Firstly, please don't take the below suggestions a criticism, we've got so much value out of the library to date that these are comparatively tiny issues.
Secondly, not using the handler also means having to re-implement all the things the handler does ( ssoSessionStore, SetConfigurationId, etc), the alternative is make the devs stop using the authN/AuthZ patterns they've become accustomed to in .NET Core. It also means I don't have keep abreast of all the changes made to the library... sort of defeats the purpose of using it otherwise.
Thirdly, the issue with adding these to the options is that these concerns may not be application scoped. Relay state and SSOOptions may be a function of the request, AuthProperties, and/or partner.
Microsoft's implementation of various Authentication handler events is a good example of allowing access to the working documents during the processing steps.
For example, with the OpenIdConnect handler, just before it's about to redirect to the identity provider, it raises a OnRedirectToIdentityProvider event with a lot of information. We can modify parts of the message before dispatch, or even take complete control.
In this specific example of the relay state, if you where to raise, say, an OnInitiateSSO event, suppying an InitiateSSOContext model, then see could modify anything relevant and allow the handler to use the updating information before dispatching to the provider. Possible properties on the context would be RelayState (read/write), SSOOptions (nullable/read/write), AuthenticationProperties (read only), PartnerName (readonly)
Given we can inject an ISamlServiceProviderEvents ( but unfortunately not in version 3.0.0 we're using ), it makes it possible to pull in other system dependencies so the Event system is really the best place for it.
At the moment I have to subclass the SamlAuthenticationHandler and reimplement SamlAuthenticationExtensions.AddSaml so I can inject an implementation of ISamlServiceProviderEvents
The same approach could be used in the SLO and Artifacts flows. I would like to add a bit more traceability for audit and logging purposes and really don't want to have to interpret the XML ( from the OnReceiveMessage event! ).
Essentially, look for all the places where the default handlers have omitted arguments and consider raising events. Subclassing all the real implementation just to tweak a few values on a methods here and there, especially when you don't have all the information at the time you need it ( i.e. AuthenticaionProperties ).