What? Are they mad? The Exchange 2013 OWA (Outlook Web App or Outlook Web Access) does not support CNG certificates?
It has been nearly eight years since 2006 when the new CNG (Cryptography Next Generation) was first introduced in Windows Vista with the goal to be the primary cryptography interface which should asap replace the older CryptoAPI and its CSP (Cryptographic Service Provider) modules.
Since then, not even SQL Server 2012 R2 supports the CNG certificates yet. Not that I would ask them to support ECDH certificates. But they do not support RSA certificates in CNG storage either.
Today, I have found that not even Exchange Server 2013 OWA web application supports RSA certificates stored in CNG's Microsoft Software Key Storage Provider. Although IIS supports the new type of private key storage provider since its very beginning in Windows Vista and Windows 2008, and although you definitely can call Enable-ExchangeCertificate -Services IIS with a CNG certificate thumbprint, you will not be able to connect to OWA nor ECP afterwards.
Although the certificate gets bound to the IIS web site well, and although client browser can connect to the HTTPS without problems, the logon screen just does not let you further and pops up again with the same password prompt like if you didn't provide correct passoword (although it does not even tell you the password is wrong - it just restarts plain).
You can also observe the following error event in Application event log of the OWA server (an Invalid provider type specified error):
Event log: Application
Event source: MSExchange Front End HTTP Proxy
Task category: Core
Event ID: 3
Event type: error
Message: [Owa] An internal server error occurred. The unhandled exception was: System.Security.Cryptography.CryptographicException: Invalid provider type specified.
at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
at Microsoft.Exchange.HttpProxy.FbaModule.ParseCadataCookies(HttpApplication httpApplication)
at Microsoft.Exchange.HttpProxy.FbaModule.OnBeginRequestInternal(HttpApplication httpApplication)
at Microsoft.Exchange.HttpProxy.ProxyModule.<>c__DisplayClassa.b__9()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(TryDelegate tryDelegate, FilterDelegate filterDelegate, CatchDelegate catchDelegate)
The reason lies probably in the constructor of the RSACryptoServiceProvider class which accepts the CspParameters parameter. This CspParameter object has another constructor with providerType parameter. They send some into it a value which is invalid in regard to the Microsoft Software Key Storage Provider.
How to determine it is really your case? How to find out which cryptographic provider actually stores the private key for your certificate?
Go to the command line on the affected Exchange server and run the following command:
CERTUTIL -repairstore my *
What you should see in its output will be the Subject, Serial Number, Issuer and Cert Hash (the thumbprint) of all certificates stored in the local computer certificate store which have an associated private key. For each certificate, you will find a field named Provider which actually contains the name of the cryptographic provider which stores the private key.
The error appears if the certificate's private key is stored by the Microsoft Software Key Storage Provider which is actually one of the "new" CNG providers.
Solution?
Make sure the Provider used to store the certificate private key is either the Microsoft RSA SChannel Cryptographic Provider or the Microsoft Software Key Storage Provider, which I have both tested successfully and I can say that the same certificate works if its private key is stored in these old CSP providers instead of the CNG.
In case you issue the certificate from AD CS, you need to configure and publish a certificate template such that it is either compatible with Windows XP and Windows 2003 clients (meaning template version 1 or template version 2) or that uses the so called Legacy Cryptographig Service Provider category on the Cryptography tab in case you are dealing with AD CS on Windows 2012 and newer.
If you selected the provider category to be Key Storage Provider instead, the private key would go into CNG storage and the Exchange OWA malfunction would appear.
If you create (offline) custom certificate requests from the Certificates MMC console to obtain a public certificate by uploading the textual certificate REQ file to a public certification authority (CA), select the (no template) Legacy key option.
If you already have the certificate issued with a private key in CNG storage, the simples way would be to recreate the certificate request from scratch with the same parameters except for the private key storage provider. You cannot just resend the original request, because its private key would have already been bound to the CNG storage since the time of the request creation.