Ondrej Sevecek's English Pages

Always, always do the following three steps (details follow, I just want to stress the importance of all the commands at the very beginning) after you export your leaf certificate into leafCertificate.cer file:

certutil -urlfetch -verify leafCertificate.cer
certutil -user -urlfetch -verify leafCertificate.cer
certutil -url leafCertificate.cer

If you want to be 100% sure everything is in order, you also start command line under system account and do the same under SYSTEM and Network Service context again.

Note that you must reference the leafCertificate.cer path in an absolute path form here.
Note also that you must run the commands separately, not that you copy and paste them all at once, or run them all from a single BAT file.

psexec -s certutil -urlfetch -verify c:/temp/leafCertificate.cer

psexec -i -s certutil -url c:/temp/leafCertificate.cer

psexec -u "nt authority\networkservice" certutil -urlfetch -verify c:/temp/leafCertificate.cer

psexec -u "nt authority\networkservice" certutil -user -urlfetch -verify c:/temp/leafCertificate.cer

psexec -i -d -u "nt authority\networkservice" certutil -url c:\temp\leafCertificate.cer

Background first

With certificate based technologies and internal l PKI overall growing penetration in Microsoft based networks, it becomes more and more frequent problem to be able to completelly verify internal AD CS (Active Directory Certificate Services) CRL availability and validity. In more general terms we will talk about verifying and especially troubleshooting of verification of certificate revocation either through CRL (certificate revocation list) or by OCSP (online certificate status protocol).

The error which demonstrates these problems is:

The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613) = CRYPT_E_REVOCATION_OFFLINE

Notete: I will mainly refer to the revocation information by shorter term CRL. Certificate revocation list is the actual thing a CA produces. Clients can download the CRL and verify whether a certificate is listed or not. Because the CRL contains all revoked certificates (actually only their serial numbers, each entry taking about 90 bytes), it can be large, sometimes in order of kBs or even MBs. You can see the slight nonsense - to verify validity of a single certificate you might download several hundreds kBs. To better optimize network traffic, there is a newer standard called OCSP (online certificate status protocol). CAs can publish this HTTP web service and allow for online validation where client just sends certificate serial number and receives its status. The important point here is, that because CAs themselves publish only CRLs, the OCSP web service verifies the revocation information from CRLs as well. Phylosophically, I can thus call the revocation information simply "CRL", although I will talk about OCSP as well.

Most services require successful CRL validation to trust and use the certificate in question. Clients of quite any TLS/SSL based, IPSec based or EAP and PEAP server verify server certificate's revocation by default. Although current Internet Explorer does not block access to HTTPS web pages for which it cannot verify revocation, it at least displays warning dialog box and yellow address bar. In case of IPSec client, the default is also to verify, but allow IKE establishment even if no CRL is available. Other require CRL validation to allow the certificate use at all, although you can usually disable certificate revocation in registry.

CRL is verified for digitally signed executable files and scripts, digitally signed documents or signed and encrypted mail certificates, as well as for client EFS encryption and recovery certificates as well as for BitLocker recovery certificates.

If you use client certificates for authentication to some TLS/SSL/EAP/PEAP or Kerberos services, the server part of the channel verifies CRL of client certificate as well. In these cases, we have CRL validation on both sides - on the client against validity of the server certificate, and on the server side against validity of the client certificate.

Anyway, all applications at least try to check CRL. If not available, it may result in unpleasant timeouts and delays in session establishement. What we want is smooth online CRL and/or OCSP availability.

Note that both CRL and OCSP responses may be (and are usually) cached on computers that perform the validation. If a valid response is found in local cache, most services will not go to network again. Note the term "valid response" and "most services". If the response expires or in case of some services (such as EAP/PEAP client or IPHTTPS), validation is always done online. Thus we still need smooth online validation, no talking about it.

How leaf certificates contain CRL and OCSP paths

Usual certificate hierarchy includes some root CA, may be several intermediate CAs, always one issuing CA (which may be identical to the root CA in case of a single CA path) and finally the end-point leaf certificate. The leaf certificate (also endpoint or end-entity certificate) is the certificate which web servers use, which are loaded into smart cards for user logon, they are those that you use to sing an email or document etc.

The leaf certificate is always what we will start with when checking revocation. Common mistake is to start with some CA certificate, in the worst case, with the root CA.

Note: how do you distinguish between a root CA certificate and any sub-certificate? Root CA certificate contains the same value in its Subject and Issuer fields and thus is also called self-signed certificate. In any other sub-certificate, the two Subject and Issuer fields contain different values.

We start with the leaf certificates. Always. And we need to verify all CRL and OCSP paths which are found in all the certificates in the certifice hierarchy starting with the leaf certificate and proceeding through all upper CA certificates. Except for the root CA certificate. Although the root CA certificate may contain CRL and/or OCSP paths, they have no sense in root certificates and are never verified. If you see any error with CRL or OCSP download at the root certificate level, you may usually ignore it.

If you look at a certificate (remember to look into the leaf certificate), you will find CRL paths in the CDP extension (CRL distribution point extension). You may also find the OCSP path in AIA extension (authority information access extension).

HTTP and LDAP paths and authentication

Although there are more options, they are seldom used. CRLs can be available at HTTP paths and at LDAP paths, which is also the default for internal AD CS deployments. OCSP has only one transport, HTTP.

LDAP path will usually require client to be authenticated with a domain account, while HTTP may be configured to provide the information anonymously.

Note: both CRLs and OCSP responses are digitally signed. It means that it is not a security issue to offer them without authentication, without any further encryption (such as HTTPS) etc. The contents of CRLs and OCSP responses is also generally considered public. They do not contain any information that would compromise privacy. Both work solely with serial numbers of certificate and do not publicise not even the revoked certificates in all.

To download CRL from an authentication LDAP location, the client must be either domain user or domain member machine and must be able to authenticate with its DCs with either Kerberos or NTLM, which may not always be possible. For example, if the client is in the internet, it will not usually have DCs available. For instance, if you also use smart card logon or 802.1x, the client might not be authenticable yet before he actually authenticates with the authentication method :-)

From this point, HTTP is usually better. There are two important things to implement and verify with HTTP CRL and OCSP publishing:

• make the CRL and OCSP work anonymously
• ensure all certificates contain publicly resolvable FQDN paths that point to the distribution locations

There is another problem with LDAP. Replication latency. AD CS can publish CRLs to Active Directory LDAP. It does so by default. You may have several Active Directory DCs, regionally distributed and with long replication delays and high latency. Your AD CS server publishes CRL to just a single DC from which the CRLs must replicate to other DCs. On the other hand, clients attempt to download CRLs from their local DC only.

This may produce chaotic, random and latent revocation validation errors with LDAP distribution. Even from this point, it is better to implement at least one HTTP distribution location, even if it is not the primary location that would be placed in the certificate

In order to download from HTTP, client machine or user profile can be configured with HTTP proxy. Even HTTP proxies may require authentication! They may require authentication even for machines not only for users. Operating system may have different HTTP proxy setting than a user. You verify user's proxy setting in Internet Explorer. You can also verify machine proxy settings with NETSH WINHTTP SHOW PROXY since Windows Vista or with PROXYCFG with on Windows 2003 or Windows XP and older.

Our tools

CRL is downloaded as normal files from HTTP. If the CRL path is HTTP, you can always try Internet Explorer and just download the file. But we need some better tool.

The best tool for this chore is CERTUTIL. CERTUTIL is available since Windows Vista in-box with the operating system. For Windows 2003 and Windows XP, you must install it as part of the Administrative (Administration) Tools Pack (adminpak.msi). Do not copy it from a newer edition - it may not work as expected, one issue may be found in the following article.

Sometimes, you will have to go even deeper. The you can download Microsoft Network Monitor and see what happens on the wire.

CERTUTIL and our -URLFETCH -VERIFY and -URL switches

The tool has two command line switches - one is urlfetch verify, the other is url. The urlfetch verify tool displays a detailed output log which may be very good for troubleshooting, but may be unnecessarily complex for novices. I also like the url tool which displays a nice GUI dialog box and allows you to retry downloads. This is handy in case you are debugging some firewall or other communication path and need to attempt the downloads recurrently.

It is also easier to trigger CRL or OCSP download with the url switch when you troubleshoot with Network Monitor, because it does not download revocation for all the CA certificates in the certificate paths. The urlfetch verify switch on the other hand verifies all revocation from the whole certificate path.

Note: although I reffer to it as "urlfetch verify" switch, they are actually two switches. If you used just the -verify switch, CERTUTIL would not download any response which it would find in local cache. Because of this, always run CERTUTIL with both the -urlfetch and -verify switches.

Both swtiches (the url and the urlfetch verify) also differ in HTTP libraries they use.

CERTUTIL and the -USER switch

The tool has yet another command line switch. The -user switch. It instructs the tool to use user registry, certificate stores and response caches when validating paths, CRL and OCSP responses and certificates.

Both machine and user profile contain separate certificate and CA stores. They also contain separate CRL and OCSP caches. As contents of machine and user certificate stores and caches may differ, you always do the following three checks and verify all their results.

First step is to export the leaf certificate. Once again. The leaf certificate, not any CA certificate from the chain, not especially the root CA certificate. Then run the following:

certutil -urlfetch -verify leafCertificate.cer
certutil -user -urlfetch -verify leafCertificate.cerercertutil -url leafCertificate.cer

SYSTEM context

It is important to understand that the previous discussion assumed you were working under the exact context of a user identity which experiences any troubles. There are many system components that work with certificates under SYSTEM or Network Service identities and perform certificate validation on their own. There are circumstances in which user's validation may work well while it may fail for the system identities.

Such services include VPN clients such as SSTP, L2TP, IKEv2, or IPv6 tunnel called IPHTTPS or IPSec based communications. The network authentication 802.1x also verifies certificates under SYSTEM account regardless you authenticate with a user and not machine certificate. It comes as an even more logical fact in case a server component verifies client certificates.

In such situations, you might not be able to verify everything completelly without running the test under SYSTEM and Network Service accounts as well.

Note that you must reference the leafCertificate.cer path in an absolute path form here.
Note also that you must run the commands separately, not that you copy and paste them all at once, or run them all from a single BAT file.

psexec -s certutil -urlfetch -verify c:/temp/leafCertificate.cer

psexec -i -s certutil -url c:/temp/leafCertificate.cer

psexec -u "nt authority\networkservice" certutil -urlfetch -verify c:/temp/leafCertificate.cer

psexec -u "nt authority\networkservice" certutil -user -urlfetch -verify c:/temp/leafCertificate.cer

psexec -i -d -u "nt authority\networkservice" certutil -url c:\temp\leafCertificate.cer

Common issues - HTTP CRL download error 12029 caused by invalid proxy

As one the most common examples - you may experience the following download error in case of HTTP CRL publishing:

Error retrieving URL: A connection with the server could not be established 0x80072efd (WIN32: 12029)
Error 12029, 0x2efd: ERROR_WINHTTP_CANNOT_CONNECT
Error 12029, 0x2efd: ERROR_INTERNET_CANNOT_CONNECT

This error may be caused by various connectivity issues, timeouts or firewalls blocking the connections. Often, it is problem of incorrectly configured HTTP proxy, or proxy not being configured at all. The issue often exists only for local system trying to download CRL while the CRL download works fine for user applications.

There are actually two distinct proxy configurations for WINHTTP (or WININET) libraries. Windows components, .NET framework and also various third party Windows-based applications use WININET API to access HTTP services. CRL validation is no exception. The API once came with Internet Explorer, but since the very times of Windows NT is an integral part of operating system distribution. It has two separate proxy configurations.

WININET proxy configuration for regular user accounts:

• these proxy settings work for user induced connections only. Operating system components running under SYSTEM, Network Service, Local Service or the various NT SERVICE or IIS APPPOOL virtual accounts do not use the user proxy setting.
• user proxy configuration is strictly on per-user bases and may differ among various user accounts on the same computer.
• supports static proxy setting or autoconfiguration (web proxy autoconfiguration) with DNS and DHCP discovery or static WPAD scripts.
• supports single-sign-on (SSO) user authentication with Windows Integrated authentication using default user credentials (those that user has used to log on to the operating system interactively) or with saved credentials. You can display list of saved credentials with one of the following command line commands:
  control keymgr.dll
  rundll32.exe keymgr.dll, KRShowKeyMgr
• you configure proxy settings manually using Internet Explorer's control panel Internet Settings, Connections tab. The proxy settings may aslo come from a Group Policy Object (GPO)
• Some GPO settings are enforced (Administrative Templates), which means that you cannot change the setting locally using the Internet Settings control panel, while some GPO settings (Internet Explorer Maintainance or Group Policy Preferences) apply only at the time when user logs. In that case users can change them temporarily. If you want to see Group Policy results on a particular computer, you can do so with one of the following commands:
  gpresult /h report.htm
  rsop.msc

WININET proxy configuration for system components:

• system components which run under SYSTEM, Network Service, Local Service or the various NT SERVICE or IIS APPPOOL virtual accounts have their own proxy configuration which is separate from that of individual user accounts.
• system proxy settings DO NOT support WPAD scripts nor web proxy autoconfiguration with DNS or DHCP discovery.
• system proxy settings cannot be set with Group Policy, unless you used a startup stript or a scheduled task or some System Center Configuration Manager (SCCM) application package
• system can authenticate agains its proxy with Windows Integrated authentication and its AD computer account if it is member of a domain.
• you can display current system WININET proxy settings from command line with the following commands on Windows XP/2003 or Windows Vista and newer respectively:
  proxycfg
  netsh winhttp show proxy
• you can change the proxy settings with the same commands on Windows XP/2003 and Windows Vista and newer respectively:
  proxycfg -p to set a static proxy
  proxycfg -d to delete proxy setting and access HTTP directly
  netsh winhttp set proxy to set a static proxy
  netsh winhttp reset proxy to delete proxy setting and access HTTP directly

If you need to troubleshoot further and were not able to assess or resolve the issue with proxy settings, you can use Microsoft Network Monitor to look at the actual packet traffic on wire.

Common issues - HTTP CRL download failures due to various HTTP errors

Similar error to that of the previously mentioned error 12029 can be found and translated with the error lookup tool:

Error 12007, 0x2ee7, 0x80072ee7: ERROR_WINHTTP_NAME_NOT_RESOLVED

The error is quite self-descriptive - the machine cannot resolve DNS name of the target CRL publication URL. Note that is is wise to put only FQDN publicly resolvable DNS names in CRL distribution point extension (CDP).

Error 12178, 0x2f92, 0x80072f92: ERROR_WINHTTP_AUTO_PROXY_SERVICE_ERROR
Error 12166, 0x2f86, 0x80072f86: ERROR_WINHTTP_BAD_AUTO_PROXY_SCRIPT
Error 12167, 0x2f87, 0x80072f87: ERROR_WINHTTP_UNABLE_TO_DOWNLOAD_SCRIPT
Error 12180, 0x2f94, 0x80072f94: ERROR_WINHTTP_AUTODETECTION_FAILED

The previous errors may appear only for user invoked WININET (also known WINHTTP) connections which support web proxy autodiscovery (autodetection) with DNS or DHCP discovery or with static WPAD proxy scripts. In order to resolve the errors, you should either correct the problem with your wpad autodiscovery or change proxy settings to static.

Error 12015, 0x2eef, 0x80072eef: ERROR_WINHTTP_LOGIN_FAILURE
Error 12044, 0x2f0c, 0x80072f0c: ERROR_WINHTTP_CLIENT_AUTH_CERT_NEEDED

These error may indicate authentication failure either at local HTTP proxy or more frequently at the target HTTP web server which hosts the CRL. Generally, it is better to not require any authentication at the CRL distribution URLs. CRLs are digitally signed and also contain no private information so that you do not risk much exposing them to unauthenticated public access.