In Part 1 we mentioned that a lot of tools handle HTTP SSL with different results than OpenSSL tools. Now we’ll review our findings in hopes that others will learn from our mistakes.
CA bundles
Before we talk about the tooling, let’s say a few words about CA bundles, as they played a big role in our issue. And again, some tools handle CA bundles differently.
What is an CA Bundle in the first place? A CA bundle is a file that contains root and intermediate certificates. The end-entity certificate along with a CA bundle constitutes the certificate chain.
We had an old script for creating such a bundle which downloaded all intermediate certificates from the internet and then built a bundle. But after changes in chain it was not constructing the full chain and some certificates were missing.
This new bundle got through our validation because we were only checking the expiration date of the final certificate. But the HTTP proxy was unable to use this bundle and used a self-signed certificate instead.
Python and requests (pip) lesson learned
First let’s talk about Python, its package manager pip, and the library requests. We found out that pip uses requests internally to do HTTP related work. That doesn’t sound bad, but we’ve discovered that the requests library comes with pre-packaged root SSL certificates and ignores the system certificates.
This was the reason why using host’s root certificates in CI fixed OpenSSL but did not fix all the jobs from failing.
We found out that requests using its own pre-packaged SSL certificates can be disabled by using the environment variable REQUESTS_CA_BUNDLE. Although I would not recommend blindly disabling it everywhere. I think it is better for our CI as we take care only for system SSL certificates from now on. Read more about this topic in requests documentation.
We set the environment variable for all CI jobs like this:
REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
Where /etc/ssl/certs/ is mounted from the host into the CI Docker container.
OpenSSL
OpenSSL and its openssl command is the one which from our experience will always use /etc/ssl/ storage. Beware when updating root certificates you have to run the command update-ca-certificates to build a root certificates bundle which is then being used.
To check what certificate and whether is it valid on HTTP sites you can use the following command:
openssl s_client -connect <host>:443 -servername example.com
The option -servername is required only when you want to target one particular server with <host> address.
You will then get the following output:
CONNECTED(00000003)
depth=3 C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
verify return:1
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = <your domain>
verify return:1
---
Certificate chain
0 s:/CN=<your domain>
i:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
3 s:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
4 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=/CN=<your domain>
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 7952 bytes and written 451 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 08CAF4005E2A289B7119C614A0C5B347A6D22B114F46B11142FB278618795423
Session-ID-ctx:
Master-Key: 565403F2C40E946DE08C59FD6A647E5156C55C93BFEE04E70D6543A8CB750C39D0CD326670BD4D85BD9590BB15935999
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1595254569
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed
Where you can see:
- After CONNECTED(00000003) your certificate chain. You have to watch carefully for verify return:1 so you know that certificate in a chain is valid. (OpenSSL will exit with a success code if end certificate is valid but one of the chain certificate is not)
- Just before the HTTP response Verify return code: 0 (ok) which means that end certificate is valid.
To validate a SSL certificate bundle the same way our proxy server does, we can use following command:
openssl verify -CApath /etc/ssl/certs/ <CABundle.pem>
Where CABundle.pem is the bundle file you are then uploading to your servers. Just beware that when some certificates are valid in OpenSSL it does not mean that other tooling won’t validate them differently.
Curl
Curl is a great tool for working with SSL on HTTP. To check if a server returns a valid certificate you can use following command:
curl -sv --resolve example.com:443:192.168.0.1 https://example.com
Beware when you want to target one particular server you have to use --resolve option.
Which will output for example:
curl -sv --resolve example.com:443:192.168.0.1 https://example.com * Added example.com:443:192.168.0.1 to DNS cache * Rebuilt URL to: https://example.com/ * Hostname example.com was found in DNS cache * Trying 192.168.0.1... * Connected to example.com (192.168.0.1) port 443 (#0) * found 128 certificates in /etc/ssl/certs/ca-certificates.crt * found 517 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * SSL: certificate subject name (<your domain>) does not match target host name 'example.com' * Closing connection 0
From our experience it is important to test it with Curl as well as with OpenSSL as those two tools’ results can be different.
What we have found is that Curl can use a different SSL root certificate storage. Which can be checked by the following command:
curl-config --ca
Some Docker images have Curl compiled differently then we are used to. You can always check how your Curl was compiled by the following command:
curl-config --config
For more information on SSL and Curl you can refer to this article: https://curl.haxx.se/docs/sslcerts.html
Summary
We are aware that a lot of applications/packages use, for example, Python’s requests, libcurl, libssl inside and all of these work a bit differently.
We now have a properly documented script for creating an SSL certificate bundle with proper validation (which works the same as validation on our proxy servers). Where the script looks like this:
#!/bin/bash # Build a certificate bundle. Run this script and feed it with a ZIP file from, e.g. Comodo. # # Usage: # ./build.sh # # The file `STAR_intranet_ourdomain_com.zip` must exist. set -ex unzip STAR_intranet_ourdomain_com.zip # Contains these two files # 4135 Stored 4135 0% 2019-03-12 00:00 440106c0 STAR_intranet_ourdomain_com.ca-bundle # 2065 Stored 2065 0% 2020-06-02 00:00 711c38d2 STAR_intranet_ourdomain_com.crt mv STAR_intranet_ourdomain_com.ca-bundle CABundle.pem mv STAR_intranet_ourdomain_com.crt \*.intranet.ourdomain.com.crt cat \*.intranet.ourdomain.com.crt CABundle.pem > \*.intranet.ourdomain.com.crt-bundle # The following commands will check that the created bundle is valid, # in the same way as the ingress proxy checks it. openssl verify -CApath /etc/ssl/certs/ CABundle.pem openssl verify -CAfile CABundle.pem \*.intranet.ourdomain.com.crt-bundle set +ex
