I was contributing to the FFmpeg project recently. They keep their source code in a Git repo, accessed via SSL. I had an awkward error message:

SSL reported: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target

The problem was that my tool handling the SSL communication lacked the SSL certificate which validated the communication with the project. I could dismiss the error and proceed without validating the SSL security. The better solution was to supply the right SSL certificate to the communication tool, so that it could validate the SSL security with no awkwardness. Here’s how I accomplished that.  This post is offered as search engine fodder, in hopes that others will benefit from these instructions.

Problem

LiClipse is the developer tool I use for writing Python code. Based on the Eclipse IDE, it accepts numerous plugins to support other programming languages like Java and C, and related tools. One of those plugins is EGit, which interacts with the Git version control system.

When I attempted to perform a “git pull” (retrieve updated source code) from git.ffmpeg.org using EGit under LiClipse, an error dialogue would appear with the following message:

Provide information for https://git.ffmpeg.org/ffmpeg.git
A secure connection to https://git.ffmpeg.org/ffmpeg.git could not be established because the server's certificate could not be validated.
SSL reported: PKIX path building failed:  sun.security.provider.certpath.SunCertPathBuilderException: unable to  find valid certification path to requested target
Do you want to skip SSL verification for this server?
Skip SSL verification for this single git operation    [ ]
Skip SSL verification for git operations for repository /my/workspace/ffmpeg/.git    [ ]
Always skip SSL verification for this server from now on    [ ]
      [Cancel]   [OK]

(“[ ]” indicates a check box, and “[OK]” indicates a dialogue button reading, “OK”.) If I would check the box for “Skip SSL verification for this single git operation” and press the “OK” button, then the git pull would succeed, but without validating the SSL security. I didn’t want to keep checking the skip box. I also didn’t want to skip SSL verification entirely.

This problem occurred in November 2017. I was using LiClipse version 4.4.0.201711281546, running on JRE version 1.8.0_77-b03, on Mac OS X El Capitan 10.11.6. You may be using different components, which require you to take different steps to accomplish this task.

Diagnosis

So what does that error dialogue tell us? What do the references to “valid certification path” and “PKIX path building failed” mean?

First some explanation. Eclipse is mostly Java language code, and it runs on a JRE (Java Runtime Environment). “SSL” refers to a communications protocol. It was later renamed to “TLS”, but I’ll use “SSL” here. It turns out that the SSL communications is handled by the JRE: “PKIX” refers to a working group which came up with the certificate system used by SSL, and “sun.security.provider” indicates that the Java SE Security architecture is handling the communications. Part of SSL security is authenticating that, when you ask to connect to a certain internet site like “git.ffmpeg.org”, you are really reaching that site, and not an imposter. A “certificate” is a small block of data which identifies a destination name, and proves the correctness of that data through a digital signature from a service which issued that certificate. If you trust the issuer, and the signature on the certificate is valid, you can trust what the resulting certificate says. The issuing service itself has a certificate, signed in turn by its own issuing service. Certificates link to their issuers in a “chain of trust“. Some certificate, and some issuer, anchors the chain. This is known as the “root certificate”. The JRE includes copies of several root certificates, in the JRE Trust Store. It is stored in a file, within the JRE’s Java Home directory, at ./lib/security/cacerts ). The JRE also includes a tool ./bin/keytool, which adds and removes certificates from the Trust Store. But the JRE doesn’t include every root certificate which matters.

What that error dialogue tells us is that LiClipse and EGit asked the JRE’s security code to connect certificates in a chain of trust, from the certificate for the git.ffmpeg.org server, to some certificate in the JRE Trust Store. This failed, because the Trust Store lacked the necessary certificate. The solution is to add git.ffmpeg.org’s certificate to the local JRE Trust Store. The JRE’s keytool will let us add it. (If you trust that people behind the root certificate of the server have only authenticated certificates for good actors, you can instead add the root certificate from the chain of trust ending in the server’s certificate. If your stakes are high, you probably should get a better security briefing than just this blog post, before trusting that much.)

So, what we need to do is:

  1. Get the SSL certificate for our target server,  git.ffmpeg.org, or that chain’s root certificate
  2. Identify the JRE used by LiClipse, and the location of its Java Home directory
  3. Understand how to use the keytool in that JRE
  4. Locate the Trust Store in that JRE
  5. Use the keytool to add the new SSL certificate to the LiClipse JRE’s Trust Store
  6. Test that the change was effective

Get the SSL certificate

First, we get the SSL certificate for our target server, git.ffmpeg.org. Or, we can get the root “Certificate Authority” (CA) certificate from the server certificate’s chain of trust. If you get the server’s certificate, you mark just that server as trusted. If you get the CA root certificate, you can then use any server with a certificate issued directly or indirectly from that CA, which will probably be many more servers. However, maybe you don’t trust everything that every CA does. Making this tradeoff is beyond the scope of these instructions.

A straightforward way to get the target server’s SSL certificate is via the OpenSSL command-line tool.

% openssl s_client -connect git.ffmpeg.org:443 </dev/null 2>/dev/null >cert.crt

(Details: openssl s_client is a reference implementation of an SSL/TLS protocol client, which communicates correctly to servers and is helpful for diagnosis.  s_client reads HTTP commands from stdin, so the “</dev/null” notation makes stdin a null file. The “2>/dev/null” notation throws away any output to stderr, so that it doesn’t get mixed up in the regular output. The parameter of the -connect option is the host name, ‘:’, port name. Port 443 is the standard port for the https protocol. Source: a SuperUser answer.)

This stores the target server’s certificate in the file cert.crt. It will be about 2182 bytes or so in size. It will look something like this:

-----BEGIN CERTIFICATE-----
MIIGBDCCBOygAwIBAgISA/dw6A9zk496P+FouEc0W3OyMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
... [ 29 lines omitted ] ...
7U4xF6Csg3X76Xx35kIVO/JJOhoDbGIydD1Cya7dba9ZhNFa+KU1uiyu5AX+i4fd
bCXI4Ukqzwc=
-----END CERTIFICATE-----

Alternatively, you can download the target server’s certificate using the Firefox web browser (v 57 or so). Navigate to https://git.ffmpeg.org. Follow Mozilla’s instructions to view the site’s certificate.

  1. Click on the “(i)” icon at the left of the location bar, then the right-arrow, then the “Security” tab, and click the “View Certificate” button. The Certificate Viewer appears.
  2. Click on the “Details” tab.
  3. In the top pane, “Certificate Hierarchy”, click either the bottom entry for the server, or the top entry for the root.
  4. Then click the “Export…” button. A “File Save” dialogue appears, with a filename ending in “.crt”.
  5. Save the file, which contains the certificate, someplace accessible.

On Chrome and Safari, it appears that it is not possible to download the certificate itself, but you can get a text file with the certificate information, readable but not reusable.

The root CA certificate is harder to get. The -showcerts option to openssl s_client will print out the trust chain. Contrary to the documentation’s claim, this is just the certificates sent by the server. Servers usually don’t send the root CA certificate at the root of their trust chain. (See OpenSSL issue s_client -showcerts man text misleading: “all certificates in the chain”, #4933.) However, this option does print out the name on the root CA certificate. It is the issuer of the final certificate sent by the server.

% openssl s_client -connect git.ffmpeg.org:443 -showcerts </dev/null 2>/dev/null
CONNECTED(00000003)
---
Certificate chain
 0 s:/CN=ffmpeg.org
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIGBDCCBOygAwIBAgISAzCih69KsBB6DxJc3koSpgrMMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
... [ 29 lines omitted ] ...
Gi010DGiJpnEM3LIcrsokySWppACKkBCcEW3dlAL/kX+8CQrtT+ns8OtAC2RYuMF
jGjs0Nphih0=
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
... [ 29 lines omitted ] ...
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---

The root CA certificate is identified as: “/O=Digital Signature Trust Co./CN=DST Root CA X3”.

There are a couple places to look for collections of root CA certificates. Each browser includes a collection of the certificates it trusts, and it may be possible to find the file with that collection. But the easiest source I found was the certificates installed together with openssl, by MacPorts.  These are in the OpenSSL data location, a directory for which the location is compiled into OpenSSL. Look for a file named cert.pem and for a directory named cert/ .  (See  What certificate authorities does OpenSSL recognize?, by Paul Heinlein, for more information.) On my computer, it looks like:

% openssl version -d OPENSSLDIR: "/opt/local/etc/openssl"

To list the paths of all certificate files trusted by OpenSSL, you can use this command. (Source: my answer to How to list certificates, trusted by OpenSSL? on StackOverflow.)

% find -H `openssl version -d | sed -E 's/OPENSSLDIR: "([^"]*)"/\1/'`/(cert.pem|certs) \
-type f -exec ls -l {} \+
lrwxr-xr-x  1 root  admin  40 29 Nov 02:05 /opt/local/etc/openssl/cert.pem -> /opt/local/share/curl/curl-ca-bundle.crt

(You can use a different command in place of “ls -l {}” in the above if you want to do something different with the certificate files.) So, this shows that my OpenSSL installation reuses curl-ca-bundle.crt from the tool “cUrl”.)  Looking at that file, I see that it marks each certificate with the same name I saw in the /CN field from server’s certificate. This command uses grep to find and print that certificate. You can omit the -text option to leave off the text version at the beginning:

% find -H `openssl version -d | sed -E 's/OPENSSLDIR: "([^"]*)"/\1/'`/(cert.pem|certs) \
-type f -exec grep -B 1 -A 19 'DST Root CA X3' {} \+ | openssl x509 -text
Certificate:
... [ 4 lines omitted ] ...
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
        Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
        Subject Public Key Info:
... [ 45 lines omitted ] ...
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
... [ 11 lines omitted ] ...
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

The certificate text from the  “–BEGIN CERTIFICATE–” line to the “–END CERTIFICATE–” line is the root CA certificate which you could choose to install in LiClipse. To get just that part into a file, remove the -text option from the above command, and pipe the result into a file with a name like “DST Root CA X3.crt”.

Identify the LiClipse JRE

Certificates are installed in the the Java Runtime Environment (JRE) used by LiClipse. The next task is to identify this JRE, and the location of its Java Home directory.

Fortunately, the question Does LiClipse (for Mac) include its own copy of the JRE? was answered on Stack Overflow.  Yes, LiClipse for Mac does include its own JRE. The Java Home directory is at ./jre/Contents/Home within the application bundle. The ./bin subdirectory has executables, like the main java executable.

% cd /Applications/LiClipse\ 4.0.0/LiClipse.app/jre/Contents/Home
% ls -F
COPYRIGHT               THIRDPARTYLICENSEREADME.txt     man/
LICENSE                 Welcome.html                release
README                  bin/
THIRDPARTYLICENSEREADME-JAVAFX.txt* lib/
% bin/java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

The LiClipse JRE’s keytool

The Java JRE includes a command-line utility, keytool, to manage keys for that JRE. This utility is located at ./bin/keytool within the JRE. You can find out more by reading the keytool documentation. You can also run keytool using its -help option, and read the keytool man page (Use “man -M man/” to get the JRE’s man page, rather than the system’s.) And, while many web pages about keytool will assume Windows and say “keytool.exe”, on the Mac you of course use no “.exe” extension.

% cd /Applications/LiClipse\ 4.0.0/LiClipse.app/jre/Contents/Home
% man -M man/ keytool
... [man page omitted] ...
% bin/keytool -help
Key and Certificate Management Tool
... [20 lines omitted] ...
Use "keytool -command_name -help" for usage of command_name

This keytool is what we use to add the key to the Trust Store.

The LiClipse JRE’s Trust Store

A JRE will look for keys in various locations. The default is ~/.keystore , a location tied to the current user, which all JRE’s for that user can consult. There is nothing at this location by default. A fallback location is under the Java Home directory of the JRE, a file ./lib/security/cacerts .

These locations are part of a larger story about the Java Security Architecture which I won’t get into here. You can start learning it by reading the Terms section of the keytool documentation, and the Java Cryptography Architecture (JCA) Reference Guide. You will see the term “Key Store”, which is related to the “Trust Store”. Sometimes the terms are used interchangeably. For our purposes, “Trust Store” is the correct term.

The expedient choice is to modify the JRE’s local Trust Store. The path is above. The cacerts section of the keytool documentation tells us that its initial password is “changeit”. For the LiClipse JRE, that is likely still its password.

Add the new root certificate to the LiClipse JRE’s Trust Store

All that remains is to put these pieces together. Use the keytool to add the new server SSL certificate, or root CA certificate, to the LiClipse JRE’s Trust Store.

% cd /Applications/LiClipse\ 4.0.0/LiClipse.app/jre/Contents/Home
% bin/keytool -import -alias FFmpeg.org -file cert.crt -keystore lib/security/cacerts -storepass changeit
Owner: CN=ffmpeg.org
Issuer: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Serial number: 3f770e80f73938f7a3fe168b847345b73b2
Valid from: Tue Oct 31 23:22:03 PDT 2017 until: Mon Jan 29 22:22:03 PST 2018
Certificate fingerprints:
	 MD5:  61:57:B5:6B:56:08:B9:ED:E8:EC:EC:85:A9:CA:1A:F4
	 SHA1: 75:1B:86:CD:1E:34:7C:13:2F:49:E2:6D:73:A0:4F:05:09:11:7B:72
	 SHA256: EB:FA:E4:3F:CB:22:31:9F:97:7B:FA:E4:79:D6:90:7F:E0:20:3E:DA:E8:6F:C3:F0:38:55:F7:C0:1E:0D:6A:33
	 Signature algorithm name: SHA256withRSA
	 Version: 3
....
Trust this certificate? [no]:  yes
Certificate was added to keystore

The keytool invocation will be clear from reading the man page. To summarise: -import commands the keytool to add the certificate, -alias gives a title to the certificate in the store, -file gives the path of the file from which to read the certificate, -keystore gives the path to the keystore (without it, the default ~/.keystore would be used), and -storepass gives the password of that keystore. You could add the option -noprompt to cut down on the tool’s commentary, and the option -trustcacerts to stop the question about trusting the certificate, but for a use like this, I like having the cross-checking.

The same invocation will import the root CA certificate, if you prefer to do that.

Test

Next, we test that the change was effective. Attempt to perform a “git pull” from ffmpeg.org using EGit under LiClipse again.  This time, no error dialogue appears. That indicates the JRE’s Security code to connect certificates in a chain of trust, from the certificate for the git.ffmpeg.org server, to some certificate in the JRE Trust Store. (If you installed the server’s certificate in the JRE Trust Store, then this was a 1-certificate chain.)

Conclusion

Some questions lead to simple answers. Some lead to deep, but narrowly bounded, answers. This question lead to an answer which require linking several pieces of knowledge, though no part was terribly deep by itself. I hope this post provides a connected answer which helps someone else in a similar situation.

There are Q&A sites, like StackOverflow, which are excellent for the simple and the deep-but-bounded answers. When I asked this question on StackOverflow, How can I add a certificate to LiClipse to permit access to a git repo via SSL?, it got little attention, much less an answer. I was even awarded a “Tumbleweed” badge for “a question with zero score, no answers, no comments, and low views for a week.” Clearly this is not the kind of question which even that site answers easily. Fortunately, having written this post, I am well prepared to answer my own question over on StackOverflow!