Ive been struggling to get an application running on z/OS to use TLS with certificate authentication.

Now it is working, I despair at how the TLS has been implemented, and it feels that the development did not understand how the product would be used; or may be TLS has moved on since the product was initially written. Java on the other hand seems to have all the facilities for a good implementation, but they are not always used, resulting in a poor end user experience. It feels that the designers only expected one, or only a few users.

Experience shows…

The first time you do a journey, you are pleased if you get to the destination with no major problems. Once you have done the journey a couple of times you know the best options and what to avoid. For example when your plane arrives at an airport, and there is shuttle train to get you to the main terminal and immigration, if the escalator at the main terminal is near the front of the train – then get on at the front of the train, and you will be ahead of the hundreds of people from the train.

I learned when in a performance role, that it takes three attempts to write a performance report. After the first attempt you understand the real question you should be asking. The second attempt you get answers to these questions. The third attempt you try to add value and explanation. Many a time I’ve looked at someone else’s performance report and asked myself “what does this mean, how do I use this information?”. I spent a week doing some tests, and the outcome was a sentence “This performance tuning option has very little effect. Just leave it as the default”. This is so much more useful than a chart showing a flat line.

In the same way, a developer designing a program which uses TLS may be very happy having got one client to talk to the server using TLS. You need the experience to understand that there are different certificate types (Elliptic Curve, RSA), and different key sizes. It is easier to administer the CA certificates, if you have one enterprise wide keyring for them, and every one uses the same keyring; rather than one keyring for each server instance.

Where are certificate stored?

There can be two types of store for certificates:

• Keystore – this contains the certificates each with its private key. Each group of servers may have its own keystore, with just the private keys it needs. So one key store for production CICS regions, one for test CICS regions, another key store for internal web servers.
• Trust store. This contains the public keys of Certificate Authority certificate authorities, and any self-signed certificate from clients. The trust store is used to validate certificates which have been sent to the application. You could have one trust store for the whole LPAR, and every one uses it.

In many TLS implementations only one store is used, called the keystore, and has both the Key store and Trust store keys in it. This is easy to implement, but not a good design.

High level overview of the TLS handshake

The handshake

• The client initiates the handshake, passing up information like I understand the following cipher specs…; I understand certificates created using Elliptic Curves (or RSA) (or both).
• The server responds with ,
  • use this cipher spec…,
  • optionally (usually) here is my certificate which matches your criteria (Elliptic Curve or RSA)
  • optionally – please send your certificate which matches these criteria… here are the CA’s I know about
• The client checks the server’s certificate is valid either because the certificate is in the trust store , or the certificate for the Certificate Authority is in the trust store, and can be used to validate the certificate.
• If asked, the client picks a certificate from the key store and sends it. It should pick a certificate with a CA from the list which was sent down.
• The server checks the client’s certificate is valid either because the certificate is in the trust store or the certificate for the Certificate Authority is in the trust store which validates the certificate.

Which certificate does the server send?

There are several implementations used in different products:

• Take the first valid certificate in the ring. Easy to code, but causes problems. If I replace the certificate, by removing and readding it, then a different certificate may be used, as what was the second certificate in the keyring, will now be the first in the keyring, and this will be used instead. This may have a different type of certificate (Elliptic Curve or RSA), and clients may then fail to connect.
  • If the certificate has expired it will not be used, even though it looks the first.
• Specify the default. There is one default. If you remove and add a certificate you can specify which certificate becomes the default. Not all clients may like the certificate type, so you may have to change the client certificate to match.
• Pick the best option. Java has an function which says give me the list of certificate in the keyring which match the following, this might be EC or RSA or some other option. With this, different clients can get a different server certificate.

You could have one keystore for all your servers, or a keystore for CICS, and other for a Liberty web server, another for MQ web server.

I think that the keystore should contain only the keys for the certificates it needs to use. Do not have the CICS, Liberty, MQ certificate all in the same keystore if the certificates are all different. It would allow CICS to use the MQ certificate.

For added security certificates can have the name of the server, or the IP address of the server, and the client can check it matches the server’s host. You might not want to share a keyring between different z/OS images, because they may have a different IP address.

Client – picking the private certificate from the keystore

• Java has chooseClientAlias(keyType[], issuers[]...) and chooseServerAlias(keyType[], issuers[]...) which does all the work for you. You could still decide to pick the first or default certificate.
• If you use z/OS System SSL it looks like you have to loop through the key store yourself to find the best one with gsk_get_certificate_info() and gsk_get_certificate_algorithms(). I can see why it is easier to pick the first, or the default certificate.

What should I use for my trust store?

The trust store has the certificates for all CAs you expect to use. If you start working with a different company you might need to add their CA to your trust store.
If you have one trust store for all users across the z/OS image, you add the certificate to it, and every one has access to it – easy.

If you have multiple trust stores, you’ll need to add the certificate to all the trust stores that need it. This may be a small number, for example, production and test.

If you use your keystore as a trust store then you will have to add the new CA to every trust store.

As certificate expire you will need to replace them. The amount of work depends on how many stores you have to update. Having one trust store makes this easier.

Why have I told you this?

I had a client server working using the DEFAULT certificate on z/OS. I wanted to use this keyring for a different application, which used the FIRST certificate in the keyring. This was a special-case certificate, and not to be used by most clients. I had to remove and re-add certificates in the keyring so the default certificate became the first certificate. It took over a day to identify the problem and fix it!