Category: Private Key

Understanding a Wireshark TLS handshake trace

I’ve done a lot of work using TLS, and Wireshark is a great tool for displaying the flows of data. The problem is understanding what the output shows! This blog post shows what to look at.

Note: Flows with GREASE values should be ignored. This is designed to make sure the remote end can handle unexpected values, by sending up unsupported values.

Summary of the flows of the handshake

See here for an overview.

  • Client to Server: Client Hello
  • Server to Client: Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
  • Client to Server: Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
  • Server to Client: Change Cipher Spec
  • Server to Client: Finished

Client to Server: Client Hello

This displays

Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Version: TLS 1.2 (0x0303)
            Random: c5ec1cf320f2f77ba6f3f84e7fd7d1f946400484a56918a5e499b0e98be2a07f
            Session ID Length: 32
            Session ID: a839f3fcd1d7f4394c6e5bd0279da5c75458b8a81f35c5890107007cc921fa7b
            Cipher Suites Length: 32
            ▶Cipher Suites (16 suites)
            Compression Methods Length: 1
            ▶Compression Methods (1 method)j             
            Extensions Length: 429
            ▶Extension: signature_algorithms 
            Extension: Reserved (GREASE) (len=0)
            ▶Extension: supported_versions (len=7)
            ...

If you expand supported versions you get

 Extension: supported_versions (len=7)
       ...
       Supported Version: TLS 1.3 (0x0304)
       Supported Version: TLS 1.2 (0x0303)

Which shows you the TLS versions that the client supports.

If you expand Cipher Specs you get

Cipher Suites (16 suites)
   Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
   Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
   Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
   Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
   Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
   ...

This is the list of cipher specs in preference order, that the client supports.

The signature_algorithms defines which signature and hash algorithms are supported for the server’s digital certificate.

Server to Client: Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done

The server sends down several blocks of data. These are logically distinct blocks of data – but tend to arrive in one block.

  • Server Hello, Here are some parameters for the handshake
  • Certificate, Here is my certificate(s)
  • Server Key Exchange,
  • Certificate Request, Please send me a certificate matching the enclosed parameters
  • Server Hello Done

Server to Client: Server Hello 

Handshake Protocol: Server Hello
  Handshake Type: Server Hello (2)
  Version: TLS 1.2 (0x0303)
  Random: ...
  Session ID Length: 32
  Session ID: ...
  Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
  ...

It shows which version of TLS has been picked.

The Cipher Suite is the value picked by the server.

Server to Client: Certificate

This is present if the server is configured to send down the server’s certificate.

Handshake Protocol: Certificate
  Handshake Type: Certificate (11)
     ...
     Certificates Length: 1608
       Certificates (1608 bytes)
          Certificate Length: 734
          ▼Certificate: .. (commonName=10.1.1.2,...organizationName=NISTECCTEST)
             ▶signedCertificate
                serialNumber: 0x7a
                ▶signature (sha256WithRSAEncryption)
                ▶issuer: ...
                ▶validity ...
                ▼subject: rdnSequence (0)
                   RDNSequence (organizationName=NISTECCTEST)
                   RDNSequence (organizationalUnitName=SSS)
                   RDNSequence (commonName=10.1.1.2)
                ▶subjectPublicKeyInfo
                ▶extensions: 5 items
                algorithmIdentifier (sha256WithRSAEncryption)
                    ...
          Certificate Length: 868
          ▶Certificate:  ...

From this we can see the certificates sent down from the server.

  • The certificate name is specified at many levels; including on the Certificate and subject.
  • The CA (issuer) which signed each certificate.
  • The validity dates of the certificate
  • The extensions, which include how the certificate can be used; key usage, Subject Alternative Name ( IP address) etc.

Check this is what you are expecting, and that the CA is in your trust store.

Server to Client: Server Key Exchange

I have never used the information in this area

Server to Client: Certificate Request

Handshake Protocol: Certificate Request
   Certificate types count: 3
      ▼Certificate types (3 types)
         Certificate type: RSA Sign (1)
         Certificate type: ECDSA Sign (64)
         Certificate type: DSS Sign (2)
      ▼Signature Hash Algorithms Length: 26
         Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
         Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
         ... 
      ▼Distinguished Names Length: 246
         Distinguished Names (246 bytes)
             Distinguished Name: (commonName=COLIN4Certification Authority,...)
             Distinguished Name: (commonName=DocZosCA,...)
             Distinguished Name: (commonName=SSCA256...)

This is saying please send me a certificate:

  • The certificate must be one of certificate-type elements; RSA, ECDSA or DSS,
  • and signed by one of the algorithms listed,
  • and have a CA signer of COLIN4Certification Authority…, DocZosCA… or SSCA256… .

The CA signers are taken from the server’s trust store.

Server to Client: Server Hello Done

That’s the server side of the work done… almost

Client to Server: Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished

Client to Server: Certificate

The client needs to pick a certificate from those in its key store. Depending on the application

  • a browser may display a list of valid certificates for the end user to select
  • a program may use the information sent down in the handshake to pick the best certificate
  • a program may return what the configuration specified (such as default)
  • a program may just return the first certificate in the key file.
Handshake Protocol: Certificate
   Handshake Type: Certificate (11)
   Certificates Length: 683
   ▼Certificates (683 bytes)
     Certificate Length: 680
     ▼Certificate: … (commonName=docec384,...countryName=GB)
       ▼signedCertificate
          serialNumber: 0x029b
          ▶signature (ecdsa-with-SHA384)
          ▼issuer: rdnSequence (0)
          ▶rdnSequence: 4 items (commonName=SSCA256,OU=CA,iO=DOC,C=GB)
            ....
          ▶validity...
          ▼subject: rdnSequence (0)
              ▶RDNSequence item: 1 item (countryName=GB)
              ▶RDNSequence item: 1 item (organizationName=Doc2)
              ▶RDNSequence item: 1 item (commonName=docec384)
          ▶subjectPublicKeyInfo
          ▶extensions: 6 items...
          algorithmIdentifier (ecdsa-with-SHA384)
          ...

We can see

  • the algorithm identifier is ecdsa-with-SHA384 which is OK
  • the issuer is CN=SSA256,ou=CA,C=GB which is in the list of CAs passed from the server.
  • the signature is ecdsa – which is OK

Client to Server: Client Key Exchange

I haven’t used this. If the handshake gets this far, the set-up is good.

Client to Server: Certificate Verify

I have never used this. It is used to send up some data encrypted with the client’s private key, to check the server can decrypt it.

Client to Server: Change Cipher Spec

Response to the server. I have never used this. Business as usual, it periodically changes the Cipher Spec

Finished

Server to Client: Change Cipher Spec

Each direction has its own cipher spec. This is the server saying to the client … ok… I’m using the new cipher spec.

Server to Client: Finished

The final – final completion.

Problems

When an invalid CA was used at the client end, the server sent down

Transport Layer Security
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Unknown CA)
        Content Type: Alert (21)
        Version: TLS 1.2 (0x0303)
        Alert Message
            Level: Fatal (2)
            Description: Unknown CA (48)

This has a list of the codes and their meaning.

Some implementations of TLS are good, some are not so good.

Ive been struggling to get an application running on z/OS to use TLS with certificate authentication.

Now it is working, I despair at how the TLS has been implemented, and it feels that the development did not understand how the product would be used; or may be TLS has moved on since the product was initially written. Java on the other hand seems to have all the facilities for a good implementation, but they are not always used, resulting in a poor end user experience. It feels that the designers only expected one, or only a few users.

Experience shows…

The first time you do a journey, you are pleased if you get to the destination with no major problems. Once you have done the journey a couple of times you know the best options and what to avoid. For example when your plane arrives at an airport, and there is shuttle train to get you to the main terminal and immigration, if the escalator at the main terminal is near the front of the train – then get on at the front of the train, and you will be ahead of the hundreds of people from the train.

I learned when in a performance role, that it takes three attempts to write a performance report. After the first attempt you understand the real question you should be asking. The second attempt you get answers to these questions. The third attempt you try to add value and explanation. Many a time I’ve looked at someone else’s performance report and asked myself “what does this mean, how do I use this information?”. I spent a week doing some tests, and the outcome was a sentence “This performance tuning option has very little effect. Just leave it as the default”. This is so much more useful than a chart showing a flat line.

In the same way, a developer designing a program which uses TLS may be very happy having got one client to talk to the server using TLS. You need the experience to understand that there are different certificate types (Elliptic Curve, RSA), and different key sizes. It is easier to administer the CA certificates, if you have one enterprise wide keyring for them, and every one uses the same keyring; rather than one keyring for each server instance.

Where are certificate stored?

There can be two types of store for certificates:

  • Keystore – this contains the certificates each with its private key. Each group of servers may have its own keystore, with just the private keys it needs. So one key store for production CICS regions, one for test CICS regions, another key store for internal web servers.
  • Trust store. This contains the public keys of Certificate Authority certificate authorities, and any self-signed certificate from clients. The trust store is used to validate certificates which have been sent to the application. You could have one trust store for the whole LPAR, and every one uses it.

In many TLS implementations only one store is used, called the keystore, and has both the Key store and Trust store keys in it. This is easy to implement, but not a good design.

High level overview of the TLS handshake

The handshake

  • The client initiates the handshake, passing up information like I understand the following cipher specs…; I understand certificates created using Elliptic Curves (or RSA) (or both).
  • The server responds with ,
    • use this cipher spec…,
    • optionally (usually) here is my certificate which matches your criteria (Elliptic Curve or RSA)
    • optionally – please send your certificate which matches these criteria… here are the CA’s I know about
  • The client checks the server’s certificate is valid either because the certificate is in the trust store , or the certificate for the Certificate Authority is in the trust store, and can be used to validate the certificate.
  • If asked, the client picks a certificate from the key store and sends it. It should pick a certificate with a CA from the list which was sent down.
  • The server checks the client’s certificate is valid either because the certificate is in the trust store or the certificate for the Certificate Authority is in the trust store which validates the certificate.

Which certificate does the server send?

There are several implementations used in different products:

  • Take the first valid certificate in the ring. Easy to code, but causes problems. If I replace the certificate, by removing and readding it, then a different certificate may be used, as what was the second certificate in the keyring, will now be the first in the keyring, and this will be used instead. This may have a different type of certificate (Elliptic Curve or RSA), and clients may then fail to connect.
    • If the certificate has expired it will not be used, even though it looks the first.
  • Specify the default. There is one default. If you remove and add a certificate you can specify which certificate becomes the default. Not all clients may like the certificate type, so you may have to change the client certificate to match.
  • Pick the best option. Java has an function which says give me the list of certificate in the keyring which match the following, this might be EC or RSA or some other option. With this, different clients can get a different server certificate.

You could have one keystore for all your servers, or a keystore for CICS, and other for a Liberty web server, another for MQ web server.

I think that the keystore should contain only the keys for the certificates it needs to use. Do not have the CICS, Liberty, MQ certificate all in the same keystore if the certificates are all different. It would allow CICS to use the MQ certificate.

For added security certificates can have the name of the server, or the IP address of the server, and the client can check it matches the server’s host. You might not want to share a keyring between different z/OS images, because they may have a different IP address.

Client – picking the private certificate from the keystore

What should I use for my trust store?

The trust store has the certificates for all CAs you expect to use. If you start working with a different company you might need to add their CA to your trust store.
If you have one trust store for all users across the z/OS image, you add the certificate to it, and every one has access to it – easy.

If you have multiple trust stores, you’ll need to add the certificate to all the trust stores that need it. This may be a small number, for example, production and test.

If you use your keystore as a trust store then you will have to add the new CA to every trust store.

As certificate expire you will need to replace them. The amount of work depends on how many stores you have to update. Having one trust store makes this easier.

Why have I told you this?

I had a client server working using the DEFAULT certificate on z/OS. I wanted to use this keyring for a different application, which used the FIRST certificate in the keyring. This was a special-case certificate, and not to be used by most clients. I had to remove and re-add certificates in the keyring so the default certificate became the first certificate. It took over a day to identify the problem and fix it!

One minute explanation of public keys and private keys.

This topic is part of the “one minute” series of posts, which give an overview of a topic with enough information to be able to understand the concepts without going too deep.

You may have a banking application on your phone which you use to communicate with the bank’s server.

Setting up a secure connection

There are several parts to establishing a secure connection:

  • You need to ensure the back-end your application is connecting to is really to your bank – and not a bad actor’s system.
  • The bank needs to check that you are who you say you are – and not a bad actor impersonating you and trying to steal your money.
  • Agree what cipher techniques you will use so a bad actor cannot read your traffic
  • Some integrity techniques so you can be sure what you send to the bank is the same data as received by the bank, and a bad actor has not replaced your traffic with some evil traffic.

All the parts above reply on encryption and decryption

Introduction to encryption and decryption.

The basic concept of encryption is you have plaintext data, a function, and an encryption key. (I think of a sausage machine, into which you plug a specially shaped key). You feed the plaintext data into the function+key and out comes encrypted data.

The basic concept of decryption is to take the encrypted data, pass it through a function and the decryption key, and out comes the original plaintext.

For some encryption, the encryption key is the same as (or similar to) the decryption key. For example Change the letters A -> P, B -> $, C -> 9…. in this case if you have the encryption key, it is easy to determine the decryption key ( P -> A, $ -> B, 9 -> C). This simple technique is not very strong, it is easy to break and find the encryption/decryption key.

There are some mathematical functions that make it very hard to find the inverse key. Even if you know the encryption key – you will not be able to guess the decryption key. (You may not be able to guess, but large governments with limitless computer resources may be able to find the decryption key). These keys tend to be very large numbers with 100’s of digits.

Public and private keys

As described above you have an encryption key, and a different decryption key. Pick one, and make it generally available to the general PUBLIC. Keep the other key very PRIVATE – because anyone with this private key can decrypt the data. Hence PUBLIC and PRIVATE keys.

  • If you have some data and encrypt it with my PUBLIC key. Only someone with my PRIVATE key can decrypt it. Knowing the public key does not allow you to decrypt it.
  • If I have some data, and encrypt with my private key, then anyone with my public key can decrypt it. So what – every one has access to my public key. If my public key decrypts the data you know that it came from me (or someone with my private key). This statement is important; it is worth reading the statement again.

If you have a private/public key pair, and I have a public/private key pair we can do interesting things:

  • I encrypt with my private key, and then encrypt with your public key. Then only you can decrypt it (with your private key), and then you decrypt with my public key – so you know it came from me – and only you can read it.
  • You encrypt with my public key, then your private key. I can decrypt it with your public key (and so I know it came from you), and then I decrypt with my private key to get the original message and I know it came from you. Note: Anyone can tell that you send me a message because they can use your public key to decrypt it, but they cannot see what is inside the message as it is still encrypted.
  • Turn around the encryption. You encrypt with your private key, then my public key. I can decrypt it with my private key and your public key (and so know it came from you). I get the original message and know it came from you. Note: No one can tell who send me the data, as they cannot encrypt it because they do not have my private key.
  • If you understand these concepts you are doing well.

Integrity checking

One use of the private/public keys is for integrity checking:

For a payload, I calculate the checksum (a complex calculation on the data, such that any change to the data will give a different checksum calculation result). I take this checksum value and encrypt it with my private key, and attach it to the data. I send the data and encrypted checksum to you. You do the same calculation. You decrypt the encrypted checkum and it should be the same as your checksum calculation. If they match you know the data came from me, and is what I sent.

A bad actor could intercept the traffic from last week, and replay it this week. The above integrity check would show the data came from me, but not that it was last weeks data. There are other techniques for this.