How do I get my client talking to the server with a signed certificate

Signed certificates are very common, but I was asked how I connected my laptop to my server, in the scenario “one up” from a trivial example.

Basic concepts

  • A private/public key pair are generated on a machine. The private stays on the machine (securely). The public key can be sent anywhere.
  • A certificate has ( amongst other stuff)
    • Your name
    • Address
    • Public key
    • Validity dates

Getting a signed certificate

When you create a certificate: it does a checksum of the contents of the certificate, encrypts the checksum with your private key, and attaches this encrypted value to the certificate.

Conceptually, you go to your favourite Certificate Authority (UKCA) building and they Sign it

  • They check your passport and gas bill with the details of your certificate.
  • They attach the UKCA public key to your certificate.
  • They do a checksum of the combined documents.
  • They encrypt the checksum with the the UKCA private key, and stick this on the combined document.

You now have a signed certificate, which you can send it to anyone who cares.

Using it

When I receive it, and use it

  • my system compares my copy of the UKCA public certificate with the one in your certificate – it matches!
  • Using (either) UKCA public certificate – decrypt the encrypted checksum
  • Do the same checksum calculation – and the two values should match.
  • If they match I know I can trust the information in the certificate.

This means the checking of the certificate requires the CA certificate that signed it.

To use a (Linux) certificate on z/OS you either need to

  • issue the RACF GENCERT command on the Linux .csr file, export it, then download it to Linux. The certificate will contain the z/OS CA’s certificate.
  • import the Linux CA certificate into RACF (This is the easy, do once solution.)

then

  • connect the CA certificate to your keyring, and usually restart your server.

Setting up my system

If the CA certificate is not on your system, you need to import it from a dataset.

You can use FTP, or use cut and paste to the dataset.

Once you have the CA certificate in your RACF database you can connect it to your keyring.

Create my Linux CA and copy it to z/OS

CA="docca256"
casubj=" -subj /C=GB/O=DOC/OU=CA/CN=LINUXDOCCA2564"
days="-days 1095"
rm $CA.pem $CA.key.pem

openssl ecparam -name prime256v1 -genkey -noout -out $CA.key.pem

openssl req -x509 -sha384 -config caca.config -key $CA.key.pem -keyform pem -nodes $casubj -out $CA.pem -outform PEM $days

openssl x509 -in $CA.pem -text -noout|less

Where my caca.config has

####################################################################
[ req ]
distinguished_name = ca_distinguished_name
x509_extensions = ca_extensions
prompt = no

authorityKeyIdentifier = keyid:always,issuer:always

[ca_distinguished_name ]
[ ca_extensions ]

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
basicConstraints = critical,CA:TRUE, pathlen:0
keyUsage = keyCertSign, digitalSignature,cRLSign

Running the command gave

Certificate:
Data:
...
Issuer: C = GB, O = DOC, OU = CA, CN = LINUXDOCCA256
...
Subject: C = GB, O = DOC, OU = CA, CN = LINUXDOCCA256
...
X509v3 extensions:
...
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Key Usage:
Digital Signature, Certificate Sign, CRL Sign
...

Where it has CA:TRUE and X509v3 Key Usage:Certificate Sign

Which allows this to be used to sign certificates.

Installing the CA certificate on z/OS

You need to copy the docca256.pem file from Linux to a z/OS dataset (Fixed block, lrecl 80, blksize 80) you can use FTP or cut and paste. I used dataset COLIN.DOCCA256.PEM.

Import it into z/OS, and connect it to the START1.MYRING keyring as a CERTAUTH.

//COLRACF  JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT CHECKCERT('COLIN.DOCCA256.PEM')

*RACDCERT DELETE (LABEL('LINUXDOCA256')) CERTAUTH
RACDCERT ADD('COLIN.DOCCA256.PEM') -
CERTAUTH WITHLABEL('LINUXDOCA256') TRUST

RACDCERT CONNECT(CERTAUTH LABEL('LINUXDOCA256') -
RING(MYRING) USAGE(CERTAUTH)) ID(START1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh
/*

Once you have connected the CA to the keyring, you need to get the server to reread the keyring, or restart the server.

Getting my Linux certificate signed by z/OS

This works, but is a bit tedious for a large number of certificates.

I created a certificate request file using

timeout="--connect-timeout 10"
enddate="-enddate 20290130164600Z"

ext="-extensions end_user"

name="docec384Pass2"
key="$name.key.pem"
cert="$name.pem"
p12="$name.p12"
subj="-subj /C=GB/O=Doc3/CN="$name
rm $name.key.pem
rm $name.csr
rm $name.pem
passin="-passin file:password.file"
passout="-passout file:password.file"
md="-md sha384"
policy="-policy signing_policy"
caconfig="-config ca2.config"
caextensions="-extensions clientServer"


openssl ecparam -name secp384r1 -genkey -noout -out $name.key.pem
openssl req -config openssl.config -new -key $key -out $name.csr -outform PEM -$subj $passin $passout

The certificate request file docec384Pass2.csr looks like

-----BEGIN CERTIFICATE REQUEST----- 
MIIBpzCCAS0CAQAwNDELMAkGA1UEBhMCR0IxDTALBgNVBAoMBERvYzMxFjAUBgNV
...
Tmmvu/nqe0wTc/jJuC4c/QJt+BQ1SYMxz9LiYjBXZuOZkpDdUieZDbbEew==
-----END CERTIFICATE REQUEST-----

With words CERTIFICATE REQUEST in the header and trailer records.

Create a dataset(COLIN.DOCLCERT.CSR) with the contents. It needs to be a sequential FB, LRECL 80 dataset.

  • Delete the old one
  • Generate the certificate using the information in the .csr. Sign it with the z/OS CA certificate
  • Export it to a dataset.
//IBMRACF2 JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *

RACDCERT ID(COLIN ) DELETE(LABEL('LINUXCERT'))

RACDCERT ID(COLIN) GENCERT('COLIN.DOCLCERT.CSR') -
SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -
WITHLABEL('LINUXCERT')

RACDCERT ID(COLIN) LIST(label('LINUXCERT'))
RACDCERT EXPORT(label('LINUXCERT')) -
id(COLIN) -
format(CERTB64 ) -
password('password') -
DSN('COLIN.DOCLCERT.PEM' )

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Then you can download COLIN.DOCLCERT.PEM to a file on Linux and use it. I used cut and paste to create a file docec384Pass2.zpem

I used it like

set -x 
name="colinpaice"
name="colinpaice"
name="docec384Pass2"
insecure=" "
insecure="--insecure"
timeout="--connect-timeout 100"
url="https://10.1.1.2:10443"
trace="--trace curl.trace.txt"

cert="--cert ./$name.zpem:password"
key="--key $name.key.pem"

curl -v $cert $key $url --verbose $timeout $insecure --tlsv1.2 $trace

Using wireshark I can see CA certificates being send from z/OS, and the docec384Pass2.lpem used; signed by a z/OS CA certificate.

Using the certificate in the Chrome browser.

  • In Chrome settings, search for cert.
  • Click security
  • Scroll down to Manage certificates, and select it
  • Select customised
  • Select import, and then select the file.
    • When I generated the file with the Linux CA it had a file type of .pem
    • When I signed it on z/OS, then downloaded it with a type of.zpem, I had to select all files (because the defaults are *.pem,*.csr,*.der..)

Understanding a Wireshark TLS handshake trace

I’ve done a lot of work using TLS, and Wireshark is a great tool for displaying the flows of data. The problem is understanding what the output shows! This blog post shows what to look at.

Note: Flows with GREASE values should be ignored. This is designed to make sure the remote end can handle unexpected values, by sending up unsupported values.

Summary of the flows of the handshake

See here for an overview.

  • Client to Server: Client Hello
  • Server to Client: Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
  • Client to Server: Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
  • Server to Client: Change Cipher Spec
  • Server to Client: Finished

Client to Server: Client Hello

This displays

Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Version: TLS 1.2 (0x0303)
            Random: c5ec1cf320f2f77ba6f3f84e7fd7d1f946400484a56918a5e499b0e98be2a07f
            Session ID Length: 32
            Session ID: a839f3fcd1d7f4394c6e5bd0279da5c75458b8a81f35c5890107007cc921fa7b
            Cipher Suites Length: 32
            ▶Cipher Suites (16 suites)
            Compression Methods Length: 1
            ▶Compression Methods (1 method)j             
            Extensions Length: 429
            ▶Extension: signature_algorithms 
            Extension: Reserved (GREASE) (len=0)
            ▶Extension: supported_versions (len=7)
            ...

If you expand supported versions you get

 Extension: supported_versions (len=7)
       ...
       Supported Version: TLS 1.3 (0x0304)
       Supported Version: TLS 1.2 (0x0303)

Which shows you the TLS versions that the client supports.

If you expand Cipher Specs you get

Cipher Suites (16 suites)
   Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
   Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
   Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
   Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
   Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
   ...		        

This is the list of cipher specs in preference order, that the client supports.

The signature_algorithms defines which signature and hash algorithms are supported for the server’s digital certificate.

Server to Client: Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done

The server sends down several blocks of data. These are logically distinct blocks of data – but tend to arrive in one block.

  • Server Hello, Here are some parameters for the handshake
  • Certificate, Here is my certificate(s)
  • Server Key Exchange,
  • Certificate Request, Please send me a certificate matching the enclosed parameters
  • Server Hello Done

Server to Client: Server Hello

Handshake Protocol: Server Hello
  Handshake Type: Server Hello (2)
  Version: TLS 1.2 (0x0303)
  Random: ...
  Session ID Length: 32
  Session ID: ...
  Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
  ...  

It shows which version of TLS has been picked.

The Cipher Suite is the value picked by the server.

Server to Client: Certificate

This is present if the server is configured to send down the server’s certificate.

Handshake Protocol: Certificate
  Handshake Type: Certificate (11)
     ...
     Certificates Length: 1608
       Certificates (1608 bytes)
          Certificate Length: 734
          ▼Certificate: .. (commonName=10.1.1.2,...organizationName=NISTECCTEST)
             ▶signedCertificate
                serialNumber: 0x7a
                ▶signature (sha256WithRSAEncryption)
                ▶issuer: ...
                ▶validity ...
                ▼subject: rdnSequence (0)
                   RDNSequence (organizationName=NISTECCTEST)
                   RDNSequence (organizationalUnitName=SSS)
                   RDNSequence (commonName=10.1.1.2)
                ▶subjectPublicKeyInfo
                ▶extensions: 5 items
                algorithmIdentifier (sha256WithRSAEncryption)
                    ...
          Certificate Length: 868
          ▶Certificate:  ...

From this we can see the certificates sent down from the server.

  • The certificate name is specified at many levels; including on the Certificate and subject.
  • The CA (issuer) which signed each certificate.
  • The validity dates of the certificate
  • The extensions, which include how the certificate can be used; key usage, Subject Alternative Name ( IP address) etc.

Check this is what you are expecting, and that the CA is in your trust store.

Server to Client: Server Key Exchange

I have never used the information in this area

Server to Client: Certificate Request

Handshake Protocol: Certificate Request
   Certificate types count: 3
      ▼Certificate types (3 types)
         Certificate type: RSA Sign (1)
         Certificate type: ECDSA Sign (64)
         Certificate type: DSS Sign (2)
      ▼Signature Hash Algorithms Length: 26
         Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
         Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
         ... 
      ▼Distinguished Names Length: 246
         Distinguished Names (246 bytes)
             Distinguished Name: (commonName=COLIN4Certification Authority,...)
             Distinguished Name: (commonName=DocZosCA,...)
             Distinguished Name: (commonName=SSCA256...)

This is saying please send me a certificate:

  • The certificate must be one of certificate-type elements; RSA, ECDSA or DSS,
  • and signed by one of the algorithms listed,
  • and have a CA signer of COLIN4Certification Authority…, DocZosCA… or SSCA256… .

The CA signers are taken from the server’s trust store.

Server to Client: Server Hello Done

That’s the server side of the work done… almost

Client to Server: Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished

Client to Server: Certificate

The client needs to pick a certificate from those in its key store. Depending on the application

  • a browser may display a list of valid certificates for the end user to select
  • a program may use the information sent down in the handshake to pick the best certificate
  • a program may return what the configuration specified (such as default)
  • a program may just return the first certificate in the key file.
Handshake Protocol: Certificate
   Handshake Type: Certificate (11)
   Certificates Length: 683
   ▼Certificates (683 bytes)
     Certificate Length: 680
     ▼Certificate: … (commonName=docec384,...countryName=GB)
       ▼signedCertificate
          serialNumber: 0x029b
          ▶signature (ecdsa-with-SHA384)
          ▼issuer: rdnSequence (0)
          ▶rdnSequence: 4 items (commonName=SSCA256,OU=CA,iO=DOC,C=GB)
            ....
          ▶validity...
          ▼subject: rdnSequence (0)
              ▶RDNSequence item: 1 item (countryName=GB)
              ▶RDNSequence item: 1 item (organizationName=Doc2)
              ▶RDNSequence item: 1 item (commonName=docec384)
          ▶subjectPublicKeyInfo
          ▶extensions: 6 items...
          algorithmIdentifier (ecdsa-with-SHA384)
          ...

We can see

  • the algorithm identifier is ecdsa-with-SHA384 which is OK
  • the issuer is CN=SSA256,ou=CA,C=GB which is in the list of CAs passed from the server.
  • the signature is ecdsa – which is OK

Client to Server: Client Key Exchange

I haven’t used this. If the handshake gets this far, the set-up is good.

Client to Server: Certificate Verify

I have never used this. It is used to send up some data encrypted with the client’s private key, to check the server can decrypt it.

Client to Server: Change Cipher Spec

Response to the server. I have never used this. Business as usual, it periodically changes the Cipher Spec

Finished

Server to Client: Change Cipher Spec

Each direction has its own cipher spec. This is the server saying to the client … ok… I’m using the new cipher spec.

Server to Client: Finished

The final – final completion.

Problems

When an invalid CA was used at the client end, the server sent down

Transport Layer Security
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Unknown CA)
        Content Type: Alert (21)
        Version: TLS 1.2 (0x0303)
        Alert Message
            Level: Fatal (2)
            Description: Unknown CA (48)

This has a list of the codes and their meaning.

Some implementations of TLS are good, some are not so good.

Ive been struggling to get an application running on z/OS to use TLS with certificate authentication.

Now it is working, I despair at how the TLS has been implemented, and it feels that the development did not understand how the product would be used; or may be TLS has moved on since the product was initially written. Java on the other hand seems to have all the facilities for a good implementation, but they are not always used, resulting in a poor end user experience. It feels that the designers only expected one, or only a few users.

Experience shows…

The first time you do a journey, you are pleased if you get to the destination with no major problems. Once you have done the journey a couple of times you know the best options and what to avoid. For example when your plane arrives at an airport, and there is shuttle train to get you to the main terminal and immigration, if the escalator at the main terminal is near the front of the train – then get on at the front of the train, and you will be ahead of the hundreds of people from the train.

I learned when in a performance role, that it takes three attempts to write a performance report. After the first attempt you understand the real question you should be asking. The second attempt you get answers to these questions. The third attempt you try to add value and explanation. Many a time I’ve looked at someone else’s performance report and asked myself “what does this mean, how do I use this information?”. I spent a week doing some tests, and the outcome was a sentence “This performance tuning option has very little effect. Just leave it as the default”. This is so much more useful than a chart showing a flat line.

In the same way, a developer designing a program which uses TLS may be very happy having got one client to talk to the server using TLS. You need the experience to understand that there are different certificate types (Elliptic Curve, RSA), and different key sizes. It is easier to administer the CA certificates, if you have one enterprise wide keyring for them, and every one uses the same keyring; rather than one keyring for each server instance.

Where are certificate stored?

There can be two types of store for certificates:

  • Keystore – this contains the certificates each with its private key. Each group of servers may have its own keystore, with just the private keys it needs. So one key store for production CICS regions, one for test CICS regions, another key store for internal web servers.
  • Trust store. This contains the public keys of Certificate Authority certificate authorities, and any self-signed certificate from clients. The trust store is used to validate certificates which have been sent to the application. You could have one trust store for the whole LPAR, and every one uses it.

In many TLS implementations only one store is used, called the keystore, and has both the Key store and Trust store keys in it. This is easy to implement, but not a good design.

High level overview of the TLS handshake

The handshake

  • The client initiates the handshake, passing up information like I understand the following cipher specs…; I understand certificates created using Elliptic Curves (or RSA) (or both).
  • The server responds with ,
    • use this cipher spec…,
    • optionally (usually) here is my certificate which matches your criteria (Elliptic Curve or RSA)
    • optionally – please send your certificate which matches these criteria… here are the CA’s I know about
  • The client checks the server’s certificate is valid either because the certificate is in the trust store , or the certificate for the Certificate Authority is in the trust store, and can be used to validate the certificate.
  • If asked, the client picks a certificate from the key store and sends it. It should pick a certificate with a CA from the list which was sent down.
  • The server checks the client’s certificate is valid either because the certificate is in the trust store or the certificate for the Certificate Authority is in the trust store which validates the certificate.

Which certificate does the server send?

There are several implementations used in different products:

  • Take the first valid certificate in the ring. Easy to code, but causes problems. If I replace the certificate, by removing and readding it, then a different certificate may be used, as what was the second certificate in the keyring, will now be the first in the keyring, and this will be used instead. This may have a different type of certificate (Elliptic Curve or RSA), and clients may then fail to connect.
    • If the certificate has expired it will not be used, even though it looks the first.
  • Specify the default. There is one default. If you remove and add a certificate you can specify which certificate becomes the default. Not all clients may like the certificate type, so you may have to change the client certificate to match.
  • Pick the best option. Java has an function which says give me the list of certificate in the keyring which match the following, this might be EC or RSA or some other option. With this, different clients can get a different server certificate.

You could have one keystore for all your servers, or a keystore for CICS, and other for a Liberty web server, another for MQ web server.

I think that the keystore should contain only the keys for the certificates it needs to use. Do not have the CICS, Liberty, MQ certificate all in the same keystore if the certificates are all different. It would allow CICS to use the MQ certificate.

For added security certificates can have the name of the server, or the IP address of the server, and the client can check it matches the server’s host. You might not want to share a keyring between different z/OS images, because they may have a different IP address.

Client – picking the private certificate from the keystore

What should I use for my trust store?

The trust store has the certificates for all CAs you expect to use. If you start working with a different company you might need to add their CA to your trust store.
If you have one trust store for all users across the z/OS image, you add the certificate to it, and every one has access to it – easy.

If you have multiple trust stores, you’ll need to add the certificate to all the trust stores that need it. This may be a small number, for example, production and test.

If you use your keystore as a trust store then you will have to add the new CA to every trust store.

As certificate expire you will need to replace them. The amount of work depends on how many stores you have to update. Having one trust store makes this easier.

Why have I told you this?

I had a client server working using the DEFAULT certificate on z/OS. I wanted to use this keyring for a different application, which used the FIRST certificate in the keyring. This was a special-case certificate, and not to be used by most clients. I had to remove and re-add certificates in the keyring so the default certificate became the first certificate. It took over a day to identify the problem and fix it!

How to accept a certificate if I do not have their CA?

Someone contacted me saying that they had a z/OS application which was trying to access a remote server using TLS, but the connection was failing because the incoming certificate could not be validated.

Or or to put it another way, a certificate and its CA arrived at my system; how do I get the CA into my keyring to allow validation.

One way of getting the certificate is to use a browser. I used chrome.

  • enter the URL and press enter. It should display a page with a padlock at the front of the URL
  • click on this padlock
  • it should say “connection is secure”
  • click on “certificate is valid”, it displays certificate information
  • click on details. It should list the certificate and any CA certificates
  • click on the CA of interest
  • click Export and save the file (as base 64 encoded ASCII)
  • this will create a file containing —–BEGIN CERTIFICATE—– … —–END CERTIFICATE—–
  • check that this CA certificate is valid, and does not belong to a bad guy
  • upload this to a sequential file on z/OS
  • “check” it
//COLRACFI JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT CHECKCERT('COLIN.IMPORT.CA.PEM') 
  • Add it to RACF database
RACDCERT CERTAUTH     ADD(''COLIN.IMPORT.CA.PEM') - 
  WITHLABEL('MYBANK.CA') TRUST 
  • connect it to the keyring, CICSID/CICSRING
RACDCERT   CONNECT(CERTAUTH    LABEL('MYBANK.CA') - 
  RING(CICSRING) USAGE(CERTAUTH)) ID(CICSID) 

Getting a system ssl trace for httpd server on z/OS

I had a problem getting the httpd server to work on z/OS. It did not like my certificate – but reported messages about ICSF security not being set up. I got to learn a lot about traces, but could not find how to trace System SSL (GSK) and httpd.

I specified SSLTRACE in my vhost*.conf file which gave me a little information – but not at the System SSL Level.

Other applications using System SSL, can set environment variables

GSK_TRACE=0xff 
GSK_TRACE_FILE=/tmp/gskssl.%.trc 

but this does not work with http. (I think the environment variables are not passed on to any spawned thread).

You have to use the gsksrvr task, and collect the trace through CTRACE.

Set up gsk trace.

I have used gsk trace before, and described setting it up.

I had to create a parmlib member

TRACEOPTS 
          WTRSTART(ctwtr) 
          on 
          wtr(ctwtr) 
jobname(httpcp) 
          OPTIONS('LEVEL=255,JOBSUFFIX=ANY') 

I start my http server with the S HTTPCP command, and specified httpcp in the jobname of the parmlib.

I got out no trace. I tried the various jobnames until it produced a trace. My trace was produced from jobname httpcp8! I could not find a way of displaying which of my httpcp* job was used, so I had to try them all.

If I had had a long name eg httpcpxx then specifying jobname(httpcpxx) should have worked.

Starting the trace

I used

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON

where my parmlib member was CTGSKOK

It produced

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON
IEE252I MEMBER CTGSKON FOUND IN USER.Z24C.PARMLIB
GSK01040I SSL component trace started.
ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE
SUCCESSFULLY EXECUTED.

and the CTWTR started up.

What is the status of the trace?

d trace,comp=gsksrvr

gave me

IEE843I 08.32.25  TRACE DISPLAY       
  COMPONENT     MODE BUFFER HEAD SUBS                                  
 --------------------------------------------------------------        
  GSKSRVR       ON   0064K                                             
     ASIDS      *NOT SUPPORTED*                                        
     JOBNAMES   HTTPCP8 ,HTTPCP1 ,HTTPCP2 ,HTTPCP3 ,HTTPCP4 ,          
                HTTPCP5 ,HTTPCP6 ,HTTPCP7                              
     OPTIONS    LEVEL=255                                              
     WRITER     CTWTR                                                  

Run my test

When I ran my test, some System SSL messages were produced on the console from the gsksrvr address space

GSK01047I SSL component trace started for HTTPCP8/STC01000.
GSK01050I SSL component trace started for HTTPCP8/STC01000/05010022.

Stop the trace

TRACE CT,OFF,COMP=GSKSRVR

Wait until you get

GSK01041I SSL component trace ended.

from the gsksrvr address space, and stop the trace writer.

D TRACE,WTR=ALL
TRACE CT,WTRSTOP=CTWTR

This seems to take a few seconds to run. It outputs

IEF196I AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA : 
IEF196I           IBMUSER.CTRACE1 
ITT111I CTRACE WRITER CTWTR TERMINATED BECAUSE OF A WTRSTOP REQUEST.  
IEF404I CTWTR - ENDED - TIME=08.40.21                                                                 

Format the trace

You need to use IPCS to format it

  • =0 – and enter the data set name (IBMUSER.CTRACE1)
  • =6 – to get to the ipcs command screen
  • dropd – to tell IPCS to forget any historical information it may know about for the dataset
  • CTRACE COMP(GSKSRVR) full – this displays any System SSL CTRACE data
  • m PF8 – go to the bottom of the data
  • report view – to go into ISPF View mode on the data set
  • X ALL – to hide all of the data
  • f SSL_ERROR ALL – this shows any error codes
  • if you get any lines displayed, you can tab down to the hidden command and use the line prefix command f4 to display the first 4 hidden lines.

The errors I got were

 S0W1      MESSAGE   00000004  08:35:55.049451  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_ec_token_private_key_sign 
   ICSF service failure: CSFPPKS retCode = 0x8, rsnCode = 0x2b00 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.049733  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_sign_data 
   crypto_ec_sign_data() failed: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050012  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  construct_tls13_certificate_verify_message 
   Unable to generate certificate verify message: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050393  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  send_tls13_alert 
   Sent TLS 1.3 alert 51 to ::ffff:10.1.0.2.45432. 
                                                                                                              

The Alert 51 matches what my browser received.

File /usr/include/gskcms.h had #define CMSERR_ICSF_SERVICE_FAILURE 0x03353084

CSFPPKS is PKCS #11 Private Key Sign.

The return code 0x2b00 (from here) gives:

User action: You might need to re-create the token by using the PKA key token build or PKA key import callable service or regenerate the key values on another platform.

or in other words, it did not like my certificate created with NISTECC SIZE(256) but did like NISTECC SIZE(521).

One minute… what is a digital certificate?

This blog post covers

Why use certificates?

Certificates can be used as an authentication mechanism. It allows identity information to be sent over a network, and the remote end to validate the information before using it. For this to work, we need a “Certificate Authority” who can validate the certificate. When the remote end gets the “certified certificate” it checks the certification and validity and if successful can use this. Checking can involve sending an electronic request to the Certificate Authority asking if the certificate is still valid.

What is a certificate?

A certificate has several components (conceptually I think of a certificate as an envelope with several bits of paper within it).

  • A public key. See One minute explanation of public keys and private keys.
  • Information about the owner, such as name and address or organization
  • Meta information – such as it can be used for document signing (but not certificate signing), and validity dates.
  • Signing information.

Information about the owner

A certificate has information about the “owner”. Common attributes include

  • Common Name CN=Colin Paice
  • Organisation O=My Org
  • Organisational Unit OU=TEST
  • Country C=GB

Meta information.

The certificate contains information on how it should be used. For example

  • This certificate can be used as a server’s certificate – but not as a client certificate.
  • This certificate can be used for signature but not encryption.
  • This certificate can be used to sign other certificate (see below).

A server (or client) could decide to ignore some of these attributes. A proper server will honour the information, and if the client’s certificate does not have the “can be used by a client” attribute set, it will reject the certificate.

How do you trust a certificate?

If you have been sent a certificate – how do you tell if it is genuine? This was one of the problems in the early days of secure communications.

This problem was solved by having an organisation we both trust, and having this organisation “sign” a certificate.

Imagine there is a UK Certificate Authority (UKCA). To get your certificate approved…

  • you take your certificate, your passport, and proof of address to the Certificate Authority (UKCA).
  • A clerk in the CA, checks that the name in your certificate matches your passport, and the address in the certificate matches the proof of address your provided.
  • The clerk can check “the permissions” within your certificate, such as this certificate can/cannot be used as a certificate authority, or can/cannot be used as a server. The CA may not care!
  • The CA take an electronic copy of your certificate and perform a checksum calculation on the contents. If the certificate is changed in any way, the checksum calculation will be a different number.
  • The CA then encrypts the checksum calculation, prints it out, and staples the encrypted value and a copy of the CA’s public certificate to the back of your certificate.
  • The certificate has now been signed by the CA.

You send me a copy of your certificate and attachement created by the CA.

When I get a copy of your certificate (and attachment),

  • I check the CA’s public certificate that you sent me is the same as the copy I already have.
  • I do the same check sum calculation as the CA authority did.
  • I use the CA’s public certificate to decrypt the CA’s encrypted checksum.
  • The two checksum values should match!

If the checksum values match, then I can trust that the information in the certificate is the same as the information you showed to the CA when getting you certificate “signed”. Of course if you provided a fake passport, all I know is that what I see is what the CA saw.

Intermediate CAs and certificate chains

Of course it costs you money to get your certificate signed by a CA. To reduce this cost you can set up your own enterprise Certificate Authority. You create a certificate called CN=MYORGCA, and set the attribute set to allow it to sign other certificates. You take this the the UKCA and get it signed.

When Joe wants his personal certificate signed, he come to you,

  • you check Joe’s corporate ID badge has the same name as in Joe’s certificate
  • you check any other data fits within your enterprise standards, such as expiry date.
  • you use your enterprise CA certificate(CN=MYORGCA) to sign Joe’ certificate.
  • Joe get a package with
    • Joe’s certificate,
    • the checksum from the enterprise signing, and the enterprise public certificate.
    • a copy of the UKCA’s certificate and encrypted checksum for your enterprise’s CA.

When I get Joe’s certificate and attachments.

  • I can see if I have a copy of your enterprise public certificate.
  • No I do not, but I do have a copy of the UKCA public certificate,
  • I validate the your enterprise’s public certificate, using the UKCA certificate
  • If your enterprises public certificate validates successfully, I can then use it to validate Joe’s certificate.

This means I can validate certificates sent to me, as long as I have at least one of the certificates in the chain.

You could take this further and have a departmental CA authority.

Is the certificate still valid?

If someone leaves your organisation you want to ensure that any certificates issued to that person cannot be used.

You can have an expiry date in the certificate, so within a year or two the certificate will expire. This is not good enough, and you want to make the invalidate the certificate as soon as possible. This can be done through an Online Certificate Status Protocol (OCSP) attribute in the certificate. Basically this says go and ask the URL if this certificate if it is still valid.
How do you know who to ask? A certificate has url information within in it Authority Information Access: OCSP – URI:http://ColinsCert.Checker.com/ . The back end serving the URL is typically an LDAP server.

If you think about this for a few minutes this may not seem a good idea. For every secure connection handshake, you have to issue a query to a remote server somewhere to see if the certificate is still valid. This will seriously affect performance. This was solved by having a “OCSP response valid time”, sent as part of the response from the OCSP server. This known as stapling (I picture this as stapling a valid-until ticket to a certificate). The OCSP server can say “This certificate is valid, and assume it is valid for n hours”. The first time the client uses the certificate it may take longer because of the OCSP checks. For successive requests , the client can then use the certificate without doing any OCSP checks – until the time period has expired.

Do I trust the sender?

A bad actor could have copied the certificate from a valid server, and presented it to the client. The client checks it, and it looks ok, but in reality it came from a bad server.

You can include Subject Alternate Name (or SAN) in the certificate. This is the URL or IP address of the server. When the certificate is sent from the server, the client checks that the IP address of the server is in the list of SANs in the certificate. If it does not match then the certificate is rejected. Example SANs DNS:www.example.com, IP:10.1.1.2

If the DNS:name is used, this will also require a connection to a DNS to get back the IP address, which will add time to TLS hand shake to the server.

You should use SAN for the server, but not for the client. A Server will have a fixed, or limited choice of values. A client can get a random IP address and so is unlikely to match a value in the certificate.