Configuring RSEAPI on z/OS to use TLS

The RSEAPI server is the Apache Tomcat server plus RSEAPI specific stuff. If you know how to configure Tomcat, you know most of what you need. The Tomcat customising is documented here.

This post follows on from Getting REST to work into z/OS. I was unclear at first how to correctly specify overrides. I’ve blogged an article Passing parameters to Java program to show how some parameters are specified as RSEAPI_KEYSTORE_FILE=… and other parameters are specified as -Djava.protocol.handler.pkgs=…

See Java Parameters for how I configured RSEAPI to be able to flip configuration options.

Update your level of Java

I had various problems getting TLS to work with RSEAPI.

  • TLSv1.3 was not supported on the level of Java V8 I originally had.
  • I had to override the /etc/zexpl/java.security file so that it understood keyrings of the format safkeyring://START1/MQRING

When I refreshed the level of Java (to SR8 FP6 dated June 2023), things worked much better. I would recommend getting a level of Java shipped within the last year.

I tested this by changing rseapi.env to include

export JAVA_HOME="/usr/lpp/java/new/J8.0_64" 
export LIBPATH="$JAVA_HOME/bin:$JAVA_HOME/bin/classic:"$LIBPATH 
export PATH="$JAVA_HOME/bin:"$PATH 

Without this I got various Java problems, such as an unresolved dependency.

Getting RSE to work with TLS was not trivial

The original version of RSEAPI was v1.0.5 (see /usr/lpp/IBM/rseapi/tomcat.base/bin/current_version.txt) Another version is available in GITHUB with a version of v1.1.0 created 7 July 2022 which I worked with.

TLS configuration changes

RSEAPI supports only one port. To use TLS change the procedure to use SECURE=’true’, (or override it at startup).

The RESAPI proc has the location of the configuration files. Mine says /etc/zexpl.

The main file to edit is /etc/zexpl/rseapi.env . The sample has a lot of commented out statements. I added at the bottom

RSEAPI_KEYSTORE_FILE="safkeyring://START1/MQRING " 
RSEAPI_KEYSTORE_TYPE="JCERACFKS" 
RSEAPI_KEYSTORE_PASS="password" 
RSEAPI_USING_ATTLS=false 
RSEAPI_SSL_ENABLED_PROTOCOLS=TLSv1.2

Which server certificate to use?

By default, the first key read from the keystore will be used. See certificateKeyAlias in Apache Tomcat configuration.

To change this I edited /usr/lpp/IBM/rseapi/tomcat.base/conf/sserver.xml and added

 <Connector port="${port.http}" protocol="${http.protocol}" 
   certificateKeyAlias="${serverCert}" 
   ...
/> 

I then edited /etc/zexpl/rseapi.env and added

d3=" -DserverCert=NISTECCTEST" 
JAVA_OPTS="$d3 " 
CATALINA_OPTS=$JAVA_OPTS 
export JAVA_OPTS 
export CATALINA_OPTS 

Using TLSv1.3

I blogged configuring Java to support TLSV1.3 a separate post TLS 1.3, Java and z/OS.

The additional RSEAPI specific configuration was

RSEAPI_HTTP_PROTOCOL=HTTP/1.1 
RSEAPI_SSL_ENABLED_PROTOCOLS=TLSv1.3 
RSEAPI_SSL_CIPHERS=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA_POLY1305_SHA256 
RSEAPI_KEYSTORE_FILE="safkeyring://START1/MQRING " 
RSEAPI_KEYSTORE_TYPE="JCERACFKS" 
RSEAPI_KEYSTORE_PASS="password" 
RSEAPI_USING_ATTLS=false 

You can use RSEAPI_SSL_ENABLED_PROTOCOLS=TLSv1.3,TLSV1.2 to get both TLS 1.2 and 1.3 support.

ClientAuth Support

The web browser (Tomcat) has support for requiring clients to specify a certificate as part of the TLS handshake.

I got this to work by editing /usr/lpp/IBM/rseapi/tomcat.base/conf/sserver.xml, and changing clientAuth=”false” to clientAuth=”{$clientAuth}”

<Connector port="${port.http}" protocol="${http.protocol}" 
     clientAuth="${clientAuth}" sslProtocol="TLS" 
...
>

and setting the value to “required” or specifying the value in the startup options:

d1="-DclientAuth=required"
t1=" -Djavax.net.ssl.trustStoreType=JCERACFKS" 
t2=" -Djavax.net.ssl.trustStore=safkeyring://START1/MQRING"
JAVA_OPTS=" $d1 $t1 $t2" 
CATALINA_OPTS=$JAVA_OPTS  
export JAVA_OPTS 
export CATALINA_OPTS 

The …trustStoreType and … trustStore provide the defaults if non are specified in the sserver.xml.

Use of keystore and trust store

The use of a trust store to store the CA certificates, and any self signed certificates is recommended. The keystore then contains just the private keys needed by the server. This means you can have one trust store per LPAR, which saves administratio.

If you use a combined trust key and trust store, and this is shared by applications, then applications may get access to private certificates used by other application, so is not as secure.

The tomcat documentation describes the truststore* parameters. These are in the in <Connector…. within file /usr/lpp/IBM/rseapi/tomcat.base/conf/sserver.xml .

For example

<SSLHostConfig 
       protocols="${ssl.enabled.protocols}"> 
       <Certificate type="EC" 
          certificateKeyAlias="NISTECCTEST" 
          certificateKeystoreFile="${keystoreFile}" 
          certificateKeystorePassword="${keystorePass}" 
          truststoreType="${trustStoreType}"                     
          truststoreFile="${trustStoreFile}"
          truststorePassword="${trustStorePass}"
      /> 
</SSLHostConfig>

and specify -Dxx where xx is the value in ${xx} such as -DtrustStoreType=”JCERACFKS” . You can hard code the values.

Setup problems

I had a variety of problems. Most were solved by going to a newer level of Java or RSEAPI. For example earlier versions did not support TLSv1.3

Authority issue

I got the message

Caused by: java.lang.IllegalArgumentException: The private key of NEWTECCTEST is not available or no authority to access the private key

This was caused by the certificate belonged to a userid START1, but I was running RSEAPI on userid STCRSE. For userid STCRSE to be able to access the private certificate part of the certificate of another userid’s certificate, the STCRSE userid need UPDATE access to the keyring profile.

My keyring was safkeyring://START1/MQRING. I needed

permit START1.MQRING.LST   class(RDATALIB) access(update) id(STCRSE)
setropts raclist(rdatalib) refresh

TLS 1.3, Java and z/OS

I was failing to get this working with Java 8 – despite it being supported in Java 8.

I eventually found a support statement which says

Enables TLS v1.3 protocol (defined by RFC 8446) in SR6 FP25

however java -version gave me

Java -version gives build 8.0.6.16 – pmz6480sr6fp16-20200902_01(SR6 FP16))

I had Fix Pack 16 and the support is in Fix Pack 25 (from 2020)

Download a more up to date version

The IBM Support page Java SDK Products on z/OS lists the supported version of Java. There is a link to IBM Semeru Runtime Certified Edition for z/OS (Semeru is free production-ready binaries built with the OpenJDK class libraries and the Eclipse OpenJ9 JVM).

When I tried to download V8 SR8, I got a window saying “You cannot download this due to export restrictions”. Next day, I tried again and it worked, the droids working in the background much have found I was a good guy.

Check your java.security file

When I used my own copy of the java.security file it did not work.

I had

security.provider.1=com.ibm.crypto.provider.IBMJCE 
security.provider.2=com.ibm.crypto.plus.provider.IBMJCEPlus 

The latest fix pack Java had

security.provider.1=com.ibm.jsse2.IBMJSSEProvider2 
security.provider.2=com.ibm.crypto.provider.IBMJCE 
security.provider.3=com.ibm.crypto.plus.provider.IBMJCEPlus 

The error messages I got were

SEVERE org.apache.catalina.util.LifecycleBase.handleSubClassException
Failed to initialize component Connector HTTP/1.1-6800

org.apache.catalina.LifecycleException: Protocol handler initialization failed Caused by: java.lang.IllegalArgumentException: None of the protocols specified are supported by the SSL engine : TLSv1.3

As the support for TLSv1.3 was in JSSE, it is clear (with hindsight) that I needed to specific the provider com.ibm.jsse2.IBMJSSEProvider2!

Some implementations of TLS are good, some are not so good.

Ive been struggling to get an application running on z/OS to use TLS with certificate authentication.

Now it is working, I despair at how the TLS has been implemented, and it feels that the development did not understand how the product would be used; or may be TLS has moved on since the product was initially written. Java on the other hand seems to have all the facilities for a good implementation, but they are not always used, resulting in a poor end user experience. It feels that the designers only expected one, or only a few users.

Experience shows…

The first time you do a journey, you are pleased if you get to the destination with no major problems. Once you have done the journey a couple of times you know the best options and what to avoid. For example when your plane arrives at an airport, and there is shuttle train to get you to the main terminal and immigration, if the escalator at the main terminal is near the front of the train – then get on at the front of the train, and you will be ahead of the hundreds of people from the train.

I learned when in a performance role, that it takes three attempts to write a performance report. After the first attempt you understand the real question you should be asking. The second attempt you get answers to these questions. The third attempt you try to add value and explanation. Many a time I’ve looked at someone else’s performance report and asked myself “what does this mean, how do I use this information?”. I spent a week doing some tests, and the outcome was a sentence “This performance tuning option has very little effect. Just leave it as the default”. This is so much more useful than a chart showing a flat line.

In the same way, a developer designing a program which uses TLS may be very happy having got one client to talk to the server using TLS. You need the experience to understand that there are different certificate types (Elliptic Curve, RSA), and different key sizes. It is easier to administer the CA certificates, if you have one enterprise wide keyring for them, and every one uses the same keyring; rather than one keyring for each server instance.

Where are certificate stored?

There can be two types of store for certificates:

  • Keystore – this contains the certificates each with its private key. Each group of servers may have its own keystore, with just the private keys it needs. So one key store for production CICS regions, one for test CICS regions, another key store for internal web servers.
  • Trust store. This contains the public keys of Certificate Authority certificate authorities, and any self-signed certificate from clients. The trust store is used to validate certificates which have been sent to the application. You could have one trust store for the whole LPAR, and every one uses it.

In many TLS implementations only one store is used, called the keystore, and has both the Key store and Trust store keys in it. This is easy to implement, but not a good design.

High level overview of the TLS handshake

The handshake

  • The client initiates the handshake, passing up information like I understand the following cipher specs…; I understand certificates created using Elliptic Curves (or RSA) (or both).
  • The server responds with ,
    • use this cipher spec…,
    • optionally (usually) here is my certificate which matches your criteria (Elliptic Curve or RSA)
    • optionally – please send your certificate which matches these criteria… here are the CA’s I know about
  • The client checks the server’s certificate is valid either because the certificate is in the trust store , or the certificate for the Certificate Authority is in the trust store, and can be used to validate the certificate.
  • If asked, the client picks a certificate from the key store and sends it. It should pick a certificate with a CA from the list which was sent down.
  • The server checks the client’s certificate is valid either because the certificate is in the trust store or the certificate for the Certificate Authority is in the trust store which validates the certificate.

Which certificate does the server send?

There are several implementations used in different products:

  • Take the first valid certificate in the ring. Easy to code, but causes problems. If I replace the certificate, by removing and readding it, then a different certificate may be used, as what was the second certificate in the keyring, will now be the first in the keyring, and this will be used instead. This may have a different type of certificate (Elliptic Curve or RSA), and clients may then fail to connect.
    • If the certificate has expired it will not be used, even though it looks the first.
  • Specify the default. There is one default. If you remove and add a certificate you can specify which certificate becomes the default. Not all clients may like the certificate type, so you may have to change the client certificate to match.
  • Pick the best option. Java has an function which says give me the list of certificate in the keyring which match the following, this might be EC or RSA or some other option. With this, different clients can get a different server certificate.

You could have one keystore for all your servers, or a keystore for CICS, and other for a Liberty web server, another for MQ web server.

I think that the keystore should contain only the keys for the certificates it needs to use. Do not have the CICS, Liberty, MQ certificate all in the same keystore if the certificates are all different. It would allow CICS to use the MQ certificate.

For added security certificates can have the name of the server, or the IP address of the server, and the client can check it matches the server’s host. You might not want to share a keyring between different z/OS images, because they may have a different IP address.

Client – picking the private certificate from the keystore

What should I use for my trust store?

The trust store has the certificates for all CAs you expect to use. If you start working with a different company you might need to add their CA to your trust store.
If you have one trust store for all users across the z/OS image, you add the certificate to it, and every one has access to it – easy.

If you have multiple trust stores, you’ll need to add the certificate to all the trust stores that need it. This may be a small number, for example, production and test.

If you use your keystore as a trust store then you will have to add the new CA to every trust store.

As certificate expire you will need to replace them. The amount of work depends on how many stores you have to update. Having one trust store makes this easier.

Why have I told you this?

I had a client server working using the DEFAULT certificate on z/OS. I wanted to use this keyring for a different application, which used the FIRST certificate in the keyring. This was a special-case certificate, and not to be used by most clients. I had to remove and re-add certificates in the keyring so the default certificate became the first certificate. It took over a day to identify the problem and fix it!

How to accept a certificate if I do not have their CA?

Someone contacted me saying that they had a z/OS application which was trying to access a remote server using TLS, but the connection was failing because the incoming certificate could not be validated.

Or or to put it another way, a certificate and its CA arrived at my system; how do I get the CA into my keyring to allow validation.

One way of getting the certificate is to use a browser. I used chrome.

  • enter the URL and press enter. It should display a page with a padlock at the front of the URL
  • click on this padlock
  • it should say “connection is secure”
  • click on “certificate is valid”, it displays certificate information
  • click on details. It should list the certificate and any CA certificates
  • click on the CA of interest
  • click Export and save the file (as base 64 encoded ASCII)
  • this will create a file containing —–BEGIN CERTIFICATE—– … —–END CERTIFICATE—–
  • check that this CA certificate is valid, and does not belong to a bad guy
  • upload this to a sequential file on z/OS
  • “check” it
//COLRACFI JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT CHECKCERT('COLIN.IMPORT.CA.PEM') 
  • Add it to RACF database
RACDCERT CERTAUTH     ADD(''COLIN.IMPORT.CA.PEM') - 
  WITHLABEL('MYBANK.CA') TRUST 
  • connect it to the keyring, CICSID/CICSRING
RACDCERT   CONNECT(CERTAUTH    LABEL('MYBANK.CA') - 
  RING(CICSRING) USAGE(CERTAUTH)) ID(CICSID) 

Collecting a wire-shark trace with TLS active for a browser

With TLS, the data transfer is encrypted. With TLS 1.3, part of the handshake is encrypted. In Wireshark this shows up as Encrypted Application Data.

You can configure wireshark to decrypt the data flow . You need to set an environment variable, and start the application from the terminal window, to pick up the environment variable.

export SSLKEYLOGFILE=$HOME/sslkeylog.log
google-chrome

This will cause the application( the google-chrome browser) to start and to write the TLS key data to the file.

Configure Wireshark to use this file:

  • Edit
  • Preferences
  • Expand the Protocols twistie
  • Scroll down to TLS ( typing T gets you near it)
  • Enter the Master-Secret log file name mine is /home/colinpaice/sslkeylog.log

Now, if you run the test you should get the data decrypted

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
   Change Cipher Spec Message
 TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
  Opaque Type: Application Data (23)
    [Content Type: Alert (21)]
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

Strong keys may be bad for your blood pressure.

I’ve just spent a couple of days trying to get a web server to use an Elliptic key with size 521. It works on Firefox, Curl, and OpenSSL, but not on Chrome. Weaker keys work of size 256 and 384 work fine, except in some cases a key size of 256 gave did not work on z/OS.

I’ve since found a page going back to 2015 discussing the dropping of several EC cipher specs. There is also discussion “NSA Suite B says that NSA uses curves P-256 and P-384” which (to some people) implied that other elliptic curves should not be used, and so were removed. There is another discussion that P256 is as strong as P521, but requires less CPU, and so P521 should not be used.

So the short answer is use Elliptic curves with key size 256 or 256, and do not use the others.

I was trying to get the Apache HTTPD server on z/OS to use a certificate, and it failed. I tried using a similar certificate from the openssl Simple server on Linux and it failed, so I’m guessing that my Chrome version (Version 114.0.5735.198 (Official Build) (64-bit)) does not support it.

The problem

In Chrome I got

This site can’t provide a secure connection 10.1.1.2 sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

This is not entirely true, as the browser sent the Alert response to the server and it was not an ERR_SSL_PROTOCOL_ERROR. After this the browser code returned its caller saying “failed” . Using Wireshark, capturing encrypted data showed the data flowing from the browser to the server to port 443),

Transmission Control Protocol, Src Port: 38188, Dst Port: 443

Transport Layer Security
  TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    Content Type: Change Cipher Spec (20)
    Change Cipher Spec Message
  TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
    Opaque Type: Application Data (23)
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

In the Apache HTTPD error.log it had

SSL0209E: SSL Handshake Failed, ERROR processing cryptography. [10.1.0.2:38188 -> 10.1.1.2:443] 

Which is not very helpful. The problem occurred on the browser; all the server sees is Alert code 51.

Taking a system SSL trace on z/OS(GSK Server) gave me

Job HTTPCP8   Process 0501003C  Thread 00000032  read_tls13_alert 
  TLS 1.3 alert 51 received from ::ffff:10.1.0.2.38188. 

Diagnostics

Using tools like gsktrace on z/OS, and Wireshark to see the flow over the network. I could see that an alert was sent from the Chrome browser to server with type Decrypt Error(51). This meant it was a problem at the browser end, not the server end.

Wireshark capturing encrypted data.

If you use TLS 1.3 once the “change cipher spec” has been issued, all traffic is encrypted, and by default Wireshark cannot read it. For example

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
     Change Cipher Spec Message
 TLSv1.3 Record Layer: Application Data Protocol: http-over-tls
   Opaque Type: Application Data (23)
     Version: TLS 1.2 (0x0303)
       Encrypted Application Data: 489be8a0976798290766c9ee158c24f5863d18
      [Application Data Protocol: http-over-tls]d

and you cannot tell this is reporting an error.

Wireshark can decrypt this. You need to set an environment variable, and start the application from the terminal window, to pickup the environment variable.

export SSLKEYLOGFILE=$HOME/sslkeylog.log
google-chrome

This will cause the application( the google-chrome browser) to start and to write the TLS key data to the file.

Configure Wireshark to use this file:

  • Edit
  • Preferences
  • Expand the Protocols twistie
  • Scroll down to TLS ( typing T gets you near it)
  • Enter the Master-Secret log file name mine is /home/colinpaice/sslkeylog.log

Now, if you run the test you should get the data decrypted

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
   Change Cipher Spec Message
 TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
  Opaque Type: Application Data (23)
    [Content Type: Alert (21)]
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

Diagnostics that were not helpful.

Chrome trace

Starting Chrome from a terminal session

google-chrome --enable-logging --v=1

The output was in ./.config/google-chrome/chrome_debug.log.

Using grep ERROR ./.config/google-chrome/chrome_debug.log gave me

...:ERROR:nss_util.cc(357)] After loading Root Certs, loaded==false: NSS error code: -8018
...:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -107
...:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -107

and no other useful information besides net_error -107. For these codes see here. 107 is an unhelpful message NET_ERROR SSL_PROTOCOL_ERROR.

Wireshark gave me decrypt error.

z/OS system SSL trace

Using GSKSRVR and CTRACE on z/OS. See here. This gave me

S0W1      MESSAGE   00000004  17:20:47.690772  SSL_ERROR 
  Job HTTPCP8   Process 0501003C  Thread 00000032  read_tls13_alert 
  TLS 1.3 alert 51 received from ::ffff:10.1.0.2.60830. 

Which shows the alert came from the browser.

Another way of getting system SSL trace.

The documentation says you can collect trace by changing the configuration. This allows you to trace just the pages/scripts that you want.

Creating a certificate on z/OS

On z/OS I used

RACDCERT ID(START1) GENCERT -                            
  SUBJECTSDN(CN('10.1.1.2') - 
             O('NISTECCTEST') - 
             OU('SSS')) -                                
   ALTNAME(IP(10.1.1.2))-                                
   NISTECC - 
   KEYUSAGE(   HANDSHAKE     )  - 
   SIZE(384) - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -               
   WITHLABEL('NISTECCTEST')     - 
                                                         

With different sizes 256,284, and 521. The IBM documentation says For NISTECC keys, valid key sizes are 192, 224, 256, 384, and 521 bits. I had problems with key size 521 bits.

On Linux I used

timeout="--connect-timeout 10"
enddate="-enddate 20240130164600Z" 
ext="-extensions end_user"
name="docec384"
key="$name.key.pem"
cert="$name.pem"

subj="-subj /C=GB/O=Doc2/CN="$name 
CA="docca256"
cafiles="-cert $CA.pem -keyfile $CA.key.pem "
rm $name.key.pem
rm $name.csr
rm $name.pem
passin="-passin file:password.file"
passout="-passout file:password.file"
md="-md sha384"
policy="-policy signing_policy"
caconfig="-config ca2.config"
caextensions="-extensions clientServer"
config="-config openssl.config"

openssl ecparam -name secp384r1 -genkey -noout -out $name.key.pem 
openssl req $config -new -key $key -out $name.csr -outform PEM -$subj $passin $passout 
openssl ca $caconfig $policy $ext $md $cafiles -out $cert -in $name.csr $enddate $caextensions 

openssl x509 -in $name.pem -text -noout|lessf

I used this as a template with different flavours of -name secp384r1

  • -name secp384r1
  • -name prime256v1 which worked
  • -name secp521r1 which did not work

You can get a list of valid names from the command openssl ecparam -list_curves.

Running a Linux web server

I tried using the openssl s_server, and could also reproduce this problem (with a much faster turnaround). I used

tls="-tls1_3  "
#cert=" -cert ./docec256.pem -key ./docec256.key.pem"
#cert=" -cert ./docecgen.pem -certform pem -key docecgen.key.pem -keyform pem" 
#cert=" -cert ./docec521.pem -certform pem -key docec521.key.pem -keyform pem"
cert=" -cert ./docec384.pem -certform pem -key docec384.key.pem -keyform pem" 
CA="-chainCAfile ./docca256.pem"
cipher1="-cipher TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384"

cipher=$cipher1
port="-port 4433 "

strict="-x509_strict -strict"
debug="-trace"

#  ca="-CAfile ./zpdt.ca.pem "
openssl s_server $port $tls  $cert $cipher $CA $debug $strict -www  

Chrome startup options

You can start chrome from a terminal with many startup options. See here

Getting a system ssl trace for httpd server on z/OS

I had a problem getting the httpd server to work on z/OS. It did not like my certificate – but reported messages about ICSF security not being set up. I got to learn a lot about traces, but could not find how to trace System SSL (GSK) and httpd.

I specified SSLTRACE in my vhost*.conf file which gave me a little information – but not at the System SSL Level.

Other applications using System SSL, can set environment variables

GSK_TRACE=0xff 
GSK_TRACE_FILE=/tmp/gskssl.%.trc 

but this does not work with http. (I think the environment variables are not passed on to any spawned thread).

You have to use the gsksrvr task, and collect the trace through CTRACE.

Set up gsk trace.

I have used gsk trace before, and described setting it up.

I had to create a parmlib member

TRACEOPTS 
          WTRSTART(ctwtr) 
          on 
          wtr(ctwtr) 
jobname(httpcp) 
          OPTIONS('LEVEL=255,JOBSUFFIX=ANY') 

I start my http server with the S HTTPCP command, and specified httpcp in the jobname of the parmlib.

I got out no trace. I tried the various jobnames until it produced a trace. My trace was produced from jobname httpcp8! I could not find a way of displaying which of my httpcp* job was used, so I had to try them all.

If I had had a long name eg httpcpxx then specifying jobname(httpcpxx) should have worked.

Starting the trace

I used

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON

where my parmlib member was CTGSKOK

It produced

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON
IEE252I MEMBER CTGSKON FOUND IN USER.Z24C.PARMLIB
GSK01040I SSL component trace started.
ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE
SUCCESSFULLY EXECUTED.

and the CTWTR started up.

What is the status of the trace?

d trace,comp=gsksrvr

gave me

IEE843I 08.32.25  TRACE DISPLAY       
  COMPONENT     MODE BUFFER HEAD SUBS                                  
 --------------------------------------------------------------        
  GSKSRVR       ON   0064K                                             
     ASIDS      *NOT SUPPORTED*                                        
     JOBNAMES   HTTPCP8 ,HTTPCP1 ,HTTPCP2 ,HTTPCP3 ,HTTPCP4 ,          
                HTTPCP5 ,HTTPCP6 ,HTTPCP7                              
     OPTIONS    LEVEL=255                                              
     WRITER     CTWTR                                                  

Run my test

When I ran my test, some System SSL messages were produced on the console from the gsksrvr address space

GSK01047I SSL component trace started for HTTPCP8/STC01000.
GSK01050I SSL component trace started for HTTPCP8/STC01000/05010022.

Stop the trace

TRACE CT,OFF,COMP=GSKSRVR

Wait until you get

GSK01041I SSL component trace ended.

from the gsksrvr address space, and stop the trace writer.

D TRACE,WTR=ALL
TRACE CT,WTRSTOP=CTWTR

This seems to take a few seconds to run. It outputs

IEF196I AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA : 
IEF196I           IBMUSER.CTRACE1 
ITT111I CTRACE WRITER CTWTR TERMINATED BECAUSE OF A WTRSTOP REQUEST.  
IEF404I CTWTR - ENDED - TIME=08.40.21                                                                 

Format the trace

You need to use IPCS to format it

  • =0 – and enter the data set name (IBMUSER.CTRACE1)
  • =6 – to get to the ipcs command screen
  • dropd – to tell IPCS to forget any historical information it may know about for the dataset
  • CTRACE COMP(GSKSRVR) full – this displays any System SSL CTRACE data
  • m PF8 – go to the bottom of the data
  • report view – to go into ISPF View mode on the data set
  • X ALL – to hide all of the data
  • f SSL_ERROR ALL – this shows any error codes
  • if you get any lines displayed, you can tab down to the hidden command and use the line prefix command f4 to display the first 4 hidden lines.

The errors I got were

 S0W1      MESSAGE   00000004  08:35:55.049451  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_ec_token_private_key_sign 
   ICSF service failure: CSFPPKS retCode = 0x8, rsnCode = 0x2b00 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.049733  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_sign_data 
   crypto_ec_sign_data() failed: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050012  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  construct_tls13_certificate_verify_message 
   Unable to generate certificate verify message: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050393  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  send_tls13_alert 
   Sent TLS 1.3 alert 51 to ::ffff:10.1.0.2.45432. 
                                                                                                              

The Alert 51 matches what my browser received.

File /usr/include/gskcms.h had #define CMSERR_ICSF_SERVICE_FAILURE 0x03353084

CSFPPKS is PKCS #11 Private Key Sign.

The return code 0x2b00 (from here) gives:

User action: You might need to re-create the token by using the PKA key token build or PKA key import callable service or regenerate the key values on another platform.

or in other words, it did not like my certificate created with NISTECC SIZE(256) but did like NISTECC SIZE(521).

One minute… what is a digital certificate?

This blog post covers

Why use certificates?

Certificates can be used as an authentication mechanism. It allows identity information to be sent over a network, and the remote end to validate the information before using it. For this to work, we need a “Certificate Authority” who can validate the certificate. When the remote end gets the “certified certificate” it checks the certification and validity and if successful can use this. Checking can involve sending an electronic request to the Certificate Authority asking if the certificate is still valid.

What is a certificate?

A certificate has several components (conceptually I think of a certificate as an envelope with several bits of paper within it).

  • A public key. See One minute explanation of public keys and private keys.
  • Information about the owner, such as name and address or organization
  • Meta information – such as it can be used for document signing (but not certificate signing), and validity dates.
  • Signing information.

Information about the owner

A certificate has information about the “owner”. Common attributes include

  • Common Name CN=Colin Paice
  • Organisation O=My Org
  • Organisational Unit OU=TEST
  • Country C=GB

Meta information.

The certificate contains information on how it should be used. For example

  • This certificate can be used as a server’s certificate – but not as a client certificate.
  • This certificate can be used for signature but not encryption.
  • This certificate can be used to sign other certificate (see below).

A server (or client) could decide to ignore some of these attributes. A proper server will honour the information, and if the client’s certificate does not have the “can be used by a client” attribute set, it will reject the certificate.

How do you trust a certificate?

If you have been sent a certificate – how do you tell if it is genuine? This was one of the problems in the early days of secure communications.

This problem was solved by having an organisation we both trust, and having this organisation “sign” a certificate.

Imagine there is a UK Certificate Authority (UKCA). To get your certificate approved…

  • you take your certificate, your passport, and proof of address to the Certificate Authority (UKCA).
  • A clerk in the CA, checks that the name in your certificate matches your passport, and the address in the certificate matches the proof of address your provided.
  • The clerk can check “the permissions” within your certificate, such as this certificate can/cannot be used as a certificate authority, or can/cannot be used as a server. The CA may not care!
  • The CA take an electronic copy of your certificate and perform a checksum calculation on the contents. If the certificate is changed in any way, the checksum calculation will be a different number.
  • The CA then encrypts the checksum calculation, prints it out, and staples the encrypted value and a copy of the CA’s public certificate to the back of your certificate.
  • The certificate has now been signed by the CA.

You send me a copy of your certificate and attachement created by the CA.

When I get a copy of your certificate (and attachment),

  • I check the CA’s public certificate that you sent me is the same as the copy I already have.
  • I do the same check sum calculation as the CA authority did.
  • I use the CA’s public certificate to decrypt the CA’s encrypted checksum.
  • The two checksum values should match!

If the checksum values match, then I can trust that the information in the certificate is the same as the information you showed to the CA when getting you certificate “signed”. Of course if you provided a fake passport, all I know is that what I see is what the CA saw.

Intermediate CAs and certificate chains

Of course it costs you money to get your certificate signed by a CA. To reduce this cost you can set up your own enterprise Certificate Authority. You create a certificate called CN=MYORGCA, and set the attribute set to allow it to sign other certificates. You take this the the UKCA and get it signed.

When Joe wants his personal certificate signed, he come to you,

  • you check Joe’s corporate ID badge has the same name as in Joe’s certificate
  • you check any other data fits within your enterprise standards, such as expiry date.
  • you use your enterprise CA certificate(CN=MYORGCA) to sign Joe’ certificate.
  • Joe get a package with
    • Joe’s certificate,
    • the checksum from the enterprise signing, and the enterprise public certificate.
    • a copy of the UKCA’s certificate and encrypted checksum for your enterprise’s CA.

When I get Joe’s certificate and attachments.

  • I can see if I have a copy of your enterprise public certificate.
  • No I do not, but I do have a copy of the UKCA public certificate,
  • I validate the your enterprise’s public certificate, using the UKCA certificate
  • If your enterprises public certificate validates successfully, I can then use it to validate Joe’s certificate.

This means I can validate certificates sent to me, as long as I have at least one of the certificates in the chain.

You could take this further and have a departmental CA authority.

Is the certificate still valid?

If someone leaves your organisation you want to ensure that any certificates issued to that person cannot be used.

You can have an expiry date in the certificate, so within a year or two the certificate will expire. This is not good enough, and you want to make the invalidate the certificate as soon as possible. This can be done through an Online Certificate Status Protocol (OCSP) attribute in the certificate. Basically this says go and ask the URL if this certificate if it is still valid.
How do you know who to ask? A certificate has url information within in it Authority Information Access: OCSP – URI:http://ColinsCert.Checker.com/ . The back end serving the URL is typically an LDAP server.

If you think about this for a few minutes this may not seem a good idea. For every secure connection handshake, you have to issue a query to a remote server somewhere to see if the certificate is still valid. This will seriously affect performance. This was solved by having a “OCSP response valid time”, sent as part of the response from the OCSP server. This known as stapling (I picture this as stapling a valid-until ticket to a certificate). The OCSP server can say “This certificate is valid, and assume it is valid for n hours”. The first time the client uses the certificate it may take longer because of the OCSP checks. For successive requests , the client can then use the certificate without doing any OCSP checks – until the time period has expired.

Do I trust the sender?

A bad actor could have copied the certificate from a valid server, and presented it to the client. The client checks it, and it looks ok, but in reality it came from a bad server.

You can include Subject Alternate Name (or SAN) in the certificate. This is the URL or IP address of the server. When the certificate is sent from the server, the client checks that the IP address of the server is in the list of SANs in the certificate. If it does not match then the certificate is rejected. Example SANs DNS:www.example.com, IP:10.1.1.2

If the DNS:name is used, this will also require a connection to a DNS to get back the IP address, which will add time to TLS hand shake to the server.

You should use SAN for the server, but not for the client. A Server will have a fixed, or limited choice of values. A client can get a random IP address and so is unlikely to match a value in the certificate.

One minute explanation of public keys and private keys.

This topic is part of the “one minute” series of posts, which give an overview of a topic with enough information to be able to understand the concepts without going too deep.

You may have a banking application on your phone which you use to communicate with the bank’s server.

Setting up a secure connection

There are several parts to establishing a secure connection:

  • You need to ensure the back-end your application is connecting to is really to your bank – and not a bad actor’s system.
  • The bank needs to check that you are who you say you are – and not a bad actor impersonating you and trying to steal your money.
  • Agree what cipher techniques you will use so a bad actor cannot read your traffic
  • Some integrity techniques so you can be sure what you send to the bank is the same data as received by the bank, and a bad actor has not replaced your traffic with some evil traffic.

All the parts above reply on encryption and decryption

Introduction to encryption and decryption.

The basic concept of encryption is you have plaintext data, a function, and an encryption key. (I think of a sausage machine, into which you plug a specially shaped key). You feed the plaintext data into the function+key and out comes encrypted data.

The basic concept of decryption is to take the encrypted data, pass it through a function and the decryption key, and out comes the original plaintext.

For some encryption, the encryption key is the same as (or similar to) the decryption key. For example Change the letters A -> P, B -> $, C -> 9…. in this case if you have the encryption key, it is easy to determine the decryption key ( P -> A, $ -> B, 9 -> C). This simple technique is not very strong, it is easy to break and find the encryption/decryption key.

There are some mathematical functions that make it very hard to find the inverse key. Even if you know the encryption key – you will not be able to guess the decryption key. (You may not be able to guess, but large governments with limitless computer resources may be able to find the decryption key). These keys tend to be very large numbers with 100’s of digits.

Public and private keys

As described above you have an encryption key, and a different decryption key. Pick one, and make it generally available to the general PUBLIC. Keep the other key very PRIVATE – because anyone with this private key can decrypt the data. Hence PUBLIC and PRIVATE keys.

  • If you have some data and encrypt it with my PUBLIC key. Only someone with my PRIVATE key can decrypt it. Knowing the public key does not allow you to decrypt it.
  • If I have some data, and encrypt with my private key, then anyone with my public key can decrypt it. So what – every one has access to my public key. If my public key decrypts the data you know that it came from me (or someone with my private key). This statement is important; it is worth reading the statement again.

If you have a private/public key pair, and I have a public/private key pair we can do interesting things:

  • I encrypt with my private key, and then encrypt with your public key. Then only you can decrypt it (with your private key), and then you decrypt with my public key – so you know it came from me – and only you can read it.
  • You encrypt with my public key, then your private key. I can decrypt it with your public key (and so I know it came from you), and then I decrypt with my private key to get the original message and I know it came from you. Note: Anyone can tell that you send me a message because they can use your public key to decrypt it, but they cannot see what is inside the message as it is still encrypted.
  • Turn around the encryption. You encrypt with your private key, then my public key. I can decrypt it with my private key and your public key (and so know it came from you). I get the original message and know it came from you. Note: No one can tell who send me the data, as they cannot encrypt it because they do not have my private key.
  • If you understand these concepts you are doing well.

Integrity checking

One use of the private/public keys is for integrity checking:

For a payload, I calculate the checksum (a complex calculation on the data, such that any change to the data will give a different checksum calculation result). I take this checksum value and encrypt it with my private key, and attach it to the data. I send the data and encrypted checksum to you. You do the same calculation. You decrypt the encrypted checkum and it should be the same as your checksum calculation. If they match you know the data came from me, and is what I sent.

A bad actor could intercept the traffic from last week, and replay it this week. The above integrity check would show the data came from me, but not that it was last weeks data. There are other techniques for this.

How do I format an encoded name from a digital certificate?

I had a string CN=myserver,O=test,C=us in a certificate, which has been encoded

Offset   :                                       EBCDIC          ASCII 
00000000 : 302F310B 30090603 55040613 02757331  ................ 0/1.0...U....us1  
00000010 : 0D300B06 0355040A 13047465 73743111  ................ .0...U....test1.  
00000020 : 300F0603 55040313 086D7973 65727665  ........._`..... 0...U....myserve  
00000030 : 72                                  .                r                 
    

where you can see the encoded string in ASCII on the right. How do I extract the string CN=myserver,O=test,C=us from this?

This format is used for Subjects and Issuers (and other types).

The hex string has an ASN.1 encoding. It usually starts with a 0x30. See here for a good introduction to ASN.1 encoding.

  • 30 means this is a sequence
  • 2f is the length of the data following
  • etc

On z/OS you can use system ssl to decode this back to a printable string. You need two steps.

  1. Convert to internal format,
  2. Create the printable EBCDIC string from the internal format.

Step1 create internal format of the data

#pragma runopts(POSIX(ON)) 
#include <gskcms.h> 
#include <gskssl.h>  
void x509_to_string(char * pData, // pointer to the data
                     int lData)    // length of the data
{ 
   x509_name    X509;   // internal format.
   int rc; 
   gsk_buffer cert;     // intermediate format
   if (lData == 0) 
   { 
      printf("no data"); 
      return; 
   } 
   cert.length=   lData; 
   cert.data  =   pData ; 
   // convert from 0x302F310B 30090603 to internal 
   rc = gsk_decode_name( 
       & cert, 
       & X509); 
   if ( rc != 0) 
   { 
     //use gsk_strerror(rc) to return a string.
     printf("\nInternal error:gsk_decode_name %s\n", 
                  gsk_strerror(rc)); 
     return; 
   } 

Convert from internal format to an EBCDIC string.

   char * pName ;       // output value   
   // convert from internal to CN=COLIN,C=GB 
   rc = gsk_name_to_dn( 
              &X509, 
              &pName); 
   // free the intermediate value regardless of rc
   gsk_free_name(&X509); 
   if ( rc != 0) 
     { 
       printf("\nInternal error:gsk_name_dn %s\n", 
                    gsk_strerror(rc)); 
       return; 
     } 
   printf("%s.",pName); 

   //free the string 
   gsk_free_string(pName); 
} 

Convert from EBCDIC string to intermediate or certificate format.

There are the reverse functions.

  • gsk_dn_to_name( pName, &X509);
  • rc = gsk_encode_name( &X509, & cert);

Both C=GB,CN=colin and c=GB,cn=colin produce

00000000 : 301D310E 300C0603 55040313 05636F6C …………..?% 0.1.0…U….col
00000010 : 696E310B 30090603 55040613 024742 .>…………. in1.0…U….GB

JCL

I needed binder input options.

INCLUDE GSK(GSKCMS31) 
INCLUDE GSK(GSKSSL) 

and JCL

//COMPILE  EXEC PROC=EDCCB, 
//       LIBPRFX=&LIBPRFX, 
//       CPARM='OPTFILE(DD:SYSOPTF)', 
// BPARM='SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31,AC=1' 
//COMPILE.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEEH.SYS.H 
//               DD DISP=SHR,DSN=&LIBPRFX..SCEEH.H 
//               DD DISP=SHR,DSN=SYS1.SIEAHDR.H 
...
//COMPILE.SYSIN  DD DSN=&SOURCE(&PROG),DISP=SHR 
//COMPILE.SYS    DD DISP=SHR,DSN=&LIBPRFX..SCEEH.SYS.H 
//COMPILE.SYSOPTF DD DISP=SHR,DSN=&SOURCE(CCOPTS) 
//* 
//* Bind the module 
//* 
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB(&PROG) 
//BIND.SYSLIB  DD DISP=SHR,DSN=&LIBPRFX..SCEELKED 
//             DD DISP=SHR,DSN=SYS1.SIEALNKE 
//BIND.GSK     DD DISP=SHR,DSN=SYS1.SIEALNKE 
...
//BIND.SYSIN DD DISP=SHR,DSN=&SOURCE(&BINDOPTS) 

Where the gskcms.h and gskssl.h are in DSN=SYS1.SIEAHDR.H or /usr/include/, and the binder stubs are in DSN=SYS1.SIEALNKE.