Collecting a wire-shark trace with TLS active for a browser

With TLS, the data transfer is encrypted. With TLS 1.3, part of the handshake is encrypted. In Wireshark this shows up as Encrypted Application Data.

You can configure wireshark to decrypt the data flow . You need to set an environment variable, and start the application from the terminal window, to pick up the environment variable.

export SSLKEYLOGFILE=$HOME/sslkeylog.log
google-chrome

This will cause the application( the google-chrome browser) to start and to write the TLS key data to the file.

Configure Wireshark to use this file:

  • Edit
  • Preferences
  • Expand the Protocols twistie
  • Scroll down to TLS ( typing T gets you near it)
  • Enter the Master-Secret log file name mine is /home/colinpaice/sslkeylog.log

Now, if you run the test you should get the data decrypted

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
   Change Cipher Spec Message
 TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
  Opaque Type: Application Data (23)
    [Content Type: Alert (21)]
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

Strong keys may be bad for your blood pressure.

I’ve just spent a couple of days trying to get a web server to use an Elliptic key with size 521. It works on Firefox, Curl, and OpenSSL, but not on Chrome. Weaker keys work of size 256 and 384 work fine, except in some cases a key size of 256 gave did not work on z/OS.

I’ve since found a page going back to 2015 discussing the dropping of several EC cipher specs. There is also discussion “NSA Suite B says that NSA uses curves P-256 and P-384” which (to some people) implied that other elliptic curves should not be used, and so were removed. There is another discussion that P256 is as strong as P521, but requires less CPU, and so P521 should not be used.

So the short answer is use Elliptic curves with key size 256 or 256, and do not use the others.

I was trying to get the Apache HTTPD server on z/OS to use a certificate, and it failed. I tried using a similar certificate from the openssl Simple server on Linux and it failed, so I’m guessing that my Chrome version (Version 114.0.5735.198 (Official Build) (64-bit)) does not support it.

The problem

In Chrome I got

This site can’t provide a secure connection 10.1.1.2 sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

This is not entirely true, as the browser sent the Alert response to the server and it was not an ERR_SSL_PROTOCOL_ERROR. After this the browser code returned its caller saying “failed” . Using Wireshark, capturing encrypted data showed the data flowing from the browser to the server to port 443),

Transmission Control Protocol, Src Port: 38188, Dst Port: 443

Transport Layer Security
  TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    Content Type: Change Cipher Spec (20)
    Change Cipher Spec Message
  TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
    Opaque Type: Application Data (23)
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

In the Apache HTTPD error.log it had

SSL0209E: SSL Handshake Failed, ERROR processing cryptography. [10.1.0.2:38188 -> 10.1.1.2:443] 

Which is not very helpful. The problem occurred on the browser; all the server sees is Alert code 51.

Taking a system SSL trace on z/OS(GSK Server) gave me

Job HTTPCP8   Process 0501003C  Thread 00000032  read_tls13_alert 
  TLS 1.3 alert 51 received from ::ffff:10.1.0.2.38188. 

Diagnostics

Using tools like gsktrace on z/OS, and Wireshark to see the flow over the network. I could see that an alert was sent from the Chrome browser to server with type Decrypt Error(51). This meant it was a problem at the browser end, not the server end.

Wireshark capturing encrypted data.

If you use TLS 1.3 once the “change cipher spec” has been issued, all traffic is encrypted, and by default Wireshark cannot read it. For example

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
     Change Cipher Spec Message
 TLSv1.3 Record Layer: Application Data Protocol: http-over-tls
   Opaque Type: Application Data (23)
     Version: TLS 1.2 (0x0303)
       Encrypted Application Data: 489be8a0976798290766c9ee158c24f5863d18
      [Application Data Protocol: http-over-tls]d

and you cannot tell this is reporting an error.

Wireshark can decrypt this. You need to set an environment variable, and start the application from the terminal window, to pickup the environment variable.

export SSLKEYLOGFILE=$HOME/sslkeylog.log
google-chrome

This will cause the application( the google-chrome browser) to start and to write the TLS key data to the file.

Configure Wireshark to use this file:

  • Edit
  • Preferences
  • Expand the Protocols twistie
  • Scroll down to TLS ( typing T gets you near it)
  • Enter the Master-Secret log file name mine is /home/colinpaice/sslkeylog.log

Now, if you run the test you should get the data decrypted

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
   Change Cipher Spec Message
 TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
  Opaque Type: Application Data (23)
    [Content Type: Alert (21)]
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

Diagnostics that were not helpful.

Chrome trace

Starting Chrome from a terminal session

google-chrome --enable-logging --v=1

The output was in ./.config/google-chrome/chrome_debug.log.

Using grep ERROR ./.config/google-chrome/chrome_debug.log gave me

...:ERROR:nss_util.cc(357)] After loading Root Certs, loaded==false: NSS error code: -8018
...:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -107
...:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -107

and no other useful information besides net_error -107. For these codes see here. 107 is an unhelpful message NET_ERROR SSL_PROTOCOL_ERROR.

Wireshark gave me decrypt error.

z/OS system SSL trace

Using GSKSRVR and CTRACE on z/OS. See here. This gave me

S0W1      MESSAGE   00000004  17:20:47.690772  SSL_ERROR 
  Job HTTPCP8   Process 0501003C  Thread 00000032  read_tls13_alert 
  TLS 1.3 alert 51 received from ::ffff:10.1.0.2.60830. 

Which shows the alert came from the browser.

Another way of getting system SSL trace.

The documentation says you can collect trace by changing the configuration. This allows you to trace just the pages/scripts that you want.

Creating a certificate on z/OS

On z/OS I used

RACDCERT ID(START1) GENCERT -                            
  SUBJECTSDN(CN('10.1.1.2') - 
             O('NISTECCTEST') - 
             OU('SSS')) -                                
   ALTNAME(IP(10.1.1.2))-                                
   NISTECC - 
   KEYUSAGE(   HANDSHAKE     )  - 
   SIZE(384) - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -               
   WITHLABEL('NISTECCTEST')     - 
                                                         

With different sizes 256,284, and 521. The IBM documentation says For NISTECC keys, valid key sizes are 192, 224, 256, 384, and 521 bits. I had problems with key size 521 bits.

On Linux I used

timeout="--connect-timeout 10"
enddate="-enddate 20240130164600Z" 
ext="-extensions end_user"
name="docec384"
key="$name.key.pem"
cert="$name.pem"

subj="-subj /C=GB/O=Doc2/CN="$name 
CA="docca256"
cafiles="-cert $CA.pem -keyfile $CA.key.pem "
rm $name.key.pem
rm $name.csr
rm $name.pem
passin="-passin file:password.file"
passout="-passout file:password.file"
md="-md sha384"
policy="-policy signing_policy"
caconfig="-config ca2.config"
caextensions="-extensions clientServer"
config="-config openssl.config"

openssl ecparam -name secp384r1 -genkey -noout -out $name.key.pem 
openssl req $config -new -key $key -out $name.csr -outform PEM -$subj $passin $passout 
openssl ca $caconfig $policy $ext $md $cafiles -out $cert -in $name.csr $enddate $caextensions 

openssl x509 -in $name.pem -text -noout|lessf

I used this as a template with different flavours of -name secp384r1

  • -name secp384r1
  • -name prime256v1 which worked
  • -name secp521r1 which did not work

You can get a list of valid names from the command openssl ecparam -list_curves.

Running a Linux web server

I tried using the openssl s_server, and could also reproduce this problem (with a much faster turnaround). I used

tls="-tls1_3  "
#cert=" -cert ./docec256.pem -key ./docec256.key.pem"
#cert=" -cert ./docecgen.pem -certform pem -key docecgen.key.pem -keyform pem" 
#cert=" -cert ./docec521.pem -certform pem -key docec521.key.pem -keyform pem"
cert=" -cert ./docec384.pem -certform pem -key docec384.key.pem -keyform pem" 
CA="-chainCAfile ./docca256.pem"
cipher1="-cipher TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384"

cipher=$cipher1
port="-port 4433 "

strict="-x509_strict -strict"
debug="-trace"

#  ca="-CAfile ./zpdt.ca.pem "
openssl s_server $port $tls  $cert $cipher $CA $debug $strict -www  

Chrome startup options

You can start chrome from a terminal with many startup options. See here

Getting a system ssl trace for httpd server on z/OS

I had a problem getting the httpd server to work on z/OS. It did not like my certificate – but reported messages about ICSF security not being set up. I got to learn a lot about traces, but could not find how to trace System SSL (GSK) and httpd.

I specified SSLTRACE in my vhost*.conf file which gave me a little information – but not at the System SSL Level.

Other applications using System SSL, can set environment variables

GSK_TRACE=0xff 
GSK_TRACE_FILE=/tmp/gskssl.%.trc 

but this does not work with http. (I think the environment variables are not passed on to any spawned thread).

You have to use the gsksrvr task, and collect the trace through CTRACE.

Set up gsk trace.

I have used gsk trace before, and described setting it up.

I had to create a parmlib member

TRACEOPTS 
          WTRSTART(ctwtr) 
          on 
          wtr(ctwtr) 
jobname(httpcp) 
          OPTIONS('LEVEL=255,JOBSUFFIX=ANY') 

I start my http server with the S HTTPCP command, and specified httpcp in the jobname of the parmlib.

I got out no trace. I tried the various jobnames until it produced a trace. My trace was produced from jobname httpcp8! I could not find a way of displaying which of my httpcp* job was used, so I had to try them all.

If I had had a long name eg httpcpxx then specifying jobname(httpcpxx) should have worked.

Starting the trace

I used

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON

where my parmlib member was CTGSKOK

It produced

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON
IEE252I MEMBER CTGSKON FOUND IN USER.Z24C.PARMLIB
GSK01040I SSL component trace started.
ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE
SUCCESSFULLY EXECUTED.

and the CTWTR started up.

What is the status of the trace?

d trace,comp=gsksrvr

gave me

IEE843I 08.32.25  TRACE DISPLAY       
  COMPONENT     MODE BUFFER HEAD SUBS                                  
 --------------------------------------------------------------        
  GSKSRVR       ON   0064K                                             
     ASIDS      *NOT SUPPORTED*                                        
     JOBNAMES   HTTPCP8 ,HTTPCP1 ,HTTPCP2 ,HTTPCP3 ,HTTPCP4 ,          
                HTTPCP5 ,HTTPCP6 ,HTTPCP7                              
     OPTIONS    LEVEL=255                                              
     WRITER     CTWTR                                                  

Run my test

When I ran my test, some System SSL messages were produced on the console from the gsksrvr address space

GSK01047I SSL component trace started for HTTPCP8/STC01000.
GSK01050I SSL component trace started for HTTPCP8/STC01000/05010022.

Stop the trace

TRACE CT,OFF,COMP=GSKSRVR

Wait until you get

GSK01041I SSL component trace ended.

from the gsksrvr address space, and stop the trace writer.

D TRACE,WTR=ALL
TRACE CT,WTRSTOP=CTWTR

This seems to take a few seconds to run. It outputs

IEF196I AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA : 
IEF196I           IBMUSER.CTRACE1 
ITT111I CTRACE WRITER CTWTR TERMINATED BECAUSE OF A WTRSTOP REQUEST.  
IEF404I CTWTR - ENDED - TIME=08.40.21                                                                 

Format the trace

You need to use IPCS to format it

  • =0 – and enter the data set name (IBMUSER.CTRACE1)
  • =6 – to get to the ipcs command screen
  • dropd – to tell IPCS to forget any historical information it may know about for the dataset
  • CTRACE COMP(GSKSRVR) full – this displays any System SSL CTRACE data
  • m PF8 – go to the bottom of the data
  • report view – to go into ISPF View mode on the data set
  • X ALL – to hide all of the data
  • f SSL_ERROR ALL – this shows any error codes
  • if you get any lines displayed, you can tab down to the hidden command and use the line prefix command f4 to display the first 4 hidden lines.

The errors I got were

 S0W1      MESSAGE   00000004  08:35:55.049451  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_ec_token_private_key_sign 
   ICSF service failure: CSFPPKS retCode = 0x8, rsnCode = 0x2b00 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.049733  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_sign_data 
   crypto_ec_sign_data() failed: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050012  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  construct_tls13_certificate_verify_message 
   Unable to generate certificate verify message: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050393  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  send_tls13_alert 
   Sent TLS 1.3 alert 51 to ::ffff:10.1.0.2.45432. 
                                                                                                              

The Alert 51 matches what my browser received.

File /usr/include/gskcms.h had #define CMSERR_ICSF_SERVICE_FAILURE 0x03353084

CSFPPKS is PKCS #11 Private Key Sign.

The return code 0x2b00 (from here) gives:

User action: You might need to re-create the token by using the PKA key token build or PKA key import callable service or regenerate the key values on another platform.

or in other words, it did not like my certificate created with NISTECC SIZE(256) but did like NISTECC SIZE(521).

One minute… what is a digital certificate?

This blog post covers

Why use certificates?

Certificates can be used as an authentication mechanism. It allows identity information to be sent over a network, and the remote end to validate the information before using it. For this to work, we need a “Certificate Authority” who can validate the certificate. When the remote end gets the “certified certificate” it checks the certification and validity and if successful can use this. Checking can involve sending an electronic request to the Certificate Authority asking if the certificate is still valid.

What is a certificate?

A certificate has several components (conceptually I think of a certificate as an envelope with several bits of paper within it).

  • A public key. See One minute explanation of public keys and private keys.
  • Information about the owner, such as name and address or organization
  • Meta information – such as it can be used for document signing (but not certificate signing), and validity dates.
  • Signing information.

Information about the owner

A certificate has information about the “owner”. Common attributes include

  • Common Name CN=Colin Paice
  • Organisation O=My Org
  • Organisational Unit OU=TEST
  • Country C=GB

Meta information.

The certificate contains information on how it should be used. For example

  • This certificate can be used as a server’s certificate – but not as a client certificate.
  • This certificate can be used for signature but not encryption.
  • This certificate can be used to sign other certificate (see below).

A server (or client) could decide to ignore some of these attributes. A proper server will honour the information, and if the client’s certificate does not have the “can be used by a client” attribute set, it will reject the certificate.

How do you trust a certificate?

If you have been sent a certificate – how do you tell if it is genuine? This was one of the problems in the early days of secure communications.

This problem was solved by having an organisation we both trust, and having this organisation “sign” a certificate.

Imagine there is a UK Certificate Authority (UKCA). To get your certificate approved…

  • you take your certificate, your passport, and proof of address to the Certificate Authority (UKCA).
  • A clerk in the CA, checks that the name in your certificate matches your passport, and the address in the certificate matches the proof of address your provided.
  • The clerk can check “the permissions” within your certificate, such as this certificate can/cannot be used as a certificate authority, or can/cannot be used as a server. The CA may not care!
  • The CA take an electronic copy of your certificate and perform a checksum calculation on the contents. If the certificate is changed in any way, the checksum calculation will be a different number.
  • The CA then encrypts the checksum calculation, prints it out, and staples the encrypted value and a copy of the CA’s public certificate to the back of your certificate.
  • The certificate has now been signed by the CA.

You send me a copy of your certificate and attachement created by the CA.

When I get a copy of your certificate (and attachment),

  • I check the CA’s public certificate that you sent me is the same as the copy I already have.
  • I do the same check sum calculation as the CA authority did.
  • I use the CA’s public certificate to decrypt the CA’s encrypted checksum.
  • The two checksum values should match!

If the checksum values match, then I can trust that the information in the certificate is the same as the information you showed to the CA when getting you certificate “signed”. Of course if you provided a fake passport, all I know is that what I see is what the CA saw.

Intermediate CAs and certificate chains

Of course it costs you money to get your certificate signed by a CA. To reduce this cost you can set up your own enterprise Certificate Authority. You create a certificate called CN=MYORGCA, and set the attribute set to allow it to sign other certificates. You take this the the UKCA and get it signed.

When Joe wants his personal certificate signed, he come to you,

  • you check Joe’s corporate ID badge has the same name as in Joe’s certificate
  • you check any other data fits within your enterprise standards, such as expiry date.
  • you use your enterprise CA certificate(CN=MYORGCA) to sign Joe’ certificate.
  • Joe get a package with
    • Joe’s certificate,
    • the checksum from the enterprise signing, and the enterprise public certificate.
    • a copy of the UKCA’s certificate and encrypted checksum for your enterprise’s CA.

When I get Joe’s certificate and attachments.

  • I can see if I have a copy of your enterprise public certificate.
  • No I do not, but I do have a copy of the UKCA public certificate,
  • I validate the your enterprise’s public certificate, using the UKCA certificate
  • If your enterprises public certificate validates successfully, I can then use it to validate Joe’s certificate.

This means I can validate certificates sent to me, as long as I have at least one of the certificates in the chain.

You could take this further and have a departmental CA authority.

Is the certificate still valid?

If someone leaves your organisation you want to ensure that any certificates issued to that person cannot be used.

You can have an expiry date in the certificate, so within a year or two the certificate will expire. This is not good enough, and you want to make the invalidate the certificate as soon as possible. This can be done through an Online Certificate Status Protocol (OCSP) attribute in the certificate. Basically this says go and ask the URL if this certificate if it is still valid.
How do you know who to ask? A certificate has url information within in it Authority Information Access: OCSP – URI:http://ColinsCert.Checker.com/ . The back end serving the URL is typically an LDAP server.

If you think about this for a few minutes this may not seem a good idea. For every secure connection handshake, you have to issue a query to a remote server somewhere to see if the certificate is still valid. This will seriously affect performance. This was solved by having a “OCSP response valid time”, sent as part of the response from the OCSP server. This known as stapling (I picture this as stapling a valid-until ticket to a certificate). The OCSP server can say “This certificate is valid, and assume it is valid for n hours”. The first time the client uses the certificate it may take longer because of the OCSP checks. For successive requests , the client can then use the certificate without doing any OCSP checks – until the time period has expired.

Do I trust the sender?

A bad actor could have copied the certificate from a valid server, and presented it to the client. The client checks it, and it looks ok, but in reality it came from a bad server.

You can include Subject Alternate Name (or SAN) in the certificate. This is the URL or IP address of the server. When the certificate is sent from the server, the client checks that the IP address of the server is in the list of SANs in the certificate. If it does not match then the certificate is rejected. Example SANs DNS:www.example.com, IP:10.1.1.2

If the DNS:name is used, this will also require a connection to a DNS to get back the IP address, which will add time to TLS hand shake to the server.

You should use SAN for the server, but not for the client. A Server will have a fixed, or limited choice of values. A client can get a random IP address and so is unlikely to match a value in the certificate.

One minute explanation of public keys and private keys.

This topic is part of the “one minute” series of posts, which give an overview of a topic with enough information to be able to understand the concepts without going too deep.

You may have a banking application on your phone which you use to communicate with the bank’s server.

Setting up a secure connection

There are several parts to establishing a secure connection:

  • You need to ensure the back-end your application is connecting to is really to your bank – and not a bad actor’s system.
  • The bank needs to check that you are who you say you are – and not a bad actor impersonating you and trying to steal your money.
  • Agree what cipher techniques you will use so a bad actor cannot read your traffic
  • Some integrity techniques so you can be sure what you send to the bank is the same data as received by the bank, and a bad actor has not replaced your traffic with some evil traffic.

All the parts above reply on encryption and decryption

Introduction to encryption and decryption.

The basic concept of encryption is you have plaintext data, a function, and an encryption key. (I think of a sausage machine, into which you plug a specially shaped key). You feed the plaintext data into the function+key and out comes encrypted data.

The basic concept of decryption is to take the encrypted data, pass it through a function and the decryption key, and out comes the original plaintext.

For some encryption, the encryption key is the same as (or similar to) the decryption key. For example Change the letters A -> P, B -> $, C -> 9…. in this case if you have the encryption key, it is easy to determine the decryption key ( P -> A, $ -> B, 9 -> C). This simple technique is not very strong, it is easy to break and find the encryption/decryption key.

There are some mathematical functions that make it very hard to find the inverse key. Even if you know the encryption key – you will not be able to guess the decryption key. (You may not be able to guess, but large governments with limitless computer resources may be able to find the decryption key). These keys tend to be very large numbers with 100’s of digits.

Public and private keys

As described above you have an encryption key, and a different decryption key. Pick one, and make it generally available to the general PUBLIC. Keep the other key very PRIVATE – because anyone with this private key can decrypt the data. Hence PUBLIC and PRIVATE keys.

  • If you have some data and encrypt it with my PUBLIC key. Only someone with my PRIVATE key can decrypt it. Knowing the public key does not allow you to decrypt it.
  • If I have some data, and encrypt with my private key, then anyone with my public key can decrypt it. So what – every one has access to my public key. If my public key decrypts the data you know that it came from me (or someone with my private key). This statement is important; it is worth reading the statement again.

If you have a private/public key pair, and I have a public/private key pair we can do interesting things:

  • I encrypt with my private key, and then encrypt with your public key. Then only you can decrypt it (with your private key), and then you decrypt with my public key – so you know it came from me – and only you can read it.
  • You encrypt with my public key, then your private key. I can decrypt it with your public key (and so I know it came from you), and then I decrypt with my private key to get the original message and I know it came from you. Note: Anyone can tell that you send me a message because they can use your public key to decrypt it, but they cannot see what is inside the message as it is still encrypted.
  • Turn around the encryption. You encrypt with your private key, then my public key. I can decrypt it with my private key and your public key (and so know it came from you). I get the original message and know it came from you. Note: No one can tell who send me the data, as they cannot encrypt it because they do not have my private key.
  • If you understand these concepts you are doing well.

Integrity checking

One use of the private/public keys is for integrity checking:

For a payload, I calculate the checksum (a complex calculation on the data, such that any change to the data will give a different checksum calculation result). I take this checksum value and encrypt it with my private key, and attach it to the data. I send the data and encrypted checksum to you. You do the same calculation. You decrypt the encrypted checkum and it should be the same as your checksum calculation. If they match you know the data came from me, and is what I sent.

A bad actor could intercept the traffic from last week, and replay it this week. The above integrity check would show the data came from me, but not that it was last weeks data. There are other techniques for this.

How do I format an encoded name from a digital certificate?

I had a string CN=myserver,O=test,C=us in a certificate, which has been encoded

Offset   :                                       EBCDIC          ASCII 
00000000 : 302F310B 30090603 55040613 02757331  ................ 0/1.0...U....us1  
00000010 : 0D300B06 0355040A 13047465 73743111  ................ .0...U....test1.  
00000020 : 300F0603 55040313 086D7973 65727665  ........._`..... 0...U....myserve  
00000030 : 72                                  .                r                 
    

where you can see the encoded string in ASCII on the right. How do I extract the string CN=myserver,O=test,C=us from this?

This format is used for Subjects and Issuers (and other types).

The hex string has an ASN.1 encoding. It usually starts with a 0x30. See here for a good introduction to ASN.1 encoding.

  • 30 means this is a sequence
  • 2f is the length of the data following
  • etc

On z/OS you can use system ssl to decode this back to a printable string. You need two steps.

  1. Convert to internal format,
  2. Create the printable EBCDIC string from the internal format.

Step1 create internal format of the data

#pragma runopts(POSIX(ON)) 
#include <gskcms.h> 
#include <gskssl.h>  
void x509_to_string(char * pData, // pointer to the data
                     int lData)    // length of the data
{ 
   x509_name    X509;   // internal format.
   int rc; 
   gsk_buffer cert;     // intermediate format
   if (lData == 0) 
   { 
      printf("no data"); 
      return; 
   } 
   cert.length=   lData; 
   cert.data  =   pData ; 
   // convert from 0x302F310B 30090603 to internal 
   rc = gsk_decode_name( 
       & cert, 
       & X509); 
   if ( rc != 0) 
   { 
     //use gsk_strerror(rc) to return a string.
     printf("\nInternal error:gsk_decode_name %s\n", 
                  gsk_strerror(rc)); 
     return; 
   } 

Convert from internal format to an EBCDIC string.

   char * pName ;       // output value   
   // convert from internal to CN=COLIN,C=GB 
   rc = gsk_name_to_dn( 
              &X509, 
              &pName); 
   // free the intermediate value regardless of rc
   gsk_free_name(&X509); 
   if ( rc != 0) 
     { 
       printf("\nInternal error:gsk_name_dn %s\n", 
                    gsk_strerror(rc)); 
       return; 
     } 
   printf("%s.",pName); 

   //free the string 
   gsk_free_string(pName); 
} 

Convert from EBCDIC string to intermediate or certificate format.

There are the reverse functions.

  • gsk_dn_to_name( pName, &X509);
  • rc = gsk_encode_name( &X509, & cert);

Both C=GB,CN=colin and c=GB,cn=colin produce

00000000 : 301D310E 300C0603 55040313 05636F6C …………..?% 0.1.0…U….col
00000010 : 696E310B 30090603 55040613 024742 .>…………. in1.0…U….GB

JCL

I needed binder input options.

INCLUDE GSK(GSKCMS31) 
INCLUDE GSK(GSKSSL) 

and JCL

//COMPILE  EXEC PROC=EDCCB, 
//       LIBPRFX=&LIBPRFX, 
//       CPARM='OPTFILE(DD:SYSOPTF)', 
// BPARM='SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31,AC=1' 
//COMPILE.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEEH.SYS.H 
//               DD DISP=SHR,DSN=&LIBPRFX..SCEEH.H 
//               DD DISP=SHR,DSN=SYS1.SIEAHDR.H 
...
//COMPILE.SYSIN  DD DSN=&SOURCE(&PROG),DISP=SHR 
//COMPILE.SYS    DD DISP=SHR,DSN=&LIBPRFX..SCEEH.SYS.H 
//COMPILE.SYSOPTF DD DISP=SHR,DSN=&SOURCE(CCOPTS) 
//* 
//* Bind the module 
//* 
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB(&PROG) 
//BIND.SYSLIB  DD DISP=SHR,DSN=&LIBPRFX..SCEELKED 
//             DD DISP=SHR,DSN=SYS1.SIEALNKE 
//BIND.GSK     DD DISP=SHR,DSN=SYS1.SIEALNKE 
...
//BIND.SYSIN DD DISP=SHR,DSN=&SOURCE(&BINDOPTS) 

Where the gskcms.h and gskssl.h are in DSN=SYS1.SIEAHDR.H or /usr/include/, and the binder stubs are in DSN=SYS1.SIEALNKE.

Setting up Linux to z/OS certificates

Several times I have had to set up certificates between Linux and z/OS and struggled for a day to get them working. Once you are familiar with doing it – it is easy. As the last time I needed to do this was over a year ago, I’ve forgotten some of the details. This blog post is to help me remember what I need to do, and to help other who struggle with this.

I’m ignoring self signed.

Basic TLS

A certificate contains

  • who it belongs to, such as CN=COLIN,O=SSS
  • the date range the certificate is valid
  • a public key
  • meta data about the key: What algorithm does the public key use, what parameters were used in the key generation, for example, algorithm=RSA, Keysize=2048.

There is a private key.

  • If you encrypt using the private key, you can use the public key to decrypt it.
  • If you encrypt using the public key, you can use the private key to decrypt it.
  • If you encrypt something with my public key, and then encrypt it with your private key. I know it came from you (or someone with your private key) and only I (or someone with my private key) can decrypt it.

Anyone can have the public key. You keep the private key secure.

Certificate Authority. This is used in validating the trust of certificates. You send your certificate to the CA, The CA does a checksum of your data, and encrypts this checksum with the CA private key. It returns your original data appended with the encrypted checksum, and information about the CA, and what was used to calculate the checksum. If someone else has the CA public key, they can do the opposite process. Do the checksum calculation, and decrypt the checksum value in the certificate, using the CA public key. If they match you know it was signed by the CA. This is known as signing the certificate.

To be able to validate a certificate sent to it, the client end needs the CA of the server end. The server needs the CA of the client end to be able to validate the client’s certificate.

During the handshake to establish the TLS connection there is a flow like

  • Establish the cipher spec to use
  • Server sends down its certificate, the client checks it
  • Servers sends down “Certificate request”, and these are the certificate(CAs) I know about
  • The client goes through it’s list of certificates (usually only one), to find the first certificate with a CA in the list sent from the server.
  • sends the client certificate to the server
    • The server checks the certificate. For example the server may be set up to accept a subset of valid algorithms, for example TLS 1.2, and Elliptic Curve. If a certificate is sent up using RSA, then this is not accepted
    • The server checks the signature of the certificate, finds the CA name, checks in the trust store for this CA, and validates the signature. Depending on the application it may check all the CA’s in the CA chain.

What do you need for the handshake to work

  • You need to have a Certificate Authority to sign certificates. In the CA certificate are some flags that say this is a CA.
  • You need to send the public key of each CA to the other end. You normally need to do this just once, and keep using the same certificates for all your TLS work.
  • You need to have a key store/trust store/keyring to hold certificates.
  • On z/OS
    • you may have a keyring for different projects, for example MQ, and TN3270.
    • You need to connect the client CA into each keyring where it will be used.
  • You need to check that the certificates are compatible with the remote end, such as Algorithm etc.

Openssl files

When using openssl, you can store common information in a configuration file. See here. This configuration file has some required options, and some optional options where you can specify common options you frequently use.

If you are using the openssl req command (for example), by default it will look for a section called [req]. This can in turn point to other sections. Using this file you can specify most of your fields in one place, and just override the specific ones.

Create a CA certificate on Linux

I have a bash file docca.sh file on Linux.

CA=”docca256″
casubj=” -subj /C=GB/O=DOC/OU=CA/CN=SSCA256″
days=”-days 1095″
rm $CA.pem $CA.key.pem

openssl ecparam -name prime256v1 -genkey -noout -out $CA.key.pem1

openssl req -x509 -sha384 -config caca.config -key $CA.key.pem2 -keyform pem -nodes $casubj -out $CA.pem3 -outform PEM $days

openssl x509 -in $CA.pem -text -noout|less4

This

  1. creates a private key (docca256.key.pem)
  2. self signs it. For any parameters not specified, it uses the configuration file caca.config and section “req” (signing request) within it.
  3. produces a public certificate in docca256.pem. This file will need to be sent to the backend servers. You can use cut and paste or FTP as ASCII.
  4. displays the x509 data

The caca.config file has

[ req ]
distinguished_name = ca_distinguished_name
x509_extensions = ca_extensions

prompt = no

authorityKeyIdentifier = keyid:always,issuer:always

[ca_distinguished_name ]
# C=GB
# O=DOC
# OU=Stromness
# CN=SSSCA4

####################################
[ ca_extensions ]

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
basicConstraints = critical,CA:TRUE, pathlen:0
keyUsage = keyCertSign, digitalSignature,cRLSign

The distinguished_name = ca_distinguished_name says go and look in the file for a section [ca_distinguished_name], and x509_extensions = ca_extensions says go and look for a section called [ca_extensions]. You can specify your own names, for example I could have used section1, and s2.

When prompt = yes, openssl takes as defaults the values in the distinguished_name section. When prompt = no, the distinguished_name is still required – but the contents of the section are ignored.

The values in the x509_extensions are defined here.

Creating an Elliptic Curve certificate on Linux

I used another bash script docecadad.sh, to document an ElliptiCal certificate for userid ADCD. It uses the CA defined above.

name="docecadcd"
key="$name.key.pem"
cert="$name.pem"
subj="-subj /C=GB/O=Doc/CN="$name
CA="docca256"
cafiles="-cert $CA.pem -keyfile $CA.key.pem "

enddate="-enddate 20240130164600Z"
passin="-passin file:password.file"
passout="-passout file:password.file"

rm $name.key.pem
rm $name.csr
rm $name.pem

#define a certificate with elliptic key with size 256

openssl ecparam -name prime256v1 -genkey -noout -out $name.key.pem 
#create a certificate request (ie hello CA please sign this)
openssl req -config openssl.config -new -key $key -out $name.csr -outform PEM -$subj $passin $passout

# sign it.

caconfig="-config ca2.config"
policy="-policy signing_policy"
extensions="-extensions clientServer"

md="-md sha384"

openssl ca $caconfig $policy $md $cafiles -out $cert -in $name.csr $enddate $extensions

# display it 
openssl x509 -in $name.pem -text -noout|less

Where the openssl.config file has

[ req ]
default_bits       = 2048

distinguished_name = server_distinguished_name
req_extensions     = server_req_extensions
string_mask        = utf8only
subjectKeyIdentifier   = hash
#extendedKeyUsage     = critical, codeSigning


[ server_req_extensions ]

subjectKeyIdentifier = hash
# subjectAltName       = DNS:localhost, IP:127.0.0.1, IP:127.0.0.6
# nsComment            = "OpenSSL"
keyUsage             = critical, nonRepudiation, digitalSignature
# extendedKeyUsage     = critical, OCSPSigning, codeSigning
subjectKeyIdentifier   = hash 

[ server_distinguished_name ]
#c=GB
#o=SSS
#cn=mqweb
  • See above for the distinguished_name value.
  • req_extensions says use the section [server_req_extensions]

The ca2.config file used to sign it has

HOME            = .
RANDFILE        = $ENV::HOME/.rnd

####################################################################
[ ca ]
default_ca    = CA_default      # The default ca section
####################################################################
[ CA_default ]
default_days     = 1000         # How long to certify for
default_crl_days = 30           # How long before next CRL
#default_md       = sha1       # Use public key default MD
default_md       = sha256       # Use public key default MD
preserve         = no           # Keep passed DN ordering

x509_extensions = ca_extensions # The extensions to add to the cert

email_in_dn     = no            # Don't concat the email in the DN
copy_extensions = copy          # Required to copy SANs from CSR to cert

#defaults
base_dir      = .
certificate   = $base_dir/cacert.pem   # The CA certifcate
private_key   = $base_dir/cakey.pem    # The CA private key
new_certs_dir = $base_dir              # Location for new certs after signing
database      = $base_dir/index.txt    # Database index file
serial        = $base_dir/serial.txt   # The current serial number

unique_subject = no  # Set to 'no' to allow creation of
                     # several certificates with same subject.


####################################################################
[ ca_extensions ]

subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always, issuer:always
basicConstraints       = critical,CA:TRUE, pathlen:0
keyUsage               = nonRepudiation

####################################################################

[ signing_policy ]
countryName            = optional
stateOrProvinceName    = optional
localityName           = optional
organizationName       = optional
organizationalUnitName = optional
commonName             = supplied

[ clientServer ]

keyUsage               = digitalSignature, keyAgreement, digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName         = DNS:localhost, IP:127.0.0.1, 
extendedKeyUsage       = serverAuth,clientAuth
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always, issuer:always
nsComment  = "clientserver"

The policy (my [ signing_policy] ) must have entries in it to create a valid Subject Distinguished name. Without it, I got a strange RACF code (0x0be8044d).

Send the CA to z/OS and import it

You need to send the CA public certificate to z/OS. This file looks like

-----BEGIN CERTIFICATE-----                                      
MIIFbTCCA1WgAwIBAgIUJw+gLLSFxqCyTdIyEUWyQ/g9JnEwDQYJKoZIhvcNAQEL.
...
-----END CERTIFICATE----- 

You can FTP the file, or create the file and use cut and paste. The file needs to be a sequential dataset with format VB. My file is VB, lrecl=256,blksize=6233. For the FTP I used

put docca256.pem ‘colin.docca256.pem’

You need to import this into RACF, and connect it to the keyrings.

//IBMRACF  JOB 1,MSGCLASS=H                                     
//S1  EXEC PGM=IKJEFT01,REGION=0M                               
//SYSPRINT DD SYSOUT=*                                          
//SYSTSPRT DD SYSOUT=*                                          
//SYSTSIN DD *                                                  
RACDCERT CHECKCERT('COLIN.DOCCA256.PEM') 
/*                       

The CHECKCERT gave me

Certificate 1:                                                          
                                                                        
  Start Date: 2022/10/09 11:45:43                                       
  End Date:   2025/10/08 11:45:43                                       
  Serial Number:                                                        
       >782A62948699FF3FB00238FB296E4A647B7DF07C<                       
  Issuer's Name:                                                        
       >CN=SSCA256.OU=CA.O=DOC.C=GB<                                    
  Subject's Name:                                                       
       >CN=SSCA256.OU=CA.O=DOC.C=GB<                                    
  Signing Algorithm: sha384ECDSA                                        
  Key Usage: HANDSHAKE, CERTSIGN                                        
  Key Type: NIST ECC                                                    
  Key Size: 256                                                         
                                                                        

Which matches what I expected, and gave me information about the certificate – ECC, 256, and signed with SHA 384 ECDSA, (from the -sha384 parameter above).

Define it to RACF and connect it to the user’s keyring

racdcert list  (label('Linux-CA256')) CERTAUTh 
RACDCERT DELETE  (LABEL('LINUXDOCCA256'))    CERTAUTH             
              
RACDCERT ADD('COLIN.DOCCA256.PEM') -                            
          CERTAUTH  WITHLABEL('LINUXDOCA256') TRUST 

RACDCERT ID(START1) CONNECT(RING(MQRING)-        
                            CERTAUTH     -       
                            LABEL('LINUXDOCCA256'))                     

If you delete a certificate, then it is removed from all keyrings. Once you have re-added it you need to reconnect it to all the keyrings. If you list the label (racdcert list (label(‘Linux-CA256’)) certauth) it will display where it is used, so you can read it.

Download the z/OS CA certificate

I downloaded the z/OS exported certificate in .pem format. it looks like

-----BEGIN CERTIFICATE-----                                       
MIIDYDCCAkigAwIBAgIBADANBgkqhkiG9w0BAQsFADAwMQ4wDAYDVQQKEwVDT0xJ  
... 
+9TRng==                                                          
-----END CERTIFICATE-----

You can use ftp or cut and paste. I created doczosca.pem.

Use it!

Before I could use it, I had to set up the server’s certificate, and download the z/OS CA certificate

I set up a bash script scl2.sh

name=docecadcd
host="-connect 10.1.1.2:4000"
CA="--CAfile doczosca.pem "

openssl s_client $host  $CA -cert $name.pem -certform PEM -key $name.key.pem -keyform PEM
 

Setting up z/OS for TLS clients

There is a lot of configuration needed when setting up TLS(SSL) between a server and a client. There are many options and it is easy to misconfigure. The diagnostic information you get when the TLS handshake fails is usually insufficient to identify any problems.

You need the following on z/OS:

  • One or more Certificate Authority certificates. You can create and use your own for testing. If you want to work with external sites you’ll need a proper (external) CA, but for validation and proof of concept you can create your own CA. You could set up a top level CA CN=CA,O=MYORG, and another one (signed by CA=CA,O=MYORG), called CN=CA,OU=TEST,O=MYORG. Either or both of the public CA certificates will need to be sent to the clients in imported into their keystore.
  • A private/public key, signed by a CA, (such as signed by CA=CA,OU=TEST,O=MYORG).
  • The private key is associated with a userid.
    • The signing operation takes the data (the public key), does a hash sum calculation on the data, encrypts this hash sum, and stores the encrypted hash value, and CA public certificate with the original data. To check the signature, the receiving application compares the CA with its local copy, if that matches, does the same checksum calculation, decodes the encrypted hash sum – and checks the decrypted and locally calculated values match.
    • A certificate is created using one from a list of algorithms. (For example, Elliptic Curve, RSA). When the certificate is sent to the client, the client needs to support the algorithm. Either end can be configured, for example, to support Elliptic Curve, but not RSA.
  • A keyring to contain your private key(s) – this can also contain CA public certificates of the partners (clients or servers).
  • A “site” keyring (public keystore, or trust ring) which holds the public CA certificates of all the other sites you work with. If you have only one keyring per user or application, you need to update each of them if you need to an a new CA to your environment. Many applications are only designed to work with one keyring. Java applications tend to have a key store(for the private key) and a trust store for the CAs.
  • Some applications can support more than one private certificate on a keyring. The certificate needs to match what the client can support.
  • For certificates which are sent to your server, you need a copy of the CA(s) used to sign the incoming certificate. If you have a copy of the CA, then you can validate any certificate that the CA signed. This means you do not have to have a copy of the public certificate of every client. You just need the CA.
    • Some application need access to just one CA in the chain, other applications need access to all certificates in the CA chain.
  • As part of the TLS handshake
    • the client sends up a list of the valid cipher specs it supports (which algorithms, and size of key)
    • the server sends down a subset of the list of cipher spec to use (from the client’s list)
    • the server can also send down its certificate, which contains information such as the distinguished name CN=zSERVER, OU=TEST, O= MYORG, and host name.
    • the client can validate these names – to make sure the host name in the certificate matches the host, and what it was expecting.
    • if requested, the client can send up its certificate for identification. The server can validate the certificate, and can optionally map it to a userid on the server.
  • A userid can be given permission to read certificate in another user’s keyring. A userid needs a higher level of authority to be able to access the private key in another id’s keyring.

Create the Certificate Authority

//IBMRACF  JOB 1,MSGCLASS=H                               
//S1  EXEC PGM=IKJEFT01,REGION=0M                         
//SYSPRINT DD SYSOUT=*                                    
//SYSTSPRT DD SYSOUT=*                                    
//SYSTSIN DD * 
RACDCERT certauth LIST(label('DOCZOSCA')) 
RACDCERT CERTAUTH DELETE(LABEL('DOCZOSCA'))               
RACDCERT GENCERT  -                                         
  CERTAUTH -                                                
  SUBJECTSDN(CN('DocZosCA')- 
             O('COLIN') -                                   
             OU('CA')) - 
  NOTAFTER(   DATE(2027-07-02  ))-                          
  KEYUSAGE(   CERTSIGN )  -      
  SIZE(2048) -                                              
  WITHLABEL('DOCZOSCA') 
/*
//                 

This certificate is created against “user” CERTAUTH. Keyusage CERTSIGN means it can be used to sign certificates. “user” CERTAUTH is often displayed internally as “irrcerta”.

Once it has been created the certificate should be connected to every ring that may use it, see below.

Export the CA certificate to a file so, clients can access it

RACDCERT CERTAUTH EXPORT(LABEL('DOCZOSCA')) -
  DSN('IBMUSER.CERT.DOC.CA.PEM') -
  FORMAT(CERTB64) -
  PASSWORD('password')

The file looks like

-----BEGIN CERTIFICATE-----                                        
MIIDYDCCAkigAwIBAgIBADANBgkqhkiG9w0BAQsFADAwMQ4wDAYDVQQKEwVDT0xJ   
TjELMAkGA1UECxMCQ0ExETAPBgNVBAMTCERvY1pvc0NBMB4XDTIyMTAwOTAwMDAw 
...  

This can be sent to the clients, so they can validate certificates sent from the server. This file could be sent using cut and paste, or FTP.

Create the keyring for user START1.

The instructions below lists the ring first – in case you need to know what it was before you deleted it”

RACDCERT LISTRING(TN3270)  ID(START1) 

RACDCERT DELRING(TN3270) ID(START1) 

RACDCERT ADDRING(TN3270) ID(START1)                                                          

RACDCERT LISTRING(TN3270)  ID(START1) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 

Connect the CA to every keyring that needs to use it

RACDCERT ID(START1) CONNECT(RING(TN3270) - 
                            CERTAUTH LABEL('DOCZOSCA'))

Create a user certificate and sign it on z/OS

This creates a certificate and gets is signed – as one operation. You can create a certificate, export it, sent it off to a remote CA, import it, and add it to a userid.

RACDCERT ID(START1) DELETE(LABEL('NISTECC521')) 
                                                                
RACDCERT ID(START1) GENCERT -                                   
  SUBJECTSDN(CN('10.1.1.2') - 
             O('NISTECC521') -                                  
             OU('SSS')) -                                       
   ALTNAME(IP(10.1.1.2))-                                       
   NISTECC - 
   KEYUSAGE(HANDSHAKE) - 
   SIZE(521) - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -                      
   WITHLABEL('NISTECC521')     -                                
                                                                
RACDCERT id(START1) ALTER(LABEL('NISTECC521'))TRUST             

RACDCERT ID(START1) CONNECT(RING(TN3270) -                      
                            ID(START1)  -                       
                            DEFAULT  - 
                            LABEL('NISTECC521') )               
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                    
RACDCERT LIST(LABEL('NISTECC521' )) ID(START1)                  
RACDCERT LISTRING(TN3270)  ID(START1)                           

This creates a certificate with type Elliptic Curve (NISTECC) with a key size of 521. It is signed with the CA certificate created above.

The ALTNAME, is a field the client can verify that the Source Name in the certificate matches the IP address of the connection.

It is connected to the user’s keyring as the DEFAULT. The default certificate is used if the label of a certificate is not specified when using the keyring.

Give a user access to the keyring

PERMIT START1.TN3270.LST CLASS(RDATALIB)  -    
    ID(COLIN )  ACCESS(UPDATE )                          
SETROPTS RACLIST(RDATALIB) refresh                       
  • Update access give userid COLIN access to the private key.
  • Read access only gives access to the public keys in the ring.

You would typically give a group of userids access, not just to individual userids.

Import the client’s CA’s used to sign the client certificates

This is the opposite to Export the CA certificate to a file so clients can access it above.

Copy the certificate to z/OS. This can be done using FTP or cut and paste.

Use it!

I used it in AT-TLS

TTLSConnectionAdvancedParms       TNCOonAdvParms 
{ 
 ServerCertificateLabel  NISTECC521
 ...
} 
TTLSSignatureParms                TNESigParms 
{ 
   CLientECurves Any 
} 
TTLSEnvironmentAction                 TNEA 
{ 
  HandshakeRole                       ServerWithClientAuth 
  TTLSKeyringParms 
  { 
    Keyring                   start1/TN3270 
  } 
...
} 

Getting my z/OS client to talk to a remote server over TLS

I looked into this because someone asked me about using the mail client on z/OS talking to a mail server somewhere in the internet.

My z/OS running on zPDT under Linux did not have a working connection to the outside world.

Configuring Linux to pass the z/OS traffic to the external network.

A ping to the server, got no response back.

Using Wireshark on the wireless connection, I could see my request from z/OS coming via Linux and going to the outside network. The source IP address was 10.1.1.1 – a “local only” address, and so the server could not send a response back to me.

The routing through Linux worked because I had a default route; ip route gave me

default via 192.168.xxx.xxx4 dev wlxd037450ab7ac proto dhcp metric 600 
10.1.0.0/24 dev eno1 proto kernel scope link src 10.1.0.3 metric 100 
10.1.0.0/24 via 10.1.0.3 dev eno1 proto static metric 100 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1 

To get the “right” IP address passed to the external network, was surprisingly easy. It was documented in the zPDT book.

I created a script

echo ‘Your firewall must be enabled for this command to be meaningful’
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -F FORWARD
iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o wlxd037450ab7ac -j MASQUERADE
# iptables -I INPUT -p tcp --dport 3270 -j ACCEPT

and used sudo to execute it.

I started the firewall

sudo ufw status
sudo ufw enable

I could then successfully ping the address

When I used Wireshark on the connection between Linux and z/OS, the source IP address was 10.1.1.2. When I used Wireshark on the wireless connection between Linux and the external network, the IP address was 192.168.xxx.xxx. This shows the effect of the iptables commands, it changed the 10.1.1.2 to a 192.168.xxx.xxx – and back again on the replies.

Getting the server’s certificate

I configured the AT-TLS definitions, for TLS support, specifying the choice of cipher specs I wanted to be used, and specified the keyring.

My z/OS client connected to the remote server. I could see the “client hello” flow, and the “server hello” response. In the server hello flow were the CA certificates the server supported. The client validates the certificates by checking the client’s key store.

I did not have the matching CA certificate in my keystore, and so my client failed to validate the certificate sent down, and the connection ended.

I could do it properly, and get the certificate from the official source for the server, but this was a test system, and I was happy to trust what was sent down in the handshake.

You can use wireshark or openssl to get the certificate

With openssl

openssl s_client -showcerts -tls1_2 -starttls smtp -connect smtp.gmail.com:587 | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’

You’ll need to interupt it (Ctrl-C) and paste the data into a file (google.pem)

You can display the contents of the certificate using

openssl x509 -in google.pem -text -noout|less

With Wireshark

display the certificates in the Server Hello flow.

right click on the certificate and use right click, export packet bytes. I created a file user.crt. This is a binary file.

The openssl command (on Linux)

openssl x509 -inform der -in user.crt -text > usercert.pem

displays the file in .pem format.

Put the certificate into the keyring

If you are using something that needs a keyring, you need to import the certificate into RACF and connect it to a keyring. If you are using something that uses openssl, such as Python, you need a .pem file in Unix Services.

I created a file on z/OS (VB) COLIN.USERCERT.PEM, and copied the .pem file into it (between and including the BEGIN CERTIFICATE and END CERTIFICATE lines).

I imported it, and connected it to my keyring using

//IBMRACF  JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=*                                           
//SYSTSPRT DD SYSOUT=*                                           
//SYSTSIN DD *                                                   
RACDCERT CHECKCERT('COLIN.USERCERT.PEM')                         
RACDCERT DELETE  (LABEL('USERTRUST.ORG'))    CERTAUTH            
RACDCERT ADD('COLIN.USERCERT.PEM')        -                      
   CERTAUTH  WITHLABEL('USERTRUST.ORG') TRUST 
RACDCERT ID(START1) CONNECT(RING(TN3270) - 
                            CERTAUTH    - 
                            LABEL('USERTRUST.ORG')) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                     
                                                                 
RACDCERT LISTRING(TN3270)  ID(START1)                            
/*                                                              

Once I had refreshed my PAGENT job, and restarted my client code, I was able to establish a TLS session.

Client certificate

In the TLS handshake, the mail servers did not send down a request for the client certificate. As no certificate was requested, or sent up to the server, no certificate authentication was done. You need to check your mail server for it authentication process – it may just be userid and password. This information is sent once the TLS session has been established – so it flows encrypted.

Debugging AT-TLS session problems

I deliberately misconfigured AT-TLS to see how easy it would be to identify and resolve the problems from an AT-TLS perspective. It turned out worse than I expected. There is little information on the z/OS to help you.

I configured TTLSEnvironmentAction {trace 255 } (see the bottom of this blog) and refreshed the PAGENT. I had configured SYSLOGD so records for *.TCPIP.*.* went to /var/log/TCPIP.

I reran my MQ client application and got

  • from MQ on Linux, in file /var/mqm/errors/AMQERR01.LOG return code 2393 (MQRC_SSL_INITIALIZATION_ERROR).
  • On Linux there was a file /var/mqm/trace/AMQ.SSL.TRC – which only IBM can format!
  • From TCPIP on z/OS EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900 JOBNAME: CSQ9CHIN RULE: REMOTE-TO-CSQ1 USERID: START1 GRPID: 0000001B ENVID: 0000000B CONNID: 0000006E This give
    • the address of my client,
    • the name of the chinit
    • which AT-TLS rule was used

The message EZD1287I TTLS Error RC: 402 Initial Handshake pointed me to Cryptographic Services System Secure Sockets Layer Programming – No SSL cipher specifications. The first reason was

The client and server cipher specifications do not contain at least one value in common. Client and server cipher specifications might be limited depending on which System SSL FMIDs are installed. See Cipher suite definitions for more information. Server cipher specifications are dependent on the type of algorithms that are used by the server certificate (RSA, DSA, ECDSA, or Diffie-Hellman), which might limit the options available during cipher negotiation.

MQ Trace

I took an MQ trace and formatted it. I used grep to find which file had “Cipher” in it.

Within this file I searched for Start of GSKit TLS Handshake Transcript.

This had information sent to the server as part of the handshake, and further down it had the reason code. You can see from the example that the fields and their values have been displayed (so cipher spec 003c is displayed as tls_rsa_with_aes_128_cbc_sha256)

Start of GSKit TLS Handshake Transcript (1119 bytes)
 <client_hello>
 client_version 
 TLSV12
 random 
   gsksslDissector_32Bits
   7f9d66d8
   gsksslDissector_Opaque
   Length: 28
   3E 5B 45 66 EE A3 C1 9F FB 81 0C 2F 38 19 DF 95     >[Ef......./8...
   5A 1B 54 CC B8 CB B6 C9 87 39 5E 88                 Z.T......9^.
 session_id 
 Length: 00
 cipher_suites 
 Length: 04
 00 FF 00 3C                                         ...<
 tls_ri_scsv,tls_rsa_with_aes_128_cbc_sha256
 compression_methods 
 Length: 01
 00                                                  .
 Extensions
 Length: 74
 00 0D 00 18 00 16 06 01 05 01 04 01 03 01 02 01     ................
 06 03 05 03 04 03 03 03 02 03 02 02 00 00 00 2A     ...............*
 00 28 00 00 25 73 79 73 74 65 6D 32 65 2D 64 65     .(..%system2e-de
 66 32 65 2D 73 76 72 63 6F 6E 6E 2E 63 68 6C 2E     f2e-svrconn.chl.
 6D 71 2E 69 62 6D 2E 63 6F 6D                       mq.ibm.com
  Extension Count: 2
  signature_algorithms 13
   rsa:sha512,rsa:sha384,rsa:sha256,rsa:sha224,rsa:sha1,  
   ecdsa:sha512,ecdsa:sha384,ecdsa:sha256,ecdsa:sha224,
   ecdsa:sha1,dsa:sha1
  server_name 0
   system2e-def2e-svrconn.chl.mq.ibm.com
End of GSKit TLS Handshake Transcript
{  rriEvent
 ...
 RetCode = 20009665, rc1 = 420, rc2 = 0, Comment1='SYSTEM.DEF.SVRCONN', 
 Comment2='gsk_secure_soc_init', Comment3='10.1.1.2(1414)'
 ...
}

With this trace, I am able to see what was sent to z/OS.

The AT-TLS Trace

The trace ( configured in syslogd to be in /var/log/TCPIP) had a one line entry with (I’ve reformatted it to make it easier to read).

Map CONNID: 0000006B 
LOCAL: 10.1.1.2..1414 
REMOTE:
10.1.0.2..53898 
JOBNAME: CSQ9CHIN 
USERID: START1 
TYPE: InBound 
STATUS: Enabled 
RULE: REMOTE-TO-CSQ1 
ACTIONS:
CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A

and data

RC: 0 Connection Init
Initial Handshake ACTIONS: CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A HS-Server
RC: 0 Call GSK_SECURE_SOCKET_OPEN - 00000052FD6228F0
RC: 0 Set GSK_FD(300) - 000000000000006B
RC: 0 Set GSK_USER_DATA(200) - 000000007EC32430
RECV CIPHER 160303007B 

and one loooong record with

RECV CIPHER 
010000770303749ED51D8DC7794EE6AC36B01FD115F38A4B0812D35 
C80A5F95DB840C35735CA00000400FF003C0100004A000D00180016 
060105010401030102010603050304030303020302020000002A002 
800002573797374656D32652D64656632652D737672636F6E6E2E63 
686C2E6D712E69626D2E636F6D 
SEND CIPHER 15030300020228 

From the AT-TLS trace of the data received from the client, it is the data as received, and has not been split down into useful fields.

I could not find any documentation on how to format this string. It is not easy to create a program to format this (and get it right), for example converting cipher spec 003c to TLS_RSA_WITH_AES_128_CBC_SHA256. However I have a REXX exec which works in ISPF and decodes the data into fields, but not the contents of the fields – so the cipher spec is reported as 003c

I had some success taking this data, and creating a file which Wireshark could process. See Wireshark – using external data: Bodging a hex dump file. This was not always successful, as it looks like the data is truncated, and can have non hex data in the hex stream.

Note, the System SSL server started task, GSKSRVR, can capture System SSL trace. The output is like

Job TCPIP     Process 0101001D  Thread 00000001  read_v3_client_hello            
Received CLIENT-HELLO message                                                  

with no detailed information

Tracing just one session

If you have a busy system you could get trace data for many sessions. You may want to set up a TLS rule, so you use a “debug port”, or you specify the remote host IP address, and port, using information from the error message

EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900

And dont forget…

And do not forget to reset the TTLSEnvironmentAction entry to reset the trace, and to refresh the PAGENT.