Category: TLS

How do I look at TLS 1.3 handshakes?

With TLS before 1.3, encryption of the data on the session occurred after the handshake had completed, so the handshake was visible in Wireshark. With TLS 1.3 the traffic is encrypted after the “Client Hello”, so you cannot immediately see the remainder of the handshake.

Tools like OpenSSL, can write out the magic data needed for decryption. For example

openssl s_client -keylogfile /tmp/kl -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile /home/colinpaice/ssl/ssl2/colinpaice.pem

This writes information to the specified file, in my case /tmp/kl.

The file has data like

SSL/TLS secrets log file, generated by OpenSSL
SERVER_HANDSHAKE_TRAFFIC_SECRET 05a42762…
EXPORTER_SECRET 05a42762dc…
SERVER_TRAFFIC_SECRET_0 05a4276…
CLIENT_HANDSHAKE_TRAFFIC_SECRET 05a42762dc…
CLIENT_TRAFFIC_SECRET_0 05a42762d…

On Linux (Ubuntu) you can tell Wireshark to use this through

edit -> preferences -> protocols -> SSL -> (pre)-master-secret log filename

Different versions of Wireshark have TLS or SSL, use whichever one is available to you.

Specify the name of your file (/tmp/kl in my case) and Wireshark will be able to decrypt the data.

Setting up certificate authentication in LDAP.

This started off as part of a small task, when I had half an hour gap before lunch. The whole end-to-end of getting TLS and LDAP, with certificate authentication took me several weeks to set up. Now I know the traps, it takes about 10 minutes.

I describe setting up TLS and LDAP (without certificate authentication) here. Get that working before trying certificate authentication.

Setting up the simplest case of an RSA certificate on the client and an RSA certificate on the server, was pretty easy to set up. Using an Elliptic Curve certificate to and RSA certificate on the server seems impossible, it eventually worked!

I created “What cipher specs should I use?” because most of my problems, were due to using the wrong cipher specs, or the right cipher specs, but it the wrong order!

Logging on

You can logon to LDAP and specify a userid (DN) and password, for example

ldapsearch -h 127.0.0.1 -D “cn=Admin, o=Your Company” -w secret -b “o=Your Company” “(objectclass=*)” aclEntry

If you use -w ? it will prompt for your password, so it is not visible.

You can also use a certificate to logon, so you do not need the password, you just need the private key (or in my case the USB dongle with my encrypted Hardware Security Module(HSM) keystore on it).

Note: If you have your TLS private key in a file, and people can copy that file, they can impersonate you! You need to protect the file, bearing in mind your corporate IT department may be able to view any backups etc that you have. Someone would need to steal my USB dongle to use my private key and logon.

Understanding the TLS 1.2 handshake and authentication.

Skip to first steps if you are keen to implement without understanding the background,

There are several stages to establishing a TLS connection and authentication.

  1. Agree the protocols for setting up the session, for example which sort of encryption, and the key size. This provides the privacy on the connection.
  2. The server sends down its certificate, and the client authenticates it
  3. The client sends up its certificate and the server authenticates it
  4. The Distinguished Name(DN)from the client certificate is looked up in the z/OS security manager, and the associated userid is looked up.
  5. The DN is used to look in the access control lists (ACLs) do determine the access the requester has to the data.

Agree the protocols

The client the sends the protocols it supports to the server. This is a list of numbers, and each number has a meaning.

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_GCM_SHA384 (0xc087)
…
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
…

The part before the WITH covers

  1. The technique used to agree a key (ECDHE, DHE,RSA)
  2. The key type Elliptic Curve (ECDSA) or RSA

There must be at least one key type matching the certificate type.

  • If you have an Elliptic Curve server certificate, then you need to have records with TLS…ECDSA…
  • If you have an RSA server certificate you need to have records with TLS…RSA…

At the server, the list sent from the client is merged with the list from the server, for example if the server had

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0250039

and the client sent C024, C025, C02C, then common elements are C024, C025. These two cipher specs are TLS_ECDHE_ECDSA_WITH… so the server’s certificate needs to be ECDSA – or an Elliptic Certificate.

If no entries match you get a message like “no cipher found”.

Authenticate the server

The server sends information down to the client.

  1. The cipher suite to be used.
  2. Information for setting up the encryption of the traffic for the session.
  3. The server’s certificate, and any CA certificates in the chain.
  4. The client then validates the certificate
    1. It use the signature algorithm in the certificate. This is the information after the “WITH” in the cipher spec. In my client’s certificate it has signing algorithm sha256WithRSAEncryption. There must be a cipher spec in the list ending in TLS…RSA_WITH… ending in SHA256
    2. It checks the signature of the CA, by checking its signing algorithm in the GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) See here.

Send the client certificate to the server

If the client certificate is wanted, the servers sends the “certificate request”.

  1. It sends down the certificate type it accepts, for example RSA, ECDSA, DSS. The client then sends its certificate of that type. If the there is no match, no certificate is sent. Note:For a long time I could not get a client certificate with Elliptic Curve, to work when the server had an RSA certificate. This was due to bad parameters on the z/OS end.
  2. There is a list of signature Hash Algorithms (eg rsa_pkcs1_sha256 (0x0401) ecdsa_secp256r1_sha256 ( 0x0403).
  3. It lists the Distinguished Names of the Certificate Authorities or Self Signed certificate.
  4. With the certificate type,the signature Hash and the Certificate Authorities, the client looks in its keystore for a certificate matching these parameters; then sends the first certificate that matches – or sends “no certificate”.
  5. It sends the Certificate Verify Signature Algorithm to the server. For my Elliptic Curve this was ecdsa_secp256r1_sha256 (0x0403). This has to be in the GSK_TLS_SIG_ALG_PAIRS in the z/OS LDAP environment file. (It took me days to find this problem!)

The server authenticates the client’s certificate

The server looks inside the certificate to see how the certificate was signed, for example using

openssl x509 -in ecec.pem -text -noout

This gave Signature Algorithm: ecdsa-with-SHA256. The server needs the matching cipher spec in the GSK_V3_CIPHER_SPECS_EXPANDED environment variable.

The client’s certificate will have been signed – either by a CA, or self signed. The Signature algorithm of the CA must be in the GSK_TLS_SIG_ALG_PAIRS environment variable. The values are defined here.

For example my CA had Signature Algorithm: ecdsa-with-SHA384. This requires 0503 (SHA-384 with ECDSA) to be in the GSK_TLS_SIG_ALG_PAIRS list. If this was missing I got

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

from ldapsearch on Ubuntu. It took me another day or so to find this problem.

As you can see there are a lot of things you need to configure and to get right before it works!

First steps – using a client certificate

Firstly set up the TLS session so you can use certificates to connect to LDAP. This took me about a week, because of configuration problems, but finally it worked. I was able to connect from Ubuntu to z/OS.

Set up TLS between the client and the z/OS server, as described here.

Setup Ubuntu

I created an RSA certificate on Ubuntu using a shell script.

name=”rsaca256″
ca=”carsa1024″

openssl genpkey -out $name.key.pem -algorithm RSA -pkeyopt rsa_keygen_bits:2048

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name -passin file:password.file -passout file:password.file
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

I set up an Elliptic Curve

enddate=”-enddate 20220409174001Z”

password=” -passin file:password.file -passout file:password.file”
name=”ecec”
rm $name.key.pem
rm $name.csr
rm $name.pem
ca=”ca256″ # sign with this

openssl ecparam -name secp256r1 -genkey -noout -out $name.key.pem

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name $password
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

openssl x509 -in $name.pem -text -noout|less

# Some tools need a .p12 file of the certificate
#openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $ca.pem -chain -name $name -passout file:password.file -passin file:password.file

Openldap uses a configuration file, for example ldaprc.

I set up my ldaprc configuration file on Ubuntu.

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_REQCERT demand
TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem
TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem

Where

  • TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem was the z/OS CA certificate exported from z/OS and downloaded. It is needed to verify a certificate sent from z/OS.
  • TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem this file contains the private key
  • TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem this file contains the public key
  • You can specify TLS_CIPHER_SUITE high to use the listed cipher specs. See the openssl ciphers command, for example
    • openssl ciphers -v -V -s -tls1_2
    • openssl ciphers -v -V -s -tls1_3
    • I omitted TLS_CIPHER_SUITE for my testing.

You need the distinguished name (DN) of the certificate. You can display the subject distinguished name DN using the command openssl x509 -in secp521r.pem -text -noout. This gave me output which included:

Subject: C = GB, O = cpwebuser, CN = secp521r

I sent the Linux CA file, ca256.pem, to z/OS.

Set up z/OS

Configure LDAP.

In my LDAP configuration file I had

sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING
sslCertificate ZZZZ
commThreads 10
allowAnonymousBinds off
sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

The sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED says look in the environment file, and take the value from there. As this is the default, you can omit it from the file.

The default GSK_V3_CIPHER_SPECS_EXPANDED value in the environment file is 003500380039002F00320033 which maps to

  1. 0035 TLS_RSA_WITH_AES_256_CBC_SHA
  2. 0038 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
  3. 0039 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  4. 002f TLS_RSA_WITH_AES_128_CBC_SHA
  5. 0032 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
  6. 0033 TLS_DHE_RSA_WITH_AES_128_CBC_SHA

Which are all very old (part of deprecated SSL, TLS 1.0, TLS 1.1) so you need to set this variable in your environment file. See the next section for what I used.

Update your environment file

I use the following value

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023130313011302

This supports

  1. C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  2. C02B TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  3. C030 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  4. C02F TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  5. C024 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
  6. C023 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  7. 1303 TLS_CHACHA20_POLY1305_SHA256 TLS 1.3
  8. 1301 TLS_AES_128_GCM_SHA256 TLS 1.3
  9. 1302 TLS_AES_256_GCM_SHA384 TLS 1.3

Put the strongest certificates first. For example, the following did not work.

GSK_V3_CIPHER_SPECS_EXPANDED=0035….C02F

It caused 0035 to be used. The Certificate-Types list had only RSA and DSS, and did not include ECDSA, so the client was not able to send its Elliptic Curve certificate.

When I used

GSK_V3_CIPHER_SPECS_EXPANDED=C02F….0035

it caused C02F to be used. This caused the certificate-types list to include the ECDSA, and so the client could send its Elliptic Curve certificate. This problem took me a week to find!

I specified

GSK_TLS_SIG_ALG_PAIRS=0601050104010301080608050804

For the certificate checking algorithms. These are for SHA = 224, 256, 384, 512 and certificate types = RSA, ECDSA, and DSA.

Add the CA from Linux and add it to the keyring

The Linux CA was uploaded to z/OS as COLIN.CARSA102.PEM.

Add it to the RACF keystore as a CERTAUTH certificate, and connect it to the keyring.

RACDCERT CHECKCERT('COLIN.CARSA102.PEM') 

RACDCERT DELETE (LABEL('LINUXCA')) CERTAUTH

RACDCERT ADD('COLIN.CARSA102.PEM') -
  CERTAUTH WITHLABEL('LINUXCA') TRUST

RACDCERT ID(START1) CONNECT(RING(MQRING)-
           CERTAUTH  LABEL('LINUXCA'))

SETROPTS RACLIST(DIGTCERT) REFRESH

Map the Distinguished Name to a z/OS userid.

You have to map the DN to a userid. In the example below, it maps the DN in the SDNFILTER to the userid ADCDA. This userid must be valid, for example if the userid is revoked, the authentication will fail (with an unhelpful message).

* remove any previous instance
RACDCERT DELMAP(LABEL('LINUXSecp2' )) ID(ADCDA)

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

RACDCERT MAP ID(ADCDA )  - 
   SDNFILTER('CN=secp521r.O=cpwebuser.C=GB')  - 
   WITHLABEL('LINUXSecp2') 

RACDCERT LISTMAP ID(ADCDA) 

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

Note the ‘.’ between the attributes, and the attributes are squashed up, in the right order.

I did not need to refresh LDAP as the change was automatically picked up.

I could now logon to LDAP using the certificate – but could do nothing.

Give the DN access to LDAP

I gave it access, using ldapmodify passing the file with

dn: o=Your Company
changetype: modify
add: aclEntry
aclEntry : access-id:CN=SECP521R,O=CPWEBUSER,C=GB:
 object:ad:normal:grant:rscw:sensitive:rsc

Note: The DN can be in mixed case, or upper case.

I could now logon and issue queries using a certificate for authentication and authorisation.

Tracing problems

You can get trace output to the job log using the command

f GLDSRV,debug error+CONNS+LDAPBE

This traces

  • any errors
  • CONNS shows the cipher spec being used
  • LDAPBE shows the DN, and SAF user etc being used.

You can turn trace off with

f GLDSRV,debug 0

You can get the low level trace by using the gsktrace=0xff in your environment file.

Using z/OS LDAP with TLS

This started as a mini project before lunch, and now, two weeks later, I have got all of the bits to work.

As I was struggling with this, I wondered why no one had crossed this swamp before me. Is it because no one wants to cross it, or it is too difficult to do so? I think it was just difficult to do so – but I can now provide a map!

It took me a couple of hours to get LDAP on z/OS to use TLS from my Linux machine. The challenge was finding the right options. I then tried using Elliptical Certificates, and got this working, then was partially successful in using TLS 1.3.

As part of this I wrote Authentication using certificates with LDAP, which is where it was really difficult.

For clients, I used ldapsearch (from openldap ) on Ubuntu, and ldapsearch on z/OS.

See the product documentation here.

I’ve put a section “What cipher specs should I use” in a separate blog post.

Configure the LDAP environment file

I had to add TLS V1_2 and TLSV1_3 (and disable the deprecated TLS 1.1)

GSK_PROTOCOL_TLSV1_1=off
GSK_PROTOCOL_TLSV1_2=on
#GSK_PROTOCOL_TLSV1_3=on

to the environment file, because they default to off. Without these options it defaults to SSL – which is so old, my certificates and keys did not support it. I disabled TLS_1_1 because it is old.

I added

GSK_TLS_SIG_ALG_PAIRS=0601060305010401030108060805080405030403

This is used when validating certificates. The signature algorithm of the CA must be in this list see GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) (see here), get things working, then remove the ones I did not want to use; rather than try to add the ones I wanted to use.

I initially added

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0251303130113020039

This is a list of the cipher specs it will accept. You should try to restrict the certificates in your organisation, to Elliptical Certificates, and a limited configuration for any CA’s. For example have

  • all client’s certificates defined as Elliptic Curve type prime256v1, signed with ecdsa-with-SHA256.
  • All internal CAs defined as Elliptic Curve type prime256v1 signed with ecdsa-with-SHA256.

When using certificate authentication I needed the following for it all to work

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023

The environment variables

GSK_SERVER_TLS_KEY_SHARES
GSK_CLIENT_TLS_KEY_SHARES

these are used only when using TLS 1.3. I’ll blog about using TLS 1.3 at a later date.

While you are investigating problems you can use the environment variable GSK_TRACE=0xff to write a trace. You can specify GSK_TRACE_FILE to direct the trace to a particular file. The default location is /tmp/gskssl.%.trc where % is the thread id. You format it with

gsktrace /tmp/gskssl.65567.trc > out

oedit out

I used

GSK_TRACE=0xff
GSK_TRACE_FILE=/tmp/gsktrace.ldap

So I could have a shell script to format and display the trace.

gsktrace gsktrace.ldap > out
oedit out

Change the LDAP configuration file.

I used

sslCipherSpecs all
sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING

I specified a port for ldaps. You can use

ldapsearch -H ldaps://10.1.1.2:389

This is preferred to specifying host and port

ldapsearch -h 10.1.1.2 -p 389

A port can process TLS and non-TLS requests, so definition for port 1389 may not needed.

I specified a keyring sslKeyRingFile START1/MQRING. This is owned by a different userid to the started task, so needs more RACF definitions. See below for the definitions.

Define the keyring and keyring

I defined an RSA certificate and an EC certificate. I controlled which was used from the configuration file.

* create the ring
RACDCERT DELRING(MQRING) ID(START1)
RACDCERT ADDRING(MQRING) ID(START1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the server's RSA certificate

RACDCERT ID(START1) DELETE(LABEL('ZZZZ'))
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('ZZZZ') - 
             O('SERVER') - 
             OU('SSS')) - 
   ALTNAME(IP(10.1.1.2))- 
   SIZE(4096)- 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   RSA - 
   WITHLABEL('ZZZZ') - 
   KEYUSAGE(         DATAENCRYPT,DOCSIGN,HANDSHAKE         ) 

RACDCERT id(START1) ALTER(LABEL('ZZZZ'))TRUST 
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('ZZZZ') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the Elliptic Certificate

RACDCERT ID(START1) DELETE(LABEL('SERVEREC')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('SERVEREC') - 
             O('ADCD') - 
             OU('TEST')) - 
   ALTNAME(IP(10.1.1.2)) - 
   SIZE(521) - 
   NISTECC - 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   KEYUSAGE(HANDSHAKE         ,KEYAGREE) - 
   NOTAFTER(   DATE(2024-12-29))- 
   WITHLABEL('SERVEREC') 
RACDCERT id(START1)  ALTER(LABEL('SERVEREC'))TRUST
                                                        
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('SERVEREC') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 
                                                             
RACDCERT LISTRING(MQRING) ID(START1)

Permit the LDAP server’s userid to access the private certificate in the keyring

Because the LDAP userid does not own the keyring, the server userid needs update access to be able to extract the private key.

RDEFINE RDATALIB START1.MQRING.LST UACC(NONE) 
PERMIT START1.MQRING.LST CLASS(RDATALIB) ID(GLDSRV) -
    ACCESS(CONTROL)
SETROPTS RACLIST(RDATALIB) REFRESH

For a userid to be able to use its own keyring, it only need read access to be able to use the private certificate.

Set up the client userid

I set up the userid in the z/OS LDAP server, so I could use the z/OS userid and password. See here.

Test it out

LDAP checks the TLS parameters, when a TLS request arrives. It does not check at startup . I used openssl s_client to connect, and display information about the TLS handshake. See Debugging TLS – an easier way.

Setting up the client on Ubuntu

I set up a file ldaprc containing the ldap configuration. You can enter all of the parameters on the command line, but the ldaprc file makes it easier. My file had

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_PROTOCOL_MIN 3.2
TLS_REQCERT demand
#TLS_CERT /home/colinpaice/ssl/ssl2/rsaca256.pem
#TLS_KEY /home/colinpaice/ssl/ssl2/rsaca256.key.pem

The #TLSCERT etc are ignored. No certificate is sent to the server (unless the -Y external option is set).

Enable GSK trace on z/OS

I enabled an LDAP trace to //SYSPRINT on z/OS, using f GLDSRV,debug error+CONNS+LDAPBE

Where

  • error reports errors
  • Conns reports connection information, such as the cipher spec used.
  • LDAPBE reports on the communication to the back end, for example it reports the SAFUSER

Run a query on Linux

I used ldapsearch on Ubuntu

ldapsearch -H ldaps://10.1.1.2:1389 -D “cn=ibmuser, o=Your Company” -w panthe0n -b “o=Your Company” “(objectclass=*)” aclEntry

The trace on z/OS GLDSRV included

CONNS srv_ssl_connect()644: SSL Cipher Specs
00000000: c3f0f0f8 *C008 *

and we can see that SSL was used, and cipher spec C008 was used.

Because I had set up my userid to map use a SAF userid, in the trace I had

LDAPBE srv_process_bind_request()939: do_return_bind msgID=1, connID=8, bindDN=‘CN=IBMUSER, O=YOUR COMPANY’, safUserID=’IBMUSER’, dnList=0x0, grpList=0x0, rc=0

Debugging TLS – an easier way

I had been trying to get an IBM product on Linux to talk to LDAP using TLS and certificate authentication; where I give a certificate instead of a LDAP userid and password. Basically I got “computer say no”.

IBM products use GSKIT to manage keystores and TLS between sessions. With Java you can get out a trace of the TLS conversation. You can get out a trace on z/OS, but not on other platforms. After a day I stumbled on a different approach.

Gettting a gskit trace on z/OS.

Getting the trace was easy, understanding it was harder.

In the LDAP environment file I added GSK_TRACE=0xff. Once I had TLS to LDAP working, I used GSK_TRACE=0x04 to trace just errors. Some “errors” are recorded in the GSKTRACE as “INFO”. For example an Elliptic curve is only supported in TLS 1.3.

By default the trace goes to /tmp/gskssl.%.trc. where % is the thread id.

To format it go into USS and use the command gsktrace gskssl.83951642.trc > out, and edit the file “out”.

The linux end.

I found a great program openssl s_client. This starts a TLS handshake and prints out in easy to understand format, what is going on. For example

The command

openssl s_client -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -x509_strict

gave the flow of the data.

stderr output

The stderr output included

depth=1 O = COLIN, OU = TEST, CN = COLIN4Certification Authority
verify error:num=19:self signed certificate in certificate chain

This is the name of the CA.

Following this was an error message. Note: it is at the top of the output – not the bottom as I expected.

SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1528:SSL alert number 80

See below for a discussion about the error.

stdout output

Certificate chain
  0 s:O = SERVER, OU = SSS, CN = ZZZZ
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority
  1 s:O = COLIN, OU = TEST, CN = COLIN4Certification Authority 
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority

This says

  • the certificate SERVER was signed by COLIN4.Certificate Authority
  • the certificate COLIN4Certificate Authority was self signed

The server certificate comes next

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Followed by the identity of the server

subject=O = SERVER, OU = SSS, CN = ZZZZ
issuer=O = COLIN, OU = TEST, CN = COLIN4Certification Authority

Next comes the list of Certificate Authority or self signed certificate in the keyring 

Acceptable client certificate CA names
O = ADCD, OU = TEST, CN = MQWEB2 
O = COLIN, OU = TEST, CN = COLIN4Certification Authority
C = GB, O = SSS, OU = CA, CN = SSCARSA1024
C = GB, O = SSS, OU = CA, CN = SSCA256

Any client certificate must be signed by one of those CAs (or self signed certificates).

Then comes the client certificate types the server will accept

  • Client Certificate Types: RSA sign, DSA sign

Note: This does not include ECDSA, so elliptic certificates are not supported in this configuration.

After this is the signature algorithms the server will accept

  • Requested Signature Algorithms: RSA+SHA512: ECDSA+SHA512: RSA+SHA384: ECDSA+SHA384: RSA+SHA256: ECDSA+SHA256: DSA+SHA256: RSA+SHA224: ECDSA+SHA224: DSA+SHA224: RSA+SHA1: ECDSA+SHA1: DSA+SHA1

It gives information on the session so far

SSL handshake has read 2493 bytes and written 1537 bytes
Verification error: self signed certificate in certificate chain

and information about the key sent down. It was defined with SIZE(4096) RSA.

New, SSLv3, Cipher is AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated

and about the TLS session

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA
    Session-ID: 0401001A0A0100029...
    Session-ID-ctx: 
    Master-Key: 32D5B4AD162F0403E323DB0...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1634115022
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no

The -msg option gave me 

>>> ??? [length 0005]
    16 03 01 01 31
>>> TLS 1.3, Handshake [length 0131], ClientHello
    ...
<<< ??? [length 0005]
    ... 
<<< TLS 1.3, Handshake [length 0051], ServerHello
    ...
<<< TLS 1.2, Handshake [length 0839], Certificate
    ...
<<< TLS 1.2, Handshake [length 0123], CertificateRequest
    ...
<<< TLS 1.2, Handshake [length 0004], ServerHelloDone
    ... 
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 021c], Certificate
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0206], ClientKeyExchange
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0050], CertificateVerify
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, ChangeCipherSpec [length 0001]
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0010], Finished
    ...
<<< ??? [length 0005]
    ...
<<< TLS 1.2, Alert [length 0002], fatal internal_error
    02 50

The fatal errors are described here. 0x02 is fatal error, 0x50= 80 = internal error. internal_error(80). Which is not very helpful.

In the LDAP log I got

GLD1116E Unable to initialize an SSL connection with 10.1.0.2: 434 – Certificate key is not compatible with cipher suite.

In the gsktrace on z/OS I got

cms_validate_certificate_mode_int(): Validating CN=ecec,O=cpwebuser,C=GB ENTRY
INFO get_issuer_certificate(): Using issuer CN=SSCA256,OU=CA,O=SSS,C=GB

ERROR read_v3_certificate(): Client certificate key type 13 not allowed for SSL V3 cipher 0x0035

  • Client certificate key type 13. Looking in /usr/include/gskcms.h under X.509 data types, was x509_alg_ecPublicKey.
  • SSL V3 cipher 0x0035 is listed here, as TLS_RSA_WITH_AES_256_CBC_SHA = 256-bit AES encryption with SHA-1 message authentication and RSA key exchange.
  • x509_alg_ecPublicKey is elliptic public key – which is incompatible with RSA key exchange.

When I used an RSA certificate

openssl s_client -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/rsaca256.pem -key /home/colinpaice/ssl/ssl2/rsaca256.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -verify_return_error -policy_print -x509_strict

it worked

Whoops I left my private key unprotected.

I had set up my certificates and private key so I could use them to work with a web server on z/OS. I set them up on Linux and copied them to z/OS. My tests all worked. It wasn’t till this morning when I thought, those test should have failed.

With Python and the requests package you can send REST requests to a backend server, using digital certificates for authentication. You configure which private key to use, but you are unable to specify the password for the private key. As this worked, there must be no password on the key file – whoops.

This is a bit like leaving your front door unlocked with a note “If we are not here, please come in and make yourself at home”.

To generate a private key I had used

openssl ecparam -name prime256v1 -genkey -noout -out $name.key.pem

but there is no way of specifying a password. With hindsight it was obvious there was no password protection.

I had to use

openssl ecparam -name prime256v1 -genkey -noout | openssl ec -out $name.key.pem -passout file:password.file

Where the first part of the command generated a private key, and wrote it to the output stream. The second part does nothing with the data, but wrote it to the output file using the password in password.file.

When I used this private key, the python requests failed because it needed a password!

I should have remembered

I remember helping with some security testing many years ago.

  • Userid1 was able to use resource1, good, that worked
  • Userid2 was able to use resource2, good, that worked
  • Userid1 was able to use resource2, good ,that worked – hang on… that should have failed.

The person customising security had missed one configuration step, so all users had access to all resources, and the test plan did not have any negative testing.

Using jConsole and with z/OS Liberty web server

I wanted to get some monitoring information out from z/OSMF using jConsole on my Ubuntu machine. Eventually this worked, but I had a few problems on the way. The same technique can be used for base Liberty, MQWeb, z/OSMF and ZOWE all of which are based on Liberty.

Configuring z/OSMF

I changed the z/OSMF configuration to include

<featureManager> 
  <feature>restConnector-2.0</feature> 
</featureManager>

and restarted the server.

In the stdout (or message log) will be something like

CWWKX0103I: The JMX REST connector is running and is available at the following service
URL: service:jmx:rest://sss.com:10443/IBMJMXConnectorREST

You need the URL. The message above gave service:jmx:rest://sss.com:10443/IBMJMXConnectorREST, but I needed to use service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST .

The port number came from the httpEndpoint with id=”defaultHttpEndpoint” . I have another httpEndpoint with port 24993, and this also worked with jConsole.

Set up jConsole

I set up a script for jConsole

k1='-J-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/adcdc.p12'
k2='-J-Djavax.net.ssl.keyStorePassword=password'
k3='-J-Djavax.net.ssl.keyStoreType=pkcs12'
t1='-J-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/zca.jks'
t2='-J-Djavax.net.ssl.trustStorePassword=password'
t3='-J-Djavax.net.ssl.trustStoreType=jks'
d='-J-Djavax.net.debug=ssl:handshake'
d=' '
de='-debug'
de=' '
s='service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST'
jconsole $de $s $k1 $k2 $k3 $t1 $t2 $t3 $d

Where

  • the -J .. parameters are passed through to java,
  • the -Djava… are the standard set of parameters to define the key stores on the Linux

Running this script gave a pop up window with

Secure connection failed. Retry insecurely?

The connection to service:jmxLret://10.1.1.2:10443/IBMJMXConnectorREST could not be made using SSL.

Would you like to try without SSL?

This was because of the exception

java.io.IOException: jmx.remote.credentials not provided.

I could not see how to pass userid and password to jConsole.

I then used Cntrl+N to create a new connection and entered Username: and Password: which jConsole requires. After a short delay of a few seconds jConsole responded with a graphs of Heap Memory Usage, and Threads in use. You can then select from the Measurement Beans.

The TLS setup

In the keystore I had a certificate which I had used to talk to a Liberty instance before.

This was signed, and the CA certificate had been imported into the key trust keyring on z/OS, for that HttpEndPoint.

The server responded with a server certificate (“CN=SERVER,O=SSS,C=GB”) which had been signed on z/OS. The signing certificate had been exported from z/OS and downloaded to Linux

I created a jks key trust store using this certificate, using the command

keytool -importcert -file temp4ca.pem -keystore zca.jks -storetype jks -storepass password

and used this trust store to validate the server certificate sent down from z/OS.

This worked with jConsole.

I created a pkcs12 keystore using keytool

keytool -importcert -file temp4ca.pem -keystore zca2.p12 -storetype pkcs12 -storepass password

Which also worked.

Problems using a .p12 trust store

I used

runmqakm -keydb -create -db zca.p12 -type pkcs12 -pw password
runmqakm -cert -add -file temp4ca.pem -db zca.p12 -type pkcs12 -pw password -label tempca
runmqakm -cert -details -db zca.p12 -type pkcs12 -pw password -label tempca

to create a pkcs12 keystore and import the z/OS CA certificate. The -details option displayed it.

When I tried to use it, jConsole produced the message (after the Cntl+N)

Secure connection failed. Retry insecurely?

The connection to service:jmxLret://10.1.1.2:10443/IBMJMXConnectorREST could not be made using SSL.

Would you like to try without SSL?

I used Ctrl-N as before, and got the same message.

Using

d=’-J-Djavax.net.debug=ssl:handshake’

and rerunning the script, produced a TLS trace. At the bottom was

VMPanel.connect, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

%% Invalidated:…
VMPanel.connect, SEND TLSv1.2 ALERT: fatal, description = internal_error

Using a trace at the server, gave the unhelpful , SEND TLSv1.2 ALERT:

Using openssl also failed. Create the .p12 keystore

openssl pkcs12 -export -out zca.p12 -in temp4ca.pem -name zCA -nokeys

and rerun the jconsole script, and it failed the same way.

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

It looks like runmqakm and openssl do not create a valid trust store with an imported certificate.

Additional diagnostics

When the trust store created by keytool was used; at the top of the TLS trace output was

System property jdk.tls.client.cipherSuites is set to ‘null’
…
Ignoring disabled cipher suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
trustStore is: /home/colinpaice/ssl/ssl2/zca.p12
trustStore type is : pkcs12
trustStore provider is :
init truststore
adding as trusted cert:
Subject: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Issuer: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Algorithm: RSA; Serial number: 0x0
Valid from Tue Jul 14 00:00:00 BST 2020 until Fri Jul 02 23:59:59 BST 2021

keyStore is : /home/colinpaice/ssl/ssl2/adcdc.p12
keyStore type is : pkcs12
keyStore provider is :
init keystore
init keymanager of type SunX509

When the runmqakm or openssl was used, the green entries were missing.

When I used runmqakm to create the pkcs12 keystore

runmqakm -cert -details -db zca.p12 -type p12 -pw password -label tempca

listed the certificate successfully.

When I used keytool to list the contents

keytool -list -keystore zca.p12 -storetype pkcs12 -storepass password
Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 0 entries

When I created the key store with keytool, both runmqakm and keytool displayed the certificate.

The problem looks like Java is only able to process the imported CA certificates when keytool was used to create the trust store.

When the burglar alarm goes off – would you just mute it?

I’m sure you’ve all been told to ignore links like

“You have wun $1,000,000 click hear to download a dodgy application to steal your money” or

“We have been unable to deliver a parcel to you. Click here to rearrange delivery and pay the excess money” but how about
“Your connection is not private, Net::err_cert_invalid – click here to accept it“

Collect 10 points if you did not fall for any of these.

I would like to talk about the last one, about your connection is not private; because I saw a web site recommending “click here to accept”.

You get messages like this if there are problems with the certificate. Common problems are

  • Your browser does not recognise the certificate, because it has not been signed by any CA in your browser’s trust store. This looks like someone is sending you an invalid certificate. It may be your enterprise security team need to update all browsers with the missing certificate, but they should have spotted this problem not you.
  • The certificate sent from the server has “valid host addresses are…. 10.1.1.2, my sys.com” and this has been sent from a different site. This could be because bad guys have intercepted your request and played “Man in the middle” between you and the server.
  • Less common, is where the certificate has parameters which your browser cannot handle. For example very old cipher specs, and your browser only supports TLS 1.2 or above.
  • It could be caused by a server using a self signed certificate. It may be acceptable to use this in a test environment, but it is not secure – as anyone could have created it. If it is signed, then you can check the digital signature.

All of these should set off the intruder alarm.

If you get a message like this – do not click on the accept button. This is just like muting the burglar alarm, and ignoring the warning – bad guys could steal your personal details – and you would not know.

Many web servers are configured with a bad certificate, for example they may specify an alternate name of 127.0.0.1, but not “localhost”. It is much easier (though wrong) to tell people to accept the certificate, than to fix the certificate.

Do not trust it

The blog post went on to say that you can download the certificate which has been sent to you and install it in your key store. I felt like screaming “NO!” and attacking the screen with a hammer!

Imaging, twenty years ago, you were a bank manager, and a man comes in, and says “I am a policeman, and we are doing a survey about your bank’s safe and where you keep the money. I would like my colleague “fingers” to go down inspect it. He has to work anonymously, which is why he is wearing a mask, and gloves, and he is wearing a black and white striped shirt. I’ll come back tomorrow”. You let “fingers” go down to the safe on his own, and say goodbye to the nice policeman. An hour later you find your safe is empty and there is no sign of the two men.

Your first mistake was to accept the policeman was a policeman. You should have checked his credentials before doing anything else. If he was fake, then any identification papers would be fake. You need to talk to your contacts in the local police station, and check if your visitor is valid or bogus (while he is sitting in your office).

Only in rare situations, such as a test environment, would you extract the certificate from what was sent to you, and install it in your key store. If you install it, then later, another request can come in from the bad guys, and be validated by the CA you just installed.

If you need a CA, you go through the proper corporate process. For example as part of the overnight maintenance on your machine, or download it securely from a known site. Emailing may not be good enough if your machine has been hacked.

So, in summary, today’s lesson is

  1. Do not “accept” bad certificates when using web site. Fix the the server.
  2. Get any additional certificates through an approved process – do not accept what the server is giving you.

In the words of the Police Constable in Dixon of Dock Green. “Evening all – mind how you go”.

Overview and challenges of using end to end message encryption (AMS)

MQ provides end to end protection of messages. This covers

  • Integrity – where the message can still be read – but any changes to the message are detected. You can check the id of the person that sent it.
  • Encryption – where the payload is encrypted and only the ids with the correct private key can decrypt it.
  • Privacy – A combination of Integrity and Encryption, where only specified ids can decrypt it, and you can check the id of the person that signed it.

The messages are protected on the network, in memory and on disk. You do not need to use a channel with TLS for protection (but you may want to use TLS for authentication), as the data is protected before sending.

Any headers that are involved in ‘routing’ like MQMD, XQH, RFH2, etc are skipped by AMS because the queue manager still needs to know what to do with the messages in terms of delivery and selection before they are unprotected. Other message headers like PCF command headers and custom application defined headers/formats can still be protected. Sensitive data should not be held in message properties as that data is used in routing/selection. Thanks to Jon Rumsey for the correction to this.s

Background about encryption and signing of data

You may need to read some of the following section more than once, because it reference private keys and public keys and it is hard to work out what is happening.

The encryption techniques used are special. There is a private key and a public key. (Think of your passport as a private key, and your name and address is your public key). You can use functions

  • You do encrypt(private key, payload) to produce an encrypted payload, send it to me, and then I use decrypt(public key, encrypted payload) to recreate the original payload. I know it came from you, because I needed your public key to decrypt it.
  • I do encrypt(public key, payload) to produce an “encrypted” payload, and send it to you. Only you can use decrypt(private key, encrypted payload) with your private key to recreate the original payload.

With this

  • I can encrypt data using your public key that only someone with access to the private key (you) can decrypt
  • I encrypt with my private key, and anyone with my public key can decrypt it – and you know it came from me – because you needed my public key.

Signing is simply taking a checksum of the data, and encrypting the value with the signer’s key. When you come to check the signature, you do the same check sum calculation, decrypt the “signed” value using my public key, and the values should match. The payload would look something like “Signed by CN=Colin,O=SSS,C=GB” encrypted checksum=2294567, my certificate; followed by the original data.

We can take this further. I sign the data with my private key, and then encrypt the data(including the signed data) with your public key, (keep up at the back there – I said it was hard to follow). When you process it, only you can decrypt it because it needs your private key, and you can check the signed data, and see it came from me. I have sent the message encrypted, and have mutual authentication.

With encryption a symmetric key is used to encrypt the data, because it is faster, and about as secure as encryption with an asymmetric key. When trying to break into enciphered data, the more data you have, the easier it is to break. This is why you can reuse the symmetric key for multiple records, but you should change it, for example after 10 MB has been processed (I don’t know the best figure – it is all a matter of risk).

Sending a message, many to many.

Sending a message from one person to another person looks fairly easy. It gets a bit harder when you have potentially many people in my organisation who are allowed to send a protected message to your organisation, and there may be many people who are allowed to get, and decrypt it.

Let the people in my organisation be CN=A,O=MYORG, and CN=B, O=MYORG, and the people in your organisation be CN=X,O=YOURORG, CN=Z,O=YOURORG.

If person CN=A wants to send an encrypted message so it can be decrypted by CN=X or CN=Z the processing is as follows.

  • Do a checksum calculation on the record.
  • Encrypt this checksum using CN=A’s private certificate.
  • Put a new field of CN=A,O=MYORG|| encrypted checksum value, CN=A’s public certificate before the record.
  • Create a symmetric key.
  • Encrypt the (signed data and original payload) with the symmetric key.
  • Take the symmetric key and encrypt it for CN=X,O=YOURORG using CN=X’s public certificate. Create a field of the Distinguished Name || encrypted key.
  • Take the symmetric key and encrypt it for CN=Z,O=YOURORG using CN=Z’s public certificate. Create a field of the Distinguished Name || encrypted key.
  • Build a record of the two “encryption fields” and the encrypted data
  • Put the data in a message and send it to the recipient(s).

Getting an encrypted message

The person with CN=X gets the message.

  • Scan down the message looking for CN=X,O=YOURORG. If found then extract the encrypted symmetric key. Decrypt this value with the CN=X’s private key. This gives you the key that was used to encrypt the customer data.
  • Decrypt the message using the key you just decrypted.
  • Extract the first field and see it is signed data. We can see that the checksum was done by CN=A,O=MYORG.
  • Check this name with your list of IDs that can send you messages.
  • Validate the signer’s certificate. Either use a CA, or if the certificate was self signed, check with the copy in your trust store. Note this certificate includes the public key.
  • Take the checksum value in the message and decrypt it using CN=A,O=MYORG’s public key from the payload. Do the checksum of the remainder of the message and compare the values – they should match.

Setting up your enterprise

You need to know the list of potential recipients of messages for each queue.

At the receiving end, if you want to check the message came from an authorised id, you need to know the list of potential senders.

If you’ve managed to keep up so far – it gets worse!

The management of these lists can be difficult. If someone joins my team, I would need to tell the other organisations that they need to add CN=NEWPERSON, O=MYORG into their list of potential senders, (and remove CN=IVE_RETIRED, O=MYORG. It will take a week to get this change through their change management system. Until this is done, the message could be rejected as coming from an unauthorised person.

In the same way, you need to update the AMS setmqspl definitions to add CN=NEWPERSON, O=MYORG to each queue, and when people join or leave the other enterprises you’ll need to update your list of potential senders.

Do not worry too much – there is a solution, see below.

Creating the policies

With MQ, the security policy is defined using the SETMQ Security PoLicy command (setmqspl) for example to define a queue at MYORG to send message to YOURORG. The -a option is for who the message is for

setmqspl -m PRODA -p SENDQUEUE -s SHA256 -e AES128
-a "CN=X, O=YOOURORG"
-a "CN=Y, O=YOOURORG"
-a "CN=Z, O=YOOURORG"

At YOURORG for the GETQUEUE you need the following. The -r option is who you can receive a message from

setmqspl -m YOQM -p GETQUEUE 
-r "CN=A, O=MYORG"
-r "CN=B, O=MYORG"
-r "CN=JO, C=GB,O=SOMEOTHERORG"

To change an entry you have to run whole command, you cannot add one “-a” or “-r”. I would keep a file for these, update file, and process the whole file. For two queues – use two files.

You also have to consider how to update the definitions for multiple queue managers. On z/OS I would have the MQ commands inline, and use a variable queue manager name &QM, which you set in the JCL. For midrange you could have a shell script and pass the queue manager name as a parameter.

What keys do I need?

The encryption is done using RSA, and so needs a certificate with an RSA private key.

When the message is signed

Each id putting a message needs access to a private key – this could be one key shared by multiple users.

When getting a message that has been signed, the recipients will need to verify the certificate sent within the signing data; either a CA in the signing chain, or the public certificate (for self signed certificates).

When encrypting the message

The id putting the message needs access to the public key of each recipients in the list.

The id getting a message which has been encrypted, needs access to the private key.

Maintaining the public keys

For each person in the -r or -a list you will need the digital certificate, and public key in a keystore or a keyring (on z/OS).

Each id on z/OS will need their own keyring. There is a keyring for the queue manager;s AMS address space with the public certificates on it.

On midrange MQ, each user has a keystore, with a combination of public keys, and private key(s).

Time to step back and think about what you want.

To me, it is clear that having every possible user having their own DN, and certificate quickly gets impractical. If you are working with people outside of your organisation do you want them to know the people’s names within your organisation?

You may want to have a departmental certificate for example CN=PAYROLL,O=MYORG, which members of your payroll team all use. You can control access to the queue using standard MQ security policies.

An intermediate configuration would be to have a proxy or server applications to change input messages from “partner protection” to “myorg protection”, and the reply messages changed from “myorg protection” to “partner protection”

  • You have an AMS protected input queue called EXTERNAL_IN. This has been configured to specify the DNs of the systems that can send you messages, so might specify CN=RECEIPTSPAYABLE,O=YOURORG as a valid sender.
  • If the get is successful, then put to a queue called INTERNAL_IN, specifying the DNs of the individuals in your Accounts department who are allowed to get one of the protected messages. If someone joins the Accounts department, you only need to change the configuration within your queue manager.
  • Someone (or an automated task) gets the message from INTERNAL_IN queue, it is decrypted automatically, the application does the processing and puts a message the AMS protected queue YOUR_BANK_INTERNAL, where it is encrypted by AMS.
  • Another proxy/server application in your enterprise gets the messages from YOUR_BANK_INTERNAL, checks the DN of the sender, and that they were authorised to put messages to the queue.
  • The server then puts to YOUR_BANK_EXTERNAL, using a DN of “CN=PAYROLL,O=MYORG”, and it is encrypted for the recipient.

This way you can isolate internal processing and external processing. Updating the internal list is within your control. You should not need to update the external lists (and certificates) once established.

Handling whoops’s

Some problems will result in the messages removed from the queue and put to a special error queue. You need to have a process in place to handle this. I had messages put to this queue when a userid tried to get an encrypted message, but the message had not been encrypted for this userid’s DN.

Other AMS blog posts

pkcs11? pkcs12? .cms? .jks? .p12? .nss? which type of keystore should I use?

There are at least four types of keystore, and after an email exchange with someone, it became clear that the different types of keystore and how to use them, are not widely understood. It is one of those topics, that once you are an expert, this topic is obvious still difficult to understand. When you have little knowledge and just want to get a job done, it can be very confusing.

I’ve learned a lot from writing this post

  • There is no one tool to manage all your keys and certificate
  • You may need one keystore for your queue manager, another keystore for your C clients, another keystore for your Java clients, a keystore for a web browser and something else for curl!
  • You might need to use 3 tools to manage your keystores!

This post has sections on

This post started off as a few lines, then I though I had better explain the pre-req knowledge, and then the pre-req knowledge to the the pre-req knowlegde. I hope it provides a clear story.

What’s the difference between a trust store and a key store – when they are both called key stores?

You need a store to contain the certificates needed to check any certificates sent to the application. These are usually certificate authority certificates, but can also be self signed certificates. This store is known as a trust store – it is used to check that a certificate is trusted. None of this data is confidential, most of the information is in the public domain. One trust store could be shared by all applications, which makes the maintenance of it much easier than a trust store for each application/server.

You need a store to keep the private key used for encryption/decryption, this is known as the key store. This needs to be kept confidential. The keystore may have just one private key. You can use smart keys, or external keystores such as special USB devices.

Some products use just one store with the trust store and the private certificates combined. They also call this the keystore. If you want to isolate the keystores, you need multiple stores. This, in turn, means that if you update one trust certificate, you have multiple stores to update.

What does a store contain?

A keystore can contain

  • private information – such as you secret key which you use for encryption and decryption
  • public information, such as your Distinguished Name (CN=colin,O=SSS), and they key needed to encrypt data for me
  • trust data, if you send me a public key – can I validate it has not been tampered with?

What format is this data in?

This data is typically in one of three formats

  • Plain text
  • Binary
  • Binary portable (64 bit encoding)

Plain text

If you print a certificate or key, it comes out like

Certificate:
  Data:
    Version: 3 (0x2)
      Serial Number: 379 (0x17b)
      Signature Algorithm: ecdsa-with-SHA256
      Issuer: C=GB, O=SSS, OU=CA, CN=SSCA256
      Validity
        Not Before: Feb  9 09:35:07 2021 GMT
        Not After : Apr  9 17:40:01 2022 GMT
        Subject: C=GB, O=cpwebuser, CN=ecec
        Subject Public Key Info:
          Public Key Algorithm: id-ecPublicKey
            Public-Key: (256 bit)
            pub:
               04:5d:15:78:83:e0:ab:af:78:69:2f:14:51:fe:29:
            ...
                ASN1 OID: prime256v1
                NIST CURVE: P-256
...

Binary

The data is encoded using Abstract Syntax Notation(asn). For example you get a string of fields like

  • The following fields are a sequence
  • of length 24
  • it is a string format
  • the first item has length 8
  • the attribute type is 06 03 55 04 06 which means Country
  • the attribute value is GB
  • the second item is …

This would describe the C=GB in C=GB, O=SSS, OU=CA, CN=SSCA256.

Binary portable (base 64 encoding)

With the binary format, the data is a hex string, which is not very portable. For example if you FTP it to a remote site, you may get data conversion, and new lines changed. Often the data is converted to an intermediate form called base 64 encoding. Every 3 hex characters are converted to 4 “printable” characters A-Z,a-z,0-9 and some special characters.

The file looks like

-----BEGIN CERTIFICATE-----
MIICDjCCAbSgAwIBAgICAXswCgYIKoZIzj0EAwIwOjELMAkGA1UEBhMCR0IxDDAK
BgNVBAoMA1NTUzELMAkGA1UECwwCQ0ExEDAOBgNVBAMMB1NTQ0EyNTYwHhcNMjEw
...
hEzE3hSzUvbfLUKnua64AiEAnyKhrmBjXCVQ834VfjhNHYqjaN1PzXBE8Cv3jlWJ
9bA=
-----END CERTIFICATE-----

What are the keystore types?

When Secure Sockets Layer was being developed each major player came up with their own format for storing the certificate and key information.

  • Java had files with type .jks (java key store) for example keystore.jks
  • IBM had files with type cms (certificate management system) with file types like zzserver.crl, zzserver.kdb, zzserver.rdb, zzserver.sth. It also uses intermediate files such as .arm, which has base 64 encoding of a certifcate, as shown above.
  • Netscape had files in a NSS database, for example files in the directory /home/colinpaice/.pki/nssdb, are files cert9.db and key4.db. key4 is the keystore database (storing keys), and cert9 is the trust store database (storing certificates).
  • OpenSSL developed .p12 files which can contain certificates and keys. It also has .pem (base 64 encoded) and .der (binary) files for individual certificate, private key, and public key files .
  • Windows has .pfx files.
  • Smart cards where the keystore is on a special USB type device or other external Hardware Security Module, each have their own format keystore.

Standards were developed to work with keystores

When can different keystore types be used?

With C programs on mid range and z/OS, IBM products use GSKIT from IBM. On z/OS you can store them in the z/OS security manager (for example RACF) or on the Hardware Security Module in the processor.

Java program can use most types of keystore. You may need to configure the java.security configuration file with an entry like

  • security.provider.X=com.ibm.security.cmskeystore.CMSProvider
  • security.provider.Y=com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl /home/colinpaice/mq/nitrokey.cfg

The first line includes support for cms files, the second includes support for a pkcs11 external keystore on an HSM (including its configuration file).

For Java programs, you configure the keystore using start-up options. The following example defines the keystore as an external USB HSM keystore, and the trust store is a openssl .p12 store.

-Djavax.net.ssl.keyStore=NONE
-Djavax.net.ssl.keyStoreType=pkcs11
-Djavax.net.ssl.keyStorePassword=12345678

-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/mytrust.p12
-Djavax.net.ssl.trustStorePassword=password
-Djavax.net.ssl.trustStoreType=pkcs12

These definitions along with the java.security, and its override, show the keystore is a pkcs11, and there is an entry in the java.security (above) pointing to the configuration file /home/colinpaice/mq/nitrokey.cfg for the pkcs11 definition. You can override or extend the options in the java.security file using configuration using

-Djava.security.properties=/home/colinpaice/mq/colin.java.properties

Do these keystores share information?

No, if you have a keystore (or a USB HSM) it shares no information with any other keystore. For example you could unplug the USB keystore, take it to another machine and plug it in, and the certificate etc will be available. You could send a .jks file, to yet another machine, and it could be used.

What can you use to administer the keystore.

IBM provided

You can use GSKCapiCmd_64 from IBM GSKIT to manage certificates and keys.

MQ has runmqakm as a command line tool which invokes GSKCapiCMD_64 under the covers.

You can also use the MQ command strmqikm to have a GUI to manage your keys and certificates. strmqikm just invokes the GSKIT ikeyman program.

These solutions feel a little dated, as I could not find if they support modern certificates capabilities; OCSP, and elliptic curves. They had a few bugs as well.

I’ve used openssl to create the “modern” private key and certificates, then imported them into the keystores using the IBM tools.

Oracle provided

With Java there is keytool. This supports a variety of keystores (depending on what has been configured in its java.security file), including

  • Java Key Stores .jks. This keystore can only be processed by java
  • Java Cryptography Extension KeyStore – a stronger version of .jks. This keystore can only be processed by java
  • pkcs12 .p12 and .pfx. These keystores can be used by Java, C and other languages, with the right API. For example Curl uses a C api to access the keystores.
  • pkcs11 – keystores on smart devices
  • nss – netscape security.

It does not support cms format keystores used by GSKIT.

Openssl

This is open source and is active, keeping up with the trends in security. Openssl deals with the building blocks; private keys, certificates etc but does not handle keystores very well. You can create manage certificates, but you may need other tools to put them into keystores.

opensc for managing smart keys and other pkcs#11 devices

(Opensc is open Smart Card.) If you have a smart key, or external keystore on an HSM, there may be hardware specific libraries for accessing the keystore, or the open source code may be supported. This can provide drivers for other tools, such as the java tools. You can also use pkcs11-tool to directly administer keys and certificates on an HSM device.

Other tools

  • pk12util from Netscape is used to mange keys in the NSS keystore databases
  • certutil from Netscape is used to manage keys and certificate in the NSS database.
  • Firefox and Chrome browsers can be used to update the NSS keystore used by the browsers.

What tool would I use when?

Creating the private key

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool -keypairgen
  • To generate an elliptic curve, use openssl ecparam
  • To generate an RSA, use openssl genpkey
  • Use keytool keypairgen to generate an elliptic or RSA key
  • runmqakm and strmqikm(ikeyman) do not seem to support elliptic curves, but support RSA keys as part of a certificate request (see below)

Create a certificate request

The certificate request takes (or creates) the public key, creates a certificate with the DN ( eg CN=Colin,C=GB,O=SSS and creates the certificate request file.

  • openssl req takes a file from the keypairgen
  • keytool can create a certificate request, I could not see how to use the keypair gen private key as part of this
  • runmqakm and strmqikm (ikeyman) can create a request, but I do not think it supports all flavours of private key.

Sign a request

I expect most people will want a tool which you can run as a script

  • openssl ca – you can include the optional attributes, with this.
  • keytool – I could not see how to use this to sign a certificate. The documentation suggest using openssl x509.
  • runmqakm – you can use runmqakm -cert -sign .
  • strmqikm (ikeyman) I could not see how to sign a request using this, and it is a GUI.

Receive the signed certificate into the keystore

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool –write-object
  • Use openssl pkcs12 to create a pkcs12 (.p12) keystore using the private key and signed certificate
  • Use keytool -importcert. This can support most keystore types, depending on the configuration in the java.security file.
  • runmqakm and strmqikm (ikeyman) import ( receive) the certificate and store it in the keystore.

Update the browser’s keystore

The browsers have an nss format store.

  • pk12util -i …imports a .p12 keystores into the nss “sql:” keystore. The sql: keystore is a Netscape internal format store. (you use certutil … to remove entries from the nss keystore).
  • You can use the browser’s facilities to import certificates and keys into the browsers (nss) keystore.

Receiving a public certificate into a keystore.

If you have a Certificate Authority or a self signed certificate you want to put into your trust store.

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool –write-object
  • Openssl is not very good at adding new entries to an existing .p12 file
  • runmqakm -cert -add -file ca256.pem…
  • strmqikm, select Signer Certificates, then select Add
  • Use keytool -importkeystore.  To import a .p12 store, or a .pem file I have a shell script

ks=” -destkeystore mytrust.p12″
dest=”-deststoretype pkcs12 -deststorepass password”
src=”-srcstoretype PKCS12 -srcstorepass password”
keytool -importkeystore $ks $dest -srckeystore ca1024.p12 $src
or
keytool -import $ks $dest -file carsa1024.pem 

Which keystore should I used for which application?

All applications can use smart cards and external keystores through the standard key stores.

  • A queue manager on midrange uses cms format files, so you need to use the runmqakm or strmqikm similar files
  • A queue manager on z/OS can use the z/OS security manager (eg RACF), or the cms keystore files.
  • Firefox and Chrome browsers use the NSS format keystores.
  • Java clients can use a variety of keystores, .jks, .p12, smart cards
  • A C, .Net etc client use cms format keystores
  • Curl can use .pem files (from openssl) and .p12 files
  • A web server has trust stores, and keystores. You configure the supported formats in the java.security file. It can use jks, .p12 and smart cards.

Summary

It all looks a bit of a mess, and you need to know a lot to get your job done.

Debugging external smart cards and external pkcs11 keystores.

There is an open source package (opensc) which provides access to smart cards and external keystores. It provides some good tools for diagnosing problems.

There is a wiki with good information.

Opensc return codes are here, and the printable text is here

Monitor traffic to and from the device.

You can monitor the traffic to and from the device by using an intermediate “spy” module which displays the traffic.

In your configuration (for example a CCDT), where you specified the name of the module /usr/lib64/pkcs11/opensc-pkcs11.so, replace this with /usr/lib64/pkcs11/pkcs11-spy.so. Specify the environment variable

export PKCS11SPY=/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so

The spy module is invoked, prints out the parameters, and then invokes the module specified in the environment variable.

The output is like

19: C_Login
2021-03-10 14:22:47.947
[in] hSession = 0x21fc030
[in] userType = CKU_USER
[in] pPin[ulPinLen] 00000000021fb2a0 / 8
00000000 5B C7 E7 BB E5 FC 6A BE […..j.
Returned: 160 CKR_PIN_INCORRECT

Detailed internal trace

You can specify the environment variable OPENSC_DEBUG to give a very detailed trace. The higher the number the more detailed the trac.

export OPENSC_DEBUG=9

and use unset OPENSC_DEBUG to reset it.

You can use OPENSC_CONF to specify a configuration file with more parameters, such as file name for the output.

The output from this trace (showing a logon with pin number 12345678) is like

0x7f96e2dca740 14:13:16.756 [opensc-pkcs11] framework-pkcs15.c:1494:pkcs15_login: pkcs15-login: userType 0x1, PIN length 8
0x7f96e2dca740 14:13:16.756 [opensc-pkcs11] pkcs15-pin.c:301:sc_pkcs15_verify_pin: called
….
0x7f96e2dca740 14:13:16.757 [opensc-pkcs11] reader-pcsc.c:283:pcsc_transmit: reader ‘Nitrokey Nitrokey HSM (DENK01051600000 ) 00 00’
0x7f96e2dca740 14:13:16.757 [opensc-pkcs11] reader-pcsc.c:284:pcsc_transmit:
Outgoing APDU (13 bytes):
00 20 00 81 08 31 32 33 34 35 36 37 38 . …12345678

GSKIT return codes

If you are using the MQ C Client interface, this uses GSKIT. There is documentation for the z/OS version, and the return codes are here.