Category: TLS

Using z/OS LDAP with TLS

This started as a mini project before lunch, and now, two weeks later, I have got all of the bits to work.

As I was struggling with this, I wondered why no one had crossed this swamp before me. Is it because no one wants to cross it, or it is too difficult to do so? I think it was just difficult to do so – but I can now provide a map!

It took me a couple of hours to get LDAP on z/OS to use TLS from my Linux machine. The challenge was finding the right options. I then tried using Elliptical Certificates, and got this working, then was partially successful in using TLS 1.3.

As part of this I wrote Authentication using certificates with LDAP, which is where it was really difficult.

For clients, I used ldapsearch (from openldap ) on Ubuntu, and ldapsearch on z/OS.

See the product documentation here.

I’ve put a section “What cipher specs should I use” in a separate blog post.

Configure the LDAP environment file

I had to add TLS V1_2 and TLSV1_3 (and disable the deprecated TLS 1.1)

GSK_PROTOCOL_TLSV1_1=off
GSK_PROTOCOL_TLSV1_2=on
#GSK_PROTOCOL_TLSV1_3=on

to the environment file, because they default to off. Without these options it defaults to SSL – which is so old, my certificates and keys did not support it. I disabled TLS_1_1 because it is old.

I added

GSK_TLS_SIG_ALG_PAIRS=0601060305010401030108060805080405030403

This is used when validating certificates. The signature algorithm of the CA must be in this list see GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) (see here), get things working, then remove the ones I did not want to use; rather than try to add the ones I wanted to use.

I initially added

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0251303130113020039

This is a list of the cipher specs it will accept. You should try to restrict the certificates in your organisation, to Elliptical Certificates, and a limited configuration for any CA’s. For example have

  • all client’s certificates defined as Elliptic Curve type prime256v1, signed with ecdsa-with-SHA256.
  • All internal CAs defined as Elliptic Curve type prime256v1 signed with ecdsa-with-SHA256.

When using certificate authentication I needed the following for it all to work

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023

The environment variables

GSK_SERVER_TLS_KEY_SHARES
GSK_CLIENT_TLS_KEY_SHARES

these are used only when using TLS 1.3. I’ll blog about using TLS 1.3 at a later date.

While you are investigating problems you can use the environment variable GSK_TRACE=0xff to write a trace. You can specify GSK_TRACE_FILE to direct the trace to a particular file. The default location is /tmp/gskssl.%.trc where % is the thread id. You format it with

gsktrace /tmp/gskssl.65567.trc > out

oedit out

I used

GSK_TRACE=0xff
GSK_TRACE_FILE=/tmp/gsktrace.ldap

So I could have a shell script to format and display the trace.

gsktrace gsktrace.ldap > out
oedit out

Change the LDAP configuration file.

I used

sslCipherSpecs all
sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING

I specified a port for ldaps. You can use

ldapsearch -H ldaps://10.1.1.2:389

This is preferred to specifying host and port

ldapsearch -h 10.1.1.2 -p 389

A port can process TLS and non-TLS requests, so definition for port 1389 may not needed.

I specified a keyring sslKeyRingFile START1/MQRING. This is owned by a different userid to the started task, so needs more RACF definitions. See below for the definitions.

Define the keyring and keyring

I defined an RSA certificate and an EC certificate. I controlled which was used from the configuration file.

* create the ring
RACDCERT DELRING(MQRING) ID(START1)
RACDCERT ADDRING(MQRING) ID(START1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the server's RSA certificate

RACDCERT ID(START1) DELETE(LABEL('ZZZZ'))
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('ZZZZ') - 
             O('SERVER') - 
             OU('SSS')) - 
   ALTNAME(IP(10.1.1.2))- 
   SIZE(4096)- 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   RSA - 
   WITHLABEL('ZZZZ') - 
   KEYUSAGE(         DATAENCRYPT,DOCSIGN,HANDSHAKE         ) 

RACDCERT id(START1) ALTER(LABEL('ZZZZ'))TRUST 
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('ZZZZ') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the Elliptic Certificate

RACDCERT ID(START1) DELETE(LABEL('SERVEREC')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('SERVEREC') - 
             O('ADCD') - 
             OU('TEST')) - 
   ALTNAME(IP(10.1.1.2)) - 
   SIZE(521) - 
   NISTECC - 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   KEYUSAGE(HANDSHAKE         ,KEYAGREE) - 
   NOTAFTER(   DATE(2024-12-29))- 
   WITHLABEL('SERVEREC') 
RACDCERT id(START1)  ALTER(LABEL('SERVEREC'))TRUST
                                                        
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('SERVEREC') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 
                                                             
RACDCERT LISTRING(MQRING) ID(START1)

Permit the LDAP server’s userid to access the private certificate in the keyring

Because the LDAP userid does not own the keyring, the server userid needs update access to be able to extract the private key.

RDEFINE RDATALIB START1.MQRING.LST UACC(NONE) 
PERMIT START1.MQRING.LST CLASS(RDATALIB) ID(GLDSRV) -
    ACCESS(CONTROL)
SETROPTS RACLIST(RDATALIB) REFRESH

For a userid to be able to use its own keyring, it only need read access to be able to use the private certificate.

Set up the client userid

I set up the userid in the z/OS LDAP server, so I could use the z/OS userid and password. See here.

Test it out

LDAP checks the TLS parameters, when a TLS request arrives. It does not check at startup . I used openssl s_client to connect, and display information about the TLS handshake. See Debugging TLS – an easier way.

Setting up the client on Ubuntu

I set up a file ldaprc containing the ldap configuration. You can enter all of the parameters on the command line, but the ldaprc file makes it easier. My file had

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_PROTOCOL_MIN 3.2
TLS_REQCERT demand
#TLS_CERT /home/colinpaice/ssl/ssl2/rsaca256.pem
#TLS_KEY /home/colinpaice/ssl/ssl2/rsaca256.key.pem

The #TLSCERT etc are ignored. No certificate is sent to the server (unless the -Y external option is set).

Enable GSK trace on z/OS

I enabled an LDAP trace to //SYSPRINT on z/OS, using f GLDSRV,debug error+CONNS+LDAPBE

Where

  • error reports errors
  • Conns reports connection information, such as the cipher spec used.
  • LDAPBE reports on the communication to the back end, for example it reports the SAFUSER

Run a query on Linux

I used ldapsearch on Ubuntu

ldapsearch -H ldaps://10.1.1.2:1389 -D “cn=ibmuser, o=Your Company” -w panthe0n -b “o=Your Company” “(objectclass=*)” aclEntry

The trace on z/OS GLDSRV included

CONNS srv_ssl_connect()644: SSL Cipher Specs
00000000: c3f0f0f8 *C008 *

and we can see that SSL was used, and cipher spec C008 was used.

Because I had set up my userid to map use a SAF userid, in the trace I had

LDAPBE srv_process_bind_request()939: do_return_bind msgID=1, connID=8, bindDN=‘CN=IBMUSER, O=YOUR COMPANY’, safUserID=’IBMUSER’, dnList=0x0, grpList=0x0, rc=0

Debugging TLS – an easier way

I had been trying to get an IBM product on Linux to talk to LDAP using TLS and certificate authentication; where I give a certificate instead of a LDAP userid and password. Basically I got “computer say no”.

IBM products use GSKIT to manage keystores and TLS between sessions. With Java you can get out a trace of the TLS conversation. You can get out a trace on z/OS, but not on other platforms. After a day I stumbled on a different approach.

Gettting a gskit trace on z/OS.

Getting the trace was easy, understanding it was harder.

In the LDAP environment file I added GSK_TRACE=0xff. Once I had TLS to LDAP working, I used GSK_TRACE=0x04 to trace just errors. Some “errors” are recorded in the GSKTRACE as “INFO”. For example an Elliptic curve is only supported in TLS 1.3.

By default the trace goes to /tmp/gskssl.%.trc. where % is the thread id.

To format it go into USS and use the command gsktrace gskssl.83951642.trc > out, and edit the file “out”.

The linux end.

I found a great program openssl s_client. This starts a TLS handshake and prints out in easy to understand format, what is going on. For example

The command

openssl s_client -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -x509_strict

gave the flow of the data.

stderr output

The stderr output included

depth=1 O = COLIN, OU = TEST, CN = COLIN4Certification Authority
verify error:num=19:self signed certificate in certificate chain

This is the name of the CA.

Following this was an error message. Note: it is at the top of the output – not the bottom as I expected.

SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1528:SSL alert number 80

See below for a discussion about the error.

stdout output

Certificate chain
  0 s:O = SERVER, OU = SSS, CN = ZZZZ
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority
  1 s:O = COLIN, OU = TEST, CN = COLIN4Certification Authority 
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority

This says

  • the certificate SERVER was signed by COLIN4.Certificate Authority
  • the certificate COLIN4Certificate Authority was self signed

The server certificate comes next

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Followed by the identity of the server

subject=O = SERVER, OU = SSS, CN = ZZZZ
issuer=O = COLIN, OU = TEST, CN = COLIN4Certification Authority

Next comes the list of Certificate Authority or self signed certificate in the keyring 

Acceptable client certificate CA names
O = ADCD, OU = TEST, CN = MQWEB2 
O = COLIN, OU = TEST, CN = COLIN4Certification Authority
C = GB, O = SSS, OU = CA, CN = SSCARSA1024
C = GB, O = SSS, OU = CA, CN = SSCA256

Any client certificate must be signed by one of those CAs (or self signed certificates).

Then comes the client certificate types the server will accept

  • Client Certificate Types: RSA sign, DSA sign

Note: This does not include ECDSA, so elliptic certificates are not supported in this configuration.

After this is the signature algorithms the server will accept

  • Requested Signature Algorithms: RSA+SHA512: ECDSA+SHA512: RSA+SHA384: ECDSA+SHA384: RSA+SHA256: ECDSA+SHA256: DSA+SHA256: RSA+SHA224: ECDSA+SHA224: DSA+SHA224: RSA+SHA1: ECDSA+SHA1: DSA+SHA1

It gives information on the session so far

SSL handshake has read 2493 bytes and written 1537 bytes
Verification error: self signed certificate in certificate chain

and information about the key sent down. It was defined with SIZE(4096) RSA.

New, SSLv3, Cipher is AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated

and about the TLS session

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA
    Session-ID: 0401001A0A0100029...
    Session-ID-ctx: 
    Master-Key: 32D5B4AD162F0403E323DB0...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1634115022
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no

The -msg option gave me 

>>> ??? [length 0005]
    16 03 01 01 31
>>> TLS 1.3, Handshake [length 0131], ClientHello
    ...
<<< ??? [length 0005]
    ... 
<<< TLS 1.3, Handshake [length 0051], ServerHello
    ...
<<< TLS 1.2, Handshake [length 0839], Certificate
    ...
<<< TLS 1.2, Handshake [length 0123], CertificateRequest
    ...
<<< TLS 1.2, Handshake [length 0004], ServerHelloDone
    ... 
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 021c], Certificate
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0206], ClientKeyExchange
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0050], CertificateVerify
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, ChangeCipherSpec [length 0001]
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0010], Finished
    ...
<<< ??? [length 0005]
    ...
<<< TLS 1.2, Alert [length 0002], fatal internal_error
    02 50

The fatal errors are described here. 0x02 is fatal error, 0x50= 80 = internal error. internal_error(80). Which is not very helpful.

In the LDAP log I got

GLD1116E Unable to initialize an SSL connection with 10.1.0.2: 434 – Certificate key is not compatible with cipher suite.

In the gsktrace on z/OS I got

cms_validate_certificate_mode_int(): Validating CN=ecec,O=cpwebuser,C=GB ENTRY
INFO get_issuer_certificate(): Using issuer CN=SSCA256,OU=CA,O=SSS,C=GB

ERROR read_v3_certificate(): Client certificate key type 13 not allowed for SSL V3 cipher 0x0035

  • Client certificate key type 13. Looking in /usr/include/gskcms.h under X.509 data types, was x509_alg_ecPublicKey.
  • SSL V3 cipher 0x0035 is listed here, as TLS_RSA_WITH_AES_256_CBC_SHA = 256-bit AES encryption with SHA-1 message authentication and RSA key exchange.
  • x509_alg_ecPublicKey is elliptic public key – which is incompatible with RSA key exchange.

When I used an RSA certificate

openssl s_client -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/rsaca256.pem -key /home/colinpaice/ssl/ssl2/rsaca256.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -verify_return_error -policy_print -x509_strict

it worked

Whoops I left my private key unprotected.

I had set up my certificates and private key so I could use them to work with a web server on z/OS. I set them up on Linux and copied them to z/OS. My tests all worked. It wasn’t till this morning when I thought, those test should have failed.

With Python and the requests package you can send REST requests to a backend server, using digital certificates for authentication. You configure which private key to use, but you are unable to specify the password for the private key. As this worked, there must be no password on the key file – whoops.

This is a bit like leaving your front door unlocked with a note “If we are not here, please come in and make yourself at home”.

To generate a private key I had used

openssl ecparam -name prime256v1 -genkey -noout -out $name.key.pem

but there is no way of specifying a password. With hindsight it was obvious there was no password protection.

I had to use

openssl ecparam -name prime256v1 -genkey -noout | openssl ec -out $name.key.pem -passout file:password.file

Where the first part of the command generated a private key, and wrote it to the output stream. The second part does nothing with the data, but wrote it to the output file using the password in password.file.

When I used this private key, the python requests failed because it needed a password!

I should have remembered

I remember helping with some security testing many years ago.

  • Userid1 was able to use resource1, good, that worked
  • Userid2 was able to use resource2, good, that worked
  • Userid1 was able to use resource2, good ,that worked – hang on… that should have failed.

The person customising security had missed one configuration step, so all users had access to all resources, and the test plan did not have any negative testing.

Using jConsole and with z/OS Liberty web server

I wanted to get some monitoring information out from z/OSMF using jConsole on my Ubuntu machine. Eventually this worked, but I had a few problems on the way. The same technique can be used for base Liberty, MQWeb, z/OSMF and ZOWE all of which are based on Liberty.

Configuring z/OSMF

I changed the z/OSMF configuration to include

<featureManager> 
  <feature>restConnector-2.0</feature> 
</featureManager>

and restarted the server.

In the stdout (or message log) will be something like

CWWKX0103I: The JMX REST connector is running and is available at the following service
URL: service:jmx:rest://sss.com:10443/IBMJMXConnectorREST

You need the URL. The message above gave service:jmx:rest://sss.com:10443/IBMJMXConnectorREST, but I needed to use service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST .

The port number came from the httpEndpoint with id=”defaultHttpEndpoint” . I have another httpEndpoint with port 24993, and this also worked with jConsole.

Set up jConsole

I set up a script for jConsole

k1='-J-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/adcdc.p12'
k2='-J-Djavax.net.ssl.keyStorePassword=password'
k3='-J-Djavax.net.ssl.keyStoreType=pkcs12'
t1='-J-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/zca.jks'
t2='-J-Djavax.net.ssl.trustStorePassword=password'
t3='-J-Djavax.net.ssl.trustStoreType=jks'
d='-J-Djavax.net.debug=ssl:handshake'
d=' '
de='-debug'
de=' '
s='service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST'
jconsole $de $s $k1 $k2 $k3 $t1 $t2 $t3 $d

Where

  • the -J .. parameters are passed through to java,
  • the -Djava… are the standard set of parameters to define the key stores on the Linux

Running this script gave a pop up window with

Secure connection failed. Retry insecurely?

The connection to service:jmxLret://10.1.1.2:10443/IBMJMXConnectorREST could not be made using SSL.

Would you like to try without SSL?

This was because of the exception

java.io.IOException: jmx.remote.credentials not provided.

I could not see how to pass userid and password to jConsole.

I then used Cntrl+N to create a new connection and entered Username: and Password: which jConsole requires. After a short delay of a few seconds jConsole responded with a graphs of Heap Memory Usage, and Threads in use. You can then select from the Measurement Beans.

The TLS setup

In the keystore I had a certificate which I had used to talk to a Liberty instance before.

This was signed, and the CA certificate had been imported into the key trust keyring on z/OS, for that HttpEndPoint.

The server responded with a server certificate (“CN=SERVER,O=SSS,C=GB”) which had been signed on z/OS. The signing certificate had been exported from z/OS and downloaded to Linux

I created a jks key trust store using this certificate, using the command

keytool -importcert -file temp4ca.pem -keystore zca.jks -storetype jks -storepass password

and used this trust store to validate the server certificate sent down from z/OS.

This worked with jConsole.

I created a pkcs12 keystore using keytool

keytool -importcert -file temp4ca.pem -keystore zca2.p12 -storetype pkcs12 -storepass password

Which also worked.

Problems using a .p12 trust store

I used

runmqakm -keydb -create -db zca.p12 -type pkcs12 -pw password
runmqakm -cert -add -file temp4ca.pem -db zca.p12 -type pkcs12 -pw password -label tempca
runmqakm -cert -details -db zca.p12 -type pkcs12 -pw password -label tempca

to create a pkcs12 keystore and import the z/OS CA certificate. The -details option displayed it.

When I tried to use it, jConsole produced the message (after the Cntl+N)

Secure connection failed. Retry insecurely?

The connection to service:jmxLret://10.1.1.2:10443/IBMJMXConnectorREST could not be made using SSL.

Would you like to try without SSL?

I used Ctrl-N as before, and got the same message.

Using

d=’-J-Djavax.net.debug=ssl:handshake’

and rerunning the script, produced a TLS trace. At the bottom was

VMPanel.connect, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

%% Invalidated:…
VMPanel.connect, SEND TLSv1.2 ALERT: fatal, description = internal_error

Using a trace at the server, gave the unhelpful , SEND TLSv1.2 ALERT:

Using openssl also failed. Create the .p12 keystore

openssl pkcs12 -export -out zca.p12 -in temp4ca.pem -name zCA -nokeys

and rerun the jconsole script, and it failed the same way.

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

It looks like runmqakm and openssl do not create a valid trust store with an imported certificate.

Additional diagnostics

When the trust store created by keytool was used; at the top of the TLS trace output was

System property jdk.tls.client.cipherSuites is set to ‘null’
…
Ignoring disabled cipher suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
trustStore is: /home/colinpaice/ssl/ssl2/zca.p12
trustStore type is : pkcs12
trustStore provider is :
init truststore
adding as trusted cert:
Subject: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Issuer: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Algorithm: RSA; Serial number: 0x0
Valid from Tue Jul 14 00:00:00 BST 2020 until Fri Jul 02 23:59:59 BST 2021

keyStore is : /home/colinpaice/ssl/ssl2/adcdc.p12
keyStore type is : pkcs12
keyStore provider is :
init keystore
init keymanager of type SunX509

When the runmqakm or openssl was used, the green entries were missing.

When I used runmqakm to create the pkcs12 keystore

runmqakm -cert -details -db zca.p12 -type p12 -pw password -label tempca

listed the certificate successfully.

When I used keytool to list the contents

keytool -list -keystore zca.p12 -storetype pkcs12 -storepass password
Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 0 entries

When I created the key store with keytool, both runmqakm and keytool displayed the certificate.

The problem looks like Java is only able to process the imported CA certificates when keytool was used to create the trust store.

When the burglar alarm goes off – would you just mute it?

I’m sure you’ve all been told to ignore links like

“You have wun $1,000,000 click hear to download a dodgy application to steal your money” or

“We have been unable to deliver a parcel to you. Click here to rearrange delivery and pay the excess money” but how about
“Your connection is not private, Net::err_cert_invalid – click here to accept it“

Collect 10 points if you did not fall for any of these.

I would like to talk about the last one, about your connection is not private; because I saw a web site recommending “click here to accept”.

You get messages like this if there are problems with the certificate. Common problems are

  • Your browser does not recognise the certificate, because it has not been signed by any CA in your browser’s trust store. This looks like someone is sending you an invalid certificate. It may be your enterprise security team need to update all browsers with the missing certificate, but they should have spotted this problem not you.
  • The certificate sent from the server has “valid host addresses are…. 10.1.1.2, my sys.com” and this has been sent from a different site. This could be because bad guys have intercepted your request and played “Man in the middle” between you and the server.
  • Less common, is where the certificate has parameters which your browser cannot handle. For example very old cipher specs, and your browser only supports TLS 1.2 or above.
  • It could be caused by a server using a self signed certificate. It may be acceptable to use this in a test environment, but it is not secure – as anyone could have created it. If it is signed, then you can check the digital signature.

All of these should set off the intruder alarm.

If you get a message like this – do not click on the accept button. This is just like muting the burglar alarm, and ignoring the warning – bad guys could steal your personal details – and you would not know.

Many web servers are configured with a bad certificate, for example they may specify an alternate name of 127.0.0.1, but not “localhost”. It is much easier (though wrong) to tell people to accept the certificate, than to fix the certificate.

Do not trust it

The blog post went on to say that you can download the certificate which has been sent to you and install it in your key store. I felt like screaming “NO!” and attacking the screen with a hammer!

Imaging, twenty years ago, you were a bank manager, and a man comes in, and says “I am a policeman, and we are doing a survey about your bank’s safe and where you keep the money. I would like my colleague “fingers” to go down inspect it. He has to work anonymously, which is why he is wearing a mask, and gloves, and he is wearing a black and white striped shirt. I’ll come back tomorrow”. You let “fingers” go down to the safe on his own, and say goodbye to the nice policeman. An hour later you find your safe is empty and there is no sign of the two men.

Your first mistake was to accept the policeman was a policeman. You should have checked his credentials before doing anything else. If he was fake, then any identification papers would be fake. You need to talk to your contacts in the local police station, and check if your visitor is valid or bogus (while he is sitting in your office).

Only in rare situations, such as a test environment, would you extract the certificate from what was sent to you, and install it in your key store. If you install it, then later, another request can come in from the bad guys, and be validated by the CA you just installed.

If you need a CA, you go through the proper corporate process. For example as part of the overnight maintenance on your machine, or download it securely from a known site. Emailing may not be good enough if your machine has been hacked.

So, in summary, today’s lesson is

  1. Do not “accept” bad certificates when using web site. Fix the the server.
  2. Get any additional certificates through an approved process – do not accept what the server is giving you.

In the words of the Police Constable in Dixon of Dock Green. “Evening all – mind how you go”.

Overview and challenges of using end to end message encryption (AMS)

MQ provides end to end protection of messages. This covers

  • Integrity – where the message can still be read – but any changes to the message are detected. You can check the id of the person that sent it.
  • Encryption – where the payload is encrypted and only the ids with the correct private key can decrypt it.
  • Privacy – A combination of Integrity and Encryption, where only specified ids can decrypt it, and you can check the id of the person that signed it.

The messages are protected on the network, in memory and on disk. You do not need to use a channel with TLS for protection (but you may want to use TLS for authentication), as the data is protected before sending.

Any headers that are involved in ‘routing’ like MQMD, XQH, RFH2, etc are skipped by AMS because the queue manager still needs to know what to do with the messages in terms of delivery and selection before they are unprotected. Other message headers like PCF command headers and custom application defined headers/formats can still be protected. Sensitive data should not be held in message properties as that data is used in routing/selection. Thanks to Jon Rumsey for the correction to this.s

Background about encryption and signing of data

You may need to read some of the following section more than once, because it reference private keys and public keys and it is hard to work out what is happening.

The encryption techniques used are special. There is a private key and a public key. (Think of your passport as a private key, and your name and address is your public key). You can use functions

  • You do encrypt(private key, payload) to produce an encrypted payload, send it to me, and then I use decrypt(public key, encrypted payload) to recreate the original payload. I know it came from you, because I needed your public key to decrypt it.
  • I do encrypt(public key, payload) to produce an “encrypted” payload, and send it to you. Only you can use decrypt(private key, encrypted payload) with your private key to recreate the original payload.

With this

  • I can encrypt data using your public key that only someone with access to the private key (you) can decrypt
  • I encrypt with my private key, and anyone with my public key can decrypt it – and you know it came from me – because you needed my public key.

Signing is simply taking a checksum of the data, and encrypting the value with the signer’s key. When you come to check the signature, you do the same check sum calculation, decrypt the “signed” value using my public key, and the values should match. The payload would look something like “Signed by CN=Colin,O=SSS,C=GB” encrypted checksum=2294567, my certificate; followed by the original data.

We can take this further. I sign the data with my private key, and then encrypt the data(including the signed data) with your public key, (keep up at the back there – I said it was hard to follow). When you process it, only you can decrypt it because it needs your private key, and you can check the signed data, and see it came from me. I have sent the message encrypted, and have mutual authentication.

With encryption a symmetric key is used to encrypt the data, because it is faster, and about as secure as encryption with an asymmetric key. When trying to break into enciphered data, the more data you have, the easier it is to break. This is why you can reuse the symmetric key for multiple records, but you should change it, for example after 10 MB has been processed (I don’t know the best figure – it is all a matter of risk).

Sending a message, many to many.

Sending a message from one person to another person looks fairly easy. It gets a bit harder when you have potentially many people in my organisation who are allowed to send a protected message to your organisation, and there may be many people who are allowed to get, and decrypt it.

Let the people in my organisation be CN=A,O=MYORG, and CN=B, O=MYORG, and the people in your organisation be CN=X,O=YOURORG, CN=Z,O=YOURORG.

If person CN=A wants to send an encrypted message so it can be decrypted by CN=X or CN=Z the processing is as follows.

  • Do a checksum calculation on the record.
  • Encrypt this checksum using CN=A’s private certificate.
  • Put a new field of CN=A,O=MYORG|| encrypted checksum value, CN=A’s public certificate before the record.
  • Create a symmetric key.
  • Encrypt the (signed data and original payload) with the symmetric key.
  • Take the symmetric key and encrypt it for CN=X,O=YOURORG using CN=X’s public certificate. Create a field of the Distinguished Name || encrypted key.
  • Take the symmetric key and encrypt it for CN=Z,O=YOURORG using CN=Z’s public certificate. Create a field of the Distinguished Name || encrypted key.
  • Build a record of the two “encryption fields” and the encrypted data
  • Put the data in a message and send it to the recipient(s).

Getting an encrypted message

The person with CN=X gets the message.

  • Scan down the message looking for CN=X,O=YOURORG. If found then extract the encrypted symmetric key. Decrypt this value with the CN=X’s private key. This gives you the key that was used to encrypt the customer data.
  • Decrypt the message using the key you just decrypted.
  • Extract the first field and see it is signed data. We can see that the checksum was done by CN=A,O=MYORG.
  • Check this name with your list of IDs that can send you messages.
  • Validate the signer’s certificate. Either use a CA, or if the certificate was self signed, check with the copy in your trust store. Note this certificate includes the public key.
  • Take the checksum value in the message and decrypt it using CN=A,O=MYORG’s public key from the payload. Do the checksum of the remainder of the message and compare the values – they should match.

Setting up your enterprise

You need to know the list of potential recipients of messages for each queue.

At the receiving end, if you want to check the message came from an authorised id, you need to know the list of potential senders.

If you’ve managed to keep up so far – it gets worse!

The management of these lists can be difficult. If someone joins my team, I would need to tell the other organisations that they need to add CN=NEWPERSON, O=MYORG into their list of potential senders, (and remove CN=IVE_RETIRED, O=MYORG. It will take a week to get this change through their change management system. Until this is done, the message could be rejected as coming from an unauthorised person.

In the same way, you need to update the AMS setmqspl definitions to add CN=NEWPERSON, O=MYORG to each queue, and when people join or leave the other enterprises you’ll need to update your list of potential senders.

Do not worry too much – there is a solution, see below.

Creating the policies

With MQ, the security policy is defined using the SETMQ Security PoLicy command (setmqspl) for example to define a queue at MYORG to send message to YOURORG. The -a option is for who the message is for

setmqspl -m PRODA -p SENDQUEUE -s SHA256 -e AES128
-a "CN=X, O=YOOURORG"
-a "CN=Y, O=YOOURORG"
-a "CN=Z, O=YOOURORG"

At YOURORG for the GETQUEUE you need the following. The -r option is who you can receive a message from

setmqspl -m YOQM -p GETQUEUE 
-r "CN=A, O=MYORG"
-r "CN=B, O=MYORG"
-r "CN=JO, C=GB,O=SOMEOTHERORG"

To change an entry you have to run whole command, you cannot add one “-a” or “-r”. I would keep a file for these, update file, and process the whole file. For two queues – use two files.

You also have to consider how to update the definitions for multiple queue managers. On z/OS I would have the MQ commands inline, and use a variable queue manager name &QM, which you set in the JCL. For midrange you could have a shell script and pass the queue manager name as a parameter.

What keys do I need?

The encryption is done using RSA, and so needs a certificate with an RSA private key.

When the message is signed

Each id putting a message needs access to a private key – this could be one key shared by multiple users.

When getting a message that has been signed, the recipients will need to verify the certificate sent within the signing data; either a CA in the signing chain, or the public certificate (for self signed certificates).

When encrypting the message

The id putting the message needs access to the public key of each recipients in the list.

The id getting a message which has been encrypted, needs access to the private key.

Maintaining the public keys

For each person in the -r or -a list you will need the digital certificate, and public key in a keystore or a keyring (on z/OS).

Each id on z/OS will need their own keyring. There is a keyring for the queue manager;s AMS address space with the public certificates on it.

On midrange MQ, each user has a keystore, with a combination of public keys, and private key(s).

Time to step back and think about what you want.

To me, it is clear that having every possible user having their own DN, and certificate quickly gets impractical. If you are working with people outside of your organisation do you want them to know the people’s names within your organisation?

You may want to have a departmental certificate for example CN=PAYROLL,O=MYORG, which members of your payroll team all use. You can control access to the queue using standard MQ security policies.

An intermediate configuration would be to have a proxy or server applications to change input messages from “partner protection” to “myorg protection”, and the reply messages changed from “myorg protection” to “partner protection”

  • You have an AMS protected input queue called EXTERNAL_IN. This has been configured to specify the DNs of the systems that can send you messages, so might specify CN=RECEIPTSPAYABLE,O=YOURORG as a valid sender.
  • If the get is successful, then put to a queue called INTERNAL_IN, specifying the DNs of the individuals in your Accounts department who are allowed to get one of the protected messages. If someone joins the Accounts department, you only need to change the configuration within your queue manager.
  • Someone (or an automated task) gets the message from INTERNAL_IN queue, it is decrypted automatically, the application does the processing and puts a message the AMS protected queue YOUR_BANK_INTERNAL, where it is encrypted by AMS.
  • Another proxy/server application in your enterprise gets the messages from YOUR_BANK_INTERNAL, checks the DN of the sender, and that they were authorised to put messages to the queue.
  • The server then puts to YOUR_BANK_EXTERNAL, using a DN of “CN=PAYROLL,O=MYORG”, and it is encrypted for the recipient.

This way you can isolate internal processing and external processing. Updating the internal list is within your control. You should not need to update the external lists (and certificates) once established.

Handling whoops’s

Some problems will result in the messages removed from the queue and put to a special error queue. You need to have a process in place to handle this. I had messages put to this queue when a userid tried to get an encrypted message, but the message had not been encrypted for this userid’s DN.

Other AMS blog posts

pkcs11? pkcs12? .cms? .jks? .p12? .nss? which type of keystore should I use?

There are at least four types of keystore, and after an email exchange with someone, it became clear that the different types of keystore and how to use them, are not widely understood. It is one of those topics, that once you are an expert, this topic is obvious still difficult to understand. When you have little knowledge and just want to get a job done, it can be very confusing.

I’ve learned a lot from writing this post

  • There is no one tool to manage all your keys and certificate
  • You may need one keystore for your queue manager, another keystore for your C clients, another keystore for your Java clients, a keystore for a web browser and something else for curl!
  • You might need to use 3 tools to manage your keystores!

This post has sections on

This post started off as a few lines, then I though I had better explain the pre-req knowledge, and then the pre-req knowledge to the the pre-req knowlegde. I hope it provides a clear story.

What’s the difference between a trust store and a key store – when they are both called key stores?

You need a store to contain the certificates needed to check any certificates sent to the application. These are usually certificate authority certificates, but can also be self signed certificates. This store is known as a trust store – it is used to check that a certificate is trusted. None of this data is confidential, most of the information is in the public domain. One trust store could be shared by all applications, which makes the maintenance of it much easier than a trust store for each application/server.

You need a store to keep the private key used for encryption/decryption, this is known as the key store. This needs to be kept confidential. The keystore may have just one private key. You can use smart keys, or external keystores such as special USB devices.

Some products use just one store with the trust store and the private certificates combined. They also call this the keystore. If you want to isolate the keystores, you need multiple stores. This, in turn, means that if you update one trust certificate, you have multiple stores to update.

What does a store contain?

A keystore can contain

  • private information – such as you secret key which you use for encryption and decryption
  • public information, such as your Distinguished Name (CN=colin,O=SSS), and they key needed to encrypt data for me
  • trust data, if you send me a public key – can I validate it has not been tampered with?

What format is this data in?

This data is typically in one of three formats

  • Plain text
  • Binary
  • Binary portable (64 bit encoding)

Plain text

If you print a certificate or key, it comes out like

Certificate:
  Data:
    Version: 3 (0x2)
      Serial Number: 379 (0x17b)
      Signature Algorithm: ecdsa-with-SHA256
      Issuer: C=GB, O=SSS, OU=CA, CN=SSCA256
      Validity
        Not Before: Feb  9 09:35:07 2021 GMT
        Not After : Apr  9 17:40:01 2022 GMT
        Subject: C=GB, O=cpwebuser, CN=ecec
        Subject Public Key Info:
          Public Key Algorithm: id-ecPublicKey
            Public-Key: (256 bit)
            pub:
               04:5d:15:78:83:e0:ab:af:78:69:2f:14:51:fe:29:
            ...
                ASN1 OID: prime256v1
                NIST CURVE: P-256
...

Binary

The data is encoded using Abstract Syntax Notation(asn). For example you get a string of fields like

  • The following fields are a sequence
  • of length 24
  • it is a string format
  • the first item has length 8
  • the attribute type is 06 03 55 04 06 which means Country
  • the attribute value is GB
  • the second item is …

This would describe the C=GB in C=GB, O=SSS, OU=CA, CN=SSCA256.

Binary portable (base 64 encoding)

With the binary format, the data is a hex string, which is not very portable. For example if you FTP it to a remote site, you may get data conversion, and new lines changed. Often the data is converted to an intermediate form called base 64 encoding. Every 3 hex characters are converted to 4 “printable” characters A-Z,a-z,0-9 and some special characters.

The file looks like

-----BEGIN CERTIFICATE-----
MIICDjCCAbSgAwIBAgICAXswCgYIKoZIzj0EAwIwOjELMAkGA1UEBhMCR0IxDDAK
BgNVBAoMA1NTUzELMAkGA1UECwwCQ0ExEDAOBgNVBAMMB1NTQ0EyNTYwHhcNMjEw
...
hEzE3hSzUvbfLUKnua64AiEAnyKhrmBjXCVQ834VfjhNHYqjaN1PzXBE8Cv3jlWJ
9bA=
-----END CERTIFICATE-----

What are the keystore types?

When Secure Sockets Layer was being developed each major player came up with their own format for storing the certificate and key information.

  • Java had files with type .jks (java key store) for example keystore.jks
  • IBM had files with type cms (certificate management system) with file types like zzserver.crl, zzserver.kdb, zzserver.rdb, zzserver.sth. It also uses intermediate files such as .arm, which has base 64 encoding of a certifcate, as shown above.
  • Netscape had files in a NSS database, for example files in the directory /home/colinpaice/.pki/nssdb, are files cert9.db and key4.db. key4 is the keystore database (storing keys), and cert9 is the trust store database (storing certificates).
  • OpenSSL developed .p12 files which can contain certificates and keys. It also has .pem (base 64 encoded) and .der (binary) files for individual certificate, private key, and public key files .
  • Windows has .pfx files.
  • Smart cards where the keystore is on a special USB type device or other external Hardware Security Module, each have their own format keystore.

Standards were developed to work with keystores

  • pkcs11 -the programming interface to create and manipulate cryptographic tokens (a token where the secret is a cryptographic key).
  • pkcs12 – defines an archive file format for storing many cryptography objects as a single file. It is commonly used to bundle a private key with its X.509 certificate or to bundle all the members of a chain of trust. Openssl uses one certificate per file, you can have multiple certificates in a pks12 file is you use keytool.

When can different keystore types be used?

With C programs on mid range and z/OS, IBM products use GSKIT from IBM. On z/OS you can store them in the z/OS security manager (for example RACF) or on the Hardware Security Module in the processor.

Java program can use most types of keystore. You may need to configure the java.security configuration file with an entry like

  • security.provider.X=com.ibm.security.cmskeystore.CMSProvider
  • security.provider.Y=com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl /home/colinpaice/mq/nitrokey.cfg

The first line includes support for cms files, the second includes support for a pkcs11 external keystore on an HSM (including its configuration file).

For Java programs, you configure the keystore using start-up options. The following example defines the keystore as an external USB HSM keystore, and the trust store is a openssl .p12 store.

-Djavax.net.ssl.keyStore=NONE
-Djavax.net.ssl.keyStoreType=pkcs11
-Djavax.net.ssl.keyStorePassword=12345678

-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/mytrust.p12
-Djavax.net.ssl.trustStorePassword=password
-Djavax.net.ssl.trustStoreType=pkcs12

These definitions along with the java.security, and its override, show the keystore is a pkcs11, and there is an entry in the java.security (above) pointing to the configuration file /home/colinpaice/mq/nitrokey.cfg for the pkcs11 definition. You can override or extend the options in the java.security file using configuration using

-Djava.security.properties=/home/colinpaice/mq/colin.java.properties

Do these keystores share information?

No, if you have a keystore (or a USB HSM) it shares no information with any other keystore. For example you could unplug the USB keystore, take it to another machine and plug it in, and the certificate etc will be available. You could send a .jks file, to yet another machine, and it could be used.

What can you use to administer the keystore.

IBM provided

You can use GSKCapiCmd_64 from IBM GSKIT to manage certificates and keys.

MQ has runmqakm as a command line tool which invokes GSKCapiCMD_64 under the covers.

You can also use the MQ command strmqikm to have a GUI to manage your keys and certificates. strmqikm just invokes the GSKIT ikeyman program.

These solutions feel a little dated, as I could not find if they support modern certificates capabilities; OCSP, and elliptic curves. They had a few bugs as well.

I’ve used openssl to create the “modern” private key and certificates, then imported them into the keystores using the IBM tools.

Oracle provided

With Java there is keytool. This supports a variety of keystores (depending on what has been configured in its java.security file), including

  • Java Key Stores .jks. This keystore can only be processed by java
  • Java Cryptography Extension KeyStore – a stronger version of .jks. This keystore can only be processed by java
  • pkcs12 .p12 and .pfx. These keystores can be used by Java, C and other languages, with the right API. For example Curl uses a C api to access the keystores.
  • pkcs11 – keystores on smart devices
  • nss – netscape security.

It does not support cms format keystores used by GSKIT.

Openssl

This is open source and is active, keeping up with the trends in security. Openssl deals with the building blocks; private keys, certificates etc but does not handle keystores very well. You can create manage certificates, but you may need other tools to put them into keystores.

opensc for managing smart keys and other pkcs#11 devices

(Opensc is open Smart Card.) If you have a smart key, or external keystore on an HSM, there may be hardware specific libraries for accessing the keystore, or the open source code may be supported. This can provide drivers for other tools, such as the java tools. You can also use pkcs11-tool to directly administer keys and certificates on an HSM device.

Other tools

  • pk12util from Netscape is used to mange keys in the NSS keystore databases
  • certutil from Netscape is used to manage keys and certificate in the NSS database.
  • Firefox and Chrome browsers can be used to update the NSS keystore used by the browsers.

What tool would I use when?

Creating the private key

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool -keypairgen
  • To generate an elliptic curve, use openssl ecparam
  • To generate an RSA, use openssl genpkey
  • Use keytool keypairgen to generate an elliptic or RSA key
  • runmqakm and strmqikm(ikeyman) do not seem to support elliptic curves, but support RSA keys as part of a certificate request (see below)

Create a certificate request

The certificate request takes (or creates) the public key, creates a certificate with the DN ( eg CN=Colin,C=GB,O=SSS and creates the certificate request file.

  • openssl req takes a file from the keypairgen
  • keytool can create a certificate request, I could not see how to use the keypair gen private key as part of this
  • runmqakm and strmqikm (ikeyman) can create a request, but I do not think it supports all flavours of private key.

Sign a request

I expect most people will want a tool which you can run as a script

  • openssl ca – you can include the optional attributes, with this.
  • keytool – I could not see how to use this to sign a certificate. The documentation suggest using openssl x509.
  • runmqakm – you can use runmqakm -cert -sign .
  • strmqikm (ikeyman) I could not see how to sign a request using this, and it is a GUI.

Receive the signed certificate into the keystore

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool –write-object
  • Use openssl pkcs12 to create a pkcs12 (.p12) keystore using the private key and signed certificate
  • Use keytool -importcert. This can support most keystore types, depending on the configuration in the java.security file.
  • runmqakm and strmqikm (ikeyman) import ( receive) the certificate and store it in the keystore.

Update the browser’s keystore

The browsers have an nss format store.

  • pk12util -i …imports a .p12 keystores into the nss “sql:” keystore. The sql: keystore is a Netscape internal format store. (you use certutil … to remove entries from the nss keystore).
  • You can use the browser’s facilities to import certificates and keys into the browsers (nss) keystore.

Receiving a public certificate into a keystore.

If you have a Certificate Authority or a self signed certificate you want to put into your trust store.

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool –write-object
  • Openssl is not very good at adding new entries to an existing .p12 file
  • runmqakm -cert -add -file ca256.pem…
  • strmqikm, select Signer Certificates, then select Add
  • Use keytool -importkeystore.  To import a .p12 store, or a .pem file I have a shell script

ks=” -destkeystore mytrust.p12″
dest=”-deststoretype pkcs12 -deststorepass password”
src=”-srcstoretype PKCS12 -srcstorepass password”
keytool -importkeystore $ks $dest -srckeystore ca1024.p12 $src
or
keytool -import $ks $dest -file carsa1024.pem 

Which keystore should I used for which application?

All applications can use smart cards and external keystores through the standard key stores.

  • A queue manager on midrange uses cms format files, so you need to use the runmqakm or strmqikm similar files
  • A queue manager on z/OS can use the z/OS security manager (eg RACF), or the cms keystore files.
  • Firefox and Chrome browsers use the NSS format keystores.
  • Java clients can use a variety of keystores, .jks, .p12, smart cards
  • A C, .Net etc client use cms format keystores
  • Curl can use .pem files (from openssl) and .p12 files
  • A web server has trust stores, and keystores. You configure the supported formats in the java.security file. It can use jks, .p12 and smart cards.

Summary

It all looks a bit of a mess, and you need to know a lot to get your job done.

Debugging external smart cards and external pkcs11 keystores.

There is an open source package (opensc) which provides access to smart cards and external keystores. It provides some good tools for diagnosing problems.

There is a wiki with good information.

Opensc return codes are here, and the printable text is here

Monitor traffic to and from the device.

You can monitor the traffic to and from the device by using an intermediate “spy” module which displays the traffic.

In your configuration (for example a CCDT), where you specified the name of the module /usr/lib64/pkcs11/opensc-pkcs11.so, replace this with /usr/lib64/pkcs11/pkcs11-spy.so. Specify the environment variable

export PKCS11SPY=/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so

The spy module is invoked, prints out the parameters, and then invokes the module specified in the environment variable.

The output is like

19: C_Login
2021-03-10 14:22:47.947
[in] hSession = 0x21fc030
[in] userType = CKU_USER
[in] pPin[ulPinLen] 00000000021fb2a0 / 8
00000000 5B C7 E7 BB E5 FC 6A BE […..j.
Returned: 160 CKR_PIN_INCORRECT

Detailed internal trace

You can specify the environment variable OPENSC_DEBUG to give a very detailed trace. The higher the number the more detailed the trac.

export OPENSC_DEBUG=9

and use unset OPENSC_DEBUG to reset it.

You can use OPENSC_CONF to specify a configuration file with more parameters, such as file name for the output.

The output from this trace (showing a logon with pin number 12345678) is like

0x7f96e2dca740 14:13:16.756 [opensc-pkcs11] framework-pkcs15.c:1494:pkcs15_login: pkcs15-login: userType 0x1, PIN length 8
0x7f96e2dca740 14:13:16.756 [opensc-pkcs11] pkcs15-pin.c:301:sc_pkcs15_verify_pin: called
….
0x7f96e2dca740 14:13:16.757 [opensc-pkcs11] reader-pcsc.c:283:pcsc_transmit: reader ‘Nitrokey Nitrokey HSM (DENK01051600000 ) 00 00’
0x7f96e2dca740 14:13:16.757 [opensc-pkcs11] reader-pcsc.c:284:pcsc_transmit:
Outgoing APDU (13 bytes):
00 20 00 81 08 31 32 33 34 35 36 37 38 . …12345678

GSKIT return codes

If you are using the MQ C Client interface, this uses GSKIT. There is documentation for the z/OS version, and the return codes are here.

Getting a Java program to use a TLS MQ channel, and use an external USB keystore.

This project started off trying to get a java program to use an HSM (external keystore on a USB). This was not well documented, so I have documented my path.

I had set up my keystore on the USB HSM device as described here. This post covers the MQ side of it.

MQ provides some JMS samples.

I could use JmsBrowser using local bindings with the jms.sh script.

. /opt/mqm/java/bin/setjmsenv64
java -Djava.library.path=/opt/mqm/java/lib64 JmsBrowser -m QMA -d CP0000

To use a client channel (QMACLIENT) was easy, the program is configured to allow a channel name to be passed as a parameter using the -l option.

. /opt/mqm/java/bin/setjmsenv64
java -Djava.library.path=/opt/mqm/java/lib64 JmsBrowser -m QMA -d CP0000 -h localhost -p 1414 -l QMACLIENT

To use a TLS channel (QMAQCLIENTTLS) proved harder

. /opt/mqm/java/bin/setjmsenv64
java -Djava.library.path=/opt/mqm/java/lib64 JmsBrowser -m QMA -d CP0000 -h localhost -p 1414 -l QMAQCLIENTTLS

This gave me

CC=2;RC=2397;AMQ9641: Remote CipherSpec error for channel ‘QMAQCLIENTTLS’ to host ”. [3=QMAQCLIENTTLS]

It was hard to find where to specify the CipherSpec. The documentation said “Using the environment variables MQCHLLIB to specify the directory where the table is located, and MQCHLTAB to specify the file name of the table.” I tried this and got the same error.

I collected a trace using the Java option -Dcom.ibm.msg.client.commonservices.trace.status=ON , but the trace showed the cipher spec was null.

I gave up with this program and used an enhanced version of the program. JMS programs can use JNDI as a file repository of configuration information and the program can extract the information. For example you can configure Connection Factories(cf) which are MQ connections, and Q objects – which are queues.

Use JNDI to store parameters

JMS programs can use a JNDI interface to access configuration parameters stored in an external file.
I used this article to get me started.

My JMSAdmin.config had

INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory
PROVIDER_URL=file:///home/colinpaice/mq/JNDI-Directory
SECURITY_AUTHENTICATION=none

I used the following commands to define a connection factory called CF1, using the channel, QMACLIENT, and a queue reference called MYQUEUE pointing to queue CP0000.

mkdir JNDI-Directory
/opt/mqm/java/bin/JMSAdmin -v -cfg JMSAdmin.config
define cf(CF1) transport(client) channel(QMACLIENT)
define q(MYQUEUE) queue(CP0000)
end

I used a bash script to run the program, using the JmsJndiBrowser sample (JmsBrowser with JNDI support).

. /opt/mqm/java/bin/setjmsenv64
java -Djava.library.path=/opt/mqm/java/lib64 JmsJndiBrowser -i file:///home/colinpaice/mq/JNDI-Directory -c CF1 -d MYQUEUE

This failed with

JMSCMQ0001: IBM MQ call failed with compcode ‘2’ (‘MQCC_FAILED’) reason ‘2400’ (‘MQRC_UNSUPPORTED_CIPHER_SUITE’).

Which was a surprise to me. I looked in the trace, and there was nothing obviously wrong. The channel was not configured for TLS.

I used the following command in the JMS Admin tool, to specify a TLS channel and configure the cipher suite

alter cf(CF1) channel(QMAQCLIENTTLS) SSLCIPHERSUITE(ANY_TLS12)

Note. If I displayed cf(CF1) it gave me SSLCIPHERSUITE(*TLS12).

This gave me unable to find valid certification path to requested target, which was good progress because it meant the cipher spec had been accepted.

I added in the Java parameters to identify the key store and trust store

. /opt/mqm/java/bin/setjmsenv64

kss=”-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ecec.p12″
ksp=”-Djavax.net.ssl.keyStorePassword=password”
kst=”-Djavax.net.ssl.keyStoreType=pkcs12″
tss=”-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/dantrust.p12″
tsp=”-Djavax.net.ssl.trustStorePassword=password”
tst=”-Djavax.net.ssl.trustStoreType=pkcs12″
ks=”$kss $ksp $kst”
ts=”$tss $tsp $tst”

java $ks $ts -Djava.library.path=/opt/mqm/java/lib64 JmsJndiBrowser -i file:///home/colinpaice/mq/JNDI-Directory -c CF1 -d MYQUEUE

and it worked successfully.

Use the pkcs11 HSM external keystore.

I changed the file to specify the external keystore (with format pkcs11) on a USB device rather than a file on disk (format pkcs12).

kss=”-Djavax.net.ssl.keyStore=NONE”
kst=”-Djavax.net.ssl.keyStoreType=pkcs11″
ksp=”-Djavax.net.ssl.keyStorePassword=12345678″

but this gave

find /java.security.KeyStoreException: pkcs11 not found
java.security.NoSuchAlgorithmException: pkcs11 KeyStore not available

You have to tell Java where to find the pkcs11 code. This is configured in the java.security file. You can find it using find $JAVA_HOME/ -name java.security . This gave me the file name /usr/lib/jvm/java-8-oracle/jre/lib/security/java.security

It had the list of security providers,

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC

and is missing an entry for pkcs11.

You can edit this file, or override it. To override it, create a file like colin.java.properties with

security.provider.10=SunPKCS11 /home/colinpaice/mq/nitrokey.cfg

where

  • .10 is the next in sequence after the .9 in the java.security file
  • SunPKCS11 says use the Sun module.
  • /home/colinpaice/mq/nitrokey.cfg use this configuration file for the pkcs11 device.

Tell Java to use this override file, by using the Java option -Djava.security.properties=/home/colinpaice/mq/colin.java.properties.

The configuration file name for the pkcs11 is /home/colinpaice/mq/nitrokey.cfg with content

name = nitrokey
library = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
slot=0

With these parameters the output from the JmsJndiBrowser command was

Initial context found!
Browse starts
No more messages
SUCCESS

Success – and it only took me half a day! The hardest part was known what to specify for the security.provider…… parameter.

Using the runmqakm commands and an HSM (but not strmqikm).

I tried to use strmqikm but it gave an exception.

You can use some of the runmqakm commands you know and love, to access a certificate with an HSM. For example

The command to list the database available to the runmqakm command,

runmqakm -keydb -list -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

Gives

/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so : UserPIN (mytoken)

You can then use the token label UserPIN (mytoken) and password to use the key store, for example

runmqakm -cert -list all -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678

gives

Certificates found
* default, - personal, ! trusted, # secret key
-	my_key3

and

runmqakm -cert -details -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678
-label my_key3

displays the details of the certificate with label my_key3.

If the -tokenlabel was wrong or the -pw was wrong, I got the unhelpful messages

  • CTGSK3026W The key file “pkcs11” does not exist or cannot be read.
  • CTGSK2137W The label does not exist on the PKCS#11 device.

Create your certificate request

The following command create a new RSA private-public key pair and a PKCS10 certificate request. The documentation for runmqakm says it supports RSA. If you want to use an Elliptic Curve you will need to use an alternative method, for example openssl.

runmqakm -certreq -create -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678
-dn “cn=colin,o=SSS” -file runmq.csr -label runmqlab -size 1024

Sign it

openssl ca -config openssl-ca-user.cnf -policy signing_policy -md sha256 -cert carsa1024.pem -keyfile carsa1024.key.pem -out runmq.pem -in runmq.csr

Store it back into the HSM keystore

I could not get the runmqakm command to receive the signed certificate and store it into the HSM keystore.

runmqakm -cert -receive -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -tokenlabel “UserPIN (mytoken)” -file runmq.pem -pw 12345678

It failed with

CTGSK3034W The certificate request created for the certificate is not in the key database.

I could use

openssl x509 -inform pem -outform der -in runmq.pem -out runmq.der
pkcs11-tool –write-object runmq.der –type cert –label “runmqlab” -l –pin 12345678

The openssl command converts the file from .pem format, to .der format as .der format is required by pkcs11-tool.

Using strmqikm – the theory

If you want to use the strmqikm GUI, you have to configure the java.security file. For example edit /opt/mqm/java/jre64/jre/lib/security/java.security and add the next security.provider in the list.

security.provider.12=com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl /home/colinpaice/mq/nitrokey.cfg

Where /home/colinpaice/mq/nitrokey.cfg is the configuration file, with

name = nitrokey
library = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
slot=0

You can then use Ctrl+O, which brings up a pop up with “Key database type”. In this list should be PKCS11Config, if not check your java.security file. Select this, leave File Name and Location empty, and click “OK”. It pops up “Open Cryptographic Token” with the “Token Label” value taken from the configuration file name = nitrokey. This is strange as the runmqakm command uses a TokenLabel of “UserPIN (mytoken)”.

In practice…

I then got an exception java.lang.RuntimeException: PKCS11KeyStore.java: findSigner(): Failure while executing cobj.getX509Certificate(certFactory, session), and strmqikm ended.