Setting up TLS on MQ Midrange

The weather is still cold up in Orkney (north of Scotland), and as I had been working on documenting TLS connections to web servers and how to get it working, I thought I would look at migrating my “enterprise” ( two queue managers and a couple of clients) to use TLS 1.3.  It took me a couple of days to set up a TLS 1.2 client to queue manager.  I spent a long time with my head in my hands thinking “why is this not working!”.
There is lots of documentation, from IBM and others, it was hard to find the best set of instructions.  I would classify the information I found as “how to dip your toe into the water” rather than “how to swim”.  At times I felt like this was a puzzle where the instructions say “There are false clues to misdirect you, and some of the clues have spelling mistakes – just to make it more interesting”. 

As well as this post I’ve created a web page of updates and suggestions to the IBM messages, with hints on where to look, and which traps to avoid.

As I was editing this document I found an excellent document on setting up MQ Clients and TLS with a java program.  The blog post below does not start from scratch, it takes an existing queue manager, and existing certificates and gets a C sample program to work.

Initial observations on key and trust stores.

Having been deep into the TLS specification and flows I know that there are two logical stores:

1. The key store is where you store the private key used by this application.  The store typically has just one private key.  If someone has access to this store they can encrypt decrypted data using the key.
2. The trust store. This stores the certificates needed to validate any certificates sent to it.  This typically has the Certificate Authorities, and any self signed keys (which you will used only in a sandbox environment).  You can have one trust store for all of your production applications – and other trust stores for your test environment.  Sharing a trust store is good practice, as this means you update a certificate, and every user gets the update.  If you have many trust stores you have to update them all when there is an update for a certificate.

MQ uses one store which is the trust store plus any private keys.  The disadvantage of this you have more work to keep the stores up to date with changes to certificates.

You could have a group of production queue managers sharing the same store.  You could give them each queue manager their own private key – or they could all use the same private key.  In the early days of MQ, each queue manager had a unique  certificate label (alias).  In the keystore for queue manager QMA, the label was  ibmwebspheremqqma.  You can now specify the label(alias) using the CERTLABL queue manager attribute, and on a channel.

In my work with the Liberty Web Server, I used key stores with key store types of .jks (deprecated)  and .p12 (for pkcs12).  On Midrange,  MQ uses a .CMS format used by IBM GSKIT.  On z/OS you use RACF or other z security products.   Java clients can use a variety of stores: .jks, .pem,  .p12; the list is what Java supports. Other clients, using the the GSKIT interface can only use .CMS stores.

You can import key stores into a .CMS store, so it is not a big effort to create a .CMS store – but it is an extra step to keep it up to date.

Before you start planning

(Not to be confused with doing the planning after you have started going in the wrong direction).

You need to think about what certificates and cipher suites names you can use.  See here.

Setting up TLS

I followed the IBM documentation but this was not very helpful.  For example it says “Create a key database file in the location that is specified in the queue manager’s Key repository attribute” but does not tell you how to do it.

I wanted to reuse the certificates I already had, so telling me how to create new certificate was not very useful.

The 30 second hints before you start, to avoid some of the holes in the road that I hit.

1. Get clients connecting to your queue manager before trying to convert to TLS.
2. Explicitly specify all  MQ environment variables,  or explicitly unset them.  For example some variables override mqclient.ini parameters, and some do not.  I spent hours debugging a problem because I had an environment variable set globally for another application.
3. Check you do not have MQSERVER environment variable specified, or ServerConnectionParms in the mqclient.ini.
4. Specify CERTLABL on each channel, to allow different cipher specs and certifcates to be used on different channels.
5. Create a client (group) specific CCDT either with JSON or mqscutil for this group of clients.  Using a queue manager generated CCDT  may not give you the channel you wanted.
6. The server’s certificate must be compatible with the cipher spec and both ends of the channel.    For example all RSA or all Elliptic curve.  This is easy to get wrong.
   1. The cipher spec of ANY_TLS12 etc available in V9.1.x make this much easier, as this uses a list of certificates.
   2. If you can use this at either end then do so.
7. Each end needs read access to the key store files.
8. You can create new certificates just for MQ, or reuse existing certificates.
   1. Java support can use a variety of key stores.
   2. Other programs need a .CMS format key store.
9. Always check after making a change, as runmqsc upper cases values unless they are quoted.
10. Remember to use refresh security type(ssl) after changing any of the queue manager’s SSL parameters, or changed the keystore.
11. If you think it has worked, use DIS CHS(…) on the channel and make sure has picked the correct channel.  By default it finds the first channel in the CCDT for the queue manager.  For me this was not configured for TLS – so it appeared worked !!!

My journey.

What tools can help me set up my .cms keystore?

1. GUI you can use strmqikm (STaRtMQ Interactive KeyManagement).
2. Command line runmqckm (RUNMQCommandlineKeyManagement).
   
3. For strong cipher keys, (eg Elliptic Curve public keys ), and Federal Information Processing Standards (FIPS) use the command line program runmqakm.  If you always use it instead of runmqckm, you will get muscle memory.  It provides a superset of functions over runmqckm.
   

The command line commands give the options available with the commands.  So you do not need the online documentation.  For example

runmqakm -keydb shows the options available for the -keydb option.

CTGSK3000W An action must be specified for this object.-Command usage-
Object Action Description
------ ------ -----------
-keydb 
   -changepw Change the password for a key database
   -convert Convert the format of a key database
   -create Create a key database
   -delete Delete a key database
   -expiry Display password expiry [Deprecated]
   -list Currently supported types of key database.
   -stashpw Stash the password of a key database into a file

Create the keystore for the server. 

I used a bash script to make it easier to change a parameter and rerun the commands.

The script

1. deletes the old keystore
2. creates the keystore
3. adds in the Certificate Authority and self signed certificates
4. imports the certificate and private key used in the handshake
5. lists the contents of the keystore
6. sets the authorities to the keystore.

db=”zzserver.kdb “
runmqakm -keydb -delete $db -pw password
# create the cms keystore
runmqakm -keydb -create -db $db -type cms -pw password -stash
# add the CA certificate
runmqakm -cert -add -file /home/colinpaice/ssl/ssl2/ca256.pem -type cms -stashed -db $db -label ca256
# and the self signed  called ss in the ss.p12 keystore
runmqakm -cert -import -file ~/ssl/ssl2/ss.p12 -type pkcs12 -pw password -target $db -target_stashed -label ss -new_label SS_SERVER
# add the certificate and private key into the store
#  it has a label ecc in in a pkcs12 file in ~/ssl/ssl2/ecec.p12
runmqakm -cert -import -file ~/ssl/ssl2/ecec.p12 -type pkcs12 -pw password -target $db -target_stashed -label ecec -new_label ECEC_SERVER
runmqakm -cert -list all -db $db -type cms -stashed
# give mqm access to the files
chown :mqm zzserver.*
chmod g+r zzserver.*

This created 4 files zzserver.crl zzserver.kdb zzserver.rdb zzserver.sth.

Check mqm can access it.  Run as mqm id

sudo -u mqm runmqakm -cert -list all -db zzserver.kdb -type cms -stashed

Define the keystore to the queue manager

Use the mqsc commands

alter qmgr SSLKEYR(‘/home/colinpaice/mq/zzserver’) CERTLABL(‘ECEC_SERVER’)
refresh security type(SSL)

Note the quotes around the values, without them the text is converted to upper case.

Define the channels for the TLS connection

I had an existing client connection QMACLIENT, and wanted to create a new channel QMAQCLIENTTLS.

I used the mqsc commands

• define chl(QMAQCLIENTTLS) chltype(CLNTCONN) like(QMACLIENT)
• define chl(QMAQCLIENTTLS) chltype(SVRCONN) like(QMACLIENT)
• alt chl(QMAQCLIENTTLS) chltype(SVRCONN) SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
• alt chl(QMAQCLIENTTLS) chltype(CLNTCONN) SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)

In theory I could use the system generated CCDT table which contains client information, but this did not work in practice.  See below.

I copied the AMQCLCHL.TAB to my working directory.

cp /var/mqm/qmgrs/QMA/@ipcc/AMQCLCHL.TAB .

Edit the mqclient.ini

I changed the mqclient.ini file to include

CHANNELS:
   ServerConnectionParms=QMAQCLIENTTLS/TCP/127.0.0.1(1414)
   ChannelDefinitionDirectory=/home/colinpaice/mq
   ChannelDefinitionFile=AMQCLCHL.TAB
SSL:
  CertificateLabel=ECEC_P
  SSLKeyRepository=/home/colinpaice/mq/zzclient

My first run

I used a a bash script which ended up as

# display time - useful for looking at trace 
date
# remove the old client trace files
sudo rm /var/mqm/trace/AMQ*

export MQCLNTCF=/home/colinpaice/mq/mqclient.ini
export MQSSLKEYR=/home/colinpaice/mq/zzclient
export MQCHLLIB=/home/colinpaice/mq
export MQCHLTAB=AMQCLCHL.TAB

strmqtrc -e 
/opt/mqm/samp/bin/amqsgetc CP0000 QMA
endmqtrc -e

This gave me

MQCONNX ended with reason code 2393

Problems I experienced

Trying to use ServerConnectionParms

I got 2393 SSL Initialization error, and less /var/mqm/errors/*01* had

AMQ9641E: Remote CipherSpec error for channel ‘QMAQCLIENTTLS’ to host
‘localhost (127.0.0.1)(1414)’.

EXPLANATION:
The remote end of channel ‘QMAQCLIENTTLS’ on host ‘localhost (127.0.0.1)(1414)’ has indicated a CipherSpec error ‘SSLCIPH(‘ ‘) -> SSLCIPH(????)’. The channel did not start.

Note the blank cipher spec. My mqclient.ini had ServerConnectionParms=QMAQCLIENTTLS/TCP/127.0.0.1(1414) which cannot be used for TLS channels.  See ServerConnectionParms. It overrides the ChannelDefinitionFile

I commented out this statement, so the CCDT would be used.

Wrong channel used from the CCDT.

The program seemed to work successfully, until I used DIS CHS(QMA*) when I saw the wrong, a non TLS channel, was being used.

When the CCDT is used, the entries are searched for the first entry which matches the queue manager name.

I found this by taking a trace, see Debugging MQ client connection problems:channel name.

The first channel matching was QMACLIENT which is non TLS.   I had to create a CCDT just for the  TLS channel.

You can have a json format file, see here for the attribute names or a binary ccdt from runmqsc.

Creating a ccdt file using runmqsc

You can use the runmqsc -n command to display or manipulate a binary ccdt file.

I created a file using the same parameters as the queue manager version.

The file it uses depends on the environment variables  for example

export MQCHLLIB=/home/colinpaice/mq
export MQCHLTAB=COLIN2.TAB
runmqsc -n

Set the environment variable in the run shell, and remove the statement from the mqclient.ini .

The wrong keystore was used

I had both an environment variable, and an entry in the mqclient.ini  (SSL: SSLKeyRepository=/home/colinpaice/mq/zzcolin)  for the keystore.

I debugged this using  Debugging MQ client connection problems:keystore.

The environment variable had precedence over the mqclient.ini.

I had mistyped the certificate label for the client

The server’s /var/mqm/qmgrs/QMA/errors/*01* file had

• AMQ9638E: SSL communications error for channel ‘????’.
• AMQ9999E: Channel ‘????’ to host ‘127.0.0.1’ ended abnormally.

The clients /var/mqm/errors/*01* had

• AMQ9716E: Remote SSL certificate revocation status check failed for channel
  ‘QMAQCLIENTTLS’.

• AMQ9642E: No SSL or TLS certificate for channel …

I had inconsistent Cipher specs.

As part of the handshake there is logic like..

• Client sends “I want to negotiate in  the following languages…”
• The Server looks in its list of languages and makes sure there is at least one in common.
• The server asks its certificate do you speak any of the following languages.

If any of the above fail you get the AMQ9616E: The CipherSpec … proposed by remote host … is not enabled error message.

The client can send up

• One cipher spec for example specify  SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
• A list of cipher specs SSLCIPH(ANY_TS12)

The server may have

• One cipher specified SSLCIPH(ECDHE_ECDSA_AES_128_CBC_SHA256)
• A list of cipher specs SSLCIPHR(ANY_TS12)

The server’s certificate could be, for example,

• Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (384 bit)

or

• Subject Public Key Info: Public Key Algorithm: rsaEncryption  RSA Public-Key: (2048 bit)

The bold text must match the agreed cipher spec.

Display connection information

Once the connection has been established you can use the DIS CHS(…) all command on the server using runmqsc.
This has information like

SECPROT(TLSV12)
SSLCERTI(CN=SSCA256,OU=CA,O=SSS,C=GB)
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(SERIALNUMBER=01:79,CN=rsaca256,O=cpwebuser,C=GB,UNSTRUCTURED...)
SSLRKEYS(0) STATUS(RUNNING)

This shows that the cipher spec used is TLS_RSA_WITH_AES_128_CBC_SHA256 and the DN from the client was had a DN with CN=rsaca256… .

I could not find the equivalent command on the client end.

You can get this from the client trace see debugging mq client connection problems:server cert