Getting SSL/TLS to work on MQ on z/OS

After I succeeded in getting TLS 1.3 to run on MQ  midrange 9.2, I thought I would try it on z/OS.  I had not used TLS on z/OS for about 10 years, so it was almost like coming to the topic with very rusty knowledge.

I searched the Knowledge centre and found no relevant hits – lots of hits which were not relevant.  I eventually found an SSL related keyword, and this got me to the topic  Working with SSL/TLS on z/OS.   I think this is well documented.  It covered all the things I had to do.

The remained of this post covers the bits not covered by the documentation.

Define SSLTASKS.

You need to define SSLTASKS to be able to use TLS on z/OS.  See the comments here. I used

%CSQ9 ALTER QMGR SSLTASKS(5)

You need to restart the CHINIT if you change the value of SSLTASKS.

Set up the keyring for the queue manager. 

See here.  This post show how to create the keyring and import a CA from z/OS, and import a CA  from a Linux system.

If you alter the keyring or certlabl you just need a refresh security type(SSL) command to pick up the changes.

Defining the channel

I tried to define the channel, as this failed for security reasons, I’ve given the RACF setup I had to do.

In this section I defined the specific commands for example DEFINE.CHANNEL.   I could have defined DEFINE.* to allow all define commands.

I used a channel called TLS, and define the resource CSQ9.CHANNEL.TLS* to allow my ID to define TLS, TLS1 etc

The command %CSQ9 DEF CHL(TLS) CHLTYPE(SVRCONN) gave me

ICH408I USER(CSQOPR ) GROUP(SYS1 ) NAME(COLIN PAICE ) 167
CSQ9.DEFINE.CHANNEL CL(MQCMDS )
WARNING: INSUFFICIENT AUTHORITY – TEMPORARY ACCESS ALLOWED
ACCESS INTENT(ALTER ) ACCESS ALLOWED(NONE )

I used the RACF commands in a batch job.

 /* RDELETE MQCMDS CSQ9.DEFINE.CHANNEL
RDEF MQCMDS CSQ9.DEFINE.CHANNEL UACC(NONE)
PERMIT CSQ9.DEFINE.CHANNEL CLASS(MQCMDS ) –
        ID(COLIN,IBMUSER) ACCESS(ALTER )

I also set up CSQ9.DELETE.CHANNEL and CSQ9.ALTER.CHANNEL in a similar way, so my userid could maintain the channels.

I refreshed MQ security %CSQ9 refresh security to pick up the changes.

I reissued the command %CSQ9 DEF CHL(TLS ) CHLTYPE(SVRCONN) and got

ICH408I USER(COLIN ) GROUP(SYS1 ) NAME(COLIN PAICE )
CSQ9.CHANNEL.TLS CL(MQADMIN )
PROFILE NOT FOUND – REQUIRED FOR AUTHORITY CHECKING
ACCESS INTENT(ALTER ) ACCESS ALLOWED(NONE )

I used the RACF commands in a batch job.

  /* RDELETE MQADMIN CSQ9.CHANNEL.TLS*
RDEF MQADMIN CSQ9.CHANNEL.TLS* UACC(NONE) WARNING
PERMIT CSQ9.CHANNEL.TLS* CLASS(MQADMIN) –    
        ID(COLIN,IBMUSER) ACCESS(ALTER )
SETROPTS RACLIST(MQADMIN) REFRESH

I issued the commands

%CSQ9 refresh security
%CSQ9 DEF CHL(TLS ) CHLTYPE(SVRCONN)

And successfully defined the channel.

I changed the cipher spec.

I selected a cipher spec from the list.

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(ECDHE_RSA_AES_128_CBC_SHA256)

When I started the channel I got

CSQX631E … CSQXRESP Cipher specifications differ,  channel TLS local=ECDHE_RSA_AES_128_CBC_SHA256 remote=TLS_RSA_WITH_AES_256_GCM_SHA384
connection 10.1.0.2 (10.1.0.2)

This was clear;  I love clear messages.

I decided to change the z/OS end

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(TLS_RSA_WITH_AES_256_GCM_SHA384 )

and the client connected successfully.

With MQ 9.2 I could (and did) change this to

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(ANY_TLS12)

and the client worked successfully.  The ANY_TLS12.  provides a wide list of supported cipher specifications, includes TLS_RSA_WITH_AES_256_GCM_SHA384  and ECDHE_RSA_AES_128_CBC_SHA256. 

When I am ready to support TLS 1.3 I will use ANY_TLS12_OR_HIGHER and ANY_TLS13_OR_HIGHER.

Connect a client to it!

I had had my client connect to a midrange queue manager, so I had working client environment.  See here for the journey.

I created a .json file for the CCDT connection to z/OS.  I specified

{ "channel":
  [{
    "name": "TLS",
    "clientConnection":
    {
      "connection":
      [{
        "host": "10.1.1.2",
        "port": 1414
       }],
      "queueManager": "CSQ9"
    },
    "transmissionSecurity":
    {
      "cipherSpecification": "ANY_TLS12",
      "certificateLabel": "rsaca256_client",
      "certificatePeerName": ""
    },
    "type": "clientConnection"
  }]
}

When it connected I got messages

+CSQX511I %CSQ9 CSQXRESP Channel TLS started connection 10.1.0.2
ICH408I USER(COLINPAI) GROUP( ) NAME(??? )
LOGON/JOB INITIATION – USER AT TERMINAL NOT RACF-DEFINED
IRR012I VERIFICATION FAILED. USER PROFILE NOT FOUND.
+CSQX512I %CSQ9 CSQXRESP Channel TLS no longer active connection 10.1.0.2

COLINPAI came from the userid on the Linux machine (colinpaice) upper cased and truncated. This id will be flowed and used as the MCAUSER if you don’t set it to anything else, using CHLAUTH for example  (Thanks to Morag for this information).

Enable chlauth

To be able to map from the DN in a certificate to a z/OS userid you have to use MQ CHLAUTH.  See  Mapping a client user ID to an MCAUSER user ID.

Check it is enabled at the queue manager level and enable it it needed.

%CSQ9 DIS QMGR CHLAUTH
%CSQ9 ALTER QMGR CHLAUTH(ENABLED)

Define a mapping from certificate to userid

I used

//S1 EXEC PGM=CSQUTIL,PARM='CSQ9' 
//STEPLIB  DD DSN=COLIN.MQ921.SCSQLOAD,DISP=SHR 
//         DD DSN=COLIN.MQ921.SCSQANLE,DISP=SHR 
//SYSPRINT DD SYSOUT=* 
//SYSIN   DD * 
 COMMAND DDNAME(COMMAND) 
//COMMAND DD * 
 SET CHLAUTH('TLS') + 
     TYPE(SSLPEERMAP) SSLPEER('O="cpwebuser"') + 
     ACTION(REPLACE)   + 
     MCAUSER(ADCDD ) CHCKCLNT(ASQMGR) 
/* 

This says for channel TLS,  take the Organisation(O=..)  from the certificate, and if it is cpwebuser then set the ID to ADCDD.

Check it works

Once the channel had started I used

%CSQ( DIS CHS(TLS)
it displayed the following, where I have removed lines which are not relevant to TLS and added some comments

  • CHSTATUS(TLS)
  • CHLTYPE(SVRCONN)
  • SECPROT(TLSV12) – this is the level of the protocol
  • SSLCERTI(CN=SSCARSA1024,OU=CA,O=SSS,C=GB)- this is the DN of the issuer of the SSLPEER certificate (below)
  • SSLCERTU(START1) – the IBM documentation says “The local user ID associated with the remote certificate.”  I dont know where this comes from.. how to change it, or where it is used.
  • SSLCIPH(TLS_RSA_WITH_AES_256_GCM_SHA384) – The negotiated cipher spec
  • SSLRKEYS(0) -The number of successful TLS key resets.
  • SSLKEYTI() -The time on which the previous successful TLS secret key was reset.  The secret key has not been reset
  • SSLKEYDA() -The date on which the previous successful TLS secret key was reset.  The secret key has not been reset
  • SSLPEER(SERIALNUMBER=01:90,CN=rsaca256,O=cpwebuser,C=GB, UNSTRUCTUREDNAME=openssl_ca_user_cnf.keyAgreement2, UNSTRUCTUREDNAME=localhost, UNSTRUCTUREDADDRESS=127.0.0.1) . This is information from the certificate at the remote end.
  • MCAUSER(ADCDD) – This is the userid (set by the CHLAUTH above) used by this channel.
  • LOCLADDR(10.1.1.2(1414)) – This is the address the connection came in from.  This value will be different it you have different IP stacks and different listener ports.

Taking the brakes off ZFS on z/OS – move it to OMVS

From z/OS 2.2 there is a performance advantage in running the ZFS file system as part of OMVS, rather than its own address space.  The IBM documentation says When running zFS in the OMVS address space, each file system vnode operation (such as creating a directory entry, removing a directory entry, or reading from a file) will have better overall performance. Each operation will take the same amount of time while inside zFS itself. The performance benefit occurs because z/OS UNIX can call zFS for each operation in a more efficient manner.  This will be relevant when you application is doing a lot of file IO – for example using a web server.

This move is not documented – but it is really easy!  It is mentioned here. Instructions are hidden in the installation instructions here.

Before I started

The IBM doc says You can determine if zFS is in its own address space by issuing D OMVS,PFS. If the output shows an ASNAME value, zFS is running as a colony address space.

OMVS     0010 ACTIVE             OMVS=(00,01,BP,IZ,RZ,BB)                
PFS CONFIGURATION INFORMATION
PFS TYPE ENTRY ASNAME DESC ST START/EXIT TIME
...
ZFS IOEFSCM ZFS LOCAL A 2021/02/17 17.35.06

The steps I took…

  1. I added KERNELSTACKS(ABOVE) to USER.Z24A.PARMLIB(BPXPRM00).
  2. Being ultra cautious I re-ipled.
  3. The documentation talks about putting IOEZPRM DD in OMVS, then goes on to say As the preferred alternative to the IOEZPRM DDNAME specification, delete the IOEZPRM DDNAME and use the IOEPRMxx parmlib member.  So I did not change the OMVS proc.  When I reipled it worked and I got the message IOEZ00374I No IOEZPRM DD specified in OMVS proc. Parmlib search being used. 
  4. I edited USER.Z24A.PARMLIB(BPXPRM00) and removed the ASNAME in  FILESYSTYPE TYPE(ZFS) ENTRYPOINT(IOEFSCM)ASNAME(ZFS) Well I actually made a copy of the original line and put it between /* and */, then deleted the text.
  5. I reipled.

Afterwards

The  D OMVS,PFS command now gives  N/A instead of the Address Space Name

OMVS     0010 ACTIVE             OMVS=(00,01,BP,IZ,RZ,BB)                
PFS CONFIGURATION INFORMATION                                         
 PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
...
  ZFS       IOEFSCM    N/A       LOCAL     A     2021/02/17 17.55.47  

Easy!

The hardest part was making sure I had an IPLable SARES1 in case I got it wrong!

Issuing commands…

I used to issue commands like f zfs,query,all. Now that the ZFS address space does not exist, you need to use  f omvs,pfs=zfs,query,all.

Setting up the MQ keyring on z/OS

I wanted to connect some clients to my z/OS queue manager over a TLS channel.  This post describes how I set up the z/OS keyring with the certificates.

Define the keyring.

The CSQ9 CHINIT runs with a userid of START1, so I defined a keyring belonging to that id.

I set up a dataset  called IBM.MQCSQ9.KEYRING to keep all of my JCL in for the CSQ9 queue manager.  This makes it easier to clone the definitions for another queue manager.

The definitions create the keyring, and add the z/OS CA certificate (CERTAUTH ADCD_CA) to it.

//IBMRACF JOB 1,MSGCLASS=H 
//* Use JCL for the RACF definitions
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
/*RACDCERT DELRING(MQRING) ID(START1)
RACDCERT ADDRING(MQRING) ID(START1)
RACDCERT ID(START1 ) -
   CONNECT(RING(MQRING) LABEL('ADCD-CA') CERTAUTH)
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Define the queue manager’s certificate.   This uses an Elliptic curve with key size of 256.

 /*RACDCERT ID(START1) DELETE(LABEL('CSQ9CERT')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('CSQ9CERT')  - 
             O('ADCD') - 
             OU('TEST')) - 
   SIZE(256) - 
   NISTECC - 
   SIGNWITH (CERTAUTH LABEL('ADCD-CA')) - 
ALTNAME(IP(10.1.1.9)) - NOTAFTER( DATE(2021-12-29))- WITHLABEL('CSQ9CERT')
RACDCERT id(START1) ALTER(LABEL('CSQ9CERT'))TRUST RACDCERT ID(START1) CONNECT(RING(MQRING) - LABEL('CSQ9CERT') USAGE(PERSONAL)) RACDCERT LISTRING(MQRING) ID(START1) RACDCERT LIST(LABEL('CSQ9CERT' )) ID(START1) SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

It need ALTNAME(IP(10.1.1.2)) ( or similar) because some browsers check this name, with the IP address of the server.

 

Configure the queue manager

%CSQ9 ALTER QMGR SSLKEYR(START1/MQRING) CERTLABL(CSQ9CERT)

Note if you use mixed case keyring you need to put the value in quotes.

Export the CA certificate from z/OS and sent to the client machine

RACDCERT CERTAUTH EXPORT(LABEL('ADCD-CA'))- 
DSN('IBMUSER.CERT.ADCDCA.PEM') -
FORMAT(CERTB64) -
PASSWORD('password')

The data set  IBMUSER.CERT.ADCDCA.PEM  contained text, and the first line is —–BEGIN CERTIFICATE—– .   Send this file to the client machine, for example using FTP.  I sent it as zos.adcdca.pem.

Import this to the keystore

runmqakm -cert -add -file zos.adcdca.pem -type cms -stashed -db zzclient.kdb -label zosca

Upload  the certificates from Linux to z/OS.

On Linux, my CA certificiate was in a *.pem file where the first line was —–BEGIN CERTIFICATE—–.  Send this to z/OS.  I used FTP.

Import the CA into the keyring.

The command adds an existing certificate CARSA1024 for userid START1.  The CONNECT USAGE(CERTAUT) defines this as a CA certificate (without the need to have the certificate belong to CERTAUTH userid).

  /*RACDCERT DELETE  ( LABEL('CARSA1024')) ID(START1) 
 SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 
                                                                   
 RACDCERT ADD('IBMUSER.CARS1024.PEM')  - 
   ID(START1)  WITHLABEL('CARSA1024') 
 RACDCERT ID(START1) CONNECT(RING(MQRING    ) - 
                             USAGE(CERTAUTH) - 
                             ID(START1)  - 
                             LABEL('CARSA1024') 
 racdcert listring(MQRING ) id(start1) 
 SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 

Refresh the queue manager

%CSQ9 refresh security type(SSL)

I’m thinking of using MQ MinimumRSAKeySize. What do I need to plan for?

If you enable this option in the qm.ini or mqclient.ini, it can have problems which are hard to diagnose.

This option restricts

  1. the key size of certificates with a type of RSA
  2. the key size of any CA certificates  with a type of RSA, used to sign a certificate (whether the certificate is RSA or not).

For example with MinimumRSAKeySize=2048 you cannot use a certificate or CA generated with openssl genpkey -out $name.key.pem -algorithm RSA -pkeyopt rsa_keygen_bits:1024.

On the client machine in /var/mqm/errors/*01* I got AMQ9633E: Bad SSL certificate for channel ….

On z/OS I got CSQX620E … CSQXRESP System SSL error, channel … connection … function ‘gsk_secure_socket_init’ RC=541.  Code 451 means “bad certificate” was received from the remote end.

How do I check?

There is no easy way of displaying the details of all of the certificates in a key store.

Midrange.

You can use the following command to list the labels all of the certificates

runmqakm -cert -list all -db zzclient.kdb -type cms -stashed -v

Then use the following command to display the details of each label in turn (zosca in the example)

runmqakm -cert -details -db zzclient.kdb -type cms -stashed -label zosca

This displays information like the example below for a CA certificate of type RSA and key size 1024.

Label : zosca
Key Size : 1024
Version : X509 V3
Serial : 00
Issuer : "CN=z/OSCertification Authority,OU=TEST,O=ADCD"
Subject : "CN=z/OSCertification Authority,OU=TEST,O=ADCD"
Not Before : 7 July 2020 00:00:00 GMT+01:00
Not After : 7 July 2021 23:59:59 GMT+01:00
Public Key
    ...
Public Key Type : RSA (1.2.840.113549.1.1.1)
  

or for an Elliptic certificate with key size 256.

Label : ca256
Key Size : 256
Version : X509 V3
Serial : ...
Issuer : CN=SSCA256,OU=CA,O=SSS,C=GB
Subject : CN=SSCA256,OU=CA,O=SSS,C=GB
Not Before : 7 February 2021 11:24:56 GMT
Not After : 7 February 2024 11:24:56 GMT
Public Key
    ... 
Public Key Type :  EC_ecPublicKey 

You can only check the certificates that are in your key store, not certificates that are sent as part of the handshake. 

z/OS

The listring command displays the contents of the ring (owner and label).

RACDCERT LISTRING(MQRING) ID(START1)

The list command displays the details of a certificate.

RACDCERT certauth LIST(label(‘ADCD-CA’))

displays information like, for the RSA certificate with a small key size,

 Label: ADCD-CA                                                       
 Certificate ID: 2QiJmZmDhZmjgcHEw8Rgw8FA                             
 Status: TRUST                                                        
 Start Date: 2020/07/06 23:00:00                                      
 End Date:   2021/07/07 22:59:59                                      
 Serial Number: ...                                                
 Issuer's Name:                                                      
      >CN=z/OSCertification Authority.OU=TEST.O=ADCD<                 
 Subject's Name:                                                      
      >CN=z/OSCertification Authority.OU=TEST.O=ADCD<                 
 Signing Algorithm: sha1RSA                                           
 Key Usage: CERTSIGN                                                  
 Key Type: RSA                                                        
 Key Size: 1024                                                       

Certificates signed by this CA would not work if MinimumRSAKeySize=2048 was specified.

Are there any requirements on the certificates used by MQ?

Having set up web servers using digital certificates I know some components are sensitive to the content of a certificate.  For example

  1. Some web browsers want the IP address in the  Alternate Name to match the IP address of the server.
  2. Some clients check that the server’s certificate has been signed with “server auth” – saying it can be used as a server’s certificate.

The short answer is no, there are no requirements. The longer answer is maybe.

For the chinit

I used the following to define certificate for z/OS.

RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('CSQ9CERT')  - 
             O('ADCD') - 
             OU('TEST')) - 
  SIZE(2048) - 
  RSA     - 
  SIGNWITH (CERTAUTH LABEL('ADCD-CA')) - 
NOTAFTER( DATE(2021-12-29))- WITHLABEL('CSQ9CERT') RACDCERT id(START1) ALTER(LABEL('CSQ9CERT'))TRUST RACDCERT ID(START1) CONNECT(RING(MQRING) - LABEL('CSQ9CERT') USAGE(PERSONAL))
RACDCERT LISTRING(MQRING) ID(START1) RACDCERT LIST(LABEL('CSQ9CERT' )) ID(START1) SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Things to be careful about

  1. Check the NOTAFTER date
  2. It needs to be TRUST
  3. The CONNECT of the certificate to the keyring needs USAGE(PERSONAL).  It it does not have it you get CSQX645E +cpf CSQXRESP Certificate … missing for channel …
  4. If the certificate is RSA, or the CA certificate is RSA, then check the keysize of the certificate and the CA.  On MQ 9.2 the size needs to be larger than the MinimumRSAKeySize any QM.INI or mqclient.ini file SSL stanza, in any queue manager or client.

For the MQ Web server

Any keyusage from no keysusage to KEYUSAGE( DATAENCRYPT, DOCSIGN, HANDSHAKE) works. Just do not use KEYUSAGE(CERTSIGN) .

With CERTSIGN Chrome gives NET::ERR_CERT_INVALID, and the TLS trace gives BAD_CERTIFICATE or UNKNOWN_CERTIFICATE. See here.

Colins updates to MQ messages

As I was trying to get TLS to work on midrange, I had many MQ error messages. Sometimes the messages were a bit vague “you’ve had a problem. Resolve it and restart the channel”.

Below is the list of messages I’ve added comments to. I’ve done it as a blog post as well-known search engines are not finding the pages.

Mid range

z/OS

Client

Getting MQ clients to TLS 1.3

King Midas was a greedy king and was granted a wish by Dionysus, the god of wine.   The king wished that every thing he touched turned into gold. His wish was granted, and every thing he touched turned to gold.  As a result he died of starvation.     There is also a saying that you cannot make a silk purse out of a pig’s ear.  I feel I have been gifted with the ability that anything I touch, breaks; as they say, making a pig’s ear out of a silk purse.

I thought changing my queue manager from using TLS 1.2 to TLS 1.3 would be easy.  This blog post describes my journey.  It was easy – I just had one problem which was easy to solve once I had spent a couple of hours work to identify the problem.

A short history of cipher specs in MQ.

In TLS 1.2 a cipher has a name like TLS _ KeyexchangeAlgorithm _  Keytype _ WITH _ BulkEncryption _ HashFunction.   The values have to be consistent at both ends, and the Keytype must match the certificate used during the hand shake. Names like  ECDHE_ECDSA_AES_128_CBC_SHA256 are the same as  of TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, with the TLS_ chopped off and shortened.  See here.

During the TLS 1.2 handshake the client sends a list of cipher suites it supports.  The server has a list of cipher suites it supports.  The lists are merged and any cipher specs in both lists are used in the subsequent processing.   The lists can have 1 or more elements in them.

With the server, it is a bit more subtle than this. The client sends its list of cipher suites to the server.  The server has a list of cipher suites it will accepts.    In MQ 9.2 the qm.ini has

  •  AllowTLSV13 whether to add the TLS 1.3 cipher specs to the standard list.   It defaults to yes.
  • AllowedCipherSpecs which allows you to specify which cipher specs can be used in the handshake.  It has a default list.

The first cipher spec which is in both the client and server lists is passed to the next stage of processing; checking with the svrconn channel’s SSLCIPHR parameter.

In MQ 9.1 and before, the channel lists had only one element in them.  This meant  the cipher spec TLS_A_B_WITH_C_D had to be specified at each end of the channel.  If you wanted to change the cipher spec you had to change the svrconn channel definition, and all of the client’s definitions to use the new cipher spec – and then restart all of the channels.  This was difficult to do, and hard to roll back.

In 9.1.1 the ANY_TLS12 list was implemented.   This is a list of cipher specs, for example TLS_A_B_WITH_C_D, TLS_A_B_WITH_E_F, TLS_X_Y_WITH_C_D, TLS_X_Y_WITH_E_F.

During the handshake the client passed its cipher spec to the server.   If the cipher spec was in this list the handshake would work.   This meant client1 could use TLS_A_B_WITH_C_D and client2 could use a different cipher spec TLS_X_Y_WITH_E_F.
Client1 can easily change to use a stronger cipher spec TLS_X_Y_WITH_E_F.  It just has to change its definition and restart.

This means you can do a phased change of clients using a channel.

You can specify the order of the cipher specification in the AllowedCipherSpecs list.    Usually the order is strong to weak.

I recommend using the ONLY_TLSxx_OR_ABOVE so any supported cipher specs can be used.

Client Server Supported
ANY_TLS12 ANY_TLS12 Yes TLS 1.2
ANY_TLS12 ANY_TLS12_OR_ABOVE Yes TLS 1.2
ANY_TLS12 ANY_TLS13 No
ANY_TLS12 ANY_TLS13_OR_ABOVE No
ANY_TLS12_OR_ABOVE ANY_TLS12 No.  It will try default to TLS 1.3 and fail
ANY_TLS12_OR_ABOVE ANY_TLS12_OR_ABOVE Yes TLS 1.2 or 1.3, Default will be 1.3
ANY_TLS12_OR_ABOVE ANY_TLS13 Yes TLS 1.3 – but not TLS 1.4 when it comes out
ANY_TLS12_OR_ABOVE ANY_TLS13_OR_ABOVE Yes TLS 1.3

How to migrate clients to newer levels of TLS.

Now that MQ supports list of cipher specs the migration should be pretty easy.

Take the scenario where two different customers,  needs to connect to my queue manager.  Before 9.1.x it would have been very difficult to coordinate changes between the customers, who may be in different time zone.

The steps are

  1. Change the svrconn channel to the highest available list, for example
    1. With MQ 9.1.1 you can use ANY_TLS12
    2. With MQ 9.1.4 you can use ANY_TLS_12_OR_HIGHER
    3. With MQ 9.2 you can specify ANY_TLS13_OR_HIGHER (but do not use this till you have finished upgrading to TLS 1.3).
  2. Do this for all queue managers the client connects to.
  3. For the first client, change the client definition to use the server supported level. If the client code is at 9.1.4 or higher then should can use ANY_TLS_12_OR_HIGHER (best), or give a specific TLS 1.x certificate .  See here for the list. The client cipher spec value must not be greater than the svrconn’s SSLCIPH value on all queue managers.
  4. Restart your client.  If there are problems you can revert to the previous value and restart the client.
  5. Use DIS CHS(…) ALL on the server.  It will list information like SSLCIPH(TLS_CHACHA20_POLY1305_SHA256).   This is a TLS 1.3 cipher spec.  Find any cipher specs which are not TLS 1.3 and fix then.
  6. Once you had done this for all clients using the channel, and for all queue managers using this channel, you should then be able to change the server definition to be ANY_TLS_13_OR_HIGHER and it should continue to work.
  7. You do not have to change the client to ANY_TLS13_OR_HIGHER, but you can as part of annual upgrade.  It prevents people creating a server with a TLS lower than 1.3, and having clients connect to it.

What happened to me…

I set ANY_TLS_12_OR_HIGHER at the client, and ANY_TLS_12 at the server – so the wrong way round (but I did not know this at the time).

During the handshake the client agreed with the queue manager to use TLS_CHACHA20_POLY1305_SHA256,  a TLS 1.3 cipher spec.   This was then checked by the channel code which was ANY_TLS12.   The “agreed” cipher spec was not supported so gave me the client AMQ9641E message.

EXPLANATION: The remote end of channel ‘…’ on host …’ has indicated a CipherSpec error ‘SSLCIPH(TLS_CHACHA20_POLY1305_SHA256 ) -> SSLCIPH(… )’. The channel did not start.

and the server message

AMQ9631E: The CipherSpec negotiated during the SSL handshake does not match the required CipherSpec for channel ‘…’.

EXPLANATION:
There is a mismatch between the CipherSpecs on the local and remote ends of channel ‘ECRSA1024’. The channel will not run until this mismatch is resolved. The CipherSpec required in the local channel definition is ‘ANY_TLS12’. The name of the CipherSpec negotiated during the SSL handshake is ‘TLS_CHACHA20_POLY1305_SHA256’.

Action to resolve it

  1. If I changed the svrconn channel to ANY_TLS12_OR_HIGHER it worked, and the session used a TLS 1.3 cipher spec.
  2. I could have changed the client to have ANY_TLS12 but I could potentially have a lot of clients to change.  Changing the svrconn was much simpler.

What are these message classes in MQ 9.2 on z/OS?

I noticed that the MQ 9.2 documentation for z/OS refers to different message classes.  This is nothing very exciting. From the JCL it says

This sample shows 8 other page sets, three for each major class of message, and one for each other class:

  • 00 – object definitions
  • 01 – system-related messages
  • 02-04 – important long-lived messages
  • 05-07 – short-lived messages
  • 08 – miscellaneous messages

Basically it means provide multiple page sets for the major classes of messages. This gives more capacity and can spread the I/O work load.

Setting up TLS on MQ Midrange

The weather is still cold up in Orkney (north of Scotland), and as I had been working on documenting TLS connections to web servers and how to get it working, I thought I would look at migrating my “enterprise” ( two queue managers and a couple of clients) to use TLS 1.3.  It took me a couple of days to set up a TLS 1.2 client to queue manager.  I spent a long time with my head in my hands thinking “why is this not working!”.
There is lots of documentation, from IBM and others, it was hard to find the best set of instructions.  I would classify the information I found as “how to dip your toe into the water” rather than “how to swim”.  At times I felt like this was a puzzle where the instructions say “There are false clues to misdirect you, and some of the clues have spelling mistakes – just to make it more interesting”. 

As well as this post I’ve created a web page of updates and suggestions to the IBM messages, with hints on where to look, and which traps to avoid.

As I was editing this document I found an excellent document on setting up MQ Clients and TLS with a java program.  The blog post below does not start from scratch, it takes an existing queue manager, and existing certificates and gets a C sample program to work.

Initial observations on key and trust stores.

Having been deep into the TLS specification and flows I know that there are two logical stores:

  1. The key store is where you store the private key used by this application.  The store typically has just one private key.  If someone has access to this store they can encrypt decrypted data using the key.
  2. The trust store. This stores the certificates needed to validate any certificates sent to it.  This typically has the Certificate Authorities, and any self signed keys (which you will used only in a sandbox environment).  You can have one trust store for all of your production applications – and other trust stores for your test environment.  Sharing a trust store is good practice, as this means you update a certificate, and every user gets the update.  If you have many trust stores you have to update them all when there is an update for a certificate.

MQ uses one store which is the trust store plus any private keys.  The disadvantage of this you have more work to keep the stores up to date with changes to certificates.

You could have a group of production queue managers sharing the same store.  You could give them each queue manager their own private key – or they could all use the same private key.  In the early days of MQ, each queue manager had a unique  certificate label (alias).  In the keystore for queue manager QMA, the label was  ibmwebspheremqqma.  You can now specify the label(alias) using the CERTLABL queue manager attribute, and on a channel.

In my work with the Liberty Web Server, I used key stores with key store types of .jks (deprecated)  and .p12 (for pkcs12).  On Midrange,  MQ uses a .CMS format used by IBM GSKIT.  On z/OS you use RACF or other z security products.   Java clients can use a variety of stores: .jks, .pem,  .p12; the list is what Java supports. Other clients, using the the GSKIT interface can only use .CMS stores.

You can import key stores into a .CMS store, so it is not a big effort to create a .CMS store – but it is an extra step to keep it up to date.

Before you start planning

(Not to be confused with doing the planning after you have started going in the wrong direction).

You need to think about what certificates and cipher suites names you can use.  See here.

Setting up TLS

I followed the IBM documentation but this was not very helpful.  For example it says “Create a key database file in the location that is specified in the queue manager’s Key repository attribute” but does not tell you how to do it.

I wanted to reuse the certificates I already had, so telling me how to create new certificate was not very useful.

The 30 second hints before you start, to avoid some of the holes in the road that I hit.

  1. Get clients connecting to your queue manager before trying to convert to TLS.
  2. Explicitly specify all  MQ environment variables,  or explicitly unset them.  For example some variables override mqclient.ini parameters, and some do not.  I spent hours debugging a problem because I had an environment variable set globally for another application.
  3. Check you do not have MQSERVER environment variable specified, or ServerConnectionParms in the mqclient.ini.
  4. Specify CERTLABL on each channel, to allow different cipher specs and certifcates to be used on different channels.
  5. Create a client (group) specific CCDT either with JSON or mqscutil for this group of clients.  Using a queue manager generated CCDT  may not give you the channel you wanted.
  6. The server’s certificate must be compatible with the cipher spec and both ends of the channel.    For example all RSA or all Elliptic curve.  This is easy to get wrong.
    1. The cipher spec of ANY_TLS12 etc available in V9.1.x make this much easier, as this uses a list of certificates.
    2. If you can use this at either end then do so.
  7. Each end needs read access to the key store files.
  8. You can create new certificates just for MQ, or reuse existing certificates.
    1. Java support can use a variety of key stores.
    2. Other programs need a .CMS format key store.
  9. Always check after making a change, as runmqsc upper cases values unless they are quoted.
  10. Remember to use refresh security type(ssl) after changing any of the queue manager’s SSL parameters, or changed the keystore.
  11. If you think it has worked, use DIS CHS(…) on the channel and make sure has picked the correct channel.  By default it finds the first channel in the CCDT for the queue manager.  For me this was not configured for TLS – so it appeared worked !!!

My journey.

What tools can help me set up my .cms keystore?

  1. GUI you can use strmqikm (STaRtMQ Interactive KeyManagement).
  2. Command line runmqckm (RUNMQCommandlineKeyManagement).
  3. For strong cipher keys, (eg Elliptic Curve public keys ), and Federal Information Processing Standards (FIPS) use the command line program runmqakm.  If you always use it instead of runmqckm, you will get muscle memory.  It provides a superset of functions over runmqckm.

The command line commands give the options available with the commands.  So you do not need the online documentation.  For example

runmqakm -keydb shows the options available for the -keydb option.

CTGSK3000W An action must be specified for this object.-Command usage-
Object Action Description
------ ------ -----------
-keydb
-changepw Change the password for a key database
-convert Convert the format of a key database
-create Create a key database
-delete Delete a key database
-expiry Display password expiry [Deprecated]
-list Currently supported types of key database.
-stashpw Stash the password of a key database into a file

Create the keystore for the server. 

I used a bash script to make it easier to change a parameter and rerun the commands.

The script

  1. deletes the old keystore
  2. creates the keystore
  3. adds in the Certificate Authority and self signed certificates
  4. imports the certificate and private key used in the handshake
  5. lists the contents of the keystore
  6. sets the authorities to the keystore.

db=”zzserver.kdb “
runmqakm -keydb -delete $db -pw password
# create the cms keystore
runmqakm -keydb -create -db $db -type cms -pw password -stash
# add the CA certificate
runmqakm -cert -add -file /home/colinpaice/ssl/ssl2/ca256.pem -type cms -stashed -db $db -label ca256
# and the self signed  called ss in the ss.p12 keystore
runmqakm -cert -import -file ~/ssl/ssl2/ss.p12 -type pkcs12 -pw password -target $db -target_stashed -label ss -new_label SS_SERVER
# add the certificate and private key into the store
#  it has a label ecc in in a pkcs12 file in ~/ssl/ssl2/ecec.p12
runmqakm -cert -import -file ~/ssl/ssl2/ecec.p12 -type pkcs12 -pw password -target $db -target_stashed -label ecec -new_label ECEC_SERVER
runmqakm -cert -list all -db $db -type cms -stashed
# give mqm access to the files
chown :mqm zzserver.*
chmod g+r zzserver.*

This created 4 files zzserver.crl zzserver.kdb zzserver.rdb zzserver.sth.

Check mqm can access it.  Run as mqm id

sudo -u mqm runmqakm -cert -list all -db zzserver.kdb -type cms -stashed

Define the keystore to the queue manager

Use the mqsc commands

alter qmgr SSLKEYR(‘/home/colinpaice/mq/zzserver’) CERTLABL(‘ECEC_SERVER’)
refresh security type(SSL)

Note the quotes around the values, without them the text is converted to upper case.

Define the channels for the TLS connection

I had an existing client connection QMACLIENT, and wanted to create a new channel QMAQCLIENTTLS.

I used the mqsc commands

  • define chl(QMAQCLIENTTLS) chltype(CLNTCONN) like(QMACLIENT)
  • define chl(QMAQCLIENTTLS) chltype(SVRCONN) like(QMACLIENT)
  • alt chl(QMAQCLIENTTLS) chltype(SVRCONN) SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
  • alt chl(QMAQCLIENTTLS) chltype(CLNTCONN) SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)

In theory I could use the system generated CCDT table which contains client information, but this did not work in practice.  See below.

I copied the AMQCLCHL.TAB to my working directory.

cp /var/mqm/qmgrs/QMA/@ipcc/AMQCLCHL.TAB .

Edit the mqclient.ini

I changed the mqclient.ini file to include

CHANNELS:
ServerConnectionParms=QMAQCLIENTTLS/TCP/127.0.0.1(1414)
ChannelDefinitionDirectory=/home/colinpaice/mq
ChannelDefinitionFile=AMQCLCHL.TAB
SSL:
CertificateLabel=ECEC_P
SSLKeyRepository=/home/colinpaice/mq/zzclient

My first run

I used a a bash script which ended up as

# display time - useful for looking at trace 
date
# remove the old client trace files
sudo rm /var/mqm/trace/AMQ*

export MQCLNTCF=/home/colinpaice/mq/mqclient.ini
export MQSSLKEYR=/home/colinpaice/mq/zzclient
export MQCHLLIB=/home/colinpaice/mq
export MQCHLTAB=AMQCLCHL.TAB

strmqtrc -e
/opt/mqm/samp/bin/amqsgetc CP0000 QMA
endmqtrc -e

This gave me

MQCONNX ended with reason code 2393

Problems I experienced

Trying to use ServerConnectionParms

I got 2393 SSL Initialization error, and less /var/mqm/errors/*01* had

AMQ9641E: Remote CipherSpec error for channel ‘QMAQCLIENTTLS’ to host
‘localhost (127.0.0.1)(1414)’.

EXPLANATION:
The remote end of channel ‘QMAQCLIENTTLS’ on host ‘localhost (127.0.0.1)(1414)’ has indicated a CipherSpec error ‘SSLCIPH(‘ ‘) -> SSLCIPH(????)’. The channel did not start.

Note the blank cipher spec. My mqclient.ini had ServerConnectionParms=QMAQCLIENTTLS/TCP/127.0.0.1(1414) which cannot be used for TLS channels.  See ServerConnectionParms. It overrides the ChannelDefinitionFile

I commented out this statement, so the CCDT would be used.

Wrong channel used from the CCDT.

The program seemed to work successfully, until I used DIS CHS(QMA*) when I saw the wrong, a non TLS channel, was being used.

When the CCDT is used, the entries are searched for the first entry which matches the queue manager name.

I found this by taking a trace, see Debugging MQ client connection problems:channel name.

The first channel matching was QMACLIENT which is non TLS.   I had to create a CCDT just for the  TLS channel.

You can have a json format file, see here for the attribute names or a binary ccdt from runmqsc.

Creating a ccdt file using runmqsc

You can use the runmqsc -n command to display or manipulate a binary ccdt file.

I created a file using the same parameters as the queue manager version.

The file it uses depends on the environment variables  for example

export MQCHLLIB=/home/colinpaice/mq
export MQCHLTAB=COLIN2.TAB
runmqsc -n

Set the environment variable in the run shell, and remove the statement from the mqclient.ini .

The wrong keystore was used

I had both an environment variable, and an entry in the mqclient.ini  (SSL: SSLKeyRepository=/home/colinpaice/mq/zzcolin)  for the keystore.

I debugged this using  Debugging MQ client connection problems:keystore.

The environment variable had precedence over the mqclient.ini.

I had mistyped the certificate label for the client

The server’s /var/mqm/qmgrs/QMA/errors/*01* file had

  • AMQ9638E: SSL communications error for channel ‘????’.
  • AMQ9999E: Channel ‘????’ to host ‘127.0.0.1’ ended abnormally.

The clients /var/mqm/errors/*01* had

I had inconsistent Cipher specs.

As part of the handshake there is logic like..

  • Client sends “I want to negotiate in  the following languages…”
  • The Server looks in its list of languages and makes sure there is at least one in common.
  • The server asks its certificate do you speak any of the following languages.

If any of the above fail you get the AMQ9616E: The CipherSpec … proposed by remote host … is not enabled error message.

The client can send up

  • One cipher spec for example specify  SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
  • A list of cipher specs SSLCIPH(ANY_TS12)

The server may have

  • One cipher specified SSLCIPH(ECDHE_ECDSA_AES_128_CBC_SHA256)
  • A list of cipher specs SSLCIPHR(ANY_TS12)

The server’s certificate could be, for example,

  • Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (384 bit)

or

  • Subject Public Key Info: Public Key Algorithm: rsaEncryption  RSA Public-Key: (2048 bit)

The bold text must match the agreed cipher spec.

Display connection information

Once the connection has been established you can use the DIS CHS(…) all command on the server using runmqsc.
This has information like

SECPROT(TLSV12)
SSLCERTI(CN=SSCA256,OU=CA,O=SSS,C=GB)
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(SERIALNUMBER=01:79,CN=rsaca256,O=cpwebuser,C=GB,UNSTRUCTURED...)
SSLRKEYS(0) STATUS(RUNNING)

This shows that the cipher spec used is TLS_RSA_WITH_AES_128_CBC_SHA256 and the DN from the client was had a DN with CN=rsaca256… .

I could not find the equivalent command on the client end.

You can get this from the client trace see debugging mq client connection problems:server cert

Which Cipher Spec and certificate type should I use for the MQ Server TLS 1.2?

Having got a TLS client to MQ working, I know much more about this topic than before I started.    Below is  some information I wish I had known before starting my project.   The discussion applies mainly to TLS 1.2, as there are significant changes and simplification in TLS 1.3.

Planning

Enterprise wide

  1. Do you have any standards you need to follow for TLS definitions, for example what levels of certificate are allowed, what cipher specs are allowed, what cipher specs are disallowed.
  2. Organisation often have multiple queue managers for availability reasons.  If a client can talk to a variety of queue managers the definitions need to be consistent.  Do you give each queue manager a unique certificate with a similar DN, or does each queue manager use the same certificate?  If you give each queue manager their own DN, you may have to do additional work for SSLPEER and CHLAUTH definitions.  If you create a new queue manager then existing clients should not have to change their definitions.
  3. You may have clients with different certificate formats, and DN formats, for example from different enterprises accessing your queue manager.  Your server needs to be able to support these.   You may have no control of the DN’s of clients outside of your domain.
  4. You can have the client authenticating with the server, where the server checks details from the DN in the clients certificate.
  5. You can have the client authenticate the server.    The server sends a certificate to the client, for the client to validate.
    1. You can specify a certificate on the svrconn channel definition at the server, and the DN from this certificate will be used.
    2. If there is no certificate on the channel definition, the queue manager’s certificate is used to provide the DN.
  6. You need to ensure that the CA certificates (and any self signed certificates) are stored in the server’s and/or client’s keystore.

Queue manager specific

  1. Each channel can have a different certificate.    You need to decide if you want the certificate type of RSA or EC.   Before MQ 9.1.1 it was  difficult to change it once it had been established. It does not matter how the certificate is signed (EC or RSA), or what message digest (eg -md SHA256) is used. You can restrict what RSA key sizes you accept for a certificate or CA.  See MinimumRSAKeySize.  It is easier to change your certifcate if you can use the ANY_TLS... cipher specs lists in MQ 9.1.1 and later.
  2. The cipher spec used by each channel must support the server’s certificate type.  Different channels can have different symmetrical encryption and hash algorithm values in the cipher suite.   For example one channel could have TLS_RSA_WITH_AES_256_GCM_SHA384 and another channel have TLS_RSA_WITH_AES_256_CBC_SHA256 both support a certificate type of RSA, but have different symmetrical encryption and hash algorithm values.
  3. Before MQ 9.1 each you could only specify one cipher spec value for a channel.  It had to be the same at each end, and so all clients for a channel had to have the same cipher spec.
  4. With 9.1.1 the server and the clients can support ANY_TLSxx which is a list of available cipher specs.    This means different clients can now have have different cipher specs for the same channel.

Which certificate is used on the server?

A certificate can be used for two purposes

  1. Specifying what key exchange algorithm to use
  2. For authentication

Certificates in an MQ server are used in two places

  1. For inbound channels the CERTLABL(…) certificate is used as part of the handshake to identify how the key exchange will be done. If a channel does not have a CERTLABL then it will use the queue manager defined value.   The certificate could be generated with certificate type RSA or Elliptic Curve (ECDSA).  See ALTER QMGR here.
  2. What is sent down to the client for authentication.  The certificate is checked and the DN extracted. The certificate type is not used.  The documentation says
    1. Channel attribute CERTLABL:Certificate label for this channel to use.  The label identifies which personal certificate in the key repository is sent to the remote peer. If this attribute is blank, the certificate is determined by the queue manager CERTLABL parameter.
    2. Note that inbound channels only send the configured certificate if the IBM® MQ version of the remote peer fully supports (requests) certificate label configuration, and the channel is using a TLS CipherSpec. See Interoperability of Elliptic Curve and RSA CipherSpecs for further information.

Setting up the handshake.

Both ends of a TLS connection need to agree the technique that will be used to agree secret keys.   The value_WITH of the cipher spec names give information about what they support.  For example  

  • TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384  supports ECDSA keys
  • TLS_RSA_WITH_AES_256_GCM_SHA384 supports RSA keys. 

You generate a certificate with a type of RSA or EC.  You can display the certificate type using

runmqakm -cert -details -db zzserver.kdb -type cms -stashed -label RSA_SERVER

This gives information like

  • Public Key Type : EC_ecPublicKey  Parameters: namedCurve: EC_NamedCurve_secp256r1
  • Public Key Type : RSA

The client sends up a list of one or more cipher specs it can support.   The server has a list of one or more cipher specs it supports.   The common subset is used to check the server’s certificate. If the certificate is not compatible with the subset then the request will fail with AMQ9616E: The CipherSpec … proposed by remote host …. is not enabled.

Client Server Certificate Valid?
TLS….RSA_WITH… TLS…RSA_WITH… RSA Yes
TLS….RSA_WITH… TLS…EC_WITH… RSA No
TLS….RSA_WITH… TLS…EC_WITH… EC No
TLS….RSA_WITH…,
TLS…EC_WITH…
TLS…RSA_WITH…

 

 

RSA Yes
TLS….RSA_WITH…,
TLS…EC…
TLS….RSA_WITH…,
TLS…EC_WTH…
RSA Yes
TLS….RSA_WITH..,
TLS…EC_WITH…
TLS….RSA_WTH…,
TLS…EC_WITh…
EC Yes

Once the key exchange parameters are agreed, the other parameters have to match. For example with

    • TLS_RSA_WITH_AES_256_GCM_SHA384
    • TLS_RSA_WITH_AES_256_CBC_SHA256

The symmetric data encryption techniques GCM/CBC do not match, nor do the hashing algorithms SHA256 and SHA384.   This causes message AMQ9631E.

You can specify only one value in the SSLCIPH.   This can be one cipher spec, or the name of a list, for example ANY_TLS12.  You can restrict what is used by using the AllowedCipherSpecs  option.

The documentation, for example Migrating existing security configurations to use the ANY_TLS12_OR_HIGHER CipherSpec is a little simplistic where it says change the value of the CipherSpec for the channel definition at each end, and then restart the channels for the change to take effect. It can be challenging updating and restarting 1000 channels cross multiple organisations; it is better to do a rolling upgrade.

Before MQ V9.1.x you had to specify the same cipher spec at both ends of a channel.  In MQ 9.1.x you can specify ANY_TLS12 at the server to accept any of the cipher specs in a list.