Are there any requirements on the certificates used by MQ?

Having set up web servers using digital certificates I know some components are sensitive to the content of a certificate.  For example

  1. Some web browsers want the IP address in the  Alternate Name to match the IP address of the server.
  2. Some clients check that the server’s certificate has been signed with “server auth” – saying it can be used as a server’s certificate.

The short answer is no, there are no requirements. The longer answer is maybe.

For the chinit

I used the following to define certificate for z/OS.

RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('CSQ9CERT')  - 
             O('ADCD') - 
             OU('TEST')) - 
  SIZE(2048) - 
  RSA     - 
  SIGNWITH (CERTAUTH LABEL('ADCD-CA')) - 
NOTAFTER( DATE(2021-12-29))- WITHLABEL('CSQ9CERT') RACDCERT id(START1) ALTER(LABEL('CSQ9CERT'))TRUST RACDCERT ID(START1) CONNECT(RING(MQRING) - LABEL('CSQ9CERT') USAGE(PERSONAL))
RACDCERT LISTRING(MQRING) ID(START1) RACDCERT LIST(LABEL('CSQ9CERT' )) ID(START1) SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Things to be careful about

  1. Check the NOTAFTER date
  2. It needs to be TRUST
  3. The CONNECT of the certificate to the keyring needs USAGE(PERSONAL).  It it does not have it you get CSQX645E +cpf CSQXRESP Certificate … missing for channel …
  4. If the certificate is RSA, or the CA certificate is RSA, then check the keysize of the certificate and the CA.  On MQ 9.2 the size needs to be larger than the MinimumRSAKeySize any QM.INI or mqclient.ini file SSL stanza, in any queue manager or client.

For the MQ Web server

Any keyusage from no keysusage to KEYUSAGE( DATAENCRYPT, DOCSIGN, HANDSHAKE) works. Just do not use KEYUSAGE(CERTSIGN) .

With CERTSIGN Chrome gives NET::ERR_CERT_INVALID, and the TLS trace gives BAD_CERTIFICATE or UNKNOWN_CERTIFICATE. See here.

Colins updates to MQ messages

As I was trying to get TLS to work on midrange, I had many MQ error messages. Sometimes the messages were a bit vague “you’ve had a problem. Resolve it and restart the channel”.

Below is the list of messages I’ve added comments to. I’ve done it as a blog post as well-known search engines are not finding the pages.

Mid range

z/OS

Client

Getting MQ clients to TLS 1.3

King Midas was a greedy king and was granted a wish by Dionysus, the god of wine.   The king wished that every thing he touched turned into gold. His wish was granted, and every thing he touched turned to gold.  As a result he died of starvation.     There is also a saying that you cannot make a silk purse out of a pig’s ear.  I feel I have been gifted with the ability that anything I touch, breaks; as they say, making a pig’s ear out of a silk purse.

I thought changing my queue manager from using TLS 1.2 to TLS 1.3 would be easy.  This blog post describes my journey.  It was easy – I just had one problem which was easy to solve once I had spent a couple of hours work to identify the problem.

A short history of cipher specs in MQ.

In TLS 1.2 a cipher has a name like TLS _ KeyexchangeAlgorithm _  Keytype _ WITH _ BulkEncryption _ HashFunction.   The values have to be consistent at both ends, and the Keytype must match the certificate used during the hand shake. Names like  ECDHE_ECDSA_AES_128_CBC_SHA256 are the same as  of TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, with the TLS_ chopped off and shortened.  See here.

During the TLS 1.2 handshake the client sends a list of cipher suites it supports.  The server has a list of cipher suites it supports.  The lists are merged and any cipher specs in both lists are used in the subsequent processing.   The lists can have 1 or more elements in them.

With the server, it is a bit more subtle than this. The client sends its list of cipher suites to the server.  The server has a list of cipher suites it will accepts.    In MQ 9.2 the qm.ini has

  •  AllowTLSV13 whether to add the TLS 1.3 cipher specs to the standard list.   It defaults to yes.
  • AllowedCipherSpecs which allows you to specify which cipher specs can be used in the handshake.  It has a default list.

The first cipher spec which is in both the client and server lists is passed to the next stage of processing; checking with the svrconn channel’s SSLCIPHR parameter.

In MQ 9.1 and before, the channel lists had only one element in them.  This meant  the cipher spec TLS_A_B_WITH_C_D had to be specified at each end of the channel.  If you wanted to change the cipher spec you had to change the svrconn channel definition, and all of the client’s definitions to use the new cipher spec – and then restart all of the channels.  This was difficult to do, and hard to roll back.

In 9.1.1 the ANY_TLS12 list was implemented.   This is a list of cipher specs, for example TLS_A_B_WITH_C_D, TLS_A_B_WITH_E_F, TLS_X_Y_WITH_C_D, TLS_X_Y_WITH_E_F.

During the handshake the client passed its cipher spec to the server.   If the cipher spec was in this list the handshake would work.   This meant client1 could use TLS_A_B_WITH_C_D and client2 could use a different cipher spec TLS_X_Y_WITH_E_F.
Client1 can easily change to use a stronger cipher spec TLS_X_Y_WITH_E_F.  It just has to change its definition and restart.

This means you can do a phased change of clients using a channel.

You can specify the order of the cipher specification in the AllowedCipherSpecs list.    Usually the order is strong to weak.

I recommend using the ONLY_TLSxx_OR_ABOVE so any supported cipher specs can be used.

Client Server Supported
ANY_TLS12 ANY_TLS12 Yes TLS 1.2
ANY_TLS12 ANY_TLS12_OR_ABOVE Yes TLS 1.2
ANY_TLS12 ANY_TLS13 No
ANY_TLS12 ANY_TLS13_OR_ABOVE No
ANY_TLS12_OR_ABOVE ANY_TLS12 No.  It will try default to TLS 1.3 and fail
ANY_TLS12_OR_ABOVE ANY_TLS12_OR_ABOVE Yes TLS 1.2 or 1.3, Default will be 1.3
ANY_TLS12_OR_ABOVE ANY_TLS13 Yes TLS 1.3 – but not TLS 1.4 when it comes out
ANY_TLS12_OR_ABOVE ANY_TLS13_OR_ABOVE Yes TLS 1.3

How to migrate clients to newer levels of TLS.

Now that MQ supports list of cipher specs the migration should be pretty easy.

Take the scenario where two different customers,  needs to connect to my queue manager.  Before 9.1.x it would have been very difficult to coordinate changes between the customers, who may be in different time zone.

The steps are

  1. Change the svrconn channel to the highest available list, for example
    1. With MQ 9.1.1 you can use ANY_TLS12
    2. With MQ 9.1.4 you can use ANY_TLS_12_OR_HIGHER
    3. With MQ 9.2 you can specify ANY_TLS13_OR_HIGHER (but do not use this till you have finished upgrading to TLS 1.3).
  2. Do this for all queue managers the client connects to.
  3. For the first client, change the client definition to use the server supported level. If the client code is at 9.1.4 or higher then should can use ANY_TLS_12_OR_HIGHER (best), or give a specific TLS 1.x certificate .  See here for the list. The client cipher spec value must not be greater than the svrconn’s SSLCIPH value on all queue managers.
  4. Restart your client.  If there are problems you can revert to the previous value and restart the client.
  5. Use DIS CHS(…) ALL on the server.  It will list information like SSLCIPH(TLS_CHACHA20_POLY1305_SHA256).   This is a TLS 1.3 cipher spec.  Find any cipher specs which are not TLS 1.3 and fix then.
  6. Once you had done this for all clients using the channel, and for all queue managers using this channel, you should then be able to change the server definition to be ANY_TLS_13_OR_HIGHER and it should continue to work.
  7. You do not have to change the client to ANY_TLS13_OR_HIGHER, but you can as part of annual upgrade.  It prevents people creating a server with a TLS lower than 1.3, and having clients connect to it.

What happened to me…

I set ANY_TLS_12_OR_HIGHER at the client, and ANY_TLS_12 at the server – so the wrong way round (but I did not know this at the time).

During the handshake the client agreed with the queue manager to use TLS_CHACHA20_POLY1305_SHA256,  a TLS 1.3 cipher spec.   This was then checked by the channel code which was ANY_TLS12.   The “agreed” cipher spec was not supported so gave me the client AMQ9641E message.

EXPLANATION: The remote end of channel ‘…’ on host …’ has indicated a CipherSpec error ‘SSLCIPH(TLS_CHACHA20_POLY1305_SHA256 ) -> SSLCIPH(… )’. The channel did not start.

and the server message

AMQ9631E: The CipherSpec negotiated during the SSL handshake does not match the required CipherSpec for channel ‘…’.

EXPLANATION:
There is a mismatch between the CipherSpecs on the local and remote ends of channel ‘ECRSA1024’. The channel will not run until this mismatch is resolved. The CipherSpec required in the local channel definition is ‘ANY_TLS12’. The name of the CipherSpec negotiated during the SSL handshake is ‘TLS_CHACHA20_POLY1305_SHA256’.

Action to resolve it

  1. If I changed the svrconn channel to ANY_TLS12_OR_HIGHER it worked, and the session used a TLS 1.3 cipher spec.
  2. I could have changed the client to have ANY_TLS12 but I could potentially have a lot of clients to change.  Changing the svrconn was much simpler.

What are these message classes in MQ 9.2 on z/OS?

I noticed that the MQ 9.2 documentation for z/OS refers to different message classes.  This is nothing very exciting. From the JCL it says

This sample shows 8 other page sets, three for each major class of message, and one for each other class:

  • 00 – object definitions
  • 01 – system-related messages
  • 02-04 – important long-lived messages
  • 05-07 – short-lived messages
  • 08 – miscellaneous messages

Basically it means provide multiple page sets for the major classes of messages. This gives more capacity and can spread the I/O work load.

Setting up TLS on MQ Midrange

The weather is still cold up in Orkney (north of Scotland), and as I had been working on documenting TLS connections to web servers and how to get it working, I thought I would look at migrating my “enterprise” ( two queue managers and a couple of clients) to use TLS 1.3.  It took me a couple of days to set up a TLS 1.2 client to queue manager.  I spent a long time with my head in my hands thinking “why is this not working!”.
There is lots of documentation, from IBM and others, it was hard to find the best set of instructions.  I would classify the information I found as “how to dip your toe into the water” rather than “how to swim”.  At times I felt like this was a puzzle where the instructions say “There are false clues to misdirect you, and some of the clues have spelling mistakes – just to make it more interesting”. 

As well as this post I’ve created a web page of updates and suggestions to the IBM messages, with hints on where to look, and which traps to avoid.

As I was editing this document I found an excellent document on setting up MQ Clients and TLS with a java program.  The blog post below does not start from scratch, it takes an existing queue manager, and existing certificates and gets a C sample program to work.

Initial observations on key and trust stores.

Having been deep into the TLS specification and flows I know that there are two logical stores:

  1. The key store is where you store the private key used by this application.  The store typically has just one private key.  If someone has access to this store they can encrypt decrypted data using the key.
  2. The trust store. This stores the certificates needed to validate any certificates sent to it.  This typically has the Certificate Authorities, and any self signed keys (which you will used only in a sandbox environment).  You can have one trust store for all of your production applications – and other trust stores for your test environment.  Sharing a trust store is good practice, as this means you update a certificate, and every user gets the update.  If you have many trust stores you have to update them all when there is an update for a certificate.

MQ uses one store which is the trust store plus any private keys.  The disadvantage of this you have more work to keep the stores up to date with changes to certificates.

You could have a group of production queue managers sharing the same store.  You could give them each queue manager their own private key – or they could all use the same private key.  In the early days of MQ, each queue manager had a unique  certificate label (alias).  In the keystore for queue manager QMA, the label was  ibmwebspheremqqma.  You can now specify the label(alias) using the CERTLABL queue manager attribute, and on a channel.

In my work with the Liberty Web Server, I used key stores with key store types of .jks (deprecated)  and .p12 (for pkcs12).  On Midrange,  MQ uses a .CMS format used by IBM GSKIT.  On z/OS you use RACF or other z security products.   Java clients can use a variety of stores: .jks, .pem,  .p12; the list is what Java supports. Other clients, using the the GSKIT interface can only use .CMS stores.

You can import key stores into a .CMS store, so it is not a big effort to create a .CMS store – but it is an extra step to keep it up to date.

Before you start planning

(Not to be confused with doing the planning after you have started going in the wrong direction).

You need to think about what certificates and cipher suites names you can use.  See here.

Setting up TLS

I followed the IBM documentation but this was not very helpful.  For example it says “Create a key database file in the location that is specified in the queue manager’s Key repository attribute” but does not tell you how to do it.

I wanted to reuse the certificates I already had, so telling me how to create new certificate was not very useful.

The 30 second hints before you start, to avoid some of the holes in the road that I hit.

  1. Get clients connecting to your queue manager before trying to convert to TLS.
  2. Explicitly specify all  MQ environment variables,  or explicitly unset them.  For example some variables override mqclient.ini parameters, and some do not.  I spent hours debugging a problem because I had an environment variable set globally for another application.
  3. Check you do not have MQSERVER environment variable specified, or ServerConnectionParms in the mqclient.ini.
  4. Specify CERTLABL on each channel, to allow different cipher specs and certifcates to be used on different channels.
  5. Create a client (group) specific CCDT either with JSON or mqscutil for this group of clients.  Using a queue manager generated CCDT  may not give you the channel you wanted.
  6. The server’s certificate must be compatible with the cipher spec and both ends of the channel.    For example all RSA or all Elliptic curve.  This is easy to get wrong.
    1. The cipher spec of ANY_TLS12 etc available in V9.1.x make this much easier, as this uses a list of certificates.
    2. If you can use this at either end then do so.
  7. Each end needs read access to the key store files.
  8. You can create new certificates just for MQ, or reuse existing certificates.
    1. Java support can use a variety of key stores.
    2. Other programs need a .CMS format key store.
  9. Always check after making a change, as runmqsc upper cases values unless they are quoted.
  10. Remember to use refresh security type(ssl) after changing any of the queue manager’s SSL parameters, or changed the keystore.
  11. If you think it has worked, use DIS CHS(…) on the channel and make sure has picked the correct channel.  By default it finds the first channel in the CCDT for the queue manager.  For me this was not configured for TLS – so it appeared worked !!!

My journey.

What tools can help me set up my .cms keystore?

  1. GUI you can use strmqikm (STaRtMQ Interactive KeyManagement).
  2. Command line runmqckm (RUNMQCommandlineKeyManagement).
  3. For strong cipher keys, (eg Elliptic Curve public keys ), and Federal Information Processing Standards (FIPS) use the command line program runmqakm.  If you always use it instead of runmqckm, you will get muscle memory.  It provides a superset of functions over runmqckm.

The command line commands give the options available with the commands.  So you do not need the online documentation.  For example

runmqakm -keydb shows the options available for the -keydb option.

CTGSK3000W An action must be specified for this object.-Command usage-
Object Action Description
------ ------ -----------
-keydb
-changepw Change the password for a key database
-convert Convert the format of a key database
-create Create a key database
-delete Delete a key database
-expiry Display password expiry [Deprecated]
-list Currently supported types of key database.
-stashpw Stash the password of a key database into a file

Create the keystore for the server. 

I used a bash script to make it easier to change a parameter and rerun the commands.

The script

  1. deletes the old keystore
  2. creates the keystore
  3. adds in the Certificate Authority and self signed certificates
  4. imports the certificate and private key used in the handshake
  5. lists the contents of the keystore
  6. sets the authorities to the keystore.

db=”zzserver.kdb “
runmqakm -keydb -delete $db -pw password
# create the cms keystore
runmqakm -keydb -create -db $db -type cms -pw password -stash
# add the CA certificate
runmqakm -cert -add -file /home/colinpaice/ssl/ssl2/ca256.pem -type cms -stashed -db $db -label ca256
# and the self signed  called ss in the ss.p12 keystore
runmqakm -cert -import -file ~/ssl/ssl2/ss.p12 -type pkcs12 -pw password -target $db -target_stashed -label ss -new_label SS_SERVER
# add the certificate and private key into the store
#  it has a label ecc in in a pkcs12 file in ~/ssl/ssl2/ecec.p12
runmqakm -cert -import -file ~/ssl/ssl2/ecec.p12 -type pkcs12 -pw password -target $db -target_stashed -label ecec -new_label ECEC_SERVER
runmqakm -cert -list all -db $db -type cms -stashed
# give mqm access to the files
chown :mqm zzserver.*
chmod g+r zzserver.*

This created 4 files zzserver.crl zzserver.kdb zzserver.rdb zzserver.sth.

Check mqm can access it.  Run as mqm id

sudo -u mqm runmqakm -cert -list all -db zzserver.kdb -type cms -stashed

Define the keystore to the queue manager

Use the mqsc commands

alter qmgr SSLKEYR(‘/home/colinpaice/mq/zzserver’) CERTLABL(‘ECEC_SERVER’)
refresh security type(SSL)

Note the quotes around the values, without them the text is converted to upper case.

Define the channels for the TLS connection

I had an existing client connection QMACLIENT, and wanted to create a new channel QMAQCLIENTTLS.

I used the mqsc commands

  • define chl(QMAQCLIENTTLS) chltype(CLNTCONN) like(QMACLIENT)
  • define chl(QMAQCLIENTTLS) chltype(SVRCONN) like(QMACLIENT)
  • alt chl(QMAQCLIENTTLS) chltype(SVRCONN) SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
  • alt chl(QMAQCLIENTTLS) chltype(CLNTCONN) SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)

In theory I could use the system generated CCDT table which contains client information, but this did not work in practice.  See below.

I copied the AMQCLCHL.TAB to my working directory.

cp /var/mqm/qmgrs/QMA/@ipcc/AMQCLCHL.TAB .

Edit the mqclient.ini

I changed the mqclient.ini file to include

CHANNELS:
ServerConnectionParms=QMAQCLIENTTLS/TCP/127.0.0.1(1414)
ChannelDefinitionDirectory=/home/colinpaice/mq
ChannelDefinitionFile=AMQCLCHL.TAB
SSL:
CertificateLabel=ECEC_P
SSLKeyRepository=/home/colinpaice/mq/zzclient

My first run

I used a a bash script which ended up as

# display time - useful for looking at trace 
date
# remove the old client trace files
sudo rm /var/mqm/trace/AMQ*

export MQCLNTCF=/home/colinpaice/mq/mqclient.ini
export MQSSLKEYR=/home/colinpaice/mq/zzclient
export MQCHLLIB=/home/colinpaice/mq
export MQCHLTAB=AMQCLCHL.TAB

strmqtrc -e
/opt/mqm/samp/bin/amqsgetc CP0000 QMA
endmqtrc -e

This gave me

MQCONNX ended with reason code 2393

Problems I experienced

Trying to use ServerConnectionParms

I got 2393 SSL Initialization error, and less /var/mqm/errors/*01* had

AMQ9641E: Remote CipherSpec error for channel ‘QMAQCLIENTTLS’ to host
‘localhost (127.0.0.1)(1414)’.

EXPLANATION:
The remote end of channel ‘QMAQCLIENTTLS’ on host ‘localhost (127.0.0.1)(1414)’ has indicated a CipherSpec error ‘SSLCIPH(‘ ‘) -> SSLCIPH(????)’. The channel did not start.

Note the blank cipher spec. My mqclient.ini had ServerConnectionParms=QMAQCLIENTTLS/TCP/127.0.0.1(1414) which cannot be used for TLS channels.  See ServerConnectionParms. It overrides the ChannelDefinitionFile

I commented out this statement, so the CCDT would be used.

Wrong channel used from the CCDT.

The program seemed to work successfully, until I used DIS CHS(QMA*) when I saw the wrong, a non TLS channel, was being used.

When the CCDT is used, the entries are searched for the first entry which matches the queue manager name.

I found this by taking a trace, see Debugging MQ client connection problems:channel name.

The first channel matching was QMACLIENT which is non TLS.   I had to create a CCDT just for the  TLS channel.

You can have a json format file, see here for the attribute names or a binary ccdt from runmqsc.

Creating a ccdt file using runmqsc

You can use the runmqsc -n command to display or manipulate a binary ccdt file.

I created a file using the same parameters as the queue manager version.

The file it uses depends on the environment variables  for example

export MQCHLLIB=/home/colinpaice/mq
export MQCHLTAB=COLIN2.TAB
runmqsc -n

Set the environment variable in the run shell, and remove the statement from the mqclient.ini .

The wrong keystore was used

I had both an environment variable, and an entry in the mqclient.ini  (SSL: SSLKeyRepository=/home/colinpaice/mq/zzcolin)  for the keystore.

I debugged this using  Debugging MQ client connection problems:keystore.

The environment variable had precedence over the mqclient.ini.

I had mistyped the certificate label for the client

The server’s /var/mqm/qmgrs/QMA/errors/*01* file had

  • AMQ9638E: SSL communications error for channel ‘????’.
  • AMQ9999E: Channel ‘????’ to host ‘127.0.0.1’ ended abnormally.

The clients /var/mqm/errors/*01* had

I had inconsistent Cipher specs.

As part of the handshake there is logic like..

  • Client sends “I want to negotiate in  the following languages…”
  • The Server looks in its list of languages and makes sure there is at least one in common.
  • The server asks its certificate do you speak any of the following languages.

If any of the above fail you get the AMQ9616E: The CipherSpec … proposed by remote host … is not enabled error message.

The client can send up

  • One cipher spec for example specify  SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
  • A list of cipher specs SSLCIPH(ANY_TS12)

The server may have

  • One cipher specified SSLCIPH(ECDHE_ECDSA_AES_128_CBC_SHA256)
  • A list of cipher specs SSLCIPHR(ANY_TS12)

The server’s certificate could be, for example,

  • Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (384 bit)

or

  • Subject Public Key Info: Public Key Algorithm: rsaEncryption  RSA Public-Key: (2048 bit)

The bold text must match the agreed cipher spec.

Display connection information

Once the connection has been established you can use the DIS CHS(…) all command on the server using runmqsc.
This has information like

SECPROT(TLSV12)
SSLCERTI(CN=SSCA256,OU=CA,O=SSS,C=GB)
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(SERIALNUMBER=01:79,CN=rsaca256,O=cpwebuser,C=GB,UNSTRUCTURED...)
SSLRKEYS(0) STATUS(RUNNING)

This shows that the cipher spec used is TLS_RSA_WITH_AES_128_CBC_SHA256 and the DN from the client was had a DN with CN=rsaca256… .

I could not find the equivalent command on the client end.

You can get this from the client trace see debugging mq client connection problems:server cert

Which Cipher Spec and certificate type should I use for the MQ Server TLS 1.2?

Having got a TLS client to MQ working, I know much more about this topic than before I started.    Below is  some information I wish I had known before starting my project.   The discussion applies mainly to TLS 1.2, as there are significant changes and simplification in TLS 1.3.

Planning

Enterprise wide

  1. Do you have any standards you need to follow for TLS definitions, for example what levels of certificate are allowed, what cipher specs are allowed, what cipher specs are disallowed.
  2. Organisation often have multiple queue managers for availability reasons.  If a client can talk to a variety of queue managers the definitions need to be consistent.  Do you give each queue manager a unique certificate with a similar DN, or does each queue manager use the same certificate?  If you give each queue manager their own DN, you may have to do additional work for SSLPEER and CHLAUTH definitions.  If you create a new queue manager then existing clients should not have to change their definitions.
  3. You may have clients with different certificate formats, and DN formats, for example from different enterprises accessing your queue manager.  Your server needs to be able to support these.   You may have no control of the DN’s of clients outside of your domain.
  4. You can have the client authenticating with the server, where the server checks details from the DN in the clients certificate.
  5. You can have the client authenticate the server.    The server sends a certificate to the client, for the client to validate.
    1. You can specify a certificate on the svrconn channel definition at the server, and the DN from this certificate will be used.
    2. If there is no certificate on the channel definition, the queue manager’s certificate is used to provide the DN.
  6. You need to ensure that the CA certificates (and any self signed certificates) are stored in the server’s and/or client’s keystore.

Queue manager specific

  1. Each channel can have a different certificate.    You need to decide if you want the certificate type of RSA or EC.   Before MQ 9.1.1 it was  difficult to change it once it had been established. It does not matter how the certificate is signed (EC or RSA), or what message digest (eg -md SHA256) is used. You can restrict what RSA key sizes you accept for a certificate or CA.  See MinimumRSAKeySize.  It is easier to change your certifcate if you can use the ANY_TLS... cipher specs lists in MQ 9.1.1 and later.
  2. The cipher spec used by each channel must support the server’s certificate type.  Different channels can have different symmetrical encryption and hash algorithm values in the cipher suite.   For example one channel could have TLS_RSA_WITH_AES_256_GCM_SHA384 and another channel have TLS_RSA_WITH_AES_256_CBC_SHA256 both support a certificate type of RSA, but have different symmetrical encryption and hash algorithm values.
  3. Before MQ 9.1 each you could only specify one cipher spec value for a channel.  It had to be the same at each end, and so all clients for a channel had to have the same cipher spec.
  4. With 9.1.1 the server and the clients can support ANY_TLSxx which is a list of available cipher specs.    This means different clients can now have have different cipher specs for the same channel.

Which certificate is used on the server?

A certificate can be used for two purposes

  1. Specifying what key exchange algorithm to use
  2. For authentication

Certificates in an MQ server are used in two places

  1. For inbound channels the CERTLABL(…) certificate is used as part of the handshake to identify how the key exchange will be done. If a channel does not have a CERTLABL then it will use the queue manager defined value.   The certificate could be generated with certificate type RSA or Elliptic Curve (ECDSA).  See ALTER QMGR here.
  2. What is sent down to the client for authentication.  The certificate is checked and the DN extracted. The certificate type is not used.  The documentation says
    1. Channel attribute CERTLABL:Certificate label for this channel to use.  The label identifies which personal certificate in the key repository is sent to the remote peer. If this attribute is blank, the certificate is determined by the queue manager CERTLABL parameter.
    2. Note that inbound channels only send the configured certificate if the IBM® MQ version of the remote peer fully supports (requests) certificate label configuration, and the channel is using a TLS CipherSpec. See Interoperability of Elliptic Curve and RSA CipherSpecs for further information.

Setting up the handshake.

Both ends of a TLS connection need to agree the technique that will be used to agree secret keys.   The value_WITH of the cipher spec names give information about what they support.  For example  

  • TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384  supports ECDSA keys
  • TLS_RSA_WITH_AES_256_GCM_SHA384 supports RSA keys. 

You generate a certificate with a type of RSA or EC.  You can display the certificate type using

runmqakm -cert -details -db zzserver.kdb -type cms -stashed -label RSA_SERVER

This gives information like

  • Public Key Type : EC_ecPublicKey  Parameters: namedCurve: EC_NamedCurve_secp256r1
  • Public Key Type : RSA

The client sends up a list of one or more cipher specs it can support.   The server has a list of one or more cipher specs it supports.   The common subset is used to check the server’s certificate. If the certificate is not compatible with the subset then the request will fail with AMQ9616E: The CipherSpec … proposed by remote host …. is not enabled.

Client Server Certificate Valid?
TLS….RSA_WITH… TLS…RSA_WITH… RSA Yes
TLS….RSA_WITH… TLS…EC_WITH… RSA No
TLS….RSA_WITH… TLS…EC_WITH… EC No
TLS….RSA_WITH…,
TLS…EC_WITH…
TLS…RSA_WITH…

 

 

RSA Yes
TLS….RSA_WITH…,
TLS…EC…
TLS….RSA_WITH…,
TLS…EC_WTH…
RSA Yes
TLS….RSA_WITH..,
TLS…EC_WITH…
TLS….RSA_WTH…,
TLS…EC_WITh…
EC Yes

Once the key exchange parameters are agreed, the other parameters have to match. For example with

    • TLS_RSA_WITH_AES_256_GCM_SHA384
    • TLS_RSA_WITH_AES_256_CBC_SHA256

The symmetric data encryption techniques GCM/CBC do not match, nor do the hashing algorithms SHA256 and SHA384.   This causes message AMQ9631E.

You can specify only one value in the SSLCIPH.   This can be one cipher spec, or the name of a list, for example ANY_TLS12.  You can restrict what is used by using the AllowedCipherSpecs  option.

The documentation, for example Migrating existing security configurations to use the ANY_TLS12_OR_HIGHER CipherSpec is a little simplistic where it says change the value of the CipherSpec for the channel definition at each end, and then restart the channels for the change to take effect. It can be challenging updating and restarting 1000 channels cross multiple organisations; it is better to do a rolling upgrade.

Before MQ V9.1.x you had to specify the same cipher spec at both ends of a channel.  In MQ 9.1.x you can specify ANY_TLS12 at the server to accept any of the cipher specs in a list.

Debugging MQ client connection problems

I had lots of problems trying to get a client using TLS to connect to the queue manager, from set up errors, to unclear documentation.   It took me days to get my first channel set up.
IBM does not provide a program on midrange to format the GSKIT trace.  Someone suggested I use Wireshark  network packet analyser to monitor the traffic on the network.  I’ve given some examples of the handshake for TLS 12, TLS 13.

Below are some things you can do to check your set up is as you expect.

Review the error log

This is in /var/mqm/errors/AMQERR01.LOG
It does not provide all the information needed to identify the parameters used in the handshake, so you need to use the trace.

Turn on the MQ client trace

strmqtrc -e
start your program
endmqtrc -e

Format the trace

The client trace files are in /var/mqm/trace

  • cd /var/mqm/trace
  • dspmqtrc *.TRC

This will create several *.FMT files

Find the trace file for the connection

grep -F PeerName *.FMT

will list the file with the trace entries in it.  My file was  AMQ7232.0.FMT .

Check the channel name.

From the output of the grep -F PeerName *.FMT above, check the Channel Name: is what you expect? Check the remote IP address and port.  If you are using a CCDT it will take the first channel name which matches the queue manager.  This may not be the channel name you were expecting.    I had channels QMACLIENT and QMACLIENTTLS in the CCDT.   QMACLIENT was chosen instead of QMACLIENTTLS.

Check the mqclient.ini being used

grep -F mqclient.ini AMQ7232.0.FMT |grep -F FileName

Check the CCDT, (Channel Definition File) being used

grep -F ChannelDefinition AMQ7232.0.FMT

Adding stanza (ChannelDefinitionDirectory) length (19)
Adding stanza (ChannelDefinitionFile) length (10)
ChannelDefinitionDirectory = '/home/colinpaice/mq'
ChannelDefinitionFile = 'COLIN2.TAB'
Adding stanza (ChannelDefinitionFile) length (10)
Using ChannelDefinitionDirectory / MQCHLLIB value of /home/colinpaice/mq
Using ChannelDefinitionFile / MQCHLTAB value of RE.TAB
Adding stanza (ChannelDefinitionDirectory) length (19)
Adding stanza (ChannelDefinitionFile) length (10)
Using ChannelDefinitionDirectory / MQCHLLIB value of /home/colinpaice/mq
Using ChannelDefinitionFile / MQCHLTAB value of RE.TAB

The last entries show what was used.

Check the keystore being used

grep -Fi SSLKeyR *.FMT

Display the certlabl.

grep -Fi CertificateLabel *.FMT

It will display many records. They key ones are

  • MQCD CertificateLabel ‘rsaca256_client’
  • Saved CertificateLabel ‘rsaca256_client’

If these are missing certlabl has not been specified.

Display the channel definiton

Edit or browse the file and locate “CD “.

This will locate the MQCD (see the MQ documentation).

Interesting offsets in the CD are

  • 0x0000 Client channel name
  • 0x0060 Queue manager name
  • 0x00c0 IP address
  • 0x0698 Cipher Spec
  • 0x0780 TLSCertificate alias name in the client’s keystore.

Display Client Hello

You may have information about the TLS handshake.  This is called Client_hello, and Server_Hello.

Locate <client_hello> if found, it will have data like (some data removed).

client_version 
TLSV12
random ... 
session_id ... 
cipher_suites 
  tls_ri_scsv,tls_rsa_with_aes_128_cbc_sha256
compression_methods ...                              .
Extensions...
signature_algorithms 13
   rsa:sha512,rsa:sha384,rsa:sha256,rsa:sha224,...
server_name 
  qmaqclienttls.chl.mq.ibm.com
End of GSKit TLS Handshake Transcript

You need to understand the TLS handshake to fully understand this.

  • TLSV12 minimum level of TLS supported
  • cipher_suites these are what the client like
  • signature_algorithms  these are what the client will accept
  • server_name is “sni” information identifying the channel

What is my client connected to?

On a queue manager you an issue the DIS CHS(..)  all and get

SECPROT(TLSV12)
SSLCERTI(CN=SSCA256,OU=CA,O=SSS,C=GB)
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLPEER(SERIALNUMBER=01:79,CN=rsaca256,O=cpwebuser,C=GB,…)

I could not find a command to display the same information for the clients perspective.

You can look in the trace file for SSLCERTI for example

SSLCERTI(‘CN=SS,O=SSS,C=GB’)
SSLPEER(‘SERIALNUMBER=73:CB:2B:…,CN=SS,O=SSS,C=GB’)

This gives the server’s DN and Certificate issuer, and peer=subject.  As the peer has the same DN as the SSLCERTI this shows it is a self signed certificate.

Upgrading to MQ 9.2.1 on Ubuntu using apt was easy, but got messy.

Installing MQ 9.2.1 over an earlier release of MQ was almost easy!

I started here and downloaded the developer edition into ~/MQ921.

    • I had to cd ~/MQ921/MQSERVER to run the license.sh.
    • I could not edit a file in /etc/apt/sources.list.d because I was not authorised, using sudo gedit failed with:
      • Invalid MIT-MAGIC-COOKIE-1 keyUnable to init server: Could not connect: Connection refused.   Gtk-WARNING **: 10:34:55.390: cannot open display: :0
      • I could have used xhost +local:  to fix this problem.
    • I used the filename  ibmmq921-install.list instead of the suggested  ibmmq-install.list because I already had an ibmmq.list file in the directory.
    • I created  ibmmq921-install.list in my home directory and then used sudo mv ibmmq921-install.list /etc/apt/sources.list.d/ .
    • Initially I forgot to do sudo apt update – so the install did not find the updated file.
    • When I ran sudo apt update it displayed 10 packages can be upgraded. Run ‘apt list –upgradable’ to see them.
    • I ran sudo apt update and it displayed 
ibmmq-client/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-gskit/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-java/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-jre/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-man/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-runtime/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-samples/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-sdk/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-server/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-web/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]

    • I ran sudo apt upgrade it ran for a short while and ended successfully.
    • The command dspmqver showed Version 9.2.1.0 – so success.
    • Starting the queue manager strmqm QMA gave IBM MQ queue manager ‘QMA’ started using V9.2.1.0

It got messy

After I had successfully installed MQ, the next time I ran sudo apt update it gave me lots of messages like 

Get:1 file:/home/colinpaice/MQ921/MQServer ./ InRelease
Ign:1 file:/home/colinpaice/MQ921/MQServer ./ InRelease
Get:2 file:/home/colinpaice/MQ921/MQServer ./ Release
Ign:2 file:/home/colinpaice/MQ921/MQServer ./ Release

By adding in the file, means apt checks it every time.  I could not see how any updates were going to appear in the files, so I removed the file from /etc/apt/sources.list.d/

I also deleted the files I downloaded (890 MB) and the unzipped directory (927MB).  (I needed to use sudo rm -r MQ921 because the files have write-protection.

Not very tidy.

 

 

Control unit cache section (SMF 42.2) statistics appear strange and unloved. Should I use them? No!

Why should I not use them?

The SMF 42 subtype 2 control unit cache records seem to have strange data in them – and seem to add little value.  The RMF 74 subtype 5 – Cache Subsystem Device Activity include most of the data in the 74.2,  and contains additional data.  The 74.5 record layout has good documentation.  The documentation for 42.2 does not.

Why do I think the record is unloved?

Some of the record formats are non standard. 

  1. Many SMF records use a time since midnight in hundred’s of a second.  SMF 42.2 does it in seconds (so I had to change my formatting routine – sigh).
  2. SMF dates tend to be packed decimal.   SMF 42.2 a date is documented as “Year, in the form 0cyy, where c is 0 for 19xx and 1 for 20xx, and yy”.  It is really just the year – 1900.  The value x79 is decimal 121 so 1900 + 121 is 2021 – this year.
  3. It has two data sections “Statistics gathered from last update period” and Statistics gathered from current update period.  The current is 90 seconds after the “last” ( should that be previous interval?).  I dont know what data these sections contain. Is it for the whole duration between the SMF records, or just the last 90 seconds of each interval.
  4. It has a field “Fast write bypasses per minute (an integer).” for each of the intervals.  I don’t know what it means especially when you consider the interval between the two sections is 90 seconds.
  5. The SMF records were produced every 30 minutes.  I dont know how to get the data for the other 27 minutes.
  6. There are sections for each SMS managed volume.   The data basically says “Is Cache and Fast write” enabled for this volume.  Yes/Yes. So what?

As this has a subtype of 2 I expect this record was produced 30+ years ago when people did not really know what they wanted in the SMF records.  I expect Development quietly ignored this subtype, and created other subtypes with useful data.  This is why I think you should use other records instead of this 42.2 records.

 

Do storage controllers pining having anything to do with a Norwegian Fiords?

No – it means I need to wear my glasses!

I was looking at the disk storage controller statistics and saw references to pinned storage.  I naturally thought this was a reference to the Monty Python dead parrot sketch where the Norwegian Blue was “pining for the Fiords”.

Ive often wondered what Pinned storage was.  I found “Pinned data cannot be removed from the cache by the storage machine because of a hardware failure”.  I think that if the amount of pinned storage is greater than zero then phone someone.

There are other fields “Unavailable storage”. 

These are examples of providing data – but no information.   You need to know that if these fields are non zero, this is a problem and you need to do something about it – just displaying a number has little value.  I would have hoped that that documentation would have said, in big bold letters: If these numbers are not zero then take action!