Are there any requirements on the certificates used by MQ?

Having set up web servers using digital certificates I know some components are sensitive to the content of a certificate.  For example

  1. Some web browsers want the IP address in the  Alternate Name to match the IP address of the server.
  2. Some clients check that the server’s certificate has been signed with “server auth” – saying it can be used as a server’s certificate.

The short answer is no, there are no requirements. The longer answer is maybe.

For the chinit

I used the following to define certificate for z/OS.

RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('CSQ9CERT')  - 
             O('ADCD') - 
             OU('TEST')) - 
  SIZE(2048) - 
  RSA     - 
  SIGNWITH (CERTAUTH LABEL('ADCD-CA')) - 
NOTAFTER( DATE(2021-12-29))- WITHLABEL('CSQ9CERT') RACDCERT id(START1) ALTER(LABEL('CSQ9CERT'))TRUST RACDCERT ID(START1) CONNECT(RING(MQRING) - LABEL('CSQ9CERT') USAGE(PERSONAL))
RACDCERT LISTRING(MQRING) ID(START1) RACDCERT LIST(LABEL('CSQ9CERT' )) ID(START1) SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Things to be careful about

  1. Check the NOTAFTER date
  2. It needs to be TRUST
  3. The CONNECT of the certificate to the keyring needs USAGE(PERSONAL).  It it does not have it you get CSQX645E +cpf CSQXRESP Certificate … missing for channel …
  4. If the certificate is RSA, or the CA certificate is RSA, then check the keysize of the certificate and the CA.  On MQ 9.2 the size needs to be larger than the MinimumRSAKeySize any QM.INI or mqclient.ini file SSL stanza, in any queue manager or client.

For the MQ Web server

Any keyusage from no keysusage to KEYUSAGE( DATAENCRYPT, DOCSIGN, HANDSHAKE) works. Just do not use KEYUSAGE(CERTSIGN) .

With CERTSIGN Chrome gives NET::ERR_CERT_INVALID, and the TLS trace gives BAD_CERTIFICATE or UNKNOWN_CERTIFICATE. See here.

Getting MQ clients to TLS 1.3

King Midas was a greedy king and was granted a wish by Dionysus, the god of wine.   The king wished that every thing he touched turned into gold. His wish was granted, and every thing he touched turned to gold.  As a result he died of starvation.     There is also a saying that you cannot make a silk purse out of a pig’s ear.  I feel I have been gifted with the ability that anything I touch, breaks; as they say, making a pig’s ear out of a silk purse.

I thought changing my queue manager from using TLS 1.2 to TLS 1.3 would be easy.  This blog post describes my journey.  It was easy – I just had one problem which was easy to solve once I had spent a couple of hours work to identify the problem.

A short history of cipher specs in MQ.

In TLS 1.2 a cipher has a name like TLS _ KeyexchangeAlgorithm _  Keytype _ WITH _ BulkEncryption _ HashFunction.   The values have to be consistent at both ends, and the Keytype must match the certificate used during the hand shake. Names like  ECDHE_ECDSA_AES_128_CBC_SHA256 are the same as  of TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, with the TLS_ chopped off and shortened.  See here.

During the TLS 1.2 handshake the client sends a list of cipher suites it supports.  The server has a list of cipher suites it supports.  The lists are merged and any cipher specs in both lists are used in the subsequent processing.   The lists can have 1 or more elements in them.

With the server, it is a bit more subtle than this. The client sends its list of cipher suites to the server.  The server has a list of cipher suites it will accepts.    In MQ 9.2 the qm.ini has

  •  AllowTLSV13 whether to add the TLS 1.3 cipher specs to the standard list.   It defaults to yes.
  • AllowedCipherSpecs which allows you to specify which cipher specs can be used in the handshake.  It has a default list.

The first cipher spec which is in both the client and server lists is passed to the next stage of processing; checking with the svrconn channel’s SSLCIPHR parameter.

In MQ 9.1 and before, the channel lists had only one element in them.  This meant  the cipher spec TLS_A_B_WITH_C_D had to be specified at each end of the channel.  If you wanted to change the cipher spec you had to change the svrconn channel definition, and all of the client’s definitions to use the new cipher spec – and then restart all of the channels.  This was difficult to do, and hard to roll back.

In 9.1.1 the ANY_TLS12 list was implemented.   This is a list of cipher specs, for example TLS_A_B_WITH_C_D, TLS_A_B_WITH_E_F, TLS_X_Y_WITH_C_D, TLS_X_Y_WITH_E_F.

During the handshake the client passed its cipher spec to the server.   If the cipher spec was in this list the handshake would work.   This meant client1 could use TLS_A_B_WITH_C_D and client2 could use a different cipher spec TLS_X_Y_WITH_E_F.
Client1 can easily change to use a stronger cipher spec TLS_X_Y_WITH_E_F.  It just has to change its definition and restart.

This means you can do a phased change of clients using a channel.

You can specify the order of the cipher specification in the AllowedCipherSpecs list.    Usually the order is strong to weak.

I recommend using the ONLY_TLSxx_OR_ABOVE so any supported cipher specs can be used.

Client Server Supported
ANY_TLS12 ANY_TLS12 Yes TLS 1.2
ANY_TLS12 ANY_TLS12_OR_ABOVE Yes TLS 1.2
ANY_TLS12 ANY_TLS13 No
ANY_TLS12 ANY_TLS13_OR_ABOVE No
ANY_TLS12_OR_ABOVE ANY_TLS12 No.  It will try default to TLS 1.3 and fail
ANY_TLS12_OR_ABOVE ANY_TLS12_OR_ABOVE Yes TLS 1.2 or 1.3, Default will be 1.3
ANY_TLS12_OR_ABOVE ANY_TLS13 Yes TLS 1.3 – but not TLS 1.4 when it comes out
ANY_TLS12_OR_ABOVE ANY_TLS13_OR_ABOVE Yes TLS 1.3

How to migrate clients to newer levels of TLS.

Now that MQ supports list of cipher specs the migration should be pretty easy.

Take the scenario where two different customers,  needs to connect to my queue manager.  Before 9.1.x it would have been very difficult to coordinate changes between the customers, who may be in different time zone.

The steps are

  1. Change the svrconn channel to the highest available list, for example
    1. With MQ 9.1.1 you can use ANY_TLS12
    2. With MQ 9.1.4 you can use ANY_TLS_12_OR_HIGHER
    3. With MQ 9.2 you can specify ANY_TLS13_OR_HIGHER (but do not use this till you have finished upgrading to TLS 1.3).
  2. Do this for all queue managers the client connects to.
  3. For the first client, change the client definition to use the server supported level. If the client code is at 9.1.4 or higher then should can use ANY_TLS_12_OR_HIGHER (best), or give a specific TLS 1.x certificate .  See here for the list. The client cipher spec value must not be greater than the svrconn’s SSLCIPH value on all queue managers.
  4. Restart your client.  If there are problems you can revert to the previous value and restart the client.
  5. Use DIS CHS(…) ALL on the server.  It will list information like SSLCIPH(TLS_CHACHA20_POLY1305_SHA256).   This is a TLS 1.3 cipher spec.  Find any cipher specs which are not TLS 1.3 and fix then.
  6. Once you had done this for all clients using the channel, and for all queue managers using this channel, you should then be able to change the server definition to be ANY_TLS_13_OR_HIGHER and it should continue to work.
  7. You do not have to change the client to ANY_TLS13_OR_HIGHER, but you can as part of annual upgrade.  It prevents people creating a server with a TLS lower than 1.3, and having clients connect to it.

What happened to me…

I set ANY_TLS_12_OR_HIGHER at the client, and ANY_TLS_12 at the server – so the wrong way round (but I did not know this at the time).

During the handshake the client agreed with the queue manager to use TLS_CHACHA20_POLY1305_SHA256,  a TLS 1.3 cipher spec.   This was then checked by the channel code which was ANY_TLS12.   The “agreed” cipher spec was not supported so gave me the client AMQ9641E message.

EXPLANATION: The remote end of channel ‘…’ on host …’ has indicated a CipherSpec error ‘SSLCIPH(TLS_CHACHA20_POLY1305_SHA256 ) -> SSLCIPH(… )’. The channel did not start.

and the server message

AMQ9631E: The CipherSpec negotiated during the SSL handshake does not match the required CipherSpec for channel ‘…’.

EXPLANATION:
There is a mismatch between the CipherSpecs on the local and remote ends of channel ‘ECRSA1024’. The channel will not run until this mismatch is resolved. The CipherSpec required in the local channel definition is ‘ANY_TLS12’. The name of the CipherSpec negotiated during the SSL handshake is ‘TLS_CHACHA20_POLY1305_SHA256’.

Action to resolve it

  1. If I changed the svrconn channel to ANY_TLS12_OR_HIGHER it worked, and the session used a TLS 1.3 cipher spec.
  2. I could have changed the client to have ANY_TLS12 but I could potentially have a lot of clients to change.  Changing the svrconn was much simpler.

Setting up TLS on MQ Midrange

The weather is still cold up in Orkney (north of Scotland), and as I had been working on documenting TLS connections to web servers and how to get it working, I thought I would look at migrating my “enterprise” ( two queue managers and a couple of clients) to use TLS 1.3.  It took me a couple of days to set up a TLS 1.2 client to queue manager.  I spent a long time with my head in my hands thinking “why is this not working!”.
There is lots of documentation, from IBM and others, it was hard to find the best set of instructions.  I would classify the information I found as “how to dip your toe into the water” rather than “how to swim”.  At times I felt like this was a puzzle where the instructions say “There are false clues to misdirect you, and some of the clues have spelling mistakes – just to make it more interesting”. 

As well as this post I’ve created a web page of updates and suggestions to the IBM messages, with hints on where to look, and which traps to avoid.

As I was editing this document I found an excellent document on setting up MQ Clients and TLS with a java program.  The blog post below does not start from scratch, it takes an existing queue manager, and existing certificates and gets a C sample program to work.

Initial observations on key and trust stores.

Having been deep into the TLS specification and flows I know that there are two logical stores:

  1. The key store is where you store the private key used by this application.  The store typically has just one private key.  If someone has access to this store they can encrypt decrypted data using the key.
  2. The trust store. This stores the certificates needed to validate any certificates sent to it.  This typically has the Certificate Authorities, and any self signed keys (which you will used only in a sandbox environment).  You can have one trust store for all of your production applications – and other trust stores for your test environment.  Sharing a trust store is good practice, as this means you update a certificate, and every user gets the update.  If you have many trust stores you have to update them all when there is an update for a certificate.

MQ uses one store which is the trust store plus any private keys.  The disadvantage of this you have more work to keep the stores up to date with changes to certificates.

You could have a group of production queue managers sharing the same store.  You could give them each queue manager their own private key – or they could all use the same private key.  In the early days of MQ, each queue manager had a unique  certificate label (alias).  In the keystore for queue manager QMA, the label was  ibmwebspheremqqma.  You can now specify the label(alias) using the CERTLABL queue manager attribute, and on a channel.

In my work with the Liberty Web Server, I used key stores with key store types of .jks (deprecated)  and .p12 (for pkcs12).  On Midrange,  MQ uses a .CMS format used by IBM GSKIT.  On z/OS you use RACF or other z security products.   Java clients can use a variety of stores: .jks, .pem,  .p12; the list is what Java supports. Other clients, using the the GSKIT interface can only use .CMS stores.

You can import key stores into a .CMS store, so it is not a big effort to create a .CMS store – but it is an extra step to keep it up to date.

Before you start planning

(Not to be confused with doing the planning after you have started going in the wrong direction).

You need to think about what certificates and cipher suites names you can use.  See here.

Setting up TLS

I followed the IBM documentation but this was not very helpful.  For example it says “Create a key database file in the location that is specified in the queue manager’s Key repository attribute” but does not tell you how to do it.

I wanted to reuse the certificates I already had, so telling me how to create new certificate was not very useful.

The 30 second hints before you start, to avoid some of the holes in the road that I hit.

  1. Get clients connecting to your queue manager before trying to convert to TLS.
  2. Explicitly specify all  MQ environment variables,  or explicitly unset them.  For example some variables override mqclient.ini parameters, and some do not.  I spent hours debugging a problem because I had an environment variable set globally for another application.
  3. Check you do not have MQSERVER environment variable specified, or ServerConnectionParms in the mqclient.ini.
  4. Specify CERTLABL on each channel, to allow different cipher specs and certifcates to be used on different channels.
  5. Create a client (group) specific CCDT either with JSON or mqscutil for this group of clients.  Using a queue manager generated CCDT  may not give you the channel you wanted.
  6. The server’s certificate must be compatible with the cipher spec and both ends of the channel.    For example all RSA or all Elliptic curve.  This is easy to get wrong.
    1. The cipher spec of ANY_TLS12 etc available in V9.1.x make this much easier, as this uses a list of certificates.
    2. If you can use this at either end then do so.
  7. Each end needs read access to the key store files.
  8. You can create new certificates just for MQ, or reuse existing certificates.
    1. Java support can use a variety of key stores.
    2. Other programs need a .CMS format key store.
  9. Always check after making a change, as runmqsc upper cases values unless they are quoted.
  10. Remember to use refresh security type(ssl) after changing any of the queue manager’s SSL parameters, or changed the keystore.
  11. If you think it has worked, use DIS CHS(…) on the channel and make sure has picked the correct channel.  By default it finds the first channel in the CCDT for the queue manager.  For me this was not configured for TLS – so it appeared worked !!!

My journey.

What tools can help me set up my .cms keystore?

  1. GUI you can use strmqikm (STaRtMQ Interactive KeyManagement).
  2. Command line runmqckm (RUNMQCommandlineKeyManagement).
  3. For strong cipher keys, (eg Elliptic Curve public keys ), and Federal Information Processing Standards (FIPS) use the command line program runmqakm.  If you always use it instead of runmqckm, you will get muscle memory.  It provides a superset of functions over runmqckm.

The command line commands give the options available with the commands.  So you do not need the online documentation.  For example

runmqakm -keydb shows the options available for the -keydb option.

CTGSK3000W An action must be specified for this object.-Command usage-
Object Action Description
------ ------ -----------
-keydb
-changepw Change the password for a key database
-convert Convert the format of a key database
-create Create a key database
-delete Delete a key database
-expiry Display password expiry [Deprecated]
-list Currently supported types of key database.
-stashpw Stash the password of a key database into a file

Create the keystore for the server. 

I used a bash script to make it easier to change a parameter and rerun the commands.

The script

  1. deletes the old keystore
  2. creates the keystore
  3. adds in the Certificate Authority and self signed certificates
  4. imports the certificate and private key used in the handshake
  5. lists the contents of the keystore
  6. sets the authorities to the keystore.

db=”zzserver.kdb “
runmqakm -keydb -delete $db -pw password
# create the cms keystore
runmqakm -keydb -create -db $db -type cms -pw password -stash
# add the CA certificate
runmqakm -cert -add -file /home/colinpaice/ssl/ssl2/ca256.pem -type cms -stashed -db $db -label ca256
# and the self signed  called ss in the ss.p12 keystore
runmqakm -cert -import -file ~/ssl/ssl2/ss.p12 -type pkcs12 -pw password -target $db -target_stashed -label ss -new_label SS_SERVER
# add the certificate and private key into the store
#  it has a label ecc in in a pkcs12 file in ~/ssl/ssl2/ecec.p12
runmqakm -cert -import -file ~/ssl/ssl2/ecec.p12 -type pkcs12 -pw password -target $db -target_stashed -label ecec -new_label ECEC_SERVER
runmqakm -cert -list all -db $db -type cms -stashed
# give mqm access to the files
chown :mqm zzserver.*
chmod g+r zzserver.*

This created 4 files zzserver.crl zzserver.kdb zzserver.rdb zzserver.sth.

Check mqm can access it.  Run as mqm id

sudo -u mqm runmqakm -cert -list all -db zzserver.kdb -type cms -stashed

Define the keystore to the queue manager

Use the mqsc commands

alter qmgr SSLKEYR(‘/home/colinpaice/mq/zzserver’) CERTLABL(‘ECEC_SERVER’)
refresh security type(SSL)

Note the quotes around the values, without them the text is converted to upper case.

Define the channels for the TLS connection

I had an existing client connection QMACLIENT, and wanted to create a new channel QMAQCLIENTTLS.

I used the mqsc commands

  • define chl(QMAQCLIENTTLS) chltype(CLNTCONN) like(QMACLIENT)
  • define chl(QMAQCLIENTTLS) chltype(SVRCONN) like(QMACLIENT)
  • alt chl(QMAQCLIENTTLS) chltype(SVRCONN) SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
  • alt chl(QMAQCLIENTTLS) chltype(CLNTCONN) SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)

In theory I could use the system generated CCDT table which contains client information, but this did not work in practice.  See below.

I copied the AMQCLCHL.TAB to my working directory.

cp /var/mqm/qmgrs/QMA/@ipcc/AMQCLCHL.TAB .

Edit the mqclient.ini

I changed the mqclient.ini file to include

CHANNELS:
ServerConnectionParms=QMAQCLIENTTLS/TCP/127.0.0.1(1414)
ChannelDefinitionDirectory=/home/colinpaice/mq
ChannelDefinitionFile=AMQCLCHL.TAB
SSL:
CertificateLabel=ECEC_P
SSLKeyRepository=/home/colinpaice/mq/zzclient

My first run

I used a a bash script which ended up as

# display time - useful for looking at trace 
date
# remove the old client trace files
sudo rm /var/mqm/trace/AMQ*

export MQCLNTCF=/home/colinpaice/mq/mqclient.ini
export MQSSLKEYR=/home/colinpaice/mq/zzclient
export MQCHLLIB=/home/colinpaice/mq
export MQCHLTAB=AMQCLCHL.TAB

strmqtrc -e
/opt/mqm/samp/bin/amqsgetc CP0000 QMA
endmqtrc -e

This gave me

MQCONNX ended with reason code 2393

Problems I experienced

Trying to use ServerConnectionParms

I got 2393 SSL Initialization error, and less /var/mqm/errors/*01* had

AMQ9641E: Remote CipherSpec error for channel ‘QMAQCLIENTTLS’ to host
‘localhost (127.0.0.1)(1414)’.

EXPLANATION:
The remote end of channel ‘QMAQCLIENTTLS’ on host ‘localhost (127.0.0.1)(1414)’ has indicated a CipherSpec error ‘SSLCIPH(‘ ‘) -> SSLCIPH(????)’. The channel did not start.

Note the blank cipher spec. My mqclient.ini had ServerConnectionParms=QMAQCLIENTTLS/TCP/127.0.0.1(1414) which cannot be used for TLS channels.  See ServerConnectionParms. It overrides the ChannelDefinitionFile

I commented out this statement, so the CCDT would be used.

Wrong channel used from the CCDT.

The program seemed to work successfully, until I used DIS CHS(QMA*) when I saw the wrong, a non TLS channel, was being used.

When the CCDT is used, the entries are searched for the first entry which matches the queue manager name.

I found this by taking a trace, see Debugging MQ client connection problems:channel name.

The first channel matching was QMACLIENT which is non TLS.   I had to create a CCDT just for the  TLS channel.

You can have a json format file, see here for the attribute names or a binary ccdt from runmqsc.

Creating a ccdt file using runmqsc

You can use the runmqsc -n command to display or manipulate a binary ccdt file.

I created a file using the same parameters as the queue manager version.

The file it uses depends on the environment variables  for example

export MQCHLLIB=/home/colinpaice/mq
export MQCHLTAB=COLIN2.TAB
runmqsc -n

Set the environment variable in the run shell, and remove the statement from the mqclient.ini .

The wrong keystore was used

I had both an environment variable, and an entry in the mqclient.ini  (SSL: SSLKeyRepository=/home/colinpaice/mq/zzcolin)  for the keystore.

I debugged this using  Debugging MQ client connection problems:keystore.

The environment variable had precedence over the mqclient.ini.

I had mistyped the certificate label for the client

The server’s /var/mqm/qmgrs/QMA/errors/*01* file had

  • AMQ9638E: SSL communications error for channel ‘????’.
  • AMQ9999E: Channel ‘????’ to host ‘127.0.0.1’ ended abnormally.

The clients /var/mqm/errors/*01* had

I had inconsistent Cipher specs.

As part of the handshake there is logic like..

  • Client sends “I want to negotiate in  the following languages…”
  • The Server looks in its list of languages and makes sure there is at least one in common.
  • The server asks its certificate do you speak any of the following languages.

If any of the above fail you get the AMQ9616E: The CipherSpec … proposed by remote host … is not enabled error message.

The client can send up

  • One cipher spec for example specify  SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
  • A list of cipher specs SSLCIPH(ANY_TS12)

The server may have

  • One cipher specified SSLCIPH(ECDHE_ECDSA_AES_128_CBC_SHA256)
  • A list of cipher specs SSLCIPHR(ANY_TS12)

The server’s certificate could be, for example,

  • Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (384 bit)

or

  • Subject Public Key Info: Public Key Algorithm: rsaEncryption  RSA Public-Key: (2048 bit)

The bold text must match the agreed cipher spec.

Display connection information

Once the connection has been established you can use the DIS CHS(…) all command on the server using runmqsc.
This has information like

SECPROT(TLSV12)
SSLCERTI(CN=SSCA256,OU=CA,O=SSS,C=GB)
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(SERIALNUMBER=01:79,CN=rsaca256,O=cpwebuser,C=GB,UNSTRUCTURED...)
SSLRKEYS(0) STATUS(RUNNING)

This shows that the cipher spec used is TLS_RSA_WITH_AES_128_CBC_SHA256 and the DN from the client was had a DN with CN=rsaca256… .

I could not find the equivalent command on the client end.

You can get this from the client trace see debugging mq client connection problems:server cert

Which Cipher Spec and certificate type should I use for the MQ Server TLS 1.2?

Having got a TLS client to MQ working, I know much more about this topic than before I started.    Below is  some information I wish I had known before starting my project.   The discussion applies mainly to TLS 1.2, as there are significant changes and simplification in TLS 1.3.

Planning

Enterprise wide

  1. Do you have any standards you need to follow for TLS definitions, for example what levels of certificate are allowed, what cipher specs are allowed, what cipher specs are disallowed.
  2. Organisation often have multiple queue managers for availability reasons.  If a client can talk to a variety of queue managers the definitions need to be consistent.  Do you give each queue manager a unique certificate with a similar DN, or does each queue manager use the same certificate?  If you give each queue manager their own DN, you may have to do additional work for SSLPEER and CHLAUTH definitions.  If you create a new queue manager then existing clients should not have to change their definitions.
  3. You may have clients with different certificate formats, and DN formats, for example from different enterprises accessing your queue manager.  Your server needs to be able to support these.   You may have no control of the DN’s of clients outside of your domain.
  4. You can have the client authenticating with the server, where the server checks details from the DN in the clients certificate.
  5. You can have the client authenticate the server.    The server sends a certificate to the client, for the client to validate.
    1. You can specify a certificate on the svrconn channel definition at the server, and the DN from this certificate will be used.
    2. If there is no certificate on the channel definition, the queue manager’s certificate is used to provide the DN.
  6. You need to ensure that the CA certificates (and any self signed certificates) are stored in the server’s and/or client’s keystore.

Queue manager specific

  1. Each channel can have a different certificate.    You need to decide if you want the certificate type of RSA or EC.   Before MQ 9.1.1 it was  difficult to change it once it had been established. It does not matter how the certificate is signed (EC or RSA), or what message digest (eg -md SHA256) is used. You can restrict what RSA key sizes you accept for a certificate or CA.  See MinimumRSAKeySize.  It is easier to change your certifcate if you can use the ANY_TLS... cipher specs lists in MQ 9.1.1 and later.
  2. The cipher spec used by each channel must support the server’s certificate type.  Different channels can have different symmetrical encryption and hash algorithm values in the cipher suite.   For example one channel could have TLS_RSA_WITH_AES_256_GCM_SHA384 and another channel have TLS_RSA_WITH_AES_256_CBC_SHA256 both support a certificate type of RSA, but have different symmetrical encryption and hash algorithm values.
  3. Before MQ 9.1 each you could only specify one cipher spec value for a channel.  It had to be the same at each end, and so all clients for a channel had to have the same cipher spec.
  4. With 9.1.1 the server and the clients can support ANY_TLSxx which is a list of available cipher specs.    This means different clients can now have have different cipher specs for the same channel.

Which certificate is used on the server?

A certificate can be used for two purposes

  1. Specifying what key exchange algorithm to use
  2. For authentication

Certificates in an MQ server are used in two places

  1. For inbound channels the CERTLABL(…) certificate is used as part of the handshake to identify how the key exchange will be done. If a channel does not have a CERTLABL then it will use the queue manager defined value.   The certificate could be generated with certificate type RSA or Elliptic Curve (ECDSA).  See ALTER QMGR here.
  2. What is sent down to the client for authentication.  The certificate is checked and the DN extracted. The certificate type is not used.  The documentation says
    1. Channel attribute CERTLABL:Certificate label for this channel to use.  The label identifies which personal certificate in the key repository is sent to the remote peer. If this attribute is blank, the certificate is determined by the queue manager CERTLABL parameter.
    2. Note that inbound channels only send the configured certificate if the IBM® MQ version of the remote peer fully supports (requests) certificate label configuration, and the channel is using a TLS CipherSpec. See Interoperability of Elliptic Curve and RSA CipherSpecs for further information.

Setting up the handshake.

Both ends of a TLS connection need to agree the technique that will be used to agree secret keys.   The value_WITH of the cipher spec names give information about what they support.  For example  

  • TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384  supports ECDSA keys
  • TLS_RSA_WITH_AES_256_GCM_SHA384 supports RSA keys. 

You generate a certificate with a type of RSA or EC.  You can display the certificate type using

runmqakm -cert -details -db zzserver.kdb -type cms -stashed -label RSA_SERVER

This gives information like

  • Public Key Type : EC_ecPublicKey  Parameters: namedCurve: EC_NamedCurve_secp256r1
  • Public Key Type : RSA

The client sends up a list of one or more cipher specs it can support.   The server has a list of one or more cipher specs it supports.   The common subset is used to check the server’s certificate. If the certificate is not compatible with the subset then the request will fail with AMQ9616E: The CipherSpec … proposed by remote host …. is not enabled.

Client Server Certificate Valid?
TLS….RSA_WITH… TLS…RSA_WITH… RSA Yes
TLS….RSA_WITH… TLS…EC_WITH… RSA No
TLS….RSA_WITH… TLS…EC_WITH… EC No
TLS….RSA_WITH…,
TLS…EC_WITH…
TLS…RSA_WITH…

 

 

RSA Yes
TLS….RSA_WITH…,
TLS…EC…
TLS….RSA_WITH…,
TLS…EC_WTH…
RSA Yes
TLS….RSA_WITH..,
TLS…EC_WITH…
TLS….RSA_WTH…,
TLS…EC_WITh…
EC Yes

Once the key exchange parameters are agreed, the other parameters have to match. For example with

    • TLS_RSA_WITH_AES_256_GCM_SHA384
    • TLS_RSA_WITH_AES_256_CBC_SHA256

The symmetric data encryption techniques GCM/CBC do not match, nor do the hashing algorithms SHA256 and SHA384.   This causes message AMQ9631E.

You can specify only one value in the SSLCIPH.   This can be one cipher spec, or the name of a list, for example ANY_TLS12.  You can restrict what is used by using the AllowedCipherSpecs  option.

The documentation, for example Migrating existing security configurations to use the ANY_TLS12_OR_HIGHER CipherSpec is a little simplistic where it says change the value of the CipherSpec for the channel definition at each end, and then restart the channels for the change to take effect. It can be challenging updating and restarting 1000 channels cross multiple organisations; it is better to do a rolling upgrade.

Before MQ V9.1.x you had to specify the same cipher spec at both ends of a channel.  In MQ 9.1.x you can specify ANY_TLS12 at the server to accept any of the cipher specs in a list.

Debugging MQ client connection problems

I had lots of problems trying to get a client using TLS to connect to the queue manager, from set up errors, to unclear documentation.   It took me days to get my first channel set up.
IBM does not provide a program on midrange to format the GSKIT trace.  Someone suggested I use Wireshark  network packet analyser to monitor the traffic on the network.  I’ve given some examples of the handshake for TLS 12, TLS 13.

Below are some things you can do to check your set up is as you expect.

Review the error log

This is in /var/mqm/errors/AMQERR01.LOG
It does not provide all the information needed to identify the parameters used in the handshake, so you need to use the trace.

Turn on the MQ client trace

strmqtrc -e
start your program
endmqtrc -e

Format the trace

The client trace files are in /var/mqm/trace

  • cd /var/mqm/trace
  • dspmqtrc *.TRC

This will create several *.FMT files

Find the trace file for the connection

grep -F PeerName *.FMT

will list the file with the trace entries in it.  My file was  AMQ7232.0.FMT .

Check the channel name.

From the output of the grep -F PeerName *.FMT above, check the Channel Name: is what you expect? Check the remote IP address and port.  If you are using a CCDT it will take the first channel name which matches the queue manager.  This may not be the channel name you were expecting.    I had channels QMACLIENT and QMACLIENTTLS in the CCDT.   QMACLIENT was chosen instead of QMACLIENTTLS.

Check the mqclient.ini being used

grep -F mqclient.ini AMQ7232.0.FMT |grep -F FileName

Check the CCDT, (Channel Definition File) being used

grep -F ChannelDefinition AMQ7232.0.FMT

Adding stanza (ChannelDefinitionDirectory) length (19)
Adding stanza (ChannelDefinitionFile) length (10)
ChannelDefinitionDirectory = '/home/colinpaice/mq'
ChannelDefinitionFile = 'COLIN2.TAB'
Adding stanza (ChannelDefinitionFile) length (10)
Using ChannelDefinitionDirectory / MQCHLLIB value of /home/colinpaice/mq
Using ChannelDefinitionFile / MQCHLTAB value of RE.TAB
Adding stanza (ChannelDefinitionDirectory) length (19)
Adding stanza (ChannelDefinitionFile) length (10)
Using ChannelDefinitionDirectory / MQCHLLIB value of /home/colinpaice/mq
Using ChannelDefinitionFile / MQCHLTAB value of RE.TAB

The last entries show what was used.

Check the keystore being used

grep -Fi SSLKeyR *.FMT

Display the certlabl.

grep -Fi CertificateLabel *.FMT

It will display many records. They key ones are

  • MQCD CertificateLabel ‘rsaca256_client’
  • Saved CertificateLabel ‘rsaca256_client’

If these are missing certlabl has not been specified.

Display the channel definiton

Edit or browse the file and locate “CD “.

This will locate the MQCD (see the MQ documentation).

Interesting offsets in the CD are

  • 0x0000 Client channel name
  • 0x0060 Queue manager name
  • 0x00c0 IP address
  • 0x0698 Cipher Spec
  • 0x0780 TLSCertificate alias name in the client’s keystore.

Display Client Hello

You may have information about the TLS handshake.  This is called Client_hello, and Server_Hello.

Locate <client_hello> if found, it will have data like (some data removed).

client_version 
TLSV12
random ... 
session_id ... 
cipher_suites 
  tls_ri_scsv,tls_rsa_with_aes_128_cbc_sha256
compression_methods ...                              .
Extensions...
signature_algorithms 13
   rsa:sha512,rsa:sha384,rsa:sha256,rsa:sha224,...
server_name 
  qmaqclienttls.chl.mq.ibm.com
End of GSKit TLS Handshake Transcript

You need to understand the TLS handshake to fully understand this.

  • TLSV12 minimum level of TLS supported
  • cipher_suites these are what the client like
  • signature_algorithms  these are what the client will accept
  • server_name is “sni” information identifying the channel

What is my client connected to?

On a queue manager you an issue the DIS CHS(..)  all and get

SECPROT(TLSV12)
SSLCERTI(CN=SSCA256,OU=CA,O=SSS,C=GB)
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLPEER(SERIALNUMBER=01:79,CN=rsaca256,O=cpwebuser,C=GB,…)

I could not find a command to display the same information for the clients perspective.

You can look in the trace file for SSLCERTI for example

SSLCERTI(‘CN=SS,O=SSS,C=GB’)
SSLPEER(‘SERIALNUMBER=73:CB:2B:…,CN=SS,O=SSS,C=GB’)

This gives the server’s DN and Certificate issuer, and peer=subject.  As the peer has the same DN as the SSLCERTI this shows it is a self signed certificate.

If a .jks is not allowed, how do I create a .p12 for all of my certificates?

If you try to create a trust store with a format of Java Key Store you get a message saying you should not be using a propriety format. After some digging around, I have found out how to create a database with a format of pkcs12 (or .p12 as it is also known).

To make a PKCS12 keystore (including the private key)

You can use open ssl to create the two parts, (public and private), and get the public part signed.

You can then create a .p12entry using

password=”-passout file:password.file -passin file:password.file”
name=”colincert”
ca=”-CAfile capem -chain “

openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12  -name $name  $password $ca

To make a key store, importing the .p12s with the private keys,  use

rm mykey.p12
ks=” -importkeystore -destkeystore mytrust.p12″
dest=”-deststoretype pkcs12 -deststorepass password”
src=”-srcstoretype PKCS12 -srcstorepass password”

keytool $ks $src $dest -srckeystore ecec.p12 
keytool $ks $src $dest -srckeystore ss.p12
keytool -list -v -keystore mykey.p12 -storepass password

By default it uses the alias from the -name in the openssl command.

Using the -list command without the verbose option (-v) gives

keytool -list -keystore mykey.p12 -storepass password

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains … entries

ecec, 21-Jan-2021, PrivateKeyEntry,
Certificate fingerprint (SHA1): C3:3D:98:D1:45:DF:44:F0:13:FE:87:D9:3E:CC:22:33:A8:D6:24:91
mqweb, 21-Jan-2021, PrivateKeyEntry, 

So we can see this is a PKCS12 keystore with entries with a Private Key in them.

To make a trust store

A trust store only has the public parts, it does not have the private key, so creating an intermediate .p12 file does not work, you have to use the public .pem files.

rm mytrust.p12
ks=” -import -keystore mytrust.p12 -deststoretype pkcs12″
password=
-storepass:file password.file”

keytool $ks $password -file ecec.pem -alias ecec
keytool $ks  $password -file ss.pem -alias ss

keytool -list -v -keystore mytrust.p12 $password

This example uses the password stored in the file password.file, rather than specifying it in the command.

You have to supply the -alias, as it defaults to “mykey”.  If you add multiple entries with the default, you only get one entry in the keystore.

Using the -list command without the verbose option (-v) gives

keytool -list -keystore mytrust.p12 -storepass:file password.file
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

ecec, 21-Jan-2021, trustedCertEntry,
Certificate fingerprint (SHA1): C3:3D:98:D1:45:DF:44:F0:13:FE:87:D9:3E:CC:22:33:A8:D6:24:91
ss, 21-Jan-2021, trustedCertEntry,
Certificate fingerprint (SHA1): E9:D1:69:49:5B:D3:B4:3C:E8:44:5A:C1:2C:A2:D6:5D:FB:47:61:E7

So we can see this is a PKCS12 keystore, and has the expected entries which are Trusted Certificate Entries.

 

How do I used Linux to manage my corporate certificates?

Having used z/OS to be my corporate Certificate Authority I thought I would use Linux to be a corporate CA, and manage z/OS certificates.  For more information on Certificate Authorities, and signing on certificates see here.

Setting up your Corporate CA up on Linux

At the top of the CA certificate hierarchy is a self signed certificate.

Create the CA self signed certificate

openssl req -x509 -config openssl-ca.cnf -newkey rsa:4096 -days 3000 -nodes -subj “/C=GB/O=SSS/OU=CA/CN=SSCA8” -out cacert.pem -keyout cacert.key.pem -outform PEM  addext basicConstraints=”critical,CA:TRUE, pathlen:0″ -addext keyUsage=”keyCertSign, digitalSignature”

This creates a certificate with

  1. -x509 says make it self signed – so my enterprise master CA
  2. using 4096 rsa encryption
  3. a subject “/C=GB/O=SSS/OU=CA/CN=SSCA8”
  4. valid for 3000 days
  5. Not DES encryption ( -nodes) of the output files
  6. the who and public key are stored in cacert.pem
  7. the private key is stored in cacert.key.pem
  8. using format pem
  9. extra parameters CA:TRUE and keyusage…

Display it

openssl x509 -in cacert.pem -text -noout|less

Create a personal  certificate on Linux and sign it.

Create a personal certificate on Linux  and get it signed byt the CA created above.

I set up a shell script to do the work

name=”adcdd”
subj=’-subj “/C=GB/O=cpwebuser/CN=adcdd” ‘
#Passwords are stored in a file called password.file
passwords=”-passin file:password.file -passout file:password.file”

#clean out the old foils
rm $name.key.pem

rm $name.csr
rm $name.pem

CA=”cacert”

# generate a private certificate using Elliptic curve and type secp256r1
openssl ecparam -name secp256r1 -genkey -noout -out $name.key.pem

#create a certificate signing request (CSR)
openssl req -new -key $name.key.pem -out $name.csr -outform PEM $subj $passwords

#sign it – or send it off to be signed. Get the $name.pem back from the request.  
openssl ca -config openssl-ca-user.cnf -md sha384 -out $name.pem -cert cacert.pem -keyfile cacert.key.pem -policy signing_policy -extensions signing_mqweb -md sha256 -infiles $name.csr

 

#Get the  *.pem file back, if required,  and merge the files to form the .p12 file
openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $CA.pem -chain -name $name $passwords

I stored common information in configuration files, such as openssl-ca-user.cnf .  This had a section called signing_policy,  and signing_mqweb which had

[ signing_mqweb ]

keyUsage = digitalSignature
subjectAltName = DNS:localhost, IP:127.0.0.1
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

Use Linux to be the Certificate Authority for my z/OS RACDCERT certificates.

Create a certificate on z/OS.

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
 
RACDCERT ID(START1) DELETE(LABEL('MYCERTL'))
/* create the certificate - note it is not signed
RACDCERT ID(START1) GENCERT -
SUBJECTSDN(CN('MYCERTL') -
O('SSS') -
OU('SSS')) -
ALTNAME(IP(10.1.1.2) -
DOMAIN('WWW.ME2.COM') )-
SIZE(4096) -
RSA -
WITHLABEL('MYCERTL')

/* convert this to a certificate request and output it
RACDCERT GENREQ (LABEL('MYCERTL')) ID(START1) -
DSN('IBMUSER.CERT.MYCERTL.CSR')
/

The first line of the data set is —–BEGIN NEW CERTIFICATE REQUEST—– which is what I expect for a Certificate Signing Request.

FTP this down to the Linux machine as mycertl.csr and use the openssl ca command.  This uses the cacert.*.pem files created above

openssl ca -config openssl-ca-user.cnf -md sha384 -out mycertl.pem -notext -cert cacert.pem -keyfile cacert.key.pem -policy signing_policy -extensions signing_mqweb -md sha256 -infiles mycertl.csr

Note: My openssl-ca-user.cnf  is given below.

I carefully checked the details displayed, and replied y to both questions.

This produced

Using configuration from openssl-ca-user.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
organizationName :PRINTABLE:'SSS'
organizationalUnitName:PRINTABLE:'SSS'
commonName :PRINTABLE:'MYCERTL'
Certificate is to be certified until Oct 14 15:28:45 2023 GMT (1000 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

If the option -notext is not specified, the output file contains the readable interpretation of the certificate.  Specify -notext to get the output file where the first line is —–BEGIN CERTIFICATE—–

Upload this output file to z/OS (for example “put mycertl.pem ‘IBMUSER.CERT.MYCERTL.PEM’ ” ).

Check the contents before you add it to the RACF keystore

RACDCERT CHECKCERT(‘IBMUSER.CERT.MYCERTL.PEM’)

Add the certificate to the keystore

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT ADD('IBMUSER.CERT.MYCERTL.PEM') -
ID(START1) WITHLABEL('MYCERTL)
/*

The command racdcert list (label(‘MYCERTL’))  id(start1) shows the certificate has NOTRUST, so will not be visible on any keyring. You need

RACDCERT id(START1) ALTER(LABEL(MYCERTL’))TRUST

and will need to connect it to any keyrings.

Upload the CA certificate into the RACF database.

You will also need to upload the public key to the RACF database as a CERTAUTH, and connect it to any ring that uses a certificate signed by the Linux CA.

FTP the certificate file, cacert.pem (created above), to z/OS as text.  Once you have FTPed the file, check the first line is “—–BEGIN CERTIFICATE—–“

Add it to RACF

You can add the certificate owned by a userid ( rather than certauth).   The certificate needs to be connected to the keyring as usage(CERTAUTH).

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
/* delete it if needed
RACDCERT DELETE (LABEL('Linux-CA256')) ID(START1)

RACDCERT id(start1) ADD('IBMUSER.CA256.PEM') -
WITHLABEL('Linux-CA256') TRUST

RACDCERT CONNECT(id(start1) LABEL('Linux-CA256') -
RING(TRUST) USAGE(CERTAUTH)) ID(START1)

RACDCERT CONNECT(id(start1) LABEL('Linux-CA256') -
RING(DANRING) USAGE(CERTAUTH)) ID(START1)

RACDCERT LISTRING(TRUST ) ID(START1)

racdcert list (label('Linux-CA256')) id(start1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

 

My openssl-ca-user.cnf file.

HOME = .
RANDFILE = $ENV::HOME/.rnd

####################################################################
[ ca ]
default_ca = CA_default # The default ca section

[ CA_default ]
default_days = 1000 # How long to certify for
default_crl_days = 30 # How long before next CRL

default_md = sha256 # Use public key default MD
preserve = no # Keep passed DN ordering

x509_extensions = ca_extensions # The extensions to add to the cert

email_in_dn = no # Don’t concat the email in the DN
copy_extensions = copy # Required to copy SANs from CSR to cert

base_dir = .
certificate = $base_dir/cacert.pem # The CA certifcate
private_key = $base_dir/cakey.pem # The CA private key
new_certs_dir = $base_dir # Location for new certs after signing
database = $base_dir/index.txt # Database index file
serial = $base_dir/serial.txt # The current serial number

unique_subject = no # Set to ‘no’ to allow creation of
# several certificates with same subject.

[ signing_policy ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

##########

[ signing_mqweb ]

subjectAltName = DNS:localhost, IP:127.0.0.1
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

 

How do I used z/OS to manage my corporate certificates?

With my first, (and second) reading of the RACF documentation it looked like z/OS could not act as my corporate Certificate Authority.  I was written up a blog post saying how to do it using Linux as my CA, when I found a couple of hints to show that z/OS could be used as a certificate authority.

The short answer on how to do it.

Create a end user certificate on z/OS

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('MYCERT') - 
             O('SS') - 
             OU('SS')) - 
   ALTNAME(IP(10.1.1.2) - 
           DOMAIN('WWW.ME.COM') )- 
   SIZE(4096) - 
   SIGNWITH (CERTAUTH LABEL('MY-CA1')) - 
   RSA - 
   WITHLABEL('MYCERT')

This creates a certificate START1.MYCERT signed by CERTAUTH.MY-CA1

How to process a certificate created on Linux

I created a certificate on Linux in a script (so I could use a variable as part of the file name).

name=”ec2″
passwords=
“-passin file:password.file -passout file:password.file
# generate a private key in rsa format
openssl genrsa -out $name.key.pem 2048
# make a certificate request to send off to get it signed
openssl req -config mqwebserver.config -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=mqwebec”  $passwords

This creates a Certificate Server Request file ec2.csr, and a private key in ec2.key.pem.

I FTP’d the .csr file to z/OS.   This is a text file, where the first line is —–BEGIN CERTIFICATE REQUEST—– so you need to FTP in text format.

With RACF you can generate a certificate by passing the information in a data set, or by specifying it on the RACDCERT GENCERT command.

I used the following job to sign the certificate using Certificate Authority DAN-CA1, passing the information in from the .CSR data set I had previously uploaded.

//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT ID(START1) GENCERT('IBMUSER.IBMUSER.EC2.CSR') - 
   SIGNWITH (CERTAUTH LABEL('MY-CA1')) - 
   WITHLABEL('MYCERT3') 

RACDCERT LIST(LABEL('MYCERT3')) ID(START1)
 
RACDCERT ID(START1) EXPORT(LABEL('MYCERT3'))- 
   DSN('IBMUSER.CERT.MYCERT3.PEM') - 
   FORMAT(CERTB64) 

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 

This reads the data set specified in the GENCERT() command, and takes all of the certificate parameters from it.  It is signed by the CERTAUTH with label(‘MY-CA1’) and stored in the RACF database under id(START1) with label(MYCERT3).

THE EXPORT command exports the signed certificate to a dataset.

I then FTP’d the exported data set IBMUSER.CERT.MYCERT3.PEM back to Linux.

I merged the two parts back together to create a .p12 combined file with

openssl pkcs12 -export -inkey ec2.key.pem -in dancert3.pem -out mycert3.p12 -name mycert3 -passout file:password.file -passin file:password.file

and displayed the .p12 file with

pk12util -l mycert3.p12

This gave me

Certificate(has private key):
Data:
  Version: 3 (0x2)
  Serial Number: 4 (0x4)
  Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
  Issuer: "CN=MYCA1,OU=SSS,O=SSS"
  Validity:
    Not Before: Sun Jan 17 00:00:00 2021
    Not After : Mon Jan 17 23:59:59 2022
  Subject: "CN=mqwebec,O=cpwebuser,C=GB"
  Subject Public Key Info:
    Public Key Algorithm: PKCS #1 RSA Encryption
    RSA Public Key:
    Modulus:...

    Exponent: 65537 (0x10001)
  Signed Extensions:
    Name: Certificate Comment
      Comment: "Generated by the Security Server for z/OS (RACF)"

    Name: Certificate Subject Alt Name
      DNS name: "localhost"
      IP Address: 127.0.0.1

  Name: Certificate Key Usage
    Critical: True
    Usages: Digital Signature
    Non-Repudiation
    Key Encipherment

  Name: Certificate Subject Key ID
    Data:...

  Name: Certificate Authority Key Identifier
    Key ID:...

  Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
    Signature:...
    Fingerprint (SHA-256):...
    Fingerprint (SHA1):...Friendly Name: dancert3

  Key(shrouded):
  Friendly Name: mycert3

  Encryption algorithm: PKCS #12 V2 PBE With SHA-1 And 3KEY Triple DES-CBC

Note: I tried exporting the file on z/OS using FORMAT(PKCS7B64) instead of FORMAT(CERTB64).
The openssl pkcs12 -export command failed with

unable to load certificates
error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149:
error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:309:Type=X509_CINF
error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=cert_info, Type=X509
:error:0907400D:PEM routines:PEM_X509_INFO_read_bio:ASN1 lib:../crypto/pem/pem_info.c:196:

If you get these messages download it in the proper format.

The longer answer…

When setting up certificates you have several options

  1. Use a Self Signed certificate.  This offers very little security, as anyone can create a certificate one.   This should only be used for testing, though I think you should never use them, as you need to test with signed certificates because of the additional work that needs to be done.  Imagine conversation with your manager “Did you test this?”  “No, I tested something else”.
  2. Get a certificate from recognised authority.   You pay for each certificate you get.  As the authority’s certificate is generally available you can use your certificate as soon as you get it.
  3. Set up your own authority you create your own high level certificate such as CN=TOPLEVELCA O=MYORG, which in turn creates other certificates such as CN=TESTCA, O=MYORG, and CN=PRODUCTIONCA O=MYORG.  You can create as many certificate as you wish for example using the TESTCA for certificates that can be used in test.
  4. A combination of both, you get a High level Certificate Authority certificate signed by a recognised authority, and then used that  to set up your own certificates.

What does a “certificate” contain?

There are two parts

  1. The certificate identified the “who”, along with the public key.
  2. The private key used in encryption.

What does “signing”  mean?

You take a block of data, and do a calculation on it, for example a checksum.  The signer takes the checksum and encrypts it using its private key.  The original block of data then has the encrypted checksum, and the signers public key appended to it.

To check the signature,

  1. You compare the signers public key with your copy of it and check it matches.  If they match you decrypt the encrypted checksum.
  2. You perform the same calculation on the original payload and check your calculation with the decrypted version.  If they match you know the data is good.

Signing a certificate is doing this with a certificate as the block of data.

If you have a corporate CA certificate, and a departmental CA certificate, the departmental certificate has been signed by the corporate CA.  When you sign a block of data, it will have the departmental certificate, and the corporate certificate appended.

How does a Certificate Authority work?

At the top of the Certificate Authority hierarchy is a generally available self signed certificate with its public key.

  • I create a personal certificate with  “who” and the public key, and the private key.
  • I send the personal certificate with  “who” and the public key, known as a Certificate Signing Request(CSR) to my security department, who sign it and return it.
  • I merge the signed “who” and the public key, with the private key, and store it in my key store.

How to define a midrange self signed certificate and get it into z/OS

Although I disapprove of using self signed certificates, (it is like having a secure lock on one door, but you leave another door open).   Sometimes there is a need to use one on the journey to using signed certificates – just to get the environment working.

Create the certificate on Linux

On Linux, I set up a bash script to create the self signed certificate.  (I used a bash script so I can parametrize values, and “name” is used in many places.)

subj=’-subj /C=GB/O=SSS/CN=SS’
addext=’-addext keyUsage=keyCertSign,digitalSignature ‘
name=’ss’
passwords=”

#remove the old files for this cert
rm $name.pem $name.key.pem $name.p12

#create the  certificate and the private key
openssl req -x509 -newkey rsa:4096 -nodes -out $name.pem -keyout $name.key.pem $subj $addext

 

#Compine the two files into a p12 (PKCS12) file.
openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -name $name $passwords

#display it
openssl x509 -in $name.pem -text -noout|less

This creates the certificate file, the key file and a .p12 file to be used by my application.

FTP the certificate file, ss.pem, to z/OS as text.  Once you have FTPed the file, check the first line is “—–BEGIN CERTIFICATE—–“

Add it to RACF

 

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT DELETE (LABEL('MYSS')) ID(START1)
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

RACDCERT ADD('IBMUSER.MY.SS.PEM') -
ID(START1) WITHLABEL('MYSS')

/* connect it to START1.TRUSTRING
RACDCERT ID(START1) CONNECT(RING(TRUSTRING ) -
USAGE(CERTAUTH) -
ID(START1) LABEL('MYSS') )

racdcert list (label('MYSS')) CERTAUTH

racdcert listring(TRUSTRING) id(start1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh
/*

The list command gave

Digital certificate information for START1:
Label: MYSS
Certificate ID: 2QiJmZmDhZmjgcTB1eLi
Status: TRUST
Start Date: 2021/01/18 14:14:38
End Date: 2021/02/17 14:14:38
Serial Number:
>73CB2B49D0E56BBEAA95EC8EA01FCCC549A97BA8<
Issuer's Name:
>CN=SS.O=SSS.C=GB<
Subject's Name:
>CN=SS.O=SSS.C=GB<
Signing Algorithm: sha256RSA
Key Type: RSA
Key Size: 4096
Private Key: NO
Ring Associations:
Ring Owner: START1
Ring:
>TRUSTRING<

You can see this does not have the private key.   The issuer’s name matches the Subject’s name, so this shows it is self signed.

Restart your web server to pick up the changes to the trust keyring.

Change your client to point to the .p12 file, and specify the password.

-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.keyStore=/home/colin/checkTLS/ss.p12
-Djavax.net.ssl.keyStoreType=pkcs12

Restart your client.

Other ways of creating your client certificate

Above I used

openssl req -x509 -newkey rsa:4096 -nodes -out $name.pem -keyout $name.key.pem $subj $addext

to create a request.   This is actually two operations

  1. Create a private key
  2. Create a certificate

For some combination of parameters it may be easier to do this in two steps

Create the private key.  This creates a private key using an elliptic curve.

openssl ecparam -name secp384r1 -genkey -noout -out $name.key.pem

Create the certificate using the private key

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name $password

A practical path to installing Liberty and z/OS Connect servers – 6 Enabling TLS

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Configuring TLS

  1. You can configure the server to creates a keystore file on its first use. This creates a self signed certificate. This is good enough to provide encryption of the traffic. Certificates sent from the client are ignored as the trust store does not have the Certificate Authority certificate to validate them.
  2. You can use your site’s keystore and trust store. The server can use them to process certificate sent from the client for authentication.

Decide how you want to authenticate

Most of the functions require an https connection. This will require a keystore.

You can decide if

  1. The server uses the client’s certificate for authentication,
    1. if that does not work then use userid and password
    2. if that does not work, then fail the request; there is no fall back to userid and password.
  2. The server does not use the clients certificate.
    1. You can configure that userid and password will used for authentication
    2. There is no authentication

Have the server create a keystore.

You can get Liberty to create a keystore for you. This creates a self signed certificate and is used to encrypt the traffic between client and server. This is a good start, while you validate the set up, but is not a good long term solution.

Create keystore.xml with

<server>
<keyStore id="defaultKeyStore" password="${keystore_password}" /> 

<ssl clientAuthentication="false" 
    clientAuthenticationSupported="false" 
    keyStoreRef="defaultKeyStore" 
    id="defaultSSLSettings" 
    sslProtocol="TLSv1.2" 
/> 
</server>

Add to the bottom of the server.xml file

 <include location="${server.config.dir}/keystore.xml"/>

If you have keyStore id=”defaultKeyStore”, (it must be defaultKeyStore) and do not have a keystore defined, the the server will create the keystore in the default location (${server.output.dir}/resources/security/key.p12) with the password taken from the server.env file.  See here.

Restart the server.

I got the messages

CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started 
and is now listening for requests on host 10.1.3.10  
(IPv4: 10.1.3.10) port 9443.

Showing TLS was active, and listening on the 9443 port.

If the keystore was created, you will get messages like

[AUDIT   ] CWPKI0803A: SSL certificate created in 87.578 seconds. 
SSL key file: /var/zosconnect/servers/d3/resources/security/key.p12 
[INFO    ] Successfully loaded default keystore: 
/var/zosconnect/servers/d3/resources/security/key.p12 of type: PKCS12

The certificate has a problem (a bug). It has been generated with CN:localhost, O:ibm: ou:d3 where d3 is the server name. The Subject Alternative Name (SAN) is DNS:localhost. It should have a SAN of the server’s IP address (10.3.1.10 in my case).

Clients check the SAN and compare it with the server’s IP address.

  1. Chrome complain. “Your connection is not private NET:ERROR_CERT_AUTHORITY_INVALID”, and the option to accept it
  2. Firefox gives “Warning: Potential Security Risk Ahead”, and the option to accept it.
  3. Z/OS explorer gives a Server certificate alert pop up, saying “Host:10.1.3.10 does not match certificate:localhost” and gives two buttons Decline or Accept.
  4. With curl I got SSL_ERROR_SYSCALL.

You can accept it, and use it until you have your own keystores set up. You can also reset this decision.

Using a RACF keyring as the keystore.

You can use a file based keystore or a RACF keying.  Below are the definitions for my RACF keyrings. The started task userid is START1. The keystore (containing the private key for the server is keyring START1/KEY. The server should use key ZZZZ.

The trust store, containing the Certificate Authority certificates and any self signed certificates from clients, is START/TRUST.

The <ssl.. /> points to the different keystores, so it makes sense to keep all these definitions in one file.  You may already have a file of these definitions which you can use from another Liberty server.

<server>

<sslDefault sslRef="defaultSSLSettings"/> 
<ssl clientAuthentication="true" 
    clientAuthenticationSupported="true" 
    id="defaultSSLSettings" keyStoreRef="racfKeyStore"  
    serverKeyAlias="ZZZZ" 
    sslProtocol="TLSv1.2" 
    trustStoreRef="racfTrustStore"/> 
                                                                                                                  
  <keyStore filebased="false" id="racfKeyStore" 
     location="safkeyring://START1/KEY" 
     password="password" 
     readOnly="true" 
     type="JCERACFKS"/> 
                                                                                                                  
  <keyStore filebased="false" id="racfTrustStore" 
     location="safkeyring://START1/TRUST" 
     password="password" 
     readOnly="true" 
     type="JCERACFKS"/>                                                                                                                  
</server>

This sets clientAuthentication=”true” and clientAuthenticationSupported=”true”

Specify if you want to use a client certificate for authentication

If you specify clientAuthenticationSupported=”true”… the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.

The default keystore will not be able to validate any certificates sent from the client. When connecting to Chrome with certificates set up, I got an FFDC and messages

  • [INFO ] FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: java.security.cert.CertPathValidatorException: The certificate issued by CN=SSCA8, OU=CA, O=SSS, C=GB is not trusted; internal cause is: java.security.cert.CertPathValidatorException:
  • [ERROR ] CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired.

If you specify clientAuthentication=”false” (the default) the server does not request that a client send a certificate during the handshake.

If you specify <webAppSecurity allowFailOverToBasicAuth=”true” />  the client certificate connection is not used or it fails,

  1. if  you specify<webAppSecurity allowFailOverToBasicAuth=”true” /> the user will be prompted for userid and password
  2. If you specify <webAppSecurity allowFailOverToBasicAuth= false > or not specified, the connection will fail.

If a userid and password can be used, the first time a browser uses the server it will be prompted for userid and password. As part of the handshake, the LTPA2 cookie is sent from the server. This has the userid and password encrypted within it. If you close down the browser and restart it (not just restart it from within the browser) you will be prompted again for userid and password. You can also be prompted for userid and password once the LPTA cookie has expired.

If you are using z/OS explorer and get a code 401, unauthorised, you may be using a certificate credential ( format userid@CertificateAuthority(CommonName)) rather than a userid and password with format of just the userid eg COLIN. Use “Set Credentials” to change credentials.

You can see what userid is being used for the requests, from the …/logs/http_access.log file.

To make it even more complex you can have different keystores for different connections or ports.  See here. But I would not try that just yet.

Map client certificates to a SAF userid

If you are using certificate authentication you will need to map the certificate to a userid using the RACDCERT MAP command.

Testing it

If the server starts successfully you can use a web browser with URL

  http:/10.1.3.10:9443/zosConnect/api-docs

and it should display json data.

If you get “Context Root Not Found” or code 404 you should wait and retry, as the https processing code is active, but the code to process the requests is not yet active.

Review the contents of …/servers/…/logs/http_access.log to see the request being issued and the http completion code.

If you have problems connecting clients over TLS add -Djavax.net.debug=ssl:handshake to the jvm.options file and restart the server.

If you connect to the z/OS Explorer, and logon to the z/OS Connect EE Server, you should have a folder for APIs and Services – which may have no elements.