With MQ, if a message cannot be successfully delivered, it can be put on a Dead Letter Queue for later processing.

You can have multiple queues

• The system dead letter queue, where the MQ puts messages it cannot processed,
• Application dead letter queues, and application can put messages to a queue,
• The AMS dead letter queue for messages which had errors during get or put, for example a certificate mismatch.

Messages can be put to these queues for a variety of reasons.

• Transient problems
  • If a channel is putting a message to a queue, and the queue is full, then the channel can put the message to the Dead Letter Queue. The DLQ handler can then try to put the message to the original queue, and retry a number of times after an interval. If the queue full condition was transient, then the DLQ handler is likely to succeed. If an application stops processing a queue, you can get quickly get thousands of messages on the DLQ queue.
  • The queue is put disabled. A queue can be set to put disabled, for example to stop messages from going onto a queue during queue maintenance. Once the maintenance has been done the queue can have put enabled.
• Administration
  • The putting channel is not authorised to put to the queue, so the message gets put to the DLQ. An administrator needs to check to see if the putter is allowed to put the message. If so, fix the security and put the message back on original queue. If not remove the message, and educate the developer.
  • An AMS protected message has a problem, for example an unauthorised user has signed a message, or the id getting the message does not have a certificate to decrypt a message. You need to resolve any local certificate problems, or send the original message back to the requester saying it is in error.
• Application
  • The message is too large for the queue. The administrator needs to educate the developer and/or make the queue max message size larger.

You may have a policy that non persistent messages for a particular queue which end up on the dead letter queue should be purged. Persistent message for another queue should have special treatment.

You may want administrators to be able to look at the meta data about a message, destination queue, MSGID, the list of recipients who can decrypt a message; but not to look at the message content.

Setting up your environment to cover these areas need considerable planning.

Implementing a solution

You want to try to keep the main DLQ close to empty, for example if your DLQ fills up with non persistent inquiries, then putting an important persistent message to the DLQ may fail.

You can use the runmqdlq program on midrange or CSQUDLQH on z/OS, to specify rules for automatic processing of messages on the DLQ.

You can select on attributes like original destination queue name, the reason why the message was on the DLQ, userid in the MQMD; and specify an action

• Retry the put to the original queue
• Move to another queue
• Purge it
• Leave it

When a message is processed on the DLQ, the rules are applied, and the action of the first matching rule is applied. For example

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(RETRY) RETRY(5)
DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(FWD) FWD(MYQUEUEOVERFLOW) HEADER(YES)

This says that if a messages destination was MYQUEUE, and the reason code was MQRC_Q_FULL, it retries the put to the queue, at most 5 times. After 5 attempts, the first rule is skipped, the second rule is used, and the message is forwarded to the queue MYQUEUEOVERFLOW keeping the DLQ header.

DEST(INQ*) PERSIST(MQPER_NON_PERSISTENT ACTION(DISCARD)

For message destination INQ* and non persistent messages, then just discard them.

DEST(INQ*) PERSIST(MQPER_PERSISTENT ACTION(LEAVE))

For message destination INQ* and persistent messages, then just leave them on the queue, for some other processing.

If runmqdlq or CSQUDLQH is restarted, then all processing is reset.

Generic rules

For transient type problems you may want to consider

• Non persistent messages for a set of queues get purged, dont even try to put them back on the queue.
• Persistent messages for INVOICE* queues get moved to INVOICE_DLQ queue, where you have another DLQ monitor running on the queue.

For administrator type problems

• You could pass non persistent messages to an admin_DLQ_NP queue, and have a program which reads the meta data, and prints it to a file, then deletes the original message
• You could pass persistent messages to an admin_DLQ_P queue
  • have a program which reads the meta data, and prints it to a file, and leaves the message on the queue.
  • Using the meta information resolve the problem.
  • Have another program which takes the msgid and correlid as input parameters, then puts the message on the original queue. (If there is only one message, you could use the default DLQ handler to do this.)

For AMS problems

• This is complicated by having to use a different queue. If the DLQ handler tries to put to the AMS protected queue, it will be “protected” (enciphered) again. You need to use put the message to an alias queue, with the original queue as the target. On midrange Java and C clients can disable AMS processing, either by using an environment variable, or through the MQCLIENT.ini file. See here.
• This is also complicated by possibly needing access to information in the payload, such as the list of recipients, and decrypting the message to get the DN of the signer.
  • Once you have resolved the problem, have another program which takes the msgid and correlid as input parameters, and puts the message on the alias queue (if there is only one message you could use the default DLQ handler to do this).

How do I check it I have got it right?

It is worth putting a process in place to monitor the depth of the dead letter queue, and if it does not become empty a few a minutes, display the contents of the queue, and add rules to handle the residual messages.

I do not think that IBM provides a list of return codes of messages that it puts onto the DLQ, I think you’ll have to go through the list (over 500!), and put a rule in place for each one. If an application invents its own return codes, you may need rules for these as well.

My quick look at the list includes

Common problems

2053 (0805) (RC2053): MQRC_Q_FULL
2056 (0808) (RC2056): MQRC_Q_SPACE_NOT_AVAILABLE
2071 (0817) (RC2071): MQRC_STORAGE_NOT_AVAILABLE

Other problems
2030 (07EE) (RC2030): MQRC_MSG_TOO_BIG_FOR_Q
2031 (07EF) (RC2031): MQRC_MSG_TOO_BIG_FOR_Q_MGR
2033 (07F1) (RC2033): MQRC_NO_MSG_AVAILABLE
2051 (0803) (RC2051): MQRC_PUT_INHIBITED
2052 (0804) (RC2052): MQRC_Q_DELETED
2053 (0805) (RC2053): MQRC_Q_FULL
2056 (0808) (RC2056): MQRC_Q_SPACE_NOT_AVAILABLE
2071 (0817) (RC2071): MQRC_STORAGE_NOT_AVAILABLE
2102 (0836) (RC2102): MQRC_RESOURCE_PROBLEM
2120 (0848) (RC2120): MQRC_CONVERTED_MSG_TOO_BIG
2141 (085D) (RC2141): MQRC_DLH_ERROR
2142 (085E) (RC2142): MQRC_HEADER_ERROR
2148 (0864) (RC2148): MQRC_IIH_ERROR
2149 (0865) (RC2149): MQRC_PCF_ERROR
2150 (0866) (RC2150): MQRC_DBCS_ERROR
2342 (0926) (RC2342): MQRC_DB2_NOT_AVAILABLE
2345 (0929) (RC2345): MQRC_CF_NOT_AVAILABLE
2348 (092C) (RC2348): MQRC_CF_STRUC_AUTH_FAILED
2349 (092D) (RC2349): MQRC_CF_STRUC_ERROR
2362 (093A) (RC2362): MQRC_BACKOUT_THRESHOLD_REACHED

You may want common ones, such as queue full and not authorised, on a per queue basis, and all the less common ones, such as all of the z/OS ones all putting to one “admin” queue.

If AMS has problems with a protected message, AMS can put the message on the SYSTEM.PROTECTION.ERROR.QUEUE queue. This blog post discusses what you can do about it. I consider this a hard problem – not in the same league as trying to simulate the beginnings of the Universe – more like climbing Ben Nevis mountain in Scotland, when you are only used to walking down to the shops.

What are the problems?

There are several problems you need to consider

1. Why is the message on the queue? Is it a problem with the putting application, or with the getting environment?
2. Which user had the problem. For example it may not be obvious which application instance had a problem, if applications come in through one channel, and many users have the same MCA userid.
3. What you need to do about it to get the message reprocessed, and prevent future problems.

Why is the message on the queue

The messages could be on the queue because

• The certificate was signed, but the DN of the signer is not in the setmqspl list of authorised signers (-a). This is an example of an MQ configuration problem
• The user getting a message was not able to verify the signers certificate sent in the message, for example it is missing the CA of the signer, or missing the signers self signed certificate. This is an example of a user’s configuration problem.
• The message was encrypted, but the user getting the message did not have access to a private certificate to allow the message to be decrypted. The user’s DN needs to be added to the recipients when the message is put and enciphered (or the user needs to be stopped from getting messages from this queue). This is an example of the putting of the message message is missing configuration information.

What other information is there to help me?

If you know which id had the problem, there should be information in the error logs. For example a problems within a Java JMS client program may write to mqjms.log.0. A local application may write to the queue manager’s error log, for example /var/mqm/qmgrs/QMA/error/*01*

In the mqjms.log.0 I got

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.prot.MessageProtectionBCImpl
The receiver of this encrypted message is not on the message recipient list ‘CN=ja2,O=aaaa,C=GB’
The certificate of a user that is receiving a message is not on the message RecipientsInfo list.

Verify that the user is on a recipients list in a security policy definition.

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.intercept.JmqiGetInterceptorImpl
The IBM MQ Advanced Message Security Java interceptor failed to unprotect the received message.
An error occurred when the IBM MQ Advanced Message Security Java interceptor was unprotecting the received message.

See subsequent messages in the exception for more details about the cause of the error

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.service.EseMQServiceImpl
The IBM MQ Advanced Message Security interceptor has put a defective message on error handling queue ‘SYSTEM.PROTECTION.ERROR.QUEUE ‘.

(On z/OS the messages are less helpful.)

On the SYSTEM.PROTECTION.ERROR.QUEUE queue there was a message with a Dead Letter Header (DLH) with a reason 2063 0x0000080f MQRC_SECURITY_ERROR.

From the MQMD you can see the time the message was put to the queue, the putting application, and the user identification. This userIdentified may have been set for example by the channel MCAUSER, or CHLAUTH, and so you do not always know where the original request came from.

If you are testing then you will know what caused the error.

If you are in a production like environment, you know the application, and as you will have configured all user keystores the same you may not need to know which specific user caused the problem. If there is a problem with a missing certificate, then you fix the problem, redeploy the keystore to all your users (as part of your automated process) and try again.

How do you tell what the problem is.

Your systems administrator needs a process for extracting meta data about messages, while keeping the application payload protected. You could build a process around displaying the recipients and signer from How do I find the recipients and signer of an AMS message? The systems administrator needs to know

• the original queue name with the problem
• the time, date, and user that put the message
• the msgid and correlid of the message – so if you put it back on the queue, you know which message to process.
• from the message, the type of protection: Integrity (it was signed), Encryption (it was encrypted), Privacy (it was encrypted with a signed payload)
• any id information from the message, such as recipient DN’s, and the signer DN. See this post.
• you may have to have some special processing to decrypt the Privacy payload, just to extract the signer information.

If this process can be automated, then any application content can be kept secure.

With this information and your “up to date application work book” (do you have one of these?) , you should be able to identify the problem.

Once you have fixed the root cause of the problem….

The fix may be to change the setmqspl to add an authorised signer, or to add a certificate to the recipients keystore, you need to get the message reprocessed.

• You get the specific message from the message id and correlid.
• You need to remove the DLQ header from the message.
• You need some special set up for the queue. If you try to put to the original queue, it will get the AMS protection again, for example re-encrypted or resigned. You need a queue alias so you put to the queue alias bypassing any AMS processing.

There are lots of things you need to consider, which is why I consider this a hard problem.

This application would be a good example where message handle is used to “move” the message.

MQCRTMH(hConn,&cmho,&hMsg,&CompCode,&Reason);

gmo.MsgHandle = hMsg;

MQGET(hConn,….);

pmo.Action = MQACTP_FORWARD;

pmo.OriginalMsgHandle =hmsg

MQPUT(…)

I found Learn to code the MQ Message Property MQI calls from MQGEM software useful in understanding message handle and message properties, and how to delete the DLQ header.

Other AMS blog posts

• Overview and challenges of using end to end message encryption (AMS).
• Certificate and keyring management for AMS.
• Installing AMS on z/OS.
• What certificates do I need for AMS with a client going to z/OS?
• checkAMS: program to check your AMS defintions are consistent with z/OS keyring
• Adding more users to AMS.
• Installing AMS on midrange for end to end message protection
• Understanding setmqspl to set up AMS definitions.
• How to administer AMS policies, and use the set policy command.
• Brain check and reset, needed for AMS keystores on midrange.
• Using runamscred to encrypt the passwords in the keystore.conf file.
• How do I find the recipients and signer of an AMS message?
• I have a message on the AMS DLQ – what can I do about it? and How do I process messages on the dead letter queue (DLQ)?

When you have a problem trying to decrypt/get a protected message, the message can end up on the AMS dead letter queue. This blog post explains how to look into the message and find the list of recipient, and the signer of the data.

This started off as a short blog post, then as I added more background and more detailed instructions it just grew!

The sections are

• Background on the format of the protected message
• 30 second overview of ASN.1 encoding
• What does signed data look like?
• What does encrypted data look like?
• How to map Issuer+ serial number to subject
• Decrypting the data
• Using Java to look at the data

Background on the format of the protected message.

The protected message conforms to the PKCS#7 protocol as defined here.

The specification defines six formats, of which three are:

• signed data
• envelopedData where data is wrapped in an opaque envelope – it has been encrypted
• signedAndEnveloped.

These may look like the AMS classes: Integerity, Confidential, and Privacy; close – but no cigar. AMS does the signing before encryption, and so only signed data, and envelopedData are used. To see who signed a privacy message, you have to decrypt it.

I wrote a program to browse the messages on a queue, and write the protected data out to a Linux file. You can use openssl tools to explore file.

There is a PDMQ header at the front of the data. This has a variable length

• at offset 0 is “PDMQ” in ASCII.
• like other variable length MQ structures, at offset 8 is an MQLONG StrucLength. Don’t forget you may need to convert from big-endian to little-endian on some platforms.
• The data after the header is the data is ASN.1 encoded, starting with x’30’.

30 second overview of ASN.1 encoding

ASN.1 encoding takes data like “typeOfRequest,{[id1,key1],[id2,key2]} and converts it into

• a SEQUENCE (the items have an order) of typeOfRequest, keys
• where keys are a SET (because it does not matter if the order is id1, id2, or id2, id1)
• ID is an integer…

Terms like countryName C=GB map into a String of type 2.5.4.6 with length 2 and value GB.

When this is decoded it take 2.5.4.6 and says this is CountryName. You can look up the id here. There is a document with popular ones in it.

You can use openssl to format a file of ASN.1 encoded data and interpret the values. For example

openssl asn1parse -inform DER -in colin.pkj -i -offset 104 > asn1

says

• openssl asn1parse this command
• -inform DER in format is decimal encoded (not base 64)
• -in colin.pkj this is the input file
• -i says indent the output to prettify it
• -offset 104 skip over the PDMQ header
• > asn1 write the output to the file asn1

When I changed my program to skip over the PMDQ header and write out just the data, I didn’t need the -offset parameter.

I’ve also used this to display the contents of a certificate in .der format.

What does signed data look like?

The spec says signed data looks like

ContentInfo ::= SEQUENCE {
     contentType ContentType,
     content
}
where contentType is SignedData... 
SignedData ::= SEQUENCE {
     version Version,
     digestAlgorithms DigestAlgorithmIdentifiers,
     contentInfo ContentInfo,
     certificates
     signerInfos SignerInfos
 }

The start of the output of the openssl asn1parse command for an integrity message contained

    0:d=0  hl=4 l=1723 cons: SEQUENCE          
    4:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-signedData
   15:d=1  hl=4 l=1708 cons:  cont [ 0 ]        
   19:d=2  hl=4 l=1704 cons:   SEQUENCE          
   23:d=3  hl=2 l=   1 prim:    INTEGER           :01
   26:d=3  hl=2 l=  15 cons:    SET               
   28:d=4  hl=2 l=  13 cons:     SEQUENCE          
   30:d=5  hl=2 l=   9 prim:      OBJECT            :sha256
   41:d=5  hl=2 l=   0 prim:      NULL              
   43:d=3  hl=2 l=  20 cons:    SEQUENCE          
   45:d=4  hl=2 l=   9 prim:     OBJECT            :pkcs7-data
   56:d=4  hl=2 l=   7 cons:     cont [ 0 ]        
   58:d=5  hl=2 l=   5 prim:      OCTET STRING      :AAAAA

The columns are

• offset from the start
• d= nesting depth, so a sequence within a set, within a sequence would be d=3
• dl= length of length field
• l= length of the field
• cons|prim. I think of these as
  • cons is construction … SEQ, SET etc.
  • prim is primary data
• SEQUENCE, OBJECT, INTEGER type of value

Looking at the output

• at offset 4 we have ContentType which is :pkcs7-signedData
• at offset 23 we have the SignedData version :01
• at offset 30 we have a SET of digestAlgorithms, there is one in the SEQUENCE and it has the value :sha256
• at offset 45 we have contentInfo :pkcs7-data
• at offset 58 we have the message data, the very boring AAAAA.

After the data we have signer info

SignerInfo ::= SEQUENCE {
     version Version,
     issuerAndSerialNumber IssuerAndSerialNumber,
     digestAlgorithm DigestAlgorithmIdentifier,
     authenticatedAttributes
       [0] IMPLICIT Attributes OPTIONAL,
     digestEncryptionAlgorithm
       DigestEncryptionAlgorithmIdentifier,
     encryptedDigest EncryptedDigest,
     unauthenticatedAttributes
       [1] IMPLICIT Attributes OPTIONAL }

Extracting the interesting fields out of the data (to make it easier to read)

• Certificate serial number: 01B0
• Issuer:
  • countryName:GB,
  • organizationName:SSS,
  • organizationalUnitName: CA,
  • commonName:SSCARSA1024
• Not before time: 210328172922Z
• Not after time: 231223172922Z
• Subject:
  • countryName:GB,
  • organizationName:aaaa,
  • commonName: ja2
• digestAlgorithm: rsaEncryption
• X509v3 Key Usage:….
• X509v3 Subject Alternative Name…
• digestEncryptedAlgorithm: sha256WithRSAEncryption
• …

What does encrypted data look like?

The spec says

EnvelopedData ::= SEQUENCE {
     version Version,
     recipientInfos RecipientInfos,
     encryptedContentInfo EncryptedContentInfo }

   RecipientInfos ::= SET OF RecipientInfo

   EncryptedContentInfo ::= SEQUENCE {
     contentType ContentType,
     contentEncryptionAlgorithm
       ContentEncryptionAlgorithmIdentifier,
     encryptedContent
       [0] IMPLICIT EncryptedContent OPTIONAL }

Instead of listing the DN’s of the recipients, it listed CA certificate + serial number (which equates to the same thing). As a self signed has signed itself, there is an entry for the self signed.

4:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-envelopedData

23:d=3 hl=2 l= 1 prim: INTEGER :00

42:d=8 hl=2 l= 9 cons: SEQUENCE
self signed 
C=GB,O=AAA,CN=colinPaice,  serial=606439D5
44:d=9 hl=2 l= 3 prim:     OBJECT          :countryName
49:d=9 hl=2 l= 2 prim:     PRINTABLESTRING :GB

57:d=9 hl=2 l= 3 prim:    OBJECT            :organizationName
62:d=9 hl=2 l= 3 prim:    PRINTABLESTRING   :AAA  

71:d=9 hl=2 l= 3 prim:    OBJECT            :commonName
76:d=9 hl=2 l= 11 prim:   PRINTABLESTRING   :colin paice
89:d=6 hl=2 l= 4 prim:    INTEGER           :606439D5

97:d=6 hl=2 l= 9 prim:    OBJECT            :rsaEncryption

the following is the encrypted key for this user
110:d=5 hl=3 l= 128 prim: OCTET STRING [HEX DUMP]:861C...F2
241:d=4 hl=4 l= 348 cons: SEQUENCE
245:d=5 hl=2 l= 1 prim: INTEGER :00
248:d=5 hl=2 l= 68 cons: SEQUENCE

This is the issuer C=GB,O=SSS,OU=CA,CN=SSCARSA1024, serial 1B0
256:d=9 hl=2 l= 3 prim:    OBJECT          :countryName
261:d=9 hl=2 l= 2 prim:    PRINTABLESTRING :GB
269:d=9 hl=2 l= 3 prim:    OBJECT          :organizationName
274:d=9 hl=2 l= 3 prim:    UTF8STRING      :SSS
283:d=9 hl=2 l= 3 prim:    OBJECT          :organizationalUnitName
288:d=9 hl=2 l= 2 prim:    UTF8STRING      :CA
296:d=9 hl=2 l= 3 prim:    OBJECT          :commonName
301:d=9 hl=2 l= 11 prim:   UTF8STRING      :SSCARSA1024
314:d=6 hl=2 l= 2 prim:   INTEGER          :01B0
318:d=5 hl=2 l=13 cons: SEQUENCE          
320:d=6 hl=2 l= 9 prim:  OBJECT            :rsaEncryption
the following is the encrypted key for this user
333:d=5 hl=4 l=256 prim:OCTET STRING      [HEX DUMP]:35..7BF

This is for Issuer O=TEMP, OU=TEST,CN=TEMP4Certification Authority Serial = 5D
593:d=4 hl=4 l= 354 cons: SEQUENCE          
597:d=5 hl=2 l=   1 prim: INTEGER           :00
608:d=9 hl=2 l=   3 prim:   OBJECT            :organizationName
613:d=9 hl=2 l=   4 prim:   PRINTABLESTRING   :TEMP
623:d=9 hl=2 l=   3 prim:   OBJECT            :organizationalUnitName
628:d=9 hl=2 l=   4 prim:   PRINTABLESTRING   :TEST
638:d=9 hl=2 l=   3 prim:   OBJECT            :commonName
643:d=9 hl=2 l=  28 prim:   PRINTABLESTRING   :TEMP4Certification Authority
673:d=6 hl=2 l=   1 prim:  INTEGER           :5D
678:d=6 hl=2 l=   9 prim:  OBJECT            :rsaEncryption
the following is the encrypted key for this user
691:d=5 hl=4 l= 256 prim:OCTET STRING      [HEX DUMP]:6A...68
951:d=3 hl=2 l=  76 cons: SEQUENCE          
953:d=4 hl=2 l=   9 prim:  OBJECT            :pkcs7-data
964:d=4 hl=2 l=  29 cons:  SEQUENCE          
966:d=5 hl=2 l=   9 prim:   OBJECT            :aes-128-cbc
977:d=5 hl=2 l=  16 prim:   OCTET STRING      [HEX DUMP]:9D...543A

There were three recipients configured for the queue (setmqspl -r option)

• CN=ja2, O=aaaa, C=GB. See offset 256 to 333, giving the Issuer, the certificate serial number, encryption type, and the hex dump of the encrypted key.
  • Serial Number: 01 B0
  • Issued by: CN=SSCARSA1024, OU=CA, O=SSS, C=GB
• CN=colin paice, O=AAA, C=GB See offset 44 to 110, giving the Issuer (this is self signed), the certificate serial number, encryption type, and the hex dump of thea encrypted key.
  • Serial Number: 60 64 39 D5
  • Issued by: CN=colin paice, O=AAA, C=GB
• CN=ADCDB,O=SSS. See offset 597 to 691, giving the Issuer, the certificate serial number, encryption type, and the hex dump of the encrypted key.
  • Serial Number: 5D
  • Issued by: CN=TEMP4Certification Authority, OU=TEST, O=TEMP

It was not easy to take the issuer information and map this to the Subject. I had to go through the whole of my keystore to find the information.

How to map Issuer+ serial to subject

For .jks keystores you can use

keytool -list -keystore ja2.jks -storepass zpassword -v |grep ‘Serial\|Issuer:\|Owner:’ | less

and search for the serial Issuer CN, and the serial number which follows it. For example

Owner: CN=ja2, O=aaaa, C=GB
Issuer: CN=SSCARSA1024, OU=CA, O=SSS, C=GB
Serial number: 1b0

For .CMS I used a bash shell script to extract the contents, then display the details of each label.

ks="/home/colinpaice/mq/zzserver.kdb"
pw="password"

z=runmqakm -cert -list -db $ks -pw $pw | awk '{print $2}'
for x in $z;do
     runmqakm -cert -details -db $ks -pw $pw -label $x |grep 'Serial\|Issuer\|Subject'
done

What encryption algorithm was used?

At offset 966 is

966:d=5 hl=2 l= 9 prim: OBJECT :aes-128-cbc

The aes128 matches the -e AES128 option on the queue.

What does privacy look like?

Just like encrypted. it has

955:d=4 hl=2 l= 9 prim: OBJECT :pkcs7-signedData

instead of

953:d=4 hl=2 l=9 prim: OBJECT:pkcs7-data

The signing data is encrypted, so cannot be viewed.

Decrypting the data

If you have a file of the data, you can use openssl cms to decrypt it. The example below uses the private key for ja2 in ja2.key.pem. ( I knew it was ja2’s key file from the information above).

openssl cms -decrypt -in colin.pkj -inform der -inkey ~/ssl/ssl2/ja2.key.pem -out colin.der

Now that it is decrypted, display the signed package

openssl asn1parse -inform DER -in colin.der -i

and I can see information about the signer: the issuer, the signer and the signing time; and the payload.

Using Java to look at the data

You can use Java to process the message, using code from bouncycastle (used by MQ).

Read the message as a byte array (data) and use

import org.bouncycastle.asn1.ASN1InputStream;
import org.bouncycastle.asn1.ASN1Primitive;
import org.bouncycastle.asn1.util.ASN1Dump;   
....  
ASN1InputStream ais = new ASN1InputStream(data);
ASN1Primitive obj = ais.readObject();
System.out.println(ASN1Dump.dumpAsString(obj, true));
       

This produces output like the ans1parse above but without the conversion of 2.5.4.6 to CountryName etc. You can either guess the meaning, or look it up here.

ObjectIdentifier(1.2.840.113549.1.7.2)
 Tagged [0]
  Sequence
   Integer(1)
   Set
    Sequence
     ObjectIdentifier(2.16.840.1.101.3.4.2.1)
     NULL
   Sequence
    ObjectIdentifier(1.2.840.113549.1.7.1)
    Tagged [0]
     DER Octet String[3] 
      424242 BBB
    Tagged [0]
   Sequence
    Sequence
     Tagged [0]
      Integer(2)
      Integer(432)
      Sequence
       ObjectIdentifier(1.2.840.113549.1.1.11)
       NULL
      Sequence
       Set
        Sequence
         ObjectIdentifier(2.5.4.6)
         PrintableString(GB) 
       Set
        Sequence
         ObjectIdentifier(2.5.4.10)
          UTF8String(SSS) 
       Set
       Sequence
        ObjectIdentifier(2.5.4.11)
        UTF8String(CA) 
       Set
        Sequence
         ObjectIdentifier(2.5.4.3)
         UTF8String(SSCARSA1024) 
       Sequence
        UTCTime(210328172922GMT+00:00) 
        UTCTime(231223172922GMT+00:00) 

Other AMS blog posts

• Overview and challenges of using end to end message encryption (AMS).
• Certificate and keyring management for AMS.
• Installing AMS on z/OS.
• What certificates do I need for AMS with a client going to z/OS?
• checkAMS: program to check your AMS defintions are consistent with z/OS keyring
• Adding more users to AMS.
• Installing AMS on midrange for end to end message protection
• Understanding setmqspl to set up AMS definitions.
• How to administer AMS policies, and use the set policy command.
• Brain check and reset, needed for AMS keystores on midrange.
• Using runamscred to encrypt the passwords in the keystore.conf file.
• How do I find the recipients and signer of an AMS message?
• I have a message on the AMS DLQ – what can I do about it? and How do I process messages on the dead letter queue (DLQ)?

Having got a basic AMS set up working (on z/OS), it takes a bit of planning (and getting your head round it) to add more users and get the certificates in the right place when you have more than one queue manager. For example for someone to get an encrypted message the putting userid on the remote queue manager needs the getter’s public certificate. If there are 1000 potential putters, you start to see the complexity of the problem.

As I have tried to implement AMS across my baby enterprise I’ve realised you need a process for maintaining the key stores. When using multiple queue managers you have to remember which queue manger needs which options on the setmqspl command.

Basic key store management

z/OS key rings

On z/OS, each local userid has its own private key on its own keyring, with all of the shared public keys on the xxxxAMSM userid’s keyring (userid/drq.ams.keyring). Update the xxxxAMSM’x keyring, and every one picks up the change – great – easy.

You can have multiple queue managers in a single z/OS image or a sysplex, each with the xxxxAMSM address space. If they use the same started task userid, they will share the same keyring. (So to have queue managers use a different keyring, they will need a different userid.)

If any of the users are outside of the RACF key ring, you may need to export public certificates and send them to other environments: different sysplexes, or different midrange machines. The certificate will need to be imported on these other machines.

Midrange key stores

With midrange, a userid needs a key store with the private and public keys. With multiple machines you need a process to update and distribute key stores. For example:

• Create a key store in a central site. Do all maintenance to this key store, adding and removing certificates as required.
• Securely distribute this to all machines that are using AMS.
• The key store which has been distributed may have the public key for the user. You need to remove it, from the downloaded keystore and add the private key from the local machine.
  • runmqakm -cert -delete -db /home/colinpaice/.mqs/shared.kdb -type kdb -pw passw0rd -label COLIN
  • runmqakm -cert -export -db $private.kdb -pw passw0rd -label COLIN -target COLIN.key.p12 -target-type pkcs12 -target_pw zpassword
  • runmqakm -cert -import -target /home/colinpaice/.mqs/shared.kdb -target_type kdb -pw zpassword -file COLIN.key.p12 -type p12
  • when you import the private key you can use -label COLIN -new_label COLIN2 to give it a new label.
  • You have to make the key the default using runmqakm -cert -setdefault (even though this is deprecated), because without this, the first certificate in the keystore is used..
• Rename the shared key stores; current to old, new to current.
• Restart the client.
• Check it all works, delete the old key store.

If you add a new user, you may need to send the user’s public key to the cental site, to get it distributed to the mid range machines, and send it to the z/OS systems to add to the xxxxAMSM key rings.

In my case I just used one client, and updated the client’s key store directly.

Adding a new user for putting messages to a queue

Signed messages

When the data is signed, the recipient needs:

• the Certificate Authority of the putter (the signer) – which the recipient may already have
• or, when the putter has used a self signed certificate, the recipient needs the public part of the putter’s self signed certificate.

When using a CA signed certificate you may not need to export anything if the receiving end already has the CA certificate in the key store or keyring.

To export and import the Certificate authority.

• To export a CA certificate from z/OS, certificate see here, the certificate owner will typically be CERTAUTH
• To export a putter’s mid range CA certificate, see here.
• To import a CA certificate on z/OS, the certificate owner could be a userid, site, or CERTAUTH (check with your security administrator). You might not have authority to use SITE or CERTAUTH.
• To import a CA certificate on mid range, see here.

When using a self signed certificate (not a good idea for production)

• export the putter’s public certificate
• import it into the midrange key stores
• import it to any other z/OS systems, and add it to the xxxxAMSM keyring.

You can use

• To export a putter’s z/OS certificate see here.
• To export a putter’s mid range certificate, see here.
• To import a certificate on z/OS for a getter, see here. Once the certificate has been imported into the xxxxAMSM keyring it is available to all uses of the queue manager, and does not need to be imported again.
• To import a certificate on mid range for a getter, see here.

Specify the list of authorised DNs

At the getting end, if the setmqspl -a option is used to specify the list of authorised signers, then the DN from the putter needs to be added to the list of authorised signers (-a…) . You should use dspmqspl -m mq -p q -export to get the current definition for the queue and use setmqspl to add the new -a data to that.

On z/OS, when you have used setmqspl to update the AMS configuration, you need to use the F xxxxAMSM,REFRESH ALL command.

Putting encrypted messages

The putter needs a private key in the keystore or key ring – though this is not used when encrypting messages.

At the putting end, the queue needs to be configured with the list of recipient DNs setmqspl … -r … .

The public certificates for these DNs need to be in the putter’s xxxxAMSM keyring, or in the mid range user’s key store.

• To export the recipients’s z/OS certificate see here. You could store this in a central file so you can just reuse it rather than having to export it every time.
• To export the recipients’s mid range certificate, see here. You could store this in a central file so you can just reuse it rather than having to export it every time.
• To import the certificate on z/OS for the putter, see here. You should only need to do this once per keyring and user.
• To import the certificate on mid range for the putter, see here. This is where you need a process for a centralised key store containing all of the public keys your organisation needs, and distributing this to all of the users. You may have a cms key store (for C programs) and a JKS key store for Java programs.

Adding a new user for getting messages from a queue

Getting signed messages

If other userids are getting signed messages from this queue there may be no additional configuration on the getting queue manager.

If you are using setmqspl … -a … see Specify the list of authorised DNs above. It may already be set if other userids are configured to get messages from this queue.

When using a CA signer certificate you may not need to export anything if the receiving end already has the CA certificate in the key store or keyring.

If the getting end does not have the CA see To export and import the Certificate authority above.

If the putter is using self signed, see When using a self signed certificate (not a good idea for production) above.

Getting encrypted messages

The getter’s private key is used to decrypt the message. Each putter needs to have access to the public key of the getter.

On the system doing the MQPUT,

• the recipient’s DN needs to be configured on the putting queue manager using setmqspl with the -r (recipient) option for the queue
• each user doing the MQPUT will need access to the recipient’s public certificate in its key store or the xxxxAMSM keyring.

Specify the list of recipient DNs

On the putting queue manager you need to add the recipient to the list of the recipients for message. Use the command dspmqspl -m QMA -p AMSQ -export to display the existing configuration for the AMSQ queue, and add the DN of the new user to the list using the setmqspl command.

You need to do this on all squeue managers which can be putting to this queue manager

On z/OS, when you have used setmqspl to update the AMS configuration, you need to use the F xxxxAMSM,REFRESH ALL command.

Get the recipients public certificate in the putters’ keyrings and key stores.

• To add a z/OS user’s public certificate to the xxxxAMSM’s keyring in the same RACF database, see here.
• To export the z/OS certificate from a recipient, see here.
• To export the mid range certificate from a recipient, see here.
• To import a certificate for putters on z/OS, see here.
• To import a certificate for putters on mid range, see here.

Other AMS blog posts

• Overview and challenges of using end to end message encryption (AMS).
• Certificate and keyring management for AMS.
• Installing AMS on z/OS.
• What certificates do I need for AMS with a client going to z/OS?
• checkAMS: program to check your AMS defintions are consistent with z/OS keyring
• Adding more users to AMS.
• Installing AMS on midrange for end to end message protection
• Understanding setmqspl to set up AMS definitions.
• How to administer AMS policies, and use the set policy command.
• Brain check and reset, needed for AMS keystores on midrange.
• Using runamscred to encrypt the passwords in the keystore.conf file.
• How do I find the recipients and signer of an AMS message?
• I have a message on the AMS DLQ – what can I do about it? and How do I process messages on the dead letter queue (DLQ)?

I started writing up how to use AMS, and found I had written a lot about how to manage certificate and key rings from a z/OS perspective. It made more sense to write this up in its own blog post.

When using AMS, you may have to export public certificates to other z/OS images, and to mid range machines.

Basic key store management

z/OS key rings

On z/OS, each local userid has its own private key on its own keyring, with all of the shared public keys on the xxxxAMSM userid’s keyring. Update the xxxxAMSM’x keyring, and every one picks up the change – great – easy.

The certificate used to sign is the default certificate in the user’s keyring.

You can have multiple queue managers in a single z/OS image or a sysplex, each with the xxxxAMSM address space. If they use the same started task userid, they will share the same keyring. (So to have queue managers use a different keyring, they will need a different userid.)

Midrange key stores

With midrange, a userid needs a key store with the private and public keys. With multiple machines you need a process to update and distribute key stores. For example:

• Create a key store in a central site. Do all maintenance to this key store, adding and removing certificates as required.
• Securely distribute this to all machines tat use AMS.
• On each user’s machine add the user’s private key to this new copy of the key store.
• Rename the key stores; current to old, new to current.
• Restart the client.
• Check it all works, delete the old key store.

In my case I just used one client, and updated the client’s key store directly.

A user can point to different key stores, so you need keep in mind that it is the user’s certificate, not the userid that does signing, encryption etc..

When do I need to export a public certificate?

Signing

• If the certificate is self signed, the recipients will need a copy of the public certificate
• If the certificate is signed by a Certificate Authority the recipient needs a copy of the CA certificate, and may already have it.

As a recipient of encrypted data

• Each sender needs the public certificate (and public key) of each recipient, to be able to encrypt the payload.

Connecting the certificate in the same RACF database.

If the xxxxAMSM keyring is in the same RACF database as the the user’s certificate you just need to connect it to the keyring

RACDCERT ID(START1) connect(ID(ADCDC) LABEL(‘AMSZ’) –
RING(drq.ams.keyring) USAGE(SITE))

Exporting a certificate from a RACF database.

You need to export the certificate to a dataset. You can use binary format, or base64 encoded. I use base64 encoded.

RACDCERT ID(ADCDC) EXPORT(LABEL(‘AMSZ’))-
DSN(‘ADCDC.AMSZ.PEM’) FORMAT(CERTB64) PASSWORD(‘password’)

This file needs to be sent to the other systems and imported into the key stores.

Import a certificate into z/OS

For z/OS you need to import the public certificate to a userid. You might have a userid just for these certificate, or a userid for enterprise (BANK1, BANK2) etc. I’ve used ADCDA below.

The following example uses a certificate (for ja2) sent from a Linux machine. It was FTPed to z/OS into a data set ADCD.JA2.PEM.

#delete any old instance
RACDCERT ID(ADCDA) DELETE (LABEL(‘LINUXJA2’))

RACDCERT ADD(‘ADCD.JA2.PEM’) –
ID(ADCDA) WITHLABEL(‘LINUXJA2’) TRUST

RACDCERT ID(START1) CONNECT(RING(drq.ams.keyring ) –
ID(ADCDA) LABEL(‘LINUXJA2’))

RACDCERT LIST (LABEL(‘LINUXJA2’)) ID(ADCDA)

SETROPTS RACLIST(DIGTCERT) REFRESH

Extract a certificate from a mid-range key store

To extract from a CMS key store.

runmqakm -cert -extract -db my.kdb -pw passw0rd -label COLIN -target LINUXID.pem -format ascii

To extract from a .jks key store

runmqckm -cert -extract -db my.jks -pw zpassword -label COLIN -target LINUXID.PEM -format ascii -type jks

FTP the file to the remote systems and import it.

Importing a certificate into a mid-range key store

Follow the key store update process to add it to each midrange key store. The certificate was exported from z/OS and FTPed to the Linux machine as file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem

For the .jks key store (used by Java).

/opt/mqm/bin/runmqckm -cert -add -db trust.jks -type jks -file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem -label zADCDC -pw zpassword

#It is worth checking the certificate to make sure the certificate chain is present.

/opt/mqm/bin/runmqckm -cert -validate -db trust.jks -type jks -pw zpassword -label zADCDC

For a CMS keystore

/opt/mqm/bin/runmqckm -cert -add -db $key.kdb -file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem -label zADCDB -pw passw0rd

Other AMS blog posts

• Overview and challenges of using end to end message encryption (AMS).
• Certificate and keyring management for AMS.
• Installing AMS on z/OS.
• What certificates do I need for AMS with a client going to z/OS?
• checkAMS: program to check your AMS defintions are consistent with z/OS keyring
• Adding more users to AMS.
• Installing AMS on midrange for end to end message protection
• Understanding setmqspl to set up AMS definitions.
• How to administer AMS policies, and use the set policy command.
• Brain check and reset, needed for AMS keystores on midrange.
• Using runamscred to encrypt the passwords in the keystore.conf file.
• How do I find the recipients and signer of an AMS message?
• I have a message on the AMS DLQ – what can I do about it? and How do I process messages on the dead letter queue (DLQ)?

Every year the security boundaries are being pushed to protect more resources. For example

• Encrypt DNS lookup requests. Without this, bad guys can see what sites you are looking at. Or, depending on your view point, good guys can see what sites a bad guy is looking at.
• Encryption of cookies in http traffic. In https, cookies are encrypted, but some sites just use https for the logon, and then use http for normal traffic.
• Padding of strings, so the length of data cannot be seen. If you send up an encrypted password and the length is 4 characters, you can guess that the password is short.

MQ CCDT

The CCDT in MQ defines parameters for a client connection, it contains, for example,

• The connection host address and port
• The cipher specification
• The certificate label to use
• The certificate peer name (what is in the certificate sent from the server)

You may want to protect this information, so bad guys cannot intercept and change it.

The CCDT can be stored in the local file system, or it can be accessed from the network using http, or ftp.

This file will need to be updated for example, when there are changes to the configuration of the back-end servers, or the cipher spec is being upgraded.

Local file system.

You need to be careful how you update the local file system. Most enterprises have technology which securely distribute updates to client machines, for example with operating fixes, application upgrades, and configuration information.

Emailing a file, or simple FTP may not be secure enough for some organisations. Secure FTP (SFTP) or FTPS may be good enough.

Getting the CCDT from the network

Using HTTP and having the CCDT on a central site has many advantages. You update it, and the next time the client starts, it will phone home, and get the updated CCDT. You may want to use a secure site so only production machines can access the production CCDT files, and only test machines can access the test CCDT files.

The MQ documentation has

Unauthenticated connections

export MQCHLLIB=ftp://myhost.sample.com/var/mqm/qmgrs/QMGR/@ipcc
export MQCHLLIB=http://myhost.sample.com/var/mqm/qmgrs/QMGR/@ipcc
export MQCHLLIB=file:///var/mqm/qmgrs/QMGR/@ipcc

Authenticated connections

export MQCHLLIB=ftp://myuser:password@myhost.sample.com/var/mqm/qmgrs/QMGR/@ipcc
export MQCHLLIB=http://myuser:password@myhost.sample.com/var/mqm/qmgrs/QMGR/@ipcc

And says : If you want to use authenticated connections you must, as with JMS, provide the user name and password encoded in the URL.

To me these are not very secure, as they could be intercepted, and you could be sending userid and password on an potentially unencrypted connection.

In my Java program I managed to use https://127.0.0.1:9444/myccdt.json, to get the CCDT from my little Python https server. The url can be http, https, ftp, or file, so the only two “secure” ones are https: and file:, but the MQ documentation does not mention https.

I could not get a C program to work when I used export MQCCDTURL=”https://127.0.0.1:9443/myccdt.json”. It hung during the TLS handshake.

To make your CCDT secure, you are stuck with getting the file securely to the machine and using export MQCCDTURL=”file:///…”, which means it will updated whenever your machine gets the automatic updates – perhaps once a week.

Remember you can never be too paranoid when it comes to security. If someone steals your wallet or purse you know. If someone copies your identity file you may never know.

At first glance, how difficult can it be to understand a command with only five operands? As I do not usually write about easy topics, you may be able to guess that there is some subtlety to it.

Background to AMS protection

There are three protection models in AMS. See the IBM Knowledge centre.

1. Integrity.
2. Encryption.
3. Privacy = Encryption and Integrity.

Integrity

A checksum is made of the message contents, and the check sum is encrypted with the senders private key. The sender’s public key, the encrypted checksum, and the original payload are put into a message and put to a queue. This can be sent to a remote queue manager, or remain on the local queue manager. The recipient, performs the same checksum calculation on the original payload, decrypts the embedded checksum value (using information from the public certificate). If the checksums match – the messages is unchanged we have integrity

With Integrity, the getting queue manager knows the DN of the userid doing the signing.

With integrity, you need to specify which checksum technique to use, and at the getting end, you can optionally specify a list of authorised senders, and AMS will check the DN of the sender is in the list.

Confidentiality

The message contents are encrypted with a symmetric key. The key is then encrypted for the list of recipients, using the each recipient’s public key. A package of [(recipient, encrypted key) (recipient, encrypted key).. encrypted message] is put to a queue. When the message is got, the getter locates the (recipient, encrypted key) for it, uses its private certificate to decrypt the key to the data, and uses this to decrypt the message content.

With encryption you need to specify which encryption technique to use, and the list of recipients. For each recipient, AMS goes to the key store at the putting end to get the public certificate.

Privacy

A combination of Integrity and Confidentiality. The data is encrypted, and signed. The getting queue manager knows the DN of the sender.

setmqspl syntax

The syntax is

• setmqspl
• -m name queue manager name
• -policy name the queue name – known as the policy name
• -e value which cipher spec to use – or none
• -r DN a list of 1 or more recipient’s DN
• -s value the signing algorithm to use or none
• -a DN a list of 0 or more acceptable signers DN values (the putter’s DN).

I’ll focus on the -e, -r, -s and -a values.

Looking at a distributed MQ environment, should make it clearer

We have a putter queue manager. Messages go to another queue manager (or stay on the same queue manager) where the messages are got.

Integrity

At the putting queue manager you need

• setmqspl -m qmname -policy queuename -s algorithm
• Pick an algorithm from MD5, SHA1, SHA256, SHA384, SHA512, where the SHAnnn are more secure (it is all relative) but MD5 may be faster. The algorithm you pick will be used.
• -r and -a values are ignored from an integrity and signing perspective

At the getting end you need

• setmqspl -m qmname -policy queuename -s algorithm <-a DN… >
• You can specify any algorithm, (as long as the value is not NONE). This says expect a signed payload. The value is not used.
• The -a DN1 -a DN2… the optional list of values are the authorised DNs from the certificate of the userid that put (and signed) the message. If none are specified, then no checking is done. If there is at least one specified, then the signer’s DN is checked. If it is in the list, the message is returned to the application, if it is not in the list, the message get is rolled back, or put on the SYSTEM.PROTECTION.ERROR.QUEUE queue.

Confidentiality

At the putting end you need

• setmqspl -m qmname -policy queuename -e algorithm -r DN… .
• Pick a digital encryption algorithm from RC2,DES,3DES,AES128,AES256 , where the AESnnn are better than the others. The algorithm you pick will be used.
• The -r value is the DN of a potential recipient. The payload key is encrypted for each DN in the list.

At the getting end you need

• setmqspl -m qmname -policy queuename -e algorithm .
• You can specify any algorithm (as long as the value is not NONE), then this says expect an encrypted message. The value is not used.
• Any -r DN values will be ignored, as they are not relevant. The recipients are identified by the presence of the encrypted key in the payload.

Privacy

See above under Integrity and Confidentially

Specifying a DN

The documentation says

• DN attribute names must be in uppercase.
• Commas must be used as a name separators.
• To avoid command interpreter errors, place quotation marks around the DNs.
• The attribute values in the DN are case sensitive so, for example, CN=USERID1 is different from CN=userid1.

There is a convention (not a standard as such) that there is a hierarchy in the terms, for example starting with CN= on the left, and C= on the right. Some key stores, such as RACF will arrange the parts of the DN into this order.

AMS allows blanks between parts, for example “CN=COLIN , O=SSS”. These padding blanks are not present in certificates, but significant blanks such as within “CN=COLIN PAICE” are kept.

When trying to resolve AMS “missing certificates” problems, I found it easier when I had the parts of the DN in the right order, and did not have extra blanks between terms. It means I could just cut and paste the DN when searching in the key store.

Other AMS blog posts

• Overview and challenges of using end to end message encryption (AMS).
• Certificate and keyring management for AMS.
• Installing AMS on z/OS.
• What certificates do I need for AMS with a client going to z/OS?
• checkAMS: program to check your AMS defintions are consistent with z/OS keyring
• Adding more users to AMS.
• Installing AMS on midrange for end to end message protection
• Understanding setmqspl to set up AMS definitions.
• How to administer AMS policies, and use the set policy command.
• Brain check and reset, needed for AMS keystores on midrange.
• Using runamscred to encrypt the passwords in the keystore.conf file.
• How do I find the recipients and signer of an AMS message?
• I have a message on the AMS DLQ – what can I do about it? and How do I process messages on the dead letter queue (DLQ)?