I have a message on the AMS DLQ – what can I do about it?

If AMS has problems with a protected message, AMS can put the message on the SYSTEM.PROTECTION.ERROR.QUEUE queue. This blog post discusses what you can do about it. I consider this a hard problem – not in the same league as trying to simulate the beginnings of the Universe – more like climbing Ben Nevis mountain in Scotland, when you are only used to walking down to the shops.

What are the problems?

There are several problems you need to consider

1. Why is the message on the queue? Is it a problem with the putting application, or with the getting environment?
2. Which user had the problem. For example it may not be obvious which application instance had a problem, if applications come in through one channel, and many users have the same MCA userid.
3. What you need to do about it to get the message reprocessed, and prevent future problems.

Why is the message on the queue

The messages could be on the queue because

• The certificate was signed, but the DN of the signer is not in the setmqspl list of authorised signers (-a). This is an example of an MQ configuration problem
• The user getting a message was not able to verify the signers certificate sent in the message, for example it is missing the CA of the signer, or missing the signers self signed certificate. This is an example of a user’s configuration problem.
• The message was encrypted, but the user getting the message did not have access to a private certificate to allow the message to be decrypted. The user’s DN needs to be added to the recipients when the message is put and enciphered (or the user needs to be stopped from getting messages from this queue). This is an example of the putting of the message message is missing configuration information.

What other information is there to help me?

If you know which id had the problem, there should be information in the error logs. For example a problems within a Java JMS client program may write to mqjms.log.0. A local application may write to the queue manager’s error log, for example /var/mqm/qmgrs/QMA/error/*01*

In the mqjms.log.0 I got

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.prot.MessageProtectionBCImpl
The receiver of this encrypted message is not on the message recipient list ‘CN=ja2,O=aaaa,C=GB’
The certificate of a user that is receiving a message is not on the message RecipientsInfo list.

Verify that the user is on a recipients list in a security policy definition.

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.intercept.JmqiGetInterceptorImpl
The IBM MQ Advanced Message Security Java interceptor failed to unprotect the received message.
An error occurred when the IBM MQ Advanced Message Security Java interceptor was unprotecting the received message.

See subsequent messages in the exception for more details about the cause of the error

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.service.EseMQServiceImpl
The IBM MQ Advanced Message Security interceptor has put a defective message on error handling queue ‘SYSTEM.PROTECTION.ERROR.QUEUE ‘.

(On z/OS the messages are less helpful.)

On the SYSTEM.PROTECTION.ERROR.QUEUE queue there was a message with a Dead Letter Header (DLH) with a reason 2063 0x0000080f MQRC_SECURITY_ERROR.

From the MQMD you can see the time the message was put to the queue, the putting application, and the user identification. This userIdentified may have been set for example by the channel MCAUSER, or CHLAUTH, and so you do not always know where the original request came from.

If you are testing then you will know what caused the error.

If you are in a production like environment, you know the application, and as you will have configured all user keystores the same you may not need to know which specific user caused the problem. If there is a problem with a missing certificate, then you fix the problem, redeploy the keystore to all your users (as part of your automated process) and try again.

How do you tell what the problem is.

Your systems administrator needs a process for extracting meta data about messages, while keeping the application payload protected. You could build a process around displaying the recipients and signer from How do I find the recipients and signer of an AMS message? The systems administrator needs to know

• the original queue name with the problem
• the time, date, and user that put the message
• the msgid and correlid of the message – so if you put it back on the queue, you know which message to process.
• from the message, the type of protection: Integrity (it was signed), Encryption (it was encrypted), Privacy (it was encrypted with a signed payload)
• any id information from the message, such as recipient DN’s, and the signer DN. See this post.
• you may have to have some special processing to decrypt the Privacy payload, just to extract the signer information.

If this process can be automated, then any application content can be kept secure.

With this information and your “up to date application work book” (do you have one of these?) , you should be able to identify the problem.

Once you have fixed the root cause of the problem….

The fix may be to change the setmqspl to add an authorised signer, or to add a certificate to the recipients keystore, you need to get the message reprocessed.

• You get the specific message from the message id and correlid.
• You need to remove the DLQ header from the message.
• You need some special set up for the queue. If you try to put to the original queue, it will get the AMS protection again, for example re-encrypted or resigned. You need a queue alias so you put to the queue alias bypassing any AMS processing.

There are lots of things you need to consider, which is why I consider this a hard problem.

This application would be a good example where message handle is used to “move” the message.

MQCRTMH(hConn,&cmho,&hMsg,&CompCode,&Reason);

gmo.MsgHandle = hMsg;

MQGET(hConn,….);

pmo.Action = MQACTP_FORWARD;

pmo.OriginalMsgHandle =hmsg

MQPUT(…)

I found Learn to code the MQ Message Property MQI calls from MQGEM software useful in understanding message handle and message properties, and how to delete the DLQ header.

Other AMS blog posts

• Overview and challenges of using end to end message encryption (AMS).
• Certificate and keyring management for AMS.
• Installing AMS on z/OS.
• What certificates do I need for AMS with a client going to z/OS?
• checkAMS: program to check your AMS defintions are consistent with z/OS keyring
• Adding more users to AMS.
• Installing AMS on midrange for end to end message protection
• Understanding setmqspl to set up AMS definitions.
• How to administer AMS policies, and use the set policy command.
• Brain check and reset, needed for AMS keystores on midrange.
• Using runamscred to encrypt the passwords in the keystore.conf file.
• How do I find the recipients and signer of an AMS message?
• I have a message on the AMS DLQ – what can I do about it? and How do I process messages on the dead letter queue (DLQ)?