Which came first, the chicken or the checksum.

The ability to sign Java jar files, or z/OS modules, has been around for many years. Using this the loader checks the digital signature in the object. This digital signature is a checksum of the object, and this checksum is encrypted and stored with the object. At load time, the loader calculates the checksum, and decrypts the checksum in the object and checks they match.

MQ now supports this for some of its objects; downloadable .zip, .tar and .gz files.

For some of these you need to download the public key to use. This raises the problem that an evil person may have taken the object, removed the official signing information, and added their own stuff. You then download their public certificate – see it works, it must be official.

To prevent this you can do the checksum on the public certificate, and make that available along with the official public key. (This is the chicken and egg problem. You need the certificate to be able to check the main code, and how to you check the certificate, without a certificate to check?)

On Linux you can calculate the checksum of a file using

sha256sum 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz

this gives 53c34cd374d7b08522423533ef2019b4aa0109a595fbaeab8ee6f927cb6c93ad, which is the same as the value on the IBM site. So this matches.

The IBM MQ code signatures page says IBM MQ public certificates, checksums, and .sig files are available from https://ibm.biz/mq92signatures. On this signatures page it says

release level: 9.2.4.0-IBM-MQ-Sigs-Certs 
Continuous Delivery: 9.2.4 IBM MQ file signatures, checksums and certificates

Platforms:  AIX 64-bit, pSeries, Linux 64-bit,x86_64, Linux 64-bit,zSeries, Linux PPC64LE, Windows 64-bit, x86, z/OS

This page is an httpS page, with the certificate issued by a proper Certificate Authority, and trusted third party. If you trust this CA, you can trust the IBM page.

When you click download, it downloads

  • 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz.sha256sum – this file content has 53c34cd374d7b08522423533ef2019b4aa0109a595fbaeab8ee6f927cb6c93ad
  • 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz

The value in the sha256sum file matches the value of the sha256sum 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz command.

As you can trust the security chain from the web page, through to the downloads, you can trust the .gz file.

Jar signing

Java has had the capability to sign a jar for at least 10 years.

The jarsigner command takes a jar file, a keystore with private key and calculates the checksum. It then encrypts it, and creates some files in the jar. For example

jarsigner -keystore trust.jks -storepass zpassword checkTLS.jar signer

This uses

  • the keystore called trust.jks,
  • with password zpassword
  • the checkTLS.jar file
  • and uses the certificate with alias name signer. This certificate must have extendedKeyUsage with codeSigning.

The jar file now has some additional files which can be seen using jar -tvf checkTLS.jar command.

  • META-INF/SIGNER.SF . This is the Signature File.
  • META-INF/SIGNER.EC .This is the public key to be used.

Where SIGNER is the name of the alias of the private key in the keystore, used to sign the jar file. The jar file can be signed many times by different private keys.

To verify the signature you can use

  • jarsigner -verify checkTLS.jar
  • jarsigner -verbose -certs -verify checkTLS.jar

The jarsigner -verbose -certs -verify checkTLS.jar gave me

- Signed by "CN=signer, O=cpwebuser, C=GB"
    Digest algorithm: SHA-256
    Signature algorithm: SHA256withECDSA, 256-bit key

jar verified.

Warning: 
This jar contains entries whose certificate chain is invalid. Reason: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
This jar contains signatures that do not include a timestamp. Without a timestamp, users may not be able to validate this jar after any of the signer certificates expire (as early as 2024-11-03).

The signer certificate will expire on 2024-11-03.

This shows that the jar file is consistent with the checksumming, but the certificate cannot be validated.

I can tell it which keystore to use to validate the certificate, using

jarsigner –keystore trust.jks -certs -verify checkTLS.jar

With the -verbose option you also get (with some of the output rearranged for clarity). The “s” or “sm” at the front of an object entry is s=signature verified, and m=entry listed in the manifest.

s = signature was verified 
m = entry is listed in manifest
k = at least one certificate was found in keystore
i = at least one certificate was found in identity scope


s 1402 Wed Dec 22 14:27:52 GMT 2021 META-INF/MANIFEST.MF

  >>> Signer
  X.509, CN=signer, O=cpwebuser, C=GB
  [certificate is valid from 22/12/21 14:51 to 30/01/25 16:46]
  X.509, CN=SSCA256, OU=CA, O=SSS, C=GB
  [certificate is valid from 04/11/21 15:48 to 03/11/24 15:48]

  [Invalid certificate chain: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

        1579 Wed Dec 22 16:12:04 GMT 2021 META-INF/SIGNER.SF
        1373 Wed Dec 22 16:12:04 GMT 2021 META-INF/SIGNER.EC

sm  54 Sat Jan 30 14:48:52 GMT 2021 checkTLS/checkTLS.manifest

  >>> Signer
  X.509, CN=signer, O=cpwebuser, C=GB
  [certificate is valid from 22/12/21 14:51 to 30/01/25 16:46]
  X.509, CN=SSCA256, OU=CA, O=SSS, C=GB
  [certificate is valid from 04/11/21 15:48 to 03/11/24 15:48]

  [Invalid certificate chain: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

When I downloaded the MQ 9.2.4 client and ran the jarsigner …. -verbose command the output included

sm 642 Thu Nov 04 16:01:46 GMT 2021 wlp/lib/extract/IFixUtils$ParsedIFix.class

[entry was signed on 04/11/21 17:58]
>>> Signer
X.509, CN=International Business Machines Corporation, OU=IBM CCSS, O=International Business Machines Corporation, L=Armonk, ST=New York, C=US
[certificate is valid from 25/08/21 01:00 to 26/08/23 00:59]
X.509, CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1, O="DigiCert, Inc.", C=US
[certificate is valid from 29/04/21 01:00 to 29/04/36 00:59]
      X.509, CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US
[trusted certificate]
... 

This shows that the certificate used to sign the component of the jar file was signed by CN=International Business Machines Corporation, which was in turn signed by CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1. The jarsigner program was able to use its public certificate to validate the CA, and so validate the IBM certficiate, and so validate the checksum.

Colin’s list of MQ messages

This blog post is my annotations to MQ messages, containing descriptions of what I did to get an MQ message, and what I did to fix the problem. It is meant for web search programs, rather than humans.

I will extend it as I experience problems.

AMQ7026E: A principal or group name was invalid.

I could display an auth entry

dspmqaut -m qml -t qmgr -g “cn=dynamic, o=Your Company”


Entity cn=dynamic, o=Your Company has the following authorizations for object qml:
connect

but not delete it

setmqaut -m qml -t qmgr -g “cn=dynamic,o=Your Company” -connect

AMQ7026E: A principal or group name was invalid.

I had set this up using LDAP, but MQ was not able to find the record in LDAP. This is because I had changed the configuration. There was an LDAP search

(&(objectClass=groupOfNames)(OU=cn=dynamic, o=Your Company))

It could not be found because there was no LDAP entry with objectClass=groupOfNames with an entry (OU=cn=dynamic, o=Your Company)

I had changed

DEFINE AUTHINFO(MYLDAP) +
AUTHTYPE(IDPWLDAP) +
GRPFIELD(sn)

to

GRPFIELD(ou)

So when MQ came to delete it, it looked for

(&(objectClass=groupOfNames)(OU=cn=dynamic, o=Your Company))

instead of

(&(objectClass=groupOfNames)(SN=cn=dynamic, o=Your Company))

and could not delete it.

I changed the LDAP group to add the OU, and the SETMQAUT command worked.

AMQ5532E: Error authorizing entity in LDAP

EXPLANATION:
The LDAP authorization service has failed in the ldap_first_entry call while
trying to find user or group ‘NULL’. Returned count is 0. Additional context is
‘cn=mqadmin,ou=groups,o=your Company’.

Colin’s comments

I had defined a dynamic group, by specifying the group name as part of the LDAP user entry.

When I changed the authinfo object yo have nestgrp(yes), I got the above message because there was no record for cn=mqadmin,ou=groups,o=your Company’.

Define the record with the appropriate object class as defined by the MQ AUTHINFO attribute CLASSGRP. (CLASSGRP(‘groupOfNames’) in my case).

AMQ5530E: Error from LDAP authentication and authorization service

EXPLANATION:
The LDAP authentication and authorization service has failed. The
‘ldap_ssl_environment_init’ call returned error 113 : ‘SSL initialization call
failed’. The context string is ‘keyfile=”/var/mqm/qmgrs/qml/ssl/key.kdb”
SSL/TLS rc=408 (ERROR BAD KEYFILE PASSWORD)’. Additional code is 0.

Colin’s comments.

I had changed the keyring using alter qmgr CERTLABL(ECRSA1024) SSLKEYR(‘/home/colinpaice/mq/zzserver’)

AMQ5530E: Error from LDAP authentication and authorization service

EXPLANATION:
The LDAP authentication and authorization service has failed. The
‘ldap_simple_bind’ call returned error 49 : ‘Invalid credentials’. The context
string is ‘10.1.1.2:389 ‘. Additional code is 0.

Colin’s comments

An anonymous logon to LDAP was attempted (LDAPUSER and LDAPPWD omitted) and the LDAP server had allowAnonymousBinds off.

Specify userid and password.


Understanding display authrec match(…) on midrange.

I found the documentation for the display authrec match(…) command hard to understand. There are ambiguous backward references (the profile … which profile?), too many ‘and’s, and I think some ‘or’s are missing. Below is how I interpret it.

  • Match exact: Select the record where the specified profile name specified is an exact match for a setmqauth record
  • Match profile: Display the setmqaut records which would be used to compute permissions, for the specified profile, and (specified userid or specified group).
  • Match membership:
    • For the specified userid do Match: profile() for the userid
    • For each group the specified userid is in, do Match: profile() for that group

I see the processing is in two stages

  1. Stage 1 extract the autrecs for the specified profile name
  2. Stage 2 filter the list using the specified userid or group.

Stage 1: extract the authrecs matching the specified profile name.

For match exact

Select the record where the specified profile name specified is an exact match for a setmqauth record. Profile(‘CP.**.99’) will match only ‘CP.**.99’.

For match profile and membership

If the specified profile has a generic then treat this as match(exact).

If the specified profile has no generics then extract all records which would apply when checking this profile.

For example for queue CP.AA.BB.99 might return

  • profile CP.AA.BB.99 (entity colinpaice) – this userid created the queue
  • profile CP.AA.BB.99 (entity mqm) – this entry is create when the queue is created
  • profile CP.**.99 (entity testuser) – this was done via setmqaut – not selected because less specific generic profile
  • profile CP.*.BB.99 (entity testuser) – this was done via setmqaut.

Stage 2. Filter the records depending on the specified userid or group.

Take the set of records from stage 1 and filter them. You can specify the principal (userid) or group. Note: If your qm.ini has SecurityPolicy=group then even if you have specified you setmqauth with a userid, it will use a group instead. This may mean that displaying a userid may give no results.

Match exact

Compare the specified entity and entity type with those in the the records. If they match display the record.

Match profile

Compare the specified entity, and entity type with those in the the records. If they match display the record.

Match membership

Compare the specified entity, and entity type with those in the the records. If they match display the record.

If the entity type in the record is group, and the specified userid is a member of the group then display it.

Examples

Match(profile) principal()

dis authrec profile(CP.AA.99) objtype(queue) match(profile)   principal('testuser')
AMQ8459I: Not found.    

There is no profile defined for the userid ‘testuser’

Match(profile) group()

dis authrec profile(CP.AA.99) objtype(queue) match(profile)   group('testuser')
   PROFILE(CP.*.99)                        ENTITY(testuser)  

There was a setmqauth -m -n “CP.*.99” -t queue -p testuser +get . Because of the qm.ini setting, and userid authorisations were converted to group authorisations. On some Unix systems, when a userid is created, it creates a group with the same name, and connects the userid to the group.

dis authrec profile(CP.AA.99) objtype(queue) match(profile)   group('test') 
AMQ8864I: Display authority record details.
    PROFILE(**) ENTITY(test) AUTHLIST(DSP,INQ) 

All members of group test, have Display and Inquire permissions on any queue.

Match(membership) principal()

dis authrec profile(CP.AA.99) objtype(queue) match(membership) principal('testuser')
AMQ8864I: Display authority record details.
   PROFILE(CP.*.99)                        ENTITY(testuser)
                         
AMQ8864I: Display authority record details.
   PROFILE(**)                             ENTITY(test)  

The display match(membership) combines all of the above. Any specific records, plus records for any group testuser is in.

Match(membership) group()

dis authrec profile(CP.AA.99) objtype(queue) match(membership) group('testuser')
AMQ8864I: Display authority record details.
   PROFILE(CP.*.99)                        ENTITY(testuser)

Because the group is specified, then this acts the same as match(profile) group(‘testuser’).

Match(profile) no group nor principal

dis authrec profile(CP.AA.99) objtype(queue) match(profile)   
AMQ8864I: Display authority record details.
   PROFILE(CP.AA.99)                       ENTITY(colinpaice)
 
AMQ8864I: Display authority record details.
   PROFILE(CP.AA.99)                       ENTITY(mqm)
 
AMQ8864I: Display authority record details.
   PROFILE(CP.*.99)                        ENTITY(testuser)
                
AMQ8864I: Display authority record details.
   PROFILE(CP.**.99)                       ENTITY(testuser)
                 
AMQ8864I: Display authority record details.
   PROFILE(**)                             ENTITY(test)

No userids or groups were specified, so all relevant autrecs for the profile CP.AA.99 are displayed.

Observation: There is a profile for PROFILE(CP.**.99) ENTITY(testuser) which does not show up when dis authrec profile(CP.AA.99) objtype(queue) match(membership) principal(‘testuser’) is used.

This is because with generic profiles, only the most specific generic profile is used, see Profile Priorities.

How easy is it to display security information on midrange.

If you thought it was easy, I would disagree with you.

While investigating a security problem (why my code worked, when it should have got a not authorised return code) I struggled to answer questions like

There are links to my solutions for some of the problems.

Morag contacted me and said:

I wrote a little blog post about using DISPLAY ENTAUTH – hope it helps. See here:

Morag

What commands are there?

There are four MQ commands providing similar functions

Definition of terms

Profile name

This can be the name of a queue, or a string which is used for comparisons and can include single * or double **. For example

  • PAYROLL for a particular queue,
  • PAYROLL.*.INPUT, which would apply to objects PAYROLL.A.INPUT, PAYROLL.B.INPUT, but not PAYROLL.A.B.INPUT, nor PAYROLL.INPUT.
  • PAYROLL.**.INPUT which would apply to PAYROLL.A.INPUT, PAYROLL.B.INPUT, PAYROLL.A.B.INPUT, but not PAYROLL.INPUT.
  • ** used as a fall back if there are no other definitions.

Example profiles

  • setmqaut -m QMA -n PAYROLL -t queue -g testuser +put
  • setmqaut -m QMA -n ‘PAYROLL.*.INPUT’ -t queue -g testuser +put
  • setmqaut -m QMA -n ‘PAYROLL.**.INPUT’ -t queue -g testuser +put
  • setmqaut -m QMA -n ‘**’ -t queue -g MQAdmin +chg +dlt

On Unix you need the single quotes around any profile which includes *, to prevent the shell replacing it with a list of file names.

You need to be careful about quoting strings with mixed case data. dspmqaut and setmqaut do not need quoting. For runmqsc display|set authrec the mixed case fields need quoting.

A profile is created

  • by the setmqaut command,
  • the runmqsc set authrec command,
  • when the object is created. In which case, the queue manager creates an entry for the mqm group, and for the principal group of the user that created the object. Userid ibmsys1 created a queue, and there were entries for group mqm, and group zpdt. These definitions have full access. (The userid that created a queue can put and get messages from the queue – and you cannot separate administration of the queue, and usage of the queue.)

Userid and group

Unix has the concept of User Private Groups (UPG). A userid can have a group with the same name as the userid, which makes the security just a touch more confusing.

You can use the id command to list which groups a userid is in

id ibmsys1
uid=1001(ibmsys1) gid=1006(zpdt) groups=1006(zpdt),27(sudo),1007(MQAdmin)

Userid ibmsys1 with numeric id 1001 is a member of 3 groups: zpdt, sudo and MQAdmin.

You can specify the authority to be given to a userid – but it looks like the userid is not used, but the default group is used! See Explicit giving a userid access – does not give the userid access. This could just be a Linux feature.

Displaying information.

dspmqaut -m QMA -n ‘PAYROLL.**.INPUT’ -t queue -g testuser

This lists the information for the group testuser and the profile PAYROLL.**.INPUT’. It reports “put” access.

What access does the group ibmsys1 have?

dspmqaut -m QMA -n ‘PAYROLL’ -t queue -g ibmsys1

Entity ibmsys1 has the following authorizations for object PAYROLL:

 

 

Shows the group has no access defined.

What access does the userid (principal) ibmsys1 have?

dspmqaut -m QMA -n ‘PAYROLL’ -t queue -p ibmsys1

Entity ibmsys1 has the following authorizations for object PAYROLL:
dlt
chg

This is because userid ibmsys1 has MQAdmin as one of its group.

Adding -e option to the dmpmqaut command (Display all profiles used to calculate the cumulative authority that the entity has to the object specified in -n Profile) shows which profiles will be used.

dmpmqaut -m QMA -n ‘PAYROLL’ -t queue -p ibmsys1 -e
profile: **
object type: queue
entity: MQAdmin
entity type: group
authority: dlt chg

We can see that the profile used to determine the access permissions came from the “**” profile statement for group(entity) MQAdmin.

Explicit giving a userid access – does not give the userid access.

On my Linux Ubuntu machine, using the following command to explicitly give a userid access; gives it to the userid’s principal group.

setmqaut -m QMA -n PAYROLL -t queue -p ibmsys1 +put

dmpmqaut -m QMA -n PAYROLL -t queue -p ibmsys1 -e
profile: PAYROLL
object type: queue
entity: zpdt
entity type: group
authority: put

profile: **
object type: queue
entity: MQAdmin
entity type: group
authority: dlt chg

This shows the group zpdt got the access – not the userid.

For this userid we can see that there was a specific profile “PAYROLL” for the zpdt group giving put permission, and the generic profile for the MQAdmin group giving inquire and display permissions.

The display authrec command

If you use dis authrec profile(PAYROLL) it will display all possible profiles including the profiles for queues, process etc. Ensure all mixed case values are surrounded by single quotes.

dis authrec profile(PAYROLL) objtype(queue)

gave

AMQ8864I: Display authority record details.
  PROFILE(PAYROLL) ENTITY(colinpaice)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
  AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

AMQ8864I: Display authority record details.
  PROFILE(PAYROLL) ENTITY(mqm)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
  AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

AMQ8864I: Display authority record details.
  PROFILE(PAYROLL) ENTITY(testuser)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
 AUTHLIST(PUT)

AMQ8864I: Display authority record details.
  PROFILE(**) ENTITY(test) ENTTYPE(GROUP) OBJTYPE(QUEUE) AUTHLIST(DSP,INQ) 

AMQ8864I: Display authority   record details. 
  PROFILE(**) ENTITY(MQAdmin)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
  AUTHLIST(CHG,DLT)

but

dis authrec profile(PAYROLL) objtype(queue) group(MQAdmin)

gives not found however, using single quotes,

dis authrec profile(PAYROLL) objtype(queue) group(‘MQAdmin’)

gives

AMQ8864I: Display authority record details.
  PROFILE(**) ENTITY(MQAdmin)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
  AUTHLIST(CHG,DLT)

List all objects

You can use

runmqsc dis authrec type(queue)

to display all queue records with (up to three lines per entry) see above for an example.

You can use the dmpmqaut -l option Dump only the profile name and type. Use this option to generate a terse list of all defined profile names and types.

dmpmqaut -m QMA -l -t queue

Which gives output like

profile: PAYROLL, object type: queue
profile: PAYROLL.*.INPUT, object type: queue
profile: PAYROLL.**.INPUT, object type: queue

Unfortunately this is output to stderr, so you cannot immediately pipe this into grep or other commands.

You can use

dmpmqaut -m QMA -l -t queue 3>&1- 1>&2 2>&3|grep PAYROLL

The dmpmqaut command says you can use generics and wild cards in the name described Using OAM generic profiles on AIX®, Linux®, and Windows systems. I could not get this to work, for example using PAYROLL.**

The runmqsc DIS AUTHREC command says profile can have The name of the object or generic profile for which to display the authority records.

I think the documentation means “the string used in the profile definition, not a generic search argument.”


For example I wanted to use a qualifier ‘*’, but the command

dmpmqaut -m QMA -n ‘CP.*’

gives

No matching authority records.

runmqsc dis entauth – display the authorizations an entity has to a specified object

This very useful command was pointed out to me by Morag. It has been in MQ since MQ 7.5.

It allows you to ask what permissions a userid or group has to a specific resource. For example

A typical display

dis entauth principal(‘testuser’) objtype(queue) objname(CP0000)
AMQ8866I: Display entity authority details.
OBJNAME(CP0000) ENTITY(testuser)
ENTTYPE(PRINCIPAL) OBJTYPE(QUEUE)
AUTHLIST(BROWSE,CRT,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

Queue does not exist

dis entauth principal(‘testuser’) objtype(queue) objname(CP000ZZZZZZ)
AMQ8147E: IBM MQ object CP000ZZZZZZ not found.

Silly me – I missed the quotes off from around the userid


dis entauth principal(testuser) objtype(queue) objname(CP0000)
3 : dis entauth principal(testuser) objtype(queue) objname(CP0000)
AMQ8871E: Entity, principal or group not known.

Dump mq configuration

You can use the dmpmqcfg command to dump out the authrec records.

dmpmqcfg -m QMA -x authrec -o 1line

gives output like (where there is one line for each SET… statement).

SET AUTHREC PROFILE(‘CP.*.00’) GROUP(‘colinpaice’) OBJTYPE(QUEUE) AUTHADD(GET,PUT)
SET AUTHREC PROFILE(‘PAYROLL.**.INPUT’) GROUP(‘testuser’) OBJTYPE(QUEUE) AUTHADD(PUT)
SET AUTHREC PROFILE(‘@class’) GROUP(‘colinpaice’) OBJTYPE(QUEUE) AUTHADD(CRT)

The -o 1line option creates one line of output for each record. You can use platform tools to process the data, for example use grep to filter lines, and parse the parts between ‘ ‘. See an example python script at the bottom.

The documentation says -n [*|ObjectName] Filter the definitions produced by object or profile name, the object/profile name can contain a single asterisk. But it didnt work for me.

My basic queries

Is this user authorised to use this queue?

You can use

dspmqaut -m QMA -n PAYROLL.A -t queue -p ibmsys1

If the queue does not exist you will get AMQ7085E: Object PAYROLL.A, type queue not found.

Or you can use the runmqsc command

dis entauth principal(‘testuser’) objtype(queue) objname(CP0000)

What security profiles are involved with this user and that queue.

Or which profiles gave a user access to the queue?

Display the userids id and groups. If userid is in group mqm, it will get all access regardless of the setmqaut settings.

Display all the profiles involved with a userid -e … or group -g …

dmpmqaut -m QMA -n PAYROLL.A -t queue -e

Display all the permission a user has with a userid -p … or group -g …

dspmqaut -m QMA -n ‘CP.ZZ.00’ -t queue -p testuser
Entity testuser has the following authorizations for object CP.ZZ.00:
inq
crt
dsp

Display the profiles which determine the permissions for a user

dmpmqaut -m QMA -n ‘CP.ZZ.00’ -t queue -e -p testuser
profile: **
object type: queue
entity: test
entity type: group
authority: inq dsp

profile: @class
object type: queue
entity: test
entity type: group
authority: crt

The inq and dsp permissions came from the “**” profile for the test group.

Who is authorised to this queue?

This is a tricky one. I could find two hard ways of

dmpmqaut -m QMA -n ‘CP.ZZ.00’ -t queue 2> file

dmpmqcfg -m QMA -x authrec -o 1line > file

Then post-processing the output to extract group name and permissions, then merging the records.

For example to see the members of a group.

getent group |grep mqm

gives
mqm:x:1003:colinpaice,ibmsys1

I want to audit the access to the queues beginning with CP.

Another tricky one.

  • dmpmqcfg -m QMA -n ‘CP*’ -t queue -o 1line -x object > queuelist
  • This gives a list of all queues starting with CP. Process this list to select each queue in turn
  • dmpmqaut -m QMA -n $queuename -t queue -e > accesslist
  • This accesslist file has the profile name, the group and the permissions.
  • Take each group name, and extract the members of the group.
  • For each member of the list use
  • dspmqaut -m QMA -n $queue -t queue -p $userid
  • Take this list extract the permissions, and produce a pretty report.

If there is a better way I would love to hear it.

A partial solution is to take the output of

dmpmqaut -m QMA -l -t queue 2>list

dmpmqcfg -m QMA -x authrec -o 1line > file

Then use tools like grep to extract the records of interest, then use other tools to issue dmpmqaut -m QMA -n ‘….’ -t queue, capture the output and process it.

But this means you have to interpret generic profiles like ‘CP.**.INPUT’ yourself.

Sample code for post processing dmpmqcfg output

Below is a small Python program that takes the output from dmpmqcfg and processes it to replace groups with users in the groups, and gives one line per userid,permission,profile, source_group

#
# Copyright (c) 2020 Stromness Software Solutions.
# 
# All rights reserved. This program and the accompanying materials
# are made available under the terms of the Eclipse Public License v1.0
# which accompanies this distribution, and is available at
# http://www.eclipse.org/legal/epl-v10.html
# 
# * Contributors:
# *   Colin Paice - Initial Contribution
# 
from sys import stdin
import grp

for line in stdin:
  sline= line.strip()
  s = sline.split(' ')
  if s[0] != "SET":
     continue
  p = s[2][9:-2] # get the profile
  g = s[3][7:-2] # get the group 
  try:
    ginfo = grp.getgrnam(g)
    ids = ginfo.gr_mem 
    if len(ids) == 0:  # group can be empty
       ids = [g]  
  except KeyError:
    ids = [g] # group not found - must be userid
  t = s[4][8:-1] #type

  a = s[5][8:-1]  # get the authorities
  aa = a.split(",")
# Output userid authority profile type group_it_came_from
  for aaa in aa:
    for i in ids:
       print(i,aaa,p,t.g) 

Invoke it as

cat data | python3 ~/python/sec.py |sort -k1,3|uniq |grep -v mqm 

grep -v mqm removes all accesses due to group mqm membership

Example output

colinpaice SET SYSTEM.SELECTION.VALIDATION.QUEUE QUEUE 
colinpaice SET TEMP QUEUE colinpaice
colinpaice SET XQMB QUEUE colinpaice
colinpaice SUB SYSTEM.ADMIN.TOPIC TOPIC colinpaice
testuser PUT COLIN_* QUEUE test
testuser PUT CP0000 QUEUE testuser

How to administer AMS policies, and use the set policy command.

I had been using the setmqspl command (on z/OS and midrange) to manage my AMS policies. This command has the drawback that if you want to change a policy, for example add a new recipipient, you had to specify the whole command. Jon Rumsey pointed out the mid range MQSC commands “set policy” and “display policy” which allow you to add, delete, or replace; recipients and signers.

Examples of midrange runmqsc set policy command

Exporting parameters

If you want to keep a copy of the AMS definitions you can use display policy command, but this gives output like RECIP(CN=BBB,C=GB), without quotes. The set policy command needs the value within single quotes. The dmpmqcfg command does not support AMS policies.

To be able to capture the output so you can reuse it, you need to use the dspmqspl -export command. This gives output like

setmqspl -m QMA -p ABC -s SHA512 -e AES256 -r “CN=BBB,C=GB” -c 0 -t 0

This gives the parameters if a format that can be used directly.

Add or remove recipients or signers

Using runmqsc define a policy using the default action(replace)

set policy(ABC) signalg(SHA512) recip(‘CN=AAA,C=GB’)  ENCALG(AES256) 

You can add a new recipient

set policy(ABC) signalg(SHA512) recip(‘CN=BBB,C=GB’) ENCALG(AES256) action(ADD)

You can now display it

DIS policy (ABC)

AMQ9086I: Display IBM MQ Advanced Message Security policy details.
POLICY(ABC) SIGNALG(SHA512)
ENCALG(AES256)
RECIP(CN=BBB,C=GB)
RECIP(CN=AAA,C=GB)
KEYREUSE(DISABLED)
ENFORCE

You can delete a recipient

set policy(ABC) SIGNALG(SHA256) ENCALG(AES128) RECIP(‘CN=AAA,C=GB’) action(remove)

and display it

DIS policy(Abc)
AMQ9086I: Display IBM MQ Advanced Message Security policy details.
POLICY(ABC) SIGNALG(SHA512)
ENCALG(AES256) RECIP(CN=BBB,C=GB)


KEYREUSE(DISABLED)
ENFORCE

You have to specify SIGNALG and/or ENCALG each time, but for action(REMOVE|ADD) it can have any valid value (except NONE). The value is only used when ACTION(REPLACE) is used, or ACTION() is omitted. The following will add the recipient, and not change the signalg or encalg values.

set policy(ABC) recip(‘CN=CCC,C=GB’) action(ADD) signalg(MD5) encalg(RC2)

You can specify multiple RECIP

set policy(ABC) signalg(SHA512) recip(‘CN=BBB,C=GB’) recip(‘CN=DDD,C=GB’) ENCALG(AES256) action(ADD)

or multiple signers

set policy(ABC) signalg(SHA512) signer(‘CN=BBB,C=GB’) signer(‘CN=DDD,C=GB’) ENCALG(AES256) action(ADD)

or multiple signers and recipients.

Changing other parameters

If want to change an algorithm, the tolerate|enforce that every message must be protected, or the key reuse, then you must use the action(replace), and specify all the parameters, so it might be easier to use setmqspl -m … -policy … -export, and output it to a file, then modify the file.

Administering AMS on z/OS

On z/OS (and mid-range) you have dspmqspl and setmqspl commands. With the setmqspl command, you replace the entire statement.

It is good practice to have a PDSE with all of your definitions in, one member per policy, or perhaps all policies in one member – depending on how many policies you have. If you have a problem with your queue manager, you have a copy of the definitions.

Another good practice is to take a copy of a definition before you make the change (and keep it unchanged), so you can roll back to it if you need to undo a change.

You can use the export command, to output all policies, or a selected policy. You can have this going into a sequential data set or a PDSE member. You might want to have two copies,

  1. The before image – from before the change
  2. The copy you update.

Of course you could always use the previous copy, but you cannot tell if someone has updated the definitions outside of your change control system, so taking a copy of the existing definitions is a good idea. You could always compare the previous copy, with the copy you just created to check there were no unauthorised changes.

You may want to make the same change to multiple queue managers, so having updates in a PDSE member is a good way of doing it. Just change the queue manager name and rerun the job.

On z/OS, remember to use the refresh command on the AMS address space for it to pick up any changes.

Other AMS blog posts

How do I process messages on the dead letter queue (DLQ)?

I was setting up security on my system, and using AMS to protect messages. I kept getting messages on the Dead Letter Queues. As messages on the DLQ have been around from before MQ V1 was shipped (they hit this problem in development), I was expecting that to process them would be easy. There are some good bits and some not so good bits with the IBM supplied solution. I was reminded of a “call and response narration” game we enjoyed in the pub from when I was a student which went ..

They are building a house in the street – (audience) Boo!
A public house – (audience)Hooray!
They don’t sell beer – (audience) Boo!
They give it away – (audience) Hooray!

For a supplied Dead Letter Queue handler it goes…

MQ provides a Dead Letter Handler program (runmqdlq) – Hooray!
On z/OS (CSQUDLQH) and midrange (runmqdlq). – Hooray.
It is rule based and can handle many scenarios – Hoorary!
But not some of the difficult ones – Boo!
The provide a set of sample programs on mid range (amqsdlq) – Hooray!
But they are not well documented, didn’t build straight off, and not available on z/OS – Boo.
It can process many similar messages in one go- Hooray,
But not process just one message – Boo.

Why are messages put on the DLQ?

If a local application tries to put a message to a queue, and the queue is full then the application gets a return code, and takes an action. The message is not lost – it wasn’t created, and the DLQ was not used. If a message comes in from another queue manager, and the channel tries putting the message and gets queue full, it cannot just throw the message away. It puts it onto the DLQ.

Messages could be put on a DLQ for many reasons.

  • A message came in from a remote queue manager and was put to a local queue, but the queue was at max depth, so was put to the DLQ. This may be due to a short lived problem. The DLQ handler can process the DLQ queue, and every 60 seconds try moving the message from the DLQ back to the original. You can configure the rules so if it tries 5 times and fails, then it moves the message to a different queue.
  • A message came in from a remote queue queue manager, but the channel userid was not authorised to put to the queue. In this case retrying every 60 seconds is unlikely to solve the problem. The administrator needs to take an action, such as grant access and retry the put, or remove the message.
  • When AMS is used, if an ID tries to get the message and there are problems, such as the ID of the signer of the message is not authorised, the message is put to the SYSTEM.PROTECTION.ERROR.QUEUE queue. To resolve this, the AMS configuration needs to be changed, or the message moved to a quarantine queue. Once the configuration has been changed, put the message back on the queue for retry.

The runmqdlq handler provided with MQ

This is a bit of a strange beast. It is rule based so you can configure rules to select messages with certain properies and take actions, such as retry, or move to a different queue.

The program on midrange is runmqdlq, and on z/OS CSQUDLQH.

The syntax for runmqdlq is

runmqdlq [-u userid] MYDEAD.QUEUE QMA <qrule.rul

you have to pipe the file into stdin, until an empty line is processed. I would have preferred a -f filename option.

To end runmqdlq, set the input queue to get(DISABLED) because Ctrl-C does not work.

It processes message silently, unless there are any problems, for example I got

Dead-letter queue handler unable to put message: Rule 6 Reason 2035.

I had several problem messages on the DLQ, but I could not specify one message and get runmqdlq to process it, so I had to write a program to move one message to a different queue, then I could use runmqdlq. There is lots of good stuff in runmqdlq, but doesn’t quite do the job.

Understanding the rules.

The rules are the same for z/OS as mid-range.

Messages are read from the specified DLQ queue, and processed with a set of rules. The rules are described here. You can select on properties in the MQMD or the DLQ header. For example

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(RETRY) RETRY(5)

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(FWD) FWD(MYQUEUEOVERFLOW) HEADER(YES)

DEST(INQ*) PERSIST(MQPER_NON_PERSISTENT ACTION(DISCARD)

DEST(INQ*) PERSIST(MQPER_PERSISTENT ACTION(LEAVE)

Runmqdlq wakes up on new messages, and scans the queue periodically (the default RETRYINT is 60 seconds). It keeps track of messages on the queue, for example how many times it has retried an operation. For each message it scans the rules until it finds the first matching rule, then takes the action.

For for the rules above

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(RETRY) RETRY(5)

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(FWD) FWD(MYQUEUEOVERFLOW) HEADER(YES)

If a messages destination was MYQUEUE, and the reason code was MQRC_Q_FULL, it retries the put to the queue, at most 5 times. After 5 attempts, the next time the first rule is skipped, the second rule is used, and the message is forwarded to the queue MYQUEUEOVERFLOW keeping the DLQ header.

DEST(INQ*) PERSIST(MQPER_NON_PERSISTENT ACTION(DISCARD)

For message destination INQ* and non persistent messages, then just discard them.

DEST(INQ*) PERSIST(MQPER_PERSISTENT ACTION(LEAVE)

For message destination INQ* and persistent messages, then just leave them on the queue, for some other processing.

If runmqdlq is restarted, then all processing is reset, as all state information is kept in memory.

You should have a strategy for processing the DLQ.

For example, see Planning for MQ Dead Letter Queue handling, because you do not want thousands of non persistent inquiry messages filling up the DLQ, and preventing important persistent messages from being put onto the DLQ.

You may want to provide an audit trail of messages on the DLQ, so when someone phones up and says “MQ has lost my message”, you can look in the DLQ error logs, and say, “no… it is still in MQ, on the PENDING_SECURITY_ACTION queue, waiting for the security people to give the userid permission to process the message”.

Writing your own DLQ handler

While the MQ provided program is pretty good, there are times when you need a bit more, for example

  • Writing an audit message for each message processed, and what action was taken.
  • Printing out information about the message, such as queue name, putter, reason code etc
  • Moving one message, based on message ID or Correlid to another queue.

A one pass application is not difficult to create, it is a typical server application. A multi-pass application is much harder as you need to remember which messages have been processed.

  • I do not know if it is better to get with convert or not, especially if you are using AMS.
  • Print message information. You can use printMD from the amqsbcg0.c sample to print the MD.
  • You can create a similar function for printing the DLQ header. You may have to handle conversion yourself, for example big-indian/little endian numbers
  • You can print a hex string such as msgid using

for (ii = 0 ; ii < sizeof(msgid) ; ii++)
printf(“%02hhX”,msgid[ii])

  • If you specify a msgid as a parameter, you can read a hex string into a byte array using the following. The arrray had to be unsigned char to for it to work,otherwise you get negative numbers

unsignchar msgid[24];
int i;
for (i = 0; i < sizeof(msgid); i++)
{
sscanf(pIn + (i * 2), “%2hhx”, &msgid[i]);
}

Remove the DLQ header if needed.

mqoo_server =… MQOO_SAVE_ALL_CONTEXT ;

MQGET(hConn,
serverHandle,
&mqmd,
&mqgmo,
lBuffer,
pBuffer,
&messageLength,
&mqcc,
&mqrc);

// move the format and CCSID from the DLQ back to the mqmd
memcpy(&mqmd.Format,&pMQDLH -> Format,sizeof(mqmd.Format));
memcpy(&mqmd.CodedCharSetId,&pMQDLH -> CodedCharSetId,sizeof(mqmd.CodedCharSetId));

mqpmo.Options += MQPMO_PASS_ALL_CONTEXT;
mqpmo.Context = serverHandle;
long lDLQH = sizeof(MQDLH);

MQPUT1( hConn,
&replyOD ,
&mqmd ,
&mqpmo,
messageLength -lDLQH, // reduce the data by the size of the DLQ
pBuffer+lDLQH,// point past the DLQ
&mqcc,
&mqrc );

You can teach an old MQ program(mer) new tricks!

I wrote a program which could be used with local bindings on Linux, or as a client. Doing what I have done for 25 years, and following the IBM documentation I had a makefile with a create for each type.

gcc -m64 -o mer me.o -L/opt/mqm/lib64 -Wl,-rpath=/opt/mqm/lib64 -Wl,-rpath=/usr/lib64 -lmqm
gcc -m64 -o merc me.o -L/opt/mqm/lib64 -Wl,-rpath=/opt/mqm/lib64 -Wl,-rpath=/usr/lib64 -lmqic

Where -lmqm was for local bindings, and -lmqic was for client bindings.

For about the last 10 years, you have only needed one executable, not two!

Thanks to Morag Hughson of MQGem who pointed this out and said You can make a client connection using something linked with mqm.lib. Just set MQ_CONNECT_TYPE to CLIENT. See here.

I only need one program mer, and do not need the client version merc. I used

export MQ_CONNECT_TYPE=CLIENT
export MQCCDTURL=/home/colinpaice/c/ccdt.json
./mer CSQ9 CP0000

and it worked! (First time)

This support has been there since MQ 7.1, so as long as you have compiled your programs with MQ 7.1 or later you can use this support.

I’ll drop an email to Hursley because the documentation for generating a program says, for example

C client application, 64-bit, non-threaded

gcc -m64 -o amqsputC_64 amqsput0.c -I MQ_INSTALLATION_PATH/inc -L MQ_INSTALLATION_PATH/lib64 -Wl,-rpath=MQ_INSTALLATION_PATH/lib64 -Wl,-rpath=/usr/lib64 -lmqic

C server application, 64-bit, non-threaded

gcc -m64 -o amqsput_64 amqsput0.c -I MQ_INSTALLATION_PATH/inc -L MQ_INSTALLATION_PATH/lib64 -Wl,-rpath=MQ_INSTALLATION_PATH/lib64 -Wl,-rpath=/usr/lib64 -lmqm

It would be good if they told you about this great facility, and not only have it hidden away.

You could just build it once, and set the environment variable.

Using it

The documentation for MQ_CONNECT_TYPE says this is for MQCONNX.

If your application uses MQCONNX, then it will try local, then try as a client (using MQCCDTURL environment variable), and you do not even need to specify MQ_CONNECT_TYPE. You can force it to use local or client by speciying MQ_CONNECT_TYPE.

My application was using the old style of MQCONN. For this to work I had to specify MQ_CONNECT_TYPE=CLIENT (and the MQCCDTURL).

You also might consider upgrading your application so you use MQCONNX instead of the MQCONN. All you need is

  • MQCNO cno = {MQCNO_DEFAULT}; /* Connect Options*/
  • cno.Options = … ;
  • change MQCONN to MQCONNX and add the &cno.

plus testing it(for several weeks) of course.

Convert MQCONN to MQCONNX and you get connection to the local machine or to a client automatically – you do not need the MQ_CONNECT_TYPE.

See, you can get an old application to do new tricks.

Planning for MQ Dead Letter Queue handling.

With MQ, if a message cannot be successfully delivered, it can be put on a Dead Letter Queue for later processing.

You can have multiple queues

  • The system dead letter queue, where the MQ puts messages it cannot processed,
  • Application dead letter queues, and application can put messages to a queue,
  • The AMS dead letter queue for messages which had errors during get or put, for example a certificate mismatch.

Messages can be put to these queues for a variety of reasons.

  • Transient problems
    • If a channel is putting a message to a queue, and the queue is full, then the channel can put the message to the Dead Letter Queue. The DLQ handler can then try to put the message to the original queue, and retry a number of times after an interval. If the queue full condition was transient, then the DLQ handler is likely to succeed. If an application stops processing a queue, you can get quickly get thousands of messages on the DLQ queue.
    • The queue is put disabled. A queue can be set to put disabled, for example to stop messages from going onto a queue during queue maintenance. Once the maintenance has been done the queue can have put enabled.
  • Administration
    • The putting channel is not authorised to put to the queue, so the message gets put to the DLQ. An administrator needs to check to see if the putter is allowed to put the message. If so, fix the security and put the message back on original queue. If not remove the message, and educate the developer.
    • An AMS protected message has a problem, for example an unauthorised user has signed a message, or the id getting the message does not have a certificate to decrypt a message. You need to resolve any local certificate problems, or send the original message back to the requester saying it is in error.
  • Application
    • The message is too large for the queue. The administrator needs to educate the developer and/or make the queue max message size larger.

You may have a policy that non persistent messages for a particular queue which end up on the dead letter queue should be purged. Persistent message for another queue should have special treatment.

You may want administrators to be able to look at the meta data about a message, destination queue, MSGID, the list of recipients who can decrypt a message; but not to look at the message content.

Setting up your environment to cover these areas need considerable planning.

Implementing a solution

You want to try to keep the main DLQ close to empty, for example if your DLQ fills up with non persistent inquiries, then putting an important persistent message to the DLQ may fail.

You can use the runmqdlq program on midrange or CSQUDLQH on z/OS, to specify rules for automatic processing of messages on the DLQ.

You can select on attributes like original destination queue name, the reason why the message was on the DLQ, userid in the MQMD; and specify an action

  • Retry the put to the original queue
  • Move to another queue
  • Purge it
  • Leave it

When a message is processed on the DLQ, the rules are applied, and the action of the first matching rule is applied. For example

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(RETRY) RETRY(5)
DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(FWD) FWD(MYQUEUEOVERFLOW) HEADER(YES)

This says that if a messages destination was MYQUEUE, and the reason code was MQRC_Q_FULL, it retries the put to the queue, at most 5 times. After 5 attempts, the first rule is skipped, the second rule is used, and the message is forwarded to the queue MYQUEUEOVERFLOW keeping the DLQ header.

DEST(INQ*) PERSIST(MQPER_NON_PERSISTENT ACTION(DISCARD)

For message destination INQ* and non persistent messages, then just discard them.

DEST(INQ*) PERSIST(MQPER_PERSISTENT ACTION(LEAVE))

For message destination INQ* and persistent messages, then just leave them on the queue, for some other processing.

If runmqdlq or CSQUDLQH is restarted, then all processing is reset.

Generic rules

For transient type problems you may want to consider

  • Non persistent messages for a set of queues get purged, dont even try to put them back on the queue.
  • Persistent messages for INVOICE* queues get moved to INVOICE_DLQ queue, where you have another DLQ monitor running on the queue.

For administrator type problems

  • You could pass non persistent messages to an admin_DLQ_NP queue, and have a program which reads the meta data, and prints it to a file, then deletes the original message
  • You could pass persistent messages to an admin_DLQ_P queue
    • have a program which reads the meta data, and prints it to a file, and leaves the message on the queue.
    • Using the meta information resolve the problem.
    • Have another program which takes the msgid and correlid as input parameters, then puts the message on the original queue. (If there is only one message, you could use the default DLQ handler to do this.)

For AMS problems

  • This is complicated by having to use a different queue. If the DLQ handler tries to put to the AMS protected queue, it will be “protected” (enciphered) again. You need to use put the message to an alias queue, with the original queue as the target. On midrange Java and C clients can disable AMS processing, either by using an environment variable, or through the MQCLIENT.ini file. See here.
  • This is also complicated by possibly needing access to information in the payload, such as the list of recipients, and decrypting the message to get the DN of the signer.
    • Once you have resolved the problem, have another program which takes the msgid and correlid as input parameters, and puts the message on the alias queue (if there is only one message you could use the default DLQ handler to do this).

How do I check it I have got it right?

It is worth putting a process in place to monitor the depth of the dead letter queue, and if it does not become empty a few a minutes, display the contents of the queue, and add rules to handle the residual messages.

I do not think that IBM provides a list of return codes of messages that it puts onto the DLQ, I think you’ll have to go through the list (over 500!), and put a rule in place for each one. If an application invents its own return codes, you may need rules for these as well.

My quick look at the list includes

Common problems

2053 (0805) (RC2053): MQRC_Q_FULL
2056 (0808) (RC2056): MQRC_Q_SPACE_NOT_AVAILABLE
2071 (0817) (RC2071): MQRC_STORAGE_NOT_AVAILABLE

Other problems
2030 (07EE) (RC2030): MQRC_MSG_TOO_BIG_FOR_Q
2031 (07EF) (RC2031): MQRC_MSG_TOO_BIG_FOR_Q_MGR
2033 (07F1) (RC2033): MQRC_NO_MSG_AVAILABLE
2051 (0803) (RC2051): MQRC_PUT_INHIBITED
2052 (0804) (RC2052): MQRC_Q_DELETED
2053 (0805) (RC2053): MQRC_Q_FULL
2056 (0808) (RC2056): MQRC_Q_SPACE_NOT_AVAILABLE
2071 (0817) (RC2071): MQRC_STORAGE_NOT_AVAILABLE
2102 (0836) (RC2102): MQRC_RESOURCE_PROBLEM
2120 (0848) (RC2120): MQRC_CONVERTED_MSG_TOO_BIG
2141 (085D) (RC2141): MQRC_DLH_ERROR
2142 (085E) (RC2142): MQRC_HEADER_ERROR
2148 (0864) (RC2148): MQRC_IIH_ERROR
2149 (0865) (RC2149): MQRC_PCF_ERROR
2150 (0866) (RC2150): MQRC_DBCS_ERROR
[z/OS]2342 (0926) (RC2342): MQRC_DB2_NOT_AVAILABLE
[z/OS]2345 (0929) (RC2345): MQRC_CF_NOT_AVAILABLE
[z/OS]2348 (092C) (RC2348): MQRC_CF_STRUC_AUTH_FAILED
[z/OS]2349 (092D) (RC2349): MQRC_CF_STRUC_ERROR
2362 (093A) (RC2362): MQRC_BACKOUT_THRESHOLD_REACHED

You may want common ones, such as queue full and not authorised, on a per queue basis, and all the less common ones, such as all of the z/OS ones all putting to one “admin” queue.

Building amqsdlq sample Dead Letter Handler.

MQ provides a sample Dead Letter Queue Handler in /opt/mqm/samp/dlq/ (on Ubuntu). It looks and behaves just like runmqdlq. I think trying to extend it, for example to filter on MSGID or CORRELID would be difficult, and it would be easier to write a small program just to process one message with input queue, and pass filter parameters.

Build it

There are no instructions in the IBM documentation on how to build this. I used the following

  • create a directory to contain the code, for example mkdir ~/dlq
  • go to this directory cd ~/dlq
  • copy the code from MQ cp -r /opt/mqm/samp/dlq/* .
  • make the directory rw chmod +w ~/dlq
  • install a yacc compiler sudo apt install byacc
  • issue the make command make all -f amqodqx.mk

If you get

gcc -m64 -I. -I/opt/mqm/inc -c -o amqodqka.o amqodqka.c
Assembler messages:
Fatal error: can’t create amqodqka.o: Permission denied

Your directory is not read/write. Use the chmod command.

If you get

yacc amqodqma.y
make: yacc: Command not found

You need to install a yacc compiler. I used sudo apt install byacc .

I dont undestand why I got this message, because the build for the yacc files is commented out. It may be there is a default make for yacc.

The YACC ( compiler generator) produces a c file. If you are not changing the keywords, you could rename the .y and .l files, so they are not used, and so use the shipped c files. In this case you will not need the yacc compiler.

Run it

  • export ODQ_MSG=amqsdlq.msg This is needed for the message catalog
  • ./amqsdlq SYSTEM.PROTECTION.ERROR.QUEUE QMA < /home/colinpaice/mqams/dlq.rul
    • for a while it failed with a syntax error in line 1 of the file, I rebuilt it, then suddenly it worked.
    • Ctrl-c ended it
  • ./amqsdlq ? gave me a segmentation core dump

I changed the make file to build a client

cc -m64 -o amqsdlqc… -lmqic_r

You should ensure you use a TLS channel if you do run this as a client.

I have a message on the AMS DLQ – what can I do about it?

If AMS has problems with a protected message, AMS can put the message on the SYSTEM.PROTECTION.ERROR.QUEUE queue. This blog post discusses what you can do about it. I consider this a hard problem – not in the same league as trying to simulate the beginnings of the Universe – more like climbing Ben Nevis mountain in Scotland, when you are only used to walking down to the shops.

What are the problems?

There are several problems you need to consider

  1. Why is the message on the queue? Is it a problem with the putting application, or with the getting environment?
  2. Which user had the problem. For example it may not be obvious which application instance had a problem, if applications come in through one channel, and many users have the same MCA userid.
  3. What you need to do about it to get the message reprocessed, and prevent future problems.

Why is the message on the queue

The messages could be on the queue because

  • The certificate was signed, but the DN of the signer is not in the setmqspl list of authorised signers (-a). This is an example of an MQ configuration problem
  • The user getting a message was not able to verify the signers certificate sent in the message, for example it is missing the CA of the signer, or missing the signers self signed certificate. This is an example of a user’s configuration problem.
  • The message was encrypted, but the user getting the message did not have access to a private certificate to allow the message to be decrypted. The user’s DN needs to be added to the recipients when the message is put and enciphered (or the user needs to be stopped from getting messages from this queue). This is an example of the putting of the message message is missing configuration information.

What other information is there to help me?

If you know which id had the problem, there should be information in the error logs. For example a problems within a Java JMS client program may write to mqjms.log.0. A local application may write to the queue manager’s error log, for example /var/mqm/qmgrs/QMA/error/*01*

In the mqjms.log.0 I got

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.prot.MessageProtectionBCImpl
The receiver of this encrypted message is not on the message recipient list ‘CN=ja2,O=aaaa,C=GB’
The certificate of a user that is receiving a message is not on the message RecipientsInfo list.

Verify that the user is on a recipients list in a security policy definition.

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.intercept.JmqiGetInterceptorImpl
The IBM MQ Advanced Message Security Java interceptor failed to unprotect the received message.
An error occurred when the IBM MQ Advanced Message Security Java interceptor was unprotecting the received message.

See subsequent messages in the exception for more details about the cause of the error

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.service.EseMQServiceImpl
The IBM MQ Advanced Message Security interceptor has put a defective message on error handling queue ‘SYSTEM.PROTECTION.ERROR.QUEUE ‘.

(On z/OS the messages are less helpful.)

On the SYSTEM.PROTECTION.ERROR.QUEUE queue there was a message with a Dead Letter Header (DLH) with a reason 2063 0x0000080f MQRC_SECURITY_ERROR.

From the MQMD you can see the time the message was put to the queue, the putting application, and the user identification. This userIdentified may have been set for example by the channel MCAUSER, or CHLAUTH, and so you do not always know where the original request came from.

If you are testing then you will know what caused the error.

If you are in a production like environment, you know the application, and as you will have configured all user keystores the same you may not need to know which specific user caused the problem. If there is a problem with a missing certificate, then you fix the problem, redeploy the keystore to all your users (as part of your automated process) and try again.

How do you tell what the problem is.

Your systems administrator needs a process for extracting meta data about messages, while keeping the application payload protected. You could build a process around displaying the recipients and signer from How do I find the recipients and signer of an AMS message? The systems administrator needs to know

  • the original queue name with the problem
  • the time, date, and user that put the message
  • the msgid and correlid of the message – so if you put it back on the queue, you know which message to process.
  • from the message, the type of protection: Integrity (it was signed), Encryption (it was encrypted), Privacy (it was encrypted with a signed payload)
  • any id information from the message, such as recipient DN’s, and the signer DN. See this post.
  • you may have to have some special processing to decrypt the Privacy payload, just to extract the signer information.

If this process can be automated, then any application content can be kept secure.

With this information and your “up to date application work book” (do you have one of these?) , you should be able to identify the problem.

Once you have fixed the root cause of the problem….

The fix may be to change the setmqspl to add an authorised signer, or to add a certificate to the recipients keystore, you need to get the message reprocessed.

  • You get the specific message from the message id and correlid.
  • You need to remove the DLQ header from the message.
  • You need some special set up for the queue. If you try to put to the original queue, it will get the AMS protection again, for example re-encrypted or resigned. You need a queue alias so you put to the queue alias bypassing any AMS processing.

There are lots of things you need to consider, which is why I consider this a hard problem.

This application would be a good example where message handle is used to “move” the message.

MQCRTMH(hConn,&cmho,&hMsg,&CompCode,&Reason);

gmo.MsgHandle = hMsg;

MQGET(hConn,….);

pmo.Action = MQACTP_FORWARD;

pmo.OriginalMsgHandle =hmsg

MQPUT(…)

I found Learn to code the MQ Message Property MQI calls from MQGEM software useful in understanding message handle and message properties, and how to delete the DLQ header.

Other AMS blog posts