Compiling a C program on z/OS

You can compile programs in USS, or with JCL. I tend to prefer JCL, but do use USS (but it takes time to get it the command right). It took me several attempts to compile and bind a program that uses USS services.

I thought people might be interested in the JCL I use, and the C Compiler options I specified.

I’ll give the JCL (so you can see how much you understand of it) then I’ll annotate it

//ADCDC4 JOB 1,MSGCLASS=H,COND=(4,LE)
//S1 JCLLIB ORDER=CBC.SCCNPRC
// SET LOADLIB=IBMUSER.LOAD
// SET LIBPRFX=CEE
//COMPILE EXEC PROC=EDCCB,
// LIBPRFX=&LIBPRFX,
// CPARM='OPTFILE(DD:SYSOPTF),NOLSEARCH,LSEARCH(/usr/include/)',
// BPARM='SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31'
//COMPILE.SYSOPTF DD *
LIST,SOURCE
aggregate(offsethex) 
xref
SEARCH(//'ADCD.C.H',//'SYS1.SIEAHDR.H')
TEST
RENT ILP32
OE
INFO(PAR,USE)
NOMARGINS EXPMAC SHOWINC XREF
LANGLVL(EXTENDED) sscom dll
DEFINE(_ALL_SOURCE)
DEBUG/*
//COMPILE.SYSIN DD *
#pragma linkage(IRRSDL00 ,OS)
#line 26
#pragma runopts(POSIX(ON))
/*Include standard libraries */
#include <stdio.h> 
#include <stdlib.h> 
#include <string.h> 
#include <stdarg.h> 

int main( int argc, char *argv??(??))
{
printf("I'm here program %s. ",argv[0]);
for (int i = 1;i < argc; i++)
printf("Arg %d %s",i,argv[i]);
printf("\n");
}//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB.
//BIND.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEELKED
//BIND.OBJLIB DD DISP=SHR,DSN=COLIN.OBJLIB
//BIND.GSK DD DISP=SHR,DSN=SYS1.SIEALNKE
//BIND.CSS DD DISP=SHR,DSN=SYS1.CSSLIB
//BIND.SYSIN DD *
INCLUDE CSS(IRRSDL00)
NAME MAINPROG(R)
//ISTEST EXEC PGM=MAINPROG,REGION=0M,PARM='PARM1, Parm2'
//STEPLIB DD DISP=SHR,DSN=&LOADLIB
//SYSPRINT DD SYSOUT=,DCB=(LRECL=200) //SYSOUT DD SYSOUT=
//SYSERR DD SYSOUT=*


Annotate JCL

The compile options are defined here

  • //ADCDC4 JOB 1,MSGCLASS=H,COND=(4,LE) If the return code from each step is less equal to 4, then it does the next step. If the compile fails with return code 8, the job stops.
  • //S1 JCLLIB ORDER=CBC.SCCNPRC Use the procedures from this library
  • // SET LOADLIB=IBMUSER.LOAD A symbol used to say where to store the load module
  • // SET LIBPRFX=CEE This is used by the Compile procedure
  • //COMPILE EXEC PROC=EDCCB, Execute the C (component EDC) Compile and Bind
  • // LIBPRFX=&LIBPRFX, using this library prefix (defined above)
  • // CPARM=’OPTFILE(DD:SYSOPTF),LSEARCH(/usr/include/)’, Read C options from //SYSOPTF, and use /usr/Include to find header files
  • // BPARM=’SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31′ Binder parameters
  • //COMPILE.SYSOPTF DD * These additional C compiler options
  • LIST, Give the assembler output
  • SOURCE display the source of the program (useful for showing compile errors)
  • aggregate(offsethex) Show c structures with hex offsets
  • xref show where thing are used
  • SEARCH(//’ADCD.C.H’,//’SYS1.SIEAHDR.H’) Look for C header files in these libraries
  • TEST produce information for some debug tools.
  • RENT Produce re-entrant code
  • ILP32 produce 32 bit code ( compare to option LP64)
  • OE Use Posix standards when looking for header files
  • INFO(PAR,USE) Print out information messages. Par=Emits warning messages on unused. parameters. USE=Emits information about usage of variables ( eg defined but not used).
  • NOMARGINS Which columns to use. Default is columns 1-72
  • EXPMAC expand all macros in the source
  • SHOWINC display the source of any include in the output
  • LANGLVL(EXTENDED) Use language extensions, for example use of long long.
  • sscom Use of Slash Slash COMents “// comments…”
  • dll Produce code which can be used in a DLL. Useful for some USS type programs
  • DEFINE(_ALL_SOURCE) Create a #define ALL_SOURCE variable
  • DEBUG produce debugging information, such as line numbers in stack traces. It turns optimisation off.
  • /*
  • //COMPILE.SYSIN DD * Main program follows
  • #pragma linkage(IRRSDL00 ,OS) A C program called a z/OS function called IRRSDL00. This says it uses a standard z/OS parameter list
  • #line 26 This tells the C compiler to reset its line number – so the error messages come with the correct line number.
  • #pragma runopts(POSIX(ON)) This is so posix (Uss) posix functions can be used
  • /*Include standard libraries */
  • #include <stdio.h>
  • #include <stdlib.h>
  • #include <string.h>
  • #include <stdarg.h>
  • int main( int argc, char *argv??(??))
  • {
  • printf(“I’m here program %s. “,argv[0]);
  • for (int i = 1;i < argc; i++)
  • printf(“Arg %d %s”,i,argv[i]);
  • printf(“\n”);
  • }
  • //BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB. Where to store the output from the bind
  • //BIND.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEELKED Use these libraries to resolve external routines
  • //BIND.OBJLIB DD DISP=SHR,DSN=COLIN.OBJLIB Load the compiled output from here
  • //BIND.CSS DD DISP=SHR,DSN=SYS1.CSSLIB My program needs a z/OS specific function. Get it from here
  • //BIND.SYSIN DD *
    • INCLUDE CSS(IRRSDL00) My program needed the z/OS callable service IRRSDL00 which is in the library SYS1.CSSLIBs
  • NAME MAINPROG(R) This is the name of the program. It is stored in //BIND.SYSLMOD above
  • //ISTEST EXEC PGM=MAINPROG,REGION=0M,PARM=’PARM1, Parm2′
  • //STEPLIB DD DISP=SHR,DSN=&LOADLIB
  • //SYSPRINT DD SYSOUT=,DCB=(LRECL=200) it prints long lines of output up to 200 chars long
  • //SYSOUT DD SYSOUT=
  • //SYSERR DD SYSOUT=*

checkAMS: program to check your AMS defintions are consistent with z/OS keyring

A C program to verify that the certificates in MQ AMS configuration are in a RACF keyring. See here.

Overview of program

With AMS you specify the Distinquished Names(DN) of users who are allowed to sign or encrypt MQ messages. The certificates for these DN’s need to be in the xxxxAMSM’s drq.ams.keyring. If they are not present, or have problems, such as they are not valid, the messages from AMS are not very helpful. The messages are as helpful as “one of the DN’s in the configuration has a problem but I am not telling you which DN it was, nor what the problem was”.

CheckAMS has two parts:

  1. Provide a useful list of information in the keyring
  2. Takes the output of the AMS dspmqspl command, and checks the DN’s are in the key store

Provide a useful list of the contents of a keyring.

With the RACDCERT commands you can list the contents of a keyring, for example owner and label; and you can display details about a certificate, such as the DN of the subject, and the Certificate Authority, but you cannot issue one command to display all the important information, nor ask, “is the DN for this issuer in the keystore”.

Example output from checkAMS, listing certificates in keyring:

Subject CN=SSCARSA1024,OU=CA,O=SSS,C=GB                                                         
Issuer  CN=SSCARSA1024,OU=CA,O=SSS,C=GB                                                         
Self signed                                                                                     
Valid date range 21/02/13 12:32:33 to 24/02/13 12:32:33                                         
Owner irrcerta/LINUXCA                                                                          
Usage:Certauth Status:Trust                                                                     
                                                                                                           
Subject CN=colin,OU=longou,O=SSS                                                                
Issuer  CN=TEMP4Certification Authority,OU=TEST,O=TEMP                                          
Valid date range 21/03/25 00:00:00 to 22/03/25 23:59:59                                         
Owner COLIN/TEST                                                                                
Usage:Site Status:Trust      

The first certificate is owned by irrcerta and has label LINUXA. Userid irrcerta means it belongs to CERTAUTH. The certificate is self signed, and has a long validity date. It has a usage of CERTAUTH, and is trusted.

The second certificate belongs to userid COLIN, it has label TEST. It has a subject DN of Subject CN=colin,OU=longou,O=SSS, and was issued by CN=TEMP4Certification Authority,OU=TEST,O=TEMP. It has a usage of Site, and is trusted.

Check the AMS set up

The program takes as input the output of the dspmqspl -m… -export command, and checks the DN against certificates in the keyring.

Example output

Userid START1, ring drq.ams.keyring                                                                                  
* Exported on Mon Mar 29 09:23:31 2021                                                                               
                                                                                                                      
dspmqspl -m CSQ9  -export                                                                                          
setmqspl -m CSQ9                                                                                                     
 -p AMSQ                                                                                                             
 -s SHA256                                                                                                           
 -a "CN=COLIN,O=SSS"                                                                                                 
   Owner COLIN/AMS Usage:Site Status:Trust Valid date range 21/03/21 00:00:00 to 22/03/21 18:45:00                  
 -a "O=aaaa, C=GB,CN=ja2"                                                                                            
 ! O=aaaa,C=GB,CN=ja2 Not found in key ring                                                                           
 -e AES256                                                                                                           
 -r "CN=COLIN,O=SSS"                                                                                                 
  Owner COLIN/AMS Usage:Site Status:Trust Valid date range 21/03/21 00:00:00 to 22/03/21 18:45:00                  
 -r "CN=ADCDB,O=SSS"

This shows the keyring was START1/drq.ams.keyring.

It prints out the exported file, and for the -a and -r records, it adds information about the certificate, or reports if it is not found.

It reports that “CN=COLIN,O=SSS” was found, the certificate belongs to userid COLIN,label AMS, it has usage of Site, it is trusted, and has a valid date.

It also reports O=aaaa,C=GB,CN=ja2 Not found in key ring This is because the definition in AMS has the wrong order. The standard order is CN=ja2,O=aaaa,c=GB. This certificate is in the keyring , but the program could not find it. I could not see a way of converting bad format DNs to good DNs.

Contents of package.

The package is on git.

FTP the amscheck.xmit.bit to z/OS as binary. Then use TSO receive indsn(amscheck.xmit) to create the load module in a PDS.

Upload runamsch, ccasmch, asmcheck. and parmlist.h to a PDS.

Edit and submit runamsch. It runs dspmqspl and puts the output into a temporary file. The parm PARM=’START1 drq.ams.keyring’ is for userid START1 and the keyring drq.ams.keyring. Your userid will need access to the userid’s keyring.

if you want to compile the program

If you want to compile the program, you can edit ccasmch, and change the SYSIN, and where the header file is imported from.

How do I process messages on the dead letter queue (DLQ)?

I was setting up security on my system, and using AMS to protect messages. I kept getting messages on the Dead Letter Queues. As messages on the DLQ have been around from before MQ V1 was shipped (they hit this problem in development), I was expecting that to process them would be easy. There are some good bits and some not so good bits with the IBM supplied solution. I was reminded of a “call and response narration” game we enjoyed in the pub from when I was a student which went ..

They are building a house in the street – (audience) Boo!
A public house – (audience)Hooray!
They don’t sell beer – (audience) Boo!
They give it away – (audience) Hooray!

For a supplied Dead Letter Queue handler it goes…

MQ provides a Dead Letter Handler program (runmqdlq) – Hooray!
On z/OS (CSQUDLQH) and midrange (runmqdlq). – Hooray.
It is rule based and can handle many scenarios – Hoorary!
But not some of the difficult ones – Boo!
The provide a set of sample programs on mid range (amqsdlq) – Hooray!
But they are not well documented, didn’t build straight off, and not available on z/OS – Boo.
It can process many similar messages in one go- Hooray,
But not process just one message – Boo.

Why are messages put on the DLQ?

If a local application tries to put a message to a queue, and the queue is full then the application gets a return code, and takes an action. The message is not lost – it wasn’t created, and the DLQ was not used. If a message comes in from another queue manager, and the channel tries putting the message and gets queue full, it cannot just throw the message away. It puts it onto the DLQ.

Messages could be put on a DLQ for many reasons.

  • A message came in from a remote queue manager and was put to a local queue, but the queue was at max depth, so was put to the DLQ. This may be due to a short lived problem. The DLQ handler can process the DLQ queue, and every 60 seconds try moving the message from the DLQ back to the original. You can configure the rules so if it tries 5 times and fails, then it moves the message to a different queue.
  • A message came in from a remote queue queue manager, but the channel userid was not authorised to put to the queue. In this case retrying every 60 seconds is unlikely to solve the problem. The administrator needs to take an action, such as grant access and retry the put, or remove the message.
  • When AMS is used, if an ID tries to get the message and there are problems, such as the ID of the signer of the message is not authorised, the message is put to the SYSTEM.PROTECTION.ERROR.QUEUE queue. To resolve this, the AMS configuration needs to be changed, or the message moved to a quarantine queue. Once the configuration has been changed, put the message back on the queue for retry.

The runmqdlq handler provided with MQ

This is a bit of a strange beast. It is rule based so you can configure rules to select messages with certain properies and take actions, such as retry, or move to a different queue.

The program on midrange is runmqdlq, and on z/OS CSQUDLQH.

The syntax for runmqdlq is

runmqdlq [-u userid] MYDEAD.QUEUE QMA <qrule.rul

you have to pipe the file into stdin, until an empty line is processed. I would have preferred a -f filename option.

To end runmqdlq, set the input queue to get(DISABLED) because Ctrl-C does not work.

It processes message silently, unless there are any problems, for example I got

Dead-letter queue handler unable to put message: Rule 6 Reason 2035.

I had several problem messages on the DLQ, but I could not specify one message and get runmqdlq to process it, so I had to write a program to move one message to a different queue, then I could use runmqdlq. There is lots of good stuff in runmqdlq, but doesn’t quite do the job.

Understanding the rules.

The rules are the same for z/OS as mid-range.

Messages are read from the specified DLQ queue, and processed with a set of rules. The rules are described here. You can select on properties in the MQMD or the DLQ header. For example

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(RETRY) RETRY(5)

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(FWD) FWD(MYQUEUEOVERFLOW) HEADER(YES)

DEST(INQ*) PERSIST(MQPER_NON_PERSISTENT ACTION(DISCARD)

DEST(INQ*) PERSIST(MQPER_PERSISTENT ACTION(LEAVE)

Runmqdlq wakes up on new messages, and scans the queue periodically (the default RETRYINT is 60 seconds). It keeps track of messages on the queue, for example how many times it has retried an operation. For each message it scans the rules until it finds the first matching rule, then takes the action.

For for the rules above

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(RETRY) RETRY(5)

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(FWD) FWD(MYQUEUEOVERFLOW) HEADER(YES)

If a messages destination was MYQUEUE, and the reason code was MQRC_Q_FULL, it retries the put to the queue, at most 5 times. After 5 attempts, the next time the first rule is skipped, the second rule is used, and the message is forwarded to the queue MYQUEUEOVERFLOW keeping the DLQ header.

DEST(INQ*) PERSIST(MQPER_NON_PERSISTENT ACTION(DISCARD)

For message destination INQ* and non persistent messages, then just discard them.

DEST(INQ*) PERSIST(MQPER_PERSISTENT ACTION(LEAVE)

For message destination INQ* and persistent messages, then just leave them on the queue, for some other processing.

If runmqdlq is restarted, then all processing is reset, as all state information is kept in memory.

You should have a strategy for processing the DLQ.

For example, see Planning for MQ Dead Letter Queue handling, because you do not want thousands of non persistent inquiry messages filling up the DLQ, and preventing important persistent messages from being put onto the DLQ.

You may want to provide an audit trail of messages on the DLQ, so when someone phones up and says “MQ has lost my message”, you can look in the DLQ error logs, and say, “no… it is still in MQ, on the PENDING_SECURITY_ACTION queue, waiting for the security people to give the userid permission to process the message”.

Writing your own DLQ handler

While the MQ provided program is pretty good, there are times when you need a bit more, for example

  • Writing an audit message for each message processed, and what action was taken.
  • Printing out information about the message, such as queue name, putter, reason code etc
  • Moving one message, based on message ID or Correlid to another queue.

A one pass application is not difficult to create, it is a typical server application. A multi-pass application is much harder as you need to remember which messages have been processed.

  • I do not know if it is better to get with convert or not, especially if you are using AMS.
  • Print message information. You can use printMD from the amqsbcg0.c sample to print the MD.
  • You can create a similar function for printing the DLQ header. You may have to handle conversion yourself, for example big-indian/little endian numbers
  • You can print a hex string such as msgid using

for (ii = 0 ; ii < sizeof(msgid) ; ii++)
printf(“%02hhX”,msgid[ii])

  • If you specify a msgid as a parameter, you can read a hex string into a byte array using the following. The arrray had to be unsigned char to for it to work,otherwise you get negative numbers

unsignchar msgid[24];
int i;
for (i = 0; i < sizeof(msgid); i++)
{
sscanf(pIn + (i * 2), “%2hhx”, &msgid[i]);
}

Remove the DLQ header if needed.

mqoo_server =… MQOO_SAVE_ALL_CONTEXT ;

MQGET(hConn,
serverHandle,
&mqmd,
&mqgmo,
lBuffer,
pBuffer,
&messageLength,
&mqcc,
&mqrc);

// move the format and CCSID from the DLQ back to the mqmd
memcpy(&mqmd.Format,&pMQDLH -> Format,sizeof(mqmd.Format));
memcpy(&mqmd.CodedCharSetId,&pMQDLH -> CodedCharSetId,sizeof(mqmd.CodedCharSetId));

mqpmo.Options += MQPMO_PASS_ALL_CONTEXT;
mqpmo.Context = serverHandle;
long lDLQH = sizeof(MQDLH);

MQPUT1( hConn,
&replyOD ,
&mqmd ,
&mqpmo,
messageLength -lDLQH, // reduce the data by the size of the DLQ
pBuffer+lDLQH,// point past the DLQ
&mqcc,
&mqrc );

You can teach an old MQ program(mer) new tricks!

I wrote a program which could be used with local bindings on Linux, or as a client. Doing what I have done for 25 years, and following the IBM documentation I had a makefile with a create for each type.

gcc -m64 -o mer me.o -L/opt/mqm/lib64 -Wl,-rpath=/opt/mqm/lib64 -Wl,-rpath=/usr/lib64 -lmqm
gcc -m64 -o merc me.o -L/opt/mqm/lib64 -Wl,-rpath=/opt/mqm/lib64 -Wl,-rpath=/usr/lib64 -lmqic

Where -lmqm was for local bindings, and -lmqic was for client bindings.

For about the last 10 years, you have only needed one executable, not two!

Thanks to Morag Hughson of MQGem who pointed this out and said You can make a client connection using something linked with mqm.lib. Just set MQ_CONNECT_TYPE to CLIENT. See here.

I only need one program mer, and do not need the client version merc. I used

export MQ_CONNECT_TYPE=CLIENT
export MQCCDTURL=/home/colinpaice/c/ccdt.json
./mer CSQ9 CP0000

and it worked! (First time)

This support has been there since MQ 7.1, so as long as you have compiled your programs with MQ 7.1 or later you can use this support.

I’ll drop an email to Hursley because the documentation for generating a program says, for example

C client application, 64-bit, non-threaded

gcc -m64 -o amqsputC_64 amqsput0.c -I MQ_INSTALLATION_PATH/inc -L MQ_INSTALLATION_PATH/lib64 -Wl,-rpath=MQ_INSTALLATION_PATH/lib64 -Wl,-rpath=/usr/lib64 -lmqic

C server application, 64-bit, non-threaded

gcc -m64 -o amqsput_64 amqsput0.c -I MQ_INSTALLATION_PATH/inc -L MQ_INSTALLATION_PATH/lib64 -Wl,-rpath=MQ_INSTALLATION_PATH/lib64 -Wl,-rpath=/usr/lib64 -lmqm

It would be good if they told you about this great facility, and not only have it hidden away.

You could just build it once, and set the environment variable.

Using it

The documentation for MQ_CONNECT_TYPE says this is for MQCONNX.

If your application uses MQCONNX, then it will try local, then try as a client (using MQCCDTURL environment variable), and you do not even need to specify MQ_CONNECT_TYPE. You can force it to use local or client by speciying MQ_CONNECT_TYPE.

My application was using the old style of MQCONN. For this to work I had to specify MQ_CONNECT_TYPE=CLIENT (and the MQCCDTURL).

You also might consider upgrading your application so you use MQCONNX instead of the MQCONN. All you need is

  • MQCNO cno = {MQCNO_DEFAULT}; /* Connect Options*/
  • cno.Options = … ;
  • change MQCONN to MQCONNX and add the &cno.

plus testing it(for several weeks) of course.

Convert MQCONN to MQCONNX and you get connection to the local machine or to a client automatically – you do not need the MQ_CONNECT_TYPE.

See, you can get an old application to do new tricks.

Planning for MQ Dead Letter Queue handling.

With MQ, if a message cannot be successfully delivered, it can be put on a Dead Letter Queue for later processing.

You can have multiple queues

  • The system dead letter queue, where the MQ puts messages it cannot processed,
  • Application dead letter queues, and application can put messages to a queue,
  • The AMS dead letter queue for messages which had errors during get or put, for example a certificate mismatch.

Messages can be put to these queues for a variety of reasons.

  • Transient problems
    • If a channel is putting a message to a queue, and the queue is full, then the channel can put the message to the Dead Letter Queue. The DLQ handler can then try to put the message to the original queue, and retry a number of times after an interval. If the queue full condition was transient, then the DLQ handler is likely to succeed. If an application stops processing a queue, you can get quickly get thousands of messages on the DLQ queue.
    • The queue is put disabled. A queue can be set to put disabled, for example to stop messages from going onto a queue during queue maintenance. Once the maintenance has been done the queue can have put enabled.
  • Administration
    • The putting channel is not authorised to put to the queue, so the message gets put to the DLQ. An administrator needs to check to see if the putter is allowed to put the message. If so, fix the security and put the message back on original queue. If not remove the message, and educate the developer.
    • An AMS protected message has a problem, for example an unauthorised user has signed a message, or the id getting the message does not have a certificate to decrypt a message. You need to resolve any local certificate problems, or send the original message back to the requester saying it is in error.
  • Application
    • The message is too large for the queue. The administrator needs to educate the developer and/or make the queue max message size larger.

You may have a policy that non persistent messages for a particular queue which end up on the dead letter queue should be purged. Persistent message for another queue should have special treatment.

You may want administrators to be able to look at the meta data about a message, destination queue, MSGID, the list of recipients who can decrypt a message; but not to look at the message content.

Setting up your environment to cover these areas need considerable planning.

Implementing a solution

You want to try to keep the main DLQ close to empty, for example if your DLQ fills up with non persistent inquiries, then putting an important persistent message to the DLQ may fail.

You can use the runmqdlq program on midrange or CSQUDLQH on z/OS, to specify rules for automatic processing of messages on the DLQ.

You can select on attributes like original destination queue name, the reason why the message was on the DLQ, userid in the MQMD; and specify an action

  • Retry the put to the original queue
  • Move to another queue
  • Purge it
  • Leave it

When a message is processed on the DLQ, the rules are applied, and the action of the first matching rule is applied. For example

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(RETRY) RETRY(5)
DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(FWD) FWD(MYQUEUEOVERFLOW) HEADER(YES)

This says that if a messages destination was MYQUEUE, and the reason code was MQRC_Q_FULL, it retries the put to the queue, at most 5 times. After 5 attempts, the first rule is skipped, the second rule is used, and the message is forwarded to the queue MYQUEUEOVERFLOW keeping the DLQ header.

DEST(INQ*) PERSIST(MQPER_NON_PERSISTENT ACTION(DISCARD)

For message destination INQ* and non persistent messages, then just discard them.

DEST(INQ*) PERSIST(MQPER_PERSISTENT ACTION(LEAVE)

For message destination INQ* and persistent messages, then just leave them on the queue, for some other processing.

If runmqdlq or CSQUDLQH is restarted, then all processing is reset.

Generic rules

For transient type problems you may want to consider

  • Non persistent messages for a set of queues get purged, dont even try to put them back on the queue.
  • Persistent messages for INVOICE* queues get moved to INVOICE_DLQ queue, where you have another DLQ monitor running on the queue.

For administrator type problems

  • You could pass non persistent messages to an admin_DLQ_NP queue, and have a program which reads the meta data, and prints it to a file, then deletes the original message
  • You could pass persistent messages to an admin_DLQ_P queue
    • have a program which reads the meta data, and prints it to a file, and leaves the message on the queue.
    • Using the meta information resolve the problem.
    • Have another program which takes the msgid and correlid as input parameters, then puts the message on the original queue. (If there is only one message, you could use the default DLQ handler to do this.)

For AMS problems

  • This is complicated by having to use a different queue. If the DLQ handler tries to put to the AMS protected queue, it will be “protected” (enciphered) again. You need to use put the message to an alias queue, with the original queue as the target. On midrange Java and C clients can disable AMS processing, either by using an environment variable, or through the MQCLIENT.ini file. See here.
  • This is also complicated by possibly needing access to information in the payload, such as the list of recipients, and decrypting the message to get the DN of the signer.
    • Once you have resolved the problem, have another program which takes the msgid and correlid as input parameters, and puts the message on the alias queue (if there is only one message you could use the default DLQ handler to do this).

How do I check it I have got it right?

It is worth putting a process in place to monitor the depth of the dead letter queue, and if it does not become empty a few a minutes, display the contents of the queue, and add rules to handle the residual messages.

I do not think that IBM provides a list of return codes of messages that it puts onto the DLQ, I think you’ll have to go through the list (over 500!), and put a rule in place for each one. If an application invents its own return codes, you may need rules for these as well.

My quick look at the list includes

Common problems

2053 (0805) (RC2053): MQRC_Q_FULL
2056 (0808) (RC2056): MQRC_Q_SPACE_NOT_AVAILABLE
2071 (0817) (RC2071): MQRC_STORAGE_NOT_AVAILABLE

Other problems
2030 (07EE) (RC2030): MQRC_MSG_TOO_BIG_FOR_Q
2031 (07EF) (RC2031): MQRC_MSG_TOO_BIG_FOR_Q_MGR
2033 (07F1) (RC2033): MQRC_NO_MSG_AVAILABLE
2051 (0803) (RC2051): MQRC_PUT_INHIBITED
2052 (0804) (RC2052): MQRC_Q_DELETED
2053 (0805) (RC2053): MQRC_Q_FULL
2056 (0808) (RC2056): MQRC_Q_SPACE_NOT_AVAILABLE
2071 (0817) (RC2071): MQRC_STORAGE_NOT_AVAILABLE
2102 (0836) (RC2102): MQRC_RESOURCE_PROBLEM
2120 (0848) (RC2120): MQRC_CONVERTED_MSG_TOO_BIG
2141 (085D) (RC2141): MQRC_DLH_ERROR
2142 (085E) (RC2142): MQRC_HEADER_ERROR
2148 (0864) (RC2148): MQRC_IIH_ERROR
2149 (0865) (RC2149): MQRC_PCF_ERROR
2150 (0866) (RC2150): MQRC_DBCS_ERROR
[z/OS]2342 (0926) (RC2342): MQRC_DB2_NOT_AVAILABLE
[z/OS]2345 (0929) (RC2345): MQRC_CF_NOT_AVAILABLE
[z/OS]2348 (092C) (RC2348): MQRC_CF_STRUC_AUTH_FAILED
[z/OS]2349 (092D) (RC2349): MQRC_CF_STRUC_ERROR
2362 (093A) (RC2362): MQRC_BACKOUT_THRESHOLD_REACHED

You may want common ones, such as queue full and not authorised, on a per queue basis, and all the less common ones, such as all of the z/OS ones all putting to one “admin” queue.

Building amqsdlq sample Dead Letter Handler.

MQ provides a sample Dead Letter Queue Handler in /opt/mqm/samp/dlq/ (on Ubuntu). It looks and behaves just like runmqdlq. I think trying to extend it, for example to filter on MSGID or CORRELID would be difficult, and it would be easier to write a small program just to process one message with input queue, and pass filter parameters.

Build it

There are no instructions in the IBM documentation on how to build this. I used the following

  • create a directory to contain the code, for example mkdir ~/dlq
  • go to this directory cd ~/dlq
  • copy the code from MQ cp -r /opt/mqm/samp/dlq/* .
  • make the directory rw chmod +w ~/dlq
  • install a yacc compiler sudo apt install byacc
  • issue the make command make all -f amqodqx.mk

If you get

gcc -m64 -I. -I/opt/mqm/inc -c -o amqodqka.o amqodqka.c
Assembler messages:
Fatal error: can’t create amqodqka.o: Permission denied

Your directory is not read/write. Use the chmod command.

If you get

yacc amqodqma.y
make: yacc: Command not found

You need to install a yacc compiler. I used sudo apt install byacc .

I dont undestand why I got this message, because the build for the yacc files is commented out. It may be there is a default make for yacc.

The YACC ( compiler generator) produces a c file. If you are not changing the keywords, you could rename the .y and .l files, so they are not used, and so use the shipped c files. In this case you will not need the yacc compiler.

Run it

  • export ODQ_MSG=amqsdlq.msg This is needed for the message catalog
  • ./amqsdlq SYSTEM.PROTECTION.ERROR.QUEUE QMA < /home/colinpaice/mqams/dlq.rul
    • for a while it failed with a syntax error in line 1 of the file, I rebuilt it, then suddenly it worked.
    • Ctrl-c ended it
  • ./amqsdlq ? gave me a segmentation core dump

I changed the make file to build a client

cc -m64 -o amqsdlqc… -lmqic_r

You should ensure you use a TLS channel if you do run this as a client.

I have a message on the AMS DLQ – what can I do about it?

If AMS has problems with a protected message, AMS can put the message on the SYSTEM.PROTECTION.ERROR.QUEUE queue. This blog post discusses what you can do about it. I consider this a hard problem – not in the same league as trying to simulate the beginnings of the Universe – more like climbing Ben Nevis mountain in Scotland, when you are only used to walking down to the shops.

What are the problems?

There are several problems you need to consider

  1. Why is the message on the queue? Is it a problem with the putting application, or with the getting environment?
  2. Which user had the problem. For example it may not be obvious which application instance had a problem, if applications come in through one channel, and many users have the same MCA userid.
  3. What you need to do about it to get the message reprocessed, and prevent future problems.

Why is the message on the queue

The messages could be on the queue because

  • The certificate was signed, but the DN of the signer is not in the setmqspl list of authorised signers (-a). This is an example of an MQ configuration problem
  • The user getting a message was not able to verify the signers certificate sent in the message, for example it is missing the CA of the signer, or missing the signers self signed certificate. This is an example of a user’s configuration problem.
  • The message was encrypted, but the user getting the message did not have access to a private certificate to allow the message to be decrypted. The user’s DN needs to be added to the recipients when the message is put and enciphered (or the user needs to be stopped from getting messages from this queue). This is an example of the putting of the message message is missing configuration information.

What other information is there to help me?

If you know which id had the problem, there should be information in the error logs. For example a problems within a Java JMS client program may write to mqjms.log.0. A local application may write to the queue manager’s error log, for example /var/mqm/qmgrs/QMA/error/*01*

In the mqjms.log.0 I got

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.prot.MessageProtectionBCImpl
The receiver of this encrypted message is not on the message recipient list ‘CN=ja2,O=aaaa,C=GB’
The certificate of a user that is receiving a message is not on the message RecipientsInfo list.

Verify that the user is on a recipients list in a security policy definition.

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.intercept.JmqiGetInterceptorImpl
The IBM MQ Advanced Message Security Java interceptor failed to unprotect the received message.
An error occurred when the IBM MQ Advanced Message Security Java interceptor was unprotecting the received message.

See subsequent messages in the exception for more details about the cause of the error

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.service.EseMQServiceImpl
The IBM MQ Advanced Message Security interceptor has put a defective message on error handling queue ‘SYSTEM.PROTECTION.ERROR.QUEUE ‘.

(On z/OS the messages are less helpful.)

On the SYSTEM.PROTECTION.ERROR.QUEUE queue there was a message with a Dead Letter Header (DLH) with a reason 2063 0x0000080f MQRC_SECURITY_ERROR.

From the MQMD you can see the time the message was put to the queue, the putting application, and the user identification. This userIdentified may have been set for example by the channel MCAUSER, or CHLAUTH, and so you do not always know where the original request came from.

If you are testing then you will know what caused the error.

If you are in a production like environment, you know the application, and as you will have configured all user keystores the same you may not need to know which specific user caused the problem. If there is a problem with a missing certificate, then you fix the problem, redeploy the keystore to all your users (as part of your automated process) and try again.

How do you tell what the problem is.

Your systems administrator needs a process for extracting meta data about messages, while keeping the application payload protected. You could build a process around displaying the recipients and signer from How do I find the recipients and signer of an AMS message? The systems administrator needs to know

  • the original queue name with the problem
  • the time, date, and user that put the message
  • the msgid and correlid of the message – so if you put it back on the queue, you know which message to process.
  • from the message, the type of protection: Integrity (it was signed), Encryption (it was encrypted), Privacy (it was encrypted with a signed payload)
  • any id information from the message, such as recipient DN’s, and the signer DN. See this post.
  • you may have to have some special processing to decrypt the Privacy payload, just to extract the signer information.

If this process can be automated, then any application content can be kept secure.

With this information and your “up to date application work book” (do you have one of these?) , you should be able to identify the problem.

Once you have fixed the root cause of the problem….

The fix may be to change the setmqspl to add an authorised signer, or to add a certificate to the recipients keystore, you need to get the message reprocessed.

  • You get the specific message from the message id and correlid.
  • You need to remove the DLQ header from the message.
  • You need some special set up for the queue. If you try to put to the original queue, it will get the AMS protection again, for example re-encrypted or resigned. You need a queue alias so you put to the queue alias bypassing any AMS processing.

There are lots of things you need to consider, which is why I consider this a hard problem.

This application would be a good example where message handle is used to “move” the message.

MQCRTMH(hConn,&cmho,&hMsg,&CompCode,&Reason);

gmo.MsgHandle = hMsg;

MQGET(hConn,….);

pmo.Action = MQACTP_FORWARD;

pmo.OriginalMsgHandle =hmsg

MQPUT(…)

I found Learn to code the MQ Message Property MQI calls from MQGEM software useful in understanding message handle and message properties, and how to delete the DLQ header.

How do I find the recipients and signer of an AMS message?

When you have a problem trying to decrypt/get a protected message, the message can end up on the AMS dead letter queue. This blog post explains how to look into the message and find the list of recipient, and the signer of the data.

This started off as a short blog post, then as I added more background and more detailed instructions it just grew!

The sections are

Background on the format of the protected message.

The protected message conforms to the PKCS#7 protocol as defined here.

The specification defines six formats, of which three are:

  • signed data
  • envelopedData where data is wrapped in an opaque envelope – it has been encrypted
  • signedAndEnveloped.

These may look like the AMS classes: Integerity, Confidential, and Privacy; close – but no cigar. AMS does the signing before encryption, and so only signed data, and envelopedData are used. To see who signed a privacy message, you have to decrypt it.

I wrote a program to browse the messages on a queue, and write the protected data out to a Linux file. You can use openssl tools to explore file.

There is a PDMQ header at the front of the data. This has a variable length

  • at offset 0 is “PDMQ” in ASCII.
  • like other variable length MQ structures, at offset 8 is an MQLONG StrucLength. Don’t forget you may need to convert from big-endian to little-endian on some platforms.
  • The data after the header is the data is ASN.1 encoded, starting with x’30’.

30 second overview of ASN.1 encoding

ASN.1 encoding takes data like “typeOfRequest,{[id1,key1],[id2,key2]} and converts it into

  • a SEQUENCE (the items have an order) of typeOfRequest, keys
  • where keys are a SET (because it does not matter if the order is id1, id2, or id2, id1)
  • ID is an integer…

Terms like countryName C=GB map into a String of type 2.5.4.6 with length 2 and value GB.

When this is decoded it take 2.5.4.6 and says this is CountryName. You can look up the id here. There is a document with popular ones in it.

You can use openssl to format a file of ASN.1 encoded data and interpret the values. For example

openssl asn1parse -inform DER -in colin.pkj -i -offset 104 > asn1

says

  • openssl asn1parse this command
  • -inform DER in format is decimal encoded (not base 64)
  • -in colin.pkj this is the input file
  • -i says indent the output to prettify it
  • -offset 104 skip over the PDMQ header
  • > asn1 write the output to the file asn1

When I changed my program to skip over the PMDQ header and write out just the data, I didn’t need the -offset parameter.

I’ve also used this to display the contents of a certificate in .der format.


What does signed data look like?

The spec says signed data looks like

ContentInfo ::= SEQUENCE {
     contentType ContentType,
     content
}
where contentType is SignedData... 
SignedData ::= SEQUENCE {
     version Version,
     digestAlgorithms DigestAlgorithmIdentifiers,
     contentInfo ContentInfo,
     certificates
     signerInfos SignerInfos
 }

The start of the output of the openssl asn1parse command for an integrity message contained

    0:d=0  hl=4 l=1723 cons: SEQUENCE          
    4:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-signedData
   15:d=1  hl=4 l=1708 cons:  cont [ 0 ]        
   19:d=2  hl=4 l=1704 cons:   SEQUENCE          
   23:d=3  hl=2 l=   1 prim:    INTEGER           :01
   26:d=3  hl=2 l=  15 cons:    SET               
   28:d=4  hl=2 l=  13 cons:     SEQUENCE          
   30:d=5  hl=2 l=   9 prim:      OBJECT            :sha256
   41:d=5  hl=2 l=   0 prim:      NULL              
   43:d=3  hl=2 l=  20 cons:    SEQUENCE          
   45:d=4  hl=2 l=   9 prim:     OBJECT            :pkcs7-data
   56:d=4  hl=2 l=   7 cons:     cont [ 0 ]        
   58:d=5  hl=2 l=   5 prim:      OCTET STRING      :AAAAA

The columns are

  • offset from the start
  • d= nesting depth, so a sequence within a set, within a sequence would be d=3
  • dl= length of length field
  • l= length of the field
  • cons|prim. I think of these as
    • cons is construction … SEQ, SET etc.
    • prim is primary data
  • SEQUENCE, OBJECT, INTEGER type of value

Looking at the output

  • at offset 4 we have ContentType which is :pkcs7-signedData
  • at offset 23 we have the SignedData version :01
  • at offset 30 we have a SET of digestAlgorithms, there is one in the SEQUENCE and it has the value :sha256
  • at offset 45 we have contentInfo :pkcs7-data
  • at offset 58 we have the message data, the very boring AAAAA.

After the data we have signer info

SignerInfo ::= SEQUENCE {
     version Version,
     issuerAndSerialNumber IssuerAndSerialNumber,
     digestAlgorithm DigestAlgorithmIdentifier,
     authenticatedAttributes
       [0] IMPLICIT Attributes OPTIONAL,
     digestEncryptionAlgorithm
       DigestEncryptionAlgorithmIdentifier,
     encryptedDigest EncryptedDigest,
     unauthenticatedAttributes
       [1] IMPLICIT Attributes OPTIONAL }

Extracting the interesting fields out of the data (to make it easier to read)

  • Certificate serial number: 01B0
  • Issuer:
    • countryName:GB,
    • organizationName:SSS,
    • organizationalUnitName: CA,
    • commonName:SSCARSA1024
  • Not before time: 210328172922Z
  • Not after time: 231223172922Z
  • Subject:
    • countryName:GB,
    • organizationName:aaaa,
    • commonName: ja2
  • digestAlgorithm: rsaEncryption
  • X509v3 Key Usage:….
  • X509v3 Subject Alternative Name…
  • digestEncryptedAlgorithm: sha256WithRSAEncryption

What does encrypted data look like?

The spec says

EnvelopedData ::= SEQUENCE {
     version Version,
     recipientInfos RecipientInfos,
     encryptedContentInfo EncryptedContentInfo }

   RecipientInfos ::= SET OF RecipientInfo

   EncryptedContentInfo ::= SEQUENCE {
     contentType ContentType,
     contentEncryptionAlgorithm
       ContentEncryptionAlgorithmIdentifier,
     encryptedContent
       [0] IMPLICIT EncryptedContent OPTIONAL }

Instead of listing the DN’s of the recipients, it listed CA certificate + serial number (which equates to the same thing). As a self signed has signed itself, there is an entry for the self signed.


4:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-envelopedData

23:d=3 hl=2 l= 1 prim: INTEGER :00

42:d=8 hl=2 l= 9 cons: SEQUENCE
self signed 
C=GB,O=AAA,CN=colinPaice,  serial=606439D5
44:d=9 hl=2 l= 3 prim:     OBJECT          :countryName
49:d=9 hl=2 l= 2 prim:     PRINTABLESTRING :GB

57:d=9 hl=2 l= 3 prim:    OBJECT            :organizationName
62:d=9 hl=2 l= 3 prim:    PRINTABLESTRING   :AAA  

71:d=9 hl=2 l= 3 prim:    OBJECT            :commonName
76:d=9 hl=2 l= 11 prim:   PRINTABLESTRING   :colin paice
89:d=6 hl=2 l= 4 prim:    INTEGER           :606439D5

97:d=6 hl=2 l= 9 prim:    OBJECT            :rsaEncryption

the following is the encrypted key for this user
110:d=5 hl=3 l= 128 prim: OCTET STRING [HEX DUMP]:861C...F2
241:d=4 hl=4 l= 348 cons: SEQUENCE
245:d=5 hl=2 l= 1 prim: INTEGER :00
248:d=5 hl=2 l= 68 cons: SEQUENCE

This is the issuer C=GB,O=SSS,OU=CA,CN=SSCARSA1024, serial 1B0
256:d=9 hl=2 l= 3 prim:    OBJECT          :countryName
261:d=9 hl=2 l= 2 prim:    PRINTABLESTRING :GB
269:d=9 hl=2 l= 3 prim:    OBJECT          :organizationName
274:d=9 hl=2 l= 3 prim:    UTF8STRING      :SSS
283:d=9 hl=2 l= 3 prim:    OBJECT          :organizationalUnitName
288:d=9 hl=2 l= 2 prim:    UTF8STRING      :CA
296:d=9 hl=2 l= 3 prim:    OBJECT          :commonName
301:d=9 hl=2 l= 11 prim:   UTF8STRING      :SSCARSA1024
314:d=6 hl=2 l= 2 prim:   INTEGER          :01B0
318:d=5 hl=2 l=13 cons: SEQUENCE          
320:d=6 hl=2 l= 9 prim:  OBJECT            :rsaEncryption
the following is the encrypted key for this user
333:d=5 hl=4 l=256 prim:OCTET STRING      [HEX DUMP]:35..7BF

This is for Issuer O=TEMP, OU=TEST,CN=TEMP4Certification Authority Serial = 5D
593:d=4 hl=4 l= 354 cons: SEQUENCE          
597:d=5 hl=2 l=   1 prim: INTEGER           :00
608:d=9 hl=2 l=   3 prim:   OBJECT            :organizationName
613:d=9 hl=2 l=   4 prim:   PRINTABLESTRING   :TEMP
623:d=9 hl=2 l=   3 prim:   OBJECT            :organizationalUnitName
628:d=9 hl=2 l=   4 prim:   PRINTABLESTRING   :TEST
638:d=9 hl=2 l=   3 prim:   OBJECT            :commonName
643:d=9 hl=2 l=  28 prim:   PRINTABLESTRING   :TEMP4Certification Authority
673:d=6 hl=2 l=   1 prim:  INTEGER           :5D
678:d=6 hl=2 l=   9 prim:  OBJECT            :rsaEncryption
the following is the encrypted key for this user
691:d=5 hl=4 l= 256 prim:OCTET STRING      [HEX DUMP]:6A...68
951:d=3 hl=2 l=  76 cons: SEQUENCE          
953:d=4 hl=2 l=   9 prim:  OBJECT            :pkcs7-data
964:d=4 hl=2 l=  29 cons:  SEQUENCE          
966:d=5 hl=2 l=   9 prim:   OBJECT            :aes-128-cbc
977:d=5 hl=2 l=  16 prim:   OCTET STRING      [HEX DUMP]:9D...543A

There were three recipients configured for the queue (setmqspl -r option)

  • CN=ja2, O=aaaa, C=GB. See offset 256 to 333, giving the Issuer, the certificate serial number, encryption type, and the hex dump of the encrypted key.
    • Serial Number: 01 B0
    • Issued by: CN=SSCARSA1024, OU=CA, O=SSS, C=GB
  • CN=colin paice, O=AAA, C=GB See offset 44 to 110, giving the Issuer (this is self signed), the certificate serial number, encryption type, and the hex dump of thea encrypted key.
    • Serial Number: 60 64 39 D5
    • Issued by: CN=colin paice, O=AAA, C=GB
  • CN=ADCDB,O=SSS. See offset 597 to 691, giving the Issuer, the certificate serial number, encryption type, and the hex dump of the encrypted key.
    • Serial Number: 5D
    • Issued by: CN=TEMP4Certification Authority, OU=TEST, O=TEMP

It was not easy to take the issuer information and map this to the Subject. I had to go through the whole of my keystore to find the information.

How to map Issuer+ serial to subject

For .jks keystores you can use

keytool -list -keystore ja2.jks -storepass zpassword -v |grep ‘Serial\|Issuer:\|Owner:’ | less

and search for the serial Issuer CN, and the serial number which follows it. For example

Owner: CN=ja2, O=aaaa, C=GB
Issuer: CN=SSCARSA1024, OU=CA, O=SSS, C=GB
Serial number: 1b0

For .CMS I used a bash shell script to extract the contents, then display the details of each label.

ks="/home/colinpaice/mq/zzserver.kdb"
pw="password"

z=runmqakm -cert -list -db $ks -pw $pw | awk '{print $2}'
for x in $z;do
     runmqakm -cert -details -db $ks -pw $pw -label $x |grep 'Serial\|Issuer\|Subject'
done

What encryption algorithm was used?

At offset 966 is

966:d=5 hl=2 l= 9 prim: OBJECT :aes-128-cbc

The aes128 matches the -e AES128 option on the queue.

What does privacy look like?

Just like encrypted. it has

955:d=4 hl=2 l= 9 prim: OBJECT :pkcs7-signedData

instead of

953:d=4 hl=2 l=9 prim: OBJECT:pkcs7-data

The signing data is encrypted, so cannot be viewed.

Decrypting the data

If you have a file of the data, you can use openssl cms to decrypt it. The example below uses the private key for ja2 in ja2.key.pem. ( I knew it was ja2’s key file from the information above).

openssl cms -decrypt -in colin.pkj -inform der -inkey ~/ssl/ssl2/ja2.key.pem -out colin.der

Now that it is decrypted, display the signed package

openssl asn1parse -inform DER -in colin.der -i

and I can see information about the signer: the issuer, the signer and the signing time; and the payload.

Using Java to look at the data

You can use Java to process the message, using code from bouncycastle (used by MQ).

Read the message as a byte array (data) and use

import org.bouncycastle.asn1.ASN1InputStream;
import org.bouncycastle.asn1.ASN1Primitive;
import org.bouncycastle.asn1.util.ASN1Dump;   
....  
ASN1InputStream ais = new ASN1InputStream(data);
ASN1Primitive obj = ais.readObject();
System.out.println(ASN1Dump.dumpAsString(obj, true));
       

This produces output like the ans1parse above but without the conversion of 2.5.4.6 to CountryName etc. You can either guess the meaning, or look it up here.


ObjectIdentifier(1.2.840.113549.1.7.2)
 Tagged [0]
  Sequence
   Integer(1)
   Set
    Sequence
     ObjectIdentifier(2.16.840.1.101.3.4.2.1)
     NULL
   Sequence
    ObjectIdentifier(1.2.840.113549.1.7.1)
    Tagged [0]
     DER Octet String[3] 
      424242 BBB
    Tagged [0]
   Sequence
    Sequence
     Tagged [0]
      Integer(2)
      Integer(432)
      Sequence
       ObjectIdentifier(1.2.840.113549.1.1.11)
       NULL
      Sequence
       Set
        Sequence
         ObjectIdentifier(2.5.4.6)
         PrintableString(GB) 
       Set
        Sequence
         ObjectIdentifier(2.5.4.10)
          UTF8String(SSS) 
       Set
       Sequence
        ObjectIdentifier(2.5.4.11)
        UTF8String(CA) 
       Set
        Sequence
         ObjectIdentifier(2.5.4.3)
         UTF8String(SSCARSA1024) 
       Sequence
        UTCTime(210328172922GMT+00:00) 
        UTCTime(231223172922GMT+00:00) 

Adding more users to AMS

Having got a basic AMS set up working (on z/OS), it takes a bit of planning (and getting your head round it) to add more users and get the certificates in the right place when you have more than one queue manager. For example for someone to get an encrypted message the putting userid on the remote queue manager needs the getter’s public certificate. If there are 1000 potential putters, you start to see the complexity of the problem.

As I have tried to implement AMS across my baby enterprise I’ve realised you need a process for maintaining the key stores. When using multiple queue managers you have to remember which queue manger needs which options on the setmqspl command.

Basic key store management

z/OS key rings

On z/OS, each local userid has its own private key on its own keyring, with all of the shared public keys on the xxxxAMSM userid’s keyring (userid/drq.ams.keyring). Update the xxxxAMSM’x keyring, and every one picks up the change – great – easy.

You can have multiple queue managers in a single z/OS image or a sysplex, each with the xxxxAMSM address space. If they use the same started task userid, they will share the same keyring. (So to have queue managers use a different keyring, they will need a different userid.)

If any of the users are outside of the RACF key ring, you may need to export public certificates and send them to other environments: different sysplexes, or different midrange machines. The certificate will need to be imported on these other machines.

Midrange key stores

With midrange, a userid needs a key store with the private and public keys. With multiple machines you need a process to update and distribute key stores. For example:

  • Create a key store in a central site. Do all maintenance to this key store, adding and removing certificates as required.
  • Securely distribute this to all machines that are using AMS.
  • The key store which has been distributed may have the public key for the user. You need to remove it, from the downloaded keystore and add the private key from the local machine.
    • runmqakm -cert -delete -db /home/colinpaice/.mqs/shared.kdb -type kdb -pw passw0rd -label COLIN
    • runmqakm -cert -export -db $private.kdb -pw passw0rd -label COLIN -target COLIN.key.p12 -target-type pkcs12 -target_pw zpassword
    • runmqakm -cert -import -target /home/colinpaice/.mqs/shared.kdb -target_type kdb -pw zpassword -file COLIN.key.p12 -type p12
    • when you import the private key you can use -label COLIN -new_label COLIN2 to give it a new label.
    • You have to make the key the default using runmqakm -cert -setdefault (even though this is deprecated), because without this, the first certificate in the keystore is used..
  • Rename the shared key stores; current to old, new to current.
  • Restart the client.
  • Check it all works, delete the old key store.

If you add a new user, you may need to send the user’s public key to the cental site, to get it distributed to the mid range machines, and send it to the z/OS systems to add to the xxxxAMSM key rings.

In my case I just used one client, and updated the client’s key store directly.

Adding a new user for putting messages to a queue

Signed messages

When the data is signed, the recipient needs:

  • the Certificate Authority of the putter (the signer) – which the recipient may already have
  • or, when the putter has used a self signed certificate, the recipient needs the public part of the putter’s self signed certificate.

When using a CA signed certificate you may not need to export anything if the receiving end already has the CA certificate in the key store or keyring.

To export and import the Certificate authority.

  • To export a CA certificate from z/OS, certificate see here, the certificate owner will typically be CERTAUTH
  • To export a putter’s mid range CA certificate, see here.
  • To import a CA certificate on z/OS, the certificate owner could be a userid, site, or CERTAUTH (check with your security administrator). You might not have authority to use SITE or CERTAUTH.
  • To import a CA certificate on mid range, see here.

When using a self signed certificate (not a good idea for production)

  • export the putter’s public certificate
  • import it into the midrange key stores
  • import it to any other z/OS systems, and add it to the xxxxAMSM keyring.

You can use

  • To export a putter’s z/OS certificate see here.
  • To export a putter’s mid range certificate, see here.
  • To import a certificate on z/OS for a getter, see here. Once the certificate has been imported into the xxxxAMSM keyring it is available to all uses of the queue manager, and does not need to be imported again.
  • To import a certificate on mid range for a getter, see here.

Specify the list of authorised DNs

At the getting end, if the setmqspl -a option is used to specify the list of authorised signers, then the DN from the putter needs to be added to the list of authorised signers (-a…) . You should use dspmqspl -m mq -p q -export to get the current definition for the queue and use setmqspl to add the new -a data to that.

On z/OS, when you have used setmqspl to update the AMS configuration, you need to use the F xxxxAMSM,REFRESH ALL command.

Putting encrypted messages

The putter needs a private key in the keystore or key ring – though this is not used when encrypting messages.

At the putting end, the queue needs to be configured with the list of recipient DNs setmqspl … -r … .

The public certificates for these DNs need to be in the putter’s xxxxAMSM keyring, or in the mid range user’s key store.

  • To export the recipients’s z/OS certificate see here. You could store this in a central file so you can just reuse it rather than having to export it every time.
  • To export the recipients’s mid range certificate, see here. You could store this in a central file so you can just reuse it rather than having to export it every time.
  • To import the certificate on z/OS for the putter, see here. You should only need to do this once per keyring and user.
  • To import the certificate on mid range for the putter, see here. This is where you need a process for a centralised key store containing all of the public keys your organisation needs, and distributing this to all of the users. You may have a cms key store (for C programs) and a JKS key store for Java programs.

Adding a new user for getting messages from a queue

Getting signed messages

If other userids are getting signed messages from this queue there may be no additional configuration on the getting queue manager.

If you are using setmqspl … -a … see Specify the list of authorised DNs above. It may already be set if other userids are configured to get messages from this queue.

When using a CA signer certificate you may not need to export anything if the receiving end already has the CA certificate in the key store or keyring.

If the getting end does not have the CA see To export and import the Certificate authority above.

If the putter is using self signed, see When using a self signed certificate (not a good idea for production) above.

Getting encrypted messages

The getter’s private key is used to decrypt the message. Each putter needs to have access to the public key of the getter.

On the system doing the MQPUT,

  • the recipient’s DN needs to be configured on the putting queue manager using setmqspl with the -r (recipient) option for the queue
  • each user doing the MQPUT will need access to the recipient’s public certificate in its key store or the xxxxAMSM keyring.
Specify the list of recipient DNs

On the putting queue manager you need to add the recipient to the list of the recipients for message. Use the command dspmqspl -m QMA -p AMSQ -export to display the existing configuration for the AMSQ queue, and add the DN of the new user to the list using the setmqspl command.

You need to do this on all squeue managers which can be putting to this queue manager

On z/OS, when you have used setmqspl to update the AMS configuration, you need to use the F xxxxAMSM,REFRESH ALL command.

Get the recipients public certificate in the putters’ keyrings and key stores.
  • To add a z/OS user’s public certificate to the xxxxAMSM’s keyring in the same RACF database, see here.
  • To export the z/OS certificate from a recipient, see here.
  • To export the mid range certificate from a recipient, see here.
  • To import a certificate for putters on z/OS, see here.
  • To import a certificate for putters on mid range, see here.

Certificate and keyring management for AMS.

I started writing up how to use AMS, and found I had written a lot about how to manage certificate and key rings from a z/OS perspective. It made more sense to write this up in its own blog post.

When using AMS, you may have to export public certificates to other z/OS images, and to mid range machines.

Basic key store management

z/OS key rings

On z/OS, each local userid has its own private key on its own keyring, with all of the shared public keys on the xxxxAMSM userid’s keyring. Update the xxxxAMSM’x keyring, and every one picks up the change – great – easy.

The certificate used to sign is the default certificate in the user’s keyring.

You can have multiple queue managers in a single z/OS image or a sysplex, each with the xxxxAMSM address space. If they use the same started task userid, they will share the same keyring. (So to have queue managers use a different keyring, they will need a different userid.)

Midrange key stores

With midrange, a userid needs a key store with the private and public keys. With multiple machines you need a process to update and distribute key stores. For example:

  • Create a key store in a central site. Do all maintenance to this key store, adding and removing certificates as required.
  • Securely distribute this to all machines tat use AMS.
  • On each user’s machine add the user’s private key to this new copy of the key store.
  • Rename the key stores; current to old, new to current.
  • Restart the client.
  • Check it all works, delete the old key store.

In my case I just used one client, and updated the client’s key store directly.

A user can point to different key stores, so you need keep in mind that it is the user’s certificate, not the userid that does signing, encryption etc..

When do I need to export a public certificate?

Signing

  • If the certificate is self signed, the recipients will need a copy of the public certificate
  • If the certificate is signed by a Certificate Authority the recipient needs a copy of the CA certificate, and may already have it.

As a recipient of encrypted data

  • Each sender needs the public certificate (and public key) of each recipient, to be able to encrypt the payload.

Connecting the certificate in the same RACF database.

If the xxxxAMSM keyring is in the same RACF database as the the user’s certificate you just need to connect it to the keyring

RACDCERT ID(START1) connect(ID(ADCDC) LABEL(‘AMSZ’) –
RING(drq.ams.keyring) USAGE(SITE))

Exporting a certificate from a RACF database.

You need to export the certificate to a dataset. You can use binary format, or base64 encoded. I use base64 encoded.

RACDCERT ID(ADCDC) EXPORT(LABEL(‘AMSZ’))-
DSN(‘ADCDC.AMSZ.PEM’) FORMAT(CERTB64) PASSWORD(‘password’)

This file needs to be sent to the other systems and imported into the key stores.

Import a certificate into z/OS

For z/OS you need to import the public certificate to a userid. You might have a userid just for these certificate, or a userid for enterprise (BANK1, BANK2) etc. I’ve used ADCDA below.

The following example uses a certificate (for ja2) sent from a Linux machine. It was FTPed to z/OS into a data set ADCD.JA2.PEM.

#delete any old instance
RACDCERT ID(ADCDA) DELETE (LABEL(‘LINUXJA2’))

RACDCERT ADD(‘ADCD.JA2.PEM’) –
ID(ADCDA) WITHLABEL(‘LINUXJA2’) TRUST

RACDCERT ID(START1) CONNECT(RING(drq.ams.keyring ) –
ID(ADCDA) LABEL(‘LINUXJA2’))

RACDCERT LIST (LABEL(‘LINUXJA2’)) ID(ADCDA)

SETROPTS RACLIST(DIGTCERT) REFRESH

Extract a certificate from a mid-range key store

To extract from a CMS key store.

runmqakm -cert -extract -db my.kdb -pw passw0rd -label COLIN -target LINUXID.pem -format ascii

To extract from a .jks key store

runmqckm -cert -extract -db my.jks -pw zpassword -label COLIN -target LINUXID.PEM -format ascii -type jks

FTP the file to the remote systems and import it.

Importing a certificate into a mid-range key store

Follow the key store update process to add it to each midrange key store. The certificate was exported from z/OS and FTPed to the Linux machine as file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem

For the .jks key store (used by Java).

/opt/mqm/bin/runmqckm -cert -add -db trust.jks -type jks -file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem -label zADCDC -pw zpassword

#It is worth checking the certificate to make sure the certificate chain is present.

/opt/mqm/bin/runmqckm -cert -validate -db trust.jks -type jks -pw zpassword -label zADCDC

For a CMS keystore

/opt/mqm/bin/runmqckm -cert -add -db $key.kdb -file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem -label zADCDB -pw passw0rd