checkAMS: program to check your AMS defintions are consistent with z/OS keyring

A C program to verify that the certificates in MQ AMS configuration are in a RACF keyring. See here.

Overview of program

With AMS you specify the Distinquished Names(DN) of users who are allowed to sign or encrypt MQ messages. The certificates for these DN’s need to be in the xxxxAMSM’s drq.ams.keyring. If they are not present, or have problems, such as they are not valid, the messages from AMS are not very helpful. The messages are as helpful as “one of the DN’s in the configuration has a problem but I am not telling you which DN it was, nor what the problem was”.

CheckAMS has two parts:

  1. Provide a useful list of information in the keyring
  2. Takes the output of the AMS dspmqspl command, and checks the DN’s are in the key store

Provide a useful list of the contents of a keyring.

With the RACDCERT commands you can list the contents of a keyring, for example owner and label; and you can display details about a certificate, such as the DN of the subject, and the Certificate Authority, but you cannot issue one command to display all the important information, nor ask, “is the DN for this issuer in the keystore”.

Example output from checkAMS, listing certificates in keyring:

Subject CN=SSCARSA1024,OU=CA,O=SSS,C=GB                                                         
Issuer  CN=SSCARSA1024,OU=CA,O=SSS,C=GB                                                         
Self signed                                                                                     
Valid date range 21/02/13 12:32:33 to 24/02/13 12:32:33                                         
Owner irrcerta/LINUXCA                                                                          
Usage:Certauth Status:Trust                                                                     
                                                                                                           
Subject CN=colin,OU=longou,O=SSS                                                                
Issuer  CN=TEMP4Certification Authority,OU=TEST,O=TEMP                                          
Valid date range 21/03/25 00:00:00 to 22/03/25 23:59:59                                         
Owner COLIN/TEST                                                                                
Usage:Site Status:Trust      

The first certificate is owned by irrcerta and has label LINUXA. Userid irrcerta means it belongs to CERTAUTH. The certificate is self signed, and has a long validity date. It has a usage of CERTAUTH, and is trusted.

The second certificate belongs to userid COLIN, it has label TEST. It has a subject DN of Subject CN=colin,OU=longou,O=SSS, and was issued by CN=TEMP4Certification Authority,OU=TEST,O=TEMP. It has a usage of Site, and is trusted.

Check the AMS set up

The program takes as input the output of the dspmqspl -m… -export command, and checks the DN against certificates in the keyring.

Example output

Userid START1, ring drq.ams.keyring                                                                                  
* Exported on Mon Mar 29 09:23:31 2021                                                                               
                                                                                                                      
dspmqspl -m CSQ9  -export                                                                                          
setmqspl -m CSQ9                                                                                                     
 -p AMSQ                                                                                                             
 -s SHA256                                                                                                           
 -a "CN=COLIN,O=SSS"                                                                                                 
   Owner COLIN/AMS Usage:Site Status:Trust Valid date range 21/03/21 00:00:00 to 22/03/21 18:45:00                  
 -a "O=aaaa, C=GB,CN=ja2"                                                                                            
 ! O=aaaa,C=GB,CN=ja2 Not found in key ring                                                                           
 -e AES256                                                                                                           
 -r "CN=COLIN,O=SSS"                                                                                                 
  Owner COLIN/AMS Usage:Site Status:Trust Valid date range 21/03/21 00:00:00 to 22/03/21 18:45:00                  
 -r "CN=ADCDB,O=SSS"

This shows the keyring was START1/drq.ams.keyring.

It prints out the exported file, and for the -a and -r records, it adds information about the certificate, or reports if it is not found.

It reports that “CN=COLIN,O=SSS” was found, the certificate belongs to userid COLIN,label AMS, it has usage of Site, it is trusted, and has a valid date.

It also reports O=aaaa,C=GB,CN=ja2 Not found in key ring This is because the definition in AMS has the wrong order. The standard order is CN=ja2,O=aaaa,c=GB. This certificate is in the keyring , but the program could not find it. I could not see a way of converting bad format DNs to good DNs.

Contents of package.

The package is on git.

FTP the amscheck.xmit.bit to z/OS as binary. Then use TSO receive indsn(amscheck.xmit) to create the load module in a PDS.

Upload runamsch, ccasmch, asmcheck. and parmlist.h to a PDS.

Edit and submit runamsch. It runs dspmqspl and puts the output into a temporary file. The parm PARM=’START1 drq.ams.keyring’ is for userid START1 and the keyring drq.ams.keyring. Your userid will need access to the userid’s keyring.

if you want to compile the program

If you want to compile the program, you can edit ccasmch, and change the SYSIN, and where the header file is imported from.

Planning for MQ Dead Letter Queue handling.

With MQ, if a message cannot be successfully delivered, it can be put on a Dead Letter Queue for later processing.

You can have multiple queues

  • The system dead letter queue, where the MQ puts messages it cannot processed,
  • Application dead letter queues, and application can put messages to a queue,
  • The AMS dead letter queue for messages which had errors during get or put, for example a certificate mismatch.

Messages can be put to these queues for a variety of reasons.

  • Transient problems
    • If a channel is putting a message to a queue, and the queue is full, then the channel can put the message to the Dead Letter Queue. The DLQ handler can then try to put the message to the original queue, and retry a number of times after an interval. If the queue full condition was transient, then the DLQ handler is likely to succeed. If an application stops processing a queue, you can get quickly get thousands of messages on the DLQ queue.
    • The queue is put disabled. A queue can be set to put disabled, for example to stop messages from going onto a queue during queue maintenance. Once the maintenance has been done the queue can have put enabled.
  • Administration
    • The putting channel is not authorised to put to the queue, so the message gets put to the DLQ. An administrator needs to check to see if the putter is allowed to put the message. If so, fix the security and put the message back on original queue. If not remove the message, and educate the developer.
    • An AMS protected message has a problem, for example an unauthorised user has signed a message, or the id getting the message does not have a certificate to decrypt a message. You need to resolve any local certificate problems, or send the original message back to the requester saying it is in error.
  • Application
    • The message is too large for the queue. The administrator needs to educate the developer and/or make the queue max message size larger.

You may have a policy that non persistent messages for a particular queue which end up on the dead letter queue should be purged. Persistent message for another queue should have special treatment.

You may want administrators to be able to look at the meta data about a message, destination queue, MSGID, the list of recipients who can decrypt a message; but not to look at the message content.

Setting up your environment to cover these areas need considerable planning.

Implementing a solution

You want to try to keep the main DLQ close to empty, for example if your DLQ fills up with non persistent inquiries, then putting an important persistent message to the DLQ may fail.

You can use the runmqdlq program on midrange or CSQUDLQH on z/OS, to specify rules for automatic processing of messages on the DLQ.

You can select on attributes like original destination queue name, the reason why the message was on the DLQ, userid in the MQMD; and specify an action

  • Retry the put to the original queue
  • Move to another queue
  • Purge it
  • Leave it

When a message is processed on the DLQ, the rules are applied, and the action of the first matching rule is applied. For example

DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(RETRY) RETRY(5)
DESTQ(MYQUEUE) REASON(MQRC_Q_FULL) ACTION(FWD) FWD(MYQUEUEOVERFLOW) HEADER(YES)

This says that if a messages destination was MYQUEUE, and the reason code was MQRC_Q_FULL, it retries the put to the queue, at most 5 times. After 5 attempts, the first rule is skipped, the second rule is used, and the message is forwarded to the queue MYQUEUEOVERFLOW keeping the DLQ header.

DEST(INQ*) PERSIST(MQPER_NON_PERSISTENT ACTION(DISCARD)

For message destination INQ* and non persistent messages, then just discard them.

DEST(INQ*) PERSIST(MQPER_PERSISTENT ACTION(LEAVE)

For message destination INQ* and persistent messages, then just leave them on the queue, for some other processing.

If runmqdlq or CSQUDLQH is restarted, then all processing is reset.

Generic rules

For transient type problems you may want to consider

  • Non persistent messages for a set of queues get purged, dont even try to put them back on the queue.
  • Persistent messages for INVOICE* queues get moved to INVOICE_DLQ queue, where you have another DLQ monitor running on the queue.

For administrator type problems

  • You could pass non persistent messages to an admin_DLQ_NP queue, and have a program which reads the meta data, and prints it to a file, then deletes the original message
  • You could pass persistent messages to an admin_DLQ_P queue
    • have a program which reads the meta data, and prints it to a file, and leaves the message on the queue.
    • Using the meta information resolve the problem.
    • Have another program which takes the msgid and correlid as input parameters, then puts the message on the original queue. (If there is only one message, you could use the default DLQ handler to do this.)

For AMS problems

  • This is complicated by having to use a different queue. If the DLQ handler tries to put to the AMS protected queue, it will be “protected” (enciphered) again. You need to use put the message to an alias queue, with the original queue as the target. On midrange Java and C clients can disable AMS processing, either by using an environment variable, or through the MQCLIENT.ini file. See here.
  • This is also complicated by possibly needing access to information in the payload, such as the list of recipients, and decrypting the message to get the DN of the signer.
    • Once you have resolved the problem, have another program which takes the msgid and correlid as input parameters, and puts the message on the alias queue (if there is only one message you could use the default DLQ handler to do this).

How do I check it I have got it right?

It is worth putting a process in place to monitor the depth of the dead letter queue, and if it does not become empty a few a minutes, display the contents of the queue, and add rules to handle the residual messages.

I do not think that IBM provides a list of return codes of messages that it puts onto the DLQ, I think you’ll have to go through the list (over 500!), and put a rule in place for each one. If an application invents its own return codes, you may need rules for these as well.

My quick look at the list includes

Common problems

2053 (0805) (RC2053): MQRC_Q_FULL
2056 (0808) (RC2056): MQRC_Q_SPACE_NOT_AVAILABLE
2071 (0817) (RC2071): MQRC_STORAGE_NOT_AVAILABLE

Other problems
2030 (07EE) (RC2030): MQRC_MSG_TOO_BIG_FOR_Q
2031 (07EF) (RC2031): MQRC_MSG_TOO_BIG_FOR_Q_MGR
2033 (07F1) (RC2033): MQRC_NO_MSG_AVAILABLE
2051 (0803) (RC2051): MQRC_PUT_INHIBITED
2052 (0804) (RC2052): MQRC_Q_DELETED
2053 (0805) (RC2053): MQRC_Q_FULL
2056 (0808) (RC2056): MQRC_Q_SPACE_NOT_AVAILABLE
2071 (0817) (RC2071): MQRC_STORAGE_NOT_AVAILABLE
2102 (0836) (RC2102): MQRC_RESOURCE_PROBLEM
2120 (0848) (RC2120): MQRC_CONVERTED_MSG_TOO_BIG
2141 (085D) (RC2141): MQRC_DLH_ERROR
2142 (085E) (RC2142): MQRC_HEADER_ERROR
2148 (0864) (RC2148): MQRC_IIH_ERROR
2149 (0865) (RC2149): MQRC_PCF_ERROR
2150 (0866) (RC2150): MQRC_DBCS_ERROR
[z/OS]2342 (0926) (RC2342): MQRC_DB2_NOT_AVAILABLE
[z/OS]2345 (0929) (RC2345): MQRC_CF_NOT_AVAILABLE
[z/OS]2348 (092C) (RC2348): MQRC_CF_STRUC_AUTH_FAILED
[z/OS]2349 (092D) (RC2349): MQRC_CF_STRUC_ERROR
2362 (093A) (RC2362): MQRC_BACKOUT_THRESHOLD_REACHED

You may want common ones, such as queue full and not authorised, on a per queue basis, and all the less common ones, such as all of the z/OS ones all putting to one “admin” queue.

I have a message on the AMS DLQ – what can I do about it?

If AMS has problems with a protected message, AMS can put the message on the SYSTEM.PROTECTION.ERROR.QUEUE queue. This blog post discusses what you can do about it. I consider this a hard problem – not in the same league as trying to simulate the beginnings of the Universe – more like climbing Ben Nevis mountain in Scotland, when you are only used to walking down to the shops.

What are the problems?

There are several problems you need to consider

  1. Why is the message on the queue? Is it a problem with the putting application, or with the getting environment?
  2. Which user had the problem. For example it may not be obvious which application instance had a problem, if applications come in through one channel, and many users have the same MCA userid.
  3. What you need to do about it to get the message reprocessed, and prevent future problems.

Why is the message on the queue

The messages could be on the queue because

  • The certificate was signed, but the DN of the signer is not in the setmqspl list of authorised signers (-a). This is an example of an MQ configuration problem
  • The user getting a message was not able to verify the signers certificate sent in the message, for example it is missing the CA of the signer, or missing the signers self signed certificate. This is an example of a user’s configuration problem.
  • The message was encrypted, but the user getting the message did not have access to a private certificate to allow the message to be decrypted. The user’s DN needs to be added to the recipients when the message is put and enciphered (or the user needs to be stopped from getting messages from this queue). This is an example of the putting of the message message is missing configuration information.

What other information is there to help me?

If you know which id had the problem, there should be information in the error logs. For example a problems within a Java JMS client program may write to mqjms.log.0. A local application may write to the queue manager’s error log, for example /var/mqm/qmgrs/QMA/error/*01*

In the mqjms.log.0 I got

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.prot.MessageProtectionBCImpl
The receiver of this encrypted message is not on the message recipient list ‘CN=ja2,O=aaaa,C=GB’
The certificate of a user that is receiving a message is not on the message RecipientsInfo list.

Verify that the user is on a recipients list in a security policy definition.

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.intercept.JmqiGetInterceptorImpl
The IBM MQ Advanced Message Security Java interceptor failed to unprotect the received message.
An error occurred when the IBM MQ Advanced Message Security Java interceptor was unprotecting the received message.

See subsequent messages in the exception for more details about the cause of the error

5 April 2021 at 14:53:23 BST[main] com.ibm.mq.ese.service.EseMQServiceImpl
The IBM MQ Advanced Message Security interceptor has put a defective message on error handling queue ‘SYSTEM.PROTECTION.ERROR.QUEUE ‘.

(On z/OS the messages are less helpful.)

On the SYSTEM.PROTECTION.ERROR.QUEUE queue there was a message with a Dead Letter Header (DLH) with a reason 2063 0x0000080f MQRC_SECURITY_ERROR.

From the MQMD you can see the time the message was put to the queue, the putting application, and the user identification. This userIdentified may have been set for example by the channel MCAUSER, or CHLAUTH, and so you do not always know where the original request came from.

If you are testing then you will know what caused the error.

If you are in a production like environment, you know the application, and as you will have configured all user keystores the same you may not need to know which specific user caused the problem. If there is a problem with a missing certificate, then you fix the problem, redeploy the keystore to all your users (as part of your automated process) and try again.

How do you tell what the problem is.

Your systems administrator needs a process for extracting meta data about messages, while keeping the application payload protected. You could build a process around displaying the recipients and signer from How do I find the recipients and signer of an AMS message? The systems administrator needs to know

  • the original queue name with the problem
  • the time, date, and user that put the message
  • the msgid and correlid of the message – so if you put it back on the queue, you know which message to process.
  • from the message, the type of protection: Integrity (it was signed), Encryption (it was encrypted), Privacy (it was encrypted with a signed payload)
  • any id information from the message, such as recipient DN’s, and the signer DN. See this post.
  • you may have to have some special processing to decrypt the Privacy payload, just to extract the signer information.

If this process can be automated, then any application content can be kept secure.

With this information and your “up to date application work book” (do you have one of these?) , you should be able to identify the problem.

Once you have fixed the root cause of the problem….

The fix may be to change the setmqspl to add an authorised signer, or to add a certificate to the recipients keystore, you need to get the message reprocessed.

  • You get the specific message from the message id and correlid.
  • You need to remove the DLQ header from the message.
  • You need some special set up for the queue. If you try to put to the original queue, it will get the AMS protection again, for example re-encrypted or resigned. You need a queue alias so you put to the queue alias bypassing any AMS processing.

There are lots of things you need to consider, which is why I consider this a hard problem.

This application would be a good example where message handle is used to “move” the message.

MQCRTMH(hConn,&cmho,&hMsg,&CompCode,&Reason);

gmo.MsgHandle = hMsg;

MQGET(hConn,….);

pmo.Action = MQACTP_FORWARD;

pmo.OriginalMsgHandle =hmsg

MQPUT(…)

I found Learn to code the MQ Message Property MQI calls from MQGEM software useful in understanding message handle and message properties, and how to delete the DLQ header.

How do I find the recipients and signer of an AMS message?

When you have a problem trying to decrypt/get a protected message, the message can end up on the AMS dead letter queue. This blog post explains how to look into the message and find the list of recipient, and the signer of the data.

This started off as a short blog post, then as I added more background and more detailed instructions it just grew!

The sections are

Background on the format of the protected message.

The protected message conforms to the PKCS#7 protocol as defined here.

The specification defines six formats, of which three are:

  • signed data
  • envelopedData where data is wrapped in an opaque envelope – it has been encrypted
  • signedAndEnveloped.

These may look like the AMS classes: Integerity, Confidential, and Privacy; close – but no cigar. AMS does the signing before encryption, and so only signed data, and envelopedData are used. To see who signed a privacy message, you have to decrypt it.

I wrote a program to browse the messages on a queue, and write the protected data out to a Linux file. You can use openssl tools to explore file.

There is a PDMQ header at the front of the data. This has a variable length

  • at offset 0 is “PDMQ” in ASCII.
  • like other variable length MQ structures, at offset 8 is an MQLONG StrucLength. Don’t forget you may need to convert from big-endian to little-endian on some platforms.
  • The data after the header is the data is ASN.1 encoded, starting with x’30’.

30 second overview of ASN.1 encoding

ASN.1 encoding takes data like “typeOfRequest,{[id1,key1],[id2,key2]} and converts it into

  • a SEQUENCE (the items have an order) of typeOfRequest, keys
  • where keys are a SET (because it does not matter if the order is id1, id2, or id2, id1)
  • ID is an integer…

Terms like countryName C=GB map into a String of type 2.5.4.6 with length 2 and value GB.

When this is decoded it take 2.5.4.6 and says this is CountryName. You can look up the id here. There is a document with popular ones in it.

You can use openssl to format a file of ASN.1 encoded data and interpret the values. For example

openssl asn1parse -inform DER -in colin.pkj -i -offset 104 > asn1

says

  • openssl asn1parse this command
  • -inform DER in format is decimal encoded (not base 64)
  • -in colin.pkj this is the input file
  • -i says indent the output to prettify it
  • -offset 104 skip over the PDMQ header
  • > asn1 write the output to the file asn1

When I changed my program to skip over the PMDQ header and write out just the data, I didn’t need the -offset parameter.

I’ve also used this to display the contents of a certificate in .der format.


What does signed data look like?

The spec says signed data looks like

ContentInfo ::= SEQUENCE {
     contentType ContentType,
     content
}
where contentType is SignedData... 
SignedData ::= SEQUENCE {
     version Version,
     digestAlgorithms DigestAlgorithmIdentifiers,
     contentInfo ContentInfo,
     certificates
     signerInfos SignerInfos
 }

The start of the output of the openssl asn1parse command for an integrity message contained

    0:d=0  hl=4 l=1723 cons: SEQUENCE          
    4:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-signedData
   15:d=1  hl=4 l=1708 cons:  cont [ 0 ]        
   19:d=2  hl=4 l=1704 cons:   SEQUENCE          
   23:d=3  hl=2 l=   1 prim:    INTEGER           :01
   26:d=3  hl=2 l=  15 cons:    SET               
   28:d=4  hl=2 l=  13 cons:     SEQUENCE          
   30:d=5  hl=2 l=   9 prim:      OBJECT            :sha256
   41:d=5  hl=2 l=   0 prim:      NULL              
   43:d=3  hl=2 l=  20 cons:    SEQUENCE          
   45:d=4  hl=2 l=   9 prim:     OBJECT            :pkcs7-data
   56:d=4  hl=2 l=   7 cons:     cont [ 0 ]        
   58:d=5  hl=2 l=   5 prim:      OCTET STRING      :AAAAA

The columns are

  • offset from the start
  • d= nesting depth, so a sequence within a set, within a sequence would be d=3
  • dl= length of length field
  • l= length of the field
  • cons|prim. I think of these as
    • cons is construction … SEQ, SET etc.
    • prim is primary data
  • SEQUENCE, OBJECT, INTEGER type of value

Looking at the output

  • at offset 4 we have ContentType which is :pkcs7-signedData
  • at offset 23 we have the SignedData version :01
  • at offset 30 we have a SET of digestAlgorithms, there is one in the SEQUENCE and it has the value :sha256
  • at offset 45 we have contentInfo :pkcs7-data
  • at offset 58 we have the message data, the very boring AAAAA.

After the data we have signer info

SignerInfo ::= SEQUENCE {
     version Version,
     issuerAndSerialNumber IssuerAndSerialNumber,
     digestAlgorithm DigestAlgorithmIdentifier,
     authenticatedAttributes
       [0] IMPLICIT Attributes OPTIONAL,
     digestEncryptionAlgorithm
       DigestEncryptionAlgorithmIdentifier,
     encryptedDigest EncryptedDigest,
     unauthenticatedAttributes
       [1] IMPLICIT Attributes OPTIONAL }

Extracting the interesting fields out of the data (to make it easier to read)

  • Certificate serial number: 01B0
  • Issuer:
    • countryName:GB,
    • organizationName:SSS,
    • organizationalUnitName: CA,
    • commonName:SSCARSA1024
  • Not before time: 210328172922Z
  • Not after time: 231223172922Z
  • Subject:
    • countryName:GB,
    • organizationName:aaaa,
    • commonName: ja2
  • digestAlgorithm: rsaEncryption
  • X509v3 Key Usage:….
  • X509v3 Subject Alternative Name…
  • digestEncryptedAlgorithm: sha256WithRSAEncryption

What does encrypted data look like?

The spec says

EnvelopedData ::= SEQUENCE {
     version Version,
     recipientInfos RecipientInfos,
     encryptedContentInfo EncryptedContentInfo }

   RecipientInfos ::= SET OF RecipientInfo

   EncryptedContentInfo ::= SEQUENCE {
     contentType ContentType,
     contentEncryptionAlgorithm
       ContentEncryptionAlgorithmIdentifier,
     encryptedContent
       [0] IMPLICIT EncryptedContent OPTIONAL }

Instead of listing the DN’s of the recipients, it listed CA certificate + serial number (which equates to the same thing). As a self signed has signed itself, there is an entry for the self signed.


4:d=1 hl=2 l= 9 prim: OBJECT :pkcs7-envelopedData

23:d=3 hl=2 l= 1 prim: INTEGER :00

42:d=8 hl=2 l= 9 cons: SEQUENCE
self signed 
C=GB,O=AAA,CN=colinPaice,  serial=606439D5
44:d=9 hl=2 l= 3 prim:     OBJECT          :countryName
49:d=9 hl=2 l= 2 prim:     PRINTABLESTRING :GB

57:d=9 hl=2 l= 3 prim:    OBJECT            :organizationName
62:d=9 hl=2 l= 3 prim:    PRINTABLESTRING   :AAA  

71:d=9 hl=2 l= 3 prim:    OBJECT            :commonName
76:d=9 hl=2 l= 11 prim:   PRINTABLESTRING   :colin paice
89:d=6 hl=2 l= 4 prim:    INTEGER           :606439D5

97:d=6 hl=2 l= 9 prim:    OBJECT            :rsaEncryption

the following is the encrypted key for this user
110:d=5 hl=3 l= 128 prim: OCTET STRING [HEX DUMP]:861C...F2
241:d=4 hl=4 l= 348 cons: SEQUENCE
245:d=5 hl=2 l= 1 prim: INTEGER :00
248:d=5 hl=2 l= 68 cons: SEQUENCE

This is the issuer C=GB,O=SSS,OU=CA,CN=SSCARSA1024, serial 1B0
256:d=9 hl=2 l= 3 prim:    OBJECT          :countryName
261:d=9 hl=2 l= 2 prim:    PRINTABLESTRING :GB
269:d=9 hl=2 l= 3 prim:    OBJECT          :organizationName
274:d=9 hl=2 l= 3 prim:    UTF8STRING      :SSS
283:d=9 hl=2 l= 3 prim:    OBJECT          :organizationalUnitName
288:d=9 hl=2 l= 2 prim:    UTF8STRING      :CA
296:d=9 hl=2 l= 3 prim:    OBJECT          :commonName
301:d=9 hl=2 l= 11 prim:   UTF8STRING      :SSCARSA1024
314:d=6 hl=2 l= 2 prim:   INTEGER          :01B0
318:d=5 hl=2 l=13 cons: SEQUENCE          
320:d=6 hl=2 l= 9 prim:  OBJECT            :rsaEncryption
the following is the encrypted key for this user
333:d=5 hl=4 l=256 prim:OCTET STRING      [HEX DUMP]:35..7BF

This is for Issuer O=TEMP, OU=TEST,CN=TEMP4Certification Authority Serial = 5D
593:d=4 hl=4 l= 354 cons: SEQUENCE          
597:d=5 hl=2 l=   1 prim: INTEGER           :00
608:d=9 hl=2 l=   3 prim:   OBJECT            :organizationName
613:d=9 hl=2 l=   4 prim:   PRINTABLESTRING   :TEMP
623:d=9 hl=2 l=   3 prim:   OBJECT            :organizationalUnitName
628:d=9 hl=2 l=   4 prim:   PRINTABLESTRING   :TEST
638:d=9 hl=2 l=   3 prim:   OBJECT            :commonName
643:d=9 hl=2 l=  28 prim:   PRINTABLESTRING   :TEMP4Certification Authority
673:d=6 hl=2 l=   1 prim:  INTEGER           :5D
678:d=6 hl=2 l=   9 prim:  OBJECT            :rsaEncryption
the following is the encrypted key for this user
691:d=5 hl=4 l= 256 prim:OCTET STRING      [HEX DUMP]:6A...68
951:d=3 hl=2 l=  76 cons: SEQUENCE          
953:d=4 hl=2 l=   9 prim:  OBJECT            :pkcs7-data
964:d=4 hl=2 l=  29 cons:  SEQUENCE          
966:d=5 hl=2 l=   9 prim:   OBJECT            :aes-128-cbc
977:d=5 hl=2 l=  16 prim:   OCTET STRING      [HEX DUMP]:9D...543A

There were three recipients configured for the queue (setmqspl -r option)

  • CN=ja2, O=aaaa, C=GB. See offset 256 to 333, giving the Issuer, the certificate serial number, encryption type, and the hex dump of the encrypted key.
    • Serial Number: 01 B0
    • Issued by: CN=SSCARSA1024, OU=CA, O=SSS, C=GB
  • CN=colin paice, O=AAA, C=GB See offset 44 to 110, giving the Issuer (this is self signed), the certificate serial number, encryption type, and the hex dump of thea encrypted key.
    • Serial Number: 60 64 39 D5
    • Issued by: CN=colin paice, O=AAA, C=GB
  • CN=ADCDB,O=SSS. See offset 597 to 691, giving the Issuer, the certificate serial number, encryption type, and the hex dump of the encrypted key.
    • Serial Number: 5D
    • Issued by: CN=TEMP4Certification Authority, OU=TEST, O=TEMP

It was not easy to take the issuer information and map this to the Subject. I had to go through the whole of my keystore to find the information.

How to map Issuer+ serial to subject

For .jks keystores you can use

keytool -list -keystore ja2.jks -storepass zpassword -v |grep ‘Serial\|Issuer:\|Owner:’ | less

and search for the serial Issuer CN, and the serial number which follows it. For example

Owner: CN=ja2, O=aaaa, C=GB
Issuer: CN=SSCARSA1024, OU=CA, O=SSS, C=GB
Serial number: 1b0

For .CMS I used a bash shell script to extract the contents, then display the details of each label.

ks="/home/colinpaice/mq/zzserver.kdb"
pw="password"

z=runmqakm -cert -list -db $ks -pw $pw | awk '{print $2}'
for x in $z;do
     runmqakm -cert -details -db $ks -pw $pw -label $x |grep 'Serial\|Issuer\|Subject'
done

What encryption algorithm was used?

At offset 966 is

966:d=5 hl=2 l= 9 prim: OBJECT :aes-128-cbc

The aes128 matches the -e AES128 option on the queue.

What does privacy look like?

Just like encrypted. it has

955:d=4 hl=2 l= 9 prim: OBJECT :pkcs7-signedData

instead of

953:d=4 hl=2 l=9 prim: OBJECT:pkcs7-data

The signing data is encrypted, so cannot be viewed.

Decrypting the data

If you have a file of the data, you can use openssl cms to decrypt it. The example below uses the private key for ja2 in ja2.key.pem. ( I knew it was ja2’s key file from the information above).

openssl cms -decrypt -in colin.pkj -inform der -inkey ~/ssl/ssl2/ja2.key.pem -out colin.der

Now that it is decrypted, display the signed package

openssl asn1parse -inform DER -in colin.der -i

and I can see information about the signer: the issuer, the signer and the signing time; and the payload.

Using Java to look at the data

You can use Java to process the message, using code from bouncycastle (used by MQ).

Read the message as a byte array (data) and use

import org.bouncycastle.asn1.ASN1InputStream;
import org.bouncycastle.asn1.ASN1Primitive;
import org.bouncycastle.asn1.util.ASN1Dump;   
....  
ASN1InputStream ais = new ASN1InputStream(data);
ASN1Primitive obj = ais.readObject();
System.out.println(ASN1Dump.dumpAsString(obj, true));
       

This produces output like the ans1parse above but without the conversion of 2.5.4.6 to CountryName etc. You can either guess the meaning, or look it up here.


ObjectIdentifier(1.2.840.113549.1.7.2)
 Tagged [0]
  Sequence
   Integer(1)
   Set
    Sequence
     ObjectIdentifier(2.16.840.1.101.3.4.2.1)
     NULL
   Sequence
    ObjectIdentifier(1.2.840.113549.1.7.1)
    Tagged [0]
     DER Octet String[3] 
      424242 BBB
    Tagged [0]
   Sequence
    Sequence
     Tagged [0]
      Integer(2)
      Integer(432)
      Sequence
       ObjectIdentifier(1.2.840.113549.1.1.11)
       NULL
      Sequence
       Set
        Sequence
         ObjectIdentifier(2.5.4.6)
         PrintableString(GB) 
       Set
        Sequence
         ObjectIdentifier(2.5.4.10)
          UTF8String(SSS) 
       Set
       Sequence
        ObjectIdentifier(2.5.4.11)
        UTF8String(CA) 
       Set
        Sequence
         ObjectIdentifier(2.5.4.3)
         UTF8String(SSCARSA1024) 
       Sequence
        UTCTime(210328172922GMT+00:00) 
        UTCTime(231223172922GMT+00:00) 

Adding more users to AMS

Having got a basic AMS set up working (on z/OS), it takes a bit of planning (and getting your head round it) to add more users and get the certificates in the right place when you have more than one queue manager. For example for someone to get an encrypted message the putting userid on the remote queue manager needs the getter’s public certificate. If there are 1000 potential putters, you start to see the complexity of the problem.

As I have tried to implement AMS across my baby enterprise I’ve realised you need a process for maintaining the key stores. When using multiple queue managers you have to remember which queue manger needs which options on the setmqspl command.

Basic key store management

z/OS key rings

On z/OS, each local userid has its own private key on its own keyring, with all of the shared public keys on the xxxxAMSM userid’s keyring (userid/drq.ams.keyring). Update the xxxxAMSM’x keyring, and every one picks up the change – great – easy.

You can have multiple queue managers in a single z/OS image or a sysplex, each with the xxxxAMSM address space. If they use the same started task userid, they will share the same keyring. (So to have queue managers use a different keyring, they will need a different userid.)

If any of the users are outside of the RACF key ring, you may need to export public certificates and send them to other environments: different sysplexes, or different midrange machines. The certificate will need to be imported on these other machines.

Midrange key stores

With midrange, a userid needs a key store with the private and public keys. With multiple machines you need a process to update and distribute key stores. For example:

  • Create a key store in a central site. Do all maintenance to this key store, adding and removing certificates as required.
  • Securely distribute this to all machines that are using AMS.
  • The key store which has been distributed may have the public key for the user. You need to remove it, from the downloaded keystore and add the private key from the local machine.
    • runmqakm -cert -delete -db /home/colinpaice/.mqs/shared.kdb -type kdb -pw passw0rd -label COLIN
    • runmqakm -cert -export -db $private.kdb -pw passw0rd -label COLIN -target COLIN.key.p12 -target-type pkcs12 -target_pw zpassword
    • runmqakm -cert -import -target /home/colinpaice/.mqs/shared.kdb -target_type kdb -pw zpassword -file COLIN.key.p12 -type p12
    • when you import the private key you can use -label COLIN -new_label COLIN2 to give it a new label.
    • You have to make the key the default using runmqakm -cert -setdefault (even though this is deprecated), because without this, the first certificate in the keystore is used..
  • Rename the shared key stores; current to old, new to current.
  • Restart the client.
  • Check it all works, delete the old key store.

If you add a new user, you may need to send the user’s public key to the cental site, to get it distributed to the mid range machines, and send it to the z/OS systems to add to the xxxxAMSM key rings.

In my case I just used one client, and updated the client’s key store directly.

Adding a new user for putting messages to a queue

Signed messages

When the data is signed, the recipient needs:

  • the Certificate Authority of the putter (the signer) – which the recipient may already have
  • or, when the putter has used a self signed certificate, the recipient needs the public part of the putter’s self signed certificate.

When using a CA signed certificate you may not need to export anything if the receiving end already has the CA certificate in the key store or keyring.

To export and import the Certificate authority.

  • To export a CA certificate from z/OS, certificate see here, the certificate owner will typically be CERTAUTH
  • To export a putter’s mid range CA certificate, see here.
  • To import a CA certificate on z/OS, the certificate owner could be a userid, site, or CERTAUTH (check with your security administrator). You might not have authority to use SITE or CERTAUTH.
  • To import a CA certificate on mid range, see here.

When using a self signed certificate (not a good idea for production)

  • export the putter’s public certificate
  • import it into the midrange key stores
  • import it to any other z/OS systems, and add it to the xxxxAMSM keyring.

You can use

  • To export a putter’s z/OS certificate see here.
  • To export a putter’s mid range certificate, see here.
  • To import a certificate on z/OS for a getter, see here. Once the certificate has been imported into the xxxxAMSM keyring it is available to all uses of the queue manager, and does not need to be imported again.
  • To import a certificate on mid range for a getter, see here.

Specify the list of authorised DNs

At the getting end, if the setmqspl -a option is used to specify the list of authorised signers, then the DN from the putter needs to be added to the list of authorised signers (-a…) . You should use dspmqspl -m mq -p q -export to get the current definition for the queue and use setmqspl to add the new -a data to that.

On z/OS, when you have used setmqspl to update the AMS configuration, you need to use the F xxxxAMSM,REFRESH ALL command.

Putting encrypted messages

The putter needs a private key in the keystore or key ring – though this is not used when encrypting messages.

At the putting end, the queue needs to be configured with the list of recipient DNs setmqspl … -r … .

The public certificates for these DNs need to be in the putter’s xxxxAMSM keyring, or in the mid range user’s key store.

  • To export the recipients’s z/OS certificate see here. You could store this in a central file so you can just reuse it rather than having to export it every time.
  • To export the recipients’s mid range certificate, see here. You could store this in a central file so you can just reuse it rather than having to export it every time.
  • To import the certificate on z/OS for the putter, see here. You should only need to do this once per keyring and user.
  • To import the certificate on mid range for the putter, see here. This is where you need a process for a centralised key store containing all of the public keys your organisation needs, and distributing this to all of the users. You may have a cms key store (for C programs) and a JKS key store for Java programs.

Adding a new user for getting messages from a queue

Getting signed messages

If other userids are getting signed messages from this queue there may be no additional configuration on the getting queue manager.

If you are using setmqspl … -a … see Specify the list of authorised DNs above. It may already be set if other userids are configured to get messages from this queue.

When using a CA signer certificate you may not need to export anything if the receiving end already has the CA certificate in the key store or keyring.

If the getting end does not have the CA see To export and import the Certificate authority above.

If the putter is using self signed, see When using a self signed certificate (not a good idea for production) above.

Getting encrypted messages

The getter’s private key is used to decrypt the message. Each putter needs to have access to the public key of the getter.

On the system doing the MQPUT,

  • the recipient’s DN needs to be configured on the putting queue manager using setmqspl with the -r (recipient) option for the queue
  • each user doing the MQPUT will need access to the recipient’s public certificate in its key store or the xxxxAMSM keyring.
Specify the list of recipient DNs

On the putting queue manager you need to add the recipient to the list of the recipients for message. Use the command dspmqspl -m QMA -p AMSQ -export to display the existing configuration for the AMSQ queue, and add the DN of the new user to the list using the setmqspl command.

You need to do this on all squeue managers which can be putting to this queue manager

On z/OS, when you have used setmqspl to update the AMS configuration, you need to use the F xxxxAMSM,REFRESH ALL command.

Get the recipients public certificate in the putters’ keyrings and key stores.
  • To add a z/OS user’s public certificate to the xxxxAMSM’s keyring in the same RACF database, see here.
  • To export the z/OS certificate from a recipient, see here.
  • To export the mid range certificate from a recipient, see here.
  • To import a certificate for putters on z/OS, see here.
  • To import a certificate for putters on mid range, see here.

Certificate and keyring management for AMS.

I started writing up how to use AMS, and found I had written a lot about how to manage certificate and key rings from a z/OS perspective. It made more sense to write this up in its own blog post.

When using AMS, you may have to export public certificates to other z/OS images, and to mid range machines.

Basic key store management

z/OS key rings

On z/OS, each local userid has its own private key on its own keyring, with all of the shared public keys on the xxxxAMSM userid’s keyring. Update the xxxxAMSM’x keyring, and every one picks up the change – great – easy.

The certificate used to sign is the default certificate in the user’s keyring.

You can have multiple queue managers in a single z/OS image or a sysplex, each with the xxxxAMSM address space. If they use the same started task userid, they will share the same keyring. (So to have queue managers use a different keyring, they will need a different userid.)

Midrange key stores

With midrange, a userid needs a key store with the private and public keys. With multiple machines you need a process to update and distribute key stores. For example:

  • Create a key store in a central site. Do all maintenance to this key store, adding and removing certificates as required.
  • Securely distribute this to all machines tat use AMS.
  • On each user’s machine add the user’s private key to this new copy of the key store.
  • Rename the key stores; current to old, new to current.
  • Restart the client.
  • Check it all works, delete the old key store.

In my case I just used one client, and updated the client’s key store directly.

A user can point to different key stores, so you need keep in mind that it is the user’s certificate, not the userid that does signing, encryption etc..

When do I need to export a public certificate?

Signing

  • If the certificate is self signed, the recipients will need a copy of the public certificate
  • If the certificate is signed by a Certificate Authority the recipient needs a copy of the CA certificate, and may already have it.

As a recipient of encrypted data

  • Each sender needs the public certificate (and public key) of each recipient, to be able to encrypt the payload.

Connecting the certificate in the same RACF database.

If the xxxxAMSM keyring is in the same RACF database as the the user’s certificate you just need to connect it to the keyring

RACDCERT ID(START1) connect(ID(ADCDC) LABEL(‘AMSZ’) –
RING(drq.ams.keyring) USAGE(SITE))

Exporting a certificate from a RACF database.

You need to export the certificate to a dataset. You can use binary format, or base64 encoded. I use base64 encoded.

RACDCERT ID(ADCDC) EXPORT(LABEL(‘AMSZ’))-
DSN(‘ADCDC.AMSZ.PEM’) FORMAT(CERTB64) PASSWORD(‘password’)

This file needs to be sent to the other systems and imported into the key stores.

Import a certificate into z/OS

For z/OS you need to import the public certificate to a userid. You might have a userid just for these certificate, or a userid for enterprise (BANK1, BANK2) etc. I’ve used ADCDA below.

The following example uses a certificate (for ja2) sent from a Linux machine. It was FTPed to z/OS into a data set ADCD.JA2.PEM.

#delete any old instance
RACDCERT ID(ADCDA) DELETE (LABEL(‘LINUXJA2’))

RACDCERT ADD(‘ADCD.JA2.PEM’) –
ID(ADCDA) WITHLABEL(‘LINUXJA2’) TRUST

RACDCERT ID(START1) CONNECT(RING(drq.ams.keyring ) –
ID(ADCDA) LABEL(‘LINUXJA2’))

RACDCERT LIST (LABEL(‘LINUXJA2’)) ID(ADCDA)

SETROPTS RACLIST(DIGTCERT) REFRESH

Extract a certificate from a mid-range key store

To extract from a CMS key store.

runmqakm -cert -extract -db my.kdb -pw passw0rd -label COLIN -target LINUXID.pem -format ascii

To extract from a .jks key store

runmqckm -cert -extract -db my.jks -pw zpassword -label COLIN -target LINUXID.PEM -format ascii -type jks

FTP the file to the remote systems and import it.

Importing a certificate into a mid-range key store

Follow the key store update process to add it to each midrange key store. The certificate was exported from z/OS and FTPed to the Linux machine as file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem

For the .jks key store (used by Java).

/opt/mqm/bin/runmqckm -cert -add -db trust.jks -type jks -file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem -label zADCDC -pw zpassword

#It is worth checking the certificate to make sure the certificate chain is present.

/opt/mqm/bin/runmqckm -cert -validate -db trust.jks -type jks -pw zpassword -label zADCDC

For a CMS keystore

/opt/mqm/bin/runmqckm -cert -add -db $key.kdb -file /home/colinpaice/mqamsclient/zADCD.AMSZ.pem -label zADCDB -pw passw0rd

What certificates do I need for AMS with a client going to z/OS

The MQ documentation usually shows a local application doing a put and get, to show “how simple” it is.
Most people have a more complex environment where they have a mixture of z/OS and midrange, and clients into both.

This blog posts shows how I got clients to work with AMS on z/OS. My starting point was a working z/OS queue manager with working Java client able to put and get messages from a queue. I then enabled AMS on the z/OS queue manager.

Getting the z/OS end working

The userids on z/OS used a CA signed certificate (which is more typical in production than a self signed certificate).

Set up the xxxxAMSM keystore

My AMSM adress space has a started task userid of START1.

The xxxxAMSM address space needs a keystore, and the CA for the certificates on z/OS

#RACDCERT ID(START1) DELRING(drq.ams.keyring)
RACDCERT ID(START1) ADDRING(drq.ams.keyring)
RACDCERT ID(START1) CONNECT(CERTAUTH LABEL(‘TEMP-CA’) –
RING(drq.ams.keyring))
RACDCERT LISTRING(drq.ams.keyring) ID(START1)

Set up the keyring and certificate for the tso userid

Userid COLIN needs a keyring and an RSA certificate to identify COLIN. It is signed by “TEMP-CA”.

#RACDCERT ID(COLIN) DELRING(drq.ams.keyring)

RACDCERT ID(COLIN) ADDRING(drq.ams.keyring)

#RACDCERT ID(COLIN) DELETE(LABEL(‘AMS’))

#It defaults to certificate type RSA
RACDCERT ID(COLIN) GENCERT –
SUBJECTSDN(CN(‘COLIN’) O(‘SSS’)) –
SIZE(2048) –
SIGNWITH (CERTAUTH LABEL(‘TEMP-CA’)) –
WITHLABEL(‘AMS’)

RACDCERT id(COLIN) ALTER(LABEL(‘AMS’))TRUST

# add the certificate to user’s keyring
RACDCERT ID(COLIN) CONNECT(RING(drq.ams.keyring ) –
ID(COLIN) –
DEFAULT –
LABEL(‘AMS’))

# Connect the user’s public certificate to the xxxxAMSM keyring
#Usage(SITE) only exposes the public certificate, the private certificate
# is not visible in the keyring
RACDCERT ID(START1) CONNECT(ID(COLIN) LABEL(‘AMS’) –
RING(drq.ams.keyring) USAGE(SITE))

RACDCERT LISTRING(drq.ams.keyring) ID(COLIN)

RACDCERT ID(COLIN) LIST

# export it so we can send public key to Linux machine.

RACDCERT ID(COLIN) EXPORT(LABEL(‘AMS’))-
DSN(‘COLIN.AMS.PEM’) –
FORMAT(CERTB64) –
PASSWORD(‘password’)

# give the userid access to the keyring

permit IRR.DIGTCERT.LISTRING class(facility) access(read) id(COLIN)

setropts raclist(facility) refresh

FTP the COLIN.AMS.PEM certificate to Linux as ASCII.

Make the queue AMS protected

You need to use setmqspl to define the level of protection

// EXPORT SYMLIST=* 
// SET QM=CSQ9
//CSQ40CFG EXEC PGM=CSQ0UTIL, 
//         PARM='ENVAR("_CEE_ENVFILE_S=DD:ENVARS") /' 
//STEPLIB  DD DSN=COLIN.MQ921.SCSQANLE,DISP=SHR 
//         DD DSN=COLIN.MQ921.SCSQAUTH,DISP=SHR 
//ENVARS   DD DSN=COLIN.MQ921.SCSQPROC(CSQ40ENV),DISP=SHR 
//EXPORT   DD SYSOUT=* 
//SYSPRINT DD SYSOUT=* 
//SYSIN    DD *,,SYMBOLS=(JCLONLY)  
 setmqspl -m &QM.
   -p AMSQ 
   -remove 
 setmqspl -m &QM.
   -p AMSQ 
   -s SHA256 
   -e AES256 
   -a CN=COLIN,O=SSS 
   -a "CN=testuser, O=aaaa, C=GB" 
   -r "CN=testuser, O=aaaa, C=GB" 
   -r CN=COLIN,O=SSS 
dspmqspl -m &QM. 
dspmqspl -m &QM. -export 

Tell the xxxxAMSM address space to pick up the changes

You can use refresh all | refresh policy to get the the xxxxAMSM address space to use the changed policy information.

F xxxxAMSM,REFRESH ALL

or

F xxxxAMSM,REFRESH POLICY

Set up Linux

Create the keystore.conf

By default the configuration file used by AMS is in $HOME/.mqs/keystore.conf . I used different files, so I could test different scenarios with just one userid. I could set up a files called Alice.conf and Bob.conf so my ID can have different roles.

The location of the keystore.conf is available using

  • export MQS_KEYSTORE_CONF=/home/colinpaice/mqamsclient/ks.conf for C program and
  • -DMQS_KEYSTORE_CONF=/home/colinpaice/mqamsclient/ks.conf for java programs

Create a file for the configuration (/home/colinpaice/mqamsclient/ks.conf)

JKS.keystore = /home/colinpaice/ssl/ssl2/trust.jks
JKS.certificate =testuser

JKS.encrypted = no
JKS.keystore_pass = zpassword
JKS.key_pass = zpassword

# in V9 JKS.provider is ignored
# JKS.provider = IBMJCE

The options are documented here.

The key_pass is to access the private key within the keystore, see -keypass here. Because you are storing the clear text password in the file, it is recommended to use runamscred to encrypt the passwords.

Create the Linux keystore and certificate

I have scripts which generate my certificates, so I modified an existing one. I created a certificate with CN=testuser,O=aaaa,C=GB.

name=”testuser”
rm $name.pem $name.key.pem $key.csr $key.p12

# create an RSA private key
openssl genpkey -out $name.key.pem -algorithm RSA -pkeyopt rsa_keygen_bits:2048

# Create a certificate requests – so we can get it signed
openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=aaaa/CN=”$name -passin file:password.file -passout file:password.file

#use this CA
ca=”carsa1024″

# sign it
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

#Create the .p12 file so we can import it into the .jks
openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $ca.pem -chain -name $name -passout file:password.file -passin file:password.file

# remove the old one and import it
/opt/mqm/bin/runmqckm -cert -delete -db trust.jks -type jks -label $name -pw zpassword

/opt/mqm/bin/runmqckm -cert -import -target trust.jks -target_type jks -file $name.p12 -label $name -pw password -target_pw zpassword

#Add the z/OS user’s public key, so we can encrypt with the certificate and public key
/opt/mqm/bin/runmqckm -cert -add -db trust.jks -type jks -file zCOLIN.PEM -label zcolin -pw zpassword

# extract the Linux certificate and public key, along with its CA.
runmqckm -cert -extract -db trust.jks -pw zpassword -label $name -target $name.PEM -format ascii -type jks

exit

FTP the Linux public key to z/OS and add it to the keyring

The public key for testuser was extracted to testuser.PEM ( $name.PEM above). FTP this to z/OS (in ASCII). This became file COLIN.TESTUSER.PEM on z/OS.

To simplify the administration, you could setup an ID to “own” the certificates for each enterprise you work with. For example certificates from YOURORG are attached to ID(YOURORG).

I’ve used id ADCDA.

#Try to delete it if needed
#RACDCERT ID(ADCDA) DELETE (LABEL(‘LINUXTESTUSER’))


RACDCERT ADD(‘COLIN.TESTUSER.PEM’) –
ID(ADCDA) WITHLABEL(‘LINUXTESTUSER’) TRUST

# Connect it to the xxxxAMSM’s keyring
RACDCERT ID(START1) CONNECT(RING(drq.ams.keyring ) –
ID(ADCDA) LABEL(‘LINUXTESTUSER’))

Tell xxxxAMAM to pick up the updated keystore

You can use refresh all | refresh keystore to refresh the xxxxAMSM keyring information

f CSQ9AMSM,REFRESH ALL

or

f CSQ9AMSM,REFRESH keyring

Run the java client

. /opt/mqm/java/bin/setjmsenv64

#Java does not used export MQS_KEYSTORE_CONF=/home/colinpaice/mqamsclient/ks.conf

ksc=”-DMQS_KEYSTORE_CONF=/home/colinpaice/mqamsclient/ks.conf”
lib=”-Djava.library.path=/opt/mqm/java/lib64″

#java $lib $ksc JmsBrowser -m CSQ9 -d AMSQ -h 10.1.1.2 -p 1414 -l SYSTEM.DEF.SVRCONN

#java $lib $ksc JmsConsumer -m CSQ9 -d AMSQ -h 10.1.1.2 -p 1414 -l SYSTEM.DEF.SVRCONN

java $lib $ksc JmsProducer -m CSQ9 -d AMSQ -h 10.1.1.2 -p 1414 -l SYSTEM.DEF.SVRCONN

Any error messages are written to file mqjms.log.0 .

Installing AMS on z/OS

Before you start to configure AMS on z/OS, you need to understand if you are licensed for AMS, see here. You need to understand this so you know what value to use in your AMSPROD. If you use the wrong value, your billing may be wrong.

I used the instructions here. There are a lot of steps, but they are clear, and worked. The AMS address space is started by the queue manager, once all the set up has been done. If you try to start it by hand, it will fail.

If you have to put in change requests for changes to security profiles you may want to build one big list for the security team to do (or perhaps better,create the JCL with all of the commands, and ask the security team to run the commands for you). The security team may have views about the granularity of access to keyrings (using FACILITY (a user can access any keyring) or RDATALIB (you give access to a specific ring <ringowner>.<ringName>.LST)).

In Enable Advanced Message Security, I created a new zparm module, for example CSQZAMSP with the SPLCAP=YES and the AMSPROD=xxxx value.

I started the queue manager with

%csq9 start qmgr parm(csqzamsp)

If you need to start the queue manager without AMS, just use your previous version.

Once MQ starts, and the AMS address space starts.

If you want to change the AMS policy you need to use CSQ0UTIL.

If you change the keyring, or the AMS policy you need to refresh the AMS address space.

F qmgrAMSM,REFRESH KEYRING
F qmgrAMSM,REFESH POLICY
F qmgrAMSM,REFRSH ALL   	

If you cancel the xxxxAMSM address space the queue manager will end.

Generation of certificates.

AMS only supports RSA certificates, so you cannot use Elliptic Curves, you must use GENCERT.. RSA ( or let it default to RSA).

Installing AMS on midrange for end to end message protection

Installing AMS on the server.

I tried to follow the instructions here but they use .rpm files. I had downloaded the MQ Developer Edition from here.

The download was mqadv_dev921_ubuntu_x86-64.tar.gz.

I extracted the ./ibmmq-ams_9.2.1.0_amd64.deb file, and used

sudo apt install ./ibmmq-ams_9.2.1.0_amd64.deb

I don’t know what you need to install just for the client.

Testing the setup

A scenario is documented Quick Start Guide for AMS on AIX. There is not a quick start guide for Linux; and the AIX guide wasn’t very quick. You have to create additional userids (which my “Enterprise security team” will not allow) do additional setup for these ids, and did not tell you how to clean up afterwards.

Below is my “quicker start to configuring AMS on Ubuntu”

Even quicker start up guide for AMS on Linux (and other Unix platforms), and this would be a good start for Windows

The examples in the IBM documentation implement the classic picture of Alice sending an encrypted message to Bob. This requires a lot of set up, new queue manager, userids etc – most of which are not needed when you use AMS.

In the example below, it takes a working system and adds AMS functions to an existing queue.

Note: although Structure of the keystore.config file refers to the different key store parameters, for example CMS and JKS, the C client needs cms keystore.

A userid using AMS needs a key store and a certificate; and a file to hold configuration information.

In summary the steps are:

  • Initial set up – to define shell variables
  • Define a queue to be protected, and an alias
  • Check that your userid can put and get messages to/from this queue
  • Define the keystore and certificate
  • Change the queue to have AMS protection
  • Check the id can put to the queue
  • Checking the message on the queue is protected
  • Getting the message

I used a shell script, and commented/uncommented statements as I progressed. I did this as it took several iterations before it all worked. It was easy to mistype a parameter, and hard to tell when you had made a mistake.

Initial set up – to define shell variables

It is less error prone to use environment variables to hold constants to avoid spelling mistakes, or using the wrong case.

export dn=”CN=CCP,C=GB,O=MINE”
export mydir=”$HOME/.mqs”
export keystore=$d/”colins”
export label=”COLINS

# create the default directory
mkdir $mydir -p

# optionally clear out this directory
# rm $mydir/*

#create the AMS configuration file
echo “cms.keystore = $key” > $mydir/keystore.conf
echo “cms.certificate = $label” >> $mydir/keystore.conf

Define a queue to be protected and an alias

If you access the protected queue you will get the AMS protection. If you define the queue as the target of an alias queue, you can bypass protection and see the message on the queue, so you can check it is protected as you expect.

You can use runmqsc to

define ql (AMSQ)
define qalias(AAMSQ) targq(AMSQ)

Or use a command in a shell script

qm=”QMA”
q=”AMSQ”

echo “define ql ($q)” | runmqsc $qm
echo “define qalias(A$q) targq($q)” |runmqsc $qm

Check that your userid can put and get messages to/from this queue

Use the amqsput sample to put data, until a null line is entered, then use amqsget to get the messages, and leave the queue empty. Note the amqsput does not prompt for input, just enter the data and press enter; no data and enter ends it.

/opt/mqm/samp/bin/amqsput AMQS QMA
/opt/mqm/samp/bin/amqsget AMQS QMA

Resolve any problems. Do this before your protect the queue, so you know you have a working environment.

Define the keystore and certificate

C programs need a cms keystore. Using the environment variables defined above

runmqakm -keydb -create -db $key.kdb -pw passw0rd -stash
runmqakm -cert -create -db $key.kdb -pw passw0rd -label $label -dn $dn -default_cert yes

Change the queue to have AMS protection

setmqspl -m $qm -p $q -s SHA1
dspmqspl -m $qm -p $q

#remove it if required
#setmqspl -m $qm -p $q -remove

Check the userid can still put to the queue

Now the setmqspl command has been issued for the queue, any put will create a protected message. It uses the default environment variable MQS_KEYSTORE_CONF pointing to “$HOME/.mqs” to point to the configuration file $HOME/.mqs/keystore.conf (created above).

/opt/mqm/samp/bin/amqsput $q $qm

Enter some data (AAAAA) and press return, then press return again to give null data.

If this was not successful, resolve the problems. See below for some help on this.

Check the message on the queue is protected.

A couple of times I though the AMS set up had worked, as the put and get both worked without error. I later found that the messages were not protected. I like to check the messages on the queue are protected.

You can browse the message from the alias queue, it should contain additional data before the message, including distinquished name of the signer.

/opt/mqm/samp/bin/amqsbcg $qm $q

This displays the data in hex. The data should start with the header PDMQ showing there is protected data on the queue. Within the PDMQ header, should be the encoded signing information.

If you get the same data as you put (AAAAA) then the protection has failed; check the set up.

Example data showing the content of the queue after the data has been signed.

00000000:  5044 4D51 0200 0200 7000 0000 7000 0000           'PDMQ....p...p...'
00000010:  0400 0000 B804 0000 0800 0000 0000 0000           '................'
00000020:  4D51 5354 5220 2020 0000 0000 0000 0000           'MQSTR   ........'
00000030:  0000 0000 0000 0000 414D 5351 2020 2020           '........AMSQ    '
00000040:  2020 2020 2020 2020 2020 2020 2020 2020           '                '
00000050:  2020 2020 2020 2020 2020 2020 2020 2020           '                '
00000060:  2020 2020 2020 2020 0000 0000 0000 0000           '        ........'
00000070:  3082 0527 0609 2A86 4886 F70D 0107 02A0           '0..'..*.H.......'
00000080:  8205 1830 8205 1402 0101 310F 300D 0609           '...0......1.0...'
00000090:  6086 4801 6503 0402 0305 0030 1706 092A           '`.H.e......0...*'
000000A0:  8648 86F7 0D01 0701 A00A 0408 4141 4141           '.H..........AAAA'
000000B0:  4141 4141 A082 0300 3082 02FC 3082 01E4           'AAAA....0...0...'
000000C0:  A003 0201 0202 0859 7651 E018 1827 E030           '.......YvQ...'.0'
000000D0:  0D06 092A 8648 86F7 0D01 0105 0500 301C           '...*.H........0.'
000000E0:  310B 3009 0603 5504 0613 0247 4231 0D30           '1.0...U....GB1.0'
000000F0:  0B06 0355 0403 1304 4343 5032 301E 170D           '...U....CCP20...'
00000100:  3231 3033 3239 3134 3433 3335 5A17 0D32           '210329144335Z..2'
00000110:  3230 3333 3031 3434 3333 355A 301C 310B           '20330144335Z0.1.'
00000120:  3009 0603 5504 0613 0247 4231 0D30 0B06           '0...U....GB1.0..'
00000130:  0355 0403 1304 4343 5032 3082 0122 300D           '.U....CCP20.."0.'
00000140:  0609 2A86 4886 F70D 0101 0105 0003 8201           '..*.H...........'

Where

  • PDMQ is the AMS header
  • AMSQ is the original queue name
  • AAAAAAAA is the original message data – note this is visible because encryption is not being used
  • GB comes from C=GB in the DN. This is part of the encoded DN
  • CCP2 comes from CN=CCP2 in the DN. This is part of the encoded DN

Destructively get the message

/opt/mqm/samp/bin/amqsget AMQS QMA

Should display the original message.

If this was not successful, resolve the problems. See below for some help on this.

Encrypt the messages

If you specify “use encryption with AES256”, and “this id” as a recipient, you will get encrypted messages on the queue

setmqspl -m $qm -p $q -e AES256 -s SHA512 -r $dn

and repeat the put, and browse from the alias queue, you will see the the PDMQ in the header, and parts of the signing DN in the header, as before. The message payload will be encrypted.

Giving another user access or using a different DN and certificate

By default, the configuration file ~/.mqs/keystore.conf contains information as to which key store, and which key within the key store to use.

You can override the location of the config file using the environment variable MQS_KEYSTORE_CONF .

This means one userid can use a different keystore.conf file file for each application, or you can have a group of userids using the same keystore.conf and so the same certificate keystore file, and certificate.

If it did not work…

Check the error logs, for example the /var/mqm/qmgrs/QMA/errors/*01* .

Most of the errors I introduced had a clear message description of the problem.

GSKIT return codes are described here.

While setting up, it is worth clearing the queue if you get errors, for example there may be a message left on the queue which you are not expecting.

Here is the full script I use ( with extra stuff in it)


d="$HOME/.mqs"
key=$d/"colin"
mkdir $d -p
rm $d/*
dn="CN=CCP2,C=GB"
qm="QMA"
q="AMSQ"
rm "COLIN.pem"
echo "define ql ("$q")" | runmqsc $qm
echo "define qalias(A$q) targq($q)" |runmqsc $qm
runmqakm -keydb -create -db $key.kdb -pw passw0rd -stash

# chmod +r $key
runmqakm -cert -create  -db $key.kdb -pw passw0rd  -label COLIN -dn $dn -default_cert yes
runmqakm -cert -extract -db $key.kdb -pw passw0rd  -label COLIN -target COLIN.pem  -format ascii 
/opt/mqm/bin/runmqckm -cert -add  -db $key.jks -type jks  -file ~/ssl/ssl2/zADCDB.pem -label zADCDB -pw passw0rd
chmod 600 $key.kdb
# chmod 777 $key.kdb

echo "cms.keystore = $key" > $d/keystore.conf
echo "cms.certificate = COLIN" >> $d/keystore.conf 
chmod 777 $d/keystore.conf 
cat $d/keystore.conf 
setmqspl -m $qm -p $q   -remove

# setmqspl -m $qm -p $q   -s SHA1 -a $dn 
setmqspl -m $qm -p $q  -s SHA512 
# setmqspl -m $qm -p $q  -e AES256  -s SHA512 -r $dn
dspmqspl -m $qm -export 
#sudo rm /var/mqm/trace/*
# strmqtrc -m $qm

export MQS_KEYSTORE_CONF=/home/colinpaice/.mqs/keystore.conf
printenv |grep MQ
# print echo in bold 
BOLD='\033[1m'
NC='\033[0m' # No Color

echo "${BOLD}type some data and press enter, then press enter again${NC}"
/opt/mqm/samp/bin/amqsput $q  $qm
#display it in hex from qalias
opt/mqm/samp/bin/amqsbcg A$q $qm 
# /opt/mqm/samp/bin/amqsgbr $qm $q  
/opt/mqm/samp/bin/amqsget $q $qm

# less /var/mqm/qmgrs/$qm/errors/*01*


Pushing the security envelope – protect the ccdt

Every year the security boundaries are being pushed to protect more resources. For example

  • Encrypt DNS lookup requests. Without this, bad guys can see what sites you are looking at. Or, depending on your view point, good guys can see what sites a bad guy is looking at.
  • Encryption of cookies in http traffic. In https, cookies are encrypted, but some sites just use https for the logon, and then use http for normal traffic.
  • Padding of strings, so the length of data cannot be seen. If you send up an encrypted password and the length is 4 characters, you can guess that the password is short.

MQ CCDT

The CCDT in MQ defines parameters for a client connection, it contains, for example,

  • The connection host address and port
  • The cipher specification
  • The certificate label to use
  • The certificate peer name (what is in the certificate sent from the server)

You may want to protect this information, so bad guys cannot intercept and change it.

The CCDT can be stored in the local file system, or it can be accessed from the network using http, or ftp.

This file will need to be updated for example, when there are changes to the configuration of the back-end servers, or the cipher spec is being upgraded.

Local file system.

You need to be careful how you update the local file system. Most enterprises have technology which securely distribute updates to client machines, for example with operating fixes, application upgrades, and configuration information.

Emailing a file, or simple FTP may not be secure enough for some organisations. Secure FTP (SFTP) or FTPS may be good enough.

Getting the CCDT from the network

Using HTTP and having the CCDT on a central site has many advantages. You update it, and the next time the client starts, it will phone home, and get the updated CCDT. You may want to use a secure site so only production machines can access the production CCDT files, and only test machines can access the test CCDT files.

The MQ documentation has

Unauthenticated connections

export MQCHLLIB=ftp://myhost.sample.com/var/mqm/qmgrs/QMGR/@ipcc
export MQCHLLIB=http://myhost.sample.com/var/mqm/qmgrs/QMGR/@ipcc
export MQCHLLIB=file:///var/mqm/qmgrs/QMGR/@ipcc

Authenticated connections

export MQCHLLIB=ftp://myuser:password@myhost.sample.com/var/mqm/qmgrs/QMGR/@ipcc
export MQCHLLIB=http://myuser:password@myhost.sample.com/var/mqm/qmgrs/QMGR/@ipcc

And says : If you want to use authenticated connections you must, as with JMS, provide the user name and password encoded in the URL.

To me these are not very secure, as they could be intercepted, and you could be sending userid and password on an potentially unencrypted connection.

In my Java program I managed to use https://127.0.0.1:9444/myccdt.json, to get the CCDT from my little Python https server. The url can be http, https, ftp, or file, so the only two “secure” ones are https: and file:, but the MQ documentation does not mention https.

I could not get a C program to work when I used export MQCCDTURL=”https://127.0.0.1:9443/myccdt.json”. It hung during the TLS handshake.

To make your CCDT secure, you are stuck with getting the file securely to the machine and using export MQCCDTURL=”file:///…”, which means it will updated whenever your machine gets the automatic updates – perhaps once a week.

Remember you can never be too paranoid when it comes to security. If someone steals your wallet or purse you know. If someone copies your identity file you may never know.