Overview and challenges of using end to end message encryption (AMS)

MQ provides end to end protection of messages. This covers

• Integrity – where the message can still be read – but any changes to the message are detected. You can check the id of the person that sent it.
• Encryption – where the payload is encrypted and only the ids with the correct private key can decrypt it.
• Privacy – A combination of Integrity and Encryption, where only specified ids can decrypt it, and you can check the id of the person that signed it.

The messages are protected on the network, in memory and on disk. You do not need to use a channel with TLS for protection (but you may want to use TLS for authentication), as the data is protected before sending.

Any headers that are involved in ‘routing’ like MQMD, XQH, RFH2, etc are skipped by AMS because the queue manager still needs to know what to do with the messages in terms of delivery and selection before they are unprotected. Other message headers like PCF command headers and custom application defined headers/formats can still be protected. Sensitive data should not be held in message properties as that data is used in routing/selection. Thanks to Jon Rumsey for the correction to this.s

Background about encryption and signing of data

You may need to read some of the following section more than once, because it reference private keys and public keys and it is hard to work out what is happening.

The encryption techniques used are special. There is a private key and a public key. (Think of your passport as a private key, and your name and address is your public key). You can use functions

• You do encrypt(private key, payload) to produce an encrypted payload, send it to me, and then I use decrypt(public key, encrypted payload) to recreate the original payload. I know it came from you, because I needed your public key to decrypt it.
• I do encrypt(public key, payload) to produce an “encrypted” payload, and send it to you. Only you can use decrypt(private key, encrypted payload) with your private key to recreate the original payload.

With this

• I can encrypt data using your public key that only someone with access to the private key (you) can decrypt
• I encrypt with my private key, and anyone with my public key can decrypt it – and you know it came from me – because you needed my public key.

Signing is simply taking a checksum of the data, and encrypting the value with the signer’s key. When you come to check the signature, you do the same check sum calculation, decrypt the “signed” value using my public key, and the values should match. The payload would look something like “Signed by CN=Colin,O=SSS,C=GB” encrypted checksum=2294567, my certificate; followed by the original data.

We can take this further. I sign the data with my private key, and then encrypt the data(including the signed data) with your public key, (keep up at the back there – I said it was hard to follow). When you process it, only you can decrypt it because it needs your private key, and you can check the signed data, and see it came from me. I have sent the message encrypted, and have mutual authentication.

With encryption a symmetric key is used to encrypt the data, because it is faster, and about as secure as encryption with an asymmetric key. When trying to break into enciphered data, the more data you have, the easier it is to break. This is why you can reuse the symmetric key for multiple records, but you should change it, for example after 10 MB has been processed (I don’t know the best figure – it is all a matter of risk).

Sending a message, many to many.

Sending a message from one person to another person looks fairly easy. It gets a bit harder when you have potentially many people in my organisation who are allowed to send a protected message to your organisation, and there may be many people who are allowed to get, and decrypt it.

Let the people in my organisation be CN=A,O=MYORG, and CN=B, O=MYORG, and the people in your organisation be CN=X,O=YOURORG, CN=Z,O=YOURORG.

If person CN=A wants to send an encrypted message so it can be decrypted by CN=X or CN=Z the processing is as follows.

• Do a checksum calculation on the record.
• Encrypt this checksum using CN=A’s private certificate.
• Put a new field of CN=A,O=MYORG|| encrypted checksum value, CN=A’s public certificate before the record.
• Create a symmetric key.
• Encrypt the (signed data and original payload) with the symmetric key.
• Take the symmetric key and encrypt it for CN=X,O=YOURORG using CN=X’s public certificate. Create a field of the Distinguished Name || encrypted key.
• Take the symmetric key and encrypt it for CN=Z,O=YOURORG using CN=Z’s public certificate. Create a field of the Distinguished Name || encrypted key.
• Build a record of the two “encryption fields” and the encrypted data
• Put the data in a message and send it to the recipient(s).

Getting an encrypted message

The person with CN=X gets the message.

• Scan down the message looking for CN=X,O=YOURORG. If found then extract the encrypted symmetric key. Decrypt this value with the CN=X’s private key. This gives you the key that was used to encrypt the customer data.
• Decrypt the message using the key you just decrypted.
• Extract the first field and see it is signed data. We can see that the checksum was done by CN=A,O=MYORG.
• Check this name with your list of IDs that can send you messages.
• Validate the signer’s certificate. Either use a CA, or if the certificate was self signed, check with the copy in your trust store. Note this certificate includes the public key.
• Take the checksum value in the message and decrypt it using CN=A,O=MYORG’s public key from the payload. Do the checksum of the remainder of the message and compare the values – they should match.

Setting up your enterprise

You need to know the list of potential recipients of messages for each queue.

At the receiving end, if you want to check the message came from an authorised id, you need to know the list of potential senders.

If you’ve managed to keep up so far – it gets worse!

The management of these lists can be difficult. If someone joins my team, I would need to tell the other organisations that they need to add CN=NEWPERSON, O=MYORG into their list of potential senders, (and remove CN=IVE_RETIRED, O=MYORG. It will take a week to get this change through their change management system. Until this is done, the message could be rejected as coming from an unauthorised person.

In the same way, you need to update the AMS setmqspl definitions to add CN=NEWPERSON, O=MYORG to each queue, and when people join or leave the other enterprises you’ll need to update your list of potential senders.

Do not worry too much – there is a solution, see below.

Creating the policies

With MQ, the security policy is defined using the SETMQ Security PoLicy command (setmqspl) for example to define a queue at MYORG to send message to YOURORG. The -a option is for who the message is for

setmqspl -m PRODA -p SENDQUEUE -s SHA256 -e AES128
-a "CN=X, O=YOOURORG"
-a "CN=Y, O=YOOURORG"
-a "CN=Z, O=YOOURORG"

At YOURORG for the GETQUEUE you need the following. The -r option is who you can receive a message from

setmqspl -m YOQM -p GETQUEUE 
-r "CN=A, O=MYORG"
-r "CN=B, O=MYORG"
-r "CN=JO, C=GB,O=SOMEOTHERORG"

To change an entry you have to run whole command, you cannot add one “-a” or “-r”. I would keep a file for these, update file, and process the whole file. For two queues – use two files.

You also have to consider how to update the definitions for multiple queue managers. On z/OS I would have the MQ commands inline, and use a variable queue manager name &QM, which you set in the JCL. For midrange you could have a shell script and pass the queue manager name as a parameter.

What keys do I need?

The encryption is done using RSA, and so needs a certificate with an RSA private key.

When the message is signed

Each id putting a message needs access to a private key – this could be one key shared by multiple users.

When getting a message that has been signed, the recipients will need to verify the certificate sent within the signing data; either a CA in the signing chain, or the public certificate (for self signed certificates).

When encrypting the message

The id putting the message needs access to the public key of each recipients in the list.

The id getting a message which has been encrypted, needs access to the private key.

Maintaining the public keys

For each person in the -r or -a list you will need the digital certificate, and public key in a keystore or a keyring (on z/OS).

Each id on z/OS will need their own keyring. There is a keyring for the queue manager;s AMS address space with the public certificates on it.

On midrange MQ, each user has a keystore, with a combination of public keys, and private key(s).

Time to step back and think about what you want.

To me, it is clear that having every possible user having their own DN, and certificate quickly gets impractical. If you are working with people outside of your organisation do you want them to know the people’s names within your organisation?

You may want to have a departmental certificate for example CN=PAYROLL,O=MYORG, which members of your payroll team all use. You can control access to the queue using standard MQ security policies.

An intermediate configuration would be to have a proxy or server applications to change input messages from “partner protection” to “myorg protection”, and the reply messages changed from “myorg protection” to “partner protection”

• You have an AMS protected input queue called EXTERNAL_IN. This has been configured to specify the DNs of the systems that can send you messages, so might specify CN=RECEIPTSPAYABLE,O=YOURORG as a valid sender.
• If the get is successful, then put to a queue called INTERNAL_IN, specifying the DNs of the individuals in your Accounts department who are allowed to get one of the protected messages. If someone joins the Accounts department, you only need to change the configuration within your queue manager.
• Someone (or an automated task) gets the message from INTERNAL_IN queue, it is decrypted automatically, the application does the processing and puts a message the AMS protected queue YOUR_BANK_INTERNAL, where it is encrypted by AMS.
• Another proxy/server application in your enterprise gets the messages from YOUR_BANK_INTERNAL, checks the DN of the sender, and that they were authorised to put messages to the queue.
• The server then puts to YOUR_BANK_EXTERNAL, using a DN of “CN=PAYROLL,O=MYORG”, and it is encrypted for the recipient.

This way you can isolate internal processing and external processing. Updating the internal list is within your control. You should not need to update the external lists (and certificates) once established.

Handling whoops’s

Some problems will result in the messages removed from the queue and put to a special error queue. You need to have a process in place to handle this. I had messages put to this queue when a userid tried to get an encrypted message, but the message had not been encrypted for this userid’s DN.

Other AMS blog posts

• Overview and challenges of using end to end message encryption (AMS).
• Certificate and keyring management for AMS.
• Installing AMS on z/OS.
• What certificates do I need for AMS with a client going to z/OS?
• checkAMS: program to check your AMS defintions are consistent with z/OS keyring
• Adding more users to AMS.
• Installing AMS on midrange for end to end message protection
• Understanding setmqspl to set up AMS definitions.
• How to administer AMS policies, and use the set policy command.
• Brain check and reset, needed for AMS keystores on midrange.
• Using runamscred to encrypt the passwords in the keystore.conf file.
• How do I find the recipients and signer of an AMS message?
• I have a message on the AMS DLQ – what can I do about it? and How do I process messages on the dead letter queue (DLQ)?