Should all red flags be green?

This question came out of a discussion on an MQ forum, where the question was if MQ does one time delivery, how come he got the same message twice?

Different sorts of MQ gets.

There are a variety of patterns for getting a message.

  • Destructive get out of sync-point. One application can get the message. It is removed from the system. As part of the MQGET logic there is a commit of the get so once it has gone it has gone. This is usually used for non persistent message. Persistent messages are usually processed within sync-point, but there are valid cases when the get of a persistent out of sync-point is valid.
  • Destructive get within sync-point. One application can get the message. The queue manger holds a lock on the message which makes it invisible to other applications. When the commit is issued (either explicitly or implicitly) , the message is deleted. If the application rolls back (either implicitly of explicitly) the message becomes visible on the queue again, and the lock released.
  • Browse. One or more applications can get the message when using the get-with-browse option. Sync-point does not come into the picture, because there are no changes to the message.
  • One problem with get-with-browse is you can have many application instances browsing the queue, and they may do the same work on a message, wasting resources. To help with this, there is cooperative browse. This is effectively browse and hide. This allows a queue monitor application to browse the message, and start a transaction saying process “this” message. A second instance of the queue monitor will not see the message. If the message has not been got within a specified time interval the “hide” is removed, and so the message becomes visible. See Avoiding repeated delivery of browsed messages.

The customer’s question was, that as the get was destructive, how come the message was processed twice – could this be a bug in MQ?

The careful reader may have spotted why a message can be got twice.

Why the message was processed “twice”.

Consider an application which does the following

MQGET destructive, in sync-point

Write “processed message id …. ” to the log
Update DB2 record
Commit

You might the see following in the log

processed message id x’aabbccdd01′.
processed message id .x’aabbccdd02′.
processed message id x’eeffccdd17′. .

Expanding the transaction to give more details

MQGET destructive, in sync-point
Write “processed message id …. ” to the log
Update DB2 record

If DB2 update worked then commit
else backout

If there was a DB2 problem, you could get the following on the log:

processed message id x’aabbccdd01′.
processed message id x’aabbccdd01′.
processed message id x’eeffccdd17′. .

You then say “Ah Ha – MQ delivered the message twice”. Which is true, but you should be saying “Ah Ha – MQ delivered the message but the application didn’t want it. The second time MQ delivered it, the application processed it”. Perhaps change the MQ phrase to “MQ does one time successful delivery“.

Why is this blog post called Should all red flags be green?

A proper programmer (compared to a coder), will treat a non successful transaction as a red flag, and take an action because it is an abnormal situation. For example write a message to an error log

  1. Transaction ABCD rolled back because “DB2 deadlock on ACCOUNTS table”
  2. Transaction ABCD rolled back because “MQ PUT to REPLYQUEUE failed – queue full”
  3. Transaction ABCD rolled back because “CICS is shutting down”

The Architects and systems programmers can look at these messages and take action.

For example with DB2, investigate the lock held duration. Can you reduce the time the lock is held, perhaps by rearranging with within a unit of work, for example “MQGET, MQPUT reply, DB2 update, commit” instead of “MQGET, DB2 update, MQPUT of reply, commit.

For MQ queue full, make the maximum queue depth bigger, or find out why the queue wasn’t being drained.

CICS shutting down.You may always get some reasons to rollback.

Once you have put an action plan in place to reduce the number of red flags, you can mark the action item complete, change its status from red to green and keep the project managers happy (who love green closed action items).

Note: This may be a never ending task!

After thought

In the online discussion, Morag pointed out that perhaps the same message was put twice. Which would show the same symptoms. This could have been due to a put out of syncpoint, and the transaction rolled back.

Debugging AT-TLS session problems

I deliberately misconfigured AT-TLS to see how easy it would be to identify and resolve the problems from an AT-TLS perspective. It turned out worse than I expected. There is little information on the z/OS to help you.

I configured TTLSEnvironmentAction {trace 255 } (see the bottom of this blog) and refreshed the PAGENT. I had configured SYSLOGD so records for *.TCPIP.*.* went to /var/log/TCPIP.

I reran my MQ client application and got

  • from MQ on Linux, in file /var/mqm/errors/AMQERR01.LOG return code 2393 (MQRC_SSL_INITIALIZATION_ERROR).
  • On Linux there was a file /var/mqm/trace/AMQ.SSL.TRC – which only IBM can format!
  • From TCPIP on z/OS EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900 JOBNAME: CSQ9CHIN RULE: REMOTE-TO-CSQ1 USERID: START1 GRPID: 0000001B ENVID: 0000000B CONNID: 0000006E This give
    • the address of my client,
    • the name of the chinit
    • which AT-TLS rule was used

The message EZD1287I TTLS Error RC: 402 Initial Handshake pointed me to Cryptographic Services System Secure Sockets Layer Programming – No SSL cipher specifications. The first reason was

The client and server cipher specifications do not contain at least one value in common. Client and server cipher specifications might be limited depending on which System SSL FMIDs are installed. See Cipher suite definitions for more information. Server cipher specifications are dependent on the type of algorithms that are used by the server certificate (RSA, DSA, ECDSA, or Diffie-Hellman), which might limit the options available during cipher negotiation.

MQ Trace

I took an MQ trace and formatted it. I used grep to find which file had “Cipher” in it.

Within this file I searched for Start of GSKit TLS Handshake Transcript.

This had information sent to the server as part of the handshake, and further down it had the reason code. You can see from the example that the fields and their values have been displayed (so cipher spec 003c is displayed as tls_rsa_with_aes_128_cbc_sha256)

Start of GSKit TLS Handshake Transcript (1119 bytes)
 <client_hello>
 client_version 
 TLSV12
 random 
   gsksslDissector_32Bits
   7f9d66d8
   gsksslDissector_Opaque
   Length: 28
   3E 5B 45 66 EE A3 C1 9F FB 81 0C 2F 38 19 DF 95     >[Ef......./8...
   5A 1B 54 CC B8 CB B6 C9 87 39 5E 88                 Z.T......9^.
 session_id 
 Length: 00
 cipher_suites 
 Length: 04
 00 FF 00 3C                                         ...<
 tls_ri_scsv,tls_rsa_with_aes_128_cbc_sha256
 compression_methods 
 Length: 01
 00                                                  .
 Extensions
 Length: 74
 00 0D 00 18 00 16 06 01 05 01 04 01 03 01 02 01     ................
 06 03 05 03 04 03 03 03 02 03 02 02 00 00 00 2A     ...............*
 00 28 00 00 25 73 79 73 74 65 6D 32 65 2D 64 65     .(..%system2e-de
 66 32 65 2D 73 76 72 63 6F 6E 6E 2E 63 68 6C 2E     f2e-svrconn.chl.
 6D 71 2E 69 62 6D 2E 63 6F 6D                       mq.ibm.com
  Extension Count: 2
  signature_algorithms 13
   rsa:sha512,rsa:sha384,rsa:sha256,rsa:sha224,rsa:sha1,  
   ecdsa:sha512,ecdsa:sha384,ecdsa:sha256,ecdsa:sha224,
   ecdsa:sha1,dsa:sha1
  server_name 0
   system2e-def2e-svrconn.chl.mq.ibm.com
End of GSKit TLS Handshake Transcript
{  rriEvent
 ...
 RetCode = 20009665, rc1 = 420, rc2 = 0, Comment1='SYSTEM.DEF.SVRCONN', 
 Comment2='gsk_secure_soc_init', Comment3='10.1.1.2(1414)'
 ...
}

With this trace, I am able to see what was sent to z/OS.

The AT-TLS Trace

The trace ( configured in syslogd to be in /var/log/TCPIP) had a one line entry with (I’ve reformatted it to make it easier to read).

Map CONNID: 0000006B 
LOCAL: 10.1.1.2..1414 
REMOTE:
10.1.0.2..53898 
JOBNAME: CSQ9CHIN 
USERID: START1 
TYPE: InBound 
STATUS: Enabled 
RULE: REMOTE-TO-CSQ1 
ACTIONS:
CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A

and data

RC: 0 Connection Init
Initial Handshake ACTIONS: CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A HS-Server
RC: 0 Call GSK_SECURE_SOCKET_OPEN - 00000052FD6228F0
RC: 0 Set GSK_FD(300) - 000000000000006B
RC: 0 Set GSK_USER_DATA(200) - 000000007EC32430
RECV CIPHER 160303007B 

and one loooong record with

RECV CIPHER 
010000770303749ED51D8DC7794EE6AC36B01FD115F38A4B0812D35 
C80A5F95DB840C35735CA00000400FF003C0100004A000D00180016 
060105010401030102010603050304030303020302020000002A002 
800002573797374656D32652D64656632652D737672636F6E6E2E63 
686C2E6D712E69626D2E636F6D 
SEND CIPHER 15030300020228 

From the AT-TLS trace of the data received from the client, it is the data as received, and has not been split down into useful fields.

I could not find any documentation on how to format this string. It is not easy to create a program to format this (and get it right), for example converting cipher spec 003c to TLS_RSA_WITH_AES_128_CBC_SHA256. However I have a REXX exec which works in ISPF and decodes the data into fields, but not the contents of the fields – so the cipher spec is reported as 003c

I had some success taking this data, and creating a file which Wireshark could process. See Wireshark – using external data: Bodging a hex dump file. This was not always successful, as it looks like the data is truncated, and can have non hex data in the hex stream.

Note, the System SSL server started task, GSKSRVR, can capture System SSL trace. The output is like

Job TCPIP     Process 0101001D  Thread 00000001  read_v3_client_hello            
Received CLIENT-HELLO message                                                  

with no detailed information

Tracing just one session

If you have a busy system you could get trace data for many sessions. You may want to set up a TLS rule, so you use a “debug port”, or you specify the remote host IP address, and port, using information from the error message

EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900

And dont forget…

And do not forget to reset the TTLSEnvironmentAction entry to reset the trace, and to refresh the PAGENT.

Trying to use PCF and decode the output?

I struggled to decode PCF output; for example decode PCF type 1203 and its value 20. MQ provide most of what you need, there is just one little link in the chain which is missing

In the MQ provided CMQSTRC header file are routines for converting the values to strings

For example

char *MQSYSP_STR (MQLONG v)                                   
{                                                             
  char *c;                                                    
  switch (v)                                                  
  {                                                           
  case          0: c = "MQSYSP_NO"; break;                    
  case          1: c = "MQSYSP_YES"; break;                   
  case          2: c = "MQSYSP_EXTENDED"; break;              
  case         10: c = "MQSYSP_TYPE_INITIAL"; break;          
  case         11: c = "MQSYSP_TYPE_SET"; break;              
  case         12: c = "MQSYSP_TYPE_LOG_COPY"; break;         
  case         13: c = "MQSYSP_TYPE_LOG_STATUS"; break;       
  case         14: c = "MQSYSP_TYPE_ARCHIVE_TAPE"; break;     
  case         20: c = "MQSYSP_ALLOC_BLK"; break;             

so if you know this your type is a System Parameter, you can use MQSYSP_STR(20) to get back the data item MQSYSP_ALLOC_BLK.

The bit that is missing is the mapping between PCF type and the function call.

In GitHub I’ve created this mapping for all of the PCF types (MQMAP.h). This has for example

MQMAP(MQIA_APPL_TYPE, MQAT_STR),  //     1
MQMAP(MQIA_DEF_INPUT_OPEN_OPTION, MQOO_STR),  //     4
MQMAP(MQIA_DEF_PERSISTENCE, MQPER_STR),  //     5
MQMAP(MQIA_DEFINITION_TYPE, MQQDT_STR),  //     7

I also provide some routines to help you call this and prettify the value.

I’ve also provided some code so you just need to issue

getPCFValue(MQLONG what, 
            MQLONG value, 
            char **pWhat, 
            char **pValue, 
            char **pPValue);

Where

  • what is the PCF data type (MQIA_TRIGGER_TYPE)
  • value is the PCF value (for example 3)
  • pWhat gets the name of the PCF data type (“Trigger_Type”)
  • pValue gets the value returned from the MQ provided function(“MQTT_DEPTH”)
  • pPValue gets the prettified value, with the prefix removed, and the remained made more readable(“Depth”)

This will allow you to run along PCF data and display all the data and values in a similar manner to a display command.

The data in MQMAP.h is in numerical sequence, so I can use a binary search to quickly find the mapping function. I also provide a small C function which takes the MQMAP.h file, checks it is in order and displays it, so it can be sorted.

What does MQRCCF_PARM_CONFLICT mean?

I’ve been using PCF (from Python) and have been getting MQRCCF_PARM_CONFLICT. For example use MQCMD_CHANGE_Q_MGR and set MQIA_TCP_CHANNELS to 201.

The documentation does not help. It says

Explanation

Incompatible parameters or parameter values.

The parameters or parameter values for a command are incompatible. One of the following occurred:

  1. A parameter was not specified that is required by another parameter or parameter value.
    • The MQIA_TCP_CHANNELS can be used on its own – so it is not this one.
  2. A parameter or parameter value was specified that is not allowed with some other parameter or parameter value.
    • The MQIA_TCP_CHANNELS was specified on its own – so it is not this one.
  3. The values for two specified parameters were not both blank or non-blank.
    • Only one parameter was specified – so not this one.
  4. The values for two specified parameters were incompatible.
    • Only one parameter was specified – so not this one.

I found the problem by issuing the command and seeing what the response was.

QM01 ALTER QMGR TCPCHL(201)

gave

CSQM150I QM01 CSQMAMMS ‘TCPCHL’ AND ‘MAXCHL’ VALUES ARE INCOMPATIBLE
CSQ9023E QM01 CSQMAMMS ‘ ALTER QMGR’ ABNORMAL COMPLETION

So I would add reason

5. The specified value is inconsistent with the configuration.

“The system resource RLIMIT_NOFILE is set at an unusually low level for IBM MQ”

I had this problem, and fixed it, and now I’ve upgraded to Ubuntu 20.04 it has come back again.

There are various hints like IBM Support telling me to change /etc/security/limits.conf, but these did not solve the problem for me.

I understand the words below, but not the overall picture. If you get this problem – and the standard changes do not work – try the /etc/systemd/user.conf change as well.

Changes to /etc/security/limits.conf apply to non GUI command windows and limits.conf is not used when systemd is running.

Recent changes to systemd include

  • The Linux kernel’s current default RLIMIT_NOFILE resource limit for userspace processes is set to 1024 (soft) and 4096 (hard). Previously, systemd passed this on unmodified to all processes it forked off. With this systemd release the hard limit systemd passes on is increased to 512K, overriding the kernel’s defaults and substantially increasing the number of simultaneous file descriptors unprivileged userspace processes can allocate …

For graphical login.  command windows (such as GNOME on Linux) you need.. (I’m copying)

Add the following line to /etc/systemd/user.conf:

DefaultLimitNOFILE=65535

That change works, but only affecting the soft limit. (Leaving us capped with a hard limit of 4096 still.) In order to affect the hard limit, we must modify /etc/systemd/system.conf with the same changes.

Hell is PCF.

In Dante’s book “Inferno”, the gates of hell have “Abandon all hope, ye who enter here” inscribed upon them. The book describes Dante’s journey to the deepest levels of hell.

I had a similar journey, not so far, trying to use PCFs in common code.

I thought I knew how to use MQ PCFs to display and extract information about MQ and its objects, but “fools rush in where angels fear to tread”.

Thanks to Morag Hugson of MQGEM for her help in getting me through this journey.

The kindergarten course on using PCF

Do not use PCF, use the display commands and parse the output. Parsing the output is a simpler problem to solve than trying to use PCF.

The high school course.

Instead of using a command DIS Q(…) you use PCF; create a message with a series of control blocks, send the request to a system queue, and get the data back in a similar format.

There is a header control block to define the function, and 0 or more control blocks each of which define a field.

  • The first control block, the header block, defines the request. It has an integer request field, such as MQCMD_INQUIRE_Q.
  • The second and following control blocks are field control blocks, they have
    • A field to define the type of data, for example character string, byte string (may contain hex 00s), integer, 64 bit integer, or arrays of data.
    • A field to define which field is defined, for example the integer MQCA_Q_NAME
    • For strings and byte strings, the length of the data.
    • For strings the code page of the data.
    • The data.

The request is put to a system queue and the response comes back in the reply_to queue you specified. This has a similar format as the request:

  • There is a header giving the request type, (MQCMD_INQUIRE_Q), the condition and reason code, and a count of the fields returned.
  • There are 0 or more field control blocks – the same as above.

Your program can chain down the field control blocks and can display the fields.

The program from the first year high school will report data like

field type 20 has value 1

which is not very usable.

Using the supplied header file CMQSTRC, you can call MQIA_STR(20) and get back “MQIA_Q_TYPE”. So now you know this field is the Queue Type data.

Now you know it is a Q_Type you can use the function MQQT_STR(1) and get back the string “MQQT_LOCAL”. You can determine the queue type value as a string using

if (field_type=MQIA_Q_TYPE ) 
    queue_type= MQQT_STR(value)
else if (field_type=MQIACF_PUBSUB_PROPERTIES,)                
    property_type=MQPSPROP_STR(value)
else if (field_type...)  

You need a bit of code to match up the field type to the routing you need to call to get the data back. I am working on a function getValue(20,0) to get back “MQIA_Q_TYPE”, and “MQQT_LOCAL”. One call which returns, the data type and its character representation of the value, which will make life much easier for the programmer. There are over 200 field types and it is hard to find the mapping for the value.

In early days of MQ there were two choices of header. You issued a header with type MQCFT_COMMAND and got back one reply with a header type MQCFT_RESPONSE, and all of the data.

Not too difficult so far.

Going down to the next level of hell with z/OS. The request “inquire_chinit” goes to the command server, which sends back “Request received”. The command server then sends the request to the chinit, which sends the response back to the command server, which sends it on to your reply to queue.

You now have to manage at least two reply messages, one for the “Request received”, and one or more for the data.

The next complication which takes you down to the next level of hell, is Shared Queue on z/OS.

You can issue the PCF request saying “hello all queue managers, please give me information about the queue status as you see it for queue COLINPAICEQ”. This request can now go to multiple queue managers, and you get multiple replies back which you have to manage.

This is not well documented in the official documentation. It jumps straight to the PhD class.

I’ll try to explain the data you get back with some examples.

Example 1 Inquire queue manager.

I sent the PCF request with MQCMD_INQUIRE_Q_MGR.

Two messages were returned. They had the same msgid and correlid

The first message was type MQCFT_XR_ITEM with sequence number 1, the second message was MQCFT_XR_SUMMARY with sequence number 2.

First message

  • Type MQCFT_XR_ITEM: Message is an extended response to an Inquire command. It contains item data.
  • Command Request MQCMD_INQUIRE_Q_MGR
  • Condition code 0
  • Reason code 0
  • Sequence number 1
  • It is not last message in set.
  • Parameter count 107

Data fields

  1. MQBACF_RESPONSE_ID … a 24 character string with hexadecimal values. Think of this as a correlator field, so you can match all the responses for this request.
  2. MQCACF_RESPONSE_Q_MGR_NAME : ‘CSQ9’. This is the queue manager which sent the response.
  3. MQCA_Q_MGR_NAME : ‘CSQ9’
  4. MQCA_Q_MGR_DESC : ‘CSQ9, IBM MQ for z/OS – V9.0.1’
  5. MQIA_PLATFORM : MQPL_ZOS
  6. MQIA_CPI_LEVEL : 100
  7. MQIA_COMMAND_LEVEL : MQCMDL_LEVEL_924

Second message

Header

  • Type: MQCFT_XR_SUMMARY . Message is an extended response to a command. It contains summary information.
  • Request INQ_QMGR
  • Condition code 0
  • Reason code 0
  • Sequence number 2
  • It is the last message in set.
  • Number of parameters 2
  1. MQBACF_RESPONSE_ID: …. same as above.
  2. MQCACF_RESPONSE_Q_MGR_NAME: ‘CSQ9’

These two fields are used to correlate requests. I think …XR_SUMMARY really means end of messages in this set.

Example 2 Inquire chinit.

I sent the PCF request with MQCMD_INQUIRE_CHANNEL_INIT.

Five messages were returned. They all had the same msgid and correlid

The first response message was type MQCFT_XR_MSG with sequence number 1.

The the remaining four messages were part of one set. They had

  1. Queue manager information, such as number of dispatchers.
  2. Information about the TCP/IP listener
  3. Information about the LU 62 listener
  4. The “end” or summary record

First set, single message

  • Type 17 MQCFT_XR_MSG: Message is an extended response to a command. It contains informational or error details.
  • Command MQCMD_INQUIRE_CHANNEL_INIT
  • Condition code 0
  • Reason code 0
  • Sequence number 1
  • It is the last message in set.
  • 4 field control blocks were returned.

Field control blocks

  1. MQBACF_RESPONSE_ID: Value: 24 bytes of data (with the queue manager name in it
  2. MQCACF_RESPONSE_Q_MGR_NAME: 48 character queue manager name
  3. MQIACF_COMMAND_INFO : MQCMDI_COMMAND_ACCEPTED
  4. MQBACF_RESPONSE_SET: Same value as the first field control block

Set 2, first message

Header

  • Type: MQCFT_XR_ITEM: Message is an extended response to an Inquire command. It contains item data.
  • Request Command MQCMD_INQUIRE_CHANNEL_INIT
  • Reason code 0
  • Sequence number 1
  • It is not the last message in set.
  • Parameter count 20.

Field control blocks – information about the chinit

  1. MQBACF_RESPONSE_ID: Value: 24 bytes of data (with the queue manager name in it)
  2. MQCACF_RESPONSE_Q_MGR_NAME: 48 character queue manager name
  3. MQIACF_CHINIT_STATUS : MQSVC_STATUS_RUNNING
  4. MQIACH_ADAPS_STARTED : 8
  5. MQIACH_ADAPS_MAX :8
  6. MQIACH_DISPS_STARTED :5
  7. MQIACH_DISPS_MAX :5

Set 2, second message

Header

  • Type: MQCFT_XR_ITEM Message is an extended response to an Inquire command. It contains item data.
  • Request Command MQCMD_INQUIRE_CHANNEL_INIT
  • Reason code 0
  • Sequence number 2
  • It is not the last message in set.
  • Parameter count 7.

Field control blocks – status of TCP listener

  1. MQBACF_RESPONSE_ID …
  2. MQCACF_RESPONSE_Q_MGR_NAME : b’CSQ9′
  3. MQIACH_LISTENER_STATUS : MQSVC_STATUS_RUNNING
  4. MQIACH_XMIT_PROTOCOL_TYPE : MQXPT_TCP
  5. MQIACH_INBOUND_DISP: MQINBD_Q_MGR
  6. MQIACH_PORT_NUMBER :1414
  7. MQCACH_IP_ADDRESS : b’*’

Set 2, third message

Header

  • Type: MQCFT_XR_ITEM Message is an extended response to an Inquire command. It contains item data.
  • Request Command MQCMD_INQUIRE_CHANNEL_INIT
  • Reason code 0
  • Sequence number 2
  • It is not the last message in set
  • Parameter count 5.

Field control blocks – status of LU62 listener

  1. MQBACF_RESPONSE_ID ….
  2. MQCACF_RESPONSE_Q_MGR_NAME : b’CSQ9′
  3. MQIACH_LISTENER_STATUS : MQSVC_STATUS_STOPPED
  4. MQIACH_XMIT_PROTOCOL_TYPE : MQXPT_LU62
  5. MQIACH_INBOUND_DISP MQINBD_Q_MGR

Set 2, fourth (and last) message

Header

  • Type: 18 MQCFT_XR_SUMMARY Message is an extended response to a command. It contains summary information.
  • Request Command MQCMD_INQUIRE_CHANNEL_INIT
  • Reason code 0
  • Sequence number 4
  • It is the last message in set.
  • Parameter count 2.

Field control blocks

  1. MQBACF_RESPONSE_ID …
  2. MQCACF_RESPONSE_Q_MGR_NAME : ‘CSQ9’

I do not have access to a QSG, so cannot document the records for a QSG wide query.

Creating the data fields

If you want to create a PCF command to change a queue, you need the PCF header to describe the command, and data fields to identify the queue, and changed attributes.

The header is defined with control block MQCFH (not MQPCFH, I keep trying to use). There are several fields, the key one is Command. In this case we need MQCMD_CHANGE_Q. All request begin with MQCMD. There is MQCMD_INQUIRE_Q for displaying information about a queue.

There is a good post on PCF fields, called Morag’s Quirk’s #14: MQCFH Versions which contains some good information on data types and their usage.

There is a variety of data field record types, such as Byte string, Character string, Integer; and different usages

The field can be

  • passed with the command, (a selector), and returned. For example MQIA_, MQCA_
  • returned (but not passed through with a command); for queue manager for example MQCACF, and MQIACF) and for the CHINIT for example (MQCACH and MQIACH)

or to look at it a different way…

  • MQIA_ are the set used on MQINQ – and thus defined in cmqc.h. They can also be used in PCF rather than defining a second ID for the same parameter.
  • MQIACF_  are the set used solely for PCF – and thus defined in cmqcfc.h. They are not used on MQINQ.
  • MQIACH_ are the set used for channel commands, they are defined in cmqcfc.h.

The same logic is for for MQCA_/MQCACF_/MQCACH_.

Each field has a unique number, and the field types are generally grouped within a range, but not all of them. The monitoring integers and 64 bit integers are all mixed up.

To find the string representation of the unique number you can call the formatting functions (*MQBACF_STR, *MQCA_STR, *MQCACF_STR, *MQCACH_STR, *MQIA_STR, *MQIACF_STR, *MQIACH_STR, *MQIAMO_STR, *MQIAMO64_STR) in CMQSTRC.h. These routines return a string matching the number or the empty string ”. I found I have to go through them each in turn until there is a match.

In more detail.

  • Individual fields
    • A byte string (MQCFBS) . This is a string which can include hexadecimal characters, such as a message id or correlid. It does not get converted to a different code page. There are fields
      • MQBACF_ (Byte Parameter Types ) and are in the range 7001 to 7039.
    • A character string(MQCFST). This is a string such as queue name. This has a code page associated with it. This allows you to convert from ASCII to EBCDIC (or other conversion). There are fields
      • MQCA_ (Character Attribute Selectors) and are in the range 2001 to 2999 range (and 4000) for queue manager values .
      • MQCACF_ (Character Queue Manager Parameter Types) in the range 3000 to 3211 + a couple in the 55xx range.
      • MQCACH_ (Character Channel Parameter Types) in the range 3500 to 3573
    • An integer (MQCFIN), such as queue depth, or the value to mean “Enabled” or “Disabled”. There are fields
      • MQIA_, (Integer Attribute Selectors) integers in the range of 1 to 275 ( and 2000).
      • MQIACF_, (Integer Queue Manager Parameter Types) and are number in the range of 1 to 1440.
      • MQIACH_, (Integer Channel Types) and in the range 1501 to 1646.
    • A 64 bit integer(MQCFIN64). For example MQIAMO64_GET_BYTES, the number of bytes got as a integer 64 number. The values are intermixed with MQIAMO_ numbers. These are only used by the monitoring code – there are no MQIA*64 for anything else
  • Lists of fields
    • 64 bit integers(MQCFIL64). These are used in the midrange monitoring
    • integers(MQCFIL)
    • character strings(MQCFSL)
  • A group of (sub) parameters (MQCFGRP).
    • For example with application activity. This has a several groups. One of these groups is “Operations” which contains a set of PCF data for Operation type, Operation time, QMGR name).
    • They are also used in command events – but not used in command server messages.
    • These have a type field of MQGACF_ and are in the range 8001 to 9000
  • Filters – which provides the same function as the “WHERE” clause on a command.
    • Passing in a byte string (MQCFBF),
    • Passing a string (MQCFSF)
    • Passing in an integer string(MQCFIF).

Using PCF

It takes a while to understand how to use PCF. I found I had to write a lot of code to handle the PCF data and covert from numeric values to useful character strings so as to produce useful output. It does all work – it is just hard to get it to work.
To make it just a touch more complex, you may need to worry about data conversion and strings having different length when doing conversion, but I’ll leave this “as an exercise for the reader”1.


(1)Quote: The phrase is sometimes used as a joke. When a writer gets to the difficult part of a problem and he doesn’t want to take the trouble to solve it, or doesn’t know how to solve, he says this is “left as an exercise for the reader”, as if it was too trivial to waste space in the book discussing, when really the issue is that he can’t figure out the answer himself.

What was new in the MQ API?

I wanted to know what new features were available in different releases of MQ, but could not find the information documented.

I’ve taken the CMQC C header file from different releases on z/OS and shown the differences!

From MQ V8.0 to V9.0.1

CMQC

MQCNO_CURRENT_VERSION          5      
MQCNO_VERSION_6                6 
MQCNO_CURRENT_VERSION          6

MQCNO_CURRENT_LENGTH           188    
MQCNO_LENGTH_6                 208 
MQCNO_CURRENT_LENGTH           208 

MQAT_MCAST_PUBLISH             36 
MQAT_AMQP                      37 
MQAT_DEFAULT                   2 

MQOT_PROT_POLICY               1019 
MQOT_TT_CHANNEL                1020 
MQOT_AMQP_CHANNEL              1021 
MQOT_AUTH_REC                  1022 
MQ_AMQP_CLIENT_ID_LENGTH       256
MQRC_STORAGE_MEDIUM_FULL       2192
MQRC_ADMIN_TOPIC_STRING_ERROR  2598 
MQRC_AMQP_NOT_AVAILABLE        2599 
MQRC_CCDT_URL_ERROR            2600
MQADOPT_CHECK_CHANNEL_NAME     8  

MQCMDL_CURRENT_LEVEL           800                         
MQCMDL_LEVEL_801               801 
MQCMDL_LEVEL_802               802 
MQCMDL_LEVEL_900               900 
MQCMDL_LEVEL_901               901 
MQCMDL_LEVEL_902               902 
MQCMDL_LEVEL_903               903 
MQCMDL_LEVEL_904               904 
MQCMDL_LEVEL_905               905 
MQCMDL_CURRENT_LEVEL           905 

MQPL_APPLIANCE                 28
MQCAP_EXPIRED                  2 

 /* Media Image Scheduling */ 
MQMEDIMGSCHED_MANUAL           0 
MQMEDIMGSCHED_AUTO             1 
                                                                 
 /* Automatic Media Image Interval */ 
MQMEDIMGINTVL_OFF              0 
                                                                 
 /* Automatic Media Image Log Length */ 
MQMEDIMGLOGLN_OFF              0 
                                                                 
 /* Media Image Recoverability */ 
MQIMGRCOV_NO                   0 
MQIMGRCOV_YES                  1 
MQIMGRCOV_AS_Q_MGR             2

MQCA_AMQP_SSL_CIPHER_SUITES    2137 
MQCA_AMQP_VERSION              2136  
  
MQCA_LAST_USED                 2135     
MQCA_LAST_USED                 2137   

MQIA_ADVANCED_CAPABILITY       273 
MQIA_AMQP_CAPABILITY           265 

MQIA_AUTHENTICATION_METHOD     266
MQIA_KEY_REUSE_COUNT           267 

MQIA_LAST_USED                 264               
MQIA_LAST_USED                 273   

/* Key reuse count */ 
MQKEY_REUSE_DISABLED           0 
MQKEY_REUSE_UNLIMITED          (-1)  

struct tagMQCNO { 
  ...
  PMQCHAR    CCDTUrlPtr;           /* Address of CCDT URL string */ 
MQLONG     CCDTUrlOffset;        /* Offset of CCDT URL string */ 
MQLONG     CCDTUrlLength;        /* Length of CCDT URL */ 
MQBYTE8    Reserved;             /* Reserved */ 
  /* Ver:6 */  
}
...
MQCNO_DEFAULT ....                                                       
                                              

CMQCFC

MQCMD_INQUIRE_AMQP_CAPABILITY  216 
MQCMD_AMQP_DIAGNOSTICS         217

MQRCCF_CLWL_EXIT_NAME_ERROR    3374 
MQRCCF_SERVICE_NAME_ERROR      3375 
MQRCCF_REMOTE_CHL_TYPE_ERROR   3376 
MQRCCF_TOPIC_RESTRICTED        3377 
MQRCCF_CURRENT_LOG_EXTENT      3378 
MQRCCF_LOG_EXTENT_NOT_FOUND    3379 
MQRCCF_LOG_NOT_REDUCED         3380 
MQRCCF_LOG_EXTENT_ERROR        3381 
MQRCCF_ACCESS_BLOCKED          3382 

MQ_ENTITY_NAME_LENGTH          64             
MQ_ENTITY_NAME_LENGTH          1024  


MQIAMO_MONITOR_CLASS           839 
MQIAMO_MONITOR_TYPE            840 
MQIAMO_MONITOR_ELEMENT         841 
MQIAMO_MONITOR_DATATYPE        842 
MQIAMO_MONITOR_FLAGS           843 
MQIAMO64_QMGR_OP_DURATION      844 
MQIAMO64_MONITOR_INTERVAL      845 
MQIAMO_LAST_USED               845 

/* Defined values for MQIAMO_MONITOR_FLAGS */ 
MQIAMO_MONITOR_FLAGS_NONE      0 
MQIAMO_MONITOR_FLAGS_OBJNAME   1 
                                                       
/* Defined values for MQIAMO_MONITOR_DATATYPE */ 
MQIAMO_MONITOR_UNIT            1 
MQIAMO_MONITOR_DELTA           2 
MQIAMO_MONITOR_HUNDREDTHS      100 
MQIAMO_MONITOR_KB              1024 
MQIAMO_MONITOR_PERCENT         10000 
MQIAMO_MONITOR_MICROSEC        1000000 
MQIAMO_MONITOR_MB              1048576 
MQIAMO_MONITOR_GB              100000000 

MQIACF_AMQP_ATTRS              1401 
MQIACF_AMQP_DIAGNOSTICS_TYPE   1406 

MQIACF_SYSP_MAX_CONC_OFFLOADS  1412          
MQIACF_LAST_USED               1412          
MQIACF_AUTH_REC_TYPE           1412 
MQIACF_SYSP_MAX_CONC_OFFLOADS  1413 
MQIACF_SYSP_ZHYPERWRITE        1414 
MQIACF_Q_MGR_STATUS_LOG        1415 
MQIACF_ARCHIVE_LOG_SIZE        1416 
MQIACF_MEDIA_LOG_SIZE          1417 
MQIACF_RESTART_LOG_SIZE        1418 
MQIACF_REUSABLE_LOG_SIZE       1419 
MQIACF_LOG_IN_USE              1420 
MQIACF_LOG_UTILIZATION         1421 
MQIACF_LOG_REDUCTION           1422 
MQIACF_LAST_USED               1422 

MQIACH_LAST_USED               1643  
MQIACH_AMQP_KEEP_ALIVE         1644 
MQIACH_SECURITY_PROTOCOL       1645 
MQIACH_LAST_USED               1645

MQCAMO_LAST_USED               2712  
MQCAMO_MONITOR_CLASS           2713 
MQCAMO_MONITOR_TYPE            2714 
MQCAMO_MONITOR_DESC            2715 
MQCAMO_LAST_USED               2715 
                                             
MQCACF_LAST_USED               3206  
MQCACF_AMQP_CLIENT_ID          3207 
MQCACF_ARCHIVE_LOG_EXTENT_NAME 3208 
MQCACF_LAST_USED               3208 

MQCACH_LAST_USED               3570  
MQCACH_TOPIC_ROOT              3571 
MQCACH_LAST_USED               3571

MQGACF_LAST_USED               8014
MQGACF_MONITOR_CLASS           8015 
MQGACF_MONITOR_TYPE            8016 
MQGACF_MONITOR_ELEMENT         8017 
MQGACF_LAST_USED               8017 
  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
MQACT_REDUCE_LOG               10 
MQACT_ARCHIVE_LOG              11 
                                             

MQEVO_REST                     8 

MQRQ_FAILOVER_PERMITTED        30 
MQRQ_FAILOVER_NOT_PERMITTED    31 
MQRQ_STANDBY_ACTIVATED         32 

MQLDAP_AUTHORMD_SRCHGRPSN      3 

/* Authentication Method */ 
MQAUTHENTICATE_OS              0 
MQAUTHENTICATE_PAM             1 
                                                
/* Reduce Log Options */ 
MQLR_ONE                       1 
MQLR_AUTO                      (-1) 
MQLR_MAX                       (-2) 
                                             

From MQ 9.0.1 to MQ 9.1.1

CMQC

MQCNO_CURRENT_VERSION          6          
MQCNO_VERSION_7                7 
MQCNO_CURRENT_VERSION          7 
 
MQCNO_CURRENT_LENGTH           208            
MQCNO_LENGTH_7                 240 
MQCNO_CURRENT_LENGTH           240 

MQCNO_GENERATE_CONN_TAG        0x00200000

MQAN_NONE 
MQAN_NONE_ARRAY
MQRC_INCOMPLETE_TRANSACTION    2147  
MQRC_Q_MGR_RECONNECT_REQUESTED 2601
MQRC_SUB_JOIN_NOT_ALTERABLE    29440

/* Max queue file size values */ 
MQQFS_DEFAULT                  (-1) 

MQCMDL_CURRENT_LEVEL           905               
MQCMDL_LEVEL_910               910 
MQCMDL_LEVEL_911               911 
MQCMDL_LEVEL_912               912 
MQCMDL_LEVEL_913               913 
MQCMDL_LEVEL_914               914 
MQCMDL_LEVEL_915               915 
MQCMDL_CURRENT_LEVEL           915 

MQIA_LAST_USED                 273          
MQIA_LAST_USED                 274   

MQIA_MAX_Q_FILE_SIZE           274  

struct tagMQCNO { 
  ...   
  MQCHAR28   ApplName;             /* Application name */ 
  MQBYTE4    Reserved2;            /* Reserved */ 
  /* Ver:7 */ 
};
MQCNO_DEFAULT.... 

CMQCFC

 MQCMD_AMQP_DIAGNOSTICS         217 
 MQCMD_INTER_Q_MGR_STATUS       218 
 MQCMD_INTER_Q_MGR_BALANCE      219 
 MQCMD_INQUIRE_APPL_STATUS      220 

 MQRCCF_PS_REQUIRED_MQUC        3383 
 MQRCCF_OBJECT_ALREADY_EXISTS   4001 

 MQRCCF_APPL_STATUS_NOT_FOUND   4097 
 MQCFT_STATUS                   27 

MQBACF_LAST_USED               7035   
MQBACF_REQUEST_ID              7036 
MQBACF_PROPERTIES_DATA         7037 
MQBACF_CONN_TAG                7038 
MQBACF_LAST_USED               7038 

MQIACF_LAST_USED               1422   
MQIACF_IGNORE_STATE            1423 
MQIACF_MOVABLE_APPL_COUNT      1424 
MQIACF_APPL_INFO_ATTRS         1425 
MQIACF_APPL_MOVABLE            1426 
MQIACF_REMOTE_QMGR_ACTIVE      1427 
MQIACF_APPL_INFO_TYPE          1428 
MQIACF_APPL_INFO_APPL          1429 
MQIACF_APPL_INFO_QMGR          1430 
MQIACF_APPL_INFO_LOCAL         1431 
MQIACF_APPL_IMMOVABLE_COUNT    1432 
MQIACF_BALANCED                1433 
MQIACF_BALSTATE                1434 
MQIACF_APPL_IMMOVABLE_REASON   1435 
MQIACF_DS_ENCRYPTED            1436 
MQIACF_CUR_Q_FILE_SIZE         1437 
MQIACF_CUR_MAX_FILE_SIZE       1438 
MQIACF_LAST_USED               1438 

MQIACH_LAST_USED               1645         
MQIACH_SPL_PROTECTION          1646 
MQIACH_LAST_USED               1646 

MQCACF_LAST_USED               3208         
MQCACF_APPL_IMMOVABLE_DATE     3209 
MQCACF_APPL_IMMOVABLE_TIME     3210 
MQCACF_LAST_USED               3210 

MQGACF_LAST_USED               8017         
MQGACF_APPL_STATUS             8018 
MQGACF_CHANGED_APPLS           8019 
MQGACF_ALL_APPLS               8020 
MQGACF_APPL_BALANCE            8021 
MQGACF_LAST_USED               8021 
                                            
MQIS_NO                        0 
MQIS_YES                       1 

/* Movable Options */ 
MQAPPL_IMMOVABLE               0 
MQAPPL_MOVABLE                 1 
                                    
/* Active Options */ 
MQACTIVE_NO                    0 
MQACTIVE_YES                   1 
                                    
/* Balance Options */ 
MQBALANCED_NO                  0 
MQBALANCED_YES                 1 
MQBALANCED_NOT_APPLICABLE      2 
MQBALANCED_UNKNOWN             3 
                                    
/* Balance State */ 
MQBALSTATE_NOT_APPLICABLE      0 
MQBALSTATE_LOW                 1 
MQBALSTATE_OK                  2 
MQBALSTATE_HIGH                3 
MQBALSTATE_UNKNOWN             4 

  /* Immovable Reasons */ 
MQIMMREASON_NONE               0 
MQIMMREASON_NOT_CLIENT         1 
MQIMMREASON_NOT_RECONNECTABLE  2 
MQIMMREASON_MOVING             3 
MQIMMREASON_APPLNAME_CHANGED   4 

From MQ 9.1.1 to MQ 9.2.2

CMQC

MQ_NHA_INSTANCE_NAME_LENGTH    48 

MQCMDL_CURRENT_LEVEL           915    
MQCMDL_LEVEL_920               920 
MQCMDL_LEVEL_921               921 
MQCMDL_LEVEL_922               922 
MQCMDL_CURRENT_LEVEL           922 

CMQCF

MQCACF_APPL_IMMOVABLE_TIME     3210 
MQCACF_LAST_USED               3210       
MQCACF_NHA_INSTANCE_NAME       3211 
MQCACF_LAST_USED               3211

MQRQ_STANDBY_ACTIVATED         32 
MQRQ_REPLICA_ACTIVATED         33 

From MQ 9.2.2. to 9.2.4

CMQC

MQBNO structure and  MQBNO_ constants

MQCNO_CURRENT_VERSION          7
MQCNO_VERSION_8                8 
MQCNO_CURRENT_VERSION          8

MQCNO_CURRENT_LENGTH           240    
MQCNO_LENGTH_8                 252 
MQCNO_CURRENT_LENGTH           252 

MQ_TEMPORARY_Q_PREFIX_LENGTH   32 
MQRC_BNO_ERROR                 2602 
MQRC_OUTBOUND_SNI_NOT_VALID    2603 

/* Streaming Queue Quality of Service Values */ 
MQST_BEST_EFFORT               0 
MQST_MUST_DUP                  1 

MQCMDL_CURRENT_LEVEL           922       
MQCMDL_LEVEL_923               923 
MQCMDL_LEVEL_924               924 
MQCMDL_CURRENT_LEVEL           924 

MQCA_LAST_USED                 2137
MQCA_LAST_USED                 2138 

MQIA_LAST_USED                 274      
MQIA_LAST_USED                 275 

MQIA_STREAM_QUEUE_QOS          275 

struct tagMQCNO { 
   ...     
   PMQBNO     BalanceParmsPtr;      /* Balance Parameter Pointer */ 
   MQLONG     BalanceParmsOffset;   /* Balance Parameter Offset */ 
   MQBYTE4    Reserved3;            /* Reserved */ 
   /* Ver:8 */ 
 };   

CMQCFC

MQRCCF_STREAMQ_DEST_NOT_SUPP   3384 
MQRCCF_STREAMQ_DEST_CONFLICT   3385 
MQRCCF_STREAMQ_NOT_SUPPORTED   3386 
MQRCCF_STREAMQ_CONFLICT        3387 
 
MQBACF_LAST_USED               7038    
MQBACF_MQBNO_STRUCT            7039 
MQBACF_LAST_USED               7039 
  
MQIACF_SYSP_SMF_STAT_TIME_MINS 1199 
MQIACF_SYSP_TRACE_CLASS        1200 
  
MQIACF_LAST_USED               1438
MQIACF_BALANCING_TYPE          1439 
MQIACF_BALANCING_OPTIONS       1440 
MQIACF_BALANCING_TIMEOUT       1441 
MQIACF_SYSP_SMF_STAT_TIME_SECS 1442 
MQIACF_SYSP_SMF_ACCT_TIME_MINS 1443 
MQIACF_SYSP_SMF_ACCT_TIME_SECS 1444 
MQIACF_LAST_USED               1444 

MQIACH_AUTH_INFO_TYPES         1622 

MQCACH_LAST_USED               3571      
MQCACH_TEMPORARY_MODEL_Q       3572 
MQCACH_TEMPORARY_Q_PREFIX      3573 
MQCACH_LAST_USED               3573 
                                         
MQIMMREASON_APPLNAME_CHANGED   4 
MQIMMREASON_IN_TRANSACTION     5 
MQIMMREASON_AWAITS_REPLY       6 

Where’s my publish?

Like many things I do, it all started with an easy problem. I had an application which subscribed to a topic, it published on a topic string, then got the published message. Another application subscribed on the same topic string but did not get the message. I’ll leave clues for the experts to spot as I explain the progress I made.

The first complication was the subscribing and publishing application was in Python on z/OS (which runs in ASCII), and the other application was written in C using the default EBCDIC. I had to spend time working out how to convert from ASCII to EBCDIC. (When using CHARV you can specify a string and its code page and have MQ do it for you automatically – easy once you know which code page to use.) For other strings I had to convert manually using the A2E function from C run time.

There are some commands for displaying status of pub sub, it took a little while to understand them and give me the data I wanted.

For example

My application did a subscription with topic string “/currency/rate/EUR/GBP”.
The command gave

DIS TPSTATUS(‘/currency/rate/EUR/GBP’)
CSQM297I … CSQMDRTC NO TPSTATUS FOUND MATCHING REQUEST CRITERIA

Once I had opened the topic using the topic string, I got

CSQM201I … CSQMDRTC DIS TPSTATUS DETAILS
TPSTATUS(/currency/rate/EUR/GBP)
TYPE(TOPIC)
DEFPRESP(SYNC)
PUB(ENABLED)
SUB(ENABLED)
ADMIN()
RETAINED(NO)
PUBCOUNT(0)
SUBCOUNT(1)
PUBSCOPE(ALL)
SUBSCOPE(ALL)

I set up the subscription using Python

topic_string = ‘/currency/rate/EUR/GBP’
sub_desc = pymqi.SD() # create the basic subscription
sub_desc[‘Options’] = pymqi.CMQC.MQSO_CREATE + pymqi.CMQC.MQSO_RESUME +
pymqi.CMQC.MQSO_DURABLE + pymqi.CMQC.MQSO_MANAGED
sub_desc.set_vs(‘SubName’, “MYSUB” )
sub_desc.set_vs(‘ObjectString’, topic_string)

The code to get the subscribed message was

get_opts = pymqi.GMO(
MsgHandle=hMsg.msg_handle,
Options=pymqi.CMQC.MQGMO_NO_SYNCPOINT +
pymqi.CMQC.MQGMO_CONVERT +
pymqi.CMQC.MQGMO_PROPERTIES_IN_HANDLE +
pymqi.CMQC.MQGMO_FAIL_IF_QUIESCING +
pymqi.CMQC.MQGMO_WAIT)
get_opts[‘WaitInterval’] = 1000
data = sub.get(None, pymqi.md(), get_opts)
print(‘Received data: [%s]’ % data)

The code to publish to a topic was

msg = ‘1.3961’
topic = pymqi.Topic(qmgr, topic_string=topic_string)
topic.open(open_opts=pymqi.CMQC.MQOO_OUTPUT)
topic.close()

The Python program was subscribe, publish to topic, get the message.

When I ran the code, no messages were published. I tried changing the topic_string to ‘/currency/rate/EUR/USD’, that had no effect. Eventually I tracked down a code page problem in my code; I fixed that – but the program still did not publish a message.

The more knowledgable people reading this may have spotted one problem.

The first time the program ran, the subscription code created a SUB(MYSUB) with the given topic string (‘/currency/rate/EUR/USD’). When I changed the topic string to ‘/currency/rate/EUR/GBP’, and published a message, the subscription was using the original topic string with USD!, but publishing to GBP. Any changes to the SUBscription are ignored unless MQSO_ALTER is specified.

I deleted the SUB and reran my application. Displaying the subscription (MYSUB) gave

CSQM201I %CSQ9 CSQMDRTC DIS SUB DETAILS 934
SUB(MYSUB)
SUBID(C3E2D8D4C3E2D8F94040404040404040DAFEED99504CE540)
DURABLE(YES)
SUBTYPE(API)
TOPICSTR(/currency/rate/EUR/GBP)
SUBLEVEL(1)

I can display the topic string and see what is subscribed to it

dis tpstatus(‘/currency/rate/EUR/GBP’) type(sub)

This gave two subscriptions:

TPSTATUS(/currency/rate/EUR/GBP)
TYPE(SUB)
SUBID(C3E2D8D4C3E2D8F94040404040404040DAFE062AED26E2C0)
DURABLE(YES)
SUBTYPE(ADMIN)
SUBUSER(IBMUSER)

NUMMSGS(12)

and

TPSTATUS(/currency/rate/EUR/GBP)
TYPE(SUB)
SUBID(C3E2D8D4C3E2D8F94040404040404040DAFC73649B3C7200)
DURABLE(YES)
SUBTYPE(ADMIN)
SUBUSER(IBMUSER)

NUMMSGS(0)

So one subscription has seen 12 messages, the other subscription has seen 0 messages. This does not look like a wrong topic string, as they are resported as part of the display tpstatus.

I could not find a way of displaying all the information about a subscription in one command. I had to use DIS TPSTATUS.. TYPE(SUB), then use cut and paste to issue the DIS SUB SUBID(….). (It would be much easier for the poor end users if there was an ALL option to display all this data in one go.)

For example

SUB(COLIN)
SUBID(C3E2D8D4C3E2D8F94040404040404040DAFC73649B3C7200)
DURABLE(YES)
SUBTYPE(ADMIN)
DISTYPE(RESOLVED)
TOPICSTR(/currency/rate/EUR/GBP)
DEST(CP0000)

DESTQMGR()
DESTCORL(C3E2D8D4C3E2D8F94040404040404040DAFC73649B3C7200)
TOPICOBJ()
PUBACCT(00000000000000000000000000000000000000000000000000000000…)
USERDATA(CZZZ)
SUBUSER(IBMUSER)
SUBLEVEL(1)

So we can see from this subscription the topic string being subscribed on, and the queue name being used.

This subscription had no messages published to it; see the DISPLAY TPSTATUS TYPE(SUB) field NUMMSGS. Those of you with a PhD in PubSub may have spotted the problem; it is very well hidden. It is all to do with PubLevel and SubLevel. (The documentation is very confusing, it implies a subscription can have more than one sublevel).

I think of it as logic within publish which says for each subscription

If SubLevel <= PubLevel then publish the message
else skip the publish
  • The subscription from the python code had SubLevel = 0. (This was latent bug – it should have been set to 1!
  • The SubLevel from my DEFINE SUB command was the default 1.
  • The PubLevel from the Python code was 0 – this was the problem.

So the publish to my manual DEFINE SUB was skipped, and the publish to the Python code worked because PubLevel = 0 matched SubLevel = 0

The message is being published – but it is not still not there.

When I restarted my queue manager and ran the test, the Python application got no message found, which was a surprise to me. When I reran it, with debug code enabled – the get worked, and carried on working all day. Next day the same problem occurred. Very strange – is the debug code affecting the messages ?

I then remembered the default syncpoint behaviour (one of those face palm moments). On midrange the default is to put out out sync-point, so the messages are immediately available. On z/OS the default is to put within sync-point. The messages are only visible after a commit, or the implicit commit at MQDISC. On z/OS the messages were in syncpoint, and there was no commit. When I ran my program the second time, it got the message from the first run, and so on!

I put the commit in – and it all worked.

Check list of what to check

Check you are subscribing to the correct topic string.

Plan carefully how you define your topic(strings).

If you want to have an application define a subscription, consider some defensive programming.

  • MQSO_CREATE will create a subscription based on the specified data. If it already exists report an error.
  • MQSO_CREATE + MQSO_RESUME will create a subscription based on the specified data. If it already exists it will not report an error. A second usage will not change the subscription.
  • MQSO_CREATE + MQSO_ALTER will create a subscription based on the specified data. A second usage will update the subscription. This could be considered dangerous if you have multiple applications running concurrently changing the topic string or other parameters.
  • MQSO_RESUME does not create a subscription, not updates it. it uses the information in the subscription object.

It would be safer for an Administrator to define the MQSUB, and the applications to refer to it. Do not allow application users to be able to define a subscription. They should use the pre-defined MQSUB and add the topic string to give more granular data. For example I could use a null object and have a topic string of /TOP/SECRET/DATA/SPIESIN/ENGLAND. If you haven’t set up security properly then I could access the data.

Check the PubLevel and the SubLevel.

If you are not using this functionality set PubLevel to 9 and SubLevel to 1 ( the defaults).

You can DISPLAY SUB(….) SUBLEVEL to see the specified value.

The PubLevel is specified in the MQPMO.PubLevel. There is no display command for this.

Check the sync-point options.

You should always, always, always specify MQPMO_NO_SYNCPOINT, or MQPMO_SYNCPOINT and the commit. Do not let it default in case someone ports your code to or from z/OS. You can waste days hunting down this problem.

Using LDAP with MQ multiplatform and nested groups.

This blog post follows on from Using LDAP with multi platform.

Nested groups can be used to simplify administration

It is good practice to grant authority to access resources using groups, rather than giving access to individual userids. For example if an application uses 10 queues, and a new person joins the team you can either connect the new id to one group, or give the id access to the 10 queues.

Using nested groups takes this further. Imagine the payroll queues are managed by the GRPAYROLL access group, the HR queues managed by the GRHR access group, and the Finance queue managed by the GRFINANCE access group. The MQ system programmers can manage any queue. You could give GRMQADMIN access to the Payroll queues, HR queues, and Finance queues. Or you say the GRMQADMIN is the super set and includes a reference to GRPAYROLL, GRHR, and GRFINANCE. If you define a HR queue, you just give GRHR access to it, and the MQ Administration team get access to it “for free”.

You can take this further, and have a group GRHR_GR1, and GRHR_GR2 which give access to a subset of HR queues. The GRHR group could include both of these.

The model of this is a tree where a group high up in the tree incorporates the groups lower down the tree.

There are two ways of defining groups in LDAP.

1. Define a group record, and list the members of the group. This is called a static group

dn:cn=GR1,OU=groups,o=myorg
member:cn=user1,ou=users,o=myorg
member:cn=user2,ou=users,o=myorg

To retrieve the groups for a user, you issue a query for all groups which have the given member data.

2. Specify the group name as part of the user’s record, this is known as a dynamic group.

dn=cn=user1,ou=users,o=myorg
group=cn=group1,ou=groups,o=myorg
group=cn=group2,ou=groups,o=myorg

For this you retrieve the “group” attributes from the record. Note: The group attribute does not exist in LDAP, you have to use a different one ( I used the st attribute).

Nested group support.

There is an MQ AUTHINFO parameter NESTGRP. The documentation says

  • NESTGRP Group nesting.
    • YES The group list is searched recursively to enumerate all the groups to which a user belongs.

Static groups and nested group support.

My MQ ADUTHINFO had


AUTHORMD(SEARCHGRP) +
BASEDNG(‘ou=groups,o=your Company’) +
CLASSGRP(‘groupOfNames’) +
GRPFIELD(sn) +
FINDGRP(‘member’) +
NESTGRP(yes) +

On my system the userid ibmuser is a “member” of two groups mqstatic and mqstatic2

dn: cn=mqstatic,ou=groups,o=your Company
objectclass: groupOfNames
cn: mqstatic
ou:groups.
member: cn=ibmuser,o=your Company
member: cn=adcdb,o=your Company

and

dn: cn=mqstatic2,ou=groups,o=your Company
objectclass: groupOfNames
cn: mqstatic2
ou:groups.
member: cn=ibmuser,o=your Company

There is another group, which refers to the mqstatic group.

dn: cn=mega,ou=groups,o=your Company
objectclass: groupOfNames
cn: mega
ou:groups.
member: cn=mqstatic,ou=groups,o=your Company
member: cn=gg,ou=groups,o=your Company

As part of the userid to group mapping, MQ issues the query What groups have member: cn=ibmuser,o=your Company. The response is

  • dn: cn=mqstatic2,ou=groups,o=your Company
  • dn: cn=mqstatic,ou=groups,o=your Company

When NESTGRP(YES) is specified MQ then queries all groups that have cn=mqstatic2,ou=groups,o=your Company, and does another query for dn: cn=mqstatic,ou=groups,o=your Company

and gets the response dn: cn=mega,ou=groups,o=your Company.

The cn=ibmuser is associated with the three groups, and those three groups are used for access checking.

For a data model I see it as

dn=cn=GRHR_GR1
isusedby: CN=GRHR

and

dn:cn=GRHR
isusedby: CN=MQADMIN

and

dn: cn=mqadmin
member:cn=user1

The “isusedby” attribute name, is in reality “member”, which I feel is very confusing, as member implies membership, and there is no membership involved. This is an LDAP naming problem – not MQ’s.

This model feels upside down (or inside out) (or both). It takes several cups of tea with biscuits to draw up the groups and the connections definitions you need. Put the kettle on, and try defining the groups for the MQADMIN, GRFINANCE, and GRFINANCEGR1, GRFINANCEGR2 groups; three people in MQADMIN, and two people in finance; then add a definition for GRFINANCEGRN.

Using dynamic groups and nested group support

This feels more the right way up – but it is upside down compared to static groups.

My authinfo had

AUTHORMD(SEARCHUSR) +
FINDGRP(‘st’) +
BASEDNU(‘o=Your Company’) +
CLASSGRP(‘person’) +

With my userid I had

dn: cn=user1, o=Your Company
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: ibm-nativeAuthentication
objectclass: inetOrgPerson
departmentNumber: mqadmin
cn: ibmuser
sn: ibmuser
ou: test
st: cn=group,ou=groups,o=your Company
st: cn=mqadmin,ou=groups,o=your Company
ibm-nativeId: ibmuser

Where st: defined the groups.

For the entry for cn=group, I defined a nested group called cn=deepergroup.

dn: cn=group,ou=groups,o=your Company
objectclass: top
objectclass: person
objectclass: organizationalPerson
cn: cpgroup
ou: groups
st : cn=deepergroup,o=your Company
sn : groupgroup

You need an entry for cn=deepergroup… in LDAP

dn: cn=deepergroupou=groups,o=your Company
objectclass: person
cn: mega
sn:mega

# no st attribute, so there is no more nesting

I define the cn=mqadmin group to LDAP with no “sn” entry, because it has no nesting.

MQ issues a query to LDAP asking for the group definitions ( st in my case) for the given user. For each group that gets returned, query that group for any more group (st in my case) definitions, etc.

Together these give three groups for the cn=ibmuser:

  • cn=mqadmin,ou=groups,o=your Company
  • cn=deepergroup,ou=groups,o=your Company
  • cn=group,ou=groups,o=your Company

Dynamic group data model

The data model for dynamic group feels more natural, where the “is in” and “includes” are the attribute you chose, “st” in my case.

  • dn=cn=user1
    • is in: cn=group1…
  • dn=cn=group1…
    • includes: cn=groupdeep

Enabling nestgrp(yes) with dynamic groups may throw errors

If you start with

dn=cn=user1,…
group=cn=group1,..
group=cn=group2,…

With nestgrp(no) this defines the userid is in two groups.

If you enable nestgrp(yes) then when the user id is used, MQ checks there are entries in LDAP for group=cn=group1,.. and group=cn=group2,… If the definitions do not exist, you get the MQ message:

AMQ5532E: Error authorizing entity in LDAP

EXPLANATION:
The LDAP authorization service has failed in the ldap_first_entry call while trying to find user or group ‘NULL’. Returned count is 0. Additional context is ‘cn=group1,… ‘.

To solve the problem I defined the two groups to LDAP.

For group1 I defined

dn:cn=group1,…
group:cn=deepergroup…

and I defined the deepergroup group (to keep mq happy) with no group entries (to indicate end of the nesting).

dn:cn=deepergroup,…

Using LDAP with MQ multi platform.

MQ multiplatform can use LDAP as a userid and group repository, so you can logon to any machine where MQ is running, and use your corporate userid and password.

I’ve logged on to MQ on my Linux machine, and used my z/OS userid and password. It was pretty easy to set up (I had prior experience of using LDAP) but although it didn’t quite behave as I thought it would – I thought this was pretty clever.

Once you have installed LDAP, started it, and created your directory structure, (user, groups) and access permissions, you can start to use it. I’ve documented some of the initial settup here. It covers some of the concepts referred to below.

I used LDAP (Tivoli Directory Server) on z/OS as my LDAP server.

Contents

I’ve also written using LDAP with MQ and nested groups (MQ NESTGRP).

Using LDAP from MQ Multiplatform

The IBM documentation for this is so-so. It gives examples, but the examples didn’t work for me, but they were enough go point me in the right direction.

Start here.

I created a LDAP.MQSC file with

DEFINE AUTHINFO(MYLDAP) +
AUTHTYPE(IDPWLDAP) +
CONNAME(‘10.1.1.2(389)’) +
AUTHORMD(SEARCHGRP) +
BASEDNG(‘o=Your Company’) +
BASEDNU(‘o=Your Company’) +
LDAPUSER(‘cn=adcda, o=Your Company’) +
LDAPPWD(‘adcdapw1’) +
SECCOMM(NO) +
CLASSUSR(‘ibm-nativeAuthentication’) +
CLASSGRP(‘groupOfNames’) +
GRPFIELD(sn) +
SHORTUSR(sn) +
REPLACE

ALTER QMGR CONNAUTH(MYLDAP)

REFRESH SECURITY TYPE (CONNAUTH)
* ALTER QMGR CONNAUTH(SYSTEM.DEFAULT.AUTHINFO.IDPWOS)

Where the key fields for connecting to LDAP are

  • conname – the IP address of the LDAP server.
  • ldapuser and ldappwd – userid and password to access LDAP.
  • seccomm – use TLS/SSL to contact the LDAP server. I used “no” while setting this up.

the key fields for identifying users are

  • basednu – the subtree to be used for userids, for example all users are one level under ou=user,o=myorg.
  • classusr – is the objectclass attribute to identify the userid. The default is inetOrgPerson.
  • shortusr – the dn identifiers are too long. MQ needs IDs with 12 characters or less. This attribute says which attribute to use.

the key fields for identifying which groups an id belongs to

  • authormd – how to search for the authorisation.
  • basedng – the subtree to be used for groups, for example ou=group,o=myorg.
  • classgrp – the objectType which objects must have to be recognised as a group
  • grpfield – the simple name of the group
  • findgrp – what to filter for. For example ‘member’

Being a careful person, I started an interactive runmqsc session in one terminal, and used runmqsc … < LDAP.MQSC in another window. This way if there were problem I could use the interactive session to reset the QMGR CONNAUTH (as in the comment above). I know that the userid that started the queue manager, does not need a password; so if you issued strmqm qma you can use runmqsc qma without a userid and password. It gets harder if your id is not the id the queue manager is running under.

LDAP definition of my logon userid

The userid I wanted to use with MQ was defined

dn: cn=ibmuser, o=Your Company
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: ibm-nativeAuthentication
objectclass: inetOrgPerson
cn: ibmuser
sn: ibmuser
ou: test
st: cn=group,ou=groups,o=your Company
st: cn=mqadmin,ou=groups,o=your Company
ibm-nativeId: ibmuser

The authinfo data has

BASEDNU(‘o=Your Company‘) +
CLASSUSR(‘ibm-nativeAuthentication‘) +
SHORTUSR(sn) +

When I try to logon with userid ibmuser, MQ issues an LDAP query for the record with

  • sn=ibmuser
  • with an object class = ‘ibm-nativeAuthentication (which provides the RACF support for userid and password)
  • in the subtree o=Your Company

Check the LDAP configuration when things go wrong

It took me a few hours to determine why I could logon with one id, but not another id. Some LDAP entries worked, and some did not. It turned out to be an Access Control List (ACL) set up problem, where the LDAPUSER userid was not authorised to see some of the records. With the above AUTHINFO object, the query that MQ uses to check authorisation is like

ldapsearch -h 127.0.0.1 -D “cn=adcda, o=Your Company” -w ? -b “o=Your Company” “&(objectClass=ibm-nativeAuthentication)(sn=zadcdc)”

Where the parameters match up with the authinfo object above, and zadcdc is the userid trying to logon.

If you get no data back, get an authorised person (cn=ibmuser…) to issue the command for the user problem userid zadcdc:

ldapsearch -h 127.0.0.1 -D “cn=ibmuser, o=Your Company” -w ? -b “o=Your Company” “&(objectClass=*)(sn=zadcdc)”aclentry aclsource

The aclentry will give you the userids or groups who are authorised to use the entry, and the access they have.

The aclsource tells you which node in the tree the ACL was inherited from (for example aclsource=o=Your Company says it came from the root node). I had set up an ACL for my zadcdc which did not include my LDAPUSER.

Setting up MQ connect authorities

You can issue the command

setmqaut -m qml -t qmgr -p ibmuser +connect

to give ibmuser connect authority.

You can use LDAP groups for example

setmqaut -m qml -t qmgr -g “cn=mqadmin,ou=groups,o=your Company” +connect.

How do you set up a group in LDAP?

This is where it gets interesting. You can define a static group with its list of members, or create a dynamic group which is more flexible and “modern” (where modern is within the last 30 years).

Using a static group (with a list of members defined in it)

You can define a static group in LDAP using

dn: cn=mqstatic,ou=groups,o=your Company
objectclass: groupOfNames
ou:groups.
member: cn=ibmuser,o=your Company
member: cn=adcdb,o=your Company

It has two members.

When the userid authenticates, the queue manager asks LDAP for the groups that the userid is in; using the AUTHINFO definitions CLASSGRP(‘groupOfNames’) + GRPFIELD(…) FINDGRP(‘…’) a query is done for the groups which have the userid id. For example with

BASEDNG(‘ou=groups,o=your Company’) +
CLASSGRP(‘groupOfNames’) +
FINDGRP(‘member’) +

and cn=ibmuser o=your Company. The query is

(&(objectClass=groupOfNames)(member=cn=ibmuser, o=Your Company) in subtree (ou=groups,o=your Company)

The ldap search asking to return the member attribute

ldapsearch … -b “ou=groups,o=your Company” “(&(objectClass=groupOfNames) (member=cn=ibmuser, o=Your Company)) ” cn

gave two group names –

cn=mqstatic2,ou=groups,o=your Company
cn=mqstatic,ou=groups,o=your Company

This gives the information as to which groups a user is in. MQ then saves the userid and group information, and does not need to go to LDAP the next time the userid needs access checking.

Knowing which groups a userid is in, MQ can then decide on the access by comparing with the setmqaut definitions.

Using a dynamic group – or using a group attributer in the user record.

Instead of a group having a list of members, you can add information to the user’s record.

For example

dn: cn=adcda, o=Your Company
changetype: add
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: ibm-nativeAuthentication
sn: adcda
group: cn=group,o=your Company
cn: mq
ibm-nativeId: adcda

Unfortunately there is no “group” attribute defined in the LDAP schema, so I had to find another attribute to use. I used st for state. I used st: cn=group,o=your Company instead of group: cn=group,o=your company.

In my MQ AUTHINFO definition I had

BASEDNU(‘o=Your Company’) +
CLASSUSR(‘ibm-nativeAuthentication’) +
SHORTUSR(sn) +
AUTHORMD(SEARCHUSR) +
FINDGRP(st,‘) +
BASEDNG(‘ou=zzzzz,o=your Company’) +

MQ did an LDAP search using this information

ldapsearch… -b “o=Your Company” “&(objectClass=(ibm-nativeAuthentication)
(sn=ibmuser))” st

Which is just a display of the userid information – and return the fields with the attribute you specified(st). It returned

cn=ibmuser, o=Your Company
st=cn=group,ou=groups,o=your Company
st=cn=mqadmin,ou=groups,o=your Company

This tells MQ to use the groups

cn=group,ou=groups,o=your Company and cn=mqadmin,ou=groups,o=your Company.

You can use the setmqaut command to give the group access

setmqaut -m qml -t qmgr -g “cn=mqadmin,ou=groups,o=your Company” +connect

Once this was done, the cn=ibmuser could connect to MQ using groups.

Can I use LDAP to hold my setauth information?

Only the group and userid information are held in LDAP, all the other information is held in the queue manager. You cannot use LDAP to have your setmqaut configuration in LDAP, and shared by multiple queue managers. You still have to use setmqaut to set up each queue manager access.

Giving userids access to MQ objects

You can use the setmqaut command or the set authrec runmqsc command to give principals or groups access to resources.

For example

setmqaut -m qml -n CP0000 -t queue -g “cn=mqstatic,ou=groups,o=your Company” +inq

I’ve changed the definitions in LDAP – when will they get picked up?

Changes get picked up when

  • the queue manager is restarted
  • when the resfresh security, refresh security type(authserv) or the refresh security type(connauth) command is issued.