if you are interested in getting MQ Statistics into a CSV format, then Bobbee Broderick, has extended the pymqi package with Program to produce CSV file of MQ Statistics. I haven’t used it, but I said I would advertise it.
Category: MQ on Midrange
Should all red flags be green?
This question came out of a discussion on an MQ forum, where the question was if MQ does one time delivery, how come he got the same message twice?
Different sorts of MQ gets.
There are a variety of patterns for getting a message.
- Destructive get out of sync-point. One application can get the message. It is removed from the system. As part of the MQGET logic there is a commit of the get so once it has gone it has gone. This is usually used for non persistent message. Persistent messages are usually processed within sync-point, but there are valid cases when the get of a persistent out of sync-point is valid.
- Destructive get within sync-point. One application can get the message. The queue manger holds a lock on the message which makes it invisible to other applications. When the commit is issued (either explicitly or implicitly) , the message is deleted. If the application rolls back (either implicitly of explicitly) the message becomes visible on the queue again, and the lock released.
- Browse. One or more applications can get the message when using the get-with-browse option. Sync-point does not come into the picture, because there are no changes to the message.
- One problem with get-with-browse is you can have many application instances browsing the queue, and they may do the same work on a message, wasting resources. To help with this, there is cooperative browse. This is effectively browse and hide. This allows a queue monitor application to browse the message, and start a transaction saying process “this” message. A second instance of the queue monitor will not see the message. If the message has not been got within a specified time interval the “hide” is removed, and so the message becomes visible. See Avoiding repeated delivery of browsed messages.
The customer’s question was, that as the get was destructive, how come the message was processed twice – could this be a bug in MQ?
The careful reader may have spotted why a message can be got twice.
Why the message was processed “twice”.
Consider an application which does the following
MQGET destructive, in sync-point
Write “processed message id …. ” to the log
Update DB2 record
Commit
You might the see following in the log
processed message id x’aabbccdd01′.
processed message id .x’aabbccdd02′.
processed message id x’eeffccdd17′. .
Expanding the transaction to give more details
MQGET destructive, in sync-point
Write “processed message id …. ” to the log
Update DB2 recordIf DB2 update worked then commit
else backout
If there was a DB2 problem, you could get the following on the log:
processed message id x’aabbccdd01′.
processed message id x’aabbccdd01′.
processed message id x’eeffccdd17′. .
You then say “Ah Ha – MQ delivered the message twice”. Which is true, but you should be saying “Ah Ha – MQ delivered the message but the application didn’t want it. The second time MQ delivered it, the application processed it”. Perhaps change the MQ phrase to “MQ does one time successful delivery“.
Why is this blog post called Should all red flags be green?
A proper programmer (compared to a coder), will treat a non successful transaction as a red flag, and take an action because it is an abnormal situation. For example write a message to an error log
- Transaction ABCD rolled back because “DB2 deadlock on ACCOUNTS table”
- Transaction ABCD rolled back because “MQ PUT to REPLYQUEUE failed – queue full”
- Transaction ABCD rolled back because “CICS is shutting down”
The Architects and systems programmers can look at these messages and take action.
For example with DB2, investigate the lock held duration. Can you reduce the time the lock is held, perhaps by rearranging with within a unit of work, for example “MQGET, MQPUT reply, DB2 update, commit” instead of “MQGET, DB2 update, MQPUT of reply, commit.
For MQ queue full, make the maximum queue depth bigger, or find out why the queue wasn’t being drained.
CICS shutting down.You may always get some reasons to rollback.
Once you have put an action plan in place to reduce the number of red flags, you can mark the action item complete, change its status from red to green and keep the project managers happy (who love green closed action items).
Note: This may be a never ending task!
After thought
In the online discussion, Morag pointed out that perhaps the same message was put twice. Which would show the same symptoms. This could have been due to a put out of syncpoint, and the transaction rolled back.
Debugging AT-TLS session problems
I deliberately misconfigured AT-TLS to see how easy it would be to identify and resolve the problems from an AT-TLS perspective. It turned out worse than I expected. There is little information on the z/OS to help you.
I configured TTLSEnvironmentAction {trace 255 } (see the bottom of this blog) and refreshed the PAGENT. I had configured SYSLOGD so records for *.TCPIP.*.* went to /var/log/TCPIP.
I reran my MQ client application and got
- from MQ on Linux, in file /var/mqm/errors/AMQERR01.LOG return code 2393 (MQRC_SSL_INITIALIZATION_ERROR).
- On Linux there was a file /var/mqm/trace/AMQ.SSL.TRC – which only IBM can format!
- From TCPIP on z/OS EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900 JOBNAME: CSQ9CHIN RULE: REMOTE-TO-CSQ1 USERID: START1 GRPID: 0000001B ENVID: 0000000B CONNID: 0000006E This give
- the address of my client,
- the name of the chinit
- which AT-TLS rule was used
The message EZD1287I TTLS Error RC: 402 Initial Handshake pointed me to Cryptographic Services System Secure Sockets Layer Programming – No SSL cipher specifications. The first reason was
The client and server cipher specifications do not contain at least one value in common. Client and server cipher specifications might be limited depending on which System SSL FMIDs are installed. See Cipher suite definitions for more information. Server cipher specifications are dependent on the type of algorithms that are used by the server certificate (RSA, DSA, ECDSA, or Diffie-Hellman), which might limit the options available during cipher negotiation.
MQ Trace
I took an MQ trace and formatted it. I used grep to find which file had “Cipher” in it.
Within this file I searched for Start of GSKit TLS Handshake Transcript.
This had information sent to the server as part of the handshake, and further down it had the reason code. You can see from the example that the fields and their values have been displayed (so cipher spec 003c is displayed as tls_rsa_with_aes_128_cbc_sha256)
Start of GSKit TLS Handshake Transcript (1119 bytes) <client_hello> client_version TLSV12 random gsksslDissector_32Bits 7f9d66d8 gsksslDissector_Opaque Length: 28 3E 5B 45 66 EE A3 C1 9F FB 81 0C 2F 38 19 DF 95 >[Ef......./8... 5A 1B 54 CC B8 CB B6 C9 87 39 5E 88 Z.T......9^. session_id Length: 00 cipher_suites Length: 04 00 FF 00 3C ...< tls_ri_scsv,tls_rsa_with_aes_128_cbc_sha256 compression_methods Length: 01 00 . Extensions Length: 74 00 0D 00 18 00 16 06 01 05 01 04 01 03 01 02 01 ................ 06 03 05 03 04 03 03 03 02 03 02 02 00 00 00 2A ...............* 00 28 00 00 25 73 79 73 74 65 6D 32 65 2D 64 65 .(..%system2e-de 66 32 65 2D 73 76 72 63 6F 6E 6E 2E 63 68 6C 2E f2e-svrconn.chl. 6D 71 2E 69 62 6D 2E 63 6F 6D mq.ibm.com Extension Count: 2 signature_algorithms 13 rsa:sha512,rsa:sha384,rsa:sha256,rsa:sha224,rsa:sha1, ecdsa:sha512,ecdsa:sha384,ecdsa:sha256,ecdsa:sha224, ecdsa:sha1,dsa:sha1 server_name 0 system2e-def2e-svrconn.chl.mq.ibm.com End of GSKit TLS Handshake Transcript
{ rriEvent
...
RetCode = 20009665, rc1 = 420, rc2 = 0, Comment1='SYSTEM.DEF.SVRCONN',
Comment2='gsk_secure_soc_init', Comment3='10.1.1.2(1414)'
...
}
With this trace, I am able to see what was sent to z/OS.
The AT-TLS Trace
The trace ( configured in syslogd to be in /var/log/TCPIP) had a one line entry with (I’ve reformatted it to make it easier to read).
Map CONNID: 0000006B LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53898 JOBNAME: CSQ9CHIN USERID: START1 TYPE: InBound STATUS: Enabled RULE: REMOTE-TO-CSQ1 ACTIONS: CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A
and data
RC: 0 Connection Init Initial Handshake ACTIONS: CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A HS-Server RC: 0 Call GSK_SECURE_SOCKET_OPEN - 00000052FD6228F0 RC: 0 Set GSK_FD(300) - 000000000000006B RC: 0 Set GSK_USER_DATA(200) - 000000007EC32430 RECV CIPHER 160303007B
and one loooong record with
RECV CIPHER 010000770303749ED51D8DC7794EE6AC36B01FD115F38A4B0812D35 C80A5F95DB840C35735CA00000400FF003C0100004A000D00180016 060105010401030102010603050304030303020302020000002A002 800002573797374656D32652D64656632652D737672636F6E6E2E63 686C2E6D712E69626D2E636F6D
SEND CIPHER 15030300020228
From the AT-TLS trace of the data received from the client, it is the data as received, and has not been split down into useful fields.
I could not find any documentation on how to format this string. It is not easy to create a program to format this (and get it right), for example converting cipher spec 003c to TLS_RSA_WITH_AES_128_CBC_SHA256. However I have a REXX exec which works in ISPF and decodes the data into fields, but not the contents of the fields – so the cipher spec is reported as 003c
I had some success taking this data, and creating a file which Wireshark could process. See Wireshark – using external data: Bodging a hex dump file. This was not always successful, as it looks like the data is truncated, and can have non hex data in the hex stream.
Note, the System SSL server started task, GSKSRVR, can capture System SSL trace. The output is like
Job TCPIP Process 0101001D Thread 00000001 read_v3_client_hello Received CLIENT-HELLO message
with no detailed information
Tracing just one session
If you have a busy system you could get trace data for many sessions. You may want to set up a TLS rule, so you use a “debug port”, or you specify the remote host IP address, and port, using information from the error message
EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900 …
And dont forget…
And do not forget to reset the TTLSEnvironmentAction entry to reset the trace, and to refresh the PAGENT.
Trying to use PCF and decode the output?
I struggled to decode PCF output; for example decode PCF type 1203 and its value 20. MQ provide most of what you need, there is just one little link in the chain which is missing
In the MQ provided CMQSTRC header file are routines for converting the values to strings
For example
char *MQSYSP_STR (MQLONG v)
{
char *c;
switch (v)
{
case 0: c = "MQSYSP_NO"; break;
case 1: c = "MQSYSP_YES"; break;
case 2: c = "MQSYSP_EXTENDED"; break;
case 10: c = "MQSYSP_TYPE_INITIAL"; break;
case 11: c = "MQSYSP_TYPE_SET"; break;
case 12: c = "MQSYSP_TYPE_LOG_COPY"; break;
case 13: c = "MQSYSP_TYPE_LOG_STATUS"; break;
case 14: c = "MQSYSP_TYPE_ARCHIVE_TAPE"; break;
case 20: c = "MQSYSP_ALLOC_BLK"; break;
so if you know this your type is a System Parameter, you can use MQSYSP_STR(20) to get back the data item MQSYSP_ALLOC_BLK.
The bit that is missing is the mapping between PCF type and the function call.
In GitHub I’ve created this mapping for all of the PCF types (MQMAP.h). This has for example
MQMAP(MQIA_APPL_TYPE, MQAT_STR), // 1 MQMAP(MQIA_DEF_INPUT_OPEN_OPTION, MQOO_STR), // 4 MQMAP(MQIA_DEF_PERSISTENCE, MQPER_STR), // 5 MQMAP(MQIA_DEFINITION_TYPE, MQQDT_STR), // 7
I also provide some routines to help you call this and prettify the value.
I’ve also provided some code so you just need to issue
getPCFValue(MQLONG what,
MQLONG value,
char **pWhat,
char **pValue,
char **pPValue);
Where
- what is the PCF data type (MQIA_TRIGGER_TYPE)
- value is the PCF value (for example 3)
- pWhat gets the name of the PCF data type (“Trigger_Type”)
- pValue gets the value returned from the MQ provided function(“MQTT_DEPTH”)
- pPValue gets the prettified value, with the prefix removed, and the remained made more readable(“Depth”)
This will allow you to run along PCF data and display all the data and values in a similar manner to a display command.
The data in MQMAP.h is in numerical sequence, so I can use a binary search to quickly find the mapping function. I also provide a small C function which takes the MQMAP.h file, checks it is in order and displays it, so it can be sorted.
What does MQRCCF_PARM_CONFLICT mean?
I’ve been using PCF (from Python) and have been getting MQRCCF_PARM_CONFLICT. For example use MQCMD_CHANGE_Q_MGR and set MQIA_TCP_CHANNELS to 201.
The documentation does not help. It says
Explanation
Incompatible parameters or parameter values.
The parameters or parameter values for a command are incompatible. One of the following occurred:
- A parameter was not specified that is required by another parameter or parameter value.
- The MQIA_TCP_CHANNELS can be used on its own – so it is not this one.
- A parameter or parameter value was specified that is not allowed with some other parameter or parameter value.
- The MQIA_TCP_CHANNELS was specified on its own – so it is not this one.
- The values for two specified parameters were not both blank or non-blank.
- Only one parameter was specified – so not this one.
- The values for two specified parameters were incompatible.
- Only one parameter was specified – so not this one.
I found the problem by issuing the command and seeing what the response was.
QM01 ALTER QMGR TCPCHL(201)
gave
CSQM150I QM01 CSQMAMMS ‘TCPCHL’ AND ‘MAXCHL’ VALUES ARE INCOMPATIBLE
CSQ9023E QM01 CSQMAMMS ‘ ALTER QMGR’ ABNORMAL COMPLETION
So I would add reason
5. The specified value is inconsistent with the configuration.
“The system resource RLIMIT_NOFILE is set at an unusually low level for IBM MQ”
I had this problem, and fixed it, and now I’ve upgraded to Ubuntu 20.04 it has come back again.
There are various hints like IBM Support telling me to change /etc/security/limits.conf, but these did not solve the problem for me.
I understand the words below, but not the overall picture. If you get this problem – and the standard changes do not work – try the /etc/systemd/user.conf change as well.
Changes to /etc/security/limits.conf apply to non GUI command windows and limits.conf is not used when systemd is running.
Recent changes to systemd include
- The Linux kernel’s current default RLIMIT_NOFILE resource limit for userspace processes is set to 1024 (soft) and 4096 (hard). Previously, systemd passed this on unmodified to all processes it forked off. With this systemd release the hard limit systemd passes on is increased to 512K, overriding the kernel’s defaults and substantially increasing the number of simultaneous file descriptors unprivileged userspace processes can allocate …
For graphical login. command windows (such as GNOME on Linux) you need.. (I’m copying)
Add the following line to /etc/systemd/user.conf:
DefaultLimitNOFILE=65535
That change works, but only affecting the soft limit. (Leaving us capped with a hard limit of 4096 still.) In order to affect the hard limit, we must modify /etc/systemd/system.conf with the same changes.
Hell is PCF.
In Dante’s book “Inferno”, the gates of hell have “Abandon all hope, ye who enter here” inscribed upon them. The book describes Dante’s journey to the deepest levels of hell.
I had a similar journey, not so far, trying to use PCFs in common code.
I thought I knew how to use MQ PCFs to display and extract information about MQ and its objects, but “fools rush in where angels fear to tread”.
Thanks to Morag Hugson of MQGEM for her help in getting me through this journey.
The kindergarten course on using PCF
Do not use PCF, use the display commands and parse the output. Parsing the output is a simpler problem to solve than trying to use PCF.
The high school course.
Instead of using a command DIS Q(…) you use PCF; create a message with a series of control blocks, send the request to a system queue, and get the data back in a similar format.
There is a header control block to define the function, and 0 or more control blocks each of which define a field.
- The first control block, the header block, defines the request. It has an integer request field, such as MQCMD_INQUIRE_Q.
- The second and following control blocks are field control blocks, they have
- A field to define the type of data, for example character string, byte string (may contain hex 00s), integer, 64 bit integer, or arrays of data.
- A field to define which field is defined, for example the integer MQCA_Q_NAME
- For strings and byte strings, the length of the data.
- For strings the code page of the data.
- The data.
The request is put to a system queue and the response comes back in the reply_to queue you specified. This has a similar format as the request:
- There is a header giving the request type, (MQCMD_INQUIRE_Q), the condition and reason code, and a count of the fields returned.
- There are 0 or more field control blocks – the same as above.
Your program can chain down the field control blocks and can display the fields.
The program from the first year high school will report data like
field type 20 has value 1
which is not very usable.
Using the supplied header file CMQSTRC, you can call MQIA_STR(20) and get back “MQIA_Q_TYPE”. So now you know this field is the Queue Type data.
Now you know it is a Q_Type you can use the function MQQT_STR(1) and get back the string “MQQT_LOCAL”. You can determine the queue type value as a string using
if (field_type=MQIA_Q_TYPE )
queue_type= MQQT_STR(value)
else if (field_type=MQIACF_PUBSUB_PROPERTIES,)
property_type=MQPSPROP_STR(value)
else if (field_type...)
You need a bit of code to match up the field type to the routing you need to call to get the data back. I am working on a function getValue(20,0) to get back “MQIA_Q_TYPE”, and “MQQT_LOCAL”. One call which returns, the data type and its character representation of the value, which will make life much easier for the programmer. There are over 200 field types and it is hard to find the mapping for the value.
In early days of MQ there were two choices of header. You issued a header with type MQCFT_COMMAND and got back one reply with a header type MQCFT_RESPONSE, and all of the data.
Not too difficult so far.
Going down to the next level of hell with z/OS. The request “inquire_chinit” goes to the command server, which sends back “Request received”. The command server then sends the request to the chinit, which sends the response back to the command server, which sends it on to your reply to queue.
You now have to manage at least two reply messages, one for the “Request received”, and one or more for the data.
The next complication which takes you down to the next level of hell, is Shared Queue on z/OS.
You can issue the PCF request saying “hello all queue managers, please give me information about the queue status as you see it for queue COLINPAICEQ”. This request can now go to multiple queue managers, and you get multiple replies back which you have to manage.
This is not well documented in the official documentation. It jumps straight to the PhD class.
I’ll try to explain the data you get back with some examples.
Example 1 Inquire queue manager.
I sent the PCF request with MQCMD_INQUIRE_Q_MGR.
Two messages were returned. They had the same msgid and correlid
The first message was type MQCFT_XR_ITEM with sequence number 1, the second message was MQCFT_XR_SUMMARY with sequence number 2.
First message
Header
- Type MQCFT_XR_ITEM: Message is an extended response to an Inquire command. It contains item data.
- Command Request MQCMD_INQUIRE_Q_MGR
- Condition code 0
- Reason code 0
- Sequence number 1
- It is not last message in set.
- Parameter count 107
Data fields
- MQBACF_RESPONSE_ID … a 24 character string with hexadecimal values. Think of this as a correlator field, so you can match all the responses for this request.
- MQCACF_RESPONSE_Q_MGR_NAME : ‘CSQ9’. This is the queue manager which sent the response.
- MQCA_Q_MGR_NAME : ‘CSQ9’
- MQCA_Q_MGR_DESC : ‘CSQ9, IBM MQ for z/OS – V9.0.1’
- MQIA_PLATFORM : MQPL_ZOS
- MQIA_CPI_LEVEL : 100
- MQIA_COMMAND_LEVEL : MQCMDL_LEVEL_924
- …
Second message
Header
- Type: MQCFT_XR_SUMMARY . Message is an extended response to a command. It contains summary information.
- Request INQ_QMGR
- Condition code 0
- Reason code 0
- Sequence number 2
- It is the last message in set.
- Number of parameters 2
- MQBACF_RESPONSE_ID: …. same as above.
- MQCACF_RESPONSE_Q_MGR_NAME: ‘CSQ9’
These two fields are used to correlate requests. I think …XR_SUMMARY really means end of messages in this set.
Example 2 Inquire chinit.
I sent the PCF request with MQCMD_INQUIRE_CHANNEL_INIT.
Five messages were returned. They all had the same msgid and correlid
The first response message was type MQCFT_XR_MSG with sequence number 1.
The the remaining four messages were part of one set. They had
- Queue manager information, such as number of dispatchers.
- Information about the TCP/IP listener
- Information about the LU 62 listener
- The “end” or summary record
First set, single message
Header
- Type 17 MQCFT_XR_MSG: Message is an extended response to a command. It contains informational or error details.
- Command MQCMD_INQUIRE_CHANNEL_INIT
- Condition code 0
- Reason code 0
- Sequence number 1
- It is the last message in set.
- 4 field control blocks were returned.
Field control blocks
- MQBACF_RESPONSE_ID: Value: 24 bytes of data (with the queue manager name in it
- MQCACF_RESPONSE_Q_MGR_NAME: 48 character queue manager name
- MQIACF_COMMAND_INFO : MQCMDI_COMMAND_ACCEPTED
- MQBACF_RESPONSE_SET: Same value as the first field control block
Set 2, first message
Header
- Type: MQCFT_XR_ITEM: Message is an extended response to an Inquire command. It contains item data.
- Request Command MQCMD_INQUIRE_CHANNEL_INIT
- Reason code 0
- Sequence number 1
- It is not the last message in set.
- Parameter count 20.
Field control blocks – information about the chinit
- MQBACF_RESPONSE_ID: Value: 24 bytes of data (with the queue manager name in it)
- MQCACF_RESPONSE_Q_MGR_NAME: 48 character queue manager name
- MQIACF_CHINIT_STATUS : MQSVC_STATUS_RUNNING
- MQIACH_ADAPS_STARTED : 8
- MQIACH_ADAPS_MAX :8
- MQIACH_DISPS_STARTED :5
- MQIACH_DISPS_MAX :5
- …
Set 2, second message
Header
- Type: MQCFT_XR_ITEM Message is an extended response to an Inquire command. It contains item data.
- Request Command MQCMD_INQUIRE_CHANNEL_INIT
- Reason code 0
- Sequence number 2
- It is not the last message in set.
- Parameter count 7.
Field control blocks – status of TCP listener
- MQBACF_RESPONSE_ID …
- MQCACF_RESPONSE_Q_MGR_NAME : b’CSQ9′
- MQIACH_LISTENER_STATUS : MQSVC_STATUS_RUNNING
- MQIACH_XMIT_PROTOCOL_TYPE : MQXPT_TCP
- MQIACH_INBOUND_DISP: MQINBD_Q_MGR
- MQIACH_PORT_NUMBER :1414
- MQCACH_IP_ADDRESS : b’*’
Set 2, third message
Header
- Type: MQCFT_XR_ITEM Message is an extended response to an Inquire command. It contains item data.
- Request Command MQCMD_INQUIRE_CHANNEL_INIT
- Reason code 0
- Sequence number 2
- It is not the last message in set
- Parameter count 5.
Field control blocks – status of LU62 listener
- MQBACF_RESPONSE_ID ….
- MQCACF_RESPONSE_Q_MGR_NAME : b’CSQ9′
- MQIACH_LISTENER_STATUS : MQSVC_STATUS_STOPPED
- MQIACH_XMIT_PROTOCOL_TYPE : MQXPT_LU62
- MQIACH_INBOUND_DISP MQINBD_Q_MGR
Set 2, fourth (and last) message
Header
- Type: 18 MQCFT_XR_SUMMARY Message is an extended response to a command. It contains summary information.
- Request Command MQCMD_INQUIRE_CHANNEL_INIT
- Reason code 0
- Sequence number 4
- It is the last message in set.
- Parameter count 2.
Field control blocks
- MQBACF_RESPONSE_ID …
- MQCACF_RESPONSE_Q_MGR_NAME : ‘CSQ9’
I do not have access to a QSG, so cannot document the records for a QSG wide query.
Creating the data fields
If you want to create a PCF command to change a queue, you need the PCF header to describe the command, and data fields to identify the queue, and changed attributes.
The header is defined with control block MQCFH (not MQPCFH, I keep trying to use). There are several fields, the key one is Command. In this case we need MQCMD_CHANGE_Q. All request begin with MQCMD. There is MQCMD_INQUIRE_Q for displaying information about a queue.
There is a good post on PCF fields, called Morag’s Quirk’s #14: MQCFH Versions which contains some good information on data types and their usage.
There is a variety of data field record types, such as Byte string, Character string, Integer; and different usages
The field can be
- passed with the command, (a selector), and returned. For example MQIA_, MQCA_
- returned (but not passed through with a command); for queue manager for example MQCACF, and MQIACF) and for the CHINIT for example (MQCACH and MQIACH)
or to look at it a different way…
- MQIA_ are the set used on MQINQ – and thus defined in cmqc.h. They can also be used in PCF rather than defining a second ID for the same parameter.
- MQIACF_ are the set used solely for PCF – and thus defined in cmqcfc.h. They are not used on MQINQ.
- MQIACH_ are the set used for channel commands, they are defined in cmqcfc.h.
The same logic is for for MQCA_/MQCACF_/MQCACH_.
Each field has a unique number, and the field types are generally grouped within a range, but not all of them. The monitoring integers and 64 bit integers are all mixed up.
To find the string representation of the unique number you can call the formatting functions (*MQBACF_STR, *MQCA_STR, *MQCACF_STR, *MQCACH_STR, *MQIA_STR, *MQIACF_STR, *MQIACH_STR, *MQIAMO_STR, *MQIAMO64_STR) in CMQSTRC.h. These routines return a string matching the number or the empty string ”. I found I have to go through them each in turn until there is a match.
In more detail.
- Individual fields
- A byte string (MQCFBS) . This is a string which can include hexadecimal characters, such as a message id or correlid. It does not get converted to a different code page. There are fields
- MQBACF_ (Byte Parameter Types ) and are in the range 7001 to 7039.
- A character string(MQCFST). This is a string such as queue name. This has a code page associated with it. This allows you to convert from ASCII to EBCDIC (or other conversion). There are fields
- MQCA_ (Character Attribute Selectors) and are in the range 2001 to 2999 range (and 4000) for queue manager values .
- MQCACF_ (Character Queue Manager Parameter Types) in the range 3000 to 3211 + a couple in the 55xx range.
- MQCACH_ (Character Channel Parameter Types) in the range 3500 to 3573
- An integer (MQCFIN), such as queue depth, or the value to mean “Enabled” or “Disabled”. There are fields
- MQIA_, (Integer Attribute Selectors) integers in the range of 1 to 275 ( and 2000).
- MQIACF_, (Integer Queue Manager Parameter Types) and are number in the range of 1 to 1440.
- MQIACH_, (Integer Channel Types) and in the range 1501 to 1646.
- A 64 bit integer(MQCFIN64). For example MQIAMO64_GET_BYTES, the number of bytes got as a integer 64 number. The values are intermixed with MQIAMO_ numbers. These are only used by the monitoring code – there are no MQIA*64 for anything else
- A byte string (MQCFBS) . This is a string which can include hexadecimal characters, such as a message id or correlid. It does not get converted to a different code page. There are fields
- Lists of fields
- 64 bit integers(MQCFIL64). These are used in the midrange monitoring
- integers(MQCFIL)
- character strings(MQCFSL)
- A group of (sub) parameters (MQCFGRP).
- For example with application activity. This has a several groups. One of these groups is “Operations” which contains a set of PCF data for Operation type, Operation time, QMGR name).
- They are also used in command events – but not used in command server messages.
- These have a type field of MQGACF_ and are in the range 8001 to 9000
- Filters – which provides the same function as the “WHERE” clause on a command.
- Passing in a byte string (MQCFBF),
- Passing a string (MQCFSF)
- Passing in an integer string(MQCFIF).
Using PCF
It takes a while to understand how to use PCF. I found I had to write a lot of code to handle the PCF data and covert from numeric values to useful character strings so as to produce useful output. It does all work – it is just hard to get it to work.
To make it just a touch more complex, you may need to worry about data conversion and strings having different length when doing conversion, but I’ll leave this “as an exercise for the reader”1.
(1)Quote: The phrase is sometimes used as a joke. When a writer gets to the difficult part of a problem and he doesn’t want to take the trouble to solve it, or doesn’t know how to solve, he says this is “left as an exercise for the reader”, as if it was too trivial to waste space in the book discussing, when really the issue is that he can’t figure out the answer himself.
What was new in the MQ API?
I wanted to know what new features were available in different releases of MQ, but could not find the information documented.
I’ve taken the CMQC C header file from different releases on z/OS and shown the differences!
From MQ V8.0 to V9.0.1
CMQC
MQCNO_CURRENT_VERSION 5MQCNO_VERSION_6 6 MQCNO_CURRENT_VERSION 6MQCNO_CURRENT_LENGTH 188MQCNO_LENGTH_6 208 MQCNO_CURRENT_LENGTH 208 MQAT_MCAST_PUBLISH 36 MQAT_AMQP 37 MQAT_DEFAULT 2 MQOT_PROT_POLICY 1019 MQOT_TT_CHANNEL 1020 MQOT_AMQP_CHANNEL 1021 MQOT_AUTH_REC 1022 MQ_AMQP_CLIENT_ID_LENGTH 256 MQRC_STORAGE_MEDIUM_FULL 2192 MQRC_ADMIN_TOPIC_STRING_ERROR 2598 MQRC_AMQP_NOT_AVAILABLE 2599 MQRC_CCDT_URL_ERROR 2600 MQADOPT_CHECK_CHANNEL_NAME 8MQCMDL_CURRENT_LEVEL 800MQCMDL_LEVEL_801 801 MQCMDL_LEVEL_802 802 MQCMDL_LEVEL_900 900 MQCMDL_LEVEL_901 901 MQCMDL_LEVEL_902 902 MQCMDL_LEVEL_903 903 MQCMDL_LEVEL_904 904 MQCMDL_LEVEL_905 905 MQCMDL_CURRENT_LEVEL 905 MQPL_APPLIANCE 28 MQCAP_EXPIRED 2 /* Media Image Scheduling */ MQMEDIMGSCHED_MANUAL 0 MQMEDIMGSCHED_AUTO 1 /* Automatic Media Image Interval */ MQMEDIMGINTVL_OFF 0 /* Automatic Media Image Log Length */ MQMEDIMGLOGLN_OFF 0 /* Media Image Recoverability */ MQIMGRCOV_NO 0 MQIMGRCOV_YES 1 MQIMGRCOV_AS_Q_MGR 2 MQCA_AMQP_SSL_CIPHER_SUITES 2137 MQCA_AMQP_VERSION 2136MQCA_LAST_USED 2135MQCA_LAST_USED 2137 MQIA_ADVANCED_CAPABILITY 273 MQIA_AMQP_CAPABILITY 265 MQIA_AUTHENTICATION_METHOD 266 MQIA_KEY_REUSE_COUNT 267MQIA_LAST_USED 264MQIA_LAST_USED 273 /* Key reuse count */ MQKEY_REUSE_DISABLED 0 MQKEY_REUSE_UNLIMITED (-1) struct tagMQCNO { ... PMQCHAR CCDTUrlPtr; /* Address of CCDT URL string */ MQLONG CCDTUrlOffset; /* Offset of CCDT URL string */ MQLONG CCDTUrlLength; /* Length of CCDT URL */ MQBYTE8 Reserved; /* Reserved */ /* Ver:6 */ } ... MQCNO_DEFAULT ....
CMQCFC
MQCMD_INQUIRE_AMQP_CAPABILITY 216 MQCMD_AMQP_DIAGNOSTICS 217 MQRCCF_CLWL_EXIT_NAME_ERROR 3374 MQRCCF_SERVICE_NAME_ERROR 3375 MQRCCF_REMOTE_CHL_TYPE_ERROR 3376 MQRCCF_TOPIC_RESTRICTED 3377 MQRCCF_CURRENT_LOG_EXTENT 3378 MQRCCF_LOG_EXTENT_NOT_FOUND 3379 MQRCCF_LOG_NOT_REDUCED 3380 MQRCCF_LOG_EXTENT_ERROR 3381 MQRCCF_ACCESS_BLOCKED 3382MQ_ENTITY_NAME_LENGTH 64MQ_ENTITY_NAME_LENGTH 1024 MQIAMO_MONITOR_CLASS 839 MQIAMO_MONITOR_TYPE 840 MQIAMO_MONITOR_ELEMENT 841 MQIAMO_MONITOR_DATATYPE 842 MQIAMO_MONITOR_FLAGS 843 MQIAMO64_QMGR_OP_DURATION 844 MQIAMO64_MONITOR_INTERVAL 845 MQIAMO_LAST_USED 845 /* Defined values for MQIAMO_MONITOR_FLAGS */ MQIAMO_MONITOR_FLAGS_NONE 0 MQIAMO_MONITOR_FLAGS_OBJNAME 1 /* Defined values for MQIAMO_MONITOR_DATATYPE */ MQIAMO_MONITOR_UNIT 1 MQIAMO_MONITOR_DELTA 2 MQIAMO_MONITOR_HUNDREDTHS 100 MQIAMO_MONITOR_KB 1024 MQIAMO_MONITOR_PERCENT 10000 MQIAMO_MONITOR_MICROSEC 1000000 MQIAMO_MONITOR_MB 1048576 MQIAMO_MONITOR_GB 100000000 MQIACF_AMQP_ATTRS 1401 MQIACF_AMQP_DIAGNOSTICS_TYPE 1406MQIACF_SYSP_MAX_CONC_OFFLOADS 1412 MQIACF_LAST_USED 1412MQIACF_AUTH_REC_TYPE 1412 MQIACF_SYSP_MAX_CONC_OFFLOADS 1413 MQIACF_SYSP_ZHYPERWRITE 1414 MQIACF_Q_MGR_STATUS_LOG 1415 MQIACF_ARCHIVE_LOG_SIZE 1416 MQIACF_MEDIA_LOG_SIZE 1417 MQIACF_RESTART_LOG_SIZE 1418 MQIACF_REUSABLE_LOG_SIZE 1419 MQIACF_LOG_IN_USE 1420 MQIACF_LOG_UTILIZATION 1421 MQIACF_LOG_REDUCTION 1422 MQIACF_LAST_USED 1422MQIACH_LAST_USED 1643MQIACH_AMQP_KEEP_ALIVE 1644 MQIACH_SECURITY_PROTOCOL 1645 MQIACH_LAST_USED 1645MQCAMO_LAST_USED 2712MQCAMO_MONITOR_CLASS 2713 MQCAMO_MONITOR_TYPE 2714 MQCAMO_MONITOR_DESC 2715 MQCAMO_LAST_USED 2715MQCACF_LAST_USED 3206MQCACF_AMQP_CLIENT_ID 3207 MQCACF_ARCHIVE_LOG_EXTENT_NAME 3208 MQCACF_LAST_USED 3208MQCACH_LAST_USED 3570MQCACH_TOPIC_ROOT 3571 MQCACH_LAST_USED 3571MQGACF_LAST_USED 8014MQGACF_MONITOR_CLASS 8015 MQGACF_MONITOR_TYPE 8016 MQGACF_MONITOR_ELEMENT 8017 MQGACF_LAST_USED 8017 - - - - - - - - - - - - - - - MQACT_REDUCE_LOG 10 MQACT_ARCHIVE_LOG 11 MQEVO_REST 8 MQRQ_FAILOVER_PERMITTED 30 MQRQ_FAILOVER_NOT_PERMITTED 31 MQRQ_STANDBY_ACTIVATED 32 MQLDAP_AUTHORMD_SRCHGRPSN 3 /* Authentication Method */ MQAUTHENTICATE_OS 0 MQAUTHENTICATE_PAM 1 /* Reduce Log Options */ MQLR_ONE 1 MQLR_AUTO (-1) MQLR_MAX (-2)
From MQ 9.0.1 to MQ 9.1.1
CMQC
MQCNO_CURRENT_VERSION 6 MQCNO_VERSION_7 7 MQCNO_CURRENT_VERSION 7 MQCNO_CURRENT_LENGTH 208 MQCNO_LENGTH_7 240 MQCNO_CURRENT_LENGTH 240 MQCNO_GENERATE_CONN_TAG 0x00200000 MQAN_NONE MQAN_NONE_ARRAY MQRC_INCOMPLETE_TRANSACTION 2147 MQRC_Q_MGR_RECONNECT_REQUESTED 2601 MQRC_SUB_JOIN_NOT_ALTERABLE 29440 /* Max queue file size values */ MQQFS_DEFAULT (-1)MQCMDL_CURRENT_LEVEL 905MQCMDL_LEVEL_910 910 MQCMDL_LEVEL_911 911 MQCMDL_LEVEL_912 912 MQCMDL_LEVEL_913 913 MQCMDL_LEVEL_914 914 MQCMDL_LEVEL_915 915 MQCMDL_CURRENT_LEVEL 915MQIA_LAST_USED 273MQIA_LAST_USED 274 MQIA_MAX_Q_FILE_SIZE 274 struct tagMQCNO { ... MQCHAR28 ApplName; /* Application name */ MQBYTE4 Reserved2; /* Reserved */ /* Ver:7 */ }; MQCNO_DEFAULT....
CMQCFC
MQCMD_AMQP_DIAGNOSTICS 217 MQCMD_INTER_Q_MGR_STATUS 218 MQCMD_INTER_Q_MGR_BALANCE 219 MQCMD_INQUIRE_APPL_STATUS 220 MQRCCF_PS_REQUIRED_MQUC 3383 MQRCCF_OBJECT_ALREADY_EXISTS 4001 MQRCCF_APPL_STATUS_NOT_FOUND 4097 MQCFT_STATUS 27MQBACF_LAST_USED 7035MQBACF_REQUEST_ID 7036 MQBACF_PROPERTIES_DATA 7037 MQBACF_CONN_TAG 7038 MQBACF_LAST_USED 7038MQIACF_LAST_USED 1422MQIACF_IGNORE_STATE 1423 MQIACF_MOVABLE_APPL_COUNT 1424 MQIACF_APPL_INFO_ATTRS 1425 MQIACF_APPL_MOVABLE 1426 MQIACF_REMOTE_QMGR_ACTIVE 1427 MQIACF_APPL_INFO_TYPE 1428 MQIACF_APPL_INFO_APPL 1429 MQIACF_APPL_INFO_QMGR 1430 MQIACF_APPL_INFO_LOCAL 1431 MQIACF_APPL_IMMOVABLE_COUNT 1432 MQIACF_BALANCED 1433 MQIACF_BALSTATE 1434 MQIACF_APPL_IMMOVABLE_REASON 1435 MQIACF_DS_ENCRYPTED 1436 MQIACF_CUR_Q_FILE_SIZE 1437 MQIACF_CUR_MAX_FILE_SIZE 1438 MQIACF_LAST_USED 1438MQIACH_LAST_USED 1645MQIACH_SPL_PROTECTION 1646 MQIACH_LAST_USED 1646 MQCACF_LAST_USED 3208MQCACF_APPL_IMMOVABLE_DATE 3209 MQCACF_APPL_IMMOVABLE_TIME 3210 MQCACF_LAST_USED 3210MQGACF_LAST_USED 8017MQGACF_APPL_STATUS 8018 MQGACF_CHANGED_APPLS 8019 MQGACF_ALL_APPLS 8020 MQGACF_APPL_BALANCE 8021 MQGACF_LAST_USED 8021 MQIS_NO 0 MQIS_YES 1 /* Movable Options */ MQAPPL_IMMOVABLE 0 MQAPPL_MOVABLE 1 /* Active Options */ MQACTIVE_NO 0 MQACTIVE_YES 1 /* Balance Options */ MQBALANCED_NO 0 MQBALANCED_YES 1 MQBALANCED_NOT_APPLICABLE 2 MQBALANCED_UNKNOWN 3 /* Balance State */ MQBALSTATE_NOT_APPLICABLE 0 MQBALSTATE_LOW 1 MQBALSTATE_OK 2 MQBALSTATE_HIGH 3 MQBALSTATE_UNKNOWN 4 /* Immovable Reasons */ MQIMMREASON_NONE 0 MQIMMREASON_NOT_CLIENT 1 MQIMMREASON_NOT_RECONNECTABLE 2 MQIMMREASON_MOVING 3 MQIMMREASON_APPLNAME_CHANGED 4
From MQ 9.1.1 to MQ 9.2.2
CMQC
MQ_NHA_INSTANCE_NAME_LENGTH 48MQCMDL_CURRENT_LEVEL 915MQCMDL_LEVEL_920 920 MQCMDL_LEVEL_921 921 MQCMDL_LEVEL_922 922 MQCMDL_CURRENT_LEVEL 922
CMQCF
MQCACF_APPL_IMMOVABLE_TIME 3210 MQCACF_LAST_USED 3210MQCACF_NHA_INSTANCE_NAME 3211 MQCACF_LAST_USED 3211 MQRQ_STANDBY_ACTIVATED 32 MQRQ_REPLICA_ACTIVATED 33
From MQ 9.2.2. to 9.2.4
CMQC
MQBNO structure and MQBNO_ constantsMQCNO_CURRENT_VERSION 7MQCNO_VERSION_8 8 MQCNO_CURRENT_VERSION 8 MQCNO_CURRENT_LENGTH 240 MQCNO_LENGTH_8 252 MQCNO_CURRENT_LENGTH 252 MQ_TEMPORARY_Q_PREFIX_LENGTH 32 MQRC_BNO_ERROR 2602 MQRC_OUTBOUND_SNI_NOT_VALID 2603 /* Streaming Queue Quality of Service Values */ MQST_BEST_EFFORT 0 MQST_MUST_DUP 1MQCMDL_CURRENT_LEVEL 922MQCMDL_LEVEL_923 923 MQCMDL_LEVEL_924 924 MQCMDL_CURRENT_LEVEL 924 MQCA_LAST_USED 2137MQCA_LAST_USED 2138MQIA_LAST_USED 274MQIA_LAST_USED 275 MQIA_STREAM_QUEUE_QOS 275 struct tagMQCNO { ... PMQBNO BalanceParmsPtr; /* Balance Parameter Pointer */ MQLONG BalanceParmsOffset; /* Balance Parameter Offset */ MQBYTE4 Reserved3; /* Reserved */ /* Ver:8 */ };
CMQCFC
MQRCCF_STREAMQ_DEST_NOT_SUPP 3384 MQRCCF_STREAMQ_DEST_CONFLICT 3385 MQRCCF_STREAMQ_NOT_SUPPORTED 3386 MQRCCF_STREAMQ_CONFLICT 3387MQBACF_LAST_USED 7038MQBACF_MQBNO_STRUCT 7039 MQBACF_LAST_USED 7039 MQIACF_SYSP_SMF_STAT_TIME_MINS 1199 MQIACF_SYSP_TRACE_CLASS 1200MQIACF_LAST_USED 1438MQIACF_BALANCING_TYPE 1439 MQIACF_BALANCING_OPTIONS 1440 MQIACF_BALANCING_TIMEOUT 1441 MQIACF_SYSP_SMF_STAT_TIME_SECS 1442 MQIACF_SYSP_SMF_ACCT_TIME_MINS 1443 MQIACF_SYSP_SMF_ACCT_TIME_SECS 1444 MQIACF_LAST_USED 1444 MQIACH_AUTH_INFO_TYPES 1622MQCACH_LAST_USED 3571MQCACH_TEMPORARY_MODEL_Q 3572 MQCACH_TEMPORARY_Q_PREFIX 3573 MQCACH_LAST_USED 3573 MQIMMREASON_APPLNAME_CHANGED 4 MQIMMREASON_IN_TRANSACTION 5 MQIMMREASON_AWAITS_REPLY 6
Where’s my publish?
Like many things I do, it all started with an easy problem. I had an application which subscribed to a topic, it published on a topic string, then got the published message. Another application subscribed on the same topic string but did not get the message. I’ll leave clues for the experts to spot as I explain the progress I made.
The first complication was the subscribing and publishing application was in Python on z/OS (which runs in ASCII), and the other application was written in C using the default EBCDIC. I had to spend time working out how to convert from ASCII to EBCDIC. (When using CHARV you can specify a string and its code page and have MQ do it for you automatically – easy once you know which code page to use.) For other strings I had to convert manually using the A2E function from C run time.
There are some commands for displaying status of pub sub, it took a little while to understand them and give me the data I wanted.
For example
My application did a subscription with topic string “/currency/rate/EUR/GBP”.
The command gave
DIS TPSTATUS(‘/currency/rate/EUR/GBP’)
CSQM297I … CSQMDRTC NO TPSTATUS FOUND MATCHING REQUEST CRITERIA
Once I had opened the topic using the topic string, I got
CSQM201I … CSQMDRTC DIS TPSTATUS DETAILS
TPSTATUS(/currency/rate/EUR/GBP)
TYPE(TOPIC)
DEFPRESP(SYNC)
PUB(ENABLED)
SUB(ENABLED)
ADMIN()
RETAINED(NO)
PUBCOUNT(0)
SUBCOUNT(1)
PUBSCOPE(ALL)
SUBSCOPE(ALL)
…
I set up the subscription using Python
topic_string = ‘/currency/rate/EUR/GBP’
sub_desc = pymqi.SD() # create the basic subscription
sub_desc[‘Options’] = pymqi.CMQC.MQSO_CREATE + pymqi.CMQC.MQSO_RESUME +
pymqi.CMQC.MQSO_DURABLE + pymqi.CMQC.MQSO_MANAGED
sub_desc.set_vs(‘SubName’, “MYSUB” )
sub_desc.set_vs(‘ObjectString’, topic_string)
The code to get the subscribed message was
get_opts = pymqi.GMO(
MsgHandle=hMsg.msg_handle,
Options=pymqi.CMQC.MQGMO_NO_SYNCPOINT +
pymqi.CMQC.MQGMO_CONVERT +
pymqi.CMQC.MQGMO_PROPERTIES_IN_HANDLE +
pymqi.CMQC.MQGMO_FAIL_IF_QUIESCING +
pymqi.CMQC.MQGMO_WAIT)
get_opts[‘WaitInterval’] = 1000
data = sub.get(None, pymqi.md(), get_opts)
print(‘Received data: [%s]’ % data)
The code to publish to a topic was
msg = ‘1.3961’
topic = pymqi.Topic(qmgr, topic_string=topic_string)
topic.open(open_opts=pymqi.CMQC.MQOO_OUTPUT)
topic.close()
The Python program was subscribe, publish to topic, get the message.
When I ran the code, no messages were published. I tried changing the topic_string to ‘/currency/rate/EUR/USD’, that had no effect. Eventually I tracked down a code page problem in my code; I fixed that – but the program still did not publish a message.
The more knowledgable people reading this may have spotted one problem.
The first time the program ran, the subscription code created a SUB(MYSUB) with the given topic string (‘/currency/rate/EUR/USD’). When I changed the topic string to ‘/currency/rate/EUR/GBP’, and published a message, the subscription was using the original topic string with USD!, but publishing to GBP. Any changes to the SUBscription are ignored unless MQSO_ALTER is specified.
I deleted the SUB and reran my application. Displaying the subscription (MYSUB) gave
CSQM201I %CSQ9 CSQMDRTC DIS SUB DETAILS 934
SUB(MYSUB)
SUBID(C3E2D8D4C3E2D8F94040404040404040DAFEED99504CE540)
DURABLE(YES)
SUBTYPE(API)
TOPICSTR(/currency/rate/EUR/GBP)
SUBLEVEL(1)
…
I can display the topic string and see what is subscribed to it
dis tpstatus(‘/currency/rate/EUR/GBP’) type(sub)
This gave two subscriptions:
TPSTATUS(/currency/rate/EUR/GBP)
TYPE(SUB)
SUBID(C3E2D8D4C3E2D8F94040404040404040DAFE062AED26E2C0)
DURABLE(YES)
SUBTYPE(ADMIN)
SUBUSER(IBMUSER)
…
NUMMSGS(12)
and
TPSTATUS(/currency/rate/EUR/GBP)
TYPE(SUB)
SUBID(C3E2D8D4C3E2D8F94040404040404040DAFC73649B3C7200)
DURABLE(YES)
SUBTYPE(ADMIN)
SUBUSER(IBMUSER)
…
NUMMSGS(0)
So one subscription has seen 12 messages, the other subscription has seen 0 messages. This does not look like a wrong topic string, as they are reported as part of the display tpstatus.
I could not find a way of displaying all the information about a subscription in one command. I had to use DIS TPSTATUS.. TYPE(SUB), then use cut and paste to issue the DIS SUB SUBID(….). (It would be much easier for the poor end users if there was an ALL option to display all this data in one go.)
For example
SUB(COLIN)
SUBID(C3E2D8D4C3E2D8F94040404040404040DAFC73649B3C7200)
DURABLE(YES)
SUBTYPE(ADMIN)
DISTYPE(RESOLVED)
TOPICSTR(/currency/rate/EUR/GBP)
DEST(CP0000)
DESTQMGR()
DESTCORL(C3E2D8D4C3E2D8F94040404040404040DAFC73649B3C7200)
TOPICOBJ()
PUBACCT(00000000000000000000000000000000000000000000000000000000…)
USERDATA(CZZZ)
SUBUSER(IBMUSER)
SUBLEVEL(1)
…
So we can see from this subscription the topic string being subscribed on, and the queue name being used.
This subscription had no messages published to it; see the DISPLAY TPSTATUS TYPE(SUB) field NUMMSGS. Those of you with a PhD in PubSub may have spotted the problem; it is very well hidden. It is all to do with PubLevel and SubLevel. (The documentation is very confusing, it implies a subscription can have more than one sublevel).
I think of it as logic within publish which says for each subscription
If SubLevel <= PubLevel then publish the message else skip the publish
- The subscription from the python code had SubLevel = 0. (This was latent bug – it should have been set to 1!
- The SubLevel from my DEFINE SUB command was the default 1.
- The PubLevel from the Python code was 0 – this was the problem.
So the publish to my manual DEFINE SUB was skipped, and the publish to the Python code worked because PubLevel = 0 matched SubLevel = 0
The message is being published – but it is not still not there.
When I restarted my queue manager and ran the test, the Python application got no message found, which was a surprise to me. When I reran it, with debug code enabled – the get worked, and carried on working all day. Next day the same problem occurred. Very strange – is the debug code affecting the messages ?
I then remembered the default syncpoint behaviour (one of those face palm moments). On midrange the default is to put out out sync-point, so the messages are immediately available. On z/OS the default is to put within sync-point. The messages are only visible after a commit, or the implicit commit at MQDISC. On z/OS the messages were in syncpoint, and there was no commit. When I ran my program the second time, it got the message from the first run, and so on!
I put the commit in – and it all worked.
Check list of what to check
Check you are subscribing to the correct topic string.
Plan carefully how you define your topic(strings).
If you want to have an application define a subscription, consider some defensive programming.
- MQSO_CREATE will create a subscription based on the specified data. If it already exists report an error.
- MQSO_CREATE + MQSO_RESUME will create a subscription based on the specified data. If it already exists it will not report an error. A second usage will not change the subscription.
- MQSO_CREATE + MQSO_ALTER will create a subscription based on the specified data. A second usage will update the subscription. This could be considered dangerous if you have multiple applications running concurrently changing the topic string or other parameters.
- MQSO_RESUME does not create a subscription, not updates it. it uses the information in the subscription object.
It would be safer for an Administrator to define the MQSUB, and the applications to refer to it. Do not allow application users to be able to define a subscription. They should use the pre-defined MQSUB and add the topic string to give more granular data. For example I could use a null object and have a topic string of /TOP/SECRET/DATA/SPIESIN/ENGLAND. If you haven’t set up security properly then I could access the data.
Check the PubLevel and the SubLevel.
If you are not using this functionality set PubLevel to 9 and SubLevel to 1 (the defaults).
You can DISPLAY SUB(….) SUBLEVEL to see the specified value.
The PubLevel is specified in the MQPMO.PubLevel. There is no display command for this.
Check the sync-point options.
You should always, always, always specify MQPMO_NO_SYNCPOINT, or MQPMO_SYNCPOINT and the commit. Do not let it default in case someone ports your code to or from z/OS. You can waste days hunting down this problem.
Using LDAP with MQ multiplatform and nested groups.
This blog post follows on from Using LDAP with multi platform.
Nested groups can be used to simplify administration
It is good practice to grant authority to access resources using groups, rather than giving access to individual userids. For example if an application uses 10 queues, and a new person joins the team you can either connect the new id to one group, or give the id access to the 10 queues.
Using nested groups takes this further. Imagine the payroll queues are managed by the GRPAYROLL access group, the HR queues managed by the GRHR access group, and the Finance queue managed by the GRFINANCE access group. The MQ system programmers can manage any queue. You could give GRMQADMIN access to the Payroll queues, HR queues, and Finance queues. Or you say the GRMQADMIN is the super set and includes a reference to GRPAYROLL, GRHR, and GRFINANCE. If you define a HR queue, you just give GRHR access to it, and the MQ Administration team get access to it “for free”.
You can take this further, and have a group GRHR_GR1, and GRHR_GR2 which give access to a subset of HR queues. The GRHR group could include both of these.
The model of this is a tree where a group high up in the tree incorporates the groups lower down the tree.
There are two ways of defining groups in LDAP.
1. Define a group record, and list the members of the group. This is called a static group
dn:cn=GR1,OU=groups,o=myorg
member:cn=user1,ou=users,o=myorg
member:cn=user2,ou=users,o=myorg
To retrieve the groups for a user, you issue a query for all groups which have the given member data.
2. Specify the group name as part of the user’s record, this is known as a dynamic group.
dn=cn=user1,ou=users,o=myorg
group=cn=group1,ou=groups,o=myorg
group=cn=group2,ou=groups,o=myorg
For this you retrieve the “group” attributes from the record. Note: The group attribute does not exist in LDAP, you have to use a different one ( I used the st attribute).
Nested group support.
There is an MQ AUTHINFO parameter NESTGRP. The documentation says
- NESTGRP Group nesting.
- YES The group list is searched recursively to enumerate all the groups to which a user belongs.
Static groups and nested group support.
My MQ ADUTHINFO had
AUTHORMD(SEARCHGRP) +
BASEDNG(‘ou=groups,o=your Company’) +
CLASSGRP(‘groupOfNames’) +
GRPFIELD(sn) +
FINDGRP(‘member’) +
NESTGRP(yes) +
On my system the userid ibmuser is a “member” of two groups mqstatic and mqstatic2
dn: cn=mqstatic,ou=groups,o=your Company
objectclass: groupOfNames
cn: mqstatic
ou:groups.
member: cn=ibmuser,o=your Company
member: cn=adcdb,o=your Company
and
dn: cn=mqstatic2,ou=groups,o=your Company
objectclass: groupOfNames
cn: mqstatic2
ou:groups.
member: cn=ibmuser,o=your Company
There is another group, which refers to the mqstatic group.
dn: cn=mega,ou=groups,o=your Company
objectclass: groupOfNames
cn: mega
ou:groups.
member: cn=mqstatic,ou=groups,o=your Company
member: cn=gg,ou=groups,o=your Company
As part of the userid to group mapping, MQ issues the query What groups have member: cn=ibmuser,o=your Company. The response is
- dn: cn=mqstatic2,ou=groups,o=your Company
- dn: cn=mqstatic,ou=groups,o=your Company
When NESTGRP(YES) is specified MQ then queries all groups that have cn=mqstatic2,ou=groups,o=your Company, and does another query for dn: cn=mqstatic,ou=groups,o=your Company
and gets the response dn: cn=mega,ou=groups,o=your Company.
The cn=ibmuser is associated with the three groups, and those three groups are used for access checking.
For a data model I see it as
dn=cn=GRHR_GR1
isusedby: CN=GRHR
and
dn:cn=GRHR
isusedby: CN=MQADMIN
and
dn: cn=mqadmin
member:cn=user1
The “isusedby” attribute name, is in reality “member”, which I feel is very confusing, as member implies membership, and there is no membership involved. This is an LDAP naming problem – not MQ’s.
This model feels upside down (or inside out) (or both). It takes several cups of tea with biscuits to draw up the groups and the connections definitions you need. Put the kettle on, and try defining the groups for the MQADMIN, GRFINANCE, and GRFINANCEGR1, GRFINANCEGR2 groups; three people in MQADMIN, and two people in finance; then add a definition for GRFINANCEGRN.
Using dynamic groups and nested group support
This feels more the right way up – but it is upside down compared to static groups.
My authinfo had
AUTHORMD(SEARCHUSR) +
FINDGRP(‘st’) +
BASEDNU(‘o=Your Company’) +
CLASSGRP(‘person’) +
…
With my userid I had
dn: cn=user1, o=Your Company
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: ibm-nativeAuthentication
objectclass: inetOrgPerson
departmentNumber: mqadmin
cn: ibmuser
sn: ibmuser
ou: test
st: cn=group,ou=groups,o=your Company
st: cn=mqadmin,ou=groups,o=your Company
ibm-nativeId: ibmuser
Where st: defined the groups.
For the entry for cn=group, I defined a nested group called cn=deepergroup.
dn: cn=group,ou=groups,o=your Company
objectclass: top
objectclass: person
objectclass: organizationalPerson
cn: cpgroup
ou: groups
st : cn=deepergroup,o=your Company
sn : groupgroup
You need an entry for cn=deepergroup… in LDAP
dn: cn=deepergroupou=groups,o=your Company
objectclass: person
cn: mega
sn:mega# no st attribute, so there is no more nesting
I define the cn=mqadmin group to LDAP with no “sn” entry, because it has no nesting.
MQ issues a query to LDAP asking for the group definitions ( st in my case) for the given user. For each group that gets returned, query that group for any more group (st in my case) definitions, etc.
Together these give three groups for the cn=ibmuser:
- cn=mqadmin,ou=groups,o=your Company
- cn=deepergroup,ou=groups,o=your Company
- cn=group,ou=groups,o=your Company
Dynamic group data model
The data model for dynamic group feels more natural, where the “is in” and “includes” are the attribute you chose, “st” in my case.
- dn=cn=user1
- is in: cn=group1…
- dn=cn=group1…
- includes: cn=groupdeep
Enabling nestgrp(yes) with dynamic groups may throw errors
If you start with
dn=cn=user1,…
group=cn=group1,..
group=cn=group2,…
With nestgrp(no) this defines the userid is in two groups.
If you enable nestgrp(yes) then when the user id is used, MQ checks there are entries in LDAP for group=cn=group1,.. and group=cn=group2,… If the definitions do not exist, you get the MQ message:
AMQ5532E: Error authorizing entity in LDAP
EXPLANATION:
The LDAP authorization service has failed in the ldap_first_entry call while trying to find user or group ‘NULL’. Returned count is 0. Additional context is ‘cn=group1,… ‘.
To solve the problem I defined the two groups to LDAP.
For group1 I defined
dn:cn=group1,…
group:cn=deepergroup…
and I defined the deepergroup group (to keep mq happy) with no group entries (to indicate end of the nesting).
dn:cn=deepergroup,…