Tag: MQ

My cluster channel definition worked – why?

This question came up during some education on clustering.

The problem arose because she had typed in the wrong name for the channel, yet clustering worked – she could send a message to a remote clustered queue. Most people who have created sender/receiver channel pairs have experienced the frustration of the channel not starting because the name was not quite the same – Ohs and zeros, different punctuation marks etc. So it was a surprise that when she made a mistake it worked!

Let me take a simple example. We have three queue managers in a cluster:

  • QM_REPOS is the full repository. It has a CLUSRCVR Channel called TO_REPOS, and a conname of REPOS(1414).
  • A queue manager called QMA. This has
    • a cluster sender channel called TO_REPOS, and a conname of REPOS(1414) which matches QM_REPOS
    • a cluster receiver channel called TO_QMA, conname(SYSA(1414))
  • A queue manager QMB. This has
    • a cluster queue called TEST
    • a cluster sender channel called TO_REPOS, and a conname of REPOS(1414) which matches QM_REPOS
    • a cluster receiver channel called BAD_SPELLING conname(SYSB(1414))

When QMB joined the cluster it sent information about the queue manager to the cluster repository. The repository then stores the data in the form of a logical table

QNAME=TEST, use_channel_name = BAD_SPELLING conname(SYSB(1414))

When QMA wants to use the TEST queue, it finds it does not have the information, and so asks the repository, which sends down all the data for QNAME=TEST. QMA then caches this data locally. QMA then picks an entry (from the list of one entry), and dynamically creates a cluster sender channel called BAD_SPELLING with conname(SYSB(1414)). Because BAD_SPELLING is the same as what QMB sent to the repository it matches – and so works! Pretty amazing eh!

How can I tell what Ive got?

In this excellent article by David Ware of IBM, it talks about cluster channel. Ive copied some of it below.

Using the DISPLAY CLUSQMGR command. Each CLUSQMGR entry represents how the local queue manager sees the other queue managers in each of the clusters it is a member of. You get a separate entry per cluster irrespective of whether namelists have been used on the cluster channels or not. The entry contains two particularly useful attributes for this discussion, QMTYPE and DEFTYPE.

QMTYPE simply shows if the other queue manager is a full repository (‘REPOS’) for the cluster or a partial repository (‘NORMAL’).

DEFTYPE shows you how the relationship between the queue managers has been established, based on what cluster channels have been defined. DEFTYPE has a number of rather cryptic values, CLUSSDR, CLUSSDRA, CLUSSDRB and CLUSRCVR. I’ll summarize them here:

DEFTYPE values

CLUSRCVR: This is the entry for the local queue manager in each cluster it has a cluster receiver channel defined.

Any CLUSSDR* value means this entry represents a remote queue manager in a cluster. The different values however help you understand how the local queue manager came to know about it:

CLUSSDRA: This is a remote cluster queue manager that the local queue manager has no manually defined cluster sender channel for it, but has been told about it by someone else, either by the remote queue manager itself (typically because this queue manager is a full repository) or because a full repository has told this queue manager about it as it needs to communicate with it for some reason.

CLUSSDRB: This means the local queue manager has a manually defined cluster sender channel which has been used to establish contact with the target queue manager and that queue manager has accepted it from the point of view of the cluster. The target could be a full or a partial repository, although as I’ve already said you really only want it to be a full repository at the other end.

CLUSSDR: This means the local queue manager has manually defined a cluster sender channel to the remote queue manager but the initial cluster handshake between them has not yet completed. This may be because the channel has never started, perhaps because the target is not running or the configuration is incorrect. It could also mean the channel has successfully started but the target queue manager did not like the cluster information provided, for example a cluster name was set in the cluster sender channel definition that does not match the target’s cluster membership. Once the handshake has been performed, the DEFTYPE should change to CLUSSDRB, so in a healthy system CLUSSDR should only be a transitory state.


Many MQ admin only do half a job

I think I can do many jobs very quickly – such as put the rubbish out. My wife says I am terrible at doing jobs because I am not a completer finisher. The task is “put the bag of rubbish out, and put a new bag in the bin’. I typically only do the first part and go and do something else.

As an MQ administrator it is easy to only do half a task. For example, define a sender channel. “It is so easy, I could do it in my sleep” – or so you think. There are the following steps

  1. Type in the define channel command
  2. Display the channel and check it to make sure it is as you expected. For example sometimes data has to be in quotes MCA(‘mqm’). MCA(mqm) without quotes is converted to upper case MQM which is a different userid.
  3. You need to define a transmission queue. Many people stop here.
  4. You need to start the channel and make sure it starts successfully, and you haven’t specified the character O rather than the number 0 and other typing problems.
  5. Set up the triggering rule, so a message on the XMITQ causes the channel to start.
  6. Cause a message to be put to the xmit queue, and test to make sure the channel is triggered.
  7. Set up events such as queue high, so if there are messages over a certain age, or the queue depth is over 100 then product an alert for automation.
  8. Update the monitor tools to include this channel.
  9. Update your reports, so you collect and report on message/hour or day on this channel.
  10. Update your documentation.

I expect most MQ Admins would get 2 out of 10 – could do better!

This is why check lists are very useful.

Cleaning up after your application programmers.

Some people are Application Pros. This is not a compliment – it is short for half an Application Programmer, because they only develop half an application.

Symptoms of this are the application queue depths are high (100 may be high), the queue never becomes empty, or a large msgage for the queue.

Consider an application which sends a request to a backend and expects a response. Normally the response comes back in time, the application gets it, and the reply is deleted.
If there is a delay, the application gets a no-message found (reason 2033), and returns. The reply message arrives a short while later.

  • If the message is non persistent, it should have an expiry interval and will get removed automatically. No problem.
  • If the message is persistent with an expiry interval. This is not good practice. It should either be non persistent with expiry or persistent without expiry.
  • If the message does not have an expiry interval. You need a process to do things with the messages. Select messages older than a certain time, so you do not process the ones which are being processed. The action may be to create a compensating request, or to update a table to remove a “pending” flag in a record, and then to remove the message from the queue.

As a systems programmer or MQ administrator you do not just want to clear the queue – as it is not your data (in case you delete a queue with your pay cheque in it – or other important information). The application owner should have the responsibility for processing the data.

If you have the “Application Pro” you need to help them, by giving them a process to offload old messages.

MQ V8 provides a program dmpmqmsg which unloads (or loads) mq messages from a queue.

MQGEM has a similar product. See here. It has a superset of functions – for example dump all queues with a specified prefix, and create output file names with today’s date in it etc. Have a look – is is worth it!

You could use the IBM version as follows

  • dspmqmsg -m queue manager name
  • -I input queue name (upper case -I for get delete, lower case -i for get browse)
  • -f file name. This should be something like qm_name.Queue_name.Date_time
  • -T1:2:3 select messages older than 1 Day, 2 Hours, 3 Minutes

You need to protect the output file, to restrict who can view it. By default the permissions are -rw-r–r–.

You could encrypt it with

  • openssl enc -AES-256-CBC -in infile -out outfile.enc -pass pass:longpassword
  • rm infile

Zip with a password is better than nothing – but it is relatively insecure.

Changing permissions to “only owner has read” – still allows super users to read the file.

If any messages have been offloaded – you need to raise an alert against the development team a) to tell them about the problem and b) to remind them to get a proper process in place.

How big is your estate?

I was sent a question from someone who had read my blog, and this reminded me of a conversation I had with a fellow traveler whilst on holiday.

The lady spoke about her housing problem. At first I thought she meant, the roof leaked, or there were 10 people per room, but no, it was totally different. She had the house in Wimbledon, London; the flat in Marble Arch, London; the Manor house in the Cotswolds (the Cotswolds is a very pretty part of England, with old stone houses, and where many people ride horses for pleasure); and her husband had recently been made responsible for the shooting lodge and estate in Scotland. The housing problem was that she had been asked to look into the holiday cottages and see what could be done with them. She said that after a brief visit, she had found 9 out of the 10 properties, some were very old, some were very small. They paid tax on the houses even if they were empty. Should they knock down the worst houses and improve the remainder – for example put in better insulation and make them more energy efficient, or make two small cottages into one larger cottage.

The emailed question I had was something similar, she had been promoted and moved to a different part of the company and was now in change of MQ. She found there were no good records about the number of queue managers and what they were used for. The question was – how do you tell the size of the MQ estate – and if there was “old iron” which should be disposed of?

I heard that in California, the state said your data centre could use no more power than it uses today. So if you wanted to add a new server – you had to take out an old one, or take out an old server, and add two energy efficient servers.

What questions do you need to ask?

  • How many queue managers do you have
  • How much work are they doing?
  • Are they current?
  • Do you have licenses for them?
  • Can I get rid of any?

How many queue managers do I have?

Most enterprise customers have each machine set up with an agent which reports what software is installed on the machine, and stores the information in a central database.

This only reports on machines it knows about; if you have old machines in the dusty basement which no one knows about – they may not be in the database.

If you do not have this enterprise software running an agent, you can logon to the machine and issue

  • dspmqver to tell you level of software and the license type
  • dspmq to tell you which queue managers are configured, and if they are running

How much work are they doing

For the high level view you can use information provided by MQ

Messages on the SYSTEM.ADMIN.STATISTICS.QUEUE report information at the queue manager level such as number of puts, gets, bytes put etc. Your enterprise software may already be consuming these and may (or may not) be storing the information in a central repository. If so, you cannot get these messages.

In MQ V9 there are also statistics produced a pub/sub model which allow multiple programs to get statistics and accounting data. You subscribe to what you want – and they get published to your queue. This allows you to run in parallel with the enterprise tools.

Good metrics are

  • Number of puts per day. This includes MQPUT, MQPUT1 to a queue and publishes and puts using a topic Later on when you are looking at consolidating queue managers, you may want plots of messages put per hour. The number of gets may not be so useful, as you usually have gets which return no messages, and get with browse, both of which increase the overall number of gets.
  • Amount of data put per day.  This is useful to get a measure of message size.

Are they current?

The enterprise tools running an agent on your machines should be able to tell you the versions of software running on the machine.

As described above, you can use dspmqver

You can also issue the Display ChannelStatus command, and this will report the level of MQ at the remote end of the channel, then try to find the remote end!

Do you have licenses for them?

Your enterprise tools running the agents on your machine should collect the information, or you can use the dspmqver command.

You need to check you are running the right license, for example not using a development license in production.

Can I get rid of any?

As Shakespeare’s Hamlet says “Ay, there’s the rub”, meaning this is a difficult problem.

Some things you need to think about

  • A server may be little used during most of the year – but very heavily used at year end – be it Christmas, or financial year. You may need to monitor for a year before deciding to decommission it
  • You may want consolidate several little used queue managers on old iron onto one newer server, to reduce the number of servers, and to use newer, energy efficient hardware.
  • Migrate old releases to newer releases of software.
  • The cost of the license depends on the size of the machine. If MQ is on a large machine and doing no work, consider moving it to a smaller machine, or reduce the capacity of the virtualised server.
  • If you plan to merge servers you need to ensure the new server has enough capacity, CPU, RAM, disk bandwidth and network bandwidth, and the disk and network have acceptable response times. Assume that all workloads will peak at the same time, to ensure you have enough peak capacity.

Some gotcha’s when using stats and accounting with MQ midrange

I’ve been writing some Python code to process the messages on SYSTEM.ADMIN.STATISTICS.QUEUE and SYSTEM.ADMIN.ACCOUNTING.QUEUE queues and to create tabular data in a .csv file. I fell over some problems, and I thought I would pass on my experiences.

I used /opt/mqm/samp/bin/amqsevt -m QMA -b -q SYSTEM.ADMIN.STATISTICS.QUEUE -o json -w 1 |jq . > statfile.txt

to read the message into json, and use jq to format them better, and write the output to the file statfile.txt.

I used the MQSA Python scripts which will soon be going up on MQTools GitHub repository, to actually produce the .csv files.

From here you can see the content of the messages.

Gotchas..

Not all fields are reported.

In the documentation, some fields are marked as “Always” present the others are marked “When available”. So fields like QName are always present, but PutCount is “When available” This means you cannot just go along the fields in the records and put the first field in column 1, the second field in column 2 etc. You have to put the “qname” data in the “qname” column etc, and you may have to skip columns if you do not have the fields. You also need to set up the column headers with all the possible fields. I used Python csv.DictWriter to write the data.

Many “fields” have multiple values.

For example “puts” has number of non persistent, number of persistent, for example puts[10,4]. I found it easier to add these together and provide one field puts[14]. Other fields, such as set have 8 fields, for queues, namelist…. Again, I added these up to provide one value.

Be careful when combining fields.

Finding the maximum message size put, PutMaxBytes is easy – it is the maximum of the non persistent and persistent values. The PutMinBytes is harder. If there were no persistent messages put, then the value in PutMinBytes will be zero! You need to factor this into your calculations. You need

  • If PutCount fields are present, and putMinBytes fields are present:
  • if PutCount[0] == 0 : # we did none
  • PutMinBytes = PutMinBytes[1] # use the other value
  • else: if PutCount[1] == 0:
  • PutMinBytes = PutMinBytes[0]
  • # else there were puts of P and NP messages so look at both
  • else: PutMinBytes = Min(PutMinBytes[0],PutMinBytes[1])
  • # we did no puts
  • else: PutMinBytes = 0 # theses fields were missing, or we did no puts.

Similarly merging TimeOnQAvg values needs care.

If the fields are present, and total number of gets > 0 :

TimeOnQAvg = (GetCount[0] * TimeOnQAvg[0]

+ GetCount[1] * TimeOnQAvg[1])

/(GetCount[0]+GetCount[1])

Date and time format.

The default format is “startDate”: “2019-02-27” and “startTime”: “17.03.41”. I changed these to “startDate”: “2019/02/27″ and “startTime”: “17:03:41”

Add up the put information

At a high level, you just want to know the total number messages put, and the total number of bytes put – and not if it came from a put, put1, or a topic put. To do this you need to do calculations like

puts_total = 0
put_names = {“puts”, “put1s”, “topicPuts”, “topicPut1s”}
for each element in put_names
if element is present in the data
puts_total += value
create element[“putsTotal”] = puts_total
put_bytes = 0
Do the same with elements = {“putBytes”, “topicPutBytes”}
create element[“putsBytesTotal”] = puts_bytes

Some tools like a date-time field.

I combined the above fields into one field. I effectively used startDate||’/’||startTime and passed this into the data-time code with the de-format string “%Y-%m-%d/%H.%M.%S” to parse it and produce a date time object.

Statistics messages are not produced at well defined times.

They are produced at an interval specified by the qmgr attribute STATINT. Mine was set to 600 (seconds). This produced records at 17:04:06, 17:14:06, 17:24:06. I stopped the queue manager and restarted it, and the next records came out at 17:32:37, 17:42:37, 17:52:37, and the times do not line up.

You need to think how to use this data. You may want to produce a chart showing usage over a day. The vertical access is the number of puts, the horizontal axis is hours. What do you do with a record which was from 19:45 to 20:05, produced at 20:05 ?

You could just record it in 20:00 to 21:00 bucket. This makes the peak look as it is was at 20:00 to 21:00 – not 19:00 to 20:00, so your graphs give the wrong message.

You could distribute the messages according to the time spent in each hour, so from 1945 to 2000 is 15 minutes, and from 2000 to 2005 is 5 minutes. You split the data in a ratio of 15:5 between the 19:00-20:00 bucket, and the 20:00 to 21:00 bucket. This is more more accurate, but still misleading. If you stop trading at 19:59:59. Splitting the data across the buckets will show usage past 20:00 which may not be true

You have to calculate the duration of the record.

Each record has a section called eventData with Start Date, Start Time, and End Date and Time. You can now calculate the duration yourself. As I needed date-time objects for another reason, I found it easiest to create the StartDateTime, and EndDateTime objects and then say duration = EndDateTime – StartDateTime. Rather than calculating EndTime – StartTime. If this is < 0 then add 24 hours.

Epoch needs to be changed.

You get data like

“eventCreation”: {
“timeStamp”: “2019-02-26T12:07:44Z”,
“epoch”: 1551182864
},

Epoch is the number of seconds from 1/1/1970. You can format it for example on Ubuntu, date –date @1551182864 gave Tue 26 Feb 12:07:44 GMT 2019. Unfortunately you cannot use this as a time stamp in some spread sheets, because they use a different day 0!, and so using “format as date” on the epoch integer give you the wrong answer. On Libre Office the calculation is epoch/86400+25569, and format as date time

Formatting the data in a spread sheet.

When you import the data into a spread sheet, you have to specify the formatting for some columns. To automate this, you may want to write a macro to do this automatically for you.

Using tools like Elasticsearch and Kibana.

Documents in Elastic search require a document type and unique id.

The document type could be MQ.Queue_Accounting, MQ.MQI accounting, MQ.Queue_Statistics, MQ.MQI_Statistics.

The unique id could be

  • qmgr,eventCreation_timeStamp for qmgr MQI statistics
  • qmgr,eventCreation_timeStamp.queueName for queue statistics
  • qmgr,eventCreation_timeStamp.processId.threadID for qmgr MQI accounting
  • qmgr,eventCreation_timeStamp.queueName.processId.threadID for queue accounting.

You can now easily display MQ PCF data using Python

For many years I have been frustrated with displaying data provided by MQ midrange in PCF format, as there was nothing to help you use it. Yes, you could display it, but it was a bit like saying … here is a bucket of bits, now build your own program from it to process the data – good luck.

I had started to write some C code to process it, but then I retired from IBM, and since then have found that Python is a brilliant systems management language.

I have put up on Github, some Python code which does the following.

You can create PCF command, such as INQUIRE QUEUES. It can then parse the output into “English” and you can then do things with it. By “English” I mean

Q_TYPE: LOCAL rather than the value 1
DEF_BIND: BIND_ON_OPEN rather than the value 16384

There are examples of what you can do with it.

get_pcf.py. You specify the connection details, and the queue, and it returns the data in json format which you can pipe into another command.

queues.py issues the INQUIRE QUEUE command. This outputs the data in json format. queues2.py then writes it into a file, one file per queue, one line per attribute. This is great

  • you do not have to worry about trying to parse the output from runmqsc
  • you can use standard tools on the file
  • you can do more…

There is a diff.py sample you give it a list of files and it tells you the difference in the queue definitions (while ignoring attributes like change date), for example

CP0000.yml CP0001.yml : Q_NAME CP0000 / CP0001
CP0000.yml CP0001.yml : Q_DESC Main queue / None
CP0000.yml CP0001.yml : MAX_Q_DEPTH 2000 / 5000
CP0000.yml CP0001.yml : Q_DEPTH_HIGH_EVENT ENABLED / DISABLED

There is standards.py which allows you to check attributes in the .yml file meet your corporate standards!

There is events.py and events2.py so you can now process the events produced by the define, delete and alter commands, and see who made the change what the change was, and when the change was made.

I am working on making the stats and accounting usable, so you can create a .csv file with useful data in it, or pass it into Kibana and other tools. So watch this space.

I would welcome any comments of feedback. Ive had one already. When using Eclipse and Python it supports text completion – so if you typed in MQ.INQ… it should give you a list of options to pick from.

These tools build on top of the excellent pymqi package which provide the MQAPI for Python programs. You use pymqi to put and get messages, then use the mqtools package to process the data.

How to get hold of it…

The README has instructions on how to download it. If there is enough interest I’ll package it up so PIP INSTALL can find it.

Ive got nothing in my events queue!

I was looking to test out my Python code and was checking the PCF messages produces – or in my case was not being produced. I did some playing around.

There is a DIS QMGR EVENT command (not eventS) this gave me

QMNAME(QMA) AUTHOREV(DISABLED)
CHLEV(DISABLED) CMDEV(DISABLED)
CONFIGEV(ENABLED) INHIBTEV(DISABLED)
LOCALEV(DISABLED) LOGGEREV(DISABLED)
PERFMEV(ENABLED) REMOTEEV(DISABLED)
SSLEV(DISABLED) STRSTPEV(ENABLED)

So the reason I was not getting many events is they mostly turned off.

I used runmqsc and exploited the command-complete facility.
I typed alter qmgr chl and pressed tab for it to complete it ALTER QMGR CHLEV(ENABLED). I then enter AUTHO and pressed tab and so on.
Then, when the command was completed, I pressed enter – only for it to complain about LOGGEREV was only valid when using linear logging.

The whole command was

ALTER QMGR CHLEV(ENABLED) AUTHOREV(ENABLED) CONFIGEV(ENABLED) INHIBTEV(ENABLED) LOCALEV(ENABLED) PERFMEV(ENABLED) REMOTEEV(ENABLED) SSLEV(ENABLED) STRSTPEV(ENABLED)

What do change events look like?

I was using my Python programs for processing MQ PCF data, and was testing them out with different sorts of MQ data. I found the documentation for change events in the IBM knowledge centre is a bit sparse, so here is some information on it.

I did

  • define QL(DELETEME)
  • alter ql(DELETEME) descr(‘change comment’)
  • delete QL(DELETEME)

For define and delete there is one event message created. For alter there are two messages created, one contains before data the other has the after data.

The MD looks like

"MQMD":{
     "StrucId":"MD  ",
     "Version":1,
     "Report":"NONE",
     "MsgType":"DATAGRAM",
     "Expiry":-1,
     "Feedback":"NONE",
     "Encoding":546,
     "CodedCharSetId":1208,
     "Format":"MQEVENT ",
     "Priority":0,
     "Persistence":"NOT_PERSISTENT",
     "MsgId":"0x414d5120514d4120202020202020202020…",
     "CorrelId":"0x414d5120514d4120202020202020202020…",
     "BackoutCount":0,
     "ReplyToQ":"  ",
     "ReplyToQMgr":"QMA",
     "UserIdentifier":"            ",
     "AccountingToken":"0x0000…",
     "ApplIdentityData":"  ",
     "PutApplType":"QMGR",
     "PutApplName":"QMA   ",
     "PutDate":"20190217",
     "PutTime":"11141373",
     "ApplOriginData":"    ",
     "GroupId":"0x000000...",
     "MsgSeqNumber":1,
     "Offset":0,
     "MsgFlags":0,
     "OriginalLength":-1
   },

When a change event occurs the MD for the “after” message is effectively the same but note the MsgID is different, and the CorrelId is the same, so you will need to clear the MsgId and keep the CorrelID when getting the second message.

For the create command the PCF header was

"Header":{
     "Type":"EVENT",
     "StrucLength":36,
     "Version":2,
     "Command":"CONFIG_EVENT",
     "MsgSeqNumber":1,
     "Control":"LAST",
     "CompCode":0,
     "Reason":2367,  "CONFIG_CREATE_OBJECT"
     "ParameterCount":58,
     },

For the delete command the PCF header was

"Header":{
     "Type":"EVENT",
     "StrucLength":36,
     "Version":2,
     "Command":"CONFIG_EVENT",
     "MsgSeqNumber":1,
     "Control":"LAST",
     "CompCode":0,
     "Reason":2369, "CONFIG_DELETE_OBJECT
     "ParameterCount":58

}

For the alter/change command the PCF headers were

"Header":{
     "Type":"EVENT",
     "StrucLength":36,
     "Version":2,
     "Command":"CONFIG_EVENT",
     "MsgSeqNumber":1,
     "Control":"NOT_LAST",
     "CompCode":0,
     "Reason":2368,
     "ParameterCount":58    
   },

and

"Header":{
     "Type":"EVENT",
     "StrucLength":36,
     "Version":2,
     "Command":"CONFIG_EVENT",
     "MsgSeqNumber":2,
     "Control":"LAST",
     "CompCode":0,
     "Reason":2368, "CONFIG_CHANGE_OBJECT"
     "ParameterCount":58
   },

The differences are the MsgSeqNumber value and Control being NOT_LAST or LAST.

The data is common to all requests. You need to compare the fields in both the records for the change event, to see which are different. With the data in Python dict – this was a trivial exercise.

"Data":{

    "EVENT_USER_ID":"colinpaice",

    "EVENT_ORIGIN":"CONSOLE",

    "EVENT_Q_MGR":"QMA",

    "OBJECT_TYPE":"Q",

    "Q_NAME":"DELETEME",

    "Q_DESC":"",

    …

    "Q_TYPE":"LOCAL"

  }

after the change

"Data":{
     "EVENT_USER_ID":"colinpaice",
     "EVENT_ORIGIN":"CONSOLE",
     "EVENT_Q_MGR":"QMA",
     "OBJECT_TYPE":"Q",
     "Q_NAME":"DELETEME",
     "Q_DESC":"change comment",
     …
     "ALTERATION_DATE":"2019-02-17",
     "ALTERATION_TIME":"11.14.33",
     …
     "Q_TYPE":"LOCAL"
   }

When I printed out the changed data, the first time ALTERATION_DATA and ALTERATION_TIME were both displayed. The second time, only ALTERATION_TIME was displayed. I thought this was a bug in my program. After checking my program, it was obvious… If I change the queue 10 minutes later – the ALTERATION_DATE does not change – so remember to report both of these, and not just the change values.

Baby Python scripts doing powerful work with MQ

I found PyMqi is an interface from Python to MQ. This is really powerful, and I’m am extending it to be even more amazing!

In this blog post, I give examples of what you can do.

  • Issue PCF commands and get responses back in words rather than internal codes ( so CHANNEL_NAME instead of 3501)
  • Saving the output of DISPLAY commands into files
  • Using these files to compare definitions and highlight differences.
  • Check these files conform to corporate standards.
  • Print out from the command event queue, and the stats event queue etc


I can use some python code to display information via PCF

  • # connect to MQ
  • qmgr = pymqi.connect( queue_manager,”QMACLIENT”,”127.0.0.1(1414)”)
  • # I want to inquire on all SYSTEM.* channels
  • prefix = b”SYSTEM.*”
  • # This PCF request
  • args = {pymqi.CMQCFC.MQCACH_CHANNEL_NAME: prefix}
  • pcf = pymqi.PCFExecute(qmgr)
  • # go execute it
  • response = pcf.MQCMD_INQUIRE_CHANNEL(args)

This is pretty impressive as a C program would take over 1000 lines to do the same!

This comes back with data like

  • 3501: b’SYSTEM.AUTO.RECEIVER’,
  • 1511: 3,
  • 2027: b’2018-08-16 ‘,
  • 2028: b’13.32.15′,
  • 1502: 50

which is cryptic even for experts because you need to know 3501 is the value of the type of data for “CHANNEL_NAME”.

I have some python code which converts this to..

  • ‘CHANNEL_NAME’: ‘SYSTEM.AUTO.RECEIVER’,
  • ‘CHANNEL_TYPE’: ‘RECEIVER’,
  • ‘ALTERATION_DATE’: ‘2018-08-16’,
  • ‘ALTERATION_TIME’: ‘13.32.15’
  • ‘BATCH_SIZE’: 50
  • …

for which you only need a kinder garden level of MQ knowledge to understand it. It converts 3501 to CHANNEL_NAME, and 3 into RECEIVER

With a few lines of python I can write this data out so each queue is a file on disk in YAML format.

A yaml file for a queue looks like

  • Q_NAME: TEMP
  • Q_TYPE: LOCAL
  • ACCOUNTING_Q: Q_MGR
  • ALTERATION_DATE: ‘2019-02-03’
  • ALTERATION_TIME: 18.15.52
  • BACKOUT_REQ_Q_NAME: ”

Now it gets exciting! (really)

Now it is in YAML, I can write small Python scripts to do clever things. For example

Compare queue definitions

  • from ruamel.yaml import YAML
  • import sys
  • yaml=YAML()
  • q1 = sys.argv[1] # get the first queue name
  • ignore = [“ALTERATION_DATE”,”ALTERATION_TIME”,
  • “CREATION_DATE”,”CREATION_TIME”]
  • in1 = open(q1, ‘r’) # open the first queue
  • data1 = yaml.load(in1) # and read the contents in
  • for i in range(2,len(sys.argv)): # for all of the passed in filenames
  • q2=sys.argv[i] # get the name of the file
  • in2 = open(q2, ‘r’) # open the file
  • data2 = yaml.load(in2) # read it in
  • for e in data1: # for each parameter in file 1
  • x1 = data1[e] # get the value from file 1
  • x2 = data2[e] # get the value from the other file
  • if not e in ignore: # some parameters we want to ignore
  • if x1 != x2: # if the parameters are different
  • print(q1,q2,”:”,e,x1,”/”,x2) # print out the queuenames, keywork and values

From this it prints out the differences

  • queues/CP0000.yml queues/CP0001.yml : Q_NAME CP0000 / CP0001
  • queues/CP0000.yml queues/CP0001.yml : OPEN_INPUT_COUNT 1 / 0
  • queues/CP0000.yml queues/CP0001.yml : MONITORING_Q Q_MGR / HIGH
  • queues/CP0000.yml queues/CP0001.yml : OPEN_OUTPUT_COUNT 1 / 0
  • queues/CP0000.yml queues/CP0002.yml : Q_NAME CP0000 / CP0002
  • queues/CP0000.yml queues/CP0002.yml : OPEN_INPUT_COUNT 1 / 0
  • queues/CP0000.yml queues/CP0002.yml : OPEN_OUTPUT_COUNT 1 / 0

I thought pretty impressive for 20 lines of code.

and another script -for checking standards

  • from ruamel.yaml import YAML
  • import sys
  • yaml=YAML()
  • q1 = sys.argv[1] # get the queue name
  • # define the variables to check
  • lessthan = {“MAX_Q_DEPTH”:100}
  • ne = {“INHIBIT_PUT”:”PUT_ALLOWED”,”INHIBIT_GET”: “GET_ALLOWED”}
  • in1 = open(q1, ‘r’) # open the first queue
  • data = yaml.load(in1) # and read the contents in
  • # for each element in the LessThan dictionary (MAX_QDEPTH), check with the
  • # data read from the file.
  • # if the data in the file is “lessthan” the value (100)
  • # print print out the name of the queue and the values
  • for i in lessthan: # just MAX_Q_DEPTH in this case
  • if data1[i] < lessthant[i] : print(q1,i,data[i],”Field in error. It should be less than <“,lessthan[i])
  • # if the values are not equal
  • for i in ne: # INHIBUT_PUT and #INHIBIT_GET
  • if data[i] != ne[i] : print(q1,i,data[i],”field is not equal to “,lt[i])

the output is 

queues/CP0000.yml
MAX_Q_DEPTH 5000 Field in error. It should be < 100

Display command events

difference Q_NAME CP0000 CP0000 ALTERATION_DATE 2019-02-07 2019-02-11

difference Q_NAME CP0000 CP0000 ALTERATION_TIME 20.48.24 21.29.23

difference Q_NAME CP0000 CP0000 MAX_Q_DEPTH 4000 2000

With my journey so far – Python seems to be a clear winner in providing the infrastructure for managing queue managers.