Featured

When is activity trace enabled?

I found the documentation for activity trace was not clear as to the activity trace settings.

In mqat.ini you can provide information as to what applications (including channels) you want traced.

For example

applicationTrace:
ApplClass=USER
ApplName=progput
Trace=OFF

This file and trace value are checked when the application connects.  If you have TRACE=ON when the application connects, and you change it to TRACE=OFF, it will continue tracing.

If you have TRACE=OFF specified, and the application connects, changing it to TRACE=ON will not produce any records.

With

  • TRACE=ON, the application will be traced
  • TRACE=OFF the application will not be traced
  • TRACE= or omitted then the tracing depends on alter qmgr ACTVTRC(ON|OFF).   For a long running transaction using alter qmgr to turn it on, and then off, you will get trace records for the application from in the gap.

If you have

applicationTrace:
ApplClass=USER 
ApplName=prog* 
Trace=OFF

applicationTrace:
ApplClass=USER 
ApplName=progp*
Trace=ON

then program progput will have trace turned on because the definition is more specific.

You could have

applicationTrace:
ApplClass=USER 
ApplName=progzzz
Trace=OFF

applicationTrace:
ApplClass=USER 
ApplName=prog*
Trace=

to  be able to turn trace on for all programs beginning with prog, but not to trace progzzz.

 

Thanks to Morag of MQGEM  who got in contact with me, and said  long running tasks are notified of a change to the mqat.ini file, if the file has changed, and a queue manager attributed has been changed – even if it is changed to the same variable.

This and lots of other great info about activity trace (a whole presentation’s worth of information) is available here.

Whoops my QM emergency recovery procedures did not recover QM in an emergency!

I was working with someone, and we managed to kill a test queue manager on midrange.  I suggested we test out the “emergency procedures” as if this was production and see if we could get “production” back in 30 minutes.

We learned so much from this exercise, so we are now working on a new emergency recovery procedure.

What killed the queue manager

The whole experience started when we thought had better clean up some of the MQ recovery logs.   With circular logging the when the last log fills up it overwrites the first one.  This is fine for many people but it means you may not be able to recover a queue if it is damaged.
We had linear logging, where the logs are not overwritten, MQ just keeps creating new logs.   You can recover queues if they are damaged, because you can go back through the logs.
As our disk was filling up someone deleted some old logs – which were over a week old – and were “obviously” not needed.

MQ was shut down, and restarted – and failed to start.

Lesson 1:  With linear logging you are meant to use the rcdmqimg command which copies queue contents to the log.   You get a message telling you which logs are needed for object recovery, and which logs are needed for restart.   This information is also in the  AMQERRxx.LOG.  You cannot just delete old logs as they may still be needed.

Issue the command at least daily.

Lesson 2:  HA disks do not give you HA.   The disks were mirrored to the backup site – also to the DR site.  The delete file command was reliably done on all disk copies.  We could not start MQ on any site because of the missing file.  We should have had a second queue manager.

These HA disk solutions and active/standby give you faster access to your data, in  a multi site environment, they do not give you High Availability

Initial panic – what options do we have

Lesson 3: Your instructions on how to recover from critical situations need to be readily available.  They should be tested regularly.    We could not find any. You need a process to follow which works, and you have timings for.  So you do not have a half hour discussion “should we restore from backup?”, “how long will it take?”, “will it work?”, “how do we restore from backup”.   The optimum solution may be to shoot  the queue manager and recreate it.  This may be the optimum route to getting MQ “production” back.  You should not have to make critical decisions under pressure, the decision path should have been documented when you have the luxury of time.

Lesson 4: you need to capture the output of every command you are doing.  Support teams will ask “please send me the error logs”.  You do not want to have to copy an paste all of your terminal data.  Linux has a “script” command which does this.  They could not email me the log showing the problems, so we had to have a conference call, and “share screens” to see what was going on, which made it hard for me to look at the problem “up a bit, down a bit – too far”.  All of which extended the recovery period.

Lesson 5: “Let’s restore from the backups”  These backups were taken  12 hours before and were not current, and we did not know how to restore them.
(Little thought, should backups be taken when a QM is down, or do you get integrity issues because files and logs were backed up at different times? – I know z/OS can handle this – Feedback from Matt at IBM.  Yes the queue manager should be shut down for backups – so you need two or three queue managers in your environment.

Make sure you backup your SSL keystore.

Let’s recreate the queue manager

Lesson  6: Do you have any special actions to delete multi instance queue managers.

Do you need linux people to do anything with the shared disks?

Lesson 7: Save key queue manager configuration files.  When you delete a queue manager instance – it deletes any qm.in and MQAT.ini files – you need them as they may have additional customising, for example SSL information.
Of course you are backing these files up -and of course you (personally)  have tested that you can  recover them from the backup.

Copy qm.in and MQAT.ini to a safe location before you delete the queue manager.

Lesson 8:  Ensure people have enough authority to be able to do all the tasks – or have an emergency “break glass userid”.  Many sites only allow operations people to access production with change capability.

Lesson 9:  You need to know how the  create queue manager command and parameters used to create the queue manager.
Some queue manager options can be changed after the queue manager has been started.  Others cannot – for example linear logging|circular logging.  Size of log files etc.

You need to have saved the original command used with all of the options.   Do not forget that when you did it the first time it was MQ V7.5 – you are now migrated to MQ V9, so it should work OK!

Lesson 10: Copy the qm.ini files etc and overwrite the newly created ones.

Start the queue manager.

Lesson 11:  Customize the queue manager.  You need to have a file of all of your objects queues and channels etc.  You may have a file which you use to create all new queue managers, but this may not  be up to date.  It is better to run dmpmqcfg every day to dump the definitions to get the “current” state of the objects which you can reload.
The -o 1line option is useful as then you can use grep to select objects with all the parameters.

Lesson 12: In your emergency recreate document  note how long each stage takes.  One step, closing down the queue manager took several minutes.  We were discussing if was looping or not – and should we cancel it.  Eventually it shut down.  It would have been better to know this stage takes 5 minutes.

Lesson 13: Document the expected output from each stage – and highlight any stage which gives warnings or errors.  We ran runmqsc with a file of definitions, and it reported 7 errors.  We wasted time checking these out.  Afterward we were told “We always get those”.

Lesson 14: Do you need to do work for your multi instance queue managers?

Getting the queue manager back into “production”. 

Lesson 15: Resetting receiver channel sequence numbers.   Sender and receiver channels will have the wrong sequence number.  You can reset the sender channels yourself.  Receiver channels are a bit harder, as the “other end” has to reset the sequence number. You can  either

  • Contact the people responsible for the other end (you do have their contact details dont you?) and
  • ask them to reset the channel,

or you wait till their queue manager sends you a message – and then you get notified of the sequence number mismatch, and can use reset channel to reset your number to the expected number.   The channel will retry and this time it will  work.  This means you need to sit by your computer, waiting for these events.  Maybe no messages will be sent over the weekend, and so you can logon first thing Monday morning to catch the events.

Lesson 16: Your SSL keystore is still available isnt it?

Lesson 17: Is every one who has the on-call phone familiar with this procedure, and has practiced it?

Lesson 18: People need to be familiar with the tools on the platform.  You may normally use notepad to edit files on your personal work station.  On the production box you only have “vi”.

Overall – this is one process that needs to work – and to get your queue manager up in the optimum time.  You need to practice it, and get it right.

Summary

You need to practice emergency recovery situations

I used to do Scuba diving.  You learn, and have to practice “ditch and retrieve” where you take your kit off under water and have to put it on again.   Once I needed to do this in the sea.  It was dark, I got caught in a fishing net, so I had to take my kit off, untangle it (by touch), and put it on again.  If I had not practiced this I would not be here today.

 

Checking the daisy chain around my MQ network

Young children collect flowers and chain them to make a circle and so make a daisy chain.

People also talk about daisy chaining electrical extension leads together to make a very long lead out of lots of small leads.

In MQ we can also have daisy chains.  One use is to check all of the links are working, and there are no delays on the channels.

If an application puts a message onto the Clustered Request Queue(BQ) on QMA, it goes around and the reply can be got from the Reply queue, then we have checked all the links are working; we have daisy chained the requests.

DaisyChainOnce I Once I got it working the definitions were simple.

 

On QMB

DEFINE QR(BQ) RQMNAME(QMC) RNAME(CQ) CLUSTER(MYCLUSTER)

On QMC

DEFINE QREMOTE(CQ) CLUSTER(MYCLUSTER) RQMNAME(QMA) RNAME(REPLY)

On QMA

DEFINE QL(REPLY)

Once we have set the definitions up I could can use the MQ utility dspmqrte to show us the path.  For example

  • dspmqrte puts a message to the BQ .  This is a clustered queued on QMB,  It reports the queue BQ is being used, and stores the message on SYSTEM.CLUSTER.XMIT.QUEUE.
  • On QMA the channel TO.B gets the message from the SCTQ and sends it
  • On QMB the channel TO.B says put this to BQ, which is defined as CQ, and stores it on the SCTQ.
  • On QMB the channel TO.C gets the message from the SCTQ and sends it to QMC.
  • On QMC the channel TO.C says put this to CQ, which is defined as REPLY,  and stores if on SCTQ
  • On QMC the channel TO.A gets the message from the SCTQ and sends it to QMA.
  • On QMA the channel TO.A puts it to the Reply queue.

I used dspmqrte -m QMA -q BQ …, and it worked like magic.   I requested summary information(-v summary) and I got the following output, which shows the intermediate queues used.

AMQ8653I: DSPMQRTE command started with options ‘-m QMA -qBQ -rqCP0001 -rqm QMA -v summary -d yes -w5’.
AMQ8659I: DSPMQRTE command successfully put a message on queue ‘SYSTEM.CLUSTER.TRANSMIT.QUEUE’, queue manager ‘QMA’.
AMQ8674I: DSPMQRTE command is now waiting for information to display.
AMQ8666I: Queue ‘SYSTEM.CLUSTER.TRANSMIT.QUEUE’ on queue manager ‘QMA’.
AMQ8666I: Queue ‘SYSTEM.CLUSTER.TRANSMIT.QUEUE’ on queue manager ‘QMB’.
AMQ8666I: Queue ‘SYSTEM.CLUSTER.TRANSMIT.QUEUE’ on queue manager ‘QMC’.
AMQ8666I: Queue ‘REPLY’ on queue manager ‘QMA’.
AMQ8652I: DSPMQRTE command has finished.

Note, specifying RQMNAME is not required, and clustering will pick a queue manager which hosts the queue.  This means that you may be testing a different path to what you expected.  By using it you specify the route.

When I stopped QMC and retried the dspmqrte command , I got

AMQ8653I: DSPMQRTE command started with options ‘-m QMA -qBQ -rqCP0001 -rqm QMA -v summary -d yes -w5’.
AMQ8659I: DSPMQRTE command successfully put a message on queue ‘SYSTEM.CLUSTER.TRANSMIT.QUEUE’, queue manager ‘QMA’.
AMQ8674I: DSPMQRTE command is now waiting for information to display.
AMQ8666I: Queue ‘SYSTEM.CLUSTER.TRANSMIT.QUEUE’ on queue manager ‘QMA’.
AMQ8666I: Queue ‘SYSTEM.CLUSTER.TRANSMIT.QUEUE’ on queue manager ‘QMB’.
AMQ8652I: DSPMQRTE command has finished.

It does not report that there were any problems;  it just did not report two hops.

To see if there are problem, I think the best thing to do is pipe the output into a file

dspmqrte… > today

and compare this with a good day.

diff today goodday -d  gave me the differences – so I could see there was a problem because I was missing

> AMQ8666I: Queue ‘SYSTEM.CLUSTER.TRANSMIT.QUEUE’ on queue manager ‘QMC’.
> AMQ8666I: Queue ‘REPLY’ on queue manager ‘QMA’.


I had tried to define a clustered queue alias queues instead of a remote queue.  I got responses like

Feedback: UnknownAliasBaseQ, MQRC_UNKNOWN_ALIAS_BASE_Q, RC2082.

Should I use dynamic queues? – probably not for high volume production work

I’ve been looking into temporary dynamic queues on midrange MQ.  People tend to use these as a scratch queue which can be deleted when it has been finished with.   Rather than use a shared reply queue, and get messages by msgid or correlid, you can just say get next message, as you know there will only be messages for your application instance  on the  queue.

The application team may save money by using a dynamic instead of a common replyTo queue,  but you are likely to pay for it else where – including management time.

Using dynamic queues may be easy to program – but can have some serious implications.

Sometimes a dynamic queue is used because it saves the programmer about 5 minutes of coding by not specify nor testing gets specifying  msgid or correlid.
One customer had a paranoid architect  who did not want an application instance to have access to messages from another instance.  With a well designed, coded, and reviewed application this should not occur – or use IBM MQ AMS to encrypt the messages.

I had a email from someone who said they are having problems with statistics on midrange because of the  large number of dynamic queue – more of this later.   I think they said they get statistics  for 10 million dynamic queues a day, hence the question if they should change their applications.

This temp dynamic queue can be used for

  • a one-of:  MQOPEN of a model queue to create a  dynamic queue. Then  MQPUT of request, the reply comes back to the dynamic queue, MQCLOSE of the reply queue, which deletes the queue.
  • open the queue in the morning, use it all day when there is no work to do. The application ends, and deletes the queue.

You get the messages, close the queue, and the queue disappears – easy – what can go wrong?

I’ll cover

  • the impact on the applications of using dynamic queues,
  • the impact of systems management.

The impact of applications using dynamic queues

The documentation says The queues are deleted when the application that issued the MQOPEN call that created the queue closes the queue or terminates. … If the queue is in use at this time (by the creating, or another application), the queue is marked as being logically deleted, and is only physically deleted when closed by the last application using the queue.

For performance reasons, a receiver channel does not use MQPUT1 to put the messages to the queue, it does MQOPEN, MQPUT, MQPUT…  as is many cases the same few queues are used. and MQOPEN, MQPUT, MQPUT is more efficient.  After some time, if the queue has not been used the channel will close the queue.
The implications of this are that if the application is a one-of,  the program started and finished in 1 second,  a channel may keep hold of the queue. You think there is only one dynamic queue in use at any time, but if you have 1000 channels – there is potentially a lot of queues being only logically deleted and still around.

Conceptually the queue manager has to created the queue every time a dynamic queue is needed.   In practice the queue manager has a pool of these which can be serially reused and so keep the costs down.   This is fine for a small number of queues.

I did some totally unscientific measurement on my laptop to give a indication of the impact of using dynamic queues.  The applications I used are completely non typical.  The measurement give an indication and are meant to make you think about using dynamic queues rather than giving information you can use.

First measurement open the queues

Have a thread open a queue, 20,00 times.  The application does 2000 opens, report statistics,  wait 1 second, repeat.

Queue type Average CPU per open Average Elapsed time uSeconds per open
Permanent First 2000 : 7
Last 2000 : 8
First 2000 : 64
Last 2000:  101
Dynamic First 2000 : 8
Last 2000 : 8
First 2000 : 183
Last 2000: 303

The CPU usage looks similar, but the elapsed time increases when dynamic queues are used.

While the first program had its dynamic queues open, a second instance was run, the first 2000 took  290 microseconds, the last 2000 took 343 microsecond per open on average, so there is an increase in open time depending on the number of temp dynamic queues which are open.  The increase is not just dependent of the number of queues opened per thread.

Second measurement – close the queues.

The queues were closed

  • test 2A, in the order they were opened,
  • test 2B,  in reverse order of opening.
Queue type Order Average CPU Average Elapsed time uSeconds
Permanent First opened, first closed First 2000 : 284 Last 2000 : 11 First 2000 : 1533
Last 2000: 83
Permanent Last opened, first closed First 2000: 5
Last: 2000: 8
First 2000 : 58
Last 2000: 65
Dynamic First opened, first closed First 2000 : 284 Last 2000 : 16 First 2000 : 1533
Last 2000: 783
Dynamic Last opened, first closed First 2000: 8
Last: 2000: 8
First 2000: 295
Last: 2000: 736

When the queue is last-opened-first-closed this is much cheaper than first-opened-first- closed.  It feels like there is a linear list of handles.  It starts at the last opened, and scans backwards looking for the specific handle.

  • An application should not have hundreds of queues open.
  • If it does, the cost will depend, on where the handle is in the list – the “cheap end”, or the “expensive end”.  You may have no control over this, so expect significant variation in the cost and elapsed time of performing the MQCLOSE on the queue.
  • The dynamic queues took longer to close than the permanent queues
  • In general it is quicker and uses less CPU time to open a permanent queue  than using a dynamic queue.   If you open and shut a queue 100 times a second this may be relevant.  If you open and close them once a day, and keep them open all day,  this is not so important.

Overwhelmed by statistics data.

All good production environment should be collecting statistics on queue usage.  If the STATQ attribute is ON ( either for the queue, or for the queue manager) then a statistics record will be produced every STATINT seconds.

Each time I ran my test  I had over 200 messages on SYSTEM.ADMIN.STATISTICS.QUEUE, and 20,000 records, one for each dynamic queue (100 record per statistics message).
For STATINT of 5 minutes, if I had opened the queues, and left them open, in  a 10 hour day this would be 12 * 10  * 20,000 which is 2.4 million records.  (If it reported the model queue name, instead of the generated name this would be just 120 records and would be so much more useful)!.
When I ran my program many times in the 5 minute interval I got many * 20,000 rows.

You take these records and store them in your statistics database, and instead of having one database row saying “at 0805 queue PAYROLL had 10000 MQPUTS”, you have 10,000 rows saying “At 0805 Queue AMQ.5CF22BF6214…. had one MQPUT”.   As you keep your data for at least 13 months (so you can look back at last year’s peak periods), you have a lot of data and wasted space.

What you can do.

  1. Turn off stats collection for these dynamic queues – not very helpful.
  2. Post process the data and aggregate all of the dynamic queues for a period into one record (select (sum(puts),….) where qname like ‘AMQ%’….) – and then delete the individual records.  Extra work which could be avoided if you did not use dynamic queues.
  3. Use a model queue name for each business application so you get  MOBILE.5CF22BF62144A7A4, and CreditCheck.5CF22BF62144A7A4 instead of AMQ.5CF22BF62144A7A4 to make it easier to see where the queues are used.
  4. Change the applications to use a shared reply queue. This may be the hardest one to do.
    1. Ensure the applications get their message by MSGID or CORRELID, and not just get next.
    2. Use a common reply queue names to provide isolation – such as MOBILE.REPLY and CREDITCHECK.REPLY.
    3. You may have to implement best practices of cleaning up old messages.   If a dynamic queue was closed any messages on the queue would be deleted.  As you are not deleting the common queue – you need to clean these up yourself.   This can occur when an application timed out and then the reply arrived.

In summary

The applications team may have saved some money  by using a dynamic queue instead of a shared reply queue – but you will spend much more money, on CPU, disk space for database tables.   Then of course there is the management time spent discussing why the reporting of dynamic queue usage is not so useful.

 

 

 

 

Buried options to dspmqrte make it more useful, but…

I was bemoaning that I dspmqrte did not actually tell me when the message arrived on the queue (COA), and  when the application did an MQGET ( message delivered, COD).

I piped the output from the dspmqrte command into a few lines of python and got

AMQ8659I: DSPMQRTE command successfully put a message on queue SYSTEM.CLUSTER.TRANSMIT.QUEUE , queue manager QMA .
AMQ8674I: DSPMQRTE command is now waiting for information to display.
Put QM:QMA QN:CSERVER RemoteQMName:QMC
Get QM:QMA QN:SYSTEM.CLUSTER.TRANSMIT.QUEUE 
Send QM:QMA RQMN:QMC Chl:CL.QMC ChlT:ClusSdr 
Receive QM:QMC RQMN:QMA Chl:CL.QMC 
Discard QM:QMC QN:CSERVER
AMQ8652I: DSPMQRTE command has finished.

We can see the message got there because it was discarded.

I changed my back end application to ignore messages with

MsgType :MQMT_DATAGRAM  and Format :MQADMIN.

Now with the command

dspmqrte -m QMA -q CSERVER -rq CP0001 -rqm QMA -d yes -v outline -w5 -ro activity,coa,cod

I got additional messages

AMQ8654I: Trace-route message arrived on queue manager QMC .
AMQ8662I: Trace-route message delivered on queue manager QMC .

When I stopped my backend application I got the message arrived – but not the message delivered, as expected.

For a small of work in the application I got a huge improvement in problems diagnostic tools.    I was really please with my progress, and had a cup of tea to celebrate.

The title said “but…”.  This is because the message arrived, and message delivered are not displayed in real time, but are produced after the time out interval specified in the -w option had expired.  I had hoped to use dspmqrte to tell me where the delays were in route to the queue.
So all I can tell is the message got there – but not how long it took.  What a wasted opportunity to provide useful information.

So get your applications changed to ignore these trace route messages – and use the -ro COA and COD to tell you if your messages are being processed in a timely manner!


Update.
If you specify the -d yes option the message can be got by your application.
If you then mqput this message to the next application in a sequence of applications, you will get the channel activity for sending it on, and the COA and COD messages sent back when it arrives at each hop.
Unfortunately, the dspmqrte program displays the last COA, and the last COD messages.

I sent the request CSERVER on QMC.   The application sent this to queue CP0000 on QMA.  The COA and COD message displayed were for QMA.  The COA and COD messages do not tell you which queue the event applies to!

 

 

 

 

 

 

Sorting out the MQ application trace knotted spaghetti.

You can turn on report = MQRO_ACTIVITY to get activity traces sent to a queue.   This shows the hops and activity of your message.
You can create your own trace route messages to be sent to a remote queue, and get back the hops to get to the queue, or you can use the dspmqrte command to do this for you.

Which ever way you do it, the result is a collection of messages in your specified reply queue.  The problem is how do you untangle the messages.  It is not easy with for a single message.  If you are getting these activity messages every 10 seconds from multiple transactions, you quickly  get knotted spaghetti!   To entangle the spaghetti even more, you could have a central site processing these data from many queue managers, so you get data from multiple messages, and multiple queue managers.

You can get a message for part of your application or transaction.  For example,

  • a message with information about the first 10 MQ verbs your program uses.
  • a message for the sender channel with the MQGET and the send for the local queue manager, and the remote queue manager will send a message with the channel’s receive and MQPUT.

The easy bit – messages for activity on your queue manager.

The event message has a header section.  This has information including

QueueManager: ‘QMA’
Host Name: ‘colinpaice’
SeqNumber: 0
ApplicationName: ‘amqsact’
ApplicationPid: 28683
UserId: ‘colinpaice’

From  QueueManager: ‘QMA‘ and Host Name: ‘colinpaice‘, you know which machine and queue manager you are on.

From ApplicationPid: 28683 SeqNumber: 0, you can see the records for this applications Process ID, and the sequence number.   This happens to be for a program ApplicationName: ‘amqsact’ and UserId: ‘colinpaice’.  I dont know when the sequence number wraps.  If the application ends, and the same process is reused,  I would expect the sequence number to be reset to 0.

You may have many threads running in a process , such as for a web server.  For each MQ operation  there is information for example

MQI Operation: 0
Operation Id: MQXF_PUT
ApplicationTid: 81
OperationDate: ‘2019-05-25′
OperationTime: ’14:28:18’
High Res Time: 1558790898843979
QMgr Operation Duration: 114

We can see that this is for Task ID 81.

So to tie up all of the activity for a program, you have to select the records with the same ApplicationPid, and check the SeqNumber to make sure you are not missing records.  Then you can locate the record with the same TID.
You also need to remember that a thread behaviour can be complex ( like adding meat balls to the spaghetti).  Because of thread pooling, an application may finish with the thread, and the thread can be reused.  If a thread is not being used, it can be deleted, so you will get MQBACK and MQDISC occurring after a period of time.

It is similar for channels

For a sending channel you get the following fields.

QueueManager: ‘QMA’
Host Name: ‘colinpaice’
SeqNumber: 1723
ApplicationName: ‘runmqchl’
Application Type: MQAT_QMGR
ApplicationPid: 5157
ConnName: ‘127.0.0.1(1416)’
Channel Type: MQCHT_CLUSSDR

MQI Operation: 0
Operation Id: MQXF_GET
ApplicationTid: 1

For a  receiving channel you get

QueueManager: ‘QMA’
Host Name: ‘colinpaice’

SeqNumber: 1746
ApplicationName: ‘amqrmppa’
ApplicationPid: 4509
UserId: ‘colinpaice’

Channel Name: ‘CL.QMA’
ConnName: ‘127.0.0.1’
Channel Type: MQCHT_CLUSRCVR

MQI Operation: 0
Operation Id: MQXF_OPEN
ApplicationTid: 5
….
MQI Operation: 1
Operation Id: MQXF_PUT
ApplicationTid: 5
….

As there can be many receiver channels with the same name for example an Receiver MCA channel, you should be able to use the CONNAME IP address to identify the channel being used.

They may have the same or different ApplicationPid.

It might be easier just search all of the channels for the messages with matching msgid and correlid!

 

 

 

 

 

 

Using mqmd report activity options properly, is very difficult.

MQ has the ability to trace the activity of a message to its destination.  As chart-ware, this  statement this is 100% accurate and looks like a great facility. However, as with many bits of chart-ware the implementation is very difficult.  I can see why people tend not to use this function.

What MQ facilities are used

  • With an MQMD you can set a report option MQRO_COA, confirm on arrival.  This says when the message is put to the (remote) queue, send a copy of the MQMD (optionally + payload) with msgtype of report, to the replyTo queue and replyTo queue manager .
  • With an MQMD you can set a report option MQRO_COD, confirm on delivery.  This says when the message application  process the message on the (remote) queue, send a copy of the MQMD (optionally + payload) with msgtype of report, to the replyTo queue and replyTo queue manager.
  • With an MQMD you can set a report option MQRO_ACTIVITY.  When the message moves through the network, the channels reports the get and send, and the receive and put of the message.   The messages produced have an embedded  PCF message within the payload.  (A message of format Embedded PCF,  MQHEPCF, allow you to have a PCF followed by other data, for example application data).
  • Your application does some work and sends a reply to the replyTo queue and replyTo queue manager with msgtype of reply.  (For example “Transfer £1,000,000 to Colin Paice – has been done”)

You tie all of these pieces together with the msgid and correlid of the original message.

When a message is put on a transmission queue, it has a transmission header, with its own MQMD etc.  Your message is embedded within this transmission queue message, so your msgid and correlid are still available.

What information is available to you?

  1. When your message is put to the queue, the MQMD has the time the message was put.
  2. From the activity record for the sending channel, you have the time the message was got from the transmission queue, and from the embedded message you have the put time .  You can now calculate the duration spent on the transmission queue.
  3. From the activity record for the sending channel, you have the time the data was sent over the network.
  4. From the activity record for the receiving channel, you have the time the data was received over the network.   This should be close to the time it was sent.  If not then the clocks on the two systems may not be in sync, and you can calculate a correction factor.
  5. From the activity record for the receiving channel you have the time the message was put to the queue.
  6. From the Confirm on Arrival record, this should be the same (or close) to the time the receiving channel put the message to the queue.  Apply the correction factor.
  7. From the Confirm on Delivery  record, you know what time the message was got by the application, and so you can calculate duration =  Time of Delivery – Time of Arrival, to see how long the message was on the server queue.    If the message is persistent, then this will include time for the channel to force the data to the log, and do disk IO.   You should be able to calculate the duration between the message being put, and the time it is processed to see the total life time of the message.

The time of day values are in the format HHMMSShh where hh is hundredths of a second.  This allows you to get granularity down to 0.01 of a second.

You now have a detailed picture of where time is being spent en route to the remote queue.  Is it the send channel being slow to the message,  or does the message gets to the remote end quickly, and the application is slow to process it.  Wow – really useful stuff.

It now starts to get messy.

These messages arrive at the specified reply to queue, and reply to queue manager.

To support using these report options, your application needs some logic like

If this is a reply message, then carry on as usual.

Else move message using  MQPUT1 to MY.ACTIVITY.PROCESSING.QUEUE, so you can process these outside of the application.

How do you process MY.ACTIVITY.PROCESSING.QUEUE ?

This is the hard bit.
You have a mixture of PCF type messages, and your original message, (perhaps with a format of MQSTR) with the MQMD saying “report message”.

MQ provides programs which can process some PCF format messages on a queue, and another program for printing (in hex) non PCF messages.  There is no program which can handle both message types and do the calculations to extract all of the useful data.

Im working on getting my Python programs to do all of this processing.

If I have made this more complex than necessary, because I have overlooked some capability, please let me know.


A big Thank You to Morag of MQGEM who pointed me to MO71.  This can format out any PCF message, and it has a trace route viewer

Select a queue manager on the main window as your source queue manager and then choose Action->Trace Message from the menu. Fill in your target queue and queue manager, and press send. You’ll get a picture like this:-

You can view the data is several ways, right click on the picture and choose Display Type.  The Display Options part of the right click context menu changes what is shown in the above picture. Another display type that might be useful is a breakdown of the steps by looking at Display Type->Nodes:-