Why is msgage() empty?

I have a queue with messages on it – but msgage ( of the oldest message on the queue) is empty.

To get this data, and last put/get time etc you need to alter ql(xx) monq(yy) where yy is not “off”. For the queue you may want to alter the queue to LOW, MEDIUM, HIGH. If the value is QMGR, you need to use DIS QMGR MONQ. You could change the QMGR value to LOW … but this would then apply to all queues – which you may or may not want. I think it should it always be LOW… so when you have a problem you have the data to help you diagnose the problem.

Cleaning up after your application programmers.

Some people are Application Pros. This is not a compliment – it is short for half an Application Programmer, because they only develop half an application.

Symptoms of this are the application queue depths are high (100 may be high), the queue never becomes empty, or a large msgage for the queue.

Consider an application which sends a request to a backend and expects a response. Normally the response comes back in time, the application gets it, and the reply is deleted.
If there is a delay, the application gets a no-message found (reason 2033), and returns. The reply message arrives a short while later.

If the message is non persistent, it should have an expiry interval and will get removed automatically. No problem.
If the message is persistent with an expiry interval. This is not good practice. It should either be non persistent with expiry or persistent without expiry.
If the message does not have an expiry interval. You need a process to do things with the messages. Select messages older than a certain time, so you do not process the ones which are being processed. The action may be to create a compensating request, or to update a table to remove a “pending” flag in a record, and then to remove the message from the queue.

As a systems programmer or MQ administrator you do not just want to clear the queue – as it is not your data (in case you delete a queue with your pay cheque in it – or other important information). The application owner should have the responsibility for processing the data.

If you have the “Application Pro” you need to help them, by giving them a process to offload old messages.

MQ V8 provides a program dmpmqmsg which unloads (or loads) mq messages from a queue.

MQGEM has a similar product. See here. It has a superset of functions – for example dump all queues with a specified prefix, and create output file names with today’s date in it etc. Have a look – is is worth it!

You could use the IBM version as follows

dspmqmsg -m queue manager name
-I input queue name (upper case -I for get delete, lower case -i for get browse)
-f file name. This should be something like qm_name.Queue_name.Date_time
-T1:2:3 select messages older than 1 Day, 2 Hours, 3 Minutes

You need to protect the output file, to restrict who can view it. By default the permissions are -rw-r–r–.

You could encrypt it with

openssl enc -AES-256-CBC -in infile -out outfile.enc -pass pass:longpassword
rm infile

Zip with a password is better than nothing – but it is relatively insecure.

Changing permissions to “only owner has read” – still allows super users to read the file.

If any messages have been offloaded – you need to raise an alert against the development team a) to tell them about the problem and b) to remind them to get a proper process in place.

How big is your estate?

I was sent a question from someone who had read my blog, and this reminded me of a conversation I had with a fellow traveler whilst on holiday.

The lady spoke about her housing problem. At first I thought she meant, the roof leaked, or there were 10 people per room, but no, it was totally different. She had the house in Wimbledon, London; the flat in Marble Arch, London; the Manor house in the Cotswolds (the Cotswolds is a very pretty part of England, with old stone houses, and where many people ride horses for pleasure); and her husband had recently been made responsible for the shooting lodge and estate in Scotland. The housing problem was that she had been asked to look into the holiday cottages and see what could be done with them. She said that after a brief visit, she had found 9 out of the 10 properties, some were very old, some were very small. They paid tax on the houses even if they were empty. Should they knock down the worst houses and improve the remainder – for example put in better insulation and make them more energy efficient, or make two small cottages into one larger cottage.

The emailed question I had was something similar, she had been promoted and moved to a different part of the company and was now in change of MQ. She found there were no good records about the number of queue managers and what they were used for. The question was – how do you tell the size of the MQ estate – and if there was “old iron” which should be disposed of?

I heard that in California, the state said your data centre could use no more power than it uses today. So if you wanted to add a new server – you had to take out an old one, or take out an old server, and add two energy efficient servers.

What questions do you need to ask?

How many queue managers do you have
How much work are they doing?
Are they current?
Do you have licenses for them?
Can I get rid of any?

How many queue managers do I have?

Most enterprise customers have each machine set up with an agent which reports what software is installed on the machine, and stores the information in a central database.

This only reports on machines it knows about; if you have old machines in the dusty basement which no one knows about – they may not be in the database.

If you do not have this enterprise software running an agent, you can logon to the machine and issue

dspmqver to tell you level of software and the license type
dspmq to tell you which queue managers are configured, and if they are running

How much work are they doing

For the high level view you can use information provided by MQ

Messages on the SYSTEM.ADMIN.STATISTICS.QUEUE report information at the queue manager level such as number of puts, gets, bytes put etc. Your enterprise software may already be consuming these and may (or may not) be storing the information in a central repository. If so, you cannot get these messages.

In MQ V9 there are also statistics produced a pub/sub model which allow multiple programs to get statistics and accounting data. You subscribe to what you want – and they get published to your queue. This allows you to run in parallel with the enterprise tools.

Good metrics are

Number of puts per day. This includes MQPUT, MQPUT1 to a queue and publishes and puts using a topic Later on when you are looking at consolidating queue managers, you may want plots of messages put per hour. The number of gets may not be so useful, as you usually have gets which return no messages, and get with browse, both of which increase the overall number of gets.
Amount of data put per day. This is useful to get a measure of message size.

Are they current?

The enterprise tools running an agent on your machines should be able to tell you the versions of software running on the machine.

As described above, you can use dspmqver

You can also issue the Display ChannelStatus command, and this will report the level of MQ at the remote end of the channel, then try to find the remote end!

Do you have licenses for them?

Your enterprise tools running the agents on your machine should collect the information, or you can use the dspmqver command.

You need to check you are running the right license, for example not using a development license in production.

Can I get rid of any?

As Shakespeare’s Hamlet says “Ay, there’s the rub”, meaning this is a difficult problem.

Some things you need to think about

A server may be little used during most of the year – but very heavily used at year end – be it Christmas, or financial year. You may need to monitor for a year before deciding to decommission it
You may want consolidate several little used queue managers on old iron onto one newer server, to reduce the number of servers, and to use newer, energy efficient hardware.
Migrate old releases to newer releases of software.
The cost of the license depends on the size of the machine. If MQ is on a large machine and doing no work, consider moving it to a smaller machine, or reduce the capacity of the virtualised server.
If you plan to merge servers you need to ensure the new server has enough capacity, CPU, RAM, disk bandwidth and network bandwidth, and the disk and network have acceptable response times. Assume that all workloads will peak at the same time, to ensure you have enough peak capacity.