Should all red flags be green?

This question came out of a discussion on an MQ forum, where the question was if MQ does one time delivery, how come he got the same message twice?

Different sorts of MQ gets.

There are a variety of patterns for getting a message.

  • Destructive get out of sync-point. One application can get the message. It is removed from the system. As part of the MQGET logic there is a commit of the get so once it has gone it has gone. This is usually used for non persistent message. Persistent messages are usually processed within sync-point, but there are valid cases when the get of a persistent out of sync-point is valid.
  • Destructive get within sync-point. One application can get the message. The queue manger holds a lock on the message which makes it invisible to other applications. When the commit is issued (either explicitly or implicitly) , the message is deleted. If the application rolls back (either implicitly of explicitly) the message becomes visible on the queue again, and the lock released.
  • Browse. One or more applications can get the message when using the get-with-browse option. Sync-point does not come into the picture, because there are no changes to the message.
  • One problem with get-with-browse is you can have many application instances browsing the queue, and they may do the same work on a message, wasting resources. To help with this, there is cooperative browse. This is effectively browse and hide. This allows a queue monitor application to browse the message, and start a transaction saying process “this” message. A second instance of the queue monitor will not see the message. If the message has not been got within a specified time interval the “hide” is removed, and so the message becomes visible. See Avoiding repeated delivery of browsed messages.

The customer’s question was, that as the get was destructive, how come the message was processed twice – could this be a bug in MQ?

The careful reader may have spotted why a message can be got twice.

Why the message was processed “twice”.

Consider an application which does the following

MQGET destructive, in sync-point

Write “processed message id …. ” to the log
Update DB2 record
Commit

You might the see following in the log

processed message id x’aabbccdd01′.
processed message id .x’aabbccdd02′.
processed message id x’eeffccdd17′. .

Expanding the transaction to give more details

MQGET destructive, in sync-point
Write “processed message id …. ” to the log
Update DB2 record

If DB2 update worked then commit
else backout

If there was a DB2 problem, you could get the following on the log:

processed message id x’aabbccdd01′.
processed message id x’aabbccdd01′.
processed message id x’eeffccdd17′. .

You then say “Ah Ha – MQ delivered the message twice”. Which is true, but you should be saying “Ah Ha – MQ delivered the message but the application didn’t want it. The second time MQ delivered it, the application processed it”. Perhaps change the MQ phrase to “MQ does one time successful delivery“.

Why is this blog post called Should all red flags be green?

A proper programmer (compared to a coder), will treat a non successful transaction as a red flag, and take an action because it is an abnormal situation. For example write a message to an error log

  1. Transaction ABCD rolled back because “DB2 deadlock on ACCOUNTS table”
  2. Transaction ABCD rolled back because “MQ PUT to REPLYQUEUE failed – queue full”
  3. Transaction ABCD rolled back because “CICS is shutting down”

The Architects and systems programmers can look at these messages and take action.

For example with DB2, investigate the lock held duration. Can you reduce the time the lock is held, perhaps by rearranging with within a unit of work, for example “MQGET, MQPUT reply, DB2 update, commit” instead of “MQGET, DB2 update, MQPUT of reply, commit.

For MQ queue full, make the maximum queue depth bigger, or find out why the queue wasn’t being drained.

CICS shutting down.You may always get some reasons to rollback.

Once you have put an action plan in place to reduce the number of red flags, you can mark the action item complete, change its status from red to green and keep the project managers happy (who love green closed action items).

Note: This may be a never ending task!

After thought

In the online discussion, Morag pointed out that perhaps the same message was put twice. Which would show the same symptoms. This could have been due to a put out of syncpoint, and the transaction rolled back.

One thought on “Should all red flags be green?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s