Midrange accounting and stats are sometimes pretty useless

I was looking into a problem between two queue managers, and I thought I would use the stats and accounting to see what was going on.   These were pretty well useless.  This post should be subtitled, “why you need connection pooling in your web server,  for all the wrong reasons“.

As I walked along this morning I pictured a man in a big chair stroking a white car saying “For you Mr Bond, you will not know which applications used MQ, and you will not know what they did.  For you, your struggle to understand what is going on, is over”.

Which queue were used?

The first problem was that the stats and accounting only display the mapped queue name, not the opened name.

I had two cluster queues QC1 and QC2.  I opened them and put a few messages to each queue.  These two queues map down to the SYSTEM.CLUSTER.TRANSMIT.QUEUE – or what ever has been configured for the cluster.

In the accounting data, I had “Queue_Name” :SYSTEM.CLUSTER.TRANSMIT.QUEUE  opened twice.  There was no entry for QC1 nor for QC2.

It works the same way  if you have a QALIAS.  If you have queue QA pointing to QL.  When you open QA, the accounting record is for QL.

This applies to the statistics records as well.

Which transaction was it?

I had the transaction running as an MDB in my web server.

For my application logic in the onMessage method,  which processed the passed message and put the reply, I had

,”Appl_Name” :CF3Name  – this is the connection pool name used for putting the message
,”Process_Id” :9987
,”Thread_Id” :42
,”User_Identifier” :colinpaice

For the listener side of the application,  I had

,”Appl_Name” :weblogic.Server
,”Process_Id” :9987
,”Thread_Id” :43
,”User_Identifier” :colinpaice
“Queue_Name” :JMSQIN

Where JMSQIN is the name of the queue the MDB was listening on.

They are both for the same process id (9987) but they are on different threads.

I could only find out what was running from the name of the input queue.

I could not easily tie the two lots of accounting data together.  Fortunately I had a dedicated connection pool for the MDB, so I could find it that way.  If I had different MDBs using the same connection pool, I would not be able to tell which MDB did which requests.

In someways this is expected.  If the listener is using a connection pool – there could be different MDBs serially using the same thread in the  connection pool, so it cannot give the name of the MDB, as the MDB name will change.

So to help you understand what is happening in your MQ applications in a web server, you should use connection pooling so you can identify which application issued the requests.  Connect pooling also gives you performance benefits.


I have an uncommitted message on the queue – but no one has the queue open. Which application is causing the problem?

The above question was raised by an MQ administrator who is not an MQ programmer, the second question was – is this a bug in MQ.

To application programmers the answers may appear be obvious, but has some subtleties.

  1. If you use MQPUT1 to put a message within syncpoint, the queue is open for a very short time.
  2. If you use MQOPEN, MQPUT to put the message within sycnpoint ,…. do some work MQCLOSE

You should then do a commit to make it happen.

If the application does not do a commit, but returns,  and the thread ends, the queue manager will do a commit or backout ( depending on a platform) as part of end of task clean up so you should not get an uncommitted unit of work.

So why is there an uncommitted message after the application “ends”.

The transaction was running on a web server, and the connections were being reused, and so in concept the threads run until the web server is shut down.

If the application is running as a Message Driven Bean, there is logic which does

  1. The message listener waits for a message and gets it.
  2. The message is passed to the onMessage method of the application
  3. The application puts  a message and returns.
  4. Being an MDB there is an MQ Commit which commits both the get and the update.

In this case the message will be committed, and you do not get an uncommitted message.

Hmmm , the plot thickens…

It looks like there was a java application which was not an MDB.   The queue name had “BACKOUT.QUEUE” as part of the name.

I expect there was an MQGET and there was a problem with the message, or there was a problem doing an MQPUT, and, in either case,  the message was written to the backout queue.  I am guessing that this is in error handling code which very rarely gets used, it was unlikely to have been tested properly.

The application needs to put in an explicit commit statement.

What commands are there to help me?

On my test system I paused my application after the close, and before the commit.


dis qstatus(COLINMON) ALL 

AMQ8450I: Display queue status details.


So no instances with the queue open, but one uncommitted message.

Use the dspmqtrn command

dspmqtrn -i -m QMA
There are no matching prepared or heuristically completed transactions.

dspmqtrn -e -m QMA
There are no matching prepared or heuristically completed transactions.

These commands display the indoubt status.  You only get indoubt when there are two resource managers involved for example MQ and DB2.

dspmqtrn -q -a -m QMA> aa

-a means display all transactions

-q mesage display the first 100 objects used by each transactions

In the aa file ( I searched for COLINMON)  was

UOWSTDA(2019-12-10) UOWSTTI(08.48.37)
PID(9380) TID(8)


The QMURID(0.24589) and TranNum(0,24589) refer to the same value, the position in the log.

Using the CONN information

dis CONN(5F51EF5D045CDE24) all 

AMQ8276I: Display Connection details.
PID(9380) TID(8) 
APPLDESC(IBM MQ Channel) APPLTAG(jpcf/myprog)
USERID(colinpaice) UOWLOG( )
UOWSTDA(2019-12-10) UOWSTTI(08.48.37)

you could also have issued

DIS CONN(*) all WHERE(QMURID,EQ,0.24589)


When the web server was used, the APPLTAG type info was “WebLogic” instead of jpcf/myprog.  All I could tell was which web server had been running the transaction.

UOWSTDA(2019-12-10) UOWSTTI(08.48.37) tells the start date and time of the unit of work.
UOWLOGDA( ) UOWLOGTI( ) tells the start date and time the connected first wrote to the log.    As there is no date and time, there was no write to the log, so in this case the message was non persistent.


What a lot of information from a seemingly obvious question!   Thanks to Gwydion for help with some of the trickier bits of this.

Should I make my web server Highly Available ?

If you are using XA on midrange  then connect to just one queue manager, because  anything else is not supported.  If your connection goes to a different queue manager, then it cannot recover a unit of work.
The only exception to that rule is if you are connecting to a QSG on z/OS. When using a QSG, all of the queue managers in the QSG can resolve each others transactions – this means that the transaction recovery issue goes away.
Be careful using multiple queue managers.
For the listener you want to connect to just one queue manager.  If you have a choice of two queue managers, all threads may connect to QM1, and there be no connection to QM2.  If you have a second web server instance, this may also connect to QM1, and not QM2.  It is better to have WS1 connect to QM1 and WS2 to connect to WS2 – so called production lines.
If you need to shut down QM1, then shut down WS1,  do not try to cross wire it
For the MDB logic connection factory if you had two possible queue managers, you may get a spread of connections, from all on QM1, to all on QM2, or a mixture.
If the WS frees connections (for example they are idle), then when there is a load needing more connections, it could get a connection from QM1 or QM2.
All in all – keep it simple,  and have each web server connect to just one queue manager.


Whoops -deploying MDB in weblogic

I was quite happily using my MDB in webLogic, but when I changed its configuration, it did not pick up the new changes.  It took a day to find out why,  and I have learned much more about deploying MDBS.

My connection factory was using SYSTEM.DEF.SVRCONN, I changed it to use a different client channel. I stopped SYSTEM.DEF.SVRCONN, ( so I could check the change had worked), and restarted the webLogic instance.  I was surprised when my MDB failed to start, because the channel was stopped.   The MDB was trying to use that channel.  It took a lot of head scratching to get it to work as I expected.

  • I had messages like <BEA-015073>  Message-Driven Bean …  is configured with unknown activation-config-property name failIfQuiesce.  This message is wrong, failIfQuiesce is supported by the IBM Resource adapter.
  • I had the same message with activation-config-property name cfLookup.   This was my problem.  I should have specified connectionFactoryLookup.
  • If you have <activation-config-property-name>connectionFactoryLookup… (specified in the ejb-jar) any other parameters you specify in the ejb-jar.xml file are ignored.
  • If you do not specify a connectionFactoryLookup, nor properties in the ejb-jar.xml file, defaults are provided, see Configuring the resource adapter for inbound communication.  In my case I had not specified  activation-config-property-name channel, and this defaulted to SYSTEM.DEF.SVRCONN, which is why it continued to use that channel.
  • It worth putting <activation-config-property-name>applicationName … in your definitions so you can see what you are using.
    • dis qstatus(JMSQ3) type(handle) gave me APPLTAG(CF3Name) so I can tell which definitions are being used.
    • If you get APPLTAG(weblogic.Server) then you are taking the defaults.
  • The Oracle documentation  says the precedence order is as below.    I do not think this is 100% accurate. (I could not specify some of the parameters on the weblogic-ejb-jar.xml file).  I didnt try the java program.
    1. properties set in the weblogic-ejb-jar.xml deployment descriptor
    2. activation-config-property properties set in the ejb-jar.xml deployment descriptor
    3. activationConfigProperty annotation properties in the java program.

What do I need to specify?

As a minimum you need to use connectionFactoryLookup or  specify

  1. applicationName – so you can identify which definitions are being used
  2. channel – which channel to use
  3. failIfQuiesce
  4. hostName
  5. port


The ejb-jar.xml file is in the META-INF directory.  Change  the ejb-jar.xml or  weblogic-ejb-jar.xml file. IUpdate the jar file using a command like jar -uvf MDB4.jar  META-INF/ejb-jar.xml,   and redeploy it.

Did the JMS architects get it wrong? Possibly!

I was looking into a performance problem where a web server was doing 1 million MQCONNects  day!  During my investigations I found that the original designers did not design it properly because of the performance overhead, and so the JMS architects had to fix it up by creating connection factories.

Below, I cover

  • the performance problem
  • writing my own connection factory
  • the problems of writing my own connection factory
  • it might just be easier to use a connection factory provided by your web server or JMS provider.

What is an EJB and an MDB?

In a web server you can have Enterprise Java Beans (EJBs) which are a package of java applications doing enterprise type work ( get a message, update a database, send a reply) conforming to an specific Programming Interface.

A Message Driven Bean is an EJB which responds to messages.  A message listener applications gets messages from a queue – and passes the message to the MDB. In non EJB terms this is just like a trigger monitor, starting a transaction and passing the data.

When you create the MDB you have a configuration file  which allows a message listener to be created, and the parameters to use to connect to MQ, and  how many threads etc can be running concurrently.

The MDB, to process the message has a basic structure of

  • method onMessage – this is given the message
  • ejbCreate – when the EJB is created, you can do initialisation here
  • ejbRemove – when the EJB is about to be deleted, you can do clean up here

and looks like

public class MDB
implements MessageListener, MessageDrivenBean
  private static final long serialVersionUID = -8070254332864574796L;
  public void onMessage(Message message)
    Connection connection = null;
    Session session = null;
    MessageProducer producer = null;

      InitialContext ctx = new InitialContext();
      ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF2");
      Destination dest = message.getJMSReplyTo();
      connection = cf.createConnection();
      session = connection.createSession(false, 1);
      producer = session.createProducer(dest);
      TextMessage response = session.createTextMessage("test response message from the WMQ_IVT_MDB");
    catch (Exception je)
      System.out.println("Something went wrong." + je);

  public void ejbCreate() { }
  public void ejbRemove() { }

Within the onMessage method is logic connection = cf.createConnection();
Under the covers this createConnection() does an MQCONN.  With 1 million messages a day – this is 1 million MQCONNs!
There was also an MQOPEN, an MQCLOSE,  and an MQDISC before the onMessage() method returned.  This results in a huge performance overhead.

Attempts to fix the performance

People quickly found that this model was not efficient, and they came up with ways to improve the performance.

The idea of a connection pool was developed.  Instead of doing an the createConnection() doing an MQCONN etc.  this code was changed, to not do the MQDISC, but to keep the Queue Manager handle. When the next MQCONN occurred, if there was one of these spare Queue Manager handles it would use it.

These connection pools are sophisticated.   For example they can free up connections if they had not been used for a time period, so reducing the overall number of Queue Manager connections in use, they can slowly increase the number of connections in the pool so there is not a sudden spike in requests.

I found it difficult to set up a connection pooling in Oracle’s webLogic webserver,  so I thought I would write my own.   This was pretty easy, but then I discovered some complications.

Writing my own “connection pooling”.

It was clear to me that having connection = cf.createConnection();     from the ejbCreate() method rather than in the onMessage() method would be much more efficient.

  • Move variables from being method variables to instance variables so they persisted across onMessage() calls.
  • Create a connect() method which actually does the createConnection().  This can be called in the ejbCreate setup routine, and from the onMessage() if the connection variable is null.
  • Have code in the ejbRemove to the connection, session etc.
public class IVTMDB
implements MessageListener, MessageDrivenBean
  private static final long serialVersionUID = -337338331639L;
  // create long lasting instance variables 
  Connection connection = null;
  Session session = null;
  MessageProducer producer = null;
  InitialContext ctx = null;
  ConnectionFactory cf = null;
  // new method to do the connect
  public void connect(){
  try {
     ctx = new InitialContext();
     cf = (ConnectionFactory)ctx.lookup("CF2");
     connection = cf.createConnection();
     session = connection.createSession(false, 1); 
    catch (Exception je)

 public void ejbCreate() {

This worked well.    The number of MQCONN dropped to about 10 per day for many puts.

The problem looked solved, until I tried to shut down the queue manager.

If no work was being processed, then the MDBs were not being called, and so there were no MQ requests being made, and so the connections stayed active, and the queue manager did not shut down.

To solve this you can set  cf.setExceptionListener(..) which will get notified if there are problems with the connection, and so you can close the connection etc.
You need to consider what to do if the connection cannot be made;  do you wait for a short time period, or do you return an error.

I found that this “simple” way of avoiding all of the MQCONNects was quickly getting much more complex.  It was going to be much easier in the long term to get the provided connection factories working.   This was another challenge, taking about a week.  See the next few blog entries on how I did this for webLogic.

MDBs activation specs and @things in the java program

While struggling with getting MDBs working, and looking at examples, I saw examples where they defined JMS resources within the java program using @….  statements, and could not see how they worked.  These are called annotations. The documentation on the web assumes you know about annotations  when explaining annotations!  They, in fact, are pretty simple, let me explain.

Annotations start with an @ character, and the information can be stored within the .class file as meta-data.  Programs can extract and use this meta data.

You can have java code like
private javax.jms... myMQ;

A program, for example,  your program, an analysis program or a web server, can issue request like

  • load class information
  • from the meta data list all fields with @resource defined.
  • do things with the list

One example would be to specify a JNDI lookup of java:customerMQ and return it into the field myMQ.

Another example from the IBM documentation

  name = "JMSSampleMDB",
  activationConfig = 
       propertyName  = "destinationType", 
       propertyValue = "javax.jms.Queue"),
       propertyName  = "destination", 
	propertyValue = "jndi_INPUT_Q")                         

The resource adapter has code which does

  • load your MDB program
  • get the MessageDriven stuff.
    • within this, locate the activationConfig records
      • within these, locate the ActivationConfigProperty propertyName and propertyValue, and merge the data with the data in the ejb-jar.xml file.


With the definitions in your java program, and the definitions in the MDB configuration you can configure a complete set of options for MDB.  I think the definitions in the java program override the MDB configuration.

How do I see what data there is?

You can extract this meta-data using a method like (see here)

public void getAnnotations(Class inclass){
    for(Field field : inclass.getDeclaredFields()){
        Class type = field.getType();
        String name = field.getName();
        field.getDeclaredAnnotations(); //do something with these

Use the javap command to display the data.

To display the annotations you can usethe command, where ….class is the name of your class file.

javap -v .....class

My java program had

import javax.annotation.Resource;
@Resource(lookup = "java:app/jms/myappTopic")
String colin = "ZZZZZ";

The javap command gave

java.lang.String colin;
  descriptor: Ljava/lang/String;
  0: #14(#15=s#16)
#14 = Utf8 Ljavax/annotation/Resource;
#15 = Utf8 lookup
#16 = Utf8 java:app/jms/myappTopic

from which we get

java.lang.String colin ... 
  javax/annotation/Resource (lookup = java:app/jms/myappTopic).

which matches the source code.

Different annotation types are confusing.

As well as providing meta-information on variables and classes, java also uses annotations to modify the java compiler behaviour.   For example

  • By putting @Deprecated infront of a method, the method can be flagged when used, as deprecated, and you should not use it
  • @SuppressWarnings(“unchecked”) tells the java compiler NOT to produce an error message for the unchecked condition.  See here for a list of warning conditions.