I was looking into a problem between two queue managers, and I thought I would use the stats and accounting to see what was going on.   These were pretty well useless.  This post should be subtitled, “why you need connection pooling in your web server,  for all the wrong reasons“.

As I walked along this morning I pictured a man in a big chair stroking a white car saying “For you Mr Bond, you will not know which applications used MQ, and you will not know what they did.  For you, your struggle to understand what is going on, is over”.

Which queue were used?

The first problem was that the stats and accounting only display the mapped queue name, not the opened name.

I had two cluster queues QC1 and QC2.  I opened them and put a few messages to each queue.  These two queues map down to the SYSTEM.CLUSTER.TRANSMIT.QUEUE – or what ever has been configured for the cluster.

In the accounting data, I had “Queue_Name” :SYSTEM.CLUSTER.TRANSMIT.QUEUE  opened twice.  There was no entry for QC1 nor for QC2.

It works the same way  if you have a QALIAS.  If you have queue QA pointing to QL.  When you open QA, the accounting record is for QL.

This applies to the statistics records as well.

Which transaction was it?

I had the transaction running as an MDB in my web server.

For my application logic in the onMessage method,  which processed the passed message and put the reply, I had

,”Appl_Name” :CF3Name  – this is the connection pool name used for putting the message
,”Process_Id” :9987
,”Thread_Id” :42
,”User_Identifier” :colinpaice
“Queue_Name” :SYSTEM.CLUSTER.TRANSMIT.QUEUE

For the listener side of the application,  I had

,”Appl_Name” :weblogic.Server
,”Process_Id” :9987
,”Thread_Id” :43
,”User_Identifier” :colinpaice
“Queue_Name” :JMSQIN

Where JMSQIN is the name of the queue the MDB was listening on.

They are both for the same process id (9987) but they are on different threads.

I could only find out what was running from the name of the input queue.

I could not easily tie the two lots of accounting data together.  Fortunately I had a dedicated connection pool for the MDB, so I could find it that way.  If I had different MDBs using the same connection pool, I would not be able to tell which MDB did which requests.

In someways this is expected.  If the listener is using a connection pool – there could be different MDBs serially using the same thread in the  connection pool, so it cannot give the name of the MDB, as the MDB name will change.

So to help you understand what is happening in your MQ applications in a web server, you should use connection pooling so you can identify which application issued the requests.  Connect pooling also gives you performance benefits.

The above question was raised by an MQ administrator who is not an MQ programmer, the second question was – is this a bug in MQ.

To application programmers the answers may appear be obvious, but has some subtleties.

1. If you use MQPUT1 to put a message within syncpoint, the queue is open for a very short time.
2. If you use MQOPEN, MQPUT to put the message within sycnpoint ,…. do some work MQCLOSE

You should then do a commit to make it happen.

If the application does not do a commit, but returns,  and the thread ends, the queue manager will do a commit or backout ( depending on a platform) as part of end of task clean up so you should not get an uncommitted unit of work.

So why is there an uncommitted message after the application “ends”.

The transaction was running on a web server, and the connections were being reused, and so in concept the threads run until the web server is shut down.

If the application is running as a Message Driven Bean, there is logic which does

1. The message listener waits for a message and gets it.
2. The message is passed to the onMessage method of the application
3. The application puts  a message and returns.
4. Being an MDB there is an MQ Commit which commits both the get and the update.

In this case the message will be committed, and you do not get an uncommitted message.

Hmmm , the plot thickens…

It looks like there was a java application which was not an MDB.   The queue name had “BACKOUT.QUEUE” as part of the name.

I expect there was an MQGET and there was a problem with the message, or there was a problem doing an MQPUT, and, in either case,  the message was written to the backout queue.  I am guessing that this is in error handling code which very rarely gets used, it was unlikely to have been tested properly.

The application needs to put in an explicit commit statement.

What commands are there to help me?

On my test system I paused my application after the close, and before the commit.

DIS QSTATUS(COLINMON) gave me

dis qstatus(COLINMON) ALL 

AMQ8450I: Display queue status details.
QUEUE(COLINMON) TYPE(QUEUE) CURDEPTH(1048) IPPROCS(0)  OPPROCS(0) UNCOM(1)

So no instances with the queue open, but one uncommitted message.

Use the dspmqtrn command

dspmqtrn -i -m QMA
There are no matching prepared or heuristically completed transactions.

dspmqtrn -e -m QMA
There are no matching prepared or heuristically completed transactions.

These commands display the indoubt status.  You only get indoubt when there are two resource managers involved for example MQ and DB2.

dspmqtrn -q -a -m QMA> aa

-a means display all transactions

-q mesage display the first 100 objects used by each transactions

In the aa file ( I searched for COLINMON)  was

TranNum(0,24589)
TRANSTATE(ACTIVE)
UOWLOGDA( ) UOWLOGTI( )
UOWSTDA(2019-12-10) UOWSTTI(08.48.37)
UOWLOG( )
EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
CONN(5F51EF5D045CDE24)
PID(9380) TID(8)
APPLTAG(jpcf/myprog)
APPLDESC(IBM MQ Channel)
CHANNEL(QMACLIENT)
CONNAME(127.0.0.1)
QMURID(0.24589)
USERID(colinpaice)
OBJECT(COLINMON )

The QMURID(0.24589) and TranNum(0,24589) refer to the same value, the position in the log.

Using the CONN information

dis CONN(5F51EF5D045CDE24) all 

AMQ8276I: Display Connection details.
CONN(5F51EF5D045CDE24) 
EXTCONN(414D5143514D41202020202020202020)
TYPE(CONN) 
PID(9380) TID(8) 
APPLDESC(IBM MQ Channel) APPLTAG(jpcf/myprog)
APPLTYPE(USER) ASTATE(NONE)
CHANNEL(QMACLIENT) CLIENTID( )
CONNAME(127.0.0.1) 
CONNOPTS(MQCNO_SHARED_BINDING,MQCNO_GENERATE_CONN_TAG)
USERID(colinpaice) UOWLOG( )
UOWSTDA(2019-12-10) UOWSTTI(08.48.37)
UOWLOGDA( ) UOWLOGTI( )
URTYPE(QMGR) 
EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
QMURID(0.24589) UOWSTATE(ACTIVE)
CONNTAG(MQCT5F51EF5D045CDE24QMA_2018-08-16_13.32.14jpcf/myprog)

you could also have issued

DIS CONN(*) all WHERE(QMURID,EQ,0.24589)

When the web server was used, the APPLTAG type info was “WebLogic” instead of jpcf/myprog.  All I could tell was which web server had been running the transaction.

UOWSTDA(2019-12-10) UOWSTTI(08.48.37) tells the start date and time of the unit of work.
UOWLOGDA( ) UOWLOGTI( ) tells the start date and time the connected first wrote to the log.    As there is no date and time, there was no write to the log, so in this case the message was non persistent.

What a lot of information from a seemingly obvious question!   Thanks to Gwydion for help with some of the trickier bits of this.

I was looking into a performance problem where a web server was doing 1 million MQCONNects  day!  During my investigations I found that the original designers did not design it properly because of the performance overhead, and so the JMS architects had to fix it up by creating connection factories.

Below, I cover

• the performance problem
• writing my own connection factory
• the problems of writing my own connection factory
• it might just be easier to use a connection factory provided by your web server or JMS provider.

What is an EJB and an MDB?

In a web server you can have Enterprise Java Beans (EJBs) which are a package of java applications doing enterprise type work ( get a message, update a database, send a reply) conforming to an specific Programming Interface.

A Message Driven Bean is an EJB which responds to messages.  A message listener applications gets messages from a queue – and passes the message to the MDB. In non EJB terms this is just like a trigger monitor, starting a transaction and passing the data.

When you create the MDB you have a configuration file  which allows a message listener to be created, and the parameters to use to connect to MQ, and  how many threads etc can be running concurrently.

The MDB, to process the message has a basic structure of

• method onMessage – this is given the message
• ejbCreate – when the EJB is created, you can do initialisation here
• ejbRemove – when the EJB is about to be deleted, you can do clean up here

and looks like

public class MDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -8070254332864574796L;
  public void onMessage(Message message)
  {
    Connection connection = null;
    Session session = null;
    MessageProducer producer = null;

    try
    {
      InitialContext ctx = new InitialContext();
      ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF2");
      Destination dest = message.getJMSReplyTo();
     
      connection = cf.createConnection();
      session = connection.createSession(false, 1);
      producer = session.createProducer(dest);
      
      TextMessage response = session.createTextMessage("test response message from the WMQ_IVT_MDB");
      response.setJMSCorrelationID(message.getJMSMessageID());
      producer.send(response);
      return;
    }
    catch (Exception je)
    {
      System.out.println("Something went wrong." + je);
    }
  }

  public void ejbCreate() { }
  public void ejbRemove() { }
 }

Within the onMessage method is logic connection = cf.createConnection();
Under the covers this createConnection() does an MQCONN.  With 1 million messages a day – this is 1 million MQCONNs!
There was also an MQOPEN, an MQCLOSE,  and an MQDISC before the onMessage() method returned.  This results in a huge performance overhead.

Attempts to fix the performance

People quickly found that this model was not efficient, and they came up with ways to improve the performance.

The idea of a connection pool was developed.  Instead of doing an the createConnection() doing an MQCONN etc.  this code was changed, to not do the MQDISC, but to keep the Queue Manager handle. When the next MQCONN occurred, if there was one of these spare Queue Manager handles it would use it.

These connection pools are sophisticated.   For example they can free up connections if they had not been used for a time period, so reducing the overall number of Queue Manager connections in use, they can slowly increase the number of connections in the pool so there is not a sudden spike in requests.

I found it difficult to set up a connection pooling in Oracle’s webLogic webserver,  so I thought I would write my own.   This was pretty easy, but then I discovered some complications.

Writing my own “connection pooling”.

It was clear to me that having connection = cf.createConnection();     from the ejbCreate() method rather than in the onMessage() method would be much more efficient.

• Move variables from being method variables to instance variables so they persisted across onMessage() calls.
• Create a connect() method which actually does the createConnection().  This can be called in the ejbCreate setup routine, and from the onMessage() if the connection variable is null.
• Have code in the ejbRemove to the connection, session etc.

public class IVTMDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -337338331639L;
  // create long lasting instance variables 
  Connection connection = null;
  Session session = null;
  MessageProducer producer = null;
  InitialContext ctx = null;
  ConnectionFactory cf = null;
  // new method to do the connect
  public void connect(){
  try {
     ctx = new InitialContext();
     cf = (ConnectionFactory)ctx.lookup("CF2");
     connection = cf.createConnection();
     session = connection.createSession(false, 1); 
    }
    catch (Exception je)
    {...
    } 
 }

 public void ejbCreate() {
   connect();
 }
...
}

This worked well.    The number of MQCONN dropped to about 10 per day for many puts.

The problem looked solved, until I tried to shut down the queue manager.

If no work was being processed, then the MDBs were not being called, and so there were no MQ requests being made, and so the connections stayed active, and the queue manager did not shut down.

To solve this you can set  cf.setExceptionListener(..) which will get notified if there are problems with the connection, and so you can close the connection etc.
You need to consider what to do if the connection cannot be made;  do you wait for a short time period, or do you return an error.

I found that this “simple” way of avoiding all of the MQCONNects was quickly getting much more complex.  It was going to be much easier in the long term to get the provided connection factories working.   This was another challenge, taking about a week.  See the next few blog entries on how I did this for webLogic.