Featured

When is activity trace enabled?

I found the documentation for activity trace was not clear as to the activity trace settings.

In mqat.ini you can provide information as to what applications (including channels) you want traced.

For example

applicationTrace:
ApplClass=USER
ApplName=progput
Trace=OFF

This file and trace value are checked when the application connects.  If you have TRACE=ON when the application connects, and you change it to TRACE=OFF, it will continue tracing.

If you have TRACE=OFF specified, and the application connects, changing it to TRACE=ON will not produce any records.

With

  • TRACE=ON, the application will be traced
  • TRACE=OFF the application will not be traced
  • TRACE= or omitted then the tracing depends on alter qmgr ACTVTRC(ON|OFF).   For a long running transaction using alter qmgr to turn it on, and then off, you will get trace records for the application from in the gap.

If you have

applicationTrace:
ApplClass=USER 
ApplName=prog* 
Trace=OFF

applicationTrace:
ApplClass=USER 
ApplName=progp*
Trace=ON

then program progput will have trace turned on because the definition is more specific.

You could have

applicationTrace:
ApplClass=USER 
ApplName=progzzz
Trace=OFF

applicationTrace:
ApplClass=USER 
ApplName=prog*
Trace=

to  be able to turn trace on for all programs beginning with prog, but not to trace progzzz.

 

Thanks to Morag of MQGEM  who got in contact with me, and said  long running tasks are notified of a change to the mqat.ini file, if the file has changed, and a queue manager attributed has been changed – even if it is changed to the same variable.

This and lots of other great info about activity trace (a whole presentation’s worth of information) is available here.

Whoops -deploying MDB in weblogic

I was quite happily using my MDB in webLogic, but when I changed its configuration, it did not pick up the new changes.  It took a day to find out why,  and I have learned much more about deploying MDBS.

My connection factory was using SYSTEM.DEF.SVRCONN, I changed it to use a different client channel. I stopped SYSTEM.DEF.SVRCONN, ( so I could check the change had worked), and restarted the webLogic instance.  I was surprised when my MDB failed to start, because the channel was stopped.   The MDB was trying to use that channel.  It took a lot of head scratching to get it to work as I expected.

  • I had messages like <BEA-015073>  Message-Driven Bean …  is configured with unknown activation-config-property name failIfQuiesce.  This message is wrong, failIfQuiesce is supported by the IBM Resource adapter.
  • I had the same message with activation-config-property name cfLookup.   This was my problem.  I should have specified connectionFactoryLookup.
  • If you have <activation-config-property-name>connectionFactoryLookup… (specified in the ejb-jar) any other parameters you specify in the ejb-jar.xml file are ignored.
  • If you do not specify a connectionFactoryLookup, nor properties in the ejb-jar.xml file, defaults are provided, see Configuring the resource adapter for inbound communication.  In my case I had not specified  activation-config-property-name channel, and this defaulted to SYSTEM.DEF.SVRCONN, which is why it continued to use that channel.
  • It worth putting <activation-config-property-name>applicationName … in your definitions so you can see what you are using.
    • dis qstatus(JMSQ3) type(handle) gave me APPLTAG(CF3Name) so I can tell which definitions are being used.
    • If you get APPLTAG(weblogic.Server) then you are taking the defaults.
  • The Oracle documentation  says the precedence order is as below.    I do not think this is 100% accurate. (I could not specify some of the parameters on the weblogic-ejb-jar.xml file).  I didnt try the java program.
    1. properties set in the weblogic-ejb-jar.xml deployment descriptor
    2. activation-config-property properties set in the ejb-jar.xml deployment descriptor
    3. activationConfigProperty annotation properties in the java program.

What do I need to specify?

As a minimum you need to use connectionFactoryLookup or  specify

  1. applicationName – so you can identify which definitions are being used
  2. channel – which channel to use
  3. failIfQuiesce
  4. hostName
  5. port

 

The ejb-jar.xml file is in the META-INF directory.  Change  the ejb-jar.xml or  weblogic-ejb-jar.xml file. IUpdate the jar file using a command like jar -uvf MDB4.jar  META-INF/ejb-jar.xml,   and redeploy it.

How do I get a client to disconnect?

I had a question from a customer who asked how they can reduce the number of client connections in use.  They had tried setting a disconnect interval (DISCINT) on the channel, but the connections were like weeds – you kill them off, and they grow back again.

DISCINT is “the length of time after which a channel closes down, if no message arrives during that period”.  This sounds perfect for most people.   The application is in an MQGET, and if no messages arrive, the channel can be disconnected, and the application gets connection broken.   The application can then decide to disconnect or reconnect.
If the application is not in an MQGET, then it will get notified of the broken connection next time it tries to use MQ.

Independent applications

Many applications are well written in that when they get Connection Broken, they just reconnect again, and so the DISCINT has no effect on reducing the number of connections. This may be good for availability but not for resource usage.   It may be good to have 1000 application instances running the day, but perhaps not overnight when there is no work to do.   Ive seen instances where the applications do an MQGET every minute, and with 1000 instances this can use a lot of CPU and doing no useful work.  In this case you want unused application instances to stop, and be restarted when needed.

You cannot use triggering with client connections (unless you have a very smart trigger monitor to produce an event which says start a client program over there).

Use automation periodically check the queue depth, and number of input handles. If there is a high queue depth, or a low number of handles(eg 2)  then start more application instances, across your back-end servers.  Your applications can then disconnect if they have not received a message within say 10 minutes.  This should keep the right number of application instances active.

An administrator should be able to get this automation set up, but getting the application to connect could be a challenge, as this requires the application developer to change the code!

Running under a web server

If your applications are running under a web server you may have mis-configured connection pools.  You can specify the initial size of the pool, and this many connections are made.  As more connections are needed, then more can be added to the pool until the pool maximum is reached. You should specify a time out value, so periodically the pool gets cleaned up, and unused connections are removed, until the pool is back to the initial size.  You should review the initial size of the pools ( is it too large), and the value of the time out value.

This should just be an administrative change.

Good luck, you may be successful in reducing the number of client connections, but do not set your hopes too high.

WebLogic message does not have authorization to view the logs

I was using JMX to display the connectionPool statistics  from webLogic, and kept getting messages

<Error> <Diagnostics> <BEA-320084> <The user principals=[] does not have authorization to view the logs.>

I solved this by using the webLogic console

  • Click “Security Realms” in the Domain Stucture box on the left hand side of the home page
  • Click on the name of your realm (myrealm)
  • Click on “Roles and Policies”
  • Click on “Realm Policies”
  • Expand Domain and select “View Policy Conditions” for “View Log”
  • Click on Add Condition
  • Select the role
  • Click on Finish
  • Click on Save

The change is available immediately on “Save”.

Did the JMS architects get it wrong? Possibly!

I was looking into a performance problem where a web server was doing 1 million MQCONNects  day!  During my investigations I found that the original designers did not design it properly because of the performance overhead, and so the JMS architects had to fix it up by creating connection factories.

Below, I cover

  • the performance problem
  • writing my own connection factory
  • the problems of writing my own connection factory
  • it might just be easier to use a connection factory provided by your web server or JMS provider.

What is an EJB and an MDB?

In a web server you can have Enterprise Java Beans (EJBs) which are a package of java applications doing enterprise type work ( get a message, update a database, send a reply) conforming to an specific Programming Interface.

A Message Driven Bean is an EJB which responds to messages.  A message listener applications gets messages from a queue – and passes the message to the MDB. In non EJB terms this is just like a trigger monitor, starting a transaction and passing the data.

When you create the MDB you have a configuration file  which allows a message listener to be created, and the parameters to use to connect to MQ, and  how many threads etc can be running concurrently.

The MDB, to process the message has a basic structure of

  • method onMessage – this is given the message
  • ejbCreate – when the EJB is created, you can do initialisation here
  • ejbRemove – when the EJB is about to be deleted, you can do clean up here

and looks like

public class MDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -8070254332864574796L;
  public void onMessage(Message message)
  {
    Connection connection = null;
    Session session = null;
    MessageProducer producer = null;

    try
    {
      InitialContext ctx = new InitialContext();
      ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF2");
      Destination dest = message.getJMSReplyTo();
     
      connection = cf.createConnection();
      session = connection.createSession(false, 1);
      producer = session.createProducer(dest);
      
      TextMessage response = session.createTextMessage("test response message from the WMQ_IVT_MDB");
      response.setJMSCorrelationID(message.getJMSMessageID());
      producer.send(response);
      return;
    }
    catch (Exception je)
    {
      System.out.println("Something went wrong." + je);
    }
  }

  public void ejbCreate() { }
  public void ejbRemove() { }
 }

Within the onMessage method is logic connection = cf.createConnection();
Under the covers this createConnection() does an MQCONN.  With 1 million messages a day – this is 1 million MQCONNs!
There was also an MQOPEN, an MQCLOSE,  and an MQDISC before the onMessage() method returned.  This results in a huge performance overhead.

Attempts to fix the performance

People quickly found that this model was not efficient, and they came up with ways to improve the performance.

The idea of a connection pool was developed.  Instead of doing an the createConnection() doing an MQCONN etc.  this code was changed, to not do the MQDISC, but to keep the Queue Manager handle. When the next MQCONN occurred, if there was one of these spare Queue Manager handles it would use it.

These connection pools are sophisticated.   For example they can free up connections if they had not been used for a time period, so reducing the overall number of Queue Manager connections in use, they can slowly increase the number of connections in the pool so there is not a sudden spike in requests.

I found it difficult to set up a connection pooling in Oracle’s webLogic webserver,  so I thought I would write my own.   This was pretty easy, but then I discovered some complications.

Writing my own “connection pooling”.

It was clear to me that having connection = cf.createConnection();     from the ejbCreate() method rather than in the onMessage() method would be much more efficient.

  • Move variables from being method variables to instance variables so they persisted across onMessage() calls.
  • Create a connect() method which actually does the createConnection().  This can be called in the ejbCreate setup routine, and from the onMessage() if the connection variable is null.
  • Have code in the ejbRemove to the connection, session etc.
public class IVTMDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -337338331639L;
  // create long lasting instance variables 
  Connection connection = null;
  Session session = null;
  MessageProducer producer = null;
  InitialContext ctx = null;
  ConnectionFactory cf = null;
  // new method to do the connect
  public void connect(){
  try {
     ctx = new InitialContext();
     cf = (ConnectionFactory)ctx.lookup("CF2");
     connection = cf.createConnection();
     session = connection.createSession(false, 1); 
    }
    catch (Exception je)
    {...
    } 
 }

 public void ejbCreate() {
   connect();
 }
...
}

This worked well.    The number of MQCONN dropped to about 10 per day for many puts.

The problem looked solved, until I tried to shut down the queue manager.

If no work was being processed, then the MDBs were not being called, and so there were no MQ requests being made, and so the connections stayed active, and the queue manager did not shut down.

To solve this you can set  cf.setExceptionListener(..) which will get notified if there are problems with the connection, and so you can close the connection etc.
You need to consider what to do if the connection cannot be made;  do you wait for a short time period, or do you return an error.

I found that this “simple” way of avoiding all of the MQCONNects was quickly getting much more complex.  It was going to be much easier in the long term to get the provided connection factories working.   This was another challenge, taking about a week.  See the next few blog entries on how I did this for webLogic.

MDBs activation specs and @things in the java program

While struggling with getting MDBs working, and looking at examples, I saw examples where they defined JMS resources within the java program using @….  statements, and could not see how they worked.  These are called annotations. The documentation on the web assumes you know about annotations  when explaining annotations!  They, in fact, are pretty simple, let me explain.

Annotations start with an @ character, and the information can be stored within the .class file as meta-data.  Programs can extract and use this meta data.

You can have java code like
@Resource(lookup="java:customerMQ")
private javax.jms... myMQ;

A program, for example,  your program, an analysis program or a web server, can issue request like

  • load class information
  • from the meta data list all fields with @resource defined.
  • do things with the list

One example would be to specify a JNDI lookup of java:customerMQ and return it into the field myMQ.

Another example from the IBM documentation

@MessageDriven(
  name = "JMSSampleMDB",
  activationConfig = 
  {
    @ActivationConfigProperty(
       propertyName  = "destinationType", 
       propertyValue = "javax.jms.Queue"),
 
    @ActivationConfigProperty(
       propertyName  = "destination", 
	propertyValue = "jndi_INPUT_Q")                         
   }
)

The resource adapter has code which does

  • load your MDB program
  • get the MessageDriven stuff.
    • within this, locate the activationConfig records
      • within these, locate the ActivationConfigProperty propertyName and propertyValue, and merge the data with the data in the ejb-jar.xml file.

 

With the definitions in your java program, and the definitions in the MDB configuration you can configure a complete set of options for MDB.  I think the definitions in the java program override the MDB configuration.

How do I see what data there is?

You can extract this meta-data using a method like (see here)

public void getAnnotations(Class inclass){
    for(Field field : inclass.getDeclaredFields()){
        Class type = field.getType();
        String name = field.getName();
        field.getDeclaredAnnotations(); //do something with these
    }

Use the javap command to display the data.

To display the annotations you can usethe command, where ….class is the name of your class file.

javap -v .....class

My java program had

import javax.annotation.Resource;
.....
@Resource(lookup = "java:app/jms/myappTopic")
String colin = "ZZZZZ";

The javap command gave

java.lang.String colin;
  descriptor: Ljava/lang/String;
  flags:
  RuntimeVisibleAnnotations:
  0: #14(#15=s#16)
...
#14 = Utf8 Ljavax/annotation/Resource;
#15 = Utf8 lookup
#16 = Utf8 java:app/jms/myappTopic

from which we get

java.lang.String colin ... 
  javax/annotation/Resource (lookup = java:app/jms/myappTopic).

which matches the source code.

Different annotation types are confusing.

As well as providing meta-information on variables and classes, java also uses annotations to modify the java compiler behaviour.   For example

  • By putting @Deprecated infront of a method, the method can be flagged when used, as deprecated, and you should not use it
  • @SuppressWarnings(“unchecked”) tells the java compiler NOT to produce an error message for the unchecked condition.  See here for a list of warning conditions.

WebSphere Liberty connectionPool statistics

This blog post explains how to get and understand statistics from WebSphere Liberty on connectionPool usage.

In your MDB application you can have code like

 InitialContext ctx = new InitialContext();
ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF3");

This says lookup the connection defined by CF3 and issue MQCONN for this connection.

In WebSphere Liberty you defined connection information in server.xml.  For example

<jmsConnectionFactory jndiName="CF3" id="CF3ID">
  <connectionManager maxPoolSize="2" connectionTimeout="7s"/> 
  <properties.wmqJms 
   queueManager="QMA"
   transportType="BINDINGS"
   applicationName="Hello"/>
</jmsConnectionFactory>

The maxPoolSize gives the maximum number of connections available in this pool.

If server.xml has

<featureManager>
   <feature>monitor-1.0</feature>
</featureManager>

then you can get out statistics on connectionPools using the JMX interface.

In ./usr/servers/test/jvm.options I had

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Which defined the JMX port as 9010, and so I can get information through this port.

Looking at the output

There is documentation here on the connectionPool statistics.

You can use jconsole to get the JMX data, but this is not very usable, so I used jmxquery, which is part of a python package.  I installed it using pip install jmxquery.

I used the command

java -jar jmxquery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi -u admin c -p admin -q ‘WebSphere:*’ > outputfile

-q ‘WebSphere:*’  means give all records belonging to the WebSphere component.  If you say -q ‘*:*’ you get statistics for all components, see the bottom of the blog post.  Example output is given below.

This command wrote all of the output to file outputfile.  I then used grep to extract the relevant records.

grep WebSphere:type=ConnectionPoolStats,name outputfile

If you change a parameter in server.xml for the jmsConnectionPool, the pool is deleted, recreated, and the JMX data is reset.   If the pool has been reset, or not been used, statistics for that pool are not available.  On the first use of the pool the pool is created, and JMX statistics are available.

The JMX data for connectionPools

The data was like

WebSphere:type=ConnectionPoolStats,name=CF3/CreateCount (Long) = 2

The detailed records for WebSphere:type=ConnectionPoolStats,name=CF3 are

  • CreateCount (Long) = 2   this is the number of connections created,
  • DestroyCount (Long) = 0 this is the number of connections released because the pool was purged,
  • WaitTime (Double) = 76.36986301369863  there were insufficient threads.  For those threads that had to wait, this is the average wait time before a connection became available,
  • InUseTime (Double) = 18.905405405405407 the threads were active this time on average,
  • WaitTimeDetails/count (Long) = 98 requests because had to wait,
  • WaitTimeDetails/description (String) = Missing,
  • WaitTimeDetails/maximumValue (Long) = 110  the maximum wait time in milliseconds,
  • WaitTimeDetails/mean (Double) = 78.13265306122449 the average wait time,
  • WaitTimeDetails/minimumValue (Long) = 16 the minimum wait time,
  • WaitTimeDetails/standardDeviation (Double) = 16.474205982730254 the standard deviation,
  • WaitTimeDetails/total (Double) = 7657.0 in milliseconds.  7657/(number of waits 98) = average 78.13 (above),
  • WaitTimeDetails/unit (String) = UNKNOWN looks like a bug – this should be milliseconds,
  • WaitTimeDetails/variance (Double) = 271.82426517365184 ,
  • ManagedConnectionCount (Long) = 2  The total number of managed connections in the free, shared, and unshared pools,
  • ConnectionHandleCount (Long) = 0  this is the current handles in use,
  • FreeConnectionCount (Long) = 2  this is the number of connections in the pool, but not in use,
  • InUseTimeDetails/count (Long) = 101 – number of requests for a connection (ctx.lookup(“CF3”)),
  • InUseTimeDetails/description (String) = Missing,
  • InUseTimeDetails/maximumValue (Long) = 53 the maximum time the connection as in use in milliseconds,
  • InUseTimeDetails/mean (Double) = 18.099009900990097  the average time the connections were in use in milliseconds,
  • InUseTimeDetails/minimumValue (Long) = 10  the minimum time the connections were in use in milliseconds,
  • InUseTimeDetails/standardDeviation (Double) = 5.63923216261808,
  • InUseTimeDetails/total (Double) = 1828.0  in milliseconds.   This value(1828)/(number of connections used 101) gives the mean value 18.09 above,
  • InUseTimeDetails/unit (String) = UNKNOWN.

Note the order of the record can vary, for example CreateCount, can be first, or nearly last.

After a time interval aged connections can be released.  When there is sufficient workload to need more connections, they will be created as needed.  If the CreateCount increases significantly during the day, you may either have an irregular workload, or you need to increase you connectionTimeout value, to smooth out the connect/disconnect.

Having WaitTimeDetails/count=0 is good.  If this number is large in comparison to InUseTimeDetails/total then the pool is too small.

Other data you can get from JMX

  • IBM MQ:type=CommonServices
  • java.lang:type=ClassLoading
  • java.lang:type=Compilation
  • java.lang:type=GarbageCollector
  • java.lang:type=Memory
  • java.lang:type=Threading
  • osgi.core:
  • JMImplementation:type=MBeanServerDelegate
  • java.util.logging:type=Logging
  • java.nio:type=BufferPool,name=direct
  • java.lang:type=MemoryManager
  • java.lang:type=MemoryPool,name=Code Cache
  • java.lang:type=OperatingSystem
  • java.lang:type=Runtime
  • WebSphere:feature=apiDiscovery,name=APIDiscovery
  • WebSphere:feature=kernel,name=ServerInfo
  • WebSphere:type=JvmStats
  • WebSphere:type=ThreadPoolStats
  • WebSphere:type=ConnectionPoolStats ( as described above)
  • WebSphere:service=com.ibm.websphere.application.ApplicationMBean,name=CCP

How do I make my MDB transactional?

I found from the application trace  that my MDB was doing MQGET, MQCMIT in the listener, and MQOPEN, MQPUT, MQCLOSE and no MQCMIT in my application.    Digging into this I found that the MQPUT was NO_SYNCPOINT, which was a surprise to me!

My application had session = connection.createSession(true, 1); // true = transactional. So I expected it to work.

The ejb-jar.xml had

enterprise-beans
  message-driven
    transaction-type Container
...
assembly-descriptor
  container-transaction
    trans-attribute NotSupported

I changed NotSupported to Required and it worked.

 

The application trace for the Listener part of the MDB gave me

Operation      CompCode MQRC HObj (ObjName) 
MQXF_XASTART            0000 -
MQXF_GET       MQCC_OK  0000    2 (JMSQ2 )
MQXF_XAEND              0000 -
MQXF_XAPREPARE          0000 -
MQXF_XACOMMIT           0000 -

The trace for the application part of the MDB gave me

Operation                    CompCode MQRC HObj (ObjName)
MQXF_XASTART                             0000         –
MQXF_OPEN             MQCC_OK   0000         2 (CP0000 )
MQXF_PUT                MQCC_OK   0000          2 (CP0000 )
MQXF_CLOSE           MQCC_OK   0000          2 (CP0000 )
MQXF_XAEND                                0000         –
MQXF_XAPREPARE                       0000 –
MQXF_XACOMMIT                        0000 –

and the put options had _SYNCPOINT.

I had read documentation saying that you needed to have XAConnectionFactory instead of ConnectionFactory.  I could not get this work,  but found it was not needed for JMS;  it may be needed for JDBC.