Should I make my web server Highly Available ?

If you are using XA on midrange  then connect to just one queue manager, because  anything else is not supported.  If your connection goes to a different queue manager, then it cannot recover a unit of work.
The only exception to that rule is if you are connecting to a QSG on z/OS. When using a QSG, all of the queue managers in the QSG can resolve each others transactions – this means that the transaction recovery issue goes away.
Be careful using multiple queue managers.
For the listener you want to connect to just one queue manager.  If you have a choice of two queue managers, all threads may connect to QM1, and there be no connection to QM2.  If you have a second web server instance, this may also connect to QM1, and not QM2.  It is better to have WS1 connect to QM1 and WS2 to connect to WS2 – so called production lines.
If you need to shut down QM1, then shut down WS1,  do not try to cross wire it
For the MDB logic connection factory if you had two possible queue managers, you may get a spread of connections, from all on QM1, to all on QM2, or a mixture.
If the WS frees connections (for example they are idle), then when there is a load needing more connections, it could get a connection from QM1 or QM2.
All in all – keep it simple,  and have each web server connect to just one queue manager.

 

Whoops -deploying MDB in weblogic

I was quite happily using my MDB in webLogic, but when I changed its configuration, it did not pick up the new changes.  It took a day to find out why,  and I have learned much more about deploying MDBS.

My connection factory was using SYSTEM.DEF.SVRCONN, I changed it to use a different client channel. I stopped SYSTEM.DEF.SVRCONN, ( so I could check the change had worked), and restarted the webLogic instance.  I was surprised when my MDB failed to start, because the channel was stopped.   The MDB was trying to use that channel.  It took a lot of head scratching to get it to work as I expected.

  • I had messages like <BEA-015073>  Message-Driven Bean …  is configured with unknown activation-config-property name failIfQuiesce.  This message is wrong, failIfQuiesce is supported by the IBM Resource adapter.
  • I had the same message with activation-config-property name cfLookup.   This was my problem.  I should have specified connectionFactoryLookup.
  • If you have <activation-config-property-name>connectionFactoryLookup… (specified in the ejb-jar) any other parameters you specify in the ejb-jar.xml file are ignored.
  • If you do not specify a connectionFactoryLookup, nor properties in the ejb-jar.xml file, defaults are provided, see Configuring the resource adapter for inbound communication.  In my case I had not specified  activation-config-property-name channel, and this defaulted to SYSTEM.DEF.SVRCONN, which is why it continued to use that channel.
  • It worth putting <activation-config-property-name>applicationName … in your definitions so you can see what you are using.
    • dis qstatus(JMSQ3) type(handle) gave me APPLTAG(CF3Name) so I can tell which definitions are being used.
    • If you get APPLTAG(weblogic.Server) then you are taking the defaults.
  • The Oracle documentation  says the precedence order is as below.    I do not think this is 100% accurate. (I could not specify some of the parameters on the weblogic-ejb-jar.xml file).  I didnt try the java program.
    1. properties set in the weblogic-ejb-jar.xml deployment descriptor
    2. activation-config-property properties set in the ejb-jar.xml deployment descriptor
    3. activationConfigProperty annotation properties in the java program.

What do I need to specify?

As a minimum you need to use connectionFactoryLookup or  specify

  1. applicationName – so you can identify which definitions are being used
  2. channel – which channel to use
  3. failIfQuiesce
  4. hostName
  5. port

 

The ejb-jar.xml file is in the META-INF directory.  Change  the ejb-jar.xml or  weblogic-ejb-jar.xml file. IUpdate the jar file using a command like jar -uvf MDB4.jar  META-INF/ejb-jar.xml,   and redeploy it.

Did the JMS architects get it wrong? Possibly!

I was looking into a performance problem where a web server was doing 1 million MQCONNects  day!  During my investigations I found that the original designers did not design it properly because of the performance overhead, and so the JMS architects had to fix it up by creating connection factories.

Below, I cover

  • the performance problem
  • writing my own connection factory
  • the problems of writing my own connection factory
  • it might just be easier to use a connection factory provided by your web server or JMS provider.

What is an EJB and an MDB?

In a web server you can have Enterprise Java Beans (EJBs) which are a package of java applications doing enterprise type work ( get a message, update a database, send a reply) conforming to an specific Programming Interface.

A Message Driven Bean is an EJB which responds to messages.  A message listener applications gets messages from a queue – and passes the message to the MDB. In non EJB terms this is just like a trigger monitor, starting a transaction and passing the data.

When you create the MDB you have a configuration file  which allows a message listener to be created, and the parameters to use to connect to MQ, and  how many threads etc can be running concurrently.

The MDB, to process the message has a basic structure of

  • method onMessage – this is given the message
  • ejbCreate – when the EJB is created, you can do initialisation here
  • ejbRemove – when the EJB is about to be deleted, you can do clean up here

and looks like

public class MDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -8070254332864574796L;
  public void onMessage(Message message)
  {
    Connection connection = null;
    Session session = null;
    MessageProducer producer = null;

    try
    {
      InitialContext ctx = new InitialContext();
      ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF2");
      Destination dest = message.getJMSReplyTo();
     
      connection = cf.createConnection();
      session = connection.createSession(false, 1);
      producer = session.createProducer(dest);
      
      TextMessage response = session.createTextMessage("test response message from the WMQ_IVT_MDB");
      response.setJMSCorrelationID(message.getJMSMessageID());
      producer.send(response);
      return;
    }
    catch (Exception je)
    {
      System.out.println("Something went wrong." + je);
    }
  }

  public void ejbCreate() { }
  public void ejbRemove() { }
 }

Within the onMessage method is logic connection = cf.createConnection();
Under the covers this createConnection() does an MQCONN.  With 1 million messages a day – this is 1 million MQCONNs!
There was also an MQOPEN, an MQCLOSE,  and an MQDISC before the onMessage() method returned.  This results in a huge performance overhead.

Attempts to fix the performance

People quickly found that this model was not efficient, and they came up with ways to improve the performance.

The idea of a connection pool was developed.  Instead of doing an the createConnection() doing an MQCONN etc.  this code was changed, to not do the MQDISC, but to keep the Queue Manager handle. When the next MQCONN occurred, if there was one of these spare Queue Manager handles it would use it.

These connection pools are sophisticated.   For example they can free up connections if they had not been used for a time period, so reducing the overall number of Queue Manager connections in use, they can slowly increase the number of connections in the pool so there is not a sudden spike in requests.

I found it difficult to set up a connection pooling in Oracle’s webLogic webserver,  so I thought I would write my own.   This was pretty easy, but then I discovered some complications.

Writing my own “connection pooling”.

It was clear to me that having connection = cf.createConnection();     from the ejbCreate() method rather than in the onMessage() method would be much more efficient.

  • Move variables from being method variables to instance variables so they persisted across onMessage() calls.
  • Create a connect() method which actually does the createConnection().  This can be called in the ejbCreate setup routine, and from the onMessage() if the connection variable is null.
  • Have code in the ejbRemove to the connection, session etc.
public class IVTMDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -337338331639L;
  // create long lasting instance variables 
  Connection connection = null;
  Session session = null;
  MessageProducer producer = null;
  InitialContext ctx = null;
  ConnectionFactory cf = null;
  // new method to do the connect
  public void connect(){
  try {
     ctx = new InitialContext();
     cf = (ConnectionFactory)ctx.lookup("CF2");
     connection = cf.createConnection();
     session = connection.createSession(false, 1); 
    }
    catch (Exception je)
    {...
    } 
 }

 public void ejbCreate() {
   connect();
 }
...
}

This worked well.    The number of MQCONN dropped to about 10 per day for many puts.

The problem looked solved, until I tried to shut down the queue manager.

If no work was being processed, then the MDBs were not being called, and so there were no MQ requests being made, and so the connections stayed active, and the queue manager did not shut down.

To solve this you can set  cf.setExceptionListener(..) which will get notified if there are problems with the connection, and so you can close the connection etc.
You need to consider what to do if the connection cannot be made;  do you wait for a short time period, or do you return an error.

I found that this “simple” way of avoiding all of the MQCONNects was quickly getting much more complex.  It was going to be much easier in the long term to get the provided connection factories working.   This was another challenge, taking about a week.  See the next few blog entries on how I did this for webLogic.

MDBs activation specs and @things in the java program

While struggling with getting MDBs working, and looking at examples, I saw examples where they defined JMS resources within the java program using @….  statements, and could not see how they worked.  These are called annotations. The documentation on the web assumes you know about annotations  when explaining annotations!  They, in fact, are pretty simple, let me explain.

Annotations start with an @ character, and the information can be stored within the .class file as meta-data.  Programs can extract and use this meta data.

You can have java code like
@Resource(lookup="java:customerMQ")
private javax.jms... myMQ;

A program, for example,  your program, an analysis program or a web server, can issue request like

  • load class information
  • from the meta data list all fields with @resource defined.
  • do things with the list

One example would be to specify a JNDI lookup of java:customerMQ and return it into the field myMQ.

Another example from the IBM documentation

@MessageDriven(
  name = "JMSSampleMDB",
  activationConfig = 
  {
    @ActivationConfigProperty(
       propertyName  = "destinationType", 
       propertyValue = "javax.jms.Queue"),
 
    @ActivationConfigProperty(
       propertyName  = "destination", 
	propertyValue = "jndi_INPUT_Q")                         
   }
)

The resource adapter has code which does

  • load your MDB program
  • get the MessageDriven stuff.
    • within this, locate the activationConfig records
      • within these, locate the ActivationConfigProperty propertyName and propertyValue, and merge the data with the data in the ejb-jar.xml file.

 

With the definitions in your java program, and the definitions in the MDB configuration you can configure a complete set of options for MDB.  I think the definitions in the java program override the MDB configuration.

How do I see what data there is?

You can extract this meta-data using a method like (see here)

public void getAnnotations(Class inclass){
    for(Field field : inclass.getDeclaredFields()){
        Class type = field.getType();
        String name = field.getName();
        field.getDeclaredAnnotations(); //do something with these
    }

Use the javap command to display the data.

To display the annotations you can usethe command, where ….class is the name of your class file.

javap -v .....class

My java program had

import javax.annotation.Resource;
.....
@Resource(lookup = "java:app/jms/myappTopic")
String colin = "ZZZZZ";

The javap command gave

java.lang.String colin;
  descriptor: Ljava/lang/String;
  flags:
  RuntimeVisibleAnnotations:
  0: #14(#15=s#16)
...
#14 = Utf8 Ljavax/annotation/Resource;
#15 = Utf8 lookup
#16 = Utf8 java:app/jms/myappTopic

from which we get

java.lang.String colin ... 
  javax/annotation/Resource (lookup = java:app/jms/myappTopic).

which matches the source code.

Different annotation types are confusing.

As well as providing meta-information on variables and classes, java also uses annotations to modify the java compiler behaviour.   For example

  • By putting @Deprecated infront of a method, the method can be flagged when used, as deprecated, and you should not use it
  • @SuppressWarnings(“unchecked”) tells the java compiler NOT to produce an error message for the unchecked condition.  See here for a list of warning conditions.