WebLogic message does not have authorization to view the logs

I was using JMX to display the connectionPool statistics  from webLogic, and kept getting messages

<Error> <Diagnostics> <BEA-320084> <The user principals=[] does not have authorization to view the logs.>

I solved this by using the webLogic console

  • Click “Security Realms” in the Domain Stucture box on the left hand side of the home page
  • Click on the name of your realm (myrealm)
  • Click on “Roles and Policies”
  • Click on “Realm Policies”
  • Expand Domain and select “View Policy Conditions” for “View Log”
  • Click on Add Condition
  • Select the role
  • Click on Finish
  • Click on Save

The change is available immediately on “Save”.

Did the JMS architects get it wrong? Possibly!

I was looking into a performance problem where a web server was doing 1 million MQCONNects  day!  During my investigations I found that the original designers did not design it properly because of the performance overhead, and so the JMS architects had to fix it up by creating connection factories.

Below, I cover

  • the performance problem
  • writing my own connection factory
  • the problems of writing my own connection factory
  • it might just be easier to use a connection factory provided by your web server or JMS provider.

What is an EJB and an MDB?

In a web server you can have Enterprise Java Beans (EJBs) which are a package of java applications doing enterprise type work ( get a message, update a database, send a reply) conforming to an specific Programming Interface.

A Message Driven Bean is an EJB which responds to messages.  A message listener applications gets messages from a queue – and passes the message to the MDB. In non EJB terms this is just like a trigger monitor, starting a transaction and passing the data.

When you create the MDB you have a configuration file  which allows a message listener to be created, and the parameters to use to connect to MQ, and  how many threads etc can be running concurrently.

The MDB, to process the message has a basic structure of

  • method onMessage – this is given the message
  • ejbCreate – when the EJB is created, you can do initialisation here
  • ejbRemove – when the EJB is about to be deleted, you can do clean up here

and looks like

public class MDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -8070254332864574796L;
  public void onMessage(Message message)
  {
    Connection connection = null;
    Session session = null;
    MessageProducer producer = null;

    try
    {
      InitialContext ctx = new InitialContext();
      ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF2");
      Destination dest = message.getJMSReplyTo();
     
      connection = cf.createConnection();
      session = connection.createSession(false, 1);
      producer = session.createProducer(dest);
      
      TextMessage response = session.createTextMessage("test response message from the WMQ_IVT_MDB");
      response.setJMSCorrelationID(message.getJMSMessageID());
      producer.send(response);
      return;
    }
    catch (Exception je)
    {
      System.out.println("Something went wrong." + je);
    }
  }

  public void ejbCreate() { }
  public void ejbRemove() { }
 }

Within the onMessage method is logic connection = cf.createConnection();
Under the covers this createConnection() does an MQCONN.  With 1 million messages a day – this is 1 million MQCONNs!
There was also an MQOPEN, an MQCLOSE,  and an MQDISC before the onMessage() method returned.  This results in a huge performance overhead.

Attempts to fix the performance

People quickly found that this model was not efficient, and they came up with ways to improve the performance.

The idea of a connection pool was developed.  Instead of doing an the createConnection() doing an MQCONN etc.  this code was changed, to not do the MQDISC, but to keep the Queue Manager handle. When the next MQCONN occurred, if there was one of these spare Queue Manager handles it would use it.

These connection pools are sophisticated.   For example they can free up connections if they had not been used for a time period, so reducing the overall number of Queue Manager connections in use, they can slowly increase the number of connections in the pool so there is not a sudden spike in requests.

I found it difficult to set up a connection pooling in Oracle’s webLogic webserver,  so I thought I would write my own.   This was pretty easy, but then I discovered some complications.

Writing my own “connection pooling”.

It was clear to me that having connection = cf.createConnection();     from the ejbCreate() method rather than in the onMessage() method would be much more efficient.

  • Move variables from being method variables to instance variables so they persisted across onMessage() calls.
  • Create a connect() method which actually does the createConnection().  This can be called in the ejbCreate setup routine, and from the onMessage() if the connection variable is null.
  • Have code in the ejbRemove to the connection, session etc.
public class IVTMDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -337338331639L;
  // create long lasting instance variables 
  Connection connection = null;
  Session session = null;
  MessageProducer producer = null;
  InitialContext ctx = null;
  ConnectionFactory cf = null;
  // new method to do the connect
  public void connect(){
  try {
     ctx = new InitialContext();
     cf = (ConnectionFactory)ctx.lookup("CF2");
     connection = cf.createConnection();
     session = connection.createSession(false, 1); 
    }
    catch (Exception je)
    {...
    } 
 }

 public void ejbCreate() {
   connect();
 }
...
}

This worked well.    The number of MQCONN dropped to about 10 per day for many puts.

The problem looked solved, until I tried to shut down the queue manager.

If no work was being processed, then the MDBs were not being called, and so there were no MQ requests being made, and so the connections stayed active, and the queue manager did not shut down.

To solve this you can set  cf.setExceptionListener(..) which will get notified if there are problems with the connection, and so you can close the connection etc.
You need to consider what to do if the connection cannot be made;  do you wait for a short time period, or do you return an error.

I found that this “simple” way of avoiding all of the MQCONNects was quickly getting much more complex.  It was going to be much easier in the long term to get the provided connection factories working.   This was another challenge, taking about a week.  See the next few blog entries on how I did this for webLogic.

MDBs activation specs and @things in the java program

While struggling with getting MDBs working, and looking at examples, I saw examples where they defined JMS resources within the java program using @….  statements, and could not see how they worked.  These are called annotations. The documentation on the web assumes you know about annotations  when explaining annotations!  They, in fact, are pretty simple, let me explain.

Annotations start with an @ character, and the information can be stored within the .class file as meta-data.  Programs can extract and use this meta data.

You can have java code like
@Resource(lookup="java:customerMQ")
private javax.jms... myMQ;

A program, for example,  your program, an analysis program or a web server, can issue request like

  • load class information
  • from the meta data list all fields with @resource defined.
  • do things with the list

One example would be to specify a JNDI lookup of java:customerMQ and return it into the field myMQ.

Another example from the IBM documentation

@MessageDriven(
  name = "JMSSampleMDB",
  activationConfig = 
  {
    @ActivationConfigProperty(
       propertyName  = "destinationType", 
       propertyValue = "javax.jms.Queue"),
 
    @ActivationConfigProperty(
       propertyName  = "destination", 
	propertyValue = "jndi_INPUT_Q")                         
   }
)

The resource adapter has code which does

  • load your MDB program
  • get the MessageDriven stuff.
    • within this, locate the activationConfig records
      • within these, locate the ActivationConfigProperty propertyName and propertyValue, and merge the data with the data in the ejb-jar.xml file.

 

With the definitions in your java program, and the definitions in the MDB configuration you can configure a complete set of options for MDB.  I think the definitions in the java program override the MDB configuration.

How do I see what data there is?

You can extract this meta-data using a method like (see here)

public void getAnnotations(Class inclass){
    for(Field field : inclass.getDeclaredFields()){
        Class type = field.getType();
        String name = field.getName();
        field.getDeclaredAnnotations(); //do something with these
    }

Use the javap command to display the data.

To display the annotations you can usethe command, where ….class is the name of your class file.

javap -v .....class

My java program had

import javax.annotation.Resource;
.....
@Resource(lookup = "java:app/jms/myappTopic")
String colin = "ZZZZZ";

The javap command gave

java.lang.String colin;
  descriptor: Ljava/lang/String;
  flags:
  RuntimeVisibleAnnotations:
  0: #14(#15=s#16)
...
#14 = Utf8 Ljavax/annotation/Resource;
#15 = Utf8 lookup
#16 = Utf8 java:app/jms/myappTopic

from which we get

java.lang.String colin ... 
  javax/annotation/Resource (lookup = java:app/jms/myappTopic).

which matches the source code.

Different annotation types are confusing.

As well as providing meta-information on variables and classes, java also uses annotations to modify the java compiler behaviour.   For example

  • By putting @Deprecated infront of a method, the method can be flagged when used, as deprecated, and you should not use it
  • @SuppressWarnings(“unchecked”) tells the java compiler NOT to produce an error message for the unchecked condition.  See here for a list of warning conditions.

WebSphere Liberty connectionPool statistics

This blog post explains how to get and understand statistics from WebSphere Liberty on connectionPool usage.

In your MDB application you can have code like

 InitialContext ctx = new InitialContext();
ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF3");

This says lookup the connection defined by CF3 and issue MQCONN for this connection.

In WebSphere Liberty you defined connection information in server.xml.  For example

<jmsConnectionFactory jndiName="CF3" id="CF3ID">
  <connectionManager maxPoolSize="2" connectionTimeout="7s"/> 
  <properties.wmqJms 
   queueManager="QMA"
   transportType="BINDINGS"
   applicationName="Hello"/>
</jmsConnectionFactory>

The maxPoolSize gives the maximum number of connections available in this pool.

If server.xml has

<featureManager>
   <feature>monitor-1.0</feature>
</featureManager>

then you can get out statistics on connectionPools using the JMX interface.

In ./usr/servers/test/jvm.options I had

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Which defined the JMX port as 9010, and so I can get information through this port.

Looking at the output

There is documentation here on the connectionPool statistics.

You can use jconsole to get the JMX data, but this is not very usable, so I used jmxquery, which is part of a python package.  I installed it using pip install jmxquery.

I used the command

java -jar jmxquery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi -u admin c -p admin -q ‘WebSphere:*’ > outputfile

-q ‘WebSphere:*’  means give all records belonging to the WebSphere component.  If you say -q ‘*:*’ you get statistics for all components, see the bottom of the blog post.  Example output is given below.

This command wrote all of the output to file outputfile.  I then used grep to extract the relevant records.

grep WebSphere:type=ConnectionPoolStats,name outputfile

If you change a parameter in server.xml for the jmsConnectionPool, the pool is deleted, recreated, and the JMX data is reset.   If the pool has been reset, or not been used, statistics for that pool are not available.  On the first use of the pool the pool is created, and JMX statistics are available.

The JMX data for connectionPools

The data was like

WebSphere:type=ConnectionPoolStats,name=CF3/CreateCount (Long) = 2

The detailed records for WebSphere:type=ConnectionPoolStats,name=CF3 are

  • CreateCount (Long) = 2   this is the number of connections created,
  • DestroyCount (Long) = 0 this is the number of connections released because the pool was purged,
  • WaitTime (Double) = 76.36986301369863  there were insufficient threads.  For those threads that had to wait, this is the average wait time before a connection became available,
  • InUseTime (Double) = 18.905405405405407 the threads were active this time on average,
  • WaitTimeDetails/count (Long) = 98 requests because had to wait,
  • WaitTimeDetails/description (String) = Missing,
  • WaitTimeDetails/maximumValue (Long) = 110  the maximum wait time in milliseconds,
  • WaitTimeDetails/mean (Double) = 78.13265306122449 the average wait time,
  • WaitTimeDetails/minimumValue (Long) = 16 the minimum wait time,
  • WaitTimeDetails/standardDeviation (Double) = 16.474205982730254 the standard deviation,
  • WaitTimeDetails/total (Double) = 7657.0 in milliseconds.  7657/(number of waits 98) = average 78.13 (above),
  • WaitTimeDetails/unit (String) = UNKNOWN looks like a bug – this should be milliseconds,
  • WaitTimeDetails/variance (Double) = 271.82426517365184 ,
  • ManagedConnectionCount (Long) = 2  The total number of managed connections in the free, shared, and unshared pools,
  • ConnectionHandleCount (Long) = 0  this is the current handles in use,
  • FreeConnectionCount (Long) = 2  this is the number of connections in the pool, but not in use,
  • InUseTimeDetails/count (Long) = 101 – number of requests for a connection (ctx.lookup(“CF3”)),
  • InUseTimeDetails/description (String) = Missing,
  • InUseTimeDetails/maximumValue (Long) = 53 the maximum time the connection as in use in milliseconds,
  • InUseTimeDetails/mean (Double) = 18.099009900990097  the average time the connections were in use in milliseconds,
  • InUseTimeDetails/minimumValue (Long) = 10  the minimum time the connections were in use in milliseconds,
  • InUseTimeDetails/standardDeviation (Double) = 5.63923216261808,
  • InUseTimeDetails/total (Double) = 1828.0  in milliseconds.   This value(1828)/(number of connections used 101) gives the mean value 18.09 above,
  • InUseTimeDetails/unit (String) = UNKNOWN.

Note the order of the record can vary, for example CreateCount, can be first, or nearly last.

After a time interval aged connections can be released.  When there is sufficient workload to need more connections, they will be created as needed.  If the CreateCount increases significantly during the day, you may either have an irregular workload, or you need to increase you connectionTimeout value, to smooth out the connect/disconnect.

Having WaitTimeDetails/count=0 is good.  If this number is large in comparison to InUseTimeDetails/total then the pool is too small.

Other data you can get from JMX

  • IBM MQ:type=CommonServices
  • java.lang:type=ClassLoading
  • java.lang:type=Compilation
  • java.lang:type=GarbageCollector
  • java.lang:type=Memory
  • java.lang:type=Threading
  • osgi.core:
  • JMImplementation:type=MBeanServerDelegate
  • java.util.logging:type=Logging
  • java.nio:type=BufferPool,name=direct
  • java.lang:type=MemoryManager
  • java.lang:type=MemoryPool,name=Code Cache
  • java.lang:type=OperatingSystem
  • java.lang:type=Runtime
  • WebSphere:feature=apiDiscovery,name=APIDiscovery
  • WebSphere:feature=kernel,name=ServerInfo
  • WebSphere:type=JvmStats
  • WebSphere:type=ThreadPoolStats
  • WebSphere:type=ConnectionPoolStats ( as described above)
  • WebSphere:service=com.ibm.websphere.application.ApplicationMBean,name=CCP

How do I make my MDB transactional?

I found from the application trace  that my MDB was doing MQGET, MQCMIT in the listener, and MQOPEN, MQPUT, MQCLOSE and no MQCMIT in my application.    Digging into this I found that the MQPUT was NO_SYNCPOINT, which was a surprise to me!

My application had session = connection.createSession(true, 1); // true = transactional. So I expected it to work.

The ejb-jar.xml had

enterprise-beans
  message-driven
    transaction-type Container
...
assembly-descriptor
  container-transaction
    trans-attribute NotSupported

I changed NotSupported to Required and it worked.

 

The application trace for the Listener part of the MDB gave me

Operation      CompCode MQRC HObj (ObjName) 
MQXF_XASTART            0000 -
MQXF_GET       MQCC_OK  0000    2 (JMSQ2 )
MQXF_XAEND              0000 -
MQXF_XAPREPARE          0000 -
MQXF_XACOMMIT           0000 -

The trace for the application part of the MDB gave me

Operation                    CompCode MQRC HObj (ObjName)
MQXF_XASTART                             0000         –
MQXF_OPEN             MQCC_OK   0000         2 (CP0000 )
MQXF_PUT                MQCC_OK   0000          2 (CP0000 )
MQXF_CLOSE           MQCC_OK   0000          2 (CP0000 )
MQXF_XAEND                                0000         –
MQXF_XAPREPARE                       0000 –
MQXF_XACOMMIT                        0000 –

and the put options had _SYNCPOINT.

I had read documentation saying that you needed to have XAConnectionFactory instead of ConnectionFactory.  I could not get this work,  but found it was not needed for JMS;  it may be needed for JDBC.

On Weblogic why isnt my MDB scaling past 10 instances?

This is another tale of one step back,  two steps sideways.  I was trying to understand why the JMX data on the MDBs was not as I expected, and why I was not getting tasks waiting.  I am still working on that little problem, but in passing I found I could not get my MDBs to scale.  I have rewritten parts of this post multiple times, as I understand more of it.  I believe the concepts are correct, but the implementation may be different to what I have described.

There are three parts to an MDB.

  1. A thread gets a message from the queue
  2. The message is passed to the application “OnMessage() method of the application
  3. The application uses a connection factory to get a connection to the queue manager and to the send the reply.

Expanding this to provide more details.

Thread pools are used to reuse the MQ connection, as the MQCONN and MQDISC are expensive operations.  By using a thread pool, the repeated MQCONN and MQDISC can be avoided.

There is a specific pool for the application, and when threads are released from this pool, they are put into a general pool.   Periodically  threads can be removed from the general pool, by issuing MQDISC, and then deleting the thread.

Get the message from the queue

The thread has two modes of operation Async consume – or plain old fashioned MQGET.

If the channel has SHARECNV(0) there is a  listener thread which browses the queue, and waits a short period( for example 5 seconds)  for a message.  There is a short wait, so that the thread can take action if required ( for example stop running).  This means if there is no traffic there is an empty MQGET every 5 seconds.   This can be expensive.

If the channel has SHARECNV(>0) then Asyn consume is used.  Internally there is a thread which browses the queue, and multiple threads which can get the message.

The maximum number of threads which can get messages is defined in the ejb-jar.xml activation-config-property-name maxPoolDepth  value.

These threads are in a pool called EJBPoolRuntime.  Each MDB has a thread pool of this name, but from the JMX data you can identify the pool as the JMS descriptor has a descriptor like MessageDrivenEJBRuntime=WMQ_IVT_MDB, Name=WMQ_IVT_MDB, ApplicationRuntime=MDB3, Type=EJBPoolRuntime, EJBComponentRuntime=MDB3/… where my MDB was called MDB3.

The parameters are defined in the ejb-jar.xml file.   The definitions are documented here.  The example below shows how to get from a queue called JMSQ2, and there will be no more than 37 threads able to get a message.

ejb-jar
  enterprise-beans
    message-driven
      activation-config
        activation-config-property>  
          activation-config-property-name maxPoolDepth  
            activation-config-property-value 37
          activation-config-property-name destination 
            activation-config-property-value JMSQ2
 

Note:  I did get messages like the following messages which I ignored ( as I think they are produced in error)

    • <Warning> <EJB> <BEA-015073> <Message-Driven Bean WMQ_IVT_MDB(Application: MDB3, EJBComponent: MDB3.jar) is configured with unknown activation-config-property name maxPoolDepth>
    • <Warning> <EJB> <BEA-015073> <Message-Driven Bean WMQ_IVT_MDB(Application: MDB3, EJBComponent: MDB3.jar) is configured with unknown activation-config-property name destination>

The default value of maxPoolDepth is 10 – this explains why I only had  10 threads getting messages from the queue.

Passing the message to the application for processing.

Once a message is available it will pass it to the OnMessage method of the application. There is some weblogic specific code, which seems to add little value. The concepts of this are

  1. There is an array of handles/Beans of size max-beans-in-free-pool.
  2. When the first message is processed, create “initial-beans-in-free-pool” beans and populate the array invoke the EJBCreate() method of the application.
  3. When a message arrives, then find a free element in this array,
    1. If the slot has a bean, then use it
    2. else allocate a bean and store it in the slot.   This allocation invokes the EJBCreate() method of the application.  On my laptop it took a seconds to allocate a new bean, which means there is a lag when responding to a spike in workload.
    3. call the OnMessage() method of the application.
  4. If all of the slots are in use – then wait.
  5. On return from the OnMessage() flag the entry as free
  6. Every idle-timeout-seconds scan the array, and free beans to make the current size the same as the in initial-beans-in-free-pool.  As part of this the EJBRemove() method of the application is invoked.

The definitions are documented here.

weblogic-ejb
  weblogic-enterprise-bean-jar
    pool    
      max-beans-in-free-pool 47
      initial-beans-in-free-pool 17 
      idle-timeout-seconds 60

I could find no benefit in using this pool.

The default max-beans-in-free-pool is 1000 which feels large enough.  You should make the initial-beans-in-free-pool the same or larger than the number of threads getting messages, see maxPoolDepth above.

If this value is too small, then periodically the pool will be purged down to the initial-beans-in-free-pool and then beans will be allocated as needed.  You will get a periodic drop in throughput.

Note the term max-beans-in-free-pool is not entirely accurate: the maximum number of threads for the pool is current threads in pool + active threads.   The term max-beans-in-free-pool  is accurate when there are no threads in use.

In the JMX statistics data, there is information on this pool.   The data name is likecom.bea:ServerRuntime=AdminServer2, MessageDrivenEJBRuntime=WMQ_IVT_MDB, Name=WMQ_IVT_MDB, ApplicationRuntime=MDB3, Type=EJBPoolRuntime, where WMQ_IVT_MDB is the display name of the MDB, and MDB3 is the name of the jar file.  This allows you to identify the pool for each MDB.

Get a connection and send the reply – the application connectionFactory pool.

The application typically needs to issue an MQCONN, MQOPEN of the reply to queue, put the message, and issue MQDISC before returning.   This MQCONN, MQDISC is expensive so a pool is used to save the queue manager connection handle between calls.  The connections are saved in a thread pool.

In the MDB java application there is code like ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF3”);

Where the connectionFactory CF3 is defined in the resource Adapter configuration.

The connectionFactory cf can then be used when putting messages.

The logic is like

  • If there is a free thread in the connectionFactory pool then use it
  • else there is no free thread in the connectionFactory pool
    • if the number of threads in the connectionFactory pool at the maximum value, then throw an exception
    • else create a new thread (doing an MQCONN etc)
  • when the java program issues connection.close(), return the thread to the connectionFactory pool.

It looks like the queue handle is not cached, so there is an MQOPEN… MQCLOSE of the reply queue for every request.

You configure the connectionFactory resource pool from: Home, Deployments, click on your resource adapter, click on the Configuration tab, click on the + in front of the javax.jms, ConnectionFactory, click on the connectionFactory name, click on the Connection Pool tab, specify the parameters and click on the save button.
Note: You have to stop and restart the server or redeploy the application to pick up changes!

This pool size needs to have enough capacity to handle the case when all input threads are busy with an MQGET.

JMX provides statistics with a description like com.bea: ServerRuntime=AdminServer2, Name=CF3, ApplicationRuntime=colinra, Type=ConnectorConnectionPoolRuntime, ConnectorComponentRuntime=colinra_colin where CF3 is the name of the connection pool defined to the resource adapter, colinra is the name I gave to the resource adapter when I installed it, colin.rar is the name of the resource adapter file.

Changing userids

The application connectionFactory pool can be used by different MDBs.  You need to make sure this pool has enough capacity for all the MDBs using it.

If the pool is used by MDBs running with different userids, then when a thread is obtained, it the thread was last used for a different userid, then the thread has to issue MQDISC and MQCONN with the current userid, this defeats the purpose of having a connection pool.

To prevent this you should have a connection pool for MDBs running with the same userid.

Getting a userid from the general pool may have the same problem, so you should ensure your pools have a maxium limit which is suitable for expected peak processing, and initial-beans-in-free-pool for your normal processing.   This should reduce the need to switching userids.

Cleaning up the connectionFactory

When the connectionFactory is configured, you can specify

  • Shrink Frequency Seconds:
  • Shrink Enabled: true|false

These parameters effectively say after after the “Shrink Frequency Seconds”, if the number of threads in the connectionFactory pool is larger than the initial pool size, then end threads (doing an MQDISC) to reduce the number of threads to the  initial pool size.   If the initial pool size is badly chosen you may get 20 threads ending, so there are 20 MQDISC, and because of the load, 20 threads are immediately created to handle the workload.  During this period  there will be insufficient threads to handle the workload, so you will get a blip in the throughput.

If you have one connectionFactory pool being used by a high importance MDB and by a low importance MDB, it could be that the high importance MDB is impacted by this “release/acquire”, and the low priority MDB is not affected.  Consider isolating the connectionFactory pools and specify the appropriate initial pool size.

To find out what was going on I used

  • DIS QSTATUS(inputqueue)  to see the number of open handles.   This is  listener count(1) + current number of threads doing MQGETS, so with maxPoolDepth = 19, this value was up to 20.
  • I changed my MDB application to display the instance number when it was deployed.
    •  ...
       private final static AtomicInteger count = new AtomicInteger(0); 
      ...
      public void ejbCreate() {
        SimpleDateFormat sdftime = new SimpleDateFormat("HH:mm:ss.SSS"); 
        Timestamp now = new Timestamp(System.currentTimeMillis());
        instance = count.addAndGet(1); 
        System.out.println(sdftime.format(now) +":" + this.getClass().getSimpleName()+":EJBCreate:"+instance);
      }
      public void ejbRemove()
      {
        System.out.println(this.getClass().getSimpleName()+":EJBRemove:"+instance 
                           +" messages processed "+messageCount);
        count.decrementAndGet();
      }

This gave me a message which told me when the instance was created, so I could see when it was started.   I could then see more instances created as the workload increased.

07:16:50.520:IVTMDB:EJBCreate:0

  • By using a client connection, I could specify the appltag for the connection pool and so see the number of MQCONNs from the application connectionFactory.

What happens if I get the numbers wrong?

  1. If the input queue is slow to process messages, or the depth is often too high, you may have a problem.
  2. If ejb-jar.xml maxPoolDepth is too small, this will limit the number of messages you can process concurrently.
  3. The weblogic max-beans-in-free-pool is too small. If the all the beans in the pool(array) are busy, consider making the pool bigger.   The requests queue in the listeners waiting for a free MDB instance.   However the JMX data has fields with names like “Wait count”.   In my tests these were always zero, so I think these fields are of no value.
  4. The number of connections in the connectionFactory is too small.  If the number of requests exceeded the pool size the MDB instance got an exception.  MQJCA1011: Failed to allocate a JMS connection.  You need to change the resource adapter definition Max Capacity for the connectionFactory pool size.
  5. If you find you have many MQDISC and MQCONNs happening at the same instance, consider increasing the initial size of the connectionFactory pool.
  6. Make the initial values suitable for your average workload.  This will prevent  the periodic destroy and recreate of the connections and beans.

 

You may want to have more than one weblogic server for availability and scalability.

You could also deploy the same application with a different MDB name, so if you want to stop and restart an MDB, you have another MDB processing messages.

Are all your jms messages persistent?

While debugging my application to see why it was so slow, I found from the MQ activity trace that my replies were all persistent.

The first problem was that by default all jms messages are persistent, so I used

int deliveryMode = message.getJMSDeliveryMode();

to get the persistence of the input message,

and used the obvious code to set the JMSDeliveryMode,

TextMessage response = session.createTextMessage("my reply");
response.setJMSDeliveryMode(deliveryMode);

to set it the same as the input message.  I reran my test and the reply was still persistent.

Eventually I found you need

producer = session.createProducer(dest);
producer.setDeliveryMode(deliveryMode);

And this worked!  It is all explained here.

How do I check?

You can either check your code (bearing in mind that this may be hidden by the productivity tools you use ( Swing, Camel etc)), or turn on activity trace for a couple of seconds to check.