WebLogic message does not have authorization to view the logs

I was using JMX to display the connectionPool statistics  from webLogic, and kept getting messages

<Error> <Diagnostics> <BEA-320084> <The user principals=[] does not have authorization to view the logs.>

I solved this by using the webLogic console

  • Click “Security Realms” in the Domain Stucture box on the left hand side of the home page
  • Click on the name of your realm (myrealm)
  • Click on “Roles and Policies”
  • Click on “Realm Policies”
  • Expand Domain and select “View Policy Conditions” for “View Log”
  • Click on Add Condition
  • Select the role
  • Click on Finish
  • Click on Save

The change is available immediately on “Save”.

Did the JMS architects get it wrong? Possibly!

I was looking into a performance problem where a web server was doing 1 million MQCONNects  day!  During my investigations I found that the original designers did not design it properly because of the performance overhead, and so the JMS architects had to fix it up by creating connection factories.

Below, I cover

  • the performance problem
  • writing my own connection factory
  • the problems of writing my own connection factory
  • it might just be easier to use a connection factory provided by your web server or JMS provider.

What is an EJB and an MDB?

In a web server you can have Enterprise Java Beans (EJBs) which are a package of java applications doing enterprise type work ( get a message, update a database, send a reply) conforming to an specific Programming Interface.

A Message Driven Bean is an EJB which responds to messages.  A message listener applications gets messages from a queue – and passes the message to the MDB. In non EJB terms this is just like a trigger monitor, starting a transaction and passing the data.

When you create the MDB you have a configuration file  which allows a message listener to be created, and the parameters to use to connect to MQ, and  how many threads etc can be running concurrently.

The MDB, to process the message has a basic structure of

  • method onMessage – this is given the message
  • ejbCreate – when the EJB is created, you can do initialisation here
  • ejbRemove – when the EJB is about to be deleted, you can do clean up here

and looks like

public class MDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -8070254332864574796L;
  public void onMessage(Message message)
  {
    Connection connection = null;
    Session session = null;
    MessageProducer producer = null;

    try
    {
      InitialContext ctx = new InitialContext();
      ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF2");
      Destination dest = message.getJMSReplyTo();
     
      connection = cf.createConnection();
      session = connection.createSession(false, 1);
      producer = session.createProducer(dest);
      
      TextMessage response = session.createTextMessage("test response message from the WMQ_IVT_MDB");
      response.setJMSCorrelationID(message.getJMSMessageID());
      producer.send(response);
      return;
    }
    catch (Exception je)
    {
      System.out.println("Something went wrong." + je);
    }
  }

  public void ejbCreate() { }
  public void ejbRemove() { }
 }

Within the onMessage method is logic connection = cf.createConnection();
Under the covers this createConnection() does an MQCONN.  With 1 million messages a day – this is 1 million MQCONNs!
There was also an MQOPEN, an MQCLOSE,  and an MQDISC before the onMessage() method returned.  This results in a huge performance overhead.

Attempts to fix the performance

People quickly found that this model was not efficient, and they came up with ways to improve the performance.

The idea of a connection pool was developed.  Instead of doing an the createConnection() doing an MQCONN etc.  this code was changed, to not do the MQDISC, but to keep the Queue Manager handle. When the next MQCONN occurred, if there was one of these spare Queue Manager handles it would use it.

These connection pools are sophisticated.   For example they can free up connections if they had not been used for a time period, so reducing the overall number of Queue Manager connections in use, they can slowly increase the number of connections in the pool so there is not a sudden spike in requests.

I found it difficult to set up a connection pooling in Oracle’s webLogic webserver,  so I thought I would write my own.   This was pretty easy, but then I discovered some complications.

Writing my own “connection pooling”.

It was clear to me that having connection = cf.createConnection();     from the ejbCreate() method rather than in the onMessage() method would be much more efficient.

  • Move variables from being method variables to instance variables so they persisted across onMessage() calls.
  • Create a connect() method which actually does the createConnection().  This can be called in the ejbCreate setup routine, and from the onMessage() if the connection variable is null.
  • Have code in the ejbRemove to the connection, session etc.
public class IVTMDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -337338331639L;
  // create long lasting instance variables 
  Connection connection = null;
  Session session = null;
  MessageProducer producer = null;
  InitialContext ctx = null;
  ConnectionFactory cf = null;
  // new method to do the connect
  public void connect(){
  try {
     ctx = new InitialContext();
     cf = (ConnectionFactory)ctx.lookup("CF2");
     connection = cf.createConnection();
     session = connection.createSession(false, 1); 
    }
    catch (Exception je)
    {...
    } 
 }

 public void ejbCreate() {
   connect();
 }
...
}

This worked well.    The number of MQCONN dropped to about 10 per day for many puts.

The problem looked solved, until I tried to shut down the queue manager.

If no work was being processed, then the MDBs were not being called, and so there were no MQ requests being made, and so the connections stayed active, and the queue manager did not shut down.

To solve this you can set  cf.setExceptionListener(..) which will get notified if there are problems with the connection, and so you can close the connection etc.
You need to consider what to do if the connection cannot be made;  do you wait for a short time period, or do you return an error.

I found that this “simple” way of avoiding all of the MQCONNects was quickly getting much more complex.  It was going to be much easier in the long term to get the provided connection factories working.   This was another challenge, taking about a week.  See the next few blog entries on how I did this for webLogic.

Are your mirrored file systems consistent?

It started with a question “Several years ago you told us about checking your MQ disks are consistent,  can you provide us with a link to any documentation please?”.

I’ll explain why this is important and what you need to do to ensure you have data integrity and you do not lose data integrity when you go to a backup site.

With some applications that write to multiple files, the order that data is actually written to the disk does not matter.  For example when you print data, it often stays in a buffer, and is written out when the buffer is full.

A transaction manager

With programs that handle transactions (a transaction manager) it is critical that writes to disk are done in the order they are issued.  If the writes are not in the correct order then if there if the system crashes and tries to restore the transaction the recovery may be missing key data  (“it has taken the money from your account..  it cannot see who should get the money?”) and so data integrity is lost.

With local disks, the sequence is

  • Write to file1,
  • Wait for confirmation that the IO has completed
  • Write to file2,
  • Wait for confirmation that the IO has completed

Consider the case where file1 and file 2 are on different file systems.  For example file1 could be transaction log, file2 could be queue data.  (Picture file system1 on slow disks, and file system 2 on fast disks – so IO for file 2 is faster than IO to file 1).

With mirrored disks with synchronous replication, the sequence is

  • Write to file1 local copy; send data to remote site,  write to file1, send back OK when completed
  • Wait for confirmation that both IOs have completed
  • Write to file2 local copy,send data to remote site,  write to file2, send back OK when completed
  • Wait for confirmation that both IOs have completed

With synchronous replication the two locations need to be within 10s of kilometers.  The response time of the file write depends on the distance.

With Asynchronous replication the two locations can be 100s of kilometers apart.

In this case the sequence is

  • Write to file1 local copy; send data to remote site,  write to file1, send back OK when completed
  • Wait for confirmation that the local IO has completed.
  • Write to file2 local copy,send data to remote site,  write to file2, send back OK when completed
  • Wait for confirmation that the local IO has completed.

The disk subsystems manages the responses coming back from the remote end.

For capacity reasons there are usually multiple paths between the two sites.  It is possible that the data for file 2 gets there before the data for file1.  If the writes are done in the wrong order, this could be bad news.

Consistency group

The architecture of the mirroring systems have the concept of a consistency group.   You define one or more consistency groups.  You put file systems into a consistence group.  For any files in the consistency group the write order will be honoured.  So in the case above, if the two files are in the same consistency group, it will wait, write the data to file 1, then write to file 2.  This gives a solution with data integrity.

The lurking problem.

Someone needs to define the file systems to each consistency group.   The storage manager may have said

  • “all file systems are part of one consistency group”.
  • “”production data is in one consistency group, test data is in another consistency group”
  • “I’ll guess, and hope people tell me their requirements”

How will I know if I have a problem?

The sure fire way of finding out if you have a problem is to lose a site ( for example a power outage).  For 99 times out of a 100 it may be fine, and then one time in a hundred, you find you cannot restart your systems on the other site.  This is clearly the wrong time to find out.

Check with your storage administrator and give them information about the file systems that need to be part of the same consistency group.

Practice your fail over – perhaps weekly – at least monthly.

How to get my enterprise talking to your enterprise over MQ.

No,  this is not about Star Trek’s “beam me up Scottie”, but setting up MQ to MQ communications.

Of course you will want

  1. it to scale – you should be able to define more channels if you need more throughput than one channel can provide
  2. be secure – you can use TLS to protect the data on the channels
  3. be highly available – this is the trickier challenge which I’ll cover below.

Should I use clustering?

At first glance you may think clustering would be a good solution.   Many institutions are unhappy with being in a cluster with their business partner, so it is rare to have two enterprises both in the same cluster.

This means you are usually looking a gateway between the two institutions, which means point-to-point.

There are two parts to being highly available,

  1. How long does it take before messages in the queue manager can flow out of the queue manager?
  2. How long does it take before new work can flow?

How long does it take before messages in the queue manager can flow out of the queue manager?

If your queue manager has ended, the messages are not available until the queue manager (and the channels) area available again.

If you are using z/OS and shared queue etc – other queue managers in the queue sharing group should be able to process the work.

If you are using mid-range MQ, you an use multi instance queue managers. One queue manager is active, the other is in standby,  partially initialized, ready to take over once a failure has been detected.

How long does it take before new work can flow?

If you have only one queue manager, you have to wait until it restarts before work can flow.  If you have more than one queue manager, you can switch message to avoid the down queue manager, by stopping channels into your gateway queue manager.  With clustering within your organization the messages should then flow to another gateway queue manager.

This will much quicker than doing an emergency recreate of your queue manager!

 

What are the JMS connection factories on webLogic doing?

As part of my long running activity to find out what is causing 1 million MQCONNects a day from an Oracle  webLogic web server, I have found out how to monitor what is going on inside the webLogic instance.

Most web servers support a Java Management eXtension (JMX) interface.  You can use gui tools like jconsole to do an ad-hoc display of the management beans – but these are   not practical for long term monitoring.

Ive listed the data from the JMX query, what the data means, and also documented how I got the data.

There are three components to an MDB

  1. A listening task waits to be notified about messages arriving on the queue.   As far as I can tell there is one thread doing this work.  If you need more I think you need to create a second MDB.
  2. Multiple MDB threads which get woken up by the listener task, get a message and invoke the MDB application OnMessage() method with the message.  The data for this part has Type=EJBPoolRuntime.
  3. The application part which processes the message – typically it connects to MQ and puts a reply, and disconnects.  This application specified which connection pool to be used using ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF3”);  The data for this part has Type=ConnectorConnectionPoolRuntime. 

 

The multiple MDB threads.

This data came from the JMX record with com.bea:ServerRuntime=AdminServer2,MessageDrivenEJBRuntime=WMQ_IVT_MDB,Name=WMQ_IVT_MDB,ApplicationRuntime=MDB3,Type=EJBPoolRuntime,EJBComponentRuntime=MDB3

Where

  1.  ServerRuntime=AdminServer2. This is where the application runs
  2. MessageDrivenEJBRuntime=WMQ_IVT_MDB comes from  display-name WMQ_IVT_MDB in the ejb-jar.xml file.,Name=WMQ_IVT_MDB,
  3. ApplicationRuntime=MDB3, This is the name of the deployed .jar file.
  4. Type=EJBPoolRuntime, this is the type for the MDB threads
  5. EJBComponentRuntime=MDB3, this is the name of the deployed .jar file.

From experiementing with MDBs and adjusting parameters my picture of how the EJB thread pool works is as follows

  1. There is a general free pool for threads.
  2. There is a pool for the EJB.
  3. When a message arrives the listener thread gets a thread from the EJB pool, and executes the OnMessage() method on the thread.
  4. If there are not available threads,
    1. it waits for a free thread.
    2. if, over a period of seconds, most of the requests have had to wait for a thread, then start a new thread
      1. it gets a thread from the general pool, if none available it creates a new thread.
      2. it executes the MDB application ejbCreate() method on the thread just obtained.
      3. It executes the OnMessage() method on the thread.
      4. Puts the thread into the EJB pool as a free thread.
    3. In the ejbCreate() method, I had it write to the job log.  I could see threads started, after
  5. Periodically the EJB pool is cleaned up,  for threads which have been idle for a specific time interval
    1. execute ejbRemove() on the thread
    2. return the thread to the general free pool
  6. Periodically the general pool is cleaned up
    1. Threads which have not been used for a while are disconnected from the queue manager, and deleted.

The data came from the JMX with the defintion

com.bea:ServerRuntime=AdminServer2, Name=WMQ_IVT_MDB, ApplicationRuntime=MDB3, Type=MessageDrivenEJBRuntime, EJBComponentRuntime=MDB3

 

See here for the documentation.

  • AccessTotalCount = 28180,   There were this many messages processed, and  so this many OnMessage() methods executed
  • BeansInUseCount  = 2.  Deprecated
  • BeansInUseCurrentCount  = 1  One thread is currently active.
  • DestroyedTotalCount   = 0.   
  • IdleBeansCount  = 4 There are 4 free thread available in the EJB pool
  • MissTotalCount  = 19.  For 19 (out of the 28180) requests, they failed to get a free thread, and so a new thread was obtained.
  • Name  = WMQ_IVT_MDB
  • PooledBeansCurrentCount  = 3.   The numer of free threads inthe pool.
  • TimeoutTotalCount  = 0.  Number of requests timed out waiting for a thread from the free pool.
  • Type  = EJBPoolRuntime.   You can use Type=EJBPoolRuntime in the JMX query string.
  • WaiterCurrentCount  = 0. The number of requests waiting for a thread in the pool.   This value should always be zero.
  • WaiterTotalCount = 0. The total number of threads that have had to wait in the pool.  This value should always be zero, if it is non zero you need to tune your connection pool.

The data for the connection factory.

See here for some documentation.

  • ActiveConnectionsCurrentCount (Inte
  • ger) = 9.  How many connections were in use when the JMX request was issued.
  • ActiveConnectionsHighCount (Integer) = 10. The highest number of connections in use since the connection pool was started or reset
  • AverageActiveUsage (Integer) = 0. This value is always zero.
  • CapacityIncrement (Integer) = 1.   This is the value the connection pool is configured with
  • CloseCount (Long) = 5682.    How many connection.close() requests have been issued.   This will be less than or equal to ConnectionsMatchedTotalCount below.
  • ConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ConnectionFactoryImpl.   This is the name of the connection class being used
  • ConnectionFactoryName (Null).  I could not find where to change this, or if it has any value
  • ConnectionIdleProfileCount (Integer) = 0.   This was always zero.
  • ConnectionIdleProfiles (Null).   This was always zero.
  • ConnectionLeakProfileCount (Integer) = 0.   This was always zero.  I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • ConnectionLeakProfiles (Null).  This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • ConnectionProfilingEnabled (Boolean) = false.   I could not see how to change this, or what value it adds
  • Connections (ObjectName[]) = [Ljavax.management.ObjectName;@299a06ac]  This will be an internal object within Java.
  • ConnectionsCreatedTotalCount (Integer) = 21. This is the count of instances started.   You could get connection started, connection freed, connection started, connection freed.   This would could  2 connections started.
  • ConnectionsDestroyedByErrorTotalCount (Integer) = 0.
  • ConnectionsDestroyedByShrinkingTotalCount (Integer) = 11.  After a time period, idle-timeout-seconds in weblogic-ejb-jar.xml, connections that have not processed messages for this time period are released back to the general pool.
  • ConnectionsDestroyedTotalCount (Integer) = 11.  This seems to be the same as the previous item
  • ConnectionsMatchedTotalCount (Integer) = 5670.    This many requests for a connection with matching userid etc were got from the pool
  •  ConnectionsRejectedTotalCount (Integer) = 0.  I dont know what this is
  • ConnectorEisType (String) = Java Message Service
  • CurrentCapacity (Long) = 10.   This is the size of the pool
  • EISResourceId (String) = type=<eis>, application=colin, module=colin, eis=Java
  • Message Service, destinationId=CF2
  • FreeConnectionsCurrentCount (Integer) = 1.   There is currently one free connection in the pool
  • FreeConnectionsHighCount (Integer) = 10.  This is the highest number of free connections in the pool – when there was no requests for the connection pool.
  • FreePoolSizeHighWaterMark (Long) = 10.  I dont know the difference between this and the previous item.
  • FreePoolSizeLowWaterMark (Long) = 0.   This is the lowest number of free connections – it is the gap between in use and the maximum pool size.
  • HealthState (Null).    I do not think connection pools for MDBs support this
  • HighestNumWaiters (Long) = 0.   This value was always zero.  Even when I reduced the size of the connection pool, and specified 50 threads could wait, this value was still 0
  • InitialCapacity (Integer) = 1.   This is the specified value.   This is used  to set a lower limit of connections in the pool.  Setting the initial capacity to 5 means there will always be at least 5 connections in the pool, and the pool will not be shrunk below this value
  • JNDIName (String) = CF2.  This is what is specified in the connection pool definition.   This label is used in the application ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF2”);
  • Key (String) = CF2.  This seems to the be same as the JNDI name.
  • LastShrinkTime (Long) = 1565334529469.  This is the number of milliseconds POSIX time (since Jan 1st 1970).  To format it use  time_t now = t/1000; strftime(buff, 20, “%H:%M:%S”, localtime(&now));
  • LogFileName (Null).  You can specify logging for this adapter.  Deployments -> resource adapter -> configuration -> Outbound Connection Pools, javax.jsm.ConnectionFactory, select the connection pool,  Logging tab.  When this was active I got nothing logged in it.
  • LogFileStreamOpened (Boolean) = false
  • LoggingEnabled (Boolean) = true
  • LogRuntime (Null)
  • ManagedConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl.  This is the name of the resouce adapter class.
  • MaxCapacity (Integer) = 20.   This is the maximum size of the pool.
  • MaxIdleTime (Integer) = 0.   This was always zero for me.
  • MCFClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl.   This is the ManagedConnectionFactoryClassName. Same as above.
  • Name (String) = CF2.  Another connection pool identifier
  • NumberDetectedIdle (Integer) = 0.   This was always zero.
  • NumberDetectedLeaks (Integer) = 0.  This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • NumUnavailableCurrentCount (Integer) = 9.   This feels like the number threads waiting while the connection pool creates a connection in the connection pool.
  • NumUnavailableHighCount (Integer) = 10.   This feels like the number threads waiting while the connection pool creates a connection in the connection pool.
  • NumWaiters (Long) = 0.   This was always zero – even through I restricited the number of connections in the pool.  I got a java exception, unable to get a connection, rather than have the thread wait.
  • NumWaitersCurrentCount (Integer) = 0.  See above.
  • Parent (ObjectName) = com.bea:ServerRuntime=AdminServer2,Name=colin, ApplicationRuntime=colin, Type=ConnectorComponentRuntime
  • PoolName (String) = CF2.   Yet another field with the name of the connection pool/
  • PoolSizeHighWaterMark (Long) = 10.   This is the highest number of connections used in the pool
  • PoolSizeLowWaterMark (Long) = 0.       This is the lowest number of connecitons used in pool
  • ProxyOn (Boolean) = false.    I could not find what this means
  • RecycledTotal (Integer) = 0.    This was always zero for me.
  • ResourceAdapterLinkRefName (Null).   You can specify this field name in your MDB definition.
  • RuntimeTransactionSupport (String) = NoTransaction.  This defines if your connections are part of a unit of work or not. NoTransaction means out of syncpoint.    You may want to specify. XATransaction.
  • ShrinkCountDownTime (Integer) = 340.  This is how long before a scan of the pool to remove any threads in the pool which have done no work for the specified interval.
  • ShrinkingEnabled (Boolean) = true
  • ShrinkPeriodMinutes (Integer) = 15.  You can specify the interval on the resource adapter definition  (Note: you specify it in seconds – it gets reported in minutes)
  • State (String) = Running.    This connection pool is started.
  • Testable (Boolean) = false.   You can defined some resources as “testable”
  • TransactionSupport (String) = NoTransaction.   This is the same as RuntimeTransactionSupport above.
  • Type (String) = ConnectorConnectionPoolRuntime.   This defines the object.   You can use the Type=ConnectorConnectionPoolRuntime  in the JMX query.

From this I can see there is a pool called CF2 which has maximum of 10 connections.  The maximum connections used was 10, the lowest was 0.

There were Connections Matched Total Count  = 5670 requests for a connection from the pool.

The pool has shrunk more than once as it has Connections Destroyed By Shrinking Total Count  = 11 connections.

Using this data, you can now plot usage over time and see if you need to increase ( or decrease) the size of the pool, or the parameters to tune when the pool is shrunk.

I do not know enough about JMX to tell if the “high” and “low” value are reset on each query, or if you can use JMX to reset them periodically. These high and low value may have little value, if they are since the webLogic instance started (6 months ago).

The data fields are mentioned here.

How I got the data

There is a python package called JMXQuery which has a .jar file which allows you to query information in a JMX server.  The output is in json format so you can use your favourite tools (python) to quickly convert this to other format, such as .csv .

The command I used was

java  -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u webLogic
-p passw0rd -q “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”

which breaks down as follows

  • java – invoke java
  • -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” – this jar file
  • -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi – this is the url of my webLogic server
  • -json – output it in json format
  • -u webLogic -p passw0rd -q  – userid and password
  • “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”
    • com.bea is the bean type
    • The admin server was called AdminServer2
    • The connection factory was CF2
    • The resource adapter is installed under “Deployments”  as colin
    • Type=ConnectorConnectionPoolRuntime is the type of bean

I then used |jq . |grep -v mBean > bb to convert the one line json to one field per line, dropped the mBean value, and put it to a file.  The output was like

[
 {
   "attribute": "Connections",
   "attributeType": "ObjectName[]",
   "value": "[Ljavax.management.ObjectName;@299a06ac"
 },
 {
   "attribute": "FreeConnectionsCurrentCount",
   "attributeType": "Integer",
   "value": 4
},
etc

You can put generics in for example

java  -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u readonly -p read0nly -q “com.bea:ApplicationRuntime=*,ConnectorComponentRuntime=*,Name=”CP*”,ServerRuntime=*,Type=ConnectorConnectionPoolRuntime” e=*,Type=ConnectorConnectionPoolRuntime” > aa

 

Where this uses a userid set up as a monitor id, “*” has been specified for many values, and only give objects beginning with “CP”.  Note blanks have meaning. “,,,=*, Name=…” looks for an object with blank,N,a,m,e,

Data when not using a resource adapter

The above information was for a resource adapter.  When an EJB 2 MDB is deployed (non resource adapter)

from –q  “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3,
MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,
Name=WMQ_IVT_MDB_JMSQ1,ServerRuntime=AdminServer2,Type=EJBPoolRuntime”

Where the weblogic-jar-xml has

<weblogic-ejb-jar> <ejb-name>WMQ_IVT_MDB</ejb-name>
<destination-jndi-name>JMSQ1</destination-jndi-name>

The data was

BeansInUseCount (Integer) = 0
PooledBeansCurrentCount (Integer) = 10
IdleBeansCount (Integer) = 10
BeansInUseCurrentCount (Integer) = 0
DestroyedTotalCount (Long) = 0
WaiterCurrentCount (Integer) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
MissTotalCount (Long) = 7
AccessTotalCount (Long) = 28
Type (String) = EJBPoolRuntime
TimeoutTotalCount (Long) = 0
WaiterTotalCount (Long) = 0

With

-q “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3, MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,Name=WMQ_IVT_MDB_JMSQ1,
ServerRuntime=AdminServer2,Type=EJBTransactionRuntime

the output was

TransactionsRolledBackTotalCount (Long) = 0
TransactionsTimedOutTotalCount (Long) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
Type (String) = EJBTransactionRuntime
TransactionsCommittedTotalCount (Long) = 0

Configuring Oracle WebLogic to use MQ Message Drive Beans

I wanted to use webLogic web server from Oracle to look into a performance problem with Java Message Driven Beans using IBM MQ. Setting this up was harder than I expected, because of lack of documentation. Some documentation gave half the story – but nothing gave the complete story.
It took me several days to configure Oracle webLogic server with IBM MQ. I went down many rabbit holes, and had many problems, before finding that, once you know what to ignore, it is not too difficult.
I used

  1. A queue manager accessible from a client connection
  2. A “server queue” Q1
  3. A reply to queue CP0000
  4. A MDB application – I used an IBM IVT file as the basis of my program. This was an existing MDB jar file which had some a webLogic configuration file added to the .jar file
  5. An existing JNDI repository configured with a connection factory and “server queue”.
    1.  My definitions were stored in a file file:/home/colinpaice/JNDI-Directory and this needs the java class com.sun.jndi.fscontext.RefFSContextFactory to access it.
    2.  The JNDI repository had a connection factory CF1 which points to the channel definition, and a queue definition JMSQ1 mapping to queue Q1 in the queue manager.
  6.  A webLogic server. I used WebLogic Server Version: 12.2.1.3.0, also called 12c. I created a new domain, and used this. I logged onto the admin console using URL http://127.0.0.1:7001/console/ .

Note: The JNDI repository is configured in two places.

  1. Within the MDB configuration definitions.
  2.  Within MDB java program logic

Create a profile script for webLogic

I created a configuration script,  here, where I defined Java overrides and gave the webLogic server access to the MQ libraries.
I configured it to use JMX (Java Management eXtensions) which allow you to use a utility called jconsole to display information about what is happening within the web server.

I used the cd command to get to the weblogic directory and used
.    sh profile.sh
to execute the profile

Start the webLogic server

From the weblogic directory
./wls12213/user_projects/domains/colin/startWebLogic.sh 1>a 2>b
1>a and 2>b is useful so you can use another command window to search for lines in the output.

Go to the webLogic admin console in a web browser http://127.0.0.1:7001/console.

Setting up the JMS Server

You do not need to define a webLogic JMS Server to use IBM MQ.

Understanding terms and concepts

I used the webLogic Admin console, taking default names when offered.
A group of deployable MQ resources are defined within a “JMS Module”. You can define different modules and deploy them in different places.

I found that sometimes when I changed the configuration (for example to correct a spelling mistake) the server ignored the update till I restarted the server – despite the web server saying there were no pending changes.

It was safer to restart the server after I changed the configuration. It felt like that when an application is deployed, it remembers the configuration at the time of deployment, rather than looking it up each time.

WebLogic adds another level of indirection to the application. It took me a while to sort this out.

1. The application uses the name of a connection factory, MYCF
2. MYCF is looked up in the “JMS Modules” “ForeignConnectionFactory” “Local JNDI Name” to give the “Remote JNDI Name” (REMCF).
3. “ForeignConnectionFactory”.General gives the location of the JNDI repository to use.
4. The “Remote JNDI Name” (REMCF) is looked up in the JNDI repository to give the channel name and other parameters. (Are you keeping up?)

I found it easiest to call everything the same name eg CF1, and it made understanding the configuration much easier.

My application prints out information about the connection factory. So I could see
com.ibm.mq.jms.MQConnectionFactor – it was MQ, not the webLogic connection factory
XMSC_WMQ_CHANNEL :- SYSTEM.DEF.SVRCONN
XMSC_WMQ_QUEUE_MANAGER :- QMA

Define MQ resources to webLogic.

Home -> Messaging → JMS Modules

New

Name : SystemModule-0
Next

Target: select from the pull down, Adminserver
Next
Select “Would you like to add resources to this JMS system module?”
Finish

Select the JMS module you just created (SystemModule-0)
Summary of resources
New
Select: Foreign server
Next

Name: ForeignServer-0

Finish

Click ForeignServer-0

JNDI Initial Context Factory: com.sun.jndi.fscontext.RefFSContextFactory
JNDI Connection URL: file:/home/colinpaice/JNDI-Directory
credential leave emtpy
Save

Click ‘Connection Factories’ tab
New
Name: ForeignConnectionFactory-0
LocalJNDI name: CF1
This is what my application running on webLogic will use. See the weblogic-ejb-jar.xml file below.

Remote JNDI name: CF1
This is the name in the JNDI repostitory
user name webLogic
The user name that is used to logon to admin console
password
Confirm password

Save

You can use “view changes and restarts” to see what webLogic thinks needs to be restarted.

I found it safer to restart the server, as sometimes changes were not picked up. Ctrl-c from the command window running the webLogic server seems to work, but it is better to use a different window and use
./wls12213/user_projects/domains/base_domain/bin/stopWebLogic.sh
then back to the orignal window and issue

./wls12213/user_projects/domains/base_domain/startWebLogic.sh 1>a 2>b

The application

The application, here, was based on the IBM provided sample for MQ on WAS

in the directory MDB
… META-INF/
… META-INF/MANIFEST.MF
… META-INF/weblogic-ejb-jar.xml
… META-INF/ejb-jar.xml
… ejbs/
… ejbs/IVTMDB.class
… ejbs/IVTMDB.java
…javax.ejb-api-3.2.jar

The ejb-jar.xml file is here.

Each web server seems to have their own way of configuring application. JBoss web server has its own file, and WAS has configuration information in the server.xml file. The webLogic file is given below

… META-INF/weblogic-ejb-jar.xml had

<?xml version=”1.0″?>
<!DOCTYPE weblogic-ejb-jar
PUBLIC ‘-//BEA Systems, Inc.//DTD WebLogic 8.1.0 EJB//EN’
http://www.bea.com/servers/wls810/dtd/weblogic-ejb-jar.dtd’&gt;

<weblogic-ejb-jar>

<weblogic-enterprise-bean>
<!– I think the value of this ejb-name matches the value in META-INF/ejb-jar.xml ->
<ejb-name>WMQ_IVT_MDB</ejb-name>

<message-driven-descriptor>
<!– define the number of listener threads that can be used, maximum and initial –>
<pool>
<max-beans-in-free-pool>10</max-beans-in-free-pool>
<initial-beans-in-free-pool>3</initial-beans-in-free-pool>
</pool>

<!– JNDI defintitions →
<!– Specify the class needed to process the JNDI repository –>
<initial-context-factory>com.sun.jndi.fscontext.RefFSContextFactory

</initial-context-factory>
<!– specify which JNDI repository to use –>
<provider-url>file:/home/colinpaice/JNDI-Directory</provider-url>

<!– look in the JNDI file for the definitions with JMSQ1 –>
<destination-jndi-name>JMSQ1</destination-jndi-name>

<!– look in the JNDI file for the connection factory → MQ channel etc–>
<connection-factory-jndi-name>CF1</connection-factory-jndi-name>

</message-driven-descriptor>
</weblogic-enterprise-bean>
</weblogic-ejb-jar>

Compile the program

javac -cp /opt/mqm/java/lib/*:javax.ejb-api-3.2.jar: ejbs/IVTMDB.java

The javax.ejb-api-3.2.jar came from
https://download.jar-download.com/cache_jars/javax.ejb/javax.ejb-api/3.2/jar_files.zip

Build the jar file

jar -cvf MDB3.jar ejbs/IVTMDB.class META-INF/weblogic-ejb-jar.xml META-INF/ejb-jar.xml

jar -tvf MDB3.jar gave

0 Fri Aug 02 07:08:38 BST 2019 META-INF/
69 Fri Aug 02 07:08:38 BST 2019 META-INF/MANIFEST.MF
3572 Fri Aug 02 07:05:10 BST 2019 ejbs/IVTMDB.class
1197 Wed Jul 31 17:43:08 BST 2019 META-INF/weblogic-ejb-jar.xml
3696 Tue Jul 30 15:16:52 BST 2019 META-INF/ejb-jar.xml

Deploy the MDB

Using the webLogic admin console
Home → Deployment resources → Deployments

Install
enter the path to the directory with your jar file
select jar file
Install as application
Next
Name MDB3
Finish

Check the webLogic error log for problems.

For example, the log showed

<BEA-160228><AppMerge failed to merge your application. If you are running AppMerge on the command-line, merge again with the -verbose option for more details. See the error message(s) below.>

I went into a command window.
Set up the environment using
.      ./wls12213/wlserver/server/bin/setWLSEnv.sh
and ran

java weblogic.appmerge -verbose ~/temp/MDB/MDB3.jar
I fixed any problems.

( If you get Error: Could not find or load main class weblogic.appmerge, you did not enter the setWLSEnv.sh command properly, check the . before the command.)

When it successfully deployed, the webLogic log had
EJBCreate
EJBCreate
EJBCreate
This was 3 instances ( as defined in the weblogic-ejb-jar.xml file) of my program starting up and printing out a message.

Try it out!

My JNDI file had JMSQ1 → Q1.  I ran a program to put a message, to queue Q1, specifying a reply to queue, and it worked!

I put a message to Q1, and a response was returned.

Thanks to Paul Titherage who gave me good advice on getting the Java program compiled, and other pointers to the correct path.

The fog around Message Driven Beans (MDBs)

While playing with Message Driven Beans and looking at performance problems, I have learned a lot about MDBs and tuning.   The documentation was only partly useful.   There is a lot of documentation  – some concepts are explained 1000 times, but other areas are undocumented – or just wrong.  I’ll try to fill the holes in this blog post.  If I am wrong – please tell me!

This blog post is just one step on the path to how to configure webLogic server to avoid millions of connects a day, and to see if you need to do any tuning ( from an MQ perspective)

I’ll cover

  1. How MDBs work
  2. Different types of Enterprise Java bean standards, and how configuration of EJBs have changed over the years.
  3. Use of resource adapters.

Message listener

We should all be familiar with an application that does an MQGET, a database update, and a commit – which commits both units of work.  A transaction manager manages the units of work with each resource manager (MQ, and database).

The same concept applies to the MDB listener running in a web server.  The listener gets a message from one queue manager, and calls the MDB on a different thread, with the message content.  The MDB connects to a (possibly different) queue manager puts the reply and returns. The listener does a commit. The web server acting as a transaction manager coordinates the commit in both queue managers.  If the application does a roll back – the whole request can be backed out (if the MDB is set up for this).

You can configured the listener tasks so the gets are in syncpoint or not, and the MDB can do its processing in sycnpoint or not.

There are two distinct parts to the MDB.

The listener task described above, and the application code.   This application code does not use the same MQ connection as the listener code.   This may be obvious to most people, but I found documents on the web discussing performance tuning, which said create a connection pool for the listener side of the MDB (which may connect once a day and so a connection pool may not be needed) and did not mention a connection pool for the application side (which does 1 million puts, causing 1 million connects a day and so needs a connection pool!).

The Enterprise Java Bean architecture.

From the dawn of web servers, the architecture has followed the classic pattern.

  1. This is great – every one develops their own way of doing things.  Let each web server manufacturer have their own way of doing this.    Let JMS be different to database access.
  2. The realization that this is not a good idea and we need standards and commonality as there is basically one problem.
  3. Now we have standards and people going along the road together, we find there are improvements we need to make.  It could also be that there is a new leader who says “let us go in a different direction – the grass is greener over here”.  You have to maintain the old way of doing things, and also swallow the new way of doing things.

How does this impact MDBs?

EJB 2.0

With EJB 2. MDBs you have information stored in a JNDI repository.   This could be file based on your local machine, or in LDAP.  In this repository you have

  • a queue resource MYJMSQ and information about it, the MQ Queue name is SERVER, and other parameters.
  • connection information MYCONN, this contains the channel name, host name, port etc.

You configure your MDB with

  1. The name of the class (your program) to use, and the method to be passed messages (this is usually onMessage)
  2. The location of the JNDI repository
  3. What sort of repository it is (what Java class you need to communicate with it)
  4. What connection to use
  5. What queue to use.

The ejb-jar.xml file has “standard” information including the class name  and the method name.
Each web server has their own way of passing in the other information.  For example webLogic has a file weblogic-ejb-jar.xml with information about the location of the JNDI repository, the JNDI connection (MYCONN), and JNDI queue name(MYJMSQ).   It also has information about the connection pool for the listener.  All these files are packaged into the .jar file for the MDB.

IBM’s Liberty web server stores the information elsewhere.

The application has code like

ctx = new InitialContext();
cf = (ConnectionFactory)ctx.lookup(“CF1”);

which says look in the same JNDI repository, for connection CF1.   This could be same connection name as the listener is using, it could be different.  I could not find a way of specifying connection pool information for this connection, so by default every message put to MQ requires an MQCONN and MQDISC.

The application can get the destination name of the replyTo Queue, and uses it

Destination dest = message.getJMSReplyTo();
producer = session.createProducer(dest);
producer.send(response);

The Destination dest will have a value like queue://QMA/QUEUENAME

EJB 3.0

With EJB version 3, a lot has changed.

  1. There is a standard Java Connector Adapter( JCA) I think version 1.5 is current.
  2. The hard work is now done by using a resource adapter (ra)
  3. This has a different way of passing configuration information
  4. You can now hard code data in your program (which seems a step backwards to me)

Key information is stored in activationSpecs.  You can code it in your java application program.   (The resource adapter can query all the activationConfigProperties from a class and then process it), or have it in the .xml configuration files.

For example with the information in your application.java file.

@MessageDriven(
  name = "JMSSampleMDB",
  activationConfig = {
    @ActivationConfigProperty(propertyName  = "destinationType", 
               propertyValue = "javax.jms.Queue"),
     @ActivationConfigProperty(propertyName  = "destination", 
	       propertyValue = "INPUT_Q")                         
  }
)
public class JMSSampleMDB implements MessageListener{
 @TransactionAttribute(value = TransactionAttributeType.REQUIRED)
  public void onMessage(Message message) {...
	 }
}

It can be stored in the ejb-jar.xml file

<activation-config>
  <activation-config-property> 
    <activation-config-property-name>connectionfactory</activation-config-property-name> 
    <activation-config-property-value>CF3</activation-config-property-value>
  </activation-config-property>
  <activation-config-property> 
    <activation-config-property-name>destination</activation-config-property-name> 
    <activation-config-property-value>JMSQ2</activation-config-property-value>
  </activation-config-property>
</activation-config>

The parameters CF3 and JMSQ2 are configured as part of the resource adapter, and not in JNDI.  Each resource adapter has an ra.xml file.  Weblogic stores its specific information in weblogic-ra.xml.  IBM Liberty stores it in the server.xml file.

Once you have installed the resource adapter you should then be able to add additional connections and queues to it.

The resoure adapter takes the definitions and puts them into the JNDI tree within the web server.   Your application can continue to use

ctx = new InitialContext();
cf = (ConnectionFactory)ctx.lookup(“CF1”);

Using this way you can configure your connection as part of a connection pool- and avoid the MQCONN and MQDISC request.

I believe you can define an MDB using definitions in JNDI – and the application part of the MDB can use the resource adapter.

As far as I have been able to find out – if you want to use connection pooling for your application program you need to use a resource adapter.  I’ll blog this in a separate topic.  It is complex and quirky, and will need a lot of description