Any port in a storm? No.

Ive just spent a day resolving a problem with specifying a port value trying to connect to MQ.

I had

public long port = 1414;
String channel = “MYCHANNEL”;
String hostname = “127.0.0.1”;
Hashtable<String, Object> h = new Hashtable<String, Object>();
h.put(MQConstants.PORT_PROPERTY, dd.port);h.put(MQConstants.CHANNEL_PROPERTY, channel);
h.put(MQConstants.TRANSPORT_PROPERTY, MQConstants.TRANSPORT_MQSERIES_CLIENT);
h.put(MQConstants.HOST_NAME_PROPERTY, hostname);
queueManager = new MQQueueManager(“QMA”,h);

(did you spot the problem?)

This failed with

MQConnection to QMA com.ibm.mq.MQException: MQJE001: Completion Code ‘2’, Reason ‘2538’.
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host ‘127.0.0.1(0)’ rejected.

This is saying it tried to connect with port 0!

I tried

String port = “1414”;, that failed the same way.

If I used

MQEnvironment.port=”1414″; it worked.

This was tough to resolve, as there is no documentation to help me.

Someone suggested public int port = 1414; and it worked!  What a way to spend a nice autumn day.

Whoops -deploying MDB in weblogic

I was quite happily using my MDB in webLogic, but when I changed its configuration, it did not pick up the new changes.  It took a day to find out why,  and I have learned much more about deploying MDBS.

My connection factory was using SYSTEM.DEF.SVRCONN, I changed it to use a different client channel. I stopped SYSTEM.DEF.SVRCONN, ( so I could check the change had worked), and restarted the webLogic instance.  I was surprised when my MDB failed to start, because the channel was stopped.   The MDB was trying to use that channel.  It took a lot of head scratching to get it to work as I expected.

  • I had messages like <BEA-015073>  Message-Driven Bean …  is configured with unknown activation-config-property name failIfQuiesce.  This message is wrong, failIfQuiesce is supported by the IBM Resource adapter.
  • I had the same message with activation-config-property name cfLookup.   This was my problem.  I should have specified connectionFactoryLookup.
  • If you have <activation-config-property-name>connectionFactoryLookup… (specified in the ejb-jar) any other parameters you specify in the ejb-jar.xml file are ignored.
  • If you do not specify a connectionFactoryLookup, nor properties in the ejb-jar.xml file, defaults are provided, see Configuring the resource adapter for inbound communication.  In my case I had not specified  activation-config-property-name channel, and this defaulted to SYSTEM.DEF.SVRCONN, which is why it continued to use that channel.
  • It worth putting <activation-config-property-name>applicationName … in your definitions so you can see what you are using.
    • dis qstatus(JMSQ3) type(handle) gave me APPLTAG(CF3Name) so I can tell which definitions are being used.
    • If you get APPLTAG(weblogic.Server) then you are taking the defaults.
  • The Oracle documentation  says the precedence order is as below.    I do not think this is 100% accurate. (I could not specify some of the parameters on the weblogic-ejb-jar.xml file).  I didnt try the java program.
    1. properties set in the weblogic-ejb-jar.xml deployment descriptor
    2. activation-config-property properties set in the ejb-jar.xml deployment descriptor
    3. activationConfigProperty annotation properties in the java program.

What do I need to specify?

As a minimum you need to use connectionFactoryLookup or  specify

  1. applicationName – so you can identify which definitions are being used
  2. channel – which channel to use
  3. failIfQuiesce
  4. hostName
  5. port

 

The ejb-jar.xml file is in the META-INF directory.  Change  the ejb-jar.xml or  weblogic-ejb-jar.xml file. IUpdate the jar file using a command like jar -uvf MDB4.jar  META-INF/ejb-jar.xml,   and redeploy it.

WebLogic message does not have authorization to view the logs

I was using JMX to display the connectionPool statistics  from webLogic, and kept getting messages

<Error> <Diagnostics> <BEA-320084> <The user principals=[] does not have authorization to view the logs.>

I solved this by using the webLogic console

  • Click “Security Realms” in the Domain Stucture box on the left hand side of the home page
  • Click on the name of your realm (myrealm)
  • Click on “Roles and Policies”
  • Click on “Realm Policies”
  • Expand Domain and select “View Policy Conditions” for “View Log”
  • Click on Add Condition
  • Select the role
  • Click on Finish
  • Click on Save

The change is available immediately on “Save”.

Did the JMS architects get it wrong? Possibly!

I was looking into a performance problem where a web server was doing 1 million MQCONNects  day!  During my investigations I found that the original designers did not design it properly because of the performance overhead, and so the JMS architects had to fix it up by creating connection factories.

Below, I cover

  • the performance problem
  • writing my own connection factory
  • the problems of writing my own connection factory
  • it might just be easier to use a connection factory provided by your web server or JMS provider.

What is an EJB and an MDB?

In a web server you can have Enterprise Java Beans (EJBs) which are a package of java applications doing enterprise type work ( get a message, update a database, send a reply) conforming to an specific Programming Interface.

A Message Driven Bean is an EJB which responds to messages.  A message listener applications gets messages from a queue – and passes the message to the MDB. In non EJB terms this is just like a trigger monitor, starting a transaction and passing the data.

When you create the MDB you have a configuration file  which allows a message listener to be created, and the parameters to use to connect to MQ, and  how many threads etc can be running concurrently.

The MDB, to process the message has a basic structure of

  • method onMessage – this is given the message
  • ejbCreate – when the EJB is created, you can do initialisation here
  • ejbRemove – when the EJB is about to be deleted, you can do clean up here

and looks like

public class MDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -8070254332864574796L;
  public void onMessage(Message message)
  {
    Connection connection = null;
    Session session = null;
    MessageProducer producer = null;

    try
    {
      InitialContext ctx = new InitialContext();
      ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF2");
      Destination dest = message.getJMSReplyTo();
     
      connection = cf.createConnection();
      session = connection.createSession(false, 1);
      producer = session.createProducer(dest);
      
      TextMessage response = session.createTextMessage("test response message from the WMQ_IVT_MDB");
      response.setJMSCorrelationID(message.getJMSMessageID());
      producer.send(response);
      return;
    }
    catch (Exception je)
    {
      System.out.println("Something went wrong." + je);
    }
  }

  public void ejbCreate() { }
  public void ejbRemove() { }
 }

Within the onMessage method is logic connection = cf.createConnection();
Under the covers this createConnection() does an MQCONN.  With 1 million messages a day – this is 1 million MQCONNs!
There was also an MQOPEN, an MQCLOSE,  and an MQDISC before the onMessage() method returned.  This results in a huge performance overhead.

Attempts to fix the performance

People quickly found that this model was not efficient, and they came up with ways to improve the performance.

The idea of a connection pool was developed.  Instead of doing an the createConnection() doing an MQCONN etc.  this code was changed, to not do the MQDISC, but to keep the Queue Manager handle. When the next MQCONN occurred, if there was one of these spare Queue Manager handles it would use it.

These connection pools are sophisticated.   For example they can free up connections if they had not been used for a time period, so reducing the overall number of Queue Manager connections in use, they can slowly increase the number of connections in the pool so there is not a sudden spike in requests.

I found it difficult to set up a connection pooling in Oracle’s webLogic webserver,  so I thought I would write my own.   This was pretty easy, but then I discovered some complications.

Writing my own “connection pooling”.

It was clear to me that having connection = cf.createConnection();     from the ejbCreate() method rather than in the onMessage() method would be much more efficient.

  • Move variables from being method variables to instance variables so they persisted across onMessage() calls.
  • Create a connect() method which actually does the createConnection().  This can be called in the ejbCreate setup routine, and from the onMessage() if the connection variable is null.
  • Have code in the ejbRemove to the connection, session etc.
public class IVTMDB
implements MessageListener, MessageDrivenBean
{
  private static final long serialVersionUID = -337338331639L;
  // create long lasting instance variables 
  Connection connection = null;
  Session session = null;
  MessageProducer producer = null;
  InitialContext ctx = null;
  ConnectionFactory cf = null;
  // new method to do the connect
  public void connect(){
  try {
     ctx = new InitialContext();
     cf = (ConnectionFactory)ctx.lookup("CF2");
     connection = cf.createConnection();
     session = connection.createSession(false, 1); 
    }
    catch (Exception je)
    {...
    } 
 }

 public void ejbCreate() {
   connect();
 }
...
}

This worked well.    The number of MQCONN dropped to about 10 per day for many puts.

The problem looked solved, until I tried to shut down the queue manager.

If no work was being processed, then the MDBs were not being called, and so there were no MQ requests being made, and so the connections stayed active, and the queue manager did not shut down.

To solve this you can set  cf.setExceptionListener(..) which will get notified if there are problems with the connection, and so you can close the connection etc.
You need to consider what to do if the connection cannot be made;  do you wait for a short time period, or do you return an error.

I found that this “simple” way of avoiding all of the MQCONNects was quickly getting much more complex.  It was going to be much easier in the long term to get the provided connection factories working.   This was another challenge, taking about a week.  See the next few blog entries on how I did this for webLogic.

Are your mirrored file systems consistent?

It started with a question “Several years ago you told us about checking your MQ disks are consistent,  can you provide us with a link to any documentation please?”.

I’ll explain why this is important and what you need to do to ensure you have data integrity and you do not lose data integrity when you go to a backup site.

With some applications that write to multiple files, the order that data is actually written to the disk does not matter.  For example when you print data, it often stays in a buffer, and is written out when the buffer is full.

A transaction manager

With programs that handle transactions (a transaction manager) it is critical that writes to disk are done in the order they are issued.  If the writes are not in the correct order then if there if the system crashes and tries to restore the transaction the recovery may be missing key data  (“it has taken the money from your account..  it cannot see who should get the money?”) and so data integrity is lost.

With local disks, the sequence is

  • Write to file1,
  • Wait for confirmation that the IO has completed
  • Write to file2,
  • Wait for confirmation that the IO has completed

Consider the case where file1 and file 2 are on different file systems.  For example file1 could be transaction log, file2 could be queue data.  (Picture file system1 on slow disks, and file system 2 on fast disks – so IO for file 2 is faster than IO to file 1).

With mirrored disks with synchronous replication, the sequence is

  • Write to file1 local copy; send data to remote site,  write to file1, send back OK when completed
  • Wait for confirmation that both IOs have completed
  • Write to file2 local copy,send data to remote site,  write to file2, send back OK when completed
  • Wait for confirmation that both IOs have completed

With synchronous replication the two locations need to be within 10s of kilometers.  The response time of the file write depends on the distance.

With Asynchronous replication the two locations can be 100s of kilometers apart.

In this case the sequence is

  • Write to file1 local copy; send data to remote site,  write to file1, send back OK when completed
  • Wait for confirmation that the local IO has completed.
  • Write to file2 local copy,send data to remote site,  write to file2, send back OK when completed
  • Wait for confirmation that the local IO has completed.

The disk subsystems manages the responses coming back from the remote end.

For capacity reasons there are usually multiple paths between the two sites.  It is possible that the data for file 2 gets there before the data for file1.  If the writes are done in the wrong order, this could be bad news.

Consistency group

The architecture of the mirroring systems have the concept of a consistency group.   You define one or more consistency groups.  You put file systems into a consistence group.  For any files in the consistency group the write order will be honoured.  So in the case above, if the two files are in the same consistency group, it will wait, write the data to file 1, then write to file 2.  This gives a solution with data integrity.

The lurking problem.

Someone needs to define the file systems to each consistency group.   The storage manager may have said

  • “all file systems are part of one consistency group”.
  • “”production data is in one consistency group, test data is in another consistency group”
  • “I’ll guess, and hope people tell me their requirements”

How will I know if I have a problem?

The sure fire way of finding out if you have a problem is to lose a site ( for example a power outage).  For 99 times out of a 100 it may be fine, and then one time in a hundred, you find you cannot restart your systems on the other site.  This is clearly the wrong time to find out.

Check with your storage administrator and give them information about the file systems that need to be part of the same consistency group.

Practice your fail over – perhaps weekly – at least monthly.

How to get my enterprise talking to your enterprise over MQ.

No,  this is not about Star Trek’s “beam me up Scottie”, but setting up MQ to MQ communications.

Of course you will want

  1. it to scale – you should be able to define more channels if you need more throughput than one channel can provide
  2. be secure – you can use TLS to protect the data on the channels
  3. be highly available – this is the trickier challenge which I’ll cover below.

Should I use clustering?

At first glance you may think clustering would be a good solution.   Many institutions are unhappy with being in a cluster with their business partner, so it is rare to have two enterprises both in the same cluster.

This means you are usually looking a gateway between the two institutions, which means point-to-point.

There are two parts to being highly available,

  1. How long does it take before messages in the queue manager can flow out of the queue manager?
  2. How long does it take before new work can flow?

How long does it take before messages in the queue manager can flow out of the queue manager?

If your queue manager has ended, the messages are not available until the queue manager (and the channels) area available again.

If you are using z/OS and shared queue etc – other queue managers in the queue sharing group should be able to process the work.

If you are using mid-range MQ, you an use multi instance queue managers. One queue manager is active, the other is in standby,  partially initialized, ready to take over once a failure has been detected.

How long does it take before new work can flow?

If you have only one queue manager, you have to wait until it restarts before work can flow.  If you have more than one queue manager, you can switch message to avoid the down queue manager, by stopping channels into your gateway queue manager.  With clustering within your organization the messages should then flow to another gateway queue manager.

This will much quicker than doing an emergency recreate of your queue manager!

 

What are the JMS connection factories on webLogic doing?

As part of my long running activity to find out what is causing 1 million MQCONNects a day from an Oracle  webLogic web server, I have found out how to monitor what is going on inside the webLogic instance.

Most web servers support a Java Management eXtension (JMX) interface.  You can use gui tools like jconsole to do an ad-hoc display of the management beans – but these are   not practical for long term monitoring.

Ive listed the data from the JMX query, what the data means, and also documented how I got the data.

There are three components to an MDB

  1. A listening task waits to be notified about messages arriving on the queue.   As far as I can tell there is one thread doing this work.  If you need more I think you need to create a second MDB.
  2. Multiple MDB threads which get woken up by the listener task, get a message and invoke the MDB application OnMessage() method with the message.  The data for this part has Type=EJBPoolRuntime.
  3. The application part which processes the message – typically it connects to MQ and puts a reply, and disconnects.  This application specified which connection pool to be used using ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF3”);  The data for this part has Type=ConnectorConnectionPoolRuntime. 

 

The multiple MDB threads.

This data came from the JMX record with com.bea:ServerRuntime=AdminServer2,MessageDrivenEJBRuntime=WMQ_IVT_MDB,Name=WMQ_IVT_MDB,ApplicationRuntime=MDB3,Type=EJBPoolRuntime,EJBComponentRuntime=MDB3

Where

  1.  ServerRuntime=AdminServer2. This is where the application runs
  2. MessageDrivenEJBRuntime=WMQ_IVT_MDB comes from  display-name WMQ_IVT_MDB in the ejb-jar.xml file.,Name=WMQ_IVT_MDB,
  3. ApplicationRuntime=MDB3, This is the name of the deployed .jar file.
  4. Type=EJBPoolRuntime, this is the type for the MDB threads
  5. EJBComponentRuntime=MDB3, this is the name of the deployed .jar file.

From experiementing with MDBs and adjusting parameters my picture of how the EJB thread pool works is as follows

  1. There is a general free pool for threads.
  2. There is a pool for the EJB.
  3. When a message arrives the listener thread gets a thread from the EJB pool, and executes the OnMessage() method on the thread.
  4. If there are not available threads,
    1. it waits for a free thread.
    2. if, over a period of seconds, most of the requests have had to wait for a thread, then start a new thread
      1. it gets a thread from the general pool, if none available it creates a new thread.
      2. it executes the MDB application ejbCreate() method on the thread just obtained.
      3. It executes the OnMessage() method on the thread.
      4. Puts the thread into the EJB pool as a free thread.
    3. In the ejbCreate() method, I had it write to the job log.  I could see threads started, after
  5. Periodically the EJB pool is cleaned up,  for threads which have been idle for a specific time interval
    1. execute ejbRemove() on the thread
    2. return the thread to the general free pool
  6. Periodically the general pool is cleaned up
    1. Threads which have not been used for a while are disconnected from the queue manager, and deleted.

The data came from the JMX with the defintion

com.bea:ServerRuntime=AdminServer2, Name=WMQ_IVT_MDB, ApplicationRuntime=MDB3, Type=MessageDrivenEJBRuntime, EJBComponentRuntime=MDB3

 

See here for the documentation.

  • AccessTotalCount = 28180,   There were this many messages processed, and  so this many OnMessage() methods executed
  • BeansInUseCount  = 2.  Deprecated
  • BeansInUseCurrentCount  = 1  One thread is currently active.
  • DestroyedTotalCount   = 0.   
  • IdleBeansCount  = 4 There are 4 free thread available in the EJB pool
  • MissTotalCount  = 19.  For 19 (out of the 28180) requests, they failed to get a free thread, and so a new thread was obtained.
  • Name  = WMQ_IVT_MDB
  • PooledBeansCurrentCount  = 3.   The numer of free threads inthe pool.
  • TimeoutTotalCount  = 0.  Number of requests timed out waiting for a thread from the free pool.
  • Type  = EJBPoolRuntime.   You can use Type=EJBPoolRuntime in the JMX query string.
  • WaiterCurrentCount  = 0. The number of requests waiting for a thread in the pool.   This value should always be zero.
  • WaiterTotalCount = 0. The total number of threads that have had to wait in the pool.  This value should always be zero, if it is non zero you need to tune your connection pool.

The data for the connection factory.

See here for some documentation.

  • ActiveConnectionsCurrentCount (Inte
  • ger) = 9.  How many connections were in use when the JMX request was issued.
  • ActiveConnectionsHighCount (Integer) = 10. The highest number of connections in use since the connection pool was started or reset
  • AverageActiveUsage (Integer) = 0. This value is always zero.
  • CapacityIncrement (Integer) = 1.   This is the value the connection pool is configured with
  • CloseCount (Long) = 5682.    How many connection.close() requests have been issued.   This will be less than or equal to ConnectionsMatchedTotalCount below.
  • ConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ConnectionFactoryImpl.   This is the name of the connection class being used
  • ConnectionFactoryName (Null).  I could not find where to change this, or if it has any value
  • ConnectionIdleProfileCount (Integer) = 0.   This was always zero.
  • ConnectionIdleProfiles (Null).   This was always zero.
  • ConnectionLeakProfileCount (Integer) = 0.   This was always zero.  I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • ConnectionLeakProfiles (Null).  This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • ConnectionProfilingEnabled (Boolean) = false.   I could not see how to change this, or what value it adds
  • Connections (ObjectName[]) = [Ljavax.management.ObjectName;@299a06ac]  This will be an internal object within Java.
  • ConnectionsCreatedTotalCount (Integer) = 21. This is the count of instances started.   You could get connection started, connection freed, connection started, connection freed.   This would could  2 connections started.
  • ConnectionsDestroyedByErrorTotalCount (Integer) = 0.
  • ConnectionsDestroyedByShrinkingTotalCount (Integer) = 11.  After a time period, idle-timeout-seconds in weblogic-ejb-jar.xml, connections that have not processed messages for this time period are released back to the general pool.
  • ConnectionsDestroyedTotalCount (Integer) = 11.  This seems to be the same as the previous item
  • ConnectionsMatchedTotalCount (Integer) = 5670.    This many requests for a connection with matching userid etc were got from the pool
  •  ConnectionsRejectedTotalCount (Integer) = 0.  I dont know what this is
  • ConnectorEisType (String) = Java Message Service
  • CurrentCapacity (Long) = 10.   This is the size of the pool
  • EISResourceId (String) = type=<eis>, application=colin, module=colin, eis=Java
  • Message Service, destinationId=CF2
  • FreeConnectionsCurrentCount (Integer) = 1.   There is currently one free connection in the pool
  • FreeConnectionsHighCount (Integer) = 10.  This is the highest number of free connections in the pool – when there was no requests for the connection pool.
  • FreePoolSizeHighWaterMark (Long) = 10.  I dont know the difference between this and the previous item.
  • FreePoolSizeLowWaterMark (Long) = 0.   This is the lowest number of free connections – it is the gap between in use and the maximum pool size.
  • HealthState (Null).    I do not think connection pools for MDBs support this
  • HighestNumWaiters (Long) = 0.   This value was always zero.  Even when I reduced the size of the connection pool, and specified 50 threads could wait, this value was still 0
  • InitialCapacity (Integer) = 1.   This is the specified value.   This is used  to set a lower limit of connections in the pool.  Setting the initial capacity to 5 means there will always be at least 5 connections in the pool, and the pool will not be shrunk below this value
  • JNDIName (String) = CF2.  This is what is specified in the connection pool definition.   This label is used in the application ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF2”);
  • Key (String) = CF2.  This seems to the be same as the JNDI name.
  • LastShrinkTime (Long) = 1565334529469.  This is the number of milliseconds POSIX time (since Jan 1st 1970).  To format it use  time_t now = t/1000; strftime(buff, 20, “%H:%M:%S”, localtime(&now));
  • LogFileName (Null).  You can specify logging for this adapter.  Deployments -> resource adapter -> configuration -> Outbound Connection Pools, javax.jsm.ConnectionFactory, select the connection pool,  Logging tab.  When this was active I got nothing logged in it.
  • LogFileStreamOpened (Boolean) = false
  • LoggingEnabled (Boolean) = true
  • LogRuntime (Null)
  • ManagedConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl.  This is the name of the resouce adapter class.
  • MaxCapacity (Integer) = 20.   This is the maximum size of the pool.
  • MaxIdleTime (Integer) = 0.   This was always zero for me.
  • MCFClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl.   This is the ManagedConnectionFactoryClassName. Same as above.
  • Name (String) = CF2.  Another connection pool identifier
  • NumberDetectedIdle (Integer) = 0.   This was always zero.
  • NumberDetectedLeaks (Integer) = 0.  This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • NumUnavailableCurrentCount (Integer) = 9.   This feels like the number threads waiting while the connection pool creates a connection in the connection pool.
  • NumUnavailableHighCount (Integer) = 10.   This feels like the number threads waiting while the connection pool creates a connection in the connection pool.
  • NumWaiters (Long) = 0.   This was always zero – even through I restricited the number of connections in the pool.  I got a java exception, unable to get a connection, rather than have the thread wait.
  • NumWaitersCurrentCount (Integer) = 0.  See above.
  • Parent (ObjectName) = com.bea:ServerRuntime=AdminServer2,Name=colin, ApplicationRuntime=colin, Type=ConnectorComponentRuntime
  • PoolName (String) = CF2.   Yet another field with the name of the connection pool/
  • PoolSizeHighWaterMark (Long) = 10.   This is the highest number of connections used in the pool
  • PoolSizeLowWaterMark (Long) = 0.       This is the lowest number of connecitons used in pool
  • ProxyOn (Boolean) = false.    I could not find what this means
  • RecycledTotal (Integer) = 0.    This was always zero for me.
  • ResourceAdapterLinkRefName (Null).   You can specify this field name in your MDB definition.
  • RuntimeTransactionSupport (String) = NoTransaction.  This defines if your connections are part of a unit of work or not. NoTransaction means out of syncpoint.    You may want to specify. XATransaction.
  • ShrinkCountDownTime (Integer) = 340.  This is how long before a scan of the pool to remove any threads in the pool which have done no work for the specified interval.
  • ShrinkingEnabled (Boolean) = true
  • ShrinkPeriodMinutes (Integer) = 15.  You can specify the interval on the resource adapter definition  (Note: you specify it in seconds – it gets reported in minutes)
  • State (String) = Running.    This connection pool is started.
  • Testable (Boolean) = false.   You can defined some resources as “testable”
  • TransactionSupport (String) = NoTransaction.   This is the same as RuntimeTransactionSupport above.
  • Type (String) = ConnectorConnectionPoolRuntime.   This defines the object.   You can use the Type=ConnectorConnectionPoolRuntime  in the JMX query.

From this I can see there is a pool called CF2 which has maximum of 10 connections.  The maximum connections used was 10, the lowest was 0.

There were Connections Matched Total Count  = 5670 requests for a connection from the pool.

The pool has shrunk more than once as it has Connections Destroyed By Shrinking Total Count  = 11 connections.

Using this data, you can now plot usage over time and see if you need to increase ( or decrease) the size of the pool, or the parameters to tune when the pool is shrunk.

I do not know enough about JMX to tell if the “high” and “low” value are reset on each query, or if you can use JMX to reset them periodically. These high and low value may have little value, if they are since the webLogic instance started (6 months ago).

The data fields are mentioned here.

How I got the data

There is a python package called JMXQuery which has a .jar file which allows you to query information in a JMX server.  The output is in json format so you can use your favourite tools (python) to quickly convert this to other format, such as .csv .

The command I used was

java  -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u webLogic
-p passw0rd -q “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”

which breaks down as follows

  • java – invoke java
  • -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” – this jar file
  • -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi – this is the url of my webLogic server
  • -json – output it in json format
  • -u webLogic -p passw0rd -q  – userid and password
  • “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”
    • com.bea is the bean type
    • The admin server was called AdminServer2
    • The connection factory was CF2
    • The resource adapter is installed under “Deployments”  as colin
    • Type=ConnectorConnectionPoolRuntime is the type of bean

I then used |jq . |grep -v mBean > bb to convert the one line json to one field per line, dropped the mBean value, and put it to a file.  The output was like

[
 {
   "attribute": "Connections",
   "attributeType": "ObjectName[]",
   "value": "[Ljavax.management.ObjectName;@299a06ac"
 },
 {
   "attribute": "FreeConnectionsCurrentCount",
   "attributeType": "Integer",
   "value": 4
},
etc

You can put generics in for example

java  -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u readonly -p read0nly -q “com.bea:ApplicationRuntime=*,ConnectorComponentRuntime=*,Name=”CP*”,ServerRuntime=*,Type=ConnectorConnectionPoolRuntime” e=*,Type=ConnectorConnectionPoolRuntime” > aa

 

Where this uses a userid set up as a monitor id, “*” has been specified for many values, and only give objects beginning with “CP”.  Note blanks have meaning. “,,,=*, Name=…” looks for an object with blank,N,a,m,e,

Data when not using a resource adapter

The above information was for a resource adapter.  When an EJB 2 MDB is deployed (non resource adapter)

from –q  “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3,
MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,
Name=WMQ_IVT_MDB_JMSQ1,ServerRuntime=AdminServer2,Type=EJBPoolRuntime”

Where the weblogic-jar-xml has

<weblogic-ejb-jar> <ejb-name>WMQ_IVT_MDB</ejb-name>
<destination-jndi-name>JMSQ1</destination-jndi-name>

The data was

BeansInUseCount (Integer) = 0
PooledBeansCurrentCount (Integer) = 10
IdleBeansCount (Integer) = 10
BeansInUseCurrentCount (Integer) = 0
DestroyedTotalCount (Long) = 0
WaiterCurrentCount (Integer) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
MissTotalCount (Long) = 7
AccessTotalCount (Long) = 28
Type (String) = EJBPoolRuntime
TimeoutTotalCount (Long) = 0
WaiterTotalCount (Long) = 0

With

-q “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3, MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,Name=WMQ_IVT_MDB_JMSQ1,
ServerRuntime=AdminServer2,Type=EJBTransactionRuntime

the output was

TransactionsRolledBackTotalCount (Long) = 0
TransactionsTimedOutTotalCount (Long) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
Type (String) = EJBTransactionRuntime
TransactionsCommittedTotalCount (Long) = 0