How long will it take my queue manager to fail over and restart on midrange?

Following on from my blog post and making sure your file systems are part of a consistency group – so the data is consistent after a restart, the next question is

“how long will it take to fail over?”.

There are two areas you need to look at

  1. The time to detect and outage.  This can be broken down into the time the active queue manager releases the lock, and the time taken for the release of the lock to be reflected to the standby system.  You need to test and measure this time.  For example you may need to adjust your network configuration.
  2. The time taken to restart the queue manager.  There is an excellent blog post on this from Ant at IBM.
    1. The blog post talks about the rate at which clients can connect to MQ.  Yes MQ can support 10,000 client connections.  But if it takes a significant time to reconnect them all, you may want multiple queue managers, and have parallelism
    2. Avoid deep queues.  In my time at IBM I saw many customers with thousands of messages on a queue with an age over 1 year old!  You need to clean up the queue.  Your applications team should have a process that runs perhaps once a day which cleans up all old messages.  For example there was a getting application instance which timed out and terminated, then the reply arrived.
    3. During normal running most of the IO is write – and this often goes into cache, so very fast.   During recovery the IO is reading from disk which might be from rotating disks rather than solid state.

One lesson from this is you need to test the recover.   You need to test a realistic scenario – take the worst case of number of connections, with peak throughput, and then pull the plug on the active system.

Another lesson is you need to do this regularly – for example monthly as configurations can change.

Are your mirrored file systems consistent?

It started with a question “Several years ago you told us about checking your MQ disks are consistent,  can you provide us with a link to any documentation please?”.

I’ll explain why this is important and what you need to do to ensure you have data integrity and you do not lose data integrity when you go to a backup site.

With some applications that write to multiple files, the order that data is actually written to the disk does not matter.  For example when you print data, it often stays in a buffer, and is written out when the buffer is full.

A transaction manager

With programs that handle transactions (a transaction manager) it is critical that writes to disk are done in the order they are issued.  If the writes are not in the correct order then if there if the system crashes and tries to restore the transaction the recovery may be missing key data  (“it has taken the money from your account..  it cannot see who should get the money?”) and so data integrity is lost.

With local disks, the sequence is

  • Write to file1,
  • Wait for confirmation that the IO has completed
  • Write to file2,
  • Wait for confirmation that the IO has completed

Consider the case where file1 and file 2 are on different file systems.  For example file1 could be transaction log, file2 could be queue data.  (Picture file system1 on slow disks, and file system 2 on fast disks – so IO for file 2 is faster than IO to file 1).

With mirrored disks with synchronous replication, the sequence is

  • Write to file1 local copy; send data to remote site,  write to file1, send back OK when completed
  • Wait for confirmation that both IOs have completed
  • Write to file2 local copy,send data to remote site,  write to file2, send back OK when completed
  • Wait for confirmation that both IOs have completed

With synchronous replication the two locations need to be within 10s of kilometers.  The response time of the file write depends on the distance.

With Asynchronous replication the two locations can be 100s of kilometers apart.

In this case the sequence is

  • Write to file1 local copy; send data to remote site,  write to file1, send back OK when completed
  • Wait for confirmation that the local IO has completed.
  • Write to file2 local copy,send data to remote site,  write to file2, send back OK when completed
  • Wait for confirmation that the local IO has completed.

The disk subsystems manages the responses coming back from the remote end.

For capacity reasons there are usually multiple paths between the two sites.  It is possible that the data for file 2 gets there before the data for file1.  If the writes are done in the wrong order, this could be bad news.

Consistency group

The architecture of the mirroring systems have the concept of a consistency group.   You define one or more consistency groups.  You put file systems into a consistence group.  For any files in the consistency group the write order will be honoured.  So in the case above, if the two files are in the same consistency group, it will wait, write the data to file 1, then write to file 2.  This gives a solution with data integrity.

The lurking problem.

Someone needs to define the file systems to each consistency group.   The storage manager may have said

  • “all file systems are part of one consistency group”.
  • “”production data is in one consistency group, test data is in another consistency group”
  • “I’ll guess, and hope people tell me their requirements”

How will I know if I have a problem?

The sure fire way of finding out if you have a problem is to lose a site ( for example a power outage).  For 99 times out of a 100 it may be fine, and then one time in a hundred, you find you cannot restart your systems on the other site.  This is clearly the wrong time to find out.

Check with your storage administrator and give them information about the file systems that need to be part of the same consistency group.

Practice your fail over – perhaps weekly – at least monthly.

Using webLogic web server and the IBM MQ resource adapter.

Ive documented the steps I took to configure webLogic to use the IBM MQ Resource Adapter, so I could use connection pools and avoid 1 million MQCONNs a day.
The document grew and grew, and I found it easiest to create a PDF document, here.

The document covers

  • Getting the IBM Resource Adapter from IBM
  • Adding the webLogic specific file into the Resource Adapter.
  • Deploying the resource adapter
  • Adding additional configurations to the Resource Adapter
  • Changing your MDB deployment to use the Resource Adapter instead of native JMS
  • How  you tell if a connnectionFactory is using the Resource Adapter and not native JMS.

I provide a sample weblogic-ra file to get you started.

Making MQ swing with Spring. Tuning Camel and Spring (to get rid of the 1 second wait in every message).

Spring is a framework which sits on top of Apache Camel, which runs on Oracle WebLogic web server.   It simplifies writing java applications for processing messages.

I was involved in tracking down some performance problems when the round trip time for a simple application was over 1 second – coming from a z/OS background I thought 50 milliseconds was a long time – so over 1 second is an age!

The application is basically a Message Drive Bean.  It does

  1. Listening applications get a message from a queue.  There  can be multiple listening threads
  2. Get an application thread
  3. Pass it via an internal queue to the application running on another thread – the listening thread is blocked while the application is running.
  4. This application sends (puts) a message to an MQ queue for the backend server and waits for the response.
  5. Return to the listening application
  6. Free the application thread.

As this is essentially the same as a classic MDB, we had configured the number of application threads in the thread pool the same number as the listener thread pool.

Shortage of threads

The symptoms of the problem looked like a shortage of threads problem.

When we increased the number of threads in the application pool ( we gave it 4* the number of listener threads) The response time dropped – good news.  I dont know how many threads you need – is it n+1 or 2* n. I’ll leave finding the right number  as an exercise for the reader!

The hard coded 1 second wait before get

One symptom we saw was the queue depth on the replyTo queue on the server was deeper than normal.

For the reply coming back from the server, I believe there is one thread getting from the queue.

When the reply to queue is not shared

The application thread has sent the request off, and is now waiting.  This getting thread does an MQ destructive GET with wait.  When the message arrives, it looks at the content, and decides which application thread is waiting for the reply, and wakes it up.

When the reply queue is shared between instances

For example you have configured two instances for availability.  The above logic is not used.

Instance1 cannot destructively get a message because the message could be destined for instance2.  Similarly instance2 cannot get destructively get the message because it could be destined for instance1.

One way to solve this would be to do a get next browse of the message, and if it is for the instance do a get_message_under_cursor.  This works great for MQ, but not other message providers which do not have this capability.

The implementation used  is to use polling!

If there are 3 applications tasks waiting for a replies, reply1, reply3, reply4.  The logic is as follows

  1. For each reply id, use MQ message selectors to try to get reply1, reply3, reply4.   This is not a get by messageID or correlID – for which the queue manager has indexes to quickly locate the message, this is a get with message selector, which means look in every message on the queue.
  2. For any messages found – pass them to the waiting applications
  3. Do an MQGET with a message selector which does not exist – with a wait time of receiveTimeout (defaults to 1 second).  Every message is scanned looking for the message selector string, and it is not found.

Looking at a time line – in seconds and tenths of seconds
0.1 send request1, wait for reply
0.2 getting task does MQGET  wait for 1 second with non exising selector
0.2 send request 2, wait for reply
0.3 reply1 comes back
0.4 send request3, wait for reply
0.5 reply2 comes back
0.6 reply3 comes back
nothing happens
1.2 getting task waiting for 1 second times out
1.2 getting tasks gets reply1 and posts task1
1.2 getting tasks gets reply2 and posts task2
1.3 getting tasks gets reply3 and posts task3

So although  reply1 was back within 0.2 seconds (0.3 – 0.1) it was not got until time 1.2.  The message had been waiting on the queue for 0.9 seconds.

  • Total wait time for reply1 was  1.1 seconds.
  • Total wait time for reply2 was  1.2 – 0.2 = 1.0 seconds
  • Total wait time for reply3  was 1.3 – 0.4 = 0.9 seconds

Wow – what a response time killer!

You can tune this time by specifying receiveTimeout.  If you make it smaller, the wait time will be shorter, so messages will be processed faster, but the CPU cost will go up as more empty gets are being done.

This solution does not scale.

You have had a slow down, and there are now 1000 messages on this queue. (990 of these are unprocessed messages, due to timeout .  There is no task waiting for them – they never get processed – nor do they expire!)

  • MQGET for reply1.  This scans all 1000 messages – looking in each message for the message with the matching selector.  This takes 0.2 seconds.
  • MQGET for reply2. This scans all 1000 messages – looking in each message for the message with the matching selector.  This takes 0.2 seconds.
  • You have 10 threads waiting for messages, so each message has to wait for typically 10 * 0.2 seconds = 2 seconds a message!

 

What can you do about it.

See the Camel documentation Request-Reply over JMS, parameters, Exclusive, concurrentConsumers, and Request-Reply over JMS Using an Exclusive Fixed Reply Queue 

  1. Avoid sharing the queue.  Give each instance its own queue.  Set the Exclusive flag
  2. Tune the ReceiveTime out – making it a shorter time interval can increase the CPU as you are doing more empty gets.  You might want to set it, to a value which is 95% percentile time between the send to the server, and the reply comes back.  So if the average time is 40 ms, set it to 60 ms or 80 ms.
  3. If you are going to share the queue, make sure you clean out old messages from the queue – for example use expiry, or have a process which periodically scans the queue and moves messages older than 1 minute.
  4. Did I mention avoid sharing the queue.
  5. If you get into a vicious spiral where the response time gets longer and longer, and the reply queue from the server gets deeper – be brave and purge server reply queue.
  6. Avoid sharing the queue.

 

How to get my enterprise talking to your enterprise over MQ.

No,  this is not about Star Trek’s “beam me up Scottie”, but setting up MQ to MQ communications.

Of course you will want

  1. it to scale – you should be able to define more channels if you need more throughput than one channel can provide
  2. be secure – you can use TLS to protect the data on the channels
  3. be highly available – this is the trickier challenge which I’ll cover below.

Should I use clustering?

At first glance you may think clustering would be a good solution.   Many institutions are unhappy with being in a cluster with their business partner, so it is rare to have two enterprises both in the same cluster.

This means you are usually looking a gateway between the two institutions, which means point-to-point.

There are two parts to being highly available,

  1. How long does it take before messages in the queue manager can flow out of the queue manager?
  2. How long does it take before new work can flow?

How long does it take before messages in the queue manager can flow out of the queue manager?

If your queue manager has ended, the messages are not available until the queue manager (and the channels) area available again.

If you are using z/OS and shared queue etc – other queue managers in the queue sharing group should be able to process the work.

If you are using mid-range MQ, you an use multi instance queue managers. One queue manager is active, the other is in standby,  partially initialized, ready to take over once a failure has been detected.

How long does it take before new work can flow?

If you have only one queue manager, you have to wait until it restarts before work can flow.  If you have more than one queue manager, you can switch message to avoid the down queue manager, by stopping channels into your gateway queue manager.  With clustering within your organization the messages should then flow to another gateway queue manager.

This will much quicker than doing an emergency recreate of your queue manager!

 

What are the JMS connection factories on webLogic doing?

As part of my long running activity to find out what is causing 1 million MQCONNects a day from an Oracle  webLogic web server, I have found out how to monitor what is going on inside the webLogic instance.

Most web servers support a Java Management eXtension (JMX) interface.  You can use gui tools like jconsole to do an ad-hoc display of the management beans – but these are   not practical for long term monitoring.

Ive listed the data from the JMX query, what the data means, and also documented how I got the data.

There are three components to an MDB

  1. A listening task waits to be notified about messages arriving on the queue.   As far as I can tell there is one thread doing this work.  If you need more I think you need to create a second MDB.
  2. Multiple MDB threads which get woken up by the listener task, get a message and invoke the MDB application OnMessage() method with the message.  The data for this part has Type=EJBPoolRuntime.
  3. The application part which processes the message – typically it connects to MQ and puts a reply, and disconnects.  This application specified which connection pool to be used using ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF3”);  The data for this part has Type=ConnectorConnectionPoolRuntime. 

 

The multiple MDB threads.

This data came from the JMX record with com.bea:ServerRuntime=AdminServer2,MessageDrivenEJBRuntime=WMQ_IVT_MDB,Name=WMQ_IVT_MDB,ApplicationRuntime=MDB3,Type=EJBPoolRuntime,EJBComponentRuntime=MDB3

Where

  1.  ServerRuntime=AdminServer2. This is where the application runs
  2. MessageDrivenEJBRuntime=WMQ_IVT_MDB comes from  display-name WMQ_IVT_MDB in the ejb-jar.xml file.,Name=WMQ_IVT_MDB,
  3. ApplicationRuntime=MDB3, This is the name of the deployed .jar file.
  4. Type=EJBPoolRuntime, this is the type for the MDB threads
  5. EJBComponentRuntime=MDB3, this is the name of the deployed .jar file.

From experiementing with MDBs and adjusting parameters my picture of how the EJB thread pool works is as follows

  1. There is a general free pool for threads.
  2. There is a pool for the EJB.
  3. When a message arrives the listener thread gets a thread from the EJB pool, and executes the OnMessage() method on the thread.
  4. If there are not available threads,
    1. it waits for a free thread.
    2. if, over a period of seconds, most of the requests have had to wait for a thread, then start a new thread
      1. it gets a thread from the general pool, if none available it creates a new thread.
      2. it executes the MDB application ejbCreate() method on the thread just obtained.
      3. It executes the OnMessage() method on the thread.
      4. Puts the thread into the EJB pool as a free thread.
    3. In the ejbCreate() method, I had it write to the job log.  I could see threads started, after
  5. Periodically the EJB pool is cleaned up,  for threads which have been idle for a specific time interval
    1. execute ejbRemove() on the thread
    2. return the thread to the general free pool
  6. Periodically the general pool is cleaned up
    1. Threads which have not been used for a while are disconnected from the queue manager, and deleted.

The data came from the JMX with the defintion

com.bea:ServerRuntime=AdminServer2, Name=WMQ_IVT_MDB, ApplicationRuntime=MDB3, Type=MessageDrivenEJBRuntime, EJBComponentRuntime=MDB3

 

See here for the documentation.

  • AccessTotalCount = 28180,   There were this many messages processed, and  so this many OnMessage() methods executed
  • BeansInUseCount  = 2.  Deprecated
  • BeansInUseCurrentCount  = 1  One thread is currently active.
  • DestroyedTotalCount   = 0.   
  • IdleBeansCount  = 4 There are 4 free thread available in the EJB pool
  • MissTotalCount  = 19.  For 19 (out of the 28180) requests, they failed to get a free thread, and so a new thread was obtained.
  • Name  = WMQ_IVT_MDB
  • PooledBeansCurrentCount  = 3.   The numer of free threads inthe pool.
  • TimeoutTotalCount  = 0.  Number of requests timed out waiting for a thread from the free pool.
  • Type  = EJBPoolRuntime.   You can use Type=EJBPoolRuntime in the JMX query string.
  • WaiterCurrentCount  = 0. The number of requests waiting for a thread in the pool.   This value should always be zero.
  • WaiterTotalCount = 0. The total number of threads that have had to wait in the pool.  This value should always be zero, if it is non zero you need to tune your connection pool.

The data for the connection factory.

See here for some documentation.

  • ActiveConnectionsCurrentCount (Inte
  • ger) = 9.  How many connections were in use when the JMX request was issued.
  • ActiveConnectionsHighCount (Integer) = 10. The highest number of connections in use since the connection pool was started or reset
  • AverageActiveUsage (Integer) = 0. This value is always zero.
  • CapacityIncrement (Integer) = 1.   This is the value the connection pool is configured with
  • CloseCount (Long) = 5682.    How many connection.close() requests have been issued.   This will be less than or equal to ConnectionsMatchedTotalCount below.
  • ConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ConnectionFactoryImpl.   This is the name of the connection class being used
  • ConnectionFactoryName (Null).  I could not find where to change this, or if it has any value
  • ConnectionIdleProfileCount (Integer) = 0.   This was always zero.
  • ConnectionIdleProfiles (Null).   This was always zero.
  • ConnectionLeakProfileCount (Integer) = 0.   This was always zero.  I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • ConnectionLeakProfiles (Null).  This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • ConnectionProfilingEnabled (Boolean) = false.   I could not see how to change this, or what value it adds
  • Connections (ObjectName[]) = [Ljavax.management.ObjectName;@299a06ac]  This will be an internal object within Java.
  • ConnectionsCreatedTotalCount (Integer) = 21. This is the count of instances started.   You could get connection started, connection freed, connection started, connection freed.   This would could  2 connections started.
  • ConnectionsDestroyedByErrorTotalCount (Integer) = 0.
  • ConnectionsDestroyedByShrinkingTotalCount (Integer) = 11.  After a time period, idle-timeout-seconds in weblogic-ejb-jar.xml, connections that have not processed messages for this time period are released back to the general pool.
  • ConnectionsDestroyedTotalCount (Integer) = 11.  This seems to be the same as the previous item
  • ConnectionsMatchedTotalCount (Integer) = 5670.    This many requests for a connection with matching userid etc were got from the pool
  •  ConnectionsRejectedTotalCount (Integer) = 0.  I dont know what this is
  • ConnectorEisType (String) = Java Message Service
  • CurrentCapacity (Long) = 10.   This is the size of the pool
  • EISResourceId (String) = type=<eis>, application=colin, module=colin, eis=Java
  • Message Service, destinationId=CF2
  • FreeConnectionsCurrentCount (Integer) = 1.   There is currently one free connection in the pool
  • FreeConnectionsHighCount (Integer) = 10.  This is the highest number of free connections in the pool – when there was no requests for the connection pool.
  • FreePoolSizeHighWaterMark (Long) = 10.  I dont know the difference between this and the previous item.
  • FreePoolSizeLowWaterMark (Long) = 0.   This is the lowest number of free connections – it is the gap between in use and the maximum pool size.
  • HealthState (Null).    I do not think connection pools for MDBs support this
  • HighestNumWaiters (Long) = 0.   This value was always zero.  Even when I reduced the size of the connection pool, and specified 50 threads could wait, this value was still 0
  • InitialCapacity (Integer) = 1.   This is the specified value.   This is used  to set a lower limit of connections in the pool.  Setting the initial capacity to 5 means there will always be at least 5 connections in the pool, and the pool will not be shrunk below this value
  • JNDIName (String) = CF2.  This is what is specified in the connection pool definition.   This label is used in the application ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF2”);
  • Key (String) = CF2.  This seems to the be same as the JNDI name.
  • LastShrinkTime (Long) = 1565334529469.  This is the number of milliseconds POSIX time (since Jan 1st 1970).  To format it use  time_t now = t/1000; strftime(buff, 20, “%H:%M:%S”, localtime(&now));
  • LogFileName (Null).  You can specify logging for this adapter.  Deployments -> resource adapter -> configuration -> Outbound Connection Pools, javax.jsm.ConnectionFactory, select the connection pool,  Logging tab.  When this was active I got nothing logged in it.
  • LogFileStreamOpened (Boolean) = false
  • LoggingEnabled (Boolean) = true
  • LogRuntime (Null)
  • ManagedConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl.  This is the name of the resouce adapter class.
  • MaxCapacity (Integer) = 20.   This is the maximum size of the pool.
  • MaxIdleTime (Integer) = 0.   This was always zero for me.
  • MCFClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl.   This is the ManagedConnectionFactoryClassName. Same as above.
  • Name (String) = CF2.  Another connection pool identifier
  • NumberDetectedIdle (Integer) = 0.   This was always zero.
  • NumberDetectedLeaks (Integer) = 0.  This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
  • NumUnavailableCurrentCount (Integer) = 9.   This feels like the number threads waiting while the connection pool creates a connection in the connection pool.
  • NumUnavailableHighCount (Integer) = 10.   This feels like the number threads waiting while the connection pool creates a connection in the connection pool.
  • NumWaiters (Long) = 0.   This was always zero – even through I restricited the number of connections in the pool.  I got a java exception, unable to get a connection, rather than have the thread wait.
  • NumWaitersCurrentCount (Integer) = 0.  See above.
  • Parent (ObjectName) = com.bea:ServerRuntime=AdminServer2,Name=colin, ApplicationRuntime=colin, Type=ConnectorComponentRuntime
  • PoolName (String) = CF2.   Yet another field with the name of the connection pool/
  • PoolSizeHighWaterMark (Long) = 10.   This is the highest number of connections used in the pool
  • PoolSizeLowWaterMark (Long) = 0.       This is the lowest number of connecitons used in pool
  • ProxyOn (Boolean) = false.    I could not find what this means
  • RecycledTotal (Integer) = 0.    This was always zero for me.
  • ResourceAdapterLinkRefName (Null).   You can specify this field name in your MDB definition.
  • RuntimeTransactionSupport (String) = NoTransaction.  This defines if your connections are part of a unit of work or not. NoTransaction means out of syncpoint.    You may want to specify. XATransaction.
  • ShrinkCountDownTime (Integer) = 340.  This is how long before a scan of the pool to remove any threads in the pool which have done no work for the specified interval.
  • ShrinkingEnabled (Boolean) = true
  • ShrinkPeriodMinutes (Integer) = 15.  You can specify the interval on the resource adapter definition  (Note: you specify it in seconds – it gets reported in minutes)
  • State (String) = Running.    This connection pool is started.
  • Testable (Boolean) = false.   You can defined some resources as “testable”
  • TransactionSupport (String) = NoTransaction.   This is the same as RuntimeTransactionSupport above.
  • Type (String) = ConnectorConnectionPoolRuntime.   This defines the object.   You can use the Type=ConnectorConnectionPoolRuntime  in the JMX query.

From this I can see there is a pool called CF2 which has maximum of 10 connections.  The maximum connections used was 10, the lowest was 0.

There were Connections Matched Total Count  = 5670 requests for a connection from the pool.

The pool has shrunk more than once as it has Connections Destroyed By Shrinking Total Count  = 11 connections.

Using this data, you can now plot usage over time and see if you need to increase ( or decrease) the size of the pool, or the parameters to tune when the pool is shrunk.

I do not know enough about JMX to tell if the “high” and “low” value are reset on each query, or if you can use JMX to reset them periodically. These high and low value may have little value, if they are since the webLogic instance started (6 months ago).

The data fields are mentioned here.

How I got the data

There is a python package called JMXQuery which has a .jar file which allows you to query information in a JMX server.  The output is in json format so you can use your favourite tools (python) to quickly convert this to other format, such as .csv .

The command I used was

java  -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u webLogic
-p passw0rd -q “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”

which breaks down as follows

  • java – invoke java
  • -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” – this jar file
  • -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi – this is the url of my webLogic server
  • -json – output it in json format
  • -u webLogic -p passw0rd -q  – userid and password
  • “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”
    • com.bea is the bean type
    • The admin server was called AdminServer2
    • The connection factory was CF2
    • The resource adapter is installed under “Deployments”  as colin
    • Type=ConnectorConnectionPoolRuntime is the type of bean

I then used |jq . |grep -v mBean > bb to convert the one line json to one field per line, dropped the mBean value, and put it to a file.  The output was like

[
 {
   "attribute": "Connections",
   "attributeType": "ObjectName[]",
   "value": "[Ljavax.management.ObjectName;@299a06ac"
 },
 {
   "attribute": "FreeConnectionsCurrentCount",
   "attributeType": "Integer",
   "value": 4
},
etc

You can put generics in for example

java  -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u readonly -p read0nly -q “com.bea:ApplicationRuntime=*,ConnectorComponentRuntime=*,Name=”CP*”,ServerRuntime=*,Type=ConnectorConnectionPoolRuntime” e=*,Type=ConnectorConnectionPoolRuntime” > aa

 

Where this uses a userid set up as a monitor id, “*” has been specified for many values, and only give objects beginning with “CP”.  Note blanks have meaning. “,,,=*, Name=…” looks for an object with blank,N,a,m,e,

Data when not using a resource adapter

The above information was for a resource adapter.  When an EJB 2 MDB is deployed (non resource adapter)

from –q  “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3,
MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,
Name=WMQ_IVT_MDB_JMSQ1,ServerRuntime=AdminServer2,Type=EJBPoolRuntime”

Where the weblogic-jar-xml has

<weblogic-ejb-jar> <ejb-name>WMQ_IVT_MDB</ejb-name>
<destination-jndi-name>JMSQ1</destination-jndi-name>

The data was

BeansInUseCount (Integer) = 0
PooledBeansCurrentCount (Integer) = 10
IdleBeansCount (Integer) = 10
BeansInUseCurrentCount (Integer) = 0
DestroyedTotalCount (Long) = 0
WaiterCurrentCount (Integer) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
MissTotalCount (Long) = 7
AccessTotalCount (Long) = 28
Type (String) = EJBPoolRuntime
TimeoutTotalCount (Long) = 0
WaiterTotalCount (Long) = 0

With

-q “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3, MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,Name=WMQ_IVT_MDB_JMSQ1,
ServerRuntime=AdminServer2,Type=EJBTransactionRuntime

the output was

TransactionsRolledBackTotalCount (Long) = 0
TransactionsTimedOutTotalCount (Long) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
Type (String) = EJBTransactionRuntime
TransactionsCommittedTotalCount (Long) = 0

Configuring Oracle WebLogic to use MQ Message Drive Beans

I wanted to use webLogic web server from Oracle to look into a performance problem with Java Message Driven Beans using IBM MQ. Setting this up was harder than I expected, because of lack of documentation. Some documentation gave half the story – but nothing gave the complete story.
It took me several days to configure Oracle webLogic server with IBM MQ. I went down many rabbit holes, and had many problems, before finding that, once you know what to ignore, it is not too difficult.
I used

  1. A queue manager accessible from a client connection
  2. A “server queue” Q1
  3. A reply to queue CP0000
  4. A MDB application – I used an IBM IVT file as the basis of my program. This was an existing MDB jar file which had some a webLogic configuration file added to the .jar file
  5. An existing JNDI repository configured with a connection factory and “server queue”.
    1.  My definitions were stored in a file file:/home/colinpaice/JNDI-Directory and this needs the java class com.sun.jndi.fscontext.RefFSContextFactory to access it.
    2.  The JNDI repository had a connection factory CF1 which points to the channel definition, and a queue definition JMSQ1 mapping to queue Q1 in the queue manager.
  6.  A webLogic server. I used WebLogic Server Version: 12.2.1.3.0, also called 12c. I created a new domain, and used this. I logged onto the admin console using URL http://127.0.0.1:7001/console/ .

Note: The JNDI repository is configured in two places.

  1. Within the MDB configuration definitions.
  2.  Within MDB java program logic

Create a profile script for webLogic

I created a configuration script,  here, where I defined Java overrides and gave the webLogic server access to the MQ libraries.
I configured it to use JMX (Java Management eXtensions) which allow you to use a utility called jconsole to display information about what is happening within the web server.

I used the cd command to get to the weblogic directory and used
.    sh profile.sh
to execute the profile

Start the webLogic server

From the weblogic directory
./wls12213/user_projects/domains/colin/startWebLogic.sh 1>a 2>b
1>a and 2>b is useful so you can use another command window to search for lines in the output.

Go to the webLogic admin console in a web browser http://127.0.0.1:7001/console.

Setting up the JMS Server

You do not need to define a webLogic JMS Server to use IBM MQ.

Understanding terms and concepts

I used the webLogic Admin console, taking default names when offered.
A group of deployable MQ resources are defined within a “JMS Module”. You can define different modules and deploy them in different places.

I found that sometimes when I changed the configuration (for example to correct a spelling mistake) the server ignored the update till I restarted the server – despite the web server saying there were no pending changes.

It was safer to restart the server after I changed the configuration. It felt like that when an application is deployed, it remembers the configuration at the time of deployment, rather than looking it up each time.

WebLogic adds another level of indirection to the application. It took me a while to sort this out.

1. The application uses the name of a connection factory, MYCF
2. MYCF is looked up in the “JMS Modules” “ForeignConnectionFactory” “Local JNDI Name” to give the “Remote JNDI Name” (REMCF).
3. “ForeignConnectionFactory”.General gives the location of the JNDI repository to use.
4. The “Remote JNDI Name” (REMCF) is looked up in the JNDI repository to give the channel name and other parameters. (Are you keeping up?)

I found it easiest to call everything the same name eg CF1, and it made understanding the configuration much easier.

My application prints out information about the connection factory. So I could see
com.ibm.mq.jms.MQConnectionFactor – it was MQ, not the webLogic connection factory
XMSC_WMQ_CHANNEL :- SYSTEM.DEF.SVRCONN
XMSC_WMQ_QUEUE_MANAGER :- QMA

Define MQ resources to webLogic.

Home -> Messaging → JMS Modules

New

Name : SystemModule-0
Next

Target: select from the pull down, Adminserver
Next
Select “Would you like to add resources to this JMS system module?”
Finish

Select the JMS module you just created (SystemModule-0)
Summary of resources
New
Select: Foreign server
Next

Name: ForeignServer-0

Finish

Click ForeignServer-0

JNDI Initial Context Factory: com.sun.jndi.fscontext.RefFSContextFactory
JNDI Connection URL: file:/home/colinpaice/JNDI-Directory
credential leave emtpy
Save

Click ‘Connection Factories’ tab
New
Name: ForeignConnectionFactory-0
LocalJNDI name: CF1
This is what my application running on webLogic will use. See the weblogic-ejb-jar.xml file below.

Remote JNDI name: CF1
This is the name in the JNDI repostitory
user name webLogic
The user name that is used to logon to admin console
password
Confirm password

Save

You can use “view changes and restarts” to see what webLogic thinks needs to be restarted.

I found it safer to restart the server, as sometimes changes were not picked up. Ctrl-c from the command window running the webLogic server seems to work, but it is better to use a different window and use
./wls12213/user_projects/domains/base_domain/bin/stopWebLogic.sh
then back to the orignal window and issue

./wls12213/user_projects/domains/base_domain/startWebLogic.sh 1>a 2>b

The application

The application, here, was based on the IBM provided sample for MQ on WAS

in the directory MDB
… META-INF/
… META-INF/MANIFEST.MF
… META-INF/weblogic-ejb-jar.xml
… META-INF/ejb-jar.xml
… ejbs/
… ejbs/IVTMDB.class
… ejbs/IVTMDB.java
…javax.ejb-api-3.2.jar

The ejb-jar.xml file is here.

Each web server seems to have their own way of configuring application. JBoss web server has its own file, and WAS has configuration information in the server.xml file. The webLogic file is given below

… META-INF/weblogic-ejb-jar.xml had

<?xml version=”1.0″?>
<!DOCTYPE weblogic-ejb-jar
PUBLIC ‘-//BEA Systems, Inc.//DTD WebLogic 8.1.0 EJB//EN’
http://www.bea.com/servers/wls810/dtd/weblogic-ejb-jar.dtd’&gt;

<weblogic-ejb-jar>

<weblogic-enterprise-bean>
<!– I think the value of this ejb-name matches the value in META-INF/ejb-jar.xml ->
<ejb-name>WMQ_IVT_MDB</ejb-name>

<message-driven-descriptor>
<!– define the number of listener threads that can be used, maximum and initial –>
<pool>
<max-beans-in-free-pool>10</max-beans-in-free-pool>
<initial-beans-in-free-pool>3</initial-beans-in-free-pool>
</pool>

<!– JNDI defintitions →
<!– Specify the class needed to process the JNDI repository –>
<initial-context-factory>com.sun.jndi.fscontext.RefFSContextFactory

</initial-context-factory>
<!– specify which JNDI repository to use –>
<provider-url>file:/home/colinpaice/JNDI-Directory</provider-url>

<!– look in the JNDI file for the definitions with JMSQ1 –>
<destination-jndi-name>JMSQ1</destination-jndi-name>

<!– look in the JNDI file for the connection factory → MQ channel etc–>
<connection-factory-jndi-name>CF1</connection-factory-jndi-name>

</message-driven-descriptor>
</weblogic-enterprise-bean>
</weblogic-ejb-jar>

Compile the program

javac -cp /opt/mqm/java/lib/*:javax.ejb-api-3.2.jar: ejbs/IVTMDB.java

The javax.ejb-api-3.2.jar came from
https://download.jar-download.com/cache_jars/javax.ejb/javax.ejb-api/3.2/jar_files.zip

Build the jar file

jar -cvf MDB3.jar ejbs/IVTMDB.class META-INF/weblogic-ejb-jar.xml META-INF/ejb-jar.xml

jar -tvf MDB3.jar gave

0 Fri Aug 02 07:08:38 BST 2019 META-INF/
69 Fri Aug 02 07:08:38 BST 2019 META-INF/MANIFEST.MF
3572 Fri Aug 02 07:05:10 BST 2019 ejbs/IVTMDB.class
1197 Wed Jul 31 17:43:08 BST 2019 META-INF/weblogic-ejb-jar.xml
3696 Tue Jul 30 15:16:52 BST 2019 META-INF/ejb-jar.xml

Deploy the MDB

Using the webLogic admin console
Home → Deployment resources → Deployments

Install
enter the path to the directory with your jar file
select jar file
Install as application
Next
Name MDB3
Finish

Check the webLogic error log for problems.

For example, the log showed

<BEA-160228><AppMerge failed to merge your application. If you are running AppMerge on the command-line, merge again with the -verbose option for more details. See the error message(s) below.>

I went into a command window.
Set up the environment using
.      ./wls12213/wlserver/server/bin/setWLSEnv.sh
and ran

java weblogic.appmerge -verbose ~/temp/MDB/MDB3.jar
I fixed any problems.

( If you get Error: Could not find or load main class weblogic.appmerge, you did not enter the setWLSEnv.sh command properly, check the . before the command.)

When it successfully deployed, the webLogic log had
EJBCreate
EJBCreate
EJBCreate
This was 3 instances ( as defined in the weblogic-ejb-jar.xml file) of my program starting up and printing out a message.

Try it out!

My JNDI file had JMSQ1 → Q1.  I ran a program to put a message, to queue Q1, specifying a reply to queue, and it worked!

I put a message to Q1, and a response was returned.

Thanks to Paul Titherage who gave me good advice on getting the Java program compiled, and other pointers to the correct path.

The fog around Message Driven Beans (MDBs)

While playing with Message Driven Beans and looking at performance problems, I have learned a lot about MDBs and tuning.   The documentation was only partly useful.   There is a lot of documentation  – some concepts are explained 1000 times, but other areas are undocumented – or just wrong.  I’ll try to fill the holes in this blog post.  If I am wrong – please tell me!

This blog post is just one step on the path to how to configure webLogic server to avoid millions of connects a day, and to see if you need to do any tuning ( from an MQ perspective)

I’ll cover

  1. How MDBs work
  2. Different types of Enterprise Java bean standards, and how configuration of EJBs have changed over the years.
  3. Use of resource adapters.

Message listener

We should all be familiar with an application that does an MQGET, a database update, and a commit – which commits both units of work.  A transaction manager manages the units of work with each resource manager (MQ, and database).

The same concept applies to the MDB listener running in a web server.  The listener gets a message from one queue manager, and calls the MDB on a different thread, with the message content.  The MDB connects to a (possibly different) queue manager puts the reply and returns. The listener does a commit. The web server acting as a transaction manager coordinates the commit in both queue managers.  If the application does a roll back – the whole request can be backed out (if the MDB is set up for this).

You can configured the listener tasks so the gets are in syncpoint or not, and the MDB can do its processing in sycnpoint or not.

There are two distinct parts to the MDB.

The listener task described above, and the application code.   This application code does not use the same MQ connection as the listener code.   This may be obvious to most people, but I found documents on the web discussing performance tuning, which said create a connection pool for the listener side of the MDB (which may connect once a day and so a connection pool may not be needed) and did not mention a connection pool for the application side (which does 1 million puts, causing 1 million connects a day and so needs a connection pool!).

The Enterprise Java Bean architecture.

From the dawn of web servers, the architecture has followed the classic pattern.

  1. This is great – every one develops their own way of doing things.  Let each web server manufacturer have their own way of doing this.    Let JMS be different to database access.
  2. The realization that this is not a good idea and we need standards and commonality as there is basically one problem.
  3. Now we have standards and people going along the road together, we find there are improvements we need to make.  It could also be that there is a new leader who says “let us go in a different direction – the grass is greener over here”.  You have to maintain the old way of doing things, and also swallow the new way of doing things.

How does this impact MDBs?

EJB 2.0

With EJB 2. MDBs you have information stored in a JNDI repository.   This could be file based on your local machine, or in LDAP.  In this repository you have

  • a queue resource MYJMSQ and information about it, the MQ Queue name is SERVER, and other parameters.
  • connection information MYCONN, this contains the channel name, host name, port etc.

You configure your MDB with

  1. The name of the class (your program) to use, and the method to be passed messages (this is usually onMessage)
  2. The location of the JNDI repository
  3. What sort of repository it is (what Java class you need to communicate with it)
  4. What connection to use
  5. What queue to use.

The ejb-jar.xml file has “standard” information including the class name  and the method name.
Each web server has their own way of passing in the other information.  For example webLogic has a file weblogic-ejb-jar.xml with information about the location of the JNDI repository, the JNDI connection (MYCONN), and JNDI queue name(MYJMSQ).   It also has information about the connection pool for the listener.  All these files are packaged into the .jar file for the MDB.

IBM’s Liberty web server stores the information elsewhere.

The application has code like

ctx = new InitialContext();
cf = (ConnectionFactory)ctx.lookup(“CF1”);

which says look in the same JNDI repository, for connection CF1.   This could be same connection name as the listener is using, it could be different.  I could not find a way of specifying connection pool information for this connection, so by default every message put to MQ requires an MQCONN and MQDISC.

The application can get the destination name of the replyTo Queue, and uses it

Destination dest = message.getJMSReplyTo();
producer = session.createProducer(dest);
producer.send(response);

The Destination dest will have a value like queue://QMA/QUEUENAME

EJB 3.0

With EJB version 3, a lot has changed.

  1. There is a standard Java Connector Adapter( JCA) I think version 1.5 is current.
  2. The hard work is now done by using a resource adapter (ra)
  3. This has a different way of passing configuration information
  4. You can now hard code data in your program (which seems a step backwards to me)

Key information is stored in activationSpecs.  You can code it in your java application program.   (The resource adapter can query all the activationConfigProperties from a class and then process it), or have it in the .xml configuration files.

For example with the information in your application.java file.

@MessageDriven(
  name = "JMSSampleMDB",
  activationConfig = {
    @ActivationConfigProperty(propertyName  = "destinationType", 
               propertyValue = "javax.jms.Queue"),
     @ActivationConfigProperty(propertyName  = "destination", 
	       propertyValue = "INPUT_Q")                         
  }
)
public class JMSSampleMDB implements MessageListener{
 @TransactionAttribute(value = TransactionAttributeType.REQUIRED)
  public void onMessage(Message message) {...
	 }
}

It can be stored in the ejb-jar.xml file

<activation-config>
  <activation-config-property> 
    <activation-config-property-name>connectionfactory</activation-config-property-name> 
    <activation-config-property-value>CF3</activation-config-property-value>
  </activation-config-property>
  <activation-config-property> 
    <activation-config-property-name>destination</activation-config-property-name> 
    <activation-config-property-value>JMSQ2</activation-config-property-value>
  </activation-config-property>
</activation-config>

The parameters CF3 and JMSQ2 are configured as part of the resource adapter, and not in JNDI.  Each resource adapter has an ra.xml file.  Weblogic stores its specific information in weblogic-ra.xml.  IBM Liberty stores it in the server.xml file.

Once you have installed the resource adapter you should then be able to add additional connections and queues to it.

The resoure adapter takes the definitions and puts them into the JNDI tree within the web server.   Your application can continue to use

ctx = new InitialContext();
cf = (ConnectionFactory)ctx.lookup(“CF1”);

Using this way you can configure your connection as part of a connection pool- and avoid the MQCONN and MQDISC request.

I believe you can define an MDB using definitions in JNDI – and the application part of the MDB can use the resource adapter.

As far as I have been able to find out – if you want to use connection pooling for your application program you need to use a resource adapter.  I’ll blog this in a separate topic.  It is complex and quirky, and will need a lot of description

 

Java whoops causes performance problem.

I had not realized how easy it is to write a bad MQ Java program. When I worked for IBM, I had seen java applications doing 1 million MQCONNs a day – and people complained because it was slow! I accidentally managed to recreate this when looking into another problem.

I’ll show the bad programming example – which may be typical of many applications, and I’ll show what MQ requests the various JMS methods do, and how you can tell if your application is misbehaving.

My bad java program

If I had a java program which does the following

public class ColinDemo{
  public static void doit(){
    while ( i< 100)
    {
      colinMQ(...)
    }
  } 
  colinMQ() { 
     MQQueueManager queueManager = new MQQueueManager("qMgrName");
  ...  
     queueManager.close()
  }
}

Every time colinMQ is entered it creates a new connection to MQ.   When queueManager.close() is executed, there will be an MQDISC

The program as written will cause 100 MQCONNs and 100 MQDISC… whoops.

QueueManager needs to be defined as a class variable for example

public class ColinDemo{
MQQueueManager queueManager = Null;

Then colinMQ() needs a check like
if queueManager == Null then
queueManager = new MQQueueManager(“qMgrName”);
or

public class ColinDemo{
  public static void doit(){
    MQQueueManager queueManager;
    queueManager = initColinMQ(...)
    while ( i< 100)
    {
       colinMQ(queueManager...)
    }
    closeqm(queueManager);
  }
  
  MQQueueManager initColinMQ(...){
  return MQQueueManager("qMgrName")
  } 
  colinMQ(qm... ) { 
  ...
  }
  closeqm(qm)
  {
    qm.close();
  }
}

That is all pretty basic stuff, which is obvious once you see it. The more subtle bug is when an application calls ColinDemo.doit() multiple times. On exit from ColinDemo queueManager  will be closed and will cause it do do an MQDISC, the next call to ColinDemo.doit() will require another MQCONN.

You need to have ColinDemo return instance data. For example in your highest level program you say

ColinDemo cd = new ColinDemo(“qMgrName”)

then you can use cd.doit()… instead of ColinDemo.doit(), and of course you need a cd.close() to close the connection once it has been finished with.

When you are using JMS you need to have
connection = cf.createConnection();
session = connection.createSession(…);
consumer = session.createConsumer(destination);
defined as instance variables so they are set when the thread is started in the morning, and are not changed or de-allocated at the end of the day when the high level program ends.

For example in your top level JMS program

ColinDemo cd = new ColinDemo(“MYQMGR”);
for (i=0 ;i <100;i++) 
{
Message m = cd.getMessage() 
}

with

public class ColinDemo{
  private Connection connection = null;
  private Session session = null;
  private Destination destination = null;
  private MessageConsumer consumer = null;
  private Context context = null;
  private JmsConnectionFactory cf = null;
  private String contextFactory = null; 
  private Hashtable<String, String> environment;

  ColinDemo(String QMGRNAME)
  {
  // Instantiate the initial context
  contextFactory = "com.sun.jndi.fscontext.RefFSContextFactory";
  environment = new Hashtable<String, String>();

  environment.put(Context.INITIAL_CONTEXT_FACTORY,contextFactory);
  environment.put(Context.PROVIDER_URL, initialContextUrl);
  context = new InitialDirContext(environment);
  cf = (JmsConnectionFactory)
  context.lookup(connectionFactoryFromJndi);
  // Lookup the destination
  destination = (JmsDestination) 
  context.lookup(destinationFromJndi);

  connection = cf.createConnection();
  session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
  consumer = session.createConsumer(destination);

  // Start the connection
  connection.start();
  }

  public Message getMessage()
  { 
    Message message = consumer.receive(timeout);
  } 
}

What goes on under the covers.

A basic JMS program looks like

String contextFactory = "com.sun.jndi.fscontext.RefFSContextFactory";
Hashtable<String, String> environment = new Hashtable<String, String>();
environment.put(Context.INITIAL_CONTEXT_FACTORY, contextFactory);
environment.put(Context.PROVIDER_URL, initialContextUrl);
context = new InitialDirContext(environment);
cf = (JmsConnectionFactory) context.lookup(connectionFactoryFromJndi);

// Lookup the destination
destination = (JmsDestination) context.lookup(destinationFromJndi);

connection = cf.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
consumer = session.createConsumer(destination);

// Start the connection
connection.start();
// get a message 
Message message = consumer.receive(timeout);

Using the distributed MQ activity trace we can see the various MQ verbs which are issued.
The program started at 09:00:30 and ended at 09:00:45

connection = cf.createConnection();
has the following accounting trace records
Tid Date       Time     Operation  CompCode MQRC    HObj (ObjName) 
001 2019-07-26 09:00:30 MQXF_CONNX MQCC_OK  0000    -
001 2019-07-26 09:00:30 MQXF_OPEN  MQCC_OK  0000    2    ( ) 
001 2019-07-26 09:00:30 MQXF_INQ   MQCC_OK  0000    2    ( )
001 2019-07-26 09:00:30 MQXF_CLOSE MQCC_OK  0000    2    ( )
001 2019-07-26 09:00:45 MQXF_DISC  MQCC_OK  0000    -

We can see the transaction starting at 09:00:30 and ending at 09:00:45.

This is querying the queue manager for

MQIA_COMMAND_LEVEL (31)
MQIA_PLATFORM (32)
MQIA_CODED_CHAR_SET_ID (2)
MQCA_Q_MGR_NAME (2015)
MQCA_Q_MGR_IDENTIFIER (2032)
MQIA_ADVANCED_CAPABILITY (273)
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

produces the following trace records

Tid Date       Time     Operation  CompCode MQRC    HObj (ObjName) 
001 2019-07-26 09:00:30 MQXF_CONNX MQCC_OK  0000 -
001 2019-07-26 09:00:45 MQXF_DISC  MQCC_OK  0000 -

consumer = session.createConsumer(destination);

adds the following to the session data.

Tid Date       Time     Operation  CompCode MQRC HObj (ObjName) 
001 2019-07-26 09:00:30 MQXF_OPEN  MQCC_OK  0000 2    (Q1 ) 
001 2019-07-26 09:00:45 MQXF_CLOSE MQCC_OK  0000 2    (Q1 )

connection.start();

adds nothing

Message message = consumer.receive(timeout);

adds the following tace data to the session data.

Tid Date       Time     Operation CompCode MQRC HObj (ObjName)
001 2019-07-26 09:00:30 MQXF_GET  MQCC_OK  0000 2    (Q1 )

So by having
connection = cf.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
consumer = session.createConsumer(destination);

Issued just once, you will save many MQCONN, MQDISC, MQOPEN, MQCLOSE and MQINQ calls.
If you see a pattern of

Tid Date       Time     Operation  CompCode MQRC HObj (ObjName) 
001 2019-07-26 09:00:30 MQXF_CONNX MQCC_OK  0000 -
001 2019-07-26 09:00:30 MQXF_OPEN  MQCC_OK  0000 2    ( ) 
001 2019-07-26 09:00:30 MQXF_INQ   MQCC_OK  0000 2    ( )
001 2019-07-26 09:00:30 MQXF_CLOSE MQCC_OK  0000 2    ( )
001 2019-07-26 09:00:45 MQXF_DISC  MQCC_OK  0000 -

You are doing too many connects – and need to look into it!

I am working on some other posts for making your Message Driven Bean applications running in a web server more efficient.

 

Thanks to Roger Lacroix for helping me get the java right.  His comments reminded me of the rules of computing

  • If you open it, close it
  • If you allocate it, free it
  • Clean up after you.
  • Be a good citizen

On the web there are some good example, and some bad examples.  One good example had

try{
    queueManager = new MQQueueManager("qMgrName");
    doWork(queueManager)    
    ...
   } catch (Exception je)
   {...
   }
finally{
    try{ queueManager.close()}
    catch (Exception je){...}
   }

When the execution left the try{} statement, the statement in the finally was always executed, so you could be sure that the queueManager object will always closed.

Other examples did not have code to close the connection, and the connection was left open until the program terminated which could be months later!  Be a good citizen, free it when you do not need it.

 

 

Why is my java application doing multiple gets because the receive buffer is too small?

When investigating a MQ JMS performance problem, I noticed that there was a pattern of

MQGET – 2080 (0820) (RC2080): MQRC_TRUNCATED_MSG_FAILED

followed by an MQGET with a bigger buffer which worked.

Being a performance person these double gets did not look right to me.

I spoke to Paul Titheridge of IBM who said this was fixed back in MQ V6 with a java property

com.ibm.mq.jmqi.defaultMaxMsgSize

If you specify

 -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=8192
in the Java start-up you will get an 8KB buffer instead of the default 4KB.
Easy when you know how!