How to get my enterprise talking to your enterprise over MQ.

No,  this is not about Star Trek’s “beam me up Scottie”, but setting up MQ to MQ communications.

Of course you will want

  1. it to scale – you should be able to define more channels if you need more throughput than one channel can provide
  2. be secure – you can use TLS to protect the data on the channels
  3. be highly available – this is the trickier challenge which I’ll cover below.

Should I use clustering?

At first glance you may think clustering would be a good solution.   Many institutions are unhappy with being in a cluster with their business partner, so it is rare to have two enterprises both in the same cluster.

This means you are usually looking a gateway between the two institutions, which means point-to-point.

There are two parts to being highly available,

  1. How long does it take before messages in the queue manager can flow out of the queue manager?
  2. How long does it take before new work can flow?

How long does it take before messages in the queue manager can flow out of the queue manager?

If your queue manager has ended, the messages are not available until the queue manager (and the channels) area available again.

If you are using z/OS and shared queue etc – other queue managers in the queue sharing group should be able to process the work.

If you are using mid-range MQ, you an use multi instance queue managers. One queue manager is active, the other is in standby,  partially initialized, ready to take over once a failure has been detected.

How long does it take before new work can flow?

If you have only one queue manager, you have to wait until it restarts before work can flow.  If you have more than one queue manager, you can switch message to avoid the down queue manager, by stopping channels into your gateway queue manager.  With clustering within your organization the messages should then flow to another gateway queue manager.

This will much quicker than doing an emergency recreate of your queue manager!

 

What are the JMS connection factories on webLogic doing?

As part of my long running activity to find out what is causing 1 million MQCONNects a day from an Oracle  webLogic web server, I have found out how to monitor what is going on inside the webLogic instance.

Most web servers support a Java Management eXtension (JMX) interface.  You can use gui tools like jconsole to do an ad-hoc display of the management beans – but these are   not practical for long term monitoring.

Ive listed the data from the JMX query, and also documented how I got the data

The data

  • ActiveConnectionsCurrentCount (Integer) = 9
  • ActiveConnectionsHighCount (Integer) = 10
  • AverageActiveUsage (Integer) = 0
  • CapacityIncrement (Integer) = 1
  • CloseCount (Long) = 5682
  • ConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ConnectionFactoryImpl
  • ConnectionFactoryName (Null)
  • ConnectionIdleProfileCount (Integer) = 0
  • ConnectionIdleProfiles (Null)
  • ConnectionLeakProfileCount (Integer) = 0
  • ConnectionLeakProfiles (Null)
  • ConnectionProfilingEnabled (Boolean) = false
  • Connections (ObjectName[]) = [Ljavax.management.ObjectName;@299a06ac
  • ConnectionsCreatedTotalCount (Integer) = 21
  • ConnectionsDestroyedByErrorTotalCount (Integer) = 0
  • ConnectionsDestroyedByShrinkingTotalCount (Integer) = 11
  • ConnectionsDestroyedTotalCount (Integer) = 11
  • ConnectionsMatchedTotalCount (Integer) = 5670
  • ConnectionsRejectedTotalCount (Integer) = 0
  • ConnectorEisType (String) = Java Message Service
  • CurrentCapacity (Long) = 10
  • EISResourceId (String) = type=<eis>, application=colin, module=colin, eis=Java
  • Message Service, destinationId=CF2
  • FreeConnectionsCurrentCount (Integer) = 1
  • FreeConnectionsHighCount (Integer) = 10
  • FreePoolSizeHighWaterMark (Long) = 10
  • FreePoolSizeLowWaterMark (Long) = 0
  • HealthState (Null)
  • HighestNumWaiters (Long) = 0
  • InitialCapacity (Integer) = 1
  • JNDIName (String) = CF2
  • Key (String) = CF2
  • LastShrinkTime (Long) = 1565334529469
  • LogFileName (Null)
  • LogFileStreamOpened (Boolean) = false
  • LoggingEnabled (Boolean) = true
  • LogRuntime (Null)
  • ManagedConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl
  • MaxCapacity (Integer) = 20
  • MaxIdleTime (Integer) = 0
  • MCFClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl
  • Name (String) = CF2
  • NumberDetectedIdle (Integer) = 0
  • NumberDetectedLeaks (Integer) = 0
  • NumUnavailableCurrentCount (Integer) = 9
  • NumUnavailableHighCount (Integer) = 10
  • NumWaiters (Long) = 0
  • NumWaitersCurrentCount (Integer) = 0
  • Parent (ObjectName) = com.bea:ServerRuntime=AdminServer2,Name=colin, ApplicationRuntime=colin, Type=ConnectorComponentRuntime
  • PoolName (String) = CF2
  • PoolSizeHighWaterMark (Long) = 10
  • PoolSizeLowWaterMark (Long) = 0
  • ProxyOn (Boolean) = false
  • RecycledTotal (Integer) = 0
  • ResourceAdapterLinkRefName (Null)
  • RuntimeTransactionSupport (String) = NoTransaction
  • ShrinkCountDownTime (Integer) = 340
  • ShrinkingEnabled (Boolean) = true
  • ShrinkPeriodMinutes (Integer) = 15
  • State (String) = Running
  • Testable (Boolean) = false
  • TransactionSupport (String) = NoTransaction
  • Type (String) = ConnectorConnectionPoolRuntime

From this I can see there is a pool called CF2 which has maximum of 10 connections.  The maximum connections used was 10, the lowest was 0.

There were Connections Matched Total Count  = 5670 requests for a connection from the pool.

The pool has shrunk more than once as it has Connections Destroyed By Shrinking Total Count  = 11 connections.

Using this data, you can now plot usage over time and see if you need to increase ( or decrease) the size of the pool, or the parameters to tune when the pool is shrunk.

I do not know enough about JMX to tell if the “high” and “low” value are reset on each query, or if you can use JMX to reset them periodically. These high and low value may have little value, if they are since the webLogic instance started (6 months ago).

The data fields are mentioned here.

How I got the data

There is a python package called JMXQuery which has a .jar file which allows you to query information in a JMX server.  The output is in json format so you can use your favourite tools (python) to quickly convert this to other format, such as .csv .

The command I used was

java -Xss409600k -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u webLogic
-p passw0rd -q “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”

which breaks down as follows

  • java -Xss409600k – invoke java with this heap size
  • -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” – this jar file
  • -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi – this is the url of my webLogic server
  • -json – output it in json format
  • -u webLogic -p passw0rd -q  – userid and password
  • “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”
    • com.bea is the bean type
    • The admin server was called AdminServer2
    • The connection factory was CF2
    • The resource adapter is installed under “Deployments”  as colin
    • Type=ConnectorConnectionPoolRuntime is the type of bean

I then used |jq . |grep -v mBean > bb to convert the one line json to one field per line, dropped the mBean value, and put it to a file.  The output was like

[
 {
   "attribute": "Connections",
   "attributeType": "ObjectName[]",
   "value": "[Ljavax.management.ObjectName;@299a06ac"
 },
 {
   "attribute": "FreeConnectionsCurrentCount",
   "attributeType": "Integer",
   "value": 4
},
etc

You can put generics in for example

java -Xss409600k -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u readonly -p read0nly -q “com.bea:ApplicationRuntime=*,ConnectorComponentRuntime=*,Name=”CP*”,ServerRuntime=*,Type=ConnectorConnectionPoolRuntime” e=*,Type=ConnectorConnectionPoolRuntime” > aa

 

Where this uses a userid set up as a monitor id, “*” has been specified for many values, and only give objects beginning with “CP”.  Note blanks have meaning. “,,,=*, Name=…” looks for an object with blank,N,a,m,e,

Data when not using a resource adapter

The above information was for a resource adapter.  When an EJB 2 MDB is deployed (non resource adapter)

from –q  “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3,
MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,
Name=WMQ_IVT_MDB_JMSQ1,ServerRuntime=AdminServer2,Type=EJBPoolRuntime”

Where the weblogic-jar-xml has

<weblogic-ejb-jar> <ejb-name>WMQ_IVT_MDB</ejb-name>
<destination-jndi-name>JMSQ1</destination-jndi-name>

The data was

BeansInUseCount (Integer) = 0
PooledBeansCurrentCount (Integer) = 10
IdleBeansCount (Integer) = 10
BeansInUseCurrentCount (Integer) = 0
DestroyedTotalCount (Long) = 0
WaiterCurrentCount (Integer) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
MissTotalCount (Long) = 7
AccessTotalCount (Long) = 28
Type (String) = EJBPoolRuntime
TimeoutTotalCount (Long) = 0
WaiterTotalCount (Long) = 0

With

-q “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3, MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,Name=WMQ_IVT_MDB_JMSQ1,
ServerRuntime=AdminServer2,Type=EJBTransactionRuntime

the output was

TransactionsRolledBackTotalCount (Long) = 0
TransactionsTimedOutTotalCount (Long) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
Type (String) = EJBTransactionRuntime
TransactionsCommittedTotalCount (Long) = 0

Configuring Oracle WebLogic to use MQ Message Drive Beans

I wanted to use webLogic web server from Oracle to look into a performance problem with Java Message Driven Beans using IBM MQ. Setting this up was harder than I expected, because of lack of documentation. Some documentation gave half the story – but nothing gave the complete story.
It took me several days to configure Oracle webLogic server with IBM MQ. I went down many rabbit holes, and had many problems, before finding that, once you know what to ignore, it is not too difficult.
I used

  1. A queue manager accessible from a client connection
  2. A “server queue” Q1
  3. A reply to queue CP0000
  4. A MDB application – I used an IBM IVT file as the basis of my program. This was an existing MDB jar file which had some a webLogic configuration file added to the .jar file
  5. An existing JNDI repository configured with a connection factory and “server queue”.
    1.  My definitions were stored in a file file:/home/colinpaice/JNDI-Directory and this needs the java class com.sun.jndi.fscontext.RefFSContextFactory to access it.
    2.  The JNDI repository had a connection factory CF1 which points to the channel definition, and a queue definition JMSQ1 mapping to queue Q1 in the queue manager.
  6.  A webLogic server. I used WebLogic Server Version: 12.2.1.3.0, also called 12c. I created a new domain, and used this. I logged onto the admin console using URL http://127.0.0.1:7001/console/ .

Note: The JNDI repository is configured in two places.

  1. Within the MDB configuration definitions.
  2.  Within MDB java program logic

Create a profile script for webLogic

I created a configuration script,  here, where I defined Java overrides and gave the webLogic server access to the MQ libraries.
I configured it to use JMX (Java Management eXtensions) which allow you to use a utility called jconsole to display information about what is happening within the web server.

I used the cd command to get to the weblogic directory and used
.    sh profile.sh
to execute the profile

Start the webLogic server

From the weblogic directory
./wls12213/user_projects/domains/colin/startWebLogic.sh 1>a 2>b
1>a and 2>b is useful so you can use another command window to search for lines in the output.

Go to the webLogic admin console in a web browser http://127.0.0.1:7001/console.

Setting up the JMS Server

You do not need to define a webLogic JMS Server to use IBM MQ.

Understanding terms and concepts

I used the webLogic Admin console, taking default names when offered.
A group of deployable MQ resources are defined within a “JMS Module”. You can define different modules and deploy them in different places.

I found that sometimes when I changed the configuration (for example to correct a spelling mistake) the server ignored the update till I restarted the server – despite the web server saying there were no pending changes.

It was safer to restart the server after I changed the configuration. It felt like that when an application is deployed, it remembers the configuration at the time of deployment, rather than looking it up each time.

WebLogic adds another level of indirection to the application. It took me a while to sort this out.

1. The application uses the name of a connection factory, MYCF
2. MYCF is looked up in the “JMS Modules” “ForeignConnectionFactory” “Local JNDI Name” to give the “Remote JNDI Name” (REMCF).
3. “ForeignConnectionFactory”.General gives the location of the JNDI repository to use.
4. The “Remote JNDI Name” (REMCF) is looked up in the JNDI repository to give the channel name and other parameters. (Are you keeping up?)

I found it easiest to call everything the same name eg CF1, and it made understanding the configuration much easier.

My application prints out information about the connection factory. So I could see
com.ibm.mq.jms.MQConnectionFactor – it was MQ, not the webLogic connection factory
XMSC_WMQ_CHANNEL :- SYSTEM.DEF.SVRCONN
XMSC_WMQ_QUEUE_MANAGER :- QMA

Define MQ resources to webLogic.

Home -> Messaging → JMS Modules

New

Name : SystemModule-0
Next

Target: select from the pull down, Adminserver
Next
Select “Would you like to add resources to this JMS system module?”
Finish

Select the JMS module you just created (SystemModule-0)
Summary of resources
New
Select: Foreign server
Next

Name: ForeignServer-0

Finish

Click ForeignServer-0

JNDI Initial Context Factory: com.sun.jndi.fscontext.RefFSContextFactory
JNDI Connection URL: file:/home/colinpaice/JNDI-Directory
credential leave emtpy
Save

Click ‘Connection Factories’ tab
New
Name: ForeignConnectionFactory-0
LocalJNDI name: CF1
This is what my application running on webLogic will use. See the weblogic-ejb-jar.xml file below.

Remote JNDI name: CF1
This is the name in the JNDI repostitory
user name webLogic
The user name that is used to logon to admin console
password
Confirm password

Save

You can use “view changes and restarts” to see what webLogic thinks needs to be restarted.

I found it safer to restart the server, as sometimes changes were not picked up. Ctrl-c from the command window running the webLogic server seems to work, but it is better to use a different window and use
./wls12213/user_projects/domains/base_domain/bin/stopWebLogic.sh
then back to the orignal window and issue

./wls12213/user_projects/domains/base_domain/startWebLogic.sh 1>a 2>b

The application

The application, here, was based on the IBM provided sample for MQ on WAS

in the directory MDB
… META-INF/
… META-INF/MANIFEST.MF
… META-INF/weblogic-ejb-jar.xml
… META-INF/ejb-jar.xml
… ejbs/
… ejbs/IVTMDB.class
… ejbs/IVTMDB.java
…javax.ejb-api-3.2.jar

The ejb-jar.xml file is here.

Each web server seems to have their own way of configuring application. JBoss web server has its own file, and WAS has configuration information in the server.xml file. The webLogic file is given below

… META-INF/weblogic-ejb-jar.xml had

<?xml version=”1.0″?>
<!DOCTYPE weblogic-ejb-jar
PUBLIC ‘-//BEA Systems, Inc.//DTD WebLogic 8.1.0 EJB//EN’
http://www.bea.com/servers/wls810/dtd/weblogic-ejb-jar.dtd’&gt;

<weblogic-ejb-jar>

<weblogic-enterprise-bean>
<!– I think the value of this ejb-name matches the value in META-INF/ejb-jar.xml ->
<ejb-name>WMQ_IVT_MDB</ejb-name>

<message-driven-descriptor>
<!– define the number of listener threads that can be used, maximum and initial –>
<pool>
<max-beans-in-free-pool>10</max-beans-in-free-pool>
<initial-beans-in-free-pool>3</initial-beans-in-free-pool>
</pool>

<!– JNDI defintitions →
<!– Specify the class needed to process the JNDI repository –>
<initial-context-factory>com.sun.jndi.fscontext.RefFSContextFactory

</initial-context-factory>
<!– specify which JNDI repository to use –>
<provider-url>file:/home/colinpaice/JNDI-Directory</provider-url>

<!– look in the JNDI file for the definitions with JMSQ1 –>
<destination-jndi-name>JMSQ1</destination-jndi-name>

<!– look in the JNDI file for the connection factory → MQ channel etc–>
<connection-factory-jndi-name>CF1</connection-factory-jndi-name>

</message-driven-descriptor>
</weblogic-enterprise-bean>
</weblogic-ejb-jar>

Compile the program

javac -cp /opt/mqm/java/lib/*:javax.ejb-api-3.2.jar: ejbs/IVTMDB.java

The javax.ejb-api-3.2.jar came from
https://download.jar-download.com/cache_jars/javax.ejb/javax.ejb-api/3.2/jar_files.zip

Build the jar file

jar -cvf MDB3.jar ejbs/IVTMDB.class META-INF/weblogic-ejb-jar.xml META-INF/ejb-jar.xml

jar -tvf MDB3.jar gave

0 Fri Aug 02 07:08:38 BST 2019 META-INF/
69 Fri Aug 02 07:08:38 BST 2019 META-INF/MANIFEST.MF
3572 Fri Aug 02 07:05:10 BST 2019 ejbs/IVTMDB.class
1197 Wed Jul 31 17:43:08 BST 2019 META-INF/weblogic-ejb-jar.xml
3696 Tue Jul 30 15:16:52 BST 2019 META-INF/ejb-jar.xml

Deploy the MDB

Using the webLogic admin console
Home → Deployment resources → Deployments

Install
enter the path to the directory with your jar file
select jar file
Install as application
Next
Name MDB3
Finish

Check the webLogic error log for problems.

For example, the log showed

<BEA-160228><AppMerge failed to merge your application. If you are running AppMerge on the command-line, merge again with the -verbose option for more details. See the error message(s) below.>

I went into a command window.
Set up the environment using
.      ./wls12213/wlserver/server/bin/setWLSEnv.sh
and ran

java weblogic.appmerge -verbose ~/temp/MDB/MDB3.jar
I fixed any problems.

( If you get Error: Could not find or load main class weblogic.appmerge, you did not enter the setWLSEnv.sh command properly, check the . before the command.)

When it successfully deployed, the webLogic log had
EJBCreate
EJBCreate
EJBCreate
This was 3 instances ( as defined in the weblogic-ejb-jar.xml file) of my program starting up and printing out a message.

Try it out!

My JNDI file had JMSQ1 → Q1.  I ran a program to put a message, to queue Q1, specifying a reply to queue, and it worked!

I put a message to Q1, and a response was returned.

Thanks to Paul Titherage who gave me good advice on getting the Java program compiled, and other pointers to the correct path.

The fog around Message Driven Beans (MDBs)

While playing with Message Driven Beans and looking at performance problems, I have learned a lot about MDBs and tuning.   The documentation was only partly useful.   There is a lot of documentation  – some concepts are explained 1000 times, but other areas are undocumented – or just wrong.  I’ll try to fill the holes in this blog post.  If I am wrong – please tell me!

This blog post is just one step on the path to how to configure webLogic server to avoid millions of connects a day, and to see if you need to do any tuning ( from an MQ perspective)

I’ll cover

  1. How MDBs work
  2. Different types of Enterprise Java bean standards, and how configuration of EJBs have changed over the years.
  3. Use of resource adapters.

Message listener

We should all be familiar with an application that does an MQGET, a database update, and a commit – which commits both units of work.  A transaction manager manages the units of work with each resource manager (MQ, and database).

The same concept applies to the MDB listener running in a web server.  The listener gets a message from one queue manager, and calls the MDB on a different thread, with the message content.  The MDB connects to a (possibly different) queue manager puts the reply and returns. The listener does a commit. The web server acting as a transaction manager coordinates the commit in both queue managers.  If the application does a roll back – the whole request can be backed out (if the MDB is set up for this).

You can configured the listener tasks so the gets are in syncpoint or not, and the MDB can do its processing in sycnpoint or not.

There are two distinct parts to the MDB.

The listener task described above, and the application code.   This application code does not use the same MQ connection as the listener code.   This may be obvious to most people, but I found documents on the web discussing performance tuning, which said create a connection pool for the listener side of the MDB (which may connect once a day and so a connection pool may not be needed) and did not mention a connection pool for the application side (which does 1 million puts, causing 1 million connects a day and so needs a connection pool!).

The Enterprise Java Bean architecture.

From the dawn of web servers, the architecture has followed the classic pattern.

  1. This is great – every one develops their own way of doing things.  Let each web server manufacturer have their own way of doing this.    Let JMS be different to database access.
  2. The realization that this is not a good idea and we need standards and commonality as there is basically one problem.
  3. Now we have standards and people going along the road together, we find there are improvements we need to make.  It could also be that there is a new leader who says “let us go in a different direction – the grass is greener over here”.  You have to maintain the old way of doing things, and also swallow the new way of doing things.

How does this impact MDBs?

EJB 2.0

With EJB 2. MDBs you have information stored in a JNDI repository.   This could be file based on your local machine, or in LDAP.  In this repository you have

  • a queue resource MYJMSQ and information about it, the MQ Queue name is SERVER, and other parameters.
  • connection information MYCONN, this contains the channel name, host name, port etc.

You configure your MDB with

  1. The name of the class (your program) to use, and the method to be passed messages (this is usually onMessage)
  2. The location of the JNDI repository
  3. What sort of repository it is (what Java class you need to communicate with it)
  4. What connection to use
  5. What queue to use.

The ejb-jar.xml file has “standard” information including the class name  and the method name.
Each web server has their own way of passing in the other information.  For example webLogic has a file weblogic-ejb-jar.xml with information about the location of the JNDI repository, the JNDI connection (MYCONN), and JNDI queue name(MYJMSQ).   It also has information about the connection pool for the listener.  All these files are packaged into the .jar file for the MDB.

IBM’s Liberty web server stores the information elsewhere.

The application has code like

ctx = new InitialContext();
cf = (ConnectionFactory)ctx.lookup(“CF1”);

which says look in the same JNDI repository, for connection CF1.   This could be same connection name as the listener is using, it could be different.  I could not find a way of specifying connection pool information for this connection, so by default every message put to MQ requires an MQCONN and MQDISC.

The application can get the destination name of the replyTo Queue, and uses it

Destination dest = message.getJMSReplyTo();
producer = session.createProducer(dest);
producer.send(response);

The Destination dest will have a value like queue://QMA/QUEUENAME

EJB 3.0

With EJB version 3, a lot has changed.

  1. There is a standard Java Connector Adapter( JCA) I think version 1.5 is current.
  2. The hard work is now done by using a resource adapter (ra)
  3. This has a different way of passing configuration information
  4. You can now hard code data in your program (which seems a step backwards to me)

Key information is stored in activationSpecs.  You can code it in your java application program.   (The resource adapter can query all the activationConfigProperties from a class and then process it), or have it in the .xml configuration files.

For example with the information in your application.java file.

@MessageDriven(
  name = "JMSSampleMDB",
  activationConfig = {
    @ActivationConfigProperty(propertyName  = "destinationType", 
               propertyValue = "javax.jms.Queue"),
     @ActivationConfigProperty(propertyName  = "destination", 
	       propertyValue = "INPUT_Q")                         
  }
)
public class JMSSampleMDB implements MessageListener{
 @TransactionAttribute(value = TransactionAttributeType.REQUIRED)
  public void onMessage(Message message) {...
	 }
}

It can be stored in the ejb-jar.xml file

<activation-config>
  <activation-config-property> 
    <activation-config-property-name>connectionfactory</activation-config-property-name> 
    <activation-config-property-value>CF3</activation-config-property-value>
  </activation-config-property>
  <activation-config-property> 
    <activation-config-property-name>destination</activation-config-property-name> 
    <activation-config-property-value>JMSQ2</activation-config-property-value>
  </activation-config-property>
</activation-config>

The parameters CF3 and JMSQ2 are configured as part of the resource adapter, and not in JNDI.  Each resource adapter has an ra.xml file.  Weblogic stores its specific information in weblogic-ra.xml.  IBM Liberty stores it in the server.xml file.

Once you have installed the resource adapter you should then be able to add additional connections and queues to it.

The resoure adapter takes the definitions and puts them into the JNDI tree within the web server.   Your application can continue to use

ctx = new InitialContext();
cf = (ConnectionFactory)ctx.lookup(“CF1”);

Using this way you can configure your connection as part of a connection pool- and avoid the MQCONN and MQDISC request.

I believe you can define an MDB using definitions in JNDI – and the application part of the MDB can use the resource adapter.

As far as I have been able to find out – if you want to use connection pooling for your application program you need to use a resource adapter.  I’ll blog this in a separate topic.  It is complex and quirky, and will need a lot of description

 

Java whoops causes performance problem.

I had not realized how easy it is to write a bad MQ Java program. When I worked for IBM, I had seen java applications doing 1 million MQCONNs a day – and people complained because it was slow! I accidentally managed to recreate this when looking into another problem.

I’ll show the bad programming example – which may be typical of many applications, and I’ll show what MQ requests the various JMS methods do, and how you can tell if your application is misbehaving.

My bad java program

If I had a java program which does the following

public class ColinDemo{
  public static void doit(){
    while ( i< 100)
    {
      colinMQ(...)
    }
  } 
  colinMQ() { 
     MQQueueManager queueManager = new MQQueueManager("qMgrName");
  ...  
     queueManager.close()
  }
}

Every time colinMQ is entered it creates a new connection to MQ.   When queueManager.close() is executed, there will be an MQDISC

The program as written will cause 100 MQCONNs and 100 MQDISC… whoops.

QueueManager needs to be defined as a class variable for example

public class ColinDemo{
MQQueueManager queueManager = Null;

Then colinMQ() needs a check like
if queueManager == Null then
queueManager = new MQQueueManager(“qMgrName”);
or

public class ColinDemo{
  public static void doit(){
    MQQueueManager queueManager;
    queueManager = initColinMQ(...)
    while ( i< 100)
    {
       colinMQ(queueManager...)
    }
    closeqm(queueManager);
  }
  
  MQQueueManager initColinMQ(...){
  return MQQueueManager("qMgrName")
  } 
  colinMQ(qm... ) { 
  ...
  }
  closeqm(qm)
  {
    qm.close();
  }
}

That is all pretty basic stuff, which is obvious once you see it. The more subtle bug is when an application calls ColinDemo.doit() multiple times. On exit from ColinDemo queueManager  will be closed and will cause it do do an MQDISC, the next call to ColinDemo.doit() will require another MQCONN.

You need to have ColinDemo return instance data. For example in your highest level program you say

ColinDemo cd = new ColinDemo(“qMgrName”)

then you can use cd.doit()… instead of ColinDemo.doit(), and of course you need a cd.close() to close the connection once it has been finished with.

When you are using JMS you need to have
connection = cf.createConnection();
session = connection.createSession(…);
consumer = session.createConsumer(destination);
defined as instance variables so they are set when the thread is started in the morning, and are not changed or de-allocated at the end of the day when the high level program ends.

For example in your top level JMS program

ColinDemo cd = new ColinDemo(“MYQMGR”);
for (i=0 ;i <100;i++) 
{
Message m = cd.getMessage() 
}

with

public class ColinDemo{
  private Connection connection = null;
  private Session session = null;
  private Destination destination = null;
  private MessageConsumer consumer = null;
  private Context context = null;
  private JmsConnectionFactory cf = null;
  private String contextFactory = null; 
  private Hashtable<String, String> environment;

  ColinDemo(String QMGRNAME)
  {
  // Instantiate the initial context
  contextFactory = "com.sun.jndi.fscontext.RefFSContextFactory";
  environment = new Hashtable<String, String>();

  environment.put(Context.INITIAL_CONTEXT_FACTORY,contextFactory);
  environment.put(Context.PROVIDER_URL, initialContextUrl);
  context = new InitialDirContext(environment);
  cf = (JmsConnectionFactory)
  context.lookup(connectionFactoryFromJndi);
  // Lookup the destination
  destination = (JmsDestination) 
  context.lookup(destinationFromJndi);

  connection = cf.createConnection();
  session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
  consumer = session.createConsumer(destination);

  // Start the connection
  connection.start();
  }

  public Message getMessage()
  { 
    Message message = consumer.receive(timeout);
  } 
}

What goes on under the covers.

A basic JMS program looks like

String contextFactory = "com.sun.jndi.fscontext.RefFSContextFactory";
Hashtable<String, String> environment = new Hashtable<String, String>();
environment.put(Context.INITIAL_CONTEXT_FACTORY, contextFactory);
environment.put(Context.PROVIDER_URL, initialContextUrl);
context = new InitialDirContext(environment);
cf = (JmsConnectionFactory) context.lookup(connectionFactoryFromJndi);

// Lookup the destination
destination = (JmsDestination) context.lookup(destinationFromJndi);

connection = cf.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
consumer = session.createConsumer(destination);

// Start the connection
connection.start();
// get a message 
Message message = consumer.receive(timeout);

Using the distributed MQ activity trace we can see the various MQ verbs which are issued.
The program started at 09:00:30 and ended at 09:00:45

connection = cf.createConnection();
has the following accounting trace records
Tid Date       Time     Operation  CompCode MQRC    HObj (ObjName) 
001 2019-07-26 09:00:30 MQXF_CONNX MQCC_OK  0000    -
001 2019-07-26 09:00:30 MQXF_OPEN  MQCC_OK  0000    2    ( ) 
001 2019-07-26 09:00:30 MQXF_INQ   MQCC_OK  0000    2    ( )
001 2019-07-26 09:00:30 MQXF_CLOSE MQCC_OK  0000    2    ( )
001 2019-07-26 09:00:45 MQXF_DISC  MQCC_OK  0000    -

We can see the transaction starting at 09:00:30 and ending at 09:00:45.

This is querying the queue manager for

MQIA_COMMAND_LEVEL (31)
MQIA_PLATFORM (32)
MQIA_CODED_CHAR_SET_ID (2)
MQCA_Q_MGR_NAME (2015)
MQCA_Q_MGR_IDENTIFIER (2032)
MQIA_ADVANCED_CAPABILITY (273)
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

produces the following trace records

Tid Date       Time     Operation  CompCode MQRC    HObj (ObjName) 
001 2019-07-26 09:00:30 MQXF_CONNX MQCC_OK  0000 -
001 2019-07-26 09:00:45 MQXF_DISC  MQCC_OK  0000 -

consumer = session.createConsumer(destination);

adds the following to the session data.

Tid Date       Time     Operation  CompCode MQRC HObj (ObjName) 
001 2019-07-26 09:00:30 MQXF_OPEN  MQCC_OK  0000 2    (Q1 ) 
001 2019-07-26 09:00:45 MQXF_CLOSE MQCC_OK  0000 2    (Q1 )

connection.start();

adds nothing

Message message = consumer.receive(timeout);

adds the following tace data to the session data.

Tid Date       Time     Operation CompCode MQRC HObj (ObjName)
001 2019-07-26 09:00:30 MQXF_GET  MQCC_OK  0000 2    (Q1 )

So by having
connection = cf.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
consumer = session.createConsumer(destination);

Issued just once, you will save many MQCONN, MQDISC, MQOPEN, MQCLOSE and MQINQ calls.
If you see a pattern of

Tid Date       Time     Operation  CompCode MQRC HObj (ObjName) 
001 2019-07-26 09:00:30 MQXF_CONNX MQCC_OK  0000 -
001 2019-07-26 09:00:30 MQXF_OPEN  MQCC_OK  0000 2    ( ) 
001 2019-07-26 09:00:30 MQXF_INQ   MQCC_OK  0000 2    ( )
001 2019-07-26 09:00:30 MQXF_CLOSE MQCC_OK  0000 2    ( )
001 2019-07-26 09:00:45 MQXF_DISC  MQCC_OK  0000 -

You are doing too many connects – and need to look into it!

I am working on some other posts for making your Message Driven Bean applications running in a web server more efficient.

 

Thanks to Roger Lacroix for helping me get the java right.  His comments reminded me of the rules of computing

  • If you open it, close it
  • If you allocate it, free it
  • Clean up after you.
  • Be a good citizen

On the web there are some good example, and some bad examples.  One good example had

try{
    queueManager = new MQQueueManager("qMgrName");
    doWork(queueManager)    
    ...
   } catch (Exception je)
   {...
   }
finally{
    try{ queueManager.close()}
    catch (Exception je){...}
   }

When the execution left the try{} statement, the statement in the finally was always executed, so you could be sure that the queueManager object will always closed.

Other examples did not have code to close the connection, and the connection was left open until the program terminated which could be months later!  Be a good citizen, free it when you do not need it.

 

 

Why is my java application doing multiple gets because the receive buffer is too small?

When investigating a MQ JMS performance problem, I noticed that there was a pattern of

MQGET – 2080 (0820) (RC2080): MQRC_TRUNCATED_MSG_FAILED

followed by an MQGET with a bigger buffer which worked.

Being a performance person these double gets did not look right to me.

I spoke to Paul Titheridge of IBM who said this was fixed back in MQ V6 with a java property

com.ibm.mq.jmqi.defaultMaxMsgSize

If you specify

 -Dcom.ibm.mq.jmqi.defaultMaxMsgSize=8192
in the Java start-up you will get an 8KB buffer instead of the default 4KB.
Easy when you know how!

Showing what the C client is doing and how long it took.

I was asked to get involved in a dispute between the application team, and the MQ team about a slow C application running as a client. The application team felt it was MQ, the MQ team felt it was somewhere else. We had lots of feelings but no facts. I took a client trace, post processed it, and could point the finger of blame. I then spent more time digging into the trace and could see where the time is being spent.

Collecting a trace

We used strmqtrc -e -d 0 to turn on the trace. -d 0 says collect 0 length data buffers.
We ran the application and used endmqtrc -e to stop the trace
As we were on Linux we had to use the dspmqtrc *.TRC to format the trace. You may have lots of file on /var/mqm/trace. The trace of the client will have the userid associated with the application. Files with userid mqm come from the queue manager.

If you look at the trace file there is so much data is it temping to give up.
At the top of the file is a head and includes a line “Program Name :- …” so you can check this is the correct file.

The trace data looks like
20:41:03.511815 17291.5      :      { MQGET
20:41:03.511819 17291.5      :       -{ reqGetConn
20:41:03.511822 17291.5 RSESS:000007 -} reqGetConn rc=OK FunctionTime=3
20:41:03.511826 17291.5 RSESS:000007{ MQGET
  • 20:41:03.511815 is a time stamp
  • 17291.5 is processid.threadid
  • the 17291 is the same as the file name AMQ17291.0.FMT
  • {, -{,–{ shows the nesting level of the function so { is the top level, -{ is the next level deeper etc

Using the command
grep -e ” {” -e ” }” *291*.FMT|less
extracts just high level MQ API calls with the output from different threads intermixed

20:41:03.637143 17291.8 : } MQPUT rc=OK FunctionTime=874
20:41:03.637146 17291.8 : { MQGET
20:41:03.641930 17291.6 : } MQGET rc=OK FunctionTime=4784
20:41:03.641949 17291.6 : { MQPUT

If you look in the trace file near the entry times you will see the MQ control blocks, such as the MQMD, and MQOD so you can identify which queues are being used etc..

grep -e ” {” -e ” }” *291*.FMT| grep 17291.8 |less
shows you the high level data in the trace file for just the thread 8.

(1) 20:41:06.069828 17291.8 : { MQGET
(2) 20:41:06.105633 17291.8 : } MQGET rc=OK FunctionTime=6784
(3) 20:41:06.105656 17291.8 : { MQPUT
(4) 20:41:06.139679 17291.8 : } MQPUT rc=OK FunctionTime=482
(5) 20:41:06.953967 17291.8 : { MQGET

Note. The FunctionTime is often mis calculated and should be ignored.

  • The first MQGET took (2) – (1) 20:41:06.105633 – 20:41:06.069828 = 0.035806 seconds.
  • After the get to the following MQPUT it took (3) – (2) 20:41:06.105656 – 20:41:06.105633 = 0.000023 seconds.
  • The MQPUT took (4)-(3) 20:41:06.139679 – 20:41:06.105656 = 0.034023 seconds
  • After the MQPUT to the following MQGET (5) – (4) it took 20:41:06.953967 – 20:41:06.139679 = 0.814288 seconds

In summary the MQ requests took about 35 milliseconds, between the get and the put was 0.023 milliseconds, but between the put and the following get it took 814 milliseconds. This shows that most of the time was outside of MQ.

This was enough evidence to show the applications people to look else where.

Script to process the trace file and pull out the times in send and waiting for data

It is hard work extract the time stamps and calculating the time differences, so I wrote a simple Python script to do this for me.

This script gave me

(1) 20:41:04.400034 17291.6 : { MQPUT 0.000017
(2) 20:41:04.404946 17291.6 : } MQPUT 0.004912 sendTime 0.000960 waitTIme 0.003281
(3) 20:41:04.404949 17291.6 : { MQGET 0.000003
(4) 20:41:04.416523 17291.6 : } MQGET 0.011574 sendTime 0.000050 waitTIme 0.011167

(2) shows that the MQPUT took 0.004912 seconds, of which 0.000960 seconds was spent doing TCP send requests, and 0.003281 seconds was spent waiting for the request to be executed, and the response sent back. The message was large, so there were multiple send requests.
(3) shows the time between the end of the previous MQPUT, and the MQGET request was 0.000003 seconds.
(4) show the time of the MQGET . Overall it took 0.011574 seconds, of which 0.000050 was doing the send (there was only one send request), and 0.011167 seconds was spent waiting for the request to be executed, and the response sent back.

Digging deeper to see what is going on under the covers

Having a performance background I thought I would dig a bit deeper into this.

The overall picture I have is of getting to the airport. Do I take a taxi or take the bus?
It is usually better for everyone if people take the bus, because it is more efficient overall, and can reduce the amount of traffic. The journey may be a little longer because the bus has to stop to set down and pick up people. If you need to minimise the journey time, I would take a taxi. It may mean I get their faster than the bus – but it may not, if the taxi is queued behind the bus and cannot overtake.

The basic flow of a channel is
1. The application issues an MQ request
2. 1 or more TCP send requests pass the MQ verb and any data (such as message payload for an MQPUT). No more than 64 KB is sent per send request, so big messages will require multiple sends.
3. The last request includes the “end of data” flag.
4. The request is processed
5. The response and any data is sent back, a thread does one or more TCP receives to get the response and any data (for example the message payload for an MQGET request)
6. When all the data has been received, the request returns back to the application

For many years the application sent the data and blocked waiting for the response. If you had an application server, each thread had its own connection. This could lead to a very large number of connections. This configuration is called sharecnv(0)

With shared conversations introduced in MQ V7 there is one TCP connection and threads within a process can use it. The number of threads that can share it is defined with sharecnv(n).
The logic is very similar
1. The application issues an MQ request
2. The application does lock:TCP send:unlock. Multiple threads can send data, so you may get sends for multiple threads intermingled; send for thread 1, send for thread 2, send for thread 1, send for thread 3 etc..
3. The last request for the application request says “end of data”
4. The application thread is suspended.
5. There is a “reply thread” which issues the TCP receive requests.
6. When all the data has been returned for an application, the application thread is woken up.
This has the advantage that there may be only one connection, for multiple threads in the process.  This can be important if you have a very large number of threads using MQ.

With sharecnv(0) the channel was blocked during the MQ request.
There are two (or more) threads, one for sending, and a reply thread, means that asynchronous put, and asynchronous get can be supported.

How does the performance compare?

Sending data….

The TCP send request just passes the data into TCP buffers, so is quick. The TCP protocol says that after a number of bytes have been sent, TCP will wait till it has had confirmation that the earlier bytes have been received at the remote end. So looking at the TCP level you may have send(fast) send(fast) send(fast) send(slow – waiting for the ack from the far end), send (fast) send(slow)

This is the same for sharecnv(0) and sharecnv(> 0).

If you had sharecnv(0), so no sharing of the connection, the channel would do multiple sends, then perhaps do a database update. You may have other threads in the process also doing multiple sends and then perhaps a database update. If you did a packet trace of the network link you might see data for connection 1, data for connection 2, data for connection 1, data for connection 3 – just as for the sharecnv>0 case, but these requests may be done in parallel – depending on the configuration

So for sharecnv(0) and sharecnv(>0) you should expect similar send data rates.

Receiving data

The channels with sharecnv(0) have a dedicated connection. With sharecnv(>1) there is one reply connection shared by the users. With a low to medium throughput this should give the similar throughput as multiple channels with sharecnv(0). Because of the way the low level TCP protocol works, I can see that sometimes a channel defined with sharecnv(>1) may be faster!

With high throughput data may be delayed getting onto the shared connection, whereas with the sharecnv(0) channels the data can be flowing over connections in parallel.

What value should I use in sharecnv(n)

  1. If you want to use the new asynchronous functions, n must be greater than 0.
  2.  If you want the maximum possible throughput and minimum response time – at the cost of a larger number of connections then use n = 1.
  3.  If your throughput requirements are lower, and the applications can tolerate a small increase in response time then a larger n value will mean you need fewer connections – and so your queue manager can support more application instances.