mqweb what’s the difference between the message API and the admin API?

At first glance it looks like the answer is in the question.  You can use

  • the messaging REST API put and get messages
  • the admin REST API to administer queue manager objects

In a couple of places the IBM documentation says you can use the messaging API to administer your objects, which is true at the general sense, but not the specific sense.  Until I hit a problem I thought there was one “messaging REST API” with different flavors of syntax.

Security

The admin API authorisation is managed through <security-role name=”MQWebAdmin”> and <security-role name=”MQWebAdminRO”> sections in the mqwebuser.xml file.

The messaging API authorisation is managed through <security-role name=”MQWebUser”> sections.

Access to resources is done using the Alternate Userid.  I can see in the activity trace that the userid is colinpaice(the id mqweb is running under), but the open of a queue was done with alternate userid testuser.  When I tried to browse messages on a queue, I got a message saying my userid did not have the correct authority. I used setmqaut, and mqsc command refresh security(*) to resolve it.

Cost of the admin interface

The admin interface has a request like

https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*

which returns all of the attributes of the queue CP0000.  From the activity trace we can see

  • MQCONN + MQDISC
  • MQOPEN, MQINQ, MQCLOSE of the manager object – twice
  • MQOPEN, MQPUT, MQCLOSE to the SYSTEM.ADMIN.COMMAND.QUEUE
  • MQOPEN, MQGET, MQCLOSE to the SYSTEM.REST.REPLY.QUEUE
  • MQCMIT
  • MQBACK – the JMQI code always does this to be sure that there is no outstanding unit of work,

The most expensive request is the MQCONNect.

Using the admin interface is fine for administration because changes to objects are usually done infrequently.   If you are considering the admin interface to monitor objects, for example plot queue depths over time, the mq rest API may not be the best solution.

Cost of the messaging interface

The messaging API interface uses connection pooling.   When the application does an MQDISC, the connection is returned to a pool, and can be reused if the same userid does an MQCONN.  If the connection is not used for a period, it can be removed from the pool and an MQDISC done to release the connection.    This should eliminate frequent MQCONN and MQDISCs.

From the activity trace we see

MQOPEN, MQGET,MQGET,MQCLOSE of the queue, and no MQCONN.

There will be an MQCONN, is there is no connection available for that userid in the pool, but this should be infrequent.

Getting MQConsole (brower interface to administer MQ via mqweb) working

It was a new year, as I sat in my basement cave while the gale force winds blow around the house, I thought I would try to use the new MQWeb and MQConsole, and see how it stands up to “the Paice treatement”.    The MQWeb allows you to administer MQ from a web browser, or a rest interface (for example using CURL or Python).  This technology has been around for a few years now.  I know it is being enhanced every few months through the continuous delivery channel.

The installation and getting started reminded me of an old car belonging to my father.  The car was not easy to get started (it had a starting handle!), but once it was started it worked pretty well.

Getting it up and running in a test sandbox took about 1 hour.   It took me about two week to get mqweb set up properly using digital certificates, and to document how I did it.  Being security related, there must be a team which tries to make it as hard as possible to diagnose problems so as not to provide useful information to a hacker.  It also took a while to  work out how to use mqweb  in an enterprise where you have multiple machines and have to support many users.  It also feels a bit buggy and some of it was frustrating, but as it is being continuously improved, I am sure it will get better.

Ive written some blog posts

I had MQ 9.1.3 running on my laptop running Ubuntu 18.04.

Getting it installed and up and running.

Initially I followed the  9.1 instructions here.   After lots of clicking and guessing I got to this page which gave me some instructions (but they were not very helpful). There are various mistakes on the page such as var/mqm/web should be /var/mqm/web.  I ignored the instructions and simply used sudo apt install /home/colinpaice/…/ibmmq-web_9.1.3.0_amd64.deb to install it.

The configuration file /opt/mqm/web/mq/samp/configuration/basic_registry.xml has predefined userids and the configuration is suitable to have an initial look at the MQWEB.

I used

cp /opt/mqm/web/mq/samp/configuration/basic_registry.xml 
/var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml

to copy the configuration file.

Starting and stopping the mqweb

The strmqweb command failed for me.   This was strange  as commands like strmqm works.  This is because there is a symbolic link /usr/bin/strmqm which points to /opt/mqm/bin/strmqm, but no link for the mqweb commands.

See here  which explains there is a /usr/bin/strmqm → /opt/mqm/bin/strmqm , but not for the mqweb stuff. I think this is an IBM Whoops.

I created these myself using

sudo ln -s /opt/mqm/bin/dspmqweb /usr/bin/dspmqweb
sudo ln -s /opt/mqm/bin/endmqweb /usr/bin/endmqweb
sudo ln -s /opt/mqm/bin/setmqweb /usr/bin/setmqweb
sudo ln -s /opt/mqm/bin/strmqweb /usr/bin/strmqweb

The configuration file is deep down a directory tree.

I created  a symbolic link to the file using

ln -s /var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml web.xml

so I can do  gedit ~/web.xml

and if you forget where the file really is, use ls -l web.xml

I used the strmqweb command to start the mqweb server.

I used dspmqweb and got

MQWB1124I: Server ‘mqweb’ is running.

MQWB1123E: The status of the mqweb server applications cannot be determined.  A request was made to read the status of the deployed mqweb server applications, however the data appears corrupt. This may indicate that there is already an mqweb server started on this system, probably related to another IBM MQ instance.

The MQWB1123E message only happened occasionally – I think it is a timing problem and can be ignored.

I stopped the mqweb instance using endmqweb

Log files

There is a file /var/mqm/web/installations/Installation1/servers/mqweb/logs/console.log  which has audit type statement in it.

There is a file /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log which has more messages (including time stamps).   This file is more useful.

I defined a symbolic link to this file, to make debugging easier.

ln -s/var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log messages.log 

When the strmqweb command is issued,

  • it deletes the previous console.log
  • it rename the messages.log to a file with a time stamp in the file name
  • it deletes any other message logs files.

After starting and stopping the web server several times the only files I had were

  • messages_20.01.05_15.18.50.0.log
  • messages.log
  • console.log

You may want to put the strmqweb command in a shell script which saves away any message and console files.

The command dspmqweb gives output like

MQWB1124I: Server 'mqweb' is running.
URLS:
  https://localhost:9443/ibmmq/rest/v1/
  https://localhost:9443/ibmmq/console/

This tells you which URL you need to use.

Note: port 9443 is the default port for WebSphere Liberty Profile.  If it is in use you will have to configure a different port.

First logon

I logged on to Firefox browser using the address https://localhost:9443/ibmmq/console/Make sure you the https in https:… .  If you use http: without the https, the logon fails with message “The connection was reset”.

Using https:… gave me big error screen and

Warning: Potential Security Risk Ahead 
localhost:9443 uses an invalid security certificate.
The certificate is not trusted because it is self-signed.
Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT

While  you are exploring the mqconsole, you can accept this.  To fix it properly is a big piece of work.  See my other blog posts.

I signed on using userid mqadmin and password mqadmin and it showed the queue managers.

Select the row of an active queue manager. The table header changes to give options.  Select properties to display the queue manager properties.

The queue manager attributes do not refresh in real time

You have to go back to the queue manager table and re-display the data.   This is not a big problem as the attributes do not typically change frequently.  I noticed this when I changed an accounting parameter, and the attribute page did not show the change.

Adding widgets to the dashboard.

There are two ways of adding widgets for MQ objects.

  1. From the list of queue manages, select a queue manager, then on the title like, click on the “…”  (more actions) button and select “Add new dashboard tab”. This creates a dashboard with all of the MQ Objects defined, MQ Queues, client connections, MCA connections, listeners etc. You can select and delete widgets you do not need.
  2. Click on the “Add widget” button.

It may be quicker and easier to use the first option to add all widgets and delete the widgets you do not need.

Create more tabs

At the top of the browser window next to the “tab”,  click on the “+”. This defines a new dashboard, use the add widget button to select the widget you want to define.

Each userid has their own dashboard (tab layout and widget)

See the next topic if you want people to have the same dashboard.

Export the dashboard for enterprise deploy or backup

At the top of the screen is an icon with three vertical dots for dashboard settings. You can export the dashboard and widgets to a JSON file.

  • You can change the queue manager names and import it on another queue manager.  This is useful to enterprise users who have to support many queue managers in a similar environment.  Being a JSON file you can process the JSON to change queue manager names.  I could not find a way of importing it except from a web page.  This make it challenging to deploy automatically.
  • You can have another user import it, so they get the same dashboard.  If it changes, they have to import it manually.
  • You may want to export your dashboard every week and back it up.

Using the widgets

I clicked on the “Queue on …”  widget.

I clicked on the “Queue depth” column for queues, and it quickly sorted the queue depth.

I could see I had a total of 33 non system queues. By clicking on the settings wheel, I could select “show system objects”.

If you select the settings wheel, you can select a different queue manager.  By changing this you could have one tab showing queues on different queue managers on the machine, and another tab showing channels on different queue managers on the machine.  You could also have a tab per queue manager, and have queues and channels for one queue manager on that tab.

I could refresh a widget by using the refresh icon.

There is a search box at the top of each widget. It searches for the value in any column. So typing in 003 gave me queue CP00003 and DEEPQ with depth 1000003.
At the bottom of the widget it said Total: 90 Filtered:2

If you select a row, the search box changes and give you a list of actions.

  • Delete queue
  • Properties
  • Put message
  • Browse message
  • More actions → Manage authority records
  • 1 item selected
  • Deselect

You can select all the objects in a widget by typing “a” , or to deselect using “shift a”.  Note: it selects all items – not just the filtered items. For example I typed “a” and the header line said “33 objects selected”. At the bottom of the widget is said total 33 filtered 8.  So be careful if you were thinking of doing bulk changes on all objects.

I was unable to select more than one object, using the cursor keys.

It was easy to delete widgets by selecting the X icon.

You can move the widgets around by grabbing the title line and dragging it.

If you hover on the title line of a widget, a pencil icon appears which allows you to rename the widget.

You can control how many widgets are displayed per line by clicking on the down arrow in the tab (at the top of the page) and selecting how many columns to use.  This is a not very smart.

  • I selected 5 column layout.
  • It did not reflow the widgets automatically.  Each line had 2 widgets and lots of space to the right.  I could drag a widget to the top line.  If I then went to 2 column layout, and back to 5 column layout – I got back to two widgets per line
  • If you select an item, the search box becomes a list of icons.  With a narrow widget, you only get the as many icons as fit in the space, for example you do not get the  “…” (more action) icon.
  • The formatting within a table is not very smart. I had a truncated queue name SYSTEM.ADMIN.CH and lots of space for the queue depth. I think the data is displayed in a table and the columns are the same width, and not changeable.

It may be better to have no more than 2 or 3 widgets per line.

Using operating system security.

The basic mqweb configuration file used hard coded userids mqadmin with password mqadmin. This is not very secure.

You can use the operating system userids and passwords using a different configuration file

I used

  • cp /opt/mqm/web/mq/samp/configuration/local_os_registry.xml  /var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml
  • chmod o+w /var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml
    • to give me update access to the file.

I changed my file to have

<enterpriseApplication id="com.ibm.mq.console">
  <application-bnd>
    <security-role name="MQWebAdmin">
      <user name="colinpaice" realm="defaultRealm"/>
    </security-role>
    <security-role name="MQWebAdminRO">
      <group name="test"/>
    </security-role>
   </application-bnd>
</enterpriseApplication>

Notes.

  • The realm=”defaultRealm” is to do with Jave Enterprise Edition security. Just specify it.
  • Each security-role name section must be unique. I specified <security-role name=”MQWebAdminRO”>… twice. Only the last one was used, I was hoping it would be cumulative.
  • You can specify multiple <user …> or <group… > lines.

See here  and here  for pointers to the IBM documentation.

Managing mqwebuser.xml

You can include files into the mqwebuser.xml files using the xml

<include optional="true" location="pathname/filename"/>
or
<include optional="true" location="url"/>

You can put groups of definitions in one file and have them included.

For example in the file payroll.xml have

<group name="mqsysprog"/>
<group name="payroll"/>

For each of the configuration files for the payroll queue managers have

<security-role name="MQWebAdmin">
  <include optional="true" location="payroll.xml"/> </security-role> <security-role name="MQWebAdminRO"> <group name="test"/> </security-role>

How do I check what role I have?

At the top right of your browser window is a porthole with a circle in it. Click on this, and then click on “about”. It gave me

Principal:colinpaice - Administrator (Password Authentication)
A different userid gave
Principal:testuser - Read-Only Administrator (Password Authentication)

Can I have the logon time out?

Yes, you set a time out value using the ltpaExpiration value. See here.

Use dspmqweb properties -a|grep ltpaEx  and note the ltpaExpiration value.

Use  setmqweb properties -k ltpaExpiration -v time    to set the time in minutes.

Note:

  • After you are logged on for this time period, your session is cancelled and you have to logon again, this happens whether the session is busy or idle.
  • The setmqweb command updates the mqwebuser.xml file on disk. If you were editing the file you will need to reload the file from disk and reapply the changes.
  • The above setmqweb command added <variable name=”ltpaExpiration” value=”10″/> to the mqwebuser.xml file. You could just update the file yourself and avoid this concurrent  update problem.

There is one timeout value for all users, so if you have a screen displaying charts from mqweb, this will also time out.

If you are using certificates to provide authentication

  • your session will be dropped, and automatically reconnected.
  • you cannot logoff – you have to drop the browser tab
  • in the top right of your page the icon will be a black circle with a which “i” in it.  If you are not using certificates this will be a porthole with a circle in it.

Stackoverflow: What throughput can a standalone Java program achieve?

There was a question on the MQ section on StackOverflow

I have a standalone multi threaded java application which listen messages from IBM MQ.
Current system take around 500ms for processing of 1 message after it read from queue and till it commit.
I want to know how many messages I can consume

  • Concurrently:
  • Max number of messages can be processed? or throttle limit

A good meaty performance question I thought.  Let me break this into pieces.

Current system take around 500ms for processing of 1 message after it read from queue and till it commit.

Processing one messages and commit should take about 10 milliseconds or less( say 30 ms for a two phase commit).    There is clearly something else going on.  Fix this first.

  1. A long database call.   This could be due to database locking, or a badly designed statement, for example a query which needs to access thousands or millions of rows.
  2. A request to a server far far away
  3. A file system with the speed of writing an illuminated letter to parchment

How many messages I can consume: Concurrently:

Take the worst case of using persistent messages, which require log IO during commit.

For one thread, processing multiple messages before doing a commit means the thread can do more work.  Consider a get taking 1 millisecond, and a commit taking 10 ms. This is one message processed every 11 ms.  If you did 50 gets – taking 50 ms and a commit taking 10 ms, this is 50 messages in 50 + 10 ms which equates to one message every 1.2 milliseconds almost 10 times faster.    This is how channels can send messages efficiently.   There is a “sweet spot” of messages per commit to give you maximum data processed per second.   This depends on the message size, logging rates and other factors.  For a 100MB message it is one message per commit.  For 10KB messages,  this may be 1000 messages per commit.

This may be selfish

This is clearly a great improvement, but possibly selfish.  If the application logic is a get followed by a database insert, followed by a commit, then doing 50 gets, 50 inserts and a commit, will work much faster.  The down side is that the database requests will keep locks until the commit.  These locks may prevent other applications from accessing data, either the recently inserted  records, page locks, or index locks. So overall MQ throughput goes up – but the business transaction suffers.    You need to understand the database and find the optimum number of requests per commit for your business transaction.

How long before the data is visible?

Rather than have one thread process 1000 messages per commit (taking 1010 ms) you may want to have multiple threads processing 10 messages per commit – taking 20 ms.  This means that the data in the database (or replies etc) are visible earlier.    This may be important to your business transaction if you have to worry about response time.

Parallel  threads

  1. Using more threads should improve throughput, unless this is delayed by external factors – such as database locks.
  2. One customer found one thread was optimum because there was no database delays.

How many messages I can consume: Max number of messages can be processed? or throttle limit

There are papers written on this but here is a one minute overview

As fast as the queue manager can process data

  1. The rate at which MQ can write its logs
  2. Keep queue data in memory – ( buffer pools on z/OS, queue buffer on midrange), so few messages on the queue.

Threads

  1. Having parallel threads gives you better throughput than one thread.  You get overlapped writing to the log, the units of work are shorter in duration, you can get parallel IO.
  2. You may be limited by the network.   Having multiple threads from an application means the network can be better utilized.  One thread can be receiving data down the wire, while another thread is waiting in commit.
  3. You may be limited by where your programs run – eg short of CPU, or slow IO (for your System.out.println statements)

Application design

  1. You may get delays due to serialization if all thread are using the same queue.
  2. Remove the debug printf or System.out.println statements.
  3. Using a queue per business application is better than all applications sharing the same queue
  4. Using one reply to queue per web server may be better than a shared reply to queue – especially if you use Apache Camel.
  5. Use get first if possible.  Avoid scans of the queues.

 

The short answer….

You should be able to get thousands of 1KB messages a second through your Java application when using multiple threads.

 

How do I get a client to disconnect?

I had a question from a customer who asked how they can reduce the number of client connections in use.  They had tried setting a disconnect interval (DISCINT) on the channel, but the connections were like weeds – you kill them off, and they grow back again.

DISCINT is “the length of time after which a channel closes down, if no message arrives during that period”.  This sounds perfect for most people.   The application is in an MQGET, and if no messages arrive, the channel can be disconnected, and the application gets connection broken.   The application can then decide to disconnect or reconnect.
If the application is not in an MQGET, then it will get notified of the broken connection next time it tries to use MQ.

Independent applications

Many applications are well written in that when they get Connection Broken, they just reconnect again, and so the DISCINT has no effect on reducing the number of connections. This may be good for availability but not for resource usage.   It may be good to have 1000 application instances running the day, but perhaps not overnight when there is no work to do.   Ive seen instances where the applications do an MQGET every minute, and with 1000 instances this can use a lot of CPU and doing no useful work.  In this case you want unused application instances to stop, and be restarted when needed.

You cannot use triggering with client connections (unless you have a very smart trigger monitor to produce an event which says start a client program over there).

Use automation periodically check the queue depth, and number of input handles. If there is a high queue depth, or a low number of handles(eg 2)  then start more application instances, across your back-end servers.  Your applications can then disconnect if they have not received a message within say 10 minutes.  This should keep the right number of application instances active.

An administrator should be able to get this automation set up, but getting the application to connect could be a challenge, as this requires the application developer to change the code!

Running under a web server

If your applications are running under a web server you may have mis-configured connection pools.  You can specify the initial size of the pool, and this many connections are made.  As more connections are needed, then more can be added to the pool until the pool maximum is reached. You should specify a time out value, so periodically the pool gets cleaned up, and unused connections are removed, until the pool is back to the initial size.  You should review the initial size of the pools ( is it too large), and the value of the time out value.

This should just be an administrative change.

Good luck, you may be successful in reducing the number of client connections, but do not set your hopes too high.

WebSphere Liberty connectionPool statistics

This blog post explains how to get and understand statistics from WebSphere Liberty on connectionPool usage.

In your MDB application you can have code like

 InitialContext ctx = new InitialContext();
ConnectionFactory cf = (ConnectionFactory)ctx.lookup("CF3");

This says lookup the connection defined by CF3 and issue MQCONN for this connection.

In WebSphere Liberty you defined connection information in server.xml.  For example

<jmsConnectionFactory jndiName="CF3" id="CF3ID">
  <connectionManager maxPoolSize="2" connectionTimeout="7s"/> 
  <properties.wmqJms 
   queueManager="QMA"
   transportType="BINDINGS"
   applicationName="Hello"/>
</jmsConnectionFactory>

The maxPoolSize gives the maximum number of connections available in this pool.

If server.xml has

<featureManager>
   <feature>monitor-1.0</feature>
</featureManager>

then you can get out statistics on connectionPools using the JMX interface.

In ./usr/servers/test/jvm.options I had

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Which defined the JMX port as 9010, and so I can get information through this port.

Looking at the output

There is documentation here on the connectionPool statistics.

You can use jconsole to get the JMX data, but this is not very usable, so I used jmxquery, which is part of a python package.  I installed it using pip install jmxquery.

I used the command

java -jar jmxquery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi -u admin c -p admin -q ‘WebSphere:*’ > outputfile

-q ‘WebSphere:*’  means give all records belonging to the WebSphere component.  If you say -q ‘*:*’ you get statistics for all components, see the bottom of the blog post.  Example output is given below.

This command wrote all of the output to file outputfile.  I then used grep to extract the relevant records.

grep WebSphere:type=ConnectionPoolStats,name outputfile

If you change a parameter in server.xml for the jmsConnectionPool, the pool is deleted, recreated, and the JMX data is reset.   If the pool has been reset, or not been used, statistics for that pool are not available.  On the first use of the pool the pool is created, and JMX statistics are available.

The JMX data for connectionPools

The data was like

WebSphere:type=ConnectionPoolStats,name=CF3/CreateCount (Long) = 2

The detailed records for WebSphere:type=ConnectionPoolStats,name=CF3 are

  • CreateCount (Long) = 2   this is the number of connections created,
  • DestroyCount (Long) = 0 this is the number of connections released because the pool was purged,
  • WaitTime (Double) = 76.36986301369863  there were insufficient threads.  For those threads that had to wait, this is the average wait time before a connection became available,
  • InUseTime (Double) = 18.905405405405407 the threads were active this time on average,
  • WaitTimeDetails/count (Long) = 98 requests because had to wait,
  • WaitTimeDetails/description (String) = Missing,
  • WaitTimeDetails/maximumValue (Long) = 110  the maximum wait time in milliseconds,
  • WaitTimeDetails/mean (Double) = 78.13265306122449 the average wait time,
  • WaitTimeDetails/minimumValue (Long) = 16 the minimum wait time,
  • WaitTimeDetails/standardDeviation (Double) = 16.474205982730254 the standard deviation,
  • WaitTimeDetails/total (Double) = 7657.0 in milliseconds.  7657/(number of waits 98) = average 78.13 (above),
  • WaitTimeDetails/unit (String) = UNKNOWN looks like a bug – this should be milliseconds,
  • WaitTimeDetails/variance (Double) = 271.82426517365184 ,
  • ManagedConnectionCount (Long) = 2  The total number of managed connections in the free, shared, and unshared pools,
  • ConnectionHandleCount (Long) = 0  this is the current handles in use,
  • FreeConnectionCount (Long) = 2  this is the number of connections in the pool, but not in use,
  • InUseTimeDetails/count (Long) = 101 – number of requests for a connection (ctx.lookup(“CF3”)),
  • InUseTimeDetails/description (String) = Missing,
  • InUseTimeDetails/maximumValue (Long) = 53 the maximum time the connection as in use in milliseconds,
  • InUseTimeDetails/mean (Double) = 18.099009900990097  the average time the connections were in use in milliseconds,
  • InUseTimeDetails/minimumValue (Long) = 10  the minimum time the connections were in use in milliseconds,
  • InUseTimeDetails/standardDeviation (Double) = 5.63923216261808,
  • InUseTimeDetails/total (Double) = 1828.0  in milliseconds.   This value(1828)/(number of connections used 101) gives the mean value 18.09 above,
  • InUseTimeDetails/unit (String) = UNKNOWN.

Note the order of the record can vary, for example CreateCount, can be first, or nearly last.

After a time interval aged connections can be released.  When there is sufficient workload to need more connections, they will be created as needed.  If the CreateCount increases significantly during the day, you may either have an irregular workload, or you need to increase you connectionTimeout value, to smooth out the connect/disconnect.

Having WaitTimeDetails/count=0 is good.  If this number is large in comparison to InUseTimeDetails/total then the pool is too small.

Other data you can get from JMX

  • IBM MQ:type=CommonServices
  • java.lang:type=ClassLoading
  • java.lang:type=Compilation
  • java.lang:type=GarbageCollector
  • java.lang:type=Memory
  • java.lang:type=Threading
  • osgi.core:
  • JMImplementation:type=MBeanServerDelegate
  • java.util.logging:type=Logging
  • java.nio:type=BufferPool,name=direct
  • java.lang:type=MemoryManager
  • java.lang:type=MemoryPool,name=Code Cache
  • java.lang:type=OperatingSystem
  • java.lang:type=Runtime
  • WebSphere:feature=apiDiscovery,name=APIDiscovery
  • WebSphere:feature=kernel,name=ServerInfo
  • WebSphere:type=JvmStats
  • WebSphere:type=ThreadPoolStats
  • WebSphere:type=ConnectionPoolStats ( as described above)
  • WebSphere:service=com.ibm.websphere.application.ApplicationMBean,name=CCP

How do I make my MDB transactional?

I found from the application trace  that my MDB was doing MQGET, MQCMIT in the listener, and MQOPEN, MQPUT, MQCLOSE and no MQCMIT in my application.    Digging into this I found that the MQPUT was NO_SYNCPOINT, which was a surprise to me!

My application had session = connection.createSession(true, 1); // true = transactional. So I expected it to work.

The ejb-jar.xml had

enterprise-beans
  message-driven
    transaction-type Container
...
assembly-descriptor
  container-transaction
    trans-attribute NotSupported

I changed NotSupported to Required and it worked.

 

The application trace for the Listener part of the MDB gave me

Operation      CompCode MQRC HObj (ObjName) 
MQXF_XASTART            0000 -
MQXF_GET       MQCC_OK  0000    2 (JMSQ2 )
MQXF_XAEND              0000 -
MQXF_XAPREPARE          0000 -
MQXF_XACOMMIT           0000 -

The trace for the application part of the MDB gave me

Operation                    CompCode MQRC HObj (ObjName)
MQXF_XASTART                             0000         –
MQXF_OPEN             MQCC_OK   0000         2 (CP0000 )
MQXF_PUT                MQCC_OK   0000          2 (CP0000 )
MQXF_CLOSE           MQCC_OK   0000          2 (CP0000 )
MQXF_XAEND                                0000         –
MQXF_XAPREPARE                       0000 –
MQXF_XACOMMIT                        0000 –

and the put options had _SYNCPOINT.

I had read documentation saying that you needed to have XAConnectionFactory instead of ConnectionFactory.  I could not get this work,  but found it was not needed for JMS;  it may be needed for JDBC.

On Weblogic why isnt my MDB scaling past 10 instances?

This is another tale of one step back,  two steps sideways.  I was trying to understand why the JMX data on the MDBs was not as I expected, and why I was not getting tasks waiting.  I am still working on that little problem, but in passing I found I could not get my MDBs to scale.  I have rewritten parts of this post multiple times, as I understand more of it.  I believe the concepts are correct, but the implementation may be different to what I have described.

There are three parts to an MDB.

  1. A thread gets a message from the queue
  2. The message is passed to the application “OnMessage() method of the application
  3. The application uses a connection factory to get a connection to the queue manager and to the send the reply.

Expanding this to provide more details.

Thread pools are used to reuse the MQ connection, as the MQCONN and MQDISC are expensive operations.  By using a thread pool, the repeated MQCONN and MQDISC can be avoided.

There is a specific pool for the application, and when threads are released from this pool, they are put into a general pool.   Periodically  threads can be removed from the general pool, by issuing MQDISC, and then deleting the thread.

Get the message from the queue

The thread has two modes of operation Async consume – or plain old fashioned MQGET.

If the channel has SHARECNV(0) there is a  listener thread which browses the queue, and waits a short period( for example 5 seconds)  for a message.  There is a short wait, so that the thread can take action if required ( for example stop running).  This means if there is no traffic there is an empty MQGET every 5 seconds.   This can be expensive.

If the channel has SHARECNV(>0) then Asyn consume is used.  Internally there is a thread which browses the queue, and multiple threads which can get the message.

The maximum number of threads which can get messages is defined in the ejb-jar.xml activation-config-property-name maxPoolDepth  value.

These threads are in a pool called EJBPoolRuntime.  Each MDB has a thread pool of this name, but from the JMX data you can identify the pool as the JMS descriptor has a descriptor like MessageDrivenEJBRuntime=WMQ_IVT_MDB, Name=WMQ_IVT_MDB, ApplicationRuntime=MDB3, Type=EJBPoolRuntime, EJBComponentRuntime=MDB3/… where my MDB was called MDB3.

The parameters are defined in the ejb-jar.xml file.   The definitions are documented here.  The example below shows how to get from a queue called JMSQ2, and there will be no more than 37 threads able to get a message.

ejb-jar
  enterprise-beans
    message-driven
      activation-config
        activation-config-property>  
          activation-config-property-name maxPoolDepth  
            activation-config-property-value 37
          activation-config-property-name destination 
            activation-config-property-value JMSQ2
 

Note:  I did get messages like the following messages which I ignored ( as I think they are produced in error)

    • <Warning> <EJB> <BEA-015073> <Message-Driven Bean WMQ_IVT_MDB(Application: MDB3, EJBComponent: MDB3.jar) is configured with unknown activation-config-property name maxPoolDepth>
    • <Warning> <EJB> <BEA-015073> <Message-Driven Bean WMQ_IVT_MDB(Application: MDB3, EJBComponent: MDB3.jar) is configured with unknown activation-config-property name destination>

The default value of maxPoolDepth is 10 – this explains why I only had  10 threads getting messages from the queue.

Passing the message to the application for processing.

Once a message is available it will pass it to the OnMessage method of the application. There is some weblogic specific code, which seems to add little value. The concepts of this are

  1. There is an array of handles/Beans of size max-beans-in-free-pool.
  2. When the first message is processed, create “initial-beans-in-free-pool” beans and populate the array invoke the EJBCreate() method of the application.
  3. When a message arrives, then find a free element in this array,
    1. If the slot has a bean, then use it
    2. else allocate a bean and store it in the slot.   This allocation invokes the EJBCreate() method of the application.  On my laptop it took a seconds to allocate a new bean, which means there is a lag when responding to a spike in workload.
    3. call the OnMessage() method of the application.
  4. If all of the slots are in use – then wait.
  5. On return from the OnMessage() flag the entry as free
  6. Every idle-timeout-seconds scan the array, and free beans to make the current size the same as the in initial-beans-in-free-pool.  As part of this the EJBRemove() method of the application is invoked.

The definitions are documented here.

weblogic-ejb
  weblogic-enterprise-bean-jar
    pool    
      max-beans-in-free-pool 47
      initial-beans-in-free-pool 17 
      idle-timeout-seconds 60

I could find no benefit in using this pool.

The default max-beans-in-free-pool is 1000 which feels large enough.  You should make the initial-beans-in-free-pool the same or larger than the number of threads getting messages, see maxPoolDepth above.

If this value is too small, then periodically the pool will be purged down to the initial-beans-in-free-pool and then beans will be allocated as needed.  You will get a periodic drop in throughput.

Note the term max-beans-in-free-pool is not entirely accurate: the maximum number of threads for the pool is current threads in pool + active threads.   The term max-beans-in-free-pool  is accurate when there are no threads in use.

In the JMX statistics data, there is information on this pool.   The data name is likecom.bea:ServerRuntime=AdminServer2, MessageDrivenEJBRuntime=WMQ_IVT_MDB, Name=WMQ_IVT_MDB, ApplicationRuntime=MDB3, Type=EJBPoolRuntime, where WMQ_IVT_MDB is the display name of the MDB, and MDB3 is the name of the jar file.  This allows you to identify the pool for each MDB.

Get a connection and send the reply – the application connectionFactory pool.

The application typically needs to issue an MQCONN, MQOPEN of the reply to queue, put the message, and issue MQDISC before returning.   This MQCONN, MQDISC is expensive so a pool is used to save the queue manager connection handle between calls.  The connections are saved in a thread pool.

In the MDB java application there is code like ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF3”);

Where the connectionFactory CF3 is defined in the resource Adapter configuration.

The connectionFactory cf can then be used when putting messages.

The logic is like

  • If there is a free thread in the connectionFactory pool then use it
  • else there is no free thread in the connectionFactory pool
    • if the number of threads in the connectionFactory pool at the maximum value, then throw an exception
    • else create a new thread (doing an MQCONN etc)
  • when the java program issues connection.close(), return the thread to the connectionFactory pool.

It looks like the queue handle is not cached, so there is an MQOPEN… MQCLOSE of the reply queue for every request.

You configure the connectionFactory resource pool from: Home, Deployments, click on your resource adapter, click on the Configuration tab, click on the + in front of the javax.jms, ConnectionFactory, click on the connectionFactory name, click on the Connection Pool tab, specify the parameters and click on the save button.
Note: You have to stop and restart the server or redeploy the application to pick up changes!

This pool size needs to have enough capacity to handle the case when all input threads are busy with an MQGET.

JMX provides statistics with a description like com.bea: ServerRuntime=AdminServer2, Name=CF3, ApplicationRuntime=colinra, Type=ConnectorConnectionPoolRuntime, ConnectorComponentRuntime=colinra_colin where CF3 is the name of the connection pool defined to the resource adapter, colinra is the name I gave to the resource adapter when I installed it, colin.rar is the name of the resource adapter file.

Changing userids

The application connectionFactory pool can be used by different MDBs.  You need to make sure this pool has enough capacity for all the MDBs using it.

If the pool is used by MDBs running with different userids, then when a thread is obtained, it the thread was last used for a different userid, then the thread has to issue MQDISC and MQCONN with the current userid, this defeats the purpose of having a connection pool.

To prevent this you should have a connection pool for MDBs running with the same userid.

Getting a userid from the general pool may have the same problem, so you should ensure your pools have a maxium limit which is suitable for expected peak processing, and initial-beans-in-free-pool for your normal processing.   This should reduce the need to switching userids.

Cleaning up the connectionFactory

When the connectionFactory is configured, you can specify

  • Shrink Frequency Seconds:
  • Shrink Enabled: true|false

These parameters effectively say after after the “Shrink Frequency Seconds”, if the number of threads in the connectionFactory pool is larger than the initial pool size, then end threads (doing an MQDISC) to reduce the number of threads to the  initial pool size.   If the initial pool size is badly chosen you may get 20 threads ending, so there are 20 MQDISC, and because of the load, 20 threads are immediately created to handle the workload.  During this period  there will be insufficient threads to handle the workload, so you will get a blip in the throughput.

If you have one connectionFactory pool being used by a high importance MDB and by a low importance MDB, it could be that the high importance MDB is impacted by this “release/acquire”, and the low priority MDB is not affected.  Consider isolating the connectionFactory pools and specify the appropriate initial pool size.

To find out what was going on I used

  • DIS QSTATUS(inputqueue)  to see the number of open handles.   This is  listener count(1) + current number of threads doing MQGETS, so with maxPoolDepth = 19, this value was up to 20.
  • I changed my MDB application to display the instance number when it was deployed.
    •  ...
       private final static AtomicInteger count = new AtomicInteger(0); 
      ...
      public void ejbCreate() {
        SimpleDateFormat sdftime = new SimpleDateFormat("HH:mm:ss.SSS"); 
        Timestamp now = new Timestamp(System.currentTimeMillis());
        instance = count.addAndGet(1); 
        System.out.println(sdftime.format(now) +":" + this.getClass().getSimpleName()+":EJBCreate:"+instance);
      }
      public void ejbRemove()
      {
        System.out.println(this.getClass().getSimpleName()+":EJBRemove:"+instance 
                           +" messages processed "+messageCount);
        count.decrementAndGet();
      }

This gave me a message which told me when the instance was created, so I could see when it was started.   I could then see more instances created as the workload increased.

07:16:50.520:IVTMDB:EJBCreate:0

  • By using a client connection, I could specify the appltag for the connection pool and so see the number of MQCONNs from the application connectionFactory.

What happens if I get the numbers wrong?

  1. If the input queue is slow to process messages, or the depth is often too high, you may have a problem.
  2. If ejb-jar.xml maxPoolDepth is too small, this will limit the number of messages you can process concurrently.
  3. The weblogic max-beans-in-free-pool is too small. If the all the beans in the pool(array) are busy, consider making the pool bigger.   The requests queue in the listeners waiting for a free MDB instance.   However the JMX data has fields with names like “Wait count”.   In my tests these were always zero, so I think these fields are of no value.
  4. The number of connections in the connectionFactory is too small.  If the number of requests exceeded the pool size the MDB instance got an exception.  MQJCA1011: Failed to allocate a JMS connection.  You need to change the resource adapter definition Max Capacity for the connectionFactory pool size.
  5. If you find you have many MQDISC and MQCONNs happening at the same instance, consider increasing the initial size of the connectionFactory pool.
  6. Make the initial values suitable for your average workload.  This will prevent  the periodic destroy and recreate of the connections and beans.

 

You may want to have more than one weblogic server for availability and scalability.

You could also deploy the same application with a different MDB name, so if you want to stop and restart an MDB, you have another MDB processing messages.