What are the JMS connection factories on webLogic doing?

As part of my long running activity to find out what is causing 1 million MQCONNects a day from an Oracle  webLogic web server, I have found out how to monitor what is going on inside the webLogic instance.

Most web servers support a Java Management eXtension (JMX) interface.  You can use gui tools like jconsole to do an ad-hoc display of the management beans – but these are   not practical for long term monitoring.

Ive listed the data from the JMX query, and also documented how I got the data

The data

  • ActiveConnectionsCurrentCount (Integer) = 9
  • ActiveConnectionsHighCount (Integer) = 10
  • AverageActiveUsage (Integer) = 0
  • CapacityIncrement (Integer) = 1
  • CloseCount (Long) = 5682
  • ConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ConnectionFactoryImpl
  • ConnectionFactoryName (Null)
  • ConnectionIdleProfileCount (Integer) = 0
  • ConnectionIdleProfiles (Null)
  • ConnectionLeakProfileCount (Integer) = 0
  • ConnectionLeakProfiles (Null)
  • ConnectionProfilingEnabled (Boolean) = false
  • Connections (ObjectName[]) = [Ljavax.management.ObjectName;@299a06ac
  • ConnectionsCreatedTotalCount (Integer) = 21
  • ConnectionsDestroyedByErrorTotalCount (Integer) = 0
  • ConnectionsDestroyedByShrinkingTotalCount (Integer) = 11
  • ConnectionsDestroyedTotalCount (Integer) = 11
  • ConnectionsMatchedTotalCount (Integer) = 5670
  • ConnectionsRejectedTotalCount (Integer) = 0
  • ConnectorEisType (String) = Java Message Service
  • CurrentCapacity (Long) = 10
  • EISResourceId (String) = type=<eis>, application=colin, module=colin, eis=Java
  • Message Service, destinationId=CF2
  • FreeConnectionsCurrentCount (Integer) = 1
  • FreeConnectionsHighCount (Integer) = 10
  • FreePoolSizeHighWaterMark (Long) = 10
  • FreePoolSizeLowWaterMark (Long) = 0
  • HealthState (Null)
  • HighestNumWaiters (Long) = 0
  • InitialCapacity (Integer) = 1
  • JNDIName (String) = CF2
  • Key (String) = CF2
  • LastShrinkTime (Long) = 1565334529469
  • LogFileName (Null)
  • LogFileStreamOpened (Boolean) = false
  • LoggingEnabled (Boolean) = true
  • LogRuntime (Null)
  • ManagedConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl
  • MaxCapacity (Integer) = 20
  • MaxIdleTime (Integer) = 0
  • MCFClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl
  • Name (String) = CF2
  • NumberDetectedIdle (Integer) = 0
  • NumberDetectedLeaks (Integer) = 0
  • NumUnavailableCurrentCount (Integer) = 9
  • NumUnavailableHighCount (Integer) = 10
  • NumWaiters (Long) = 0
  • NumWaitersCurrentCount (Integer) = 0
  • Parent (ObjectName) = com.bea:ServerRuntime=AdminServer2,Name=colin, ApplicationRuntime=colin, Type=ConnectorComponentRuntime
  • PoolName (String) = CF2
  • PoolSizeHighWaterMark (Long) = 10
  • PoolSizeLowWaterMark (Long) = 0
  • ProxyOn (Boolean) = false
  • RecycledTotal (Integer) = 0
  • ResourceAdapterLinkRefName (Null)
  • RuntimeTransactionSupport (String) = NoTransaction
  • ShrinkCountDownTime (Integer) = 340
  • ShrinkingEnabled (Boolean) = true
  • ShrinkPeriodMinutes (Integer) = 15
  • State (String) = Running
  • Testable (Boolean) = false
  • TransactionSupport (String) = NoTransaction
  • Type (String) = ConnectorConnectionPoolRuntime

From this I can see there is a pool called CF2 which has maximum of 10 connections.  The maximum connections used was 10, the lowest was 0.

There were Connections Matched Total Count  = 5670 requests for a connection from the pool.

The pool has shrunk more than once as it has Connections Destroyed By Shrinking Total Count  = 11 connections.

Using this data, you can now plot usage over time and see if you need to increase ( or decrease) the size of the pool, or the parameters to tune when the pool is shrunk.

I do not know enough about JMX to tell if the “high” and “low” value are reset on each query, or if you can use JMX to reset them periodically. These high and low value may have little value, if they are since the webLogic instance started (6 months ago).

The data fields are mentioned here.

How I got the data

There is a python package called JMXQuery which has a .jar file which allows you to query information in a JMX server.  The output is in json format so you can use your favourite tools (python) to quickly convert this to other format, such as .csv .

The command I used was

java -Xss409600k -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u webLogic
-p passw0rd -q “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”

which breaks down as follows

  • java -Xss409600k – invoke java with this heap size
  • -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” – this jar file
  • -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi – this is the url of my webLogic server
  • -json – output it in json format
  • -u webLogic -p passw0rd -q  – userid and password
  • “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”
    • com.bea is the bean type
    • The admin server was called AdminServer2
    • The connection factory was CF2
    • The resource adapter is installed under “Deployments”  as colin
    • Type=ConnectorConnectionPoolRuntime is the type of bean

I then used |jq . |grep -v mBean > bb to convert the one line json to one field per line, dropped the mBean value, and put it to a file.  The output was like

[
 {
   "attribute": "Connections",
   "attributeType": "ObjectName[]",
   "value": "[Ljavax.management.ObjectName;@299a06ac"
 },
 {
   "attribute": "FreeConnectionsCurrentCount",
   "attributeType": "Integer",
   "value": 4
},
etc

You can put generics in for example

java -Xss409600k -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u readonly -p read0nly -q “com.bea:ApplicationRuntime=*,ConnectorComponentRuntime=*,Name=”CP*”,ServerRuntime=*,Type=ConnectorConnectionPoolRuntime” e=*,Type=ConnectorConnectionPoolRuntime” > aa

 

Where this uses a userid set up as a monitor id, “*” has been specified for many values, and only give objects beginning with “CP”.  Note blanks have meaning. “,,,=*, Name=…” looks for an object with blank,N,a,m,e,

Data when not using a resource adapter

The above information was for a resource adapter.  When an EJB 2 MDB is deployed (non resource adapter)

from –q  “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3,
MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,
Name=WMQ_IVT_MDB_JMSQ1,ServerRuntime=AdminServer2,Type=EJBPoolRuntime”

Where the weblogic-jar-xml has

<weblogic-ejb-jar> <ejb-name>WMQ_IVT_MDB</ejb-name>
<destination-jndi-name>JMSQ1</destination-jndi-name>

The data was

BeansInUseCount (Integer) = 0
PooledBeansCurrentCount (Integer) = 10
IdleBeansCount (Integer) = 10
BeansInUseCurrentCount (Integer) = 0
DestroyedTotalCount (Long) = 0
WaiterCurrentCount (Integer) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
MissTotalCount (Long) = 7
AccessTotalCount (Long) = 28
Type (String) = EJBPoolRuntime
TimeoutTotalCount (Long) = 0
WaiterTotalCount (Long) = 0

With

-q “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3, MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,Name=WMQ_IVT_MDB_JMSQ1,
ServerRuntime=AdminServer2,Type=EJBTransactionRuntime

the output was

TransactionsRolledBackTotalCount (Long) = 0
TransactionsTimedOutTotalCount (Long) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
Type (String) = EJBTransactionRuntime
TransactionsCommittedTotalCount (Long) = 0

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s