Error connecting to JMX endpoint: Unsupported protocol: rest

I got this error message when I was trying to use JMX into the WLP web server, when using the restConnector-2.0 interface in Liberty.

The documentation was not that helpful. Oneproblem was I suffered from the curse of cut and paste, and used a Windows environment variable %JAVA_HOME% instead of using the Linux $JAVA_HOME. Another problem was caused documentation saying add the jar to the class path, and then the class path was ignored.

True documentation.

When you use the -jar option, the JAR file is the source of all user classes, and other user class path settings are ignored. See here.

Unhelpful documentation

Configuring secure JMX connection to Liberty says

jconsole -J-Djava.class.path=...;%WLP_HOME%/clients/restConnector.jar

This was for Windows – on Linux it would be $WLP_HOME – except on my system $WLP_HOME was not set.

jconsole

Some of the jar files are in $JAVA_HOME, you can use the environment variable, or specify it yourself. Note %JAVA_HOME% is windows, so be careful when you use cut and paste.

jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar
jconsole -J-Djava.class.path=/usr/lib/jvm/java-8-openjdk-amd64/lib/jconsole.jar:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar

If the window displays “Secure connection failed” restart jconsole and use the -debug option. For me this gave “java.io.IOException jmx.remote.credentials not provided. Set to a String[2] {user,password}”. I entered my userid and password, and this connected successfully.

I could not connect using my .jar file.

I was using JMXQuery to extract the data. I was getting the Error connecting to JMX endpoint: Unsupported protocol: rest message.
Adding the jar file to my class path did not help, as the class path is ignored when using java the -jar parameter.

How to fix it

There are two ways of fixing this.

Put the required jar file in the extensions path, not the class path
Use the java -classpath… instead of specifying java -jar

1. You need to have the jar for in the extensions path, not in the class path.

See How classes are found in the Java documentation. It says

Bootstrap classes – Classes that comprise the Java platform, including the classes in rt.jar and several other important jar files.
Extension classes – Classes that use the Java Extension mechanism. These are bundled as .jar files located in the extensions directory.
User classes – Classes defined by developers and third parties that do not take advantage of the extension mechanism. You identify the location of these classes using the -classpath option on the command line (the preferred method) or by using the CLASSPATH environment variable.

There is a system property java.ext.dirs which gives the location of the Extension classes. On my system this was

/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext

The /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext: is java dependent. You should not put your files in this directory
/usr/java/packages/lib/ext This is for “user” extensions.

The directory /usr/java/packages/lib/ext did not exist on my system, so I had to do, create it, copy the web connection jar file to it, and grant permissions on it.

sudo mkdir -p /usr/java/packages/lib/ext
sudo cp /opt/mqm/web/clients/restConnector.jar /usr/java/packages/lib/ext/
sudo chmod 444 /usr/java/packages/lib/ext/restConnector.jar

2. Use a -classpath – not a -jar

In a .jar file there is a META-INF/MANIFEST.MF file which includes information on the entry point.

Manifest-Version: 1.0
Main-Class: src.Client

Instead of using the -jar option to point to the jar, you can use the -classpath to point to the jar and explicitly specify the entry point. For example

java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar  
src/Client
service:jmx:rest://localhost:9443/IBMJMXConnectorREST

java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar – use the classpath option, and specify the needed jar files. Client.jar is my program. /opt/mqm/web/clients/restConnector.jar is the Liberty provided jar.
src/Client – the “entry point” class to use
service:jmx:rest://localhost:9443/IBMJMXConnectorREST – the url to use.
note the absence of a -jar option.

In the manifest of the Client.jar file it had Main-Class: src.Client. This is for the the src/Client.java source file. This ties up with the src/Client.class as seen when you use the command jar -tvf Client.jar .

Accessing JMX data in Liberty server, securely.

I thought I would complete the work I did with using JMX in the mqweb server. It was another example of Hofstadter’s Law:

It always takes longer than you expect, even when you take into account Hofstadter’s Law.

I spent a lot of time looking for things on the web, expecting them to be obvious, only to find that the things do not behave as expected. I could not find them, because they were not there. For example I expected to be able to configure the JMX server to use my OS userid and password. I could have a file with userids and passwords, or lookup in LDAP, but not my normal userid.

Getting started

I found there are two ways of getting the JMX data from the mqweb server.

Use of the native JMX support
Using the Liberty REST API

I think the REST API is easier to set up and is more secure.

I’ll document a high level overview of the two approaches, and how to configure them

Overview of using the native JMX support.

To use this, you configure parameters in the jvm.options file, including a port solely for JMX.

You can use TLS certificates to set up a secure link between the client and the server.

You can decide if you want to logon with userid and password. If you do you can set up

A file with userids and passwords; and a file with userids and permitted access. The documentation talks about userids like monitorRole and controlRole. You have to put a process in place to periodically change these passwords.
Use and LDAP server to do userid validation and to get the access.
I could not find how to use your operating system userid and password for authentication.
I could not find how to use the DN as authorization.

If your certificate is valid (either because it is signed by a CA, or there is a copy of a self signed certificate in the trust store), this is good enough for the checking. You can enable userid and password checking, but this solution feels weak, as you have to do extra work to manage it properly; you do not have a single signon. Not all tools support using userid and password, for example I could not pass userid and password on the jconsole command.

Overview of Using the Liberty REST API

As with the MQ REST API you can issue an HTTP request and get data back. See here. For example

curl –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password https://localhost:9443/IBMJMXConnectorREST/mbeans/WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes=*

There is a small amount of configuration you need to do – less than with the native JMX support. The data comes back as JSON (as you might expect) and also includes a time stamp, which is very useful when post processing.

You define <administrator-role><user>..</user></administrator-role> in a similar way to setting up authorisation for mqconsole and mqrest. It takes the cn= value from your certificate as the userid, so you can give individual access.

“Securely” is a good laugh.

There are different levels of (in)security.

If you are using the native JMX support

You can have no passwords or access checks needed. The data is read only, and is not sensitive.
You can set up userid(s) and passwords in a file
You cannot use the operating system userid and password
You can use LDAP to check the userid and password, and get the role for that userid
You can use TLS, so anyone with a valid certificate can access the data
You can use TLS and use the userid and password in a file to determine access
You can use TLS and LDAP to get the role for that certificate

If you are using the WLP REST support

You can specify a userid and password
You can use a certificate, and the Common Name is used as the userid
You can specify in the configuration file, what access userids, or groups have

You can use TLS to protect your communications to and from the server.

Java leaks passwords

You need to be aware that your client machine may leak information. For example I ran a Java program to issue JMX requests from a script.

I could use the linux command ps -ef to display information about my request

ps -ef |grep JMX

gave me

colinpa+ 1871 1870 79 10:27 pts/2 00:00:01 java … -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12 -Djavax.net.ssl.keyStorePassword=password … -username monitorRole -password password …

This exposed the password to my keystore and password to my userid! I could not find a way of having all these java system parameters in a file.

I found export JAVA_TOOL_OPTIONS=”-D…” and this get picked up, but then java displays the variables as in Picked up JAVA_TOOL_OPTIONS: …

jconsole

Some programs have been designed to protect information for example jconsole you can put your system properties in a file

-J-Dcom.sun.management.config.file=ConfigFilePath

and so keep your parameters secure, but I could not get this to work.

Curl can be configured not to display parameters

With curl you have a command like

curl -n –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password

which gives away your password. If you do not specify it inline, you get prompted for it.

You can put your parameters in a config file, for example curl.config,

cacert ./cacert.pem 
cert ./colinpaice.pem:password 
key colinpaice.key.pem 
cookie cookie.jar.txt 
cookie-jar cookie.jar.txt 
url https://127.0.0.1:9443/ibmmq/rest/v1/login

and use

curl –config curl.config

Easy!

Protecting key files

It is important to protect the certificate file (with the important private key) so it is accessible by just the owner. The linux command ls -ltr colinpaice.p12 gives

-rw------- 1 colinpaice colinpaice 4146 Jan 31 17:56 colinpaice.p12

Of course anyone with super user authority has access to this file!

Getting useful information out of JMX data

The data coming from Liberty WebServer through the JMX interface provides some data, but it is not very useful, and it may become inaccurate over time.

I’ll cover

Getting a useful mean value
Getting a more accurate long term mean
Data gets more inaccurate over time
Standard deviation (this may only be of interest to a few people)

For example from JMX, the reported mean time for mqconsole transactions was 9.9 milliseconds – this is for all requests since the mqweb server was started. Over the previous minute the average measured time, for a 10 second period was 7, or 8 milliseconds, so well below the 9.9 reported.

This is because the mean time includes any initial start up time. The maximum transaction time, at the start of the run, was over 2 seconds. This will bias the mean.

You can process the data to extract some useful information, and I show below how to get out useful response time values.

You get the following data (and example values) from mqweb through the JMX interface.

ResponseTimeDetails/count (Long) = 20
ResponseTimeDetails/description (String) = Average Response Time for servlet
ResponseTimeDetails/maximumValue (Long) = 3060146565 – in nanoseconds (see below for the unit)
ResponseTimeDetails/mean (Double) = 4.336789965E8 – in nanoseconds
ResponseTimeDetails/minimumValue (Long) = 2474556 – in nanoseconds
ResponseTimeDetails/standardDeviation (Double) = 9.089057964078983E8 – in nanoseconds
ResponseTimeDetails/total (Double) = 8.67357993E9 – used in calculations
ResponseTimeDetails/unit (String) = ns – the unit ns = nanoseconds
ResponseTimeDetails/variance (Double) = 8.319076762335653 – used in calculations

Getting a useful mean value

To produce these numbers, the count of the response times and the sum of the transaction response times are accumulated within the Liberty Server. To calculate the mean value you calculate sum/count. This gives you the overall mean time. If you obtain the data periodically you can manipulate the data to provide some useful statistics.

Let the count and sum at time T1 be Count1, and Sum1, and at time T2 Count2, and Sum2.
You can now calculate (Sum2- Sum1)/(Count2 – Count1) to get the average for that period. For example the reported mean was 0.016 ms, but the calculated value gave 0.008 ms. You can also calculate (Count2 – Count1)/(T2-T1) to give a rate of requests per second. These are much more useful than the raw data. I suggest collecting the data every minute.

Getting a more accurate long term mean

The first rest request and console request take a long time because the java code has to be loaded in etc. In one test the duration of the first request was 50 times the duration of the other requests. A better “mean” value is to ignore the duration of the first request.

The improved mean is (JMX mean * JMX count – JMX Maximum value) /(JMX Count-1), or JMXMean – (JMXMaximum/JMXCount) .

Data gets more inaccurate over time

The total time is stored as a floating point double. As you add small numbers to big numbers, the small numbers may be ignored. Let me try to explain.

Consider a float which has precision of 3, so you could have 1.23 E2 = 1230. Add 1 to this, and you get 1231 which is 1.23 E2 with a precision of 3 – the number we started with.

The total time is in nanoseconds so 1 second is stored as 1.0 E9. With 100 of these a second, and 1 hour( 3600 seconds) for 100 hours is 360,000,000, or 3.6 E8 seconds. * 1.0 E9 nano seconds. = 3.6E17 nano seconds. The precision of most float numbers is 16, so with this 3.6 E17 we have lost the odd nanosecond. I do not think this is a big enough problem to worry about – unless you are running for years without restarting the server.

The variance uses the time**2 value. So with the maximum time above 599482097 nano seconds. Time **2 is 3.593787846×10¹⁷ and you are already losing data. I expect the variance will not be used very often, so I think this can be ignored.

If the times were in microseconds instead of nano seconds, this would not be a problem.

Getting a useful standard deviation (this may only be of interest to a few people)

The standard deviation gives a measure of the spread of the data, a small standard deviation means the data is in a narrow band, a larger standard deviation means it is spread over a wider band. Often 95% of the values are within plus or minus 3 * standard deviations from the mean, so anything outside this range would considered an outlier, or unusual statistic.

I set up some data, a mixture of 10 values 9, 10, 11, the standard deviation was 0.73. I changed one value to 20, and the standard deviation changed to 3.3, indicating a wide spread of values.

With a mixture of 100 values 9,10,11, the standard deviation was 0.71. I changed one value to 20, and the standard deviation changed to 1.2, so a much smaller value, most of the data was still around 10 – just one outside the range.

With a lot of data, the standard deviation converges on a value, and “unusual” numbers make little difference to the value. I think that the standard deviation over an extended period is not that useful, especially if you get periodic variations such as busy time, and overnight.

You calculate the standard deviation as the square root of the variance. The variance is (Sum of (values**2) – (mean ** 2)) /number of measurements.

With data

ResponseTimeDetails/count (Long) = 203
ResponseTimeDetails/mean (Double) = 6420785.187192118 – nanoseconds
ResponseTimeDetails/variance (Double) = 1.7113341125320868E15 – used in calculations

Variance = 1.7113341125320868E15 = ( (Sum of (values**2) – (6420785.187192118 ** 2)) / 203

So (Sum of (values**2)) = 3.474420513264337e+17

You can now do the sums as with the mean, above:

At time T1, the ssquares1 is the sum of (values**2) at time T2, the ssquares2 is the sum of (values**2).

You can now calculate ssquares2 – ssquares2, and used that to estimate the variance, and so the standard deviation of the data in the range T1 to T2, I’ll leave the details to the end user.

For the advanced user, you can use the mean for the interval – rather than the long term mean. Good luck.

What are the JMS connection factories on webLogic doing?

As part of my long running activity to find out what is causing 1 million MQCONNects a day from an Oracle webLogic web server, I have found out how to monitor what is going on inside the webLogic instance.

Most web servers support a Java Management eXtension (JMX) interface. You can use gui tools like jconsole to do an ad-hoc display of the management beans – but these are not practical for long term monitoring.

Ive listed the data from the JMX query, what the data means, and also documented how I got the data.

There are three components to an MDB

A listening task waits to be notified about messages arriving on the queue. As far as I can tell there is one thread doing this work. If you need more I think you need to create a second MDB.
Multiple MDB threads which get woken up by the listener task, get a message and invoke the MDB application OnMessage() method with the message. The data for this part has Type=EJBPoolRuntime.
The application part which processes the message – typically it connects to MQ and puts a reply, and disconnects. This application specified which connection pool to be used using ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF3”); The data for this part has Type=ConnectorConnectionPoolRuntime.

The multiple MDB threads.

This data came from the JMX record with com.bea:ServerRuntime=AdminServer2,MessageDrivenEJBRuntime=WMQ_IVT_MDB,Name=WMQ_IVT_MDB,ApplicationRuntime=MDB3,Type=EJBPoolRuntime,EJBComponentRuntime=MDB3

Where

ServerRuntime=AdminServer2. This is where the application runs
MessageDrivenEJBRuntime=WMQ_IVT_MDB comes from display-name WMQ_IVT_MDB in the ejb-jar.xml file.,Name=WMQ_IVT_MDB,
ApplicationRuntime=MDB3, This is the name of the deployed .jar file.
Type=EJBPoolRuntime, this is the type for the MDB threads
EJBComponentRuntime=MDB3, this is the name of the deployed .jar file.

From experiementing with MDBs and adjusting parameters my picture of how the EJB thread pool works is as follows

There is a general free pool for threads.
There is a pool for the EJB.
When a message arrives the listener thread gets a thread from the EJB pool, and executes the OnMessage() method on the thread.
If there are not available threads,
1. it waits for a free thread.
2. if, over a period of seconds, most of the requests have had to wait for a thread, then start a new thread
  1. it gets a thread from the general pool, if none available it creates a new thread.
  2. it executes the MDB application ejbCreate() method on the thread just obtained.
  3. It executes the OnMessage() method on the thread.
  4. Puts the thread into the EJB pool as a free thread.
3. In the ejbCreate() method, I had it write to the job log. I could see threads started, after
Periodically the EJB pool is cleaned up, for threads which have been idle for a specific time interval
1. execute ejbRemove() on the thread
2. return the thread to the general free pool
Periodically the general pool is cleaned up
1. Threads which have not been used for a while are disconnected from the queue manager, and deleted.

The data came from the JMX with the defintion

com.bea:ServerRuntime=AdminServer2, Name=WMQ_IVT_MDB, ApplicationRuntime=MDB3, Type=MessageDrivenEJBRuntime, EJBComponentRuntime=MDB3

See here for the documentation.

AccessTotalCount = 28180, There were this many messages processed, and so this many OnMessage() methods executed
BeansInUseCount = 2. Deprecated
BeansInUseCurrentCount = 1 One thread is currently active.
DestroyedTotalCount = 0.
IdleBeansCount = 4 There are 4 free thread available in the EJB pool
MissTotalCount = 19. For 19 (out of the 28180) requests, they failed to get a free thread, and so a new thread was obtained.
Name = WMQ_IVT_MDB
PooledBeansCurrentCount = 3. The numer of free threads inthe pool.
TimeoutTotalCount = 0. Number of requests timed out waiting for a thread from the free pool.
Type = EJBPoolRuntime. You can use Type=EJBPoolRuntime in the JMX query string.
WaiterCurrentCount = 0. The number of requests waiting for a thread in the pool. This value should always be zero.
WaiterTotalCount = 0. The total number of threads that have had to wait in the pool. This value should always be zero, if it is non zero you need to tune your connection pool.

The data for the connection factory.

See here for some documentation.

ActiveConnectionsCurrentCount (Inte
ger) = 9. How many connections were in use when the JMX request was issued.
ActiveConnectionsHighCount (Integer) = 10. The highest number of connections in use since the connection pool was started or reset
AverageActiveUsage (Integer) = 0. This value is always zero.
CapacityIncrement (Integer) = 1. This is the value the connection pool is configured with
CloseCount (Long) = 5682. How many connection.close() requests have been issued. This will be less than or equal to ConnectionsMatchedTotalCount below.
ConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ConnectionFactoryImpl. This is the name of the connection class being used
ConnectionFactoryName (Null). I could not find where to change this, or if it has any value
ConnectionIdleProfileCount (Integer) = 0. This was always zero.
ConnectionIdleProfiles (Null). This was always zero.
ConnectionLeakProfileCount (Integer) = 0. This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
ConnectionLeakProfiles (Null). This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
ConnectionProfilingEnabled (Boolean) = false. I could not see how to change this, or what value it adds
Connections (ObjectName[]) = [Ljavax.management.ObjectName;@299a06ac] This will be an internal object within Java.
ConnectionsCreatedTotalCount (Integer) = 21. This is the count of instances started. You could get connection started, connection freed, connection started, connection freed. This would could 2 connections started.
ConnectionsDestroyedByErrorTotalCount (Integer) = 0.
ConnectionsDestroyedByShrinkingTotalCount (Integer) = 11. After a time period, idle-timeout-seconds in weblogic-ejb-jar.xml, connections that have not processed messages for this time period are released back to the general pool.
ConnectionsDestroyedTotalCount (Integer) = 11. This seems to be the same as the previous item
ConnectionsMatchedTotalCount (Integer) = 5670. This many requests for a connection with matching userid etc were got from the pool
ConnectionsRejectedTotalCount (Integer) = 0. I dont know what this is
ConnectorEisType (String) = Java Message Service
CurrentCapacity (Long) = 10. This is the size of the pool
EISResourceId (String) = type=<eis>, application=colin, module=colin, eis=Java
Message Service, destinationId=CF2
FreeConnectionsCurrentCount (Integer) = 1. There is currently one free connection in the pool
FreeConnectionsHighCount (Integer) = 10. This is the highest number of free connections in the pool – when there was no requests for the connection pool.
FreePoolSizeHighWaterMark (Long) = 10. I dont know the difference between this and the previous item.
FreePoolSizeLowWaterMark (Long) = 0. This is the lowest number of free connections – it is the gap between in use and the maximum pool size.
HealthState (Null). I do not think connection pools for MDBs support this
HighestNumWaiters (Long) = 0. This value was always zero. Even when I reduced the size of the connection pool, and specified 50 threads could wait, this value was still 0
InitialCapacity (Integer) = 1. This is the specified value. This is used to set a lower limit of connections in the pool. Setting the initial capacity to 5 means there will always be at least 5 connections in the pool, and the pool will not be shrunk below this value
JNDIName (String) = CF2. This is what is specified in the connection pool definition. This label is used in the application ConnectionFactory cf = (ConnectionFactory)ctx.lookup(“CF2”);
Key (String) = CF2. This seems to the be same as the JNDI name.
LastShrinkTime (Long) = 1565334529469. This is the number of milliseconds POSIX time (since Jan 1st 1970). To format it use time_t now = t/1000; strftime(buff, 20, “%H:%M:%S”, localtime(&now));
LogFileName (Null). You can specify logging for this adapter. Deployments -> resource adapter -> configuration -> Outbound Connection Pools, javax.jsm.ConnectionFactory, select the connection pool, Logging tab. When this was active I got nothing logged in it.
LogFileStreamOpened (Boolean) = false
LoggingEnabled (Boolean) = true
LogRuntime (Null)
ManagedConnectionFactoryClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl. This is the name of the resouce adapter class.
MaxCapacity (Integer) = 20. This is the maximum size of the pool.
MaxIdleTime (Integer) = 0. This was always zero for me.
MCFClassName (String) = com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl. This is the ManagedConnectionFactoryClassName. Same as above.
Name (String) = CF2. Another connection pool identifier
NumberDetectedIdle (Integer) = 0. This was always zero.
NumberDetectedLeaks (Integer) = 0. This was always zero. I believe this is to do with connections not being returned to a pool, perhaps connection.close() was not issued.
NumUnavailableCurrentCount (Integer) = 9. This feels like the number threads waiting while the connection pool creates a connection in the connection pool.
NumUnavailableHighCount (Integer) = 10. This feels like the number threads waiting while the connection pool creates a connection in the connection pool.
NumWaiters (Long) = 0. This was always zero – even through I restricited the number of connections in the pool. I got a java exception, unable to get a connection, rather than have the thread wait.
NumWaitersCurrentCount (Integer) = 0. See above.
Parent (ObjectName) = com.bea:ServerRuntime=AdminServer2,Name=colin, ApplicationRuntime=colin, Type=ConnectorComponentRuntime
PoolName (String) = CF2. Yet another field with the name of the connection pool/
PoolSizeHighWaterMark (Long) = 10. This is the highest number of connections used in the pool
PoolSizeLowWaterMark (Long) = 0. This is the lowest number of connecitons used in pool
ProxyOn (Boolean) = false. I could not find what this means
RecycledTotal (Integer) = 0. This was always zero for me.
ResourceAdapterLinkRefName (Null). You can specify this field name in your MDB definition.
RuntimeTransactionSupport (String) = NoTransaction. This defines if your connections are part of a unit of work or not. NoTransaction means out of syncpoint. You may want to specify. XATransaction.
ShrinkCountDownTime (Integer) = 340. This is how long before a scan of the pool to remove any threads in the pool which have done no work for the specified interval.
ShrinkingEnabled (Boolean) = true
ShrinkPeriodMinutes (Integer) = 15. You can specify the interval on the resource adapter definition (Note: you specify it in seconds – it gets reported in minutes)
State (String) = Running. This connection pool is started.
Testable (Boolean) = false. You can defined some resources as “testable”
TransactionSupport (String) = NoTransaction. This is the same as RuntimeTransactionSupport above.
Type (String) = ConnectorConnectionPoolRuntime. This defines the object. You can use the Type=ConnectorConnectionPoolRuntime in the JMX query.

From this I can see there is a pool called CF2 which has maximum of 10 connections. The maximum connections used was 10, the lowest was 0.

There were Connections Matched Total Count = 5670 requests for a connection from the pool.

The pool has shrunk more than once as it has Connections Destroyed By Shrinking Total Count = 11 connections.

Using this data, you can now plot usage over time and see if you need to increase ( or decrease) the size of the pool, or the parameters to tune when the pool is shrunk.

I do not know enough about JMX to tell if the “high” and “low” value are reset on each query, or if you can use JMX to reset them periodically. These high and low value may have little value, if they are since the webLogic instance started (6 months ago).

The data fields are mentioned here.

How I got the data

There is a python package called JMXQuery which has a .jar file which allows you to query information in a JMX server. The output is in json format so you can use your favourite tools (python) to quickly convert this to other format, such as .csv .

The command I used was

java -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u webLogic
-p passw0rd -q “com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”

which breaks down as follows

java – invoke java
-jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” – this jar file
-url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi – this is the url of my webLogic server
-json – output it in json format
-u webLogic -p passw0rd -q – userid and password
“com.bea:ServerRuntime=AdminServer2,Name=CF2,ApplicationRuntime=colin, Type=ConnectorConnectionPoolRuntime,ConnectorComponentRuntime=colin”
- com.bea is the bean type
- The admin server was called AdminServer2
- The connection factory was CF2
- The resource adapter is installed under “Deployments” as colin
- Type=ConnectorConnectionPoolRuntime is the type of bean

I then used |jq . |grep -v mBean > bb to convert the one line json to one field per line, dropped the mBean value, and put it to a file. The output was like

[
 {
   "attribute": "Connections",
   "attributeType": "ObjectName[]",
   "value": "[Ljavax.management.ObjectName;@299a06ac"
 },
 {
   "attribute": "FreeConnectionsCurrentCount",
   "attributeType": "Integer",
   "value": 4
},
etc

You can put generics in for example

java -jar “/usr/local/lib/python3.6/dist-packages/jmxquery/JMXQuery-0.1.8.jar” -url service:jmx:rmi:///jndi/rmi://127.0.0.1:8091/jmxrmi -json -u readonly -p read0nly -q “com.bea:ApplicationRuntime=*,ConnectorComponentRuntime=*,Name=”CP*”,ServerRuntime=*,Type=ConnectorConnectionPoolRuntime” e=*,Type=ConnectorConnectionPoolRuntime” > aa

Where this uses a userid set up as a monitor id, “*” has been specified for many values, and only give objects beginning with “CP”. Note blanks have meaning. “,,,=*, Name=…” looks for an object with blank,N,a,m,e,

Data when not using a resource adapter

The above information was for a resource adapter. When an EJB 2 MDB is deployed (non resource adapter)

from –q “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3,
MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,
Name=WMQ_IVT_MDB_JMSQ1,ServerRuntime=AdminServer2,Type=EJBPoolRuntime”

Where the weblogic-jar-xml has

<weblogic-ejb-jar> <ejb-name>WMQ_IVT_MDB</ejb-name>
<destination-jndi-name>JMSQ1</destination-jndi-name>

The data was

BeansInUseCount (Integer) = 0
PooledBeansCurrentCount (Integer) = 10
IdleBeansCount (Integer) = 10
BeansInUseCurrentCount (Integer) = 0
DestroyedTotalCount (Long) = 0
WaiterCurrentCount (Integer) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
MissTotalCount (Long) = 7
AccessTotalCount (Long) = 28
Type (String) = EJBPoolRuntime
TimeoutTotalCount (Long) = 0
WaiterTotalCount (Long) = 0

With

-q “com.bea:ApplicationRuntime=MDB3,EJBComponentRuntime=MDB3, MessageDrivenEJBRuntime=WMQ_IVT_MDB_JMSQ1,Name=WMQ_IVT_MDB_JMSQ1,
ServerRuntime=AdminServer2,Type=EJBTransactionRuntime“

the output was

TransactionsRolledBackTotalCount (Long) = 0
TransactionsTimedOutTotalCount (Long) = 0
Name (String) = WMQ_IVT_MDB_JMSQ1
Type (String) = EJBTransactionRuntime
TransactionsCommittedTotalCount (Long) = 0