Using Activity Trace to show a picture of which queues and queue managers your application used.

I used the midrange MQ activity trace to show what my simple application, putting a request to a cluster server queue and getting the reply, was doing.  As a proof of concept (200 lines of Python), I  produced the following

This output is a .png format.   You can create it as an HTML image, and have the nodes and links as clickable html links.

Ive ignored any SYSTEM.* queues, so the SYSTEM.CLUSTER.TRANSMIT.QUEUE does not appear.

The red arrows show the “high level” flow between queue managers at the “architectural”, hand waving level.

  • The application oemput on QMA did a put to a clustered queue CSERVER, there is an instance of the queue on QMB and QMC.   There is a red line from QMA.oemput to the queue CSERVER on QMB and QMC
  • The server programs, server running on QMB and QMC put the reply message to queue CP0000 on queue manager A

The blue arrows show puts to the application specified queue name – even though this may map to the S.C.T.Q.  There are two blue lines from QMA.oemput because one message went to QMC.CSERVER, and another went to QMB.CSERVER

The yellow lines show the path the message took between queue managers.  The message was put by QMA.oemput to queue CSERVER; under the covers this was put to the SCTQ.  From the accounting trace record this shows the remote queue manager and queue name:  the the yellow line links them.

The black line is getting from the local queue

The green line is the get from the underlying queue.  So if I had a reply queue called CP0000, with a QAlias of QAREPLY. If the application does a get from QAREPLY,  There would be a black line to CP0000, and a green line to QAREPLY

How did I get this?

I used the midrange activity trace.

On QMA I had in mqat.ini

applicationTrace:
ApplClass=USER # Application type
ApplName=oemp* # Application name (may be wildcarded)
Trace=ON # Activity trace switch for application
ActivityInterval=30 # Time interval between trace messages
ActivityCount=10 # Number of operations between trace msgs
TraceLevel=MEDIUM  # Amount of data traced for each operation
TraceMessageData=0 # Amount of message data traced

I turned on the activity trace using the runmqsc command

ALTER QMGR ACTVTRC(ON)

I ran some work load, and turned the trace off few seconds later.

I processed the trace data into a json file using

/opt/mqm/samp/bin/amqsevt -m QMA -q SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE -w 1 -o json > aa.json

I captured the trace on QMB, then on QMC, so I had three files aa.json, bb.json, cc.json.  Although I captured these at different times, I could have collected them all at the same time.

jq is a “sed” like processor for processing json data.   I used it to process these json files and produce one output file which the Python json support can handle.

jq . --slurp aa.json bb.json cc.json  > all.json

The two small python files are zipped here. AT.

I used ATJson.py python script to process the all.json file and extract out key data in the following format:

server,COLIN,127.0.0.1,QMC,Put1,CP0000,SYSTEM.CLUSTER.TRANSMIT.QUEUE,QMC,CP0000,QMA, 400

  • server, the name of the application program
  • COLIN, the channel name, or “Local”
  • 127.0.0.1, the IP address, or “Local”
  • QMC, on this queue manager
  • Put1, the verb
  • CP0000, the name of the object used by the application
  • SYSTEM.CLUSTER.TRANSMIT.QUEUE, the queue actually used, under the covers
  • QMC, which queue manager is the SCTQ on
  • CP0000, the destination (remote) queue name
  • QMA, the destination queue manager
  • 400 the number of times this was used, so 400 puts to this queue.

I had another python program Process.py which took this table and used python graphviz to draw the graph of the contents.  This produces a file with DOT  (graph descriptor language)parameters, and used one of the many programs to draw the chart.

This shows you what can be done, it is not a turn-key solution, but I am willing to spend a bit of time making it easier to use, so you can automate it.  If so please send me your Activity Trace data, and I’ll see what I can do.

My mid-range accounting is missing data

Like many of the questions I get asked, it started off as a simple question.

Over a day, what people and machines are using this queue?”  Simple I said.  Just use the accounting data.

Problem 1

I had a program which puts a message to a clustered queue on a remote queue manager.  The message flows to the server and a reply is sent back.  I thought I would check it works.  No it didn’t.

The accounting data has my program, but it recorded a message put to the SYSTEM.CLUSTER.TRANSMIT.QUEUE, and a message got from my reply queue, so I cannot tell what queue was used, but I can tell how many bytes were processed

Problem 2

My application doing just a publish did not show up in the accounting data.

I used the samples amqspub, and amqssub to publish, and subscribe respectively.

The accounting data had a record for the amqssub, getting from a queue, but there was no record for the publishing application.

As Morag kindly pointed out, you can tell how many publishes you did from the MQI accounting data, but not to which topic.

Ive started looking at the activity trace, but this looks like a heavyweight solution to the simple question.

Problem 3

Someone else pointed out that if you want to know the channel name, and IP address of the application using a queue, you have to do some extra work.

  • Collect the MQI accounting
  • Process queue accounting and MQI accounting, and use the common field connectionId to merge the data

Using amqsevt to display the accounting data in json format you get

“eventData” : {
“queueMgrName” : “QMC”,
“startDate” : “2020-04-17”,
“startTime” : “09.43.23”,
“endDate” : “2020-04-17”,
“endTime” : “10.18.58”,
“commandLevel” : 913,
“connectionId” : “414D5143514D432020202020202020201361995E0A24A923”,
“sequenceNumber” : 1,
“applName” : “server”,
“processId” : 29103,
“threadId” : 11,
“userIdentifier” : “colinpaice”,
   “connDate” : “2020-04-17”,
   “connTime” : “09.12.16”,
   “connectionName” : “127.0.0.1”,
   “channelName” : “COLIN”,

Where the bold text is for the data only in the MQI accounting information.

It looks like the MQI accounting record is produced before the Queue accounting record.

You will need to save the connectionId key with the connection specific information.  When the queue accounting record is processed, use the connectionId to look up the information and append it to the queue record.  If many queues are being used, you may get multiple queue accounting records – so you cannot simply delete the connection key and connection specific information.

 

mqweb – what’s the difference between MQWebAdmin* and MQWebUser roles?

The MQWebAdmin, MQWebAdminRO and MQWebUser roles can all be used for the admin REST API. That’s why MQWebUser is not called MQWebMessaging – it’s not just for messaging. The difference between them is the user ID that’s checked by the qmgr for operations performed via the REST API.
  • Operations performed by users in the MQWebAdmin* roles take place under the context of the mqweb server user ID.
  • Operation performed by users in the MQWebUser role take place under the context of the user logged into the REST API.

Thanks to Gwydion for this enlightenment.

mqweb – who did what to what, when, and how long did it take?

You can provide an audit trail of the http requests coming into your server.  This is described in the base liberty document, and works in mqweb.

Within the httpEndoint tag you can add

<httpEndpoint host=”${httpHost}” httpPort=”${httpPort}” httpsPort=”${httpsPort}” id=”defaultHttpEndpoint”>
<httpOptions removeServerHeader=”false”/>
<accessLogging enabled=”true” filePath=”${server.output.dir}/logs/http_access.log”
logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U X:%{X}W r:”%r” s:%s’
maxFileSize=20

maxFiles=0
/>
</httpEndpoint>

See here for information on the accessLogging,  here for the syntax of the <accessLogging…> and here for the logFormat format options.

From the logFormat page

%a
Remote IP address
%A
Local IP address
%b
Response size in bytes excluding headers
%B
Response size in bytes excluding headers. 0 is printed instead of – if no value is found.
%{CookieName}C
The request cookie specified within the brackets, or if the brackets are not included, prints all of the request cookies.
%D
The elapsed time of the request – millisecond accuracy, microsecond precision
%h
Remote host
%{HeaderName}i
HeaderName header value from the request
%m
Request method
%{HeaderName}o
HeaderName header value from the response
%q
Output the query string with any password escaped
%r
First line of the request
%{R}W
Service time of the request from the moment the request is received until the first set of bytes of the response is sent – millisecond accuracy, microsecond precision
%s
Status code of the response
%t
NCSA format of the start time of the request
%(t)W
The current time when the message to the access log is queued to be logged in normal NCSA format
%u
Remote user according to the WebSphere Application Server specific $WSRU header
%U
URL Path, not including the query string
%{X}W
Cross Component Tracing (XCT) Context ID

So

logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U  r:”%r” s:%s’

gave me

  • a:127.0.0.1 remote IP address
  • A:127.0.0.1 local host
  • D:225960 duration in millisecond
  • h:127.0.0.1 remote host
  • HeaderHost:127.0.0.1:9443 the Host header
  • HeaderOrigin:- the Origin header (missing in this case)
  • m:GET request method
  • R:186930 Service time of the request from the moment the request is received until the first set of bytes of the response is sent
  • t:[27/Feb/2020:17:04:27 +0000] NCSA format of the start time of the request
  • u:colinpaice remote user
  • U:/ibmmq/console url path
  • r:”GET /ibmmq/console HTTP/1.1″ First line of the request
  • s:302 Status code of the response

There are other log formatting options available, I picked those I thought were most useful.

Note when using the MQConsole from a browser, the interface is chatty. I had 20 request to refresh one window.

Other ways of formatting the data

  • I separated each field with a ‘,’ and could read it into a spread sheet.
  • You could configure your log format string to produce the output in JSON format, to make it easier to post process.

What does mqRestCorsMaxAgeInSeconds in mqweb mean?

Ive blogged about CORS, and how this allows you to list sites that are permitted to use scripts to send request to the mqweb server.

I struggled with understanding what value mqRestCorsMaxAgeInSeconds has, as it did not behave as expected (this was due to a bug).

If you have a CORS transaction there is an OPTIONS request, followed by the actual DELETE/GET/POST requests.
The OPTIONS request checks that the request is coming from an authorised list of URLs, and that the parameters are valid.  The OPTIONS information can be cached.

If the check is successful then the real request can be issued.  If the requests occur in a short time, then you can have OPTIONS, DELETE, DELETE, DELETE, using the cached values.  If there is a “long” time interval between the requests you may get OPTIONS, DELETE, gap, OPTIONS, DELETE.

The OPTIONS information can be cached for up to the minimum of the mqweb mqRestCorsMaxAgeInSeconds  time, and the browser time.

For Chrome the maximum time interval is 600 seconds.  If no time is specified in the OPTIONS response, then 5 seconds is used.

There is a bug in the Liberty base code which sends down the header Access-Control-Allow-Max-Age: …, when the browser is expecting Access-Control-Max-Age.   Because of this, the default time of 5 seconds is used in Chrome.

This should not have a major impact.  For those applications using scripts to send multiple REST API request, there will be more frequent OPTIONS requests – every 5 seconds instead of up to 600 seconds.  These extra flowes are invisible to the scripts and the end user.

What value should I use?

Chrome has a maximum of 600 seconds, with a default of 5 seconds.

Firefox has a maximum of 24 hours (86400 seconds).

Setting it to 600 seconds sounds reasonable to me.

Making changes to mqRestCorsMaxAgeInSeconds

If you change mqRestCorsMaxAgeInSeconds you have to restart the mqweb server.

I do not get caching!

When researching this topic I found every GET request had an OPTIONS request, rather than the OPTIONS, GET, GET.   A quick search on the internet showed me the Chrome option ( F1 -> Settings and preferences) “Disable Cache ( while DevTOOLS is open)” was enabled. I deselected this, and I got the caching.

How do I tell if my puts and gets are in or out of syncpoint?

This question came up from someone who is new to MQ.  it is good to find people people who are new to MQ.

As usual there is an easy answer, but some cases you need to think about.

There is no command you can use to show this information.  You can find it it out using the accounting information.

Typically an application does

  • MQCONN
  • MQOPEN queue
  • MQPUT|MQGET
  • Possibly  a commit
  • MQCLOSE
  • MQDISC

If the accounting data shows that there was a COMMIT request, and the commit took about 1 millisecond, then there is a good chance that the put or get was in syncpoint.

If gets more interesting if you have

  • MQPUT of messages out of syncpoint (for example and audit trail – we got to this point.  Out of syncpoint so it exists even after a rollback)
  • MQPUT within syncpoint
  • MQCOMMIT

If the accounting information shows one commit, you cannot tell if it was one or two messages within syncpoint.

If the message is persistent and out of syncpoint, then under the covers it does “MQPUT|MQGET – and commit this requests”.  On z/OS a put or get typically take 10 microseconds, a log I/O takes about  1 ms, so if you see an MQPUT or MQGET with a long response time (1 millisecond), then it might have been a put or get of a message outside of syncpoint.
On z/OS you can also get the logging time for a put or get, so if you see the logging time for a put or get, this is a persistent message out of syncpoint.

A non persistent message does not write to the log on disk, all updates are kept in memory, so you cannot tell from the duration of the request if it was in or out of syncpoint.    Best practice is that non persistent messages are usually processed out of syncpoint.

If you see persistent messages are put or got, and the commit time is microseconds,  then the commit did not do disk IO, so there was no work to commit, and so the put or get was out of syncpoint.

And lastly, you may see no commit request in the accounting data.   This is bad practice, as it is assuming that the MQDISC will cause a commit or roll back, so it would happen invisibly.  An application should explicitly issue a commit if messages are within syncpoint.

As I said – pretty simple – but with some complexities.

 

 

 

Error connecting to JMX endpoint: Unsupported protocol: rest

I got this error message when I was trying to use JMX into the WLP web server, when using the restConnector-2.0 interface in Liberty.

The documentation was not that helpful.  Oneproblem was I suffered from the curse of cut and paste, and used a Windows environment variable %JAVA_HOME%  instead of using the Linux $JAVA_HOME.  Another problem was caused documentation saying add the jar to the class path, and then the class path was ignored.

True documentation.

When you use the  -jar option, the JAR file is the source of all user classes, and other user class path settings are ignored.  See here.

Unhelpful documentation

Configuring secure JMX connection to Liberty   says

jconsole -J-Djava.class.path=...;%WLP_HOME%/clients/restConnector.jar

This was for Windows – on Linux it would be $WLP_HOME – except on my system $WLP_HOME was not set.

jconsole

Some of the jar files are in $JAVA_HOME, you can use the environment variable, or specify it yourself.  Note %JAVA_HOME% is windows, so be careful when you use cut and paste.

  • jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar
  • jconsole -J-Djava.class.path=/usr/lib/jvm/java-8-openjdk-amd64/lib/jconsole.jar:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar

If the window displays “Secure connection failed” restart jconsole and use the -debug option.   For me this gave  “java.io.IOException jmx.remote.credentials not provided. Set to a String[2] {user,password}”.  I entered my userid and password, and this connected successfully.

I could not connect using my .jar file.

I was using JMXQuery to extract the data.  I was getting the Error connecting to JMX endpoint: Unsupported protocol: rest message.
Adding the jar file to my class path did not help, as the class path is ignored when using java the -jar parameter.

How to fix it

There are two ways of fixing this.

  1. Put the required jar file in the extensions path, not the class path
  2. Use the java -classpath…  instead of specifying java -jar

1. You need to have the jar for in the extensions path, not in the class path.

See How classes are found  in the Java documentation.  It says

  • Bootstrap classes – Classes that comprise the Java platform, including the classes in rt.jar and several other important jar files.
  • Extension classes – Classes that use the Java Extension mechanism. These are bundled as .jar files located in the extensions directory.
  • User classes – Classes defined by developers and third parties that do not take advantage of the extension mechanism. You identify the location of these classes using the -classpath option on the command line (the preferred method) or by using the CLASSPATH environment variable.

There is a system property java.ext.dirs  which gives the location of the Extension classes. On my system this was

/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext

  • The /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext: is java dependent. You should not put your files in this directory
  • /usr/java/packages/lib/ext This is for “user” extensions.

The directory /usr/java/packages/lib/ext did not exist on my system, so I had to do, create it, copy the web connection jar file to it, and grant permissions on it.

  • sudo mkdir -p /usr/java/packages/lib/ext
  • sudo cp /opt/mqm/web/clients/restConnector.jar /usr/java/packages/lib/ext/
  • sudo chmod 444 /usr/java/packages/lib/ext/restConnector.jar

2. Use a -classpath – not a -jar

In a .jar file there is a META-INF/MANIFEST.MF file which includes information on the entry point.

Manifest-Version: 1.0
Main-Class: src.Client

Instead of using the -jar option to point to the jar, you can use the -classpath to point to the jar and explicitly specify the entry point.  For example

java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar  
src/Client
service:jmx:rest://localhost:9443/IBMJMXConnectorREST
  • java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar –  use the classpath option, and specify the needed jar files.  Client.jar is my program.  /opt/mqm/web/clients/restConnector.jar  is the Liberty provided jar.
  • src/Client – the “entry point” class to use
  • service:jmx:rest://localhost:9443/IBMJMXConnectorREST – the url to use.
  • note the absence of a -jar option.

In the manifest of the Client.jar file it had Main-Class: src.Client.   This is for the the src/Client.java source file.   This ties up with the src/Client.class as seen when you use the command jar -tvf Client.jar .

Use the Liberty REST API to access the JMX data in Liberty

Rather than set up traditional JMX, where you specify the JMX port etc, you can use the REST support provided in LIberty to access the JMX data.   The rest support is easier to set up.

The Liberty documentation recommends that you do not have the native JMX support (configured in jvm.options), and the Liberty REST support for JMX configured at the same time.

The REST request to get the statistics worked and was easy to use.   I could not get the “traditional JMX interface”, such as jconsole to work with the REST interface.  See below.

Configure the server:  mqwebuser.xml

In mqwebuser.xml  add the support

<featureManager>
  <feature>restConnector-2.0</feature>
</featureManager>

Set up authorisation with

<administrator-role>
   <user>colinpaice</user>
   <group>MQADMIN</group>
</administrator-role>

<reader-role>
  <user>John</user>
</reader-role>

As the mqconsole and rest statistics are read only, then it may be better to set up every user as a reader-role.

As with the MQ support, it will use the userid if specified, or the Common Name from the digital certificate.

Using Curl and the rest API

I used

curl --cacert ./cacert.pem --cert-type P12 
--cert colinpaice.p12:password
-url https : //localhost:9443/IBMJMXConnectorREST/mbeans/...

Where … was  WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes/ResponseTimeDetails  . This gives the JMX statistics for the mq.console.

The data comes back as JSON (as you might expect) for example

"name": "ResponseTimeDetails",
  "value": {
    "value": {
      "count": "99",
      "description": "Average Response Time for servlet",
      "maximumValue": "3183755861",
      "mean": "1.116053166969697E8",
      "minimumValue": "1777114",
      "reading": {
        "count": "99",
        "maximumValue": "3183755861",
        "mean": "1.116053166969697E8",
        "minimumValue": "1777114",
        "standardDeviation": "4.360373971884932E8",
        "timestamp": "1580819294060",
        "total": "1.1048926353E10",
        "unit": "ns",
        "variance": "1.90128611746915776E17"
      },
      "standardDeviation": "3.218674128849494E8",
      "total": "1.1048926353E10",
      "unit": "ns",
      "variance": "1.63102693370991648E17"
    ...

As well as the data I have covered before, you also get the time stamp value.  This is the value in milliseconds from a well known time.

I used the python to convert the timestamp (1580978634610) to a date time

import datetime
s = 1580978634610 / 1000.0
print(datetime.datetime.fromtimestamp(s).strftime('%Y-%m-%d %H:%M:%S.%f'))

to give  2020-02-04 12:28:14.060000.

Which URL to use for traditional JMX?

The IBM documentation says the url to access the JMX data using “traditional JMX” is in file /var/mqm/web/installations/Installation1/servers/mqweb/log/state/com.ibm.ws.jmx.rest.address.  For me this was service:jmx:rest://localhost:9443/IBMJMXConnectorREST .

Client set up: Using the “traditional JMX interface” did not work for me

The Configuring secure JMX connection to Liberty page says you can use this url in jconsole and other tools.  I could not get this to work.   I got  messages like Error connecting to JMX endpoint: Unsupported protocol: rest.  This page gives a lot of information on JMX, and towards the end of the second page it says Error connecting to JMX endpoint: Unsupported protocol: xxxx is likely to be a problem with the class path.  I used  -verbose=class, and did not see the jar file being loaded.

What else can you do with the REST interface?

Show what is available

You may get data like

WebSphere%3Aname%3Dcom.ibm.mq.console.javax.ws.rs.core.Application %2C type%3DServletStats

The punctuation has been “escaped” so you need to change

  • %3A- to  :
  • %3D to :
  • %2C to ,

and the string becomes the familiar

WebSphere: name=com.ibm.mq.console.javax.ws.rs.core.Application, type=ServletStats

Setting up Liberty(as used in mqweb) to use native JMX

Setting up the server side is well documented in the Oracle Monitoring and Management Using JMX Technology documentation.  Using it from a client is not so well documented.

Server set up

The  Liberty jvm.options file needs parameters.  Note the port=9010 is used  by clients accessing the data.

To provide insecure access from only the local machine

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9010 
-Dcom.sun.management.jmxremote.local.only=true 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false

To provide securer access using TLS

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true

# the following statements point to the same key store as
# used by mqweb server.   This could be different.
-Djavax.net.ssl.keyStoreType=PKCS12
-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/mqweb.p12
-Djavax.net.ssl.keyStorePassword=password
# the following statements point to the same trust store as
# used by mqweb server.   This could be different.
# if you used self signed certificates you could have a keystore
# just for the JMX users
-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks
-Djavax.net.ssl.trustStorePassword=zpassword
-Djavax.net.ssl.trustStoreType=JKS

# The following defines the userid and password file
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.password.file=/home/colinpaice/ssl/ssl2/jmxremote.password

# The following defines the access a userid can have
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.access.file=/home/colinpaice/ssl/ssl2/jmxremote.access

jmxremote.password has

# specify actual password instead of the text password
monitorRole password
controlRole password

jmxremote.access has

# The "monitorRole" role has readonly access.
# The "controlRole" role has readwrite access.
monitorRole readonly
controlRole readwrite

Client set up

jconsole

You cannot pass a  userid and password when the jconsole command, so you have to disable authentication in the jvm.options file

-Dcom.sun.management.jmxremote.authenticate=false

The parameters for jconsole have  -J on them, as in -J-D…. .  jconsole removes the -J and uses the rest of the parameters when invoking the JVM.

I could not get jconsole to recognize a config file using the -J-Dcom.sun.management.config.file = /path/to/jmxremote.properties , so I wrote a bash script to make it easier to change parameters.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file"
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-J-Djavax.net.debug=ssl:handshake"
jconsole -J$ssl1 -J$ssl2 -J$ssl3 -J$ssl4 -J$ssl5 -J$ssl6 -J$ssl7 $ssl8 127.0.0.1:9010

The option “-J-Djavax.net.debug=ssl:handshake” gives a verb verbose trace of the ssl flows for the handshake.

The option -J-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file enables the jconsole logging.  I did not find the output very useful.

There is information the logger in general here,  and on the file logger, here.

The logging.file had

Logging.properties

handlers= java.util.logging.FileHandler
// , java.util.logging.ConsoleHandler2

java.util.logging.FileHandler.pattern=/home/colinpaice/JMXQuery/java/log.%g.file
java.util.logging.FileHandler.limit=50000
java.util.logging.FileHandler.count=2
java.util.logging.FileHandler.level=ALL
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

// .level = INFO
// logger.level = FINEST
.level = FINEST
// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose...
javax.level= FINER
javax.management.level = FINER
javax.management.remote.*     = FINER 
javax.management.remote.level = FINER
javax.management.remote.misc.level  = FINER
javax.management.remote.rmi.level= FINER

Using jmxquery

I used a bash shell script to run the command, as it was easier to manage, and I could not find a way of having the java system properties in a file.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ibmsys1.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"les
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file" 
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-Djavax.net.debug=ssl:handshake"
jar="-jar JMXQuery.jar"
user="-username monitorRole -password password"
url="-url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi"
parms=" -q   WebSphere:*  -count 2 -every 2"
java $ssl1 $ssl2 $ssl3 $ssl4 $ssl5 $ssl6 $ssl7 $ssl8 $jar $url $user $parms

 

Accessing JMX data in Liberty server, securely.

I thought  I would complete the work I did with using JMX in the mqweb server.   It was another example of Hofstadter’s Law:

It always takes longer than you expect, even when you take into account Hofstadter’s Law.

I spent a lot of time looking for things on the web, expecting them to be obvious, only to find that the things do not behave as expected.  I could not find them, because they were not there.  For example I expected to be able to configure the JMX server to use my OS userid and password.  I could have a file with userids and passwords, or lookup in LDAP, but not my normal userid.

Getting started

I found there are two ways of getting the JMX data from the mqweb server.

  1. Use of the native JMX support
  2. Using the Liberty REST API

I think the REST API is easier to set up and is more secure.

I’ll document a high level overview of the two approaches, and how to configure them

Overview of using the native JMX support.

To use this, you configure parameters in the jvm.options file, including a port solely for JMX.

You can use TLS certificates to set up a secure link between the client and the server.

You can decide if you want to logon with userid and password.  If you do you can set up

  1. A file with userids and passwords; and a file with userids and permitted access.   The documentation talks about userids like monitorRole and controlRole.   You have to put a process in place to periodically change these passwords.
  2. Use and LDAP server to do userid validation and to get the access.
  3. I could not find how to use your operating system userid and password for authentication.
  4. I could not find how to use the DN as authorization.

If your certificate is valid (either because it is signed by a CA, or there is a copy of a self signed certificate in the trust store), this is good enough for the checking.   You can enable userid and password checking, but this solution feels weak, as you have to do extra work to manage it properly;  you do not have a single signon.  Not all tools support using userid and password, for example I could not pass userid and password on the jconsole command.

Overview of Using the Liberty REST API

As with the MQ REST API you can issue an HTTP request and get data back.  See here.  For example

curl –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password https://localhost:9443/IBMJMXConnectorREST/mbeans/WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes=*

There is a small amount of configuration you need to do – less than with the native JMX support.  The data comes back as JSON (as you might expect) and also includes a time stamp, which is very useful when post processing.

You define <administrator-role><user>..</user></administrator-role>  in a similar way to setting up authorisation for mqconsole and mqrest.  It takes the cn= value from your certificate as the userid, so you can give individual access.

“Securely” is a good laugh.

There are different levels of (in)security.

If you are using the native JMX support

  • You can have no passwords or access checks needed.  The data is read only, and is not sensitive.
  • You can set up userid(s) and passwords in a file
  • You cannot use the operating system userid and password
  • You can use LDAP to check the userid and password, and get the role for that userid
  • You can use TLS, so anyone with a valid certificate can access the data
  • You can use TLS and use the userid and password in a file to determine access
  • You can use TLS and LDAP to get the role for that certificate

If you are using the WLP REST support

  • You can specify a userid and password
  • You can use a certificate, and the Common Name is used as the userid
  • You can specify in the configuration file, what access userids, or groups have

You can use TLS to protect your communications to and from the server.

Java leaks passwords

You need to be aware that your client machine may leak information.  For example I ran a  Java program to issue JMX requests from a script.

I could use the linux command ps -ef to display information about my request

ps -ef |grep JMX

gave me

colinpa+ 1871 1870 79 10:27 pts/2 00:00:01 java …  -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12 -Djavax.net.ssl.keyStorePassword=password …  -username monitorRole -password password

This exposed the password to my keystore and password to my userid!  I could not find a way of having all these java system parameters in a file.

I found export JAVA_TOOL_OPTIONS=”-D…”  and this get picked up, but then java displays the variables as in Picked up JAVA_TOOL_OPTIONS: …

jconsole

Some programs have been designed to protect information for example jconsole you can put your system properties in a file

-J-Dcom.sun.management.config.file=ConfigFilePath

and so keep your parameters secure, but I could not get this to work.

Curl can be configured not to display parameters

With curl you have a command like

curl -n –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password

which gives away your password.  If you do not specify it inline, you get prompted for it.

You can put your parameters in a config file, for example curl.config,

cacert ./cacert.pem 
cert ./colinpaice.pem:password 
key colinpaice.key.pem 
cookie cookie.jar.txt 
cookie-jar cookie.jar.txt 
url https://127.0.0.1:9443/ibmmq/rest/v1/login

and use

curl –config curl.config

Easy!

Protecting key files

It is important to protect the certificate file (with the important private key) so it is accessible by just the owner.  The linux command  ls -ltr colinpaice.p12 gives

-rw------- 1 colinpaice colinpaice 4146 Jan 31 17:56 colinpaice.p12

Of course anyone with super user authority has access to this file!