mqweb – who did what to what, when, and how long did it take?

You can provide an audit trail of the http requests coming into your server.  This is described in the base liberty document, and works in mqweb.

Within the httpEndoint tag you can add

<httpEndpoint host=”${httpHost}” httpPort=”${httpPort}” httpsPort=”${httpsPort}” id=”defaultHttpEndpoint”>
<httpOptions removeServerHeader=”false”/>
<accessLogging enabled=”true” filePath=”${server.output.dir}/logs/http_access.log”
logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U X:%{X}W r:”%r” s:%s’
maxFileSize=20

maxFiles=0
/>
</httpEndpoint>

See here for information on the accessLogging,  here for the syntax of the <accessLogging…> and here for the logFormat format options.

From the logFormat page

%a
Remote IP address
%A
Local IP address
%b
Response size in bytes excluding headers
%B
Response size in bytes excluding headers. 0 is printed instead of – if no value is found.
%{CookieName}C
The request cookie specified within the brackets, or if the brackets are not included, prints all of the request cookies.
%D
The elapsed time of the request – millisecond accuracy, microsecond precision
%h
Remote host
%{HeaderName}i
HeaderName header value from the request
%m
Request method
%{HeaderName}o
HeaderName header value from the response
%q
Output the query string with any password escaped
%r
First line of the request
%{R}W
Service time of the request from the moment the request is received until the first set of bytes of the response is sent – millisecond accuracy, microsecond precision
%s
Status code of the response
%t
NCSA format of the start time of the request
%(t)W
The current time when the message to the access log is queued to be logged in normal NCSA format
%u
Remote user according to the WebSphere Application Server specific $WSRU header
%U
URL Path, not including the query string
%{X}W
Cross Component Tracing (XCT) Context ID

So

logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U  r:”%r” s:%s’

gave me

  • a:127.0.0.1 remote IP address
  • A:127.0.0.1 local host
  • D:225960 duration in millisecond
  • h:127.0.0.1 remote host
  • HeaderHost:127.0.0.1:9443 the Host header
  • HeaderOrigin:- the Origin header (missing in this case)
  • m:GET request method
  • R:186930 Service time of the request from the moment the request is received until the first set of bytes of the response is sent
  • t:[27/Feb/2020:17:04:27 +0000] NCSA format of the start time of the request
  • u:colinpaice remote user
  • U:/ibmmq/console url path
  • r:”GET /ibmmq/console HTTP/1.1″ First line of the request
  • s:302 Status code of the response

There are other log formatting options available, I picked those I thought were most useful.

Note when using the MQConsole from a browser, the interface is chatty. I had 20 request to refresh one window.

Other ways of formatting the data

  • I separated each field with a ‘,’ and could read it into a spread sheet.
  • You could configure your log format string to produce the output in JSON format, to make it easier to post process.

Is mqweb secure enough? Depends on what you mean by secure enough.

When doing research into the MQ Console and REST interface I was some times surprised at how “insecure” the mqweb was, but with a bit more thought it seemed more reasonable.

One of the questions I had was, is authentication with certificates better than using userid and password?  See here.

What do I mean by secure?

You can buy a safe with a combination lock to secure your valuables.    It will not be 100% secure.   It will have a statement like “This safe will provide protection from fire for up to 1 hour”.  The number of combinations is limited, and so it will take someone some hours to try them all.   They may be lucky and guess it first go!  An insecure safe would have one combination wheel.  The more wheels, the longer it should take to find the combination.

Digital certificates are considered a good way of providing “secure” communication.  However if you are careless with your private information, then other people have easy access to your secrets.  The private information for some encryption types is based on multiplying two very large prime numbers together.  The theory being it will take you a long time to factorize  the product .  If I was a government, I would take the “book of prime numbers” and go through each number, and multiply it be each other number in the book, and generate a reverse dictionary of factor1 * factor 2 -> public key.    You need a lot of computing power as the numbers are very large, but computing power is cheap, and you only need to do it once.  Take the public key, and look up the factors.  Of course it is not that simple, you might need a dictionary for each encryption type, as well as a lot of CPU power.  These certificates give some security, but are not totally secure.

You can have locks on your house, but if you leave the doors unlocked – it is easy to break in.  People often lock the front door, but leave the back door or a window open.

Security is down to how long it will take someone to access the information.  All components have to be secure; you cannot be “half secure”.

What do I want to secure?

This is where it starts to get murky.  Some of the HTTP protocol protection depends on “state change”, so changing a field in a database is a state change, creating an object is a state change, but browsing a record is not a state change.

With the MQ REST API an HTTP GET maps to a browse;  an HTTP POST is a change or put;  and an HTTP DELETE is a destructive get.  With the standard HTTP protection POST and DELETE have more protection than GET.

I consider being able to browse a queue which has financial transaction information, as insecure.  You can obtain bank account details, which could be used in a different sort of attack.   In mqweb, the protection for browse is weaker than for POST and DELETE.

Surprise number 1 – which certificate should the browser use?

My first surprise when using digital certificates for identification was how the browser decides which certificate to use.

I said here

If client authentication is used, the certificates are read from the server’s trust store. The DN of the self signed certificates and the DN of the CA certificates are extracted and sent down to the browser.

The browser takes this list and checks through the “user” certificates in its key store, for those self signed certificates with a matching DN, or those certificates which have been signed by the CA.

The list of matching certificates is displayed for the user to select from.

From then on, communication to that https://url:port will use that certificate.  If you create a new tab within the browser instance, and work with the same site, the selected certificate will be used – without prompting.

You can only change the certificate by restarting the browser and selecting a different certificate.  You cannot logoff.

Surprise number 2 – it is easy to browse the queue with no additional security.

In an HTML page,  I used the html link <a href=”https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message “> direct link https ></a> .

If I had already accessed the server then this displayed the content of the next message on the queue, with no further security checks.

If I opened another tab within the browser, displayed the page and clicked on the link, I also got the next message content displayed.

I used a file on my laptop with the link in it – and it displayed the message content.  It was easy to create a web page in an editor and so display the content of the queue.

The CSRF checks are not applicable – as these only occur when the request is a change of state, POST, DELETE etc.

Using Curl, and changing the GET request to a DELETE I received (as expected)

{“error”: [{
  “action”: “Resubmit the request with the correct CSRF header.”,
  “completionCode”: 0,
  “explanation”: “The REST API request cannot be completed because the CSRF header was omitted.”,
  “message”: “MQWB0100E: The CSRF header ‘ibm-mq-rest-csrf-token’ was omitted from the request.”,
  “msgId”: “MQWB0100E”,
  “reasonCode”: 0,
  “type”: “rest”
 }]}’

This is not a Cross Origin request – as it is in one page – so the CORS checks are not performed.   Putting the backend request into a java script did cause the CORS checking as expected.

Turning off client Certificate Authentication helped

I changed the <ssl…  clientAuthenticationSupported=”true”> to “false” to use userid and password.

With my <a href=”https://localhost:9445/ib…  > ” > direct link https ></a>  I got

{"error": [{
  "action": "Provide credentials using a client certificate, LTPA security token, or username 
   and password via HTTP basic authentication header. 
   On z/OS, if the mqweb server has been configured for SAF authentication, 
   check the messages.log file for messages indicating that SAF authentication is not available. 
   Start the Liberty angel process if it is not already running. 
   You might need to restart the mqweb server for any changes to take effect.",
  "completionCode": 0,
  "explanation": "The REST API request cannot be completed because credentials were omitted from the request. 
   On z/OS, if the mqweb server has been configured for SAF authentication, 
   this can be caused by the Liberty angel process not being active.",
  "message": "MQWB0104E: The REST API request to 
  'https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message' is not authenticated.",
  "msgId": "MQWB0104E",
  "reasonCode": 0,
  "type": "rest"
}]}

I tried putting my userid and password in the URL as in
<a href=”https://colinpaice:mypassword@localhost:9445/ib…> “> direct link https ></a> . This gave me the same message.  In particular the message says Provide credentials using …  or username and password via HTTP basic authentication header.  So it needs credentials in  a header not inline.

As raw html cannot insert HTTP headers, it means this is more secure.   To insert headers you need a script, and then the CORS processing will capture it.

I used the logon url https://127.0.0.1:9445/ibmmq/console/login.html this prompted me for userid and password, then displayed the console.

So is using userid and password the right thing to use?

Yes and No.

  1. Yes – you know when you are accessing the back end, but then it logs you off, and you have to logon again after ltpaExpiration minutes.
  2. With long running displays, such as the monitors in the operations room – you do not want to be logging on every hour.  In this case certificates are the right answer.

See is it better to use certificate authentication or userid and password? for information.

 

Making it more secure.

I have shown that if people are authorised to process messages on a queue as part of their normal job, they could easily write a web page to browse the queue.

If the queue was access as part of a client  program accessing the queue over a channel, the end user could write or obtain a program to process the queue – but this would be harder than just writing a web page.    This is like having a combination safe with one wheel, and having multiple wheels – you need a lot more effort.

You cannot secure mqweb for message access to a subset of queues – it is all or nothing.

I would disable the end users access to message data – and just use the rest API for administration.

This means in the mqwebuser.xml the <security-role name=”MQWebUser”> … </security-role> has no users who are allowed to process sensitive queues.  You need to prevent them using curl and other tools, so

<enterpriseApplication id=”com.ibm.mq.rest”> … and <enterpriseApplication id=”com.ibm.mq.console”>…. should have no users in <security-role name=”MQWebUser”>… </security-role>

Ban the special subject.

I would also explicitly specify the groups of users – and not use the <special-subject type=”ALL_AUTHENTICATED_USERS”/>.

Best practice says not to specify user names in the security roles, but to put the user names in groups and specify the groups.   It make it much easier to manage.  For example the manager in charge of the Payroll team, can administer the PayrollGroup, and the MQ Admin team does not need to edit the mqwebuser.xml file.

mqweb- is it better to use certificate authentication or userid and password?

Like may questions the answer is it depends.

You can set up your mqweb environment

  1. Certificates are required – you cannot use userid and passwords
  2. If a valid certificate is found use that, else prompt for userid and password
  3. Do not use certificates – just use userid and passwords.

This is described in the Liberty documentation.

It says for the <ssl…> tag

  1. If you specify clientAuthentication="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake does not succeed.
  2. If you specify clientAuthenticationSupported="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.
  3. If you do not specify either clientAuthentication or clientAuthenticationSupported, or you specify clientAuthentication="false" or clientAuthenticationSupported="false", the server does not request that a client send a certificate during the handshake.

No matter how you logon, the web server creates a LTPA* cookie with encrypted information about the logon.   The mqwebuser.xml attribute ltpaExpiration says how long this cookie is valid for.  If the time is exceeded you have to logon again, and it generates a new cookie.

Logging on

Certificate authorisation

If you are using certificates, the server sends down a list of valid self signed certificates or CA certificates, and the browser compares this list with the certificates in its store.   If the certificate in the browser’s store is acceptable, it is added to a list

The list is displayed, and the end user selects a certificate.  For subsequent requests with the same  http://url:port combination, in the same or a different tab within the browser window, the same certificate is used, and the browser does not prompt.

If the ltpa cookie expires then the same certificate is used, invisibly to the end user.  A new cookie is generate and sent down.

If there are no valid certificates in the list, then depending clientAuthentication, it may prompt for a userid and password.

You cannot logoff from the page as there is no option to do so.

Userid and password.

When userid and password are passed up in a header (not in the URL as some web servers accept) the ltpa token is returned.  When the ltpa token expires, the user is prompted to re-logon.   They need to enter the userid and password again.  You can also logoff from the web page.

What are the implications of certificates and userid with password

Certificate logon

  • The web browser can logon, and although the ltpa token expires, it is automatically logged on again.   This is useful for a “big screen monitor” in the operations room, where you want the display active all day.
  • Someone can create a file with some HTML in it to browse a queue  (if authorised).  So if they are authorised to the queue for their normal job, they could easily create a web page to display the queue contents.  This could be the end user, or an evil person saying “click this link”.

userid logon

  • If the user is using the mqconsole web pages, then periodically they will get logged off – and they will have to logon again.
  • If they try to access a queue from an HTML page, they will be asked for userid and password.  Depending on what the end user is doing, this may give them notification than there is some unusual activity.  If they are hacking they will enter the userid and password anyway.
  • The person may have access to the same information without using the mqweb console interface, so there is no change to the threat level.

What do I recommend?

Tricky.

  • I think long running monitors need to use certificates
  • End users should logon with userid and password, with an expiry period of about an hour or less.
  • You may want to have a Certificate  Authority just for the mqweb browsers, to stop your corporate certificates from being used to access the mqweb server.

Because it is easy to create an html page and browse a queue, I would allow administration over mqrest, and not allow processing messages over REST.

As I said in another posting

  • The admin API authorisation is managed through <security-role name=”MQWebAdmin”> and <security-role name=”MQWebAdminRO”> sections in the mqwebuser.xml file.
  • The messaging API authorisation is managed through <security-role name=”MQWebUser”> sections.

To stop messaging over the rest interface, have no entries in the <security-role name=”MQWebUser”> section.  This applies to both certificates, and userid and password.

(Personally I would have called<security-role name=”MQWebUser”> as <security-role name=”MQWebMessaging”>  to make the roles clearer. 

Update from Gwydion…

The MQWebAdmin, MQWebAdminRO and MQWebUser roles can all be used for the admin REST API. That’s why MQWebUser is not called MQWebMessaging – it’s not just for messaging. The difference between them is the user ID that’s checked by the qmgr for operations performed via the REST API.
  • Operations performed by users in the MQWebAdmin* roles take place under the context of the mqweb server user ID.
  • Operation performed by users in the MQWebUser role take place under the context of the user logged into the REST API.
To stop messaging over the REST API completely, the mqRestMessagingEnabled should be set to false in the mqweb server configuration.

Can evil websites get to your mqweb – using java script to get to the backend server

With HTML and scripting people would write scripts and get you to execute them, and so access your personal information, and steal your money.  Over time security has been improved to make this harder, and now you have to explicitly say which web sites can run scripts which use the mqweb interface to access MQ to put and get messages.

One way of protecting the access is using Cross Origin Resource Sharing, or CORS.  I explained the basics of CORS here.  I struggled getting it to work with a web browser.

  • browsers have been improved and what worked last year may no longer work now, and the documentation does not reflect the newer browsers.
  • Chrome carefully changes your hand crafted HTTP headers, so what is sent up may not what you expected.

I’ll go through three examples, and then show how it gets more difficult, and what you can do to identify problems.

Note: If you use a web page from a file:// then the origin is ‘null’, and this will fail any checks in the backend, as the checks compare the passed origin to the list of acceptable urls.

I used Dev Tools in Chrome (Alt+Ctrl+i) to display information including the headers flowing.

Simple HTML

With the following within your web page,

<a href=”https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/…/message
> direct link ></a>

It issues the REST request, returns the data from the queue, and displays it.

From the headers we see

Simple HTML: Request headers

The important request headers were

And no origin header.

Simple HTML: Response headers

The response headers have

  • HTTP/1.1 200 OK
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent
  • Content-Length: 1024
  • Set-Cookie: LtpaToken2_1 ….

There are no Allow….  headers, so this indicates the response is not a valid cross origin response.   The request came from one page, so there was no cross origin request.

You can see the MQ attributes returned, ibm-mq*.

Invoking request from java script.

My HTML page had

<html
<head>
<script src=”http://localhost:8884/src.js ” text=”text/javascript”></script >
</head>
<body>
<button onclick=”src()””>Get message</button>
</body>

I had the src() script downloaded from http://localhost:8884/src.js, and inline within the html file.  Both worked.

function src()
{
  fetch("https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message",
  { 
     method :'DELETE',
     headers: {
                'Origin': 'https://localhost:888499'
                ,'Access-Control-Request-Headers' : 'Content-Type' 
                ,'ibm-mq-rest-csrf-token' : '99'
              } 
   }
  ) // end of fetch
   .then((response) => response.text() )
   .then(x => {document.write("OUTPUT:"+x); } 
  ) 
  .catch(error =>  {console.log("Error from src:" + error);});
}

When the button was pressed, the script was executed,  there was an OPTIONS request (and response), and a DELETE request (and response).   It returned a message and displayed it.

In more detail, the flows were:

JavaScript OPTIONS request

The OPTIONS request had headers

This is doing a preflight check, saying it intends to issue a DELETE request from Origin  http://localhost:8889, the url of the web page.

JavaScript OPTIONS response headers

The OPTIONS response headers were the same as before but with additional ones

  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8889
  • Access-Control-Allow-Max-Age: 91
  • Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE
  • Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

This means, the script is allowed to issue a request from http://localhost:8889 using a request method in the list {GET, POST, PATCH, PUT, DELETE } and the valid headers that can be used are in the list of headers.

The Access-Control-Allow-Max-Age: 91 came from my mqwebuser.xml file, <variable name=”mqRestCorsMaxAgeInSeconds” value=”91″/>.

After this there was a DELETE request to get the message.

JavaScript DELETE request headers

JavaScript DELETE response headers

The response included the CORS headers, and included the headers from the non CORS situation

  • HTTP/1.1 200 OK
  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8889
  • Access-Control-Expose-Headers: Content-Language, Content-Length, Content-Type, Location, ibm-mq-qmgrs, ibm-mq-md-messageId, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers
  • Content-Type: text/plain; charset=utf-8
  • Content-Language: en-GB
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent
  • Content-Length: 1024
  • Set-Cookie: LtpaToken2_….

Because the Access-* headers are present, this is a CORS response.

The message content was displayed in a browser window.

Link to another page

I set up a link <a href=”http://localhost:8884/page.html”/>localhost 8884</a>  to execute a page on another web server.  When this executed, it issued the java script request as before.  The Origin was Origin: http://localhost:8884 – so the page where the script executed.

What happens if CORS is not set up?

If the http://localhost:8889 is not in the list in the mqwebuser.xml file,

No data was  displayed.   The Chrome browser Developer tools ( Alt+Ctrl+i) displays a message

Access to fetch at ‘ https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

The OPTIONS Request header has, as before

  • HTTP/1.1 200 OK
  • Host: localhost:9445
  • Access-Control-Request-Method: DELETE
  • Origin: http://localhost:8889
  • Access-Control-Request-Headers: ibm-mq-rest-csrf-token

but the OPTIONS Response header has no Access-* headers.

No DELETE request was issued.

The HTTP/1.1 200 OK is the same for all cases – it means the request was successful.

Trying to be clever

I read the documentation on the web, and much of it was very helpful, but some of it is no longer true.  It was hard to get it working, because every things has to be right for it to work.

Unsupported header.

I added an extra header, which is a valid CORS thing to do – but the back end has to support it.  With hindsight it makes no sense to add headers that will be ignored by the server.

headers: {
     'Origin': 'https://localhost:888499'
     , 'Access-Control-Request-Headers' : 'Content-Type' 
     , 'colin':'TRUE' 
     ,'ibm-mq-rest-csrf-token' : '99'
}

This sent up a header with

Access-Control-Request-Headers: colin,ibm-mq-rest-csrf-token

Header “colin” is not in the list of accepted headers, header ibm-mq-rest-csrf-token is in the list:

Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

The Developer tools message was the same as before,

Access to fetch at ‘https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

I removed the unsupported header and it worked.

Other headers do not work

The request header ‘Access-Control-Allow-Method’ : ‘DELETE’ is a valid header.

When this was used, the request headers included

  • Access-Control-Request-Headers: access-control-allow-method,ibm-mq-rest-csrf-token
  • Access-Control-Request-Method: DELETE

As before  Access-Control-Request-Method is not in the list of Access-Control-Allow-Headers, so the request fails.

The Access-Control-Request-Method: DELETE is not needed, as the method: DELETE defines what will be used.

Using Access-Control-Request-Headers to add more headers does not work, if the header is not in the list of valid parameters.

The true origin is used

I tried using the header ‘Origin’: ‘https://localhost:888499 ‘ – as it did with my curl – and this was ignored.  The true Origin from the web page was used (I am glad to say, otherwise it would make the whole protection scheme worthless)

Some options ignored

I found  the “fetch” options credentials:, redirect: , and cache:, were all ignored by Chrome.

Some  cookies  were used.

The LtpaToken2_… was sent up and down

Problems in mqwebuser.xml file

I misconfigured the configuration file with <variable name=”mqRestCorsMaxAgeInSeconds” value=”AA91″/>

This gave me the familair message

Access to fetch at ‘https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

So if you get this message it is worth checking the configuration file for problems.

Can I trace it?

I used <variable name=”traceSpec” value=”*=audit:CorsService=finest”/> in the mqwebuser.xml file to provide information about the CORS processing, and what was coming in, and what was being checked.

Can evil websites get to your mqweb – understanding CORS

In the beginning was the html, and the html was good;  then we had html and scripts  which could only do things on the page, which was also good; then we had scripts which could reach out to other websites – and that’s where the problems began.  It was easy for evil developers to get you to click on an innocent looking page, which had a script which jumped into a different tab of your browser where you had your banking window open ,  or  to executed a script ; and steal all your money.

The browsers were improved to stop evil scripts from accessing a server, and then they were improved again so the server could say “stuff coming from this list of web sites is OK, I trust them.   One implementation is called CORS.  There is a good description here.   It says

Cross-Origin Resource Sharing (CORS) is a mechanism that uses additional HTTP headers to tell browsers to give a web application running at one origin, access to selected resources from a different origin. A web application executes a cross-origin HTTP request when it requests a resource that has a different origin (domain, protocol, or port) from its own.

I do not trust things that “just work”, I like to see evidence of it.   This lack of trust comes from working with a group of young testers who came to IBM Hursley to test their code.   All of their tests ran cleanly – and thought they had a few days spare to go to London sightseeing.  I stopped the server they were meant to be using (without telling them) and the tests carried on running successfully!   It turned out they had been testing their code with a dummy program acting as the server.    They removed this code, reran their tests  and most of them failed – and had to stay an extra week.

I also remember changing a config file, and being surprised when my changed worked first time.   After a cup of tea (an invaluable thinking aid) I put some spelling mistakes in the file ; and it carried on running.  Why? I was using the wrong config file.

I played with CORS and wanted to get things to fail, as well as to work.   This was a good choice, as I had many failures.

I’ll document how I got curl to work and demonstrate CORS , and document how I got a web browser to work – a real challenge

mqweb implements CORS, so you can configure mqweb to give a list of websites which may access your server.

The documentation  is not very clear.  It says

where allowedOrigins specifies the origin that you want to allow cross-origin requests from. You can use an asterisk surrounded by double quotation marks, “*”, to allow all cross-origin requests. You can enter more than one origin in a comma-separated list, surrounded by double quotation marks. To allow no cross-origin requests, enter empty quotation marks as the value for allowedOrigins.

My observations,

  • You cannot use generics, so http://127.0.0.1:* is the same as “*” – or allow all cross-origin requests
  • You must specify {scheme:address:port} so http or https,  the url with // at the front, and the specific port number
  • The match is an string equality test, so the case, spacing and values must be the same

How does an HTTP request work?

When you click on a web page, data is sent to the back end server.   The following data is exchanged

  • the request
  • request headers
  • your data going to the server
  • response headers
  • response data – such as the content of web page.

In more detail…

Request

Request Headers

  • accept: for example  text/html, application/xml
  • accept-languages: en-GB
  • dnt:  1  this is “do not track me”
  • user-agent:  Chromium
  • cookie: bcookie=”….”

Response headers

  • status: 200
  • status: 200 OK
  • set-cookie ….
  • server: nginx
  • ….

Response data

This might be the web page to be displayed.  It can include script, images etc.

What is the origin of the page?

If your web page, invoked a script, for example from clicking a button, an “Origin” header is added, for example Origin: http://localhost:8884 which is the address of the web server hosting the page.   The backend server checks to see if this header is present, and looks up site in the authorised list.

If the Origin is acceptable, it sends down additional response headers (CORS Headers), so the browser (or your program) knows and can use the web site.

As part of the handshake, the browser can send up an “OPTIONS” request (instead of a get/delete etc), with a header saying the browser will be doing a GET/DELETE etc from this origin.    If there is a positive response, where the additional CORS headers are send back then the get/delete is allowed.   If the CORS headers are not present in the response, then the request will not be permitted.   This is called a preflight check – just like having your boarding pass checked at the gate before you get on the plane.

Does it work?

If there is no Origin header in the request, the backend server thinks it is all same domain, and does no CORS checks.

This CORS support is really aimed at web browsers, as the web browsers will automatically add headers for you.  If you are using curl or other tools to create your own request, you specify exactly the headers you want, so if you omit the Origins header, the server will not check.

My mqwebuser.xml file had <variable name=”mqRestCorsAllowedOrigins” value=”https://9999.0.0.1:19442,http://localhost:8884”/&gt;

So origins for https://9999.0.0.1:19442 and http://localhost:8884 are permitted.

I used a configuration file for curl (because command parameters did not work passing the headers) and had in curl.parms

cacert ./cacert.pem
cert ./colinpaice.pem:password
key ./colinpaice.key.pem
cookie cookie.jar.txt
cookie-jar cookie.jar.txt
request OPTIONS
header “Access-Control-Request-Method: DELETE”
header “Access-Control-Request-Headers: Content-Type”
header “Origin: https://9999.0.0.1:19442
header “ibm-mq-rest-csrf-token : COLINCSRF”
include

I used the command

curl –verbose –config curl.parms –url https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message

The headers were displayed, and I had

OPTIONS /ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message HTTP/1.1
> Host: localhost:9445
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: LtpaToken2_….
> Access-Control-Request-Method: DELETE
> Access-Control-Request-Headers: Content-Type
> Origin: https://9999.0.0.1:19442
> ibm-mq-rest-csrf-token : COLINCSRF

The response headers were

< HTTP/1.1 200 OK
< X-Powered-By: Servlet/3.1
< X-XSS-Protection: 1;mode=block
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src ‘none’; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’; connect-src ‘self’; img-src ‘self’; style-src ‘self’ ‘unsafe-inline’; font-src ‘self’
< Cache-Control: no-cache, no-store, must-revalidate
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: https://9999.0.0.1:19442
< Access-Control-Allow-Max-Age: 90
< Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE
< Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

The response header Access-Control-Allow-Origin: https://9999.0.0.1:19442 shows that requests with origin https://9999.0.0.1:19442 is acceptable.

When I used a different “Origin”,   I did not get any Access-Control-Allow-* headers.  So from the absence,  I could tell the request was not support from the different origin.

The Access-Control-Allow-Headers is a list of header names which can be sent, so the header ibm-mq-rest-csrf-token : COLINCSRF is valid, but “COLIN:value”  is not valid, because ibm-mq-rest-csrf-token  is in the Access-Control-Allow-Headers list, and COLIN is not in the list.

The Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE says my application can use any of the methods in the list.

Using a web browser.

If you use HTML in a local file, then the “Origin” is null, and so does not match any elements in the authorised list.  I had to set up my own web server – which was easy to do using Python.

Using the web page at the bottom of this posting, I pointed my web browser at the web server.   It displayed a button.  I pressed it, and it invoked a script. Using the developer mode ( Al+Ctrl +i) in Chrome could see network flows etc.

The request headers had

  • Host: localhost:9445 This is where my webserver is hosted
  • ibm-mq-rest-csrf-token : 99  I specified this header value
  • Origin: http://localhost:8884  this overrode the value I had specified in the headers.

The response headers included

  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8884
  • Access-Control-Expose-Headers: Content-Language, Content-Length, Content-Type, Location, ibm-mq-qmgrs, ibm-mq-md-messageId, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent

We can see from

  • the Access-Control-* headers we know this has validated for http://localhost:8884
  • the Access-Control-Expose-Headers we can see what headers will be accepted
  • ibm-mq-md-persistence:  persistence. For the returned messages, it was a persistent message

The web page used

<HTML>
<HEAD>
<TITLE>Call a  mqweb rest API </TITLE>
<script>
  function local()
  {
    fetch("https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message" 
      {
         method :'DELETE',
         headers: {
                    'Origin': 'https://localhost:888499'
                    , 'Access-Control-Request-Headers' : 'Content-Type'
                    ,'ibm-mq-rest-csrf-token' : '99'
                  }
     }
    )
    .then((response) => response.text() )
    .then(x => {  document.write("OUTPUT:"+x);    } )
    .catch(error => { console.log("Booo:" + error);});
  }
</script> /head> <body> <button onclick="local()"">press me</button> </body> </html>

 

For information about “fetch(…,…)” see here.

For information about the “.then(…) ” see here.

Use the Liberty REST API to access the JMX data in Liberty

Rather than set up traditional JMX, where you specify the JMX port etc, you can use the REST support provided in LIberty to access the JMX data.   The rest support is easier to set up.

The Liberty documentation recommends that you do not have the native JMX support (configured in jvm.options), and the Liberty REST support for JMX configured at the same time.

The REST request to get the statistics worked and was easy to use.   I could not get the “traditional JMX interface”, such as jconsole to work with the REST interface.  See below.

Configure the server:  mqwebuser.xml

In mqwebuser.xml  add the support

<featureManager>
  <feature>restConnector-2.0</feature>
</featureManager>

Set up authorisation with

<administrator-role>
   <user>colinpaice</user>
   <group>MQADMIN</group>
</administrator-role>

<reader-role>
  <user>John</user>
</reader-role>

As the mqconsole and rest statistics are read only, then it may be better to set up every user as a reader-role.

As with the MQ support, it will use the userid if specified, or the Common Name from the digital certificate.

Using Curl and the rest API

I used

curl --cacert ./cacert.pem --cert-type P12 
--cert colinpaice.p12:password
-url https : //localhost:9443/IBMJMXConnectorREST/mbeans/...

Where … was  WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes/ResponseTimeDetails  . This gives the JMX statistics for the mq.console.

The data comes back as JSON (as you might expect) for example

"name": "ResponseTimeDetails",
  "value": {
    "value": {
      "count": "99",
      "description": "Average Response Time for servlet",
      "maximumValue": "3183755861",
      "mean": "1.116053166969697E8",
      "minimumValue": "1777114",
      "reading": {
        "count": "99",
        "maximumValue": "3183755861",
        "mean": "1.116053166969697E8",
        "minimumValue": "1777114",
        "standardDeviation": "4.360373971884932E8",
        "timestamp": "1580819294060",
        "total": "1.1048926353E10",
        "unit": "ns",
        "variance": "1.90128611746915776E17"
      },
      "standardDeviation": "3.218674128849494E8",
      "total": "1.1048926353E10",
      "unit": "ns",
      "variance": "1.63102693370991648E17"
    ...

As well as the data I have covered before, you also get the time stamp value.  This is the value in milliseconds from a well known time.

I used the python to convert the timestamp (1580978634610) to a date time

import datetime
s = 1580978634610 / 1000.0
print(datetime.datetime.fromtimestamp(s).strftime('%Y-%m-%d %H:%M:%S.%f'))

to give  2020-02-04 12:28:14.060000.

Which URL to use for traditional JMX?

The IBM documentation says the url to access the JMX data using “traditional JMX” is in file /var/mqm/web/installations/Installation1/servers/mqweb/log/state/com.ibm.ws.jmx.rest.address.  For me this was service:jmx:rest://localhost:9443/IBMJMXConnectorREST .

Client set up: Using the “traditional JMX interface” did not work for me

The Configuring secure JMX connection to Liberty page says you can use this url in jconsole and other tools.  I could not get this to work.   I got  messages like Error connecting to JMX endpoint: Unsupported protocol: rest.  This page gives a lot of information on JMX, and towards the end of the second page it says Error connecting to JMX endpoint: Unsupported protocol: xxxx is likely to be a problem with the class path.  I used  -verbose=class, and did not see the jar file being loaded.

What else can you do with the REST interface?

Show what is available

You may get data like

WebSphere%3Aname%3Dcom.ibm.mq.console.javax.ws.rs.core.Application %2C type%3DServletStats

The punctuation has been “escaped” so you need to change

  • %3A- to  :
  • %3D to :
  • %2C to ,

and the string becomes the familiar

WebSphere: name=com.ibm.mq.console.javax.ws.rs.core.Application, type=ServletStats

Setting up Liberty(as used in mqweb) to use native JMX

Setting up the server side is well documented in the Oracle Monitoring and Management Using JMX Technology documentation.  Using it from a client is not so well documented.

Server set up

The  Liberty jvm.options file needs parameters.  Note the port=9010 is used  by clients accessing the data.

To provide insecure access from only the local machine

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9010 
-Dcom.sun.management.jmxremote.local.only=true 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false

To provide securer access using TLS

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true

# the following statements point to the same key store as
# used by mqweb server.   This could be different.
-Djavax.net.ssl.keyStoreType=PKCS12
-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/mqweb.p12
-Djavax.net.ssl.keyStorePassword=password
# the following statements point to the same trust store as
# used by mqweb server.   This could be different.
# if you used self signed certificates you could have a keystore
# just for the JMX users
-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks
-Djavax.net.ssl.trustStorePassword=zpassword
-Djavax.net.ssl.trustStoreType=JKS

# The following defines the userid and password file
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.password.file=/home/colinpaice/ssl/ssl2/jmxremote.password

# The following defines the access a userid can have
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.access.file=/home/colinpaice/ssl/ssl2/jmxremote.access

jmxremote.password has

# specify actual password instead of the text password
monitorRole password
controlRole password

jmxremote.access has

# The "monitorRole" role has readonly access.
# The "controlRole" role has readwrite access.
monitorRole readonly
controlRole readwrite

Client set up

jconsole

You cannot pass a  userid and password when the jconsole command, so you have to disable authentication in the jvm.options file

-Dcom.sun.management.jmxremote.authenticate=false

The parameters for jconsole have  -J on them, as in -J-D…. .  jconsole removes the -J and uses the rest of the parameters when invoking the JVM.

I could not get jconsole to recognize a config file using the -J-Dcom.sun.management.config.file = /path/to/jmxremote.properties , so I wrote a bash script to make it easier to change parameters.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file"
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-J-Djavax.net.debug=ssl:handshake"
jconsole -J$ssl1 -J$ssl2 -J$ssl3 -J$ssl4 -J$ssl5 -J$ssl6 -J$ssl7 $ssl8 127.0.0.1:9010

The option “-J-Djavax.net.debug=ssl:handshake” gives a verb verbose trace of the ssl flows for the handshake.

The option -J-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file enables the jconsole logging.  I did not find the output very useful.

There is information the logger in general here,  and on the file logger, here.

The logging.file had

Logging.properties

handlers= java.util.logging.FileHandler
// , java.util.logging.ConsoleHandler2

java.util.logging.FileHandler.pattern=/home/colinpaice/JMXQuery/java/log.%g.file
java.util.logging.FileHandler.limit=50000
java.util.logging.FileHandler.count=2
java.util.logging.FileHandler.level=ALL
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

// .level = INFO
// logger.level = FINEST
.level = FINEST
// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose...
javax.level= FINER
javax.management.level = FINER
javax.management.remote.*     = FINER 
javax.management.remote.level = FINER
javax.management.remote.misc.level  = FINER
javax.management.remote.rmi.level= FINER

Using jmxquery

I used a bash shell script to run the command, as it was easier to manage, and I could not find a way of having the java system properties in a file.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ibmsys1.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"les
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file" 
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-Djavax.net.debug=ssl:handshake"
jar="-jar JMXQuery.jar"
user="-username monitorRole -password password"
url="-url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi"
parms=" -q   WebSphere:*  -count 2 -every 2"
java $ssl1 $ssl2 $ssl3 $ssl4 $ssl5 $ssl6 $ssl7 $ssl8 $jar $url $user $parms

 

Accessing JMX data in Liberty server, securely.

I thought  I would complete the work I did with using JMX in the mqweb server.   It was another example of Hofstadter’s Law:

It always takes longer than you expect, even when you take into account Hofstadter’s Law.

I spent a lot of time looking for things on the web, expecting them to be obvious, only to find that the things do not behave as expected.  I could not find them, because they were not there.  For example I expected to be able to configure the JMX server to use my OS userid and password.  I could have a file with userids and passwords, or lookup in LDAP, but not my normal userid.

Getting started

I found there are two ways of getting the JMX data from the mqweb server.

  1. Use of the native JMX support
  2. Using the Liberty REST API

I think the REST API is easier to set up and is more secure.

I’ll document a high level overview of the two approaches, and how to configure them

Overview of using the native JMX support.

To use this, you configure parameters in the jvm.options file, including a port solely for JMX.

You can use TLS certificates to set up a secure link between the client and the server.

You can decide if you want to logon with userid and password.  If you do you can set up

  1. A file with userids and passwords; and a file with userids and permitted access.   The documentation talks about userids like monitorRole and controlRole.   You have to put a process in place to periodically change these passwords.
  2. Use and LDAP server to do userid validation and to get the access.
  3. I could not find how to use your operating system userid and password for authentication.
  4. I could not find how to use the DN as authorization.

If your certificate is valid (either because it is signed by a CA, or there is a copy of a self signed certificate in the trust store), this is good enough for the checking.   You can enable userid and password checking, but this solution feels weak, as you have to do extra work to manage it properly;  you do not have a single signon.  Not all tools support using userid and password, for example I could not pass userid and password on the jconsole command.

Overview of Using the Liberty REST API

As with the MQ REST API you can issue an HTTP request and get data back.  See here.  For example

curl –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password https://localhost:9443/IBMJMXConnectorREST/mbeans/WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes=*

There is a small amount of configuration you need to do – less than with the native JMX support.  The data comes back as JSON (as you might expect) and also includes a time stamp, which is very useful when post processing.

You define <administrator-role><user>..</user></administrator-role>  in a similar way to setting up authorisation for mqconsole and mqrest.  It takes the cn= value from your certificate as the userid, so you can give individual access.

“Securely” is a good laugh.

There are different levels of (in)security.

If you are using the native JMX support

  • You can have no passwords or access checks needed.  The data is read only, and is not sensitive.
  • You can set up userid(s) and passwords in a file
  • You cannot use the operating system userid and password
  • You can use LDAP to check the userid and password, and get the role for that userid
  • You can use TLS, so anyone with a valid certificate can access the data
  • You can use TLS and use the userid and password in a file to determine access
  • You can use TLS and LDAP to get the role for that certificate

If you are using the WLP REST support

  • You can specify a userid and password
  • You can use a certificate, and the Common Name is used as the userid
  • You can specify in the configuration file, what access userids, or groups have

You can use TLS to protect your communications to and from the server.

Java leaks passwords

You need to be aware that your client machine may leak information.  For example I ran a  Java program to issue JMX requests from a script.

I could use the linux command ps -ef to display information about my request

ps -ef |grep JMX

gave me

colinpa+ 1871 1870 79 10:27 pts/2 00:00:01 java …  -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12 -Djavax.net.ssl.keyStorePassword=password …  -username monitorRole -password password

This exposed the password to my keystore and password to my userid!  I could not find a way of having all these java system parameters in a file.

I found export JAVA_TOOL_OPTIONS=”-D…”  and this get picked up, but then java displays the variables as in Picked up JAVA_TOOL_OPTIONS: …

jconsole

Some programs have been designed to protect information for example jconsole you can put your system properties in a file

-J-Dcom.sun.management.config.file=ConfigFilePath

and so keep your parameters secure, but I could not get this to work.

Curl can be configured not to display parameters

With curl you have a command like

curl -n –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password

which gives away your password.  If you do not specify it inline, you get prompted for it.

You can put your parameters in a config file, for example curl.config,

cacert ./cacert.pem 
cert ./colinpaice.pem:password 
key colinpaice.key.pem 
cookie cookie.jar.txt 
cookie-jar cookie.jar.txt 
url https://127.0.0.1:9443/ibmmq/rest/v1/login

and use

curl –config curl.config

Easy!

Protecting key files

It is important to protect the certificate file (with the important private key) so it is accessible by just the owner.  The linux command  ls -ltr colinpaice.p12 gives

-rw------- 1 colinpaice colinpaice 4146 Jan 31 17:56 colinpaice.p12

Of course anyone with super user authority has access to this file!

mqweb – displaying the secret statistics

Yes, mqweb does provide statistics; through the standard JMX interface provided as part of the base Liberty function.  I expect most people do not know they are available.   The data gets less useful over time, for example you get the “average time” since the mqweb started, rather than the last minute.  See here on how to extract useful information from the data, and show useful averages.

You need in mqwebuser.xml .

<featureManager>
<feature>monitor-1.0</feature>
</featureManager>

and in jvm.options

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

These options should be suitable in your test environment.  You will want to change them for production.

You need the port number (9010 in the above example) when you extract jmx data.

How do you display the data?

For a quick sniff, (no good for extracting data and plotting charts)  you can use jconsole.  Use remote connection localhost:9010 .  it does not display all of the data.

I found jmxquery very useful.  I updated the github version to fix a bug which caused a loop.  See here.

The query is

java -jar JMXQuery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi -q ‘WebSphere:*’

You get data on

  • WebSphere:type=JvmStats
  • WebSphere:type=ThreadPoolStats,name=Default Executor
  • WebSphere:type=ServletStats,name=com.ibm.mq.console.javax.ws.rs.core.Application/AppName (String) = com.ibm.mq.console
  • WebSphere:type=ServletStats,name=com.ibm.mq.rest.javax.ws.rs.core.Application/AppName (String) = com.ibm.mq.rest

To get ‘console’  and ‘ResponseTime’ data I used

java -jar JMXQuery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi 
-q 'WebSphere:*' 
-count 20 -every 60 
|grep --line-buffered console
|grep --line-buffered ResponseTime
|python3 mqweb.py
  • Where
  • java -jar JMXQuery.jar – invoke the program
  • -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi – with this url and the above port number from the jvm.options
  • -q ‘WebSphere:*’ – give me only data for this components
  • -count 20 -every 60 – my extensions giving a record every 60 seconds, and doing 20 of them
  • |grep –line-buffered console – only pull out the console records, ( ignore the ‘rest’ records).  The –line-buffered tells grep to flush it immediately
  • |grep –line-buffered ResponseTime – only interested in this detailed level
  • |python3 mqweb.py – pass it into the python program.  This calculates the delta between records and prints out the count and mean value for that interval

If you are collecting data in real time from a stream, you need to ensure any processing is unbufferred.  Often the default behavior is to accumulate the data in a big buffer, and write the buffer when it is full.  Check any filters you use, for example grep –line-buffered.

One line of output from the JMXQuery program for the console activity is

WebSphere:type=ServletStats,name=com.ibm.mq.console.javax.ws.rs.core.Application/ResponseTimeDetails/count (Long) = 5

JvmStats

This comes under WebSphere:type=JvmStats/  See here.

  • FreeMemory (Long) = 13350536
  • ProcessCPU (Double) = 0.658040027605245
  • UsedMemory (Long) = 75131256
  • GcTime (Long) = 539
  • UpTime (Long) = 1048277
  • GcCount (Long) = 118
  • Heap (Long) = 88473600
  • FreeMemory (Long) = 13350536
  • ProcessCPU (Double) = 0.658040027605245
  • UsedMemory (Long) = 75131256
  • GcTime (Long) = 539
  • UpTime (Long) = 1048277
  • GcCount (Long) = 118

ThreadPoolStats

This comes under WebSphere:type=ThreadPoolStats,name=Default Executor/

  • PoolSize (Integer) = 8
  • ActiveThreads (Integer) = 2
  • PoolName (String) = Default Executor

See here.

com.ibm.mq.console and  com.ibm.mq.rest

The data is similar between them.  One has name=com.ibm.mq.console.javax.ws.rs.core.Application, the other has name=com.ibm.mq.rest.javax.ws.rs.core.Application

The data (in italics) with my comments in plain font are

  • AppName (String) = com.ibm.mq.console
  • RequestCountDetails/currentValue (Long) = 116
  • RequestCountDetails/description (String) = This shows number of requests to a servlet
  • RequestCountDetails/unit (String) = ns – this looks like a bug as it is a count not nanoseconds
  • RequestCount (Long) = 116
  • ServletName (String) = javax.ws.rs.core.Application
  • ResponseTimeDetails/count (Long) = 116
  • ResponseTimeDetails/description (String) = Average Response Time for servlet
  • ResponseTimeDetails/maximumValue (Long) = 3060146565 – in nanoseconds ( see below for the unit)
  • ResponseTimeDetails/mean (Double) = 8.796846855172414E7 – in nanoseconds
  • ResponseTimeDetails/minimumValue (Long) = 793871 – in nanoseconds
  • ResponseTimeDetails/standardDeviation (Double) = 4.198572684166255E8
  • ResponseTimeDetails/total (Double) = 1.0204342352E10 – in nanoseconds – used in calculations
  • ResponseTimeDetails/unit (String) = ns  – this is the units.  ns is nanoseconds
  • ResponseTimeDetails/variance (Double) = 1.64064538075292096E17 –  used in calculations
  • Description (String) = Report Servlet Stats for specified Servlet and application.
  • ResponseTime (Double) = 8.796846855172414E7 – same as the ResponseTimeDetails

So we can see that there were

  • 116 console requests since the mqweb server was started
  • the units are ns (nano seconds)
  • the console requests taking an average of 8.796846855172414E7  nanoseconds, 0.0879 seconds with
  • a standard deviation of  4.136787844033974E7 – nanoseconds = 0.04198 seconds
  • the maximum value was 3 060 146 565 ns = 3.060 seconds
  • the minimum time was  793 871 ns or 0.000793 seconds

Some other data, showing how it changed over time

Data Values Later values Much later values
Number of requests 82 3590 22920
Average (seconds) 0.158 0.0108 0.0099
Standard deviation (seconds) 0.487 0.034 0.035
Maximum (seconds) 2.3 2.3 2.3
Minimum(seconds) 0.001 0.0001 0.0001

Notes:

  • There is data only once a request has been processed, so if you have not run a rest request, there will be no JMX data for rest activity.
  • These values are from start of the mqweb server. I did not see them reset, so you could have a data for a whole week or more.
  • The maximum was from the first requests to run.  I expect this includes the “warm up” costs,  of loading the code and JITing it.
  • The average values are from the start, so will be impacted by peaks and troughs.

For the each mqconsole window, there are two console counts every 10 seconds.  Any charts are refreshed every 10 seconds, so  I think this is a “I am still here, please send me any data you have for me”.

Data for rest

I started my mqweb server, and ran a python program which opened a connected and got three messages.

  • Maximum time 0.3486
  • Minimum time 0.0026
  • Calculate the other one 0.0028

Because the first request takes a long time, you can adjust for this in your calculations to get a truer mean.

For example

I reran the script and processed 100 messages.  The average time of these was 0.003 seconds.

  • Maximum 0.3486
  • Mean 0.00637
  • Count 103
  • Total 0.656

The calculations are

  • Mean * count =  0.656 (which matches Total as expected)
  • Subtract the maximum, first time value 0.656 – 0.349 = 0.307
  • Calculate the improved mean value ignoring the first value,  0.307 /(103 -1) = 0.003

So the adjusted mean time is 0.003 seconds – compared to the 0.006 which the JMX stats report.

 

Getting useful information out of JMX data

The data coming from Liberty WebServer through the JMX interface  provides some data, but it is not very useful, and it may become inaccurate over time.

I’ll cover

  1. Getting a useful mean value
  2. Getting a more accurate long term mean
  3. Data gets more inaccurate over time
  4. Standard deviation (this may only be of interest to a few people)

For example from JMX, the reported  mean time for mqconsole transactions  was 9.9 milliseconds – this is for all requests since the mqweb server was started.   Over the previous minute the average measured time, for a 10 second period was 7, or 8 milliseconds, so well below the 9.9 reported.

This is because the mean time includes any initial start up time.   The maximum transaction time, at the start of the run, was over 2 seconds.   This will bias the mean.

You can process the data to extract some useful information, and I show below how to get out useful response time values.

You get the following data (and example values) from mqweb through the JMX interface.

ResponseTimeDetails/count (Long) = 20
ResponseTimeDetails/description (String) = Average Response Time for servlet
ResponseTimeDetails/maximumValue (Long) = 3060146565 
– in nanoseconds (see below for the unit)
ResponseTimeDetails/mean (Double) = 4.336789965E8
– in nanoseconds
ResponseTimeDetails/minimumValue (Long) = 2474556
– in nanoseconds
ResponseTimeDetails/standardDeviation (Double) = 9.089057964078983E8
– in nanoseconds
ResponseTimeDetails/total (Double) = 8.67357993E9
– used in calculations
ResponseTimeDetails/unit (String) = ns
– the unit ns = nanoseconds
ResponseTimeDetails/variance (Double) = 8.319076762335653
– used in calculations

Getting a useful mean value

To produce these numbers, the count of the response times and the sum of the transaction response times are accumulated within the Liberty Server.  To calculate the mean value you calculate sum/count.   This gives you the overall mean time.  If you obtain the data periodically you can manipulate the data to provide some useful statistics.

Let the count and sum at time T1 be Count1, and Sum1, and at time T2 Count2, and Sum2.
You can now calculate (Sum2- Sum1)/(Count2 – Count1) to get the average for that period.  For example the reported mean was 0.016 ms, but the calculated value gave 0.008 ms.  You can also calculate (Count2 – Count1)/(T2-T1) to give a rate of requests per second.   These are much more useful than the raw data.  I suggest collecting the data every minute.

Getting a more accurate long term mean

The first rest request and console request take a long time because the java code has to be loaded in etc.  In one test the duration of the first request was 50 times the duration of the other requests.  A better “mean” value is to ignore the duration of the first request.

The improved mean is (JMX mean * JMX count  – JMX Maximum value) /(JMX Count-1), or JMXMean – (JMXMaximum/JMXCount) .

Data gets more inaccurate over time

The total time is stored as a floating point double.  As you add small numbers to big numbers, the small numbers may be ignored.  Let me try to explain.

Consider a float which has precision of 3, so you could have 1.23 E2 = 1230.  Add 1 to this, and you get 1231 which is 1.23 E2 with a precision of 3 – the number we started with.

The total time is in nanoseconds so 1 second is stored as 1.0 E9.  With 100 of these a second, and 1 hour( 3600 seconds) for 100 hours is 360,000,000, or 3.6 E8 seconds.  * 1.0 E9 nano seconds. = 3.6E17 nano seconds.   The precision  of most float numbers is 16, so with this 3.6 E17 we have lost the odd nanosecond.    I do not think this is a big enough problem to worry about – unless you are running for years without restarting the server.

The variance uses the time**2 value.  So with the maximum time above 599482097 nano seconds. Time **2 is 3.593787846×10¹⁷ and you are already losing data.  I expect the variance will not be used very often, so I think this can be ignored.

If the times were in microseconds instead of nano seconds, this would not be a problem.

Getting a useful standard deviation (this may only be of interest to a few people)

The standard deviation gives a measure of the spread of the data, a small standard deviation means the data is in a narrow band, a larger standard deviation means it is spread over a wider band.  Often 95% of the values are within plus or minus 3 * standard deviations from the mean, so anything outside this range would considered an outlier, or unusual statistic.

I set up some data, a mixture of  10  values 9, 10, 11,  the standard deviation was 0.73.    I changed one value to 20, and the standard deviation changed to 3.3, indicating a wide spread of values.

With a mixture of 100 values 9,10,11, the standard deviation was 0.71.   I changed one value to 20, and the standard deviation changed to 1.2, so a much smaller value, most of the data was still around 10 – just one outside the range.

With a lot of data, the standard deviation converges on a value, and “unusual” numbers make little difference to the value.  I think that the standard deviation over an extended period is not that useful, especially if you get periodic variations such as busy time, and overnight.

You calculate the standard deviation as the square root of the variance.   The variance is (Sum of (values**2) – (mean ** 2)) /number of measurements.

With data

ResponseTimeDetails/count (Long) = 203
ResponseTimeDetails/mean (Double) = 6420785.187192118 nanoseconds
ResponseTimeDetails/variance (Double) = 1.7113341125320868E15 – used in calculations

Variance  = 1.7113341125320868E15 =  ( (Sum of (values**2) – (6420785.187192118 ** 2)) / 203

So (Sum of (values**2)) =   3.474420513264337e+17

You can now do the sums as with the mean, above:

At time T1, the ssquares1 is the sum of (values**2)   at time T2, the ssquares2 is the sum of (values**2).

You can now calculate ssquares2 – ssquares2, and used that to estimate the variance, and so the standard deviation of the data in the range T1 to T2, I’ll leave the details to the end user.

For the advanced user,  you can use the mean for the interval – rather than the long term mean.  Good luck.