mqweb- is it better to use certificate authentication or userid and password?

Like may questions the answer is it depends.

You can set up your mqweb environment

  1. Certificates are required – you cannot use userid and passwords
  2. If a valid certificate is found use that, else prompt for userid and password
  3. Do not use certificates – just use userid and passwords.

This is described in the Liberty documentation.

It says for the <ssl…> tag

  1. If you specify clientAuthentication="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake does not succeed.
  2. If you specify clientAuthenticationSupported="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.
  3. If you do not specify either clientAuthentication or clientAuthenticationSupported, or you specify clientAuthentication="false" or clientAuthenticationSupported="false", the server does not request that a client send a certificate during the handshake.

No matter how you logon, the web server creates a LTPA* cookie with encrypted information about the logon.   The mqwebuser.xml attribute ltpaExpiration says how long this cookie is valid for.  If the time is exceeded you have to logon again, and it generates a new cookie.

Logging on

Certificate authorisation

If you are using certificates, the server sends down a list of valid self signed certificates or CA certificates, and the browser compares this list with the certificates in its store.   If the certificate in the browser’s store is acceptable, it is added to a list

The list is displayed, and the end user selects a certificate.  For subsequent requests with the same  http://url:port combination, in the same or a different tab within the browser window, the same certificate is used, and the browser does not prompt.

If the ltpa cookie expires then the same certificate is used, invisibly to the end user.  A new cookie is generate and sent down.

If there are no valid certificates in the list, then depending clientAuthentication, it may prompt for a userid and password.

You cannot logoff from the page as there is no option to do so.

Userid and password.

When userid and password are passed up in a header (not in the URL as some web servers accept) the ltpa token is returned.  When the ltpa token expires, the user is prompted to re-logon.   They need to enter the userid and password again.  You can also logoff from the web page.

What are the implications of certificates and userid with password

Certificate logon

  • The web browser can logon, and although the ltpa token expires, it is automatically logged on again.   This is useful for a “big screen monitor” in the operations room, where you want the display active all day.
  • Someone can create a file with some HTML in it to browse a queue  (if authorised).  So if they are authorised to the queue for their normal job, they could easily create a web page to display the queue contents.  This could be the end user, or an evil person saying “click this link”.

userid logon

  • If the user is using the mqconsole web pages, then periodically they will get logged off – and they will have to logon again.
  • If they try to access a queue from an HTML page, they will be asked for userid and password.  Depending on what the end user is doing, this may give them notification than there is some unusual activity.  If they are hacking they will enter the userid and password anyway.
  • The person may have access to the same information without using the mqweb console interface, so there is no change to the threat level.

What do I recommend?

Tricky.

  • I think long running monitors need to use certificates
  • End users should logon with userid and password, with an expiry period of about an hour or less.
  • You may want to have a Certificate  Authority just for the mqweb browsers, to stop your corporate certificates from being used to access the mqweb server.

Because it is easy to create an html page and browse a queue, I would allow administration over mqrest, and not allow processing messages over REST.

As I said in another posting

  • The admin API authorisation is managed through <security-role name=”MQWebAdmin”> and <security-role name=”MQWebAdminRO”> sections in the mqwebuser.xml file.
  • The messaging API authorisation is managed through <security-role name=”MQWebUser”> sections.

To stop messaging over the rest interface, have no entries in the <security-role name=”MQWebUser”> section.  This applies to both certificates, and userid and password.

(Personally I would have called<security-role name=”MQWebUser”> as <security-role name=”MQWebMessaging”>  to make the roles clearer. 

Update from Gwydion…

The MQWebAdmin, MQWebAdminRO and MQWebUser roles can all be used for the admin REST API. That’s why MQWebUser is not called MQWebMessaging – it’s not just for messaging. The difference between them is the user ID that’s checked by the qmgr for operations performed via the REST API.
  • Operations performed by users in the MQWebAdmin* roles take place under the context of the mqweb server user ID.
  • Operation performed by users in the MQWebUser role take place under the context of the user logged into the REST API.
To stop messaging over the REST API completely, the mqRestMessagingEnabled should be set to false in the mqweb server configuration.

What does mqRestCorsMaxAgeInSeconds in mqweb mean?

Ive blogged about CORS, and how this allows you to list sites that are permitted to use scripts to send request to the mqweb server.

I struggled with understanding what value mqRestCorsMaxAgeInSeconds has, as it did not behave as expected (this was due to a bug).

If you have a CORS transaction there is an OPTIONS request, followed by the actual DELETE/GET/POST requests.
The OPTIONS request checks that the request is coming from an authorised list of URLs, and that the parameters are valid.  The OPTIONS information can be cached.

If the check is successful then the real request can be issued.  If the requests occur in a short time, then you can have OPTIONS, DELETE, DELETE, DELETE, using the cached values.  If there is a “long” time interval between the requests you may get OPTIONS, DELETE, gap, OPTIONS, DELETE.

The OPTIONS information can be cached for up to the minimum of the mqweb mqRestCorsMaxAgeInSeconds  time, and the browser time.

For Chrome the maximum time interval is 600 seconds.  If no time is specified in the OPTIONS response, then 5 seconds is used.

There is a bug in the Liberty base code which sends down the header Access-Control-Allow-Max-Age: …, when the browser is expecting Access-Control-Max-Age.   Because of this, the default time of 5 seconds is used in Chrome.

This should not have a major impact.  For those applications using scripts to send multiple REST API request, there will be more frequent OPTIONS requests – every 5 seconds instead of up to 600 seconds.  These extra flowes are invisible to the scripts and the end user.

What value should I use?

Chrome has a maximum of 600 seconds, with a default of 5 seconds.

Firefox has a maximum of 24 hours (86400 seconds).

Setting it to 600 seconds sounds reasonable to me.

Making changes to mqRestCorsMaxAgeInSeconds

If you change mqRestCorsMaxAgeInSeconds you have to restart the mqweb server.

I do not get caching!

When researching this topic I found every GET request had an OPTIONS request, rather than the OPTIONS, GET, GET.   A quick search on the internet showed me the Chrome option ( F1 -> Settings and preferences) “Disable Cache ( while DevTOOLS is open)” was enabled. I deselected this, and I got the caching.

Can evil websites get to your mqweb – using java script to get to the backend server

With HTML and scripting people would write scripts and get you to execute them, and so access your personal information, and steal your money.  Over time security has been improved to make this harder, and now you have to explicitly say which web sites can run scripts which use the mqweb interface to access MQ to put and get messages.

One way of protecting the access is using Cross Origin Resource Sharing, or CORS.  I explained the basics of CORS here.  I struggled getting it to work with a web browser.

  • browsers have been improved and what worked last year may no longer work now, and the documentation does not reflect the newer browsers.
  • Chrome carefully changes your hand crafted HTTP headers, so what is sent up may not what you expected.

I’ll go through three examples, and then show how it gets more difficult, and what you can do to identify problems.

Note: If you use a web page from a file:// then the origin is ‘null’, and this will fail any checks in the backend, as the checks compare the passed origin to the list of acceptable urls.

I used Dev Tools in Chrome (Alt+Ctrl+i) to display information including the headers flowing.

Simple HTML

With the following within your web page,

<a href=”https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/…/message
> direct link ></a>

It issues the REST request, returns the data from the queue, and displays it.

From the headers we see

Simple HTML: Request headers

The important request headers were

And no origin header.

Simple HTML: Response headers

The response headers have

  • HTTP/1.1 200 OK
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent
  • Content-Length: 1024
  • Set-Cookie: LtpaToken2_1 ….

There are no Allow….  headers, so this indicates the response is not a valid cross origin response.   The request came from one page, so there was no cross origin request.

You can see the MQ attributes returned, ibm-mq*.

Invoking request from java script.

My HTML page had

<html
<head>
<script src=”http://localhost:8884/src.js ” text=”text/javascript”></script >
</head>
<body>
<button onclick=”src()””>Get message</button>
</body>

I had the src() script downloaded from http://localhost:8884/src.js, and inline within the html file.  Both worked.

function src()
{
  fetch("https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message",
  { 
     method :'DELETE',
     headers: {
                'Origin': 'https://localhost:888499'
                ,'Access-Control-Request-Headers' : 'Content-Type' 
                ,'ibm-mq-rest-csrf-token' : '99'
              } 
   }
  ) // end of fetch
   .then((response) => response.text() )
   .then(x => {document.write("OUTPUT:"+x); } 
  ) 
  .catch(error =>  {console.log("Error from src:" + error);});
}

When the button was pressed, the script was executed,  there was an OPTIONS request (and response), and a DELETE request (and response).   It returned a message and displayed it.

In more detail, the flows were:

JavaScript OPTIONS request

The OPTIONS request had headers

This is doing a preflight check, saying it intends to issue a DELETE request from Origin  http://localhost:8889, the url of the web page.

JavaScript OPTIONS response headers

The OPTIONS response headers were the same as before but with additional ones

  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8889
  • Access-Control-Allow-Max-Age: 91
  • Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE
  • Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

This means, the script is allowed to issue a request from http://localhost:8889 using a request method in the list {GET, POST, PATCH, PUT, DELETE } and the valid headers that can be used are in the list of headers.

The Access-Control-Allow-Max-Age: 91 came from my mqwebuser.xml file, <variable name=”mqRestCorsMaxAgeInSeconds” value=”91″/>.

After this there was a DELETE request to get the message.

JavaScript DELETE request headers

JavaScript DELETE response headers

The response included the CORS headers, and included the headers from the non CORS situation

  • HTTP/1.1 200 OK
  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8889
  • Access-Control-Expose-Headers: Content-Language, Content-Length, Content-Type, Location, ibm-mq-qmgrs, ibm-mq-md-messageId, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers
  • Content-Type: text/plain; charset=utf-8
  • Content-Language: en-GB
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent
  • Content-Length: 1024
  • Set-Cookie: LtpaToken2_….

Because the Access-* headers are present, this is a CORS response.

The message content was displayed in a browser window.

Link to another page

I set up a link <a href=”http://localhost:8884/page.html”/>localhost 8884</a>  to execute a page on another web server.  When this executed, it issued the java script request as before.  The Origin was Origin: http://localhost:8884 – so the page where the script executed.

What happens if CORS is not set up?

If the http://localhost:8889 is not in the list in the mqwebuser.xml file,

No data was  displayed.   The Chrome browser Developer tools ( Alt+Ctrl+i) displays a message

Access to fetch at ‘ https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

The OPTIONS Request header has, as before

  • HTTP/1.1 200 OK
  • Host: localhost:9445
  • Access-Control-Request-Method: DELETE
  • Origin: http://localhost:8889
  • Access-Control-Request-Headers: ibm-mq-rest-csrf-token

but the OPTIONS Response header has no Access-* headers.

No DELETE request was issued.

The HTTP/1.1 200 OK is the same for all cases – it means the request was successful.

Trying to be clever

I read the documentation on the web, and much of it was very helpful, but some of it is no longer true.  It was hard to get it working, because every things has to be right for it to work.

Unsupported header.

I added an extra header, which is a valid CORS thing to do – but the back end has to support it.  With hindsight it makes no sense to add headers that will be ignored by the server.

headers: {
     'Origin': 'https://localhost:888499'
     , 'Access-Control-Request-Headers' : 'Content-Type' 
     , 'colin':'TRUE' 
     ,'ibm-mq-rest-csrf-token' : '99'
}

This sent up a header with

Access-Control-Request-Headers: colin,ibm-mq-rest-csrf-token

Header “colin” is not in the list of accepted headers, header ibm-mq-rest-csrf-token is in the list:

Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

The Developer tools message was the same as before,

Access to fetch at ‘https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

I removed the unsupported header and it worked.

Other headers do not work

The request header ‘Access-Control-Allow-Method’ : ‘DELETE’ is a valid header.

When this was used, the request headers included

  • Access-Control-Request-Headers: access-control-allow-method,ibm-mq-rest-csrf-token
  • Access-Control-Request-Method: DELETE

As before  Access-Control-Request-Method is not in the list of Access-Control-Allow-Headers, so the request fails.

The Access-Control-Request-Method: DELETE is not needed, as the method: DELETE defines what will be used.

Using Access-Control-Request-Headers to add more headers does not work, if the header is not in the list of valid parameters.

The true origin is used

I tried using the header ‘Origin’: ‘https://localhost:888499 ‘ – as it did with my curl – and this was ignored.  The true Origin from the web page was used (I am glad to say, otherwise it would make the whole protection scheme worthless)

Some options ignored

I found  the “fetch” options credentials:, redirect: , and cache:, were all ignored by Chrome.

Some  cookies  were used.

The LtpaToken2_… was sent up and down

Problems in mqwebuser.xml file

I misconfigured the configuration file with <variable name=”mqRestCorsMaxAgeInSeconds” value=”AA91″/>

This gave me the familair message

Access to fetch at ‘https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

So if you get this message it is worth checking the configuration file for problems.

Can I trace it?

I used <variable name=”traceSpec” value=”*=audit:CorsService=finest”/> in the mqwebuser.xml file to provide information about the CORS processing, and what was coming in, and what was being checked.

Can evil websites get to your mqweb – understanding CORS

In the beginning was the html, and the html was good;  then we had html and scripts  which could only do things on the page, which was also good; then we had scripts which could reach out to other websites – and that’s where the problems began.  It was easy for evil developers to get you to click on an innocent looking page, which had a script which jumped into a different tab of your browser where you had your banking window open ,  or  to executed a script ; and steal all your money.

The browsers were improved to stop evil scripts from accessing a server, and then they were improved again so the server could say “stuff coming from this list of web sites is OK, I trust them.   One implementation is called CORS.  There is a good description here.   It says

Cross-Origin Resource Sharing (CORS) is a mechanism that uses additional HTTP headers to tell browsers to give a web application running at one origin, access to selected resources from a different origin. A web application executes a cross-origin HTTP request when it requests a resource that has a different origin (domain, protocol, or port) from its own.

I do not trust things that “just work”, I like to see evidence of it.   This lack of trust comes from working with a group of young testers who came to IBM Hursley to test their code.   All of their tests ran cleanly – and thought they had a few days spare to go to London sightseeing.  I stopped the server they were meant to be using (without telling them) and the tests carried on running successfully!   It turned out they had been testing their code with a dummy program acting as the server.    They removed this code, reran their tests  and most of them failed – and had to stay an extra week.

I also remember changing a config file, and being surprised when my changed worked first time.   After a cup of tea (an invaluable thinking aid) I put some spelling mistakes in the file ; and it carried on running.  Why? I was using the wrong config file.

I played with CORS and wanted to get things to fail, as well as to work.   This was a good choice, as I had many failures.

I’ll document how I got curl to work and demonstrate CORS , and document how I got a web browser to work – a real challenge

mqweb implements CORS, so you can configure mqweb to give a list of websites which may access your server.

The documentation  is not very clear.  It says

where allowedOrigins specifies the origin that you want to allow cross-origin requests from. You can use an asterisk surrounded by double quotation marks, “*”, to allow all cross-origin requests. You can enter more than one origin in a comma-separated list, surrounded by double quotation marks. To allow no cross-origin requests, enter empty quotation marks as the value for allowedOrigins.

My observations,

  • You cannot use generics, so http://127.0.0.1:* is the same as “*” – or allow all cross-origin requests
  • You must specify {scheme:address:port} so http or https,  the url with // at the front, and the specific port number
  • The match is an string equality test, so the case, spacing and values must be the same

How does an HTTP request work?

When you click on a web page, data is sent to the back end server.   The following data is exchanged

  • the request
  • request headers
  • your data going to the server
  • response headers
  • response data – such as the content of web page.

In more detail…

Request

Request Headers

  • accept: for example  text/html, application/xml
  • accept-languages: en-GB
  • dnt:  1  this is “do not track me”
  • user-agent:  Chromium
  • cookie: bcookie=”….”

Response headers

  • status: 200
  • status: 200 OK
  • set-cookie ….
  • server: nginx
  • ….

Response data

This might be the web page to be displayed.  It can include script, images etc.

What is the origin of the page?

If your web page, invoked a script, for example from clicking a button, an “Origin” header is added, for example Origin: http://localhost:8884 which is the address of the web server hosting the page.   The backend server checks to see if this header is present, and looks up site in the authorised list.

If the Origin is acceptable, it sends down additional response headers (CORS Headers), so the browser (or your program) knows and can use the web site.

As part of the handshake, the browser can send up an “OPTIONS” request (instead of a get/delete etc), with a header saying the browser will be doing a GET/DELETE etc from this origin.    If there is a positive response, where the additional CORS headers are send back then the get/delete is allowed.   If the CORS headers are not present in the response, then the request will not be permitted.   This is called a preflight check – just like having your boarding pass checked at the gate before you get on the plane.

Does it work?

If there is no Origin header in the request, the backend server thinks it is all same domain, and does no CORS checks.

This CORS support is really aimed at web browsers, as the web browsers will automatically add headers for you.  If you are using curl or other tools to create your own request, you specify exactly the headers you want, so if you omit the Origins header, the server will not check.

My mqwebuser.xml file had <variable name=”mqRestCorsAllowedOrigins” value=”https://9999.0.0.1:19442,http://localhost:8884”/&gt;

So origins for https://9999.0.0.1:19442 and http://localhost:8884 are permitted.

I used a configuration file for curl (because command parameters did not work passing the headers) and had in curl.parms

cacert ./cacert.pem
cert ./colinpaice.pem:password
key ./colinpaice.key.pem
cookie cookie.jar.txt
cookie-jar cookie.jar.txt
request OPTIONS
header “Access-Control-Request-Method: DELETE”
header “Access-Control-Request-Headers: Content-Type”
header “Origin: https://9999.0.0.1:19442
header “ibm-mq-rest-csrf-token : COLINCSRF”
include

I used the command

curl –verbose –config curl.parms –url https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message

The headers were displayed, and I had

OPTIONS /ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message HTTP/1.1
> Host: localhost:9445
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: LtpaToken2_….
> Access-Control-Request-Method: DELETE
> Access-Control-Request-Headers: Content-Type
> Origin: https://9999.0.0.1:19442
> ibm-mq-rest-csrf-token : COLINCSRF

The response headers were

< HTTP/1.1 200 OK
< X-Powered-By: Servlet/3.1
< X-XSS-Protection: 1;mode=block
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src ‘none’; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’; connect-src ‘self’; img-src ‘self’; style-src ‘self’ ‘unsafe-inline’; font-src ‘self’
< Cache-Control: no-cache, no-store, must-revalidate
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: https://9999.0.0.1:19442
< Access-Control-Allow-Max-Age: 90
< Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE
< Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

The response header Access-Control-Allow-Origin: https://9999.0.0.1:19442 shows that requests with origin https://9999.0.0.1:19442 is acceptable.

When I used a different “Origin”,   I did not get any Access-Control-Allow-* headers.  So from the absence,  I could tell the request was not support from the different origin.

The Access-Control-Allow-Headers is a list of header names which can be sent, so the header ibm-mq-rest-csrf-token : COLINCSRF is valid, but “COLIN:value”  is not valid, because ibm-mq-rest-csrf-token  is in the Access-Control-Allow-Headers list, and COLIN is not in the list.

The Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE says my application can use any of the methods in the list.

Using a web browser.

If you use HTML in a local file, then the “Origin” is null, and so does not match any elements in the authorised list.  I had to set up my own web server – which was easy to do using Python.

Using the web page at the bottom of this posting, I pointed my web browser at the web server.   It displayed a button.  I pressed it, and it invoked a script. Using the developer mode ( Al+Ctrl +i) in Chrome could see network flows etc.

The request headers had

  • Host: localhost:9445 This is where my webserver is hosted
  • ibm-mq-rest-csrf-token : 99  I specified this header value
  • Origin: http://localhost:8884  this overrode the value I had specified in the headers.

The response headers included

  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8884
  • Access-Control-Expose-Headers: Content-Language, Content-Length, Content-Type, Location, ibm-mq-qmgrs, ibm-mq-md-messageId, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent

We can see from

  • the Access-Control-* headers we know this has validated for http://localhost:8884
  • the Access-Control-Expose-Headers we can see what headers will be accepted
  • ibm-mq-md-persistence:  persistence. For the returned messages, it was a persistent message

The web page used

<HTML>
<HEAD>
<TITLE>Call a  mqweb rest API </TITLE>
<script>
  function local()
  {
    fetch("https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message" 
      {
         method :'DELETE',
         headers: {
                    'Origin': 'https://localhost:888499'
                    , 'Access-Control-Request-Headers' : 'Content-Type'
                    ,'ibm-mq-rest-csrf-token' : '99'
                  }
     }
    )
    .then((response) => response.text() )
    .then(x => {  document.write("OUTPUT:"+x);    } )
    .catch(error => { console.log("Booo:" + error);});
  }
</script> /head> <body> <button onclick="local()"">press me</button> </body> </html>

 

For information about “fetch(…,…)” see here.

For information about the “.then(…) ” see here.

How do I tell if my puts and gets are in or out of syncpoint?

This question came up from someone who is new to MQ.  it is good to find people people who are new to MQ.

As usual there is an easy answer, but some cases you need to think about.

There is no command you can use to show this information.  You can find it it out using the accounting information.

Typically an application does

  • MQCONN
  • MQOPEN queue
  • MQPUT|MQGET
  • Possibly  a commit
  • MQCLOSE
  • MQDISC

If the accounting data shows that there was a COMMIT request, and the commit took about 1 millisecond, then there is a good chance that the put or get was in syncpoint.

If gets more interesting if you have

  • MQPUT of messages out of syncpoint (for example and audit trail – we got to this point.  Out of syncpoint so it exists even after a rollback)
  • MQPUT within syncpoint
  • MQCOMMIT

If the accounting information shows one commit, you cannot tell if it was one or two messages within syncpoint.

If the message is persistent and out of syncpoint, then under the covers it does “MQPUT|MQGET – and commit this requests”.  On z/OS a put or get typically take 10 microseconds, a log I/O takes about  1 ms, so if you see an MQPUT or MQGET with a long response time (1 millisecond), then it might have been a put or get of a message outside of syncpoint.
On z/OS you can also get the logging time for a put or get, so if you see the logging time for a put or get, this is a persistent message out of syncpoint.

A non persistent message does not write to the log on disk, all updates are kept in memory, so you cannot tell from the duration of the request if it was in or out of syncpoint.    Best practice is that non persistent messages are usually processed out of syncpoint.

If you see persistent messages are put or got, and the commit time is microseconds,  then the commit did not do disk IO, so there was no work to commit, and so the put or get was out of syncpoint.

And lastly, you may see no commit request in the accounting data.   This is bad practice, as it is assuming that the MQDISC will cause a commit or roll back, so it would happen invisibly.  An application should explicitly issue a commit if messages are within syncpoint.

As I said – pretty simple – but with some complexities.

 

 

 

Error connecting to JMX endpoint: Unsupported protocol: rest

I got this error message when I was trying to use JMX into the WLP web server, when using the restConnector-2.0 interface in Liberty.

The documentation was not that helpful.  Oneproblem was I suffered from the curse of cut and paste, and used a Windows environment variable %JAVA_HOME%  instead of using the Linux $JAVA_HOME.  Another problem was caused documentation saying add the jar to the class path, and then the class path was ignored.

True documentation.

When you use the  -jar option, the JAR file is the source of all user classes, and other user class path settings are ignored.  See here.

Unhelpful documentation

Configuring secure JMX connection to Liberty   says

jconsole -J-Djava.class.path=...;%WLP_HOME%/clients/restConnector.jar

This was for Windows – on Linux it would be $WLP_HOME – except on my system $WLP_HOME was not set.

jconsole

Some of the jar files are in $JAVA_HOME, you can use the environment variable, or specify it yourself.  Note %JAVA_HOME% is windows, so be careful when you use cut and paste.

  • jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar
  • jconsole -J-Djava.class.path=/usr/lib/jvm/java-8-openjdk-amd64/lib/jconsole.jar:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar

If the window displays “Secure connection failed” restart jconsole and use the -debug option.   For me this gave  “java.io.IOException jmx.remote.credentials not provided. Set to a String[2] {user,password}”.  I entered my userid and password, and this connected successfully.

I could not connect using my .jar file.

I was using JMXQuery to extract the data.  I was getting the Error connecting to JMX endpoint: Unsupported protocol: rest message.
Adding the jar file to my class path did not help, as the class path is ignored when using java the -jar parameter.

How to fix it

There are two ways of fixing this.

  1. Put the required jar file in the extensions path, not the class path
  2. Use the java -classpath…  instead of specifying java -jar

1. You need to have the jar for in the extensions path, not in the class path.

See How classes are found  in the Java documentation.  It says

  • Bootstrap classes – Classes that comprise the Java platform, including the classes in rt.jar and several other important jar files.
  • Extension classes – Classes that use the Java Extension mechanism. These are bundled as .jar files located in the extensions directory.
  • User classes – Classes defined by developers and third parties that do not take advantage of the extension mechanism. You identify the location of these classes using the -classpath option on the command line (the preferred method) or by using the CLASSPATH environment variable.

There is a system property java.ext.dirs  which gives the location of the Extension classes. On my system this was

/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext

  • The /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext: is java dependent. You should not put your files in this directory
  • /usr/java/packages/lib/ext This is for “user” extensions.

The directory /usr/java/packages/lib/ext did not exist on my system, so I had to do, create it, copy the web connection jar file to it, and grant permissions on it.

  • sudo mkdir -p /usr/java/packages/lib/ext
  • sudo cp /opt/mqm/web/clients/restConnector.jar /usr/java/packages/lib/ext/
  • sudo chmod 444 /usr/java/packages/lib/ext/restConnector.jar

2. Use a -classpath – not a -jar

In a .jar file there is a META-INF/MANIFEST.MF file which includes information on the entry point.

Manifest-Version: 1.0
Main-Class: src.Client

Instead of using the -jar option to point to the jar, you can use the -classpath to point to the jar and explicitly specify the entry point.  For example

java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar  
src/Client
service:jmx:rest://localhost:9443/IBMJMXConnectorREST
  • java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar –  use the classpath option, and specify the needed jar files.  Client.jar is my program.  /opt/mqm/web/clients/restConnector.jar  is the Liberty provided jar.
  • src/Client – the “entry point” class to use
  • service:jmx:rest://localhost:9443/IBMJMXConnectorREST – the url to use.
  • note the absence of a -jar option.

In the manifest of the Client.jar file it had Main-Class: src.Client.   This is for the the src/Client.java source file.   This ties up with the src/Client.class as seen when you use the command jar -tvf Client.jar .

Use the Liberty REST API to access the JMX data in Liberty

Rather than set up traditional JMX, where you specify the JMX port etc, you can use the REST support provided in LIberty to access the JMX data.   The rest support is easier to set up.

The Liberty documentation recommends that you do not have the native JMX support (configured in jvm.options), and the Liberty REST support for JMX configured at the same time.

The REST request to get the statistics worked and was easy to use.   I could not get the “traditional JMX interface”, such as jconsole to work with the REST interface.  See below.

Configure the server:  mqwebuser.xml

In mqwebuser.xml  add the support

<featureManager>
  <feature>restConnector-2.0</feature>
</featureManager>

Set up authorisation with

<administrator-role>
   <user>colinpaice</user>
   <group>MQADMIN</group>
</administrator-role>

<reader-role>
  <user>John</user>
</reader-role>

As the mqconsole and rest statistics are read only, then it may be better to set up every user as a reader-role.

As with the MQ support, it will use the userid if specified, or the Common Name from the digital certificate.

Using Curl and the rest API

I used

curl --cacert ./cacert.pem --cert-type P12 
--cert colinpaice.p12:password
-url https : //localhost:9443/IBMJMXConnectorREST/mbeans/...

Where … was  WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes/ResponseTimeDetails  . This gives the JMX statistics for the mq.console.

The data comes back as JSON (as you might expect) for example

"name": "ResponseTimeDetails",
  "value": {
    "value": {
      "count": "99",
      "description": "Average Response Time for servlet",
      "maximumValue": "3183755861",
      "mean": "1.116053166969697E8",
      "minimumValue": "1777114",
      "reading": {
        "count": "99",
        "maximumValue": "3183755861",
        "mean": "1.116053166969697E8",
        "minimumValue": "1777114",
        "standardDeviation": "4.360373971884932E8",
        "timestamp": "1580819294060",
        "total": "1.1048926353E10",
        "unit": "ns",
        "variance": "1.90128611746915776E17"
      },
      "standardDeviation": "3.218674128849494E8",
      "total": "1.1048926353E10",
      "unit": "ns",
      "variance": "1.63102693370991648E17"
    ...

As well as the data I have covered before, you also get the time stamp value.  This is the value in milliseconds from a well known time.

I used the python to convert the timestamp (1580978634610) to a date time

import datetime
s = 1580978634610 / 1000.0
print(datetime.datetime.fromtimestamp(s).strftime('%Y-%m-%d %H:%M:%S.%f'))

to give  2020-02-04 12:28:14.060000.

Which URL to use for traditional JMX?

The IBM documentation says the url to access the JMX data using “traditional JMX” is in file /var/mqm/web/installations/Installation1/servers/mqweb/log/state/com.ibm.ws.jmx.rest.address.  For me this was service:jmx:rest://localhost:9443/IBMJMXConnectorREST .

Client set up: Using the “traditional JMX interface” did not work for me

The Configuring secure JMX connection to Liberty page says you can use this url in jconsole and other tools.  I could not get this to work.   I got  messages like Error connecting to JMX endpoint: Unsupported protocol: rest.  This page gives a lot of information on JMX, and towards the end of the second page it says Error connecting to JMX endpoint: Unsupported protocol: xxxx is likely to be a problem with the class path.  I used  -verbose=class, and did not see the jar file being loaded.

What else can you do with the REST interface?

Show what is available

You may get data like

WebSphere%3Aname%3Dcom.ibm.mq.console.javax.ws.rs.core.Application %2C type%3DServletStats

The punctuation has been “escaped” so you need to change

  • %3A- to  :
  • %3D to :
  • %2C to ,

and the string becomes the familiar

WebSphere: name=com.ibm.mq.console.javax.ws.rs.core.Application, type=ServletStats