Whoops my certificate has expired – what do I need to do?

The amount of work depends on what has expired.

  • If the root CA has expired then you need to reissue all certificate which include the root CA, and update trust stores
  • If the enterprise CA has expired you need to renew all certificates signed by the enterprise CA, and update trust stores with the new certificate
  • If your personal certificate has expired, you need to renew that certificate.

The amount of work also depends on the size of your enterprise.

I’ll list the work need if the certificate is going to expire next week – as this is more work than if it has already expired

If your personal certificate is going to expire

You need to

  • renew it, create a new private key and certificate request
  • or extend the date of the existing certificate  – by re-signing it
  • update the keystores which include it.  This could be those used by java programs ( .p12, or .jks) as well as any browser keystore (.nssdb)

To renew your personal certificate

You can just recreate it.  If you are considering to use a stronger algorithms, such as Elliptic Curves, bear in mind that the servers need to be able to support what you specify.

Depending on how your openssl has been configured, if you have unique_subject = yes you may need to use openssl revoke… to remove the old personal certificate from the configuration.   This is to ensure there is only one certificate with the same distinguished name, and online checking of certificate validity (Certificate Revocation Lists and Online Status Certificate Protocol) can be used.  If you have unique_subject = no, you can recreate the certificate without revoking it.

  • openssl genpkey  to generate a new private key
  • openssl req to create a request

You do not need to recreate it – you can just sign it again.

You then need to do normal process of update the users of the certificate.

  • openssl ca to sign the request
  • openssl pkcs12 -export to create the .p12 file
  • You may want to deploy it only when the machine is attached to the local network.  If you download it over wireless, or email, these may be compromised, and the bad guys can just use the new copy.  Downloading it over a wired connection eliminates this risk.
  • certutil -D $sql -n $shortName to remove the certificate from the browser’s keystore
  • pk12util -i $shortName.p12 $sql -W password to add the certificate to the browser’s keystore
  • /opt/mqm/bin/runmqckm -cert -delete  to remove a certificate from a keystore (.jks)
  • /opt/mqm/bin/runmqckm -cert -import to add a certificate to a keystore(.jks)
  • restart the servers to pick up the new certificate

Rather than deleting the certificate from the keystore and adding the new one with the same short name,  you may want to make a copy of the old certificate from the keystore, or simply backup the whole keystore – as it should not change once frequently.  If there are problems, restore from the backup.

To renew your enterprise certificate

This requires a lot more work than updating a personal certificate.

Just like the personal certificate you can recreate it, or just sign it again.

You need to phase in the new certificate as it may take days or weeks to deploy it successfully.

The client’s trust store needs the CA certificate matching the CA used by the server’s certificate.

If you have two servers, and change servers to use the new CA certificate, the client’s trust store content will change over time.

  • The old CA certificate – before any work is done
  • Both old and new CA certificates during the migration period.  The server using the old CA will use the old CA in the client.   The server which has been migrated to use the new CA will use the new CA in the client.
  • Just the new CA certificate.  Once the migration has finished. The old certificate can be removed when all of the servers have been migrated, or the certificate has expired.

A key store and a trust store can have certificate with the same distinguished names, but with a different short name or alias name.  My enterprise is called SSS, and the CA for my enterprise is CASSS… .  My trust store has

  • the CA with C=GB,O=SSS,OU=CA,CN=CASSS and short name CASSS2016 and
  • the CA with C=GB,O=SSS,OU=CA,CN=CASSS and short name CASSS2020 and

When you maintain a keystore you use the short name, for my example CASSS2016 or CASSS2020.

During the TLS handshake, when the server’s certificate is sent to the client, it is checked against all of the trusted CA certificates in the keystore, so having two certificates with the same DN does not matter.

The steps to deploy your new CA are

  • Get the new CA.  Either create a new CA using the latest algorithms and cipher suites (the better solution), or have the old certificate resigned.
  • For each user and server
    • Deploy the CA certificate into the trust stores with a new name – eg CASSS2020
  • For each user and server
    • Renew or recreate each personal and server certificate and sign it with the new CA
    • Deploy it securely to each user and server and replace the old personal certificate
    • You may want to deploy the private key only when attached to the local network.  If you download it over wireless, or email, these may be compromised, and the bad guys can just use the new copy.  Downloading it over a wired connection eliminates this risk.
  • For the server create java.security rules to disable weaker algorithms and cipher specs.   It is easy to undo this change.
    • This should identify any user not using the latest cipher suites and certificates.  These rules can be relaxed while problems are fixed.
  • For each user and server
    • Make a copy of the old CA certificate prior to deletion
    • Delete the old CA certificate from the trust stores eg CASSS
    • After a validation period delete the copy of the old certificate

Start all over again!

You can see how much work you need to do when you renew your enterprise CA, so make sure you  renew it in plenty of time – months rather than days.

mqweb – who did what to what, when, and how long did it take?

You can provide an audit trail of the http requests coming into your server.  This is described in the base liberty document, and works in mqweb.

Within the httpEndoint tag you can add

<httpEndpoint host=”${httpHost}” httpPort=”${httpPort}” httpsPort=”${httpsPort}” id=”defaultHttpEndpoint”>
<httpOptions removeServerHeader=”false”/>
<accessLogging enabled=”true” filePath=”${server.output.dir}/logs/http_access.log”
logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U X:%{X}W r:”%r” s:%s’


See here for information on the accessLogging,  here for the syntax of the <accessLogging…> and here for the logFormat format options.

From the logFormat page

Remote IP address
Local IP address
Response size in bytes excluding headers
Response size in bytes excluding headers. 0 is printed instead of – if no value is found.
The request cookie specified within the brackets, or if the brackets are not included, prints all of the request cookies.
The elapsed time of the request – millisecond accuracy, microsecond precision
Remote host
HeaderName header value from the request
Request method
HeaderName header value from the response
Output the query string with any password escaped
First line of the request
Service time of the request from the moment the request is received until the first set of bytes of the response is sent – millisecond accuracy, microsecond precision
Status code of the response
NCSA format of the start time of the request
The current time when the message to the access log is queued to be logged in normal NCSA format
Remote user according to the WebSphere Application Server specific $WSRU header
URL Path, not including the query string
Cross Component Tracing (XCT) Context ID


logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U  r:”%r” s:%s’

gave me

  • a: remote IP address
  • A: local host
  • D:225960 duration in millisecond
  • h: remote host
  • HeaderHost: the Host header
  • HeaderOrigin:- the Origin header (missing in this case)
  • m:GET request method
  • R:186930 Service time of the request from the moment the request is received until the first set of bytes of the response is sent
  • t:[27/Feb/2020:17:04:27 +0000] NCSA format of the start time of the request
  • u:colinpaice remote user
  • U:/ibmmq/console url path
  • r:”GET /ibmmq/console HTTP/1.1″ First line of the request
  • s:302 Status code of the response

There are other log formatting options available, I picked those I thought were most useful.

Note when using the MQConsole from a browser, the interface is chatty. I had 20 request to refresh one window.

Other ways of formatting the data

  • I separated each field with a ‘,’ and could read it into a spread sheet.
  • You could configure your log format string to produce the output in JSON format, to make it easier to post process.

Is mqweb secure enough? Depends on what you mean by secure enough.

When doing research into the MQ Console and REST interface I was some times surprised at how “insecure” the mqweb was, but with a bit more thought it seemed more reasonable.

One of the questions I had was, is authentication with certificates better than using userid and password?  See here.

What do I mean by secure?

You can buy a safe with a combination lock to secure your valuables.    It will not be 100% secure.   It will have a statement like “This safe will provide protection from fire for up to 1 hour”.  The number of combinations is limited, and so it will take someone some hours to try them all.   They may be lucky and guess it first go!  An insecure safe would have one combination wheel.  The more wheels, the longer it should take to find the combination.

Digital certificates are considered a good way of providing “secure” communication.  However if you are careless with your private information, then other people have easy access to your secrets.  The private information for some encryption types is based on multiplying two very large prime numbers together.  The theory being it will take you a long time to factorize  the product .  If I was a government, I would take the “book of prime numbers” and go through each number, and multiply it be each other number in the book, and generate a reverse dictionary of factor1 * factor 2 -> public key.    You need a lot of computing power as the numbers are very large, but computing power is cheap, and you only need to do it once.  Take the public key, and look up the factors.  Of course it is not that simple, you might need a dictionary for each encryption type, as well as a lot of CPU power.  These certificates give some security, but are not totally secure.

You can have locks on your house, but if you leave the doors unlocked – it is easy to break in.  People often lock the front door, but leave the back door or a window open.

Security is down to how long it will take someone to access the information.  All components have to be secure; you cannot be “half secure”.

What do I want to secure?

This is where it starts to get murky.  Some of the HTTP protocol protection depends on “state change”, so changing a field in a database is a state change, creating an object is a state change, but browsing a record is not a state change.

With the MQ REST API an HTTP GET maps to a browse;  an HTTP POST is a change or put;  and an HTTP DELETE is a destructive get.  With the standard HTTP protection POST and DELETE have more protection than GET.

I consider being able to browse a queue which has financial transaction information, as insecure.  You can obtain bank account details, which could be used in a different sort of attack.   In mqweb, the protection for browse is weaker than for POST and DELETE.

Surprise number 1 – which certificate should the browser use?

My first surprise when using digital certificates for identification was how the browser decides which certificate to use.

I said here

If client authentication is used, the certificates are read from the server’s trust store. The DN of the self signed certificates and the DN of the CA certificates are extracted and sent down to the browser.

The browser takes this list and checks through the “user” certificates in its key store, for those self signed certificates with a matching DN, or those certificates which have been signed by the CA.

The list of matching certificates is displayed for the user to select from.

From then on, communication to that https://url:port will use that certificate.  If you create a new tab within the browser instance, and work with the same site, the selected certificate will be used – without prompting.

You can only change the certificate by restarting the browser and selecting a different certificate.  You cannot logoff.

Surprise number 2 – it is easy to browse the queue with no additional security.

In an HTML page,  I used the html link <a href=”https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message “> direct link https ></a> .

If I had already accessed the server then this displayed the content of the next message on the queue, with no further security checks.

If I opened another tab within the browser, displayed the page and clicked on the link, I also got the next message content displayed.

I used a file on my laptop with the link in it – and it displayed the message content.  It was easy to create a web page in an editor and so display the content of the queue.

The CSRF checks are not applicable – as these only occur when the request is a change of state, POST, DELETE etc.

Using Curl, and changing the GET request to a DELETE I received (as expected)

{“error”: [{
  “action”: “Resubmit the request with the correct CSRF header.”,
  “completionCode”: 0,
  “explanation”: “The REST API request cannot be completed because the CSRF header was omitted.”,
  “message”: “MQWB0100E: The CSRF header ‘ibm-mq-rest-csrf-token’ was omitted from the request.”,
  “msgId”: “MQWB0100E”,
  “reasonCode”: 0,
  “type”: “rest”

This is not a Cross Origin request – as it is in one page – so the CORS checks are not performed.   Putting the backend request into a java script did cause the CORS checking as expected.

Turning off client Certificate Authentication helped

I changed the <ssl…  clientAuthenticationSupported=”true”> to “false” to use userid and password.

With my <a href=”https://localhost:9445/ib…  > ” > direct link https ></a>  I got

{"error": [{
  "action": "Provide credentials using a client certificate, LTPA security token, or username 
   and password via HTTP basic authentication header. 
   On z/OS, if the mqweb server has been configured for SAF authentication, 
   check the messages.log file for messages indicating that SAF authentication is not available. 
   Start the Liberty angel process if it is not already running. 
   You might need to restart the mqweb server for any changes to take effect.",
  "completionCode": 0,
  "explanation": "The REST API request cannot be completed because credentials were omitted from the request. 
   On z/OS, if the mqweb server has been configured for SAF authentication, 
   this can be caused by the Liberty angel process not being active.",
  "message": "MQWB0104E: The REST API request to 
  'https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message' is not authenticated.",
  "msgId": "MQWB0104E",
  "reasonCode": 0,
  "type": "rest"

I tried putting my userid and password in the URL as in
<a href=”https://colinpaice:mypassword@localhost:9445/ib…> “> direct link https ></a> . This gave me the same message.  In particular the message says Provide credentials using …  or username and password via HTTP basic authentication header.  So it needs credentials in  a header not inline.

As raw html cannot insert HTTP headers, it means this is more secure.   To insert headers you need a script, and then the CORS processing will capture it.

I used the logon url this prompted me for userid and password, then displayed the console.

So is using userid and password the right thing to use?

Yes and No.

  1. Yes – you know when you are accessing the back end, but then it logs you off, and you have to logon again after ltpaExpiration minutes.
  2. With long running displays, such as the monitors in the operations room – you do not want to be logging on every hour.  In this case certificates are the right answer.

See is it better to use certificate authentication or userid and password? for information.


Making it more secure.

I have shown that if people are authorised to process messages on a queue as part of their normal job, they could easily write a web page to browse the queue.

If the queue was access as part of a client  program accessing the queue over a channel, the end user could write or obtain a program to process the queue – but this would be harder than just writing a web page.    This is like having a combination safe with one wheel, and having multiple wheels – you need a lot more effort.

You cannot secure mqweb for message access to a subset of queues – it is all or nothing.

I would disable the end users access to message data – and just use the rest API for administration.

This means in the mqwebuser.xml the <security-role name=”MQWebUser”> … </security-role> has no users who are allowed to process sensitive queues.  You need to prevent them using curl and other tools, so

<enterpriseApplication id=”com.ibm.mq.rest”> … and <enterpriseApplication id=”com.ibm.mq.console”>…. should have no users in <security-role name=”MQWebUser”>… </security-role>

Ban the special subject.

I would also explicitly specify the groups of users – and not use the <special-subject type=”ALL_AUTHENTICATED_USERS”/>.

Best practice says not to specify user names in the security roles, but to put the user names in groups and specify the groups.   It make it much easier to manage.  For example the manager in charge of the Payroll team, can administer the PayrollGroup, and the MQ Admin team does not need to edit the mqwebuser.xml file.

mqweb- is it better to use certificate authentication or userid and password?

Like may questions the answer is it depends.

You can set up your mqweb environment

  1. Certificates are required – you cannot use userid and passwords
  2. If a valid certificate is found use that, else prompt for userid and password
  3. Do not use certificates – just use userid and passwords.

This is described in the Liberty documentation.

It says for the <ssl…> tag

  1. If you specify clientAuthentication="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake does not succeed.
  2. If you specify clientAuthenticationSupported="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.
  3. If you do not specify either clientAuthentication or clientAuthenticationSupported, or you specify clientAuthentication="false" or clientAuthenticationSupported="false", the server does not request that a client send a certificate during the handshake.

No matter how you logon, the web server creates a LTPA* cookie with encrypted information about the logon.   The mqwebuser.xml attribute ltpaExpiration says how long this cookie is valid for.  If the time is exceeded you have to logon again, and it generates a new cookie.

Logging on

Certificate authorisation

If you are using certificates, the server sends down a list of valid self signed certificates or CA certificates, and the browser compares this list with the certificates in its store.   If the certificate in the browser’s store is acceptable, it is added to a list

The list is displayed, and the end user selects a certificate.  For subsequent requests with the same  http://url:port combination, in the same or a different tab within the browser window, the same certificate is used, and the browser does not prompt.

If the ltpa cookie expires then the same certificate is used, invisibly to the end user.  A new cookie is generate and sent down.

If there are no valid certificates in the list, then depending clientAuthentication, it may prompt for a userid and password.

You cannot logoff from the page as there is no option to do so.

Userid and password.

When userid and password are passed up in a header (not in the URL as some web servers accept) the ltpa token is returned.  When the ltpa token expires, the user is prompted to re-logon.   They need to enter the userid and password again.  You can also logoff from the web page.

What are the implications of certificates and userid with password

Certificate logon

  • The web browser can logon, and although the ltpa token expires, it is automatically logged on again.   This is useful for a “big screen monitor” in the operations room, where you want the display active all day.
  • Someone can create a file with some HTML in it to browse a queue  (if authorised).  So if they are authorised to the queue for their normal job, they could easily create a web page to display the queue contents.  This could be the end user, or an evil person saying “click this link”.

userid logon

  • If the user is using the mqconsole web pages, then periodically they will get logged off – and they will have to logon again.
  • If they try to access a queue from an HTML page, they will be asked for userid and password.  Depending on what the end user is doing, this may give them notification than there is some unusual activity.  If they are hacking they will enter the userid and password anyway.
  • The person may have access to the same information without using the mqweb console interface, so there is no change to the threat level.

What do I recommend?


  • I think long running monitors need to use certificates
  • End users should logon with userid and password, with an expiry period of about an hour or less.
  • You may want to have a Certificate  Authority just for the mqweb browsers, to stop your corporate certificates from being used to access the mqweb server.

Because it is easy to create an html page and browse a queue, I would allow administration over mqrest, and not allow processing messages over REST.

As I said in another posting

  • The admin API authorisation is managed through <security-role name=”MQWebAdmin”> and <security-role name=”MQWebAdminRO”> sections in the mqwebuser.xml file.
  • The messaging API authorisation is managed through <security-role name=”MQWebUser”> sections.

To stop messaging over the rest interface, have no entries in the <security-role name=”MQWebUser”> section.  This applies to both certificates, and userid and password.

(Personally I would have called<security-role name=”MQWebUser”> as <security-role name=”MQWebMessaging”>  to make the roles clearer. 

Update from Gwydion…

The MQWebAdmin, MQWebAdminRO and MQWebUser roles can all be used for the admin REST API. That’s why MQWebUser is not called MQWebMessaging – it’s not just for messaging. The difference between them is the user ID that’s checked by the qmgr for operations performed via the REST API.
  • Operations performed by users in the MQWebAdmin* roles take place under the context of the mqweb server user ID.
  • Operation performed by users in the MQWebUser role take place under the context of the user logged into the REST API.
To stop messaging over the REST API completely, the mqRestMessagingEnabled should be set to false in the mqweb server configuration.

Can evil websites get to your mqweb – using java script to get to the backend server

With HTML and scripting people would write scripts and get you to execute them, and so access your personal information, and steal your money.  Over time security has been improved to make this harder, and now you have to explicitly say which web sites can run scripts which use the mqweb interface to access MQ to put and get messages.

One way of protecting the access is using Cross Origin Resource Sharing, or CORS.  I explained the basics of CORS here.  I struggled getting it to work with a web browser.

  • browsers have been improved and what worked last year may no longer work now, and the documentation does not reflect the newer browsers.
  • Chrome carefully changes your hand crafted HTTP headers, so what is sent up may not what you expected.

I’ll go through three examples, and then show how it gets more difficult, and what you can do to identify problems.

Note: If you use a web page from a file:// then the origin is ‘null’, and this will fail any checks in the backend, as the checks compare the passed origin to the list of acceptable urls.

I used Dev Tools in Chrome (Alt+Ctrl+i) to display information including the headers flowing.

Simple HTML

With the following within your web page,

<a href=”https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/…/message
> direct link ></a>

It issues the REST request, returns the data from the queue, and displays it.

From the headers we see

Simple HTML: Request headers

The important request headers were

And no origin header.

Simple HTML: Response headers

The response headers have

  • HTTP/1.1 200 OK
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent
  • Content-Length: 1024
  • Set-Cookie: LtpaToken2_1 ….

There are no Allow….  headers, so this indicates the response is not a valid cross origin response.   The request came from one page, so there was no cross origin request.

You can see the MQ attributes returned, ibm-mq*.

Invoking request from java script.

My HTML page had

<script src=”http://localhost:8884/src.js ” text=”text/javascript”></script >
<button onclick=”src()””>Get message</button>

I had the src() script downloaded from http://localhost:8884/src.js, and inline within the html file.  Both worked.

function src()
     method :'DELETE',
     headers: {
                'Origin': 'https://localhost:888499'
                ,'Access-Control-Request-Headers' : 'Content-Type' 
                ,'ibm-mq-rest-csrf-token' : '99'
  ) // end of fetch
   .then((response) => response.text() )
   .then(x => {document.write("OUTPUT:"+x); } 
  .catch(error =>  {console.log("Error from src:" + error);});

When the button was pressed, the script was executed,  there was an OPTIONS request (and response), and a DELETE request (and response).   It returned a message and displayed it.

In more detail, the flows were:

JavaScript OPTIONS request

The OPTIONS request had headers

This is doing a preflight check, saying it intends to issue a DELETE request from Origin  http://localhost:8889, the url of the web page.

JavaScript OPTIONS response headers

The OPTIONS response headers were the same as before but with additional ones

  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8889
  • Access-Control-Allow-Max-Age: 91
  • Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE
  • Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

This means, the script is allowed to issue a request from http://localhost:8889 using a request method in the list {GET, POST, PATCH, PUT, DELETE } and the valid headers that can be used are in the list of headers.

The Access-Control-Allow-Max-Age: 91 came from my mqwebuser.xml file, <variable name=”mqRestCorsMaxAgeInSeconds” value=”91″/>.

After this there was a DELETE request to get the message.

JavaScript DELETE request headers

JavaScript DELETE response headers

The response included the CORS headers, and included the headers from the non CORS situation

  • HTTP/1.1 200 OK
  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8889
  • Access-Control-Expose-Headers: Content-Language, Content-Length, Content-Type, Location, ibm-mq-qmgrs, ibm-mq-md-messageId, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers
  • Content-Type: text/plain; charset=utf-8
  • Content-Language: en-GB
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent
  • Content-Length: 1024
  • Set-Cookie: LtpaToken2_….

Because the Access-* headers are present, this is a CORS response.

The message content was displayed in a browser window.

Link to another page

I set up a link <a href=”http://localhost:8884/page.html”/>localhost 8884</a>  to execute a page on another web server.  When this executed, it issued the java script request as before.  The Origin was Origin: http://localhost:8884 – so the page where the script executed.

What happens if CORS is not set up?

If the http://localhost:8889 is not in the list in the mqwebuser.xml file,

No data was  displayed.   The Chrome browser Developer tools ( Alt+Ctrl+i) displays a message

Access to fetch at ‘ https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

The OPTIONS Request header has, as before

  • HTTP/1.1 200 OK
  • Host: localhost:9445
  • Access-Control-Request-Method: DELETE
  • Origin: http://localhost:8889
  • Access-Control-Request-Headers: ibm-mq-rest-csrf-token

but the OPTIONS Response header has no Access-* headers.

No DELETE request was issued.

The HTTP/1.1 200 OK is the same for all cases – it means the request was successful.

Trying to be clever

I read the documentation on the web, and much of it was very helpful, but some of it is no longer true.  It was hard to get it working, because every things has to be right for it to work.

Unsupported header.

I added an extra header, which is a valid CORS thing to do – but the back end has to support it.  With hindsight it makes no sense to add headers that will be ignored by the server.

headers: {
     'Origin': 'https://localhost:888499'
     , 'Access-Control-Request-Headers' : 'Content-Type' 
     , 'colin':'TRUE' 
     ,'ibm-mq-rest-csrf-token' : '99'

This sent up a header with

Access-Control-Request-Headers: colin,ibm-mq-rest-csrf-token

Header “colin” is not in the list of accepted headers, header ibm-mq-rest-csrf-token is in the list:

Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

The Developer tools message was the same as before,

Access to fetch at ‘https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

I removed the unsupported header and it worked.

Other headers do not work

The request header ‘Access-Control-Allow-Method’ : ‘DELETE’ is a valid header.

When this was used, the request headers included

  • Access-Control-Request-Headers: access-control-allow-method,ibm-mq-rest-csrf-token
  • Access-Control-Request-Method: DELETE

As before  Access-Control-Request-Method is not in the list of Access-Control-Allow-Headers, so the request fails.

The Access-Control-Request-Method: DELETE is not needed, as the method: DELETE defines what will be used.

Using Access-Control-Request-Headers to add more headers does not work, if the header is not in the list of valid parameters.

The true origin is used

I tried using the header ‘Origin’: ‘https://localhost:888499 ‘ – as it did with my curl – and this was ignored.  The true Origin from the web page was used (I am glad to say, otherwise it would make the whole protection scheme worthless)

Some options ignored

I found  the “fetch” options credentials:, redirect: , and cache:, were all ignored by Chrome.

Some  cookies  were used.

The LtpaToken2_… was sent up and down

Problems in mqwebuser.xml file

I misconfigured the configuration file with <variable name=”mqRestCorsMaxAgeInSeconds” value=”AA91″/>

This gave me the familair message

Access to fetch at ‘https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message ‘ from origin ‘http://localhost:8889 ‘ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

So if you get this message it is worth checking the configuration file for problems.

Can I trace it?

I used <variable name=”traceSpec” value=”*=audit:CorsService=finest”/> in the mqwebuser.xml file to provide information about the CORS processing, and what was coming in, and what was being checked.

Can evil websites get to your mqweb – understanding CORS

In the beginning was the html, and the html was good;  then we had html and scripts  which could only do things on the page, which was also good; then we had scripts which could reach out to other websites – and that’s where the problems began.  It was easy for evil developers to get you to click on an innocent looking page, which had a script which jumped into a different tab of your browser where you had your banking window open ,  or  to executed a script ; and steal all your money.

The browsers were improved to stop evil scripts from accessing a server, and then they were improved again so the server could say “stuff coming from this list of web sites is OK, I trust them.   One implementation is called CORS.  There is a good description here.   It says

Cross-Origin Resource Sharing (CORS) is a mechanism that uses additional HTTP headers to tell browsers to give a web application running at one origin, access to selected resources from a different origin. A web application executes a cross-origin HTTP request when it requests a resource that has a different origin (domain, protocol, or port) from its own.

I do not trust things that “just work”, I like to see evidence of it.   This lack of trust comes from working with a group of young testers who came to IBM Hursley to test their code.   All of their tests ran cleanly – and thought they had a few days spare to go to London sightseeing.  I stopped the server they were meant to be using (without telling them) and the tests carried on running successfully!   It turned out they had been testing their code with a dummy program acting as the server.    They removed this code, reran their tests  and most of them failed – and had to stay an extra week.

I also remember changing a config file, and being surprised when my changed worked first time.   After a cup of tea (an invaluable thinking aid) I put some spelling mistakes in the file ; and it carried on running.  Why? I was using the wrong config file.

I played with CORS and wanted to get things to fail, as well as to work.   This was a good choice, as I had many failures.

I’ll document how I got curl to work and demonstrate CORS , and document how I got a web browser to work – a real challenge

mqweb implements CORS, so you can configure mqweb to give a list of websites which may access your server.

The documentation  is not very clear.  It says

where allowedOrigins specifies the origin that you want to allow cross-origin requests from. You can use an asterisk surrounded by double quotation marks, “*”, to allow all cross-origin requests. You can enter more than one origin in a comma-separated list, surrounded by double quotation marks. To allow no cross-origin requests, enter empty quotation marks as the value for allowedOrigins.

My observations,

  • You cannot use generics, so* is the same as “*” – or allow all cross-origin requests
  • You must specify {scheme:address:port} so http or https,  the url with // at the front, and the specific port number
  • The match is an string equality test, so the case, spacing and values must be the same

How does an HTTP request work?

When you click on a web page, data is sent to the back end server.   The following data is exchanged

  • the request
  • request headers
  • your data going to the server
  • response headers
  • response data – such as the content of web page.

In more detail…


Request Headers

  • accept: for example  text/html, application/xml
  • accept-languages: en-GB
  • dnt:  1  this is “do not track me”
  • user-agent:  Chromium
  • cookie: bcookie=”….”

Response headers

  • status: 200
  • status: 200 OK
  • set-cookie ….
  • server: nginx
  • ….

Response data

This might be the web page to be displayed.  It can include script, images etc.

What is the origin of the page?

If your web page, invoked a script, for example from clicking a button, an “Origin” header is added, for example Origin: http://localhost:8884 which is the address of the web server hosting the page.   The backend server checks to see if this header is present, and looks up site in the authorised list.

If the Origin is acceptable, it sends down additional response headers (CORS Headers), so the browser (or your program) knows and can use the web site.

As part of the handshake, the browser can send up an “OPTIONS” request (instead of a get/delete etc), with a header saying the browser will be doing a GET/DELETE etc from this origin.    If there is a positive response, where the additional CORS headers are send back then the get/delete is allowed.   If the CORS headers are not present in the response, then the request will not be permitted.   This is called a preflight check – just like having your boarding pass checked at the gate before you get on the plane.

Does it work?

If there is no Origin header in the request, the backend server thinks it is all same domain, and does no CORS checks.

This CORS support is really aimed at web browsers, as the web browsers will automatically add headers for you.  If you are using curl or other tools to create your own request, you specify exactly the headers you want, so if you omit the Origins header, the server will not check.

My mqwebuser.xml file had <variable name=”mqRestCorsAllowedOrigins” value=”https://9999.0.0.1:19442,http://localhost:8884”/&gt;

So origins for https://9999.0.0.1:19442 and http://localhost:8884 are permitted.

I used a configuration file for curl (because command parameters did not work passing the headers) and had in curl.parms

cacert ./cacert.pem
cert ./colinpaice.pem:password
key ./colinpaice.key.pem
cookie cookie.jar.txt
cookie-jar cookie.jar.txt
request OPTIONS
header “Access-Control-Request-Method: DELETE”
header “Access-Control-Request-Headers: Content-Type”
header “Origin: https://9999.0.0.1:19442
header “ibm-mq-rest-csrf-token : COLINCSRF”

I used the command

curl –verbose –config curl.parms –url https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message

The headers were displayed, and I had

OPTIONS /ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message HTTP/1.1
> Host: localhost:9445
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: LtpaToken2_….
> Access-Control-Request-Method: DELETE
> Access-Control-Request-Headers: Content-Type
> Origin: https://9999.0.0.1:19442
> ibm-mq-rest-csrf-token : COLINCSRF

The response headers were

< HTTP/1.1 200 OK
< X-Powered-By: Servlet/3.1
< X-XSS-Protection: 1;mode=block
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src ‘none’; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’; connect-src ‘self’; img-src ‘self’; style-src ‘self’ ‘unsafe-inline’; font-src ‘self’
< Cache-Control: no-cache, no-store, must-revalidate
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: https://9999.0.0.1:19442
< Access-Control-Allow-Max-Age: 90
< Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE
< Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

The response header Access-Control-Allow-Origin: https://9999.0.0.1:19442 shows that requests with origin https://9999.0.0.1:19442 is acceptable.

When I used a different “Origin”,   I did not get any Access-Control-Allow-* headers.  So from the absence,  I could tell the request was not support from the different origin.

The Access-Control-Allow-Headers is a list of header names which can be sent, so the header ibm-mq-rest-csrf-token : COLINCSRF is valid, but “COLIN:value”  is not valid, because ibm-mq-rest-csrf-token  is in the Access-Control-Allow-Headers list, and COLIN is not in the list.

The Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE says my application can use any of the methods in the list.

Using a web browser.

If you use HTML in a local file, then the “Origin” is null, and so does not match any elements in the authorised list.  I had to set up my own web server – which was easy to do using Python.

Using the web page at the bottom of this posting, I pointed my web browser at the web server.   It displayed a button.  I pressed it, and it invoked a script. Using the developer mode ( Al+Ctrl +i) in Chrome could see network flows etc.

The request headers had

  • Host: localhost:9445 This is where my webserver is hosted
  • ibm-mq-rest-csrf-token : 99  I specified this header value
  • Origin: http://localhost:8884  this overrode the value I had specified in the headers.

The response headers included

  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8884
  • Access-Control-Expose-Headers: Content-Language, Content-Length, Content-Type, Location, ibm-mq-qmgrs, ibm-mq-md-messageId, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent

We can see from

  • the Access-Control-* headers we know this has validated for http://localhost:8884
  • the Access-Control-Expose-Headers we can see what headers will be accepted
  • ibm-mq-md-persistence:  persistence. For the returned messages, it was a persistent message

The web page used

<TITLE>Call a  mqweb rest API </TITLE>
  function local()
         method :'DELETE',
         headers: {
                    'Origin': 'https://localhost:888499'
                    , 'Access-Control-Request-Headers' : 'Content-Type'
                    ,'ibm-mq-rest-csrf-token' : '99'
    .then((response) => response.text() )
    .then(x => {  document.write("OUTPUT:"+x);    } )
    .catch(error => { console.log("Booo:" + error);});
</script> /head> <body> <button onclick="local()"">press me</button> </body> </html>


For information about “fetch(…,…)” see here.

For information about the “.then(…) ” see here.

Use the Liberty REST API to access the JMX data in Liberty

Rather than set up traditional JMX, where you specify the JMX port etc, you can use the REST support provided in LIberty to access the JMX data.   The rest support is easier to set up.

The Liberty documentation recommends that you do not have the native JMX support (configured in jvm.options), and the Liberty REST support for JMX configured at the same time.

The REST request to get the statistics worked and was easy to use.   I could not get the “traditional JMX interface”, such as jconsole to work with the REST interface.  See below.

Configure the server:  mqwebuser.xml

In mqwebuser.xml  add the support


Set up authorisation with



As the mqconsole and rest statistics are read only, then it may be better to set up every user as a reader-role.

As with the MQ support, it will use the userid if specified, or the Common Name from the digital certificate.

Using Curl and the rest API

I used

curl --cacert ./cacert.pem --cert-type P12 
--cert colinpaice.p12:password
-url https : //localhost:9443/IBMJMXConnectorREST/mbeans/...

Where … was  WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes/ResponseTimeDetails  . This gives the JMX statistics for the mq.console.

The data comes back as JSON (as you might expect) for example

"name": "ResponseTimeDetails",
  "value": {
    "value": {
      "count": "99",
      "description": "Average Response Time for servlet",
      "maximumValue": "3183755861",
      "mean": "1.116053166969697E8",
      "minimumValue": "1777114",
      "reading": {
        "count": "99",
        "maximumValue": "3183755861",
        "mean": "1.116053166969697E8",
        "minimumValue": "1777114",
        "standardDeviation": "4.360373971884932E8",
        "timestamp": "1580819294060",
        "total": "1.1048926353E10",
        "unit": "ns",
        "variance": "1.90128611746915776E17"
      "standardDeviation": "3.218674128849494E8",
      "total": "1.1048926353E10",
      "unit": "ns",
      "variance": "1.63102693370991648E17"

As well as the data I have covered before, you also get the time stamp value.  This is the value in milliseconds from a well known time.

I used the python to convert the timestamp (1580978634610) to a date time

import datetime
s = 1580978634610 / 1000.0
print(datetime.datetime.fromtimestamp(s).strftime('%Y-%m-%d %H:%M:%S.%f'))

to give  2020-02-04 12:28:14.060000.

Which URL to use for traditional JMX?

The IBM documentation says the url to access the JMX data using “traditional JMX” is in file /var/mqm/web/installations/Installation1/servers/mqweb/log/state/com.ibm.ws.jmx.rest.address.  For me this was service:jmx:rest://localhost:9443/IBMJMXConnectorREST .

Client set up: Using the “traditional JMX interface” did not work for me

The Configuring secure JMX connection to Liberty page says you can use this url in jconsole and other tools.  I could not get this to work.   I got  messages like Error connecting to JMX endpoint: Unsupported protocol: rest.  This page gives a lot of information on JMX, and towards the end of the second page it says Error connecting to JMX endpoint: Unsupported protocol: xxxx is likely to be a problem with the class path.  I used  -verbose=class, and did not see the jar file being loaded.

What else can you do with the REST interface?

Show what is available

You may get data like

WebSphere%3Aname%3Dcom.ibm.mq.console.javax.ws.rs.core.Application %2C type%3DServletStats

The punctuation has been “escaped” so you need to change

  • %3A- to  :
  • %3D to :
  • %2C to ,

and the string becomes the familiar

WebSphere: name=com.ibm.mq.console.javax.ws.rs.core.Application, type=ServletStats