Category: mqweb

Can evil websites get to your mqweb – understanding CORS

In the beginning was the html, and the html was good;  then we had html and scripts  which could only do things on the page, which was also good; then we had scripts which could reach out to other websites – and that’s where the problems began.  It was easy for evil developers to get you to click on an innocent looking page, which had a script which jumped into a different tab of your browser where you had your banking window open ,  or  to executed a script ; and steal all your money.

The browsers were improved to stop evil scripts from accessing a server, and then they were improved again so the server could say “stuff coming from this list of web sites is OK, I trust them.   One implementation is called CORS.  There is a good description here.   It says

Cross-Origin Resource Sharing (CORS) is a mechanism that uses additional HTTP headers to tell browsers to give a web application running at one origin, access to selected resources from a different origin. A web application executes a cross-origin HTTP request when it requests a resource that has a different origin (domain, protocol, or port) from its own.

I do not trust things that “just work”, I like to see evidence of it.   This lack of trust comes from working with a group of young testers who came to IBM Hursley to test their code.   All of their tests ran cleanly – and thought they had a few days spare to go to London sightseeing.  I stopped the server they were meant to be using (without telling them) and the tests carried on running successfully!   It turned out they had been testing their code with a dummy program acting as the server.    They removed this code, reran their tests  and most of them failed – and had to stay an extra week.

I also remember changing a config file, and being surprised when my changed worked first time.   After a cup of tea (an invaluable thinking aid) I put some spelling mistakes in the file ; and it carried on running.  Why? I was using the wrong config file.

I played with CORS and wanted to get things to fail, as well as to work.   This was a good choice, as I had many failures.

I’ll document how I got curl to work and demonstrate CORS , and document how I got a web browser to work – a real challenge

mqweb implements CORS, so you can configure mqweb to give a list of websites which may access your server.

The documentation  is not very clear.  It says

where allowedOrigins specifies the origin that you want to allow cross-origin requests from. You can use an asterisk surrounded by double quotation marks, “*”, to allow all cross-origin requests. You can enter more than one origin in a comma-separated list, surrounded by double quotation marks. To allow no cross-origin requests, enter empty quotation marks as the value for allowedOrigins.

My observations,

  • You cannot use generics, so http://127.0.0.1:* is the same as “*” – or allow all cross-origin requests
  • You must specify {scheme:address:port} so http or https,  the url with // at the front, and the specific port number
  • The match is an string equality test, so the case, spacing and values must be the same

How does an HTTP request work?

When you click on a web page, data is sent to the back end server.   The following data is exchanged

  • the request
  • request headers
  • your data going to the server
  • response headers
  • response data – such as the content of web page.

In more detail…

Request

Request Headers

  • accept: for example  text/html, application/xml
  • accept-languages: en-GB
  • dnt:  1  this is “do not track me”
  • user-agent:  Chromium
  • cookie: bcookie=”….”
  • …

Response headers

  • status: 200
  • status: 200 OK
  • set-cookie ….
  • server: nginx
  • ….

Response data

This might be the web page to be displayed.  It can include script, images etc.

What is the origin of the page?

If your web page, invoked a script, for example from clicking a button, an “Origin” header is added, for example Origin: http://localhost:8884 which is the address of the web server hosting the page.   The backend server checks to see if this header is present, and looks up site in the authorised list.

If the Origin is acceptable, it sends down additional response headers (CORS Headers), so the browser (or your program) knows and can use the web site.

As part of the handshake, the browser can send up an “OPTIONS” request (instead of a get/delete etc), with a header saying the browser will be doing a GET/DELETE etc from this origin.    If there is a positive response, where the additional CORS headers are send back then the get/delete is allowed.   If the CORS headers are not present in the response, then the request will not be permitted.   This is called a preflight check – just like having your boarding pass checked at the gate before you get on the plane.

Does it work?

If there is no Origin header in the request, the backend server thinks it is all same domain, and does no CORS checks.

This CORS support is really aimed at web browsers, as the web browsers will automatically add headers for you.  If you are using curl or other tools to create your own request, you specify exactly the headers you want, so if you omit the Origins header, the server will not check.

My mqwebuser.xml file had <variable name=”mqRestCorsAllowedOrigins” value=”https://9999.0.0.1:19442,http://localhost:8884”/&gt;

So origins for https://9999.0.0.1:19442 and http://localhost:8884 are permitted.

I used a configuration file for curl (because command parameters did not work passing the headers) and had in curl.parms

cacert ./cacert.pem
cert ./colinpaice.pem:password
key ./colinpaice.key.pem
cookie cookie.jar.txt
cookie-jar cookie.jar.txt
request OPTIONS
header “Access-Control-Request-Method: DELETE”
header “Access-Control-Request-Headers: Content-Type”
header “Origin: https://9999.0.0.1:19442 ”
header “ibm-mq-rest-csrf-token : COLINCSRF”
include

I used the command

curl –verbose –config curl.parms –url https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message

The headers were displayed, and I had

OPTIONS /ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message HTTP/1.1
> Host: localhost:9445
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: LtpaToken2_….
> Access-Control-Request-Method: DELETE
> Access-Control-Request-Headers: Content-Type
> Origin: https://9999.0.0.1:19442
> ibm-mq-rest-csrf-token : COLINCSRF

The response headers were

< HTTP/1.1 200 OK
< X-Powered-By: Servlet/3.1
< X-XSS-Protection: 1;mode=block
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src ‘none’; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’; connect-src ‘self’; img-src ‘self’; style-src ‘self’ ‘unsafe-inline’; font-src ‘self’
< Cache-Control: no-cache, no-store, must-revalidate
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: https://9999.0.0.1:19442
< Access-Control-Allow-Max-Age: 90
< Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE
< Access-Control-Allow-Headers: Accept-Charset, Accept-Language, Authorization, Content-Type, ibm-mq-rest-csrf-token, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers

The response header Access-Control-Allow-Origin: https://9999.0.0.1:19442 shows that requests with origin https://9999.0.0.1:19442 is acceptable.

When I used a different “Origin”,   I did not get any Access-Control-Allow-* headers.  So from the absence,  I could tell the request was not support from the different origin.

The Access-Control-Allow-Headers is a list of header names which can be sent, so the header ibm-mq-rest-csrf-token : COLINCSRF is valid, but “COLIN:value”  is not valid, because ibm-mq-rest-csrf-token  is in the Access-Control-Allow-Headers list, and COLIN is not in the list.

The Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE says my application can use any of the methods in the list.

Using a web browser.

If you use HTML in a local file, then the “Origin” is null, and so does not match any elements in the authorised list.  I had to set up my own web server – which was easy to do using Python.

Using the web page at the bottom of this posting, I pointed my web browser at the web server.   It displayed a button.  I pressed it, and it invoked a script. Using the developer mode ( Al+Ctrl +i) in Chrome could see network flows etc.

The request headers had

  • Host: localhost:9445 This is where my webserver is hosted
  • ibm-mq-rest-csrf-token : 99  I specified this header value
  • Origin: http://localhost:8884  this overrode the value I had specified in the headers.

The response headers included

  • Access-Control-Allow-Credentials: true
  • Access-Control-Allow-Origin: http://localhost:8884
  • Access-Control-Expose-Headers: Content-Language, Content-Length, Content-Type, Location, ibm-mq-qmgrs, ibm-mq-md-messageId, ibm-mq-md-correlationId, ibm-mq-md-expiry, ibm-mq-md-persistence, ibm-mq-md-replyTo, ibm-mq-rest-mft-total-transfers
  • ibm-mq-md-expiry: unlimited
  • ibm-mq-md-messageId: 414d5120514d412020202020202020204c27165d04a98a25
  • ibm-mq-md-persistence: persistent

We can see from

  • the Access-Control-* headers we know this has validated for http://localhost:8884
  • the Access-Control-Expose-Headers we can see what headers will be accepted
  • ibm-mq-md-persistence:  persistence. For the returned messages, it was a persistent message

The web page used

<HTML>
<HEAD>
<TITLE>Call a  mqweb rest API </TITLE>
<script>
  function local()
  {
    fetch("https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message" 
      {
         method :'DELETE',
         headers: {
                    'Origin': 'https://localhost:888499'
                    , 'Access-Control-Request-Headers' : 'Content-Type'
                    ,'ibm-mq-rest-csrf-token' : '99'
                  }
     }
    )
    .then((response) => response.text() )
    .then(x => {  document.write("OUTPUT:"+x);    } )
    .catch(error => { console.log("Booo:" + error);});
  }
</script> /head> <body> <button onclick="local()"">press me</button> </body> </html>

 

For information about “fetch(…,…)” see here.

For information about the “.then(…) ” see here.

Error connecting to JMX endpoint: Unsupported protocol: rest

I got this error message when I was trying to use JMX into the WLP web server, when using the restConnector-2.0 interface in Liberty.

The documentation was not that helpful.  Oneproblem was I suffered from the curse of cut and paste, and used a Windows environment variable %JAVA_HOME%  instead of using the Linux $JAVA_HOME.  Another problem was caused documentation saying add the jar to the class path, and then the class path was ignored.

True documentation.

When you use the  -jar option, the JAR file is the source of all user classes, and other user class path settings are ignored.  See here.

Unhelpful documentation

Configuring secure JMX connection to Liberty   says

jconsole -J-Djava.class.path=...;%WLP_HOME%/clients/restConnector.jar

This was for Windows – on Linux it would be $WLP_HOME – except on my system $WLP_HOME was not set.

jconsole

Some of the jar files are in $JAVA_HOME, you can use the environment variable, or specify it yourself.  Note %JAVA_HOME% is windows, so be careful when you use cut and paste.

  • jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar
  • jconsole -J-Djava.class.path=/usr/lib/jvm/java-8-openjdk-amd64/lib/jconsole.jar:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/opt/mqm/web/clients/restConnector.jar

If the window displays “Secure connection failed” restart jconsole and use the -debug option.   For me this gave  “java.io.IOException jmx.remote.credentials not provided. Set to a String[2] {user,password}”.  I entered my userid and password, and this connected successfully.

I could not connect using my .jar file.

I was using JMXQuery to extract the data.  I was getting the Error connecting to JMX endpoint: Unsupported protocol: rest message.
Adding the jar file to my class path did not help, as the class path is ignored when using java the -jar parameter.

How to fix it

There are two ways of fixing this.

  1. Put the required jar file in the extensions path, not the class path
  2. Use the java -classpath…  instead of specifying java -jar

1. You need to have the jar for in the extensions path, not in the class path.

See How classes are found  in the Java documentation.  It says

  • Bootstrap classes – Classes that comprise the Java platform, including the classes in rt.jar and several other important jar files.
  • Extension classes – Classes that use the Java Extension mechanism. These are bundled as .jar files located in the extensions directory.
  • User classes – Classes defined by developers and third parties that do not take advantage of the extension mechanism. You identify the location of these classes using the -classpath option on the command line (the preferred method) or by using the CLASSPATH environment variable.

There is a system property java.ext.dirs  which gives the location of the Extension classes. On my system this was

/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext

  • The /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext: is java dependent. You should not put your files in this directory
  • /usr/java/packages/lib/ext This is for “user” extensions.

The directory /usr/java/packages/lib/ext did not exist on my system, so I had to do, create it, copy the web connection jar file to it, and grant permissions on it.

  • sudo mkdir -p /usr/java/packages/lib/ext
  • sudo cp /opt/mqm/web/clients/restConnector.jar /usr/java/packages/lib/ext/
  • sudo chmod 444 /usr/java/packages/lib/ext/restConnector.jar

2. Use a -classpath – not a -jar

In a .jar file there is a META-INF/MANIFEST.MF file which includes information on the entry point.

Manifest-Version: 1.0
Main-Class: src.Client

Instead of using the -jar option to point to the jar, you can use the -classpath to point to the jar and explicitly specify the entry point.  For example

java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar  
src/Client
service:jmx:rest://localhost:9443/IBMJMXConnectorREST
  • java -cp ./Client.jar:/opt/mqm/web/clients/restConnector.jar –  use the classpath option, and specify the needed jar files.  Client.jar is my program.  /opt/mqm/web/clients/restConnector.jar  is the Liberty provided jar.
  • src/Client – the “entry point” class to use
  • service:jmx:rest://localhost:9443/IBMJMXConnectorREST – the url to use.
  • note the absence of a -jar option.

In the manifest of the Client.jar file it had Main-Class: src.Client.   This is for the the src/Client.java source file.   This ties up with the src/Client.class as seen when you use the command jar -tvf Client.jar .

Use the Liberty REST API to access the JMX data in Liberty

Rather than set up traditional JMX, where you specify the JMX port etc, you can use the REST support provided in LIberty to access the JMX data.   The rest support is easier to set up.

The Liberty documentation recommends that you do not have the native JMX support (configured in jvm.options), and the Liberty REST support for JMX configured at the same time.

The REST request to get the statistics worked and was easy to use.   I could not get the “traditional JMX interface”, such as jconsole to work with the REST interface.  See below.

Configure the server:  mqwebuser.xml

In mqwebuser.xml  add the support

<featureManager>
  <feature>restConnector-2.0</feature>
</featureManager>

Set up authorisation with

<administrator-role>
   <user>colinpaice</user>
   <group>MQADMIN</group>
</administrator-role>

<reader-role>
  <user>John</user>
</reader-role>

As the mqconsole and rest statistics are read only, then it may be better to set up every user as a reader-role.

As with the MQ support, it will use the userid if specified, or the Common Name from the digital certificate.

Using Curl and the rest API

I used

curl --cacert ./cacert.pem --cert-type P12 
--cert colinpaice.p12:password
-url https : //localhost:9443/IBMJMXConnectorREST/mbeans/...

Where … was  WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes/ResponseTimeDetails  . This gives the JMX statistics for the mq.console.

The data comes back as JSON (as you might expect) for example

"name": "ResponseTimeDetails",
  "value": {
    "value": {
      "count": "99",
      "description": "Average Response Time for servlet",
      "maximumValue": "3183755861",
      "mean": "1.116053166969697E8",
      "minimumValue": "1777114",
      "reading": {
        "count": "99",
        "maximumValue": "3183755861",
        "mean": "1.116053166969697E8",
        "minimumValue": "1777114",
        "standardDeviation": "4.360373971884932E8",
        "timestamp": "1580819294060",
        "total": "1.1048926353E10",
        "unit": "ns",
        "variance": "1.90128611746915776E17"
      },
      "standardDeviation": "3.218674128849494E8",
      "total": "1.1048926353E10",
      "unit": "ns",
      "variance": "1.63102693370991648E17"
    ...

As well as the data I have covered before, you also get the time stamp value.  This is the value in milliseconds from a well known time.

I used the python to convert the timestamp (1580978634610) to a date time

import datetime
s = 1580978634610 / 1000.0
print(datetime.datetime.fromtimestamp(s).strftime('%Y-%m-%d %H:%M:%S.%f'))

to give  2020-02-04 12:28:14.060000.

Which URL to use for traditional JMX?

The IBM documentation says the url to access the JMX data using “traditional JMX” is in file /var/mqm/web/installations/Installation1/servers/mqweb/log/state/com.ibm.ws.jmx.rest.address.  For me this was service:jmx:rest://localhost:9443/IBMJMXConnectorREST .

Client set up: Using the “traditional JMX interface” did not work for me

The Configuring secure JMX connection to Liberty page says you can use this url in jconsole and other tools.  I could not get this to work.   I got  messages like Error connecting to JMX endpoint: Unsupported protocol: rest.  This page gives a lot of information on JMX, and towards the end of the second page it says Error connecting to JMX endpoint: Unsupported protocol: xxxx is likely to be a problem with the class path.  I used  -verbose=class, and did not see the jar file being loaded.

What else can you do with the REST interface?

Show what is available

You may get data like

WebSphere%3Aname%3Dcom.ibm.mq.console.javax.ws.rs.core.Application %2C type%3DServletStats

The punctuation has been “escaped” so you need to change

  • %3A- to  :
  • %3D to :
  • %2C to ,

and the string becomes the familiar

WebSphere: name=com.ibm.mq.console.javax.ws.rs.core.Application, type=ServletStats

Setting up Liberty(as used in mqweb) to use native JMX

Setting up the server side is well documented in the Oracle Monitoring and Management Using JMX Technology documentation.  Using it from a client is not so well documented.

Server set up

The  Liberty jvm.options file needs parameters.  Note the port=9010 is used  by clients accessing the data.

To provide insecure access from only the local machine

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9010 
-Dcom.sun.management.jmxremote.local.only=true 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false

To provide securer access using TLS

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true

# the following statements point to the same key store as
# used by mqweb server.   This could be different.
-Djavax.net.ssl.keyStoreType=PKCS12
-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/mqweb.p12
-Djavax.net.ssl.keyStorePassword=password
# the following statements point to the same trust store as
# used by mqweb server.   This could be different.
# if you used self signed certificates you could have a keystore
# just for the JMX users
-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks
-Djavax.net.ssl.trustStorePassword=zpassword
-Djavax.net.ssl.trustStoreType=JKS

# The following defines the userid and password file
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.password.file=/home/colinpaice/ssl/ssl2/jmxremote.password

# The following defines the access a userid can have
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.access.file=/home/colinpaice/ssl/ssl2/jmxremote.access

jmxremote.password has

# specify actual password instead of the text password
monitorRole password
controlRole password

jmxremote.access has

# The "monitorRole" role has readonly access.
# The "controlRole" role has readwrite access.
monitorRole readonly
controlRole readwrite

Client set up

jconsole

You cannot pass a  userid and password when the jconsole command, so you have to disable authentication in the jvm.options file

-Dcom.sun.management.jmxremote.authenticate=false

The parameters for jconsole have  -J on them, as in -J-D…. .  jconsole removes the -J and uses the rest of the parameters when invoking the JVM.

I could not get jconsole to recognize a config file using the -J-Dcom.sun.management.config.file = /path/to/jmxremote.properties , so I wrote a bash script to make it easier to change parameters.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file"
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-J-Djavax.net.debug=ssl:handshake"
jconsole -J$ssl1 -J$ssl2 -J$ssl3 -J$ssl4 -J$ssl5 -J$ssl6 -J$ssl7 $ssl8 127.0.0.1:9010

The option “-J-Djavax.net.debug=ssl:handshake” gives a verb verbose trace of the ssl flows for the handshake.

The option -J-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file enables the jconsole logging.  I did not find the output very useful.

There is information the logger in general here,  and on the file logger, here.

The logging.file had

Logging.properties

handlers= java.util.logging.FileHandler
// , java.util.logging.ConsoleHandler2

java.util.logging.FileHandler.pattern=/home/colinpaice/JMXQuery/java/log.%g.file
java.util.logging.FileHandler.limit=50000
java.util.logging.FileHandler.count=2
java.util.logging.FileHandler.level=ALL
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

// .level = INFO
// logger.level = FINEST
.level = FINEST
// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose...
javax.level= FINER
javax.management.level = FINER
javax.management.remote.*     = FINER 
javax.management.remote.level = FINER
javax.management.remote.misc.level  = FINER
javax.management.remote.rmi.level= FINER

Using jmxquery

I used a bash shell script to run the command, as it was easier to manage, and I could not find a way of having the java system properties in a file.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ibmsys1.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"les
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file" 
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-Djavax.net.debug=ssl:handshake"
jar="-jar JMXQuery.jar"
user="-username monitorRole -password password"
url="-url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi"
parms=" -q   WebSphere:*  -count 2 -every 2"
java $ssl1 $ssl2 $ssl3 $ssl4 $ssl5 $ssl6 $ssl7 $ssl8 $jar $url $user $parms

 

Accessing JMX data in Liberty server, securely.

I thought  I would complete the work I did with using JMX in the mqweb server.   It was another example of Hofstadter’s Law:

It always takes longer than you expect, even when you take into account Hofstadter’s Law.

I spent a lot of time looking for things on the web, expecting them to be obvious, only to find that the things do not behave as expected.  I could not find them, because they were not there.  For example I expected to be able to configure the JMX server to use my OS userid and password.  I could have a file with userids and passwords, or lookup in LDAP, but not my normal userid.

Getting started

I found there are two ways of getting the JMX data from the mqweb server.

  1. Use of the native JMX support
  2. Using the Liberty REST API

I think the REST API is easier to set up and is more secure.

I’ll document a high level overview of the two approaches, and how to configure them

Overview of using the native JMX support.

To use this, you configure parameters in the jvm.options file, including a port solely for JMX.

You can use TLS certificates to set up a secure link between the client and the server.

You can decide if you want to logon with userid and password.  If you do you can set up

  1. A file with userids and passwords; and a file with userids and permitted access.   The documentation talks about userids like monitorRole and controlRole.   You have to put a process in place to periodically change these passwords.
  2. Use and LDAP server to do userid validation and to get the access.
  3. I could not find how to use your operating system userid and password for authentication.
  4. I could not find how to use the DN as authorization.

If your certificate is valid (either because it is signed by a CA, or there is a copy of a self signed certificate in the trust store), this is good enough for the checking.   You can enable userid and password checking, but this solution feels weak, as you have to do extra work to manage it properly;  you do not have a single signon.  Not all tools support using userid and password, for example I could not pass userid and password on the jconsole command.

Overview of Using the Liberty REST API

As with the MQ REST API you can issue an HTTP request and get data back.  See here.  For example

curl –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password https://localhost:9443/IBMJMXConnectorREST/mbeans/WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes=*

There is a small amount of configuration you need to do – less than with the native JMX support.  The data comes back as JSON (as you might expect) and also includes a time stamp, which is very useful when post processing.

You define <administrator-role><user>..</user></administrator-role>  in a similar way to setting up authorisation for mqconsole and mqrest.  It takes the cn= value from your certificate as the userid, so you can give individual access.

“Securely” is a good laugh.

There are different levels of (in)security.

If you are using the native JMX support

  • You can have no passwords or access checks needed.  The data is read only, and is not sensitive.
  • You can set up userid(s) and passwords in a file
  • You cannot use the operating system userid and password
  • You can use LDAP to check the userid and password, and get the role for that userid
  • You can use TLS, so anyone with a valid certificate can access the data
  • You can use TLS and use the userid and password in a file to determine access
  • You can use TLS and LDAP to get the role for that certificate

If you are using the WLP REST support

  • You can specify a userid and password
  • You can use a certificate, and the Common Name is used as the userid
  • You can specify in the configuration file, what access userids, or groups have

You can use TLS to protect your communications to and from the server.

Java leaks passwords

You need to be aware that your client machine may leak information.  For example I ran a  Java program to issue JMX requests from a script.

I could use the linux command ps -ef to display information about my request

ps -ef |grep JMX

gave me

colinpa+ 1871 1870 79 10:27 pts/2 00:00:01 java …  -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12 -Djavax.net.ssl.keyStorePassword=password …  -username monitorRole -password password …

This exposed the password to my keystore and password to my userid!  I could not find a way of having all these java system parameters in a file.

I found export JAVA_TOOL_OPTIONS=”-D…”  and this get picked up, but then java displays the variables as in Picked up JAVA_TOOL_OPTIONS: …

jconsole

Some programs have been designed to protect information for example jconsole you can put your system properties in a file

-J-Dcom.sun.management.config.file=ConfigFilePath

and so keep your parameters secure, but I could not get this to work.

Curl can be configured not to display parameters

With curl you have a command like

curl -n –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password

which gives away your password.  If you do not specify it inline, you get prompted for it.

You can put your parameters in a config file, for example curl.config,

cacert ./cacert.pem 
cert ./colinpaice.pem:password 
key colinpaice.key.pem 
cookie cookie.jar.txt 
cookie-jar cookie.jar.txt 
url https://127.0.0.1:9443/ibmmq/rest/v1/login

and use

curl –config curl.config

Easy!

Protecting key files

It is important to protect the certificate file (with the important private key) so it is accessible by just the owner.  The linux command  ls -ltr colinpaice.p12 gives

-rw------- 1 colinpaice colinpaice 4146 Jan 31 17:56 colinpaice.p12

Of course anyone with super user authority has access to this file!

mqweb – displaying the secret statistics

Yes, mqweb does provide statistics; through the standard JMX interface provided as part of the base Liberty function.  I expect most people do not know they are available.   The data gets less useful over time, for example you get the “average time” since the mqweb started, rather than the last minute.  See here on how to extract useful information from the data, and show useful averages.

You need in mqwebuser.xml .

<featureManager>
<feature>monitor-1.0</feature>
</featureManager>

and in jvm.options

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

These options should be suitable in your test environment.  You will want to change them for production.

You need the port number (9010 in the above example) when you extract jmx data.

How do you display the data?

For a quick sniff, (no good for extracting data and plotting charts)  you can use jconsole.  Use remote connection localhost:9010 .  it does not display all of the data.

I found jmxquery very useful.  I updated the github version to fix a bug which caused a loop.  See here.

The query is

java -jar JMXQuery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi -q ‘WebSphere:*’

You get data on

  • WebSphere:type=JvmStats
  • WebSphere:type=ThreadPoolStats,name=Default Executor
  • WebSphere:type=ServletStats,name=com.ibm.mq.console.javax.ws.rs.core.Application/AppName (String) = com.ibm.mq.console
  • WebSphere:type=ServletStats,name=com.ibm.mq.rest.javax.ws.rs.core.Application/AppName (String) = com.ibm.mq.rest

To get ‘console’  and ‘ResponseTime’ data I used

java -jar JMXQuery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi 
-q 'WebSphere:*' 
-count 20 -every 60 
|grep --line-buffered console
|grep --line-buffered ResponseTime
|python3 mqweb.py
  • Where
  • java -jar JMXQuery.jar – invoke the program
  • -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi – with this url and the above port number from the jvm.options
  • -q ‘WebSphere:*’ – give me only data for this components
  • -count 20 -every 60 – my extensions giving a record every 60 seconds, and doing 20 of them
  • |grep –line-buffered console – only pull out the console records, ( ignore the ‘rest’ records).  The –line-buffered tells grep to flush it immediately
  • |grep –line-buffered ResponseTime – only interested in this detailed level
  • |python3 mqweb.py – pass it into the python program.  This calculates the delta between records and prints out the count and mean value for that interval

If you are collecting data in real time from a stream, you need to ensure any processing is unbufferred.  Often the default behavior is to accumulate the data in a big buffer, and write the buffer when it is full.  Check any filters you use, for example grep –line-buffered.

One line of output from the JMXQuery program for the console activity is

WebSphere:type=ServletStats,name=com.ibm.mq.console.javax.ws.rs.core.Application/ResponseTimeDetails/count (Long) = 5

JvmStats

This comes under WebSphere:type=JvmStats/  See here.

  • FreeMemory (Long) = 13350536
  • ProcessCPU (Double) = 0.658040027605245
  • UsedMemory (Long) = 75131256
  • GcTime (Long) = 539
  • UpTime (Long) = 1048277
  • GcCount (Long) = 118
  • Heap (Long) = 88473600
  • FreeMemory (Long) = 13350536
  • ProcessCPU (Double) = 0.658040027605245
  • UsedMemory (Long) = 75131256
  • GcTime (Long) = 539
  • UpTime (Long) = 1048277
  • GcCount (Long) = 118

ThreadPoolStats

This comes under WebSphere:type=ThreadPoolStats,name=Default Executor/

  • PoolSize (Integer) = 8
  • ActiveThreads (Integer) = 2
  • PoolName (String) = Default Executor

See here.

com.ibm.mq.console and  com.ibm.mq.rest

The data is similar between them.  One has name=com.ibm.mq.console.javax.ws.rs.core.Application, the other has name=com.ibm.mq.rest.javax.ws.rs.core.Application

The data (in italics) with my comments in plain font are

  • AppName (String) = com.ibm.mq.console
  • RequestCountDetails/currentValue (Long) = 116
  • RequestCountDetails/description (String) = This shows number of requests to a servlet
  • RequestCountDetails/unit (String) = ns – this looks like a bug as it is a count not nanoseconds
  • RequestCount (Long) = 116
  • ServletName (String) = javax.ws.rs.core.Application
  • ResponseTimeDetails/count (Long) = 116
  • ResponseTimeDetails/description (String) = Average Response Time for servlet
  • ResponseTimeDetails/maximumValue (Long) = 3060146565 – in nanoseconds ( see below for the unit)
  • ResponseTimeDetails/mean (Double) = 8.796846855172414E7 – in nanoseconds
  • ResponseTimeDetails/minimumValue (Long) = 793871 – in nanoseconds
  • ResponseTimeDetails/standardDeviation (Double) = 4.198572684166255E8
  • ResponseTimeDetails/total (Double) = 1.0204342352E10 – in nanoseconds – used in calculations
  • ResponseTimeDetails/unit (String) = ns  – this is the units.  ns is nanoseconds
  • ResponseTimeDetails/variance (Double) = 1.64064538075292096E17 –  used in calculations
  • Description (String) = Report Servlet Stats for specified Servlet and application.
  • ResponseTime (Double) = 8.796846855172414E7 – same as the ResponseTimeDetails

So we can see that there were

  • 116 console requests since the mqweb server was started
  • the units are ns (nano seconds)
  • the console requests taking an average of 8.796846855172414E7  nanoseconds, 0.0879 seconds with
  • a standard deviation of  4.136787844033974E7 – nanoseconds = 0.04198 seconds
  • the maximum value was 3 060 146 565 ns = 3.060 seconds
  • the minimum time was  793 871 ns or 0.000793 seconds

Some other data, showing how it changed over time

DataValuesLater valuesMuch later values
Number of requests82359022920
Average (seconds)0.1580.01080.0099
Standard deviation (seconds)0.4870.0340.035
Maximum (seconds)2.32.32.3
Minimum(seconds)0.0010.00010.0001

Notes:

  • There is data only once a request has been processed, so if you have not run a rest request, there will be no JMX data for rest activity.
  • These values are from start of the mqweb server. I did not see them reset, so you could have a data for a whole week or more.
  • The maximum was from the first requests to run.  I expect this includes the “warm up” costs,  of loading the code and JITing it.
  • The average values are from the start, so will be impacted by peaks and troughs.

For the each mqconsole window, there are two console counts every 10 seconds.  Any charts are refreshed every 10 seconds, so  I think this is a “I am still here, please send me any data you have for me”.

Data for rest

I started my mqweb server, and ran a python program which opened a connected and got three messages.

  • Maximum time 0.3486
  • Minimum time 0.0026
  • Calculate the other one 0.0028

Because the first request takes a long time, you can adjust for this in your calculations to get a truer mean.

For example

I reran the script and processed 100 messages.  The average time of these was 0.003 seconds.

  • Maximum 0.3486
  • Mean 0.00637
  • Count 103
  • Total 0.656

The calculations are

  • Mean * count =  0.656 (which matches Total as expected)
  • Subtract the maximum, first time value 0.656 – 0.349 = 0.307
  • Calculate the improved mean value ignoring the first value,  0.307 /(103 -1) = 0.003

So the adjusted mean time is 0.003 seconds – compared to the 0.006 which the JMX stats report.

 

Getting useful information out of JMX data

The data coming from Liberty WebServer through the JMX interface  provides some data, but it is not very useful, and it may become inaccurate over time.

I’ll cover

  1. Getting a useful mean value
  2. Getting a more accurate long term mean
  3. Data gets more inaccurate over time
  4. Standard deviation (this may only be of interest to a few people)

For example from JMX, the reported  mean time for mqconsole transactions  was 9.9 milliseconds – this is for all requests since the mqweb server was started.   Over the previous minute the average measured time, for a 10 second period was 7, or 8 milliseconds, so well below the 9.9 reported.

This is because the mean time includes any initial start up time.   The maximum transaction time, at the start of the run, was over 2 seconds.   This will bias the mean.

You can process the data to extract some useful information, and I show below how to get out useful response time values.

You get the following data (and example values) from mqweb through the JMX interface.

ResponseTimeDetails/count (Long) = 20
ResponseTimeDetails/description (String) = Average Response Time for servlet
ResponseTimeDetails/maximumValue (Long) = 3060146565  – in nanoseconds (see below for the unit)
ResponseTimeDetails/mean (Double) = 4.336789965E8 – in nanoseconds
ResponseTimeDetails/minimumValue (Long) = 2474556 – in nanoseconds
ResponseTimeDetails/standardDeviation (Double) = 9.089057964078983E8 – in nanoseconds
ResponseTimeDetails/total (Double) = 8.67357993E9 – used in calculations
ResponseTimeDetails/unit (String) = ns – the unit ns = nanoseconds
ResponseTimeDetails/variance (Double) = 8.319076762335653 – used in calculations

Getting a useful mean value

To produce these numbers, the count of the response times and the sum of the transaction response times are accumulated within the Liberty Server.  To calculate the mean value you calculate sum/count.   This gives you the overall mean time.  If you obtain the data periodically you can manipulate the data to provide some useful statistics.

Let the count and sum at time T1 be Count1, and Sum1, and at time T2 Count2, and Sum2.
You can now calculate (Sum2- Sum1)/(Count2 – Count1) to get the average for that period.  For example the reported mean was 0.016 ms, but the calculated value gave 0.008 ms.  You can also calculate (Count2 – Count1)/(T2-T1) to give a rate of requests per second.   These are much more useful than the raw data.  I suggest collecting the data every minute.

Getting a more accurate long term mean

The first rest request and console request take a long time because the java code has to be loaded in etc.  In one test the duration of the first request was 50 times the duration of the other requests.  A better “mean” value is to ignore the duration of the first request.

The improved mean is (JMX mean * JMX count  – JMX Maximum value) /(JMX Count-1), or JMXMean – (JMXMaximum/JMXCount) .

Data gets more inaccurate over time

The total time is stored as a floating point double.  As you add small numbers to big numbers, the small numbers may be ignored.  Let me try to explain.

Consider a float which has precision of 3, so you could have 1.23 E2 = 1230.  Add 1 to this, and you get 1231 which is 1.23 E2 with a precision of 3 – the number we started with.

The total time is in nanoseconds so 1 second is stored as 1.0 E9.  With 100 of these a second, and 1 hour( 3600 seconds) for 100 hours is 360,000,000, or 3.6 E8 seconds.  * 1.0 E9 nano seconds. = 3.6E17 nano seconds.   The precision  of most float numbers is 16, so with this 3.6 E17 we have lost the odd nanosecond.    I do not think this is a big enough problem to worry about – unless you are running for years without restarting the server.

The variance uses the time**2 value.  So with the maximum time above 599482097 nano seconds. Time **2 is 3.593787846×10¹⁷ and you are already losing data.  I expect the variance will not be used very often, so I think this can be ignored.

If the times were in microseconds instead of nano seconds, this would not be a problem.

Getting a useful standard deviation (this may only be of interest to a few people)

The standard deviation gives a measure of the spread of the data, a small standard deviation means the data is in a narrow band, a larger standard deviation means it is spread over a wider band.  Often 95% of the values are within plus or minus 3 * standard deviations from the mean, so anything outside this range would considered an outlier, or unusual statistic.

I set up some data, a mixture of  10  values 9, 10, 11,  the standard deviation was 0.73.    I changed one value to 20, and the standard deviation changed to 3.3, indicating a wide spread of values.

With a mixture of 100 values 9,10,11, the standard deviation was 0.71.   I changed one value to 20, and the standard deviation changed to 1.2, so a much smaller value, most of the data was still around 10 – just one outside the range.

With a lot of data, the standard deviation converges on a value, and “unusual” numbers make little difference to the value.  I think that the standard deviation over an extended period is not that useful, especially if you get periodic variations such as busy time, and overnight.

You calculate the standard deviation as the square root of the variance.   The variance is (Sum of (values**2) – (mean ** 2)) /number of measurements.

With data

ResponseTimeDetails/count (Long) = 203
ResponseTimeDetails/mean (Double) = 6420785.187192118 – nanoseconds
ResponseTimeDetails/variance (Double) = 1.7113341125320868E15 – used in calculations

Variance  = 1.7113341125320868E15 =  ( (Sum of (values**2) – (6420785.187192118 ** 2)) / 203

So (Sum of (values**2)) =   3.474420513264337e+17

You can now do the sums as with the mean, above:

At time T1, the ssquares1 is the sum of (values**2)   at time T2, the ssquares2 is the sum of (values**2).

You can now calculate ssquares2 – ssquares2, and used that to estimate the variance, and so the standard deviation of the data in the range T1 to T2, I’ll leave the details to the end user.

For the advanced user,  you can use the mean for the interval – rather than the long term mean.  Good luck.

 

mqweb error messages and symptoms of TLS setup problems

I deliberately caused TLS set up errors, and noted the symptoms.  Ive recorded them below; the article is not meant to be read, but indexed by search engines.

There are three sections

  1. Problems with server certificates
  2. Problems with the client certificate
  3. Chrome messages, and possible causes of the problems.

The mqweb messages.log reported problems that the mqweb server saw.   For me this was in file /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log

Problems with the server certificate

Problem: mqwebuser.xml serverKeyAlias name not in the keystore

This can be caused by the certificate being in the keyring but not visible, or cannot be validated.

The RACF command RACDCERT LISTRING(KEYRING) ID(IZUSVR) will list the contents of the keyring. For example it gives ZZZZ ID(START1).  You can then use

RACDCERT LIST(LABEL(‘ZZZZ’ )) ID(START1).   This gives output including

Status: TRUST
Start Date: 2020/12/17 00:00:00
End Date: 2021/12/17 23:59:59

Check it has STATUS:TRUST and the dates are valid.  If you make a change, check it afterwards.  Several times I got the change wrong!

Check the CA for the certificate is in the keystore; you need the key, and the CA in the keystore.

Message log:

  • Failed validating certificate paths
  • E CWPKI0024E: The certificate alias mqweb specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore /home/colinpaice/ssl/ssl2/mqweb.p12.
  • I FFDC1015I: An FFDC Incident has been created: “com.ibm.wsspi.channelfw.exception.ChannelException: java.lang.IllegalArgumentException: CWPKI0024E: The certificate alias mqweb specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore /home/colinpaice/ssl/ssl2/mqweb.p12. com.ibm.ws.channel.ssl.internal.SSLConnectionLink 238″ at ffdc_….

curl:

* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* curl (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443
* stopped the pause stream!
* Closing connection 0

chrome:

This site can’t be reached.  ERR_CONNECTION_CLOSED

Problem:  The host certificate is self signed and not in the client keystore

Problem:  The host certificate is signed but the signer certificate is not in  the client keystore

Message log:

Nothing.

curl:

* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate

Chrome: in browser

NET::ERR_CERT_AUTHORITY_INVALID

Click on the Not Secure in the url, to display the certificate which was sent down.

If it is signed, make a note of the “issued by” Common Name(CN), and the  Organisation(0) and look up the value of Organisation in the “Authorities” section of “Manage Certificates”.

Use the chrome url chrome://settings/certificates .  Authorities tab

  1. if it is not present, import it
  2. it it is present and UNTRUSTED, edit it, and tick the “Trust this certificate for identifying web sites”

Chrome log:

ERROR:cert_verify_proc_nss.cc(1011)] CERT_PKIXVerifyCert for localhost failed err=-8179

From here  -8179 is Peer’s certificate issuer is not recognized.

Firefox:  browser

SEC_ERROR_UNKNOWN_ISSUER

Action import the CA signing certificate into the keystore and make it trusted.

Problem: curl: The host certificate is self signed and you use the –insecure option

curl

* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=GB; O=aaaa; CN=testuser
* start date: Jan 20 17:39:37 2020 GMT
* expire date: Feb 19 17:39:37 2020 GMT
* issuer: C=GB; O=aaaa; CN=testuser
* SSL certificate verify result: self signed certificate (18), continuing anyway.

Problem: Chrome:  The host certificate is self signed and is not trusted

Chrome browser

This site can’t be reached
localhost unexpectedly closed the connection.
ERR_CONNECTION_CLOSED

Debugging

  • I could find nothing that told me what certificate was being used.  The Chrome network trace just gave “net_error = -100 (ERR_CONNECTION_CLOSED)“.
  • Use certutil -L $sql  to list the contents of your browsers keystore.   The certificate needs “P,…” permissions.
  •  Or use the chrome url chrome://settings/certificates  and display “your certificates”. Pick the likely one, if it says “UNTRUSTED” then this may be the problem.   View the certificate, and check it, for example under details, there may be a comment describing its use.
  •  Defined the server certificate as trusted using certutil -M $sql -n name -t “P,,” 
  • Restart the web browser.

Problem: The  CA signer server certificate had the wrong subjectAltName

curl:

* subjectAltName does not match 127.0.0.1
* SSL: no alternative certificate subject name matches target host name ‘127.0.0.1’

Chrome:

NET::ERR_CERT_COMMON_NAME_INVALID
From the “Not Secure” in front of the URL, display the certificate, and check the extenstions, especially Certificate Subject Alternative Names.

Chrome log:

ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -200
From here -200 is  CERT_INVALID

Problem: The mqweb server certifcate has expired

curl:

* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
curl: (60) SSL certificate problem: certificate has expired

chrome:

while Chrome running:   web page reports Lost communication with the server.  Could not establish communication with the server. Check your network connections and refresh your browser

restart browser, get “Your connection is not private NET::ERR_CERT_DATE_INVALID”

message.log.  Chrome session was working, then server certificate expired

  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

Problem: The mqweb server certificate is missing extendedKeyUsage = serverAuth

curl:

* SSL certificate problem: unsupported certificate purpose
curl: (60) SSL certificate problem: unsupported certificate purpose

Chrome:

Your connection is not private
Attackers might be trying to steal your information from localhost (for example, passwords, messages or credit cards).
NET::ERR_CERT_INVALID

Chrome log:

CERT_PKIXVerifyCert for localhost failed err=-8101
From here  -8101 is Certificate type not approved for application.

ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -207
From here -207 is CERT_INVALID

Problems with the server ca certificate

Problem: The trust store has an expired CA.

curl:

* gnutls_handshake() failed: The TLS connection was non-properly terminated.
…
pycurl.error: (35, ‘gnutls_handshake() failed: The TLS connection was non-properly terminated.’)

Problems with the client certificate

Problem: There is no suitable certificate in the client keystore.

For example

  1. There are no “Your certificates” in the browsers keystore
  2. There is a certificate, but has a CA which was not passed down from the server trust keystore
  3. As part of the TLS handshake any self signed certificates are read from the server trust keystore and sent down.  None were found in the “Your certificates”

Curl:

  • * gnutls_handshake() failed: The TLS connection was non-properly terminated.
  • pycurl.error: (35, ‘gnutls_handshake() failed: The TLS connection was non-properly terminated.’)

These messages basically mean the server just ended the connection

Chrome:

ERR_CONNECTION_CLOSED

For a test site, change <ssl clientAuthentication=”true” to false.  Restart mqweb, restart the web browser.  If it prompts for userid and password, the certificate sent from the server was OK.  It is the certificate sent up to the server that has a problem.

Reset false back to true.

Messages in messages.log:

None.

How to debug it.

Check the logs/ffdc directory.  I found I had an ffdc with Stack Dump = java.security.cert.CertPathValidatorException: The certificate issued by CN=SSCA8, OU=CA, O=SSS, C=GB is not trusted; internal cause is:   java.security.cert.CertPathValidatorException: Signature does not match.

Using Chrome trace

When I repeated the investigations, I got different records in the Chromium trace.  One included

--> net_error = -110 (ERR_SSL_CLIENT_AUTH_CERT_NEEDED)

Using the mqweb server java trace – which traces the whole server

See the Oracle Debugging SSL/TLS Connections page and an IBM page.  I could not see how to trace just “the problem”.

With -Djavax.net.debug=ssl:handshake in the jvm.options file, and restarting the mweb server I got

 *** ServerHelloDone
Default Executor-thread-8, WRITE: TLSv1.2 Handshake, length = 3054
Default Executor-thread-2, READ: TLSv1.2 Handshake, length = 7
*** Certificate chain
***
Default Executor-thread-2, fatal error: 40: null cert chain

When it worked I had

*** ServerHelloDone
Default Executor-thread-7, WRITE: TLSv1.2 Handshake, length = 3054
Default Executor-thread-15, READ: TLSv1.2 Handshake, length = 2433
*** Certificate chain
chain [0] = […. the  certificates

…

Found trusted certificate:

When there was no certificate sent up,  it reported null cert chain.

Problem: The client certificate is self signed and not in the server’s trust store

curl:

* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443

Chrome:

ERR_CONNECTION_CLOSED

Messages in messages.log:

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_08.29.27.0.log
  •  E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=testuser, O=aaaa, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target
  •  I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: unable to find valid certification path to requested target com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_08.29.27.1.log
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

 

Problem: Invalid cn=, the cn value is not a valid userid.

curl message

{“error”: [{

  • “action”: “Provide credentials using a client certificate, LTPA security token, or username and password via HTTP basic authentication header. On z/OS, if the mqweb server has been configured for SAF authentication, check the messages.log file for messages indicating that SAF authentication is not available. Start the Liberty angel process if it is not already running. You might need to restart the mqweb server for any changes to take effect.”,
  • “completionCode”: 0,
  •  “explanation”: “The REST API request cannot be completed because credentials were omitted from the request. On z/OS, if the mqweb server has been configured for SAF authentication, this can be caused by the Liberty angel process not being active.”,
  • “message”: “MQWB0104E: The REST API request to ‘https://127.0.0.1:9443/ibmmq/rest/v1/login ‘ is not authenticated.”,
  • “msgId”: “MQWB0104E”,
  • “reasonCode”: 0,
  • “type”: “rest”

chrome:

It gives you a window to enter userid and password.   This looks like a bug as I have <webAppSecurity allowFailOverToBasicAuth=”false”/>.  It takes the userid and password.

Messages in  messages.log:

R com.ibm.websphere.security.CertificateMapFailedException
and 100 lines of stack trace

The certificate causing the problems, nor the userid is listed – so pretty useless.

Problem: Client certificate missing “extendedKeyUsage = clientAuth”  during signing.

curl message

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
curl session hangs…
* Operation timed out after 300506 milliseconds with 0 out of 0 bytes received

Chrome

ERR_CONNECTION_CLOSED

message in messages.log:

  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: Extended key usage does not permit use for TLS client authentication
  •  I FFDC1015I: An FFDC Incident has been created: “java.lang.NullPointerException com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.28_17.11.10.1.log

ffdc in /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log/ffdc

Exception = java.lang.NullPointerException
Source = com.ibm.ws.ssl.core.WSX509TrustManager
probeid = checkClientTrusted
Stack Dump = java.lang.NullPointerException
at com.ibm.ws.ssl.core.WSX509TrustManager.checkClientTrusted(WSX509TrustManager.java:202)

Problem: Client certificate missing “keyUsage = digitalSignature”  during signing.

curl message

* TLSv1.2 (OUT), TLS handshake, Finished (20):
* Operation timed out after 300509 milliseconds with 0 out of 0 bytes received

message in messages.log

  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: KeyUsage does not allow digital signatures
  • FFDC1015I: An FFDC Incident has been created: “java.lang.NullPointerException com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted”
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

ffdc in /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log/ffdc

Exception = java.lang.NullPointerException
Source = com.ibm.ws.ssl.core.WSX509TrustManager
probeid = checkClientTrusted
Stack Dump = java.lang.NullPointerException
at com.ibm.ws.ssl.core.WSX509TrustManager.checkClientTrusted(WSX509TrustManager.java:202)

Chrome:

  • If there is one or more certificates in the keystore, the list of valid certificates does not include the problem one.
  • If there is only the problem certificate in the keystore, you get
    This site can’t be reached.
    localhost unexpectedly closed the connection.
    ERR_CONNECTION_CLOSED

CA Signed client certificate has expired

curl:

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443
* stopped the pause stream!
* Closing connection 0

Chrome:

This site can’t be reached
localhost unexpectedly closed the connection.
ERR_CONNECTION_CLOSED

message in messages.log:

for curl.

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathValidatorException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020; internal cause is:
    java.security.cert.CertificateExpiredException: NotAfter: Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.0.log
  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path validation failed: java.security.cert.CertPathValidatorException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020; internal cause is:
    java.security.cert.CertificateExpiredException: NotAfter: Thu Jan 30 16:46:00 GMT 2020
  •  I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.1.log

for chrome:

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: The cer
    tificate expired at Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.1.log
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

Bad requests

HTTP request was issued – it should have been HTTPS

curl:

curl:(52) Empty reply from server

messages.log:

E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

The client certificate cannot be verified because it is too weak.

Chrome:  ERR_BAD_SSL_CLIENT_AUTH_CERT

Firefox:  An error occurred during a connection to …  security library: memory allocation failure.  Error code: SEC_ERROR_NO_MEMORY

Reason:

The selected client certificate cannot be validated.  For example it has been created with Elliptic Curve sect409k1.   This is considered weak see here.  The signature is not in the list of acceptable signatures.

Display the certificate and compare it with the list of weak signatures.  A TLS handshake trace may help identify this.  Create a new certificate with a supported signature, and import it.

Problem the CA signing is too weak.

For example signing with sha1RSA, when Chrome expects SHA256RSA or stronger.

Chrome:  NET::ERR_CERT_WEAK_SIGNATURE_ALGORITHM

Firefox: I didnt get the error

Action: Use stronger signing.  For example on z/OS use RSA SIZE(2048)

Firefox errors

Your computer clock is set to … . Make sure your computer is set to the correct date, time, and time zone in your system settings, and then refresh …

If your clock is already set to the right time, the web site is likely misconfigured, and there is nothing you can do to resolve the issue. You can notify the web site’s administrator about the problem.

… uses an invalid security certificate.

The certificate is not trusted because the issuer certificate has expired.

Error code: SEC_ERROR_EXPIRED_ISSUER_CERTIFICATE

Reason:

The CA certificate in the trust store has expired.  The a valid CA certificate may have been sent down with the server’s certificate, but the validation failed.

Action:

  1. From Warning: Potential Security Risk Ahead -> Advanced -> View certificate. It will have the certificate.  Note Issuer -> Organisation and common name
  2. Use Firefox preferences-> view certificates.   Select authorities.  Search for the Organisation from the previous line.  Display the certificate with the matching common name.  Replace it and restart the browser.   Replace the certificate through firefox or use this to locate the directory containing the cert9.db.

Error code: SSL_ERROR_CERTIFICATE_UNKNOWN_ALERT

The backend may get java.security.cert.CertPathValidatorException: signature check failed.

One reason, the certificate being used by firefox was signed by an invalid CA, for example the CA had expired.

Action:

  1. Check Firefox preferences-> certificates, and check “Ask you every time” is selected, repeat the connection and display information about the certificate.  It will give you the issuer, but no more information than that.
  2. Regenerate the certificate, import into Firefox, restart Firefox.

Chrome errors

Chrome has more stricter checks than curl.  These are from Chrome browser.

NET::ERR_CONNECTION_CLOSED

  • mqwebuser.xml serverKeyAlias name not in the keystore
  • The host certificate is self signed and is not trusted
  • The client certificate is self signed and not in the server’s trust store
  • Client certificate missing “extendedKeyUsage = clientAuth”  during signing.
  • CA Signed client certificate has expired
  • Client certificate missing “keyUsage = digitalSignature”  during signing.

NET::ERR_CERT_COMMON_NAME_INVALID

  • missing x509 extensions in the server certificate
  • invalid subjectAltName in x509 extensions, for example IP:127.0.0.11  instead of IP:127.0.0.1

NET::ERR_CERT_INVALID

  • missing extendedKeyUsage = serverAuth in x509 extensions

NET::ERR_CERT_AUTHORITY_INVALID

  • Certificate is not peer.  Need certutil -M $sql -n $name -t “P,,” to change the certificate to be a trusted peer
  • Server’s self signed not found in the browser keystore.
  • The CA from the server does not match the certificate in the browsers’ keystore.  It may have the same name,  but check validity dates, finger prints etc.  Check very carefully.

NET::ERR_CERT_DATE_INVALID

  • The mqweb server certificate has expired.

CWPKI0024E: The certificate alias …  specified by the
property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore safkeyring://…/….

The z/OS certificate is not in the keyring, or it is in the keyring and needs to have TRUST

Make the change, stop and restart the web browser

Firefox:  PR_END_OF_FILE_ERROR

Slow backend server.

MQWEB on z/OS

 CWWKS2932I: The unauthorized version of the SAF user registry is activated.
Authentication will proceed using unauthorized native services.

Check at the top of the message log for.  CWWKB0104I: Authorized service group SAFCRED is not available.

Reason: When the web server was started the SAFCRED service was not available.   This could be caused by security not set up properly.

Fix the security.  For example here

CWWKS2930W: A SAF authentication attempt using authorized SAF services was rejected because the server is not authorized to authorized to access the APPL-ID MQWEB. Authentication will proceed using unauthorized SAF services.

Problem:  the profile with class(SERVER) and profile(BBG.SECPFX.MQWEB) is missing
Action:  the define profile matching the APPL-ID.

RDEFINE SERVER BBG.SECPFX.MQWEB
PERMIT BBG.SECPFX.MQWEB  CLASS(SERVER) ID(START1) ACC(READ)
SETROPTS RACLIST(SERVER) refresh

Restart MQWEB server.

CWWKS2960W: Cannot create the default credential for SAF authorization of unauthenticated users.

All authorization checks for unauthenticated users will fail.
The default credential could not be created due to the following error:

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user WSGUEST has insufficient authority to access APPL-ID MQWEB.

SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

PERMIT MQWEB CLASS(APPL) ACCESS(READ) ID(MQWSGUEST)
SETROPTS RACLIST(APPL) REFRESH

CWPKI0022E: SSL HANDSHAKE FAILURE:

A signer with SubjectDN CN=colinpaicesECp256r1, O=cpwebuser,
C=GB was sent from the target host. The signer might need to be added to local trust store safkeyring://…/…,
located in SSL configurate on alias defaultSSLConfig.
The extended error message from the SSL handshake exception is:

Unexpected error: java.security. InvalidAlgorithmParameterException:
the trustAnchors parameter must be non-empty

The full error was

CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with Subject DN  CN=colinpaice, O=HW, C=GB was sent from the target host.  The signer might need to be added to local trust store  safkeyring://START1/TRUST, located in SSL configuration alias izuSSLConfig. The extended error message from the SSL  handshake exception is: Unexpected error:  java.security.InvalidAlgorithmParameterException: the  trustAnchors parameter must be non-empt.

The problem was that the started task userid did not have update access to the trust keyring.  There was an FFDC in the log file at startup showing this.  Part of this was I assumed the wrong userid for the started task.  The z/OS Command D A,IZUSVR1 gave me th userid, which I then checked., and found it had no access.

ERROR: SEC_ERROR_REUSED_ISSUER_AND_SERIAL

I got this on a slow backend system.  I shut down the web server and restarted it, and it ran OK without the message.

ICH408I USER( ) GROUP( ) NAME()
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(…)
SUBJECT(CN=.. .O=… C=GB) ISSUER(….)

The certificate came in, but there was no mapping for it.

Use RACDCERT command to map it to a userid.

RACDCERT MAP ID(IBMUSER) –
SDNFILTER(‘CN…. ‘)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

Firefox SEC_ERROR_BAD_SIGNATURE

Dont know what caused it.  I deleted the CA and readded it and it worked. 

Others

CWWKO0801E:

Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common.

Problem:

There was no serverKeyAlias specified in the <ssl … tag.

CWPKI0024E:

The certificate alias… specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore safkeyring://…/… .

Problem

  • The certificate was not in the keyring
  • It was NOTRUST
  • It had expired
  • The CA for the certificate was not in the keyring.

MQWB0107E: Unable to parse the request data due to exception


A JSONObject text must begin with ‘{‘ at 0 [character 1 line 1]’.”,
Explanation: The REST API request failed as the data in the request payload could not be parsed.

I got this because I used a HTTP POST request instead of a HTTP GET request.


mqweb – what to do when you cannot get TLS to work?

It is hard to debug setup problems in mqweb.   I found it easiest to not use the mqweb trace, but diagnose problems from the client side.

You need to understand many TLS concepts.  I’ve documented a lot of information here: Understanding the TLS concepts for using certificates to authenticate in mqweb.

I found the easiest way to debug my mqconsole TLS setup, was to use extract the certificates from my browser’s key store and use curl’s verbose, or trace functions.   I’ve documented here how to get a Chrome trace.

I caused all of the common “user errors” and have documented the messages or symptoms I got, these are in this post.

Have you tried turning it off and on again?

The first thing you need to do if you have problems when you are configuring certificates is to restart mqweb, and your browser.   This is because updates to the keystores are not picked up till the mqweb or browser is restarted.  The Chrome and Firefox browsers, remember the certificate used, and logon this on again – so restart the browser to reset every thing.  With Chrome, I set up a bookmark url chrome://restart .

Once you have set up your first connection,  you should not need to change the mqweb server, as you will have set up the mqweb server certificate, and the CA certificate(s) to certify clients.  If you are using self signed,  you will have to import the SS certificate into the trust store, and restart the mqweb server (not good for high availability).

I found if I started chrome from a command window, instead of clicking on an icon, I got out some diagnostic messages to the command window.   These messages were slightly more useful than generic messages like “NET::ERR_CERT_AUTHORITY_INVALID”

Useful Chrome urls

  • chrome://restart
  • chrome://settings/certificates
  • chrome://net-export/ – for collecting a Chrome trace

Getting started

If you are using .pem files (for example openssl) you can use these with no further work.

If you have a .p12 (pkcs12) format keystore, you can use this with no further work.

If you are using a browser with its nssdb database, you need to extract the certificate and private key, and any CA certificates you use.  It is easy to extract a certificate and key  into a .p12 keystore.

Extract the certificate and private key from your browser’s keystore

Curl can use the browser’s key store directly if it has been compiled with NSS (instead of openssl).  “Curl -V”, built with openssl gave me “libcurl/7.58.0 OpenSSL/1.1.1″, someone else’s curl, built with NSS had “libcurl/7.19.7 NSS/3.14.3.0″.  If you do not have curl with NSS support you need to extract the certificate and key from the browsers keystore.

  • Check where your Chrome profile is.  In the Chrome browser, use the url chrome://version .   On one Chrome instance this was  /home/colinpaice/snap/chromium/986/.pki/nssdb .  On a different Chrome instance, the keystore was /home/colinpaice/.pki/nssdb .
  • Export your certificate and keystore
    • pk12util -o colinpaicex.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -n colinpaice -W password
    • pk12util – invoke this program
    • -o colinpaicex.p12  – create this pkcs12 store
    • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/  – from this repository
    • -n colinpaice  – with this name
    • -W password  – and give it this password
  • If you have created your own certificate authority, you need to extract the certificate if you do not already have it.  Firstly list the contents to remind yourself what the CA certificate is called, then extract the certificate (‘myCACert’ in my case)
    • certutil -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -L
      • This gives “Certificate Nickname ” and “Trust Attributes”.   Your CA should have a trust Attribute of “C”.
    • certutil -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -L -n “myCACert” -a >outcacert.pem
    • certutil – this program
    • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ – this key store
    • -L  – list
    • -n “myCACert”  – this name
    • -a – ASCII output
    • >outcacert.pem  – create this file

Issue the curl request

You can use the .p12 file, or the certificate.pem and the key.pem file

Example output

If you use the option — verbose  you get a lot of information for example, a successful request has

  • * Trying 127.0.0.1…
  • * TCP_NODELAY set
  • * ALPN, offering h2
  • * ALPN, offering http/1.1
  • * successfully set certificate verify locations:
  • * CAfile: ./outcacert.pem
  • CApath: /etc/ssl/certs
  • * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • * TLSv1.2 (IN), TLS handshake, Certificate (11):
  • * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • * TLSv1.2 (IN), TLS handshake, Request CERT (13):
  • * TLSv1.2 (IN), TLS handshake, Server finished (14):
  • * TLSv1.2 (OUT), TLS handshake, Certificate (11):
  • * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
  • * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • * TLSv1.2 (OUT), TLS handshake, Finished (20):
  • * TLSv1.2 (IN), TLS handshake, Finished (20):
  • * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
  • * ALPN, server did not agree to a protocol
  • * Server certificate:
  • *   subject: C=GB; O=cpwebuser; CN=mqweb5
  • *   start date: Jan 20 17:53:59 2020 GMT
  • *   expire date: Oct 16 17:53:59 2022 GMT
  • *   subjectAltName: host “127.0.0.1” matched cert’s IP address!
  • *   issuer: C=GB; O=SSS; OU=CA; CN=SSCA7
  • *  SSL certificate verify ok.
  • > GET /ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* HTTP/1.1
  • > Host: 127.0.0.1:9443

See here for an overview of the TLS handshake.   The amount of progress down the list of steps in the hand shake give you a clue as to where the problem may be.  If it is around “TLS handshake, Client Hello (1)”.  This is likely to be a problem with the server certificate.

The numbers as in TLS handshake, CERT verify (15): are the id number of the request, 15 is CERT verify.

A “Finished” message is always sent immediately after a change cipher spec message to verify that the key exchange and authentication processes were successful.  More checks are done after this.

If you use ‑‑trace filename.txt instead of ‑‑verbose you get the same data as displayed as with ‑‑verbose, plus the data flowing up and down the connection.  I found ‑‑verbose had sufficient details to resolve the problems.

mqweb – performance notes

  • I found facilities in Liberty which can improve the performance of your mqweb server by 1% – ish, by using http/2 protocol and ALPN
  • Ive documented where time is spent in the mq rest exchange.

Use of http/2 and ALPN to improve performance.

According to Wikipedia, Application-Layer Protocol Negotiation (ALPN) is a Transport Layer Security (TLS) extension that allows the application layer to negotiate which protocol should be performed over a secure connection in a manner that avoids additional round trips and which is independent of the application-layer protocols. It is needed by secure HTTP/2 connections, which improves the compression of web pages and reduces their latency compared to HTTP/1.x.

mqweb configuration.

This is a liberty web browser configuration, see this page.

For example

 <httpEndpoint id="defaultHttpEndpoint"
   host="${httpHost}" 
   httpPort="${httpPort}"
   httpsPort="${httpsPort}"
   protocolVersion="http/2"
   >
   <httpOptions removeServerHeader="false"/>

</httpEndpoint>

Client configuration

Most web  browsers support this with no additional configuration needed.

With curl you specify ––http2.

With curl, ALPN is enabled by default (as long as curl is built with the ALPN support).

With the curl ––verbose option on a curl request,  you get

  • * ALPN, offering h2 – this tells you that curl has the support for http2.
  • * ALPN, offering http/1.1

and one of

  • * ALPN, server did not agree to a protocol
  • * ALPN, server accepted to use h2

The “* ALPN, server accepted to use h2” says that mqweb is configured for http2.

With pycurl you specify

 c.setopt(pycurl.SSL_ENABLE_ALPN,1)
 c.setopt(pycurl.HTTP_VERSION,pycurl.CURL_HTTP_VERSION_2_0)

Performance test

I did a quick performance test of a pycurl program getting a 1024 byte message (1024 * the character ‘x’) using TLS certificates.

HTTP support Amount of “application data” sent Total data sent.
http/1.1 2414 7151
http/2 2320 7097

So a slight reduction in the number of bytes send when using http/2.

The time to get 10 messages was 55 ms with http/2, and 77ms with http/1.1,  though there was significant variation in repeated measurements, so I would not rely on these measurements.

Where is the time being spent?

cURL and pycurl can report the times from the underlying libcurl package.  See TIMES here.

The times (from the start of the request) are

  • Name lookup
  • Connect
  • Application connect
  • Pre transfer
  • Start transfer
  • Total time

Total time- Start transfer = duration of application data transfer.

Connect duration = Connect Time – Name lookup Time etc.

For a pycurl request getting two messages from a queue the durations were

Duration in microseconds First message Second messages
Name_lookup 4265 32
Connect 53 3
APP Connect 18985 0
Pre Transfer 31 42
Start Transfer 12644 11036
Transfer of application data 264 235

Most of the time is spent setting up the connection, if the same connection can be reused, then the second and successive requests are much faster.

In round numbers, the first message took 50 ms, successive messages took between 10 and 15 ms.