Setting up Liberty(as used in mqweb) to use native JMX

Setting up the server side is well documented in the Oracle Monitoring and Management Using JMX Technology documentation.  Using it from a client is not so well documented.

Server set up

The  Liberty jvm.options file needs parameters.  Note the port=9010 is used  by clients accessing the data.

To provide insecure access from only the local machine

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9010 
-Dcom.sun.management.jmxremote.local.only=true 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false

To provide securer access using TLS

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true

# the following statements point to the same key store as
# used by mqweb server.   This could be different.
-Djavax.net.ssl.keyStoreType=PKCS12
-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/mqweb.p12
-Djavax.net.ssl.keyStorePassword=password
# the following statements point to the same trust store as
# used by mqweb server.   This could be different.
# if you used self signed certificates you could have a keystore
# just for the JMX users
-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks
-Djavax.net.ssl.trustStorePassword=zpassword
-Djavax.net.ssl.trustStoreType=JKS

# The following defines the userid and password file
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.password.file=/home/colinpaice/ssl/ssl2/jmxremote.password

# The following defines the access a userid can have
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.access.file=/home/colinpaice/ssl/ssl2/jmxremote.access

jmxremote.password has

# specify actual password instead of the text password
monitorRole password
controlRole password

jmxremote.access has

# The "monitorRole" role has readonly access.
# The "controlRole" role has readwrite access.
monitorRole readonly
controlRole readwrite

Client set up

jconsole

You cannot pass a  userid and password when the jconsole command, so you have to disable authentication in the jvm.options file

-Dcom.sun.management.jmxremote.authenticate=false

The parameters for jconsole have  -J on them, as in -J-D…. .  jconsole removes the -J and uses the rest of the parameters when invoking the JVM.

I could not get jconsole to recognize a config file using the -J-Dcom.sun.management.config.file = /path/to/jmxremote.properties , so I wrote a bash script to make it easier to change parameters.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file"
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-J-Djavax.net.debug=ssl:handshake"
jconsole -J$ssl1 -J$ssl2 -J$ssl3 -J$ssl4 -J$ssl5 -J$ssl6 -J$ssl7 $ssl8 127.0.0.1:9010

The option “-J-Djavax.net.debug=ssl:handshake” gives a verb verbose trace of the ssl flows for the handshake.

The option -J-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file enables the jconsole logging.  I did not find the output very useful.

There is information the logger in general here,  and on the file logger, here.

The logging.file had

Logging.properties

handlers= java.util.logging.FileHandler
// , java.util.logging.ConsoleHandler2

java.util.logging.FileHandler.pattern=/home/colinpaice/JMXQuery/java/log.%g.file
java.util.logging.FileHandler.limit=50000
java.util.logging.FileHandler.count=2
java.util.logging.FileHandler.level=ALL
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

// .level = INFO
// logger.level = FINEST
.level = FINEST
// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose...
javax.level= FINER
javax.management.level = FINER
javax.management.remote.*     = FINER 
javax.management.remote.level = FINER
javax.management.remote.misc.level  = FINER
javax.management.remote.rmi.level= FINER

Using jmxquery

I used a bash shell script to run the command, as it was easier to manage, and I could not find a way of having the java system properties in a file.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ibmsys1.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"les
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file" 
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-Djavax.net.debug=ssl:handshake"
jar="-jar JMXQuery.jar"
user="-username monitorRole -password password"
url="-url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi"
parms=" -q   WebSphere:*  -count 2 -every 2"
java $ssl1 $ssl2 $ssl3 $ssl4 $ssl5 $ssl6 $ssl7 $ssl8 $jar $url $user $parms

 

Accessing JMX data in Liberty server, securely.

I thought  I would complete the work I did with using JMX in the mqweb server.   It was another example of Hofstadter’s Law:

It always takes longer than you expect, even when you take into account Hofstadter’s Law.

I spent a lot of time looking for things on the web, expecting them to be obvious, only to find that the things do not behave as expected.  I could not find them, because they were not there.  For example I expected to be able to configure the JMX server to use my OS userid and password.  I could have a file with userids and passwords, or lookup in LDAP, but not my normal userid.

Getting started

I found there are two ways of getting the JMX data from the mqweb server.

  1. Use of the native JMX support
  2. Using the Liberty REST API

I think the REST API is easier to set up and is more secure.

I’ll document a high level overview of the two approaches, and how to configure them

Overview of using the native JMX support.

To use this, you configure parameters in the jvm.options file, including a port solely for JMX.

You can use TLS certificates to set up a secure link between the client and the server.

You can decide if you want to logon with userid and password.  If you do you can set up

  1. A file with userids and passwords; and a file with userids and permitted access.   The documentation talks about userids like monitorRole and controlRole.   You have to put a process in place to periodically change these passwords.
  2. Use and LDAP server to do userid validation and to get the access.
  3. I could not find how to use your operating system userid and password for authentication.
  4. I could not find how to use the DN as authorization.

If your certificate is valid (either because it is signed by a CA, or there is a copy of a self signed certificate in the trust store), this is good enough for the checking.   You can enable userid and password checking, but this solution feels weak, as you have to do extra work to manage it properly;  you do not have a single signon.  Not all tools support using userid and password, for example I could not pass userid and password on the jconsole command.

Overview of Using the Liberty REST API

As with the MQ REST API you can issue an HTTP request and get data back.  See here.  For example

curl –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password https://localhost:9443/IBMJMXConnectorREST/mbeans/WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes=*

There is a small amount of configuration you need to do – less than with the native JMX support.  The data comes back as JSON (as you might expect) and also includes a time stamp, which is very useful when post processing.

You define <administrator-role><user>..</user></administrator-role>  in a similar way to setting up authorisation for mqconsole and mqrest.  It takes the cn= value from your certificate as the userid, so you can give individual access.

“Securely” is a good laugh.

There are different levels of (in)security.

If you are using the native JMX support

  • You can have no passwords or access checks needed.  The data is read only, and is not sensitive.
  • You can set up userid(s) and passwords in a file
  • You cannot use the operating system userid and password
  • You can use LDAP to check the userid and password, and get the role for that userid
  • You can use TLS, so anyone with a valid certificate can access the data
  • You can use TLS and use the userid and password in a file to determine access
  • You can use TLS and LDAP to get the role for that certificate

If you are using the WLP REST support

  • You can specify a userid and password
  • You can use a certificate, and the Common Name is used as the userid
  • You can specify in the configuration file, what access userids, or groups have

You can use TLS to protect your communications to and from the server.

Java leaks passwords

You need to be aware that your client machine may leak information.  For example I ran a  Java program to issue JMX requests from a script.

I could use the linux command ps -ef to display information about my request

ps -ef |grep JMX

gave me

colinpa+ 1871 1870 79 10:27 pts/2 00:00:01 java …  -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12 -Djavax.net.ssl.keyStorePassword=password …  -username monitorRole -password password

This exposed the password to my keystore and password to my userid!  I could not find a way of having all these java system parameters in a file.

I found export JAVA_TOOL_OPTIONS=”-D…”  and this get picked up, but then java displays the variables as in Picked up JAVA_TOOL_OPTIONS: …

jconsole

Some programs have been designed to protect information for example jconsole you can put your system properties in a file

-J-Dcom.sun.management.config.file=ConfigFilePath

and so keep your parameters secure, but I could not get this to work.

Curl can be configured not to display parameters

With curl you have a command like

curl -n –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password

which gives away your password.  If you do not specify it inline, you get prompted for it.

You can put your parameters in a config file, for example curl.config,

cacert ./cacert.pem 
cert ./colinpaice.pem:password 
key colinpaice.key.pem 
cookie cookie.jar.txt 
cookie-jar cookie.jar.txt 
url https://127.0.0.1:9443/ibmmq/rest/v1/login

and use

curl –config curl.config

Easy!

Protecting key files

It is important to protect the certificate file (with the important private key) so it is accessible by just the owner.  The linux command  ls -ltr colinpaice.p12 gives

-rw------- 1 colinpaice colinpaice 4146 Jan 31 17:56 colinpaice.p12

Of course anyone with super user authority has access to this file!

mqweb – displaying the secret statistics

Yes, mqweb does provide statistics; through the standard JMX interface provided as part of the base Liberty function.  I expect most people do not know they are available.   The data gets less useful over time, for example you get the “average time” since the mqweb started, rather than the last minute.  See here on how to extract useful information from the data, and show useful averages.

You need in mqwebuser.xml .

<featureManager>
<feature>monitor-1.0</feature>
</featureManager>

and in jvm.options

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

These options should be suitable in your test environment.  You will want to change them for production.

You need the port number (9010 in the above example) when you extract jmx data.

How do you display the data?

For a quick sniff, (no good for extracting data and plotting charts)  you can use jconsole.  Use remote connection localhost:9010 .  it does not display all of the data.

I found jmxquery very useful.  I updated the github version to fix a bug which caused a loop.  See here.

The query is

java -jar JMXQuery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi -q ‘WebSphere:*’

You get data on

  • WebSphere:type=JvmStats
  • WebSphere:type=ThreadPoolStats,name=Default Executor
  • WebSphere:type=ServletStats,name=com.ibm.mq.console.javax.ws.rs.core.Application/AppName (String) = com.ibm.mq.console
  • WebSphere:type=ServletStats,name=com.ibm.mq.rest.javax.ws.rs.core.Application/AppName (String) = com.ibm.mq.rest

To get ‘console’  and ‘ResponseTime’ data I used

java -jar JMXQuery.jar -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi 
-q 'WebSphere:*' 
-count 20 -every 60 
|grep --line-buffered console
|grep --line-buffered ResponseTime
|python3 mqweb.py
  • Where
  • java -jar JMXQuery.jar – invoke the program
  • -url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi – with this url and the above port number from the jvm.options
  • -q ‘WebSphere:*’ – give me only data for this components
  • -count 20 -every 60 – my extensions giving a record every 60 seconds, and doing 20 of them
  • |grep –line-buffered console – only pull out the console records, ( ignore the ‘rest’ records).  The –line-buffered tells grep to flush it immediately
  • |grep –line-buffered ResponseTime – only interested in this detailed level
  • |python3 mqweb.py – pass it into the python program.  This calculates the delta between records and prints out the count and mean value for that interval

If you are collecting data in real time from a stream, you need to ensure any processing is unbufferred.  Often the default behavior is to accumulate the data in a big buffer, and write the buffer when it is full.  Check any filters you use, for example grep –line-buffered.

One line of output from the JMXQuery program for the console activity is

WebSphere:type=ServletStats,name=com.ibm.mq.console.javax.ws.rs.core.Application/ResponseTimeDetails/count (Long) = 5

JvmStats

This comes under WebSphere:type=JvmStats/  See here.

  • FreeMemory (Long) = 13350536
  • ProcessCPU (Double) = 0.658040027605245
  • UsedMemory (Long) = 75131256
  • GcTime (Long) = 539
  • UpTime (Long) = 1048277
  • GcCount (Long) = 118
  • Heap (Long) = 88473600
  • FreeMemory (Long) = 13350536
  • ProcessCPU (Double) = 0.658040027605245
  • UsedMemory (Long) = 75131256
  • GcTime (Long) = 539
  • UpTime (Long) = 1048277
  • GcCount (Long) = 118

ThreadPoolStats

This comes under WebSphere:type=ThreadPoolStats,name=Default Executor/

  • PoolSize (Integer) = 8
  • ActiveThreads (Integer) = 2
  • PoolName (String) = Default Executor

See here.

com.ibm.mq.console and  com.ibm.mq.rest

The data is similar between them.  One has name=com.ibm.mq.console.javax.ws.rs.core.Application, the other has name=com.ibm.mq.rest.javax.ws.rs.core.Application

The data (in italics) with my comments in plain font are

  • AppName (String) = com.ibm.mq.console
  • RequestCountDetails/currentValue (Long) = 116
  • RequestCountDetails/description (String) = This shows number of requests to a servlet
  • RequestCountDetails/unit (String) = ns – this looks like a bug as it is a count not nanoseconds
  • RequestCount (Long) = 116
  • ServletName (String) = javax.ws.rs.core.Application
  • ResponseTimeDetails/count (Long) = 116
  • ResponseTimeDetails/description (String) = Average Response Time for servlet
  • ResponseTimeDetails/maximumValue (Long) = 3060146565 – in nanoseconds ( see below for the unit)
  • ResponseTimeDetails/mean (Double) = 8.796846855172414E7 – in nanoseconds
  • ResponseTimeDetails/minimumValue (Long) = 793871 – in nanoseconds
  • ResponseTimeDetails/standardDeviation (Double) = 4.198572684166255E8
  • ResponseTimeDetails/total (Double) = 1.0204342352E10 – in nanoseconds – used in calculations
  • ResponseTimeDetails/unit (String) = ns  – this is the units.  ns is nanoseconds
  • ResponseTimeDetails/variance (Double) = 1.64064538075292096E17 –  used in calculations
  • Description (String) = Report Servlet Stats for specified Servlet and application.
  • ResponseTime (Double) = 8.796846855172414E7 – same as the ResponseTimeDetails

So we can see that there were

  • 116 console requests since the mqweb server was started
  • the units are ns (nano seconds)
  • the console requests taking an average of 8.796846855172414E7  nanoseconds, 0.0879 seconds with
  • a standard deviation of  4.136787844033974E7 – nanoseconds = 0.04198 seconds
  • the maximum value was 3 060 146 565 ns = 3.060 seconds
  • the minimum time was  793 871 ns or 0.000793 seconds

Some other data, showing how it changed over time

Data Values Later values Much later values
Number of requests 82 3590 22920
Average (seconds) 0.158 0.0108 0.0099
Standard deviation (seconds) 0.487 0.034 0.035
Maximum (seconds) 2.3 2.3 2.3
Minimum(seconds) 0.001 0.0001 0.0001

Notes:

  • There is data only once a request has been processed, so if you have not run a rest request, there will be no JMX data for rest activity.
  • These values are from start of the mqweb server. I did not see them reset, so you could have a data for a whole week or more.
  • The maximum was from the first requests to run.  I expect this includes the “warm up” costs,  of loading the code and JITing it.
  • The average values are from the start, so will be impacted by peaks and troughs.

For the each mqconsole window, there are two console counts every 10 seconds.  Any charts are refreshed every 10 seconds, so  I think this is a “I am still here, please send me any data you have for me”.

Data for rest

I started my mqweb server, and ran a python program which opened a connected and got three messages.

  • Maximum time 0.3486
  • Minimum time 0.0026
  • Calculate the other one 0.0028

Because the first request takes a long time, you can adjust for this in your calculations to get a truer mean.

For example

I reran the script and processed 100 messages.  The average time of these was 0.003 seconds.

  • Maximum 0.3486
  • Mean 0.00637
  • Count 103
  • Total 0.656

The calculations are

  • Mean * count =  0.656 (which matches Total as expected)
  • Subtract the maximum, first time value 0.656 – 0.349 = 0.307
  • Calculate the improved mean value ignoring the first value,  0.307 /(103 -1) = 0.003

So the adjusted mean time is 0.003 seconds – compared to the 0.006 which the JMX stats report.

 

Getting useful information out of JMX data

The data coming from Liberty WebServer through the JMX interface  provides some data, but it is not very useful, and it may become inaccurate over time.

I’ll cover

  1. Getting a useful mean value
  2. Getting a more accurate long term mean
  3. Data gets more inaccurate over time
  4. Standard deviation (this may only be of interest to a few people)

For example from JMX, the reported  mean time for mqconsole transactions  was 9.9 milliseconds – this is for all requests since the mqweb server was started.   Over the previous minute the average measured time, for a 10 second period was 7, or 8 milliseconds, so well below the 9.9 reported.

This is because the mean time includes any initial start up time.   The maximum transaction time, at the start of the run, was over 2 seconds.   This will bias the mean.

You can process the data to extract some useful information, and I show below how to get out useful response time values.

You get the following data (and example values) from mqweb through the JMX interface.

ResponseTimeDetails/count (Long) = 20
ResponseTimeDetails/description (String) = Average Response Time for servlet
ResponseTimeDetails/maximumValue (Long) = 3060146565 
– in nanoseconds (see below for the unit)
ResponseTimeDetails/mean (Double) = 4.336789965E8
– in nanoseconds
ResponseTimeDetails/minimumValue (Long) = 2474556
– in nanoseconds
ResponseTimeDetails/standardDeviation (Double) = 9.089057964078983E8
– in nanoseconds
ResponseTimeDetails/total (Double) = 8.67357993E9
– used in calculations
ResponseTimeDetails/unit (String) = ns
– the unit ns = nanoseconds
ResponseTimeDetails/variance (Double) = 8.319076762335653
– used in calculations

Getting a useful mean value

To produce these numbers, the count of the response times and the sum of the transaction response times are accumulated within the Liberty Server.  To calculate the mean value you calculate sum/count.   This gives you the overall mean time.  If you obtain the data periodically you can manipulate the data to provide some useful statistics.

Let the count and sum at time T1 be Count1, and Sum1, and at time T2 Count2, and Sum2.
You can now calculate (Sum2- Sum1)/(Count2 – Count1) to get the average for that period.  For example the reported mean was 0.016 ms, but the calculated value gave 0.008 ms.  You can also calculate (Count2 – Count1)/(T2-T1) to give a rate of requests per second.   These are much more useful than the raw data.  I suggest collecting the data every minute.

Getting a more accurate long term mean

The first rest request and console request take a long time because the java code has to be loaded in etc.  In one test the duration of the first request was 50 times the duration of the other requests.  A better “mean” value is to ignore the duration of the first request.

The improved mean is (JMX mean * JMX count  – JMX Maximum value) /(JMX Count-1), or JMXMean – (JMXMaximum/JMXCount) .

Data gets more inaccurate over time

The total time is stored as a floating point double.  As you add small numbers to big numbers, the small numbers may be ignored.  Let me try to explain.

Consider a float which has precision of 3, so you could have 1.23 E2 = 1230.  Add 1 to this, and you get 1231 which is 1.23 E2 with a precision of 3 – the number we started with.

The total time is in nanoseconds so 1 second is stored as 1.0 E9.  With 100 of these a second, and 1 hour( 3600 seconds) for 100 hours is 360,000,000, or 3.6 E8 seconds.  * 1.0 E9 nano seconds. = 3.6E17 nano seconds.   The precision  of most float numbers is 16, so with this 3.6 E17 we have lost the odd nanosecond.    I do not think this is a big enough problem to worry about – unless you are running for years without restarting the server.

The variance uses the time**2 value.  So with the maximum time above 599482097 nano seconds. Time **2 is 3.593787846×10¹⁷ and you are already losing data.  I expect the variance will not be used very often, so I think this can be ignored.

If the times were in microseconds instead of nano seconds, this would not be a problem.

Getting a useful standard deviation (this may only be of interest to a few people)

The standard deviation gives a measure of the spread of the data, a small standard deviation means the data is in a narrow band, a larger standard deviation means it is spread over a wider band.  Often 95% of the values are within plus or minus 3 * standard deviations from the mean, so anything outside this range would considered an outlier, or unusual statistic.

I set up some data, a mixture of  10  values 9, 10, 11,  the standard deviation was 0.73.    I changed one value to 20, and the standard deviation changed to 3.3, indicating a wide spread of values.

With a mixture of 100 values 9,10,11, the standard deviation was 0.71.   I changed one value to 20, and the standard deviation changed to 1.2, so a much smaller value, most of the data was still around 10 – just one outside the range.

With a lot of data, the standard deviation converges on a value, and “unusual” numbers make little difference to the value.  I think that the standard deviation over an extended period is not that useful, especially if you get periodic variations such as busy time, and overnight.

You calculate the standard deviation as the square root of the variance.   The variance is (Sum of (values**2) – (mean ** 2)) /number of measurements.

With data

ResponseTimeDetails/count (Long) = 203
ResponseTimeDetails/mean (Double) = 6420785.187192118 nanoseconds
ResponseTimeDetails/variance (Double) = 1.7113341125320868E15 – used in calculations

Variance  = 1.7113341125320868E15 =  ( (Sum of (values**2) – (6420785.187192118 ** 2)) / 203

So (Sum of (values**2)) =   3.474420513264337e+17

You can now do the sums as with the mean, above:

At time T1, the ssquares1 is the sum of (values**2)   at time T2, the ssquares2 is the sum of (values**2).

You can now calculate ssquares2 – ssquares2, and used that to estimate the variance, and so the standard deviation of the data in the range T1 to T2, I’ll leave the details to the end user.

For the advanced user,  you can use the mean for the interval – rather than the long term mean.  Good luck.

 

mqweb error messages and symptoms of TLS setup problems

I deliberately caused TLS set up errors, and noted the symptoms.  Ive recorded them below; the article is not meant to be read, but indexed by search engines.

There are three sections

  1. Problems with server certificates
  2. Problems with the client certificate
  3. Chrome messages, and possible causes of the problems.

The mqweb messages.log reported problems that the mqweb server saw.   For me this was in file /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log

Problems with the server certificate

Problem: mqwebuser.xml serverKeyAlias name not in the keystore

Message log:

  • E CWPKI0024E: The certificate alias mqweb specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore /home/colinpaice/ssl/ssl2/mqweb.p12.
  • I FFDC1015I: An FFDC Incident has been created: “com.ibm.wsspi.channelfw.exception.ChannelException: java.lang.IllegalArgumentException: CWPKI0024E: The certificate alias mqweb specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore /home/colinpaice/ssl/ssl2/mqweb.p12. com.ibm.ws.channel.ssl.internal.SSLConnectionLink 238″ at ffdc_….

curl:

* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* curl (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443
* stopped the pause stream!
* Closing connection 0

chrome:

This site can’t be reached.  ERR_CONNECTION_CLOSED

Problem:  The host certificate is self signed and not in the client keystore

Problem:  The host certificate is signed but the signer certificate is the client keystore

Message log:

Nothing.

curl:

* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate

Chrome: in browser

NET::ERR_CERT_AUTHORITY_INVALID

Click on the Not Secure in the url, to display the certificate which was sent down.

If it is signed, make a note of the “issued by” Common Name(CN), and the  Organisation(0) and look up the value of Organisation in the “Authorities” section of “Manage Certificates”.

Chrome log:

ERROR:cert_verify_proc_nss.cc(1011)] CERT_PKIXVerifyCert for localhost failed err=-8179

From here  -8179 is Peer’s certificate issuer is not recognized.

Problem: curl: The host certificate is self signed and you use the –insecure option

curl

* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=GB; O=aaaa; CN=testuser
* start date: Jan 20 17:39:37 2020 GMT
* expire date: Feb 19 17:39:37 2020 GMT
* issuer: C=GB; O=aaaa; CN=testuser
* SSL certificate verify result: self signed certificate (18), continuing anyway.

Problem: Chrome:  The host certificate is self signed and is not trusted

Chrome browser

This site can’t be reached
localhost unexpectedly closed the connection.
ERR_CONNECTION_CLOSED

Debugging

  • I could find nothing that told me what certificate was being used.  The Chrome network trace just gave “net_error = -100 (ERR_CONNECTION_CLOSED)“.
  • Use certutil -L $sql  to list the contents of your browsers keystore.   The certificate needs “P,…” permissions.
  •  Or use the chrome url chrome://settings/certificates  and display “your certificates”. Pick the likely one, if it says “UNTRUSTED” then this may be the problem.   View the certificate, and check it, for example under details, there may be a comment describing its use.
  •  Defined the server certificate as trusted using certutil -M $sql -n name -t “P,,” 
  • Restart the web browser.

Problem: The  CA signer server certificate had the wrong subjectAltName

curl:

* subjectAltName does not match 127.0.0.1
* SSL: no alternative certificate subject name matches target host name ‘127.0.0.1’

Chrome:

NET::ERR_CERT_COMMON_NAME_INVALID
From the “Not Secure” in front of the URL, display the certificate, and check the extenstions, especially Certificate Subject Alternative Names.

Chrome log:

ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -200
From here -200 is  CERT_INVALID

Problem: The mqweb server certifcate has expired

curl:

* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
curl: (60) SSL certificate problem: certificate has expired

chrome:

while Chrome running:   web page reports Lost communication with the server.  Could not establish communication with the server. Check your network connections and refresh your browser

restart browser, get “Your connection is not private NET::ERR_CERT_DATE_INVALID”

message.log.  Chrome session was working, then server certificate expired

  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

Problem: The mqweb server certificate is missing extendedKeyUsage = serverAuth

curl:

* SSL certificate problem: unsupported certificate purpose
curl: (60) SSL certificate problem: unsupported certificate purpose

Chrome:

Your connection is not private
Attackers might be trying to steal your information from localhost (for example, passwords, messages or credit cards).
NET::ERR_CERT_INVALID

Chrome log:

CERT_PKIXVerifyCert for localhost failed err=-8101
From here  -8101 is Certificate type not approved for application.

ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -207
From here -207 is CERT_INVALID

 

Problems with the server ca certificate

Problem: The trust store has an expired CA.

curl:

* gnutls_handshake() failed: The TLS connection was non-properly terminated.

pycurl.error: (35, ‘gnutls_handshake() failed: The TLS connection was non-properly terminated.’)

Problems with the client certificate

Problem: There is no suitable certificate in the client keystore.

For example

  1. There are no “Your certificates” in the browsers keystore
  2. There is a certificate, but has a CA which was not passed down from the server trust keystore
  3. As part of the TLS handshake any self signed certificates are read from the server trust keystore and sent down.  None were found in the “Your certificates”

Curl:

  • * gnutls_handshake() failed: The TLS connection was non-properly terminated.
  • pycurl.error: (35, ‘gnutls_handshake() failed: The TLS connection was non-properly terminated.’)

These messages basically mean the server just ended the connection

Chrome:

ERR_CONNECTION_CLOSED

For a test site, change <ssl clientAuthentication=”true” to false.  Restart mqweb, restart the web browser.  If it prompts for userid and password, the certificate sent from the server was OK.  It is the certificate sent up to the server that has a problem.

Reset false back to true.

Messages in messages.log:

None.

How to debug it.

Check the logs/ffdc directory.  I found I had an ffdc with Stack Dump = java.security.cert.CertPathValidatorException: The certificate issued by CN=SSCA8, OU=CA, O=SSS, C=GB is not trusted; internal cause is:   java.security.cert.CertPathValidatorException: Signature does not match.

Using Chrome trace

When I repeated the investigations, I got different records in the Chromium trace.  One included

--> net_error = -110 (ERR_SSL_CLIENT_AUTH_CERT_NEEDED)

Using the mqweb server java trace – which traces the whole server

See the Oracle Debugging SSL/TLS Connections page and an IBM page.  I could not see how to trace just “the problem”.

With -Djavax.net.debug=ssl:handshake in the jvm.options file, and restarting the mweb server I got

 *** ServerHelloDone
Default Executor-thread-8, WRITE: TLSv1.2 Handshake, length = 3054
Default Executor-thread-2, READ: TLSv1.2 Handshake, length = 7
*** Certificate chain
***
Default Executor-thread-2, fatal error: 40: null cert chain

When it worked I had

*** ServerHelloDone
Default Executor-thread-7, WRITE: TLSv1.2 Handshake, length = 3054
Default Executor-thread-15, READ: TLSv1.2 Handshake, length = 2433
*** Certificate chain
chain [0] = […. the  certificates

Found trusted certificate:

When there was no certificate sent up,  it reported null cert chain.

Problem: The client certificate is self signed and not in the server’s trust store

curl:

* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443

Chrome:

ERR_CONNECTION_CLOSED

Messages in messages.log:

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_08.29.27.0.log
  •  E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=testuser, O=aaaa, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target
  •  I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: unable to find valid certification path to requested target com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_08.29.27.1.log
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

Problem: Invalid cn=, the cn value is not a valid userid.

curl message

{“error”: [{

  • “action”: “Provide credentials using a client certificate, LTPA security token, or username and password via HTTP basic authentication header. On z/OS, if the mqweb server has been configured for SAF authentication, check the messages.log file for messages indicating that SAF authentication is not available. Start the Liberty angel process if it is not already running. You might need to restart the mqweb server for any changes to take effect.”,
  • “completionCode”: 0,
  •  “explanation”: “The REST API request cannot be completed because credentials were omitted from the request. On z/OS, if the mqweb server has been configured for SAF authentication, this can be caused by the Liberty angel process not being active.”,
  • “message”: “MQWB0104E: The REST API request to ‘https://127.0.0.1:9443/ibmmq/rest/v1/login ‘ is not authenticated.”,
  • “msgId”: “MQWB0104E”,
  • “reasonCode”: 0,
  • “type”: “rest”

chrome:

It gives you a window to enter userid and password.   This looks like a bug as I have <webAppSecurity allowFailOverToBasicAuth=”false”/>.  It takes the userid and password.

Messages in  messages.log:

R com.ibm.websphere.security.CertificateMapFailedException
and 100 lines of stack trace

The certificate causing the problems, nor the userid is listed – so pretty useless.

Problem: Client certificate missing “extendedKeyUsage = clientAuth”  during signing.

curl message

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
curl session hangs…
* Operation timed out after 300506 milliseconds with 0 out of 0 bytes received

Chrome

ERR_CONNECTION_CLOSED

message in messages.log:

  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: Extended key usage does not permit use for TLS client authentication
  •  I FFDC1015I: An FFDC Incident has been created: “java.lang.NullPointerException com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.28_17.11.10.1.log

ffdc in /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log/ffdc

Exception = java.lang.NullPointerException
Source = com.ibm.ws.ssl.core.WSX509TrustManager
probeid = checkClientTrusted
Stack Dump = java.lang.NullPointerException
at com.ibm.ws.ssl.core.WSX509TrustManager.checkClientTrusted(WSX509TrustManager.java:202)

Problem: Client certificate missing “keyUsage = digitalSignature”  during signing.

curl message

* TLSv1.2 (OUT), TLS handshake, Finished (20):
* Operation timed out after 300509 milliseconds with 0 out of 0 bytes received

message in messages.log

  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: KeyUsage does not allow digital signatures
  • FFDC1015I: An FFDC Incident has been created: “java.lang.NullPointerException com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted”
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

ffdc in /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log/ffdc

Exception = java.lang.NullPointerException
Source = com.ibm.ws.ssl.core.WSX509TrustManager
probeid = checkClientTrusted
Stack Dump = java.lang.NullPointerException
at com.ibm.ws.ssl.core.WSX509TrustManager.checkClientTrusted(WSX509TrustManager.java:202)

Chrome:

  • If there is one or more certificates in the keystore, the list of valid certificates does not include the problem one.
  • If there is only the problem certificate in the keystore, you get
    This site can’t be reached.
    localhost unexpectedly closed the connection.
    ERR_CONNECTION_CLOSED

CA Signed client certificate has expired

curl:

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443
* stopped the pause stream!
* Closing connection 0

Chrome:

This site can’t be reached
localhost unexpectedly closed the connection.
ERR_CONNECTION_CLOSED

message in messages.log:

for curl.

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathValidatorException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020; internal cause is:
    java.security.cert.CertificateExpiredException: NotAfter: Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.0.log
  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path validation failed: java.security.cert.CertPathValidatorException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020; internal cause is:
    java.security.cert.CertificateExpiredException: NotAfter: Thu Jan 30 16:46:00 GMT 2020
  •  I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.1.log

for chrome:

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: The cer
    tificate expired at Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.1.log
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

 

 

Bad requests

HTTP request was issued – it should have been HTTPS

curl:

curl:(52) Empty reply from server

messages.log:

E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

 

Chrome errors

Chrome has more stricter checks than curl.  These are from Chrome browser.

NET::ERR_CONNECTION_CLOSED

  • mqwebuser.xml serverKeyAlias name not in the keystore
  • The host certificate is self signed and is not trusted
  • The client certificate is self signed and not in the server’s trust store
  • Client certificate missing “extendedKeyUsage = clientAuth”  during signing.
  • CA Signed client certificate has expired
  • Client certificate missing “keyUsage = digitalSignature”  during signing.

NET::ERR_CERT_COMMON_NAME_INVALID

  • missing x509 extensions in the server certificate
  • invalid subjectAltName in x509 extensions, for example IP:127.0.0.11  instead of IP:127.0.0.1

NET::ERR_CERT_INVALID

  • missing extendedKeyUsage = serverAuth in x509 extensions

NET::ERR_CERT_AUTHORITY_INVALID

  • Certificate is not peer.  Need certutil -M $sql -n $name -t “P,,” to change the certificate to be a trusted peer
  • Server’s self signed not found in the browser keystore.

NET::ERR_CERT_DATE_INVALID

  • The mqweb server certifcate has expired

mqweb – what to do when you cannot get TLS to work?

It is hard to debug setup problems in mqweb.   I found it easiest to not use the mqweb trace, but diagnose problems from the client side.

You need to understand many TLS concepts.  I’ve documented a lot of information here: Understanding the TLS concepts for using certificates to authenticate in mqweb.

I found the easiest way to debug my mqconsole TLS setup, was to use extract the certificates from my browser’s key store and use curl’s verbose, or trace functions.   I’ve documented here how to get a Chrome trace.

I caused all of the common “user errors” and have documented the messages or symptoms I got, these are in this post.

Have you tried turning it off and on again?

The first thing you need to do if you have problems when you are configuring certificates is to restart mqweb, and your browser.   This is because updates to the keystores are not picked up till the mqweb or browser is restarted.  The Chrome and Firefox browsers, remember the certificate used, and logon this on again – so restart the browser to reset every thing.  With Chrome, I set up a bookmark url chrome://restart .

Once you have set up your first connection,  you should not need to change the mqweb server, as you will have set up the mqweb server certificate, and the CA certificate(s) to certify clients.  If you are using self signed,  you will have to import the SS certificate into the trust store, and restart the mqweb server (not good for high availability).

I found if I started chrome from a command window, instead of clicking on an icon, I got out some diagnostic messages to the command window.   These messages were slightly more useful than generic messages like “NET::ERR_CERT_AUTHORITY_INVALID”

Useful Chrome urls

  • chrome://restart
  • chrome://settings/certificates
  • chrome://net-export/ – for collecting a Chrome trace

Getting started

If you are using .pem files (for example openssl) you can use these with no further work.

If you have a .p12 (pkcs12) format keystore, you can use this with no further work.

If you are using a browser with its nssdb database, you need to extract the certificate and private key, and any CA certificates you use.  It is easy to extract a certificate and key  into a .p12 keystore.

Extract the certificate and private key from your browser’s keystore

Curl can use the browser’s key store directly if it has been compiled with NSS (instead of openssl).  “Curl -V”, built with openssl gave me “libcurl/7.58.0 OpenSSL/1.1.1″, someone else’s curl, built with NSS had “libcurl/7.19.7 NSS/3.14.3.0″.  If you do not have curl with NSS support you need to extract the certificate and key from the browsers keystore.

  • Check where your Chrome profile is.  In the Chrome browser, use the url chrome://version .   On one Chrome instance this was  /home/colinpaice/snap/chromium/986/.pki/nssdb .  On a different Chrome instance, the keystore was /home/colinpaice/.pki/nssdb .
  • Export your certificate and keystore
    • pk12util -o colinpaicex.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -n colinpaice -W password
    • pk12util – invoke this program
    • -o colinpaicex.p12  – create this pkcs12 store
    • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/  – from this repository
    • -n colinpaice  – with this name
    • -W password  – and give it this password
  • If you have created your own certificate authority, you need to extract the certificate if you do not already have it.  Firstly list the contents to remind yourself what the CA certificate is called, then extract the certificate (‘myCACert’ in my case)
    • certutil -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -L
      • This gives “Certificate Nickname ” and “Trust Attributes”.   Your CA should have a trust Attribute of “C”.
    • certutil -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -L -n “myCACert” -a >outcacert.pem
    • certutil – this program
    • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ – this key store
    • -L  – list
    • -n “myCACert”  – this name
    • -a – ASCII output
    • >outcacert.pem  – create this file

Issue the curl request

You can use the .p12 file, or the certificate.pem and the key.pem file

Example output

If you use the option — verbose  you get a lot of information for example, a successful request has

  • * Trying 127.0.0.1…
  • * TCP_NODELAY set
  • * ALPN, offering h2
  • * ALPN, offering http/1.1
  • * successfully set certificate verify locations:
  • * CAfile: ./outcacert.pem
  • CApath: /etc/ssl/certs
  • * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • * TLSv1.2 (IN), TLS handshake, Certificate (11):
  • * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • * TLSv1.2 (IN), TLS handshake, Request CERT (13):
  • * TLSv1.2 (IN), TLS handshake, Server finished (14):
  • * TLSv1.2 (OUT), TLS handshake, Certificate (11):
  • * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
  • * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • * TLSv1.2 (OUT), TLS handshake, Finished (20):
  • * TLSv1.2 (IN), TLS handshake, Finished (20):
  • * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
  • * ALPN, server did not agree to a protocol
  • * Server certificate:
  • *   subject: C=GB; O=cpwebuser; CN=mqweb5
  • *   start date: Jan 20 17:53:59 2020 GMT
  • *   expire date: Oct 16 17:53:59 2022 GMT
  • *   subjectAltName: host “127.0.0.1” matched cert’s IP address!
  • *   issuer: C=GB; O=SSS; OU=CA; CN=SSCA7
  • *  SSL certificate verify ok.
  • > GET /ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* HTTP/1.1
  • > Host: 127.0.0.1:9443

See here for an overview of the TLS handshake.   The amount of progress down the list of steps in the hand shake give you a clue as to where the problem may be.  If it is around “TLS handshake, Client Hello (1)”.  This is likely to be a problem with the server certificate.

The numbers as in TLS handshake, CERT verify (15): are the id number of the request, 15 is CERT verify.

A “Finished” message is always sent immediately after a change cipher spec message to verify that the key exchange and authentication processes were successful.  More checks are done after this.

If you use ‑‑trace filename.txt instead of ‑‑verbose you get the same data as displayed as with ‑‑verbose, plus the data flowing up and down the connection.  I found ‑‑verbose had sufficient details to resolve the problems.

mqweb – how to get a chrome browser trace

How to get a chrome trace

See Troubleshooting Chrome network issues  and the description here on how to collect a trace.

  • Open a tab with the chrome://net-export/ url.
  • Click start logging to disk
  • Select a file location
  • In another tab select the mqweb url
  • Click on the “stop” button in the window
  • If you select show file – it opens the json file.   This has all the information you need to process the file, but it is much easier to use the provided tools
  • The filename is given for example “FILE: /home/colinpaice/Downloads/chrome-net-export-log.json
  • Click on “The log file can be loaded using the netlog_viewer.” link.   This gets you to a page which says
  • This app loads NetLog files generated by Chromium’s chrome://net-export. Log data is processed and visualized entirely on the client side (your browser). Data is never uploaded to a remote endpoint.
  • Select  https://netlog-viewer.appspot.com/ to invoke the formatter.
  • Drag your netlog file, or use “choose file”
  • Select events, and this displays all of the traffic
  • In the search bar at the top enter your port 9443, or error
  • You get a list like
  • NONE HOST_RESOLVER_IMPL_REQUEST
    1083 URL_REQUEST https://127.0.0.1:9443/ibmmq/console/
    1084 DISK_CACHE_ENTRY
    1085 HTTP_STREAM_JOB_CONTROLLER https://127.0.0.1:9443/
    1086 HTTP_STREAM_JOB https://127.0.0.1:9443/
  • If the background  is pale green – it is good.  If it is pink (pale red) there was a problem.
  • Click on a line and it displays trace information in a window.  For example the first URL_REQUEST gave
    • t= 8 [st= 8]        HTTP_STREAM_JOB_CONTROLLER_BOUND
                          --> source_dependency = 1089 (HTTP_STREAM_JOB_CONTROLLER)
      t=65 [st=65]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                          --> source_dependency = 1090 (HTTP_STREAM_JOB)
      t=65 [st=65]     -HTTP_STREAM_REQUEST
      t=65 [st=65]      URL_REQUEST_DELEGATE_SSL_CERTIFICATE_ERROR  [dt=1]
      t=66 [st=66]      CANCELLED
                        --> net_error = -200 (ERR_CERT_COMMON_NAME_INVALID)
      t=66 [st=66]   -URL_REQUEST_START_JOB
                      --> net_error = -200 (ERR_CERT_COMMON_NAME_INVALID)
      t=66 [st=66]    URL_REQUEST_DELEGATE_RESPONSE_STARTED  [dt=0]
      t=66 [st=66] -REQUEST_ALIVE
      
    • SSL_CONNECT_JOB gave me
      1087: SSL_CONNECT_JOB
      ssl/127.0.0.1:9443
      Start Time: 2020-01-29 08:41:25.699
      t= 1 [st= 0] +CONNECT_JOB  [dt=64]
      t= 1 [st= 0]    SOCKET_POOL_CONNECT_JOB_CREATED
                      --> backup_job = false
                      --> group_id = "ssl/127.0.0.1:9443"
      t= 1 [st= 0]   +SSL_CONNECT_JOB_CONNECT  [dt=64]
      t= 1 [st= 0]     +TRANSPORT_CONNECT_JOB_CONNECT  [dt=0]
      t= 1 [st= 0]        HOST_RESOLVER_IMPL_REQUEST  [dt=0]
                          --> address_family = 0
                          --> allow_cached_response = true
                          --> host = "127.0.0.1:9443"
                          --> is_speculative = false
      t= 1 [st= 0]        CONNECT_JOB_SET_SOCKET
      t= 1 [st= 0]     -TRANSPORT_CONNECT_JOB_CONNECT
      t=65 [st=64]      CONNECT_JOB_SET_SOCKET
      t=65 [st=64]   -SSL_CONNECT_JOB_CONNECT
                      --> net_error = -200 (ERR_CERT_COMMON_NAME_INVALID)
      t=65 [st=64] -CONNECT_JOB
      

Understanding Chromium trace and performance data

I found this link very useful to explain the developer information, such as trace, performance etc.