mqweb – what to do when you cannot get TLS to work?

It is hard to debug setup problems in mqweb.   I found it easiest to not use the mqweb trace, but diagnose problems from the client side.

You need to understand many TLS concepts.  I’ve documented a lot of information here: Understanding the TLS concepts for using certificates to authenticate in mqweb.

I found the easiest way to debug my mqconsole TLS setup, was to use extract the certificates from my browser’s key store and use curl’s verbose, or trace functions.   I’ve documented here how to get a Chrome trace.

I caused all of the common “user errors” and have documented the messages or symptoms I got, these are in this post.

Have you tried turning it off and on again?

The first thing you need to do if you have problems when you are configuring certificates is to restart mqweb, and your browser.   This is because updates to the keystores are not picked up till the mqweb or browser is restarted.  The Chrome and Firefox browsers, remember the certificate used, and logon this on again – so restart the browser to reset every thing.  With Chrome, I set up a bookmark url chrome://restart .

Once you have set up your first connection,  you should not need to change the mqweb server, as you will have set up the mqweb server certificate, and the CA certificate(s) to certify clients.  If you are using self signed,  you will have to import the SS certificate into the trust store, and restart the mqweb server (not good for high availability).

I found if I started chrome from a command window, instead of clicking on an icon, I got out some diagnostic messages to the command window.   These messages were slightly more useful than generic messages like “NET::ERR_CERT_AUTHORITY_INVALID”

Useful Chrome urls

• chrome://restart
• chrome://settings/certificates
• chrome://net-export/ – for collecting a Chrome trace

Getting started

If you are using .pem files (for example openssl) you can use these with no further work.

If you have a .p12 (pkcs12) format keystore, you can use this with no further work.

If you are using a browser with its nssdb database, you need to extract the certificate and private key, and any CA certificates you use.  It is easy to extract a certificate and key  into a .p12 keystore.

Extract the certificate and private key from your browser’s keystore

Curl can use the browser’s key store directly if it has been compiled with NSS (instead of openssl).  “Curl -V”, built with openssl gave me “libcurl/7.58.0 OpenSSL/1.1.1″, someone else’s curl, built with NSS had “libcurl/7.19.7 NSS/3.14.3.0″.  If you do not have curl with NSS support you need to extract the certificate and key from the browsers keystore.

• Check where your Chrome profile is.  In the Chrome browser, use the url chrome://version .   On one Chrome instance this was  /home/colinpaice/snap/chromium/986/.pki/nssdb .  On a different Chrome instance, the keystore was /home/colinpaice/.pki/nssdb .
• Export your certificate and keystore
  • pk12util -o colinpaicex.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -n colinpaice -W password
  • pk12util – invoke this program
    
  • -o colinpaicex.p12  – create this pkcs12 store
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/  – from this repository
  • -n colinpaice  – with this name
  • -W password  – and give it this password
• If you have created your own certificate authority, you need to extract the certificate if you do not already have it.  Firstly list the contents to remind yourself what the CA certificate is called, then extract the certificate (‘myCACert’ in my case)
  • certutil -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -L
    • This gives “Certificate Nickname ” and “Trust Attributes”.   Your CA should have a trust Attribute of “C”.
  • certutil -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -L -n “myCACert” -a >outcacert.pem
    
  • certutil – this program
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ – this key store
  • -L  – list
  • -n “myCACert”  – this name
  • -a – ASCII output
  • >outcacert.pem  – create this file

Issue the curl request

You can use the .p12 file, or the certificate.pem and the key.pem file

• Use the .p12 file and the CA certificate with curl
  • curl –cacert ./outcacert.pem –cert-type P12 –cert colinpaicex.p12:password “https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* ” –verbose
  • curl
  • –cacert ./outcacert.pem use this ca certificate
  • –cert-type P12 the keystore  is a P12 (PKCS12)
  • –cert colinpaicex.p12:password this is the keystore and password
  •   “https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* “
    this is the url
  • –verbose provide status messages
• Or use the certificate file, the keystore file and the CA certificate file.
  • curl –cacert ./cacert.pem –cert ./colinpaice.pem:password –key colinpaice.key.pem “https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* ” –verbose
  • curl
  • –cacert ./cacert.pem use this ca certificate
  • –cert ./colinpaice.pem:password use this certificate and password
  • —key colinpaice.key.pem  this is the private key
  • “https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* ”
    this is the url
  • –verbose provide status messages

Example output

If you use the option — verbose  you get a lot of information for example, a successful request has

• * Trying 127.0.0.1…
• * TCP_NODELAY set
• * ALPN, offering h2
• * ALPN, offering http/1.1
• * successfully set certificate verify locations:
• * CAfile: ./outcacert.pem
• CApath: /etc/ssl/certs
• * TLSv1.3 (OUT), TLS handshake, Client hello (1):
• * TLSv1.2 (IN), TLS handshake, Certificate (11):
• * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• * TLSv1.2 (IN), TLS handshake, Request CERT (13):
• * TLSv1.2 (IN), TLS handshake, Server finished (14):
• * TLSv1.2 (OUT), TLS handshake, Certificate (11):
• * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
• * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
• * TLSv1.2 (OUT), TLS handshake, Finished (20):
• * TLSv1.2 (IN), TLS handshake, Finished (20):
• * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
• * ALPN, server did not agree to a protocol
• * Server certificate:
• *   subject: C=GB; O=cpwebuser; CN=mqweb5
• *   start date: Jan 20 17:53:59 2020 GMT
• *   expire date: Oct 16 17:53:59 2022 GMT
• *   subjectAltName: host “127.0.0.1” matched cert’s IP address!
• *   issuer: C=GB; O=SSS; OU=CA; CN=SSCA7
• *  SSL certificate verify ok.
• > GET /ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* HTTP/1.1
• > Host: 127.0.0.1:9443

See here for an overview of the TLS handshake.   The amount of progress down the list of steps in the hand shake give you a clue as to where the problem may be.  If it is around “TLS handshake, Client Hello (1)”.  This is likely to be a problem with the server certificate.

The numbers as in TLS handshake, CERT verify (15): are the id number of the request, 15 is CERT verify.

A “Finished” message is always sent immediately after a change cipher spec message to verify that the key exchange and authentication processes were successful.  More checks are done after this.

If you use ‑‑trace filename.txt instead of ‑‑verbose you get the same data as displayed as with ‑‑verbose, plus the data flowing up and down the connection.  I found ‑‑verbose had sufficient details to resolve the problems.