Why cant I logoff from mqconsole?

If you are using mqweb using certificates to identify yourself, if you logoff, or close the tab, then open a new tab, you will get a session using the same certificate as before.

This little problem has been a tough one to investigate, and turns out to be lack of function in Chromium browser.

The scenario is you connect to mqweb using a digital certificate. You want to logoff and logon again with a different certificate, for example you do most of your work with a read only userid, and want to logon with a more powerful id to make a change.  You click logoff, and your screen flashes and logs you on again with the same userid as before.

At first glance this may look like a security hole, but if someone has access to your web browser, then the can click on the mqweb site, and just pick a certificate – so it is no different.

Under the covers,  the TLS handshake can pass up the previous session ID.   If the server  recognises this, then it is a short handshake instead of a full hand shake, so helping performance.

To reset the certificate if you are using Firefox

To clear your SSL session state in Firefox choose History -> Clear Recent History… and then select “Active Logins” and click “OK”. Then the next time you connect to your SSL server Firefox will prompt for which certificate to use, you may need to reset the URL.

You should check Firefox preferences, certificates, “Ask you every time” is selected, rather than “Select one automatically”.

Chrome does not support this reset of the certificate.

There has been discussion over the last 9 years along the lines of, seeing as Internet Explorer, and Firefox have there, should we do it to met the end user demand?

If you set up an additional browser instance, you get the same problem. With Chrome you have to close down all instances of the browser and restart chrome to be able to select a different certificate.

It looks like there is code which has a cache of url, and certificate to use.   If you open up another tab using the same IP address you will reuse the same certificate.

If you localhost instead of 127.0.0.1 – it will prompt for certificate, and then cache it, so you can have one tab open with one certificate, and another tab, with a different URL and another certificate.

Whoops my certificate has expired – what do I need to do?

The amount of work depends on what has expired.

  • If the root CA has expired then you need to reissue all certificate which include the root CA, and update trust stores
  • If the enterprise CA has expired you need to renew all certificates signed by the enterprise CA, and update trust stores with the new certificate
  • If your personal certificate has expired, you need to renew that certificate.

The amount of work also depends on the size of your enterprise.

I’ll list the work need if the certificate is going to expire next week – as this is more work than if it has already expired

If your personal certificate is going to expire

You need to

  • renew it, create a new private key and certificate request
  • or extend the date of the existing certificate  – by re-signing it
  • update the keystores which include it.  This could be those used by java programs ( .p12, or .jks) as well as any browser keystore (.nssdb)

To renew your personal certificate

You can just recreate it.  If you are considering to use a stronger algorithms, such as Elliptic Curves, bear in mind that the servers need to be able to support what you specify.

Depending on how your openssl has been configured, if you have unique_subject = yes you may need to use openssl revoke… to remove the old personal certificate from the configuration.   This is to ensure there is only one certificate with the same distinguished name, and online checking of certificate validity (Certificate Revocation Lists and Online Status Certificate Protocol) can be used.  If you have unique_subject = no, you can recreate the certificate without revoking it.

  • openssl genpkey  to generate a new private key
  • openssl req to create a request

You do not need to recreate it – you can just sign it again.

You then need to do normal process of update the users of the certificate.

  • openssl ca to sign the request
  • openssl pkcs12 -export to create the .p12 file
  • You may want to deploy it only when the machine is attached to the local network.  If you download it over wireless, or email, these may be compromised, and the bad guys can just use the new copy.  Downloading it over a wired connection eliminates this risk.
  • certutil -D $sql -n $shortName to remove the certificate from the browser’s keystore
  • pk12util -i $shortName.p12 $sql -W password to add the certificate to the browser’s keystore
  • /opt/mqm/bin/runmqckm -cert -delete  to remove a certificate from a keystore (.jks)
  • /opt/mqm/bin/runmqckm -cert -import to add a certificate to a keystore(.jks)
  • restart the servers to pick up the new certificate

Rather than deleting the certificate from the keystore and adding the new one with the same short name,  you may want to make a copy of the old certificate from the keystore, or simply backup the whole keystore – as it should not change once frequently.  If there are problems, restore from the backup.

To renew your enterprise certificate

This requires a lot more work than updating a personal certificate.

Just like the personal certificate you can recreate it, or just sign it again.

You need to phase in the new certificate as it may take days or weeks to deploy it successfully.

The client’s trust store needs the CA certificate matching the CA used by the server’s certificate.

If you have two servers, and change servers to use the new CA certificate, the client’s trust store content will change over time.

  • The old CA certificate – before any work is done
  • Both old and new CA certificates during the migration period.  The server using the old CA will use the old CA in the client.   The server which has been migrated to use the new CA will use the new CA in the client.
  • Just the new CA certificate.  Once the migration has finished. The old certificate can be removed when all of the servers have been migrated, or the certificate has expired.

A key store and a trust store can have certificate with the same distinguished names, but with a different short name or alias name.  My enterprise is called SSS, and the CA for my enterprise is CASSS… .  My trust store has

  • the CA with C=GB,O=SSS,OU=CA,CN=CASSS and short name CASSS2016 and
  • the CA with C=GB,O=SSS,OU=CA,CN=CASSS and short name CASSS2020 and

When you maintain a keystore you use the short name, for my example CASSS2016 or CASSS2020.

During the TLS handshake, when the server’s certificate is sent to the client, it is checked against all of the trusted CA certificates in the keystore, so having two certificates with the same DN does not matter.

The steps to deploy your new CA are

  • Get the new CA.  Either create a new CA using the latest algorithms and cipher suites (the better solution), or have the old certificate resigned.
  • For each user and server
    • Deploy the CA certificate into the trust stores with a new name – eg CASSS2020
  • For each user and server
    • Renew or recreate each personal and server certificate and sign it with the new CA
    • Deploy it securely to each user and server and replace the old personal certificate
    • You may want to deploy the private key only when attached to the local network.  If you download it over wireless, or email, these may be compromised, and the bad guys can just use the new copy.  Downloading it over a wired connection eliminates this risk.
  • For the server create java.security rules to disable weaker algorithms and cipher specs.   It is easy to undo this change.
    • This should identify any user not using the latest cipher suites and certificates.  These rules can be relaxed while problems are fixed.
  • For each user and server
    • Make a copy of the old CA certificate prior to deletion
    • Delete the old CA certificate from the trust stores eg CASSS
    • After a validation period delete the copy of the old certificate

Start all over again!

You can see how much work you need to do when you renew your enterprise CA, so make sure you  renew it in plenty of time – months rather than days.

mqweb error messages and symptoms of TLS setup problems

I deliberately caused TLS set up errors, and noted the symptoms.  Ive recorded them below; the article is not meant to be read, but indexed by search engines.

There are three sections

  1. Problems with server certificates
  2. Problems with the client certificate
  3. Chrome messages, and possible causes of the problems.

The mqweb messages.log reported problems that the mqweb server saw.   For me this was in file /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log

Problems with the server certificate

Problem: mqwebuser.xml serverKeyAlias name not in the keystore

Message log:

  • E CWPKI0024E: The certificate alias mqweb specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore /home/colinpaice/ssl/ssl2/mqweb.p12.
  • I FFDC1015I: An FFDC Incident has been created: “com.ibm.wsspi.channelfw.exception.ChannelException: java.lang.IllegalArgumentException: CWPKI0024E: The certificate alias mqweb specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore /home/colinpaice/ssl/ssl2/mqweb.p12. com.ibm.ws.channel.ssl.internal.SSLConnectionLink 238″ at ffdc_….

curl:

* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* curl (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443
* stopped the pause stream!
* Closing connection 0

chrome:

This site can’t be reached.  ERR_CONNECTION_CLOSED

Problem:  The host certificate is self signed and not in the client keystore

Problem:  The host certificate is signed but the signer certificate is not in  the client keystore

Message log:

Nothing.

curl:

* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate

Chrome: in browser

NET::ERR_CERT_AUTHORITY_INVALID

Click on the Not Secure in the url, to display the certificate which was sent down.

If it is signed, make a note of the “issued by” Common Name(CN), and the  Organisation(0) and look up the value of Organisation in the “Authorities” section of “Manage Certificates”.

Use the chrome url chrome://settings/certificates .  Authorities tab

    1. if it is not present, import it
    2. it it is present and UNTRUSTED, edit it, and tick the “Trust this certificate for identifying web sites”

Chrome log:

ERROR:cert_verify_proc_nss.cc(1011)] CERT_PKIXVerifyCert for localhost failed err=-8179

From here  -8179 is Peer’s certificate issuer is not recognized.

Firefox:  browser

SEC_ERROR_UNKNOWN_ISSUER

Action import the CA signing certificate into the keystore and make it trusted.

Problem: curl: The host certificate is self signed and you use the –insecure option

curl

* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=GB; O=aaaa; CN=testuser
* start date: Jan 20 17:39:37 2020 GMT
* expire date: Feb 19 17:39:37 2020 GMT
* issuer: C=GB; O=aaaa; CN=testuser
* SSL certificate verify result: self signed certificate (18), continuing anyway.

Problem: Chrome:  The host certificate is self signed and is not trusted

Chrome browser

This site can’t be reached
localhost unexpectedly closed the connection.
ERR_CONNECTION_CLOSED

Debugging

  • I could find nothing that told me what certificate was being used.  The Chrome network trace just gave “net_error = -100 (ERR_CONNECTION_CLOSED)“.
  • Use certutil -L $sql  to list the contents of your browsers keystore.   The certificate needs “P,…” permissions.
  •  Or use the chrome url chrome://settings/certificates  and display “your certificates”. Pick the likely one, if it says “UNTRUSTED” then this may be the problem.   View the certificate, and check it, for example under details, there may be a comment describing its use.
  •  Defined the server certificate as trusted using certutil -M $sql -n name -t “P,,” 
  • Restart the web browser.

Problem: The  CA signer server certificate had the wrong subjectAltName

curl:

* subjectAltName does not match 127.0.0.1
* SSL: no alternative certificate subject name matches target host name ‘127.0.0.1’

Chrome:

NET::ERR_CERT_COMMON_NAME_INVALID
From the “Not Secure” in front of the URL, display the certificate, and check the extenstions, especially Certificate Subject Alternative Names.

Chrome log:

ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -200
From here -200 is  CERT_INVALID

Problem: The mqweb server certifcate has expired

curl:

* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
curl: (60) SSL certificate problem: certificate has expired

chrome:

while Chrome running:   web page reports Lost communication with the server.  Could not establish communication with the server. Check your network connections and refresh your browser

restart browser, get “Your connection is not private NET::ERR_CERT_DATE_INVALID”

message.log.  Chrome session was working, then server certificate expired

  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

Problem: The mqweb server certificate is missing extendedKeyUsage = serverAuth

curl:

* SSL certificate problem: unsupported certificate purpose
curl: (60) SSL certificate problem: unsupported certificate purpose

Chrome:

Your connection is not private
Attackers might be trying to steal your information from localhost (for example, passwords, messages or credit cards).
NET::ERR_CERT_INVALID

Chrome log:

CERT_PKIXVerifyCert for localhost failed err=-8101
From here  -8101 is Certificate type not approved for application.

ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -207
From here -207 is CERT_INVALID

 

Problems with the server ca certificate

Problem: The trust store has an expired CA.

curl:

* gnutls_handshake() failed: The TLS connection was non-properly terminated.

pycurl.error: (35, ‘gnutls_handshake() failed: The TLS connection was non-properly terminated.’)

Problems with the client certificate

Problem: There is no suitable certificate in the client keystore.

For example

  1. There are no “Your certificates” in the browsers keystore
  2. There is a certificate, but has a CA which was not passed down from the server trust keystore
  3. As part of the TLS handshake any self signed certificates are read from the server trust keystore and sent down.  None were found in the “Your certificates”

Curl:

  • * gnutls_handshake() failed: The TLS connection was non-properly terminated.
  • pycurl.error: (35, ‘gnutls_handshake() failed: The TLS connection was non-properly terminated.’)

These messages basically mean the server just ended the connection

Chrome:

ERR_CONNECTION_CLOSED

For a test site, change <ssl clientAuthentication=”true” to false.  Restart mqweb, restart the web browser.  If it prompts for userid and password, the certificate sent from the server was OK.  It is the certificate sent up to the server that has a problem.

Reset false back to true.

Messages in messages.log:

None.

How to debug it.

Check the logs/ffdc directory.  I found I had an ffdc with Stack Dump = java.security.cert.CertPathValidatorException: The certificate issued by CN=SSCA8, OU=CA, O=SSS, C=GB is not trusted; internal cause is:   java.security.cert.CertPathValidatorException: Signature does not match.

Using Chrome trace

When I repeated the investigations, I got different records in the Chromium trace.  One included

--> net_error = -110 (ERR_SSL_CLIENT_AUTH_CERT_NEEDED)

Using the mqweb server java trace – which traces the whole server

See the Oracle Debugging SSL/TLS Connections page and an IBM page.  I could not see how to trace just “the problem”.

With -Djavax.net.debug=ssl:handshake in the jvm.options file, and restarting the mweb server I got

 *** ServerHelloDone
Default Executor-thread-8, WRITE: TLSv1.2 Handshake, length = 3054
Default Executor-thread-2, READ: TLSv1.2 Handshake, length = 7
*** Certificate chain
***
Default Executor-thread-2, fatal error: 40: null cert chain

When it worked I had

*** ServerHelloDone
Default Executor-thread-7, WRITE: TLSv1.2 Handshake, length = 3054
Default Executor-thread-15, READ: TLSv1.2 Handshake, length = 2433
*** Certificate chain
chain [0] = […. the  certificates

Found trusted certificate:

When there was no certificate sent up,  it reported null cert chain.

Problem: The client certificate is self signed and not in the server’s trust store

curl:

* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443

Chrome:

ERR_CONNECTION_CLOSED

Messages in messages.log:

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_08.29.27.0.log
  •  E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=testuser, O=aaaa, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target
  •  I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: unable to find valid certification path to requested target com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_08.29.27.1.log
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

Problem: Invalid cn=, the cn value is not a valid userid.

curl message

{“error”: [{

  • “action”: “Provide credentials using a client certificate, LTPA security token, or username and password via HTTP basic authentication header. On z/OS, if the mqweb server has been configured for SAF authentication, check the messages.log file for messages indicating that SAF authentication is not available. Start the Liberty angel process if it is not already running. You might need to restart the mqweb server for any changes to take effect.”,
  • “completionCode”: 0,
  •  “explanation”: “The REST API request cannot be completed because credentials were omitted from the request. On z/OS, if the mqweb server has been configured for SAF authentication, this can be caused by the Liberty angel process not being active.”,
  • “message”: “MQWB0104E: The REST API request to ‘https://127.0.0.1:9443/ibmmq/rest/v1/login ‘ is not authenticated.”,
  • “msgId”: “MQWB0104E”,
  • “reasonCode”: 0,
  • “type”: “rest”

chrome:

It gives you a window to enter userid and password.   This looks like a bug as I have <webAppSecurity allowFailOverToBasicAuth=”false”/>.  It takes the userid and password.

Messages in  messages.log:

R com.ibm.websphere.security.CertificateMapFailedException
and 100 lines of stack trace

The certificate causing the problems, nor the userid is listed – so pretty useless.

Problem: Client certificate missing “extendedKeyUsage = clientAuth”  during signing.

curl message

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
curl session hangs…
* Operation timed out after 300506 milliseconds with 0 out of 0 bytes received

Chrome

ERR_CONNECTION_CLOSED

message in messages.log:

  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: Extended key usage does not permit use for TLS client authentication
  •  I FFDC1015I: An FFDC Incident has been created: “java.lang.NullPointerException com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.28_17.11.10.1.log

ffdc in /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log/ffdc

Exception = java.lang.NullPointerException
Source = com.ibm.ws.ssl.core.WSX509TrustManager
probeid = checkClientTrusted
Stack Dump = java.lang.NullPointerException
at com.ibm.ws.ssl.core.WSX509TrustManager.checkClientTrusted(WSX509TrustManager.java:202)

Problem: Client certificate missing “keyUsage = digitalSignature”  during signing.

curl message

* TLSv1.2 (OUT), TLS handshake, Finished (20):
* Operation timed out after 300509 milliseconds with 0 out of 0 bytes received

message in messages.log

  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: KeyUsage does not allow digital signatures
  • FFDC1015I: An FFDC Incident has been created: “java.lang.NullPointerException com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted”
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

ffdc in /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log/ffdc

Exception = java.lang.NullPointerException
Source = com.ibm.ws.ssl.core.WSX509TrustManager
probeid = checkClientTrusted
Stack Dump = java.lang.NullPointerException
at com.ibm.ws.ssl.core.WSX509TrustManager.checkClientTrusted(WSX509TrustManager.java:202)

Chrome:

  • If there is one or more certificates in the keystore, the list of valid certificates does not include the problem one.
  • If there is only the problem certificate in the keystore, you get
    This site can’t be reached.
    localhost unexpectedly closed the connection.
    ERR_CONNECTION_CLOSED

CA Signed client certificate has expired

curl:

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443
* stopped the pause stream!
* Closing connection 0

Chrome:

This site can’t be reached
localhost unexpectedly closed the connection.
ERR_CONNECTION_CLOSED

message in messages.log:

for curl.

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathValidatorException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020; internal cause is:
    java.security.cert.CertificateExpiredException: NotAfter: Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.0.log
  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path validation failed: java.security.cert.CertPathValidatorException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020; internal cause is:
    java.security.cert.CertificateExpiredException: NotAfter: Thu Jan 30 16:46:00 GMT 2020
  •  I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.1.log

for chrome:

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: The cer
    tificate expired at Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.1.log
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

Bad requests

HTTP request was issued – it should have been HTTPS

curl:

curl:(52) Empty reply from server

messages.log:

E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

 

The client certificate cannot be verified because it is too weak.

Chrome:  ERR_BAD_SSL_CLIENT_AUTH_CERT

Firefox:  An error occurred during a connection to …  security library: memory allocation failure.  Error code: SEC_ERROR_NO_MEMORY

Reason:

The selected client certificate cannot be validated.  For example it has been created with Elliptic Curve sect409k1.   This is considered weak see here.  The signature is not in the list of acceptable signatures.

Display the certificate and compare it with the list of weak signatures.  A TLS handshake trace may help identify this.  Create a new certificate with a supported signature, and import it.

 

Problem the CA signing is too weak.

For example signing with sha1RSA, when Chrome expects SHA256RSA or stronger.

Chrome:  NET::ERR_CERT_WEAK_SIGNATURE_ALGORITHM

Firefox: I didnt get the error

Action: Use stronger signing.  For example on z/OS use RSA SIZE(2048)

Firefox errors

Your computer clock is set to … . Make sure your computer is set to the correct date, time, and time zone in your system settings, and then refresh …

If your clock is already set to the right time, the web site is likely misconfigured, and there is nothing you can do to resolve the issue. You can notify the web site’s administrator about the problem.

… uses an invalid security certificate.

The certificate is not trusted because the issuer certificate has expired.

Error code: SEC_ERROR_EXPIRED_ISSUER_CERTIFICATE

Reason:

The CA certificate in the trust store has expired.  The a valid CA certificate may have been sent down with the server’s certificate, but the validation failed.

Action:

  1. From Warning: Potential Security Risk Ahead -> Advanced -> View certificate. It will have the certificate.  Note Issuer -> Organisation and common name
  2. Use Firefox preferences-> view certificates.   Select authorities.  Search for the Organisation from the previous line.  Display the certificate with the matching common name.  Replace it and restart the browser.   Replace the certificate through firefox or use this to locate the directory containing the cert9.db.

Error code: SSL_ERROR_CERTIFICATE_UNKNOWN_ALERT

The backend may get java.security.cert.CertPathValidatorException: signature check failed.

One reason, the certificate being used by firefox was signed by an invalid CA, for example the CA had expired.

Action:

  1. Check Firefox preferences-> certificates, and check “Ask you every time” is selected, repeat the connection and display information about the certificate.  It will give you the issuer, but no more information than that.
  2. Regenerate the certificate, import into Firefox, restart Firefox.

Chrome errors

Chrome has more stricter checks than curl.  These are from Chrome browser.

NET::ERR_CONNECTION_CLOSED

  • mqwebuser.xml serverKeyAlias name not in the keystore
  • The host certificate is self signed and is not trusted
  • The client certificate is self signed and not in the server’s trust store
  • Client certificate missing “extendedKeyUsage = clientAuth”  during signing.
  • CA Signed client certificate has expired
  • Client certificate missing “keyUsage = digitalSignature”  during signing.

NET::ERR_CERT_COMMON_NAME_INVALID

  • missing x509 extensions in the server certificate
  • invalid subjectAltName in x509 extensions, for example IP:127.0.0.11  instead of IP:127.0.0.1

NET::ERR_CERT_INVALID

  • missing extendedKeyUsage = serverAuth in x509 extensions

NET::ERR_CERT_AUTHORITY_INVALID

  • Certificate is not peer.  Need certutil -M $sql -n $name -t “P,,” to change the certificate to be a trusted peer
  • Server’s self signed not found in the browser keystore.
  • The CA from the server does not match the certificate in the browsers’ keystore.  It may have the same name,  but check validity dates, finger prints etc.  Check very carefully.

NET::ERR_CERT_DATE_INVALID

  • The mqweb server certificate has expired.

 

CWPKI0024E: The certificate alias …  specified by the
property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore safkeyring://…/….

The z/OS certificate is not in the keyring, or it is in the keyring and needs to have TRUST

Make the change, stop and restart the web browser

 

Firefox:  PR_END_OF_FILE_ERROR

Slow backend server.

 

MQWEB on z/OS

 CWWKS2932I: The unauthorized version of the SAF user registry is activated.
Authentication will proceed using unauthorized native services.

Check at the top of the message log for.  CWWKB0104I: Authorized service group SAFCRED is not available.

Reason: When the web server was started the SAFCRED service was not available.   This could be caused by security not set up properly.

Fix the security.  For example here

CWWKS2930W: A SAF authentication attempt using authorized SAF services was rejected because the server is not authorized to
authorized to access the APPL-ID MQWEB. Authentication will proceed using unauthorized SAF services.

Problem:  the profile with class(SERVER) and profile(BBG.SECPFX.MQWEB) is missing
Action:  the define profile matching the APPL-ID.

RDEFINE SERVER BBG.SECPFX.MQWEB
PERMIT BBG.SECPFX.MQWEB  CLASS(SERVER) ID(START1) ACC(READ)
SETROPTS RACLIST((SERVER) refresh

Restart MQWEB server.

CWWKS2960W: Cannot create the default credential for SAF authorization of unauthenticated users.

All authorization checks for unauthenticated users will fail.
The default credential could not be created due to the following error:

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user WSGUEST has insufficient authority to access APPL-ID MQWEB.

SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

PERMIT MQWEB CLASS(APPL) ACCESS(READ) ID(MQWSGUEST)
SETROPTS RACLIST(APPL) REFRESH

 

CWPKI0022E: SSL HANDSHAKE FAILURE:

A signer with SubjectDN CN=colinpaicesECp256r1, O=cpwebuser,
C=GB was sent from the target host. The signer might need to be added to local trust store safkeyring://…/…,
located in SSL configurate on alias defaultSSLConfig.
The extended error message from the SSL handshake exception is:

Unexpected error: java.security.InvalidAlgorithmParameterException:
the trustAnchors parameter must be non-empty

Problem: The CA certificate was added to the keyring via SITE, and not CERTAUTH.

Action: Delete the CA, import it as a CERTAUTH certificate, connect it to the trust ring.

 

ERROR: SEC_ERROR_REUSED_ISSUER_AND_SERIAL

I got this on a slow backend system.  I shut down the web server and restarted it, and it ran OK without the message.

 

ICH408I USER( ) GROUP( ) NAME()
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(…)
SUBJECT(CN=.. .O=… C=GB) ISSUER(….)

The certificate came in, but there was no mapping for it.

Use RACDCERT command to map it to a userid.

RACDCERT MAP ID(IBMUSER) –
SDNFILTER(‘CN…. ‘)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

 

 

Using IBM api explorer to display the syntax of mq rest commands.

You can use an url like https://localhost:9443/ibm/api/explorer  to display the mqweb api documentation.  The documentation is in swagger format, available in a web browser.   This is the same information you get from a REST request see here.

If you enter the url in the web browser, and you are using certificate authentication, you will be prompted with a list of valid certificates.  Any certificate is valid.   Even one with a CN=unknown was accepted.   The information is read only so this is not an issue.

The web page has

  • API Discovery
  • channel
  • login

If you click login it gives

  • DELETE /ibmmq/rest/v1/login  
    • Logs out a user
  • GET   /ibmmq/rest/v1/login
    • Queries the authenticated user and their roles.
  • POST  /ibmmq/rest/v1/login
    • Logs out a user

If you click on an item it displays more information, in a nice, easy to read format.  Much easier to read than if you retrieved it using the REST API.

What it doesn’t tell you

It does not tell you to put ?attributes=*&status=* on the end of your query to get queue attributes, and qstatus returned, as in https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=*See here for more information.

Using the url as a REST request.

Using the url from curl and –user colinpaice:password gave

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:9443

Using certificate authentication nothing was returned.

Using certificate authentication and the –include option (include headers)

url –include https://localhost:9443/ibm/api/explorer –cacert cacert.pem –cert colinpaice.pem:password –key colinpaice.key.pem

gave

HTTP/1.1 401 Unauthorized

So using the explorer url is not allowed.

mqweb – how to use the REST api commands.

You can issue MQ commands using the REST API vai the mqweb server, for example with python or curl.  The problem I had was trying to remember the syntax of the commands.  You can access this information from your program by using the “swagger” interface.  You can can access it from a browser (easiest), or from a REST request (for the experts).

I found it useful to use a curl query logon request to check the validity of the certificates I was using.  I could use the –trace curl.trace.txt to capture the data flows.  If this worked, problems were likely to be in the browser set up, rather than the certificate set up.

Note: Swagger is the name of a protocol, not an acronym.

You need to configure mqweb to use Swagger.

<featureManager>
  <feature>apiDiscovery-1.0</feature>
</featureManager>

Then use a request like (for certificate authentication)

curl -i https://localhost:9443/ibm/api/docs –cacert cacert.pem –cert colinpaice.pem:password –key colinpaice.key.pem > swagger.out

where

  • curl – issue the command
  • -i  – include the http headers in the response.  This is the same flag as –include
  • https://localhost:9443/ibm/api/docs – this url
  • –cacert cacert.pem – use this CA certificate
  • –cert colinpaice.pem:password – use this user certificate
  • –key colinpaice.key.pem – use this user private key
  • > swagger.out – write the output to this file

If you are using self signed you should use the –insecure option otherwise you get

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

If you try to userid and password you get

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:9443

The swagger output

The output contains

{
  "swagger" : "2.0",
  "info" : {
    "description" : "Discover REST APIs available within Liberty",
    "version" : "1.0.0",
    "title" : "Liberty REST APIs"
},
  "host" : "localhost:9443",
  "tags" : [ {
    ...
    "name" : "qmgr"
    ...
    "name" : "login"
    ...
    "name" : "API Discovery",
    "description" : "APIs available from the API Discovery feature"
} ],
...

We can see there is a login command.  You have to search the json for the data with “tags”  : [“login”].

The section for login has

"/ibmmq/rest/v1/login" : {
  "get" : {
    "tags" : [ "login" ],
    "summary" : "Queries the authenticated user and their roles.",
    "description" : "Retrieves details of the authenticated user and their roles.",
    "operationId" : "getLogin",
    "produces" : [ "application/json; charset=utf-8" ], 
    "parameters" : [ ],
    "responses" : {
      "200" : {
         "description" : "A JSONArray that contains JSONObjects that describe the authenticated user.",
         "schema" : {
            "$ref" : "#/definitions/UserAttributePojo"
              }
     },
...

“get”

Within the partial url /ibmmq/rest/v1/login are sections “get”, “post”, and “delete”:

  • “get” –   Queries the authenticated user and their roles  – as the “summary” tag says
  • “post”  – Authenticates a user and returns an authentication token.
  • “delete” – Deletes the authentication token of the logged in user.

You need to pick the section for what you want to do within “get”, “post”, “delete”

“tags” : [ “login” ]

This is for the request.

“200”

This is one of the list of return codes from the http request, and the data each returns.

This returns data as defined in the schema, “$ref” : “#/definitions/UserAttributePojo” .  Look in the rest of the document for this section called definitions, with UserAttributePojo underneath.

“definitions”: UserAttributePojo

 "definitions" : {

...
 "UserAttributePojo" : {
    "type" : "object",
    "required" : [ "name", "role" ],
    "properties" : {
      "name" : {
      "type" : "string"
      },
     "role" : {
       "type" : "array",
        "items" : {
          "type" : "string"
         }
      }
  }

Two strings will be returned, name and role.

Issue the command

From the data in the swagger output,  we can see  “host” : “localhost:9443” and “/ibmmq/rest/v1/login” : { “get” : {

build the command

https://localhost:9443/ibmmq/rest/v1/login –request get

my command is now

curl -i https://localhost:9443/ibmmq/rest/v1/login –cacert cacert.pem –cert colinpaice.pem:password –key colinpaice.key.pem –trace trace.txt –request get

note: -request get is inferred and can be allowed to default.

and it returns

{"user": [{
  "name": "colinpaice",
  "role": [
    "MQWebAdmin",
    "MQWebAdminRO",
    "MQWebUser"
  ]
}]}

If you want to pass data with quotes in it – for example json data. Create a file, and use that
for example
curl -i …  –request post -H “Content-Type: application/json” -d @curl.json.txt 

or use

-d "{\"param\":\"value\"}"

which is not so easy.

Setting up the end user self signed certificate for mqweb

This post describes how to set up self signed certificates to authenticate end user’s access to mqweb.

You can use self signed, which is fine for test and small environments, or use signed certificate which are suitable for production,  and typical environments.  Using certificates means you do not need to specify userid and password.

The userid is taken from the CN part of the subject, and this userid is used to grant access depending on the configuration in the mqwebuser.xml file.

Information about certificates used for authentication are stored in the trust store.  For a CA signed certificate, you only need the CA certificates, not the individual certificates.  With self signed, you need a copy of the individual self signed certificate in the mqweb trust store.

Create the trust store if required.

/opt/mqm/bin/runmqckm -keydb -create -db trust.jks -pw zpassword -type jks

You need to do this once.

Create the self signed certificate for the end user

openssl req -x509 -config openssl-client2.cnf -newkey rsa:2048 -out testuser.pem -keyout testuser.key.pem -subj “/C=GB/O=aaaa/CN=testuser” -extensions ss_extensions -passin file:password.file -passout file:password.file

  • openssl req -x509  –x509 makes this a self signed request
  • -config openssl-client2.cnf – use this config file
  • -newkey rsa:2048 – create a new private key with 2048 bits rsa key
  • -out testuser.pem – put the request in this file
  • -outform PEM – with this format
  • -keyout testuser.key.pem – put the key in this file
  • -subj “/C=GB/O=aaaa/CN=testuser” – this is the DN.   The CN= is the userid used by mqweb to determine the role.  It must match the case of userid
  • -extensions ss_extensions – see below
  • -passin file:password.file -passout file:password.file – passwords are in this file
[  ss_extensions  ]

subjectKeyIdentifier = hash
#Note: there is a bug in Chrome where it does 
# not accept certificates if basicConstraints
# is specified
# basicConstraints   = CA:false
subjectAltName       = DNS:localhost, IP:127.0.0.1
nsComment            = "OpenSSL Self signed Client"
keyUsage             = critical, keyEncipherment
extendedKeyUsage     = critical, clientAuth

Create an intermediate pkcs12 keystore so it can be imported to your browser.

You need to import the certificate and private key into the browser’s keystore.   The only way I found of doing this was via an intermediate pkcs12 keystore (with extension .p12).   If you import the certificate and key from the web browser, it will expect a .p12 file.

openssl pkcs12 -export -in testuser.pem -inkey testuser.key.pem -out testuser.p12 -name “testuser” -passin file:password.file -passout file:password.file

  • openssl pkcs12 – request to process a pkcs12 keystore
  • -export – to create it
  • -inkey testuser.key.pem – this private key
  • -in testuser.pem – this certificate returned from the CA
  • -out ssks.p12 – the name of the key store which is created
  • -name testuser – create this name in the keystore
  • -passout file:password.file -passin file:password.file – use these passwords

Import the intermediate keystore into the trust store

/opt/mqm/bin/runmqckm -cert -import -target trust.jks -target_type jks -file testuser.p12 -label testuser -pw password -target_pw zpassword

  • /opt/mqm/bin/runmqckm – run this command
  • -cert – we want to process a certificate
  • -import – w want to import a file
  • -target trust.jks – this is the mqweb trust store
  • -target_type jks – it is a jks store
  • -file testuser.p12 – input file
  • -label testuser – this is the label in the trust.jks keystore
  • -pw password – the password of the testuser.p12
  • -target_pw zpassword – the password of the trust.jks keystore

In the message.log I had

E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=testuser, O=aaaa, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path validation failed: java.security.cert.CertPathValidatorException: The certificate issued by CN=testuser, O=aaaa, C=GB is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error

When I restarted mqweb and the certificate was accepted.

I had the same message when I did not import the certificate into the trust store.

Import the temporary keystore into the Chrome keystore

pk12util -i testuser.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -W password

  • pk12util   – this command
  • -i  testuser.p12  – from this keystore
  • -d sql:/home/colinpaice/snap/chromium/971/.pki/nssdb/ – into this key store
  • -W password  – using this password (for the temporary .p12 file)

Remove the intermediate file

rm testuser.p12

Update the mqweb configuration

<webAppSecurity allowFailOverToBasicAuth="false" />
<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/ssl2/mqweb.p12" 
          type="pkcs12" 
          password="password"/>

<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/ssl2/trust.jks" 
          type="JKS" 
          password="password"/>

<ssl     id="defaultSSLConfig" 
         keyStoreRef="defaultKeyStore" serverKeyAlias="mqweb" 
         trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
         clientAuthentication="true" 
         clientAuthenticationSupported="true" 
/>

Stop mqweb

You need to restart mqweb so it picks up any changes to the trust store.

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

No messages are produced in  /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log if the trust store was opened successfully.

Use a command like grep ” E ” messages.log  and check for messages like

CWPKI0033E: The keystore located at /home/colinpaice/ssl/ssl2/trust.jks did not load because of the following error: Keystore was tampered with, or password was incorrect

Try using it in Chrome

You need to restart Chrome to pick up the changes to its keystore.  Use the url chrome://restart/

Use the url chrome://settings/certificates , to check your certificate is present under “Your certificates”. If not use url chrome://version to display the profile being used, and that it matches the store used in the pk12util command above.

Try connecting to mqweb using a url like https://127.0.0.1:9443/ibmmq/console/ .

You should be logged on with no password request. In the top right hand corner of the screen you should have a black circle with a white “i” in it.   This shows you are logged on with certificates.

Setting up the end user CA signed certificate for mqweb

You want to use certificates to authenticate access to a mqweb server.  You can use self signed, which is fine for test and small environments, or use signed certificate which are suitable for production,  and typical environments.  Using certificates means you do not need to specify userid and password.

The userid is taken from the CN part of the subject, and this userid is used to grant access depending on the configuration in the mqwebuser.xml file.

This section assumes you have set up your mqweb using a certificate authority.

Information about certificates used for authentication are stored in the trust store.  For a CA signed certificate, you only need the CA certificates, not the individual certificates.  With self signed, you need a copy of the individual self signed certificate.

Create the trust store if required.

/opt/mqm/bin/runmqckm -keydb -create -db trust.jks -pw zpassword -type jks

You need to do this once.

Add the CA certificate to the trust store

/opt/mqm/bin/runmqckm -cert -add -db trust.jks -file cacert.pem -label CACert -type jks -pw zpassword

You need to do this for each CA certificate you want to add, giving each CA a unique label.

You need to restart mqweb so it picks up any changes to the trust store, but as you will be changing the mqwebuser.xml – the restart can wait will later.

Create the certificate request for the end user

openssl req -config client.cnf -newkey rsa:2048 -out colinpaice.csr -outform PEM -keyout colinpaice.key.pem -subj “/C=GB/O=cpwebuser/CN=colinpaice” -extensions client_extensions -passin file:password.file -passout file:password.file

  • openssl req – the absence of -x509 makes this a certificate request
  • -config client.cnf – use this config file
  • -newkey rsa:2048 – create a new private key with 2048 bits rsa key
  • -out colinpaice.csr – put the request in this file
  • -outform PEM – with this format
  • -keyout colinpaice.key.pem – put the key in this file
  • -subj “/C=GB/O=cpwebuser/CN=colinpaice” – this is the DN.   The CN= is the userid used by mqweb to determine the role.  It must match the case of userid
  • -extensions client_extensions – see below
  • -passin file:password.file -passout file:password.file – passwords are in this file
[ client_extensions ]

subjectKeyIdentifier = hash
# basicConstraints = CA:FALSE
subjectAltName       = DNS:localhost, IP:127.0.0.1
nsComment = "OpenSSL ColinClient"
keyUsage = critical, nonRepudiation,digitalSignature,
extendedKeyUsage = critical, clientAuth

You need to do this for each user.

Sign it

Send the certificate request to your CA.  You can use the following command to sign it.

openssl ca -config openssl-ca-user.cnf -policy signing_policy -extensions signing_req -md sha256 -keyfile cacert.key.pem -out colinpaice.pem -infiles colinpaice.csr

  • openssl ca – the signing request
  • -config openssl-ca-user.cnf – use this config file
  • -policy signing_policy – defines the requirements for the DN.  See below
  • -extensions signing_req – see below
  • -md sha256 – what encryption to be used for the message digest
  • -keyfile cacert.key.pem – the CA authorities private key
  • -out colinpaice.pem – where the output goes
  • -infiles colinpaice.csr – the input file that needs signing

Send the *.pem file back to the requestor.

You need to do this for each user.

The signing policy allows the CA administrator to define which elements are required in the DN.

[ signing_policy ]
organizationName = supplied
commonName = supplied

The certificate needs extensions which say how the certificate can be used.

[ signing_req ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints    = CA:FALSE
keyUsage            = digitalSignature
extendedKeyUsage    =  clientAuth

Create an intermediate pkcs12 keystore so certificate can be imported

You need to import the certificate and private key into the browser’s keystore.   The only way I found of doing this was via an intermediate pkcs12 keystore (with extension .p12).   If you import the certificate and key from the web browser, it will expect a .p12 file.

openssl pkcs12 -export -inkey colinpaice.key.pem -in colinpaice.pem -out colinpaice.p12 -CAfile cacert.pem -chain -name colinpaice -passout file:password.file -passin file:password.file

  • openssl pkcs12 – request to process a pkcs12 keystore
  • -export – to create it
  • -inkey colinpaice.key.pem – this private key
  • -in colinpaice.pem – this certificate returned from the CA
  • -out colinpaice.p12 – the name of the temporary key store which is created
  • -CAfile cacert.pem – use this CA certificate
  • -chain – include any CA certificates with the certificate and key
  • -name colinpaice – create this name in the keystore
  • -passout file:password.file -passin file:password.file – use these passwords

Import the temporary keystore into the Chrome keystore

pk12util -i colinpaice.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -W password

  • pk12util   – this command
  • -i  colinpaice.p12  – from the temporary keystore you just created
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ – into this key store
  • -W password  – using this password (for the temporary .p12 file)

Remove the intermediate file

rm colinpaice.p12

You do not need to import the certificate into the mqweb trust store.

Update the mqweb configuration if required

<webAppSecurity allowFailOverToBasicAuth="false" />
<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/ssl2/mqweb.p12" 
          type="pkcs12" 
          password="password"/>

<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/ssl2/trust.jks" 
          type="JKS" 
          password="password"/>

<ssl     id="defaultSSLConfig" 
         keyStoreRef="defaultKeyStore" serverKeyAlias="mqweb" 
         trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
         clientAuthentication="true" 
         clientAuthenticationSupported="true" 
/>

Stop mqweb

It is better to stop and restart mqweb if you change the xml config files, otherwise you can get strange errors.

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

No messages are produced in  /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log if the trust store was opened successfully.

Use a command like grep ” E ” messages.log  and check for messages like

CWPKI0033E: The keystore located at /home/colinpaice/ssl/ssl2/trust.jks did not load because of the following error: Keystore was tampered with, or password was incorrect

Try using it in Chrome

You need to restart Chrome to pick up the changes to its keystore.  Use the url chrome://restart/

Use the url chrome://settings/certificates , to check your certificate is present under “Your certificates”. If not use url chrome://version to display the profile being used, and that it matches the store used in the pk12util command above.

Try connecting to mqweb using a url like https://127.0.0.1:9443/ibmmq/console/ .

You should be logged on with no password request. In the top right hand corner of the screen you should have a black circle with a white “i” in it.   This shows you are logged on with certificates.  Click on this to show how you are logged on.

Setting up a self signed certificate for the mqweb server end

When using mqweb with certificates you can use

  • a self signed certificate to identify the server
  • a CA signed certificate to identify the server

You can use certificates to authenticate…

  • a self signed certificate at the client end
  • a CA signed certificate at the client end

This post explains how I set up mqweb to use a self  signed certificate at the server, and to import the certificate into my Chrome browser.

The tasks are

  • create the self signed certificate
  • create the keystore and import the certificate
  • update the mqwebuser.xml file
  • import the certificate into the browser keystore

Create the openssl config file

You configure parameters in sections in a config file, and use a command parameter to identify which sections to use.

For the self signed certificiate I set up ss_extensions

[ ss_extensions ]

subjectKeyIdentifier = hash
#Note: there is a bug in Chrome where it does 
# not accept certificates if basicConstraints
# is specified
# basicConstraints = CA:false

subjectAltName = DNS:localhost, IP:127.0.0.1
nsComment = "OpenSSL ColinClientSS28"
keyUsage = critical, keyEncipherment
extendedKeyUsage = critical, serverAuth

Create the self signed certificate

I set up a shell script to automate the tasks

Create the self signed certificate

openssl req -x509 -config openssl-client2.cnf -newkey rsa:2048 -out ss.pem -keyout ss.key.pem -subj “/C=GB/O=aaaa/CN=colinpaice” -extensions ss_extensions -passin file:password.file -passout file:password.file

  • -config openssl-client2.cnf  – the location of the openssl configutation file ( see -extensions)
  • -newkey rsa:2048 – define a self signed certiticate
  • -out ss.pem – where the certificate is stored
  • -keyout ss.key.pem– where the private key is stored
  • -subj “/C=GB/O=aaaa/CN=colinpaice” – this is the DN of the certificate.  I Used O=aaaa so it was at the top of any list of certificates
  • -extensions ss_extensions see above
  • -passin file:password.file -passout file:password.file – openssl uses passwords. The file has two lines each with a password in it.

delete the old keystore

rm ssks.p12

  • delete the old keystore

create the keystore and import the certificate

openssl pkcs12 -export -in ss.pem -inkey ss.key.pem -out ssks.p12 -name “server” -passin file:password.file -passout file:password.file

  • pkcs12 -export – create the pkcs12 keystore
  • -in ss.pem -inkey ss.key.pem – the two files which are imported into the keystore
  • -out ssks.p12 – this is the keystore used by mqweb
  • -name “server”– this is the label given to the certificate in the keystore, used in serverKeyAlias in the mqweb xml configuration
  • -passin file:password.file -passout file:password.file – this gives the passwords to use

Other commands I used, to display information about the certificate

  • openssl x509 -purpose -in ss.pem -inform PEM -nocert
  • openssl x509 -in ss.pem -noout -ext subjectAltName
  • openssl x509 -in ss.pem -noout -ext keyUsage
  • openssl x509 -in ss.pem -noout -ext extendedKeyUsage

Update the mqweb configutation file

<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/ssks.p12"  
          type="pkcs12" 
          password="password"/> 
<!-- this is used for authentication with user certificates
<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/key.jks" 
          type="JKS" 
          password="zpassword"/>
-->
<ssl id="defaultSSLConfig" 
     keyStoreRef="defaultKeyStore" 
     serverKeyAlias="server" 
     clientAuthentication="false" 
     clientAuthenticationSupported="false" 
/>
<!--trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
-->

Stop mqweb

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

Check /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log for messages like

[14/01/20 09:12:15:730 GMT] 00000024 com.ibm.ws.ssl.config.WSKeyStore E CWPKI0033E: The keystore located at /home/colinpaice/ssl/ssks.p12 did not load because of the following error: keystore password was incorrect.

Import it into Chrome

If you do not import it into the web browser, the web browser will complain and give you the option to accept it as untrusted.  If this is just a test system this may be acceptable.  If you want to be able to trust the web server, you need to import the certificate into the browser’s keystore, as trusted.

I have several profiles for Chrome.  At one point it hickuped and created a new profile.

Find the keystore

In Chrome use the url chrome://version/ this gives the profile path, for example /home/colinpaice/snap/chromium/986/.config/chromium/Default

Remove the old certificate

certutil -D -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb -n server

  • certutil -D – delete the certificate
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb – from this keystore directory
  • -n server with this name

Add the new certificate

certutil -A -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb -t “P,,” -n server -i $name.pem

  • certutil -A – add a certificate
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb – into this keystore directory
  • -t “P,,” – give it these permissions.
    • P says Trusted Peer certificate.   The certificate appears in Chrome under “servers”
  • -n server – with this name
  • -i ss.pem – and this certificate

Tell Chrome to pickup the changes

Use the url chrome://restart to restart chrome

Try using it

use the url like https://127.0.0.1:9443/ibmmq/console/

You should get the IBM MQ Console – login

 

Certificates in Chrome

You can import certificates without using the certutil command.

You can display the certificates in Chrome using the url chrome://settings/certificates .

If you click on “Your certificates” or “Servers”, and then click on import, you can import a certificate.

I selected the directory, and then the ss.pem file.   The certificate ended up in “Others” under

org-aaaa

  • UNTRUSTED colinpaice

Use the url chrome://restart to restart chrome

Use the url like https://127.0.0.1:9443/ibmmq/console/

This gave me the “Your connection is not private”,  NET::ERR_CERT_AUTHORITY_INVALID.

The only way I found of making it trusted was to use the certutil command

certutil -L -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb

This listed the certificates in the store.   The one I had just added was colinpaice #2

certutil -M -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb -n “colinpaice #2” -t P,,

This modifies the certificate to be a Trusted Peer.  Restart Chrome, and the certificate appeared in the “Servers” section.

Using the mqweb url now worked successfully

Setting up a CA signed certificate for the mqweb server end

When using mqweb with certificates you can use

  • a self signed certificate to identify the server
  • a CA signed certificate to identify the server

You can use certificates to authenticate…

  • a self signed certificate at the client end
  • a CA signed certificate at the client end

This post explains how I set up mqweb to use a CA  signed certificate at the server, and to import the CA into my Chrome browser.

The steps are

  • Create your certificate authority certificate
  • Create the certificate request for mqweb server
  • Sign the request
  • Create the mqweb keystore and import the mqweb certificate
  • Import the CA into the web browser keystore if required

Create your certificate authority certificate

If you do not already have a certificate authority and a process for signing certificates you need to set these up.   To do it properly, you should create a certificate request and have an external CA sign it.

The following command creates a self signed certificate.   This is your CA authority certificate and private key.

openssl req -x509 -config openssl-ca.cnf -newkey rsa:4096 -subj “/C=GB/O=SSS/OU=CA/CN=SSCA” -nodes  -out cacert.pem -keyout cacert.key.pem -outform PEM

  • openssl req -x509 – create a self signed certificate request.  -x509 says self signed.
  • -config openssl-ca.cnf – use this file for the definitions
  • -newkey rsa:4096 – generate a new key
  • -nodes  – do not encrypt the private keys.  I do not know if this should be specified or not.
  • -subj “/C=GB/O=SSS/OU=CA/CN=SSCA” –  with this DN
  • -out cacert.pem – output the certificate.   This is used when signing.    This file is sent to all users.
  • -keyout cacert.key.pem – output the private key.  This is used when signing.  This files stays on the machine.
  • -outform PEM – in this format

In the config file, the important options I had were

[ req_extensions ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer
basicConstraints = critical,CA:TRUE, pathlen:0
keyUsage = keyCertSign, digitalSignature

You need to distribute the cacert.pem certificate to all your users, so they import it into their keystores.

Create the certificate request for mqweb server

The following command creates a certificate request which will be sent to the CA to sign.

openssl req -config mqwebserver.config -newkey rsa:2048 -out mqweb.csr -outform PEM -keyout mqweb.key.pem -subj “/C=GB/O=cpwebuser/CN=mqweb” -passin file:password.file -passout file:password.file

  • openssl req – as this request does not have -x509 specified it is for a certificate request
  • -config mqwebserver.config – use definitions in the specified  file
  • -newkey rsa:2048 – create a new  certificate request and private key with a 2048 bit  RSA key
  • -out mqweb.csr – use this output file for the request to be sent to the CA
  • -outform PEM – use pem format
  • -keyout mqweb.key.pem – put the private key in this file
  • -subj “/C=GB/O=cpwebuser/CN=mqweb” – with this distinguished name. It can have any values which meet your enterprise standards.
  • -passin file:password.file -passout file:password.file – use the passwords in the file(s).  The file:… says use this file. You can specify a password instead.  As the same file is used twice, two lines in the file are used.

In the config file, the important options were

[ req_extensions ]
subjectKeyIdentifier = hash
basicConstraints = CA:FALSE
subjectAltName = DNS:localhost, IP:127.0.0.1
nsComment = "OpenSSL mqweb server"
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = critical, serverAuth

Sign the request

Send the *.csr file to your CA.

Initial setup of ca signing process

If this is the first time you are using your CA you need to set up some files.  The files are referred to in the config file used by openssl ca command

touch index.txt
echo '01' > serial.txt

  • index.txt  is the certificate database index file.  This name is used in the config file option database =… .  For the format of this file see here. 
  • serial.txt contains the current serial number of the certificate. This name is used in the config file option serial =… .

Sign the certificate request

This takes the .csr file and signs it.

openssl ca -config casign.config -md sha384 -out mqweb.pem -cert cacert.pem -keyfile cacert.key.pem -infiles mqweb.csr

  • openssl ca – do the ca signing
  • -config casign.config – using the specified config file
  • -md sha384 – what message digest strength to use
  • -out mqweb.pem – put the signed certificate in this file
  • -cert cacert.pem – sign it with this ca file
  • -keyfile cacert.key.pem – sign it with this ca private  file
  • -infiles mqweb.csr – this is the input certificate request file

This displays the certificate, so check it is correct.  You get prompted

  • Sign the certificate? [y/n]:y
  • 1 out of 1 certificate requests certified, commit? [y/n]y

In the config file the important section is

[ ca_extensions ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:FALSE
keyUsage = keyEncipherment
extendedKeyUsage = serverAuth

Send the signed certificate back to the requestor.

Create the keystore to be used by mqweb and import the certificate

You can delete the certificate in the keystore using runmqckm -cert -delete -db mqweb.p12 -pw password -label mqweb .  This is not required.

openssl pkcs12 -export -inkey mqweb.key.pem -in mqweb.pem -out mqweb.p12 -CAfile cacert.pem -chain -name mqweb -passout file:password.file -passin file:password.file

This command creates the keystore (if requried) and imports the signed certificate and private key into the store with the specified name.   If a certificate exists with the same name, it is replaced.

  • openssl pkcs12 -export – this says a pkcs12 file will be created
  • -inkey mqweb.key.pem – using this private key file
  • -in mqweb.pem – and this signed certificate
  • -out mqweb.p12 – output put it to this pkcs12 keystore, used by mqweb
  • -CAfile cacert.pem – using this CA certificate
  • -chain – include all of the certificates needed when adding the certificate to the key store
  • -name mqweb – create this name in the keystore.  It is used to identify the key in the key store.
  • -passout file:password.file -passin file:password.file – use these password files

There is no config file for this operation.

Use chmod and chown to protect this keystore file from unauthorised access.

Change the mqweb configuration file.

<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/mqweb.p12"  
          type="pkcs12" 
          password="password"/> 
<!-- the trust store is used when authenticating 
<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/key.jks" 
          type="JKS" 
          password="zpassword"/>
-->
<ssl id="defaultSSLConfig" 
     keyStoreRef="defaultKeyStore" 
     serverKeyAlias="mqweb" 
     clientAuthentication="false" 
     clientAuthenticationSupported="false" 
/>
<!--trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
-->

The keystore name and server key alias which identifies which certificate to use,  are highlighted.

Stop mqweb

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

Check /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log for

Successfully loaded default keystore: /home/colinpaice/ssl/ssl2/mqweb.p12 of type: pkcs12.   This means it has successfully opened the keystore.

If you do not get this message use a command like grep ” E ” messages.log  and check for messages like

E CWPKI0033E: The keystore located at …. did not load because of the following error: keystore password was incorrect.

Import the CA certificate into Chrome

You need to do this once for every CA certificate

I have several profiles for Chrome.  At one point it hickup-ed and created a new profile, my scripts carried on updating the old profile until I realized a different profile was now being used.

Find the keystore

In Chrome use the url chrome://version/ this gives the profile path, for example /home/colinpaice/snap/chromium/986/.config/chromium/Default

You can remove the old certificate CA

certutil -D -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb -n myCACert

  • certutil -D – delete the certificate
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb – from this keystore directory
  • -n  myCACertr with this name

Add the new CA certificate

certutil -A -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb -t “C,,”  -i cacert.pem -n myCACert

  • certutil -A – add a certificate
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb – into this keystore directory
  • -t “C,,” – give it these permissions.
    • C says Trusted CA.   The certificate appears in Chrome under “certificate authorities”
  • -i cacert.pem – import this certificate
  • -n myCACert – with this name

Tell Chrome to pickup the changes

Use the url chrome://restart to restart chrome

Try using it.   Use the url like https://127.0.0.1:9443/ibmmq/console/

You should get the IBM MQ Console – login

Are my digital certificates still valid and are they slowing down my channel start?

Digital certificates are great. They allow program to program communication so each end can get information to identify the other end, and the programs can then communicate securely, with encryption, or just checking the payload has not changed.

A certificate is basically a file with two parts (or two files)  – a public certificate and a private key. You can publicize the public part to any one who wants it (which is why is is called the public part). Anyone with the private key can use it to say they are you. (If you can get access to the private key, then you can impersonate the identity)

There are times when you want to say this certificate is not longer valid. For example when I worked at IBM, I had a certificate on my laptop to access the IBM mail servers.

  • If my laptop was stolen, IBM would need to revoke the certificate.
  • When I retired from IBM, IBM revoked my certificate to prevent me from trying to access my IBM mail using my old certificate from my personal laptop.

Managing these certificate is difficult. There could be billions of certificates in use today.

Your server should validate every connection request to ensure the certificate sent from the client is still valid.

A client should validate the certificate sent by the server to ensure that it is connecting to a valid server.

In the early days of certificates, there was a big list of revoked certificates – the Certificate Revocation List(CRL). If a certificate is on the list then it has been revoked. You tend to have an LDAP server within your firewall which contains these lists of revoked certificates.

This was a step in the right direction, but it is difficult to keep these lists up to date, when you consider how many certificates are in use today, and how many organizations generate certificates. How often do I need to refresh my list? If the CRL server was to refresh it every day, it may be up to one day out of date, and report “this certificate is ok” – when it had been revoked.

These days there is a technique called Online Certificate Status Protocol (OCSP). Basically this says go and ask the site which issued this certificate if it is still valid. This is a good idea – and they say the simple ideas are the best.
How do you know who to ask? A certificate can have url information within in it Authority Information Access: OCSP – URI:http://ColinsCert.Checker.com/, or you can specify URL information in the queue manager configuration for those certificates without the Authority Information Access(AIA) information.

Often the URL in the certificate is outside of your organization, and outside of your firewall. To access the OCSP site you may need to have an SSL Proxy server which has access through the firewall.

You can configure MQ to use a (one) OCSP server for those certificates not using AIA information.  If your organization is a multinational company, you may be working with other companies who use different Certificate Authorities.   If you have certificates from more than one CA, you will not be able use MQ to check all of them to see if they are still valid.  You may want to set up an offline job which runs periodically and checks the validity of the certificates.

Starting the MQ channel can be slow

When an TLS/SSL MQ channel is started, you can use OCSP or CRL to check that a certificate is valid. This means sending a request to a remote server and waiting for a response.   The LDAP server for CRL requests is likely to be within your domain,  as your organization manages it. The OCSP  server could be outside of your control, and in the external network.  If this server is slow, or the access to the server is slow, the channel will be slow to start.  For many customers the network time within a site is under 1 millisecond, between two sites could be 50 ms. Going to an external site the time could be seconds – and dependent on the traffic to the external site.
This time may be acceptable when starting the channel, first thing in the morning, but restarting a channel during a busy period can cause a spike in traffic because no messages flow while the channel is starting. For example

ChannelwithRestart

No messages will flow while the channel is starting, and this delay will add to the round trip time of the messages.

How do I check to see if I have a problem

This is tough. MQ does not provide any information. I used MQ internal trace when debugging problems, but you cannot run with trace enabled during normal running.

There are two parts to the validation request. The time to get to and from the server, and the time to process your request once it has got to the server.

You can use TCP Ping to get to the server (or to the proxy server).  If you are using a proxy server you cannot “ping” through the proxy server.

Openssl provides a many functions for creating and managing certificates.

You can use the command

time openssl s_client -connect server:port
or
time openssl s_client -connect -proxy host:port server:port

The “time” is a linux command which tells your the duration in milliseconds of the command following it.

The openssl s_client is a powerful ssl client program. The -connect… tries connecting to server:port. You can specify -proxy host:port to use the proxy.

The server at the remote end may not recognize the request – but you will get the response time of the request.

Running this on my laptop I got

time openssl s_client -connect 127.0.0.1:8888
CONNECTED(00000005)
140566846001600:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:332:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 5 bytes and written 233 bytes

Verification: OK

real 0m0.012s

The request took 0.012 seconds.

My openssl OCSP server reported

OCSP Response Data:
OCSP Response Status: malformedrequest (0x1)

You can use openssl ocsp …. to send a certificate validation request to the OCSP server and check the validity of the system – but you would have to first extract the certificates from the iKeyman keystore.

How do I test this?

I’ve created the instructions I used:

More information about OCSP.

There is a good article in the IBM Knowledge Centre here .

The article says
To check the revocation status of a digital certificate using OCSP, IBM MQ determines which OCSP responder to contact in one of two ways:

  1. Using a URL specified in an authentication information object or specified by a client application.
  2. Using the AuthorityInfoAccess (AIA) certificate extension in the certificate to be checked.

Configure a QM for OSCP checking for certificates without AIA information.

You can configure a queue manager to do OCSP checking, for those certificates without AIA information within the certificate.

There is a queue manager attribute SSLCRLNL ( SSL Certificate RevocationList Name List) which points to a name list.   This name list has a list of AUTHINFO object names.
The name list can have up to one AUTHINFO object for OCSP checking and up to 10 AUTHINFO objects for CRL checking.

You define an AUTHINFO object to define the URL of an OCSP server.

Define AUTHINFO(MYOSCP) AUTHTYPE(OCSP) OCSPURL(‘HTTP://MyOSCP.server’)

Create a name list, and add the AUTHINFO to it.

Use alter qmgr  SSLCRLNL(name) and refresh security type(SSL)
You need to change the qm.ini file SSL stanza of the queue manager configuration file see here.

OCSPAuthentication=REQUIRED|WARN|OPTIONAL
OCSPCheckExtensions= YES|NO
SSLHTTPProxyName= string

If you have a fire wall around your network, you can use SSLHTTPProxyName to get through your fire wall.

There is some good information here.

Configure client OSCP checking for certificates without AIA information.

You need the CCDT created by a queue manager rather than a JSON CCDT.
When you configure a queue manager with the AUTHINFO objects and the queue manager SSLCRLNL attributes, the information is copied to the CCDT.

This CCDT is in the usual location, for example the /prefix/qmgrs/QUEUEMANAGERNAME/@ipcc directory.

You can use a CCDT from one queue manager, when accessing other queue managers.

You need to make the CCDT file available to the client machines, for example email or FTP,  or use URL access to the CCDT.

You also should configure the mqclient.ini file see here.

How do I check to see if my certificates have AIA information.

You can use the iKeyman GUI to display details about the certificate, or a command line like

/opt/mqm/bin/runmqckm -cert -details -db key.kdb -pw password -label CLIENT

This gives output like

Label: CLIENT
Key Size: 2048
Version: X509 V3
Serial Number: 01
Issued by: ….

Extensions:
– AuthorityInfoAccess: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
[accessMethod: ocsp
accessLocation: URIName: http://ocsp
server.my.host/
]
]

Can I turn this OCSP checking off ?

For example if you think you have a problem with OCSP server response time.

In the mqclient.ini you can set

  • ClientRevocationChecks = DISABLED No attempt is made to load certificate revocation configuration from the CCDT and no certificate revocation checking is done
  • OCSPCheckExtensions = NO   This says ignore the URL in the AIA information within a certificate.

See SSL stanza of the client configuration file.

In the qm.ini you can set

  • OCSPCheckExtensions=NO  This says ignore the URL in the AIA information within a certificate.
  • alter qmgr SSLCRLNL(‘ ‘) and refresh security type(SSL)

See SSL stanza of the queue manager configuration file and Revoked certificates and OCSP.

How do I tell if I have a problem with OCSP?

There are no events or messages which tell you the response time of requests.

You may get message AMQ9716

AMQ9716
Remote SSL certificate revocation status check failed for channel …
Explanation
IBM MQ failed to determine the revocation status of the remote SSL certificate for one of the following reasons:
(a) The channel was unable to contact any of the CRL servers or OCSP responders for the certificate.
(b) None of the OCSP responders contacted knows the revocation status of the certificate.
(c) An OCSP response was received, but the digital signature of the response could not be verified.
You can change the queue manager configuration to not produce these messages, by setting ClientRevocationChecks = OPTIONAL
From this message you cannot tell if the request got to the server.
The easiest way may be to ask the network people to take a packet trace to the URL(s) and review the time of the requests and the responses.

Using the AuthorityInfoAccess (AIA) certificate extension in the certificate.

You can create certificates containing the URL needed to validate the certificate. Most of the IBM MQ documentation assumes you have already have a certificate with this information in it.
You can use openssl to create a certificate with AIA information, and import it into the iKeyman keystore.   See here.

You cannot use IBM GSKIT program iKeyman to generate this data because it does not support it. You can use iKeyman to display the information once it is inside the keystore.

Timing the validation request

Openssl has a command to validate a certificate for example

openssl ocsp -CAfile cacert.pem -issuer cacert.pem -cert servercert.pem -url http://OCSP.server.com:port -resp_text

You can use the linux time command, for example

time openssl ocsp -CAfile cacert.pem -issuer cacert.pem -cert servercert.pem -url http://OCSP.server.com:port -resp_text

you get

Response verify OK
servercert.pem: good
real 0m0.040s

The time taken to go to a OCSP server on the same machine is 40 milliseconds. The time for a ping to 127.0.0.1 was also 40 ms.

Thanks…

Thanks to Morag of MQGEM, and Gwydion at IBM for helping me get my head round this topic.