Category: TLS

mqweb- is it better to use certificate authentication or userid and password?

Like may questions the answer is it depends.

You can set up your mqweb environment

  1. Certificates are required – you cannot use userid and passwords
  2. If a valid certificate is found use that, else prompt for userid and password
  3. Do not use certificates – just use userid and passwords.

This is described in the Liberty documentation.

It says for the <ssl…> tag

  1. If you specify clientAuthentication="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake does not succeed.
  2. If you specify clientAuthenticationSupported="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.
  3. If you do not specify either clientAuthentication or clientAuthenticationSupported, or you specify clientAuthentication="false" or clientAuthenticationSupported="false", the server does not request that a client send a certificate during the handshake.

No matter how you logon, the web server creates a LTPA* cookie with encrypted information about the logon.   The mqwebuser.xml attribute ltpaExpiration says how long this cookie is valid for.  If the time is exceeded you have to logon again, and it generates a new cookie.

Logging on

Certificate authorisation

If you are using certificates, the server sends down a list of valid self signed certificates or CA certificates, and the browser compares this list with the certificates in its store.   If the certificate in the browser’s store is acceptable, it is added to a list

The list is displayed, and the end user selects a certificate.  For subsequent requests with the same  http://url:port combination, in the same or a different tab within the browser window, the same certificate is used, and the browser does not prompt.

If the ltpa cookie expires then the same certificate is used, invisibly to the end user.  A new cookie is generate and sent down.

If there are no valid certificates in the list, then depending clientAuthentication, it may prompt for a userid and password.

You cannot logoff from the page as there is no option to do so.

Userid and password.

When userid and password are passed up in a header (not in the URL as some web servers accept) the ltpa token is returned.  When the ltpa token expires, the user is prompted to re-logon.   They need to enter the userid and password again.  You can also logoff from the web page.

What are the implications of certificates and userid with password

Certificate logon

  • The web browser can logon, and although the ltpa token expires, it is automatically logged on again.   This is useful for a “big screen monitor” in the operations room, where you want the display active all day.
  • Someone can create a file with some HTML in it to browse a queue  (if authorised).  So if they are authorised to the queue for their normal job, they could easily create a web page to display the queue contents.  This could be the end user, or an evil person saying “click this link”.

userid logon

  • If the user is using the mqconsole web pages, then periodically they will get logged off – and they will have to logon again.
  • If they try to access a queue from an HTML page, they will be asked for userid and password.  Depending on what the end user is doing, this may give them notification than there is some unusual activity.  If they are hacking they will enter the userid and password anyway.
  • The person may have access to the same information without using the mqweb console interface, so there is no change to the threat level.

What do I recommend?

Tricky.

  • I think long running monitors need to use certificates
  • End users should logon with userid and password, with an expiry period of about an hour or less.
  • You may want to have a Certificate  Authority just for the mqweb browsers, to stop your corporate certificates from being used to access the mqweb server.

Because it is easy to create an html page and browse a queue, I would allow administration over mqrest, and not allow processing messages over REST.

As I said in another posting

  • The admin API authorisation is managed through <security-role name=”MQWebAdmin”> and <security-role name=”MQWebAdminRO”> sections in the mqwebuser.xml file.
  • The messaging API authorisation is managed through <security-role name=”MQWebUser”> sections.

To stop messaging over the rest interface, have no entries in the <security-role name=”MQWebUser”> section.  This applies to both certificates, and userid and password.

(Personally I would have called<security-role name=”MQWebUser”> as <security-role name=”MQWebMessaging”>  to make the roles clearer. 

Update from Gwydion…

The MQWebAdmin, MQWebAdminRO and MQWebUser roles can all be used for the admin REST API. That’s why MQWebUser is not called MQWebMessaging – it’s not just for messaging. The difference between them is the user ID that’s checked by the qmgr for operations performed via the REST API.
  • Operations performed by users in the MQWebAdmin* roles take place under the context of the mqweb server user ID.
  • Operation performed by users in the MQWebUser role take place under the context of the user logged into the REST API.
To stop messaging over the REST API completely, the mqRestMessagingEnabled should be set to false in the mqweb server configuration.

Setting up Liberty(as used in mqweb) to use native JMX

Setting up the server side is well documented in the Oracle Monitoring and Management Using JMX Technology documentation.  Using it from a client is not so well documented.

Server set up

The  Liberty jvm.options file needs parameters.  Note the port=9010 is used  by clients accessing the data.

To provide insecure access from only the local machine

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9010 
-Dcom.sun.management.jmxremote.local.only=true 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false

To provide securer access using TLS

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true

# the following statements point to the same key store as
# used by mqweb server.   This could be different.
-Djavax.net.ssl.keyStoreType=PKCS12
-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/mqweb.p12
-Djavax.net.ssl.keyStorePassword=password
# the following statements point to the same trust store as
# used by mqweb server.   This could be different.
# if you used self signed certificates you could have a keystore
# just for the JMX users
-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks
-Djavax.net.ssl.trustStorePassword=zpassword
-Djavax.net.ssl.trustStoreType=JKS

# The following defines the userid and password file
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.password.file=/home/colinpaice/ssl/ssl2/jmxremote.password

# The following defines the access a userid can have
# Only the owner can have access to it
-Dcom.sun.management.jmxremote.access.file=/home/colinpaice/ssl/ssl2/jmxremote.access

jmxremote.password has

# specify actual password instead of the text password
monitorRole password
controlRole password

jmxremote.access has

# The "monitorRole" role has readonly access.
# The "controlRole" role has readwrite access.
monitorRole readonly
controlRole readwrite

Client set up

jconsole

You cannot pass a  userid and password when the jconsole command, so you have to disable authentication in the jvm.options file

-Dcom.sun.management.jmxremote.authenticate=false

The parameters for jconsole have  -J on them, as in -J-D…. .  jconsole removes the -J and uses the rest of the parameters when invoking the JVM.

I could not get jconsole to recognize a config file using the -J-Dcom.sun.management.config.file = /path/to/jmxremote.properties , so I wrote a bash script to make it easier to change parameters.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file"
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-J-Djavax.net.debug=ssl:handshake"
jconsole -J$ssl1 -J$ssl2 -J$ssl3 -J$ssl4 -J$ssl5 -J$ssl6 -J$ssl7 $ssl8 127.0.0.1:9010

The option “-J-Djavax.net.debug=ssl:handshake” gives a verb verbose trace of the ssl flows for the handshake.

The option -J-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file enables the jconsole logging.  I did not find the output very useful.

There is information the logger in general here,  and on the file logger, here.

The logging.file had

Logging.properties

handlers= java.util.logging.FileHandler
// , java.util.logging.ConsoleHandler2

java.util.logging.FileHandler.pattern=/home/colinpaice/JMXQuery/java/log.%g.file
java.util.logging.FileHandler.limit=50000
java.util.logging.FileHandler.count=2
java.util.logging.FileHandler.level=ALL
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

// .level = INFO
// logger.level = FINEST
.level = FINEST
// Use FINER or FINEST for javax.management.remote.level - FINEST is
// very verbose...
javax.level= FINER
javax.management.level = FINER
javax.management.remote.*     = FINER 
javax.management.remote.level = FINER
javax.management.remote.misc.level  = FINER
javax.management.remote.rmi.level= FINER

Using jmxquery

I used a bash shell script to run the command, as it was easier to manage, and I could not find a way of having the java system properties in a file.

ssl1="-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ibmsys1.p12"
ssl2="-Djavax.net.ssl.keyStorePassword=password"
ssl3="-Djavax.net.ssl.keyStoreType=pkcs12"les
ssl4="-Djava.util.logging.config.file=/home/colinpaice/JMXQuery/java/logging.file" 
ssl5="-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/trust.jks"
ssl6="-Djavax.net.ssl.trustStorePassword=zpassword"
ssl7="-Djavax.net.ssl.trustStoreType=jks"
ssl8="-Djavax.net.debug=ssl:handshake"
jar="-jar JMXQuery.jar"
user="-username monitorRole -password password"
url="-url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi"
parms=" -q   WebSphere:*  -count 2 -every 2"
java $ssl1 $ssl2 $ssl3 $ssl4 $ssl5 $ssl6 $ssl7 $ssl8 $jar $url $user $parms

 

Accessing JMX data in Liberty server, securely.

I thought  I would complete the work I did with using JMX in the mqweb server.   It was another example of Hofstadter’s Law:

It always takes longer than you expect, even when you take into account Hofstadter’s Law.

I spent a lot of time looking for things on the web, expecting them to be obvious, only to find that the things do not behave as expected.  I could not find them, because they were not there.  For example I expected to be able to configure the JMX server to use my OS userid and password.  I could have a file with userids and passwords, or lookup in LDAP, but not my normal userid.

Getting started

I found there are two ways of getting the JMX data from the mqweb server.

  1. Use of the native JMX support
  2. Using the Liberty REST API

I think the REST API is easier to set up and is more secure.

I’ll document a high level overview of the two approaches, and how to configure them

Overview of using the native JMX support.

To use this, you configure parameters in the jvm.options file, including a port solely for JMX.

You can use TLS certificates to set up a secure link between the client and the server.

You can decide if you want to logon with userid and password.  If you do you can set up

  1. A file with userids and passwords; and a file with userids and permitted access.   The documentation talks about userids like monitorRole and controlRole.   You have to put a process in place to periodically change these passwords.
  2. Use and LDAP server to do userid validation and to get the access.
  3. I could not find how to use your operating system userid and password for authentication.
  4. I could not find how to use the DN as authorization.

If your certificate is valid (either because it is signed by a CA, or there is a copy of a self signed certificate in the trust store), this is good enough for the checking.   You can enable userid and password checking, but this solution feels weak, as you have to do extra work to manage it properly;  you do not have a single signon.  Not all tools support using userid and password, for example I could not pass userid and password on the jconsole command.

Overview of Using the Liberty REST API

As with the MQ REST API you can issue an HTTP request and get data back.  See here.  For example

curl –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password https://localhost:9443/IBMJMXConnectorREST/mbeans/WebSphere:name=com.ibm.mq.console.javax.ws.rs.core.Application,type=ServletStats/attributes=*

There is a small amount of configuration you need to do – less than with the native JMX support.  The data comes back as JSON (as you might expect) and also includes a time stamp, which is very useful when post processing.

You define <administrator-role><user>..</user></administrator-role>  in a similar way to setting up authorisation for mqconsole and mqrest.  It takes the cn= value from your certificate as the userid, so you can give individual access.

“Securely” is a good laugh.

There are different levels of (in)security.

If you are using the native JMX support

  • You can have no passwords or access checks needed.  The data is read only, and is not sensitive.
  • You can set up userid(s) and passwords in a file
  • You cannot use the operating system userid and password
  • You can use LDAP to check the userid and password, and get the role for that userid
  • You can use TLS, so anyone with a valid certificate can access the data
  • You can use TLS and use the userid and password in a file to determine access
  • You can use TLS and LDAP to get the role for that certificate

If you are using the WLP REST support

  • You can specify a userid and password
  • You can use a certificate, and the Common Name is used as the userid
  • You can specify in the configuration file, what access userids, or groups have

You can use TLS to protect your communications to and from the server.

Java leaks passwords

You need to be aware that your client machine may leak information.  For example I ran a  Java program to issue JMX requests from a script.

I could use the linux command ps -ef to display information about my request

ps -ef |grep JMX

gave me

colinpa+ 1871 1870 79 10:27 pts/2 00:00:01 java …  -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/colinpaice.p12 -Djavax.net.ssl.keyStorePassword=password …  -username monitorRole -password password …

This exposed the password to my keystore and password to my userid!  I could not find a way of having all these java system parameters in a file.

I found export JAVA_TOOL_OPTIONS=”-D…”  and this get picked up, but then java displays the variables as in Picked up JAVA_TOOL_OPTIONS: …

jconsole

Some programs have been designed to protect information for example jconsole you can put your system properties in a file

-J-Dcom.sun.management.config.file=ConfigFilePath

and so keep your parameters secure, but I could not get this to work.

Curl can be configured not to display parameters

With curl you have a command like

curl -n –cacert ./cacert.pem –cert-type P12 –cert colinpaice.p12:password

which gives away your password.  If you do not specify it inline, you get prompted for it.

You can put your parameters in a config file, for example curl.config,

cacert ./cacert.pem 
cert ./colinpaice.pem:password 
key colinpaice.key.pem 
cookie cookie.jar.txt 
cookie-jar cookie.jar.txt 
url https://127.0.0.1:9443/ibmmq/rest/v1/login

and use

curl –config curl.config

Easy!

Protecting key files

It is important to protect the certificate file (with the important private key) so it is accessible by just the owner.  The linux command  ls -ltr colinpaice.p12 gives

-rw------- 1 colinpaice colinpaice 4146 Jan 31 17:56 colinpaice.p12

Of course anyone with super user authority has access to this file!

mqweb error messages and symptoms of TLS setup problems

I deliberately caused TLS set up errors, and noted the symptoms.  Ive recorded them below; the article is not meant to be read, but indexed by search engines.

There are three sections

  1. Problems with server certificates
  2. Problems with the client certificate
  3. Chrome messages, and possible causes of the problems.

The mqweb messages.log reported problems that the mqweb server saw.   For me this was in file /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log

Problems with the server certificate

Problem: mqwebuser.xml serverKeyAlias name not in the keystore

This can be caused by the certificate being in the keyring but not visible, or cannot be validated.

The RACF command RACDCERT LISTRING(KEYRING) ID(IZUSVR) will list the contents of the keyring. For example it gives ZZZZ ID(START1).  You can then use

RACDCERT LIST(LABEL(‘ZZZZ’ )) ID(START1).   This gives output including

Status: TRUST
Start Date: 2020/12/17 00:00:00
End Date: 2021/12/17 23:59:59

Check it has STATUS:TRUST and the dates are valid.  If you make a change, check it afterwards.  Several times I got the change wrong!

Check the CA for the certificate is in the keystore; you need the key, and the CA in the keystore.

Message log:

  • Failed validating certificate paths
  • E CWPKI0024E: The certificate alias mqweb specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore /home/colinpaice/ssl/ssl2/mqweb.p12.
  • I FFDC1015I: An FFDC Incident has been created: “com.ibm.wsspi.channelfw.exception.ChannelException: java.lang.IllegalArgumentException: CWPKI0024E: The certificate alias mqweb specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore /home/colinpaice/ssl/ssl2/mqweb.p12. com.ibm.ws.channel.ssl.internal.SSLConnectionLink 238″ at ffdc_….

curl:

* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* curl (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443
* stopped the pause stream!
* Closing connection 0

chrome:

This site can’t be reached.  ERR_CONNECTION_CLOSED

Problem:  The host certificate is self signed and not in the client keystore

Problem:  The host certificate is signed but the signer certificate is not in  the client keystore

Message log:

Nothing.

curl:

* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate

Chrome: in browser

NET::ERR_CERT_AUTHORITY_INVALID

Click on the Not Secure in the url, to display the certificate which was sent down.

If it is signed, make a note of the “issued by” Common Name(CN), and the  Organisation(0) and look up the value of Organisation in the “Authorities” section of “Manage Certificates”.

Use the chrome url chrome://settings/certificates .  Authorities tab

  1. if it is not present, import it
  2. it it is present and UNTRUSTED, edit it, and tick the “Trust this certificate for identifying web sites”

Chrome log:

ERROR:cert_verify_proc_nss.cc(1011)] CERT_PKIXVerifyCert for localhost failed err=-8179

From here  -8179 is Peer’s certificate issuer is not recognized.

Firefox:  browser

SEC_ERROR_UNKNOWN_ISSUER

Action import the CA signing certificate into the keystore and make it trusted.

Problem: curl: The host certificate is self signed and you use the –insecure option

curl

* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=GB; O=aaaa; CN=testuser
* start date: Jan 20 17:39:37 2020 GMT
* expire date: Feb 19 17:39:37 2020 GMT
* issuer: C=GB; O=aaaa; CN=testuser
* SSL certificate verify result: self signed certificate (18), continuing anyway.

Problem: Chrome:  The host certificate is self signed and is not trusted

Chrome browser

This site can’t be reached
localhost unexpectedly closed the connection.
ERR_CONNECTION_CLOSED

Debugging

  • I could find nothing that told me what certificate was being used.  The Chrome network trace just gave “net_error = -100 (ERR_CONNECTION_CLOSED)“.
  • Use certutil -L $sql  to list the contents of your browsers keystore.   The certificate needs “P,…” permissions.
  •  Or use the chrome url chrome://settings/certificates  and display “your certificates”. Pick the likely one, if it says “UNTRUSTED” then this may be the problem.   View the certificate, and check it, for example under details, there may be a comment describing its use.
  •  Defined the server certificate as trusted using certutil -M $sql -n name -t “P,,” 
  • Restart the web browser.

Problem: The  CA signer server certificate had the wrong subjectAltName

curl:

* subjectAltName does not match 127.0.0.1
* SSL: no alternative certificate subject name matches target host name ‘127.0.0.1’

Chrome:

NET::ERR_CERT_COMMON_NAME_INVALID
From the “Not Secure” in front of the URL, display the certificate, and check the extenstions, especially Certificate Subject Alternative Names.

Chrome log:

ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -200
From here -200 is  CERT_INVALID

Problem: The mqweb server certifcate has expired

curl:

* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
curl: (60) SSL certificate problem: certificate has expired

chrome:

while Chrome running:   web page reports Lost communication with the server.  Could not establish communication with the server. Check your network connections and refresh your browser

restart browser, get “Your connection is not private NET::ERR_CERT_DATE_INVALID”

message.log.  Chrome session was working, then server certificate expired

  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

Problem: The mqweb server certificate is missing extendedKeyUsage = serverAuth

curl:

* SSL certificate problem: unsupported certificate purpose
curl: (60) SSL certificate problem: unsupported certificate purpose

Chrome:

Your connection is not private
Attackers might be trying to steal your information from localhost (for example, passwords, messages or credit cards).
NET::ERR_CERT_INVALID

Chrome log:

CERT_PKIXVerifyCert for localhost failed err=-8101
From here  -8101 is Certificate type not approved for application.

ERROR:ssl_client_socket_impl.cc(935)] handshake failed; returned -1, SSL error code 1, net_error -207
From here -207 is CERT_INVALID

Problems with the server ca certificate

Problem: The trust store has an expired CA.

curl:

* gnutls_handshake() failed: The TLS connection was non-properly terminated.
…
pycurl.error: (35, ‘gnutls_handshake() failed: The TLS connection was non-properly terminated.’)

Problems with the client certificate

Problem: There is no suitable certificate in the client keystore.

For example

  1. There are no “Your certificates” in the browsers keystore
  2. There is a certificate, but has a CA which was not passed down from the server trust keystore
  3. As part of the TLS handshake any self signed certificates are read from the server trust keystore and sent down.  None were found in the “Your certificates”

Curl:

  • * gnutls_handshake() failed: The TLS connection was non-properly terminated.
  • pycurl.error: (35, ‘gnutls_handshake() failed: The TLS connection was non-properly terminated.’)

These messages basically mean the server just ended the connection

Chrome:

ERR_CONNECTION_CLOSED

For a test site, change <ssl clientAuthentication=”true” to false.  Restart mqweb, restart the web browser.  If it prompts for userid and password, the certificate sent from the server was OK.  It is the certificate sent up to the server that has a problem.

Reset false back to true.

Messages in messages.log:

None.

How to debug it.

Check the logs/ffdc directory.  I found I had an ffdc with Stack Dump = java.security.cert.CertPathValidatorException: The certificate issued by CN=SSCA8, OU=CA, O=SSS, C=GB is not trusted; internal cause is:   java.security.cert.CertPathValidatorException: Signature does not match.

Using Chrome trace

When I repeated the investigations, I got different records in the Chromium trace.  One included

--> net_error = -110 (ERR_SSL_CLIENT_AUTH_CERT_NEEDED)

Using the mqweb server java trace – which traces the whole server

See the Oracle Debugging SSL/TLS Connections page and an IBM page.  I could not see how to trace just “the problem”.

With -Djavax.net.debug=ssl:handshake in the jvm.options file, and restarting the mweb server I got

 *** ServerHelloDone
Default Executor-thread-8, WRITE: TLSv1.2 Handshake, length = 3054
Default Executor-thread-2, READ: TLSv1.2 Handshake, length = 7
*** Certificate chain
***
Default Executor-thread-2, fatal error: 40: null cert chain

When it worked I had

*** ServerHelloDone
Default Executor-thread-7, WRITE: TLSv1.2 Handshake, length = 3054
Default Executor-thread-15, READ: TLSv1.2 Handshake, length = 2433
*** Certificate chain
chain [0] = […. the  certificates

…

Found trusted certificate:

When there was no certificate sent up,  it reported null cert chain.

Problem: The client certificate is self signed and not in the server’s trust store

curl:

* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443

Chrome:

ERR_CONNECTION_CLOSED

Messages in messages.log:

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_08.29.27.0.log
  •  E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=testuser, O=aaaa, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target
  •  I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: unable to find valid certification path to requested target com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_08.29.27.1.log
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

 

Problem: Invalid cn=, the cn value is not a valid userid.

curl message

{“error”: [{

  • “action”: “Provide credentials using a client certificate, LTPA security token, or username and password via HTTP basic authentication header. On z/OS, if the mqweb server has been configured for SAF authentication, check the messages.log file for messages indicating that SAF authentication is not available. Start the Liberty angel process if it is not already running. You might need to restart the mqweb server for any changes to take effect.”,
  • “completionCode”: 0,
  •  “explanation”: “The REST API request cannot be completed because credentials were omitted from the request. On z/OS, if the mqweb server has been configured for SAF authentication, this can be caused by the Liberty angel process not being active.”,
  • “message”: “MQWB0104E: The REST API request to ‘https://127.0.0.1:9443/ibmmq/rest/v1/login ‘ is not authenticated.”,
  • “msgId”: “MQWB0104E”,
  • “reasonCode”: 0,
  • “type”: “rest”

chrome:

It gives you a window to enter userid and password.   This looks like a bug as I have <webAppSecurity allowFailOverToBasicAuth=”false”/>.  It takes the userid and password.

Messages in  messages.log:

R com.ibm.websphere.security.CertificateMapFailedException
and 100 lines of stack trace

The certificate causing the problems, nor the userid is listed – so pretty useless.

Problem: Client certificate missing “extendedKeyUsage = clientAuth”  during signing.

curl message

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
curl session hangs…
* Operation timed out after 300506 milliseconds with 0 out of 0 bytes received

Chrome

ERR_CONNECTION_CLOSED

message in messages.log:

  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: Extended key usage does not permit use for TLS client authentication
  •  I FFDC1015I: An FFDC Incident has been created: “java.lang.NullPointerException com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.28_17.11.10.1.log

ffdc in /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log/ffdc

Exception = java.lang.NullPointerException
Source = com.ibm.ws.ssl.core.WSX509TrustManager
probeid = checkClientTrusted
Stack Dump = java.lang.NullPointerException
at com.ibm.ws.ssl.core.WSX509TrustManager.checkClientTrusted(WSX509TrustManager.java:202)

Problem: Client certificate missing “keyUsage = digitalSignature”  during signing.

curl message

* TLSv1.2 (OUT), TLS handshake, Finished (20):
* Operation timed out after 300509 milliseconds with 0 out of 0 bytes received

message in messages.log

  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: KeyUsage does not allow digital signatures
  • FFDC1015I: An FFDC Incident has been created: “java.lang.NullPointerException com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted”
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

ffdc in /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log/ffdc

Exception = java.lang.NullPointerException
Source = com.ibm.ws.ssl.core.WSX509TrustManager
probeid = checkClientTrusted
Stack Dump = java.lang.NullPointerException
at com.ibm.ws.ssl.core.WSX509TrustManager.checkClientTrusted(WSX509TrustManager.java:202)

Chrome:

  • If there is one or more certificates in the keystore, the list of valid certificates does not include the problem one.
  • If there is only the problem certificate in the keystore, you get
    This site can’t be reached.
    localhost unexpectedly closed the connection.
    ERR_CONNECTION_CLOSED

CA Signed client certificate has expired

curl:

* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9443
* stopped the pause stream!
* Closing connection 0

Chrome:

This site can’t be reached
localhost unexpectedly closed the connection.
ERR_CONNECTION_CLOSED

message in messages.log:

for curl.

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathValidatorException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020; internal cause is:
    java.security.cert.CertificateExpiredException: NotAfter: Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.0.log
  • E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=colinpaice, O=cpwebuser, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path validation failed: java.security.cert.CertPathValidatorException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020; internal cause is:
    java.security.cert.CertificateExpiredException: NotAfter: Thu Jan 30 16:46:00 GMT 2020
  •  I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: The certificate expired at Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.1.log

for chrome:

  • I FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertificateException: The cer
    tificate expired at Thu Jan 30 16:46:00 GMT 2020 com.ibm.ws.ssl.core.WSX509TrustManager checkClientTrusted” at ffdc_20.01.30_17.16.11.1.log
  • E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: null cert chain

Bad requests

HTTP request was issued – it should have been HTTPS

curl:

curl:(52) Empty reply from server

messages.log:

E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

The client certificate cannot be verified because it is too weak.

Chrome:  ERR_BAD_SSL_CLIENT_AUTH_CERT

Firefox:  An error occurred during a connection to …  security library: memory allocation failure.  Error code: SEC_ERROR_NO_MEMORY

Reason:

The selected client certificate cannot be validated.  For example it has been created with Elliptic Curve sect409k1.   This is considered weak see here.  The signature is not in the list of acceptable signatures.

Display the certificate and compare it with the list of weak signatures.  A TLS handshake trace may help identify this.  Create a new certificate with a supported signature, and import it.

Problem the CA signing is too weak.

For example signing with sha1RSA, when Chrome expects SHA256RSA or stronger.

Chrome:  NET::ERR_CERT_WEAK_SIGNATURE_ALGORITHM

Firefox: I didnt get the error

Action: Use stronger signing.  For example on z/OS use RSA SIZE(2048)

Firefox errors

Your computer clock is set to … . Make sure your computer is set to the correct date, time, and time zone in your system settings, and then refresh …

If your clock is already set to the right time, the web site is likely misconfigured, and there is nothing you can do to resolve the issue. You can notify the web site’s administrator about the problem.

… uses an invalid security certificate.

The certificate is not trusted because the issuer certificate has expired.

Error code: SEC_ERROR_EXPIRED_ISSUER_CERTIFICATE

Reason:

The CA certificate in the trust store has expired.  The a valid CA certificate may have been sent down with the server’s certificate, but the validation failed.

Action:

  1. From Warning: Potential Security Risk Ahead -> Advanced -> View certificate. It will have the certificate.  Note Issuer -> Organisation and common name
  2. Use Firefox preferences-> view certificates.   Select authorities.  Search for the Organisation from the previous line.  Display the certificate with the matching common name.  Replace it and restart the browser.   Replace the certificate through firefox or use this to locate the directory containing the cert9.db.

Error code: SSL_ERROR_CERTIFICATE_UNKNOWN_ALERT

The backend may get java.security.cert.CertPathValidatorException: signature check failed.

One reason, the certificate being used by firefox was signed by an invalid CA, for example the CA had expired.

Action:

  1. Check Firefox preferences-> certificates, and check “Ask you every time” is selected, repeat the connection and display information about the certificate.  It will give you the issuer, but no more information than that.
  2. Regenerate the certificate, import into Firefox, restart Firefox.

Chrome errors

Chrome has more stricter checks than curl.  These are from Chrome browser.

NET::ERR_CONNECTION_CLOSED

  • mqwebuser.xml serverKeyAlias name not in the keystore
  • The host certificate is self signed and is not trusted
  • The client certificate is self signed and not in the server’s trust store
  • Client certificate missing “extendedKeyUsage = clientAuth”  during signing.
  • CA Signed client certificate has expired
  • Client certificate missing “keyUsage = digitalSignature”  during signing.

NET::ERR_CERT_COMMON_NAME_INVALID

  • missing x509 extensions in the server certificate
  • invalid subjectAltName in x509 extensions, for example IP:127.0.0.11  instead of IP:127.0.0.1

NET::ERR_CERT_INVALID

  • missing extendedKeyUsage = serverAuth in x509 extensions

NET::ERR_CERT_AUTHORITY_INVALID

  • Certificate is not peer.  Need certutil -M $sql -n $name -t “P,,” to change the certificate to be a trusted peer
  • Server’s self signed not found in the browser keystore.
  • The CA from the server does not match the certificate in the browsers’ keystore.  It may have the same name,  but check validity dates, finger prints etc.  Check very carefully.

NET::ERR_CERT_DATE_INVALID

  • The mqweb server certificate has expired.

CWPKI0024E: The certificate alias …  specified by the
property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore safkeyring://…/….

The z/OS certificate is not in the keyring, or it is in the keyring and needs to have TRUST

Make the change, stop and restart the web browser

Firefox:  PR_END_OF_FILE_ERROR

Slow backend server.

MQWEB on z/OS

 CWWKS2932I: The unauthorized version of the SAF user registry is activated.
Authentication will proceed using unauthorized native services.

Check at the top of the message log for.  CWWKB0104I: Authorized service group SAFCRED is not available.

Reason: When the web server was started the SAFCRED service was not available.   This could be caused by security not set up properly.

Fix the security.  For example here

CWWKS2930W: A SAF authentication attempt using authorized SAF services was rejected because the server is not authorized to authorized to access the APPL-ID MQWEB. Authentication will proceed using unauthorized SAF services.

Problem:  the profile with class(SERVER) and profile(BBG.SECPFX.MQWEB) is missing
Action:  the define profile matching the APPL-ID.

RDEFINE SERVER BBG.SECPFX.MQWEB
PERMIT BBG.SECPFX.MQWEB  CLASS(SERVER) ID(START1) ACC(READ)
SETROPTS RACLIST(SERVER) refresh

Restart MQWEB server.

CWWKS2960W: Cannot create the default credential for SAF authorization of unauthenticated users.

All authorization checks for unauthenticated users will fail.
The default credential could not be created due to the following error:

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user WSGUEST has insufficient authority to access APPL-ID MQWEB.

SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

PERMIT MQWEB CLASS(APPL) ACCESS(READ) ID(MQWSGUEST)
SETROPTS RACLIST(APPL) REFRESH

CWPKI0022E: SSL HANDSHAKE FAILURE:

A signer with SubjectDN CN=colinpaicesECp256r1, O=cpwebuser,
C=GB was sent from the target host. The signer might need to be added to local trust store safkeyring://…/…,
located in SSL configurate on alias defaultSSLConfig.
The extended error message from the SSL handshake exception is:

Unexpected error: java.security. InvalidAlgorithmParameterException:
the trustAnchors parameter must be non-empty

The full error was

CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with Subject DN  CN=colinpaice, O=HW, C=GB was sent from the target host.  The signer might need to be added to local trust store  safkeyring://START1/TRUST, located in SSL configuration alias izuSSLConfig. The extended error message from the SSL  handshake exception is: Unexpected error:  java.security.InvalidAlgorithmParameterException: the  trustAnchors parameter must be non-empt.

The problem was that the started task userid did not have update access to the trust keyring.  There was an FFDC in the log file at startup showing this.  Part of this was I assumed the wrong userid for the started task.  The z/OS Command D A,IZUSVR1 gave me th userid, which I then checked., and found it had no access.

ERROR: SEC_ERROR_REUSED_ISSUER_AND_SERIAL

I got this on a slow backend system.  I shut down the web server and restarted it, and it ran OK without the message.

ICH408I USER( ) GROUP( ) NAME()
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(…)
SUBJECT(CN=.. .O=… C=GB) ISSUER(….)

The certificate came in, but there was no mapping for it.

Use RACDCERT command to map it to a userid.

RACDCERT MAP ID(IBMUSER) –
SDNFILTER(‘CN…. ‘)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

Firefox SEC_ERROR_BAD_SIGNATURE

Dont know what caused it.  I deleted the CA and readded it and it worked. 

Others

CWWKO0801E:

Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common.

Problem:

There was no serverKeyAlias specified in the <ssl … tag.

CWPKI0024E:

The certificate alias… specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore safkeyring://…/… .

Problem

  • The certificate was not in the keyring
  • It was NOTRUST
  • It had expired
  • The CA for the certificate was not in the keyring.

MQWB0107E: Unable to parse the request data due to exception


A JSONObject text must begin with ‘{‘ at 0 [character 1 line 1]’.”,
Explanation: The REST API request failed as the data in the request payload could not be parsed.

I got this because I used a HTTP POST request instead of a HTTP GET request.


mqweb – what to do when you cannot get TLS to work?

It is hard to debug setup problems in mqweb.   I found it easiest to not use the mqweb trace, but diagnose problems from the client side.

You need to understand many TLS concepts.  I’ve documented a lot of information here: Understanding the TLS concepts for using certificates to authenticate in mqweb.

I found the easiest way to debug my mqconsole TLS setup, was to use extract the certificates from my browser’s key store and use curl’s verbose, or trace functions.   I’ve documented here how to get a Chrome trace.

I caused all of the common “user errors” and have documented the messages or symptoms I got, these are in this post.

Have you tried turning it off and on again?

The first thing you need to do if you have problems when you are configuring certificates is to restart mqweb, and your browser.   This is because updates to the keystores are not picked up till the mqweb or browser is restarted.  The Chrome and Firefox browsers, remember the certificate used, and logon this on again – so restart the browser to reset every thing.  With Chrome, I set up a bookmark url chrome://restart .

Once you have set up your first connection,  you should not need to change the mqweb server, as you will have set up the mqweb server certificate, and the CA certificate(s) to certify clients.  If you are using self signed,  you will have to import the SS certificate into the trust store, and restart the mqweb server (not good for high availability).

I found if I started chrome from a command window, instead of clicking on an icon, I got out some diagnostic messages to the command window.   These messages were slightly more useful than generic messages like “NET::ERR_CERT_AUTHORITY_INVALID”

Useful Chrome urls

  • chrome://restart
  • chrome://settings/certificates
  • chrome://net-export/ – for collecting a Chrome trace

Getting started

If you are using .pem files (for example openssl) you can use these with no further work.

If you have a .p12 (pkcs12) format keystore, you can use this with no further work.

If you are using a browser with its nssdb database, you need to extract the certificate and private key, and any CA certificates you use.  It is easy to extract a certificate and key  into a .p12 keystore.

Extract the certificate and private key from your browser’s keystore

Curl can use the browser’s key store directly if it has been compiled with NSS (instead of openssl).  “Curl -V”, built with openssl gave me “libcurl/7.58.0 OpenSSL/1.1.1″, someone else’s curl, built with NSS had “libcurl/7.19.7 NSS/3.14.3.0″.  If you do not have curl with NSS support you need to extract the certificate and key from the browsers keystore.

  • Check where your Chrome profile is.  In the Chrome browser, use the url chrome://version .   On one Chrome instance this was  /home/colinpaice/snap/chromium/986/.pki/nssdb .  On a different Chrome instance, the keystore was /home/colinpaice/.pki/nssdb .
  • Export your certificate and keystore
    • pk12util -o colinpaicex.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -n colinpaice -W password
    • pk12util – invoke this program
    • -o colinpaicex.p12  – create this pkcs12 store
    • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/  – from this repository
    • -n colinpaice  – with this name
    • -W password  – and give it this password
  • If you have created your own certificate authority, you need to extract the certificate if you do not already have it.  Firstly list the contents to remind yourself what the CA certificate is called, then extract the certificate (‘myCACert’ in my case)
    • certutil -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -L
      • This gives “Certificate Nickname ” and “Trust Attributes”.   Your CA should have a trust Attribute of “C”.
    • certutil -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -L -n “myCACert” -a >outcacert.pem
    • certutil – this program
    • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ – this key store
    • -L  – list
    • -n “myCACert”  – this name
    • -a – ASCII output
    • >outcacert.pem  – create this file

Issue the curl request

You can use the .p12 file, or the certificate.pem and the key.pem file

Example output

If you use the option — verbose  you get a lot of information for example, a successful request has

  • * Trying 127.0.0.1…
  • * TCP_NODELAY set
  • * ALPN, offering h2
  • * ALPN, offering http/1.1
  • * successfully set certificate verify locations:
  • * CAfile: ./outcacert.pem
  • CApath: /etc/ssl/certs
  • * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • * TLSv1.2 (IN), TLS handshake, Certificate (11):
  • * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • * TLSv1.2 (IN), TLS handshake, Request CERT (13):
  • * TLSv1.2 (IN), TLS handshake, Server finished (14):
  • * TLSv1.2 (OUT), TLS handshake, Certificate (11):
  • * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
  • * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • * TLSv1.2 (OUT), TLS handshake, Finished (20):
  • * TLSv1.2 (IN), TLS handshake, Finished (20):
  • * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
  • * ALPN, server did not agree to a protocol
  • * Server certificate:
  • *   subject: C=GB; O=cpwebuser; CN=mqweb5
  • *   start date: Jan 20 17:53:59 2020 GMT
  • *   expire date: Oct 16 17:53:59 2022 GMT
  • *   subjectAltName: host “127.0.0.1” matched cert’s IP address!
  • *   issuer: C=GB; O=SSS; OU=CA; CN=SSCA7
  • *  SSL certificate verify ok.
  • > GET /ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* HTTP/1.1
  • > Host: 127.0.0.1:9443

See here for an overview of the TLS handshake.   The amount of progress down the list of steps in the hand shake give you a clue as to where the problem may be.  If it is around “TLS handshake, Client Hello (1)”.  This is likely to be a problem with the server certificate.

The numbers as in TLS handshake, CERT verify (15): are the id number of the request, 15 is CERT verify.

A “Finished” message is always sent immediately after a change cipher spec message to verify that the key exchange and authentication processes were successful.  More checks are done after this.

If you use ‑‑trace filename.txt instead of ‑‑verbose you get the same data as displayed as with ‑‑verbose, plus the data flowing up and down the connection.  I found ‑‑verbose had sufficient details to resolve the problems.

Getting mqweb into production

You’ve got mqweb working,  you can now do administration using the REST API, or use a web browser in your sandbox environment to manage a queue manager.  You now want to get it ready for production – so where do you start?

I’ll document some of the things you need to do.  But to set the the scene, consider your environment

  • Production and test
  • Two major applications, accounts and payroll
  • You have multiple machines for each application, providing high availability and scalability
  • Teams of people
    • The MQ administration team who can do anything
    • The MQ RO administration team who can change the test systems, but have read only access to production
    • The applications teams who can change their test environment, but only have read only access to production
  • You will use signed certificates (because this is production)  and not use passwords.
  • People will get the same dashboard,  to make training and use easier.
  • You want to be able to quickly tell if a dashboard is for production or test, and accounts and payroll
  • You want to script deployment, so you deployment to production can be done with no manual involvement.
  • You want a secure, available solution.

The areas you need to consider are

  • the mqwebuser.xml file
  • the keystore for the mqweb certificate
  • the trust store for the authorisation certificates
  • the dashboard for each user
  • each user’s certificate store with their private keys
  • Displaying the statistics on the mq console and REST requests.

Setting up security

It is better to give access using groups rather than by individual ids.

  • If some one joins or leaves a team, you have to update one group, rather than many configuration files.
    • This is easier to do, and is easier to audit
  • The control is in the right place.  For example the manager of the accounts team should mange the accounts group.  The MQ team should not be doing userid administration on the accounts group.

You will need groups for

  • MQ Systems Administrators who can administer production and test machines
  • MQ Systems  RO Administrators,  who can administer test machines, and have read access to production machines.
  • Payroll – the applications manager may want more granular groups.
  • Accounts  – the applications manager may want more granular groups.

You will need to set up the groups on each machine (you may well have this already).

Queue security

REST users need get and put access to SYSTEM.REST.REPLY.QUEUE.

For example

setmqaut -m QMA -n SYSTEM.REST.REPLY.QUEUE -t q -g test +get +put

then runmqsc refresh security

Set up the mqwebuser.xml file

The same file can be used for the different machines for “Accounts – production”, and a similar file for “Accounts – test” etc.

You may want to use “include”  files, so have one file imbedded in more than one mqwebuser.xml file.

Do not use the setmqweb command.   This will update the copy on the machine, and it will be out of sync with the master copy in your repository.

Define roles

The production environment for payroll may have

 <security-role name="MQWebAdmin">
   <group name="MQSA"/>
</security-role>

<security-role name="MQWebAdminRO">
  <group name="MQSARO"/>
  <group name="PAYROLL"/>
</security-role>

The test environment for payroll may have

<security-role name="MQWebAdmin">
   <group name="MQSA"/>
   <group name="MQSARO"/>
   <group name="PAYROLL"/>
</security-role>

<security-role name="MQWebAdminRO">
  <!-- none -all admin users can change test-->
</security-role>

Define http settings

By default mqweb is set up for localhost only.  You will need to have

  • <variable name=”httpHost” value=”hostname” />

where hostname specifies the IP address, domain name server (DNS) host name with domain name suffix, or the DNS host name of the server where IBM MQ is installed. Use an asterisk, *, to specify all available network interfaces.

You may need to change the port value from

  • <variable name=”httpsPort” value=”9443″/>

Define the keystore in mqwebuser.xml

Decide on the names and location of the key stores

  • <keyStore id=”defaultKeyStore” location=”/home/mq/payrollproductionkeystore.p12” type=”pkcs12″ password=”{aes}AMsUYgpOjy+rxR7f/7wnAfw1gZNBdpx8RpxfwjeIG8Wj”/>
  • <keyStore id=”defaultTrustStore” location=”/home/mq/payrollproductiontruststore.jks” type=”JKS” password=”{aes}AJOmiC8YKMFZwHlfJrI2//f2Keb/nGc7E7/ojSj37I/5″/>

Encrypt the keystore passwords  using the /opt/mqm/web/bin/securityUtility command. See here.

Ensure the deployment process gives the files the appropriate access.  The key store includes the private key, so needs to be protected.  The trust store should only have information in the “public” domain, such as certificates and no private keys, so could be universally read.

Set up the keystores

The keystore has the certificate and private key which identifies the server.  The certificate needs the subjectAltName specified which has a list of valid url names and IP addresses.
You need to decide if you want one certificate per server, and so have several certificates

subjectAltName = DNS:payroll1, IP:10.4.6.1

or several systems in the list, and have one certificate

subjectAltName = DNS:payroll1, DNS:payroll2, IP:10.4.6.1,  IP:10.9.5.4

You may want to create the keystore on your build environment, and securely deploy it to the run time machines, or send the .p12 file across and import it.  I think creating the keystore and deploying it is more secure.

If you change the keystore you have to restart mqweb to pickup changes.

Set up the trust store.

The trust store is used to validate certificates sent from the client for authentication.  In an ideal work, this will have just one CA certificate.  You may have more than one CA.  If you have self signed certificates this creates a management problem.

You may be able to use the same trust store for all your environments.   The access control is done by the security-roles in the mqwebuser.xml, not by the trust store.

The cn from the certificate is used as the userid. So both

cn=colinpaice,ou=test,o=sss and cn=colinpaice,ou=prod,o=sss are valid, and would extract userid colinpaice.

If the trust store is changed, the mqweb server needs to be restarted.

End user certificates

Each user will need a certificate to be able to access the mqweb server.  This needs to be signed by your CA, and needs to be set to trusted.  You should have this set up already.

If you have more than one valid certificate in the browser store, you will be prompted to pick one.   This is used until the browser is restarted.

You can configure mqweb to log off users after a period.   If you are using certificates, the browser will automatically log you on again!

Dashboard

The dashboard is the layout of the mqweb window, the tabs in the window, and the widgets on the tabs.

You will generally want users to have the template you define, and not have to create their own. So the Payroll team use the payroll dashboard, and the MQ admin team use the MQADMIN dashboard.

Create a dashboard and use export to create a json file.   You can store in your configuration repository.   You can change queue manager names as you deploy it for example change QMPayroll1 to QMPayroll2.

On the MQ machines these files are stored in the  /var/mqm/web/installations/Installation1/angular.persistence directory.

You can put your templates for that machine in this directory, and use symbolic links for a userid to their dashboard. For example

ln -s common.json colinpaice.json

If the dashboard.json is made read only, then people will not be allowed to change it online.

 

Is this dashboard for production or test?

I could not find a way of customise the colours of a page, so you cannot easily tell which is production and which is test etc.

I need a secure available solution.

You can use userids and passwords, or certificates to provide authentication.

You need to protect access to MQ objects

You need to protect the files used by mqweb, especially the key store, and the mqwebuser.xml

If you update the mqwebsuser.xml file, it will pickup up changes a short while later (seconds rather than minute).

If you change the keystore or trust store you need to restart mqweb to pick up the changes.   This should take about 10s of  seconds.

Deploy scripts

All of the configuration can be done with scripts.  For example extract your mqwebuser.xml file, make machine specific changes and deploy it.

You can create the keystores in your secure build environment and deploy them.

House keeping

  • You should check /var/mqm/web/installations/Installation1/servers/mqweb/logs/ffdc daily for any files, and raise a PMR with PMR if you get any exceptions.
  • Check /var/mqm/web/installations/Installation1/servers/mqweb/ daily.  I was getting large (700MB) dumps in this directory, which caused my machine to go short on disk space.
  • Display the server certificate expiry date (any any CA certificates) and put a date in your diary to check (and renew) them.
  • Your enterprise should have a process for renewing personal certificates

Someone joins the department

  • Connect them to the appropriate group on all machines
  • Give them a symbolic link to the appropriate dashboard file, in /var/mqm/web/installations/Installation1/angular.persistence

Collect statistics on the MQ console and REST requests, and the JVM

See these posts

Using IBM api explorer to display the syntax of mq rest commands.

You can use an url like https://localhost:9443/ibm/api/explorer  to display the mqweb api documentation.  The documentation is in swagger format, available in a web browser.   This is the same information you get from a REST request see here.

If you enter the url in the web browser, and you are using certificate authentication, you will be prompted with a list of valid certificates.  Any certificate is valid.   Even one with a CN=unknown was accepted.   The information is read only so this is not an issue.

The web page has

  • API Discovery
  • channel
  • …
  • login
  • …

If you click login it gives

  • DELETE /ibmmq/rest/v1/login  
    • Logs out a user
  • GET   /ibmmq/rest/v1/login
    • Queries the authenticated user and their roles.
  • POST  /ibmmq/rest/v1/login
    • Logs out a user

If you click on an item it displays more information, in a nice, easy to read format.  Much easier to read than if you retrieved it using the REST API.

What it doesn’t tell you

It does not tell you to put ?attributes=*&status=* on the end of your query to get queue attributes, and qstatus returned, as in https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*&status=* .  See here for more information.

Using the url as a REST request.

Using the url from curl and –user colinpaice:password gave

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:9443

Using certificate authentication nothing was returned.

Using certificate authentication and the –include option (include headers)

url –include https://localhost:9443/ibm/api/explorer –cacert cacert.pem –cert colinpaice.pem:password –key colinpaice.key.pem

gave

HTTP/1.1 401 Unauthorized

So using the explorer url is not allowed.

mqweb – how to use the REST api commands.

You can issue MQ commands using the REST API vai the mqweb server, for example with python or curl.  The problem I had was trying to remember the syntax of the commands.  You can access this information from your program by using the “swagger” interface.  You can can access it from a browser (easiest), or from a REST request (for the experts).

I found it useful to use a curl query logon request to check the validity of the certificates I was using.  I could use the –trace curl.trace.txt to capture the data flows.  If this worked, problems were likely to be in the browser set up, rather than the certificate set up.

Note: Swagger is the name of a protocol, not an acronym.

You need to configure mqweb to use Swagger.

<featureManager>
  <feature>apiDiscovery-1.0</feature>
</featureManager>

Then use a request like (for certificate authentication)

curl -i https://localhost:9443/ibm/api/docs –cacert cacert.pem –cert colinpaice.pem:password –key colinpaice.key.pem > swagger.out

where (note the ‘–‘ may be double – – )

  • curl – issue the command
  • -i  – include the http headers in the response.  This is the same flag as –include
  • https://localhost:9443/ibm/api/docs – this url
  • –cacert cacert.pem – use this CA certificate
  • –cert colinpaice.pem:password – use this user certificate
  • –key colinpaice.key.pem – use this user private key
  • > swagger.out – write the output to this file

If you are using self signed you should use the –insecure option otherwise you get

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

If you try to userid and password you get

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:9443

The swagger output

The output contains

{
  "swagger" : "2.0",
  "info" : {
    "description" : "Discover REST APIs available within Liberty",
    "version" : "1.0.0",
    "title" : "Liberty REST APIs"
},
  "host" : "localhost:9443",
  "tags" : [ {
    ...
    "name" : "qmgr"
    ...
    "name" : "login"
    ...
    "name" : "API Discovery",
    "description" : "APIs available from the API Discovery feature"
} ],
...

We can see there is a login command.  You have to search the json for the data with “tags”  : [“login”].

The section for login has

"/ibmmq/rest/v1/login" : {
  "get" : {
    "tags" : [ "login" ],
    "summary" : "Queries the authenticated user and their roles.",
    "description" : "Retrieves details of the authenticated user and their roles.",
    "operationId" : "getLogin",
    "produces" : [ "application/json; charset=utf-8" ], 
    "parameters" : [ ],
    "responses" : {
      "200" : {
         "description" : "A JSONArray that contains JSONObjects that describe the authenticated user.",
         "schema" : {
            "$ref" : "#/definitions/UserAttributePojo"
              }
     },
...

“get”

Within the partial url /ibmmq/rest/v1/login are sections “get”, “post”, and “delete”:

  • “get” –   Queries the authenticated user and their roles  – as the “summary” tag says
  • “post”  – Authenticates a user and returns an authentication token.
  • “delete” – Deletes the authentication token of the logged in user.

You need to pick the section for what you want to do within “get”, “post”, “delete”

“tags” : [ “login” ]

This is for the request.

“200”

This is one of the list of return codes from the http request, and the data each returns.

This returns data as defined in the schema, “$ref” : “#/definitions/UserAttributePojo” .  Look in the rest of the document for this section called definitions, with UserAttributePojo underneath.

“definitions”: UserAttributePojo

 "definitions" : {

...
 "UserAttributePojo" : {
    "type" : "object",
    "required" : [ "name", "role" ],
    "properties" : {
      "name" : {
      "type" : "string"
      },
     "role" : {
       "type" : "array",
        "items" : {
          "type" : "string"
         }
      }
  }

Two strings will be returned, name and role.

Issue the command

From the data in the swagger output,  we can see  “host” : “localhost:9443” and “/ibmmq/rest/v1/login” : { “get” : {

build the command

https://localhost:9443/ibmmq/rest/v1/login –request get

my command is now

curl -i https://localhost:9443/ibmmq/rest/v1/login –cacert cacert.pem –cert colinpaice.pem:password –key colinpaice.key.pem –trace trace.txt –request get

note: -request get is inferred and can be allowed to default.

and it returns

{"user": [{
  "name": "colinpaice",
  "role": [
    "MQWebAdmin",
    "MQWebAdminRO",
    "MQWebUser"
  ]
}]}

If you want to pass data with quotes in it – for example json data. Create a file, and use that
for example
curl -i …  –request post -H “Content-Type: application/json” -d @curl.json.txt 

or use

-d "{\"param\":\"value\"}"

which is not so easy.

Setting up the end user self signed certificate for mqweb

This post describes how to set up self signed certificates to authenticate end user’s access to mqweb.

You can use self signed, which is fine for test and small environments, or use signed certificate which are suitable for production,  and typical environments.  Using certificates means you do not need to specify userid and password.

The userid is taken from the CN part of the subject, and this userid is used to grant access depending on the configuration in the mqwebuser.xml file.

Information about certificates used for authentication are stored in the trust store.  For a CA signed certificate, you only need the CA certificates, not the individual certificates.  With self signed, you need a copy of the individual self signed certificate in the mqweb trust store.

Create the trust store if required.

/opt/mqm/bin/runmqckm -keydb -create -db trust.jks -pw zpassword -type jks

You need to do this once.

Create the self signed certificate for the end user

openssl req -x509 -config openssl-client2.cnf -newkey rsa:2048 -out testuser.pem -keyout testuser.key.pem -subj “/C=GB/O=aaaa/CN=testuser” -extensions ss_extensions -passin file:password.file -passout file:password.file

  • openssl req -x509  –x509 makes this a self signed request
  • -config openssl-client2.cnf – use this config file
  • -newkey rsa:2048 – create a new private key with 2048 bits rsa key
  • -out testuser.pem – put the request in this file
  • -outform PEM – with this format
  • -keyout testuser.key.pem – put the key in this file
  • -subj “/C=GB/O=aaaa/CN=testuser” – this is the DN.   The CN= is the userid used by mqweb to determine the role.  It must match the case of userid
  • -extensions ss_extensions – see below
  • -passin file:password.file -passout file:password.file – passwords are in this file
[  ss_extensions  ]

subjectKeyIdentifier = hash
#Note: there is a bug in Chrome where it does 
# not accept certificates if basicConstraints
# is specified
# basicConstraints   = CA:false
subjectAltName       = DNS:localhost, IP:127.0.0.1
nsComment            = "OpenSSL Self signed Client"
keyUsage             = critical, keyEncipherment
extendedKeyUsage     = critical, clientAuth

Create an intermediate pkcs12 keystore so it can be imported to your browser.

You need to import the certificate and private key into the browser’s keystore.   The only way I found of doing this was via an intermediate pkcs12 keystore (with extension .p12).   If you import the certificate and key from the web browser, it will expect a .p12 file.

openssl pkcs12 -export -in testuser.pem -inkey testuser.key.pem -out testuser.p12 -name “testuser” -passin file:password.file -passout file:password.file

  • openssl pkcs12 – request to process a pkcs12 keystore
  • -export – to create it
  • -inkey testuser.key.pem – this private key
  • -in testuser.pem – this certificate returned from the CA
  • -out ssks.p12 – the name of the key store which is created
  • -name testuser – create this name in the keystore
  • -passout file:password.file -passin file:password.file – use these passwords

Import the intermediate keystore into the trust store

/opt/mqm/bin/runmqckm -cert -import -target trust.jks -target_type jks -file testuser.p12 -label testuser -pw password -target_pw zpassword

  • /opt/mqm/bin/runmqckm – run this command
  • -cert – we want to process a certificate
  • -import – w want to import a file
  • -target trust.jks – this is the mqweb trust store
  • -target_type jks – it is a jks store
  • -file testuser.p12 – input file
  • -label testuser – this is the label in the trust.jks keystore
  • -pw password – the password of the testuser.p12
  • -target_pw zpassword – the password of the trust.jks keystore

In the message.log I had

E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=testuser, O=aaaa, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path validation failed: java.security.cert.CertPathValidatorException: The certificate issued by CN=testuser, O=aaaa, C=GB is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error

When I restarted mqweb and the certificate was accepted.

I had the same message when I did not import the certificate into the trust store.

Import the temporary keystore into the Chrome keystore

pk12util -i testuser.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -W password

  • pk12util   – this command
  • -i  testuser.p12  – from this keystore
  • -d sql:/home/colinpaice/snap/chromium/971/.pki/nssdb/ – into this key store
  • -W password  – using this password (for the temporary .p12 file)

Remove the intermediate file

rm testuser.p12

Update the mqweb configuration

<webAppSecurity allowFailOverToBasicAuth="false" />
<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/ssl2/mqweb.p12" 
          type="pkcs12" 
          password="password"/>

<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/ssl2/trust.jks" 
          type="JKS" 
          password="password"/>

<ssl     id="defaultSSLConfig" 
         keyStoreRef="defaultKeyStore" serverKeyAlias="mqweb" 
         trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
         clientAuthentication="true" 
         clientAuthenticationSupported="true" 
/>

Stop mqweb

You need to restart mqweb so it picks up any changes to the trust store.

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

No messages are produced in  /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log if the trust store was opened successfully.

Use a command like grep ” E ” messages.log  and check for messages like

CWPKI0033E: The keystore located at /home/colinpaice/ssl/ssl2/trust.jks did not load because of the following error: Keystore was tampered with, or password was incorrect

Try using it in Chrome

You need to restart Chrome to pick up the changes to its keystore.  Use the url chrome://restart/

Use the url chrome://settings/certificates , to check your certificate is present under “Your certificates”. If not use url chrome://version to display the profile being used, and that it matches the store used in the pk12util command above.

Try connecting to mqweb using a url like https://127.0.0.1:9443/ibmmq/console/ .

You should be logged on with no password request. In the top right hand corner of the screen you should have a black circle with a white “i” in it.   This shows you are logged on with certificates.

Setting up the end user CA signed certificate for mqweb

You want to use certificates to authenticate access to a mqweb server.  You can use self signed, which is fine for test and small environments, or use signed certificate which are suitable for production,  and typical environments.  Using certificates means you do not need to specify userid and password.

The userid is taken from the CN part of the subject, and this userid is used to grant access depending on the configuration in the mqwebuser.xml file.

This section assumes you have set up your mqweb using a certificate authority.

Information about certificates used for authentication are stored in the trust store.  For a CA signed certificate, you only need the CA certificates, not the individual certificates.  With self signed, you need a copy of the individual self signed certificate.

Create the trust store if required.

/opt/mqm/bin/runmqckm -keydb -create -db trust.jks -pw zpassword -type jks

You need to do this once.

Add the CA certificate to the trust store

/opt/mqm/bin/runmqckm -cert -add -db trust.jks -file cacert.pem -label CACert -type jks -pw zpassword

You need to do this for each CA certificate you want to add, giving each CA a unique label.

You need to restart mqweb so it picks up any changes to the trust store, but as you will be changing the mqwebuser.xml – the restart can wait will later.

Create the certificate request for the end user

openssl req -config client.cnf -newkey rsa:2048 -out colinpaice.csr -outform PEM -keyout colinpaice.key.pem -subj “/C=GB/O=cpwebuser/CN=colinpaice” -extensions client_extensions -passin file:password.file -passout file:password.file

  • openssl req – the absence of -x509 makes this a certificate request
  • -config client.cnf – use this config file
  • -newkey rsa:2048 – create a new private key with 2048 bits rsa key
  • -out colinpaice.csr – put the request in this file
  • -outform PEM – with this format
  • -keyout colinpaice.key.pem – put the key in this file
  • -subj “/C=GB/O=cpwebuser/CN=colinpaice” – this is the DN.   The CN= is the userid used by mqweb to determine the role.  It must match the case of userid
  • -extensions client_extensions – see below
  • -passin file:password.file -passout file:password.file – passwords are in this file
[ client_extensions ]

subjectKeyIdentifier = hash
# basicConstraints = CA:FALSE
subjectAltName       = DNS:localhost, IP:127.0.0.1
nsComment = "OpenSSL ColinClient"
keyUsage = critical, nonRepudiation,digitalSignature,
extendedKeyUsage = critical, clientAuth

You need to do this for each user.

Sign it

Send the certificate request to your CA.  You can use the following command to sign it.

openssl ca -config openssl-ca-user.cnf -policy signing_policy -extensions signing_req -md sha256 -keyfile cacert.key.pem -out colinpaice.pem -infiles colinpaice.csr

  • openssl ca – the signing request
  • -config openssl-ca-user.cnf – use this config file
  • -policy signing_policy – defines the requirements for the DN.  See below
  • -extensions signing_req – see below
  • -md sha256 – what encryption to be used for the message digest
  • -keyfile cacert.key.pem – the CA authorities private key
  • -out colinpaice.pem – where the output goes
  • -infiles colinpaice.csr – the input file that needs signing

Send the *.pem file back to the requestor.

You need to do this for each user.

The signing policy allows the CA administrator to define which elements are required in the DN.

[ signing_policy ]
organizationName = supplied
commonName = supplied

The certificate needs extensions which say how the certificate can be used.

[ signing_req ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints    = CA:FALSE
keyUsage            = digitalSignature
extendedKeyUsage    =  clientAuth

Create an intermediate pkcs12 keystore so certificate can be imported

You need to import the certificate and private key into the browser’s keystore.   The only way I found of doing this was via an intermediate pkcs12 keystore (with extension .p12).   If you import the certificate and key from the web browser, it will expect a .p12 file.

openssl pkcs12 -export -inkey colinpaice.key.pem -in colinpaice.pem -out colinpaice.p12 -CAfile cacert.pem -chain -name colinpaice -passout file:password.file -passin file:password.file

  • openssl pkcs12 – request to process a pkcs12 keystore
  • -export – to create it
  • -inkey colinpaice.key.pem – this private key
  • -in colinpaice.pem – this certificate returned from the CA
  • -out colinpaice.p12 – the name of the temporary key store which is created
  • -CAfile cacert.pem – use this CA certificate
  • -chain – include any CA certificates with the certificate and key
  • -name colinpaice – create this name in the keystore
  • -passout file:password.file -passin file:password.file – use these passwords

Import the temporary keystore into the Chrome keystore

pk12util -i colinpaice.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -W password

  • pk12util   – this command
  • -i  colinpaice.p12  – from the temporary keystore you just created
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ – into this key store
  • -W password  – using this password (for the temporary .p12 file)

Remove the intermediate file

rm colinpaice.p12

You do not need to import the certificate into the mqweb trust store.

Update the mqweb configuration if required

<webAppSecurity allowFailOverToBasicAuth="false" />
<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/ssl2/mqweb.p12" 
          type="pkcs12" 
          password="password"/>

<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/ssl2/trust.jks" 
          type="JKS" 
          password="password"/>

<ssl     id="defaultSSLConfig" 
         keyStoreRef="defaultKeyStore" serverKeyAlias="mqweb" 
         trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
         clientAuthentication="true" 
         clientAuthenticationSupported="true" 
/>

Stop mqweb

It is better to stop and restart mqweb if you change the xml config files, otherwise you can get strange errors.

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

No messages are produced in  /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log if the trust store was opened successfully.

Use a command like grep ” E ” messages.log  and check for messages like

CWPKI0033E: The keystore located at /home/colinpaice/ssl/ssl2/trust.jks did not load because of the following error: Keystore was tampered with, or password was incorrect

Try using it in Chrome

You need to restart Chrome to pick up the changes to its keystore.  Use the url chrome://restart/

Use the url chrome://settings/certificates , to check your certificate is present under “Your certificates”. If not use url chrome://version to display the profile being used, and that it matches the store used in the pk12util command above.

Try connecting to mqweb using a url like https://127.0.0.1:9443/ibmmq/console/ .

You should be logged on with no password request. In the top right hand corner of the screen you should have a black circle with a white “i” in it.   This shows you are logged on with certificates.  Click on this to show how you are logged on.