mqweb – how to get a chrome browser trace

How to get a chrome trace

See Troubleshooting Chrome network issues  and the description here on how to collect a trace.

  • Open a tab with the chrome://net-export/ url.
  • Click start logging to disk
  • Select a file location
  • In another tab select the mqweb url
  • Click on the “stop” button in the window
  • If you select show file – it opens the json file.   This has all the information you need to process the file, but it is much easier to use the provided tools
  • The filename is given for example “FILE: /home/colinpaice/Downloads/chrome-net-export-log.json
  • Click on “The log file can be loaded using the netlog_viewer.” link.   This gets you to a page which says
  • This app loads NetLog files generated by Chromium’s chrome://net-export. Log data is processed and visualized entirely on the client side (your browser). Data is never uploaded to a remote endpoint.
  • Select  https://netlog-viewer.appspot.com/ to invoke the formatter.
  • Drag your netlog file, or use “choose file”
  • Select events, and this displays all of the traffic
  • In the search bar at the top enter your port 9443, or error
  • You get a list like
  • NONE HOST_RESOLVER_IMPL_REQUEST
    1083 URL_REQUEST https://127.0.0.1:9443/ibmmq/console/
    1084 DISK_CACHE_ENTRY
    1085 HTTP_STREAM_JOB_CONTROLLER https://127.0.0.1:9443/
    1086 HTTP_STREAM_JOB https://127.0.0.1:9443/
  • If the background  is pale green – it is good.  If it is pink (pale red) there was a problem.
  • Click on a line and it displays trace information in a window.  For example the first URL_REQUEST gave
    • t= 8 [st= 8]        HTTP_STREAM_JOB_CONTROLLER_BOUND
                          --> source_dependency = 1089 (HTTP_STREAM_JOB_CONTROLLER)
      t=65 [st=65]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                          --> source_dependency = 1090 (HTTP_STREAM_JOB)
      t=65 [st=65]     -HTTP_STREAM_REQUEST
      t=65 [st=65]      URL_REQUEST_DELEGATE_SSL_CERTIFICATE_ERROR  [dt=1]
      t=66 [st=66]      CANCELLED
                        --> net_error = -200 (ERR_CERT_COMMON_NAME_INVALID)
      t=66 [st=66]   -URL_REQUEST_START_JOB
                      --> net_error = -200 (ERR_CERT_COMMON_NAME_INVALID)
      t=66 [st=66]    URL_REQUEST_DELEGATE_RESPONSE_STARTED  [dt=0]
      t=66 [st=66] -REQUEST_ALIVE
      
    • SSL_CONNECT_JOB gave me
      1087: SSL_CONNECT_JOB
      ssl/127.0.0.1:9443
      Start Time: 2020-01-29 08:41:25.699
      t= 1 [st= 0] +CONNECT_JOB  [dt=64]
      t= 1 [st= 0]    SOCKET_POOL_CONNECT_JOB_CREATED
                      --> backup_job = false
                      --> group_id = "ssl/127.0.0.1:9443"
      t= 1 [st= 0]   +SSL_CONNECT_JOB_CONNECT  [dt=64]
      t= 1 [st= 0]     +TRANSPORT_CONNECT_JOB_CONNECT  [dt=0]
      t= 1 [st= 0]        HOST_RESOLVER_IMPL_REQUEST  [dt=0]
                          --> address_family = 0
                          --> allow_cached_response = true
                          --> host = "127.0.0.1:9443"
                          --> is_speculative = false
      t= 1 [st= 0]        CONNECT_JOB_SET_SOCKET
      t= 1 [st= 0]     -TRANSPORT_CONNECT_JOB_CONNECT
      t=65 [st=64]      CONNECT_JOB_SET_SOCKET
      t=65 [st=64]   -SSL_CONNECT_JOB_CONNECT
                      --> net_error = -200 (ERR_CERT_COMMON_NAME_INVALID)
      t=65 [st=64] -CONNECT_JOB
      

Understanding Chromium trace and performance data

I found this link very useful to explain the developer information, such as trace, performance etc.

mqweb charts look like “work in progress”

One of the widgets you can add to a dashboard page is a chart.   This can subscribe to the published monitoring data, and displays it as a line chart.

The topics are described in my post  What data is available with the Published Monitoring data, and include

  • Platform central processing units
      • CPU performance – platform wide  …
    • CPU performance – running queue manager …
  • Platform persistent data stores
    • Disk usage – platform wide …
    • Disk usage – running queue managers …
    • Disk usage – queue manager recovery logs …
  • API usage statistics
    • MQCONNs and MQDISCs …
    • etc
  • API per-queue usage statistics
    • MQOPEN and MQCLOSE …
    • etc

The data is published every 10 seconds or so, and the charts are refreshed around every 10 seconds.

These charts seem to be a work in progress or demonstration of technology.

After you’ve added the widget you need to click in the wheel to configure the chart.

You can select the data you want to display from drop downs

  • Select top left, for the “resource class” (major category), top right for the “resource type” (minor category) second row left for the “resource element” (detail), second row right for any object.
  • For those topics that need an object, such as a queue name, you must give  a specific object, such as CP9999, not as CP999*.
  • If you change what is being displayed, you have to select all of the data, for example,  I was collecting API usage statistics, get count per queue, and wanted to collect API usage statistics (on all queues).  I found I was changing the major category, clicking save, and getting the wrong data displayed.
    • Click on the cog
    • Select API resource class: “API usage statistics
    • Resource type: defaults to “MQCONN and MQDISC“, so select “MQGET
    • The Resource element defaults to “Interval total destructive get- count”.  I want this, so select Save.

View finder=select time range

From the cog, you can select or hide the “view finder”.  I would have called “view finder”  “select time range”.  If you “show” view finder, instead of one chart 6 cm high, the main chart is squashed into 5 cm, and there is a 1 cm squashed version below it.  It took me a while to find what this view finder is for.  If you click+hold on the “view finder” graph, and drag left or right,  the mini chart becomes a grey box with a slider.  Drag, right or left,  allows you to select the range which is displayed in the big chart.

  • To reset the window click somewhere else in the view finder chart.
  • As the small chart displays the whole time period available to you, you can drag the slider to an interesting area to allow you to drill into it.
  • You can click and drag the time range bar/gap in the view finder, so you get the same time duration, but at an earlier or later time.
  • As more data is added to the right hand side of the chart,  the time slide moves to the left over time
  • I could not find how to display just the last 5 minutes worth of data, as the window moved.
  • With the <variable name=”ltpaExpiration” value=”30″/> configuration, you get logged off after 30 minutes.  With certificates you get logged on again, but the time interval is reset, and you lose the historical data.  In this case you get no more than 30 minutes worth of data displayed.

Have data for more than one queue manager on a chart

  • If you have more than one queue manager active, you can display data from more than one queue manager.
  • A queue manager has to be active to be able to add it to a chart.   You can then stop the queue manager, and the chart will remember your selection.
  • You can select a colour for each queue manager, so for example have QMA in red, and QMB in blue, on the same chart.
  • On a chart, if you click on the circle in front of a queue manager, the circle changes from solid to empty. Click again and it goes solid.   When circle is solid the data is displayed on the chart, when it is empty, the data for that queue manager is not displayed.
    • The displayed time range may change.
  • If you move your cursor over the chart, a grey line will appear, and jump to data points, so you can see the data at that point.  It only jumped to data for one colour.
  • The number at the top of the y axis is the maximum value  displayed.

Some descriptions could be clearer

  • Some of the descriptions could be clearer, for example “interval total destructive get – count” is clear,  but “Failed MQGET – count”, is presumable for the interval as well.  I think these come from the published data.   (I found it easier to create better descriptions when I processed the published monitoring data)

The data for multiple queue managers may not be synchronised

  • For example “Platform CPU,  CPU performance – platform wide” when I had two queue managers, gave me two lines, which tend to follow each other.   The data is collected at different times (for example 10:00:03, and 10:00:07), so the data is different at each point.
  • The answer is easy – for system wide metrics just select one queue manager.

You can rename charts

  • If you hover over the title, you can click on the pencil to rename the widget.  If you clear the name it resets it to the default.
  • I changed it to “COLINs LAPTOP CPU”, then, later,  changed the chart to disk space usage.  The title “COLINs LAPTOP CPU ” was no longer relevant, so I clicked on the pencil icon, and cleared the title, and got the default chart description back.

Refresh window

  • If you refresh the window, all the charts are reset and historical data may be lost.
  • If a queue manager was stopped and restarted, refreshing the window will cause the subscription for all of the charts in the window to be reissued.

Some “hmm interesting” observations.

  • Some of the y axis data was strange.   When starting to collect some data, I had number of gets 2050, when the number of gets in the last hour was about 5.   This is from the published data.  Since data was published for the queue (over 1 week ago) there were 2050 gets from the queue since them.   The published data reported this value, and reset it.
  • Because I had <variable name=”ltpaExpiration” value=”30″/>, after 30 minutes the windows gets logged off.  Because I was using digital certificates it automatically logs on again.
  • I stopped one queue manager and restarted it.  In some charts, the data for that queue manager stopped being displayed. On other charts it was displayed successfully.
    • You need to go to the settings cog, and click save.   This reissues the subscription.
  • If you click on a the box with an arrow in it (“browse data”), by the cog, you can display the data for that chart.  Select a queue manager from the pull down list.   If you type “a” you can select all of the data – you cannot copy it to the clip board.
    • If you click on the column heading ( Timestamp or Data) you can sort ascending or descending.
  • If you select API stats for a specific queue, it does not display which queue is being displayed.
  • Sometimes data is missing.  I could see a line was missing some data.   Using the “browse data” box, I could see one queue manager had data from 13:14,  another had data from 13:15.   Both queue managers were running while I had my lunch.
  • The chart “MQ trace file system – bytes in use” reported 16KB of data – when I had over 160KB of *.trc data.  If if was for the file system, then it is a very small file system.  I do not understand this metric.

mqweb what’s the difference between the message API and the admin API?

At first glance it looks like the answer is in the question.  You can use

  • the messaging REST API put and get messages
  • the admin REST API to administer queue manager objects

In a couple of places the IBM documentation says you can use the messaging API to administer your objects, which is true at the general sense, but not the specific sense.  Until I hit a problem I thought there was one “messaging REST API” with different flavors of syntax.

Security

The admin API authorisation is managed through <security-role name=”MQWebAdmin”> and <security-role name=”MQWebAdminRO”> sections in the mqwebuser.xml file.

The messaging API authorisation is managed through <security-role name=”MQWebUser”> sections.

Access to resources is done using the Alternate Userid.  I can see in the activity trace that the userid is colinpaice(the id mqweb is running under), but the open of a queue was done with alternate userid testuser.  When I tried to browse messages on a queue, I got a message saying my userid did not have the correct authority. I used setmqaut, and mqsc command refresh security(*) to resolve it.

Cost of the admin interface

The admin interface has a request like

https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=*

which returns all of the attributes of the queue CP0000.  From the activity trace we can see

  • MQCONN + MQDISC
  • MQOPEN, MQINQ, MQCLOSE of the manager object – twice
  • MQOPEN, MQPUT, MQCLOSE to the SYSTEM.ADMIN.COMMAND.QUEUE
  • MQOPEN, MQGET, MQCLOSE to the SYSTEM.REST.REPLY.QUEUE
  • MQCMIT
  • MQBACK – the JMQI code always does this to be sure that there is no outstanding unit of work,

The most expensive request is the MQCONNect.

Using the admin interface is fine for administration because changes to objects are usually done infrequently.   If you are considering the admin interface to monitor objects, for example plot queue depths over time, the mq rest API may not be the best solution.

Cost of the messaging interface

The messaging API interface uses connection pooling.   When the application does an MQDISC, the connection is returned to a pool, and can be reused if the same userid does an MQCONN.  If the connection is not used for a period, it can be removed from the pool and an MQDISC done to release the connection.    This should eliminate frequent MQCONN and MQDISCs.

From the activity trace we see

MQOPEN, MQGET,MQGET,MQCLOSE of the queue, and no MQCONN.

There will be an MQCONN, is there is no connection available for that userid in the pool, but this should be infrequent.

Getting mqweb into production

You’ve got mqweb working,  you can now do administration using the REST API, or use a web browser in your sandbox environment to manage a queue manager.  You now want to get it ready for production – so where do you start?

I’ll document some of the things you need to do.  But to set the the scene, consider your environment

  • Production and test
  • Two major applications, accounts and payroll
  • You have multiple machines for each application, providing high availability and scalability
  • Teams of people
    • The MQ administration team who can do anything
    • The MQ RO administration team who can change the test systems, but have read only access to production
    • The applications teams who can change their test environment, but only have read only access to production
  • You will use signed certificates (because this is production)  and not use passwords.
  • People will get the same dashboard,  to make training and use easier.
  • You want to be able to quickly tell if a dashboard is for production or test, and accounts and payroll
  • You want to script deployment, so you deployment to production can be done with no manual involvement.
  • You want a secure, available solution.

The areas you need to consider are

  • the mqwebuser.xml file
  • the keystore for the mqweb certificate
  • the trust store for the authorisation certificates
  • the dashboard for each user
  • each user’s certificate store with their private keys
  • Displaying the statistics on the mq console and REST requests.

Setting up security

It is better to give access using groups rather than by individual ids.

  • If some one joins or leaves a team, you have to update one group, rather than many configuration files.
    • This is easier to do, and is easier to audit
  • The control is in the right place.  For example the manager of the accounts team should mange the accounts group.  The MQ team should not be doing userid administration on the accounts group.

You will need groups for

  • MQ Systems Administrators who can administer production and test machines
  • MQ Systems  RO Administrators,  who can administer test machines, and have read access to production machines.
  • Payroll – the applications manager may want more granular groups.
  • Accounts  – the applications manager may want more granular groups.

You will need to set up the groups on each machine (you may well have this already).

Queue security

REST users need get and put access to SYSTEM.REST.REPLY.QUEUE.

For example

setmqaut -m QMA -n SYSTEM.REST.REPLY.QUEUE -t q -g test +get +put

then runmqsc refresh security

Set up the mqwebuser.xml file

The same file can be used for the different machines for “Accounts – production”, and a similar file for “Accounts – test” etc.

You may want to use “include”  files, so have one file imbedded in more than one mqwebuser.xml file.

Do not use the setmqweb command.   This will update the copy on the machine, and it will be out of sync with the master copy in your repository.

Define roles

The production environment for payroll may have

 <security-role name="MQWebAdmin">
   <group name="MQSA"/>
</security-role>

<security-role name="MQWebAdminRO">
  <group name="MQSARO"/>
  <group name="PAYROLL"/>
</security-role>

The test environment for payroll may have

<security-role name="MQWebAdmin">
   <group name="MQSA"/>
   <group name="MQSARO"/>
   <group name="PAYROLL"/>
</security-role>

<security-role name="MQWebAdminRO">
  <!-- none -all admin users can change test-->
</security-role>

Define http settings

By default mqweb is set up for localhost only.  You will need to have

  • <variable name=”httpHost” value=”hostname” />

where hostname specifies the IP address, domain name server (DNS) host name with domain name suffix, or the DNS host name of the server where IBM MQ is installed. Use an asterisk, *, to specify all available network interfaces.

You may need to change the port value from

  • <variable name=”httpsPort” value=”9443″/>

Define the keystore in mqwebuser.xml

Decide on the names and location of the key stores

  • <keyStore id=”defaultKeyStore” location=”/home/mq/payrollproductionkeystore.p12” type=”pkcs12″ password=”{aes}AMsUYgpOjy+rxR7f/7wnAfw1gZNBdpx8RpxfwjeIG8Wj”/>
  • <keyStore id=”defaultTrustStore” location=”/home/mq/payrollproductiontruststore.jks” type=”JKS” password=”{aes}AJOmiC8YKMFZwHlfJrI2//f2Keb/nGc7E7/ojSj37I/5″/>

Encrypt the keystore passwords  using the /opt/mqm/web/bin/securityUtility command. See here.

Ensure the deployment process gives the files the appropriate access.  The key store includes the private key, so needs to be protected.  The trust store should only have information in the “public” domain, such as certificates and no private keys, so could be universally read.

Set up the keystores

The keystore has the certificate and private key which identifies the server.  The certificate needs the subjectAltName specified which has a list of valid url names and IP addresses.
You need to decide if you want one certificate per server, and so have several certificates

subjectAltName = DNS:payroll1, IP:10.4.6.1

or several systems in the list, and have one certificate

subjectAltName = DNS:payroll1, DNS:payroll2, IP:10.4.6.1,  IP:10.9.5.4

You may want to create the keystore on your build environment, and securely deploy it to the run time machines, or send the .p12 file across and import it.  I think creating the keystore and deploying it is more secure.

If you change the keystore you have to restart mqweb to pickup changes.

Set up the trust store.

The trust store is used to validate certificates sent from the client for authentication.  In an ideal work, this will have just one CA certificate.  You may have more than one CA.  If you have self signed certificates this creates a management problem.

You may be able to use the same trust store for all your environments.   The access control is done by the security-roles in the mqwebuser.xml, not by the trust store.

The cn from the certificate is used as the userid. So both

cn=colinpaice,ou=test,o=sss and cn=colinpaice,ou=prod,o=sss are valid, and would extract userid colinpaice.

If the trust store is changed, the mqweb server needs to be restarted.

End user certificates

Each user will need a certificate to be able to access the mqweb server.  This needs to be signed by your CA, and needs to be set to trusted.  You should have this set up already.

If you have more than one valid certificate in the browser store, you will be prompted to pick one.   This is used until the browser is restarted.

You can configure mqweb to log off users after a period.   If you are using certificates, the browser will automatically log you on again!

Dashboard

The dashboard is the layout of the mqweb window, the tabs in the window, and the widgets on the tabs.

You will generally want users to have the template you define, and not have to create their own. So the Payroll team use the payroll dashboard, and the MQ admin team use the MQADMIN dashboard.

Create a dashboard and use export to create a json file.   You can store in your configuration repository.   You can change queue manager names as you deploy it for example change QMPayroll1 to QMPayroll2.

On the MQ machines these files are stored in the  /var/mqm/web/installations/Installation1/angular.persistence directory.

You can put your templates for that machine in this directory, and use symbolic links for a userid to their dashboard. For example

ln -s common.json colinpaice.json

If the dashboard.json is made read only, then people will not be allowed to change it online.

 

Is this dashboard for production or test?

I could not find a way of customise the colours of a page, so you cannot easily tell which is production and which is test etc.

I need a secure available solution.

You can use userids and passwords, or certificates to provide authentication.

You need to protect access to MQ objects

You need to protect the files used by mqweb, especially the key store, and the mqwebuser.xml

If you update the mqwebsuser.xml file, it will pickup up changes a short while later (seconds rather than minute).

If you change the keystore or trust store you need to restart mqweb to pick up the changes.   This should take about 10s of  seconds.

Deploy scripts

All of the configuration can be done with scripts.  For example extract your mqwebuser.xml file, make machine specific changes and deploy it.

You can create the keystores in your secure build environment and deploy them.

House keeping

  • You should check /var/mqm/web/installations/Installation1/servers/mqweb/logs/ffdc daily for any files, and raise a PMR with PMR if you get any exceptions.
  • Check /var/mqm/web/installations/Installation1/servers/mqweb/ daily.  I was getting large (700MB) dumps in this directory, which caused my machine to go short on disk space.
  • Display the server certificate expiry date (any any CA certificates) and put a date in your diary to check (and renew) them.
  • Your enterprise should have a process for renewing personal certificates

Someone joins the department

  • Connect them to the appropriate group on all machines
  • Give them a symbolic link to the appropriate dashboard file, in /var/mqm/web/installations/Installation1/angular.persistence

Collect statistics on the MQ console and REST requests, and the JVM

See these posts

Setting up the end user self signed certificate for mqweb

This post describes how to set up self signed certificates to authenticate end user’s access to mqweb.

You can use self signed, which is fine for test and small environments, or use signed certificate which are suitable for production,  and typical environments.  Using certificates means you do not need to specify userid and password.

The userid is taken from the CN part of the subject, and this userid is used to grant access depending on the configuration in the mqwebuser.xml file.

Information about certificates used for authentication are stored in the trust store.  For a CA signed certificate, you only need the CA certificates, not the individual certificates.  With self signed, you need a copy of the individual self signed certificate in the mqweb trust store.

Create the trust store if required.

/opt/mqm/bin/runmqckm -keydb -create -db trust.jks -pw zpassword -type jks

You need to do this once.

Create the self signed certificate for the end user

openssl req -x509 -config openssl-client2.cnf -newkey rsa:2048 -out testuser.pem -keyout testuser.key.pem -subj “/C=GB/O=aaaa/CN=testuser” -extensions ss_extensions -passin file:password.file -passout file:password.file

  • openssl req -x509  –x509 makes this a self signed request
  • -config openssl-client2.cnf – use this config file
  • -newkey rsa:2048 – create a new private key with 2048 bits rsa key
  • -out testuser.pem – put the request in this file
  • -outform PEM – with this format
  • -keyout testuser.key.pem – put the key in this file
  • -subj “/C=GB/O=aaaa/CN=testuser” – this is the DN.   The CN= is the userid used by mqweb to determine the role.  It must match the case of userid
  • -extensions ss_extensions – see below
  • -passin file:password.file -passout file:password.file – passwords are in this file
[  ss_extensions  ]

subjectKeyIdentifier = hash
#Note: there is a bug in Chrome where it does 
# not accept certificates if basicConstraints
# is specified
# basicConstraints   = CA:false
subjectAltName       = DNS:localhost, IP:127.0.0.1
nsComment            = "OpenSSL Self signed Client"
keyUsage             = critical, keyEncipherment
extendedKeyUsage     = critical, clientAuth

Create an intermediate pkcs12 keystore so it can be imported to your browser.

You need to import the certificate and private key into the browser’s keystore.   The only way I found of doing this was via an intermediate pkcs12 keystore (with extension .p12).   If you import the certificate and key from the web browser, it will expect a .p12 file.

openssl pkcs12 -export -in testuser.pem -inkey testuser.key.pem -out testuser.p12 -name “testuser” -passin file:password.file -passout file:password.file

  • openssl pkcs12 – request to process a pkcs12 keystore
  • -export – to create it
  • -inkey testuser.key.pem – this private key
  • -in testuser.pem – this certificate returned from the CA
  • -out ssks.p12 – the name of the key store which is created
  • -name testuser – create this name in the keystore
  • -passout file:password.file -passin file:password.file – use these passwords

Import the intermediate keystore into the trust store

/opt/mqm/bin/runmqckm -cert -import -target trust.jks -target_type jks -file testuser.p12 -label testuser -pw password -target_pw zpassword

  • /opt/mqm/bin/runmqckm – run this command
  • -cert – we want to process a certificate
  • -import – w want to import a file
  • -target trust.jks – this is the mqweb trust store
  • -target_type jks – it is a jks store
  • -file testuser.p12 – input file
  • -label testuser – this is the label in the trust.jks keystore
  • -pw password – the password of the testuser.p12
  • -target_pw zpassword – the password of the trust.jks keystore

In the message.log I had

E CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN CN=testuser, O=aaaa, C=GB was sent from the target host. The signer might need to be added to local trust store /home/colinpaice/ssl/ssl2/trust.jks, located in SSL configuration alias defaultSSLConfig. The extended error message from the SSL handshake exception is: PKIX path validation failed: java.security.cert.CertPathValidatorException: The certificate issued by CN=testuser, O=aaaa, C=GB is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error

When I restarted mqweb and the certificate was accepted.

I had the same message when I did not import the certificate into the trust store.

Import the temporary keystore into the Chrome keystore

pk12util -i testuser.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -W password

  • pk12util   – this command
  • -i  testuser.p12  – from this keystore
  • -d sql:/home/colinpaice/snap/chromium/971/.pki/nssdb/ – into this key store
  • -W password  – using this password (for the temporary .p12 file)

Remove the intermediate file

rm testuser.p12

Update the mqweb configuration

<webAppSecurity allowFailOverToBasicAuth="false" />
<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/ssl2/mqweb.p12" 
          type="pkcs12" 
          password="password"/>

<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/ssl2/trust.jks" 
          type="JKS" 
          password="password"/>

<ssl     id="defaultSSLConfig" 
         keyStoreRef="defaultKeyStore" serverKeyAlias="mqweb" 
         trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
         clientAuthentication="true" 
         clientAuthenticationSupported="true" 
/>

Stop mqweb

You need to restart mqweb so it picks up any changes to the trust store.

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

No messages are produced in  /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log if the trust store was opened successfully.

Use a command like grep ” E ” messages.log  and check for messages like

CWPKI0033E: The keystore located at /home/colinpaice/ssl/ssl2/trust.jks did not load because of the following error: Keystore was tampered with, or password was incorrect

Try using it in Chrome

You need to restart Chrome to pick up the changes to its keystore.  Use the url chrome://restart/

Use the url chrome://settings/certificates , to check your certificate is present under “Your certificates”. If not use url chrome://version to display the profile being used, and that it matches the store used in the pk12util command above.

Try connecting to mqweb using a url like https://127.0.0.1:9443/ibmmq/console/ .

You should be logged on with no password request. In the top right hand corner of the screen you should have a black circle with a white “i” in it.   This shows you are logged on with certificates.

Setting up the end user CA signed certificate for mqweb

You want to use certificates to authenticate access to a mqweb server.  You can use self signed, which is fine for test and small environments, or use signed certificate which are suitable for production,  and typical environments.  Using certificates means you do not need to specify userid and password.

The userid is taken from the CN part of the subject, and this userid is used to grant access depending on the configuration in the mqwebuser.xml file.

This section assumes you have set up your mqweb using a certificate authority.

Information about certificates used for authentication are stored in the trust store.  For a CA signed certificate, you only need the CA certificates, not the individual certificates.  With self signed, you need a copy of the individual self signed certificate.

Create the trust store if required.

/opt/mqm/bin/runmqckm -keydb -create -db trust.jks -pw zpassword -type jks

You need to do this once.

Add the CA certificate to the trust store

/opt/mqm/bin/runmqckm -cert -add -db trust.jks -file cacert.pem -label CACert -type jks -pw zpassword

You need to do this for each CA certificate you want to add, giving each CA a unique label.

You need to restart mqweb so it picks up any changes to the trust store, but as you will be changing the mqwebuser.xml – the restart can wait will later.

Create the certificate request for the end user

openssl req -config client.cnf -newkey rsa:2048 -out colinpaice.csr -outform PEM -keyout colinpaice.key.pem -subj “/C=GB/O=cpwebuser/CN=colinpaice” -extensions client_extensions -passin file:password.file -passout file:password.file

  • openssl req – the absence of -x509 makes this a certificate request
  • -config client.cnf – use this config file
  • -newkey rsa:2048 – create a new private key with 2048 bits rsa key
  • -out colinpaice.csr – put the request in this file
  • -outform PEM – with this format
  • -keyout colinpaice.key.pem – put the key in this file
  • -subj “/C=GB/O=cpwebuser/CN=colinpaice” – this is the DN.   The CN= is the userid used by mqweb to determine the role.  It must match the case of userid
  • -extensions client_extensions – see below
  • -passin file:password.file -passout file:password.file – passwords are in this file
[ client_extensions ]

subjectKeyIdentifier = hash
# basicConstraints = CA:FALSE
subjectAltName       = DNS:localhost, IP:127.0.0.1
nsComment = "OpenSSL ColinClient"
keyUsage = critical, nonRepudiation,digitalSignature,
extendedKeyUsage = critical, clientAuth

You need to do this for each user.

Sign it

Send the certificate request to your CA.  You can use the following command to sign it.

openssl ca -config openssl-ca-user.cnf -policy signing_policy -extensions signing_req -md sha256 -keyfile cacert.key.pem -out colinpaice.pem -infiles colinpaice.csr

  • openssl ca – the signing request
  • -config openssl-ca-user.cnf – use this config file
  • -policy signing_policy – defines the requirements for the DN.  See below
  • -extensions signing_req – see below
  • -md sha256 – what encryption to be used for the message digest
  • -keyfile cacert.key.pem – the CA authorities private key
  • -out colinpaice.pem – where the output goes
  • -infiles colinpaice.csr – the input file that needs signing

Send the *.pem file back to the requestor.

You need to do this for each user.

The signing policy allows the CA administrator to define which elements are required in the DN.

[ signing_policy ]
organizationName = supplied
commonName = supplied

The certificate needs extensions which say how the certificate can be used.

[ signing_req ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints    = CA:FALSE
keyUsage            = digitalSignature
extendedKeyUsage    =  clientAuth

Create an intermediate pkcs12 keystore so certificate can be imported

You need to import the certificate and private key into the browser’s keystore.   The only way I found of doing this was via an intermediate pkcs12 keystore (with extension .p12).   If you import the certificate and key from the web browser, it will expect a .p12 file.

openssl pkcs12 -export -inkey colinpaice.key.pem -in colinpaice.pem -out colinpaice.p12 -CAfile cacert.pem -chain -name colinpaice -passout file:password.file -passin file:password.file

  • openssl pkcs12 – request to process a pkcs12 keystore
  • -export – to create it
  • -inkey colinpaice.key.pem – this private key
  • -in colinpaice.pem – this certificate returned from the CA
  • -out colinpaice.p12 – the name of the temporary key store which is created
  • -CAfile cacert.pem – use this CA certificate
  • -chain – include any CA certificates with the certificate and key
  • -name colinpaice – create this name in the keystore
  • -passout file:password.file -passin file:password.file – use these passwords

Import the temporary keystore into the Chrome keystore

pk12util -i colinpaice.p12 -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ -W password

  • pk12util   – this command
  • -i  colinpaice.p12  – from the temporary keystore you just created
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb/ – into this key store
  • -W password  – using this password (for the temporary .p12 file)

Remove the intermediate file

rm colinpaice.p12

You do not need to import the certificate into the mqweb trust store.

Update the mqweb configuration if required

<webAppSecurity allowFailOverToBasicAuth="false" />
<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/ssl2/mqweb.p12" 
          type="pkcs12" 
          password="password"/>

<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/ssl2/trust.jks" 
          type="JKS" 
          password="password"/>

<ssl     id="defaultSSLConfig" 
         keyStoreRef="defaultKeyStore" serverKeyAlias="mqweb" 
         trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
         clientAuthentication="true" 
         clientAuthenticationSupported="true" 
/>

Stop mqweb

It is better to stop and restart mqweb if you change the xml config files, otherwise you can get strange errors.

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

No messages are produced in  /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log if the trust store was opened successfully.

Use a command like grep ” E ” messages.log  and check for messages like

CWPKI0033E: The keystore located at /home/colinpaice/ssl/ssl2/trust.jks did not load because of the following error: Keystore was tampered with, or password was incorrect

Try using it in Chrome

You need to restart Chrome to pick up the changes to its keystore.  Use the url chrome://restart/

Use the url chrome://settings/certificates , to check your certificate is present under “Your certificates”. If not use url chrome://version to display the profile being used, and that it matches the store used in the pk12util command above.

Try connecting to mqweb using a url like https://127.0.0.1:9443/ibmmq/console/ .

You should be logged on with no password request. In the top right hand corner of the screen you should have a black circle with a white “i” in it.   This shows you are logged on with certificates.  Click on this to show how you are logged on.

Setting up a self signed certificate for the mqweb server end

When using mqweb with certificates you can use

  • a self signed certificate to identify the server
  • a CA signed certificate to identify the server

You can use certificates to authenticate…

  • a self signed certificate at the client end
  • a CA signed certificate at the client end

This post explains how I set up mqweb to use a self  signed certificate at the server, and to import the certificate into my Chrome browser.

The tasks are

  • create the self signed certificate
  • create the keystore and import the certificate
  • update the mqwebuser.xml file
  • import the certificate into the browser keystore

Create the openssl config file

You configure parameters in sections in a config file, and use a command parameter to identify which sections to use.

For the self signed certificiate I set up ss_extensions

[ ss_extensions ]

subjectKeyIdentifier = hash
#Note: there is a bug in Chrome where it does 
# not accept certificates if basicConstraints
# is specified
# basicConstraints = CA:false

subjectAltName = DNS:localhost, IP:127.0.0.1
nsComment = "OpenSSL ColinClientSS28"
keyUsage = critical, keyEncipherment
extendedKeyUsage = critical, serverAuth

Create the self signed certificate

I set up a shell script to automate the tasks

Create the self signed certificate

openssl req -x509 -config openssl-client2.cnf -newkey rsa:2048 -out ss.pem -keyout ss.key.pem -subj “/C=GB/O=aaaa/CN=colinpaice” -extensions ss_extensions -passin file:password.file -passout file:password.file

  • -config openssl-client2.cnf  – the location of the openssl configutation file ( see -extensions)
  • -newkey rsa:2048 – define a self signed certiticate
  • -out ss.pem – where the certificate is stored
  • -keyout ss.key.pem– where the private key is stored
  • -subj “/C=GB/O=aaaa/CN=colinpaice” – this is the DN of the certificate.  I Used O=aaaa so it was at the top of any list of certificates
  • -extensions ss_extensions see above
  • -passin file:password.file -passout file:password.file – openssl uses passwords. The file has two lines each with a password in it.

delete the old keystore

rm ssks.p12

  • delete the old keystore

create the keystore and import the certificate

openssl pkcs12 -export -in ss.pem -inkey ss.key.pem -out ssks.p12 -name “server” -passin file:password.file -passout file:password.file

  • pkcs12 -export – create the pkcs12 keystore
  • -in ss.pem -inkey ss.key.pem – the two files which are imported into the keystore
  • -out ssks.p12 – this is the keystore used by mqweb
  • -name “server”– this is the label given to the certificate in the keystore, used in serverKeyAlias in the mqweb xml configuration
  • -passin file:password.file -passout file:password.file – this gives the passwords to use

Other commands I used, to display information about the certificate

  • openssl x509 -purpose -in ss.pem -inform PEM -nocert
  • openssl x509 -in ss.pem -noout -ext subjectAltName
  • openssl x509 -in ss.pem -noout -ext keyUsage
  • openssl x509 -in ss.pem -noout -ext extendedKeyUsage

Update the mqweb configutation file

<keyStore id="defaultKeyStore" 
          location="/home/colinpaice/ssl/ssks.p12"  
          type="pkcs12" 
          password="password"/> 
<!-- this is used for authentication with user certificates
<keyStore id="defaultTrustStore" 
          location="/home/colinpaice/ssl/key.jks" 
          type="JKS" 
          password="zpassword"/>
-->
<ssl id="defaultSSLConfig" 
     keyStoreRef="defaultKeyStore" 
     serverKeyAlias="server" 
     clientAuthentication="false" 
     clientAuthenticationSupported="false" 
/>
<!--trustStoreRef="defaultTrustStore" sslProtocol="TLSv1.2"
-->

Stop mqweb

/opt/mqm/bin/endmqweb

Start mqweb

/opt/mqm/bin/strmqweb

Check /var/mqm/web/installations/Installation1/servers/mqweb/logs/messages.log for messages like

[14/01/20 09:12:15:730 GMT] 00000024 com.ibm.ws.ssl.config.WSKeyStore E CWPKI0033E: The keystore located at /home/colinpaice/ssl/ssks.p12 did not load because of the following error: keystore password was incorrect.

Import it into Chrome

If you do not import it into the web browser, the web browser will complain and give you the option to accept it as untrusted.  If this is just a test system this may be acceptable.  If you want to be able to trust the web server, you need to import the certificate into the browser’s keystore, as trusted.

I have several profiles for Chrome.  At one point it hickuped and created a new profile.

Find the keystore

In Chrome use the url chrome://version/ this gives the profile path, for example /home/colinpaice/snap/chromium/986/.config/chromium/Default

Remove the old certificate

certutil -D -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb -n server

  • certutil -D – delete the certificate
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb – from this keystore directory
  • -n server with this name

Add the new certificate

certutil -A -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb -t “P,,” -n server -i $name.pem

  • certutil -A – add a certificate
  • -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb – into this keystore directory
  • -t “P,,” – give it these permissions.
    • P says Trusted Peer certificate.   The certificate appears in Chrome under “servers”
  • -n server – with this name
  • -i ss.pem – and this certificate

Tell Chrome to pickup the changes

Use the url chrome://restart to restart chrome

Try using it

use the url like https://127.0.0.1:9443/ibmmq/console/

You should get the IBM MQ Console – login

 

Certificates in Chrome

You can import certificates without using the certutil command.

You can display the certificates in Chrome using the url chrome://settings/certificates .

If you click on “Your certificates” or “Servers”, and then click on import, you can import a certificate.

I selected the directory, and then the ss.pem file.   The certificate ended up in “Others” under

org-aaaa

  • UNTRUSTED colinpaice

Use the url chrome://restart to restart chrome

Use the url like https://127.0.0.1:9443/ibmmq/console/

This gave me the “Your connection is not private”,  NET::ERR_CERT_AUTHORITY_INVALID.

The only way I found of making it trusted was to use the certutil command

certutil -L -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb

This listed the certificates in the store.   The one I had just added was colinpaice #2

certutil -M -d sql:/home/colinpaice/snap/chromium/986/.pki/nssdb -n “colinpaice #2” -t P,,

This modifies the certificate to be a Trusted Peer.  Restart Chrome, and the certificate appeared in the “Servers” section.

Using the mqweb url now worked successfully