Category: mqweb

Colin’s list of MQ messages

This blog post is my annotations to MQ messages, containing descriptions of what I did to get an MQ message, and what I did to fix the problem. It is meant for web search programs, rather than humans.

I will extend it as I experience problems.

AMQ7026E: A principal or group name was invalid.

I could display an auth entry

dspmqaut -m qml -t qmgr -g “cn=dynamic, o=Your Company”


Entity cn=dynamic, o=Your Company has the following authorizations for object qml:
connect

but not delete it

setmqaut -m qml -t qmgr -g “cn=dynamic,o=Your Company” -connect

AMQ7026E: A principal or group name was invalid.

I had set this up using LDAP, but MQ was not able to find the record in LDAP. This is because I had changed the configuration. There was an LDAP search

(&(objectClass=groupOfNames)(OU=cn=dynamic, o=Your Company))

It could not be found because there was no LDAP entry with objectClass=groupOfNames with an entry (OU=cn=dynamic, o=Your Company)

I had changed

DEFINE AUTHINFO(MYLDAP) +
AUTHTYPE(IDPWLDAP) +
GRPFIELD(sn)

to

GRPFIELD(ou)

So when MQ came to delete it, it looked for

(&(objectClass=groupOfNames)(OU=cn=dynamic, o=Your Company))

instead of

(&(objectClass=groupOfNames)(SN=cn=dynamic, o=Your Company))

and could not delete it.

I changed the LDAP group to add the OU, and the SETMQAUT command worked.

AMQ5532E: Error authorizing entity in LDAP

EXPLANATION:
The LDAP authorization service has failed in the ldap_first_entry call while
trying to find user or group ‘NULL’. Returned count is 0. Additional context is
‘cn=mqadmin,ou=groups,o=your Company’.

Colin’s comments

I had defined a dynamic group, by specifying the group name as part of the LDAP user entry.

When I changed the authinfo object yo have nestgrp(yes), I got the above message because there was no record for cn=mqadmin,ou=groups,o=your Company’.

Define the record with the appropriate object class as defined by the MQ AUTHINFO attribute CLASSGRP. (CLASSGRP(‘groupOfNames’) in my case).

AMQ5530E: Error from LDAP authentication and authorization service

EXPLANATION:
The LDAP authentication and authorization service has failed. The
‘ldap_ssl_environment_init’ call returned error 113 : ‘SSL initialization call
failed’. The context string is ‘keyfile=”/var/mqm/qmgrs/qml/ssl/key.kdb”
SSL/TLS rc=408 (ERROR BAD KEYFILE PASSWORD)’. Additional code is 0.

Colin’s comments.

I had changed the keyring using alter qmgr CERTLABL(ECRSA1024) SSLKEYR(‘/home/colinpaice/mq/zzserver’)

AMQ5530E: Error from LDAP authentication and authorization service

EXPLANATION:
The LDAP authentication and authorization service has failed. The
‘ldap_simple_bind’ call returned error 49 : ‘Invalid credentials’. The context
string is ‘10.1.1.2:389 ‘. Additional code is 0.

Colin’s comments

An anonymous logon to LDAP was attempted (LDAPUSER and LDAPPWD omitted) and the LDAP server had allowAnonymousBinds off.

Specify userid and password.

API: 2460 (099C) (RC2460): MQRC_HMSG_ERROR during get

I had this during an MQGET when the GMO was not using Version:4.

This would apply to PMO not using Version:3.

Reason 2464: FAILED: MQRC_IMPO_ERROR

MQIMPO impo = {MQIMPO_DEFAULT};

I got this return code because I had a C program on z/OS and compiled it using the ASCII option. This meant the IMPO eye catcher was ASCII 0x494D504F)  instead of EBCDIC 0xC9D4D7D6.

MQIMPO impo = {MQIMPO_DEFAULT};

// if we are in ascii mode we need to convert eyecatcher from ASCII to
// EBCDIC
__a2e_l(impo.StrucId,sizeof(impo.StrucId));

This also applies to

2482: FAILED: MQRC_PD_ERROR

2440 (0988) (RC2440): MQRC_SUB_NAME_ERROR

I got this because my data was in EBCDIC, but I had specified the code page as 437 ( ASCII).

AMQ9669E The PKCS #11 token could not be found

AMQ9669E The PKCS #11 token could not be found. Severity 30 : Error Explanation The PKCS #11 driver failed to find the token specified to MQ in the PKCS #11 token label field of the GSK_PKCS11 SSL CryptoHardware parameter. The channel is <insert_3>; in some cases its name cannot be determined and so is shown as ‘????’. The channel did not start. Response Ensure that the PKCS #11 token exists with the label specified. Restart the channel.

I got this when I had the following in my mqclient.ini

SSL:
…
SSLCryptoHardware=GSK_PKCS11=/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so\;UserPIN (mytoken)\;12345678\;SYMMETRIC_CIPHER_ON\;

I commented out the SSLCryptoHardware… (I used #SSLCryptoHardware…) and it worked.

Z/OS messages

CSQY010E %CSQ9 CSQYASCP LOAD MODULE CSQYARIB IS NOT AT THE CORRECT
RELEASE LEVEL

If you get CSQYARIB then this can be caused by running Beta code past its validity date.

CSQX630E %CSQ9 CSQXRESP Channel ???? requires SSL

The chinit needs some SSL tasks and a keyring

%csq9 ALTER QMGR SSLTASKS(5) SSLKEYR(MQRING)

Restart the CHINIT.

Return codes

3015 (0BC7) (RC3015): MQRCCF_CFST_PARM_ID_ERR

IBM Documentation: Explanation. Parameter identifier is not valid. The MQCFST Parameter field value was not valid.

Colin’s comment. I passed a value which was valid, but the context was not valid. For example I issued a PCF command to start SMDSCONN. On the console the command gave

CSQM174E M801 CSQMSSMD ‘SMDSCONN’ KEYWORD IS NOT ALLOWED WITH CFLEVEL(4) – THIS KEYWORD REQUIRES CFLEVEL(5)

3229 (0C9D) (RC3220) MQRCCF_PARM_VALUE_ERROR

I got this trying to use MQCMD_INQUIRE_Q.

I had used qtype=MQOT_LOCAL_Q = 1004 when I should have used MQQT_LOCAL = 1

The PCF return message told me the incorrect parameter and value

2019 (07E3) (RC2019): MQRC_HOBJ_ERROR

I got the MQRC_HOBJ_ERROR when using MQSUB to subscribe to a topic. This is described here.

Colin’s comment

I got this because the queue I was using was not consistent with the subscription definition. For example

  • The subscription was using a managed subscription and I was using a queue
  • The queue name I specified in the queue handle did not match the queue name in the subscription

You can use the DISPLAY SUB(..) DEST DESTCLAS

A managed subscription will have a DEST(SYSTEM.MANAGED.DURABLE…) and DESTCLAS(MANAGED).

When using a queue you will have DEST(COLINSUBQ) DISTYPE(RESOLVED)


Have a good REST and save a fortune in CPU with Python

Following on from Have a good REST and save a fortune in CPU. The post gives some guidance on reducing the costs of using Liberty based servers from a Python program.

Certificate set up

I used certificate authentication from Linux to z/OS. I used

  • A certificate defined on Linux using Openssl.
  • I sent the Linux CA certificate to z/OS and imported it to the TRUST keyring.
  • I created a certificate on z/OS and installed it into the KEY keyring.
  • I exported the z/OS CA, sent it to Linux, and created a file called tempca.pem.

Python set up

Define the names of the user certificate private key, and certificate

cf=”colinpaicesECp256r1.pem”
kf=”colinpaicesECp256r1.key.pem”
cpcert=(cf,kf)

Define the name of the certificate for validating the server’s certificate

v=’tempca.pem’

Set up a cookie jar to hold the cookies sent down from the server

jar = requests.cookies.RequestsCookieJar()

Define the URL and request

geturl =”https://10.1.1.2:9443/ibmmq/rest/v1/admin/qmgr/ “

Define the headers

import base64
useridPassword = base64.b64encode(b’colin:passworm’)
my_header = {
‘Content-Type’: ‘application/json’,
‘Authorization’: useridPassword,
‘ibm-mq-rest-csrf-token’ : ‘ ‘
}

An example flow of two requests, using two connections

For example using python

s = requests
response1 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert)
response2 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert)

creates two session, each has a TLS handshake, issue a request, get a response and end.

An example of two requests using one session

For example using python

s = requests.Session()
response1 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert1)
response2 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert2)

The initial request has one expensive TLS handshake, the second request reuses the session.

Reusing this session means there was only one expensive Client Hello,Server Hello exchange for the whole conversation.

Even though the second request specified a different set of certificates, the certificates from when the session was established, using cpcert1 were used. (No surprise here as the certificates are only used when the session is established).

For the authentication, in both cases the first requests received a cookie with the LtpaToken2 cookie in it.

When this was passed up on successive requests, the userid information from the first request was used.

What is the difference?

I ran a workload of a single thread doing 200 requests. The ratios are important, not the absolute values.

Shared sessionOne session per requests
TCP flows to server1 11
CPU cost1 5
Elapsed time16

Have a good REST and save a fortune in CPU

The REST protocol is a common programming model with the internet. It is basically a one shot model, which scales and has high availability, but can have a very high CPU cost. There are things you can do to reduce the CPU cost. Also, the MQWeb server, has implemented some changes to reduce the cost. See here for the MQ documentation.

The post gives some guidance on reducing the costs, for Liberty based servers.

The traditional model and the REST model

The traditional application model may have a client and a flow to the server

  • Connect to the server and authenticate
  • Debit my account by £500 within syncpoint
  • Credit your account by £500 within syncpoint
  • Commit the transaction
  • Do the next transaction etc
  • At the end of the day, disconnect from the server.

The REST model would be

  • Connect to the server and authenticate and do (Debit my account by £500, credit your account by £500), disconnect

This model has the advantage that it scales. When you initiate a transaction it can go to any one of the available back-end servers. This spreads the load and improves availability.

With the traditional model, the clients connects any available server at the start of day stays connected all day. If a new server becomes available during the day, it may get no workload.

The downside of the REST model is the cost. Establishing a connection and authenticating can be very expensive. I could not find much useful documentation on how to reduce these costs.

There are two parts of getting a REST connection.

  • Establishing the connection
  • Authentication

Establishing the connection

You can have each REST request use a new session for every REST request each of which which involves a full TLS handshake. Two requests could go to different servers, or go to the same server.

You can issue multiple REST request over the same session, to the same backend server.

On my little z/OS, using z/OSMF it takes

  • about 1 second to create a new connection and issue a request and terminate
  • about 0.1 seconds to use the shared session, per REST request.

Establishing the TLS session is expensive, as there is a lot of computation to generate the keys.

For MQWEB, the results are very similar.

Authentication

Once the session has been established each REST request requires authentication.

If you are using userid and password, the values are checked with z/OS.

If you are using client certificate authentication the Subject DN is looked up in the security manager, and if there is a DN to userid mapping, the userid is returned.

Once you have a valid userid, the userid’s access can be obtained from the security manager.

All of these values can be cached in the Liberty web server. So the first time a certificate or userid is used, it will take a longer than successive times.

Information about authentication is then encrypted and passed back in the REST response as the LtpaToken2 cookie.

If a REST request passes the cookie back to the server, then the information in the cookie is used by the server, and fewer checks need to be done.

This cookie can expire, and when it does expire the userid and password, or the certificate DN is checked as before, and the cookie will be updated.

If you do not send the LtpaToken2 cookie, this will cause the passed authentication information to be revalidated. If you want to change userid, do not send the the cookie.

Is any of this documented?

There is not a lot of documentation. There is information Configuring the authentication cache in Liberty.

There is a parameter javax.net.ssl.sessionCacheSize. If this is not set the default is 20480.

What they don’t tell you about using a REST interface.

After I stumbled on a change to my Python program which gave 10 times the throughput to a Web Server, I realised that I knew only a little about using REST. It is the difference between the knowledge to get a Proof Of Concept working, and the knowledge to run properly in production; it is the difference between one request a minute to 100 requests a second.

This blog post compares REST and traditional client server and suggests ways of using REST in production. The same arguments also apply to long running classical client server applications.

A REST request is a stateless, self contained request which you send to the back-end server, and get one response back. It is also known as a one shot request. Traditional client server applications can send several requests to the back-end as part of a unit of work.

In the table below I compare an extreme REST transaction, and an extreme traditional Client Server

AttributeRESTClient Server
ConnectionCreate a new connection for every request.Connect once, stay connected all day, reuse the session, disconnect at end of day.
Workload BalancingThe request can select from any available server, and so on average, requests will be spread across all connections. If a new server is added, then it will get used.The application connects to a server and stays connected. If the session ends and restarts, it may select a different server.
If a new server is added, it may not be used.
AuthenticationEach request needs authentication. If the userid is invalidated, the request will fail. Note that servers cache userid information, so it may take minutes before the request is
re-authenticated.		Authentication is done as part of the connection. If the userid is invalidated during the day, the application will carry on working until it restarts.
IdentificationBoth userid+password, and client certificate can be used to give the userid.Both userid+password, and client certificate can be used to give the userid. If you want to change which identity is used, you should disconnect and reconnect.
CostIt is very expensive to create a new connection. It is even more expensive when using TLS, because of the generation of the secret key. As a result it is very very expensive to use REST requests.The expensive create connection is done once, at start of day. Successive request do not have this overhead, so are much cheaper
Renew TLS session keyBecause there is only one transfer per connection you do not need to renew the encryption key.Using the same session key for a whole day is weak, as it makes it easier to break it. Renewing the session key after an amount of data has been processed, or after a time period is good practice.
RequestSome requests are suitable for packaging in one request, for example where just one server is involved.This can support more complex requests, for example DB2 on system A, and MQ on system B.
Number of connectionsThe connection is active only when it is used.The connection is active even though it has not been used for a long time. This can waste resources, and prevent other connections from being made to the server.
StatisticsYou get an SMF record for every request. Creating an SMF record costs CPU.You get one SMF record for collection of work, so reducing the overall costs. The worst case is one SMF record for the whole day.

What are good practices for using REST (and Client Server) in production?

Do not have a new connection for every request. Create a session which can be reused for perhaps 50 requests or ten minutes, depending on workload. This has the advantages :

  • You reduce the costs of creating the new connection for every request, by reusing the session.
  • You get workload balancing. With the connection ending and being recreated periodically, you will get the connections spread across all available connections. You should randomise the time a connection is active for, so you do not get a lot of time-out activity occurring at the same time
  • You get the re-authentication regularly.
  • The TLS key is renewed periodically.
  • You avoid the long running connections doing nothing.
  • For a REST request you may get fewer SMF records, for a Client-Server you get more SMF requests, and so more granular data.

How can I do this?

With Java you can open a connection, and have the client control how long it is open for.

With Python and the requests package, you can use

s = requests.Session()
res = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert)
…
res = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert)
etc

With Curl you can reuse the session.

Do I need to worry if my throughput is low?

No, If you are likely to have only one request to a server, and so cannot benefit from having multiple requests per connection you might just as well stay with a “one shot” and not use any of the tuning suggestions.

Running a python rest application on z/OS

I installed Python and co-req packaged on my z/OS system, described here. I wanted to run a REST workload into z/OSMF. I could have used, Liberty, z/OS Connect or MQWEB as the backend.

It makes use of the python requests package.

Initial script

#!/usr/bin/env python3 
import requests 
from timeit import default_timer as timer 
import urllib3 
my_header = { 
  'Connection': 'keep-alive', 
  'Content-Type': 'application/json', 
  'Cache-Control': 'max-age=0', 
  'Authorization': 'Basic Y395aW46cGFu67GhlMG4=', 
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 
  'Accept-Language': 'en-GB,en;q=0.5', 
  'DNT': '1', 
  'Connection': 'keep-alive', 
  'Upgrade-Insecure-Requests': '1', 
  'Cache-Control': 'max-age=0' 
}
urllib3.disable_warnings() 
geturl ="https://10.1.1.2:29443/zosmf/rest/mvssubs?ssid=IZUG" 
jar = requests.cookies.RequestsCookieJar() 
start=timer() 
res = requests.get(geturl,headers=my_header,verify=False,cookies=jar) 
end=timer() 
print("duration=",end-start) 
if res.status_code != 200: 
  print(res.status_code) 
print("Output=",res.text) 
jar=  res.cookies

Comments on the python script

  • Authorization’: ‘Basic Y395aW46cGFu67GhlMG4=’ is the 64 bit encoding of the userid and password. Which is trivial(!) to decode.
  • urllib3.disable_warnings() Without this set you get a message InsecureRequestWarning: Unverified HTTPS request is being made to host ‘10.1.1.2’. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings. This is because the certificate sent down from the server has not been validated.
  • jar= res.cookies says save the cookies into the jar dictionary, for future use

The output was

duration= 1.210928201675415
output= {“items”:[
{“subsys”:”IZUG”, “active”:true, “dynamic”:true, “funcs”:[10]}
],”JSONversion”:1}

Verifying TLS certificate

With urllib3.disable_warnings() present it will cause error warnings to be suppressed.
When this statement is not present, there will be warnings about certificate problems.

In the statement res = requests.get(geturl,headers=my_header,verify=False,cookies=jar) verify is either “False” or the name of a CA .pem file containing the CA certificates. I used verify=ABC and got

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:

I got the error because “ABC” is not a valid file, and the verification could not be done.

I exported the CA certificate used by the server using

RACDCERT CERTAUTH   EXPORT(LABEL('TEMP-CA')) -         
       DSN('IBMUSER.CERT.TEMP.CA.PEM')   -             
       FORMAT(CERTB64) -                               
       PASSWORD('password')

I could only get verify=…. to work with a USS file, so I had to copy the dataset IBMUSER.CERT.TEMP.CA.PEM into a USS file CACert.pem. Then when I used

res = requests.get(geturl,headers=my_header,verify=CACert.pem,cookies=jar)

it worked fine.

Using a client certificate.

You cannot use RACF certificate with the requests facility, because the underlying code does not support it. You have to use .pem style certificate.

The support does not allow you to specify a password for the private key, so this is not very secure.

You define

cf=”colinpaicesECp256r1.pem”
kf=”colinpaicesECp256r1.key.pem”
cpcert=(cf,kf)

where

  • cf is the name of the file with the certificate in it
  • kf is the name of the file with the private key in it
  • cpcert is the python tuple.

If your certificate file also includes the private key, you do not need the kf, just use cpcert=(cf).

You use it

res = requests.get(geturl,headers=my_header,verify=CACert.pem,cookies=jar,cert=cpcert)

I tried exporting a certificate from z/OS using RACDCERT EXPORT … format(PKCS12B64), copying it to a uss file, and using that, but it did not work. The file could not be read.

I tried creating a private key with a password (to make it more secure) but when I used it I got the message

urllib3.exceptions.SSLError: Client private key is encrypted, password is required

There is a package request_pkcs12 which provides support for a password on the certificate. https://github.com/m-click/requests_pkcs12. I did not use this, I recreated my certificate and private key without a password.

I tried running on Linux using my Hardware Security Module (which plugs into a USB socket). This also failed as I could not enter the PIN for the device.

Compare the response time to running across the network.

I ran the same python script on z/OS and on Linux. The round trip time of the rest request was

  • 1.41 seconds on z/OS
  • 0.92 seconds on Linux.

I think I’ll run my tests from Linux in future.

I cut the CPU cost of doing nothing.

I was running z/OSMF and saw that the CPU costs where high when it was sitting there doing nothing. I managed to reduce the CPU costs by more than half. This would apply to other Liberty based web servers, such as MQWEB, and z/OS Connect.

I could see from the MVS system trace there was a lot of activity creating a thread, and deleting a thread, a lot of costs associated with these activities, such as allocating and freeing storage.

I increased the number of threads so that this allocating a thread and delete a thread activity disappeared.

In the xml configuration file (based from server.xml) was the default

<executor name=”LargeThreadPool” id=”default” coreThreads=”100″
maxThreads=”0″ keepAlive=”60s” stealPolicy=”STRICT”
rejectedWorkPolicy=”CALLER_RUNS” />

I changed this to

<executor name=”LargeThreadPool” id=”default”
coreThreads=”300″ maxThreads=”600″ keepAlive=”60s”
stealPolicy=”STRICT” rejectedWorkPolicy=”CALLER_RUNS” />

and restarted the server.

The options are documented here. There is an option keepAlive which defaults to 60 seconds. If a thread has been idle for this time, the thread is a candidate to be freed to reduce the pool back to corethreads size.

I was alerted to this problem when I looked at an MVS system trace. This is described here.

There is a discussion how sun thread pools work in this post. It is not obvious. This may or may not be how this executor works.

What value should you use?

This is a hard question, as Liberty does not provide this information directly.

I used the Health Checker connects from Eclipse to the JVM and extracts information about the JVM and applications.

This shows that at rest there was a lot of activity. I increased it to 250 threads and restarted the server and got

So better … but still some activity. I increased it to 300 threads, and the graph was flat.

I set up USER.Z24A.PROCLIB(CEEOPT) with

RPTSTG(ON),
RPTOPTS(ON)

in my z/OSMF job I had

//CEEOPTS DD DISP=SHR,DSN=USER.Z24A.PROCLIB(CEEOPT)

This printed out a lot of useful information about the stack and heap usage. It the bottom it said

Largest number of threads concurrently active: 397

The number of threads includes threads from the pool I had specified, plus other threads that z/OSMF creates. The health check showed there were 372 threads, event though coreThreads was set to “300”.

I also used jconsole to display information about the highest thread usage. The URL was service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST. It displays peak threads and live threads.

Security

I found the security of both jconsole, and health check, was weak (userid and password). I was unable to successfully set up a TLS certificate logon to the server.

The information from rptstg was only available at shutdown.

Why does increasing the number of threads reduce the CPU when idle?

The thread pool has logic to remove unused threads and shrink it to the coreThreads size. If the pool size is too small it has to create threads and delete threads according to the load. See here. The keepAlive mentioned at the top is how long a thread can be idle for, before it can be considered a candidate for deletion.

Summary

Monitor the CPU used when idle and see if increasing the threadpool to 300 helps.

Debugging external smart cards and external pkcs11 keystores.

There is an open source package (opensc) which provides access to smart cards and external keystores. It provides some good tools for diagnosing problems.

There is a wiki with good information.

Opensc return codes are here, and the printable text is here

Monitor traffic to and from the device.

You can monitor the traffic to and from the device by using an intermediate “spy” module which displays the traffic.

In your configuration (for example a CCDT), where you specified the name of the module /usr/lib64/pkcs11/opensc-pkcs11.so, replace this with /usr/lib64/pkcs11/pkcs11-spy.so. Specify the environment variable

export PKCS11SPY=/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so

The spy module is invoked, prints out the parameters, and then invokes the module specified in the environment variable.

The output is like

19: C_Login
2021-03-10 14:22:47.947
[in] hSession = 0x21fc030
[in] userType = CKU_USER
[in] pPin[ulPinLen] 00000000021fb2a0 / 8
00000000 5B C7 E7 BB E5 FC 6A BE […..j.
Returned: 160 CKR_PIN_INCORRECT

Detailed internal trace

You can specify the environment variable OPENSC_DEBUG to give a very detailed trace. The higher the number the more detailed the trac.

export OPENSC_DEBUG=9

and use unset OPENSC_DEBUG to reset it.

You can use OPENSC_CONF to specify a configuration file with more parameters, such as file name for the output.

The output from this trace (showing a logon with pin number 12345678) is like

0x7f96e2dca740 14:13:16.756 [opensc-pkcs11] framework-pkcs15.c:1494:pkcs15_login: pkcs15-login: userType 0x1, PIN length 8
0x7f96e2dca740 14:13:16.756 [opensc-pkcs11] pkcs15-pin.c:301:sc_pkcs15_verify_pin: called
….
0x7f96e2dca740 14:13:16.757 [opensc-pkcs11] reader-pcsc.c:283:pcsc_transmit: reader ‘Nitrokey Nitrokey HSM (DENK01051600000 ) 00 00’
0x7f96e2dca740 14:13:16.757 [opensc-pkcs11] reader-pcsc.c:284:pcsc_transmit:
Outgoing APDU (13 bytes):
00 20 00 81 08 31 32 33 34 35 36 37 38 . …12345678

GSKIT return codes

If you are using the MQ C Client interface, this uses GSKIT. There is documentation for the z/OS version, and the return codes are here.

Using the runmqakm commands and an HSM (but not strmqikm).

I tried to use strmqikm but it gave an exception.

You can use some of the runmqakm commands you know and love, to access a certificate with an HSM. For example

The command to list the database available to the runmqakm command,

runmqakm -keydb -list -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

Gives

/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so : UserPIN (mytoken)

You can then use the token label UserPIN (mytoken) and password to use the key store, for example

runmqakm -cert -list all -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678

gives

Certificates found
* default, - personal, ! trusted, # secret key
-	my_key3

and

runmqakm -cert -details -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678
-label my_key3

displays the details of the certificate with label my_key3.

If the -tokenlabel was wrong or the -pw was wrong, I got the unhelpful messages

  • CTGSK3026W The key file “pkcs11” does not exist or cannot be read.
  • CTGSK2137W The label does not exist on the PKCS#11 device.

Create your certificate request

The following command create a new RSA private-public key pair and a PKCS10 certificate request. The documentation for runmqakm says it supports RSA. If you want to use an Elliptic Curve you will need to use an alternative method, for example openssl.

runmqakm -certreq -create -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678
-dn “cn=colin,o=SSS” -file runmq.csr -label runmqlab -size 1024

Sign it

openssl ca -config openssl-ca-user.cnf -policy signing_policy -md sha256 -cert carsa1024.pem -keyfile carsa1024.key.pem -out runmq.pem -in runmq.csr

Store it back into the HSM keystore

I could not get the runmqakm command to receive the signed certificate and store it into the HSM keystore.

runmqakm -cert -receive -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -tokenlabel “UserPIN (mytoken)” -file runmq.pem -pw 12345678

It failed with

CTGSK3034W The certificate request created for the certificate is not in the key database.

I could use

openssl x509 -inform pem -outform der -in runmq.pem -out runmq.der
pkcs11-tool –write-object runmq.der –type cert –label “runmqlab” -l –pin 12345678

The openssl command converts the file from .pem format, to .der format as .der format is required by pkcs11-tool.

Using strmqikm – the theory

If you want to use the strmqikm GUI, you have to configure the java.security file. For example edit /opt/mqm/java/jre64/jre/lib/security/java.security and add the next security.provider in the list.

security.provider.12=com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl /home/colinpaice/mq/nitrokey.cfg

Where /home/colinpaice/mq/nitrokey.cfg is the configuration file, with

name = nitrokey
library = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
slot=0

You can then use Ctrl+O, which brings up a pop up with “Key database type”. In this list should be PKCS11Config, if not check your java.security file. Select this, leave File Name and Location empty, and click “OK”. It pops up “Open Cryptographic Token” with the “Token Label” value taken from the configuration file name = nitrokey. This is strange as the runmqakm command uses a TokenLabel of “UserPIN (mytoken)”.

In practice…

I then got an exception java.lang.RuntimeException: PKCS11KeyStore.java: findSigner(): Failure while executing cobj.getX509Certificate(certFactory, session), and strmqikm ended.

Using a hardware security module USB as a keystore for a browser.

Background to certificates and keystores

When using TLS(SSL) you have two keystores

  • A keystore for holding the public part and private key of your certificate
  • A trust store which holds the public keys of certificate sent to you which you need to authenticate.

Your certificate has two parts

  • The private key which contains information needed to encrypt information you send. This needs to be kept private.
  • The public part,which has information that is needed to decrypt information you have encrypted, along with information such as your Distinguished Dame (DN) such as CN=ColinPaice C=GB,O=StromnessSoftware

The process of creating a signed certificate is

  • Create a private key and public key. This can be done using an external device Hardware Security Module (HSM), such as the Nitrogen HSM USB, or software, for example using OPENSSL. This produces a private key file, and a certificate request file containing the public information.
  • Send the public information to your certificate authority which signs it, and returns it
  • Import the signed public certificate into your keystore.

Creating a certificate using an HSM as the key repository

I used openssl to process my certificates, I’ve discussed the openssl setup here.

I use a bash script because it is easy to parametrize, and makes it easy to rerun until it works. I’ll give the script, then explain what it does

  • enddate=”-enddate 20240130164600Z”
  • name=”hw”
  • rm $name.key.pem
  • rm $name.csr
  • rm $name.pem
  • ca=”carsa1024″
  • pkcs11-tool –keypairgen –key-type rsa:2048 –login –pin 648219 –label “my_key3”
  • OPENSSL_CONF=eccert.config openssl req -new -engine pkcs11 -keyform engine -key label_my_key3 -out $name.csr -sha256 -subj “/C=GB/O=HW/CN=colinpaice” -nodes
  • openssl ca -config openssl-ca-user.cnf -policy signing_policy -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate
  • openssl x509 -inform pem -outform der -in $name.pem -out $name.der
  • pkcs11-tool –write-object $name.der –type cert –label “my_key3” -l –pin 648219

What does the script do ?

enddate=”-enddate 20240130164600Z”

This sets the end date for the certificate – the end date is set when it is signed.

name=”hw”

This is used within the script to ensure the correct files are being used.

Remove old intermediate files
  • rm $name.key.pem
  • rm $name.csr
  • rm $name.pem
ca=”carsa1024″

Define the name of the CA files to use at signing time. The $ca.pem and $ca.key.pem are both needed.

pkcs11-tool –keypairgen –key-type rsa:2048 –login –pin 648219 –label “my_key3”
  • pkcs11-tool use this tool
  • –keypairgen to create a key pair (private and public pair)
  • –key-type rsa:2048 use this key type and key length
  • –login –pin 648219 login with the pin number
  • –label “my_key3” use this label to identify the key
OPENSSL_CONF=eccert.config openssl req -new -engine pkcs11 -keyform engine -key label_my_key3 -out $name.csr -sha256 -subj “/C=GB/O=HW/CN=colinpaice”
  • OPENSSL_CONF=eccert.config this sets up the openssl config file. Having -config eccert.config does not work. See here.
  • openssl
  • req this is to create a certificate requests – create a .csr.
  • -new it is a new request
  • -engine pkcs11 use the named engine, pkcs11, defined to the system
  • -keyform engine this says use the engine (HSM). Other choices are der and pem
  • -key label_my_key3 go to the engine and look for the my_key3 label
  • -out $name.csr create this request file with this name.
  • -sha256 using this signature
  • -subj “/C=GB/O=HW/CN=colinpaice” the name to go in the certificate. It uses colinpaice as the certificate will be used to authenticate with the mq web server, and this is the userid the mq web server should use.

Send the .csr file to the CA for signing (which is the same machine in my case).

openssl ca -config openssl-ca-user.cnf -policy signing_policy -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate
  • openssl ca Use this command to sign the certificate
  • -config openssl-ca-user.cnf use this configuration file
  • -policy signing_policy use this policy within the config file
  • -md sha256 use this for the message digest
  • -cert $ca.pem use the public certificate of the CA
  • -keyfile $ca.key.pem use this private key of the CA to encrypt information about the csr request’s certificate
  • -out $name.pem whee to store the output
  • -in $name.csr the input .csr request
  • $enddate specify the certificate expiry date – set at the top of the script

Send the signed certificate back to the requester.b

openssl x509 -inform pem -outform der -in $name.pem -out $name.der

The pkcs11-tool uses .der files so convert the .pem file to .der format

  • openssl x509
  • -inform pem input format
  • -outform der output format
  • -in $name.pem hw.pem
  • -out $name.der hw.der
pkcs11-tool –write-object $name.der –type cert –label “my_key3” -l –pin 648219

Read the signed certificate and write it to the HSM

  • pkcs11-tool
  • –write-object $name.der write onto the HSM the file hw.der coverted above
  • –type cert import type (cert|pubkey|privkey)
  • –label “my_key3” use this name
  • -l –pin 648219 and logon with this pin number

Define the HSM to Chrome browser

Stop the browser because you need to update the keystore.
The command was issued in the home directory, because key store is in the home directory/.pki .

modutil -dbdir sql:.pki/nssdb/ -add “my_HSM” -libfile opensc-pkcs11.so

  • modutil use this command
  • -dbdir sql:.pki/nssdb/ to up date this keystore (in ~)
  • -add “my_HSM” give it this name
  • -libfile opensc-pkcs11.so and use this file to communicate to it

Display the contents of the browser’s keystore

modutil -dbdir sql:.pki/nssdb/ -list

This gave me

Listing of PKCS #11 Modules
 NSS Internal PKCS #11 Module
...
 Mozilla Root Certs
 library name: /usr/lib/x86_64-linux-gnu/nss/libnssckbi.so
...
my_HSM
 library name: opensc-pkcs11.so
    uri: pkcs11:library-manufacturer=OpenSC%20Project;library-description=OpenSC%20smartcard%20framework;library-version=0.17
  slots: 1 slot attached
 status: loaded
 slot: Nitrokey Nitrokey HSM (DENK01051600000         ) 00 00
 token: UserPIN (SmartCard-HSM)
   uri: pkcs11:token=UserPIN%20(SmartCard-HSM);manufacturer=www.CardContact.de;serial=DENK0105160;model=PKCS%2315%20emulated

Restart the browser.

Use an URL which needs a certificate for authentication.

The browser prompts for the pin number (twice), and displays the list of valid certificate CNs. Pick one. When I connected to the mqweb server, I had 3 certificates displayed. I had to remember which one I wanted from the Issuer’s CN and serial number. For example

SubjectIssuerSerial
colinpaiceSSCARSA1024019c
ibmsys1SSCARSA1024019a
170594SSCARSA10240197
Select a certificate

(Having a CA just for HSM keys, such as SSSCAHSM would make it more obvious.)

Setting up digital certificates for identification in your enterprise.

You can use digital certificate for authentication, for example you can logon onto the MQ Web server using a certificate to identify you, and you do not have to enter a userid or password.

Many systems have Multi Factor Authentication (MFA) to logon which usually means you authenticate with something you have, and with something you know. Something you have is the private certificate, something you know is userid and password.

At the bottom I discuss having an external device for your keystore to make your keystore more secure.

General background and information

  • Your certificate has a private key (which should not leave your machine), and a public part, which anyone can have.
  • You can have a key store which has your private key in it. This is often just a file which could be copied to another machine. This is not a very secure way of keeping your certificates, as there is usually a stash file with the password in it, which could easily be copied along with the keystore.
  • You have a trust store which contains the public part of the certificates you want to validate (demonstrate trust) with. This is usually a set of Certificate Authority public keys, and any self signed certificates. The information in these certificates is commonly available and can be world read. You will want to protect this for write, so people cannot insert CAs from the bad guys.
  • You can use Hardware Security Module, a piece of hardware which can store your private keys, and does encryption for you. This is a secure way of keeping your certificates. You need physical access to the machine to be able to physically access the HSM hardware.
  • Certificates are based on trust. When I create a public certificate, I can get this signed by a Certificate Authority. When I send my public certificate to you, and you have the same Certificate Authority, you can check what I sent you using the Certificate Authority. My public certificate give information on how to decrypt stuff I send you.
  • When a connection is made between a client and a server. The server sends down its certificate for the client to validate and accept, and the client can then send up a certificate for the server to validate and accept. This is known as the handshake
  • A certificate has a Distinguished Name. This is like “CN=COLIN,OU=TEST,O=SSS.ORG” so my Common Name is COLIN, The Organizational Unit is TEST, and my Organization is SSS.ORG.
    • Some products like the mid-range MQ Web Server map the CN to a userid.
    • As part of the logon a client or server can check the certificate sent to it, for example allow any certificate with OU=TEST, and O=SSS.ORG.

Planning for TLS and certificate

Consider a simple scenario of two MQ Servers, and people from my.org and your.org want to work with MQ. Leaving aside the task of creating the certificate, you need to decide

  • What name hierarchy you want, for example CN=”COLIN PAICE”, OU=TEST, C=GB, O=SSS.ORG,
    • do you want to have a CN with a name in it, or a userid, or a personnel number. This is used by the MQWeb as a userid. You could have CN=MQPROD1, etc to give each server its own CN.
    • Do you want to have the country code in it C=GB? What happens if someone moves country. You might decide to have servers with CN=MQPROD1,OU=PROD… or OU=TEST… .
  • What CA hierarchy do you want. You could have a CA for OU=PROD, O=SSS.ORG at the PROD level, or CN=CA,O=SSS.ORG at the organisation level. Some servers can check the issuer is OU=PROD, O=SSS.ORG and so only allow certificates signed by that CA. Someone connecting with a certificate signed with OU=TEST,O=SSS.ORG would not be allowed access.
  • You could give each server the same DN, for example CN=MQSERVER,OU=PROD,O=SSS.ORG, or individual ones CN=MQSERVER1,OU=PROD,O=SSS.ORG
  • You can have a server check that a certificate is still valid by using Online Certificate Status Protocol (OCSP). After the handshake, a request goes to a remote server asking if the certificate is still valid. Ive written a blog post Are my digital certificates still valid and are they slowing down my channel start? z/OS does not support OCSP. MQ on z/OS supports a LDAP repository of Certificate Revocation Lists. If you intend to use OCSP you need to set up the OCSP infrastructure.
  • With the MQ mover, you can set up CHLAUTH records to allow or disallow DN’s or CA certificates.
  • The clients from my.org have a DN like CN=COLIN,OU=TEST,O=myorg.com. The clients from your.org have a DN like CN=170594,c=GB,o=your.org. You cannot have one string (SSLPEER) to allow both format certificates.
    • For connections to the chinit(mover) you can use CHLAUTH to give find grained control.
    • For the MQWeb on z/OS you can control which certificates (or Issuers) map to a userid.
    • For mid-range MQWEB you have no control beyond a successful handshake. CN=COLIN,o=MY.ORG, and CN=COLIN,o=YOUR.ORG would both map to userid COLIN even though they are from different organisations. The CN is used as a userid, and you map userids or groups to security profiles.

Setting up your certificates

As your private key should not leave your machine, the standard way of generating a certificate is

  • The client machine creates a certificate request. This has the public certificate, and the private key.
  • The public certificate is sent to the appropriate authority (a department in your organization) which signs the certificate. Signing the certificate consists of doing a check sum of the public certificate, encrypting the check sum value, and packaging the public certificate, the encrypted checksum, and the CA public certificate into one file. This file is sent back to the requester
  • The originator reads the package stores it in a keystore, and uses this as its public key.
  • Often this request for a certificate is allowed only when the machine is connected locally to the network, rather than over the internet. This means people need to bring their portable machines into the office to renew a certificate.

If you create the private certificate centrally and email it to the end user, someone who is snooping on the email will get a copy of it!

A machine can have more than one keystore and a keystore can have one or more certificates. With some servers you can configure the default certificate to use. If not they the “best” certificate is chosen. This could depend on the strength and selection of the cipher specs.

What if’s

Once you have set up your certificate strategy it is difficult to change it, so it is worth setting up a prototype to make sure the end to end solutions work, then throwing the prototype away and starting again.

You need to consider how to solve problems like

  • What if someone leaves my organisation, how do I inactivate the certificate
  • What happens of someone loses their laptop, how do I inactivate the certificate
  • Certificates have expiry dates. What do I need to do to renew the certificate before it expires – for example you could email the owner and tell them to bring the laptop to the office to renew the certificate
  • What happens if a CA expires?
  • Someone joins the department how do I update the access lists. Usually this is done using a repository like LDAP.
  • Are the CHLAUTH records restrictive enough to prevent the wrong people from getting access, but broad enough that you do not need to change them when someone joins the organisation.
  • What if you open up your business to a new organisation with a different standard of DN? What do you need to change to support it.

Use of physical keystores.

You can have a physical keystore to store your private key. This can range from a USB device up to integrated devices.
With these people cannot just copy the keystore and stash file, they need physical access to the device.

You need to plan how these will be used in your organisation for example you have two machines for HA reasons. Each has a USB store. Does each machine need its own private key? How do you handle disaster recovery when someone loses/breaks the keystore.

Physical keystores can have have a secure export and import capability. You configure a key onto the device, for example saying it needs 3 partial keys, needing three people to enter their portion of it. When you export the key, it comes out encrypted.

In this scenario the configuration process could be

  • Configure the first device. 3 people enter their password.
  • Create a private key
  • Export the private key and send it to the second machine. It is encrypted so can safely be sent.
  • Go to the second machine, and configure the second device.
  • As before, the three people have to configure the device.
  • Import the encrypted certificate to the device
  • Go the the next machine etc.
  • In some cases you can say that n out of m people are required to configure the device. So any 3 out of a team of 6 is enough.