Using jConsole and with z/OS Liberty web server

I wanted to get some monitoring information out from z/OSMF using jConsole on my Ubuntu machine. Eventually this worked, but I had a few problems on the way. The same technique can be used for base Liberty, MQWeb, z/OSMF and ZOWE all of which are based on Liberty.

Configuring z/OSMF

I changed the z/OSMF configuration to include

<featureManager> 
  <feature>restConnector-2.0</feature> 
</featureManager> 

and restarted the server.

In the stdout (or message log) will be something like

CWWKX0103I: The JMX REST connector is running and is available at the following service
URL: service:jmx:rest://sss.com:10443/IBMJMXConnectorREST

You need the URL. The message above gave service:jmx:rest://sss.com:10443/IBMJMXConnectorREST, but I needed to use service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST .

The port number came from the httpEndpoint with id=”defaultHttpEndpoint” . I have another httpEndpoint with port 24993, and this also worked with jConsole.

Set up jConsole

I set up a script for jConsole

k1='-J-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/adcdc.p12'
k2='-J-Djavax.net.ssl.keyStorePassword=password'
k3='-J-Djavax.net.ssl.keyStoreType=pkcs12'
t1='-J-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/zca.jks'
t2='-J-Djavax.net.ssl.trustStorePassword=password'
t3='-J-Djavax.net.ssl.trustStoreType=jks'
d='-J-Djavax.net.debug=ssl:handshake'
d=' '
de='-debug'
de=' '
s='service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST'
jconsole $de $s $k1 $k2 $k3 $t1 $t2 $t3 $d

Where

  • the -J .. parameters are passed through to java,
  • the -Djava… are the standard set of parameters to define the key stores on the Linux

Running this script gave a pop up window with

Secure connection failed. Retry insecurely?

The connection to service:jmxLret://10.1.1.2:10443/IBMJMXConnectorREST could not be made using SSL.

Would you like to try without SSL?

This was because of the exception

java.io.IOException: jmx.remote.credentials not provided.

I could not see how to pass userid and password to jConsole.

I then used Cntrl+N to create a new connection and entered Username: and Password: which jConsole requires. After a short delay of a few seconds jConsole responded with a graphs of Heap Memory Usage, and Threads in use. You can then select from the Measurement Beans.

The TLS setup

In the keystore I had a certificate which I had used to talk to a Liberty instance before.

This was signed, and the CA certificate had been imported into the key trust keyring on z/OS, for that HttpEndPoint.

The server responded with a server certificate (“CN=SERVER,O=SSS,C=GB”) which had been signed on z/OS. The signing certificate had been exported from z/OS and downloaded to Linux

I created a jks key trust store using this certificate, using the command

keytool -importcert -file temp4ca.pem -keystore zca.jks -storetype jks -storepass password

and used this trust store to validate the server certificate sent down from z/OS.

This worked with jConsole.

I created a pkcs12 keystore using keytool

keytool -importcert -file temp4ca.pem -keystore zca2.p12 -storetype pkcs12 -storepass password

Which also worked.

Problems using a .p12 trust store

I used

runmqakm -keydb -create -db zca.p12 -type pkcs12 -pw password
runmqakm -cert -add -file temp4ca.pem -db zca.p12 -type pkcs12 -pw password -label tempca
runmqakm -cert -details -db zca.p12 -type pkcs12 -pw password -label tempca

to create a pkcs12 keystore and import the z/OS CA certificate. The -details option displayed it.

When I tried to use it, jConsole produced the message (after the Cntl+N)

Secure connection failed. Retry insecurely?

The connection to service:jmxLret://10.1.1.2:10443/IBMJMXConnectorREST could not be made using SSL.

Would you like to try without SSL?

I used Ctrl-N as before, and got the same message.

Using

d=’-J-Djavax.net.debug=ssl:handshake’

and rerunning the script, produced a TLS trace. At the bottom was

VMPanel.connect, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

%% Invalidated:…
VMPanel.connect, SEND TLSv1.2 ALERT: fatal, description = internal_error

Using a trace at the server, gave the unhelpful , SEND TLSv1.2 ALERT:

Using openssl also failed. Create the .p12 keystore

openssl pkcs12 -export -out zca.p12 -in temp4ca.pem -name zCA -nokeys

and rerun the jconsole script, and it failed the same way.

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

It looks like runmqakm and openssl do not create a valid trust store with an imported certificate.

Additional diagnostics

When the trust store created by keytool was used; at the top of the TLS trace output was

System property jdk.tls.client.cipherSuites is set to ‘null’

Ignoring disabled cipher suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
trustStore is: /home/colinpaice/ssl/ssl2/zca.p12
trustStore type is : pkcs12

trustStore provider is :
init truststore
adding as trusted cert:
Subject: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Issuer: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Algorithm: RSA; Serial number: 0x0
Valid from Tue Jul 14 00:00:00 BST 2020 until Fri Jul 02 23:59:59 BST 2021

keyStore is : /home/colinpaice/ssl/ssl2/adcdc.p12
keyStore type is : pkcs12
keyStore provider is :
init keystore
init keymanager of type SunX509

When the runmqakm or openssl was used, the green entries were missing.

When I used runmqakm to create the pkcs12 keystore

runmqakm -cert -details -db zca.p12 -type p12 -pw password -label tempca

listed the certificate successfully.

When I used keytool to list the contents

keytool -list -keystore zca.p12 -storetype pkcs12 -storepass password
Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 0 entries

When I created the key store with keytool, both runmqakm and keytool displayed the certificate.

The problem looks like Java is only able to process the imported CA certificates when keytool was used to create the trust store.

When the burglar alarm goes off – would you just mute it?

I’m sure you’ve all been told to ignore links like

You have wun $1,000,000 click hear to download a dodgy application to steal your money” or

We have been unable to deliver a parcel to you. Click here to rearrange delivery and pay the excess money” but how about
Your connection is not private, Net::err_cert_invalid – click here to accept it

Collect 10 points if you did not fall for any of these.

I would like to talk about the last one, about your connection is not private; because I saw a web site recommending “click here to accept”.

You get messages like this if there are problems with the certificate. Common problems are

  • Your browser does not recognise the certificate, because it has not been signed by any CA in your browser’s trust store. This looks like someone is sending you an invalid certificate. It may be your enterprise security team need to update all browsers with the missing certificate, but they should have spotted this problem not you.
  • The certificate sent from the server has “valid host addresses are…. 10.1.1.2, my sys.com” and this has been sent from a different site. This could be because bad guys have intercepted your request and played “Man in the middle” between you and the server.
  • Less common, is where the certificate has parameters which your browser cannot handle. For example very old cipher specs, and your browser only supports TLS 1.2 or above.
  • It could be caused by a server using a self signed certificate. It may be acceptable to use this in a test environment, but it is not secure – as anyone could have created it. If it is signed, then you can check the digital signature.

All of these should set off the intruder alarm.

If you get a message like this – do not click on the accept button. This is just like muting the burglar alarm, and ignoring the warning – bad guys could steal your personal details – and you would not know.

Many web servers are configured with a bad certificate, for example they may specify an alternate name of 127.0.0.1, but not “localhost”. It is much easier (though wrong) to tell people to accept the certificate, than to fix the certificate.

Do not trust it

The blog post went on to say that you can download the certificate which has been sent to you and install it in your key store. I felt like screaming “NO!” and attacking the screen with a hammer!

Imaging, twenty years ago, you were a bank manager, and a man comes in, and says “I am a policeman, and we are doing a survey about your bank’s safe and where you keep the money. I would like my colleague “fingers” to go down inspect it. He has to work anonymously, which is why he is wearing a mask, and gloves, and he is wearing a black and white striped shirt. I’ll come back tomorrow”. You let “fingers” go down to the safe on his own, and say goodbye to the nice policeman. An hour later you find your safe is empty and there is no sign of the two men.

Your first mistake was to accept the policeman was a policeman. You should have checked his credentials before doing anything else. If he was fake, then any identification papers would be fake. You need to talk to your contacts in the local police station, and check if your visitor is valid or bogus (while he is sitting in your office).

Only in rare situations, such as a test environment, would you extract the certificate from what was sent to you, and install it in your key store. If you install it, then later, another request can come in from the bad guys, and be validated by the CA you just installed.

If you need a CA, you go through the proper corporate process. For example as part of the overnight maintenance on your machine, or download it securely from a known site. Emailing may not be good enough if your machine has been hacked.

So, in summary, today’s lesson is

  1. Do not “accept” bad certificates when using web site. Fix the the server.
  2. Get any additional certificates through an approved process – do not accept what the server is giving you.

In the words of the Police Constable in Dixon of Dock Green. “Evening all – mind how you go”.

Overview and challenges of using end to end message encryption (AMS)

MQ provides end to end protection of messages. This covers

  • Integrity – where the message can still be read – but any changes to the message are detected. You can check the id of the person that sent it.
  • Encryption – where the payload is encrypted and only the ids with the correct private key can decrypt it.
  • Privacy – A combination of Integrity and Encryption, where only specified ids can decrypt it, and you can check the id of the person that signed it.

The messages are protected on the network, in memory and on disk. You do not need to use a channel with TLS for protection (but you may want to use TLS for authentication), as the data is protected before sending.

Any headers that are involved in ‘routing’ like MQMD, XQH, RFH2, etc are skipped by AMS because the queue manager still needs to know what to do with the messages in terms of delivery and selection before they are unprotected. Other message headers like PCF command headers and custom application defined headers/formats can still be protected. Sensitive data should not be held in message properties as that data is used in routing/selection. Thanks to Jon Rumsey for the correction to this.s

Background about encryption and signing of data

You may need to read some of the following section more than once, because it reference private keys and public keys and it is hard to work out what is happening.

The encryption techniques used are special. There is a private key and a public key. (Think of your passport as a private key, and your name and address is your public key). You can use functions

  • You do encrypt(private key, payload) to produce an encrypted payload, send it to me, and then I use decrypt(public key, encrypted payload) to recreate the original payload. I know it came from you, because I needed your public key to decrypt it.
  • I do encrypt(public key, payload) to produce an “encrypted” payload, and send it to you. Only you can use decrypt(private key, encrypted payload) with your private key to recreate the original payload.

With this

  • I can encrypt data using your public key that only someone with access to the private key (you) can decrypt
  • I encrypt with my private key, and anyone with my public key can decrypt it – and you know it came from me – because you needed my public key.

Signing is simply taking a checksum of the data, and encrypting the value with the signer’s key. When you come to check the signature, you do the same check sum calculation, decrypt the “signed” value using my public key, and the values should match. The payload would look something like “Signed by CN=Colin,O=SSS,C=GB” encrypted checksum=2294567, my certificate; followed by the original data.

We can take this further. I sign the data with my private key, and then encrypt the data(including the signed data) with your public key, (keep up at the back there – I said it was hard to follow). When you process it, only you can decrypt it because it needs your private key, and you can check the signed data, and see it came from me. I have sent the message encrypted, and have mutual authentication.

With encryption a symmetric key is used to encrypt the data, because it is faster, and about as secure as encryption with an asymmetric key. When trying to break into enciphered data, the more data you have, the easier it is to break. This is why you can reuse the symmetric key for multiple records, but you should change it, for example after 10 MB has been processed (I don’t know the best figure – it is all a matter of risk).

Sending a message, many to many.

Sending a message from one person to another person looks fairly easy. It gets a bit harder when you have potentially many people in my organisation who are allowed to send a protected message to your organisation, and there may be many people who are allowed to get, and decrypt it.

Let the people in my organisation be CN=A,O=MYORG, and CN=B, O=MYORG, and the people in your organisation be CN=X,O=YOURORG, CN=Z,O=YOURORG.

If person CN=A wants to send an encrypted message so it can be decrypted by CN=X or CN=Z the processing is as follows.

  • Do a checksum calculation on the record.
  • Encrypt this checksum using CN=A’s private certificate.
  • Put a new field of CN=A,O=MYORG|| encrypted checksum value, CN=A’s public certificate before the record.
  • Create a symmetric key.
  • Encrypt the (signed data and original payload) with the symmetric key.
  • Take the symmetric key and encrypt it for CN=X,O=YOURORG using CN=X’s public certificate. Create a field of the Distinguished Name || encrypted key.
  • Take the symmetric key and encrypt it for CN=Z,O=YOURORG using CN=Z’s public certificate. Create a field of the Distinguished Name || encrypted key.
  • Build a record of the two “encryption fields” and the encrypted data
  • Put the data in a message and send it to the recipient(s).

Getting an encrypted message

The person with CN=X gets the message.

  • Scan down the message looking for CN=X,O=YOURORG. If found then extract the encrypted symmetric key. Decrypt this value with the CN=X’s private key. This gives you the key that was used to encrypt the customer data.
  • Decrypt the message using the key you just decrypted.
  • Extract the first field and see it is signed data. We can see that the checksum was done by CN=A,O=MYORG.
  • Check this name with your list of IDs that can send you messages.
  • Validate the signer’s certificate. Either use a CA, or if the certificate was self signed, check with the copy in your trust store. Note this certificate includes the public key.
  • Take the checksum value in the message and decrypt it using CN=A,O=MYORG’s public key from the payload. Do the checksum of the remainder of the message and compare the values – they should match.

Setting up your enterprise

You need to know the list of potential recipients of messages for each queue.

At the receiving end, if you want to check the message came from an authorised id, you need to know the list of potential senders.

If you’ve managed to keep up so far – it gets worse!

The management of these lists can be difficult. If someone joins my team, I would need to tell the other organisations that they need to add CN=NEWPERSON, O=MYORG into their list of potential senders, (and remove CN=IVE_RETIRED, O=MYORG. It will take a week to get this change through their change management system. Until this is done, the message could be rejected as coming from an unauthorised person.

In the same way, you need to update the AMS setmqspl definitions to add CN=NEWPERSON, O=MYORG to each queue, and when people join or leave the other enterprises you’ll need to update your list of potential senders.

Do not worry too much – there is a solution, see below.

Creating the policies

With MQ, the security policy is defined using the SETMQ Security PoLicy command (setmqspl) for example to define a queue at MYORG to send message to YOURORG. The -a option is for who the message is for

setmqspl -m PRODA -p SENDQUEUE -s SHA256 -e AES128
-a "CN=X, O=YOOURORG"
-a "CN=Y, O=YOOURORG"
-a "CN=Z, O=YOOURORG"

At YOURORG for the GETQUEUE you need the following. The -r option is who you can receive a message from

setmqspl -m YOQM -p GETQUEUE 
-r "CN=A, O=MYORG"
-r "CN=B, O=MYORG"
-r "CN=JO, C=GB,O=SOMEOTHERORG"

To change an entry you have to run whole command, you cannot add one “-a” or “-r”. I would keep a file for these, update file, and process the whole file. For two queues – use two files.

You also have to consider how to update the definitions for multiple queue managers. On z/OS I would have the MQ commands inline, and use a variable queue manager name &QM, which you set in the JCL. For midrange you could have a shell script and pass the queue manager name as a parameter.

What keys do I need?

The encryption is done using RSA, and so needs a certificate with an RSA private key.

When the message is signed

Each id putting a message needs access to a private key – this could be one key shared by multiple users.

When getting a message that has been signed, the recipients will need to verify the certificate sent within the signing data; either a CA in the signing chain, or the public certificate (for self signed certificates).

When encrypting the message

The id putting the message needs access to the public key of each recipients in the list.

The id getting a message which has been encrypted, needs access to the private key.

Maintaining the public keys

For each person in the -r or -a list you will need the digital certificate, and public key in a keystore or a keyring (on z/OS).

Each id on z/OS will need their own keyring. There is a keyring for the queue manager;s AMS address space with the public certificates on it.

On midrange MQ, each user has a keystore, with a combination of public keys, and private key(s).

Time to step back and think about what you want.

To me, it is clear that having every possible user having their own DN, and certificate quickly gets impractical. If you are working with people outside of your organisation do you want them to know the people’s names within your organisation?

You may want to have a departmental certificate for example CN=PAYROLL,O=MYORG, which members of your payroll team all use. You can control access to the queue using standard MQ security policies.

An intermediate configuration would be to have a proxy or server applications to change input messages from “partner protection” to “myorg protection”, and the reply messages changed from “myorg protection” to “partner protection”

  • You have an AMS protected input queue called EXTERNAL_IN. This has been configured to specify the DNs of the systems that can send you messages, so might specify CN=RECEIPTSPAYABLE,O=YOURORG as a valid sender.
  • If the get is successful, then put to a queue called INTERNAL_IN, specifying the DNs of the individuals in your Accounts department who are allowed to get one of the protected messages. If someone joins the Accounts department, you only need to change the configuration within your queue manager.
  • Someone (or an automated task) gets the message from INTERNAL_IN queue, it is decrypted automatically, the application does the processing and puts a message the AMS protected queue YOUR_BANK_INTERNAL, where it is encrypted by AMS.
  • Another proxy/server application in your enterprise gets the messages from YOUR_BANK_INTERNAL, checks the DN of the sender, and that they were authorised to put messages to the queue.
  • The server then puts to YOUR_BANK_EXTERNAL, using a DN of “CN=PAYROLL,O=MYORG”, and it is encrypted for the recipient.

This way you can isolate internal processing and external processing. Updating the internal list is within your control. You should not need to update the external lists (and certificates) once established.

Handling whoops’s

Some problems will result in the messages removed from the queue and put to a special error queue. You need to have a process in place to handle this. I had messages put to this queue when a userid tried to get an encrypted message, but the message had not been encrypted for this userid’s DN.

Other AMS blog posts

pkcs11? pkcs12? .cms? .jks? .p12? .nss? which type of keystore should I use?

There are at least four types of keystore, and after an email exchange with someone, it became clear that the different types of keystore and how to use them, are not widely understood. It is one of those topics, that once you are an expert, this topic is obvious still difficult to understand. When you have little knowledge and just want to get a job done, it can be very confusing.

I’ve learned a lot from writing this post

  • There is no one tool to manage all your keys and certificate
  • You may need one keystore for your queue manager, another keystore for your C clients, another keystore for your Java clients, a keystore for a web browser and something else for curl!
  • You might need to use 3 tools to manage your keystores!

This post has sections on

This post started off as a few lines, then I though I had better explain the pre-req knowledge, and then the pre-req knowledge to the the pre-req knowlegde. I hope it provides a clear story.

What’s the difference between a trust store and a key store – when they are both called key stores?

You need a store to contain the certificates needed to check any certificates sent to the application. These are usually certificate authority certificates, but can also be self signed certificates. This store is known as a trust store – it is used to check that a certificate is trusted. None of this data is confidential, most of the information is in the public domain. One trust store could be shared by all applications, which makes the maintenance of it much easier than a trust store for each application/server.

You need a store to keep the private key used for encryption/decryption, this is known as the key store. This needs to be kept confidential. The keystore may have just one private key. You can use smart keys, or external keystores such as special USB devices.

Some products use just one store with the trust store and the private certificates combined. They also call this the keystore. If you want to isolate the keystores, you need multiple stores. This, in turn, means that if you update one trust certificate, you have multiple stores to update.

What does a store contain?

A keystore can contain

  • private information – such as you secret key which you use for encryption and decryption
  • public information, such as your Distinguished Name (CN=colin,O=SSS), and they key needed to encrypt data for me
  • trust data, if you send me a public key – can I validate it has not been tampered with?

What format is this data in?

This data is typically in one of three formats

  • Plain text
  • Binary
  • Binary portable (64 bit encoding)

Plain text

If you print a certificate or key, it comes out like

Certificate:
  Data:
    Version: 3 (0x2)
      Serial Number: 379 (0x17b)
      Signature Algorithm: ecdsa-with-SHA256
      Issuer: C=GB, O=SSS, OU=CA, CN=SSCA256
      Validity
        Not Before: Feb  9 09:35:07 2021 GMT
        Not After : Apr  9 17:40:01 2022 GMT
        Subject: C=GB, O=cpwebuser, CN=ecec
        Subject Public Key Info:
          Public Key Algorithm: id-ecPublicKey
            Public-Key: (256 bit)
            pub:
               04:5d:15:78:83:e0:ab:af:78:69:2f:14:51:fe:29:
            ...
                ASN1 OID: prime256v1
                NIST CURVE: P-256
...

Binary

The data is encoded using Abstract Syntax Notation(asn). For example you get a string of fields like

  • The following fields are a sequence
  • of length 24
  • it is a string format
  • the first item has length 8
  • the attribute type is 06 03 55 04 06 which means Country
  • the attribute value is GB
  • the second item is …

This would describe the C=GB in C=GB, O=SSS, OU=CA, CN=SSCA256.

Binary portable (base 64 encoding)

With the binary format, the data is a hex string, which is not very portable. For example if you FTP it to a remote site, you may get data conversion, and new lines changed. Often the data is converted to an intermediate form called base 64 encoding. Every 3 hex characters are converted to 4 “printable” characters A-Z,a-z,0-9 and some special characters.

The file looks like

-----BEGIN CERTIFICATE-----
MIICDjCCAbSgAwIBAgICAXswCgYIKoZIzj0EAwIwOjELMAkGA1UEBhMCR0IxDDAK
BgNVBAoMA1NTUzELMAkGA1UECwwCQ0ExEDAOBgNVBAMMB1NTQ0EyNTYwHhcNMjEw
...
hEzE3hSzUvbfLUKnua64AiEAnyKhrmBjXCVQ834VfjhNHYqjaN1PzXBE8Cv3jlWJ
9bA=
-----END CERTIFICATE-----

What are the keystore types?

When Secure Sockets Layer was being developed each major player came up with their own format for storing the certificate and key information.

  • Java had files with type .jks (java key store) for example keystore.jks
  • IBM had files with type cms (certificate management system) with file types like zzserver.crl, zzserver.kdb, zzserver.rdb, zzserver.sth. It also uses intermediate files such as .arm, which has base 64 encoding of a certifcate, as shown above.
  • Netscape had files in a NSS database, for example files in the directory /home/colinpaice/.pki/nssdb, are files cert9.db and key4.db. key4 is the keystore database (storing keys), and cert9 is the trust store database (storing certificates).
  • OpenSSL developed .p12 files which can contain certificates and keys. It also has .pem (base 64 encoded) and .der (binary) files for individual certificate, private key, and public key files .
  • Windows has .pfx files.
  • Smart cards where the keystore is on a special USB type device or other external Hardware Security Module, each have their own format keystore.

Standards were developed to work with keystores

When can different keystore types be used?

With C programs on mid range and z/OS, IBM products use GSKIT from IBM. On z/OS you can store them in the z/OS security manager (for example RACF) or on the Hardware Security Module in the processor.

Java program can use most types of keystore. You may need to configure the java.security configuration file with an entry like

  • security.provider.X=com.ibm.security.cmskeystore.CMSProvider
  • security.provider.Y=com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl /home/colinpaice/mq/nitrokey.cfg

The first line includes support for cms files, the second includes support for a pkcs11 external keystore on an HSM (including its configuration file).

For Java programs, you configure the keystore using start-up options. The following example defines the keystore as an external USB HSM keystore, and the trust store is a openssl .p12 store.

-Djavax.net.ssl.keyStore=NONE
-Djavax.net.ssl.keyStoreType=pkcs11
-Djavax.net.ssl.keyStorePassword=12345678

-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/mytrust.p12
-Djavax.net.ssl.trustStorePassword=password
-Djavax.net.ssl.trustStoreType=pkcs12

These definitions along with the java.security, and its override, show the keystore is a pkcs11, and there is an entry in the java.security (above) pointing to the configuration file /home/colinpaice/mq/nitrokey.cfg for the pkcs11 definition. You can override or extend the options in the java.security file using configuration using

-Djava.security.properties=/home/colinpaice/mq/colin.java.properties

Do these keystores share information?

No, if you have a keystore (or a USB HSM) it shares no information with any other keystore. For example you could unplug the USB keystore, take it to another machine and plug it in, and the certificate etc will be available. You could send a .jks file, to yet another machine, and it could be used.

What can you use to administer the keystore.

IBM provided

You can use GSKCapiCmd_64 from IBM GSKIT to manage certificates and keys.

MQ has runmqakm as a command line tool which invokes GSKCapiCMD_64 under the covers.

You can also use the MQ command strmqikm to have a GUI to manage your keys and certificates. strmqikm just invokes the GSKIT ikeyman program.

These solutions feel a little dated, as I could not find if they support modern certificates capabilities; OCSP, and elliptic curves. They had a few bugs as well.

I’ve used openssl to create the “modern” private key and certificates, then imported them into the keystores using the IBM tools.

Oracle provided

With Java there is keytool. This supports a variety of keystores (depending on what has been configured in its java.security file), including

  • Java Key Stores .jks. This keystore can only be processed by java
  • Java Cryptography Extension KeyStore – a stronger version of .jks. This keystore can only be processed by java
  • pkcs12 .p12 and .pfx. These keystores can be used by Java, C and other languages, with the right API. For example Curl uses a C api to access the keystores.
  • pkcs11 – keystores on smart devices
  • nss – netscape security.

It does not support cms format keystores used by GSKIT.

Openssl

This is open source and is active, keeping up with the trends in security. Openssl deals with the building blocks; private keys, certificates etc but does not handle keystores very well. You can create manage certificates, but you may need other tools to put them into keystores.

opensc for managing smart keys and other pkcs#11 devices

(Opensc is open Smart Card.) If you have a smart key, or external keystore on an HSM, there may be hardware specific libraries for accessing the keystore, or the open source code may be supported. This can provide drivers for other tools, such as the java tools. You can also use pkcs11-tool to directly administer keys and certificates on an HSM device.

Other tools

  • pk12util from Netscape is used to mange keys in the NSS keystore databases
  • certutil from Netscape is used to manage keys and certificate in the NSS database.
  • Firefox and Chrome browsers can be used to update the NSS keystore used by the browsers.

What tool would I use when?

Creating the private key

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool -keypairgen
  • To generate an elliptic curve, use openssl ecparam
  • To generate an RSA, use openssl genpkey
  • Use keytool keypairgen to generate an elliptic or RSA key
  • runmqakm and strmqikm(ikeyman) do not seem to support elliptic curves, but support RSA keys as part of a certificate request (see below)

Create a certificate request

The certificate request takes (or creates) the public key, creates a certificate with the DN ( eg CN=Colin,C=GB,O=SSS and creates the certificate request file.

  • openssl req takes a file from the keypairgen
  • keytool can create a certificate request, I could not see how to use the keypair gen private key as part of this
  • runmqakm and strmqikm (ikeyman) can create a request, but I do not think it supports all flavours of private key.

Sign a request

I expect most people will want a tool which you can run as a script

  • openssl ca – you can include the optional attributes, with this.
  • keytool – I could not see how to use this to sign a certificate. The documentation suggest using openssl x509.
  • runmqakm – you can use runmqakm -cert -sign .
  • strmqikm (ikeyman) I could not see how to sign a request using this, and it is a GUI.

Receive the signed certificate into the keystore

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool –write-object
  • Use openssl pkcs12 to create a pkcs12 (.p12) keystore using the private key and signed certificate
  • Use keytool -importcert. This can support most keystore types, depending on the configuration in the java.security file.
  • runmqakm and strmqikm (ikeyman) import ( receive) the certificate and store it in the keystore.

Update the browser’s keystore

The browsers have an nss format store.

  • pk12util -i …imports a .p12 keystores into the nss “sql:” keystore. The sql: keystore is a Netscape internal format store. (you use certutil … to remove entries from the nss keystore).
  • You can use the browser’s facilities to import certificates and keys into the browsers (nss) keystore.

Receiving a public certificate into a keystore.

If you have a Certificate Authority or a self signed certificate you want to put into your trust store.

  • When using a smart device, for example an external keystore on an USB, use the device driver or pkcs11-tool –write-object
  • Openssl is not very good at adding new entries to an existing .p12 file
  • runmqakm -cert -add -file ca256.pem…
  • strmqikm, select Signer Certificates, then select Add
  • Use keytool -importkeystore.  To import a .p12 store, or a .pem file I have a shell script

ks=” -destkeystore mytrust.p12″
dest=”-deststoretype pkcs12 -deststorepass password”
src=”-srcstoretype PKCS12 -srcstorepass password”
keytool -importkeystore $ks $dest -srckeystore ca1024.p12 $src
or
keytool -import $ks $dest -file carsa1024.pem 

Which keystore should I used for which application?

All applications can use smart cards and external keystores through the standard key stores.

  • A queue manager on midrange uses cms format files, so you need to use the runmqakm or strmqikm similar files
  • A queue manager on z/OS can use the z/OS security manager (eg RACF), or the cms keystore files.
  • Firefox and Chrome browsers use the NSS format keystores.
  • Java clients can use a variety of keystores, .jks, .p12, smart cards
  • A C, .Net etc client use cms format keystores
  • Curl can use .pem files (from openssl) and .p12 files
  • A web server has trust stores, and keystores. You configure the supported formats in the java.security file. It can use jks, .p12 and smart cards.

Summary

It all looks a bit of a mess, and you need to know a lot to get your job done.

Debugging external smart cards and external pkcs11 keystores.

There is an open source package (opensc) which provides access to smart cards and external keystores. It provides some good tools for diagnosing problems.

There is a wiki with good information.

Opensc return codes are here, and the printable text is here

Monitor traffic to and from the device.

You can monitor the traffic to and from the device by using an intermediate “spy” module which displays the traffic.

In your configuration (for example a CCDT), where you specified the name of the module /usr/lib64/pkcs11/opensc-pkcs11.so, replace this with /usr/lib64/pkcs11/pkcs11-spy.so. Specify the environment variable

export PKCS11SPY=/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so

The spy module is invoked, prints out the parameters, and then invokes the module specified in the environment variable.

The output is like

19: C_Login
2021-03-10 14:22:47.947
[in] hSession = 0x21fc030
[in] userType = CKU_USER
[in] pPin[ulPinLen] 00000000021fb2a0 / 8
00000000 5B C7 E7 BB E5 FC 6A BE […..j.
Returned: 160 CKR_PIN_INCORRECT

Detailed internal trace

You can specify the environment variable OPENSC_DEBUG to give a very detailed trace. The higher the number the more detailed the trac.

export OPENSC_DEBUG=9

and use unset OPENSC_DEBUG to reset it.

You can use OPENSC_CONF to specify a configuration file with more parameters, such as file name for the output.

The output from this trace (showing a logon with pin number 12345678) is like

0x7f96e2dca740 14:13:16.756 [opensc-pkcs11] framework-pkcs15.c:1494:pkcs15_login: pkcs15-login: userType 0x1, PIN length 8
0x7f96e2dca740 14:13:16.756 [opensc-pkcs11] pkcs15-pin.c:301:sc_pkcs15_verify_pin: called
….
0x7f96e2dca740 14:13:16.757 [opensc-pkcs11] reader-pcsc.c:283:pcsc_transmit: reader ‘Nitrokey Nitrokey HSM (DENK01051600000 ) 00 00’
0x7f96e2dca740 14:13:16.757 [opensc-pkcs11] reader-pcsc.c:284:pcsc_transmit:
Outgoing APDU (13 bytes):
00 20 00 81 08 31 32 33 34 35 36 37 38 . …12345678

GSKIT return codes

If you are using the MQ C Client interface, this uses GSKIT. There is documentation for the z/OS version, and the return codes are here.

Getting a Java program to use a TLS MQ channel, and use an external USB keystore.

This project started off trying to get a java program to use an HSM (external keystore on a USB). This was not well documented, so I have documented my path.

I had set up my keystore on the USB HSM device as described here. This post covers the MQ side of it.

MQ provides some JMS samples.

I could use JmsBrowser using local bindings with the jms.sh script.

. /opt/mqm/java/bin/setjmsenv64
java -Djava.library.path=/opt/mqm/java/lib64 JmsBrowser -m QMA -d CP0000

To use a client channel (QMACLIENT) was easy, the program is configured to allow a channel name to be passed as a parameter using the -l option.

. /opt/mqm/java/bin/setjmsenv64
java -Djava.library.path=/opt/mqm/java/lib64 JmsBrowser -m QMA -d CP0000 -h localhost -p 1414 -l QMACLIENT

To use a TLS channel (QMAQCLIENTTLS) proved harder

. /opt/mqm/java/bin/setjmsenv64
java -Djava.library.path=/opt/mqm/java/lib64 JmsBrowser -m QMA -d CP0000 -h localhost -p 1414 -l QMAQCLIENTTLS

This gave me

CC=2;RC=2397;AMQ9641: Remote CipherSpec error for channel ‘QMAQCLIENTTLS’ to host ”. [3=QMAQCLIENTTLS]

It was hard to find where to specify the CipherSpec. The documentation said “Using the environment variables MQCHLLIB to specify the directory where the table is located, and MQCHLTAB to specify the file name of the table.” I tried this and got the same error.

I collected a trace using the Java option -Dcom.ibm.msg.client.commonservices.trace.status=ON , but the trace showed the cipher spec was null.

I gave up with this program and used an enhanced version of the program. JMS programs can use JNDI as a file repository of configuration information and the program can extract the information. For example you can configure Connection Factories(cf) which are MQ connections, and Q objects – which are queues.

Use JNDI to store parameters

JMS programs can use a JNDI interface to access configuration parameters stored in an external file.
I used this article to get me started.

My JMSAdmin.config had

INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory
PROVIDER_URL=file:///home/colinpaice/mq/JNDI-Directory
SECURITY_AUTHENTICATION=none

I used the following commands to define a connection factory called CF1, using the channel, QMACLIENT, and a queue reference called MYQUEUE pointing to queue CP0000.

mkdir JNDI-Directory
/opt/mqm/java/bin/JMSAdmin -v -cfg JMSAdmin.config
define cf(CF1) transport(client) channel(QMACLIENT)
define q(MYQUEUE) queue(CP0000)
end

I used a bash script to run the program, using the JmsJndiBrowser sample (JmsBrowser with JNDI support).

. /opt/mqm/java/bin/setjmsenv64
java -Djava.library.path=/opt/mqm/java/lib64 JmsJndiBrowser -i file:///home/colinpaice/mq/JNDI-Directory -c CF1 -d MYQUEUE

This failed with

JMSCMQ0001: IBM MQ call failed with compcode ‘2’ (‘MQCC_FAILED’) reason ‘2400’ (‘MQRC_UNSUPPORTED_CIPHER_SUITE’).

Which was a surprise to me. I looked in the trace, and there was nothing obviously wrong. The channel was not configured for TLS.

I used the following command in the JMS Admin tool, to specify a TLS channel and configure the cipher suite

alter cf(CF1) channel(QMAQCLIENTTLS) SSLCIPHERSUITE(ANY_TLS12)

Note. If I displayed cf(CF1) it gave me SSLCIPHERSUITE(*TLS12).

This gave me unable to find valid certification path to requested target, which was good progress because it meant the cipher spec had been accepted.

I added in the Java parameters to identify the key store and trust store

. /opt/mqm/java/bin/setjmsenv64

kss=”-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ecec.p12″
ksp=”-Djavax.net.ssl.keyStorePassword=password”
kst=”-Djavax.net.ssl.keyStoreType=pkcs12″
tss=”-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/dantrust.p12″
tsp=”-Djavax.net.ssl.trustStorePassword=password”
tst=”-Djavax.net.ssl.trustStoreType=pkcs12″
ks=”$kss $ksp $kst”
ts=”$tss $tsp $tst”

java $ks $ts -Djava.library.path=/opt/mqm/java/lib64 JmsJndiBrowser -i file:///home/colinpaice/mq/JNDI-Directory -c CF1 -d MYQUEUE

and it worked successfully.

Use the pkcs11 HSM external keystore.

I changed the file to specify the external keystore (with format pkcs11) on a USB device rather than a file on disk (format pkcs12).

kss=”-Djavax.net.ssl.keyStore=NONE”
kst=”-Djavax.net.ssl.keyStoreType=pkcs11″
ksp=”-Djavax.net.ssl.keyStorePassword=12345678″

but this gave

find /java.security.KeyStoreException: pkcs11 not found
java.security.NoSuchAlgorithmException: pkcs11 KeyStore not available

You have to tell Java where to find the pkcs11 code. This is configured in the java.security file. You can find it using find $JAVA_HOME/ -name java.security . This gave me the file name /usr/lib/jvm/java-8-oracle/jre/lib/security/java.security

It had the list of security providers,

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC

and is missing an entry for pkcs11.

You can edit this file, or override it. To override it, create a file like colin.java.properties with

security.provider.10=SunPKCS11 /home/colinpaice/mq/nitrokey.cfg

where

  • .10 is the next in sequence after the .9 in the java.security file
  • SunPKCS11 says use the Sun module.
  • /home/colinpaice/mq/nitrokey.cfg use this configuration file for the pkcs11 device.

Tell Java to use this override file, by using the Java option -Djava.security.properties=/home/colinpaice/mq/colin.java.properties.

The configuration file name for the pkcs11 is /home/colinpaice/mq/nitrokey.cfg with content

name = nitrokey
library = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
slot=0

With these parameters the output from the JmsJndiBrowser command was

Initial context found!
Browse starts
No more messages
SUCCESS

Success – and it only took me half a day! The hardest part was known what to specify for the security.provider…… parameter.

Using the runmqakm commands and an HSM (but not strmqikm).

I tried to use strmqikm but it gave an exception.

You can use some of the runmqakm commands you know and love, to access a certificate with an HSM. For example

The command to list the database available to the runmqakm command,

runmqakm -keydb -list -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

Gives

/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so : UserPIN (mytoken)

You can then use the token label UserPIN (mytoken) and password to use the key store, for example

runmqakm -cert -list all -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678

gives

Certificates found
* default, - personal, ! trusted, # secret key
-	my_key3

and

runmqakm -cert -details -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678
-label my_key3

displays the details of the certificate with label my_key3.

If the -tokenlabel was wrong or the -pw was wrong, I got the unhelpful messages

  • CTGSK3026W The key file “pkcs11” does not exist or cannot be read.
  • CTGSK2137W The label does not exist on the PKCS#11 device.

Create your certificate request

The following command create a new RSA private-public key pair and a PKCS10 certificate request. The documentation for runmqakm says it supports RSA. If you want to use an Elliptic Curve you will need to use an alternative method, for example openssl.

runmqakm -certreq -create -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
-tokenlabel “UserPIN (mytoken)” -pw 12345678
-dn “cn=colin,o=SSS” -file runmq.csr -label runmqlab -size 1024

Sign it

openssl ca -config openssl-ca-user.cnf -policy signing_policy -md sha256 -cert carsa1024.pem -keyfile carsa1024.key.pem -out runmq.pem -in runmq.csr

Store it back into the HSM keystore

I could not get the runmqakm command to receive the signed certificate and store it into the HSM keystore.

runmqakm -cert -receive -crypto /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -tokenlabel “UserPIN (mytoken)” -file runmq.pem -pw 12345678

It failed with

CTGSK3034W The certificate request created for the certificate is not in the key database.

I could use

openssl x509 -inform pem -outform der -in runmq.pem -out runmq.der
pkcs11-tool –write-object runmq.der –type cert –label “runmqlab” -l –pin 12345678

The openssl command converts the file from .pem format, to .der format as .der format is required by pkcs11-tool.

Using strmqikm – the theory

If you want to use the strmqikm GUI, you have to configure the java.security file. For example edit /opt/mqm/java/jre64/jre/lib/security/java.security and add the next security.provider in the list.

security.provider.12=com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl /home/colinpaice/mq/nitrokey.cfg

Where /home/colinpaice/mq/nitrokey.cfg is the configuration file, with

name = nitrokey
library = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
slot=0

You can then use Ctrl+O, which brings up a pop up with “Key database type”. In this list should be PKCS11Config, if not check your java.security file. Select this, leave File Name and Location empty, and click “OK”. It pops up “Open Cryptographic Token” with the “Token Label” value taken from the configuration file name = nitrokey. This is strange as the runmqakm command uses a TokenLabel of “UserPIN (mytoken)”.

In practice…

I then got an exception java.lang.RuntimeException: PKCS11KeyStore.java: findSigner(): Failure while executing cobj.getX509Certificate(certFactory, session), and strmqikm ended.

Using a hardware security module USB as a keystore for a browser.

Background to certificates and keystores

When using TLS(SSL) you have two keystores

  • A keystore for holding the public part and private key of your certificate
  • A trust store which holds the public keys of certificate sent to you which you need to authenticate.

Your certificate has two parts

  • The private key which contains information needed to encrypt information you send. This needs to be kept private.
  • The public part,which has information that is needed to decrypt information you have encrypted, along with information such as your Distinguished Dame (DN) such as CN=ColinPaice C=GB,O=StromnessSoftware

The process of creating a signed certificate is

  • Create a private key and public key. This can be done using an external device Hardware Security Module (HSM), such as the Nitrogen HSM USB, or software, for example using OPENSSL. This produces a private key file, and a certificate request file containing the public information.
  • Send the public information to your certificate authority which signs it, and returns it
  • Import the signed public certificate into your keystore.

Creating a certificate using an HSM as the key repository

I used openssl to process my certificates, I’ve discussed the openssl setup here.

I use a bash script because it is easy to parametrize, and makes it easy to rerun until it works. I’ll give the script, then explain what it does

  • enddate=”-enddate 20240130164600Z”
  • name=”hw”
  • rm $name.key.pem
  • rm $name.csr
  • rm $name.pem
  • ca=”carsa1024″
  • pkcs11-tool –keypairgen –key-type rsa:2048 –login –pin 648219 –label “my_key3”
  • OPENSSL_CONF=eccert.config openssl req -new -engine pkcs11 -keyform engine -key label_my_key3 -out $name.csr -sha256 -subj “/C=GB/O=HW/CN=colinpaice” -nodes
  • openssl ca -config openssl-ca-user.cnf -policy signing_policy -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate
  • openssl x509 -inform pem -outform der -in $name.pem -out $name.der
  • pkcs11-tool –write-object $name.der –type cert –label “my_key3” -l –pin 648219

What does the script do ?

enddate=”-enddate 20240130164600Z”

This sets the end date for the certificate – the end date is set when it is signed.

name=”hw”

This is used within the script to ensure the correct files are being used.

Remove old intermediate files
  • rm $name.key.pem
  • rm $name.csr
  • rm $name.pem
ca=”carsa1024″

Define the name of the CA files to use at signing time. The $ca.pem and $ca.key.pem are both needed.

pkcs11-tool –keypairgen –key-type rsa:2048 –login –pin 648219 –label “my_key3”
  • pkcs11-tool use this tool
  • –keypairgen to create a key pair (private and public pair)
  • –key-type rsa:2048 use this key type and key length
  • –login –pin 648219 login with the pin number
  • –label “my_key3” use this label to identify the key
OPENSSL_CONF=eccert.config openssl req -new -engine pkcs11 -keyform engine -key label_my_key3 -out $name.csr -sha256 -subj “/C=GB/O=HW/CN=colinpaice”
  • OPENSSL_CONF=eccert.config this sets up the openssl config file. Having -config eccert.config does not work. See here.
  • openssl
  • req this is to create a certificate requests – create a .csr.
  • -new it is a new request
  • -engine pkcs11 use the named engine, pkcs11, defined to the system
  • -keyform engine this says use the engine (HSM). Other choices are der and pem
  • -key label_my_key3 go to the engine and look for the my_key3 label
  • -out $name.csr create this request file with this name.
  • -sha256 using this signature
  • -subj “/C=GB/O=HW/CN=colinpaice” the name to go in the certificate. It uses colinpaice as the certificate will be used to authenticate with the mq web server, and this is the userid the mq web server should use.

Send the .csr file to the CA for signing (which is the same machine in my case).

openssl ca -config openssl-ca-user.cnf -policy signing_policy -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate
  • openssl ca Use this command to sign the certificate
  • -config openssl-ca-user.cnf use this configuration file
  • -policy signing_policy use this policy within the config file
  • -md sha256 use this for the message digest
  • -cert $ca.pem use the public certificate of the CA
  • -keyfile $ca.key.pem use this private key of the CA to encrypt information about the csr request’s certificate
  • -out $name.pem whee to store the output
  • -in $name.csr the input .csr request
  • $enddate specify the certificate expiry date – set at the top of the script

Send the signed certificate back to the requester.b

openssl x509 -inform pem -outform der -in $name.pem -out $name.der

The pkcs11-tool uses .der files so convert the .pem file to .der format

  • openssl x509
  • -inform pem input format
  • -outform der output format
  • -in $name.pem hw.pem
  • -out $name.der hw.der
pkcs11-tool –write-object $name.der –type cert –label “my_key3” -l –pin 648219

Read the signed certificate and write it to the HSM

  • pkcs11-tool
  • –write-object $name.der write onto the HSM the file hw.der coverted above
  • –type cert import type (cert|pubkey|privkey)
  • –label “my_key3” use this name
  • -l –pin 648219 and logon with this pin number

Define the HSM to Chrome browser

Stop the browser because you need to update the keystore.
The command was issued in the home directory, because key store is in the home directory/.pki .

modutil -dbdir sql:.pki/nssdb/ -add “my_HSM” -libfile opensc-pkcs11.so

  • modutil use this command
  • -dbdir sql:.pki/nssdb/ to up date this keystore (in ~)
  • -add “my_HSM” give it this name
  • -libfile opensc-pkcs11.so and use this file to communicate to it

Display the contents of the browser’s keystore

modutil -dbdir sql:.pki/nssdb/ -list

This gave me

Listing of PKCS #11 Modules
 NSS Internal PKCS #11 Module
...
 Mozilla Root Certs
 library name: /usr/lib/x86_64-linux-gnu/nss/libnssckbi.so
...
my_HSM
 library name: opensc-pkcs11.so
    uri: pkcs11:library-manufacturer=OpenSC%20Project;library-description=OpenSC%20smartcard%20framework;library-version=0.17
  slots: 1 slot attached
 status: loaded
 slot: Nitrokey Nitrokey HSM (DENK01051600000         ) 00 00
 token: UserPIN (SmartCard-HSM)
   uri: pkcs11:token=UserPIN%20(SmartCard-HSM);manufacturer=www.CardContact.de;serial=DENK0105160;model=PKCS%2315%20emulated 

Restart the browser.

Use an URL which needs a certificate for authentication.

The browser prompts for the pin number (twice), and displays the list of valid certificate CNs. Pick one. When I connected to the mqweb server, I had 3 certificates displayed. I had to remember which one I wanted from the Issuer’s CN and serial number. For example

SubjectIssuerSerial
colinpaiceSSCARSA1024019c
ibmsys1SSCARSA1024019a
170594SSCARSA10240197
Select a certificate

(Having a CA just for HSM keys, such as SSSCAHSM would make it more obvious.)

Using openssl with an HSM keystore, and opensc pkcs11 engines.

A Hardware Security Module (HSM) is an external device, such as USB plugin which can securely store keystores, and do other encrpyption work. I used a Nitrokey which uses open source software.

Create the key on the HSM

pkcs11-tool –keypairgen –key-type EC:prime256v1 –login –pin 12345678 –label “my_key3”

Create the certificate request using openssl

To use the opensc pkcs11 driver for an HSM you need to pass parameters to the driver.

A typical openssl command to create a certificate request, using a pre existing private key, is

OPENSSL_CONF=hw.config openssl req -new -x509 -engine pkcs11 -keyform engine -key slot_0-label_my_key3 -sha256 -out hw.pem -subj “/C=CB/O=HW/CN=HW”

The fields for the HSM device are:

  • -engine pkcs11 . This tells openssl which exernal device to use. Use the command openssl engine -vvv -tt pkcs11 to display information about the pkcs11 engine
  • -keyform engine it needs to be “engine” to use the HSM.
  • -key xxxx where xxxx can be in the format
    • n:m where n is the slot number (“where the HSM device is plugged into – the first device is usually 0) and m is the id of the certificate, public key, or private key within the device
    • slot_n-id_m where where n is the slot number (“where the HSM device is plugged into – usually 0) and m is the id within the device
    • id_m m is the id within the device (slot_n defaults to slot_0)
    • slot_n-label_name where where n is the slot number (“where the hsm device is plugged into – usually 0) and name is the keyname within the HSM
    • label_name where name is the keyname within the HSM (and slot_n defaults to slot_0)

Note the id and labels are strings, so the id value 0100 is different from the value 100.

The command pkcs11-tool -O gave

Using slot 0 with a present token (0x0)
Public Key Object; RSA 2048 bits
  label:      Private Key
  ID:         0200
  Usage:      encrypt, verify, wrap
Public Key Object; RSA 2048 bits
  label:      my_key
  ID:         0100
  Usage:      encrypt, verify, wrap
Public Key Object; RSA 2048 bits
  label:      my_key2
  ID:         0101
  Usage:      encrypt, verify, wrap
Public Key Object; RSA 2048 bits
  label:      my_key3
  ID:         0fea4632d386a7f2469eec44daeafa84a1dbd8e2
  Usage:      encrypt, verify, wrap

So I could use for -key

  • 0:0100
  • id_0100
  • slot_0-id_100
  • slot_0-label_my_key
  • label_my_key

The code for this is here.

The contents of the openssl config file for HSM devices

I’ve displayed the bits you need for the “engine” or HSM processing.

openssl_conf = openssl_def

[openssl_def]
engines = engine_section

[engine_section]
pkcs11 = pkcs11_section
 
[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/lib/x86_64-linux-gnu/engines-1.1/libpkcs11.so
MODULE_PATH    = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
PIN = 648219
# init = 0

The documentation for the config file is https://www.openssl.org/docs/man1.1.1/man5/config.html .

The use of an HSM is driven from an “engine”.

The relevant statements in the config file are

  • openssl_conf openssl searches the config file for this entry by default. It describes which other sections should always be processed. If you use openssl req …, openssl will also process the section [req]. My file says use the section called openssl_def.
  • [openssl_def] this contains the names of sections to be processed. In this case it says process the engine_section.
  • [engine_section] this lists all of the engine sections to be processed. For a pkcs11 engine process I have called the section pkcs11_section.
  • [pkcs11_section] this lists the individual entries for the pkcs11 engine.
    • The engine_id is optional
    • The dynamic path is the code for the HSM. If specified, this must come first. For the default see openssl version -e or openssl version -a.
    • You can pass parameters through the HSM. Use the openssl engine -vvv -tt pkcs11 to list them. Note they are in upper case.
    • The MODULE_PATH in upper case is passed through to the device.
    • The PIN in upper case is passed through to the HSM. Without this, you are prompted for the pin
    • Using VERBOSE = EMPTY gives more information from the HSM device.

Problems using the config file.

The following command worked

OPENSSL_CONF=eccert.config openssl req -new -x509 -engine pkcs11 -keyform engine -key id_0101 -sha256 -out hw.pem -subj “/C=CB/O=HW/CN=HW”

The following command failed

openssl req -config eccert.config -new -x509 -engine pkcs11 -keyform engine -key id_0101 -sha256 -out hw.pem -subj “/C=CB/O=HW/CN=HW”

It looks like the config file is being processed twice, so you have to use the OPENSSL_CONF variable rather than the -config option.

The error messages are

  • engine “pkcs11” set.
  • Using configuration from eccert.config
  • Error configuring OpenSSL modules
  • engine routines:engine_list_add:conflicting engine id:../crypto/engine/eng_list.c :63. I think this is reporting it has already been processed and exists in the list of engines.
  • engine routines:ENGINE_add:internal list error:../crypto/engine/eng_list.c :223:
  • engine routines:dynamic_load:conflicting engine id:../crypto/engine/eng_dyn.c :502:
  • engine routines:int_engine_configure:engine configuration error:../crypto/engine/eng_cnf.c :141: section=pkcs11_section, name=dynamic_path, value=/usr/lib/x86_64-linux-gnu/engines-1.1/libpkcs11.so These are the parts of my configuration file.
  • configuration file routines:module_run:module initialization error:../crypto/conf/conf_mod.c: 174:module=engines, value=engine_section2, retcode=-1

Some other error messages

With no MODULE_PATH in the configuration file

  • engine “pkcs11” set.
  • No private keys found.
  • PKCS11_get_private_key returned NULL
  • cannot load Private Key from engine
  • pkcs11 engine:ctx_load_privkey:object not found:eng_back.c:876:
  • engine routines:ENGINE_load_private_key:failed loading private key:../crypto/engine/eng_pkey.c:78:
  • unable to load Private Key

Initial setup for using a keystore on a HSM USB stick.

You can use a keystore on disk, but this inherently insecure, as people with administrator access to the machine, can copy the keystore. Using an external device (such as a USB Hardware Security Module) as a keystore, is more secure as you need physical access to the machine to physically access the keystore. If you have 3 failed attempts to access the keystore using a PIN code, the device locks up.

I found this document a good high level introduction to smart keys.

This post describes the initial set up for using the Hardware Security Module from Nitrokey for securely storing my digital certificates. It comes as a USB device. I chose it because it cost under 80 euros. There are other suppliers, such as yubico , and other suppliers but either they did not supply a price, or it was “call us and to discuss it”.

I found the Instructions that came with it via here, and a user blog very useful.

The Nitrokey HSM is open sourced, and uses open source facilities.

Software needed to use the key.

My machine is Linux Ubuntu 18.04.
You need software installed to configure it.

sudo apt install opensc pcscd pcsc-tools

To be able to use openssl you need an “engine” interface.

sudo apt install libengine-pkcs11-openssl

Once install you need to start it

sudo systemctl start pcscd
sudo systemctl status pcscd

gives

● pcscd.service – PC/SC Smart Card Daemon
Loaded: loaded (/lib/systemd/system/pcscd.service; indirect; vendor preset: enabled)
Active: active (running) since Tue 2021-03-02 08:49:27 GMT; 2s ago

Display it

opensc-tool –list-readers

Gave me

#Detected readers (pcsc)
Nr. Card Features Name
0   Yes           Nitrokey Nitrokey HSM (DENK01051600000 ) 00 00

This shows the card is in

  • slot number 0. You may need this number when configuring keys, for example in openssl. This number is like a USB port number.
  • it is a physical card
  • it has no features listed
  • it comes from NitroKey and is a Hardware Security Module with the given serial number.

Set up

The device has

  • a device pin (SO-PIN) which is needed for administration, such as reinitialising the device or setting a user pin. This is 16 hex characters
  • a user pin to allow users access to modify keys. The user pin is a 6-15 digit string.

You need to consider how you use your device. You can have it self contained, and the private information is private to the device. This may be acceptable for a test device, but not in production, where you want to securely backup the keystore, and securely shared the key store between different machines. This can be done using Device Key Encryption Key (DKEK). The DKEK key is a 256-Bit AES key.

You can configure this so that you need more than one person to be able to enable a new device with this DKEK. You can configure n out of m people are needed. This is described here under Using key backup and restore.

You can use sc-hsm-tool – smart card utility for SmartCard-HSM, to

  • Initialize token, removing all existing keys, certificates and files.
  • Create a DKEK share encrypted under a password and save it to the file given as parameter.
  • Read and decrypt DKEK share and import into SmartCard-HSM
  • Define device pin for initialization
  • Force removal of existing key, description and certificate.
  • Define the token label to be used in –initialize.
  • Backup a private key to an encrypted external file. (Using the DKEK).
  • Restore a private key from an external encrypted file into the device, using the DKEK

You need to initialise the device see here.

I used

sc-hsm-tool –initialize –so-pin 3537363231383830 –pin 648219 –dkek-shares 1 –label mytoken
sc-hsm-tool –create-dkek-share dkek-share-1.pbe
sc-hsm-tool –import-dkek-share dkek-share-1.pbe

The command pkcs11-tool -L gave

Available slots:
Slot 0 (0x0): Nitrokey Nitrokey HSM (DENK01051600000         ) 00 00
  token label        : UserPIN (mytoken)
  token manufacturer : www.CardContact.de
  token model        : PKCS#15 emulated
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 24.13
  firmware version   : 3.4
  serial num         : DENK0105160
  pin min/max        : 6/15

The token label : UserPIN (mytoken) has the name I entered in the –label option above. When using MQ and GSKIT with this device,I needed to identify device with “UserPIN (mytoken)” not just “mytoken”.

You can create a private key using

pkcs11-tool –keypairgen –key-type rsa:2048 –id 10 –label “my_key”

Using slot 0 with a present token (0x0)
Key pair generated:
Private Key Object; RSA
label: Private Key
ID: 10
Usage: decrypt, sign, unwrap
Public Key Object; RSA 2048 bits
label: Private Key
ID: 10
Usage: encrypt, verify, wrap

You can omit the -id, and it will generate a (long) id for you. You can list the objects (in-use slots) in the device

pkcs11-tool -O

You can delete the one we just created

pkcs11-tool -l –pin 648219 –delete-object –type privkey –id 10

You can delete -type with privkey, pubkey and cert