Debugging MQ client connection problems

I had lots of problems trying to get a client using TLS to connect to the queue manager, from set up errors, to unclear documentation.   It took me days to get my first channel set up.
IBM does not provide a program on midrange to format the GSKIT trace.  Someone suggested I use Wireshark  network packet analyser to monitor the traffic on the network.  I’ve given some examples of the handshake for TLS 12, TLS 13.

Below are some things you can do to check your set up is as you expect.

Review the error log

This is in /var/mqm/errors/AMQERR01.LOG
It does not provide all the information needed to identify the parameters used in the handshake, so you need to use the trace.

Turn on the MQ client trace

strmqtrc -e
start your program
endmqtrc -e

Format the trace

The client trace files are in /var/mqm/trace

  • cd /var/mqm/trace
  • dspmqtrc *.TRC

This will create several *.FMT files

Find the trace file for the connection

grep -F PeerName *.FMT

will list the file with the trace entries in it.  My file was  AMQ7232.0.FMT .

Check the channel name.

From the output of the grep -F PeerName *.FMT above, check the Channel Name: is what you expect? Check the remote IP address and port.  If you are using a CCDT it will take the first channel name which matches the queue manager.  This may not be the channel name you were expecting.    I had channels QMACLIENT and QMACLIENTTLS in the CCDT.   QMACLIENT was chosen instead of QMACLIENTTLS.

Check the mqclient.ini being used

grep -F mqclient.ini AMQ7232.0.FMT |grep -F FileName

Check the CCDT, (Channel Definition File) being used

grep -F ChannelDefinition AMQ7232.0.FMT

Adding stanza (ChannelDefinitionDirectory) length (19)
Adding stanza (ChannelDefinitionFile) length (10)
ChannelDefinitionDirectory = '/home/colinpaice/mq'
ChannelDefinitionFile = 'COLIN2.TAB'
Adding stanza (ChannelDefinitionFile) length (10)
Using ChannelDefinitionDirectory / MQCHLLIB value of /home/colinpaice/mq
Using ChannelDefinitionFile / MQCHLTAB value of RE.TAB
Adding stanza (ChannelDefinitionDirectory) length (19)
Adding stanza (ChannelDefinitionFile) length (10)
Using ChannelDefinitionDirectory / MQCHLLIB value of /home/colinpaice/mq
Using ChannelDefinitionFile / MQCHLTAB value of RE.TAB

The last entries show what was used.

Check the keystore being used

grep -Fi SSLKeyR *.FMT

Display the certlabl.

grep -Fi CertificateLabel *.FMT

It will display many records. They key ones are

  • MQCD CertificateLabel ‘rsaca256_client’
  • Saved CertificateLabel ‘rsaca256_client’

If these are missing certlabl has not been specified.

Display the channel definiton

Edit or browse the file and locate “CD “.

This will locate the MQCD (see the MQ documentation).

Interesting offsets in the CD are

  • 0x0000 Client channel name
  • 0x0060 Queue manager name
  • 0x00c0 IP address
  • 0x0698 Cipher Spec
  • 0x0780 TLSCertificate alias name in the client’s keystore.

Display Client Hello

You may have information about the TLS handshake.  This is called Client_hello, and Server_Hello.

Locate <client_hello> if found, it will have data like (some data removed).

client_version 
TLSV12
random ... 
session_id ... 
cipher_suites 
  tls_ri_scsv,tls_rsa_with_aes_128_cbc_sha256
compression_methods ...                              .
Extensions...
signature_algorithms 13
   rsa:sha512,rsa:sha384,rsa:sha256,rsa:sha224,...
server_name 
  qmaqclienttls.chl.mq.ibm.com
End of GSKit TLS Handshake Transcript

You need to understand the TLS handshake to fully understand this.

  • TLSV12 minimum level of TLS supported
  • cipher_suites these are what the client like
  • signature_algorithms  these are what the client will accept
  • server_name is “sni” information identifying the channel

What is my client connected to?

On a queue manager you an issue the DIS CHS(..)  all and get

SECPROT(TLSV12)
SSLCERTI(CN=SSCA256,OU=CA,O=SSS,C=GB)
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLPEER(SERIALNUMBER=01:79,CN=rsaca256,O=cpwebuser,C=GB,…)

I could not find a command to display the same information for the clients perspective.

You can look in the trace file for SSLCERTI for example

SSLCERTI(‘CN=SS,O=SSS,C=GB’)
SSLPEER(‘SERIALNUMBER=73:CB:2B:…,CN=SS,O=SSS,C=GB’)

This gives the server’s DN and Certificate issuer, and peer=subject.  As the peer has the same DN as the SSLCERTI this shows it is a self signed certificate.

Upgrading to MQ 9.2.1 on Ubuntu using apt was easy, but got messy.

Installing MQ 9.2.1 over an earlier release of MQ was almost easy!

I started here and downloaded the developer edition into ~/MQ921.

    • I had to cd ~/MQ921/MQSERVER to run the license.sh.
    • I could not edit a file in /etc/apt/sources.list.d because I was not authorised, using sudo gedit failed with:
      • Invalid MIT-MAGIC-COOKIE-1 keyUnable to init server: Could not connect: Connection refused.   Gtk-WARNING **: 10:34:55.390: cannot open display: :0
      • I could have used xhost +local:  to fix this problem.
    • I used the filename  ibmmq921-install.list instead of the suggested  ibmmq-install.list because I already had an ibmmq.list file in the directory.
    • I created  ibmmq921-install.list in my home directory and then used sudo mv ibmmq921-install.list /etc/apt/sources.list.d/ .
    • Initially I forgot to do sudo apt update – so the install did not find the updated file.
    • When I ran sudo apt update it displayed 10 packages can be upgraded. Run ‘apt list –upgradable’ to see them.
    • I ran sudo apt update and it displayed 
ibmmq-client/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-gskit/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-java/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-jre/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-man/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-runtime/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-samples/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-sdk/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-server/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]
ibmmq-web/unknown 9.2.1.0 amd64 [upgradable from: 9.1.5.0]

    • I ran sudo apt upgrade it ran for a short while and ended successfully.
    • The command dspmqver showed Version 9.2.1.0 – so success.
    • Starting the queue manager strmqm QMA gave IBM MQ queue manager ‘QMA’ started using V9.2.1.0

It got messy

After I had successfully installed MQ, the next time I ran sudo apt update it gave me lots of messages like 

Get:1 file:/home/colinpaice/MQ921/MQServer ./ InRelease
Ign:1 file:/home/colinpaice/MQ921/MQServer ./ InRelease
Get:2 file:/home/colinpaice/MQ921/MQServer ./ Release
Ign:2 file:/home/colinpaice/MQ921/MQServer ./ Release

By adding in the file, means apt checks it every time.  I could not see how any updates were going to appear in the files, so I removed the file from /etc/apt/sources.list.d/

I also deleted the files I downloaded (890 MB) and the unzipped directory (927MB).  (I needed to use sudo rm -r MQ921 because the files have write-protection.

Not very tidy.

 

 

Control unit cache section (SMF 42.2) statistics appear strange and unloved. Should I use them? No!

Why should I not use them?

The SMF 42 subtype 2 control unit cache records seem to have strange data in them – and seem to add little value.  The RMF 74 subtype 5 – Cache Subsystem Device Activity include most of the data in the 74.2,  and contains additional data.  The 74.5 record layout has good documentation.  The documentation for 42.2 does not.

Why do I think the record is unloved?

Some of the record formats are non standard. 

  1. Many SMF records use a time since midnight in hundred’s of a second.  SMF 42.2 does it in seconds (so I had to change my formatting routine – sigh).
  2. SMF dates tend to be packed decimal.   SMF 42.2 a date is documented as “Year, in the form 0cyy, where c is 0 for 19xx and 1 for 20xx, and yy”.  It is really just the year – 1900.  The value x79 is decimal 121 so 1900 + 121 is 2021 – this year.
  3. It has two data sections “Statistics gathered from last update period” and Statistics gathered from current update period.  The current is 90 seconds after the “last” ( should that be previous interval?).  I dont know what data these sections contain. Is it for the whole duration between the SMF records, or just the last 90 seconds of each interval.
  4. It has a field “Fast write bypasses per minute (an integer).” for each of the intervals.  I don’t know what it means especially when you consider the interval between the two sections is 90 seconds.
  5. The SMF records were produced every 30 minutes.  I dont know how to get the data for the other 27 minutes.
  6. There are sections for each SMS managed volume.   The data basically says “Is Cache and Fast write” enabled for this volume.  Yes/Yes. So what?

As this has a subtype of 2 I expect this record was produced 30+ years ago when people did not really know what they wanted in the SMF records.  I expect Development quietly ignored this subtype, and created other subtypes with useful data.  This is why I think you should use other records instead of this 42.2 records.

 

Do storage controllers pining having anything to do with a Norwegian Fiords?

No – it means I need to wear my glasses!

I was looking at the disk storage controller statistics and saw references to pinned storage.  I naturally thought this was a reference to the Monty Python dead parrot sketch where the Norwegian Blue was “pining for the Fiords”.

Ive often wondered what Pinned storage was.  I found “Pinned data cannot be removed from the cache by the storage machine because of a hardware failure”.  I think that if the amount of pinned storage is greater than zero then phone someone.

There are other fields “Unavailable storage”. 

These are examples of providing data – but no information.   You need to know that if these fields are non zero, this is a problem and you need to do something about it – just displaying a number has little value.  I would have hoped that that documentation would have said, in big bold letters: If these numbers are not zero then take action!

Making diff even better

I was using diff option -y  on Linux to compare two files side by side,  but it did not display the fill width of the window.

diff -W $(( $(tput cols) - 2 )) -y file1 file2 |less

solved it.

It went from being like

javax.net.ssl|AL   |    javax.net.ssl|DE
javax.net.ssl|IN   |    javax.net.ssl|DE
javax.net.ssl|DE   <
javax.net.ssl|DE   <
javax.net.ssl|DE   <
"ClientHello": {        "ClientHello": {

to


javax.net.ssl|ALL|01|main|2021-01-29 16:39:00.297 GMT|Si   |    javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:15.405 GMT|
javax.net.ssl|INFO|01|main|2021-01-29 16:39:00.297 GMT|A   |    javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:15.407 GMT|
javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:00.297 GMT|   <
javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:00.298 GMT|   <
javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:00.299 GMT|   <
"ClientHello": {                                                "ClientHello": {
  "client version"      : "TLSv1.2",                              "client version"      : "TLSv1.2",

diff -W $(( 120))…. used a terminal width of 120 or each section 60 wide.

Finding the nugget of useful information in a TLS(SSL) trace

Understanding the TLS trace, for example trying to get a client to use a web server over TLS, is a difficulty-squared problem.

In this post, I give some tools to reduce the amount of trace data, and provide and annotated trace so you can understand what is going on, and spot the errors.   I’ve also annotated the trace with common user errors, and links to possible error causes.

 This post, and the referenced pages are still work in progress while I sort out some of the little problems that creep in.  Please send me comments on any mistakes or suggestions for improvements (or additional reasons why a handshake fails).

Understanding the TLS trace is hard.

It is hard enough to understand what the trace is showing, but it is made even more difficult to use.

  1. Where there are concurrent threads running; the trace records are interleaved, and it can be hard to tell which data belongs to which thread.
  2. The formatting is sometimes poor. Instead of giving one line with a list of 50 comma separated numbers, it gives you 100 lines with either a number, or a comma.  And when you have two threads doing this, it is a nightmare trying to work out what data belongs to which thread.  (But I usually ignore these records).
  3. The trace tends to be “provide all information that might possibly be useful”, rather than provide information needed by most people to resolve why a client cannot connect to the server.  For example the trace gives a print out of the encrypted data – in hex!
  4. You turn the trace on with a java -D… parameter.  Other applications, such as web servers, have different ways of turning the trace on.    I could not find a way of turning it on and off dynamically so you can get a lot of output if you have to run for a long period.  The output may go into trace files which wrap.
  5. Different implementations have  slightly different trace formats.

All these factors made it very difficult to understand the trace and find the cause of your problems.

What can you do to understand it.

This page from Java walks through a trace, but I dont find it very helpful.  This one which also covers TLS V1.3 is better.

Do not despair! 

  • On z/OS I created an edit macro, which I use to delete or transform data.  It reduces an 8000 line spool file down to 800 lines.  See ztrace.rexx.
  • On Linux I have a python script  which does the same. See tls.py.

In some traces sections are delimited by ***…. *** to make it easier to see the structure.

To find problems look for *** at the start of the line, or “exception”.

You may need to look at both ends of the trace to understand the problem.   One end may get a response “TLSv1.2 ALERT: warning, description = …” and you need to examine the other end to find the reason for the message.

Annotated trace file

I have taken a trace file from each end, and annotated then, so you can see what the flow is, and how and where the data is used.  I have colour coded some of the flows, and included some common errors (in red) , with a link to possible solutions.
Some lines have hover text.

If you have suggestions for additional information – or reasons why things do not work – please tell me and I’ll update the documentation.  

The trace is from a Linux client going to a Liberty on z/OS.

  1. Server starts up – and waits for a connection from a client
  2. Client starts up, and sends a “Client Hello” request to the server
  3. Server wakes up, processes the request
  4. Server sends “ServerHello” to the client
  5. Optional. If the server wants client authentication,  server send the “client Authentication request”
  6. *** ServerHelloDone. It has finished the processing, send the data and wait for the reply.
  7. Client wakes up and processes the “ServerHello”, optionally sends back the “CertificationResponse”, and sends verify.
  8. Servers processes the verify and ends the handshake.

 

 

If a .jks is not allowed, how do I create a .p12 for all of my certificates?

If you try to create a trust store with a format of Java Key Store you get a message saying you should not be using a propriety format. After some digging around, I have found out how to create a database with a format of pkcs12 (or .p12 as it is also known).

To make a PKCS12 keystore (including the private key)

You can use open ssl to create the two parts, (public and private), and get the public part signed.

You can then create a .p12entry using

password=”-passout file:password.file -passin file:password.file”
name=”colincert”
ca=”-CAfile capem -chain “

openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12  -name $name  $password $ca

To make a key store, importing the .p12s with the private keys,  use

rm mykey.p12
ks=” -importkeystore -destkeystore mytrust.p12″
dest=”-deststoretype pkcs12 -deststorepass password”
src=”-srcstoretype PKCS12 -srcstorepass password”

keytool $ks $src $dest -srckeystore ecec.p12 
keytool $ks $src $dest -srckeystore ss.p12
keytool -list -v -keystore mykey.p12 -storepass password

By default it uses the alias from the -name in the openssl command.

Using the -list command without the verbose option (-v) gives

keytool -list -keystore mykey.p12 -storepass password

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains … entries

ecec, 21-Jan-2021, PrivateKeyEntry,
Certificate fingerprint (SHA1): C3:3D:98:D1:45:DF:44:F0:13:FE:87:D9:3E:CC:22:33:A8:D6:24:91
mqweb, 21-Jan-2021, PrivateKeyEntry, 

So we can see this is a PKCS12 keystore with entries with a Private Key in them.

To make a trust store

A trust store only has the public parts, it does not have the private key, so creating an intermediate .p12 file does not work, you have to use the public .pem files.

rm mytrust.p12
ks=” -import -keystore mytrust.p12 -deststoretype pkcs12″
password=
-storepass:file password.file”

keytool $ks $password -file ecec.pem -alias ecec
keytool $ks  $password -file ss.pem -alias ss

keytool -list -v -keystore mytrust.p12 $password

This example uses the password stored in the file password.file, rather than specifying it in the command.

You have to supply the -alias, as it defaults to “mykey”.  If you add multiple entries with the default, you only get one entry in the keystore.

Using the -list command without the verbose option (-v) gives

keytool -list -keystore mytrust.p12 -storepass:file password.file
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

ecec, 21-Jan-2021, trustedCertEntry,
Certificate fingerprint (SHA1): C3:3D:98:D1:45:DF:44:F0:13:FE:87:D9:3E:CC:22:33:A8:D6:24:91
ss, 21-Jan-2021, trustedCertEntry,
Certificate fingerprint (SHA1): E9:D1:69:49:5B:D3:B4:3C:E8:44:5A:C1:2C:A2:D6:5D:FB:47:61:E7

So we can see this is a PKCS12 keystore, and has the expected entries which are Trusted Certificate Entries.

 

How do I used Linux to manage my corporate certificates?

Having used z/OS to be my corporate Certificate Authority I thought I would use Linux to be a corporate CA, and manage z/OS certificates.  For more information on Certificate Authorities, and signing on certificates see here.

Setting up your Corporate CA up on Linux

At the top of the CA certificate hierarchy is a self signed certificate.

Create the CA self signed certificate

openssl req -x509 -config openssl-ca.cnf -newkey rsa:4096 -days 3000 -nodes -subj “/C=GB/O=SSS/OU=CA/CN=SSCA8” -out cacert.pem -keyout cacert.key.pem -outform PEM  addext basicConstraints=”critical,CA:TRUE, pathlen:0″ -addext keyUsage=”keyCertSign, digitalSignature”

This creates a certificate with

  1. -x509 says make it self signed – so my enterprise master CA
  2. using 4096 rsa encryption
  3. a subject “/C=GB/O=SSS/OU=CA/CN=SSCA8”
  4. valid for 3000 days
  5. Not DES encryption ( -nodes) of the output files
  6. the who and public key are stored in cacert.pem
  7. the private key is stored in cacert.key.pem
  8. using format pem
  9. extra parameters CA:TRUE and keyusage…

Display it

openssl x509 -in cacert.pem -text -noout|less

Create a personal  certificate on Linux and sign it.

Create a personal certificate on Linux  and get it signed by the CA created above.

I set up a shell script to do the work

name=”adcdd”
subj=’-subj “/C=GB/O=cpwebuser/CN=adcdd” ‘
#Passwords are stored in a file called password.file
passwords=”-passin file:password.file -passout file:password.file”

#clean out the old foils
rm $name.key.pem

rm $name.csr
rm $name.pem

CA=”cacert”

# generate a private certificate using Elliptic curve and type secp256r1
openssl ecparam -name secp256r1 -genkey -noout -out $name.key.pem

#create a certificate signing request (CSR)
openssl req -new -key $name.key.pem -out $name.csr -outform PEM $subj $passwords

#sign it – or send it off to be signed. Get the $name.pem back from the request.  
openssl ca -config openssl-ca-user.cnf -md sha384 -out $name.pem -cert cacert.pem -keyfile cacert.key.pem -policy signing_policy -extensions signing_mqweb -md sha256 -infiles $name.csr

 

#Get the  *.pem file back, if required,  and merge the files to form the .p12 file
openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $CA.pem -chain -name $name $passwords

I stored common information in configuration files, such as openssl-ca-user.cnf .  This had a section called signing_policy,  and signing_mqweb which had

[ signing_mqweb ]

keyUsage = digitalSignature
subjectAltName = DNS:localhost, IP:127.0.0.1
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

Use Linux to be the Certificate Authority for my z/OS RACDCERT certificates.

Create a certificate on z/OS.

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
 
RACDCERT ID(START1) DELETE(LABEL('MYCERTL'))
/* create the certificate - note it is not signed
RACDCERT ID(START1) GENCERT -
SUBJECTSDN(CN('MYCERTL') -
O('SSS') -
OU('SSS')) -
ALTNAME(IP(10.1.1.2) -
DOMAIN('WWW.ME2.COM') )-
SIZE(4096) -
RSA -
WITHLABEL('MYCERTL')

/* convert this to a certificate request and output it
RACDCERT GENREQ (LABEL('MYCERTL')) ID(START1) -
DSN('IBMUSER.CERT.MYCERTL.CSR')
/

The first line of the data set is —–BEGIN NEW CERTIFICATE REQUEST—– which is what I expect for a Certificate Signing Request.

FTP this down to the Linux machine as mycertl.csr and use the openssl ca command.  This uses the cacert.*.pem files created above

openssl ca -config openssl-ca-user.cnf -md sha384 -out mycertl.pem -notext -cert cacert.pem -keyfile cacert.key.pem -policy signing_policy -extensions signing_mqweb -md sha256 -infiles mycertl.csr

Note: My openssl-ca-user.cnf  is given below.

I carefully checked the details displayed, and replied y to both questions.

This produced

Using configuration from openssl-ca-user.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
organizationName :PRINTABLE:'SSS'
organizationalUnitName:PRINTABLE:'SSS'
commonName :PRINTABLE:'MYCERTL'
Certificate is to be certified until Oct 14 15:28:45 2023 GMT (1000 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

If the option -notext is not specified, the output file contains the readable interpretation of the certificate.  Specify -notext to get the output file where the first line is —–BEGIN CERTIFICATE—–

Upload this output file to z/OS (for example “put mycertl.pem ‘IBMUSER.CERT.MYCERTL.PEM’ ” ).

Check the contents before you add it to the RACF keystore

RACDCERT CHECKCERT(‘IBMUSER.CERT.MYCERTL.PEM’)

Add the certificate to the keystore

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT ADD('IBMUSER.CERT.MYCERTL.PEM') -
ID(START1) WITHLABEL('MYCERTL)
/*

The command racdcert list (label(‘MYCERTL’))  id(start1) shows the certificate has NOTRUST, so will not be visible on any keyring. You need

RACDCERT id(START1) ALTER(LABEL(MYCERTL’))TRUST

and will need to connect it to any keyrings.

Upload the CA certificate into the RACF database.

You will also need to upload the public key to the RACF database as a CERTAUTH, and connect it to any ring that uses a certificate signed by the Linux CA.

FTP the certificate file, cacert.pem (created above), to z/OS as text.  Once you have FTPed the file, check the first line is “—–BEGIN CERTIFICATE—–“

Add it to RACF

You can add the certificate owned by a userid ( rather than certauth).   The certificate needs to be connected to the keyring as usage(CERTAUTH).

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
/* delete it if needed
RACDCERT DELETE (LABEL('Linux-CA256')) ID(START1)

RACDCERT id(start1) ADD('IBMUSER.CA256.PEM') -
WITHLABEL('Linux-CA256') TRUST

RACDCERT CONNECT(id(start1) LABEL('Linux-CA256') -
RING(TRUST) USAGE(CERTAUTH)) ID(START1)

RACDCERT CONNECT(id(start1) LABEL('Linux-CA256') -
RING(DANRING) USAGE(CERTAUTH)) ID(START1)

RACDCERT LISTRING(TRUST ) ID(START1)

racdcert list (label('Linux-CA256')) id(start1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

 

My openssl-ca-user.cnf file.

HOME = .
RANDFILE = $ENV::HOME/.rnd

####################################################################
[ ca ]
default_ca = CA_default # The default ca section

[ CA_default ]
default_days = 1000 # How long to certify for
default_crl_days = 30 # How long before next CRL

default_md = sha256 # Use public key default MD
preserve = no # Keep passed DN ordering

x509_extensions = ca_extensions # The extensions to add to the cert

email_in_dn = no # Don’t concat the email in the DN
copy_extensions = copy # Required to copy SANs from CSR to cert

base_dir = .
certificate = $base_dir/cacert.pem # The CA certifcate
private_key = $base_dir/cakey.pem # The CA private key
new_certs_dir = $base_dir # Location for new certs after signing
database = $base_dir/index.txt # Database index file
serial = $base_dir/serial.txt # The current serial number

unique_subject = no # Set to ‘no’ to allow creation of
# several certificates with same subject.

[ signing_policy ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

##########

[ signing_mqweb ]

subjectAltName = DNS:localhost, IP:127.0.0.1
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

 

Useful linux commands

Someone told me about a useful linux command, for using z/OS… so I thought I would pass on some one-liners to help other people.

sshfs colin@10.1.1.2: ~/mountpointz

This mounts a remote file system to give local access to files on a remote box.  This allows you to use gedit remotely, including z/OS files in USS.  For example gedit mountpointz/mqweb/servers/mqweb/mqwebuser.xml .

For z/OS the files must be tagged.   Use  chtag -p name to see the tag.  For me it needs a tag of  “t ISO8859-1 T=on” .  Without the tag it display the file in the wrong code page.

parcellite

This keeps a history of your clipboard, with hot keys.

Linux settings -> keyboard mapping

Ctrl+1 give me the x3270 with “tso@” in the title.   The command executed is wmctrl -a tso@
Ctrl+2 give me the x3270 with “colin@” in the title.   The command executed is wmctrl -a colin@

I start the x3270 with

x3270 -model 5 colin@localhost:3270 &
x3270 -model 5 tso@localhost:3270 &

model 5 makes the screen 132 characters wide by 80 deep

Map the 3270 keyboard to define the uss escape key

See x3270 – where’s the money key?

Gedit hot keys

See here

  • Find a string Ctrl-F , next string Ctrl-G, previous Ctrl-Shift-G
  • Top Ctrl-home, bottom Ctrl-End
  • Move left/right one word Alt- <- or Alt–>
  • Start/end of line Home/end or  Ctrl-pgup,Ctrl-pgdn
  • Next session Ctrl-Alt Pgup, Ctrl-Alt-Pgdn
  • Create new gedit window Ctrl-N
  • Create new tab Ctrl-T

 

How do I used z/OS to manage my corporate certificates?

With my first, (and second) reading of the RACF documentation it looked like z/OS could not act as my corporate Certificate Authority.  I was written up a blog post saying how to do it using Linux as my CA, when I found a couple of hints to show that z/OS could be used as a certificate authority.

The short answer on how to do it.

Create a end user certificate on z/OS

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('MYCERT') - 
             O('SS') - 
             OU('SS')) - 
   ALTNAME(IP(10.1.1.2) - 
           DOMAIN('WWW.ME.COM') )- 
   SIZE(4096) - 
   SIGNWITH (CERTAUTH LABEL('MY-CA1')) - 
   RSA - 
   WITHLABEL('MYCERT')

This creates a certificate START1.MYCERT signed by CERTAUTH.MY-CA1

How to process a certificate created on Linux

I created a certificate on Linux in a script (so I could use a variable as part of the file name).

name=”ec2″
passwords=
“-passin file:password.file -passout file:password.file
# generate a private key in rsa format
openssl genrsa -out $name.key.pem 2048
# make a certificate request to send off to get it signed
openssl req -config mqwebserver.config -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=mqwebec”  $passwords

This creates a Certificate Server Request file ec2.csr, and a private key in ec2.key.pem.

I FTP’d the .csr file to z/OS.   This is a text file, where the first line is —–BEGIN CERTIFICATE REQUEST—– so you need to FTP in text format.

With RACF you can generate a certificate by passing the information in a data set, or by specifying it on the RACDCERT GENCERT command.

I used the following job to sign the certificate using Certificate Authority DAN-CA1, passing the information in from the .CSR data set I had previously uploaded.

//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT ID(START1) GENCERT('IBMUSER.IBMUSER.EC2.CSR') - 
   SIGNWITH (CERTAUTH LABEL('MY-CA1')) - 
   WITHLABEL('MYCERT3') 

RACDCERT LIST(LABEL('MYCERT3')) ID(START1)
 
RACDCERT ID(START1) EXPORT(LABEL('MYCERT3'))- 
   DSN('IBMUSER.CERT.MYCERT3.PEM') - 
   FORMAT(CERTB64) 

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 

This reads the data set specified in the GENCERT() command, and takes all of the certificate parameters from it.  It is signed by the CERTAUTH with label(‘MY-CA1’) and stored in the RACF database under id(START1) with label(MYCERT3).

THE EXPORT command exports the signed certificate to a dataset.

I then FTP’d the exported data set IBMUSER.CERT.MYCERT3.PEM back to Linux.

I merged the two parts back together to create a .p12 combined file with

openssl pkcs12 -export -inkey ec2.key.pem -in dancert3.pem -out mycert3.p12 -name mycert3 -passout file:password.file -passin file:password.file

and displayed the .p12 file with

pk12util -l mycert3.p12

This gave me

Certificate(has private key):
Data:
  Version: 3 (0x2)
  Serial Number: 4 (0x4)
  Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
  Issuer: "CN=MYCA1,OU=SSS,O=SSS"
  Validity:
    Not Before: Sun Jan 17 00:00:00 2021
    Not After : Mon Jan 17 23:59:59 2022
  Subject: "CN=mqwebec,O=cpwebuser,C=GB"
  Subject Public Key Info:
    Public Key Algorithm: PKCS #1 RSA Encryption
    RSA Public Key:
    Modulus:...

    Exponent: 65537 (0x10001)
  Signed Extensions:
    Name: Certificate Comment
      Comment: "Generated by the Security Server for z/OS (RACF)"

    Name: Certificate Subject Alt Name
      DNS name: "localhost"
      IP Address: 127.0.0.1

  Name: Certificate Key Usage
    Critical: True
    Usages: Digital Signature
    Non-Repudiation
    Key Encipherment

  Name: Certificate Subject Key ID
    Data:...

  Name: Certificate Authority Key Identifier
    Key ID:...

  Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
    Signature:...
    Fingerprint (SHA-256):...
    Fingerprint (SHA1):...Friendly Name: dancert3

  Key(shrouded):
  Friendly Name: mycert3

  Encryption algorithm: PKCS #12 V2 PBE With SHA-1 And 3KEY Triple DES-CBC

Note: I tried exporting the file on z/OS using FORMAT(PKCS7B64) instead of FORMAT(CERTB64).
The openssl pkcs12 -export command failed with

unable to load certificates
error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149:
error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:309:Type=X509_CINF
error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=cert_info, Type=X509
:error:0907400D:PEM routines:PEM_X509_INFO_read_bio:ASN1 lib:../crypto/pem/pem_info.c:196:

If you get these messages download it in the proper format.

The longer answer…

When setting up certificates you have several options

  1. Use a Self Signed certificate.  This offers very little security, as anyone can create a certificate one.   This should only be used for testing, though I think you should never use them, as you need to test with signed certificates because of the additional work that needs to be done.  Imagine conversation with your manager “Did you test this?”  “No, I tested something else”.
  2. Get a certificate from recognised authority.   You pay for each certificate you get.  As the authority’s certificate is generally available you can use your certificate as soon as you get it.
  3. Set up your own authority you create your own high level certificate such as CN=TOPLEVELCA O=MYORG, which in turn creates other certificates such as CN=TESTCA, O=MYORG, and CN=PRODUCTIONCA O=MYORG.  You can create as many certificate as you wish for example using the TESTCA for certificates that can be used in test.
  4. A combination of both, you get a High level Certificate Authority certificate signed by a recognised authority, and then used that  to set up your own certificates.

What does a “certificate” contain?

There are two parts

  1. The certificate identified the “who”, along with the public key.
  2. The private key used in encryption.

What does “signing”  mean?

You take a block of data, and do a calculation on it, for example a checksum.  The signer takes the checksum and encrypts it using its private key.  The original block of data then has the encrypted checksum, and the signers public key appended to it.

To check the signature,

  1. You compare the signers public key with your copy of it and check it matches.  If they match you decrypt the encrypted checksum.
  2. You perform the same calculation on the original payload and check your calculation with the decrypted version.  If they match you know the data is good.

Signing a certificate is doing this with a certificate as the block of data.

If you have a corporate CA certificate, and a departmental CA certificate, the departmental certificate has been signed by the corporate CA.  When you sign a block of data, it will have the departmental certificate, and the corporate certificate appended.

How does a Certificate Authority work?

At the top of the Certificate Authority hierarchy is a generally available self signed certificate with its public key.

  • I create a personal certificate with  “who” and the public key, and the private key.
  • I send the personal certificate with  “who” and the public key, known as a Certificate Signing Request(CSR) to my security department, who sign it and return it.
  • I merge the signed “who” and the public key, with the private key, and store it in my key store.