Category: openssl

How do you validate files on z/OS, at install time and long term?

You’ve been asked to install some files. You are not sure of their provenance, or if they have been changed from what the author wrote, and what you have received. How can you check these files?

You have some systems which are meant to be identical how can you easily check this?

One way of doing these activities is to calculate a digital hash of the file, sign the hash value, and use it.

See background for information on public/private keys.

Digitally signing is to take value, encrypt it with your private key and send it out with your certificate. To verify the signature you use the public key from the certificate to decrypt the data. It should match your copy of the value. If it matches you know it came from the certificate owner.

On z/OS you can sign load modules. These are signed at bind time, and, if configured, are checked at execution time.

Packages like openssl, and GPG (Pretty Good Privacy) have the facilities to sign objects.

This blog post covers

Installing packages

On the author’s machine

You have the Certificate Authorities file ca256.pem. This is available to every one. This could have been provided by a commercial Certificate Authority.

You have a public certificate which you will send with your package.

You have a private key matching the public certificate.

On the recipient’s machine

You have the Certificate Authority file ca256.pem.

How you do it

Create the package

You want to package data.file and distribute it. Calculate the digest of the file, and digitaly sign it

openssl dgst -sign cert.key.pem -out data.file.signature  data.file

The signature is put into data.file.signature. The signature data is in binary.

Send the following to the recipient

  • data.file
  • data.file.signature
  • your public certificate matching the private key used to sign the data. (colins.cert.pem)

Validate the package

Validate the certificate against the CA.pem 

openssl verify -CAfile ca256.pem colins.cert.pem 

If this works, you can trust the certificate.

Extract the public key from the certificate

openssl x509 -pubkey -noout -in colins.cert.pem > colins.pubkey

Validate the checksum of the file

This uses the public key extracted above

openssl dgst -verify ./colins.pubkey  -signature data.file.signature  data.file

z/OS dataset

You can use z/OS dataset, for example 

openssl dgst -sign z256.key.pem -out upa.signature "//'USER.Z31B.PARMLIB(AUTORCP)'"

where you need the double quotes and the single quotes.

Package up a package

You could create a shell script to work on a directory. For example

#!/bin/bash
for filename in ./*; do 
    [ -e "$filename" ] || continue 
    echo $filename 
    FILE="$(basename "${filename}")" 
    openssl dgst -sign z256.key.pem -out signatures/$FILE.signature $filename 
done

When I ran this, the signatures directory contained the signatures of the files in the current directory.

Long term validation

You’ve validated that the files you have installed are the correct ones. Has anyone changed them since they’ve been installed?


You can issue the command to calculate the digest(hash) of a file.

openssl dgst index.txt >> dgst.txt

Which says calculate the digest – but do not sign it. The output is

SHA256(index.txt)= 1c6e0089a3ceebddf1f8e475c164162c06d7d58f29cc0b2d4c230e4e7a79cbce
SHA256(aa.txt)= d8b1fb09ac7649b61d13ca9cde72851037a83c0bca60a8545310645bb0b3da7d

You can now periodically reissue the commands, and check the value are the same as they were previously. If the values have changed – the files have changed. You can also extend this (with a small bit of Python or shell code) to include the system name

SYS1 2026/04/22 SHA256(index.txt)= 1c6e0089a3ceebddf1f8e475c...e7a79cbce
SYS2 2026/04/22 SHA256(index.txt)= d8b1fb09ac7649b61d13ca9cd...bb0b3da7d

If those files are meant to be the same on both of the systems, they clearly are not. If you run the digest command weekly you will be able to see approximately when the file was changed. It may be that maintenance was applied to one system, and the other system was overlooked.

How do I get my client talking to the server with a signed certificate

Signed certificates are very common, but I was asked how I connected my laptop to my server, in the scenario “one up” from a trivial example.

Basic concepts

  • A private/public key pair are generated on a machine. The private stays on the machine (securely). The public key can be sent anywhere.
  • A certificate has ( amongst other stuff)
    • Your name
    • Address
    • Public key
    • Validity dates

Getting a signed certificate

When you create a certificate: it does a checksum of the contents of the certificate, encrypts the checksum with your private key, and attaches this encrypted value to the certificate.

Conceptually, you go to your favourite Certificate Authority (UKCA) building and they Sign it

  • They check your passport and gas bill with the details of your certificate.
  • They attach the UKCA public key to your certificate.
  • They do a checksum of the combined documents.
  • They encrypt the checksum with the the UKCA private key, and stick this on the combined document.

You now have a signed certificate, which you can send it to anyone who cares.

Using it

When I receive it, and use it

  • my system compares my copy of the UKCA public certificate with the one in your certificate – it matches!
  • Using (either) UKCA public certificate – decrypt the encrypted checksum
  • Do the same checksum calculation – and the two values should match.
  • If they match I know I can trust the information in the certificate.

This means the checking of the certificate requires the CA certificate that signed it.

To use a (Linux) certificate on z/OS you either need to

  • issue the RACF GENCERT command on the Linux .csr file, export it, then download it to Linux. The certificate will contain the z/OS CA’s certificate.
  • import the Linux CA certificate into RACF (This is the easy, do once solution.)

then

  • connect the CA certificate to your keyring, and usually restart your server.

Setting up my system

If the CA certificate is not on your system, you need to import it from a dataset.

You can use FTP, or use cut and paste to the dataset.

Once you have the CA certificate in your RACF database you can connect it to your keyring.

Create my Linux CA and copy it to z/OS

CA="docca256"
casubj=" -subj /C=GB/O=DOC/OU=CA/CN=LINUXDOCCA2564"
days="-days 1095"
rm $CA.pem $CA.key.pem

openssl ecparam -name prime256v1 -genkey -noout -out $CA.key.pem

openssl req -x509 -sha384 -config caca.config -key $CA.key.pem -keyform pem -nodes $casubj -out $CA.pem -outform PEM  $days

openssl x509 -in $CA.pem -text -noout|less

Where my caca.config has 

####################################################################
[ req ]
distinguished_name = ca_distinguished_name
x509_extensions    = ca_extensions
prompt             = no

authorityKeyIdentifier = keyid:always,issuer:always

[ca_distinguished_name ]
[ ca_extensions ]

subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always 
basicConstraints       = critical,CA:TRUE, pathlen:0
keyUsage               = keyCertSign, digitalSignature,cRLSign

Running the command gave

Certificate:
    Data:
        ...
        Issuer: C = GB, O = DOC, OU = CA, CN = LINUXDOCCA256
        ...
        Subject: C = GB, O = DOC, OU = CA, CN = LINUXDOCCA256
        ...
        X509v3 extensions:
           ...
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage:
               Digital Signature, Certificate Sign, CRL Sign
            ...

Where it has CA:TRUE and X509v3 Key Usage:Certificate Sign

Which allows this to be used to sign certificates.

Installing the CA certificate on z/OS

You need to copy the docca256.pem file from Linux to a z/OS dataset (Fixed block, lrecl 80, blksize 80) you can use FTP or cut and paste. I used dataset COLIN.DOCCA256.PEM.

Import it into z/OS, and connect it to the START1.MYRING keyring as a CERTAUTH.

//COLRACF  JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT CHECKCERT('COLIN.DOCCA256.PEM') 
                                                             
 *RACDCERT DELETE  (LABEL('LINUXDOCA256'))    CERTAUTH 
RACDCERT ADD('COLIN.DOCCA256.PEM')        - 
   CERTAUTH  WITHLABEL('LINUXDOCA256') TRUST 

RACDCERT   CONNECT(CERTAUTH    LABEL('LINUXDOCA256') - 
  RING(MYRING) USAGE(CERTAUTH)) ID(START1) 

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 
/*

Once you have connected the CA to the keyring, you need to get the server to reread the keyring, or restart the server.

Getting my Linux certificate signed by z/OS

This works, but is a bit tedious for a large number of certificates.

I created a certificate request file using

timeout="--connect-timeout 10"
enddate="-enddate 20290130164600Z" 

ext="-extensions end_user"

name="docec384Pass2"
key="$name.key.pem"
cert="$name.pem"
p12="$name.p12"
subj="-subj /C=GB/O=Doc3/CN="$name 
rm $name.key.pem
rm $name.csr
rm $name.pem
passin="-passin file:password.file"
passout="-passout file:password.file"
md="-md sha384"
policy="-policy signing_policy"
caconfig="-config ca2.config"
caextensions="-extensions clientServer"


openssl ecparam -name  secp384r1 -genkey -noout -out $name.key.pem 
openssl req -config openssl.config -new -key $key -out $name.csr -outform PEM -$subj $passin $passout

The certificate request file docec384Pass2.csr looks like

-----BEGIN CERTIFICATE REQUEST----- 
MIIBpzCCAS0CAQAwNDELMAkGA1UEBhMCR0IxDTALBgNVBAoMBERvYzMxFjAUBgNV 
...
Tmmvu/nqe0wTc/jJuC4c/QJt+BQ1SYMxz9LiYjBXZuOZkpDdUieZDbbEew== 
-----END CERTIFICATE REQUEST-----

With words CERTIFICATE REQUEST in the header and trailer records.

Create a dataset(COLIN.DOCLCERT.CSR) with the contents. It needs to be a sequential FB, LRECL 80 dataset.

  • Delete the old one
  • Generate the certificate using the information in the .csr. Sign it with the z/OS CA certificate
  • Export it to a dataset.
//IBMRACF2 JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD *                                                    

RACDCERT ID(COLIN ) DELETE(LABEL('LINUXCERT')) 
                                                                    
RACDCERT ID(COLIN) GENCERT('COLIN.DOCLCERT.CSR') - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) - 
   WITHLABEL('LINUXCERT') 
                                                                    
RACDCERT ID(COLIN)   LIST(label('LINUXCERT')) 
RACDCERT EXPORT(label('LINUXCERT'))    - 
    id(COLIN)                          - 
    format(CERTB64 )  - 
    password('password')    - 
    DSN('COLIN.DOCLCERT.PEM' ) 

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Then you can download COLIN.DOCLCERT.PEM to a file on Linux and use it. I used cut and paste to create a file docec384Pass2.zpem

I used it like

set -x 
name="colinpaice"
name="colinpaice"
name="docec384Pass2"
insecure=" "
insecure="--insecure"
timeout="--connect-timeout 100"
url="https://10.1.1.2:10443" 
trace="--trace curl.trace.txt"	
 	
cert="--cert  ./$name.zpem:password"
key="--key $name.key.pem"

curl -v $cert $key $url --verbose $timeout $insecure --tlsv1.2  $trace

Using wireshark I can see CA certificates being send from z/OS, and the docec384Pass2.lpem used; signed by a z/OS CA certificate.

Using the certificate in the Chrome browser.

  • In Chrome settings, search for cert.
  • Click security
  • Scroll down to Manage certificates, and select it
  • Select customised
  • Select import, and then select the file.
    • When I generated the file with the Linux CA it had a file type of .pem
    • When I signed it on z/OS, then downloaded it with a type of.zpem, I had to select all files (because the defaults are *.pem,*.csr,*.der..)

Setting up CRL on z/OS LDAP

Certificate Revocation Lists (CRLs) are an old (and deprecated) way of checking the validity of certificates. I thought this would be a good way of understanding LDAP on z/OS.

As usual this had some surprises for me,such as having to recreate my CA, and certificates that it had issued, because it was missing the crlSign attribute.

The MQ documentation was a good start (I could not find any other documentation).

How do you know if your certificate is valid?

  • The old way of doing this (from about 2008!) was to have a list containing the Certificate Authority’s certificate, and a list of the certificate serial numbers which were revoked. This was called the Certificate Revocation List(CRL). In some cases this might just be a file, but typically this was stored in LDAP. Periodically you would refresh this list.
  • If may be that an intermediate CA was revoked. As well as a Certificate Revocation List you had Authority Revocation List(ARL) (from Certificate Authority).
  • People realised that refreshing the CRL every day was leaving a certificate exposed for up to a day. Also this file could get very large.
  • These days the checks are done “online”, using an Online Certificate Status Protocol(OCSP). A certificate has an extension with information “ask this site… if I am still valid”.
  • This has been extended to support the server caching the response to the OCSP request, and sending the OCSP status down to the client so the client knows how long the status of the certificate is valid for. Eventually the OCSP status expires, and the OCSP status has to be renewed.

Revoking a certificate

I wanted to revoke a certificate that I had created on Linux, so it could not be used for authentication.

This was pretty painless, except for a couple of “ahh” moments.

I set up a bash script to revoke one of my client’s certificates, and create the file of revoked certificates.

password=” -passin file:password.file -passout file:password.file”
name=”ecec”
ca=”ca256″ # sign with this
config=”-config openssl-ca-user.cnf”
policy=”-policy signing_policy”
reason=”-crl_reason cessationOfOperation”

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem $reason –revoke $name.pem

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem –gencrl -out crl.pem

This revokes the ecec.pem file, which was signed by ca256.*. I picked a reason cessationOfOperation from the list of reasons.

The index.txt file for the CA, shows the certificate is revoked “R”

R 200229103805Z 211102120448Z, cessationOfOperation 492DF59CE0479E9523C9355077E1DD6F83C4B01E unknown /C=GB/O=cpwebuser/CN=ecec

The second command (–gencrl) produced a file crl.pem (base 64 encoded) with a list of revoked certificates:

—–BEGIN X509 CRL—–
MIIBIzCByQIBATAKBggqhk…
3vs3Bcdkrw==
—–END X509 CRL—–

You can display the file contents using

openssl crl -text -in crl.pem

For example 

Certificate Revocation List (CRL):
    Version 2 (0x1)
    Signature Algorithm: ecdsa-with-SHA256
    Issuer: C = GB, O = SSS, OU = CA, CN = SSCA256
    Last Update: Nov 2 15:33:19 2021 GMT
    Next Update: Dec 2 15:33:19 2021 GMT
Revoked Certificates:
  Serial Number: 01
    Revocation Date: Jun 19 10:46:32 2019 GMT
  Serial Number: 023F
    Revocation Date: Nov 2 11:55:32 2021 GMT
  Serial Number: 0240
    Revocation Date: Nov 2 15:33:10 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Serial Number: 492DF59CE0479E9523C9355077E1DD6F83C4B01E
    Revocation Date: Nov 2 12:04:48 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Signature Algorithm: ecdsa-with-SHA256
    30:44:02:20:21:f4:70:11:42:85:22:c8:41:b9:95:d2:3d:9e:
    0a:b4:69:fa:a4:e0:37:81:05:c7:e8:98:43:a1:dc:67:58:6c:
    02:20:14:33:bd:2e:e1:57:3b:76:22:57:b9:b2:c4:a7:29:a3:
    3c:b1:4e:5a:a3:13:bc:13:61:e2:9e:a5:7c:64:b1:b1

I ftped the crl.pem file to z/OS.

Revoking a certificate on RACF

RACF does not have the support for revoking certificates. You are meant to use z/OS® Cryptographic Services PKI Services, to manage your certificates – which looks a non trivial installation.

Configure LDAP.

The CRL information is stored under the DN of the CA. My CA is called

C = GB, O = SSS, OU = CA, CN = SSCA256

So I need an entry in the LDAP tree for cn=SSCA256, ou=CA, o=SSS, c=GB.

My ldap configuration included

database LDBM GLDBLD31/GLDBLD64
useNativeAuth all
suffix “o=Your Company”
databaseDirectory /var/ldap/ldbm

the suffix means that all entries are below “o=Your Company” for example

dn: cn=ibmuser, o=Your Company

To support the CA, I added

suffix “c=GB” to the configuration file and restarted LDAP.

I created the elements for the CA name by using an .ldif file with.

dn: c=GB
objectclass: country
c: GB

dn: o=SSS, c=GB
o: SSS
objectclass: top
objectclass: organization

dn: ou=CA, o=SSS, c=GB
ou: CA

and the CA specific CRL information

dn: cn=SSCA256, ou=CA, o=SSS, c=GB
cn: SSCA256
objectclass: cRLDistributionPoint
certificateRevocationList;binary:< file:///crl1.ldif

This page says the z/OS LDAP client LDIF parser does not support the file:// URL format. 

I modified the ldif to be 

dn: CN=SSCA256,OU=CA,O=SSS,C=GB 
cn: SSCA256 
objectclass: cRLDistributionPoint 
objectclass: certificationAuthority 
cacertificate:: 
 MIIB2DCCAX+gAwIBAgI
...
certificateRevocationList:: 
 MIIBRTCB7QIBATA
...

Where the data contents are indented one character.

I copied the contents .pem file from the CA, and the content of the CRL.pem file into the .ldif file.

I used ldapmodify… to update LDAP with the .ldif files.

I tried using the revoked certificate on Linux to issue an LDAPsearch – and it worked! (where it should have failed).

The gsktrace had

Checking revoked status CN=ecec,O=cpwebuser,C=GB

CRL cannot be located

More configuration on z/OS

The CRL processing works as follows

  • I use a certificate to connect to LDAP.
  • LDAP calls GSKit to verify the certificate, map it to a RACF userid etc.
  • If GSKIT has been configured for CRL checking, it issue a request to the CRL server to get the CRL for the CA. This could be my LDAP, or another LDAP somewhere else. It took a little while to work out, my client came in to LDAP, called GSKIT, which called the same LDAP!

Configure GSKIT

In the LDAP environment file I added

GSK_LDAP_SERVER=127.0.0.1
GSK_LDAP_PORT=389
GSK_LDAP_USER=cn=ldapid,o=Your Company
GSK_LDAP_PASSWORD=passwr0d

This does a query asking for the CA certificate, and the CRL data for my Certificate Authority’s DN.

This did not work, I got the message in the LDAP job log

GLD1116E Unable to initialize an SSL connection 8 – Certificate validation error.

In the GSK trace I had

ERROR check_crl_issuer_extensions(): crlSign bit is not set in KeyUsage
ERROR check_revoked(): Unable to verify CRL issuer extensions: Error 0x03353026

My Linux CA was not defined properly. I was missing cRLSign in my CA definition

keyUsage = keyCertSign, digitalSignature,cRLSign

I did the following

  • Recreated my CA certificate
  • Sent the updated CA to my backend systems.
  • Update the certificate on z/OS and added it to the keyrings
  • Recreated the user certificate on Linux
  • Tried the ldapsearch to z/OS

To my pleasant surprise this failed to work – as I hoped.

My Ubuntu client had

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

and there was a message in the LDAP joblog

GLD1116E Unable to initialize an SSL connection with 10.1.0.2: 431 – Certificate is revoked.

In my gsktrace file I had

ERROR check_revoked(): RFC 2459 Revoked certificate check failed: Error 0x03353041

and from 03353041 you get

03353041 Certificate is revoked.

Explanation: A certificate is revoked and cannot be used.
User response: Obtain a new certificate from the certification authority.

You have not finished yet!

Now you know the process, of getting a CRL from the CA, and updating LDAP with it you need to consider some more things.

  • You need automation to generate the CRL, and update LDAP.
  • You may want multiple LDAP servers for your organisation.
  • You should consider a DN userid for your GSK, which can only access the CA information, and no other, so even if people saw the userid and password, they could not do much with it.
  • You need to protect the data, so the default access is none, but your GSK DN can read it, and you automatic process can update it.

Hindsight.

Since I got it working, I realised that I should have a better name for my CAs, such as O= SSS, OU=CA, c=GB, CN = SSCA256 instead of C = GB, O = SSS, OU = CA, CN = SSCA256.

It is worth going through the whole processed to find out the problems, so you minimise the amount of rework. For example missing keyUsage = cRLSign, caused a lot of rework (several times till I got it right).