Don’t try this at home, PKI certificates.

I was trying to generate z/OS certificates which I could use to check out certificate revocation. I can do it in Linux – no problem. Getting it to work on z/OS was the challenge and I don’t think it can be done.

Meanwhile, back at the ranch (‘at home’), running z/OS on the zPDT environment on Linux, I found the PKI Server environment does not support PKI for key generation on zPDT or ZD&T, because zPDT does not support ICSF TKDS.

The longer story of why it will not work.

The RACF command RACDCERT GENCERT creates a basic certificate which is suitable for many uses. It does not support the extensions, such as specifying a URI for OCSP checking of the certificate.
The PKI product has these capabilities, and together with ICSF it can store the keys in the ICSF data sets. This product seems complex to set up (two web servers), and a GUI interface instead of a command interface.

There is an API, the RACF callable services R_PKISERV, which allows you to issue API requests to administer certificate.
You can use a SAF interface and pass in a public certificate, certificate request, or use PKI to generate a full certificate with all of the optional fields etc, and do full life cycle management with it.

I could not get this to work, and when I started the PKI SERVER, it reported


This in turn pointed me to ICSF and the TKDS (Token Key Data Set), not being set up – it needed a master key. You enter this master key on a TKE (Trusted Key Entry) workstation which sits inside the z hardware. I was running on zPDT, and following the trail, the zPDT documentation said PKCS#11 (Token Key Data Set) is not available with zPDT. This means it looks like I cannot create certificates which support OCSP on my z/OS.

How to delete a RACF group

It only took me 30 minutes to do so!

I was creating some RACF definitions for a product, and being a good citizen,I wanted to have a script which cleans up after me, and deletes anything I defined. This all worked fine, except for trying to delete a RACF group. For example

DELGROUP  PKIGRP3                            
IKJ56702I INVALID GROUP, PKIGRP3            
ADDGROUP PKIGRP3                            

Hmm the message is not helpful – PKIGRP3 >IS< valid.

IKJ56702I INVALID invalid data
Explanation: The user entered invalid data.

I had a mini project to find out why it was not being deleted. If I list the group it gives

LG PKIGRP3                                                                                
INFORMATION FOR GROUP PKIGRP3                                                            
    SUPERIOR GROUP=SYS1         OWNER=COLIN       CREATED=21.315                          
    NO INSTALLATION DATA                                                                  
    NO MODEL DATA SET                                                                    
    NO SUBGROUPS                                                                          
    USER(S)=      ACCESS=      ACCESS COUNT=      UNIVERSAL ACCESS=                      
      IBMUSER       USE           000000               NONE                              
         CONNECT ATTRIBUTES=NONE                                                          
         REVOKE DATE=NONE                 RESUME DATE=NONE                                

If I remove the userid from the group the delete group works


Easy when you know why.

Having to remove all of the users from a group before deleting the group means I cannot just have code to delete the userids I had created, and delete the groups I created. I’ve raised an RFE on this.

Old dogs, new tricks, ISPF compare

When looking for a comparison facility for Unix files, I stumbled across ISPF edit compare function.

If you are editing a file you can use the compare * command to show the changes you made to the file in the current session. This gave me

000001 line 1
====== line 3
.OAAAA line 2
.OAAAB line 4
.OAAAC line 5


  • line 1 was unchanged
  • line 3 was deleted
  • line 2,4,5 were added

You can then use “mm” or “mdd …. mdd” prefix commands on the ====== to apply the changes to the file, and use “d” or “dd … dd” on the .O prefix lines to delete the lines.

You can use the compare command, and be prompted for options.
I specified a Unix file, and could see the differences.

You can set the exclude option (at the bottom of the panel), which says exclude all the lines, but display ‘n’ lines around changes, where ‘n’ is specified at the top of the panel.

One funny….

I noticed that compare was display “blank” lines in Unix files as being changed. The original file had a null line, with just an end of line. When I customised the file and saved it, ISPF replaces an “end of line” with “blank end of line”. As these do not match, they show up as being changed.

The solution is to edit the original file with null lines, make a dummy change, and save the file. The file will no longer have null lines, and only your changes will be visible.

ASCII files in OMVS

I found I was unable to compare python source files in OMVS, because they were ASCII files. (When the files were tagged as ASCII, I could edit them).

You cannot use ISPF compare if the files are ASCII (non EBCDIC). You can do

Compare the files then use the following to transform them back.

Using RACF ids in LDAP

I stumbled across this useful way of defining userids to LDAP.

In your LDAP configuration file, define the SDBM database

database sdbm GLDBSD31/GLDBSD64
suffix o=myracf

This says for all Distinguished Names (DN) ending in o=myracf, then go to the RACF (SAF) database.

If my DN is RACFID=COLIN,PROFILETYPE=user,o=myracf it will use this. I do not need to set up a special DN in LDAP.

I used

ldapsearch -h -D “RACFID=colin,PROFILETYPE=user,o=myracf” -w ? -b “o=myracf” “(objectClass=*)”

To list all the userids and groups in RACF.

For those IDs which map to a SAF userid, for example

  • defined in LDAP with the attribute ibm-nativeId:
  • using a SAF userid directly RACFID=COLIN,PROFILETYPE=user,o=myracf,
  • via a certificate and RACDCERT MAP

that userid is used to issue the command. For example userid COLIN is a member of group SYS1, and can display information from commands like TSO LU ANOTHER.

Another ID with no special authority returned no data from the ldapsearch command above.

The SDBM backend is virtual directory and is mostly read-only, so the update operations are usually not allowed. RACF configuration is used to restrict a user’s authority to SDBM  and ACLs are not used.
SDBM only supports RACF user IDs or user IDs with RACF mappings, so a user must map to a SAF user to be able to query the RACF data in the SDMB database.

Possible queries

This table gives some LDAP queries.

  • -b “o=myracf” “(objectclass=*)” list all users groups etc
  • b “profiletype=User,o=myracf” “(objectclass=*)” list all users
  • -b “profiletype=Group,o=myracf” “(objectclass=*)” list all users
  • -b “profiletype=Connect,o=myracf” “(objectclass=*)” list all user and group connections
  • -b “racfid=colin,profiletype=user,o=myracf” “(objectclass=*)” list everything about racfid COLIN
  • -b “cn=setropts,o=myracf” “(objectclass=*)” did not work for me
  • -b “profiletype=facility,o=myracf” “(objectclass=*)” did not work for me

Setting up CRL on z/OS LDAP

Certificate Revocation Lists (CRLs) are an old (and deprecated) way of checking the validity of certificates. I thought this would be a good way of understanding LDAP on z/OS.

As usual this had some surprises for me,such as having to recreate my CA, and certificates that it had issued, because it was missing the crlSign attribute.

The MQ documentation was a good start (I could not find any other documentation).

How do you know if your certificate is valid?

  • The old way of doing this (from about 2008!) was to have a list containing the Certificate Authority’s certificate, and a list of the certificate serial numbers which were revoked. This was called the Certificate Revocation List(CRL). In some cases this might just be a file, but typically this was stored in LDAP. Periodically you would refresh this list.
  • If may be that an intermediate CA was revoked. As well as a Certificate Revocation List you had Authority Revocation List(ARL) (from Certificate Authority).
  • People realised that refreshing the CRL every day was leaving a certificate exposed for up to a day. Also this file could get very large.
  • These days the checks are done “online”, using an Online Certificate Status Protocol(OCSP). A certificate has an extension with information “ask this site… if I am still valid”.
  • This has been extended to support the server caching the response to the OCSP request, and sending the OCSP status down to the client so the client knows how long the status of the certificate is valid for. Eventually the OCSP status expires, and the OCSP status has to be renewed.

Revoking a certificate

I wanted to revoke a certificate that I had created on Linux, so it could not be used for authentication.

This was pretty painless, except for a couple of “ahh” moments.

I set up a bash script to revoke one of my client’s certificates, and create the file of revoked certificates.

password=” -passin file:password.file -passout file:password.file”
ca=”ca256″ # sign with this
config=”-config openssl-ca-user.cnf”
policy=”-policy signing_policy”
reason=”-crl_reason cessationOfOperation”

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem $reason –revoke $name.pem

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem –gencrl -out crl.pem

This revokes the ecec.pem file, which was signed by ca256.*. I picked a reason cessationOfOperation from the list of reasons.

The index.txt file for the CA, shows the certificate is revoked “R”

R 200229103805Z 211102120448Z, cessationOfOperation 492DF59CE0479E9523C9355077E1DD6F83C4B01E unknown /C=GB/O=cpwebuser/CN=ecec

The second command (–gencrl) produced a file crl.pem (base 64 encoded) with a list of revoked certificates:

—–BEGIN X509 CRL—–
—–END X509 CRL—–

You can display the file contents using

openssl crl -text -in crl.pem

For example

Certificate Revocation List (CRL):
    Version 2 (0x1)
    Signature Algorithm: ecdsa-with-SHA256
    Issuer: C = GB, O = SSS, OU = CA, CN = SSCA256
    Last Update: Nov 2 15:33:19 2021 GMT
    Next Update: Dec 2 15:33:19 2021 GMT
Revoked Certificates:
  Serial Number: 01
    Revocation Date: Jun 19 10:46:32 2019 GMT
  Serial Number: 023F
    Revocation Date: Nov 2 11:55:32 2021 GMT
  Serial Number: 0240
    Revocation Date: Nov 2 15:33:10 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Serial Number: 492DF59CE0479E9523C9355077E1DD6F83C4B01E
    Revocation Date: Nov 2 12:04:48 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Signature Algorithm: ecdsa-with-SHA256

I ftped the crl.pem file to z/OS.

Revoking a certificate on RACF

RACF does not have the support for revoking certificates. You are meant to use z/OS® Cryptographic Services PKI Services, to manage your certificates – which looks a non trivial installation.

Configure LDAP.

The CRL information is stored under the DN of the CA. My CA is called

C = GB, O = SSS, OU = CA, CN = SSCA256

So I need an entry in the LDAP tree for cn=SSCA256, ou=CA, o=SSS, c=GB.

My ldap configuration included

useNativeAuth all
suffix “o=Your Company”
databaseDirectory /var/ldap/ldbm

the suffix means that all entries are below “o=Your Company” for example

dn: cn=ibmuser, o=Your Company

To support the CA, I added

suffix “c=GB” to the configuration file and restarted LDAP.

I created the elements for the CA name by using an .ldif file with.

dn: c=GB
objectclass: country
c: GB

dn: o=SSS, c=GB
o: SSS
objectclass: top
objectclass: organization

dn: ou=CA, o=SSS, c=GB
ou: CA

and the CA specific CRL information

dn: cn=SSCA256, ou=CA, o=SSS, c=GB
cn: SSCA256
objectclass: cRLDistributionPoint
certificateRevocationList;binary:< file:///crl1.ldif

This page says the z/OS LDAP client LDIF parser does not support the file:// URL format. 

I modified the ldif to be

cn: SSCA256 
objectclass: cRLDistributionPoint 
objectclass: certificationAuthority 

Where the data contents are indented one character.

I copied the contents .pem file from the CA, and the content of the CRL.pem file into the .ldif file.

I used ldapmodify… to update LDAP with the .ldif files.

I tried using the revoked certificate on Linux to issue an LDAPsearch – and it worked! (where it should have failed).

The gsktrace had

Checking revoked status CN=ecec,O=cpwebuser,C=GB

CRL cannot be located

More configuration on z/OS

The CRL processing works as follows

  • I use a certificate to connect to LDAP.
  • LDAP calls GSKit to verify the certificate, map it to a RACF userid etc.
  • If GSKIT has been configured for CRL checking, it issue a request to the CRL server to get the CRL for the CA. This could be my LDAP, or another LDAP somewhere else. It took a little while to work out, my client came in to LDAP, called GSKIT, which called the same LDAP!

Configure GSKIT

In the LDAP environment file I added

GSK_LDAP_USER=cn=ldapid,o=Your Company

This does a query asking for the CA certificate, and the CRL data for my Certificate Authority’s DN.

This did not work, I got the message in the LDAP job log

GLD1116E Unable to initialize an SSL connection 8 – Certificate validation error.

In the GSK trace I had

ERROR check_crl_issuer_extensions(): crlSign bit is not set in KeyUsage
ERROR check_revoked(): Unable to verify CRL issuer extensions: Error 0x03353026

My Linux CA was not defined properly. I was missing cRLSign in my CA definition

keyUsage = keyCertSign, digitalSignature,cRLSign

I did the following

  • Recreated my CA certificate
  • Sent the updated CA to my backend systems.
  • Update the certificate on z/OS and added it to the keyrings
  • Recreated the user certificate on Linux
  • Tried the ldapsearch to z/OS

To my pleasant surprise this failed to work – as I hoped.

My Ubuntu client had

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

and there was a message in the LDAP joblog

GLD1116E Unable to initialize an SSL connection with 431 – Certificate is revoked.

In my gsktrace file I had

ERROR check_revoked(): RFC 2459 Revoked certificate check failed: Error 0x03353041

and from 03353041 you get

03353041 Certificate is revoked.

Explanation: A certificate is revoked and cannot be used.
User response: Obtain a new certificate from the certification authority.

You have not finished yet!

Now you know the process, of getting a CRL from the CA, and updating LDAP with it you need to consider some more things.

  • You need automation to generate the CRL, and update LDAP.
  • You may want multiple LDAP servers for your organisation.
  • You should consider a DN userid for your GSK, which can only access the CA information, and no other, so even if people saw the userid and password, they could not do much with it.
  • You need to protect the data, so the default access is none, but your GSK DN can read it, and you automatic process can update it.


Since I got it working, I realised that I should have a better name for my CAs, such as O= SSS, OU=CA, c=GB, CN = SSCA256 instead of C = GB, O = SSS, OU = CA, CN = SSCA256.

It is worth going through the whole processed to find out the problems, so you minimise the amount of rework. For example missing keyUsage = cRLSign, caused a lot of rework (several times till I got it right).

Using z/OS LDAP with TLS 1.3

This blog post started as one line in the Using z/OS LDAP with TLS post. After it took two days to get working, I thought it deserved its own page.

Displaying the handshake

Using Wireshark to display TLS traffic is harder with TLS 1.3, as most of the handshake is now encrypted. See here on how to display it.

What supports LDAP and TLS 1.3?

I could not get Openldap on Ubuntu 18.04 to use TLS 1.3 – I do not think it has the support.

I could use the z/OS LDAPSEARCH with TLS 1.3, see below.

Java should work with the right set up.

Updating LDAP server on z/OS

Updates to enable TLS V1.3 protocol support for GSKIT has some good information.

You need to add the following to the configuration file


# TLS 1.3 only supports sslFipsState off
slFipsState off

Below are the environment variables I used for the LDAP server on z/OS


I think this is the super set of data, and should be used as a starter list. You can remove unwanted cipher specs once you have it working. It is easier doing it this way compared to adding those you need (because it is hard to find what you need).


You need to add the TLS 1.3 cipher specs to the GSK_V3_CIPHER_SPECS_EXPANDED in the environment file.

For example



This variable needs to include 0806,0805,0804.

Key share

TLSv1.3 uses client and server key-shares to facilitate the encryption of TLSv1.3 handshake messages and to determine the key exchange algorithms.

You give a preference as to which elliptic curve to use.

You need to list the key shared values (from this list (table 5)), where  secp256r1 has the value 0023.

GSK_SERVER_TLS_KEY_SHARES is used by the server. The client has GSK_CLIENT_TLS_KEY_SHARES




Remove any definitions in the keyshares which are not supported by TLS 1.3 (0019 and 0021) (even though you have GSK_PROTOCOL_TLSV1_2=on)

With these my LDAP started.

Once you have LDAP working, you can review the cipher specs, and remove the ones you no longer use. You should plan to remove the weaker cipher specs. If you remove a cipher spec and it is still needed, clients will fail to connect. You will need to replace the cipher specs and restart the server.

Using openssl s_client on Linux to establish a session to LDAP on z/OS.

Openssl s_client sends the TLS 1.3 supported cipher specs as part of the initial client hello handshake, so you can see the flows.

When I did this I got

s_client fails with SSL alert number 42.

From the z/OS GSK trace I can see

ERROR check_cert_extensions_3280_and_later(): authorityKeyIdentifier extension is missing
ERROR validate_certificate_basics(): Unable to verify certificate extensions: Error 0x0335306f
ERROR validate_certificate_mode(): Unable to validate certificate: Error 0x0335306f
ERROR cms_validate_certificate_mode_int(): Unable to validate certificate: Error 0x0335306f

ERROR read_tls13_certificate(): Unable to validate peer certificate: Error 0x0335306f

ERROR send_tls13_alert(): Sent TLS 1.3 alert 42 to[42514]

Where 0x0335306f (search for 0335306f) is


A certificate extension that is mandatory for the certificate to be used for the required purpose has not been found.

User response

Ensure that the certificate chain is correct and complies with the validation mode defined for the connection. Collect a System SSL trace containing the error and then contact your service representative if the error persists.

A certificate authority can use multiple certificates to sign a certificate, for example you are refreshing or upgrading the certificate. Having the authorityKeyIdentifier says which certificate was used to sign it.
On the web there are documents which recommend its use, for example with openssl, the configuration file should have.

[ v3_ca ]


On z/OS with RACF you get this attribute when using RACDCERT GENCERT … SIGNWITH() .

On my system I had misconfigured it

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer

I had to

  • create a new CA with the above in the Openssl configuration file
  • reissue the end user certificate
  • update all my servers with the new CA
  • update my LDAP CRL server

Wow – what a lot of work for a user error.

When I had done the above work, I used the

openssl x509 -in rsaca256.pem -text -noout

to list the end user certificate, and there was additional information

X509v3 Authority Key Identifier:

Using openLDAP ldapsearch on Ubuntu 18.04

I had problems getting ldapsearch on Linux to work with TLS 1.3, because it was difficult to get TLS 1.3 support – it did not send the TLS 1.3 cipher specs as part of the Client Hello. The documentation describes TLS_CIPHER_SUITE option. The Openssl syntax did not work, and the GnuTLS Syntax did work but I could not get it working with TLS 1.3 – perhaps it is not supported.

Using ldapsearch from z/OS

This was easy to set up, and I could use the gsktrace from each end to observe the flows.

I set up a bash script

export GSK_TRACE=0xff
export GSK_TRACE_FILE=./colin.gsktrace
export GSK_PROTOCOL_TLSV1_1=”off”
export GSK_PROTOCOL_TLSV1_2=”off”
export GSK_PROTOCOL_TLSV1_3=”on”
export GSK_CLIENT_TLS_KEY_SHARES=”003000290025002400230021″
export GSK_TLS_SIG_ALG_PAIRS=”0601050104010301080608050804050304030603″
export GSK_CLIENT_ECURVE_LIST=”00230024002500290030″
export GSK_V3_CIPHER_SPECS_EXPANDED=”130213011303C02BC02CC030C028C02aC024C026C02E”
debug=”-d all”
debug=”-d error+conns”
host=”-h -p 1389″
host=”-h -p 9443″
ldapsearch $host $tls $debug -b “o=Your Company” “(objectclass=*)” aclEntry

gsktrace colin.* > out
oedit out

I defined an environment variable more than once, so I just needed to move the line down to change what was used. For example by swapping over the “tls” lines I could quickly use a different certificate.

I added the commands to format the gsktrace and display it.


Because I used the debug value “-d error+conns” I got

CONNS ldap_ssl_socket_init()1600: 4-byte ciphers format will be used
CONNS ldap_ssl_socket_init()1625: No 4-byte cipher specs specified in LDAP handle
CONNS ldap_ssl_socket_init()1689: SSL Protocol Version = 621
CONNS ldap_ssl_socket_init()1700: SSL Cipher Specs
Address: 1FA36946 Length: 4 (x’4′)
00000000: f1f3f0f3 *1303 *
ERROR ldap_parse_pwdpolicy_response_int()856: No server controls o=Your Company

I think these messages are OK, and do not show a problem.

You can see that cipher spec 1303 was used.

Other information

TLS 1.3 uses one of the following ciphers

  • TLS_AES_128_GCM_SHA256 -1301
  • TLS_AES_256_GCM_SHA384 – 1302
  • TLS_CHACHA20_POLY1305_SHA256 1303
  • TLS_AES_128_CCM_SHA256 -1304
  • TLS_AES_128_CCM_8_SHA256 -1305

How do I look at TLS 1.3 handshakes?

With TLS before 1.3, encryption of the data on the session occurred after the handshake had completed, so the handshake was visible in Wireshark. With TLS 1.3 the traffic is encrypted after the “Client Hello”, so you cannot immediately see the remainder of the handshake.

Tools like OpenSSL, can write out the magic data needed for decryption. For example

openssl s_client -keylogfile /tmp/kl -connect -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile /home/colinpaice/ssl/ssl2/colinpaice.pem

This writes information to the specified file, in my case /tmp/kl.

The file has data like

SSL/TLS secrets log file, generated by OpenSSL

On Linux (Ubuntu) you can tell Wireshark to use this through

edit -> preferences -> protocols -> SSL -> (pre)-master-secret log filename

Different versions of Wireshark have TLS or SSL, use whichever one is available to you.

Specify the name of your file (/tmp/kl in my case) and Wireshark will be able to decrypt the data.

Setting up certificate authentication in LDAP.

This started off as part of a small task, when I had half an hour gap before lunch. The whole end-to-end of getting TLS and LDAP, with certificate authentication took me several weeks to set up. Now I know the traps, it takes about 10 minutes.

I describe setting up TLS and LDAP (without certificate authentication) here. Get that working before trying certificate authentication.

Setting up the simplest case of an RSA certificate on the client and an RSA certificate on the server, was pretty easy to set up. Using an Elliptic Curve certificate to and RSA certificate on the server seems impossible, it eventually worked!

I created “What cipher specs should I use?” because most of my problems, were due to using the wrong cipher specs, or the right cipher specs, but it the wrong order!

Logging on

You can logon to LDAP and specify a userid (DN) and password, for example

ldapsearch -h -D “cn=Admin, o=Your Company” -w secret -b “o=Your Company” “(objectclass=*)” aclEntry

If you use -w ? it will prompt for your password, so it is not visible.

You can also use a certificate to logon, so you do not need the password, you just need the private key (or in my case the USB dongle with my encrypted Hardware Security Module(HSM) keystore on it).

Note: If you have your TLS private key in a file, and people can copy that file, they can impersonate you! You need to protect the file, bearing in mind your corporate IT department may be able to view any backups etc that you have. Someone would need to steal my USB dongle to use my private key and logon.

Understanding the TLS 1.2 handshake and authentication.

Skip to first steps if you are keen to implement without understanding the background,

There are several stages to establishing a TLS connection and authentication.

  1. Agree the protocols for setting up the session, for example which sort of encryption, and the key size. This provides the privacy on the connection.
  2. The server sends down its certificate, and the client authenticates it
  3. The client sends up its certificate and the server authenticates it
  4. The Distinguished Name(DN)from the client certificate is looked up in the z/OS security manager, and the associated userid is looked up.
  5. The DN is used to look in the access control lists (ACLs) do determine the access the requester has to the data.

Agree the protocols

The client the sends the protocols it supports to the server. This is a list of numbers, and each number has a meaning.


TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)

The part before the WITH covers

  1. The technique used to agree a key (ECDHE, DHE,RSA)
  2. The key type Elliptic Curve (ECDSA) or RSA

There must be at least one key type matching the certificate type.

  • If you have an Elliptic Curve server certificate, then you need to have records with TLS…ECDSA…
  • If you have an RSA server certificate you need to have records with TLS…RSA…

At the server, the list sent from the client is merged with the list from the server, for example if the server had


and the client sent C024, C025, C02C, then common elements are C024, C025. These two cipher specs are TLS_ECDHE_ECDSA_WITH… so the server’s certificate needs to be ECDSA – or an Elliptic Certificate.

If no entries match you get a message like “no cipher found”.

Authenticate the server

The server sends information down to the client.

  1. The cipher suite to be used.
  2. Information for setting up the encryption of the traffic for the session.
  3. The server’s certificate, and any CA certificates in the chain.
  4. The client then validates the certificate
    1. It use the signature algorithm in the certificate. This is the information after the “WITH” in the cipher spec. In my client’s certificate it has signing algorithm sha256WithRSAEncryption. There must be a cipher spec in the list ending in TLS…RSA_WITH… ending in SHA256
    2. It checks the signature of the CA, by checking its signing algorithm in the GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) See here.

Send the client certificate to the server

If the client certificate is wanted, the servers sends the “certificate request”.

  1. It sends down the certificate type it accepts, for example RSA, ECDSA, DSS. The client then sends its certificate of that type. If the there is no match, no certificate is sent. Note:For a long time I could not get a client certificate with Elliptic Curve, to work when the server had an RSA certificate. This was due to bad parameters on the z/OS end.
  2. There is a list of signature Hash Algorithms (eg rsa_pkcs1_sha256 (0x0401) ecdsa_secp256r1_sha256 ( 0x0403).
  3. It lists the Distinguished Names of the Certificate Authorities or Self Signed certificate.
  4. With the certificate type,the signature Hash and the Certificate Authorities, the client looks in its keystore for a certificate matching these parameters; then sends the first certificate that matches – or sends “no certificate”.
  5. It sends the Certificate Verify Signature Algorithm to the server. For my Elliptic Curve this was ecdsa_secp256r1_sha256 (0x0403). This has to be in the GSK_TLS_SIG_ALG_PAIRS in the z/OS LDAP environment file. (It took me days to find this problem!)

The server authenticates the client’s certificate

The server looks inside the certificate to see how the certificate was signed, for example using

openssl x509 -in ecec.pem -text -noout

This gave Signature Algorithm: ecdsa-with-SHA256. The server needs the matching cipher spec in the GSK_V3_CIPHER_SPECS_EXPANDED environment variable.

The client’s certificate will have been signed – either by a CA, or self signed. The Signature algorithm of the CA must be in the GSK_TLS_SIG_ALG_PAIRS environment variable. The values are defined here.

For example my CA had Signature Algorithm: ecdsa-with-SHA384. This requires 0503 (SHA-384 with ECDSA) to be in the GSK_TLS_SIG_ALG_PAIRS list. If this was missing I got

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

from ldapsearch on Ubuntu. It took me another day or so to find this problem.

As you can see there are a lot of things you need to configure and to get right before it works!

First steps – using a client certificate

Firstly set up the TLS session so you can use certificates to connect to LDAP. This took me about a week, because of configuration problems, but finally it worked. I was able to connect from Ubuntu to z/OS.

Set up TLS between the client and the z/OS server, as described here.

Setup Ubuntu

I created an RSA certificate on Ubuntu using a shell script.


openssl genpkey -out $name.key.pem -algorithm RSA -pkeyopt rsa_keygen_bits:2048

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name -passin file:password.file -passout file:password.file
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

I set up an Elliptic Curve

enddate=”-enddate 20220409174001Z”

password=” -passin file:password.file -passout file:password.file”
rm $name.key.pem
rm $name.csr
rm $name.pem
ca=”ca256″ # sign with this

openssl ecparam -name secp256r1 -genkey -noout -out $name.key.pem

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name $password
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

openssl x509 -in $name.pem -text -noout|less

# Some tools need a .p12 file of the certificate
#openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $ca.pem -chain -name $name -passout file:password.file -passin file:password.file

Openldap uses a configuration file, for example ldaprc.

I set up my ldaprc configuration file on Ubuntu.

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://
TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem
TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem


  • TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem was the z/OS CA certificate exported from z/OS and downloaded. It is needed to verify a certificate sent from z/OS.
  • TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem this file contains the private key
  • TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem this file contains the public key
  • You can specify TLS_CIPHER_SUITE high to use the listed cipher specs. See the openssl ciphers command, for example
    • openssl ciphers -v -V -s -tls1_2
    • openssl ciphers -v -V -s -tls1_3
    • I omitted TLS_CIPHER_SUITE for my testing.

You need the distinguished name (DN) of the certificate. You can display the subject distinguished name DN using the command openssl x509 -in secp521r.pem -text -noout. This gave me output which included:

Subject: C = GB, O = cpwebuser, CN = secp521r

I sent the Linux CA file, ca256.pem, to z/OS.

Set up z/OS

Configure LDAP.

In my LDAP configuration file I had

sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING
sslCertificate ZZZZ
commThreads 10
allowAnonymousBinds off

The sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED says look in the environment file, and take the value from there. As this is the default, you can omit it from the file.

The default GSK_V3_CIPHER_SPECS_EXPANDED value in the environment file is 003500380039002F00320033 which maps to

  1. 0035 TLS_RSA_WITH_AES_256_CBC_SHA
  4. 002f TLS_RSA_WITH_AES_128_CBC_SHA

Which are all very old (part of deprecated SSL, TLS 1.0, TLS 1.1) so you need to set this variable in your environment file. See the next section for what I used.

Update your environment file

I use the following value


This supports

  7. 1303 TLS_CHACHA20_POLY1305_SHA256 TLS 1.3
  8. 1301 TLS_AES_128_GCM_SHA256 TLS 1.3
  9. 1302 TLS_AES_256_GCM_SHA384 TLS 1.3

Put the strongest certificates first. For example, the following did not work.


It caused 0035 to be used. The Certificate-Types list had only RSA and DSS, and did not include ECDSA, so the client was not able to send its Elliptic Curve certificate.

When I used


it caused C02F to be used. This caused the certificate-types list to include the ECDSA, and so the client could send its Elliptic Curve certificate. This problem took me a week to find!

I specified


For the certificate checking algorithms. These are for SHA = 224, 256, 384, 512 and certificate types = RSA, ECDSA, and DSA.

Add the CA from Linux and add it to the keyring

The Linux CA was uploaded to z/OS as COLIN.CARSA102.PEM.

Add it to the RACF keystore as a CERTAUTH certificate, and connect it to the keyring.






Map the Distinguished Name to a z/OS userid.

You have to map the DN to a userid. In the example below, it maps the DN in the SDNFILTER to the userid ADCDA. This userid must be valid, for example if the userid is revoked, the authentication will fail (with an unhelpful message).

* remove any previous instance


   SDNFILTER('CN=secp521r.O=cpwebuser.C=GB')  - 



Note the ‘.’ between the attributes, and the attributes are squashed up, in the right order.

I did not need to refresh LDAP as the change was automatically picked up.

I could now logon to LDAP using the certificate – but could do nothing.

Give the DN access to LDAP

I gave it access, using ldapmodify passing the file with

dn: o=Your Company
changetype: modify
add: aclEntry
aclEntry : access-id:CN=SECP521R,O=CPWEBUSER,C=GB:

Note: The DN can be in mixed case, or upper case.

I could now logon and issue queries using a certificate for authentication and authorisation.

Tracing problems

You can get trace output to the job log using the command


This traces

  • any errors
  • CONNS shows the cipher spec being used
  • LDAPBE shows the DN, and SAF user etc being used.

You can turn trace off with

f GLDSRV,debug 0

You can get the low level trace by using the gsktrace=0xff in your environment file.

Using z/OS LDAP with TLS

This started as a mini project before lunch, and now, two weeks later, I have got all of the bits to work.

As I was struggling with this, I wondered why no one had crossed this swamp before me. Is it because no one wants to cross it, or it is too difficult to do so? I think it was just difficult to do so – but I can now provide a map!

It took me a couple of hours to get LDAP on z/OS to use TLS from my Linux machine. The challenge was finding the right options. I then tried using Elliptical Certificates, and got this working, then was partially successful in using TLS 1.3.

As part of this I wrote Authentication using certificates with LDAP, which is where it was really difficult.

For clients, I used ldapsearch (from openldap ) on Ubuntu, and ldapsearch on z/OS.

See the product documentation here.

I’ve put a section “What cipher specs should I use” in a separate blog post.

Configure the LDAP environment file

I had to add TLS V1_2 and TLSV1_3 (and disable the deprecated TLS 1.1)


to the environment file, because they default to off. Without these options it defaults to SSL – which is so old, my certificates and keys did not support it. I disabled TLS_1_1 because it is old.

I added


This is used when validating certificates. The signature algorithm of the CA must be in this list see GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) (see here), get things working, then remove the ones I did not want to use; rather than try to add the ones I wanted to use.

I initially added


This is a list of the cipher specs it will accept. You should try to restrict the certificates in your organisation, to Elliptical Certificates, and a limited configuration for any CA’s. For example have

  • all client’s certificates defined as Elliptic Curve type prime256v1, signed with ecdsa-with-SHA256.
  • All internal CAs defined as Elliptic Curve type prime256v1 signed with ecdsa-with-SHA256.

When using certificate authentication I needed the following for it all to work


The environment variables


these are used only when using TLS 1.3. I’ll blog about using TLS 1.3 at a later date.

While you are investigating problems you can use the environment variable GSK_TRACE=0xff to write a trace. You can specify GSK_TRACE_FILE to direct the trace to a particular file. The default location is /tmp/gskssl.%.trc where % is the thread id. You format it with

gsktrace /tmp/gskssl.65567.trc > out

oedit out

I used


So I could have a shell script to format and display the trace.

gsktrace gsktrace.ldap > out
oedit out

Change the LDAP configuration file.

I used

sslCipherSpecs all
sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING

I specified a port for ldaps. You can use

ldapsearch -H ldaps://

This is preferred to specifying host and port

ldapsearch -h -p 389

A port can process TLS and non-TLS requests, so definition for port 1389 may not needed.

I specified a keyring sslKeyRingFile START1/MQRING. This is owned by a different userid to the started task, so needs more RACF definitions. See below for the definitions.

Define the keyring and keyring

I defined an RSA certificate and an EC certificate. I controlled which was used from the configuration file.

* create the ring


* and the server's RSA certificate

             O('SERVER') - 
             OU('SSS')) - 
   RSA - 

                            ID(START1)  - 
                            LABEL('ZZZZ') DEFAULT) 

* and the Elliptic Certificate

             O('ADCD') - 
             OU('TEST')) - 
   SIZE(521) - 
   NOTAFTER(   DATE(2024-12-29))- 
                            ID(START1)  - 
                            LABEL('SERVEREC') DEFAULT) 

Permit the LDAP server’s userid to access the private certificate in the keyring

Because the LDAP userid does not own the keyring, the server userid needs update access to be able to extract the private key.


For a userid to be able to use its own keyring, it only need read access to be able to use the private certificate.

Set up the client userid

I set up the userid in the z/OS LDAP server, so I could use the z/OS userid and password. See here.

Test it out

LDAP checks the TLS parameters, when a TLS request arrives. It does not check at startup . I used openssl s_client to connect, and display information about the TLS handshake. See Debugging TLS – an easier way.

Setting up the client on Ubuntu

I set up a file ldaprc containing the ldap configuration. You can enter all of the parameters on the command line, but the ldaprc file makes it easier. My file had

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://
#TLS_CERT /home/colinpaice/ssl/ssl2/rsaca256.pem
#TLS_KEY /home/colinpaice/ssl/ssl2/rsaca256.key.pem

The #TLSCERT etc are ignored. No certificate is sent to the server (unless the -Y external option is set).

Enable GSK trace on z/OS

I enabled an LDAP trace to //SYSPRINT on z/OS, using f GLDSRV,debug error+CONNS+LDAPBE


  • error reports errors
  • Conns reports connection information, such as the cipher spec used.
  • LDAPBE reports on the communication to the back end, for example it reports the SAFUSER

Run a query on Linux

I used ldapsearch on Ubuntu

ldapsearch -H ldaps:// -D “cn=ibmuser, o=Your Company” -w panthe0n -b “o=Your Company” “(objectclass=*)” aclEntry

The trace on z/OS GLDSRV included

CONNS srv_ssl_connect()644: SSL Cipher Specs
00000000: c3f0f0f8 *C008 *

and we can see that SSL was used, and cipher spec C008 was used.

Because I had set up my userid to map use a SAF userid, in the trace I had

LDAPBE srv_process_bind_request()939: do_return_bind msgID=1, connID=8, bindDN=‘CN=IBMUSER, O=YOUR COMPANY’, safUserID=’IBMUSER’, dnList=0x0, grpList=0x0, rc=0

Debugging TLS – an easier way

I had been trying to get an IBM product on Linux to talk to LDAP using TLS and certificate authentication; where I give a certificate instead of a LDAP userid and password. Basically I got “computer say no”.

IBM products use GSKIT to manage keystores and TLS between sessions. With Java you can get out a trace of the TLS conversation. You can get out a trace on z/OS, but not on other platforms. After a day I stumbled on a different approach.

Gettting a gskit trace on z/OS.

Getting the trace was easy, understanding it was harder.

In the LDAP environment file I added GSK_TRACE=0xff. Once I had TLS to LDAP working, I used GSK_TRACE=0x04 to trace just errors. Some “errors” are recorded in the GSKTRACE as “INFO”. For example an Elliptic curve is only supported in TLS 1.3.

By default the trace goes to /tmp/gskssl.%.trc. where % is the thread id.

To format it go into USS and use the command gsktrace gskssl.83951642.trc > out, and edit the file “out”.

The linux end.

I found a great program openssl s_client. This starts a TLS handshake and prints out in easy to understand format, what is going on. For example

The command

openssl s_client -connect -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -x509_strict

gave the flow of the data.

stderr output

The stderr output included

depth=1 O = COLIN, OU = TEST, CN = COLIN4Certification Authority
verify error:num=19:self signed certificate in certificate chain

This is the name of the CA.

Following this was an error message. Note: it is at the top of the output – not the bottom as I expected.

SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1528:SSL alert number 80

See below for a discussion about the error.

stdout output

Certificate chain
  0 s:O = SERVER, OU = SSS, CN = ZZZZ
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority
  1 s:O = COLIN, OU = TEST, CN = COLIN4Certification Authority 
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority 

This says

  • the certificate SERVER was signed by COLIN4.Certificate Authority
  • the certificate COLIN4Certificate Authority was self signed

The server certificate comes next


Followed by the identity of the server

subject=O = SERVER, OU = SSS, CN = ZZZZ
issuer=O = COLIN, OU = TEST, CN = COLIN4Certification Authority

Next comes the list of Certificate Authority or self signed certificate in the keyring

Acceptable client certificate CA names
O = COLIN, OU = TEST, CN = COLIN4Certification Authority
C = GB, O = SSS, OU = CA, CN = SSCARSA1024
C = GB, O = SSS, OU = CA, CN = SSCA256

Any client certificate must be signed by one of those CAs (or self signed certificates).

Then comes the client certificate types the server will accept

  • Client Certificate Types: RSA sign, DSA sign

Note: This does not include ECDSA, so elliptic certificates are not supported in this configuration.

After this is the signature algorithms the server will accept

  • Requested Signature Algorithms: RSA+SHA512: ECDSA+SHA512: RSA+SHA384: ECDSA+SHA384: RSA+SHA256: ECDSA+SHA256: DSA+SHA256: RSA+SHA224: ECDSA+SHA224: DSA+SHA224: RSA+SHA1: ECDSA+SHA1: DSA+SHA1

It gives information on the session so far

SSL handshake has read 2493 bytes and written 1537 bytes
Verification error: self signed certificate in certificate chain

and information about the key sent down. It was defined with SIZE(4096) RSA.

New, SSLv3, Cipher is AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated

and about the TLS session

    Protocol  : TLSv1.2
    Cipher    : AES256-SHA
    Session-ID: 0401001A0A0100029...
    Master-Key: 32D5B4AD162F0403E323DB0...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1634115022
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no

The -msg option gave me

>>> ??? [length 0005]
    16 03 01 01 31
>>> TLS 1.3, Handshake [length 0131], ClientHello
<<< ??? [length 0005]
<<< TLS 1.3, Handshake [length 0051], ServerHello
<<< TLS 1.2, Handshake [length 0839], Certificate
<<< TLS 1.2, Handshake [length 0123], CertificateRequest
<<< TLS 1.2, Handshake [length 0004], ServerHelloDone
>>> ??? [length 0005]
>>> TLS 1.2, Handshake [length 021c], Certificate
>>> ??? [length 0005]
>>> TLS 1.2, Handshake [length 0206], ClientKeyExchange
>>> ??? [length 0005]
>>> TLS 1.2, Handshake [length 0050], CertificateVerify
>>> ??? [length 0005]
>>> TLS 1.2, ChangeCipherSpec [length 0001]
>>> ??? [length 0005]
>>> TLS 1.2, Handshake [length 0010], Finished
<<< ??? [length 0005]
<<< TLS 1.2, Alert [length 0002], fatal internal_error
    02 50

The fatal errors are described here. 0x02 is fatal error, 0x50= 80 = internal error. internal_error(80). Which is not very helpful.

In the LDAP log I got

GLD1116E Unable to initialize an SSL connection with 434 – Certificate key is not compatible with cipher suite.

In the gsktrace on z/OS I got

cms_validate_certificate_mode_int(): Validating CN=ecec,O=cpwebuser,C=GB ENTRY
INFO get_issuer_certificate(): Using issuer CN=SSCA256,OU=CA,O=SSS,C=GB

ERROR read_v3_certificate(): Client certificate key type 13 not allowed for SSL V3 cipher 0x0035

  • Client certificate key type 13. Looking in /usr/include/gskcms.h under X.509 data types, was x509_alg_ecPublicKey.
  • SSL V3 cipher 0x0035 is listed here, as TLS_RSA_WITH_AES_256_CBC_SHA = 256-bit AES encryption with SHA-1 message authentication and RSA key exchange.
  • x509_alg_ecPublicKey is elliptic public key – which is incompatible with RSA key exchange.

When I used an RSA certificate

openssl s_client -connect -cert /home/colinpaice/ssl/ssl2/rsaca256.pem -key /home/colinpaice/ssl/ssl2/rsaca256.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -verify_return_error -policy_print -x509_strict

it worked