Being too strong can be a waste of time

The short story is to use Elliptic curves with size 256 or 384, and avoid wasting time investigating why using other sizes doesn’t always work.

The long story – part 1 the server

I had a RACF server certificate defined as

SIZE(521) NISTECC …

and using Chrome browser it failed. I did a lot of digging around and could see the gsktrace had

EXIT gsk_get_ec_parameters_info(): <— Exit status 0x00000000 (0) EC curve type 34, key size 521
ERROR send_v3_alert(): Sent SSL V3 alert 40 to 10.1.0.2[38736]

INFO edit_ciphers(): Server certificate ec curve 0034 not in supported ecurve tls extension. EC cipher suites disabled

Where curve type 34, also known as 0x0019 = secp521r.

When I changed the size to 256 it worked.

Digging in to this, the Client Hello part of the TLS handshake sent from Chrome had

Supported Groups (4 groups)
Supported Group: Reserved (GREASE) (0x2a2a)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)

This is missing the 0x0019.

For Firefox, which worked, it had

Supported Groups (6 groups)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)
Supported Group: secp521r1 (0x0019)
Supported Group: ffdhe2048 (0x0100)
Supported Group: ffdhe3072 (0x0101)

So after a couple of hours working on this ( and an overnight sleep-on-it) the solution was to use secp256r1 instead of secp521.

The long story – part 2 – the client

I also tried using a client certificate which was defined as a secp521r1. This has similar problems. When I defined it as secp256r1 or secp384r1 it worked.

The gsktrace included

INFO read_v3_certificate(): Certificate key algorithm 13, Signature algorithm 84
ENTRY gsk_get_ec_parameters_info(): —> keyInfo size 12
EXIT gsk_get_ec_parameters_info(): <— Exit status 0x00000000 (0) EC curve type 34, key size 521
ERROR read_v3_certificate(): Client certificate elliptic curve not in clients supported elliptic curve list

I also know more about the TLS hand shake – which has more twists and turns every time I look at it!

Certificate validation on LDAP using OCSP

It is good practice to validate certificates when security is important. There are typically two ways of doing this.

  • Have a central server with a list of all revoked certificates. This server has a Certificate Revocation List(CRL) which has to be maintained. This solution tends to be deprecated
  • When a certificate is created add an extension with “check using this URL… to check if the certificate is valid”. This field is added when the certificate is signed by the Certificate Authority. A request is sent to the OCSP server, and a response sent back. This technique is known as Online Certificate Status Protocol (OCSP). You need an OCSP server for each certificate authority.

The blog post explains about OCSP and some of the challenges in using it.

You can configure a certificate to have OCSP support.

When you create a signed certificate, the CA signs the certificate.

For example

openssl req -config eccert.config … -out $name.csr
openssl ca -config openssl-ca-user.cnf -policy signing_policy -in $name.csr -extensions xxxx

In the config file are different sections, in my xxx section is

[ xxxx ]
authorityInfoAccess = OCSP;URI:http://10.1.0.2:2000

If the server (LDAP in this case) is configured for OCSP, then when it sees the AIA extension, it sends a request to the OCSP server at the URI (10.1.0.2:2000) which responds saying “good” , “revoked” or “unknown”.

OCSP server.

Openssl provides a simple ocsp server. You give it

  • the port it is to use
  • the CA public certificate (so the incoming certificate can be validated against the CA)
  • a file for that CA of the certificates it has issued, and it they are valid or not.

You can also configure the LDAP server to give a default URI, for those certificates that do not have the Authority Info Access(AIA) section. Typically this only works if you have one Certificate Authority, or you have a smart OCSP server which can handle the data for multiple CAs.

For example if your openssl OSCP server is configured for CA with DN:CN=CA1,o=MYORG, and you send down a request for DN:CN=CA2,o=myorg, it will not recognize it.

If you use the AIA extension, then you can have two different OCSP servers, one for each CA.

How does it work?

There are two ways of doing OCSP checking.

  • The client looks at the certificate and sees there is an OCSP extension. The client sends a request to the OCSP server to check the certificate is valid.
  • As part of the TLS handshake send the extension to the server to say “please do OCSP checking for me”. The server issues the request to the OCSP server, and can cache the response in the server. The server also sends the OCSP status to the client as part of the TLS handshake. The next request for the certificate can use the cached response. It tends to improve performance, as it reduces the number of requests to the OCSP server, and the responses are cached in the server. This is known as OCSP stapling.

Does using OCSP kill performance?

Yes and no. I’ll cover this after “caching the response”

Caching the responses

The OCSP server may be within your organization, or it may be be external. The time to send a request and get the response back may range from milliseconds to seconds.

Servers can typically cache the response to an OCSP query.

The OCSP may update it’s certificate list every hour, or perhaps every day, so if it is refreshed once a day, your LDAP server, does not need to refresh its data more than once a day.

The openssl OCSP server has an option -nmin minutes| -ndays days which is how often to reread the file. It sends a response

SingleResponse
  certID
    hashAlgorithm (SHA-1)
    issuerNameHash: 157b5dd0bcdee5b5428e063cf29a1f4e45be7499
    issuerKeyHash: 5830af55c7b4d49fc9e3fac91441ef1fd7b215e3
    serialNumber: 587
  certStatus: good (0)
  thisUpdate: 2021-11-05 10:42:58 (UTC)
  nextUpdate: 2021-11-05 10:43:58 (UTC)

From this the requester knows when the cached value is no longer valid, and needs to contact the OCSP server again for that certificate. From this we can see that the response is valid for 60 seconds.

When the certificate was revoked the output was

Online Certificate Status Protocol
  responseStatus: successful (0)
  responseBytes
    ResponseType Id: 1.3.6.1.5.5.7.48.1.1 (id-pkix-ocsp-basic)
    BasicOCSPResponse
      tbsResponseData
        responderID: byName (1)
          byName: 0...
          producedAt: 2021-11-06 15:15:58 (UTC)
            responses: 1 item
              SingleResponse
                certID...
                  certStatus: revoked (1)
                    revoked
                      revocationTime: 2021-11-06 15:12:30 (UTC)
                      revocationReason: cessationOfOperation (5)
                    thisUpdate: 2021-11-06 15:15:58 (UTC)
                    nextUpdate: 2021-11-06 15:16:58 (UTC)
            responseExtensions: 1 item
              Extension
                Id: 1.3.6.1.5.5.7.48.1.2 (id-pkix-ocsp-nonce)
                ReOcspNonce: a7e59577b7d2b3a2
      signatureAlgorithm (ecdsa-with-SHA256)
        Padding: 0
        signature: 3046022100fb264d4c5dbbbf45cce752a8263c4f01631441...
      certs: 1 item...

This information is available to the client.

What about performance?

If you write your application properly the impact of OCSP can be minimized.

For example consider the scenario where you are using REST requests, and go to one of a number of server.

  • Your application starts
  • It needs to check the validity of the certificate. Send a request to the OCSP server. This could take a long time (seconds). Some applications have a time-out of 15 seconds waiting for a response.
  • The response comes back saying “certificate Good – and valid for 1 hour.
  • Send a request to the server, and get the response back
  • Issue another request (with another TLS handshake)
  • After 1 hour – resend the request to the OCSP server to re-validate.
  • Etc

OCSP stapling

OCSP stapling is very common now. Before this, clients themselves used to check the validity of the user’s certificate by contacting the OCSP server. With OCSP stapling, a request is put into the TLS handshake which says “please do the OCSP checks for me and send me the output”. This allows the server to cache the information.

Think of the old days of checking in at the airport, when the person checking you in, would staple “checked-in by agent 46” to your paper ticket.

The client requests this by adding the “status_request” extension to the TLS clientHello handshake.

The server sends down, as part of the “serverHello” the information it received from the OCSP server.

Note. ldapsearch, from openssl, sends up the status_request, but does not handle the response, I get

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1) additional info: (unknown error code)

openssl s_client does not send the status_request extension, so does not participate in the OCSP checking.

Java does support this, and a Java application can get the OCSP response message using the getStatusResponses method on the from the ExtendedSSLSession. I believe you can decode it using Bouncycastle.

Setting up LDAP on z/OS to support OCSP

I added the following to a working LDAP system

Environment

GSK_OCSP_URL=http://10.1.0.2:2000
GSK_OCSP_ENABLE=ON
GSK_OCSP_CLIENT_CACHE_SIZE=100
GSK_REVOCATION_SECURITY_LEVEL=MEDIUM
GSK_SERVER_OCSP_STAPLING=OFF
GSK_OCSP_RESPONSE_SIGALG_PAIRS=0603060105030501040304020401

#GSK_OCSP_RESPONSE_SIGALG_PAIRS=0806080508040603060105030501040304020401

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Stapling

I set GSK_SERVER_OCSP_STAPLING=OFF because ldapsearch on Ubunutu did not work with the ENDENTITY value.

Signature Algorithms

If GSK_OCSP_RESPONSE_SIGALG_PAIRS included any of 0806 0805 0804, I got messages

GLD1160E Unable to initialize the LDAP client SSL support: Error 113, Reason -99.
GLD1063E Unable to initialize the SSL environment: 466 – Signature algorithm pair is not valid.

In the trace I had 03353003 Cryptographic algorithm is not supported, despite these being listed in the documentation.

Nonce

The nonce is used to reduce replay attacks. In your request to the OCSP serve you include a nonce (string of data). You expect this in the response message.

The default in LDAP is off!

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Configuration

In the configuration file I had

sslKeyRingFile START1/MQRING

sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

Setting up the OCSP server on Linux

My CA was on Linux – address 10.1.0.2.

I set up a bash script to run the openssl OCSP server on Linx,

ca=”-CA ca256.pem”
index=”-index index.txt”
port=”-port 2000″
#rsigner=”-rsigner rsaca256.pem –rkey rsaca256.key.pem”
#rsigner=”-rsigner ecec.pem –rkey ecec.key.pem”
rsigner=”-rsigner ss.pem –rkey ss.key.pem”
nextUpdate=”-nmin 1″
openssl ocsp $index $ca $port $rsigner $nextUpdate

You need

  • CA … for the Certificate Authority .pem file
  • -index index.txt, for the status of the certificates issued by the CA
  • -port … a port to use
  • -rsigner… the public key to be used when responding
  • -rkey … the private key for encrypting the response.
  • -nmin … how often the index file is refreshed. Typically this value might be an hour or more.

It does not log any activity, so I had to use Wireshark to trace the network traffic.

Problems using a certificate signed by the CA, for encrypting the response.

The rsigner certificate needs to have

Extended Key Usage: critical, OCSP Signing

Without this I got the following in the gsktrace

ERROR check_ocsp_signer_extensions(): extended keyUsage does not allow OCSP Signing

A self signed certificate worked OK.

Testing it

I had several challenges when testing it

  • ldapsearch on Linux sends up the “I support OCSP stapling”, but it objects to the response, and ends with unknown error code.
  • openssl s_client does not send the OCSP flag, and so the certificate does not get validated.
  • Java worked. I used this as a basis, and made a few changes to reflect my system. I needed to use the following optuons to run it
    • -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ecec.p12
    • -Djavax.net.ssl.keyStorePassword=password
    • -Djavax.net.ssl.keyStoreType=pkcs12
    • -Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/dantrust.p12
    • -Djavax.net.ssl.trustStorePassword=password
    • -Djavax.net.ssl.trustStoreType=pkcs12

Once I change GSK_SERVER_OCSP_STAPLING=ENDENTITY to GSK_SERVER_OCSP_STAPLING=OFF, I was able to use LDAPSEARCH.

Some OCSP certificates didn’t work

In my OCSP server, I used a certificate signed by the CA, for encrypting the response back to LDAP.

In the GSKtrace I got

ERROR find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
ERROR crypto_ec_token_public_key_verify(): ICSF service failure: CSFPPKV retCode = 0x4, rsnCode = 0x2af8
ERROR crypto_ec_token_public_key_verify(): Signature failed verification
ERROR crypto_verify_data_signature(): crypto_ec_verify_data_signature() failed: Error 0x03353004

There are two connected problems here

  1. Find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
  2. retcode 0x2af8 (11000) The digital signature verify ICSF callable service completed successfully but the supplied digital signature failed verification.8 (11000). I did not have the correct CA certificate for the OCSP certificate in the LDAP keyring.

Setting up CRL on z/OS LDAP

Certificate Revocation Lists (CRLs) are an old (and deprecated) way of checking the validity of certificates. I thought this would be a good way of understanding LDAP on z/OS.

As usual this had some surprises for me,such as having to recreate my CA, and certificates that it had issued, because it was missing the crlSign attribute.

The MQ documentation was a good start (I could not find any other documentation).

How do you know if your certificate is valid?

  • The old way of doing this (from about 2008!) was to have a list containing the Certificate Authority’s certificate, and a list of the certificate serial numbers which were revoked. This was called the Certificate Revocation List(CRL). In some cases this might just be a file, but typically this was stored in LDAP. Periodically you would refresh this list.
  • If may be that an intermediate CA was revoked. As well as a Certificate Revocation List you had Authority Revocation List(ARL) (from Certificate Authority).
  • People realised that refreshing the CRL every day was leaving a certificate exposed for up to a day. Also this file could get very large.
  • These days the checks are done “online”, using an Online Certificate Status Protocol(OCSP). A certificate has an extension with information “ask this site… if I am still valid”.
  • This has been extended to support the server caching the response to the OCSP request, and sending the OCSP status down to the client so the client knows how long the status of the certificate is valid for. Eventually the OCSP status expires, and the OCSP status has to be renewed.

Revoking a certificate

I wanted to revoke a certificate that I had created on Linux, so it could not be used for authentication.

This was pretty painless, except for a couple of “ahh” moments.

I set up a bash script to revoke one of my client’s certificates, and create the file of revoked certificates.

password=” -passin file:password.file -passout file:password.file”
name=”ecec”
ca=”ca256″ # sign with this
config=”-config openssl-ca-user.cnf”
policy=”-policy signing_policy”
reason=”-crl_reason cessationOfOperation”

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem $reason –revoke $name.pem

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem –gencrl -out crl.pem

This revokes the ecec.pem file, which was signed by ca256.*. I picked a reason cessationOfOperation from the list of reasons.

The index.txt file for the CA, shows the certificate is revoked “R”

R 200229103805Z 211102120448Z, cessationOfOperation 492DF59CE0479E9523C9355077E1DD6F83C4B01E unknown /C=GB/O=cpwebuser/CN=ecec

The second command (–gencrl) produced a file crl.pem (base 64 encoded) with a list of revoked certificates:

—–BEGIN X509 CRL—–
MIIBIzCByQIBATAKBggqhk…
3vs3Bcdkrw==
—–END X509 CRL—–

You can display the file contents using

openssl crl -text -in crl.pem

For example

Certificate Revocation List (CRL):
    Version 2 (0x1)
    Signature Algorithm: ecdsa-with-SHA256
    Issuer: C = GB, O = SSS, OU = CA, CN = SSCA256
    Last Update: Nov 2 15:33:19 2021 GMT
    Next Update: Dec 2 15:33:19 2021 GMT
Revoked Certificates:
  Serial Number: 01
    Revocation Date: Jun 19 10:46:32 2019 GMT
  Serial Number: 023F
    Revocation Date: Nov 2 11:55:32 2021 GMT
  Serial Number: 0240
    Revocation Date: Nov 2 15:33:10 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Serial Number: 492DF59CE0479E9523C9355077E1DD6F83C4B01E
    Revocation Date: Nov 2 12:04:48 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Signature Algorithm: ecdsa-with-SHA256
    30:44:02:20:21:f4:70:11:42:85:22:c8:41:b9:95:d2:3d:9e:
    0a:b4:69:fa:a4:e0:37:81:05:c7:e8:98:43:a1:dc:67:58:6c:
    02:20:14:33:bd:2e:e1:57:3b:76:22:57:b9:b2:c4:a7:29:a3:
    3c:b1:4e:5a:a3:13:bc:13:61:e2:9e:a5:7c:64:b1:b1

I ftped the crl.pem file to z/OS.

Revoking a certificate on RACF

RACF does not have the support for revoking certificates. You are meant to use z/OS® Cryptographic Services PKI Services, to manage your certificates – which looks a non trivial installation.

Configure LDAP.

The CRL information is stored under the DN of the CA. My CA is called

C = GB, O = SSS, OU = CA, CN = SSCA256

So I need an entry in the LDAP tree for cn=SSCA256, ou=CA, o=SSS, c=GB.

My ldap configuration included

database LDBM GLDBLD31/GLDBLD64
useNativeAuth all
suffix “o=Your Company”
databaseDirectory /var/ldap/ldbm

the suffix means that all entries are below “o=Your Company” for example

dn: cn=ibmuser, o=Your Company

To support the CA, I added

suffix “c=GB” to the configuration file and restarted LDAP.

I created the elements for the CA name by using an .ldif file with.

dn: c=GB
objectclass: country
c: GB

dn: o=SSS, c=GB
o: SSS
objectclass: top
objectclass: organization

dn: ou=CA, o=SSS, c=GB
ou: CA

and the CA specific CRL information

dn: cn=SSCA256, ou=CA, o=SSS, c=GB
cn: SSCA256
objectclass: cRLDistributionPoint
certificateRevocationList;binary:< file:///crl1.ldif

This page says the z/OS LDAP client LDIF parser does not support the file:// URL format. 

I modified the ldif to be

dn: CN=SSCA256,OU=CA,O=SSS,C=GB 
cn: SSCA256 
objectclass: cRLDistributionPoint 
objectclass: certificationAuthority 
cacertificate:: 
 MIIB2DCCAX+gAwIBAgI
...
certificateRevocationList:: 
 MIIBRTCB7QIBATA
...

Where the data contents are indented one character.

I copied the contents .pem file from the CA, and the content of the CRL.pem file into the .ldif file.

I used ldapmodify… to update LDAP with the .ldif files.

I tried using the revoked certificate on Linux to issue an LDAPsearch – and it worked! (where it should have failed).

The gsktrace had

Checking revoked status CN=ecec,O=cpwebuser,C=GB

CRL cannot be located

More configuration on z/OS

The CRL processing works as follows

  • I use a certificate to connect to LDAP.
  • LDAP calls GSKit to verify the certificate, map it to a RACF userid etc.
  • If GSKIT has been configured for CRL checking, it issue a request to the CRL server to get the CRL for the CA. This could be my LDAP, or another LDAP somewhere else. It took a little while to work out, my client came in to LDAP, called GSKIT, which called the same LDAP!

Configure GSKIT

In the LDAP environment file I added

GSK_LDAP_SERVER=127.0.0.1
GSK_LDAP_PORT=389
GSK_LDAP_USER=cn=ldapid,o=Your Company
GSK_LDAP_PASSWORD=passwr0d

This does a query asking for the CA certificate, and the CRL data for my Certificate Authority’s DN.

This did not work, I got the message in the LDAP job log

GLD1116E Unable to initialize an SSL connection 8 – Certificate validation error.

In the GSK trace I had

ERROR check_crl_issuer_extensions(): crlSign bit is not set in KeyUsage
ERROR check_revoked(): Unable to verify CRL issuer extensions: Error 0x03353026

My Linux CA was not defined properly. I was missing cRLSign in my CA definition

keyUsage = keyCertSign, digitalSignature,cRLSign

I did the following

  • Recreated my CA certificate
  • Sent the updated CA to my backend systems.
  • Update the certificate on z/OS and added it to the keyrings
  • Recreated the user certificate on Linux
  • Tried the ldapsearch to z/OS

To my pleasant surprise this failed to work – as I hoped.

My Ubuntu client had

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

and there was a message in the LDAP joblog

GLD1116E Unable to initialize an SSL connection with 10.1.0.2: 431 – Certificate is revoked.

In my gsktrace file I had

ERROR check_revoked(): RFC 2459 Revoked certificate check failed: Error 0x03353041

and from 03353041 you get

03353041 Certificate is revoked.

Explanation: A certificate is revoked and cannot be used.
User response: Obtain a new certificate from the certification authority.

You have not finished yet!

Now you know the process, of getting a CRL from the CA, and updating LDAP with it you need to consider some more things.

  • You need automation to generate the CRL, and update LDAP.
  • You may want multiple LDAP servers for your organisation.
  • You should consider a DN userid for your GSK, which can only access the CA information, and no other, so even if people saw the userid and password, they could not do much with it.
  • You need to protect the data, so the default access is none, but your GSK DN can read it, and you automatic process can update it.

Hindsight.

Since I got it working, I realised that I should have a better name for my CAs, such as O= SSS, OU=CA, c=GB, CN = SSCA256 instead of C = GB, O = SSS, OU = CA, CN = SSCA256.

It is worth going through the whole processed to find out the problems, so you minimise the amount of rework. For example missing keyUsage = cRLSign, caused a lot of rework (several times till I got it right).

Using z/OS LDAP with TLS 1.3

This blog post started as one line in the Using z/OS LDAP with TLS post. After it took two days to get working, I thought it deserved its own page.

Displaying the handshake

Using Wireshark to display TLS traffic is harder with TLS 1.3, as most of the handshake is now encrypted. See here on how to display it.

What supports LDAP and TLS 1.3?

I could not get Openldap on Ubuntu 18.04 to use TLS 1.3 – I do not think it has the support.

I could use the z/OS LDAPSEARCH with TLS 1.3, see below.

Java should work with the right set up.

Updating LDAP server on z/OS

Updates to enable TLS V1.3 protocol support for GSKIT has some good information.

You need to add the following to the configuration file

sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

# TLS 1.3 only supports sslFipsState off
slFipsState off

Below are the environment variables I used for the LDAP server on z/OS

GSK_TRACE=0xff
GSK_TRACE_FILE=/tmp/gsktrace.ldap
GSK_PROTOCOL_TLSV1_1=off
GSK_PROTOCOL_TLSV1_2=off
GSK_PROTOCOL_TLSV1_3=on
GSK_SERVER_TLS_KEY_SHARES=00300029002500240023
GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023130313011302
GSK_TLS_SIG_ALG_PAIRS=060105010401030108060805080405030403

I think this is the super set of data, and should be used as a starter list. You can remove unwanted cipher specs once you have it working. It is easier doing it this way compared to adding those you need (because it is hard to find what you need).

GSK_V3_CIPHER_SPECS_EXPANDED

You need to add the TLS 1.3 cipher specs to the GSK_V3_CIPHER_SPECS_EXPANDED in the environment file.

For example

GSK_V3_CIPHER_SPECS_EXPANDED=13011302C028C0271303

GSK_TLS_SIG_ALG_PAIRS

This variable needs to include 0806,0805,0804.

Key share

TLSv1.3 uses client and server key-shares to facilitate the encryption of TLSv1.3 handshake messages and to determine the key exchange algorithms.

You give a preference as to which elliptic curve to use.

You need to list the key shared values (from this list (table 5)), where  secp256r1 has the value 0023.

GSK_SERVER_TLS_KEY_SHARES is used by the server. The client has GSK_CLIENT_TLS_KEY_SHARES

GSK_SERVER_TLS_KEY_SHARES=00300029002500240023

or

GSK_CLIENT_TLS_KEY_SHARES=00300029002500240023

Remove any definitions in the keyshares which are not supported by TLS 1.3 (0019 and 0021) (even though you have GSK_PROTOCOL_TLSV1_2=on)

With these my LDAP started.

Once you have LDAP working, you can review the cipher specs, and remove the ones you no longer use. You should plan to remove the weaker cipher specs. If you remove a cipher spec and it is still needed, clients will fail to connect. You will need to replace the cipher specs and restart the server.

Using openssl s_client on Linux to establish a session to LDAP on z/OS.

Openssl s_client sends the TLS 1.3 supported cipher specs as part of the initial client hello handshake, so you can see the flows.

When I did this I got

s_client fails with SSL alert number 42.

From the z/OS GSK trace I can see

ERROR check_cert_extensions_3280_and_later(): authorityKeyIdentifier extension is missing
ERROR validate_certificate_basics(): Unable to verify certificate extensions: Error 0x0335306f
ERROR validate_certificate_mode(): Unable to validate certificate: Error 0x0335306f
ERROR cms_validate_certificate_mode_int(): Unable to validate certificate: Error 0x0335306f

ERROR read_tls13_certificate(): Unable to validate peer certificate: Error 0x0335306f

ERROR send_tls13_alert(): Sent TLS 1.3 alert 42 to 10.1.0.2[42514]

Where 0x0335306f (search for 0335306f) is

Explanation

A certificate extension that is mandatory for the certificate to be used for the required purpose has not been found.

User response

Ensure that the certificate chain is correct and complies with the validation mode defined for the connection. Collect a System SSL trace containing the error and then contact your service representative if the error persists.

A certificate authority can use multiple certificates to sign a certificate, for example you are refreshing or upgrading the certificate. Having the authorityKeyIdentifier says which certificate was used to sign it.
On the web there are documents which recommend its use, for example with openssl, the configuration file should have.

[ v3_ca ]
subjectKeyIdentifier=hash

authorityKeyIdentifier=keyid:always,issuer:always

On z/OS with RACF you get this attribute when using RACDCERT GENCERT … SIGNWITH() .

On my system I had misconfigured it

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer


I had to

  • create a new CA with the above in the Openssl configuration file
  • reissue the end user certificate
  • update all my servers with the new CA
  • update my LDAP CRL server

Wow – what a lot of work for a user error.

When I had done the above work, I used the

openssl x509 -in rsaca256.pem -text -noout

to list the end user certificate, and there was additional information

X509v3 Authority Key Identifier:
keyid:B3:0C:14:E9:C4:7D:8D:C1:45:9D:57:CE:19:8C:6B:0A:6C:EF:00:78
DirName:/C=GB/O=SSS/OU=CA/CN=SSCARSA1024
serial:03:DB:6B:09:E5:38:41:7D:8C:F3:CB:62:CA:1A:DD:63:CE:7E:C6:24

Using openLDAP ldapsearch on Ubuntu 18.04

I had problems getting ldapsearch on Linux to work with TLS 1.3, because it was difficult to get TLS 1.3 support – it did not send the TLS 1.3 cipher specs as part of the Client Hello. The documentation describes TLS_CIPHER_SUITE option. The Openssl syntax did not work, and the GnuTLS Syntax did work but I could not get it working with TLS 1.3 – perhaps it is not supported.

Using ldapsearch from z/OS

This was easy to set up, and I could use the gsktrace from each end to observe the flows.

I set up a bash script

export GSK_TRACE=0xff
export GSK_TRACE_FILE=./colin.gsktrace
export GSK_PROTOCOL_TLSV1_1=”off”
export GSK_PROTOCOL_TLSV1_2=”off”
export GSK_PROTOCOL_TLSV1_3=”on”
export GSK_CLIENT_TLS_KEY_SHARES=”003000290025002400230021″
export GSK_TLS_SIG_ALG_PAIRS=”0601050104010301080608050804050304030603″
export LDAP_SSL_CIPHER_FORMAT=”CHAR4″
export GSK_CLIENT_ECURVE_LIST=”00230024002500290030″
export GSK_V3_CIPHER_SPECS_EXPANDED=”130213011303C02BC02CC030C028C02aC024C026C02E”
debug=”-d all”
debug=”-d error+conns”
tls=”-Z -K adcda/MQRING -N ADCDRSA -S EXTERNAL”
tls=”-Z -K adcda/MQRING -N ADCDEC -S EXTERNAL”
host=”-h 127.0.0.1 -p 1389″
host=”-h 10.1.0.2 -p 9443″
ldapsearch $host $tls $debug -b “o=Your Company” “(objectclass=*)” aclEntry

gsktrace colin.* > out
oedit out

I defined an environment variable more than once, so I just needed to move the line down to change what was used. For example by swapping over the “tls” lines I could quickly use a different certificate.

I added the commands to format the gsktrace and display it.

Output

Because I used the debug value “-d error+conns” I got

CONNS ldap_ssl_socket_init()1600: 4-byte ciphers format will be used
CONNS ldap_ssl_socket_init()1625: No 4-byte cipher specs specified in LDAP handle
CONNS ldap_ssl_socket_init()1689: SSL Protocol Version = 621
CONNS ldap_ssl_socket_init()1700: SSL Cipher Specs
Address: 1FA36946 Length: 4 (x’4′)
00000000: f1f3f0f3 *1303 *
ERROR ldap_parse_pwdpolicy_response_int()856: No server controls o=Your Company

I think these messages are OK, and do not show a problem.

You can see that cipher spec 1303 was used.

Other information

TLS 1.3 uses one of the following ciphers

  • TLS_AES_128_GCM_SHA256 -1301
  • TLS_AES_256_GCM_SHA384 – 1302
  • TLS_CHACHA20_POLY1305_SHA256 1303
  • TLS_AES_128_CCM_SHA256 -1304
  • TLS_AES_128_CCM_8_SHA256 -1305

How do I look at TLS 1.3 handshakes?

With TLS before 1.3, encryption of the data on the session occurred after the handshake had completed, so the handshake was visible in Wireshark. With TLS 1.3 the traffic is encrypted after the “Client Hello”, so you cannot immediately see the remainder of the handshake.

Tools like OpenSSL, can write out the magic data needed for decryption. For example

openssl s_client -keylogfile /tmp/kl -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile /home/colinpaice/ssl/ssl2/colinpaice.pem

This writes information to the specified file, in my case /tmp/kl.

The file has data like

SSL/TLS secrets log file, generated by OpenSSL
SERVER_HANDSHAKE_TRAFFIC_SECRET 05a42762…
EXPORTER_SECRET 05a42762dc…
SERVER_TRAFFIC_SECRET_0 05a4276…
CLIENT_HANDSHAKE_TRAFFIC_SECRET 05a42762dc…
CLIENT_TRAFFIC_SECRET_0 05a42762d…

On Linux (Ubuntu) you can tell Wireshark to use this through

edit -> preferences -> protocols -> SSL -> (pre)-master-secret log filename

Different versions of Wireshark have TLS or SSL, use whichever one is available to you.

Specify the name of your file (/tmp/kl in my case) and Wireshark will be able to decrypt the data.

Setting up certificate authentication in LDAP.

This started off as part of a small task, when I had half an hour gap before lunch. The whole end-to-end of getting TLS and LDAP, with certificate authentication took me several weeks to set up. Now I know the traps, it takes about 10 minutes.

I describe setting up TLS and LDAP (without certificate authentication) here. Get that working before trying certificate authentication.

Setting up the simplest case of an RSA certificate on the client and an RSA certificate on the server, was pretty easy to set up. Using an Elliptic Curve certificate to and RSA certificate on the server seems impossible, it eventually worked!

I created “What cipher specs should I use?” because most of my problems, were due to using the wrong cipher specs, or the right cipher specs, but it the wrong order!

Logging on

You can logon to LDAP and specify a userid (DN) and password, for example

ldapsearch -h 127.0.0.1 -D “cn=Admin, o=Your Company” -w secret -b “o=Your Company” “(objectclass=*)” aclEntry

If you use -w ? it will prompt for your password, so it is not visible.

You can also use a certificate to logon, so you do not need the password, you just need the private key (or in my case the USB dongle with my encrypted Hardware Security Module(HSM) keystore on it).

Note: If you have your TLS private key in a file, and people can copy that file, they can impersonate you! You need to protect the file, bearing in mind your corporate IT department may be able to view any backups etc that you have. Someone would need to steal my USB dongle to use my private key and logon.

Understanding the TLS 1.2 handshake and authentication.

Skip to first steps if you are keen to implement without understanding the background,

There are several stages to establishing a TLS connection and authentication.

  1. Agree the protocols for setting up the session, for example which sort of encryption, and the key size. This provides the privacy on the connection.
  2. The server sends down its certificate, and the client authenticates it
  3. The client sends up its certificate and the server authenticates it
  4. The Distinguished Name(DN)from the client certificate is looked up in the z/OS security manager, and the associated userid is looked up.
  5. The DN is used to look in the access control lists (ACLs) do determine the access the requester has to the data.

Agree the protocols

The client the sends the protocols it supports to the server. This is a list of numbers, and each number has a meaning.

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_GCM_SHA384 (0xc087)

TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)

The part before the WITH covers

  1. The technique used to agree a key (ECDHE, DHE,RSA)
  2. The key type Elliptic Curve (ECDSA) or RSA

There must be at least one key type matching the certificate type.

  • If you have an Elliptic Curve server certificate, then you need to have records with TLS…ECDSA…
  • If you have an RSA server certificate you need to have records with TLS…RSA…

At the server, the list sent from the client is merged with the list from the server, for example if the server had

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0250039

and the client sent C024, C025, C02C, then common elements are C024, C025. These two cipher specs are TLS_ECDHE_ECDSA_WITH… so the server’s certificate needs to be ECDSA – or an Elliptic Certificate.

If no entries match you get a message like “no cipher found”.

Authenticate the server

The server sends information down to the client.

  1. The cipher suite to be used.
  2. Information for setting up the encryption of the traffic for the session.
  3. The server’s certificate, and any CA certificates in the chain.
  4. The client then validates the certificate
    1. It use the signature algorithm in the certificate. This is the information after the “WITH” in the cipher spec. In my client’s certificate it has signing algorithm sha256WithRSAEncryption. There must be a cipher spec in the list ending in TLS…RSA_WITH… ending in SHA256
    2. It checks the signature of the CA, by checking its signing algorithm in the GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) See here.

Send the client certificate to the server

If the client certificate is wanted, the servers sends the “certificate request”.

  1. It sends down the certificate type it accepts, for example RSA, ECDSA, DSS. The client then sends its certificate of that type. If the there is no match, no certificate is sent. Note:For a long time I could not get a client certificate with Elliptic Curve, to work when the server had an RSA certificate. This was due to bad parameters on the z/OS end.
  2. There is a list of signature Hash Algorithms (eg rsa_pkcs1_sha256 (0x0401) ecdsa_secp256r1_sha256 ( 0x0403).
  3. It lists the Distinguished Names of the Certificate Authorities or Self Signed certificate.
  4. With the certificate type,the signature Hash and the Certificate Authorities, the client looks in its keystore for a certificate matching these parameters; then sends the first certificate that matches – or sends “no certificate”.
  5. It sends the Certificate Verify Signature Algorithm to the server. For my Elliptic Curve this was ecdsa_secp256r1_sha256 (0x0403). This has to be in the GSK_TLS_SIG_ALG_PAIRS in the z/OS LDAP environment file. (It took me days to find this problem!)

The server authenticates the client’s certificate

The server looks inside the certificate to see how the certificate was signed, for example using

openssl x509 -in ecec.pem -text -noout

This gave Signature Algorithm: ecdsa-with-SHA256. The server needs the matching cipher spec in the GSK_V3_CIPHER_SPECS_EXPANDED environment variable.

The client’s certificate will have been signed – either by a CA, or self signed. The Signature algorithm of the CA must be in the GSK_TLS_SIG_ALG_PAIRS environment variable. The values are defined here.

For example my CA had Signature Algorithm: ecdsa-with-SHA384. This requires 0503 (SHA-384 with ECDSA) to be in the GSK_TLS_SIG_ALG_PAIRS list. If this was missing I got

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

from ldapsearch on Ubuntu. It took me another day or so to find this problem.

As you can see there are a lot of things you need to configure and to get right before it works!

First steps – using a client certificate

Firstly set up the TLS session so you can use certificates to connect to LDAP. This took me about a week, because of configuration problems, but finally it worked. I was able to connect from Ubuntu to z/OS.

Set up TLS between the client and the z/OS server, as described here.

Setup Ubuntu

I created an RSA certificate on Ubuntu using a shell script.

name=”rsaca256″
ca=”carsa1024″

openssl genpkey -out $name.key.pem -algorithm RSA -pkeyopt rsa_keygen_bits:2048

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name -passin file:password.file -passout file:password.file
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

I set up an Elliptic Curve

enddate=”-enddate 20220409174001Z”

password=” -passin file:password.file -passout file:password.file”
name=”ecec”
rm $name.key.pem
rm $name.csr
rm $name.pem
ca=”ca256″ # sign with this

openssl ecparam -name secp256r1 -genkey -noout -out $name.key.pem

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name $password
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

openssl x509 -in $name.pem -text -noout|less

# Some tools need a .p12 file of the certificate
#openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $ca.pem -chain -name $name -passout file:password.file -passin file:password.file

Openldap uses a configuration file, for example ldaprc.

I set up my ldaprc configuration file on Ubuntu.

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_REQCERT demand
TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem
TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem

Where

  • TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem was the z/OS CA certificate exported from z/OS and downloaded. It is needed to verify a certificate sent from z/OS.
  • TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem this file contains the private key
  • TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem this file contains the public key
  • You can specify TLS_CIPHER_SUITE high to use the listed cipher specs. See the openssl ciphers command, for example
    • openssl ciphers -v -V -s -tls1_2
    • openssl ciphers -v -V -s -tls1_3
    • I omitted TLS_CIPHER_SUITE for my testing.

You need the distinguished name (DN) of the certificate. You can display the subject distinguished name DN using the command openssl x509 -in secp521r.pem -text -noout. This gave me output which included:

Subject: C = GB, O = cpwebuser, CN = secp521r

I sent the Linux CA file, ca256.pem, to z/OS.

Set up z/OS

Configure LDAP.

In my LDAP configuration file I had

sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING
sslCertificate ZZZZ
commThreads 10
allowAnonymousBinds off
sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

The sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED says look in the environment file, and take the value from there. As this is the default, you can omit it from the file.

The default GSK_V3_CIPHER_SPECS_EXPANDED value in the environment file is 003500380039002F00320033 which maps to

  1. 0035 TLS_RSA_WITH_AES_256_CBC_SHA
  2. 0038 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
  3. 0039 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  4. 002f TLS_RSA_WITH_AES_128_CBC_SHA
  5. 0032 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
  6. 0033 TLS_DHE_RSA_WITH_AES_128_CBC_SHA

Which are all very old (part of deprecated SSL, TLS 1.0, TLS 1.1) so you need to set this variable in your environment file. See the next section for what I used.

Update your environment file

I use the following value

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023130313011302

This supports

  1. C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  2. C02B TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  3. C030 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  4. C02F TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  5. C024 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
  6. C023 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  7. 1303 TLS_CHACHA20_POLY1305_SHA256 TLS 1.3
  8. 1301 TLS_AES_128_GCM_SHA256 TLS 1.3
  9. 1302 TLS_AES_256_GCM_SHA384 TLS 1.3

Put the strongest certificates first. For example, the following did not work.

GSK_V3_CIPHER_SPECS_EXPANDED=0035….C02F

It caused 0035 to be used. The Certificate-Types list had only RSA and DSS, and did not include ECDSA, so the client was not able to send its Elliptic Curve certificate.

When I used

GSK_V3_CIPHER_SPECS_EXPANDED=C02F….0035

it caused C02F to be used. This caused the certificate-types list to include the ECDSA, and so the client could send its Elliptic Curve certificate. This problem took me a week to find!

I specified

GSK_TLS_SIG_ALG_PAIRS=0601050104010301080608050804

For the certificate checking algorithms. These are for SHA = 224, 256, 384, 512 and certificate types = RSA, ECDSA, and DSA.

Add the CA from Linux and add it to the keyring

The Linux CA was uploaded to z/OS as COLIN.CARSA102.PEM.

Add it to the RACF keystore as a CERTAUTH certificate, and connect it to the keyring.

RACDCERT CHECKCERT('COLIN.CARSA102.PEM') 

RACDCERT DELETE (LABEL('LINUXCA')) CERTAUTH

RACDCERT ADD('COLIN.CARSA102.PEM') -
  CERTAUTH WITHLABEL('LINUXCA') TRUST

RACDCERT ID(START1) CONNECT(RING(MQRING)-
           CERTAUTH  LABEL('LINUXCA'))

SETROPTS RACLIST(DIGTCERT) REFRESH

Map the Distinguished Name to a z/OS userid.

You have to map the DN to a userid. In the example below, it maps the DN in the SDNFILTER to the userid ADCDA. This userid must be valid, for example if the userid is revoked, the authentication will fail (with an unhelpful message).

* remove any previous instance
RACDCERT DELMAP(LABEL('LINUXSecp2' )) ID(ADCDA)

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

RACDCERT MAP ID(ADCDA )  - 
   SDNFILTER('CN=secp521r.O=cpwebuser.C=GB')  - 
   WITHLABEL('LINUXSecp2') 

RACDCERT LISTMAP ID(ADCDA) 

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH 

Note the ‘.’ between the attributes, and the attributes are squashed up, in the right order.

I did not need to refresh LDAP as the change was automatically picked up.

I could now logon to LDAP using the certificate – but could do nothing.

Give the DN access to LDAP

I gave it access, using ldapmodify passing the file with

dn: o=Your Company
changetype: modify
add: aclEntry
aclEntry : access-id:CN=SECP521R,O=CPWEBUSER,C=GB:
 object:ad:normal:grant:rscw:sensitive:rsc

Note: The DN can be in mixed case, or upper case.

I could now logon and issue queries using a certificate for authentication and authorisation.

Tracing problems

You can get trace output to the job log using the command

f GLDSRV,debug error+CONNS+LDAPBE

This traces

  • any errors
  • CONNS shows the cipher spec being used
  • LDAPBE shows the DN, and SAF user etc being used.

You can turn trace off with

f GLDSRV,debug 0

You can get the low level trace by using the gsktrace=0xff in your environment file.

Using z/OS LDAP with TLS

This started as a mini project before lunch, and now, two weeks later, I have got all of the bits to work.

As I was struggling with this, I wondered why no one had crossed this swamp before me. Is it because no one wants to cross it, or it is too difficult to do so? I think it was just difficult to do so – but I can now provide a map!

It took me a couple of hours to get LDAP on z/OS to use TLS from my Linux machine. The challenge was finding the right options. I then tried using Elliptical Certificates, and got this working, then was partially successful in using TLS 1.3.

As part of this I wrote Authentication using certificates with LDAP, which is where it was really difficult.

For clients, I used ldapsearch (from openldap ) on Ubuntu, and ldapsearch on z/OS.

See the product documentation here.

I’ve put a section “What cipher specs should I use” in a separate blog post.

Configure the LDAP environment file

I had to add TLS V1_2 and TLSV1_3 (and disable the deprecated TLS 1.1)

GSK_PROTOCOL_TLSV1_1=off
GSK_PROTOCOL_TLSV1_2=on
#GSK_PROTOCOL_TLSV1_3=on

to the environment file, because they default to off. Without these options it defaults to SSL – which is so old, my certificates and keys did not support it. I disabled TLS_1_1 because it is old.

I added

GSK_TLS_SIG_ALG_PAIRS=0601060305010401030108060805080405030403

This is used when validating certificates. The signature algorithm of the CA must be in this list see GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) (see here), get things working, then remove the ones I did not want to use; rather than try to add the ones I wanted to use.

I initially added

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0251303130113020039

This is a list of the cipher specs it will accept. You should try to restrict the certificates in your organisation, to Elliptical Certificates, and a limited configuration for any CA’s. For example have

  • all client’s certificates defined as Elliptic Curve type prime256v1, signed with ecdsa-with-SHA256.
  • All internal CAs defined as Elliptic Curve type prime256v1 signed with ecdsa-with-SHA256.

When using certificate authentication I needed the following for it all to work

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023

The environment variables

GSK_SERVER_TLS_KEY_SHARES
GSK_CLIENT_TLS_KEY_SHARES

these are used only when using TLS 1.3. I’ll blog about using TLS 1.3 at a later date.

While you are investigating problems you can use the environment variable GSK_TRACE=0xff to write a trace. You can specify GSK_TRACE_FILE to direct the trace to a particular file. The default location is /tmp/gskssl.%.trc where % is the thread id. You format it with

gsktrace /tmp/gskssl.65567.trc > out

oedit out

I used

GSK_TRACE=0xff
GSK_TRACE_FILE=/tmp/gsktrace.ldap

So I could have a shell script to format and display the trace.

gsktrace gsktrace.ldap > out
oedit out

Change the LDAP configuration file.

I used

sslCipherSpecs all
sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING

I specified a port for ldaps. You can use

ldapsearch -H ldaps://10.1.1.2:389

This is preferred to specifying host and port

ldapsearch -h 10.1.1.2 -p 389

A port can process TLS and non-TLS requests, so definition for port 1389 may not needed.

I specified a keyring sslKeyRingFile START1/MQRING. This is owned by a different userid to the started task, so needs more RACF definitions. See below for the definitions.

Define the keyring and keyring

I defined an RSA certificate and an EC certificate. I controlled which was used from the configuration file.

* create the ring
RACDCERT DELRING(MQRING) ID(START1)
RACDCERT ADDRING(MQRING) ID(START1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the server's RSA certificate

RACDCERT ID(START1) DELETE(LABEL('ZZZZ'))
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('ZZZZ') - 
             O('SERVER') - 
             OU('SSS')) - 
   ALTNAME(IP(10.1.1.2))- 
   SIZE(4096)- 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   RSA - 
   WITHLABEL('ZZZZ') - 
   KEYUSAGE(         DATAENCRYPT,DOCSIGN,HANDSHAKE         ) 

RACDCERT id(START1) ALTER(LABEL('ZZZZ'))TRUST 
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('ZZZZ') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the Elliptic Certificate

RACDCERT ID(START1) DELETE(LABEL('SERVEREC')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('SERVEREC') - 
             O('ADCD') - 
             OU('TEST')) - 
   ALTNAME(IP(10.1.1.2)) - 
   SIZE(521) - 
   NISTECC - 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   KEYUSAGE(HANDSHAKE         ,KEYAGREE) - 
   NOTAFTER(   DATE(2024-12-29))- 
   WITHLABEL('SERVEREC') 
RACDCERT id(START1)  ALTER(LABEL('SERVEREC'))TRUST
                                                        
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('SERVEREC') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 
                                                             
RACDCERT LISTRING(MQRING) ID(START1) 

Permit the LDAP server’s userid to access the private certificate in the keyring

Because the LDAP userid does not own the keyring, the server userid needs update access to be able to extract the private key.

RDEFINE RDATALIB START1.MQRING.LST UACC(NONE) 
PERMIT START1.MQRING.LST CLASS(RDATALIB) ID(GLDSRV) -
    ACCESS(CONTROL)
SETROPTS RACLIST(RDATALIB) REFRESH  

For a userid to be able to use its own keyring, it only need read access to be able to use the private certificate.

Set up the client userid

I set up the userid in the z/OS LDAP server, so I could use the z/OS userid and password. See here.

Test it out

LDAP checks the TLS parameters, when a TLS request arrives. It does not check at startup . I used openssl s_client to connect, and display information about the TLS handshake. See Debugging TLS – an easier way.

Setting up the client on Ubuntu

I set up a file ldaprc containing the ldap configuration. You can enter all of the parameters on the command line, but the ldaprc file makes it easier. My file had

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_PROTOCOL_MIN 3.2
TLS_REQCERT demand
#TLS_CERT /home/colinpaice/ssl/ssl2/rsaca256.pem
#TLS_KEY /home/colinpaice/ssl/ssl2/rsaca256.key.pem

The #TLSCERT etc are ignored. No certificate is sent to the server (unless the -Y external option is set).

Enable GSK trace on z/OS

I enabled an LDAP trace to //SYSPRINT on z/OS, using f GLDSRV,debug error+CONNS+LDAPBE

Where

  • error reports errors
  • Conns reports connection information, such as the cipher spec used.
  • LDAPBE reports on the communication to the back end, for example it reports the SAFUSER

Run a query on Linux

I used ldapsearch on Ubuntu

ldapsearch -H ldaps://10.1.1.2:1389 -D “cn=ibmuser, o=Your Company” -w panthe0n -b “o=Your Company” “(objectclass=*)” aclEntry

The trace on z/OS GLDSRV included

CONNS srv_ssl_connect()644: SSL Cipher Specs
00000000: c3f0f0f8 *C008 *

and we can see that SSL was used, and cipher spec C008 was used.

Because I had set up my userid to map use a SAF userid, in the trace I had

LDAPBE srv_process_bind_request()939: do_return_bind msgID=1, connID=8, bindDN=‘CN=IBMUSER, O=YOUR COMPANY’, safUserID=’IBMUSER’, dnList=0x0, grpList=0x0, rc=0

Debugging TLS – an easier way

I had been trying to get an IBM product on Linux to talk to LDAP using TLS and certificate authentication; where I give a certificate instead of a LDAP userid and password. Basically I got “computer say no”.

IBM products use GSKIT to manage keystores and TLS between sessions. With Java you can get out a trace of the TLS conversation. You can get out a trace on z/OS, but not on other platforms. After a day I stumbled on a different approach.

Gettting a gskit trace on z/OS.

Getting the trace was easy, understanding it was harder.

In the LDAP environment file I added GSK_TRACE=0xff. Once I had TLS to LDAP working, I used GSK_TRACE=0x04 to trace just errors. Some “errors” are recorded in the GSKTRACE as “INFO”. For example an Elliptic curve is only supported in TLS 1.3.

By default the trace goes to /tmp/gskssl.%.trc. where % is the thread id.

To format it go into USS and use the command gsktrace gskssl.83951642.trc > out, and edit the file “out”.

The linux end.

I found a great program openssl s_client. This starts a TLS handshake and prints out in easy to understand format, what is going on. For example

The command

openssl s_client -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -x509_strict

gave the flow of the data.

stderr output

The stderr output included

depth=1 O = COLIN, OU = TEST, CN = COLIN4Certification Authority
verify error:num=19:self signed certificate in certificate chain

This is the name of the CA.

Following this was an error message. Note: it is at the top of the output – not the bottom as I expected.

SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1528:SSL alert number 80

See below for a discussion about the error.

stdout output

Certificate chain
  0 s:O = SERVER, OU = SSS, CN = ZZZZ
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority
  1 s:O = COLIN, OU = TEST, CN = COLIN4Certification Authority 
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority 

This says

  • the certificate SERVER was signed by COLIN4.Certificate Authority
  • the certificate COLIN4Certificate Authority was self signed

The server certificate comes next

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Followed by the identity of the server

subject=O = SERVER, OU = SSS, CN = ZZZZ
issuer=O = COLIN, OU = TEST, CN = COLIN4Certification Authority

Next comes the list of Certificate Authority or self signed certificate in the keyring

Acceptable client certificate CA names
O = ADCD, OU = TEST, CN = MQWEB2 
O = COLIN, OU = TEST, CN = COLIN4Certification Authority
C = GB, O = SSS, OU = CA, CN = SSCARSA1024
C = GB, O = SSS, OU = CA, CN = SSCA256

Any client certificate must be signed by one of those CAs (or self signed certificates).

Then comes the client certificate types the server will accept

  • Client Certificate Types: RSA sign, DSA sign

Note: This does not include ECDSA, so elliptic certificates are not supported in this configuration.

After this is the signature algorithms the server will accept

  • Requested Signature Algorithms: RSA+SHA512: ECDSA+SHA512: RSA+SHA384: ECDSA+SHA384: RSA+SHA256: ECDSA+SHA256: DSA+SHA256: RSA+SHA224: ECDSA+SHA224: DSA+SHA224: RSA+SHA1: ECDSA+SHA1: DSA+SHA1

It gives information on the session so far


SSL handshake has read 2493 bytes and written 1537 bytes
Verification error: self signed certificate in certificate chain

and information about the key sent down. It was defined with SIZE(4096) RSA.

New, SSLv3, Cipher is AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated

and about the TLS session

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA
    Session-ID: 0401001A0A0100029...
    Session-ID-ctx: 
    Master-Key: 32D5B4AD162F0403E323DB0...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1634115022
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no

The -msg option gave me

>>> ??? [length 0005]
    16 03 01 01 31
>>> TLS 1.3, Handshake [length 0131], ClientHello
    ...
<<< ??? [length 0005]
    ... 
<<< TLS 1.3, Handshake [length 0051], ServerHello
    ...
<<< TLS 1.2, Handshake [length 0839], Certificate
    ...
<<< TLS 1.2, Handshake [length 0123], CertificateRequest
    ...
<<< TLS 1.2, Handshake [length 0004], ServerHelloDone
    ... 
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 021c], Certificate
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0206], ClientKeyExchange
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0050], CertificateVerify
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, ChangeCipherSpec [length 0001]
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0010], Finished
    ...
<<< ??? [length 0005]
    ...
<<< TLS 1.2, Alert [length 0002], fatal internal_error
    02 50

The fatal errors are described here. 0x02 is fatal error, 0x50= 80 = internal error. internal_error(80). Which is not very helpful.

In the LDAP log I got

GLD1116E Unable to initialize an SSL connection with 10.1.0.2: 434 – Certificate key is not compatible with cipher suite.

In the gsktrace on z/OS I got

cms_validate_certificate_mode_int(): Validating CN=ecec,O=cpwebuser,C=GB ENTRY
INFO get_issuer_certificate(): Using issuer CN=SSCA256,OU=CA,O=SSS,C=GB

ERROR read_v3_certificate(): Client certificate key type 13 not allowed for SSL V3 cipher 0x0035

  • Client certificate key type 13. Looking in /usr/include/gskcms.h under X.509 data types, was x509_alg_ecPublicKey.
  • SSL V3 cipher 0x0035 is listed here, as TLS_RSA_WITH_AES_256_CBC_SHA = 256-bit AES encryption with SHA-1 message authentication and RSA key exchange.
  • x509_alg_ecPublicKey is elliptic public key – which is incompatible with RSA key exchange.

When I used an RSA certificate

openssl s_client -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/rsaca256.pem -key /home/colinpaice/ssl/ssl2/rsaca256.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -verify_return_error -policy_print -x509_strict

it worked

Whoops I left my private key unprotected.

I had set up my certificates and private key so I could use them to work with a web server on z/OS. I set them up on Linux and copied them to z/OS. My tests all worked. It wasn’t till this morning when I thought, those test should have failed.

With Python and the requests package you can send REST requests to a backend server, using digital certificates for authentication. You configure which private key to use, but you are unable to specify the password for the private key. As this worked, there must be no password on the key file – whoops.

This is a bit like leaving your front door unlocked with a note “If we are not here, please come in and make yourself at home”.

To generate a private key I had used

openssl ecparam -name prime256v1 -genkey -noout -out $name.key.pem

but there is no way of specifying a password. With hindsight it was obvious there was no password protection.

I had to use

openssl ecparam -name prime256v1 -genkey -noout | openssl ec -out $name.key.pem -passout file:password.file

Where the first part of the command generated a private key, and wrote it to the output stream. The second part does nothing with the data, but wrote it to the output file using the password in password.file.

When I used this private key, the python requests failed because it needed a password!

I should have remembered

I remember helping with some security testing many years ago.

  • Userid1 was able to use resource1, good, that worked
  • Userid2 was able to use resource2, good, that worked
  • Userid1 was able to use resource2, good ,that worked – hang on… that should have failed.

The person customising security had missed one configuration step, so all users had access to all resources, and the test plan did not have any negative testing.

Using jConsole and with z/OS Liberty web server

I wanted to get some monitoring information out from z/OSMF using jConsole on my Ubuntu machine. Eventually this worked, but I had a few problems on the way. The same technique can be used for base Liberty, MQWeb, z/OSMF and ZOWE all of which are based on Liberty.

Configuring z/OSMF

I changed the z/OSMF configuration to include

<featureManager> 
  <feature>restConnector-2.0</feature> 
</featureManager> 

and restarted the server.

In the stdout (or message log) will be something like

CWWKX0103I: The JMX REST connector is running and is available at the following service
URL: service:jmx:rest://sss.com:10443/IBMJMXConnectorREST

You need the URL. The message above gave service:jmx:rest://sss.com:10443/IBMJMXConnectorREST, but I needed to use service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST .

The port number came from the httpEndpoint with id=”defaultHttpEndpoint” . I have another httpEndpoint with port 24993, and this also worked with jConsole.

Set up jConsole

I set up a script for jConsole

k1='-J-Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/adcdc.p12'
k2='-J-Djavax.net.ssl.keyStorePassword=password'
k3='-J-Djavax.net.ssl.keyStoreType=pkcs12'
t1='-J-Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/zca.jks'
t2='-J-Djavax.net.ssl.trustStorePassword=password'
t3='-J-Djavax.net.ssl.trustStoreType=jks'
d='-J-Djavax.net.debug=ssl:handshake'
d=' '
de='-debug'
de=' '
s='service:jmx:rest://10.1.1.2:10443/IBMJMXConnectorREST'
jconsole $de $s $k1 $k2 $k3 $t1 $t2 $t3 $d

Where

  • the -J .. parameters are passed through to java,
  • the -Djava… are the standard set of parameters to define the key stores on the Linux

Running this script gave a pop up window with

Secure connection failed. Retry insecurely?

The connection to service:jmxLret://10.1.1.2:10443/IBMJMXConnectorREST could not be made using SSL.

Would you like to try without SSL?

This was because of the exception

java.io.IOException: jmx.remote.credentials not provided.

I could not see how to pass userid and password to jConsole.

I then used Cntrl+N to create a new connection and entered Username: and Password: which jConsole requires. After a short delay of a few seconds jConsole responded with a graphs of Heap Memory Usage, and Threads in use. You can then select from the Measurement Beans.

The TLS setup

In the keystore I had a certificate which I had used to talk to a Liberty instance before.

This was signed, and the CA certificate had been imported into the key trust keyring on z/OS, for that HttpEndPoint.

The server responded with a server certificate (“CN=SERVER,O=SSS,C=GB”) which had been signed on z/OS. The signing certificate had been exported from z/OS and downloaded to Linux

I created a jks key trust store using this certificate, using the command

keytool -importcert -file temp4ca.pem -keystore zca.jks -storetype jks -storepass password

and used this trust store to validate the server certificate sent down from z/OS.

This worked with jConsole.

I created a pkcs12 keystore using keytool

keytool -importcert -file temp4ca.pem -keystore zca2.p12 -storetype pkcs12 -storepass password

Which also worked.

Problems using a .p12 trust store

I used

runmqakm -keydb -create -db zca.p12 -type pkcs12 -pw password
runmqakm -cert -add -file temp4ca.pem -db zca.p12 -type pkcs12 -pw password -label tempca
runmqakm -cert -details -db zca.p12 -type pkcs12 -pw password -label tempca

to create a pkcs12 keystore and import the z/OS CA certificate. The -details option displayed it.

When I tried to use it, jConsole produced the message (after the Cntl+N)

Secure connection failed. Retry insecurely?

The connection to service:jmxLret://10.1.1.2:10443/IBMJMXConnectorREST could not be made using SSL.

Would you like to try without SSL?

I used Ctrl-N as before, and got the same message.

Using

d=’-J-Djavax.net.debug=ssl:handshake’

and rerunning the script, produced a TLS trace. At the bottom was

VMPanel.connect, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

%% Invalidated:…
VMPanel.connect, SEND TLSv1.2 ALERT: fatal, description = internal_error

Using a trace at the server, gave the unhelpful , SEND TLSv1.2 ALERT:

Using openssl also failed. Create the .p12 keystore

openssl pkcs12 -export -out zca.p12 -in temp4ca.pem -name zCA -nokeys

and rerun the jconsole script, and it failed the same way.

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

It looks like runmqakm and openssl do not create a valid trust store with an imported certificate.

Additional diagnostics

When the trust store created by keytool was used; at the top of the TLS trace output was

System property jdk.tls.client.cipherSuites is set to ‘null’

Ignoring disabled cipher suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
trustStore is: /home/colinpaice/ssl/ssl2/zca.p12
trustStore type is : pkcs12

trustStore provider is :
init truststore
adding as trusted cert:
Subject: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Issuer: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Algorithm: RSA; Serial number: 0x0
Valid from Tue Jul 14 00:00:00 BST 2020 until Fri Jul 02 23:59:59 BST 2021

keyStore is : /home/colinpaice/ssl/ssl2/adcdc.p12
keyStore type is : pkcs12
keyStore provider is :
init keystore
init keymanager of type SunX509

When the runmqakm or openssl was used, the green entries were missing.

When I used runmqakm to create the pkcs12 keystore

runmqakm -cert -details -db zca.p12 -type p12 -pw password -label tempca

listed the certificate successfully.

When I used keytool to list the contents

keytool -list -keystore zca.p12 -storetype pkcs12 -storepass password
Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 0 entries

When I created the key store with keytool, both runmqakm and keytool displayed the certificate.

The problem looks like Java is only able to process the imported CA certificates when keytool was used to create the trust store.