Using PKI Server with the HTTPD web interface.

This post follows on from configuring PKI Server, and explains how to configure the HTTPD server, explains how to use it, and gives some hints on debugging it when it goes wrong.

Having tried to get this working (and fixing the odd bug) I feel that this area is not as well designed as it could have been, and I could not get parts of it to work.

For example

  • You cannot generate browser based certificate request because the <keygen> html tag was removed around 2017, and the web page fails. See here. You can use 1-Year PKI Generated Key Certificate instead, so not a big problem now we know.
  • The TLS cipher specs did not have the cipher specs I was using.
  • I was expecting a simple URL like https://10.1.1.2/PKIServer/Admin. You have to use https://10.1.1.2/PKIServ/ssl-cgi-bin/camain.rexx, which exposes the structure of the files. You can go directly go to the Admin URL using https://10.1.1.2/PKIServ/ssl-cgi-bin/auth/admmain.rexx, which is not very elegant.
  • For an end user to request a certificate you have to use https://10.1.1.2/Customers/ssl-cgi-bin/camain.rexx.
  • There seem to be few security checks.
    • I managed to get into the administrative panels and display information using a certificate mapping to a z/OS userid, and with no authority!
    • There are no authority checks for people requesting a certificate. This may not be an exposure as the person giving the OK should be checking the request.
    • There were no security checks for administration functions. (It is easy to add them(
  • You can configure HTTPD to use certificates for authentication and fall back to userid and password.
  • There is no FallbackResource specified. This is a default page which is displayed if you get the URL wrong.
  • The web pages are generated dynamically. These feel over engineered. There was a problem with one of the supplied pages, but after some time trying to resolve the problem, I gave up.

I’ll discuss how to use the web interface, then I’ll cover the improvements I made to make the HTTP configuration files meet my requirements, and give some guidance on debugging.

You may want to use a HTTPD server just for PKI Server, or if you want to share, then I suggest you allocate a TLS port just for PKI Server.

URL

The URL looks like

https://10.1.1.2:443/PKIServ/ssl-cgi-bin/camain.rexx

where (see Overview of port usage below for more explanation)

  • 10.1.1.2 is the address of my server
  • port 443 is for TLS with userid and password authentication
  • PKIServ is the part of the configuration. If you have multiple CA’s this will be CA dependant.
  • ssl-cgi-bin is the “directory” where …
  • camain.rexx the Rexx program that does the work.

With https:10.1.1.2:443/Customers/ssl-cgi-bin/camain.rexx this uses the same camain.rexx as for PKIServ, but in the template for displaying data, it uses a section with the same name (Customers) as the URL.

Overview of port usage

There are three default ports set up in the HTTPD server for PKI Server. I found the port set-up confusing, and not well document. I’ve learned (by trial and error) that

  • port 80 (the default for non https requests) for unauthenticated requests, with no TLS session protection. All data flows as clear text. You many not want to use port 80.
  • port 443 (the default for https requests) for authentication with userid and password, with TLS session protection
  • port 1443 for certificate authentication, with TLS Session protection. Using https://10.1.1.2:443/PKIServ/clientauth-cgi/auth/admmain.rexx, internally this gets mapped to https://10.1.1.2:1443/PKIServ/clientauth-cgi-bin/auth/admmain.rexx. I cannot see the need for this port and its configuration.

and for the default configuration

  • port:/PKIServ/xxx is for administrators
  • port:/Customers/xxx is for an end user.

and xxx is

  • clientauth-cgi. This uses TLS for session encryption. Port 1443 runs with user SAFRunAs PKISERV. All updates are done using the PKISERVD userid, this means you do not need to set up the admin authority for each userid. There is no security checking enabled. I was able to process certificates from a userid with no authority!
  • ssl-cgi-bin. This uses port TLS and 443. I had to change the file to be SAFRunAs %%CERTIF%% as $$CLIENT$$ is invalid. You have to give each administrator ID access to the appropriate security profiles.
  • public-cgi. This is used by some insecure requests, such as print a certificate.

I think the only one you should use is ssl-cgi-bin.

Accessing the services

You can start using

These both give a page with

  • Administration Page. This may prompt for your userid and password, and gives you a page
  • Customer’s Home Page. This gives a page https://10.1.1.2/Customers/ssl-cgi-bin/camain.rexx? called PKI Services Certificate Generation Application. This has functions like
    • Request a new certificate using a model
    • Pickup a previously requested certificate
    • Renew or revoke a previously issued browser certificate

Note: You cannot use https://10.1.1.2:1443/PKIServ/ssl-cgi-bin/camain.rexx, as 1443 is not configured for this. I could access the admin panel directly using https://10.1.1.2:1443/PKIServ/ssl-cgi-bin/auth/admmain.rexx

I changed the 443 definition to support client and password authentication by using

  • SSLClientAuth Optional . This will cause the end user to use a certificate if one is available.
  • SAFRunAs %%CERTIF%% . This says use the Certificate authentication when available, if not prompt for userid and password.

Certificate requests

I was able to use the admin interface and display all certificate requests.

Request a new certificate using a model.

I tried to use the model “1 Year PKI SSL Browser Certificate“. This asks the browser to generate a private/public key (rather than the PKIServer generating them). This had a few problems. Within the page is a <KEYGEN> tag which is not supported in most browsers. It gave me

  • The field “Select a key size” does not have anything to select, or type.
  • Clicking submit request gave me IKYI003I PKI Services CGI error in careq.rexx: PublicKey is a required field. Please use back button to try again or report the problem to admin person to

I was able to use a “1 Year PKI Generated Key Certificate

The values PKIServ and Customer are hard-coded within some of the files.

If you want to use more than one CA, read z/OS PKI Services: Quick Set-up for Multiple CAs. Use this book if you want to change “PKIServ” and “Customer”.

Colin’s HTTPD configuration files.

Because I had problems with getting the supplied files to work, I found it easier to restructure, parameterise and extend the provided files.

I’ve put these files up to github.

Basic restructure

I restructured and parametrised the files. The new files are

  • pki.conf. You edit this to define your variables.
  • 80.conf contains the definitions for a general end user, not using TLS. So the session is not encrypted. Not recommended.
  • 443.conf the definitions for the TLS port. You should not need to edit this while you are getting started. If you want to use multiple Certificate Authorities, then you need to duplicate some sections, and add definitions to the pki.conf file. See here.
  • 1443.conf the definitions for the TLS port for the client-auth path. You should not need to edit this while you are getting started. If you want to use multiple Certificate Authorities, then you need to duplicate some sections, and add definitions to the pki.conf file. See here.
  • Include conf/pkisetenv.conf to set some environment variables.
  • pkissl.conf. The SSL definitions have been moved to this file, and it has an updated list of cipher specs.

The top level configuration file pki.conf

The top level file is pki.conf. It has several sections

system wide

# define system wide stuff
# define my host name

Define sdn 10.1.1.2
Define PKIAppRoot /usr/lpp/pkiserv
Define PKIKeyRing START1/MQRING
Define PKILOG “/u/mqweb3/conf”

# The following is the default
#Define PKISAFAPPL “OMVSAPPL”
Define PKISAFAPPL “ZZZ”
Define serverCert “SERVEREC”
Define pkidir “/usr/lpp/pkiserv”

#the format of the trace entry
Define elf “[%{u}t] %E: %M”

Defined the CA specific stuff

# This defines the path of PKIServ or Customers as part of the URL
# This is used in a regular expression to map URLs to executables.
Define CA1 PKIServ|Customers
Define CA1PATH “_PKISERV_CONFIG_PATH_PKIServ /etc/pkiserv”

#Define the port for TLS
Define CA1Port 443

# specify the groups which can use the admin facility
Define CA1AdminAuth ” Require saf-group SYS1 “

other stuff

LogLevel debug
ErrorLog “${PKILOG}/zzzz.log”
ErrorLogFormat “${elf}”
# uncomment these if you want the traces
#Define _PKISERV_CMP_TRACE 0xff
#Define _PKISERV_CMP_TRACE_FILE /tmp/pkicmp.%.trc
#Define _PKISERV_EST_TRACE 0xff
#Define _PKISERV_EST_TRACE_FILE /tmp/pkiest.%.trc

#Include the files
Include conf/80.conf
Include conf/1443.conf
Include conf/443.conf

The TLS configuration file

The file 443.conf has several parts. It uses the parametrised values above, for example ${pkidir} is substituted with /usr/lpp/pkiserv/. When getting started you should not need to edit this file.

Listen ${CA1Port}
<VirtualHost *:${CA1Port}>

#define the log file for this port
ErrorLog “${PKILOG}/z${CA1Port}.log


DocumentRoot “${pkidr}”
LogLevel Warn
ErrorLogFormat “${elf}”

Include conf/pkisetenv.conf
Include conf/pkissl.conf
KeyFile /saf ${PKIKeyRing}
SSLClientAuth Optional
#SSLClientAuth None

RewriteEngine On

# display a default page if there are problems
# I created it in ${PKIAppRoot}/PKIServ,
# (/usr/lpp/pkiserv/PKIServ/index.html)
FallbackResource “index.html”

Below the definitions for one CA are defined. If you want a second CA, then duplicate the definitions,and change CA1 to CA2.

Notes on following section.

# Start of definitions for a CA

<IfDefine CA1>
SetEnv ${CA1PATH}
RewriteRule ¬/(${CA1})/ssl-cgi/(.) https://${sdn}/$1/ssl-cgi-bin/$2 [R,NE]

RewriteRule ¬/(${CA1})/clientauth-cgi/(.) https://${sdn}:1443/$1/clientauth-cgi-bin/$2 [R,NE,L]
ScriptAliasMatch ¬/(${CA1})/adm(.).rexx(.) “${PKIAppRoot}/PKIServ/ssl-cgi-bin/auth/adm$2.rexx$3
ScriptAliasMatch ¬/(${CA1})/Admin “${PKIAppRoot}/PKIServ/ssl-cgi-bin/auth/admmain.rexx”
ScriptAliasMatch ¬/(${CA1})/EU “${PKIAppRoot}/PKIServ/ssl-cgi-bin/camain.rexx”
ScriptAliasMatch ¬/(${CA1})/(public-cgi|ssl-cgi-bin)/(.*) “${PKIAppRoot}/PKIServ/$2/$3”
<LocationMatch “¬/(${CA1})/clientauth-cgi-bin/auth/pkicmp”>
CharsetOptions NoTranslateRequestBodies
</LocationMatch>
<LocationMatch “¬/(${CA1})/ssl-cgi-bin(/(auth|surrogateauth))?/cagetcert.rexx”>
Charsetoptions TranslateAllMimeTypes
</LocationMatch>
<IfDefine>

#End of definitions for CA1

Grouping the statements for a CA in one place means it is very easy to change it to use multiple CA’s, just repeat the section between <IfDefine…> and</IfDefine> and change CA1 to CA2.

The third part has definitions for controlling access to a directory. I added more some security information, and changed $$CLIENT$$ to %%CLIENT%%. This is a subset of the file, for illustration

# The User will be prompted to enter a RACF User ID
#and password and will use the same RACF User ID
# and password to access files in this directory
<Directory ${PKIAppRoot}/PKIServ/ssl-cgi-bin/auth>
AuthName AuthenticatedUser
AuthType Basic
AuthBasicProvider saf
Require valid-user

#Users must have access to the SAF APPLID to work
# ZZZ in my case
# it defaults to OMVSAPPL
<IfDefine PKISAFAPPL>
SAFAPPLID ${PKISAFAPPL}
</IfDefine>

# IBM Provided has $$CLIENT$$ where it should have %%CLIENT%%
# SAFRunAs $$CLIENT$$
# The following says use certificate if available else prompt for
# userid and password
SAFRunAs %%CERTIF%%
</Directory>…

Debugging hints and tips

I spent a lot of time investigating problems, and getting the definitions right.

Whenever I made a change, I used

s COLWEB,action=’restart’

to cause the running instance of HTTPD server to stop and restart. Any errors in the configuration are reported in the job which has the action=’restart’. It is easy to overlook configuration problems, and then spend time wondering why your change has not been picked up.

I edited the envvars file, and added code to rename and delete logs. For example rm z443.log.save, and mv z443.log z443.log.save .

I found it useful to have

<VirtualHost *:443>
DocumentRoot “${pkidr}”
ErrorLog “${PKILOG}/z443.log
ErrorLogFormat “${elf}”
LogLevel Warn


Where

  • Error logs is where the logs for this virtual host (port 443) are stored. I like to have one per port.
  • The format is defined in the variable Define elf “[%{c}t] %E: %M” in the pki.conf file. The c is compact time (2021-11-27 17:19:09). If you use %{cu}t you also get microseconds. I could not find where you just get the time, and no date.
  • LogLevel Warn. When trying to debug the RewriteRule and ScriptAlias I used LogLevel trace6. I also used LogLevel Debug authz_core_module:Trace6 which sets the default to Debug, but the authorization checking to Trace6.

With LogLevel Debug, I got a lot of good TLS diagnostics

Validating ciphers for server: S0W1, port: 443
No ciphers enabled for SSLV2
SSL0320I: Using SSLv3,TLSv1.0,TLSv1.1,TLSv1.2,TLSv1.3 Cipher: TLS_RSA_WITH_AES_128_GCM_SHA256(9C)

TLSv10 disabled, not setting ciphers
TLSv11 disabled, not setting ciphers
TLSv13 disabled, not setting ciphers
env_init entry (generation 2)
VirtualHost S0W1:443 is the default and only vhost

Then for each web session

Cert Body Len: 872
Serial Number: 02:63
Distinguished name CN=secp256r1,O=cpwebuser,C=GB
Country: GB
Organization: cpwebuser
Common Name: secp256r1
Issuer’s Distinguished Name: CN=SSCA256,OU=CA,O=SSS,C=GB
Issuer’s Country: GB
Issuer’s Organization: SSS
Issuer’s Organization Unit: CA
Issuer’s Common Name: SSCA256
[500865c0f0] SSL2002I: Session ID: A…AAE= (new)
[500865c0f0] [33620012] Peer certificate: DN [CN=secp256r1,O=cpwebuser,C=GB], SN [02:63], Issuer [CN=SSCA256,OU=CA,O=SSS,C=GB]

With LogLevel Trace6 I got information about the RewriteRule, for example we can see /Customers/EU was mapped to /usr/lpp/pkiserv/PKIServ/ssl-cgi-bin/camain.rexx

applying pattern ‘¬/(PKIServ|Customers)/clientauth-cgi/(.*)’ to uri ‘/Customers/EU’

AH01626: authorization result of Require all granted: granted
AH01626: authorization result of : granted

should_translate_request: r->handler=cgi-script r->uri=/Customers/EU r->filename=/usr/lpp/pkiserv/PKIServ/ssl-cgi-bin/camain.rexx dcpath=/

uri: /Customers/EU file: /usr/lpp/pkiserv/PKIServ/ssl-cgi-bin/camain.rexx method: 0 imt: (unknown) flags: 00 IBM-1047->ISO8859-1

# and the output

Headers from script ‘camain.rexx’:
Status: 200 OK
Status line from script ‘camain.rexx’: 200 OK
Content-Type: text/html
X-Frame-Options: SAMEORIGIN
Cache-Control: no-store, no-cache

Certificate validation on LDAP using OCSP

It is good practice to validate certificates when security is important. There are typically two ways of doing this.

  • Have a central server with a list of all revoked certificates. This server has a Certificate Revocation List(CRL) which has to be maintained. This solution tends to be deprecated
  • When a certificate is created add an extension with “check using this URL… to check if the certificate is valid”. This field is added when the certificate is signed by the Certificate Authority. A request is sent to the OCSP server, and a response sent back. This technique is known as Online Certificate Status Protocol (OCSP). You need an OCSP server for each certificate authority.

The blog post explains about OCSP and some of the challenges in using it.

You can configure a certificate to have OCSP support.

When you create a signed certificate, the CA signs the certificate.

For example

openssl req -config eccert.config … -out $name.csr
openssl ca -config openssl-ca-user.cnf -policy signing_policy -in $name.csr -extensions xxxx

In the config file are different sections, in my xxx section is

[ xxxx ]
authorityInfoAccess = OCSP;URI:http://10.1.0.2:2000

If the server (LDAP in this case) is configured for OCSP, then when it sees the AIA extension, it sends a request to the OCSP server at the URI (10.1.0.2:2000) which responds saying “good” , “revoked” or “unknown”.

OCSP server.

Openssl provides a simple ocsp server. You give it

  • the port it is to use
  • the CA public certificate (so the incoming certificate can be validated against the CA)
  • a file for that CA of the certificates it has issued, and it they are valid or not.

You can also configure the LDAP server to give a default URI, for those certificates that do not have the Authority Info Access(AIA) section. Typically this only works if you have one Certificate Authority, or you have a smart OCSP server which can handle the data for multiple CAs.

For example if your openssl OSCP server is configured for CA with DN:CN=CA1,o=MYORG, and you send down a request for DN:CN=CA2,o=myorg, it will not recognize it.

If you use the AIA extension, then you can have two different OCSP servers, one for each CA.

How does it work?

There are two ways of doing OCSP checking.

  • The client looks at the certificate and sees there is an OCSP extension. The client sends a request to the OCSP server to check the certificate is valid.
  • As part of the TLS handshake send the extension to the server to say “please do OCSP checking for me”. The server issues the request to the OCSP server, and can cache the response in the server. The server also sends the OCSP status to the client as part of the TLS handshake. The next request for the certificate can use the cached response. It tends to improve performance, as it reduces the number of requests to the OCSP server, and the responses are cached in the server. This is known as OCSP stapling.

Does using OCSP kill performance?

Yes and no. I’ll cover this after “caching the response”

Caching the responses

The OCSP server may be within your organization, or it may be be external. The time to send a request and get the response back may range from milliseconds to seconds.

Servers can typically cache the response to an OCSP query.

The OCSP may update it’s certificate list every hour, or perhaps every day, so if it is refreshed once a day, your LDAP server, does not need to refresh its data more than once a day.

The openssl OCSP server has an option -nmin minutes| -ndays days which is how often to reread the file. It sends a response

SingleResponse
  certID
    hashAlgorithm (SHA-1)
    issuerNameHash: 157b5dd0bcdee5b5428e063cf29a1f4e45be7499
    issuerKeyHash: 5830af55c7b4d49fc9e3fac91441ef1fd7b215e3
    serialNumber: 587
  certStatus: good (0)
  thisUpdate: 2021-11-05 10:42:58 (UTC)
  nextUpdate: 2021-11-05 10:43:58 (UTC)

From this the requester knows when the cached value is no longer valid, and needs to contact the OCSP server again for that certificate. From this we can see that the response is valid for 60 seconds.

When the certificate was revoked the output was

Online Certificate Status Protocol
  responseStatus: successful (0)
  responseBytes
    ResponseType Id: 1.3.6.1.5.5.7.48.1.1 (id-pkix-ocsp-basic)
    BasicOCSPResponse
      tbsResponseData
        responderID: byName (1)
          byName: 0...
          producedAt: 2021-11-06 15:15:58 (UTC)
            responses: 1 item
              SingleResponse
                certID...
                  certStatus: revoked (1)
                    revoked
                      revocationTime: 2021-11-06 15:12:30 (UTC)
                      revocationReason: cessationOfOperation (5)
                    thisUpdate: 2021-11-06 15:15:58 (UTC)
                    nextUpdate: 2021-11-06 15:16:58 (UTC)
            responseExtensions: 1 item
              Extension
                Id: 1.3.6.1.5.5.7.48.1.2 (id-pkix-ocsp-nonce)
                ReOcspNonce: a7e59577b7d2b3a2
      signatureAlgorithm (ecdsa-with-SHA256)
        Padding: 0
        signature: 3046022100fb264d4c5dbbbf45cce752a8263c4f01631441...
      certs: 1 item...

This information is available to the client.

What about performance?

If you write your application properly the impact of OCSP can be minimized.

For example consider the scenario where you are using REST requests, and go to one of a number of server.

  • Your application starts
  • It needs to check the validity of the certificate. Send a request to the OCSP server. This could take a long time (seconds). Some applications have a time-out of 15 seconds waiting for a response.
  • The response comes back saying “certificate Good – and valid for 1 hour.
  • Send a request to the server, and get the response back
  • Issue another request (with another TLS handshake)
  • After 1 hour – resend the request to the OCSP server to re-validate.
  • Etc

OCSP stapling

OCSP stapling is very common now. Before this, clients themselves used to check the validity of the user’s certificate by contacting the OCSP server. With OCSP stapling, a request is put into the TLS handshake which says “please do the OCSP checks for me and send me the output”. This allows the server to cache the information.

Think of the old days of checking in at the airport, when the person checking you in, would staple “checked-in by agent 46” to your paper ticket.

The client requests this by adding the “status_request” extension to the TLS clientHello handshake.

The server sends down, as part of the “serverHello” the information it received from the OCSP server.

Note. ldapsearch, from openssl, sends up the status_request, but does not handle the response, I get

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1) additional info: (unknown error code)

openssl s_client does not send the status_request extension, so does not participate in the OCSP checking.

Java does support this, and a Java application can get the OCSP response message using the getStatusResponses method on the from the ExtendedSSLSession. I believe you can decode it using Bouncycastle.

Setting up LDAP on z/OS to support OCSP

I added the following to a working LDAP system

Environment

GSK_OCSP_URL=http://10.1.0.2:2000
GSK_OCSP_ENABLE=ON
GSK_OCSP_CLIENT_CACHE_SIZE=100
GSK_REVOCATION_SECURITY_LEVEL=MEDIUM
GSK_SERVER_OCSP_STAPLING=OFF
GSK_OCSP_RESPONSE_SIGALG_PAIRS=0603060105030501040304020401

#GSK_OCSP_RESPONSE_SIGALG_PAIRS=0806080508040603060105030501040304020401

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Stapling

I set GSK_SERVER_OCSP_STAPLING=OFF because ldapsearch on Ubunutu did not work with the ENDENTITY value.

Signature Algorithms

If GSK_OCSP_RESPONSE_SIGALG_PAIRS included any of 0806 0805 0804, I got messages

GLD1160E Unable to initialize the LDAP client SSL support: Error 113, Reason -99.
GLD1063E Unable to initialize the SSL environment: 466 – Signature algorithm pair is not valid.

In the trace I had 03353003 Cryptographic algorithm is not supported, despite these being listed in the documentation.

Nonce

The nonce is used to reduce replay attacks. In your request to the OCSP serve you include a nonce (string of data). You expect this in the response message.

The default in LDAP is off!

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Configuration

In the configuration file I had

sslKeyRingFile START1/MQRING

sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

Setting up the OCSP server on Linux

My CA was on Linux – address 10.1.0.2.

I set up a bash script to run the openssl OCSP server on Linx,

ca=”-CA ca256.pem”
index=”-index index.txt”
port=”-port 2000″
#rsigner=”-rsigner rsaca256.pem –rkey rsaca256.key.pem”
#rsigner=”-rsigner ecec.pem –rkey ecec.key.pem”
rsigner=”-rsigner ss.pem –rkey ss.key.pem”
nextUpdate=”-nmin 1″
openssl ocsp $index $ca $port $rsigner $nextUpdate

You need

  • CA … for the Certificate Authority .pem file
  • -index index.txt, for the status of the certificates issued by the CA
  • -port … a port to use
  • -rsigner… the public key to be used when responding
  • -rkey … the private key for encrypting the response.
  • -nmin … how often the index file is refreshed. Typically this value might be an hour or more.

It does not log any activity, so I had to use Wireshark to trace the network traffic.

Problems using a certificate signed by the CA, for encrypting the response.

The rsigner certificate needs to have

Extended Key Usage: critical, OCSP Signing

Without this I got the following in the gsktrace

ERROR check_ocsp_signer_extensions(): extended keyUsage does not allow OCSP Signing

A self signed certificate worked OK.

Testing it

I had several challenges when testing it

  • ldapsearch on Linux sends up the “I support OCSP stapling”, but it objects to the response, and ends with unknown error code.
  • openssl s_client does not send the OCSP flag, and so the certificate does not get validated.
  • Java worked. I used this as a basis, and made a few changes to reflect my system. I needed to use the following optuons to run it
    • -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ecec.p12
    • -Djavax.net.ssl.keyStorePassword=password
    • -Djavax.net.ssl.keyStoreType=pkcs12
    • -Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/dantrust.p12
    • -Djavax.net.ssl.trustStorePassword=password
    • -Djavax.net.ssl.trustStoreType=pkcs12

Once I change GSK_SERVER_OCSP_STAPLING=ENDENTITY to GSK_SERVER_OCSP_STAPLING=OFF, I was able to use LDAPSEARCH.

Some OCSP certificates didn’t work

In my OCSP server, I used a certificate signed by the CA, for encrypting the response back to LDAP.

In the GSKtrace I got

ERROR find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
ERROR crypto_ec_token_public_key_verify(): ICSF service failure: CSFPPKV retCode = 0x4, rsnCode = 0x2af8
ERROR crypto_ec_token_public_key_verify(): Signature failed verification
ERROR crypto_verify_data_signature(): crypto_ec_verify_data_signature() failed: Error 0x03353004

There are two connected problems here

  1. Find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
  2. retcode 0x2af8 (11000) The digital signature verify ICSF callable service completed successfully but the supplied digital signature failed verification.8 (11000). I did not have the correct CA certificate for the OCSP certificate in the LDAP keyring.

How do I look at TLS 1.3 handshakes?

With TLS before 1.3, encryption of the data on the session occurred after the handshake had completed, so the handshake was visible in Wireshark. With TLS 1.3 the traffic is encrypted after the “Client Hello”, so you cannot immediately see the remainder of the handshake.

Tools like OpenSSL, can write out the magic data needed for decryption. For example

openssl s_client -keylogfile /tmp/kl -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile /home/colinpaice/ssl/ssl2/colinpaice.pem

This writes information to the specified file, in my case /tmp/kl.

The file has data like

SSL/TLS secrets log file, generated by OpenSSL
SERVER_HANDSHAKE_TRAFFIC_SECRET 05a42762…
EXPORTER_SECRET 05a42762dc…
SERVER_TRAFFIC_SECRET_0 05a4276…
CLIENT_HANDSHAKE_TRAFFIC_SECRET 05a42762dc…
CLIENT_TRAFFIC_SECRET_0 05a42762d…

On Linux (Ubuntu) you can tell Wireshark to use this through

edit -> preferences -> protocols -> SSL -> (pre)-master-secret log filename

Different versions of Wireshark have TLS or SSL, use whichever one is available to you.

Specify the name of your file (/tmp/kl in my case) and Wireshark will be able to decrypt the data.

Setting up certificate authentication in LDAP.

This started off as part of a small task, when I had half an hour gap before lunch. The whole end-to-end of getting TLS and LDAP, with certificate authentication took me several weeks to set up. Now I know the traps, it takes about 10 minutes.

I describe setting up TLS and LDAP (without certificate authentication) here. Get that working before trying certificate authentication.

Setting up the simplest case of an RSA certificate on the client and an RSA certificate on the server, was pretty easy to set up. Using an Elliptic Curve certificate to and RSA certificate on the server seems impossible, it eventually worked!

I created “What cipher specs should I use?” because most of my problems, were due to using the wrong cipher specs, or the right cipher specs, but it the wrong order!

Logging on

You can logon to LDAP and specify a userid (DN) and password, for example

ldapsearch -h 127.0.0.1 -D “cn=Admin, o=Your Company” -w secret -b “o=Your Company” “(objectclass=*)” aclEntry

If you use -w ? it will prompt for your password, so it is not visible.

You can also use a certificate to logon, so you do not need the password, you just need the private key (or in my case the USB dongle with my encrypted Hardware Security Module(HSM) keystore on it).

Note: If you have your TLS private key in a file, and people can copy that file, they can impersonate you! You need to protect the file, bearing in mind your corporate IT department may be able to view any backups etc that you have. Someone would need to steal my USB dongle to use my private key and logon.

Understanding the TLS 1.2 handshake and authentication.

Skip to first steps if you are keen to implement without understanding the background,

There are several stages to establishing a TLS connection and authentication.

  1. Agree the protocols for setting up the session, for example which sort of encryption, and the key size. This provides the privacy on the connection.
  2. The server sends down its certificate, and the client authenticates it
  3. The client sends up its certificate and the server authenticates it
  4. The Distinguished Name(DN)from the client certificate is looked up in the z/OS security manager, and the associated userid is looked up.
  5. The DN is used to look in the access control lists (ACLs) do determine the access the requester has to the data.

Agree the protocols

The client the sends the protocols it supports to the server. This is a list of numbers, and each number has a meaning.

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_GCM_SHA384 (0xc087)

TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)

The part before the WITH covers

  1. The technique used to agree a key (ECDHE, DHE,RSA)
  2. The key type Elliptic Curve (ECDSA) or RSA

There must be at least one key type matching the certificate type.

  • If you have an Elliptic Curve server certificate, then you need to have records with TLS…ECDSA…
  • If you have an RSA server certificate you need to have records with TLS…RSA…

At the server, the list sent from the client is merged with the list from the server, for example if the server had

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0250039

and the client sent C024, C025, C02C, then common elements are C024, C025. These two cipher specs are TLS_ECDHE_ECDSA_WITH… so the server’s certificate needs to be ECDSA – or an Elliptic Certificate.

If no entries match you get a message like “no cipher found”.

Authenticate the server

The server sends information down to the client.

  1. The cipher suite to be used.
  2. Information for setting up the encryption of the traffic for the session.
  3. The server’s certificate, and any CA certificates in the chain.
  4. The client then validates the certificate
    1. It use the signature algorithm in the certificate. This is the information after the “WITH” in the cipher spec. In my client’s certificate it has signing algorithm sha256WithRSAEncryption. There must be a cipher spec in the list ending in TLS…RSA_WITH… ending in SHA256
    2. It checks the signature of the CA, by checking its signing algorithm in the GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) See here.

Send the client certificate to the server

If the client certificate is wanted, the servers sends the “certificate request”.

  1. It sends down the certificate type it accepts, for example RSA, ECDSA, DSS. The client then sends its certificate of that type. If the there is no match, no certificate is sent. Note:For a long time I could not get a client certificate with Elliptic Curve, to work when the server had an RSA certificate. This was due to bad parameters on the z/OS end.
  2. There is a list of signature Hash Algorithms (eg rsa_pkcs1_sha256 (0x0401) ecdsa_secp256r1_sha256 ( 0x0403).
  3. It lists the Distinguished Names of the Certificate Authorities or Self Signed certificate.
  4. With the certificate type,the signature Hash and the Certificate Authorities, the client looks in its keystore for a certificate matching these parameters; then sends the first certificate that matches – or sends “no certificate”.
  5. It sends the Certificate Verify Signature Algorithm to the server. For my Elliptic Curve this was ecdsa_secp256r1_sha256 (0x0403). This has to be in the GSK_TLS_SIG_ALG_PAIRS in the z/OS LDAP environment file. (It took me days to find this problem!)

The server authenticates the client’s certificate

The server looks inside the certificate to see how the certificate was signed, for example using

openssl x509 -in ecec.pem -text -noout

This gave Signature Algorithm: ecdsa-with-SHA256. The server needs the matching cipher spec in the GSK_V3_CIPHER_SPECS_EXPANDED environment variable.

The client’s certificate will have been signed – either by a CA, or self signed. The Signature algorithm of the CA must be in the GSK_TLS_SIG_ALG_PAIRS environment variable. The values are defined here.

For example my CA had Signature Algorithm: ecdsa-with-SHA384. This requires 0503 (SHA-384 with ECDSA) to be in the GSK_TLS_SIG_ALG_PAIRS list. If this was missing I got

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

from ldapsearch on Ubuntu. It took me another day or so to find this problem.

As you can see there are a lot of things you need to configure and to get right before it works!

First steps – using a client certificate

Firstly set up the TLS session so you can use certificates to connect to LDAP. This took me about a week, because of configuration problems, but finally it worked. I was able to connect from Ubuntu to z/OS.

Set up TLS between the client and the z/OS server, as described here.

Setup Ubuntu

I created an RSA certificate on Ubuntu using a shell script.

name=”rsaca256″
ca=”carsa1024″

openssl genpkey -out $name.key.pem -algorithm RSA -pkeyopt rsa_keygen_bits:2048

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name -passin file:password.file -passout file:password.file
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

I set up an Elliptic Curve

enddate=”-enddate 20220409174001Z”

password=” -passin file:password.file -passout file:password.file”
name=”ecec”
rm $name.key.pem
rm $name.csr
rm $name.pem
ca=”ca256″ # sign with this

openssl ecparam -name secp256r1 -genkey -noout -out $name.key.pem

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name $password
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

openssl x509 -in $name.pem -text -noout|less

# Some tools need a .p12 file of the certificate
#openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $ca.pem -chain -name $name -passout file:password.file -passin file:password.file

Openldap uses a configuration file, for example ldaprc.

I set up my ldaprc configuration file on Ubuntu.

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_REQCERT demand
TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem
TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem

Where

  • TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem was the z/OS CA certificate exported from z/OS and downloaded. It is needed to verify a certificate sent from z/OS.
  • TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem this file contains the private key
  • TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem this file contains the public key
  • You can specify TLS_CIPHER_SUITE high to use the listed cipher specs. See the openssl ciphers command, for example
    • openssl ciphers -v -V -s -tls1_2
    • openssl ciphers -v -V -s -tls1_3
    • I omitted TLS_CIPHER_SUITE for my testing.

You need the distinguished name (DN) of the certificate. You can display the subject distinguished name DN using the command openssl x509 -in secp521r.pem -text -noout. This gave me output which included:

Subject: C = GB, O = cpwebuser, CN = secp521r

I sent the Linux CA file, ca256.pem, to z/OS.

Set up z/OS

Configure LDAP.

In my LDAP configuration file I had

sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING
sslCertificate ZZZZ
commThreads 10
allowAnonymousBinds off
sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

The sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED says look in the environment file, and take the value from there. As this is the default, you can omit it from the file.

The default GSK_V3_CIPHER_SPECS_EXPANDED value in the environment file is 003500380039002F00320033 which maps to

  1. 0035 TLS_RSA_WITH_AES_256_CBC_SHA
  2. 0038 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
  3. 0039 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  4. 002f TLS_RSA_WITH_AES_128_CBC_SHA
  5. 0032 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
  6. 0033 TLS_DHE_RSA_WITH_AES_128_CBC_SHA

Which are all very old (part of deprecated SSL, TLS 1.0, TLS 1.1) so you need to set this variable in your environment file. See the next section for what I used.

Update your environment file

I use the following value

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023130313011302

This supports

  1. C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  2. C02B TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  3. C030 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  4. C02F TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  5. C024 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
  6. C023 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  7. 1303 TLS_CHACHA20_POLY1305_SHA256 TLS 1.3
  8. 1301 TLS_AES_128_GCM_SHA256 TLS 1.3
  9. 1302 TLS_AES_256_GCM_SHA384 TLS 1.3

Put the strongest certificates first. For example, the following did not work.

GSK_V3_CIPHER_SPECS_EXPANDED=0035….C02F

It caused 0035 to be used. The Certificate-Types list had only RSA and DSS, and did not include ECDSA, so the client was not able to send its Elliptic Curve certificate.

When I used

GSK_V3_CIPHER_SPECS_EXPANDED=C02F….0035

it caused C02F to be used. This caused the certificate-types list to include the ECDSA, and so the client could send its Elliptic Curve certificate. This problem took me a week to find!

I specified

GSK_TLS_SIG_ALG_PAIRS=0601050104010301080608050804

For the certificate checking algorithms. These are for SHA = 224, 256, 384, 512 and certificate types = RSA, ECDSA, and DSA.

Add the CA from Linux and add it to the keyring

The Linux CA was uploaded to z/OS as COLIN.CARSA102.PEM.

Add it to the RACF keystore as a CERTAUTH certificate, and connect it to the keyring.

RACDCERT CHECKCERT('COLIN.CARSA102.PEM') 

RACDCERT DELETE (LABEL('LINUXCA')) CERTAUTH

RACDCERT ADD('COLIN.CARSA102.PEM') -
  CERTAUTH WITHLABEL('LINUXCA') TRUST

RACDCERT ID(START1) CONNECT(RING(MQRING)-
           CERTAUTH  LABEL('LINUXCA'))

SETROPTS RACLIST(DIGTCERT) REFRESH

Map the Distinguished Name to a z/OS userid.

You have to map the DN to a userid. In the example below, it maps the DN in the SDNFILTER to the userid ADCDA. This userid must be valid, for example if the userid is revoked, the authentication will fail (with an unhelpful message).

* remove any previous instance
RACDCERT DELMAP(LABEL('LINUXSecp2' )) ID(ADCDA)

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

RACDCERT MAP ID(ADCDA )  - 
   SDNFILTER('CN=secp521r.O=cpwebuser.C=GB')  - 
   WITHLABEL('LINUXSecp2') 

RACDCERT LISTMAP ID(ADCDA) 

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH 

Note the ‘.’ between the attributes, and the attributes are squashed up, in the right order.

I did not need to refresh LDAP as the change was automatically picked up.

I could now logon to LDAP using the certificate – but could do nothing.

Give the DN access to LDAP

I gave it access, using ldapmodify passing the file with

dn: o=Your Company
changetype: modify
add: aclEntry
aclEntry : access-id:CN=SECP521R,O=CPWEBUSER,C=GB:
 object:ad:normal:grant:rscw:sensitive:rsc

Note: The DN can be in mixed case, or upper case.

I could now logon and issue queries using a certificate for authentication and authorisation.

Tracing problems

You can get trace output to the job log using the command

f GLDSRV,debug error+CONNS+LDAPBE

This traces

  • any errors
  • CONNS shows the cipher spec being used
  • LDAPBE shows the DN, and SAF user etc being used.

You can turn trace off with

f GLDSRV,debug 0

You can get the low level trace by using the gsktrace=0xff in your environment file.

Using z/OS LDAP with TLS

This started as a mini project before lunch, and now, two weeks later, I have got all of the bits to work.

As I was struggling with this, I wondered why no one had crossed this swamp before me. Is it because no one wants to cross it, or it is too difficult to do so? I think it was just difficult to do so – but I can now provide a map!

It took me a couple of hours to get LDAP on z/OS to use TLS from my Linux machine. The challenge was finding the right options. I then tried using Elliptical Certificates, and got this working, then was partially successful in using TLS 1.3.

As part of this I wrote Authentication using certificates with LDAP, which is where it was really difficult.

For clients, I used ldapsearch (from openldap ) on Ubuntu, and ldapsearch on z/OS.

See the product documentation here.

I’ve put a section “What cipher specs should I use” in a separate blog post.

Configure the LDAP environment file

I had to add TLS V1_2 and TLSV1_3 (and disable the deprecated TLS 1.1)

GSK_PROTOCOL_TLSV1_1=off
GSK_PROTOCOL_TLSV1_2=on
#GSK_PROTOCOL_TLSV1_3=on

to the environment file, because they default to off. Without these options it defaults to SSL – which is so old, my certificates and keys did not support it. I disabled TLS_1_1 because it is old.

I added

GSK_TLS_SIG_ALG_PAIRS=0601060305010401030108060805080405030403

This is used when validating certificates. The signature algorithm of the CA must be in this list see GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) (see here), get things working, then remove the ones I did not want to use; rather than try to add the ones I wanted to use.

I initially added

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0251303130113020039

This is a list of the cipher specs it will accept. You should try to restrict the certificates in your organisation, to Elliptical Certificates, and a limited configuration for any CA’s. For example have

  • all client’s certificates defined as Elliptic Curve type prime256v1, signed with ecdsa-with-SHA256.
  • All internal CAs defined as Elliptic Curve type prime256v1 signed with ecdsa-with-SHA256.

When using certificate authentication I needed the following for it all to work

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023

The environment variables

GSK_SERVER_TLS_KEY_SHARES
GSK_CLIENT_TLS_KEY_SHARES

these are used only when using TLS 1.3. I’ll blog about using TLS 1.3 at a later date.

While you are investigating problems you can use the environment variable GSK_TRACE=0xff to write a trace. You can specify GSK_TRACE_FILE to direct the trace to a particular file. The default location is /tmp/gskssl.%.trc where % is the thread id. You format it with

gsktrace /tmp/gskssl.65567.trc > out

oedit out

I used

GSK_TRACE=0xff
GSK_TRACE_FILE=/tmp/gsktrace.ldap

So I could have a shell script to format and display the trace.

gsktrace gsktrace.ldap > out
oedit out

Change the LDAP configuration file.

I used

sslCipherSpecs all
sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING

I specified a port for ldaps. You can use

ldapsearch -H ldaps://10.1.1.2:389

This is preferred to specifying host and port

ldapsearch -h 10.1.1.2 -p 389

A port can process TLS and non-TLS requests, so definition for port 1389 may not needed.

I specified a keyring sslKeyRingFile START1/MQRING. This is owned by a different userid to the started task, so needs more RACF definitions. See below for the definitions.

Define the keyring and keyring

I defined an RSA certificate and an EC certificate. I controlled which was used from the configuration file.

* create the ring
RACDCERT DELRING(MQRING) ID(START1)
RACDCERT ADDRING(MQRING) ID(START1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the server's RSA certificate

RACDCERT ID(START1) DELETE(LABEL('ZZZZ'))
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('ZZZZ') - 
             O('SERVER') - 
             OU('SSS')) - 
   ALTNAME(IP(10.1.1.2))- 
   SIZE(4096)- 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   RSA - 
   WITHLABEL('ZZZZ') - 
   KEYUSAGE(         DATAENCRYPT,DOCSIGN,HANDSHAKE         ) 

RACDCERT id(START1) ALTER(LABEL('ZZZZ'))TRUST 
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('ZZZZ') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the Elliptic Certificate

RACDCERT ID(START1) DELETE(LABEL('SERVEREC')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('SERVEREC') - 
             O('ADCD') - 
             OU('TEST')) - 
   ALTNAME(IP(10.1.1.2)) - 
   SIZE(521) - 
   NISTECC - 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   KEYUSAGE(HANDSHAKE         ,KEYAGREE) - 
   NOTAFTER(   DATE(2024-12-29))- 
   WITHLABEL('SERVEREC') 
RACDCERT id(START1)  ALTER(LABEL('SERVEREC'))TRUST
                                                        
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('SERVEREC') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 
                                                             
RACDCERT LISTRING(MQRING) ID(START1) 

Permit the LDAP server’s userid to access the private certificate in the keyring

Because the LDAP userid does not own the keyring, the server userid needs update access to be able to extract the private key.

RDEFINE RDATALIB START1.MQRING.LST UACC(NONE) 
PERMIT START1.MQRING.LST CLASS(RDATALIB) ID(GLDSRV) -
    ACCESS(CONTROL)
SETROPTS RACLIST(RDATALIB) REFRESH  

For a userid to be able to use its own keyring, it only need read access to be able to use the private certificate.

Set up the client userid

I set up the userid in the z/OS LDAP server, so I could use the z/OS userid and password. See here.

Test it out

LDAP checks the TLS parameters, when a TLS request arrives. It does not check at startup . I used openssl s_client to connect, and display information about the TLS handshake. See Debugging TLS – an easier way.

Setting up the client on Ubuntu

I set up a file ldaprc containing the ldap configuration. You can enter all of the parameters on the command line, but the ldaprc file makes it easier. My file had

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_PROTOCOL_MIN 3.2
TLS_REQCERT demand
#TLS_CERT /home/colinpaice/ssl/ssl2/rsaca256.pem
#TLS_KEY /home/colinpaice/ssl/ssl2/rsaca256.key.pem

The #TLSCERT etc are ignored. No certificate is sent to the server (unless the -Y external option is set).

Enable GSK trace on z/OS

I enabled an LDAP trace to //SYSPRINT on z/OS, using f GLDSRV,debug error+CONNS+LDAPBE

Where

  • error reports errors
  • Conns reports connection information, such as the cipher spec used.
  • LDAPBE reports on the communication to the back end, for example it reports the SAFUSER

Run a query on Linux

I used ldapsearch on Ubuntu

ldapsearch -H ldaps://10.1.1.2:1389 -D “cn=ibmuser, o=Your Company” -w panthe0n -b “o=Your Company” “(objectclass=*)” aclEntry

The trace on z/OS GLDSRV included

CONNS srv_ssl_connect()644: SSL Cipher Specs
00000000: c3f0f0f8 *C008 *

and we can see that SSL was used, and cipher spec C008 was used.

Because I had set up my userid to map use a SAF userid, in the trace I had

LDAPBE srv_process_bind_request()939: do_return_bind msgID=1, connID=8, bindDN=‘CN=IBMUSER, O=YOUR COMPANY’, safUserID=’IBMUSER’, dnList=0x0, grpList=0x0, rc=0

Debugging TLS – an easier way

I had been trying to get an IBM product on Linux to talk to LDAP using TLS and certificate authentication; where I give a certificate instead of a LDAP userid and password. Basically I got “computer say no”.

IBM products use GSKIT to manage keystores and TLS between sessions. With Java you can get out a trace of the TLS conversation. You can get out a trace on z/OS, but not on other platforms. After a day I stumbled on a different approach.

Gettting a gskit trace on z/OS.

Getting the trace was easy, understanding it was harder.

In the LDAP environment file I added GSK_TRACE=0xff. Once I had TLS to LDAP working, I used GSK_TRACE=0x04 to trace just errors. Some “errors” are recorded in the GSKTRACE as “INFO”. For example an Elliptic curve is only supported in TLS 1.3.

By default the trace goes to /tmp/gskssl.%.trc. where % is the thread id.

To format it go into USS and use the command gsktrace gskssl.83951642.trc > out, and edit the file “out”.

The linux end.

I found a great program openssl s_client. This starts a TLS handshake and prints out in easy to understand format, what is going on. For example

The command

openssl s_client -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -x509_strict

gave the flow of the data.

stderr output

The stderr output included

depth=1 O = COLIN, OU = TEST, CN = COLIN4Certification Authority
verify error:num=19:self signed certificate in certificate chain

This is the name of the CA.

Following this was an error message. Note: it is at the top of the output – not the bottom as I expected.

SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1528:SSL alert number 80

See below for a discussion about the error.

stdout output

Certificate chain
  0 s:O = SERVER, OU = SSS, CN = ZZZZ
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority
  1 s:O = COLIN, OU = TEST, CN = COLIN4Certification Authority 
    i:O = COLIN, OU = TEST, CN = COLIN4Certification Authority 

This says

  • the certificate SERVER was signed by COLIN4.Certificate Authority
  • the certificate COLIN4Certificate Authority was self signed

The server certificate comes next

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Followed by the identity of the server

subject=O = SERVER, OU = SSS, CN = ZZZZ
issuer=O = COLIN, OU = TEST, CN = COLIN4Certification Authority

Next comes the list of Certificate Authority or self signed certificate in the keyring

Acceptable client certificate CA names
O = ADCD, OU = TEST, CN = MQWEB2 
O = COLIN, OU = TEST, CN = COLIN4Certification Authority
C = GB, O = SSS, OU = CA, CN = SSCARSA1024
C = GB, O = SSS, OU = CA, CN = SSCA256

Any client certificate must be signed by one of those CAs (or self signed certificates).

Then comes the client certificate types the server will accept

  • Client Certificate Types: RSA sign, DSA sign

Note: This does not include ECDSA, so elliptic certificates are not supported in this configuration.

After this is the signature algorithms the server will accept

  • Requested Signature Algorithms: RSA+SHA512: ECDSA+SHA512: RSA+SHA384: ECDSA+SHA384: RSA+SHA256: ECDSA+SHA256: DSA+SHA256: RSA+SHA224: ECDSA+SHA224: DSA+SHA224: RSA+SHA1: ECDSA+SHA1: DSA+SHA1

It gives information on the session so far


SSL handshake has read 2493 bytes and written 1537 bytes
Verification error: self signed certificate in certificate chain

and information about the key sent down. It was defined with SIZE(4096) RSA.

New, SSLv3, Cipher is AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated

and about the TLS session

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA
    Session-ID: 0401001A0A0100029...
    Session-ID-ctx: 
    Master-Key: 32D5B4AD162F0403E323DB0...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1634115022
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no

The -msg option gave me

>>> ??? [length 0005]
    16 03 01 01 31
>>> TLS 1.3, Handshake [length 0131], ClientHello
    ...
<<< ??? [length 0005]
    ... 
<<< TLS 1.3, Handshake [length 0051], ServerHello
    ...
<<< TLS 1.2, Handshake [length 0839], Certificate
    ...
<<< TLS 1.2, Handshake [length 0123], CertificateRequest
    ...
<<< TLS 1.2, Handshake [length 0004], ServerHelloDone
    ... 
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 021c], Certificate
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0206], ClientKeyExchange
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0050], CertificateVerify
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, ChangeCipherSpec [length 0001]
    ...
>>> ??? [length 0005]
    ...
>>> TLS 1.2, Handshake [length 0010], Finished
    ...
<<< ??? [length 0005]
    ...
<<< TLS 1.2, Alert [length 0002], fatal internal_error
    02 50

The fatal errors are described here. 0x02 is fatal error, 0x50= 80 = internal error. internal_error(80). Which is not very helpful.

In the LDAP log I got

GLD1116E Unable to initialize an SSL connection with 10.1.0.2: 434 – Certificate key is not compatible with cipher suite.

In the gsktrace on z/OS I got

cms_validate_certificate_mode_int(): Validating CN=ecec,O=cpwebuser,C=GB ENTRY
INFO get_issuer_certificate(): Using issuer CN=SSCA256,OU=CA,O=SSS,C=GB

ERROR read_v3_certificate(): Client certificate key type 13 not allowed for SSL V3 cipher 0x0035

  • Client certificate key type 13. Looking in /usr/include/gskcms.h under X.509 data types, was x509_alg_ecPublicKey.
  • SSL V3 cipher 0x0035 is listed here, as TLS_RSA_WITH_AES_256_CBC_SHA = 256-bit AES encryption with SHA-1 message authentication and RSA key exchange.
  • x509_alg_ecPublicKey is elliptic public key – which is incompatible with RSA key exchange.

When I used an RSA certificate

openssl s_client -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/rsaca256.pem -key /home/colinpaice/ssl/ssl2/rsaca256.key.pem -CAfile ~/ssl/ssl2/colinpaice.pem -verify_return_error -policy_print -x509_strict

it worked

Which cipher specs should I use?

I spent a couple of weeks trying to get different flavours of TLS to work with an LDAP server, as part of client certificate authentication. Although I knew something of the TLS handshake before I started, I now know much more. In this blog post I’ll try to explain some of the best practices which will make your life easier, and how to avoid some of the problems.

The short answer is use the following cipher specs.

GSK_V3_CIPHER_SPECS_EXPANDED=C02C,C02B,C030,C02,1301,1302,1303

Backgroup

There are different levels of TLS.

  • TLS 1.3 is the latest, and supports a small subset of cipher specs. The TLS 1.3 handshake is more efficient than earlier versions (fewer network flows).
  • TLS 1.2 is very popular, it supports a wide selection of cipher specs, some of which are considered weak.
  • TLS 1.1 and TLS 1.0 are older versions of TLS, and should no longer be used.
  • SSL – this is so old, you should move to TLS 1.2 or 1.3

Using ciphers names and numbers.

Programs like LDAP and GSKIT refer to 4 character numbers for certificates. In the description below, I give the numbers and the names of the cipher specs.

When using GSKIT you might specify the cipher specs with an environment variable GSK..=”C02BC02F”, to specify cipher specs C02B and C02F.

There is a good openssl command

openssl ciphers -v -V
openssl ciphers -v -V high
openssl ciphers -v -V -s -tls1_3
openssl ciphers -v -V -s -tls1_2

Which lists all of the cipher specs in decreasing strength order, along with some interpretation of the values, for example

0xC0,0x2B – ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD

Shows

  • 0xC0 0x2B is C02B
  • ECDHE-ECDSA-AES128-GCM-SHA256 the description
  • TLSv1.2 – this is applicable to TLS 1.2
  • Kx – the key exchange is ECDH
  • Au – the authentiation is ECDSA
  • Enc – the encryption is AES -128 GCM
  • Mac – the hashing value is AEAD.

openssl ciphers -v -V high gives the high strength values.

openssl ciphers -v -V -s -tls1_3 gives the TLS 1.3 cipher specs.

TLS 1.3

This is easy. The people who developed this selected only a few, strong cipher specs.

  • 1301 TLS_AES_128_GCM_SHA256
  • 1302 TLS_AES_256_GCM_SHA384
  • 1303 TLS_CHACHA20_POLY1305_SHA256

TLS 1.3 uses Elliptic Curves as standard, for example Curve 25519 or secp256r1.

TLS 1.3 does not (currently) support the following

  • 1304 TLS_AES_128_CCM_SHA256    
  • 1305 TLS_AES_128_CCM_8_SHA256.

TLS 1.2

There is a big list of supported cipher specs. The recommended list is a much smaller list.

The cipher spec name has several parts

  1. The handshake protocol
  2. The authentication(certificate) type
  3. The technique for symmetric encryption
  4. The technique for doing checksum, ( hash or MAC)

From my own investigation, and searching the internet, I have found the following guidance.

Authentication(Certificate) type

You can create certificates with certificate types of RSA or Digital Signature Algorithm(DSA), Elliptic Curve (and DSA).

I recommend having an Elliptic Curve (+DSA) certificate as the server certificate because it is stronger and better than the others.

This means using cipher specs like TLS_…_ECDSA_WITH….

Handshake prototols

  1. Diffie-Hellman is better than RSA.
  2. Use TLS_ECDH… over TLS_DH… (Diffie-Hellman using Elliptic Curve)
  3. Use TLS_ECDHE_ (Elliptic Curve Diffie-Hellman with Ephemeral) over TLS_ECDH_ (Elliptic Curve Diffie-Hellman; Ephemeral is better)

This means use cipher suites

  1. TLS_ECDHE_ECDSA_WITH_
  2. TLS_ECDH_ECDSA_WITH_

Symmetric encryption algorithms

This is the information after the WITH_

AES is better than DES or 3DES.

Use cipher suite

  1. TLS_…_…_WITH_AES_256_…_…
  2. TLS_…_…_WITH_AES_128_…_…

Block data encryption

GCM is better than CCM which is better than CBC. (For example GCM calculations can exploit multiple processor pipelines whereas CBC does not exploit multiple CPUs).

AEAD ciphers include GCM and ChaCha20-Poly1305(available in TLS 1.3).

SHA384 is stronger than SHA256 which is stronger than SHA. I saw some comments that SHA384 is better than SHA512 because of problems if a bad guy changes the size of the file when SHA512 is used.

  1. TLS_…_…_WITH_AES_256_GCM_SHA384
  2. TLS_…_…_WITH_AES_128_GCM_SHA256

List of cipher specs

Below are the cipher specs – sorted, good at the top. As a general rule, bigger numbers (C02C) are better than small numbers (0006).

The documentation usually lists the cipher specs in numerical order – which makes it hard to select the ones you need!

When you use these to specify cipher specs, put the strong ones at the front. This is because gskit takes the first acceptable cipher spec in the list, where you want the strongest acceptable cipher spec. If you have a weak cipher spec at the front of the list, you may use that over a more secure cipher spec. This was a major problem for me.

I found specifying the first four ( C02C,C02B,C030,C02f) and the TLS 1.3 (1301,1302,1303) worked well for me.

Elliptic Curve Diffie-Hellman Ephemeral, Elliptic Curve Certificate,

C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
C02B TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
C030 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
C02F TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

C024 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
C023 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
C00A TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
C009 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
C008 TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA

Elliptic Curve Diffie-Hellman Ephemeral, RSA Certificate

C028 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
C027 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
C014 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
C013 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
C012 TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA

Elliptic Curve Diffie-Hellman Ephemeral, Elliptic Curve Certificate,

C02E TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
C02D TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256

C026 TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
C025 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
C005 TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
C004 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
C003 TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA

Elliptic Curve Diffie-Hellman Ephemeral, RSA Certificate,

C032 TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
C031 TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256

C02A TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
C029 TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
C00F TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
C00E TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
C00D TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA

Diffie-Hellman Ephemeral, DSS Certificate,

00A3 TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
00A2 TLS_DHE_DSS_WITH_AES_128_GCM_SHA256

006A TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
0040 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
0038 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
0032 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
0013 TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
0012 TLS_DHE_DSS_WITH_DES_CBC_SHA

Diffie-Hellman Ephemeral, RSA Certificate

009F TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
009E TLS_DHE_RSA_WITH_AES_128_GCM_SHA256

006B TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
0067 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256

0039 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
0033 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
0016 TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
0015 TLS_DHE_RSA_WITH_DES_CBC_SHA

Diffie-Hellman Ephemeral, DSS Certificate,

00A1 TLS_DH_RSA_WITH_AES_256_GCM_SHA384
00A4 TLS_DH_DSS_WITH_AES_128_GCM_SHA256
00A5 TLS_DH_DSS_WITH_AES_256_GCM_SHA384
00A0 TLS_DH_RSA_WITH_AES_128_GCM_SHA256

0068 TLS_DH_DSS_WITH_AES_256_CBC_SHA256
003E TLS_DH_DSS_WITH_AES_128_CBC_SHA256
0036 TLS_DH_DSS_WITH_AES_256_CBC_SHA
0030 TLS_DH_DSS_WITH_AES_128_CBC_SHA
000D TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA
000C TLS_DH_DSS_WITH_DES_CBC_SHA

Diffie-Hellman Ephemeral, RSA Certificate,

0069 TLS_DH_RSA_WITH_AES_256_CBC_SHA256
003F TLS_DH_RSA_WITH_AES_128_CBC_SHA256
0037 TLS_DH_RSA_WITH_AES_256_CBC_SHA
0031 TLS_DH_RSA_WITH_AES_128_CBC_SHA
0010 TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA
000F TLS_DH_RSA_WITH_DES_CBC_SHA

RSA handshake RSA certificate

009D TLS_RSA_WITH_AES_256_GCM_SHA384
009C TLS_RSA_WITH_AES_128_GCM_SHA256

003D TLS_RSA_WITH_AES_256_CBC_SHA256
003C TLS_RSA_WITH_AES_128_CBC_SHA256
0035 TLS_RSA_WITH_AES_256_CBC_SHA
002F TLS_RSA_WITH_AES_128_CBC_SHA
000A TLS_RSA_WITH_3DES_EDE_CBC_SHA
0009 TLS_RSA_WITH_DES_CBC_SHA
0006 TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5