Category: TLS

Getting my z/OS client to talk to a remote server over TLS

I looked into this because someone asked me about using the mail client on z/OS talking to a mail server somewhere in the internet.

My z/OS running on zPDT under Linux did not have a working connection to the outside world.

Configuring Linux to pass the z/OS traffic to the external network.

A ping to the server, got no response back.

Using Wireshark on the wireless connection, I could see my request from z/OS coming via Linux and going to the outside network. The source IP address was 10.1.1.1 – a “local only” address, and so the server could not send a response back to me.

The routing through Linux worked because I had a default route; ip route gave me 

default via 192.168.xxx.xxx4 dev wlxd037450ab7ac proto dhcp metric 600 
10.1.0.0/24 dev eno1 proto kernel scope link src 10.1.0.3 metric 100 
10.1.0.0/24 via 10.1.0.3 dev eno1 proto static metric 100 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1

To get the “right” IP address passed to the external network, was surprisingly easy. It was documented in the zPDT book.

I created a script 

echo ‘Your firewall must be enabled for this command to be meaningful’
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -F FORWARD
iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o wlxd037450ab7ac -j MASQUERADE
# iptables -I INPUT -p tcp --dport 3270 -j ACCEPT

and used sudo to execute it.

I started the firewall

sudo ufw status
sudo ufw enable

I could then successfully ping the address

When I used Wireshark on the connection between Linux and z/OS, the source IP address was 10.1.1.2. When I used Wireshark on the wireless connection between Linux and the external network, the IP address was 192.168.xxx.xxx. This shows the effect of the iptables commands, it changed the 10.1.1.2 to a 192.168.xxx.xxx – and back again on the replies.

Getting the server’s certificate

I configured the AT-TLS definitions, for TLS support, specifying the choice of cipher specs I wanted to be used, and specified the keyring.

My z/OS client connected to the remote server. I could see the “client hello” flow, and the “server hello” response. In the server hello flow were the CA certificates the server supported. The client validates the certificates by checking the client’s key store.

I did not have the matching CA certificate in my keystore, and so my client failed to validate the certificate sent down, and the connection ended.

I could do it properly, and get the certificate from the official source for the server, but this was a test system, and I was happy to trust what was sent down in the handshake.

You can use wireshark or openssl to get the certificate

With openssl

openssl s_client -showcerts -tls1_2 -starttls smtp -connect smtp.gmail.com:587 | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’

You’ll need to interupt it (Ctrl-C) and paste the data into a file (google.pem)

You can display the contents of the certificate using

openssl x509 -in google.pem -text -noout|less

With Wireshark

display the certificates in the Server Hello flow.

right click on the certificate and use right click, export packet bytes. I created a file user.crt. This is a binary file.

The openssl command (on Linux)

openssl x509 -inform der -in user.crt -text > usercert.pem

displays the file in .pem format.

Put the certificate into the keyring

If you are using something that needs a keyring, you need to import the certificate into RACF and connect it to a keyring. If you are using something that uses openssl, such as Python, you need a .pem file in Unix Services.

I created a file on z/OS (VB) COLIN.USERCERT.PEM, and copied the .pem file into it (between and including the BEGIN CERTIFICATE and END CERTIFICATE lines).

I imported it, and connected it to my keyring using

//IBMRACF  JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=*                                           
//SYSTSPRT DD SYSOUT=*                                           
//SYSTSIN DD *                                                   
RACDCERT CHECKCERT('COLIN.USERCERT.PEM')                         
RACDCERT DELETE  (LABEL('USERTRUST.ORG'))    CERTAUTH            
RACDCERT ADD('COLIN.USERCERT.PEM')        -                      
   CERTAUTH  WITHLABEL('USERTRUST.ORG') TRUST 
RACDCERT ID(START1) CONNECT(RING(TN3270) - 
                            CERTAUTH    - 
                            LABEL('USERTRUST.ORG')) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                     
                                                                 
RACDCERT LISTRING(TN3270)  ID(START1)                            
/*

Once I had refreshed my PAGENT job, and restarted my client code, I was able to establish a TLS session.

Client certificate

In the TLS handshake, the mail servers did not send down a request for the client certificate. As no certificate was requested, or sent up to the server, no certificate authentication was done. You need to check your mail server for it authentication process – it may just be userid and password. This information is sent once the TLS session has been established – so it flows encrypted.

Debugging AT-TLS session problems

I deliberately misconfigured AT-TLS to see how easy it would be to identify and resolve the problems from an AT-TLS perspective. It turned out worse than I expected. There is little information on the z/OS to help you.

I configured TTLSEnvironmentAction {trace 255 } (see the bottom of this blog) and refreshed the PAGENT. I had configured SYSLOGD so records for *.TCPIP.*.* went to /var/log/TCPIP.

I reran my MQ client application and got

  • from MQ on Linux, in file /var/mqm/errors/AMQERR01.LOG return code 2393 (MQRC_SSL_INITIALIZATION_ERROR).
  • On Linux there was a file /var/mqm/trace/AMQ.SSL.TRC – which only IBM can format!
  • From TCPIP on z/OS EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900 JOBNAME: CSQ9CHIN RULE: REMOTE-TO-CSQ1 USERID: START1 GRPID: 0000001B ENVID: 0000000B CONNID: 0000006E This give
    • the address of my client,
    • the name of the chinit
    • which AT-TLS rule was used

The message EZD1287I TTLS Error RC: 402 Initial Handshake pointed me to Cryptographic Services System Secure Sockets Layer Programming – No SSL cipher specifications. The first reason was

The client and server cipher specifications do not contain at least one value in common. Client and server cipher specifications might be limited depending on which System SSL FMIDs are installed. See Cipher suite definitions for more information. Server cipher specifications are dependent on the type of algorithms that are used by the server certificate (RSA, DSA, ECDSA, or Diffie-Hellman), which might limit the options available during cipher negotiation.

MQ Trace

I took an MQ trace and formatted it. I used grep to find which file had “Cipher” in it.

Within this file I searched for Start of GSKit TLS Handshake Transcript.

This had information sent to the server as part of the handshake, and further down it had the reason code. You can see from the example that the fields and their values have been displayed (so cipher spec 003c is displayed as tls_rsa_with_aes_128_cbc_sha256)

Start of GSKit TLS Handshake Transcript (1119 bytes)
 <client_hello>
 client_version 
 TLSV12
 random 
   gsksslDissector_32Bits
   7f9d66d8
   gsksslDissector_Opaque
   Length: 28
   3E 5B 45 66 EE A3 C1 9F FB 81 0C 2F 38 19 DF 95     >[Ef......./8...
   5A 1B 54 CC B8 CB B6 C9 87 39 5E 88                 Z.T......9^.
 session_id 
 Length: 00
 cipher_suites 
 Length: 04
 00 FF 00 3C                                         ...<
 tls_ri_scsv,tls_rsa_with_aes_128_cbc_sha256
 compression_methods 
 Length: 01
 00                                                  .
 Extensions
 Length: 74
 00 0D 00 18 00 16 06 01 05 01 04 01 03 01 02 01     ................
 06 03 05 03 04 03 03 03 02 03 02 02 00 00 00 2A     ...............*
 00 28 00 00 25 73 79 73 74 65 6D 32 65 2D 64 65     .(..%system2e-de
 66 32 65 2D 73 76 72 63 6F 6E 6E 2E 63 68 6C 2E     f2e-svrconn.chl.
 6D 71 2E 69 62 6D 2E 63 6F 6D                       mq.ibm.com
  Extension Count: 2
  signature_algorithms 13
   rsa:sha512,rsa:sha384,rsa:sha256,rsa:sha224,rsa:sha1,  
   ecdsa:sha512,ecdsa:sha384,ecdsa:sha256,ecdsa:sha224,
   ecdsa:sha1,dsa:sha1
  server_name 0
   system2e-def2e-svrconn.chl.mq.ibm.com
End of GSKit TLS Handshake Transcript
{  rriEvent
 ...
 RetCode = 20009665, rc1 = 420, rc2 = 0, Comment1='SYSTEM.DEF.SVRCONN', 
 Comment2='gsk_secure_soc_init', Comment3='10.1.1.2(1414)'
 ...
}

With this trace, I am able to see what was sent to z/OS.

The AT-TLS Trace

The trace ( configured in syslogd to be in /var/log/TCPIP) had a one line entry with (I’ve reformatted it to make it easier to read).

Map CONNID: 0000006B 
LOCAL: 10.1.1.2..1414 
REMOTE:
10.1.0.2..53898 
JOBNAME: CSQ9CHIN 
USERID: START1 
TYPE: InBound 
STATUS: Enabled 
RULE: REMOTE-TO-CSQ1 
ACTIONS:
CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A

and data

RC: 0 Connection Init
Initial Handshake ACTIONS: CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A HS-Server
RC: 0 Call GSK_SECURE_SOCKET_OPEN - 00000052FD6228F0
RC: 0 Set GSK_FD(300) - 000000000000006B
RC: 0 Set GSK_USER_DATA(200) - 000000007EC32430
RECV CIPHER 160303007B

and one loooong record with

RECV CIPHER 
010000770303749ED51D8DC7794EE6AC36B01FD115F38A4B0812D35 
C80A5F95DB840C35735CA00000400FF003C0100004A000D00180016 
060105010401030102010603050304030303020302020000002A002 
800002573797374656D32652D64656632652D737672636F6E6E2E63 
686C2E6D712E69626D2E636F6D 
SEND CIPHER 15030300020228

From the AT-TLS trace of the data received from the client, it is the data as received, and has not been split down into useful fields.

I could not find any documentation on how to format this string. It is not easy to create a program to format this (and get it right), for example converting cipher spec 003c to TLS_RSA_WITH_AES_128_CBC_SHA256. However I have a REXX exec which works in ISPF and decodes the data into fields, but not the contents of the fields – so the cipher spec is reported as 003c

I had some success taking this data, and creating a file which Wireshark could process. See Wireshark – using external data: Bodging a hex dump file. This was not always successful, as it looks like the data is truncated, and can have non hex data in the hex stream.

Note, the System SSL server started task, GSKSRVR, can capture System SSL trace. The output is like 

Job TCPIP     Process 0101001D  Thread 00000001  read_v3_client_hello            
Received CLIENT-HELLO message

with no detailed information

Tracing just one session

If you have a busy system you could get trace data for many sessions. You may want to set up a TLS rule, so you use a “debug port”, or you specify the remote host IP address, and port, using information from the error message

EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900 …

And dont forget…

And do not forget to reset the TTLSEnvironmentAction entry to reset the trace, and to refresh the PAGENT.

Wireshark – using external data

Export the trace in hex format

  • Capture your trace.
  • Select the line(s) of interest
  • File -> Export Packet Dissections -> As Plain Text
  • On the next screen
    • Select the directory and file name
    • Select the Export as “Plain text” – it adds .txt to the file name you chose above, if required.
    • Select the packet range
    • Packet format: Bytes (and only bytes)
    • Save

If you select Packet Format : Bytes you get output like

0000  00 d8 61 e9 31 2a 8c 16 45 36 f4 8a 08 00 45 10   ..a.1*..E6....E.
0010  00 78 39 c5 40 00 40 06 ec a4 0a 01 00 02 0a 01   .x9.@.@.........
0020  00 03 bc 3e 00 16 0d 5b 33 ab 44 ae 0b 8f 80 18   ...>...[3.D.....

If you select Packet Format: Details

You get 

No.     Time           Source                Destination           Dst port port   Protocol Length Info
      1 0.000000000    10.1.0.2              10.1.0.3              22       48190  SSH      134    Client: Encrypted packet (len=68)

Frame 1: 134 bytes on wire (1072 bits), 134 bytes captured (1072 bits) on interface enp0s31f6, id 0
Ethernet II, Src: LCFCHeFe_36:f4:8a (8c:16:45:36:f4:8a), Dst: Micro-St_e9:31:2a (00:d8:61:e9:31:2a)
Internet Protocol Version 4, Src: 10.1.0.2, Dst: 10.1.0.3
Transmission Control Protocol, Src Port: 48190, Dst Port: 22, Seq: 1, Ack: 1, Len: 68
SSH Protocol

If you specify If you select Packet Format: Details + Bytes you get both sets of output in the file.

If you try to import this files back into Wire Shark, it should work, as it will ignore lines which do not begin with a hexadecimal offset.

Import a hex dump file

Using the “bytes” file created above

  • File -> Import from Hex Dump
  • File /u/colin/aaa.txt
  • Offsets: Hexadecimal
  • Encapsulation type: Ethernet

The data is displayed as if it was real time capture.

The Wireshark documentation on importing data is here.

Information in the imported file.

Various bits of information are not in the hex dump file, and WireShark creates (fakes) them. For example

  • The packet number is the data record number of the data in the file.
  • The time between packets is an incrementing microsecond counter, 0.000000, 0.000001, 0.000002 etc
  • The packet length is taken from the length of the data record.
  • The “arrival time” is the time the data was read from the file.
  • The encapsulation type is taken from the import page.
  • However information such as source, destination, port, destination port are taken from the data.

Bodging a hex dump file

I had a AT-TLS trace from PAGENT which was a string of raw hex data, with no interpretation of the TLS data. AT-TLS does not provide any way of formatting this data.

I was partially successful in using WireShark to process this data, and decode the TLS fields.

The first part of the data is the IP and TCP header info, followed by the TLS data.

I obtained a hex dump of a TLS handshake and took the first x42 bytes.

0000  8c 16 45 36 f4 8a 00 d8 61 e9 31 2a 08 00 45 00 
0010  xx xx 01 3d 00 00 3f 06 5c ef 0a 01 01 02 0a 01 
0020  00 02 05 86 b8 dc 34 b1 03 17 57 c4 a5 0c 80 18
0030  0f fc 1d bf 00 00 01 01 08 0a 9d cf 5c b7 db 69 
0040  e9 0d

I then wrote an ISPF editor rexx program to take the TLS trace and convert into the similar format, such as

0042   16 03 03 08 8E 02 00 00 4D 03 03 62 A8 75 18 C9
0052   5D 61 E1 1B 71 40 6A 6B 95 A8 F9 E5 E8 3A 83 AB
....
07E2   F0 61 98 51 92 4B 2E 0A 29

Then changed the xx xx to the length of the data – 12. So xx xx becomes (x7ea – 12) = x07dd.

The data could sometimes be imported.

If anyone is interested I can send them the ISPF rexx exec.

The trace from PAGENT is some times incomplete or wrong. I had long records with

0A29C..TRC at the end, which is clearly not hex data.

Hints on bodging.

It took me several hours to get the first successful import of TLS data. Some of the problems I had were

  • The offsets were wrong. I sometimes had the wrong offset, or the same offset more than once
  • The xx xx needs to be correct.
  • In the PAGENT trace file I had data like 160303007B which is of length 10 – but only length 5 hex characters. The length would be 00 47 ( 00 42 + 5)

Being too strong can be a waste of time

The short story is to use Elliptic curves with size 256 or 384, and avoid wasting time investigating why using other sizes doesn’t always work.

The long story – part 1 the server

I had a RACF server certificate defined as

SIZE(521) NISTECC …

and using Chrome browser it failed. I did a lot of digging around and could see the gsktrace had

EXIT gsk_get_ec_parameters_info(): <— Exit status 0x00000000 (0) EC curve type 34, key size 521
ERROR send_v3_alert(): Sent SSL V3 alert 40 to 10.1.0.2[38736]

INFO edit_ciphers(): Server certificate ec curve 0034 not in supported ecurve tls extension. EC cipher suites disabled

Where curve type 34, also known as 0x0019 = secp521r.

When I changed the size to 256 it worked.

Digging in to this, the Client Hello part of the TLS handshake sent from Chrome had

Supported Groups (4 groups)
Supported Group: Reserved (GREASE) (0x2a2a)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)

This is missing the 0x0019.

For Firefox, which worked, it had

Supported Groups (6 groups)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)
Supported Group: secp521r1 (0x0019)
Supported Group: ffdhe2048 (0x0100)
Supported Group: ffdhe3072 (0x0101)

So after a couple of hours working on this ( and an overnight sleep-on-it) the solution was to use secp256r1 instead of secp521.

The long story – part 2 – the client

I also tried using a client certificate which was defined as a secp521r1. This has similar problems. When I defined it as secp256r1 or secp384r1 it worked.

The gsktrace included

INFO read_v3_certificate(): Certificate key algorithm 13, Signature algorithm 84
ENTRY gsk_get_ec_parameters_info(): —> keyInfo size 12
EXIT gsk_get_ec_parameters_info(): <— Exit status 0x00000000 (0) EC curve type 34, key size 521
ERROR read_v3_certificate(): Client certificate elliptic curve not in clients supported elliptic curve list

I also know more about the TLS hand shake – which has more twists and turns every time I look at it!

Certificate validation on LDAP using OCSP

It is good practice to validate certificates when security is important. There are typically two ways of doing this.

  • Have a central server with a list of all revoked certificates. This server has a Certificate Revocation List(CRL) which has to be maintained. This solution tends to be deprecated
  • When a certificate is created add an extension with “check using this URL… to check if the certificate is valid”. This field is added when the certificate is signed by the Certificate Authority. A request is sent to the OCSP server, and a response sent back. This technique is known as Online Certificate Status Protocol (OCSP). You need an OCSP server for each certificate authority.

The blog post explains about OCSP and some of the challenges in using it.

You can configure a certificate to have OCSP support.

When you create a signed certificate, the CA signs the certificate.

For example

openssl req -config eccert.config … -out $name.csr
openssl ca -config openssl-ca-user.cnf -policy signing_policy -in $name.csr -extensions xxxx

In the config file are different sections, in my xxx section is

[ xxxx ]
authorityInfoAccess = OCSP;URI:http://10.1.0.2:2000

If the server (LDAP in this case) is configured for OCSP, then when it sees the AIA extension, it sends a request to the OCSP server at the URI (10.1.0.2:2000) which responds saying “good” , “revoked” or “unknown”.

OCSP server.

Openssl provides a simple ocsp server. You give it

  • the port it is to use
  • the CA public certificate (so the incoming certificate can be validated against the CA)
  • a file for that CA of the certificates it has issued, and it they are valid or not.

You can also configure the LDAP server to give a default URI, for those certificates that do not have the Authority Info Access(AIA) section. Typically this only works if you have one Certificate Authority, or you have a smart OCSP server which can handle the data for multiple CAs.

For example if your openssl OSCP server is configured for CA with DN:CN=CA1,o=MYORG, and you send down a request for DN:CN=CA2,o=myorg, it will not recognize it.

If you use the AIA extension, then you can have two different OCSP servers, one for each CA.

How does it work?

There are two ways of doing OCSP checking.

  • The client looks at the certificate and sees there is an OCSP extension. The client sends a request to the OCSP server to check the certificate is valid.
  • As part of the TLS handshake send the extension to the server to say “please do OCSP checking for me”. The server issues the request to the OCSP server, and can cache the response in the server. The server also sends the OCSP status to the client as part of the TLS handshake. The next request for the certificate can use the cached response. It tends to improve performance, as it reduces the number of requests to the OCSP server, and the responses are cached in the server. This is known as OCSP stapling.

Does using OCSP kill performance?

Yes and no. I’ll cover this after “caching the response”

Caching the responses

The OCSP server may be within your organization, or it may be be external. The time to send a request and get the response back may range from milliseconds to seconds.

Servers can typically cache the response to an OCSP query.

The OCSP may update it’s certificate list every hour, or perhaps every day, so if it is refreshed once a day, your LDAP server, does not need to refresh its data more than once a day.

The openssl OCSP server has an option -nmin minutes| -ndays days which is how often to reread the file. It sends a response

SingleResponse
  certID
    hashAlgorithm (SHA-1)
    issuerNameHash: 157b5dd0bcdee5b5428e063cf29a1f4e45be7499
    issuerKeyHash: 5830af55c7b4d49fc9e3fac91441ef1fd7b215e3
    serialNumber: 587
  certStatus: good (0)
  thisUpdate: 2021-11-05 10:42:58 (UTC)
  nextUpdate: 2021-11-05 10:43:58 (UTC)

From this the requester knows when the cached value is no longer valid, and needs to contact the OCSP server again for that certificate. From this we can see that the response is valid for 60 seconds.

When the certificate was revoked the output was

Online Certificate Status Protocol
  responseStatus: successful (0)
  responseBytes
    ResponseType Id: 1.3.6.1.5.5.7.48.1.1 (id-pkix-ocsp-basic)
    BasicOCSPResponse
      tbsResponseData
        responderID: byName (1)
          byName: 0...
          producedAt: 2021-11-06 15:15:58 (UTC)
            responses: 1 item
              SingleResponse
                certID...
                  certStatus: revoked (1)
                    revoked
                      revocationTime: 2021-11-06 15:12:30 (UTC)
                      revocationReason: cessationOfOperation (5)
                    thisUpdate: 2021-11-06 15:15:58 (UTC)
                    nextUpdate: 2021-11-06 15:16:58 (UTC)
            responseExtensions: 1 item
              Extension
                Id: 1.3.6.1.5.5.7.48.1.2 (id-pkix-ocsp-nonce)
                ReOcspNonce: a7e59577b7d2b3a2
      signatureAlgorithm (ecdsa-with-SHA256)
        Padding: 0
        signature: 3046022100fb264d4c5dbbbf45cce752a8263c4f01631441...
      certs: 1 item...

This information is available to the client.

What about performance?

If you write your application properly the impact of OCSP can be minimized.

For example consider the scenario where you are using REST requests, and go to one of a number of server.

  • Your application starts
  • It needs to check the validity of the certificate. Send a request to the OCSP server. This could take a long time (seconds). Some applications have a time-out of 15 seconds waiting for a response.
  • The response comes back saying “certificate Good – and valid for 1 hour.
  • Send a request to the server, and get the response back
  • Issue another request (with another TLS handshake)
  • After 1 hour – resend the request to the OCSP server to re-validate.
  • Etc

OCSP stapling

OCSP stapling is very common now. Before this, clients themselves used to check the validity of the user’s certificate by contacting the OCSP server. With OCSP stapling, a request is put into the TLS handshake which says “please do the OCSP checks for me and send me the output”. This allows the server to cache the information.

Think of the old days of checking in at the airport, when the person checking you in, would staple “checked-in by agent 46” to your paper ticket.

The client requests this by adding the “status_request” extension to the TLS clientHello handshake.

The server sends down, as part of the “serverHello” the information it received from the OCSP server.

Note. ldapsearch, from openssl, sends up the status_request, but does not handle the response, I get

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1) additional info: (unknown error code)

openssl s_client does not send the status_request extension, so does not participate in the OCSP checking.

Java does support this, and a Java application can get the OCSP response message using the getStatusResponses method on the from the ExtendedSSLSession. I believe you can decode it using Bouncycastle.

Setting up LDAP on z/OS to support OCSP

I added the following to a working LDAP system

Environment

GSK_OCSP_URL=http://10.1.0.2:2000
GSK_OCSP_ENABLE=ON
GSK_OCSP_CLIENT_CACHE_SIZE=100
GSK_REVOCATION_SECURITY_LEVEL=MEDIUM
GSK_SERVER_OCSP_STAPLING=OFF
GSK_OCSP_RESPONSE_SIGALG_PAIRS=0603060105030501040304020401

#GSK_OCSP_RESPONSE_SIGALG_PAIRS=0806080508040603060105030501040304020401

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Stapling

I set GSK_SERVER_OCSP_STAPLING=OFF because ldapsearch on Ubunutu did not work with the ENDENTITY value.

Signature Algorithms

If GSK_OCSP_RESPONSE_SIGALG_PAIRS included any of 0806 0805 0804, I got messages

GLD1160E Unable to initialize the LDAP client SSL support: Error 113, Reason -99.
GLD1063E Unable to initialize the SSL environment: 466 – Signature algorithm pair is not valid.

In the trace I had 03353003 Cryptographic algorithm is not supported, despite these being listed in the documentation.

Nonce

The nonce is used to reduce replay attacks. In your request to the OCSP serve you include a nonce (string of data). You expect this in the response message.

The default in LDAP is off!

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Configuration

In the configuration file I had

sslKeyRingFile START1/MQRING

sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

Setting up the OCSP server on Linux

My CA was on Linux – address 10.1.0.2.

I set up a bash script to run the openssl OCSP server on Linx,

ca=”-CA ca256.pem”
index=”-index index.txt”
port=”-port 2000″
#rsigner=”-rsigner rsaca256.pem –rkey rsaca256.key.pem”
#rsigner=”-rsigner ecec.pem –rkey ecec.key.pem”
rsigner=”-rsigner ss.pem –rkey ss.key.pem”
nextUpdate=”-nmin 1″
openssl ocsp $index $ca $port $rsigner $nextUpdate

You need

  • CA … for the Certificate Authority .pem file
  • -index index.txt, for the status of the certificates issued by the CA
  • -port … a port to use
  • -rsigner… the public key to be used when responding
  • -rkey … the private key for encrypting the response.
  • -nmin … how often the index file is refreshed. Typically this value might be an hour or more.

It does not log any activity, so I had to use Wireshark to trace the network traffic.

Problems using a certificate signed by the CA, for encrypting the response.

The rsigner certificate needs to have

Extended Key Usage: critical, OCSP Signing

Without this I got the following in the gsktrace

ERROR check_ocsp_signer_extensions(): extended keyUsage does not allow OCSP Signing

A self signed certificate worked OK.

Testing it

I had several challenges when testing it

  • ldapsearch on Linux sends up the “I support OCSP stapling”, but it objects to the response, and ends with unknown error code.
  • openssl s_client does not send the OCSP flag, and so the certificate does not get validated.
  • Java worked. I used this as a basis, and made a few changes to reflect my system. I needed to use the following optuons to run it
    • -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ecec.p12
    • -Djavax.net.ssl.keyStorePassword=password
    • -Djavax.net.ssl.keyStoreType=pkcs12
    • -Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/dantrust.p12
    • -Djavax.net.ssl.trustStorePassword=password
    • -Djavax.net.ssl.trustStoreType=pkcs12

Once I change GSK_SERVER_OCSP_STAPLING=ENDENTITY to GSK_SERVER_OCSP_STAPLING=OFF, I was able to use LDAPSEARCH.

Some OCSP certificates didn’t work

In my OCSP server, I used a certificate signed by the CA, for encrypting the response back to LDAP.

In the GSKtrace I got

ERROR find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
ERROR crypto_ec_token_public_key_verify(): ICSF service failure: CSFPPKV retCode = 0x4, rsnCode = 0x2af8
ERROR crypto_ec_token_public_key_verify(): Signature failed verification
ERROR crypto_verify_data_signature(): crypto_ec_verify_data_signature() failed: Error 0x03353004

There are two connected problems here

  1. Find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
  2. retcode 0x2af8 (11000) The digital signature verify ICSF callable service completed successfully but the supplied digital signature failed verification.8 (11000). I did not have the correct CA certificate for the OCSP certificate in the LDAP keyring.

Setting up CRL on z/OS LDAP

Certificate Revocation Lists (CRLs) are an old (and deprecated) way of checking the validity of certificates. I thought this would be a good way of understanding LDAP on z/OS.

As usual this had some surprises for me,such as having to recreate my CA, and certificates that it had issued, because it was missing the crlSign attribute.

The MQ documentation was a good start (I could not find any other documentation).

How do you know if your certificate is valid?

  • The old way of doing this (from about 2008!) was to have a list containing the Certificate Authority’s certificate, and a list of the certificate serial numbers which were revoked. This was called the Certificate Revocation List(CRL). In some cases this might just be a file, but typically this was stored in LDAP. Periodically you would refresh this list.
  • If may be that an intermediate CA was revoked. As well as a Certificate Revocation List you had Authority Revocation List(ARL) (from Certificate Authority).
  • People realised that refreshing the CRL every day was leaving a certificate exposed for up to a day. Also this file could get very large.
  • These days the checks are done “online”, using an Online Certificate Status Protocol(OCSP). A certificate has an extension with information “ask this site… if I am still valid”.
  • This has been extended to support the server caching the response to the OCSP request, and sending the OCSP status down to the client so the client knows how long the status of the certificate is valid for. Eventually the OCSP status expires, and the OCSP status has to be renewed.

Revoking a certificate

I wanted to revoke a certificate that I had created on Linux, so it could not be used for authentication.

This was pretty painless, except for a couple of “ahh” moments.

I set up a bash script to revoke one of my client’s certificates, and create the file of revoked certificates.

password=” -passin file:password.file -passout file:password.file”
name=”ecec”
ca=”ca256″ # sign with this
config=”-config openssl-ca-user.cnf”
policy=”-policy signing_policy”
reason=”-crl_reason cessationOfOperation”

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem $reason –revoke $name.pem

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem –gencrl -out crl.pem

This revokes the ecec.pem file, which was signed by ca256.*. I picked a reason cessationOfOperation from the list of reasons.

The index.txt file for the CA, shows the certificate is revoked “R”

R 200229103805Z 211102120448Z, cessationOfOperation 492DF59CE0479E9523C9355077E1DD6F83C4B01E unknown /C=GB/O=cpwebuser/CN=ecec

The second command (–gencrl) produced a file crl.pem (base 64 encoded) with a list of revoked certificates:

—–BEGIN X509 CRL—–
MIIBIzCByQIBATAKBggqhk…
3vs3Bcdkrw==
—–END X509 CRL—–

You can display the file contents using

openssl crl -text -in crl.pem

For example 

Certificate Revocation List (CRL):
    Version 2 (0x1)
    Signature Algorithm: ecdsa-with-SHA256
    Issuer: C = GB, O = SSS, OU = CA, CN = SSCA256
    Last Update: Nov 2 15:33:19 2021 GMT
    Next Update: Dec 2 15:33:19 2021 GMT
Revoked Certificates:
  Serial Number: 01
    Revocation Date: Jun 19 10:46:32 2019 GMT
  Serial Number: 023F
    Revocation Date: Nov 2 11:55:32 2021 GMT
  Serial Number: 0240
    Revocation Date: Nov 2 15:33:10 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Serial Number: 492DF59CE0479E9523C9355077E1DD6F83C4B01E
    Revocation Date: Nov 2 12:04:48 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Signature Algorithm: ecdsa-with-SHA256
    30:44:02:20:21:f4:70:11:42:85:22:c8:41:b9:95:d2:3d:9e:
    0a:b4:69:fa:a4:e0:37:81:05:c7:e8:98:43:a1:dc:67:58:6c:
    02:20:14:33:bd:2e:e1:57:3b:76:22:57:b9:b2:c4:a7:29:a3:
    3c:b1:4e:5a:a3:13:bc:13:61:e2:9e:a5:7c:64:b1:b1

I ftped the crl.pem file to z/OS.

Revoking a certificate on RACF

RACF does not have the support for revoking certificates. You are meant to use z/OS® Cryptographic Services PKI Services, to manage your certificates – which looks a non trivial installation.

Configure LDAP.

The CRL information is stored under the DN of the CA. My CA is called

C = GB, O = SSS, OU = CA, CN = SSCA256

So I need an entry in the LDAP tree for cn=SSCA256, ou=CA, o=SSS, c=GB.

My ldap configuration included

database LDBM GLDBLD31/GLDBLD64
useNativeAuth all
suffix “o=Your Company”
databaseDirectory /var/ldap/ldbm

the suffix means that all entries are below “o=Your Company” for example

dn: cn=ibmuser, o=Your Company

To support the CA, I added

suffix “c=GB” to the configuration file and restarted LDAP.

I created the elements for the CA name by using an .ldif file with.

dn: c=GB
objectclass: country
c: GB

dn: o=SSS, c=GB
o: SSS
objectclass: top
objectclass: organization

dn: ou=CA, o=SSS, c=GB
ou: CA

and the CA specific CRL information

dn: cn=SSCA256, ou=CA, o=SSS, c=GB
cn: SSCA256
objectclass: cRLDistributionPoint
certificateRevocationList;binary:< file:///crl1.ldif

This page says the z/OS LDAP client LDIF parser does not support the file:// URL format. 

I modified the ldif to be 

dn: CN=SSCA256,OU=CA,O=SSS,C=GB 
cn: SSCA256 
objectclass: cRLDistributionPoint 
objectclass: certificationAuthority 
cacertificate:: 
 MIIB2DCCAX+gAwIBAgI
...
certificateRevocationList:: 
 MIIBRTCB7QIBATA
...

Where the data contents are indented one character.

I copied the contents .pem file from the CA, and the content of the CRL.pem file into the .ldif file.

I used ldapmodify… to update LDAP with the .ldif files.

I tried using the revoked certificate on Linux to issue an LDAPsearch – and it worked! (where it should have failed).

The gsktrace had

Checking revoked status CN=ecec,O=cpwebuser,C=GB

CRL cannot be located

More configuration on z/OS

The CRL processing works as follows

  • I use a certificate to connect to LDAP.
  • LDAP calls GSKit to verify the certificate, map it to a RACF userid etc.
  • If GSKIT has been configured for CRL checking, it issue a request to the CRL server to get the CRL for the CA. This could be my LDAP, or another LDAP somewhere else. It took a little while to work out, my client came in to LDAP, called GSKIT, which called the same LDAP!

Configure GSKIT

In the LDAP environment file I added

GSK_LDAP_SERVER=127.0.0.1
GSK_LDAP_PORT=389
GSK_LDAP_USER=cn=ldapid,o=Your Company
GSK_LDAP_PASSWORD=passwr0d

This does a query asking for the CA certificate, and the CRL data for my Certificate Authority’s DN.

This did not work, I got the message in the LDAP job log

GLD1116E Unable to initialize an SSL connection 8 – Certificate validation error.

In the GSK trace I had

ERROR check_crl_issuer_extensions(): crlSign bit is not set in KeyUsage
ERROR check_revoked(): Unable to verify CRL issuer extensions: Error 0x03353026

My Linux CA was not defined properly. I was missing cRLSign in my CA definition

keyUsage = keyCertSign, digitalSignature,cRLSign

I did the following

  • Recreated my CA certificate
  • Sent the updated CA to my backend systems.
  • Update the certificate on z/OS and added it to the keyrings
  • Recreated the user certificate on Linux
  • Tried the ldapsearch to z/OS

To my pleasant surprise this failed to work – as I hoped.

My Ubuntu client had

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

and there was a message in the LDAP joblog

GLD1116E Unable to initialize an SSL connection with 10.1.0.2: 431 – Certificate is revoked.

In my gsktrace file I had

ERROR check_revoked(): RFC 2459 Revoked certificate check failed: Error 0x03353041

and from 03353041 you get

03353041 Certificate is revoked.

Explanation: A certificate is revoked and cannot be used.
User response: Obtain a new certificate from the certification authority.

You have not finished yet!

Now you know the process, of getting a CRL from the CA, and updating LDAP with it you need to consider some more things.

  • You need automation to generate the CRL, and update LDAP.
  • You may want multiple LDAP servers for your organisation.
  • You should consider a DN userid for your GSK, which can only access the CA information, and no other, so even if people saw the userid and password, they could not do much with it.
  • You need to protect the data, so the default access is none, but your GSK DN can read it, and you automatic process can update it.

Hindsight.

Since I got it working, I realised that I should have a better name for my CAs, such as O= SSS, OU=CA, c=GB, CN = SSCA256 instead of C = GB, O = SSS, OU = CA, CN = SSCA256.

It is worth going through the whole processed to find out the problems, so you minimise the amount of rework. For example missing keyUsage = cRLSign, caused a lot of rework (several times till I got it right).

Using z/OS LDAP with TLS 1.3

This blog post started as one line in the Using z/OS LDAP with TLS post. After it took two days to get working, I thought it deserved its own page.

Displaying the handshake

Using Wireshark to display TLS traffic is harder with TLS 1.3, as most of the handshake is now encrypted. See here on how to display it.

What supports LDAP and TLS 1.3?

I could not get Openldap on Ubuntu 18.04 to use TLS 1.3 – I do not think it has the support.

I could use the z/OS LDAPSEARCH with TLS 1.3, see below.

Java should work with the right set up.

Updating LDAP server on z/OS

Updates to enable TLS V1.3 protocol support for GSKIT has some good information.

You need to add the following to the configuration file

sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

# TLS 1.3 only supports sslFipsState off
slFipsState off

Below are the environment variables I used for the LDAP server on z/OS

GSK_TRACE=0xff
GSK_TRACE_FILE=/tmp/gsktrace.ldap
GSK_PROTOCOL_TLSV1_1=off
GSK_PROTOCOL_TLSV1_2=off
GSK_PROTOCOL_TLSV1_3=on
GSK_SERVER_TLS_KEY_SHARES=00300029002500240023
GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023130313011302
GSK_TLS_SIG_ALG_PAIRS=060105010401030108060805080405030403

I think this is the super set of data, and should be used as a starter list. You can remove unwanted cipher specs once you have it working. It is easier doing it this way compared to adding those you need (because it is hard to find what you need).

GSK_V3_CIPHER_SPECS_EXPANDED

You need to add the TLS 1.3 cipher specs to the GSK_V3_CIPHER_SPECS_EXPANDED in the environment file.

For example

GSK_V3_CIPHER_SPECS_EXPANDED=13011302C028C0271303

GSK_TLS_SIG_ALG_PAIRS

This variable needs to include 0806,0805,0804.

Key share

TLSv1.3 uses client and server key-shares to facilitate the encryption of TLSv1.3 handshake messages and to determine the key exchange algorithms.

You give a preference as to which elliptic curve to use.

You need to list the key shared values (from this list (table 5)), where  secp256r1 has the value 0023.

GSK_SERVER_TLS_KEY_SHARES is used by the server. The client has GSK_CLIENT_TLS_KEY_SHARES

GSK_SERVER_TLS_KEY_SHARES=00300029002500240023

or

GSK_CLIENT_TLS_KEY_SHARES=00300029002500240023

Remove any definitions in the keyshares which are not supported by TLS 1.3 (0019 and 0021) (even though you have GSK_PROTOCOL_TLSV1_2=on)

With these my LDAP started.

Once you have LDAP working, you can review the cipher specs, and remove the ones you no longer use. You should plan to remove the weaker cipher specs. If you remove a cipher spec and it is still needed, clients will fail to connect. You will need to replace the cipher specs and restart the server.

Using openssl s_client on Linux to establish a session to LDAP on z/OS.

Openssl s_client sends the TLS 1.3 supported cipher specs as part of the initial client hello handshake, so you can see the flows.

When I did this I got

s_client fails with SSL alert number 42.

From the z/OS GSK trace I can see

ERROR check_cert_extensions_3280_and_later(): authorityKeyIdentifier extension is missing
ERROR validate_certificate_basics(): Unable to verify certificate extensions: Error 0x0335306f
ERROR validate_certificate_mode(): Unable to validate certificate: Error 0x0335306f
ERROR cms_validate_certificate_mode_int(): Unable to validate certificate: Error 0x0335306f

ERROR read_tls13_certificate(): Unable to validate peer certificate: Error 0x0335306f

ERROR send_tls13_alert(): Sent TLS 1.3 alert 42 to 10.1.0.2[42514]

Where 0x0335306f (search for 0335306f) is

Explanation

A certificate extension that is mandatory for the certificate to be used for the required purpose has not been found.

User response

Ensure that the certificate chain is correct and complies with the validation mode defined for the connection. Collect a System SSL trace containing the error and then contact your service representative if the error persists.

A certificate authority can use multiple certificates to sign a certificate, for example you are refreshing or upgrading the certificate. Having the authorityKeyIdentifier says which certificate was used to sign it.
On the web there are documents which recommend its use, for example with openssl, the configuration file should have.

[ v3_ca ]
subjectKeyIdentifier=hash

authorityKeyIdentifier=keyid:always,issuer:always

On z/OS with RACF you get this attribute when using RACDCERT GENCERT … SIGNWITH() .

On my system I had misconfigured it

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer


I had to

  • create a new CA with the above in the Openssl configuration file
  • reissue the end user certificate
  • update all my servers with the new CA
  • update my LDAP CRL server

Wow – what a lot of work for a user error.

When I had done the above work, I used the

openssl x509 -in rsaca256.pem -text -noout

to list the end user certificate, and there was additional information

X509v3 Authority Key Identifier:
keyid:B3:0C:14:E9:C4:7D:8D:C1:45:9D:57:CE:19:8C:6B:0A:6C:EF:00:78
DirName:/C=GB/O=SSS/OU=CA/CN=SSCARSA1024
serial:03:DB:6B:09:E5:38:41:7D:8C:F3:CB:62:CA:1A:DD:63:CE:7E:C6:24

Using openLDAP ldapsearch on Ubuntu 18.04

I had problems getting ldapsearch on Linux to work with TLS 1.3, because it was difficult to get TLS 1.3 support – it did not send the TLS 1.3 cipher specs as part of the Client Hello. The documentation describes TLS_CIPHER_SUITE option. The Openssl syntax did not work, and the GnuTLS Syntax did work but I could not get it working with TLS 1.3 – perhaps it is not supported.

Using ldapsearch from z/OS

This was easy to set up, and I could use the gsktrace from each end to observe the flows.

I set up a bash script

export GSK_TRACE=0xff
export GSK_TRACE_FILE=./colin.gsktrace
export GSK_PROTOCOL_TLSV1_1=”off”
export GSK_PROTOCOL_TLSV1_2=”off”
export GSK_PROTOCOL_TLSV1_3=”on”
export GSK_CLIENT_TLS_KEY_SHARES=”003000290025002400230021″
export GSK_TLS_SIG_ALG_PAIRS=”0601050104010301080608050804050304030603″
export LDAP_SSL_CIPHER_FORMAT=”CHAR4″
export GSK_CLIENT_ECURVE_LIST=”00230024002500290030″
export GSK_V3_CIPHER_SPECS_EXPANDED=”130213011303C02BC02CC030C028C02aC024C026C02E”
debug=”-d all”
debug=”-d error+conns”
tls=”-Z -K adcda/MQRING -N ADCDRSA -S EXTERNAL”
tls=”-Z -K adcda/MQRING -N ADCDEC -S EXTERNAL”
host=”-h 127.0.0.1 -p 1389″
host=”-h 10.1.0.2 -p 9443″
ldapsearch $host $tls $debug -b “o=Your Company” “(objectclass=*)” aclEntry

gsktrace colin.* > out
oedit out

I defined an environment variable more than once, so I just needed to move the line down to change what was used. For example by swapping over the “tls” lines I could quickly use a different certificate.

I added the commands to format the gsktrace and display it.

Output

Because I used the debug value “-d error+conns” I got

CONNS ldap_ssl_socket_init()1600: 4-byte ciphers format will be used
CONNS ldap_ssl_socket_init()1625: No 4-byte cipher specs specified in LDAP handle
CONNS ldap_ssl_socket_init()1689: SSL Protocol Version = 621
CONNS ldap_ssl_socket_init()1700: SSL Cipher Specs
Address: 1FA36946 Length: 4 (x’4′)
00000000: f1f3f0f3 *1303 *
ERROR ldap_parse_pwdpolicy_response_int()856: No server controls o=Your Company

I think these messages are OK, and do not show a problem.

You can see that cipher spec 1303 was used.

Other information

TLS 1.3 uses one of the following ciphers

  • TLS_AES_128_GCM_SHA256 -1301
  • TLS_AES_256_GCM_SHA384 – 1302
  • TLS_CHACHA20_POLY1305_SHA256 1303
  • TLS_AES_128_CCM_SHA256 -1304
  • TLS_AES_128_CCM_8_SHA256 -1305

How do I look at TLS 1.3 handshakes?

With TLS before 1.3, encryption of the data on the session occurred after the handshake had completed, so the handshake was visible in Wireshark. With TLS 1.3 the traffic is encrypted after the “Client Hello”, so you cannot immediately see the remainder of the handshake.

Tools like OpenSSL, can write out the magic data needed for decryption. For example

openssl s_client -keylogfile /tmp/kl -connect 10.1.1.2:1389 -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile /home/colinpaice/ssl/ssl2/colinpaice.pem

This writes information to the specified file, in my case /tmp/kl.

The file has data like

SSL/TLS secrets log file, generated by OpenSSL
SERVER_HANDSHAKE_TRAFFIC_SECRET 05a42762…
EXPORTER_SECRET 05a42762dc…
SERVER_TRAFFIC_SECRET_0 05a4276…
CLIENT_HANDSHAKE_TRAFFIC_SECRET 05a42762dc…
CLIENT_TRAFFIC_SECRET_0 05a42762d…

On Linux (Ubuntu) you can tell Wireshark to use this through

edit -> preferences -> protocols -> SSL -> (pre)-master-secret log filename

Different versions of Wireshark have TLS or SSL, use whichever one is available to you.

Specify the name of your file (/tmp/kl in my case) and Wireshark will be able to decrypt the data.

Setting up certificate authentication in LDAP.

This started off as part of a small task, when I had half an hour gap before lunch. The whole end-to-end of getting TLS and LDAP, with certificate authentication took me several weeks to set up. Now I know the traps, it takes about 10 minutes.

I describe setting up TLS and LDAP (without certificate authentication) here. Get that working before trying certificate authentication.

Setting up the simplest case of an RSA certificate on the client and an RSA certificate on the server, was pretty easy to set up. Using an Elliptic Curve certificate to and RSA certificate on the server seems impossible, it eventually worked!

I created “What cipher specs should I use?” because most of my problems, were due to using the wrong cipher specs, or the right cipher specs, but it the wrong order!

Logging on

You can logon to LDAP and specify a userid (DN) and password, for example

ldapsearch -h 127.0.0.1 -D “cn=Admin, o=Your Company” -w secret -b “o=Your Company” “(objectclass=*)” aclEntry

If you use -w ? it will prompt for your password, so it is not visible.

You can also use a certificate to logon, so you do not need the password, you just need the private key (or in my case the USB dongle with my encrypted Hardware Security Module(HSM) keystore on it).

Note: If you have your TLS private key in a file, and people can copy that file, they can impersonate you! You need to protect the file, bearing in mind your corporate IT department may be able to view any backups etc that you have. Someone would need to steal my USB dongle to use my private key and logon.

Understanding the TLS 1.2 handshake and authentication.

Skip to first steps if you are keen to implement without understanding the background,

There are several stages to establishing a TLS connection and authentication.

  1. Agree the protocols for setting up the session, for example which sort of encryption, and the key size. This provides the privacy on the connection.
  2. The server sends down its certificate, and the client authenticates it
  3. The client sends up its certificate and the server authenticates it
  4. The Distinguished Name(DN)from the client certificate is looked up in the z/OS security manager, and the associated userid is looked up.
  5. The DN is used to look in the access control lists (ACLs) do determine the access the requester has to the data.

Agree the protocols

The client the sends the protocols it supports to the server. This is a list of numbers, and each number has a meaning.

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_GCM_SHA384 (0xc087)
…
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
…

The part before the WITH covers

  1. The technique used to agree a key (ECDHE, DHE,RSA)
  2. The key type Elliptic Curve (ECDSA) or RSA

There must be at least one key type matching the certificate type.

  • If you have an Elliptic Curve server certificate, then you need to have records with TLS…ECDSA…
  • If you have an RSA server certificate you need to have records with TLS…RSA…

At the server, the list sent from the client is merged with the list from the server, for example if the server had

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0250039

and the client sent C024, C025, C02C, then common elements are C024, C025. These two cipher specs are TLS_ECDHE_ECDSA_WITH… so the server’s certificate needs to be ECDSA – or an Elliptic Certificate.

If no entries match you get a message like “no cipher found”.

Authenticate the server

The server sends information down to the client.

  1. The cipher suite to be used.
  2. Information for setting up the encryption of the traffic for the session.
  3. The server’s certificate, and any CA certificates in the chain.
  4. The client then validates the certificate
    1. It use the signature algorithm in the certificate. This is the information after the “WITH” in the cipher spec. In my client’s certificate it has signing algorithm sha256WithRSAEncryption. There must be a cipher spec in the list ending in TLS…RSA_WITH… ending in SHA256
    2. It checks the signature of the CA, by checking its signing algorithm in the GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) See here.

Send the client certificate to the server

If the client certificate is wanted, the servers sends the “certificate request”.

  1. It sends down the certificate type it accepts, for example RSA, ECDSA, DSS. The client then sends its certificate of that type. If the there is no match, no certificate is sent. Note:For a long time I could not get a client certificate with Elliptic Curve, to work when the server had an RSA certificate. This was due to bad parameters on the z/OS end.
  2. There is a list of signature Hash Algorithms (eg rsa_pkcs1_sha256 (0x0401) ecdsa_secp256r1_sha256 ( 0x0403).
  3. It lists the Distinguished Names of the Certificate Authorities or Self Signed certificate.
  4. With the certificate type,the signature Hash and the Certificate Authorities, the client looks in its keystore for a certificate matching these parameters; then sends the first certificate that matches – or sends “no certificate”.
  5. It sends the Certificate Verify Signature Algorithm to the server. For my Elliptic Curve this was ecdsa_secp256r1_sha256 (0x0403). This has to be in the GSK_TLS_SIG_ALG_PAIRS in the z/OS LDAP environment file. (It took me days to find this problem!)

The server authenticates the client’s certificate

The server looks inside the certificate to see how the certificate was signed, for example using

openssl x509 -in ecec.pem -text -noout

This gave Signature Algorithm: ecdsa-with-SHA256. The server needs the matching cipher spec in the GSK_V3_CIPHER_SPECS_EXPANDED environment variable.

The client’s certificate will have been signed – either by a CA, or self signed. The Signature algorithm of the CA must be in the GSK_TLS_SIG_ALG_PAIRS environment variable. The values are defined here.

For example my CA had Signature Algorithm: ecdsa-with-SHA384. This requires 0503 (SHA-384 with ECDSA) to be in the GSK_TLS_SIG_ALG_PAIRS list. If this was missing I got

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

from ldapsearch on Ubuntu. It took me another day or so to find this problem.

As you can see there are a lot of things you need to configure and to get right before it works!

First steps – using a client certificate

Firstly set up the TLS session so you can use certificates to connect to LDAP. This took me about a week, because of configuration problems, but finally it worked. I was able to connect from Ubuntu to z/OS.

Set up TLS between the client and the z/OS server, as described here.

Setup Ubuntu

I created an RSA certificate on Ubuntu using a shell script.

name=”rsaca256″
ca=”carsa1024″

openssl genpkey -out $name.key.pem -algorithm RSA -pkeyopt rsa_keygen_bits:2048

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name -passin file:password.file -passout file:password.file
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

I set up an Elliptic Curve

enddate=”-enddate 20220409174001Z”

password=” -passin file:password.file -passout file:password.file”
name=”ecec”
rm $name.key.pem
rm $name.csr
rm $name.pem
ca=”ca256″ # sign with this

openssl ecparam -name secp256r1 -genkey -noout -out $name.key.pem

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name $password
openssl ca -config openssl-ca-user.cnf -policy signing_policy $ext -md sha256 -cert $ca.pem -keyfile $ca.key.pem -out $name.pem -in $name.csr $enddate -extensions clientServer

openssl x509 -in $name.pem -text -noout|less

# Some tools need a .p12 file of the certificate
#openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $ca.pem -chain -name $name -passout file:password.file -passin file:password.file

Openldap uses a configuration file, for example ldaprc.

I set up my ldaprc configuration file on Ubuntu.

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_REQCERT demand
TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem
TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem

Where

  • TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem was the z/OS CA certificate exported from z/OS and downloaded. It is needed to verify a certificate sent from z/OS.
  • TLS_KEY /home/colinpaice/ssl/ssl2/secp521r.key.pem this file contains the private key
  • TLS_CERT /home/colinpaice/ssl/ssl2/secp521r.pem this file contains the public key
  • You can specify TLS_CIPHER_SUITE high to use the listed cipher specs. See the openssl ciphers command, for example
    • openssl ciphers -v -V -s -tls1_2
    • openssl ciphers -v -V -s -tls1_3
    • I omitted TLS_CIPHER_SUITE for my testing.

You need the distinguished name (DN) of the certificate. You can display the subject distinguished name DN using the command openssl x509 -in secp521r.pem -text -noout. This gave me output which included:

Subject: C = GB, O = cpwebuser, CN = secp521r

I sent the Linux CA file, ca256.pem, to z/OS.

Set up z/OS

Configure LDAP.

In my LDAP configuration file I had

sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING
sslCertificate ZZZZ
commThreads 10
allowAnonymousBinds off
sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

The sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED says look in the environment file, and take the value from there. As this is the default, you can omit it from the file.

The default GSK_V3_CIPHER_SPECS_EXPANDED value in the environment file is 003500380039002F00320033 which maps to

  1. 0035 TLS_RSA_WITH_AES_256_CBC_SHA
  2. 0038 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
  3. 0039 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  4. 002f TLS_RSA_WITH_AES_128_CBC_SHA
  5. 0032 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
  6. 0033 TLS_DHE_RSA_WITH_AES_128_CBC_SHA

Which are all very old (part of deprecated SSL, TLS 1.0, TLS 1.1) so you need to set this variable in your environment file. See the next section for what I used.

Update your environment file

I use the following value

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023130313011302

This supports

  1. C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  2. C02B TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  3. C030 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  4. C02F TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  5. C024 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
  6. C023 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  7. 1303 TLS_CHACHA20_POLY1305_SHA256 TLS 1.3
  8. 1301 TLS_AES_128_GCM_SHA256 TLS 1.3
  9. 1302 TLS_AES_256_GCM_SHA384 TLS 1.3

Put the strongest certificates first. For example, the following did not work.

GSK_V3_CIPHER_SPECS_EXPANDED=0035….C02F

It caused 0035 to be used. The Certificate-Types list had only RSA and DSS, and did not include ECDSA, so the client was not able to send its Elliptic Curve certificate.

When I used

GSK_V3_CIPHER_SPECS_EXPANDED=C02F….0035

it caused C02F to be used. This caused the certificate-types list to include the ECDSA, and so the client could send its Elliptic Curve certificate. This problem took me a week to find!

I specified

GSK_TLS_SIG_ALG_PAIRS=0601050104010301080608050804

For the certificate checking algorithms. These are for SHA = 224, 256, 384, 512 and certificate types = RSA, ECDSA, and DSA.

Add the CA from Linux and add it to the keyring

The Linux CA was uploaded to z/OS as COLIN.CARSA102.PEM.

Add it to the RACF keystore as a CERTAUTH certificate, and connect it to the keyring.

RACDCERT CHECKCERT('COLIN.CARSA102.PEM') 

RACDCERT DELETE (LABEL('LINUXCA')) CERTAUTH

RACDCERT ADD('COLIN.CARSA102.PEM') -
  CERTAUTH WITHLABEL('LINUXCA') TRUST

RACDCERT ID(START1) CONNECT(RING(MQRING)-
           CERTAUTH  LABEL('LINUXCA'))

SETROPTS RACLIST(DIGTCERT) REFRESH

Map the Distinguished Name to a z/OS userid.

You have to map the DN to a userid. In the example below, it maps the DN in the SDNFILTER to the userid ADCDA. This userid must be valid, for example if the userid is revoked, the authentication will fail (with an unhelpful message).

* remove any previous instance
RACDCERT DELMAP(LABEL('LINUXSecp2' )) ID(ADCDA)

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

RACDCERT MAP ID(ADCDA )  - 
   SDNFILTER('CN=secp521r.O=cpwebuser.C=GB')  - 
   WITHLABEL('LINUXSecp2') 

RACDCERT LISTMAP ID(ADCDA) 

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

Note the ‘.’ between the attributes, and the attributes are squashed up, in the right order.

I did not need to refresh LDAP as the change was automatically picked up.

I could now logon to LDAP using the certificate – but could do nothing.

Give the DN access to LDAP

I gave it access, using ldapmodify passing the file with

dn: o=Your Company
changetype: modify
add: aclEntry
aclEntry : access-id:CN=SECP521R,O=CPWEBUSER,C=GB:
 object:ad:normal:grant:rscw:sensitive:rsc

Note: The DN can be in mixed case, or upper case.

I could now logon and issue queries using a certificate for authentication and authorisation.

Tracing problems

You can get trace output to the job log using the command

f GLDSRV,debug error+CONNS+LDAPBE

This traces

  • any errors
  • CONNS shows the cipher spec being used
  • LDAPBE shows the DN, and SAF user etc being used.

You can turn trace off with

f GLDSRV,debug 0

You can get the low level trace by using the gsktrace=0xff in your environment file.

Using z/OS LDAP with TLS

This started as a mini project before lunch, and now, two weeks later, I have got all of the bits to work.

As I was struggling with this, I wondered why no one had crossed this swamp before me. Is it because no one wants to cross it, or it is too difficult to do so? I think it was just difficult to do so – but I can now provide a map!

It took me a couple of hours to get LDAP on z/OS to use TLS from my Linux machine. The challenge was finding the right options. I then tried using Elliptical Certificates, and got this working, then was partially successful in using TLS 1.3.

As part of this I wrote Authentication using certificates with LDAP, which is where it was really difficult.

For clients, I used ldapsearch (from openldap ) on Ubuntu, and ldapsearch on z/OS.

See the product documentation here.

I’ve put a section “What cipher specs should I use” in a separate blog post.

Configure the LDAP environment file

I had to add TLS V1_2 and TLSV1_3 (and disable the deprecated TLS 1.1)

GSK_PROTOCOL_TLSV1_1=off
GSK_PROTOCOL_TLSV1_2=on
#GSK_PROTOCOL_TLSV1_3=on

to the environment file, because they default to off. Without these options it defaults to SSL – which is so old, my certificates and keys did not support it. I disabled TLS_1_1 because it is old.

I added

GSK_TLS_SIG_ALG_PAIRS=0601060305010401030108060805080405030403

This is used when validating certificates. The signature algorithm of the CA must be in this list see GSK_TLS_SIG_ALG_PAIRS parameter. I found it easiest to specify most of the available options 0601060305010503040104030402″. (0601, 0603…) (see here), get things working, then remove the ones I did not want to use; rather than try to add the ones I wanted to use.

I initially added

GSK_V3_CIPHER_SPECS_EXPANDED=006BC006C007C008c024c023c0251303130113020039

This is a list of the cipher specs it will accept. You should try to restrict the certificates in your organisation, to Elliptical Certificates, and a limited configuration for any CA’s. For example have

  • all client’s certificates defined as Elliptic Curve type prime256v1, signed with ecdsa-with-SHA256.
  • All internal CAs defined as Elliptic Curve type prime256v1 signed with ecdsa-with-SHA256.

When using certificate authentication I needed the following for it all to work

GSK_V3_CIPHER_SPECS_EXPANDED=C02CC02BC030C02FC024C023

The environment variables

GSK_SERVER_TLS_KEY_SHARES
GSK_CLIENT_TLS_KEY_SHARES

these are used only when using TLS 1.3. I’ll blog about using TLS 1.3 at a later date.

While you are investigating problems you can use the environment variable GSK_TRACE=0xff to write a trace. You can specify GSK_TRACE_FILE to direct the trace to a particular file. The default location is /tmp/gskssl.%.trc where % is the thread id. You format it with

gsktrace /tmp/gskssl.65567.trc > out

oedit out

I used

GSK_TRACE=0xff
GSK_TRACE_FILE=/tmp/gsktrace.ldap

So I could have a shell script to format and display the trace.

gsktrace gsktrace.ldap > out
oedit out

Change the LDAP configuration file.

I used

sslCipherSpecs all
sslMapCertificate check fail
sslAuth serverClientAuth
listen ldap://:389
listen ldaps://:1389
sslKeyRingFile START1/MQRING

I specified a port for ldaps. You can use

ldapsearch -H ldaps://10.1.1.2:389

This is preferred to specifying host and port

ldapsearch -h 10.1.1.2 -p 389

A port can process TLS and non-TLS requests, so definition for port 1389 may not needed.

I specified a keyring sslKeyRingFile START1/MQRING. This is owned by a different userid to the started task, so needs more RACF definitions. See below for the definitions.

Define the keyring and keyring

I defined an RSA certificate and an EC certificate. I controlled which was used from the configuration file.

* create the ring
RACDCERT DELRING(MQRING) ID(START1)
RACDCERT ADDRING(MQRING) ID(START1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the server's RSA certificate

RACDCERT ID(START1) DELETE(LABEL('ZZZZ'))
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('ZZZZ') - 
             O('SERVER') - 
             OU('SSS')) - 
   ALTNAME(IP(10.1.1.2))- 
   SIZE(4096)- 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   RSA - 
   WITHLABEL('ZZZZ') - 
   KEYUSAGE(         DATAENCRYPT,DOCSIGN,HANDSHAKE         ) 

RACDCERT id(START1) ALTER(LABEL('ZZZZ'))TRUST 
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('ZZZZ') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

* and the Elliptic Certificate

RACDCERT ID(START1) DELETE(LABEL('SERVEREC')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('SERVEREC') - 
             O('ADCD') - 
             OU('TEST')) - 
   ALTNAME(IP(10.1.1.2)) - 
   SIZE(521) - 
   NISTECC - 
   SIGNWITH (CERTAUTH LABEL('COLIN-CA')) - 
   KEYUSAGE(HANDSHAKE         ,KEYAGREE) - 
   NOTAFTER(   DATE(2024-12-29))- 
   WITHLABEL('SERVEREC') 
RACDCERT id(START1)  ALTER(LABEL('SERVEREC'))TRUST
                                                        
RACDCERT ID(START1) CONNECT(RING(MQRING) - 
                            ID(START1)  - 
                            LABEL('SERVEREC') DEFAULT) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 
                                                             
RACDCERT LISTRING(MQRING) ID(START1)

Permit the LDAP server’s userid to access the private certificate in the keyring

Because the LDAP userid does not own the keyring, the server userid needs update access to be able to extract the private key.

RDEFINE RDATALIB START1.MQRING.LST UACC(NONE) 
PERMIT START1.MQRING.LST CLASS(RDATALIB) ID(GLDSRV) -
    ACCESS(CONTROL)
SETROPTS RACLIST(RDATALIB) REFRESH

For a userid to be able to use its own keyring, it only need read access to be able to use the private certificate.

Set up the client userid

I set up the userid in the z/OS LDAP server, so I could use the z/OS userid and password. See here.

Test it out

LDAP checks the TLS parameters, when a TLS request arrives. It does not check at startup . I used openssl s_client to connect, and display information about the TLS handshake. See Debugging TLS – an easier way.

Setting up the client on Ubuntu

I set up a file ldaprc containing the ldap configuration. You can enter all of the parameters on the command line, but the ldaprc file makes it easier. My file had

TLS_CACERT /home/colinpaice/ssl/ssl2/colinca.pem
URI ldaps://10.1.1.2:1389
TLS_PROTOCOL_MIN 3.2
TLS_REQCERT demand
#TLS_CERT /home/colinpaice/ssl/ssl2/rsaca256.pem
#TLS_KEY /home/colinpaice/ssl/ssl2/rsaca256.key.pem

The #TLSCERT etc are ignored. No certificate is sent to the server (unless the -Y external option is set).

Enable GSK trace on z/OS

I enabled an LDAP trace to //SYSPRINT on z/OS, using f GLDSRV,debug error+CONNS+LDAPBE

Where

  • error reports errors
  • Conns reports connection information, such as the cipher spec used.
  • LDAPBE reports on the communication to the back end, for example it reports the SAFUSER

Run a query on Linux

I used ldapsearch on Ubuntu

ldapsearch -H ldaps://10.1.1.2:1389 -D “cn=ibmuser, o=Your Company” -w panthe0n -b “o=Your Company” “(objectclass=*)” aclEntry

The trace on z/OS GLDSRV included

CONNS srv_ssl_connect()644: SSL Cipher Specs
00000000: c3f0f0f8 *C008 *

and we can see that SSL was used, and cipher spec C008 was used.

Because I had set up my userid to map use a SAF userid, in the trace I had

LDAPBE srv_process_bind_request()939: do_return_bind msgID=1, connID=8, bindDN=‘CN=IBMUSER, O=YOUR COMPANY’, safUserID=’IBMUSER’, dnList=0x0, grpList=0x0, rc=0