Extending the HTTPD server on z/OS

The HTTPD web server is the Apache web server ported to run on z/OS. It runs in Unix Services, and behaves like a proper z/OS program, for example it can use z/OS userids and keyrings.

The configuration is easy, it is text driven (rather than XML), can imbed other configuration files, and can substitute variables.

I found the Apache documentation was as good, but the z/OS documentation was not very good. I prefer baby steps, taking the smallest system and adding functions, rather than configure everything and be disappointed when fails to work.

This post follows on Getting started with httpd server on z/OS and describes how to configure your first web page. Other posts on HTTPD server

Baby step number 2 – extending it

My HTTPD instance directory is /u/mqweb3/ .

I like to keep any changes I make to a configuration file, in a different file, and include this file in the original file. This way, if the original file changes, I just have to add the include statement rather than “diff” the my config file with the new config file. I also like to logically group changes, so my TLS configuration are in the tls.conf file, my definitions for port 8800 are in a file 8800.conf.

In the /u/mqweb3/conf is the httpd.conf file.

I edited this, and inserted

Include conf/colin.conf

at the bottom of the file.

I created /u/mqweb3/conf/colin.conf with

LogLevel debug

ErrorLog “/u/mqweb3/conf/error.log”

LoadModule rewrite_module modules/mod_rewrite.so
LoadModule authnz_saf_module modules/mod_authnz_saf.so
LoadModule ibm_ssl_module modules/mod_ibm_ssl.so

<Location /server-status>
AuthName “Colins Page”
AuthType Basic
AuthBasicProvider saf
Require valid-user
AuthSAFExpiration “EXPIRED! oldpw/newpw/newpw”
AuthSAFReEnter “Enter new password one more time”
CharsetSourceEnc IBM-1047
CharsetDefault ISO8859-1
SetHandler server-status
</Location>

The LoadModules provide the SAF support for logon

Shutdown the server (use P HTTPPCP6) – do not just cancel it – as there are several address spaces running for the server.

Restart the server (or do S httpcp,action=’restart’ ), fix any problems and try logging on to

http://10.1.1.2:8300/server-status

This should prompt for userid and password, and display the status of the server. The web browsers remember the userid and password, so if you want to reuse the page it will not prompt you for the userid and password. To change to a different userid and password you will need to restart the browser.

While playing with pages and logging on, I found the curl request

curl -u colin:password -i http://10.1.1.2:8830/xxxx.html

a good way of checking the page out, and logging on each time, as the password is not saved.

If you get

BPXP015I HFS PROGRAM /usr/lpp/ihsa_zos/bin/httpd IS NOT MARKED PROGRAM
CONTROLLED.
BPXP014I ENVIRONMENT MUST BE CONTROLLED FOR SERVER (BPX.SERVER)
PROCESSING.

You need to use the command

extattr +p /usr/lpp/ihsa_zos/bin/httpd

Create your a virtual host (container)

The HTTPD server can support multiple ports, and treat them as isolated environments. These are known as Virtual Hosts.

In my colin.conf I add

Include conf/vhost8831.conf

I created a file vhost8831.conf

Listen 8831
<VirtualHost *.8831>

<Location /xxxx.html>

#ServerName Colins.com
AuthName colinvh
AuthType Basic
AuthBasicProvider saf
#Require valid-user

Require saf-user COLIN JOE

# CharsetSourceEnc IBM-1047
# CharsetDefault ISO8859-1
# SetHandler server-status
</Location>

<Directory “/u/mweb3/htdocs”>
Require saf-user COLIN JOE
# Require saf-group SYS1

DocumentRoot “/u/mqweb3/htdocs”
#DirectoryIndex index_ihs.html

</Directory>
ErrorLog “/u/mqweb3/conf/zz.log”
ErrorLogFormat “[%t] [%l] %F: %E: [client %a] %M”

</VirtualHost>

Only userids COLIN and JOE are authorised to this (http://10.1.1.2:8831/xxxx.html service).

Restart the server

If you are authorised to use this service you will get

The requested URL was not found on this server.

Because this has not been set up yet.

The DocumentRoot works with the URL to identify a file.

The URL

http://10.1.1.2:8831/xxxx.html

will look for xxxx.html in DocumentRoot so it looks for file /u/mqweb3/htdocs/xxxx.html .

Baby steps 3 – create a page.

Create a file /u/mqweb3/htdocs/xxxx.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4//EN"> 
<html lang="en"> 
<head> 
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> 
<title>Colin's page</title> 
</head> 
<body width="778" height="556" background="images/ihs/background.gif" resize="no" scroll="no"> 
<h1>Colins header</h1> 
<p>Colin</p> 
</body> 
</html>

Retry the web browser page.

When this is displayed you get

Where

  • The title maps to the page heading
  • The background image comes from the body

Getting started with HTTPD server on z/OS

Apache web server

The HTTPD web server is the Apache web server ported to run on z/OS. It runs in Unix Services, and behaves like a proper z/OS program, for example it can use z/OS userids and keyrings. It starts in seconds!

The configuration is easy, it is text driven (rather than XML), can imbed other configuration files, and can substitute variables.

I found the Apache documentation was very good, but the z/OS documentation was not as good. I prefer baby steps, taking the smallest system and adding functions, rather than configure everything and be disappointed when it fails to work first time.

Other posts

Getting started

I used the IBM HTTP Server – Powered by Apache Version 9 PDF document. The text below is an addition to the IBM documentation, not a replacement. I’m trying to fill the holes in the documentation.

The product comes pre-installed. Mine was in /usr/lpp/ihsa_zos/bin/httpd

You need a directory for your HTTPD instance.

You need the userid to run the server.

ADDUSER WEB2 NAME(‘COLWEB’) NOPASSWORD –
OMVS(AUTOUID ASSIZE(256000000) THREADS(512) –
Program(‘/bin/sh’) home(‘/u/mqweb3/’))

CONNECT IBMUSER GROUP(COLWEB )
CONNECT WEB2 GROUP(COLWEB )

When you set up the userid, it is better to use OMVS(AUTOUID .. than to give a specific numeric id, similarly use OMVS(AUTOGID… for the group.

The userid needs OMVS, PROGRAM= /bin/sh, and a home directory.

Setup the system with environment file

The /usr/lpp/ihsa_zos/bin/envvars file allows you to set up an image wide environment file. You can rename /usr/lpp/ihsa_zos/bin/envvars-std to /usr/lpp/ihsa_zos/bin/envvars

I set up this file with

#/!bin/sh
IHS=/usr/lpp/ihsa_zos
LIBPATH=$IHS/lib:$IHS/modules:$IHS
PATH=$IHS/bin
_EDC_ADD_ERRNO2=1
_BPX_SHAREAS=NO
_BPX_BATCH_SPAWN=YES
GSK_SSL_HW_DETECT_MESSAGE=1
LC_ALL=En_US.IBM-1047

Create the instance

Follow the instructions in the documentation.

cd /usr/lpp/ihsa*
umask 022
bin/install_ihs /u/mqweb3 8300

Note down the port you specified (8300) as you will need it when you try to connect to the server.

The documentation says switch to the instance directory and issue apachectl -v. This failed for me because the path and libpath were not set up. I set up the envvars file (above) and it worked.

/usr/lpp/ihs*/bin/apachectl -v

When I ran it, it produced

test: /usr/lpp/ihsa_zos/bin/apachectl 49: FSUM7351 not found
[: /usr/lpp/ihsa_zos/bin/apachectl 74: FSUM7351 not found
[: /usr/lpp/ihsa_zos/bin/apachectl 87: FSUM7351 not found
[: /usr/lpp/ihsa_zos/bin/apachectl 92: FSUM7351 not found
Server version: IBM_HTTP_Server/9.0.5.5 (Unix) (SMP/E, 64-bit)
Server built: Jun 10 2020 16:22:51

This is because it cannot find “echo” in the path. Check in the /usr/lpp/ihsa_zos/bin/envvars file. I had to use

LIBPATH=$IHS/lib:$IHS/modules:$IHS:/bin:/usr/sbin 
PATH=$IHS/bin:/bin:/usr/sbin

The apachectl takes the following options, so you can display the configuration, or modify the start.

Options:
-D name : define a name for use in directives
-d directory : specify an initial ServerRoot
-f file : specify an alternate ServerConfigFile
-C “directive” : process directive before reading config files
-c “directive” : process directive after reading config files
-e level : show startup errors of level (see LogLevel)
-E file : log startup errors to file
-v : show version number
-V : show compile settings
-h : list available command line options (this page)
-l : list compiled in modules
-L : list available configuration directives
-t -D DUMP_VHOSTS : show parsed vhost settings
-t -D DUMP_RUN_CFG : show parsed run settings
-S : a synonym for -t -D DUMP_VHOSTS -D DUMP_RUN_CFG
-t -D DUMP_MODULES : show all loaded modules
-M : a synonym for -t -D DUMP_MODULES
-t -D DUMP_SSL_CONFIG: show parsed SSL vhost configurations
-t -D DUMP_SSL_CIPHERS: show all known SSL ciphers
-t -D DUMP_UNIFIED_CONFIG: show configuration with all includes merged
-t -D DUMP_INCLUDES: show all included configuration files
-t : run syntax check for config files
-T : start without DocumentRoot(s) check
-X : debug mode (only one worker, do not detach)

-t -DDUMP_CONFIG is very useful as it shows what you have configured after any <If…> and after variable substitution. I use this as a standalone command to see what’s configured.

Note:I had problems using //STDENV in the started task, so I had to use the envvars file.

Create the JCL procedure

See the documentation.

//HTTPCP PROC ACTION='start',
// DIR='/usr/lpp/ihsa_zos',
// CONF='/u/mqweb3/conf/httpd.conf'
//*---------------------------------------------------------
//IHS EXEC PGM=BPXBATCH,REGION=0M,
// PARM='SH &DIR/bin/apachectl -k &ACTION -f &CONF -DNO_DETACH ',
// MEMLIMIT=1236M
//STDOUT DD SYSOUT=H
//STDERR DD SYSOUT=H
// PEND

BPXBATCH needs REGION=0M, and at least MEMLIMIT=1236M

Define the STARTED task to RACF.

RDEFINE STARTED HTTPCP* STDATA(USER(WEB2)
SETROPTS RACLIST(STARTED ) REFRESH

Start the started task

S HTTPCP

You cannot just type P HTTPCP. In the syslog I had

CRIHS0001I IHS S0W1 is active. 83951827 0.0.0.0:8300 unspecified:-1.
Use jobname HTTPCP6 for console commands.

This means you have to issue P HTTPCP6 to stop it.

You can also use

s HTTPCP,action=’start’
s HTTPCP,action=’stop’

To start and stop the server.
Note: When you run these commands it checks the syntax of the configuration file, and if there is a problem, then the command is not executed. I kept wondering why my HTTPD instance was not shutting down; it was because I had a configuration error, and so the stop request was being ignored.

If it starts (there are no helpful messages saying success) try connecting a web browser to it in my case

http://10.1.1.2:8300

This gave me page with links to IBM sites. If you get here – you have done the first baby step. You cannot do much with the server.

Being too strong can be a waste of time

The short story is to use Elliptic curves with size 256 or 384, and avoid wasting time investigating why using other sizes doesn’t always work.

The long story – part 1 the server

I had a RACF server certificate defined as

SIZE(521) NISTECC …

and using Chrome browser it failed. I did a lot of digging around and could see the gsktrace had

EXIT gsk_get_ec_parameters_info(): <— Exit status 0x00000000 (0) EC curve type 34, key size 521
ERROR send_v3_alert(): Sent SSL V3 alert 40 to 10.1.0.2[38736]

INFO edit_ciphers(): Server certificate ec curve 0034 not in supported ecurve tls extension. EC cipher suites disabled

Where curve type 34, also known as 0x0019 = secp521r.

When I changed the size to 256 it worked.

Digging in to this, the Client Hello part of the TLS handshake sent from Chrome had

Supported Groups (4 groups)
Supported Group: Reserved (GREASE) (0x2a2a)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)

This is missing the 0x0019.

For Firefox, which worked, it had

Supported Groups (6 groups)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)
Supported Group: secp521r1 (0x0019)
Supported Group: ffdhe2048 (0x0100)
Supported Group: ffdhe3072 (0x0101)

So after a couple of hours working on this ( and an overnight sleep-on-it) the solution was to use secp256r1 instead of secp521.

The long story – part 2 – the client

I also tried using a client certificate which was defined as a secp521r1. This has similar problems. When I defined it as secp256r1 or secp384r1 it worked.

The gsktrace included

INFO read_v3_certificate(): Certificate key algorithm 13, Signature algorithm 84
ENTRY gsk_get_ec_parameters_info(): —> keyInfo size 12
EXIT gsk_get_ec_parameters_info(): <— Exit status 0x00000000 (0) EC curve type 34, key size 521
ERROR read_v3_certificate(): Client certificate elliptic curve not in clients supported elliptic curve list

I also know more about the TLS hand shake – which has more twists and turns every time I look at it!

Certificate validation on LDAP using OCSP

It is good practice to validate certificates when security is important. There are typically two ways of doing this.

  • Have a central server with a list of all revoked certificates. This server has a Certificate Revocation List(CRL) which has to be maintained. This solution tends to be deprecated
  • When a certificate is created add an extension with “check using this URL… to check if the certificate is valid”. This field is added when the certificate is signed by the Certificate Authority. A request is sent to the OCSP server, and a response sent back. This technique is known as Online Certificate Status Protocol (OCSP). You need an OCSP server for each certificate authority.

The blog post explains about OCSP and some of the challenges in using it.

You can configure a certificate to have OCSP support.

When you create a signed certificate, the CA signs the certificate.

For example

openssl req -config eccert.config … -out $name.csr
openssl ca -config openssl-ca-user.cnf -policy signing_policy -in $name.csr -extensions xxxx

In the config file are different sections, in my xxx section is

[ xxxx ]
authorityInfoAccess = OCSP;URI:http://10.1.0.2:2000

If the server (LDAP in this case) is configured for OCSP, then when it sees the AIA extension, it sends a request to the OCSP server at the URI (10.1.0.2:2000) which responds saying “good” , “revoked” or “unknown”.

OCSP server.

Openssl provides a simple ocsp server. You give it

  • the port it is to use
  • the CA public certificate (so the incoming certificate can be validated against the CA)
  • a file for that CA of the certificates it has issued, and it they are valid or not.

You can also configure the LDAP server to give a default URI, for those certificates that do not have the Authority Info Access(AIA) section. Typically this only works if you have one Certificate Authority, or you have a smart OCSP server which can handle the data for multiple CAs.

For example if your openssl OSCP server is configured for CA with DN:CN=CA1,o=MYORG, and you send down a request for DN:CN=CA2,o=myorg, it will not recognize it.

If you use the AIA extension, then you can have two different OCSP servers, one for each CA.

How does it work?

There are two ways of doing OCSP checking.

  • The client looks at the certificate and sees there is an OCSP extension. The client sends a request to the OCSP server to check the certificate is valid.
  • As part of the TLS handshake send the extension to the server to say “please do OCSP checking for me”. The server issues the request to the OCSP server, and can cache the response in the server. The server also sends the OCSP status to the client as part of the TLS handshake. The next request for the certificate can use the cached response. It tends to improve performance, as it reduces the number of requests to the OCSP server, and the responses are cached in the server. This is known as OCSP stapling.

Does using OCSP kill performance?

Yes and no. I’ll cover this after “caching the response”

Caching the responses

The OCSP server may be within your organization, or it may be be external. The time to send a request and get the response back may range from milliseconds to seconds.

Servers can typically cache the response to an OCSP query.

The OCSP may update it’s certificate list every hour, or perhaps every day, so if it is refreshed once a day, your LDAP server, does not need to refresh its data more than once a day.

The openssl OCSP server has an option -nmin minutes| -ndays days which is how often to reread the file. It sends a response

SingleResponse
  certID
    hashAlgorithm (SHA-1)
    issuerNameHash: 157b5dd0bcdee5b5428e063cf29a1f4e45be7499
    issuerKeyHash: 5830af55c7b4d49fc9e3fac91441ef1fd7b215e3
    serialNumber: 587
  certStatus: good (0)
  thisUpdate: 2021-11-05 10:42:58 (UTC)
  nextUpdate: 2021-11-05 10:43:58 (UTC)

From this the requester knows when the cached value is no longer valid, and needs to contact the OCSP server again for that certificate. From this we can see that the response is valid for 60 seconds.

When the certificate was revoked the output was

Online Certificate Status Protocol
  responseStatus: successful (0)
  responseBytes
    ResponseType Id: 1.3.6.1.5.5.7.48.1.1 (id-pkix-ocsp-basic)
    BasicOCSPResponse
      tbsResponseData
        responderID: byName (1)
          byName: 0...
          producedAt: 2021-11-06 15:15:58 (UTC)
            responses: 1 item
              SingleResponse
                certID...
                  certStatus: revoked (1)
                    revoked
                      revocationTime: 2021-11-06 15:12:30 (UTC)
                      revocationReason: cessationOfOperation (5)
                    thisUpdate: 2021-11-06 15:15:58 (UTC)
                    nextUpdate: 2021-11-06 15:16:58 (UTC)
            responseExtensions: 1 item
              Extension
                Id: 1.3.6.1.5.5.7.48.1.2 (id-pkix-ocsp-nonce)
                ReOcspNonce: a7e59577b7d2b3a2
      signatureAlgorithm (ecdsa-with-SHA256)
        Padding: 0
        signature: 3046022100fb264d4c5dbbbf45cce752a8263c4f01631441...
      certs: 1 item...

This information is available to the client.

What about performance?

If you write your application properly the impact of OCSP can be minimized.

For example consider the scenario where you are using REST requests, and go to one of a number of server.

  • Your application starts
  • It needs to check the validity of the certificate. Send a request to the OCSP server. This could take a long time (seconds). Some applications have a time-out of 15 seconds waiting for a response.
  • The response comes back saying “certificate Good – and valid for 1 hour.
  • Send a request to the server, and get the response back
  • Issue another request (with another TLS handshake)
  • After 1 hour – resend the request to the OCSP server to re-validate.
  • Etc

OCSP stapling

OCSP stapling is very common now. Before this, clients themselves used to check the validity of the user’s certificate by contacting the OCSP server. With OCSP stapling, a request is put into the TLS handshake which says “please do the OCSP checks for me and send me the output”. This allows the server to cache the information.

Think of the old days of checking in at the airport, when the person checking you in, would staple “checked-in by agent 46” to your paper ticket.

The client requests this by adding the “status_request” extension to the TLS clientHello handshake.

The server sends down, as part of the “serverHello” the information it received from the OCSP server.

Note. ldapsearch, from openssl, sends up the status_request, but does not handle the response, I get

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1) additional info: (unknown error code)

openssl s_client does not send the status_request extension, so does not participate in the OCSP checking.

Java does support this, and a Java application can get the OCSP response message using the getStatusResponses method on the from the ExtendedSSLSession. I believe you can decode it using Bouncycastle.

Setting up LDAP on z/OS to support OCSP

I added the following to a working LDAP system

Environment

GSK_OCSP_URL=http://10.1.0.2:2000
GSK_OCSP_ENABLE=ON
GSK_OCSP_CLIENT_CACHE_SIZE=100
GSK_REVOCATION_SECURITY_LEVEL=MEDIUM
GSK_SERVER_OCSP_STAPLING=OFF
GSK_OCSP_RESPONSE_SIGALG_PAIRS=0603060105030501040304020401

#GSK_OCSP_RESPONSE_SIGALG_PAIRS=0806080508040603060105030501040304020401

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Stapling

I set GSK_SERVER_OCSP_STAPLING=OFF because ldapsearch on Ubunutu did not work with the ENDENTITY value.

Signature Algorithms

If GSK_OCSP_RESPONSE_SIGALG_PAIRS included any of 0806 0805 0804, I got messages

GLD1160E Unable to initialize the LDAP client SSL support: Error 113, Reason -99.
GLD1063E Unable to initialize the SSL environment: 466 – Signature algorithm pair is not valid.

In the trace I had 03353003 Cryptographic algorithm is not supported, despite these being listed in the documentation.

Nonce

The nonce is used to reduce replay attacks. In your request to the OCSP serve you include a nonce (string of data). You expect this in the response message.

The default in LDAP is off!

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Configuration

In the configuration file I had

sslKeyRingFile START1/MQRING

sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

Setting up the OCSP server on Linux

My CA was on Linux – address 10.1.0.2.

I set up a bash script to run the openssl OCSP server on Linx,

ca=”-CA ca256.pem”
index=”-index index.txt”
port=”-port 2000″
#rsigner=”-rsigner rsaca256.pem –rkey rsaca256.key.pem”
#rsigner=”-rsigner ecec.pem –rkey ecec.key.pem”
rsigner=”-rsigner ss.pem –rkey ss.key.pem”
nextUpdate=”-nmin 1″
openssl ocsp $index $ca $port $rsigner $nextUpdate

You need

  • CA … for the Certificate Authority .pem file
  • -index index.txt, for the status of the certificates issued by the CA
  • -port … a port to use
  • -rsigner… the public key to be used when responding
  • -rkey … the private key for encrypting the response.
  • -nmin … how often the index file is refreshed. Typically this value might be an hour or more.

It does not log any activity, so I had to use Wireshark to trace the network traffic.

Problems using a certificate signed by the CA, for encrypting the response.

The rsigner certificate needs to have

Extended Key Usage: critical, OCSP Signing

Without this I got the following in the gsktrace

ERROR check_ocsp_signer_extensions(): extended keyUsage does not allow OCSP Signing

A self signed certificate worked OK.

Testing it

I had several challenges when testing it

  • ldapsearch on Linux sends up the “I support OCSP stapling”, but it objects to the response, and ends with unknown error code.
  • openssl s_client does not send the OCSP flag, and so the certificate does not get validated.
  • Java worked. I used this as a basis, and made a few changes to reflect my system. I needed to use the following optuons to run it
    • -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ecec.p12
    • -Djavax.net.ssl.keyStorePassword=password
    • -Djavax.net.ssl.keyStoreType=pkcs12
    • -Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/dantrust.p12
    • -Djavax.net.ssl.trustStorePassword=password
    • -Djavax.net.ssl.trustStoreType=pkcs12

Once I change GSK_SERVER_OCSP_STAPLING=ENDENTITY to GSK_SERVER_OCSP_STAPLING=OFF, I was able to use LDAPSEARCH.

Some OCSP certificates didn’t work

In my OCSP server, I used a certificate signed by the CA, for encrypting the response back to LDAP.

In the GSKtrace I got

ERROR find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
ERROR crypto_ec_token_public_key_verify(): ICSF service failure: CSFPPKV retCode = 0x4, rsnCode = 0x2af8
ERROR crypto_ec_token_public_key_verify(): Signature failed verification
ERROR crypto_verify_data_signature(): crypto_ec_verify_data_signature() failed: Error 0x03353004

There are two connected problems here

  1. Find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
  2. retcode 0x2af8 (11000) The digital signature verify ICSF callable service completed successfully but the supplied digital signature failed verification.8 (11000). I did not have the correct CA certificate for the OCSP certificate in the LDAP keyring.

More about certificate authentication with LDAP.

Having got certificate authentication to work with LDAP on z/OS, see here, I found there are some more things you can do with it certificate authentication.

I had a certificate with Distinguished Name CN=ecec,O=cpwebuser,C=GB. This mapped to a RACF userid ADCDA.

I can get LDAP to generate a different DN racfid=ADCDA,profiletype=USER,o=myracf, based on this RACF userid. I can then use this DN in Access Control Lists etc..

Setting up LDAP

In my configuration file I set the sslMapCertificate line, added a section for SDBM, and restarted the server.

sslMapCertificate add fail

databaseDirectory /var/ldap/ldbm
#———– SDBM —————-
database sdbm GLDBSD31/GLDBSD64
suffix o=myracf

I set up the mapping of certificate to userid.

RACDCERT LISTMAP ID(ADCDA)

RACDCERT DELMAP(LABEL(‘LINUXECEC’ )) ID(ADCDA)

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

RACDCERT MAP ID(ADCDA ) –
SDNFILTER(‘CN=ecec.O=cpwebuser.C=GB’) –
WITHLABEL(‘LINUXECEC’)

RACDCERT LISTMAP ID(ADCDA)

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

The result

When the certificate was used to connect to LDAP, it generated a DN, DN=’RACFID=ADCDA,PROFILETYPE=USER,O=MYRACF‘, where ADCDA is the userid from the certificate mapping, and O=MYRACF comes from the suffix statement in the SDBM section.

If sslMapCertificate replace is specified, the DN from the certificate is discarded and replaced with the generated DN.

If sslMapCertificate add is specified, both the DN from the certificate and the generated DN, are used in the ACL check. The certificate DN is used in auditing etc

Things to think about

You might want to use sslMapCertificate ADD rather than replace, so audit statements have the original DN, and thus provide a better audit trail.

Don’t try this at home, PKI certificates.

I was trying to generate z/OS certificates which I could use to check out certificate revocation. I can do it in Linux – no problem. Getting it to work on z/OS was the challenge and I don’t think it can be done.

Meanwhile, back at the ranch (‘at home’), running z/OS on the zPDT environment on Linux, I found the PKI Server environment does not support PKI for key generation on zPDT or ZD&T, because zPDT does not support ICSF TKDS.

The longer story of why it will not work.

The RACF command RACDCERT GENCERT creates a basic certificate which is suitable for many uses. It does not support the extensions, such as specifying a URI for OCSP checking of the certificate.
The PKI product has these capabilities, and together with ICSF it can store the keys in the ICSF data sets. This product seems complex to set up (two web servers), and a GUI interface instead of a command interface.

There is an API, the RACF callable services R_PKISERV, which allows you to issue API requests to administer certificate.
You can use a SAF interface and pass in a public certificate, certificate request, or use PKI to generate a full certificate with all of the optional fields etc, and do full life cycle management with it.

I could not get this to work, and when I started the PKI SERVER, it reported

IKYP040I PKI SERVICES DOES NOT HAVE KEY GENERATION CAPABILITY

This in turn pointed me to ICSF and the TKDS (Token Key Data Set), not being set up – it needed a master key. You enter this master key on a TKE (Trusted Key Entry) workstation which sits inside the z hardware. I was running on zPDT, and following the trail, the zPDT documentation said PKCS#11 (Token Key Data Set) is not available with zPDT. This means it looks like I cannot create certificates which support OCSP on my z/OS.

How to delete a RACF group

It only took me 30 minutes to do so!

I was creating some RACF definitions for a product, and being a good citizen,I wanted to have a script which cleans up after me, and deletes anything I defined. This all worked fine, except for trying to delete a RACF group. For example

                                       
DELGROUP  PKIGRP3                            
IKJ56702I INVALID GROUP, PKIGRP3            
                                     
ADDGROUP PKIGRP3                            
IKJ56702I INVALID GROUP, PKIGRP3 

Hmm the message is not helpful – PKIGRP3 >IS< valid.

IKJ56702I INVALID invalid data
Explanation: The user entered invalid data.

I had a mini project to find out why it was not being deleted. If I list the group it gives

LG PKIGRP3                                                                                
INFORMATION FOR GROUP PKIGRP3                                                            
    SUPERIOR GROUP=SYS1         OWNER=COLIN       CREATED=21.315                          
    NO INSTALLATION DATA                                                                  
    NO MODEL DATA SET                                                                    
    TERMUACC                                                                              
    NO SUBGROUPS                                                                          
    USER(S)=      ACCESS=      ACCESS COUNT=      UNIVERSAL ACCESS=                      
      IBMUSER       USE           000000               NONE                              
         CONNECT ATTRIBUTES=NONE                                                          
         REVOKE DATE=NONE                 RESUME DATE=NONE                                

If I remove the userid from the group the delete group works

REMOVE IBMUSER GROUP(PKIGRP3)
READY
DELGROUP PKIGRP3
READY

Easy when you know why.

Having to remove all of the users from a group before deleting the group means I cannot just have code to delete the userids I had created, and delete the groups I created. I’ve raised an RFE on this.

Old dogs, new tricks, ISPF compare

When looking for a comparison facility for Unix files, I stumbled across ISPF edit compare function.

If you are editing a file you can use the compare * command to show the changes you made to the file in the current session. This gave me 

000001 line 1
====== line 3
.OAAAA line 2
.OAAAB line 4
.OAAAC line 5

Where

  • line 1 was unchanged
  • line 3 was deleted
  • line 2,4,5 were added

You can then use “mm” or “mdd …. mdd” prefix commands on the ====== to apply the changes to the file, and use “d” or “dd … dd” on the .O prefix lines to delete the lines.

You can use the compare command, and be prompted for options.
I specified a Unix file, and could see the differences.

You can set the exclude option (at the bottom of the panel), which says exclude all the lines, but display ‘n’ lines around changes, where ‘n’ is specified at the top of the panel.

One funny….

I noticed that compare was display “blank” lines in Unix files as being changed. The original file had a null line, with just an end of line. When I customised the file and saved it, ISPF replaces an “end of line” with “blank end of line”. As these do not match, they show up as being changed.

The solution is to edit the original file with null lines, make a dummy change, and save the file. The file will no longer have null lines, and only your changes will be visible.

ASCII files in OMVS

I found I was unable to compare python source files in OMVS, because they were ASCII files. (When the files were tagged as ASCII, I could edit them).

You cannot use ISPF compare if the files are ASCII (non EBCDIC). You can do

Compare the files then use the following to transform them back.

Using RACF ids in LDAP

I stumbled across this useful way of defining userids to LDAP.

In your LDAP configuration file, define the SDBM database

database sdbm GLDBSD31/GLDBSD64
suffix o=myracf

This says for all Distinguished Names (DN) ending in o=myracf, then go to the RACF (SAF) database.

If my DN is RACFID=COLIN,PROFILETYPE=user,o=myracf it will use this. I do not need to set up a special DN in LDAP.

I used

ldapsearch -h 127.0.0.1 -D “RACFID=colin,PROFILETYPE=user,o=myracf” -w ? -b “o=myracf” “(objectClass=*)”

To list all the userids and groups in RACF.

For those IDs which map to a SAF userid, for example

  • defined in LDAP with the attribute ibm-nativeId:
  • using a SAF userid directly RACFID=COLIN,PROFILETYPE=user,o=myracf,
  • via a certificate and RACDCERT MAP

that userid is used to issue the command. For example userid COLIN is a member of group SYS1, and can display information from commands like TSO LU ANOTHER.

Another ID with no special authority returned no data from the ldapsearch command above.

The SDBM backend is virtual directory and is mostly read-only, so the update operations are usually not allowed. RACF configuration is used to restrict a user’s authority to SDBM  and ACLs are not used.
SDBM only supports RACF user IDs or user IDs with RACF mappings, so a user must map to a SAF user to be able to query the RACF data in the SDMB database.

Possible queries

This table gives some LDAP queries.

  • -b “o=myracf” “(objectclass=*)” list all users groups etc
  • –b “profiletype=User,o=myracf” “(objectclass=*)” list all users
  • -b “profiletype=Group,o=myracf” “(objectclass=*)” list all users
  • -b “profiletype=Connect,o=myracf” “(objectclass=*)” list all user and group connections
  • -b “racfid=colin,profiletype=user,o=myracf” “(objectclass=*)” list everything about racfid COLIN
  • -b “cn=setropts,o=myracf” “(objectclass=*)” did not work for me
  • -b “profiletype=facility,o=myracf” “(objectclass=*)” did not work for me

Setting up CRL on z/OS LDAP

Certificate Revocation Lists (CRLs) are an old (and deprecated) way of checking the validity of certificates. I thought this would be a good way of understanding LDAP on z/OS.

As usual this had some surprises for me,such as having to recreate my CA, and certificates that it had issued, because it was missing the crlSign attribute.

The MQ documentation was a good start (I could not find any other documentation).

How do you know if your certificate is valid?

  • The old way of doing this (from about 2008!) was to have a list containing the Certificate Authority’s certificate, and a list of the certificate serial numbers which were revoked. This was called the Certificate Revocation List(CRL). In some cases this might just be a file, but typically this was stored in LDAP. Periodically you would refresh this list.
  • If may be that an intermediate CA was revoked. As well as a Certificate Revocation List you had Authority Revocation List(ARL) (from Certificate Authority).
  • People realised that refreshing the CRL every day was leaving a certificate exposed for up to a day. Also this file could get very large.
  • These days the checks are done “online”, using an Online Certificate Status Protocol(OCSP). A certificate has an extension with information “ask this site… if I am still valid”.
  • This has been extended to support the server caching the response to the OCSP request, and sending the OCSP status down to the client so the client knows how long the status of the certificate is valid for. Eventually the OCSP status expires, and the OCSP status has to be renewed.

Revoking a certificate

I wanted to revoke a certificate that I had created on Linux, so it could not be used for authentication.

This was pretty painless, except for a couple of “ahh” moments.

I set up a bash script to revoke one of my client’s certificates, and create the file of revoked certificates.

password=” -passin file:password.file -passout file:password.file”
name=”ecec”
ca=”ca256″ # sign with this
config=”-config openssl-ca-user.cnf”
policy=”-policy signing_policy”
reason=”-crl_reason cessationOfOperation”

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem $reason –revoke $name.pem

openssl ca $config $policy $ext –cert $ca.pem -keyfile $ca.key.pem –gencrl -out crl.pem

This revokes the ecec.pem file, which was signed by ca256.*. I picked a reason cessationOfOperation from the list of reasons.

The index.txt file for the CA, shows the certificate is revoked “R”

R 200229103805Z 211102120448Z, cessationOfOperation 492DF59CE0479E9523C9355077E1DD6F83C4B01E unknown /C=GB/O=cpwebuser/CN=ecec

The second command (–gencrl) produced a file crl.pem (base 64 encoded) with a list of revoked certificates:

—–BEGIN X509 CRL—–
MIIBIzCByQIBATAKBggqhk…
3vs3Bcdkrw==
—–END X509 CRL—–

You can display the file contents using

openssl crl -text -in crl.pem

For example 

Certificate Revocation List (CRL):
    Version 2 (0x1)
    Signature Algorithm: ecdsa-with-SHA256
    Issuer: C = GB, O = SSS, OU = CA, CN = SSCA256
    Last Update: Nov 2 15:33:19 2021 GMT
    Next Update: Dec 2 15:33:19 2021 GMT
Revoked Certificates:
  Serial Number: 01
    Revocation Date: Jun 19 10:46:32 2019 GMT
  Serial Number: 023F
    Revocation Date: Nov 2 11:55:32 2021 GMT
  Serial Number: 0240
    Revocation Date: Nov 2 15:33:10 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Serial Number: 492DF59CE0479E9523C9355077E1DD6F83C4B01E
    Revocation Date: Nov 2 12:04:48 2021 GMT
    CRL entry extensions:
      X509v3 CRL Reason Code:
        Cessation Of Operation
  Signature Algorithm: ecdsa-with-SHA256
    30:44:02:20:21:f4:70:11:42:85:22:c8:41:b9:95:d2:3d:9e:
    0a:b4:69:fa:a4:e0:37:81:05:c7:e8:98:43:a1:dc:67:58:6c:
    02:20:14:33:bd:2e:e1:57:3b:76:22:57:b9:b2:c4:a7:29:a3:
    3c:b1:4e:5a:a3:13:bc:13:61:e2:9e:a5:7c:64:b1:b1

I ftped the crl.pem file to z/OS.

Revoking a certificate on RACF

RACF does not have the support for revoking certificates. You are meant to use z/OS® Cryptographic Services PKI Services, to manage your certificates – which looks a non trivial installation.

Configure LDAP.

The CRL information is stored under the DN of the CA. My CA is called

C = GB, O = SSS, OU = CA, CN = SSCA256

So I need an entry in the LDAP tree for cn=SSCA256, ou=CA, o=SSS, c=GB.

My ldap configuration included

database LDBM GLDBLD31/GLDBLD64
useNativeAuth all
suffix “o=Your Company”
databaseDirectory /var/ldap/ldbm

the suffix means that all entries are below “o=Your Company” for example

dn: cn=ibmuser, o=Your Company

To support the CA, I added

suffix “c=GB” to the configuration file and restarted LDAP.

I created the elements for the CA name by using an .ldif file with.

dn: c=GB
objectclass: country
c: GB

dn: o=SSS, c=GB
o: SSS
objectclass: top
objectclass: organization

dn: ou=CA, o=SSS, c=GB
ou: CA

and the CA specific CRL information

dn: cn=SSCA256, ou=CA, o=SSS, c=GB
cn: SSCA256
objectclass: cRLDistributionPoint
certificateRevocationList;binary:< file:///crl1.ldif

This page says the z/OS LDAP client LDIF parser does not support the file:// URL format. 

I modified the ldif to be 

dn: CN=SSCA256,OU=CA,O=SSS,C=GB 
cn: SSCA256 
objectclass: cRLDistributionPoint 
objectclass: certificationAuthority 
cacertificate:: 
 MIIB2DCCAX+gAwIBAgI
...
certificateRevocationList:: 
 MIIBRTCB7QIBATA
...

Where the data contents are indented one character.

I copied the contents .pem file from the CA, and the content of the CRL.pem file into the .ldif file.

I used ldapmodify… to update LDAP with the .ldif files.

I tried using the revoked certificate on Linux to issue an LDAPsearch – and it worked! (where it should have failed).

The gsktrace had

Checking revoked status CN=ecec,O=cpwebuser,C=GB

CRL cannot be located

More configuration on z/OS

The CRL processing works as follows

  • I use a certificate to connect to LDAP.
  • LDAP calls GSKit to verify the certificate, map it to a RACF userid etc.
  • If GSKIT has been configured for CRL checking, it issue a request to the CRL server to get the CRL for the CA. This could be my LDAP, or another LDAP somewhere else. It took a little while to work out, my client came in to LDAP, called GSKIT, which called the same LDAP!

Configure GSKIT

In the LDAP environment file I added

GSK_LDAP_SERVER=127.0.0.1
GSK_LDAP_PORT=389
GSK_LDAP_USER=cn=ldapid,o=Your Company
GSK_LDAP_PASSWORD=passwr0d

This does a query asking for the CA certificate, and the CRL data for my Certificate Authority’s DN.

This did not work, I got the message in the LDAP job log

GLD1116E Unable to initialize an SSL connection 8 – Certificate validation error.

In the GSK trace I had

ERROR check_crl_issuer_extensions(): crlSign bit is not set in KeyUsage
ERROR check_revoked(): Unable to verify CRL issuer extensions: Error 0x03353026

My Linux CA was not defined properly. I was missing cRLSign in my CA definition

keyUsage = keyCertSign, digitalSignature,cRLSign

I did the following

  • Recreated my CA certificate
  • Sent the updated CA to my backend systems.
  • Update the certificate on z/OS and added it to the keyrings
  • Recreated the user certificate on Linux
  • Tried the ldapsearch to z/OS

To my pleasant surprise this failed to work – as I hoped.

My Ubuntu client had

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1)
additional info: A TLS fatal alert has been received.

and there was a message in the LDAP joblog

GLD1116E Unable to initialize an SSL connection with 10.1.0.2: 431 – Certificate is revoked.

In my gsktrace file I had

ERROR check_revoked(): RFC 2459 Revoked certificate check failed: Error 0x03353041

and from 03353041 you get

03353041 Certificate is revoked.

Explanation: A certificate is revoked and cannot be used.
User response: Obtain a new certificate from the certification authority.

You have not finished yet!

Now you know the process, of getting a CRL from the CA, and updating LDAP with it you need to consider some more things.

  • You need automation to generate the CRL, and update LDAP.
  • You may want multiple LDAP servers for your organisation.
  • You should consider a DN userid for your GSK, which can only access the CA information, and no other, so even if people saw the userid and password, they could not do much with it.
  • You need to protect the data, so the default access is none, but your GSK DN can read it, and you automatic process can update it.

Hindsight.

Since I got it working, I realised that I should have a better name for my CAs, such as O= SSS, OU=CA, c=GB, CN = SSCA256 instead of C = GB, O = SSS, OU = CA, CN = SSCA256.

It is worth going through the whole processed to find out the problems, so you minimise the amount of rework. For example missing keyUsage = cRLSign, caused a lot of rework (several times till I got it right).