PKI Server error messages

AH00526: … \xac

I got AH00526:

Syntax error on line … of…: RewriteRule: bad argument line ‘\xac/…

This was because I had ^ and I needed ¬ in the documents.

The USS command chtag -p /u/mqweb3/conf/443.conf gave me

untagged T=off /u/mqweb3/conf/443.conf

My 3270 emulator code page was Bracket CP 037 modified.

Using the ISPF edit command hex on, showed the correct hex data is x’5f’. This can display as ¬ or as ^ depending on your 3270 emulator.

IKYC901I Error 76677164 initializing ICL: The CA certificate in the ICL does not match the one in the keyring

I got this when I redefined my userids and keyrings. I recreated the VSAM files. Display the ICL VSAM file

export PATH=/usr/lpp/pkiserv/bin/
export LIBPATH=/usr/lpp/pkiserv/lib
export NLSPATH=/usr/lpp/pkiserv/lib//usr/lpp/nls/msg/%L/%N
/usr/lpp/pkiserv/bin/iclview -d \’PKISRVD.VSAM.ICL\’

Need to escape the data set name.

My file was empty, so I recreated the VSAM data sets.

IKYP022I Unable to register PKI Services for restart: Error 12, Reason 0x160

This server has not been set up for ARM (Automatic Restart Management). The return code makes no sense to me.

Ignore it.

IKYC009I LDAP post unsuccessful for object id = 101, state = 0x2150000, status =
581500960: No such object
IKYP039E DIRECTORY POST UNSUCCESSFUL. ERROR CODE = 581500960

Unable to get the CA DN from the LDAP server. Check the suffix (eg CN=PKICA,OU=SSS,O=ZZUR COMPANY

IKYP040I PKI SERVICES DOES NOT HAVE KEY GENERATION CAPABILITY

You are missing the TokenName such as

[SAF]
KeyRing=PKISRVD/CARING
TokenName=PKISRVD.PKIToken

in /etc/pkiserv/pkiserv.conf.

This in turn caused rc 0 safrc 8 racfrc 8 racfrs 64 with function GENCERT in IRRSPX00 (R_pkiserve).

In the PKISERVD log (with debug trace turned on)

POLICY IKYK001I Unexpected PKCS#11 icsfpkcs11::genKeyPair return code 0x190. The request is not processed.

CORE IKYC010I Error 791740499 returned from CP_NewKeysCreate: Unable to generate or store a public/private key pair through ICSF

CORE IKYC010I Error 791740499 returned from JNH_create_certificate: Unable to generate or store a public/private key pair through ICSF

httpd: SSL0222W: SSL Handshake Failed, No ciphers specified (no shared
ciphers or no shared protocols).

During a TLS handshake there was no matching certificate found for the client.

I added

SSLCipherSpec TLS_AES_128_GCM_SHA256
SSLCipherSpec TLS_AES_256_GCM_SHA384
SSLCipherSpec TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
SSLCipherSpec TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
SSLCipherSpec TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
SSLCipherSpec TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
SSLCipherSpec TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
SSLCipherSpec TLS_RSA_WITH_AES_128_GCM_SHA256
SSLCipherSpec TLS_RSA_WITH_AES_256_GCM_SHA384
SSLCipherSpec TLS_RSA_WITH_AES_128_CBC_SHA
SSLCipherSpec TLS_RSA_WITH_AES_256_CBC_SHA
SSLCipherSpec TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

ICH408I USER(…) GROUP(…) NAME(ADCDA )NOT AUTHORIZED TO ADMINISTER DIGITAL CERTIFICATES OR CERTIFICATE REQUESTS. READ DENIED

and

IKYI002I SAF Service IRRSPX00 Returned SAF RC = 8 RACF RC = 8 RACF RSN = 8 Request denied, not authorized.

The user issuing the request was not authorised to IRR.RPKISERV.PKIADMIN CLASS(FACILITY).

Note what the message says

  • READ DENIED
  • UPDATE DENIED

Use

tso rlist facility irr.RPKISERV.PKIADmin auth

and connect the userid ( if required) to a group or give the required access with

PERMIT IRR.RPKISERV.PKIADMIN CLASS(FACILITY)
ID(ADCDA ) ACCESS(read )

setropts raclist(FACILITY) refresh

(163) EDC5163I SAF/RACF
extract error. (errno2 = 0x0BE8081C ): …


pthread_security_applid_np(__CREATE_SECURITY_ENV,
__CERTIFICATE_IDENTITY, 32, …, NULL, 0,… returned -1,
errno 163 errno2 be8081c 0x0be8081

The userid being used is revoked

IEW2646W 5383 ESD RMODE(24) CONFLICTS WITH USER-SPECIFIED RMODE(ANY) FOR SECTION …. CLASS B_TEXT.
IEW2646W 5383 ESD RMODE(24) CONFLICTS WITH USER-SPECIFIED RMODE(ANY) FOR SECTION … CLASS B_LIT.

I got these trying to bind a C program. I also had an assembler stub which caused this problem.

I added the RMODE and AMODE to my assembler program and cured the problem

CALLPRTF RMODE ANY
CALLPRTF AMODE ANY

CALLPRTF CSECT

HTTPD configuring the config file

The HTTPD server has a configuration file. This has some great facilities for making it easy to manage the contents of the configuration file. In summary these are

  • Ability to include files into the configuration
  • Define and use symbols
  • If then..
  • Use of regular expressions

Use include files

You can use Include

  • Include /usr/colin/a.conf an explicit, fully qualified path.
  • Include conf/ssl.conf relative to the ServerRoot directory (which you specify at start up).
  • Include conf/vhosts/*.conf multiple files in the directory.

If the files do not exist, the server will fail to start.

You can also use IncludeOptional if the files do not exist, the processing continues

Define and use symbols

You can use Define

  • Define symbol
  • Define symbol myvalue

Use the symbol

Define myvalue “mydata”

DocumentRoot “/var/%{myvalue}”

If symbol has been Defined

You can check to see if a symbol has been defined

<IfDefine>symbol>

</IfDefine>

<IfDefine !symbol>

</IfDefine>

Use a symbol in an if

<If %{v} == “yes”>

</if>

If processing

You can use

  • <IfDefined.. to determine if a symbol has been defined
  • <If to use value
  • <IfDirective. Check to see if the directive was specified in a -D parameter at startup.

You can do

<If … >

</If>

<ElseIf…>

</Elseif>
<Else>

</Else>

You can do

<If ..>
Error ” message “
</If>

This produces

AH00526: Syntax error on line … of /u/mqweb3/conf…

and ends. It does not continue past this.

Regular expressions

Within many of the statements you can use regular expressions, see Apache Documentation and Wikipedia. For example, all .gif and .jpg files under any “images” directory could be written as “/images/.*(jpg|gif)$“.

Packaging files

If you have

Listen 1834
<VirtualHost *:1834>

</VirtualHost>
<VirtualHost *:1834>

</VirtualHost>

The second <VirtualHost is ignored.

Multiple <Location..> for the same resource name do not work.

Try to put common definitions in one file and include them where needed, for example the list of TLS certificates.

If the same value is used in multiple configuration files, make it a variable and set it once. If it is changed, it will be picked up automatically. You can display the configuration to see what is actually being used.

HTTPD, SAFAPPL and protecting web resources

The HTTPD server can check a userid’s access to a RACF APPLID to enforce checks on resources.

Setting it up to give access seemed trivial, setting it up to deny access took longer.

In my VirtualHost I had

SAFAPPLID ZZZ
AuthType Basic
AuthBasicProvider saf
SAFRunAs %%CLIENT%%
Require saf-user ADCDA
Require saf-group SYS1

This says

  • userids must have read access to the APPL profile ZZZ.
  • a request should include the userid and password as part of the request.
  • the userid must be ADCDA or in group SYS1.

If the RACF profile is not set up (or not set up properly) then access defaults to yes.

Setup the profile

rdefine APPL ZZZ uacc(NONE) NOTIFY(COLIN)
setropts raclist(APPL) refresh

The NOTIFY is to notify a user(COLIN) when a user is denied access to the resource. This is useful while testing to check authentication is working. A failed attempt gave me

ICH70004I USER(ADCDB) GROUP(ADCDGR) NAME(COLIN PAICE) ATTEMPTED ‘READ’ ACCESS OF ‘ZZZ’

You do not get a message if a user does not have the right access (as you do with other resources), so the NOTIFY seems the only way of finding out there is a problem.

If I logged on with certificate, the same checks were done.

To give a user access, (actually it is better to give the user’s group access)

permit ZZZ class(APPL) ID(WEB2) access(READ)
setropts raclist(APPL) refresh

Problems with SAFAPPLID

The SAFAPPLID statement is meant to be supported in directory, virtual host, and server sections, but it only accepted it in the <Directory… section.

For example the following fails to parse

<virtualHost *:8833>
SAFAPPLID ZZZ

with

AH00526: Syntax error on line 11 of /u/mqweb3/conf/notls.conf: SAFAPPLID not allowed here

Originally I defined APPL ZZZZZZZZ, but used ZZZZZZZ (7 Z’z not 8). And the application continued to have access to HTTPD. By specifying NOTIFY(COLIN) this notified me when the request failed.

With

<VirtualHost …>
LogLevel debug
ErrorLog “/u/mqweb3/conf/yy.log”

I got the following in the yy.log file

pthread_security_applid_np(__CREATE_SECURITY_ENV, __USERID_IDENTITY, 5, colin, …, 0, ZZZ) returned OK

From this I can see the userid “colin”, the SAFAPPLID “ZZZ”, and the return code “OK”.

Getting HTTPD server to work with TLS on z/OS

This is one of a series of blog posts on HTTPD.

The HTTPD web server is the Apache web server ported to run on z/OS. It runs in Unix Services, and behaves like a proper z/OS program, for example it can use z/OS userids and keyrings.

One catch is that there is Apache SSL, and IBM SSL. For example SSLProtocolEnable is part of IBM SSL support, and does not exist in Apache SSL; and SSLVerifyClient exists in Apache SSL, and not in IBM SSL support.

You need to know which options you need to use. With the wrong options you will get a message like

AH00526: Syntax error on line .. of … Invalid command … , perhaps misspelled or defined by a module not included in the server configuration

This post describes how to get TLS to work with the HTTPD server.

You can use TLS to encrypt the session data, and you can use TLS to use the client certificate as authentication.

The IBM httpd TLS options are described here.

Before you start

  • You need a keyring with certificates. I won’t cover this, as it well documented. I had problems using elliptic keys with size other than 256 and 384. See here.
  • You need to select a port for the TLS sessions. The default for TLS is 443. You may wish to use another port for isolation, and ease of management and configuration.
  • I set up a self contained Virtual Host for the TLS stuff, you do not need to do this.
  • Consider putting your common TLS definitions in a file and including it where needed. For example the list of TLS Ciphers, and the keyring. If you want to change the parameters, you change it once, and restart the server.
  • You can define the SSL parameters in the main server section of the configuration, or within a virtual host. A definition within a virtual host overrides the main server definitions.

Establish a TLS session to encrypt the session data.

Set up permissions

The started task userid needs access to read the keyring. Because WEB2 is not the owner of the ring (START1 is the owner), WEB2 needs CONTROL to get access to the private key.

permit START1.MQRING.LST class(RDATALIB) ID(WEB2) ACCESS(CONTROL)
setropts raclist(RDATALIB) refresh

PERMIT CSFOWH CLASS(CSFSERV) ID(WEB2) ACCESS(READ)
SETROPTS RACLIST(CSFSERV) REFRESH

I was using keyring START1.MQRING,and the httpd server userid is WEB2.

Setup the configuration

For the z/OS SSL support you need

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so

The Apache module mod_ssl.so does not exist on z/OS, so it, and the facilities it provides cannot be used.

I set up a configuration file tls.conf (and used Include conf/tls.conf in my colin.conf)

Listen 8832
<VirtualHost *:8832>
SSLEnable

# SSLTrace
<Location /xxxx.html>

</Location>

ErrorLog “/u/mqweb3/conf/tls.log”
ErrorLogFormat “[%{%X}t]![%l] %F: %E: [client %a] %M”

KeyFile /saf START1/MQRING#

# SSLVerifyClient None

SSLClientAuth optional

SSLProtocolEnable TLSv12
SSLServerCert SERVEREC
SSLCipherSpec TLS_AES_128_GCM_SHA256
SSLCipherSpec TLS_AES_256_GCM_SHA384

# TLS 1.3 cipher specs
# SSLCipherSpec TLS_CHACHA20_POLY1305_SHA256
# SSLCipherSpec TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
# SSLCipherSpec TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
SSLCipherSpec TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
SSLCipherSpec TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
SSLCipherSpec TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
SSLCipherSpec TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
SSLCipherSpec TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
SSLCipherSpec TLS_RSA_WITH_AES_128_GCM_SHA256
SSLCipherSpec TLS_RSA_WITH_AES_256_GCM_SHA384
SSLCipherSpec TLS_RSA_WITH_AES_128_CBC_SHA
SSLCipherSpec TLS_RSA_WITH_AES_256_CBC_SHA
SSLCipherSpec TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
SSLClientAuth none
TLS13Options ServerKeyRefresh=604800

</VirtualHost>

Debugging it

You can either have an SSL VirtualHost wide trace, or a server wide SSL trace (or both)

VirtualHost wide trace

You can use SSLTrace (see above) in your virtual host and it writes it to the error log for that virtual host.

Server wide trace

I changed the environment to create a GSK trace. I added

export GSK_TRACE=0x0f
export GSK_TRACE_FILE=/u/mqweb3/conf/httpd.gsktrace

to /usr/lpp/ihsa_zos/bin/envvars .

You use the gskit command gsktrace /u/mqweb3/conf/httpd.gsktrace > gsk_out to format it.

Testing it

If the httpd server starts, try connecting to it from a web browser. Don’t for get to use https:

https://10.1.1.2:8832

If you get

Internal Server Error

This means the TLS handshake worked, you just have not set up the backend application.

When I used

https://10.1.1.2:8832/xxxx.html

(matching the entry in my VirtualHost definition) I was prompted for userid and password.

Using a client certificate for authentication and identification.

You can use a certificate on a client to authenticate with the server, without having to enter a userid and password. The server needs the CA from the client to be able to authenticate the client certificate. I had problems using elliptic keys on the client with size other than 256 and 384. See here.
You need to set up the certificate in RACF to map from the certificate to a userid.

Map from certificate to userid

My certificate had DN CN=secp256r1,O=cpwebuser,C=GB.

I used the following to map it to userid ADCDA

RACDCERT LISTMAP ID(ADCDA)
RACDCERT DELMAP(LABEL(‘SECP256R1 )) ID(ADCDA)
RACDCERT MAP ID(ADCDA ) –
SDNFILTER(‘CN=secp256r1.O=cpwebuser.C=GB’) –
WITHLABEL(‘SECP256r1’)
RACDCERT LISTMAP ID(ADCDA)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

Note: In the certificate the DN is CN=secp256r1,O=cpwebuser, in the RACF command, the comma is replace with a period CN=secp256r1.O… I get it wrong every time!

To force the client to send a certificate you need

SSLClientAuth Required

instead of SSLClientAuth none in your <VirtualHost>…</VirtualHost>.

You also need to specify SAFRunAs

<Location /xxxx.html>
AuthName colinvh
AuthType Basic
AuthBasicProvider saf
SAFRunAs %%CERTIF_REQ%%
Require saf-user ADCDA
Require saf-user COLIN
</Location

The documentation said SAFRunAs can be in “directory, virtual host, server config” , I could only get it to be accepted in the location or the directory statement.

My certificate mapped to ADCDA userid, and so with this certificate I can display page xxxx.html.

SAFAPPLID didn’t work at first.

You can use SAFAPPLID you can say that a user needs access to a profile in the APPL class, for example PAYROLL. The default is OMVSAPPL.

Initially I could not get SAFAPPLID to work. This was due to a set up error. See here for more information.

AuthName didn’t work

When HTTPD prompts you for a userid and password, it is meant to display the authname as the title of the popup window, so you know which userid and password to specify. It didn’t display it for me. I could tell from the network traffic that the AuthName was sent down to the Chromium Browser as WWW-Authenticate: Basic realm=”colinvh“. I believe this is for integrity reasons – someone could change the value, and get you to enter “the wrong” credentials.

Now you’ve go it to work

You should consider

  • How many ports do you need to support.
  • Moving your TLS definitions into one configuration file, and include this where needed.
  • Removing the weak SSL cipher specs.
  • Moving to TLS 1.2 or above.

Documentation

HTTPD server – what is configured?

This is one of a series of blog posts on HTTPD.

You can run an Apache command, or display in the web browser, the active configuration with variables substituted, and If statements processed.

The server can dump configuration at startup. The best option was

-t -DDUMP_CONFIG

which displays the configuration with symbols substituted and after If processing. You can use this as part of your startup, or standalone, or as a web page. The web page displays more information.

/usr/lpp/ihs*/bin/apachectl -f /u/mqweb3/conf/httpd.conf -t -DDUMP_CONFIG > aa

This displays lots of good information. Below is an illustrative subset of what is available

  • Server settings
    • Server build:Jun 10 2020 16:22:51
    • Host Name/port:10.1.1.2:8831
    • Server Root: /u/mqweb3
  • Information about each module, for example ‘Module Name:mod_rewrite.c
    • Module directives, a list of all the actions supported eg
      • ‘RewriteRule – an URL-applied regexp-pattern and a substitution URL
    • Current configuration for that module, with If statements processed, and variables substituted
      • In file: /u/mqweb3/conf/80.conf
  • Information about each module, for example ‘Module Name:mod_ibm_ssl.c
    • Module directives, a list of all the actions supported eg
      • ‘SSLEnable – SSL is enabled for this server
    • Current configuration for that module, with If statements processed, and variables substituted. Line 10 was ‘Keyfile /saf ${pkiRing}’
      • In file: /u/mqweb3/conf/pkissl.conf   
      • 1:   SSLEnable  
      • 10:   Keyfile /saf START1/MQRING
      • etc

To run it as a web page, the configuration is below.

LoadModule info_module modules/mod_info.so
<Location /server-info>
SetHandler server-info
AuthType Basic
AuthBasicProvider saf
SAFRunAs PKISERV
</Location>

and the URL was

http://10.1.1.2:8831/server-info

Extending the HTTPD server on z/OS

The HTTPD web server is the Apache web server ported to run on z/OS. It runs in Unix Services, and behaves like a proper z/OS program, for example it can use z/OS userids and keyrings.

The configuration is easy, it is text driven (rather than XML), can imbed other configuration files, and can substitute variables.

I found the Apache documentation was as good, but the z/OS documentation was not very good. I prefer baby steps, taking the smallest system and adding functions, rather than configure everything and be disappointed when fails to work.

This post follows on Getting started with httpd server on z/OS and describes how to configure your first web page. Other posts on HTTPD server

Baby step number 2 – extending it

My HTTPD instance directory is /u/mqweb3/ .

I like to keep any changes I make to a configuration file, in a different file, and include this file in the original file. This way, if the original file changes, I just have to add the include statement rather than “diff” the my config file with the new config file. I also like to logically group changes, so my TLS configuration are in the tls.conf file, my definitions for port 8800 are in a file 8800.conf.

In the /u/mqweb3/conf is the httpd.conf file.

I edited this, and inserted

Include conf/colin.conf

at the bottom of the file.

I created /u/mqweb3/conf/colin.conf with

LogLevel debug

ErrorLog “/u/mqweb3/conf/error.log”

LoadModule rewrite_module modules/mod_rewrite.so
LoadModule authnz_saf_module modules/mod_authnz_saf.so
LoadModule ibm_ssl_module modules/mod_ibm_ssl.so

<Location /server-status>
AuthName “Colins Page”
AuthType Basic
AuthBasicProvider saf
Require valid-user
AuthSAFExpiration “EXPIRED! oldpw/newpw/newpw”
AuthSAFReEnter “Enter new password one more time”
CharsetSourceEnc IBM-1047
CharsetDefault ISO8859-1
SetHandler server-status
</Location>

The LoadModules provide the SAF support for logon

Shutdown the server (use P HTTPPCP6) – do not just cancel it – as there are several address spaces running for the server.

Restart the server (or do S httpcp,action=’restart’ ), fix any problems and try logging on to

http://10.1.1.2:8300/server-status

This should prompt for userid and password, and display the status of the server. The web browsers remember the userid and password, so if you want to reuse the page it will not prompt you for the userid and password. To change to a different userid and password you will need to restart the browser.

While playing with pages and logging on, I found the curl request

curl -u colin:password -i http://10.1.1.2:8830/xxxx.html

a good way of checking the page out, and logging on each time, as the password is not saved.

If you get

BPXP015I HFS PROGRAM /usr/lpp/ihsa_zos/bin/httpd IS NOT MARKED PROGRAM
CONTROLLED.
BPXP014I ENVIRONMENT MUST BE CONTROLLED FOR SERVER (BPX.SERVER)
PROCESSING.

You need to use the command

extattr +p /usr/lpp/ihsa_zos/bin/httpd

Create your a virtual host (container)

The HTTPD server can support multiple ports, and treat them as isolated environments. These are known as Virtual Hosts.

In my colin.conf I add

Include conf/vhost8831.conf

I created a file vhost8831.conf

Listen 8831
<VirtualHost *.8831>

<Location /xxxx.html>

#ServerName Colins.com
AuthName colinvh
AuthType Basic
AuthBasicProvider saf
#Require valid-user

Require saf-user COLIN JOE

# CharsetSourceEnc IBM-1047
# CharsetDefault ISO8859-1
# SetHandler server-status
</Location>

<Directory “/u/mweb3/htdocs”>
Require saf-user COLIN JOE
# Require saf-group SYS1

DocumentRoot “/u/mqweb3/htdocs”
#DirectoryIndex index_ihs.html

</Directory>
ErrorLog “/u/mqweb3/conf/zz.log”
ErrorLogFormat “[%t] [%l] %F: %E: [client %a] %M”

</VirtualHost>

Only userids COLIN and JOE are authorised to this (http://10.1.1.2:8831/xxxx.html service).

Restart the server

If you are authorised to use this service you will get

The requested URL was not found on this server.

Because this has not been set up yet.

The DocumentRoot works with the URL to identify a file.

The URL

http://10.1.1.2:8831/xxxx.html

will look for xxxx.html in DocumentRoot so it looks for file /u/mqweb3/htdocs/xxxx.html .

Baby steps 3 – create a page.

Create a file /u/mqweb3/htdocs/xxxx.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4//EN"> 
<html lang="en"> 
<head> 
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> 
<title>Colin's page</title> 
</head> 
<body width="778" height="556" background="images/ihs/background.gif" resize="no" scroll="no"> 
<h1>Colins header</h1> 
<p>Colin</p> 
</body> 
</html> 

Retry the web browser page.

When this is displayed you get

Where

  • The title maps to the page heading
  • The background image comes from the body

Getting started with HTTPD server on z/OS

Apache web server

The HTTPD web server is the Apache web server ported to run on z/OS. It runs in Unix Services, and behaves like a proper z/OS program, for example it can use z/OS userids and keyrings. It starts in seconds!

The configuration is easy, it is text driven (rather than XML), can imbed other configuration files, and can substitute variables.

I found the Apache documentation was very good, but the z/OS documentation was not as good. I prefer baby steps, taking the smallest system and adding functions, rather than configure everything and be disappointed when it fails to work first time.

Other posts

Getting started

I used the IBM HTTP Server – Powered by Apache Version 9 PDF document. The text below is an addition to the IBM documentation, not a replacement. I’m trying to fill the holes in the documentation.

The product comes pre-installed. Mine was in /usr/lpp/ihsa_zos/bin/httpd

You need a directory for your HTTPD instance.

You need the userid to run the server.

ADDUSER WEB2 NAME(‘COLWEB’) NOPASSWORD –
OMVS(AUTOUID ASSIZE(256000000) THREADS(512) –
Program(‘/bin/sh’) home(‘/u/mqweb3/’))

CONNECT IBMUSER GROUP(COLWEB )
CONNECT WEB2 GROUP(COLWEB )

When you set up the userid, it is better to use OMVS(AUTOUID .. than to give a specific numeric id, similarly use OMVS(AUTOGID… for the group.

The userid needs OMVS, PROGRAM= /bin/sh, and a home directory.

Setup the system with environment file

The /usr/lpp/ihsa_zos/bin/envvars file allows you to set up an image wide environment file. You can rename /usr/lpp/ihsa_zos/bin/envvars-std to /usr/lpp/ihsa_zos/bin/envvars

I set up this file with

#/!bin/sh
IHS=/usr/lpp/ihsa_zos
LIBPATH=$IHS/lib:$IHS/modules:$IHS
PATH=$IHS/bin
_EDC_ADD_ERRNO2=1
_BPX_SHAREAS=NO
_BPX_BATCH_SPAWN=YES
GSK_SSL_HW_DETECT_MESSAGE=1
LC_ALL=En_US.IBM-1047

Create the instance

Follow the instructions in the documentation.

cd /usr/lpp/ihsa*
umask 022
bin/install_ihs /u/mqweb3 8300

Note down the port you specified (8300) as you will need it when you try to connect to the server.

The documentation says switch to the instance directory and issue apachectl -v. This failed for me because the path and libpath were not set up. I set up the envvars file (above) and it worked.

/usr/lpp/ihs*/bin/apachectl -v

When I ran it, it produced

test: /usr/lpp/ihsa_zos/bin/apachectl 49: FSUM7351 not found
[: /usr/lpp/ihsa_zos/bin/apachectl 74: FSUM7351 not found
[: /usr/lpp/ihsa_zos/bin/apachectl 87: FSUM7351 not found
[: /usr/lpp/ihsa_zos/bin/apachectl 92: FSUM7351 not found
Server version: IBM_HTTP_Server/9.0.5.5 (Unix) (SMP/E, 64-bit)
Server built: Jun 10 2020 16:22:51

The FSUM7351 messages are OK. For example the apachectl script has

if test -f /usr/lpp/ihsa_zos/bin/envvars; then
. /usr/lpp/ihsa_zos/bin/envvars
fi

This checks the file exists before invoking it.

The apachectl takes the following options, so you can display the configuration, or modify the start.

Options:
-D name : define a name for use in directives
-d directory : specify an initial ServerRoot
-f file : specify an alternate ServerConfigFile
-C “directive” : process directive before reading config files
-c “directive” : process directive after reading config files
-e level : show startup errors of level (see LogLevel)
-E file : log startup errors to file
-v : show version number
-V : show compile settings
-h : list available command line options (this page)
-l : list compiled in modules
-L : list available configuration directives
-t -D DUMP_VHOSTS : show parsed vhost settings
-t -D DUMP_RUN_CFG : show parsed run settings
-S : a synonym for -t -D DUMP_VHOSTS -D DUMP_RUN_CFG
-t -D DUMP_MODULES : show all loaded modules
-M : a synonym for -t -D DUMP_MODULES
-t -D DUMP_SSL_CONFIG: show parsed SSL vhost configurations
-t -D DUMP_SSL_CIPHERS: show all known SSL ciphers
-t -D DUMP_UNIFIED_CONFIG: show configuration with all includes merged
-t -D DUMP_INCLUDES: show all included configuration files
-t : run syntax check for config files
-T : start without DocumentRoot(s) check
-X : debug mode (only one worker, do not detach)

-t -DDUMP_CONFIG is very useful as it shows what you have configured after any <If…> and after variable substitution. I use this as a standalone command to see what’s configured.

Note:I had problems using //STDENV in the started task, so I had to use the envvars file.

Create the JCL procedure

See the documentation.

//HTTPCP PROC ACTION='start',
// DIR='/usr/lpp/ihsa_zos',
// CONF='/u/mqweb3/conf/httpd.conf'
//*---------------------------------------------------------
//IHS EXEC PGM=BPXBATCH,REGION=0M,
// PARM='SH &DIR/bin/apachectl -k &ACTION -f &CONF -DNO_DETACH ',
// MEMLIMIT=1236M
//STDOUT DD SYSOUT=H
//STDERR DD SYSOUT=H
// PEND

BPXBATCH needs REGION=0M, and at least MEMLIMIT=1236M

Define the STARTED task to RACF.

RDEFINE STARTED HTTPCP* STDATA(USER(WEBSRV)
SETROPTS RACLIST(STARTED ) REFRESH

Start the started task

S HTTPCP

You cannot just type P HTTPCP. In the syslog I had

CRIHS0001I IHS S0W1 is active. 83951827 0.0.0.0:8300 unspecified:-1.
Use jobname HTTPCP6 for console commands.

This means you have to issue P HTTPCP6 to stop it.

You can also use

s HTTPCP,action=’start’
s HTTPCP,action=’stop’

To start and stop the server.
Note: When you run these commands it checks the syntax of the configuration file, and if there is a problem, then the command is not executed. I kept wondering why my HTTPD instance was not shutting down; it was because I had a configuration error, and so the stop request was being ignored.

If it starts (there are no helpful messages saying success) try connecting a web browser to it in my case

http://10.1.1.2:8300

This gave me page with links to IBM sites. If you get here – you have done the first baby step. You cannot do much with the server.

Being too strong can be a waste of time

The short story is to use Elliptic curves with size 256 or 384, and avoid wasting time investigating why using other sizes doesn’t always work.

The long story – part 1 the server

I had a RACF server certificate defined as

SIZE(521) NISTECC …

and using Chrome browser it failed. I did a lot of digging around and could see the gsktrace had

EXIT gsk_get_ec_parameters_info(): <— Exit status 0x00000000 (0) EC curve type 34, key size 521
ERROR send_v3_alert(): Sent SSL V3 alert 40 to 10.1.0.2[38736]

INFO edit_ciphers(): Server certificate ec curve 0034 not in supported ecurve tls extension. EC cipher suites disabled

Where curve type 34, also known as 0x0019 = secp521r.

When I changed the size to 256 it worked.

Digging in to this, the Client Hello part of the TLS handshake sent from Chrome had

Supported Groups (4 groups)
Supported Group: Reserved (GREASE) (0x2a2a)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)

This is missing the 0x0019.

For Firefox, which worked, it had

Supported Groups (6 groups)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)
Supported Group: secp521r1 (0x0019)
Supported Group: ffdhe2048 (0x0100)
Supported Group: ffdhe3072 (0x0101)

So after a couple of hours working on this ( and an overnight sleep-on-it) the solution was to use secp256r1 instead of secp521.

The long story – part 2 – the client

I also tried using a client certificate which was defined as a secp521r1. This has similar problems. When I defined it as secp256r1 or secp384r1 it worked.

The gsktrace included

INFO read_v3_certificate(): Certificate key algorithm 13, Signature algorithm 84
ENTRY gsk_get_ec_parameters_info(): —> keyInfo size 12
EXIT gsk_get_ec_parameters_info(): <— Exit status 0x00000000 (0) EC curve type 34, key size 521
ERROR read_v3_certificate(): Client certificate elliptic curve not in clients supported elliptic curve list

I also know more about the TLS hand shake – which has more twists and turns every time I look at it!

Certificate validation on LDAP using OCSP

It is good practice to validate certificates when security is important. There are typically two ways of doing this.

  • Have a central server with a list of all revoked certificates. This server has a Certificate Revocation List(CRL) which has to be maintained. This solution tends to be deprecated
  • When a certificate is created add an extension with “check using this URL… to check if the certificate is valid”. This field is added when the certificate is signed by the Certificate Authority. A request is sent to the OCSP server, and a response sent back. This technique is known as Online Certificate Status Protocol (OCSP). You need an OCSP server for each certificate authority.

The blog post explains about OCSP and some of the challenges in using it.

You can configure a certificate to have OCSP support.

When you create a signed certificate, the CA signs the certificate.

For example

openssl req -config eccert.config … -out $name.csr
openssl ca -config openssl-ca-user.cnf -policy signing_policy -in $name.csr -extensions xxxx

In the config file are different sections, in my xxx section is

[ xxxx ]
authorityInfoAccess = OCSP;URI:http://10.1.0.2:2000

If the server (LDAP in this case) is configured for OCSP, then when it sees the AIA extension, it sends a request to the OCSP server at the URI (10.1.0.2:2000) which responds saying “good” , “revoked” or “unknown”.

OCSP server.

Openssl provides a simple ocsp server. You give it

  • the port it is to use
  • the CA public certificate (so the incoming certificate can be validated against the CA)
  • a file for that CA of the certificates it has issued, and it they are valid or not.

You can also configure the LDAP server to give a default URI, for those certificates that do not have the Authority Info Access(AIA) section. Typically this only works if you have one Certificate Authority, or you have a smart OCSP server which can handle the data for multiple CAs.

For example if your openssl OSCP server is configured for CA with DN:CN=CA1,o=MYORG, and you send down a request for DN:CN=CA2,o=myorg, it will not recognize it.

If you use the AIA extension, then you can have two different OCSP servers, one for each CA.

How does it work?

There are two ways of doing OCSP checking.

  • The client looks at the certificate and sees there is an OCSP extension. The client sends a request to the OCSP server to check the certificate is valid.
  • As part of the TLS handshake send the extension to the server to say “please do OCSP checking for me”. The server issues the request to the OCSP server, and can cache the response in the server. The server also sends the OCSP status to the client as part of the TLS handshake. The next request for the certificate can use the cached response. It tends to improve performance, as it reduces the number of requests to the OCSP server, and the responses are cached in the server. This is known as OCSP stapling.

Does using OCSP kill performance?

Yes and no. I’ll cover this after “caching the response”

Caching the responses

The OCSP server may be within your organization, or it may be be external. The time to send a request and get the response back may range from milliseconds to seconds.

Servers can typically cache the response to an OCSP query.

The OCSP may update it’s certificate list every hour, or perhaps every day, so if it is refreshed once a day, your LDAP server, does not need to refresh its data more than once a day.

The openssl OCSP server has an option -nmin minutes| -ndays days which is how often to reread the file. It sends a response

SingleResponse
  certID
    hashAlgorithm (SHA-1)
    issuerNameHash: 157b5dd0bcdee5b5428e063cf29a1f4e45be7499
    issuerKeyHash: 5830af55c7b4d49fc9e3fac91441ef1fd7b215e3
    serialNumber: 587
  certStatus: good (0)
  thisUpdate: 2021-11-05 10:42:58 (UTC)
  nextUpdate: 2021-11-05 10:43:58 (UTC)

From this the requester knows when the cached value is no longer valid, and needs to contact the OCSP server again for that certificate. From this we can see that the response is valid for 60 seconds.

When the certificate was revoked the output was

Online Certificate Status Protocol
  responseStatus: successful (0)
  responseBytes
    ResponseType Id: 1.3.6.1.5.5.7.48.1.1 (id-pkix-ocsp-basic)
    BasicOCSPResponse
      tbsResponseData
        responderID: byName (1)
          byName: 0...
          producedAt: 2021-11-06 15:15:58 (UTC)
            responses: 1 item
              SingleResponse
                certID...
                  certStatus: revoked (1)
                    revoked
                      revocationTime: 2021-11-06 15:12:30 (UTC)
                      revocationReason: cessationOfOperation (5)
                    thisUpdate: 2021-11-06 15:15:58 (UTC)
                    nextUpdate: 2021-11-06 15:16:58 (UTC)
            responseExtensions: 1 item
              Extension
                Id: 1.3.6.1.5.5.7.48.1.2 (id-pkix-ocsp-nonce)
                ReOcspNonce: a7e59577b7d2b3a2
      signatureAlgorithm (ecdsa-with-SHA256)
        Padding: 0
        signature: 3046022100fb264d4c5dbbbf45cce752a8263c4f01631441...
      certs: 1 item...

This information is available to the client.

What about performance?

If you write your application properly the impact of OCSP can be minimized.

For example consider the scenario where you are using REST requests, and go to one of a number of server.

  • Your application starts
  • It needs to check the validity of the certificate. Send a request to the OCSP server. This could take a long time (seconds). Some applications have a time-out of 15 seconds waiting for a response.
  • The response comes back saying “certificate Good – and valid for 1 hour.
  • Send a request to the server, and get the response back
  • Issue another request (with another TLS handshake)
  • After 1 hour – resend the request to the OCSP server to re-validate.
  • Etc

OCSP stapling

OCSP stapling is very common now. Before this, clients themselves used to check the validity of the user’s certificate by contacting the OCSP server. With OCSP stapling, a request is put into the TLS handshake which says “please do the OCSP checks for me and send me the output”. This allows the server to cache the information.

Think of the old days of checking in at the airport, when the person checking you in, would staple “checked-in by agent 46” to your paper ticket.

The client requests this by adding the “status_request” extension to the TLS clientHello handshake.

The server sends down, as part of the “serverHello” the information it received from the OCSP server.

Note. ldapsearch, from openssl, sends up the status_request, but does not handle the response, I get

ldap_sasl_interactive_bind_s: Can’t contact LDAP server (-1) additional info: (unknown error code)

openssl s_client does not send the status_request extension, so does not participate in the OCSP checking.

Java does support this, and a Java application can get the OCSP response message using the getStatusResponses method on the from the ExtendedSSLSession. I believe you can decode it using Bouncycastle.

Setting up LDAP on z/OS to support OCSP

I added the following to a working LDAP system

Environment

GSK_OCSP_URL=http://10.1.0.2:2000
GSK_OCSP_ENABLE=ON
GSK_OCSP_CLIENT_CACHE_SIZE=100
GSK_REVOCATION_SECURITY_LEVEL=MEDIUM
GSK_SERVER_OCSP_STAPLING=OFF
GSK_OCSP_RESPONSE_SIGALG_PAIRS=0603060105030501040304020401

#GSK_OCSP_RESPONSE_SIGALG_PAIRS=0806080508040603060105030501040304020401

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Stapling

I set GSK_SERVER_OCSP_STAPLING=OFF because ldapsearch on Ubunutu did not work with the ENDENTITY value.

Signature Algorithms

If GSK_OCSP_RESPONSE_SIGALG_PAIRS included any of 0806 0805 0804, I got messages

GLD1160E Unable to initialize the LDAP client SSL support: Error 113, Reason -99.
GLD1063E Unable to initialize the SSL environment: 466 – Signature algorithm pair is not valid.

In the trace I had 03353003 Cryptographic algorithm is not supported, despite these being listed in the documentation.

Nonce

The nonce is used to reduce replay attacks. In your request to the OCSP serve you include a nonce (string of data). You expect this in the response message.

The default in LDAP is off!

GSK_OCSP_NONCE_GENERATION_ENABLE=ON
GSK_OCSP_NONCE_CHECK_ENABLE=ON
GSK_OCSP_NONCE_SIZE=8

Configuration

In the configuration file I had

sslKeyRingFile START1/MQRING

sslCipherSpecs GSK_V3_CIPHER_SPECS_EXPANDED

Setting up the OCSP server on Linux

My CA was on Linux – address 10.1.0.2.

I set up a bash script to run the openssl OCSP server on Linx,

ca=”-CA ca256.pem”
index=”-index index.txt”
port=”-port 2000″
#rsigner=”-rsigner rsaca256.pem –rkey rsaca256.key.pem”
#rsigner=”-rsigner ecec.pem –rkey ecec.key.pem”
rsigner=”-rsigner ss.pem –rkey ss.key.pem”
nextUpdate=”-nmin 1″
openssl ocsp $index $ca $port $rsigner $nextUpdate

You need

  • CA … for the Certificate Authority .pem file
  • -index index.txt, for the status of the certificates issued by the CA
  • -port … a port to use
  • -rsigner… the public key to be used when responding
  • -rkey … the private key for encrypting the response.
  • -nmin … how often the index file is refreshed. Typically this value might be an hour or more.

It does not log any activity, so I had to use Wireshark to trace the network traffic.

Problems using a certificate signed by the CA, for encrypting the response.

The rsigner certificate needs to have

Extended Key Usage: critical, OCSP Signing

Without this I got the following in the gsktrace

ERROR check_ocsp_signer_extensions(): extended keyUsage does not allow OCSP Signing

A self signed certificate worked OK.

Testing it

I had several challenges when testing it

  • ldapsearch on Linux sends up the “I support OCSP stapling”, but it objects to the response, and ends with unknown error code.
  • openssl s_client does not send the OCSP flag, and so the certificate does not get validated.
  • Java worked. I used this as a basis, and made a few changes to reflect my system. I needed to use the following optuons to run it
    • -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/ecec.p12
    • -Djavax.net.ssl.keyStorePassword=password
    • -Djavax.net.ssl.keyStoreType=pkcs12
    • -Djavax.net.ssl.trustStore=/home/colinpaice/ssl/ssl2/dantrust.p12
    • -Djavax.net.ssl.trustStorePassword=password
    • -Djavax.net.ssl.trustStoreType=pkcs12

Once I change GSK_SERVER_OCSP_STAPLING=ENDENTITY to GSK_SERVER_OCSP_STAPLING=OFF, I was able to use LDAPSEARCH.

Some OCSP certificates didn’t work

In my OCSP server, I used a certificate signed by the CA, for encrypting the response back to LDAP.

In the GSKtrace I got

ERROR find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
ERROR crypto_ec_token_public_key_verify(): ICSF service failure: CSFPPKV retCode = 0x4, rsnCode = 0x2af8
ERROR crypto_ec_token_public_key_verify(): Signature failed verification
ERROR crypto_verify_data_signature(): crypto_ec_verify_data_signature() failed: Error 0x03353004

There are two connected problems here

  1. Find_ocsp_signer_in_certificate_chain(): Unable to locate signing certificate.
  2. retcode 0x2af8 (11000) The digital signature verify ICSF callable service completed successfully but the supplied digital signature failed verification.8 (11000). I did not have the correct CA certificate for the OCSP certificate in the LDAP keyring.

More about certificate authentication with LDAP.

Having got certificate authentication to work with LDAP on z/OS, see here, I found there are some more things you can do with it certificate authentication.

I had a certificate with Distinguished Name CN=ecec,O=cpwebuser,C=GB. This mapped to a RACF userid ADCDA.

I can get LDAP to generate a different DN racfid=ADCDA,profiletype=USER,o=myracf, based on this RACF userid. I can then use this DN in Access Control Lists etc..

Setting up LDAP

In my configuration file I set the sslMapCertificate line, added a section for SDBM, and restarted the server.

sslMapCertificate add fail

databaseDirectory /var/ldap/ldbm
#———– SDBM —————-
database sdbm GLDBSD31/GLDBSD64
suffix o=myracf

I set up the mapping of certificate to userid.

RACDCERT LISTMAP ID(ADCDA)

RACDCERT DELMAP(LABEL(‘LINUXECEC’ )) ID(ADCDA)

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

RACDCERT MAP ID(ADCDA ) –
SDNFILTER(‘CN=ecec.O=cpwebuser.C=GB’) –
WITHLABEL(‘LINUXECEC’)

RACDCERT LISTMAP ID(ADCDA)

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

The result

When the certificate was used to connect to LDAP, it generated a DN, DN=’RACFID=ADCDA,PROFILETYPE=USER,O=MYRACF‘, where ADCDA is the userid from the certificate mapping, and O=MYRACF comes from the suffix statement in the SDBM section.

If sslMapCertificate replace is specified, the DN from the certificate is discarded and replaced with the generated DN.

If sslMapCertificate add is specified, both the DN from the certificate and the generated DN, are used in the ACL check. The certificate DN is used in auditing etc

Things to think about

You might want to use sslMapCertificate ADD rather than replace, so audit statements have the original DN, and thus provide a better audit trail.