What’s my Appache HTTPD configuration?

I was trying to understand how my HTTPD configuration worked, and which file something was in. I was pleasantly surprised to find how easy it was to get this information. Rather than just list the configuration files as they were read, the data is displayed by function. This means I can see all the configuration information about “Environment variables”, or SAF.

It displays all the configuration keywords. For example for Module Name:mod_ibm_ssl.c it displays

  • SSLDisable – SSL is disabled for this server
  • SSLSNIMap – Add a new SNI/label mapping
  • SSLEnable – SSL is enabled for this server
  • SSLServerCert – Certificate within keyfile for this server
  • SSLClientAuth – The type of Client Authentication

Values are substituted, in my configuration files I had

colin.conf:define  urls  https://10.1.1.2   
...
RewriteRule "¬/(AdmRootca|Rootca)/ssl-cgi/(.*) ${urls}/$1/ssl-cgi-bin/$2 [R,NE,L] " 

This was displayed as

RewriteRule "^/(AdmRootca|Rootca)/ssl-cgi/(.*) https://10.1.1.2/$1/ssl-cgi-bin/$2 [R,NE,L] "

and I could check my configuration was correct.

You can change the startup to display the configuration at startup… but…

The documentation says you can add -DDUMP_CONFIG to the startup parameters. But when I tried this, it made the parameter list too long for the PARM=…

I had to change

//HTTPCP   PROC ACTION='start  -DDUMP_CONFIG ', 
//         DIR='/usr/lpp/ihsa_zos', 
//         CONF='/u/mqweb2/conf/httpd.conf'
//IHS      EXEC PGM=BPXBATCH,REGION=0M, 
//  PARM='SH &DIR/bin/apachectl -k &ACTION -f &CONF -DNO_DETACH   ', 
// MEMLIMIT=1236M  

to

//HTTPCP   PROC 
//  EXPORT SYMLIST=* 
//  SET ACTION='start  -DDUMP_CONFIG ' 
//  SET DIR='/usr/lpp/ihsa_zos' 
//  SET CONF='/u/mqweb2/conf/httpd.conf' 
//IHS      EXEC PGM=BPXBATCH,REGION=0M,PARMDD=PARMDD, 
// MEMLIMIT=1236M 
//PARMDD DD  *,SYMBOLS=EXECSYS 
SH &DIR/bin/apachectl 
 -k &ACTION 
 -f &CONF -DNO_DETACH 
 -DDUMP_CONFIG 
/* 

Then I got the configuration in STDOUT – though without the HTML links.

To display it online, you need to configure HTTPD

In my colin.conf file I had

LoadModule info_module modules/mod_info.so 

<IfModule mod_info.c> 
<Location /info> 
    SetHandler server-info 
#   Require all denied 
#   To allow access from a specific IP: 
#   Require ip 192.168.1 
</Location>
</IfModule> 

Of course you’ll enable security once it works…

I used

https://10.1.1.2:443/info

and it displayed the configuration. It has sections on

  • Loaded modules (I was pleased to see mod_info.so in the list)
  • Server settings
  • Hooks and providers
  • What I call “function type parameters” with the relevant configuration. For example for mod_alias.c it shows
    • Module directives
      • Alias – a fakename and a realname, or a realname in a Location
      • ScriptAlias – a fakename and a realname, or a realname in a Location
      • Redirect – an optional status, then document to be redirected and destination URL
      • Current configuration – and the information applicable to the function. For Module Name:mod_alias.c
        • In file: /u/mqweb2/conf/httpd.conf 
          • 467: Alias /icons/ “/u/mqweb2/icons/” 
          • 483: ScriptAlias /cgi-bin/ “/u/mqweb2/cgi-bin/”
        • In file: /u/mqweb2/conf/vhost80.conf  
          • 23: <VirtualHost *:80>  
          • 44:   ScriptAliasMatch /(PKIServ|Customers)/public-cgi/(.*) /usr/lpp/pkiserv/PKIServ/public-cgi/$
          • 2    : </VirtualHost>
      • For Module Name:mod_env.c
        • In file: /u/mqweb2/conf/vhost80.conf  
          • 23: <VirtualHost *:80>  
          • 27:   SetEnv LIBPATH “/usr/lpp/pkiserv/lib:/usr/lpp/ihsa_zos/lib:/usr/lpp/ihsa_zos/modules:/usr/lpp/ihsa_zos”  
          • 28:   SetEnv _PKISERV_CONFIG_PATH “/etc/pkiserv”    
          • : </VirtualHost>
        • In file: /u/mqweb2/conf/vhost443.conf  
          • 27: <VirtualHost *:443>  
          • 28:   SetEnv LIBPATH “/usr/lpp/pkiserv/lib:/usr/lpp/ihsa_zos/lib:/usr/lpp/ihsa_zos/modules:/usr/lpp/ihsa_zos”  
          • 29:   SetEnv _PKISERV_CONFIG_PATH “/etc/pkiserv”    
          • : </VirtualHost>

This is all very well thought through and easy to use.

Strong keys may be bad for your blood pressure.

I’ve just spent a couple of days trying to get a web server to use an Elliptic key with size 521. It works on Firefox, Curl, and OpenSSL, but not on Chrome. Weaker keys work of size 256 and 384 work fine, except in some cases a key size of 256 gave did not work on z/OS.

I’ve since found a page going back to 2015 discussing the dropping of several EC cipher specs. There is also discussion “NSA Suite B says that NSA uses curves P-256 and P-384” which (to some people) implied that other elliptic curves should not be used, and so were removed. There is another discussion that P256 is as strong as P521, but requires less CPU, and so P521 should not be used.

So the short answer is use Elliptic curves with key size 256 or 256, and do not use the others.

I was trying to get the Apache HTTPD server on z/OS to use a certificate, and it failed. I tried using a similar certificate from the openssl Simple server on Linux and it failed, so I’m guessing that my Chrome version (Version 114.0.5735.198 (Official Build) (64-bit)) does not support it.

The problem

In Chrome I got

This site can’t provide a secure connection 10.1.1.2 sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

This is not entirely true, as the browser sent the Alert response to the server and it was not an ERR_SSL_PROTOCOL_ERROR. After this the browser code returned its caller saying “failed” . Using Wireshark, capturing encrypted data showed the data flowing from the browser to the server to port 443),

Transmission Control Protocol, Src Port: 38188, Dst Port: 443

Transport Layer Security
  TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    Content Type: Change Cipher Spec (20)
    Change Cipher Spec Message
  TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
    Opaque Type: Application Data (23)
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

In the Apache HTTPD error.log it had

SSL0209E: SSL Handshake Failed, ERROR processing cryptography. [10.1.0.2:38188 -> 10.1.1.2:443] 

Which is not very helpful. The problem occurred on the browser; all the server sees is Alert code 51.

Taking a system SSL trace on z/OS(GSK Server) gave me

Job HTTPCP8   Process 0501003C  Thread 00000032  read_tls13_alert 
  TLS 1.3 alert 51 received from ::ffff:10.1.0.2.38188. 

Diagnostics

Using tools like gsktrace on z/OS, and Wireshark to see the flow over the network. I could see that an alert was sent from the Chrome browser to server with type Decrypt Error(51). This meant it was a problem at the browser end, not the server end.

Wireshark capturing encrypted data.

If you use TLS 1.3 once the “change cipher spec” has been issued, all traffic is encrypted, and by default Wireshark cannot read it. For example

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
     Change Cipher Spec Message
 TLSv1.3 Record Layer: Application Data Protocol: http-over-tls
   Opaque Type: Application Data (23)
     Version: TLS 1.2 (0x0303)
       Encrypted Application Data: 489be8a0976798290766c9ee158c24f5863d18
      [Application Data Protocol: http-over-tls]d

and you cannot tell this is reporting an error.

Wireshark can decrypt this. You need to set an environment variable, and start the application from the terminal window, to pickup the environment variable.

export SSLKEYLOGFILE=$HOME/sslkeylog.log
google-chrome

This will cause the application( the google-chrome browser) to start and to write the TLS key data to the file.

Configure Wireshark to use this file:

  • Edit
  • Preferences
  • Expand the Protocols twistie
  • Scroll down to TLS ( typing T gets you near it)
  • Enter the Master-Secret log file name mine is /home/colinpaice/sslkeylog.log

Now, if you run the test you should get the data decrypted

Transport Layer Security
 TLSv1.3 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
   Content Type: Change Cipher Spec (20)
   Change Cipher Spec Message
 TLSv1.3 Record Layer: Alert (Level: Fatal, Description: Decrypt Error)
  Opaque Type: Application Data (23)
    [Content Type: Alert (21)]
    Alert Message
      Level: Fatal (2)
      Description: Decrypt Error (51)

Diagnostics that were not helpful.

Chrome trace

Starting Chrome from a terminal session

google-chrome --enable-logging --v=1

The output was in ./.config/google-chrome/chrome_debug.log.

Using grep ERROR ./.config/google-chrome/chrome_debug.log gave me

...:ERROR:nss_util.cc(357)] After loading Root Certs, loaded==false: NSS error code: -8018
...:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -107
...:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -107

and no other useful information besides net_error -107. For these codes see here. 107 is an unhelpful message NET_ERROR SSL_PROTOCOL_ERROR.

Wireshark gave me decrypt error.

z/OS system SSL trace

Using GSKSRVR and CTRACE on z/OS. See here. This gave me

S0W1      MESSAGE   00000004  17:20:47.690772  SSL_ERROR 
  Job HTTPCP8   Process 0501003C  Thread 00000032  read_tls13_alert 
  TLS 1.3 alert 51 received from ::ffff:10.1.0.2.60830. 

Which shows the alert came from the browser.

Another way of getting system SSL trace.

The documentation says you can collect trace by changing the configuration. This allows you to trace just the pages/scripts that you want.

Creating a certificate on z/OS

On z/OS I used

RACDCERT ID(START1) GENCERT -                            
  SUBJECTSDN(CN('10.1.1.2') - 
             O('NISTECCTEST') - 
             OU('SSS')) -                                
   ALTNAME(IP(10.1.1.2))-                                
   NISTECC - 
   KEYUSAGE(   HANDSHAKE     )  - 
   SIZE(384) - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -               
   WITHLABEL('NISTECCTEST')     - 
                                                         

With different sizes 256,284, and 521. The IBM documentation says For NISTECC keys, valid key sizes are 192, 224, 256, 384, and 521 bits. I had problems with key size 521 bits.

On Linux I used

timeout="--connect-timeout 10"
enddate="-enddate 20240130164600Z" 
ext="-extensions end_user"
name="docec384"
key="$name.key.pem"
cert="$name.pem"

subj="-subj /C=GB/O=Doc2/CN="$name 
CA="docca256"
cafiles="-cert $CA.pem -keyfile $CA.key.pem "
rm $name.key.pem
rm $name.csr
rm $name.pem
passin="-passin file:password.file"
passout="-passout file:password.file"
md="-md sha384"
policy="-policy signing_policy"
caconfig="-config ca2.config"
caextensions="-extensions clientServer"
config="-config openssl.config"

openssl ecparam -name secp384r1 -genkey -noout -out $name.key.pem 
openssl req $config -new -key $key -out $name.csr -outform PEM -$subj $passin $passout 
openssl ca $caconfig $policy $ext $md $cafiles -out $cert -in $name.csr $enddate $caextensions 

openssl x509 -in $name.pem -text -noout|lessf

I used this as a template with different flavours of -name secp384r1

  • -name secp384r1
  • -name prime256v1 which worked
  • -name secp521r1 which did not work

You can get a list of valid names from the command openssl ecparam -list_curves.

Running a Linux web server

I tried using the openssl s_server, and could also reproduce this problem (with a much faster turnaround). I used

tls="-tls1_3  "
#cert=" -cert ./docec256.pem -key ./docec256.key.pem"
#cert=" -cert ./docecgen.pem -certform pem -key docecgen.key.pem -keyform pem" 
#cert=" -cert ./docec521.pem -certform pem -key docec521.key.pem -keyform pem"
cert=" -cert ./docec384.pem -certform pem -key docec384.key.pem -keyform pem" 
CA="-chainCAfile ./docca256.pem"
cipher1="-cipher TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384"

cipher=$cipher1
port="-port 4433 "

strict="-x509_strict -strict"
debug="-trace"

#  ca="-CAfile ./zpdt.ca.pem "
openssl s_server $port $tls  $cert $cipher $CA $debug $strict -www  

Chrome startup options

You can start chrome from a terminal with many startup options. See here

IKJ56500I COMMAND … NOT FOUND. Macro does not exist

Due to some random typing, when I try to edit a member I keep getting

  • IKJ56500I COMMAND M NOT FOUND
  • Macro does not exist

After some playing around, I found this was caused by having a list of PDS members, using the prefix command E, and putting something(M) in the Prompt column.

To fix it, use E and put / in the Prompt column. This gives the EDIT Entry Panel and shows Initial Macro..M . Blank out the M, and press PF3 to clear it.

Easy when you know how.

I expected this to show up under ISPF 2 (Edit) but the initial Macro field was blank.