z/OS systems-ssl strange behaviour with environment variables

I was trying to use system ssl to write a program to use native z/O TLS facilities. I wasted a couple of hours because it said it could not find my keyring. Then when I collected a trace, it sometimes did not find the file – which did exist as I could list it.

If I used

//START1   EXEC PGM=GSKMAIN,REGION=0M, 
//*  PARM='4000' 
// PARM=('ENVAR("_CEE_ENVFILE=DD:STDENV")/4000') 
//STDENV DD PATH='/u/ibmuser/gskparms'

When the USS file had

GSK_TRACE_FILE=/tmp/zzztrace.file 
GSK_TRACE=0xff 
GSK_KEYRING_FILE=START1/TN3270

This worked file

When I used

//START1   EXEC PGM=GSKMAIN,REGION=0M, 
//*  PARM='4000' 
// PARM=('ENVAR("_CEE_ENVFILE=DD:STDENV")/4000') 
//STDENV  DD * 
GSK_TRACE_FILE=/tmp/zzztrace.file 
GSK_TRACE=0xff 
GSK_KEYRING_FILE=START1/TN3270 
/*

This failed to work.

If I looked in the trace file I had

ENTRY gsk_open_keyring(): ---> Keyring 'START1/TN3270                       '

Where it had taken the whole length of the line – and so START1/TN3270 padded with blanks was not found.

The trace file was not /tmp/zzztrace.file, it was /tmp/zzztrace.file padded with lots of blanks!

The answer is to use a environment file in USS, not in JCL or a data set.

Getting a system ssl trace for httpd server on z/OS

I had a problem getting the httpd server to work on z/OS. It did not like my certificate – but reported messages about ICSF security not being set up. I got to learn a lot about traces, but could not find how to trace System SSL (GSK) and httpd.

I specified SSLTRACE in my vhost*.conf file which gave me a little information – but not at the System SSL Level.

Other applications using System SSL, can set environment variables

GSK_TRACE=0xff 
GSK_TRACE_FILE=/tmp/gskssl.%.trc

but this does not work with http. (I think the environment variables are not passed on to any spawned thread).

You have to use the gsksrvr task, and collect the trace through CTRACE.

Set up gsk trace.

I have used gsk trace before, and described setting it up.

I had to create a parmlib member

TRACEOPTS 
          WTRSTART(ctwtr) 
          on 
          wtr(ctwtr) 
jobname(httpcp) 
          OPTIONS('LEVEL=255,JOBSUFFIX=ANY')

I start my http server with the S HTTPCP command, and specified httpcp in the jobname of the parmlib.

I got out no trace. I tried the various jobnames until it produced a trace. My trace was produced from jobname httpcp8! I could not find a way of displaying which of my httpcp* job was used, so I had to try them all.

If I had had a long name eg httpcpxx then specifying jobname(httpcpxx) should have worked.

Starting the trace

I used

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON

where my parmlib member was CTGSKOK

It produced

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON
IEE252I MEMBER CTGSKON FOUND IN USER.Z24C.PARMLIB
GSK01040I SSL component trace started.
ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE
SUCCESSFULLY EXECUTED.

and the CTWTR started up.

What is the status of the trace?

d trace,comp=gsksrvr

gave me

IEE843I 08.32.25  TRACE DISPLAY       
  COMPONENT     MODE BUFFER HEAD SUBS                                  
 --------------------------------------------------------------        
  GSKSRVR       ON   0064K                                             
     ASIDS      *NOT SUPPORTED*                                        
     JOBNAMES   HTTPCP8 ,HTTPCP1 ,HTTPCP2 ,HTTPCP3 ,HTTPCP4 ,          
                HTTPCP5 ,HTTPCP6 ,HTTPCP7                              
     OPTIONS    LEVEL=255                                              
     WRITER     CTWTR

Run my test

When I ran my test, some System SSL messages were produced on the console from the gsksrvr address space

GSK01047I SSL component trace started for HTTPCP8/STC01000.
GSK01050I SSL component trace started for HTTPCP8/STC01000/05010022.

Stop the trace

TRACE CT,OFF,COMP=GSKSRVR

Wait until you get

GSK01041I SSL component trace ended.

from the gsksrvr address space, and stop the trace writer.

D TRACE,WTR=ALL
TRACE CT,WTRSTOP=CTWTR

This seems to take a few seconds to run. It outputs

IEF196I AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA : 
IEF196I           IBMUSER.CTRACE1 
ITT111I CTRACE WRITER CTWTR TERMINATED BECAUSE OF A WTRSTOP REQUEST.  
IEF404I CTWTR - ENDED - TIME=08.40.21

Format the trace

You need to use IPCS to format it

=0 – and enter the data set name (IBMUSER.CTRACE1)
=6 – to get to the ipcs command screen
dropd – to tell IPCS to forget any historical information it may know about for the dataset
CTRACE COMP(GSKSRVR) full – this displays any System SSL CTRACE data
m PF8 – go to the bottom of the data
report view – to go into ISPF View mode on the data set
X ALL – to hide all of the data
f SSL_ERROR ALL – this shows any error codes
if you get any lines displayed, you can tab down to the hidden command and use the line prefix command f4 to display the first 4 hidden lines.

The errors I got were

 S0W1      MESSAGE   00000004  08:35:55.049451  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_ec_token_private_key_sign 
   ICSF service failure: CSFPPKS retCode = 0x8, rsnCode = 0x2b00 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.049733  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_sign_data 
   crypto_ec_sign_data() failed: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050012  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  construct_tls13_certificate_verify_message 
   Unable to generate certificate verify message: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050393  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  send_tls13_alert 
   Sent TLS 1.3 alert 51 to ::ffff:10.1.0.2.45432.

The Alert 51 matches what my browser received.

File /usr/include/gskcms.h had #define CMSERR_ICSF_SERVICE_FAILURE 0x03353084

CSFPPKS is PKCS #11 Private Key Sign.

The return code 0x2b00 (from here) gives:

User action: You might need to re-create the token by using the PKA key token build or PKA key import callable service or regenerate the key values on another platform.

or in other words, it did not like my certificate created with NISTECC SIZE(256) but did like NISTECC SIZE(521).

z/OS PAGENT error messages and system SSL codes

Message

SYSERR :001: plfm_kernel_init: socket(INET, DGRAM, 0), failed, errno=EDC5112I Resource temporarily unavailable., errno2=74610296

OBJERR :001: init_PEP_and_kernel: Kernel initialization failed for image ‘TCPIP2’,

RACF profile EZB.INITSTACK.*.TCPIP2 CLASS(SERVAUTH) was missing.

//S1 EXEC PGM=IKJEFT01,REGION=0M
//STEPLIB DD DISP=SHR,DSN=SYS1.MIGLIB
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RDEFINE SERVAUTH EZB.INITSTACK..TCPIP2 PERMIT EZB.INITSTACK..TCPIP2 CLASS(SERVAUTH)
ID(START1) ACCESS(READ)
SETROPTS RACLIST(SERVAUTH) REFRESH
/*

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file

In my image file I had a

TTLSRule
{
…

}

which is not allowed. They have to be in the TTLSConfig… file.

The error message was

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file ‘TTLSRule’ statement

402 No SSL cipher specifications.

In some situations the cipher specs default.

The message EZD1287I TTLS Error RC: 402 Initial Handshake pointed me to Cryptographic Services System Secure Sockets Layer Programming – No SSL cipher specifications. The first reason was

The client and server cipher specifications do not contain at least one value in common. Client and server cipher specifications might be limited depending on which System SSL FMIDs are installed. See Cipher suite definitions for more information. Server cipher specifications are dependent on the type of algorithms that are used by the server certificate (RSA, DSA, ECDSA, or Diffie-Hellman), which might limit the options available during cipher negotiation.

The GSK trace had GSK_CONNECT_CIPHER_SPEC(207) – C02B. Where C02B is TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

The certificate specified for the server was incompatible with this. I had Key Usage: HANDSHAKE
Key Type: NIST ECC
Key Size: 384

The one that worked had

Key Usage: HANDSHAKE
Key Type: NIST ECC
Key Size: 521

RC 416 GSK_ERR_PERMISSION_DENIED

Not authorized to access key database, PKCS #12 file, SAF key ring or z/OS PKCS #11 token.

EZD1287I TTLS Error RC: 416 Environment Master Init

ICH408I USER(START1 ) GROUP(SYS1 )
START1.TN3270.LST CL(RDATALIB)
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

Action: Give the started task userid permission to the ring.

RC 510 No acceptable key labels found

See here. The specified label was inconsistent with the client request.

The Wireshark output had a list of Cipher Suites, for example

   Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
   Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
   Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)

In my keyring I had only ECC certificates. When I created an RSA certificate it worked (well moved on a step).

RACDCERT ID(START1) GENCERT -                        
  SUBJECTSDN(CN('10.1.1.2') - 
             O('RSA2048') -                          
             OU('SSS')) -                            
   ALTNAME(IP(10.1.1.2))-                            
   RSA     - 
   SIZE(2048)  - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -           
   WITHLABEL('RSA2048')        -

RC: Code 515 Initial handshake

Code 515 Key share list is not valid

I got this when trying to use TLS 1.3.

You need

TTLSSignatureParms
{
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0030

}

or ServerKeyShareGroup

517 No matches between elliptic curve and key share lists

Need something like

TTLSSignatureParms CPESigParms
{
CLientECurves 0023
CLientECurves 0024
CLientECurves 0025
CLientECurves 0029
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0029
}

519 Required ciphers have not been specified

I had TLS 1.3 specified, but no TLS 1.3 cipher specs.

The following TLSv1.3 worked.

V3CipherSuites TLS_CHACHA20_POLY1305_SHA256

EZD1287I TTLS Error RC: 5006 Initial Handshake

See the TCPIP diagnosis reference.

EZD1287I TTLS Error RC: 5006 Initial Handshake
5006 The connection is using a TTLSEnvironmentAction statement that failed
to initialize a System SSL environment.

Use the syslog to determine why the System SSL environment failed to
initialize.
If the TTLSEnvironmentAction statement is in error, make the
necessary corrections. A System SSL environment is initialized for the
corrected TTLSEnvironmentAction statement and new connections use
that environment.
If a SAF configuration change is needed (such as changing a
certificate in the key ring), make the change and then update the
EnvironmentUserInstance parameter in the TTLSEnvironmentAction
statement to reflect a changed action. A System SSL environment
is initialized by using the modified RACF configuration and new
connections use that environment

I gave the started task permission to the keyring, and got this error.

When I restarted my application and it worked. I don’t know if this is relevant.

EZD1287I TTLS Error RC: 5003

RC 5003 will occur when the AT-TLS process is expecting an SSL message but receives a clear-text message.

System SSL 0x134cc000

The documentation for System SSL says

SSL error codes are defined internally to be ‘134CCxxx’, where xxx is the hexadecimal representation of the SSL error code. The range is ‘134CC001’ through ‘134CCFFF’. For example, SSL error code 414 would be ‘134CC19E’.

So xxx is 000 – which is no error. I got this using gsk_strerror(0) when there was no error!