Category: System SSL

z/OS PAGENT error messages and system SSL codes

Message

SYSERR :001: plfm_kernel_init: socket(INET, DGRAM, 0), failed, errno=EDC5112I Resource temporarily unavailable., errno2=74610296

OBJERR :001: init_PEP_and_kernel: Kernel initialization failed for image ‘TCPIP2’,

RACF profile EZB.INITSTACK.*.TCPIP2 CLASS(SERVAUTH) was missing.

//S1 EXEC PGM=IKJEFT01,REGION=0M
//STEPLIB DD DISP=SHR,DSN=SYS1.MIGLIB
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RDEFINE SERVAUTH EZB.INITSTACK..TCPIP2 PERMIT EZB.INITSTACK..TCPIP2 CLASS(SERVAUTH)
ID(START1) ACCESS(READ)
SETROPTS RACLIST(SERVAUTH) REFRESH
/*

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file

In my image file I had a

TTLSRule
{
…

}

which is not allowed. They have to be in the TTLSConfig… file.

The error message was

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file ‘TTLSRule’ statement

402 No SSL cipher specifications.

In some situations the cipher specs default.

The message EZD1287I TTLS Error RC: 402 Initial Handshake pointed me to Cryptographic Services System Secure Sockets Layer Programming – No SSL cipher specifications. The first reason was

The client and server cipher specifications do not contain at least one value in common. Client and server cipher specifications might be limited depending on which System SSL FMIDs are installed. See Cipher suite definitions for more information. Server cipher specifications are dependent on the type of algorithms that are used by the server certificate (RSA, DSA, ECDSA, or Diffie-Hellman), which might limit the options available during cipher negotiation.

The GSK trace had GSK_CONNECT_CIPHER_SPEC(207) – C02B. Where C02B is TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

The certificate specified for the server was incompatible with this. I had Key Usage: HANDSHAKE
Key Type: NIST ECC
Key Size: 384

The one that worked had

Key Usage: HANDSHAKE
Key Type: NIST ECC
Key Size: 521

RC 416 GSK_ERR_PERMISSION_DENIED


Not authorized to access key database, PKCS #12 file, SAF key ring or z/OS PKCS #11 token.

EZD1287I TTLS Error RC: 416 Environment Master Init

ICH408I USER(START1 ) GROUP(SYS1 )
START1.TN3270.LST CL(RDATALIB)
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

Action: Give the started task userid permission to the ring.

RC 510 No acceptable key labels found

See here. The specified label was inconsistent with the client request.

The Wireshark output had a list of Cipher Suites, for example

   Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
   Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
   Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)

In my keyring I had only ECC certificates. When I created an RSA certificate it worked (well moved on a step).

RACDCERT ID(START1) GENCERT -                        
  SUBJECTSDN(CN('10.1.1.2') - 
             O('RSA2048') -                          
             OU('SSS')) -                            
   ALTNAME(IP(10.1.1.2))-                            
   RSA     - 
   SIZE(2048)  - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -           
   WITHLABEL('RSA2048')        -

RC: Code 515 Initial handshake

Code 515 Key share list is not valid

I got this when trying to use TLS 1.3.

You need

TTLSSignatureParms
{
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0030

}

or ServerKeyShareGroup

517 No matches between elliptic curve and key share lists

Need something like

TTLSSignatureParms CPESigParms
{
CLientECurves 0023
CLientECurves 0024
CLientECurves 0025
CLientECurves 0029
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0029
}

519 Required ciphers have not been specified

I had TLS 1.3 specified, but no TLS 1.3 cipher specs.

The following TLSv1.3 worked.

V3CipherSuites TLS_CHACHA20_POLY1305_SHA256

EZD1287I TTLS Error RC: 5006 Initial Handshake

See the TCPIP diagnosis reference.

EZD1287I TTLS Error RC: 5006 Initial Handshake
5006 The connection is using a TTLSEnvironmentAction statement that failed
to initialize a System SSL environment.

  • Use the syslog to determine why the System SSL environment failed to
    initialize.
  • If the TTLSEnvironmentAction statement is in error, make the
    necessary corrections. A System SSL environment is initialized for the
    corrected TTLSEnvironmentAction statement and new connections use
    that environment.
  • If a SAF configuration change is needed (such as changing a
    certificate in the key ring), make the change and then update the
    EnvironmentUserInstance parameter in the TTLSEnvironmentAction
    statement to reflect a changed action. A System SSL environment
    is initialized by using the modified RACF configuration and new
    connections use that environment

I gave the started task permission to the keyring, and got this error.

When I restarted my application and it worked. I don’t know if this is relevant.

EZD1287I TTLS Error RC: 5003

RC 5003 will occur when the AT-TLS process is expecting an SSL message but receives a clear-text message.

System SSL 0x134cc000

The documentation for System SSL says

SSL error codes are defined internally to be ‘134CCxxx’, where xxx is the hexadecimal representation of the SSL error code. The range is ‘134CC001’ through ‘134CCFFF’. For example, SSL error code 414 would be ‘134CC19E’.

So xxx is 000 – which is no error. I got this using gsk_strerror(0) when there was no error!

GSK trace and TCPIP

To track down a TLS setup problem with TN3270 and AT-TLS, I had to collect a System SSL Trace. You can collect data and have it output to the joblog, or the console. This is not good practice as it can produce a lot of output, and it is hard to read the data.

This blog post follows on from Collecting a TCPIP Packet trace on z/OS.

You can use the same CTRACE writer procedure for all your CTRACE work. It is better not to have more than one trace writing to it at a time.

Erik Janssen sent me an email on how to take the CTRACE file and process it in Wireshark. I’ve put his comments at the bottom.

You need to start the GSK Server task

See SSL Server task

My GSKSRVR JCL looks like

//GSKSRVR  PROC  REGSIZE=256M,OUTCLASS='A' 
//*-------------------------------------------------------------------- 
//GO       EXEC  PGM=GSKSRVR,REGION=&REGSIZE,TIME=1440, 
//  PARM=('ENVAR("HOME=/etc/gskssl/server"),TERM(DUMP)                 X
//             / 1>DD:STDOUT 2>DD:STDERR') 
//STDOUT   DD  SYSOUT=&OUTCLASS,DCB=LRECL=250, 
//  FREE=END,SPIN=UNALLOC 
//STDERR   DD  SYSOUT=&OUTCLASS,DCB=LRECL=250, 
//  FREE=END,SPIN=UNALLOC 
//SYSOUT   DD  SYSOUT=&OUTCLASS, 
//  FREE=END,SPIN=UNALLOC 
//CEEDUMP  DD  SYSOUT=&OUTCLASS, 
//  FREE=END,SPIN=UNALLOC

I did not change any parameters, or defaults.

Start it

s GSKSRVR

It gives

GSK01001I System SSL version 4.4, Service level OA59115.
GSK01003I SSL server initialization complete.

Within the TELNET (TN3270) parameters I had

TelnetParms         ; ATTLS defined secure port 
 TTLSPort 2023      ; 
 DEBUG CONN DETAIL CTRACE 
 DEBUG CONN trace  CTRACE 
 DEBUG TRACE CTRACE
 ConnType Secure    ; Client chooses secure or nonsecure connection. 
EndTelnetParms

This limits the trace to only port 2023, other ports, such as 0023 were unaffected.

I set up parmlib member for GSK trace, member CPGSKON

TRACEOPTS 
          WTRSTART(ctwtr) 
          on 
          jobname(TCPIP) 
          wtr(ctwtr) 
          OPTIONS('LEVEL=255')

I turned trace on using

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON

This gave

IEE252I MEMBER CTGSKON FOUND IN USER.Z24C.PARMLIB
GSK01040I SSL component trace started.
ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE SUCCESSFULLY EXECUTED.

ITT110I INITIALIZATION OF CTRACE WRITER CTWTR COMPLETE.

The CTWTR procedure was started was part of this command.

You stop the trace with

trace CT,off,COMP=GSKSRVR

Wait for the message

ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE SUCCESSFULLY EXECUTED.

And stop the CTWTR writer using

TRACE CT,WTRSTOP=CTWTR

You should get messages

ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE SUCCESSFULLY EXECUTED.
IEE839I ST=(ON,0001M,00005M) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,064K)
IEF196I AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA :
IEF196I IBMUSER.CTRACE1
AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA :
IBMUSER.CTRACE1

ITT111I CTRACE WRITER CTWTR TERMINATED BECAUSE OF A WTRSTOP REQUEST.

If there is data in the file, you can use IPCS to format it.

The IPCS command is

CTRACE COMP(GSKSRVR) FULL ALL

You get output like

S0W1      MESSAGE   00000008  15:19:52.221020  SSL_INFO                 
  Job TCPIP     Process 0201003E  Thread 00000001  read_v3_client_hello 
  Received CLIENT-HELLO message                                         
                                                                        
S0W1      DUMP      00000020  15:19:52.221414  SSL_ASCII_DUMP           
  Job TCPIP     Process 0201003E  Thread 00000001  read_v3_client_hello 
  CLIENT-HELLO message                                                  
    00000000: 010001FC 03030C33 81E156C3 DF4693DD   *.......3..V..F..*  
    00000010: 621CA636 8A75BDF0 26F3DE8F 831B78BC   *b..6.u..&.....x.*
...
S0W1      MESSAGE   00000008  15:19:52.226897  SSL_INFO                  
  Job TCPIP     Process 0201003E  Thread 00000001  edit_ciphers          
  Using server certificate 'ZZZZ'                                        
...
S0W1      MESSAGE   00000008  15:19:52.223132  SSL_INFO                         
  Job TCPIP     Process 0201003E  Thread 00000001  read_v3_extended_client_hello
  Elliptical curve 0030 is being skipped as it is only supported in TLS 1.3

Getting the data into Wireshark format.

Wireshark is an excellent tool for doing network trace, and decoding the flows. You can also is it to process the CTRACE data. Thanks for Eril Janssen for the instructions below on how to get from CTRACE to Wireshark.

You need to use the CTRACE SNIFFER option. This writes to the file with DDNAME of SNIFFER

Allocate a VB 1600 LRECL dataset and allocate it to a ddname, for example SNIFFER
ALLOC DD(SNIFFER) DS(<some dsname>) SHR
From IPCS you can now:

CTRACE COMP(SYSTCPDA) GMT SHORT  OPTIONS((SNIFFER(1600,TCPDUMP) NOREASSEMBLY)) DS(‘<the trace ds>’)


The formatted trace will now be in <some dsname>. Do a binary transfer to your pc and you show be able to open the trace in wireshark.

I tend to use IPCS in batch (so I do not get the lock on the trace file, (grin)).

Problems

I was using CTRACE, the IPCS, then CTRACE etc. Sometimes the CTRACE writer would not start (or stop). This can be because you are still in IPCS, and IPCS has a lock on the dataset, or you have quit from IPCS, and TSO still has a lock on the data set. I used =X to get out of the ISPF session, and this usually freed it up.

I could it was easiest to run IPCS in batch – it saved lots of typing, and starting/stopping CTRACE when I had got it wrong.

There may still be a timing window, where the trace writer would not stop. When I stopped gsksrvr, the trace writer stopped.