Category: System SSL

Using System SSL trace to debug AMS problems.

When using MQ AMS to encrypt MQ messages, it has been very frustrating to get messages like

CSQ0217E %MQPA CSQ0COPN Failed to process object ‘recipient public key certificate in recipient keyring’

and not know what the problem is. (Especially when the cause is recipient public key certificate is not in recipient keyring.

System SSL which provides all of the encryption has a trace capability. You enable it when you start AMS, so you have the overhead in normal operation.

It sometimes writes data to a file in Unix, other times you need to start GKSRVR and collect the CTRACE.


With AMS the trace data is written to a file in Unix. Use the unix gsktrace command to format it.

gsktrace gskssl.16777257.trc > gsk

Configuring the trace

In the AMS JCL procedure //ENVARS DD DSN=….,DISP=SHR

I have

####################################################################### 
# 
#  GSKIT Trace 
# 
#  If you are having problems with certificates and missing entries in 
#  key stores you can enable GSKIT trace to help diagnose the problems. 
# 
#  To enable/disable GSKIT trace, un-comment and update the following 
#  environment variables as required. 
# 
#  GSK_TRACE_FILE - Specifies the name of the trace file in Unix 
#                   System Services (USS). Insert the correct user 
#                   name and ensure you set the correct permissions 
#                   on the directories and files to allow the 
#                   user ID in use by IBM MQ for z/OS Advanced 
#                   Message Security to write the resulting trace 
#                   file. The current process identifier is included 
#                   as part of the trace file name when the name 
#                   contains a percent sign (%). The gsktrace command 
#                   can be entered in USS to format the trace: 
# 
#                   e.g. gsktrace gskssl.84017302.tr 
# 
#  GSK_TRACE      - Specifies a bit mask that enables System SSL trace 
#                   options to trace gsk_* calls. All trace options are 
#                   enabled if the bit mask is 0xff and all trace 
#                   options are disabled if the bit mask is 0x00. 
                                                                              
GSK_TRACE_FILE=/tmp/gskssl.%.trc 
#SK_TRACE=0xff     everything    including payloads 
GSK_TRACE=0x0f     just control stuff

If you specify GSK_TRACE=0xff you get a lot of data – including all the application data ( message content + any headers etc).

I found GSK_TRACE=0x0f provides what I need.

A first look at the data

There is data like

08/06/2026-07:00:49 Thd-0 ENTRY gsk_open_keyring(): ---> Keyring 'STCMQ/drq.ams.keyring' 
08/06/2026-07:00:49 Thd-0 ENTRY crypto_generate_random_bytes(): ---> Length 20 
08/06/2026-07:00:49 Thd-0 EXIT crypto_generate_random_bytes(): <--- Exit status 0x00000000 (0) 
08/06/2026-07:00:49 Thd-0 INFO gsk_open_keyring(): Record 'AMS' is the default key 
08/06/2026-07:00:49 Thd-0 INFO gsk_open_keyring(): Identifier 1 assigned to 'AMS' 
08/06/2026-07:00:49 Thd-0 ENTRY gsk_decode_certificate(): ---> 
08/06/2026-07:00:49 Thd-0 EXIT gsk_decode_certificate(): <--- Exit status 0x00000000 (0) 
08/06/2026-07:00:49 Thd-0 INFO gsk_open_keyring(): Identifier 2 assigned to 'NEW-CA' 
08/06/2026-07:00:49 Thd-0 ENTRY gsk_decode_certificate(): ---> 
08/06/2026-07:00:49 Thd-0 EXIT gsk_decode_certificate(): <--- Exit status 0x00000000 (0) 
08/06/2026-07:00:49 Thd-0 INFO gsk_open_keyring(): Identifier 3 assigned to 'DOCZOSCA' 
08/06/2026-07:00:49 Thd-0 ENTRY gsk_decode_certificate(): ---> 
08/06/2026-07:00:49 Thd-0 EXIT gsk_decode_certificate(): <--- Exit status 0x00000000 (0) 
08/06/2026-07:00:49 Thd-0 INFO gsk_open_keyring(): Identifier 4 assigned to 'AMS2' 
08/06/2026-07:00:49 Thd-0 ENTRY gsk_decode_certificate(): ---> 
08/06/2026-07:00:49 Thd-0 EXIT gsk_decode_certificate(): <--- Exit status 0x00000000 (0) 
08/06/2026-07:00:49 Thd-0 ENTRY gsk_decode_certificate_extension(): ---> Decoding
08/06/2026-07:00:49 Thd-0 EXIT gsk_decode_certificate_extension(): <--- Exit status 
08/06/2026-07:00:49 Thd-0 ENTRY gsk_decode_certificate_extension(): ---> Decoding
08/06/2026-07:00:49 Thd-0 EXIT gsk_decode_certificate_extension(): <--- Exit status 
08/06/2026-07:00:49 Thd-0 INFO gsk_build_issuer_chains(): Record 'NEW-CA' is issuer for ...
08/06/2026-07:00:49 Thd-0 INFO gsk_build_issuer_chains(): Record 'NEW-CA' is self-signed 
08/06/2026-07:00:49 Thd-0 INFO gsk_build_issuer_chains(): Record 'DOCZOSCA' is self-signed 
08/06/2026-07:00:49 Thd-0 ENTRY gsk_decode_certificate_extension(): ---> Decoding ...
08/06/2026-07:00:49 Thd-0 EXIT gsk_decode_certificate_extension(): <--- Exit status ... 
08/06/2026-07:00:49 Thd-0 INFO gsk_build_issuer_chains(): No issuer found for record 'AMS2' 
08/06/2026-07:00:49 Thd-0 EXIT gsk_open_keyring(): <--- Exit status 0x00000000 (0) Handle...
  • I ignore the date
  • I use the time to match the records to when the problem occurred. It is HH:MM:SS granularity
  • Thd-0, Thd-15. Work runs on AMS tasks. A task can run work for any userid. I’ve often found one thread does the processing for the putters and the getters. You have to guess when one piece of work ends, and the next starts.
  • ENTRY/EXIT/INFO this is the entry, exit, or just some information about the function
  • gsk_open_keyring() is the function. This is described in Cryptographic Services System Secure Sockets Layer Programming (SC14-7495-60) which I think is only available in PDF format.

The trace shows records between

  • ENTRY gsk_open_keyring(): —> Keyring ‘STCMQ/drq.ams.keyring’
  • EXIT gsk_open_keyring(): <— Exit status 0x00000000

Showing the function gsk_open_keyring(): worked successfully.

It shows the elements on the keyring

INFO gsk_open_keyring(): Record 'AMS' is the default key
INFO gsk_open_keyring(): Identifier 1 assigned to 'AMS'
INFO gsk_open_keyring(): Identifier 2 assigned to 'NEW-CA'
INFO gsk_open_keyring(): Identifier 3 assigned to 'DOCZOSCA'
INFO gsk_open_keyring(): Identifier 4 assigned to 'AMS2'

How do you find the problem?

You might have the time of day when the problem occurred. The trace is only HH:MM:SS, so there may be a lot of records.

I use ISPF edit.

  • X ALL
  • Find ‘Exit status’ all
  • exclude ‘Exit status 0x00000000’ all

This shows just the Exit status which are non zero. This should localise the problem.

Periodically it uses a new file – so check you are using the latest file.

Some records showing a problem

 ENTRY gsk_validate_certificate(): ---> Data source count=1 
 ENTRY gsk_validate_certificate_mode(): ---> Data source count=1,...
 ENTRY cms_validate_certificate_mode_int(): ---> Data source count=1,...
 INFO cms_validate_certificate_mode_int(): validate root=566,...
 ENTRY gsk_name_to_dn(): ---> 
 EXIT gsk_name_to_dn(): <--- Exit status 0x00000000 (0) DN: CN=START1,O=AMS,C=TEST 
 INFO cms_validate_certificate_mode_int(): Validating CN=START1,O=AMS,C=TEST 
 INFO cms_validate_certificate_mode_int(): No Signature Algorithm List Provided 
 ENTRY gsk_decode_certificate_extension(): ---> Decoding certificate extension type 2 
 EXIT gsk_decode_certificate_extension(): <--- Exit status 0x00000000 (0) 
 INFO get_issuer_certificate(): Using AuthorityKeyIdentifier to locate issuer 
 ENTRY gsk_name_to_dn(): ---> 
 EXIT gsk_name_to_dn(): <--- Exit status 0x00000000 (0) DN: CN=NEW-CA2,OU=TEST,O=COLIN 
 INFO get_issuer_certificate(): Using issuer CN=NEW-CA2,OU=TEST,O=COLIN 
 ENTRY gsk_get_record_by_subject_mode(): ---> Handle 275787C0 
 EXIT gsk_get_record_by_subject_mode(): <--- Exit status 0x0335300e (53817358) 
 ERROR validate_certificate(): Unable to get issuer certificate: Error 0x03353024 
 ERROR validate_certificate_mode(): Unable to validate certificate: Error 0x03353024

This shows data is being processed from the message.

  • ENTRY gsk_name_to_dn(): —> shows data was passed in to routine to convert from the transmission (internal) format DN to a readable name
  • EXIT gsk_name_to_dn(): <— Exit status 0x00000000 (0) DN: CN=START1,O=AMS,C=TEST The output name is CN=START1,O=AMS,C=TEST
  • INFO get_issuer_certificate(): get the issuer name from the certificate
  • EXIT gsk_name_to_dn(): <— Exit status 0x00000000 (0) DN: CN=NEW-CA2,OU=TEST,O=COLIN this is the decoded issuer name
  • INFO get_issuer_certificate(): Using issuer CN=NEW-CA2,OU=TEST,O=COLIN look for the certificate with this DN in the keyring
  • EXIT gsk_get_record_by_subject_mode(): <— Exit status 0x0335300e It could not find the record in the keyring with the given subject DN.
  • Looking in the documentation gives 0335300E : Record not found.

First steps in using system ssl

System SSL also known as GS kit provides an API for doing

  • TLS handshakes
  • Keyring manipulation
  • Encryption and signing of data

This blog post covers background information for using GSKit. I wrote it because I could not find useful samples to get me started.

Introduction

It took me some time to understand how System SSL has been written. Now I understand it, I think it is a good API. They can be more than one way of doing something, so working out what you want to to can be hard.

Use of structures

It took me a while to understand the structures. I thought they were a little complex, and started designing my own equivalent, but I then found my “improved” definition were very similar to the original ones.

For example, I want to process some application data. It has an address and a length.
With my simplistic view I would define

  • char * pBuffer;
  • int lBuffer;

You could the use these in your application.

GSKIT provides the same files in a structure.

gsk_buffer mybuffer;
mybuffer.data = "ABCD";  // data is a void * pointing to the data
mybuffer.length = 4;     //

I can then do

gsk_buffer in;
gsk_buffer out;
in.data = ...
in.length = ...
gsk_rc gsk_copy_buffer(*in,*out)

For gsk_buffers that are output from a GSKIT call you need to free the contents.

gsk_free_buffer(&in);

This will release any storage which had been allocated to the structure, and set the values to 0.

Do not use this for gsk_buffers you created, because it will free the storage the data is using.

Arrays of objects

Some API functions take one certificate, and others take an array of certificates.

You define one certificate-key 

 pkcs_cert_key cert_key;

For an array of certificate keys you use

// Define the array (of 2)
pkcs_cert_key cert_key[2]; 
// Initialise it
cert_key[0].certificate =  cert; 
cert_key[0].privateKey  =  pRecord -> privateKey;
cert_key[1].certificate =  cert1; 
cert_key[1].privateKey  =  pRecord1 -> privateKey;
// Define the structure of array of keys
pkcs_cert_keys cert_keys; 
cert_keys.count = 2; 
cert_keys.certKeys = &cert_key[0];

Processing arrays of objects

void print_certs(pkcs_certificates * pcs)
{
  // pkcs_certificate has a count and array of certs
  // iterate over them all
  int i;
  pkcs_certificate * pc = & pcs-> certificates[0];
  for (i = 1; i <= pcs-> count;i++)
  {
    printf("Certificate %i\n",i);
    printf("==========\n");
    print_cert(pc);
    pc ++;
    printf(" \n");
  }  
                  
}

Where print_cert() is the program I wrote which prints information about one certificate.

Useful functions

Most gskit functions return a gsk_status. You can print the interpretation of this using

if ( gskrc != 0)
  printf("Error return: %s\n" gsk_strerror(gskrc));

Definitions are in

  • /usr/include/gsktypes
  • /usr/include/gskcms.h

First steps in using system ssl: compiling a program

System SSL also known as GS kit provides an API for doing

  • TLS handshakes
  • Keyring manipulation
  • Encryption and signing of data

See First steps in using system ssl gives some introductions to gskit.

The blog post provides the script I used to compile my programs. Not everything may be relevant.

name=main 
export _C89_CCMODE=1 
p5="-I.  -I/usr/include " 
p7="-Wc,DLL " 
p8="-Wc,LIST(c.lst),SOURCE,XREX,SHOWINC" 
p8=" "  # no listing
/bin/xlc $p5  $p7 $p8  -c $name.c -o $name.o 
l1="-Wl,LIST=ALL,MAP,XREF         " 
bind=" /usr/lib/GSKCMS31.x /usr/lib/GSKSSL.x " 
/bin/xlc $name.o ${bind}  -o $name -v $l1 1>bind.lst

The output from the C compile goes into c.lst, and from the bind it goes into bind.lst

The gsk includes are in /usr/include.

For the bind you need /usr/lib/GSKCMS31.x /usr/lib/GSKSSL.x

Example of system ssl to display keyring information

System SSL also known as GS kit provides an API for doing

  • TLS handshakes
  • Keyring manipulation
  • Encryption and signing of data

This blog post covers how to access a keyring and use the information in it. I wrote it because I could not find useful samples to get me started.

Introduction

It took me some time to understand how System SSL has been written. Now I understand it, I think it is a good API. They can be more than one way of doing something, so working out what you want to to can be hard.

List the certificates in a keyring

// #include <gskssl.h>  // only used for TLS and SSL
#include <gskcms.h> 
#include <stdio.h> 
#include <stdlib.h> 
#include <string.h> 
#include <errno.h> 
#include <printhex.h>
int main(int argc, char * argv[]) { 
 int rc; 
 gsk_status gskrc; 
 gsk_handle hRing ; 
 int        count = 0;// the number of certs is put here
 char * ring = "START1/TN3270";
 printf("Open keyring %s\n",ring);
 gskrc=  gsk_open_keyring (ring, 
                &hRing, 
                &count ); 
 if (gskrc != 0)
 {
   printf("gsk_open_keyring %s\n",  gsk_strerror(gskrc)); 
   return 8;
 }
 printf("Number of certs in the ring %i\n",count); 
 //
 // Get the default label (if it exists)
 // 
 char * default_label;
 gskrc = gsk_get_default_label(hRing,&default_label);
    
 if (gskrc != 0)
 { 
   else printf("gsk_get_default_label error %s\n",gsk_strerror(gskrc));
   return 8;
 }
 printf("gsk_get_default_label %s\n",default_label);
 // 
 // now get the record with this label (or by number)
 //
 gskdb_record * pRecord = 0;
 gskrc =  gsk_get_record_by_label(hRing,default_label,&pRecord);   
 //gskrc = gsk_get_record_by_id(hRing,4,&pRecord); // by number
 if (gskrc != 0)
 {
    printf("gsk_get_record_by_label worked %s\n",gsk_strerror(gskrc));
    return 8;
 }  
 // The value is a half work enum
 printf("Record Type %hi\n",pRecord ->recordType);
 #include <readLabels.h> 
 #include <listall.>
 gskrc = gsk_close_database(&hRing); 
 return 0 ;
}

ReadLabels

With gsk_get_record_labels you can request those with private keys (1) or those without private keys(0)

If you want all certificates on the ring use without private_keys(0).

gskrc = gsk_get_default_label ( hRing,&default_label);
if (gskrc == 0)
    printf("gsk_get_default_label %s\n",default_label);
else printf("gsk_get_default_label error %s\n",gsk_strerror(gskrc));
char ** pl = 0;
int countL = 0;int whichRecords = 0;
printf("/nGet records with no private key (%i)\n",whichRecords);
gskrc = gsk_get_record_labels (hRing, 
           whichRecords, // those with public keys only
                  //  1 for this with private key 
           &countL, // How many returned
           &pl);  // char *** labels
if ( gskrc != 0)
{
  printf("gsk_get_record_labels%s\n",gsk_strerror(gskrc));
  return 8
}  
for (int j = 0; j<countL ;j++)
   {
     printf("Label %i,%s\n",j,*pl);
     pl++;
   }
    
printf("============\n");

It took me some time to get the list of labels printed out. The value is a char ***;

  • If you had char * p; p is a pointer to a string so is a char *.
  • char * q[10]; pl = &q[0]
  • pl is a pointer to the elements in the array. pl is addr(a char * ) so char **;
  • the address of pl is passed to the function, so it is addr( a char **) so is a char ***
  • In the loop we get what pl points to (*pl). This is a pointer to a string. so we have printf(“…”,*pl);

Listall

printf("Number of records %i\n",count);
for (int j = 1 ; j<count+1 ;j++) // starts at 1
    {
       gskrc = gsk_get_record_by_index(hRing,j,&pRecord);
       printf("Record %i:n",j);
       printRecord(pRecord);    
    }
    
printf("============\n");

printRecord

void printRecord( gskdb_record * pRecord)
{
  printf("Record Type %hi\n",pRecord ->recordType);
  printf("Record Flags %2.2x\n",pRecord ->recordFlags);
  printf("Record ID %i\n",pRecord ->recordId);
  printf("Record Issuer ID %i\n",pRecord ->issuerRecordId );
  printf("Record Label %s\n",pRecord ->recordLabel);
  printf("============\n");
}

Output

The program printed out 

Open keyring START1/TN3270
Number of certs in the ring 9
gsk_get_default_label NISTEC256
gsk_get_record_by_label worked
precrd 2  20a2e6e0
Sizeof gskdb_record_type 2
gsk_get_default_label NISTEC256

Get records with no private key (0)

Label 0,DOCZOSCA
Label 1,JUN24CA
Label 2,DocZosCADSA
Label 3,CA256

============
Number of records 9
Record 1:
Record Type 2
Record Flags 80
Record ID 1
Record Issuer ID 1
Record Label DOCZOSCA
...

Record 9:
Record Type 3
Record Flags c0
Record ID 9
Record Issuer ID 1
Record Label NISTEC256
============

  • The value of Record 1 type gskdb_rectype_certificate = 2, /* Certificate */
  • Flags is
    • GSKDB_RECFLAG_TRUSTED = The certificate is trusted ( 0x80);
    • GSKDB_RECFLAG_DEFAULT = This is the default key
  • ID 1 – record 1
  • Issuer ID 1 – it is self signed
  • Label DOCZOSCA
Record 9:
Record Type 3
Record Flags c0
Record ID 9
Record Issuer ID 1
Record Label NISTEC256

  • The record type is 3 gskdb_rectype_certKey = 3, /* Certificate with key */
  • Flags is
    • GSKDB_RECFLAG_TRUSTED = The certificate is trusted.
    • GSKDB_RECFLAG_DEFAULT = This is the default key
  • ID 9 – it is record 9 in the keyring
  • Issuer ID 1- it was signed by the certificate with position 1 (DOCZOSCA)
  • Its label is NISTEC256

Example of using system ssl to print certificate information

The structure containing information about a certificate has a very deep nesting of structures and arrays. Fortunately for the most common information, gskit provides an interface.

There is a standards document giving the layout of the certificate, and what the fields mean.

The structures are in /usr/include/gskcms.h. Some of the data types are in /usr/include/gsktypes.h .

The certificate structure

  • The structure pkcs_certificate>s has an array of pkcs_certificate and a count of the elements.
  • The structure pkcs_certificate has a certificate type and an x509_certificate.
  • The structure x509_certificate has
    • a tbsCertificate
    • signature algorithm
  • The x509_tbs_certificate structure (tbs from To Be Signed) has
    • the certificate version
    • the certificate serial number
    • the signature algorithm of the certificate
    • the issuer
    • the validity period
    • the subject.

The fields

  • The certificate serial number. This a hexadecimal value. The definition has a pointer to the data, and a length of the data. The length is at least one.
  • The signature algorithm of the certificate. This includes
    • The signature type. This is a C enum, such as x509_alg_sha256WithRsaEncryption = 25
  • The issuer is a complex nest of structure. Fortunately gskit provide a function gsk_name_to_dn(structure,name). The output “name” is a printable EBCDIC string like CN=DocZosCA,OU=CA,O=COLIN
  • The validity period. This structure contains two time-value fields, not before, and not after.
    • You can format a time value field using
      • char dest[18];
      • strftime(dest,18,”%y/%m/%d %H:%M:%S”,timeval);
  • The subject is like the issuer. Use the function gsk_name_to_dn to convert it to printable EBCDIC. For example CN=10.1.1.2,OU=SSS,O=NISTECC521

Printing the structures

I have some functions like

void print_certs(pkcs_certificates * pcs)
{
  // pkcs_certificate has a count and array of certs
  // iterate over them all
  int i;
  pkcs_certificate * pc = & pcs-> certificates[0];
  for (i = 1; i <= pcs-> count;i++)
  {
    printf("Certificate %i\n",i);
    printf("==========\n");
    print_cert(pc);
    pc ++;
    printf(" \n");
  }  
                  
}

which invokes

void print_cert(pkcs_certificate * pc )
{
    print_x509_algorithm_type(pc ->u.certificate. signatureAlgorithm
            .type,0);    
    // Serial number
    char output[60]; // into here
    // hexof is written by Colin Paice      
    printf("Serial number Hex     :%s\n",hexof(
        pc ->u.certificate.tbsCertificate.serialNumber.data,
        pc ->u.certificate.tbsCertificate.serialNumber.length,
        &output[0])    );
    
    // get the subject and issuer names
    gsk_status gskrc; 
    char * pName;   // returned here - must free it
    pName   = "Unknown"; 
    gskrc =  gsk_name_to_dn(
            & pc ->u.certificate.tbsCertificate.subject,
            &pName); 
  
    if (gskrc != 0)  
        printf("Subject gsk_name_to_dn  %s\n",  gsk_strerror(gskrc));
    
    printf("Subject name          :%s\n",pName);
    gsk_free_string(pName);           
    
  // extract the issuer,       
    pName   = "Unknown"; 
    gskrc =  gsk_name_to_dn (
        & pc ->u.certificate.tbsCertificate.issuer,
        &pName);      
    if (gskrc != 0) 
        printf("Issuer gsk_name_to_dn  %s\n,  gsk_strerror(gskrc)); 
    printf("Issuers name          :%s\n",pName); 
    gsk_free_string(pName); 
    
    // extract the valid times.
    char   buffer[18]; 
    printf("Not Before            :%s\n",         
        ccpformattime ( 
            & pc ->u.certificate.tbsCertificate.validity.notBefore, 
            &buffer[0]     ) );
     printf("Not After             :%s\n",
        ccpformattime ( 
            & pc ->u.certificate.tbsCertificate.validity.notAfter, 
            &buffer[0] ));
    
      print_x509_extensions(&pc -> 
             u.certificate.tbsCertificate.extensions);
}

Process DER ASN.1

Some data is encoded in ASN.1 format.

This is of format

  • 1 byte of type
  • 1 or more bytes of length
  • Value, where value can be
    • Integer data
    • ASCII string data
    • …
    • A sequence of things ..

For example

16304765 6E657261… is

  • Type 16 is an ASCII string
  • of length 0x30
  • with value 0x4765… which translates to EBCDIC Genera…

30068704 0A010102

  • Type 30 is a sequence (it happens to be a sequence of one element)
  • Of length 6
    • The value is a dotted decimal IP address of length 4
    • with value 0A010102 – which is 10.1.1.2.
Certificate 2
==========
Certificate algorithm :x509_alg_sha256WithRsaEncryption
Serial number Hex     :68
Subject name          :CN=10.1.1.2,OU=SSS,O=NISTEC256
Issuers name          :CN=DocZosCA,OU=CA,O=COLIN
Not Before            :26/04/05 00:00:00
Not After             :27/04/05 23:59:59
Extension 1
============
Extension type:x509_ext_unknown
DER IA5String: Generated by the Security Server for z/OS (RACF)
 
Extension 2
============
Extension type:x509_ext_subjectAltName
DER Dotted address: 0x0A010102 
 
Extension 3
============
Extension type:x509_ext_keyUsage
DER BitString: 0x0780
Sample output

Extension 4
============
Extension type:x509_ext_subjectKeyIdentifier
DER Octet: 0xF421DD55 F00BEE6E E81B97D2 1D5556AB A671EC59 
 
Extension 5
============
Extension type:x509_ext_authorityKeyIdentifier
DER Context: 0xCAD12CE6 AACB598C 6E9595CD D812C90A C9C4C289

Example of using system ssl to sign some data

System SSL, also known as GS kit, provides an API for doing

  • TLS handshakes
  • Keyring manipulation
  • Encryption and signing of data

This blog post covers how to sign data. I wrote it because I could not find useful samples to get me started.

Signing data

To digitally sign data, you

  • perform a check sum calculation on the data,
  • encrypt the checksum with a private key,
  • package up make the data, the encrypted checkum, and the originator’s public certificate.

The recipient

  • checks the validity of the certificate
  • extracts the public key
  • decrypts the encrypted checksum
  • does a checksum of the enclosed data
  • compares the two checksums.

The checksums should match.

How the defining structures work together

Reference to fields

The definition of a record read from a keyring is

typedef struct _gskdb_record { 
    gskdb_record_type           recordType; 
    gsk_octet                   recordFlags; 
    gsk_octet                   rsvd_1[1]; 
    gsk_int32                   recordId; 
    gsk_int32                   issuerRecordId; 
    char *                      recordLabel; 
    union { 
        x509_certificate        certificate; 
        pkcs_cert_request       request; 
    } u; 
    pkcs_private_key_info       privateKey; 
    gsk_octet                   rsvd_2[16]; 
} gskdb_record;

This has information about certificate, and the private key.

If you have a variable pRecord as a pointer to this structure, you can refer to the private key as pRecord->privateKey.

Because this structure can be used for both a certificate and a certificate request to refer to a certificate you have to use pRecord->u.certificate to locate the certificate.

Arrays of objects

Some API functions take one certificate, and others take an array of certificates. See Arrays of objects.

To sign some data

printf("....SIGN the data\n");         
//  pRecord points to the default record 
gsk_buffer indata; 
gsk_buffer outdata; 
// Create the data to sign
indata.data = "ABCDEF"; 
indata.length = 6; 
outdata.data  ="OUTDATA "; // dummy data
outdata.length = 6 ;    
// Define the certificate(s) to be used
pkcs_certificate cert; 
cert.type = pkcs_certtype_certificate  ; 
cert.u. certificate = pRecord -> u.certificate;
pkcs_cert_key cert_key[1];
cert_key[0].certificate =  cert; 
cert_key[0].privateKey  =  pRecord -> privateKey; 
// create our structure of an array of keys
// Note the structure is pkcs_cert_key>s< which contains 
// one or more cert_key structures.
pkcs_cert_keys cert_keys; 
cert_keys.count = 1; 
cert_keys.certKeys = &cert_key[0]; 
  
// pkcs_certificates certs; 
                
gsk_process_option po; 
po.enforce_keyusage = 0; 
po.enforce_content_length = 0; 
po.enforce_keyparity = 0; 
po.create_detached_signature  = 0; 
gskrc =  gsk_make_signed_data_msg_extended 
        (po, // no options
         0, // version 
         x509_alg_sha1Digest, // algoritm 
         1, // include certs in the output
         &cert_keys, // the certs to use
         0, // no ca certs 
         &indata, 
         0, // no attrib signers 
         &outdata); 
          
if (gskrc == 0) 
   printf("gsk_make_signed_data_msg_extended worked\n"); 
else 
{ 
    printf("gsk_make_signed_data_msg_extended. %8.8x %s\n",
           gskrc,gsk_strerror(gskrc)); 
    return 8; 
} 
printf("Outdata length %i\n",outdata.length);
printHex(stdout,outdata.data,outdata.length);

This produced

00000000 : 3082048C 06092A86 4886F70D 010702A0  ................ .b.....f.f7..... 
00000010 : 82047D30 82047902 0100310B 30090605  ................ b.'.b.`......... 
00000020 : 2B0E0302 1A050030 1506092A 864886F7  ................ ............f.f7 
00000030 : 0D010701 A0080406 C1C2C3C4 C5C6A082  ................ ........ABCDEF.b 
00000040 : 03023082 02FE3082 01E6A003 02010202  ................ ...b...b.W...... 
00000050 : 0168300D 06092A86 4886F70D 01010B05  ................ .......f.f7..... 
00000060 : 00303031 0E300C06 0355040A 1305434F  ................ ...............| 
00000070 : 4C494E31 0B300906 0355040B 13024341  ................ <.+............. 
...
00000460 : B6CBA3F3 48B5C9CD 76063F2D 007840A7  ................ ..t3..I....... x 
00000470 : 0C8EF6FE 5442459E 47F6C1D0 12F1247E  ................ ..6......6A}.1.= 
00000480 : 0DC79B03 7723D129 078D1DB9 812F6EB9  ................ .G....J.....a.>.

In the data is

  • ABCDE – the original data
  • 3082048c – I recognise this as ANS-1 encoding. (ASN.1) is a standard interface description language (IDL) for defining data structures that can be serialized and deserialized in a cross-platform way. It is broadly used in telecommunications and computer networking, and especially in cryptography.
    • For example 048c is the length of data following

Once the signature has been obtained, it can be externalised (such as writing to a file) and sent to a remote site.

Because the data is in hexadecimal you may wish to convert it to EBCDIC for transport. You can use the 

gsk_encode_base64 (gsk_buffer input_data,gsk_buffer_output_data);

to encode the data and use the gsk_decode_base64() to decode it.

Validate the data

The recipient can check that the signature and certificates are valid.

// Process the validation of the signature
// The list of input signatures 
pkcs_certificates certs; 
certs.count = 1;
certs.certificates = &cert;
gsk_attributes_signers as; 
gsk_buffer temp;  // process the data into temp.
printf("============READSIGNED======\n"); 
gsk_boolean ul; // were local certificates used?
gskrc = gsk_read_signed_data_msg_extended( 
         po, // no option 
         NULL,  // no local certificates provided / local_certificates, 
         &outdata,  // the signed data  
         &ul, // Set if certs were set locally or not 
         NULL, // returned cert not needed // msg_certificates 
         NULL, // signer_certificates 
         &as, // attrbute signers 
         &temp); 
     if (gskrc == 0) 
     { 
         printf("gsk_read_signed_data_msg_extended worked\n"); 
         printf("Use local certificates %i\n",ul);
         printHex(stdout,temp.data,temp.length); 
     } 
     else 
     { 
         printf("gsk_read_signed_data_msg_extended. %8.8x %s\n",
                gskrc,gsk_strerror(gskrc)); 
         return 8; 
     }

When this executed it produced

gsk_read_signed_data_msg_extended worked
Use local certificates 0
00000000 : C1C2C3C4 C5C6                        ......           ABCDEF

which matches the original input data.

With 

gskrc = gsk_read_signed_data_msg_extended( 
         po, // no option 
         NULL,  // no local certificates provided / local_certificates, 
         ...

Only certificates in the data were used. “Use local certificates” was 0.

When

gskrc = gsk_read_signed_data_msg_extended( 
         po, // no option 
         &certs, // use local certificates provided
         ...

it used the certificate(s) from my keyring. “Use local certificates” was 1.

My final code was

pkcs_certificates msgcerts;
pkcs_certificates signercerts;
gsk_attributes_signers as; 
gskrc = gsk_read_signed_data_msg_extended( 
         po, // no option 
         &certs, // local_certificates, 
         &outdata,  // stread output
         &ul, // Set if certs were set locally or not 
         &msgcerts,  // returned msg_certificates 
         &signercerts, // signer_certificates 
         &as, // attrbute signers 
         &temp); 
if (gskrc == 0) 
{ 
    printf("gsk_read_signed_data_msg_extended worked\n"); 
    printf("Use local certificates %i\n",ul);
    printHex(stdout,temp.data,temp.length); 
    printf("====MSGCERTS+++\n");
    // print_pkcs_certificates(&msgcerts);
    print_certs(&msgcerts);
    printf("====Signers++\n");
    print_certs(&signercerts);
    // print_pkcs_certificates(&signercerts);
} 
else 
{ 
    printf("gsk_read_signed_data_msg_extended. %8.8x %s\n",
           gskrc,gsk_strerror(gskrc)); 
    return 8; 
}

z/OS systems-ssl strange behaviour with environment variables

I was trying to use system ssl to write a program to use native z/O TLS facilities. I wasted a couple of hours because it said it could not find my keyring. Then when I collected a trace, it sometimes did not find the file – which did exist as I could list it.

If I used

//START1   EXEC PGM=GSKMAIN,REGION=0M, 
//*  PARM='4000' 
// PARM=('ENVAR("_CEE_ENVFILE=DD:STDENV")/4000') 
//STDENV DD PATH='/u/ibmuser/gskparms'

When the USS file had

GSK_TRACE_FILE=/tmp/zzztrace.file 
GSK_TRACE=0xff 
GSK_KEYRING_FILE=START1/TN3270

This worked file

When I used

//START1   EXEC PGM=GSKMAIN,REGION=0M, 
//*  PARM='4000' 
// PARM=('ENVAR("_CEE_ENVFILE=DD:STDENV")/4000') 
//STDENV  DD * 
GSK_TRACE_FILE=/tmp/zzztrace.file 
GSK_TRACE=0xff 
GSK_KEYRING_FILE=START1/TN3270 
/*

This failed to work.

If I looked in the trace file I had

ENTRY gsk_open_keyring(): ---> Keyring 'START1/TN3270                       '

Where it had taken the whole length of the line – and so START1/TN3270 padded with blanks was not found.

The trace file was not /tmp/zzztrace.file, it was /tmp/zzztrace.file padded with lots of blanks!

The answer is to use a environment file in USS, not in JCL or a data set.

Getting a system ssl trace for httpd server on z/OS

I had a problem getting the httpd server to work on z/OS. It did not like my certificate – but reported messages about ICSF security not being set up. I got to learn a lot about traces, but could not find how to trace System SSL (GSK) and httpd.

I specified SSLTRACE in my vhost*.conf file which gave me a little information – but not at the System SSL Level.

Other applications using System SSL, can set environment variables 

GSK_TRACE=0xff 
GSK_TRACE_FILE=/tmp/gskssl.%.trc

but this does not work with http. (I think the environment variables are not passed on to any spawned thread).

You have to use the gsksrvr task, and collect the trace through CTRACE.

Set up gsk trace.

I have used gsk trace before, and described setting it up.

I had to create a parmlib member

TRACEOPTS 
          WTRSTART(ctwtr) 
          on 
          wtr(ctwtr) 
jobname(httpcp) 
          OPTIONS('LEVEL=255,JOBSUFFIX=ANY')

I start my http server with the S HTTPCP command, and specified httpcp in the jobname of the parmlib.

I got out no trace. I tried the various jobnames until it produced a trace. My trace was produced from jobname httpcp8! I could not find a way of displaying which of my httpcp* job was used, so I had to try them all.

If I had had a long name eg httpcpxx then specifying jobname(httpcpxx) should have worked.

Starting the trace

I used 

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON

where my parmlib member was CTGSKOK

It produced

TRACE CT,ON,COMP=GSKSRVR,PARM=CTGSKON
IEE252I MEMBER CTGSKON FOUND IN USER.Z24C.PARMLIB
GSK01040I SSL component trace started.
ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE
SUCCESSFULLY EXECUTED.

and the CTWTR started up.

What is the status of the trace?

d trace,comp=gsksrvr

gave me 

IEE843I 08.32.25  TRACE DISPLAY       
  COMPONENT     MODE BUFFER HEAD SUBS                                  
 --------------------------------------------------------------        
  GSKSRVR       ON   0064K                                             
     ASIDS      *NOT SUPPORTED*                                        
     JOBNAMES   HTTPCP8 ,HTTPCP1 ,HTTPCP2 ,HTTPCP3 ,HTTPCP4 ,          
                HTTPCP5 ,HTTPCP6 ,HTTPCP7                              
     OPTIONS    LEVEL=255                                              
     WRITER     CTWTR

Run my test

When I ran my test, some System SSL messages were produced on the console from the gsksrvr address space

GSK01047I SSL component trace started for HTTPCP8/STC01000.
GSK01050I SSL component trace started for HTTPCP8/STC01000/05010022.

Stop the trace

TRACE CT,OFF,COMP=GSKSRVR

Wait until you get

GSK01041I SSL component trace ended.

from the gsksrvr address space, and stop the trace writer.

D TRACE,WTR=ALL
TRACE CT,WTRSTOP=CTWTR

This seems to take a few seconds to run. It outputs

IEF196I AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA : 
IEF196I           IBMUSER.CTRACE1 
ITT111I CTRACE WRITER CTWTR TERMINATED BECAUSE OF A WTRSTOP REQUEST.  
IEF404I CTWTR - ENDED - TIME=08.40.21

Format the trace

You need to use IPCS to format it

  • =0 – and enter the data set name (IBMUSER.CTRACE1)
  • =6 – to get to the ipcs command screen
  • dropd – to tell IPCS to forget any historical information it may know about for the dataset
  • CTRACE COMP(GSKSRVR) full – this displays any System SSL CTRACE data
  • m PF8 – go to the bottom of the data
  • report view – to go into ISPF View mode on the data set
  • X ALL – to hide all of the data
  • f SSL_ERROR ALL – this shows any error codes
  • if you get any lines displayed, you can tab down to the hidden command and use the line prefix command f4 to display the first 4 hidden lines.

The errors I got were

 S0W1      MESSAGE   00000004  08:35:55.049451  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_ec_token_private_key_sign 
   ICSF service failure: CSFPPKS retCode = 0x8, rsnCode = 0x2b00 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.049733  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  crypto_sign_data 
   crypto_ec_sign_data() failed: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050012  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  construct_tls13_certificate_verify_message 
   Unable to generate certificate verify message: Error 0x03353084 
                                                                                                              
 S0W1      MESSAGE   00000004  08:35:55.050393  SSL_ERROR 
   Job HTTPCP8   Process 05010022  Thread 00000005  send_tls13_alert 
   Sent TLS 1.3 alert 51 to ::ffff:10.1.0.2.45432.

The Alert 51 matches what my browser received.

File /usr/include/gskcms.h had #define CMSERR_ICSF_SERVICE_FAILURE 0x03353084

CSFPPKS is PKCS #11 Private Key Sign.

The return code 0x2b00 (from here) gives:

User action: You might need to re-create the token by using the PKA key token build or PKA key import callable service or regenerate the key values on another platform.

or in other words, it did not like my certificate created with NISTECC SIZE(256) but did like NISTECC SIZE(521).

z/OS PAGENT error messages and system SSL codes

Message

SYSERR :001: plfm_kernel_init: socket(INET, DGRAM, 0), failed, errno=EDC5112I Resource temporarily unavailable., errno2=74610296

OBJERR :001: init_PEP_and_kernel: Kernel initialization failed for image ‘TCPIP2’,

RACF profile EZB.INITSTACK.*.TCPIP2 CLASS(SERVAUTH) was missing.

//S1 EXEC PGM=IKJEFT01,REGION=0M
//STEPLIB DD DISP=SHR,DSN=SYS1.MIGLIB
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RDEFINE SERVAUTH EZB.INITSTACK..TCPIP2 PERMIT EZB.INITSTACK..TCPIP2 CLASS(SERVAUTH)
ID(START1) ACCESS(READ)
SETROPTS RACLIST(SERVAUTH) REFRESH
/*

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file

In my image file I had a

TTLSRule
{
…

}

which is not allowed. They have to be in the TTLSConfig… file.

The error message was

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file ‘TTLSRule’ statement

WARNING:006: pinit_fetch_policy_profile: Ignoring unknown image config file ‘TLSConfig’ statement

This is saying the TLSConfig statement is not recognised. It should be TTLSConfig! This took me a couple of hours to resolve because I had commented out TTLSConfig with … #TLSConfig and replaced the # with a blank.

402 No SSL cipher specifications.

In some situations the cipher specs default.

The message EZD1287I TTLS Error RC: 402 Initial Handshake pointed me to Cryptographic Services System Secure Sockets Layer Programming – No SSL cipher specifications. The first reason was

The client and server cipher specifications do not contain at least one value in common. Client and server cipher specifications might be limited depending on which System SSL FMIDs are installed. See Cipher suite definitions for more information. Server cipher specifications are dependent on the type of algorithms that are used by the server certificate (RSA, DSA, ECDSA, or Diffie-Hellman), which might limit the options available during cipher negotiation.

The GSK trace had GSK_CONNECT_CIPHER_SPEC(207) – C02B. Where C02B is TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

The certificate specified for the server was incompatible with this. I had Key Usage: HANDSHAKE
Key Type: NIST ECC
Key Size: 384

The one that worked had

Key Usage: HANDSHAKE
Key Type: NIST ECC
Key Size: 521

RC 416 GSK_ERR_PERMISSION_DENIED


Not authorized to access key database, PKCS #12 file, SAF key ring or z/OS PKCS #11 token.

EZD1287I TTLS Error RC: 416 Environment Master Init

ICH408I USER(START1 ) GROUP(SYS1 )
START1.TN3270.LST CL(RDATALIB)
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

Action: Give the started task userid permission to the ring.

RC 510 No acceptable key labels found

See here. The specified label was inconsistent with the client request.

The Wireshark output had a list of Cipher Suites, for example

   Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
   Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
   Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)

In my keyring I had only ECC certificates. When I created an RSA certificate it worked (well moved on a step).

RACDCERT ID(START1) GENCERT -                        
  SUBJECTSDN(CN('10.1.1.2') - 
             O('RSA2048') -                          
             OU('SSS')) -                            
   ALTNAME(IP(10.1.1.2))-                            
   RSA     - 
   SIZE(2048)  - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -           
   WITHLABEL('RSA2048')        -

RC: Code 515 Initial handshake

Code 515 Key share list is not valid

I got this when trying to use TLS 1.3.

You need

TTLSSignatureParms
{
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0030

}

or ServerKeyShareGroup

517 No matches between elliptic curve and key share lists

Need something like

TTLSSignatureParms CPESigParms
{
CLientECurves 0023
CLientECurves 0024
CLientECurves 0025
CLientECurves 0029
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0029
}

519 Required ciphers have not been specified

I had TLS 1.3 specified, but no TLS 1.3 cipher specs.

The following TLSv1.3 worked.

V3CipherSuites TLS_CHACHA20_POLY1305_SHA256

EZD1287I TTLS Error RC: 5006 Initial Handshake

See the TCPIP diagnosis reference.

EZD1287I TTLS Error RC: 5006 Initial Handshake
5006 The connection is using a TTLSEnvironmentAction statement that failed
to initialize a System SSL environment.

  • Use the syslog to determine why the System SSL environment failed to
    initialize.
  • If the TTLSEnvironmentAction statement is in error, make the
    necessary corrections. A System SSL environment is initialized for the
    corrected TTLSEnvironmentAction statement and new connections use
    that environment.
  • If a SAF configuration change is needed (such as changing a
    certificate in the key ring), make the change and then update the
    EnvironmentUserInstance parameter in the TTLSEnvironmentAction
    statement to reflect a changed action. A System SSL environment
    is initialized by using the modified RACF configuration and new
    connections use that environment

I gave the started task permission to the keyring, and got this error.

When I restarted my application and it worked. I don’t know if this is relevant.

EZD1287I TTLS Error RC: 5003

RC 5003 will occur when the AT-TLS process is expecting an SSL message but receives a clear-text message.

System SSL 0x134cc000

The documentation for System SSL says

SSL error codes are defined internally to be ‘134CCxxx’, where xxx is the hexadecimal representation of the SSL error code. The range is ‘134CC001’ through ‘134CCFFF’. For example, SSL error code 414 would be ‘134CC19E’.

So xxx is 000 – which is no error. I got this using gsk_strerror(0) when there was no error!