Understanding ICSF fixed and variable format keys and how not to get confused when using them.

This is part of the project to set up ICSF so I can create an encrypted dataset on one system, and use it on a different system. This involves setting up keys for encryption, public/private keys and sending stuff between the two systems.

Dataset encryption keys are symmetric and reside in the CKDS.

It took me a while to understand the implications of the fixed and variable format keys, and lots of head scratching when things did not work as expected.

This post is a very simplistic view of the topic – but it should give you enough information to get started with ICSF.

One thing you need to know about (and then quickly forget) is that there are two format of keys.

  1. Fixed length – an example of this is a DATA key.
  2. Variable length – an example of this is is a CIPHER key.

I just think of there being two versions of APIs – Version 1 for Data and version 2 for Cipher. Any new project should use variable length keys.

You can use either DATA or CIPHER to encrypt a dataset.

Each type of key has its own APIs, so you cannot use a fixed key in an API designed for variable length keys.

Most ICSF APIs have “rules” which are like passing parameters to a command. This is an array of 8 character strings such as “AES “,”PKCS-1.2”,OP “. This string indicates

  • AES – encrypt an AES keyType
  • PKCS-1.2 using PKCS
  • OP “Operational” for use on this system.

When using the APIs’ to export keys, both ends must have matching configuration

For example with a DATA key any one of the following.

  1. AES,PKCS-12
  2. AES,PKCSOAEP
  3. AES,PKCSOAEP,SHA-512

I think the last set of parameters is the strongest.

For CIPHER, this worked

  1. AES, PKOAEP2, SHA-256

When programming I found it easier to create some helper routines to reduced the complexity of the APIs. For example I created a ADDKEY routine to specify PKDS|CKDS, the key name, and the buffer contents. As a result I had a high amount of reuse, and my main programs were very compact.

Below I give the API calls for

  • Using variable length Cipher keys
    • create the skeleton using CSNBKTB2
    • create the data key using CSNBKGN2
    • export the data key using CSNDSYX
    • import the data key (on another system) using CSNDSYI2
  • Using fixed length data keys
    • create the data key using CSNDSYG
    • export the data key using CSNDSYX
    • import the data key (on another system) using CSNDSYI

Using variable length Cipher keys

I see the variable length keys as an evolution in key management from the fixed length keys.

For example you can store the name of the key within the key (though I do not know when or how this is used).

Create the skeleton

I built a skeleton using CSNBKTB2. You can optionally pass in

  • a key name
  • user data

I did not set these (I set the lengths to zero (&zero)).

I passed in the rule

  • char 8 rule[4] = “INTERNAL”,”AES “,”CIPHER “, “ANY-MODE”

The ANY-MODE was required for the encryption to work.

Create the cipher key

I then created the key using CSNBKGN2 and passed in

  • char8 rule[2] = {“AES “,”OP “};
  • keyLength = 256 ; why use a weaker key?
  • char8 keyType1 = {“TOKEN “}; This says use the data passed in from the skeleton token.
  • char8 keyType2 = {” “};
  • The skeleton data

This returns a block of data (the AES Cipher key encrypted with the local key).

I added it to the local keystore using CSNBKRC2.

Export the cipher key

I exported the key using a public certificate with CSNDSYX. I passed

  • the name of the public key.
  • char8 rule_array[3] = { “AES “,”PKOAEP2 “,”SHA-256 “}

It returns a block of data containing the AES cipher key encrypted with the public key.

I wrote this data to a file (in binary) and sent it to my remote system.

Import the cipher key at the remote system

I read the file into a buffer, and used CSNDSYI2 to decrypt the contents using the private key, and encrypting it with the local key.

I passed

  • char8 rule_array[2] ={“AES “,”PKOAEP2 “}
  • the name of the private key
  • the buffer

it returned a buffer containing the re-encrypted key

I added it to the keystore on the remote system using CSNBKRC2.

__________________________________________________

If you want to use fixed length keys(why do you?)…

Using fixed length data keys

You can create a data key, and export it at a later date, or you can generate it and export it at the same time.

I feel more comfortable about exporting it when it is needed, in case the “old” copy is out of data.

If you export it from the live system you know it is current.

Create the data key

I used use CSNDSYG

with

  • a char 64 key name
  • char8 rule[3] = {“AES “, “PKCS-1.2″| PKCSOAEP, “OP “}; If you use PKCSOAEP you can specify SHA-512

This creates a data key in the local repository.

I think PKCSOAEP is better than PKCS-1.2 as is is more recent.

You can also get it to create an RSA enciphered version of the key using a public certificate (as part of key creation) This can be written to a file, and the file sent to the remote system. I tend not to use this, but export the key at the time when wanted. This way you can be sure you have the correct key.

Export the data key

Use CSNDSYX with similar parameters as for the create (CSNDSYG) to create an encrypted version of the Data key.

CSNDSYX returns a block of storage with the encrypted key in it. You can write this to a file in binary or create a base64 encoding of it.

Import the data key

Use CSNDSYI with

  • The block of encrypted data (which you can read from a file)
  • rule = “AES “, “PKCS-1.2″|| “PKCSOAEP”. If you use PKCSOAEP you can specify SHA-512
  • The parameters must match the sending end
  • The name of the private key in the PKDS to be used to decrypt the data

This returns a block of data which is the data key, encrypted with the local system’s key.

You can then add it to the CKDS using CSNBKRC2

ICSF: My first C application.

As part of setting up data set encryption to send data sets between two sites. I wrote some programs to help securely send the symmetric key exchange between two systems using private and public keys.

Basis flow to get a symmetric key on two systems to allow data set encryption and decryption, using private key and public key.

The steps to allow me to send an encrypted dataset to a remote system, using PKI are as follows

  1. On the remote system generate a private/public key in the PKDS
  2. Extract the public key and send it to the local system
  3. Import the public key into the local PKDS
  4. On the local system, generate a symmetric AES key for encrypting data sets.
  5. Export the key, encrypt it using the public key, into a buffer.
  6. Write the buffer to a file
  7. Send the file to the remote system.
  8. Read the file, decrypt the buffer using the private key, re-encrypt with the local master key, store it in the local CKDS.
  9. Displayed the key at each end and compared the
  10. Use the symmetric key to decrypt a dataset.

The programs

I wrote some helper programs for example read a key from the CKDS returning the data; create an entry in the PKDS with a specified key and a blob of data containing the token.

Create a private public key

I could not find a batch utility to define a private key, but could use the ISPF panels.

I started writing my own program to do this…

I used API function CSNDPKB. I successfully created a Private/Public key using Elliptic Curve (ECC).

I could display this using the ISPF panels 5 UTILITY->6 PKDS KEYS → 1. This lists all of the keys. Use the line command K to display it. ( D is for delete!). It displays attributes like Algorithm: ECC, size 512, Sections: PRIVATE PUBLIC.

I later found out that ECC cannot be used to encrypt a symmetric AESkey. “Use the Symmetric Key Export callable service to transfer an application-supplied AES, DES or variable length symmetric key token key from encryption under a master key to encryption under an application supplied RSA public key or AES EXPORTER key. I could not find how to generate an EXPORTER key with public certificate.

I tried using CSNDPKB to create an RSA private key. You have to complete a “Key Value Structure”. I struggled to complete this for RSA because I needed to specify “prime number p”,and “prime number q”. Ive since found how to do this. I tried an alternate approach of using the ISPF panels.

On the remote system, I used the ISPF panels to create a PKI key. 5 UTILITY → 6 PKDS KEYS → 6 Generate PKA keys. I created a new RSA key with bit length 4096, as bit length 512 was too weak.

Having created my private/public key, I used the ISPF panels to export the public key to a data set, sent it to my local system and import it using the ISPF panels. This public key does not need to be sent securely.

  • You can extract the contents of this file using RACDCERT ADD(‘COLIN.EXPORT.ECC1’) ID(COLIN). This gave me a key with
Certificate ID: 2QXD1tPJ1dPBwsXT8PDw8PDw8PFA
Status: TRUST
Start Date: 2021/08/29 09:42:44
End Date: 2041/08/29 09:42:44
Serial Number:
>00< 
Issuer's Name: >CN=ECCB512Z< 
Subject's Name: >CN=ECCB512Z<
Signing Algorithm: sha256ECDSA
Key Type: Brainpool ECC
Key Size: 512
Private Key: NO
Ring Associations:

I had now had a private key at the remote end, and the public certificate at the local end.

I can encrypt at the local end using the public key, and decrypt it at the remote end using the private key.

Create the AES symmetric key on the local end.

I found I could not use AES type DATA, and so had to use AES type CIPHER.

If I used KGUP utility with

ADD TYPE(CIPHER ) ALGORITHM(AES) LENGTH(32) LAB(AESCIPHER)

This could not be used to encrypt because it gave IEC143I 213-85 RC=X’00000008′, RSN=X’0000085E’

85E (2142) The key usage attributes of the variable-length key token does not allow the requested operation. For example, the request might have been to encrypt data, but encryption is not allowed, or the request might have been to use the ECB cipher mode, but that mode is not allowed.

When I displayed through the ISPF panels (5.5.1) it it showed

Key Usage: ENCRYPT DECRYPT CBC

If I generated an AES key using the API, I found that ENCRYPT DECRYPT ANY-MODE worked, but I could not see how to set ANY-MODE using KGUP.

Program to generate an AES key

 // build the skeleton.   It is returned in pToken
 rc = skeletonAES(&pToken,& lToken); 
 if ( rc != 0 ) return rc; 
 
 // input: the skeleton 
 // output: the token 
 rc = GENAES2(pToken,&lToken); 
 if ( rc != 0 ) return rc; 

 // add it to the CKDS                                                           
 rc = doAdd2("C",pKey,pToken,lToken,pDelete); 
 if ( rc != 0 ) return rc; 
 return rc; 

skeletonAES program

// this program allocates storage, uses it, and passes it back
// to the caller. 
int skeletonAES(char ** pData, int * lData) 
 { 
  int rc; 
  int rs; 
  int zero = 0; 
  int rule_count                 =  4 ; 
  char8 rule[ 4] ={ 
   "INTERNAL","AES     ","CIPHER  ",    "ANY-MODE" 
   }; 
  char * pRule = & rule[0][0];
  int lKeyToken=725; 
  char * pKeyToken; 
  pKeyToken = (char *) malloc(lKeyToken); 
  CSNBKTB2(   
           &rc,         /* return code             */ 
           &rs,         /* reason code             */ 
           &zero,       /* exit data length        */ 
           0,           /* exit data[]             */ 
           &rule_count, /* rule array count        */ 
           pRule,       /* rule array[]            */ 
           &zero,       /* clear key bit length    */ 
           0,           /* key value[]             */ 
           &zero,       /* key name length         */ 
           0,           /* key name[64]            */ 
           &zero,       /* user assoc data length  */ 
           0,           /* user associated data[]  */ 
           &zero,       /* token data length       */ 
           0,           /* token data[]            */ 
           &zero,       /* service data length     */ 
           0,           /* service data[]          */ 
           &lKeyToken,  /* target key token length */ 
           pKeyToken ); /* target key token[]      */ 
 if ( rc > 0) 
 { 
    printf("CSNBKTB2 rc %i rs %i %s\n",rc,rs,csfgetrc(rs)); 
    return rc; 
 } 
 
   free (pData); // get rid of the passed in block 
// return the length
 *lData = lKeyToken;
//pass back the address of the token
 *pData = pKeyToken; 
 printf("skeletonAES key length %i\n",lKeyToken); 
// printHex(stdout,&keyToken,lKeyToken); 
   printAES((char *) pKeyToken,lKeyToken); 
   return 0; 
} 

Create program keygenerate

// this takes the data passed in, and uses it
// The length is updated.
int keyGenerate2( char * pData, int *  lData) 
{ 
  int rc; 
  int rs; 
  int zero = 0; 
  int rule_count                 =  2; 
  // key type AES and used for OP (on this system 
  char8 rule[2]  = {"AES     ","OP      "}; 
  char * pRule = &rule[0][0];
  int keyLength = 256 ;//  AES 256  - why use any other? 
                                                                            
  char8  keyType1 = {"TOKEN   "}; 
  char8  keyType2 = {"        "}; 
  int i64 = 64; 
  int l725 = 725; 

 CSNBKGN2( 
         &rc  ,        /* return code             */ 
         &rs  ,        /* reason code             */ 
         &zero,        /* exit data length        */ 
         0,            /* exit data[]             */ 
         &rule_count,  /* rule array count        */ 
         pRule,        /* rule array[]            */ 
         &keyLength,   /* clear key bit length    */ 
(char *) &keyType1   , /* key type1[8]            */ 
(char *) &keyType2   , /* key type2[8]            */ 
         &zero,        /* key name1 length        */ 
         0,            /* key name1[64]           */ 
         &zero,        /* key name2 length        */ 
         0,            /* key name2[64]           */ 
         &zero,        /* user assoc data1 length */ 
         0,            /* user associated data1[] */ 
         &zero,        /* user assoc data2 length */ 
         0,            /* user associated data2[] */ 
         &zero,        /* KEK id1 length          */ 
         0,            /* KEK id1[]               */ 
         &zero,        /* KEK id2 length          */ 
         0,            /* KEK id2[]               */ 
         &l725,        /* output key id1 length   */ 
 (char *)pData,        /* output key id1[]        */ 
         &zero    ,    /* output key id2 length   */ 
         0          ); /* output key id2[]        */ 
  if ( rc > 0) 
  { 
    printf("CSNBKGN2 rc %i rs %i %s\n",rc,rs,csfgetrc(rs)); 
    return rc; 
  } 
  *lData = l725; 
  // the same buffer is used,so no need to set pData
  return rc; 
} 

Using these and my add2(…) functions I could create my AES for CIPHER with ANY-MODE. (See ANY-MODE above).

Export the AES key

To export the AES key, you need the name of the public key.

You need a different set of rules for a DATA key and a CIPHER key.

 int   lData; 
 char * pData; 
 rc =doExportAES (pKey,pPublic,pType,&pData, &lData); 
 if (rc > 0 ) return rc; 
 rc = writeKey(dd,pData,lData); 
 if (rc > 0 ) return rc; 

I created a routine which read the key, and one to write it to a file.

The parameters to doExportAES are

  • dd name “dd:CERT would use //CERT .. in the JCL
  • key is a char64 left justified key name in the CKDS
  • pPublic a char 64 left justified name of the public key to use. It must exit in the PKDS.
  • A pointer to the string to containing the data
  • The length of the returned data.
int doExportAES (char * pKey, char * pPublic, char * pType, 
                 char ** pData, int * lData)  
{ 
  int rc; 
  int rs; 
  int i64 = 64; 
  int zero = 0; 
  int rule_count                 =  3; 
  char8 rule_Cipher[3] = 
                     {"AES     ", 
                      "PKOAEP2 ", 
                      "SHA-256 "}; 
  char * pRule =   rule_Cipher[0][0];
  int lOut = 900; 
  char * pOut; 
  pOut = (char * ) malloc(lOut); 
  printf("Source Key user   %64.64s.\n",key  ); 
  printf("Transport key     %64.64s.\n",encrypt); 

   CSNDSYX ( 
            &rc,                /* return code            */ 
            &rs ,               /* reason code            */ 
            &zero,              /* exit data length       */ 
            0    ,              /* exit data[]            */ 
            &rule_count,        /* rule array count       */ 
            pRule,              /* rule array[]           */ 
            &i64   ,            /* source key    length,  */ 
            pKey  ,             /* source key             */ 
            &i64,               /* RSA public key length  */ 
            pPublic,            /* RSA public key token   */ 
            &lOut,              /* exported key length    */ 
            pOut           );   /* exported key           */ 

if( rc > 0) 
  printf("CSNDSYX  rc %i rs %i %s\n",rc,rs,csfgetrc(rs)); 
if (rc != 0) return rc; 

// return the exported data

 *pData = pOut; 
* lData = lOut; 
   return 0; 
} 

Import the certificate at the remote end

I created IMPAES

  rc=  read(dd,&pData,&lData);         // returns a buffer  from the file
  printf("readCert rc %i data length %i\n",rc,lData ); 
  if ( rc != 0) return 8; 

  rc = doImportAES(pKey,pPrivateKey,&pData,&lData); 
  if ( rc != 0) return 8; 
  printf("Returned buffer size %i\n",lData); 
  rc = doAdd2("C",  pKey,pData,lData); 
  if ( rc != 0 ) return rc; 
  return 0; 

read opens the ddname , and gets back the (binary) data in pData of length lData.

This get passed to doImportAES which uses the private key pointed to by pPrivateKey (a 64 char left justified string). This then returns the AES key (encrypted with the local key) which is then added using do ADD2.

DoImportAES.

/* --------------------------------------------------------------- *
/* Import public key                                               *
/* --------------------------------------------------------------- *
int doImportAES (char * pKey, char * pDecryptKey, 
    char ** pData, int * lData) 
{ 
  int rc; 
  int rs; 
  int zero = 0; 
  int i64 = 64; 
  int rule_count                 =  2; 
  char8 rule_array[2]            ; 
  memcpy(&rule_array[0],"AES     ",8); 
  memcpy(&rule_array[1],"PKOAEP2 ",8); 
  int lOutput = 725 ; 
  char * pOutput = (char * ) malloc(lOutput); 
  if ( pOutput == 0) 
  { 
    printf("malloc for %i failed in IMPAES\n",lOutput); 
  } 
  printf("Key %64.64s.\n",pKey); 
  printf("Decrypt Key %64.64s.\n",pDecryptKey); 
  printf("lInput  %i\n",*lData     ); 
//printHex(stdout,* pData,32); 
  CSNDSYI2 ( 
              &rc,           /* return code            */ 
              &rs ,          /* reason code            */ 
              &zer           /* exit data length       */ 
              0    ,         /* exit data[]            */ 
              &rule_count,   /* rule array count       */ 
   (char *)   &rule_array[0],/* rule array[]           */ 
              lData     ,    /* length of input data   */ 
            * pData,         /* input data             */ 
              &i64,          /* cca DES token length,  */ 
   (char *)   pDecryptKey ,  /* cca DES token[64]      */ 
              &i64,          /* key name length        */ 
   (char *)   pKey,          /* key name[]             */ 
              &lOutput    ,  /* Imported key length    */ 
   (char *)   pOutput   );   /* Imported key           */ 
// if( rc > 0) 
   printf("CSNDSYI2 rc %i rs %i %s\n",rc,rs,csfgetrc(rs)); 
 if (rc != 0) return rc; 

 // get rid of input buffer 
   printf("CSNDSYI2 data lengt %i\n",lOutput); 
   free (*pData); 
// and update with the new data
   *pData =  pOutput; 
   *lData = lOutput; 
 return 0; 
 } 

All this creates a key in the CKDS on the remote system which I could use to decrypt a data set.

Summary

It took a long time to write these programs, because I did not know the path to take, and went down many dead ends. Once you know the concepts and know which ICSF functions and options you need, it is not too difficult.

Having these helper routines, I was able to create a program to generate a symmetric key using Diffi-Hellman in about 2 hours! (Most of this time was reading the documentation).

ICSF – background to writing my first ICSF application.

I was exploring data set encryption, and wanted to copy some data to an encrypted data set, and move to a new z/OS system (on a different USB). I used this as an exercise to learn about ICSF.

Overview

ICSF has lots of APIs to do wonderful things with encryption such as checking PIN numbers and encrypting packets of data. It also has some commands to define keys etc.

It was easy to set up data set encryption on one system. It was hard to set it up as a typical customer, where you had to set up two independent systems and send files between them.

The administration guide describes a scenario of two ICSF systems establishing initial transport keys. This was a bit clumsy. You have to edit a file, extract a string of hex digits and use a courier to send them to the remote system.

I set myself the challenge to use private and public keys to securely transfer the keys from one system to another. I could not find any commands to help me do this – so I had to write my own.

It took me much longer than expected, partly because I am not a good C programmer, but also because the path was not clear, and it took a while to understand what I needed to do. For example some APIs work with a key with type of DATA, and some do not. I ended up using a key type of CIPHER because that was the only one I could get to work. I’ve since got DATA to work.

Background ICSF information.

It took me a few days to understand the ICSF environments.

  • You have Symmetric keys – where a key is used for encryption, and the same key is used for decryption. These keys are stored in the Cryptographic Key Data Set (CKDS). These have two formats (and use different APIs) –
    • fixed length
    • variable length – this is a follow-on from fixed lengths and should be used instead of fixed lengths.
  • You have Asymmetric keys – for example private and public PKI keys. Everyone can have access to the public key. Only you have access to your private key. These are stored in the PKDS (Public Key Data Set). There are different flavours of PKI keys using different algorithm, and key sizes.
  • The ICSF API documentation refers to labels and tokens
    • Labels are a 64 character string used to identify the record in the +KDS.
    • Tokens represent the actual key, and is a structure (or blob of data). For example an AES token has information on the hash algorithm used, the encryption algorithm used, what sort of key it is (data, cipher, exporter). You can generate a token, use it, and not write it to the +KDS.
    • Sometimes a field in the API can be the label, or the token. If the length is 64 it is a label, otherwise it is a token.
  • Keys
    • Internal Keys in the +KDS are encrypted by a master level of encryption. If you you want to decrypt using a particular key. The (encrypted) key is extracted from the key store, and passed to the hardware encryption. The hardware decrypts the key, then uses it to decrypt the data. The unencrypted key does not leave the hardware.
    • External keys are keys that have not been encrypted, or are encrypted with a encrypted under a key-encrypting key (KEK) in the +KDS other than the master key. To use it you have to give the name/token of the KEK along with the name/token of external key.
  • You can say “extract the key, and encrypt it using another key before giving it to me”. You can then send the encrypted key to another system, which can decrypt the key and use it. The plain text of the key is never seen.
  • You can do things in the API which you are not authorised to do in the ISPF panels.
  • I was able to extract AES keys, encrypt them with Public/Private keys for transportation, and import them at the remote system. This is not considered good practice because the IBM cryptographic standards say you should keep keys used for encrypting data separate from those used to transmit keys. These are known as Key Transport Keys, and you have Exported and Importer keys. The documentation talks about Key Encryption Keys (KEKs) which is another name for the Key Transport Keys. You should have these for production, but I managed without them for my investigations. All these “keys” in a sentence made my head ache!

The hardest part

I found the hardest part about using ICSF was knowing which options to specify. For example you can use a public/private key to extract a key used for data set encryption, send it to the remote site, and decrypt it and add it to the system. How should you set it up? The answers are all in the documentation – somewhere, there is no getting started guide. The questions I had included:

What encryption technique and key strength should I use for data set encryption?

ICSF supports two techniques for symmetric encryption

  1. DES – Data Encryption Standard
  2. AES – Advanced Encryption Standard. This is often considered to be better than DES.

With each of these you can have different strength keys.

To encrypt data sets you need to use AES 256.

What sort of private/public key do I need

You can create a private/public pair of keys using

  1. RSA
    1. Modulus-exponent form (with a variety of different length modulus)
    2. Chinese Remainder Theorem(RCT) (with a variety of different length modulus)
  2. ECC (Elliptical curves)
    1. With a variety of curves: NIST, Prime, Brainpool
    2. And a variety of sizes
  3. CRYSTALS-Dilithium. Not having heard of this, I looked it up. It serves as a controlling agent in the faster than light warp drive in Star Trek, as well as being a technique for encryption.

Only RSA is supported to encrypt an AES key to send to a remote system using PKI.

What encryption technique should I use?

There are different ways of encrypting using an RSA key.

  1. PKCS–1.2 – RSA DSI PKCS #1 block type 02
  2. PKCSOAEP – RSA DSI PKCS #1V2 OAEP
  3. PKOAEP2 – RSA DSI PKCS #1 v2.1 RSAES-OAEP documentation (Not valid with DATA keys)

3. is better than 2. which is better than 1.

I use PKOAEP2 for CIPHER keys, and PKCSOAEP for DATA keys.

It would be very good if there was a documented decision tree giving the best advice on the options to use, along the lines of

  1. If you want to do data set encryption use AES256 and use CIPHER rather than DATA.
  2. If you want to encrypt a key to send to a remote system use RSA, with key size > … (eg 4096)
  3. …..

ICSF programming model

I had used the RACF callable services R_datalib (Certificate Data Library) to manage digital certificates. There is one module, and you pass parameters to define what to do, function_code = 0x01: get first, function code 0x0a: delete keyring. The parameters are similar.

With ICSF there are many functions, and the parameters are different. Although I was expecting a function “create public key and store it in the database with this name”, there may be times when you want to create a public key, use it and not store it in the database.

When you consider the options, using just one API does not work.

Low level functions

ICSF provides functions for each logical function. for example:

  1. create key (several flavours)
  2. store key in *KDS (several flavours)
  3. extract key (several flavours)
  4. encrypt data using this public key token (big blob of data), or use this label of a public key in the PKDS.

Rules

You pass configuration through to routines using “rules”. For example

typedef char char8[8];

int rule_count = 4 ;

char8 rule[ 4] ={ “AES “, “INTERNAL”, “CIPHER “, “ANY-MODE” };

These are like passing parameters through a command interface. It was easy to do – and worked well.

These parameters mean

  1. Create an AES key (the choice is AES or HMAC)
  2. Create an INTERNAL format token which will be used on this machine.
  3. Create a key with type of CIPHER. There is a choice of 8 options.
  4. ANY-MODE: Specifies that this key can be used for any encryption mode. The default is CBC:Specifies that this key can be used for cipher block chaining.

Using the rules was easy – finding which rules to use was hard (as I mentioned above).

Return and reason codes

Each function provides a return code, and a reason code. There is no service to call which returns a short string describing the error from the reason code. I wrote my own.

Exit programs

You can pass information through to an exit program – this feels like a very advanced topic. I haven’t used it.

Using the functions

The ICSF functions call stub services (which jump into the ICSF address space). These services do not have much function in them. You should not have to recompile your programs, or rebind them if you use a different level of ICSF.

C coding

With C routines you might code

printf(“the number is %\n”,i);

and pass “i” in.

With the stub interface you have to pass in the address of the variables for example

int zero = 0;

rc= CSNDPKB (&rc, &rs, &zero…)…

I found it easiest to put each ICSF function into its own function and call that, so hiding a lot of the ICSF API, and using common C techniques.

Rexx coding

It is pretty easy to write in Rexx. The hardest part is knowing which options to use.

The disadvantage of using Rexx, is that people could turn trace on, and display sensitive information.

Writing my own helper interface

I created routines like rc= skeletonAES(char ** pData, int * lData) {} to do the work, and return storage with the data in it.

Memory management: Within this routine I did a malloc() request for the storage. It passed the storage to ICSF, and returned the address back to the caller. I could have passed in a block of storage, but this means the higher level function needs to know length of the storage required. I think having the lower level function allocate the storage is better, because the routine knows the length it needs and makes it self contained.

int skeletonAES(char ** pData, int * lData) {

int lStorage = 725;
char *pStorage = malloc(lStorage);
CSNBKTB2(&rc, &rs,
…
     &zero,         /* service data length     */ 
     0,             /* service data[] ignored  */ 
     &lKeyToken,    /* target key token length */ 
     pKeyToken   ); /* target key token[]      */ 

*pData = pStorage;
* lData = lStorage;  // return the  true length 
                     //  of data returned.
return rc 
}

Once I had finished with the storage I could use free(…) to release the block.

What helper routines did I create?

I found it useful to create some helper routines.

Read the *KDS

rc=read(“C”,pKey, pData, lData );

rc= read(“P”,pKey, pData, lData );

to read the record and return a pointer to the data.
Internally

  • If “C” then use CSNBKRR2.
  • If “P” then use CSNDKRR.

I wrote some Rexx helper functions: readCKDS(label) and readPKDS(label), returning the return code, reason code, and the data.

doExists

rc=doExists(“C”,pKey );

rc= doExists(“P”,pKey );

to read the record and return if the record was not found.
If “C” then use CSNBKRR2.

If “P” then use CSNDKRR.

This is actually a wrapper around read, passing back the return code.

Add

rc = doAdd2(“C”,pKey,pReplace,Token ,lToken );

If “C” then use CSNBKRC2.

If “P” then use CSNDKRC.

If the add got the return code “already exists”, and pReplace = “yes” , then delete the record and retry the add.

In rexx I wrote addCKDS(label,data) and addPKDS(label,data).

PrintHex(stdout,pData,lData)

To print a dump like format of a block of storage.

PrintAES(pData,lData))

to print a summary of the AES token.

WriteKey

rc = writekey(dd,pData,lData)

This writes the data to a file specified by dd. Where dd is a string used in fopen, such as “dd:cert”.

The dataset was a variable length, with just one on record in it.

ReadKey

rc = readkey(dd,&pData,&lData)

This reads one record from the file specified by dd. Where dd is a string used in fopen, such as “dd:cert”. It allocates storage, and passes it back to the caller.

Other helpers

  • Export PKI public certificate
  • Import PKI public certificate
  • Create AES Skeleton for CIPHER (Variable format key)
  • Create AES Skeleton for DATA(Fixed format key)
  • Generate symmetric key using Diffi-Hellman
  • Export AES CIPHER key (Variable format key)
  • Export AES DATA key(Fixed format key)
  • Import AES CIPHER key (Variable format key)
  • Import AES DATA key(Fixed format key)

_____________________________________________________

Whoops, when I sent my encrypted data set, it wasn’t encrypted.

It is very easy to expose sensitive data when you send it off site, while thinking it is protected.

As an end user, the use of encrypted data sets is transparent. You do not know if a data set (or ZFS file system) is encrypted unless you explicitly check. It is easy just to copy some data and not realise you have gone from an encrypted data set to an unencrypted data set. It reminds me of someone who had to encrypt all of their data on CD’s, and left the decryption password inside the CD case.

I’ve been migrating from z/OS on one USB drive to a different z/OS level on a different USB drive. Rather than take the easy way, I thought I would try to do it properly, as if I had a typical customer’s environment.

I had managed to set up data set encryption so my userid COLIN got errors trying to access the data set: IEC143I 213-85 … RC=X’00000008′,RSN=X’00003E84′ and a RACF message saying my userid did not have access to they key – great encryption works.

I used my systems programmer userid and used XMIT to backup the dataset to a file, so I could FTP it off the z/OS image. Expecting it to still be encrypted. This worked fine.

As a test, I used receive indsn(….) and restored the “backup”. My userid COLIN was able to read the restored data set with no problems! Whoops – no encryption!

What happens under the covers, is my systems programmer userid was authorised to read the dataset, and so the XMIT command read it – and created the output dataset using the unencrypted data. Obvious once you think about it.

FTP will work the same way. You can use FTP over TLS – but at the remote end it will not be encrypted unless you have set up SMS access routines to encrypt on data set name.

TRANSMIT with Encipher didn’t work

The TRANSMIT(XMIT) command has an option ENCIPHER which invokes IDCAMS REPRO to copy the file and encrypt it.

xmit a.a encipher dsn(‘IBMUSER.ENC’) outdsn(‘IBMUSER.ENC.x’)

INMX100A Enter ENCIPHER options for AMS REPRO command +

The encipher options are described here. I could not get it to work. ICSF need to run in COMPAT(YES) in order to use the old PCF code – but changing to that did not help. It only supports keys with length 1-8 characters whereas ICSF supports key 64 character key names.

I could not see a way of automatically passing parameters for the ENCIPHER options, so I could not use this in a batch job to backup all my personal dataset. This technique looks about 25 years old.

ADRDSSU (DFDSS) did work

I used

//STEPT03 EXEC PGM=ADRDSSU,REGION=0M
//SYSPRINT DD SYSOUT=*
//TARGET DD DSN=COLIN.ADR.BACKUP,DISP=OLD,
// SPACE=(CYL,(1,1))
//SYSIN DD *
DUMP DATASET(INCLUDE(‘IBMUSER.ENC’)) –
OUTDDNAME(TARGET) –
ALLEXCP
/*

to produce a backup. I could restore from this backup using

//STEPT03 EXEC PGM=ADRDSSU,REGION=0M
//SYSPRINT DD SYSOUT=*
//DUMPED DD DSN=COLIN.ADR.BACKUP,DISP=SHR
//SYSIN DD *
RESTORE DATASET( –
INCLUDE(IBMUSER.ENC)) –
INDDNAME(DUMPED) –
RENAME(IBMUSER.ENC,IBMUSER.ENC3)
/*

which restored from the backup, and renamed IBMUSER.ENC, to IBMUSER.ENC3.

When my COLIN userid tried to access the restored dataset IBMUSER.ENC3 it got the same messages as it did with the original dataset. In this case the backup and restored recreated the encrypted data.

ADRDSSU (DFDSS) did work – bonus

When dumping the original data set you can specify the option RSA( RSA1PUB) where RSA1 is the public key of an RSA key. When you restore it you specify the private key RSA(RSA1PRIV).

These keys are in the ICSF CKDS.

This protects the backup data sets contents from being read.

But…

If you use omit the RSA() on the restore, it uses the same label as was used for the dump. This gave me

2B2C An incorrect PKA token was supplied. The service requires a private key token of the correct type.

because it tried to use my public key. On my remote system when the label pointed to the machines private key it worked.

What can you do?

I am not sure what you can do to prevent this.

  • The transmit command generates a temporary dataset. You could set up a profile for all temporary data sets, to be encrypted. Temporary data sets have a name like SYSyyddd.Thhmmss.RA00.jjobname.
  • You can restrict the use of the TSO TRANSMIT command, which will limit the opportunity for accidents to happen, but accidents may still occur.
  • You could write an exit for the TSO TRANSMIT command which takes the data set name, does a catalog lookup to determine to see if the data set is encrypted, and terminates if it is encrypted.
  • Rename the TRANSMIT and receive command. Create new TRANSMIT and RECEIVE commands which dump using DFDSS and then send the output file.

With FTP you can use the client EZAFCCMD exit, and the server FTCHKCMD exit which can do the same checks.

These exits are not trivial!

The lesson…

If you have been using XMIT or FTP to send your datasets off-site you will need to change. Your organisation may have enabled data set encryption without your knowledge (as it is transparent to authorised users).

It is very easy to accidentally expose sensitive data when you send it off site, while thinking it is protected.

Using a long TSO command from batch is easy

I wanted to run a rexx exec with a long string of parameters from JCL, and had problems because the line was too long. One evening, I couldn’t fix it, but when I came back next day I fixed it in seconds.

Using the PARM statement and symbol substitution

// SET REP='-REPLACE '
// SET KEY='-KEY REXXDH '
// SET PR='-PRIVATE ECCB512Z '
// SET PU='-PUBLIC ECCB512PUB '
//S1 EXEC PGM=IKJEFT01,REGION=0M,PARM='GAESDH &PR &PU &KEY &REP'
//SYSEXEC DD DISP=SHR,DSN=COLIN.ICSF.REXX
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD DUMMY
/*

This worked fine as long as the length of the PARM was under 100 characters long, after substitution. it looks for GAESDH in //SYSEXEC

Using SYSTSIN

The TSO command reference says

When it is necessary to continue to the next line, use a plus (+) or minus (-) sign as the last character of the line you want to continue.

That worked

//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSEXEC DD DISP=SHR,DSN=COLIN.ICSF.REXX 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
EXEC 'COLIN.ICSF.REXX(GAESDH)' '-KEY REXXDH -PRIVATE RSA1 - 
-PUBLIC RSA2PUB -REPLACE ' 
// 

Ending it with a comma does not work! (After struggling yesterday, it took 10 seconds this morning to find this was my problem).

Using SYSTSIN and JCL overrides

I tried using the JCL variables, and substitute them in the SYSTSIN file.

//E1 EXPORT SYMLIST=*
// SET REP='-REPLACE '
// SET KEY='-KEY REXXDH '
// SET PR='-PRIVATE ECCB512Z '
// SET PU='-PUBLIC ECCB512PUB '
//S1 EXEC PGM=IKJEFT01,REGION=0M PARM='GAESDH &PR &PU &KEY &REP'
//SYSEXEC DD DISP=SHR,DSN=COLIN.ICSF.REXX
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD ,SYMBOLS=JCLONLY 
EXEC 'COLIN.ICSF.REXX(GAESDH)' ' &KEY &PR &PU &REP' 
/*

This gave me

IEC020I 001-4,IBMREXDH,S1 ,SYSTSIN ,JES

Because the line after substitution was greater than 80 bytes long.

Putting it all together

//E1 EXPORT SYMLIST=*
// SET REP='-REPLACE '
// SET KEY='-KEY REXXDH '
// SET PR='-PRIVATE ECCB512Z '
// SET PU='-PUBLIC ECCB512PUB '
//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSEXEC DD DISP=SHR,DSN=COLIN.ICSF.REXX
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *,SYMBOLS=JCLONLY 
EXEC 'COLIN.ICSF.REXX(GAESDH)' '-KEY REXXDH - 
  &PR &PU - 
  &REP 
/*

This also worked. I find this a cleaner way of using JCL, for example I could have

//E1 EXPORT SYMLIST=*
// SET REP='-REPLACE '
// SET KEY='-KEY REXXDH '
// SET PR='-PRIVATE ECCB512Z '
//T1 SET PU='-PUBLIC TRY1 '
//T2  SET PU='-PUBLIC TRY2 '
// SET PU='-PUBLIC TRY3 '
// SET PU='-PUBLIC ECCB512PUB '
//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSEXEC DD DISP=SHR,DSN=COLIN.ICSF.REXX
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *,SYMBOLS=JCLONLY 
EXEC 'COLIN.ICSF.REXX(GAESDH)' '-KEY REXXDH - 
  &PR &PU - 
  &REP 
/*

The last definition of a variable wins. I could move line T1 down and run the JCL, then move line T2 down and run the JCL.

It is then easy to “pick and mix” your variables.

JCL with long PARM strings

Thanks to Morag who pointed me to the MQGEM blog post. I can also use PARMSDD. For example

//E1 EXPORT SYMLIST=*
// SET REP='-REPLACE '
// SET KEY='-KEY REXXDH '
// SET PR='-PRIVATE ECCB512Z '
// SET PU='-PUBLIC ECCB512PUB '
//S1 EXEC PGM=IKJEFT01,REGION=0M,PARMDD=PARMS
//PARMS DD *,SYMBOLS=JCLONLY 
  GAESDH -KEY REXXDH 
  &PR &PU 
  &REP 
//SYSEXEC DD DISP=SHR,DSN=COLIN.ICSF.REXX
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD DUMMY

This gives you a nice long parameter, with no quotes or continuation characters

ICSF return codes – not for humans

Using the ISPF ICSF functions I got

OUTPUT KEY DS NOT EMPTY (TO BE ABLE TO WRITE TO THE OUTPUT KEY DATA SET IT MUST BE EMPTY)

This was with the option 6, and trying to reinitialise the CSF.SCSFCKDS and CSF.SCSFPKDS.

The data sets had already been initialised. If you want to start with a new pair of data sets, you will need to delete and recreate them.

Below are some of the errors I experienced using ICSF

IKYC010I Error 791740530 returned from CP_NewKeysCreate: PKI Services can not generate certificates with secure keys.
IKYC010I Error 791740530 returned from JNH_create_certificate: PKI Services can not generate certificates with secure keys.

I had in pkiserv.conf

# Should the CA generate secure keys in the Token Data Set (TKDS) 
# when it has key generation capability?
# Valid SecureKey values are:
# T - True indicates secure keys are generated in the TKDS
# F - False (or absence of this keyword) indicates clear keys
# will be generated in the TKDS. Note: Installation
# configuration policy may override the ability to create
# clear keys causing clear key requests to create secure
# keys.
# If TokenName is not specified, the SecureKey keyword
# will be ignored.
SecureKey=T

I changed this to F and I could generate certificates.

When CSF was restarted I got the health check exception

CSFH0054I Check for clear keys in the CKDS, PKDS, and TKDS.                                                               

Active TKDS: COLIN.SCSFTKDS
---------------------------------------------------------
PKISRVD.PKITOKEN 00000001T
PKISRVD.PKITOKEN 00000002T
PKISRVD.PKITOKEN 00000003T
PKISRVD.PKITOKEN 00000004T

zPDT does not support PKCS #11 instructions, so you have to live with this.

INSUFFICIENT ACCESS AUTHORITY TO PERFORM THIS FUNCTION. CSF1TRC FAILED WITH
RETURN CODE: 00000008 REASON CODE: 00016000

I got this when trying to define a token PKISRVD.Z2

  • I had access to CSF1TRC
  • The reason was there was no RDEFINE CRYPTOZ SO.PKISRVD.Z2
  • I did not need access to it – the profile just had to exist
  • It needs the SO on the front.
  • To be able to use it, you also need USER.PKISRVD.Z2 and be given authority to it.

IEC143I 213-85 RC=X’00000008′,RSN=X’0000271C’

The return codes as documented in ICSF Application programmers guide – appendix A.

271C (10012)

Colin’s answer.

It could not find the key. Perhaps the CKDS was updated using the KGUP utility. Try refreshing the CKDS (either in batch or using the ISPF panels). For example

//REFRESHE EXEC PGM=CSFEUTIL,PARM=’CSF.SCSFCKDS,REFRESH’
//REFRESHF EXEC PGM=CSFPUTIL,PARM=’REFRESH,CSF.SCSFPKDS.NEW’

Perhaps you are trying to encrypt a data set with a non symmetric key – for example a PKI.

IEC143I 213-8,RC=X’00000008′,RSN=X’00000BF3′

The return codes as documented in ICSF Application programmers guide – appendix A.

BF3 (3059) The provided key_identifier refers to an encrypted variable-length CCA key token or a key label of an encrypted variable-length CCA key token. The key-management field in the CCA token does not allow its use in high performance encrypted key operations.


User action: Supply a key token or the label of a key token with the required key-management settings.

Colin’s comments

With CSNBKTB2 I got the 0xbf3 when ‘XPRTCPAC’ was missing. For example I needed rule_array = ‘INTERNAL’||’AES ‘||’CIPHER ‘||’ANY-MODE’||’XPRTCPAC’

The doc for AES CIPHER says XPRTCPAC Allow export to CPACF protected key format.

I also got this trying to use an EXPORTER or an IMPORTER key. This does not support XPRTCPAC.

Defining the cipher using ISPF worked. Using KGUP I needed

ADD TYPE(CIPHER) ALGORITHM(AES), 
KEYUSAGE(ANY-MODE) LENGTH(32),
KEYMGT(XPRTCPAC),
LABEL(COLINCIPHER2 )

I also got this BF3 when the key was archived.

IEC143I 213-85, RC=X’00000008′,RSN=X’00000BFB’

The provided symmetric key label refers to an encrypted CCA key token, and the CSFKEYS profile covering it does not allow its use in high performance encrypted key operations.

User action: Contact your ICSF or RACF administrator if you need to use this key in calls to Symmetric Key Encipher (CSNBSYE) or Symmetric Key Decipher (CSNBSYD). Otherwise, use Encipher (CSNBENC) or Decipher (CSNBDEC) instead.

Colin’s answer

Define the profile with the bold text

RDEFINE CSFKEYS DES5 UACC(NONE) –
ICSF(SYMCPACFWRAP(YES) SYMCPACFRET(YES))

IEC143I 213-85, RC=X’00000008′,RSN=X’0000272C’

The return codes as documented in ICSF Application programmers guide – appendix A.

272C (10028)

Colin’s comment

I got this when I tried to use

ADD TYPE(DECIPHER) ALGORITHM(DES) LABEL(DES5) CLEAR or
ADD TYPE(ENCIPHER) ALGORITHM(DES) LABEL(DES5) CLEAR

The following worked

ADD TYPE(CIPHER) ALGORITHM(DES) LABEL(DES5) CLEAR

as did

ADD TYPE(CIPHER) ALGORITHM(AES),
KEYUSAGE(ANY-MODE) LENGTH(32),
KEYMGT(XPRTCPAC),
LABEL(COLINCIPHER2 )

Colin’s answer.

I had defined the cipher as DECIPHER.

CSFM655I AN ARCHIVED RECORD COLINCIPHER IN THE ACTIVE CKDS WAS REFERENCED.

I got this when I used an archived key, and had KEYARCHMSG(YES) specified in my ICSF startup.

CSFG1094 TRANSKEY label TOO WEAK.

Colin’s comment

I was trying to use TRANSKEY but the length of the transkey is shorter than the key being defined, for example

ADD LABEL(ATOB) TYPE(EXPORTER) CLEAR LENGTH(16) ALGORITHM(DES)

ADD LABEL(KEY2) TYPE(DATA) LENGTH(24)TRANSKEY(ATOB) ALGORITHM(DES)

Instead of LENGTH(16) I had to use DOUBLEO.

IEC143I 213-86

During open processing for an encrypted extended format data set, on return from the ICSF service used to process the key label associated with the data set, the system detected that the encryption type of the data key associated with the key label was not of a supported encryption type. Only encryption keys of type AES256 are supported for extended format data sets.

Colin’s comment

Using

ADD TYPE(DATA) ALGORITHM(AES) LABEL(AES5) LENGTH(32)

Using ISPF panels 5 UTILITY, 5 CKDS KEYS, 7 Generate AES DATA keys, Enter the CKDS record label for the new AES DATA key.

With AES key bit length: 128 or 192, I got this message. Using 256 worked.

with length(32) works. With length(16) it gives 213-86

Update: Only encryption keys of type AES256 are supported for ANY data set encryption.

I also got this abend code when the data set was not extended format.

IEC143I 213-85

COLINAES CL(CSFKEYS ) INSUFFICIENT ACCESS AUTHORITY COLIN.ENCR.DSN,
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
RC=X’00000008′,RSN=X’00003E84′

Colin’s comments

The userid did not have access to the profile COLINAES of type CSFKEYS in ICSF. The data set was COLIN.ENCR.DSN.

RACF class XFACILIT CSF.XCSFKEY.ENABLE.AES was defined, which says do more detailed security checks, but the XCSFKEY profile was not found. RDEFINE XCSFKEY * UACC(NONE) WARNING

IEC143I 213-85, RC=X’00000008′,RSN=X’0000085E’

The key usage attributes of the variable-length key token does not allow the requested operation. For example, the request might have been to encrypt data, but encryption is not allowed, or the request might have been to use the ECB cipher mode, but that mode is not allowed.
User action: Use the variable-length key token in a manner consistent with its usage attributes or create a new key token with the desired attributes

Colin’s comments 1

I got this when I had

ADD TYPE(CIPHER ) ALGORITHM(AES) LENGTH(32) LAB(AESCI)

Changing it to type(DATA) worked.

Colin’s comments 2

  • I had a C program and used CSNBKTB2. When CBC was defaulted I got the 85E, when I used ANY-MODE it worked.
  • I had another program which used rule_array = ‘INTERNAL’||’AES ‘||’CIPHER ‘||’XPRTCPAC’||’ANY-MODE’||’ENCRYPT ‘. Without ENCYPT it worked. With both ENCRYPT and DECRYPT it worked.

Colin’s comment 3.

A type(cipher) with key used(any) worked; with key used(cbc) it didn’t work.

IEC143I 213-91

During open processing for an encrypted extended format sequential data set, the system determined that the data key in the CKDS associated with the key label for the data set is not the same data key used to encrypt the data set.

I created an encrypted dataset, then deleted and recreated the cipher key. This means the dataset could not be processed as the key was now lost.

Return codes

02f (47) A source key token is unusable because it contains data that is not valid or undefined.

I tried to extract the key of a skeleton key token, when there was none. CSNBKYT2 and the KEY-LEN option

3d (61) The keyword supplied with the key_type parameter is not valid.

I was using CSNBKGN2 and wrongly specified INPORTER instead of IMPORTER

048 ( 72 ) The value specified for length parameter for a key token, key, or text field is not valid.

Colin’s comments.

I got this in CSNDSYI2 when using a private key with a small key size(1024). When I used a private key with key size of 4096 it worked.

09B ( 155 ) The value that the generated_key_identifier parameter specifies is not valid,

or it is not consistent with the value that the key_form parameter specifies.

Colin’s comments

Case 1.

I was trying to generate an IMPORTER and an EXPORTER key. I used CSNBKTB2 to build a skeleton. When I used CSNBKGN2 to generate the token. I got this return code. I think this is because I did not provide a Transport Encryption Key (KEK)

When I used CSNDEDH passing the output from CSNBKTB2, the private key label, and the public key label, it worked, and I could add it to the CKDS using CSNBKRC2.

Case 2.

CSNBKGN2 only accepts skeletons created with type = CIPHER, HMAC, or MAC. See table 77. Trying to use a skeleton for EXPORTER or IMPORTER give you this message.

Case 3.

In CSNBKTB2 I had specified

‘INTERNAL’||’AES ‘||’CIPHER ‘||’XPRTCPAC’||’ANY-MODE’||’DECRYPT ‘

CSNBKGN2 gave me rs 155. Remove the DECRYPT and it worked

Case 4.
I was using CSNBKGN2 with pkeyType1 = “TOKEN “; I needed to change this to AES for it to work.

Case 5. I had generated a skeleton with TYPE=EXTERNAL. When I came to generate the key using this skeleton, it was mode=OP – or INTERNAL! I changed the skeleton and it worked.

If MODE=OP the skeleton must have INTERNAL, if MODE=EX the skeleton must have EXTERNAL.

The opposite may give the same return code – eg skeleton with type=internal, and generate key with non OP mode.

F6 ( 246 ) Not documented

I got

  • CSNDKRC  add pkds getting 0 246

I got this with

rc = 'FFFFFFFF'x
rs = 'FFFFFFFF'x
ADDRESS LINKPGM "CSNDKRC",
  'myrc' 'myrs' ,
 ...

because I had not initialised myrc and myrs.

7FB ( 2040 ) Bad data

This check is based on the first byte in the key identifier parameter. The key identifier provided is either an internal token, where an external or null token was required; or an external or null token, where an internal token was required. The token provided may be none of these, and, therefore, the parameter is not a key identifier at all.

Colin’s comment

  • Check you are passing in the right data! I had misspelt a variable.
  • I was trying to import a PKI public certificate – when it was an AES exported certificate
  • I was trying to use a PKI a private CCA key token of type ECC. CSNDSYI2 only accepts … key enciphered under an RSA public key or AES EXPORTER key.
  • You are trying to use CSNDSYI2 for a DATA key when you should be using CSNDSYI.
  • I was trying to add a key to the ckds, with the wrong format. It had been generated as exporter/importer/external token.

806 ( 2054 ) Invalid RSA enciphered key cryptogram; OAEP optional encoding parameters failed validation.

Colin’s comments

I got this when I used the wrong private key to decrypt a key in CSNDSYI2. When I used the correct key it worked.

829 ( 2089 ) The algorithm does not match the algorithm of the key identifier

The algorithm does not match the algorithm of the key identifier.
User action: Make sure the rule_array keywords specified are valid for the type of key specified. Refer to the rule_array parameter described in this publication under the appropriate callable service for the valid values.

Colin’s comment.

  • I got this because I had a private key created as an ECC. Where it was expecting an RSA key.
  • CSNDSYX trying to use an PKI public key with ECC…. to encryption under an application supplied RSA public key or AES EXPORTER key.

825 (2095) The value in the key_form parameter is incompatible with the value in the key_type parameter.

I was using CSNBKGN2. I had pkeyType1=”EXPORTER” . I had rule_array Key Form (required) as “EX “. I change pkeyType1 to CIPHER, and got past it.

86A ( 2154 ) Bad key type

At least one key token passed to this callable service does not have the required key type for the specified
function.

Colin’s comments.

  • I got this trying to use an Importer key instead of an Exporter key.
  • I got this trying to use a DH key when an RSA key was expected. The requirements were CSNDSYX: RSA public key or AES EXPORTER

86E ( 2158 ) Not in the books

I got this doing Diffie-Hellman key exchange CSNDEDH using a private key and a public key.

  • With private ECC Curve: PRIME Bits 521 and public ECC Curve: PRIME Bits 384 I got reason code 2158.
  • With private ECC Curve: PRIME Bits 521 and public ECC Curve: PRIME Bits 521 I got reason code 0.
  • With private ECC Curve: PRIME Bits 521 and public ECC Curve: BRAINPOOL Bits 521 I got reason code 2158.
  • With private ECC Curve: BRAINPOOL Bits 521 and public ECC Curve: BRAINPOOL Bits 521 I got reason code 0.

It looks like you have to have matching curve type, and matching size (in bits) for it to work. The documentation under ECC Diffie-Hellman (CSNDEDH and CSNFEDH) says

The ECC curve type and size must be the same as the type (Prime, Brainpool, or Koblitz) and size of
the ECC key-token specified by the public key identifier parameter.

BF9 (3065) Duplicate key

ICSF Duplicate key token policy checking is active. The caller is requesting to add a token to the key data set (CKDS or PKDS as appropriate) that already exists within the key data set. The request fails.

I got this when adding a key to the PKDS and the key name field within the record already exists in the data base. I tried to add key ATOB, with key name TEMP1 – but TEMP1 already existed.

DC9 ( 3529 ) Bad label

A key identifier was supplied to a callable service as a key token or the label of a key token in a key data set. Either the key type of the key or the algorithm of the key is unsupported by the cryptographic features available to ICSF.

Colin’s comment

Perhaps you specified a label name – when it did not exit.

PKA Key Generate (CSNDPKG):generated_key_token_length: The length of the generated key token or label for the generated key token.

I assumed you could give it a label, and it would store the data under that label.

2B30 ( 11056 ) The input PKA token contains length fields that are not valid.


User action: Re-create the key token.

Colin’s comment
2B30 (11056) The skeleton_key_identifier_length field is not valid.
User action:  Check  the skeleton_key_identifier_length and skeleton_key_identifier  (returned from CSNDPKB fields key_token_length,key_token)

Return code 8 reason 2AF8 ( 11000 ) The value specified for length parameter for a key token, key, or text field is not valid.


User action: Correct the appropriate length field parameter. For example I had target_key_identifier_length as 1000, but the documentation said The maximum value is 725 bytes.

Colin’s comment

Make sure you pass the address of the length eg &size, not the size itself.

Make sure you are adding to the correct database. If you try to add a PKI to a CKDS you will get this reason code.

I specified the length of a key – but for DES the length had to be zero.

Make fields bigger. I got this with

RSA_enciphered_key_length: The length of the RSA_enciphered_key parameter. This service updates this field with the actual length of the RSA_enciphered_key it generates. The maximum size is 512 bytes.

I had to make it 530 before it worked. Note when I came to check this at a later data – it all worked perfectly and I did not need to make it bigger!)

With CSNBKGN2 I have specified a key length of 64 when I had OP… with OP the length must b zero.

I specified a key length of zero when a value like 256 was required.

In create skeleton, I specified a key length, but did not specify KEY_CLR in the rules.

If key length = 0, when it should be a positive integer.

CSFPTRL: rc:8 rs:11000

No more data. When looping through you get rs 3017, buffer too small, meaning there is more data to come. rs = 0 means last record. rs = 11000 means past the end of the data.

271C ( 10012 ) A key label was supplied for a key identifier parameter.

This label is the label of a key in the in-storage CKDS or PKDS. A key record with that label (and the specific type if required by the ICSF callable service) could not be found. So it could be in the CKDS or PKDS, but with the wrong type.

For a retained key label, this error code is also returned if the key is not found in the CCA coprocessor specified in the PKDS record.

I also got this when using CSNDSYX and did not have rule_array with AESKW.

Colin’s comment.

  • I had specified a key of type data (which existed in the CKDS), but it was expecting a key of type Exporter, so was not found and could not find the label in the PKDS).
  • CSNDSYX trying to use an PKI public key with ECC…. to encryption under an application supplied RSA public key or AES EXPORTER key.
  • You specified a key, but the key was not char[64] and had garbage in the value. This can occur if you use a C null terminated string.
  • I was using CSNBKGN2 and used a rule I_rule[2] = {“AES “,”IN “} instead of IM.

2740 ( 10048 ) The key_type parameter does not contain one of the valid types for the service or the keyword TOKEN.

Colin’s comment

I was trying to use CSNDEDH which required a private key and a public key of type ECC. I had specified an RSA key.

3E80(16000) RACF failed your request to use this service or PKCS #11 token. This may be caused by the CSFSERV or CRYPTOZ class.

Colin’s comments.

Check the job log. This userid does not have read access to CSFOWH CLASS(CSFSERV )

IRRD117I Unexpected ICSF CSFPPKS return code x’00000008′ and reason code x’00002B00′. The request is not processed.

The doc says

The public or private key values are not valid (for example, the modulus or an exponent is zero or the exponent is even) or the key could not have created the signature (for example, the modulus value is less than the signature value). In any case, the key cannot be used to verify the signature.

I had

RACDCERT ID(COLIN) GENCERT -                                   
  SUBJECTSDN(CN('TRYPKI') - 
             O('NISTEC256') - 
             OU('SSS')) - 
   NISTECC - 
   KEYUSAGE(   HANDSHAKE     )  - 
   SIZE(256 ) - 
   WITHLABEL('TRYPKI') 

RACDCERT ID(COLIN)          - 
    GENREQ(LABEL('TRYPKI'))      - 
    DSN('COLIN.PKI.GENREQ') 

Changing size to 384 worked.

IRRD117I Unexpected ICSF CSFPTRC return code x’0000000C’ and reason code x’00000BCF’.

See BCF. The PKCS #11 TKDS is not available for processing

CSFC0116 CONTROL BLOCK VALIDATION ERROR. RETURN CODE = 8, REASON CODE = 36.

I got this when reinitialising csf. The data sets were not quoted, and used a data set name of ‘.CSF.CSFCKDS.NEW’ (starting with a period).

Action: Enclose the names in quotes.

CSFY0076 VSAM ERROR OCCURRED PROCESSING DD CSFCKDS. VSAM FEEDBACK CODE = 93080010

I think this is saying it cannot get the CKDS exclusive. I stopped CSF and the job ran.

CSFY0076 VSAM ERROR OCCURRED PROCESSING DD CSFCKDS. VSAM FEEDBACK CODE = A9080010

I think code A9080010 is record not found. It matches the message in CSFDIAG

CSFG0264 ENTRY TEMP IMPORTER NOT FOUND ON CKDS. ADD NOT PERFORMED

CSFKGUP return code 36

I ran a KGUP program, got return code 36 and no other output. csfg0002 says

36: State of special secure mode on the coprocessor is not the same as the state that is specified in the PARM field of the EXEC JCL statement.

CSFG0986 CKDS IS NOT USABLE.

I was using the wrong CKDS. I had two and was swapping between them.

CSFG0074 SYNTAX ERROR IN CONTROL STATEMENT

I had

ADD TYPE(CIPHER  ) ALGORITHM(AES), 
CLEAR KEY(9C96548DEC7D5057,4B95D3B09E8D75D6,BF6C497593E5E31D,
B13AC2AF14331483),
LAB(COLINCLEAR)

with the wrong number of digits in one of the key parts. I had 483>0< at the of the statement.

CSNDSYX rc 8 rs 16004 RACF

I was trying to use key AESCIPHER and exporting key KEYBTOA. I got CSNDSYX rc 8 rs 16004 RACF failed your request to use the key label or token.

Not authorised

For rc 16004 I got

AESCIPHER CL(CSFKEYS )
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

Give the userid access to the resource.

I needed this for the Key to be exported and the Key Exporting Key.

Missing XCSFKEY profile

After a GTF RACF trace I found I did not have a profile for class XCSFKEY profile AESCIPHER. I defined this, and the request worked.

In this page, it says Profiles in the XCSFKEY class are used in authorization checks only when the Symmetric Key Export service (CSNDSYX, CSNFSYX, or CSNDSXD) is called. See Increasing the level of authority required to export symmetric keys for additional information.

I needed this profile because I had (a long time ago) defined a class XFACILIT profile CSF.XCSFKEY.ENABLE.AES. See table 1.

Can you send a secret to Mars?

I’ve been reading a book on cryptography, and playing with encryption of data between two sites, and these got me thinking about the history of cryptography.

Hundreds of years ago cryptography was used to send state secrets. With a lot of data, experts could crack and read your correspondence. You could have a “one time pad” where you used a key just once, which helped. The biggest problem was key distribution. In theory you could send the keys (or the pad of keys) through the post, but if people are intercepting and reading your mail – they get to see the keys!

With computers and mathematicians, sending keys to someone has got easier. It is harder to break, but not impossible.

I saw a document recently which said send your key on paper, in an envelope, by courier and not over the network. If people can read documents wrapped in jars from 2000 years ago in a CAT scanner – reading through an envelope should be easy. Also, there may be no rocket going to Mars to take your letter.

There are two common techniques used today for two ends to get a common secret: RSA and Diffie-Hellman

RSA

With RSA you have a private key (which only you has access to) and a public key – which every one can have access to. If you want to send me some data then you encrypt it with my public key and send the data to me. I use my private key and can then decrypt it.

Is this good enough ?

If you can afford to send someone to Mars, you can afford to have a team of programmers to create every possible private-public key combination and have a look up – if the public key is x then the private key is X. If you advertise a public key then some people can decrypt your message. If we do not advertise which private/public key to use then it makes it harder. But how do we negotiate in secret which private/public key to use?

Diffie-Hellman

From a mathematical perspective I found this is much more interesting. We want to communicate to get a secret key, knowing that people can monitor the network and see what we are sending.

It relies on some number theory properties

  • (x **a ) **b = (x**b) **a for example (a**2) **3 = (a* a*) * (a* a) * (a*a) = (a * a * a) * ( a * a * a) = (a**3) **2
  • modular arithmetic x mod y is the remainder if you divide x by y. 17 mod 5 is 2
  • combining these (x ** a) mod y = (x mod y ) **a mod y. x**a could have hundreds or millions of digits. “x mod y” is less than y, so may fit in 64 bit integers.
  • We agree two numbers x and y, which the spy can snoop. I think of a big number a, you think of a big number b.
  • I calculate (x **a) mod y = X and send it to you. You take X and calculate X ** b mod y.
  • You calculate x **b mod y = Y and send it to me. I take Y and calculate Y**a mod y
  • The two numbers are the same, and we can use it as our secret key. The secret agent who does not know a or b cannot calculate the secret.

The secret agents listening on the connection do not know which values of a and b we used, and there could be an infinite number of ‘a’s which all give the same value of (x **a) mod y. Typically people use a and b with thousands of digits.

If you have an x with thousands of digits, and an a with thousands of digits x **a will have millions of digits so you have to have special routines to do the calculations. Fortunately the mod y calculation reduces the size down to a more manageable number – with only thousands of digits.

Who said mathematics was boring?

Is this good enough ?

If you use small numbers then it is easy to crack. Today’s thinking is using more than 2048 bits will be hard to crack.

Backwards migration problem with load modules? – try using DLLs

I had a problem where I compiled a program on z/OS 2.4. When it ran on z/OS 2.3 it abended. This was because I had included some z/OS code with my load module, which was not compatible with back level versions of z/OS.
I didn’t have a z/OS 2.3 to run on …. what could I do? A quick search showed me that using DLL’s was the right answer. You may be using this today without knowing it, for example your C program does not contain the whole printf executable, it just contains a link to it.

What did I do?

My old JCL had

//COMPILE EXEC PROC=EDCCB,….
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB.
//BIND.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEELKED
//BIND.OBJLIB DD DISP=SHR,DSN=COLIN.OBJLIB
//BIND.GSK DD DISP=SHR,DSN=SYS1.SIEALNKE
//BIND.CSS DD DISP=SHR,DSN=SYS1.CSSLIB
//BIND.SYSIN DD *
INCLUDE GSK(GSKCMS31)
INCLUDE GSK(GSKSSL)
INCLUDE CSS(IRRSDL00)
NAME AMSCHECK(R)

Where the INCLUDE GSK(..) copied in the (z/OS 2.4 specific ) code needed to execute.

My new JCL had

//BIND.SYSIN DD *
SETOPT PARM(CASE=MIXED)
include /usr/lpp/gskssl/lib/GSKSSL.x
include /usr/lpp/gskssl/lib/GSKCMS31.x

INCLUDE CSS(IRRSDL00)
NAME AMSCHECK(R)

How does it work?

The GSKSSL.x has

include /usr/lpp/gskssl/lib/GSKSSL.x
IMPORT CODE,’GSKSSL’,’gsk_attribute_get_buffer’
IMPORT CODE,’GSKSSL’,’gsk_attribute_get_cert_info’

This says gsk_attribute_get_buffer is in module GSKSSL, use dynamic resolve at execution time.

When the program is executed, the loader looks for entries which have pending dynamic resolve. In this case it sees gsk_attribute_get_buffer, It loads the module GSKSSL from the executing system, looks inside it, locates the gsk_attribute_get_buffer code and resolves the address.

On my z/OS 2.4 system it loaded the 2.4 version of GSKSSL, on the z/OS 2.3 system it loaded the 2.3 version of GSKSSL. As long as the parameters to the calls are consistent it will work.

Does this work for my own code?

Yes. You have to build it a certain way. Look for side-deck in the C user guide and C Programming guide.

Are there any other benefits?

Yes – your load modules are smaller, because the imported code is obtained at run time, and not included in your load module. Also, as Charles M. pointed out, another benefit is that you are no longer violating IBM’s license by shipping z/OS licensed code as part of your product.

What can I ship?

Charles Mills pointed me to this documented list of what you can do. See here under Redistribution of Code Routines. Which starts

This document includes information about certain callable service stub and linkage-assist (stub) routines and other routines that are contained in specific data sets and file system directories that are intended to be bound, link-edited, or otherwise included with code and run on z/OS systems. With your authorized use of z/OS, you can include these routines into your modules and distribute your
modules with the included routines for the purposes of developing, using, marketing, and distributing programs that conform to the documented programming interfaces for z/OS, provided that each routine is included in its entirety, including any IBM copyright statements.

It then goes on and lists which libraries you can use.

One Minute MVS – ICSF. It might be better if they finished it.

This is another post in the series of “One Minute MVS” which aims to give the basics you need to be able to get started with a topic.

The IBM documentation says: ICSF provides support for

  • The ANSI Data Encryption Algorithm (DES) and Advanced Encryption Standard (AES) encryption and decryption
  • DES key management and transport
  • AES key management and transport
  • Financial services including PINs, payment card industry transactions and ATMs
  • Public key operations including key generation, digital signatures and wrapping symmetric keys for transport
  • MAC and hash generation
  • Acceleration of handshake and frame encryption for SSL
  • PKCS #11 API

which has too many buzz words for me.

The manual Getting Started with z/OS Data Set Encryption is a very useful book.

My interpretation of what ICSF is:

ICSF…

  • Can create and store Public and Private certificates (as used in SSL and TLS). RACF can store its certificates in ICSF.
  • Can store symmetric keys used to encrypt data – such as data sets. Note: if you are using TLS the actual encryption of data over the network is done with a symmetric key.
  • Mange the hardware, tamper proof, keystores provided with z hardware. (You have to access the physical z machine to enter the master cryptographic keys).
  • To check credit card PIN number and other checks.
  • You can configure ICSF in a group of datasets, and switch to a different set.

ICSF facilities

ICSF has

  • A callable services API to allow you to call ICSF services from a program. There is a header file /usr/include/csfbext.h and SYS1.SIEAHDR.H which defines the function parameters.
  • Some ISPF panels to help you mange the ICSF entities.
  • Some batch command interfaces.

ICSF is not very usable

I found that ICSF was not very usable. For example

  • The ISPF panels (see here) are not intuitive. You can update the ICSF datasets in batch. You then have to refresh the in-memory copy.
    • For example, to refresh the CKDS data set and list it, using the ISPF panels for ICSF
      • 2 KDS MANAGEMENT -> 1 CKDS MANAGEMENT -> 1 CKDS OPERATIONS -> 2 REFRESH – Activate an updated CKDS.
      • Then PF3, 3 times.
    • To list the contents
      • 5 UTILITY -> 5 CKDS KEYS -> 1 List and manage all records
    • I would have had a page for KDS, and had a refresh option, and a list option on the same page.
  • I expected to be able to use ICSF using commands. It looks like I have to write programs to use the API! Some REXX programs are available for a subset of the function. See dataset CSF.SCSFCLI0 .
  • There is a lack of consistency. Two utilities doing similar things one has PARM=’REFESH,name’ the other has PARM=’name,REFRESH’. The utility for PKDS is CSFPUTIL. The utility for CKDS is CSFEUTIL.

It feels like ICSF has not yet been finished, more of an “Here are some API’s – good luck as you are on your own”, than a guide for the new user.

I’m writing some C programs to do some basic definitions, and pass parameters. I should not need to do this.

ICSF concepts

There are asymmetric keys such as private key and its public key. If you encrypt some text with a public key you need the private key to decrypt it. If you encrypt with the private key – you need the public key to decrypt it.

There are symmetric keys where the same key is used at each end. For example encrypt: change A to s, B to !; decrypt change s to A, ! to B.

Asymmetric keys are usually used in negotiating or sending a symmetric key to the remote end.

Symmetric keys are usually used to encrypt the payload. It is good practice to change the symmetric key periodically to make it harder for someone to break the cipher.

ICSF has

  • PKDS (Public Key Data Set) for storing Private and Public Asymmetric keys.
  • CKDS (Cryptographic Key Data Set) which is used to store Symmetric keys
  • TKDS (Token Key Data Set). When you are using keys stored in the hardware cryptographic facility, you have a token to reference the data.

The data in these key data sets, may be encrypted, for example by the hardware cryptographic facility. You can configure ICSF so your keys are encrypted, and in normal operation they are not available in clear text, as they are encrypted by the tamper proof hardware, and are used within the hardware, where they are decrypted and used.

Keys can be in one of several state

  • Active. The key can be used to process data. It is within its start and end dates (if present)
  • Archive. The key cannot be used to process data. For example you have backed up an encrypted data set to tape. If you delete the key, the data cannot be processed. If they key is archived then it cannot be used. If you need to access the backed up dataset, you can change the status to Active for the duration of the work.
  • Inactive – A key which has not been archived, and is outside of the start and end dates (if present).
  • Pre-active – I cannot find what this is. It is mentioned in the ISPF panels.

Configuration

You can configure which ICSF data sets are current using, and other parameters via SYS1.PARMLIB(CSFPRMxx).

When you start the CSF procedure you specify the xx. For example in SYS1.PROCLIB concatenation member CSF,

//CSF PROC PRM=00
//CSF EXEC PGM=CSFINIT,PARM=&PRM,…..

S CSF,PRM=CP

You can use the SETICSF operator command to change some parameters for the duration of the CSF task.

You can use D ICSF, for example d ICSF,kds to give

CKDS CSF.SCSFCKDS FORMAT=KDSR SYSPLEX=N MKVPs=DES AES
PKDS CSF.SCSFPKDS FORMAT=VARIABLE SYSPLEX=N MKVPs=RSA ECC
No TKDS was provided.

To change use a different CSFPRMxx, you have to stop and restart CSF, specifying the CSFPRM suffix.

You need to plan your ICSF usage

You can set up data set encryption so when you create a data set, the data is automatically encrypted. You define a key and give it a label. If you delete the key, the data cannot be read. To be able to process the dataset you need RACF access to the dataset, and RACF access to the key.

The label is associated with the data set (so it you send the dataset to a remote system, it will still have the same label).

Most organisations say you much change your password periodically, typically every month. For similar reasons organisations say you should define a new key and use it, typically every month. This is called Key Rotation, where you roll over your key to a new value.

I could have a key with label ColinDSKey, and encrypt my data sets with it. I encrypt data set COLIN.AUG2021.LOG with this key. Next month if I create a new key and reuse the same label, I can encrypt the data set COLIN.SEPT2021.LOG and use the new key. However I will be unable to read COLIN.AUG2021.LOG, because ColinDSKey now has a new value.

I’ve seen presentations which say ” just re-encrypt all your datasets with the new key”. This sounds like a lot of work and a major disruption.

Another approach is to change the key label, for example ColinAug2021DSKey. When I generate a new key, it has a label ColinSept2021DSKey. I configure datasets to use the new label. Datasets with the old label can still be used, as long as the key exists. You can tell ICSF to archive the label (and key), so it cannot be used. If it is needed you make the key active, use the data set, and re archive the key.

To set the label for a data set, you can

  • Specify it in JCL. So you may have to change your JCL every month, or use a symbolic such as the month and year.

// SET KEY=’OUTAESKEY’
//S4 EXEC PGM=IKJEFT01,REGION=0M
//SYSTSPRT DD DSN=IBMUSER.ENC,DISP=(MOD,CATLG),SPACE=(CYL,(1,1)),
// DSKEYLBL=&KEY,DSNTYPE=EXTREQ,DCB=(LRECL=132,BLKSIZE=3200)
//SYSTSIN DD *
LU *
/*

Use SMS and have rules to generate the label depending on your profile and the name of the data set.

Define a RACF profile for example ADDSD ‘PROTECT.*’ UACC(NONE) DFP(DATAKEY(AES2)) says for all datasets with a HLQ of PROTECT, then use key AES2.

When a dataset is allocated you get a message

IGD17150I DATA SET PROTECT.ENC IS ELIGIBLE FOR ACCESS METHOD ENCRYPTION. KEY LABEL IS (AES5)

You also need to think about encrypting your databases, which is another jump in complexity. As an old text book said “We’ll leave this as an exercise for the reader”.

To make the planning just a little! more complex; If you send an encrypted data set to another z/OS system, it will have the same label as when it was originally created, so you need the keys sent (securely) to the remote system for the data set to be processed, and coordinate naming conventions.

As the section header says You need to plan your ICSF usage. ICSF and data set encryption needs a lot of planning.

Rexx program to invoke ICSF panels

I used the following to invoke the ISPF panels for ICSF. I call the exec CSF.

To invoke it, from an ISPF screen type TSO CSF.

/* Rexx */                                                
/* IBMs ICSF */                                           
 address ispexec                                          
"LIBDEF ISPPLIB DATASET ID('CSF.SCSFPNL0') STACK"         
"LIBDEF ISPMLIB DATASET ID('CSF.SCSFMSG0') STACK"         
"LIBDEF ISPSLIB DATASET ID('CSF.SCSFSKL0') STACK"         
"LIBDEF ISPTLIB DATASET ID('CSF.SCSFTLIB') STACK"         
address tso "ALTLIB ACTIVATE APPLICATION(CLIST)           
            DATASET('CSF.SCSFCLI0')"                      
                                                          
"SELECT PANEL(CSF@PRIM) NEWAPPL(CSF) PASSLIB"             
                                                          
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)"        
"LIBDEF ISPSLIB"                                          
"LIBDEF ISPPLIB"                                          
"LIBDEF ISPMLIB"                                          
"LIBDEF ISPTLIB"                                          

One Minute MVS – tuning stack and heap pools

These days many applications use a stack and heap to manage storage used by an application. For C and Cobol programs on z/OS these use the C run time facilities. As Java uses the C run time facilities, it also uses the stack and heap.

If the stack and heap are not configured appropriately it can lead to an increase in CPU. With the introduction of 64 bit storage, tuning the heap pools and stack is no longer critical. You used to have to carefully manage the stack and heap pool sizes so you didn’t run out of storage.

The 5 second information on what to check, is the number of segments freed for the stack and heap should be zero. If the value is large then a lot of CPU is being used to manage the storage.

The topics are

Kinder garden background to stack.

When a C (main) program starts, it needs storage for the variables uses in the program. For example

int i;
for (ii=0;ii<3:ii++)
{}

char * p = malloc(1024);

The variables ii and p are variables within the function, and will be on the functions stack. p is a pointer.

The block of storage from the malloc(1024) will be obtained from the heap, and its address stored in p.

When the main program calls a function the function needs storage for the variables it uses. This can be done in several ways

  1. Each function uses a z/OS GETMAIN request on entry, to allocate storage, and a z/OS FREEMAIN request on exit. These storage requests are expensive.
  2. The main program has a block of storage which functions can use. For the main program uses bytes 0 to 1500 of this block, and the first function needs 500 bytes, so uses bytes 1501 to 2000. If this function calls another function, the lower level function uses storage from 2001 on wards. This is what usually happens, it is very efficient, and is known as a “stack”.

Intermediate level for stack

It starts to get interesting when initial block of storage allocated in the main program is not big enough.

There are several approaches to take when this occurs

  1. Each function does a storage GETMAIN on entry, and FREEMAIN on exit. This is expensive.
  2. Allocate another big block of storage, so successive functions now use this block, just like in the kinder garden case. When functions return to the one that caused a new block to be allocated,
    1. this new block is freed. This is not as expensive as the previous case.
    2. this block is retained, and stored for future requests. This is the cheapest case. However a large block has been allocated, and may never be used again.

How big a block should it allocate?

When using a stack, the size of the block to allocate is the larger of the user specified size, and the size required for the function. If the specified secondary size was 16KB, and a function needs 20KB of storage, then it will allocate at least 20KB.

How do I get the statistics?

For your C programs you can specify options in the #PRAGMA statement or, the easier way, is to specify it through JCL. You specify C run time options through //CEEOPTS … For example

//CEEOPTS DD *
STACK(2K,12K,ANYWHERE,FREE,2K,2K)
RPTSTG(ON)

Where

  • STACK(…) is the size of the stack
  • RPTSTG(ON) says collect and display statistics.

There is a small overhead in collecting the data.

The output is like:

STACK statistics:                                                
  Initial size:                                2048     
  Increment size:                             12288     
  Maximum used by all concurrent threads:  16218808     
  Largest used by any thread:              16218808     
  Number of segments allocated:                2004     
  Number of segments freed:                    2002     

Interpreting the stack statistics

From the above data

  • This shows the initial stack size was 2KB and an increment of 12KB.
  • The stack was extended 2004 times.
  • Because the statement had STACK(2K,12K,ANYWHERE,FREE,2K,2K), when the secondary extension became free it was FREEMAINed back to z/OS.

When KEEP was used instead of FREE, the storage was not returned back to z/OS.

The statistics looked like

STACK statistics:                                                
  Initial size:                               2048     
  Increment size:                            12288     
  Maximum used by all concurrent thread:  16218808     
  Largest used by any thread:             16218808     
  Number of segments allocated:               1003     
  Number of segments freed:                      0     

What to check for and what to set

For most systems, the key setting is KEEP, so that freed blocks are not released. You can see this a) from the definition b) Number of segments freed is 0.

If a request to allocate a new segment fails, then the C run time can try releasing segments that are not in use. If this happens the “”segments freed” will be incremented.

Check that the “segments freed” is zero, and if not, investigate why not.

When a program is running for a long time, a small number of “segments allocated” is not a problem.

Make the initial size larger, closer to the “Largest used of any thread” may improve the storage utilisation. With smaller segments there is likely to be unused space, which was too small for a functions request, causing the next segment to be used. So a better definition would be

STACK(16M,12K,ANYWHERE,KEEP,2K,2K)

Which gave

STACK statistics:                                                          
  Initial size:                                     16777216               
  Increment size:                                      12288               
  Maximum used by all concurrent threads:           16193752               
  Largest used by any thread:                       16193752               
  Number of segments allocated:                            1               
  Number of segments freed:                                0               

Which shows that just one segment was allocated.


Kinder garden background to heap

When there is a malloc() request in C, or a new … in Java, the storage may exist outside of the function. The storage is obtained from the heap.

The heap has blocks of storage which can be reused. The blocks may all be of the same size, or or different sizes. It uses CPU time to scan free blocks looking for the best one to reuse. With more blocks it can use increasing amounts of CPU.

There are heap pools which avoids the costs of searching for the “right” block. It uses a pools of blocks. For example:

  1. there is a heap pool with 1KB fixed size blocks
  2. there is another heap pool with 16KB blocks
  3. there is another heap pool with 256 KB blocks.

If there is a malloc request for 600 bytes, a block will be taken from the 1KB heap pool.

If there is a malloc request for 32KB, a block would be used from the 256KB pool.

If there is a malloc request for 512KB, it will issue a GETMAIN request.

Intermediate level for heap

If there is a request for a block of heap storage, and there is no free storage, a large segment of storage can be obtained, and divided up into blocks for the stack. If the heap has 1KB blocks, and a request for another block fails, it may issue a GETMAIN request for 100 * 1KB and then add 100 blocks of 1KB to the heap. As storage is freed, the blocks are added to the free list in the heap pool.

There is the same logic as for the stack, about returning storage.

  1. If KEEP is specified, then any storage that is released, stays in the thread pool. This is the cheapest solution.
  2. If FREE is specified, then when all the blocks in an additional segment have been freed, then free the segment back to the z/OS. This is more expensive than KEEP, as you may get frequent GETMAIN and FREEMAIN requests.

How many heap pools do I need and of what size blocks?

There is usually a range of block sizes used in a heap. The C run time supports up to 12 cell sizes. Using a Liberty Web server, there was a range of storage requests, from under 8 bytes to 64KB.

With most requests there will frequently be space wasted. If you want a block which is 16 bytes long, but the pool with the smallest block size is 1KB – most of the storage is wasted.
The C run time gives you suggestions on the configuration of the heap pools, the initial size of the pool and the size of the blocks in the pool.

Defining a heap pool

How to define a heap pool is defined here.

You specify the size of overall size of storage in the heap using the HEAP statement. For example for a 16MB total heap size.

HEAP(16M,32768,ANYWHERE,FREE,8192,4096)

You then specify the pool sizes


HEAPPOOL(ON,32,1,64,2,128,4,256,1,1024,7,4096,1,0)

The figures in bold are the size of the blocks in the pool.

  • 32,1 says maximum size of blocks in the pool is 32 bytes, allocate 1% of the heap size to this pool
  • 64,2 says maximum size of blocks in the pool is 64 bytes, allocate 2% of the heap size to this pool
  • 128,4 says maximum size of blocks in the pool is 128 bytes, allocate 4% of the heap size to this pool
  • 256,1 says maximum size of blocks in the pool is 256 bytes, allocate 1% of the heap size to this pool
  • 1024,7 says maximum size of blocks in the pool is 1024 bytes, allocate 7% of the heap size to this pool
  • 4096,1 says maximum size of blocks in the pool is 4096 bytes, allocate 1% of the heap size to this pool
  • 0 says end of definition.

Note, the percentages do not have to add up to 100%.

For example, with the CEEOPTS

HEAP(16M,32768,ANYWHERE,FREE,8192,4096)
HEAPPOOLS(ON,32,50,64,1,128,1,256,1,1024,7,4096,1,0)

After running my application, the data in //SYSOUT is


HEAPPOOLS Summary:                                                         
  Specified Element   Extent   Cells Per  Extents    Maximum      Cells In 
  Cell Size Size      Percent  Extent     Allocated  Cells Used   Use      
  ------------------------------------------------------------------------ 
       32        40    50      209715           0           0           0 
       64        72      1        2330           1        1002           2 
      128       136      1        1233           0           0           0 
      256       264      1         635           0           0           0 
     1024      1032      7        1137           1           2           0 
     4096      4104      1          40           1           1           1 
  ------------------------------------------------------------------------ 

For the cell size of 32, 50% of the pool was allocated to it,

Each block has a header, and the total size of the 32 byte block is 40 bytes. The number of 40 bytes units in 50% of 16 MB is 8MB/40 = 209715, so these figures match up.

(Note with 64 bit heap pools, you just specify the absolute number you want – not a percentage of anything).

Within the program there was a loop doing malloc(50). This uses cell pool with size 64 bytes. 1002 blocks(cells) were used.

The output also has

Suggested Percentages for current Cell Sizes:
HEAPP(ON,32,1,64,1,128,1,256,1,1024,1,4096,1,0)


Suggested Cell Sizes:
HEAPP(ON,56,,280,,848,,2080,,4096,,0)

I found this confusing and not well documented. It is another of the topics that once you understand it it make sense.

Suggested Percentages for current Cell Sizes

The first “suggested… ” values are the suggestions for the size of the pools if you do not change the size of the cells.

I had specified 50% for the 32 byte cell pool. As this cell pool was not used ( 0 allocated cells) then it suggests making this as 1%, so the suggestion is HEAPP(ON,32,1

You could cut and paste this into you //CEEOPTS statement.

Suggested Cell Sizes

The C run times has a profile of all the sizes of blocks used, and has suggested some better cell sizes. For example as I had no requests for storage less than 32 bytes, making it bigger makes sense. For optimum storage usage, it suggests of using sizes of 56, 280,848,2080,4096 bytes.

Note it does not give suggested number of blocks. I think this is poor design. Because it knows the profile it could have an attempt at specifying the numbers.

If you want to try this definition, you need to add some values such as

HEAPP(ON,56,1,280,1,848,1,2080,1,4096,1,0)

Then rerun your program, and see what percentage figures it recommends, update the figures, and test again. Not the easiest way of working.

What to check for and what to set

There can be two heap pools. One for 64 bit storage ( HEAPPOOL64) the other for 31 bit storage (HEAPPOOL).

The default configuration should be “KEEP”, so any storage obtained is kept and not freed. This saves the cost of expensive GETMAINS and FREEMAINs.

If the address space is constrained for storage, the C run time can go round each heap pool and free up segments which are in use.

The value “Number of segments freed” for each heap should be 0. If not, find out why (has the pool been specified incorrectly, or was there a storage shortage).

You can specify how big each pool is

  • for HEAPPOOL the HEAP size, and the percentage to be allocated to each pool – so two numbers to change
  • for HEAPPOOL64 you specify the size of each pool directly.

The sizes you specify are not that sensitive, as the pools will grow to meet the demand. Allocating one large block is cheaper that allocating 50 smaller blocks – but for a server, this different can be ignored.

With a 4MB heap specified

HEAP(4M,32768,ANYWHERE,FREE,8192,4096)
HEAPP(ON,56,1,280,1,848,1,2080,1,4096,1,0)

the heap report was

 HEAPPOOLS Summary: 
   Specified Element   Extent   Cells Per  Extents    Maximum      Cells In 
   Cell Size Size      Percent  Extent     Allocated  Cells Used   Use 
   ------------------------------------------------------------------------ 
        56        64      1         655           2        1002           2 
       280       288      1         145           1           1           0 
       848       856      1          48           1           1           0 
      2080      2088      1          20           1           1           1 
      4096      4104      1          10           0           0           0 
   ------------------------------------------------------------------------ 
   Suggested Percentages for current Cell Sizes: 
     HEAPP(ON,56,2,280,1,848,1,2080,1,4096,1,0) 

With a small(16KB) heap specified

HEAP(16K,32768,ANYWHERE,FREE,8192,4096)
HEAPP(ON,56,1,280,1,848,1,2080,1,4096,1,0)

The output was

HEAPPOOLS Summary:                                                            
  Specified Element   Extent   Cells Per  Extents    Maximum      Cells In    
  Cell Size Size      Percent  Extent     Allocated  Cells Used   Use         
  ------------------------------------------------------------------------    
       56        64      1           4         251        1002           2    
      280       288      1           4           1           1           0    
      848       856      1           4           1           1           0    
     2080      2088      1           4           1           1           1    
     4096      4104      1           4           0           0           0    
  ------------------------------------------------------------------------    
  Suggested Percentages for current Cell Sizes:                               
    HEAPP(ON,56,90,280,2,848,6,2080,13,4096,1,0)                             

and we can see it had to allocate 251 extents for all the request.

Once the system has “warmed up” there should not be a major difference in performance. I would allocate the heap to be big enough to start with, and avoid extensions.

With the C run time there are heaps as well as heap pools. My C run time report gave

64bit User HEAP statistics:
31bit User HEAP statistics:
24bit User HEAP statistics:
64bit Library HEAP statistics:
31bit Library HEAP statistics:
24bit Library HEAP statistics:
64bit I/O HEAP statistics:
31bit I/O HEAP statistics:
24bit I/O HEAP statistics:

You should check all of these and make the initial size the same as the suggested recommended size. This way the storage will be allocated at startup, and you avoid problems of a request to expand the heap failing due to lack of storage during a buys period.

Advanced level for heap

While the above discussion is suitable for many workloads, especially if they are single threaded. It can get more complex when there are multiple thread using the heappools.

If you have a “hot” or highly active pool you can get contention when obtaining and releasing blocks from the heap pool. You can define multiple pools for an element size. For example

HEAPP(ON,(56,4),1,280,1,848,1,2080,1,4096,1,0)

The (56,4) says make 4 pools with block size of 56 bytes.

The output has

HEAPPOOLS Summary:                                                          
  Specified Element   Extent   Cells Per  Extents    Maximum      Cells In  
  Cell Size Size      Percent  Extent     Allocated  Cells Used   Use       
  ------------------------------------------------------------------------  
       56       64     1           4         251        1002           2  
       56       64     1           4           0           0           0  
       56       64     1           4           0           0           0  
       56       64     1           4           0           0           0  
      280       288      1           4           1           1           0  
      848       856      1           4           1           1           0  
     2080      2088      1           4           1           1           1  
     4096      4104      1           4           0           0           0  
  ------------------------------------------------------------------------  

We can see there are now 4 pools with cell size of 56 bytes. The documentation says Multiple pools are allocated with the same cell size and a portion of the threads are assigned to allocate cells out of each of the pools.

If you have 16 threads you might expect 4 threads to be allocated to each pool.

How do you know if you have a “hot” pool.

You cannot tell from the summary, as you just get the maximum cells used.

In the report is the count of requests for different storage ranges.

Pool  2     size:   160 Get Requests:           777707 
  Successful Get Heap requests:    81-   88                 77934 
  Successful Get Heap requests:    89-   96                 59912 
  Successful Get Heap requests:    97-  104                 47233 
  Successful Get Heap requests:   105-  112                 60263 
  Successful Get Heap requests:   113-  120                 80064 
  Successful Get Heap requests:   121-  128                302815 
  Successful Get Heap requests:   129-  136                 59762 
  Successful Get Heap requests:   137-  144                 43744 
  Successful Get Heap requests:   145-  152                 17307 
  Successful Get Heap requests:   153-  160                 28673
Pool  3     size:   288 Get Requests:            65642  

I used ISPF edit, to process the report.

By extracting the records with size: you get the count of requests per pool.

Pool  1     size:    80 Get Requests:           462187 
Pool  2     size:   160 Get Requests:           777707 
Pool  3     size:   288 Get Requests:            65642 
Pool  4     size:   792 Get Requests:            18293 
Pool  5     size:  1520 Get Requests:            23861 
Pool  6     size:  2728 Get Requests:            11677 
Pool  7     size:  4400 Get Requests:            48943 
Pool  8     size:  8360 Get Requests:            18646 
Pool  9     size: 14376 Get Requests:             1916 
Pool 10     size: 24120 Get Requests:             1961 
Pool 11     size: 37880 Get Requests:             4833 
Pool 12     size: 65536 Get Requests:              716 
Requests greater than the largest cell size:               1652 

It might be worth splitting Pool 2 and seeing if makes a difference in CPU usage at peak time. If it has a benefit, try Pool 1.

You can also sort the “Successful Heap requests” count, and see what range has the most requests. I don’t know what you would use this information for, unless you were investigating why so much storage was being used.

Ph D level for heap

For high use application on boxes with many CPUs you can get contention for storage at the hardware cache level.

Before a CPU can use storage, it has to get the 256 byte cache line into the processor cache. If two CPU’s are fighting for storage in the same 256 bytes the throughput goes down.

By specifying

HEAPP(ALIGN….

It ensures each block is isolated in its own cache line. This can lead to an increase in virtual storage, but you should get improved throughput at the high end. It may make very little difference when there is little load, or on an LPAR with few engines.