What are the ICSF bits?

I struggled for a while to understand the different parts of ICSF.

Definition of terms

You use a symmetric key for encrypting and decrypting data. You use the same key for encryption and decryption.

An asymmetric key has two parts. You encrypt with one, and decrypt with the other. If you encrypt with a key, you cannot use your key to decrypt it. You need the other part. You pick one part, and keep it secure – this is called the private key. The other part you can make this available to every one – the public key.

A certificate is a collecting of information. It contains a public key, and information about the owner of the certificate. I think of a certificate as a paper envelope in which you put the public key, they owner information, and other information, such as validity dates.

Hardware encryption. These days the calculations for encryption can be offloaded to special cryptographic hardware.

Cryptographic keys can be stored in hardware. The hardware can have a cryptographic key, which can be used to encrypt data before storing it in the hardware.

You can request the hardware to extract the data, and encrypt it with another key. This way the encryption key is never seen outside of the hardware.

ICSF provides an API to use z/OS hardware cryptographic functions, and some facilities to mange the keys.

ICSF

A symmetric key can have different algorithms. Typically RSA and AES. RSA came first, but AES is considered better as it takes more work to crack, and uses less resources when encrypting and decrypting data.

With ICSF, the symmetric keys are stored (encrypted) within the ICSF CKDS database. The keys are referenced to using tokens. You can have:

  • A null token – used as place holder.
  • A fixed length token, this is an old format token.
  • A variable length token. As cryptography developed over time, all the information could not be stored in a fixed length, and so variable length tokens were introduced.

With ICSF, Asymmetric keys are called PKA (Public Key algorithms).

An asymmetric key can have different algorithms. Typically RSA, Elliptic Curve (EC), and various Crystal methods (these are based on complex calculations based on crystal lattices).

The asymmetric keys are stored (encrypted) within the ICSF PKDS databases. The keys are referenced to using tokens.

As with symmetric keys, the early asymmetric tokens were a fixed length. As cryptography developed, variable length tokens were introduced.

Symmetric ciphers in more depth

ICSF has two forms of symmetric cipher for encrypting user data

  • DATA – I think of this as version 1. The key value can be extracted as cleartext – this is not very secure.
  • CIPHER – I think this as version 2. The key value is always encrypted – either within ICSF, encrypted on the hardware, or with a transmission key.

Best practice is to use CIPHER instead of DATA. There is an older API function for generating the DATA keys, and a newer API function (CSNBKTB2) for the CIPHER keys. The newer function has more capability. For example, you can specify options that allow/prohibit exporting the key, and which encryption modes can be used, such as cipher block chaining(CBC).

  • AES has one type of cipher: CIPHER. This can be used for data set encryption and in encrypt API (CSNBENC), and the decrypt API (CSNBDEC).
  • DES has three types of cipher
    • DATA which can be used to encrypt and decrypt. This can be used for data set encryption and in encrypt API (CSNBENC), and the decrypt API (CSNBDEC).
    • ENCIPHER. This can be used in the encrypt data API (CSNBENC).
    • DECIPHER. This can be used in the decrypt data API (CSNBDEC).

  • To use the ENCIPHER function you need a DATA, CIPHER or ENCIPHER key
  • To use the DECIPHER function you need a DATA, CIPHER or DECIPHER key
  • To use data set encryption you need DATA or CIPHER (because you need to do encryption and decryption).

Generating symmetric keys

You can use the ISPF panel interface.

For example, to create a symmetric key; from the ICSF main page select options:

  • 5 UTILITY
  • 5 CKDS KEYS – Manage keys in the CKDS
  • 8 Generate AES CIPHER keys

You specify the name of the key, the AES key bit length, which encryption mode and if it can be exported from the hardware (XPRTCPAC).

The ISPF panels provide defaults. You can see the values by using the ISPF interface to display entries in the CKDS. For example

Key usage:ENCRYPT and DECRYPT, and ANY-MODE

Key management: gives XPRTCPAC and other keys. For example XPRT-AES says the key can be exported and encrypted using an AES Key Exporting Key.

Use the CSFKGUP batch interface.

The ICSF administrators guide gives details of the utility. It uses data sets (which may not be the active ones). You need to tell ICSF to refresh the in-memory information after the data sets have been updated.

For example

//IBMICSF  JOB 1,MSGCLASS=H 
// SET CKDS=COLIN.SCSFCKDS
//KGUP EXEC PGM=CSFKGUP,PARM=('SSM')
//CSFCKDS DD DISP=OLD,DSN=&CKDS
//CSFIN DD *,LRECL=80
...
//CSFDIAG DD SYSOUT=*,LRECL=133
//CSFKEYS DD DSN=COLIN.CSFKEYS,DISP=SHR
//CSFSTMNT DD SYSOUT=*,LRECL=80
//* REFRESH THE IN MEMORY DATA
//REFRESH EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'
//

The statements in CSFIN are like

DELETE TYPE(CIPHER) LABEL(AESCIPHER) 
ADD TYPE(CIPHER ) ALGORITHM(AES) LENGTH(32),
LAB(AESCIPHER)

You can specify values for KEYUSAGE and KEYMGMT for example

KEYUSAGE(ANY-MODE) KEYMGMT(XPRTCPAC).

CSFKGUP seems to provide the same capability as the ISPF interface.

If you use CSFKGUP you need to issue a program to refresh the in memory information from the updated CKDS and TKDS data sets.

How is ICSF configured?

ICSF is started as a started task, so uses JCL in the SYS1.PROCLIB concatenation. I have member CSF in USER.Z31A.PROCLIB.

//CSF  PROC PRM=CP 
//CSF EXEC PGM=CSFINIT,PARM=&PRM,REGION=0M,TIME=1440,MEMLIMIT=NOLIMIT

The parameters are selected when the procedure is started. For example

S CSF,PRM=C2

The parameters come from member CSFPRMxx in the parmlib concatenation. My default parameters are in USER.Z31A.PARMLIB(CSFPRMCP)

CKDSN(COLIN.SCSFCKDS)                   
PKDSN(COLIN.SCSFPKDS)
TKDSN(COLIN.SCSFTKDS)
DOMAIN(0)
SSM(YES)
KEYARCHMSG(YES)

To use different data sets, you need to stop ICSF, and restart it with a different configuration member, specifying different data set names.

Using ICSF within a single image or sysplex

This blog post describes using ICSF within one system. This could be a sysplex if all systems use the same ICSF data sets. It gives some guidance on setting up your environment, and some implications of your decisions.

For some guidance on defining keys for Data Set Encryption see here(how many keys?). For a deep dive on key management which talks about changing the keys on the ICSF data sets (key rotation) see here.

Introduction

This section gives some general information about using ICSF, keys and data set encryption.

  • You set up a symmetric key for data set encryption.
  • You can specify which key a data set should use, when the data set is created.
    • You can configure a RACF profile to specify the key name in the DFP information.
      • When you allocate a data set which uses this RACF profile, the key is extracted from the DFP information and used.
    • The data set must be SMS-managed extended format. (You can configure this either in JCL, or with SMS options)
    • You can specify the keylabel in the JCL
    • You can specify in the SMS Data class. With SMS you configure ACS with statements like if the data set attributes are … then use data class xxxx, which in turn has a data set encryption key specified.
  • The key name is stored with in the catalog along with other SMS information.
  • If you change the label in the RACF profile, any new dataset which get created get the updated label, existing data sets are unchanged.
  • If you delete and recreate a key for a label, any data set which used the old label becomes unreadable.
  • You can unload and reload a key using my programs in EASYICSF on Github. ICSF does not provide this capability. When you define a key, you might unload it, and archive it away. Then, if the key gets deleted by accident, you can then recover by reloading it.
  • Keys stored by ICSF are encrypted before they are stored in the ICSF data set. The master encryption key for these data sets can be stored in the cryptographic hardware. If someone steals the ICSF data sets, they cannot use them, unless they have the master key as well.
  • You can get ICSF to use a new master password. It will take each entry in the ICSF data set, decrypt it with the old key, encrypt it with the new key, and store it back in (a new) data set. This is known as key rotation. (This is a strange use of the term rotation because there is no rotation, you just change the key.)
  • If you want to use a new key for a data set. You have to copy the contents of the data set to a new data set set up with the new key. You then rename the data set. You cannot just change the key.
  • You can specify a start and end date for a key. The start date can be today’s or any date in the future. The end date is after the value (this means you cannot specify an end date, so it can not be used today.)
  • If you want to stop a key from being used, you can set an archive flag on it. (If you delete it, any data sets using this key cannot be processed). By default the application using an archived key will fail. You can configure ICSF so that use of an archived key will work, and you get a message for the key on the ICSF joblog, and a record in SMF.
  • You can control who can use keys by a RACF profile in class CSFKEYS. If you use CSF.CSFKEYS.CONDITIONAL.ACCESS.CONTROL you can restrict what functions userids can execute – and so prevent most people from being able delete keys.
  • Copying data sets
    • If you use DFDSS backup, the encrypted data is backed up. When you restore it, it allocates a data set with the same encryption label. The intermediate file is not encrypted. You can still read the intermediate file and see the meta data – but the original file contents are still encrypted.
    • If you use IEBGENER to copy the data, if the output data set does not have an encryption label, the output data set is encrypted with the same key label as the input.
    • To specify a key label you can specify DSKEYLBL in JCL and in the TSO ALLOC command.
    • In an application
      • you can specify the key label using SVC 99 (dynamic allocation).
      • you cannot specify it using the C function fopen(). You will need to specify it in JCL and use DD:name to refer to the data set.
    • If you use other program, such as a ISPF edit or a C program, to read the file and write the contents out, by default the output file is not encrypted. The application sees the clear text, and the output is in clear text unless you have specified a key label for the output data set.

Planning

You need to consider how many keys you need. For example, one for all your data sets, or one per business function such as online banking, or stocks and shares.

Keys for some files, such as a database will be long lived – as long as the data base exists, or until you copy it with a new key. The data set may have been archived to tape, and could be recalled years later.

You need to plan who has access to the data set. For a DB2 data base data set, people will not normally need access to it, as the access is managed by DB2. The DB2 system userid will need access. Storage managers, responsible for backing up the data sets do not need access to the key, and access the encrypted data without decryption.

If someone has access to an encrypted data set, you need to ensure that any data set they create which may have sensitive information in it is also encrypted. This may mean reviewing userids and what they are used for. For example if you have a data set ENC.DATASET1 which userid COLIN can access, do you need to ensure that any data set that userid COLIN can create is encrypted?. This may mean data sets are encrypted which do not contain sensitive information. If userid COLIN can also create data sets beginning with SYS2 – you need to consider if all SYS2.* datasets need to be encrypted.

You might define a userid COLINE which can only access the encrypted data sets, and can only created data sets prefixed with COLINE – and these are all encrypted. Userid COLINE does not have access to general data sets, such as SYS1.PARMLIB.

For short lived data sets you could change the key every few months.

You need to consider backups and disaster recovery, and how your remote sites will use the datasets. The remote sites will need the same keys for the data sets, as the primary site.

Changing keys

As a general rule, you do not want to delete a key unless you are sure that is is not needed (including any data sets that may have been archived off the system). If you want to generate new keys periodically you may want to use a naming convention such as PAYROLL2024NOV, then generate a new key PAYROLL2025NOV, and update your definitions (or SMS).

Using keys

You can specify the encryption key to be used in

  1. The RACF profile used for the data set. The value is in the DFP segment.
  2. The JCL DSKEYLBL parameter on the DD statement.
  3. In the SMS data class used for the data set. This can be specified in the DATACLAS parameter on the DD statement, or selected via the ACS(Automatic Class Selection) routine.

The key is used when the data set is allocated(created).

If you change the RACF profile, the updated label will be used next time a data set is created.

If people specify the key either by application on JCL (the DSKEYLBL option on the DD statement), changing the key name will mean changing the applications and JCL to use the new key.

An ACS routine has logic like

if the data set names begins with … , and has properties …. then use the following volumes…, make it this size…, and use the Data Class … which has the Data Set Key Label…)

If you change the Data Set Key Label in the Data class, the next time the Data Class is used, it will use the updated key. This is much simpler than trying to change JCL.

It looks like it is easier to use the RACF profile or SMS if you will be changing the data set encryption key.

Creating keys

  • You can use the ICSF utility CSFKGUP to generate keys. The syntax is like ADD TYPE(CIPHER) LABEL(COLINKEY)…
  • You can use the ISPF interface to define keys. From the main panel, options 5 UTILITY -> 5 CKDS KEYS, option 7 or 8 to define a symmetric key.
  • You can use ISPF to generate statements for CSFKGUP. From the main panel option 8 KGUP – Key Generator Utility processes. This simplifies selecting the options. For example if you select the action to maintain (add) an entry, if you specify algorithm DES, it will display the valid key types for DES (eg CIPHER, DATA, ENCIPHER…). If you select algorithm AES, it will display the valid key types for AES ( eg CIPHER, DATA, EXPORTER…). If you use this method, you may still have to edit the statements manually, because not all the required parameters are specified, see below.

For some keys you need to have a RACF profile defined for the label name. For example

RDEFINE CSFKEYS DES5 UACC(NONE) –
ICSF(SYMCPACFWRAP(YES) SYMCPACFRET(YES))

This seems to be a strange way of storing this information.

Setting up data set encryption

Best practices.

Use definition like

ADD ALGORITHM(AES) TYPE(CIPHER) KEYUSAGE(ANY-MODE) LENGTH(32), 
KEYMGT(XPRTCPAC)LABEL(COLINCIPHER2 )

Where

  • ALGORITHM(AES) is better than ALGORITHM(DES). It is harder to crack, and uses less CPU.
  • TYPE(CIPHER) is better than TYPE(DATA).
  • KEYUSAGE(ANY-MODE) is needed for data set encryption with AES.
  • LENGTH(32) is the longest and strongest AES key size.
  • KEYMGT(XPRTCPAC) is needed for data set encryption.
  • LABEL(COLINCIPHER2 ) you specify your label.

If you use the ISPF panels to generate the key, you can specify all of the parameters. If you use the ISPF interface and 8 KGUP – Key Generator Utility processes. This does not specify create KEYUSAGE and KEYMGT values, which will have to be entered manually.

Afterthoughts

You need to backup the ICSF data sets.

If you need to use a (restored) data set with a different name, you will need to restart ICSF with a different parmlib member containing the different names.

ICSF does not provide a way of copying between ICSF data sets, but you can use UNLOAD and RELOAD programs in my EASYICSF on Github which allows you to copy a key from one ICSF data set to an intermediate data set, then reload it into a different,or the same, ICSF data set.

You should have a process, which you have tested, for replacing keys which may be been exposed. This may cause a long (hours) outage, so you need to have JCL and definitions written before you need them.

All I ever wanted to know about ICSF and data set encryption keys

z/OS systems have cryptographic processors, which can be used to offload expensive cryptographic operations. You can configure z/OS so if you generate a cryptographic key, you do not see the actual, clear text key (so it cannot easily be stolen). This is done by the cryptographic processors having a cryptographic key themselves, and any keys they emit, are encrypted with this hardware key. When you use your cryptographic key, it is passed to the cryptographic processors, which then decrypts the data using the hardware key, then uses your real cryptographic key.

ICSF provides an interface to do cryptography, use the cryptography processors and to mange cryptographic keys. Most z/OS user will ICSF capabilities without knowing about it.

ICSF works at the level of symmetric and asymmetric keys. You cannot use it to generate digital certificate, do TLS handshakes, nor do data set encryption (but data set encryption uses ICSF keys).

ICSF provides

  • An ISPF interface to mange keys. You can list, define, delete etc keys through the panels
  • A batch program CSFKGUP which can be used to so some of the key management. This acts on a data set. You need to tell ICSF to refresh the in-memory contents from the data set.
  • An extensive set of API’s if you want to write your own application, for example to generate a key, or to encrypt some data. It is relatively easy to write Rexx programs to use the APIS. These APIs work on the active ICSF data sets. To use different data sets you need to change the ICSF configuration.

Having spent a few weeks trying to use it I feel “how to” information is missing. I tried implementing data set encryption across disconnected system and found ICSF is missing some functions, for example batch program to export and import keys, and a batch program to securely generate shared keys on different systems.

In this blog post, I’ve tried to fill in the holes in the documentation, and provide the missing batch utility programs.

I’ve written blogs posts on

I’ve put my utilities and sample code on Github.

Using ICSF in a multi system environment

Using ICSF within a single image or sysplex discussed setting up ICSF in a single environment, where systems can share the ICSF datasets.

This blog post is when you have multiple systems which do not share the ICSF data sets.

You have some encrypted data sets which you have sent to your backup system. How do you get the ICSF keys to your backup site? I will focus on using AES CIPHER keys because AES is recommended over DES keys; and CIPHER keys are recommended over DATA keys.

The challenge of using ICSF in a multi system environment where the ICSF data sets are not shared, is setting up the first keys. This is not easy as ICSF does not provide all the functionality to do this.

If your systems have the same hardware key, you can unload and reload a key using my programs in EASYICSF on Github. You unload the key (encrypted with the hardware key) to a data set, send the data set to the remote system, and reload the key.

If your systems do not use the same hardware key, you can fairly easily set up the keys for use between systems if you can accept some clear text key value sharing.

If your systems do not use the same hardware key, the most secure way of setting up keys is where the key value is never in clear text, and so cannot be copied or used in an authorised way. This has many steps, and I’ll cover this in another blog post (Generating ICSF keys using Diffie-Hellman).

If you want to encrypt a key to send to a remote site, ICSF provides Key Encryption Key (KEK) technology. Once they KEK has been set up you can export a key from the local system, encrypting it with an exporter KEK. Send the encrypted data to the remote site, and use the matching KEK to import it. At no point is the clear text version of the key exposed. I think of a KEK as an asymmetric key for data. Having the exporter KEK does not mean you can decrypt the data.

You need to have an exporter key to get from system A to system B, and an exporter key to get from system B to system A. If you specify the key value yourself these could be the same. If you let the system generate the key value, they will be different.

Once you have keys set up, you can use ICSF Key Encryption Key (KEK) to send keys securely to another system. The key phrase is “Once you have set up…”.

How long are keys needed for ?

A cipher key used in data set encryption is needed for as long as the data set using it is needed, so potentially many years. Deleting the key means the data set cannot be read.

When you create a KEK, you create both the exporter and importer at the same time. You cannot generate one from the other. A KEK is only used when transferring keys between systems. You could have a new KEK every week, with no impact. You just need to ensure you have at least another KEK to encrypt it.

How many KEK’s do I need?

If you have three systems, Production, Backup and Disaster Recovery (DR). You could define an exporter key on production, and the same importer key on Backup and DR. This might be easy to implement, but I think I would feel more comfortable with different KEKs:

  • Exporter on Production and Importer on Backup,
  • Exporter on Production and Importer on DR,
  • Backup to DR

and the opposite; Exporter on Backup and Importer on Production etc.

I included the system names in the data set used to store the key. When I initially used the same name for all definitions, such as ‘COLIN.EXPORTED.KEK’, I found I was getting into a muddle, and had the wrong key in it.

To export and import AES CIPHER keys see export a key from CKDS into a data set so it can be imported on another system and import a key from a data set into the CKDS on Github. You could just use unload and reload a key using my programs in EASYICSF on Github!

Overview to creating keys in a multi system environment

  • If the hardware keys are the same on each system, you can use UNLOAD and RELOAD programs in my EASYICSF on Github which allows you to copy the hardware encrypted key from one ICSF data set to an intermediate data set, then reload it into an ICSF data set. See here.
  • You can generate data set encryption keys on each system using a cut and paste of the JCL. This is not very secure because the key value is visible in clear text. See here.
  • Define Key Encrypting Keys for sending keys encrypted between systems. See here for detailed instructions.
    • On the sender system define a temporary exporter key specifying the key value in clear text.
    • On the receiving system(s) define the matching importer key using the same key values (cut and paste) and a small edit.
    • Define a permanent exporter key letting the system securely generate the key value, and specify the temporary KEK to encrypt the importer key.
    • Send the encrypted permanent importer KEK to the remote system.
    • On the receiver systems use the temporary importer KEK to import the permanent key
    • Delete the temporary KEK from each system.
    • You now have KEKs on your system, and the window where there was clear text visible was small.
  • Locally generate a secure data set encryption key, and export it to the remote systems as needed. See here.
    • You generate a data set encryption key.
    • For each system you want to send this key to you export it, using the exporter KEK for that system.
    • If you want to send the key at a later date, you just export the key using the appropriate KEK.
  • Generating secure data set encryption keys, both local and encrypted with a KEK
    • ICSF has a utility CSFKGUP which, in one request, can generate a key for use locally, and export it using an exporter KEK.
    • This does not feel a very good solution to me.
      • If you want to send the key to multiple systems, you need to use the same KEK for all systems.
      • If, at a later date, you want to send the key to another system you have to export it (see above). It is easier just to export it anyway.

These may look relatively simple – but ICSF does not provide a utility to export and import the keys. You can write your own C or REXX program, or use the ones I have written. See export a key and import a key in EasyICSF on Gitub.

Detailed instructions

Use unload and reload.

This is based on unload and reload a key using my programs in EASYICSF on Github.

Unload

//IBMUNL  JOB 'COLIN',CLASS=A,REGION=0M,COND=(4,LE) RESTART=RUN 
//JOBLIB JCLLIB ORDER=(COLIN.ICSF.C,CBC.SCCNPRC)
//* COMPILE EXEC CCPROC,PROG=UNLOAD
//*
//*
// SET KEY='-key SECKEY2 '
// SET DB='-debug 8 '
// SET DB=' '
//RUN EXEC PGM=UNLOAD,REGION=0M,
// PARM='&KEY. &DB '
//STEPLIB DD DISP=SHR,DSN=COLIN.ICSFLOAD
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200)
//SYSOUT DD SYSOUT=*
//KEY DD DSN=COLIN.UNLOAD.AES,DISP=(OLD)
//* SPACE=(CYL,(1,1)),DCB=(LRECL=7000,RECFM=V)

Reload

//IBMRELOA JOB 'COLIN',CLASS=A,REGION=0M,COND=(4,LE) RESTART=RUN 
//JOBLIB JCLLIB ORDER=(COLIN.ICSF.C,CBC.SCCNPRC)
//*OMPILE EXEC CCPROC,PROG=RELOAD
//*
//*
// SET DB='-debug 8 '
// SET KEY='-key SECKEY2 '
//RUN EXEC PGM=RELOAD,REGION=0M,
// PARM='&KEY. &DB '
//STEPLIB DD DISP=SHR,DSN=COLIN.ICSFLOAD
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200)
//SYSOUT DD SYSOUT=*
//KEY DD DSN=COLIN.UNLOAD.AES,DISP=(OLD)

An easy, not very secure method, for use in a proof of concept.

You can use JCL Like

//IBMICSF JOB 1,MSGCLASS=H
//STEP10 EXEC PGM=CSFKGUP
// SET CKDS=COLIN.SCSFCKDS
//CSFCKDS DD DISP=OLD,DSN=&CKDS
//CSFIN DD *,LRECL=80
ADD TYPE(CIPHER) ALGORITHM(AES),
KEYMGT(XPRTCPAC),
CLEAR,
KEY(0123456789ABCDEF,0123456789ABCDEF,
0123456789ABCDEF,0123456789ABCDEF),
LABEL(COLINCIPHERCLEAR ) ,
KEYUSAGE(ANY-MODE)
/*
//CSFDIAG DD SYSOUT=,LRECL=133 //CSFKEYS DD SYSOUT=,LRECL=1044
//CSFSTMNT DD SYSOUT=,LRECL=80 // REFRESH THE IN MEMORY DATA
//REFRESH EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

You run this on every system where you need the data set encryption key.

You need to specify for data se

  • KEYMGT(XPRTCPAC),
  • KEYUSAGE(ANY-MODE)
  • KEY(….) You need to pick a set of random hexadecimal numbers.

You can run this job on all systems you need the data set encryption keys on, with no changes.

This is not secure because other people may be able to see the keys…

  • You may have saved the JCL containing the key value (for future use) either in a data set or on paper
  • If you ran this as a job, the values may be in the spool output. Depending on how you specified the output class.
  • Copying the values to the remote system may get intercepted.

Defining KEKs

If you want to encrypt a key to send to a remote site ICSF provides Key Encryption Key (KEK) technology. Once the KEK has been set up you can export a key from the local system, encrypting it with an exporter KEK. Send the encrypted data to the remote site, and use the matching KEK to import it. At no point is the clear text version of the key exposed.

Generate a bootstrap KEK.

This bootstrap KEK is only used to set up a permanent KEK.

In a similar way to defining the data set encryption key above you can use

//IBMICSF  JOB 1,MSGCLASS=H 
//STEP10 EXEC PGM=CSFKGUP
// SET CKDS=COLIN.SCSFCKDS
//CSFCKDS DD DISP=OLD,DSN=&CKDS
//CSFIN DD *,LRECL=80
DELETE TYPE(EXPORTER) LABEL(TEMPATOB)
ADD LABEL(TEMPATOB)TYPE(EXPORTER) CLEAR ,
ALGORITHM(AES),
KEY(3123456789ABCDEF,0123456789ABCDEF)
/*
//CSFDIAG DD SYSOUT=*,LRECL=133
//* REFRESH THE IN MEMORY DATA
//REFRESH EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

On the remote system, use the same JCL except with TYPE(IMPORTER). You will, of course, use different hex values in the key to those I gave.

Generate the long term KEK.

//IBMICSF  JOB 1,MSGCLASS=H 
//STEP10 EXEC PGM=CSFKGUP
// SET CKDS=COLIN.SCSFCKDS
//CSFCKDS DD DISP=OLD,DSN=&CKDS
//* LENGTH(32) GENERATES A 256 BIT KEY
//CSFIN DD *,LRECL=80
ADD LABEL(ATOBPERM) TYPE(EXPORTER) ,
TRANSKEY(TEMPATOB) ,
ALGORITHM(AES) LENGTH(32)
/*
//CSFDIAG DD SYSOUT=*,LRECL=133
//CSFKEYS DD DSN=COLIN.CSFKEYS,DISP=OLD
//CSFSTMNT DD SYSOUT=*,LRECL=80
//* REFRESH THE IN MEMORY DATA
//REFRESH EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

This creates a key value, stores the exporter KEK in the local CKDS encrypted with the hardware keys. It encrypts the new importer KEK with the TEMPATOB KEK and stores it in the CSFKEYS dataset.

The layout of this CSFKEYS data set is:

  • ATOBPERM the name of the object,
  • IMPORTER you need this sort of key to import it,
  • ATOB this is the KEK name used to encrypt it
  • The key for ATOBPERM with the key value encrypted. This is a hexadecimal value.

Knowing the information in the CSFKEYS data set you can import the key into the CKDS on the remote system, using the TEMPATOB importer key.

ICSF does not provide a utility for this. You can write some REXX code, or you can use my programs. See import a key from a data set into the CKDS.

Delete the temporary KEK

You can use the JCL

//IBMICSF  JOB 1,MSGCLASS=H 
//STEP10 EXEC PGM=CSFKGUP
// SET CKDS=COLIN.SCSFCKDS
//CSFCKDS DD DISP=OLD,DSN=&CKDS
//CSFIN DD *,LRECL=80
DELETE TYPE(EXPORTER) LABEL(TEMPATOB)
/*
//CSFDIAG DD SYSOUT=*,LRECL=133
//CSFKEYS DD DUMMY
//CSFSTMNT DD DUMMY
//* REFRESH THE IN MEMORY DATA
//REFRESH EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

Locally generate a secure data set encryption key

If you do not have a data set encryption key, use JCL like

Securely generate data set encryption keys.

//IBMDSENC JOB 1,MSGCLASS=H 
//STEP0 EXEC PGM=CSFKGUP
//*
// SET CKDS=COLIN.SCSFCKDS
//CSFCKDS DD DISP=OLD,DSN=&CKDS
//* LENGTH(32) GENERATES A 256 BIT KEY
//CSFIN DD *,LRECL=80
ADD TYPE(CIPHER) ALGORITHM(AES) LENGTH(32),
LABEL(DSENCRYPTION),
KEYMGT(XPRTCPAC),
KEYUSAGE(ANY-MODE)
/*
//CSFDIAG DD SYSOUT=*,LRECL=133
//CSFKEYS DD DUMMY
//CSFSTMNT DD DUMMY
//* REFRESH THE IN MEMORY DATA
//REFRESH EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'
//

You can then send the CSFKEYS data set to the remote site, and import it using the ATOBPERM importer key. See import a key from a data set into the CKDS on Github.

Using keys across systems

You can only use keys which are within their validity dates: crypto-period start date and
cypto-period end date.

When you import a key, meta data from the original site is not included, so you might need to specify start date, stop date, archive flags etc. See set meta data attributes on Github.

Generating ICSF keys using Diffie-Hellman

The Diffie-Hellman technique allows you to create keys on two system securely, with no need to share a common key. Each system needs a private/public key pair, and the public keys are available on all systems. There are many steps but it is relatively painless.

Scenario

Two systems A and B. We want to have a data set encryption key on system A copied to system B and to create an exporter key on A, and a matching importer key on B.

The steps are

  • generate private/public pairs on each system
  • use these to generate the symmetric keys.

Generate a private/public key pair on each system, and send the public key to the other system.

On System A

Use the ISPF ICSF panels

  • Options 5;6;6 Generate PKA keys, import or export public keys via certificate
    • Enter PKDS record’s label: SYSTEMA
    • Select Generate a new RSA or EC key pair record
    • Select EC NIST Curve, P521.
    • This should display PKDS Key Request Successful. Press enter to return.
    • Select Export the PKDS record’s public key to a certificate data set
    • Specify a data set name ‘COLIN.SYSTEMA’
    • Specify the common name SYSTEMA
    • Press enter
    • This should give you PKDS Public Key Export Successful
  • Send the data set ‘COLIN.SYSTEMA’ to the remote system.

On System B

  • Follow the instructions above to generate PKA keys.
    • Label Name SYSTEMB, and use export data set name COLIN.SYSTEMB, etc. Check it has worked.
    • Next, select Create a PKDS public key record from an input certificate
    • Enter the name of the data set you copied across containing the public key for SYSTEMA
    • Press enter. It should import the key
    • F3 to return to the ICSF PDKS keys page.
    • Select option 1, and press enter. Use the line command K to display the contents of the certificates. You should have the private key with sections “PRIVATE PUBLIC”, and the public key should have section “PUBLIC”
  • Send the data set ‘COLIN.SYSTEMB’ to the first system.

On System A

  • Read in the data set
  • Options 5;6;6 Generate PKA keys, import or export public keys via certificate
  • Enter PKDS record label SYSTEMB
  • Select Create a PKDS public key record from an input certificate
  • Enter the name of the data set COLIN.SYSTEMB.
  • When this has worked, display the contents of the PKDS as described above.

You now have a private/public key on each system, and each system has a copy of the other’s public key.

Generate a symmetric key on each system

Now you have the private/public key on each system, and the public key of the other system, you can create a key.

ICSF does not provide a function for this. See Generate a secure shared key on multiple systems using Diffie-Hellman in Github.

  • On SYSTEMA you need the name of the private/public pair SYSTEMA, and the name of the public certificate SYSTEMB.
  • You need a phrase which provides a seed to the encryption. This does not need to be kept confidential, but you may still wish to protect it.

Generate a CIPHER for data set encryption

//RUN      EXEC PGM=GENDH,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD *
-ptype INTERNAL,AES,CIPHER,XPRTCPAC,ANY-MODE
-key AESDHDSCIPHER
-private AAA
-public BBB
-replace Y
-party cOlinSeed
-debug 0
/*
//STEPLIB DD DISP=SHR,DSN=COLIN.LOAD
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200)
//CEEDUMP DD SYSOUT=*,DCB=(LRECL=200)
//SYSOUT DD SYSOUT=*
//SYSERR DD SYSOUT=*

On the other system change AAA and BBB, and run the JCL.

Create an exporter key on SYSTEMA

//RUN      EXEC PGM=GENDH,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD *
-ptype INTERNAL,AES,EXPORTER
-key AESDHKEK
-private AAA
-public BBB
-replace Y
-party cOlinSeed2
-debug 0
/*
//STEPLIB DD DISP=SHR,DSN=COLIN.LOAD
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200)
//CEEDUMP DD SYSOUT=*,DCB=(LRECL=200)
//SYSOUT DD SYSOUT=*
//SYSERR DD SYSOUT=*

Create a matching importer key on SYSTEMB

//RUN      EXEC PGM=GENDH,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD *
-ptype INTERNAL,AES,IMPORTER
-key AESDHKEK
-private BBB
-public AAA
-replace Y
-party cOlinSeed2
-debug 0
/*
//STEPLIB DD DISP=SHR,DSN=COLIN.LOAD
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200)
//CEEDUMP DD SYSOUT=*,DCB=(LRECL=200)
//SYSOUT DD SYSOUT=*
//SYSERR DD SYSOUT=*

You can now use exporter key AESDHKEK on system A, and importer KEK AESDHKEK on system B.

Idiot’s guide to TCPIP connectivity

I had a working TCPIP network, and made a few “improvements”. Unfortunately these improvements sometimes stopped the connectivity between systems, and I had a frustrating time understanding the problems and fixing them. The idiot in the blog post is me, for next time when I need to connect boxes together.

In concept TCPIP connectivity is simple – it is, but there are some subtle, non obvious things you need to be aware of.

As I was writing this post I found I did not know really how IPV4 worked, because it used “the wrong” IP address but still worked.

I found many ways of failing to connect to TCPIP, and some complex ways of getting it to work – I just wanted a simple way of being able to ping z/OS from my laptop. It is complicated by some definitions need to be done in order, and doing things in a different order sometimes worked, sometimes did not.

Basic TCPIP concepts that every one should know

  • The term socket is used by applications to communicate with TCP/IP, not where you connect a network/phone cable.
  • Think of a connection between two boxes. I have a yellow Ethernet cable between them. There are several terms for the where the cable is plugged in. A common term is the interface.
  • IP addresses
    • Each end of the connection has one or more IP addresses.  I think of it as having plastic labels tied to the end of the cable.
    • IPV6 addresses beginning with fe… and ff… are used by (internal use) advanced technology and can be ignored. You can use them, but the addresses may change every time the connection is started, which makes it hard to automate using them.
    • The system may generate some IPV6 addresses, but you can define your own. The system generated an address like 2a00:9999:8888:7777:894e:9876:781:32f1. Sometimes parts of these (the right hand part) are randomised (to make it harder for people to observer traffic patterns and so hack your system).
    • I use addresses like 2001:db8::f which are shorter to type.
    • On z/OS an IPV4 interface can have only one IP address. An IPV6 interface can have multiple addresses see ADDADDR. On z/OS an interface can be IPV4 or IPV6 but not both.
    • On Linux, an interface can have multiple IPV4, and multiple IPV6 addresses (but only the first IPV4 may be visible to applications)
    • For IPV6, TCP/IP can generate its own IPV6 addresses for internal processing, such as routing.
  • To get data from this machine to that machine over the yellow Ethernet cable, you have a route definition like “for this range of remote addresses use the yellow Ethernet cable, which has the address xxxx at the far end.
  • If you use TCP/IP to send a request, you usually want a response to come back. As well as defining a route to get to the remote end, you need a route defined to get from the remote machine back to the local machine. A ping request can fail because
    • The local end does not have a valid route to the remote end. The packet could be sent to the wrong place(down the wrong cable), or just discarded.
    • An intermediate box does not have a route to the remote end.
    • The remote end receives the request but does not have a route definition to send the response back to the requester.
    • An intermediate box does not have a route to the local end.
    • A firewall says no.
    • You can use the traceroute command to find the path taken to the remote end. This will tell you the path it took to get there. It does not tell you the route back. For this you need to issue the traceroute command on the remote end, and perhaps on intermediate boxes.
  • You define a route from this box using the yellow cable with label xxxx on it. The remote end of the cable has IP address….
  • You need at least two route statements
    • to get the data from the local system to the remote system,
    • the remote system needs a route statement to get to the local system.
  • You can find these address using
    • the Linux command ip -6 addr or ip -4 addr for TCP IPV6 and IPV4 respectively.
    • the z/OS command TSO NETSTAT HOME
  • Subnet: an IP V6 address has 32 hex digits. These are broken up into groups of 16 eg 2001:0DB8… This can be written as 2001:db8… The subnet specified which bits are significant when routing packets to the router. With z/OS usually the top 64 bits are used. This is written as …./64.
  • An address 2001:db8:9::1/64 is in a different subnet to address 2001:db8:8::1/64.
  • Address 2001:db8:8:1::2/64 is in the same subnet as 2001:db8:8:1::3 because only the top 64 bits count towards the subnet (2001:db8:8:1).
  • A gateway is a network point that acts as an entrance to another network. On the Internet, a node or stopping point can be either a gateway node or a host (end-point) node. Both the computers of Internet users and the computers that serve pages to users are host nodes. A gateway can have one protocol in, and output the data in a different protocol. For example I have broadband coming to my house. The gateway router converts this to TCP/IP, and converts it to wireless.

Things that you may not know

  • My end of a connection has two IP addresses defined. If I ping a remote site it uses the first IP address in its list, the remote site sees a packet of data from the first IP address in the list. You may have configured a route at the remote system to get back to your local system, but if you define your local addresses in a different order, a different IP address will be sent – and the remote end may not have a route for it.
  • If the interface at the next machine has two IP addresses 10.1.0.3 and 7.168.1.2 , I have to use the first IP address in the list defining a route sudo ip -4 route add 7.168.1.74 via 10.1.0.3 dev enp0s31f6. If I delete the first address, then I need to use the 7.168.1.2

Python could not read a data set I sent from z/OS USS.

I created a file in Unix System Services, and FTPed it down to my Linux box. I could edit it, and process it with no problems, until I came to read in the file using Python.

Python gave me

File “<frozen codecs>”, line 322, in decode
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xb8 in position 3996: invalid start byte

The Linux command file pagentn.txt gave me

pagentn.txt: ISO-8859 text

whereas other files had ASCII text.

I changed my Python program to have

with open(“/home/colinpaice/python/pagentn.txt”,encoding=”ISO-8859-1″) as file:

and it worked!

I browsed the web, and found a Python way of finding the code page of a file

import chardet    
rawdata = open(infile, 'rb').read()
result = chardet.detect(rawdata)
charenc = result['encoding']

it returned a dict with

result {‘encoding’: ‘ISO-8859-1’, ‘confidence’: 0.73, ‘language’: ”}

Why does it take an application using TCPIP, so long to start?

I had problems with a couple of applications taking over 30 seconds to start. For example FTP and the RMF DDS Server.


I found this was cause by my TCPIP resolver was misconfigured. An application can ask DNS for the IP address(10.1.1.2) or the string address (BBC.CO.UK) from the TCPIP Resolver function. On my system this was configured in ADCD.Z31A.TCPPARMS(GBLTDATA) as

LOOKUP DNS LOCAL

This says, go to the network, and ask the DNS server “out there” for information. If this request times out, use the local information. On my system the path to the DNS server was not configured, so it waited, and eventually timed out.

When I changed the LOOKUP definition to

LOOKUP LOCAL

it came up with no delays.

Debugging the “you do not have access to something, but I’m not telling you what” problem

The problem, I had a message

SSL Handshake Failed, ICSF error. Review ‘RACF CSFSERV Resource Requirements’ of the z/OS documentation.
Reason: The webservers userid does not have access to CSFSERV resource classes required for SSL.

But it does not tell me what it does not have access to.

When an application tries to access a resource, and the userid is not authorised to that resource, RACF can produce an error message, which tells you the resource name.

If the application asks “does this application have access to this resource”, then RACF produces no error message, and it is up to the application to provide a sensible and useful message.

Collecting a RACF trace

You can use a command

#SET TRACE(CLASS(CSFSERV),RACROUTE(ALL))

to turn on the trace for that class. You can also use USERID(…) and jobname(…) to further restrict what is traced.

The output goes to GTF.

s gtf,gtf
01 AHL125A  RESPECIFY TRACE OPTIONS OR REPLY U 
 1,trace=usrp                                                                                 
IEE600I REPLY TO 01 IS;TRACE=USRP                                                              
    09.53.19 STC00315  TRACE=USRP                                                                                     
02 AHL101A  SPECIFY TRACE EVENT KEYWORDS --USR=                                                
  - 09.53.27           r 2,usr=(F44),end                                                                              
    09.53.27 STC00315  IEE600I REPLY TO 02 IS;USR=(F44),END                                                           
    09.53.27 STC00315  USR=(F44),END                                                                                  
    09.53.27 STC00315  AHL103I  TRACE OPTIONS SELECTED --USR=(F44)                                                    
  | 09.53.27 STC00315 *03 AHL125A  RESPECIFY TRACE OPTIONS OR REPLY U                                                 
00- 09.53.30           r 3,u                                                                                          
    09.53.30 STC00315  IEE600I REPLY TO 03 IS;U                                                                       
    09.53.30 STC00315  U                                                                                              

Run your work.

P GTF
AHL006I GTF ACKNOWLEDGES STOP COMMAND
AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA :
SYS1.TRACE

if you do not get “THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA…” it means you did not collect any data.

Use IPCS to look at it

  • =0 and specify the trace data set name
  • if you change scope to both it will remember the data for next time
  • =6 to get you to IPCS Subcommand Entry panel
  • if this is the first time you have used this instance of the data set, you should issue the dropd command to get IPCS to forget about previous usage
  • gtf usr(all) This displays the data
  • You can process this
    • type M and press PF8 to get to the bottom of the data
    • report view will display the data in ISPF edit (view mode)
    • You can now issue commands like
    • x all
    • f code all and look for non zero return codes.
    • del all x
    • sort 30 50 to display all return codes in numerical order. You need to look at the top and the bottom.
    • make a note of the return code ( copy the line to your clipboard)
    • quit
    • report view
    • find return code

To get rid of some of the forest of unhelpful data

  • x all
  • find ‘ ‘ 1 20 all
  • find ‘+’ 1 2 all
  • delete all nx

#SET TRACE(NOCLASS,RACROUTE(ALL))

Upgrade to Ubuntu 24.04 – whoops I’ve lost my network connections

I used remote access to upgrade my Ubuntu server. After the upgrade I could not connect over Ethernet. ( I could not access over Wi-Fi, but that was a different problem).

If I pinged from either end, it did not connect.

On my server, the network server for the network connection wired, did not have a MAC address specified. I used the pull down and selected it. I do not know if this is relevant.

My laptop had address 10.1.0.2, and the server had address 10.10.0.3.

On the server the command

ip addr

gave

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
inet 10.1.0.3/24 brd 10.1.0.255 scope global noprefixroute eno1

I used the command

sudo ip route add 10.1.0.2 dev eno1

to define the route, and connectivity was temporarily restored. I could ping from either machine.

Digging deeper

The command

grep -i warn /var/log/syslog

gave

NetworkManager[1457]: <warn>  [...] keyfile: load: "/etc/NetworkManager/system-connections/eno1": failed to load connection: File permissions (100604) are insecure

ls -ltr /etc/NetworkManager/system-connections/eno1

gave

-rw----r-- 1 root root 376 Oct 21 2022 /etc/NetworkManager/system-connections/eno1

Where file permission xxx604 is -rw—-r–, which allows “other” to read the file

The command

sudo chmod o-r /etc/NetworkManager/system-connections/*

removes the “other” read permission.The next reboot gave me connectivity back.

Lessons learned

After doing an upgraded, (or perhaps once a week) display the syslog for warn and error and resolve any problems.