Using ICSF in a multi system environment

Using ICSF within a single image or sysplex discussed setting up ICSF in a single environment, where systems can share the ICSF datasets.

This blog post is when you have multiple systems which do not share the ICSF data sets.

You have some encrypted data sets which you have sent to your backup system. How do you get the ICSF keys to your backup site? I will focus on using AES CIPHER keys because AES is recommended over DES keys; and CIPHER keys are recommended over DATA keys.

The challenge of using ICSF in a multi system environment where the ICSF data sets are not shared, is setting up the first keys. This is not easy as ICSF does not provide all the functionality to do this.

If your systems have the same hardware key, you can unload and reload a key using my programs in EASYICSF on Github. You unload the key (encrypted with the hardware key) to a data set, send the data set to the remote system, and reload the key.

If your systems do not use the same hardware key, you can fairly easily set up the keys for use between systems if you can accept some clear text key value sharing.

If your systems do not use the same hardware key, the most secure way of setting up keys is where the key value is never in clear text, and so cannot be copied or used in an authorised way. This has many steps, and I’ll cover this in another blog post (Generating ICSF keys using Diffie-Hellman).

If you want to encrypt a key to send to a remote site, ICSF provides Key Encryption Key (KEK) technology. Once they KEK has been set up you can export a key from the local system, encrypting it with an exporter KEK. Send the encrypted data to the remote site, and use the matching KEK to import it. At no point is the clear text version of the key exposed. I think of a KEK as an asymmetric key for data. Having the exporter KEK does not mean you can decrypt the data.

You need to have an exporter key to get from system A to system B, and an exporter key to get from system B to system A. If you specify the key value yourself these could be the same. If you let the system generate the key value, they will be different.

Once you have keys set up, you can use ICSF Key Encryption Key (KEK) to send keys securely to another system. The key phrase is “Once you have set up…”.

How long are keys needed for ?

A cipher key used in data set encryption is needed for as long as the data set using it is needed, so potentially many years. Deleting the key means the data set cannot be read.

When you create a KEK, you create both the exporter and importer at the same time. You cannot generate one from the other. A KEK is only used when transferring keys between systems. You could have a new KEK every week, with no impact. You just need to ensure you have at least another KEK to encrypt it.

How many KEK’s do I need?

If you have three systems, Production, Backup and Disaster Recovery (DR). You could define an exporter key on production, and the same importer key on Backup and DR. This might be easy to implement, but I think I would feel more comfortable with different KEKs:

Exporter on Production and Importer on Backup,
Exporter on Production and Importer on DR,
Backup to DR

and the opposite; Exporter on Backup and Importer on Production etc.

I included the system names in the data set used to store the key. When I initially used the same name for all definitions, such as ‘COLIN.EXPORTED.KEK’, I found I was getting into a muddle, and had the wrong key in it.

To export and import AES CIPHER keys see export a key from CKDS into a data set so it can be imported on another system and import a key from a data set into the CKDS on Github. You could just use unload and reload a key using my programs in EASYICSF on Github!

Overview to creating keys in a multi system environment

If the hardware keys are the same on each system, you can use UNLOAD and RELOAD programs in my EASYICSF on Github which allows you to copy the hardware encrypted key from one ICSF data set to an intermediate data set, then reload it into an ICSF data set. See here.
You can generate data set encryption keys on each system using a cut and paste of the JCL. This is not very secure because the key value is visible in clear text. See here.
Define Key Encrypting Keys for sending keys encrypted between systems. See here for detailed instructions.
- On the sender system define a temporary exporter key specifying the key value in clear text.
- On the receiving system(s) define the matching importer key using the same key values (cut and paste) and a small edit.
- Define a permanent exporter key letting the system securely generate the key value, and specify the temporary KEK to encrypt the importer key.
- Send the encrypted permanent importer KEK to the remote system.
- On the receiver systems use the temporary importer KEK to import the permanent key
- Delete the temporary KEK from each system.
- You now have KEKs on your system, and the window where there was clear text visible was small.
Locally generate a secure data set encryption key, and export it to the remote systems as needed. See here.
- You generate a data set encryption key.
- For each system you want to send this key to you export it, using the exporter KEK for that system.
- If you want to send the key at a later date, you just export the key using the appropriate KEK.
Generating secure data set encryption keys, both local and encrypted with a KEK
- ICSF has a utility CSFKGUP which, in one request, can generate a key for use locally, and export it using an exporter KEK.
- This does not feel a very good solution to me.
  - If you want to send the key to multiple systems, you need to use the same KEK for all systems.
  - If, at a later date, you want to send the key to another system you have to export it (see above). It is easier just to export it anyway.

These may look relatively simple – but ICSF does not provide a utility to export and import the keys. You can write your own C or REXX program, or use the ones I have written. See export a key and import a key in EasyICSF on Gitub.

Detailed instructions

Use unload and reload.

This is based on unload and reload a key using my programs in EASYICSF on Github.

Unload

//IBMUNL  JOB 'COLIN',CLASS=A,REGION=0M,COND=(4,LE) RESTART=RUN 
//JOBLIB JCLLIB ORDER=(COLIN.ICSF.C,CBC.SCCNPRC) 
//* COMPILE    EXEC  CCPROC,PROG=UNLOAD 
//* 
//* 
// SET KEY='-key   SECKEY2         ' 
// SET DB='-debug 8      ' 
// SET DB='              ' 
//RUN      EXEC PGM=UNLOAD,REGION=0M, 
//   PARM='&KEY.  &DB         ' 
//STEPLIB  DD DISP=SHR,DSN=COLIN.ICSFLOAD 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=* 
//KEY      DD DSN=COLIN.UNLOAD.AES,DISP=(OLD) 
//*        SPACE=(CYL,(1,1)),DCB=(LRECL=7000,RECFM=V)

Reload

//IBMRELOA JOB 'COLIN',CLASS=A,REGION=0M,COND=(4,LE) RESTART=RUN 
//JOBLIB JCLLIB ORDER=(COLIN.ICSF.C,CBC.SCCNPRC) 
//*OMPILE    EXEC  CCPROC,PROG=RELOAD 
//* 
//* 
// SET DB='-debug 8      ' 
// SET KEY='-key   SECKEY2         ' 
//RUN      EXEC PGM=RELOAD,REGION=0M, 
//   PARM='&KEY. &DB         ' 
//STEPLIB  DD DISP=SHR,DSN=COLIN.ICSFLOAD 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=* 
//KEY      DD DSN=COLIN.UNLOAD.AES,DISP=(OLD)

An easy, not very secure method, for use in a proof of concept.

You can use JCL Like

//IBMICSF JOB 1,MSGCLASS=H
//STEP10 EXEC PGM=CSFKGUP
// SET CKDS=COLIN.SCSFCKDS
//CSFCKDS DD DISP=OLD,DSN=&CKDS
//CSFIN DD *,LRECL=80
ADD TYPE(CIPHER) ALGORITHM(AES), 
   KEYMGT(XPRTCPAC), 
   CLEAR, 
   KEY(0123456789ABCDEF,0123456789ABCDEF,
       0123456789ABCDEF,0123456789ABCDEF), 
   LABEL(COLINCIPHERCLEAR ) , 
   KEYUSAGE(ANY-MODE) 
/*
//CSFDIAG DD SYSOUT=,LRECL=133 //CSFKEYS DD SYSOUT=,LRECL=1044
//CSFSTMNT DD SYSOUT=,LRECL=80 // REFRESH THE IN MEMORY DATA
//REFRESH EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

You run this on every system where you need the data set encryption key.

You need to specify for data se

KEYMGT(XPRTCPAC),
KEYUSAGE(ANY-MODE)
KEY(….) You need to pick a set of random hexadecimal numbers.

You can run this job on all systems you need the data set encryption keys on, with no changes.

This is not secure because other people may be able to see the keys…

You may have saved the JCL containing the key value (for future use) either in a data set or on paper
If you ran this as a job, the values may be in the spool output. Depending on how you specified the output class.
Copying the values to the remote system may get intercepted.

Defining KEKs

If you want to encrypt a key to send to a remote site ICSF provides Key Encryption Key (KEK) technology. Once the KEK has been set up you can export a key from the local system, encrypting it with an exporter KEK. Send the encrypted data to the remote site, and use the matching KEK to import it. At no point is the clear text version of the key exposed.

Generate a bootstrap KEK.

This bootstrap KEK is only used to set up a permanent KEK.

In a similar way to defining the data set encryption key above you can use

//IBMICSF  JOB 1,MSGCLASS=H 
//STEP10 EXEC PGM=CSFKGUP 
//  SET CKDS=COLIN.SCSFCKDS 
//CSFCKDS DD DISP=OLD,DSN=&CKDS 
//CSFIN DD *,LRECL=80 
DELETE TYPE(EXPORTER) LABEL(TEMPATOB) 
ADD   LABEL(TEMPATOB)TYPE(EXPORTER)  CLEAR , 
 ALGORITHM(AES), 
 KEY(3123456789ABCDEF,0123456789ABCDEF) 
/* 
//CSFDIAG DD SYSOUT=*,LRECL=133 
//* REFRESH THE IN MEMORY DATA 
//REFRESH  EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

On the remote system, use the same JCL except with TYPE(IMPORTER). You will, of course, use different hex values in the key to those I gave.

Generate the long term KEK.

//IBMICSF  JOB 1,MSGCLASS=H 
//STEP10 EXEC PGM=CSFKGUP 
//  SET CKDS=COLIN.SCSFCKDS 
//CSFCKDS DD DISP=OLD,DSN=&CKDS 
//* LENGTH(32) GENERATES A 256 BIT KEY 
//CSFIN DD *,LRECL=80 
ADD   LABEL(ATOBPERM)  TYPE(EXPORTER)        , 
  TRANSKEY(TEMPATOB)  , 
  ALGORITHM(AES)  LENGTH(32) 
/* 
//CSFDIAG DD SYSOUT=*,LRECL=133 
//CSFKEYS DD DSN=COLIN.CSFKEYS,DISP=OLD
//CSFSTMNT DD SYSOUT=*,LRECL=80  
//* REFRESH THE IN MEMORY DATA 
//REFRESH  EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

This creates a key value, stores the exporter KEK in the local CKDS encrypted with the hardware keys. It encrypts the new importer KEK with the TEMPATOB KEK and stores it in the CSFKEYS dataset.

The layout of this CSFKEYS data set is:

ATOBPERM the name of the object,
IMPORTER you need this sort of key to import it,
ATOB this is the KEK name used to encrypt it
The key for ATOBPERM with the key value encrypted. This is a hexadecimal value.

Knowing the information in the CSFKEYS data set you can import the key into the CKDS on the remote system, using the TEMPATOB importer key.

ICSF does not provide a utility for this. You can write some REXX code, or you can use my programs. See import a key from a data set into the CKDS.

Delete the temporary KEK

You can use the JCL

//IBMICSF  JOB 1,MSGCLASS=H 
//STEP10 EXEC PGM=CSFKGUP 
//  SET CKDS=COLIN.SCSFCKDS 
//CSFCKDS DD DISP=OLD,DSN=&CKDS 
//CSFIN DD *,LRECL=80 
DELETE TYPE(EXPORTER) LABEL(TEMPATOB) 
/* 
//CSFDIAG DD SYSOUT=*,LRECL=133 
//CSFKEYS DD DUMMY 
//CSFSTMNT DD DUMMY 
//* REFRESH THE IN MEMORY DATA 
//REFRESH  EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

Locally generate a secure data set encryption key

If you do not have a data set encryption key, use JCL like

Securely generate data set encryption keys.

//IBMDSENC JOB 1,MSGCLASS=H 
//STEP0  EXEC PGM=CSFKGUP 
//* 
//  SET CKDS=COLIN.SCSFCKDS 
//CSFCKDS DD DISP=OLD,DSN=&CKDS 
//* LENGTH(32) GENERATES A 256 BIT KEY 
//CSFIN DD *,LRECL=80 
ADD TYPE(CIPHER) ALGORITHM(AES) LENGTH(32), 
  LABEL(DSENCRYPTION), 
  KEYMGT(XPRTCPAC), 
  KEYUSAGE(ANY-MODE) 
/*
//CSFDIAG DD SYSOUT=*,LRECL=133 
//CSFKEYS DD DUMMY
//CSFSTMNT DD DUMMY 
//* REFRESH THE IN MEMORY DATA 
//REFRESH  EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH' 
//

You can then send the CSFKEYS data set to the remote site, and import it using the ATOBPERM importer key. See import a key from a data set into the CKDS on Github.

Using keys across systems

You can only use keys which are within their validity dates: crypto-period start date and
cypto-period end date.

When you import a key, meta data from the original site is not included, so you might need to specify start date, stop date, archive flags etc. See set meta data attributes on Github.

Generating ICSF keys using Diffie-Hellman

The Diffie-Hellman technique allows you to create keys on two system securely, with no need to share a common key. Each system needs a private/public key pair, and the public keys are available on all systems. There are many steps but it is relatively painless.

Scenario

Two systems A and B. We want to have a data set encryption key on system A copied to system B and to create an exporter key on A, and a matching importer key on B.

The steps are

generate private/public pairs on each system
use these to generate the symmetric keys.

Generate a private/public key pair on each system, and send the public key to the other system.

On System A

Use the ISPF ICSF panels

Options 5;6;6 Generate PKA keys, import or export public keys via certificate
- Enter PKDS record’s label: SYSTEMA
- Select Generate a new RSA or EC key pair record
- Select EC NIST Curve, P521.
- This should display PKDS Key Request Successful. Press enter to return.
- Select Export the PKDS record’s public key to a certificate data set
- Specify a data set name ‘COLIN.SYSTEMA’
- Specify the common name SYSTEMA
- Press enter
- This should give you PKDS Public Key Export Successful
Send the data set ‘COLIN.SYSTEMA’ to the remote system.

On System B

Follow the instructions above to generate PKA keys.
- Label Name SYSTEMB, and use export data set name COLIN.SYSTEMB, etc. Check it has worked.
- Next, select Create a PKDS public key record from an input certificate
- Enter the name of the data set you copied across containing the public key for SYSTEMA
- Press enter. It should import the key
- F3 to return to the ICSF PDKS keys page.
- Select option 1, and press enter. Use the line command K to display the contents of the certificates. You should have the private key with sections “PRIVATE PUBLIC”, and the public key should have section “PUBLIC”
Send the data set ‘COLIN.SYSTEMB’ to the first system.

On System A

Read in the data set
Options 5;6;6 Generate PKA keys, import or export public keys via certificate
Enter PKDS record label SYSTEMB
Select Create a PKDS public key record from an input certificate
Enter the name of the data set COLIN.SYSTEMB.
When this has worked, display the contents of the PKDS as described above.

You now have a private/public key on each system, and each system has a copy of the other’s public key.

Generate a symmetric key on each system

Now you have the private/public key on each system, and the public key of the other system, you can create a key.

ICSF does not provide a function for this. See Generate a secure shared key on multiple systems using Diffie-Hellman in Github.

On SYSTEMA you need the name of the private/public pair SYSTEMA, and the name of the public certificate SYSTEMB.
You need a phrase which provides a seed to the encryption. This does not need to be kept confidential, but you may still wish to protect it.

Generate a CIPHER for data set encryption

//RUN      EXEC PGM=GENDH,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD * 
 -ptype INTERNAL,AES,CIPHER,XPRTCPAC,ANY-MODE 
 -key AESDHDSCIPHER 
 -private AAA 
 -public  BBB 
 -replace Y 
 -party cOlinSeed 
 -debug 0 
/* 
//STEPLIB  DD DISP=SHR,DSN=COLIN.LOAD 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200) 
//CEEDUMP  DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=* 
//SYSERR   DD SYSOUT=*

On the other system change AAA and BBB, and run the JCL.

Create an exporter key on SYSTEMA

//RUN      EXEC PGM=GENDH,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD * 
 -ptype INTERNAL,AES,EXPORTER 
 -key AESDHKEK
 -private AAA 
 -public  BBB 
 -replace Y 
 -party cOlinSeed2 
 -debug 0 
/*  
//STEPLIB  DD DISP=SHR,DSN=COLIN.LOAD 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200) 
//CEEDUMP  DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=* 
//SYSERR   DD SYSOUT=*

Create a matching importer key on SYSTEMB

//RUN      EXEC PGM=GENDH,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD * 
 -ptype INTERNAL,AES,IMPORTER 
 -key AESDHKEK
 -private BBB 
 -public  AAA 
 -replace Y 
 -party cOlinSeed2 
 -debug 0 
/* 
//STEPLIB  DD DISP=SHR,DSN=COLIN.LOAD 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200) 
//CEEDUMP  DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=* 
//SYSERR   DD SYSOUT=*

You can now use exporter key AESDHKEK on system A, and importer KEK AESDHKEK on system B.

Idiot’s guide to TCPIP connectivity

I had a working TCPIP network, and made a few “improvements”. Unfortunately these improvements sometimes stopped the connectivity between systems, and I had a frustrating time understanding the problems and fixing them. The idiot in the blog post is me, for next time when I need to connect boxes together.

In concept TCPIP connectivity is simple – it is, but there are some subtle, non obvious things you need to be aware of.

As I was writing this post I found I did not know really how IPV4 worked, because it used “the wrong” IP address but still worked.

I found many ways of failing to connect to TCPIP, and some complex ways of getting it to work – I just wanted a simple way of being able to ping z/OS from my laptop. It is complicated by some definitions need to be done in order, and doing things in a different order sometimes worked, sometimes did not.

Basic TCPIP concepts that every one should know

The term socket is used by applications to communicate with TCP/IP, not where you connect a network/phone cable.
Think of a connection between two boxes. I have a yellow Ethernet cable between them. There are several terms for the where the cable is plugged in. A common term is the interface.
IP addresses
- Each end of the connection has one or more IP addresses. I think of it as having plastic labels tied to the end of the cable.
- IPV6 addresses beginning with fe… and ff… are used by (internal use) advanced technology and can be ignored. You can use them, but the addresses may change every time the connection is started, which makes it hard to automate using them.
- The system may generate some IPV6 addresses, but you can define your own. The system generated an address like 2a00:9999:8888:7777:894e:9876:781:32f1. Sometimes parts of these (the right hand part) are randomised (to make it harder for people to observer traffic patterns and so hack your system).
- I use addresses like 2001:db8::f which are shorter to type.
- On z/OS an IPV4 interface can have only one IP address. An IPV6 interface can have multiple addresses see ADDADDR. On z/OS an interface can be IPV4 or IPV6 but not both.
- On Linux, an interface can have multiple IPV4, and multiple IPV6 addresses (but only the first IPV4 may be visible to applications)
- For IPV6, TCP/IP can generate its own IPV6 addresses for internal processing, such as routing.
To get data from this machine to that machine over the yellow Ethernet cable, you have a route definition like “for this range of remote addresses use the yellow Ethernet cable, which has the address xxxx at the far end.
If you use TCP/IP to send a request, you usually want a response to come back. As well as defining a route to get to the remote end, you need a route defined to get from the remote machine back to the local machine. A ping request can fail because
- The local end does not have a valid route to the remote end. The packet could be sent to the wrong place(down the wrong cable), or just discarded.
- An intermediate box does not have a route to the remote end.
- The remote end receives the request but does not have a route definition to send the response back to the requester.
- An intermediate box does not have a route to the local end.
- A firewall says no.
- You can use the traceroute command to find the path taken to the remote end. This will tell you the path it took to get there. It does not tell you the route back. For this you need to issue the traceroute command on the remote end, and perhaps on intermediate boxes.
You define a route from this box using the yellow cable with label xxxx on it. The remote end of the cable has IP address….
You need at least two route statements
- to get the data from the local system to the remote system,
- the remote system needs a route statement to get to the local system.
You can find these address using
- the Linux command ip -6 addr or ip -4 addr for TCP IPV6 and IPV4 respectively.
- the z/OS command TSO NETSTAT HOME
Subnet: an IP V6 address has 32 hex digits. These are broken up into groups of 16 eg 2001:0DB8… This can be written as 2001:db8… The subnet specified which bits are significant when routing packets to the router. With z/OS usually the top 64 bits are used. This is written as …./64.
An address 2001:db8:9::1/64 is in a different subnet to address 2001:db8:8::1/64.
Address 2001:db8:8:1::2/64 is in the same subnet as 2001:db8:8:1::3 because only the top 64 bits count towards the subnet (2001:db8:8:1).
A gateway is a network point that acts as an entrance to another network. On the Internet, a node or stopping point can be either a gateway node or a host (end-point) node. Both the computers of Internet users and the computers that serve pages to users are host nodes. A gateway can have one protocol in, and output the data in a different protocol. For example I have broadband coming to my house. The gateway router converts this to TCP/IP, and converts it to wireless.

Things that you may not know

My end of a connection has two IP addresses defined. If I ping a remote site it uses the first IP address in its list, the remote site sees a packet of data from the first IP address in the list. You may have configured a route at the remote system to get back to your local system, but if you define your local addresses in a different order, a different IP address will be sent – and the remote end may not have a route for it.
If the interface at the next machine has two IP addresses 10.1.0.3 and 7.168.1.2 , I have to use the first IP address in the list defining a route sudo ip -4 route add 7.168.1.74 via 10.1.0.3 dev enp0s31f6. If I delete the first address, then I need to use the 7.168.1.2

Python could not read a data set I sent from z/OS USS.

I created a file in Unix System Services, and FTPed it down to my Linux box. I could edit it, and process it with no problems, until I came to read in the file using Python.

Python gave me

File “<frozen codecs>”, line 322, in decode
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xb8 in position 3996: invalid start byte

The Linux command file pagentn.txt gave me

pagentn.txt: ISO-8859 text

whereas other files had ASCII text.

I changed my Python program to have

with open(“/home/colinpaice/python/pagentn.txt”,encoding=”ISO-8859-1″) as file:

and it worked!

I browsed the web, and found a Python way of finding the code page of a file

import chardet    
rawdata = open(infile, 'rb').read()
result = chardet.detect(rawdata)
charenc = result['encoding']

it returned a dict with

result {‘encoding’: ‘ISO-8859-1’, ‘confidence’: 0.73, ‘language’: ”}

Why does it take an application using TCPIP, so long to start?

I had problems with a couple of applications taking over 30 seconds to start. For example FTP and the RMF DDS Server.

I found this was cause by my TCPIP resolver was misconfigured. An application can ask DNS for the IP address(10.1.1.2) or the string address (BBC.CO.UK) from the TCPIP Resolver function. On my system this was configured in ADCD.Z31A.TCPPARMS(GBLTDATA) as

LOOKUP DNS LOCAL

This says, go to the network, and ask the DNS server “out there” for information. If this request times out, use the local information. On my system the path to the DNS server was not configured, so it waited, and eventually timed out.

When I changed the LOOKUP definition to

LOOKUP LOCAL

it came up with no delays.