Making diff even better

I was using diff option -y  on Linux to compare two files side by side,  but it did not display the fill width of the window.

diff -W $(( $(tput cols) - 2 )) -y file1 file2 |less

solved it.

It went from being like

javax.net.ssl|AL   |    javax.net.ssl|DE
javax.net.ssl|IN   |    javax.net.ssl|DE
javax.net.ssl|DE   <
javax.net.ssl|DE   <
javax.net.ssl|DE   <
"ClientHello": {        "ClientHello": {

to


javax.net.ssl|ALL|01|main|2021-01-29 16:39:00.297 GMT|Si   |    javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:15.405 GMT|
javax.net.ssl|INFO|01|main|2021-01-29 16:39:00.297 GMT|A   |    javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:15.407 GMT|
javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:00.297 GMT|   <
javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:00.298 GMT|   <
javax.net.ssl|DEBUG|01|main|2021-01-29 16:39:00.299 GMT|   <
"ClientHello": {                                                "ClientHello": {
  "client version"      : "TLSv1.2",                              "client version"      : "TLSv1.2",

diff -W $(( 120))…. used a terminal width of 120 or each section 60 wide.

Finding the nugget of useful information in a TLS(SSL) trace

Understanding the TLS trace, for example trying to get a client to use a web server over TLS, is a difficulty-squared problem.

In this post, I give some tools to reduce the amount of trace data, and provide and annotated trace so you can understand what is going on, and spot the errors.   I’ve also annotated the trace with common user errors, and links to possible error causes.

 This post, and the referenced pages are still work in progress while I sort out some of the little problems that creep in.  Please send me comments on any mistakes or suggestions for improvements (or additional reasons why a handshake fails).

Understanding the TLS trace is hard.

It is hard enough to understand what the trace is showing, but it is made even more difficult to use.

  1. Where there are concurrent threads running; the trace records are interleaved, and it can be hard to tell which data belongs to which thread.
  2. The formatting is sometimes poor. Instead of giving one line with a list of 50 comma separated numbers, it gives you 100 lines with either a number, or a comma.  And when you have two threads doing this, it is a nightmare trying to work out what data belongs to which thread.  (But I usually ignore these records).
  3. The trace tends to be “provide all information that might possibly be useful”, rather than provide information needed by most people to resolve why a client cannot connect to the server.  For example the trace gives a print out of the encrypted data – in hex!
  4. You turn the trace on with a java -D… parameter.  Other applications, such as web servers, have different ways of turning the trace on.    I could not find a way of turning it on and off dynamically so you can get a lot of output if you have to run for a long period.  The output may go into trace files which wrap.
  5. Different implementations have  slightly different trace formats.

All these factors made it very difficult to understand the trace and find the cause of your problems.

What can you do to understand it.

This page from Java walks through a trace, but I dont find it very helpful.  This one which also covers TLS V1.3 is better.

Do not despair! 

  • On z/OS I created an edit macro, which I use to delete or transform data.  It reduces an 8000 line spool file down to 800 lines.  See ztrace.rexx.
  • On Linux I have a python script  which does the same. See tls.py.

In some traces sections are delimited by ***…. *** to make it easier to see the structure.

To find problems look for *** at the start of the line, or “exception”.

You may need to look at both ends of the trace to understand the problem.   One end may get a response “TLSv1.2 ALERT: warning, description = …” and you need to examine the other end to find the reason for the message.

Annotated trace file

I have taken a trace file from each end, and annotated then, so you can see what the flow is, and how and where the data is used.  I have colour coded some of the flows, and included some common errors (in red) , with a link to possible solutions.
Some lines have hover text.

If you have suggestions for additional information – or reasons why things do not work – please tell me and I’ll update the documentation.  

The trace is from a Linux client going to a Liberty on z/OS.

  1. Server starts up – and waits for a connection from a client
  2. Client starts up, and sends a “Client Hello” request to the server
  3. Server wakes up, processes the request
  4. Server sends “ServerHello” to the client
  5. Optional. If the server wants client authentication,  server send the “client Authentication request”
  6. *** ServerHelloDone. It has finished the processing, send the data and wait for the reply.
  7. Client wakes up and processes the “ServerHello”, optionally sends back the “CertificationResponse”, and sends verify.
  8. Servers processes the verify and ends the handshake.

 

 

If a .jks is not allowed, how do I create a .p12 for all of my certificates?

If you try to create a trust store with a format of Java Key Store you get a message saying you should not be using a propriety format. After some digging around, I have found out how to create a database with a format of pkcs12 (or .p12 as it is also known).

To make a PKCS12 keystore (including the private key)

You can use open ssl to create the two parts, (public and private), and get the public part signed.

You can then create a .p12entry using

password=”-passout file:password.file -passin file:password.file”
name=”colincert”
ca=”-CAfile capem -chain “

openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12  -name $name  $password $ca

To make a key store, importing the .p12s with the private keys,  use

rm mykey.p12
ks=” -importkeystore -destkeystore mytrust.p12″
dest=”-deststoretype pkcs12 -deststorepass password”
src=”-srcstoretype PKCS12 -srcstorepass password”

keytool $ks $src $dest -srckeystore ecec.p12 
keytool $ks $src $dest -srckeystore ss.p12
keytool -list -v -keystore mykey.p12 -storepass password

By default it uses the alias from the -name in the openssl command.

Using the -list command without the verbose option (-v) gives

keytool -list -keystore mykey.p12 -storepass password

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains … entries

ecec, 21-Jan-2021, PrivateKeyEntry,
Certificate fingerprint (SHA1): C3:3D:98:D1:45:DF:44:F0:13:FE:87:D9:3E:CC:22:33:A8:D6:24:91
mqweb, 21-Jan-2021, PrivateKeyEntry, 

So we can see this is a PKCS12 keystore with entries with a Private Key in them.

To make a trust store

A trust store only has the public parts, it does not have the private key, so creating an intermediate .p12 file does not work, you have to use the public .pem files.

rm mytrust.p12
ks=” -import -keystore mytrust.p12 -deststoretype pkcs12″
password=
-storepass:file password.file”

keytool $ks $password -file ecec.pem -alias ecec
keytool $ks  $password -file ss.pem -alias ss

keytool -list -v -keystore mytrust.p12 $password

This example uses the password stored in the file password.file, rather than specifying it in the command.

You have to supply the -alias, as it defaults to “mykey”.  If you add multiple entries with the default, you only get one entry in the keystore.

Using the -list command without the verbose option (-v) gives

keytool -list -keystore mytrust.p12 -storepass:file password.file
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

ecec, 21-Jan-2021, trustedCertEntry,
Certificate fingerprint (SHA1): C3:3D:98:D1:45:DF:44:F0:13:FE:87:D9:3E:CC:22:33:A8:D6:24:91
ss, 21-Jan-2021, trustedCertEntry,
Certificate fingerprint (SHA1): E9:D1:69:49:5B:D3:B4:3C:E8:44:5A:C1:2C:A2:D6:5D:FB:47:61:E7

So we can see this is a PKCS12 keystore, and has the expected entries which are Trusted Certificate Entries.

 

How do I used Linux to manage my corporate certificates?

Having used z/OS to be my corporate Certificate Authority I thought I would use Linux to be a corporate CA, and manage z/OS certificates.  For more information on Certificate Authorities, and signing on certificates see here.

Setting up your Corporate CA up on Linux

At the top of the CA certificate hierarchy is a self signed certificate.

Create the CA self signed certificate

openssl req -x509 -config openssl-ca.cnf -newkey rsa:4096 -days 3000 -nodes -subj “/C=GB/O=SSS/OU=CA/CN=SSCA8” -out cacert.pem -keyout cacert.key.pem -outform PEM  addext basicConstraints=”critical,CA:TRUE, pathlen:0″ -addext keyUsage=”keyCertSign, digitalSignature”

This creates a certificate with

  1. -x509 says make it self signed – so my enterprise master CA
  2. using 4096 rsa encryption
  3. a subject “/C=GB/O=SSS/OU=CA/CN=SSCA8”
  4. valid for 3000 days
  5. Not DES encryption ( -nodes) of the output files
  6. the who and public key are stored in cacert.pem
  7. the private key is stored in cacert.key.pem
  8. using format pem
  9. extra parameters CA:TRUE and keyusage…

Display it

openssl x509 -in cacert.pem -text -noout|less

Create a personal  certificate on Linux and sign it.

Create a personal certificate on Linux  and get it signed byt the CA created above.

I set up a shell script to do the work

name=”adcdd”
subj=’-subj “/C=GB/O=cpwebuser/CN=adcdd” ‘
#Passwords are stored in a file called password.file
passwords=”-passin file:password.file -passout file:password.file”

#clean out the old foils
rm $name.key.pem

rm $name.csr
rm $name.pem

CA=”cacert”

# generate a private certificate using Elliptic curve and type secp256r1
openssl ecparam -name secp256r1 -genkey -noout -out $name.key.pem

#create a certificate signing request (CSR)
openssl req -new -key $name.key.pem -out $name.csr -outform PEM $subj $passwords

#sign it – or send it off to be signed. Get the $name.pem back from the request.  
openssl ca -config openssl-ca-user.cnf -md sha384 -out $name.pem -cert cacert.pem -keyfile cacert.key.pem -policy signing_policy -extensions signing_mqweb -md sha256 -infiles $name.csr

 

#Get the  *.pem file back, if required,  and merge the files to form the .p12 file
openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile $CA.pem -chain -name $name $passwords

I stored common information in configuration files, such as openssl-ca-user.cnf .  This had a section called signing_policy,  and signing_mqweb which had

[ signing_mqweb ]

keyUsage = digitalSignature
subjectAltName = DNS:localhost, IP:127.0.0.1
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

Use Linux to be the Certificate Authority for my z/OS RACDCERT certificates.

Create a certificate on z/OS.

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
 
RACDCERT ID(START1) DELETE(LABEL('MYCERTL'))
/* create the certificate - note it is not signed
RACDCERT ID(START1) GENCERT -
SUBJECTSDN(CN('MYCERTL') -
O('SSS') -
OU('SSS')) -
ALTNAME(IP(10.1.1.2) -
DOMAIN('WWW.ME2.COM') )-
SIZE(4096) -
RSA -
WITHLABEL('MYCERTL')

/* convert this to a certificate request and output it
RACDCERT GENREQ (LABEL('MYCERTL')) ID(START1) -
DSN('IBMUSER.CERT.MYCERTL.CSR')
/

The first line of the data set is —–BEGIN NEW CERTIFICATE REQUEST—– which is what I expect for a Certificate Signing Request.

FTP this down to the Linux machine as mycertl.csr and use the openssl ca command.  This uses the cacert.*.pem files created above

openssl ca -config openssl-ca-user.cnf -md sha384 -out mycertl.pem -notext -cert cacert.pem -keyfile cacert.key.pem -policy signing_policy -extensions signing_mqweb -md sha256 -infiles mycertl.csr

Note: My openssl-ca-user.cnf  is given below.

I carefully checked the details displayed, and replied y to both questions.

This produced

Using configuration from openssl-ca-user.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
organizationName :PRINTABLE:'SSS'
organizationalUnitName:PRINTABLE:'SSS'
commonName :PRINTABLE:'MYCERTL'
Certificate is to be certified until Oct 14 15:28:45 2023 GMT (1000 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

If the option -notext is not specified, the output file contains the readable interpretation of the certificate.  Specify -notext to get the output file where the first line is —–BEGIN CERTIFICATE—–

Upload this output file to z/OS (for example “put mycertl.pem ‘IBMUSER.CERT.MYCERTL.PEM’ ” ).

Check the contents before you add it to the RACF keystore

RACDCERT CHECKCERT(‘IBMUSER.CERT.MYCERTL.PEM’)

Add the certificate to the keystore

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT ADD('IBMUSER.CERT.MYCERTL.PEM') -
ID(START1) WITHLABEL('MYCERTL)
/*

The command racdcert list (label(‘MYCERTL’))  id(start1) shows the certificate has NOTRUST, so will not be visible on any keyring. You need

RACDCERT id(START1) ALTER(LABEL(MYCERTL’))TRUST

and will need to connect it to any keyrings.

Upload the CA certificate into the RACF database.

You will also need to upload the public key to the RACF database as a CERTAUTH, and connect it to any ring that uses a certificate signed by the Linux CA.

FTP the certificate file, cacert.pem (created above), to z/OS as text.  Once you have FTPed the file, check the first line is “—–BEGIN CERTIFICATE—–“

Add it to RACF

You can add the certificate owned by a userid ( rather than certauth).   The certificate needs to be connected to the keyring as usage(CERTAUTH).

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
/* delete it if needed
RACDCERT DELETE (LABEL('Linux-CA256')) ID(START1)

RACDCERT id(start1) ADD('IBMUSER.CA256.PEM') -
WITHLABEL('Linux-CA256') TRUST

RACDCERT CONNECT(id(start1) LABEL('Linux-CA256') -
RING(TRUST) USAGE(CERTAUTH)) ID(START1)

RACDCERT CONNECT(id(start1) LABEL('Linux-CA256') -
RING(DANRING) USAGE(CERTAUTH)) ID(START1)

RACDCERT LISTRING(TRUST ) ID(START1)

racdcert list (label('Linux-CA256')) id(start1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

 

My openssl-ca-user.cnf file.

HOME = .
RANDFILE = $ENV::HOME/.rnd

####################################################################
[ ca ]
default_ca = CA_default # The default ca section

[ CA_default ]
default_days = 1000 # How long to certify for
default_crl_days = 30 # How long before next CRL

default_md = sha256 # Use public key default MD
preserve = no # Keep passed DN ordering

x509_extensions = ca_extensions # The extensions to add to the cert

email_in_dn = no # Don’t concat the email in the DN
copy_extensions = copy # Required to copy SANs from CSR to cert

base_dir = .
certificate = $base_dir/cacert.pem # The CA certifcate
private_key = $base_dir/cakey.pem # The CA private key
new_certs_dir = $base_dir # Location for new certs after signing
database = $base_dir/index.txt # Database index file
serial = $base_dir/serial.txt # The current serial number

unique_subject = no # Set to ‘no’ to allow creation of
# several certificates with same subject.

[ signing_policy ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

##########

[ signing_mqweb ]

subjectAltName = DNS:localhost, IP:127.0.0.1
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

 

Useful linux commands

Someone told me about a useful linux command, for using z/OS… so I thought I would pass on some one-liners to help other people.

sshfs colin@10.1.1.2: ~/mountpointz

This mounts a remote file system to give local access to files on a remote box.  This allows you to use gedit remotely, including z/OS files in USS.  For example gedit mountpointz/mqweb/servers/mqweb/mqwebuser.xml .

For z/OS the files must be tagged.   Use  chtag -p name to see the tag.  For me it needs a tag of  “t ISO8859-1 T=on” .  Without the tag it display the file in the wrong code page.

parcellite

This keeps a history of your clipboard, with hot keys.

Linux settings -> keyboard mapping

Ctrl+1 give me the x3270 with “tso@” in the title.   The command executed is wmctrl -a tso@
Ctrl+2 give me the x3270 with “colin@” in the title.   The command executed is wmctrl -a colin@

I start the x3270 with

x3270 -model 5 colin@localhost:3270 &
x3270 -model 5 tso@localhost:3270 &

model 5 makes the screen 132 characters wide by 80 deep

Map the 3270 keyboard to define the uss escape key

See x3270 – where’s the money key?

Gedit hot keys

See here

  • Find a string Ctrl-F , next string Ctrl-G, previous Ctrl-Shift-G
  • Top Ctrl-home, bottom Ctrl-End
  • Move left/right one word Alt- <- or Alt–>
  • Start/end of line Home/end or  Ctrl-pgup,Ctrl-pgdn
  • Next session Ctrl-Alt Pgup, Ctrl-Alt-Pgdn
  • Create new gedit window Ctrl-N
  • Create new tab Ctrl-T

 

How do I used z/OS to manage my corporate certificates?

With my first, (and second) reading of the RACF documentation it looked like z/OS could not act as my corporate Certificate Authority.  I was written up a blog post saying how to do it using Linux as my CA, when I found a couple of hints to show that z/OS could be used as a certificate authority.

The short answer on how to do it.

Create a end user certificate on z/OS

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('MYCERT') - 
             O('SS') - 
             OU('SS')) - 
   ALTNAME(IP(10.1.1.2) - 
           DOMAIN('WWW.ME.COM') )- 
   SIZE(4096) - 
   SIGNWITH (CERTAUTH LABEL('MY-CA1')) - 
   RSA - 
   WITHLABEL('MYCERT')

This creates a certificate START1.MYCERT signed by CERTAUTH.MY-CA1

How to process a certificate created on Linux

I created a certificate on Linux in a script (so I could use a variable as part of the file name).

name=”ec2″
passwords=
“-passin file:password.file -passout file:password.file
# generate a private key in rsa format
openssl genrsa -out $name.key.pem 2048
# make a certificate request to send off to get it signed
openssl req -config mqwebserver.config -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=mqwebec”  $passwords

This creates a Certificate Server Request file ec2.csr, and a private key in ec2.key.pem.

I FTP’d the .csr file to z/OS.   This is a text file, where the first line is —–BEGIN CERTIFICATE REQUEST—– so you need to FTP in text format.

With RACF you can generate a certificate by passing the information in a data set, or by specifying it on the RACDCERT GENCERT command.

I used the following job to sign the certificate using Certificate Authority DAN-CA1, passing the information in from the .CSR data set I had previously uploaded.

//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT ID(START1) GENCERT('IBMUSER.IBMUSER.EC2.CSR') - 
   SIGNWITH (CERTAUTH LABEL('MY-CA1')) - 
   WITHLABEL('MYCERT3') 

RACDCERT LIST(LABEL('MYCERT3')) ID(START1)
 
RACDCERT ID(START1) EXPORT(LABEL('MYCERT3'))- 
   DSN('IBMUSER.CERT.MYCERT3.PEM') - 
   FORMAT(CERTB64) 

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 

This reads the data set specified in the GENCERT() command, and takes all of the certificate parameters from it.  It is signed by the CERTAUTH with label(‘MY-CA1’) and stored in the RACF database under id(START1) with label(MYCERT3).

THE EXPORT command exports the signed certificate to a dataset.

I then FTP’d the exported data set IBMUSER.CERT.MYCERT3.PEM back to Linux.

I merged the two parts back together to create a .p12 combined file with

openssl pkcs12 -export -inkey ec2.key.pem -in dancert3.pem -out mycert3.p12 -name mycert3 -passout file:password.file -passin file:password.file

and displayed the .p12 file with

pk12util -l mycert3.p12

This gave me

Certificate(has private key):
Data:
  Version: 3 (0x2)
  Serial Number: 4 (0x4)
  Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
  Issuer: "CN=MYCA1,OU=SSS,O=SSS"
  Validity:
    Not Before: Sun Jan 17 00:00:00 2021
    Not After : Mon Jan 17 23:59:59 2022
  Subject: "CN=mqwebec,O=cpwebuser,C=GB"
  Subject Public Key Info:
    Public Key Algorithm: PKCS #1 RSA Encryption
    RSA Public Key:
    Modulus:...

    Exponent: 65537 (0x10001)
  Signed Extensions:
    Name: Certificate Comment
      Comment: "Generated by the Security Server for z/OS (RACF)"

    Name: Certificate Subject Alt Name
      DNS name: "localhost"
      IP Address: 127.0.0.1

  Name: Certificate Key Usage
    Critical: True
    Usages: Digital Signature
    Non-Repudiation
    Key Encipherment

  Name: Certificate Subject Key ID
    Data:...

  Name: Certificate Authority Key Identifier
    Key ID:...

  Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
    Signature:...
    Fingerprint (SHA-256):...
    Fingerprint (SHA1):...Friendly Name: dancert3

  Key(shrouded):
  Friendly Name: mycert3

  Encryption algorithm: PKCS #12 V2 PBE With SHA-1 And 3KEY Triple DES-CBC

Note: I tried exporting the file on z/OS using FORMAT(PKCS7B64) instead of FORMAT(CERTB64).
The openssl pkcs12 -export command failed with

unable to load certificates
error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149:
error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:309:Type=X509_CINF
error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=cert_info, Type=X509
:error:0907400D:PEM routines:PEM_X509_INFO_read_bio:ASN1 lib:../crypto/pem/pem_info.c:196:

If you get these messages download it in the proper format.

The longer answer…

When setting up certificates you have several options

  1. Use a Self Signed certificate.  This offers very little security, as anyone can create a certificate one.   This should only be used for testing, though I think you should never use them, as you need to test with signed certificates because of the additional work that needs to be done.  Imagine conversation with your manager “Did you test this?”  “No, I tested something else”.
  2. Get a certificate from recognised authority.   You pay for each certificate you get.  As the authority’s certificate is generally available you can use your certificate as soon as you get it.
  3. Set up your own authority you create your own high level certificate such as CN=TOPLEVELCA O=MYORG, which in turn creates other certificates such as CN=TESTCA, O=MYORG, and CN=PRODUCTIONCA O=MYORG.  You can create as many certificate as you wish for example using the TESTCA for certificates that can be used in test.
  4. A combination of both, you get a High level Certificate Authority certificate signed by a recognised authority, and then used that  to set up your own certificates.

What does a “certificate” contain?

There are two parts

  1. The certificate identified the “who”, along with the public key.
  2. The private key used in encryption.

What does “signing”  mean?

You take a block of data, and do a calculation on it, for example a checksum.  The signer takes the checksum and encrypts it using its private key.  The original block of data then has the encrypted checksum, and the signers public key appended to it.

To check the signature,

  1. You compare the signers public key with your copy of it and check it matches.  If they match you decrypt the encrypted checksum.
  2. You perform the same calculation on the original payload and check your calculation with the decrypted version.  If they match you know the data is good.

Signing a certificate is doing this with a certificate as the block of data.

If you have a corporate CA certificate, and a departmental CA certificate, the departmental certificate has been signed by the corporate CA.  When you sign a block of data, it will have the departmental certificate, and the corporate certificate appended.

How does a Certificate Authority work?

At the top of the Certificate Authority hierarchy is a generally available self signed certificate with its public key.

  • I create a personal certificate with  “who” and the public key, and the private key.
  • I send the personal certificate with  “who” and the public key, known as a Certificate Signing Request(CSR) to my security department, who sign it and return it.
  • I merge the signed “who” and the public key, with the private key, and store it in my key store.

How to define a midrange self signed certificate and get it into z/OS

Although I disapprove of using self signed certificates, (it is like having a secure lock on one door, but you leave another door open).   Sometimes there is a need to use one on the journey to using signed certificates – just to get the environment working.

Create the certificate on Linux

On Linux, I set up a bash script to create the self signed certificate.  (I used a bash script so I can parametrize values, and “name” is used in many places.)

subj=’-subj /C=GB/O=SSS/CN=SS’
addext=’-addext keyUsage=keyCertSign,digitalSignature ‘
name=’ss’
passwords=”

#remove the old files for this cert
rm $name.pem $name.key.pem $name.p12

#create the  certificate and the private key
openssl req -x509 -newkey rsa:4096 -nodes -out $name.pem -keyout $name.key.pem $subj $addext

 

#Compine the two files into a p12 (PKCS12) file.
openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -name $name $passwords

#display it
openssl x509 -in $name.pem -text -noout|less

This creates the certificate file, the key file and a .p12 file to be used by my application.

FTP the certificate file, ss.pem, to z/OS as text.  Once you have FTPed the file, check the first line is “—–BEGIN CERTIFICATE—–“

Add it to RACF

 

//IBMRACF JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT DELETE (LABEL('MYSS')) ID(START1)
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

RACDCERT ADD('IBMUSER.MY.SS.PEM') -
ID(START1) WITHLABEL('MYSS')

/* connect it to START1.TRUSTRING
RACDCERT ID(START1) CONNECT(RING(TRUSTRING ) -
USAGE(CERTAUTH) -
ID(START1) LABEL('MYSS') )

racdcert list (label('MYSS')) CERTAUTH

racdcert listring(TRUSTRING) id(start1)

SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh
/*

The list command gave

Digital certificate information for START1:
Label: MYSS
Certificate ID: 2QiJmZmDhZmjgcTB1eLi
Status: TRUST
Start Date: 2021/01/18 14:14:38
End Date: 2021/02/17 14:14:38
Serial Number:
>73CB2B49D0E56BBEAA95EC8EA01FCCC549A97BA8<
Issuer's Name:
>CN=SS.O=SSS.C=GB<
Subject's Name:
>CN=SS.O=SSS.C=GB<
Signing Algorithm: sha256RSA
Key Type: RSA
Key Size: 4096
Private Key: NO
Ring Associations:
Ring Owner: START1
Ring:
>TRUSTRING<

You can see this does not have the private key.   The issuer’s name matches the Subject’s name, so this shows it is self signed.

Restart your web server to pick up the changes to the trust keyring.

Change your client to point to the .p12 file, and specify the password.

-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.keyStore=/home/colin/checkTLS/ss.p12
-Djavax.net.ssl.keyStoreType=pkcs12

Restart your client.

Other ways of creating your client certificate

Above I used

openssl req -x509 -newkey rsa:4096 -nodes -out $name.pem -keyout $name.key.pem $subj $addext

to create a request.   This is actually two operations

  1. Create a private key
  2. Create a certificate

For some combination of parameters it may be easier to do this in two steps

Create the private key.  This creates a private key using an elliptic curve.

openssl ecparam -name secp384r1 -genkey -noout -out $name.key.pem

Create the certificate using the private key

openssl req -config eccert.config -passin password -sha384 -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name $password

The power of exclude in ISPF edit

I hope most people know about the ISPF edit exclude facility.  You can issue commands like

X ALL /* exclude every thing */
F XXXX ALL
F YYYYY ALL WORD
DELETE ALL X /* delete everything which is excluded */
RESET
X ALL
F ZZZZ all
F AAAA all
FLIP /* make the excluded visible, and the displayed hidden */
CHANGE 'ABC' 'ZYX' ALL NX /* do not change hidden lines */

You can also exploit this in macros.  See ISREDIT XSTATUS.

/* rexx */ 
ADDRESS ISPEXEC
'ISREDIT MACRO'
"ISREDIT EXCLUDE 'AAA' ALL PREFIX" /* just like the command*/
"ISREDIT LOCATE .ZLAST" /* go to the last line */
"ISREDIT (r,c) = CURSOR" /* get the row and column number */
/* r has the row number of the last line */
do i = 1 to r
"ISREDIT (xnx) = XSTATUS "i
say "Line exclude status is" i xnx
"ISREDIT (line) = LINE " i /* extract the line */
if (...) then
"ISREDIT XSTATUS "i" = X" /* exclude it */
end

You can also see if a line has been changed this edit session:

"ISREDIT (status) = LINE_STATUS "i /* for line i */
if (substr(status,8,1) = '1') then
say "Line" i " was changed this session "

 

 

It is now so easy to have hide/show text in your html page. Auto-twisties are here.

Click me to show the rest of the information. Go on, click me

This text is hidden until the summary line is clicked.

 

This is done using

<details><summary> This is displayed</summary>this is hidden until the summary is clicked</details>

So easy and impressive isn’t it. You used to have to set up scripts etc to do this. Now it is built in. Of course,  you’ll need to format the summary so it looks clickable.  

Note: Another technique, the accordion, requires a java script for it to work. <details… works without java script.

You can also have hover test using <span title = “hovertext”>displayed text</span>.

How to backup only key data sets on z/OS

I’ve been backing up my datasets on z/OS, and wondered what the best way of doing it was.

I wanted to backup datasets containing data I wanted to keep, but did not want to backup other data sets which could easily be recreated, such as IPCS dump dataset, the output of compiles, or the SMF records.

DFDSS has a backup and restore program which is very powerful.  With it you can

  • Process data sets under a High Level Qualifier – include or exclude data sets.
  • Backup only changed data sets
  • Backup individual files in a ZFS or USS – but this is limited, you have to explicitly specify the files you want to backup.   You cannot backup a directory

You cannot backup individual members of a PDS(E).   You have to backup the whole PDS(E),   If you need to restore a member, restore the backup with a different HLQ and select the members from that.

What should I use?

I tend to use XMIT and DFDSS – the Storage Management component on z/OS. This tends to be used by the data managers as it can backup groups of data sets, volumes, etc..

Backing up using XMIT.

This has the advantage that the output file is a card image, which is a portable format.

I have a job

//MYLIBS1 JCLLIB ORDER=USER.Z24A.PROCLIB 
// SET TODAY='D201224'
//S1 EXEC PROC=BACKUP,P=USER.Z24A.PARMLIB,DD=&TODAY.
//S1 EXEC PROC=BACKUP,P=USER.Z24A.PROCLIB,DD=&TODAY.

Where

  • P is the name of the dataset
  • TODAY  – is where I set today’s date.

The backup procedure has

//BACKUP PROC P='USER.Z24A.PROCLIB',DD='UNKNOWN' 
//S1 EXEC PGM=IKJEFT01,REGION=0M,
// PARM='XMIT A.A DSN(''&P'') OUTDSN(''BACKUP.&P..&DD'')'
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
// PEND

The command that gets generated when P is USER.Z24A.PROCLIB and DD=D201224 is

XMIT A.A DSN('USER.Z24A.PROCLIB') OUTDSN('BACKUP.USER.Z24A.PROCLIB.D201224') 

This makes it easy to find the backups for a file, and a particular data.

To restore a file you use command TSO RECEIVE INDSN(‘BACKUP.USER.Z24A.PROCLIB.D201224’)  .

Using DFDSS to backup

This is a powerful program, and it is worth taking baby steps to understand it.

The basic job is

//IBMDFDSS JOB 1,MSGCLASS=H 
//S1 EXEC PGM=ADRDSSU,REGION=0M,PARM='TYPRUN=NORUN'
//TARGET DD DSN=COLIN.BACKUP.DFDSS,DISP=(MOD,CATLG),
// SPACE=(CYL,(50,50),RLSE)
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DUMP -
DATASET(INCLUDE(COLIN.JCL,COLIN.WLM,COLIN.C) -
BY(DSCHA,EQ,YES)) -
OUTDDNAME(TARGET) -
COMPRESS
/*
  • For the syntax of the dump data set command see here.
  • This dumps the specified data sets, COLIN.JCL. COLIN.WLM, COLIN.C, takes them and puts them in one file through TARGET.   TARGET is defined a dataset (COLIN.BACKUP.DFDSS).
  • This does not actually do the backup because it has TYPRUN=NORUN.
  • You can specify many filter criteria, in the BY(…) such as last reference, size, etc.  See here.
  • The BY(DSCHA,EQ,YES) says dump datasets only if they have the “changed flag” set.  The Changed flag is set when a data set was open for output.  Using ADRDSSU with the RESET option resets the changed flag.   This allows you to backup only data sets which have changed – see below.
  • It compresses the files as it backs up the files.

I did have

DATASET(INCLUDE(COLIN.**) - 
EXCLUDE(COLIN.O.**,COLIN.SMP*.**,COLIN.DDIR ) -
BY(DSCHA,EQ,YES)) -

Which says backup all data sets with the High Level Qualifier COLIN.**, but exclude the listed files.  I ran this using TYPRUN=NORUN, and this listed 100+ datasets.   Whoops, so I changed it to explicitly include the files I wanted to backup.  Once  I had determined the files I wanted to backup I removed the TYPRUN=NORUN, and backed up the datasets.

Using DFDSS to restore

You can restore from the DFDSS backups using a job like

//S1 EXEC PGM=ADRDSSU,REGION=0M,PARM='TYPRUN=NORUN' 
//TARGET DD DSN=COLIN.BACKUP.DFDSS,DISP=SHR
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
RESTORE -
DATASET(INCLUDE(COLIN.C) ) -
RENAME(COLINN) -
INDDNAME(TARGET)
/*

This says restore the files specified in the INCLUDE…  rename the HLQ to be COLINN.  From the dataset via //TARGET.

Initially I specified PARM=’TYPRUN=NORUN’ so it did not actually try to restore the files.   It reported

THE INPUT DUMP DATA SET BEING PROCESSED IS IN LOGICAL DATA SET FORMAT AND WAS CREATED BY Z/OS DFSMSDSS 
VERSION 2 RELEASE 4 MODIFICATION LEVEL 0 ON 2020.359 17:16:44
DATA SET COLINN.C WAS SELECTED
PROCESSING BYPASSED DUE TO NORUN OPTION
THE FOLLOWING DATA SETS WERE SUCCESSFULLY PROCESSED
COLIN.C

From the time stamp 2020.359 17:16:44 we can see I was using the expected backup.

Once you are happy you have the right backup, and list of data sets, you can remove the PARM=’TYPRUN=NORUN’ to restore the data.

If you have backed up COLIN.JCL, and SUE.JCL, and try to rename on restore ( so you do not overwrite existing files) it would fail because if would create COLINN.JCL and then try to create COLINN.JCL from the other file!   To get round this using INCLUDE(COLIN.**) RENAMEN(COLINN) and INCLUDE(SUE.*) renamen(SUEN) .

 

What’s in the backup?

You can use the following to list the contents  (with TYPRUN=NORUN)

RESTORE - 
DATASET(INCLUDE(**) ) -
INDDNAME(TARGET)

Note: that because this job does not have REPLACE, it will not overwrite any files.

Using advanced backup facilities.

Each dataset has a changed-flag associated with it.   If this bit is on, the data set has been changed.  You can display this in the data set section of ISMF.  Field 24 – CHG IND, or if you have access to the DCOLLECT output, it is in one of the flags.

If you use

DUMP - 
DATASET(INCLUDE(COLIN.JCL,COLIN.WLM,COLIN.C) -
BY(DSCHA,EQ,YES)) -
RESET -
OUTDDNAME(TARGET) -
COMPRESS

it will backup the data sets, and reset the changed flag.  In my case it backed up the 3 data sets I had specified.

When I reran the same job, it backup up NO data sets, giving me a message

ADR383W (001)-DTDSC(01), DATA SET COLIN.JCL NOT SELECTED, 01.
Where  01 means  The fully qualified data set name did not pass the INCLUDE, EXCLUDE, and/or BY filtering criteria.

This is because I had specified BY(DSCHA,EQ,YES)) which says filter by Data Sets with the CHAnge flag on (DSCSHA) flag on.  The first DUMP request RESET the flag, the second DUMP job skipped the data sets.

You can exploit this by backing up all data sets once a week, and just changed data sets during the week.

You might need to keep the output of the dump job in member of a PDS, so you can search for your dataset name to find the date when a backup was done which included the file.

How many backups should I keep?

This depends on if you are backing up all, or just changed files.  You can use GDG (see here) where you use a generation of dataset.  If you specify 3 generations, then when you create the 4th copy, it deletes copy 1 etc.