Month: April 2023

Setting up Linux to z/OS certificates

Several times I have had to set up certificates between Linux and z/OS and struggled for a day to get them working. Once you are familiar with doing it – it is easy. As the last time I needed to do this was over a year ago, I’ve forgotten some of the details. This blog post is to help me remember what I need to do, and to help other who struggle with this.

I’m ignoring self signed.

Basic TLS

A certificate contains

  • who it belongs to, such as CN=COLIN,O=SSS
  • the date range the certificate is valid
  • a public key
  • meta data about the key: What algorithm does the public key use, what parameters were used in the key generation, for example, algorithm=RSA, Keysize=2048.

There is a private key.

  • If you encrypt using the private key, you can use the public key to decrypt it.
  • If you encrypt using the public key, you can use the private key to decrypt it.
  • If you encrypt something with my public key, and then encrypt it with your private key. I know it came from you (or someone with your private key) and only I (or someone with my private key) can decrypt it.

Anyone can have the public key. You keep the private key secure.

Certificate Authority. This is used in validating the trust of certificates. You send your certificate to the CA, The CA does a checksum of your data, and encrypts this checksum with the CA private key. It returns your original data appended with the encrypted checksum, and information about the CA, and what was used to calculate the checksum. If someone else has the CA public key, they can do the opposite process. Do the checksum calculation, and decrypt the checksum value in the certificate, using the CA public key. If they match you know it was signed by the CA. This is known as signing the certificate.

To be able to validate a certificate sent to it, the client end needs the CA of the server end. The server needs the CA of the client end to be able to validate the client’s certificate.

During the handshake to establish the TLS connection there is a flow like

  • Establish the cipher spec to use
  • Server sends down its certificate, the client checks it
  • Servers sends down “Certificate request”, and these are the certificate(CAs) I know about
  • The client goes through it’s list of certificates (usually only one), to find the first certificate with a CA in the list sent from the server.
  • sends the client certificate to the server
    • The server checks the certificate. For example the server may be set up to accept a subset of valid algorithms, for example TLS 1.2, and Elliptic Curve. If a certificate is sent up using RSA, then this is not accepted
    • The server checks the signature of the certificate, finds the CA name, checks in the trust store for this CA, and validates the signature. Depending on the application it may check all the CA’s in the CA chain.

What do you need for the handshake to work

  • You need to have a Certificate Authority to sign certificates. In the CA certificate are some flags that say this is a CA.
  • You need to send the public key of each CA to the other end. You normally need to do this just once, and keep using the same certificates for all your TLS work.
  • You need to have a key store/trust store/keyring to hold certificates.
  • On z/OS
    • you may have a keyring for different projects, for example MQ, and TN3270.
    • You need to connect the client CA into each keyring where it will be used.
  • You need to check that the certificates are compatible with the remote end, such as Algorithm etc.

Openssl files

When using openssl, you can store common information in a configuration file. See here. This configuration file has some required options, and some optional options where you can specify common options you frequently use.

If you are using the openssl req command (for example), by default it will look for a section called [req]. This can in turn point to other sections. Using this file you can specify most of your fields in one place, and just override the specific ones.

Create a CA certificate on Linux

I have a bash file docca.sh file on Linux.

CA=”docca256″
casubj=” -subj /C=GB/O=DOC/OU=CA/CN=SSCA256″
days=”-days 1095″
rm $CA.pem $CA.key.pem

openssl ecparam -name prime256v1 -genkey -noout -out $CA.key.pem1

openssl req -x509 -sha384 -config caca.config -key $CA.key.pem2 -keyform pem -nodes $casubj -out $CA.pem3 -outform PEM $days

openssl x509 -in $CA.pem -text -noout|less4

This

  1. creates a private key (docca256.key.pem)
  2. self signs it. For any parameters not specified, it uses the configuration file caca.config and section “req” (signing request) within it.
  3. produces a public certificate in docca256.pem. This file will need to be sent to the backend servers. You can use cut and paste or FTP as ASCII.
  4. displays the x509 data

The caca.config file has

[ req ]
distinguished_name = ca_distinguished_name
x509_extensions = ca_extensions

prompt = no

authorityKeyIdentifier = keyid:always,issuer:always

[ca_distinguished_name ]
# C=GB
# O=DOC
# OU=Stromness
# CN=SSSCA4

####################################
[ ca_extensions ]

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
basicConstraints = critical,CA:TRUE, pathlen:0
keyUsage = keyCertSign, digitalSignature,cRLSign

The distinguished_name = ca_distinguished_name says go and look in the file for a section [ca_distinguished_name], and x509_extensions = ca_extensions says go and look for a section called [ca_extensions]. You can specify your own names, for example I could have used section1, and s2.

When prompt = yes, openssl takes as defaults the values in the distinguished_name section. When prompt = no, the distinguished_name is still required – but the contents of the section are ignored.

The values in the x509_extensions are defined here.

Creating an Elliptic Curve certificate on Linux

I used another bash script docecadad.sh, to document an ElliptiCal certificate for userid ADCD. It uses the CA defined above.

name="docecadcd"
key="$name.key.pem"
cert="$name.pem"
subj="-subj /C=GB/O=Doc/CN="$name
CA="docca256"
cafiles="-cert $CA.pem -keyfile $CA.key.pem "

enddate="-enddate 20240130164600Z"
passin="-passin file:password.file"
passout="-passout file:password.file"

rm $name.key.pem
rm $name.csr
rm $name.pem

#define a certificate with elliptic key with size 256

openssl ecparam -name prime256v1 -genkey -noout -out $name.key.pem 
#create a certificate request (ie hello CA please sign this)
openssl req -config openssl.config -new -key $key -out $name.csr -outform PEM -$subj $passin $passout

# sign it.

caconfig="-config ca2.config"
policy="-policy signing_policy"
extensions="-extensions clientServer"

md="-md sha384"

openssl ca $caconfig $policy $md $cafiles -out $cert -in $name.csr $enddate $extensions

# display it 
openssl x509 -in $name.pem -text -noout|less

Where the openssl.config file has 

[ req ]
default_bits       = 2048

distinguished_name = server_distinguished_name
req_extensions     = server_req_extensions
string_mask        = utf8only
subjectKeyIdentifier   = hash
#extendedKeyUsage     = critical, codeSigning


[ server_req_extensions ]

subjectKeyIdentifier = hash
# subjectAltName       = DNS:localhost, IP:127.0.0.1, IP:127.0.0.6
# nsComment            = "OpenSSL"
keyUsage             = critical, nonRepudiation, digitalSignature
# extendedKeyUsage     = critical, OCSPSigning, codeSigning
subjectKeyIdentifier   = hash 

[ server_distinguished_name ]
#c=GB
#o=SSS
#cn=mqweb
  • See above for the distinguished_name value.
  • req_extensions says use the section [server_req_extensions]

The ca2.config file used to sign it has 

HOME            = .
RANDFILE        = $ENV::HOME/.rnd

####################################################################
[ ca ]
default_ca    = CA_default      # The default ca section
####################################################################
[ CA_default ]
default_days     = 1000         # How long to certify for
default_crl_days = 30           # How long before next CRL
#default_md       = sha1       # Use public key default MD
default_md       = sha256       # Use public key default MD
preserve         = no           # Keep passed DN ordering

x509_extensions = ca_extensions # The extensions to add to the cert

email_in_dn     = no            # Don't concat the email in the DN
copy_extensions = copy          # Required to copy SANs from CSR to cert

#defaults
base_dir      = .
certificate   = $base_dir/cacert.pem   # The CA certifcate
private_key   = $base_dir/cakey.pem    # The CA private key
new_certs_dir = $base_dir              # Location for new certs after signing
database      = $base_dir/index.txt    # Database index file
serial        = $base_dir/serial.txt   # The current serial number

unique_subject = no  # Set to 'no' to allow creation of
                     # several certificates with same subject.


####################################################################
[ ca_extensions ]

subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always, issuer:always
basicConstraints       = critical,CA:TRUE, pathlen:0
keyUsage               = nonRepudiation

####################################################################

[ signing_policy ]
countryName            = optional
stateOrProvinceName    = optional
localityName           = optional
organizationName       = optional
organizationalUnitName = optional
commonName             = supplied

[ clientServer ]

keyUsage               = digitalSignature, keyAgreement, digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName         = DNS:localhost, IP:127.0.0.1, 
extendedKeyUsage       = serverAuth,clientAuth
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always, issuer:always
nsComment  = "clientserver"

The policy (my [ signing_policy] ) must have entries in it to create a valid Subject Distinguished name. Without it, I got a strange RACF code (0x0be8044d).

Send the CA to z/OS and import it

You need to send the CA public certificate to z/OS. This file looks like

-----BEGIN CERTIFICATE-----                                      
MIIFbTCCA1WgAwIBAgIUJw+gLLSFxqCyTdIyEUWyQ/g9JnEwDQYJKoZIhvcNAQEL.
...
-----END CERTIFICATE-----

You can FTP the file, or create the file and use cut and paste. The file needs to be a sequential dataset with format VB. My file is VB, lrecl=256,blksize=6233. For the FTP I used

put docca256.pem ‘colin.docca256.pem’

You need to import this into RACF, and connect it to the keyrings.

//IBMRACF  JOB 1,MSGCLASS=H                                     
//S1  EXEC PGM=IKJEFT01,REGION=0M                               
//SYSPRINT DD SYSOUT=*                                          
//SYSTSPRT DD SYSOUT=*                                          
//SYSTSIN DD *                                                  
RACDCERT CHECKCERT('COLIN.DOCCA256.PEM') 
/*

The CHECKCERT gave me 

Certificate 1:                                                          
                                                                        
  Start Date: 2022/10/09 11:45:43                                       
  End Date:   2025/10/08 11:45:43                                       
  Serial Number:                                                        
       >782A62948699FF3FB00238FB296E4A647B7DF07C<                       
  Issuer's Name:                                                        
       >CN=SSCA256.OU=CA.O=DOC.C=GB<                                    
  Subject's Name:                                                       
       >CN=SSCA256.OU=CA.O=DOC.C=GB<                                    
  Signing Algorithm: sha384ECDSA                                        
  Key Usage: HANDSHAKE, CERTSIGN                                        
  Key Type: NIST ECC                                                    
  Key Size: 256

Which matches what I expected, and gave me information about the certificate – ECC, 256, and signed with SHA 384 ECDSA, (from the -sha384 parameter above).

Define it to RACF and connect it to the user’s keyring

racdcert list  (label('Linux-CA256')) CERTAUTh 
RACDCERT DELETE  (LABEL('LINUXDOCCA256'))    CERTAUTH             
              
RACDCERT ADD('COLIN.DOCCA256.PEM') -                            
          CERTAUTH  WITHLABEL('LINUXDOCA256') TRUST 

RACDCERT ID(START1) CONNECT(RING(MQRING)-        
                            CERTAUTH     -       
                            LABEL('LINUXDOCCA256'))

If you delete a certificate, then it is removed from all keyrings. Once you have re-added it you need to reconnect it to all the keyrings. If you list the label (racdcert list (label(‘Linux-CA256’)) certauth) it will display where it is used, so you can read it.

Download the z/OS CA certificate

I downloaded the z/OS exported certificate in .pem format. it looks like

-----BEGIN CERTIFICATE-----                                       
MIIDYDCCAkigAwIBAgIBADANBgkqhkiG9w0BAQsFADAwMQ4wDAYDVQQKEwVDT0xJ  
... 
+9TRng==                                                          
-----END CERTIFICATE-----

You can use ftp or cut and paste. I created doczosca.pem.

Use it!

Before I could use it, I had to set up the server’s certificate, and download the z/OS CA certificate

I set up a bash script scl2.sh

name=docecadcd
host="-connect 10.1.1.2:4000"
CA="--CAfile doczosca.pem "

openssl s_client $host  $CA -cert $name.pem -certform PEM -key $name.key.pem -keyform PEM

Setting up z/OS for TLS clients

There is a lot of configuration needed when setting up TLS(SSL) between a server and a client. There are many options and it is easy to misconfigure. The diagnostic information you get when the TLS handshake fails is usually insufficient to identify any problems.

You need the following on z/OS:

  • One or more Certificate Authority certificates. You can create and use your own for testing. If you want to work with external sites you’ll need a proper (external) CA, but for validation and proof of concept you can create your own CA. You could set up a top level CA CN=CA,O=MYORG, and another one (signed by CA=CA,O=MYORG), called CN=CA,OU=TEST,O=MYORG. Either or both of the public CA certificates will need to be sent to the clients in imported into their keystore.
  • A private/public key, signed by a CA, (such as signed by CA=CA,OU=TEST,O=MYORG).
  • The private key is associated with a userid.
    • The signing operation takes the data (the public key), does a hash sum calculation on the data, encrypts this hash sum, and stores the encrypted hash value, and CA public certificate with the original data. To check the signature, the receiving application compares the CA with its local copy, if that matches, does the same checksum calculation, decodes the encrypted hash sum – and checks the decrypted and locally calculated values match.
    • A certificate is created using one from a list of algorithms. (For example, Elliptic Curve, RSA). When the certificate is sent to the client, the client needs to support the algorithm. Either end can be configured, for example, to support Elliptic Curve, but not RSA.
  • A keyring to contain your private key(s) – this can also contain CA public certificates of the partners (clients or servers).
  • A “site” keyring (public keystore, or trust ring) which holds the public CA certificates of all the other sites you work with. If you have only one keyring per user or application, you need to update each of them if you need to an a new CA to your environment. Many applications are only designed to work with one keyring. Java applications tend to have a key store(for the private key) and a trust store for the CAs.
  • Some applications can support more than one private certificate on a keyring. The certificate needs to match what the client can support.
  • For certificates which are sent to your server, you need a copy of the CA(s) used to sign the incoming certificate. If you have a copy of the CA, then you can validate any certificate that the CA signed. This means you do not have to have a copy of the public certificate of every client. You just need the CA.
    • Some application need access to just one CA in the chain, other applications need access to all certificates in the CA chain.
  • As part of the TLS handshake
    • the client sends up a list of the valid cipher specs it supports (which algorithms, and size of key)
    • the server sends down a subset of the list of cipher spec to use (from the client’s list)
    • the server can also send down its certificate, which contains information such as the distinguished name CN=zSERVER, OU=TEST, O= MYORG, and host name.
    • the client can validate these names – to make sure the host name in the certificate matches the host, and what it was expecting.
    • if requested, the client can send up its certificate for identification. The server can validate the certificate, and can optionally map it to a userid on the server.
  • A userid can be given permission to read certificate in another user’s keyring. A userid needs a higher level of authority to be able to access the private key in another id’s keyring.

Create the Certificate Authority

//IBMRACF  JOB 1,MSGCLASS=H                               
//S1  EXEC PGM=IKJEFT01,REGION=0M                         
//SYSPRINT DD SYSOUT=*                                    
//SYSTSPRT DD SYSOUT=*                                    
//SYSTSIN DD * 
RACDCERT certauth LIST(label('DOCZOSCA')) 
RACDCERT CERTAUTH DELETE(LABEL('DOCZOSCA'))               
RACDCERT GENCERT  -                                         
  CERTAUTH -                                                
  SUBJECTSDN(CN('DocZosCA')- 
             O('COLIN') -                                   
             OU('CA')) - 
  NOTAFTER(   DATE(2027-07-02  ))-                          
  KEYUSAGE(   CERTSIGN )  -      
  SIZE(2048) -                                              
  WITHLABEL('DOCZOSCA') 
/*
//

This certificate is created against “user” CERTAUTH. Keyusage CERTSIGN means it can be used to sign certificates. “user” CERTAUTH is often displayed internally as “irrcerta”.

Once it has been created the certificate should be connected to every ring that may use it, see below.

Export the CA certificate to a file so, clients can access it 

RACDCERT CERTAUTH EXPORT(LABEL('DOCZOSCA')) -
  DSN('IBMUSER.CERT.DOC.CA.PEM') -
  FORMAT(CERTB64) -
  PASSWORD('password')

The file looks like

-----BEGIN CERTIFICATE-----                                        
MIIDYDCCAkigAwIBAgIBADANBgkqhkiG9w0BAQsFADAwMQ4wDAYDVQQKEwVDT0xJ   
TjELMAkGA1UECxMCQ0ExETAPBgNVBAMTCERvY1pvc0NBMB4XDTIyMTAwOTAwMDAw 
...

This can be sent to the clients, so they can validate certificates sent from the server. This file could be sent using cut and paste, or FTP.

Create the keyring for user START1.

The instructions below lists the ring first – in case you need to know what it was before you deleted it”

RACDCERT LISTRING(TN3270)  ID(START1) 

RACDCERT DELRING(TN3270) ID(START1) 

RACDCERT ADDRING(TN3270) ID(START1)                                                          

RACDCERT LISTRING(TN3270)  ID(START1) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Connect the CA to every keyring that needs to use it

RACDCERT ID(START1) CONNECT(RING(TN3270) - 
                            CERTAUTH LABEL('DOCZOSCA'))

Create a user certificate and sign it on z/OS

This creates a certificate and gets is signed – as one operation. You can create a certificate, export it, sent it off to a remote CA, import it, and add it to a userid.

RACDCERT ID(START1) DELETE(LABEL('NISTECC521')) 
                                                                
RACDCERT ID(START1) GENCERT -                                   
  SUBJECTSDN(CN('10.1.1.2') - 
             O('NISTECC521') -                                  
             OU('SSS')) -                                       
   ALTNAME(IP(10.1.1.2))-                                       
   NISTECC - 
   KEYUSAGE(HANDSHAKE) - 
   SIZE(521) - 
   SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -                      
   WITHLABEL('NISTECC521')     -                                
                                                                
RACDCERT id(START1) ALTER(LABEL('NISTECC521'))TRUST             

RACDCERT ID(START1) CONNECT(RING(TN3270) -                      
                            ID(START1)  -                       
                            DEFAULT  - 
                            LABEL('NISTECC521') )               
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                    
RACDCERT LIST(LABEL('NISTECC521' )) ID(START1)                  
RACDCERT LISTRING(TN3270)  ID(START1)

This creates a certificate with type Elliptic Curve (NISTECC) with a key size of 521. It is signed with the CA certificate created above.

The ALTNAME, is a field the client can verify that the Source Name in the certificate matches the IP address of the connection.

It is connected to the user’s keyring as the DEFAULT. The default certificate is used if the label of a certificate is not specified when using the keyring.

Give a user access to the keyring

PERMIT START1.TN3270.LST CLASS(RDATALIB)  -    
    ID(COLIN )  ACCESS(UPDATE )                          
SETROPTS RACLIST(RDATALIB) refresh
  • Update access give userid COLIN access to the private key.
  • Read access only gives access to the public keys in the ring.

You would typically give a group of userids access, not just to individual userids.

Import the client’s CA’s used to sign the client certificates

This is the opposite to Export the CA certificate to a file so clients can access it above.

Copy the certificate to z/OS. This can be done using FTP or cut and paste.

Use it!

I used it in AT-TLS

TTLSConnectionAdvancedParms       TNCOonAdvParms 
{ 
 ServerCertificateLabel  NISTECC521
 ...
} 
TTLSSignatureParms                TNESigParms 
{ 
   CLientECurves Any 
} 
TTLSEnvironmentAction                 TNEA 
{ 
  HandshakeRole                       ServerWithClientAuth 
  TTLSKeyringParms 
  { 
    Keyring                   start1/TN3270 
  } 
...
}

Changing wi-fi frequency on Ubuntu

The wi-fi on two computers was giving different performance. I wondered if this was because it was using a different frequency wi-fi. I found this article useful on wi-fi. It says the higher the frequency used, the better the performance.

Display information about what the wi-fi is currently using.

The iwconfig command gave

wlp4s0    IEEE 802.11  ESSID:"BTHub6-78RQ"  
          Mode:Managed  Frequency:5.24 GHz  Access Point: 94:6A:B0:85:54:AA   
          Bit Rate=585.1 Mb/s   Tx-Power=20 dBm   
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Power Management:on
          Link Quality=41/70  Signal level=-69 dBm  
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:32   Missed beacon:0

Which shows the frequency (5.24 GHz) and bit rate (585.1 Mega bits /second (not bytes).

On the slower computer on the same wi-fi network it had 2.412 GHz.

Display information about the wireless interface

The iwlist command displays information about the wireless interface and its capabilities

iwlist wlp4s0 frequency gave

wlp4s0    32 channels in total; available frequencies :
          Channel 01 : 2.412 GHz
          Channel 02 : 2.417 GHz
          Channel 03 : 2.422 GHz
...       
          Channel 48 : 5.24 GHz
...
          Channel 132 : 5.66 GHz
          Channel 136 : 5.68 GHz
          Channel 140 : 5.7 GHz
          Current Frequency:5.24 GHz (Channel 48)

On my other machine it only listed 2.412 to 2,484 GHz.

What frequency is available?

Displaying my BT hub (http://192.168.1.254/basic) it displayed 2.4 and 5 GHz and channels 1 and 48. Thus I can use frequencies 2.412 and 5.24 GHz

Change the frequency being used

sudo iwconfig wlp4s0 freq 2.412G

Turn wi-fi off and on – and it picks up the changed value.

Defibrillators in Orkney.

Someone having a heart attack does not necessarily need a defibrillator, it is normally needed when someone suffers a cardiac arrest. If they are breathing and conscious they would not require a defibrillator.

The usual process, if someone is suspected of having a cardiac event, is to phone 999, and the call centre can advise you were the nearest defibrillator is located and supply the unlock code if it is required. Calling 999 also ensures medical assistance is on the way

Once switched on the defibrillator will talk you through the whole process and if needed guide you through CPR. There is no need to be concerned about wrongly shocking a patient as the defibrillator analyses the situation and will not allow a shock to be administered if it is not required. Depending on the type of defibrillator you may be ask to press a button to administer the shock (which tends to be the most common and preferred option) or it will happen automatically.

Getting a machine.

The defib store has been used by people in Orkney. They are used to sending the machines with the Lithium batteries to Orkeny.

Defibs recently purchased (April 2023) are the iPAD SP1 semi-automatic units by CU medical Systems and the cabinet was the defibstore 4000

The prices quoted on defib store are – iPad SP1 £1134 (including VAT) , and 4000 cabinet £598.80. Replacement pads cost £66 and battery £246 All prices include VAT, Defibstore are offering free shipping at the moment.

One advantage of the iPAD SP1 is that you can switch between infant (under 8 years old) and adult modes.

The defib should be registered with “The Circuit” which allows the emergency services to know its location and availability and they can allow public access when required. This does require a responsible person to log on to the web site at regular intervals to confirm the defib is in good order and available.

Maintenance

A weekly health check of the defibrillator (usually indicated by a green light).

A monthly check to make sure the key-press combination lock is in working order, and a light spray of WD40 to the combination lock and hinges.

Replace the pads every 2 years (£65).

Replace the battery – about 4-5 years (£100 – £200).

The external cabinet should have a main s supply so a small thermostatically controlled heater can maintain good storage conditions for the defibrillator.

zpdt and Ubuntu 22.04 -it worked!

I installed Ubuntu 22.04 on an isolated hard disk drive and installed IBM Z Development and Test Environment Personal Edition 14.0. on it.

I followed the documentation. The executable was sudo ./zdt-install-pe

The first time it ran, I allowed it to configure the network. It then prompted me with

Preconfiguration steps …

32 bit support not installed
Some of the above software dependencies are not installed

Do you want the necessary Linux dependencies for the product IBM® ZD&T Personal Edition to be installed? By entering y, all required dependencies will be installed. The list of dependencies are mentioned in the Prerequisites. You need to have access to internet and software repository to install the dependencies. Otherwise, installation will complete without dependencies, and you need to install the dependencies manually. For more information about linux prerequisites, see: https://ibm.biz/zdt_prerequisites

Y

It then hung.

When I reran it, but did not allow it to configure the network, and it ran successfully to completion in under a minute. It could have been a wi-fi problem.

I had to install x3270 (sudo apt install x3270) before starting my z/OS system. My zPDT environment was on a removable hard disk drive, so I plugged it it, started it up, and my system came up with no problems.

Performance tuning at the instruction level is weird

This post came out of a presentation on performance which I gave to some Computer Science students.

When I first joined the IBM MQ performance team, I was told that it was obvious that register/register instructions were fast, and storage/register were slow. Over time I found is not always true, I’ll explain why below…

For application performance there are some rules of thumb,

  • Use the most modern compilers, as they will have the best optimisation and use newer, faster instructions
  • Most single threading applications will gain from using the best algorithms, and storage structures, but they may gain very little from trying to tune which instructions to use.
  • Multi threading programs may get benefit from designing their programs so they do not interact at the instruction level.

You may do a lot of work to tune the code – and find it makes no difference. You change one field, and it can make a big difference. You think you understand it – and find you do not.

Background needed to understand the rest of the post.

Some of what I say is true. Some of what I say may be false – but you it will help you understand – even though it is false. For example, the table in front of me is solid and made out of wood. That is not strictly accurate. Atoms are mainly empty space. Saying the table is solid, is a good picture, but strictly inaccurate.

I spent my life working with the IBM 390 series and the description below is based on it – but I’ve changed some of it to keep it simple.

Physical architecture

The mainframe has an instruction “Move Character Long” (MVCL). If you get microscope and look at the processor chips, you will not find any circuitry which implements this instruction. This instruction is implemented in the microcode.

Your program running on a processor is a bit like Java byte codes. The microcode reads storage and finds an instruction and executes it.

For an instruction “move data from a virtual address into a register”, the execution can be broken down into steps

  1. Read memory and copy the instruction and any parameters
  2. Parse the data into operation-code, registers, and virtual storage address
  3. Jump to the appropriate code for the operation code
  4. Convert the virtual storage address into a real page address (in RAM). This is complex code. Every thread has its own address say, 4 000 000 , so you need the thread-look-up tables to get the “real address” for that thread. Your machine may be running virtualised, so the the “real address” needs a further calculation of the next level of indirection.
  5. “Lock” the register” and “lock” the real address of the data
  6. Go and get the data from storage
  7. Move the data into the register
  8. Unlock the register, unlock the real address of the data
  9. End.

Where is the data?

There is a large (TB) of RAM in the physical box. The processor “chips” are in books (think pluggable boards). The “chips” are about the size of my palm, one per book. There is cache in the books. Within the “chips” are multiple CPUs, storage management processors and more cache.

The speed of data access depends on the speed of light.

  • To get from the RAM storage to the CPU, the distance could be 100 cm – or 3 nanoseconds
  • To get from the book’s cache storage to the CPU this could be 10 cm or about 0.3 nanoseconds
  • The time for the CPU to access the memory on the chip is about 0.03 nano seconds.

The time for “Go and get the data from storage” (above) depends on where the data is. The first access may take 3 nano seconds when the data is read from RAM, if a following instruction uses the same data, it is already in the CPU cache, and so take 0.03 nanoseconds ( 100 times faster).

How is the program executed?

In the flow of an instruction (above) each stage is executed in a production line known as a pipeline. There are usually multiple instructions being processed at the same time.

While one instruction is in the stage “Convert the virtual storage address into a real page address”, another instruction is being parsed and so on.

If we had instructions

  1. Load register 5 from 4 000 000
  2. Load register 4 from register 5
  3. Clear register 6
  4. Clear register 7

Instruction 2 (Load register 4 from register 5) needs to use register 4 and cannot execute until the first instruction has finished. Instruction 2 has to has to wait until instruction 1 has finished. This shows that a register to register instruction may not be the fastest; it has to wait for a previous instruction to finish.

A clever compiler can reorder the code

  1. Load register 5 from 4 000 000
  2. Clear register 6
  3. Clear register 7
  4. Load register 4 from register 5

and so this code may execute faster because the clear register instructions can execute without changing the logic of the program. By the time the clear register instructions have finished, register 4 may be available.

If you look at code generated from a compiler, a register may be initialised many instructions away from where it is next used.

The hardware may be able to reorder these instructions; as long as they end in the correct order!

Smarter use of the storage and cache

Data is read and written to storage in “cache lines” these may be blocks of 256, 512 or 1024 byte blocks.

If you have a program with a big control block, you may get benefit from putting hot fields together. If your structure is spread across two cache lines, you may have processing like.

  • Load register 5 from cache block 1. This takes 3 ns.
  • Load register 6 from cache block 2. This takes 3 ns.

If the fields are adjacent you might get

  • Load register 5 from cache block 1. This takes 3 ns.
  • Load register 6 from cache block 1. This takes 0.03 ns because the cache block is already in the CPU’s cache.

Use faster instructions

Newer generations of CPUs can have newer and faster instructions. To load a register with the constant value – say 5. In the old days, it had to read this value from storage. Newer instructions may have the value(5) as part of the instruction – so no storage access is required, and so the instruction should be faster.

The second time round a loop may be faster than the first time around a loop.

The stages of the production line may cache data, for example converting a virtual address to a real page address. The stage may look in it’s cache – if not found then do the expensive calculation. If it is found then use the value directly.

If your program is using the same address (same page) the second time round the loop, the the real address of the data may already be in the CPU cache. The first time may have had to go to RAM, the second time the data is in the CPU cache.

This can all change

Consider the scenario where the first time round the loop was 100 times slower than later loop iterations – it may all suddenly change. Your program is un-dispatched to let someone else’s program run. When your program is re-dispatched, the cached values may no longer be available, so your program has a slow iteration while the real address of you virtual page is recalculated, and the data read in from RAM.

Multi programming interactions.

If you have multiple instances of your program running accessing shared fields, you can get interference at the instruction level.

Consider a program executing

  • Add value 1 to address 4 000 000

Part of the execution of this is to take a lock on the cache line with address 4 000 000. If another CPU executes the same instruction, the second CPU will have to wait until the first CPU has finished with it. If both CPUs are on the same chip the delay may be small (0.03 ns). If the CPUs are in different chips (in different books) it will take 0.3 nanoseconds to notify the second CPU.

If lots of CPUs are trying to access this field there will be a long access time.

You should design your program so each instance has its own cache line, so the CPUs do not compete for storage. I know of someone who did this and got 30% throughput improvement!

Configuring the hardware for virtual machines.

You should also consider how you configure your hardware. If you give each virtual machine CPUs on the same chip, then any interference should be small. If a virtual machine has CPUs in different books (so takes 0.3 nano seconds to talk to the other CPU) the interference will be larger because the requests take longer. Ive seen performance results vary by a couple of percentages because the CPUs allocated to a virtual machine were different on the second run.

Going deeper into the murk

If you have virtual machines sharing the same CPUs, this may affect your results, because your virtual machine may be un-dispatched, and another virtual machine is dispatched on the processor(s). The cached values for your virtual machine may be been overwritten.

Improving application performance – why, how ?

I’m working on a presentation on performance, for some university students, and I thought it would be worth blogging some of the content.

I had presented on what it was like working in industry, compared to working in a university environment. I explained what it is like working in a financial institutions; where you have 10,000 transactions a second, transactions response time is measured in 10s of milliseconds, and if you are down for a day you are out of business. After this they asked how you tune the applications and systems at this level of work.

Do you need to do performance tuning?

Like many questions about performance the answer is it depends….. it comes down to cost benefit analysis. How much CPU (or money) will you save if you do analysis and tuning. You could work for a month and save a couple of hundred pounds. You could work for a day and find CPU savings which means you do not need to upgrade your systems, and so save lots of money.

It is not usually worth doing performance analysis on programs which run infrequently, or are of short duration.

Obvious statements

When I joined the performance team, the previous person in the role had left a month before, and the hand over documentation was very limited. After a week or so making tentative steps into understanding work, I came to the realise the following (obvious once you think about it) statements

  • A piece of work is either using CPU or is waiting.
  • To reduce the time a piece of work takes you can either reduce the CPU used, or reduce the waiting time.
  • To reduce the CPU you need to reduce the CPU used.
  • The best I/O is no I/O
  • Caching of expensive operations can save you a lot.

Scenario

In the description below I’ll cover the moderately a simple case, and also the case where there are concurrent threads accessing data.

Concurrent activity

When you have more than one thread in your application you will need to worry about data consistency. There are locks and latches

  • Locks tend to be “long running” – from milliseconds to seconds. For example you lock a database record while updating it
  • Latches tend to be held across a block of code, for example manipulation of lists and updating pointers.

Storing data in memory

There are different ways of storing data in memory, from arrays, hash tables to binary trees. Some are easy to use, some have good performance.

Consider having a list of 10,000 names, which you have to maintain.

Array

An array is a contiguous block of memory with elements of the same size. To locate an element you calculate the offset “number of element” * size of element.

If the list is not sorted, you have to iterate over the array to find the element of interest.

If the list is sorted, you can do a binary search, for example if the array has 1000 elements, first check element 500, and see if the value is higher or lower, then select element 250 etc.

An array is easy to use, but the size is inflexible; to change the size of the array you have to allocate a new array, copy old to new, release old.

Single Linked list

This is a chain of elements, where each element points to the next, the there is a pointer to the start of the chain, and something to say end of chain ( often “next” is 0).

This is flexible, in that you can easily add elements, but to find an element you have to search along the chain and so this is not suitable for long chains.

You cannot easily delete an element from the chain.

If you have A->B->D->Q. You can add a new element G, by setting G->Q, and D->G. If there are multiple threads you need to do this under a latch.

Doubly linked lists

This is like a single linked list, but you have a back chain as well. This allows you to easily delete an element. To add an element you have to update 4 pointers.

This is a flexible list where you can add and remove element, but you have to scan it sequentially to find the element of interest, and so is not suitable for long chains.

If there are multiple threads you need to do this under a latch.

Hash tables

Hash tables are a combination of array and linked lists.

You allocate an array of suitable size, for example 4096. You hash the key to a value between 0 and 4095 and use this as the index into the array. The value of the array is a linked list of elements with the same hash value, which you scan to find the element of interest.

You need a hash table size so there are a few (up to 10 to 50) elements in the linked list. The hash function needs to produce a wide spread of values. Having a hash function which returned one value, means you would have one long linked list.

Binary trees

Binary trees are an efficient way of storing data. If there are any updates, you need to latch the tree while updates are made, which may slow down multi threaded programs.

Each node of a tree has 4 parts

  • The value of this node such as “COLIN PAICE”
  • A pointer to a node for values less than “COLIN PAICE”
  • A pointer to a node for values greater than “COLIN PAICE”
  • A pointer to the data record for this node.

If the tree is balanced the number of steps from the start of the tree to the element of interest is approximately the same for all elements.

If you add lots of elements you can get an unbalanced tree where the tree looks like a flag pole – rather than an apple tree. In this case you need to rebalanced the tree.

You do not need to worry about the size of the tree because it will grow as more elements are added.

If you rebalance the tree, this will require a latch on the tree, and the rebalancing could be expensive.

There are C run time functions such as tsearch which walks the tree and if the element exists in the tree, it returns the node. If it did not exist in the tree, it adds to the free, and returns the value.

This is not trivial to code – (but is much easier than coding a tree yourself).

You need to latch the tree when using multiple threads, which can slow down your access.

Optimising your code

Take the scenario where you write an application which is executed a 1000 times a second.

int myfunc(char * name, int cost, int discount)
{
  printf(“Values passed to myfunc %s cost discount" i\n”,name,cost,discount);
  rc= dosomething()  
  rc = 0;
  printf(“exit from myfunc %i\n”,rc);
  return rc;
}

Note: This is based on a real example, I went to a customer to help with a performance problem, and found the top user was printf() – printing out logging information. They commented this code out in all of their functions and it went 5 times faster

You can make this go faster by having a flag you set to produce trace output, so

if (global.trace ) 
    printf(“Values passed to myfunc %s cost discount" i\n”,name,cost,discount);

You could to the same for the exit printf, but you may want to be more subtle, and use

if (global.traceNZonexit  && rc != 0)
   printf(“exit from myfunc %i\n”,rc);

This is useful when the return code is 0 most of the time. It is useful if someone reports problems with the application – and you can say “there is a message access-denied” at the time of your problem.

FILE * hFILE = 0;
for ( I = 0;i < 100;i ++)
    /* create a buffer with our data in it */
    lenData =  sprintf(buffer,”userid %s, parm %s\n”, getid(), inputparm); 
    error = ….()
    if (error > 0)
    {
     hFILE = fopen(“outputfile”,”a);
     fwrite(buffer,1,lenData,fFile)
     fclose(hFile)
    }
…
}

This can be improved

  • by moving the getid() out of the loop – it does not change within the loop
  • move the lenData = sprintf.. within the error loop.
  • change the error loop
{
  ... 
  if (error > 0)
  {
     if (hFile == 0 )
     {  
        hFILE = fopen(“outputfile”,”a”);
        pUserid = strdup(getuserid());  
     } 
     fwrite(buffer,1,lenData,fFile)     
  }
...
}
if (hFile > 0) 
   fclose(hFile);

You can take this further, and have the file handle passed in to the function, so it is only opened once, rather than every time the function is invoked. 

main()
{
   struct {FILE * hFile
      …
    } threadBlock
   for(i=1,i<9999,i++)
   myprog(&threadBlock..}
   if (threadBlock →hFile != 0 )fclose(theadBlock → hFile);
   }
}
// subroutine
   myprog(threadblock * pt....){
...

  if (error > 0)
  {
     if (pt -> hFile == 0 )
     {  
        pt -> hFile= fopen(“outputfile”,”a”);       
     } 
     fwrite(buffer,1,lenData,pt -> hFile)
  }

Note: If this is a long running “production” system you may want to open the file as part of application startup to ensure the file can be opened etc, rather than find this out two days later.

Programming using AT-TLS

My mini project was to connect an openssl client to z/OS with AT-TLS only using a certificate. This was a challenging project partly because of the lack of a map and a description of what to do.

Overview

The usual way a server works with TCP/IP is using socket calls; socket(), bind(), listen() accept(), recv() and send(). You control the socket using ioctl().

This does not work with AT-TLS because ioctl() does not support the AT-TLS calls SIOCTTLSCTL; PL/I, REXX and Assembler supports it, but not C. (See here for a list of supported requests in C). I had to use a lower level set of interfaces (z/OS callable services); BPX1SOC(), BPX1BND(), BPX1LSN(), BPX1ACP(), BPX1RCV(), BPX1SND() and BPX1IOC1()

The documentation says

The application must have the ApplicationControlled parameter set to ON in their TTLSEnvironmentAdvancedParms or TTLSConnectionAdvancedParms statement. This causes AT-TLS to postpone the TLS handshake. After the connection is established, the application can issue the SIOCTTLSCTL IOCTL to get the current AT-TLS connection status and determine whether or not AT-TLS support is available on this connection.

Once the TLS session has been established, you can retrieve the userid associated with the certificate, or you can extract the client’s certificate and process it.

Once you have the userid or certificate you can use the pthread_security_applid_np(__CREATE_SECURITY_ENV…) to change the thread to a different userid. Note you have to run this as a thread – not as the main task.

The application flow

The application has the following logic

Main program

  • create a thread using pthread_create
  • wait for the thread to end – using rc =pthread_join
  • return

Thread subtask

  • Allocate a socket using bpx1soc.
  • Set the socket so it can quickly be reused. By default a port cannot be reused for a period of minutes, while waiting for a response from the client.
  • Bind the port to listen on to this socket using bpx1bnd.
  • Listen(wait for) a connection request on this socket using bpx1lsn.
  • Accept the request, using bpx1acp.
  • Issue the ioctl request using bpx1ioc to query information about the connection (TTLS_QUERY_ONLY). It returned:
    • Policy:Policy defined for connection – AT-TLS enabled and Application Controlled.
    • Type :Connection is not secure.
    • SSL Protocol Version 0 – because the session has not been established.
    • SSL Protocol Modifier 0 – because the session has not been established.
    • Rule name COLATTLJ.
    • Group Action TNGA.
    • Environment TNEA.
    • Connection TNCA.
    • Note: asking for TTLSK_Host_Status gave me “EDC5121I Invalid argument.” because this request is meaningless at this time.
  • Issue the ioctl request using BPX1IOC to start the the connection (TTLS_INIT_CONNECTION). This initiates the TLS handshake. If this is successful, it returned in the ioc control block
    • Policy:Policy defined for connection – AT-TLS enabled and Application Controlled
    • Type :Connection is secure
    • SSL Protocol Version 3
    • SSL Protocol Modifier 3
    • SSL Cipher 4X. 4X means look at the 4 byte field
    • SSL Cipher C02C. C02C is
      • TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, which is “256-bit AES in Galois Counter Mode encryption with 128-bit AEAD message authentication and ephemeral ECDH key exchange signed with an ECDSA certificateit AES in Galois Counter Mode encryption”
    • userid COLIN
  • Receive the data BPX1RCV(). You can peek to see how much data is available using flags = MSG_PEEK
  • Send a response using BPX1SND()
  • Close the remote session using BPX1CLO
  • Close the server’s socket using BPX1CLO
  • Exit the thread using pthread_exit()

Mapping certificate to userid

You can use AT-TLS to use the certificate and return the userid associated with the certificate; or you can use the pthread_security_applid_np to pass a certificate and change the thread to be the certificate owner.

You map a certificate to a userid with commands like

//COLRACF  JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT LISTMAP ID(COLIN) 
RACDCERT DELMAP(LABEL('CP'))  ID(COLIN) 
RACDCERT MAP ID(COLIN  )  - 
   SDNFILTER('CN=docec256.O=Doc.C=GB')                  - 
   IDNFILTER('CN=SSCA256.OU=CA.O=DOC.C=GB')             - 
   WITHLABEL('CP') 
RACDCERT LISTMAP ID(COLIN) 
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH 
/*

This associates the certificate with the given Subject Name (SN) value , and the Issuer’s value with the userid COLIN. See Using certificates to logon to z/OS for a high level perspective.

AT-TLS definitions

TTLSRule                      COLATTLJ 
{ 
  LocalPortRange              4000 
# Jobname                     COLCOMPI 
# Userid                      COLIN 
  Direction                   BOTH 
  TTLSGroupActionRef          TNGA 
  TTLSEnvironmentActionRef    TNEA 
  TTLSConnectionActionRef     TNCA 
} 

TTLSRule                      COLATTLS 
{ 
  LocalPortRange              4000 
# Jobname                     COLATTLS 
  Userid                      START1 
  Direction                   BOTH 
  TTLSGroupActionRef          TNGA 
  TTLSEnvironmentActionRef    TNEA 
  TTLSConnectionActionRef     TNCA 
} 

TTLSConnectionAction              TNCA 
{ 
  TTLSCipherParmsRef              TLS13TLS12 
  TTLSSignatureParmsRef           TNESigParms 
  TTLSConnectionAdvancedParmsRef  TNCOonAdvParms 
  CtraceClearText                 Off 
  Trace                           255 
} 

TTLSConnectionAdvancedParms       TNCOonAdvParms 
{ 
 ServerCertificateLabel  NISTECC521 
#ServerCertificateLabel  RSA2048 
#ccp this was added 
  ApplicationControlled         On 
  SSLv3          OFF 
  TLSv1          OFF 
  TLSv1.1        OFF 
  TLSv1.2        ON 
  TLSv1.3        OFF 
  SecondaryMap   OFF 
  HandshakeTimeout 3 
} 

TTLSSignatureParms                TNESigParms 
{ 
   CLientECurves Any 
} 

TTLSEnvironmentAction                 TNEA 
{ 
  HandshakeRole                       ServerWithClientAuth 
# HandshakeRole                       Server 
  TTLSKeyringParms 
  { 
    Keyring                   start1/TN3270 
  } 
  TTLSSignatureParmsRef       TNESigParms 
  TTLSCipherParmsRef  TLS13 
} 

TTLSCipherParms             TLS13TLS12 
{ 
#TLS 1.3 
 V3CipherSuites      TLS_CHACHA20_POLY1305_SHA256 
#V3CipherSuites      TLS_AES_256_GCM_SHA384 
#V3CipherSuites      TLS_AES_128_GCM_SHA256 
#TLS 1.2 
# NSTECC 
 V3CipherSuites      TLS_RSA_WITH_AES_256_CBC_SHA256 
 V3CipherSuites   TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 
#RSA 
}
 
TTLSCipherParms             TLS13 
{ 
 V3CipherSuites      TLS_CHACHA20_POLY1305_SHA256 
}
 
TTLSGroupAction             TNGA 
{ 
  TTLSEnabled               ON 
  trace                     255 
}

Submitting a job from userid COLIN, got definition Rule name COLATTLJ

When I used a started task COLATTLS with userid START1, I was expecting Rule name COLATTLS but got Rule name COLATTLJ. It looks like PAGENT uses the first matching rule; it matches rule PORT 4000, so used COLATTLJ.

My program printed out

Group Action TNGA.         
Environment  TNEA.         
Connection   TNCA.

which matches the definitions above.

The program printed out

BPX1SOC rv 0                                                                                         
BPX1SOC socket value 0  
                                                                             
BPX1OPT SET SO_REUSEADDR rv 0    
                                                                    
BPX1BND rv 0  
                                                                                       
BPX1LSN rv 0                                                                                         

BPX1ACP rv 1                                                                                         
After IOC   
                                                                                         
BPX1IOC rv 0                                                                                         
printATTLS rv 0                                                                                      
query.header.TTLSHdr_BytesNeeded 136                                                                 
BPX1IOC Policy:Policy defined for connection - AT-TLS enabled and Application Controlled             
BPX1IOC Conn  :Connection is not secure                                                              
BPX1IOC Type  :Server with client authentication ClientAuthType = Required                           
SSL protocol version TTLS_PROT_UNKNOWN  
Rule name    COLATTLJ.                                                                          
Group Action TNGA.                                                                              
Environment  TNEA.                                                                              
Connection   TNCA.                                                                              

TTLS_INIT_CONNECTION                                                   
BPX1SOC TTLS_INIT_CONNECTION  rv 0                                                              
After INIT_CONNECTION                                                                           
printATTLS rv 0                                                                                 
query.header.TTLSHdr_BytesNeeded 128                                                            
BPX1IOC Policy:Policy defined for connection - AT-TLS enabled and Application Controlled        
BPX1IOC Conn  :Connection is secure                                                             
BPX1IOC Type  :Server with client authentication ClientAuthType = Required                      
SSL protocol version TTLS_PROT_TLSV1_2                                                          
SSL Cipher  C02C 
                                                                               
ioc.TTLSi_Cert_Len 1080                                                                         
get cert  IOC                                                                                   
BPX1IOC Get cert rv 0   
userid 8     ADCDA                                                                
pthread_s... applid ZZZ rc = 0 userid    ADCDA.                                   
                                                                                  
BPX1RCV Peek rv 4  
                                                                
BPX1RCV bytes 4 
                                                         
BPX1SND rv 48

AT-TLS programming

In my program I had

Accept the session and invoke ATTLS

struct sockaddr_in client; /* client address information          */ 

BPX1SOC()...
BPX1OPT()... // Set SO_REUSEADDR
BPX1BND()...
BPX1LSN().. // this returns once there is a connection to the socket

int lClient = sizeof(client); 
BPX1ACP(&Socket_vector[0], 
        &lClient, 
        &client, 
        &rv, 
        &rc, 
        &rs); 
if (check("BPX1ACP",rv,rc,rs) < 0 ) // -1 is error 
  exit(4); 
int sd = rv; // save the returned value 

#include <attls.h> 
#include <attlssta.h>

Issue ATTLS query before initial TLS handshake

Member attls.h had 

// AT-TLS 
struct TTLS_IOCTL ioc;            // ioctl data structure 
memset(&ioc,0,sizeof(ioc));     //* set all unused fields to zero 
ioc.TTLSi_Req_Type = TTLS_QUERY_ONLY ; 
int command; 
                                                                     
command = SIOCTTLSCTL; 
ioc.TTLSi_Ver = TTLS_VERSION2; 
int lioc; 
lioc = sizeof(ioc); 
// 
// this is used for getting data from ATTLS 
// a header and a number of quads 
// 
memset(&query,0,sizeof(query)); 
// move the eye catcher 
memcpy(&query.header.TTLSHeaderIdent[0],TTLSHeaderIdentifier,8); 
query.header.TTLSHdr_BytesNeeded = 128; 
query.header.TTLSHdr_SetCount =  0; 
query.header.TTLSHdr_GetCount =  4;                                                                  
query.q1.TTLSQ_Key = TTLSK_TTLSRuleName  ;                                                                  
query.q2.TTLSQ_Key = TTLSK_TTLSGroupActionName;                                                                  
query.q3.TTLSQ_Key = TTLSK_TTLSEnvironmentActionName ;                                                                  
query.q4.TTLSQ_Key = TTLSK_TTLSConnectionActionName ; 
                                                                 
ioc.TTLSi_BufferPtr = (char *) &query; 
ioc.TTLSi_BufferLen = sizeof(query); 

ioc.TTLSi_Ver = TTLS_VERSION2; 
                                                        
BPX1IOC(&sd, 
        &command, 
        &lioc, 
        &ioc , 
        &rv, 
        &rc, 
        &rs); 
 
 if (check("BPX1IOC",rv,rc,rs) != 0) 
    exit(1); 
 printATTLS( &ioc,rv,rc,rs);

Issue the start connection

Member attlssta.h had

command = SIOCTTLSCTL; 
ioc.TTLSi_Ver = TTLS_VERSION2; 
lioc = sizeof(ioc); 
ioc.TTLSi_Req_Type = TTLS_INIT_CONNECTION ; 
printf("TTLS_INIT_CONNECTION\n"); 
// 
// 
memset(&query,0,sizeof(query)); 
// move the eye catcher 
memcpy(&query.header.TTLSHeaderIdent[0],TTLSHeaderIdentifier,8); 
query.header.TTLSHdr_BytesNeeded = 128; 
query.header.TTLSHdr_SetCount =  0; 
query.header.TTLSHdr_GetCount =  0; 
ioc.TTLSi_BufferPtr = 0; 
ioc.TTLSi_BufferLen = 0; 
//printHex(stdout,&query,256); 
ioc.TTLSi_Ver = TTLS_VERSION2; 
BPX1IOC(&sd, 
          &command, 
          &lioc, 
          &ioc , 
          &rv, 
          &rc, 
          &rs); 
if (check("BPX1SOC TTLS_INIT_CONNECTION ",rv,rc,rs) != 0) 
    exit(1); 
printATTLS( &ioc,rv,rc,rs); 
if (rv >= 0) 
  { // get the certificate 
    #include <ATTLSGC.h> 
  }

Get the certificate fromAT-TLS

ATTLSGC.h (get certificate from TCPIP) had 

// AT-TLS 
// 
//  Get the certificate 
// 
char * applid = "ZZZ"; 
memset(&ioc,0,sizeof(ioc));     //* set all unused fields to zero 
ioc.TTLSi_Req_Type = TTLS_QUERY_ONLY ; 
int command; 
                                                                     
command = SIOCTTLSCTL; 
ioc.TTLSi_Ver = TTLS_VERSION2; 
int lioc; 
lioc = sizeof(ioc); 
// 
// this is used for getting data from ATTLS 
// a header and a number of quads 
// 
memset(&query,0,sizeof(query)); 
// move the eye catcher 
memcpy(&query.header.TTLSHeaderIdent[0],TTLSHeaderIdentifier,8); 
query.header.TTLSHdr_BytesNeeded = 128; 
query.header.TTLSHdr_SetCount =  0; 
query.header.TTLSHdr_GetCount =  1; 

query.q1.TTLSQ_Key = TTLSK_Certificate   ; 
                                                                   
ioc.TTLSi_BufferPtr = (char *) &query; 
ioc.TTLSi_BufferLen = sizeof(query); 
ioc.TTLSi_Ver = TTLS_VERSION2;
BPX1IOC(&sd, 
          &command, 
          &lioc, 
          &ioc , 
          &rv, 
          &rc, 
          &rs); 
 printf("ioc.TTLSi_Cert_Len %d \n",ioc.TTLSi_Cert_Len); 
 printf("get cert  IOC\n"); 
 if (check("BPX1IOC Get cert",rv,rc,rs) != 0) 
    exit(1);

AT-TLS will return a userid for application OMVSAPPL, which may not be what was wanted.

Use the certificate to change the userid of the thread.

This uses pthread_security_applid_np to the userid determined from the certificate, and the specified applied.

Using userid and password the code is

rc = pthread_security_applid_np(__CREATE_SECURITY_ENV, 
           __USERID_IDENTITY, 
           5,         // length of userid
           "COLIN",   // userid
           "PASSWORD",  // password- null terminated
           0,"OMVSAPPL");

Using a certificate is a little more complicated.

// use certificate to change userid 

char * applid = "ZZZ";                                                             
struct __certificate  ct; 
ct.__cert_type = __CERT_X509; 
char * pData = (char *)  ioc.TTLSi_BufferPtr; 
           // offsets are from start of header 
ct.__cert_length = ioc.TTLSi_Cert_Len; 
ct.__cert_ptr    =& pData[query.q1.TTLSQ_Offset] ; 
//printHex(stdout,ct.__cert_ptr, 66); 
rc = pthread_security_applid_np(__CREATE_SECURITY_ENV, 
         __CERTIFICATE_IDENTITY, 
         sizeof(ct),  // size of object
         &ct,         // adress of object
         "xxxxxxxx",  // not used with certificate
         0,           // options
         applid); // this controls which applid security checks are done.
if ( rc != 0) 
  perror("pthead security"); 
switch (errno) 
 { 
 case ESRCH : 
   printf("ESRCH:" 
   "The user ID provided as input is not defined to the " 
   "security product or does not have an OMVS segment defined" 
   "\n"); 
   break; 
 } 
if (rc != 0) 
{ 
lOutBuff = sprintf(&outBuff[0], 
"pthread_s... applid %s  rc = %d errno %d %s errno2 %8.8x\n\n", 
        applid, 
        rc,errno,strerror(errno),__errno2()); 
} 
else 
{ 
  userlen = 0;  
  rc = __getuserid(&userid[0], userlen); 
  if (rc != 0) 
     printf("getuser rc %d\n",rc); 
  printf("userid %d  %*.*s\n",userlen,userlen,userlen,userid); 
    lOutBuff = sprintf(&outBuff[0], 
          "pthread_s... applid %s rc = 0 userid %*.*s.\n", 
          applid, 
          userlen,userlen,userid); 
} 
printf("%s\n",outBuff);

This certificate was mapped to userid ADCDA, and the userid ADCDA was printed. See Using certificates to logon to z/OS-Use a subject DN for the mapping.

Routine to print out the IOC and its data

int  printATTLS(struct  TTLS_IOCTL * pioc, 
                 int rv, int rc, int rs)
{ 
    if (check("printATTLS",rv,rc,rs) != 0) // check the return code
       return(8); 
    printf("query.header.TTLSHdr_BytesNeeded %d\n", 
        query.header.TTLSHdr_BytesNeeded); 
    printf("BPX1IOC Policy:%s\n",Stat_Policy[pioc->TTLSi_Stat_Policy]); 
    printf("BPX1IOC Conn  :%s\n", Stat_Conn[ pioc->TTLSi_Stat_Conn]); 
    printf("BPX1IOC Type  :%s\n", Set_type[ pioc->TTLSi_Sec_Type]); 
    char * pProt = "Unknown Protocol"; 
    switch( pioc->TTLSi_SSL_Prot) 
    { 
      case TTLS_PROT_UNKNOWN: pProt = "TTLS_PROT_UNKNOWN";break; 
      case TTLS_PROT_SSLV2  : pProt = "TTLS_PROT_SSLV2  ";break; 
      case TTLS_PROT_SSLV3  : pProt = "TTLS_PROT_SSLV3  ";break; 
      case TTLS_PROT_TLSV1  : pProt = "TTLS_PROT_TLSV1  ";break; 
      case TTLS_PROT_TLSV1_1: pProt = "TTLS_PROT_TLSV1_1";break; 
      case TTLS_PROT_TLSV1_2: pProt = "TTLS_PROT_TLSV1_2";break; 
      case TTLS_PROT_TLSV1_3: pProt = "TTLS_PROT_TLSV1_3";break; 
    }
//  printf("SSL Protocol Version  %u\n",  pioc->TTLSi_SSL_ProtVer); 
//  printf("SSL Protocol Modifier %hhu\n",  pioc->TTLSi_SSL_ProtMod); 
    printf("SSL protocol version %s\n",pProt); 
    if (pioc->TTLSi_Neg_Cipher[0] != 0 ) 
    { 
      if ( memcmp(&pioc-> TTLSi_Neg_Cipher[0] ,"4X",2) == 0) 
      printf("SSL Cipher  %4.4s\n",pioc->TTLSi_Neg_Cipher4   ); 
      else 
      printf("SSL Cipher    %2.2s\n",pioc->TTLSi_Neg_Cipher   ); 
    } 
    if (pioc->TTLSi_Neg_KeyShare[0]!= 0) 
      printf("SSL key share   %4.4s\n",pioc->TTLSi_Neg_KeyShare   ); 
    int lUserid = pioc->TTLSi_UserID_Len; 
    if (lUserid >0 ) 
    { 
      printf("userid %*.*s\n",lUserid,lUserid,&pioc->TTLSi_UserID[0]); 
    }
    if (pioc->TTLSi_BufferLen > 0 
       && pioc->TTLSi_BufferPtr > 0) 
    { 
      int len = 256; 
      if (pioc->TTLSi_BufferLen < len) 
      len = pioc->TTLSi_BufferLen; 
      //printHex(stdout,pioc->TTLSi_BufferPtr,len ); 
      char * pData = (char *) pioc->TTLSi_BufferPtr; 
                    // offsets are from start of header 
      if (query.q1.TTLSQ_Offset > 0) 
        printf("Rule name    %s.\n",&pData[query.q1.TTLSQ_Offset]); 
      else 
        printf("Rule name missing\n"); 
      if (query.q2.TTLSQ_Offset > 0) 
        printf("Group Action %s.\n",&pData[query.q2.TTLSQ_Offset]); 
      else 
        printf("Group Action missing\n"); 
      if (query.q3.TTLSQ_Offset > 0) 
        printf("Environment  %s.\n",&pData[query.q3.TTLSQ_Offset]); 
      else 
        printf("Environment  missing\n"); 
      if (query.q4.TTLSQ_Offset > 0) 
        printf("Connection   %s.\n",&pData[query.q4.TTLSQ_Offset]); 
      else 
        printf("Connection   missing\n"); 
   } 
}

Header file

// used to query data
struct  { 
  struct TTLSHeader header; 
  struct TTLSQuadruplet q1; 
  struct TTLSQuadruplet q2; 
  struct TTLSQuadruplet q3; 
  struct TTLSQuadruplet q4; 
  struct TTLSQuadruplet q5; 
  char buffer[4096]; 
} query; 
 
// used in printing IOC                                                              
 char * Stat_Policy[]={ 
    "reserved", 
    "AT-TLS function is off", 
    "No policy defined for connection", 
    "Policy defined for connection - AT-TLS not enabled", 
    "Policy defined for connection - AT-TLS enabled", 
    "Policy defined for connection - AT-TLS enabled and " 
    "Application Controlled"};
char * Stat_Conn[] = { 
      "reserved", 
      "Connection is not secure", 
      "Connection handshake in progress", 
      "Connection is secure"}; 
char * Set_type[] = { 
    "reserved", 
    "Client", 
    "Server", 
    "Server with client authentication " 
    "ClientAuthType = PassThru", 
    "Server with client authentication " 
    "ClientAuthType = Full", 
    "Server with client authentication " 
    "ClientAuthType = Required ", 
    "Server with client authentication " 
    "ClientAuthType = SAFCheck"}; 
struct TTLS_IOCTL ioc;            // ioctl data structure 
char buff[1000];                  // buffer for certificate

Aside on ClientHandshakeSNI

I spent a couple of hours trying to get this to work. I got ServerHandshakeSNIto work.

The documentation says

ClientHandshakeSNI


For TLSv1.0 protocol or later, this keyword specifies whether a client can specify a list of server names. The server chooses a certificate based on that server name list for this connection. For System SSL, the extension ID is set to GSK_TLS_SET_SNI_CLIENT_SNAMES and a flag is set in the gsk_tls_extension structure if it is required. Valid values are:

  • Required: Specifies that server name indication support must be accepted by the server. Connections fail if the server does not support server name indication.
    • Tip: When you specify ClientHandshakeSNI as required, specify SSLv3 as Off.
  • Optional -Specifies that server name indication negotiation is supported, but allows connections with servers that do not support server name indication negotiation.
  • Off – Specifies that server name indication is not supported. The function is not enabled. Connections fail if the server requires support for server name indication. This is the default.

I think this only applies when the program on z/OS is running as a client.

Using ServerHandshakeSNI

For example 

ServerHandshakeSNI Required
ServerHandshakeSNIMatch Required
ServerHandshakeSNIList COLINCLIENZ/NISTECC521
ServerHandshakeSNIList CLIENT2/BB
  • If my client uses -servername COLINCLIENZ then the certificate with label NISTECC521 will be used.
  • If my client uses -servername CLIENT2 then the BB certificate will be used
  • Any other server name (or if is omitted) the connection will fail

Verify the sender

The client can use –verify_hostname ZZZZZZ to verify the name of the host.

Write instructions for your target audience – not for yourself.

Over the last couple of weeks, I’ve been asked questions about installing two products on z/OS. I looked at the installation documentation, and it was written the way I would write it for myself – it was not written for other people to follow.

I sent some comments to one of the developers, and as the comments mainly apply to the other products as well, I thought I would write them down – for when another product comes along.

I’ve been doing some documentation of for AT-TLS which allows you to give applications TLS support, without changing the application, so I’ll focus on a product using TCP/IP.

What is the environment?

The environment can range from one person running z/OS on a laptop, to running a Parallel Sysplex where you have multiple z/OS instances running as a Single System Image; and taking it further, you can have multiple sites.

What levels of software

Within a Sysplex you can have different levels of software, for example one image at z/OS 2.4 and another image at z/OS 2.5 You tend to upgrade one system to the next release, then when this has been demonstrated to be stable, migrate the other systems in turn.

Within one z/OS image you can have multiple levels of products, for example MQ 9.2.3 and MQ 9.1. People may have multiple levels so they test the newer level, and when it looks stable, they switch to the newer level and later remove the older level. If the newer level does not work in production – they can easily switch back to the previous level.

Each version may have specific requirements.

  • If your product has an SVC, you may need an SVC for each version, unless the higher level SVC supports the lower level code.
  • If your product uses a TCP/IP port, you will need a port for each instance.

You need to ensure your product can run in this environment, with more than one version installed on an image.

How do things run?

Often z/OS images and programs run for many months. For example IPLing every three months to get the latest fixes on. Your product instance may run for 3 months before restarting. If you write message to the joblog, or have output going to the JES2 spool, you want to be able to purge old output without shutting down your instance. You can specify options to “spin” off output and make the file purge-able.

Your instance may need to be able to refresh its parameters. For example, if a key in a keyring changes, you need to close and reopen the keyring. This implies a refresh command, or the keyring is opened for each request.

Who is responsible for the system?

For me – I am the only person using the system and I am responsible for every thing.

For big systems there will be functions allocated to different departments:

  • Installation of software (getting the libraries and files to the z/OS image)
  • The z/OS systems team – creating and updating the base z/OS system
  • The Security team – this may be split into platform security(RACF), and network security
  • Data management – responsible for data, backup (and restore), migration of unused data sets to tape, ensuring there is enough disk space available.
  • Communications team – responsible for TCPIP connectivity, DNS, firewalls etc.
  • Database team – responsible for DB2 and other products
  • Liberty and z/OSMF etc built on top of Liberty.
  • MQ – responsible for MQ, and MQ to MQ connectivity.

Some responsibilities could be done by different teams, for example creating the security profile when creating a started task. This is a “security” task – but the z/OS systems programmer will usually do it.

How are systems changes managed?

Changes are usually made on a test system and migrated into production. I’ve seen a rule “nothing goes into production which has not been tested”. Some implications of this are

  • No changes are typed into production. A file can be copied into production, and a file may have symbolic substitution, such as SYSTEM=&SYSNAME. You can use cut and paste, but no typing. This eliminates problems like 0 being misread as O, and 1,i,l looking similar.
  • Changes are automated.
  • Every change needs a back-out process – and this back-out has been tested.
    • Delete is a 2 phase operation. Today you do a rename rather than a delete; next week you do the actual delete. If there is a problem with the change you can just rename it back again. Some objects have non obvious attributes, and if you recreate an object, it may be different, and not work the same way as it used to.

There are usually change review meetings. You have to write a change request, outlining

  • the change description
  • the impact on the existing system
  • the back-out plan
  • dependencies
  • which areas are affected.

You might have one change request for all areas (z/OS, security, networking), or a change request for each area, one for z/OS, one for security, one for networking.

Affected areas have to approve changes in their area.

How to write installation instructions

You need to be aware of differences between installing a product first time, and successive times. For example creating a security definition. It is easy to re-test an install, and not realise you already have security profiles set up. A pristine new image is great for testing installation because it is clean, and you have to do everything.

Instructions like

  • Task 1 – create sys1.proclib member
  • Task 2 – define security profile
  • Task 3 – allocate disk storage
  • Task 4 – define another security profile
  • Task 5 – update parmlib

may make sense when one person is doing the work, but not if there are many teams.

It is better to have a summary by role like

  • z/OS systems programmer
    • create proclib member
    • update parmlib
  • Security team
    • Define security profile 1
    • Define security profile 2
  • Storage management team
    • Allocate disk space

and have links to the specific topics. This way it is very clear what a team’s responsibilities are, and you can raise one change request per team.

This summary also gives a good road map so you can see the scale of the installation task.

It is also good to indicate if this needs to be done once only per z/OS image, or for every instance. For example

  • APF authorise the load libraries – once per z/OS image
  • Create a JCL procedure in SYS1.PROCLIB – once per instance

Some tasks for the different roles

z/OS system programmers

  • Create alias for MYPROD.* to a user catalog
  • APF authorise MYPROD…. datasets
  • Create PARMLIB entries
  • Update LNKLST and LPA
  • Update PROCLIB concatenation with product JCL
  • Create security profiles for any started tasks; which userid should be used?
  • WLM classification of the started task or job.
  • Schedule delete of any old log files older than a specified criteria
  • When multiple instances per LPAR, decide whether to use S MYSTASK1, S MYSTASK2, or S MYSTASK.T1, S MYSTASK.T2
  • Do you need to specify JESLOG SPIN to allows JES2 logs to be spun regulary, or when they are greater than a certain size, or any DD SYSOUT with SPIN?
  • ISPF
    • Add any ISPF Panels etc into logon procedures, or provide CLIST to do it.
    • Update your ISPF “extras” panel to add product to page.
  • Try to avoid SVCs. There are better ways, for example using authorized services.
  • Propagate the changes to all systems in the Sysplex.
  • What CF structures are needed. Do they have any specific characteristics, such as duplexed?
  • How much (e)CSA is needed, for each product instance.
  • Does your product need any Storage Class Memory (SCM).

Security team

  • Create groups as needed eg MYPRODSYS, MYPRODRO, and make requester’s userid group special, so they can add and remove userids to and from the groups.
  • Create a userid for the started task. Create the userid with NOPASSWORD, to prevent people logging on with the userid and password.
  • Protect the MYPROD.* datasets, for example members of group MYPRODSYS can update the datasets, members of group MYPRODRO only have read-only access.
  • Create any other profiles.
  • Create any certificate or keyrings, and give users access to them.
  • Set up profiles for who can issue operator commands against the jobs or procedures.
  • Does the product require an “applid”. For example users much have access to a specific APPL to be able to use the facilities. An application can use pthread_security_applid_np, to change the userid a thread is running on – but they must have access to an applid. The default applid is OMVSAPPL.
  • Do users needing to use this product need anything specific? Such as id(0), needing a Unix Segment, or access to any protected resources? See below for id(0).
  • If a client authenticates to the server, the server needs access to BPX.SERVER in the RACF FACILITY.
  • The started task userid may need access to BPX.DAEMON.
  • If a userid needs access to another user’s keyring, the requestor needs read access to user.ring.LST in CLASS(RDATALIB) or access to IRR.DIGTCERT.LISTRING.
  • If a userid needs access to a private key in a keyring the requester needs If a userid needs access to another user’s keyring, the requester needs control access to user.ring.LST in CLASS(RDATALIB).
  • You might need to program control data sets, for example RDEF PROGRAM * ADDMEM(‘SYS1.LINKLIB’//NOPADCHK) UACC(READ) .
  • Users may need access to ICSF class CSFSERV and CSFKEYS.
  • Use of CLASS(SURROGAT) BPX.SRV.<userid> to allow one userid to be a surrogate for another userid.
  • Use of CLASS(FACILITY) BPX.CONSOLE to remove the generation of BPXM023I messages on the syslog.

Storage team

  • How much disk space is needed once the product has been installed, for data sets, and Unix file systems. This includes product libraries and instance data, and logs which can grow without limit.
  • How much temporary space is needed during the install.
  • Where do Unix files for the product go? for example /opt/ or /var….
  • Where do instance files go. For example on image local disks, or sysplex shared disks. You have an instance on every member of the Sysplex – where you do put the instance files?
  • How much data will be produced in normal running – for example traces or logs.
  • When can the data be pruned?
  • Does the product need its own ZFS for instance data, to keep it isolated and so cannot impact other products.
  • Are any additional Storage Classes etc needing to be defined? These determine if and when datasets are migrated to tape, or get deleted.
  • Are any other definitions needed. For example for datasets MYPROD.LOG*, they need to go on the fastest disks, MYPROD.SAMPLES* can go on any disks, and could be migrated.

Database team

  • What databases, tables,indexes etc are required?
  • How much disk space is needed.
  • What volume of updates per second. Can the existing DB2 instances sustain the additional throughput?
  • What security and protection is needed at the table level and at the field level.
  • What groups are permitted to access which fields?
  • What auditing is needed?
  • Is encryption needed?

MQ

  • Do you need to uses MQ Shared Queue between queue managers?
  • How much data will be logged per second?
  • What is the space needed for the message storage, disk space, buffer pool and Coupling Facility?
  • Product specific definitions.
  • Security protection of any product specific definitions.

Networking

  • Which port(s) to use?
    • Do you need to control access to ports with the SAF resource on the PORT entry, and permit access to profile EZB.PORTACCESS.sysname.tcpname.resname
    • Use of SHAREPORT and SHAREPORTWLM
  • Use of Sysplex Distributor to route work coming in to a Sysplex to any available system?
  • Update the port list – so only specific job can use it
  • RACF profile for port?
  • Which cipher specs
  • Which level of TLS
  • Which certificates
  • Any AT-TLS profile?
  • Any firewall changes?
  • Any class of service?
  • Any changes to syslogd profile?
  • Are there any additional sites that will be accessed, and so need adding to the “allow” list.

Automation

  • If the started tasks, or jobs need to be started at IPL, create the definitions. Do they have any pre-reqs, for example requiring DB2 to be active.
  • If the jobs are shutdown during the day, should they be automatically restarted?
  • Add automation to shut down any jobs or started tasks, when the system is shutdown
  • Which product messages need to be managed – such as events requiring operator action, or events reported to the site wide monitoring team.

Operations

  • Play book for product, how to start, and stop it
  • Are there any other commands?

Monitoring

  • Any SMF data needed to be collected.
  • Any other monitoring.
  • How much additional CPU will be needed – at first, and in the long term.

Making your product secure

Many sites are ultra careful about keeping their system secure. The philosophy is give a user access for what they need to do – but no more. For example

  • They will not be comfortable installing a non IBM SVC into their system. An SVC can be used from any address space, so if there is any weakness in the SVC it could be exploiter.
  • Using id(0) (superuser) in Unix Services is not allowed. The userid needs to be given specific permission. If the code “changes userid” then services like pthread_security_applid_np() should be used; where the applid is part of the configuration. Alternatives include __login_applid. End users of this facility will need read access to the specific applid.

TLS and SSL

If you are using TLS there are other considerations

  • Any certificate you generate needs a long validity date, and JCL to recreate it when it expires.
  • If you create a Certificate Authority you need to document how to export it and distribute it to other platforms
  • Browsers and application may verify the host name, so you need to generate a certificate with a valid name. The external z/OS name may be different from the internal name.
  • You should support TLS V1.2 and TLS 1.3 Other TLS and SSL versions are deprecated.
  • It is good practice to have one keyring with the server certificate with its private key, and a “common” trust store keyring which has the Certificate Authorities for all the sites connecting to the z/OS image. If you connect to a new site, you update the common keyring, and all applications pick up the new CA. If you have one keyring just for your instance, you need to maintain multiple keyrings when a new certificate is added, one for each application.

Migrating from cc to xlc is like playing twister

I needed to compile a file in Unix System Services; I took an old make file, changed cc to xlc expecting it to compile and had lots of problems.

It feels like the documentation was well written in the days of the cc and c89 complier, and has a different beast inserted into it.

As started to write this blog post, I learned even more about compiling in Unix Services on z/OS!

Make file using cc 

cparmsa= -Wc,"SSCOM,DEBUG,DEF(MVS),DEF(_OE_SOCKETS),UNDEF(_OPEN_DEFAULT),NOOE 
cparmsb= ,SO,SHOW,LIST(),XREF,ILP32,DLL,SKIPS(HIDE)" 
syslib= -I'/usr/include' -I'/usr/include/sys'  -I"//'TCPIP.SEZACMAC'" -I"//'TCPIP.SEZANMAC'" 
all: main 
parts =  tcps.o 
main: $(parts)
  cc -o tcps  $(parts) 
                                                                                                                            
%.o: %.c 
 cc  -c -o $@   $(syslib) $(cparmsa)$(cparmsb)    -V          $< 
 
clean: 
 rm  *.o

The generated compile statement is

cc -c -o tcps.o -I’/usr/include’ -I’/usr/include/sys’ -I”//’TCPIP.SEZACMAC'” -I”//’TCPIP.SEZANMAC'” -Wc,”SSCOM,DEBUG,DEF(MVS),DEF(_OE_SOCKETS),UNDEF(_OPEN_DEFAULT),NOOE,SO, SHOW,LIST(),XREF,ILP32,DLL,SKIPS(HIDE)” -V tcps.c

Note the following

  • the -V option generates the listing. “-V produces all reports for the compiler, and binder, or prelinker, and directs them to stdout“. If you do not have -V you do not get a listing.
  • -Wc,list() says generate a list with a name like tcps.lst based on the file name being compiled. If you use list(x.lst) it does not produce any output! This is contrary to what the documentation says. (Possible bug on compiler when specifying nooe”
  • SHOW lists the included files
  • SKIPS(HIDE) omits the stuff which is not used – see below.

Make using xlc

I think the xlc compiler has bits from z/OS and bits from AIX (sensibly sharing code!). On AIX some parameters are passed using -q. You might use -qSHOWINC or -qNOSHOWINC instead of -Wc,SHOWINC

cparmsx= -Wc,"SO,SHOW,LIST(lst31),XREF,ILP32,DLL,SSCOM, 
cparmsy= DEBUG,DEF(MVS),DEF(_OE_SOCKETS),UNDEF(_OPEN_DEFAULT),NOOE" 
cparms3= -qshowinc -qso=./lst.yy  -qskips=hide -V 
syslib= -I'/usr/include' -I'/usr/include/sys'  -I"//'TCPIP.SEZACMAC'" -I"//'TCPIP.SEZANMAC'" 
all: main 
parts =  tcps.o 
main: $(parts) 
  cc -o tcps  $(parts) 
                                                                                                      
%.o: %.c 
 xlc -c -o $@   $(syslib) $(cparmsx)$(cparmsy) $(cparms3)     $< 
                                                                                                      
clean: 
 rm  *.o

This generates a statement

xlc -c -o tcps.o -I’/usr/include’ -I’/usr/include/sys’ -I”//’TCPIP.SEZACMAC'” -I”//’TCPIP.SEZANMAC'” -Wc,”SO,SHOW,LIST(lst31),XREF, ILP32,DLL, SSCOM,DEBUG,DEF(MVS),DEF(_OE_SOCKETS), UNDEF(_OPEN_DEFAULT),NOOE” -qshowinc -qso=./lst.yy -qskips=hide tcps.c

Note the -q options. You need -qso=…. to get a listing.

Any -V option is ignored, and LIST(…) is not used.

Note: There is a buglet in the compiler, specifying nooe does not always produce a listing. The above xlc statement gets round this problem.

SKIPS(SHOW|HIDE)

The SKIPS(HIDE) also known as SKIPSRC shows you what is used, and suppresses text which is not used. I found this useful trying to find the combination of #define … to get the program to compile.

For example with SKIPS(SHOW)

170 |#if 0x42040000 >= 0X220A0000                               | 672     4      
171 |    #if defined (_NO_PROTO) &&  !defined(__cplusplus)      | 673     4      
172 |        #define __new210(ret,func,parms) ret func ()       | 674     4      
173 |    #else                                                  | 675     4      
174 |        #define __new210(ret,func,parms) ret func parms    | 676     4      
175 |    #endif                                                 | 677     4      
176 |#elif !defined(__cplusplus) && !defined(_NO_NEW_FUNC_CHECK)| 678     4      
177 |       #define __new210(ret,func,parms) \                  | 679     4      
178 |        extern struct __notSupportedBeforeV2R10__ func     | 680     4      
179 |   #else                                                   | 681     4      
180 |     #define __new210(ret,func,parms)                      | 682     4      
181 |#endif                                                     | 683     4

With SKIPS(HIDE) the bold lines are not displayed, 

170 |#if 0x42040000 >= 0X220A0000                              | 629     4 
171 |    #if defined (_NO_PROTO) &&  !defined(__cplusplus)     | 630     4 
172 |        #define __new210(ret,func,parms) ret func ()      | 631     4 
173 |     else                                                 | 632     4 
175 |    #endif                                                | 633     4 
176 |                                                          | 634     4 
179 |   #else                                                  | 635     4 
181 |#endif                                                    | 636     4 
182 |#endif                                                    | 637     4

This shows

  • 170 The line number within the included file
  • 629 The line number within the file
  • 4 is the 4th included file. In the “I N C L U D E S” section it says 4 /usr/include/features.h
  • rows 174 is missing … this is the #else text which was not included
  • rows 177, 178,180 are omitted.

This makes is much easier to browse through the includes to find why you have duplicate definitions and other problems.