Category: ADCD

Using a VSAM file from another system.

I have been working with two levels of ADCD z/OS system, Z24C and Z25D. I want to be able to use VSAM files from the z24C level system on the z25D system.

With non VSAM files, it is easy. I can define an alias for a high level qualifier such as my userid COLIN which points to the user catalog with my data sets in it. It is a bit harder with VSAM files, especially where there is a file with the same name of both systems (such as CSF.CSFCKDS).

A VSAM PATH is an alias for VSAM files.

Conceptually the first step is

//IBMDEFP JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IDCAMS 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD * 
 DEF PATH - 
   (NAME(COLINQ.AUT420.AUT420D.CSI.Z24C) - 
    PATHENTRY( AUT420.AUT420D.CSI ) - 
   )                    - 
   CATALOG( USERCAT.Z24C.PRODS ) 
/*

File AUT420.AUT420D.CSI is in catalog USERCAT.Z24C.PRODS

The above JCL will create a name COLINQ.AUT420.AUT420D.CSI.Z24C which points to the file AUT420.AUT420D.CSI in catalog USERCAT.Z24C.PRODS. The entry COLINQ.AUT420.AUT420D.CSI.Z24C is put in the same catalog.

If you use ISPF 3.4 it will not find the dataset.

Create an alias for the High Level Qualifier

//IBMUSERT JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IDCAMS,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD * 
  DEFINE ALIAS (NAME(COLINQ) RELATE('USERCAT.Z24C.PRODS')) 
/*

The above JCL will create an alias COLINQ, and says to find any datasets beginning with COLINQ go and look in catalog USERCAT.Z24C.PRODS.

To import the catalog into the current system

//IBMIMPC JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IDCAMS 
//SYSPRINT DD SYSOUT=* 
//CAT  DD DISP=SHR,DSN=ADCD.LIB.JCL,VOL=SER=C4SYS1, 
//        UNIT=3390 
//SYSPRINT DD SYSOUT=A 
//SYSIN DD * 
  IMPORT - 
    OBJECTS - 
      ((USERCAT.Z24C.PRODS  - 
        VOLUME(C4SYS1) - 
        DEVICETYPE(3390))) - 
     CONNECT - 
   CATALOG(CATALOG.Z25D.MASTER) 
/* 
//

The above JCL says import the catalog USERCAT.Z24C.PRODS on volume(C4SYS1), type (3390) into the (master) catalog CATALOG.Z25D.MASTER.

If the system needs to find USERCAT.Z24C.PRODS, it has enough information to be able to find it.

What you actually do

Now that you understand the process, the process you should follow is

  • Import the catalog into the current system.
  • Define an High Level Qualifier alias to point to the catalog. I might pick COLIN4C ( for the z24C system).
  • Create a path using COLIN4C as the high level qualifier of the data set, for each VSAM file.

You should then be able to see your data set in ISPF 3.4

To access the Z24C /u ZFS files system on the Z25D system I used

 IMPORT - 
   OBJECTS - 
     ((CATALOG.Z24C.MASTER   - 
       VOLUME(C4SYS1) - 
       DEVICETYPE(3390))) - 
    CONNECT - 
  CATALOG(CATALOG.Z25D.MASTER) 

 DEFINE ALIAS (NAME(Z24CMAST) RELATE('CATALOG.Z24C.MASTER')) 

 DEFINE PATH  - 
     (NAME(Z24CMAST.ZFS.USERS     ) - 
     PATHENTRY( ZFS.USERS       )) - 
     CATALOG(CATALOG.Z24C.MASTER)

In Unix I created a directory

mkdir /u/old

The mounted the file system in ISPF option 6 TSO

mount filesystem('Z24CMAST.ZFS.USERS') mountpoint('/u/old') type(ZFS)  
mode(read)

I could then access the files from /u/old/…

NO? Migrating an ADCD z/OS release:RACF

This post is a very quick post in response to a question about migrating your RACF data to a different system. I will update it when I have more time.

This is one of a series of posts about migrating to a newer level of ADCD.

This covers RACF.

A typical z/OS migration is to take your existing system and upgrade it. With ADCD the operating system is replaced and you have to move to it, taking your data with you.

You need to think about your RACF definitions, and migrates your specific data into the newer RACF database. This will include userid, profiles, permissions and digital certificates.

Some of the changes are good practice and clean up. For example

Instead of a definition CICSTS55.* in class(STARTED), the profile is CICSTS.*.* in class(STARTED). Using the scripts described in this blog you can update your old system before you move to the new system. 

I have put files into a github repository.

Migrating certificates and keyrings

The JCL EXPCERT runs a rexx exec which create commands to export your certificates, imports them (on the new system), and recreates the keyrings (on the new system).

The exec issues the RACDCERT command to list all the keyrings for a userid, and processes that output.

//IBMCERT  JOB 1,MSGCLASS=H 
// EXPORT SYMLIST=(*) 
//ADCDA EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//OUTPUT   DD DISP=SHR,DSN=COLIN.MIG.DATA(EXPCERTD) 
//IMPORT   DD DISP=SHR,DSN=COLIN.MIG.DATA(IMPCERTD) 
//RING     DD DISP=SHR,DSN=COLIN.MIG.DATA(RINGD) 
//SYSEXEC  DD DISP=SHR,DSN=USER.Z24C.CLIST 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN  DD * 
%LRING COLIN.CERT.EXPORT   START1 PASS4ME4 
/* 
//S2    EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSIN  DD DISP=SHR,DSN=COLIN.MIG.DATA(EXPCERTD) 
//SYSTSPRT DD SYSOUT=* 
/*  

The parameters are

  • the exported certificates high level qualifier. It will create data sets like COLIN.CERT.EXPORT.C14 . Once you have imported the certificates you need to delete these data sets.
  • the userid that owns the keyrings
  • the password for the exported certificates

The output data sets are

OUTPUT: this has data like

RACDCERT EXPORT(LABEL('AdcdShrCA')) -
CERTAUTH DSN('COLIN.CERT.EXPORT.C8')-
FORMAT(PKCS12DER) password('&PASSWORD')

IMPORT: This has data like

RACDCERT ADD('COLIN.CERT.EXPORT.C8') CERTAUTH -
TRUST withlabel('AdcdShrCA') PASSWORD('&PASSWORD')

RING: this has data like

RACDCERT ADDRING(RING) ID(START1)
RACDCERT ID(START1) CONNECT(RING(RING) –
  CERTAUTH DEFAULT usage(PERSONAL ) –
  LABEL(‘AdcdShrCA’) )

Migrating userids and profiles

RACF provides a DBSYNC rexx program to take two unloaded RACF databases and generate the RACF command statements to show the differences and help you convert from one to the other. One file will have “define resource XYZ” another file will have “delete resource XYZ”.

In theory you can just run the appropriate files and merge your old definitions into the new database. I have a lot of junk in my old database and wanted to be more selective as to what I copied across, and understand the differences.

I used the DBSYNC job to create the various files. I changed the DCB to DCB=(RECFM=VB,LRECL=600,BLKSIZE=6400)

The output of ALTFILE1 looks like

/*70:339:286*/ "altgroup PKIGRP   omvs( gid(0000990031))" 
/*75:373:315*/ "altuser ADCDA owner(IBMUSER  ) dfltgrp(TEST     ) noadsp n....
  "noseclabel"

I use ISPF View on the data set, so I do not accidentally change the output.

I run the Rexx exec RACFONEL (in user.*.clist). This creates one line in the file for each logical line, and sorts the file alphabetically. For example

altuser ADCDA owner(IBMUSER  ) dfltgrp(TEST     ) noa..... noseclabel

Depending on what I am doing, I select a subset of the records for example, for my userid and use those records.

The Rexx exec RACSPLIT does the following

  • Cosmetic edits to make fields align – for example change ‘ group’ to ‘group’, changes revoke to resume to make it easier to compare records
  • Splits each line so it fits more into columns 1-80. Sometimes the lines are longer because the definition requires a long string.
  • It appends the records into the save file

Example output

altuser ZWESIUSR owner(IBMUSER  ) - 
    dfltgrp(ZWEADMIN ) - 
    noadsp - 
    nospecial - 
    nooperations - 
    nogrpacc - 
    name(zzzzz) - 
    nodata - 
    noseclabel 
altuser ZWESVUSR omvs( uid(0000990016 ) home('/apps/zowe/v10/home/z.....

If I am happy with the output, delete the top records in the file, and save it.

Do the same for the old and new RACF DBSYNC files and then use ISPF edit compare function to show the difference.

Once you know what the difference are you can run the updates for the changes you want to make to your new database.

Which of my ADCD disks should I move to my SSD device?

I’m working on moving to a newer version of ADCD, but I do not have enough space for all of the ADCD disks, on my SSD drive, so I am using an external USB device. Which of my new files should I move off the USB drive onto my SSD device for best performance?

Background

How much free space do I have on my disk?

The command

df -P /home/zPDT

gave

Filesystem   1024-blocks     Used Available Capacity Mounted on
/dev/nvme0n1p5 382985776 339351984 24105908      94% /home/zPDT

This shows there is not much free space. What is using all of the space?

ls -lSr

the -S is sort by size largest first, the -r is reverse sort, so the largest comes last.

This showed me lots of old ADCD files which I could delete. After I deleted them, df -P showed the disk was only 69% full.

zPDT “disks”

Each device as seen by zPDT is a process. For example

$ps -ef |grep 5079
colin 5079 4792 0 10:21 ? 00:00:00 awsckd --dev=0A94 --cunbr=0001

So process with pid 5079 is running a program awsckd passing in the device number 0A94

Linux statistics

You can access Linux statistics under the /proc tree.

less /proc/5079/io

gave

rchar: 251198496
wchar: 79167416
syscr: 4525
syscw: 1403
read_bytes: 78671872
write_bytes: 78655488
cancelled_write_bytes: 0

rchar: characters read

The number of bytes which this task has caused to be read from storage. This is simply the sum of bytes which this process passed to read(2) and similar system calls. It includes things such as terminal I/O and is unaffected by whether or not actual physical disk I/O was required (the read might have been satisfied from pagecache).

wchar: characters written

The number of bytes which this task has caused, or shall cause to be written to disk. Similar caveats apply here as with rchar.

read_bytes: bytes read

Attempt to count the number of bytes which this process really did cause to be fetched from the
storage layer. This is accurate for block-backed filesystems.

write_bytes: bytes written

Attempt to count the number of bytes which this process caused to be sent to the storage layer.

How to find the hot files

Use the Linux command

grep read_bytes -r /proc/*/io |sort -k2,2 -g

This finds the read_bytes for each process. It then sorts numerically (-g) and displays the output. For example

/proc/5088/io:read_bytes: 55910400
/proc/5078/io:read_bytes: 61440000
/proc/5091/io:read_bytes: 72916992
/proc/5079/io:read_bytes: 78671872
/proc/5076/io:read_bytes: 138698752
/proc/5074/io:read_bytes: 321728512

You can then display the process information

ps -ef |grep 5074

Which gave

… awsckd –dev=0A80 –cunbr=0001

From the devmap ( or z/OS) device 0A80 is C4RES1.

The disks with the most read activity were (in decreasing order) C4RES1, C4SYS1, C4PAGA, USER02, C4CFG1, C4USS1

zpdt and Ubuntu 22.04 -it worked!

I installed Ubuntu 22.04 on an isolated hard disk drive and installed IBM Z Development and Test Environment Personal Edition 14.0. on it.

I followed the documentation. The executable was sudo ./zdt-install-pe

The first time it ran, I allowed it to configure the network. It then prompted me with

Preconfiguration steps …

32 bit support not installed
Some of the above software dependencies are not installed

Do you want the necessary Linux dependencies for the product IBM® ZD&T Personal Edition to be installed? By entering y, all required dependencies will be installed. The list of dependencies are mentioned in the Prerequisites. You need to have access to internet and software repository to install the dependencies. Otherwise, installation will complete without dependencies, and you need to install the dependencies manually. For more information about linux prerequisites, see: https://ibm.biz/zdt_prerequisites

Y

It then hung.

When I reran it, but did not allow it to configure the network, and it ran successfully to completion in under a minute. It could have been a wi-fi problem.

I had to install x3270 (sudo apt install x3270) before starting my z/OS system. My zPDT environment was on a removable hard disk drive, so I plugged it it, started it up, and my system came up with no problems.

Why is my z/OS IP address changing when using zPDT, and routing does not work?

I was looking into configuring IP V6 on my z/OS running on zPDT running on Linux. I could not understand why configuring the IP V6 link between Linux and z/OS was so difficult.

IP V6 address for use within a connection are like fe80::b0b6:daff:fe64:77f5 where b0b6:daff:fe64:77f5 is based on the MAC(hwaddr). On many systems, this value does not change across IPLs – and so most of the documentation uses the “constant” value.

The connection between Linux and z/OS is a “tap” interface (a kernel virtual device) which looks like an OSA adapter to z/OS.

I found a comment

Each TAP device has a random MAC address that is used as source address.

This explains why the connection was getting a different IP address every time I ipled.

On z/OS you defined a route using this IP address, for example

BEGINRoutes 
ROUTE 2001:db8::7/128 fe80::3f:67ff:fe08:51dc   IFPORTCP6   MTU 5000 
ENDRoutes

To get round this problem you need to explicitly define an address on Linux

sudo ip -6 addr  add fec0::cccc/64 dev tap1

where cccc is for my initials!

You then put this address into the z/OS routing statements.

BEGINRoutes 
ROUTE 2001:db8::7/128 fe80::cccc   IFPORTCP6   MTU 5000 
ENDRoutes

and it works first time!

Setting up IP V6 Linux to z/OS with ADCD

This post follows on from getting TCPIP to work with ADCD (zPDT and ZD&T) and allows me to FTP to z/OS from my Linux machine. There is a Q&A Has anyone configured z/OS running on ZD&T to support IPV6 protocol? but this was not enough information for me.

I’ve written about the IP V6 concepts and how they fit with z/OS.

You need to:

  1. Configure the Linux device map to add an OSA entry in the device map. You specify the path, and z/OS device addresses.
  2. Configure z/OS to support AF_NET6. You can make this change dynamically – or just re-ipl.
  3. Configure TCPIP to add an IFCONFIG6 entry. You need the z/OS AF_NET6 change before the IFCONFIG6 change is accepted. You need to restart TCPIP (or re-ipl).
  4. Configure a VTAM TRLE pointing to the devices defined in the devmap entry. This can be configured dynamically.
  5. Define a TCPIP interface, pointing to the PORTNAME of the TRLE definition. This defines a IP V6 address. The change can be configured dynamically.
  6. Test it!

Configuring the Linux Devmap

Create the Linux devmap entry

[manager]  # tap2 define network adapter (OSA) for communication with Linux
name awsosa 0019 --path=A2 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.25.1.6 
  # QDIO mode
device 408 osa osa 
device 409 osa osa 
device 40a osa osa

This session has IP V4 address 172.25.1.6, and uses device addresses 408,409 and 40a. It uses path A2.

Restart zD&T to pick up the changes, and re-ipl z/OS.

Configuring z/OS

You have to configure both z/OS and TCPIP to enable TCPIP V6 support.

Update BPXPRM

Use D OMVS,S to show the BPXPRMxx members being used.

Update bpxprmxx with AF_INIT6, by adding the following into a BPXPRMxx member.

NETWORK DOMAINNAME(AF_INET6) 
        DOMAINNUMBER(19) 
        MAXSOCKETS(50000) 
        TYPE(INET)

Check if your AF_INET is INET or CINET (Common INET is used when you have multiple TCPIP stacks), and specify the same value.

Re IPL.

If you mis configure it

DOMAINNUMBER value 19 is required ( see DOMAINNUMBER ) When I used a different value I got

BPXF202I DOMAIN AF_INET6 WAS NOT ACTIVATED FOR FILE SYSTEM
TYPE INET. RETURN CODE = 0000045A, REASON CODE = 743A0000

and, when TCPIP was started

EZZ0695I IPCONFIG6 STATEMENT ON LINE 1 NOT VALID – IPV6 SUPPORT IS NOT
ENABLED

Where 045A is EAFNOSUPPORT The address family is not supported.

Check AF_INET6 is configured

The command D OMVS,PFS gave me

PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
 INET      EZBPFINI   N/A       SOCKETS   A     2022/09/20 04.08.00     
 NFS       GFSCINIT   NFSC      REMOTE    A     2022/09/20 04.07.23     
 ZFS       IOEFSCM    N/A       LOCAL     A     2022/09/20 04.07.19     
 AUTOMNT   BPXTAMD    N/A       LOCAL     A     2022/09/20 04.07.19     
 UDS       BPXTUINT   N/A       SOCKETS   A     2022/09/20 04.07.19     
                                                                        
PFS TYPE  DOMAIN        MAXSOCK  OPNSOCK  HIGHUSED                      
 INET     AF_INET6       50000        5         5                      
          AF_INET         64000        8         8                      
 UDS      AF_UNIX         10000        2         2

Check AF_INET6 is in the list.

Configure TCPIP

I added “include user.Z25A.tcpparms(iconfig6)” into the TCPIP PROF.

This member had just

IPCONFIG6

Restart TCPIP.

The only change when TCPIP was restarted was the additional message

EZZ0300I OPENED INCLUDE FILE ‘USER.Z25A.TCPPARMS(ICONFIG6)’

Check the configuration

On Linux the find_io command gave

FIND_IO for "colinpaice@colinpaice" 
                                                                                                
         Interface Current     MAC     IPv4       IPv6           
 Path    Name      State       Address Address    Address        
------   --------- ----------- ------- ---------  ----------------  -------------- 
  A0     tap0      UP, RUNNING fa:...  10.1.1.1   fe80::f85c:c2ff:fe0a:1415%tap0  
  A1     tap1      UP, RUNNING 5e:...  172.26.1.6 fe80::5cda:64ff:feee:eeaa%tap1  
  A2     tap2      UP, RUNNING 4a:...  172.25.1.6 fe80::4850:5fff:fe5e:87c5%tap2

Check the interface is UP, RUNNING

Define a VTAM TRLE

You need to create a VTAM TRLE resource. I invented PORTCP, and created member user.z25a.vtamlst(TRLE).

OSATRL3 VBUILD TYPE=TRL                                                 
OSATRL3E TRLE LNCTL=MPC,READ=(0408),WRITE=(0409),DATAPATH=(040A),      X
               PORTNAME=PORTCP,                                        X
               MPCLEVEL=QDIO

This uses address 0408,0409, and 040a (matching the devmap entry above)

Use V net,act,id=trle to activate it.

Note: USER.Z25A.VTAMLST is in the DD concatenation for //VTAMLST.

Use D NET,TRL to display the defined TRLs. This showed

TRLE = OSATRL3E STATUS = NEVAC

Showing the TRLE above, and the status. It becomes ACTIVE when the TCPIP interface is activated.

Create the TCPIP interface definition

IP V6 uses an interface definition instead of a link and device.

  INTERFACE IFPORTCP6  DELETE 
  INTERFACE IFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTCP 
    INTFID 7:7:7:7  
    IPADDR FD00::67:1:1

I activated these using

  • v tcpip,,stop,ifportcp6
  • v tcpip,,obey,USER.Z25A.TCPPARMS(IFACE6)
  • You might need v tcpip,,stop,ifportcp6

I found it better to stop the interface before updating it, as sometimes the updates were not all made.

Once these definitions were activated, TSO NETSTAT HOME gave

IntfName:   IFPORTCP6
  Address:  fd00::67:1:1
    Type:   Global
    Flags:
  Address:  fe80::7:7:7:7
    Type:   Link_Local
    Flags:  Autoconfigured

This shows an address fd00::67:1:1 and address fe80::7:7:7:7 based on the INTFID. If you do not specify an INTFID you get a name like fe80::a2:a201:a2:a2a2, based on the chpid (value a2). If the chpid was changed, you would get a different IP address. You can see the chpid from the Linux from_io command, or the z/OS d NET,ID=OSATRL3E,E command.

I could not get any IP address specified in the IPADDR parameter, to work. I could ping to it, but there were no responses.

The interface gets a MAC address based on the CHPID value – for example MACADDRESS: 02A2A2A2A2A2.

NETSTAT ROUTE gave

IPV4 DESTINATIONS                                                     
DESTINATION        GATEWAY         FLAGS    REFCNT     INTERFACE      
127.0.0.1/32       0.0.0.0         UH       0000000000 LOOPBACK       
172.26.1.2/32      0.0.0.0         UH       0000000000 ETH1           
172.26.1.20/32     0.0.0.0         H        0000000000 EZAZCX         
172.26.1.20/32     0.0.0.0         H        0000000000 EZASAMEMVS     
IPV6 DESTINATIONS                                                     
DESTIP:   ::1/128                                                     
  GW:     ::                                                          
  INTF:   LOOPBACK6         REFCNT:  0000000000                       
  FLGS:   UH                MTU:     65535                            
DESTIP:   FD00::67:1/128                                              
  GW:     ::                                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000                       
  FLGS:   UHS               MTU:     1492                             
DESTIP:   FD00::67:1:1/128                                            
  GW:     ::                                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000    
  FLGS:   UH                MTU:     9000             
DESTIP:   FE80::7:7:7:7/128                   
  GW:     ::                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000       
  FLGS:   UH                MTU:     9000

The Linux find_io command gave

                                                                                              
      Interface  Current    MAC       IPv4        IPv6           
 Path Name       State      Address   Address     Address        
----- --------- ----------- --------  ----------  -------------- 
...      
. 
  A0 tap0       UP, RUNNING da:...    10.1.1.1    fe80::...tap0  
  A1 tap1       UP, RUNNING 92:...    172.26.1.6  fe80::...%tap1  
  A2 tap2       UP, RUNNING 42:...    172.25.1.6  fe80::...%tap2

Update the Linux route information

I did this to try to get the IPADDR to work. It did not work, and so is this is not needed.

sudo ip -6 route add fd00::6:1:1/128 dev tap2

Test it!

Use TSO NETSTAT HOME to find the IP V6 address. For example

Address: fe80::7:7:7:7, Type: Link_Local

On Linux use the find_io command to display information about the tunnels to z/OS. Find the tapn matching the chpid being used on z/OS.
Use the

ping fe80::7:7:7:7%tap2

command to send data to z/OS.

The response to the ping will be sent back down the connection the request arrived on.

You can use the tso netstat devlinks(intfname IFPORTCP6 command (where IFPORTCP6 is my interface) to display information about just the specified interface; for example Inbound packets, BytesIn, Outbound packets, BytesOut.

You can use

FTP fe80::7:7:7:7%tap2

then use

tso NETSTAT CONN
or
tso netstat conn (port 21

to see the connections.

You can use NETSTAT ND to display the neighbours. This gave me

Query Neighbor cache for fe80::6a:ffff:feaf:c0e4
IntfName: IFPORTCP6 IntfType: IPAQENET6
LinkLayerAddr: 026AFFAFC0E4 State: Reachable
Type: Host AdvDfltRtr: No

The value fe80::6a:ffff:feaf:c0e4 matches up with the value from find_io on Linux, and

02:6a:ff:af:c0:e4 matches up with the MAC address.

IP V6 concepts and using IP V6 with ADCD

This post follows on from getting TCPIP to work with ADCD (zPDT and ZD&T) and allow me to FTP to z/OS from my Linux machine. There is a Q&A Has anyone configured z/OS running on ZD&T to support IPV6 protocol? but this was not enough information for me.

Background

With IP V4 there is a limit of the number of IP addresses available. IP V6 has many addresses, and so this should not be a problem. There is no smooth migration from IP V4 to IPV6, it is more start with IP V4, run IP V4 and V6 at the same time, move stuff from IP V4 to IP V6, – and possibly (unlikely) run with just IP V6.

Wikipedia has many good articles

  • IP V6 in general
  • IP V4 uses addresses like 192.6.7.1. IP V6 uses addresses like 2001:0db8:0000:0000:0000:ff00:0042:8329 (=2001:db8::ff00:42:8329). See here.
  • Each IP V6 has a local address (link-local) fe80::….
  • An IP V6 address can have :: to mean replace with as many zeros as needed to make this a valid IP V6 address. So 2001:0db8:0000:0000:0000:ff00:0042:8329 can be written 2001:db8::ff00:42:8329. You can only have one :: in a value.
  • TCPIP can support IPV4 and IP V6 at the same time (dual stack)
  • You can wrap an IP V4 address into an IP V6. For example ::ffff:192.0.2.128
  • For security clients often get a “temporary” (or randomised) IP address instead of a hard coded address. This uses a randomiser function with a secret key. This IP address can expire, and a new(different) IP address obtained. This can make it hard(impossible) for a server to do a reverse DNS lookup. This temporary IP address is useful, as it means you cannot be tracked by your IP address.

Other information

  • Each IP node has a IP V4 address and an IP V6 address.
  • When defining connections between systems, it looks like you need at least one IP V4 route, and at least one IP V6 route – I could be wrong.
  • An IPV6 host usually has more than one IP address.

IPV6 has reserved IP ranges

  • 2001:db8::/32 Addresses used in documentation and example source code.
  • fe80::/10 are the link-local unicast [RFC4291] addresses. Addresses within this block should not appear on the public Internet. Your router should not externalise this.
  • fd00::/7 for private internets. These are the unique-local addresses [RFC4193]. Addresses within this block should not appear by default on the public Internet. This means you can use them within your organisation.
  • ffxx is for multicast to all links matching the address. For example ff02::5 is used by the dynamic routing protocol OSPF to say to all routers in the (local) network “hello – anyone there”.

Getting started

The end of each connection needs at least one IP address. If you have 5 connections, you will have at least 5 IP addresses

What is my IP address on Linux ?

You can use hostname -I

which gave me

192.168.1.223
10.1.1.1
172.24.1.6
172.22.1.6
2a00:23c5:8888:9999:0000:1111:2222:3333
2a00:23c5:9999:0000:1111:2222:3333:4444

You can also use ifconfig or ip addr show.

You can also use the z109x find_io command.

What is my IP address on z/OS?

You can use TSO NETSTAT HOME, or the operator command V TCPIP,,NETSTAT,HOME .

This gives information like

LinkName:   ETH1
  Address:  172.26.1.2
    Flags:  Primary
IntfName:   IFPORTCP6
  Address:  fc00::67:1:1
    Type:   Global
    Flags:
  Address:  fe80::a1:a101:a1:a1a1
    Type:   Link_Local
    Flags:  Autoconfigured

This shows

  • 172.26.1.2 an IP V4 address for use within a private network, for connection ETH1.
  • fc00::67:1:1 an IP V6, unique local address, for interface IFPORTCP6.
  • fe80::a1:a101:a1:a1a1a link-local address, for interface IFPORTCP6. The a1:a1… is based on the MAC address of the device. You can override this on z/OS by specifying the INTFID.
  • Note: If you do not specify the INTFID, it will default to the MAC address. If you reconfigure the system, you may get different MAC address, and so the IP address via this interface will change. By specifying the INTFID you can specify what the IP address for this interface, which will not change if the system is reconfigured.

What are the routes on my machine?

IP V4 On Linux ip route or ip -4 route

gives

default via 192.168.1.254 dev wlp4s0 proto dhcp metric 600 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1 
169.254.0.0/16 dev wlp4s0 scope link metric 1000 
172.25.1.0/24 dev tap2 proto kernel scope link src 172.25.1.6 
172.26.1.0/24 dev tap1 proto kernel scope link src 172.26.1.6 
172.27.1.0/24 via 172.25.1.6 dev tap2 
192.168.1.0/24 dev wlp4s0 proto kernel scope link src 192.168.1.223 metric 600

IP V6 on Linux ip -6 route gives

:1 dev lo proto kernel metric 256 pref medium
2a00:xxx:xxxx:xxxx::/64 dev wlp4s0 proto ra metric 600 pref medium
fe80::/64 dev tap0 proto kernel metric 256 pref medium
fe80::/64 dev tap1 proto kernel metric 256 pref medium
fe80::/64 dev tap2 proto kernel metric 256 pref medium
fe80::/64 dev wlp4s0 proto kernel metric 600 pref medium
default via fe80::966a:b0ff:fe85:54a7 dev wlp4s0 proto ra metric 20600 pref medium

On z/OS, TSO NETSTAT route or V tcpip,,netstat,route gives

IPv4 Destinations
Destination        Gateway         Flags    Refcnt     Interface
-----------        -------         -----    ------     ---------
Default            172.25.1.6      GS       0000000000 IFPORTCP
127.0.0.1/32       0.0.0.0         UH       0000000000 LOOPBACK
172.26.1.2/32      0.0.0.0         UH       0000000000 ETH1
IPv6 Destinations
DestIP:   ::1/128
  Gw:     ::
  Intf:   LOOPBACK6         Refcnt:  0000000000
  Flgs:   UH                MTU:     65535
DestIP:   fc00::67:1:1/128
  Gw:     ::
  Intf:   IFPORTCP6         Refcnt:  0000000000
  Flgs:   UH                MTU:     9000
DestIP:   fe80::a1:a101:a1:a1a1/128
  Gw:     ::
  Intf:   IFPORTCP6         Refcnt:  0000000000
  Flgs:   UH                MTU:     9000

find_io

FIND_IO for "colinpaice@colinpaice" 
                                                                                                
      Interface  Current MAC      IPv4      IPv6           
 Path Name       State   Address  Address   Address        
----- ---------- ------- ------- ---------- -------------- 
  A0 tap0 UP, RUNNING    ea:...  10.1.1.1   fe80::e8e8:69ff:fe20:435b%tap0  
  A1 tap1 UP, RUNNING    22:...  172.26.1.6 fe80::2090:14ff:fee0:5f20%tap1  
  A2 tap2 UP, RUNNING    22:...  172.25.1.6 fe80::2047:afff:fef7:1caf%tap2

Joining it all up

To FTP from Linux to z/OS, I use

ftp fe80::a1:a101:a1:a1a1%tap1

The fe80::a1:a101:a1:a1a1 came from the z/OS NETSTAT HOME, and is the z/OS end of the connection.

Using another interface (defined with the INTFID 7:7:7:7) I could use

ftp fe80::7:7:7:7%tap2

The ip -6 route command gave me

fe80::/64 dev tap0 proto kernel metric 256 pref medium
fe80::/64 dev tap1 proto kernel metric 256 pref medium
fe80::/64 dev tap2 proto kernel metric 256 pref medium
fe80::/64 dev wlp4s0 proto kernel metric 600 pref medium

So the request for FE80…. can be routed to any of these. I know that it was configured using tunnel interface tap1, so the address to use is fe80::a1:a101:a1:a1a1%tap1.

Once the connection to FTP was established, z/OS TSO NETSTAT CONN gave me 

FTPD1    0000003D ESTBLSH                             
  LOCAL SOCKET:   FE80::A1:A101:A1:A1A1..21           
  FOREIGN SOCKET: FE80::2090:14FF:FEE0:5F20..42572

The request is processed by z/OS address FE80::A1:A101:A1:A1A1 (port 21)

This ties up with DestIP: fe80::a1:a101:a1:a1a1/128 … Intf: IFPORTCP6 which shows the request came in on interface IFPORTCP6

The request came from FE80::2090:14FF:FEE0:5F20, which ties up with fe80::2090:14ff:fee0:5f20%tap1. The request came in over the tap1 interface.

Using eclipse based tools to z/OS

Eclipse based tools like z/OS Explorer and IBM Developer for z/OS, use a server on the z/OS system called RSED.

This is available on the ZD&T (ZPDT) system, but is hidden away.

You need to mount the file system

mount filesystem(‘FEK320.HALG320.ZFS’) mountpoint(‘/usr/lpp/IBM/zexpl/’)
mode(read)type(ZFS)

Then start the server

s rsed

It’s easy when you know how.

Using R_PKISERV PKI server Callable service.

I tried to use PKI Services to generate a certificate so I could do OCSP verification. I tried using the R_PKIServ Security Service Callable API. This ultimately failed because key generation with PKI Server is not supported on my zPDT system running z/OS on my Linux system. Below are some of the things I learned about using this interface.

Most of the documentation is there and complete, it assumes you are an expert in this area, so it is a bit tough when you are new to it.

I found there are two modes of operation, (this was not clear)

  1. one is the SAF interface, and is an API for issuing the RACDCERT requests – read up on the RACDCERT GENCERT(request-dataset-name) command,
  2. The other is to use the PKI server, and to store stuff in ICSF,and not use RACF.

My zPDT system does not support PKI to generate certificates, so I cannot comment on that.

The SAF/PKI mode of operation is determined by the SIGNWITH option.

  • SIGNWITH PKI: says use PKI,
  • SIGNWITH SAF:CERTAUTH/COLIN-CA says use SAF, and the specified CA certificate.

Options for Gencert

Table 2. CertPlist for GENCERT and REQCERT defines all the options for GENCERT. Many of them apply only to PKI. (The fields have “Only valid with PKI Services requests” in the field description.) Some parameters are used to defined the parameters of a certificate, other provide information about the certificate.

For SAF, these fields provide “other information”

  • DiagInfo – this is very helpful for diagnosing problems, it gives the name of the field causing problems, see below.
  • SignWith – this defines whether SAF or PKI is used. If SAF, this is the CA certificate.
  • Userid – which ID will own the certificate
  • Label – this is the name the certificate to be stored in the RACF database.

These fields provide information for the certificate

  • CommonName
  • PublicCert – this is a Base 64 encoded certificate request you want to sign and store in RACF
  • Title
  • OrgUnit (OU)
  • Org
  • Locality
  • StateProv
  • Country
  • KeyUsage – some values are valid with SAF
  • NotBefore
  • NotAfter
  • AltIPAddr
  • AltURI

It does not matter the order you specify these components. The CN that was generated came out as

CN=Colin.T=COLINTITLE.OU=OUSSS.O=SSS.C=GB

exactly the same as if you issued the RACDCERT GENCERT command.

Diagnostic information

You have to provide a field called DiagInfo. This has some very good diagnostic information, especially when you get a return code saying “one of your parameters is not supported”. For example I got

safrc 8 racfrc 8 racfrs 52, where 52 means Incorrect field value specified in CertPlist.

The DiagInfo field layout is

  • “DiagInfo ” eye catcher
  • an integer length of the following field
  • the additional information, in my case it was “SignWith”. I had specified SignWith:PKI which was not supported.

Once the field had

“Label” specified is already in use (IRRD111I)

so you can sometimes get the RACF (RACDCERT) error message as well.

SAF interface and Public Cert

You can use this interface with a certificate request.

My certificate request was in a file with a format like

—–BEGIN CERTIFICATE REQUEST—–
MII…

…
C/l/hL4HV/iU2iX8EFr3BPlA2A==
—–END CERTIFICATE REQUEST—–

I read in the data between the Begin certificate request and the End certificate request, and passed this in as the PublicCert.

Return codes

I got safrc=8, racfrc=8, racfrsn=28 (Certificate generation provider not available). This was caused when the R_PKIServ (IRRSPX00) was called with a CA_domain specified, and there as not a pkiservd with an environment file with a matching _PKISERV_CA_DOMAIN value



Using PKI Server with the HTTPD web interface.

This post follows on from configuring PKI Server, and explains how to configure the HTTPD server, explains how to use it, and gives some hints on debugging it when it goes wrong.

Having tried to get this working (and fixing the odd bug) I feel that this area is not as well designed as it could have been, and I could not get parts of it to work.

For example

  • You cannot generate browser based certificate request because the <keygen> html tag was removed around 2017, and the web page fails. See here. You can use 1-Year PKI Generated Key Certificate instead, so not a big problem now we know.
  • The TLS cipher specs did not have the cipher specs I was using.
  • I was expecting a simple URL like https://10.1.1.2/PKIServer/Admin. You have to use https://10.1.1.2/PKIServ/ssl-cgi-bin/camain.rexx, which exposes the structure of the files. You can go directly go to the Admin URL using https://10.1.1.2/PKIServ/ssl-cgi-bin/auth/admmain.rexx, which is not very elegant.
  • For an end user to request a certificate you have to use https://10.1.1.2/Customers/ssl-cgi-bin/camain.rexx.
  • There seem to be few security checks.
    • I managed to get into the administrative panels and display information using a certificate mapping to a z/OS userid, and with no authority!
    • There are no authority checks for people requesting a certificate. This may not be an exposure as the person giving the OK should be checking the request.
    • There were no security checks for administration functions. (It is easy to add them).
  • You can configure HTTPD to use certificates for authentication and fall back to userid and password.
  • There is no FallbackResource specified. This is a default page which is displayed if you get the URL wrong.
  • The web pages are generated dynamically. These feel over engineered. There was a problem with one of the supplied pages, but after some time trying to resolve the problem, I gave up.

I’ll discuss how to use the web interface, then I’ll cover the improvements I made to make the HTTP configuration files meet my requirements, and give some guidance on debugging.

You may want to use a HTTPD server just for PKI Server, or if you want to share, then I suggest you allocate a TLS port just for PKI Server.

URL

The URL looks like

https://10.1.1.2:443/PKIServ/ssl-cgi-bin/camain.rexx

where (see Overview of port usage below for more explanation)

  • 10.1.1.2 is the address of my server
  • port 443 is for TLS with userid and password authentication
  • PKIServ is the part of the configuration. If you have multiple CA’s this will be CA dependant.
  • ssl-cgi-bin is the “directory” where …
  • camain.rexx the Rexx program that does the work.

With https:10.1.1.2:443/Customers/ssl-cgi-bin/camain.rexx this uses the same camain.rexx as for PKIServ, but in the template for displaying data, it uses a section with the same name (Customers) as the URL.

Overview of port usage

There are three default ports set up in the HTTPD server for PKI Server. I found the port set-up confusing, and not well document. I’ve learned (by trial and error) that

  • port 80 (the default for non https requests) for unauthenticated requests, with no TLS session protection. All data flows as clear text. You many not want to use port 80.
  • port 443 (the default for https requests) for authentication with userid and password, with TLS session protection
  • port 1443 for certificate authentication, with TLS Session protection. Using https://10.1.1.2:443/PKIServ/clientauth-cgi/auth/admmain.rexx, internally this gets mapped to https://10.1.1.2:1443/PKIServ/clientauth-cgi-bin/auth/admmain.rexx. I cannot see the need for this port and its configuration.

and for the default configuration

  • port:/PKIServ/xxx is for administrators
  • port:/Customers/xxx is for an end user.

and xxx is

  • clientauth-cgi. This uses TLS for session encryption. Port 1443 runs with user SAFRunAs PKISERV. All updates are done using the PKISERVD userid, this means you do not need to set up the admin authority for each userid. There is no security checking enabled. I was able to process certificates from a userid with no authority!
  • ssl-cgi-bin. This uses port TLS and 443. I had to change the file to be SAFRunAs %%CERTIF%% as $$CLIENT$$ is invalid. You have to give each administrator ID access to the appropriate security profiles.
  • public-cgi. This is used by some insecure requests, such as print a certificate.

I think the only one you should use is ssl-cgi-bin.

Accessing the services

You can start using

These both give a page with

  • Administration Page. This may prompt for your userid and password, and gives you a page
  • Customer’s Home Page. This gives a page https://10.1.1.2/Customers/ssl-cgi-bin/camain.rexx? called PKI Services Certificate Generation Application. This has functions like
    • Request a new certificate using a model
    • Pickup a previously requested certificate
    • Renew or revoke a previously issued browser certificate

Note: You cannot use https://10.1.1.2:1443/PKIServ/ssl-cgi-bin/camain.rexx, as 1443 is not configured for this. I could access the admin panel directly using https://10.1.1.2:1443/PKIServ/ssl-cgi-bin/auth/admmain.rexx

I changed the 443 definition to support client and password authentication by using

  • SSLClientAuth Optional . This will cause the end user to use a certificate if one is available.
  • SAFRunAs %%CERTIF%% . This says use the Certificate authentication when available, if not prompt for userid and password.

Certificate requests

I was able to use the admin interface and display all certificate requests.

Request a new certificate using a model.

I tried to use the model “1 Year PKI SSL Browser Certificate“. This asks the browser to generate a private/public key (rather than the PKIServer generating them). This had a few problems. Within the page is a <KEYGEN> tag which is not supported in most browsers. It gave me

  • The field “Select a key size” does not have anything to select, or type.
  • Clicking submit request gave me IKYI003I PKI Services CGI error in careq.rexx: PublicKey is a required field. Please use back button to try again or report the problem to admin person to

I was able to use a “1 Year PKI Generated Key Certificate“

The values PKIServ and Customer are hard-coded within some of the files.

If you want to use more than one CA, read z/OS PKI Services: Quick Set-up for Multiple CAs. Use this book if you want to change “PKIServ” and “Customer”.

Colin’s HTTPD configuration files.

Because I had problems with getting the supplied files to work, I found it easier to restructure, parameterise and extend the provided files.

I’ve put these files up to github.

Basic restructure

I restructured and parametrised the files. The new files are

  • pki.conf. You edit this to define your variables.
  • 80.conf contains the definitions for a general end user, not using TLS. So the session is not encrypted. Not recommended.
  • 443.conf the definitions for the TLS port. You should not need to edit this while you are getting started. If you want to use multiple Certificate Authorities, then you need to duplicate some sections, and add definitions to the pki.conf file. See here.
  • 1443.conf the definitions for the TLS port for the client-auth path. You should not need to edit this while you are getting started. If you want to use multiple Certificate Authorities, then you need to duplicate some sections, and add definitions to the pki.conf file. See here.
  • Include conf/pkisetenv.conf to set some environment variables.
  • pkissl.conf. The SSL definitions have been moved to this file, and it has an updated list of cipher specs.

The top level configuration file pki.conf

The top level file is pki.conf. It has several sections

system wide

# define system wide stuff
# define my host name

Define sdn 10.1.1.2
Define PKIAppRoot /usr/lpp/pkiserv
Define PKIKeyRing START1/MQRING
Define PKILOG “/u/mqweb3/conf”

# The following is the default
#Define PKISAFAPPL “OMVSAPPL”
Define PKISAFAPPL “ZZZ”
Define serverCert “SERVEREC”
Define pkidir “/usr/lpp/pkiserv”

#the format of the trace entry
Define elf “[%{u}t] %E: %M”

Defined the CA specific stuff

# This defines the path of PKIServ or Customers as part of the URL
# This is used in a regular expression to map URLs to executables.
Define CA1 PKIServ|Customers
Define CA1PATH “_PKISERV_CONFIG_PATH_PKIServ /etc/pkiserv”

#Define the port for TLS
Define CA1Port 443

# specify the groups which can use the admin facility
Define CA1AdminAuth ” Require saf-group SYS1 “

other stuff

LogLevel debug
ErrorLog “${PKILOG}/zzzz.log”
ErrorLogFormat “${elf}”
# uncomment these if you want the traces
#Define _PKISERV_CMP_TRACE 0xff
#Define _PKISERV_CMP_TRACE_FILE /tmp/pkicmp.%.trc
#Define _PKISERV_EST_TRACE 0xff
#Define _PKISERV_EST_TRACE_FILE /tmp/pkiest.%.trc

#Include the files
Include conf/80.conf
Include conf/1443.conf
Include conf/443.conf

The TLS configuration file

The file 443.conf has several parts. It uses the parametrised values above, for example ${pkidir} is substituted with /usr/lpp/pkiserv/. When getting started you should not need to edit this file.

Listen ${CA1Port}
<VirtualHost *:${CA1Port}>

#define the log file for this port
ErrorLog “${PKILOG}/z${CA1Port}.log


DocumentRoot “${pkidr}”
LogLevel Warn
ErrorLogFormat “${elf}”

Include conf/pkisetenv.conf
Include conf/pkissl.conf
KeyFile /saf ${PKIKeyRing}
SSLClientAuth Optional
#SSLClientAuth None

RewriteEngine On

# display a default page if there are problems
# I created it in ${PKIAppRoot}/PKIServ,
# (/usr/lpp/pkiserv/PKIServ/index.html)
FallbackResource “index.html”

Below the definitions for one CA are defined. If you want a second CA, then duplicate the definitions,and change CA1 to CA2.

Notes on following section.

# Start of definitions for a CA

<IfDefine CA1>
SetEnv ${CA1PATH}
RewriteRule ¬/(${CA1})/ssl-cgi/(.) https://${sdn}/$1/ssl-cgi-bin/$2 [R,NE]
RewriteRule ¬/(${CA1})/clientauth-cgi/(.) https://${sdn}:1443/$1/clientauth-cgi-bin/$2 [R,NE,L]
ScriptAliasMatch ¬/(${CA1})/adm(.).rexx(.) “${PKIAppRoot}/PKIServ/ssl-cgi-bin/auth/adm$2.rexx$3
ScriptAliasMatch ¬/(${CA1})/Admin “${PKIAppRoot}/PKIServ/ssl-cgi-bin/auth/admmain.rexx”
ScriptAliasMatch ¬/(${CA1})/EU “${PKIAppRoot}/PKIServ/ssl-cgi-bin/camain.rexx”
ScriptAliasMatch ¬/(${CA1})/(public-cgi|ssl-cgi-bin)/(.*) “${PKIAppRoot}/PKIServ/$2/$3”
<LocationMatch “¬/(${CA1})/clientauth-cgi-bin/auth/pkicmp”>
CharsetOptions NoTranslateRequestBodies
</LocationMatch>
<LocationMatch “¬/(${CA1})/ssl-cgi-bin(/(auth|surrogateauth))?/cagetcert.rexx”>
Charsetoptions TranslateAllMimeTypes
</LocationMatch>
<IfDefine>

#End of definitions for CA1

Grouping the statements for a CA in one place means it is very easy to change it to use multiple CA’s, just repeat the section between <IfDefine…> and</IfDefine> and change CA1 to CA2.

The third part has definitions for controlling access to a directory. I added more some security information, and changed $$CLIENT$$ to %%CLIENT%%. This is a subset of the file, for illustration

# The User will be prompted to enter a RACF User ID
#and password and will use the same RACF User ID
# and password to access files in this directory
<Directory ${PKIAppRoot}/PKIServ/ssl-cgi-bin/auth>
AuthName AuthenticatedUser
AuthType Basic
AuthBasicProvider saf
Require valid-user

#Users must have access to the SAF APPLID to work
# ZZZ in my case
# it defaults to OMVSAPPL
<IfDefine PKISAFAPPL>
SAFAPPLID ${PKISAFAPPL}
</IfDefine>

# IBM Provided has $$CLIENT$$ where it should have %%CLIENT%%
# SAFRunAs $$CLIENT$$
# The following says use certificate if available else prompt for
# userid and password
SAFRunAs %%CERTIF%%
</Directory>…

Debugging hints and tips

I spent a lot of time investigating problems, and getting the definitions right.

Whenever I made a change, I used

s COLWEB,action=’restart’

to cause the running instance of HTTPD server to stop and restart. Any errors in the configuration are reported in the job which has the action=’restart’. It is easy to overlook configuration problems, and then spend time wondering why your change has not been picked up.

I edited the envvars file, and added code to rename and delete logs. For example rm z443.log.save, and mv z443.log z443.log.save .

I found it useful to have

<VirtualHost *:443>
DocumentRoot “${pkidr}”
ErrorLog “${PKILOG}/z443.log
ErrorLogFormat “${elf}”
LogLevel Warn


Where

  • Error logs is where the logs for this virtual host (port 443) are stored. I like to have one per port.
  • The format is defined in the variable Define elf “[%{c}t] %E: %M” in the pki.conf file. The c is compact time (2021-11-27 17:19:09). If you use %{cu}t you also get microseconds. I could not find where you just get the time, and no date.
  • LogLevel Warn. When trying to debug the RewriteRule and ScriptAlias I used LogLevel trace6. I also used LogLevel Debug authz_core_module:Trace6 which sets the default to Debug, but the authorization checking to Trace6.

With LogLevel Debug, I got a lot of good TLS diagnostics

Validating ciphers for server: S0W1, port: 443
No ciphers enabled for SSLV2
SSL0320I: Using SSLv3,TLSv1.0,TLSv1.1,TLSv1.2,TLSv1.3 Cipher: TLS_RSA_WITH_AES_128_GCM_SHA256(9C)
…
TLSv10 disabled, not setting ciphers
TLSv11 disabled, not setting ciphers
TLSv13 disabled, not setting ciphers
env_init entry (generation 2)
VirtualHost S0W1:443 is the default and only vhost

Then for each web session

Cert Body Len: 872
Serial Number: 02:63
Distinguished name CN=secp256r1,O=cpwebuser,C=GB
Country: GB
Organization: cpwebuser
Common Name: secp256r1
Issuer’s Distinguished Name: CN=SSCA256,OU=CA,O=SSS,C=GB
Issuer’s Country: GB
Issuer’s Organization: SSS
Issuer’s Organization Unit: CA
Issuer’s Common Name: SSCA256
[500865c0f0] SSL2002I: Session ID: A…AAE= (new)
[500865c0f0] [33620012] Peer certificate: DN [CN=secp256r1,O=cpwebuser,C=GB], SN [02:63], Issuer [CN=SSCA256,OU=CA,O=SSS,C=GB]

With LogLevel Trace6 I got information about the RewriteRule, for example we can see /Customers/EU was mapped to /usr/lpp/pkiserv/PKIServ/ssl-cgi-bin/camain.rexx

applying pattern ‘¬/(PKIServ|Customers)/clientauth-cgi/(.*)’ to uri ‘/Customers/EU’

AH01626: authorization result of Require all granted: granted
AH01626: authorization result of : granted

should_translate_request: r->handler=cgi-script r->uri=/Customers/EU r->filename=/usr/lpp/pkiserv/PKIServ/ssl-cgi-bin/camain.rexx dcpath=/

uri: /Customers/EU file: /usr/lpp/pkiserv/PKIServ/ssl-cgi-bin/camain.rexx method: 0 imt: (unknown) flags: 00 IBM-1047->ISO8859-1

# and the output

Headers from script ‘camain.rexx’:
Status: 200 OK
Status line from script ‘camain.rexx’: 200 OK
Content-Type: text/html
X-Frame-Options: SAMEORIGIN
Cache-Control: no-store, no-cache