How do I diff on Z/OS with Unix files and directories

I wanted to compare the contents of two directories in Unix System Services on z/OS before I merged them. This took me some time to do because the documented is lacking.

With ISPF you can use SUPERC (3.13), give it two PDSs and it shows you the differences.

On Unix there is the diff command. This can compare individual files, or directories. It can display just the changes, or the changes in context.

File a

AOnly 
line1
Aline2
line3

File b

line1 
Bline2
line3
BOnly
BOnly2

Using ispf edit compare to show differences

You can use diff to show the differences in two files, but it is not easy to understand. ISPF EDIT has the compare facility. If you know two files are different you can use

  • oedit /etc/zexpl/rseapi.env
  • use the primary command compare
    • enter the fully qualified name /u/ibmuser/zexpl/rseapi.env,, in the “Name . . . . .” field. if you specify +/filename the + means the same directory.
    • press pf3 and it will show the differences

Because I tend to remove comments to make it easier to see the content, I tend to use

  • oview /etc/ssh/sshd_config you get into ISPF edit, but no changes are saved.
  • Comments start with a #. x all;f ‘#’ all 1 1; f ‘ ‘ 1 1 all;del all nx removes comment lines and blank lines
  • reset to show the hidden lines
  • compare I then specify my version of the file, and see the changes.

  • Lines like ====== TrustedUserCAKeys /etc/ssh/user_ca_key.pub are from my copy.
  • Lines in green with a line number are in both files 000006 Subsystem sftp /usr/lib/ssh/sftp-server
  • Lines like .OAAAA UseDNS yes are from the base file

Update your version, make a copy of the original, then replace the original with your version.

diff -c1 a b – show the files and the changes in context

The line prefix for input file going to output file

  • – to be removed
  • ! to be changed
  • + to be added

The command diff -c1 a b gives

*** a Tue Aug 27 02:48:56 2024              
--- b Tue Aug 27 02:50:41 2024
***************
*** 1,4 ****
- AOnly
line1
! Aline2
line3
--- 1,5 ----
line1
! Bline2
line3
+ BOnly
+ BOnly2
  • *** a Tue Aug 27 02:48:56 2024 the first file name, and last changed date
  • — b Tue Aug 27 02:50:41 2024 the second file name, and last changed date
  • *** 1,4 **** the*** show it is file 1, lines 1 to 4
  • – AOnly this line is in file a is not in file b, so would need to be removed(-) from file a
  • ! Aline2 this line is in file b – but different
  • — 1,5 —- this is file b, lines 1 to 5
  • ! Bline2 this line is also in file a – but different
  • + BOnly this line is in file b and and was additional(+) to file a

When one file exists but is empty you get output like

*** /etc/resolv.conf Wed Mar  6 11:54:50 2024                         
--- /u/ibmuser/temp/resolv.conf Thu Dec 7 05:40:24 2023
***************
*** 0 ****
--- 1,2 ----
+ nameserver 127.0.0.1
+ TCPIPJOBNAME TCPIP

which follows the rules I explained above. *** 0 **** shows the content after line 0 is empty, because the next line is — 1,2 —- from the other file.

diff a b – show just the changes

gives

1d0             
< AOnly

3c2
< Aline2
---
> Bline2

4a4,5
> BOnly
> BOnly2

The output can be split into sections. The first line of each section is like

  • 1d0 the first line of a needs to be deleted from b, line 0
  • 3c2 line 3 of a is changed from line 2 of b
  • 4a4,5 lines 4,5 of b need to be added to a

The < and > tell you which file the data came from

When data is changed it gives the lines

  • < content of file a
  • output divider
  • > content of file b

When the data is in file a and not file b

  • < contents of file a

When the data is in file b and not file a

  • > contents of file b

diff -s dir1 dir2 compare the directory contents

If you specify -s, or just specify two directories, it compares the directory content.

You can use

diff -c1 dir1 dir2 

the -c1 to display the contents (how I like it).

With the directory entries you get records like

Only in /u/ibmuser/temp: test.tar 
Common subdirectories: /etc/wbem and /u/ibmuser/temp/wbem
Only in /etc: yylex.c
diff -c1 /etc/hosts /u/ibmuser/temp/hosts
*** /etc/hosts Wed Mar 6 11:06:55 2024
--- /u/ibmuser/temp/hosts Tue Feb 28 12:43:07 2023

You can find which files are missing from /etc , by using grep ‘Only in /u/ibmuser/temp’ on the output.

It shows the command used for the individual files, and the output

diff -c1 /etc/hosts /u/ibmuser/temp/hosts
*** /etc/hosts Wed Mar 6 11:06:55 2024
--- /u/ibmuser/temp/hosts Tue Feb 28 12:43:07 2023
...

diff -s -r dir1 dir2 compare the directory contents

The -r option displays the data recursively.

Using a VSAM file from another system.

I have been working with two levels of ADCD z/OS system, Z24C and Z25D. I want to be able to use VSAM files from the z24C level system on the z25D system.

With non VSAM files, it is easy. I can define an alias for a high level qualifier such as my userid COLIN which points to the user catalog with my data sets in it. It is a bit harder with VSAM files, especially where there is a file with the same name of both systems (such as CSF.CSFCKDS).

A VSAM PATH is an alias for VSAM files.

Conceptually the first step is

//IBMDEFP JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEF PATH -
(NAME(COLINQ.AUT420.AUT420D.CSI.Z24C) -
PATHENTRY( AUT420.AUT420D.CSI ) -
) -
CATALOG( USERCAT.Z24C.PRODS )
/*

File AUT420.AUT420D.CSI is in catalog USERCAT.Z24C.PRODS

The above JCL will create a name COLINQ.AUT420.AUT420D.CSI.Z24C which points to the file AUT420.AUT420D.CSI in catalog USERCAT.Z24C.PRODS. The entry COLINQ.AUT420.AUT420D.CSI.Z24C is put in the same catalog.

If you use ISPF 3.4 it will not find the dataset.

Create an alias for the High Level Qualifier

//IBMUSERT JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IDCAMS,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEFINE ALIAS (NAME(COLINQ) RELATE('USERCAT.Z24C.PRODS'))
/*

The above JCL will create an alias COLINQ, and says to find any datasets beginning with COLINQ go and look in catalog USERCAT.Z24C.PRODS.

To import the catalog into the current system

//IBMIMPC JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//CAT DD DISP=SHR,DSN=ADCD.LIB.JCL,VOL=SER=C4SYS1,
// UNIT=3390
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
IMPORT -
OBJECTS -
((USERCAT.Z24C.PRODS -
VOLUME(C4SYS1) -
DEVICETYPE(3390))) -
CONNECT -
CATALOG(CATALOG.Z25D.MASTER)
/*
//

The above JCL says import the catalog USERCAT.Z24C.PRODS on volume(C4SYS1), type (3390) into the (master) catalog CATALOG.Z25D.MASTER.

If the system needs to find USERCAT.Z24C.PRODS, it has enough information to be able to find it.

What you actually do

Now that you understand the process, the process you should follow is

  • Import the catalog into the current system.
  • Define an High Level Qualifier alias to point to the catalog. I might pick COLIN4C ( for the z24C system).
  • Create a path using COLIN4C as the high level qualifier of the data set, for each VSAM file.

You should then be able to see your data set in ISPF 3.4

To access the Z24C /u ZFS files system on the Z25D system I used

 IMPORT - 
OBJECTS -
((CATALOG.Z24C.MASTER -
VOLUME(C4SYS1) -
DEVICETYPE(3390))) -
CONNECT -
CATALOG(CATALOG.Z25D.MASTER)

DEFINE ALIAS (NAME(Z24CMAST) RELATE('CATALOG.Z24C.MASTER'))

DEFINE PATH -
(NAME(Z24CMAST.ZFS.USERS ) -
PATHENTRY( ZFS.USERS )) -
CATALOG(CATALOG.Z24C.MASTER)

In Unix I created a directory

mkdir /u/old

The mounted the file system in ISPF option 6 TSO

mount filesystem('Z24CMAST.ZFS.USERS') mountpoint('/u/old') type(ZFS)  
mode(read)

I could then access the files from /u/old/…

NO? Migrating an ADCD z/OS release:RACF

This post is a very quick post in response to a question about migrating your RACF data to a different system. I will update it when I have more time.

This is one of a series of posts about migrating to a newer level of ADCD.

This covers RACF.

A typical z/OS migration is to take your existing system and upgrade it. With ADCD the operating system is replaced and you have to move to it, taking your data with you.

You need to think about your RACF definitions, and migrates your specific data into the newer RACF database. This will include userid, profiles, permissions and digital certificates.

Some of the changes are good practice and clean up. For example

Instead of a definition CICSTS55.* in class(STARTED), the profile is CICSTS.*.* in class(STARTED). Using the scripts described in this blog you can update your old system before you move to the new system. 

I have put files into a github repository.

Migrating certificates and keyrings

The JCL EXPCERT runs a rexx exec which create commands to export your certificates, imports them (on the new system), and recreates the keyrings (on the new system).

The exec issues the RACDCERT command to list all the keyrings for a userid, and processes that output.

//IBMCERT  JOB 1,MSGCLASS=H 
// EXPORT SYMLIST=(*)
//ADCDA EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//OUTPUT DD DISP=SHR,DSN=COLIN.MIG.DATA(EXPCERTD)
//IMPORT DD DISP=SHR,DSN=COLIN.MIG.DATA(IMPCERTD)
//RING DD DISP=SHR,DSN=COLIN.MIG.DATA(RINGD)
//SYSEXEC DD DISP=SHR,DSN=USER.Z24C.CLIST
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
%LRING COLIN.CERT.EXPORT START1 PASS4ME4
/*
//S2 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSIN DD DISP=SHR,DSN=COLIN.MIG.DATA(EXPCERTD)
//SYSTSPRT DD SYSOUT=*
/*  

The parameters are

  • the exported certificates high level qualifier. It will create data sets like COLIN.CERT.EXPORT.C14 . Once you have imported the certificates you need to delete these data sets.
  • the userid that owns the keyrings
  • the password for the exported certificates

The output data sets are

OUTPUT: this has data like

RACDCERT EXPORT(LABEL('AdcdShrCA')) -
CERTAUTH DSN('COLIN.CERT.EXPORT.C8')-
FORMAT(PKCS12DER) password('&PASSWORD')

IMPORT: This has data like

RACDCERT ADD('COLIN.CERT.EXPORT.C8') CERTAUTH -
TRUST withlabel('AdcdShrCA') PASSWORD('&PASSWORD')

RING: this has data like

RACDCERT ADDRING(RING) ID(START1)
RACDCERT ID(START1) CONNECT(RING(RING) –
  CERTAUTH DEFAULT usage(PERSONAL ) –
  LABEL(‘AdcdShrCA’) )

Migrating userids and profiles

RACF provides a DBSYNC rexx program to take two unloaded RACF databases and generate the RACF command statements to show the differences and help you convert from one to the other. One file will have “define resource XYZ” another file will have “delete resource XYZ”.

In theory you can just run the appropriate files and merge your old definitions into the new database. I have a lot of junk in my old database and wanted to be more selective as to what I copied across, and understand the differences.

I used the DBSYNC job to create the various files. I changed the DCB to DCB=(RECFM=VB,LRECL=600,BLKSIZE=6400)

The output of ALTFILE1 looks like

/*70:339:286*/ "altgroup PKIGRP   omvs( gid(0000990031))" 
/*75:373:315*/ "altuser ADCDA owner(IBMUSER ) dfltgrp(TEST ) noadsp n....
"noseclabel"

I use ISPF View on the data set, so I do not accidentally change the output.

I run the Rexx exec RACFONEL (in user.*.clist). This creates one line in the file for each logical line, and sorts the file alphabetically. For example

altuser ADCDA owner(IBMUSER  ) dfltgrp(TEST     ) noa..... noseclabel

Depending on what I am doing, I select a subset of the records for example, for my userid and use those records.

The Rexx exec RACSPLIT does the following

  • Cosmetic edits to make fields align – for example change ‘ group’ to ‘group’, changes revoke to resume to make it easier to compare records
  • Splits each line so it fits more into columns 1-80. Sometimes the lines are longer because the definition requires a long string.
  • It appends the records into the save file

Example output

altuser ZWESIUSR owner(IBMUSER  ) - 
dfltgrp(ZWEADMIN ) -
noadsp -
nospecial -
nooperations -
nogrpacc -
name(zzzzz) -
nodata -
noseclabel
altuser ZWESVUSR omvs( uid(0000990016 ) home('/apps/zowe/v10/home/z.....

If I am happy with the output, delete the top records in the file, and save it.

Do the same for the old and new RACF DBSYNC files and then use ISPF edit compare function to show the difference.

Once you know what the difference are you can run the updates for the changes you want to make to your new database.

Which of my ADCD disks should I move to my SSD device?

I’m working on moving to a newer version of ADCD, but I do not have enough space for all of the ADCD disks, on my SSD drive, so I am using an external USB device. Which of my new files should I move off the USB drive onto my SSD device for best performance?

Background

How much free space do I have on my disk?

The command

df -P /home/zPDT

gave

Filesystem   1024-blocks     Used Available Capacity Mounted on
/dev/nvme0n1p5 382985776 339351984 24105908      94% /home/zPDT

This shows there is not much free space. What is using all of the space?

ls -lSr

the -S is sort by size largest first, the -r is reverse sort, so the largest comes last.

This showed me lots of old ADCD files which I could delete. After I deleted them, df -P showed the disk was only 69% full.

zPDT “disks”

Each device as seen by zPDT is a process. For example

$ps -ef |grep 5079
colin 5079 4792 0 10:21 ? 00:00:00 awsckd --dev=0A94 --cunbr=0001

So process with pid 5079 is running a program awsckd passing in the device number 0A94

Linux statistics

You can access Linux statistics under the /proc tree.

less /proc/5079/io

gave

rchar: 251198496
wchar: 79167416
syscr: 4525
syscw: 1403
read_bytes: 78671872
write_bytes: 78655488
cancelled_write_bytes: 0

rchar: characters read

The number of bytes which this task has caused to be read from storage. This is simply the sum of bytes which this process passed to read(2) and similar system calls. It includes things such as terminal I/O and is unaffected by whether or not actual physical disk I/O was required (the read might have been satisfied from pagecache).

wchar: characters written

The number of bytes which this task has caused, or shall cause to be written to disk. Similar caveats apply here as with rchar.

read_bytes: bytes read

Attempt to count the number of bytes which this process really did cause to be fetched from the
storage layer. This is accurate for block-backed filesystems.

write_bytes: bytes written

Attempt to count the number of bytes which this process caused to be sent to the storage layer.

How to find the hot files

Use the Linux command

grep read_bytes -r /proc/*/io |sort -k2,2 -g

This finds the read_bytes for each process. It then sorts numerically (-g) and displays the output. For example

/proc/5088/io:read_bytes: 55910400
/proc/5078/io:read_bytes: 61440000
/proc/5091/io:read_bytes: 72916992
/proc/5079/io:read_bytes: 78671872
/proc/5076/io:read_bytes: 138698752
/proc/5074/io:read_bytes: 321728512

You can then display the process information

ps -ef |grep 5074

Which gave

… awsckd –dev=0A80 –cunbr=0001

From the devmap ( or z/OS) device 0A80 is C4RES1.

The disks with the most read activity were (in decreasing order) C4RES1, C4SYS1, C4PAGA, USER02, C4CFG1, C4USS1

zpdt and Ubuntu 22.04 -it worked!

I installed Ubuntu 22.04 on an isolated hard disk drive and installed IBM Z Development and Test Environment Personal Edition 14.0. on it.

I followed the documentation. The executable was sudo ./zdt-install-pe

The first time it ran, I allowed it to configure the network. It then prompted me with

Preconfiguration steps …

32 bit support not installed
Some of the above software dependencies are not installed

Do you want the necessary Linux dependencies for the product IBM® ZD&T Personal Edition to be installed? By entering y, all required dependencies will be installed. The list of dependencies are mentioned in the Prerequisites. You need to have access to internet and software repository to install the dependencies. Otherwise, installation will complete without dependencies, and you need to install the dependencies manually. For more information about linux prerequisites, see: https://ibm.biz/zdt_prerequisites

Y

It then hung.

When I reran it, but did not allow it to configure the network, and it ran successfully to completion in under a minute. It could have been a wi-fi problem.

I had to install x3270 (sudo apt install x3270) before starting my z/OS system. My zPDT environment was on a removable hard disk drive, so I plugged it it, started it up, and my system came up with no problems.

Why is my z/OS IP address changing when using zPDT, and routing does not work?

I was looking into configuring IP V6 on my z/OS running on zPDT running on Linux. I could not understand why configuring the IP V6 link between Linux and z/OS was so difficult.

IP V6 address for use within a connection are like fe80::b0b6:daff:fe64:77f5 where b0b6:daff:fe64:77f5 is based on the MAC(hwaddr). On many systems, this value does not change across IPLs – and so most of the documentation uses the “constant” value.

The connection between Linux and z/OS is a “tap” interface (a kernel virtual device) which looks like an OSA adapter to z/OS.

I found a comment

Each TAP device has a random MAC address that is used as source address.

This explains why the connection was getting a different IP address every time I ipled.

On z/OS you defined a route using this IP address, for example

BEGINRoutes 
ROUTE 2001:db8::7/128 fe80::3f:67ff:fe08:51dc   IFPORTCP6   MTU 5000 
ENDRoutes 

To get round this problem you need to explicitly define an address on Linux

sudo ip -6 addr  add fec0::cccc/64 dev tap1

where cccc is for my initials!

You then put this address into the z/OS routing statements.

BEGINRoutes 
ROUTE 2001:db8::7/128 fe80::cccc   IFPORTCP6   MTU 5000 
ENDRoutes 

and it works first time!

Setting up IP V6 Linux to z/OS with ADCD

This post follows on from getting TCPIP to work with ADCD (zPDT and ZD&T) and allows me to FTP to z/OS from my Linux machine. There is a Q&A Has anyone configured z/OS running on ZD&T to support IPV6 protocol? but this was not enough information for me.

I’ve written about the IP V6 concepts and how they fit with z/OS.

You need to:

  1. Configure the Linux device map to add an OSA entry in the device map. You specify the path, and z/OS device addresses.
  2. Configure z/OS to support AF_NET6. You can make this change dynamically – or just re-ipl.
  3. Configure TCPIP to add an IFCONFIG6 entry. You need the z/OS AF_NET6 change before the IFCONFIG6 change is accepted. You need to restart TCPIP (or re-ipl).
  4. Configure a VTAM TRLE pointing to the devices defined in the devmap entry. This can be configured dynamically.
  5. Define a TCPIP interface, pointing to the PORTNAME of the TRLE definition. This defines a IP V6 address. The change can be configured dynamically.
  6. Test it!

Configuring the Linux Devmap

Create the Linux devmap entry

[manager]  # tap2 define network adapter (OSA) for communication with Linux
name awsosa 0019 --path=A2 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.25.1.6 
  # QDIO mode
device 408 osa osa 
device 409 osa osa 
device 40a osa osa 

This session has IP V4 address 172.25.1.6, and uses device addresses 408,409 and 40a. It uses path A2.

Restart zD&T to pick up the changes, and re-ipl z/OS.

Configuring z/OS

You have to configure both z/OS and TCPIP to enable TCPIP V6 support.

Update BPXPRM

Use D OMVS,S to show the BPXPRMxx members being used.

Update bpxprmxx with AF_INIT6, by adding the following into a BPXPRMxx member.

NETWORK DOMAINNAME(AF_INET6) 
        DOMAINNUMBER(19) 
        MAXSOCKETS(50000) 
        TYPE(INET) 

Check if your AF_INET is INET or CINET (Common INET is used when you have multiple TCPIP stacks), and specify the same value.

Re IPL.

If you mis configure it

DOMAINNUMBER value 19 is required ( see DOMAINNUMBER ) When I used a different value I got

BPXF202I DOMAIN AF_INET6 WAS NOT ACTIVATED FOR FILE SYSTEM
TYPE INET. RETURN CODE = 0000045A, REASON CODE = 743A0000

and, when TCPIP was started

EZZ0695I IPCONFIG6 STATEMENT ON LINE 1 NOT VALID – IPV6 SUPPORT IS NOT
ENABLED

Where 045A is EAFNOSUPPORT The address family is not supported.

Check AF_INET6 is configured

The command D OMVS,PFS gave me

PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
 INET      EZBPFINI   N/A       SOCKETS   A     2022/09/20 04.08.00     
 NFS       GFSCINIT   NFSC      REMOTE    A     2022/09/20 04.07.23     
 ZFS       IOEFSCM    N/A       LOCAL     A     2022/09/20 04.07.19     
 AUTOMNT   BPXTAMD    N/A       LOCAL     A     2022/09/20 04.07.19     
 UDS       BPXTUINT   N/A       SOCKETS   A     2022/09/20 04.07.19     
                                                                        
PFS TYPE  DOMAIN        MAXSOCK  OPNSOCK  HIGHUSED                      
 INET     AF_INET6       50000        5         5                      
          AF_INET         64000        8         8                      
 UDS      AF_UNIX         10000        2         2                      

Check AF_INET6 is in the list.

Configure TCPIP

I added “include user.Z25A.tcpparms(iconfig6)” into the TCPIP PROF.

This member had just

IPCONFIG6

Restart TCPIP.

The only change when TCPIP was restarted was the additional message

EZZ0300I OPENED INCLUDE FILE ‘USER.Z25A.TCPPARMS(ICONFIG6)’

Check the configuration

On Linux the find_io command gave

FIND_IO for "colinpaice@colinpaice" 
                                                                                                
         Interface Current     MAC     IPv4       IPv6           
 Path    Name      State       Address Address    Address        
------   --------- ----------- ------- ---------  ----------------  -------------- 
  A0     tap0      UP, RUNNING fa:...  10.1.1.1   fe80::f85c:c2ff:fe0a:1415%tap0  
  A1     tap1      UP, RUNNING 5e:...  172.26.1.6 fe80::5cda:64ff:feee:eeaa%tap1  
  A2     tap2      UP, RUNNING 4a:...  172.25.1.6 fe80::4850:5fff:fe5e:87c5%tap2 

Check the interface is UP, RUNNING

Define a VTAM TRLE

You need to create a VTAM TRLE resource. I invented PORTCP, and created member user.z25a.vtamlst(TRLE).

OSATRL3 VBUILD TYPE=TRL                                                 
OSATRL3E TRLE LNCTL=MPC,READ=(0408),WRITE=(0409),DATAPATH=(040A),      X
               PORTNAME=PORTCP,                                        X
               MPCLEVEL=QDIO                                            

This uses address 0408,0409, and 040a (matching the devmap entry above)

Use V net,act,id=trle to activate it.

Note: USER.Z25A.VTAMLST is in the DD concatenation for //VTAMLST.

Use D NET,TRL to display the defined TRLs. This showed

TRLE = OSATRL3E STATUS = NEVAC

Showing the TRLE above, and the status. It becomes ACTIVE when the TCPIP interface is activated.

Create the TCPIP interface definition

IP V6 uses an interface definition instead of a link and device.

  INTERFACE IFPORTCP6  DELETE 
  INTERFACE IFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTCP 
    INTFID 7:7:7:7  
    IPADDR FD00::67:1:1 

I activated these using

  • v tcpip,,stop,ifportcp6
  • v tcpip,,obey,USER.Z25A.TCPPARMS(IFACE6)
  • You might need v tcpip,,stop,ifportcp6

I found it better to stop the interface before updating it, as sometimes the updates were not all made.

Once these definitions were activated, TSO NETSTAT HOME gave

IntfName:   IFPORTCP6
  Address:  fd00::67:1:1
    Type:   Global
    Flags:
  Address:  fe80::7:7:7:7
    Type:   Link_Local
    Flags:  Autoconfigured

This shows an address fd00::67:1:1 and address fe80::7:7:7:7 based on the INTFID. If you do not specify an INTFID you get a name like fe80::a2:a201:a2:a2a2, based on the chpid (value a2). If the chpid was changed, you would get a different IP address. You can see the chpid from the Linux from_io command, or the z/OS d NET,ID=OSATRL3E,E command.

I could not get any IP address specified in the IPADDR parameter, to work. I could ping to it, but there were no responses.

The interface gets a MAC address based on the CHPID value – for example MACADDRESS: 02A2A2A2A2A2.

NETSTAT ROUTE gave

IPV4 DESTINATIONS                                                     
DESTINATION        GATEWAY         FLAGS    REFCNT     INTERFACE      
127.0.0.1/32       0.0.0.0         UH       0000000000 LOOPBACK       
172.26.1.2/32      0.0.0.0         UH       0000000000 ETH1           
172.26.1.20/32     0.0.0.0         H        0000000000 EZAZCX         
172.26.1.20/32     0.0.0.0         H        0000000000 EZASAMEMVS     
IPV6 DESTINATIONS                                                     
DESTIP:   ::1/128                                                     
  GW:     ::                                                          
  INTF:   LOOPBACK6         REFCNT:  0000000000                       
  FLGS:   UH                MTU:     65535                            
DESTIP:   FD00::67:1/128                                              
  GW:     ::                                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000                       
  FLGS:   UHS               MTU:     1492                             
DESTIP:   FD00::67:1:1/128                                            
  GW:     ::                                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000    
  FLGS:   UH                MTU:     9000             
DESTIP:   FE80::7:7:7:7/128                   
  GW:     ::                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000       
  FLGS:   UH                MTU:     9000                                

The Linux find_io command gave

                                                                                              
      Interface  Current    MAC       IPv4        IPv6           
 Path Name       State      Address   Address     Address        
----- --------- ----------- --------  ----------  -------------- 
...      
. 
  A0 tap0       UP, RUNNING da:...    10.1.1.1    fe80::...tap0  
  A1 tap1       UP, RUNNING 92:...    172.26.1.6  fe80::...%tap1  
  A2 tap2       UP, RUNNING 42:...    172.25.1.6  fe80::...%tap2  

Update the Linux route information

I did this to try to get the IPADDR to work. It did not work, and so is this is not needed.

sudo ip -6 route add fd00::6:1:1/128 dev tap2

Test it!

Use TSO NETSTAT HOME to find the IP V6 address. For example

Address: fe80::7:7:7:7, Type: Link_Local

On Linux use the find_io command to display information about the tunnels to z/OS. Find the tapn matching the chpid being used on z/OS.
Use the

ping fe80::7:7:7:7%tap2

command to send data to z/OS.

The response to the ping will be sent back down the connection the request arrived on.

You can use the tso netstat devlinks(intfname IFPORTCP6 command (where IFPORTCP6 is my interface) to display information about just the specified interface; for example Inbound packets, BytesIn, Outbound packets, BytesOut.

You can use

FTP fe80::7:7:7:7%tap2

then use

tso NETSTAT CONN
or
tso netstat conn (port 21

to see the connections.

You can use NETSTAT ND to display the neighbours. This gave me

Query Neighbor cache for fe80::6a:ffff:feaf:c0e4
IntfName: IFPORTCP6 IntfType: IPAQENET6
LinkLayerAddr: 026AFFAFC0E4 State: Reachable
Type: Host AdvDfltRtr: No

The value fe80::6a:ffff:feaf:c0e4 matches up with the value from find_io on Linux, and

02:6a:ff:af:c0:e4 matches up with the MAC address.

IP V6 concepts and using IP V6 with ADCD

This post follows on from getting TCPIP to work with ADCD (zPDT and ZD&T) and allow me to FTP to z/OS from my Linux machine. There is a Q&A Has anyone configured z/OS running on ZD&T to support IPV6 protocol? but this was not enough information for me.

Background

With IP V4 there is a limit of the number of IP addresses available. IP V6 has many addresses, and so this should not be a problem. There is no smooth migration from IP V4 to IPV6, it is more start with IP V4, run IP V4 and V6 at the same time, move stuff from IP V4 to IP V6, – and possibly (unlikely) run with just IP V6.

Wikipedia has many good articles

  • IP V6 in general
  • IP V4 uses addresses like 192.6.7.1. IP V6 uses addresses like 2001:0db8:0000:0000:0000:ff00:0042:8329 (=2001:db8::ff00:42:8329). See here.
  • Each IP V6 has a local address (link-local) fe80::….
  • An IP V6 address can have :: to mean replace with as many zeros as needed to make this a valid IP V6 address. So 2001:0db8:0000:0000:0000:ff00:0042:8329 can be written 2001:db8::ff00:42:8329. You can only have one :: in a value.
  • TCPIP can support IPV4 and IP V6 at the same time (dual stack)
  • You can wrap an IP V4 address into an IP V6. For example ::ffff:192.0.2.128
  • For security clients often get a “temporary” (or randomised) IP address instead of a hard coded address. This uses a randomiser function with a secret key. This IP address can expire, and a new(different) IP address obtained. This can make it hard(impossible) for a server to do a reverse DNS lookup. This temporary IP address is useful, as it means you cannot be tracked by your IP address.

Other information

  • Each IP node has a IP V4 address and an IP V6 address.
  • When defining connections between systems, it looks like you need at least one IP V4 route, and at least one IP V6 route – I could be wrong.
  • An IPV6 host usually has more than one IP address.

IPV6 has reserved IP ranges

  • 2001:db8::/32 Addresses used in documentation and example source code.
  • fe80::/10 are the link-local unicast [RFC4291] addresses. Addresses within this block should not appear on the public Internet. Your router should not externalise this.
  • fd00::/7 for private internets. These are the unique-local addresses [RFC4193]. Addresses within this block should not appear by default on the public Internet. This means you can use them within your organisation.
  • ffxx is for multicast to all links matching the address. For example ff02::5 is used by the dynamic routing protocol OSPF to say to all routers in the (local) network “hello – anyone there”.

Getting started

The end of each connection needs at least one IP address. If you have 5 connections, you will have at least 5 IP addresses

What is my IP address on Linux ?

You can use hostname -I

which gave me

192.168.1.223
10.1.1.1
172.24.1.6
172.22.1.6
2a00:23c5:8888:9999:0000:1111:2222:3333
2a00:23c5:9999:0000:1111:2222:3333:4444

You can also use ifconfig or ip addr show.

You can also use the z109x find_io command.

What is my IP address on z/OS?

You can use TSO NETSTAT HOME, or the operator command V TCPIP,,NETSTAT,HOME .

This gives information like

LinkName:   ETH1
  Address:  172.26.1.2
    Flags:  Primary
IntfName:   IFPORTCP6
  Address:  fc00::67:1:1
    Type:   Global
    Flags:
  Address:  fe80::a1:a101:a1:a1a1
    Type:   Link_Local
    Flags:  Autoconfigured

This shows

  • 172.26.1.2 an IP V4 address for use within a private network, for connection ETH1.
  • fc00::67:1:1 an IP V6, unique local address, for interface IFPORTCP6.
  • fe80::a1:a101:a1:a1a1a link-local address, for interface IFPORTCP6. The a1:a1… is based on the MAC address of the device. You can override this on z/OS by specifying the INTFID.
  • Note: If you do not specify the INTFID, it will default to the MAC address. If you reconfigure the system, you may get different MAC address, and so the IP address via this interface will change. By specifying the INTFID you can specify what the IP address for this interface, which will not change if the system is reconfigured.

What are the routes on my machine?

IP V4 On Linux ip route or ip -4 route

gives

default via 192.168.1.254 dev wlp4s0 proto dhcp metric 600 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1 
169.254.0.0/16 dev wlp4s0 scope link metric 1000 
172.25.1.0/24 dev tap2 proto kernel scope link src 172.25.1.6 
172.26.1.0/24 dev tap1 proto kernel scope link src 172.26.1.6 
172.27.1.0/24 via 172.25.1.6 dev tap2 
192.168.1.0/24 dev wlp4s0 proto kernel scope link src 192.168.1.223 metric 600 

IP V6 on Linux ip -6 route gives

:1 dev lo proto kernel metric 256 pref medium
2a00:xxx:xxxx:xxxx::/64 dev wlp4s0 proto ra metric 600 pref medium
fe80::/64 dev tap0 proto kernel metric 256 pref medium
fe80::/64 dev tap1 proto kernel metric 256 pref medium
fe80::/64 dev tap2 proto kernel metric 256 pref medium
fe80::/64 dev wlp4s0 proto kernel metric 600 pref medium
default via fe80::966a:b0ff:fe85:54a7 dev wlp4s0 proto ra metric 20600 pref medium

On z/OS, TSO NETSTAT route or V tcpip,,netstat,route gives

IPv4 Destinations
Destination        Gateway         Flags    Refcnt     Interface
-----------        -------         -----    ------     ---------
Default            172.25.1.6      GS       0000000000 IFPORTCP
127.0.0.1/32       0.0.0.0         UH       0000000000 LOOPBACK
172.26.1.2/32      0.0.0.0         UH       0000000000 ETH1
IPv6 Destinations
DestIP:   ::1/128
  Gw:     ::
  Intf:   LOOPBACK6         Refcnt:  0000000000
  Flgs:   UH                MTU:     65535
DestIP:   fc00::67:1:1/128
  Gw:     ::
  Intf:   IFPORTCP6         Refcnt:  0000000000
  Flgs:   UH                MTU:     9000
DestIP:   fe80::a1:a101:a1:a1a1/128
  Gw:     ::
  Intf:   IFPORTCP6         Refcnt:  0000000000
  Flgs:   UH                MTU:     9000

find_io

FIND_IO for "colinpaice@colinpaice" 
                                                                                                
      Interface  Current MAC      IPv4      IPv6           
 Path Name       State   Address  Address   Address        
----- ---------- ------- ------- ---------- -------------- 
  A0 tap0 UP, RUNNING    ea:...  10.1.1.1   fe80::e8e8:69ff:fe20:435b%tap0  
  A1 tap1 UP, RUNNING    22:...  172.26.1.6 fe80::2090:14ff:fee0:5f20%tap1  
  A2 tap2 UP, RUNNING    22:...  172.25.1.6 fe80::2047:afff:fef7:1caf%tap2  

Joining it all up

To FTP from Linux to z/OS, I use

ftp fe80::a1:a101:a1:a1a1%tap1

The fe80::a1:a101:a1:a1a1 came from the z/OS NETSTAT HOME, and is the z/OS end of the connection.

Using another interface (defined with the INTFID 7:7:7:7) I could use

ftp fe80::7:7:7:7%tap2

The ip -6 route command gave me

fe80::/64 dev tap0 proto kernel metric 256 pref medium
fe80::/64 dev tap1 proto kernel metric 256 pref medium
fe80::/64 dev tap2 proto kernel metric 256 pref medium
fe80::/64 dev wlp4s0 proto kernel metric 600 pref medium

So the request for FE80…. can be routed to any of these. I know that it was configured using tunnel interface tap1, so the address to use is fe80::a1:a101:a1:a1a1%tap1.

Once the connection to FTP was established, z/OS TSO NETSTAT CONN gave me

FTPD1    0000003D ESTBLSH                             
  LOCAL SOCKET:   FE80::A1:A101:A1:A1A1..21           
  FOREIGN SOCKET: FE80::2090:14FF:FEE0:5F20..42572    

The request is processed by z/OS address FE80::A1:A101:A1:A1A1 (port 21)

This ties up with DestIP: fe80::a1:a101:a1:a1a1/128 … Intf: IFPORTCP6 which shows the request came in on interface IFPORTCP6

The request came from FE80::2090:14FF:FEE0:5F20, which ties up with fe80::2090:14ff:fee0:5f20%tap1. The request came in over the tap1 interface.

Using eclipse based tools to z/OS

Eclipse based tools like z/OS Explorer and IBM Developer for z/OS, use a server on the z/OS system called RSED.

This is available on the ZD&T (ZPDT) system, but is hidden away.

You need to mount the file system

mount filesystem(‘FEK320.HALG320.ZFS’) mountpoint(‘/usr/lpp/IBM/zexpl/’)
mode(read)type(ZFS)

Then start the server

s rsed

It’s easy when you know how.

Using R_PKISERV PKI server Callable service.

I tried to use PKI Services to generate a certificate so I could do OCSP verification. I tried using the R_PKIServ Security Service Callable API. This ultimately failed because key generation with PKI Server is not supported on my zPDT system running z/OS on my Linux system. Below are some of the things I learned about using this interface.

Most of the documentation is there and complete, it assumes you are an expert in this area, so it is a bit tough when you are new to it.

I found there are two modes of operation, (this was not clear)

  1. one is the SAF interface, and is an API for issuing the RACDCERT requests – read up on the RACDCERT GENCERT(request-dataset-name) command,
  2. The other is to use the PKI server, and to store stuff in ICSF,and not use RACF.

My zPDT system does not support PKI to generate certificates, so I cannot comment on that.

The SAF/PKI mode of operation is determined by the SIGNWITH option.

  • SIGNWITH PKI: says use PKI,
  • SIGNWITH SAF:CERTAUTH/COLIN-CA says use SAF, and the specified CA certificate.

Options for Gencert

Table 2. CertPlist for GENCERT and REQCERT defines all the options for GENCERT. Many of them apply only to PKI. (The fields have “Only valid with PKI Services requests” in the field description.) Some parameters are used to defined the parameters of a certificate, other provide information about the certificate.

For SAF, these fields provide “other information”

  • DiagInfo – this is very helpful for diagnosing problems, it gives the name of the field causing problems, see below.
  • SignWith – this defines whether SAF or PKI is used. If SAF, this is the CA certificate.
  • Userid – which ID will own the certificate
  • Label – this is the name the certificate to be stored in the RACF database.

These fields provide information for the certificate

  • CommonName
  • PublicCert – this is a Base 64 encoded certificate request you want to sign and store in RACF
  • Title
  • OrgUnit (OU)
  • Org
  • Locality
  • StateProv
  • Country
  • KeyUsage – some values are valid with SAF
  • NotBefore
  • NotAfter
  • AltIPAddr
  • AltURI

It does not matter the order you specify these components. The CN that was generated came out as

CN=Colin.T=COLINTITLE.OU=OUSSS.O=SSS.C=GB

exactly the same as if you issued the RACDCERT GENCERT command.

Diagnostic information

You have to provide a field called DiagInfo. This has some very good diagnostic information, especially when you get a return code saying “one of your parameters is not supported”. For example I got

safrc 8 racfrc 8 racfrs 52, where 52 means Incorrect field value specified in CertPlist.

The DiagInfo field layout is

  • “DiagInfo ” eye catcher
  • an integer length of the following field
  • the additional information, in my case it was “SignWith”. I had specified SignWith:PKI which was not supported.

Once the field had

“Label” specified is already in use (IRRD111I)

so you can sometimes get the RACF (RACDCERT) error message as well.

SAF interface and Public Cert

You can use this interface with a certificate request.

My certificate request was in a file with a format like

—–BEGIN CERTIFICATE REQUEST—–
MII…


C/l/hL4HV/iU2iX8EFr3BPlA2A==
—–END CERTIFICATE REQUEST—–

I read in the data between the Begin certificate request and the End certificate request, and passed this in as the PublicCert.

Return codes

I got safrc=8, racfrc=8, racfrsn=28 (Certificate generation provider not available). This was caused when the R_PKIServ (IRRSPX00) was called with a CA_domain specified, and there as not a pkiservd with an environment file with a matching _PKISERV_CA_DOMAIN value