Configuring TCPIP on z/OS is as easy as — um.

When I was young I was told that if you cannot say something good about anyone, then do not say anything; or to put it another way; focus on the positive and not the negative. Here goes…

Configuring TCPIP on z/OS

In general, some of the requirements for making changes include:

  • Being able to undo or back-out a change
  • Make the smallest change necessary
  • Changes should be simple to implement
  • Have changes being worked on in parallel
  • You implement the same change in production that was tested in earlier stages, so no typing.
  • Only authorised people are allowed to make changes.

What is good about TCPIP configuration

  • When I configured my first TCPIP system, I thought that it was easy to configure. Since then, working with more complex changes, my views have changed.
  • You can use system symbols, so you can have one configuration across many systems, and exploit symbols like SYSNAME=MVS1, or SYSNAME=MVS2. Of course where there are multiple instances of &SYSNAME – they all get the same value, which makes it even easier to configure!

How does it work

In the TCPIP procedure JCL are statements like

//SYSTCPD DSN=ADCD.Z25A.TCPPARMS(TCPDATA),DISP=SHR 
//PROFILE DISP=SHR,DSN=ADCD.Z25A.TCPPARMS(PROF2) 

SYSTCPD is used by clients or servers. This has information about how to use TCPIP: (including which TCPIP, if there is more than one).

DOMAINORIGIN  DAL-EBIS.IHOST.COM     
TCPIPJOBNAME TCPIP     
DATASETPREFIX TCPIP     

PROFILE is used to configure the links, connections etc of the TCPIP address space. The remainder of the post is about configuring this dataset.

In the profile you can define:

  • IP V4 route statements
  • IP V6 route statements
  • Which hardware connections to use, and what IP addresses these have.
  • Reserved ports
  • Startup trace configuration.

Conceptually this profile is one large file, but you can use includes such as

include adcd.Z25A.tcpparms(zconnect)

to include specific files or members. But you cannot have

BEGINRoutes 
include user.Z25A.tcpparms(routeinc) 
ENDRoutes 

because each major statement has to be totally within one file.

Operator commands to change the configuration

As well as the profile you can alter the configuration by using the operator command, such as

v tcpip,,syntax,user.z25a.tcpparms(iface6)
v tcpip,,obey,user.z25a.tcpparms(iface6)

The syntax command checks the syntax of the statements, and allows you to fix any syntax problems before you use the OBEY command.

Note: The syntax check does not check for consistency or pre requirements. I had a request which passed the syntax check but failed because a VTAM resource was not present.

You can have include files like

 INTERFACE IFPORTCP6  DELETE 
 INTERFACE IFPORTCP6 
   DEFINE IPAQENET6 
   CHPIDTYPE OSD 
   PORTNAME PORTCP 
   INTFID 7:7:7:7 
                                                                          
 BEGINRoutes 
 ; Destination      SubnetMask    FirstHop LinkName    Size 
 ROUTE 172.25.1.0   255.255.255.0  =       IFPORTCP    MTU 1492 
 ROUTE DEFAULT      172.25.1.6             IFPORTCP    MTU 1492 
 ROUTE 2a00::/64                           IFPORTCP6   MTU 1 
 ENDRoutes 
 START IFPORTCP6 

This is typical of the documentation, as it is a nice self contained example. It:

  • deletes an interface
  • defines an interface
  • defines some routes – how to route packets to different systems
  • starts the interface

This is OK for a simple, one user, type of system.

You can use commands like

v tcpip,,start…
v tcpip,,stop...

to manage resources.

The command syntax is a bit strange, rather than v tcpip,start…, or v tcpipprd,start.. you have to specify v tcpip,tcpipprd,start – passing the name of the TCPIP job as the second parameter.

Change control

Simplistic change control

You can change the file, and restart TCPIP to pick up the changes.

Better change control

To make a change

If you put sections in a file or member, you can change the member and use obeyfile command on the file. For example have a file called user.tcpip(route1)

  BEGINRoutes 
  ; Destination        SubnetMask    FirstHop   LinkName    Size 
    ROUTE 172.25.1.0     255.255.255.0    =       IFPORTCP    MTU 1492 
    ROUTE DEFAULT6       fe00::                     IFPORTCP6   MTU 1492 
  ENDRoutes

To make a change, (to replace all the static routes)

  • you copy this to user.tcpip.old(route1).
  • make the change
  • use v tcpip,,syntax,user.tcpip(route1)
  • use v tcpip,,obey,user.tcpip(route1)

If it does not work, you use obey,user.tcpip.old(route1) to undo the change, and then fix the problem.

Two stage changes

Some changes need a delete followed by a define. There is no define replace.

If you have a delete… define… in one file, this may fail because the delete is asynchronous, and the operation is ongoing after the delete command has finished. The define then fails because the object still exists.

The best way of configuring seems to be:

  • Have the define statement in one file, which can be included in the TCPIP startup profile, or used in an OBEY command;
  • Have a file with the matching delete in a file which is used in an obey command
  • At startup the define file is used.
  • To modify it, use two commands, obeyfile… delete, obeyfile… define .

Production change control

Things you may need to consider when making changes:

  • There are often changes made in parallel, for example from different departments. Some of these changes can be made independently of each other. Others changes, like beginroutes… endroutes, need to be coordinated, and done serially.
  • If you are using system variables, for example a statement like
    • HOME &HOMEIPADDRESS1 ETH1,
  • you will need to define/change the symbol on each system and activate the resource.

Where you need to be careful.

The size of the change.

When you define static routes (between BEGINROUTES and ENDRoutes) if you want to change it, you replace all of the definitions; you cannot add or delete individual routes.

Unauthorised changes.

It seems that a userid which is authorised to issue the OBEY command, can issue the command against any file, so could use

VARY TCPIP,,OBEY,EVIL.PERSON.CONFIG(change) ,

and so someone can issue an an unauthorised change. With VTAM, an authorised operator, can only activate a member in a PDS – not use a random PDS member. You can control who can update datasets in the VTAMLST concatentation. This gives better protection.

To give a userid OBEYFILE access use RACF definitions

RDEFINE OPERCMDS MVS.VARY.TCPIP.OBEYFILE UACC(NONE)
PERMIT MVS.VARY.TCPIP.OBEYFILE ACCESS(CONTROL) CLASS(OPERCMDS) GROUP(CSYSPR)

This command seems to give authority to the obeyfile command in all address spaces, eg TCPIP, TCPIP1 etc.

I could not find a way of giving a userid permission on one TCPIP image, and no permission on another TCPIP image.

It is hard to make a small change

I wanted to make a small change to the reserved ports.

If this was VTAM, with JCL like

//VTAMLST DD DISP=SHR,DSN=USER.Z25A.VTAMLST
//        DD DISP=SHR,DSN=ADCD.Z25A.VTAMLST       

where my local customising is in USER.VTAMLST and the IBM provided customising is in ADCD.VTAMLST. To make a change, I add the entry to USER.VTAMLST. I do not change the IBM provided stuff. When I upgrade to a newer level, I use the updated version of ADCD.VTAMLST and my changes are still in USER.VTAMLST.

With the TCPIP approach, I have to change the IBM configuration, if only to point to my user.TCPPARMS file.

I had to update ADCD.TCPPARMS(PROF2). When a new level is available, I have to look at the dates, and compare the members, and reapply my changes! Not just a simple step.

If it had been me, I would have conditional includes so you would have

include USER.Z25A.TCPPARMS(ports) cond
include ADCD.Z25A.TCPPARMS(ports) required

where I put my configuration of TCPIP port numbers in the user…tcpparms(ports). If this member is not found, it produces a warning. When adcd…tcpparms(ports) is processed, it adds to my port definitions. If this file does not exist, and required is specified, then TCPIP does not start up.

Of course TCPIP configuration would also have to change to allow two or more BEGINROUTES…ENDROUTES sections.

Overall

Overall, the configuration is not too bad. I think VTAM configuration works well. Once you are used to the different syntax used by TCPIP commands, TCPIP is OK, but it does feel like extra typing! I would rather have “add route” and “delete route” instead of having to replace all of the static routes. It goes against the “minimum change” aspect of change control.

I do wonder why TCPIP did a completely different configuration interface to VTAM – but it will not change now.

Collecting a tcpip packet trace on z/OS.

I needed to collect a packet trace to see the details of a ping request. There is a lot of good documentation, but it is spread around, and not linked up.

It looks like a packet trace is associated with an interface or a device/link pair (but it may not be).

Thanks to Erik Janssen who explained that you can format a packet trace suitable for Wireshark. I’ve added his comments below; and to Peter Vels who pointing out some typos which meant some of the the commands did not work!

Overview

  • The TCPIP trace uses z/OS CTRACE capabilities, which requires a procedure (or job) to write trace data into a file.
  • You need some a procedure (or a job) called the trace write, which CTRACE uses to write to a dataset.
  • You need to start the trace writer
    • You can do this with parmlib members,
    • or reply to a WTOR
  • You need to enable the TCPIP trace
  • Recreate the problem
  • Disable the TCPIP trace
  • Stop the trace writer
    • Either with a parmlib member
    • or with a WTOR
  • Use IPCS to format the trace

Create a procedure to collect the trace

See IBM documentation. My JCL is

//CTWTR PROC 
//DELETE  EXEC PGM=IEFBR14 
//TRCOUT01 DD DSNAME=IBMUSER.CTRACE1, 
// SPACE=(CYL,(10),,CONTIG),DISP=(MOD,DELETE) 
//* 
//IEFPROC EXEC PGM=ITTTRCWR,TIME=999 
//TRCOUT01 DD DSNAME=IBMUSER.CTRACE1, 
// SPACE=(CYL,(10),,CONTIG),DISP=(NEW,CATLG) 
//SYSPRINT DD SYSOUT=* 

Notes:

  • You hard code the data set name, you cannot use symbols
  • You cannot pass parameters to it.
  • Do not specify DCB information, the system determines the best blocking.
  • You can specify more than one trace output data set
  • You might want use multiple procedures such as SYS1TCP1, SYS2TCP2, SYS1TCPP etc.

Start the trace writer

Start the trace writer by hand

TRACE CT,WTRSTART=CTWTR

where CTWTR is the name of your procedure.

Activate the trace collecting for the component

TRACE CT,ON,COMP=SYSTCPDA,SUB=(TCPIP) 

Which then issues WTOR and waits for you to respond to it.

R nn,WTR=CTWTR,END 

By having a WTOR you cannot easily automate the command.

Start the trace by parmlib member

TRACE CT,ON,COMP=SYSTCPDA,SUB=(TCPIP),PARM=name

Name is a member of the parmlib concentration. The name must begin with CT. My member is CTTCPS ( CT TCP Start).

This member has

TRACEOPTS                         
  WTRSTART(CTWTR) WRAP            
  ON  WTR(CTWTR)                  

It is one command to issue. It can be used in automation or scripts because it does not use WTOR.

Once the trace has been started

You can use the DISPLAY TRACE z/OS operator command to display the status

D TRACE,WTR=ALL

gave me

  WRITER  STATUS    HEAD  COMPONENT  SUBNAME                      
 --------------------------------------------------------------   
 CTWTR    ACTIVE          SYSTCPDA  TCPIP                         

d trace,comp=all
or
d trace,comp=systcpda

gave me

 COMPONENT     MODE BUFFER HEAD SUBS 
-------------------------------------
 SYSTCPDA      OFF         HEAD    1 
    NO HEAD OPTIONS                  

Define the trace to TCPIP

V TCPIP,TCPIP,PKTTRACE,ON,…. 

Depending on your system you may need to filter what is collected. If this is a one person z/OS image, you might not need to filter.

You can trace by

  • IP address – if the source or target matches. This means you can capture the trace for just one machine.
  • Interface name
  • Link name
  • Destination port number
  • Source port
  • Protocol TCP, ICMP, UDP

You can trace by having the trace data in a file PKTTRON

PKTTRACE 
   INTFNAME=IFPORTCP6 
   PROT=ICMP 
   ON 

and member PKTTROFF

PKTTRACE 
   INTFNAME=IFPORTCP6 
   PROT=ICMP 
   OFF

and use

v tcpip,,obeyfile,USER.Z25A.TCPPARMS(PKTTRON)

or use the trace command, such as

V TCPIP,TCPIP,PKTTRACE,ON,IPADDR=xx.xxx.xxx.xxx

Is trace enabled?

Surprisingly the display trace command (D TCPIP,,DISPLAY,TRACE) does not display the trace. This command is used to display information about network management applications that use the real-time application-controlled TCP/IP trace network management interface (NMI) to obtain real-time network management data from the TCP/IP stack.

You have to display information about the devices or interfaces.

tso netstat devlinks(intfn ifportcp6

Packet Trace Setting:

   Isolate: No                      OptLatencyMode: No
   TempPrefix: All
 Packet Trace Setting:
  Protocol: ICMP              TrRecCnt: 00000000  PckLength: FULL
  Discard:  NONE
  SrcPort:  *                 DestPort: *         PortNum: *
  IpAddr:   *                 SubNet:   *

If there is no trace, then the Packet Trace section is absent.

Stop the TCPIP trace

You can do the the opposite of when you turned it on, for example

V TCPIP,TCPIP,PKTTRACE,OFF,IPADDR=xx.xxx.xxx.xxx

or use an obeyfile, such as member PKTTROFF (see above).

Stop the trace writer

You can either use

  • TRACE CT,ON,COMP=SYSTCPDA,SUB=(TCPIP) 
  • R nn,WTR=DISCONNECT,END
  • TRACE CT,WTRSTOP=CTWTR

Or

TRACE CT,OFF,COMP=SYSTCPDA,SUB=(TCPIP),PARM=CTTCPP

Where parmlib member CTTTCPP has

TRACEOPTS WTRSTOP(CTWTR) 
  OFF 

If you get

ITT038I NONE OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE
SUCCESSFULLY EXECUTED.

Then the trace is still active. Check you have stopped collecting the TCPIP trace, and you have issued the stop writer request.

You have to issue

TRACE CT,OFF,COMP=SYSTCPDA,SUB=(TCPIP),PARM=CTTCPP

a second time for the trace to stop, and the writer to stop.

The data set you specified in the WTR JCL may have some data in it.

You need to use IPCS to format it.

  • IPCS option 0 (Specify Defaults) and enter DSNAME(‘IBMUSER.CTRACE1’)  
  • IPCS option 6 (IPCS Subcommand Entry) and enter

This gives output like the following for PING6 to fe80::7:7:7:7.

From Interface    : IFPORTCP6        Device: QDIO Ethernet6   Full=104 
 Tod Clock        : 2022/09/24 11:30:32.614976                Intfx: 81
 Segment #        : 0                Flags:  Ping In                   
 Source           : fe80::4081:88ff:fec5:b624                          
 Destination      : fe80::7:7:7:7                                      
                                     Asid: 0035 TCB: 00000000          
 QID              : 1                                                  
IpHeader: Version : 6                Header Length: 40                 
 Class:           : 00               Flow: 03CE34                      
 Payload Length   : 64                                                 
 Hops             : 64               Protocol: ICMPv6                  
 Source           : fe80::4081:88ff:fec5:b624                          
 Destination      : fe80::7:7:7:7                                      
                                                                       
ICMPv6                                                                 
 Type/Code        : 80/0             ECHO Request                      
 CheckSum         : 1A35 FFFF                                          
 Id               : 0004             Seq: 1                            
 Echo Data        : 56                                                  

Notes:

  • It says From Interface. This shows data was coming from the interface into TCPIP
  • Flags: Ping In this was ping request
  • It came from fe80::4081:88ff:fec5:b624
  • Heading towards fe80::7:7:7:7 on z/OS
To Interface      : IFPORTCP6        Device: QDIO Ethernet6   Full=104  
 Tod Clock        : 2022/09/24 11:30:32.615816                Intfx: 81 
 Segment #        : 0                Flags:  Ping Out                   
 Source           : fe80::7:7:7:7                                       
 Destination      : fe80::4081:88ff:fec5:b624                           
                                     Asid: 0035 TCB: 00000000           
 Next Hop         : fe80::4081:88ff:fec5:b624                           
IpHeader: Version : 6                Header Length: 40                  
 Class:           : 00               Flow: 03CE34                       
 Payload Length   : 64                                                  
 Hops             : 255              Protocol: ICMPv6                   
 Source           : fe80::7:7:7:7                                       
 Destination      : fe80::4081:88ff:fec5:b624                           
                                                                        
ICMPv6                                                                  
 Type/Code        : 81/0             ECHO Reply                         
 CheckSum         : 1935 FFFF                                           
 Id               : 0004             Seq: 1                             
 Echo Data        : 56                                                                                                                                         

Notes:

  • It says To Interface. This shows data was going to the interface from TCPIP
  • Flags: Ping Out this was ping response
  • It came from fe80::7:7:7:7 on z/OS
  • Heading towards fe80::4081:88ff:fec5:b624. This is the same address as the input request.

The above entries are as expected.

I set up an IP address for my IP V6 interface and did a ping to it (ping6 -I tap2 2000::67:1:1:).

This gave me a trace entry

From Interface    : IFPORTCP6        Device: QDIO Ethernet6   Full=104 
 Tod Clock        : 2022/09/24 11:30:25.470408                Intfx: 81
 Segment #        : 0                Flags:  Ping In                   
 Source           : 2a00:23c5:978f:6e01:307a:2e51:9020:afc5            
 Destination      : 2000::67:1:1
...                                       

Notes:

  • It says From Interface. This shows data was coming from the interface into TCPIP
  • Flags: Ping In this was ping request
  • It came from 2a00:23c5:978f:6e01:307a:2e51:9020:afc5. This is different to the previous address of fe80::4081:88ff:fec5:b624 – on the same machine
  • Heading towards 2000::67:1:1. This tells us the data came in on the IFPORTCP6 interface as before.
  • There was no return Ping Out record, so the request was lost in TCPIP on z/OS.

Using Wireshark to look at the packet flow.

Getting the data into Wireshark format.

Wireshark is an excellent tool for doing network trace, and decoding the flows. You can also is it to process the CTRACE data. Thanks for Erik Janssen for the instructions below on how to get from CTRACE to Wireshark.

You need to use the CTRACE SNIFFER option. This writes to the file with DDNAME of SNIFFER.

Allocate a VB 8000 LRECL dataset and allocate it to a ddname, for example SNIFFER.
ALLOC DD(SNIFFER) DS(<some dsname>) SHR
From IPCS you can now:
CTRACE COMP(SYSTCPDA) OPTIONS((SNIFFER(3100,TCPDUMP) NOREASSEMBLY) GMT

The formatted trace will now be in <some dsname>.

Do a binary transfer to your pc and you show be able to open the trace in Wireshark. File -> Open.

IPCS in Batch

I tend to use IPCS in batch (so I do not get the lock on the trace file, (grin)).

//IBMIPCS JOB 1,MSGCLASS=H 
//DEL  EXEC PGM=IEFBR14 
//SNIFFER  DD DISP=(MOD,DELETE),DSN=COLIN.SNIFFER, 
//     RECFM=VB,LRECL=8000,SPACE=(CYL,(10,10)),BLKSIZE=32000 
//ALLOC  EXEC PGM=IEFBR14 
//SNIFFER  DD DISP=(NEW,CATLG),DSN=COLIN.SNIFFER, 
//     RECFM=VB,LRECL=8000,SPACE=(CYL,(10,10)),BLKSIZE=32000 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//STEPLIB  DD  DISP=SHR,DSN=SYS1.MIGLIB 
//SNIFFER  DD DISP=SHR,DSN=COLIN.SNIFFER 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSPROC    DD  DISP=SHR,DSN=USER.Z24C.CLIST 
//           DD  DISP=SHR,DSN=ADCD.Z24C.CLIST 
//           DD  DISP=SHR,DSN=SYS1.SBLSCLI0 
//IPCSPARM   DD  DISP=SHR,DSN=SYS1.PARMLIB 
//           DD  DISP=SHR,DSN=ADCD.Z24C.PARMLIB 
//IPCSTOC    DD  SYSOUT=* 
//IPCSDDIR   DD  DISP=SHR,DSN=SYS1.DDIR 
//SYSTSIN DD * 
 IPCS NOPARM 
 SETDEF DA('IBMUSER.CTRACE1')                                         
 DROPD                                                                
CTRACE COMP(SYSTCPDA) OPTIONS((SNIFFER(8000,TCPDUMP) NOREASSEMBLY) GMT
/*                                                                    
 CTRACE COMP(SYSTCPDA) SUB((TCPIP)) LOCAL FULL SUB((TCIP)) GMT SHORT 
    

Note: The CTRACE command has to fit on one line.

Collecting a SYSTCIP trace

trace ct,wtrstart=CTWTR

TRACE CT,ON,COMP=SYSTCPIP,SUB=(tcpip),PARM=ctiezbcp

trace ct,wtrstop=CTWTR

 and use

CTRACE COMP(SYSTCPIP) SUB((TCPIP)) LOCAL FULL

to process it in IPCS.

Setting up IP V6 Linux to z/OS with ADCD

This post follows on from getting TCPIP to work with ADCD (zPDT and ZD&T) and allows me to FTP to z/OS from my Linux machine. There is a Q&A Has anyone configured z/OS running on ZD&T to support IPV6 protocol? but this was not enough information for me.

I’ve written about the IP V6 concepts and how they fit with z/OS.

You need to:

  1. Configure the Linux device map to add an OSA entry in the device map. You specify the path, and z/OS device addresses.
  2. Configure z/OS to support AF_NET6. You can make this change dynamically – or just re-ipl.
  3. Configure TCPIP to add an IFCONFIG6 entry. You need the z/OS AF_NET6 change before the IFCONFIG6 change is accepted. You need to restart TCPIP (or re-ipl).
  4. Configure a VTAM TRLE pointing to the devices defined in the devmap entry. This can be configured dynamically.
  5. Define a TCPIP interface, pointing to the PORTNAME of the TRLE definition. This defines a IP V6 address. The change can be configured dynamically.
  6. Test it!

Configuring the Linux Devmap

Create the Linux devmap entry

[manager]  # tap2 define network adapter (OSA) for communication with Linux
name awsosa 0019 --path=A2 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.25.1.6 
  # QDIO mode
device 408 osa osa 
device 409 osa osa 
device 40a osa osa 

This session has IP V4 address 172.25.1.6, and uses device addresses 408,409 and 40a. It uses path A2.

Restart zD&T to pick up the changes, and re-ipl z/OS.

Configuring z/OS

You have to configure both z/OS and TCPIP to enable TCPIP V6 support.

Update BPXPRM

Use D OMVS,S to show the BPXPRMxx members being used.

Update bpxprmxx with AF_INIT6, by adding the following into a BPXPRMxx member.

NETWORK DOMAINNAME(AF_INET6) 
        DOMAINNUMBER(19) 
        MAXSOCKETS(50000) 
        TYPE(INET) 

Check if your AF_INET is INET or CINET (Common INET is used when you have multiple TCPIP stacks), and specify the same value.

Re IPL.

If you mis configure it

DOMAINNUMBER value 19 is required ( see DOMAINNUMBER ) When I used a different value I got

BPXF202I DOMAIN AF_INET6 WAS NOT ACTIVATED FOR FILE SYSTEM
TYPE INET. RETURN CODE = 0000045A, REASON CODE = 743A0000

and, when TCPIP was started

EZZ0695I IPCONFIG6 STATEMENT ON LINE 1 NOT VALID – IPV6 SUPPORT IS NOT
ENABLED

Where 045A is EAFNOSUPPORT The address family is not supported.

Check AF_INET6 is configured

The command D OMVS,PFS gave me

PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
 INET      EZBPFINI   N/A       SOCKETS   A     2022/09/20 04.08.00     
 NFS       GFSCINIT   NFSC      REMOTE    A     2022/09/20 04.07.23     
 ZFS       IOEFSCM    N/A       LOCAL     A     2022/09/20 04.07.19     
 AUTOMNT   BPXTAMD    N/A       LOCAL     A     2022/09/20 04.07.19     
 UDS       BPXTUINT   N/A       SOCKETS   A     2022/09/20 04.07.19     
                                                                        
PFS TYPE  DOMAIN        MAXSOCK  OPNSOCK  HIGHUSED                      
 INET     AF_INET6       50000        5         5                      
          AF_INET         64000        8         8                      
 UDS      AF_UNIX         10000        2         2                      

Check AF_INET6 is in the list.

Configure TCPIP

I added “include user.Z25A.tcpparms(iconfig6)” into the TCPIP PROF.

This member had just

IPCONFIG6

Restart TCPIP.

The only change when TCPIP was restarted was the additional message

EZZ0300I OPENED INCLUDE FILE ‘USER.Z25A.TCPPARMS(ICONFIG6)’

Check the configuration

On Linux the find_io command gave

FIND_IO for "colinpaice@colinpaice" 
                                                                                                
         Interface Current     MAC     IPv4       IPv6           
 Path    Name      State       Address Address    Address        
------   --------- ----------- ------- ---------  ----------------  -------------- 
  A0     tap0      UP, RUNNING fa:...  10.1.1.1   fe80::f85c:c2ff:fe0a:1415%tap0  
  A1     tap1      UP, RUNNING 5e:...  172.26.1.6 fe80::5cda:64ff:feee:eeaa%tap1  
  A2     tap2      UP, RUNNING 4a:...  172.25.1.6 fe80::4850:5fff:fe5e:87c5%tap2 

Check the interface is UP, RUNNING

Define a VTAM TRLE

You need to create a VTAM TRLE resource. I invented PORTCP, and created member user.z25a.vtamlst(TRLE).

OSATRL3 VBUILD TYPE=TRL                                                 
OSATRL3E TRLE LNCTL=MPC,READ=(0408),WRITE=(0409),DATAPATH=(040A),      X
               PORTNAME=PORTCP,                                        X
               MPCLEVEL=QDIO                                            

This uses address 0408,0409, and 040a (matching the devmap entry above)

Use V net,act,id=trle to activate it.

Note: USER.Z25A.VTAMLST is in the DD concatenation for //VTAMLST.

Use D NET,TRL to display the defined TRLs. This showed

TRLE = OSATRL3E STATUS = NEVAC

Showing the TRLE above, and the status. It becomes ACTIVE when the TCPIP interface is activated.

Create the TCPIP interface definition

IP V6 uses an interface definition instead of a link and device.

  INTERFACE IFPORTCP6  DELETE 
  INTERFACE IFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTCP 
    INTFID 7:7:7:7  
    IPADDR FD00::67:1:1 

I activated these using

  • v tcpip,,stop,ifportcp6
  • v tcpip,,obey,USER.Z25A.TCPPARMS(IFACE6)
  • You might need v tcpip,,stop,ifportcp6

I found it better to stop the interface before updating it, as sometimes the updates were not all made.

Once these definitions were activated, TSO NETSTAT HOME gave

IntfName:   IFPORTCP6
  Address:  fd00::67:1:1
    Type:   Global
    Flags:
  Address:  fe80::7:7:7:7
    Type:   Link_Local
    Flags:  Autoconfigured

This shows an address fd00::67:1:1 and address fe80::7:7:7:7 based on the INTFID. If you do not specify an INTFID you get a name like fe80::a2:a201:a2:a2a2, based on the chpid (value a2). If the chpid was changed, you would get a different IP address. You can see the chpid from the Linux from_io command, or the z/OS d NET,ID=OSATRL3E,E command.

I could not get any IP address specified in the IPADDR parameter, to work. I could ping to it, but there were no responses.

The interface gets a MAC address based on the CHPID value – for example MACADDRESS: 02A2A2A2A2A2.

NETSTAT ROUTE gave

IPV4 DESTINATIONS                                                     
DESTINATION        GATEWAY         FLAGS    REFCNT     INTERFACE      
127.0.0.1/32       0.0.0.0         UH       0000000000 LOOPBACK       
172.26.1.2/32      0.0.0.0         UH       0000000000 ETH1           
172.26.1.20/32     0.0.0.0         H        0000000000 EZAZCX         
172.26.1.20/32     0.0.0.0         H        0000000000 EZASAMEMVS     
IPV6 DESTINATIONS                                                     
DESTIP:   ::1/128                                                     
  GW:     ::                                                          
  INTF:   LOOPBACK6         REFCNT:  0000000000                       
  FLGS:   UH                MTU:     65535                            
DESTIP:   FD00::67:1/128                                              
  GW:     ::                                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000                       
  FLGS:   UHS               MTU:     1492                             
DESTIP:   FD00::67:1:1/128                                            
  GW:     ::                                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000    
  FLGS:   UH                MTU:     9000             
DESTIP:   FE80::7:7:7:7/128                   
  GW:     ::                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000       
  FLGS:   UH                MTU:     9000                                

The Linux find_io command gave

                                                                                              
      Interface  Current    MAC       IPv4        IPv6           
 Path Name       State      Address   Address     Address        
----- --------- ----------- --------  ----------  -------------- 
...      
. 
  A0 tap0       UP, RUNNING da:...    10.1.1.1    fe80::...tap0  
  A1 tap1       UP, RUNNING 92:...    172.26.1.6  fe80::...%tap1  
  A2 tap2       UP, RUNNING 42:...    172.25.1.6  fe80::...%tap2  

Update the Linux route information

I did this to try to get the IPADDR to work. It did not work, and so is this is not needed.

sudo ip -6 route add fd00::6:1:1/128 dev tap2

Test it!

Use TSO NETSTAT HOME to find the IP V6 address. For example

Address: fe80::7:7:7:7, Type: Link_Local

On Linux use the find_io command to display information about the tunnels to z/OS. Find the tapn matching the chpid being used on z/OS.
Use the

ping fe80::7:7:7:7%tap2

command to send data to z/OS.

The response to the ping will be sent back down the connection the request arrived on.

You can use the tso netstat devlinks(intfname IFPORTCP6 command (where IFPORTCP6 is my interface) to display information about just the specified interface; for example Inbound packets, BytesIn, Outbound packets, BytesOut.

You can use

FTP fe80::7:7:7:7%tap2

then use

tso NETSTAT CONN
or
tso netstat conn (port 21

to see the connections.

You can use NETSTAT ND to display the neighbours. This gave me

Query Neighbor cache for fe80::6a:ffff:feaf:c0e4
IntfName: IFPORTCP6 IntfType: IPAQENET6
LinkLayerAddr: 026AFFAFC0E4 State: Reachable
Type: Host AdvDfltRtr: No

The value fe80::6a:ffff:feaf:c0e4 matches up with the value from find_io on Linux, and

02:6a:ff:af:c0:e4 matches up with the MAC address.

IP V6 concepts and using IP V6 with ADCD

This post follows on from getting TCPIP to work with ADCD (zPDT and ZD&T) and allow me to FTP to z/OS from my Linux machine. There is a Q&A Has anyone configured z/OS running on ZD&T to support IPV6 protocol? but this was not enough information for me.

Background

With IP V4 there is a limit of the number of IP addresses available. IP V6 has many addresses, and so this should not be a problem. There is no smooth migration from IP V4 to IPV6, it is more start with IP V4, run IP V4 and V6 at the same time, move stuff from IP V4 to IP V6, – and possibly (unlikely) run with just IP V6.

Wikipedia has many good articles

  • IP V6 in general
  • IP V4 uses addresses like 192.6.7.1. IP V6 uses addresses like 2001:0db8:0000:0000:0000:ff00:0042:8329 (=2001:db8::ff00:42:8329). See here.
  • Each IP V6 has a local address (link-local) fe80::….
  • An IP V6 address can have :: to mean replace with as many zeros as needed to make this a valid IP V6 address. So 2001:0db8:0000:0000:0000:ff00:0042:8329 can be written 2001:db8::ff00:42:8329. You can only have one :: in a value.
  • TCPIP can support IPV4 and IP V6 at the same time (dual stack)
  • You can wrap an IP V4 address into an IP V6. For example ::ffff:192.0.2.128
  • For security clients often get a “temporary” (or randomised) IP address instead of a hard coded address. This uses a randomiser function with a secret key. This IP address can expire, and a new(different) IP address obtained. This can make it hard(impossible) for a server to do a reverse DNS lookup. This temporary IP address is useful, as it means you cannot be tracked by your IP address.

Other information

  • Each IP node has a IP V4 address and an IP V6 address.
  • When defining connections between systems, it looks like you need at least one IP V4 route, and at least one IP V6 route – I could be wrong.
  • An IPV6 host usually has more than one IP address.

IPV6 has reserved IP ranges

  • 2001:db8::/32 Addresses used in documentation and example source code.
  • fe80::/10 are the link-local unicast [RFC4291] addresses. Addresses within this block should not appear on the public Internet. Your router should not externalise this.
  • fd00::/7 for private internets. These are the unique-local addresses [RFC4193]. Addresses within this block should not appear by default on the public Internet. This means you can use them within your organisation.
  • ffxx is for multicast to all links matching the address. For example ff02::5 is used by the dynamic routing protocol OSPF to say to all routers in the (local) network “hello – anyone there”.

Getting started

The end of each connection needs at least one IP address. If you have 5 connections, you will have at least 5 IP addresses

What is my IP address on Linux ?

You can use hostname -I

which gave me

192.168.1.223
10.1.1.1
172.24.1.6
172.22.1.6
2a00:23c5:8888:9999:0000:1111:2222:3333
2a00:23c5:9999:0000:1111:2222:3333:4444

You can also use ifconfig or ip addr show.

You can also use the z109x find_io command.

What is my IP address on z/OS?

You can use TSO NETSTAT HOME, or the operator command V TCPIP,,NETSTAT,HOME .

This gives information like

LinkName:   ETH1
  Address:  172.26.1.2
    Flags:  Primary
IntfName:   IFPORTCP6
  Address:  fc00::67:1:1
    Type:   Global
    Flags:
  Address:  fe80::a1:a101:a1:a1a1
    Type:   Link_Local
    Flags:  Autoconfigured

This shows

  • 172.26.1.2 an IP V4 address for use within a private network, for connection ETH1.
  • fc00::67:1:1 an IP V6, unique local address, for interface IFPORTCP6.
  • fe80::a1:a101:a1:a1a1a link-local address, for interface IFPORTCP6. The a1:a1… is based on the MAC address of the device. You can override this on z/OS by specifying the INTFID.
  • Note: If you do not specify the INTFID, it will default to the MAC address. If you reconfigure the system, you may get different MAC address, and so the IP address via this interface will change. By specifying the INTFID you can specify what the IP address for this interface, which will not change if the system is reconfigured.

What are the routes on my machine?

IP V4 On Linux ip route or ip -4 route

gives

default via 192.168.1.254 dev wlp4s0 proto dhcp metric 600 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1 
169.254.0.0/16 dev wlp4s0 scope link metric 1000 
172.25.1.0/24 dev tap2 proto kernel scope link src 172.25.1.6 
172.26.1.0/24 dev tap1 proto kernel scope link src 172.26.1.6 
172.27.1.0/24 via 172.25.1.6 dev tap2 
192.168.1.0/24 dev wlp4s0 proto kernel scope link src 192.168.1.223 metric 600 

IP V6 on Linux ip -6 route gives

:1 dev lo proto kernel metric 256 pref medium
2a00:xxx:xxxx:xxxx::/64 dev wlp4s0 proto ra metric 600 pref medium
fe80::/64 dev tap0 proto kernel metric 256 pref medium
fe80::/64 dev tap1 proto kernel metric 256 pref medium
fe80::/64 dev tap2 proto kernel metric 256 pref medium
fe80::/64 dev wlp4s0 proto kernel metric 600 pref medium
default via fe80::966a:b0ff:fe85:54a7 dev wlp4s0 proto ra metric 20600 pref medium

On z/OS, TSO NETSTAT route or V tcpip,,netstat,route gives

IPv4 Destinations
Destination        Gateway         Flags    Refcnt     Interface
-----------        -------         -----    ------     ---------
Default            172.25.1.6      GS       0000000000 IFPORTCP
127.0.0.1/32       0.0.0.0         UH       0000000000 LOOPBACK
172.26.1.2/32      0.0.0.0         UH       0000000000 ETH1
IPv6 Destinations
DestIP:   ::1/128
  Gw:     ::
  Intf:   LOOPBACK6         Refcnt:  0000000000
  Flgs:   UH                MTU:     65535
DestIP:   fc00::67:1:1/128
  Gw:     ::
  Intf:   IFPORTCP6         Refcnt:  0000000000
  Flgs:   UH                MTU:     9000
DestIP:   fe80::a1:a101:a1:a1a1/128
  Gw:     ::
  Intf:   IFPORTCP6         Refcnt:  0000000000
  Flgs:   UH                MTU:     9000

find_io

FIND_IO for "colinpaice@colinpaice" 
                                                                                                
      Interface  Current MAC      IPv4      IPv6           
 Path Name       State   Address  Address   Address        
----- ---------- ------- ------- ---------- -------------- 
  A0 tap0 UP, RUNNING    ea:...  10.1.1.1   fe80::e8e8:69ff:fe20:435b%tap0  
  A1 tap1 UP, RUNNING    22:...  172.26.1.6 fe80::2090:14ff:fee0:5f20%tap1  
  A2 tap2 UP, RUNNING    22:...  172.25.1.6 fe80::2047:afff:fef7:1caf%tap2  

Joining it all up

To FTP from Linux to z/OS, I use

ftp fe80::a1:a101:a1:a1a1%tap1

The fe80::a1:a101:a1:a1a1 came from the z/OS NETSTAT HOME, and is the z/OS end of the connection.

Using another interface (defined with the INTFID 7:7:7:7) I could use

ftp fe80::7:7:7:7%tap2

The ip -6 route command gave me

fe80::/64 dev tap0 proto kernel metric 256 pref medium
fe80::/64 dev tap1 proto kernel metric 256 pref medium
fe80::/64 dev tap2 proto kernel metric 256 pref medium
fe80::/64 dev wlp4s0 proto kernel metric 600 pref medium

So the request for FE80…. can be routed to any of these. I know that it was configured using tunnel interface tap1, so the address to use is fe80::a1:a101:a1:a1a1%tap1.

Once the connection to FTP was established, z/OS TSO NETSTAT CONN gave me

FTPD1    0000003D ESTBLSH                             
  LOCAL SOCKET:   FE80::A1:A101:A1:A1A1..21           
  FOREIGN SOCKET: FE80::2090:14FF:FEE0:5F20..42572    

The request is processed by z/OS address FE80::A1:A101:A1:A1A1 (port 21)

This ties up with DestIP: fe80::a1:a101:a1:a1a1/128 … Intf: IFPORTCP6 which shows the request came in on interface IFPORTCP6

The request came from FE80::2090:14FF:FEE0:5F20, which ties up with fe80::2090:14ff:fee0:5f20%tap1. The request came in over the tap1 interface.

Understanding the Linux, VTAM and TCPIP linkage with ADCD and z/OS

This follows on from Getting TCPIP on my Linux machine to talk to z/OS ADCD running on ZPDT on my Linux machine. It shows the moving parts.

There are lot of bits of configuration which have to be mutually consistent, and some bits do not work as I expected.

There are several sections in this blog post

The different parts

The Linux devmap

In Linux the devmap has

[manager]  # tap0 define network adapter (OSA) for communication with Linux
name awsosa 0022 --path=A0 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.26.1.1 
device 400 osa osa 
device 401 osa osa 
device 402 osa osa 

The key information is

  • It uses z/OS path A0.
  • It is an OSA with QDIO (OSD).
  • It uses the tunnelling interface.
  • The IP address is 172.26.1.1. This is the IP address of the Linux end of the tunnel.
  • There are 3 z/OS devices with address 0400, 0401, 0402.
  • The devices are OSAs.
  • The control unit they are connected to is an OSA. ( A 3279 colour display would have device 3279, control unit 3274).

In ADCD.Z25A.VTAMLST(OSATRL2) is

OSATRL1 VBUILD TYPE=TRL                                                 
OSATRL1E TRLE LNCTL=MPC,READ=(0400),WRITE=(0401),DATAPATH=(0402),      X
               PORTNAME=PORTA,                                         X
               MPCLEVEL=QDIO                                            
OSATRL2E TRLE LNCTL=MPC,READ=(0404),WRITE=(0405),DATAPATH=(0406),      X
               PORTNAME=PORTB,                                         X
               MPCLEVEL=QDIO 

This defines a Transport Resource List(TRL)

  • OSATRL1E is a name which can be used in commands.
  • TRLE is a Transport Resource List Entry
  • It uses MultiPathChannel links(MPC) (more than one device)
  • It uses multiple device addresses 0400,0401,0402
  • The Open Systems Adapter(OSA) port name is PORTA
  • MPCLEVEL=QDIO Indicates that the Queued Direct I/O interface is used for an OSA-Express feature.

You can use commands like

  • D NET,TRL
  • D NET,ID=OSATRL1
  • D NET,ID=OSATRL2E,E

Example output from D NET,IDA=OSATRL1E

DISPLAY ACCEPTED                                                   
NAME = OSATRL1E, TYPE = TRLE 558                                   
STATUS= ACTIV, DESIRED STATE= ACTIV                                
TYPE = LEASED             , CONTROL = MPC , HPDT = YES             
TRL MAJOR NODE = OSATRL2                                          
MPCLEVEL = QDIO       MPCUSAGE = SHARE                            
PORTNAME = PORTA      LINKNUM =   0   OSA CODE LEVEL = 7617       
CHPID TYPE = OSD      CHPID = A0  PNETID = **NA** 
WRITE DEV = 0401 STATUS = ACTIVE     STATE = ONLINE      
READ  DEV = 0400 STATUS = ACTIVE     STATE = ONLINE 
DATA  DEV = 0402 STATUS = ACTIVE     STATE = N/A         

Key information is

  • TYPE = TRLE
  • STATUS= ACTIV
  • PORTNAME = PORTA
  • CHPID = A0
  • There are three devices 0401, 0400, 0402

As this matches the Linux configuration, it has all connected up and become active. It is known to VTAM and TCPIP as PORTA.

Within TCPIP there is a member ADCD.Z25A.TCPPARMS(ZPDTDEV1) with

DEVICE PORTA MPCIPA 
 LINK ETH1 IPAQENET PORTA 
HOME &HOMEIPADDRESS1 ETH1 
BEGINRoutes 
; Destination        SubnetMask    FirstHop       LinkName    Size 
ROUTE 9.114.209.0    255.255.255.0    =            ETH1   MTU 1492 
; Destination                      First Hop      LinkName    Size 
ROUTE DEFAULT             &DEFAULTROUTEADDR        ETH1   MTU 1492 
ENDRoutes 
START PORTA 

The key information is

  • Device PORTA – matches the VTAM TLRE definition.
  • There is a link ETH1 associated with PORTA.
  • When anyone uses this device-> link, use the Home address &HOMEIPADDRESS1. The z/OS symbol &HOMEIPADDRESS1 is “172.26.1.2”, so this value is used.
  • If TCPIP does not know where to send a packet, it sends it to &DEFAULTROUTEADDR down ETH1 link. (&DEFAULTROUTEADDR. = “172.26.1.1”)
  • The “172.26.1.1” matches the –tunnel_ip=172.26.1.1 in the Linux definition.

Background information

A link has two ends, each end defines its IP address. For example

  • for the Linux for the DEVMAP file –tunnel_ip=172.26.1.6 defines the Linux as having IP address 172.26.1.6.
  • for z/OS the HOME 172.26.1.9 ETH2 defines the z/OS as being 172.26.1.9

If you “FTP 172.26.1.9”, and use NETSTAT CONN, it gives a connection 172.26.1.9..21 and 172.26.1.6..53526 , so you can see the IP addresses (and ports) of the two ends of the connection. Note: Port 21 is the well known port for FTP.

The definitions

Devmap

The devmap definitions below, create a two connections between Linux and z/OS.

[manager]  # tap0 define network adapter (OSA) for communication with Linux QDIO
#defaults to  10.1.1.1
name awsosa 0022 --path=A0 --pathtype=OSD --tunnel_intf=y mode
device 400 osa osa 
device 401 osa osa 
device 402 osa osa 

[manager]  # tap1 define network adapter (OSA) for communication with Linux QDIO
name awsosa 0009 --path=A1 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.26.1.6 
device 404 osa osa 
device 405 osa osa 
device 406 osa osa 

The VTAM TRL definition

OSATRL1 VBUILD TYPE=TRL                                                 
OSATRL1E TRLE LNCTL=MPC,READ=(0400),WRITE=(0401),DATAPATH=(0402),      X
               PORTNAME=PORTA,                                         X
               MPCLEVEL=QDIO                                            
OSATRL2E TRLE LNCTL=MPC,READ=(0404),WRITE=(0405),DATAPATH=(0406),      X
               PORTNAME=PORTB,                                         X
               MPCLEVEL=QDIO                                            

The TCPIP definition in USER.Z25A.TCPPARMS(ZPDTDEV1)

DEVICE PORTB MPCIPA 
 LINK ETH2 IPAQENET PORTB 
HOME 172.26.1.9      ETH2 
                                                                        
DEVICE PORTA MPCIPA 
 LINK ETH1 IPAQENET PORTA 
HOME 10.1.1.2        ETH1 
                                                                        
BEGINRoutes 
; Destination        SubnetMask    FirstHop   LinkName    Size 
ROUTE 10.1.1.0       255.255.255.0    =            ETH1   MTU 1500 
ROUTE 172.26.1.0     255.255.255.0    =            ETH2   MTU 1492 
ROUTE DEFAULT                      10.1.1.1        ETH1   MTU 1492 
ENDRoutes 


START PORTA 
START PORTB 

I found that sometimes TCPIP does not 100% accept an update, and the device continues to point to old definitions.

I used the operator commands

v tcpip,,STOP,PORTA
v tcpip,,STOP,PORTB
v tcpip,,obey,USER.Z25A.TCPPARMS(ZPDTDEV1)

to activate the change.

With this I could ping and FTP to 10.1.1.2 and 127.26.1.9 (the values in the HOME)

TSO netstat home gave

MVS TCP/IP NETSTAT CS V2R5       TCPIP Name: TCPIP 
Home address list:
Address          Link             Flg
-------          ----             ---
172.26.1.9       ETH2             P
10.1.1.2         ETH1
172.26.1.20      EZASAMEMVS
127.0.0.1        LOOPBACK

Address          Interface        Flg
-------          ---------        ---
172.26.1.20      EZAZCX

and TSO NETSTAT ROUTE gave

Destination    Gateway  Flags Refcnt     Interface
-----------    -------  ----- ------     ---------
Default        10.1.1.1 UGS   0000000000 ETH1
10.1.1.0/24    0.0.0.0  US    0000000000 ETH1
10.1.1.2/32    0.0.0.0  UH    0000000000 ETH1
127.0.0.1/32   0.0.0.0  UH    0000000000 LOOPBACK
172.26.1.0/24  0.0.0.0  US    0000000000 ETH2
172.26.1.9/32  0.0.0.0  UH    0000000000 ETH2
172.26.1.20/32 0.0.0.0  H     0000000000 EZAZCX
172.26.1.20/32 0.0.0.0  H     0000000000 EZASAMEMVS

Where the flags are

  • UGS – The route is Up, uses a Gateway, is a Static route
  • US -The route is Up, is a Static route
  • US
  • UH – The route is Up, is a route to a Host (rather than a gateway)
  • H – is a route to a Host (rather than a gateway) It does not have “U” so is not up.

More complex example – Using an Interface statement

The definitions above used

DEVICE PORTB MPCIPA 
 LINK ETH2 IPAQENET PORTB 
HOME 172.26.1.9      ETH2 

You can also use the newer interface definitions which combine these

INTERFACE IFPORTCP  DELETE 
INTERFACE IFPORTCP 
  DEFINE IPAQENET 
  CHPIDTYPE OSD 
  PORTNAME PORTCP 
  IPADDR 172.25.1.1 

START IFPORTCP                                      

I activated this by V TCPIP,,OBEYFILE,USER.Z25A.TCPPARMS(IFACE)

This still needs a TRLE, such as

OSATRL3 VBUILD TYPE=TRL                                                 
OSATRL3E TRLE LNCTL=MPC,READ=(0408),WRITE=(0409),DATAPATH=(040A),      X
               PORTNAME=PORTCP,                                        X
               MPCLEVEL=QDIO                                            

Connected by the PORTNAME=PORTCP

More complex example – Using an Interface statement and different IP address ranges

I change the address of the interface

INTERFACE IFPORTCP  DELETE 
INTERFACE IFPORTCP 
  DEFINE IPAQENET 
  CHPIDTYPE OSD 
  PORTNAME PORTCP 
  IPADDR 172.27.1.1 

START IFPORTCP                                      

I activated this by

  • v tcpip,,stop,IFPORTCP
  • V TCPIP,,OBEYFILE,USER.Z25A.TCPPARMS(IFACE)
  • v tcpip,,start,IFPORTCP

TSO NETSTAT HOME now says

IntfName:   IFPORTCP
  Address:  172.27.1.1
    Flags:

Tell Linux of the new address

To use the new IP address, you need to tell Linux where to send the requests to.

sudo ip route add 172.27.1.0/24 via 172.25.1.6

This says that to get to the 172.27… you need to go via 172.25.1.6, which has been defined as

[manager]  # tap2 define network adapter (OSA) for communication with Linux
name awsosa 0019 --path=A2 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.25.1.6 
  # QDIO mode
device 408 osa osa 
device 409 osa osa 
device 40a osa osa 

z/OS needs to know how to get back from z/OS. In the routes you need

BEGINRoutes 
; Destination        SubnetMask    FirstHop   LinkName    Size 
ROUTE 172.25.1.0     255.255.255.0    =       IFPORTCP    MTU 1492 
ROUTE DEFAULT        172.25.1.6               IFPORTCP    MTU 1492 
ENDRoutesd

Which says anything for 172.25.1.* send down link/interface IFPORTCP.

If no routes match – send it to 172.25.1.6.

Lessons learned

Below are some of the lessons I learned in writing this blog post. I hope they will be useful to other people. They will be useful to me when I come to use this topic and find it does not work!

Defining Linux tunnels

On Linux the tunnels are defined by the zPDT code, you do not have to define them. If you have defined a tunnel, the zPDT will override the definitions.

Activating resources on z/OS

I created my z/OS TCP, and activated them, but they did not work.

If you define a resource which already exists, and is active, parts of the old definition is still used. I had to stop the PORT using V TCPIP,,STOP,PORTA before I could successfully activate the resource.

When I did not stop the PORT I got

PROCESSING COMMAND: VARY TCPIP,,OBEY,USER.Z25A.TCPPARMS(ZPDTDEV1)          
OPENED OBEYFILE FILE 'USER.Z25A.TCPPARMS(ZPDTDEV1)'                        
PROFILE PROCESSING BEGINNING FOR 'USER.Z25A.TCPPARMS(ZPDTDEV1)'            
DEVICE NAME PORTA ON LINE 5 IS ALREADY DEFINED                             
LINK NAME ETH1 ON LINE 6 IS ALREADY DEFINED                                
DEVICE PORTA ON LINE 15 IS ALREADY STARTED                                 
PROFILE PROCESSING COMPLETE FOR FILE 'USER.Z25A.TCPPARMS(ZPDTDEV1)'        
COMMAND VARY OBEY COMPLETED SUCCESSFULLY                                   
INITIALIZATION COMPLETE FOR DEVICE PORTB                                   

When I put a STOP PORTA in the OBEY file, I got

EZZ4308I ERROR: CODE=8010302D DURING ACTIVATION OF DEVICE PORTB. DIAGNOSTIC CODE: 02    
EZD2028I DEVICE PORTB ACTIVATION FAILED - NO DATAPATH DEVICE ADDRESSES ARE AVAILABLE    

When I repeated it – it worked. It looks like a timing issue. So you have to do it in two steps. The meaning of 302D is A ULP cannot use a QDIO device because there are no datapath channel addresses available.

VTAMTRL

When the system was IPLED the VTAM command D NET,TRL gave

DISPLAY TYPE = TRL             
---------------------------------------------------------
 TRL MAJOR NODE = OSATRL2                                
 TRLE = OSATRL1E  STATUS = ACTIV       CONTROL = MPC     
 TRLE = OSATRL2E  STATUS = NEVAC       CONTROL = MPC     
 2 TRLE(S) DISPLAYED                                     

The TRLE entry will only become active when the port is active. Defining it and starting it gave me TRLE = OSATRL2E STATUS = ACTIV CONTROL = MPC

Getting TCPIP to talk to an ADCD system.

I had a new (z/OS 2.5) ADCD system, and wanted to get FTP working. Once I had unzipped the files, and started z/OS, I could not ping or connect to the z/OS via TCPIP.

When I wanted to use z/OS on a different machine I needed to configure my laptop to give the route to the machine using and Ethernet connection

sudo ip route add 10.1.1.0/24 via 10.1.0.3 
# open a session to it
ssh -X colin@10.1.0.3 

Using my home wireless connection I could use instead

sudo ip route add 10.1.1.0/24 dev wlp4s0
# open a session to it
ssh -X colin@10.1.0.3 

The problem

I had ZD&T running on my laptop; My definitions on Linux used an IP address of 10.1.1.2, but the the shipped ADCD system has been configured for a HOME entry of 172.26.1.2.

You can either change z/OS, or you can change your Linux setup.

Changing z/OS

I wanted to get the address 10.1.1.2 on my Linux to talk to z/OS, but z/OS was not configured for this address. Using the TCPIP command NETSTAT HOME showed it was configured for a 172.26.1.2 .

In ADCD.Z25A.TCPPARMS(ZPDTDEV1) is

DEVICE PORTA MPCIPA 
 LINK ETH1 IPAQENET PORTA 
HOME &HOMEIPADDRESS1 ETH1 
BEGINRoutes 
; Destination        SubnetMask    FirstHop       LinkName    Size 
ROUTE 9.114.209.0    255.255.255.0    =            ETH1   MTU 1492 
; Destination                      First Hop      LinkName    Size 
ROUTE DEFAULT             &DEFAULTROUTEADDR        ETH1   MTU 1492 
ENDRoutes 
START PORTA 
                                                                       

The z/OS command D SYMBOLS gave

&DEFAULTROUTEADDR. = "172.26.1.1"         
&DYNXCFIPADDRESS.  = "172.26.1.20"        
&EPHEMERALPORTS.   = "10000 65534"        
&GBLRESL.          = "GBLRESOL"           
&HOMEIPADDRESS1.   = "172.26.1.2" 
&TCPPROF.          = "PROF2"                            f

Updating the address

You have several approaches.

  • Update the system symbols and re-ipl or restart TCPIP
  • Replace the symbols with the hard coded definitions and re-ipl or restart TCPIP
  • Use system symbols and restart just the relevant bits of TCPIP
  • Update the TCPIP definitions and restart just the relevant bits of TCPIP

Update the system symbols and re-ipl

See Changing z/OS system symbols is – easy – ish.

My IEASYM definition included member IEASYMAU.

I created member USER.Z25A.PARMLIB(IEASYMAU), copied member FEU.Z25A.PARMLIB(IEASYMAU) into it, and changed the definitions.

SYSDEF  SYSNAME(S0W1) 
        SYSCLONE(1A) 
        SYMDEF(&HOMEIPADDRESS1.='10.1.1.2') 
        SYMDEF(&DEFAULTROUTEADDR.='10.1.1.1') 
        SYMDEF(&DYNXCFIPADDRESS.='10.1.1.31') 
        SYMDEF(&ZCXDVIPAADDRESS.='10.1.1.32') 
        SYMDEF(&EPHEMERALPORTS.='10000 65534') 
        SYMDEF(&GBLRESL.='GBLRESAU') 
        SYMDEF(&TCPPROF='PROF2') 

I shutdown z/OS and restarted it – and then I could use PING and, once I had started FTPD, I could FTP to 10.1.1.2. If you do not want to restart z/OS see below for the long story. in Changing z/OS system symbols is – easy – ish, or “Refreshing TCP/IP definitions“, below.

Update the TCPIP definitions and restart TCPIP or re-ipl

I backed up ADCD.Z25A.TCPPARMS(ZPDTDEV1), then changed ADCD.Z25A.TCPPARMS(ZPDTDEV1) to use the hardcoded address and default router address.

Restarting TCPIP or a REIPL activated these changes – but doing this is disruptive to users.

Refreshing TCP/IP definitions.

It is not easy to refresh the TCPIP definitions. See here.

Once I had made the changes, and activated the updated definitions, I could ping to z/OS and use FTPD.

Changing Linux

I changed my devmap to have

[manager] # tap0 define network adapter (OSA) for communication with Linux
name awsosa 0022 –path=A0 –pathtype=OSD –tunnel_intf=y –tunnel_ip=172.26.1.1
device 400 osa osa –unitadd=0
device 401 osa osa –unitadd=1
device 402 osa osa –unitadd=2

The “ip route” command gave

default via 192.168.1.254 dev wlp4s0 proto dhcp metric 600
169.254.0.0/16 dev wlp4s0 scope link metric 1000
172.26.1.0/24 dev tap0 proto kernel scope link src 172.26.1.1
192.168.1.0/24 dev wlp4s0 proto kernel scope link src 192.168.1.223 metric 600

The find_io command gave

         Interface         Current          MAC     IPv4       IPv6           
 Path    Name              State            Address Address    Address        
------   ----------------  ---------------- ------- -------    --------- 
  F0     enp0s31f6         UP, NOT-RUNNING  8c:...  *          *               
  F1     wlp4s0            UP, RUNNING      d4....  192....    2a00:...  
  F2     wwan0             DOWN             a2....  *          *               
. 
  A0     tap0              UP, RUNNING      22....  172.26.1.1 fe80...
  A1     tap1              DOWN             02....  *         *               
  ...

I could ping and FTP to 172.26.1.2.

Changing TCPIP configuration on z/OS is not easy.

I wanted to change the TCPIP definitions, so I could FTP from my laptop to the zD&T system running the ADCD z/OS images.

On my one-user, self contained system, it was easiest to make the changes to TCPIP, then stop and restart TCPIP.

If course with multiple users this would be disruptive and may not be possible. You can activate changes to your configuration – but it is not trivial.

Where are the definitions?

In the TCPIP job there is a //PROFILE… statement. This was ADCD.Z25A.TCPPARMS(PROF2). In this member is the TCPIP configuration including several “include adcd.Z25A.tcpparms(…)” statements including “include adcd.Z25A.tcpparms(zpdtdev1)” which had the definitions for the connection I needed to change.

Note: TCPIP explicity includes dataset(member) name. If you copy a member from ADCD.Z25A.TCPPARMS to USER.Z25A.TCPPARMS, it will not be used unless you change the configuration to use the fully qualified name.
You may want to copy a member from ADCD.Z25A.TCPPARMS, to keep a copy of the original, then edit the ADCD.Z25A.TCPPARMS to make your changes.

You can use the OBEYFILE command to make configuration changes once TCPIP has started, this can use any dataset. You can put your new definitions in TEST.TCPPARMS, and use OBEYFILE to activate them. Once they are working as expected, copy them to the ADCD.Z25A.TCPPARMS and have them activated when TCPIP starts.

A typical OBEYFILE command is

v tcpip,tcpip,obeyfile,COLIN.TCPPARMS(DELHOME)

Refreshing TCP/IP definitions.

It is not easy to refresh the TCP/IP definitions. Restarting TCP may be an easier solution.

The definitions I wanted to change were in member zpdtdev1

DEVICE PORTA MPCIPA 
 LINK ETH1 IPAQENET PORTA 
HOME &HOMEIPADDRESS1 ETH1 
BEGINRoutes 
; Destination        SubnetMask    FirstHop       LinkName    Size 
ROUTE 9.114.209.0    255.255.255.0    =            ETH1   MTU 1492 
; Destination                      First Hop      LinkName    Size 
ROUTE DEFAULT             &DEFAULTROUTEADDR        ETH1   MTU 1492 
ENDRoutes 
START PORTA 

I needed to change the system symbols &HOMEIPADDRESS1 and &DEFAULTROUTEADDR to be hard coded values.

There is no “replace” command; you have to delete the definitions and re-add them.

From the above configuration file, the obvious statements are

stop PORTA
delete LINK ETH1
delete device PORTA

but this fails with

EZZ0395I DELETE LINK ETH1 ON LINE 2 FAILED BECAUSE LINK STATE NOT VALID
EZZ0395I DELETE DEVICE PORTA ON LINE 3 FAILED BECAUSE DEVICE HAS A LINK DEFINED

The command TSO NETSTAT DEVLINKS gives the status. This gave me

EZZ2760I DevName: PORTA DevType: MPCIPA
EZZ2766I DevStatus: Not Active CfgRouter: Non ActRouter: Unknown
EZZ2761I LnkName: ETH1 LnkType: IPAQENET LnkStatus: Not Active

The EZZ0395I message said

The link is in use. If this message was issued in response to an attempt to delete a link, the link IP address might still be defined. You must delete the link IP address from the HOME list before the link can be deleted. To remove the link IP address from the HOME list, use the VARY TCPIP,,OBEYFILE command with a profile that contains a HOME statement that does not include the home IP address that is associated with the link that you want to delete. If you specify the updated HOME statement and the DELETE LINK statement in the same VARY TCPIP,,OBEYFILE data set, the HOME statement must precede the DELETE LINK statement.

A replace option would seem a better design than the above.

TSO NETSTAT HOME gave me

EZZ2350I MVS TCP/IP NETSTAT CS V2R5       TCPIP Name: TCPIP           16:28:46
EZZ2700I Home address list:
EZZ2701I Address          Link             Flg
EZZ2702I -------          ----             ---
EZZ2703I 172.26.1.2       ETH1             P
EZZ2703I 10.1.1.31        EZASAMEMVS
EZZ2703I 127.0.0.1        LOOPBACK
 
EZZ2704I Address          Interface        Flg
EZZ2704I -------          ---------        ---
EZZ2703I 10.1.1.31        EZAZCX

I copied the key information into a file – excluding the 172.** stuff

HOME  10.1.1.31        EZASAMEMVS 
      127.0.0.1        LOOPBACK 

and used

v tcpip,tcpip,obeyfile,COLIN.TCPPARMS(DELHOME)

this gave messages

EZZ0344I PERMANENT LOOPBACK ADDRESS 127.0.0.1 SPECIFIED ON LINE 2 CANNOT BE ADDED TO THE HOME LIST     
EZZ0612I HOME ADDRESS 10.1.1.31 FOR LINK EZASAMEMVS ON LINE 1 REPLACES THE PREVIOUS ADDRESS            
EZZ0316I PROFILE PROCESSING COMPLETE FOR FILE 'USER.Z25A.TCPPARMS(DELHOME)'                            
EZZ0303I OBEYFILE FILE CONTAINS ERRORS                                                                 
EZZ0331I NO HOME ADDRESS ASSIGNED TO LINK ETH1                                                         
EZZ0619I LINK EZASAMEMVS USES DUPLICATE HOME ADDRESS 10.1.1.31                                         
EZZ0619I LINK IQDIOLNK0A01011F USES DUPLICATE HOME ADDRESS 10.1.1.31                                   
EZZ0059I VARY OBEY COMMAND FAILED: SEE PREVIOUS MESSAGES                                               

which despite the error messages, it seems to have worked as TSO NETSTAT HOME did not show ETH1.

The output from TSO NETSTAT DEVLINKS showed the deletes had worked, and the device PORTA and link ETH1 where no longer present.

I changed the TCPIP definitions and used

V tcpip,tcpip,obeyfile,ADCD.Z25A.TCPPARMS(ZPDTDEV1)

this worked, and TSO NETSTAT HOME gave

MVS TCP/IP NETSTAT CS V2R5       TCPIP Name: TCPIP           16:44:09
Home address list:
Address          Link             Flg
-------          ----             ---
10.1.1.2         ETH1             P
127.0.0.1        LOOPBACK

Address          Interface        Flg
-------          ---------        ---
10.1.1.31        EZAZCX

and ping 10.1.1.2 worked.

As I said at the top – it was was quicker to restart TCPIP.

Changing z/OS system symbols is – easy – ish.

Z/OS provides system wide symbols. This is very useful because you can have configuration with &SYSNAME. within it – and so you can have one definition – and the value depends which system you are on.

The process of changing these symbols without an IPL may be trivial – or not.

What is my system currently using?

You can use the operator command D SYMBOLS . This gives output like

IEA007I STATIC SYSTEM SYMBOL VALUES 607          
 &SYSALVL.          = "2"                        
 &SYSCLONE.         = "1A"                       
 &SYSNAME.          = "S0W1"                     
 &SYSOSLVL.         = "Z1020500"                 

Where do these come from?

The system was IPLed with parm A80GK, this points to a member LOADGK in SYS1.IPLPARM, or the PARMLIB concatenation. This member had

IODF     99 SYS1                                                   
INITSQA  0000M 0008M                                               
SYSCAT   A5SYS1113CCATALOG.Z25A.MASTER                             
SYSPARM  NZ                                                        
IEASYM   (AU,GK)                                                   
NUCLST   00                                                        
PARMLIB  USER.Z25A.PARMLIB                            A5CFG1       
PARMLIB  FEU.Z25A.PARMLIB                             A5CFG1       
PARMLIB  ADCD.Z25A.PARMLIB                            A5SYS1       
PARMLIB  SYS1.PARMLIB                                 A5RES1       
NUCLEUS  1                                                         
SYSPLEX  ADCDPL                                                    

At the start of the IPL it displays

SYS1.IPLPARM ON DEVICE 0A82 SELECTED FOR IPL PARAMETERS    
LOAD   ID GK SELECTED                                      
NUCLST ID 00 SELECTED                                      
IODF DSN = SYS1.IODF99                                     
CONFIGURATION ID = OS390   . IODF DEVICE NUMBER = 0A82     
NUCLEUS 1 SELECTED                                         
IPL DEVICE: 00A80  VOLUME: A5RES1                          
MASTER CATALOG SELECTED IS CATALOG.Z25A.MASTER             
MEMBER IEASYMAU FOUND IN FEU.Z25A.PARMLIB                  
MEMBER IEASYMGK FOUND IN FEU.Z25A.PARMLIB                  

Which matches the LOADGK member.

This shows the symbols came from definitions IEASYMAU and IEASYMGK.

You can use DISPLAY IPLINFO to get

IEE254I  13.34.36 IPLINFO DISPLAY 594                      
 SYSTEM IPLED AT 13.10.56 ON 09/12/2022                    
 RELEASE z/OS 02.05.00    LICENSE = z/OS                   
 USED LOADWS IN SYS1.IPLPARM ON 00A82                      
 ARCHLVL = 2   MTLSHARE = N                                
 IEASYM LIST = 00                                          
 IEASYS LIST = WS (OP)                                     
 IODF DEVICE: ORIGINAL(00A82) CURRENT(00A82)               
 IPL DEVICE: ORIGINAL(00A80) CURRENT(00A80) VOLUME(A5RES1) 

So you can see for this IPL, LOADWS and member IEASYM00 was used.

Changing symbols dynamically.

You can update members in the PARMLIB concatenation, but activating them gets a bit harder.

The DISPLAY PARMLIB command displays the PARMLIB concatenation for example

PARMLIB DATA SETS SPECIFIED                                             
AT IPL                                                                  
ENTRY  FLAGS  VOLUME  DATA SET                                          
  1      S    A5CFG1  USER.Z25A.PARMLIB                                 
  2      S    A5CFG1  FEU.Z25A.PARMLIB                                  
  3      S    A5SYS1  ADCD.Z25A.PARMLIB                                 
  4      S    A5RES1  SYS1.PARMLIB                                      

I copied a member from ADCD.Z25A.PARMLIB, to USER.Z25A.PARMLIB with the same name, and edited that. The next time the member is used, the copy from USER.Z25A.PARMLIB will be used. This means you keep the original unchanged, and only change the copy. You may want to have a LOADxx without the USER…PARMLIB, in case you make a mistake and the IPL fails!

The SETLOAD command can be used to refresh system symbols via a LOADxx member.

  • If you have the LOADxx members in the PARMLIB concatenation, (and so are not using SYS1.IPLPARM) the SETLOAD WS,IEASYM, command will refresh the symbols, defined by LOADWS.
  • If you have the LOADxx members in SYS1.IPLPARM, then you need to use the command like
    • SETLOAD xx,IEASYM,DSN=SYS1.IPLPARM or
    • SETLOAD xx,IEASYM,DSN=SYS1.IPLPARM,VOL=A5SYS1
    • giving the name, and optionally the volume of the dataset containing the LOADxx member.

You can use a different LOADxx to that used at IPL – so you can change your symbols after the IPL has finished.

Note: The entire set of symbols are deleted, and the specified symbols added, so you need to reload all of them. For example if you specified IEASYM (xx,yy) at IPL, using SETLOAD with just IEASYMyy, you will lose the symbols from IEASYMxx.

Running assembler control block chains in C

I needed to extract some information from z/OS in my C program. There is not a callable interface for the data, so I had to chain through z/OS control blocks.

Once you have an example to copy it is pretty easy – it just getting started which is the problem.

I have code (which starts with PSATOLD)

 #define PSA  540 
 char *TCB   = (char*)*(int*)(PSA); 
 char *TIO   = (char*)*(int*)(TCB + 12); 
 char *TIOE  = (char*)(TIO + 24) ; 
                                                            

  • At absolute address 540 (0x21C) is the address of the currently executing TCB.
  • (int *) (PSA) says treat this as an integer (4 byte) pointer.
  • * take the value of what this integer pointer points to. This is the address of the TCB
  • TCB + 12. Offset 12 (0x0c) in the TCB is the address of the Task I/O table (TCB IO)
  • (int *) says treat this as an integer ( 4 byte) pointer
  • * take the value of it to get to the TIOT
  • Offset 24 (0x18) the the location of the first TIO Entry in the control block

When I copied the code originally had char * (long * ) PSA. This worked fine on 31 bit programs but not on a 64 bit program as it uses 64 bit as an address – not 32 ! I had to use “int” to get it to work.

Another example, which prints the CPU TCB and SRB time used by each address space, is

// CVT Main anchor for many system wide control blocks
#define FLTCVT     16L
//  The first Address Space Control Block
#define CVTASCBH  564L
// the chain of ASCBs - next
#define ASCBFWDP    4L
//  offset to job info
#define ASCBEJST   64L
// the ASID of this address space
#define ASCBASID   36L


__int64 lTCB, lSRB; // could have used long long 
short ASID;  // 0x0000
char *plStor = (char*)FLTCVT;
char *plCVT  = (char*)*(int*)plStor;
char *plASCB = (char*)*(int*)(plCVT+CVTASCBH); // first ASCB
for( i=0; i<1000 & plASCB != NULL;
       i++, plASCB = (char*)*(int*)(plASCB+ASCBFWDP) )
{
  lTCB = *(__int64*)(plASCB+ASCBEJST) >> 12; // microseconds
  lSRB = *(__int64*)(plASCB+ASCBSRBT) >> 12; // microseconds
  ASID = *(short*)(plASCB+ASCBASID));
  printf("ASID=%4.4x TCB=%lld; SRB=%lld\n", ASID, lTCB, lSRB);
}