Setting up IP V6 Linux to z/OS with ADCD

This post follows on from getting TCPIP to work with ADCD (zPDT and ZD&T) and allows me to FTP to z/OS from my Linux machine. There is a Q&A Has anyone configured z/OS running on ZD&T to support IPV6 protocol? but this was not enough information for me.

I’ve written about the IP V6 concepts and how they fit with z/OS.

You need to:

  1. Configure the Linux device map to add an OSA entry in the device map. You specify the path, and z/OS device addresses.
  2. Configure z/OS to support AF_NET6. You can make this change dynamically – or just re-ipl.
  3. Configure TCPIP to add an IFCONFIG6 entry. You need the z/OS AF_NET6 change before the IFCONFIG6 change is accepted. You need to restart TCPIP (or re-ipl).
  4. Configure a VTAM TRLE pointing to the devices defined in the devmap entry. This can be configured dynamically.
  5. Define a TCPIP interface, pointing to the PORTNAME of the TRLE definition. This defines a IP V6 address. The change can be configured dynamically.
  6. Test it!

Configuring the Linux Devmap

Create the Linux devmap entry

[manager]  # tap2 define network adapter (OSA) for communication with Linux
name awsosa 0019 --path=A2 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.25.1.6 
  # QDIO mode
device 408 osa osa 
device 409 osa osa 
device 40a osa osa 

This session has IP V4 address 172.25.1.6, and uses device addresses 408,409 and 40a. It uses path A2.

Restart zD&T to pick up the changes, and re-ipl z/OS.

Configuring z/OS

You have to configure both z/OS and TCPIP to enable TCPIP V6 support.

Update BPXPRM

Use D OMVS,S to show the BPXPRMxx members being used.

Update bpxprmxx with AF_INIT6, by adding the following into a BPXPRMxx member.

NETWORK DOMAINNAME(AF_INET6) 
        DOMAINNUMBER(19) 
        MAXSOCKETS(50000) 
        TYPE(INET) 

Check if your AF_INET is INET or CINET (Common INET is used when you have multiple TCPIP stacks), and specify the same value.

Re IPL.

If you mis configure it

DOMAINNUMBER value 19 is required ( see DOMAINNUMBER ) When I used a different value I got

BPXF202I DOMAIN AF_INET6 WAS NOT ACTIVATED FOR FILE SYSTEM
TYPE INET. RETURN CODE = 0000045A, REASON CODE = 743A0000

and, when TCPIP was started

EZZ0695I IPCONFIG6 STATEMENT ON LINE 1 NOT VALID – IPV6 SUPPORT IS NOT
ENABLED

Where 045A is EAFNOSUPPORT The address family is not supported.

Check AF_INET6 is configured

The command D OMVS,PFS gave me

PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
 INET      EZBPFINI   N/A       SOCKETS   A     2022/09/20 04.08.00     
 NFS       GFSCINIT   NFSC      REMOTE    A     2022/09/20 04.07.23     
 ZFS       IOEFSCM    N/A       LOCAL     A     2022/09/20 04.07.19     
 AUTOMNT   BPXTAMD    N/A       LOCAL     A     2022/09/20 04.07.19     
 UDS       BPXTUINT   N/A       SOCKETS   A     2022/09/20 04.07.19     
                                                                        
PFS TYPE  DOMAIN        MAXSOCK  OPNSOCK  HIGHUSED                      
 INET     AF_INET6       50000        5         5                      
          AF_INET         64000        8         8                      
 UDS      AF_UNIX         10000        2         2                      

Check AF_INET6 is in the list.

Configure TCPIP

I added “include user.Z25A.tcpparms(iconfig6)” into the TCPIP PROF.

This member had just

IPCONFIG6

Restart TCPIP.

The only change when TCPIP was restarted was the additional message

EZZ0300I OPENED INCLUDE FILE ‘USER.Z25A.TCPPARMS(ICONFIG6)’

Check the configuration

On Linux the find_io command gave

FIND_IO for "colinpaice@colinpaice" 
                                                                                                
         Interface Current     MAC     IPv4       IPv6           
 Path    Name      State       Address Address    Address        
------   --------- ----------- ------- ---------  ----------------  -------------- 
  A0     tap0      UP, RUNNING fa:...  10.1.1.1   fe80::f85c:c2ff:fe0a:1415%tap0  
  A1     tap1      UP, RUNNING 5e:...  172.26.1.6 fe80::5cda:64ff:feee:eeaa%tap1  
  A2     tap2      UP, RUNNING 4a:...  172.25.1.6 fe80::4850:5fff:fe5e:87c5%tap2 

Check the interface is UP, RUNNING

Define a VTAM TRLE

You need to create a VTAM TRLE resource. I invented PORTCP, and created member user.z25a.vtamlst(TRLE).

OSATRL3 VBUILD TYPE=TRL                                                 
OSATRL3E TRLE LNCTL=MPC,READ=(0408),WRITE=(0409),DATAPATH=(040A),      X
               PORTNAME=PORTCP,                                        X
               MPCLEVEL=QDIO                                            

This uses address 0408,0409, and 040a (matching the devmap entry above)

Use V net,act,id=trle to activate it.

Note: USER.Z25A.VTAMLST is in the DD concatenation for //VTAMLST.

Use D NET,TRL to display the defined TRLs. This showed

TRLE = OSATRL3E STATUS = NEVAC

Showing the TRLE above, and the status. It becomes ACTIVE when the TCPIP interface is activated.

Create the TCPIP interface definition

IP V6 uses an interface definition instead of a link and device.

  INTERFACE IFPORTCP6  DELETE 
  INTERFACE IFPORTCP6 
    DEFINE IPAQENET6 
    CHPIDTYPE OSD 
    PORTNAME PORTCP 
    INTFID 7:7:7:7  
    IPADDR FD00::67:1:1 

I activated these using

  • v tcpip,,stop,ifportcp6
  • v tcpip,,obey,USER.Z25A.TCPPARMS(IFACE6)
  • You might need v tcpip,,stop,ifportcp6

I found it better to stop the interface before updating it, as sometimes the updates were not all made.

Once these definitions were activated, TSO NETSTAT HOME gave

IntfName:   IFPORTCP6
  Address:  fd00::67:1:1
    Type:   Global
    Flags:
  Address:  fe80::7:7:7:7
    Type:   Link_Local
    Flags:  Autoconfigured

This shows an address fd00::67:1:1 and address fe80::7:7:7:7 based on the INTFID. If you do not specify an INTFID you get a name like fe80::a2:a201:a2:a2a2, based on the chpid (value a2). If the chpid was changed, you would get a different IP address. You can see the chpid from the Linux from_io command, or the z/OS d NET,ID=OSATRL3E,E command.

I could not get any IP address specified in the IPADDR parameter, to work. I could ping to it, but there were no responses.

The interface gets a MAC address based on the CHPID value – for example MACADDRESS: 02A2A2A2A2A2.

NETSTAT ROUTE gave

IPV4 DESTINATIONS                                                     
DESTINATION        GATEWAY         FLAGS    REFCNT     INTERFACE      
127.0.0.1/32       0.0.0.0         UH       0000000000 LOOPBACK       
172.26.1.2/32      0.0.0.0         UH       0000000000 ETH1           
172.26.1.20/32     0.0.0.0         H        0000000000 EZAZCX         
172.26.1.20/32     0.0.0.0         H        0000000000 EZASAMEMVS     
IPV6 DESTINATIONS                                                     
DESTIP:   ::1/128                                                     
  GW:     ::                                                          
  INTF:   LOOPBACK6         REFCNT:  0000000000                       
  FLGS:   UH                MTU:     65535                            
DESTIP:   FD00::67:1/128                                              
  GW:     ::                                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000                       
  FLGS:   UHS               MTU:     1492                             
DESTIP:   FD00::67:1:1/128                                            
  GW:     ::                                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000    
  FLGS:   UH                MTU:     9000             
DESTIP:   FE80::7:7:7:7/128                   
  GW:     ::                                          
  INTF:   IFPORTCP6         REFCNT:  0000000000       
  FLGS:   UH                MTU:     9000                                

The Linux find_io command gave

                                                                                              
      Interface  Current    MAC       IPv4        IPv6           
 Path Name       State      Address   Address     Address        
----- --------- ----------- --------  ----------  -------------- 
...      
. 
  A0 tap0       UP, RUNNING da:...    10.1.1.1    fe80::...tap0  
  A1 tap1       UP, RUNNING 92:...    172.26.1.6  fe80::...%tap1  
  A2 tap2       UP, RUNNING 42:...    172.25.1.6  fe80::...%tap2  

Update the Linux route information

I did this to try to get the IPADDR to work. It did not work, and so is this is not needed.

sudo ip -6 route add fd00::6:1:1/128 dev tap2

Test it!

Use TSO NETSTAT HOME to find the IP V6 address. For example

Address: fe80::7:7:7:7, Type: Link_Local

On Linux use the find_io command to display information about the tunnels to z/OS. Find the tapn matching the chpid being used on z/OS.
Use the

ping fe80::7:7:7:7%tap2

command to send data to z/OS.

The response to the ping will be sent back down the connection the request arrived on.

You can use the tso netstat devlinks(intfname IFPORTCP6 command (where IFPORTCP6 is my interface) to display information about just the specified interface; for example Inbound packets, BytesIn, Outbound packets, BytesOut.

You can use

FTP fe80::7:7:7:7%tap2

then use

tso NETSTAT CONN
or
tso netstat conn (port 21

to see the connections.

You can use NETSTAT ND to display the neighbours. This gave me

Query Neighbor cache for fe80::6a:ffff:feaf:c0e4
IntfName: IFPORTCP6 IntfType: IPAQENET6
LinkLayerAddr: 026AFFAFC0E4 State: Reachable
Type: Host AdvDfltRtr: No

The value fe80::6a:ffff:feaf:c0e4 matches up with the value from find_io on Linux, and

02:6a:ff:af:c0:e4 matches up with the MAC address.

IP V6 concepts and using IP V6 with ADCD

This post follows on from getting TCPIP to work with ADCD (zPDT and ZD&T) and allow me to FTP to z/OS from my Linux machine. There is a Q&A Has anyone configured z/OS running on ZD&T to support IPV6 protocol? but this was not enough information for me.

Background

With IP V4 there is a limit of the number of IP addresses available. IP V6 has many addresses, and so this should not be a problem. There is no smooth migration from IP V4 to IPV6, it is more start with IP V4, run IP V4 and V6 at the same time, move stuff from IP V4 to IP V6, – and possibly (unlikely) run with just IP V6.

Wikipedia has many good articles

  • IP V6 in general
  • IP V4 uses addresses like 192.6.7.1. IP V6 uses addresses like 2001:0db8:0000:0000:0000:ff00:0042:8329 (=2001:db8::ff00:42:8329). See here.
  • Each IP V6 has a local address (link-local) fe80::….
  • An IP V6 address can have :: to mean replace with as many zeros as needed to make this a valid IP V6 address. So 2001:0db8:0000:0000:0000:ff00:0042:8329 can be written 2001:db8::ff00:42:8329. You can only have one :: in a value.
  • TCPIP can support IPV4 and IP V6 at the same time (dual stack)
  • You can wrap an IP V4 address into an IP V6. For example ::ffff:192.0.2.128
  • For security clients often get a “temporary” (or randomised) IP address instead of a hard coded address. This uses a randomiser function with a secret key. This IP address can expire, and a new(different) IP address obtained. This can make it hard(impossible) for a server to do a reverse DNS lookup. This temporary IP address is useful, as it means you cannot be tracked by your IP address.

Other information

  • Each IP node has a IP V4 address and an IP V6 address.
  • When defining connections between systems, it looks like you need at least one IP V4 route, and at least one IP V6 route – I could be wrong.
  • An IPV6 host usually has more than one IP address.

IPV6 has reserved IP ranges

  • 2001:db8::/32 Addresses used in documentation and example source code.
  • fe80::/10 are the link-local unicast [RFC4291] addresses. Addresses within this block should not appear on the public Internet. Your router should not externalise this.
  • fd00::/7 for private internets. These are the unique-local addresses [RFC4193]. Addresses within this block should not appear by default on the public Internet. This means you can use them within your organisation.
  • ffxx is for multicast to all links matching the address. For example ff02::5 is used by the dynamic routing protocol OSPF to say to all routers in the (local) network “hello – anyone there”.

Getting started

The end of each connection needs at least one IP address. If you have 5 connections, you will have at least 5 IP addresses

What is my IP address on Linux ?

You can use hostname -I

which gave me

192.168.1.223
10.1.1.1
172.24.1.6
172.22.1.6
2a00:23c5:8888:9999:0000:1111:2222:3333
2a00:23c5:9999:0000:1111:2222:3333:4444

You can also use ifconfig or ip addr show.

You can also use the z109x find_io command.

What is my IP address on z/OS?

You can use TSO NETSTAT HOME, or the operator command V TCPIP,,NETSTAT,HOME .

This gives information like

LinkName:   ETH1
  Address:  172.26.1.2
    Flags:  Primary
IntfName:   IFPORTCP6
  Address:  fc00::67:1:1
    Type:   Global
    Flags:
  Address:  fe80::a1:a101:a1:a1a1
    Type:   Link_Local
    Flags:  Autoconfigured

This shows

  • 172.26.1.2 an IP V4 address for use within a private network, for connection ETH1.
  • fc00::67:1:1 an IP V6, unique local address, for interface IFPORTCP6.
  • fe80::a1:a101:a1:a1a1a link-local address, for interface IFPORTCP6. The a1:a1… is based on the MAC address of the device. You can override this on z/OS by specifying the INTFID.
  • Note: If you do not specify the INTFID, it will default to the MAC address. If you reconfigure the system, you may get different MAC address, and so the IP address via this interface will change. By specifying the INTFID you can specify what the IP address for this interface, which will not change if the system is reconfigured.

What are the routes on my machine?

IP V4 On Linux ip route or ip -4 route

gives

default via 192.168.1.254 dev wlp4s0 proto dhcp metric 600 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1 
169.254.0.0/16 dev wlp4s0 scope link metric 1000 
172.25.1.0/24 dev tap2 proto kernel scope link src 172.25.1.6 
172.26.1.0/24 dev tap1 proto kernel scope link src 172.26.1.6 
172.27.1.0/24 via 172.25.1.6 dev tap2 
192.168.1.0/24 dev wlp4s0 proto kernel scope link src 192.168.1.223 metric 600 

IP V6 on Linux ip -6 route gives

:1 dev lo proto kernel metric 256 pref medium
2a00:xxx:xxxx:xxxx::/64 dev wlp4s0 proto ra metric 600 pref medium
fe80::/64 dev tap0 proto kernel metric 256 pref medium
fe80::/64 dev tap1 proto kernel metric 256 pref medium
fe80::/64 dev tap2 proto kernel metric 256 pref medium
fe80::/64 dev wlp4s0 proto kernel metric 600 pref medium
default via fe80::966a:b0ff:fe85:54a7 dev wlp4s0 proto ra metric 20600 pref medium

On z/OS, TSO NETSTAT route or V tcpip,,netstat,route gives

IPv4 Destinations
Destination        Gateway         Flags    Refcnt     Interface
-----------        -------         -----    ------     ---------
Default            172.25.1.6      GS       0000000000 IFPORTCP
127.0.0.1/32       0.0.0.0         UH       0000000000 LOOPBACK
172.26.1.2/32      0.0.0.0         UH       0000000000 ETH1
IPv6 Destinations
DestIP:   ::1/128
  Gw:     ::
  Intf:   LOOPBACK6         Refcnt:  0000000000
  Flgs:   UH                MTU:     65535
DestIP:   fc00::67:1:1/128
  Gw:     ::
  Intf:   IFPORTCP6         Refcnt:  0000000000
  Flgs:   UH                MTU:     9000
DestIP:   fe80::a1:a101:a1:a1a1/128
  Gw:     ::
  Intf:   IFPORTCP6         Refcnt:  0000000000
  Flgs:   UH                MTU:     9000

find_io

FIND_IO for "colinpaice@colinpaice" 
                                                                                                
      Interface  Current MAC      IPv4      IPv6           
 Path Name       State   Address  Address   Address        
----- ---------- ------- ------- ---------- -------------- 
  A0 tap0 UP, RUNNING    ea:...  10.1.1.1   fe80::e8e8:69ff:fe20:435b%tap0  
  A1 tap1 UP, RUNNING    22:...  172.26.1.6 fe80::2090:14ff:fee0:5f20%tap1  
  A2 tap2 UP, RUNNING    22:...  172.25.1.6 fe80::2047:afff:fef7:1caf%tap2  

Joining it all up

To FTP from Linux to z/OS, I use

ftp fe80::a1:a101:a1:a1a1%tap1

The fe80::a1:a101:a1:a1a1 came from the z/OS NETSTAT HOME, and is the z/OS end of the connection.

Using another interface (defined with the INTFID 7:7:7:7) I could use

ftp fe80::7:7:7:7%tap2

The ip -6 route command gave me

fe80::/64 dev tap0 proto kernel metric 256 pref medium
fe80::/64 dev tap1 proto kernel metric 256 pref medium
fe80::/64 dev tap2 proto kernel metric 256 pref medium
fe80::/64 dev wlp4s0 proto kernel metric 600 pref medium

So the request for FE80…. can be routed to any of these. I know that it was configured using tunnel interface tap1, so the address to use is fe80::a1:a101:a1:a1a1%tap1.

Once the connection to FTP was established, z/OS TSO NETSTAT CONN gave me

FTPD1    0000003D ESTBLSH                             
  LOCAL SOCKET:   FE80::A1:A101:A1:A1A1..21           
  FOREIGN SOCKET: FE80::2090:14FF:FEE0:5F20..42572    

The request is processed by z/OS address FE80::A1:A101:A1:A1A1 (port 21)

This ties up with DestIP: fe80::a1:a101:a1:a1a1/128 … Intf: IFPORTCP6 which shows the request came in on interface IFPORTCP6

The request came from FE80::2090:14FF:FEE0:5F20, which ties up with fe80::2090:14ff:fee0:5f20%tap1. The request came in over the tap1 interface.

Understanding the Linux, VTAM and TCPIP linkage with ADCD and z/OS

This follows on from Getting TCPIP on my Linux machine to talk to z/OS ADCD running on ZPDT on my Linux machine. It shows the moving parts.

There are lot of bits of configuration which have to be mutually consistent, and some bits do not work as I expected.

There are several sections in this blog post

The different parts

The Linux devmap

In Linux the devmap has

[manager]  # tap0 define network adapter (OSA) for communication with Linux
name awsosa 0022 --path=A0 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.26.1.1 
device 400 osa osa 
device 401 osa osa 
device 402 osa osa 

The key information is

  • It uses z/OS path A0.
  • It is an OSA with QDIO (OSD).
  • It uses the tunnelling interface.
  • The IP address is 172.26.1.1. This is the IP address of the Linux end of the tunnel.
  • There are 3 z/OS devices with address 0400, 0401, 0402.
  • The devices are OSAs.
  • The control unit they are connected to is an OSA. ( A 3279 colour display would have device 3279, control unit 3274).

In ADCD.Z25A.VTAMLST(OSATRL2) is

OSATRL1 VBUILD TYPE=TRL                                                 
OSATRL1E TRLE LNCTL=MPC,READ=(0400),WRITE=(0401),DATAPATH=(0402),      X
               PORTNAME=PORTA,                                         X
               MPCLEVEL=QDIO                                            
OSATRL2E TRLE LNCTL=MPC,READ=(0404),WRITE=(0405),DATAPATH=(0406),      X
               PORTNAME=PORTB,                                         X
               MPCLEVEL=QDIO 

This defines a Transport Resource List(TRL)

  • OSATRL1E is a name which can be used in commands.
  • TRLE is a Transport Resource List Entry
  • It uses MultiPathChannel links(MPC) (more than one device)
  • It uses multiple device addresses 0400,0401,0402
  • The Open Systems Adapter(OSA) port name is PORTA
  • MPCLEVEL=QDIO Indicates that the Queued Direct I/O interface is used for an OSA-Express feature.

You can use commands like

  • D NET,TRL
  • D NET,ID=OSATRL1
  • D NET,ID=OSATRL2E,E

Example output from D NET,IDA=OSATRL1E

DISPLAY ACCEPTED                                                   
NAME = OSATRL1E, TYPE = TRLE 558                                   
STATUS= ACTIV, DESIRED STATE= ACTIV                                
TYPE = LEASED             , CONTROL = MPC , HPDT = YES             
TRL MAJOR NODE = OSATRL2                                          
MPCLEVEL = QDIO       MPCUSAGE = SHARE                            
PORTNAME = PORTA      LINKNUM =   0   OSA CODE LEVEL = 7617       
CHPID TYPE = OSD      CHPID = A0  PNETID = **NA** 
WRITE DEV = 0401 STATUS = ACTIVE     STATE = ONLINE      
READ  DEV = 0400 STATUS = ACTIVE     STATE = ONLINE 
DATA  DEV = 0402 STATUS = ACTIVE     STATE = N/A         

Key information is

  • TYPE = TRLE
  • STATUS= ACTIV
  • PORTNAME = PORTA
  • CHPID = A0
  • There are three devices 0401, 0400, 0402

As this matches the Linux configuration, it has all connected up and become active. It is known to VTAM and TCPIP as PORTA.

Within TCPIP there is a member ADCD.Z25A.TCPPARMS(ZPDTDEV1) with

DEVICE PORTA MPCIPA 
 LINK ETH1 IPAQENET PORTA 
HOME &HOMEIPADDRESS1 ETH1 
BEGINRoutes 
; Destination        SubnetMask    FirstHop       LinkName    Size 
ROUTE 9.114.209.0    255.255.255.0    =            ETH1   MTU 1492 
; Destination                      First Hop      LinkName    Size 
ROUTE DEFAULT             &DEFAULTROUTEADDR        ETH1   MTU 1492 
ENDRoutes 
START PORTA 

The key information is

  • Device PORTA – matches the VTAM TLRE definition.
  • There is a link ETH1 associated with PORTA.
  • When anyone uses this device-> link, use the Home address &HOMEIPADDRESS1. The z/OS symbol &HOMEIPADDRESS1 is “172.26.1.2”, so this value is used.
  • If TCPIP does not know where to send a packet, it sends it to &DEFAULTROUTEADDR down ETH1 link. (&DEFAULTROUTEADDR. = “172.26.1.1”)
  • The “172.26.1.1” matches the –tunnel_ip=172.26.1.1 in the Linux definition.

Background information

A link has two ends, each end defines its IP address. For example

  • for the Linux for the DEVMAP file –tunnel_ip=172.26.1.6 defines the Linux as having IP address 172.26.1.6.
  • for z/OS the HOME 172.26.1.9 ETH2 defines the z/OS as being 172.26.1.9

If you “FTP 172.26.1.9”, and use NETSTAT CONN, it gives a connection 172.26.1.9..21 and 172.26.1.6..53526 , so you can see the IP addresses (and ports) of the two ends of the connection. Note: Port 21 is the well known port for FTP.

The definitions

Devmap

The devmap definitions below, create a two connections between Linux and z/OS.

[manager]  # tap0 define network adapter (OSA) for communication with Linux QDIO
#defaults to  10.1.1.1
name awsosa 0022 --path=A0 --pathtype=OSD --tunnel_intf=y mode
device 400 osa osa 
device 401 osa osa 
device 402 osa osa 

[manager]  # tap1 define network adapter (OSA) for communication with Linux QDIO
name awsosa 0009 --path=A1 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.26.1.6 
device 404 osa osa 
device 405 osa osa 
device 406 osa osa 

The VTAM TRL definition

OSATRL1 VBUILD TYPE=TRL                                                 
OSATRL1E TRLE LNCTL=MPC,READ=(0400),WRITE=(0401),DATAPATH=(0402),      X
               PORTNAME=PORTA,                                         X
               MPCLEVEL=QDIO                                            
OSATRL2E TRLE LNCTL=MPC,READ=(0404),WRITE=(0405),DATAPATH=(0406),      X
               PORTNAME=PORTB,                                         X
               MPCLEVEL=QDIO                                            

The TCPIP definition in USER.Z25A.TCPPARMS(ZPDTDEV1)

DEVICE PORTB MPCIPA 
 LINK ETH2 IPAQENET PORTB 
HOME 172.26.1.9      ETH2 
                                                                        
DEVICE PORTA MPCIPA 
 LINK ETH1 IPAQENET PORTA 
HOME 10.1.1.2        ETH1 
                                                                        
BEGINRoutes 
; Destination        SubnetMask    FirstHop   LinkName    Size 
ROUTE 10.1.1.0       255.255.255.0    =            ETH1   MTU 1500 
ROUTE 172.26.1.0     255.255.255.0    =            ETH2   MTU 1492 
ROUTE DEFAULT                      10.1.1.1        ETH1   MTU 1492 
ENDRoutes 


START PORTA 
START PORTB 

I found that sometimes TCPIP does not 100% accept an update, and the device continues to point to old definitions.

I used the operator commands

v tcpip,,STOP,PORTA
v tcpip,,STOP,PORTB
v tcpip,,obey,USER.Z25A.TCPPARMS(ZPDTDEV1)

to activate the change.

With this I could ping and FTP to 10.1.1.2 and 127.26.1.9 (the values in the HOME)

TSO netstat home gave

MVS TCP/IP NETSTAT CS V2R5       TCPIP Name: TCPIP 
Home address list:
Address          Link             Flg
-------          ----             ---
172.26.1.9       ETH2             P
10.1.1.2         ETH1
172.26.1.20      EZASAMEMVS
127.0.0.1        LOOPBACK

Address          Interface        Flg
-------          ---------        ---
172.26.1.20      EZAZCX

and TSO NETSTAT ROUTE gave

Destination    Gateway  Flags Refcnt     Interface
-----------    -------  ----- ------     ---------
Default        10.1.1.1 UGS   0000000000 ETH1
10.1.1.0/24    0.0.0.0  US    0000000000 ETH1
10.1.1.2/32    0.0.0.0  UH    0000000000 ETH1
127.0.0.1/32   0.0.0.0  UH    0000000000 LOOPBACK
172.26.1.0/24  0.0.0.0  US    0000000000 ETH2
172.26.1.9/32  0.0.0.0  UH    0000000000 ETH2
172.26.1.20/32 0.0.0.0  H     0000000000 EZAZCX
172.26.1.20/32 0.0.0.0  H     0000000000 EZASAMEMVS

Where the flags are

  • UGS – The route is Up, uses a Gateway, is a Static route
  • US -The route is Up, is a Static route
  • US
  • UH – The route is Up, is a route to a Host (rather than a gateway)
  • H – is a route to a Host (rather than a gateway) It does not have “U” so is not up.

More complex example – Using an Interface statement

The definitions above used

DEVICE PORTB MPCIPA 
 LINK ETH2 IPAQENET PORTB 
HOME 172.26.1.9      ETH2 

You can also use the newer interface definitions which combine these

INTERFACE IFPORTCP  DELETE 
INTERFACE IFPORTCP 
  DEFINE IPAQENET 
  CHPIDTYPE OSD 
  PORTNAME PORTCP 
  IPADDR 172.25.1.1 

START IFPORTCP                                      

I activated this by V TCPIP,,OBEYFILE,USER.Z25A.TCPPARMS(IFACE)

This still needs a TRLE, such as

OSATRL3 VBUILD TYPE=TRL                                                 
OSATRL3E TRLE LNCTL=MPC,READ=(0408),WRITE=(0409),DATAPATH=(040A),      X
               PORTNAME=PORTCP,                                        X
               MPCLEVEL=QDIO                                            

Connected by the PORTNAME=PORTCP

More complex example – Using an Interface statement and different IP address ranges

I change the address of the interface

INTERFACE IFPORTCP  DELETE 
INTERFACE IFPORTCP 
  DEFINE IPAQENET 
  CHPIDTYPE OSD 
  PORTNAME PORTCP 
  IPADDR 172.27.1.1 

START IFPORTCP                                      

I activated this by

  • v tcpip,,stop,IFPORTCP
  • V TCPIP,,OBEYFILE,USER.Z25A.TCPPARMS(IFACE)
  • v tcpip,,start,IFPORTCP

TSO NETSTAT HOME now says

IntfName:   IFPORTCP
  Address:  172.27.1.1
    Flags:

Tell Linux of the new address

To use the new IP address, you need to tell Linux where to send the requests to.

sudo ip route add 172.27.1.0/24 via 172.25.1.6

This says that to get to the 172.27… you need to go via 172.25.1.6, which has been defined as

[manager]  # tap2 define network adapter (OSA) for communication with Linux
name awsosa 0019 --path=A2 --pathtype=OSD --tunnel_intf=y --tunnel_ip=172.25.1.6 
  # QDIO mode
device 408 osa osa 
device 409 osa osa 
device 40a osa osa 

z/OS needs to know how to get back from z/OS. In the routes you need

BEGINRoutes 
; Destination        SubnetMask    FirstHop   LinkName    Size 
ROUTE 172.25.1.0     255.255.255.0    =       IFPORTCP    MTU 1492 
ROUTE DEFAULT        172.25.1.6               IFPORTCP    MTU 1492 
ENDRoutesd

Which says anything for 172.25.1.* send down link/interface IFPORTCP.

If no routes match – send it to 172.25.1.6.

Lessons learned

Below are some of the lessons I learned in writing this blog post. I hope they will be useful to other people. They will be useful to me when I come to use this topic and find it does not work!

Defining Linux tunnels

On Linux the tunnels are defined by the zPDT code, you do not have to define them. If you have defined a tunnel, the zPDT will override the definitions.

Activating resources on z/OS

I created my z/OS TCP, and activated them, but they did not work.

If you define a resource which already exists, and is active, parts of the old definition is still used. I had to stop the PORT using V TCPIP,,STOP,PORTA before I could successfully activate the resource.

When I did not stop the PORT I got

PROCESSING COMMAND: VARY TCPIP,,OBEY,USER.Z25A.TCPPARMS(ZPDTDEV1)          
OPENED OBEYFILE FILE 'USER.Z25A.TCPPARMS(ZPDTDEV1)'                        
PROFILE PROCESSING BEGINNING FOR 'USER.Z25A.TCPPARMS(ZPDTDEV1)'            
DEVICE NAME PORTA ON LINE 5 IS ALREADY DEFINED                             
LINK NAME ETH1 ON LINE 6 IS ALREADY DEFINED                                
DEVICE PORTA ON LINE 15 IS ALREADY STARTED                                 
PROFILE PROCESSING COMPLETE FOR FILE 'USER.Z25A.TCPPARMS(ZPDTDEV1)'        
COMMAND VARY OBEY COMPLETED SUCCESSFULLY                                   
INITIALIZATION COMPLETE FOR DEVICE PORTB                                   

When I put a STOP PORTA in the OBEY file, I got

EZZ4308I ERROR: CODE=8010302D DURING ACTIVATION OF DEVICE PORTB. DIAGNOSTIC CODE: 02    
EZD2028I DEVICE PORTB ACTIVATION FAILED - NO DATAPATH DEVICE ADDRESSES ARE AVAILABLE    

When I repeated it – it worked. It looks like a timing issue. So you have to do it in two steps. The meaning of 302D is A ULP cannot use a QDIO device because there are no datapath channel addresses available.

VTAMTRL

When the system was IPLED the VTAM command D NET,TRL gave

DISPLAY TYPE = TRL             
---------------------------------------------------------
 TRL MAJOR NODE = OSATRL2                                
 TRLE = OSATRL1E  STATUS = ACTIV       CONTROL = MPC     
 TRLE = OSATRL2E  STATUS = NEVAC       CONTROL = MPC     
 2 TRLE(S) DISPLAYED                                     

The TRLE entry will only become active when the port is active. Defining it and starting it gave me TRLE = OSATRL2E STATUS = ACTIV CONTROL = MPC

Getting TCPIP to talk to an ADCD system.

I had a new (z/OS 2.5) ADCD system, and wanted to get FTP working. Once I had unzipped the files, and started z/OS, I could not ping or connect to the z/OS via TCPIP.

When I wanted to use z/OS on a different machine I needed to configure my laptop to give the route to the machine using and Ethernet connection

sudo ip route add 10.1.1.0/24 via 10.1.0.3 
# open a session to it
ssh -X colin@10.1.0.3 

Using my home wireless connection I could use instead

sudo ip route add 10.1.1.0/24 dev wlp4s0
# open a session to it
ssh -X colin@10.1.0.3 

The problem

I had ZD&T running on my laptop; My definitions on Linux used an IP address of 10.1.1.2, but the the shipped ADCD system has been configured for a HOME entry of 172.26.1.2.

You can either change z/OS, or you can change your Linux setup.

Changing z/OS

I wanted to get the address 10.1.1.2 on my Linux to talk to z/OS, but z/OS was not configured for this address. Using the TCPIP command NETSTAT HOME showed it was configured for a 172.26.1.2 .

In ADCD.Z25A.TCPPARMS(ZPDTDEV1) is

DEVICE PORTA MPCIPA 
 LINK ETH1 IPAQENET PORTA 
HOME &HOMEIPADDRESS1 ETH1 
BEGINRoutes 
; Destination        SubnetMask    FirstHop       LinkName    Size 
ROUTE 9.114.209.0    255.255.255.0    =            ETH1   MTU 1492 
; Destination                      First Hop      LinkName    Size 
ROUTE DEFAULT             &DEFAULTROUTEADDR        ETH1   MTU 1492 
ENDRoutes 
START PORTA 
                                                                       

The z/OS command D SYMBOLS gave

&DEFAULTROUTEADDR. = "172.26.1.1"         
&DYNXCFIPADDRESS.  = "172.26.1.20"        
&EPHEMERALPORTS.   = "10000 65534"        
&GBLRESL.          = "GBLRESOL"           
&HOMEIPADDRESS1.   = "172.26.1.2" 
&TCPPROF.          = "PROF2"                            f

Updating the address

You have several approaches.

  • Update the system symbols and re-ipl or restart TCPIP
  • Replace the symbols with the hard coded definitions and re-ipl or restart TCPIP
  • Use system symbols and restart just the relevant bits of TCPIP
  • Update the TCPIP definitions and restart just the relevant bits of TCPIP

Update the system symbols and re-ipl

See Changing z/OS system symbols is – easy – ish.

My IEASYM definition included member IEASYMAU.

I created member USER.Z25A.PARMLIB(IEASYMAU), copied member FEU.Z25A.PARMLIB(IEASYMAU) into it, and changed the definitions.

SYSDEF  SYSNAME(S0W1) 
        SYSCLONE(1A) 
        SYMDEF(&HOMEIPADDRESS1.='10.1.1.2') 
        SYMDEF(&DEFAULTROUTEADDR.='10.1.1.1') 
        SYMDEF(&DYNXCFIPADDRESS.='10.1.1.31') 
        SYMDEF(&ZCXDVIPAADDRESS.='10.1.1.32') 
        SYMDEF(&EPHEMERALPORTS.='10000 65534') 
        SYMDEF(&GBLRESL.='GBLRESAU') 
        SYMDEF(&TCPPROF='PROF2') 

I shutdown z/OS and restarted it – and then I could use PING and, once I had started FTPD, I could FTP to 10.1.1.2. If you do not want to restart z/OS see below for the long story. in Changing z/OS system symbols is – easy – ish, or “Refreshing TCP/IP definitions“, below.

Update the TCPIP definitions and restart TCPIP or re-ipl

I backed up ADCD.Z25A.TCPPARMS(ZPDTDEV1), then changed ADCD.Z25A.TCPPARMS(ZPDTDEV1) to use the hardcoded address and default router address.

Restarting TCPIP or a REIPL activated these changes – but doing this is disruptive to users.

Refreshing TCP/IP definitions.

It is not easy to refresh the TCPIP definitions. See here.

Once I had made the changes, and activated the updated definitions, I could ping to z/OS and use FTPD.

Changing Linux

I changed my devmap to have

[manager] # tap0 define network adapter (OSA) for communication with Linux
name awsosa 0022 –path=A0 –pathtype=OSD –tunnel_intf=y –tunnel_ip=172.26.1.1
device 400 osa osa –unitadd=0
device 401 osa osa –unitadd=1
device 402 osa osa –unitadd=2

The “ip route” command gave

default via 192.168.1.254 dev wlp4s0 proto dhcp metric 600
169.254.0.0/16 dev wlp4s0 scope link metric 1000
172.26.1.0/24 dev tap0 proto kernel scope link src 172.26.1.1
192.168.1.0/24 dev wlp4s0 proto kernel scope link src 192.168.1.223 metric 600

The find_io command gave

         Interface         Current          MAC     IPv4       IPv6           
 Path    Name              State            Address Address    Address        
------   ----------------  ---------------- ------- -------    --------- 
  F0     enp0s31f6         UP, NOT-RUNNING  8c:...  *          *               
  F1     wlp4s0            UP, RUNNING      d4....  192....    2a00:...  
  F2     wwan0             DOWN             a2....  *          *               
. 
  A0     tap0              UP, RUNNING      22....  172.26.1.1 fe80...
  A1     tap1              DOWN             02....  *         *               
  ...

I could ping and FTP to 172.26.1.2.

Changing TCPIP configuration on z/OS is not easy.

I wanted to change the TCPIP definitions, so I could FTP from my laptop to the zD&T system running the ADCD z/OS images.

On my one-user, self contained system, it was easiest to make the changes to TCPIP, then stop and restart TCPIP.

If course with multiple users this would be disruptive and may not be possible. You can activate changes to your configuration – but it is not trivial.

Where are the definitions?

In the TCPIP job there is a //PROFILE… statement. This was ADCD.Z25A.TCPPARMS(PROF2). In this member is the TCPIP configuration including several “include adcd.Z25A.tcpparms(…)” statements including “include adcd.Z25A.tcpparms(zpdtdev1)” which had the definitions for the connection I needed to change.

Note: TCPIP explicity includes dataset(member) name. If you copy a member from ADCD.Z25A.TCPPARMS to USER.Z25A.TCPPARMS, it will not be used unless you change the configuration to use the fully qualified name.
You may want to copy a member from ADCD.Z25A.TCPPARMS, to keep a copy of the original, then edit the ADCD.Z25A.TCPPARMS to make your changes.

You can use the OBEYFILE command to make configuration changes once TCPIP has started, this can use any dataset. You can put your new definitions in TEST.TCPPARMS, and use OBEYFILE to activate them. Once they are working as expected, copy them to the ADCD.Z25A.TCPPARMS and have them activated when TCPIP starts.

A typical OBEYFILE command is

v tcpip,tcpip,obeyfile,COLIN.TCPPARMS(DELHOME)

Refreshing TCP/IP definitions.

It is not easy to refresh the TCP/IP definitions. Restarting TCP may be an easier solution.

The definitions I wanted to change were in member zpdtdev1

DEVICE PORTA MPCIPA 
 LINK ETH1 IPAQENET PORTA 
HOME &HOMEIPADDRESS1 ETH1 
BEGINRoutes 
; Destination        SubnetMask    FirstHop       LinkName    Size 
ROUTE 9.114.209.0    255.255.255.0    =            ETH1   MTU 1492 
; Destination                      First Hop      LinkName    Size 
ROUTE DEFAULT             &DEFAULTROUTEADDR        ETH1   MTU 1492 
ENDRoutes 
START PORTA 

I needed to change the system symbols &HOMEIPADDRESS1 and &DEFAULTROUTEADDR to be hard coded values.

There is no “replace” command; you have to delete the definitions and re-add them.

From the above configuration file, the obvious statements are

stop PORTA
delete LINK ETH1
delete device PORTA

but this fails with

EZZ0395I DELETE LINK ETH1 ON LINE 2 FAILED BECAUSE LINK STATE NOT VALID
EZZ0395I DELETE DEVICE PORTA ON LINE 3 FAILED BECAUSE DEVICE HAS A LINK DEFINED

The command TSO NETSTAT DEVLINKS gives the status. This gave me

EZZ2760I DevName: PORTA DevType: MPCIPA
EZZ2766I DevStatus: Not Active CfgRouter: Non ActRouter: Unknown
EZZ2761I LnkName: ETH1 LnkType: IPAQENET LnkStatus: Not Active

The EZZ0395I message said

The link is in use. If this message was issued in response to an attempt to delete a link, the link IP address might still be defined. You must delete the link IP address from the HOME list before the link can be deleted. To remove the link IP address from the HOME list, use the VARY TCPIP,,OBEYFILE command with a profile that contains a HOME statement that does not include the home IP address that is associated with the link that you want to delete. If you specify the updated HOME statement and the DELETE LINK statement in the same VARY TCPIP,,OBEYFILE data set, the HOME statement must precede the DELETE LINK statement.

A replace option would seem a better design than the above.

TSO NETSTAT HOME gave me

EZZ2350I MVS TCP/IP NETSTAT CS V2R5       TCPIP Name: TCPIP           16:28:46
EZZ2700I Home address list:
EZZ2701I Address          Link             Flg
EZZ2702I -------          ----             ---
EZZ2703I 172.26.1.2       ETH1             P
EZZ2703I 10.1.1.31        EZASAMEMVS
EZZ2703I 127.0.0.1        LOOPBACK
 
EZZ2704I Address          Interface        Flg
EZZ2704I -------          ---------        ---
EZZ2703I 10.1.1.31        EZAZCX

I copied the key information into a file – excluding the 172.** stuff

HOME  10.1.1.31        EZASAMEMVS 
      127.0.0.1        LOOPBACK 

and used

v tcpip,tcpip,obeyfile,COLIN.TCPPARMS(DELHOME)

this gave messages

EZZ0344I PERMANENT LOOPBACK ADDRESS 127.0.0.1 SPECIFIED ON LINE 2 CANNOT BE ADDED TO THE HOME LIST     
EZZ0612I HOME ADDRESS 10.1.1.31 FOR LINK EZASAMEMVS ON LINE 1 REPLACES THE PREVIOUS ADDRESS            
EZZ0316I PROFILE PROCESSING COMPLETE FOR FILE 'USER.Z25A.TCPPARMS(DELHOME)'                            
EZZ0303I OBEYFILE FILE CONTAINS ERRORS                                                                 
EZZ0331I NO HOME ADDRESS ASSIGNED TO LINK ETH1                                                         
EZZ0619I LINK EZASAMEMVS USES DUPLICATE HOME ADDRESS 10.1.1.31                                         
EZZ0619I LINK IQDIOLNK0A01011F USES DUPLICATE HOME ADDRESS 10.1.1.31                                   
EZZ0059I VARY OBEY COMMAND FAILED: SEE PREVIOUS MESSAGES                                               

which despite the error messages, it seems to have worked as TSO NETSTAT HOME did not show ETH1.

The output from TSO NETSTAT DEVLINKS showed the deletes had worked, and the device PORTA and link ETH1 where no longer present.

I changed the TCPIP definitions and used

V tcpip,tcpip,obeyfile,ADCD.Z25A.TCPPARMS(ZPDTDEV1)

this worked, and TSO NETSTAT HOME gave

MVS TCP/IP NETSTAT CS V2R5       TCPIP Name: TCPIP           16:44:09
Home address list:
Address          Link             Flg
-------          ----             ---
10.1.1.2         ETH1             P
127.0.0.1        LOOPBACK

Address          Interface        Flg
-------          ---------        ---
10.1.1.31        EZAZCX

and ping 10.1.1.2 worked.

As I said at the top – it was was quicker to restart TCPIP.

Using z/OSMF Network Configuration assistant for TCPIP, to define AT-TLS configuration

I initially found it hard to set up the AT-TLS configuration for MQ. The easiest way was to use the sample configurations provided by MQ. See here for an overview. I used Scenario 5 – Between an IBM MQ for z/OS queue manager and a client application running on IBM MQ for Multiplatforms.

Using the MQ samples, this took about 10 minutes once I had PAGENT and SYSLOGD set up.

I thought I would try to use the TCP provided facilities. There is a lot of documentation, but it is not easy to find what you need. It has been written as an IBM developer, rather than from an end user or task based perspective.

I then thought I would try to use the “way of the future” and the z/OS configuration tool z/OSMF. You use a browser to connect to z/OSMF and do your work through the browser interface. The z/OSMF interface has configuration tools, and workflow tools which guide you step by step through configuration.

I’ve blogged Using z/OSMF workflows for TCPIP. Using the workflow was not very successful.

The Network Configuration Assistance is used to configure the PAGENT, and I used it to define a AT-TLS configuration. Initially this was a struggle as there was no documentation to tell me how to do it. Once I had been through the configuration a couple of times, I found the correct path through the configuration progress and it is relatively painless.

I found In z/OSMF Configuration Assistant, how do I import policy data from the Policy Agent? which may or may not help you

My mission.

My mission was to configure AT-TLS and to provide two ports for use with MQ.

I wanted to do this using two people (me with two userids) and do the typical steps when changing systems, such as saving configurations before changing them, and deploying them, when I had a “change window”.

Using the Network configuration assistant (CA)

AT-TLS concepts

You need to be aware of the AT-TLS concepts when defining the configuration. From an administrator’s perspective:

  • What ports you want to protect? This is known by the CA as Traffic Descriptors. You can specify
    • An IP port
    • A range of IP ports
  • What IP addresses you want to protect.
    • The IP address. A TCP/IP stack can support different IP addresses. You can use a specific IP address. You can select on all IPV4, or all IPV6, or all addresses.
    • The name of a group of IP addresses. z/OSMF CA calls these Address Groups.
  • How do you want to protect the session. For example what levels of TLS, and what cipher specs do you want to use. This is known by the CA as Security levels.
  • The mapping of ports to protecting the session. z/OSMF calls this Requirement Maps.
  • You configure at the TCPIP stack level.
  • z/OSMF has groups of z/OS instances, with one or more z/OS instances, and you can have multiple TCPIP stacks in a z/OS instance.

Backing store

The configuration assistant(CA), stores configuration in a backing store. You can use tools to copy the current store. I found a copy of the file in /global/zosmf/data/app/CAV2R4/backingStore/save Data. I should be able to use standard backup procedures to keep a copy of the file. The resulting configuration is created in a file which is used by PAGENT.

You can copy a backing store within the CA, and so go back to the original one if you need to.

Before you start.

You should collect the information that you will be used to configure PAGENT. For example

  • Which systems and IP stacks will be used.
  • Which keyrings and certificates will be used?
  • For each port you want to protect.
  • What rules do you want, for example which cipher specs.

I found the terms used when creating the rules manually – did not map to the CA concepts, but once you understood the difference in terminology it was ok.

How to define the resources

If you define the configuration bottom up. You define all of the rules, then when you get to configure the TCPIP stack, the rules and other components should all be there, you just have to select them.

If you define the configuration top down, you define the TCPIP stack, then the rules for the stack. You have to define the TCPIP stack, then the rules, then go back and reconfigure the TCPIP stack to add the rules.

I think bottom up configuration is better while you gain experience of the tool. Once you are familiar with the tool then the top down approach will work ok, and may be how you update the configuration.

Getting started

  • Double click on the Network Configuration Assistant icon.
  • You get a page Welcome to V2R4 Configuration Assistant for z/OS Communications Server. On this page you can specify the name of the backing store. The pull down lists the existing backing stores. If you do not have a backing store create one. You can use “tools” button to copy the backing store.
  • The “getting started” gives you information on how to use the panels. I found it a little confusing at times. It displays the help in a separate window. In the Table of Contents, it has “AT/TLS – getting started”. I didn’t find the help or tutorials much use.
  • On the Welcome page, press Proceed.
    • I sometimes get “The backing store is locked by your id.” I got this after I had shutdown down z/OSMF without logging off.
    • You can use “Tools” to manage your backing store, and configuration.
    • “Select a TCP/IP technology to configure” : AT-TLS
    • The layout of the panels, make me think you create the definitions from top to bottom, and so the tabs are defined left to right. I think it is easier to define resources then create the group/image/stack.

Define the rules for which ports to be protected

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Traffic Descriptors tab.

  • Actions -> New…
  • Name COLINTD
  • Actions-> New…
  • Under the Details tab, specify the port or port range and any other information
  • Under the KeyRing tab, specify the keyring and the Certificate label or let it default to the TCPIP stack level keyring.
  • Under the Advanced tab, I let everything default.
  • Click OK
  • You can define a second port within this Traffic Descriptor
  • Click OK

You can press Save to save the current definitions.

Define which IP addresses you want to protect (optional)

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Address Groups tab.

By default it has

  • All_IPv4_Addresses
  • All_IPv6_Addresses
  • All_IP_Addresses

A TCPIP stack can host different IP addresses, one for each connector coming in. If you want to limit rules to particular stack owned IP addresses, create a definition.

  • Actions-> New
  • Name: COLINAG
  • IP Address: 10.1.1.2
  • IP Address: 10.1.1.3
  • OK

You can press Save to save the current definitions.

How do you want to protect the session.

For example what levels of TLS, and what cipher specs do you want to use.

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Security Levels tab.

  • Actions: -> New…
  • Name: COLINSL
  • Select levels of TLS you want to use
  • Next, select cipher specs. I used “Use 2019 suggested values”
  • Next – I took the default (“Do not generate FIPS 140 support“)
  • Click on Advanced settings.
    • If you want to use client authentication click the “Use client authentication” box
    • OK
  • Finish

Your definition should be in the list.

You can press Save to save the current definitions.

Mapping of ports to Session protection

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Requirement Maps tab.

  • Actions: -> New…
  • Name: COLINMAP .
  • In the mappings table,
    • use the Traffic Descriptor pull down and select the Traffic Descriptor you created above. For example COLINTD.
    • Under Security Level pull down select the security definition you created above. For example COLINSL.
  • OK

If I changed an existing definition, I had a pop-up

Modify Requirement Map.
The requirement map you are changing may be referenced in at least one connectivity rule.

Prior to making this change you may want to see which connectivity rules are referencing this requirement map. Click OK to show where used. Click Proceed to proceed with the Modify; otherwise, click Cancel.

Click OK to show where it is used.

Click Proceed

You can press Save to save the current definitions.

Create the group, z/OS instance and TCPIP Stack

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Systems tab.

  • Action: -> Add z/OS group…
  • Name: COLINGR
  • Click OK
  • Action: -> Add z/OS system image…
  • Name: COLMVSA
  • Keyring: START1.COLINRING
  • Press OK
  • I get a pop-up Proceed to the next step? Connectivity rules are configured for each TCP/IP stack. To continue with configuration you need to add a TCP/IP stack to the new z/OS system image. Do you want to add a TCP/IP stack now? Click on Proceed.
  • This gets to the Add TCP/IP stack
  • Name:TCPIP
  • OK
  • I get a pop-up. Proceed to the next Step? to continue with the configuration you should add connectivity rules to the TCP/IP stack. Do you want to be directed to the TCP/IP stack rules panel? Proceed.
    • If you cancel you can use the Actions -> rules to define the rules.
  • I get a pop-up Proceed to the Next Step? Do you want to start a wizard to create a connectivity rule? Click Proceed.
  • This gets to the Data End points where you associate the IP addresses to the stack instance.
  • Name: COLINRN
  • Select from the address group pull-down, or let it default.
  • Press Next
  • This gets to the Requirement Mapping to Stack association.
    • You can select from the existing requirements map see Mapping of ports to Session protection above, or create a new mapping.
    • You can create a new map, for example Name: COLINMP
      • Select from the Traffic Descriptor pull down
      • Select from the Security level pull down.
  • Press Next
  • You can specify advanced settings, such as Tracing, Tuning, Environment, Effective times, Handshake
  • Finish
  • Close

You can press Save to save the current definitions.

Join the bits up

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Systems tab.

  • Select a group instance
  • Actions: Install All files for this group
  • This will list the configuration files.
  • On the Configured File Name,
    • right click on the file name value, and ->Show Configuration File. This will show you the configuration as it might be deployed.
    • right click on the file name value and -> Install … . Specify the a file name and click GO.
    • Close
  • You can use
    • Actions: Install to create the configuration file
    • Actions: Show configuration file to see the generated configuration

You can now use the configuration file as input to PAGENT.

You can press Save to save the current definitions.

Extending the configuration to add a new rule

It took a while to work out how to do this, but it is really easy.

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Traffic Descriptors tab.

  • Create a new Traffic descriptor as above
  • Get back to Network Configuration Assistant (Home) -> AT-TLS page, and click on the Systems tab.
  • Select a TCPIP instance, and click Actions: -> Rules..
  • Actions: -> New
  • Connectivity rule name: rule2
  • Press Next
  • You can select from the existing requirements map see Mapping of ports to Session protection above, or create a new mapping. If you have just created a new rule, then you may not have defined a mapping, and create it “inline”.

Then install it.

Use the configuration

You need to change the PAGENT JCL to use the created configuration file. You may want to copy it to a backup, as the next time you reconfigure it can overwrite the file. Or just create a new file perhaps with a date in the filename.

If you have problems with a newly reconfigured file you need a copy of the previous, working, definitions.

Display the configuration

On many items, you can use right click -> Show where used. This will then display the group, image, stack, connectivity rules and data end points where the item is used.

Should I use this just to get started, or every time.

When I created my definitions by hand, I could put definitions in to a “Common” section, and have multiple TCPIP stacks in one configuration file. I could have small files with bits of configuration in them.

If you use the CA, “common” definitions are copied into the configuration file, and you have one configuration file per TCPIP stack instance, so you do not need to have a common section etc.

As a configuration tool, now I know how to use it, I might continue to use it – but it is slightly more complex than this.

I want to enable trace for one definition. To do so means I have to…

  • Change the configuration to set the trace. This can be difficult if someone else is in the middle of changing the configuration.
  • Deploy the whole configuration. You may pick up incomplete changes which have been made, but not deployed.
  • If a second TCPIP stack is using the configuration, this may get trace enabled if the configuration file is recreated.

Overall (my views may change tomorrow), I would use the CA to create my configuration. Then not use it again – or use it again to generate definitions which I can copy into my master configuration files. I would restructure the configuration so create small files with specific content.

Using z/OSMF workflows for TCPIP.

I found it hard to set up the AT-TLS configuration for MQ. The easiest way was to use the sample configurations provided by MQ. See here for an overview. I used Scenario 5 – Between an IBM MQ for z/OS queue manager and a client application running on IBM MQ for Multiplatforms.

This took about 10 minutes once I had PAGENT and SYSLOGD set up.

I thought I would try to use the TCP provided facilities. There is a lot of documentation, but it is not easy to find what you need. It has been written as an IBM developer, rather than from an end user perspective.

I then thought I would try to use the “way of the future” and the z/OS configuration tool z/OSMF. You use a browser to connect to z/OSMF and do your work through the browser interface. The z/OSMF interface has configuration tools, and workflow tools which guide you step by step through configuration.

I found using the workflow tools was harder than using the TCPIP documentation and TCPIP samples, and I would not recommend its use.

Ive blogged Using z/OSMF Network Configuration assistant for TCPIP, to define AT-TLS configuration. Which worked.

The workflow stuff makes the easy bit “easier”, but does not help with the hard stuff. An improvement would be to skip the workflow, and have one page of instructions saying copy samples into Proclib, and Unix; run a RACF job. We could do with a workflow to help configure syslogd, which I had a struggle to get working in a non trivial situation. For example having error messages for PAGENT go to one file, and have the TLS trace go into another file.

My mission.

My mission was to configure AT-TLS and to provide two ports for use with MQ.

I wanted to do this using two people (me with two userids) and do the typical steps when changing systems, such as saving configurations before changing them, and deploying them, when I had a “change window”.

Initial steps

z/OSMF provides facilities like ISPF, Workload management configuration, system status etc. I used Workflow.

It was hard to know where to start. I assumed (wrongly) that there would be a workflow to define the AT-TLS definitions.

It seems you use Workflow to define the PAGENT and syslogd JCL, and not for configuring the PAGENT or syslogd.

Instructions to use Workflow to configure TCPIP JCL procedures

  • Double click the workflow icon.
  • From the actions pull down, select Create workflow…
  • You need to select Workflow definition file: I could not find what I had to specify. There was no prompting. The “?” basically said “put a value here”. The help key just gave me a panel with information about using creating a workflow.
  • I found an IBM support document which says
    • Workflows for Policy-based Networking
    • ezb_pagent_setup_wizard.xml – This workflow provides the steps for setting up the Policy Agent (Pagent). Pagent is required for all of the policy-based networking technologies: IPSec, AT-TLS, IDS, PBR, and QoS. Pagent uses syslogd for logging.
    • ezb_syslogd_setup_wizard.xml – This workflow provides the steps for setting up syslogd.
    • ezb_tcpip_profile_sample_wizard.xml – This workflow provides a sample TCP/IP profile which contains common statements required to enable AT-TLS and IP Security, and additionally includes port reservation statements for running daemons.
  • I had to use the fully qualified filename /usr/lpp/zosmf/workflow/plugins/izuca/ezb_syslogd_setup_wizard.xml
  • This came up with an error in the workflow name because the default name has ‘z/OS… ‘ and ‘/’ is not a valid character. I removed the ‘/’.
  • At the bottom of the page you can Assign all steps to owner user id. I did not do this, and had to assign steps below
  • You get a list of steps that need to be done.
  • Assign the work to a userid
    • Select all of the steps, and use Actions-> Assignment and ownership -> Add assignees.
    • This displays the assigned roles. I used Actions -> add to add my SAF userid. I pressed OK and returned to the list of steps – all now assigned to me.
  • I selected the first step “define the “RACF userid for Syslogd”, Actions -> Accept .
  • Click on the task, and it gives you a window with tabs. The important tab is Perform. If this is greyed out, you have not accepted the task!
    • Fill in the details and click Next, Next etc. You can edit the contents.
    • You can save it – but you need to give a data set. It suggested SYS1(SYSLOGD). I had to change it (every time) to COLIN.ATTLS(…)
    • Next – gives you the save panel. You have to specify the dataset where you want to save it. The default was wrong for me.
    • Once saved you have to submit it manually, check the output, and edit the file if needed.
  • Back at the workflow details, it had step 1 complete (even though you may not have submitted it)
  • I accepted step 2 and started working on it.
    • It asks for Dataset HLQ – but I could not change it.
    • I stepped through the definitions – and had to type in my dataset again (why can’t it remember what I specified last time).
    • This step just creates a job with some RACF definitions in it.
  • I ran step 3 -again just creating a JCL member of definitions
  • Step 4 “Sample Syslogd Configuration Setup“. This just copies in a sample configuration.
    • “Save” did not do anything
  • Step 5 “Sample started procedure for Syslogd” creates a sample Procedure.
  • On the workflows page, it shows the workflow is 100% complete.

Having been through all of this, the create JCL did not run, one line in error was

//SYSLOGD PROC PROG=”,
// VARS=”,
// PARMS=”
//SYSLOGD EXEC PGM=&PROG., REGION=0K,TIME=NOLIMIT,
// PARM=(‘POSIX(ON) ALL31(ON)’,
// ‘ENVAR(“_CEE_ENVFILE=DD:VARS”)’,
// ‘/&PARMS.’)

  • &PROG had not been specified – you gave to go and find what you need to specified (SYSLOGD)
  • There is a blank after the &PROG., so the REGION=0K,TIME=NOLIMIT, is ignored
  • The location of the configuration (in &VARs) is not specified.

Create the PAGENT JCL

I followed the same process to create the PAGENT file.

I used file /usr/lpp/zosmf/workflow/plugins/izuca/ezb_pagent_setup_wizard.xml.

When this JCL ran, it produced messages

06/16 08:00:20 SYSERR :000: …plfm_config_medium_open: cannot open ‘/etc/pagent.conf’, errno EDC5129I No such file or directory.

You have to know to copy the configuration file from the PDS to /etc/pagent.conf.

Comments on using the workflows

This seems a lot of work to produce code which does not work. The process feels unloved. I am surprised that the problems I found have not been fixed – they are Unit Test level bugs.

I think it is far simpler to follow the documentation, for example to create the procedure. The documentation says

Update the cataloged procedure, syslogd, by copying the sample in SEZAINST(SYSLOGD) to your system or recognized PROCLIB. Specify syslogd parameters and change the data set names to suit your local configurtion See the syslog daemon section of SEZAINST(EZARACF) for SAF considerations for started procedures

The instructions could be on one side of paper, and would be quicker than using the workflow.

Debugging AT-TLS session problems

I deliberately misconfigured AT-TLS to see how easy it would be to identify and resolve the problems from an AT-TLS perspective. It turned out worse than I expected. There is little information on the z/OS to help you.

I configured TTLSEnvironmentAction {trace 255 } (see the bottom of this blog) and refreshed the PAGENT. I had configured SYSLOGD so records for *.TCPIP.*.* went to /var/log/TCPIP.

I reran my MQ client application and got

  • from MQ on Linux, in file /var/mqm/errors/AMQERR01.LOG return code 2393 (MQRC_SSL_INITIALIZATION_ERROR).
  • On Linux there was a file /var/mqm/trace/AMQ.SSL.TRC – which only IBM can format!
  • From TCPIP on z/OS EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900 JOBNAME: CSQ9CHIN RULE: REMOTE-TO-CSQ1 USERID: START1 GRPID: 0000001B ENVID: 0000000B CONNID: 0000006E This give
    • the address of my client,
    • the name of the chinit
    • which AT-TLS rule was used

The message EZD1287I TTLS Error RC: 402 Initial Handshake pointed me to Cryptographic Services System Secure Sockets Layer Programming – No SSL cipher specifications. The first reason was

The client and server cipher specifications do not contain at least one value in common. Client and server cipher specifications might be limited depending on which System SSL FMIDs are installed. See Cipher suite definitions for more information. Server cipher specifications are dependent on the type of algorithms that are used by the server certificate (RSA, DSA, ECDSA, or Diffie-Hellman), which might limit the options available during cipher negotiation.

MQ Trace

I took an MQ trace and formatted it. I used grep to find which file had “Cipher” in it.

Within this file I searched for Start of GSKit TLS Handshake Transcript.

This had information sent to the server as part of the handshake, and further down it had the reason code. You can see from the example that the fields and their values have been displayed (so cipher spec 003c is displayed as tls_rsa_with_aes_128_cbc_sha256)

Start of GSKit TLS Handshake Transcript (1119 bytes)
 <client_hello>
 client_version 
 TLSV12
 random 
   gsksslDissector_32Bits
   7f9d66d8
   gsksslDissector_Opaque
   Length: 28
   3E 5B 45 66 EE A3 C1 9F FB 81 0C 2F 38 19 DF 95     >[Ef......./8...
   5A 1B 54 CC B8 CB B6 C9 87 39 5E 88                 Z.T......9^.
 session_id 
 Length: 00
 cipher_suites 
 Length: 04
 00 FF 00 3C                                         ...<
 tls_ri_scsv,tls_rsa_with_aes_128_cbc_sha256
 compression_methods 
 Length: 01
 00                                                  .
 Extensions
 Length: 74
 00 0D 00 18 00 16 06 01 05 01 04 01 03 01 02 01     ................
 06 03 05 03 04 03 03 03 02 03 02 02 00 00 00 2A     ...............*
 00 28 00 00 25 73 79 73 74 65 6D 32 65 2D 64 65     .(..%system2e-de
 66 32 65 2D 73 76 72 63 6F 6E 6E 2E 63 68 6C 2E     f2e-svrconn.chl.
 6D 71 2E 69 62 6D 2E 63 6F 6D                       mq.ibm.com
  Extension Count: 2
  signature_algorithms 13
   rsa:sha512,rsa:sha384,rsa:sha256,rsa:sha224,rsa:sha1,  
   ecdsa:sha512,ecdsa:sha384,ecdsa:sha256,ecdsa:sha224,
   ecdsa:sha1,dsa:sha1
  server_name 0
   system2e-def2e-svrconn.chl.mq.ibm.com
End of GSKit TLS Handshake Transcript
{  rriEvent
 ...
 RetCode = 20009665, rc1 = 420, rc2 = 0, Comment1='SYSTEM.DEF.SVRCONN', 
 Comment2='gsk_secure_soc_init', Comment3='10.1.1.2(1414)'
 ...
}

With this trace, I am able to see what was sent to z/OS.

The AT-TLS Trace

The trace ( configured in syslogd to be in /var/log/TCPIP) had a one line entry with (I’ve reformatted it to make it easier to read).

Map CONNID: 0000006B 
LOCAL: 10.1.1.2..1414 
REMOTE:
10.1.0.2..53898 
JOBNAME: CSQ9CHIN 
USERID: START1 
TYPE: InBound 
STATUS: Enabled 
RULE: REMOTE-TO-CSQ1 
ACTIONS:
CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A

and data

RC: 0 Connection Init
Initial Handshake ACTIONS: CSQ1-GROUP-ACTION CSQ1-INBOUND-ENVIRONMENT-ACTION N/A HS-Server
RC: 0 Call GSK_SECURE_SOCKET_OPEN - 00000052FD6228F0
RC: 0 Set GSK_FD(300) - 000000000000006B
RC: 0 Set GSK_USER_DATA(200) - 000000007EC32430
RECV CIPHER 160303007B 

and one loooong record with

RECV CIPHER 
010000770303749ED51D8DC7794EE6AC36B01FD115F38A4B0812D35 
C80A5F95DB840C35735CA00000400FF003C0100004A000D00180016 
060105010401030102010603050304030303020302020000002A002 
800002573797374656D32652D64656632652D737672636F6E6E2E63 
686C2E6D712E69626D2E636F6D 
SEND CIPHER 15030300020228 

From the AT-TLS trace of the data received from the client, it is the data as received, and has not been split down into useful fields.

I could not find any documentation on how to format this string. It is not easy to create a program to format this (and get it right), for example converting cipher spec 003c to TLS_RSA_WITH_AES_128_CBC_SHA256. However I have a REXX exec which works in ISPF and decodes the data into fields, but not the contents of the fields – so the cipher spec is reported as 003c

I had some success taking this data, and creating a file which Wireshark could process. See Wireshark – using external data: Bodging a hex dump file. This was not always successful, as it looks like the data is truncated, and can have non hex data in the hex stream.

Note, the System SSL server started task, GSKSRVR, can capture System SSL trace. The output is like

Job TCPIP     Process 0101001D  Thread 00000001  read_v3_client_hello            
Received CLIENT-HELLO message                                                  

with no detailed information

Tracing just one session

If you have a busy system you could get trace data for many sessions. You may want to set up a TLS rule, so you use a “debug port”, or you specify the remote host IP address, and port, using information from the error message

EZD1287I TTLS Error RC: 402 Initial Handshake LOCAL: 10.1.1.2..1414 REMOTE: 10.1.0.2..53900

And dont forget…

And do not forget to reset the TTLSEnvironmentAction entry to reset the trace, and to refresh the PAGENT.

Wireshark – using external data

Export the trace in hex format

  • Capture your trace.
  • Select the line(s) of interest
  • File -> Export Packet Dissections -> As Plain Text
  • On the next screen
    • Select the directory and file name
    • Select the Export as “Plain text” – it adds .txt to the file name you chose above, if required.
    • Select the packet range
    • Packet format: Bytes (and only bytes)
    • Save

If you select Packet Format : Bytes you get output like

0000  00 d8 61 e9 31 2a 8c 16 45 36 f4 8a 08 00 45 10   ..a.1*..E6....E.
0010  00 78 39 c5 40 00 40 06 ec a4 0a 01 00 02 0a 01   .x9.@.@.........
0020  00 03 bc 3e 00 16 0d 5b 33 ab 44 ae 0b 8f 80 18   ...>...[3.D.....

If you select Packet Format: Details

You get

No.     Time           Source                Destination           Dst port port   Protocol Length Info
      1 0.000000000    10.1.0.2              10.1.0.3              22       48190  SSH      134    Client: Encrypted packet (len=68)

Frame 1: 134 bytes on wire (1072 bits), 134 bytes captured (1072 bits) on interface enp0s31f6, id 0
Ethernet II, Src: LCFCHeFe_36:f4:8a (8c:16:45:36:f4:8a), Dst: Micro-St_e9:31:2a (00:d8:61:e9:31:2a)
Internet Protocol Version 4, Src: 10.1.0.2, Dst: 10.1.0.3
Transmission Control Protocol, Src Port: 48190, Dst Port: 22, Seq: 1, Ack: 1, Len: 68
SSH Protocol

If you specify If you select Packet Format: Details + Bytes you get both sets of output in the file.

If you try to import this files back into Wire Shark, it should work, as it will ignore lines which do not begin with a hexadecimal offset.

Import a hex dump file

Using the “bytes” file created above

  • File -> Import from Hex Dump
  • File /u/colin/aaa.txt
  • Offsets: Hexadecimal
  • Encapsulation type: Ethernet

The data is displayed as if it was real time capture.

The Wireshark documentation on importing data is here.

Information in the imported file.

Various bits of information are not in the hex dump file, and WireShark creates (fakes) them. For example

  • The packet number is the data record number of the data in the file.
  • The time between packets is an incrementing microsecond counter, 0.000000, 0.000001, 0.000002 etc
  • The packet length is taken from the length of the data record.
  • The “arrival time” is the time the data was read from the file.
  • The encapsulation type is taken from the import page.
  • However information such as source, destination, port, destination port are taken from the data.

Bodging a hex dump file

I had a AT-TLS trace from PAGENT which was a string of raw hex data, with no interpretation of the TLS data. AT-TLS does not provide any way of formatting this data.

I was partially successful in using WireShark to process this data, and decode the TLS fields.

The first part of the data is the IP and TCP header info, followed by the TLS data.

I obtained a hex dump of a TLS handshake and took the first x42 bytes.

0000  8c 16 45 36 f4 8a 00 d8 61 e9 31 2a 08 00 45 00 
0010  xx xx 01 3d 00 00 3f 06 5c ef 0a 01 01 02 0a 01 
0020  00 02 05 86 b8 dc 34 b1 03 17 57 c4 a5 0c 80 18
0030  0f fc 1d bf 00 00 01 01 08 0a 9d cf 5c b7 db 69 
0040  e9 0d 

I then wrote an ISPF editor rexx program to take the TLS trace and convert into the similar format, such as

0042   16 03 03 08 8E 02 00 00 4D 03 03 62 A8 75 18 C9
0052   5D 61 E1 1B 71 40 6A 6B 95 A8 F9 E5 E8 3A 83 AB
....
07E2   F0 61 98 51 92 4B 2E 0A 29

Then changed the xx xx to the length of the data – 12. So xx xx becomes (x7ea – 12) = x07dd.

The data could sometimes be imported.

If anyone is interested I can send them the ISPF rexx exec.

The trace from PAGENT is some times incomplete or wrong. I had long records with

0A29C..TRC at the end, which is clearly not hex data.

Hints on bodging.

It took me several hours to get the first successful import of TLS data. Some of the problems I had were

  • The offsets were wrong. I sometimes had the wrong offset, or the same offset more than once
  • The xx xx needs to be correct.
  • In the PAGENT trace file I had data like 160303007B which is of length 10 – but only length 5 hex characters. The length would be 00 47 ( 00 42 + 5)

Netstat, TTLS and AT-TLS

Once a session has been established using AT-TLS to do end to end TLS encryption, you can use netstat to display information about the session, and what configuration is being used. It feels slightly incomplete, in that some of the data I expected is not available.

What sessions are using a Port? – and display TTLSPolicy information

tso netstat all (port 1414

You can use other filter statements, using port 1414 was easy to specify.

This displays the high level TTLS information, see the blue text below

EZZ2350I MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP           16:40:39
EZZ2550I Client Name: CSQ9CHIN                 Client Id: 000000BB
EZZ2551I Local Socket: 10.1.1.2..1414          Foreign Socket: 10.1.0.2..51844 
EZZ2577I   BytesIn:            0000002248        BytesOut:           0000002076
EZZ2574I   SegmentsIn:         0000000020        SegmentsOut:        0000000014
EZZ2536I   StartDate:          05/31/2022        StartTime:          16:31:54
EZZ2552I   Last Touched:       16:36:57          State:              Establsh
EZZ2553I   RcvNxt:             0626077759        SndNxt:             3598815082
EZZ2554I   ClientRcvNxt:       0626076574        ClientSndNxt:       3598812426
EZZ2555I   InitRcvSeqNum:      0626074325        InitSndSeqNum:      3598810349
EZZ2556I   CongestionWindow:   0000018720        SlowStartThreshold: 0000065535
EZZ2557I   IncomingWindowNum:  0626208746        OutgoingWindowNum:  3598877418
EZZ2558I   SndWl1:             0626077759        SndWl2:             3598815082
EZZ2559I   SndWnd:             0000062336        MaxSndWnd:          0000064256
EZZ2560I   SndUna:             3598815082        rtt_seq:            3598814997
EZZ2561I   MaximumSegmentSize: 0000001440        DSField:            00
EZZ2563I   Round-trip information:
EZZ2564I     Smooth trip time: 7.000             SmoothTripVariance: 19.000
EZZ2565I   ReXmt:              0000000000        ReXmtCount:         0000000000
EZZ2572I   DupACKs:            0000000000        RcvWnd:             0000130987
EZZ2566I   SockOpt:            88                TcpTimer:           00
EZZ2567I   TcpSig:             04                TcpSel:             40
EZZ2568I   TcpDet:             E4                TcpPol:             00
EZZ2593I   TcpPrf:             89                TcpPrf2:            20
EZZ2593I   TcpPrf3:            00
EZZ2593I   DelayAck:           Yes
EZZ2537I   QOSPolicy:          No
EZZ2545I   TTLSPolicy:         Yes
EZZ2546I     TTLSRule:         REMOTE-TO-CSQ1
EZZ2547I     TTLSGrpAction:    CSQ1-GROUP-ACTION
EZZ2548I     TTLSEnvAction:    CSQ1-INBOUND-ENVIRONMENT-ACTION
EZZ2542I   RoutingPolicy:      No
EZZ2570I   ReceiveBufferSize:  0000065536        SendBufferSize:     0000065536
EZZ2538I   ReceiveDataQueued:  0000000000
EZZ2539I   SendDataQueued:     0000000000
EZZ2611I   SendStalled:        No
EZZ2609I   Ancillary Input Queue: N/A
...

From the clientid (connection id) display any TTLS information

From the netstat allconn (port 1414 command, you get each session, and its clientid (see above for clientid 000000BB).

From the tso netstat allconn (port 1414 command, you get one line per session with the connection ID and remote IP address and port.

EZZ2350I MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP           16:55:18
EZZ2585I User Id  Conn     Local Socket           Foreign Socket         State
EZZ2586I -------  ----     ------------           --------------         -----
EZZ2587I CSQ9CHIN 000000BB 10.1.1.2..1414         10.1.0.2..51844        Establsh
EZZ2587I CSQ9CHIN 00000022 0.0.0.0..1414          0.0.0.0..0             Listen

Issue the command to display the TTLS information details about connection.

tso netstat ttls conn 000000BB detail

MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP  
ConnID: 000000bb
  JobName:      CSQ9CHIN
  LocalSocket:  10.1.1.2..1414
  RemoteSocket: 10.1.0.2..53230
  SecLevel:     TLS Version 1.2
  Cipher:       003C TLS_RSA_WITH_AES_128_CBC_SHA256
  KeyShare:     N/A
  CertUserID:   N/A
  MapType:      Primary
  FIPS140:      Off
  SessionID:    01010018 0A010002 CFEE0000 00000000
                00000000 00000000 62970B05 00000001
  SIDReuseReq:  Off
TTLSRule: REMOTE-TO-CSQ1
  Priority:       1
  LocalAddr:      All
  LocalPort:      1414
  RemoteAddr:     All
  RemotePort:     All
  JobName:        CSQ9CHIN
  Direction:      Inbound
  TTLSGrpAction:  CSQ1-GROUP-ACTION
    GroupID:                    00000007
    TTLSEnabled:                On
    CtraceClearText:            Off
    Trace:                      2
    SyslogFacility:             Daemon
    SecondaryMap:               Off
    FIPS140:                    Off
  TTLSEnvAction:  CSQ1-INBOUND-ENVIRONMENT-ACTION
    HandshakeRole:              Server
    SuiteBProfile:              Off
    MiddleBoxCompatMode:        Off
    Keyring:                    START1/MQRING
    V3CipherSuites:             003C TLS_RSA_WITH_AES_128_CBC_SHA256
    Trace:                      255
    SSLV2:                      Off
    SSLV3:                      Off
    TLSV1:                      Off
    TLSV1.1:                    Off
    TLSV1.2:                    On
    TLSV1.3:                    On
    ResetCipherTimer:           0
    ApplicationControlled:      Off
    HandshakeTimeout:           10
    CertificateLabel:           ZZZZ
    SecondaryMap:               Off
    TruncatedHMAC:              Off
    ClientMaxSSLFragment:       Off
    ServerMaxSSLFragment:       Off
    ClientHandshakeSNI:         Off
    ServerHandshakeSNI:         Off
    ClientECurves:              0021 secp224r1
                                0023 secp256r1
                                0024 secp384r1
                                0025 secp521r1
                                0019 secp192r1
                                0029 X25519
    ClientKeyShareGroups:       0023 secp256r1
    ServerKeyShareGroups:       0023 secp256r1
                                0024 secp384r1
                                0025 secp521r1
                                0029 X25519
                                0030 X448
    SignaturePairs:             0601 TLS_SIGALG_SHA512_WITH_RSA
                                0603 TLS_SIGALG_SHA512_WITH_ECDSA
                                0501 TLS_SIGALG_SHA384_WITH_RSA
                                0503 TLS_SIGALG_SHA384_WITH_ECDSA
                                0401 TLS_SIGALG_SHA256_WITH_RSA
                                0403 TLS_SIGALG_SHA256_WITH_ECDSA
                                0402 TLS_SIGALG_SHA256_WITH_DSA
                                0301 TLS_SIGALG_SHA224_WITH_RSA
                                0303 TLS_SIGALG_SHA224_WITH_ECDSA
                                0302 TLS_SIGALG_SHA224_WITH_DSA
                                0201 TLS_SIGALG_SHA1_WITH_RSA
                                0203 TLS_SIGALG_SHA1_WITH_ECDSA
                                0202 TLS_SIGALG_SHA1_WITH_DSA
                                0806 TLS_SIGALG_SHA512_WITH_RSASSA_PSS
                                0805 TLS_SIGALG_SHA384_WITH_RSASSA_PSS
                                0804 TLS_SIGALG_SHA256_WITH_RSASSA_PSS
    ClientAuthType:             Required
    CertValidationMode:         Any
    Renegotiation:              Default
    RenegotiationIndicator:     Optional
    RenegotiationCertCheck:     Off
    3DesKeyCheck:               Off
    ClientEDHGroupSize:         Legacy
    ServerEDHGroupSize:         Legacy
    PeerMinCertVersion:         Any
    PeerMinDHKeySize:           1024
    PeerMinDsaKeySize:          1024
    PeerMinECCKeySize:          192
    PeerMinRsaKeySize:          1024
    ServerScsv:                 Off
    GSK_V3_SESSION_TIMEOUT:     86400
    GSK_V3_SIDCACHE_SIZE:       512
    GSK_SESSION_TICKET_CLIENT_ENABLE:      On
    GSK_SESSION_TICKET_CLIENT_MAXSIZE:     8192
    GSK_SESSION_TICKET_SERVER_ENABLE:      On
    GSK_SESSION_TICKET_SERVER_ALGORITHM:   AESCBC128
    GSK_SESSION_TICKET_SERVER_COUNT:       2
    GSK_SESSION_TICKET_SERVER_TIMEOUT:     300
    GSK_SESSION_TICKET_SERVER_KEY_REFRESH: 300
    HttpCdpEnable:              Off
    HttpCdpProxyServerPort:     80
    HttpCdpResponseTimeout:     15
    HttpCdpMaxResponseSize:     204800
    HttpCdpCacheSize:           32
    HttpCdpCacheEntryMaxsize:   0
    OcspAiaEnable:              Off
    OcspProxyServerPort:        80
    OcspRetrieveViaGet:         Off
    OcspUrlPriority:            On
    OcspRequestSigalg:          0401 TLS_SIGALG_SHA256_WITH_RSA
    OcspClientCacheSize:        256
    OcspCliCacheEntryMaxsize:   0
    OcspNonceGenEnable:         Off
    OcspNonceCheckEnable:       Off
    OcspNonceSize:              8
    OcspResponseTimeout:        15
    OcspMaxResponseSize:        20480
    OcspServerStapling:         Off

Which AT-TLS groups are being used?

I didn’t find this information very useful. It isn’t clear what a group is. The doc says

Use the TTLSGroupAction statement to specify parameters for a Language Environment process required to support secure connections. The TTLSGroupAction statement indicates whether a selected connection should use AT-TLS security.

tso netstat ttls group
tso netstat ttls

MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP         
TTLSGrpAction                             Group ID           Conns
----------------------------------------  -----------------  -----
CSQ1-GROUP-ACTION                         0000003F               1
GrpActOff                                 00000040               0
GrpActOn                                  00000041               0
GA1                                       00000042               0

tso netstat ttls group detail

MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP         
TTLSGrpAction:   CSQ1-GROUP-ACTION
  GroupID:         0000003F
  Tasks:           4                    GroupConns:      1
  WorkQElements:   0                    SyslogQElements: 0
    Env: CSQ1-INBOUND-ENVIRONMENT-ACTION           EnvConns: 1
TTLSGrpAction:   GrpActOff
  GroupID:         00000040
  Tasks:           4                    GroupConns:      0
  WorkQElements:   0                    SyslogQElements: 0
TTLSGrpAction:   GrpActOn
  GroupID:         00000041
  Tasks:           4                    GroupConns:      0
  WorkQElements:   0                    SyslogQElements: 0
TTLSGrpAction:   GA1
  GroupID:         00000042
  Tasks:           4                    GroupConns:      0
  WorkQElements:   0                    SyslogQElements: 0