Category: TCPIP

Using the z/OS DNS on ADCD

This came out of a question. It is another of the little questions that get much bigger.

Background to Domain Name System(DNS)

DNS allows you to get an IP address from a string such as “WWW.MY.COM”.

You can have some files on your local system which provide this mapping, or you can exploit DNS Servers in the big internet.

Some people configure their system so it tries the internet first, and if that fails, uses local files.

You can do reverse DNS lookup, mapping an IP address to a string. For example you want to allow access from sites in WWW.MYFRIEND.COM. When a connection is started, you get the IP address, and can then do a reverse DNS lookup to get a name, which you can check in your “allow” list.

DNS commands for the end user

You can use the “old” tso command NSLOOKUP http://www.ibm.com, or the “new” command dig http://www.ibm.com. Neither of which seemed to give me any output!

The NSLOOKUP and DIG commands send their output to SYSOUT. In my TSO system, SYSOUT has been configured to JES. If I use SDSF, and display the output of my TSO userid, there is a SYSOUT, with the output in it!

NSLOOKUP

The NSLOOKUP command

NSLOOKUP http://www.my.com

NSLOOKUP http://www.my.com this.dns.site

NSLOOKUP 10.1.1.2

Tracing a DNS request

This does not provide much useful information! It does not tell you what happened, or what failed. It is described here.

Starting and stopping the DNS

This is not obvious. At IPL the ADCD.Z24C.PARMLIB(BPXPRM00) member has

RESOLVER_PROC(RESOLVER)

the resolver procedure must be in a data set that is specified by the IEFPDSI DD card specification of the MSTJCLxx PARMLIB member.

If you use D A,L it does not show up.

D A,RESOLVER gives you the normal output.

When I issued 

P RESOLVER
S RESOLVER

It used the RESOLVER procedure from USER.Z24C.PROCLIB, the normal concatenation.

Displaying and changing the configuration.

You can display some of the current resolver configuration using

f resolver,display

The output is like

EZZ9298I RESOLVERSETUP - USER.Z24C.TCPPARMS(GBLRESOL)                   
EZZ9298I DEFAULTTCPIPDATA - USER.Z24C.TCPPARMS(GBLTDATA)                
EZZ9298I GLOBALTCPIPDATA - /etc/resolv.conf                             
EZZ9298I DEFAULTIPNODES - ADCD.Z24C.TCPPARMS(ZPDTIPN1)                  
EZZ9298I GLOBALIPNODES - /etc/hosts                                     
EZZ9304I COMMONSEARCH                                                   
EZZ9304I CACHE                                                          
EZZ9298I CACHESIZE - 200M                                               
EZZ9298I MAXTTL - 2147483647                                            
EZZ9298I MAXNEGTTL - 2147483647                                         
EZZ9304I NOCACHEREORDER                                                 
EZZ9298I UNRESPONSIVETHRESHOLD - 25

The only way I could display all of the resolver configuration was to get a resolver trace!

//IBMRESO JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTCPT DD SYSOUT=* 
//SYSPRINT DD SYSOUT=* 
//SYSTSIN DD * 
NSLOOKUP 99.99.99.99 
/*

This gave me in //SYSTCPT

Resolver Trace Initialization Complete -> 2023/02/26 18:01:56.725504                      
res_init Parse error on line     1: /etc/resolv.conf                                      
                                                                                          
res_init Resolver values:                                                                 
 Setup file warning messages = No                                                         
 CTRACE TRACERES option = No                                                              
 Global Tcp/Ip Dataset  = /etc/resolv.conf                                                
 Default Tcp/Ip Dataset = USER.Z24C.TCPPARMS(GBLTDATA)                                    
 Local Tcp/Ip Dataset   = None                                                            
 Translation Table      = TCPIP.STANDARD.TCPXLBIN                                         
 UserId/JobName         = IBMUSER                                                         
 Caller API             = TCP/IP C Sockets                                                
 Caller Mode            = EBCDIC                                                          
 System Name            = S0W1 (from VMCF)                                                
 UnresponsiveThreshold  = 25                                                              
 (D) DataSetPrefix = TCPIP                                                                
 (D) HostName      = S0W1                                                                 
 (D) TcpIpJobName  = TCPIP                                                                
 (*) DomainOrigin  = None                                                                   
 (*) NameServer(s) = None                                                                   
 (*) NsPortAddr    = 53            (*) ResolverTimeout    = 5                               
 (*) ResolveVia    = UDP           (*) ResolverUdpRetries = 1                               
 (*) Options NDots = 1                                                                      
 (D) Trace Resolver                (*) SockNoTestStor                                       
 (D) AlwaysWto     = NO            (D) MessageCase        = MIXED                           
 (*) LookUp        = DNS LOCAL                                                              
 (*) Cache                                                                                  
 (*) NoCacheReorder                                                                         
res_init Succeeded                                                                          
res_init Started: 2023/02/26 18:01:56.794280                                                
res_init Ended: 2023/02/26 18:01:56.794305

This is documented here.

The source of the value is

  • (*) Default value
  • (A) Modified by application
  • (D) Default file (not used if the local file is found)
  • (E) Environment variable
  • (G) Global file
  • (L) Local file

This means the “LookUp = DNS LOCAL ” value came from the default value.

The resolver JCL in USER.Z24C.PROCLIB had

//SETUP DISP=SHR,DSN=USER.Z24C.TCPPARMS(GBLRESOL)

When I changed this member to have LOOKUP LOCAL DSN, and used the F RESOLVER,REFRESH command, this changed the value.

Sample hosts file

The sample host file in TCPIP.SEZAINST(HOSTS) has 

; The format of this file is documented in RFC 952, "DoD Internet 
; Host Table Specification". 
; 
; The format for entries is: 
; 
; NET : ADDR : NETNAME : 
; GATEWAY : ADDR, ALT-ADDR : HOSTNM : CPUTYPE : OPSYS : PROTOCOLS : 
; HOST : ADDR, ALT-ADDR : HOSTNM, NICKNM : CPUTYPE : OPSYS : PROTOCOLS : 
; 
; Where: 
;   ADDR, ALT-ADDR = IP address in decimal, e.g., 26.0.0.73 
;   HOSTNM, NICKNM = the fully qualified host name and any nicknames 
;   CPUTYPE = machine type (PDP-11/70, VAX-11/780, IBM-3090, C/30, etc.) 
;   OPSYS = operating system (UNIX, TOPS20, TENEX, VM/SP, etc.) 
;   PROTOCOLS = transport/service (TCP/TELNET,TCP/FTP, etc.) 
;   : (colon) = field delimiter 
;   :: (2 colons) = null field 
; *** CPUTYPE, OPSYS, and PROTOCOLS are optional fields. 
; 
;   MAKESITE does not allow continuation lines, as described in 
;   note 2 of the section "GRAMMATICAL HOST TABLE SPECIFICATION" 
;   in RFC 952.  Entries should be specified on a single line of 
;   up to a maximum of 512 characters per line. 
HOST : 129.34.128.245, 129.34.128.246 : YORKTOWN, WATSON :::: 
; 
NET  : 9.67.43.0 : RALEIGH.IBM.COM : 
; 
GATEWAY : 129.34.0.0 : YORKTOWN-GATEWAY ::::

Unix application trace

Enable the trace by issuing the Unix command

export RESOLVER_TRACE=~/trace

Run the command

pip install mfpandas

gave

-[33mWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x500B209580>: 
Failed to establish a new connection: 
[Errno 1] EDC9501I The name does not resolve for the supplied parameters.')': /simple/mfpandas/-[0m-[33m

Look at the trace file

oedit trace

gave

GetAddrInfo Started: 2025/11/25 15:58:36.890589 
GetAddrinfo Invoked with following inputs: 
   Host Name:  pypi.org 
   Service Name:  443 
   Hints parameter supplied with settings: 
       ai_family = 0, ai_flags = 0x00000000 
       ai_protocol = 0, ai_socktype = 1 
No NameServers specified, no DNS activity 
GetAddrInfo Opening Socket for IOCTLs 
 BPX1SOC:  RetVal = 0, RC = 0, Reason = 0x00000000, Type=IPv4 
 BPX1IOC:  RetVal = 0, RC = 0, Reason = 0x00000000 
GetAddrInfo Opened Socket 0x00000005 
GetAddrInfo Only IPv4 Interfaces Exist 
GetAddrInfo Searching Local Tables for IPv6 Address 
Global IpNodes Dataset  = ADCD.Z31B.TCPPARMS(ZPDTIPN1) 
Default IpNodes Dataset = ADCD.Z31B.TCPPARMS(ZPDTIPN1) 
Search order            = CommonSearch 
 BPX1ENV Get _BPXK_AUTOCVT:  RetVal = 0, RC = 0, Reason = 0x00000000 
_BPXK_AUTOCVT current value is ON 
 BPX1ENV Set _BPXK_AUTOCVT:  RetVal = 0, RC = 0, Reason = 0x00000000 
_BPXK_AUTOCVT set to OFF 
  Parse error on line   22: ADCD.Z31B.TCPPARMS(ZPDTIPN1) 
 SITETABLE from globalipnodes ADCD.Z31B.TCPPARMS(ZPDTIPN1) 
 - Lookup for pypi.org 
GetAddrInfo Searching Local Tables for IPv4 Address 
 - Lookup for pypi.org 
GetAddrInfo Searching Local Tables for IPv6 Address 
 - Lookup for pypi.org.DAL-EBIS.IHOST.COM 
GetAddrInfo Searching Local Tables for IPv4 Address 
 - Lookup for pypi.org.DAL-EBIS.IHOST.COM 
GetAddrInfo Closing IOCTL Socket 0x00000005 
 BPX1CLO:  RetVal = 0, RC = 0, Reason = 0x00000000 
GetAddrInfo Failed:  RetVal = -1, RC = 1, Reason = 0x78AE1004 
GetAddrInfo Ended: 2025/11/25 15:58:36.904995 

EDIT       /etc/hosts
Command ===>                                                      
****** ******************************************************* Top
==MSG> -Warning- The UNDO command is not available until you chang
==MSG>           your edit profile using the command RECOVERY ON. 
000001 # BEGIN ANSIBLE MANAGED BLOCK 
000002 #72.26.1.2  s0w1.dal-ebis.ihost.com S0W1 
000003 127.0.0.1 localhost 
000004 # END ANSIBLE MANAGED BLOCK 
000005 #IPAddress             Hostname   alias 
000006 151.101.128.223        pypi.org    pip

Why can’t I connect to a z/OS port?

I’ve found couple of those little problems which took me a day to resolve – but which are obvious when you understand the problem.

The problems

I was trying to connect the Health Center in Eclipse to the Health agent in Liberty on z/OS.

The first problem was the health center agent on z/OS could not connect to the port. This was due to bad TCPIP configuration

The second problem was I could not connect to it from Eclipse. I had configured the port to be on the local rather than external interface.

My setup

In my jvm.options I had

-Xhealthcenter:level=off,readonly=off,jmx=on,port=1972

Problem 1: The health center agent on z/OS could not connect to the port

In the Liberty startup output I received (after about a timeout of about a minute)

SEVERE: Health Center agent failed to start. java.io.IOException: Cannot bind to URL [rmi://S0W1:1972/jmxrmi]: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host:

Where my system is called S0W1.

It is trying to connect to system S0W1 port 1972, and failing.

TSO PING S0W1 gave

 CS 3.1: Pinging host S0W1.DAL-EBIS.IHOST.COM (172.26.1.2)
 Ping #1 timed out

This was a surprise to me – I was expecting it to be my local z/OS machine…. I do not have an interface with address 172.26.1.2. This explains why it timed out.

In my ADCD.Z31B.TCPPARMS(GBLTDATA) I had 

S0W1:   HOSTNAME   S0W1 
; 
; 
; NOTE - Use either DOMAINORIGIN/DOMAIN or SEARCH to specify your domain 
;         origin value 
; 
; DOMAINORIGIN or DOMAIN statement 
; ================================ 
; DOMAINORIGIN or DOMAIN specifies the domain origin that will be 
; appended to host names passed to the resolver.  If a host name 
; ends with a dot, then the domain origin will not be appended to the 
; host name. 
; 
DOMAINORIGIN  DAL-EBIS.IHOST.COM

Because S0W1 did not end with a dot – TCPIP put the DOMAINORIGIN on the end.

ADCD.Z31B.TCPPARMS(ZPDTIPN1)

had

172.26.1.2 S0W1.DAL-EBIS.IHOST.COM S0W1      
127.0.0.1       LOCALHOST

Which says for S0W1…. use IP address 172.26.1.2.

I changed this to

S0W1:   HOSTNAME   S0W1.
   127.0.0.1       S0W1        
   127.0.0.1       LOCALHOST

With these changes, I restarted TCPIP, and told the resolver to use the updated configuration.

F RESOLVER,REFRESH,SETUP=ADCD.Z31b.TCPPARMS(GBLRESOL)

I then got 

INFO: Health Center agent started on port 1972.

So my first success. However…

Problem 2 : I could not connect Eclipse to the port

… once I had managed to get get the server to connect to the port. When the server issues a TCPIP binds to a port, you need to specify the IP address and port. I had configured the hostname S0W1 as the local interface (127.0.0.1). When I tried to connect from Eclipse, I was trying to connect to port 1972 on interface 10.1.1.2 – which had not been configured!

The Liberty output had

WARNING: RMI TCP Accept-1972: accept loop for erverSocket[addr=0.0.0.0/0.0.0.0, localport=1972] throws java.io.IOException: EDC5122I Input/output error. (errno2=0x12B804B9)

I changed ADCD.Z31B.TCPPARMS(ZPDTIPN1) to have

10.1.1.2 S0W1
127.0.0.1 LOCALHOST

so the name S0W1 is associated with interface 10.1.1.2. I started restart TCPIP and the resolver and it manage to connect. It only took a day to resolve these problems.

Not for humans – AT-TLS and security messages

EZD1287I TTLS Error RC: 6 Initial Handshake

6 Key label is not found

My server certificate was in the keyring, but it had expired. I renewed it, and got past this.

EZD1287I TTLS 403 No certificate received from partner.

403 No certificate received from partner.

In my curl request I did not have –cert ./colinpaice.pem:password –key ./colinpaice.key.pem, when the server expected a certificate (gpmserve had CLIENT_CERT(ACCEPT)).

My z/OS had HandshakeRole ServerWithClientAuth, but the client did not provide a certificate.

EZD1287I TTLS Error RC: 428 Initial Handshake

The private key cannot be obtained from the certificate.

The server userid needs access to the keyring. If the private key belongs to the server’s userid, then the server’s userid needs read access to the keyring. If the private key belongs to a different userid, the server’s userid needs update access to the keyring. See here for more information.

EZD1287I TTLS Error RC: 435 Initial Handshake

435 Certification authority is unknown.

I got this having replaced the CA certificate. Deleting a certificate removes it from any keyring. When you recreate the CA, you need to add it to every keyring it was in. Before deleting a certificate it is worth listing it to see where it is used. I added it to my keyring and it worked!

EZD1287I TTLS Error RC: 510 No acceptable key labels found

510 No acceptable key labels found

My server certificate was in the keyring, but it did not have the capabilities to support the handshake. For example client only supported RSA, but server was ECC.

pthread_security_np, pthread_security_app_np

This function call can be used to map a certificate to a userid.

I got

ESRCH (143): The user ID provided as input is not defined to the security product or does not have an OMVS segment defined.

errno2 0be8044c. Code 044c is JRNoCertforUser, There is no userid defined for this certificate. Action: Ensure the userid is known to the SAF service.


You also get this message if the mapping from certificate to userid is missing. For example

RACDCERT DELMAP(LABEL('IBMUSER1Label))ID(IBMUSER)
RACDCERT MAP ID(IBMUSER) -
   WITHLABEL('IBMUSER1Label') -
   SDNFILTER('CN=colinpaice.O=cpwebuser.C=GB')
RACDCERT LISTMAP ID(IBMUSER)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

R_ticketserv (IRRSPK00): Parse or extract 8, 16, 28

I was using the service to generate a pass token, and got SAF 8 RACF 16 RS 28.

Unable to generate PassTicket. Verify that the secured signon (PassTicket) function and
application ID is configured properly by referring to Using PassTickets in z/OS Security Server RACF Security Administrator’s Guide.

Action

I needed to define

RALTER  PTKTDATA MYAPPL  SSIGNON(KEYMASKED(7E4304D681920260)) - 
    APPLDATA('NO REPLAY PROTECTION')

Where MYAPPL is the application name.

R_ticketserv (IRRSPK00): Parse or extract 8, 8, 16

With userid = “ADCBD” and APPL = ” MVSS0W1″ I got SAF 0 RACF 0 RS 0.

With userid = “ADCBD” and APPL = ” TSOS0W1″ I got SAF 8 RACF 8 RS 16. Not authorized to use this service.
I used

RDEFINE PTKTDATA   IRRPTAUTH.TSOS0W1.*  UACC(NONE)

and it worked

EZD1287I TTLS Error RC: 467 Initial Handshake

Running from Python – a request worked. Running from Chrome gave me the above message.
I solved it by making the connection TLS 1.3 ON and TLS 1.2 OFF

The doc (Cryptographic Services System Secure Sockets Layer Programming)

says

467 Signature algorithm not in signature algorithm pairs list.
Explanation
A signature algorithm that is used to sign a local or peer certificate is not included in the signature algorithm pairs list or the certificate signature algorithm pairs list. The server certificate chain must use signature algorithms included in the signature algorithm pairs or the certificate signature algorithm pairs that are presented by the client during the TLS handshake. The client certificate chain must use signature algorithms included in the signature algorithm pairs or the certificate signature algorithm pairs that are presented by the server during the TLS handshake. If remote partner specifies the signature algorithm pairs and the certificate signature algorithm pairs, the specified certificate signature algorithm pairs take precedence over the signature algorithm pairs. For TLS V1.2 handshakes, the GSK_TLS_CERT_SIG_ALG_PAIRS setting is only used on the client
side to indicate the signature algorithms that the client supports in the server’s certificate chain.
User response
Ensure that the signatures of the local and peer certificates in the certificate chain use signature algorithms that are present in the signature algorithm pairs list or the certificate signature algorithm pairs list that is presented by the session partner. If the certificate chain is correct, configure the client or server or both to specify all necessary signature algorithms pairs in the GSK_TLS_SIG_ALG_PAIRS or SK_TLS_CERT_SIG_ALG_PAIRS
settings to allow use of the certificate chain. If GSK_TLS_CERT_SIG_ALG_PAIRS is specified, it takes precedence while checking the signature algorithms used in the certificate chain.

Zowe: Planning: How do I protect what Zowe users can do on z/OS?

As part of my planning for Zowe, I wanted to know how I could control what Zowe users can do on z/OS. I could not find any definitions for security profiles, so how do I do it?

It took a few days thinking about this to realise I was looking a the problem the wrong way. The correct way of looking at it, is that Zowe is a transport system for getting requests from a user’s work station to z/OS. This is similar to a 3270 emulator connection to z/OS. You control what the userid can do, and do not try to control what the 3270 emulator can do.

When a userid logs on to TSO through a 3270 emulator, z/OS knows the userid of the address space, and can control access to what resources the userid can access.

When a userid logs on to Zowe there are two paths that can be taken:

  • Zowe can create a TSO address space for the userid, using the CEA facilities. The z/OS® CEA TSO/E address space manager provides services to programmatically start and manage TSO/E address spaces and provides a communications mechanism for use between the caller and the programs running in these managed address spaces.
  • A thread within Zowe can use the pthread_security and change the userid of that thread. It can use your certificate, or a userid and password to validate the user. At the end of the request it resets the userid back.

From a permissions perspective, it does not matter if request came into a TSO address space or as a result of the pthread_security request. The userid is extracted and normal SAF processing is used to manage access to a resource.

But…

Your system may have IPSEC rules which police traffic into and out of TCP/IP on z/OS, for example allow traffic from these external IP address during 0900 to 1700 Monday to Friday, and deny access at any other time.

You might need to have similar rules for connectivity through Zowe. There are several ports used by Zowe and z/OSMF. You need to review what controls you need for these ports to stop unwanted traffic from accessing your system.

How do you get into the well protected castle? – you tunnel

Before I retired, if I wanted access to the corporate systems, I would start up a tool (vpn/dialer) which set up an encrypted session to the corporate front end, and tunnel through this to get to the back end. It was transparent and was like being directly attached.

How do you set up your castle to allow authorised people in – but deny access to unauthorised access?

If you logon directly to a back-end server from a cafe or other public WiFi, your credentials etc may be in the clear. You need to use “a secure app” such as banking app, or provide a vpn or tunnel to access your system to encrypt the traffic.

I’ve been logging on to someone else’s system and most of the useful connections to this are disabled. I could access via 3270 but could not use FTP etc.

I was trying to debug a TLS problem and found suspicious evidence in the trace. A connection was being made to the TLS port on my server, and failing because it did not speak TLS. Someone had clearly found the IP address of my system, and was trying all ports to get in!

How do you protect the system and keep unwanted people out – but allow authorised people to connect. You need to do both of…

Secure tunnelling

You could use TLS to protect the conversation between client and server, but this means you have allowed the connection to get through TCP/IP and to your application before checking to see if the connection is permitted.

Port forwarding or tunnelling controls access to your system at the outer edge of your system, logically within TCP/IP, before it gets to your application. It is well described here.

I used the SSH command

ssh -N -L 9876:12.23.34.45:8765 colin@23.34.45.56 -p 2222

If I used address https://localhost:9876 in my web browser, I am connected to 12.23.34.45:8765 through system 23.34.45.56 .

Where

  • -N says port forwarding
  • my local port 9876 is linked to 12.23.34.45 port 8765
  • via IP address 23.34.45.56. Think of this as the guard house.
  • colin is the userid. You get prompted for your password.
  • use port 2222 at the guard house. Think of multiple entrances to the guard house. “General public” “special guest” “people who work here”. The port says go to the “special guest” door. The first time, you have to successfully logon with your userid and password to get your details recorded as a valid visitor to the site. You are given a token which is saved for next time you want access (think of it as a visitor badge).

What is IPSEC?

IPSEC is part of Communications Server on z/OS. It provides

  • IP filtering to control which packets can enter the system
  • IP filtering to control which packets can leave the system

You can filter on

  • packet information – allow a ping, but not FTP
  • network attributes – only allow from a list of IP address
  • time – at certain times of day. For example a normal working day – so if someone is trying to access this system at two o’clock in the morning, this would be worth investigating.

You can

  • allow
  • deny
  • log

You need to be careful setting the rules up – if you want to allow traffic in from 1.2.3.4, then you need to allow traffic out to address 1.2.3.4

My rule

I would set up an input rule like

For input port 8765 deny all access, all packet types, all hours of the day.

Using the tunnel needs a little care

The browser validates the certificate sent from the server. If there ALTNAME, the client(browser) should check that the IP address specified in the ALTNAME matches the IP address used in the original URL, and your connection has not been high jacked.

In my browser I used https://127.0.0.1:9876, where 9876 was specified in the ssl command above. The certificate at the server had ALTNAME with IP address of 127.0.0.1, and so this was considered valid. The IP ALTNAME IP adddress is usually the IP address of the server (or one of them if there are more than one) so you may get messages saying an insecure certificate is being used. You can accept this, or configure your browser to ignore these checks (which is not a good idea)

Thanks

Thanks to Lionel Dyck and Randy Rackov for their help in this blog post.

How to get a file from z/OS to a different z/OS without using FTP

I have a userid on a z/OS production system, which does not support FTP. To run my tests, I needed to get some files on to this system. Getting the files there was a challange.

The 3270 emulator has support for transferring files. It uses the IND$FILE TSO command to send data packaged as 3270 datastream As far as I can tell, this only works with data sets, not Unix files.

Creating a portable file from a data set.

You can package a data set into a FB Lrecl 80 dataset using the TSO XMIT (TRANSMIT) command.

Create a portable dataset from a Unix file.

On my home system I created a PAX dataset from a file in a Unix directory.

Use cd to get into the directory you want to package. If you specify a file name like /tmp/mypackage, the unpax will store the output in /tmp/mypackage which may not be where you want to store the data.

If you use relative directories such as ‘.’ it will unpax into a relative directory. I used the cd command to get into my working directory

pax -W "seqparms='space=(cyl,(10,10))'" -wzvf  "//'COLIN.ZOWE.PAX'" -x os390  myfile

You need both the single and double quotes around the data set name.

This created a data set with record format FB, and Lrecl 80.

A 360 MB file became a 426 CYL data set.

If you run out of space ( B37-04 abend). Delete the dataset before you reissue the pax command, otherwise the space parameters on the pax command are ignored; and increase the amount of space in the pax command.

I FTPed this down to my Linux machine in binary mode.

Send the file to the remote z/OS over 3270 emulator

Because FTP was not available I had to use the TSO facility IND$FILE. One of the options from the “file” menu was “File Transfer”.

You fill in details of the local file name, the remote data set name, and data set attributes.

In theory you need to be in TSO option 6 – where you can enter TSO commands, but when I tried this I kept getting “input field too small”. I had to exit ISPF and get into native TSO before the command worked.

The transfer rate is very slow. It sends one block at a time, and waits for the acknowledgement. With TCP/IP you can send multiple blocks before waiting for the ack, and use big blocks. For a 300MB file, I achieved 47KB per second with a 16000 block size – so not very high.

With IND$FILE, pick the biggest block size you can. I think it supports a maximum size of 32767. I got 86 KB/second with a 32767 block size with DFT mode.

For a dataset packaged with TSO XMIT

Use the TSO command RECEIVE INDSN(…) to restore the data set.

Un PAX the file to recreate it

On the production system, I use went into Unix, and used the cd command to get to the destination directory.

pax -ppx -rf  "//'COLIN.ZOWE.PAX'"

Why can’t I connect my something to my laptop over Ethernet?

I was failing to connect a Wi-fi repeater to my laptop via Ethernet. It is a very simple device. It about the size if a plug, and says connect to 192.168.11.1. I did, and it didn’t connect.

Once I spotted the problem, it was obvious.

On Linux, I had to configure the wired connection so support this address. Under IPv4, I added

Routes
192.168.11.1  |  255.255.255.0 | 10.1.0.2

and it all worked.

Simple when you know how!

Tracing encrypted data to z/OS

I had blogged Collecting a wire-shark trace with TLS active for a browser where you could specify an environment variable export SSLKEYLOGFILE=$HOME/sslkeylog.log. OpenSSL would write the key to this file, and Wireshark could decrypt the traffic using this data.

Unfortunately this only worked with RSA keys. I could not get it to work with modern Elliptic Curve keys.

I’ve updated my zWireshark program to capture AT-TLS application data in clear text from the z/OS side. It uses an IBM provided API, and captures the traffic between AT-TLS and the application.

You need to set up security profiles, for example

permit  EZB.TRCSEC.*.*.ATTLS            - 
   CL(SERVAUTH) id(ADCDB) access(READ) 
permit  EZB.TRCCTL.S0W1.TCPIP.DATTRACE  - 
   CL(SERVAUTH) id(ADCDB) access(READ) 
permit EZB.TRCCTL.S0W1.TCPIP.OPEN  - 
   CL(SERVAUTH) id(ADCDB) access(READ) 
permit EZB.TRCCTL.*.*.* CL(SERVAUTH) id(ADCDB) access(READ) 
SETROPTS RACLIST(SERVAUTH) refresh

and change the AT-TLS configuration to include CtraceClearText On

For my web browser traffic it produced (printed with the ASCII switch)

Data trace.  Data length 345.  ATTLS Clear Text.                             
<GPMSERVE  11:01:57.258946 Src 10.1.1.2 Port 8803 Dst 10.1.0.2 Port 45168    
Warning: 199 RMF-DDS-Server SeverityCode(03) Data(0)                         
Content-Location: perform_20250102110157_20250102110157.xml                  
Cache-Control: max-age=30                                                    
Date: Thu, 02 Jan 2025 11:01:57 GMT                                          
Connection: close                                                            
Content-Length: 1468                                                         
Content-Type: application/xml                                                
X-UA-Compatible: IE=edge                                                     
X-Content-Type-Options: nosniff                                              
X-XSS-Protection: 1; mode=block

When the AT-TLS option CtraceClearText was Off, the output was

Data trace.  Data length 0.  PTH_Flag.Confidential.                           
<GPMSERVE  11:27:48.733616 Src 10.1.1.2 Port 8803 Dst 10.1.0.2 Port 52860

So no confidential data was displayed.

The JCL is

//COLINC5    JOB 1,MSGCLASS=H,COND=(4,LE) 
// SET LOADLIB=COLIN.ZWIRESHA.LOAD 
//RUN      EXEC PGM=TCPDATA,REGION=0M,PARMDD=MYPARMS 
//STEPLIB  DD DISP=SHR,DSN=&LOADLIB 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=* 
//SYSERR   DD SYSOUT=* 
//* MYPARMS needs a / to split LE parms from program parms 
//*         needs a blank before each parm, because trailing blanks removed 
//MYPARMS DD * 
/ 
 --IP   10.1.0.2 
 --WAIT 60 
 --DEBUG 0 
 --DISCARD 1 
 --PRINT A 
/*

Please let me know of any problems or suggestions

Easy AT-TLS configuration and reporting

This blog post discusses AT-TLS configuration, and my EasyAT-TLS code on GitHub to make it easy to configure AT-TLS and give an compact and useful configuration report.

A lot of documentation is written by experts for experts, from the developer’s viewpoint; rather than by experts for people who just want to get the job done. It feels that AT-TLS exposes to much of the structure of how data is held internally. I found it was very easy to get lost, and configuring it is hard. z/OSMF has a “configure AT-TLS” workflow – but it just creates a workflow of the steps. It did not make it easier to configure AT-TLS.

As a general philosophy, rather than the traditional approach of “here are all the keywords in alphabetical order”, I would provide information in different topics.

  • Keywords that everyone uses, and what you will need to get started,
  • More advanced keywords that most people might use,
  • Even more advanced keywords that only experts would use.

and then list the keywords alphabetically within topic. For an inexperienced user this makes it very clear what options they need to specify to get started. If you have a configuration tool (or online documentation) have a button which allows you to specify what your experience level is, and display or configure the information at the appropriate level.

The problem

What does an AT-TLS configuration file look like?

Part of the definition for one rule is

TTLSRule                      COLATTLJ 
{ 
  LocalPortRange              4000 
  Jobname                     COLCOMPI 
  Userid                      COLIN 
  Direction                   BOTH 
  RemoteAddr                  10.1.2.2/32 
  TTLSGroupActionRef          TNGA 
  TTLSEnvironmentActionRef    TNEA 
  TTLSConnectionActionRef     TNCA 
} 
TTLSGroupAction             TNGA 
{ 
  TTLSEnabled               ON 
} 
TTLSEnvironmentAction                 TNEA 
{ 
  HandshakeRole                       ServerWithClientAuth 
  TTLSKeyringParms 
  { 
    Keyring                   start1/TN3270 
  } 
  TTLSSignatureParmsRef       TNESigParms  
} 
TTLSSignatureParms                TNESigParms 
{ 
  CLientECurves Any 
  SignaturePairs     060305030403 

}......

There is a lot of structure, with values beginning with TTLS.

Without the structure the data looks like

policyRule : COLATTLJ
LocalAddr : All
RemoteAddr : '10.1.1.2/32'
LocalPortRange : 4000-4000
JobName : COLCOMPI
UserId : COLIN
Direction : Both
TTLSEnabled : On
Trace : 255
HandshakeRole : ServerWithClientAuth
Keyring : start1/TN3270
TLSv1.1 : Off
TLSv1.2 : On
TLSv1.3 : Off
HandshakeTimeout : 3
ClientECurves : Any
ServerCertificateLabel : NISTECCTEST
V3CipherSuites : [
   003D  TLS_RSA_WITH_AES_256_CBC_SHA256,
   C02C  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
 ]

Which I find much clearer.

Once you have configured AT-TLS, you can use the Unix command pasearch to report the configuration. This gives output like

policyRule:             AZFClientRule                                  
  Rule Type:            TTLS                                           
 ...            
  Time Periods:
   Day of Month Mask:
   First to Last:       1111111111111111111111111111111
   Last to First:       1111111111111111111111111111111
   Month of Yr Mask:    111111111111    
   Day of Week Mask:    1111111  (Sunday - Saturday)
   Start Date Time:     None            
   End Date Time:       None            
 ...
  TTLS Condition Summary:                 NegativeIndicator: Off
   Local Address:
    FromAddr:           All                                           
    ToAddr:             All                                           
   Remote Address:
    FromAddr:           0.0.26.137                                    
    ToAddr:             0.0.26.137                                    
   LocalPortFrom:       0                 LocalPortTo:       0     
   RemotePortFrom:      0                 RemotePortTo:      0     
   JobName:             AZF*              UserId:                            
   ServiceDirection:    Outbound 
...

The output for one rule is over 180 lines long, and gives the configuration, including any defaults; not what is actually used. (Some fields can be configured in more than one place, so you need to know which is actually used). Compare this with my tool’s output above.

Easy AT-TLS on GitHub

Displaying a concise view of the data.

On GitHub is code which removes all of the “structure” from the pasearch report to leave just the non default data. You get, for one rule

policyRule : COLATTLJ
LocalAddr : All
RemoteAddr : '10.1.1.2/32'
LocalPortRange : 4000-4000
JobName : COLCOMPI
UserId : COLIN
Direction : Both
TTLSEnabled : On
Trace : 255
HandshakeRole : ServerWithClientAuth
Keyring : start1/TN3270
TLSv1.1 : Off
TLSv1.2 : On
TLSv1.3 : Off
HandshakeTimeout : 3
ClientECurves : Any
ServerCertificateLabel : NISTECCTEST
V3CipherSuites : [
   003D  TLS_RSA_WITH_AES_256_CBC_SHA256,
   C02C  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
 ]

formatted as a YAML file, which I find much easier to use!

The output of the pasearch file was 2296 lines, the output file from my Python program was 184 lines!

Generating AT-TLS definitions

I found it hard to generate AT-TLS definitions because you need to know the AT-TLS structure; or have an existing entry to copy. For example a keyring is defined in a TTLSKeyringParms definition in TTLSEnvironmentAction statement which is pointed to by a high level TTLSRule. See the partial example above. You should not need to know this information.

As an end user I just want to define a keyring, and let the computer put it into the “internal format”.

With AT-TLS, you can group common information into a unit, and refer to that. I found I got lost trying to work out what I was using, and what else used it. Making one small change is complex as you had to do lots of copying to generate new groups, and then you have these groups lying around which may not be used again.

Generating AT-TLS definitions the easy way

The code to extract the key information from a pasearch output produces a YAML file. The genATTLS code takes a YAML file and generates the AT-TLS input file with all of the structure that AT-TLS requires. The output file can then be used by the Policy Agent on z/OS.

I’ve written the code from an end user perspective. For example I have a YAML file which defines two rules.

---
policyRule : Rule1
Priority : 255
LocalAddr : All
RemoteAddr : All
LocalPortRange : 6794-6794
Direction : Inbound
TTLSEnabled : On
Trace : 255
HandshakeRole : ServerWithClientAuth
Keyring : start1/TN3270
TLSv1.1 : Off
TLSv1.2 : On
TLSv1.3 : Off
HandshakeTimeout : 120
ServerCertificateLabel : RSA2048
---
policyRule : Rule2
Priority : 255
LocalAddr : All
RemoteAddr : All
LocalPortRange : 6793-6793
Direction : Inbound
TTLSEnabled : On
Trace : 255
HandshakeRole : Server
Keyring : start1/TN3270
TLSv1.1 : Off
TLSv1.2 : On
TLSv1.3 : Off
CertificateLabel : RSA2048
ServerCertificateLabel : RSA2048

This can be simplified by using “BasedOn” to refer to other sections, such as the “common” rule

---
# This first section is common to the others
policyRule : common
TLSv1.1 : Off
TLSv1.2 : On
TLSv1.3 : Off
Keyring : start1/TN3270
Priority : 255
LocalAddr : All
RemoteAddr : All
Direction : Inbound
TTLSEnabled : On
Trace : 255
CertificateLabel : RSA2048
ServerCertificateLabel : RSA2048
---
policyRule : Rule1
BasedOn : common
LocalPortRange : 6794-6794
HandshakeRole : ServerWithClientAuth
HandshakeTimeout : 120
---
policyRule : Rule2
BasedOn : common
LocalPortRange : 6793-6793
HandshakeRole : Server
---

The Python script generates the definitions from this file, 110 lines of output from a 28 line input file!

It is easy to extend. You just specify the overrides

policyRule : Rule2A
BasedOn : Rule2
TLSv1.3 : On
Priority: 300

The priority:300 says this definition should be used over Rule2, because Rule2 only has priority 255.

Idiot’s guide to TCPIP connectivity

I had a working TCPIP network, and made a few “improvements”. Unfortunately these improvements sometimes stopped the connectivity between systems, and I had a frustrating time understanding the problems and fixing them. The idiot in the blog post is me, for next time when I need to connect boxes together.

In concept TCPIP connectivity is simple – it is, but there are some subtle, non obvious things you need to be aware of.

As I was writing this post I found I did not know really how IPV4 worked, because it used “the wrong” IP address but still worked.

I found many ways of failing to connect to TCPIP, and some complex ways of getting it to work – I just wanted a simple way of being able to ping z/OS from my laptop. It is complicated by some definitions need to be done in order, and doing things in a different order sometimes worked, sometimes did not.

Basic TCPIP concepts that every one should know

  • The term socket is used by applications to communicate with TCP/IP, not where you connect a network/phone cable.
  • Think of a connection between two boxes. I have a yellow Ethernet cable between them. There are several terms for the where the cable is plugged in. A common term is the interface.
  • IP addresses
    • Each end of the connection has one or more IP addresses.  I think of it as having plastic labels tied to the end of the cable.
    • IPV6 addresses beginning with fe… and ff… are used by (internal use) advanced technology and can be ignored. You can use them, but the addresses may change every time the connection is started, which makes it hard to automate using them.
    • The system may generate some IPV6 addresses, but you can define your own. The system generated an address like 2a00:9999:8888:7777:894e:9876:781:32f1. Sometimes parts of these (the right hand part) are randomised (to make it harder for people to observer traffic patterns and so hack your system).
    • I use addresses like 2001:db8::f which are shorter to type.
    • On z/OS an IPV4 interface can have only one IP address. An IPV6 interface can have multiple addresses see ADDADDR. On z/OS an interface can be IPV4 or IPV6 but not both.
    • On Linux, an interface can have multiple IPV4, and multiple IPV6 addresses (but only the first IPV4 may be visible to applications)
    • For IPV6, TCP/IP can generate its own IPV6 addresses for internal processing, such as routing.
  • To get data from this machine to that machine over the yellow Ethernet cable, you have a route definition like “for this range of remote addresses use the yellow Ethernet cable, which has the address xxxx at the far end.
  • If you use TCP/IP to send a request, you usually want a response to come back. As well as defining a route to get to the remote end, you need a route defined to get from the remote machine back to the local machine. A ping request can fail because
    • The local end does not have a valid route to the remote end. The packet could be sent to the wrong place(down the wrong cable), or just discarded.
    • An intermediate box does not have a route to the remote end.
    • The remote end receives the request but does not have a route definition to send the response back to the requester.
    • An intermediate box does not have a route to the local end.
    • A firewall says no.
    • You can use the traceroute command to find the path taken to the remote end. This will tell you the path it took to get there. It does not tell you the route back. For this you need to issue the traceroute command on the remote end, and perhaps on intermediate boxes.
  • You define a route from this box using the yellow cable with label xxxx on it. The remote end of the cable has IP address….
  • You need at least two route statements
    • to get the data from the local system to the remote system,
    • the remote system needs a route statement to get to the local system.
  • You can find these address using
    • the Linux command ip -6 addr or ip -4 addr for TCP IPV6 and IPV4 respectively.
    • the z/OS command TSO NETSTAT HOME
  • Subnet: an IP V6 address has 32 hex digits. These are broken up into groups of 16 eg 2001:0DB8… This can be written as 2001:db8… The subnet specified which bits are significant when routing packets to the router. With z/OS usually the top 64 bits are used. This is written as …./64.
  • An address 2001:db8:9::1/64 is in a different subnet to address 2001:db8:8::1/64.
  • Address 2001:db8:8:1::2/64 is in the same subnet as 2001:db8:8:1::3 because only the top 64 bits count towards the subnet (2001:db8:8:1).
  • A gateway is a network point that acts as an entrance to another network. On the Internet, a node or stopping point can be either a gateway node or a host (end-point) node. Both the computers of Internet users and the computers that serve pages to users are host nodes. A gateway can have one protocol in, and output the data in a different protocol. For example I have broadband coming to my house. The gateway router converts this to TCP/IP, and converts it to wireless.

Things that you may not know

  • My end of a connection has two IP addresses defined. If I ping a remote site it uses the first IP address in its list, the remote site sees a packet of data from the first IP address in the list. You may have configured a route at the remote system to get back to your local system, but if you define your local addresses in a different order, a different IP address will be sent – and the remote end may not have a route for it.
  • If the interface at the next machine has two IP addresses 10.1.0.3 and 7.168.1.2 , I have to use the first IP address in the list defining a route sudo ip -4 route add 7.168.1.74 via 10.1.0.3 dev enp0s31f6. If I delete the first address, then I need to use the 7.168.1.2