Category: PAGENT

Sending an email from z/OS

This started off as a question to me about TLS certificates and ended up with a Python solution.

The short version of this blog post is I could send an email to Gmail using Python – but could not use the IBM CSSMTP solution because it does not support authentication. I was also sent a link to Sending Email from z/OS using Java.

As far as I can tell, SMTP authentication does not use client certificates. If you want to use userid and password, make sure the session is encrypted using TLS.

What types of email are there?

Through JES spool

On my system with id S0W1, I can use TSO XMIT S0W1.IBMUSER and send an email.

The recipient (IBMUSER) can use TSO receive to read the email.

You can use XMIT OVERTHER.MYID to send the email over NJE to other system, such as z/OS, z/VM and z/VSE. You say (on z/OS) xmit node.userid da(‘…’). or xmit node.userid . The JES2 configuration has an IP address and port to send it to a remote spool.

JES2 destinations

You can configure destinations on JES2, and have program (“a writer”) process the node name (destination) data.

You can display the defined destinations using the JES2 command

$DDESTID

Example JCL to write to a destination,

//COPY EXEC PGM=IEBGENER 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD DUMMY 
//SYSUT2 DD SYSOUT=(A,MYDEST),SPIN=UNALLOC 
//SYSUT1 DD * 
line1
/*
//

This writes the data from SYSUT1 to SYOUT2 which goes to nodename MYDEST.

You configure CSSMP and specify ExtWrtName MYDEST, for CSSMTP to read from.

SMTP

There is Simple Mail Transport Protocol. A stream of data is provided such as

MAIL FROM:<ME@MYSYSTEM.COM> 
RCPT TO:<MYBOSS@MYSYSTEM.COMU> 
RCPT TO:<YOU@yoursystem.com> 
DATA 
Blah blah blah. 
QUIT

which is sent to the mail server over TCPIP (usually port 25 or 587).

SMTP is a popular protocol. It can support encrypted session encryption. The protocol can support different levels of authorisation.

z/OS provides CSSMTP which reads from the spool and sends the data over TCPIP to a server. However CSSMTP does not seems to support the passing of authentication information to the server – and as most mail servers want you to authenticate before sending emails – CSSMTP cannot be used for these. I know this from the documentation because the AUTH command is not listed in the list of supported SMTP commands, and AUTH does not work!

Using Python to send an email.

I used Python to send an email to my Gmail account. The script I used is 

#!/usr/bin/python 
import smtplib 
import ssl 
import sys 
from email.mime.text import MIMEText 
host = "173.194.76.108" 
sender = 'colinpaicetest@gmail.com' 
receivers = ['colinpaicemq@gmail.com','colinpaicetest@gmail.com'] 
msg = MIMEText('This is test mail2') 
msg['Subject'] = 'Test mail3' 
msg['From'] = 'colinpaice3@gmail.com' 
msg['To'] ='colinpaicemq@gmail.com , colinpaicetest@gmail.com' 
with smtplib.SMTP(host , 587) as server: 
      # if you need to validate the certificate sent to client.. you need a context
      context=ssl.create_default_context(
                 purpose=ssl.Purpose.CLIENT_AUTH,cafile="pem.pem") 
      server.starttls(context=context) 
      # server.starttls() # for servers that do not send a certificate 
      print(server.esmtp_features) # print info about the connection such as auth types
      server.login('colinpaice3@gmail.com', 'abcpasswordxyz') 
      server.sendmail(sender, receivers, msg.as_string()) 
      print("Successfully sent email")
  • context=ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH,cafile=”pem.pem”) is used to set up the CA cert for verifying any certificate sent from the server. GMAIL does not send a certificate, so the CA is not needed, and you can use server.starttls()
  • server.starttls(context=context) creates the TLS session. This is handled by PAGENT.
  • server.login(‘colinpaice3@gmail.com’, ‘ZYC123….’) does the authentication
  • server.sendmail(sender, receivers, msg.as_string()) sends the message.

The id in the sender must match the id in the login.

This does not use PAGENT or AT-TLS.

In the TLS handshake 57 cipher specs were sent to the server, and cipher spec TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xC02B) was used.

Setting up CSSMTP

Basic configuration

Even though I could not use CSSMTP because I could not authenticate, I learned a lot on my doomed journey.

In the CSSMTP configuration I had 

ExtWrtName    MYDEST             # the external writer name 
TargetServer 
{ 
   TargetIp aaa.bbb.ccc.ddd      # This IP address 
  # TargetName  
   ConnectPort   25              # port to connect to target server 
   Secure        Yes             # Transport Layer Security (TLS) 
} 
...

Each CSSMTP instance reads from one nodeName and sends to one server. If you want to support more nodeNames you need more CSSMTP instances.

Certificates

As part of the TLS handshake, some servers will send down a certificate (to authenticate the server).

If your TLS provider, such as AT-TLS, uses keyrings, you need to import certificate into RACF, and connect the CA certificate to the keyring. Your keyring needs the CA for this certificate. You can get the servers certificate from the network flow, or from an openssl command.

If you are not using keyrings, for example Python uses openssl, you need the certificate in a file in Unix Services.

z/OS PAGENT error messages and system SSL codes

Message

SYSERR :001: plfm_kernel_init: socket(INET, DGRAM, 0), failed, errno=EDC5112I Resource temporarily unavailable., errno2=74610296

OBJERR :001: init_PEP_and_kernel: Kernel initialization failed for image ‘TCPIP2’,

RACF profile EZB.INITSTACK.*.TCPIP2 CLASS(SERVAUTH) was missing.

//S1 EXEC PGM=IKJEFT01,REGION=0M
//STEPLIB DD DISP=SHR,DSN=SYS1.MIGLIB
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RDEFINE SERVAUTH EZB.INITSTACK..TCPIP2 PERMIT EZB.INITSTACK..TCPIP2 CLASS(SERVAUTH)
ID(START1) ACCESS(READ)
SETROPTS RACLIST(SERVAUTH) REFRESH
/*

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file

In my image file I had a

TTLSRule
{
…

}

which is not allowed. They have to be in the TTLSConfig… file.

The error message was

WARNING:005: ..pinit_fetch_policy_profile: Ignoring non-image config file ‘TTLSRule’ statement

402 No SSL cipher specifications.

In some situations the cipher specs default.

You have to specify them for TLS 1.3

RC: Code 515 Initial handshake

Code 515 Key share list is not valid

I got this when trying to use TLS 1.3.

You need

TTLSSignatureParms
{
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0030

}

or ServerKeyShareGroup

517 No matches between elliptic curve and key share lists

Need something like

TTLSSignatureParms CPESigParms
{
CLientECurves 0023
CLientECurves 0024
CLientECurves 0025
CLientECurves 0029
ClientKeyShareGroups 0023
ClientKeyShareGroups 0024
ClientKeyShareGroups 0025
ClientKeyShareGroups 0029
}

519 Required ciphers have not been specified

I had TLS 1.3 specified, but no TLS 1.3 cipher specs.

Secure X3270 to z/OS on ADCD

I was asked if it was possible to connect to an ADCD system using X3270 and certificates, after quite a bit of work, the answer is yes – but it may not be easy.

The process is documented in the September version of IBM ISV zPDT Guide and Reference – which works, but the SecurePort is being removed in z/OS 2.5!

How to connect using x3270.

Using a “dialed in” local terminal into an ADCD system.

You can connect to z/OS using 3270 like terminals via a 3274 type controller and use applications like TSO. This is not a TCPIP connection, so you cannot use TLS to protect the session. These terminals are created using a Linux command like

x3270 -model 5 colin@localhost:3270 &

Where the :3270 matches the zPDT configuration statement 3270port 3270.

The colin@, known as the LUName, is put in the title of the window (so I could hot key to it, by setting key Ctrl+3 to wmctrl -a colin@).

Using X3270 and TCPIP over an insecure connection

You can connect over TCPIP into the TN3270 server using a command like

x3270 -model 5 10.1.1.2:23

This uses port 23, the standard telnet port.

You can specify an LU name – but you need to look at your TN3270 configuration to see what values have been configured. If you do not know what you are doing … it is safer to omit it.

SECUREPORT will be removed in z/OS 2.5

This was the “secure” way of providing a secure connection, before AT-TLS was developed. It is documented in Appendix E of IBM ISV zPDT Guide and Reference

The z/OS documentation says SECUREPORT will be removed

Keyword no longer supported for the TN3270E Telnet server: Removal of native
TLS/SSL support from TN3270E Telnet Server, FTP Server, and DCAS.

How to configure and use TN3270

Ive written about it here.

My TN3270 configuration is member TNPO2023 is 

TelnetParms         ; ATTLS defined secure port 
 TTLSPort 2023      ; 
 DEBUG CONN DETAIL CTRACE 
 DEBUG CONN trace  CTRACE 
 ConnType Secure    ; Client chooses secure or nonsecure connection. 
EndTelnetParms 
                                                                           
BeginVTAM 
 Port 2023 
 DEFAULTLUS 
   TLS00001..TLS00030 
 ENDDEFAULTLUS 
 ALLOWAPPL TSO* DISCONNECTABLE ; Allow all users access to TSO 
             ; applications. 
             ; TSO is multiple applications all beginning with TSO, 
             ; so use the * to get them all.  
 ALLOWAPPL *      ; Allow all applications that have not been 
                  ; previously specified to be accessed.             
 USSTCP USSN 
EndVTAM

Using AT-TLS

I eventually got a secure connection using AT-TLS. Many parameters need to be correct for the TLS Handshake to work. Some combinations should work – but did not.

I’ve blogged about setting up and using AT-TLS.

The short list of requirements:

  • The z/OS certificate needs CN(‘10.1.1.2’) where 10.1.1.2 is what is used by the X3270 client. It checks the value in the certificate with the value used in the connection. ALTNAME(IP(10.1.1.2)) did not work for me. If there is a mismatch you get x3270 message “hostname mismatch(62). When I used a host name of zos as in “x3270…. zos” – and the z/OS certificate has CN(‘zos’)…. it also worked.
  • You can use RSA, and NISTECC, certificates on z/OS.
    • TLS 1.2 or TLS 1.3 , RSA size 2048 or larger
    • TLS 1.2 or TLS 1.3, NISTECC size 521
    • BPECC not supported.
  • The CA was exported, sent down to Linux and used in the x3270 command (and also the openssl s_client command). For example -cafile ./doczosca.pem .
  • x3270 does not use a certificate on Linux – the doc implies it does, but nothing gets sent to the server.
  • In the AT-TLS configuration below,
    • the three cipher spec represent TLS 1.3, NISTECC and RSA.
    • ApplicationControlled On is required
    • HandshakeRole Server is required

Below is the AT-TLS configuration for my telnet (TN…) configuration. It should work with the keyring created in Appendix E. Secure x3270 connection in IBM ISV zPDT Guide and Reference September 2022 (SG24-8205-06). See JCL. The file below is on GIT.

TTLSRule                      TN 
{ 
  LocalPortRange              2023 
  Jobname                     TN3270 
  Direction                   INBOUND 
  TTLSConnectionActionRef     TNCA   
  TTLSGroupActionRef          TNGA 
  TTLSEnvironmentActionRef    TNEA 

} 
TTLSConnectionAction              TNCA 
{ 
  TTLSCipherParmsRef              TLS13TLS12 
  TTLSSignatureParmsRef           TNESigParms 
  TTLSConnectionAdvancedParmsRef  TNCOonAdvParms 
  CtraceClearText                 Off 
  Trace                           50 
} 
TTLSConnectionAdvancedParms       TNCOonAdvParms 
{ 
 #ServerCertificateLabel  NISTECC521 
 #ServerCertificateLabel  RSA2048 
 #ccp this was added 
  ApplicationControlled         On 
  SSLv3          OFF 
  TLSv1          OFF 
  TLSv1.1        OFF 
  TLSv1.2        ON 
  TLSv1.3        OFF 
  SecondaryMap   OFF 
  HandshakeTimeout 3 
} 

TTLSGroupAction      TNGA 
{ 
  TTLSEnabled        ON 
  trace              50 
} 
TTLSKeyringParms    TNKEYRING 
{ 
  Keyring           START1/TN3270 
} 

TTLSEnvironmentAction      TNEA 
{ 
  HandshakeRole            Server 
  TTLSKeyringParms 
  { 
#    Keyring                START1/TN3270 
# Use the keyring from the ZPDT documentation chapter E
     Keyring                START1/TN3270TLSring 
  } 
  TTLSSignatureParmsRef    TNESigParms 
} 
TTLSSignatureParms         TNESigParms 
{ 
   CLientECurves Any 
} 
TTLSCipherParms             TLS13TLS12 
{ 
#TLS 1.3 
 V3CipherSuites      TLS_CHACHA20_POLY1305_SHA256 
#V3CipherSuites      TLS_AES_256_GCM_SHA384 
#V3CipherSuites      TLS_AES_128_GCM_SHA256 
#TLS 1.2 
# NSTECC 
 V3CipherSuites      TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 
#RSA 
 V3CipherSuites      TLS_RSA_WITH_AES_256_CBC_SHA256 
}

I did many system SSL traces to get it working.

Some lessons learned

  • Although you can specify a certificate and private key on the Linux x3270 command, it does not use them. If you specify them, the certificate is not passed up to to z/OS.
  • The AT-TLS configuration needs TTLSEnvironmentAction -> HandshakeRole SERVER. (Despite what the documentation says). If you specify HandshakeRole ServerWithClientAuth, then z/OS expects the client to send a certificate as part of the handshake – and as x3270 does not sent a certificate – the handshake fails.
  • x3270 checks the host name. In the z/OS certificate the DN needs a CN matching the connection name. I connected to 10.1.1.2 – I needed SUBJECTSDN(CN(‘10.1.1.2‘) … . Without this you get hostname mismatch (62).
    • You should be able to specify RACDCERT …. ALTNAME(IP(10.1.1.2)), as an alternate name, but x3270 does not recognise it.
  • x3270 supports TLS 1.3 – if you have TLS 1.3 cipher specs configured in AT-TLS, and specify TLSv1.3 ON
  • X3270 supports TLS 1.2 – if you have TLS 1.2 cipher specs configured in AT-TLS, and specify TLSv1.2 ON and TLSv1.3 OFF.
  • x3270 sends up a list of cipher suites it will accept. The first three are TLS 1.3 (TLS_AES_256_GCM_SHA384 (0x1302) ,TLS_CHACHA20_POLY1305_SHA256 (0x1303),TLS_AES_128_GCM_SHA256 (0x1301).
  • When configuring TN3270 use V3CipherSuites not V3CipherSuites4char. V3CipherSuites takes 2 character strings and long TLS_… strings. V3CipherSuites4char only take 4 character strings, and not TLS_… strings.
  • If you change the PAGENT configuration you need F PAGNET,UPDATE to pick up the changes.
  • If you make a change to the PAGENT configuration, it does not always report problems. You should use Unix command commands like pagent -t -f TN 1>a … oedit a to display the TTLS configuration for rule TN, and edit the file to check.
  • If you change a keyring, you need F CPAGENT,REFRESH to pick up the change.
  • When you start PAGENT use the -l /var/log/pagent.log -d1 options (or similar). Review the /var/log/pagent.log file and check for “WARNING” and “ERR”
  • The PAGENT configuration can have the same parameter specified at the environment level and at the connection level. The pasearch -t -f TN command displays both sets of data … you have to be careful you are checking the right set of data.

Is the port active?

Check the port is being listened to

tso netstat conn ( port 2023

gave

EZZ2585I User Id  Conn     Local Socket           Foreign Socket         State
EZZ2586I -------  ----     ------------           --------------         -----
EZZ2587I TN3270   0000000B 0.0.0.0..2023          0.0.0.0..0             Listen

Showing the port is in Listen state from jobname TN3270 (which has userid TCPIP!)

Update to show it not working!

I was asked a question about AT-TLS, and it not working. I thought it would be useful to include some Wireshark output to help people to see what is happening.

I used x3270 10.1.1.2@3023 and it reported “SSL: Host certificate verification failed: self signed certificate in certificate chain”.

In the examples below there are three records of interest

  • Client Hello – this is the first step in the handshake. If you do not have this – your client is not sending a request
  • Server Hello – the back-end has processed the Client-Hello – and, in the example below, sends down a certificate
  • Alert (from the client). The client cannot validate the certificate because it does not have the CA for the certificate.

The Wireshark trace was

The important line is the “Client Hello” in blue. This is the start of the TLS handshake. In my case the length is 512 bytes. I would expect any TLS handshake to be a couple of hundred bytes long. If you do not get a “Client Hello”, your client is not set up for TLS/SSL.

The “Server Hello” line is where the server is responding (with a certificate).

There is an error “Alert (Level: Fatal Description: Unknown CA”

Using AT-TLS and PAGENT on z/OS (with ADCD)

I wrote this post based on getting X3270 to work with TLS to z/OS. It covers some of the lessons I learned in doing so.

Policy Server

The Communications Server’s policy server allows you to define policies for: (the list below is mainly taken from the IBM documentation)

  • QoS: The overall Quality of Service provided to applications or users, in terms of elements such as throughput and delay. It might be critical to provide Business Application traffic better service during peak hours than that of FTP or web traffic.
  • IDS: Intrusion Detection Services provides the following support: scan detection and reporting, attack detection, reporting, and prevention, traffic regulation for TCP connections and UDP receive queues. You can use IDS policies to specify event conditions and the actions to take for particular events
  • IPSec: IP Security policy can be used for the following protection:
    • Protect a secure host on an internal network from unwanted network traffic
    • Provide protection for traffic between partner companies over connected networks
    • Allow secure sending of data over the Internet by providing IPSec virtual private network (VPN) support
  • Routing: Policy conditions consist of a variety of selection criteria that act as filters for policy-based routing (Routing) rules. Traffic can be filtered based on source addresses, destination addresses, source port range, destination port range, protocol, job name, security zone, and security label.
  • AT-TLS: (Application Transparent-Transport Layer Security) Provides TLS support at the TCPIP level (below the application). Using AT-TLS means applications do not need to be changed to support TLS.

The PAGENT (Policy Agent) started task runs the policy server. It can use local configuration files, or data in LDAP.

Using AT-TLS

It was a struggle to get a secure connection using AT-TLS and x3270. Many parameters need to be correct for the TLS Handshake to work. Some combinations should work – but did not, the documentation is not 100% correct. When I knew what the correct parameters were, the configuration was pretty easy! The configuration technique looks as if it was designed for a baby system. It is not easy to follow good housekeeping, and change management practices, which made the overall experience harder than I expected.

X3270 requirements

x3270 is a 3270 emulator. It can use TLS to provide a secure connection to z/OS. Although you can specify a certificate for X3270 to use – x3270 does not used the client certificate.

At a high level, the TLS flow between x3270 and AT-TLS is:

  • X3270 client sends a “client-hello” request to the server. This includes what levels of TLS are supported (TLS 1.2, TLS 1.3), and the cipher specs it supports (what encryption type,what checksum algorithm etc)
  • The server (Pagent) matches the information sent in the “Client-hello” with its configuration. For example, select a certificate which matches the encryption type, and finds a common cipher spec.
  • If there is an acceptable certificate, and cipher spec, the selected certificate and selected cipher spec are sent down to the client (“server hello”).
  • The x3270 client checks that the certificate is acceptable – the CN in the certificate matches the address of the server. (CN(10.1.1.2) matches the IP address 10.1.1.2).
  • If client authentication is requested, then the client sends its certificate to the server. This is not supported by X3270, and the server has to be configured with HandshakeRole Server.

X3270 initially sends that it can support TLS 1.2 and TLS 1.3, and a variety of cipher specs.

TN3270 requirements

Not all certificates worked. I could only get certificates to work with

  • RSA key size >= 2048
  • NISTECC leysize >= 521

To AT-TLS, this is just a certificate name in a keyring.

In the AT-TLS configuration

  • ApplicationControlled On is required. This allows TN3270 to interact with AT-TLS, for example display the cipher spec being used.
  • HandshakeRole Server is required.
  • It needs a keyring, and access to the private part of the certificate. For example CONTROL access to the RACF profile userid.ring.LST int the RDATALIB class.
  • Certificates
    • If you do not specify a certificate – it takes the keyring default.
    • You can specify a(one) certificatelabel.
    • You can specify multiple ServerCertificateLabel statements. With this you can support different certificate types, and support migration to newer certificates.

AT-TLS configuration – in general

The Communications Server as a group of products, do not have a consistent way of configuring the individual components. For example some use # as a comment delimiter, other products use ; . Configuring the Policy Server is not difficult – just different, and does not behave as other z/OS components do, or as I expected.

In my PAGENT JCL is

//STDENV -> USER.Z24C.TCPPARMS(PAGENTEN).

This file has

PAGENT_CONFIG_FILE=//'USER.Z24C.TCPPARMS(PAGENTCF)' 
LIBPATH=/usr/lib

The PAGENTCF member is like 

CommonTTLSConfig //'USER.Z24C.TCPPARMS(PAGENTCO)' 
tcpImage TCPIP   //'USER.Z24C.TCPPARMS(PAGENTT)' 
tcpImage TCPIP1  //'USER.Z24C.TCPPARMS(PAGENT1)'

File USER.Z24C.TCPPARMS(PAGENTT) has the configuration for one TCPIP image. 

TTLSConfig //'USER.Z24C.TCPPARMS(PAGENTTN)' FLUSH PURGE 
TTLSRULE ... #and other inline definitions 

QOSConfig  //'USER.Z24C.TCPPARMS(PAGENTQ1)' FLUSH PURGE 
policyAction  ...  # and other inline QOS definitions 

# and similarly for IDS, IPSec,Routing etc

You are allowed one TTLSConfig statement per TCPIP file. If you have multiple, only the last one will be used. You can have multiple TTLSRULE statements.

How the configuration works

Within a file, if you have a set of definitions with the same name, the last one will be used. For example 

TTLSEnvironmentAction                 TNEA 
{ 
  TTLSKeyringParms 
  { 
    Keyring                   START1/TN3270 
  } 
  Trace 17
} 
TTLSEnvironmentAction                 TNEA 
{ 
  TTLSKeyringParms 
  { 
    Keyring                   START2/TN3270 
  }   
}

The keyring will be START2/TN3270 and Trace is not specified.

If you use Unix command pasearch -t , it will show keyring:START2/TN3270 and no trace statement in that section. Definitions with the same name are replaced, not merged.

Having just one TTLSCONFIG file makes it harder to manage. I would like to be able to have a configuration file for each port, or have all of the TLSRULEs in one file, and TTLSCipherParms in another file, and so on. It would make it easier to manager, and perform change management on the files.

The configurations from tcpImage TCPIP2 and tcpImage TCPIP2 are isolated from each other. If you want to use common definitions between tcpImages, put them in CommonTTLSConfig file.

The commonTTLSConfig statements are processed before the tcpImage statements, so a definition set in the tcpImage file will take precedence over a common definition.

The commonTTLSConfig file seems to need to be self consistent. I added

TTLSEnvironmentAction                 TNEA 
{ 
  TTLSSignatureParmsRef       TNESigParms 
}

Without the TNESigParms definition. I got strange configuration error messages until I included the TNESigParms {..} in the file.

Using smaller units.

I remember some advice I was given… try to get all of your changes visible on one screen. Use subroutines or other ways of dividing up the code.

You can have

TTLSCipherParms 
{
   TTLSCipherParms 
   { 
      V3CipherSuites      TLS_CHACHA20_POLY1305_SHA256 
   } 
}

But the list of cipher suites could be long.

You can have a …REFerence o a set of definitions.

TTLSConnectionAction   TNCA 
{ 
  TTLSCipherParmsRef   TLS13TLS12 
} 
TTLSCipherParms        TLS13TLS12 
{ 
  #TLS 1.3 
   V3CipherSuites      TLS_CHACHA20_POLY1305_SHA256 
   #V3CipherSuites      TLS_AES_256_GCM_SHA384 
   #V3CipherSuites      TLS_AES_128_GCM_SHA256 
 #TLS 1.2 
   # NSTECC 
   V3CipherSuites   TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
   #RSA 
   V3CipherSuites      TLS_RSA_WITH_AES_256_CBC_SHA256 
}

The allows you to create a “subroutine” out of the cipherspecs. It also allows you to use the same definitions in multiple places.

Within a TCPIPIMAGE configuration

The same parameter can be specified in more than one configuration statement. The documentation says:

For parameters that can be specified in multiple action types, the value used by a connection is
determined by the following hierarchical rule set, they first found is used.

  1. If the parameter is specified in the TTLSConnectionAction statement that is the value used.
  2. If the parameter is specified in the TTLSEnvironmentAction statement that is the value used.
  3. If the parameter is specified in the TTLSGroupAction statement that is the value used.
  4. If a default value is defined, that is the value used.
  5. No value is used by AT-TLS and no parameter is explicitly passed to System SSL.

So if you have

TTLSEnvironmentAction  TNEA 
{ 
  TTLSCipherParmsRef     TLS13 
} 
TTLSConnectionAction   TNCA 
{ 
  TTLSCipherParmsRef    TLS13TLS12 
}

The the CipherParms TLS13TLS12 will be used, because Connection is used before Environment.

If you use pasearch -t … it will display the configuration with sections like 

TTLS Action:                  TNGA 
    Scope:                      Group 

TTLS Action:                  TNEA 
  Scope:                      Environment 

  TTLSCipherParms: 
    v3CipherSuites: 
      1303  TLS_CHACHA20_POLY1305_SHA256 
 ... 
TTLS Action:                  TNCA 
    Scope:                      Connection 
    TTLSCipherParms: 
     v3CipherSuites: 
      1303  TLS_CHACHA20_POLY1305_SHA256 
      C02C  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 
      003D  TLS_RSA_WITH_AES_256_CBC_SHA256

Because of the order, Scope:Connection takes precedence over Scope:Environment, the three cipher specs will be used.

When configuring AT-TLS I found it easiest if I used the Unix commands pasearch -t -f TN 1>a; oedit a and specify the rule name I was interested in (TN), then search for the last instance of the attribute of interest

PDSE or Unix file?

If you configuration is in Unix files, then you can get AT-TLS to reread this on regular basis – typically half an hour. This is useful of you want changes to be picked up automatically, but not if you want strict change control management.

AT-TLS configuration for TN3270

Below is the AT-TLS configuration for my telnet (PAGENTTN…) configuration.

TTLSRule                      TN 
{ 
  LocalPortRange              2023 
  Jobname                     TN3270 
  Direction                   INBOUND 
  TTLSConnectionActionRef     TNCA   
  TTLSGroupActionRef          TNGA 
  TTLSEnvironmentActionRef    TNEA 

} 
TTLSConnectionAction              TNCA 
{ 
  TTLSCipherParmsRef              TLS13TLS12 
  TTLSSignatureParmsRef           TNESigParms 
  TTLSConnectionAdvancedParmsRef  TNCOonAdvParms 
  CtraceClearText                 Off 
  Trace                           50 
} 
TTLSConnectionAdvancedParms       TNCOonAdvParms 
{ 
 ServerCertificateLabel  NISTECC521 
 ServerCertificateLabel  RSA2048 
#ccp this was added 
  ApplicationControlled         On 
  SSLv3          OFF 
  TLSv1          OFF 
  TLSv1.1        OFF 
  TLSv1.2        ON 
  TLSv1.3        OFF 
  SecondaryMap   OFF 
  HandshakeTimeout 3 
} 

TTLSGroupAction      TNGA 
{ 
  TTLSEnabled        ON 
  trace              50 
} 
TTLSKeyringParms    TNKEYRING 
{ 
  Keyring           START1/TN3270 
} 

TTLSEnvironmentAction      TNEA 
{ 
  HandshakeRole            Server 
  TTLSKeyringParms 
  { 
    Keyring                START1/TN3270 
  } 
  TTLSSignatureParmsRef    TNESigParms 
} 
TTLSSignatureParms         TNESigParms 
{ 
   CLientECurves Any 
} 
TTLSCipherParms             TLS13TLS12 
{ 
#TLS 1.3 
 V3CipherSuites      TLS_CHACHA20_POLY1305_SHA256 
#V3CipherSuites      TLS_AES_256_GCM_SHA384 
#V3CipherSuites      TLS_AES_128_GCM_SHA256 
#TLS 1.2 
# NSTECC 
 V3CipherSuites      TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 
#RSA 
 V3CipherSuites      TLS_RSA_WITH_AES_256_CBC_SHA256 
}

I did many system SSL traces to get it working.

Using certificates

  • If you do not specify a certificate label in the configuration, AT-TLS takes the default from the keyring.
  • You can specify a certificate label using CertificateLabel…
  • If you want to specify more than one Certificate label – for example you are migrating from one certificate to another, or you want to support an RSA certificate and an Elliptic Curve certificate then specify one or more ServerCertificateLabel … statements.

Some lessons learned

  • When configuring TN3270 use V3CipherSuites not V3CipherSuites4char. V3CipherSuites takes 2 character identifiers and long TLS_… strings. V3CipherSuites4char only take 4 character identifiers , and not TLS_… strings.
  • If you change the PAGENT configuration you need F PAGNET,UPDATE to pick up the changes.
  • If you make a change to the PAGENT configuration, it does not always report problems. You should use Unix command commands like pagent -t -f TN 1>a ;oedit a to display the TTLS configuration for rule TN, and edit the file to check.
  • If you change a keyring, you need F CPAGENT,REFRESH to pick up the change.
  • When you start PAGENT use the -l /var/log/pagent.log -d1 options. Review the /var/log/pagent.log file and check for “WARNING” and “ERR”
  • The PAGENT configuration can have the same parameter specified at the environment level and at the connection level. The pasearch -t -f TN command displays both sets of data … you have to be careful you are checking the right set of data.
  • PAGENT does not report if a section was skipped. I had two TTLSConfig statements in a TCPIP image file, and only the last one was used.
  • You need the syslogd daemon running to capture error messages in a file.

General PAGENT commands

See here for starting PAGENT (and the trace levels you can use).

Updating PAGENT

If you change the PAGENT configuration (my job is CPAGENT) you can use

F CPAGENT,UPDATE

To refresh the configuration,

F CPAGENT,REFRESH

to refresh the configuration and restart processing. Existing sessions will continue unchanged.

When configuring TN3270 you can use the Unix command pasearch -t -f TN 1>a to display just the TLS policy you are interested in (TN) .

Displaying information

You display configuration information using the Unix command pasearch…

What does a rule configuration look like?

I frequently use

pasearch -t -f TN

This shows me the TLS configuration for rule TN

What does the high level TLS object look like?

pasearch -c -t

gave me 

TTLS Policy Object:                                                     
  ConfigLocation:       Local             LDAPServer:        False      
  CommonFileName:                                                       
  ImageFileName:        //'USER.Z24C.TCPPARMS(PAGENTTN)'                
  ApplyFlush:           True              PolicyFlush:       True       
  ApplyPurge:           True              PurgePolicies:     True       
  AtomicParse:          True              DeleteOnNoflush:   False      
  DummyOnEmptyPolicy:   True              ModifyOnIDChange:  False      
  Configured:           True              UpdateInterval:    1800       
  TTLS Enabled:         True                                            
  InstanceId:           1666079380                                      
  LastPolicyChanged:    Tue Oct 18 08:49:40 2022

The CommonFileName: value was blank, which was a surprise as I had specified a file.

How to display a summary of a rule?

pasearch -t -n

gave

policyRule:             TN         
  TTLS Action:          TNGA       
  TTLS Action:          TNEA       
  TTLS Action:          TNCA

How do I display which policy or rule has been used?

I could not find a way of displaying which rules were used, and which options were used.

Using z/OSMF Network Configuration assistant for TCPIP, to define AT-TLS configuration

I initially found it hard to set up the AT-TLS configuration for MQ. The easiest way was to use the sample configurations provided by MQ. See here for an overview. I used Scenario 5 – Between an IBM MQ for z/OS queue manager and a client application running on IBM MQ for Multiplatforms.

Using the MQ samples, this took about 10 minutes once I had PAGENT and SYSLOGD set up.

I thought I would try to use the TCP provided facilities. There is a lot of documentation, but it is not easy to find what you need. It has been written as an IBM developer, rather than from an end user or task based perspective.

I then thought I would try to use the “way of the future” and the z/OS configuration tool z/OSMF. You use a browser to connect to z/OSMF and do your work through the browser interface. The z/OSMF interface has configuration tools, and workflow tools which guide you step by step through configuration.

I’ve blogged Using z/OSMF workflows for TCPIP. Using the workflow was not very successful.

The Network Configuration Assistance is used to configure the PAGENT, and I used it to define a AT-TLS configuration. Initially this was a struggle as there was no documentation to tell me how to do it. Once I had been through the configuration a couple of times, I found the correct path through the configuration progress and it is relatively painless.

My mission.

My mission was to configure AT-TLS and to provide two ports for use with MQ.

I wanted to do this using two people (me with two userids) and do the typical steps when changing systems, such as saving configurations before changing them, and deploying them, when I had a “change window”.

Using the Network configuration assistant (CA)

AT-TLS concepts

You need to be aware of the AT-TLS concepts when defining the configuration. From an administrator’s perspective:

  • What ports you want to protect? This is known by the CA as Traffic Descriptors. You can specify
    • An IP port
    • A range of IP ports
  • What IP addresses you want to protect.
    • The IP address. A TCP/IP stack can support different IP addresses. You can use a specific IP address. You can select on all IPV4, or all IPV6, or all addresses.
    • The name of a group of IP addresses. z/OSMF CA calls these Address Groups.
  • How do you want to protect the session. For example what levels of TLS, and what cipher specs do you want to use. This is known by the CA as Security levels.
  • The mapping of ports to protecting the session. z/OSMF calls this Requirement Maps.
  • You configure at the TCPIP stack level.
  • z/OSMF has groups of z/OS instances, with one or more z/OS instances, and you can have multiple TCPIP stacks in a z/OS instance.

Backing store

The configuration assistant(CA), stores configuration in a backing store. You can use tools to copy the current store. I found a copy of the file in /global/zosmf/data/app/CAV2R4/backingStore/save Data. I should be able to use standard backup procedures to keep a copy of the file. The resulting configuration is created in a file which is used by PAGENT.

You can copy a backing store within the CA, and so go back to the original one if you need to.

Before you start.

You should collect the information that you will be used to configure PAGENT. For example

  • Which systems and IP stacks will be used.
  • Which keyrings and certificates will be used?
  • For each port you want to protect.
  • What rules do you want, for example which cipher specs.

I found the terms used when creating the rules manually – did not map to the CA concepts, but once you understood the difference in terminology it was ok.

How to define the resources

If you define the configuration bottom up. You define all of the rules, then when you get to configure the TCPIP stack, the rules and other components should all be there, you just have to select them.

If you define the configuration top down, you define the TCPIP stack, then the rules for the stack. You have to define the TCPIP stack, then the rules, then go back and reconfigure the TCPIP stack to add the rules.

I think bottom up configuration is better while you gain experience of the tool. Once you are familiar with the tool then the top down approach will work ok, and may be how you update the configuration.

Getting started

  • Double click on the Network Configuration Assistant icon.
  • You get a page Welcome to V2R4 Configuration Assistant for z/OS Communications Server. On this page you can specify the name of the backing store. The pull down lists the existing backing stores. If you do not have a backing store create one. You can use “tools”button to copy the backing store.
  • The “getting started” gives you information on how to use the panels. I found it a little confusing at times. It displays the help in a separate window. In the Table of Contents, it has “AT/TLS – getting started”. I didn’t find the help or tutorials much use.
  • On the Welcome page, press Proceed.
    • I sometimes get “The backing store is locked by your id.” I got this after I had shutdown down z/OSMF without logging off.
    • You can use “Tools” to manage your backing store, and configuration.
    • “Select a TCP/IP technology to configure” : AT-TLS
    • The layout of the panels, make me think you create the definitions from top to bottom, and so the tabs are defined left to right. I think it is easier to define resources then create the group/image/stack.

Define the rules for which ports to be protected

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Traffic Descriptors tab.

  • Actions -> New…
  • Name COLINTD
  • Actions-> New…
  • Under the Details tab, specify the port or port range and any other information
  • Under the KeyRing tab, specify the keyring and the Certificate label or let it default to the TCPIP stack level keyring.
  • Under the Advanced tab, I let everything default.
  • Click OK
  • You can define a second port within this Traffic Descriptor
  • Click OK

You can press Save to save the current definitions.

Define which IP addresses you want to protect (optional)

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Address Groups tab.

By default it has

  • All_IPv4_Addresses
  • All_IPv6_Addresses
  • All_IP_Addresses

A TCPIP stack can host different IP addresses, one for each connector coming in. If you want to limit rules to particular stack owned IP addresses, create a definition.

  • Actions-> New
  • Name: COLINAG
  • IP Address: 10.1.1.2
  • IP Address: 10.1.1.3
  • OK

You can press Save to save the current definitions.

How do you want to protect the session.

For example what levels of TLS, and what cipher specs do you want to use.

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Security Levels tab.

  • Actions: -> New…
  • Name: COLINSL
  • Select levels of TLS you want to use
  • Next, then select cipher specs. I used “Use 2019 suggested values”
  • Next – I took the default (“Do not generate FIPS 140 support“)
  • Click on Advanced settings.
    • If you want to use client authentication click the “Use client authentication” box
    • OK
  • Finish

Your definition should be in the list.

You can press Save to save the current definitions.

Mapping of ports to Session protection

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Requirement Maps tab.

  • Actions: -> New…
  • Name: COLINMAP .
  • In the mappings table,
    • use the Traffic Descriptor pull down and select the Traffic Descriptor you created above. For example COLINTD.
    • Under Security Level pull down select the security definition you created above. For example COLINSL.
  • OK

If I changed an existing definition, I had a pop-up

Modify Requirement Map.
The requirement map you are changing may be referenced in at least one connectivity rule.

Prior to making this change you may want to see which connectivity rules are referencing this requirement map. Click OK to show where used. Click Proceed to proceed with the Modify; otherwise, click Cancel.

Click OK to show where it is used.

Click Proceed

You can press Save to save the current definitions.

Create the group, z/OS instance and TCPIP Stack

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Systems tab.

  • Action: -> Add z/OS group…
  • Name: COLINGR
  • Click OK
  • Action: -> Add z/OS system image…
  • Name: COLMVSA
  • Keyring: START1.COLINRING
  • Press OK
  • I get a pop-up Proceed to the next step? Connectivity rules are configured for each TCP/IP stack. To continue with configuration you need to add a TCP/IP stack to the new z/OS system image. Do you want to add a TCP/IP stack now? Click on Proceed.
  • This gets to the Add TCP/IP stack
  • Name:TCPIP
  • OK
  • I get a pop-up. Proceed to the next Step? o continue with the configuration you should add connectivity rules to the TCP/IP stack. Do you want to be directed to the TCP/IP stack rules panel? Proceed.
    • If you cancel you can use the Actions -> rules to define the rules.
  • I get a pop-up Proceed to the Next Step? Do you want to start a wizard to create a connectivity rule? Click Proceed.
  • This gets to the Data End points where you associate the IP addresses to the stack instance.
  • Name: COLINRN
  • Select from the address group pull-down, or let it default.
  • Press Next
  • This gets to the Requirement Mapping to Stack association.
    • You can select from the existing requirements map see Mapping of ports to Session protection above, or create a new mapping.
    • You can create a new map, for example Name: COLINMP
      • Select from the Traffic Descriptor pull down
      • Select from the Security level pull down.
  • Press Next
  • You can specify advanced settings, such as Tracing, Tuning, Environment, Effective times, Handshake
  • Finish
  • Close

You can press Save to save the current definitions.

Join the bits up

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Systems tab.

  • Select a group instance
  • Actions: Install All files for this group
  • This will list the configuration files.
  • On the Configured File Name,
    • right click on the file name value, and ->Show Configuration File. This will show you the configuration as it might be deployed.
    • right click on the file name value and -> Install … . Specify the a file name and click GO.
    • Close
  • You can use
    • Actions: Install to create the configuration file
    • Actions: Show configuration file to see the generated configuration

You can now use the configuration file as input to PAGENT.

You can press Save to save the current definitions.

Extending the configuration to add a new rule

It took a while to work out how to do this, but it is really easy.

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Traffic Descriptors tab.

  • Create a new Traffic descriptor as above
  • Get back to Network Configuration Assistant (Home) -> AT-TLS page, and click on the Systems tab.
  • Select a TCPIP instance, and click Actions: -> Rules..
  • Actions: -> New
  • Connectivity rule name: rule2
  • Press Next
  • You can select from the existing requirements map see Mapping of ports to Session protection above, or create a new mapping. If you have just created a new rule, then you may not have defined a mapping, and create it “inline”.

Then install it.

Use the configuration

You need to change the PAGENT JCL to use the created configuration file. You may want to copy it to a backup, as the next time you reconfigure it can overwrite the file. Or just create a new file perhaps with a date in the filename.

If you have problems with a newly reconfigured file you need a copy of the previous, working, definitions.

Display the configuration

On many items, you can use right click -> Show where used. This will then display the group, image, stack, connectivity rules and data end points where the item is used.

Should I use this just to get started, or every time.

When I created my definitions by hand, I could put definitions in to a “Common” section, and have multiple TCPIP stacks in one configuration file. I could have small files with bits of configuration in them.

If you use the CA, “common” definitions are copied into the configuration file, and you have one configuration file per TCPIP stack instance, so you do not need to have a common section etc.

As a configuration tool, now I know how to use it, I might continue to use it – but it is slightly more complex than this.

I want to enable trace for one definition. To do so means I have to…

  • Change the configuration to set the trace. This can be difficult if someone else is in the middle of changing the configuration.
  • Deploy the whole configuration. You may pick up incomplete changes which have been made, but not deployed.
  • If a second TCPIP stack is using the configuration, this may get trace enabled if the configuration file is recreated.

Overall (my views may change tomorrow), I would use the CA to create my configuration. Then not use it again – or use it again to generate definitions which I can copy into my master configuration files. I would restructure the configuration so create small files with specific content.

Using z/OSMF workflows for TCPIP.

I found it hard to set up the AT-TLS configuration for MQ. The easiest way was to use the sample configurations provided by MQ. See here for an overview. I used Scenario 5 – Between an IBM MQ for z/OS queue manager and a client application running on IBM MQ for Multiplatforms.

This took about 10 minutes once I had PAGENT and SYSLOGD set up.

I thought I would try to use the TCP provided facilities. There is a lot of documentation, but it is not easy to find what you need. It has been written as an IBM developer, rather than from an end user perspective.

I then thought I would try to use the “way of the future” and the z/OS configuration tool z/OSMF. You use a browser to connect to z/OSMF and do your work through the browser interface. The z/OSMF interface has configuration tools, and workflow tools which guide you step by step through configuration.

I found using the workflow tools was harder than using the TCPIP documentation and TCPIP samples, and I would not recommend its use.

Ive blogged Using z/OSMF Network Configuration assistant for TCPIP, to define AT-TLS configuration. Which worked.

The workflow stuff makes the easy bit “easier”, but does not help with the hard stuff. An improvement would be to skip the workflow, and have one page of instructions saying copy samples into Proclib, and Unix; run a RACF job. We could do with a workflow to help configure syslogd, which I had a struggle to get working in a non trivial situation. For example having error messages for PAGENT go to one file, and have the TLS trace go into another file.

My mission.

My mission was to configure AT-TLS and to provide two ports for use with MQ.

I wanted to do this using two people (me with two userids) and do the typical steps when changing systems, such as saving configurations before changing them, and deploying them, when I had a “change window”.

Initial steps

z/OSMF provides facilities like ISPF, Workload management configuration, system status etc. I used Workflow.

It was hard to know where to start. I assumed (wrongly) that there would be a workflow to define the AT-TLS definitions.

It seems you use Workflow to define the PAGENT and syslogd JCL, and not for configuring the PAGENT or syslogd.

Instructions to use Workflow to configure TCPIP JCL procedures

  • Double click the workflow icon.
  • From the actions pull down, select Create workflow…
  • You need to select Workflow definition file: I could not find what I had to specify. There was no prompting. The “?” basically said “put a value here”. The help key just gave me a panel with information about using creating a workflow.
  • I found an IBM support document which says
    • Workflows for Policy-based Networking
    • ezb_pagent_setup_wizard.xml – This workflow provides the steps for setting up the Policy Agent (Pagent). Pagent is required for all of the policy-based networking technologies: IPSec, AT-TLS, IDS, PBR, and QoS. Pagent uses syslogd for logging.
    • ezb_syslogd_setup_wizard.xml – This workflow provides the steps for setting up syslogd.
    • ezb_tcpip_profile_sample_wizard.xml – This workflow provides a sample TCP/IP profile which contains common statements required to enable AT-TLS and IP Security, and additionally includes port reservation statements for running daemons.
    • …
  • I had to use the fully qualified filename /usr/lpp/zosmf/workflow/plugins/izuca/ezb_syslogd_setup_wizard.xml
  • This came up with an error in the workflow name because the default name has ‘z/OS… ‘ and ‘/’ is not a valid character. I removed the ‘/’.
  • At the bottom of the page you can Assign all steps to owner user id. I did not do this, and had to assign steps below
  • You get a list of steps that need to be done.
  • Assign the work to a userid
    • Select all of the steps, and use Actions-> Assignment and ownership -> Add assignees.
    • This displays the assigned roles. I used Actions -> add to add my SAF userid. I pressed OK and returned to the list of steps – all now assigned to me.
  • I selected the first step “define the “RACF userid for Syslogd”, Actions -> Accept .
  • Click on the task, and it gives you a window with tabs. The important tab is Perform. If this is greyed out, you have not accepted the task!
    • Fill in the details and click Next, Next etc. You can edit the contents.
    • You can save it – but you need to give a data set. It suggested SYS1(SYSLOGD). I had to change it (every time) to COLIN.ATTLS(…)
    • Next – gives you the save panel. You have to specify the dataset where you want to save it. The default was wrong for me.
    • Once saved you have to submit it manually, check the output, and edit the file if needed.
  • Back at the workflow details, it had step 1 complete (even though you may not have submitted it)
  • I accepted step 2 and started working on it.
    • It asks for Dataset HLQ – but I could not change it.
    • I stepped through the definitions – and had to type in my dataset again (why can’t it remember what I specified last time).
    • This step just creates a job with some RACF definitions in it.
  • I ran step 3 -again just creating a JCL member of definitions
  • Step 4 “Sample Syslogd Configuration Setup“. This just copies in a sample configuration.
    • “Save” did not do anything
  • Step 5 “Sample started procedure for Syslogd” creates a sample Procedure.
  • On the workflows page, it shows the workflow is 100% complete.

Having been through all of this, the create JCL did not run, one line in error was

//SYSLOGD PROC PROG=”,
// VARS=”,
// PARMS=”
//SYSLOGD EXEC PGM=&PROG., REGION=0K,TIME=NOLIMIT,
// PARM=(‘POSIX(ON) ALL31(ON)’,
// ‘ENVAR(“_CEE_ENVFILE=DD:VARS”)’,
// ‘/&PARMS.’)

…

  • &PROG had not been specified – you gave to go and find what you need to specified (SYSLOGD)
  • There is a blank after the &PROG., so the REGION=0K,TIME=NOLIMIT, is ignored
  • The location of the configuration (in &VARs) is not specified.

Create the PAGENT JCL

I followed the same process to create the PAGENT file.

I used file /usr/lpp/zosmf/workflow/plugins/izuca/ezb_pagent_setup_wizard.xml.

When this JCL ran, it produced messages

06/16 08:00:20 SYSERR :000: …plfm_config_medium_open: cannot open ‘/etc/pagent.conf’, errno EDC5129I No such file or directory.

You have to know to copy the configuration file from the PDS to /etc/pagent.conf.

Comments on using the workflows

This seems a lot of work to produce code which does not work. The process feels unloved. I am surprised that the problems I found have not been fixed – they are Unit Test level bugs.

I think it is far simpler to follow the documentation, for example to create the procedure. The documentation says

Update the cataloged procedure, syslogd, by copying the sample in SEZAINST(SYSLOGD) to your system or recognized PROCLIB. Specify syslogd parameters and change the data set names to suit your local configurtion See the syslog daemon section of SEZAINST(EZARACF) for SAF considerations for started procedures

The instructions could be on one side of paper, and would be quicker than using the workflow.

Wireshark – using external data

Export the trace in hex format

  • Capture your trace.
  • Select the line(s) of interest
  • File -> Export Packet Dissections -> As Plain Text
  • On the next screen
    • Select the directory and file name
    • Select the Export as “Plain text” – it adds .txt to the file name you chose above, if required.
    • Select the packet range
    • Packet format: Bytes (and only bytes)
    • Save

If you select Packet Format : Bytes you get output like

0000  00 d8 61 e9 31 2a 8c 16 45 36 f4 8a 08 00 45 10   ..a.1*..E6....E.
0010  00 78 39 c5 40 00 40 06 ec a4 0a 01 00 02 0a 01   .x9.@.@.........
0020  00 03 bc 3e 00 16 0d 5b 33 ab 44 ae 0b 8f 80 18   ...>...[3.D.....

If you select Packet Format: Details

You get 

No.     Time           Source                Destination           Dst port port   Protocol Length Info
      1 0.000000000    10.1.0.2              10.1.0.3              22       48190  SSH      134    Client: Encrypted packet (len=68)

Frame 1: 134 bytes on wire (1072 bits), 134 bytes captured (1072 bits) on interface enp0s31f6, id 0
Ethernet II, Src: LCFCHeFe_36:f4:8a (8c:16:45:36:f4:8a), Dst: Micro-St_e9:31:2a (00:d8:61:e9:31:2a)
Internet Protocol Version 4, Src: 10.1.0.2, Dst: 10.1.0.3
Transmission Control Protocol, Src Port: 48190, Dst Port: 22, Seq: 1, Ack: 1, Len: 68
SSH Protocol

If you specify If you select Packet Format: Details + Bytes you get both sets of output in the file.

If you try to import this files back into Wire Shark, it should work, as it will ignore lines which do not begin with a hexadecimal offset.

Import a hex dump file

Using the “bytes” file created above

  • File -> Import from Hex Dump
  • File /u/colin/aaa.txt
  • Offsets: Hexadecimal
  • Encapsulation type: Ethernet

The data is displayed as if it was real time capture.

The Wireshark documentation on importing data is here.

Information in the imported file.

Various bits of information are not in the hex dump file, and WireShark creates (fakes) them. For example

  • The packet number is the data record number of the data in the file.
  • The time between packets is an incrementing microsecond counter, 0.000000, 0.000001, 0.000002 etc
  • The packet length is taken from the length of the data record.
  • The “arrival time” is the time the data was read from the file.
  • The encapsulation type is taken from the import page.
  • However information such as source, destination, port, destination port are taken from the data.

Bodging a hex dump file

I had a AT-TLS trace from PAGENT which was a string of raw hex data, with no interpretation of the TLS data. AT-TLS does not provide any way of formatting this data.

I was partially successful in using WireShark to process this data, and decode the TLS fields.

The first part of the data is the IP and TCP header info, followed by the TLS data.

I obtained a hex dump of a TLS handshake and took the first x42 bytes.

0000  8c 16 45 36 f4 8a 00 d8 61 e9 31 2a 08 00 45 00 
0010  xx xx 01 3d 00 00 3f 06 5c ef 0a 01 01 02 0a 01 
0020  00 02 05 86 b8 dc 34 b1 03 17 57 c4 a5 0c 80 18
0030  0f fc 1d bf 00 00 01 01 08 0a 9d cf 5c b7 db 69 
0040  e9 0d

I then wrote an ISPF editor rexx program to take the TLS trace and convert into the similar format, such as

0042   16 03 03 08 8E 02 00 00 4D 03 03 62 A8 75 18 C9
0052   5D 61 E1 1B 71 40 6A 6B 95 A8 F9 E5 E8 3A 83 AB
....
07E2   F0 61 98 51 92 4B 2E 0A 29

Then changed the xx xx to the length of the data – 12. So xx xx becomes (x7ea – 12) = x07dd.

The data could sometimes be imported.

If anyone is interested I can send them the ISPF rexx exec.

The trace from PAGENT is some times incomplete or wrong. I had long records with

0A29C..TRC at the end, which is clearly not hex data.

Hints on bodging.

It took me several hours to get the first successful import of TLS data. Some of the problems I had were

  • The offsets were wrong. I sometimes had the wrong offset, or the same offset more than once
  • The xx xx needs to be correct.
  • In the PAGENT trace file I had data like 160303007B which is of length 10 – but only length 5 hex characters. The length would be 00 47 ( 00 42 + 5)

Netstat, TTLS and AT-TLS

Once a session has been established using AT-TLS to do end to end TLS encryption, you can use netstat to display information about the session, and what configuration is being used. It feels slightly incomplete, in that some of the data I expected is not available.

What sessions are using a Port? – and display TTLSPolicy information

tso netstat all (port 1414

You can use other filter statements, using port 1414 was easy to specify.

This displays the high level TTLS information, see the blue text below

EZZ2350I MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP           16:40:39
EZZ2550I Client Name: CSQ9CHIN                 Client Id: 000000BB
EZZ2551I Local Socket: 10.1.1.2..1414          Foreign Socket: 10.1.0.2..51844 
EZZ2577I   BytesIn:            0000002248        BytesOut:           0000002076
EZZ2574I   SegmentsIn:         0000000020        SegmentsOut:        0000000014
EZZ2536I   StartDate:          05/31/2022        StartTime:          16:31:54
EZZ2552I   Last Touched:       16:36:57          State:              Establsh
EZZ2553I   RcvNxt:             0626077759        SndNxt:             3598815082
EZZ2554I   ClientRcvNxt:       0626076574        ClientSndNxt:       3598812426
EZZ2555I   InitRcvSeqNum:      0626074325        InitSndSeqNum:      3598810349
EZZ2556I   CongestionWindow:   0000018720        SlowStartThreshold: 0000065535
EZZ2557I   IncomingWindowNum:  0626208746        OutgoingWindowNum:  3598877418
EZZ2558I   SndWl1:             0626077759        SndWl2:             3598815082
EZZ2559I   SndWnd:             0000062336        MaxSndWnd:          0000064256
EZZ2560I   SndUna:             3598815082        rtt_seq:            3598814997
EZZ2561I   MaximumSegmentSize: 0000001440        DSField:            00
EZZ2563I   Round-trip information:
EZZ2564I     Smooth trip time: 7.000             SmoothTripVariance: 19.000
EZZ2565I   ReXmt:              0000000000        ReXmtCount:         0000000000
EZZ2572I   DupACKs:            0000000000        RcvWnd:             0000130987
EZZ2566I   SockOpt:            88                TcpTimer:           00
EZZ2567I   TcpSig:             04                TcpSel:             40
EZZ2568I   TcpDet:             E4                TcpPol:             00
EZZ2593I   TcpPrf:             89                TcpPrf2:            20
EZZ2593I   TcpPrf3:            00
EZZ2593I   DelayAck:           Yes
EZZ2537I   QOSPolicy:          No
EZZ2545I   TTLSPolicy:         Yes
EZZ2546I     TTLSRule:         REMOTE-TO-CSQ1
EZZ2547I     TTLSGrpAction:    CSQ1-GROUP-ACTION
EZZ2548I     TTLSEnvAction:    CSQ1-INBOUND-ENVIRONMENT-ACTION
EZZ2542I   RoutingPolicy:      No
EZZ2570I   ReceiveBufferSize:  0000065536        SendBufferSize:     0000065536
EZZ2538I   ReceiveDataQueued:  0000000000
EZZ2539I   SendDataQueued:     0000000000
EZZ2611I   SendStalled:        No
EZZ2609I   Ancillary Input Queue: N/A
...

From the clientid (connection id) display any TTLS information

From the netstat allconn (port 1414 command, you get each session, and its clientid (see above for clientid 000000BB).

From the tso netstat allconn (port 1414 command, you get one line per session with the connection ID and remote IP address and port.

EZZ2350I MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP           16:55:18
EZZ2585I User Id  Conn     Local Socket           Foreign Socket         State
EZZ2586I -------  ----     ------------           --------------         -----
EZZ2587I CSQ9CHIN 000000BB 10.1.1.2..1414         10.1.0.2..51844        Establsh
EZZ2587I CSQ9CHIN 00000022 0.0.0.0..1414          0.0.0.0..0             Listen

Issue the command to display the TTLS information details about connection.

tso netstat ttls conn 000000BB detail

MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP  
ConnID: 000000bb
  JobName:      CSQ9CHIN
  LocalSocket:  10.1.1.2..1414
  RemoteSocket: 10.1.0.2..53230
  SecLevel:     TLS Version 1.2
  Cipher:       003C TLS_RSA_WITH_AES_128_CBC_SHA256
  KeyShare:     N/A
  CertUserID:   N/A
  MapType:      Primary
  FIPS140:      Off
  SessionID:    01010018 0A010002 CFEE0000 00000000
                00000000 00000000 62970B05 00000001
  SIDReuseReq:  Off
TTLSRule: REMOTE-TO-CSQ1
  Priority:       1
  LocalAddr:      All
  LocalPort:      1414
  RemoteAddr:     All
  RemotePort:     All
  JobName:        CSQ9CHIN
  Direction:      Inbound
  TTLSGrpAction:  CSQ1-GROUP-ACTION
    GroupID:                    00000007
    TTLSEnabled:                On
    CtraceClearText:            Off
    Trace:                      2
    SyslogFacility:             Daemon
    SecondaryMap:               Off
    FIPS140:                    Off
  TTLSEnvAction:  CSQ1-INBOUND-ENVIRONMENT-ACTION
    HandshakeRole:              Server
    SuiteBProfile:              Off
    MiddleBoxCompatMode:        Off
    Keyring:                    START1/MQRING
    V3CipherSuites:             003C TLS_RSA_WITH_AES_128_CBC_SHA256
    Trace:                      255
    SSLV2:                      Off
    SSLV3:                      Off
    TLSV1:                      Off
    TLSV1.1:                    Off
    TLSV1.2:                    On
    TLSV1.3:                    On
    ResetCipherTimer:           0
    ApplicationControlled:      Off
    HandshakeTimeout:           10
    CertificateLabel:           ZZZZ
    SecondaryMap:               Off
    TruncatedHMAC:              Off
    ClientMaxSSLFragment:       Off
    ServerMaxSSLFragment:       Off
    ClientHandshakeSNI:         Off
    ServerHandshakeSNI:         Off
    ClientECurves:              0021 secp224r1
                                0023 secp256r1
                                0024 secp384r1
                                0025 secp521r1
                                0019 secp192r1
                                0029 X25519
    ClientKeyShareGroups:       0023 secp256r1
    ServerKeyShareGroups:       0023 secp256r1
                                0024 secp384r1
                                0025 secp521r1
                                0029 X25519
                                0030 X448
    SignaturePairs:             0601 TLS_SIGALG_SHA512_WITH_RSA
                                0603 TLS_SIGALG_SHA512_WITH_ECDSA
                                0501 TLS_SIGALG_SHA384_WITH_RSA
                                0503 TLS_SIGALG_SHA384_WITH_ECDSA
                                0401 TLS_SIGALG_SHA256_WITH_RSA
                                0403 TLS_SIGALG_SHA256_WITH_ECDSA
                                0402 TLS_SIGALG_SHA256_WITH_DSA
                                0301 TLS_SIGALG_SHA224_WITH_RSA
                                0303 TLS_SIGALG_SHA224_WITH_ECDSA
                                0302 TLS_SIGALG_SHA224_WITH_DSA
                                0201 TLS_SIGALG_SHA1_WITH_RSA
                                0203 TLS_SIGALG_SHA1_WITH_ECDSA
                                0202 TLS_SIGALG_SHA1_WITH_DSA
                                0806 TLS_SIGALG_SHA512_WITH_RSASSA_PSS
                                0805 TLS_SIGALG_SHA384_WITH_RSASSA_PSS
                                0804 TLS_SIGALG_SHA256_WITH_RSASSA_PSS
    ClientAuthType:             Required
    CertValidationMode:         Any
    Renegotiation:              Default
    RenegotiationIndicator:     Optional
    RenegotiationCertCheck:     Off
    3DesKeyCheck:               Off
    ClientEDHGroupSize:         Legacy
    ServerEDHGroupSize:         Legacy
    PeerMinCertVersion:         Any
    PeerMinDHKeySize:           1024
    PeerMinDsaKeySize:          1024
    PeerMinECCKeySize:          192
    PeerMinRsaKeySize:          1024
    ServerScsv:                 Off
    GSK_V3_SESSION_TIMEOUT:     86400
    GSK_V3_SIDCACHE_SIZE:       512
    GSK_SESSION_TICKET_CLIENT_ENABLE:      On
    GSK_SESSION_TICKET_CLIENT_MAXSIZE:     8192
    GSK_SESSION_TICKET_SERVER_ENABLE:      On
    GSK_SESSION_TICKET_SERVER_ALGORITHM:   AESCBC128
    GSK_SESSION_TICKET_SERVER_COUNT:       2
    GSK_SESSION_TICKET_SERVER_TIMEOUT:     300
    GSK_SESSION_TICKET_SERVER_KEY_REFRESH: 300
    HttpCdpEnable:              Off
    HttpCdpProxyServerPort:     80
    HttpCdpResponseTimeout:     15
    HttpCdpMaxResponseSize:     204800
    HttpCdpCacheSize:           32
    HttpCdpCacheEntryMaxsize:   0
    OcspAiaEnable:              Off
    OcspProxyServerPort:        80
    OcspRetrieveViaGet:         Off
    OcspUrlPriority:            On
    OcspRequestSigalg:          0401 TLS_SIGALG_SHA256_WITH_RSA
    OcspClientCacheSize:        256
    OcspCliCacheEntryMaxsize:   0
    OcspNonceGenEnable:         Off
    OcspNonceCheckEnable:       Off
    OcspNonceSize:              8
    OcspResponseTimeout:        15
    OcspMaxResponseSize:        20480
    OcspServerStapling:         Off

Which AT-TLS groups are being used?

I didn’t find this information very useful. It isn’t clear what a group is. The doc says

Use the TTLSGroupAction statement to specify parameters for a Language Environment process required to support secure connections. The TTLSGroupAction statement indicates whether a selected connection should use AT-TLS security.

tso netstat ttls group
tso netstat ttls

MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP         
TTLSGrpAction                             Group ID           Conns
----------------------------------------  -----------------  -----
CSQ1-GROUP-ACTION                         0000003F               1
GrpActOff                                 00000040               0
GrpActOn                                  00000041               0
GA1                                       00000042               0

tso netstat ttls group detail

MVS TCP/IP NETSTAT CS V2R4       TCPIP Name: TCPIP         
TTLSGrpAction:   CSQ1-GROUP-ACTION
  GroupID:         0000003F
  Tasks:           4                    GroupConns:      1
  WorkQElements:   0                    SyslogQElements: 0
    Env: CSQ1-INBOUND-ENVIRONMENT-ACTION           EnvConns: 1
TTLSGrpAction:   GrpActOff
  GroupID:         00000040
  Tasks:           4                    GroupConns:      0
  WorkQElements:   0                    SyslogQElements: 0
TTLSGrpAction:   GrpActOn
  GroupID:         00000041
  Tasks:           4                    GroupConns:      0
  WorkQElements:   0                    SyslogQElements: 0
TTLSGrpAction:   GA1
  GroupID:         00000042
  Tasks:           4                    GroupConns:      0
  WorkQElements:   0                    SyslogQElements: 0

Trace PAGENT and AT-TLS

Many components of TCPIP write information to syslogd. This is a process that captures the data sent to it over a UDP socket, and writes it to files in the Unix file system. If the syslogd is not active then messages may be written to the job log. When I was trying to set up AT-TLS, I had 10s of messages on the job log, each time a client tried to use AT-TLS.

The IBM documentation is not very clear, it tells you how to turn on debug, trace etc but does not clearly explain the difference, and when they are used.

It look like the PAGENT job is to take a configuration file, parse it, and pass the data to TCPIP.

If you are using AT-TLS to set up TLS channels, the trace data comes from the TCPIP address space.

Modify the PAGENT address space.

You can pass commands to the PAGENT address space.

Configuration processing.

You can control how much information is logged when parsing configuration statements. The value 127 covers most levels of information (including warnings).

F PAGENT,LOGLEVEL,LEVEL=127

You can use

F PAGENT,TRACE,LEVEL=..
F PAGENT,DEBUG,LEVEL=…

But these do not seem to control the level of trace produced.

Trace PAGENT startup and parse of the configuration

To collect the PAGENT startup and display information on the configuration file as it is processed change the started task JCL to include the -d option.


//PAGENT EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT,
// PARM=’ENVAR(“_CEE_ENVFILE_S=DD:STDENV”)/ -d 4‘

By default the output trace goes to /tmp/pagent.log. It has content like

05/29 17:17:54 EVENT :005: pzos_install_A_PolicyRule: Finished installing policy rule: ‘REMOTE-TO-CSQ1’

Trace PAGENT use of TLS

My PAGENT JCL is

//CPAGENT  PROC 
//  SET EN='ENVAR("_CEE_ENVFILE_S=DD:STDENV")' 
//PAGENT   EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT, 
//       PARM='&EN/                                      -d 4' 
//STDENV   DD DISP=SHR,DSN=USER.Z24C.TCPPARMS(PAGENTEN) 
//SYSPRINT DD SYSOUT=H 
//SYSERR   DD SYSOUT=H 
//SYSOUT   DD SYSOUT=H 
//* 
//CEEDUMP  DD SYSOUT=*,DCB=(RECFM=FB,LRECL=132,BLKSIZE=132)

With the environment file USER.Z24C.TCPPARMS(PAGENTEN) having 

_CEE_ENVFILE_COMMENT=# 
PAGENT_CONFIG_FILE=//'USER.Z24C.TCPPARMS(PAGENTCF)' 
LIBPATH=/usr/lib 
GSK_TRACE=0x00
GSK_TRACE_FILE=/var/log/GSK

You can collect the GSK calls made by PAGENT at startup by using the environment variables

GSK_TRACE=0xFF
GSK_TRACE_FILE=/var/log/GSK

Note: This turns it on for all requests! I could not find how to do selective tracing.

You have to format the trace file using

gsktrace /var/log/GSK /var/log/GSK.txt

This has about 40 lines with information like

05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-1 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-224 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-256 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-384 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-512 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES3 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 128-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 256-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES-GCM crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic accelerator is not available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic coprocessor is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Public key hardware support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Max RSA key sizes in hardware – signature 4096, encryption 4096,
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Maximum RSA token size 3500
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC clear key support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC secure key support is available. Maximum key size 521

Remember this is the PAGENT invoking GSK – but PAGENT does not do any TLS work – this is done by TCPIP.

Trace an AT-TLS connection.

You need to enable trace in the AT-TLS configuration for example

TTLSEnvironmentAction CSQ1-INBOUND-ENVIRONMENT-ACTION
{
HandshakeRole SERVER
TTLSKeyringParmsRef CSQ1-KEYRING
TTLSCipherParmsRef CSQ1-CIPHERPARM
TTLSEnvironmentAdvancedParmsRef CSQ1-ENVIRONMENT-ADVANCED
Trace 255
}

If the syslogd daemon is not configured the output goes to the TCPIP job log.

If the syslogd daemon is configured, you need to have a syslogd configuration with


*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

Where TCPIP is the TCPIP address space name, and *.*.*.* is a catch-all. It took me about a day to realize that my trace was being thrown away because I didn’t have TCPIP, and the catch-all.

The trace file has data like

May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1284I TTLS Flow GRPID: 00000021 ENVID: 00000009 CONNID: 00000053 RC: 0 Set GSK_USER_DATA(200) – 000000007F280610

May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1285I TTLS Data CONNID: 00000053 RECV CIPHER 160303007B

This shows a GSK call was made to GSK_USER_DATA which completed with return code 0, and the connection RECeiVed data which was the CIPHER specs ( 4 chars or 2 chars) 160303007b.

You need to configure the syslogd procedure.

See if SYSLOGD is running, if not, try to start it. If it does not exist…

  • Copy /usr/lpp/tcpip/samples/syslog.conf to its default configuration file /etc/syslog.conf, or another file.
  • Copy TCPIP.SEZAINST(SYSLOGD) to your proclib concatenation.
  • The program uses environment variables defined in STDENV to control operations. The default configuration file location is /etc/syslog.conf

You can configure syslog.conf for example

*.TCPIP.*.* /var/log/%Y/%m/%d/TCPIP
*.SYSLOGD.*.* /var/log/%Y/%m/%d/syslogd
*.err /var/log/%Y/%m/%d/errors

This says all messages for SYSLOGD go to a file like /var/log/2022/05/14/syslogd, and error messages go to /var/log/2022/05/14/errors

This means you get a file of messages for each day. For me, I just used /var/log/syslogd.log and /var/log/errors.log, and deleted them periodically.

I also added a the end of the file, the catchall

*.*.*.* /var/log/all.log

ISPF interface

There is an ISPF syslog browser tool which displays information about the logs, and helps you browse the logs of interest. The documentation for this is not very good.

I got this to work by experimentation. I created an exec like MYSYSLOG

/* Rexx */ 
address ispexec 
"LIBDEF ISPMLIB DATASET ID('TCPIP.SEZAMENU') STACK" 
"LIBDEF ISPPLIB DATASET ID('TCPIP.SEZAPENU') STACK" 
address tso "ALTLIB ACTIVATE APPLICATION(CLIST) 
              DATASET('TCPIP.SEZAEXEC') " 
"SELECT CMD(EZASYRGO) NEWPOOL PASSLIB NEWAPPL(EZAS)" 
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)" 
"LIBDEF ISPPLIB" 
"LIBDEF ISPMLIB"

You can execute this from ISPF option 6 or have this built into the ISPF panels.

Originally this exec was called syslogd; when I used it, I got

SYSTEM COMPLETION CODE=4C5 REASON CODE=77A53217

Where 4C5 is TCPIP, and 3217 – the program has the wrong AC (APF related). This is because there is a command syslogd which was executed in preference to my exec. When I renamed the exec to MYSYSLOG it used the exec and it worked fine!

The first panel is 

EZASYP01 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 1
Command ===>                                                  Scroll ===> PAGE
                                                                               
Enter syslogd browser options                                                  
  Recall migrated data sets ==> NO     (Yes/No) Recall data sets or not        
  Maximum hits to display   ==> 200    (1-99999) Search results to display     
  Maximum file archives     ==> 30     (0-400) Days to look for file archives  
  Display start date/time   ==> YES    (Yes/No) Retrieve start date/time       
  Display active files only ==> NO     (Yes/No) Active files only, no archives 
  DSN Prefix override value ==>                                                
                                                                               
Enter file or data set name of syslogd configuration, or select one from below:
                                                                               
  File/DS Name ==> /etc/syslog.conf
                                                                               
Press ENTER to continue, or press END PF key to exit without a selection       
                                                                               
Line commands: S Select, R Remove from list, B Browse content, E Edit content  
                                                                               
Cmd Recently used syslogd configuration file or data set name                  
--- -------------------------------------------------------------------------- 
    /etc/syslog.conf                                                          
******************************* Bottom of data ********************************

Pressing enter, gave me another panel with

EZASYP00 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 6
OPTION ===>                                                   Scroll ===> PAGE
                                                                               
Select one of the following, or press END PF key to exit the syslogd browser   
                                                                               
  1 Change current syslogd configuration file and/or options                   
  2 Guide me to a possible syslogd destination                                 
  3 Clear guide-me hits (indicated by ==> in the Cmd column)                   
  4 Search across all active syslogd files                                     
                                                                               
Current config file ==> /etc/syslog.conf                                      
                                                                               
Line commands: B Browse, A List archives, S Search active file and archives,   
               SF Search active file, SA Search archives, I File/DSN info      
                                                                    Archive    
Cmd Rule/Active UNIX file name                    Start Time        Type Avail.
--- --------------------------------------------- ----------------- ---- ------
    *.SYSLOGD*.*.*                                28 May 2022 13:31 None 0     
    /var/log/syslogd                                                          
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    *.INETD*.*.*                                  Empty       N/A   None 0     
    /var/log/inetd                                                            
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    auth.*                                        Empty       N/A   None 0     
    /var/log/auth

I could then browse the error log for SYSLOGD.

You can search for userid, strings etc, and give date ranges.

However for my small, one person usage, I found it was easier to use Unix services and use the command

oedit /var/syslogd.log

to edit the file.

Capturing the right data

If the config file you can specify options like 

*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

The entries are Userid.Jobname.facility.priority.

PAGENT AT-TLS request are reported via TCPIP.

I could not find how to filter the TCPIP data so the AT-TLS data went to one file, and other TCPIP data went to another file. For TCPIP it looks like the “facility” is either “daemon” or “auth”, which you can specify in the TTLS configuration. So not very useful.

Getting AT-TLS and PAGENT to work on z/OS – start here.

With traditional TLS applications, the application code has to issue the requests to use TLS, for example specify the keystore, and which cipher specs to use and does the encryption and decryption of the data. The application then issues TCP send and receive request as usual.

With AT-TLS, the TLS work is moved out of the application and into the TCPIP subsystem. The application just does the normal sends and receives, and TCPIP does the work of establishing the session and handling the encryption. There are rules and policies to define how the session should be established. It uses the PAGENT address space (Policy Agent) to manage the configuration.

Is it easier than having MQ or WAS Liberty do the TLS stuff? – I don’t think so. When it works it is fine. Getting it working is a challenge, because the trace and diagnostics are poor.

My other blog posts on PAGENT and AT-TLS

What is PAGENT?

Having used PAGENT to configure AT-TLS with TCPIP, I see PAGENT is a program which reads configuration information from a file – and gives the configuration to TCPIP. TCP then does the work.

General

It feels that the PAGENT setup and configuration was not designed with the z/OS environment in mind. It “breaks” so many things.

  1. You can have only one PAGENT running per LPAR – even with different name. This means you cannot have a “test” and production PAGENT in the same LPAR.
  2. PAGENT can be configured to have information on:
    1. Common Intrusion detection services (IDS).
    2. Common IP filtering, and manual and dynamic virtual private network (VPN) tunnels (IPSEC).
    3. Common Routing (Policy-based routing enables the TCP/IP stack to make routing decisions that take into account criteria other than just the destination IP address. The additional criteria can include job name, source port, destination port, protocol type (TCP or UDP), source IP address, NetAccess security zone, and multilevel secure environment security label).
    4. AT-TLS Common definitions.
    5. AT-TLS for TCPIP Image level which can have sections on
      1. IDS
      2. IPSec
      3. Qos
      4. Routing
      5. AT-TLS.
  3. As there is only one active PAGENT allowed per LPAR, you have to make your configuration changes to the production PAGENT, refresh it, and fix any configuration errors. The documentation says “make a change to production, if it doesn’t work back out the changes”!
  4. There is one initial configuration file per PAGENT, which can “include” other files. You cannot have a concatenated list of files.
  5. You cannot validate definitions before making them active. The configuration is processed only when the referenced TCPIP stack is active.
  6. Error messages do not have error message numbers, so there is no ability to look up the errors messages.
  7. It lacks good diagnostics. For example
    1. I got error message “Resource temporarily unavailable” when it could not find the security profile “EZB.INITSTACK.*.TCPIP2” on my system. The PAGENT code checks to see if the profile exists and if not, it dies quietly. It does not actually use the security profile which would cause RACF to produce a message saying missing profile.
    2. I deliberately misconfigured a file to use a file that does not exist. It just reported …processing_Stmt_TTLSConfig: processing: ‘ TTLSConfig //’USER.Z24C.TCPPARMS(BLAHBLAH)’ . It should report file not found. Some missing files get “Cannot get FILE handle for information.”

My set up

I could not find any good guidance on setting up PAGENT and AT-TLS, so I’ve documented what I did. It may not be correct…

It took about a day to understand the AT-TLS setup – as I was a typical user with typos etc which slowed me down.

Errors

I naively assumed errors would be reported in //SYSPRINT. On my system they were in /tmp/pagent.log. This file location can be configured with an Environment variable.

The output can be verbose, so I use oedit, and ISPF search

f err 15 25

to find the errors. You may find fields SYSERR or OBJERR.

When errors occur, you do not get file and line number of the error. You have to hunt around. Invalid statements are often just ignored.

With a configuration error the PAGENT job gave me a message on syslog

EZZ8438I PAGENT POLICY DEFINITIONS CONTAIN ERRORS FOR TCPIP : TTLS

In the /tmp/pagent.log file I had

05/30 07:21:12 EVENT :005: pinit_fetch_TTLS_policy_profile: Processing Image TTLS config file: ‘//’USER.Z24C.TCPPARMS(TTLS)” for image ‘TCPIP’

05/30 07:21:12 OBJERR :005: process_TTLS_attribute_table: Unknown attribute ‘ZocalAddr’ for TTLSRule

My common mistakes were

  • spelling errors for example TLSConfig instead of TTLSConfig. (I commented, then uncommented a line and lost the initial T)
  • incorrect dataset names, either the data set, or the member.

Configuration concepts

In the PAGENT configuration file, the AT-TLS specific stuff is like

CommonTTLSConfig //’USER.Z24C.TCPPARMS(TTLSCOM)’
tcpImage TCPIP //’USER.Z24C.TCPPARMS(PAGENT)’
TcpImage TCPIP2 //’USER.Z24C.TCPPARMS(PAGENTT2)’

This defines common stuff for AT-TLS in //’USER.Z24C.TCPPARMS(TTLSCOM)’, and specific TCPIP image in its own file.

The TCPIP specific file has

TTLSConfig //’USER.Z24C.TCPPARMS(TTLS2)’ FLUSH PURGE

This says the TTLS stuff is in the member TTLS2.

You can have the entry without a file or dataset name.

TTLSConfig FLUSH PURGE

This says use the definition in the CommonTTLSConfig.

You need a TTLSConfig, statement, to get AT-TLS definitions configured in the LPAR.

How to update definitions

So I did not break “production” I created a second TCPIP stack (TCPIP2), and created a configuration within PAGENT for the TCPIP2 stack. (This seems a lot of work just to validate some definitions. I raised an RFE on this, but it was declined).

When I was happy with the definitions, I merged them with the the common/production ones.

When I defined a second TCPIP (TCPIP2), the configuration statements were only parsed, when TCPIP2 was started, and so PAGENT produced the error messages once TCPIP2 was active

PAGENT has started – what next?

Pagent operator commands

You can “modify” the PAGENT address space

For example

What is my configuration?

Once you have configured PAGENT you can use the Unix command

pasearch -c 1>a

to give output like

TCP/IP pasearch CS V2R4 Image Name: TCPIP1
Date: 05/23/2022 Time: 17:34:44
PAPI Version: 14 DLL Version: 14
TTLS Policy Object:
ConfigLocation: Local LDAPServer: False
CommonFileName: //'USER.Z24C.TCPPARMS(TTLSCOM)'
ImageFileName: 


TCP/IP pasearch CS V2R4 Image Name: TCPIP2
Date: 05/23/2022 Time: 17:34:44
PAPI Version: 14 DLL Version: 14
TTLS Policy Object:
ConfigLocation: Local LDAPServer: False
CommonFileName: //'USER.Z24C.TCPPARMS(TTLSCOM)'
ImageFileName: //'USER.Z24C.TCPPARMS(TTLS2)'

The command

pasearch -p TCPIP2 1>a

gave the configuration for just the TCPIP stack TCPIP2, including

...
policyRule:             TLSCOM 
  Rule Type:            TTLS 
...
policyRule:             TLSCP3 
  Rule Type:            TTLS 
...
policyRule:             TLSCP4 
  Rule Type:            TTLS 
...

You get the definitions – but you do not know where they came from. I happen to know that TLSCOM comes from the common definition.

A definition can be in both Common and TCPIP Image files.

Instead of relying on PAGENT to report configuration errors I used the Unix command pasearch to display the configuration.

Display the configuration for a TCPIP image

Use the Unix command pasearch to display the configuration.

pasearch -p TCPIP2 >a

Display the object types configured to PAGENT

pasearch -c 1>a

TCP/IP pasearch CS V2R4 Image Name: TCPIP
Qos Policy Object:…
Ids Policy Object:…
IPSec Policy Object:…
IpFilter Policy Object:…
KeyExchange Policy Object:…
LocalDynVpn Policy Object:…
Routing Policy Object:…
TTLS Policy Object:…

TCP/IP pasearch CS V2R4 Image Name: TCPIP2
…

For example 

TTLS Policy Object:
ConfigLocation: Local LDAPServer: False
CommonFileName: //'ADCD.Z24C.TCPPARMS(TLSPOLY1)'
ImageFileName: //'ADCD.Z24C.TCPPARMS(TLSPOLY1)'
ApplyFlush: True PolicyFlush: True
ApplyPurge: True PurgePolicies: True
AtomicParse: True DeleteOnNoflush: False
DummyOnEmptyPolicy: True ModifyOnIDChange: False
Configured: True UpdateInterval: 1800
TTLS Enabled: True
InstanceId: 1653375346
LastPolicyChanged: Tue May 24 07:55:46 2022

Overall

PAGENT feels like it not of the standard that I would expect z/OS products to have. For example, you cannot validate changes before making them live, and the changes are only validated when the TCPIP stack is active.

This means you are making unvalidated changes to your production system!