Trace PAGENT and AT-TLS

Many components of TCPIP write information to syslogd. This is a process that captures the data sent to it over a UDP socket, and writes it to files in the Unix file system. If the syslogd is not active then messages may be written to the job log. When I was trying to set up AT-TLS, I had 10s of messages on the job log, each time a client tried to use AT-TLS.

The IBM documentation is not very clear, it tells you how to turn on debug, trace etc but does not clearly explain the difference, and when they are used.

It look like the PAGENT job is to take a configuration file, parse it, and pass the data to TCPIP.

If you are using AT-TLS to set up TLS channels, the trace data comes from the TCPIP address space.

Modify the PAGENT address space.

You can pass commands to the PAGENT address space.

Configuration processing.

You can control how much information is logged when parsing configuration statements. The value 127 covers most levels of information (including warnings).

F PAGENT,LOGLEVEL,LEVEL=127

You can use

F PAGENT,TRACE,LEVEL=..
F PAGENT,DEBUG,LEVEL=…

But these do not seem to control the level of trace produced.

Trace PAGENT startup and parse of the configuration

To collect the PAGENT startup and display information on the configuration file as it is processed change the started task JCL to include the -d option.


//PAGENT EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT,
// PARM=’ENVAR(“_CEE_ENVFILE_S=DD:STDENV”)/ -d 4

By default the output trace goes to /tmp/pagent.log. It has content like

05/29 17:17:54 EVENT :005: pzos_install_A_PolicyRule: Finished installing policy rule: ‘REMOTE-TO-CSQ1’

Trace PAGENT use of TLS

My PAGENT JCL is

//CPAGENT  PROC 
//  SET EN='ENVAR("_CEE_ENVFILE_S=DD:STDENV")' 
//PAGENT   EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT, 
//       PARM='&EN/                                      -d 4' 
//STDENV   DD DISP=SHR,DSN=USER.Z24C.TCPPARMS(PAGENTEN) 
//SYSPRINT DD SYSOUT=H 
//SYSERR   DD SYSOUT=H 
//SYSOUT   DD SYSOUT=H 
//* 
//CEEDUMP  DD SYSOUT=*,DCB=(RECFM=FB,LRECL=132,BLKSIZE=132) 

With the environment file USER.Z24C.TCPPARMS(PAGENTEN) having

_CEE_ENVFILE_COMMENT=# 
PAGENT_CONFIG_FILE=//'USER.Z24C.TCPPARMS(PAGENTCF)' 
LIBPATH=/usr/lib 
GSK_TRACE=0x00
GSK_TRACE_FILE=/var/log/GSK 

You can collect the GSK calls made by PAGENT at startup by using the environment variables

GSK_TRACE=0xFF
GSK_TRACE_FILE=/var/log/GSK

Note: This turns it on for all requests! I could not find how to do selective tracing.

You have to format the trace file using

gsktrace /var/log/GSK /var/log/GSK.txt

This has about 40 lines with information like

05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-1 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-224 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-256 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-384 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-512 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES3 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 128-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 256-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES-GCM crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic accelerator is not available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic coprocessor is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Public key hardware support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Max RSA key sizes in hardware – signature 4096, encryption 4096,
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Maximum RSA token size 3500
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC clear key support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC secure key support is available. Maximum key size 521

Remember this is the PAGENT invoking GSK – but PAGENT does not do any TLS work – this is done by TCPIP.

Trace an AT-TLS connection.

You need to enable trace in the AT-TLS configuration for example

TTLSEnvironmentAction CSQ1-INBOUND-ENVIRONMENT-ACTION
{
HandshakeRole SERVER
TTLSKeyringParmsRef CSQ1-KEYRING
TTLSCipherParmsRef CSQ1-CIPHERPARM
TTLSEnvironmentAdvancedParmsRef CSQ1-ENVIRONMENT-ADVANCED
Trace 255
}

If the syslogd daemon is not configured the output goes to the TCPIP job log.

If the syslogd daemon is configured, you need to have a syslogd configuration with


*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

Where TCPIP is the TCPIP address space name, and *.*.*.* is a catch-all. It took me about a day to realize that my trace was being thrown away because I didn’t have TCPIP, and the catch-all.

The trace file has data like

May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1284I TTLS Flow GRPID: 00000021 ENVID: 00000009 CONNID: 00000053 RC: 0 Set GSK_USER_DATA(200) – 000000007F280610

May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1285I TTLS Data CONNID: 00000053 RECV CIPHER 160303007B

This shows a GSK call was made to GSK_USER_DATA which completed with return code 0, and the connection RECeiVed data which was the CIPHER specs ( 4 chars or 2 chars) 160303007b.

You need to configure the syslogd procedure.

See if SYSLOGD is running, if not, try to start it. If it does not exist…

  • Copy /usr/lpp/tcpip/samples/syslog.conf to its default configuration file /etc/syslog.conf, or another file.
  • Copy TCPIP.SEZAINST(SYSLOGD) to your proclib concatenation.
  • The program uses environment variables defined in STDENV to control operations. The default configuration file location is /etc/syslog.conf

You can configure syslog.conf for example

*.TCPIP.*.* /var/log/%Y/%m/%d/TCPIP
*.SYSLOGD.*.* /var/log/%Y/%m/%d/syslogd
*.err /var/log/%Y/%m/%d/errors

This says all messages for SYSLOGD go to a file like /var/log/2022/05/14/syslogd, and error messages go to /var/log/2022/05/14/errors

This means you get a file of messages for each day. For me, I just used /var/log/syslogd.log and /var/log/errors.log, and deleted them periodically.

I also added a the end of the file, the catchall

*.*.*.* /var/log/all.log

ISPF interface

There is an ISPF syslog browser tool which displays information about the logs, and helps you browse the logs of interest. The documentation for this is not very good.

I got this to work by experimentation. I created an exec like MYSYSLOG

/* Rexx */ 
address ispexec 
"LIBDEF ISPMLIB DATASET ID('TCPIP.SEZAMENU') STACK" 
"LIBDEF ISPPLIB DATASET ID('TCPIP.SEZAPENU') STACK" 
address tso "ALTLIB ACTIVATE APPLICATION(CLIST) 
              DATASET('TCPIP.SEZAEXEC') " 
"SELECT CMD(EZASYRGO) NEWPOOL PASSLIB NEWAPPL(EZAS)" 
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)" 
"LIBDEF ISPPLIB" 
"LIBDEF ISPMLIB" 

You can execute this from ISPF option 6 or have this built into the ISPF panels.

Originally this exec was called syslogd; when I used it, I got

SYSTEM COMPLETION CODE=4C5 REASON CODE=77A53217

Where 4C5 is TCPIP, and 3217 – the program has the wrong AC (APF related). This is because there is a command syslogd which was executed in preference to my exec. When I renamed the exec to MYSYSLOG it used the exec and it worked fine!

The first panel is

EZASYP01 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 1
Command ===>                                                  Scroll ===> PAGE
                                                                               
Enter syslogd browser options                                                  
  Recall migrated data sets ==> NO     (Yes/No) Recall data sets or not        
  Maximum hits to display   ==> 200    (1-99999) Search results to display     
  Maximum file archives     ==> 30     (0-400) Days to look for file archives  
  Display start date/time   ==> YES    (Yes/No) Retrieve start date/time       
  Display active files only ==> NO     (Yes/No) Active files only, no archives 
  DSN Prefix override value ==>                                                
                                                                               
Enter file or data set name of syslogd configuration, or select one from below:
                                                                               
  File/DS Name ==> /etc/syslog.conf
                                                                               
Press ENTER to continue, or press END PF key to exit without a selection       
                                                                               
Line commands: S Select, R Remove from list, B Browse content, E Edit content  
                                                                               
Cmd Recently used syslogd configuration file or data set name                  
--- -------------------------------------------------------------------------- 
    /etc/syslog.conf                                                          
******************************* Bottom of data ********************************

Pressing enter, gave me another panel with

EZASYP00 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 6
OPTION ===>                                                   Scroll ===> PAGE
                                                                               
Select one of the following, or press END PF key to exit the syslogd browser   
                                                                               
  1 Change current syslogd configuration file and/or options                   
  2 Guide me to a possible syslogd destination                                 
  3 Clear guide-me hits (indicated by ==> in the Cmd column)                   
  4 Search across all active syslogd files                                     
                                                                               
Current config file ==> /etc/syslog.conf                                      
                                                                               
Line commands: B Browse, A List archives, S Search active file and archives,   
               SF Search active file, SA Search archives, I File/DSN info      
                                                                    Archive    
Cmd Rule/Active UNIX file name                    Start Time        Type Avail.
--- --------------------------------------------- ----------------- ---- ------
    *.SYSLOGD*.*.*                                28 May 2022 13:31 None 0     
    /var/log/syslogd                                                          
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    *.INETD*.*.*                                  Empty       N/A   None 0     
    /var/log/inetd                                                            
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    auth.*                                        Empty       N/A   None 0     
    /var/log/auth                                                             

I could then browse the error log for SYSLOGD.

You can search for userid, strings etc, and give date ranges.

However for my small, one person usage, I found it was easier to use Unix services and use the command

oedit /var/syslogd.log

to edit the file.

Capturing the right data

If the config file you can specify options like

*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

The entries are Userid.Jobname.facility.priority.

PAGENT AT-TLS request are reported via TCPIP.

I could not find how to filter the TCPIP data so the AT-TLS data went to one file, and other TCPIP data went to another file. For TCPIP it looks like the “facility” is either “daemon” or “auth”, which you can specify in the TTLS configuration. So not very useful.

One thought on “Trace PAGENT and AT-TLS

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s