Many components of TCPIP write information to syslogd. This is a process that captures the data sent to it over a UDP socket, and writes it to files in the Unix file system. If the syslogd is not active then messages may be written to the job log. When I was trying to set up AT-TLS, I had 10s of messages on the job log, each time a client tried to use AT-TLS.
The IBM documentation is not very clear, it tells you how to turn on debug, trace etc but does not clearly explain the difference, and when they are used.
It look like the PAGENT job is to take a configuration file, parse it, and pass the data to TCPIP.
If you are using AT-TLS to set up TLS channels, the trace data comes from the TCPIP address space.
Modify the PAGENT address space.
You can pass commands to the PAGENT address space.
Configuration processing.
You can control how much information is logged when parsing configuration statements. The value 127 covers most levels of information (including warnings).
F PAGENT,LOGLEVEL,LEVEL=127
You can use
F PAGENT,TRACE,LEVEL=..
F PAGENT,DEBUG,LEVEL=…
But these do not seem to control the level of trace produced.
Trace PAGENT startup and parse of the configuration
To collect the PAGENT startup and display information on the configuration file as it is processed change the started task JCL to include the -d option.
//PAGENT EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT,
// PARM=’ENVAR(“_CEE_ENVFILE_S=DD:STDENV”)/ -d 4‘
By default the output trace goes to /tmp/pagent.log. It has content like
05/29 17:17:54 EVENT :005: pzos_install_A_PolicyRule: Finished installing policy rule: ‘REMOTE-TO-CSQ1’
Trace PAGENT use of TLS
My PAGENT JCL is
//CPAGENT PROC
// SET EN='ENVAR("_CEE_ENVFILE_S=DD:STDENV")'
//PAGENT EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT,
// PARM='&EN/ -d 4'
//STDENV DD DISP=SHR,DSN=USER.Z24C.TCPPARMS(PAGENTEN)
//SYSPRINT DD SYSOUT=H
//SYSERR DD SYSOUT=H
//SYSOUT DD SYSOUT=H
//*
//CEEDUMP DD SYSOUT=*,DCB=(RECFM=FB,LRECL=132,BLKSIZE=132)
With the environment file USER.Z24C.TCPPARMS(PAGENTEN) having
_CEE_ENVFILE_COMMENT=# PAGENT_CONFIG_FILE=//'USER.Z24C.TCPPARMS(PAGENTCF)' LIBPATH=/usr/lib GSK_TRACE=0x00 GSK_TRACE_FILE=/var/log/GSK
You can collect the GSK calls made by PAGENT at startup by using the environment variables
GSK_TRACE=0xFF
GSK_TRACE_FILE=/var/log/GSK
Note: This turns it on for all requests! I could not find how to do selective tracing.
You have to format the trace file using
gsktrace /var/log/GSK /var/log/GSK.txt
This has about 40 lines with information like
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-1 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-224 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-256 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-384 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-512 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES3 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 128-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 256-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES-GCM crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic accelerator is not available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic coprocessor is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Public key hardware support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Max RSA key sizes in hardware – signature 4096, encryption 4096,
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Maximum RSA token size 3500
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC clear key support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC secure key support is available. Maximum key size 521
Remember this is the PAGENT invoking GSK – but PAGENT does not do any TLS work – this is done by TCPIP.
Trace an AT-TLS connection.
You need to enable trace in the AT-TLS configuration for example
TTLSEnvironmentAction CSQ1-INBOUND-ENVIRONMENT-ACTION
{
HandshakeRole SERVER
TTLSKeyringParmsRef CSQ1-KEYRING
TTLSCipherParmsRef CSQ1-CIPHERPARM
TTLSEnvironmentAdvancedParmsRef CSQ1-ENVIRONMENT-ADVANCED
Trace 255
}
If the syslogd daemon is not configured the output goes to the TCPIP job log.
If the syslogd daemon is configured, you need to have a syslogd configuration with
*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all
Where TCPIP is the TCPIP address space name, and *.*.*.* is a catch-all. It took me about a day to realize that my trace was being thrown away because I didn’t have TCPIP, and the catch-all.
The trace file has data like
May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1284I TTLS Flow GRPID: 00000021 ENVID: 00000009 CONNID: 00000053 RC: 0 Set GSK_USER_DATA(200) – 000000007F280610
May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1285I TTLS Data CONNID: 00000053 RECV CIPHER 160303007B
This shows a GSK call was made to GSK_USER_DATA which completed with return code 0, and the connection RECeiVed data which was the CIPHER specs ( 4 chars or 2 chars) 160303007b.
You need to configure the syslogd procedure.
See if SYSLOGD is running, if not, try to start it. If it does not exist…
- Copy /usr/lpp/tcpip/samples/syslog.conf to its default configuration file /etc/syslog.conf, or another file.
- Copy TCPIP.SEZAINST(SYSLOGD) to your proclib concatenation.
- The program uses environment variables defined in STDENV to control operations. The default configuration file location is /etc/syslog.conf
You can configure syslog.conf for example
*.TCPIP.*.* /var/log/%Y/%m/%d/TCPIP
*.SYSLOGD.*.* /var/log/%Y/%m/%d/syslogd
*.err /var/log/%Y/%m/%d/errors
This says all messages for SYSLOGD go to a file like /var/log/2022/05/14/syslogd, and error messages go to /var/log/2022/05/14/errors
This means you get a file of messages for each day. For me, I just used /var/log/syslogd.log and /var/log/errors.log, and deleted them periodically.
I also added a the end of the file, the catchall
*.*.*.* /var/log/all.log
ISPF interface
There is an ISPF syslog browser tool which displays information about the logs, and helps you browse the logs of interest. The documentation for this is not very good.
I got this to work by experimentation. I created an exec like MYSYSLOG
/* Rexx */
address ispexec
"LIBDEF ISPMLIB DATASET ID('TCPIP.SEZAMENU') STACK"
"LIBDEF ISPPLIB DATASET ID('TCPIP.SEZAPENU') STACK"
address tso "ALTLIB ACTIVATE APPLICATION(CLIST)
DATASET('TCPIP.SEZAEXEC') "
"SELECT CMD(EZASYRGO) NEWPOOL PASSLIB NEWAPPL(EZAS)"
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)"
"LIBDEF ISPPLIB"
"LIBDEF ISPMLIB"
You can execute this from ISPF option 6 or have this built into the ISPF panels.
Originally this exec was called syslogd; when I used it, I got
SYSTEM COMPLETION CODE=4C5 REASON CODE=77A53217
Where 4C5 is TCPIP, and 3217 – the program has the wrong AC (APF related). This is because there is a command syslogd which was executed in preference to my exec. When I renamed the exec to MYSYSLOG it used the exec and it worked fine!
The first panel is
EZASYP01 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 1
Command ===> Scroll ===> PAGE
Enter syslogd browser options
Recall migrated data sets ==> NO (Yes/No) Recall data sets or not
Maximum hits to display ==> 200 (1-99999) Search results to display
Maximum file archives ==> 30 (0-400) Days to look for file archives
Display start date/time ==> YES (Yes/No) Retrieve start date/time
Display active files only ==> NO (Yes/No) Active files only, no archives
DSN Prefix override value ==>
Enter file or data set name of syslogd configuration, or select one from below:
File/DS Name ==> /etc/syslog.conf
Press ENTER to continue, or press END PF key to exit without a selection
Line commands: S Select, R Remove from list, B Browse content, E Edit content
Cmd Recently used syslogd configuration file or data set name
--- --------------------------------------------------------------------------
/etc/syslog.conf
******************************* Bottom of data ********************************
Pressing enter, gave me another panel with
EZASYP00 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 6
OPTION ===> Scroll ===> PAGE
Select one of the following, or press END PF key to exit the syslogd browser
1 Change current syslogd configuration file and/or options
2 Guide me to a possible syslogd destination
3 Clear guide-me hits (indicated by ==> in the Cmd column)
4 Search across all active syslogd files
Current config file ==> /etc/syslog.conf
Line commands: B Browse, A List archives, S Search active file and archives,
SF Search active file, SA Search archives, I File/DSN info
Archive
Cmd Rule/Active UNIX file name Start Time Type Avail.
--- --------------------------------------------- ----------------- ---- ------
*.SYSLOGD*.*.* 28 May 2022 13:31 None 0
/var/log/syslogd
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
*.INETD*.*.* Empty N/A None 0
/var/log/inetd
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
auth.* Empty N/A None 0
/var/log/auth
I could then browse the error log for SYSLOGD.
You can search for userid, strings etc, and give date ranges.
However for my small, one person usage, I found it was easier to use Unix services and use the command
oedit /var/syslogd.log
to edit the file.
Capturing the right data
If the config file you can specify options like
*.TCPIP.*.* /var/log/TCPIP *.*.*. /var/log/all
The entries are Userid.Jobname.facility.priority.
PAGENT AT-TLS request are reported via TCPIP.
I could not find how to filter the TCPIP data so the AT-TLS data went to one file, and other TCPIP data went to another file. For TCPIP it looks like the “facility” is either “daemon” or “auth”, which you can specify in the TTLS configuration. So not very useful.
ColinPaice 
One thought on “Trace PAGENT and AT-TLS”