Using z/OSMF workflows for TCPIP.

I found it hard to set up the AT-TLS configuration for MQ. The easiest way was to use the sample configurations provided by MQ. See here for an overview. I used Scenario 5 – Between an IBM MQ for z/OS queue manager and a client application running on IBM MQ for Multiplatforms.

This took about 10 minutes once I had PAGENT and SYSLOGD set up.

I thought I would try to use the TCP provided facilities. There is a lot of documentation, but it is not easy to find what you need. It has been written as an IBM developer, rather than from an end user perspective.

I then thought I would try to use the “way of the future” and the z/OS configuration tool z/OSMF. You use a browser to connect to z/OSMF and do your work through the browser interface. The z/OSMF interface has configuration tools, and workflow tools which guide you step by step through configuration.

I found using the workflow tools was harder than using the TCPIP documentation and TCPIP samples, and I would not recommend its use.

Ive blogged Using z/OSMF Network Configuration assistant for TCPIP, to define AT-TLS configuration. Which worked.

The workflow stuff makes the easy bit “easier”, but does not help with the hard stuff. An improvement would be to skip the workflow, and have one page of instructions saying copy samples into Proclib, and Unix; run a RACF job. We could do with a workflow to help configure syslogd, which I had a struggle to get working in a non trivial situation. For example having error messages for PAGENT go to one file, and have the TLS trace go into another file.

My mission.

My mission was to configure AT-TLS and to provide two ports for use with MQ.

I wanted to do this using two people (me with two userids) and do the typical steps when changing systems, such as saving configurations before changing them, and deploying them, when I had a “change window”.

Initial steps

z/OSMF provides facilities like ISPF, Workload management configuration, system status etc. I used Workflow.

It was hard to know where to start. I assumed (wrongly) that there would be a workflow to define the AT-TLS definitions.

It seems you use Workflow to define the PAGENT and syslogd JCL, and not for configuring the PAGENT or syslogd.

Instructions to use Workflow to configure TCPIP JCL procedures

  • Double click the workflow icon.
  • From the actions pull down, select Create workflow…
  • You need to select Workflow definition file: I could not find what I had to specify. There was no prompting. The “?” basically said “put a value here”. The help key just gave me a panel with information about using creating a workflow.
  • I found an IBM support document which says
    • Workflows for Policy-based Networking
    • ezb_pagent_setup_wizard.xml – This workflow provides the steps for setting up the Policy Agent (Pagent). Pagent is required for all of the policy-based networking technologies: IPSec, AT-TLS, IDS, PBR, and QoS. Pagent uses syslogd for logging.
    • ezb_syslogd_setup_wizard.xml – This workflow provides the steps for setting up syslogd.
    • ezb_tcpip_profile_sample_wizard.xml – This workflow provides a sample TCP/IP profile which contains common statements required to enable AT-TLS and IP Security, and additionally includes port reservation statements for running daemons.
  • I had to use the fully qualified filename /usr/lpp/zosmf/workflow/plugins/izuca/ezb_syslogd_setup_wizard.xml
  • This came up with an error in the workflow name because the default name has ‘z/OS… ‘ and ‘/’ is not a valid character. I removed the ‘/’.
  • At the bottom of the page you can Assign all steps to owner user id. I did not do this, and had to assign steps below
  • You get a list of steps that need to be done.
  • Assign the work to a userid
    • Select all of the steps, and use Actions-> Assignment and ownership -> Add assignees.
    • This displays the assigned roles. I used Actions -> add to add my SAF userid. I pressed OK and returned to the list of steps – all now assigned to me.
  • I selected the first step “define the “RACF userid for Syslogd”, Actions -> Accept .
  • Click on the task, and it gives you a window with tabs. The important tab is Perform. If this is greyed out, you have not accepted the task!
    • Fill in the details and click Next, Next etc. You can edit the contents.
    • You can save it – but you need to give a data set. It suggested SYS1(SYSLOGD). I had to change it (every time) to COLIN.ATTLS(…)
    • Next – gives you the save panel. You have to specify the dataset where you want to save it. The default was wrong for me.
    • Once saved you have to submit it manually, check the output, and edit the file if needed.
  • Back at the workflow details, it had step 1 complete (even though you may not have submitted it)
  • I accepted step 2 and started working on it.
    • It asks for Dataset HLQ – but I could not change it.
    • I stepped through the definitions – and had to type in my dataset again (why can’t it remember what I specified last time).
    • This step just creates a job with some RACF definitions in it.
  • I ran step 3 -again just creating a JCL member of definitions
  • Step 4 “Sample Syslogd Configuration Setup“. This just copies in a sample configuration.
    • “Save” did not do anything
  • Step 5 “Sample started procedure for Syslogd” creates a sample Procedure.
  • On the workflows page, it shows the workflow is 100% complete.

Having been through all of this, the create JCL did not run, one line in error was

//SYSLOGD PROC PROG=”,
// VARS=”,
// PARMS=”
//SYSLOGD EXEC PGM=&PROG., REGION=0K,TIME=NOLIMIT,
// PARM=(‘POSIX(ON) ALL31(ON)’,
// ‘ENVAR(“_CEE_ENVFILE=DD:VARS”)’,
// ‘/&PARMS.’)

  • &PROG had not been specified – you gave to go and find what you need to specified (SYSLOGD)
  • There is a blank after the &PROG., so the REGION=0K,TIME=NOLIMIT, is ignored
  • The location of the configuration (in &VARs) is not specified.

Create the PAGENT JCL

I followed the same process to create the PAGENT file.

I used file /usr/lpp/zosmf/workflow/plugins/izuca/ezb_pagent_setup_wizard.xml.

When this JCL ran, it produced messages

06/16 08:00:20 SYSERR :000: …plfm_config_medium_open: cannot open ‘/etc/pagent.conf’, errno EDC5129I No such file or directory.

You have to know to copy the configuration file from the PDS to /etc/pagent.conf.

Comments on using the workflows

This seems a lot of work to produce code which does not work. The process feels unloved. I am surprised that the problems I found have not been fixed – they are Unit Test level bugs.

I think it is far simpler to follow the documentation, for example to create the procedure. The documentation says

Update the cataloged procedure, syslogd, by copying the sample in SEZAINST(SYSLOGD) to your system or recognized PROCLIB. Specify syslogd parameters and change the data set names to suit your local configurtion See the syslog daemon section of SEZAINST(EZARACF) for SAF considerations for started procedures

The instructions could be on one side of paper, and would be quicker than using the workflow.

Trace PAGENT and AT-TLS

Many components of TCPIP write information to syslogd. This is a process that captures the data sent to it over a UDP socket, and writes it to files in the Unix file system. If the syslogd is not active then messages may be written to the job log. When I was trying to set up AT-TLS, I had 10s of messages on the job log, each time a client tried to use AT-TLS.

The IBM documentation is not very clear, it tells you how to turn on debug, trace etc but does not clearly explain the difference, and when they are used.

It look like the PAGENT job is to take a configuration file, parse it, and pass the data to TCPIP.

If you are using AT-TLS to set up TLS channels, the trace data comes from the TCPIP address space.

Modify the PAGENT address space.

You can pass commands to the PAGENT address space.

Configuration processing.

You can control how much information is logged when parsing configuration statements. The value 127 covers most levels of information (including warnings).

F PAGENT,LOGLEVEL,LEVEL=127

You can use

F PAGENT,TRACE,LEVEL=..
F PAGENT,DEBUG,LEVEL=…

But these do not seem to control the level of trace produced.

Trace PAGENT startup and parse of the configuration

To collect the PAGENT startup and display information on the configuration file as it is processed change the started task JCL to include the -d option.


//PAGENT EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT,
// PARM=’ENVAR(“_CEE_ENVFILE_S=DD:STDENV”)/ -d 4

By default the output trace goes to /tmp/pagent.log. It has content like

05/29 17:17:54 EVENT :005: pzos_install_A_PolicyRule: Finished installing policy rule: ‘REMOTE-TO-CSQ1’

Trace PAGENT use of TLS

You can collect the GSK calls made by PAGENT at startup by using the environment variables

GSK_TRACE=0xFF
GSK_TRACE_FILE=/var/log/GSK

You have to format the trace file using

gsktrace /var/log/GSK /var/log/GSK.txt

This has about 40 lines with information like

05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-1 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-224 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-256 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-384 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-512 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES3 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 128-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 256-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES-GCM crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic accelerator is not available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic coprocessor is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Public key hardware support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Max RSA key sizes in hardware – signature 4096, encryption 4096,
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Maximum RSA token size 3500
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC clear key support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC secure key support is available. Maximum key size 521

Remember this is the PAGENT invoking GSK – but PAGENT does not do any TLS work – this is done by TCPIP.

Trace an AT-TLS connection.

You need to enable trace in the AT-TLS configuration for example

TTLSEnvironmentAction CSQ1-INBOUND-ENVIRONMENT-ACTION
{
HandshakeRole SERVER
TTLSKeyringParmsRef CSQ1-KEYRING
TTLSCipherParmsRef CSQ1-CIPHERPARM
TTLSEnvironmentAdvancedParmsRef CSQ1-ENVIRONMENT-ADVANCED
Trace 255
}

If the syslogd daemon is not configured the output goes to the TCPIP job log.

If the syslogd daemon is configured, you need to have a syslogd configuration with


*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

Where TCPIP is the TCPIP address space name, and *.*.*.* is a catch-all. It took me about a day to realize that my trace was being thrown away because I didn’t have TCPIP, and the catch-all.

The trace file has data like

May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1284I TTLS Flow GRPID: 00000021 ENVID: 00000009 CONNID: 00000053 RC: 0 Set GSK_USER_DATA(200) – 000000007F280610

May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1285I TTLS Data CONNID: 00000053 RECV CIPHER 160303007B

This shows a GSK call was made to GSK_USER_DATA which completed with return code 0, and the connection RECeiVed data which was the CIPHER specs ( 4 chars or 2 chars) 160303007b.

You need to configure the syslogd procedure.

See if SYSLOGD is running, if not, try to start it. If it does not exist…

  • Copy /usr/lpp/tcpip/samples/syslog.conf to its default configuration file /etc/syslog.conf, or another file.
  • Copy TCPIP.SEZAINST(SYSLOGD) to your proclib concatenation.
  • The program uses environment variables defined in STDENV to control operations. The default configuration file location is /etc/syslog.conf

You can configure syslog.conf for example

*.TCPIP.*.* /var/log/%Y/%m/%d/TCPIP
*.SYSLOGD.*.* /var/log/%Y/%m/%d/syslogd
*.err /var/log/%Y/%m/%d/errors

This says all messages for SYSLOGD go to a file like /var/log/2022/05/14/syslogd, and error messages go to /var/log/2022/05/14/errors

This means you get a file of messages for each day. For me, I just used /var/log/syslogd.log and /var/log/errors.log, and deleted them periodically.

I also added a the end of the file, the catchall

*.*.*.* /var/log/all.log

ISPF interface

There is an ISPF syslog browser tool which displays information about the logs, and helps you browse the logs of interest. The documentation for this is not very good.

I got this to work by experimentation. I created an exec like MYSYSLOG

/* Rexx */ 
address ispexec 
"LIBDEF ISPMLIB DATASET ID('TCPIP.SEZAMENU') STACK" 
"LIBDEF ISPPLIB DATASET ID('TCPIP.SEZAPENU') STACK" 
address tso "ALTLIB ACTIVATE APPLICATION(CLIST) 
              DATASET('TCPIP.SEZAEXEC') " 
"SELECT CMD(EZASYRGO) NEWPOOL PASSLIB NEWAPPL(EZAS)" 
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)" 
"LIBDEF ISPPLIB" 
"LIBDEF ISPMLIB" 

You can execute this from ISPF option 6 or have this built into the ISPF panels.

Originally this exec was called syslogd; when I used it, I got

SYSTEM COMPLETION CODE=4C5 REASON CODE=77A53217

Where 4C5 is TCPIP, and 3217 – the program has the wrong AC (APF related). This is because there is a command syslogd which was executed in preference to my exec. When I renamed the exec to MYSYSLOG it used the exec and it worked fine!

The first panel is

EZASYP01 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 1
Command ===>                                                  Scroll ===> PAGE
                                                                               
Enter syslogd browser options                                                  
  Recall migrated data sets ==> NO     (Yes/No) Recall data sets or not        
  Maximum hits to display   ==> 200    (1-99999) Search results to display     
  Maximum file archives     ==> 30     (0-400) Days to look for file archives  
  Display start date/time   ==> YES    (Yes/No) Retrieve start date/time       
  Display active files only ==> NO     (Yes/No) Active files only, no archives 
  DSN Prefix override value ==>                                                
                                                                               
Enter file or data set name of syslogd configuration, or select one from below:
                                                                               
  File/DS Name ==> /etc/syslog.conf
                                                                               
Press ENTER to continue, or press END PF key to exit without a selection       
                                                                               
Line commands: S Select, R Remove from list, B Browse content, E Edit content  
                                                                               
Cmd Recently used syslogd configuration file or data set name                  
--- -------------------------------------------------------------------------- 
    /etc/syslog.conf                                                          
******************************* Bottom of data ********************************

Pressing enter, gave me another panel with

EZASYP00 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 6
OPTION ===>                                                   Scroll ===> PAGE
                                                                               
Select one of the following, or press END PF key to exit the syslogd browser   
                                                                               
  1 Change current syslogd configuration file and/or options                   
  2 Guide me to a possible syslogd destination                                 
  3 Clear guide-me hits (indicated by ==> in the Cmd column)                   
  4 Search across all active syslogd files                                     
                                                                               
Current config file ==> /etc/syslog.conf                                      
                                                                               
Line commands: B Browse, A List archives, S Search active file and archives,   
               SF Search active file, SA Search archives, I File/DSN info      
                                                                    Archive    
Cmd Rule/Active UNIX file name                    Start Time        Type Avail.
--- --------------------------------------------- ----------------- ---- ------
    *.SYSLOGD*.*.*                                28 May 2022 13:31 None 0     
    /var/log/syslogd                                                          
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    *.INETD*.*.*                                  Empty       N/A   None 0     
    /var/log/inetd                                                            
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    auth.*                                        Empty       N/A   None 0     
    /var/log/auth                                                             

I could then browse the error log for SYSLOGD.

You can search for userid, strings etc, and give date ranges.

However for my small, one person usage, I found it was easier to use Unix services and use the command

oedit /var/syslogd.log

to edit the file.

Capturing the right data

If the config file you can specify options like

*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

The entries are Userid.Jobname.facility.priority.

PAGENT AT-TLS request are reported via TCPIP.

I could not find how to filter the TCPIP data so the AT-TLS data went to one file, and other TCPIP data went to another file. For TCPIP it looks like the “facility” is either “daemon” or “auth”, which you can specify in the TTLS configuration. So not very useful.

Setting up syslogd on z/OS

Many components of TCPIP write information to syslogd. This is a process that captures the data sent to it over a UDP socket, and writes it to files in the Unix file system. If the syslogd is not active then messages may be written to the job log. When I was trying to set up AT-TLS, I had 10s of messages on the job log, each time a client tried to use AT-TLS.

The IBM documentation is not very clear, it tells you how to turn on debug, trace etc but does not clearly explain the difference, and when they are used.

It look like the PAGENT’s job is to take a configuration file, parse it, and pass the configuration data to TCPIP.

If you are using AT-TLS to set up TLS channels, the trace data comes from the TCPIP address space into syslogd.

Configure syslogd

See if SYSLOGD is running, if not, try to start it. If it does not exist…

  • Copy /usr/lpp/tcpip/samples/syslog.conf to its default configuration file /etc/syslog.conf, or another file.
  • Copy TCPIP.SEZAINST(SYSLOGD) to your proclib concatenation.
  • The program uses environment variables defined in STDENV to control operations. The default configuration file location is /etc/syslog.conf .

You can configure syslog.conf for example

*.TCPIP.*.* /var/log/%Y/%m/%d/TCPIP
*.SYSLOGD.*.* /var/log/%Y/%m/%d/syslogd
*.err /var/log/%Y/%m/%d/errors

This says all messages for SYSLOGD go to a file like /var/log/2022/05/14/syslogd, and error messages go to /var/log/2022/05/14/errors

This means you get a file of messages for each day. For me, I just used /var/log/syslogd.log and /var/log/errors.log, and deleted them periodically.

Priority

Data logged to syslogd has a “priority”. For example AT-TLS trace level 32 (Data) have a priority of “debug”. You can use this for example

*.TCPIP.*.debug     /var/log/TCPIPdebug 
*.TCPIP.*.*         /var/log/TCPIP 

This says

  1. for messages from TCPIP with priority debug or higher( debug, info, notice, warning, error, crit, alert, emerg) then write the data to /var/log/TCPIPdebug .
  2. Write all messages to /var/log/TCPIP

As debug is the lowest level of priority, these statements are effectively the same.

It may be better to have

*.TCPIP.*.debug     /var/log/TCPIPdebug 
*.TCPIP.*.warning   /var/log/TCPIP 

The priority can be “none” which means do not log any messages.

How do I capture messages not handled else where?

This is a bit clumsy.

When a message arrives each of the rules are check. If the check is true the message is logged.

You can have compound checks separated by a semicolon for a rule.

For example

*.*.*.* ; *.TCPIP.*.none ; *.PAGENT.*.none /var/log/all.log

This says log all messages – but not from TCPIP or PAGENT. If you have have one file for 20 jobs, you need to have 20 statements with the semicolon.

You can spread the definition over several lines. The semicolon at the end of the line says read next line. The code is not smart enough to put the /var/log/all on its own line, so using a ‘dummy’ may make it easier to maintain.

*.*.*.*; 
       *.TCPIP.*.none; 
       *.PAGENT.*.none;
       *.DUMMY.*.none /var/log/all 

Capturing the right data

If the config file you can specify options like

*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

The entries are Userid.Jobname.facility.priority.

PAGENT AT-TLS request are reported via TCPIP.

I could not find how to filter the TCPIP data so the AT-TLS data went to one file, and other TCPIP data went to another file. For TCPIP it looks like the “facility” is either “daemon” or “auth”, which

ISPF interface

There is an ISPF syslog browser tool which displays information about the logs, and helps you browse the logs of interest. The documentation for this is not very good.

I got this to work by experimentation. I created an exec like MYSYSLOG

/* Rexx */ 
address ispexec 
"LIBDEF ISPMLIB DATASET ID('TCPIP.SEZAMENU') STACK" 
"LIBDEF ISPPLIB DATASET ID('TCPIP.SEZAPENU') STACK" 
address tso "ALTLIB ACTIVATE APPLICATION(CLIST) 
              DATASET('TCPIP.SEZAEXEC') " 
"SELECT CMD(EZASYRGO) NEWPOOL PASSLIB NEWAPPL(EZAS)" 
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)" 
"LIBDEF ISPPLIB" 
"LIBDEF ISPMLIB" 

You can execute this from ISPF option 6 or have this built into the ISPF panels.

Originally this exec was called syslogd; when I used it, I got

SYSTEM COMPLETION CODE=4C5 REASON CODE=77A53217

Where 4C5 is TCPIP’s abend code and 3217 – the program has the wrong Authrorization Code (APF related). This is because there is a command syslogd which was executed in preference to my exec. When I renamed the exec to MYSYSLOG it used the exec and it worked fine!

The first panel is

EZASYP01 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 1
Command ===>                                                  Scroll ===> PAGE
                                                                               
Enter syslogd browser options                                                  
  Recall migrated data sets ==> NO     (Yes/No) Recall data sets or not        
  Maximum hits to display   ==> 200    (1-99999) Search results to display     
  Maximum file archives     ==> 30     (0-400) Days to look for file archives  
  Display start date/time   ==> YES    (Yes/No) Retrieve start date/time       
  Display active files only ==> NO     (Yes/No) Active files only, no archives 
  DSN Prefix override value ==>                                                
                                                                               
Enter file or data set name of syslogd configuration, or select one from below:
                                                                               
  File/DS Name ==> /etc/syslog.conf
                                                                               
Press ENTER to continue, or press END PF key to exit without a selection       
                                                                               
Line commands: S Select, R Remove from list, B Browse content, E Edit content  
                                                                               
Cmd Recently used syslogd configuration file or data set name                  
--- -------------------------------------------------------------------------- 
    /etc/syslog.conf                                                          
******************************* Bottom of data ********************************

Pressing enter, gave me another panel with

EZASYP00 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 6
OPTION ===>                                                   Scroll ===> PAGE
                                                                               
Select one of the following, or press END PF key to exit the syslogd browser   
                                                                               
  1 Change current syslogd configuration file and/or options                   
  2 Guide me to a possible syslogd destination                                 
  3 Clear guide-me hits (indicated by ==> in the Cmd column)                   
  4 Search across all active syslogd files                                     
                                                                               
Current config file ==> /etc/syslog.conf                                      
                                                                               
Line commands: B Browse, A List archives, S Search active file and archives,   
               SF Search active file, SA Search archives, I File/DSN info      
                                                                    Archive    
Cmd Rule/Active UNIX file name                    Start Time        Type Avail.
--- --------------------------------------------- ----------------- ---- ------
    *.SYSLOGD*.*.*                                28 May 2022 13:31 None 0     
    /var/log/syslogd                                                          
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    *.INETD*.*.*                                  Empty       N/A   None 0     
    /var/log/inetd                                                            
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    auth.*                                        Empty       N/A   None 0     
    /var/log/auth                                                             

I could then browse the error log for SYSLOGD.

You can search for userid, strings etc, and give date ranges.

However for my small, one person usage, I found it was easier to use Unix services and use the command

oedit /var/syslogd.log

to edit the file.

you can specify in the TTLS configuration. So not very useful.