Spice up a C program by using __asm__ to include inline assembler.

The C compiler on z/OS has an extension which allows you to put assembler code inline within a C program. This function can be useful, for example accessing z/OS macros. __asm__ is very badly documented in the publications, but this post gives a good overview.

Overall the use of __asm__ works, but you have to be careful. For small bits of assembler it was quicker to use __asm__ instead of creating a small assembler program and linking that with the C program.

This blog post document some of my experiences.

Using and compiling the code

You put code in __asm__(…); , _asm(..); or asm(..); . I think these are all the same.

To use macros or copy files within the code you need the ASMLIB statement in your JCL.

//S1          JCLLIB ORDER=CBC.SCCNPRC 
//STEP1    EXEC PROC=EDCC,INFILE='COLIN.C.SOURCE(ASM)', 
//         LNGPRFX='CBC',LIBPRFX='CEE', 
//         CPARM='OPTFILE(DD:SYSOPTF)' 
//COMPILE.ASMLIB DD DSN=SYS1.MACLIB,DISP=SHR 

Basic function

The asm() instruction has the following parts

  • asm(
  • “a string of source which can contain %[symbolname] “. Each line of assembler has “\n” at the end of the line.
  • the output code will be formatted to conform to normal HLASM layout standards.
  • “:” a list of output symbols and their mapping to C variables.
  • “:” a list of symbols used as input and their mapping to C variable names.
  • “:” a list of register that may have been changed (clobbered) by this code, for example “r14” and “r15”.
  • “);”

Example code

__asm__(
“*2345678901234567890xxxxxxxxxxxxxxxxxxxxxxxx\n”
” WTO ‘%[PARMS]’ \n”
:
:
[PARMS]”s”(“zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz”)
: “r0”, “r1”, “r14”, “r15”
);

The PARMS statement is a string with a value ZZZZZ… It is used in the WTO ‘%[PARMS]\n’ statement.

Long statements – wrapping and continuation

The generated code from the above statement is

*2345678901234567890xxxxxxxxxxxxxxxxxxxxxxxx                             000023  
         WTO   'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzX 000023  
               zzz'                                                      000023  

We can see

  • the *234… starts in column 1
  • the WTO instruction is in column 10
  • because the string was very long, it has been split at column 71 and wrapped onto the next line at column 16.
  • A continuation character was inserted at column 72

This means you do not need to worry too much about the formatting of the data.

The code looks a bit buggy.

Making the WTO into an operand and a comment

__asm__(
“*2345678901234567890xxxxxxxxxxxxxxxxxxxxxxxx\n”
” WTO abc ‘%[PARMS]’ \n”
:
:
[PARMS]”s”(“zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz”)
: “r0”, “r1”, “r14”, “r15”
);

Gives a warning message

*2345678901234567890xxxxxxxxxxxxxxxxxxxxxxxx                            
         WTO   abc                     'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzX
               zzzzzzzzzzzzzzzzzzzzzzzzzzz' 

ASMA432W Continuation statement may be in error - comma omitted from continued statement.                            

What are the __asm__ parameters?

The first parameter is a C string containing the assembler instructions. Each line ends with a “\n”. You specify substitution variables using %[name] where the name is defined later.

You can use multiple lines, for example

asm(
” LA R2,%[p1] \n”
” TIMEUSED LINKAGE=SYSTEM,CPU=MIC,STORADR=(2) \n”
:

The C compiler treats the “…” “…” as one long literal constant.

There are three sets of values between the delimiters “:” “:” “:”

  • Output variables
  • Input variables
  • General Registers which your code changes

A C variable can be used for

  • Output only. This is in the output section. This has a definition with “=”. For example [p1] “=m”(pASCB). asm() may generate code to load load the value before use
  • Input and output. This is in the output section. This has a definition with “+”. For example [i1] “+m”(data), asm() may generate code to load the value before use, and store it afterwards.
  • Input only. This is in the input section. It does not have character with its definition. For example [i2] “m”(data)
  • Dummy – but used as a register. If you specify you want a register (let asm() which register), it needs a variable either to load, or to store depending if it is write or read or both. For example [rr] “=r”(val). I defined the C variable “val”.

Using C variables in the assembler code

There is a variety of different “types” of data, from memory, to offset. I could not see the difference between them. Some gave the same output. I tended just to use “m” for memory fields.

Use of variables

// this code gets the ASCB address from low core

long pASCB;
asm(” LLGT 1,548 \n”
” STG 1,%[p1] \n”
: [p1] “=m”(pASCB)
:
: “r1”
);

This (64 bit) code

  • Clears and loads register 1 with the value in address decimal 548. (The ASCB value) .
  • It stores register 1 in the the variable %[p1]
  • [p1] is defined as
    • “=” means this field is write only
    • m is a memory address
    • (pASCB) is the variable to use. The compiler replaces this with (in my case) the value 2248(4) – the address of the variable in format offset(base regiser).
  • There was no input data
  • Register r1 was “clobbered” (meaning it was changed in my assembler code).

Using constants is not quite what I expected.

printf(“ttime %ld\n”,data);
asm(” LA 1,%[i1] ccp\n”
” LA 1,%[i2] \n”
” LA 1,%[i3]\n”
:
: [i1] “i”(“999”),
[i2] “i”(998)
[i3] “i”(“=c\’ABCD\'”)
: “r1″,”r2”
);

Gives code

 LA    1,999
 LA    2,998
 LA    2,=c'ABCD'            

Using [i2] “i”(“COLIN”) gave

LA 2,COLIN
ASMA044E Undefined symbol – COLIN

An example of using registers

David Clayford posted the following

inline bool isauth() {
   int rc;
   __asm (" TESTAUTH FCTN=1"
         : "=NR:r15"(rc)
         :
         : "r1", "r14", "r15");
   return rc == 0;

This code invokes the TESTAUTH macro which gives a return code in register 15.

The “=NR: r15” (rc) means

  • = the __ASM__ only writes, it does not read the variable
  • NR : Use the named register
  • r15 use this register
  • (rc) and store it in the variable called rc

Using generated registers – or not

You specify that you want a register allocated to you by using the type “r”.

int val = 40;
asm(” LLGT %[rr],548 pASCB \n”
STG %[rr],%[p1] ZZZZZ \n”
: [p1] “=m”(pASCB)
: [rr] “r”(val)
: “r1″,”r2”
);

The lack of a “=”, “+” or “&” in front of the “r” means read only use of the register, so load the register with the value before my code.

Produces

   LGF   r6,val(,r4,2248)   - This is generated                  
   LLGT  6,548                   pASCB           
   STG   6,2240(4)               ZZZZZ           

This code has been given register 6 to use

  • It loaded the value of val into it – because I had specified it in the list of input variables value.
  • Used the same register where-ever I had specified %[rr] .

When I had specified the register as an input/output register by

: [p1] “=m”(pASCB), [rr] “+r”(val)
:
: “r1″,”r2”

The “+” says it is read and written the output code was

     LGF      r6,val(,r4,2248)   Generated                      
     LLGT     6,548              My Code pASCB                
     STG      6,2240(4)          My Code ZZZZZ 
     LGR      r0,r6              Generated                    
     LGFR     r0,r0              Generated                    
     ST       r0,val(,r4,2248)   Generated                    

So there is code generated to load the register from val, and save the value of the last of my instruction in the variable val.

Personally, I do not think I would use the “r”, but would select my own register(s) and use them.

If I wanted to used C variables, I can specify those, and explicitly load and save them.

Some instructions do not work.

char buffer[256];

asm(
” MVC [%p1](4),548 pASCB \n”

: [p1] “=m”(buffer)
:
: “r1″,”r2”
);

This fails with

: [p1] “=m”(buffer)
CCN4454 Operand must be an lvalue.

You need to use

You need to use [p1] “=m”(buffer[0]) instead of (buffer). (But this is just normal C)

The instruction then fails because

MVC 2240(4)(4),548

Is not a valid instruction.

You need to use

char buffer[256];
asm(
” LA 1,%[p1] \n”
” MVC 0(4,1),548 pASCB \n”

:
[p1] “=m”(buffer[0])
:
: “r1″,”r2”
);

Which successfully generates

  LA r1,2240(r4,)
  MVC 0(4,r1),548

Using literals

You can use assembler literals in your code, for example

asm(
” LA 1,=C’ABCD’ \n”
:
:
: “r1”
);

This works. There is a section in the listing

Start of ASM Literals
         =C'ABCD'
End of ASM Literals

Using assembler macros

When you use a macro, you need to review the generated code, and make a note of the registers it uses, then update the “clobbers” list.

asm(
” LA 2,%[p1] \n”
” TIMEUSED LINKAGE=SYSTEM,CPU=MIC,STORADR=(2) \n”

:…

This used r14,r0,r15

There was an error

BNZ   *+8  
*** ASMA307E No active USING for operand *+8         

I had to use the following to get it to work.

long long CPUUSED;
asm(
” PUSH USING \n”
” BASR 3,0 \n”
” USING *,3 \n”

” LA 2,%[p1] \n”
” TIMEUSED LINKAGE=SYSTEM,CPU=MIC,STORADR=(2) \n”
” POP USING \n”
:
[p1] “=m”(CPUUSED)
:
: “r0″,”r1″,”r2”,“r3”,r14″,”r15″
);
printf(“TIMEUSED %ld\n”,CPUUSED);

Using assembler services from a (64 bit) C program.

I wanted to provide a Python started task on z/OS to respond to the operator Stop, and Modify commands (for example to pass commands to the program).

I wrote a Python extension in C, and got the basics working. I then wanted to extend this a bit more. I learned a lot about the interface from C to assembler, and some of the newer linkage instructions.

The sections in this post are

Some of the problems I had were subtle, and the documentation did not cover them.

C provides a run time facility called __console2 which provide a write to the console, and a read from the console.

The output from the write to the console using __console2 looks like

BPXM023I (COLIN) from console2

Thanks to Morag who pointed out you get the BPXM023i(userid) if the userid does not have READ access to BPX.CONSOLE in class(FACILITY).

With the BPXM023I prefix, which I thought looked untidy and unnecessary.

To use the Modify operator command with __console2 you have to use a command like

F PYTTASK,APPL=’mydata’

Which feels wrong, as most of the rest of z/OS uses

F PYTTASK,mydata

This can be implemented using the QEDIT assembler interface.

The journey

I’ll break my journey into logical steps.

Information on programming the C to assembler interface

There is not a lot of good information available.

Calling an assembler routine from a C program.

Setting up the linkage

A 64 bit program uses the C XPLINK interface between C programs.

To use the traditional Assembler interface you need to use

#pragma linkage(QEDIT , OS)

rc = QEDIT( pMsg, 6);

C does not set the Variable Length parameter list bit (the high bit of the last parameter to ’80…) so you cannot use parameter lists with a variable length, and expect traditional applications to work. You could always pass a count of parameters, or build the parameter list yourself.

Register 1 pointed to a block of storage, for example to parameters

00000000 203920D8 00000050 082FE3A0

which is two 64 byte addresses, the address of the pMsg data, and the address of the fullword constant 6;

The C code invokes the routine by

LG r15,=V(QEDIT)(,…,…)
BALR r14,r15

even though use of BALR is deprecated.

Allocating variable storage

The z/OS services my assembler program used, need 31 bit storage.

I allocated this in my C program using

char * __ptr32 pMsg;
pMsg = (char *) __malloc31(1024);

I then passed this to my assembler routine.

Coding the assembler routine

Assembler Linkage

The basic linkage was

BSM 14,0
BAKR 14,0

…….
PR go back

This is where it started to get complicated. The BAKR… PR is a well documented and commonly used interface.

A Branch and StacK Register instruction BAKR 14,15 says branch to the address in register 15, save the value of register 14 as the return address, and save the registers and other status in the linkage stack. The code pointed to by register 15 is executed, and at the end, there is a Program Return (PR) instruction which loads registers from the linkage stack, and goes to the “return address”.

The Branch and Stack instruction BAKR 14,0 says do not branch, but save the status, and the return address. A subsequent PR instruction will go to where register 14 points to.

Unfortunately, with the BALR code in C, and the BAKR, PR does not work entirely.

You can be executing in a program with a 64 bit address instructions (such as 00000050 089790A0), in 24 or 31, or 64 bit mode.

  • In 64 bit mode, all the contents of a register are used to address the data.
  • In 31 bit mode only the bottom(right) half of the register are used to address the data – the top half is ignored
  • In 24 bit mode, only the bottom 24 bits of the register are used to address the data.

There are various instructions which change which mode the program is executing in.

When a BAKR, or BASSM ( Branch and Save, and set Mode) is used, the return address is changed to set the mode ( 24,31,64) as part of the saved data. When this address is used as a branch back – the save mode information is used to switch back to the original mode.

When BALR (or BASR) is used to branch to a routine, the return address is saved in Register 14. The mode information is not stored. When this address is used as a branch back – the “default mode” information (24 bit) is used to set the mode. This means the return tries to execute with a 24 bit address – and it fails.

You solve this problem by using a (BRANCH AND SET MODE) BSM 14,0 instruction. The value of 0 says do not branch, so this just updates register 14 with the state information. When BAKR is issued, the correct state is saved with it.

If you use the “correct” linkage you do not need to use BSM. It is only needed because the C code is using an out dated interface. It still uses this interface for compatibility with historically compiled programs.

Note: BSM 0,14 is also a common usage. It is the standard return instruction in a program entered by means of BRANCH AND SAVE AND SET MODE (BASSM) or a BRANCH AND SAVE (BAS). It means branch to the address in register 14, and set the appropriate AMODE, but do not save in the linkage stack.

Using 64 and 31 bit registers

Having grown up with 32 bit registers, it took a little while to understand the usage 64 bit registers.

In picture terms all registers are 64 bit, but you can have a piece of paper with a hole in it which only shows the right 32 bit part of it.

When using the full 64 bit registers the instructions have a G in them.

  • LR 2,3 copies the 32 bit value from register 3 into the right 32 bits of register 2
  • LGR 2,3 copies the value of the 64 bit register 3 into the 64 bit register 2

If there is a block of storage at TEST with content 12345678, ABCDEFG

  • R13 has 22222222 33333333
  • copy R13 into Reg 7. LGR R7,R13. R7 contains 22222222 33333333
  • L R7,TEST. 32 bit load of data into R7. R7 now has 22222222 12345678. This has loaded the visible “32 bit” (4 bytes) part of the register, leaving the rest unchanged.
  • LG R8,TEST. 64 bit load into Reg 8. R8 now has 12345678 ABCDEFG . The 8 bytes have been loaded.
  • “clear high R9” LLGTR R9,R9. R9 has 00000000 ……… See below.
  • L R9,TEST . 32 bit (4 bytes) load into R9. R9 now has 00000000 12345678

Before you use any register in 32 bit code you need to clear the top.

The Load Logical Thirty One Bits (LLGTR R1,R2) instruction, takes the “right hand part” of R2 and copies it to the right hand of R1, and sets the left hand part of R1 to 0. Or to rephrase it, LLGTR R2,R2 just clears the top of the register.

Using QEDIT to catch operator commands

QEDIT is an interface which allows an application to process operator commands start, modify and stop, on the address space.For example

  • S PYTASK,,,’STARTPARM’
  • f PYTASK,’lower case data’
  • p PYTASK

Internally the QEDIT code uses CIBs (Console Information Block) which have information about the operator action

(I think QEDIT comes from “editing”= removing the Queue of CIBs once they have been processed – so Q EDIT).

The interface provides an ECB for the application to wait on.

The documentation was ok, but could be clearer. For example the sample code is wrong.

In my case I wanted a Python thread which used console.get() to wait for the operator action and return the data. You then issue the console.get() to get the next operator action.

My program had logic like

  • Use the extract macro to get the address of the CIBS.
  • If this job is a started task, and it is the ‘get’ action then there will be a CIB with type=Started
    • Return the data to the requester
    • Remove the CIB from the chain
    • Return to the requester
  • Set the backlog of CIBs supported ( the number of operator requests which can be outstanding)
  • WAIT on the ECB
  • Return the data to the requester
  • Remove the CIB from the chain
  • Return to the requester

The action can be “Start”, “Modify”, or “Stop”

Can I use __ASM__ within C code to generate my own assembler?

In theory yes. The documentation is missing a lot of information. I could not get my simplest tests to work and return what I was expecting.

How do I assemble a program in Unix System Services using the c complier?

My preferred way of assembling some code is to use JCL. The next best way is to use the “as – Use the HLASM assembler to produce object files“. (For example “as -a -d xxxx.o xxxx.s 1>a”) Unfortunately I needed to use the”C” compiler xlc, because I wanted to compile a file for a Python External command.

The basic syntax is

xlc “-Wa,LIST” zonsole.s 1>a 2>b

Where

  • -Wa specifies these are options for the assembler source
  • LIST says generate a listing. By default it does not generate a listing, only the error file.
  • zonsole.s is my assembler source program
  • 1>a says put the listing into file a
  • 2>b says put the error file into file b

At the top of the listing file is

No Overriding ASMAOPT Parameters
Overriding Parameters- NODECK,OBJECT,TERM,NOLIST,NOESD,NORLD,NOXREF,FLAG(4),LIST
No Process Statements

** ASMA425N Option conflict in invocation parameters. LIST overrides an earlier setting.

The options in the -Wa are described here

Using z/OSMF Network Configuration assistant for TCPIP, to define AT-TLS configuration

I initially found it hard to set up the AT-TLS configuration for MQ. The easiest way was to use the sample configurations provided by MQ. See here for an overview. I used Scenario 5 – Between an IBM MQ for z/OS queue manager and a client application running on IBM MQ for Multiplatforms.

Using the MQ samples, this took about 10 minutes once I had PAGENT and SYSLOGD set up.

I thought I would try to use the TCP provided facilities. There is a lot of documentation, but it is not easy to find what you need. It has been written as an IBM developer, rather than from an end user or task based perspective.

I then thought I would try to use the “way of the future” and the z/OS configuration tool z/OSMF. You use a browser to connect to z/OSMF and do your work through the browser interface. The z/OSMF interface has configuration tools, and workflow tools which guide you step by step through configuration.

I’ve blogged Using z/OSMF workflows for TCPIP. Using the workflow was not very successful.

The Network Configuration Assistance is used to configure the PAGENT, and I used it to define a AT-TLS configuration. Initially this was a struggle as there was no documentation to tell me how to do it. Once I had been through the configuration a couple of times, I found the correct path through the configuration progress and it is relatively painless.

I found In z/OSMF Configuration Assistant, how do I import policy data from the Policy Agent? which may or may not help you

My mission.

My mission was to configure AT-TLS and to provide two ports for use with MQ.

I wanted to do this using two people (me with two userids) and do the typical steps when changing systems, such as saving configurations before changing them, and deploying them, when I had a “change window”.

Using the Network configuration assistant (CA)

AT-TLS concepts

You need to be aware of the AT-TLS concepts when defining the configuration. From an administrator’s perspective:

  • What ports you want to protect? This is known by the CA as Traffic Descriptors. You can specify
    • An IP port
    • A range of IP ports
  • What IP addresses you want to protect.
    • The IP address. A TCP/IP stack can support different IP addresses. You can use a specific IP address. You can select on all IPV4, or all IPV6, or all addresses.
    • The name of a group of IP addresses. z/OSMF CA calls these Address Groups.
  • How do you want to protect the session. For example what levels of TLS, and what cipher specs do you want to use. This is known by the CA as Security levels.
  • The mapping of ports to protecting the session. z/OSMF calls this Requirement Maps.
  • You configure at the TCPIP stack level.
  • z/OSMF has groups of z/OS instances, with one or more z/OS instances, and you can have multiple TCPIP stacks in a z/OS instance.

Backing store

The configuration assistant(CA), stores configuration in a backing store. You can use tools to copy the current store. I found a copy of the file in /global/zosmf/data/app/CAV2R4/backingStore/save Data. I should be able to use standard backup procedures to keep a copy of the file. The resulting configuration is created in a file which is used by PAGENT.

You can copy a backing store within the CA, and so go back to the original one if you need to.

Before you start.

You should collect the information that you will be used to configure PAGENT. For example

  • Which systems and IP stacks will be used.
  • Which keyrings and certificates will be used?
  • For each port you want to protect.
  • What rules do you want, for example which cipher specs.

I found the terms used when creating the rules manually – did not map to the CA concepts, but once you understood the difference in terminology it was ok.

How to define the resources

If you define the configuration bottom up. You define all of the rules, then when you get to configure the TCPIP stack, the rules and other components should all be there, you just have to select them.

If you define the configuration top down, you define the TCPIP stack, then the rules for the stack. You have to define the TCPIP stack, then the rules, then go back and reconfigure the TCPIP stack to add the rules.

I think bottom up configuration is better while you gain experience of the tool. Once you are familiar with the tool then the top down approach will work ok, and may be how you update the configuration.

Getting started

  • Double click on the Network Configuration Assistant icon.
  • You get a page Welcome to V2R4 Configuration Assistant for z/OS Communications Server. On this page you can specify the name of the backing store. The pull down lists the existing backing stores. If you do not have a backing store create one. You can use “tools” button to copy the backing store.
  • The “getting started” gives you information on how to use the panels. I found it a little confusing at times. It displays the help in a separate window. In the Table of Contents, it has “AT/TLS – getting started”. I didn’t find the help or tutorials much use.
  • On the Welcome page, press Proceed.
    • I sometimes get “The backing store is locked by your id.” I got this after I had shutdown down z/OSMF without logging off.
    • You can use “Tools” to manage your backing store, and configuration.
    • “Select a TCP/IP technology to configure” : AT-TLS
    • The layout of the panels, make me think you create the definitions from top to bottom, and so the tabs are defined left to right. I think it is easier to define resources then create the group/image/stack.

Define the rules for which ports to be protected

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Traffic Descriptors tab.

  • Actions -> New…
  • Name COLINTD
  • Actions-> New…
  • Under the Details tab, specify the port or port range and any other information
  • Under the KeyRing tab, specify the keyring and the Certificate label or let it default to the TCPIP stack level keyring.
  • Under the Advanced tab, I let everything default.
  • Click OK
  • You can define a second port within this Traffic Descriptor
  • Click OK

You can press Save to save the current definitions.

Define which IP addresses you want to protect (optional)

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Address Groups tab.

By default it has

  • All_IPv4_Addresses
  • All_IPv6_Addresses
  • All_IP_Addresses

A TCPIP stack can host different IP addresses, one for each connector coming in. If you want to limit rules to particular stack owned IP addresses, create a definition.

  • Actions-> New
  • Name: COLINAG
  • IP Address: 10.1.1.2
  • IP Address: 10.1.1.3
  • OK

You can press Save to save the current definitions.

How do you want to protect the session.

For example what levels of TLS, and what cipher specs do you want to use.

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Security Levels tab.

  • Actions: -> New…
  • Name: COLINSL
  • Select levels of TLS you want to use
  • Next, select cipher specs. I used “Use 2019 suggested values”
  • Next – I took the default (“Do not generate FIPS 140 support“)
  • Click on Advanced settings.
    • If you want to use client authentication click the “Use client authentication” box
    • OK
  • Finish

Your definition should be in the list.

You can press Save to save the current definitions.

Mapping of ports to Session protection

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Requirement Maps tab.

  • Actions: -> New…
  • Name: COLINMAP .
  • In the mappings table,
    • use the Traffic Descriptor pull down and select the Traffic Descriptor you created above. For example COLINTD.
    • Under Security Level pull down select the security definition you created above. For example COLINSL.
  • OK

If I changed an existing definition, I had a pop-up

Modify Requirement Map.
The requirement map you are changing may be referenced in at least one connectivity rule.

Prior to making this change you may want to see which connectivity rules are referencing this requirement map. Click OK to show where used. Click Proceed to proceed with the Modify; otherwise, click Cancel.

Click OK to show where it is used.

Click Proceed

You can press Save to save the current definitions.

Create the group, z/OS instance and TCPIP Stack

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Systems tab.

  • Action: -> Add z/OS group…
  • Name: COLINGR
  • Click OK
  • Action: -> Add z/OS system image…
  • Name: COLMVSA
  • Keyring: START1.COLINRING
  • Press OK
  • I get a pop-up Proceed to the next step? Connectivity rules are configured for each TCP/IP stack. To continue with configuration you need to add a TCP/IP stack to the new z/OS system image. Do you want to add a TCP/IP stack now? Click on Proceed.
  • This gets to the Add TCP/IP stack
  • Name:TCPIP
  • OK
  • I get a pop-up. Proceed to the next Step? to continue with the configuration you should add connectivity rules to the TCP/IP stack. Do you want to be directed to the TCP/IP stack rules panel? Proceed.
    • If you cancel you can use the Actions -> rules to define the rules.
  • I get a pop-up Proceed to the Next Step? Do you want to start a wizard to create a connectivity rule? Click Proceed.
  • This gets to the Data End points where you associate the IP addresses to the stack instance.
  • Name: COLINRN
  • Select from the address group pull-down, or let it default.
  • Press Next
  • This gets to the Requirement Mapping to Stack association.
    • You can select from the existing requirements map see Mapping of ports to Session protection above, or create a new mapping.
    • You can create a new map, for example Name: COLINMP
      • Select from the Traffic Descriptor pull down
      • Select from the Security level pull down.
  • Press Next
  • You can specify advanced settings, such as Tracing, Tuning, Environment, Effective times, Handshake
  • Finish
  • Close

You can press Save to save the current definitions.

Join the bits up

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Systems tab.

  • Select a group instance
  • Actions: Install All files for this group
  • This will list the configuration files.
  • On the Configured File Name,
    • right click on the file name value, and ->Show Configuration File. This will show you the configuration as it might be deployed.
    • right click on the file name value and -> Install … . Specify the a file name and click GO.
    • Close
  • You can use
    • Actions: Install to create the configuration file
    • Actions: Show configuration file to see the generated configuration

You can now use the configuration file as input to PAGENT.

You can press Save to save the current definitions.

Extending the configuration to add a new rule

It took a while to work out how to do this, but it is really easy.

In the page Network Configuration Assistant (Home) -> AT-TLS page, click on the Traffic Descriptors tab.

  • Create a new Traffic descriptor as above
  • Get back to Network Configuration Assistant (Home) -> AT-TLS page, and click on the Systems tab.
  • Select a TCPIP instance, and click Actions: -> Rules..
  • Actions: -> New
  • Connectivity rule name: rule2
  • Press Next
  • You can select from the existing requirements map see Mapping of ports to Session protection above, or create a new mapping. If you have just created a new rule, then you may not have defined a mapping, and create it “inline”.

Then install it.

Use the configuration

You need to change the PAGENT JCL to use the created configuration file. You may want to copy it to a backup, as the next time you reconfigure it can overwrite the file. Or just create a new file perhaps with a date in the filename.

If you have problems with a newly reconfigured file you need a copy of the previous, working, definitions.

Display the configuration

On many items, you can use right click -> Show where used. This will then display the group, image, stack, connectivity rules and data end points where the item is used.

Should I use this just to get started, or every time.

When I created my definitions by hand, I could put definitions in to a “Common” section, and have multiple TCPIP stacks in one configuration file. I could have small files with bits of configuration in them.

If you use the CA, “common” definitions are copied into the configuration file, and you have one configuration file per TCPIP stack instance, so you do not need to have a common section etc.

As a configuration tool, now I know how to use it, I might continue to use it – but it is slightly more complex than this.

I want to enable trace for one definition. To do so means I have to…

  • Change the configuration to set the trace. This can be difficult if someone else is in the middle of changing the configuration.
  • Deploy the whole configuration. You may pick up incomplete changes which have been made, but not deployed.
  • If a second TCPIP stack is using the configuration, this may get trace enabled if the configuration file is recreated.

Overall (my views may change tomorrow), I would use the CA to create my configuration. Then not use it again – or use it again to generate definitions which I can copy into my master configuration files. I would restructure the configuration so create small files with specific content.

Trace PAGENT and AT-TLS

Many components of TCPIP write information to syslogd. This is a process that captures the data sent to it over a UDP socket, and writes it to files in the Unix file system. If the syslogd is not active then messages may be written to the job log. When I was trying to set up AT-TLS, I had 10s of messages on the job log, each time a client tried to use AT-TLS.

The IBM documentation is not very clear, it tells you how to turn on debug, trace etc but does not clearly explain the difference, and when they are used.

It look like the PAGENT job is to take a configuration file, parse it, and pass the data to TCPIP.

If you are using AT-TLS to set up TLS channels, the trace data comes from the TCPIP address space.

Modify the PAGENT address space.

You can pass commands to the PAGENT address space.

Configuration processing.

You can control how much information is logged when parsing configuration statements. The value 127 covers most levels of information (including warnings).

F PAGENT,LOGLEVEL,LEVEL=127

You can use

F PAGENT,TRACE,LEVEL=..
F PAGENT,DEBUG,LEVEL=…

But these do not seem to control the level of trace produced.

Trace PAGENT startup and parse of the configuration

To collect the PAGENT startup and display information on the configuration file as it is processed change the started task JCL to include the -d option.


//PAGENT EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT,
// PARM=’ENVAR(“_CEE_ENVFILE_S=DD:STDENV”)/ -d 4

By default the output trace goes to /tmp/pagent.log. It has content like

05/29 17:17:54 EVENT :005: pzos_install_A_PolicyRule: Finished installing policy rule: ‘REMOTE-TO-CSQ1’

Trace PAGENT use of TLS

My PAGENT JCL is

//CPAGENT  PROC 
//  SET EN='ENVAR("_CEE_ENVFILE_S=DD:STDENV")' 
//PAGENT   EXEC PGM=PAGENT,REGION=0K,TIME=NOLIMIT, 
//       PARM='&EN/                                      -d 4' 
//STDENV   DD DISP=SHR,DSN=USER.Z24C.TCPPARMS(PAGENTEN) 
//SYSPRINT DD SYSOUT=H 
//SYSERR   DD SYSOUT=H 
//SYSOUT   DD SYSOUT=H 
//* 
//CEEDUMP  DD SYSOUT=*,DCB=(RECFM=FB,LRECL=132,BLKSIZE=132) 

With the environment file USER.Z24C.TCPPARMS(PAGENTEN) having

_CEE_ENVFILE_COMMENT=# 
PAGENT_CONFIG_FILE=//'USER.Z24C.TCPPARMS(PAGENTCF)' 
LIBPATH=/usr/lib 
GSK_TRACE=0x00
GSK_TRACE_FILE=/var/log/GSK 

You can collect the GSK calls made by PAGENT at startup by using the environment variables

GSK_TRACE=0xFF
GSK_TRACE_FILE=/var/log/GSK

Note: This turns it on for all requests! I could not find how to do selective tracing.

You have to format the trace file using

gsktrace /var/log/GSK /var/log/GSK.txt

This has about 40 lines with information like

05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-1 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-224 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-256 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-384 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): SHA-512 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): DES3 crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 128-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES 256-bit crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): AES-GCM crypto assist is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic accelerator is not available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Cryptographic coprocessor is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Public key hardware support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Max RSA key sizes in hardware – signature 4096, encryption 4096,
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): Maximum RSA token size 3500
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC clear key support is available
05/28/2022-17:53:30 Thd-5 INFO crypto_init(): ECC secure key support is available. Maximum key size 521

Remember this is the PAGENT invoking GSK – but PAGENT does not do any TLS work – this is done by TCPIP.

Trace an AT-TLS connection.

You need to enable trace in the AT-TLS configuration for example

TTLSEnvironmentAction CSQ1-INBOUND-ENVIRONMENT-ACTION
{
HandshakeRole SERVER
TTLSKeyringParmsRef CSQ1-KEYRING
TTLSCipherParmsRef CSQ1-CIPHERPARM
TTLSEnvironmentAdvancedParmsRef CSQ1-ENVIRONMENT-ADVANCED
Trace 255
}

If the syslogd daemon is not configured the output goes to the TCPIP job log.

If the syslogd daemon is configured, you need to have a syslogd configuration with


*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

Where TCPIP is the TCPIP address space name, and *.*.*.* is a catch-all. It took me about a day to realize that my trace was being thrown away because I didn’t have TCPIP, and the catch-all.

The trace file has data like

May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1284I TTLS Flow GRPID: 00000021 ENVID: 00000009 CONNID: 00000053 RC: 0 Set GSK_USER_DATA(200) – 000000007F280610

May 29 09:25:30 S0W1 TTLS[67174439]: 09:25:30 TCPIP EZD1285I TTLS Data CONNID: 00000053 RECV CIPHER 160303007B

This shows a GSK call was made to GSK_USER_DATA which completed with return code 0, and the connection RECeiVed data which was the CIPHER specs ( 4 chars or 2 chars) 160303007b.

You need to configure the syslogd procedure.

See if SYSLOGD is running, if not, try to start it. If it does not exist…

  • Copy /usr/lpp/tcpip/samples/syslog.conf to its default configuration file /etc/syslog.conf, or another file.
  • Copy TCPIP.SEZAINST(SYSLOGD) to your proclib concatenation.
  • The program uses environment variables defined in STDENV to control operations. The default configuration file location is /etc/syslog.conf

You can configure syslog.conf for example

*.TCPIP.*.* /var/log/%Y/%m/%d/TCPIP
*.SYSLOGD.*.* /var/log/%Y/%m/%d/syslogd
*.err /var/log/%Y/%m/%d/errors

This says all messages for SYSLOGD go to a file like /var/log/2022/05/14/syslogd, and error messages go to /var/log/2022/05/14/errors

This means you get a file of messages for each day. For me, I just used /var/log/syslogd.log and /var/log/errors.log, and deleted them periodically.

I also added a the end of the file, the catchall

*.*.*.* /var/log/all.log

ISPF interface

There is an ISPF syslog browser tool which displays information about the logs, and helps you browse the logs of interest. The documentation for this is not very good.

I got this to work by experimentation. I created an exec like MYSYSLOG

/* Rexx */ 
address ispexec 
"LIBDEF ISPMLIB DATASET ID('TCPIP.SEZAMENU') STACK" 
"LIBDEF ISPPLIB DATASET ID('TCPIP.SEZAPENU') STACK" 
address tso "ALTLIB ACTIVATE APPLICATION(CLIST) 
              DATASET('TCPIP.SEZAEXEC') " 
"SELECT CMD(EZASYRGO) NEWPOOL PASSLIB NEWAPPL(EZAS)" 
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)" 
"LIBDEF ISPPLIB" 
"LIBDEF ISPMLIB" 

You can execute this from ISPF option 6 or have this built into the ISPF panels.

Originally this exec was called syslogd; when I used it, I got

SYSTEM COMPLETION CODE=4C5 REASON CODE=77A53217

Where 4C5 is TCPIP, and 3217 – the program has the wrong AC (APF related). This is because there is a command syslogd which was executed in preference to my exec. When I renamed the exec to MYSYSLOG it used the exec and it worked fine!

The first panel is

EZASYP01 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 1
Command ===>                                                  Scroll ===> PAGE
                                                                               
Enter syslogd browser options                                                  
  Recall migrated data sets ==> NO     (Yes/No) Recall data sets or not        
  Maximum hits to display   ==> 200    (1-99999) Search results to display     
  Maximum file archives     ==> 30     (0-400) Days to look for file archives  
  Display start date/time   ==> YES    (Yes/No) Retrieve start date/time       
  Display active files only ==> NO     (Yes/No) Active files only, no archives 
  DSN Prefix override value ==>                                                
                                                                               
Enter file or data set name of syslogd configuration, or select one from below:
                                                                               
  File/DS Name ==> /etc/syslog.conf
                                                                               
Press ENTER to continue, or press END PF key to exit without a selection       
                                                                               
Line commands: S Select, R Remove from list, B Browse content, E Edit content  
                                                                               
Cmd Recently used syslogd configuration file or data set name                  
--- -------------------------------------------------------------------------- 
    /etc/syslog.conf                                                          
******************************* Bottom of data ********************************

Pressing enter, gave me another panel with

EZASYP00 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 6
OPTION ===>                                                   Scroll ===> PAGE
                                                                               
Select one of the following, or press END PF key to exit the syslogd browser   
                                                                               
  1 Change current syslogd configuration file and/or options                   
  2 Guide me to a possible syslogd destination                                 
  3 Clear guide-me hits (indicated by ==> in the Cmd column)                   
  4 Search across all active syslogd files                                     
                                                                               
Current config file ==> /etc/syslog.conf                                      
                                                                               
Line commands: B Browse, A List archives, S Search active file and archives,   
               SF Search active file, SA Search archives, I File/DSN info      
                                                                    Archive    
Cmd Rule/Active UNIX file name                    Start Time        Type Avail.
--- --------------------------------------------- ----------------- ---- ------
    *.SYSLOGD*.*.*                                28 May 2022 13:31 None 0     
    /var/log/syslogd                                                          
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    *.INETD*.*.*                                  Empty       N/A   None 0     
    /var/log/inetd                                                            
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    auth.*                                        Empty       N/A   None 0     
    /var/log/auth                                                             

I could then browse the error log for SYSLOGD.

You can search for userid, strings etc, and give date ranges.

However for my small, one person usage, I found it was easier to use Unix services and use the command

oedit /var/syslogd.log

to edit the file.

Capturing the right data

If the config file you can specify options like

*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

The entries are Userid.Jobname.facility.priority.

PAGENT AT-TLS request are reported via TCPIP.

I could not find how to filter the TCPIP data so the AT-TLS data went to one file, and other TCPIP data went to another file. For TCPIP it looks like the “facility” is either “daemon” or “auth”, which you can specify in the TTLS configuration. So not very useful.

Getting AT-TLS and PAGENT to work on z/OS – start here.

With traditional TLS applications, the application code has to issue the requests to use TLS, for example specify the keystore, and which cipher specs to use and does the encryption and decryption of the data. The application then issues TCP send and receive request as usual.

With AT-TLS, the TLS work is moved out of the application and into the TCPIP subsystem. The application just does the normal sends and receives, and TCPIP does the work of establishing the session and handling the encryption. There are rules and policies to define how the session should be established. It uses the PAGENT address space (Policy Agent) to manage the configuration.

Is it easier than having MQ or WAS Liberty do the TLS stuff? – I don’t think so. When it works it is fine. Getting it working is a challenge, because the trace and diagnostics are poor.

My other blog posts on PAGENT and AT-TLS

What is PAGENT?

Having used PAGENT to configure AT-TLS with TCPIP, I see PAGENT is a program which reads configuration information from a file – and gives the configuration to TCPIP. TCP then does the work.

General

It feels that the PAGENT setup and configuration was not designed with the z/OS environment in mind. It “breaks” so many things.

  1. You can have only one PAGENT running per LPAR – even with different name. This means you cannot have a “test” and production PAGENT in the same LPAR.
  2. PAGENT can be configured to have information on:
    1. Common Intrusion detection services (IDS).
    2. Common IP filtering, and manual and dynamic virtual private network (VPN) tunnels (IPSEC).
    3. Common Routing (Policy-based routing enables the TCP/IP stack to make routing decisions that take into account criteria other than just the destination IP address. The additional criteria can include job name, source port, destination port, protocol type (TCP or UDP), source IP address, NetAccess security zone, and multilevel secure environment security label).
    4. AT-TLS Common definitions.
    5. AT-TLS for TCPIP Image level which can have sections on
      1. IDS
      2. IPSec
      3. Qos
      4. Routing
      5. AT-TLS.
  3. As there is only one active PAGENT allowed per LPAR, you have to make your configuration changes to the production PAGENT, refresh it, and fix any configuration errors. The documentation says “make a change to production, if it doesn’t work back out the changes”!
  4. There is one initial configuration file per PAGENT, which can “include” other files. You cannot have a concatenated list of files.
  5. You cannot validate definitions before making them active. The configuration is processed only when the referenced TCPIP stack is active.
  6. Error messages do not have error message numbers, so there is no ability to look up the errors messages.
  7. It lacks good diagnostics. For example
    1. I got error message “Resource temporarily unavailable” when it could not find the security profile “EZB.INITSTACK.*.TCPIP2” on my system. The PAGENT code checks to see if the profile exists and if not, it dies quietly. It does not actually use the security profile which would cause RACF to produce a message saying missing profile.
    2. I deliberately misconfigured a file to use a file that does not exist. It just reported …processing_Stmt_TTLSConfig: processing: ‘ TTLSConfig //’USER.Z24C.TCPPARMS(BLAHBLAH)’ . It should report file not found. Some missing files get “Cannot get FILE handle for information.”

My set up

I could not find any good guidance on setting up PAGENT and AT-TLS, so I’ve documented what I did. It may not be correct…

It took about a day to understand the AT-TLS setup – as I was a typical user with typos etc which slowed me down.

Errors

I naively assumed errors would be reported in //SYSPRINT. On my system they were in /tmp/pagent.log. This file location can be configured with an Environment variable.

The output can be verbose, so I use oedit, and ISPF search

f err 15 25

to find the errors. You may find fields SYSERR or OBJERR.

When errors occur, you do not get file and line number of the error. You have to hunt around. Invalid statements are often just ignored.

With a configuration error the PAGENT job gave me a message on syslog

EZZ8438I PAGENT POLICY DEFINITIONS CONTAIN ERRORS FOR TCPIP : TTLS

In the /tmp/pagent.log file I had

05/30 07:21:12 EVENT :005: pinit_fetch_TTLS_policy_profile: Processing Image TTLS config file: ‘//’USER.Z24C.TCPPARMS(TTLS)” for image ‘TCPIP’

05/30 07:21:12 OBJERR :005: process_TTLS_attribute_table: Unknown attribute ‘ZocalAddr’ for TTLSRule

My common mistakes were

  • spelling errors for example TLSConfig instead of TTLSConfig. (I commented, then uncommented a line and lost the initial T)
  • incorrect dataset names, either the data set, or the member.

Configuration concepts

In the PAGENT configuration file, the AT-TLS specific stuff is like

CommonTTLSConfig //’USER.Z24C.TCPPARMS(TTLSCOM)’
tcpImage TCPIP //’USER.Z24C.TCPPARMS(PAGENT)’
TcpImage TCPIP2 //’USER.Z24C.TCPPARMS(PAGENTT2)’

This defines common stuff for AT-TLS in //’USER.Z24C.TCPPARMS(TTLSCOM)’, and specific TCPIP image in its own file.

The TCPIP specific file has

TTLSConfig //’USER.Z24C.TCPPARMS(TTLS2)’ FLUSH PURGE

This says the TTLS stuff is in the member TTLS2.

You can have the entry without a file or dataset name.

TTLSConfig FLUSH PURGE

This says use the definition in the CommonTTLSConfig.

You need a TTLSConfig, statement, to get AT-TLS definitions configured in the LPAR.

How to update definitions

So I did not break “production” I created a second TCPIP stack (TCPIP2), and created a configuration within PAGENT for the TCPIP2 stack. (This seems a lot of work just to validate some definitions. I raised an RFE on this, but it was declined).

When I was happy with the definitions, I merged them with the the common/production ones.

When I defined a second TCPIP (TCPIP2), the configuration statements were only parsed, when TCPIP2 was started, and so PAGENT produced the error messages once TCPIP2 was active

PAGENT has started – what next?

Pagent operator commands

You can “modify” the PAGENT address space

For example

What is my configuration?

Once you have configured PAGENT you can use the Unix command

pasearch -c 1>a

to give output like

TCP/IP pasearch CS V2R4 Image Name: TCPIP1
Date: 05/23/2022 Time: 17:34:44
PAPI Version: 14 DLL Version: 14
TTLS Policy Object:
ConfigLocation: Local LDAPServer: False
CommonFileName: //'USER.Z24C.TCPPARMS(TTLSCOM)'
ImageFileName: 


TCP/IP pasearch CS V2R4 Image Name: TCPIP2
Date: 05/23/2022 Time: 17:34:44
PAPI Version: 14 DLL Version: 14
TTLS Policy Object:
ConfigLocation: Local LDAPServer: False
CommonFileName: //'USER.Z24C.TCPPARMS(TTLSCOM)'
ImageFileName: //'USER.Z24C.TCPPARMS(TTLS2)'

The command

pasearch -f CPJES2IN > a

gave the output for just the TTLSRule CPJES2IN.

The command

pasearch -p TCPIP2 1>a

gave the configuration for just the TCPIP stack TCPIP2, including

...
policyRule:             TLSCOM 
  Rule Type:            TTLS 
...
policyRule:             TLSCP3 
  Rule Type:            TTLS 
...
policyRule:             TLSCP4 
  Rule Type:            TTLS 
...

You get the definitions – but you do not know where they came from. I happen to know that TLSCOM comes from the common definition.

A definition can be in both Common and TCPIP Image files.

Instead of relying on PAGENT to report configuration errors I used the Unix command pasearch to display the configuration.

Display the configuration for a TCPIP image

Use the Unix command pasearch to display the configuration.

pasearch -p TCPIP2 >a

Display the object types configured to PAGENT

pasearch -c 1>a

TCP/IP pasearch CS V2R4 Image Name: TCPIP
Qos Policy Object:…
Ids Policy Object:…
IPSec Policy Object:…
IpFilter Policy Object:…
KeyExchange Policy Object:…
LocalDynVpn Policy Object:…
Routing Policy Object:…
TTLS Policy Object:…

TCP/IP pasearch CS V2R4 Image Name: TCPIP2

For example

TTLS Policy Object:
ConfigLocation: Local LDAPServer: False
CommonFileName: //'ADCD.Z24C.TCPPARMS(TLSPOLY1)'
ImageFileName: //'ADCD.Z24C.TCPPARMS(TLSPOLY1)'
ApplyFlush: True PolicyFlush: True
ApplyPurge: True PurgePolicies: True
AtomicParse: True DeleteOnNoflush: False
DummyOnEmptyPolicy: True ModifyOnIDChange: False
Configured: True UpdateInterval: 1800
TTLS Enabled: True
InstanceId: 1653375346
LastPolicyChanged: Tue May 24 07:55:46 2022

Overall

PAGENT feels like it not of the standard that I would expect z/OS products to have. For example, you cannot validate changes before making them live, and the changes are only validated when the TCPIP stack is active.

This means you are making unvalidated changes to your production system!

Setting up syslogd on z/OS

Many components of TCPIP write information to syslogd. This is a process that captures the data sent to it over a UDP socket, and writes it to files in the Unix file system. If the syslogd is not active then messages may be written to the job log. When I was trying to set up AT-TLS, I had 10s of messages on the job log, each time a client tried to use AT-TLS.

The IBM documentation is not very clear, it tells you how to turn on debug, trace etc but does not clearly explain the difference, and when they are used.

It look like the PAGENT’s job is to take a configuration file, parse it, and pass the configuration data to TCPIP.

If you are using AT-TLS to set up TLS channels, the trace data comes from the TCPIP address space into syslogd.

Configure syslogd

See if SYSLOGD is running, if not, try to start it. If it does not exist…

  • Copy /usr/lpp/tcpip/samples/syslog.conf to its default configuration file /etc/syslog.conf, or another file.
  • Copy TCPIP.SEZAINST(SYSLOGD) to your proclib concatenation.
  • The program uses environment variables defined in STDENV to control operations. The default configuration file location is /etc/syslog.conf .

You can configure syslog.conf for example

*.TCPIP.*.* /var/log/%Y/%m/%d/TCPIP
*.SYSLOGD.*.* /var/log/%Y/%m/%d/syslogd
*.err /var/log/%Y/%m/%d/errors

This says all messages for SYSLOGD go to a file like /var/log/2022/05/14/syslogd, and error messages go to /var/log/2022/05/14/errors

This means you get a file of messages for each day. For me, I just used /var/log/syslogd.log and /var/log/errors.log, and deleted them periodically. My syslog.conf is

*.INETD*.*.*       /var/log/inetd 
auth.* /var/log/auth 
mail.* /var/log//mail -F 640 -D 770 
local1.err       /var/log/local1 
*.err            /var/log/errors 
*.CPAGENT.*.*       /var/log/CPAGENT 
*.TTLS*.*.*          /var/log/TTLS 
*.Pagent.*.*        /var/log/Pagent 
*.TCPIP.*.debug     /var/log/TCPIPdebug 
*.TCPIP.*.warning   /var/log/TCPIP 
*.TCPIP.*.err       /var/log/TCPIPerr 
*.TCPIP.*.info      /var/log/TCPIPinfo 
*.SYSLOGD*.*.*      /var/log/syslogd 
*.TN3270*.*.*       /var/log/tn3270 
*.SSHD*.*.*         /var/log/SSHD 

The syntax is

  • Read syntax diagramuserid.jobname.facility.priority …
  • facility.priority ….

Priority

Data logged to syslogd has a “priority”. For example AT-TLS trace level 32 (Data) have a priority of “debug”. You can use this for example

*.TCPIP.*.debug     /var/log/TCPIPdebug 
*.TCPIP.*.*         /var/log/TCPIP 

This says

  1. for messages from TCPIP with priority debug or higher( debug, info, notice, warning, error, crit, alert, emerg) then write the data to /var/log/TCPIPdebug .
  2. Write all messages to /var/log/TCPIP

As debug is the lowest level of priority, these statements are effectively the same.

It may be better to have

*.TCPIP.*.debug     /var/log/TCPIPdebug 
*.TCPIP.*.warning   /var/log/TCPIP 

The priority can be “none” which means do not log any messages.

How do I capture messages not handled else where?

This is a bit clumsy.

When a message arrives each of the rules are check. If the check is true the message is logged.

You can have compound checks separated by a semicolon for a rule.

For example

*.*.*.* ; *.TCPIP.*.none ; *.PAGENT.*.none /var/log/all.log

This says log all messages – but not from TCPIP or PAGENT. If you have have one file for 20 jobs, you need to have 20 statements with the semicolon.

You can spread the definition over several lines. The semicolon at the end of the line says read next line. The code is not smart enough to put the /var/log/all on its own line, so using a ‘dummy’ may make it easier to maintain.

*.*.*.*; 
       *.TCPIP.*.none; 
       *.PAGENT.*.none;
       *.DUMMY.*.none /var/log/all 

Capturing the right data

If the config file you can specify options like

*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

The entries are Userid.Jobname.facility.priority.

PAGENT AT-TLS request are reported via TCPIP.

I could not find how to filter the TCPIP data so the AT-TLS data went to one file, and other TCPIP data went to another file. For TCPIP it looks like the “facility” is either “daemon” or “auth”, which

ISPF interface

There is an ISPF syslog browser tool which displays information about the logs, and helps you browse the logs of interest. The documentation for this is not very good.

I got this to work by experimentation. I created an exec like MYSYSLOG

/* Rexx */ 
address ispexec 
"LIBDEF ISPMLIB DATASET ID('TCPIP.SEZAMENU') STACK" 
"LIBDEF ISPPLIB DATASET ID('TCPIP.SEZAPENU') STACK" 
address tso "ALTLIB ACTIVATE APPLICATION(CLIST) 
              DATASET('TCPIP.SEZAEXEC') " 
"SELECT CMD(EZASYRGO) NEWPOOL PASSLIB NEWAPPL(EZAS)" 
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)" 
"LIBDEF ISPPLIB" 
"LIBDEF ISPMLIB" 

You can execute this from ISPF option 6 or have this built into the ISPF panels.

Originally this exec was called syslogd; when I used it, I got

SYSTEM COMPLETION CODE=4C5 REASON CODE=77A53217

Where 4C5 is TCPIP’s abend code and 3217 – the program has the wrong Authrorization Code (APF related). This is because there is a command syslogd which was executed in preference to my exec. When I renamed the exec to MYSYSLOG it used the exec and it worked fine!

The first panel is

EZASYP01 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 1
Command ===>                                                  Scroll ===> PAGE
                                                                               
Enter syslogd browser options                                                  
  Recall migrated data sets ==> NO     (Yes/No) Recall data sets or not        
  Maximum hits to display   ==> 200    (1-99999) Search results to display     
  Maximum file archives     ==> 30     (0-400) Days to look for file archives  
  Display start date/time   ==> YES    (Yes/No) Retrieve start date/time       
  Display active files only ==> NO     (Yes/No) Active files only, no archives 
  DSN Prefix override value ==>                                                
                                                                               
Enter file or data set name of syslogd configuration, or select one from below:
                                                                               
  File/DS Name ==> /etc/syslog.conf
                                                                               
Press ENTER to continue, or press END PF key to exit without a selection       
                                                                               
Line commands: S Select, R Remove from list, B Browse content, E Edit content  
                                                                               
Cmd Recently used syslogd configuration file or data set name                  
--- -------------------------------------------------------------------------- 
    /etc/syslog.conf                                                          
******************************* Bottom of data ********************************

Pressing enter, gave me another panel with

EZASYP00 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 6
OPTION ===>                                                   Scroll ===> PAGE
                                                                               
Select one of the following, or press END PF key to exit the syslogd browser   
                                                                               
  1 Change current syslogd configuration file and/or options                   
  2 Guide me to a possible syslogd destination                                 
  3 Clear guide-me hits (indicated by ==> in the Cmd column)                   
  4 Search across all active syslogd files                                     
                                                                               
Current config file ==> /etc/syslog.conf                                      
                                                                               
Line commands: B Browse, A List archives, S Search active file and archives,   
               SF Search active file, SA Search archives, I File/DSN info      
                                                                    Archive    
Cmd Rule/Active UNIX file name                    Start Time        Type Avail.
--- --------------------------------------------- ----------------- ---- ------
    *.SYSLOGD*.*.*                                28 May 2022 13:31 None 0     
    /var/log/syslogd                                                          
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    *.INETD*.*.*                                  Empty       N/A   None 0     
    /var/log/inetd                                                            
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    auth.*                                        Empty       N/A   None 0     
    /var/log/auth                                                             

I could then browse the error log for SYSLOGD.

You can search for userid, strings etc, and give date ranges.

However for my small, one person usage, I found it was easier to use Unix services and use the command

oedit /var/syslogd.log

to edit the file.

Getting TCPIP syslogd working, and tracing PAGENT

Many components of TCPIP write information to syslogd. This is a process that captures the data sent to it over a UDP socket, and writes it to files in the Unix file system. If the syslogd is not active then messages may be written to the job log. When I was trying to set up AT-TLS, I had 10s of messages on the job log, each time a client tried to use AT-TLS.

You need to configure the syslogd procedure.

See if SYSLOGD is running, if not, try to start it. If it does not exist…

  • Copy /usr/lpp/tcpip/samples/syslog.conf to its default configuration file /etc/syslog.conf, or another file.
  • Copy TCPIP.SEZAINST(SYSLOGD) to your proclib concatenation.
  • The program uses environment variables defined in STDENV to control operations. The default configuration file location is /etc/syslog.conf

You can configure syslog.conf for example

*.SYSLOGD.*.* /var/log/%Y/%m/%d/syslogd
*.err /var/log/%Y/%m/%d/errors

This says all messages for SYSLOGD go to a file like /var/log/2022/05/14/syslogd, and error messages go to /var/log/2022/05/14/errors

This means you get a file of messages for each day. For me, I just used /var/log/syslogd.log and /var/log/errors.log, and deleted them periodically.

ISPF interface

There is an ISPF syslog browser tool which displays information about the logs, and helps you browse the logs of interest. The documentation for this is not very good.

I got this to work by experimentation. I created an exec like MYSYSLOG

/* Rexx */ 
address ispexec 
"LIBDEF ISPMLIB DATASET ID('TCPIP.SEZAMENU') STACK" 
"LIBDEF ISPPLIB DATASET ID('TCPIP.SEZAPENU') STACK" 
address tso "ALTLIB ACTIVATE APPLICATION(CLIST) 
              DATASET('TCPIP.SEZAEXEC') " 
"SELECT CMD(EZASYRGO) NEWPOOL PASSLIB NEWAPPL(EZAS)" 
address tso "ALTLIB DEACTIVATE APPLICATION(CLIST)" 
"LIBDEF ISPPLIB" 
"LIBDEF ISPMLIB" 

You can execute this from ISPF option 6 or have this built into the ISPF panels.

Originally this exec was called syslogd; when I used it, I got

SYSTEM COMPLETION CODE=4C5 REASON CODE=77A53217

Where 4C5 is TCPIP, and 3217 – the program has the wrong AC (APF related). This is because there is a command syslogd which was executed in preference to my exec. When I renamed the exec to MYSYSLOG it used the exec and it worked fine!

The first panel is

EZASYP01 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 1
Command ===>                                                  Scroll ===> PAGE
                                                                               
Enter syslogd browser options                                                  
  Recall migrated data sets ==> NO     (Yes/No) Recall data sets or not        
  Maximum hits to display   ==> 200    (1-99999) Search results to display     
  Maximum file archives     ==> 30     (0-400) Days to look for file archives  
  Display start date/time   ==> YES    (Yes/No) Retrieve start date/time       
  Display active files only ==> NO     (Yes/No) Active files only, no archives 
  DSN Prefix override value ==>                                                
                                                                               
Enter file or data set name of syslogd configuration, or select one from below:
                                                                               
  File/DS Name ==> /etc/syslog.conf
                                                                               
Press ENTER to continue, or press END PF key to exit without a selection       
                                                                               
Line commands: S Select, R Remove from list, B Browse content, E Edit content  
                                                                               
Cmd Recently used syslogd configuration file or data set name                  
--- -------------------------------------------------------------------------- 
    /etc/syslog.conf                                                          
******************************* Bottom of data ********************************

Pressing enter, gave me another panel with

EZASYP00 ----------------- z/OS CS Syslogd Browser ---------------- Row 1 of 6
OPTION ===>                                                   Scroll ===> PAGE
                                                                               
Select one of the following, or press END PF key to exit the syslogd browser   
                                                                               
  1 Change current syslogd configuration file and/or options                   
  2 Guide me to a possible syslogd destination                                 
  3 Clear guide-me hits (indicated by ==> in the Cmd column)                   
  4 Search across all active syslogd files                                     
                                                                               
Current config file ==> /etc/syslog.conf                                      
                                                                               
Line commands: B Browse, A List archives, S Search active file and archives,   
               SF Search active file, SA Search archives, I File/DSN info      
                                                                    Archive    
Cmd Rule/Active UNIX file name                    Start Time        Type Avail.
--- --------------------------------------------- ----------------- ---- ------
    *.SYSLOGD*.*.*                                28 May 2022 13:31 None 0     
    /var/log/syslogd                                                          
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    *.INETD*.*.*                                  Empty       N/A   None 0     
    /var/log/inetd                                                            
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    auth.*                                        Empty       N/A   None 0     
    /var/log/auth                                                             

I could then browse the error log for SYSLOGD.

You can search for userid, strings etc, and give date ranges.

However for my small, one person usage, I found it was easier to use Unix services and use the command

oedit /var/syslogd.log

to edit the file.

Capturing the right data

If the config file you can specify options like

*.TCPIP.*.* /var/log/TCPIP
*.*.*. /var/log/all

The entries are Userid.Jobname.facility.priority.

PAGENT AT-TLS request are reported via TCPIP.

I could not find how to filter the TCPIP data so the AT-TLS data went to one file, and other TCPIP data went to another file. For TCPIP it looks like the “facility” is either “daemon” or “auth”, which you can specify in the TTLS configuration. So not very useful.

Tracing the z/OS ZFS file system, and using ZFS commands.

I was looking into a little Java problem, and wanted to know which files were being used by my Java program. The “obvious” answer was a trace – but the IBM documentation was about 8 years out of date!

The key lesson from this post is to use commands like

f OMVS,PFS=ZFS,… instead of MODIFY ZFS,… if you have ZFS in your OMVS address space.

zFS running in the z/OS UNIX address space says

In releases before z/OS V2R2, the amount of 31-bit virtual storage that was needed by both z/OS UNIX and zFS combined would have exceeded the size of a 2 GB address space. Due to that size limitation, zFS and z/OS UNIX could not coexist in the same address space.

In z/OS V2R2, zFS caches are moved above the 2 GB bar into 64-bit storage. You can now choose to have zFS run in its own colony address space or in the address space that is used by z/OS UNIX, which is OMVS.

When running zFS in the OMVS address space, each file system vnode operation (such as creating a directory entry, removing a directory entry, or reading from a file) will have better overall performance. Each operation will take the same amount of time while inside zFS itself. The performance benefit occurs because z/OS UNIX can call zFS for each operation in a more efficient manner.

Some inherent differences exist when zFS is run in the OMVS address space.

MODIFY commands must be passed to zFS through z/OS UNIX. Use the form MODIFY OMVS,pfs=zfs,cmd. For more information, see the section on passing a MODIFY command string to a physical file system (PFS) through a logical file system (LFS) in z/OS MVS System Commands. This form of the MODIFY command can be used whether zFS is in its own address space or in the OMVS address space.

Issuing commands

So when the documentation says issue a command

Steps for tracing on zFS

If you are re-creating a problem and need to collect a zFS trace, use the following steps:

1. Allocate the trace output data set as a PDSE, RECFM=VB, LRECL=133 with a primary allocation of at least 50 cylinders and a secondary allocation of 30 cylinders.

2. Define the zFS trace output data set to zFS by either using the IOEFSPRM trace_dsn option, or dynamically by using the zfsadm config -trace_dsn command. If you use the IOEFSPRM option, zFS must be stopped and then restarted to pick up the change, unless you also dynamically activate the trace output data set with the zfsadm config -trace_dsn command.

3. When you are ready to re-create the problem, reset the zFS trace table using the MODIFY ZFS,TRACE,RESET command.

4. Re-create the problem.

5. Enter the MODIFY ZFS,TRACE,PRINT command. This formats and prints the trace table to the PDSE defined on the trace_dsn option.

You still use the Unix command to define the output destination of the trace

zfsadm config -trace_dsn ‘IBMUSER.ZFSTRACE’

but you use the following console command to cause the trace to be formatted to the file from the internal buffer.

f OMVS,PFS=ZFS,TRACE,PRINT

Easy when you know how…

Why can’t I delete the data set?

I had some data sets which I could not delete from ISPF 3;4 as I got

IEC614I SCRATCH FAILED – RC 008, DIAGNOSTIC INFORMATION IS (04060020) ..

where the error code is

04060020 – Verification of SMS flags failed; caller indicates data set is not SMS-managed, but FMT4 DSCB indicates that it is an SMS volume. This error might be caused by a damaged catalog entry. Issuing the recatalog command might clear this error.

I tried to use various DFDSS commands such move, or backup delete, but these failed on the delete.

DATA SET BACKUP.D210201.COLIN.JCL ON VOLUME A4USR1 NOT SCRATCHED, 2-08,02 

IGD17040I ERROR IN DADSM PROCESSING ON VOLUME A4USR1 FOR DATA SET  
BACKUP.D210201.COLIN.JCL HISTORIC RETURN CODE IS 8 DIAGNOSTIC INFORMATION IS 04060020              
IGD306I UNEXPECTED ERROR DURING IGGDAS02 PROCESSING RETURN CODE 4 REASON CODE 32                                             

This all looked pretty broken – until I found an easy command

//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DELETE BACKUP.D210118.USER.Z24A.CLIST
/*

That worked! Having found one way of doing it – I could take it to the next level using the MASK option.

//S1 EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DELETE BACKUP.D210215.** MASK
/*

This gave me

DELETE BACKUP.D210215.** MASK
IDC0550I ENTRY (A) BACKUP.D210215.ADCD.C.SOURCE DELETED
IDC0550I ENTRY (A) BACKUP.D210215.COLIN.C DELETED
IDC0550I ENTRY (A) BACKUP.D210215.COLIN.JCL DELETED
IDC0550I ENTRY (A) BACKUP.D210215.COLIN.RACF DELETED
IDC0550I ENTRY (A) BACKUP.D210215.COLIN.WLM DELETED
IDC0550I ENTRY (A) BACKUP.D210215.IBMUSER.JCL DELETED
IDC0550I ENTRY (A) BACKUP.D210215.IBMUSER.MQ.JCL DELETED
IDC0550I ENTRY (A) BACKUP.D210215.USER.Z24A.CLIST DELETED
IDC0550I ENTRY (A) BACKUP.D210215.USER.Z24A.PARMLIB DELETED
IDC0550I ENTRY (A) BACKUP.D210215.USER.Z24A.PROCLIB DELETED
IDC0550I ENTRY (A) BACKUP.D210215.USER.Z24A.TCPPARMS DELETED
IDC0550I ENTRY (A) BACKUP.D210215.USER.Z24A.VTAMLST DELETED
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0

I thought this was pretty useful!

Just be a bit careful and do not say DELETE BACKUP.* MASK