Using irrseq00 to extract profile information from RACF

IRRSEQ00 also known as R_ADMIN can be used by an application to issue RACF commands, or extract information from RACF. It is used by the Python interface to RACF pysear.

Using this was not difficult – but has it challenges (including a designed storage leak!).

I also had a side visit into That’s strange – the compile worked.

Challenges

The documentation explains how to search through the profiles.

The notes say

When using extract-next to return all general resource profiles for a given class, all the discrete profiles are returned, followed by the generic profiles. An output flag indicates if the returned profile is generic. A flag can be specified in the parameter list to request only the generic profiles in a given class. If only the discrete profiles are desired, check the output flag indicating whether the returned profile is generic. If it is, ignore the entry and terminate your extract-next processing.

  • To search for all of the profiles, specify a single blank as the name, and use the extract_next value.
  • There are discrete and generic profiles. If you specify flag bit 0x20000000 For extract-next requests: return the next alphabetic generic profile. This will not retrieve discrete profiles. If you do not specify this bit, you get all profiles.

This is where it gets hard.

  • The output is returned in a buffer allocated by IRRSEQ00. This is the same format as the control block used to specify parameters. After a successful request, it will contain the profile, and may return all of the segments (such as a userid’s TSO segment depending on the option specified).
  • Extract the information you are interested in.
  • Use this data as input to the irrseq00 call. I set used pParms = buffer;
  • After the next IRRSEQ00 request FREE THE STORAGE pointed to by pParms.
  • Use the data returned to you in the new buffer.
  • Loop

The problems with this are

The documentation says

The output storage is obtained in the subpool specified by the caller in the Out_message_subpool parameter. It is the responsibility of the caller to free this storage.

I do not know how to issued a FREEMAIN/STORAGE request from a C program! Because you cannot free the z/OS storage from C, you effectively get a storage leak!

I expect the developers did not think of this problem. Other RACF calls get the back in the same control block, and you get a return code if the control block is too small.

I solved this by having some assembler code in my C program see Putting assembler code inside a C program.

My program

Declare constants

 #pragma linkage(IRRSEQ00 ,OS) 
// Include standard libraries */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <inttypes.h>

int main( int argc, char *argv??(??))
{
// this structure taken from pysear is the parameter block
typedef struct {
char eyecatcher[4]; // 'PXTR'
uint32_t result_buffer_length; // result buffer length
uint8_t subpool; // subpool of result buffer
uint8_t version; // parameter list version
uint8_t reserved_1[2]; // reserved
char class_name[8]; // class name - upper case, blank pad
uint32_t profile_name_length; // length of profile name
char reserved_2[2]; // reserved
char volume[6]; // volume (for data set extract)
char reserved_3[4]; // reserved
uint32_t flags; // see flag constants below
uint32_t segment_count; // number of segments
char reserved_4[16]; // reserved
// start of extracted data
char data[1];
} generic_extract_parms_results_t;
// Note: This structure is used for both input & output.

Set up the irrseq00 parameters

I want to find all profiles for class ACCTNUM. You specify a starting profile of one blank, and use the get next request.

char work_area[1024]; 
int rc;
long SAF_RC,RACF_RC,RACF_RS;
long ALET = 0;

char Function_code = 0x20; // Extract next general resource profile
// RACF is ignored for problem state
char RACF_userid[9];
char * ACEE_ptr = 0;
RACF_userid[0]=0; // set length to 0

char Out_message_subpool = 1;
char * Out_message_string; // returned by program

generic_extract_parms_results_t parms;
memset(&parms,0,sizeof(parms));
memcpy(&parms.eyecatcher,"PXTR",4);
parms.version = 0;
memcpy(&parms.class_name,"ACCTNUM ",8);
parms.profile_name_length = 1;
parms.data[0] =' ';

char *pParms = (char *) & parms;
Function_code = 0x20; // get next resource
int i;
generic_extract_parms_results_t * pGEP;

Loop round getting the data

I knew there were only 3 discrete profiles and one generic (with a “*” in it) .

For extract-next requests, SAF_RC = 4, RACFRC = 4 and RACFRS = 4, means. there are no more profiles that the caller is authorised to extract.

 for (i=0;i < 6;i++)
{
parms.flags = 0x04000000; // get next + base only
rc=IRRSEQ00(
&work_area,
&ALET , &SAF_RC,
&ALET , &RACF_RC,
&ALET , &RACF_RS,
&Function_code,
pParms,
&RACF_userid,
&ACEE_ptr,
&Out_message_subpool,
&Out_message_string
);
pParms = Out_message_string;

pGEP = (generic_extract_parms_results_t *) pParms;
if (RACF_RC == 0 )
{
printf("return code SAF %d RACF %d RS %d %2.2x %8.8x %*.*s \n",
SAF_RC,RACF_RC,RACF_RS, Function_code, pGEP-> flags,
pGEP->profile_name_length,
pGEP-> profile_name_length, pGEP->data);
}
else
{
printf("return code SAF %d RACF %d RS %d \n",
SAF_RC,RACF_RC,RACF_RS );
break;
}
}
return 8;
}

The results were

return code SAF 0 RACF 0 RS 0 20 00000000  ACCT# 
return code SAF 0 RACF 0 RS 0 20 00000000 IZUACCT
return code SAF 0 RACF 0 RS 0 20 10000000 TESTGEN*
return code SAF 4 RACF 4 RS 4

For TESTGEN* the flag is 0x10 which is On output: indicates that the profile returned by RACF is generic. When using extract-next to cycle through profiles, the caller should not alter this bit. For the others this bit is off, meaning the profiles are discrete.

Putting assembler code inside a C program

I was using a RACF service, and the documentation casually says “the application must free the returned storage”.

C does not provide facilities to use the STORAGE RELEASE z/OS service, so I had to write some assembler code to do this. It wasn’t difficult, I just fell over many little problems.

The IBM extension to the C language ASM is defined in the documentation.

I think of asm() as a special macro.

  • You pass in strings of assembler code. asm() will format it, and split the line if it would be longer than 72 characters.
  • If you want multiple lines you should end each line with \n (standard printf in C).
  • Lines starting with “*” are treated as comments, and produce no code
  • You can start your line with just one blank, before the instruction and it will be formatted as properly aligned assembler code, wrapping onto new lines if needed.
  • You do not use the C variable names in the assembler source.
    • You specified a mapping like [length] “m”(lDeletestorage), it takes the C variable lDeletestorage, determines its storage address such as it is offset 40 from register 5 (40(5)) and assigns this value to “length”
    • In the assembler code you specify ” L 2,%[length] \n” and it substitutes the value for %[length]. This instruction becomes ” L 2,40(5) “
    • At first glance it you would think it should be able to substitute the value directly, but you need to specify meta information about the field (is it used read only or read/write, is it a storage location, or a literal string). This is why there is this indirection
  • You need to specify which registers are modified so the C code can either save/restore their values, or just use different registers.

My code

if ( pDeletestorage != 0 )
{
int freerc = 0;
#define ASM_PREFIX " SYSSTATE ARCHLVL=2 \n"
asm(
ASM_PREFIX
" L 2,%[length] \n"
" L 4,%[a] \n"
" STORAGE RELEASE,LENGTH=(2),ADDR=(4),COND=YES,RTCD=%[rc] "
: [rc] "+m"(freerc) //* output
: [length] "m"(lDeletestorage,
[a] "m"(pDeletestorage)
:"r0", "r1" , "r2" , "r4" , "r15" );
printf("Storage release rc %i\n",rc);
}
  • if ( pDeletestorage != 0 ) If there is a block of storage to release
  • {
  • int freerc = 0; define and preset the return code
  • #define ASM_PREFIX ” SYSSTATE ARCHLVL=2 \n” see below. The \n makes a new line
  • asm( generate the code
  • ASM_PREFIX insert the SYSTATE code
  • ” L 2,%[length] \n” Load register 2 with the value of the variable length defined below
  • ” L 4,%[a] \n” Load register 4 with the value of the variable a(ddress) defined below
  • ” STORAGE RELEASE,LENGTH=(2),ADDR=(4),COND=YES,RTCD=%[rc] “ issue the storage macro
  • : After the first : is a list of output variables
    • [rc] “+m”(freerc) //* output [rc] matches the name in the RTCD=%[rc]
      • “+” means Indicates that the operand is both read and written by the instruction
      • “m” is use a memory object – that is, a variable. Other options are literal values or registers.
      • (freerc) is the name of the C variable
  • : After the second : is a list of input variables
    • [length] “m”(lDeletestorage),
    • [a] “m”(pDeletestorage)
  • : After the third : is a list of register that are changed (clobbered)
    • “r0”, “r1” , “r2” , “r4” , “r15”
  • );
  • printf(“Storage release rc %i”,rc);
  • }

You have to pass the data to the macro using registers. You can either select registers yourself, or let the compiler pick them for you. See Using registers with my program as a more correct way of coding the program.

Note: If you pick the registers yourself, they may be unavailable if you compile it as a 64 bit program or as XPLINK, so using Using registers with my program is better.g

With the SYSSTATE ARCHLVL=2

This statement sets the minimum architecture to  z/Architecture (which includes Jump instructions).

The generated code was

* SYSSTATE ARCHLVL=2                                              
* THE VALUE OF SYSSTATE IS NOW SET TO ASCENV=P AMODE64=NO ARCHLVX01-SYSS
* L=2 OSREL=00000000 RMODE64=NO
L 2,1368(13)
L 4,1364(13)

STORAGE RELEASE,LENGTH=(2),ADDR=(4),COND=YES,RTCD=1372(13)
LR 0,2 .STORAGE LENGTH
LR 1,4 .ADDRESS OF STORAGE
LHI 15,X'0001' .Add in parameters
L 14,16(0,0) .CVT ADDRESS
L 14,772(14,0) .ADDR SYST LINKAGE TABLE
L 14,204(14,0) .OBTAIN LX/EX FOR RELEASE
PC 0(14) .PC TO STORAGE RTN
ST 15,1372(13) .SAVE RETURN CODE

Where

  L     2,1368(13)       
L 4,1364(13)

is loading the registers with the variable data from the C code,

and

  ST     15,1372(13)                  .SAVE RETURN CODE      

saves the return code into the C variable.

Without the SYSSTATE ARCHLVL=2

The code failed to compile because of addressability issues.

         L     2,1368(13)                                              
L 4,1364(13)
STORAGE RELEASE,LENGTH=(2),ADDR=(4),COND=YES,RTCD=1372(13)
CNOP 0,4
B IHB0001B .BRANCH AROUND DATA
*** ASMA307E No active USING for operand IHB0001B
IHB0001F DC BL1'00000000'
DC AL1(0*16) .KEY
DC AL1(0) .SUBPOOL
DC BL1'00000001' .FLAGS
IHB0001B DS 0F
LR 0,2 .STORAGE LENGTH
LR 1,4 .ADDRESS OF STORAGE
L 15,IHB0001F .CONTROL INFORMATION
*** ASMA307E No active USING for operand IHB0001F
L 14,16(0,0) .CVT ADDRESS
L 14,772(14,0) .ADDR SYST LINKAGE TABLE
L 14,204(14,0) .OBTAIN LX/EX FOR RELEASE
PC 0(14) .PC TO STORAGE RTN
ST 15,1372(13) .SAVE RETURN CODE

In days of old you had base registers to locate data and instructions. Data reference was relative to (USING) a base register. Branching within a program used one or more base register. A re-entrant program would get dynamic storage, and this would be addressed by its own base register.

Modern instructions include the Jump instruction. This says jump this many half words to the instruction. These instructions do not need a base register.

With SYSSTATE ARCHLVL=2, the generated code for the C part of the program used the jump instructions, and did not need a base register. Assembler macros that generate code the old way, need a base register.

Using old instruction, to load a value into a register, it was loaded from storage – which needed a base register to locate the storage. For example

IHB0001F DC     BL1'00000000'                                          
DC AL1(0*16) .KEY
DC AL1(0) .SUBPOOL
DC BL1'00000001' .FLAGS
...
L 15,IHB0001F .CONTROL INFORMATION

Modern instructions have the “constant” value as part of the instruction

 LHI    15,X'0001'             

This effectively clears register 15 and loads the value 0x0001 into the bottom. (To be accurate it moves 0x0001 to the bottom half, then propagates the sign bit to the upper half word. The sign bit is 0.)

Using registers

The z/OS uses registers 14,15,0,1 for linkage, and these are likely to be used when using macros. Register 3 was used by the C code to locate storage, so I used registers 2 and 4.

Using variables

Reading the documentation, it seems I should be able to say

 " STORAGE RELEASE,LENGTH=%[length],ADDR=(4),COND=YES,RTCD=%[rc] "
: ...
: [length] "m"(lDeletestorage),
[a] "m"(pDeletestorage)
:...

passing in a variable. This does not work, because C does not pass in “lDeletestorage”, but the offset and register.

For example the storage for variable lDeleteme is 1368 off register 13.

It can be used in a Load instruction

  L     2,1368(13)  

But not every where. In the macro it get substituted.

STORAGE RELEASE,LENGTH=1368(13),ADDR=(4),COND=YES,RTCD=1372(13)
L 0,=A(1368(13)) .STORAGE LENGTH
*** ASMA035S Invalid delimiter - 1368(13)

Passing back the return code using RTCD=%[rc]. The macro checks to see if the value of RTCD is a register, if not then use the value in a store instruction.

Using registers

You can specify that you want to use a register, and the ASM() will pick registers for you.

 int res = 0x12345678;        
int newRes = 55;
__asm(" SR %[rx],%[rx] clear \n"
" SR %[rx],%[ry] them \n"
" LR %[rx],%[ry] COLINS\n"

: [rx]"=r"(res) // output
: [ry]"r"(newRes) // input
);

This generates code

     L        r4,newRes(,r13,1380) 
* SR 2,2 clear
* SR 2,4 them
* LR 2,4 COLINS
SR r2,r2
SR r2,r4
LR r2,r4

LR r0,r2
ST r0,res(,r13,1376)

It has selected to use registers 2 and 4.

Where

  • L r4,newRes(,r13,1380) loads the input value into a register of its choice
  • *… these are comments of the coded instructions
  • SR.. LR.. are the generated instructions
  • LR r0,r2 , ST r0,res(,r13,1376) saves the variable defined as output back into C storage.
Using registers with my program

The compiler decides which registers to user ( r2 and r4). When I compiled this code as 64 bit (using option lp64) it used registers 6 and 7.

asm( ASM_PREFIX 
" STORAGE RELEASE,LENGTH=(%[length]),ADDR=(%[a]),COND=YES,RTCD=%[rc] "
: [rc] "+m"(freerc) //* output
: [length] "r"(lDeleteme),
[a] "r"(pDeleteme)
:"r0", "r1" , "r15" );

This generates

 *          asm( 
L r2,lDeleteme(,r13,1368)
L r4,pDeleteme(,r13,1364)
SYSSTATE ARCHLVL=2

STORAGE RELEASE,LENGTH=(2),ADDR=(4),COND=YES,RTCD=1372(13)
LR 0,2 .STORAGE LENGTH
LR 1,4 .ADDRESS OF STORAGE
LHI 15,X'0001' .Add in parameters @PCA
L 14,16(0,0) .CVT ADDRESS
L 14,772(14,0) .ADDR SYST LINKAGE TABLE
L 14,204(14,0) .OBTAIN LX/EX FOR RELEASE
PC 0(14) .PC TO STORAGE RTN
ST 15,1372(13) .SAVE RETURN CODE

Using the different data types.

This is not explained very well in the documentation.

Using literal integer constants

The code

asm( "LABEL LA     4,%[i] \n" 
:
: [i] "i"(99)
: "r4" );

Generates

 * LABEL    LA    4,99                    Colins             
LA r4,99

The label I specified on the instruction is not added to the created instruction.

Using literal string constants

Instead of using this capability, you could use C constants, and the “m” definition.

The code

asm( "LABEL LA     4,%[i] Colins \n" 
:
: [i] "i"("ABCD")
: "r4" );

Gives a compile error

10 LABEL    LA    4,ABCD                  Colins    
*** ASMA044E Undefined symbol - ABCD

The code

asm("LABEL LA     4,%[i] Colins \n" 
:
: [i] "i"("=C'ABCD'")
: "r4" );

Generates

* LABEL    LA    4,=C'ABCD'              Colins          
LA r4,0(,r3)
...
Start of ASM Literals
000360 C1C2C3C4 =C'ABCD'

Using a memory operand that is offsetable (o)

The code

asm("LABEL LA     4,%[i] Colins \n" 
"LABEL2 LA 4,%[j] Colins2\n"
:
: [i] "o"(res),
[j] "m"(res)
: "r4" );

produced

* LABEL    LA    4,1376(13)              Colins      
* LABEL2 LA 4,1376(13) Colins2
LA r4,1376(r13,)
LA r4,1376(r13,)

So it looks like the “o” operand is the same as specifying “m”.

64 bit programming

When you compile in 64 bit code (option lp64). The asm() works just as well It is better to use Using registers with my program so you do not have to guess which registers you can use.

My code

asm( ASM_PREFIX 
" STORAGE RELEASE,LENGTH=(%[l]),ADDR=(%[a]),COND=YES,RTCD=%[rc] "
: [rc] "+m"(freerc) //* output
: [l] "r"(lDeleteme),
[a] "r"(pDeleteme)
:"r0", "r1" , "r15"
);

Generated

    LLGF     r6,lDeleteme(,r4,3504) 
LG r7,pDeleteme(,r4,3496)
SYSSTATE ARCHLVL=2
...

Where the LLGF loads the 32 bit length variable into register 6, and LG loads the 64 bit address variable into register 7.

Compiling the code

I used the shell script

name=irrseq
export _C89_CCMODE=1
p1="-Wc,arch(8),target(zOSV2R3),list,source,ilp32,gonum,asm,float(ieee)"
p5=" -I. "


p7="-Wc,ASM,ASMLIB(//'SYS1.MACLIB') -Wa,LIST,RENT"

p8="-Wc,LIST(c.lst),SOURCE,NOWARN64,XREF,SHOWINC "

xlc $p1 $p5 $p7 $p8 -c $name.c -o $name.o

l1="-Wl,LIST,MAP,XREF "
/bin/xlc $name.o -o irrseq -V $l1 1>a

The important line, p7, contains

  • -Wc,ASM,ASMLIB(//’SYS1.MACLIB’) C compile options
    • ASM Enables inlined assembly code inside C/C++ programs.
    • ASMLIB Specifies assembler macro libraries to be used when assembling the assembler source code.
  • -Wa,LIST,RENT” Assembler options
    • LIST Instructs the assembler to produce a listing
    • RENT Specifies that the assembler checks for possible coding violations of program reenterability

Can I automate new SSH connections to z/OS?

I’m running on Linux, and using remote z/OS systems. Being from a performance background I hate having to waste seconds, manually starting SSH sessions to my backend systems.

I found I can automate this!

From my gnome-terminal I can issue the command

gnome-terminal --tab --working-directory=~ --title=COLINS --profile=blue  -- ssh colin@10.1.1.2 

This

  • created a new terminal session as a tab in the existing terminal window
  • did a cd ~
  • called the tab COLINS
  • selected the profile called blue (go to the hamburger of your current terminal and select profile to see what profiles you have)
  • executes the command after the ‐‐, so executes ssl colin@10.1.1.1

You can issue the command

gnome-terminal --help-all

to get a list of all options.

That’s strange – the compile worked.

I was setting up a script to compile some C code in Unix Services, and it worked – when I expected the bind to fail because I had not specified where to find a stub file.

How to compile the source

I used a shell script to compile and bind the source. I was surprised to see that it worked, because it needed some Linkedit stubs from CSSLIB. I thought I needed

export _C89_LSYSLIB=”CEE.SCEELKEX:CEE.SCEELKED:CBC.SCCNOBJ:SYS1.CSSLIB”

but it worked without it.

The script

name=irrseq 

export _C89_CCMODE=1
# export _C89_LSYSLIB="CEE.SCEELKEX:CEE.SCEELKED:CBC.SCCNOBJ:SYS1.CSSLIB"
p1="-Wc,arch(8),target(zOSV2R3),list,source,ilp32,gonum,asm,float(ieee)"
p5=" -I. "
p8="-Wc,LIST(c.lst),SOURCE,NOWARN64,XREF,SHOWINC "

xlc $p1 $p5 $p8 -c $name.c -o $name.o
# now bind it
l1="-Wl,LIST,MAP,XREF "
/bin/xlc $name.o -o irrseq -V $l1 1>binder.out

The binder output had

XL_CONFIG=/bin/../usr/lpp/cbclib/xlc/etc/xlc.cfg:xlc 
-v -Wl,LIST,MAP,XREF irrseq.o -o./irrseq
STEPLIB=NONE
_C89_ACCEPTABLE_RC=4
_C89_PVERSION=0x42040000
_C89_PSYSIX=
_C89_PSYSLIB=CEE.SCEEOBJ:CEE.SCEECPP
_C89_LSYSLIB=CEE.SCEELKEX:CEE.SCEELKED:CBC.SCCNOBJ:SYS1.CSSLIB

Where did these come from? – I was interested in SYS1.CSSLIB. It came from xlc config file below.

xlc config file

By default the compile command uses a configuration file /usr/lpp/cbclib/xlc/etc/xlc.cfg .

The key parts of this file are

* FUNCTION: z/OS V2.4 XL C/C++ Compiler Configuration file
*
* Licensed Materials - Property of IBM
* 5650-ZOS Copyright IBM Corp. 2004, 2018.
* US Government Users Restricted Rights - Use, duplication or
* disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
*

* C compiler, extended mode
xlc: use = DEFLT
...* common definitions
DEFLT: cppcomp = /usr/lpp/cbclib/xlc/exe/ccndrvr
ccomp = /usr/lpp/cbclib/xlc/exe/ccndrvr
ipacomp = /usr/lpp/cbclib/xlc/exe/ccndrvr
ipa = /bin/c89
as = /bin/c89
ld_c = /bin/c89
ld_cpp = /bin/cxx
xlC = /usr/lpp/cbclib/xlc/bin/xlc
xlCcopt = -D_XOPEN_SOURCE
sysobj = cee.sceeobj:cee.sceecpp
syslib = cee.sceelkex:cee.sceelked:cbc.sccnobj:sys1.csslib
syslib_x = cee.sceebnd2:cbc.sccnobj:sys1.csslib
exportlist_c = NONE
exportlist_cpp = cee.sceelib(c128n):cbc.sclbsid(iostream,complex)
exportlist_c_x = cee.sceelib(celhs003,celhs001)
exportlist_cpp_x = ...
exportlist_c_64 = cee.sceelib(celqs003)
exportlist_cpp_64 = ...
steplib = NONE

Where the _x entries are for xplink.

It is easy to find the answer when you know the solution.

Note:

Without the export _C89_CCMODE=1

I got

IEW2763S DE07 FILE ASSOCIATED WITH DDNAME /0000002 CANNOT BE OPENED
BECAUSE THE FILE DOES NOT EXIST OR CANNOT BE CREATED.
IEW2302E 1031 THE DATA SET SPECIFIED BY DDNAME /0000002 COULD NOT BE
FOUND, AND THUS HAS NOT BEEN INCLUDED.
FSUM3065 The LINKEDIT step ended with return code 8.

Compiling in 64 bit

It was simple to change the script to compile it in 64 bit mode, but overall this didn’t work.

p1="-Wc,arch(8),target(zOSV2R3),list,source,lp64,gonum,asm,float(ieee)" 
...
l1="-Wl,LIST,MAP,XREF -q64 "

When I compiled in 64 bit mode, and tried to bind in 31/32 bit mode (omitting the -q64 option) I got messages like

 IEW2469E 9907 THE ATTRIBUTES OF A REFERENCE TO isprint FROM SECTION irrseq#C DO
NOT MATCH THE ATTRIBUTES OF THE TARGET SYMBOL. REASON 2
...
IEW2469E 9907 THE ATTRIBUTES OF A REFERENCE TO IRRSEQ00 FROM SECTION irrseq#C
DO NOT MATCH THE ATTRIBUTES OF THE TARGET SYMBOL. REASON 3

IEW2456E 9207 SYMBOL CELQSG03 UNRESOLVED. MEMBER COULD NOT BE INCLUDED FROM
THE DESIGNATED CALL LIBRARY.
...
IEW2470E 9511 ORDERED SECTION CEESTART NOT FOUND IN MODULE.
IEW2648E 5111 ENTRY CEESTART IS NOT A CSECT OR AN EXTERNAL NAME IN THE MODULE.

IEW2469E THE ATTRIBUTES OF A REFERENCE TO … FROM SECTION … DO
NOT MATCH THE ATTRIBUTES OF THE TARGET SYMBOL. REASON x

  • Reason 2 The xplink attributes of the reference and target do not match.
  • Reason 3 Either the reference or the target is in amode 64 and the amodes do not match. The IRRSEQ00 stub is only available in 31 bit mode, my program was 64 bit amode.

IEW2456E SYMBOL CELQINPL UNRESOLVED. MEMBER COULD NOT BE INCLUDED
FROM THE DESIGNATED CALL LIBRARY.

  • The compile in 64 bit mode generates an “include…” of the 64 bit stuff needed by C. Because the binder was in 31 bit, it used the 31 bit libraries – which did not have the specified include file. When you compile in 64 bit mode you need to bind with the 64 bit libraries. The compile command sorts all this out depending on the options.
  • The libraries used when binding in 64 bit mode are syslib_x = cee.sceebnd2:cbc.sccnobj:sys1.csslib. See the xlc config file above.

IEW2470E 9511 ORDERED SECTION CEESTART NOT FOUND IN MODULE.
IEW2648E 5111 ENTRY CEESTART IS NOT A CSECT OR AN EXTERNAL NAME IN THE MODULE.

  • Compiling in 64 bit mode, generates an entry point of CELQSTRT instead of CEESTART, so the binder instructions for 31 bit programs specifying the entry point of CEESTART will fail.

Overall

Because IRRSEQ00 only comes in a 31 bit flavour, and not a 64 bit flavour, I could not call it directly from a 64 bit program, and I had to use a 32 bit compile and bind.

Accessing SMF Real Time data.

The traditional way of processing SMF data (product statistics, and audit information), is to post-process SMF datasets. This might be done hourly, or daily, (or more frequently). This means there is a delay between the records being created, and being available for processing.

With SMF Real Time, you can connect an application program to SMF, and get records from the SMF buffers, as the records are created.

Configuring SMF

SMF needs to be in logstream mode. See Many is so last year – logstreams is the way to go.

You need to configure SMF to generate the records. See the SMFPRMxx parameter in parmlib.

I created an entry dynamically using

setsmf inmem(IFASMF.INMEM,RESSIZMAX(128M),TYPE(30,42))   

Note: 128M is the smallest buffer size.

The IBM documentation Defining in-memory resources covers various topics.

Displaying information

The command

D SMF

gave me

IFA714I 11.46.50 SMF STATUS 101                
LOGSTREAM NAME BUFFERS STATUS
A-IFASMF.DEFAULT 0 CONNECTED
A-IFASMF.COLIN 0 CONNECTED
A-IFASMF.INMEM 4826066 IN-MEMORY

The command

 D SMF,M

Gave showed my Real time, in Memory resource in use

d smf,m                                                   
IFA714I 11.48.15 SMF STATUS 109
IN MEMORY CONNECTIONS
Resource: IFASMF.INMEM
Con#: 0001 Connect Time: 2026.019 10:07:20
ASID: 004B
Con#: 0002 Connect Time: 2026.019 11:48:10
ASID: 0049

The Application Programming Interface.

The API is pretty easy to use. I based my C application on the IBM example.

I called my program from Python, so that was an extra challenge.

Query

You can issue the query API request. This returns the name of the INMEM definitions available to you, and the SMF record types in the definition.

Capture the data

You need to issue

  • connect, passing the name of the INMEM definition. It returns a 16 byte token. Once the connect has completed successfully, SMF will capture the data in a buffer.
  • get, passing the token. You can specify a flag saying blocking – so the thread waits until data is available. You do not get records from before the connect.
  • If there is too much data for your application to process – or your application is slow to process the data, SMF will wrap the data, and so lose records. The application will get return code IFAINMMissedData (Meaning: Records were skipped due to buffer re-use—that is, wrapping of the data in the in-memory resource. In this case, the output buffer might not contain a valid record.) You should reissue the get.
  • disconnect, passing the token. The disconnect can be done on a different thread. If so, it notifies any thread in a blocking get request, which gets a return code IFAINMGetForcedOut.

Problems

The problems I originally had were that my SMF was not running in log stream mode.
Once I set this up, I could get data back.

I set up INMEM record for SMF 30 records, and although I submitted some batch jobs, I did not get any SMF 30 records in my program.
If I logged off TSO, I got a record. If I issued tso omvs from ISPF I got records.

I added

SUBSYS(JES2) 

to my SMFPRMLS member, and I got SMF 30 records for batch jobs.

I later changed this to be

SUBSYS(JES2,EXITS(IEFU29,IEFU83,IEFU84,IEFUJP,IEFUSO))

to be consistent wit the SUBSYS(STC… parameter)
I got SMF 30 records when logging on using SSH, from using TSO OMVS, or spawning a thread in OMVS to run in the background, for example ls &

It is curious that I do not have SUBSYS(TSO) defined – but I get entries for TSO usage.

It is OK, but…

The code works and generates records. One problem I have is how to stop my program running.

You could use a non blocking call, loop around getting records until you get no records available, then return, do an external wait, and then reloop. This puts the control in your application, but does use CPU as it loops periodically (every second perhaps) looking for records.

You could use a blocking call where the request waits until a record is available, or another thread issues the disconnect call. This means an extra programming challenge creating a thread for the blocking request to run off, and another thread to handle the disconnect request.

The first case, non blocking case, feels easier to code – but at the cost of higher CPU.

Python calling C functions

  1. You can have Python programs which are pure Python.
  2. You can call C programs that act like Python programs, using Python constructs within the C program
  3. You can call a C program from Python, and it processes parameters like a normal C program.

This blog post is about the third, calling a C program from Python, passing simple data types such as char, integers and strings.

I have based a lot of this on the well written pyzfile package by @daveyc.

The glue that makes it work is the ctypes package a “foreign function library” package.

Before you start

The blog post is called “Python calling C functions”. I tried using a z/OS stub code directly. This is not written in C, and I got.

CEE3595S DLL ... does not contain a CELQSTRT CSECT.

Which shows you must supply a C program.

The C program that I wrote, calls z/OS services. These must be defined like (or default to)

#pragma linkage(...,OS64_NOSTACK)     

Getting started

My C program has several functions including

int query() {  
return 0;
}

The compile instructions said exportall – so all functions are visible from outside of the load module.

You access this from Python using code like

lib_file = pathlib.Path(__file__).parent / "pySMFRealTime.so"
self.lib = ctypes.CDLL(str(lib_file))
...
result = self.lib.query()

Where

  • lib_file = pathlib.Path(__file__).parent / “pySMFRealTime.so” says get the file path of the .so module in the same directory as the current Python file.
  • self.lib = ctypes.CDLL(str(lib_file)) load the file and extract information.
  • result = self.lib.query() execute the query function, passing no parameters, and store any return code in the variable result

Passing simple parameters

A more realistic program, passing parameters in, and getting data back in the parameters is

int conn(const char* resource_name,  // input:  what we want to connect to
char * pOut, // output: where we return the handle
int * rc, // output: return code
int * rs, // output: reason code
int * debug) // input: pass in debug information
{
int lName = strlen(resource_name);
if (*debug >= 1)
{
printf("===resource_namen");
printHex(stdout,pFn,20);
}
...
return 0;
}

The Python code has

lib_file = pathlib.Path(__file__).parent / "pySMFRealTime.so"
self.lib = ctypes.CDLL(str(lib_file))
self.lib.conn.argtypes = [c_char_p, # the name of stream
c_char_p, # the returned buffer
ctypes.POINTER(ctypes.c_int), # rc
ctypes.POINTER(ctypes.c_int), # rs
ctypes.POINTER(ctypes.c_int), # debug
]
self.lib.conn.restype = c_int

The code to do the connection is

def conn(self,name: str,):
token = ctypes.create_string_buffer(16) # 16 byte handle
rc = ctypes.c_int(0)
rs = ctypes.c_int(0)
debug = ctypes.c_int(self.debug)
self.token = None
retcode = self.lib.conn(name.encode("cp500"),
token,
rc,
rs,
debug)
if retcode != 0:
print("returned rc",rc, "reason",rs)
print(">>>>>>>>>>>>>>>>> connect error ")
return None
print("returned rc",rc, "reason",rs)
self.token = token
return rc

The code does

  • def conn(self,name: str,): define the conn function and pass in the variable name which is a string
  • token = ctypes.create_string_buffer(16) # 16 byte handle create a 16 byte buffer and wrap it in ctypes stuff.
  • rc = ctypes.c_int(0), rs = ctypes.c_int(0), debug = ctypes.c_int(self.debug) create 3 integer variables.
  • self.token = None preset this
  • retcode = self.lib.conn( invoke the conn function
    • name.encode(“cp500”), convert the name from ASCII (all Python printable strings are in ASCII) to code page 500.
    • token, the 16 byte token defined above
    • rc, rs, debug) the three integer variables
  • if retcode != 0: print out error messages
  • print(“returned rc”,rc, “reason”,rs) print the return and reason code
  • self.token = token save the token for the next operation
  • return rc return to caller, with the return code.

Once I got my head round the ctypes… it was easy.

The C program

There are some things you need to be aware of.

  • Python is compiled with the -qascii compiler option, so all strings etc are in ASCII. The code name.encode(“cp500”), converts it from ASCII to EBCDIC. The called C program sees the data as a valid EBCDIC string (null terminated).
  • If a character string is returned, with printable text. Either your program coverts it to ASCII, or your Python calling code needs to convert it.
  • Your C program can be compiled with -qascii – or as EBCDIC(no -qascii)
    • Because Python is compiled in ASCII, the printf routines are configured to print ASCII. If your program is compiled as ASCII, printf(“ABCD”) will print as ABCD. If your program is compiled as EBCDIC printf(“ABCD”) will print garbage – because the hex values for EBCDIC ABCD are not printable as ASCII characters.
    • If your program is compiled as ASCII you can define EBCDIC constants.
      • #pragma convert(“IBM-1047”)
      • char * pEyeCatcher = “EYEC”; // EBCDIC eye catcher for control block
      • #pragma convert(pop)

Getting SSH to work to z/OS

I have two versions of z/OS, old and new(!). I had problems getting ssh to work because of key problems.

The problem

I tried to update my laptop key to the server

ssh-copy-id colin@10.1.1.2

This gave

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: ERROR: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
ERROR: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
ERROR: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
ERROR: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
ERROR: Someone could be eavesdropping on you right now (man-in-the-middle attack)!
ERROR: It is also possible that a host key has just been changed.
ERROR: The fingerprint for the ED25519 key sent by the remote host is
ERROR: SHA256:2mUOVfdSedJVQIzZiGsRkOe9Vkc1bkyuDNp5H+VrZ98.
ERROR: Please contact your system administrator.
ERROR: Add correct host key in /home/colin/.ssh/known_hosts to get rid of this message.
ERROR: Offending ED25519 key in /home/colin/.ssh/known_hosts:1
ERROR: remove with:
ERROR: ssh-keygen -f '/home/colin/.ssh/known_hosts' -R '10.1.1.2'
ERROR: Host key for 10.1.1.2 has changed and you have requested strict checking.
ERROR: Host key verification failed.

Searching the internet I got suggestions saying “delete the old line from the file”. I didn’t want to do this because it meant I would not be able to go back to the old system and work as before.

Solutions

I edited /home/colin/.ssh/known_hosts and commented out line 1, with a # at the front (the :1 above is the first line). I repeated the command and it report the same message for line :2. I commented that out as well.

I got further

colin@ColinNew:~$ ssh-copy-id colin@10.1.1.2
The authenticity of host '10.1.1.2 (10.1.1.2)' can't be established.
ED25519 key fingerprint is SHA256:2mUOVfdSedJVQIzZiGsRkOe9Vkc1bkyuDNp5H+VrZ98.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 2 key(s) remain to be installed -- if you are prompted now it is to install the new keys
colin@10.1.1.2: Permission denied (publickey,hostbased).

I had to start the SYSLOGD on z/OS to capture the output from SSHD.

In the /var/logSSHD (your’s may be different) it said

FOTS2307 User COLIN from 10.1.0.2 not allowed because not listed in AllowUsers 

In my SSHD config file /etc/ssh/sshd_config I had

# Allow specific user IDs 
AllowUsers IBMUSER

I added COLIN to the list and restarted SSHD. (I do not know how to refresh SSHD)

This time the error log had

trying public key file /u/tmp/zowet/colin/.ssh/authorized_keys 
Could not open authorized keys '/u/tmp/zowet/colin/.ssh/authorized_keys': ...

I fixed this, tried to logon, and this time it worked.

On Linux, I edited /home/colin/.ssh/known_hosts and un-commented the lines I had commented out before.
I tried the ssh command again, and it still worked!

Python calling C functions – passing structures

I’ve written how you can pass simple data from Python to a C function, see Python calling C functions.

This article explains how you can pass structures and point to buffers in the Python program. it extends Python calling C functions. It allows you to move logic from the C program to a Python program.

Using complex arguments

The examples in Python calling C functions were for using simple elements, such as Integers or strings.

I have a C structure I need to pass to a C function. The example below passes in an eye catcher, some lengths, and a buffer for the C function to use.

The C structure

typedef struct querycb {                                                         
char Eyecatcher[4]; /* Eye catcher offset 0 */
uint16_t Length; /* Length of the block 4 */
char Rsvd1[1]; /* Reserved 6 */
uint8_t Version; /* Version number 7 */
char Flags[2]; /* Flags 8 */
uint16_t Reserved8; // 10
uint32_t Count; // number returned 12
uint32_t lBuffer; // length of buffer 16
uint32_t Reservedx ; // 20
void *pBuffer; // 24
} querycb;

The Python code

# create the variables
eyec = "EYEC".encode("cp500") # char[4] eye catcher
l = 32 # uint16_t
res1 = 0 # char[1]
version = 1 # uint8_t -same as a char
flags = 0 # char[2]
res2 = 0 # uint16_t
count = 0 # uint32_t
lBuffer = 4000 # uint32_t
res3 = 0 # uint32_t
# pBuffer # void *
# allocate a buffer for the C program to use and put some data
# into it
pBuffer = ctypes.create_string_buffer(b'abcdefg',size=lBuffer)
# cast the pBuffer so it is a void *
pB = ctypes.cast(pBuffer, ctypes.c_void_p)
# use the struct.pack function. See @4shbbhhiiiP below
# @4 is 4 bytes, the eye catcher
# h half word
# bb two char fields res1, and version
# hh two half word s flags and res2
# iii three integer fields. count lBuffer and res3
# P void * pointer
# Note pB is a ctype, we need the value of it, so pB.value
p = pack("@4shbbhhiiiP", eyec,l,res1,version,flags,
res2,count,lBuffer,res3,pB.value)

#create first parm
p1 = ctypes.c_int(3) # pass in the integer 3 as an example
# create second parm
p2 = ctypes.cast(p, ctypes.c_void_p)

# invoke the function

retcode = lib.conn(p1,p2)

The C program

int conn(int * p1, char * p2) 
// int conn(int max,...)
{
typedef struct querycb {
char Eyecatcher[4]; /* Eye catcher 0 */
uint16_t Length; /* Length of the block 4 */
char Rsvd1[1]; /* Reserved 6 */
uint8_t Version; /* Version number 7 */
char Flags[2]; /* Flags 8 */
uint16_t Reserved8; // 10
uint32_t Count; // number returned 12
uint32_t lBuffer; // length of buffer 16
uint32_t Reservedx ; // 20
void *pBuffer; // 24
} querycb;

querycb * pcb = (querycb * ) p2;

printf("P1 %i\n",*p1);
printHex(stdout,p2,32);
printf("Now the structure\n")
printHex(stdout,pcb -> pBuffer,32);
return 0 ;
}

The output

P1 3
00000000 : D8D9D7C2 00200001 00000000 00000000 ..... .......... EYEC............
00000010 : 00000FA0 00000000 00000050 0901BCB0 ...........P.... ...........&....
Now the structure
00000000 : 61626364 65666700 00000000 00000000 abcdefg......... /...............
00000010 : 00000000 00000000 00000000 00000000 ................ ................

Where

  • EYEC is the passed in eye catcher
  • 00000FA0 is the length of 4000
  • 00000050 0901BCB0 is the 64 address of the structure
  • abcdefg is the data used to initialise the buffer

Observations

It took me a couple of hours to get this to work. I found it hard to get the cast, and the ctype…. functions to work successfully. There may be a better way of coding it, if so please tell me. The code works, which is the objective – but there may be better more correct ways of doing it.

Benefits

By using this technique I was able to move code from my C program to set up the structure needed by the z/OS service into C. My C program was just parse input parameters, set up the linkage for the z/OS service, and invoke the service.

If course I did not have the constants available from the C header file for the service, but that’s a different problem.

Python safely iterating

I was using a Python program to access a z/OS service, and found there were times when my code did not clean up and close the resource.

It took me an afternoon to find out how to do it. I found pyzfile by daveyc an excellent example of how to cover Python advanced topics.

pyzfile example

The documentation has

from pyzfile import *
try:
with ZFile("//'USERID.CNTL(JCL)'", "rb,type=record",encoding='cp1047') as file:
for rec in file:
print(rec)
except ZFileError as e:
print(e)

Breaking this down

Understanding the “with”

try:
with ZFile("//'USERID.CNTL(JCL)'", "rb,type=record",encoding='cp1047') as file:

...
do something with file
...
except ZFileError as e:
print(e)

When the with ZFile(…) as file: is executed the code conceptually does

  • standard set up processing
  • open the file and return the handle
  • do processing using the file handle
  • when ever it leaves the with code section perform the close activity

Note:This could have been done with

try:
open the file
...
do something
...
except:
...
finally: # do this every time
if the file was opened:
close the file

but this is not quite so tidy and compact as the with syntax

In more detail…

  • The def __init__(self,..): method is invoked and passed the parameters. It saves parameters using statements like self.p1
  • The __enter__(self): is invoked passing the instance data(self). It seems to have no other parameters.
    • In the pyzfile, the code issues return self._open(). This invokes the function _open to open the data set.
  • When the with processing completes, it invokes the function __exit__(self, exc_type, exc_value, exc_traceback): This is invoked whether the code returned normally, or got an exception.
    • In the pyzfile, the code issue executes self.close(). So however the “with” processing ends, the close is always executed

Handing errors

I’ve seen that using the “with” clause, people tend to throw exceptions when problems are found

For example with the pyfile code there is

class ZFileError(Exception):
""" ZFile exception """
def __init__(self, message: str, amrc: dict = None):
self.message = message
self.amrc = amrc
if amrc is None:
self.amrc = {}
super().__init__(self.message)

def __str__(self) -> str:
return self.message

def amrc(self):
"""
Returns the amrc dict at the time of error.

:return: The ``__amrc`` structure at the time of error.
"""
return self.amrc

class ZFile:
...
def _open(self):
...
self.handle = open...
if not self.handle:
raise ZFileError(f"Error opening file '{self.filename}':
{self.lib.zfile_strerror().decode('utf-8')}")
return self

Understanding the “for”

The code above had

    with ZFile("//'USERID.CNTL(JCL)'", "rb,type=record",encoding='cp1047') as file:
for rec in file:
print(rec)

How does the “for” work?

The key to this code are the functions

##################################################
# Iterators
##################################################
def __iter__(self):
return self

def __next__(self):
ret = self.read()
if not ret:
raise StopIteration
return ret

When the for statement is processed it processes the __next__ function. This does the work of getting the next record and returning it.

There is a lot of confusing documentation about iterators, iteration and iterables. Let’s see if my description helps clarify or just adds more confusion.

Something is iter-able of you can do iteration on it; where iteration means taking each element in turn.

In Python a list is iter-able

for l in [0,1,2,3,]
print(l)

will iterate over the list and return the element from the list

0
1
2
3

Records in a file are a bit more abstract, you cannot see the whole file, but you can say get the next record – and move through the file until there are no more records.

An iterator is the mechanism by which you iterate. Think of it as a function. The Python documentation is pretty clear.

Most people define

  def __iter__(self):
return self

For most people, just specify this. The PhD class may use something different.

The mechanism of “for” uses the __next__ function

    def __next__(self):
ret = self.read()
if not ret:
raise StopIteration
return ret

Which obtains the next element of data. If there are no more elements, then raise the StopIteration exception.

If you do not handle the StopIteration exception, then Python handles it for you and leaves the processing loop.

Conclusion

With both of these techniques “with” and “for” I could extract records from a z/OS resource.

I’ve used the “with” and “for” with yield to hide implementation detail

# create the function to read the file
def readfile(name):
try:
with ZFile(name, "rb,type=record,noseek") as file:
for rec in file:
yield rec
except ZFileError as e:
print(e)
# process the file using for ... readline()
def reader(...):
for line in readfile("//'IBMUSER.RMF'"):
do something with the data

Many is so last year – logstreams is the way to go.

I’ve been looking into the SMF Real Time, where an application program can get records directly from SMF, and not have to post-process SMF datasets or log streams. To use the real time support, SMF needs to use log streams.

What is SMF?

SMF is System Management Facility. z/OS and the subsystems can write data to SMF for post processing. Typical records are audit and accounting records from z/OS, RACF or CICS, changes to SMS, and changes to resources. Each product has one or more SMF record-type numbers allocated to it. Within each SMF record type you can have sub-types, for example the z/OS SMF 30 record has a sub-type for job start, another sub-type for job step end, and another sub-type for job end.

Display SMF options

The command

d smf

gave

   NAME                VOLSER SIZE(BLKS) %FULL  STATUS    
P-SYS1.S0W1.MAN1 B3SYS1 7200 0 ALTERNATE
S-SYS1.S0W1.MAN3 USER04 72000 1 ACTIVE

showing the dataset are being used, and giving information about the datasets

The command

d smf,o

displays all of the SMF options, and where they came from – for example a parmlib member, or from the SETSMF command.

IEE967I 08.44.41 SMF PARAMETERS 489                
MEMBER = SMFPRM00
...
SYNCVAL(00) -- DEFAULT
DUMPABND(RETRY) -- DEFAULT
INMEM(IFASMF.COLIN,TYPE(30,42),RESSIZMAX(0128M)) -- PARMLIB
SUBSYS(STC,NOTYPE(14:19,62:69,99)) -- SYS
...
STATUS(010000) -- PARMLIB
INTVAL(01) -- PARMLIB
MAXDORM(0001) -- PARMLIB
REC(PERM) -- PARMLIB
NOPROMPT -- PARMLIB
DSNAME(SYS1.S0W1.MAN3) -- PARMLIB
DSNAME(SYS1.S0W1.MAN1) -- PARMLIB

ACTIVE -- PARMLIB

The old way of recording SMF data

SMF had set of datasets it would use in turn. Typically these were named like SYS1.MANX, SYS1.MANY, or SYS1.PROD.MAN2 etc.. When the active dataset filled up, SMF would switch to the next empty dataset. You (or automation) then runs a job to either copy the records to another dataset, or post process the records; and then clear the dataset for reuse.

As computers got bigger, more work was done, more records were written and writing records to disk could not keep up.

Logstreams is the way forward.

A log stream is a stream of data which can be written to a Coupling Facility(CF) structure, or to a dataset on disk. Typically writing to a CF is faster than writing to disk.

With MANx datasets, all records were written to one dataset. With logstreams, you can configure SMF have multiple logstreams and you configure which record type(s) go to which log stream. This means you can have CICS records going to the “CICS log stream”, and RACF records going to the “RACF logstream”, and the remainder going to a default log stream.

Having multiple logstreams means data can be written to many log streams concurrently, and so avoids the bottleneck of writing to a MANx dataset.

Setting up security profiles

It took me several attempts to configure the security profiles.

Be able to define and delete logstreams

//IBMUSER1 JOB   1,MSGCLASS=H 
//KEYCERTS EXEC PGM=IKJEFT01
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RDEFINE FACILITY RESOURCE(MVSADMIN.LOGR) UACC(NONE)
permit MVSADMIN.LOGR class(FACILITY) -
access(control) ID(SYS1)
setr raclist(facility) refresh

Define individual logstreams

RDEFINE LOGSTRM IFASMF.** UACC(NONE) 
PERMIT IFASMF.** class(LOGSTRM ) -
access(ALTER ) ID(SYS1)
setr raclist(logstrm ) refresh

Giving SMF access to the logstreams

RDEFINE FACILITY IFA.IFASMF.* UACC(READ)
setr raclist(facility) refresh

Setting up logstreams

You need to set up at least one log stream. It is easy to define more and change the SMF configuation.

I used the define logstream command

//IBMLOG JOB 1,MSGCLASS=H 
//LOGDEF EXEC PGM=IXCMIAPU,REGION=4M
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DATA TYPE(LOGR) REPORT(YES)

DELETE LOGSTREAM NAME(IFASMF.DEFAULT)
DEFINE LOGSTREAM NAME(IFASMF.DEFAULT)
DESCRIPTION(SMF_LOGSTREAM)
MODEL(NO)
DASDONLY(YES)
STG_SIZE(65532)
LS_SIZE(15000)
HLQ(IXGLOGR)
HIGHOFFLOAD(80)
LOWOFFLOAD(0)
AUTODELETE(YES) /* DELETE OPTION */
OFFLOADRECALL(NO)
MAXBUFSIZE(65532)
DIAG(NO)
RETPD(1) /* DELETE 1 DAYS */
//

I also define a log stream IFASMF.COLIN

With the HLQ(IXGLOGR) definition, behind the logstreams were data sets like

Dataset                              Volume  
IXGLOGR.IFASMF.COLIN.ADCDPL *VSAM*
IXGLOGR.IFASMF.COLIN.ADCDPL.DATA USER05
IXGLOGR.IFASMF.COLIN.A0000000 *VSAM*
IXGLOGR.IFASMF.COLIN.A0000000.DATA USER04

Configure SMF

I created a member SMFPRMLS in a user.parmlib

ACTIVE                          /* ACTIVE SMF RECORDING             */ 
DSNAME(SYS1.&SYSNAME..MAN1,
SYS1.&SYSNAME..MAN3)
RECORDING(LOGSTREAM)
NOPROMPT /* DO NOT PROMPT OPERATOR */
REC(PERM) /* TYPE 17 PERM RECORDS ONLY */
MAXDORM(0001) /* WRITE IDLE BUFFER AFTER 1 SEC */
INTVAL(01) /* EVEY MINUTE */
STATUS(010000) /* WRITE SMF STATS AFTER 1 HOUR */
JWT(0400) /* 522 AFTER 30 MINUTES */
SID(&SYSNAME(1:4))
LISTDSN /* LIST DATA SET STATUS AT IPL */
DEFAULTLSNAME(IFASMF.DEFAULT)
LSNAME(IFASMF.COLIN,TYPE(30,42))

AUTHSETSMF
SYS(NOTYPE(14:19,62:69,99),EXITS(IEFU83,IEFU84,IEFACTRT,
IEFUSI,IEFUJI,IEFU29),NOINTERVAL,NODETAIL)
SUBSYS(STC,EXITS(IEFU29,IEFU83,IEFU84,IEFUJP,IEFUSO))
INMEM(IFASMF.COLI2,RESSIZMAX(128M),TYPE(30,42))

I activated it using the command

t smf=ls

When this failed, because my log stream definitions were not correct, the SMF collection defaulted to using the specified SYS1.MANx datasets.
The important bits of the SMFPRMxx file are

  • RECORDING(LOGSTREAM) – use logstreams rather than datasets
  • LSNAME(IFASMF.COLIN,TYPE(30,42)) for record types 30 and 42 write them to this log stream
  • DEFAULTLSNAME(IFASMF.DEFAULT) If there is no LSNAME for a record type – then write them to this log stream

You can issue setsmf commands to override the existing definition.

Processing SMF records

For SMF datasets

For the Use JCL like

// SET SMFPDS=SYS1.S0W1.MAN1                
// SET SMFSDS=SYS1.S0W1.MAN3
//SMFDUMP EXEC PGM=IFASMFDP
//DUMPINA DD DSN=&SMFPDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPINB DD DSN=&SMFSDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPOUT DD DISP=(NEW,CATLG),DSN=&RMF,SPACE=(CYL,(10,10))
//* DCB=(LRECL=32760,RECFM=VBS)
//* DCB=(BLKSIZE=0,LRECL=32760,RECFM=VBS)
//*UMPOUT DD DISP=SHR,DSN=IBMUSER.RMF,SPACE=(CYL,(1,1))
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
INDD(DUMPINA,OPTIONS(DUMP))
INDD(DUMPINB,OPTIONS(DUMP))
OUTDD(DUMPOUT,TYPE(42,80,30))
RELATIVEDATE(BYDAY,0,1)
START(0000)
END(2300)
/*

This processes records within the specified time range in the datasets.

For log streams

Use JCL like the following – using PGM=IFASMFDL

//IBMSMFL  JOB 1,MSGCLASS=H 
//* DUMP THE SMF DATASETS
// SET SMF=IBMUSER.SMF
//*
//S1 EXEC PGM=IEFBR14
//DUMPOUT DD DISP=(MOD,DELETE),DSN=&SMF,SPACE=(CYL,(1,1))
//*
//SMFDUMP EXEC PGM=IFASMFDL,REGION=0M
//DUMPOUT DD DISP=(NEW,CATLG),DSN=&SMF,SPACE=(CYL,(10,10))
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
LSNAME(IFASMF.COLIN,OPTIONS(DUMP))
OUTDD(DUMPOUT,TYPE(30))
RELATIVEDATE(BYDAY,0,1)
START(0000)
END(2300)
/*
//

When you specify a date range, it will read not only the active log stream datasets, but any archive ones it created, and which are available.

Display SMF

With logstream the D SMF command gave

   LOGSTREAM NAME               BUFFERS        STATUS            
A-IFASMF.DEFAULT 774 CONNECTED
A-IFASMF.COLIN 584 CONNECTED
A-IFASMF.INMEM 0 IN-MEMORY