Accessing SMF Real Time data.

The traditional way of processing SMF data (product statistics, and audit information), is to post-process SMF datasets. This might be done hourly, or daily, (or more frequently). This means there is a delay between the records being created, and being available for processing.

With SMF Real Time, you can connect an application program to SMF, and get records from the SMF buffers, as the records are created.

Configuring SMF

SMF needs to be in logstream mode. See Many is so last year – logstreams is the way to go.

You need to configure SMF to generate the records. See the SMFPRMxx parameter in parmlib.

I created an entry dynamically using

setsmf inmem(IFASMF.INMEM,RESSIZMAX(128M),TYPE(30,42))

Note: 128M is the smallest buffer size.

The IBM documentation Defining in-memory resources covers various topics.

Displaying information

The command

D SMF

gave me

IFA714I 11.46.50 SMF STATUS 101                
      LOGSTREAM NAME               BUFFERS        STATUS      
      A-IFASMF.DEFAULT                      0       CONNECTED   
      A-IFASMF.COLIN                        0       CONNECTED   
      A-IFASMF.INMEM                  4826066       IN-MEMORY

The command

 D SMF,M

Gave showed my Real time, in Memory resource in use

d smf,m                                                   
IFA714I 11.48.15 SMF STATUS 109                   
IN MEMORY CONNECTIONS                                                
  Resource: IFASMF.INMEM                                             
  Con#: 0001 Connect Time: 2026.019 10:07:20                         
  ASID: 004B                                                         
  Con#: 0002 Connect Time: 2026.019 11:48:10                         
  ASID: 0049

The Application Programming Interface.

The API is pretty easy to use. I based my C application on the IBM example.

I called my program from Python, so that was an extra challenge.

Query

You can issue the query API request. This returns the name of the INMEM definitions available to you, and the SMF record types in the definition.

Capture the data

You need to issue

connect, passing the name of the INMEM definition. It returns a 16 byte token. Once the connect has completed successfully, SMF will capture the data in a buffer.
get, passing the token. You can specify a flag saying blocking – so the thread waits until data is available. You do not get records from before the connect.
If there is too much data for your application to process – or your application is slow to process the data, SMF will wrap the data, and so lose records. The application will get return code IFAINMMissedData (Meaning: Records were skipped due to buffer re-use—that is, wrapping of the data in the in-memory resource. In this case, the output buffer might not contain a valid record.) You should reissue the get.
disconnect, passing the token. The disconnect can be done on a different thread. If so, it notifies any thread in a blocking get request, which gets a return code IFAINMGetForcedOut.

Problems

The problems I originally had were that my SMF was not running in log stream mode.
Once I set this up, I could get data back.

I set up INMEM record for SMF 30 records, and although I submitted some batch jobs, I did not get any SMF 30 records in my program.
If I logged off TSO, I got a record. If I issued tso omvs from ISPF I got records.

I added

SUBSYS(JES2)

to my SMFPRMLS member, and I got SMF 30 records for batch jobs.

I later changed this to be

SUBSYS(JES2,EXITS(IEFU29,IEFU83,IEFU84,IEFUJP,IEFUSO))

to be consistent wit the SUBSYS(STC… parameter)
I got SMF 30 records when logging on using SSH, from using TSO OMVS, or spawning a thread in OMVS to run in the background, for example ls &

It is curious that I do not have SUBSYS(TSO) defined – but I get entries for TSO usage.

It is OK, but…

The code works and generates records. One problem I have is how to stop my program running.

You could use a non blocking call, loop around getting records until you get no records available, then return, do an external wait, and then reloop. This puts the control in your application, but does use CPU as it loops periodically (every second perhaps) looking for records.

You could use a blocking call where the request waits until a record is available, or another thread issues the disconnect call. This means an extra programming challenge creating a thread for the blocking request to run off, and another thread to handle the disconnect request.

The first case, non blocking case, feels easier to code – but at the cost of higher CPU.

Many is so last year – logstreams is the way to go.

I’ve been looking into the SMF Real Time, where an application program can get records directly from SMF, and not have to post-process SMF datasets or log streams. To use the real time support, SMF needs to use log streams.

What is SMF?

SMF is System Management Facility. z/OS and the subsystems can write data to SMF for post processing. Typical records are audit and accounting records from z/OS, RACF or CICS, changes to SMS, and changes to resources. Each product has one or more SMF record-type numbers allocated to it. Within each SMF record type you can have sub-types, for example the z/OS SMF 30 record has a sub-type for job start, another sub-type for job step end, and another sub-type for job end.

Display SMF options

The command

d smf

gave

   NAME                VOLSER SIZE(BLKS) %FULL  STATUS    
 P-SYS1.S0W1.MAN1      B3SYS1      7200     0  ALTERNATE  
 S-SYS1.S0W1.MAN3      USER04     72000     1  ACTIVE

showing the dataset are being used, and giving information about the datasets

The command

d smf,o

displays all of the SMF options, and where they came from – for example a parmlib member, or from the SETSMF command.

IEE967I 08.44.41 SMF PARAMETERS 489                
        MEMBER = SMFPRM00   
        ...   
        SYNCVAL(00) -- DEFAULT                                      
        DUMPABND(RETRY) -- DEFAULT                                  
        INMEM(IFASMF.COLIN,TYPE(30,42),RESSIZMAX(0128M)) -- PARMLIB 
        SUBSYS(STC,NOTYPE(14:19,62:69,99)) -- SYS   
        ...
        STATUS(010000) -- PARMLIB            
        INTVAL(01) -- PARMLIB                
        MAXDORM(0001) -- PARMLIB             
        REC(PERM) -- PARMLIB                 
        NOPROMPT -- PARMLIB                  
        DSNAME(SYS1.S0W1.MAN3) -- PARMLIB    
        DSNAME(SYS1.S0W1.MAN1) -- PARMLIB    
        ACTIVE -- PARMLIB

The old way of recording SMF data

SMF had set of datasets it would use in turn. Typically these were named like SYS1.MANX, SYS1.MANY, or SYS1.PROD.MAN2 etc.. When the active dataset filled up, SMF would switch to the next empty dataset. You (or automation) then runs a job to either copy the records to another dataset, or post process the records; and then clear the dataset for reuse.

As computers got bigger, more work was done, more records were written and writing records to disk could not keep up.

Logstreams is the way forward.

A log stream is a stream of data which can be written to a Coupling Facility(CF) structure, or to a dataset on disk. Typically writing to a CF is faster than writing to disk.

With MANx datasets, all records were written to one dataset. With logstreams, you can configure SMF have multiple logstreams and you configure which record type(s) go to which log stream. This means you can have CICS records going to the “CICS log stream”, and RACF records going to the “RACF logstream”, and the remainder going to a default log stream.

Having multiple logstreams means data can be written to many log streams concurrently, and so avoids the bottleneck of writing to a MANx dataset.

Setting up security profiles

It took me several attempts to configure the security profiles.

Be able to define and delete logstreams

//IBMUSER1 JOB   1,MSGCLASS=H 
//KEYCERTS EXEC PGM=IKJEFT01 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN  DD * 
 RDEFINE FACILITY RESOURCE(MVSADMIN.LOGR) UACC(NONE) 
 permit  MVSADMIN.LOGR class(FACILITY)   - 
                    access(control) ID(SYS1) 
 setr raclist(facility) refresh

Define individual logstreams

RDEFINE LOGSTRM IFASMF.** UACC(NONE) 
PERMIT IFASMF.** class(LOGSTRM ) - 
                    access(ALTER  ) ID(SYS1) 
setr raclist(logstrm ) refresh

Giving SMF access to the logstreams

RDEFINE FACILITY IFA.IFASMF.* UACC(READ)
setr raclist(facility) refresh

Setting up logstreams

You need to set up at least one log stream. It is easy to define more and change the SMF configuation.

I used the define logstream command

//IBMLOG JOB 1,MSGCLASS=H 
//LOGDEF EXEC PGM=IXCMIAPU,REGION=4M 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD * 
DATA TYPE(LOGR) REPORT(YES) 
                                                                       
DELETE LOGSTREAM NAME(IFASMF.DEFAULT) 
DEFINE LOGSTREAM NAME(IFASMF.DEFAULT) 
                   DESCRIPTION(SMF_LOGSTREAM) 
                   MODEL(NO) 
                   DASDONLY(YES)         
                   STG_SIZE(65532) 
                   LS_SIZE(15000) 
                   HLQ(IXGLOGR) 
                   HIGHOFFLOAD(80) 
                   LOWOFFLOAD(0) 
                   AUTODELETE(YES)           /* DELETE OPTION */ 
                   OFFLOADRECALL(NO) 
                   MAXBUFSIZE(65532) 
                   DIAG(NO) 
                   RETPD(1)                  /* DELETE 1 DAYS */ 
//

I also define a log stream IFASMF.COLIN

With the HLQ(IXGLOGR) definition, behind the logstreams were data sets like

Dataset                              Volume  
IXGLOGR.IFASMF.COLIN.ADCDPL          *VSAM*
IXGLOGR.IFASMF.COLIN.ADCDPL.DATA     USER05
IXGLOGR.IFASMF.COLIN.A0000000        *VSAM*
IXGLOGR.IFASMF.COLIN.A0000000.DATA   USER04

Configure SMF

I created a member SMFPRMLS in a user.parmlib

ACTIVE                          /* ACTIVE SMF RECORDING             */ 
DSNAME(SYS1.&SYSNAME..MAN1, 
       SYS1.&SYSNAME..MAN3) 
RECORDING(LOGSTREAM) 
NOPROMPT                        /* DO NOT PROMPT OPERATOR           */ 
REC(PERM)                       /* TYPE 17 PERM RECORDS ONLY        */ 
MAXDORM(0001)                   /* WRITE IDLE BUFFER AFTER 1 SEC    */ 
INTVAL(01)                      /* EVEY MINUTE                      */ 
STATUS(010000)                  /* WRITE SMF STATS AFTER 1 HOUR     */ 
JWT(0400)                       /* 522 AFTER 30 MINUTES             */ 
SID(&SYSNAME(1:4)) 
LISTDSN                         /* LIST DATA SET STATUS AT IPL      */ 
DEFAULTLSNAME(IFASMF.DEFAULT) 
LSNAME(IFASMF.COLIN,TYPE(30,42)) 
AUTHSETSMF 
SYS(NOTYPE(14:19,62:69,99),EXITS(IEFU83,IEFU84,IEFACTRT, 
              IEFUSI,IEFUJI,IEFU29),NOINTERVAL,NODETAIL) 
SUBSYS(STC,EXITS(IEFU29,IEFU83,IEFU84,IEFUJP,IEFUSO)) 
INMEM(IFASMF.COLI2,RESSIZMAX(128M),TYPE(30,42))

I activated it using the command

t smf=ls

When this failed, because my log stream definitions were not correct, the SMF collection defaulted to using the specified SYS1.MANx datasets.
The important bits of the SMFPRMxx file are

RECORDING(LOGSTREAM) – use logstreams rather than datasets
LSNAME(IFASMF.COLIN,TYPE(30,42)) for record types 30 and 42 write them to this log stream
DEFAULTLSNAME(IFASMF.DEFAULT) If there is no LSNAME for a record type – then write them to this log stream

You can issue setsmf commands to override the existing definition.

Processing SMF records

For SMF datasets

For the Use JCL like

// SET SMFPDS=SYS1.S0W1.MAN1                
// SET SMFSDS=SYS1.S0W1.MAN3                
//SMFDUMP  EXEC PGM=IFASMFDP 
//DUMPINA  DD   DSN=&SMFPDS,DISP=SHR,AMP=('BUFSP=65536') 
//DUMPINB  DD   DSN=&SMFSDS,DISP=SHR,AMP=('BUFSP=65536') 
//DUMPOUT  DD   DISP=(NEW,CATLG),DSN=&RMF,SPACE=(CYL,(10,10)) 
//*             DCB=(LRECL=32760,RECFM=VBS) 
//*            DCB=(BLKSIZE=0,LRECL=32760,RECFM=VBS) 
//*UMPOUT  DD   DISP=SHR,DSN=IBMUSER.RMF,SPACE=(CYL,(1,1)) 
//SYSPRINT DD   SYSOUT=* 
//SYSIN  DD * 
  INDD(DUMPINA,OPTIONS(DUMP)) 
  INDD(DUMPINB,OPTIONS(DUMP)) 
  OUTDD(DUMPOUT,TYPE(42,80,30)) 
  RELATIVEDATE(BYDAY,0,1)
  START(0000) 
  END(2300) 
/*

This processes records within the specified time range in the datasets.

For log streams

Use JCL like the following – using PGM=IFASMFDL

//IBMSMFL  JOB 1,MSGCLASS=H 
//* DUMP THE SMF DATASETS 
// SET SMF=IBMUSER.SMF 
//* 
//S1 EXEC PGM=IEFBR14 
//DUMPOUT  DD   DISP=(MOD,DELETE),DSN=&SMF,SPACE=(CYL,(1,1)) 
//* 
//SMFDUMP   EXEC PGM=IFASMFDL,REGION=0M 
//DUMPOUT  DD   DISP=(NEW,CATLG),DSN=&SMF,SPACE=(CYL,(10,10)) 
//SYSPRINT DD   SYSOUT=* 
//SYSIN  DD * 
  LSNAME(IFASMF.COLIN,OPTIONS(DUMP)) 
  OUTDD(DUMPOUT,TYPE(30)) 
  RELATIVEDATE(BYDAY,0,1)
  START(0000) 
  END(2300) 
/* 
//

When you specify a date range, it will read not only the active log stream datasets, but any archive ones it created, and which are available.

Display SMF

With logstream the D SMF command gave

   LOGSTREAM NAME               BUFFERS        STATUS            
 A-IFASMF.DEFAULT                    774       CONNECTED         
 A-IFASMF.COLIN                      584       CONNECTED         
 A-IFASMF.INMEM                        0       IN-MEMORY

Dumping SMF data – last n day’s worth

For many years, I’ve been processing SMF data, and using the date option like DATE(2026012,2027000). Every day, I had to change it to match today’s date, and submit the job.

I’ve just discovered you can give relative dates. For example RELATIVEDATE(BYDAY,0,1), which says go back 0 days and includes 1 day – so just do today.

The output listing has, for today’s date day 19 of 2026:

IFA834I RELATIVEDATE PARAMETER RESULTS IN START DATE 2026.019, END         
                DATE 2026.019                                                
IFA836I RELATIVEDATE RANGE EXTENDS INTO FUTURE, END DATE AND TIME USED       
                IS 2026.019 11:29

You can specify BYDAY, BYWEEK, and BYMONTH.

This function has been around for years! I wonder how much time I’ve wasted on doing it the old way.