Getting a CTRACE

Component TRACE (CTRACE) is the z/OS system trace capability for z/OS components. Most z/OS components use it.

From “capturing a trace” perspective, there are two aspects.

  • Capturing the trace data
    • The trace can be an in-memory trace, which is available when a dump is taken. This is is often the default. For example by default a trace is enabled to capture errors, and the in-memory trace is used.
    • You can have a trace writer started task which writes to a data set. When you start the trace you give the name of the started task. Data is passed to the trace writer job. You can then use the trace data set in IPCS.
  • Enabling the trace for the component. Usually there are options you can specify, for example all entries, or just error entries and how big the in-memory trace should be.

To trace a z/OS component, you need to know the CTRACE component name, and what you want to trace.

I tried to capture a CTRACE of a z/OS component, and struggled, because I didn’t know the name of the component.

What are the trace component names?

The z/OS command

TRACE STATUS         

gave

IEE843I 16.10.22  TRACE DISPLAY 940                               
SYSTEM STATUS INFORMATION
ST=(ON,0001M,00005M) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,064K)
COMPONENT MODE COMPONENT MODE COMPONENT MODE COMPONENT MODE
--------------------------------------------------------------
CSF ON NFSC ON SYSGRS MIN SYSANT00 MIN
SYSJES2 SUB SYSRRS MIN SYSIEAVX MIN SYSSPI OFF
SYSJES SUB SYSHZS MIN SYSSMS OFF SYSAXR MIN
SYSDLF MIN SYSOPS MIN SYSXCF MIN SYSDUMP ON
SYSLLA MIN SYSXES ON SYSUNI OFF SYSCATLG MIN
SYSTTRC OFF SYSTCPDA SUB SYSRSM SUB SYSAOM MIN
SYSVLF MIN SYSTCPIP SUB SYSLOGR ON SYSOMVS MIN
SYSCEA MIN SYSWLM MIN SYSTCPIS SUB SYSTCPRE SUB
SYSIOS MIN SYSANTMN MIN SYSDMO MIN SYSIEFAL ON
SYSTCPOT SUB

I was after a CEA trace, and from the above, the name is SYSCEA. It is MIN, so is already active.

What is the trace’s status?

d trace,comp=SYSCEA

gave me

COMPONENT     MODE BUFFER HEAD SUBS                           
-------------------------------------------------------------
SYSCEA MIN 0002M
ASIDS *NONE*
JOBNAMES *NONE*
OPTIONS ERROR
WRITER *NONE*

So it is active, capturing errors, and writing to the in-memory trace (because there is no WRITER). I recognised the options as the defaults in parmlib member CTICEA00.

I had my own trace writer started task

Member CTWTR in proclib

//CTWTR PROC                                                                  
//DELETE EXEC PGM=IEFBR14
//TRCOUT01 DD DSNAME=IBMUSER.CTRACE1,
// SPACE=(CYL,(10),,CONTIG),DISP=(MOD,DELETE)
//*
//IEFPROC EXEC PGM=ITTTRCWR,TIME=999
//TRCOUT01 DD DSNAME=IBMUSER.CTRACE1,
// SPACE=(CYL,(10),,CONTIG),DISP=(NEW,CATLG)
//SYSPRINT DD SYSOUT=*

I started my CTRACE writer

TRACE CT,WTRSTART=CTWTR          

I created my own member CTICEACP in parmlib

TRACEOPTS 
ON
BUFSIZE(20m)
OPTIONS('ALL')
WTR(CTWTR)

The WTR ties up with my CTRACE writer started task name.

Stop the current trace

TRACE CT,OFF,COMP=SYSCEA

Start the CEA trace using my member

TRACE CT,ON,COMP=sysCEA,PARM=CTICEACP

Run the test

Stop the CEA trace

TRACE CT,OFF,COMP=SYSCEA

Stop the trace writer

TRACE CT,WTRSTOP=CTWTR     

The output from the CTWTR task gave me

IEF196I IEF142I CTWTR CTWTR - STEP WAS EXECUTED - COND CODE 0000         
IEF196I IGD104I IBMUSER.CTRACE1 RETAINED,
IEF196I DDNAME=TRCOUT01

which gives me the name of the data set IBMUSER.CTRACE1.

Use IPCS to look at the trace

  • option =0 to specify the name of the data set
  • =6
  • dropd
  • The above command clears out any old information about the data set
  • CTRACE COMP(SYSCEA) full

I had some data in the trace – but not for the problem I had…. so I need to try something else.

The advanced class.

You do not need to have a member in parmlib. You can use

TRACE CT,ON,COMP=SYSCEA

and do not specify a PARM. This will then prompt for the parameters, asid, jobname, writer and options.

Getting a trace from ICSF using CTRACE

I had a problem with a certificate and as part of trying to track down the problem, I traced all of the components. I found the documentation for tracing ICSF was not complete – it assumed you were going to take a dump.

You can collect a CTRACE from ICSF, and have it go to a CTRACE dataset.

Create the CTICSFxx member in parmlib

I copied CTICSF00, added the writer statement, and said collect ALL

TRACEOPTS 
          ON 
          BUFSIZE(2M) 
          wtr(ctwtr) 
          OPTIONS('ALL') 

Turn off the ICSF trace

TRACE CT,OFF,COMP=CSF   

Start the trace writing started task

See Create a procedure to collect the trace. The same trace writer can be used for all ctrace record.

TRACE CT,WTRSTART=CTWTR  

Start CTRACE

You need a parmlib member to contain the definitions I called mine CTICSF01

TRACEOPTS 
/*-----------------------------------------------------------------*/
/* ON OR OFF: PICK 1 */
/*-----------------------------------------------------------------*/
ON
/* OFF */
BUFSIZE(2M)
wtr(ctwtr)
/*-----------------------------------------------------------------*/
/* OPTIONS: NAMES OF FUNCTIONS TO BE TRACED, OR "ALL", OR "MIN" */
/*-----------------------------------------------------------------*/
/* OPTIONS( */
/* 'ALL' */
/* ,'KDSIO' */
/* ,'CARDIO' */
/* ,'SYSCALL' */
/* ,'DEBUG' */
/* ,'RDIO' */
/* ,'RDDATA' */
/* ,'MIN' */
/* ) */
OPTIONS('KDSIO','CARDIO','SYSCALL','RDIO','ALL')

Start the trace

TRACE CT,ON,COMP=CSF,PARM=CTICSF01

This gave me

TRACE CT,ON,COMP=CSF,PARM=CTICSF01
IEE252I MEMBER CTICSF01 FOUND IN USER.Z24C.PARMLIB
ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE SUCCESSFULLY EXECUTED.

Run your failing transaction

Stop ctrace

TRACE CT,OFF,COMP=CSF 

This gave

ITT038I ALL OF THE TRANSACTIONS REQUESTED VIA THE TRACE CT COMMAND WERE SUCCESSFULLY EXECUTED.

Stop the trace writer

TRACE CT,WTRSTOP=CTWTR

This gave messages

ITT111I CTRACE WRITER CTWTR TERMINATED BECAUSE OF A WTRSTOP REQUEST.
IEF404I CTWTR – ENDED – TIME=07.54.33
IEF196I IEF142I CTWTR CTWTR – STEP WAS EXECUTED – COND CODE 0000
IEF196I IGD104I IBMUSER.CTRACE1 RETAINED

Any data is in the dataset.

Go into IPCS.

The instructions below show how to find statements with a non zero return code.

  • =0 – and enter the data set name
  • =6 – to get to the ipcs command screen
  • dropd – to tell IPCS to forget any information it may know about for the dataset
  • CTRACE COMP(CSF) full – this displays any ICSF CTRACE data
  • m PF8 – go to the bottom of the data
  • report view – to go into ISPF View mode on the data set
  • X ALL – to hide all of the data
  • f code all – this shows any error codes
  • c ‘Return code = 00000000 ‘Return ZZZZ = 00000000’ all – use cut and paste of the string, as blanks are lost in this post
  • c ‘Condition code = 00000000’ ‘Condition ZZZZ = 00000000’ all
  • x all
  • f code all – this will show you are statements with a non zero return code.