This blog post is part of a series on the zFS file system on z/OS and how to identify performance problems.
Related posts
- zFS on z/OS concepts, from a performance perspective
- How to collect zFS statistics
- Example of zFS statistics
- zFS performance reports I would like to use on z/OS (but can’t)
How to collect the statistics data.
You can collect statistics data from zFS using
- SMF type 92 records
- Using operator commands. This should not be the normal way of collecting data, as it is verbose, and does not format well
- You can display accumulated data
- You can display and reset accumulated data
- Using a batch/tso command. You can create output datasets of the information
- You can display accumulated data
- You can display and reset accumulated data
- You can display them in RMF.
- You can write your own program to extract the records. zFS provides the code of their commands.
SMF
You need to enable SMF collection using the zfsadm command. I could only get this to work through JCL.
// SET P='config -smf_recording off'
// SET P='config -smf_recording on,10'
// SET P='config -smf_recording on'
// SET P='configquery -all'
//AGGRINFO EXEC PGM=IOEZADM,REGION=0M,
// PARM=('&P')
//SYSPRINT DD SYSOUT=H
//STDOUT DD SYSOUT=H
//STDERR DD SYSOUT=H
//SYSUDUMP DD SYSOUT=H
//CEEDUMP DD SYSOUT=H
You can use
- configquery -all to display the current configuration
- config -smf_recording on,10 to produce records every 10 minutes
- config -smf_recording on to enable SMF recording on the SMF interval broadcast
- config -smf_recording off to stop the collection of SMF data
You need to check that SMF is configured to collect the SMF 92 records. The operator command d SMF,o shows what is being collected. If it reports NOTYPE(14:19,62:69,92,99) with 92 in the list of NOTYPE, then SMF 92 records will not be collected.
You use a standard SMF job to copy the SMF data for post processing. I could not find an IBM provided formatter, so I wrote one.
I could not see how to cofigure zFS to not produce the SMF 92-11 records on individual zFS usage. I think you have to disable it at the SMF interval.
Operator command
You can issue a command at the console for example
F OMVS,PFS=ZFS,QUERY,VM
or
f ZFS,QUERY,VM
There is a lot of output, and it does not always format well on the console.
Using OMVS command line
You can use the omvs command zfsadm, for example zfsadm query -iobyaggr to display the data.
You’ll need to issue a command like
zfsadm query -iobyaggr 1>output_file
To capture the output
Using Batch
I use JCL (and move the relevant SET P statement to the bottom of the list as needed).
// SET P='config -smf_recording on,10'
// SET P='/fileinfo /u/ibmuser '
// SET P='config -smf_recording on'
// SET P='configquery all'
// SET P='config -smf_recording off'
// SET P='query -iobyaggr'
//AGGRINFO EXEC PGM=IOEZADM,REGION=0M,
// PARM=('&P')
//SYSPRINT DD SYSOUT=H
//STDOUT DD SYSOUT=H
//STDERR DD SYSOUT=H
//SYSUDUMP DD SYSOUT=H
//CEEDUMP DD SYSOUT=H
The query command has many options. I think you can only pass parameters via the parm statement. You cannot pass a list of command in//SYSIN.
Command interface
For the command interface, The values displayed are accumulated until reset, for example query -reset -iobyaggr
RMF
I started RMF, then used F RMF,START III to collect additional information.
I used the TSO RMFWDM command (RMF Work Delay Monitor). This gave me RMF Monitor III Primary Menu.
Selection S SYSPLEX Sysplex reports and Data Index
Selection I ZFSOVW zFS Overview
This gave
---------- Cache Activity ------------
System -----Wait%------ ---User--- --Vnode--- -Metadata-
I/O Lock Sleep Rate Hit% Rate Hit% Rate Hit%
S0W1 24.8 1.2 6.7 694.9 98.6 569.3 97.0 743.3 99.6
This displays the user, vnode, and Metadata data cache. The rate of activity and the cache hit ratio. High(> 95%) is good. The rate is the number of get page requests a second.
If you tab to any of the numbers and press enter, it displays more information, for example
zFS Overview - User Cache Details
The following details are available for system S0W1
Press Enter to return to the Report panel.
Size : 951M Storage fixed : NO
Total Pages : 122K
Free Pages : 98245
Segments : 4694
--------- Read --------- --------- Write --------
Rate Hit% Dly% Async Rate Hit% Dly% Sched Read% Dly%
Rate Rate
261.3 96.4 0.2 97.44 433.6 100 0.0 7.010 37.6 0.0
---------- Misc -----------
Page Reclaim Writes : 0
Fsyncs : 7
Selection 14 ZFSFS zFS File System (or zff)
RMF V2R4 zFS File System - ADCDPL Line 55 of 80
Command ===> Scroll ===> CSR
Samples: 100 Systems: 1 Date: 06/01/21 Time: 08.51.40 Range: 100 Sec
------ File System Name -------------------- I/O Resp Read XCF
System Owner Mode Size Usg% Rate Time % Rate
ZFS.S0W1.USR.MAIL
*ALL S0W1 RW 3600K 4.9 0.000 0.000 0.0 0.000
ZFS.S0W1.VAR
*ALL S0W1 RW 37M 63.2 265.1 0.033 100 0.000
ZFS.S0W1.VARWBEM
*ALL S0W1 RO 105M 33.8 0.000 0.000 0.0 0.000
If you put the cursor on any value ( except file name) you get more information.
I cound not find how to sort the data.
zFS File System Details
File System Name : JVB800.ZFS
Mount
Point :
System : *ALL Owner : S0W1 Mode : RO
-------------- Read ------------- ------------- Write -------------
--- Appl --- ---- XCF ---- Aggr --- Appl --- ---- XCF ---- Aggr
Rate Resp Rate Resp Rate Rate Resp Rate Resp Rate
Time Time Time Time
112.8 0.191 0.000 0.000 36618 0.000 0.000 0.000 0.000 0.000
Vnodes : 111 USS held vnodes : 68
Open objects : 47 Tokens : 0
User cache 4k pages : 9549 Metadata cache 8k pages : 106
ENOSPC errors : 0 Disk I/O error : 0
XCF comm. failures : 0 Cancelled operations : 0
Selection 15 ZFSKN zFS Kernel (zfk)
This gave me
RMF V2R4 zFS Kernel - ADCDPL Line 1 of 1
Command ===> Scroll ===> CSR
Samples: 100 Systems: 1 Date: 06/01/21 Time: 08.51.40 Range: 100 Sec
System - Request Rate - --- XCF Rate --- - Response Time -
Name Local Remote Local Remote Local Remote
S0W1 8599 0.000 0.000 0.000 0.054 0.000
In all these reports you can use PF10 and PF11 to scroll through time.
Annoyances
With all the output you do not get the duration of the statistics, so you are not able to display rates, for example MB/Second to a file system.
If you enable SMF, then the first record contains the accumulated data since ZFS was started, or SMF was disabled. If you try plotting the values against time – you will get a strange graph.
There is no SMF formatter provided so I’ve written my own.
You cannot pass all of the parameters to IOEZADM as the parameter field is too long, so you have to use PARMSDD=
//AGGRINFO EXEC PGM=IOEZADM,REGION=0M,
// PARMDD=PARMS
//PARMS DD *
query -reset -iobyaggregate -iobydasd -knpfs -ctkc
-usercache -iocounts -metacache -dircache -logcache
/*
ColinPaice 
3 thoughts on “How to collect zFS statistics”