It’s easy – use the chmount command.
chmount -w /usr
or
chmount -r /usr
Category: Z/OS
How much free space do my ZFS have?
I was running into a problem where my file systems were running out of space. I knew I had some spare space – but where was it?
The command
produced output like
COLIN.ZFS.ZOWE.CONFIG (R/W COMP): 1943183 K free out of total 2975040
COLIN.ZFS.ZOWE33.CONFIG (R/W COMP): 7327 K free out of total 421920
COLIN.OPENSSL.ZFS2 (R/W COMP): 2407159 K free out of total 7200000
ZFS.S0W1.SYSTEM (R/W COMP): 1263 K free out of total 1440
ZFS.Z31B.CNJ (R/O COMP): 80605 K free out of total 648000
ZFS.S0W1.WEB (R/W COMP): 9839 K free out of total 10080
You can sort it to give the largest space at the bottom.
- -n means treat the value as a number
- -k 4,4 means sort by the 4th field.
zfsadm aggrinfo |sort -n -k4,4
Gave
ZFS.Z31B.VERSION (R/W COMP): 33 K free out of total 3823200
ZFS.Z31B.ZEDC (R/O COMP): 367 K free out of total 1440
...
COLIN.OPENSSL.ZFS2 (R/W COMP): 2407159 K free out of total 7200000
COLIN.ZOPEN.ZFS (R/W COMP): 2446835 K free out of total 6144480
Using an external function
My “simple” problem was to have one application, using functions from another application, and allow me to upgrade both, and continue working. I want to write some code for openssl to call my program. I’m changing my code frequently, and I do not want to have to recompile and rebind openssl every time I change my program. (The recompile takes about 3 hours on my baby machine).
It took me a couple of days to get working; I learned a lot.
The simple solution which does not really work
I have a suite of programs called mycode, and I want to use stuff from another package which I’ll call otherstuff.
I can simply bind mycode, and otherstuff together to resolve all of the functions entry points. It works, but if I upgrade mycode, or otherstuff, I will not pick up the latest versions, and the code which is in the final load module may be incompatible with the newer versions.
Use fetch!
In mycode I can use the C function fetch, to load a load module, and give me the entry point. I then call the functions using the entry point.
This works – after a fashion. The key word is load module. A load module exists in a PDS or a PDSE. A Program Object can be in Unix or in a PDS, so the fetch solution does not work with file files in Unix.
The code worked if the module was in a load module – which I didn’t want.
I could not use fetch to load a program object from Unix.
Putting the code in a load module
I used the binder to create a load module.
xlc dummy.o-o "//'COLIN.LOAD(dummy)'" ...
To get my program to run I had to use
EXPORT STEPLIB=COLIN.LOAD
./myprog
The fetch worked, and my program was called.
Use Dynamic Link Library
The Unix eqivilent to a load module is the DLL.
When you create a DLL, two parts are created.
- The .x file which is used for C to find how to call a function (stub code)
- The .so file which contains the executable code.
The .x file
The .x file contains data like
IMPORT DATA,'cpfopen','ascii_tab'
IMPORT CODE,'cpfopen','cpfopen1'
IMPORT CODE,'cpfopen','printHex'
Which says
- there is an entry point cpfopen1 in program object cpfopen
- there is an external data object called ascii_tab in program object cpfopen
- there is an entry point printHex in program object cpfopen
This .x file contains information for C to create stub code to load the actual code.
My calling program has
#pragma linkage(cpfopen1 ,OS)
...
int myopen ;
myopen = cpfopen1("ABC","DEF");
...
This needs to be complied with option
-Wc,DLL
If you do not use the DLL option it ignores the IMPORT statement, and reports cpfopen1 and printHex are not found.
I bound it using
xlc -o fopen myfile.o cpfopen1.x V
The executed code.
You need to compile your code and specify which function names you want to make visible to other programs(export).
You can use
- the compiler option EXPORTALL or
- #pragma export(function1) #pragma export(function2) within your code.
At bind time
l1="-Wl,DLL "
xlc cpfopen1.o -o cpfopen1 -V $l1 1>a
This creates
- the executable with the name in the -o parameter (cpfopen1)
- and cpfopen1.x using the name of the object in the -o parameter
With
l1="-Wl,DLL "
xlc cpfopen1.o -o cpfopen1.so -V $l1 1>a
the file cpfopen1.x has
IMPORT CODE,'cpfopen1.so','cpfopen1'
Doing it this way, and letting the C and the binder resolve the entry point is the easy way of doing it.
Advanced DLL
You can load a DLL yourself
void * hDLL;
int (*fptr)(const char * filename, const char *mode);
hDLL = dlopen("cpfopen1", RTLD_LOCAL | RTLD_LAZY );
if (!hDLL) {
fprintf(stderr, "%s\n", dlerror());
exit(99);
}
dlerror(); /* Clear any existing error */
The find the entry point using dlsym. This uses the handle of the DLL object loaded above, and you pass the entry point name. (It might not exist).
Define a pointer to the function
int (*fptr)(const char * filename, const char *mode);
This defines a pointer to a function fptr, which returns an int, and has two char * parameters.
fptr = (int (*)(const char *, const char *)) dlsym(hDLL, "cpfopen1");
if ( fptr != 0)
{
myopen = (*fptr)("ABC","DEF");
printf("result %i\n",myopen);
}
It took me some time to get the definitions of the function parameters correct.
The called function is
#pragma linkage(cpfopen1, fetchable)
int cpfopen1(char * filename, char *mode)
{
printf(">>>>>>>>>>>>>>>In cpfopen1 %s %s\n",filename,mode);
return 7 ;
}
It was much easier to include the .x file at bind time, and let C run time sort it out.
Clean up
You should use dlclose(handle) to close the object and free its resources
Sharing status information between functions.
My high level problem is that I want to extend a very large package to be able to have my own code process some file requests, such as fopen(), fread() and fclose(). Then I wanted to share information such file name in the fclose routine.
I decided to map fopen() to cpfopen() and process the request through my own code.
The first step was easy; I changed fopen to invoke my program.
#define fopen(a,b) cpfopen(a,b)
FILE * cpfopen(const char *__restrict__filename, const char *__restrict__mode);
I include that code in the package routines, and recompile them all.
The problem is that I want information used in the fopen function, to be available in the fclose function. I could change these routines not to return a FILE * handle, but return a COLINFILE , but this makes it very messy.
At first glance the use of C global storage looks as if it may work. However because I can open two files, opening a second file will overwrite information from the first file, I need to be able to keep status on each file.
I found a neat, self contained solution – the use of name tokens.
You can define a name token pair to z/OS, and you can retrieve the token by giving the name.
How does it work?
Define the token
I did this in my cpfopen routine using IEANTCR.
- The “name” field is 16 bytes long. You can specify any value for the identifying label. In the example below I used the value COLIN.
- The “token” field is 16 bytes long. You can specify any value to be stored under the label. In the example below I used the value MYVALUE.
#pragma linkage(IEANTCR ,OS)
...
char name[16];
char token[16];
memset(&name,0,sizeof(name));
memcpy(&name[0],"COLIN",5);
memset(&token[0],0,sizeof(token));
memcpy(&token[0],"MYVALUE",7);
int task = 1; // for TCB level
int persist = 0; // Not relevant when TCB or address space
int rc; // return code passed back
IEANTCR(&task,&name,&token,&persist,&rc);
printf("Return code from ieantcr %i\n",rc);
Retrieve the token
I did this in my cpfread routine using IEANTRT.
#pragma linkage(IEANTRT ,OS)
...
char name[16];
char token[16];
memset(&name,0,sizeof(name));
memcpy(&name[0],"COLIN",5);// this label
int task = 1;
int persist = 0;
int rc;
IEANTRT(&task,name,token,&rc);
printf("IEANTRT %i\n",rc);
if ( rc == 0)
{
printf("read token %7.7s\n",token);
}
When this code ran it printed out MYVALUE, as expected.
You can have name token pairs on each TCB, or name token pairs available for any TCB within the address space, (and authorised users can make name token pairs available across address spaces).
Each name token pair is chained off a TCB/Address space/system control block. If you have many name token pairs, the CPU needed to locate an entry will increase as the number of entries increases.
In this case you have to do more work. Allocate a hash table of input value -> value, and put the address of the hash table as the value of the name token. You then use a unique token name such as “COLINSNT” to locate it the hash table.
How to bind
The SYSLIB of the bind step needs SYS1.CSSLIB to find the stub for the name token code to work.
My actual code for the fopen etc
I wanted to pass the file name used in fopen, to the fread and fclose routines. I used the FILE * handle as the identifying label.
I wanted to pass the variable length file name, and not just a constant, so I needed to make a copy of the file name, and pass the address of the copy of the name in the token.
cpfopen
#pragma linkage(IEANTCR ,OS)
...
FILE * cpfopen(const char * filename, const char *mode)
{
printf("In cpfopen %s,%s\n",filename, mode);
FILE * handle;
handle = fopen(filename,mode);
if (handle > 0)
{
printf("handle %8.8x\n",handle);
char name[16];
char token[16];
memset(&name,0,sizeof(name));
memcpy(&name[0],handle,4);
char * dupname ;
dupname = strdup(filename); // save the file name
memset(&token[0],0,sizeof(token));
memcpy(&token[0],&dupname,4); // 31 bit
int task = 1;
int persist = 0;
int rc;
IEANTCR(&task,&name,&token,&persist,&rc);
printf("Return code from ieantcr %i\n",rc);
}
return handle;
}
cpfread
#pragma linkage(IEANTRT ,OS)
...
size_t cpfread(void * buffer, size_t size, size_t count, FILE * stream)
{
char name[16];
char token[16];
memset(&name,0,sizeof(name));
memcpy(&name[0],stream,4); // use the passed stream value( handle)
char * dupname;
int task = 1;
int persist = 0;
int rc;
IEANTRT(&task,name,token,&rc);
printf("IEANTRT %i\n",rc);
if ( rc == 0) // it was found
{
// copy from the token to my variable
memcpy(&dupname,&token[0],4);
printf("read file name %s\n",dupname);
if (dupname != 0)
printf("In Read: Value from NT is %s\n",dupname);
else printf("dupname is 0\n");
}
// Do the read file read
size_t n = fread(buffer, size, count, stream);
printf("Bytes read %i\n",n);
return n;
}
cpfclose
You need to free any storage you allocated, and delete the name token pair using IEANTDL.
#pragma linkage(IEANTRT ,OS)
...
size_t cpfclose(FILE * handle)
{
char name[16];
char token[16];
memset(&name,0,sizeof(name));
memcpy(&name[0],handle,4); // use the passed stream value( handle)
char * dupname;
int task = 1;
int persist = 0;
int rc;
IEANTRT(&task,name,token,&rc);
printf("IEANTRT %i\n",rc);
if ( rc == 0) // it was found
{
// copy from the token to my variable
memcpy(&dupname,&token[0],4);
printf("close file name %s\n",dupname);
gg
}
// now the close
rc = fclose(handle);
return rc;
}
Binding in Unix on z/OS. A whole new world
I had been happy compiling and binding small programs in Unix on z/OS. It got a bit more complex as my programs got bigger and needed more files at bind time.
I stumbled into an area which I knew nothing about, (the pirate world of ar!) and which may help with doing binds.
- My simple compile and bind script
- My messy compile and bind script doing compiles, assembles and bind
- Linkedit control statements
- Using an archive file (the ar command)
- What is in the archive?
My simple compile and bind script
name=wr
rm ${name}.o
export _C89_CCMODE=1
p1="-Wc,arch(8),target(zOSV2R4),list,source,ilp32,gonum,asm,float(ieee) "
p8="-Wc,LIST(c.lst),SOURCE,NOWARN64,XREF,SHOWINC"
xlc $p1 $p8 -c $name.c
xlc $name.o -o wr -V 1>a
This script
- compiles a program (wr.c) putting the compile listing into c.lst
- binds the .o file and outputs (-o) into a file wr, and puts the output listing into a
You can use the ld command to invoke the binder, or use the xlc command. The xlc command looks at the types of the files to decide the action. For example *.c means compile *.o means bind.
My messy compile and bind script
name=colinm
rm ${name}.o
export _C89_CCMODE=1
p1="-Wc,arch(8),target(zOSV2R4),list,source,ilp32,gonum,asm,float(ieee) "
p7="-Wc,ASM,ASMLIB(//'SYS1.MACLIB') "
p8="-Wc,LIST(c.lst),SOURCE -Wa,LIST(133),RENT"
# compile it
xlc $p1 $p7 $p8 $p2 -c $name.c -o $name.o
as -mgoff -aegmrsx=a.lst BPXTPOST.s
as -mgoff -aegmrsx=a.lst BPXTWAIT.s
as -mgoff -aegmrsx=a.lst DUMMY.s
as -mgoff -aegmrsx=a.lst br14.s
as -mgoff -aegmrsx=a.lst cpfetch.s
# create a load module in a PDS
include=" BPXTPOST.o BPXTWAIT.o pcexit.o br14.o DUMMY.o DUMMY2.o mywto.o cpfetch.o "
l1="-Wl,LIST,MAP,XREF,AC=1 "
/bin/xlc $name.o $include -o "//'COLIN.LOAD(colinm)'" -V $l1 1>a
This script
- compiles colinm.c into colinm.o, puts the listing into c.lst
- assembles the files BPXTPOST… putting the listing into a.lst
- binds the list of “text files” into a load module COLIN.LOAD(COLINM), setting AC=1, and producing a load map.
Linkedit control statements
name=pcexit
export _C89_CCMODE=1
l2="-l //'CEE.SCEESPC' "
l3="-O EDCXSTRT"
# create a load module in a PDS
l1="-b LIST,MAP,XREF,AC=1 "
ss="//'CEE.SCEESPC.OBJ(EDCXSTRT)'"
ld $ss $name.o -v -o "//'COLIN.LOAD(PCEXIT)'" -V $l1 $l2 $l3 1>ab
This script generates binder like JCL statements. (ld does not use them – it is a JCL equivilent of what is used)
//* ld ---------------------------------------------------------------
//LINKEDIT EXEC PGM=IEWBLINK,
// PARM='TERM=YES,PRINT=NO,MSGLEVEL=4,STORENX=NEVER,LIST=NOIMP,XREF=YESc
// ,MAP=YES,PRINT=YES,MSGLEVEL=0,LIST,MAP,XREF,AC=1'
//LDIN0001 DD DSN=CEE.SCEESPC.OBJ(EDCXSTRT),DISP=SHR
//LDAT0001 DD DSN=CEE.SCEESPC,DISP=SHR,DCB=DSORG=DIR
//SYSLMOD DD DSN=COLIN.LOAD(PCEXIT),DISP=REP,DATACLAS=,DSNTYPE=LIBRARc
// Y,MGMTCLAS=,SPACE=,STORCLAS=,UNIT=,DCB=(RECFM=U,LRECL=0,c
// BLKSIZE=6144,DSORG=PO)
//SYSPRINT DD PATH='/dev/fd1',PATHOPTS=(ORDWR,OCREAT,OAPPEND),FILEDATAc
// =TEXT,PATHMODE=(SIROTH,SIRGRP,SIRUSR,SIWOTH,SIWGRP,SIWUSc
// R)
//SYSTERM DD PATH='/dev/fd2',PATHOPTS=(ORDWR,OCREAT,OAPPEND),FILEDATAc
// =TEXT,PATHMODE=(SIROTH,SIRGRP,SIRUSR,SIWOTH,SIWGRP,SIWUSc
// R)
//SYSLIN DD *
INCLUDE LDIN0001
INCLUDE './pcexit.o'
LIBRARY LDAT0001
ORDER EDCXSTRT
/*
Where
- -l //’CEE.SCEESPC’ becomes a binder LIBRARY statement which points to a PDS, which gets searched before SYSLIB
- -O EDXCSTRT becomes the binder ORDER statement. For some programs, some CSECTS have to be in the right order. You can specify -O name1,name2.
- -b…. pass these parameters to the binder
- //’CEE.SCEESPC.OBJ(EDCXSTRT)’ contains some precompiled code. It gets mapped to INCLUDE … from a JCL dataset
- $name.o is the precompiled object input to the binder.
Using an archive file (the ar command)
An archive file is something between a tar file, and a PDS with the output of compiles. (I have a dataset COLIN.OBJLIB in ISPF).
* compile an assembler file
as -mgoff -aegmrsx=a.lst BPXTPOST.s
ar -rc arch.a BPXTPOST.o
This
- assembles the program BPXTPOST.s, creating BPXTPOST.o
- stores it in the archive arch.a
I can then bind with it
/bin/xlc colinm.o arch.a -o out.load -V $l1 1>a
This takes the colinm.o and any .o files it needs from the arch.a archive.
What is in the archive?
The ar command
ar -tv arch.a
gives
rw-r--r-- 990021/990000 220 May 14 09:36 2026 __.SYMDEF
rw-r--r-- 990021/0 1200 Apr 27 10:29 2026 BPXTPOST.o
rw-r--r-- 990021/0 1200 Apr 28 19:18 2026 BPXTWAIT.o
rw-r--r-- 990021/0 720 May 11 15:11 2026 br14.o
The command nm (Display symbol table of object, library, or executable files)
nm -M arch.a
gives more information
BPXTPOST.o:
0 d --- --- - BPXTPOST
0 T ANY ANY - BPXTPOST
0 d --- --- - B_IDRL
0 d --- --- - B_PRV
0 d ANY ANY - B_TEXT
0 U --- --- - CEESTART
Where
- 0 is offset from the start
- d is Initialized data (bbs), local
- T is Text symbol, global
- ANY ANY – is RMODE(ANY) AMODE(ANY) Compiler_options(-)
- U is undefined
Using the nm command on an executable gave
...
128 U --- --- - B_TEXT
1312 T MIN MIN - CEEARLU
1312 U --- --- - CEEARLU
728 U --- --- - CEEBETBL
728 T MIN MIN - CEEBETBL
...
104 D --- --- - locs
152 T ANY ANY - main
152 U --- --- - main
40 U --- --- - optarg
40 D --- --- - optarg
...
we can see the offset of the main entry point is at offset 152 in the module.
Assembler programming without using using – it is all relative
Before Jump
For many years a program on z architecture looked like
MYPROG CSECT
LR 12,15
USING MYPROG,12
B OVER
DC C'*MYPROG *'
OVER DS 0H
...
LA 3,MYAUTOSTORAGE
USING MYDATA,3
L 4,FIELD2
MYDATA DSECT
FIELD1 DS F
FIELD2 DS F
To reference a field you have to do it off a register. For example
- B OVER is converted to BR 0x10 off register 12
- L 4,FIELD2 is converted to Load offset 4 from register 3
Each base register could cover 4KB, so for large programs, or large data you needed multiple registers.
Post Jump
A significant leap in the instructions was the introduction of the “… relative” instruction. For example the Jump (or as it is properly know the BRANCH RELATIVE ON CONDITION) is jump the number of half words from the start of the instruction
OFFSET
0000 MYPROG CSECT
0000 LR 12,15
0000 USING MYPROG,12
0002 J OVER (A7F4 0007)
0006 DC C'*MYPROG *'
0010 OVER DS 0H F
The Jump instruction A7M4 xxxx) has condition M (F is always branch) and xxxx in this case if 0007. The address of the instruction is 0002, so jump to h2 + 2 * 7 = 0010 — which is the label for OVER.
This is a faster instruction than the B OVER, because it has to do less work. It can jump to +- 65534 bytes, and does not use any base registers. Not using a base register means that more registers can be used to access data.
But ..
I was writing some assembler code within a C program, and hit a few snags. The C code does not make a base register available to access fields.
With code like
MYLABEL DC C'HELLO'
LA R2,MYLABEL
With base registers, this code will work and create the LA instruction with the offset of MYLABEL from the base register.
However, because there is no base register and USING, the LA instruction fails.
I found two ways of doing it when using relative instructions
Branch and save
BRAS 2,OVERWTOL
SWTO WTO TEXT=label,MF=L
OVERWTOL DS 0H
This jumps to the label OVERWTOL, and saves the “return address” in register 2. Register 2 points to label SWTO.
Load address relative
OFFSET
0004 SWTO WTO TEXT=label,MF=L
0074 LARL 2,SWTO
Which becomes load address with value -38 half words from the start of the instruction, so “offset” FFFFFFC8
This is half the solution
I was using the WTO macro to write message on the console.
WTO text=(Mydata),MF=(E,(4))
...
MYDATA DC H'5',CL8'HELLO'
This failed because there was no base register.
I had to use
LARL 2,MYDATA
WTO text=((2),MF=(E,(4))
...
MYDATA DC H'5',CL8'HELLO'
and this worked successfully.
Using IPCS to look at z/OS dumps
I’ve been trying to find a big deep problem in my on z/OS, and I’ve been getting plenty of dumps.
I thought I’d give a short list of command for people new to IPCS. It is not detailed – but at least should give you sign posts as to what you can do.
First get your dump.
On my system dumps are in data sets like SYS1.S0W1.Z31B.DMP00001.
You can issue the operator command D D,E to give information about the dumps on your system. It provides information on where the problem occurred, and registers etc. This may be enough to help you solve your problem.
You can print the EREP report, again this gives a summary of the problem, and may be enough to solve your problem. See Logrec and example output.
Using ISPF editor on the output
On many of the displays you can use the command
REPORT VIEW
to capture what is currently displayed, and display it in an ISPF view session, so you can use ISPF edit commands on it.
If you display the first 24 lines of output, and use REPORT VIEW, you will get ISPF edit with the 24 lines of output. I tend to display the report, scroll down to the bottom of the report (so it is all displayed), and then use REPORT VIEW on the complete report.
Into IPCS
From the main ISPF IPCS panel, select option 0 defaults.
Tab down to
Source ==> DSNAME('SYS1.S0W1.Z31B.DMP00001')
If you want IPCS to remember this next time you come in, change
Scope ==> LOCAL (LOCAL, GLOBAL, or BOTH)
Change local to both, and press enter.
Type =6
to issue an IPCS command.
If this is the first time you have used this dump, type
DROPD
This says drop all information you know about the current dump. This is because last week you may have had a different dump called SYS1.S0W1.Z31B.DMP00001. Any information stored about that dump does not apply to the current dump.
PF3 back to the command window.
status
This gives you a summary of the contents of the dump.
Dump Title: COMPON=BPX,COMPID=SCPX1,ISSUER=BPXMIPCE,MODULE=BPXVOTHD+00000244, ABEND=S00C4,REASON=00000004
Use
find 'DIAGNOSTIC DATA REPORT'
This gives you information on
- The symptom string. If you search on the internet with this string you may find other people have hit the same problem
- Time of Error Information
- The PSW pointing to (past) the failing instruction
- Failing instruction text: 12 bytes of data around the failure
- Translation exception address: If there was a problem with accessing the data
- The registers and access registers
- The Home, Primary and Secondary address spaces ids. (For cases where your program jumps to a different address space)
Commands
Display storage in the current ASID: IP WHERE 20E1F344
For the given address IPCS will information about which module it is in.
ASID(X'0010') 20E1F344. BPXINPVT+618344 IN EXTENDED PRIVATE
Display storage in a different ASID: ip where 20E1F344 asid(x’59’)
If your program is using multiple address spaces, you can specify the address space id.
Display 64 bit data: ip l 050_814F7F30
This displays the storage at the address in standard dump format
Display storage key of the page: ip l 050_814F7F30 display(machine)
Also displays
ASID(X'0010') ADDRESS(50_814F7F30.) KEY(00) ABSOLUTE(02_03C9AF30.)
so you can see the key of the page.
Display the offset of the storage within the display: ip l 050_814F7F30 str
The str says display it as a structure – it gives the hex offsets of each line at the start of the line
+002B0 _14F81E0. 00000000 00000000 ...
+002D0 _14F8200. 00000000 00000000 ...
Display the instructions from the page: ip l 20E1F344 I
LIST 20E1F344. ASID(X'0010') LENGTH(X'1000') INSTRUCTION
ASID(X'0010') ADDRESS(20E1F344.) KEY(00) ABSOLUTE(F9ECA344.)
20E1F354 | D28F 700C 2008 | MVC X'C'(X'90',R7),X'8'(R2)
20E1F35A | 9140 208C | TM X'8C'(R2),X'40'
20E1F35E | A774 0065 | BRC X'7',*+X'CA'
20E1F362 | 9180 208C | TM X'8C'(R2),X'80'
20E1F366 | A7E4 0008 | BRC X'E',*+X'10'
20E1F36A | A798 0004 | LHI R9,X'4'
20E1F36E | 5090 D2D0 | ST R9,X'2D0'(,R13)
20E1F372 | A7F4 0006 | BRC X'F',*+X'C'
20E1F376 | A728 0003 | LHI R2,X'3'
20E1F37A | 5020 D2D0 | ST R2,X'2D0'(,R13)
20E1F37E | 9602 D2D4 | OI X'2D4'(R13),X'02'
20E1F382 | 5898 0000 | L R9,X'0'(R8)
20E1F386 | 41E0 D2D0 | LA R14,X'2D0'(,R13)
Display storage for a given length: ip l 20E1F344 length(20)
I use
IP SETDEF LENGTH(4096)
to display 4KB of data each time.
Set defaults: IP SETDEF
The IP SETDEF command (or ISPF option 0) gives
....
/*---------------- Local Default Values for IPCS Subcommands ---------------*/
SETDEF LOCAL NOPRINT TERMINAL NOPDS /* Routing of displays */
SETDEF LOCAL FLAG(WARNING) /* Optional diagnostic messages */
SETDEF LOCAL NOCONFIRM /* Double-checking major acts */
SETDEF LOCAL NOTEST /* IPCS application testing */
SETDEF LOCAL DSNAME('SYS1.S0W1.Z31B.DMP00001')
SETDEF LOCAL LENGTH(4096) /* Default data length */
SETDEF LOCAL VERIFY /* Optional dumping of data */
SETDEF LOCAL DISPLAY( MACHINE) /* Include storage keys, .... */
SETDEF LOCAL DISPLAY( REMARK) /* Include remark text */
SETDEF LOCAL DISPLAY( REQUEST) /* Include model LIST subcommand */
SETDEF LOCAL DISPLAY(NOSTORAGE) /* Include contents of storage */
SETDEF LOCAL DISPLAY( SYMBOL) /* Include associated symbol */
SETDEF LOCAL DISPLAY( ALIGN) /* Align output to byte */
SETDEF LOCAL ASID(X'0010') /* Default address space */
If you are using multiple address spaces, you can set the default address space so you do not have to specify ASID(x’..’) every time.
Browsing interesting storage
You can use option 1 from the main IPCS panel, to browse data, and keep track of your interesting data.
You enter the address, (and address space if required) and can browse the data.
This pointer data is stored in IPCS. If you give it a remark, it will persist across sessions, and so help you later.
Once you have defined at least one pointer, you can get a list of all of the pointers you have defined. Use the S prefix command to select it.
Finding data in the dump
You can search for data in the dump. You can specify a range to be searched. Using the variable X can be useful for a repeat search
find x'019EDC90' range(0:7fffffff)
ip l x str position(-512) length(4096)
find x'019EDC90' range(x+1:7fffffff)
ip l x str position(-512) length(4096)
- The first find defines the range.
- ip l x str position(-512) length(4096) says display 512 bytes before the found location for 4096 bytes… and the half page on each side of the data
- The second find – says search from just past the previous location
- ip l x str position(-512) length(4096) as above
- I use PF12 as recall, so it is easy to display the storage, pf3, enter, pf12 etc
The output is like
LIST 8FF91C. ASID(X'0010') POSITION(X'-0200') LENGTH(X'1000') STRUCTURE
ASID(X'0010') ADDRESS(8FF71C.) KEY(00) ABSOLUTE(FA37771C.)
-001FC 008FF720. 00000000 018E0AD0 .... |.... ...
...
-0001C 008FF900. 001ED022 00000000 ... |..}.. ...
+00004 008FF920. 019EE13C 02F72A00 ... |........
+00024 008FF940. 00000000 00000001 ... |..... ...
System trace
This is where it gets harder
The command
SYSTRACE
displays information from the system trace. Some of the key fields are
0002-0010 008DC518 SVC 2 00000000_20BC9420
07041001 80000000
0002-0010 008DC518 SVCR 2 00000000_20BC9420
07041001 80000000
- 0002 is the CPU number (which I very rarely use)
- 0010 is the asid
- 008DC518 is the TCB address
- SVC/SVCR 2 is the supervisor call/supervisor call return
- 00000000_20BC9420 is the PSW where the SVC occured.
On the right of the display is a string like
E29F45B336755380
This is a Store clock value. If you use the option SYSTRACE TIME(GMT) it is displayed as
06:39:12.803669218
What is interesting?
I use REPORT VIEW get an ISPF edit session of the trace
- X ALL
- F ‘*’ 20 20 all
This gives information such as I/Os completing, but also
- *RCVY recovery code was entered
- *SVC D a dump request was made
- *PGM… an exception occurred – such as invalid storage access.
I then label each line of interest, such as putting .aa in the field on the left hand side.
- reset
- loc .aa
To locate the line of interest. Note, lines of interest will normally be at the bottom of the output – just before the dump is taken.
The system carries on processing while dump is being taken, so you will continue to get records in the system trace after the point of failure/request for a dump.
You can use standard ISPF edit commands, such as sort, exclude, delete all X, delete all NX; and use ISPF edit macros (written in Rexx) for special processing.
Note: You cannot use the IPCS command from ISPF edit sessions to display information from the dump. You have to come out of ISPF edit.
Note: Some PGM exceptions are valid – such as page not in storage – and z/OS will read it into memory. These do not have the *; they are just PGM, and are usually not interesting.
Program calls (PC) and SVCs
A program call has output like
PC ... 0 00_01A5184A 0030A LocAscb
PR ... 0 00_01A5184A 0142923C
- For every PC there is a matching return (PR) with the same address (00_01A5184A)
- The PC number is 0030a – and because this is a system value – it knows it is for the LOCASBC request.
Other subsystems, CICS, MQ, DB2 also use PC routines, but their PC number can be different between IPLs, and if you can have more than one CICS, MQ, DB2 subsystems on an LPAR.
Other requests in the SYSTRACE
- DSP this TCB was dispatched
- SSRV 11E This TCB was paused ( suspended because there was no work for it to do)
- EXT CLKC the system timer interrupted this thread
What is a PC doing?
In my trace I have
PC ... 0 00_20DDF46C 0C101
What is 0c101 doing?
Issue the commands
- SUMMARY FORMAT
- This formats most of the significant control blocks.
- Go to the bottom and use REPORT VIEW
- FIND ‘0c101’ 1 10
This gives output like
**PC INFORMATION**
AUTH
PC KEY EXEC ENTRY EXEC LATENT
NUMBER MASK ASID ADDRESS STATE PARMS
-------- ---- ---- ------- ----- ------------------ ...
0000C100 8000 0059 20811470 S 00000000 00000000 ...
0000C101 8000 0059 20811D68 S 00000000 00000000 ...
The PC number C101 goes to address space with ASID 59 – and address 20811D68
You can then use the commands
ip where 20811D68 asid(x'59')
ip l 20811D68 asid(x'59') i
to display which load module the code is in, and the instructions which will be executed.
Not for humans but for search engine – BPX
BPXF024I
You get this message if the syslogd program is not running.
BPXP015I HFS PROGRAM /usr/lpp/zosmf/lib/libIzuCommandJni.so IS NOT MARKED PROGRAM CONTROLLED. BPXP014I ENVIRONMENT MUST BE CONTROLLED FOR DAEMON (BPX.DAEMON) PROCESSING.
Use the command extattr /usr/lpp/zosmf/lib/libIzuCommandJni.so to check the Program Controlled attribute is set. Use the extattr +p…. to set it if required.
I had the wrong SAF_PREFIX(‘IZUDFLT‘) in USER.Z24A.PARMLIB(IZUPRMCP). IZUDFLT was correct.
I had other problems like invalid password when I logged onto the web browser.
Fix the problem and regenerate.
BPXO042I with D OMVS,PFS
I was expecing D OMVS,PFS or D OMVS,P to give me BPXO068I and a list of Physical File Systems.
it gives BPXO042I when the command failed.
This was due to having an HFS definition in my z/OS 3.1 system. HFS is not supported on 3.1 . I removed the definition and it worked.
BPX1SOC TTLS_INIT_CONNECTION rv -1 rc ECONNRESET(1121) rs 2007593789 (0x77a9733d) 77a9733d EDC8121I Connection reset
The bpxmtext 77a9733d gives
TCPIP
JrTtlsHandshakeFailed: AT-TLS was unable to successfully negotiate a secure
TCP connection with the remote end.
Action: Review message EZD1286I for more information about the error.
On syslog was
EZD1287I TTLS Error RC: 403 Initial Handshake
Where 403 is The required certificate was not received from the communication partner.
The Wireshark output had a Certificate flow from the client to the server. This had no certificate in it.
The reason for this was,
- the client had an RSA certificate
- the Signature Hash Algorithms sent from the server did not include RSA.
The client was thus unable to send a certificate matching the SHA.
If I specified RSA only signature pairs, I could only use an RSA certificate. An Elliptic Curve certificate (ECDSA) had the same message and error code.
BPX1BND rv -1 rc EADDRINUSE(1115) rs 1951167047 (0x744c7247) EDC8115I Address already in use.
Because a program may not know that the “FIN” (end of conversation) has got to the other end, a socket enters a TIMEWAIT state. The IBM documentation says
If the server cannot wait for one to four minutes, you can use the setsockopt() call in the server to specify SO_REUSEADDR before it issues the bind() call. In that case, the server will be able to bind its socket to the same port number it was using before, even if the TIMEWAIT period has not elapsed. However, the TCP protocol layer still prevents it from establishing a connection to the same partner socket address. As clients normally initiate connections and clients use ephemeral port numbers, the likelihood of this is low.
BPX1SND rv -1 rc EOPNOTSUPP(1112) rs 1977578120 (0x75df7288) EDC8112I Operation not supported on socket.
I got this trying to issue bpx1snd() when there was data in the receive buffer. I used bpx1rcv to read the data, and the problem went away.
I peeked at the data before getting it, so I knew the length of the data to get, and so avoided waiting for data.
char buf[4000];
int lbuff = sizeof(buf);
int alet = 0;
int flags = MSG_PEEK;
BPX1RCV( &sd, // socket desciptor
&lbuff,
&buf,
&alet,
&flags,
&rv, // -1 or number of bytes
&rc,
&rs);
printf("BPX1RCV Peek bytes %d data... n",rv );
lbuff = rv; // the number of bytes in the buffer
flags = 0 ;
BPX1RCV( &sd, // socket descriptor
&lbuff,
&buf,
&alet,
&flags,
&rv, // -1 or number of bytes
&rc,
&rs);
printf("BPX1RCV bytes %d data... n",rv );
BPXF135E RETURN CODE 00000079, REASON CODE 055B005C
I got this using the command
MOUNT FILESYSTEM(‘COLIN.ZFS2’) TYPE(ZFS) MOUNTPOINT(‘/u/ibmuser/temp’ )
code 79 is invalid. The 005b005c means already in use. Either
- COLIN.ZFS2 is already mounted
- there is something else mounted on /u/ibmuser/temp
You can use the D OMVS,F command to display the file system and where they are mounted.
BPXF135E RETURN CODE 00000081, REASON CODE 053B006C
May because the file system is mounted READ and it needs to be RDWR.
BPXMTEXT 053B006C -> JRFileNotThere: The requested file does not exist.
Problem 1
I had MOUNTPOINT(‘/u/ibmuser/test’ ) (which did not exit) not the correct MOUNTPOINT(‘/u/ibmuser/temp’ )
Problem 2
I was trying to mount it at /my. I had to go into Unix and issue mkdir /my only then could I mount the file system.
BPXF137E RETURN CODE 00000079, REASON CODE 0588002E.
THE UNMOUNT FAILED FOR FILE SYSTEM …
002E is JRFilesysNotThere. Check the file system is mounted
BPXF137E RETURN CODE 00000072, REASON CODE 058800AA
BPXF137E RETURN CODE 00000072 (the resource is busy) , REASON CODE 058800AA JRFsParentFs The file system has file systems mounted on it.
I was trying to unmount a ZFS file systemm, and got the above messages. It means you cannot unmount it, because you have other file systems attached to it. On the z/OS console it had
BPXF271I FILE SYSTEM ZFS.USERS
FAILED TO UNMOUNT BECAUSE IT CONTAINS MOUNTPOINT DIRECTORIES FOR
ONE OR MORE OTHER FILE SYSTEMS WHICH MUST BE UNMOUNTED FIRST,
INCLUDING FILE SYSTEM COLIN.ZFS2
I used
unmount filesystem('COLIN.ZFS2') Immediate
and got message on the console
IOEZ00048I Detaching aggregate COLIN.ZFS2
BPXF002I FILE SYSTEM … NOT MOUNTED. RETURN CODE = 0000009D, REASON CODE = 11B900B7
With a Physical File System (PFS) there was a bad parameter in the OSI_THREAD block. The value of threadParm.ot_modname ( program to execute) did not exist.
BPXF135E RETURN CODE 0000008B, REASON CODE 119B00B0.
Reason code 119B00B0 is JRUserNotPrivileged: The requester of the service is not privileged. Action: The service requested required a privileged user.
Run it from an authorised userid.
Where’s my dump?
I had been working on a bug and getting out a System Dump (SDUMP). Then, after a fix, I stopped getting dumps, and just got a message with a return code indicating an abend had occurred. So where was my dump?
z/OS has Dump Elimination and Analysis DAE. This keeps a note of the dumps being taken, and can be configured to say “If you get the same abend many times, don’t bother taking a dump”. This stops you getting many identical dumps, and filling your disk storage.
In IPCS option 3 UTILITY – Perform utility functions, 5 DAE – Process DAE data, you can display the contents of the DAE information. On my system the data set is SYS1.DAE.
Command ===> Scroll ===> PAGE
Enter an Action Code next to an entry.
Enter / next to an entry to choose from a list of Action Codes.
Dataset: 'SYS1.DAE'
Dumps since last DAE Display: 0 Total Dumps suppressed: 214
Events since last DAE Display: 0 Suppression rate: 81%
A Last Last Total Date of Symptom String information:
C Date System Events Dump Abend Reason Module CSECT
_ 05/01/26 S0W1 7 04/28/26 S0EC6 055B0718 BPXINPVT BPXFSMNT
_ 05/01/26 S0W1 66 04/28/26 S00C4 00000004 BPXINPVT BPXVOTHD
_ 05/01/26 S0W1 10 04/28/26 S00C4 00000004 BPXINPVT BPXVOTHD
s 05/01/26 S0W1 53 04/28/26 S00C4 00000004 BPXINPVT BPXVOTHD
Selecting a record with S gave
Date Time System Name
Last (most recent) Event: 05/01/26 19:02:16 S0W1
Dump Taken: 04/28/26 18:44:59 S0W1
Symptoms used for Dump Suppression:
MVS RETAIN
Key Key Symptom Data Explanation
MOD/ RIDS/ BPXINPVT LOAD MODULE NAME
CSECT/ RIDS/ BPXVOTHD ASSEMBLY MODULE CSECT NAME
PIDS/ PIDS/ 5752SCPX1 PRODUCT/COMPONENT IDENTIFIER
AB/S AB/S 00C4 ABEND CODE-SYSTEM
REXN/ RIDS/ BPXMIPCE RECOVERY ROUTINE CSECT NAME
FI/ VALU/H 7542A0704742A0708BB917 FAILING INSTRUCTION AREA
REGS/ REGS/ C07DC REG/PSW DIFFERENCE
HRC1/ PRCS/ 00000004 ABEND REASON CODE
SUB1/ VALU/C OPENMVS COMPONENT SUBFUNCTION
This information is called a Symptom String. It provides a very short summary of the problem. You can search the internet with this information, to see if the problem has been found before.
For me, this symptom string, there has been 66 occurrences of it in my system (see the first display).
DAE can be configured to say if you get the same symptom string, do not take a dump, as is is most probably the same problem. This means your disks are not fully of identical system dumps.
Similar dumps
In
Command ===> Scroll ===> PAGE
Enter an Action Code next to an entry.
Enter / next to an entry to choose from a list of Action Codes.
Dataset: 'SYS1.DAE'
Dumps since last DAE Display: 0 Total Dumps suppressed: 214
Events since last DAE Display: 0 Suppression rate: 81%
A Last Last Total Date of Symptom String information:
C Date System Events Dump Abend Reason Module CSECT
_ 05/01/26 S0W1 7 04/28/26 S0EC6 055B0718 BPXINPVT BPXFSMNT
_ 05/01/26 S0W1 66 04/28/26 S00C4 00000004 BPXINPVT BPXVOTHD
_ 05/01/26 S0W1 10 04/28/26 S00C4 00000004 BPXINPVT BPXVOTHD
_ 05/01/26 S0W1 53 04/28/26 S00C4 00000004 BPXINPVT BPXVOTHD
There is a record for S0EC6 055B0718, and several records for S00C4 00000004. The detailed symptom string for these S00C4 abends are similar but different. The abend occurred at a different place, so the symptom string is slightly different.
Configuring DAE
You configure DAE through parmlib members ADYSETXX
On my system member ADYSET01 stops DAE, so I get a dump for every abend.
In member ADYSET00 I have
SVCDUMP(MATCH,SUPPRESSALL,UPDATE,NOTIFY(3,30)),
Where
- MATCH: Specifies that DAE is to compare the symptoms from the current memory dump to those that have already been recorded in the DAE data set. (Coding MATCH does not indicate that DAE suppresses duplicate memory dumps or update the DAE data set.)
- SUPPRESSALL: Specifies that duplicate memory dumps are to be suppressed when all criteria for matching and suppressing memory dumps are met except for the VRADAE key.
- Update: Specifies that the DAE data set is to be updated with the results of matching.
- Notify: This sends an internal signal to system applications listening, that a dump has occurred. The default is three memory dumps in 30 minutes for a particular symptom string.
How am I meant to know what’s happened?
As well as DAE processing, a record is written to EREP/LOGREC. EREP is a repository for hardware errors, and software errors.
For example, formatting the dataset, gave me
TYPE: SOFTWARE RECORD REPORT: SOFTWARE EDIT REPORT DAY.YEAR
(PROGRAM INTERRUPT) REPORT DATE: 121.26
...
SEARCH ARGUMENT ABSTRACT
PIDS/5752SCPX1 RIDS/BPXINPVT#L RIDS/BPXVOTHD AB/S00C4 PRCS/00000004 REGS/C082C
RIDS/BPXMIPCE#R
SYMPTOM DESCRIPTION
------- -----------
PIDS/5752SCPX1 PROGRAM ID: 5752SCPX1
RIDS/BPXINPVT#L LOAD MODULE NAME: BPXINPVT
RIDS/BPXVOTHD CSECT NAME: BPXVOTHD
AB/S00C4 SYSTEM ABEND CODE: 00C4
PRCS/00000004 ABEND REASON CODE: 00000004
REGS/C082C REGISTER/PSW DIFFERENCE FOR R0C:-082C
RIDS/BPXMIPCE#R RECOVERY ROUTINE CSECT NAME: BPXMIPCE
...
TIME OF ERROR INFORMATION
PSW: 07047001 80000000 00000000 20E1F2F4
INSTRUCTION LENGTH: 04 INTERRUPT CODE: 0004
FAILING INSTRUCTION TEXT: 00175045 00005049 01B0E368
TRANSLATION EXCEPTION ADDRESS: 00000000_01B6A404
BREAKING EVENT ADDRESS: 00000000_20E1F280
AR/GR 0-1 FFF00001/00000000_00000097 00000000/00000051_01B6AC90
AR/GR 2-3 00000001/00000000_0239A000 00000000/00000051_02FCFA00
...
HOME ASID: 0010 PRIMARY ASID: 0010 SECONDARY ASID: 0056
PKM: 8040 AX: 0001 EAX: 0000
RECOVERY ROUTINE ACTION
THE RECOVERY ROUTINE RETRIED TO ADDRESS 20E1F7DA.
THE REQUESTED SVC DUMP WAS NOT TAKEN. THE DUMP WAS SUPPRESSED BY DAE.
NO LOCKS WERE REQUESTED TO BE FREED.
THE SDWA WAS REQUESTED TO BE FREED BEFORE RETRY.
The information is a superset of the information in DAE. You get the registers at the point of failure, and what recovery action was taken. (Programs can ignore some abends, or pass the decision to a program higher up the stack).
In this case I can see there was no dump taken because of DAE.
The information in the logrec record was enough for me to debug the program. I did not need an SDUMP.
Processing LOGREC
Logrec is a z/OS dataset which records information about events, such as hardware problems, and software abends. Information is written to the dataset even though the information may have been suppressed elsewhere.
JCL to print it
//IBMPEREP JOB (ACCT),'PRINT LOGREC',CLASS=A,MSGCLASS=H
//STEP EXEC PGM=IFCEREP1,PARM='CARD'
//SERLOG DD DISP=SHR,DSN=SYS1.S0W1.LOGREC
//DIRECTWK DD UNIT=SYSDA,SPACE=(CYL,10,,CONTIG)
//EREPPT DD SYSOUT=A,DCB=BLKSIZE=133
//TOURIST DD SYSOUT=A,DCB=BLKSIZE=133
//ZERLOG DD SYSOUT=A,DCB=BLKSIZE=133
//SYSIN DD *
PRINT=PS
ACC=N
ZERO=N
ENDPARM
//
To print and clear specify ZERO=Y.
The command syntax is given here.
Example output
IPL record
IPL RECORD EDIT AND PRINTING SECTION
DAY YEAR HH MM SS TH
DATE -122 26 TIME -06 05 01 52
MODEL - 1090 CPU SERIAL NO. - 011238
MVS/ESA V7 R3
IPL REASON CODE - DF DEFAULT -U-
SUBSYSTEM ID - 00 SUBSYSTEM NAME - NULL
HIGHEST STORAGE ADDRESS 7FFFFFFF
LAST ACTIVITY INFORMATION :
DAY YEAR HH MM SS TH
DATE -121 26 TIME -19 14 13 68
END OF IPL RECORD
3270 termination
DEVICE NUMBER: 000703 DAY YEAR JOB IDENTITY: VTAM
DEVICE TYPE: 3277
MODEL: 1090 HH MM SS.TH
ERROR PATH: EF-0703 CPU ID: 111238 TIME: 19 16 26.42
RECORD IS: TEMPORARY
MODE IS: 370XA
---UNIT STATUS---- SUB-CHANNEL STATUS
....
DEVICE DEPENDENT DATA
TYPE OF RECORD: CLOSEDOWN (X'20')
TERMINAL NAME: LCL703 SIO CNTR: 00000003 TEMP. ERRORS: 00
Software abend
See Where’s my dump?
How often should I clear it?
Periodically you should archive the data, so you can later do trend analysis, such as which disks are having more I/O problems than usual.
When logrec fills up, your automation can trigger a job to copy the logrec dataset, and clear it.
ColinPaice 