Help ! My ZFS has filled up

The file system I was using in Unix Services filled up – but it didn’t tell me,I just had a truncated file. I piped the output of a shell script to a file. As the file system filled up – it could not write the “file system full” message.

SInce I wrote this, Ive written What is using all the space on this file system? which discusses how to find what is using the space on a file system, rather than in a directory. These are not the same as a file system can be mounted over a sub-directory.

To solve this file system full problem, I had to explore other areas of z/OS which I was not so familiar with – ADRDSSU to move data sets, zfsadmin commands, and how to stop SMS from being too helpful.

Having made the ZFS.USERS larger, I then found IEC070I 104-204 is data set is > 4GB. So some of this blog post is wrong!

Later I found the ZFS.USERS data set had 123 extents – and could not be expanded.

The Unix Services command

df -P /u/pymqi

tells you the file system – and how full it is. This gave me

Filesystem 512-blocks   Used Available Capacity Mounted on
ZFS.USERS      204480 203954       526     100% /u

So we can see the data set is ZFS.USERS and it is 100% full.

What is using all of the space?

du -a ./ | sort -n -r | head -n 30

The command

zfsadm fsinfo ZFS.USERS

give more (too much) information, and

zfsadm aggrinfo ZFS.USERS

doesnt quite give enough info. df -P … is best

I used the command

zfsadm grow ZFS.USERS -size 144000

to make it bigger, but I got the Unix Services message

IOEZ00326E Error 133 extending ZFS.USERS

and on the system log

IOEZ00445E Error extending ZFS.USERS. 591
DFSMS return code = 104, PDF code = 204.
IOEZ00308E Aggregate ZFS.USERS failed dynamic grow, (by user COLIN).
IOEZ00323I Attempting to extend ZFS.USERS to 36000 4096 byte control intervals.
IEF196I IEC070I 104-204,OMVS,OMVS,SYS00022,0A9E,C4USS2,ZFS.USERS,
IEF196I IEC070I ZFS.USERS.DATA,CATALOG.Z24C.MASTER
IEC070I 104-204,OMVS,OMVS,SYS00022,0A9E,C4USS2,ZFS.USERS, 594
IEC070I ZFS.USERS.DATA,CATALOG.Z24C.MASTER
IOEZ00445E Error extending ZFS.USERS. DFSMS return code = 104, PDF code = 204.

Use the return codes from the IEC196I message. Search for IEC196I 104

DFSMS return code = 104, PDF code = 204 means no space on the volume.

I used ISPF 3;4 to display the volume the ZFS.USERS data set was on; C4USS2.

I used ISPF 3;4 to display what data sets were on the C4USS2 volume. If you use PF11 to can see the space allocated to each data set.

I could try moving this dataset to another volume, but that would mean unmounting it, moving it, remounting it. I thought it easier to move other dataset off the volume.

On the volume, I found a ZFS which I was not using and unmounted it

unmount filesystem(‘ZFS.Z24C.ZCX’) normal

and trying to move it, looked easy using DFDSS COPY DATASET .

//IBMUSER1 JOB 1,MSGCLASS=H
//STEP1 EXEC PGM=ADRDSSU,REGION=0M
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
COPY DATASET(INCLUDE(ZFS.Z24C.ZCX))-
ODY(C4USS1) DELETE CATALOG
/*

When I ran this job – it moved the dataset, but moved it to a USER00 volume – filling up most of the space on this volume. I had just moved the problem. SMS intercepted my request and “managed” the disk storage for me.

I added BYPASSACS and NULLSTORCLA

COPY DATASET(INCLUDE(ZFS.Z24C.ZCX))-
BYPASSACS(ZFS.Z24C.ZCX) –
NULLSTORCLAS –
ODY(C4USS2) DELETE CATALOG

and this worked.

  • BYPASSACS – do not use ACS routines to decide where to put the data set
  • NULLSTORCLAS ( or STORCLAS(xxxxx)) do not use a Storage class.
  • ODY OUTDYNAM specifies that the output DASD volume is to be dynamically allocated.

I then had enough space to be able to grow the zfs.

When I ran it on a different ZFS, I got a message

BPXF137E RETURN CODE 00000072, REASON CODE 058800AA

which means there is a file system mounted within it. I got on the console

IOEZ00048I Detaching aggregate COLIN.ZFS2

I unmounted this

unmount filesystem(‘COLIN.ZFS2’) Immediate

I could then unmount ZFS.USERS, and then move it.

Once I had moved it, I expanded it, and remounted the COLIN.ZFS2. (See the MOUNT command in parmlib for the ZFS’s

Messages when using python on z/OS

This post gives some of the error messages I received, and the actions I took to resolve the problems.

FSUM3008 Specify a file with the correct suffix (.c, .i, .s, .o, .x, .p, .I, or .a), or a corresponding data set name, instead of -o ….

I got this during a Python C extension build. You need

export _C89_CCMODE=1
export _C99_CCMODE=1
export _Ccc_CCMODE=1
export _CC_CCMODE=1
export _CXX_CCMODE=1
export _C89_CCMODE=1
export _CC_EXTRA_ARGS=1
export _CXX_EXTRA_ARGS=1
export _C89_EXTRA_ARGS=1

Before doing any builds.

Python builds

DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives.

Easy fix which no one tells you about (it took me 3 days to find this). Add

import setuptools

to the top of the file.

COLIN:/u/pymqi: >python3 -m build
No module named build.main; ‘build’ is a package and cannot be directly executed

You have a build directory in your project

https://pypi.org/search/?q=build

then install it

python3 setup.py bdist_wheel … error: invalid command ‘bdist_wheel’

I needed “import setuptools” at the top of the setup.py file. I also needed wheel to be installed.

CEE3501S The module libpython3.10.so was not found.

I was trying to do

import ctypes
from ctypes.util import find_library
testlib = ctypes.CDLL(“… “)

This file was in /u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/

You need

export LIBPATH=/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/:$LIBPATH

python3 ….

CEE3587S A call was made to a function in the AMODE 31 DLL //ADD2 from an AMODE 64 caller.

I was trying to call a C program from Python – but i was built with 31 bit mode – not 64 bit mode.

You need to compile it with LP64, and bind in 64 bit mode, and use //BIND.SYSLIB DD DISP=SHR,DSN=CEE.SCEEBND2

ImportError: CEE3512S An HFS load of module /u/tmp/py/mq.so failed. The system return code was 0000000130; the reason code was 0BDF0

I had a bind error. When I fixed it – it worked.

The IBM documentation says

A package shared library may get tagged incorrectly when using the xlc utility. Verify that the shared library is untagged by running the following line:

ls -alT

If the file is tagged, with the output being similar to the following line:

t ISO8859-1 T=on

you can remove the tag with the following command:

chtag -r <filename.so>

ERROR: Could not install packages due to an EnvironmentError: Erno 111 EDC5111I Permission denied.: ‘/u/.local’
Check the permissions.

I was trying to install a package using

python3 -m pip install /tmp/… .whl /tmp/… whl –no-cache-dir

It defaults to storing things in /u/.local. I needed

export PYTHONUSERBASE=.

Before running the command.

ImportError: CEE3512S An HFS load of module …. failed. The system return code was 0000000111; the reason code was EF076015 .

You need to use chmod +x …. to the module

SystemError: unmatched paren in format

I had a C program and was using

rv = Py_BuildValue("(blll",
...
);

but was missing a backet in "(blll)"

CEE3204S The system detected a protection exception (System Completion Code=0C4).
From compile unit TOROLABA:./Python/getargs.c at entry point vgetargskeywords at statement 1687

I had code

static char *kwlist[] = {“routcde”};
if (!PyArg_ParseTupleAndKeywords(args, keywds, “s#|i”, kwlist,

It needs to be static char *kwlist[] = {“routcde”,NULL};

From compile unit TOROLABA:./Python/getargs.c at entry point vgetargskeywords at statement … at compile unit offset ….

With code like

static char *kwlist[] = {“text”,”routecde”,NULL};
PyArg_ParseTupleAndKeywords(args, keywds, .., kwlist,…

IEW2606S 4B39 MODULE INCORPORATES VERSION 3 PROGRAM OBJECT FEATURES AND CANNOT BE SAVED IN LOAD MODULE FORMAT.

I was trying to save a DLL in a load library.

I had created the PDSE using

//PYTALL JOB 1,MSGCLASS=H
//S1 EXEC PGM=IEFBR14
//DD1 DD DISP=(MOD,DELETE),SPACE=(CYL,(1,10,10)),
// DSN=COLIN.PDSE2
//S1 EXEC PGM=IEFBR14
//DD2 DD DISP=(NEW,CATLG),SPACE=(CYL,(1,10,10)),
// DSNTYPE=(LIBRARY,1),
// DSN=COLIN.PDSE2,
// DCB=(RECFM=U,LRECL=0,BLKSIZE=6400)

I was building it in OMVS using

/bin/xlc $name.o -o //’COLIN.PDSE2($name)’

This tried to use data set COLIN.COLIN.PDSE which did not, exist, so it tried to create it, and created a PDS, not a PDSE.

The statement

/bin/xlc $name.o -o “//’COLIN.PDSE2($name)'”

With double quotes around the name worked.

Binder problems compiling a module

IEW2456E 9207 SYMBOL CEETHLOC UNRESOLVED. MEMBER COULD NOT BE INCLUDED FROM THE DESIGNATED CALL LIBRARY.
IEW2456E 9207 SYMBOL @@ROND UNRESOLVED. MEMBER COULD NOT BE INCLUDED FROM THE DESIGNATED CALL LIBRARY.
IEW2456E 9207 SYMBOL CEEROOTD UNRESOLVED. MEMBER COULD NOT BE INCLUDED FROM THE DESIGNATED CALL LIBRARY.

I was compiling a C program using XPLINK and got the above messages.
I used the following JCL

//COMPILE EXEC PROC=EDCXCB,
// LIBPRFX=&LIBPRFX,
// CPARM=’OPTFILE(DD:SYSOPTF),LSEARCH(/usr/include/)’,
// BPARM=’SIZE=(900K,124K),RENT,LIST,RMODE=ANY’
//* BPARM=’SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31,AC=1′
//COMPILE.SYSOPTF DD *

….

//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB.
//BIND.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEELKED

SCEELKED is for non XPLINK.

It needs to be

DSNAME=&LIBPRFX..SCEEBND2,DISP=SHR

EDC5061I An error occurred when attempting to define a file to the system. (errno2=0xC00B0403)

I got this trying to open a data set from from a Python program.

C00B0403: The filename argument passed to fopen() or freopen() specified dsname syntax. Allocation of a ddname for the dsname was attempted, but failed.

I used

printf(“AMRC\n”);
printHex(stdout,__amrc ,sizeof(__amrc_type));

The first word was 00000210.

Interpreting error reason codes from DYNALLOC gives

210: Meaning: Requested data set unavailable. The data set is allocated to another job and its usage attribute conflicts with this request. (dsname allocation)

I had the dataset open in a ISPF window.

EDC5129I No such file or directory. (errno2=0x05620062)

I was trying to use fopen(“DD:VB”…) where VB was not in the JCL.

When I specified a data set name “//’COLIN/VB'” it worked.

BPXM018I BPXBATCH FAILED BECAUSE SPAWN (BPX1SPN) OF /BIN/LOGIN FAILED WITH RETURN CODE 0000009D REASON CODE
0B1B0473

I got this running under PGM=BPXBATSL. When I changed it to PGM=BPXBATCH it worked.

BPXBATCH

BPXBATCH makes it easy for you to run shell scripts and executable files that reside in z/OS® UNIX files through the MVS™ job control language (JCL)…

In addition to using BPXBATCH, a user who wants to perform a local spawn without being concerned about environment setup (that is, without having to set specific environment variables, which could be overwritten if they are also set in the user’s profile) can use BPXBATSL. BPXBATSL provides users with an alternate entry point into BPXBATCH. It also forces a program to run by using a local spawn instead of fork/exec as BPXBATCH does. These actions allow the program to run faster.

can’t open file ‘//DD:STDIN’: [Errno 92] EDC5092I An I/O abend was trapped.

I was using AOPBATCH, and had PGM=AOPBATCH,PARM=’//usr/lpp/IBM/cyp/v3r8/pyz/bin/python3 //DD:STDIN’

where STDIN was DD *, trying to read from the inline data. Using //STDIN DD PATH=’/u/tmp/zos/z.py’ worked fine.

SyntaxError: Non-UTF-8 code starting with ‘\x83’ in file on line 1, but no encoding declared;

I got this using // PGM=AOPBATCH,PARM=’//usr/lpp/IBM/cyp/v3r8/pyz/bin/python3′ and letting the Python source default to //STDIN. I had to specified

// PGM=AOPBATCH,PARM=’//usr/lpp/IBM/cyp/v3r8/pyz/bin/python3 //DD:STDIN’ for it to work.

BPXM047I BPXBATCH FAILED BECAUSE SPAWN (BPX1SPN) OF
… FAILED WITH RETURN CODE 00000082 REASON CODE 0B1B0C27

Return code 82, 0000008 0x82 0x00000082 decimal 130 is Exec format error.

I got 0B1B0C27 because I had

//R EXEC PGM=BPXBATCH,REGION=0M,TIME=NOLIMIT,MEMLIMIT=NOLIMIT,
// PARM=’pgm /u/tmp/zos/y.py …’

Instead of

//R EXEC PGM=BPXBATCH,REGION=0M,TIME=NOLIMIT,MEMLIMIT=NOLIMIT,
// PARM=’pgm /usr/lpp/IBM/cyp/v3r8/pyz/bin/python3 /u/tmp/zos/y.py….’

I also got this trying to run a java program. I needed environment variable _BPX_SPAWN_SCRIPT=YES when using the BPXBATSL utility
to run the command (or a nested command).

FOPEN: EDC5129I No such file or directory. (errno2=0x05620062)

If you try to use fopen(“DD:xxxx”…) from a shell script (or BPXBATCH PARM=”pgm… ” you will get

FOPEN: EDC5129I No such file or directory. (errno2=0x05620062)

If you use fopen(“//’COLIN.VB’”…) and specify a fully qualified dataset name if will work.

fopen(“//VB”..) will put the RACF userid in front of the name. For example attempt to open “//’COLIN.VB.’”

CCN3276 Syntax error: possible missing ‘)’?

I had

48 | asm(
49 | " LA 2,%[p1] Adderss of the block \n"
50 | :
     a…………………………………………………………….
*=ERROR===========> a - CCN3276 Syntax error: possible missing ')'?
51 | : [p1] "m"(pMsg)
52 | : "r0","r1"
53 | );

This was caused by not having ASM in the compiler options.

SEVERE ERROR CCN1104: Internal error while compiling function ….. Unsupported Assembler Template. Compilation terminated.

I had

asm(
" LA 3,[%EPA] \n"
:
: [EPA] "m"(SWAEPA)
: "r1","r2"
);

with the %inside the []. It should be %[EPA]

PIP

I got

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1019)

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1019)

trying to install a product. See here.

zoneinfo/init.py … EDC5129I No such file or directory.

I was getting

u/tmp/zowet/colin/envz /lib/python3.12/site-packages/dateutil/zoneinfo/init.py:26: UserWarning: I/O error(129): EDC5129I No such file or directory. warnings.warn(“I/O error({0}): {1}”.format(e.errno, e.strerror))

I put some debug code in, and found it was trying to find dateutil-zoneinfo.tar.gz.

I tried installing various packages. Finally the following worked for me

pip uninstall python-dateutil
pip install python-dateutil
...
Successfully installed python-dateutil-2.9.0.post0

pip install tzdata
...
Successfully installed tzdata-2025.3

and it worked.

ModuleNotFoundError: No module named

My code was

import dumphex

... dumphex.dumphex()

and I got the message

ModuleNotFoundError: No module named ‘dumphex’

This was strange because the dumphex.py was in the same directory as the main python code.

Solution

It needs

from . import dumphex

Python on z/OS coding a C extension.

I was porting the pymqi code, which provides a Python interface to IBM MQ, to z/OS.

I’ve documented getting the code to build. I also had challenges trying to use it.

The code runs as ASCII!

When my C extension was built, it gets built with the ASCII option. This means any character constants are in ASCII not, EBCDIC.

My python program had

import sys
sys.path.append(‘./’)
import pymqi
queue_manager = ‘AB12’
qmgr = pymqi.connect(queue_manager)
qmgr.disconnect()

When my C code got to see the AB12 value, it was in ASCII. When the code tried to connect to the queue manager, it return with name error. This was because the value was in ASCII, and the C code expected EBCDIC.

You can take see if the program has been compiled with the ASCII flag using

#ifdef __CHARSET_LIB

… It is has been compiled with option ASCII
#else

If you use printf to display the queue manager name it will print AB12. If you display it in hex you will be 41423132 where A is x41, B is x42, 1 is x31 is 2 is x32.

In your C program you can convert this using the c function a2e with length option, for example

char EQMName[4];
memcpy(&EQMName[0],QMname,4);
__a2e_l(&EQMName[0],sizeof(EQMName));

// then use &EQMName

Converting from ASCII to EBCDIC in Python.

Within Python you have strings stored as Unicode, and byte data. If you have a byte array with x41423132 (the ASCII equivalent to AB12). You can get this in the EBCDIC format using

a=b’41423132′ # this is the byte array
m = a.decode(“ascii”) # this creates a character string
e = m.encode(‘cp500’) # this create the new byte array of the EBCDIC version .. or xC1C2F1F2

In Python you convert from a dictionary of fields into a “control block” using the pack function. You can use the “e” value above in this.
MQ control blocks have a 4 character eye catcher at the front eg “OD “. If you use the pack function and pass “OD ” you will pass the ASCII version. You will need to do the decode(‘ascii’) encode(‘CP500’) to create the EBCDIC version.

Similarly passing an object such as MQ queue name, will need to be converted to the EBCDIC version.

Converting from EBCDIC to ASCII

If you want to return character data from your C program to Python, you will need to the opposite.

For example m is a byte array retuned back from the load module.

#v is has the value b’C1C2F2F3′ (AB23)
m = v.decode(“cp500”)
a = m.encode(‘ascii’)

# a now has b’41423132′ which is the ascii equivilant

Python on z/OS – Helpful hints

This posts contains small snippets of useful information about using IBM Python on z/OS.

Setup an alias for python3

I set up an alias so I can type p3 or py instead of python3.

alias p3=”python3″
alias py=”python3″

Creating a .py file is harder than it looks

It was easy to create a .py file in Unix Services. It was hard to create a file which I could edit, would build and be compiled. For example I could execute a .py file. If I tried to compile it (as happens when you build it) I got

Compiling ‘/u/pymqi/mq2.py’…
*** Sorry: UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xa2 in position 0: invalid start byte

I found a couple of ways of solving this problem

Use

touch myfile.py
chtag -tc ISO8859-1 myfile.py
oedit myfile.py

To undo this use IBM-1047.

Copy an existing file for example

cp /u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/python3.10/site-packages/pip/main.py my.py

Then edit my.py, remove all of the content and add in my content using cut and paste.

A different approach – useful when copying packages to z/OS was to FTP a .py file from Linux as BINARY then use

chtag -tc t ISO8859-1 mynew.py

So that z/OS recognises it as ASCII.

Displaying [] in your OMVS session

I could only get ISPF to display [] by using settings, and configuring terminal type to be 3278 (not 3278A as the documentation says).

The OMVS command has a CONVERT option that lets you specify a conversion
table for converting between code pages. The table you want to specify depends
on the code pages you are using in MVS and in the shell. For example, if you are
using code page IBM-037 in MVS and code page IBM-1047 in the shell, specify the
following when you enter the TSO OMVS command:
OMVS CONVERT((BPXFX111))

For example the program

a = {“a1″,2,”a3”}
b = [“a1″,2,”a3”]
print(a)
print(b)

With OMVS CONVERT((BPXFX111)) when you run it, you get

{‘a1’, 2, ‘a3’}
[‘a1’, 2, ‘a3’]

With OMVS you get

{‘a1’, 2, ‘a3’}
Ý’a1′, 2, ‘a3’¨

How to edit source and get the brackets right

The provided .py files format ok in ISPF when using US English 037. This is different to my normal code page of Bracket CP037 Modified.

With US English 037 in ISPF editor I get

a = {“a1″,2,”a3”}
b = [“a1″,2,”a3”]

With Bracket CP037 Modified – which works for normal C

a = {“a1″,2,”a3”}
b = Ý“a1″,2,”a3”¨

What is installed

python3 -m pip list

How to install Python software in an isolated environment

If your z/OS is connected to the internet, you can use python3 -m pip install ….
It is harder if your z/OS is isolated and does not have direct connectivity.

Download the software to your workstation

I used site https://pypi.org/, found the software I was interested in (eg wheel) and downloaded it to Linux.

Upload to z/OS

I FTPed the file to z/OS in binary, keeping the same name wheel-0.37.1-py2.py3-none-any.whl

Install the software

You can install this in a virtual environment, or on the system wide Python. The file system Python is on needs to be writeable.

To display the syntax of the install command

python3 -m pip install -help

When I tried to install it using my non privileged userid I got


-[31]ERROR: Could not install packages due to an OSError: [Errno 111] EDC5111I Permission denied.: ‘/u/.local’ Check the permissions.
-[0]

I had to install it using my IBMUSER id which had more authority ( and could change any file etc), or use the option –no-cache-dir .

The command was

python3 -m pip install /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl
Looking in links: /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl
Processing /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.37.1
-[33]WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv-%5B0%5D

When installing other software I had to use options

  • -f : search for pre-req software in this directory
  • –no-cache-dir : because it tried to write to a read only cache
  • –user : to install it in the virtual environment, not the system wide python.

Uninstall the software

python3 -m pip uninstall wheel

What machine is this running on ?

My z/OS is z/OS 2.4 on zPDT on Linux.

import platform
sys.path.append('./')
import pymqi
print("machine",platform.machine())
print("platform",platform.platform())
print("arch ",platform.architecture())
print("process ",platform.processor())
print("compiler",platform.python_compiler())
print("system ",platform.system())

gives

machine 1090
platform OS-390-27.00-1090-64bit
arch (’64bit’, ”)
process
compiler Clang 4.0.1 (tags/RELEASE_401/final)
system OS/390

Using struct.pack on z/OS

I wanted to understand what pack did on z/OS. See “format characters” in the documentation. For example ‘c’ is character, ‘L’ is unsigned long. Using a statement like print(“i”,struct.pack(“i”,1)) to format a number I got


b b’\x01′
h b’\x00\x01′
H b’\x00\x01′
i b’\x00\x00\x00\x01′
I b’\x00\x00\x00\x01′
l b’\x00\x00\x00\x00\x00\x00\x00\x01′
L b’\x00\x00\x00\x00\x00\x00\x00\x01′
q b’\x00\x00\x00\x00\x00\x00\x00\x01′
Q b’\x00\x00\x00\x00\x00\x00\x00\x01′
n b’\x00\x00\x00\x00\x00\x00\x00\x01′
N b’\x00\x00\x00\x00\x00\x00\x00\x01′
P b’\x00\x00\x00\x00\x00\x00\x00\x01′

I also noticed a “funny”

z = b’AB23′
print(“4sll” , struct.pack(“4sll”,z,2,1))

gave me

4sll b’AB23\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x01′

The red text is from the long variable, the green text is the padding to make the long value on a long boundary (8).

I’ve run out of space on z/OS – what is using it all?

I hit this problem when trying to backup some data sets. There was plenty of free space on some of the volumes, but not much free space on the “user” volumes.

You can use ISPF 3;4 to list data sets on a volume or with a specified high level qualifier. You can use the sort command, for example sort tracks. This reports on number of tracks allocated sorted in descending size. You can deletes unwanted data sets.

If you go to ISPF 3;4 and specify option V for VTOC, and specify a Volume Serial, you can display information about the volume.

If you are running in Unix Services

du -a ./ | sort -n -r | head -n 30

Gives the top 30 files or directories

A short C quiz, and some gotcha’s

I’ve been looking at porting pymqi, the Python MQ interface to z/OS.

The biggest challenges where nothing to do with Pymqi.

So if you are bored after Christmas and want something stimulating… here are a few questions for you… The answers are below. I tried getting them displayed upside down, like all quality magazines; but that was too difficult.

Question 1. C question

I’ve reduced the problem I experienced,down to

int main() 
{ 
if ( 1==0 ) return 8; 
int rc; 
*=ERROR===========> CCN3275 Unexpected text 'int' encountered.
}                                                          

Hint: it works in a batch compile, using EDCCB

Question 2 binding in Unix Services

/bin/xlc a.o -L. -o b.so -Wl,INFO //’COLIN.MQ924.SCSQDEFS(CSQBRR2X)’ -Wl,dll c.x

Gave

FSUM3218 xlc: File //’COLIN.MQ924.SCSQDEFS(CSQBRR2X)’ contains an incorrect file suffix.

What do I need to do to fix it?

Question 3. Strange bind messages

Before I found the solution to problem number 2, I put the bind statements into a Unix Services file.

Using this gave me

IEW2326E 1221 THE FOLLOWING INVALID RECORD HAS BEEN SEEN:
=”lm-source” *

Copyright
IEW2326E 1221 THE FOLLOWING INVALID RECORD HAS BEEN SEEN:
IBM Corp. 2009, 2016 All Rights Reserved.

This bind statement was

cc -o mqsamp -W l,DYNAM=DLL,LP64 c.o mq.o

it worked without the mq.o

The mq.o file had

* <copyright                                                          * 
* notice="lm-source"                                                  * 
* (C) Copyright IBM Corp. 2009, 2016 All Rights Reserved.             * 
* </copyright>                                                        * 

Answer

  1. Using the cc compiler, it defaults to #pragma Langlvl(stdc89) which supports the c89 level of C. This says all variable declarations must come before any logic. This is relaxed in the c99 level, so specifying #pragma Langlvl(stdc99) cures it. You can also specify LANGLVL(EXTENDED) in the cc statement
  2. To include datasets in some of the binder options you need host file: filename with .OBJ suffix (object host file for the binder/IPA Link). When I used /bin/xlc a.o … -Wl,INFO //’COLIN.MQ924.SCSQDEFS.OBJ(CSQBRR2X)’ … it worked.
  3. The binder is not good at files in Unix Services, it likes records which are fixed block 80. The mq.o file had trailing blanks removed, and this confused it. I had to use a PDSE to get it to work.

Which came first, the chicken or the checksum.

The ability to sign Java jar files, or z/OS modules, has been around for many years. Using this the loader checks the digital signature in the object. This digital signature is a checksum of the object, and this checksum is encrypted and stored with the object. At load time, the loader calculates the checksum, and decrypts the checksum in the object and checks they match.

MQ now supports this for some of its objects; downloadable .zip, .tar and .gz files.

For some of these you need to download the public key to use. This raises the problem that an evil person may have taken the object, removed the official signing information, and added their own stuff. You then download their public certificate – see it works, it must be official.

To prevent this you can do the checksum on the public certificate, and make that available along with the official public key. (This is the chicken and egg problem. You need the certificate to be able to check the main code, and how to you check the certificate, without a certificate to check?)

On Linux you can calculate the checksum of a file using

sha256sum 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz

this gives 53c34cd374d7b08522423533ef2019b4aa0109a595fbaeab8ee6f927cb6c93ad, which is the same as the value on the IBM site. So this matches.

The IBM MQ code signatures page says IBM MQ public certificates, checksums, and .sig files are available from https://ibm.biz/mq92signatures. On this signatures page it says

release level: 9.2.4.0-IBM-MQ-Sigs-Certs 
Continuous Delivery: 9.2.4 IBM MQ file signatures, checksums and certificates

Platforms:  AIX 64-bit, pSeries, Linux 64-bit,x86_64, Linux 64-bit,zSeries, Linux PPC64LE, Windows 64-bit, x86, z/OS

This page is an httpS page, with the certificate issued by a proper Certificate Authority, and trusted third party. If you trust this CA, you can trust the IBM page.

When you click download, it downloads

  • 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz.sha256sum – this file content has 53c34cd374d7b08522423533ef2019b4aa0109a595fbaeab8ee6f927cb6c93ad
  • 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz

The value in the sha256sum file matches the value of the sha256sum 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz command.

As you can trust the security chain from the web page, through to the downloads, you can trust the .gz file.

Jar signing

Java has had the capability to sign a jar for at least 10 years.

The jarsigner command takes a jar file, a keystore with private key and calculates the checksum. It then encrypts it, and creates some files in the jar. For example

jarsigner -keystore trust.jks -storepass zpassword checkTLS.jar signer

This uses

  • the keystore called trust.jks,
  • with password zpassword
  • the checkTLS.jar file
  • and uses the certificate with alias name signer. This certificate must have extendedKeyUsage with codeSigning.

The jar file now has some additional files which can be seen using jar -tvf checkTLS.jar command.

  • META-INF/SIGNER.SF . This is the Signature File.
  • META-INF/SIGNER.EC .This is the public key to be used.

Where SIGNER is the name of the alias of the private key in the keystore, used to sign the jar file. The jar file can be signed many times by different private keys.

To verify the signature you can use

  • jarsigner -verify checkTLS.jar
  • jarsigner -verbose -certs -verify checkTLS.jar

The jarsigner -verbose -certs -verify checkTLS.jar gave me

- Signed by "CN=signer, O=cpwebuser, C=GB"
    Digest algorithm: SHA-256
    Signature algorithm: SHA256withECDSA, 256-bit key

jar verified.

Warning: 
This jar contains entries whose certificate chain is invalid. Reason: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
This jar contains signatures that do not include a timestamp. Without a timestamp, users may not be able to validate this jar after any of the signer certificates expire (as early as 2024-11-03).

The signer certificate will expire on 2024-11-03.

This shows that the jar file is consistent with the checksumming, but the certificate cannot be validated.

I can tell it which keystore to use to validate the certificate, using

jarsigner –keystore trust.jks -certs -verify checkTLS.jar

With the -verbose option you also get (with some of the output rearranged for clarity). The “s” or “sm” at the front of an object entry is s=signature verified, and m=entry listed in the manifest.

s = signature was verified 
m = entry is listed in manifest
k = at least one certificate was found in keystore
i = at least one certificate was found in identity scope


s 1402 Wed Dec 22 14:27:52 GMT 2021 META-INF/MANIFEST.MF

  >>> Signer
  X.509, CN=signer, O=cpwebuser, C=GB
  [certificate is valid from 22/12/21 14:51 to 30/01/25 16:46]
  X.509, CN=SSCA256, OU=CA, O=SSS, C=GB
  [certificate is valid from 04/11/21 15:48 to 03/11/24 15:48]

  [Invalid certificate chain: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

        1579 Wed Dec 22 16:12:04 GMT 2021 META-INF/SIGNER.SF
        1373 Wed Dec 22 16:12:04 GMT 2021 META-INF/SIGNER.EC

sm  54 Sat Jan 30 14:48:52 GMT 2021 checkTLS/checkTLS.manifest

  >>> Signer
  X.509, CN=signer, O=cpwebuser, C=GB
  [certificate is valid from 22/12/21 14:51 to 30/01/25 16:46]
  X.509, CN=SSCA256, OU=CA, O=SSS, C=GB
  [certificate is valid from 04/11/21 15:48 to 03/11/24 15:48]

  [Invalid certificate chain: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

When I downloaded the MQ 9.2.4 client and ran the jarsigner …. -verbose command the output included

sm 642 Thu Nov 04 16:01:46 GMT 2021 wlp/lib/extract/IFixUtils$ParsedIFix.class

[entry was signed on 04/11/21 17:58]
>>> Signer
X.509, CN=International Business Machines Corporation, OU=IBM CCSS, O=International Business Machines Corporation, L=Armonk, ST=New York, C=US
[certificate is valid from 25/08/21 01:00 to 26/08/23 00:59]
X.509, CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1, O="DigiCert, Inc.", C=US
[certificate is valid from 29/04/21 01:00 to 29/04/36 00:59]
      X.509, CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US
[trusted certificate]
... 

This shows that the certificate used to sign the component of the jar file was signed by CN=International Business Machines Corporation, which was in turn signed by CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1. The jarsigner program was able to use its public certificate to validate the CA, and so validate the IBM certficiate, and so validate the checksum.

Rexx to C to Rexx sample code

I’ve put up on github some sample code to demonstrate how you can write a function in C, and invoke it from Rexx. I’ve provided some glue code as Rexx uses R0 and R1 to pass parameters, and C programs only use R1.

I’ve create some small functions to use in your C program which hide the Rexx logic. For example

rc = CRexxDrop(pEnv,”ZDROP”);
rc = CRexxGet(pEnv,”InSymbol”,&buffer[0],&lBuffer);
rc = CRexxPut(pEnv,”CPPUTVar,”Colinsv”,0);
Iterate through all symbols

If you have any comments or suggestions, please let me know.

Where’s my invisible code.

In trying to get system exits written in C to work. I found my code was not being executed, even when the first instructions were a deliberate program check. I tried using the tried and trusted program AMASPZAP (known as Super Zap) for displaying the internals of a load module and zapping it – but my code was not there! Where was it hiding? When I took a dump of the address space my code was in the dump. Why was it invisible and not being executed?

HSM archives on tape

Like taking 20 minutes to recall a long unused dataset from HSM (mounting a physical tape to retrieve the data set), I had this vague memory of doing a presentation on the binder and the structure of load modules. After a cup of tea and a chocolate biscuit to help the recall, I remembered about classes etc in a load module.

Classes etc

When I joined IBM over 40 years ago you wrote your program, and used the link editor to create the load module, a single blob of instructions and data.

Things have moved on. Think about a C program, in read only memory. When you issue a load to use it, you get a pointer to the read only (the re-entrant instructions and data), and your own copy of the “global variables” or Writeable Static Area (WSA). When using the C compiler, at bind time it includes a bit of code with 24 bit addressing mode. This means you have code which runs in 31/64 bit mode, and some code resident in 24 bit mode! It is no longer a single blob of instructions and data.

Within the load module there are different classes of data for example

  • C_CODE – C code
  • C_WSA – for a C program compiled with RENT option. This is the global data which each instance gets its own private copy of
  • B_TEXT code from the assembler
  • Using the HL Assembler, you can define your own classes using CATTR.

A class has attributes, such as

  • Length.
  • Should it be loaded or not. You could store documentation in the load module, which can be used by programs, but not needed at execution time.
  • RMODE.
  • It is reentrant or not.
  • Should this code be merged or replaced with similar code. For example the C Globals section would be merged. A block of instructions would be replace.

Segments

The binder can take things with similar attributes and store them together within a segment. You can have mixed classes eg B_TEXT and C_CODE, with the same RMODE attributes etc and have them in one segment. The C_WSA needs to be in a different segment because it has different attributes.

So where was my invisible code?

I needed to change my SPZAP job to tell it to dump out the C_CODE section. By default it dumps the B_TEXT sections. You can specify C_* or B_*. See the AMASPZAP documentation.

//STEP EXEC PGM=AMASPZAP
//SYSPRINT DD SYSOUT=A
//SYSLIB DD DISP=SHR,DSN=COLIN.C.LOAD
//SYSIN DD *
DUMPT COLIN CPPROGH C_CODE
/*

This dumps out (decoding the data into instructions) load module COLIN, CSECT CPPROGH and the C_CODE class.

Why wasn’t my code executing? The code to set up the C environment was not invoking my program because I had compiled it with the wrong options!

Writing system exits in C (and compiling them).

I wanted to call a C program from Rexx to do some special processing. The C programming guide gave me some hints, but I found it was a struggle to do it. It reminded me of when I was young and my father gave me a “beginners electronics kit” where had transistors, resisters, etc. You could build a “computer” that counted to 3, and make a radio. Unfortunately the instructions that came in it were in German, and for a different model kit to what I had. As a result it was very difficult to get working, but once you knew it was easy.

In the C programming guide there were instruction like “The CSECT must be the module entry point.” without saying which CSECT to use. They gave some sample programs, but not the JCL to compile them. After many failures, (looking at dumps and traces) I found you had to compile the C programs with “NORENT” which went against many years of experience.

I was using the System Programming C facility, which can be used, for example as z/OS exits. Note: This is different to Metal C, which allows you to include assembler code in your C program.

Some background

  • These programs do not have a main() but are invoked with a z/OS type parameter list.
  • They can use C facilities, such as printf, but not LE functions.
  • You cannot use the UNIX file system functions.
  • They need to be called with the C environment set up. You cannot just branch to the entry point.
  • You can have several functions in the same source file. You branch to the one of interest.

Simple case

My C program was

#pragma environment(CPPROGH)
#pragma csect (CODE, “OEMPUT”)
int CPPROGH(int * p, evalblock * pEval, char * env) {
….
return 0;
}

The pragma environment said set up the C environment before calling executing this function. It takes the standard z/OS parameter list.

I needed some glue code to take the parameters from Rexx and store them in a parameter list for the function.

This glue codes saves parameters from R0,and 16(r1) and 20 (r1), then executes the function.

ENVA RMODE ANY
ENVA AMODE 31 
ENVA  CSECT
  ...   
  L    R3,16(R1)  a(Parmlist) 
  ST   R3,Parmlist+0 
  L    R3,20(R1)  a(evalblk) 
  L    R3,0(R3) 
  ST   R3,Parmlist+4 
  ST   R0,PARMLIST+08  A(env block) 
  OI   PARMLIST+08,X'80' 
  la   r1,parmlist 
  L     R15,=V(CPPROGH) 
  BASR  R14,R15 

I wanted this to be called from REXX, which passes parameters in R0 and R1, so I had to write some glue code to store the parameters in storage before passing them to the program.

I compiled the glue code with

//GLUE EXEC PGM=ASMA90,PARM=’DECK,NOOBJECT,LIST,XREF(SHORT),NORENT’,
// REGION=0M
//SYSLIB DD DISP=SHR,DSN=CEE.SCEEMAC
// DD DSN=SYS1.MACLIB,DISP=SHR
//SYSUT1 DD UNIT=SYSDA,SPACE=(CYL,(1,1))
//SYSPUNCH DD DISP=SHR,DSN=COLIN.C.REXX.OBJ(GLUE2)
//SYSPRINT DD SYSOUT=*
//SYSIN DD DISP=SHR,DSN=COLIN.C.REXX(GLUE2)
//*

and compiled the C code with

//S1 JCLLIB ORDER=CBC.SCCNPRC
// SET LOADLIB=COLIN.C.REXX.LOAD
// SET LIBPRFX=CEE
//COMPILE EXEC PROC=EDCCB,
// LIBPRFX=&LIBPRFX,
// CPARM=’OPTFILE(DD:SYSOPTF),NORENT‘,
// BPARM=’SIZE=(900K,124K),RENT,LIST,XREF,RMODE=ANY,AMODE=31′
//COMPILE.SYSOPTF DD DISP=SHR,DSN=COLIN.C.REXX(CPARMS)
//COMPILE.SYSIN DD DISP=SHR,DSN=COLIN.C.REXX(CPPROGHE)
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB.
//BIND.SYSLIB DD DISP=SHR,DSN=CEE.SCEESPC
// DD DISP=SHR,DSN=CEE.SCEELKED
//BIND.OBJLIB DD DISP=SHR,DSN=COLIN.C.REXX.OBJ
//BIND.SYSIN DD *
INCLUDE OBJLIB(GLUE2)
ENTRY ENVA
NAME COLIN(R)
/*

The EDCCB procedure to compile and bind, stores the object deck in a temporary file then passes this file and BIND.SYSIN into the binder.

C persistent environment.

The previous example created a C environment, ran my program, and deleted the C environment. If you want to do many calls to C functions you can set up a Persistent C environment. In this environment you do

  • From assembler, set up the environment
  • From assembler, use the environment, and call functions with your program as many times as you need
  • From assembler close down the environment,

This is well documented in the C programming guide, (but not how to compile it).

The essence of my program was

Set up the environment

L R15,=V(EDCXHOTL)
BASR R14,R15

Call my function

   LA R4,HANDLE 
   LA R5,USEFN  This has the  
   STM  R4,R5,PARMLIST 
* now the user paramaters
  ...
   OI   PARMLIST+16,X'80' 
   LA   R1,PARMLIST 
   L    R15,=V(EDCXHOTU) 
   BASR R14,R15 
...
USEFN    DC V(CPPROGH) <<  This function name

Clean up

    LA R1,PARMLIST 
    OI 0(R1),X'80' 
    L R15,=V(EDCXHOTT) 
    BASR R14,R15 

My C program was

#pragma linkage(CPPROGH,OS)
int CPPROGH(int * p, evalblock * pEval, char * env) {
printf(“in CPPROG\n”);
return 0}

In this case the pragma is LINKAGE(CPPROGH,OS). The previous, self contained code, had ENVIRONMENT(CPPROGH). You need to use the right one.

Which procedure do I use to compile?

The C books describe the various procedures, for example EDCCB for compile and BIND, and EDCCL for compile and LINKEDIT. They do the same thing. The LINKEDIT uses program HEWL to link edit. The BIND uses IEWL to invoke the binder. These are both aliases to the binder IEWBLINK.

What’s the difference between BALR and BASR?

When coding, my fingers automatically used BALR (Branch and Link Register). This worked fine, but I should have used BASR (Branch and Save Register). As the Principles of Operation (POP) says

It is recommended, however, that BRANCH AND SAVE (BAS and BASR) be used instead and that BRANCH AND LINK be avoided since it places nonzero information in bit positions 32-39 of the general register in the 24-bit addressing mode, which may lead to problems and may decrease performance.

In 31 bit mode with BALR 14,15, the return address is stored in register 14. ‘1’ followed by the 31 it address.

In 24 bit mode, the return address has other information at the top, including the condition code. Most of the time this information will be ignored.

So using BALR is not wrong, it is that BASR is better.