Python on z/OS – creating a C extension

I enjoy using Python on Linux, because it is very powerful. I thought it would be interesting to port the MQ Python interface pymqi to z/OS. This exposed many of the challenges of running Python on z/OS.

I’ll cover some of the lessons I learned in doing this work. Thanks to Steve Pitman who helped me package the extension.

Creating files that would compile was a challenge.

See here.

Compiling files.

I copied the pymqi C code to z/OS Unix Services, and tried to compile it. This was a mistake, as it took me a long time to get the compile options right. I found that using the setup.py script was the right way to go.

My directory tree

/u/pymqi
..setup.py
..include
....sample.h
..code
....pymqi
......__init__.py        
......CMQCFC.py          
......CMQXC.py           
......CMQZC.py           
......const.py           
......CMQC.py            
......pymqe.c            

Setup.py

This script needs export _C89_CCMODE=1, otherwise you get FSUM3008 message

Specify a file with the correct suffix (.c, .i, .s, .o, .x, .p, .I, or .a), or a corresponding data set name, instead of -L

import setuptools 
from distutils.core import setup, Extension 
import os 
import sysconfig 
# 
# This script needs    export _C89_CCMODE=1 
# Otherwise you get FSUM3008  messages 
# 
import os 
os.environ['_C89_CCMODE'] = '1' 
bindings_mode = 1 
version = '1.12.0' 
setup(name = 'pymqi', 
    version = version, 
    description = 'Python...', 
    platforms='OS Independent', 
    package_dir = {'': 'code'}, 
    packages = ['pymqi'],  
    py_modules = ['pymqi.CMQC', 'pymqi.CMQCFC', 'pymqi.CMQXC', 'pymqi.CMQZC'], 
    ext_modules = [Extension('pymqi.pymqe',['code/pymqi/pymqe.c'], define_macros=[('PYQMI_BINDINGS_MODE_BUILD', 
bindings_mode)], 
    include_dirs=["//'COLIN.MQ924.SCSQC370'"], 
    extra_link_args=["//'COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ2X)'"], 
      )] 
) 
# I had extra_link_args=["-Wl,INFO,LIST,MAP",.... when setting 
# this up
# I used 
# extra_compile_args=["-Wc,LIST(c.lst),XREF"], 
# to get out a listing and cross reference.

Which says

  • The package name is packages = [‘pymqi’],
  • The Python files are py_modules = [‘pymqi.CMQC’….
  • There is an extension .. ext_modules=.. with the source program code/pymqi/pymqe.c
  • It needs “//’COLIN.MQ924.SCSQC370′” to compile and “//’COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ2X)'” at bind time. This file contains the MQ Binder input
  • When I wanted the binder output – “-Wl,INFO,LIST,MAP”. This goes to the terminal. I used a ‘>’ command to pipe the output of the python3 setup build it to a file.
  • and a C listing “-Wc,LIST(c.lst),XREF”. The listing goes to c.lst

You need

  • import setuptools so that the setup bdist_wheel packaging works. You also need the wheel package installed.

Setup

There is a buglet in the compile set up. You need to specify

export _C89_CCMODE=1

Without it you get

FSUM3008 Specify a file with the correct suffix (.c, .i, .s, .o, .x, .p, .I, or .a), or a corresponding data set name, instead of -obuild/lib.os390-27.00-1090-3.8/pymqi/pymqe.so.

You also need the binder input in a data set with the correct suffix. For example .OBJ

“//’COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ2X)'”

If you do not have the correct suffix you get

FSUM3218 xlc: File //’COLIN.MQ924.SCSQDEFS(CSQBMQ2X)’ contains an incorrect file suffix.

Doing the compile and test install

I used a shell script to do the compiles and install

touch code/pymqi/*.c
rm a b c d
export _C89_CCMODE=1
#python3 setup.py clean
python3 setup.py build 1>a 2>b
python3 setup.py install 1>c 2>d

I captured the output from the setup.py jobs using 1>a etc because I could not see how to direct the binder output to a file. It comes out on the terminal – and there was a lot of it!.

Packaging the package

Python build which worked

I had to install wheel package. See How to install software in an isolated environment, or just use python3 -m pip install wheel if your z/OS image is connected to the network.

The command I used was

python3 -m pip install –user –no-cache-dir /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl-f /u/tmp/py/wheel-0.37.1-py2.py3-none

I had to add import setuptools to my setup.py file (at the top). (This converted the install package from a dist-utils to a setuptools packaging)

python3 setup.py bdist_wheel

This created a file “/u/pymqi/dist/pymqi-1.12.0-cp310-cp310-os390_27_00_1090.whl

This file is specific to python 3.10

For a wheel package, you’ll need to build it for all major versions and cannot just use one. Note that there were some issues in 3.8/3.9 with wheels that have been resolved in 3.10, so it’s recommended you to use 3.10.

Steven Pitman

This means you need to have multiple levels of Python installed, and build for each one!

This also has the operating system level (os390_27_00 – this may be constant across machines) and the hardware 1090. For this to work on other hardware, one solution would be to manually rename the file to pretend it is for a different machine, but this both not supported nor recommended, and has no guarantee to work. So it is hard to know the best thing to do. I do not have every 390 machine from IBM to do a build on !

Failing build. This built but did not install.

python3 setup.py bdist –format=tar

It built the package and create a file

./dist/pymqi-1.12.0.os390-27.00-1090.tar

This tar file is not completely readable by the z/OS tar command.

When I used tar -tf ….tar it gave

FSUMF371 Value 1641318408.0 is not valid for keyword mtime. Keyword not set.

It uses a Python tar command, not the operating system tar command.

You can display the contents using a Python program like

import tarfile
tar = tarfile.open("dist/pymqi-1.12.0.os390-27.00-1090.tar.gz")
# tar.extractall() 
for x in tar:
    print(x)

This gave output like

<TarInfo ‘.’ at 0x5008ad3880>
<TarInfo ‘./usr’ at 0x5008ad3dc0>
<TarInfo ‘./usr/lpp’ at 0x5008ad3a00>

Installing the package

From an authorised user in OMVS,

python3 -m pip install –no-cache-dir /u/pymqi/dist/pymqi-1.12.0-cp310-cp310-os390_27_00_1090.whl /u/pymqi/dist/pymqi-1.12.0-cp310-cp310-os390_27_00_1090.whl
Processing /u/pymqi/dist/pymqi-1.12.0-cp310-cp310-os390_27_00_1090.whl
Installing collected packages: pymqi
Successfully installed pymqi-1.12.0

If you do not use –no-cache-dir, you may get

-[33]WARNING: The directory ‘/u/.cache/pip’ or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo’s -H flag.-[0]

The compile options

The following text is the compile and bind options used for my code. Some of the options are pymqi specific.

/bin/xlc -DNDEBUG -O3 -qarch=10 -qlanglvl=extc99 -q64
-Wc,DLL
-D_XOPEN_SOURCE_EXTENDED
-D_UNIX03_THREADS
-D_POSIX_THREADS
-D_OPEN_SYS_FILE_EXT
-qexportall -qascii -qstrict -qnocsect
-Wa,asa,goff -Wa,xplink
-qgonumber -qenum=int
-DPYQMI_BINDINGS_MODE_BUILD=1 -I//’COLIN.MQ924.SCSQC370′
-I/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/include/python3.10
-c code/pymqi/cpmqe.c
-o build/temp.os390-27.00-1090-3.10/code/pymqi/cpmqe.o

/bin/xlc build/temp.os390-27.00-1090-3.10/code/pymqi/cpmqe.o -L.
-o build/lib.os390-27.00-1090-3.10/pymqi/cpmqe.cpython-310.so -Wl,INFO,LIST,MAP,DLL //’COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ2X)’
-Wl,dll
/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/python3.10/config-3.10/libpython3.10.x
-q64

On minute SMS

For many people SMS is something on z/OS that gets in the way of doing real work, and you find you are trying to work around it. A bit like Clippy on Window.

SMS is System Managed Storage. Before SMS, the systems programmer would be responsible for which data sets go on which volume; how often to backup these data sets, and get annoyed when the z/OS users allocated data sets inefficiently, wasting space and using the wrong data set attributes.

SMS has made the systems programmers job much easier. Instead of having a list of disk volumes which can be used by developers, and a list of volumes reserved for systems people, then telling people which disk to use; SMS allows you to say group these volumes as SYSPROGS, and these volumes as OTHER, and when you allocate a data set the system says “COLIN.*” maps to OTHER; SYS1.* map to SYSPROGS.

You can disks have which are not SMS managed, but these days, most of the disks are SMS managed.

You can also specify rules such as all SYS1.*.PARMLIB get backed up daily – and keep the last 10 copies; and do not backup these temporary files.

The system can also do house keeping and say if these user data sets have not been used for a long period, backup them and move them to slower volumes or tape.

SMS topics

SMS has different “topics”.

A data class is used when creating a data set. It influences the space and data set attributes. You could set up a data class of USERFB80 for user data sets which are Fixed block 80 records. This data class can specify optimum block size to be used, and override any value the user has specified.

A management class provides information about how often it is to be backed up, how long it is to be kept for before automatic deletion, and if it can be compressed or moved out to tape if it has not been used for a period.

A Storage class provides information about which disk volumes to use. You may have some high performing disks, which you want databases to use, and some old spinning disks, for the end users to use.

A storage group is which disks belong to which storage group. For example storage group SGCICS has volumes CICS00 through to CICS1F, storage group SGMQ has volumes SGMQS – MQ0000 through MQ004, and OLD001.

The Automatic Class Selection (ACS) is the magic which says “my datasets get fast disks”; “your data sets get slow disks, and the data sets are migrated out to tape if they haven’t been used for a week”. At a conceptual level it is like

if DSN in “SYS1.**” then STORCLAS = “SGSYS1”
if SIZE > 4052MB then DATACLAS= BIGFILE
Select
When(STORCLAS=”IMS”) then STORGRP = “SGIMS”
When(STORCLAS=”MQ”) then STORGRP=”SGMQ”
end

When are these visible?

I compiled a C program job. In the JCL output it had

IGD101I SMS ALLOCATED TO DDNAME (SYSLIN )
DSN (SYS22013.T150656.RA000.COLINC3.LOADSET.H01 )
STORCLAS (SCBASE) MGMTCLAS ( ) DATACLAS ( )
VOL SER NOS= VIO

So we can see the storage class used (SGBASE), and the data class, and management class (both not specified).

I’ve run out of space – but there is plenty of space.

I got the following message, and struggled to resolve the problem.

IGD17272I VOLUME SELECTION HAS FAILED FOR INSUFFICIENT SPACE FOR
DATA SET SYS21355.T163955.RA000.COLINMQ.TEMP.H01
JOBNAME (COLINMQ ) STEPNAME (S1 )
PROGNAME (AMATERSE) DDNAME (SYSUT2 )
REQUESTED SPACE QUANTITY = 415019 KB
STORCLAS (SCBASE) MGMTCLAS ( ) DATACLAS ( )
STORGRPS (SGBASE SGEXTEAV SGVIO )

SMS is clearly involved – but how to display more information? You can use ISPF ISMF panels to display information, but sometimes it is easier to use the DISPLAY SMS operator command. For example

What is the status of a volume

d sms,vol(C4USR1)
IGD002I DISPLAY SMS
VOLUME UNIT MVS   SYSTEM= 1   STORGRP NAME
C4USR1 0A9B ONRW          +   SGBASE
* LEGEND *
. THE STORAGE GROUP OR VOLUME IS NOT DEFINED TO THE SYSTEM
+ THE STORAGE GROUP OR VOLUME IS ENABLED
* - THE STORAGE GROUP OR VOLUME IS DISABLED
D THE STORAGE GROUP OR VOLUME IS QUIESCED
Q THE STORAGE GROUP OR VOLUME IS DISABLED FOR NEW ALLOCATIONS ONLY

Where the + in the line with the volume is described in the following lines. So the C4USR1 means the volume is enabled for SMS.

ONRW is ONline ReadWrite

What storage groups are defined?

D SMS,STORGRP(ALL)
D SMS,SG(SGBASE)

Gave

SGBASE POOL +
SPACE INFORMATION:
TOTAL SPACE = 16240MB USAGE% = 91 ALERT% = 0
TRACK-MANAGED SPACE = 16240MB USAGE% = 91 ALERT% = 0

if you want more information

D SMS,SG(SGBASE),LISTVOL

Gave

STORGRP TYPE SYSTEM= 1
SGBASE POOL +
SPACE INFORMATION:
TOTAL SPACE = 16240MB USAGE% = 91 ALERT% = 0
TRACK-MANAGED SPACE = 16240MB USAGE% = 91 ALERT% = 0 

VOLUME UNIT MVS SYSTEM= 1 STORGRP NAME
C4USR1 0A9B ONRW        + SGBASE
USER00 0A9C ONRW        + SGBASE
USER01                  + SGBASE
USER0A                  + SGBASE
...

There were lots of volumes defined … but only two were online and available.

SMS Enable/disable volumes

You can disable a volume from SMS using

V SMS,VOL(C4USR1),DISABLE

The command
D SMS,VOL(C4USR1)

now gives

VOLUME UNIT MVS    SYSTEM= 1 STORGRP NAME
C4USR1 0A9B ONRW           - SGBASE
+ THE STORAGE GROUP OR VOLUME IS ENABLED
- THE STORAGE GROUP OR VOLUME IS DISABLED

The JCL below will allocate a (temporary) data set on volume C4CFG1, and then delete it.

//IBMUSER1 JOB 1,MSGCLASS=H
//S1 EXEC PGM=IEFBR14
//DD1 DD DSN=&TEMP,DISP=(NEW,PASS),
// SPACE=(CYL,(1,1)),DCB=(LRECL=80,RECFM=FB),
// VOL=SER=(C4CFG1),
// STORCLAS=SCNOSMS
//S2 EXEC PGM=IEFBR14
//DD1 DD DSN=*.S1.DD1,DISP=(OLD,DELETE)

If I removed the STORCLAS=SCNOSMS, the data set was allocated on a different volume C4USR1 with STORCLAS (SCBASE).

Help ! My ZFS has filled up

The file system I was using in Unix Services filled up – but it didn’t tell me,I just had a truncated file. I piped the output of a shell script to a file. As the file system filled up – it could not write the “file system full” message.

To solve this file system full problem, I had to explore other areas of z/OS which I was not so familiar with – ADRDSSU to move data sets, zfsadmin commands, and how to stop SMS from being too helpful.

Having made the ZFS.USERS larger, I then found IEC070I 104-204 is data set is > 4GB. So some of this blog post is wrong!

The Unix Services command

df -P /u/pymqi

tells you the file system – and how full it is. This gave me

Filesystem 512-blocks   Used Available Capacity Mounted on
ZFS.USERS      204480 203954       526     100% /u

So we can see the data set is ZFS.USERS and it is 100% full.

The command

zfsadm fsinfo ZFS.USERS

give more (too much) information, and

zfsadm aggrinfo ZFS.USERS

doesnt quite give enough info. df -P … is best

I used the command

zfsadm grow ZFS.USERS -size 144000

to make it bigger, but I got the Unix Services message

IOEZ00326E Error 133 extending ZFS.USERS

and on the system log

IOEZ00445E Error extending ZFS.USERS. 591
DFSMS return code = 104, PDF code = 204.
IOEZ00308E Aggregate ZFS.USERS failed dynamic grow, (by user COLIN).
IOEZ00323I Attempting to extend ZFS.USERS to 36000 4096 byte control intervals.
IEF196I IEC070I 104-204,OMVS,OMVS,SYS00022,0A9E,C4USS2,ZFS.USERS,
IEF196I IEC070I ZFS.USERS.DATA,CATALOG.Z24C.MASTER
IEC070I 104-204,OMVS,OMVS,SYS00022,0A9E,C4USS2,ZFS.USERS, 594
IEC070I ZFS.USERS.DATA,CATALOG.Z24C.MASTER
IOEZ00445E Error extending ZFS.USERS. DFSMS return code = 104, PDF code = 204.

DFSMS return code = 104, PDF code = 204 means no space on the volume.

I used ISPF 3;4 to display the volume the ZFS.USERS data set was on; C4USS2.

I used ISPF 3;4 to display what data sets were on the C4USS2 volume. If you use PF11 to can see the space allocated to each data set.

I could try moving this dataset to another volume, but that would mean unmounting it, moving it, remounting it. I thought it easier to move other dataset off the volume.

On the volume, I found a ZFS which I was not using and unmounted it

unmount filesystem(‘ZFS.Z24C.ZCX’) normal

and trying to move it, looked easy using DFDSS COPY DATASET .

//IBMUSER1 JOB 1,MSGCLASS=H
//STEP1 EXEC PGM=ADRDSSU,REGION=0M
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
COPY DATASET(INCLUDE(ZFS.Z24C.ZCX))-
ODY(C4USS1) DELETE CATALOG
/*

When I ran this job – it moved the dataset, but moved it to a USER00 volume – filling up most of the space on this volume. I had just moved the problem. SMS intercepted my request and “managed” the disk storage for me.

I added BYPASSACS and NULLSTORCLA

COPY DATASET(INCLUDE(ZFS.Z24C.ZCX))-
BYPASSACS(ZFS.Z24C.ZCX) –
NULLSTORCLAS –
ODY(C4USS2) DELETE CATALOG

and this worked.

  • BYPASSACS – do not use ACS routines to decide where to put the data set
  • NULLSTORCLAS ( or STORCLAS(xxxxx)) do not use a Storage class.
  • ODY OUTDYNAM specifies that the output DASD volume is to be dynamically allocated.

I then had enough space to be able to grow the zfs.

Messages when using python on z/OS

This post gives some of the error messages I received, and the actions I took to resolve the problems.

FSUM3008 Specify a file with the correct suffix (.c, .i, .s, .o, .x, .p, .I, or .a), or a corresponding data set name, instead of -o ….

I got this during a Python C extension build. You need

export _C89_CCMODE=1
export _C99_CCMODE=1
export _Ccc_CCMODE=1
export _CC_CCMODE=1
export _CXX_CCMODE=1
export _C89_CCMODE=1
export _CC_EXTRA_ARGS=1
export _CXX_EXTRA_ARGS=1
export _C89_EXTRA_ARGS=1

Before doing any builds.

Python builds

DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives.

Easy fix which no one tells you about (it took me 3 days to find this). Add

import setuptools

to the top of the file.

COLIN:/u/pymqi: >python3 -m build
No module named build.main; ‘build’ is a package and cannot be directly executed

You have a build directory in your project

https://pypi.org/search/?q=build

then install it

python3 setup.py bdist_wheel … error: invalid command ‘bdist_wheel’

I needed “import setuptools” at the top of the setup.py file. I also needed wheel to be installed.

CEE3501S The module libpython3.10.so was not found.

I was trying to do

import ctypes
from ctypes.util import find_library
testlib = ctypes.CDLL(“… “)

This file was in /u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/

You need

export LIBPATH=/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/:$LIBPATH

python3 ….

CEE3587S A call was made to a function in the AMODE 31 DLL //ADD2 from an AMODE 64 caller.

I was trying to call a C program from Python – but i was built with 31 bit mode – not 64 bit mode.

You need to compile it with LP64, and bind in 64 bit mode, and use //BIND.SYSLIB DD DISP=SHR,DSN=CEE.SCEEBND2

ImportError: CEE3512S An HFS load of module /u/tmp/py/mq.so failed. The system return code was 0000000130; the reason code was 0BDF0

I had a bind error. When I fixed it – it worked.

The IBM documentation says

A package shared library may get tagged incorrectly when using the xlc utility. Verify that the shared library is untagged by running the following line:

ls -alT

If the file is tagged, with the output being similar to the following line:

t ISO8859-1 T=on

you can remove the tag with the following command:

chtag -r <filename.so>

Python on z/OS coding a C extension.

I was porting the pymqi code, which provides a Python interface to IBM MQ, to z/OS.

I’ve documented getting the code to build. I also had challenges trying to use it.

The code runs as ASCII!

When my C extension was built, it gets built with the ASCII option. This means any character constants are in ASCII not, EBCDIC.

My python program had

import sys
sys.path.append(‘./’)
import pymqi
queue_manager = ‘AB12’
qmgr = pymqi.connect(queue_manager)
qmgr.disconnect()

When my C code got to see the AB12 value, it was in ASCII. When the code tried to connect to the queue manager, it return with name error. This was because the value was in ASCII, and the C code expected EBCDIC.

You can take see if the program has been compiled with the ASCII flag using

#ifdef __CHARSET_LIB

… It is has been compiled with option ASCII
#else

If you use printf to display the queue manager name it will print AB12. If you display it in hex you will be 41423132 where A is x41, B is x42, 1 is x31 is 2 is x32.

In your C program you can convert this using the c function a2e with length option, for example

char EQMName[4];
memcpy(&EQMName[0],QMname,4);
__a2e_l(&EQMName[0],sizeof(EQMName));

// then use &EQMName

Converting from ASCII to EBCDIC in Python.

Within Python you have strings stored as Unicode, and byte data. If you have a byte array with x41423132 (the ASCII equivalent to AB12). You can get this in the EBCDIC format using

a=b’41423132′ # this is the byte array
m = a.decode(“ascii”) # this creates a character string
e = m.encode(‘cp500’) # this create the new byte array of the EBCDIC version .. or xC1C2F1F2

In Python you convert from a dictionary of fields into a “control block” using the pack function. You can use the “e” value above in this.
MQ control blocks have a 4 character eye catcher at the front eg “OD “. If you use the pack function and pass “OD ” you will pass the ASCII version. You will need to do the decode(‘ascii’) encode(‘CP500’) to create the EBCDIC version.

Similarly passing an object such as MQ queue name, will need to be converted to the EBCDIC version.

Converting from EBCDIC to ASCII

If you want to return character data from your C program to Python, you will need to the opposite.

For example m is a byte array retuned back from the load module.

#v is has the value b’C1C2F2F3′ (AB23)
m = v.decode(“cp500”)
a = m.encode(‘ascii’)

# a now has b’41423132′ which is the ascii equivilant

Python on z/OS – Helpful hints

This posts contains small snippets of useful information about using IBM Python on z/OS.

Creating a .py file is harder than it looks

It was easy to create a .py file in Unix Services. It was hard to create a file which I could edit, would build and be compiled. For example I could execute a .py file. If I tried to compile it (as happens when you build it) I got

Compiling ‘/u/pymqi/mq2.py’…
*** Sorry: UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xa2 in position 0: invalid start byte

I found a couple of ways of solving this problem

Use

touch myfile.py
chtag -tc t ISO8859-1 myfile.py
oedi myfile.py

Copy an existing file for example

cp /u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/python3.10/site-packages/pip/main.py my.py

Then edit my.py, remove all of the content and add in my content using cut and paste.

A different approach – useful when copying packages to z/OS was to FTP a .py file from Linux as BINARY then use

chtag -tc t ISO8859-1 mynew.py

So that z/OS recognises it as ASCII.

Displaying [] in your OMVS session

The OMVS command has a CONVERT option that lets you specify a conversion
table for converting between code pages. The table you want to specify depends
on the code pages you are using in MVS and in the shell. For example, if you are
using code page IBM-037 in MVS and code page IBM-1047 in the shell, specify the
following when you enter the TSO OMVS command:
OMVS CONVERT((BPXFX111))

For example the program

a = {“a1″,2,”a3”}
b = [“a1″,2,”a3”]
print(a)
print(b)

With OMVS CONVERT((BPXFX111)) when you run it, you get

{‘a1’, 2, ‘a3’}
[‘a1’, 2, ‘a3’]

With OMVS you get

{‘a1’, 2, ‘a3’}
Ý’a1′, 2, ‘a3’¨

How to edit source and get the brackets right

The provided .py files formast ok in ISPF when using US English 037. This is different to my normal code page of Bracket CP037 Modified.

With US English 037 in ISPF editor I get

a = {“a1″,2,”a3”}
b = [“a1″,2,”a3”]

With Bracket CP037 Modified – which works for normal C

a = {“a1″,2,”a3”}
b = Ý“a1″,2,”a3”¨

What is installed

python3 -m pip list

How to install Python software in an isolated environment

If your z/OS is connected to the internet, you can use python3 -m pip install ….
It is harder if your z/OS is isolated and does not have direct connectivity.

Download the software to your workstation

I used site https://pypi.org/, found the software I was interested in (eg wheel) and downloaded it to Linux.

Upload to z/OS

I FTPed the file to z/OS in binary, keeping the same name wheel-0.37.1-py2.py3-none-any.whl

Install the software

You can install this in a virtual environment, or on the system wide Python. The file system Python is on needs to be writeable.

To display the syntax of the install command

python3 -m pip install -help

When I tried to install it using my non privileged userid I got


-[31]ERROR: Could not install packages due to an OSError: [Errno 111] EDC5111I Permission denied.: ‘/u/.local’ Check the permissions.
-[0]

I had to install it using my IBMUSER id which had more authority ( and could change any file etc), or use the option –no-cache-dir .

The command was

python3 -m pip install /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl
Looking in links: /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl
Processing /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.37.1
-[33]WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv-%5B0%5D

When installing other software I had to use options

  • -f : search for pre-req software in this directory
  • –no-cache-dir : because it tried to write to a read only cache
  • –user : to install it in the virtual environment, not the system wide python.

Uninstall the software

python3 -m pip uninstall wheel

What machine is this running on ?

My z/OS is z/OS 2.4 on zPDT on Linux.

import platform
sys.path.append('./')
import pymqi
print("machine",platform.machine())
print("platform",platform.platform())
print("arch ",platform.architecture())
print("process ",platform.processor())
print("compiler",platform.python_compiler())
print("system ",platform.system())

gives

machine 1090
platform OS-390-27.00-1090-64bit
arch (’64bit’, ”)
process
compiler Clang 4.0.1 (tags/RELEASE_401/final)
system OS/390

Using struct.pack on z/OS

I wanted to understand what pack did on z/OS. Using a statement like print(“i”,struct.pack(“i”,1)) to format a number I got


b b’\x01′
h b’\x00\x01′
H b’\x00\x01′
i b’\x00\x00\x00\x01′
I b’\x00\x00\x00\x01′
l b’\x00\x00\x00\x00\x00\x00\x00\x01′
L b’\x00\x00\x00\x00\x00\x00\x00\x01′
q b’\x00\x00\x00\x00\x00\x00\x00\x01′
Q b’\x00\x00\x00\x00\x00\x00\x00\x01′
n b’\x00\x00\x00\x00\x00\x00\x00\x01′
N b’\x00\x00\x00\x00\x00\x00\x00\x01′
P b’\x00\x00\x00\x00\x00\x00\x00\x01′

print(“h”,struct.pack(“h”,-1)) gave
h b’\xff\xff’

I also noticed a “funny”

z = b’AB23′
print(“4sll” , struct.pack(“4sll”,z,2,1))

gave me

4sll b’AB23\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x01′

The red text is from the long variable, the green text is the padding to make the long value on a long boundary (8).

I’ve run out of space on z/OS – what is using it all?

I hit this problem when trying to backup some data sets. There was plenty of free space on some of the volumes, but not much free space on the “user” volumes.

You can use ISPF 3;4 to list data sets on a volume or with a specified high level qualifier. You can use the sort command, for example sort tracks. This reports on number of tracks allocated sorted in descending size. You can deletes unwanted data sets.

If you go to ISPF 3;4 and specify option V for VTOC, and specify a Volume Serial, you can display information about the volume.

If you are running in Unix Services

du -a ./ | sort -n -r | head -n 30

Gives the top 30 files or directories

A short C quiz, and some gotcha’s

I’ve been looking at porting pymqi, the Python MQ interface to z/OS.

The biggest challenges where nothing to do with Pymqi.

So if you are bored after Christmas and want something stimulating… here are a few questions for you… The answers are below. I tried getting them displayed upside down, like all quality magazines; but that was too difficult.

Question 1. C question

I’ve reduced the problem I experienced,down to

int main() 
{ 
if ( 1==0 ) return 8; 
int rc; 
*=ERROR===========> CCN3275 Unexpected text 'int' encountered.
}                                                          

Hint: it works in a batch compile, using EDCCB

Question 2 binding in Unix Services

/bin/xlc a.o -L. -o b.so -Wl,INFO //’COLIN.MQ924.SCSQDEFS(CSQBRR2X)’ -Wl,dll c.x

Gave

FSUM3218 xlc: File //’COLIN.MQ924.SCSQDEFS(CSQBRR2X)’ contains an incorrect file suffix.

What do I need to do to fix it?

Question 3. Strange bind messages

Before I found the solution to problem number 2, I put the bind statements into a Unix Services file.

Using this gave me

IEW2326E 1221 THE FOLLOWING INVALID RECORD HAS BEEN SEEN:
=”lm-source” *

Copyright
IEW2326E 1221 THE FOLLOWING INVALID RECORD HAS BEEN SEEN:
IBM Corp. 2009, 2016 All Rights Reserved.

This bind statement was

cc -o mqsamp -W l,DYNAM=DLL,LP64 c.o mq.o

it worked without the mq.o

The mq.o file had

* <copyright                                                          * 
* notice="lm-source"                                                  * 
* (C) Copyright IBM Corp. 2009, 2016 All Rights Reserved.             * 
* </copyright>                                                        * 

Answer

  1. Using the cc compiler, it defaults to #pragma Langlvl(stdc89) which supports the c89 level of C. This says all variable declarations must come before any logic. This is relaxed in the c99 level, so specifying #pragma Langlvl(stdc99) cures it. You can also specify LANGLVL(EXTENDED) in the cc statement
  2. To include datasets in some of the binder options you need host file: filename with .OBJ suffix (object host file for the binder/IPA Link). When I used /bin/xlc a.o … -Wl,INFO //’COLIN.MQ924.SCSQDEFS.OBJ(CSQBRR2X)’ … it worked.
  3. The binder is not good at files in Unix Services, it likes records which are fixed block 80. The mq.o file had trailing blanks removed, and this confused it. I had to use a PDSE to get it to work.

Which came first, the chicken or the checksum.

The ability to sign Java jar files, or z/OS modules, has been around for many years. Using this the loader checks the digital signature in the object. This digital signature is a checksum of the object, and this checksum is encrypted and stored with the object. At load time, the loader calculates the checksum, and decrypts the checksum in the object and checks they match.

MQ now supports this for some of its objects; downloadable .zip, .tar and .gz files.

For some of these you need to download the public key to use. This raises the problem that an evil person may have taken the object, removed the official signing information, and added their own stuff. You then download their public certificate – see it works, it must be official.

To prevent this you can do the checksum on the public certificate, and make that available along with the official public key. (This is the chicken and egg problem. You need the certificate to be able to check the main code, and how to you check the certificate, without a certificate to check?)

On Linux you can calculate the checksum of a file using

sha256sum 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz

this gives 53c34cd374d7b08522423533ef2019b4aa0109a595fbaeab8ee6f927cb6c93ad, which is the same as the value on the IBM site. So this matches.

The IBM MQ code signatures page says IBM MQ public certificates, checksums, and .sig files are available from https://ibm.biz/mq92signatures. On this signatures page it says

release level: 9.2.4.0-IBM-MQ-Sigs-Certs 
Continuous Delivery: 9.2.4 IBM MQ file signatures, checksums and certificates

Platforms:  AIX 64-bit, pSeries, Linux 64-bit,x86_64, Linux 64-bit,zSeries, Linux PPC64LE, Windows 64-bit, x86, z/OS

This page is an httpS page, with the certificate issued by a proper Certificate Authority, and trusted third party. If you trust this CA, you can trust the IBM page.

When you click download, it downloads

  • 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz.sha256sum – this file content has 53c34cd374d7b08522423533ef2019b4aa0109a595fbaeab8ee6f927cb6c93ad
  • 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz

The value in the sha256sum file matches the value of the sha256sum 9.2.4.0-IBM-MQ-Sigs-Certs.tar.gz command.

As you can trust the security chain from the web page, through to the downloads, you can trust the .gz file.

Jar signing

Java has had the capability to sign a jar for at least 10 years.

The jarsigner command takes a jar file, a keystore with private key and calculates the checksum. It then encrypts it, and creates some files in the jar. For example

jarsigner -keystore trust.jks -storepass zpassword checkTLS.jar signer

This uses

  • the keystore called trust.jks,
  • with password zpassword
  • the checkTLS.jar file
  • and uses the certificate with alias name signer. This certificate must have extendedKeyUsage with codeSigning.

The jar file now has some additional files which can be seen using jar -tvf checkTLS.jar command.

  • META-INF/SIGNER.SF . This is the Signature File.
  • META-INF/SIGNER.EC .This is the public key to be used.

Where SIGNER is the name of the alias of the private key in the keystore, used to sign the jar file. The jar file can be signed many times by different private keys.

To verify the signature you can use

  • jarsigner -verify checkTLS.jar
  • jarsigner -verbose -certs -verify checkTLS.jar

The jarsigner -verbose -certs -verify checkTLS.jar gave me

- Signed by "CN=signer, O=cpwebuser, C=GB"
    Digest algorithm: SHA-256
    Signature algorithm: SHA256withECDSA, 256-bit key

jar verified.

Warning: 
This jar contains entries whose certificate chain is invalid. Reason: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
This jar contains signatures that do not include a timestamp. Without a timestamp, users may not be able to validate this jar after any of the signer certificates expire (as early as 2024-11-03).

The signer certificate will expire on 2024-11-03.

This shows that the jar file is consistent with the checksumming, but the certificate cannot be validated.

I can tell it which keystore to use to validate the certificate, using

jarsigner –keystore trust.jks -certs -verify checkTLS.jar

With the -verbose option you also get (with some of the output rearranged for clarity). The “s” or “sm” at the front of an object entry is s=signature verified, and m=entry listed in the manifest.

s = signature was verified 
m = entry is listed in manifest
k = at least one certificate was found in keystore
i = at least one certificate was found in identity scope


s 1402 Wed Dec 22 14:27:52 GMT 2021 META-INF/MANIFEST.MF

  >>> Signer
  X.509, CN=signer, O=cpwebuser, C=GB
  [certificate is valid from 22/12/21 14:51 to 30/01/25 16:46]
  X.509, CN=SSCA256, OU=CA, O=SSS, C=GB
  [certificate is valid from 04/11/21 15:48 to 03/11/24 15:48]

  [Invalid certificate chain: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

        1579 Wed Dec 22 16:12:04 GMT 2021 META-INF/SIGNER.SF
        1373 Wed Dec 22 16:12:04 GMT 2021 META-INF/SIGNER.EC

sm  54 Sat Jan 30 14:48:52 GMT 2021 checkTLS/checkTLS.manifest

  >>> Signer
  X.509, CN=signer, O=cpwebuser, C=GB
  [certificate is valid from 22/12/21 14:51 to 30/01/25 16:46]
  X.509, CN=SSCA256, OU=CA, O=SSS, C=GB
  [certificate is valid from 04/11/21 15:48 to 03/11/24 15:48]

  [Invalid certificate chain: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

When I downloaded the MQ 9.2.4 client and ran the jarsigner …. -verbose command the output included

sm 642 Thu Nov 04 16:01:46 GMT 2021 wlp/lib/extract/IFixUtils$ParsedIFix.class

[entry was signed on 04/11/21 17:58]
>>> Signer
X.509, CN=International Business Machines Corporation, OU=IBM CCSS, O=International Business Machines Corporation, L=Armonk, ST=New York, C=US
[certificate is valid from 25/08/21 01:00 to 26/08/23 00:59]
X.509, CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1, O="DigiCert, Inc.", C=US
[certificate is valid from 29/04/21 01:00 to 29/04/36 00:59]
      X.509, CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US
[trusted certificate]
... 

This shows that the certificate used to sign the component of the jar file was signed by CN=International Business Machines Corporation, which was in turn signed by CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1. The jarsigner program was able to use its public certificate to validate the CA, and so validate the IBM certficiate, and so validate the checksum.

Rexx to C to Rexx sample code

I’ve put up on github some sample code to demonstrate how you can write a function in C, and invoke it from Rexx. I’ve provided some glue code as Rexx uses R0 and R1 to pass parameters, and C programs only use R1.

I’ve create some small functions to use in your C program which hide the Rexx logic. For example

rc = CRexxDrop(pEnv,”ZDROP”);
rc = CRexxGet(pEnv,”InSymbol”,&buffer[0],&lBuffer);
rc = CRexxPut(pEnv,”CPPUTVar,”Colinsv”,0);
Iterate through all symbols

If you have any comments or suggestions, please let me know.