What do you mean, I can’t set the maximum queue depth?

Ive been involved with MQ for the whole of its life (I worked on the very first release), and I’ve just discovered something which is really basic!

I was testing out some code, and wanted to set the max depth of the queue to a low value for the duration of a test.

I thought this was easy; just use MQSET, specify parameter MQIA_MAX_Q_DEPTH, and the value I wanted (9).

When it reported

Reason 2067: FAILED: MQRC_SELECTOR_ERROR

I added in extra code to printout the values and yes, this was the value returned. If this was a bug I was sure it would have been spotted many years ago. I checked the documentation (remember, if all else fails, read the documentation). Look, I can set get inhibit, put inhibit etc ahh – max depth was not listed.

So when you get Reason 2067: FAILED: MQRC_SELECTOR_ERROR please check the documentation.

Debugging PIP install on z/OS

I had various problems installing Python packaged on z/OS.

Firstly the Python file system was read only – that’s ok. I can try to install in a virtual environment. This also had it’s problems!

The main problem was, I had the HOME environment variable pointing to a directory which my userid did not have write access to – /u/. When I changed it to my working directory /u/tmp/pymi2, it worked with no problems.

I’ll document in the blog the debugging I did.

Preparation

I FTP’d the wheels package in binary from my Linux machine, into /tmp on z/OS.

I switched to my virtual environment using

. env/bin/activate

This gave a command line like

(env) COLIN:/u/tmp/pymqi2:

I used the command

python3 -m pip install /tmp/wheel-0.37.1-py2.py3-none-any.whl /tmp/wheel-0.37.1-py2.py3-none-any.whl –no-cache-dir

But this gave

Installing collected packages: wheel
ERROR: Could not install packages due to an EnvironmentError: [Errno 111] EDC5111I Permission denied.: ‘/u/.local’

Using the –verbose argument

python3 -m pip –verbose install /tmp/wheel-0.37.1-py2.py3-none-any.whl /tmp/wheel-0.37.1-py2.py3-none-any.whl –no-cache-dir

gave more information, but not enough to resolve the problem.
Using

export DISTUTILS_DEBUG=1
python3 -m pip –verbose install..

gave much more information including

config vars:
{‘abiflags’: ”,
‘base’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘dist_fullname’: ‘UNKNOWN-0.0.0’,
‘dist_name’: ‘UNKNOWN’,
‘dist_version’: ‘0.0.0’,
‘exec_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘platbase’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘py_version’: ‘3.8.5’,
‘py_version_nodot’: ’38’,
‘py_version_short’: ‘3.8’,
‘sys_exec_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘sys_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘userbase’: ‘/u/.local’,
‘usersite’: ‘./lib/python3.8/site-packages’}

where I could see where the /u/.local came from.

You can change the userbase (see here) using

export PYTHONUSERBASE=.

and the product installed. When I fixed my HOME environment variable to point to my working directory this also worked!

What is installed?

Using the command

python3 -m pip –verbose list

This gave

Package      Version Location                                                   Installer          
------------ ------- ---------------------------------------------------------- ---------          
cffi         1.14.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
cryptography 2.8     /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
ebcdic       1.1.1   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
numpy        1.18.2  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pip          20.2.1  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pycparser    2.20    /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
setuptools   47.1.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
six          1.15.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
wheel        0.37.1  /u/tmp/pymqi2/lib/python3.8/site-packages                  pip                
zos-util     1.0.0   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    

so we can see the wheel package was installed in my user directory.

Without the –verbose just package and version were displayed, no location or installer.

Uninstall it

I used

python3 -m pip –verbose uninstall wheel

to uninstall it

Python virtual environments on z/OS

Virtual environments are very useful as they allow you to test install components without affecting the shared libraries.

You create an environment called env with

python3 -m venv env

but this gave

Error: Command ‘Ý’/u/py/env/bin/python3’, ‘-Im’, ‘ensurepip’, ‘–upgrade’, ‘–default-pip’¨’ returned non-zero exit status 1.

I think this was because my z/OS had no direct network connection. However

python3 -m venv –without-pip env –system-site-packages

worked. I needed the –system-site-packages, so I could build the extension.

Activate the environment

The instruction said use the command env/bin/activate but this failed

env/bin/activate: FSUM9209 cannot execute: reason code = ef076015: EDC5111I Permission denied.

but

. env/bin/activate

worked. (Just . env/b*/a* worked for me.)

If you get
 FSUM7332 syntax error: got Word, expecting )    

You need to issue the command ( or put it in your .profile)

export _BPXK_AUTOCVT=ON

See here. (_BPXK_AUTOCVT Used when enabling automatic conversion of tagged files).

To get out of the virtual environment use

deactivate

To delete the environment

remove the directory

rm -r env

Use the environment

You should now be able to use the environment.

You should check that the HOME environment variable is a directory with read/write access.

You may want to set up PYTHONHOME for special python packages.

The challenges of using PIP on z/OS.

Using Python PIP on z/OS, was not as smooth as I had expected. Some problems I worked around, some I just had to live with.

PIP is the standard package installer for Python. It is documented here.

To get information about PIP.

The following command gives lots of data about PIP.

python3 -m pip debug –verbose

Valid wheel types

“Wheels” are used to create and install Python packages; see here.

For example what are the names of valid “wheel types on my system” (needed when creating a package). Amongst the data this gave, was

Compatible tags: 30
py3-none-any
py37-none-any

This means PIP install will accept a package with

pymqi-1.12.0-py3-none-any.whl

A package like pymqi-1.12.0-py3-none-zos.whl is not in the list and will not install.

Configuration parameters

python3 -m pip debug –verbose

Configuration data is stored in several places

global:
  /etc/xdg/pip/pip.conf
  /etc/pip.conf
site:
  /usr/lpp/IBM/cyp/v3r8/pyz/pip.conf
user:
  $HOME/.pip/pip.conf
  $HOME/.config/pip/pip.conf

The documentation said $VIRTUAL_ENV/pip.conf will be used. $VIRTUAL_ENV was set on my system, but did not show up in the list.

Initially my $HOME was /u, and this caused problems as my userid was not authorised to write to /u/.pip…

Check your $HOME is set to an appropriate value.

Show the configuration files: python3 -m pip config debug

env_var:                                                                   
env
global:                                                      
  /etc/xdg/pip/pip.conf, exists: False                       
  /etc/pip.conf, exists: False                               
site:                                                        
  /usr/lpp/IBM/cyp/v3r8/pyz/pip.conf, exists: False          
user:                                                        
  /u/tmp/pymqi2/.pip/pip.conf, exists: False                 
  /u/tmp/pymqi2/.config/pip/pip.conf, exists: False          

python3 -m pip config list

gave me

[33]WARNING: The directory ‘/u/.cache/pip’ or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.-[0]

because HOME was not pointing to a value writeable directory.

Updating configuration:

Once I had set HOME to a valid value, I could set configuration values.

  • python3 -m pip config –user set user.colin yes
  • python3 -m pip config –user set site.colin yes

gave me

Writing to /u/tmp/pymqi2/.config/pip/pip.conf

This file had

[site]
colin = yes

[user]
colin = yes

python3 -m pip config –global set user.colin yes

gave me ( as expected)

Writing to /etc/pip.conf
[31]ERROR: Unable to save configuration. Please report this as a bug.
PermissionError: [Errno 111] EDC5111I Permission denied.: ‘/etc/pip.conf’

I would need use a suitably authorised userid to do this.

To edit a config file

You need to specify the editor to use

python3 -m pip config –user –editor /bin/oedit edit

Python on z/OS: Creating a pure Python package.

I had problems creating a package with both Python code, and a extension module written in c (shipped as a load module .so object in Unix Services). The problems were that the package name was dependant on the name of the hardware and the level of the operating system. To produce a package for z/OS in general, I would need to build on every z/OS hardware.

Removing the C code made it much easier to package.

The setup.py file for pure Python

import setuptools 
from setuptools import setup, Extension 
import sysconfig 
bindings_mode = 1 
version = '1.12.0' 
setup(name = 'pymqi', 
    version = version, 
    description = 'Python IBM MQI Extension for IBM MQ.', 
    long_description= 'PyMQI is a Python library ', 
    author='Dariusz Suchojad', 
    author_email='pymqi@m.zato.io', 
    url='https://dsuch.github.io/pymqi/', 
    download_url='https://pypi.python.org/pypi/pymqi', 
    platforms='OS/390', 
    package_dir = {'': 'code'}, 
    packages = ['pymqi'], 
    license='Python Software Foundation License', 
    keywords=('pymqi IBM MQ WebSphere WMQ MQSeries IBM middleware'), 
    python_requires='>=3', 
    classifiers = [ 
        'Development Status :: 5 - Production/Stable', 
        'License :: OSI Approved :: Python Software Foundation License', 
        'Intended Audience :: Developers', 
        'Natural Language :: English', 
        'Operating System :: OS Independent', 
        'Programming Language :: C', 
        'Programming Language :: Python', 
        'Topic :: Software Development :: Libraries :: Python Modules', 
        'Topic :: Software Development :: Object Brokering', 
        ], 
    py_modules = ['pymqi.CMQC', 'pymqi.CMQCFC', 'pymqi.CMQXC', 'pymqi.CMQZC'], 
    ) 

Where the py_modules were in ./code/pymqi/ as CMQC.py etc.

Install the wheel package and build it

python3 setup.py bdist_wheel

This gave a package

./dist/pymqi-1.12.0-py3-none-any.whl

Which can be installed on any z/OS system.

setup tools (and so bdist_wheel) has a web page here.

Uninstall it

I used

python3 -m pip –verbose uninstall wheel

to uninstall it

List what is installed

python3 -m pip –verbose list

This gave

Package      Version Location                                                   Installer          
------------ ------- ---------------------------------------------------------- ---------          
cffi         1.14.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
cryptography 2.8     /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
ebcdic       1.1.1   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
numpy        1.18.2  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pip          20.2.1  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pycparser    2.20    /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
setuptools   47.1.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
six          1.15.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
wheel        0.37.1  /u/tmp/pymqi2/lib/python3.8/site-packages                  pip                
zos-util     1.0.0   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    

so we can see the wheel package was installed in my user directory.

Without the –verbose just package and version were displayed, no location or installer.

Why can’t I use this C function – it is there but I cannot see it.

I’ve been writing in C for over 20 years, and it is humbling when you suddenly realise how little you know of a topic.
It reminds me of when I worked for IBM, and the company wanted a skills register. The questions were along the lines of

Rate your skills in the following areas from 0 (nothing) to 10 (expert).

  • z/OS
  • DB2
  • CICS
  • etc

Overall the skills register was found to be not useful, as the rankings were inverted. If someone put themselves down as 10 – it usually meant they knew very little, they knew enough for their day to day work. If someone put themselves down as 2 they may be an expert who realises how much they do not know, or someone who honestly realises they do not know very much.

My humbling discovery was that when I ported some existing C code to run on z/OS, the functions were not visible outside of the C program. There were two reasons for this.

  • The functions were defined as static,
  • The functions were not exported.

Static functions

static int hidden(int  i)
{
  return 0;
}
int visible(int i)
{
  int x = hidden(1);
  return 0; 
}

I think that using “static” in this case is the wrong word. It does not mean static. I think “internal” would be a clearer description, but I do not think that I’ll have any success changing the C language to use “internal”.

The function “hidden” can only be used within the compiled unit. It cannot be referenced from outside of the compiled object. The “visible” function can use the “hidden” function as the code shows.
The function”visible” is potentially visible to external programs. You can load the module and execute the function.

Exported functions

You have to tell the compiler to externalise functions within the compile unit.

For example

#pragma export(COLIN)
int COLIN(char * self, int args) {
   return 8;
}

or the compiler option EXPORTALL for example

cc… -Wc,EXPORTALL

What has been exported?

When you bind (linkedit) your program, the binder and report the exported functions. You need the binder parameters XREF and DYNAM=DLL

This gave output like

IMPORT/EXPORT     TYPE    SYMBOL              DLL                 DDNAME   SEQ  MEMBER 
-------------     ------  ----------------    ----------------    -------- ---  --------- 
   IMPORT         CODE64  __a2e_l             CELQV003            CELQS003  01  CELQS003 
   IMPORT         CODE64  malloc              CELQV003            CELQS003  01  CELQS003 
   IMPORT         CODE64  CSQB3BAK            CSQBLB16            MQ        01  CSQBMQ2X 
                                                                                                   
   EXPORT         DATA64  ascii_tab 
   EXPORT         CODE64  printHex 
   EXPORT         CODE64  COLIN 
   

This shows the imported symbols, and where they came from, and what was exported.

  • ascii_tab is a table of data in 64 bit mode
  • printHex is a function in 64 bit mode
  • COLIN is a function in 64 bit mode.

How to use it.

You can use handle= dlopen(name,mode) to get the load module into storage, and functionPointer=dlsym(handle,”COLIN”) to locate the external symbol COLIN in the load module.

Why is a static function useful?

With Python external functions (written in C), it uses static functions to hide internals. For example

static PyObject * pymqe_MQCONN(
... )
...
static struct PyMethodDef pymqe_methods[] = { 
  {"MQCONN", (PyCFunction)pymqe_MQCONN,... }, 
  {"MQCONNX", (PyCFunction)pymqe_MQCONNX,... },
  ....  
}

When the external function is imported, the initialisation routine returns the pymqe_methods data to Python.

Python now knows what functions the module provides (MQCONN, MQCONNX), and the C code to be executed when the function is executed.

This means that you cannot load the module, and accidentally try to use the function pymqe_MQCONN; which I thought was good defensive programming.

How do you start a pudding race? Sago. How do you start a hire car? Read 200 pages of documentation!

I recently had to hire a car for a few days. It was a 2021 car – very new, with all the latest gizmos – but I found it was a nightmare to use! Its “configuration and use” reminded me of software product documentation. I think people who develop products tend to forget who their audience is.

“Before you start”

The first challenge was to get to the drivers seat into the right position. I’ve been used to cars where you pull up a mechanical lever underneath the seat, move the seat back and release the lever, then adjust the height. This had the improved version it was all electronic. There were three knobs on the base of the seat. One knob seemed to move the seat back and down, one knob seemed to move the seat forward and up, and one knob to make the back of the seat softer or harder. I could either move the seat up to the wheel to be able to see over the bonnet of the car (I felt like a little squirrel with my hands close to my shoulders), or I could move the seat back and down, and only be able to see under the top of the steering wheel; I compromised.

When I got to the hotel that evening, I read the instructions, and found that if you move both the knobs for positioning the seat at the same time – it controlled the height. The second day I could position it and find a comfortable driving position.

Starting the car.

You did not need to put the key in the ignition, you just pressed the “Engine Start/Stop” button to turn the car on, then press the Engine Start/Stop button again to start the engine and drive off. Good theory, bad practice. I pressed the button to start the engine, and it said “Service the car now, press OK and reset”. I could not put the car into drive and drive off. Amongst all of the buttons and knobs I could not find one called “OK” nor “reset”. Turning the car off and on, sometimes seemed to clear it. We eventually found that if you waited 10 seconds, it reset itself. Being a hybrid car, it sometimes used the battery, and sometimes used the engine. It was strange at first driving off in silence, but that’s ok.

I was waiting in a queue of parked traffic on a hill. I had turned the car on, and started the engine, put the car in drive and pressed the throttle – only to find the car running backwards towards the car behind – an almost whooops. The car was not in run mode.

Eventually my wife said “the ‘Engine Start/Stop button needs to be in blue to be able to drive off. If it is yellow, or orange it is not in drive mode” This button was hidden behind the steering wheel, so I could not see it. If I moved closer to the steering wheel, I could see it.

My wife who used to be a programmer came up with a flow diagram along the lines of

  • Turn on car, ignore all of the visual effects.
  • If it says “service now” just wait till goes away.
  • Wait until the display said “run mode” and the speedometer is displaying 0 miles per hour, then you can drive off.
  • When I tried using it before the car was ready it beeped at me, and my friends came to offer (unhelpful) suggestions!

Parking the car

With most automatics the “Park” mode is good enough when you are on flat ground, but you should put the parking brake on when you are not on flat ground.

We looked in the index of the user’s guide which said “Parking brake, see page …..” This said to use the parking brake press the parking brake with your foot, as per illustration. My car did not match the illustration (I got out of the car to look in the foot well). I pressed what looked like the parking brake button – and nothing happened. That night I read the instruction book, and that to use the “Electronic Brake Control” you have to press the brake pedal, then the EBC button.

The car automatically released the EBC when you started driving.

We got to the hotel, took our suitcases out of the boot and I tried to lock the car. It displayed a message saying “key not in car” with a yellow cigar shaped icon, and the doors were not locked. Of course the key was not in the car, I was trying to lock the car and take the key away! It was like one of those messages which says “consult your system programmer” and you shout at it ” I AM THE SYSTEMS PROGRAMMER”.

I got in the car, turned it on, turned it off, got out – and tried again. Still “no key in car”. My wife suggested getting in the car, turning it on, waiting for a few seconds then stopping it and getting out. This time it locked the car.

Is this a safe car?

I found that the abundance of safety features made the car less safe. For example when I was changing lane, there was a brief display on the multi function display. I think it was telling me I was about to move out of my lane. This distracted me from driving, as I had to look down and see what was displayed. By the time I had spotted what was displayed, it had disappeared.

When two motorways merged, I had to merged right. As I pulled out, I caught a flash out of the corner of my eye. Was this from a car I had not seen? No – it was the wing mirror displaying a car symbol to say there was a car close behind me.

The car frequently beeped, which was distracting as you had to look at the dash board to see if this was important or not.

If this was a car that you used every day it may have been a great car, but for someone using it as a hire car for two days – this was a terrible car to use.

The instructions

The instruction book has over 500 pages. It covered many models, so much of it did not apply to my car. The first chapter as about the benefits of using hybrid, and so on. The second chapter was about the 14 ways you can install a child seat in the car. The third chapter was “convenient features of your vehicle”. The fourth chapter was “Infotainment”, and finally the fifth chapter covered the controls, one of which was the “Engine Start/Stop” button”. These instructions came after “How to change the battery in the key fob”, “Theft-alarm system” and “Driver position memory system” which are not the top priority topics that people need to know.

The most important button was buried among all of the controls.

Some of the pictures on the controls were baffling, so once I got home, I looked at the instruction manual online. I failed to find what one of the pictures was. If I had read the book cover to cover then I may have found it. They must be better ways for all manufactures to write the instruction books!

This feels like one of the products I have been using on z/OS. It covers configuration, tailoring etc, and on page about 240 it say “this is how you start it”. I would consider much of the tailoring as optional, advanced tailoring.

You are meant to know how to logon. I was told “It is intuitive”. Once you know the magic incantation it was easy to use, but the documentation does not tell you the magic incantation to get started.

With the car I would provide a one sheet of A4 paper with pictures and text, covering

  • Adjusting the seat
  • Adjusting the wing mirrors and interior mirror
  • How to start the car along the lines of
    • Press the “Engine Start/Stop button”
    • If it says “service car now” wait till the message disappears
    • Wait until the display shows “0 MPH”
    • Press the brake pedal
    • Move the gear level to Drive or Reverse
    • Use the throttle to move off.
  • How to stop the car
    • Park it
    • If you want to put on the parking brake, use the foot brake, then push down the switch with the “P” on it beside the gear lever. A red ! in a triangle sign appears
    • Press the “Engine Start/Stop button” and the car should shutdown
    • Get out of the car and press the “lock” button on the remote. The wing mirrors should fold in, and the display goes black.

Writing installation instructions

If you get this far, and are responsible for writing documentation on how to install and customise software here are some thoughts on how to write the documentation

  • Know your audience. Do you expect them to be experts?
  • What is the minimum information they need to know to get started? Move any other information to the back of the document (is knowing how to change the battery in the key fob more important than knowing how to start the car?)
  • If the instructions take more than one page – are the instructions too complex, can you simplify the product?
  • How will people use the information?

Python on z/OS using a load module (and .so from Unix Services)

As part of playing with Python on z/OS I found you can call a z/OS Unix Services load module ( a .so object) from Python. It can also use a load module in a PDSE.

What sort is it?

A load module on z/OS can be used on one of two ways.

  • Load it from steplib or PATH environment variable (which uses dllopen under the covers), call the function, and pass the parameters The parameters might be a byte string ( char * in C terms), a Unicode string, integer etc. You return one value, for example an integer return code.
  • As part of a package where you use the Python “import package” statement. This gets loaded from PYTHONPATH, the current directory, and other directories (but not steplib). The parameters passed in are Python Objects, and you have to use Python functions to extract the value. You can return a complex Python object, for example a character string, return code and reason code.

This article is on the first case.

In both cases, the values passed in are in ASCII. If you use printf to display data, the printf treats your data as ASCII.

There is a good article on Python ctypes , a foreign function library for Python.

My initial program was

int add_it(int i, int j)
{
   return i+j;
}

I compiled it with a bash script

wc64=”-Wc,SO,LIST(lst64),XREF,LP64,DLL,SSCOM,EXPORT”
cc -c -o add.o ${wc64} add.c
cc -o add -V -Wl,DYNAM=DLL,LP64 add.o 1>ax 2>bx

This created a load module object “add”. Note: You need the EXPORT to make the entry point(s) visible to callers.

My python program was

import ctypes
from ctypes.util import find_library
zzmqe = ctypes.CDLL(“add”)
print(“mql”, zzmqe.add_it(2,5))

When this ran it produced

mql 7

As expected. To be consistent with Unix platforms, the load module should be called add.so, but “add” works.

Using strings is more complex

I changed the program (add2) to have strings as input

int add_one(char * a, char *b)
{
  printf("a %s\n",a);  
  printf("b %s\n",b);
  return 2 ;
}

and a Python program, passing a byte string.

import ctypes
from ctypes.util import find_library
zzmqe = ctypes.CDLL("add2")
print("mql", zzmqe.add_one(b'abc',b'aaa'))

This ran and gave output

-@abc–@aaa-mql 2

This shows that Python has converted the printf output ASCII to EBCDIC, and so the “a” and “b” in the printf statements are converted to strange characters, and the \n (new line) is treated as hex rather than a new line.

When I compiled the program with ASCII (-Wc…ASCII), the output from the Python program was

a abc
b aaa
mql 2

Displaying the data as expected.

Using a load module in a PDSE.

The JCL

//COLINC3 JOB 1,MSGCLASS=H,COND=(4,LE)
//S1 JCLLIB ORDER=CBC.SCCNPRC
// SET LOADLIB=COLIN.C.REXX.LOAD
// SET LIBPRFX=CEE
//COMPILE EXEC PROC=EDCCB,
// LIBPRFX=&LIBPRFX,
// CPARM=’OPTFILE(DD:SYSOPTF),LSEARCH(/usr/include/),RENT’,
// BPARM=’SIZE=(900K,124K),RENT,LIST,XREF,RMODE=ANY,AMODE=64
//COMPILE.SYSOPTF DD DISP=SHR,DSN=COLIN.C.REXX(CPARMS)
// DD *
EXPORT,LP64
/*
//COMPILE.SYSIN DD *
int add_one(int i, int j)
{
return i+j;
}

//COMPILE.SYSLIB DD
// DD
// DD DISP=SHR,DSN=COLIN.C.REXX
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB.
//BIND.SYSLIB DD DISP=SHR,DSN=CEE.SCEEBND2
// DD DISP=SHR,DSN=CEE.SCEELKED
// DD DISP=SHR,DSN=CEE.SCEELIB
//BIND.OBJLIB DD DISP=SHR,DSN=COLIN.C.REXX.OBJ
//BIND.SYSIN DD *
NAME ADD3(R)
/*

In Unix services

export STEPLIB=”COLIN.C.REXX.LOAD“:$STEPLIB

The python program

import ctypes
from ctypes.util import find_library
zzmqe = ctypes.CDLL(“ADD3”)
print(zzmqe)
print(dir(zzmqe))
print(“steplib”, zzmqe.add_one(2,5))

This gave

steplib 7

Byte string and character strings parameters

I set up a C program with a function COLIN

#pragma export(COLIN)
int COLIN(char * p1, int p2) { 
  printf(" p1 %4.4s\n",p1);
  printf(" p2 %i\n",p2);
  return 8;
}

This was compiled in Unix Services, and bound using JCL into a PDSE load library as member YYYYY.

I used a shell to invoke the Python script

export LIBPATH=/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/:$LIBPATH
export STEPLIB=COLIN.C.REXX.LOAD:$STEPLIB 
python3 dll.py

where the Python dll.py script was

import ctypes
testlib = ctypes.CDLL("YYYYY")
name =b'CSQ9'
i = 7
zz = testlib.COLIN(name,i)
print("return code",zz)

This displayed

p1 CSQ9
p2 7
return code 8

The name, a binary string CSQ9, was passed as a null terminated ASCII string (0x43535139 00).

When a string name = “CSQ9” was passed in, the data was in a Unicode string, hex values

00000043 00000053 00000051 00000039 00000000

You need to be sure to pass in the correct data (binary or Unicode), and be sure to handle the data in ASCII.

Is this how ‘import’ works?

This is different process to when a module is used via a Python import statement. If you passed in b’ABCD’ to a C extension which has been “imported” this would be passed as a Python Object, rather than the null terminated string 0x’4142434400′.

Python on z/OS advanced C extension

I found that to create a standard Python package with a C extension, the package has a very specific name – depending on the level of Python, the level of z/OS, the hardware the z/OS is running on. To be able to build a package for all levels of z/OS this would be a near impossible job; because I do not have access to every combination of z/OS hardware and software.

I’ve found a way to get round it. It took a few days to get it right, but it is pretty simple.

The standard way of packaging.

With the normal way of packaging the C executable module is stored deep in the Python tree, for example

/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/python3.10/site-packages/pymqi-1.12.0-py3.10-os390-27.00-1090.egg/pymqi/pymqe.cpython-310.so

The easy way of using it

You can copy this file, for example to my working directory. I copied it as pymqe.so.

My Python program was

import pymqe
print(dir(pymqe))
name = “CSQ9”
rv = pymqe.MQCONN(name)
print(“rv”,rv)

This locates pymqe*.so in the current directory. It located pymqe.so; if I renamed it to pymqe.cpython-310.so it also worked.

You can put the .so object in the PYTHONPATH environment variable, and have it picked up from there.

When the file is imported, an initialisation routine is invoked which defines all of the Python entry points. You can see them using the dir(pymqe) statement. This gave me

[‘MQBACK’, ‘MQCLOSE’, ‘MQCMIT’, ‘MQCONN’, ‘MQCONNX’, ‘MQCRTMH’, ‘MQDISC’, ‘MQGET’, ‘MQINQ’, ‘MQINQMP’, ‘MQOPEN’, ‘MQPUT’, ‘MQPUT1′,’MQSET’, ‘MQSETMP’, ‘MQSUB’, ‘doc’, ‘file’, ‘loader’, ‘mqbuild’, ‘mqlevels’, ‘name’, ‘package’, ‘spec_ _’, ‘__version’, ‘pymqe.error’]

When the module is loaded Python looks for the entry name PyInit_… where … is the name of the module. For pymqe.so it looks for PyInit_pymqe. If you rename the module to mq.so and import mq, you get

ImportError: dynamic module does not define module export function (PyInit_mq)

The rv=pymqe.MQCONN invokes the MQ function which returns a handle, a return code and a reason code. For me it printed

rv (549309464, 0, 0)

So .. overall an easy solution.

I could find no way of using a load module from a PDSE, so it looks like the PYTHONPATH, or current directory is best for this.

Python on z/OS – creating a C extension

I enjoy using Python on Linux, because it is very powerful. I thought it would be interesting to port the MQ Python interface pymqi to z/OS. This exposed many of the challenges of running Python on z/OS.

I’ll cover some of the lessons I learned in doing this work. Thanks to Steve Pitman who helped me package the extension.

IBM Open Enterprise Python for z/OS, V3.8, user’s guide is a useful book.

Creating files that would compile was a challenge.

See here.

Compiling files.

I copied the pymqi C code to z/OS Unix Services, and tried to compile it. This was a mistake, as it took me a long time to get the compile options right. I found that using the setup.py script was the right way to go.

My directory tree

/u/pymqi
..setup.py
..include
....sample.h
..code
....pymqi
......__init__.py        
......CMQCFC.py          
......CMQXC.py           
......CMQZC.py           
......const.py           
......CMQC.py            
......pymqe.c            

Setup.py

This script needs export _C89_CCMODE=1, otherwise you get FSUM3008 message

Specify a file with the correct suffix (.c, .i, .s, .o, .x, .p, .I, or .a), or a corresponding data set name, instead of -L

import setuptools 
from distutils.core import setup, Extension 
import os 
import sysconfig 
# 
# This script needs    export _C89_CCMODE=1 
# Otherwise you get FSUM3008  messages 
# 
import os 
os.environ['_C89_CCMODE'] = '1' 
bindings_mode = 1 
version = '1.12.0' 
setup(name = 'pymqi', 
    version = version, 
    description = 'Python...', 
    platforms='OS Independent', 
    package_dir = {'': 'code'}, 
    packages = ['pymqi'],  
    py_modules = ['pymqi.CMQC', 'pymqi.CMQCFC', 'pymqi.CMQXC', 'pymqi.CMQZC'], 
    ext_modules = [Extension('pymqi.pymqe',['code/pymqi/pymqe.c'], define_macros=[('PYQMI_BINDINGS_MODE_BUILD', 
bindings_mode)], 
    include_dirs=["//'COLIN.MQ924.SCSQC370'"], 
    extra_link_args=["//'COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ2X)'"], 
      )] 
) 
# I had extra_link_args=["-Wl,INFO,LIST,MAP",.... when setting 
# this up
# I used 
# extra_compile_args=["-Wc,LIST(c.lst),XREF"], 
# to get out a listing and cross reference.

Which says

  • The package name is packages = [‘pymqi’],
  • The Python files are py_modules = [‘pymqi.CMQC’….
  • There is an extension .. ext_modules=.. with the source program code/pymqi/pymqe.c
  • It needs “//’COLIN.MQ924.SCSQC370′” to compile and “//’COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ2X)'” at bind time. This file contains the MQ Binder input
  • When I wanted the binder output – “-Wl,INFO,LIST,MAP”. This goes to the terminal. I used a ‘>’ command to pipe the output of the python3 setup build it to a file.
  • and a C listing “-Wc,LIST(c.lst),XREF”. The listing goes to c.lst

You need

  • import setuptools so that the setup bdist_wheel packaging works. You also need the wheel package installed.

Setup

There is a buglet in the compile set up. You need to specify

export _C89_CCMODE=1

Without it you get

FSUM3008 Specify a file with the correct suffix (.c, .i, .s, .o, .x, .p, .I, or .a), or a corresponding data set name, instead of -obuild/lib.os390-27.00-1090-3.8/pymqi/pymqe.so.

You also need the binder input in a data set with the correct suffix. For example .OBJ

“//’COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ2X)'”

If you do not have the correct suffix you get

FSUM3218 xlc: File //’COLIN.MQ924.SCSQDEFS(CSQBMQ2X)’ contains an incorrect file suffix.

Doing the compile and test install

I used a shell script to do the compiles and install

touch code/pymqi/*.c
rm a b c d
export _C89_CCMODE=1
#python3 setup.py clean
python3 setup.py build 1>a 2>b
python3 setup.py install 1>c 2>d

I captured the output from the setup.py jobs using 1>a etc because I could not see how to direct the binder output to a file. It comes out on the terminal – and there was a lot of it!.

Packaging the package

Python build which worked

I had to install wheel package. See How to install software in an isolated environment, or just use python3 -m pip install wheel if your z/OS image is connected to the network.

The command I used was

python3 -m pip install –user –no-cache-dir /u/tmp/py/wheel-0.37.1-py2.py3-none-any.whl-f /u/tmp/py/wheel-0.37.1-py2.py3-none

I had to add import setuptools to my setup.py file (at the top). (This converted the install package from a dist-utils to a setuptools packaging)

python3 setup.py bdist_wheel

This created a file “/u/pymqi/dist/pymqi-1.12.0-cp310-cp310-os390_27_00_1090.whl

This file is specific to python 3.10

For a wheel package, you’ll need to build it for all major versions and cannot just use one. Note that there were some issues in 3.8/3.9 with wheels that have been resolved in 3.10, so it’s recommended you to use 3.10.

Steven Pitman

This means you need to have multiple levels of Python installed, and build for each one!

This also has the operating system level (os390_27_00 – this may be constant across machines) and the hardware 1090. For this to work on other hardware, one solution would be to manually rename the file to pretend it is for a different machine, but this both not supported nor recommended, and has no guarantee to work. So it is hard to know the best thing to do. I do not have every 390 machine from IBM to do a build on !

Failing build. This built but did not install.

python3 setup.py bdist –format=tar

It built the package and create a file

./dist/pymqi-1.12.0.os390-27.00-1090.tar

This tar file is not completely readable by the z/OS tar command.

When I used tar -tf ….tar it gave

FSUMF371 Value 1641318408.0 is not valid for keyword mtime. Keyword not set.

It uses a Python tar command, not the operating system tar command.

You can display the contents using a Python program like

import tarfile
tar = tarfile.open("dist/pymqi-1.12.0.os390-27.00-1090.tar.gz")
# tar.extractall() 
for x in tar:
    print(x)

This gave output like

<TarInfo ‘.’ at 0x5008ad3880>
<TarInfo ‘./usr’ at 0x5008ad3dc0>
<TarInfo ‘./usr/lpp’ at 0x5008ad3a00>

Installing the package

From an authorised user in OMVS,

python3 -m pip install –no-cache-dir /u/pymqi/dist/pymqi-1.12.0-cp310-cp310-os390_27_00_1090.whl /u/pymqi/dist/pymqi-1.12.0-cp310-cp310-os390_27_00_1090.whl
Processing /u/pymqi/dist/pymqi-1.12.0-cp310-cp310-os390_27_00_1090.whl
Installing collected packages: pymqi
Successfully installed pymqi-1.12.0

If you do not use –no-cache-dir, you may get

-[33]WARNING: The directory ‘/u/.cache/pip’ or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo’s -H flag.-[0]

The compile options

The following text is the compile and bind options used for my code. Some of the options are pymqi specific.

/bin/xlc -DNDEBUG -O3 -qarch=10 -qlanglvl=extc99 -q64
-Wc,DLL
-D_XOPEN_SOURCE_EXTENDED
-D_UNIX03_THREADS
-D_POSIX_THREADS
-D_OPEN_SYS_FILE_EXT
-qexportall -qascii -qstrict -qnocsect
-Wa,asa,goff -Wa,xplink
-qgonumber -qenum=int
-DPYQMI_BINDINGS_MODE_BUILD=1 -I//’COLIN.MQ924.SCSQC370′
-I/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/include/python3.10
-c code/pymqi/cpmqe.c
-o build/temp.os390-27.00-1090-3.10/code/pymqi/cpmqe.o

/bin/xlc build/temp.os390-27.00-1090-3.10/code/pymqi/cpmqe.o -L.
-o build/lib.os390-27.00-1090-3.10/pymqi/cpmqe.cpython-310.so -Wl,INFO,LIST,MAP,DLL //’COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ2X)’
-Wl,dll
/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/python3.10/config-3.10/libpython3.10.x
-q64