How do I use SFTP to ftp to a z/OS data set?

You cannot do it directly, you have to do a two step process.

Sending files to z/OS using SFTP

SFTP can send files to Unix Services Subsystem. It cannot send to data sets.

sftp colin@10.99.88.77

You can use commands like

  • cd to change directory on the remote system
  • lcd change directory on the local system
  • put
  • get
  • chmod
  • chown
  • exit

There is no command “bin” nor “quote…”.

Getting from a Unix Services file to a dataset.

You can use the cp command.

To copy a binary file to a dataset

For example as if you were using FTP with BIN; quote site cyl pri=1 sec=1 recfm=fb blksize=3200 lrecl=80; put mp1b.load.xmit ‘COLIN.MP1B.LOAD.XMIT’)

cp -W “seqparms=’RECFM=FB,SPACE=(500,100),LRECL=80,BLKSIZE=3200′” mp1b.load.xmit “//’COLIN.MP1B.LOAD.XMIT'”

Where the seqparms are in upper case. If they are in mixed case you get

FSUM6258 cannot open file “… “: EDC5121I Invalid argument.

I could then do TSO RECEIVE INDSN(‘COLIN.MP1B.LOAD.XMIT’).

To copy a text file to a data set

If you use SFTP to copy a text file to Unix Services, it gets sent in bin, and, on z/OS, looks like

ñà ÈÇÁ ËøÁÄÑÃÑÁÀ….

You can tag a file so Unix Services knows it is an ASCII file, using

chtag -tc ISO8859-1 aaa

This makes the file editable from Unix Services, but you cannot just use cp to copy and create a dataset, as above.

You can convert it from ASCII to EBCDIC using

iconv -f ISO8859-1 -t IBM-037 ascii_file  > ebcdic-file

Then use

cp -W “seqparms=’RECFM=VB,SPACE=(CYL,(1,1)),LRECL=800,BLKSIZE=8000′” ebcdic-file “//’COLIN.EBCDFILE'”

Using CISCO openconnect to tunnel to another system from Linux

I needed to use openconnect from CISCO to be able to logon from my Ubuntu system to someone else’s z/OS system.

This was pretty easy, but understanding some of the under the cover’s bits took a bit of time.

Basic install

  • Use sudo apt install openconnect
  • Download the VPNC script from http://www.infradead.org/ 
  • Create the configuration script
    • I saved the script as vpnc-script.sh
    • Using ls /etc/vpnc showed the directory did not exist. Create it and move the file
    • sudo mkdir /etc/vpnc/
    • sudo mv vpnc-script.sh /etc/vpnc/
    • sudo chmod +x /etc/vpnc/vpnc-script.sh
  • You need information from the owners of the vpn server.
    • vpn userid
    • vpn password
    • name of their system
    • IP address of their internal system
    • tso userid
    • tso password
  • I created a script (openc.sh), where XXXXX is the short userid, and password is your long userid:
    • printf ‘%s’ “password” | sudo openconnect –user=XXXXXX –script=/etc/vpnc/vpnc-script.sh vpn.customer.com
  • When you run openc.sh it prompts for your su password on the machine. The print… means you can store the password in the shell script. If you do not specify it, openconnect will prompt you for it.
  • Once the connection is made you can use ping 10.66.77.88, or x3270 -model 5 10.66.77.88 to access the system, where 10.66.77.88 is the IP address the owner of the vpn server gave you.

x3270

The owner of the vpn server gave me the address of the z/OS machine, my userid and password.

I then used

x3270 -model 5 10.66.77.88 to logon to the system.

Hot key

I like to hot key to my z/OS sessions. I used Ubuntu “Settings”-> Keyboard shortcuts, and added a shortcut

  • name: mvsCust
  • Command: wmctrl -a 10.66.77.88
  • Hot key: Ctrl + H

The wmctl -a says make the window active which has 10.66.77.88 in the window page title.

When I press Ctrl +H it makes the customers x3270 session the active window.

Change the x3270 colours

I wanted to change the screen colours, to distinguish it from other 3270 sessions. See Making x3270 green screens blue or red, or yellow with green bits.

FTP

I had to use SFTP colin@10.66.77.88 to ftp to the remote z/OS system (where colin is my TSO userid).

What happens with openconnect, under the covers.

The handshake has several stages

  • Establish a TLS session using the certificate from the server. Once this has completed, any traffic is encrypted. In my case I used the vpn userid and password. The vpn server can be configured to accept certificates instead of userid and password.
  • The server sends down configuration information from the vpn server’s configuration. For example
    • The IP addresses it supports , such as 10.66.0.0 and netmask 255.255.0.0
    • Any changes to the DNS configuration, so it knows to route 10.66.77.78 via the VPN session.
    • The “banner” such as “Welcome to mycom.com. Users of this system do so at their own risk”.
    • A default domain.
    • Which tunnelling device to use – such as tun0.
    • How many configuration statements.
    • Each set of configuration statements.
    • You can see this information by using the -v option on the openconnect command.
  • Using the information sent from the the vpn server, the openconnect client creates environment variables.
  • The script defined (or defaulted, for example /etc/vpnc/vpnc-script.sh) on the openconnect command is invoked, and it uses these environment variables to manage the ip and dns configuration, changing files like /etc/resolv.conf (the local DNS file).

Oh p*x, I’ve lost my changes

I have been using pax to backup the files in my Unix Services directory and needed to restore a file so I could compare it with the last version (and work out why my updates didn’t work). Unfortunately I managed to overwrite my latest version instead of creating a copy.
I backed up my directory using

pax -W “seqparms=’space=(cyl,(10,10))'” -wzvf “//’COLIN.PAX.PYMQI2′” -x os390 /u/tmp/pymqi2/

This created a data set COLIN.PAX.PYMQI2 with the give space parameters, and os390 format.
(If you do not want all the files listed, use –wzf)

To list the contents of this file use

pax -f “//’COLIN.PAX.PYMQI2′”

To display a subset of the files use

pax -f “//’COLIN.PAX.PYMQI2′” /u/tmp/pymqi2/code

which gave

/u/tmp/pymqi2/code/
/u/tmp/pymqi2/code/pymqi/
/u/tmp/pymqi2/code/pymqi/__init__.py
/u/tmp/pymqi2/code/pymqi/old__init__.old
/u/tmp/pymqi2/code/pymqi/aa

And provide more information using the -v option

drwxrwxrwx 1 COLIN    1000      0 Jan 22 17:04 /u/tmp/pymqi2/code/
drwxr-xr-x 1 COLIN    1000      0 Feb 11 13:10 /u/tmp/pymqi2/code/pymqi/
-rw-r--r-- 1 OMVSKERN 1000 133011 Feb 22 13:15 /u/tmp/pymqi2/code/pymqi/init.py
-rw-r----- 1 COLIN    1000 119592 Feb  3 12:59 /u/tmp/pymqi2/code/pymqi/old__init__.old
-rwx------ 1 OMVSKERN 1000 119565 Jan 22 16:43 /u/tmp/pymqi2/code/pymqi/aa

The whoops

To restore an individual file and overwrite the original I used the -r option.

pax -rf “//’COLIN.PAX.PYMQI2′” /u/tmp/pymqi2/pymqi/__init__.py

I was expecting the file to be restored relative to the directory I was in; No – because I had backed up the files using an absolute path it restored the file to the same place, and so it overwrote my changes to the file. I had changed to a temporary directory, but I had not realised how the command worked.

There are several ways of doing it properly.

Restore with rename

pax -rf “//’COLIN.PAX.PYMQI2′” -i /u/tmp/pymqi2/pymqi/__init__.py

The -i option means rename.

I ran the command and it prompted me to rename it

Rename “/u/tmp/pymqi2/pymqi/__init__.py” as…

/tmp/oldinit.py

Set “do not overwrite”

I could also have used the -k option which prevents the overwriting of existing files.

Rename on restore

I could also have used the rename

pax -rf “//’COLIN.PAX.PYMQI2′” -s#/u/tmp/pymqi2/pymqi#/tmp/# /u/tmp/pymqi2/pymqi/__init__.py

Where the -s#/u/tmp/pymqi2/pymqi#/tmp/# / says use the regular expression to change /u/tmp/pymqi2/pymqi to /tmp and so restore it to a different place. Note: The more obvious -s/abc/xyz/, where / is used as the delimiter, would not work, as there is a ‘/’ in the file path.

All of the above

I could have use all of the options -i -k -s…. .

A better way to backup.

I had specified an absolute directory /u/tmp/pymi2/. If I was in this directory when I did the backup I could have used

pax … -x os390 .

Where the . at the end means from this directory, and so backup a relative directory.

If I list the files I get

pax -f “//’COLIN.PAX.PYMQI2A'” ./aa
./aa

And now if I restore the file…

pax -rf “//’COLIN.PAX.PYMQI2A'” ./aa

It restored the file into my working directory /tmp/aa .

So out of all the good ways of backing up and restoring – I chose the worst one. It only took me about 2 hours to remake all the changes I had lost.

What do you mean, I can’t set the maximum queue depth?

Ive been involved with MQ for the whole of its life (I worked on the very first release), and I’ve just discovered something which is really basic!

I was testing out some code, and wanted to set the max depth of the queue to a low value for the duration of a test.

I thought this was easy; just use MQSET, specify parameter MQIA_MAX_Q_DEPTH, and the value I wanted (9).

When it reported

Reason 2067: FAILED: MQRC_SELECTOR_ERROR

I added in extra code to printout the values and yes, this was the value returned. If this was a bug I was sure it would have been spotted many years ago. I checked the documentation (remember, if all else fails, read the documentation). Look, I can set get inhibit, put inhibit etc ahh – max depth was not listed.

So when you get Reason 2067: FAILED: MQRC_SELECTOR_ERROR please check the documentation.

Debugging PIP install on z/OS

I had various problems installing Python packaged on z/OS.

Firstly the Python file system was read only – that’s ok. I can try to install in a virtual environment. This also had it’s problems!

The main problem was, I had the HOME environment variable pointing to a directory which my userid did not have write access to – /u/. When I changed it to my working directory /u/tmp/pymi2, it worked with no problems.

I’ll document in the blog the debugging I did.

Preparation

I FTP’d the wheels package in binary from my Linux machine, into /tmp on z/OS.

I switched to my virtual environment using

. env/bin/activate

This gave a command line like

(env) COLIN:/u/tmp/pymqi2:

I used the command

python3 -m pip install /tmp/wheel-0.37.1-py2.py3-none-any.whl /tmp/wheel-0.37.1-py2.py3-none-any.whl –no-cache-dir

But this gave

Installing collected packages: wheel
ERROR: Could not install packages due to an EnvironmentError: [Errno 111] EDC5111I Permission denied.: ‘/u/.local’

Using the –verbose argument

python3 -m pip –verbose install /tmp/wheel-0.37.1-py2.py3-none-any.whl /tmp/wheel-0.37.1-py2.py3-none-any.whl &x45;&x45;no-cache-dir

gave more information, but not enough to resolve the problem.
Using

export DISTUTILS_DEBUG=1
python3 -m pip &x45;&x45;verbose install..

gave much more information including

config vars:
{‘abiflags’: ”,
‘base’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘dist_fullname’: ‘UNKNOWN-0.0.0’,
‘dist_name’: ‘UNKNOWN’,
‘dist_version’: ‘0.0.0’,
‘exec_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘platbase’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘py_version’: ‘3.8.5’,
‘py_version_nodot’: ’38’,
‘py_version_short’: ‘3.8’,
‘sys_exec_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘sys_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘userbase’: ‘/u/.local’,
‘usersite’: ‘./lib/python3.8/site-packages’}

where I could see where the /u/.local came from.

You can change the userbase (see here) using

export PYTHONUSERBASE=.

and the product installed. When I fixed my HOME environment variable to point to my working directory this also worked!

What is installed?

Using the command:

python3 -m pip --verbose list

This gave

Package      Version Location                                                   Installer          
------------ ------- ---------------------------------------------------------- ---------          
cffi         1.14.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
cryptography 2.8     /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
ebcdic       1.1.1   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
numpy        1.18.2  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pip          20.2.1  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pycparser    2.20    /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
setuptools   47.1.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
six          1.15.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
wheel        0.37.1  /u/tmp/pymqi2/lib/python3.8/site-packages                  pip                
zos-util     1.0.0   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    

so we can see the wheel package was installed in my user directory.

Without the --verbose, just package and version were displayed, no location or installer.

Uninstall it

I used

python3 -m pip –verbose uninstall wheel

to uninstall it

Python virtual environments on z/OS

Virtual environments are very useful as they allow you to test install components without affecting the shared libraries.

You create an environment called env with

python3 -m venv env

but this gave

Error: Command ‘Ý’/u/py/env/bin/python3’, ‘-Im’, ‘ensurepip’, ‘–upgrade’, ‘–default-pip’¨’ returned non-zero exit status 1.

I think this was because my z/OS had no direct network connection. However

python3 -m venv –without-pip env –system-site-packages

worked. I needed the –system-site-packages, so I could build the extension.

Activate the environment

The instruction said use the command env/bin/activate but this failed

env/bin/activate: FSUM9209 cannot execute: reason code = ef076015: EDC5111I Permission denied.

but

. env/bin/activate

worked. (Just . env/b*/a* worked for me.)

If you get
 FSUM7332 syntax error: got Word, expecting )    

You need to issue the command ( or put it in your .profile)

export _BPXK_AUTOCVT=ON

See here. (_BPXK_AUTOCVT Used when enabling automatic conversion of tagged files).

To get out of the virtual environment use

deactivate

To delete the environment

remove the directory

rm -r env

Use the environment

You should now be able to use the environment.

You should check that the HOME environment variable is a directory with read/write access.

You may want to set up PYTHONHOME for special python packages.

The challenges of using PIP on z/OS.

Using Python PIP on z/OS, was not as smooth as I had expected. Some problems I worked around, some I just had to live with.

PIP is the standard package installer for Python. It is documented here.

To get information about PIP.

The following command gives lots of data about PIP.

python3 -m pip debug --verbose

Valid wheel types

“Wheels” are used to create and install Python packages; see here.

For example what are the names of valid “wheel types on my system” (needed when creating a package). Amongst the data this gave, was

Compatible tags: 30
py3-none-any
py37-none-any

This means PIP install will accept a package with

pymqi-1.12.0-py3-none-any.whl

A package like pymqi-1.12.0-py3-none-zos.whl is not in the list and will not install.

Configuration parameters

python3 -m pip debug --verbose

Configuration data is stored in several places

global:
  /etc/xdg/pip/pip.conf
  /etc/pip.conf
site:
  /usr/lpp/IBM/cyp/v3r8/pyz/pip.conf
user:
  $HOME/.pip/pip.conf
  $HOME/.config/pip/pip.conf

The documentation said $VIRTUAL_ENV/pip.conf will be used. $VIRTUAL_ENV was set on my system, but did not show up in the list.

Initially my $HOME was /u, and this caused problems as my userid was not authorised to write to /u/.pip…

Check your $HOME is set to an appropriate value.

Show the configuration files: python3 -m pip config debug

env_var:                                                                   
env
global:                                                      
  /etc/xdg/pip/pip.conf, exists: False                       
  /etc/pip.conf, exists: False                               
site:                                                        
  /usr/lpp/IBM/cyp/v3r8/pyz/pip.conf, exists: False          
user:                                                        
  /u/tmp/pymqi2/.pip/pip.conf, exists: False                 
  /u/tmp/pymqi2/.config/pip/pip.conf, exists: False          

python3 -m pip config list

gave me

[33]WARNING: The directory ‘/u/.cache/pip’ or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.-[0]

because HOME was not pointing to a value writeable directory.

Updating configuration:

Once I had set HOME to a valid value, I could set configuration values.

  • python3 -m pip config --user set user.colin yes
  • python3 -m pip config --user set site.colin yes

gave me

Writing to /u/tmp/pymqi2/.config/pip/pip.conf

This file had

[site]
colin = yes

[user]
colin = yes

python3 -m pip config --global set user.colin yes

gave me ( as expected)

Writing to /etc/pip.conf
[31]ERROR: Unable to save configuration. Please report this as a bug.
PermissionError: [Errno 111] EDC5111I Permission denied.: ‘/etc/pip.conf’

I would need use a suitably authorised userid to do this.

To edit a config file

You need to specify the editor to use

python3 -m pip config --user --editor /bin/oedit edit

Python on z/OS: Creating a pure Python package.

I had problems creating a package with both Python code, and a extension module written in c (shipped as a load module .so object in Unix Services). The problems were that the package name was dependant on the name of the hardware and the level of the operating system. To produce a package for z/OS in general, I would need to build on every z/OS hardware.

Removing the C code made it much easier to package.

The setup.py file for pure Python

import setuptools 
from setuptools import setup, Extension 
import sysconfig 
bindings_mode = 1 
version = '1.12.0' 
setup(name = 'pymqi', 
    version = version, 
    description = 'Python IBM MQI Extension for IBM MQ.', 
    long_description= 'PyMQI is a Python library ', 
    author='Dariusz Suchojad', 
    author_email='pymqi@m.zato.io', 
    url='https://dsuch.github.io/pymqi/', 
    download_url='https://pypi.python.org/pypi/pymqi', 
    platforms='OS/390', 
    package_dir = {'': 'code'}, 
    packages = ['pymqi'], 
    license='Python Software Foundation License', 
    keywords=('pymqi IBM MQ WebSphere WMQ MQSeries IBM middleware'), 
    python_requires='>=3', 
    classifiers = [ 
        'Development Status :: 5 - Production/Stable', 
        'License :: OSI Approved :: Python Software Foundation License', 
        'Intended Audience :: Developers', 
        'Natural Language :: English', 
        'Operating System :: OS Independent', 
        'Programming Language :: C', 
        'Programming Language :: Python', 
        'Topic :: Software Development :: Libraries :: Python Modules', 
        'Topic :: Software Development :: Object Brokering', 
        ], 
    py_modules = ['pymqi.CMQC', 'pymqi.CMQCFC', 'pymqi.CMQXC', 'pymqi.CMQZC'], 
    ) 

Where the py_modules were in ./code/pymqi/ as CMQC.py etc.

Install the wheel package and build it

python3 setup.py bdist_wheel

This gave a package

./dist/pymqi-1.12.0-py3-none-any.whl

Which can be installed on any z/OS system.

setup tools (and so bdist_wheel) has a web page here.

Uninstall it

I used

python3 -m pip –verbose uninstall wheel

to uninstall it

List what is installed

python3 -m pip –verbose list

This gave

Package      Version Location                                                   Installer          
------------ ------- ---------------------------------------------------------- ---------          
cffi         1.14.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
cryptography 2.8     /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
ebcdic       1.1.1   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
numpy        1.18.2  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pip          20.2.1  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pycparser    2.20    /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
setuptools   47.1.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
six          1.15.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
wheel        0.37.1  /u/tmp/pymqi2/lib/python3.8/site-packages                  pip                
zos-util     1.0.0   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    

so we can see the wheel package was installed in my user directory.

Without the –verbose just package and version were displayed, no location or installer.

Why can’t I use this C function – it is there but I cannot see it.

I’ve been writing in C for over 20 years, and it is humbling when you suddenly realise how little you know of a topic.
It reminds me of when I worked for IBM, and the company wanted a skills register. The questions were along the lines of

Rate your skills in the following areas from 0 (nothing) to 10 (expert).

  • z/OS
  • DB2
  • CICS
  • etc

Overall the skills register was found to be not useful, as the rankings were inverted. If someone put themselves down as 10 – it usually meant they knew very little, they knew enough for their day to day work. If someone put themselves down as 2 they may be an expert who realises how much they do not know, or someone who honestly realises they do not know very much.

My humbling discovery was that when I ported some existing C code to run on z/OS, the functions were not visible outside of the C program. There were two reasons for this.

  • The functions were defined as static,
  • The functions were not exported.

Static functions

static int hidden(int  i)
{
  return 0;
}
int visible(int i)
{
  int x = hidden(1);
  return 0; 
}

I think that using “static” in this case is the wrong word. It does not mean static. I think “internal” would be a clearer description, but I do not think that I’ll have any success changing the C language to use “internal”.

The function “hidden” can only be used within the compiled unit. It cannot be referenced from outside of the compiled object. The “visible” function can use the “hidden” function as the code shows.
The function”visible” is potentially visible to external programs. You can load the module and execute the function.

Exported functions

You have to tell the compiler to externalise functions within the compile unit.

For example

#pragma export(COLIN)
int COLIN(char * self, int args) {
   return 8;
}

or the compiler option EXPORTALL for example

cc… -Wc,EXPORTALL

What has been exported?

When you bind (linkedit) your program, the binder and report the exported functions. You need the binder parameters XREF and DYNAM=DLL

This gave output like

IMPORT/EXPORT     TYPE    SYMBOL              DLL                 DDNAME   SEQ  MEMBER 
-------------     ------  ----------------    ----------------    -------- ---  --------- 
   IMPORT         CODE64  __a2e_l             CELQV003            CELQS003  01  CELQS003 
   IMPORT         CODE64  malloc              CELQV003            CELQS003  01  CELQS003 
   IMPORT         CODE64  CSQB3BAK            CSQBLB16            MQ        01  CSQBMQ2X 
                                                                                                   
   EXPORT         DATA64  ascii_tab 
   EXPORT         CODE64  printHex 
   EXPORT         CODE64  COLIN 
   

This shows the imported symbols, and where they came from, and what was exported.

  • ascii_tab is a table of data in 64 bit mode
  • printHex is a function in 64 bit mode
  • COLIN is a function in 64 bit mode.

How to use it.

You can use handle= dlopen(name,mode) to get the load module into storage, and functionPointer=dlsym(handle,”COLIN”) to locate the external symbol COLIN in the load module.

Why is a static function useful?

With Python external functions (written in C), it uses static functions to hide internals. For example

static PyObject * pymqe_MQCONN(
... )
...
static struct PyMethodDef pymqe_methods[] = { 
  {"MQCONN", (PyCFunction)pymqe_MQCONN,... }, 
  {"MQCONNX", (PyCFunction)pymqe_MQCONNX,... },
  ....  
}

When the external function is imported, the initialisation routine returns the pymqe_methods data to Python.

Python now knows what functions the module provides (MQCONN, MQCONNX), and the C code to be executed when the function is executed.

This means that you cannot load the module, and accidentally try to use the function pymqe_MQCONN; which I thought was good defensive programming.

Python on z/OS using a load module (and .so from Unix Services)

As part of playing with Python on z/OS I found you can call a z/OS Unix Services load module ( a .so object) from Python. It can also use a load module in a PDSE.

What sort is it?

A load module on z/OS can be used on one of two ways.

  • Load it from steplib or PATH environment variable (which uses dllopen under the covers), call the function, and pass the parameters The parameters might be a byte string ( char * in C terms), a Unicode string, integer etc. You return one value, for example an integer return code.
  • As part of a package where you use the Python “import package” statement. This gets loaded from PYTHONPATH, the current directory, and other directories (but not steplib). The parameters passed in are Python Objects, and you have to use Python functions to extract the value. You can return a complex Python object, for example a character string, return code and reason code.

This article is on the first case.

In both cases, the values passed in are in ASCII. If you use printf to display data, the printf treats your data as ASCII.

There is a good article on Python ctypes , a foreign function library for Python.

My initial program was

int add_it(int i, int j)
{
   return i+j;
}

I compiled it with a bash script

wc64=”-Wc,SO,LIST(lst64),XREF,LP64,DLL,SSCOM,EXPORT”
cc -c -o add.o ${wc64} add.c
cc -o add -V -Wl,DYNAM=DLL,LP64 add.o 1>ax 2>bx

This created a load module object “add”. Note: You need the EXPORT to make the entry point(s) visible to callers.

My python program was

import ctypes
from ctypes.util import find_library
zzmqe = ctypes.CDLL(“add”)
print(“mql”, zzmqe.add_it(2,5))

When this ran it produced

mql 7

As expected. To be consistent with Unix platforms, the load module should be called add.so, but “add” works.

Using strings is more complex

I changed the program (add2) to have strings as input

int add_one(char * a, char *b)
{g
printf("b %s\n",b);
return 2 ;
}

and a Python program, passing a byte string.

import ctypes
from ctypes.util import find_library
zzmqe = ctypes.CDLL("add2")
print("mql", zzmqe.add_one(b'abc',b'aaa'))

This ran and gave output

-@abc–@aaa-mql 2

This shows that Python has converted the printf output ASCII to EBCDIC, and so the “a” and “b” in the printf statements are converted to strange characters, and the \n (new line) is treated as hex rather than a new line.

When I compiled the program with ASCII (-Wc…ASCII), the output from the Python program was

a abc
b aaa
mql 2

Displaying the data as expected.

Using a load module in a PDSE.

The JCL

//COLINC3 JOB 1,MSGCLASS=H,COND=(4,LE)
//S1 JCLLIB ORDER=CBC.SCCNPRC
// SET LOADLIB=COLIN.C.REXX.LOAD
// SET LIBPRFX=CEE
//COMPILE EXEC PROC=EDCCB,
// LIBPRFX=&LIBPRFX,
// CPARM=’OPTFILE(DD:SYSOPTF),LSEARCH(/usr/include/),RENT’,
// BPARM=’SIZE=(900K,124K),RENT,LIST,XREF,RMODE=ANY,AMODE=64
//COMPILE.SYSOPTF DD DISP=SHR,DSN=COLIN.C.REXX(CPARMS)
// DD *
EXPORT,LP64
/*
//COMPILE.SYSIN DD *
int add_one(int i, int j)
{
return i+j;
}

//COMPILE.SYSLIB DD
// DD
// DD DISP=SHR,DSN=COLIN.C.REXX
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB.
//BIND.SYSLIB DD DISP=SHR,DSN=CEE.SCEEBND2
// DD DISP=SHR,DSN=CEE.SCEELKED
// DD DISP=SHR,DSN=CEE.SCEELIB
//BIND.OBJLIB DD DISP=SHR,DSN=COLIN.C.REXX.OBJ
//BIND.SYSIN DD *
NAME ADD3(R)
/*

In Unix services

export STEPLIB=”COLIN.C.REXX.LOAD“:$STEPLIB

The python program

import ctypes
from ctypes.util import find_library
zzmqe = ctypes.CDLL(“ADD3”)
print(zzmqe)
print(dir(zzmqe))
print(“steplib”, zzmqe.add_one(2,5))

This gave

steplib 7

Byte string and character strings parameters

I set up a C program with a function COLIN

#pragma export(COLIN)
int COLIN(char * p1, int p2) {
printf(" p1 %4.4s\n",p1);
printf(" p2 %i\n",p2);
return 8;
}

This was compiled in Unix Services, and bound using JCL into a PDSE load library as member YYYYY.

I used a shell to invoke the Python script

export LIBPATH=/u/tmp/python/usr/lpp/IBM/cyp/v3r10/pyz/lib/:$LIBPATH
export STEPLIB=COLIN.C.REXX.LOAD:$STEPLIB 
python3 dll.py

where the Python dll.py script was

import ctypes
testlib = ctypes.CDLL("YYYYY")
name =b'CSQ9'
i = 7
zz = testlib.COLIN(name,i)
print("return code",zz)

This displayed

p1 CSQ9
p2 7
return code 8

The name, a binary string CSQ9, was passed as a null terminated ASCII string (0x43535139 00).

When a string name = “CSQ9” was passed in, the data was in a Unicode string, hex values

00000043 00000053 00000051 00000039 00000000

You need to be sure to pass in the correct data (binary or Unicode), and be sure to handle the data in ASCII.

Is this how ‘import’ works?

This is different process to when a module is used via a Python import statement. If you passed in b’ABCD’ to a C extension which has been “imported” this would be passed as a Python Object, rather than the null terminated string 0x’4142434400′.