Running some Python programs on z/OS I got the above error when using Python 11.

If seems that when the C code was compiled, an option (which I cannot find documented) says make it downward compatible.

The fix is easy…

• Mount the Python 11 file system r/w (this is a one of)
• cd  /u/ibmuser/python/v3r11/lib or what every library you are using
• ln -s libpython3.11.so libpython3.8.so
• This says… if you are looking for  libpython3.8.so … go and use libpython3.11.so.

The command ls -ltr /u/ibmuser/python/v3r11/lib/libpython* gave

-rwxr-xr-x ... Jul 15 12:09 /u/ibmuser/python/v3r11/lib/libpython3.11.so                     
lrwxrwxrwx ... Sep 6 12:11  /u/ibmuser/python/v3r11/lib/libpython3.8.so -> libpython3.11.so   

I’ve been working in some code to be able to use z/OS datasets, and DD statements. It took me a while to understand how some bits of Python work.

I also did things such as open a file, allocate a 1MB buffer, and wondered how to close the file, and release the buffer to prevent a storage leak.

The Python import

The Python import makes external functions and classes available to a program. The syntax is like

import abc as xyz

x = xyz…..

abc can be

• a file abc.py
• a directory abc
• a load module abc.so

I’ll focus on the load module.

The abc.so load module

This can define a function based approach, so you would use it like

fileHandle = zos.fopen(“colin.c”,”rb”)
data = zos.fread(fileHandle)
zos.fclose(fileHandle)

You can provide many functions. Some may return a “handle” object, such as fileHandle which is passed to other functions.

It can also be object based and the C load module external function creates a new type.

file = zos.fopen(“colin.c”,”rb”)
data = file.fread()
file.close()

The functions are associated with the object “file”, rather than the load module zos.

Internally the object is passed to the function.

Cleaning up

Within my code I had fileHandle = fopen(“datasetname”….), which allocated a 1MB buffer for the read function.

I also had fclose(fileHandle) where I closed the file and freed the buffer.

However I could also do

fileHandle = fopen(“datasetname1″….)
fileHandle = fopen(“datasetname2″….)
fileHandle = fopen(“datasetname3″….)
fclose(fileHandle)

with no intermediate fclose(), which would lead to a storage leak as the file was fclose routine was not being called.

Using a class to call a function at exit

If you have a Python class for your data you can use

def cb(self,a,b):
     self.handle =  zconsole.acb(a,b)
     atexit.register(self.clean_up,self.handle)

def clean_up(self,handle):
    if handle != None:
        zconsole.cancel(self.handle)

When function cb is used, it registers with the “at exit” routine atexit, and says, “at exit” call my routine “clean_up”, and pass the handle.

At shutdown the clean_up routine is called once for every instance, and gives the cancel code a chance to clean up.

Using a C external function and “functions”.

Within the external functions C code, is PyModuleDef which defines the module to Python.

As such there is no way to automatically get your clean up function to be called (and free my 1MB buffer).

However you can exploit the Python module state data. For example

struct {
myparm * ...
...
} myStatic;

static struct PyModuleDef zos_module = {
  PyModuleDef_HEAD_INIT,
  "zos",
  zos_doc,
  sizeof(myStatic),
  zos_methods, // the functions (methods)
  NULL, // Multi phase init. NULL -> single
  NULL, // Garbage collection traversal
  zos_clear, // Garbage collection clear
  zos_free // Garbage collection free
};

The block of state data is allocated for you, and you can issue the PyModule_GetState(PythonModule) function to get access this block.

You chain could chain your data from the state data, perhaps in a linked list.

When the clean up occurs, your “zos_free” routine will be called, and you can free all the storage you allocated and clean up.

For example

PyMODINIT_FUNC PyInit_zos(void) { 
  PyObject *d; 
                                                                                        
  /* Create the module  */ 
  mzos = PyModule_Create(&zos_module); 
  // get the state data and initialise it
  state * pState = (state * )  PyModule_GetState(mzos); 
  memcpy(pState -> eyec,"state   ",8);
  ... 
                                  
  PyDict_SetItemString(d, "__doc__", Py23Text_FromString(zos_doc)); 
  PyDict_SetItemString(d,"__version__", Py23Text_FromString(__version__)); 
                                                                                        
return mzos;

Using a C external function and “objects” or types.

With a “function based” function, you have Python code like

fileHandle = zos.fopen("myfilename"....)
data = zos.fread(fileHande)
...

With “object based” functions you have Python code like

file = zos.fopen("colin.c","rb")
data = file.fread()
file.close()

In this case the object is a Python type. There is a good description here.

As with function based code you define the attributes of the object, including the tp_dealloc function. This gets control when the object is deallocated. In the Custom_dealloc, function you can close the file and free the buffer etc.

static PyTypeObject CustomType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom.Custom",
    .tp_doc = PyDoc_STR("Custom objects"),
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_dealloc = (destructor) Custom_dealloc,
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_new = PyType_GenericNew,
};

static void
Custom_dealloc(CustomObject *self)
{
   ... // put your code here
}

static PyModuleDef custommodule = {
    PyModuleDef_HEAD_INIT,
    .m_name = "custom",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
};

PyMODINIT_FUNC
PyInit_custom(void)
{
    PyObject *m;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;
    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) <  0) {
        Py_DECREF(&CustomType);
        Py_DECREF(m);
        return NULL;
    }
    return m;
}

Note: The list of available .tp… definitions is available here.

I’ve been building various Python packages (for example pymqi for z/OS, and accessing z/OS datasets from Python). It took me a while to understand how Python import works, for example why I needed two packages, one for my load modules, and one for the Python code.

There is a lot of good documentation but I felt it was missing the end user’s view who was starting to work in this area.

The import statement

The Python import makes external functions and classes available to a program. The syntax is like

import abc as xyz

x = zyx…..

abc can be

• a file abc.py
• a directory abc
• a load module abc.so

They do the same thing, but differently

The abc.py file

This Python source file can have a class (for objects) or functions in the file. It can import other files.

The abc.pyc file

This is a compiled Python file (from abc.py).

The abc.so load module

The load module is generated from C source.

This can defined a function based approach, so you would use it like

fileHandle = zos.fopen(“colin.c”,”rb”)
data = zos.fread(fileHandle)
zos.fclose(fileHandle)

You can provide many functions. Some functions may return a “handle” object which is passed to other functions.

It can also be object based and the C code creates a new type.

hFile = zos.fopen(“colin.c”,”rb”)
data = hFile.fread()
hFile.fclose()

The function calls are attached to the object (hFile) – rather than the load module zos.

Internally the object is passed to the function.

The abc directory with __init__.py

This is known as a “regular” module package.

It has the __init__.py file, and can have other files and subdirectories.

The __init__.py is run when the package is first imported, so this can import other packages and do other initialisation.

The abc directory without __init__.py

This is the follow-on to regular module package, known as a “namespace” package. It feels a bit strange, and I guess most people do not need to know about it.

I’ll give the concept view here, and give an expanded description below.

For example you have a couple of directories

• /u/mystuff/xyz/abc.py
• /u/mystuff/xyz/a2.py
• /usr/myprod/xyz/hij.pj
• /usr/myprod/xyz/klm.pj

and when the PythonPath has both directories in it, you can use

import xyz
from xyz import abc, klm

which selects the directories in the PythonPath and imports from these.

Packages

The documentation says …

Python defines two types of packages, regular packages and namespace packages. Regular packages are traditional packages as they existed in Python 3.2 and earlier. A regular package is typically implemented as a directory containing an __init__.py file. When a regular package is imported, this __init__.py file is implicitly executed, and the objects it defines are bound to names in the package’s namespace. The __init__.py file can contain the same Python code that any other module can contain, and Python will add some additional attributes to the module when it is imported.

A Namespace package is a composite of various portions, where each portion contributes a sub-package to the parent package. Portions may reside in different locations on the file system. Portions may also be found in zip files, or where-ever else that Python searches during import. Namespace packages may or may not correspond directly to objects on the file system; they may be virtual modules that have no concrete representation.

• regular packages
• namespace packages

My view as to how they work is

Regular packages

You have PYTHONPATH pointing to a list of directories.

You want to import foo.

• For each directory on PYTHONPATH
  • If <directory>/foo/__init__.py is found, return the regular package foo
  • If <directory>/foo.{py,pyc,so,pyd} is found, return the regular package foo

If this returns with a package then import the package.

Namespace package

You have PYTHONPATH pointing to a list of directories.

You want to import foo.

• dirList = “”
• For each directory on PYTHONPATH
  • If <directory>/foo/__init__.py is found, return the regular package foo
  • If <directory>/foo.{py,pyc,so,pyd} is found, return the regular package foo
  • If “<directory>/foo/” is a directory then dirList += “<directory>/foo/“

If no package was returned, and dirList is not empty then we have a namespace package.

This can be used as follows

from foo import abc

has logic like

• for d in dirlist:
  • if d/”abc.*” exists then return d/”abc….”

This has the advantage that you can work on a sub component.

If you have PYTHONPATH = /u/colin;/usr/python, and there is a file /u/colin/foo/abc.py, the statement from foo import abc, xyz imports /u/colin/foo/abc and /usr/python/foo/xyz.py