Python import, packages and modules.

I’ve been building various Python packages (for example pymqi for z/OS, and accessing z/OS datasets from Python). It took me a while to understand how Python import works, for example why I needed two packages, one for my load modules, and one for the Python code.

There is a lot of good documentation but I felt it was missing the end user’s view who was starting to work in this area.

The import statement

The Python import makes external functions and classes available to a program. The syntax is like

import abc as xyz

x = zyx…..

abc can be

• a file abc.py
• a directory abc
• a load module abc.so

They do the same thing, but differently

The abc.py file

This Python source file can have a class (for objects) or functions in the file. It can import other files.

The abc.pyc file

This is a compiled Python file (from abc.py).

The abc.so load module

The load module is generated from C source.

This can defined a function based approach, so you would use it like

fileHandle = zos.fopen(“colin.c”,”rb”)
data = zos.fread(fileHandle)
zos.fclose(fileHandle)

You can provide many functions. Some functions may return a “handle” object which is passed to other functions.

It can also be object based and the C code creates a new type.

hFile = zos.fopen(“colin.c”,”rb”)
data = hFile.fread()
hFile.fclose()

The function calls are attached to the object (hFile) – rather than the load module zos.

Internally the object is passed to the function.

The abc directory with __init__.py

This is known as a “regular” module package.

It has the __init__.py file, and can have other files and subdirectories.

The __init__.py is run when the package is first imported, so this can import other packages and do other initialisation.

The abc directory without __init__.py

This is the follow-on to regular module package, known as a “namespace” package. It feels a bit strange, and I guess most people do not need to know about it.

I’ll give the concept view here, and give an expanded description below.

For example you have a couple of directories

• /u/mystuff/xyz/abc.py
• /u/mystuff/xyz/a2.py
• /usr/myprod/xyz/hij.pj
• /usr/myprod/xyz/klm.pj

and when the PythonPath has both directories in it, you can use

import xyz
from xyz import abc, klm

which selects the directories in the PythonPath and imports from these.

Packages

The documentation says …

Python defines two types of packages, regular packages and namespace packages. Regular packages are traditional packages as they existed in Python 3.2 and earlier. A regular package is typically implemented as a directory containing an __init__.py file. When a regular package is imported, this __init__.py file is implicitly executed, and the objects it defines are bound to names in the package’s namespace. The __init__.py file can contain the same Python code that any other module can contain, and Python will add some additional attributes to the module when it is imported.

A Namespace package is a composite of various portions, where each portion contributes a sub-package to the parent package. Portions may reside in different locations on the file system. Portions may also be found in zip files, or where-ever else that Python searches during import. Namespace packages may or may not correspond directly to objects on the file system; they may be virtual modules that have no concrete representation.

• regular packages
• namespace packages

My view as to how they work is

Regular packages

You have PYTHONPATH pointing to a list of directories.

You want to import foo.

• For each directory on PYTHONPATH
  • If <directory>/foo/__init__.py is found, return the regular package foo
  • If <directory>/foo.{py,pyc,so,pyd} is found, return the regular package foo

If this returns with a package then import the package.

Namespace package

You have PYTHONPATH pointing to a list of directories.

You want to import foo.

• dirList = “”
• For each directory on PYTHONPATH
  • If <directory>/foo/__init__.py is found, return the regular package foo
  • If <directory>/foo.{py,pyc,so,pyd} is found, return the regular package foo
  • If “<directory>/foo/” is a directory then dirList += “<directory>/foo/“

If no package was returned, and dirList is not empty then we have a namespace package.

This can be used as follows

from foo import abc

has logic like

• for d in dirlist:
  • if d/”abc.*” exists then return d/”abc….”

This has the advantage that you can work on a sub component.

If you have PYTHONPATH = /u/colin;/usr/python, and there is a file /u/colin/foo/abc.py, the statement from foo import abc, xyz imports /u/colin/foo/abc and /usr/python/foo/xyz.py