Using assembler services from a (64 bit) C program.

I wanted to provide a Python started task on z/OS to respond to the operator Stop, and Modify commands (for example to pass commands to the program).

I wrote a Python extension in C, and got the basics working. I then wanted to extend this a bit more. I learned a lot about the interface from C to assembler, and some of the newer linkage instructions.

The sections in this post are

Some of the problems I had were subtle, and the documentation did not cover them.

C provides a run time facility called __console2 which provide a write to the console, and a read from the console.

The output from the write to the console using __console2 looks like

BPXM023I (COLIN) from console2

With the BPXM023I prefix, which I thought looked untidy and unnecessary.

To use the Modify operator command with __console2 you have to use a command like

F PYTTASK,APPL=’mydata’

Which feels wrong, as most of the rest of z/OS uses

F PYTTASK,mydata

This can be implemented using the QEDIT assembler interface.

The journey

I’ll break my journey into logical steps.

Information on programming the C to assembler interface

There is not a lot of good information available.

Calling an assembler routine from a C program.

Setting up the linkage

A 64 bit program uses the C XPLINK interface between C programs.

To use the traditional Assembler interface you need to use

#pragma linkage(QEDIT , OS)

rc = QEDIT( pMsg, 6);

C does not set the Variable Length parameter list bit (the high bit of the last parameter to ’80…) so you cannot use parameter lists with a variable length, and expect traditional applications to work. You could always pass a count of parameters, or build the parameter list yourself.

Register 1 pointed to a block of storage, for example to parameters

00000000 203920D8 00000050 082FE3A0

which is two 64 byte addresses, the address of the pMsg data, and the address of the fullword constant 6;

The C code invokes the routine by

LG r15,=V(QEDIT)(,…,…)
BALR r14,r15

even though use of BALR is deprecated.

Allocating variable storage

The z/OS services my assembler program used, need 31 bit storage.

I allocated this in my C program using

char * __ptr32 pMsg;
pMsg = (char *) __malloc31(1024);

I then passed this to my assembler routine.

Coding the assembler routine

Assembler Linkage

The basic linkage was

BSM 14,0
BAKR 14,0

…….
PR go back

This is where it started to get complicated. The BAKR… PR is a well documented and commonly used interface.

A Branch and StacK Register instruction BAKR 14,15 says branch to the address in register 15, save the value of register 14 as the return address, and save the registers and other status in the linkage stack. The code pointed to by register 15 is executed, and at the end, there is a Program Return (PR) instruction which loads registers from the linkage stack, and goes to the “return address”.

The Branch and Stack instruction BAKR 14,0 says do not branch, but save the status, and the return address. A subsequent PR instruction will go to where register 14 points to.

Unfortunately, with the BALR code in C, and the BAKR, PR does not work entirely.

You can be executing in a program with a 64 bit address instructions (such as 00000050 089790A0), in 24 or 31, or 64 bit mode.

  • In 64 bit mode, all the contents of a register are used to address the data.
  • In 31 bit mode only the bottom(right) half of the register are used to address the data – the top half is ignored
  • In 24 bit mode, only the bottom 24 bits of the register are used to address the data.

There are various instructions which change which mode the program is executing in.

When a BAKR, or BASSM ( Branch and Save, and set Mode) is used, the return address is changed to set the mode ( 24,31,64) as part of the saved data. When this address is used as a branch back – the save mode information is used to switch back to the original mode.

When BALR (or BASR) is used to branch to a routine, the return address is saved in Register 14. The mode information is not stored. When this address is used as a branch back – the “default mode” information (24 bit) is used to set the mode. This means the return tries to execute with a 24 bit address – and it fails.

You solve this problem by using a (BRANCH AND SET MODE) BSM 14,0 instruction. The value of 0 says do not branch, so this just updates register 14 with the state information. When BAKR is issued, the correct state is saved with it.

If you use the “correct” linkage you do not need to use BSM. It is only needed because the C code is using an out dated interface. It still uses this interface for compatibility with historically compiled programs.

Note: BSM 0,14 is also a common usage. It is the standard return instruction in a program entered by means of BRANCH AND SAVE AND SET MODE (BASSM) or a BRANCH AND SAVE (BAS). It means branch to the address in register 14, and set the appropriate AMODE, but do not save in the linkage stack.

Using 64 and 31 bit registers

Having grown up with 32 bit registers, it took a little while to understand the usage 64 bit registers.

In picture terms all registers are 64 bit, but you can have a piece of paper with a hole in it which only shows the right 32 bit part of it.

When using the full 64 bit registers the instructions have a G in them.

  • LR 2,3 copies the 32 bit value from register 3 into the right 32 bits of register 2
  • LGR 2,3 copies the value of the 64 bit register 3 into the 64 bit register 2

If there is a block of storage at TEST with content 12345678, ABCDEFG

  • R13 has 22222222 33333333
  • copy R13 into Reg 7. LGR R7,R13. R7 contains 22222222 33333333
  • L R7,TEST. 32 bit load of data into R7. R7 now has 22222222 12345678. This has loaded the visible “32 bit” (4 bytes) part of the register, leaving the rest unchanged.
  • LG R8,TEST. 64 bit load into Reg 8. R8 now has 12345678 ABCDEFG . The 8 bytes have been loaded.
  • “clear high R9” LLGTR R9,R9. R9 has 00000000 ……… See below.
  • L R9,TEST . 32 bit (4 bytes) load into R9. R9 now has 00000000 12345678

Before you use any register in 32 bit code you need to clear the top.

The Load Logical Thirty One Bits (LLGTR R1,R2) instruction, takes the “right hand part” of R2 and copies it to the right hand of R1, and sets the left hand part of R1 to 0. Or to rephrase it, LLGTR R2,R2 just clears the top of the register.

Using QEDIT to catch operator commands

QEDIT is an interface which allows an application to process operator commands start, modify and stop, on the address space.For example

  • S PYTASK,,,’STARTPARM’
  • f PYTASK,’lower case data’
  • p PYTASK

Internally the QEDIT code uses CIBs (Console Information Block) which have information about the operator action

(I think QEDIT comes from “editing”= removing the Queue of CIBs once they have been processed – so Q EDIT).

The interface provides an ECB for the application to wait on.

The documentation was ok, but could be clearer. For example the sample code is wrong.

In my case I wanted a Python thread which used console.get() to wait for the operator action and return the data. You then issue the console.get() to get the next operator action.

My program had logic like

  • Use the extract macro to get the address of the CIBS.
  • If this job is a started task, and it is the ‘get’ action then there will be a CIB with type=Started
    • Return the data to the requester
    • Remove the CIB from the chain
    • Return to the requester
  • Set the backlog of CIBs supported ( the number of operator requests which can be outstanding)
  • WAIT on the ECB
  • Return the data to the requester
  • Remove the CIB from the chain
  • Return to the requester

The action can be “Start”, “Modify”, or “Stop”

Can I use __ASM__ within C code to generate my own assembler?

In theory yes. The documentation is missing a lot of information. I could not get my simplest tests to work and return what I was expecting.

Using PyArg_ParseTupleAndKeywords to parse data in Python external functions.

I was failing to use PyArg_ParseTupleAndKeywords succcessfully in an external Python function. It took about a day to get it to work. Below are some of the lessons learned in using this facility.

Background

When Python calls an external function (written in C) you can pass parameters, and keywords. For example a function taking one positional parameter, and keywords a,b,c,d could be used:

rc = zconsole.put( 12345 , d = 7, b = 10 )

rc = zconsole.put( 12345 , b=10, d= 7)

Where

  • 12345 is a positional parameter,
  • keyword d is set to the value 7
  • keyword b is set to the value 10
  • the keyword values a and c are not set, and the values are unchanged.

My C program had

static PyObject *console_put(PyObject *self, PyObject *args, PyObject *keywds ) {

int a = 0;
int b = 0;
int c = 0;
int d = 0;
int pos0 = 0;
int pos1 = 0;
static char *kwlist[] = {“pos0″,”pos1,”,”a”,”b”,”c”,”d”,NULL};
if (!PyArg_ParseTupleAndKeywords(args, keywds, “i|i$iiii”, kwlist,
&pos0 ,
&pos1 ,
&a , // i
&b , // i
&c , // i
&d )) // i
{
return NULL;
}

The kwlist array contains the keywords, so it includes a,b,c,d. It must also include labels for the positional parameters (I was missing this for the first day!). pos0 is for the first positional parameter. There is also an optional second position parameter pos1. The list ends with a NULL.

The PyArg_ParseTupleAndKeywords function takes

  • args passed into your function
  • keywds passed into your function
  • the format string “i|i$iiii”. This says
    • i – a required positional integer
    • | – following parameters are optional
    • i -an optional positional integer
    • $ – the following are keywords
    • iiii – 4 integers corresponding to the keywords in {“pos0″,”pos1,”,“a”,”b”,”c”,”d”,NULL};

The message

SystemError: more argument specifiers than keyword list entries (remaining format:’i’)

means the number of elements in the format string does not match the number of elements in the keywords array. I got the above error when I removed “pos0″,”pos1” from the list.

Processing strings

With well defined values like integers have a known length. Strings can have different sizes.

In the format you define s#,

  • s – return a pointer to the string
  • # – give the length of the string.

Py_ssize_t lMsg;
char * p;

static char kwlist[] = {“pos0″,”pos1″,”a”,”b”,”c”,”d”,NULL};
if (!PyArg_ParseTupleAndKeywords(args, keywds, “i|i$s#iii”, kwlist,
&pos0 ,
&pos1 ,
&p , // s this variable is a string
&lMsg , // this is the size of the string the ‘#’ above
&b , // i this variable is int
&c , // i this variable is an int
&d )) // i
{ return NULL; }
printf(“lMsg=%d\n”,lMsg);
printf(“String=%*.*s\n”,lMsg,lMsg,p);
printf(“pos0=%d\n”,pos0);

The code

rc = zconsole.put( 12345 , a = “abcde”, b = 10 )

gave

lMsg=5
String=abcde
pos0=12345

Setting up the function to use keywords

You need to set a flag to say that keywords are being used. For example

static struct PyMethodDef console_methods[] = {
{“put”, (PyCFunction)console_put,METH_KEYWORDS | METH_VARARGS, console_put_doc},
{NULL, (PyCFunction)NULL, 0, NULL} /* sentinel */
};

I want to pass a variable number of data items

You can configure PyArg_ParseTupleAndKeywords with a list containing optional fields, but how do you pass multiple fields in?

You can say parse the data, and return a field as a python object.

PyObject *rv = NULL;

if (!PyArg_ParseTupleAndKeywords(args, keywds, “O|….”, kwlist,
&rv ,
….

rc = PyTuple_Check(rv );
if (rc == 1 ) // it is a tuple
{
size = PyTuple_Size(rv );
printf(“Tuple size %d\n”,size);
}

The “O” code says return the object, rather than the string.

You can now do your own checks on the code for example get the type of the object.

  • rc = PyList_Check(t );
  • rc = PyLong_Check( t)
  • rc = PyBytes_Check(t );
  • rc = PyByteArray_Check(t);
  • rc = PyBool_Check(t);
  • rc = PyFloat_Check(t);
  • rc = PyFunction_Check(t) ;
  • rc = PySet_Check(t );
  • rc = PyTuple_Check(t );
  • rc = PyDict_Check(t );
  • rc = PyUnicode_Check(t);
  • rc = PyComplex_Check(t);

There is no “getType” function as such see here.

Once you know it’s type you can use Pyxxx_Size to get the size, or number of elements in the object.

If the data is a tuple of 3 bits of data

rc = zconsole.put((“a123456789″,”b123″,”c44” ) )

PyTuple_Check(rv) will return 1, and you can get the number of elements using PyTuple_Size which returns 3.

You can use PyArg_ParseTuple to dig into the objects you have retrieved, for example, for the object rv obtained above:

if (!PyArg_ParseTuple(rv, “s#|s#”,
&p1 , // message text
&lMsg1 , //i
&p2 , // message text
&lMsg2))
{
return NULL;
}
printf(“p1 %.s\n”,lMsg1,lMsg1,p1);
printf(“p2 %.s\n”,lMsg2,lMsg2,p2);

Parsing a fixed size tuple

You can use ‘(…)’ within the format string to process data in a tuple, so you do not need to dig into the object

if (!PyArg_ParseTupleAndKeywords(args, keywds, “(s#s#)|$iiiii”, kwlist,
&p1 , // string1
&lMsg1 , //i
&p2 , // string2
&lMsg2, //i

Getting the type of an object

There is no “getType” function as such.

You can use

int type;
type = PyType_GetFlags(Py_TYPE(rv ));
printf(“TYPE = %8.8x\n”,type);

The values of type are listed here. A constant is defined as

#define Py_TPFLAGS_TUPLE_SUBCLASS (1UL << 26)
etc

which is 0x04000000. I found it easier to use the pyTuple_Check() function than decode the fields.

Setting up Python on z/OS

Running a Python package on z/OS is pretty easy. You just have to remember to set up the environment. This one of those little tasks you do not do very often, and forget how to do it.

When you enter OMVS your .profile shell is executed (if the file exists).

Does the file exist ?

You can use the ls command to see if the file exists

ls -altr ~/.profile

where the -a option says display all the files – including those beginning with ‘.’. By default these are omitted.

This gave

-rwx------ 1 BPXROOT TSOUSER 297 May 18 08:46 /u/colin/.profile

Note the x attribute; it means the file can be executed. You can use the following command to set this attribute.

chmod +x ~/.profile

Profile contents

My .profile file has

# set up Python path 
export PATH=/usr/lpp/IBM/cyp/v3r8/pyz/bin:$PATH 
# set up MQ Libraries
export MQ=MQM.V900 
export STEPLIB=$MQ.SCSQANLE:$MQ.SCSQANLE:$MQ.SCSQAUTH:$STEPLIB 
# set up PYTHONPATH to point to where zpymqi is
export PYTHONPATH=/u/colin/:$PYTHONPATH 
      

When my Python program has import pymqi, Python will look in directory /u/colin/pymqi

Getting PyDev to work with IBM Developer for z/OS

PyDev is an integrated Development Environment for Python which works in Eclipse. It has all the capabilities you expect with an IDE.

I had problems getting PyDev to work on IBM Developer for z/OS ( ID/z). Gerald Mitchell of IBM sent me instructions which worked perfectly – so thanks to him for them. I’ve modified them slightly.

Note / Disclaimer: PyDev is not currently officially supported by the IBM Developer for z/OS product. If you have problems, IBM will not be able to help you.

The problems I had were that PyDev installed, but did not display within Eclipse. Window-> Preferences did not show PyDev, and so could not be used.

The latest PyDev requires Java 11, but ID/z in Java is at V8. PyDev 8.2.0 from February 2021 works, see PyDev Releases.

PyDev 8.2.0 is on the PyDev GitHub for install at PyDev 8.2.0 Release · fabioz/Pydev

I installed Pydev using “Install new software”. Add site

 This displayed a check box for        

  • PyDev for Eclipse 8.2.0.202102211157
  • PyDev for Eclipse Developer Resources 8.2.0.202102211157
  • PyDev Mylyn Integration 0.6.0

I selected the first one.

I got a pop up for a Security warning for unsigned content, I said “install anyway.”

I restarted the work bench. The restart may take a while as the plugins and features are integrated.

The Window -> Preferences should have PyDev in the list.

You can check the file associations now…

  • Window -> Preferences -> General -> Editors -> File Associations should have a File types: entry for *.py and the Associated Editors: section should list Python Editor as one of the options.
  •  Window -> Preferences -> PyDev should also be populated with preferences for Editor, Interpreters, PyUnit, and Scripting PyDev.
  • If you don’t set up Python via the preferences, on the first python file interaction, you may get a message that Python interpreter is not currently configured.

You can edit Python files inside the base IDz install with some of the generic file editors already installed, though they will not have Python specific language knowledge or capabilities such as content assist or syntax highlighting

To allow .py files to be edited internally, by setting the Window -> Preferences -> General -> Editors -> File Associations. Here you can choose to change the Open associated files with: option from System Editor; if none Text Editor to Text Editor as one option.
Another option is in the File types: section press Add… and then in the dialog that comes up type *.py and then press OK. (If you get a dialog message that says a file type *.py already exists that is OK: this means PyDev registered as an editor which is fine. )
In the Associated editors: section press Add… and select the Internal editors option. I would recommend then choosing Generic Text Editor, Text Editor, System z LPEX Editor, or z Systems LPEX Editor but there are several other options that may work for you. Note that you can Add… more than one editor to a a single file type, at which point if you right click on the *.py files you will have an option of which editor to use.
The one that is used when you double click or right click -> Open is the one you set to default, which you can adjust in this section by selection of one of the editors and then pressing the Default button.
Then press Apply and Close.

Example of editing a .py file

and

Running the debugger

I was not able to use the debugger as it tries to run it on my work station, not on z/OS

Understanding PCF cmdscope on z/OS

I wanted to check the response to a PCF command when cmdscope=”*” was issued.

It took a while to understand it. It feels like it was implemented by different people with different strategies. I’ve documented what I found in case someone else struggles down this path. As an extra challenge I used Python to issue the command and process the responses.

High level view of the cmdscope reply

A PCF request is sent to a server, and responses are turned to the specified reply-to-queue. These reply messages all have the same correlid, but different msgids.

  • The first message gives information about the remainder of the messages, and how many queue managers are involved.
  • The last message is a message saying “finished”, and loosely ties up with the first messages.
  • In between there are sets of messages from each queue manager. There are one or more messages in the set. Each set has its own identifier.

More information

The first message

  • This has a Response id – it is not used anywhere else.
  • Says there replies from N queue manager. The value of N is given.
  • There is a Response Set structure (A) giving the Response id for the “end of command scope” message.
  • There are one or more Response Set structures giving the Response id for each queue manager. There are N of these structures.

The last message has a Response-id which matches the Response set (A). The content of this last message is “the request (with command scope) has finished”.

Each queue manager returns a set one or more messages.

  • All messages in the set have a Response-id for the set; which is given in the first message above. The name of the queue manager providing the information is given.
  • The last message in the set has the “Last message in the set” flag set in the PCF header.

Response-id values

Each message has a Response-id field. This has a value like “CSQ M801 GM802…” where the … is hex data. The original request was to queue manager M801, and in this case the response was for queue manager M802. There are several field that look similar to this, for example “CSQ M801 RM801…”. The content of this response is not an API.

How many messages should I expect?

The answer is “it depends”.

  • You will get a matching first and last message (2)
  • You will get a set of messages for each queue manager
    • There will be one or more messages with data.
    • The number of messages with data varies. For example the Inquire archive request returns returns the values at startup. Sysp Type : Type Initial.
    • If a Set Archive command has been issue, a second record with Sysp Type : Type Set, will be present, which shows the parameters which have been changed since startup.
    • A last in set message.

You know when you have the last message when you have received the “the request (with command scope) has finished” message.

At a practical level…

I wrote code like that below, to get all the messages in a request. If is more complex than written as the PCF body may not have the structure with the data type.

cmdscopeloop = no
_______________

Loop: Get message with PCF header and PCF body
## If cmdscope was specified, keep looping until end cmdscope message
If header.type ==MQCFT_XR_MSG and
body[MQIACF_COMMAND_INFO] == MQCMDI_CMDSCOPE_ACCEPTED
then cmdscopeloop = yes

If header.type ==MQCFT_XR_MSG and
body[MQIACF_COMMAND_INFO] == MQCMDI_CMDSCOPE_COMPLETED
then cmdscopeloop = no
## Requests like inquire chinit have a command accepted
If header.type ==MQCFT_XR_MSG and
body[MQIACF_COMMAND_INFO] == MQCMDI_COMMAND_ACCEPTED
then state = “CommandAccepted”

## Most messages have
if header.type ==MQCFT_XR_ITEM
then state = “Item”

## The summary record means end of set.
##For many requests this is end of data
if header.type ==MQCFT_XR_SUMMARY
then state = “endset”

## see if we need to keep looping.
if state == “endset” and cmdscopeloop == no
then return
else loop to get more messages

The request and detailed responses

I used PCF with the command MQCMD_INQUIRE_ARCHIVE, cmdscope=”*”. I send the request to queue manager M801. There were three queue managers in the QSG: M801, M802, M803.

The data description like “Response Q Mgr Name”, is decoded from the PCF value, and post processed to make it more readable, using the MQ sample code (provided on MQ Midrange).

Values like b’M801′ is a Python byte string, or hexadecimal string. Other character data may be in UTF-8 code page.

Response (1) meta information

PCF.Type 17 MQCFT_XR_MSG
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 1
PCF.Control 1 Last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 8

  1. Response Id : CSQ M801 RM801…
  2. Response Q Mgr Name : b’M801′
  3. Command Info : Cmdscope Accepted
  4. Cmdscope Q Mgr Count : 3
  5. Response Set : CSQ M801 SM801… This is for the final “end of cmdscope” message
  6. Response Set : CSQ M801 GM801… These are for the data returned from a queue manager
  7. Response Set : CSQ M801 GM802…
  8. Response Set : CSQ M801 GM803…

Response (2), for queue manager M801. The initial values

PCF.Type 18 MQCFT_XR_ITEM
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 1
PCF.Control 0 Not last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 19

  1. Response Id : CSQ M801 GM801… Matching a response set value in the first message
  2. Response Q Mgr Name : b’M801′
  3. Sysp Type : Type Initial
  4. Sysp Archive Unit1 : b’TAPE’
    ….

19. Sysp Quiesce Interval : 5

Response (3), for queue manager M801. The fields which have been set

On queue manager M801 the command SET ARCHIVE UNIT(DISK) was issued. If the DISPLAY ARCHIVE command is issued it will display DISK under the “SET value” column.

PCF.Type 18 MQCFT_XR_ITEM
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 2
PCF.Control 0 Not last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 4

  1. Response Id : CSQ M801 GM801…
  2. Response Q Mgr Name : b’M801′
  3. Sysp Type : Type Set
  4. Sysp Archive Unit1 : b’DISK’

Response(4), for queue manager M801. End of queue manager’s data

PCF.Type 19 MQCFT_XR_SUMMARY
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 3
PCF.Control 1 Last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 2

  1. Response Id : CSQ M801 GM801…
  2. Response Q Mgr Name : b’M801′

Response (5), for queue manager M802. The initial values

PCF.Type 18 MQCFT_XR_ITEM
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 1
PCF.Control 0 Not last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 19

  1. Response Id : CSQ M801 GM802…
  2. Response Q Mgr Name : b’M802′
  3. Sysp Type : Type Initial
  4. Sysp Archive Unit1 : b’TAPE’

Response (6), for queue manager M802. End of queue manager’s data

PCF.Type 19 MQCFT_XR_SUMMARY
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 2
PCF.Control 1 Last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 2

  1. Response Id : CSQ M801 GM802…
  2. Response Q Mgr Name : b’M802′

Response (7), for queue manager M803. The initial values

PCF.Type 18 MQCFT_XR_ITEM
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 1
PCF.Control 0 Not last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 19

  1. Response Id : CSQ M801 GM803…
  2. Response Q Mgr Name : b’M803′
  3. Sysp Type : Type Initial
  4. Sysp Archive Unit1 : b’TAPE’

Response (8), for queue manager M803. End of queue manager’s data

PCF.Type 19 MQCFT_XR_SUMMARY
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 2
PCF.Control 1 Last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 2

  1. Response Id : CSQ M801 GM803…
  2. Response Q Mgr Name : b’M803′

Response (9), for queue manager M801. End of command.

PCF.Type 17 MQCFT_XR_MSG
PCF.Command 114 MQCMD_INQUIRE_ARCHIVE
PCF.MsgSeq 1
PCF.Control 1 Last message in set
PCF.CompCode 0
PCF.Reason 0
PCF.ParmCount 3

  1. CSQ M801 SM801…
  2. Response Q Mgr Name : b’M801′
  3. Command Info : Cmdscope Completed

Using parameters in Python unittest

I’ve been writing some unit tests for pymqi and fell at the first hurdle. How do I provide logon information? This cannot be hard coded because every one’s system is different. You cannot pass command line parameters to the unittest module, so how do you do it?

The second hurdle was how do I provide configuration information on what to test.
I solved these in two slightly different ways.

Logging on, or connecting to MQ

I created a separate python script called myConnect.py and put the logon information in that.

To be able to use the unit tests, you just have to customise the myConnect.py and use that.

My file does the connection, and had

import sys
import platform
import pymqi
import threading
def connect():
    queue_manager = 'CSQ9'
    qmgr = pymqi.connect(queue_manager)
    return qmgr

In my unit test script I had

import myConnect
class inq(unittest.TestCase):
    def setUp(self):
        self.qmgr = myConnect.connect()

Simple… and if I need to change my logon information, I just change one file instead of changing all of the scripts.

Passing parameters to the test.

I wanted to be able to pass a “debug option” to the tests, to be able to print information, and to pass other configuration. With the command

python3 -m unittest -v ut.inq1.test_inq_log

you cannot pass parameters.

Environment data

You can pass environment data for example, passing the variables inline

userid="user1" password="secret1" python -m unittest test

and your test.py has

user  = os.environ["userid"]
password = os.environ["password"]

External configuration file

I found it easier to set up a utparms.py and store my variables in that, for example

queues = ["CP0000","CP0001"]

I can then import and use the file.

import utparms
...
queues = utparms.queues
for q in queues:
    ....

This is different to the myConnect.py above, as the myConnect.py has logic to do the connection. The utparms.py is just a list of variable defines, with no logic.

To use this I can have a shell script like

# recreate the utmarms.py every time
cp simpleutparms.py utparms.py
python3 -m unittest -v ut.inq1.test_inq_log

or

cp reallybigutparms.py utparms.py
python3 -m unittest -v ut.inq1.test_inq_log

or dynamically create the utparms.py file.

Do not bang your head against a wall when you have a problem – go round it!

You may think these are obvious, but things are only obvious1 when you understand them. Too many people write tests where the configuration information is within the test, and not isolated, so it is clearly not obvious.


1 I remember reading about a university and the definition of obvious:

  • If professor X says it is obvious, then it is,
  • If professor Y says it is obvious, then after an hours thought it should be obvious,
  • If professor Z says it is obvious, then after a year’s work you will understand it.

Debugging PIP install on z/OS

I had various problems installing Python packaged on z/OS.

Firstly the Python file system was read only – that’s ok. I can try to install in a virtual environment. This also had it’s problems!

The main problem was, I had the HOME environment variable pointing to a directory which my userid did not have write access to – /u/. When I changed it to my working directory /u/tmp/pymi2, it worked with no problems.

I’ll document in the blog the debugging I did.

Preparation

I FTP’d the wheels package in binary from my Linux machine, into /tmp on z/OS.

I switched to my virtual environment using

. env/bin/activate

This gave a command line like

(env) COLIN:/u/tmp/pymqi2:

I used the command

python3 -m pip install /tmp/wheel-0.37.1-py2.py3-none-any.whl /tmp/wheel-0.37.1-py2.py3-none-any.whl –no-cache-dir

But this gave

Installing collected packages: wheel
ERROR: Could not install packages due to an EnvironmentError: [Errno 111] EDC5111I Permission denied.: ‘/u/.local’

Using the –verbose argument

python3 -m pip –verbose install /tmp/wheel-0.37.1-py2.py3-none-any.whl /tmp/wheel-0.37.1-py2.py3-none-any.whl –no-cache-dir

gave more information, but not enough to resolve the problem.
Using

export DISTUTILS_DEBUG=1
python3 -m pip –verbose install..

gave much more information including

config vars:
{‘abiflags’: ”,
‘base’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘dist_fullname’: ‘UNKNOWN-0.0.0’,
‘dist_name’: ‘UNKNOWN’,
‘dist_version’: ‘0.0.0’,
‘exec_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘platbase’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘py_version’: ‘3.8.5’,
‘py_version_nodot’: ’38’,
‘py_version_short’: ‘3.8’,
‘sys_exec_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘sys_prefix’: ‘/usr/lpp/IBM/cyp/v3r8/pyz’,
‘userbase’: ‘/u/.local’,
‘usersite’: ‘./lib/python3.8/site-packages’}

where I could see where the /u/.local came from.

You can change the userbase (see here) using

export PYTHONUSERBASE=.

and the product installed. When I fixed my HOME environment variable to point to my working directory this also worked!

What is installed?

Using the command

python3 -m pip –verbose list

This gave

Package      Version Location                                                   Installer          
------------ ------- ---------------------------------------------------------- ---------          
cffi         1.14.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
cryptography 2.8     /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
ebcdic       1.1.1   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
numpy        1.18.2  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pip          20.2.1  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pycparser    2.20    /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
setuptools   47.1.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
six          1.15.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
wheel        0.37.1  /u/tmp/pymqi2/lib/python3.8/site-packages                  pip                
zos-util     1.0.0   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    

so we can see the wheel package was installed in my user directory.

Without the –verbose just package and version were displayed, no location or installer.

Uninstall it

I used

python3 -m pip –verbose uninstall wheel

to uninstall it

Python virtual environments on z/OS

Virtual environments are very useful as they allow you to test install components without affecting the shared libraries.

You create an environment called env with

python3 -m venv env

but this gave

Error: Command ‘Ý’/u/py/env/bin/python3’, ‘-Im’, ‘ensurepip’, ‘–upgrade’, ‘–default-pip’¨’ returned non-zero exit status 1.

I think this was because my z/OS had no direct network connection. However

python3 -m venv –without-pip env –system-site-packages

worked. I needed the –system-site-packages, so I could build the extension.

Activate the environment

The instruction said use the command env/bin/activate but this failed

env/bin/activate: FSUM9209 cannot execute: reason code = ef076015: EDC5111I Permission denied.

but

. env/bin/activate

worked. (Just . env/b*/a* worked for me.)

If you get
 FSUM7332 syntax error: got Word, expecting )    

You need to issue the command ( or put it in your .profile)

export _BPXK_AUTOCVT=ON

See here. (_BPXK_AUTOCVT Used when enabling automatic conversion of tagged files).

To get out of the virtual environment use

deactivate

To delete the environment

remove the directory

rm -r env

Use the environment

You should now be able to use the environment.

You should check that the HOME environment variable is a directory with read/write access.

You may want to set up PYTHONHOME for special python packages.

The challenges of using PIP on z/OS.

Using Python PIP on z/OS, was not as smooth as I had expected. Some problems I worked around, some I just had to live with.

PIP is the standard package installer for Python. It is documented here.

To get information about PIP.

The following command gives lots of data about PIP.

python3 -m pip debug –verbose

Valid wheel types

“Wheels” are used to create and install Python packages; see here.

For example what are the names of valid “wheel types on my system” (needed when creating a package). Amongst the data this gave, was

Compatible tags: 30
py3-none-any
py37-none-any

This means PIP install will accept a package with

pymqi-1.12.0-py3-none-any.whl

A package like pymqi-1.12.0-py3-none-zos.whl is not in the list and will not install.

Configuration parameters

python3 -m pip debug –verbose

Configuration data is stored in several places

global:
  /etc/xdg/pip/pip.conf
  /etc/pip.conf
site:
  /usr/lpp/IBM/cyp/v3r8/pyz/pip.conf
user:
  $HOME/.pip/pip.conf
  $HOME/.config/pip/pip.conf

The documentation said $VIRTUAL_ENV/pip.conf will be used. $VIRTUAL_ENV was set on my system, but did not show up in the list.

Initially my $HOME was /u, and this caused problems as my userid was not authorised to write to /u/.pip…

Check your $HOME is set to an appropriate value.

Show the configuration files: python3 -m pip config debug

env_var:                                                                   
env
global:                                                      
  /etc/xdg/pip/pip.conf, exists: False                       
  /etc/pip.conf, exists: False                               
site:                                                        
  /usr/lpp/IBM/cyp/v3r8/pyz/pip.conf, exists: False          
user:                                                        
  /u/tmp/pymqi2/.pip/pip.conf, exists: False                 
  /u/tmp/pymqi2/.config/pip/pip.conf, exists: False          

python3 -m pip config list

gave me

[33]WARNING: The directory ‘/u/.cache/pip’ or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.-[0]

because HOME was not pointing to a value writeable directory.

Updating configuration:

Once I had set HOME to a valid value, I could set configuration values.

  • python3 -m pip config –user set user.colin yes
  • python3 -m pip config –user set site.colin yes

gave me

Writing to /u/tmp/pymqi2/.config/pip/pip.conf

This file had

[site]
colin = yes

[user]
colin = yes

python3 -m pip config –global set user.colin yes

gave me ( as expected)

Writing to /etc/pip.conf
[31]ERROR: Unable to save configuration. Please report this as a bug.
PermissionError: [Errno 111] EDC5111I Permission denied.: ‘/etc/pip.conf’

I would need use a suitably authorised userid to do this.

To edit a config file

You need to specify the editor to use

python3 -m pip config –user –editor /bin/oedit edit

Python on z/OS: Creating a pure Python package.

I had problems creating a package with both Python code, and a extension module written in c (shipped as a load module .so object in Unix Services). The problems were that the package name was dependant on the name of the hardware and the level of the operating system. To produce a package for z/OS in general, I would need to build on every z/OS hardware.

Removing the C code made it much easier to package.

The setup.py file for pure Python

import setuptools 
from setuptools import setup, Extension 
import sysconfig 
bindings_mode = 1 
version = '1.12.0' 
setup(name = 'pymqi', 
    version = version, 
    description = 'Python IBM MQI Extension for IBM MQ.', 
    long_description= 'PyMQI is a Python library ', 
    author='Dariusz Suchojad', 
    author_email='pymqi@m.zato.io', 
    url='https://dsuch.github.io/pymqi/', 
    download_url='https://pypi.python.org/pypi/pymqi', 
    platforms='OS/390', 
    package_dir = {'': 'code'}, 
    packages = ['pymqi'], 
    license='Python Software Foundation License', 
    keywords=('pymqi IBM MQ WebSphere WMQ MQSeries IBM middleware'), 
    python_requires='>=3', 
    classifiers = [ 
        'Development Status :: 5 - Production/Stable', 
        'License :: OSI Approved :: Python Software Foundation License', 
        'Intended Audience :: Developers', 
        'Natural Language :: English', 
        'Operating System :: OS Independent', 
        'Programming Language :: C', 
        'Programming Language :: Python', 
        'Topic :: Software Development :: Libraries :: Python Modules', 
        'Topic :: Software Development :: Object Brokering', 
        ], 
    py_modules = ['pymqi.CMQC', 'pymqi.CMQCFC', 'pymqi.CMQXC', 'pymqi.CMQZC'], 
    ) 

Where the py_modules were in ./code/pymqi/ as CMQC.py etc.

Install the wheel package and build it

python3 setup.py bdist_wheel

This gave a package

./dist/pymqi-1.12.0-py3-none-any.whl

Which can be installed on any z/OS system.

setup tools (and so bdist_wheel) has a web page here.

Uninstall it

I used

python3 -m pip –verbose uninstall wheel

to uninstall it

List what is installed

python3 -m pip –verbose list

This gave

Package      Version Location                                                   Installer          
------------ ------- ---------------------------------------------------------- ---------          
cffi         1.14.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
cryptography 2.8     /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
ebcdic       1.1.1   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
numpy        1.18.2  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pip          20.2.1  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
pycparser    2.20    /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
setuptools   47.1.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages pip                
six          1.15.0  /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    
wheel        0.37.1  /u/tmp/pymqi2/lib/python3.8/site-packages                  pip                
zos-util     1.0.0   /Z24C/usr/lpp/IBM/cyp/v3r8/pyz/lib/python3.8/site-packages                    

so we can see the wheel package was installed in my user directory.

Without the –verbose just package and version were displayed, no location or installer.