Setting up Python on z/OS

Running a Python package on z/OS is pretty easy. You just have to remember to set up the environment. This one of those little tasks you do not do very often, and forget how to do it.

When you enter OMVS your .profile shell is executed (if the file exists).

Does the file exist ?

You can use the ls command to see if the file exists

ls -altr ~/.profile

where the -a option says display all the files – including those beginning with ‘.’. By default these are omitted.

This gave

-rwx------ 1 BPXROOT TSOUSER 297 May 18 08:46 /u/colin/.profile

Note the x attribute; it means the file can be executed. You can use the following command to set this attribute.

chmod +x ~/.profile

Profile contents

My .profile file has

# set up Python path 
export PATH=/usr/lpp/IBM/cyp/v3r8/pyz/bin:$PATH 
# set up MQ Libraries
export MQ=MQM.V900 
export STEPLIB=$MQ.SCSQANLE:$MQ.SCSQANLE:$MQ.SCSQAUTH:$STEPLIB 
# set up PYTHONPATH to point to where zpymqi is
export PYTHONPATH=/u/colin/:$PYTHONPATH 
      

When my Python program has import pymqi, Python will look in directory /u/colin/pymqi

Getting PyDev to work with IBM Developer for z/OS

PyDev is an integrated Development Environment for Python which works in Eclipse. It has all the capabilities you expect with an IDE.

I had problems getting PyDev to work on IBM Developer for z/OS ( ID/z). Gerald Mitchell of IBM sent me instructions which worked perfectly – so thanks to him for them. I’ve modified them slightly.

Note / Disclaimer: PyDev is not currently officially supported by the IBM Developer for z/OS product. If you have problems, IBM will not be able to help you.

The problems I had were that PyDev installed, but did not display within Eclipse. Window-> Preferences did not show PyDev, and so could not be used.

The latest PyDev requires Java 11, but ID/z in Java is at V8. PyDev 8.2.0 from February 2021 works, see PyDev Releases.

PyDev 8.2.0 is on the PyDev GitHub for install at PyDev 8.2.0 Release · fabioz/Pydev

I installed Pydev using “Install new software”. Add site

 This displayed a check box for        

  • PyDev for Eclipse 8.2.0.202102211157
  • PyDev for Eclipse Developer Resources 8.2.0.202102211157
  • PyDev Mylyn Integration 0.6.0

I selected the first one.

I got a pop up for a Security warning for unsigned content, I said “install anyway.”

I restarted the work bench. The restart may take a while as the plugins and features are integrated.

The Window -> Preferences should have PyDev in the list.

You can check the file associations now…

  • Window -> Preferences -> General -> Editors -> File Associations should have a File types: entry for *.py and the Associated Editors: section should list Python Editor as one of the options.
  •  Window -> Preferences -> PyDev should also be populated with preferences for Editor, Interpreters, PyUnit, and Scripting PyDev.
  • If you don’t set up Python via the preferences, on the first python file interaction, you may get a message that Python interpreter is not currently configured.

You can edit Python files inside the base IDz install with some of the generic file editors already installed, though they will not have Python specific language knowledge or capabilities such as content assist or syntax highlighting

To allow .py files to be edited internally, by setting the Window -> Preferences -> General -> Editors -> File Associations. Here you can choose to change the Open associated files with: option from System Editor; if none Text Editor to Text Editor as one option.
Another option is in the File types: section press Add… and then in the dialog that comes up type *.py and then press OK. (If you get a dialog message that says a file type *.py already exists that is OK: this means PyDev registered as an editor which is fine. )
In the Associated editors: section press Add… and select the Internal editors option. I would recommend then choosing Generic Text Editor, Text Editor, System z LPEX Editor, or z Systems LPEX Editor but there are several other options that may work for you. Note that you can Add… more than one editor to a a single file type, at which point if you right click on the *.py files you will have an option of which editor to use.
The one that is used when you double click or right click -> Open is the one you set to default, which you can adjust in this section by selection of one of the editors and then pressing the Default button.
Then press Apply and Close.

Example of editing a .py file

and

Running the debugger

I was not able to use the debugger as it tries to run it on my work station, not on z/OS

I didn’t know that z/OS command

On one of the z/OS fora I found the following good stuff from David Mingee. Three of these were new to me.

  1. After using START command to start more than one screen, you can enter =XALL to close all screens.
  2. In ISPF enter CMD SWAPBAR ON.  This will cause a screen name to be displayed at the bottom left of your screen.  There will be a screen name for each START CMD that has been done.  The active screen will have an * e.g. *dslist .  You can go directly to any screen listed by clicking your mouse on the desired screen name.  This feature stays turned on for your next logon and until you enter SWAPBAR OFF. You can change the name of the screen using the command such as SCRNAME SDSF. By default you get names like
    1. ISR@PRIM for the main ISPF panel
    1. ISFPCU4 for SDSF
    2. DSLIST for ISPF 3;4
    3. ISRTSO for ISPF option 6 for TSO
    4. OS390S for z/OS System Programmer Primary Option Menu
  3. When editing a file do CC CC A to copy a range of lines AFTER line A, a new feature allows the use of CC CC AK  on as many lines as wanted and then A on the last location for the copy.  Lines are copied to multiple locations with one command. So do cc, cc, ak,ak,a
  4. In ISPF use command 0 for settings then go to page 2 and change 3278 screen display to 1 for DATA vs. STD. This will automatically allow your screen to display up to 120 columns vs. only 80.
  5. In SDSF change FIND LIMIT to 99999 vs default of 5000 as this will speed up log searches.
  6. Use SRCHFOR command in 3.4 for pds/pdse files to find a ‘STRING’ in all members.
  7. Use KEYS command to change your PF KEY settings, e.g. change F9 to SWAP NEXT vs. SWAP.
  8. Use ISPF APPEND command to create a personal DSLIST of files frequently used with different DSN Qualifiers.
  9. ISPF cut and paste. You can have multiple “clipboards”, and do ASCII or EBCDIC conversion. For example CUT .a .b BOARD2 APPEND UTF8 copies the lines labelled .a to .b to clipboard BOARD2.

Compiling 64 bit C programs (with MQ) – and using dataset aliases

I wanted to compile one of the MQ samples to run as a 64 bit program. The sample comes compiled as 31 bit.

How did I compile the sample to make it 64 bit?

There is documentation on building 64 bit applications; it provides a compile and link JCL example. However, I wanted to do a compile, bind and go.

//COLINCC JOB 1,MSGCLASS=H
//JOBLIB JCLLIB ORDER=(PP.CBC.ZOS204.SCCNPRC)
//COMPILE EXEC PROC=EDCQCBG,  Needed for 64 bit compile
//   LIBPRFX=CEE,   This is an alias 
//   LNGPRFX=PP.CBC.ZOS204,
//   BPARM='MAP,XREF,RENT,DYNAM=DLL',
//   CPARM='OPTFILE(DD:SYSOPTF)',
//   GPARM='M801 CP0000 1'
//COMPILE.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEEH.SYS.H
//   DD DISP=SHR,DSN=&LIBPRFX..SCEEH.H
//   DD DISP=SHR,DSN=MQM.V900.SCSQC370
//COMPILE.SYSIN DD DISP=SHR,DSN=BETACP.MP1B.JCL(CSQ4BCG1)
//COMPILE.SYS DD DISP=SHR,DSN=&LIBPRFX..SCEEH.SYS.H
//COMPILE.SYSOPTF DD *
WARN 64 RENT LP64 LO SOURCE LIST
NOMARGINS EXPMAC SHOWINC XREF
LANGLVL(EXTENDED) SSCOM DLL
DEFINE(_ALL_SOURCE)
/*
//BIND.SCSQDEFS DD DISP=SHR,DSN=MQM.V900.SCSQDEFS
//BIND.SYSIN DD *  
  INCLUDE SCSQDEFS(CSQBMQ2X)
//GO.STEPLIB   DD DISP=SHR,DSN=MQM.V900.SCSQANLE
//   DD DISP=SHR,DSN=MQM.V900.SCSQLOAD
//   DD DISP=SHR,DSN=MQM.V900.SCSQAUTH

Using ALIASes

The above JCL used

  • an alias (CEE) for some libraries. The CEE alias was defined as SYMBOLIC PP.ADLE370.&SYSLEVEL and RESOLVED PP.ADLE370.ZOS204
  • explicit High Level Qualifier PP.CBC.ZOS204 for other libraries.

If your system provides aliases for data sets, it is better to use the alias, rather than the explicit data set name (or HLQ). When the system is upgraded (to ZOS205), the symbolic will be updated from ZOS204 to ZOS205, the alias will be updated automatically, and any JCL using the alias will still work. If you used the explicit dataset name, when your system is upgraded the dataset may not be available, and you need to update your JCL.

What does an alias point to?

The easiest way I found to see what an alias points to use ISPF 3.4 to list datasets beginning with the alias, for example CEE.SCEEH, and browse it. This browsed dataset was PP.ADLE370.ZOS204.SCEEH, so the alias CEE represents PP.ADLE370.ZOS204.

You can also use a LISTCAT command. (Use ISPF 3.4 and use the I prefix command) and change the command to LISTCAT ENTRIES(CEE.* ) ALL then type EXEC on the command line to execute it.

This gives output like

LISTCAT ENTRIES(CEE.* ) ALL
...
ALIAS --------- CEE.SCEEBIND                                                      
     IN-CAT --- ICFCAT.PLEXH.CATALOG3                                             
     HISTORY                                                                      
       RELEASE----------------2     CREATION--------0000.000                      
     ENCRYPTIONDATA                                                               
       DATA SET ENCRYPTION-----(NO)                                               
     ASSOCIATIONS                                                                 
       SYMBOLIC-PP.ADLE370.&SYSLEVEL..SCEEBIND                                    
       RESOLVED-PP.ADLE370.ZOS204.SCEEBIND                                        

Problems with mqinqmp

I struggled to get MQ in Python to work with Message properties when running on z/OS. I eventually found that mqinqmp() in 64 bit mode was not always working (31 bit mode was fine).

Also the documentation for mqinqmp() is not very clear.

You pass to mqinqmp(), a property name (or a generic like %), a buffer, the size of the buffer, the length of the data, and some other parameters (such as get first, or get next). It returns the property name, the property value ( possibly truncated) , and the true length of the property..

If your buffer is too small to contain all of the property, you get reason code MQRC_PROPERTY_VALUE_TOO_BIG, and should get the true length of the property, its name, type, and length.

This worked fine when it was a 31 bit program. When I ran it as a 64 bit program, it did not return the true length, and did not return the property name.

The documentation says

DataLength Type: MQLONG – output

This is the length in bytes of the actual property value as returned in the Value area.

If DataLength is less than the property value length, DataLength is still filled in on return from the MQINQMP call. This allows the application to determine the size of the buffer required to accommodate the property value, and then reissue the call with a buffer of the appropriate size.

The DataLength is the actual length of the property. (The value returned in the Value area is truncated at Value Length).

What should happen

After you get the MQRC_PROPERTY_VALUE_TOO_BIG reason code, you reallocate the buffer with a size of DataLength and retry the request. It should work the second time.

For me the DataLength was 0, so every time around the loop it got MQRC_PROPERTY_VALUE_TOO_BIG.

What can I use to edit program source in Unix Services on z/OS?

I hit this question trying to edit some Python programs on z/OS in Unix Services.

ISPF services

The easy answer for programs which are in EBCDIC or untagged, is just use ISPF edit, and you can use the HILITE command to highlight the constants and variables in the file. You can use this command to highlite, C, REXX, JCL, XML, assembler etc.

Example output for a C program.

Unfortunately although python supports EBCDIC source files, if you want to compile and package these files, they need to be ASCII files. The Unix Services command OEDIT will detect that the file as an ASCII file, and you can edit it, but not use the HILITE command.

To display the type of the file, use the Unix command ls -Tr *.py . This gave me

t ISO8859-1   T=on  cf.py        
- untagged    T=off aa.py        

Where cf.py is tagged as an ASCII file, an aa.py is an untagged file which defaults to EBCDIC.

Download to your work station

You can download ( FTP) these files to your work station, in binary and edit them. I use gedit on Linux. This has support built in for highlighting source.

IBM developer for z/OS.

This is an eclipse based package which allows you to edit data sets, submit jobs, debug programs etc. It has build in support for C, COBOL etc. When you right click on a file, you can select which editor to use, for example LPEX, a simple editor,a remote C/C++ editor, or an editor on your work station (gedit in my case).

For a python file (*.py), Eclipse downloads the file, and the choice is the eclipse simple text editor, or your workstation standard editor (gedit).

Pydev

Pydev is an Integrate Development Environment for Python on Eclipse (IBM Developer for z/OS). I had a few problems getting it working in Eclipse. (I had to use Pydev 8.2 because of incompatible Java release problem). This provides a Python editor with highlighting, and support for “twisties” where you can hide functions etc. As an editor this worked fine for most of the things I did. There were a couple of surprises.

I could not get the debugger to work, as it tried to debug running the Python program on my laptop – not on z/OS.

Whoops who over wrote my file

With all of the non ISPF techniques there is an integrity problem. If there are two people A and B and they want to edit the file at the same time.
With ISPF and ISPF

  • A edits the file
  • B is prevented from editing the file
  • A saves the updates

If A uses ISPF and B uses IBM developer

  • A edits the file
  • B edits the file
  • A saves the file
  • When B tries to save the file, it detects the last changed time stamp is different, and asks if you wan to overwrite it

If A uses ISPF and B uses IBM developer

  • A edits the file
  • B edits the file
  • B saves the file, and as the last changed time stamp has not changed, the save is successful
  • A saves the file, and overwrites the changes B made.

If you use FTP or similar to download the files, there is no locking, and the last person to save will overwrite any previous updates from other people.

Using filters on IBM Developer to limit what is displayed

I was using IBM Developer for z/OS (known as ID/z) to use Eclipse to access files on z/OS, and so use the Eclipse editors and other facilities. The files were displayed in a window, but did not have the handy PF7 and PF8 to scroll through the list. I also had a mixture of python source files, and other files.

I knew there was a way of filtering the displays, but it was a challenge to get this to work.

The easy bit – creating a filter

Under the Remote Systems (Window -> Show view -> Remote systems) it had my connections to z/OS systems, and a tree of my USS files.

On an subdirectory in the tree, right click -> New -> Filter. This gives

  • You can specify (or browse for) a subdirectory
  • You can either filer on file name of file type.

To filter for file type

You cannot type in the file type field. Click on Select… . You can either select from the existing list, or create your own extension. I created .py. This appears in the File types filter box as py, .

If Show files only is unticked, then the filter will displayed the directories below the path you specified as well as any matching files. If this is selected, then only the filtered file names are displayed

When you press Next it prompts you for a name.

When you get back to the Remote systems, window, you should see your filter in the list. If you open the twistie you will see the files you specified.

Changing the filter

This was initially hard because I was not familiar with the Eclipse usage.

Right click on the filter, -> Properties -> Filter Strings and you can change the information in the filter. Click Apply and Close to finish.

Extending the filter

As well as changing a filter, you can add new filter strings. Click on “New filter string”, fill in the details and select “Apply and Close”.

I could not see how to delete a filter once created.

Go into

Although you have filtered the list of files, you have all of the connections, and other displayed information. If you right click a directory, or filter and “go into” you get just that directory or filter output displayed.

Select the folder with an up arrow in it to get back.

Filter pools

You can have a hierarchy of filters, so I could have a set

  • MQ filters – with all of the filter I use for developing MQ applications
  • Python filters – with the filters I use for creating python program, and python unit tests

You can display or hide this through the twistie (menu button) at the top right of the Remote Systems window.

To create a new filter pool, ignore what the documentation says. Where you have “Z/OS UNIX files”, right click on it -> new -> Filter pool.

More information

You can display more information using “Window-> Show view -> Remote Systems -> team”

Using eclipse based tools to z/OS

Eclipse based tools like z/OS Explorer and IBM Developer for z/OS, use a server on the z/OS system called RSED.

This is available on the ZD&T (ZPDT) system, but is hidden away.

You need to mount the file system

mount filesystem(‘FEK320.HALG320.ZFS’) mountpoint(‘/usr/lpp/IBM/zexpl/’)
mode(read)type(ZFS)

Then start the server

s rsed

It’s easy when you know how.

ISPF has autocomplete – just like a laptop!

On Linux I can use “ls D” and tab, and it lists Documents/ and Downloads/ . If I type “ls Doc” and press tab it uses autocomplete to give “ls Documents/”. This is a great way to save you typing.

I’ve just found the same function for z/OS. It has been in the product since at least 2004!

  • Set up a pf key to be AUTOTYPE ( type KEYS on the command line) I set PF6 to this
  • Go to ISPF 3.4 and type the start of a data set, for example COLIN.
  • press PF6, it gives me COLIN.JCL, press PF6 again, it gives me COLIN.MP1B.JCL

Of course you could always type COLIN.* and press enter, then tab down the list. But I thought AUTOTYPE was pretty neat.

Setting PF Keys 13 to 24

When I went to set my PF key, I found all 12 keys were allocated. How do I set keys 13 to 24?
If you use PFSHOW TAILOR it allows you to use 12 or 24 keys, and to control what is displayed.

I have my terminal emulator set up so shift PF5 is PF17, and so I set PF17 to AUTOTYPE.

Easy.

Using point and shoot keys in ISPF

ISPF has had hot keys for many years – I have only just got round to using them.

At the top of ISPF screens there is usually an action bar. For example

   Menu  Utilities  Compilers  Options  Status  Help                       
 ────────────────────────────────────────────────────
                   ISPF Primary Option Menu                           
 Option ===>                                                                   

Where the actions are Menu, Utilities etc. The underlined letter is the hot key for the action.

To select Utilities, type U on the command line and press the “Actions” PF key. This will do the same as tabbing to the Utilities action bar item, and pressing enter. For me PF10 is set to Actions.

You can display your PF Key settings using the keys or pfshow command.

For people who like to use short cuts, have your command line at the top, and use the “home” key. This will get you to the command line. So you can do: home, U, PF10 and quickly get to the Utilities action bar item.

Who develops standard?

When the GUI standards came out I wondered why the standard was to put the command line at the bottom, as there was no “go to command line” key, you had to do “Home”, then shift, tab (to do back tab). This was too many keystrokes for me.