Have a good REST and save a fortune in CPU with Python

Following on from Have a good REST and save a fortune in CPU. The post gives some guidance on reducing the costs of using Liberty based servers from a Python program.

Certificate set up

I used certificate authentication from Linux to z/OS. I used

  • A certificate defined on Linux using Openssl.
  • I sent the Linux CA certificate to z/OS and imported it to the TRUST keyring.
  • I created a certificate on z/OS and installed it into the KEY keyring.
  • I exported the z/OS CA, sent it to Linux, and created a file called tempca.pem.

Python set up

Define the names of the user certificate private key, and certificate

cf=”colinpaicesECp256r1.pem”
kf=”colinpaicesECp256r1.key.pem”
cpcert=(cf,kf)

Define the name of the certificate for validating the server’s certificate

v=’tempca.pem’

Set up a cookie jar to hold the cookies sent down from the server

jar = requests.cookies.RequestsCookieJar()

Define the URL and request

geturl =”https://10.1.1.2:9443/ibmmq/rest/v1/admin/qmgr/

Define the headers

import base64
useridPassword = base64.b64encode(b’colin:passworm’)
my_header = {
‘Content-Type’: ‘application/json’,
‘Authorization’: useridPassword,
‘ibm-mq-rest-csrf-token’ : ‘ ‘
}

An example flow of two requests, using two connections

For example using python

s = requests
response1 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert)
response2 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert)

creates two session, each has a TLS handshake, issue a request, get a response and end.

An example of two requests using one session

For example using python

s = requests.Session()
response1 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert1)
response2 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert2)

The initial request has one expensive TLS handshake, the second request reuses the session.

Reusing this session means there was only one expensive Client Hello,Server Hello exchange for the whole conversation.

Even though the second request specified a different set of certificates, the certificates from when the session was established, using cpcert1 were used. (No surprise here as the certificates are only used when the session is established).

For the authentication, in both cases the first requests received a cookie with the LtpaToken2 cookie in it.

When this was passed up on successive requests, the userid information from the first request was used.

What is the difference?

I ran a workload of a single thread doing 200 requests. The ratios are important, not the absolute values.

Shared sessionOne session per requests
TCP flows to server1 11
CPU cost1 5
Elapsed time16

Installing python on z/OS.

I wanted to use Python on z/OS to run a REST workload. It took a bit of time, but it worked with no major problems.

As well as installing Python, I had to install some packages so I could use the request package to issue REST requests.

This page has a good article about using python on z/OS. I used it to install additional packages.

This post has topics

There is a Python from Rocket software, and one from IBM. They may be the same; I used the IBM version.

There are instructions here. I logged on to my usual IBM web site, where my usual browser remembered my userid and password, then used https://www.ibm.com/products/open-enterprise-python-zos/pricing .

I used the web site https://www.ibm.com/docs/en/python-zos/3.9?topic=configuration-installing-configuring-pax-edition.

When I used the web site directly, I had to enter my userid and password.

The web site has a .pax file, and instructions.

The HAMB390.runnable.pax.Z file used 492418 * 512 byte blocks on z/OS.

I did not have enough space in my ZFS on z/OS, so I had to allocate a new ZFS

Allocate a ZFS

//DEFINE EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEFINE –
CLUSTER –
(NAME(COLIN.ZFS1 ) –
VOLUMES(USER00) –
LINEAR –
MEGABYTES(254 25) –
SHAREOPTIONS(3 3))
/*

Format it

//FORMATFS EXEC PGM=IOEAGFMT,REGION=0M,COND=(0,NE,DEFINE),
// PARM=(‘-aggregate COLIN.ZFS1 -compat’)
//SYSPRINT DD SYSOUT=*
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//*

Mount it

//MOUNT EXEC PGM=IKJEFT1A,COND=((0,NE,DEFINE),(0,NE,FORMATFS))
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
MOUNT FILESYSTEM(‘COLIN.ZFS1’) TYPE(ZFS) +
MOUNTPOINT(‘/colin2’) MODE(RDWR) PARM(‘AGGRGROW’) AUTOMOVE
/*

If you want to use this ZFS after an IPL you will have to mount it. If you only want to use the ZFS as a temporary ZFS you can unmount and delete it.

I created another ZFS for the unpacked Python with size MEGABYTES(550 50).

I followed the install instructions https://www.ibm.com/docs/en/python-zos/3.9?topic=configuration-installing-configuring-pax-edition using the directory /colin3.

After I had unpacked it, there was a directory usr/lpp/IBM/cyp/v3r9 with members IBM and pyz.

The path for python is /colin3/usr/lpp/IBM/cyp/v3r9/pyz/bin

You may want to consider setting up a symbolic link from /usr/lpp/pyz to the new libraries. From an authorised userid issue

ln -s /colin3/usr/lpp/IBM/cyp/v3r9/pyz /usr/lpp/pyz

Set up a profile

I set up a pythonprof file with

export LIBPATH=/usr/lpp/pyz/lib:$LIBPATH
export _BPXK_AUTOCVT=’ON’
export _CEE_RUNOPTS=’FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)’
# if you are using a bash shell, you should also set these:
export _TAG_REDIR_ERR=txt
export _TAG_REDIR_IN=txt
export _TAG_REDIR_OUT=txt
export PATH=/usr/lpp/pyz/bin:$PATH

When I wanted to run a python profile, I use . ./pythonprof to do the set up.

I then checked python worked using

/usr/lpp/pyz/bin/python3 –version

Which gave me

Python 3.9.2

Install the request package and co requisites

I used the information here to install additional packages so I could use a rest interface from python. I have copied the relevant section and expanded it.

If you want to use private keys with a password you need to use package requests_pkcs12. Use Pip download requests_pkcs12 instead of download requests below.

Our z/OS service isn’t connected to the internet, so we did it like this:

On a laptop with Python and pip installed:

a. mkdir requests
b. cd requests
c. pip download requests

This pulled the “requests” package and four dependencies: The files end in .whl

1. certifi
2. chardet
3. idna
4. urllib

Binary FTP all five to a USS directory (/u/myid/pipdl)

On z/OS, in USS in the directory we uploaded to, install all of the packages using:

. /u/adcd/pythonprof
pip3 install requests --find-links ./

Note lots of error messages as pip tries to access the internet version, but then success at the end.

You can now erase the .whl files.

I ran my python programs – and they worked!