Python and mq REST api

I found cURL a good way of using the mq REST API, but I wanted to do more.  cURL depends on a package called libcurl, which can be used by other languages.

Python seemed the next obvious place to look.

As I have found out, using digital certificates for authentication is hard to set up, and using signed certificates is even harder.  As I had done the hard work of setting up the certificates before I tried curl and Python, the curl and Python experience was pretty easy.

I looked at using the Python “request” package.   This allows you to specify most of the parameters that libcurl needs, except it does not allow you to specify the password for the user’s keystore.

I then looked at the Python package pycurl package.    This is a slightly lower level API, but got it working in an hour or so.
My whole program is below.

During the testing I got various errors, such as “77”.  These are documented here. 

The messages were clear, for example

CURLE_SSL_CACERT_BADFILE (77) Problem with reading the SSL CA cert (path? access rights?).

Which was enough to tell me where to look.

All the things you can do with curl, you can do with pycurl.

 

# program - based on code in http://pycurl.io/docs/latest/quickstart.html
import sys
import pycurl

from io import BytesIO

# header_function take from http://pycurl.io/docs/latest/quickstart.html
headers = {}
def header_function(header_line):
# HTTP standard specifies that headers are encoded in iso-8859-1.

header_line = header_line.decode('iso-8859-1')

# Header lines include the first status line (HTTP/1.x ...).
# We are going to ignore all lines that don't have a colon in them.
# This will botch headers that are split on multiple lines...
if ':' not in header_line:
  return

# Break the header line into header name and value.
name, value = header_line.split(':', 1)
print("header",name,value)

home = "/home/colinpaice/ssl/ssl2/"
ca=home+"cacert.pem"
cert=home+"testuser.pem"
key=home+"testuser.key.pem"
cookie=home+"cookie.jar.txt"
url="https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=type"
buffer = BytesIO()
c = pycurl.Curl()
print("C=",c)
try:
  # see option names here https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
  # PycURL option names are derived from libcurl
  # option names by removing the CURLOPT_ prefix. 
  c.setopt(c.URL, url) 
  c.setopt(c.WRITEDATA, buffer) 
  c.setopt(pycurl.CAINFO, ca) 
  c.setopt(pycurl.CAPATH, "") 
  c.setopt(pycurl.SSLKEY, key) 
  c.setopt(pycurl.SSLCERT, cert) 
  c.setopt(pycurl.SSL_ENABLE_ALPN,1)
  c.setopt(pycurl.HTTP_VERSION,pycurl.CURL_HTTP_VERSION_2_0)
  c.setopt(pycurl.COOKIE,cookie) 
  c.setopt(pycurl.COOKIEJAR,cookie) 
  c.setopt(pycurl.SSLKEYPASSWD , "password") 
  c.setopt(c.HEADERFUNCTION, header_function)  
# c.setopt(c.VERBOSE, True)
  c.perform() 
  c.close()
except Exception as e: 
  print("exception :",e ) 
finally: 
  print("done") 
body = buffer.getvalue() # Body is a byte string. 
# We have to know the encoding in order to print it to a text file 
# such as standard output. 
print(body.decode('iso-8859-1'))

 

Baby Python scripts doing powerful work with MQ

I found PyMqi is an interface from Python to MQ. This is really powerful, and I’m am extending it to be even more amazing!

In this blog post, I give examples of what you can do.

  • Issue PCF commands and get responses back in words rather than internal codes ( so CHANNEL_NAME instead of 3501)
  • Saving the output of DISPLAY commands into files
  • Using these files to compare definitions and highlight differences.
  • Check these files conform to corporate standards.
  • Print out from the command event queue, and the stats event queue etc


I can use some python code to display information via PCF

  • # connect to MQ
  • qmgr = pymqi.connect( queue_manager,”QMACLIENT”,”127.0.0.1(1414)”)
  • # I want to inquire on all SYSTEM.* channels
  • prefix = b”SYSTEM.*”
  • # This PCF request
  • args = {pymqi.CMQCFC.MQCACH_CHANNEL_NAME: prefix}
  • pcf = pymqi.PCFExecute(qmgr)
  • # go execute it
  • response = pcf.MQCMD_INQUIRE_CHANNEL(args)

This is pretty impressive as a C program would take over 1000 lines to do the same!

This comes back with data like

  • 3501: b’SYSTEM.AUTO.RECEIVER’,
  • 1511: 3,
  • 2027: b’2018-08-16 ‘,
  • 2028: b’13.32.15′,
  • 1502: 50

which is cryptic even for experts because you need to know 3501 is the value of the type of data for “CHANNEL_NAME”.

I have some python code which converts this to..

  • ‘CHANNEL_NAME’: ‘SYSTEM.AUTO.RECEIVER’,
  • ‘CHANNEL_TYPE’: ‘RECEIVER’,
  • ‘ALTERATION_DATE’: ‘2018-08-16’,
  • ‘ALTERATION_TIME’: ‘13.32.15’
  • ‘BATCH_SIZE’: 50

for which you only need a kinder garden level of MQ knowledge to understand it. It converts 3501 to CHANNEL_NAME, and 3 into RECEIVER

With a few lines of python I can write this data out so each queue is a file on disk in YAML format.

A yaml file for a queue looks like

  • Q_NAME: TEMP
  • Q_TYPE: LOCAL
  • ACCOUNTING_Q: Q_MGR
  • ALTERATION_DATE: ‘2019-02-03’
  • ALTERATION_TIME: 18.15.52
  • BACKOUT_REQ_Q_NAME: ”

Now it gets exciting! (really)

Now it is in YAML, I can write small Python scripts to do clever things. For example

Compare queue definitions

  • from ruamel.yaml import YAML
  • import sys
  • yaml=YAML()
  • q1 = sys.argv[1] # get the first queue name
  • ignore = [“ALTERATION_DATE”,”ALTERATION_TIME”,
  • “CREATION_DATE”,”CREATION_TIME”]
  • in1 = open(q1, ‘r’) # open the first queue
  • data1 = yaml.load(in1) # and read the contents in
  • for i in range(2,len(sys.argv)): # for all of the passed in filenames
  • q2=sys.argv[i] # get the name of the file
  • in2 = open(q2, ‘r’) # open the file
  • data2 = yaml.load(in2) # read it in
  • for e in data1: # for each parameter in file 1
  • x1 = data1[e] # get the value from file 1
  • x2 = data2[e] # get the value from the other file
  • if not e in ignore: # some parameters we want to ignore
  • if x1 != x2: # if the parameters are different
  • print(q1,q2,”:”,e,x1,”/”,x2) # print out the queuenames, keywork and values

From this it prints out the differences

  • queues/CP0000.yml queues/CP0001.yml : Q_NAME CP0000 / CP0001
  • queues/CP0000.yml queues/CP0001.yml : OPEN_INPUT_COUNT 1 / 0
  • queues/CP0000.yml queues/CP0001.yml : MONITORING_Q Q_MGR / HIGH
  • queues/CP0000.yml queues/CP0001.yml : OPEN_OUTPUT_COUNT 1 / 0
  • queues/CP0000.yml queues/CP0002.yml : Q_NAME CP0000 / CP0002
  • queues/CP0000.yml queues/CP0002.yml : OPEN_INPUT_COUNT 1 / 0
  • queues/CP0000.yml queues/CP0002.yml : OPEN_OUTPUT_COUNT 1 / 0

I thought pretty impressive for 20 lines of code.

and another script -for checking standards

  • from ruamel.yaml import YAML
  • import sys
  • yaml=YAML()
  • q1 = sys.argv[1] # get the queue name
  • # define the variables to check
  • lessthan = {“MAX_Q_DEPTH”:100}
  • ne = {“INHIBIT_PUT”:”PUT_ALLOWED”,”INHIBIT_GET”: “GET_ALLOWED”}
  • in1 = open(q1, ‘r’) # open the first queue
  • data = yaml.load(in1) # and read the contents in
  • # for each element in the LessThan dictionary (MAX_QDEPTH), check with the
  • # data read from the file.
  • # if the data in the file is “lessthan” the value (100)
  • # print print out the name of the queue and the values
  • for i in lessthan: # just MAX_Q_DEPTH in this case
  • if data1[i] < lessthant[i] : print(q1,i,data[i],”Field in error. It should be less than <“,lessthan[i])
  • # if the values are not equal
  • for i in ne: # INHIBUT_PUT and #INHIBIT_GET
  • if data[i] != ne[i] : print(q1,i,data[i],”field is not equal to “,lt[i])

the output is

queues/CP0000.yml
MAX_Q_DEPTH 5000 Field in error. It should be < 100

Display command events

difference Q_NAME CP0000 CP0000 ALTERATION_DATE 2019-02-07 2019-02-11

difference Q_NAME CP0000 CP0000 ALTERATION_TIME 20.48.24 21.29.23

difference Q_NAME CP0000 CP0000 MAX_Q_DEPTH 4000 2000

With my journey so far – Python seems to be a clear winner in providing the infrastructure for managing queue managers.