As part of writing a GTF trace formatter in Python I needed to covert a STCK value to a printable value. I could do it in C – but I did not find a Python equivalent.
from datetime import datetime # Pass in a 8 bytes value def stck(value): value = int.from_bytes(value) t = value/4096 # remove the bottom 12 bits to get value in micro seconds tsm = (t /1000000 ) - 2208988800 # // number of seconds from Jan 1 1970 as float ts = datetime.fromtimestamp(tsm) # create the timestamp print("TS",tsm,ts.isoformat()) # format it
I created a file in Unix System Services, and FTPed it down to my Linux box. I could edit it, and process it with no problems, until I came to read in the file using Python.
Python gave me
File “<frozen codecs>”, line 322, in decode UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xb8 in position 3996: invalid start byte
The Linux command file pagentn.txt gave me
pagentn.txt: ISO-8859 text
whereas other files had ASCII text.
I changed my Python program to have
with open(“/home/colinpaice/python/pagentn.txt”,encoding=”ISO-8859-1″) as file:
and it worked!
I browsed the web, and found a Python way of finding the code page of a file
I’ve been using xlsxwriter from Python to create data graphs of data, and it is great.
I had problems with adding more data to a graph, and found some aspects of this was not documented. This blog post is to help other people who are trying to do something similar.
I wanted to produce a graph like
The pink colour would normally be transparent. I have coloured it to make the explanation easier.
This shows the OMS team worked from 0900 to 1600 on Monday, and SDC worked from 10am to 12am and for a few minutes on the same day around 1700. I wanted the data to be colour coded, so OMS was brown and SDC was green.
It took a while to understand how the data should be laid out in the table
A
B
C
D
E
F
G
1
OStart1
ODur 1
SStart 1
SDur1
SInt2
SDur2
2
OMS Tue
9.0
7.0
3
OMS Wed
17.0
2.0
4
SDC Wed
10.0
2.0
7.0
0.1
Where the data
OSTart1 is the time based on hours for OMS
ODur1 is the duration for OMS, so on Wed, the time was from 1700 to 1900, and interval of 2.0 hours
SStart1 is the start time of the SDC Wed item
SDur1 is the duration of the work. The work was from 1000 to 1200
SInt2 is the interval from the end of the work to the start of the next work. It is not the start time. It is 1000 + interval of 2 hours + interval of 7 hours, or 1900
SDur2 is the duration from the start of the work. It ends at 1000+ 2 hours + 7 hours + 0.1 hours
Define the data to the chart
To get the data displayed properly I used add_series to define the data.
Categories (The labels OMS Tue, OMS Wed, SDC Wed): You have to specify the same categories for all of your data. For me, range A2:A5. Using add_series for the OMS data, and add_series for the SDC data did not display the SDC data labels. This was the key to my problems.
You define the data as columns. The first column is the time from midnight. I have coloured it pink to show you. Normally this would be fill = [{‘none’ : true } ] You use
fill = [{'color': "pink"}] # fill = [{'none': True}] chart.add_series({ 'name': "Series1, 'categories': ["Hours",1,0,4,0], 'values': ["Hours",1,1,4,1], 'fill': fill })
This specifies categories row 1, column 0 to row 4, column 0, and the column of data row 1, column 1, to row 4, column 1. (Column 0 is column A etc.)
For the second column – the brown, you use
fill = [{'color': "brown"}] chart.add_series({ 'name': "Series2, 'categories': ["Hours",1,0,4,0], 'values': ["Hours",1,2,4,2], 'fill': fill })
The categories stays the same, the superset of names.
The “values” specifies the column of data row 1, column 2, to row 4, column 2.
Because the data for SDC is missing – this is not displayed.
For the SDC data I used 4 add_series request. The first one
name:Series3
‘categories’: [“Hours”,1,0,4,0], the same as for OMS
values: row 1,column 3 to row 4 column 3
I then repeated this for columns (and Series) 4,5,6
This gave me the output I wanted.
I used Python lists and used loops to generate the data, so overall the code was fairly compact. The overall result was
I found cURL a good way of using the mq REST API, but I wanted to do more. cURL depends on a package called libcurl, which can be used by other languages.
Python seemed the next obvious place to look.
As I have found out, using digital certificates for authentication is hard to set up, and using signed certificates is even harder. As I had done the hard work of setting up the certificates before I tried curl and Python, the curl and Python experience was pretty easy.
I looked at using the Python “request” package. This allows you to specify most of the parameters that libcurl needs, except it does not allow you to specify the password for the user’s keystore.
I then looked at the Python package pycurl package. This is a slightly lower level API, but got it working in an hour or so.
My whole program is below.
During the testing I got various errors, such as “77”. These are documented here.
The messages were clear, for example
CURLE_SSL_CACERT_BADFILE (77) Problem with reading the SSL CA cert (path? access rights?).
Which was enough to tell me where to look.
All the things you can do with curl, you can do with pycurl.
# program - based on code in http://pycurl.io/docs/latest/quickstart.html
import sys
import pycurl
from io import BytesIO
# header_function take from http://pycurl.io/docs/latest/quickstart.html
headers = {}
def header_function(header_line):
# HTTP standard specifies that headers are encoded in iso-8859-1.
header_line = header_line.decode('iso-8859-1')
# Header lines include the first status line (HTTP/1.x ...).
# We are going to ignore all lines that don't have a colon in them.
# This will botch headers that are split on multiple lines...
if ':' not in header_line:
return
# Break the header line into header name and value.
name, value = header_line.split(':', 1)
print("header",name,value)
home = "/home/colinpaice/ssl/ssl2/"
ca=home+"cacert.pem"
cert=home+"testuser.pem"
key=home+"testuser.key.pem"
cookie=home+"cookie.jar.txt"
url="https://127.0.0.1:9443/ibmmq/rest/v1/admin/qmgr/QMA/queue/CP0000?attributes=type"
buffer = BytesIO()
c = pycurl.Curl()
print("C=",c)
try:
# see option names here https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
# PycURL option names are derived from libcurl
# option names by removing the CURLOPT_ prefix.
c.setopt(c.URL, url)
c.setopt(c.WRITEDATA, buffer)
c.setopt(pycurl.CAINFO, ca)
c.setopt(pycurl.CAPATH, "")
c.setopt(pycurl.SSLKEY, key)
c.setopt(pycurl.SSLCERT, cert)
c.setopt(pycurl.SSL_ENABLE_ALPN,1)
c.setopt(pycurl.HTTP_VERSION,pycurl.CURL_HTTP_VERSION_2_0)
c.setopt(pycurl.COOKIE,cookie)
c.setopt(pycurl.COOKIEJAR,cookie)
c.setopt(pycurl.SSLKEYPASSWD , "password")
c.setopt(c.HEADERFUNCTION, header_function)
# c.setopt(c.VERBOSE, True)
c.perform()
c.close()
except Exception as e:
print("exception :",e )
finally:
print("done")
body = buffer.getvalue() # Body is a byte string.
# We have to know the encoding in order to print it to a text file
# such as standard output.
print(body.decode('iso-8859-1'))
ColinPaice 