With Opentelemetry, as work flows through a system, it sends “I am here” data back to a collector, which can process it, and pass it on to other tools to display the information on graphical dashboards.
I spent several days trying to work out why I was not getting any telemetry data from my queue managers.
Basic setup
Check SMF
The telemetry data is written to SMF. You should check it is being collected. The command D SMF gives output like
IFA714I 08.45.27 SMF STATUS FRAME LAST F E SYS=VS01
LOGSTREAM NAME BUFFERS STATUS
A-IFASMF.DEFAULT 0 CONNECTED
A-IFASMF.COLIN 0 CONNECTED
A-IFASMF.INMEM 1289792 IN-MEMORY
A-IFASMF.MQOTEL 393304 IN-MEMORY
A-IFASMF.T1159 0 IN-MEMORY
This has some data in the buffers, so it looks like data is being produced.
You can check the SMF option using the command D SMF,O
This gives a lot of information including
INMEM(IFASMF.T1159,TYPE(1159),RESSIZMAX(0128M)) -- PARMLIB
INMEM(IFASMF.MQOTEL,TYPE(1158),RESSIZMAX(0128M)) -- PARMLIB
INMEM(IFASMF.INMEM,TYPE(30),RESSIZMAX(0128M)) -- PARMLIB
This shows the MQ SMF records, type 1158 are mapped to name IFASMF.MQOTEL. It may be different on your system.
Check the data gatherer is active
This is a Java program which runs on z/OS and writes to an HTTP connection. I run mine as a started task.
This program does not report any statistics, if it is using CPU, then is may be processing records.
Check the network traffic
I use TLS encryption on my connection to the Opentelemetry collector server running on Linux. Using tools like wireshark on the connection allow you to see the overall traffic, but not the content of the traffic. For example
Shows there is traffic to and from port 4317 on my Linxu box.
- The records with length of 8258 contains my Otel traffic
- There is also TCP/IP Keep Alive flows every 15 seconds or so.
On the Linux Opentelemetry collector
During setup, I had the collector write to debug, and to a file, so I could see the traffic coming down.
But no data is being produced.
There are two switches that need to be enabled.
The Opentelemetry state is sent as a message property within MQ messages.
There is a property traceparent, which identifies the high level piece of work. The data is of the format
- trace identifier ‘-‘ span identifier ‘-‘ flags
Where flags is a two byte character string such as 01.
Format of the flags
- If the rightmost bit of the flags is 0, then this signals do not collect any data.
- If the rightmost bit of the flags is 1, then do more checks.
- If the queue has an attribute OTELTRAC(ON) (either directly, or as specified at the QMGR level), then emit the otel data
- Else do nothing.
The flag in the traceparent is called SAMPLED, which I found confusing. I didn’t want to sample – I wanted OTEL information for all records!
I had the flags specified as 00 and did not get any output. When I changed it to 01 I got the OTEL data – it was as easy as that.
