Visualises the data – see the image at the top of the blog post
Can aggregates the data for Grafana to use
Visualise the data
By default it displays 20 entries
and shows information about the individual points
This shows there were two queue manager CSQ9 and MQPA, there were 8 spans (recorded data)the duration if each item, and the date. ( I think the date format is wrong .. you want hh:mm:ss – because you know what the day and date are!
If you click on the blue Trace Name – you get a detailed picture for that trace point of where the time was spent
There is a limited search capability. You can display data going back 5 minutes, last 15 minutes etc.
You can limit how many records are displayed.
Displaying 100 records over half an hour gave me
If you click on an outlier you get the details of that point
You can click on each Service & Operation and see all the attributes associated with the record. For example z/OS image name, message persistence, size of message etc.
The challenges with using Jaeger visualisation
What transaction is this for ?
I ran two different transactions at the same time. The data displayed was the same as above. I could not identify which trace record was for which transaction.
If you display the details, the “title” such as MQPA MQPUT RSERVER gives you a clue as to which transaction the data is for.
Whoops missed it
If you have a low transaction rate, then if you display the last 10 minutes, you may get all of the data.
If you are doing a high transaction rate, and need to go back half an hour, there may be so much data you cannot find the entries of interest.
I’ve spend a few weeks trying to generate and use Opentelemetry data. My salesman’s vision does not match the practice. Below is what I have learned from hands on. It shows the typical usage of displaying Opentelemetry data using Jaeger and Grafana. What I say may be wrong, if so please tell me and I’ll correct it.
As work moves through a system it reports where it is. A central collector takes this data and can display where the work item spent its time.
This shows an application putting a message to an MQ queue, on queue manager MQPA, flowing through to a server on queue manager on CSQ9, and a response flowing back.
The overall transaction time was 71 ms.
The architecture
On z/OS the OTEL data is written to SMF. There is a Java application which runs on z/OS, which reads from SMF, and sends it to an Opentelemetry collector (running on my Linux laptop).
The Opentelemetry collector generates data in the form for “standard” packages Prometheus and Grafana to process, and display dashboards of the information.
There are two data models for sending data between components
The Opentelemetry collector has a push model. The recipient is a web server, and the OTEL collector sends JSON data over a POST request to the HTTP server.
Grafana has a pull model. It periodically sends a request to its providers saying “send me data on ….”
Prometheus has a database, which has a web server for capturing the data from the Opentelemetry client. Prometheus saves the timestamp data efficiently. Prometheus provides another web server to respond to the “send me data …” from Grafana.
Prometheus accepts metric data from sources like Opentelemetry, but it does not take the raw Opentelemetry “log” data directly into Prometheus – I think of Prometheus as a database.
Grafana takes data from many sources (such as Prometheus) and displays dash boards, which typically are time sequences of data. See below.
There is another component Jaeger, which converts the data from OTEL into data suitable for Grafana.
I expect people will use Grafana for overall monitoring, and Jaeger to dig down into a time range.
Note 1: You cannot go directly from Opentelemetry to Grafana, because Opentelemetry only provides a push model. Note 2: You can write your own components to process the data from Opentelemetry.
Grafana
A typical chart showing where time was spent
There are many layer – each one is for a unique span name and dimensions (meta data). There is a pop up with information about where the graphical cursor is. The information shows for queue manager MQPA, the MQPUT to RSERVER took 0.635 milliseconds.
Overall the work took between 7 and 4 milliseconds.
You can display multiple charts in a dashboard
This shows the average times for the different spans ( MQGET COLIN), and the count of “transactions”.
The MQPUT to CSERVER was highest, and about 1.2 milliseconds.
Visualises the data – see the image at the top of the blog post.
Can aggregates the data for Grafana to use
Visualise the data in Jaeger
By default it displays 20 entries
and
This shows there were two queue manager CSQ9 and MQPA, there were 8 spans (recorded data) the duration if each item, and the date. ( I think the date format is wrong .. you want hh:mm:ss – because you know what the day and date are!
If you click on the blue Trace Name or a blue dot – you get a detailed picture of where the time was spent
