Is it my new glasses, or have my x3270 sessions gone out of focus?

I’m running on Ubuntu Resolute Racoon, and after a recent update, my x3270 sessions connecting to z/OS all look out of focus, or like my glasses are dirty.

I think that something has changed in the fonts that are used, or my screen sizes, so when data is displayed it is having to fake the font.

I define a 3270 using a command like

x3270 -model 5 -title mstcon mstcon@localhost:3270  &

where mstcon is the short name.

I needed to be able to change the size of the 3270 screen, so the fonts were clear.

Third attempt at fixing it.

The easiest way of fixing the problem was to use

export DPI=240
x3270a -model 5 -title mstcon mstcon@localhost:3270  &

This may all be connected with X11 being removed from Ubuntu 26.04, and Ubuntu using Wayland instead. The XWayland code making the system look like X11 may have a couple of hickups.

The information below is from my journey to getting a readable screen.

What is the size of my screen?

You can use the wmctrl command to display information about the screen. wmctrl interacts with the X11 window manager.

wmctrl -l -G

This lists the geometry of all the screens

0x00600169 -1 396  478  2496 1520 ColinNew mstcon
0x00c00169 -1 794  702  2496 1582 ColinNew colin
0x00e00169 -1 794  764  2496 1520 ColinNew tso

Where

ColinNew is the name of my userid
mstcon is the name of my x3270 window
396 478 is the position of the top left corner of the window,
2496 1520 is the width and height of the window

You can change the configuration using

 wmctrl -r  colin  -e  1,744,618,2496,1582

You can grab the edges of the x3270 window, and make adjust the size till it looks right, then use the wmctrl -l -G to capture the geometry – and update your wmctrl … -e command.

Or you can do it with X3270

x3270 ... -geom 2496x1582+711+618 ...

Where -geom is -geometry widthxheight+x+y .

The font size jumps!

There are a set of fonts, as you change the size of the screen, for example make it half an inch narrower, at some point it may reconfigure, and suddenly use a smaller font (with lots of space around the content). You should size the window so it is wide enough and high enough for the font size, without wasting space around the content.

My glasses?

I also had a new pair of glasses, and have to position my head to be able to get the windows in focus. Because I keep my 3270 sessions at the top of my screens, I have to tilt my head back slightly!

Update

I did the following commands


sudo apt install fonts-3270
sudo apt install xfonts-x3270-misc
sudo sudo fc-cache -f -v

And x3270 suddenly got better with readable fonts etc..

Resetting the IBMUSER password on the ZD&T z/OS 3.2 stock image

I was sent this useful document from Dan Kalmar.

The IBM documentation on the initial logon to the ZD&T image in the “standard stock image” may not always work. The information below gives an alternative way of getting in.

Environment

Emulator ZD&T Personal Edition 14.2.8 (z1091 1.12.59.17), Ubuntu host
Image IBM z/OS 3.2 Dev & Test Stock Image for ZD&T PE 14.2.20 (built 2026-09-08)
IPL ipl DE27 parm DE28A0M (SYSRES Z32VS1 at DE27, IODF/LOAD on OPEVS1 at DE28)

The problem

The stock image ships with a randomized IBMUSER password. IBM’s documented procedure for Personal sEdition is to edit the password-injection file on the ZDTDPY volume, include ZDTDPY in the device map and IPL with load parm NV:

pdsUtil ZDTDPY ZDT.PASSFILE/PASS0 --extract (edit line 1 of PASS0.txt, 8 chars)
pdsUtil ZDTDPY ZDT.PASSFILE/PASS0 --overlay
ipl DE27 parm DE28NV

On this system that did not work. ZDTDPY was online with the edited PASS0, the IPL used DE28NV, yet TSO IBMUSER still failed with:

IKJ56474I USERID OR PASSWORD IS INCORRECT OR NOT AUTHORIZED

No console message referred to ZDTDPY, so the injection step did not appear to run. RACF commands from the console are refused as well:

IRRV003I (<) YOU ARE NOT ALLOWED TO ISSUE THE LISTUSER COMMAND
             AS AN OPERATOR COMMAND

so the password cannot be reset from the console, and the usual ADCD defaults (IBMUSER / ADCDMST) are rejected too. (Whether the 14.2.20 image needs the matching 14.2.20 emulator for the injection to run is an open question with IBM support.)

Caution: do not keep retrying logons — RACF will revoke the userid after a few failures.

The way in

The image ships two pieces that together allow a job to run with authority, without anyone logging on:

SYS1.PROCLIB(SUBMIT) on OPEVS1 — a started task that copies a member to the internal reader. It is
started with S SUBMIT,DS=dsname,M=member.
IBMUSER.JCL(SSHKYCP) on STGVS1 — a sample job (shipped to install an SSH key for IBMUSER). Its JOB card has no USER=, so the submitted job runs under the SUBMIT started-task user, which has enough authority to update IBMUSER.

The members can be read and rewritten from Linux with pdsUtil, the same way PASS0 is. So we replace the SSHKYCP member with a job that issues an ALTUSER, then let the shipped SUBMIT task run it at IPL time.

Key constraint: pdsUtil –overlay is an update-in-place — the replacement file must have exactly the same number of records as the original member. SSHKYCP is 26 records, so the replacement job is padded with comment lines to 26.

Procedure

All Linux commands run as the ZD&T user (ibmsys1). The volumes directory below is this system’s (/media/mybook6/z32/volumes) — substitute your own.

Step 1. Shut down z/OS and stop the emulator

awsstop

Step 2. Back up the volume that holds IBMUSER.JCL (STGVS1 here)

cd /media/mybook6/z32/volumes
cp STGVS1 ../STGVS1.bak

Step 3. Extract the SSHKYCP member and keep an untouched copy

mkdir -p ~/z32check && cd ~/z32check
pdsUtil /media/mybook6/z32/volumes/STGVS1 IBMUSER.JCL/SSHKYCP --extract
cp SSHKYCP.txt SSHKYCP.orig.txt

–extract writes SSHKYCP.txt (ASCII) into the current directory. Note the record count reported (26).

Step 4. Replace SSHKYCP.txt with the reset job, padded to 26 records

The job below runs ALTUSER under TMP/E (IKJEFT01). Keep it at exactly 26 lines. Put your own 8-character password (uppercase + digits) in place of NEWPASS1 on the ALTUSER line:


//PWRESET JOB CLASS=A,MSGCLASS=X
//* Reset IBMUSER password. Runs under the SUBMIT STC user (no USER=).
//* Start with: S SUBMIT,DS=IBMUSER.JCL,M=SSHKYCP
//*
//* --- padding so the member stays 26 records (pdsUtil --overlay rule) ---
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//S1 EXEC PGM=IKJEFT01
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
ALTUSER IBMUSER PASSWORD(NEWPASS1) NOEXPIRED RESUME
LISTUSER IBMUSER
/*

Confirm the count before writing it back:

wc -l ~/z32check/SSHKYCP.txt # must print 26

Step 5. Write the member back to the volume

cd ~/z32check
pdsUtil /media/mybook6/z32/volumes/STGVS1 IBMUSER.JCL/SSHKYCP --overlay

Success looks like: AWSPDS055I 26 records overlayed PDS member ‘SSHKYCP’. A record-count mismatch instead gives AWSPDS054E … RC=-10 — re-check the line count.

Step 6. IPL and run the job from the master console

cd /media/mybook6/z32 && sh runzpdt_z32

Once BASE Z/OS SUBSYSTEMS HAVE BEEN STARTED appears, on the master console:

S SUBMIT,DS=IBMUSER.JCL,M=SSHKYCP

Watch for $HASP395 PWRESET ENDED (and IEF404I PWRESET – ENDED). If no ICH/IRR error appears, the ALTUSER was accepted.

Step 7. Log on and verify

Dial a TSO terminal (the stock image uses devices 0062–0065 for VTAM; 0060/0061 are MCS consoles) and log on:

TSO IBMUSER (then the new password at IKJ56476I ENTER PASSWORD)

The PWRESET job output is held in class X — its LISTUSER confirms IBMUSER’s status.

Step 8. Restore the original member

Once in TSO, put the shipped SSHKYCP back so the image matches IBM’s.

Edit IBMUSER.JCL(SSHKYCP) in ISPF and paste the original contents (SSHKYCP.orig.txt), or re-run –overlay with the saved original while z/OS is down. Then change the password the normal way and delete the volume backup.

Why this works

The submitted job inherits the SUBMIT started task’s RACF identity (no USER= on the JOB card), and on this image that identity has enough authority to issue ALTUSER against IBMUSER. Nothing bypasses RACF: the job runs as an already-authorized started-task user, which is why it can set a password that console RACF commands are blocked from setting. Everything up to Step 6 is done offline with pdsUtil, the same utility IBM’s own PASS0 procedure uses.

Level independence and suspected root cause

This workaround is independent of the emulator and z/OS release. pdsUtil, SYS1.PROCLIB(SUBMIT), a no-USER= job inheriting the started task’s RACF identity, and ALTUSER are all stable across ZD&T and z/OS levels, so the method works whether or not the PASS0 injection mechanism is present on the image.

The likely reason IBM’s documented PASS0 procedure produced nothing here: the code that reads ZDT.PASSFILE(PASS0) at IPL may require a later emulator than 14.2.8. The image is built for ZD&T 14.2.20; this host runs the 14.2.8 emulator. That would explain why ZDTDPY was online with the edited PASS0 and the IPL used DE28NV as documented, yet no console message referred to the injection and the password was unchanged. This is a hypothesis, not confirmed by IBM — the open question is whether the 14.2.20 stock image requires the matching 14.2.20 emulator for PASS0 injection (and for other functions). If so, applying the 14.2.20 fix pack is the proper fix, and the procedure above is the bridge until then.

Note

This is shared as a working note — verify each step against your own entitlement and IBM documentation. Device numbers, volume serials and paths are specific to this ZD&T system.

How to export and import a Grafana dashboard

Like many things, it is easy once you know how. The instructions on the internet usually missed one tiny critical piece of information. |←

Export a dashboard

In Grafana, select a dashboard. At the bottom of the display select |← (circled in the image). (This was the missing bit of information.)

This gives you icons
- Download
- Content outline
- ->|
Select Download. This gives

You can either Download the file, or copy to the clip board. Because Grafana is running in a browser, the downloaded file is in your normal Downloads directory. One download had a name dashboard-1791876486315.json

Importing a dashboard

In Grafana, select Dashboards

Select the +^ on the top line, and select import dashboard.

You can drag and drop a file to the area, or select from a file list.

Capturing and replaying Opentelemetry data

I wanted to create some files so that people could get Opentelemetry, and displaying data working, but also to provide sample data. There are several sites offering configuration files, but not the data to display.

This has an additional challenge. With tools like Jaeger and Grafana, you typically display data “in the last 5 minutes”, or “in the last hour”. If I provide data, then it may be viewed a year later. However with a small bit of Python code I fixed this.

The Opentelemetry collector to capture data

This takes the data from the various sources in the network, and collects the information.

The receiver

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

The exporter

exporters:
  debug:
    verbosity: normal # basic: basic, normal, detailed

  otlp_http:
    endpoint: "http://jaeger2:4318" # was 4318
    tls:
      insecure: true
  file/in:
      path: /file.in

The magic

service:
  pipelines:
    traces:
      receivers: [otlp] 
#      processors: [transform/timestamps] # I did some transformation of the data
      exporters: [debug,file/in,otlp_http]

The flow

The data came from z/O into the oltp receiver (grpc: endpoint: 0.0.0.0:4317)
The service: traces: receivers…, exporters, says pipe the data from otlp and put it to
- debug – output to the terminal
- file/in – this maps to a file definition file.in
- otlp_http – for Jaeger to use

The docker command

I ran the opentelemetry collector in Docker, and this needs some statements to map from internal names (such as file.in), to the real file.


touch file.in.json
truncate -s 0 file.in.json
chmod 777 file.in.json
docker run --rm  --name ozf \
 --volume "$(pwd)/o2.yaml":/otel-config.yaml \
  -v "$(pwd)/file.in.json":"/file.in" \
  --env COLLECTOR_OTLP_ENABLED=true \
  --publish 4317:4317 \
  --publish 4318:4318 \
  --network otel-jaeger-network \
    otel/opentelemetry-collector-contrib:latest --config otel-config.yaml

The command executed is

otel/opentelemetry-collector-contrib:latest --config otel-config.yaml

Docker will download the code if needed. It will take the latest level of code.

The truncate file -s o resets the file to empty.

The configuration is in otel-config.yaml. The option –volume “$(pwd)/o2.yaml”:/otel-config.yaml maps otel-config.yaml to the file “$(pwd)/o2.yaml”

The yaml configuration file refers to /file.in. This maps to $(pwd)/file.in.json on my file system.

You can then issue commands like

cat file.in.json | jq |less

to look at the data.

The Opentelemetry collector to replay data

As well as receiving data over http, there are other sources, one of which is oltp_json_file which reads opentelemetry data from a file

receivers:
  otlp_json_file:
    include:
      - "/input.json"
    start_at: beginning

The docker command

docker run --rm  --name ozf \
 --volume "$(pwd)/o1.yaml":"/otel-config.yaml" \
  -v "$(pwd)/otel.in.json":"/input.json" \
  --env COLLECTOR_OTLP_ENABLED=true \
  --publish 4317:4317 \
  --publish 4318:4318 \
  --network otel-jaeger-network \
    otel/opentelemetry-collector-contrib:latest --config otel-config.yaml

The data is read from file file otel.in.json in the current directory.

Transforming the date times

The JSON data has start and end times. The display tools display “recent data”, such as the “last 5 minutes”. I wrote some python code to find the earliest time stamp, calculate the data from the time stamp to now, and adjust all the timestamps.

'''
Python  script to adjust the datetimes in an opentelemety json file, so it 
looks as it the data has just been created.

This has been tested on Ubuntu

Format

python3 fixtime filename1 <filename2>

Where 
filename1 is the name of the input file
filename2 is the name of the output file.   The default is otelts.json 

'''
import csv, json
import sys
import json
from datetime import datetime
import json 
import time
earliestTime = 0
if len(sys.argv) < 1:
    print("fixtime adjusts the otel datetime stampts in otel data to look like they've just been created")
    print("The format is fixtime  <inputfile> <outputfile>")
    print("Where the defaults are  inputfile is file.in.json and the outputfile is otel.in.json ")
    sys.exit(0)
if len(sys.argv) < 2:
   inputFile =  "file.in.json" 
else:
    inputFile  = sys.argv[1]   
if len(sys.argv) < 3:
   outputFile =  "otel.in.json" 
else:
    outputFile  = sys.argv[2]       
timenow = time.time_ns() 
lineno = 0
with open(inputFile, 'r') as file, open(outputFile, 'w') as fout:

    for line in file:
        data = json.loads(line)
        # get the first time - so we can offset all of the time values
        if earliestTime == 0:  # get the first time found in a span
            for d,dvalue in data.items():  # a dict with one item, resourceSpans
                for resourceSpan in dvalue :   # it is a list - do each one
                    scopeSpans= resourceSpan["scopeSpans"]
                    for scopeSpan in scopeSpans:  #"scopeSpans": [ is a list of scopes
                            spans = scopeSpan["spans"]  # a dict
                            # print(spans)
                            for s in spans:  # each span  in the list Only element
                                # s["scopeSpan"] = scopeSpan
                                earliestTime = s["startTimeUnixNano"]
                                delta = timenow- int(earliestTime)
                                break
                            break
                    break
        
        # now process all of the data and add the delta to the time stampb       
        for d,dvalue in data.items():  # a dict with one item resourceSpans                    
            for resourceSpan in dvalue :   # it is a list - do each one
                scopeSpans= resourceSpan["scopeSpans"]                
                for scopeSpan in scopeSpans:  #"scopeSpans": [ is a list of scopes
                    spans = scopeSpan["spans"]  # a dict
                    # print(spans)
                    for s in spans:  # each span  in the list Only element
                        s["startTimeUnixNano"] = int(s["startTimeUnixNano"]) + delta 
                        s["endTimeUnixNano"] = int(s["endTimeUnixNano"])
        # and now write out the updated data  
        jsonstr = json.dumps(data)
        fout.write(jsonstr+"\n")                                 

sys.exit(0)

How does it all fit together?

I ran the otel collector in one terminal, and Jaeger in another.
I stopped both of them
Use python3 fixtime json.file.in otel.in.json This created file otel.in.json.
Started otel
Start jaeger
Look at the data in a web browser pointing to the Jaeger address and port.

Where is my MQ Opentelemetry output from z/SOS?

With Opentelemetry, as work flows through a system, it sends “I am here” data back to a collector, which can process it, and pass it on to other tools to display the information on graphical dashboards.

I spent several days trying to work out why I was not getting any telemetry data from my queue managers.

Basic setup

Check SMF

The telemetry data is written to SMF. You should check it is being collected. The command D SMF gives output like

IFA714I 08.45.27 SMF STATUS          FRAME LAST   F      E   SYS=VS01       
          LOGSTREAM NAME               BUFFERS        STATUS                
        A-IFASMF.DEFAULT                      0       CONNECTED             
        A-IFASMF.COLIN                        0       CONNECTED             
        A-IFASMF.INMEM                  1289792       IN-MEMORY             
        A-IFASMF.MQOTEL                  393304       IN-MEMORY             
        A-IFASMF.T1159                        0       IN-MEMORY

This has some data in the buffers, so it looks like data is being produced.

You can check the SMF option using the command D SMF,O

This gives a lot of information including

INMEM(IFASMF.T1159,TYPE(1159),RESSIZMAX(0128M)) -- PARMLIB     
INMEM(IFASMF.MQOTEL,TYPE(1158),RESSIZMAX(0128M)) -- PARMLIB    
INMEM(IFASMF.INMEM,TYPE(30),RESSIZMAX(0128M)) -- PARMLIB

This shows the MQ SMF records, type 1158 are mapped to name IFASMF.MQOTEL. It may be different on your system.

Check the data gatherer is active

This is a Java program which runs on z/OS and writes to an HTTP connection. I run mine as a started task.

This program does not report any statistics, if it is using CPU, then is may be processing records.

Check the network traffic

I use TLS encryption on my connection to the Opentelemetry collector server running on Linux. Using tools like wireshark on the connection allow you to see the overall traffic, but not the content of the traffic. For example

Shows there is traffic to and from port 4317 on my Linxu box.

The records with length of 8258 contains my Otel traffic
There is also TCP/IP Keep Alive flows every 15 seconds or so.

On the Linux Opentelemetry collector

During setup, I had the collector write to debug, and to a file, so I could see the traffic coming down.

But no data is being produced.

There are two switches that need to be enabled.

The Opentelemetry state is sent as a message property within MQ messages.
There is a property traceparent, which identifies the high level piece of work. The data is of the format

trace identifier ‘-‘ span identifier ‘-‘ flags

Where flags is a two byte character string such as 01.

Format of the flags

If the rightmost bit of the flags is 0, then this signals do not collect any data.
If the rightmost bit of the flags is 1, then do more checks.
- If the queue has an attribute OTELTRAC(ON) (either directly, or as specified at the QMGR level), then emit the otel data
- Else do nothing.

The flag in the traceparent is called SAMPLED, which I found confusing. I didn’t want to sample – I wanted OTEL information for all records!

I had the flags specified as 00 and did not get any output. When I changed it to 01 I got the OTEL data – it was as easy as that.

MQ, JMS and Opentelemetry

I wanted to use Opentelemetry with MQ. One option is to use JMS, and the “code free” Otel support.

This code-free-support intercepts the send and receive methods, intercepts them, and sets up the Otel support (in MQ).

See here for information on the Java Otel instrumentation.

My code

I used the IBM MQ sample JMS programs JMSProducer and JMSConsumer.

export CLASSPATH=/opt/mqm/java/lib/com.ibm.mq.allclient.jar:$CLASSPATH
export CLASSPATH=/opt/mqm/samp/jms/samples:$CLASSPATH
export CLASSPATH=/home/colin/Downloads/opentelemetry-javaagent.jar:$CLASSPATH
export OTEL_JAVAAGENT_ENABLED=true
export OTEL_JAVAAGENT_LOGGING=simple
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://10.1.0.2:4317

java -javaagent:/home/colin/Downloads/opentelemetry-javaagent.jar
-Dotel.resource.attributes=service.name=ColinsJMS
-Dotel.traces.exporter=console
/opt/mqm/samp/jms/samples/JmsProducer.java \
-m MQPA -d RSERVER -l SYSTEM.DEF.SVRCONN -h 172.26.1.2 -p 2414

This produced

Jaeger output

Which shows the message passed through MQ and was got (by the server program).

Because the sample JMS program does not allow you to specify a ReplyToQueue manager name, the server cannot send the reply back.

I changed the JMSProducer to specify a replyToQueue, and pass this as part of the message.
I used the JMSConsumer program unchanged.

The updated application

export CLASSPATH=/opt/mqm/samp/jms/samples:$CLASSPATH
export CLASSPATH=/home/colin/Downloads/opentelemetry-javaagent.jar:$CLASSPATH
export OTEL_JAVAAGENT_ENABLED=true
 # export OTEL_JAVAAGENT_LOGGING=simple # default
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_LOGS_EXPORTER=console
export OTEL_METRICS_EXPORTER=console
export OTEL_EXPORTER_OTLP_ENDPOINT=http://10.1.0.2:4317	
java -javaagent:/home/colin/Downloads/opentelemetry-javaagent.jar \
     -Dotel.resource.attributes=service.name=ColinsJMSProducer \
     -Dotel.traces.exporter=console \
     /opt/mqm/samp/jms/samples/colinProducer.java   \
-m MQPA -d RSERVER -l SYSTEM.DEF.SVRCONN -h  172.26.1.2 -p 2414 -r CCOLIN

echo "===================================================="

java -javaagent:/home/colin/Downloads/opentelemetry-javaagent.jar \
     -Dotel.resource.attributes=service.name=ColinsJMSConsumer \
     -Dotel.traces.exporter=console \
     /opt/mqm/samp/jms/samples/JmsConsumer.java \
    -m MQPA -d CCOLIN -l SYSTEM.DEF.SVRCONN -h  172.26.1.2 -p 2414

When this ran, the Otel data displayed in Jaeger was

Comments on the output.

The chart shows
- A message is put to an MQ remote queue called RSERVER on queue manager MQPA. This operation takes 20.2 milliseconds.
- The message is got from a queue called CSQ9 (a transmission queue). This flows over the network to queue manager CSQ9.
- The message is put to a queue CSERVER on queue manager CSQ9.
- An application on CSQ9 gets the message, does some processing and puts a reply to queue CCOLIN ( on queue manager MQPA). You cannot see this program in the trace.
- The mover on CSQ9 gets the message from the transmission queue YMQPA and sends it over the network
- The mover on CSQ9 puts the message to queue CCOLIN
- The above processing is very quick.
- The putting JMS program ends, and starts the JMSConsumer program.
- 3.2 seconds after the initial put, the JMSConsumer program gets the message (and the MQGET takes 3.6 ms). The start of the second Java program takes a long time (most of this 3.2 seconds)
There are no entries for the JMS programs themselves. There are only entries for the MQ on z/OS.
Although there were two JMS Java programs involved – the Jaeger output just shows the MQ processing

What next?

Although this has shown that you can easily add Otel processing to your JMS program, the output is lacking.

You can manually instrument your Java program, as described by the Opentelemetry documentation.

How do I save Grafana dash boards?

There is documentation on the internet explaining how to save (share) dashboards. It goes like “from the hamburger icon select share dashboard”. I do not have the hamburger icon. I think this is because I am using the free opensource version.

I found a pointer to a script on the internet

#!/bin/bash

x=$(curl -u admin:password  http://localhost:3000/api/search?query= )

for uid in $(curl -u admin:password  http://localhost:3000/api/search?query= | jq --raw-output '.[].uid') ; do
  echo $uid
  curl -u admin:panthe0n  "http://localhost:3000/api/dashboards/uid/$uid" > "dashboards/$uid.json"
done

The output was a JSON file. I do not think you can import it back into Grafana (because it says it does not have a matching schema), but at least you can see what the contents were, and manually add them back into Grafana.

There is a more modern API such as

curl -u admin:panthe0n  http://localhost:3000/apis/dashboard.grafana.app/v1/namespaces/default/dashboards?limit=10

which will list all dashboards and their contents.

The modern API allows you to create dashboards – but I’ll leave this till another day.

Opentelemetry: Jaeger handling clock mismatches

I am collecting Opentelemetry data from z/OS and Linux, unfortunately the clocks on the two systems are not synchronised.

When I display the data in Jaeger, instead of looking like

Where the transaction took 56.5 milliseconds, the output looks like it took over 26 seconds.

The z/OS data (MQPA and CSQ9) is all squashed to the right because the z/OS clock is out by about 26 seconds.

How to fix it

Within the Jaeger yaml configuration file you can specify

extensions:
   jaeger_query:
     max_clock_skew_adjust: 30s

This says if it looks like the data has a time skew, and the time is off by less than 30 seconds, then make the timings look sensible.

How does it work?

The Opentelemetry data has information about each span processed. One of the fields is the parent span.

If Jaeger know know that the parent span executed between 13:00:00 and 13:00:01, and the spans from z/OS were for times 13:00:20 to 13:00:21, it looks like the z/OS times are out by 20 seconds, and so Jaeger can compensate for this.

By default max_clock_skew_adjust is 0, so there is no compensation.

Understanding Prometheus queries and why they might not work

I had a query

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le) / sum(rate(traces_span_metrics_duration_milliseconds_count[1m]) )

and it returned no data. Why?

I discuss how I debugged this problem in Debugging Prometheus queries

The Prometheus documentation has a sections on queries and data.

Prometheus calculations

Data types

There are different sorts of data in Prometheus

A scalar. This is a single value. It can be a number or a string.
An array or vector. This could be an empty array [], an array with one element [44], or an array with multiple elements [1,2,3]
A time series. This has a time stamp and some data.

Basic calculations

I’ll use division as an example:

Scalar/scalar gives a scalar
Array/scalar gives an array, with result[i] = array[i]/scalar
Array/Array when the arrays are the same size. Result[i] = LeftArray[i]/RightArray[i]
Otherwise no result

Functions

The sum function takes a vector and returns a vector; for example, sum[1,2,3] = [6]
The scalar function acts on a vector and creates a scalar; for example, scalar[6] = 6
The vector function acts on a scalar and returns a vector of size 1; for example, vector(6) gives [6].

Why did the query not return any data

The query

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le) / sum(rate(traces_span_metrics_duration_milliseconds_count[1m]) )

The sum() by() produces a vector, with each element of the vector corresponding to each unique value of le (8 elements in my case)
The sum() produces a vector with only one element.

From the basic calculations above an array of size 8/array of size 1 does not produce a result, because the arrays are of different sizes.

Putting scalar() around the second query means the calculation is array/scalar, which does work

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le) / scalar(sum(rate(traces_span_metrics_duration_milliseconds_count[1m]) ))

Debugging Prometheus queries

I had a query

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le) / sum(rate(traces_span_metrics_duration_milliseconds_count[1m]) )

and it returned no data. Why?

Ive explained the reason for this in Understanding Prometheus queries and why they might not work

Getting started

In Prometheus specify a query

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le)

Select explain

and click the enable the query tree view.
This gave me

The traces_span_metrics_duration_milliseconds_bucket returned 96 results.
Hovering over the traces_spans… line shows the unique values of the span_names, le:6, exported_job etc. The image shows the unique values of the span_name.
The rate calculation gave 96 results
The sum by(le) gave 6 results – matching the value in the hover over the traces_span_metrics_duration_milliseconds_bucket

You can click on a box to get an explanation of that box. For example the sum by(le) gave

The second part of the query

The sum returns one value.

The combining the two subqueries

The first part before the query gave 6 results, but the part after the ‘/’ gave 0 results, even though the sum() gave 1 result.

Further down was

This shows the data on the left is a time an array of multiple values (for each le value), but there is only one value on the right hand side, and this is used once. Because there is no value available for the coloured entries – they are dropped.

The problem

I have a vector with multiple elements, and I want to divide it by a vector with only one element. This does not work.

You can use the scalar function to covert a vector with one element to a scalar, and the calculation vector/scalar will work

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le) / scalar(sum(rate(traces_span_metrics_duration_milliseconds_count[1m]) )
      )

and the output was