I’ve written a High level view of Opentelemetry, and visualising Opentelemetry data in Jaeger.

This blog post is about the Opentelemetry aggregated data, and displaying it in Grafana.

Prometheus accepts metric data from sources like Opentelemetry, but it does not take the raw Opentelemetry “log” data from application directly into Prometheus. There is a component System Performance Measuring(SPM) which takes the OTEL log data and creates metrics. It runs in Opentelemetry collector, or the Jaeger program. (I think running it in Opentelemetry collector is better, because running it in Jaeger did not immediatelygive me all of the records)

The SPM code takes each record

• extracts hinformation from the OTEL logs. For example the service name, the span name, and others fields that have been configured in the configuration file

Conceptually the SPM code takes all the records received in a time interval – say 10 seconds, and creates two records

• traces_span_metrics_duration_milliseconds_count
• traces_span_metrics_duration_milliseconds_sum

and copies in the default dimensions (meta data) and which additional dimensions have been configured to be included. My simplistic view would have had one record with both the sum and the count, but this is not how it is done. There is a lot of additional data for each of the generated records, but I cannot find how to use it.

See here for information

The span metrics processor works by inspecting every received span and computing the total count and the duration of spans for every unique combination of dimensions. Dimensions can be the service name, the operation, the span kind, the status code and any attribute present in the span.

You can display the data in Prometheus, but you can do so much more with Grafana. I did Unit Testing with Grafana, and my System Testing using Grafana, because Grafana can save queries and dashboards.

It will help if you are familiar with the Prometheus data model.

First query

The values traces_span_metrics_duration_milliseconds_sum and traces_span_metrics_duration_milliseconds_count are ever increasing.

To find the average value over 1 minute intervals use (see here for an explanation)

increase(traces_span_metrics_duration_milliseconds_sum[1m])/ increase(traces_span_metrics_duration_milliseconds_count[1m])

If you move the mouse over a colour, it gives you information about that point (see the purple data below). The pop-up shows it is for an MQPUT to CSERVER for queue manager (=service name) CSQ9, with a duration of 1.52 ms.

Below the chart is all the data for the items.

There is data on the chart for every combination of dimensions/parameters/meta data. These items include information like

service.name: CSQ9
span.name: MQPUT CSERVER
span.kind: SPAN_KIND_PRODUCER
status.code: STATUS_CODE_UNSET
collector.instance.id: 2e1c51fc-7f54-4da5-aa9f-9c3cac190103
w3.tracestate: rojo=MODEL – this is what I passed on the trace data.

I had some data with service.name MQPA, and different span.names

This is for all resources

You can select which records are displayed, for example

rate(traces_span_metrics_duration_milliseconds_sum{w3_tracestate= "rojo=MODEL"}[$__rate_interval])

Selects those with the property w3.tracestate = “rojo=MODEL”. I specified this by setting the MQ message propery

tracestate=rojo=MODEL  

You can control which parameters are passed from the original records into the aggregated records. See the dimensions parameter in the span_metrics: definitions.

Set up data source

You can set up different data sources. I have one for Prometheus, and one or Jaeger. Once you have configured them you can use explore to look at the data.

Set up dashboards

You can set up dashboard to display the data in different ways.

You specify the data source, and then how you want the data formatted, typically I use a timeline.

You can use Builder |code to define your query. With builder you define the query through a GUI. With code you specify the query.

You can define multiple queries which display on the same windows. You can hide/unhide the data from each query, by selecting/unselecting the query.

You could define one query for w3_tracestate= “appl=OnlineBanking”, and another for w3_tracestate= “appl=OfflineBanking”.

Time ranges

From a pull down you can select “last 5 minutes”, “last 15 minutes”, or you can specify a date and time range. This means you can go back to a previous days data.

I have it refreshing the data every 5 seconds, and the graph data gradually creeps left, until the data falls out of the displayed time range.

Data links

On each page you can set up data links. If you hover your graphic cursor over a data item, it shows the data item’s attributes in a pop up window. You can also configure data links, which appear on in the pop up window, so you can link to another URL, passing data from the data items.

In the figure below, I graphed the top 4 span_names with the highest average value. It has a data link called colins jaeger2

By clicking on the “colins jaeger2” it invoked the URL, and passed the __from and __to times, in the URL.

I was trying to make it easier to read the trace lines from Liberty. These lines can be hundreds of characters long – and it needs many scrolls right and left to display it.
If I extracted and displayed the line – it displayed garbage because the line was in ASCII, and if you just displayed the line, it displays the ASCII values.

This raised several challenges.

1. How do do you convert from ASCII to EBCDIC in an ISPF Rexx macro.
2. How do you nicely display a long line on one screen.

How do do you convert from ASCII to EBCDIC in an ISPF Rexx macro?

I could not find any easy code to copy, so I had to write code to create the ASCII to EBCDIC mapping in Rexx.

/* REXX */ 
/* 
exec to display long (Liberty) logs by flowing text 
*/ 
ADDRESS ISPEXEC 
'ISREDIT MACRO' 
 "ISREDIT autosave off     " 
 ascii0  = "000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F" 
 ebcdic0 = "................................" 
 ascii1= "202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F40" 
 ebcdic1 = " !..$.&'()*+,-./0123456789:;<=>?@" 
 ascii2 ="4142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F60" 
 ebcdic2=  "ABCDEFGHIJKLMNOPQRSTUVWXYZ(\)._." 
 ascii3 ="6162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F" 
 ebcdic3= "abcdefghijklmnopqrstuvwxyz{|}~." 
                                                                                
 ascii   = ascii0 || ascii1 || ascii2||ascii3 
 ebcdic = ebcdic0 || ebcdic1|| ebcdic2||ebcdic3 
                                                                                
 ascii = x2c(ascii) 
 /*get the current line number, and extract it  */ 
   "ISREDIT         (l) = LINENUM .ZCSR " 
   "ISREDIT      ( d )  = LINE   " l 
/* convert it to ebcdic  */ 
out = translate(d,ebcdic,ascii) 

The upper part of the code just creates the translation tables. For example ASCII 0x21i is EBCDIC “!”.

The code

"ISREDIT         (l) = LINENUM .ZCSR " 
   "ISREDIT      ( d )  = LINE   " l 
/* convert it to ebcdic from ascii  */ 
out = translate(d,ebcdic,ascii) 
say out

reads the current (.ZCSR) line of the file into variable d, converts to EBCDIC and displays it.

How do you nicely display a long line on one screen?

I used

out = translate(d,ebcdic,ascii) 
/* and display it */ 
out = strip(out) 
do  i = 1 to length(out) by 72 
    say substr(out,i,72 ) 
end 

which cuts the message into 72 byte chunks.

I enhanced this to allow me to enter a number of lines on the macro

/* REXX */ 
/* 
exec to display long (Liberty) logs by flowing text 
*/ 
ADDRESS ISPEXEC 
'ISREDIT MACRO (lines) ' 
 "ISREDIT autosave off     " 
 "ISREDIT (fcol,lcol)  = DISPLAY_COLS" /* get width of the screen */
 width = lcol - fcol 
 ascii0  = "000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F" 
 ebcdic0 = "................................" 
 ascii1= "202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F40" 
 ebcdic1 = " !..$.&'()*+,-./0123456789:;<=>?@" 
 ascii2 ="4142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F60" 
 ebcdic2=  "ABCDEFGHIJKLMNOPQRSTUVWXYZ(\)._." 
 ascii3 ="6162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F" 
 ebcdic3= "abcdefghijklmnopqrstuvwxyz{|}~." 
                                                                              
 ascii   = ascii0 || ascii1 || ascii2||ascii3 
 ebcdic = ebcdic0 || ebcdic1|| ebcdic2||ebcdic3 
                                                                              
 ascii = x2c(ascii) 
 /*get the current line number, and extract it  */ 
   "ISREDIT         (last) =  LINENUM .ZLAST" 
   "ISREDIT         (l)    = LINENUM .ZCSR " 
 if lines = "" then 
    lines = 1 
 do j = l to (l + lines  - 1) 
    if (j > last) then leave 
   "ISREDIT      ( d )  = LINE   " j 
   if rc <> 0 then leave 
    /* convert it to ebcdic  */ 
    out = translate(d,ebcdic,ascii) 
    /* and display it */ 
    out = strip(out) 
    /* display full width, based on screen size */

    do  i = 1 to length(out) by width 
        say substr(out,i,width) 
    end 
    say " " 
 end 
exit 

so if my macro is called zz, I can issue the command zz 3 and it displays 3 lines in the file

The couple of hours it took me to write this have made my life so much easier.

I remember being asked this question at my grandfather’s knee. Of course the popular answer, which is wrong, is they both weight the same. Yes, they have different weights, an ounce of gold is heavier than an ounce of paper.

An ounce of gold weights 31 grams, and an ounce of paper weight 28 grams. Gold is measured in ounces troy, and most other things, including paper, are measured in ounces avoirdupois.

A similar question is why do cars in Britain have a better miles per gallon than in the US – it is not because Americans have bigger cars. A US gallon is 3.7 litres and a UK gallon is 4.5 litres, Americans have smaller litres.

Connect the userid to the right groups.

It is better to give permissions to groups, rather than to individual userids.

If someone changes jobs, within your organisation, you just remove the userid of the person leaving from the group, and they lose all access.

If you have given access to a userid, you have to go through all resources (for example APPLs) and remove access.

When a new person joins, you just connect them to the group, and they should have access.

Zowe groups

If you are going to use Zowe, then user userid needs access to the Zowe group.

• If class(APPL) OMVSAPPL is defined, the Zowe group needs read access to it
• If users will be using z/OMSF they will need read access to the class(APPL) IZUDFLT (or what ever is specified)
• If certificate logon is being used, the certificate will need to map to a userid

Access z/OSMF

If userid will be using z/OSMF they need access to the z/OSMF group.

In the z/OSMF logs, I got

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user colin2 has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

I connected userid COLIN2 to the z/OSMF group IZUUSER. I think the change is available immediately.

Access to OMVSAPPL

An application can switch userid by using the pthread_security_np, and passing in a userid and password or a certificate (which maps to a userid).

The application can control which userid can be used by means of an resource in the APPL class.

If the resource was called MYAPPL, then pthread_security_np would use “MYAPPL”, and the userid being switched to much gave read access to MYAPPL.

The default (and only supported) resource is called OMVSAPPL.

If OMVSAPPL in class(APPL) is defined, the userid must have read access to it. z/OSMF also uses OMVSAPPL.
If OMVSAPPL in class(APPL) is not defined, no checks are done.

With Zowe you cannot specify the resource name.

Certificate logon

You can use a digital certificate from a web browser ( curl, or other tools) to authenticate to z/OS.  You need to map the certificate to a userid.

A certificate coming in can have a Distinguished Name like CN=adcdd.O=cpwebuser.C=GB  (Note the ‘.’not ‘,’ between elements).

Your userid needs to have SPECIAL define to be able to use the RACDCERT command (SPECIAL, not just GROUP-SPECIAL).

You will need a definition like (see here for the command)

RACDCERT MAP ID(ADCDD ) - 
    SDNFILTER('CN=adcdd.O=cpwebuser.C=GB') - 
    WITHLABEL('adcdd')

or a general definition for those certificate with  O=cpwebuser.C=GB, ignoring the CN part

RACDCERT MAP ID(ADCDB ) - 
   SDNFILTER('O=cpwebuser.C=GB') - 
   WITHLABEL('cpwerbusergb') 

or using the Issuing Distinguished Name (the Certificate Authority)

IDNFILTER(‘CN=TESTCA.OU=SSSCA.C=GB)

Using a generic

SDNFILTER(‘CN=a*.O=cpwebuser.C=GB’)

does not work.

If you attempt to use a certificate which is not mapped you get

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(COLIN)
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(0163)  SUBJECT(CN=adcdd.O=cpwebuser.C=GB) ISSUER(CN=SSCA8.OU=CA.O=SSS.C=GB).

It is worth defining these using JCL, because if you try to add it, and it already exists then you get a message saying it exists already.  If you know the userid, you can list the maps associated with it.   If you do not know the userid, there is no practical way of finding out – you have to logon with the certificate, and display the userid from the web browser, or extract the list of all users, and use LISTMAP on all of them.

Shared storage is a Unix concept, which works on Unix on z/OS.

It allows you to share data between threads, and between different address spaces. Java Shared Classes exploit this. Java classes can be stored in the shared storage, and “Ahead Of Time” compilation can be stored in the shared cache. Next time a JVM starts, it can look in the shared cache, and use the already compiled class.

Each shared storage segment has a key which you use in the various shared memory API calls.

There are no real restrictions, in some of the documentation on the web, it suggests using 1234 as a key! This has its limitations. A better way, used by Java is based on the inode of a file

You can display information about InterProcess Communication Status (ipcs) using the ipcs command.

You can display shared memory, and shared semaphores.

ipcs command lists…

Shared Memory:                                                           
T         ID     KEY        MODE       OWNER    GROUP                    
m       8196 0x00000000 --rw-------   WEBSRV   WEBGRP                    
m      73733 0x6137142d --rw------- OMVSKERN     SYS1                    
m      73734 0x6137e82f --rw------- ZWESVUSR ZWEADMIN                    
m     139271 0x6100410f --rw------- ZWESVUSR ZWEADMIN                    
m       8200 0x610ea00f --rw------- OMVSKERN     SYS1                    

The Java shared class files on my system are

767 -rw-r--r--   1 ZWESVUSR ZWEADMIN  ... C290M17F1A64_semaphore_zoweGW_G43L00             
768 -rw-r--r--   1 ZWESVUSR ZWEADMIN  ... C290M17F1A64_memory_zoweGW_G43L00                

Inode 767 is 0x2ff, Inode 768 is 0x300.

0x61… inode…. 2d

061 because created with t = ftok(fn,idi );

Shared Memory:                                                          
T         ID     KEY        MODE       OWNER    GROUP                   
m       8196 0x00000000 --rw-------   WEBSRV   WEBGRP                   
m     139269 0x6137142d --rw------- OMVSKERN     SYS1                   
m      73734 0x6137e82f --rw------- ZWESVUSR ZWEADMIN                   
m     139271 0x6100410f --rw------- ZWESVUSR ZWEADMIN                   
m       8200 0x610ea00f --rw------- OMVSKERN     SYS1                   
Semaphores:                                                             
T         ID     KEY        MODE       OWNER    GROUP                   
s     135172 0x8137132d --ra------- OMVSKERN     SYS1                   
s      69637 0x8137e72f --ra------- ZWESVUSR ZWEADMIN                   
s     135174 0x8100400f --ra------- ZWESVUSR ZWEADMIN                   
s      69639 0x8c10840f --ra-ra-ra- ZWESVUSR ZWEADMIN                   
s     135178 0x810e9f0f --ra------- OMVSKERN     SYS1                   
s      69643 0x8c108a0f --ra-ra-ra- ZWESVUSR ZWEADMIN                   

More details

You can use

ipcs -m -a

to display more information including size, and number of users.