OTEL data, z/OS data gatherer, Jaeger and TLS

OpenTelemetry is a technology for providing trace data as work flows between systems. It is often called OpenTel or OTel.

As work flows around a network, status(“where the work is”) information is sent to a central location, and tools at this central location can sew the data together and produce visualisations of the flow of data, and where the work was delayed.

The data is normally sent over TCP/IP.

Products/programs could emit their own data in the required format, and sent it over TCP/IP to the server. On z/OS data can be written to In-Memory SMF, and an IBM provided data collector reads the SMF records and sends the data over TCP/IP to the central site.

Typically the data is sent to a visualisation tool called Jaeger, so you can see where the delays in your applications are.

IBM Data gatherer and Jaeger can connect over http, but I could not configure Jaeger to support TLS.
I could get IBM data gatherer to communicate using TLS to OpenTel, which passed the data on the Jaeger.

The TLS implementation of IBM Data Gatherer, feels it was written by someone who does not understand TLS, and only used self signed certificates!

Tools like the OpenTelemetry project can capture the information and make it available to visualisation tools like Jaeger, and data reporting tools like Prometheus.

I’ve documented how to install the IBM Data Gatherer This blog post takes the next step of using IBM Data Gatherer with TLS to secure the traffic from z/OS.

z/OS and Data Gatherer set up for TLS

For TLS you need the certificate, and a private key. These are .pem files (not RACF certificates). To get the TLS working, I copied the files from my Linux machine. You should generate files properly with tools like openssl.

With TLS you normally have a set of CA certificates to validate any certificate received. With Data Gatherer you provide have to provide a file of the certificate(s) of the back end server – and not the Certificate Authority certificates. This makes me think the developer of the code only used self signed certificates.

In the Data Gatherer JCL I had

//   SET $TLSCLKY='/usr/colin/oemput/certs/tempcert.key.pem' 
//   SET $TLSCLCR='/usr/colin/oemput/certs/tempcert.pem' 
//   SET $TLSCERT='/usr/colin/oemput/certs/tempcert.pem'
//   SET $OTLENDP='https://10.1.0.2:4317'
//JAVAJVM  EXEC PGM=JVMLDM&VERSION,REGION=&REGSIZE,PARM='/+I' 
...
//STDENV *
...
IJO="$IJO -Djavax.net.debug=ssl:handshake " 

The PARM=’/+I’ displays the parameters passed to the program. For example

JVMJZBL1006I IBM_JAVA_OPTIONS =  -Dgrb.otel.dt.read.flag=0 
-Dgrb.otel.dt.endpoint=https://10.1.0.2:4317 -Dgrb.otel.dt.protocol=grpc 
-Dgrb.otel.dt.resource.names=IFASMF.MQOTEL -Dgrb.otel.dt.tls.enabled=true 
-Dgrb.otel.dt.mtls.enabled=false 
-Dgrb.otel.dt.tls.trusted.certs=/usr/colin/oemput/certs/tempcert.pem 
-Dgrb.otel.dt.tls.client.key=/usr/colin/oemput/certs/tempcert.key.pem 
-Dgrb.otel.dt.tls.client.cert=/usr/colin/oemput/certs/tempcert.pem 
-Dgrb.otel.dt.exporter=false -Dgrb.otel.dt.resource.buffer.size= 
-Dgrb.otel.dt.data.dump=false 
-Dfile.encoding=IBM-1047 -Dgrb.otel.dt.stats.enabled=true -Djavax.net.debug=ssl:handshake                               

The -Djavax.net.debug turns on the Java trace of TLS. It shows the TLS handshake and I/O information

Create the Otel container

See One minute – docker to docker gfor more information about configuring multiple docker containers to work with each other.

I created the docker network

docker network create otel-jaeger-network

I created a docker image

docker run --rm  --name otelcollector \
  --volume "$(pwd)/o1.yaml":/otel-config.yaml \
  -v "/mnt/Colin/ssl/ssl2/tempcert.pem":"/server.pem" \
  -v "/mnt/Colin/ssl/ssl2/tempcert.key.pem":"/server.key.pem" \
  --env COLLECTOR_OTLP_ENABLED=true \
  --publish 4318:4318 \
  --publish 4317:4317 \
  --network otel-jaeger-network \
  --network-alias otelcollector \
  otel/opentelemetry-collector --config otel-config.yaml

Where the tempcert.key.pem and tempcert.pem where the private key, and certificate, the same ones as I used on z/OS.

The configuration file o1.yaml had

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
        tls:
          cert_file: server.pem
          key_file: server.key.pem
          min_version: "1.3"  # Enforces TLS 1.3 as the minimum requirement
          max_version: "1.3"  # Locks maximum version to TLS 1.3
          cipher_suites: []
          reload_interval: "1h"
      http:
exporters:

  otlphttp/jaeger:
    endpoint: "http://jaeger2:4318"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors:
      exporters: [otlphttp/jaeger ]

The tls: section had the TLS parameters. I specified TLS v1.3. You could specify 1.2; but 1.3 is better.

The exporters: otlphttp/jaeger:endpoint: “http://jaeger2:4318″ says send data to port 4318 in docker container with the name jaeger2.

There is a lot of good information about configuring OTEL here.

Create the Jaeger container

I created the container

docker run --rm  --name jaeger2 \
  --env COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  --env COLLECTOR_OTLP_ENABLED=true \
  --publish 16686:16686 \
  --network otel-jaeger-network \
  --network-alias jaeger \
  -v "/home/colin/otel/j1.yaml":"/j1.yaml" \
  cr.jaegertracing.io/jaegertracing/jaeger:2.19.0

This container has a name of jaeger2, matching the value in the opentel container.

This maps the internal port 16686 to the external port 16686 so I could use my browser on http://localhost:16686.

With this set of configurations I was able to use TLS from z/OS.