The above packages take tracking data or metrics and can display them in dashboards. I found it a struggle to understand how they were configured, as the documentation assumes you are an expert, and I could not find any “starting from zero” documentation.

This is one of my blog posts on using the above packages with Docker. See Configuring opentelemetry, Jaeger, Prometheus, Grafana

• 101 the basics This gives the minimum configuration, and defines some of the terms
• 102 doing processing
• 103 http endpoints – this post
• problem determination

Understanding endpoints

If you think of the processing as a pipe line. Data from one or more sources go into the pipe, and at the remote end of the pipe the data can be copied to one or more destinations (fanned out).

When you define the pipeline you define

• receivers – these identify what goes into the pipe
• exporters – these identify what goes out of a pipe

Pass on data

As well as writing data to a file, or displaying it using debug, the most common pattern is to pass data on to another package

For example with Opentelemetry data it is passed it on to the Jaeger package which displays transaction delay information in near real time.

The code snippet below takes oltp data and exports it to a driver which exports otlp data over http.

The otlphttp definition sends it to the url http://jaeger2:4318. Where jaeger2 is some Docker magic which routes it to the Docker image called jaeger2.

...
exporters:
  otlphttp:
    endpoint: "http://jaeger2:4318"
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors:
      exporters: [otlphttp]   	     

The jaeger2 configuration has

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:4317"
      http:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:4318"

This defines a receiver of type otlp, for http data, read from port 4318.

Endpoints

You might see

${env:JAEGER_LISTEN_HOST:-localhost}           

This is a configuration syntax commonly used in Jaeger or OpenTelemetry to set default values for environment variables.

It evaluates as follows:

• If JAEGER_LISTEN_HOST is set: It uses the IP address or host defined in the environment variable.
• If JAEGER_LISTEN_HOST is empty or not set: It falls back to the default value of localhost.

A typical flow between containers is Opentelemetry, Jaeger, Prometheus, Grafana. You do not need to know what these are, just they are just containers which do processing.

Push and pull.

Push

For the connection between Opentelemetry and Jaeger, data is pushed to Jaeger.

• The Jaeger acts like a server, opens a TCP/IP socket, and listens for connections to it. You typically specify a network address of 0.0.0.0, which means listen on all networks on this box for the specified port. You can specify an address such as 172.26.5.0 which says only clients using addresses 172.26.5.* can connect.
• The Opentelemetry container acts like a client, and has the IP address and port of the Jaeger connection. It initiates a session to Jaeger.

Running with docker adds a layer of complexity to it. Docker uses its own (sub) network. I could run two docker environments on my laptop, and both could use port 999. Each environment has its own network, and each subnetwork can have its own ports. The client has to use the correct IP address,

To get my docker environment working consistently, I used a definiton like

endpoint: "http://jaeger2:4318"

where jaeger2 is the name of the docker image of the jaeger container. Docker then substitues the IP addess of that container.

Pull

The situation is more complex than I have written.

For the Jaeger to Prometheus connect, typically Prometheus “scrapes” the data from Jaeger. For a few hours my mental picture was Jaeger pushing data to Prometheus; so Jaeger needed the IP address of Prometheus. This is the wrong way round.

Prometheus contacts Jaeger and says “send me your data”. Prometheus is the client, and Jaeger is the server. For this connection, Prometheus has to have a definitions like

endpoint: "http://jaeger2:7654"

Receivers

With a receiver you just specify a port; to be more specific you specify an adapter and a port.

• Localhost is 127.0.0.0 is the internal adaptor
• 0.0.0.0 stands for all adaptors

You specify an endpoint like

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

The endpoint url of 0.0.0.0 seems to work. The documentation says use localhost, but this did not always work for me.

The 4317 port is an external port, I can send data to it from z/OS.

In the docker container, the internal value above is mapped to the external using

  --publish 4317:4317 

Exporters

You specify an endpoint like

exporters:

  otlp_http:
    endpoint: "http://jaeger2:4318"
    tls:
      insecure: true

This says use the “driver” to export otlp data over http. The jaeger2 directs the data to the docker image with name jaeger2. You do not need

  --publish 4318:4318 

in the docker file for this to work, because it is internal to the docker configuration.

Getting in and out of the docker environment.

Because docker runs its own sub-network, you need to configure external ports to docker.

To pass data from z/OS down to the Opentelemetry container I used port 4317 on my laptop.

The Opentelemetry configuration is

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /server.pem
          key_file: /server.key.pem
          min_version: "1.3"  # Enforces TLS 1.3 as the minimum requirement
          max_version: "1.3"  # Locks maximum version to TLS 1.3
          cipher_suites: []
          reload_interval: "1h"

The docker configuration has

 --publish 4317:4317 \

which is of the format external:internal.

This is one of my blog posts on using the above packages with Docker.

See Configuring opentelemetry, Jaeger, Prometheus, Grafana

• 101 the basics This gives the minimum configuration, and defines some of the terms
• 102 doing processing, this post
• 103 http endpoints
• problem determination

The above packages take tracking data or metrics and can display them in dashboards. I found it a struggle to understand how they were configured, as the documentation assumes you are an expert, and I could not find any “starting from zero” documentation.

This follows on from the 101 the basics.

Doing processing – counting records

With the configuration file

receivers:
  otlp_json_file:
     include:
      - /fooin.file
     start_at: beginning 
exporters:
  file/a:
      path: /fooout.file   
  file/b:
      path: /b.file       

connectors:
  count:

service:
  pipelines:
    traces:
      receivers: [otlp_json_file]
      exporters: [count]
    metrics:
      receivers: [count]
      exporters: [file/b]  

This reads the input file, and copies the data to “count”. Count is a pipeline stage which reads the input, counts the records and outputs a summary of the data to the receiver count, which maps to the file b.file.

This runs in Docker, so there is b.file maps to a real file.

 -v "$(pwd)/foob.json":"/b.file" \

The file needs to exist ( use touch foob.json) and the userid docker needs write to it. I used chmod 777 foob.josn

Another example using span_metrics

This counts the number of span records from opentelemetry collector. It is defined here.

The documentation says

This says it receives data with a type of metrics, and outputs data with a type of trace.

• traces: … exporters: [span_metrics]
• metrics: receivers: [span_metrics]

receivers:
  otlp_json_file:
     include:
      - /fooin.file
     start_at: beginning 
  
exporters:
  file/a:
      path: /fooout.file   

  file/b:
      path: /b.file       

  debug:
    verbosity: detailed # Options: basic, normal, detailed
    sampling_initial: 2
    sampling_thereafter: 10    


connectors:
  count:
  span_metrics:

processors: 
  batch:

service:
  pipelines:

    traces:
      receivers: [otlp_json_file]
      exporters: [span_metrics]
    metrics:
      receivers: [span_metrics]
      exporters: [file/b,debug]   

The above packages take tracking data or metrics and can display them in dashboards. I found it a struggle to understand how they were configured, as the documentation assumes you are an expert, and I could not find any “starting from zero” documentation.

These packages often run under Docker, which introduces additional complexity.

My mission

My missions was to take the Opentelemetry data from MQ on z/OS and display a summary of the data in Grafana, so I could see “the average transaction time over the last hour was ..”.

I can capture the data and send it down to Opentelemetry running on Ubuntu. I can display it in Jaeger so I know the basics work.

The basics

Configuration is done using YAML. This is a good interface and easy to use. Sub parameters are indented.

There is documentation on Opentelemetry collector configuration.

The configuration breaks down to

• input definitions ( receivers)
• output definitions ( exporters)
• processing

The simplest configuration file for Opentelemetry is

receivers:
  otlp_json_file:
     include:
      - /fooin.file
     start_at: beginning 
exporters:
  file:
      path: /fooout.file   
 
service:
  pipelines:
    traces:
      receivers: [otlp_json_file]
      exporters: [file]

Within the receivers and exporters you have “driver definitions” (my term). These drivers are like external functions. otlp_json_file is a driver for reading from a json file. Someone has written this (in go). It is not in the default opentelemetry package, so you have to use the package which includes these drivers.

For example the docker definition is

docker run --rm  --name otelcollector \
...
   otel/opentelemetry-collector-contrib:latest ...

where the standard package is otel/opentelemetry-collector:latest, without the -contrib.

The parameters for this driver are

     include: /fooin.file
     start_at: beginning 

• read the file fooin.file
• and start at the beginning.

There are many parameters you can specify – for example which code page the data is in. See the code on github.

In a similar way there is a “file driver” which writes data to the file with path: /fooout.file.

You cannot use a random name as a driver – it has to be available in the configuration.

Docker

When running under Docker, there is a level of indirection. In my Docker configuration I have

  -v "$(pwd)/myfoo.file":"/foo.file" 

Where /foo.file mentioned in the configuration file, and this maps to myfoo.file in the current directory. If you are using an output file, create it (use the touch myfoo.file command), and use chmod 777 myfoo.file so the container ( running under the docker userid) can access it.

The processing

There is a section

service:
  pipelines:
    traces:
      receivers: [otlp_json_file]
      exporters: [file]

Which says create a pipe line between the input receiver(s) and the output exporters. For trace data read from the device driver otlp_json_file and write it to the device driver file.

This just reads from the input file and writes the output to the output file.

A more complex example

receivers:
  otlp_json_file:
     include:
      - /fooin.file
     start_at: beginning 

exporters:
  file/a:
      path: /fooout.file   
  file/b:
      path: /b.file       
 

service:
  pipelines:
    traces:
      receivers: [otlp_json_file]
      exporters: [file/a,file/b]

This has two exporters file/a,file/b. In the exporters section, there is still the same device driver file, but there are now two of them file/a and file/b.

With only one definition you could use just file, or you could have a more descriptive definition file/mydata.

Can I just write it to the terminal?

Yes use debug

exporters:
  debug:
    verbosity: detailed # Options: basic, normal, detailed
...

 exporters: [file/b,debug]   

This can produce a lot of output, so only use it when there is only a little data.

You can use sampling_initial and sampling_thereafter to display the first few messages, then sample the rest, so you do not get flooded.

Running under docker

Within the docker definition I have code

  -v "$(pwd)/foo.file":"/fooin.file" \
  -v "$(pwd)/fooout.json":"/fooout.file" \
  -v "$(pwd)/b.json":"/b.file" \

which maps the name in the yaml, eg /fooin.file to the name outside of docker $(pwd)/foo.file, and so the file used by b.file is actually b.json