Configuring opentelemetry, Jaeger, Prometheus, Grafana – 101

The above packages take tracking data or metrics and can display them in dashboards. I found it a struggle to understand how they were configured, as the documentation assumes you are an expert, and I could not find any “starting from zero” documentation.

These packages often run under Docker, which introduces additional complexity.

My mission

My missions was to take the Opentelemetry data from MQ on z/OS and display a summary of the data in Grafana, so I could see “the average transaction time over the last hour was ..”.

I can capture the data and send it down to Opentelemetry running on Ubuntu. I can display it in Jaeger so I know the basics work.

The basics

Configuration is done using YAML. This is a good interface and easy to use. Sub parameters are indented.

The configuration breaks down to

• input definitions ( receivers)
• output definitions ( exporters)
• processing

The simplest configuration file for Opentelemetry is

receivers:
  otlp_json_file:
     include:
      - /fooin.file
     start_at: beginning 
exporters:
  file:
      path: /fooout.file   
 
service:
  pipelines:
    traces:
      receivers: [otlp_json_file]
      exporters: [file]

Within the receivers and exporters you have “driver definitions” (my term). These drivers are like external functions. otlp_json_file is a driver for reading from a json file. Someone has written this (in go). It is not in the default opentelemetry package, so you have to use the package which includes these drivers.

For example the docker definition is

docker run --rm  --name otelcollector \
...
   otel/opentelemetry-collector-contrib:latest ...

where the standard package is otel/opentelemetry-collector:latest, without the -contrib.

The parameters for this driver are

     include: /fooin.file
     start_at: beginning 

• read the file fooin.file
• and start at the beginning.

There are many parameters you can specify – for example which code page the data is in. See the code on github.

In a similar way there is a “file driver” which writes data to the file with path: /fooout.file.

You cannot use a random name as a driver – it has to be available in the configuration.

Docker

When running under Docker, there is a level of indirection. In my Docker configuration I have

  -v "$(pwd)/myfoo.file":"/foo.file" 

Where /foo.file mentioned in the configuration file, and this maps to myfoo.file in the current directory. If you are using an output file, create it (use the touch myfoo.file command), and use chmod 777 myfoo.file so the container ( running under the docker userid) can access it.

The processing

There is a section

service:
  pipelines:
    traces:
      receivers: [otlp_json_file]
      exporters: [file]

Which says create a pipe line between the input receiver(s) and the output exporters. For trace data read from the device driver otlp_json_file and write it to the device driver file.

This just reads from the input file and writes the output to the output file.

A more complex example

receivers:
  otlp_json_file:
     include:
      - /fooin.file
     start_at: beginning 

exporters:
  file/a:
      path: /fooout.file   
  file/b:
      path: /b.file       
 

service:
  pipelines:
    traces:
      receivers: [otlp_json_file]
      exporters: [file/a,file/b]

This has two exporters file/a,file/b. In the exporters section, there is still the same device driver file, but there are now two of them file/a and file/b.

With only one definition you could use just file, or you could have a more descriptive definition file/mydata.

Can I just write it to the terminal?

Yes use debug

exporters:
  debug:
    verbosity: detailed # Options: basic, normal, detailed
...

 exporters: [file/b,debug]   

This can produce a lot of output, so only use it when there is only a little data.

You can use sampling_initial and sampling_thereafter to display the first few messages, then sample the rest, so you do not get flooded.

Running under docker

Within the docker definition I have code

  -v "$(pwd)/foo.file":"/fooin.file" \
  -v "$(pwd)/fooout.json":"/fooout.file" \
  -v "$(pwd)/b.json":"/b.file" \

which maps the name in the yaml, eg /fooin.file to the name outside of docker $(pwd)/foo.file, and so the file used by b.file is actually b.json