Configuring opentelemetry, Jaeger, Prometheus, Grafana – 103 http endpoints

The above packages take tracking data or metrics and can display them in dashboards. I found it a struggle to understand how they were configured, as the documentation assumes you are an expert, and I could not find any “starting from zero” documentation.

This is one of my blog posts on using the above packages with Docker. See Configuring opentelemetry, Jaeger, Prometheus, Grafana

• 101 the basics This gives the minimum configuration, and defines some of the terms
• 102 doing processing
• 103 http endpoints – this post
• problem determination

Understanding endpoints

If you think of the processing as a pipe line. Data from one or more sources go into the pipe, and at the remote end of the pipe the data can be copied to one or more destinations (fanned out).

When you define the pipeline you define

• receivers – these identify what goes into the pipe
• exporters – these identify what goes out of a pipe

Pass on data

As well as writing data to a file, or displaying it using debug, the most common pattern is to pass data on to another package

For example with Opentelemetry data it is passed it on to the Jaeger package which displays transaction delay information in near real time.

The code snippet below takes oltp data and exports it to a driver which exports otlp data over http.

The otlphttp definition sends it to the url http://jaeger2:4318. Where jaeger2 is some Docker magic which routes it to the Docker image called jaeger2.

...
exporters:
  otlphttp:
    endpoint: "http://jaeger2:4318"
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors:
      exporters: [otlphttp]   	     

The jaeger2 configuration has

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:4317"
      http:
        endpoint: "${env:JAEGER_LISTEN_HOST:-localhost}:4318"

This defines a receiver of type otlp, for http data, read from port 4318.

Endpoints

You might see

${env:JAEGER_LISTEN_HOST:-localhost}           

This is a configuration syntax commonly used in Jaeger or OpenTelemetry to set default values for environment variables.

It evaluates as follows:

• If JAEGER_LISTEN_HOST is set: It uses the IP address or host defined in the environment variable.
• If JAEGER_LISTEN_HOST is empty or not set: It falls back to the default value of localhost.

A typical flow between containers is Opentelemetry, Jaeger, Prometheus, Grafana. You do not need to know what these are, just they are just containers which do processing.

Push and pull.

Push

For the connection between Opentelemetry and Jaeger, data is pushed to Jaeger.

• The Jaeger acts like a server, opens a TCP/IP socket, and listens for connections to it. You typically specify a network address of 0.0.0.0, which means listen on all networks on this box for the specified port. You can specify an address such as 172.26.5.0 which says only clients using addresses 172.26.5.* can connect.
• The Opentelemetry container acts like a client, and has the IP address and port of the Jaeger connection. It initiates a session to Jaeger.

Running with docker adds a layer of complexity to it. Docker uses its own (sub) network. I could run two docker environments on my laptop, and both could use port 999. Each environment has its own network, and each subnetwork can have its own ports. The client has to use the correct IP address,

To get my docker environment working consistently, I used a definiton like

endpoint: "http://jaeger2:4318"

where jaeger2 is the name of the docker image of the jaeger container. Docker then substitues the IP addess of that container.

Pull

The situation is more complex than I have written.

For the Jaeger to Prometheus connect, typically Prometheus “scrapes” the data from Jaeger. For a few hours my mental picture was Jaeger pushing data to Prometheus; so Jaeger needed the IP address of Prometheus. This is the wrong way round.

Prometheus contacts Jaeger and says “send me your data”. Prometheus is the client, and Jaeger is the server. For this connection, Prometheus has to have a definitions like

endpoint: "http://jaeger2:7654"

Receivers

With a receiver you just specify a port; to be more specific you specify an adapter and a port.

• Localhost is 127.0.0.0 is the internal adaptor
• 0.0.0.0 stands for all adaptors

You specify an endpoint like

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

The endpoint url of 0.0.0.0 seems to work. The documentation says use localhost, but this did not always work for me.

The 4317 port is an external port, I can send data to it from z/OS.

In the docker container, the internal value above is mapped to the external using

  --publish 4317:4317 

Exporters

You specify an endpoint like

exporters:

  otlp_http:
    endpoint: "http://jaeger2:4318"
    tls:
      insecure: true

This says use the “driver” to export otlp data over http. The jaeger2 directs the data to the docker image with name jaeger2. You do not need

  --publish 4318:4318 

in the docker file for this to work, because it is internal to the docker configuration.

Getting in and out of the docker environment.

Because docker runs its own sub-network, you need to configure external ports to docker.

To pass data from z/OS down to the Opentelemetry container I used port 4317 on my laptop.

The Opentelemetry configuration is

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /server.pem
          key_file: /server.key.pem
          min_version: "1.3"  # Enforces TLS 1.3 as the minimum requirement
          max_version: "1.3"  # Locks maximum version to TLS 1.3
          cipher_suites: []
          reload_interval: "1h"

The docker configuration has

 --publish 4317:4317 \

which is of the format external:internal.