Debugging Prometheus queries

I had a query

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le) / sum(rate(traces_span_metrics_duration_milliseconds_count[1m]) )

and it returned no data. Why?

Ive explained the reason for this in Understanding Prometheus queries and why they might not work

Getting started

In Prometheus specify a query

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le) 

Select explain

and click the enable the query tree view.
This gave me

• The traces_span_metrics_duration_milliseconds_bucket returned 96 results.
• Hovering over the traces_spans… line shows the unique values of the span_names, le:6, exported_job etc. The image shows the unique values of the span_name.
• The rate calculation gave 96 results
• The sum by(le) gave 6 results – matching the value in the hover over the traces_span_metrics_duration_milliseconds_bucket

You can click on a box to get an explanation of that box. For example the sum by(le) gave

The second part of the query

The sum returns one value.

The combining the two subqueries

The first part before the query gave 6 results, but the part after the ‘/’ gave 0 results, even though the sum() gave 1 result.

Further down was

This shows the data on the left is a time an array of multiple values (for each le value), but there is only one value on the right hand side, and this is used once. Because there is no value available for the coloured entries – they are dropped.

The problem

I have a vector with multiple elements, and I want to divide it by a vector with only one element. This does not work.

You can use the scalar function to covert a vector with one element to a scalar, and the calculation vector/scalar will work

sum (rate(traces_span_metrics_duration_milliseconds_bucket[1m]) ) by(le) / scalar(sum(rate(traces_span_metrics_duration_milliseconds_count[1m]) )
      )

and the output was