Exploring Prometheus

Once we have the Prometheus data source properly configured, you might be wondering what kind of data we're likely to see. Turns out, since we configured Prometheus to scrape itself, we'll get a bunch of juicy internal server metrics delivered to the scraped endpoint and stored in the Prometheus database. So, let's dive in and get an idea of what's there.

Using Explore for investigation

Selecting Explore from the left-hand side menu activates the Explore tool. Basically, Explore includes special versions of both Graph and Table panel plugins, each looking at the same data source query. Make sure to select your Prometheus data source from the dropdown, then select a metric data series by selecting up from the Metrics menu. This is probably the simplest metric available: it shows 1 if the server is up and 0 otherwise. You can see, from the following screenshot, that (obviously) our Prometheus server is up and running:

The graph shows a single series with a value of 1 and the table contains 1 in the Value #A field. You should also take note of the series label. In this case, it refers to the up metric,tagged withlocalhost:9090for the instance value andprometheus for the job value. Going back to the configuration file, we can see where the job label comes from:

 # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
 - job_name: 'prometheus'

But where does the metric itself come from, and how does Grafana know about all those metrics in the Metrics drop-down menu?

Every 5 seconds, Prometheus sends an HTTP request to a specific endpoint, http://localhost:9090/metrics. Go ahead, you can even open it in a browser. You should see a page filled with metrics data. Here are the first few lines:

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 7.057e-06
go_gc_duration_seconds{quantile="0.25"} 1.2362e-05
go_gc_duration_seconds{quantile="0.5"} 2.7312e-05
go_gc_duration_seconds{quantile="0.75"} 0.000259168
go_gc_duration_seconds{quantile="1"} 0.001861891
go_gc_duration_seconds_sum 0.006119489
go_gc_duration_seconds_count 36
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 39
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.1"} 1
...

As you can see, a lot of the metrics are simply a metric name and a value and are sometimes a duplicated metric name followed by a key-value pairing in braces called a label. In other applications, the label is called a tag, but it performs the same function, which is to attach a piece of metadata to the metric to distinguish between similar metrics.

This page of data is parsed, timestamped, and stored in the Prometheus database. When you launch the Explore tool in Grafana, the Prometheus data source plug makes a service discovery query to find out what metrics are available and based on the response, it builds a convenient menu for you.

Let's now take a look at how the go_gc_duration_seconds metric is depicted in Explore. Select go from the dropdown (go, in this case, refers to the initial portion of the metric name, called the metric's namespace). From the submenu, select gc_duration_seconds to see the metric graph:

Now, we can see that each series name includes quantile, along with instance and job. Further down in the table, quantile is treated as a field, much like in a typical database or spreadsheet. This is great, but are we limited to Prometheus metrics? Not at all!

Configuring Grafana metrics

Now that we have a handle on some of the rich metrics available in Prometheus, can we get similar metrics in Grafana? Indeed, we can, but in order to do so with the Docker versions of both Prometheus and Grafana, we need to link them over the same network. That's why we brought them up as a dual container app in Docker Compose. All containers in a Docker Compose app share a single network, complete with a DNS entry, which happens to be the container name.

Let's go ahead and update the configuration with a new job that will scrape the Grafana server. Add additional lines to scrape_configs in the prometheus.yml file (also available as prometheus-grafana.yml from this book's GitHub repository):

scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'

# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s

static_configs:
      - targets: ['localhost:9090']
  - job_name: 'grafana'

# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s

static_configs:
      - targets: ['grafana:3000']

Sending an HUP signal to the Prometheus container process should force it to re-read the configuration file. Run the following command:

          docker-compose kill -s HUP prometheus
        

Go back to the Prometheus page and check the targets at http://localhost:9090/targets to confirm that Grafana is now a target:

Let's go back to Explore and see what Grafana goodies Prometheus scraped for us.