Monitoring your application

In this documentation you will learn how to instrument Prometheus to scrape metrics from your application and then how to add a custom Grafana dashboard to visualize them using the monitoring stack of the SIGHUP Distribution.

Prerequisites

The following conditions must be true to achieve the goal of this documentation:

• An SD cluster with monitoring type set to prometheus or mimir.
• An application with a Prometheus-compatible metrics endpoint.

Scraping the metrics

The Prometheus instance included in the Monitoring module is pre-configured to scrape metrics from several places, and it can be extended using Custom Resources like PodMonitors and ServiceMonitors to scrape additional endpoints. There are other options, see Prometheus Operator API reference for more information.

Assuming that your application has an associated Service my-service with a label app: my-app, to instruct Prometheus to scrape the metrics from your application via the service, create a ServiceMonitor that selects the service via its labels, like the following example:

Example Service Monitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: my-namespace
spec:
  endpoints:
    - interval: 10s
      port: http
      scheme: http
      path: /metrics
  selector:
    matchLabels:
      app: my-app

Be sure to configure the endpoints with the ones exposed by your service. Apply the manifest to the cluster and wait for a moment for Prometheus to pick up the configuration and start scraping the new ServiceMonitor.

Once Prometheus has scraped your application, you can go to the Prometheus UI (usually https://prometheus.internal.<base domain>/ or to the Grafana "Explore" view (usually https://grafana.internal.<base domain>/explore) and do a PromQL query for your metrics to verify it.

Now that you have metrics being scraped and stored on Prometheus (or Mimir), you can proceed to create a dashboard to visualize them.

tip

See the Prometheus Getting Started Guide to learn more on this topic.

Create a Custom Grafana Dashboard

Building the dashboard itself is out of scope of this documentation, refer to Grafana's documentation on build your first dashboard and create dashboards for help on designing the dashboard.

You may also use some dashboard created from someone else instead of building your own, for example there are dashboards already available for the major frameworks and programming languages.

Once you have built or found your dashboard using the prometheus or mimir datasource (depending on your choice of Monitoring type), export it to a JSON file. Let's assume that you exported it to the my-dashboard.json file.

warning

The Grafana included with SD is stateless, if you are using this Grafana to build your dashboard it will be lost if Grafana's pod gets restarted for any reason.

Even though Grafana is state-less, it is configured to pick up Dashboards and Datasources from the cluster.

To make the Dashboard persistent and survive Grafana restarts, create a configMap with the JSON file.

The configMap must have the following characteristics:

• It can be created in any namespace (we recommend creating it in your application's namespace)
• It must have the grafana-sighup-dashboard label (the value is not important).
• Optionally, if you want to put the dashboard(s) in a specific Grafana folder, for example myfolder, add the following annotation to the configMap: grafana-folder: myfolder.

You can do this easily with a Kustomize configMapGenerator:

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: my-namespace

generatorOptions:
  labels:
    grafana-sighup-dashboard: default # This is needed so Grafana knows that this configMap contains dashboards
  annotations:
    grafana-folder: "My App" # (Optional) This will put the dashboard in the desired Folder, it will create the folder if it does not exist.
  disableNameSuffixHash: true # You can use the same configMap and when it is modified Grafana will pick up the changes, no need for the suffix.

configMapGenerator:
  - name: my-app-dashboards
    files:
      - my-dashboard.json
      # You can add more dashboards JSON files to this list

Apply the configMap to the cluster and a few moments later Grafana should pick it up and show it inside your chosen folder. When you update the configMap and the dashboard will be updated in Grafana as well.