Skip to content

Managing Alerts

Overview

This repository contains the monitoring configuration based on Prometheus, Grafana, Alertmanager, and several exporters (Node Exporter, cAdvisor, Blackbox Exporter).

System Components

Component Purpose Link
Prometheus Collects metrics from all systems Prometheus UI
Grafana Visualizes metrics and alerts Grafana UI
Alertmanager Processes alerts and sends notifications Alertmanager UI

Where to Find the Configuration

All files are located in the repository:
🔗 GitLab: prometheus-grafana

Main Directories

Path Description
provisioning/config/prometheus.yml Main Prometheus configuration (targets, Alertmanager connection, etc.)
provisioning/config/alertmanager.yml Alertmanager configuration (notification rules, email settings)
provisioning/config/rules.yml Alert rules (critical conditions, availability checks, etc.)
provisioning/config/node_exporter_rules.yml Additional rules for Node Exporter metrics
provisioning/config/blackbox.yml Blackbox Exporter configuration (service availability checks)

img.png img_1.png

How It Works

General Workflow

  1. Prometheus collects metrics from all targets (exporters, Docker containers, web services via Blackbox, etc.).
  2. If any rule triggers (e.g., a host is down, disk is 90% full), Prometheus generates an alert.
  3. The alert is sent to Alertmanager, which decides whether to notify someone and via which channel (email, messenger, etc.)
  4. All alert information is available in Grafana and Alertmanager UI.

Configuration Details

Key Alertmanager Settings:

  • All notifications are sent to: mentioneduser@quintessence.net
  • Email relay server: qcmgw-qcint (internal relay)
  • Notifications are sent for both firing and resolved alerts

How to Change Alert Recipients

  1. Open provisioning/config/alertmanager.yml.
  2. Find the receivers section and change the email address.
  3. Application will restart automatically after commit changes to git repo.

How to Add a New System to Monitoring

  1. Add the new host to prometheus.yml (in the appropriate scrape_config section).
  2. Add alert rules for this host in rules.yml if necessary.
  3. Application will restart automatically after commit changes to git repo.

Troubleshooting

What to Check Command/Location
Alertmanager Logs docker logs alertmanager
Prometheus Logs docker logs prometheus
Check Alertmanager UI Availability https://qcmon-qcint.quintessence.de/alertmanager
Current Alerts In Grafana or in Alertmanager UI
Check Metrics and Targets Prometheus Targets

Contacts

Dmitry Huks (dhuks@quintessence.net)

Changelog

Date Author Message
2026-02-25 aresnikowa Merge remote-tracking branch 'origin/master'