Overview
SPOG.AI integrates with Prometheus Alertmanager to enable centralized alert ingestion and monitoring under a specific tenant. The integration connects SPOG.AI with Prometheus via a webhook endpoint. Once configured, SPOG.AI receives alerts from Prometheus in real time, enriching incident visibility and supporting tenant-level monitoring.
Key Features
| Capability | Description |
| Real-Time Alert Ingestion | Receives alerts from Prometheus Alertmanager in real time, as they fire. |
| Webhook-Based Connectivity | Connects SPOG.AI with Prometheus through a webhook endpoint generated by SPOG.AI. |
| Tenant-Level Monitoring | Ingests alerts into the tenant’s dashboard for centralized monitoring under a specific tenant. |
| Asset Correlation | Correlates incoming alerts with internal assets. |
| Incident & Notification Workflows | Can generate incidents, dashboards, or notifications from ingested alerts. |
| Grouping & Routing Support | Uses Alertmanager’s grouping, routing, and inhibition logic so SPOG.AI receives a coherent stream of alerts. |
| Resolution Tracking | Captures resolution events when alerts move from firing to resolved. |
| Secure Webhook Traffic | Supports TLS and Basic Auth on webhook HTTP traffic, with options for custom authorization headers. |
How the Integration Works
SPOG.AI acts as a webhook receiver for Prometheus Alertmanager. When Alertmanager fires alerts, it sends HTTP POSTs to the configured SPOG webhook endpoint. SPOG.AI ingests those alerts in real time into the tenant’s dashboard, correlates them with internal assets, and can generate incidents, dashboards, or notifications. Alertmanager’s grouping, routing, and inhibition logic ensures SPOG receives a coherent stream of alerts.
Data Collected
SPOG.AI will ingest the following from Prometheus alerts:
| Category | Data Collected | Description |
| Alert Fields | labels, annotations, status (firing / resolved), startsAt, endsAt, generatorURL, fingerprint | Core attributes of each alert sent by Alertmanager. |
| Receiver | Receiver name (the SPOG receiver) | Name of the Alertmanager receiver that delivered the alert. |
| Grouping Labels | Common labels / groupLabels as defined in routing | Labels determined by your Alertmanager routing configuration. |
| Metadata | Where the alert came from, grouping, timestamps | Source and context information for each alert. |
| Resolution Events | Alerts that go from firing → resolved | Status changes showing when an alert has been resolved. |
| Grouped Alerts | Grouped alerts as per your routing config | Alerts arrive grouped according to your Alertmanager routing. |
| Real-Time Ingestion | Alerts pushed as they fire | Alerts are pushed to SPOG.AI as they fire. |
Prerequisites
Before configuring the integration, ensure the following prerequisites are met:
- You must have Organization or Tenant Admin privileges in SPOG.AI.
- You must be able to modify the Prometheus alertmanager.yml configuration and restart Alertmanager.
- Your SPOG webhook endpoint must be publicly reachable (or network reachable) by Alertmanager.
- If TLS or authentication is required on the webhook endpoint, you’ll need certificate and credential support.
- Alertmanager version should support webhook receivers, which is part of standard Alertmanager.
- For security, Alertmanager supports TLS and Basic Auth on webhook HTTP traffic.
Configuration
1. Generate Connection Details
- Log in to your SPOG.AI instance.
- Navigate to Settings → Integrations → Add Integration.
- Choose Prometheus / Alertmanager Integration and click Configure.
- SPOG will generate a Webhook Endpoint URL (e.g.https://spog.example.com/webhook/prometheus/…).
- Copy and securely store this WEBHOOK_URL.
- Copy and securely store this WEBHOOK_URL.
2. Configure Prometheus in SPOG
- Edit your alertmanager.yml (or whichever config file Alertmanager uses).
- Add a receiver for SPOG:
receivers:
- name: 'SPOG'
webhook_configs:
- url: '<WEBHOOK_URL>'- Reference that receiver in your route block (or child route) so alerts are sent to it:
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'SPOG'- (Optionally) define inhibition rules or default/global settings as needed.
- Save the configuration and restart Alertmanager to apply.
A minimal end-to-end example:
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'SPOG'
receivers:
- name: 'SPOG'
webhook_configs:
- url: '<WEBHOOK_URL>'- 6. Validate the alertmanager.yml using amtool check-config or equivalent to catch syntax or routing errors.
3. Security & Access Control
Security & Access Control
- Use HTTPS / TLS for webhook traffic to encrypt the alert payloads in transit.
- Alertmanager supports TLS + Basic Auth on webhook HTTP traffic (though that is experimental).
- Protect the webhook URL as a secret; do not expose it publicly beyond required access.
- Limit write access to
alertmanager.ymlto trusted operators only. - If using Basic Auth or custom headers in
webhook_configs.http_config.authorization, you can embed credentials (Base64, Bearer) for authentication. - In webhooks, adopt security best practices such as signing the payload (HMAC), timestamping, and rejecting replayed requests.
- Log all webhook requests in SPOG, including headers, status codes, timestamps, IP source.
- Use rate limits or firewall rules to accept only from known Alertmanager IP or network segments.
4. Network & Connectivity
- Alertmanager must be able to send an HTTP(S) POST to the SPOG webhook endpoint (port 443 or as configured).
- If your SPOG webhook is behind a firewall or NAT, ensure the public DNS / IP is reachable from the Prometheus server(s).
- Use TLS (TLS 1.2 or 1.3) for webhooks.
- If using Basic Auth or headers, ensure proxies or load balancers do not strip those headers.
- Configure timeouts (on webhook side) to respond promptly to avoid Alertmanager marking webhook failures.
- If Alertmanager clusters or HA are used, ensure all nodes can reach the SPOG endpoint.
- Monitor for network latency or issues in delivery — if webhooks fail repeatedly, SPOG should alert or log errors.
5. Validation & Testing
- Trigger a test alert in Prometheus (for example using a
--testor alert rule that always fires). - Check that the alert appears in SPOG.AI’s dashboard / integration view.
- Confirm SPOG logs capture the webhook POST and payload.
- Inspect the headers and payload JSON to make sure important labels / annotations are forwarded.
- Validate error handling:
- If the webhook URL is unreachable, Alertmanager will retry per its retry logic.
- If SPOG returns a non-2xx HTTP response, Alertmanager treats it as failure.
- Confirm grouped alerts adhere to your grouping / routing logic.
- Use
amtool check-configon your Alertmanager config to catch syntax or routing misconfigurations. - Monitor for edge cases (e.g. alerts with resolve status, duplicate alerts) and how SPOG handles them.
Uninstalling the Integration
1. In SPOG.AI, go to Settings → Integrations, find the Prometheus integration, and Remove / Uninstall it.
2. In Prometheus / Alertmanager:
- Remove the SPOG receiver from alertmanager.yml.
- Remove references to it in your route configuration.
- Restart Alertmanager.
3. Optionally, clean up firewall or DNS rules associated with the webhook endpoint.
4. Confirm no further webhook hits from Alertmanager to SPOG and that SPOG no longer shows new alerts from Prometheus.