Kubernetes · #azure · #ibm-db2 · #rocket-software

Monitoring IBM DB2 and Rocket Software on Azure VM’s with Managed Prometheus and Terraform

The Challenge The customer runs IBM DB2 databases and Rocket Software enterprise applications on Azure Virtual Machines. Their requirement was clear: complete observability with zero dependency on external monitoring platforms. No Datadog, no New Relic, no self-hosted…

The Challenge

The customer runs IBM DB2 databases and Rocket Software enterprise applications on Azure Virtual Machines. Their requirement was clear: complete observability with zero dependency on external monitoring platforms. No Datadog, no New Relic, no self-hosted Prometheus clusters, everything had to stay inside Azure, managed centrally, and deployable as code.

The result is a fully automated monitoring stack built on Azure Monitor Workspace (Managed Prometheus), Grafana Alloy, and purpose-built exporters, deployed entirely with Terraform.


Solution Overview

The architecture combines four components running on each monitored VM:

ComponentRole
IBM DB2 ExporterExposes DB2 internal metrics as Prometheus endpoints
Rocket Software ExporterExposes ESCWA / MFDS / ESI region metrics as Prometheus endpoints
Grafana AlloyScrapes all exporters (including node-exporter) and forwards metrics to Azure Monitor
Azure Monitor WorkspaceManaged Prometheus backend: stores, queries, and evaluates alert rules

A Service Principal, secured through Azure Key Vault, authenticates the metric publishing pipeline to Azure. Dashboards and alert rules are provisioned automatically via Terraform.

Architecture

How the Metrics Flow

  1. IBM DB2 Exporter runs as a service on the DB2 host. It connects to the DB2 engine using JDBC and exposes metrics on a local HTTP port – things like bufferpool hit ratios, active connections, deadlock counts, transaction log usage, and tablespace fill levels.
  2. Rocket Software Exporter connects to the ESCWA (Enterprise Server Common Web API) interface and exposes Prometheus metrics for MFDS (Micro Focus Directory Service) availability, ESI region status, communication server thread counts, and overall exporter health.
  3. Grafana Alloy acts as the collection agent on each VM. It has three scrape jobs configured:
    • node_exporter job: scrapes the host’s node-exporter on port 9100 for CPU, memory, disk, network, and filesystem metrics
    • ibm_db2 job: scrapes the DB2 Exporter endpoint
    • rocket_software job: scrapes the Rocket Software Exporter endpoint
    Alloy then forwards all collected metrics to the Azure Monitor Workspace using the Prometheus Remote Write protocol, authenticating with the Service Principal credentials sourced from Azure Key Vault.
  4. Azure Monitor Workspace receives all metrics, stores them in its Managed Prometheus backend, and continuously evaluates the 58 configured alert rules at 10-minute intervals.
  5. When a rule fires, Azure Monitor Action Groups route the alert to the right team — DB2 alerts to the DB2 operations team, Rocket Software alerts to the enterprise applications team, and infrastructure alerts to the platform team.

What Gets Monitored

IBM DB2 – 26 Alert Rules

The IBM DB2 Exporter surfaces the full operational picture of a DB2 instance:

Availability & Connectivity

  • DB2Down: instance unreachable (Severity 1)
  • Active application counts and connection pool depth

Performance

  • DB2CriticalBufferpoolHitRatio: bufferpool cache hit rate drops below threshold; this is the single most important DB2 performance indicator
  • Row operation rates: insert, update, delete spikes that indicate abnormal workload

Reliability

  • DB2DeadlocksDetected and DB2HighDeadlockRate: transaction contention
  • DB2CriticalWaitingLocks: lock queue depth indicating blocking queries
  • DB2LogUsageCritical: transaction log space, the resource that causes DB2 to halt on exhaustion
  • DB2TablespaceCritical: tablespace fill levels across all containers

Rocket Software – 5 Alert Rules

The Rocket Software Exporter monitors the ESCWA-managed application tier:

  • ESExporterDown: exporter itself is unreachable
  • MFDSDisconnected: MFDS (the central directory service for Enterprise Server) has lost connectivity
  • ESIRegionDown: one or more ESI (Enterprise Server Image) application regions are not running
  • Communication server thread failures – comms threads below minimum threshold
  • ESCWA restart required – detects when a configuration change requires ESCWA restart

Infrastructure (node-exporter) – 27 Alert Rules

Host-level metrics cover every layer of the OS:

  • Memory: available memory, memory pressure, swap usage
  • Disk: space, inodes, I/O latency, RAID drive sufficiency
  • CPU: load average, I/O wait, CPU steal, context switch rate
  • Network: bandwidth saturation, error rates, bond interface health
  • System: systemd service failures, NTP drift, kernel version changes, OOM events, hardware temperature, EDAC memory errors

Azure Dashboards

Three dashboards are provisioned directly into the Azure Portal using the native Grafana-compatible dashboard API. No separate Grafana instance is required.

DashboardKey Panels
Alloy-Node-Performance-MetricsCPU utilization, memory usage, disk I/O, network throughput per host
IBM-DB2-Exporter-DashboardConnections over time, bufferpool hit ratio, deadlock rate, log usage %, tablespace fill
Rocket-SoftwareMFDS state, ESI region count, comms threads, ESCWA health

Identity and Security

The entire metric publishing pipeline is secured with Azure-native identity:

  • An App Registration (tf-app-registration-prometheus) is created in Azure AD
  • A Service Principal is derived from the app registration with the Monitoring Metrics Publisher role scoped to the resource group – the minimum required to write metrics to the Monitor Workspace
  • The client secret is auto-generated by Terraform with a 365-day rotation cycle and stored in Azure Key Vault (kv-prometheus-monitoring)
  • Grafana Alloy reads the secret from Key Vault at startup and uses it for Remote Write authentication

This means no credentials are baked into VM images or config files. Key Vault is the single source of truth, and secret rotation is handled automatically by Terraform on the next apply cycle after expiry.


Terraform Design

The entire stack is deployed with Terraform across three logical layers:

Identity & Secrets Layer

  • Azure AD App Registration + Service Principal
  • Key Vault + Secret with auto-rotation
  • RBAC role assignments

Monitoring Backend Layer

  • Azure Monitor Workspace (the Managed Prometheus endpoint)
  • 5 Prometheus Rule Groups loading 58 alert rules from external JSON files
  • 3 Action Groups with email and webhook routing per product team

Visualization Layer

  • 3 Azure Portal dashboards deployed via the azapi provider using the preview Dashboard API (Microsoft.Dashboard/dashboards@2025-09-01-preview)

One architectural choice worth noting: alert rules are defined in JSON files in a shared directory (shared/prometheus-rules/) and loaded by Terraform using jsondecode(file(...)). This decouples rule authoring from Terraform operations – the operations team can update an alert threshold by editing a JSON file and running terraform apply, without touching any .tf code.


Why This Approach

No External SaaS — Everything runs in Azure. Metrics never leave the customer’s Azure tenant. This satisfies network isolation and data sovereignty requirements.

No Self-Hosted Prometheus: Azure Monitor Workspace handles HA, retention, and scaling. There are no Prometheus servers, Alertmanager clusters, or storage backends to operate.

Single Deployment Unit: One Terraform workspace deploys the full monitoring stack: identity, secrets, metric backend, alert rules, action groups, and dashboards. The same code structure is replicated for stage and production with different tfvars.

Product-Team Alignment: Alert routing is separated by product (DB2 team, Rocket team, platform team). Each team only receives alerts relevant to their systems.

Extensible: Adding a new exporter requires adding a scrape job to Alloy, a new JSON rule file in shared/prometheus-rules/, and a new entry in terraform.tfvars. No Terraform resource code changes needed.


Dashboards

Summary

This solution delivers enterprise-grade observability for IBM DB2 and Rocket Software on Azure, with 58 pre-built alert rules, three native dashboards, and fully automated identity management – all without a single external monitoring tool. The combination of Azure Monitor Workspace’s Managed Prometheus, Grafana Alloy’s flexible scraping, and Terraform’s infrastructure-as-code discipline makes the stack reproducible, auditable, and operationally lean.

Repo link: https://github.com/machani/azure-managed-prometheus-tf

Cya soon!

Responses

No comments yet · be the first