If you've ever set up a monitoring dashboard by hand; clicking through a UI, dragging panels around, tweaking a query, saving, refreshing, tweaking again then you already know the pain this article is about to solve.
Definition
An observability stack relies on DaemonSets to gather data from a cluster. A DaemonSet is a Kubernetes controller that ensures at least one pod runs on every (or a specified subset of) node in the cluster. Because they guarantee node-level coverage, DaemonSets are the natural fit for workloads like metrics collection, logging, tracing and anything that needs eyes on every node.
This is what powers tools like node-exporter (metrics) or fluentd/filebeat (logs). One instance per node, continuously sending data back to a central store like Prometheus or an ELK stack.
Observability as Code (OaC) is simply defining your observability stack; be it dashboards, alerts, panels, thresholds, etc, using code instead of a conventional UI. Where a traditional setup means clicking through a dashboard builder to manually create panels, OaC works the same way Infrastructure as Code (IaC) does for your infrastructure: you write a small, usually JSON-like configuration that fully describes what you want, and the tooling renders it for you.
Whether it's logging, monitoring, tracing, or alerting, all of it can be expressed in code and rendered consistently on a dashboard.
Why This Matters
Before diving into the mechanics, it's worth being upfront about why this approach is worth adopting into your current architecture.
Version control: your dashboards live in Git, not in someone's browser session. You can see exactly who changed what and when, you can also roll it back like any other code change.
Reproducibility: spinning up a new environment (staging, a new cluster, a DR region) becomes as easy as applying the same config, not manually rebuilding dashboards from memory or a screenshot.
Review and collaboration: dashboard changes can go through pull requests and reviews like everything else the team ships. No more quiet changes without reason or general communication.
Consistency across clusters: if you're running multiple clusters (dev/staging/prod, or multi-region), OaC guarantees they all observe the same things, the same way.
Faster incident response setup: spinning up a new alert or panel during an active incident becomes a
git commit, not a multi-step UI workflow while under pressure.
If you've worked with Terraform or any IaC tool, this flow will feel familiar, you're basically applying the exact same discipline to your observability layer that you already apply to your infrastructure layer.
How It Works
For this example, we'll set use a metrics/monitoring stack with Grafana + Prometheus and build dashboard panels entirely through code. Here's a link to the complete kubernetes minikube setup for a local/test environment.
Every Grafana dashboard, under the hood, is just a JSON document. At the top level, it declares metadata about the dashboard itself:
{
"title": "Cluster Overview",
"uid": "cluster-overview",
"schemaVersion": 38,
"version": 1,
"editable": true,
"timezone": "browser",
"time": {
"from": "now-6h",
"to": "now"
}
}A quick breakdown of what each field is doing:
title: the human-readable dashboard name, shown in the Grafana UI.uid: a unique identifier for the dashboard. This matters more than it looks: it's what lets you reference this exact dashboard from code (CI/CD pipelines, provisioning scripts) without relying on the title, which could change.schemaVersion: tells Grafana which version of the dashboard JSON schema this file follows, so it renders correctly across Grafana upgrades.editable: whether users can modify the dashboard from the UI after it's provisioned. Teams enforcing strict OaC often set this tofalse, so the only way to change the dashboard is through code which prevents drifts between what's committed and what's live.timezone:"browser"renders the time in the viewer's local timezone rather than a fixed one.time: the default time range shown when the dashboard loads (now-6htonowmeans "the last 6 hours" by default).
That's the dashboard shell. The actual data lives inside a panels arrays. Each panel is its own object defining a chart with the dashboard as a parent of all pannels. Here's a single panel added to the config above, which will be a child of the dashboard, tracking pod CPU usage cluster-wide:
{
"title": "Cluster Overview",
"uid": "cluster-overview",
"schemaVersion": 38,
"version": 1,
"editable": true,
"timezone": "browser",
"time": {
"from": "now-6h",
"to": "now"
},
"panels": [
{
"id": 1,
"title": "CPU Usage by Node",
"type": "timeseries",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
"targets": [
{
"expr": "sum(rate(container_cpu_usage_seconds_total[5m])) by (node)",
"legendFormat": "{{node}}"
}
]
}
]
}
Breaking down what's new here:
type: the visualisation type (timeseries,gauge,stat,table, etc.). Grafana renders the same underlying data differently depending on what you set here.gridPos: the panel's position and size on the dashboard grid (h/wfor height/width,x/yfor coordinates). This is how you lay out a dashboard entirely in code instead of dragging panels around visually.targets: the actual query (or queries) powering the panel. Theexprfield is a raw PromQL query, it sums CPU usage rate across containers, grouped by node.legendFormat: controls how each data series is labeled in the panel's legend, using template variables pulled from your query's labels ({{node}}here).
Once this JSON is committed, it can be applied via Grafana's provisioning system (a dashboards.yaml provider pointing at a folder of JSON files) or pushed through Grafana's HTTP API in a CI/CD pipeline — meaning your dashboard now deploys the same way your application code does.
The output is the image below, a customised dashboard with well defined panels.
Making This a Habit
If you're a DevOps engineer who's used to clicking through Grafana or Kibana to set things up, the upfront cost of learning the JSON schema can feel like friction for its own sake. But the payoff is massive:
The first time a dashboard gets accidentally deleted or corrupted and you restore it with
git checkoutin ten seconds instead of rebuilding it from memory, it pays for itself.The first time you spin up a new cluster and your entire observability layer comes up automatically alongside your infrastructure, it pays for itself again.
You can catch a mistake in review before it ships, instead of a 3am troubleshooting session.
Observability as Code isn't a nice-to-have layered on top of good DevOps practice, it's the natural extension of it. If your infrastructure is code, your observability should be too, let me know what you think about OaC.
Photo credit: note