Skip to content

Datadog Module

Terraform module for deploying Datadog monitoring to Kubernetes using the Datadog Operator. Provides cloud-based infrastructure monitoring, APM, and log management as an alternative to self-hosted observability.

Architecture

flowchart TB
    subgraph external [Datadog Cloud]
        DatadogSaaS[Datadog Platform]
        Dashboard[Dashboards]
        Alerts[Alerting]
        APM[APM]
    end

    subgraph k8s [Kubernetes Cluster]
        subgraph ns [Namespace: datadog]
            Operator[Datadog Operator]

            subgraph agent [DatadogAgent CR]
                ClusterAgent[Cluster Agent]
                NodeAgent[Node Agent DaemonSet]
            end

            subgraph secrets [Secrets]
                APIKey[API Key Secret]
                APPKey[APP Key Secret]
            end
        end

        subgraph workloads [Monitored Workloads]
            Pods[Application Pods]
            Nodes[Kubernetes Nodes]
            Services[Services]
        end
    end

    Operator -->|manages| agent
    agent --> secrets

    NodeAgent -->|collect metrics| Nodes
    NodeAgent -->|collect logs| Pods
    ClusterAgent -->|collect events| Services

    NodeAgent -->|send| DatadogSaaS
    ClusterAgent -->|send| DatadogSaaS

    DatadogSaaS --> Dashboard
    DatadogSaaS --> Alerts
    DatadogSaaS --> APM

Data Flow

sequenceDiagram
    participant App as Application
    participant Agent as Datadog Agent
    participant Cluster as Cluster Agent
    participant DD as Datadog Cloud

    App->>App: Generate metrics/logs/traces
    Agent->>App: Collect container metrics
    Agent->>Agent: Collect node metrics
    Cluster->>Cluster: Collect cluster events
    Agent->>DD: Ship metrics and logs
    Cluster->>DD: Ship cluster metadata
    DD->>DD: Process and store
    DD-->>User: Dashboards and alerts

Resources Created

  • kubernetes_namespace.datadog - Dedicated namespace
  • kubernetes_secret.datadog_api_key - Datadog API key
  • kubernetes_secret.datadog_app_key - Datadog APP key
  • helm_release.datadog_operator - Datadog Operator
  • kubectl_manifest.datadog_agent - DatadogAgent custom resource
  • kubectl_manifest.datadog_dashboard - Pre-configured dashboard

Variables

Name Description Default
datadog_cluster_name Cluster identifier in Datadog (required)
datadog_site Datadog site (e.g., datadoghq.com, datadoghq.eu) (required)
datadog_api_key Datadog API key (required, sensitive)
datadog_app_key Datadog APP key (required, sensitive)

Usage

1. Get Datadog Keys

  1. Log in to Datadog
  2. Go to Organization Settings > API Keys
  3. Create or copy an API key
  4. Go to Application Keys and create an APP key

2. Enable Module

Set in Terraform Cloud or .env:

TF_VAR_datadog_enable=true
TF_VAR_datadog_cluster_name="homelab"
TF_VAR_datadog_site="datadoghq.com"  # or datadoghq.eu for EU
TF_VAR_datadog_api_key="your-api-key"
TF_VAR_datadog_app_key="your-app-key"

3. Deploy

task stage2:terraform:apply

4. Verify in Datadog Console

  1. Go to Infrastructure > Kubernetes
  2. Your cluster should appear within a few minutes
  3. Check Metrics > Explorer for kubernetes.* metrics

Helm Chart

Property Value
Repository https://helm.datadoghq.com
Chart datadog-operator

Features Enabled

Feature Description
Infrastructure Monitoring Node, pod, and container metrics
Log Collection Container logs, plus each node's systemd journal. See Node Journal Logs
APM Distributed tracing (requires app instrumentation)
Cluster Agent Kubernetes events and metadata
Admission Controller Auto-injection of tracing libraries
NPM Disabled - see Cost Considerations
kube-apiserver metrics Collected every 60s instead of the default 15s, to cut agent CPU

Integration Checks From Pod Annotations

Each workload declares its own integration check in a pod annotation, so a new workload needs no change to this module. Passwords are never written into the annotation: the agent reads them from a Kubernetes Secret through the k8s.secrets secret backend, referenced as ENC[<namespace>/<secret>;<key>].

metadata:
  annotations:
    ad.datadoghq.com/<container>.checks: |
      {"redisdb": {"instances": [{"host": "%%host%%", "port": 6379, "password": "ENC[<namespace>/<secret>;<key>]"}]}}
  • <container> is the container name in the pod, not the image name.
  • The Secret must be in the check's own namespace (the pod's, or the Service's for a cluster check). The node agent, cluster agent and cluster checks runner all refuse a handle that points anywhere else.
  • The Operator grants the agents get on Secrets cluster-wide (ClusterRole datadog-datadog-agent-secret-backend). That is the cost of not listing each Secret here.
  • The shipped redisdb auto-configuration is disabled. It carries no password, so a Redis or Valkey without an annotation is not monitored.

Verify from the agent on the pod's node:

P=$(kubectl -n datadog get pod -l app.kubernetes.io/component=agent --field-selector spec.nodeName=<node> -o name)
kubectl -n datadog exec $P -c agent -- agent status | grep -A8 '    redisdb'
kubectl -n datadog exec $P -c agent -- agent secret

A resolved handle is listed under "Secrets handle resolved". A refused one logs secret not resolved.

See Datadog secrets management and Kubernetes integrations.

Node Journal Logs

The node agent also tails each node's systemd journal: kernel, kubelet, containerd, iscsid and the other host services. Container output is not duplicated, because containerd writes it to /var/log/pods rather than the journal.

  • The agent mounts /var/log/journal and /etc/machine-id from the host read-only, as the journald integration requires for a containerized agent.
  • Each log's source and service come from the first of SYSLOG_IDENTIFIER, _SYSTEMD_USER_UNIT, _SYSTEMD_UNIT and _COMM that the entry carries, so search with source:kernel or source:kubelet.
  • A new source starts at the end of the journal. Existing history is not backfilled.
  • UFW logs one kernel line per allowed packet, which was most of the control plane's journal (about 1.3M of 1.66M lines a day). The agent drops [UFW ALLOW] and [UFW AUDIT] lines before sending. [UFW BLOCK] lines still ship.

Verify from the agent on a node:

P=$(kubectl -n datadog get pod -l app.kubernetes.io/component=agent --field-selector spec.nodeName=<node> -o name)
kubectl -n datadog exec $P -c agent -- agent status | grep -A6 'journald'

Datadog Sites

Site URL Region
US1 datadoghq.com US
US3 us3.datadoghq.com US
US5 us5.datadoghq.com US
EU1 datadoghq.eu EU
AP1 ap1.datadoghq.com Asia-Pacific

Cost Considerations

Datadog pricing is based on:

  • Number of hosts
  • Log ingestion volume
  • APM spans
  • Custom metrics

For homelab use, consider:

  • Free tier: 5 hosts, limited retention
  • Reducing log volume with filtering. Node journal logs are billed as ingested logs; the UFW filter above is the main reduction
  • Disabling APM if not needed

NPM (features.npm) is disabled. It bills a per-host NPM charge monthly, and this cluster never emitted the network.*/dns.* data to justify it. Disabling it does not remove the system-probe container, which features.oomKill requires independently.

References