Datadog
Observability Platform

Agent Internal Components

Collector — runs all configured checks and gathers metrics on a 15–20 second interval. Passes output to the local Aggregator and Forwarder.

Forwarder — sends payloads to Datadog over HTTPS. Buffers metrics in memory during network outages, preventing data loss. Discards oldest data only at memory limits.

APM Agent — separate optional process collecting distributed traces. Enabled by default. Listens on port 8126.

Process Agent — collects live process and container info. Requires explicit enablement for full process monitoring.

DogStatsD — Go implementation of StatsD. Accepts custom metrics over UDP (port 8125) or Unix socket, aggregates and forwards to backend.

Data Retention

Deployment Options

Integration Check Config

YAML — conf.d/nginx.d/conf.yaml
init_config:

instances:
  - nginx_status_url: http://localhost/nginx_status
    tags:
      - service:nginx
      - env:production

K8s — Autodiscovery annotations
annotations:
  ad.datadoghq.com/nginx.check_names:  '["nginx"]'
  ad.datadoghq.com/nginx.init_configs: '[{}]'
  ad.datadoghq.com/nginx.instances:
    '[{"nginx_status_url":"http://%%host%%/nginx_status"}]'

Metric Types

Submitting Custom Metrics via DogStatsD

PYTHON
from datadog import initialize, statsd
initialize(statsd_host='localhost', statsd_port=8125)

# Gauge
statsd.gauge('app.queue.depth', 42,
  tags=['env:prod', 'service:worker'])

# Increment counter
statsd.increment('app.page.views',
  tags=['page:home'])

# Histogram (timing)
statsd.histogram('db.query.time', 0.042,
  tags=['query:get_user'])

# Distribution (global percentiles)
statsd.distribution('api.response.time', 125.3,
  tags=['endpoint:/checkout'])

APM Metric Namespaces

Enabling Log Collection

YAML — datadog.yaml
logs_enabled: true

YAML — conf.d/python.d/conf.yaml
logs:
  - type:    file
    path:    /var/log/myapp/*.log
    source:  python
    service: my-service
    tags:
      - env:production

DOCKER — label-based collection
labels:
  com.datadoghq.ad.logs:
    '[{"source":"nginx","service":"web"}]'

Log Limits

Storage Tiers

Instrumentation Methods

Key APM Features

Python APM

PYTHON
pip install ddtrace

# Auto-instrument at startup (recommended)
DD_SERVICE="my-api" DD_ENV="prod" DD_VERSION="1.2.0" \
  ddtrace-run python app.py

# Manual span creation
from ddtrace import tracer

with tracer.trace("db.query", resource="SELECT users") as span:
    span.set_tag("db.type", "postgres")
    result = db_query()

Java APM

JAVA — JVM flag
java -javaagent:/path/to/dd-java-agent.jar \
  -Ddd.service=my-app \
  -Ddd.env=production \
  -Ddd.version=1.0.0 \
  -jar app.jar

OpenTelemetry Collector

YAML — OTel Collector Datadog Exporter
exporters:
  datadog:
    api:
      key:  ${DD_API_KEY}
      site: datadoghq.com
    traces:
      compute_stats_by_span_kind: true

service:
  pipelines:
    traces:
      exporters: [datadog]
    metrics:
      exporters: [datadog]

Infrastructure Views

• Infrastructure List — every host with key metrics and tag filtering
• Host Map — hexagonal heatmap of all hosts by any metric
• Containers page — resource metrics and faceted search across containers
• Container Images — every image in your env + vulnerability data
• Orchestrator Explorer — monitor pods, deployments, namespaces
• Control Plane Monitoring — API server, scheduler, controller manager, etcd
• Live Processes — real-time process list with CPU, memory, I/O
• Network Performance Monitoring — eBPF-based traffic flow visibility

Key System Metrics

Cloud Integrations

Kubernetes Cluster Agent

The Cluster Agent efficiently gathers monitoring data from across an orchestrated cluster. It distributes check configurations to node Agents and ensures only one instance of each check runs per workload — preventing duplicate data collection across replicas.

The Cluster Agent holds configs and dispatches them to node Agents every 10 seconds. If a node Agent stops reporting, the Cluster Agent removes it from the active pool and redistributes its configurations.

Serverless

For AWS Lambda, Datadog collects metrics, traces, and logs via the Lambda Extension (preferred, runs in-process) or Lambda Forwarder (CloudWatch-based). Supports enhanced Lambda metrics, cold start detection, and X-Ray integration.

Monitor — Terraform

TERRAFORM
resource "datadog_monitor" "high_cpu" {
  name    = "High CPU Usage"
  type    = "metric alert"
  message = "CPU > 90% on {{host.name}} @pagerduty"

  query = "avg(last_5m):avg:system.cpu.user{env:production} by {host} > 90"

  monitor_thresholds {
    critical = 90
    warning  = 75
  }
  notify_no_data    = true
  no_data_timeframe = 20
  tags = ["env:production", "team:platform"]
}

Notification Channels

Monitor Configuration Reference

Service Level Objectives (SLOs)

Three SLO types:

Dashboard Types

Widget Types

Template Variables — Terraform

TERRAFORM
resource "datadog_dashboard" "service_health" {
  title       = "Service Health"
  layout_type = "ordered"

  template_variable {
    name    = "env"
    prefix  = "env"
    default = "production"
  }

  widget {
    timeseries_definition {
      request {
        q = "avg:trace.web.request.duration{$env} by {service}"
      }
      title = "Request Latency by Service"
    }
  }
}

Synthetic Test Types

Continuous Testing (CI/CD)

BASH — datadog-ci
npm install -g @datadog/datadog-ci

# Run Synthetic tests in CI pipeline
datadog-ci synthetics run-tests \
  --public-id "abc-123-xyz" \
  --apiKey $DD_API_KEY \
  --failOnCriticalErrors

Real User Monitoring (RUM)

• Session Replay — pixel-perfect video-like replay of real user sessions
• Core Web Vitals — LCP, FID, CLS tracking with custom user timings
• Error Tracking — frontend JS errors grouped and prioritized by user impact
• RUM-APM Correlation — link frontend sessions to backend distributed traces
• Funnel Analysis — track conversion rates through multi-step user flows
• RUM Recommendations — AI-powered performance improvement suggestions (Preview)
• Mobile RUM — iOS and Android performance monitoring and crash tracking

Feature Flags & A/B Testing

Datadog Feature Flags integrates with your existing feature flag provider to track flag evaluations alongside RUM data. Correlate feature flag rollouts directly with performance regressions and error spikes in the same view.

Cloud Platforms

Databases & Caches

Message Queues & Streaming

Web Servers & Proxies

DevOps & CI/CD

Alerting & Incident Management

Observability Standards

Unified Service Tagging (Required)

ENV VARS — Kubernetes pod spec
env:
  - name:  DD_ENV
    value: production
  - name:  DD_SERVICE
    value: checkout-api
  - name:  DD_VERSION
    value: 1.4.2
  - name:  DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

Tagging Best Practices

• Use lowercase key:value format consistently everywhere
• Always tag with env: (production, staging, dev)
• Always tag with service: for service-level views
• Always tag with version: for deployment tracking
• Use team: for ownership routing in monitor notifications
• Use region: and availability-zone: for geographic scoping
• Avoid high-cardinality values on metrics (user_id, request_id) — use traces for those
• Limit custom metric tag cardinality to control monthly custom metric costs
• Apply tags at the Agent level for host-wide application

Tag Sources (All Applied Automatically)

REST API — Key Endpoints

PYTHON — API client
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.monitors_api import MonitorsApi

config = Configuration()
config.api_key["apiKeyAuth"] = "<DD_API_KEY>"
config.api_key["appKeyAuth"] = "<DD_APP_KEY>"

with ApiClient(config) as api_client:
    api = MonitorsApi(api_client)
    print(api.list_monitors())

Terraform Provider

TERRAFORM — Provider setup
terraform {
  required_providers {
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3.0"
    }
  }
}
# Auth via DD_API_KEY + DD_APP_KEY env vars
provider "datadog" {
  api_url = "https://api.datadoghq.com/"
}

Terraform Resources

datadog-ci CLI

BASH
npm install -g @datadog/datadog-ci

# Upload source maps for RUM error tracking
datadog-ci sourcemaps upload ./dist \
  --service my-app --release-version 1.4.2

# Report CI test results
datadog-ci junit upload --service my-app ./test-results.xml