Datadog Integration¶

Click-Dog's packaged Datadog path includes OTLP trace export, the Application Query Analysis dashboard, the Exported User Activity dashboard, the Click-Dog: Health dashboard, and recommended monitors for exporter health and query regressions. You can send traces directly to the Datadog Agent or through an OpenTelemetry Collector. Separately, an explicit query-analysis run can post a bounded finding summary to Datadog Event Management.

Fast Path¶

Enable OTLP gRPC on the Datadog Agent.
Install Click-Dog with the guided quick start.

Run the readiness check:

sudo -u click-dog click-dog check --config /etc/click-dog/click-dog.yaml

Provision the packaged dashboards:

DD_API_KEY=... DD_APP_KEY=... click-dog create-dashboards

In Datadog, open Click-Dog: Application Query Analysis, Click-Dog: Exported User Activity, and Click-Dog: Health.

Option 1: Datadog Agent (Recommended)¶

The Datadog Agent (v7.35+) can ingest OTLP traces directly.

Configure the Datadog Agent¶

Edit datadog.yaml or use environment variables:

datadog.yaml:

otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

Environment variables (e.g., in Docker/Kubernetes):

DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT=0.0.0.0:4317

Restart the Datadog Agent after making changes.

Configure Click-Dog¶

Point click-dog at the Datadog Agent's OTLP endpoint:

These are configuration fragments

Merge the Click-Dog exporter/TLS blocks on this page into a complete config such as the minimal profile. They do not include the ClickHouse and required scheduled-monitor sections.

exporters:
  otel:
    - collector_address: localhost:4317
      service_name: click-dog-monitor

If the Datadog Agent is on a different host:

exporters:
  otel:
    - collector_address: datadog-agent.internal:4317
      service_name: click-dog-monitor

With TLS¶

If your Datadog Agent requires TLS:

exporters:
  otel:
    - collector_address: datadog-agent.internal:4317
      service_name: click-dog-monitor
      secure: true
      ca_cert: /path/to/ca.pem

Verify¶

Before sending traffic, confirm the ClickHouse data plane these dashboards rely on is actually ready:

sudo -u click-dog click-dog check --config /etc/click-dog/click-dog.yaml

This validates that system.opentelemetry_span_log and system.query_log are readable and granted, that recent spans exist, that min_trace_duration_ms isn't filtering everything out, that spans carry clickhouse.query_id when query-log enrichment or user filters are configured (otherwise that probe reports SKIP), and that normalized_query_hash is available (so the query-family widgets populate). Warn/fail lines print the exact fix.

First isolate exporter credentials, TLS, and routing without querying ClickHouse:

sudo -u click-dog click-dog test export --config /etc/click-dog/click-dog.yaml

It is sent to every configured exporter and carries click_dog.test=true, so it is easy to find or exclude in the backend. Then prove native ClickHouse trace-context propagation and span-log recovery:

sudo -u click-dog click-dog test tracing --config /etc/click-dog/click-dog.yaml

Both commands report exporter acceptance, not confirmation that Datadog has indexed the trace. On tracing PASS, use the printed trace_id for the backend search.

Then, in Datadog, navigate to APM > Traces and search for: - Service: click-dog-monitor - Look for spans with db.statement, duration_ms, and hostname attributes

Option 2: OpenTelemetry Collector¶

If you're already running an OTEL Collector, you can route click-dog traces through it.

OTEL Collector Configuration¶

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  datadog:
    api:
      key: ${DD_API_KEY}
      site: datadoghq.com      # or datadoghq.eu, etc.

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [datadog]

Click-Dog Configuration¶

exporters:
  otel:
    - collector_address: otel-collector.internal:4317
      service_name: click-dog-monitor

Example: Full Production Setup¶

clickhouse:
  host: ${CLICKHOUSE_HOST}
  port: 9440
  database: default
  username: ${CLICKHOUSE_USERNAME}
  password: ${CLICKHOUSE_PASSWORD}
  secure: true
  ca_cert: /etc/ssl/clickhouse-ca.pem

exporters:
  otel:
    - collector_address: ${DD_AGENT_HOST}:4317
      service_name: click-dog-monitor
      secure: true
      ca_cert: /etc/ssl/dd-ca.pem

monitor:
  enabled: true
  min_trace_duration_ms: 1000
  check_interval_s: 30
  max_spans_per_cycle: 500
  batch_size: 50
  batch_delay_ms: 100
  circuit_breaker:
    enabled: true
    failure_threshold: 5
  backoff:
    enabled: true

filters:
  blacklist_queries:
    - "^SELECT \\* FROM system\\."
    - "(?i)healthcheck"

log_level: info

Query-analysis events¶

The optional datadog_events destination sends one structured custom alert event when click-dog analyze queries --notify has eligible conditions. It uses the Datadog Events API v2 directly; it does not flow through the Agent, OTLP trace exporter, or OTLP self-metric exporter, and it never runs analysis on a schedule by itself.

Create a Datadog API key and application key, choose the site hostname for the same Datadog organization, and configure:

datadog_events:
  enabled: true
  site: datadoghq.com
  api_key: ${DD_API_KEY}
  application_key: ${DD_APP_KEY}
  timeout_s: 10
  environment: production
  service: click-dog

For file-mounted secrets, replace either inline key independently with api_key_file or application_key_file; do not set both forms for the same key. Click-Dog derives the endpoint as https://event-management-intake.<site>/api/v2/events, sends both Datadog authentication headers, follows no redirects, makes no retries, and accepts only an HTTP 2xx response. Treat site as trusted operator configuration: provide only your Datadog site hostname, without a scheme, path, or port.

Run analysis with a retained local report:

sudo -u click-dog click-dog analyze queries \
  --config /etc/click-dog/click-dog.yaml \
  --baseline /var/lib/click-dog/query-baseline.json \
  --format json --output /var/lib/click-dog/latest-analysis.json \
  --notify --fail-on critical

A success line means Datadog intake accepted the event. It does not prove that Datadog indexed it, evaluated an Event Monitor, or sent a downstream alert. In Event Explorer, start with:

event_type:analysis_findings environment:production service:click-dog

Events are categorized as alert with integration custom-events. Critical summaries use status error and priority 1; warning-only summaries use status warn and priority 3. Fixed tags expose source, event_type, environment, service, highest eligible severity, and the report and notification schema versions. The nested custom attributes carry the same bounded, privacy-reviewed summary as the webhook destination.

See Query Analysis for eligibility, schema, limits, and exit-code precedence, and Recommended monitors for an Event Monitor recipe.

Continue with Datadog¶

Use the focused guides for the rest of the packaged Datadog experience:

Span attributes: scheduled-mode, backfill, and log_comment fields exposed in Datadog.

Dashboards: provisioning, data-source contracts, and dashboard layouts.

Self-monitoring: OTLP health metrics, legacy OpenMetrics mapping, data-plane signals, and health endpoints.

Recommended monitors: exporter health, query regressions, and fallback alerts.