Skip to content

Datadog Dashboards

This guide is part of the Datadog integration. It covers provisioning and the data contracts behind the three packaged dashboards.

Click-dog ships three Datadog dashboards:

  • Click-Dog: Application Query Analysis (datadog-query-analysis.json): live-span dashboard for system.opentelemetry_span_log data, filtered with resource_name:query @click_dog.source:span_log and grouped by log_comment.* plus live query_log.* enrichment
  • Click-Dog: Exported User Activity (datadog-user-activity.json): searchable live-span relationships from users to databases, tables, statement operations, and access classes; an operational view of exported activity, not an audit log
  • Click-Dog: Health (datadog-clickdog-health.json): self-monitoring metrics for current export state, error/export rates, backoff, circuit breaker, and lifetime counters

Option 1: click-dog command

DD_API_KEY=... DD_APP_KEY=... click-dog create-dashboards

Option 2: Datadog API

From the repository root, import each shipped JSON definition:

for dashboard in \
  datadog-query-analysis.json \
  datadog-user-activity.json \
  datadog-clickdog-health.json
do
  curl -X POST "https://api.datadoghq.com/api/v1/dashboard" \
    -H "Content-Type: application/json" \
    -H "DD-API-KEY: ${DD_API_KEY}" \
    -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
    -d "@dashboards/${dashboard}"
done

The Application Query Analysis and Exported User Activity dashboards require click-dog v26.03.1 or newer for their @click_dog.source:span_log filter. If you cannot upgrade immediately, remove that filter from widget queries after import.

For EU or other regions, replace datadoghq.com with your site (e.g. datadoghq.eu).

Running on Kubernetes?

The dashboards only show data once ClickHouse is actually emitting spans and click-dog is exporting them. For the operator-based ClickHouse setup (enabling system.opentelemetry_span_log cluster-wide, the read-only monitoring user, sidecar vs centralized topology, and OTLP/Agent wiring), follow Kubernetes.

Query dashboard data-source contract

The shipped Application Query Analysis dashboard is live-span-only. It is covers Click-Dog's scheduled mode, where ClickHouse native spans from system.opentelemetry_span_log arrive in Datadog as click_dog.source=span_log with resource_name:query. Its filters and template variables use live-span fields:

  • @log_comment.app and @log_comment.query_name from ClickHouse client tagging
  • @query_log.user, @query_log.client_name, @query_log.normalized_query_hash, and related query_log.* enrichment joined onto live spans
  • @query_log.read_rows, @query_log.memory_usage, and other live query_log stats as numeric OTLP attributes for measure-style widgets

Compatibility: the @click_dog.source:span_log filter requires click-dog v26.03.1 or newer, or any build that emits the click_dog.source attribute. Older binaries can still export spans, but this dashboard's source filter will not match them until click-dog is upgraded. If you cannot upgrade immediately, remove @click_dog.source:span_log from widget queries after import to restore the previous resource_name:query-only matching behavior.

Backfill mode is intentionally separate. Backfill reads system.query_log and exports single-span traces named clickhouse.query with click_dog.source=query_log and db.* attributes (db.user, db.read_rows, db.memory_usage, db.normalized_query_hash, and so on). If you import only historical backfill data, the shipped query dashboard may show little or no data. For backfill analysis, use Datadog trace search or custom widgets with a query such as:

service:click-dog-monitor resource_name:clickhouse.query @click_dog.source:query_log

Group those views by backfill fields such as @db.user, @client.name, @db.normalized_query_hash, @db.tables, or @hostname.

User activity dashboard data-source contract

The shipped Exported User Activity dashboard uses the same live root-query contract as Application Query Analysis, but focuses on relationships between @query_log.user, @query_log.databases, @query_log.tables, @query_log.operation, and @query_log.access_type. Its table widgets keep a search bar visible, and its template variables provide searchable filters for each dimension.

query_log.operation is read from ClickHouse's parsed system.query_log.query_kind column and normalized to lowercase. Click-dog maps it to query_log.access_type (read, write, ddl, admin, or other) without inferring activity from row counters. In cluster query mode those two fields are emitted only when every replica exposes query_kind; other enrichment fields continue to work if that optional capability is unavailable. The click_dog.query_operation_supported self-metric is 1 when the activity dimensions are enabled and 0 when the startup capability probe disabled them.

This dashboard is not a compliance or audit surface. Its counts include only the exported/qualified stream selected by monitor.min_trace_duration_ms and the configured query/operation/IP/user filters. A complete activity record requires a dedicated system.query_log pipeline with suitable retention.

User activity dashboard layout

Row 1: Exported activity summary
  [ Exported activity ] [ Active users ] [ Reads ] [ Writes ] [ DDL ] [ Failures ]

Row 2: Activity over time
  [ Exported activity by access type ]

Rows 3–6: Searchable relationship tables
  [ User + access type ]
  [ User → database + access type ]
  [ User → table + operation ]
  [ User → operation + access type ]

Query dashboard layout

Key Metrics
  [ Exported queries (1h) ] [ p95 latency ] [ p99 latency ] [ Slow queries (>1s) ]
  [ Top user ] [ Top client ]
  [ Exported queries per minute ] [ Query latency (p50, p95, p99) ]

Failures                      (query_log enrichment)
  [ Failed queries ] [ Top failing query families ]

By ClickHouse User & Client   (query_log enrichment)
  [ Top users by exported volume ] [ Top client libraries ] [ Top users by p95 latency ]

Per-host Distribution
  [ Exported queries by host ]

Resource Usage                (query_log enrichment)
  [ Families by total query time ] [ Families by rows read (p95) ] [ Families by peak memory (p95) ]
  [ Exported queries by tables accessed ] [ ... by databases accessed ]

Exact Query Families
  [ Slowest exact query families ]

By Application                (log_comment tagging)
  [ Top apps by exported volume ] [ Top apps by p95 latency ] [ Slow queries (>1s) by app ]

By Named Query                (log_comment tagging)
  [ Top named queries by p95 latency ] [ Top named queries by exported volume ]

Volume/count widgets show the exported/qualified query stream, not total ClickHouse query volume. Scheduled mode exports only traces selected by monitor.min_trace_duration_ms and any other query/operation/IP filters, so with the quickstart default min_trace_duration_ms: 1000 these counts reflect qualifying slow-query traces. For total ClickHouse query volume, use the official Datadog ClickHouse integration.

Next: send Click-Dog health metrics to Datadog.