Skip to content

Datadog Span Attributes

This reference is part of the Datadog integration. It lists the scheduled-mode, backfill, and log_comment fields Click-Dog exposes for Datadog searches, facets, dashboards, and monitors.

Click-Dog spans appear in Datadog with these attributes:

From Scheduled Mode (span log)

Datadog Attribute Source
Service exporters.otel[].service_name config
Operation operation_name from span
Duration Computed from span start/finish timestamps
hostname ClickHouse server hostname
duration_ms Duration in milliseconds (numeric, queryable)
db.statement SQL query text
query_log.client_address Client IP, from live query_log enrichment. ClickHouse does not put a client address on span-log rows, so scheduled mode has no client.address attribute — the backfill table below does
kind Span kind (INTERNAL, SERVER, CLIENT, etc.)
query_log.query_duration_ms Query duration from live query_log enrichment (numeric)
query_log.read_rows / query_log.read_bytes Data read from live query_log enrichment (numeric)
query_log.written_rows / query_log.written_bytes Data written from live query_log enrichment (numeric)
query_log.result_rows / query_log.result_bytes Result size from live query_log enrichment (numeric)
query_log.memory_usage Peak memory use from live query_log enrichment (numeric)
query_log.exception_code ClickHouse exception code from live query_log enrichment (numeric, when non-zero)
query_log.operation Lowercase ClickHouse statement type (select, insert, alter, etc.)
query_log.access_type Coarse activity class: read, write, ddl, admin, or other
query_log.normalized_query_hash Query-family grouping hash (intentionally a string, because it is a dimension, not a measure)
query_log.databases Databases accessed from query_log enrichment (array facet)
query_log.tables Tables accessed from query_log enrichment (array facet)

The numeric query_log.* fields above are emitted as OTLP integer measures. ClickHouse stores read_bytes / written_bytes / memory_usage as UInt64, so a value above 2^63 (≈9.2 EB / 9.2×10¹⁸) is emitted as a string instead (OTLP integers are signed 64-bit). This is rare in practice, but a measure widget over these fields should tolerate the occasional string-typed point.

If you previously created Datadog facets for query_log.databases or query_log.tables while they were joined strings, recreate those facets after upgrading so new spans are indexed as multi-value array facets. Historical spans keep the old joined-string shape.

From Backfill Mode (query log)

Datadog Attribute Source
Service exporters.otel[].service_name config
Operation clickhouse.query
db.system clickhouse
db.statement SQL query text
db.query_duration_ms Query duration in ms
db.user ClickHouse user
client.address Client IP
db.databases Databases accessed
db.tables Tables accessed
db.read_rows / db.read_bytes Data read
db.memory_usage Memory used
error true if query failed

log_comment Attributes (Scheduled Mode)

If your ClickHouse clients set the log_comment query setting with JSON, click-dog extracts the keys as span attributes. For example, a query with log_comment={"app": "Dash8", "query_name": "what_watched"} produces:

Attribute Value
@log_comment.app Dash8
@log_comment.query_name what_watched

These appear as filterable facets in Datadog APM. See Span Attributes.

Dashboards & Monitors

Use these attributes to build Datadog dashboards and monitors:

Slow query monitor:

avg:trace.duration{service:click-dog-monitor} > 5000000000

Query by duration_ms attribute:

@duration_ms:>5000

Filter by ClickHouse host:

@hostname:clickhouse-prod-01

Filter by app (via log_comment):

@log_comment.app:Dash8

Next: provision and understand the packaged dashboards.