Skip to content

Splunk HEC Integration

Click-Dog can fan out span-shaped query events to Splunk HTTP Event Collector (HEC) alongside OTLP exporters. Use this when Splunk is your operational search surface or when you want ClickHouse query telemetry in an existing Splunk index.

Packaged Splunk experience

HEC export works today. Packaged Splunk dashboards, saved searches, and alert recipes are being scoped. If you are using Click-Dog with Splunk, tell us which searches or dashboards would make this production-ready via Support.

Configure Click-Dog

This is a configuration fragment

Merge this exporter block into a complete config such as the minimal profile. By itself it omits the ClickHouse connection and required scheduled-monitor threshold.

exporters:
  splunk_hec:
    - endpoint: https://splunk.internal:8088
      token_file: /etc/click-dog/splunk-hec.token
      index: clickhouse
      source: click-dog
      source_type: _json

filters:
  # Compatibility default is raw. Prefer normalized_only or none when query
  # literals may contain secrets or PII.
  query_text_mode: normalized_only

The mode is applied before HEC serialization and before multi-sink fan-out. In normalized_only, the top-level query field is omitted and a bounded normalized_query is present only when ClickHouse safely provides it. In none, both fields are absent. Neither mode falls back to raw SQL. All privacy-restricting modes also drop span attribute values carrying an HTTP query parameter before HEC serialization. See Query Text Export Modes.

Use https:// for HEC endpoints by default because the token is sent in the Authorization header. Local development receivers can use http:// only when allow_insecure_http: true is set explicitly.

token_file keeps the HEC token out of the process environment and config file. The inline token: ${SPLUNK_HEC_TOKEN} form is also supported, but token and token_file are mutually exclusive.

Endpoint and trust contract

Configure the HEC server base URL (or a proxy prefix), not the full event collector path. Click-dog always trims trailing / and appends /services/collector/event. For example, https://splunk.internal:8088 becomes https://splunk.internal:8088/services/collector/event. Supplying an endpoint that already ends in /services/collector/event double-appends that path and sends events to the wrong URL.

HEC TLS uses the service host/container's system trust store. The HEC exporter has no ca_cert setting and no client-certificate/mTLS settings. To reach a private-CA HEC endpoint, install the CA into that system trust store or place a trusted TLS proxy in front of HEC. insecure_skip_verify: true disables server certificate verification for an https:// endpoint and is only appropriate for isolated testing.

Delivery contract

Click-dog treats an HTTP 2xx response as acceptance of that request. It does not request or poll Splunk indexer acknowledgements, so a 2xx response does not prove the event was indexed. Each export call gets one HTTP attempt with no automatic retry. A failure returns to the normal click-dog cycle/backfill failure path; a later cycle or manual rerun may resend the full batch.

The HEC HTTP client has a fixed 30-second timeout. The shared monitor.export_timeout_s deadline also wraps export calls and can end one earlier; disabling that shared deadline does not remove HEC's 30-second cap.

Verify

What check verifies

Run the connectivity and data-plane checks:

sudo -u click-dog click-dog check --config /etc/click-dog/click-dog.yaml

For HEC, check derives /services/collector/health/1.0 from the configured base, sends the configured token in the Authorization header, and verifies that an HTTP server is reachable. It accepts 200, 400, or 404 as a connectivity success because some reachable deployments do not expose that health route. Therefore an ok result does not validate the token, target index, event path, or successful indexing.

To isolate HEC credentials and routing without querying ClickHouse, send a synthetic span to every configured sink:

sudo -u click-dog click-dog test export --config /etc/click-dog/click-dog.yaml

The event carries click_dog.test=true nested under the event's attributes object, so search on the flattened field: attributes.click_dog.test="true".

Then prove native ClickHouse trace propagation, span-log recovery, and complete trace delivery:

sudo -u click-dog click-dog test tracing --config /etc/click-dog/click-dog.yaml

PASS means the HEC exporter accepted the batch; it does not confirm Splunk indexing. Use the printed trace_id to search after normal ingestion delay.

Then search the configured Splunk index for events with:

source="click-dog" sourcetype="_json"

The exported payload contains ClickHouse query text, durations, host metadata, and the same redaction/truncation controls documented in Configuration.

Multiple Backends

Splunk HEC can be combined with OTLP exporters:

exporters:
  otel:
    - collector_address: datadog-agent.internal:4317
      service_name: click-dog-monitor
  splunk_hec:
    - endpoint: https://splunk.internal:8088
      token_file: /etc/click-dog/splunk-hec.token
      index: clickhouse

With multiple sinks configured, Click-Dog uses the strict all-required delivery contract: a sink failure marks the cycle as an error and the next cycle retries the full batch. See Operating Contract.