infra.observability

38 options

`infra.observability.alerts`

`infra.observability.alerts.enable`

Whether this host contributes its alert rules to the fleet. When false, none of this host’s infra.observability.alerts.rules are collected/provisioned by the grafana host. Use to opt a host out of alerting entirely (e.g. signet).

Type: boolean

Default:

true

Declared by: nix/modules/infra/observability/alerts

`infra.observability.alerts.group`

Grafana alert group name these contributed rules land in.

Type: string

Default:

"Fleet — module-contributed"

Declared by: nix/modules/infra/observability/alerts

`infra.observability.alerts.rules`

Grafana alert-rule attrsets (built via grafana.mkAlertRule) contributed by the modules enabled on this host. Mergeable — every module that wants an alert appends to it. Consumed centrally by the grafana-stack module, which gathers this across all nodes with infra.observability.alerts.enable = true.

Type: list of (attribute set)

Default:

[ ]

Example:

[ (grafana.mkAlertRule {
    uid = "svc-down";
    title = "Service down";
    expr = "up{job=\"myservice\"} == 0";
    for = "5m";
  }) ]

Declared by: nix/modules/infra/observability/alerts

`infra.observability.alloy`

`infra.observability.alloy.enable`

Whether to enable Grafana Alloy telemetry agent.

Type: boolean

Default:

false

Example:

true

Declared by: nix/modules/infra/observability/alloy

`infra.observability.alloy.extraConfig`

Extra Alloy config appended to the main config. Use for additional scrape targets.

Type: strings concatenated with "\n"

Default:

""

Declared by: nix/modules/infra/observability/alloy

`infra.observability.alloy.gateway.enable`

Whether to enable OTLP gateway mode — central OTLP ingest fanning metrics/logs/traces into Prometheus/Loki/Tempo (INFRA-47 / ADR-038).

Type: boolean

Default:

false

Example:

true

Declared by: nix/modules/infra/observability/alloy

`infra.observability.alloy.gateway.otlpGrpcAddress`

Listen address for the OTLP/gRPC receiver. Public on the internal admin net.

Type: string

Default:

"0.0.0.0:4317"

Declared by: nix/modules/infra/observability/alloy

`infra.observability.alloy.gateway.otlpHttpAddress`

Listen address for the OTLP/HTTP receiver (the PVE metric server pushes here).

Type: string

Default:

"0.0.0.0:4318"

Declared by: nix/modules/infra/observability/alloy

`infra.observability.alloy.gateway.tempoOtlpEndpoint`

OTLP/gRPC endpoint of the local Tempo trace store. Loopback + non-default port so Tempo’s receiver does not collide with the gateway’s own :4317. Must match nix/modules/tempo otlpGrpcAddress.

Type: string

Default:

"127.0.0.1:14317"

Declared by: nix/modules/infra/observability/alloy

`infra.observability.alloy.lokiUrl`

Loki push endpoint URL. Must be non-null when infra.observability.alloy is enabled (asserted).

Type: null or string

Default:

config.fleet.settings.observability.lokiPushUrl

Declared by: nix/modules/infra/observability/alloy

`infra.observability.alloy.prometheusUrl`

Prometheus remote-write endpoint URL. Must be non-null when infra.observability.alloy is enabled (asserted).

Type: null or string

Default:

config.fleet.settings.observability.prometheusRemoteWriteUrl

Declared by: nix/modules/infra/observability/alloy

`infra.observability.stack`

`infra.observability.stack.alertRulesDirs`

Consumer-supplied directories of Grafana alert-rule provisioning YAML files (*.yaml). Merged with the framework’s Nix-built alert catalog into the single provisioned alerting path.

Type: list of absolute path

Default:

[ ]

Example:

[ ./alerts ]

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.alerting.groupByAlertnameOnly`

Alert titles grouped by alertname only (the tree default is [alertname instance]): one Slack post per firing alert name across all hosts, so a single host outage that fails 25 units is one message listing them rather than 25 messages. Applied under both severity routes.

Type: list of string

Default:

[
  "Systemd Unit Failed"
]

Example:

[
  "Systemd Unit Failed"
  "Disk Low (warning)"
]

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.alerting.repeatIntervals.critical`

repeat_interval for the severity=critical route.

Type: string

Default:

"1h"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.alerting.repeatIntervals.default`

Root notification policy repeat_interval — alerts without a severity label (or with one no tier matches).

Type: string

Default:

"4h"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.alerting.repeatIntervals.warning`

repeat_interval for the severity=warning route.

Type: string

Default:

"24h"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.alerting.slack.criticalWebhookSecret`

SOPS key (in sops.defaultSopsFile) holding a SECOND Slack incoming webhook, bound to the channel that must see severity=critical alerts. Provisions contact point slack-alerts and routes criticals to it; every other severity stays on slack-fleet (services/grafana/slack_webhook). A Slack webhook is bound to one channel, so two channels need two webhooks — create the second in Slack (App → Incoming Webhooks → Add to channel) and store it with fleet devtools secrets keys add <this path> <webhook url>.

null keeps the single-channel setup. If the key is set but not yet present in the sops file, evaluation WARNS and criticals fall back to slack-fleet (so the deploy still succeeds); once the key exists the next deploy wires the second channel automatically.

Type: null or string

Default:

null

Example:

"services/grafana/slack_webhook_alerts"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.dashboardsDirs`

Consumer-supplied dashboards: Grafana folder name → directory of pre-rendered dashboard JSON files. Each entry becomes its own file provider (same mechanism as the built-in Fleet folder) and shows up as a folder of that name in the Grafana UI.

Type: attribute set of absolute path

Default:

{ }

Example:

{ "My App" = ./dashboards/my-app; }

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.dataDir`

Root directory for all observability data.

Type: string

Default:

"/var/lib/observability"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.enable`

Whether to enable Self-hosted Grafana observability stack (Grafana + Loki + Prometheus).

Type: boolean

Default:

false

Example:

true

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.extraDatasources`

Extra Grafana datasources (verbatim provisioning attrsets) appended to the built-in Prometheus/Loki/Tempo ones — e.g. read-only PostgreSQL datasources for consumer dashboards. Secrets can be referenced with $__file{…} against SOPS-provisioned paths.

Type: list of (attribute set)

Default:

[ ]

Example:

[ { name = "app-db"; type = "postgres"; url = "192.0.2.104:5432"; jsonData.sslmode = "disable"; } ]

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.grafana.domain`

Grafana server domain for URL generation. Must be non-null when infra.observability.stack is enabled (asserted).

Type: null or string

Default:

config.fleet.settings.observability.grafanaDomain

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.grafana.httpPort`

Grafana HTTP listen port.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.loki.httpPort`

Loki HTTP listen port.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.loki.retentionPeriod`

How long to retain log data.

Type: string

Default:

"90d"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.oidc.clientId`

OAuth2 client ID registered in Authentik.

Type: string

Default:

"grafana"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.oidc.enable`

Whether to enable OIDC authentication via Authentik.

Type: boolean

Default:

false

Example:

true

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.prometheus.httpPort`

Prometheus HTTP listen port.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.prometheus.retentionPeriod`

How long to retain metric data.

Type: string

Default:

"14d"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.prometheus.scrapeInterval`

Default scrape interval.

Type: string

Default:

"15s"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.smtp.enable`

Whether to enable SMTP email via Resend for Grafana alert notifications.

Type: boolean

Default:

false

Example:

true

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.smtp.fromAddress`

Sender email address for Grafana alerts. Must be non-null when smtp.enable is set (asserted).

Type: null or string

Default:

"grafana@${config.fleet.settings.domain.base}"

Declared by: nix/modules/infra/observability/stack

`infra.observability.stack.smtp.fromName`

Sender display name for Grafana alert emails.

Type: string

Default:

"Grafana"

Declared by: nix/modules/infra/observability/stack

`infra.observability.tempo`

`infra.observability.tempo.enable`

Whether to enable Tempo trace store (INFRA-47 / ADR-038).

Type: boolean

Default:

false

Example:

true

Declared by: nix/modules/infra/observability/tempo

`infra.observability.tempo.httpPort`

Tempo HTTP/query API port — the Grafana Tempo datasource target.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

Declared by: nix/modules/infra/observability/tempo

`infra.observability.tempo.mcpServer.enable`

Whether to enable Tempo’s built-in MCP server (streamable HTTP at /api/mcp on the query frontend). Lets AI tooling run TraceQL against this Tempo over MCP. Opt-in per host so it only runs where we want it.

Type: boolean

Default:

false

Example:

true

Declared by: nix/modules/infra/observability/tempo

`infra.observability.tempo.otlpGrpcAddress`

OTLP/gRPC receiver address. Loopback + non-default port: only the local Alloy gateway writes here, and it must not clash with the gateway’s own :4317. Must match infra.observability.alloy.gateway.tempoOtlpEndpoint.

Type: string

Default:

"127.0.0.1:14317"

Declared by: nix/modules/infra/observability/tempo

`infra.observability.tempo.retention`

Trace block retention (compactor block_retention). Default 7 days.

Type: string

Default:

"168h"

Declared by: nix/modules/infra/observability/tempo

Keyboard shortcuts

fleetkit