The Things Stack exports Prometheus telemetry on the
/metrics endpoint. This reference gives an overview of the most important metrics.
Warning:Metrics are not covered by our compatibility commitment. This means that metrics may be changed or removed in major, minor, or patch releases. This page only lists metrics that are considered relatively stable, but even these metrics may be changed.
Process and Go Metrics
The Things Stack uses the Prometheus instrumentation library for Go that includes a collector for the state of the process (on Linux) and metrics from the Go runtime.
The Things Stack uses the Prometheus gRPC instrumentation library that exposes metrics about gRPC method calls.
grpc_server_conns_closed_totalcan be used to see how many gRPC connections to The Things Stack are opened, and how many are currently open.
grpc_server_handled_totalcan be used to see how many RPCs to The Things Stack are started, active and finished (including response code).
grpc_server_msg_received_totalcan be used to see how many RPC messages are sent/received by the server.
Similar metrics exist for gRPC client connections opened by The Things Stack, and RPCs made by The Things Stack.
ttn_lw_license_expiry_secondscan be used to keep track of license expiry
- New in 3.13.1
ttn_lw_log_messages_totalcan be used to track the log messages written by different log namespaces at different log levels.
- Before version 3.13.1 this metric was called
- Before version 3.13.1 this metric was called
ttn_lw_events_publishes_totalcan be used to track the published events by event type.
ttn_lw_events_channel_dropped_totalcan be used to watch for dropped events, which typically indicates that a consumer (such as a user’s web browser) can’t keep up.
ttn_lw_workerpool_workers_stoppedindicate the total number of workers started, and stopped, in the worker pools.
ttn_lw_workerpool_workers_idleindicates the number of workers which are idle (not currently processing any work item) in the worker pools.
ttn_lw_workerpool_work_queue_sizeindicates the number of elements found in the queue of the worker pools.
ttn_lw_workerpool_work_droppedindicates the total amount of work processed, or dropped, by the worker pools.
For the Gateway Server:
ttn_lw_gs_connected_gatewaysindicates the number of connected gateways.
ttn_lw_gs_uplink_received_totalindicates the number of uplink messages received from gateways.
ttn_lw_gs_downlink_sent_totalindicates the number of downlink messages sent to gateways.
ttn_lw_gs_status_received_totalindicates the number of status messages received from gateways.
ttn_lw_gs_uplink_forwarded_totalindicates the number of uplink messages forwarded to the Network Server or Packet Broker.
ttn_lw_gs_uplink_dropped_totalindicates the number of uplink messages that are dropped for various reasons.
ttn_lw_gs_uplink_failed_totalindicates the number of uplink messages for which the processing has failed for various reasons.
ttn_lw_gs_downlink_tx_failed_totalindicates the number of successful transmissions, and failed transmissions for various reasons, reported by the gateways.
For the Packet Broker Agent: New in 3.11.2
ttn_lw_pba_uplink_received_totalindicates the number of uplink messages received from Packet Broker (as a home network).
ttn_lw_pba_downlink_forwarded_totalindicates the number of downlink messages sent to Packet Broker (as a home network).
ttn_lw_pba_downlink_received_totalindicates the number of downlink messages received from Packet Broker (as a forwarder).
ttn_lw_pba_uplink_forwarded_totalindicates the number of uplink messages sent to Packet Broker (as a forwarder).
For the Network Server:
ttn_lw_ns_uplink_received_totalindicates the number of uplink messages received from the Gateway Server or Packet Broker.
ttn_lw_ns_uplink_duplicates_totalindicates the number of duplicate uplinks that are merged into a single uplink.
ttn_lw_ns_uplink_processed_totalindicates the number of processed uplinks, meaning uplinks that are matched to a device.
ttn_lw_ns_uplink_forwarded_totalindicates the number of uplinks forwarded to an Application Server.
ttn_lw_ns_uplink_dropped_totalindicates the number of uplinks that are dropped for various reasons.
ttn_lw_ns_downlink_attempted_totalindicates the number of downlink attempts.
ttn_lw_ns_downlink_forwarded_totalindicates the number of downlinks forwarded to the Gateway Server.
ttn_lw_ns_uplink_gatewaysis a histogram that indicates the number of gateways that received a given uplink.
For the Application Server:
ttn_lw_as_subscriptions_stopped_totalindicate the number of started/stopped uplink subscriptions from integrations and external applications.
ttn_lw_as_pubsub_integrations_stopped_totalindicate the number of started/stopped pub/sub integrations.
ttn_lw_as_uplink_received_totalindicates the number of uplink messages received from the Network Server.
ttn_lw_as_uplink_forwarded_totalindicates the number of uplink messages forwarded to integrations and external applications.
ttn_lw_as_uplink_dropped_totalindicates the number of uplink messages dropped for various reasons.
ttn_lw_as_downlink_received_totalindicates the number of downlink messages received from integrations and external applications.
ttn_lw_as_downlink_forwarded_totalindicates the number of downlink messages forwarded to the Network Server.
ttn_lw_as_downlink_dropped_totalindicates the number of downlink messages dropped for various reasons.
ttn_lw_as_subscription_sets_stopped_totalindicates the total number of subscriptions sets started and stopped.
ttn_lw_as_packages_failed_totalindicates the total number of uplink messages processed, or which have failed processing for various reasons, by application packages.
ttn_lw_as_packages_storage_upstream_messages_droppedindicates the total number of uplink messages stored, or dropped for various reasons, by the Storage Integration.
ttn_lw_as_webhook_failed_totalindicates the total number of webhook messages sent, or failed for various reasons.
For the Join Server:
ttn_lw_js_join_rejected_totalindicate the number of accepted and rejected join requests.