Release Notes

Added

• Packet Broker gateway visibility management (default settings only). See ttn-lw-cli packetbroker home-networks gateway-visibilities --help for more information.

Changed

• The Gateway Server worker pools may now drop workers if they are idle for too long.
• FPort = 0 uplinks are no longer decoded by the Application Server, and the Network Server no longer provides the frame payload to the Application Server for these messages.

Fixed

• Emails to admins about requested OAuth clients.
• session handling for joined OTAA end devices in the Console.
• Empty Join Server address handling in end device creation form in the Console.
• Data Rate to data rate index matching for uplinks and downlinks.

Added

• RPC to query supported PHY versions for a given Band ID.
• Non-TLS LNS endpoint support.
• New config option is.user-login.disable-credentials-login that disables login with credentials (can be set per tenant).
  • In case where only one authentication provider is configured and the config option is set to true the login screen redirects to provider login.

Changed

• Update to Go 1.17.
• LBS timestamp rollover threshold.
• The Application Server worker pools may now drop workers if they are idle for too long.
• Improved error page UX in the Console.

Fixed

• Collaborator ID validation in gateway claiming form in the Console.
• Entity purge handling for non-admin users in the Console.
• URL field validation in webhook forms in the Console when value is not trimmed.
• Not rendering site header and footer for error pages in some situations.
• Not providing a copy button for error pages in some situations.
• Improved errors for invalid URLs.
• Limit length of search queries within tables in the Console to 50 to comply with API validation.
• External Join Server address handling in end device creation form in the Console.
• Updating supports_class_b field in the end device general settings page in the Console.

Added

• CLI warnings about insecure connections.
• CLI warnings about using the --all flag.
• Packet Broker network listed switch in the Console.
• Improved errors for invalid command-line flags.
• Validation of entity attributes in the Console, with regards to maximum length for keys and values.
• Support limiting concurrent data plane application gRPC and MQTT connections by application and tenant ID. See the resource-limiting configuration section for details.
• CLI command to decode raw LoRaWAN frames (ttn-lw-cli lorawan decode), useful for debugging purposes.
• Options to restore or purge deleted applications, gateways and organizations in the Console.
• Handling of default mac settings values when manually registering end devices in the Console.
• Add a new class_b_c_downlink_interval field that can be configured to set the minimum interval between a network initiated downlink (Class B & Class C) and an arbitrary downlink per device.
• Retrieve count of upstream messages from the Storage Integration by end device.
  • See the new GetStoredApplicationUpCount RPC.
  • See the new ttn-lw-cli applications storage count and ttn-lw-cli end-devices storage count CLI commands.

Changed

• The Identity Server now returns a validation error when trying to update the EUIs of an end device.
• Network Server no longer accepts RX metadata from Packet Broker if the originating forwarder network equals the current Network Server (by NetID and cluster ID, based onns.net-id and ns.cluster-id configuration). This avoids duplicate RX metadata as well as redundant downlink scheduling attempts through Packet Broker after the cluster’s Gateway Server already failed to schedule.
• Usability of the end device import function in the Console.
  • Show a per-device report when errors occur.
  • More structural changes to the process to improve UX.

Removed

• Packet Broker mutual TLS authentication; only OAuth 2.0 is supported now.
• request_details from errors in the Console.

Fixed

• Generated CLI configuration for The Things Stack Community Edition.
• End device access with limited rights in the Console.
• Parsing of ID6 encoded EUIs from Basic Station gateways.
• Warnings about unknown fields when getting or searching for gateways.
• Internal Server Errors from pkg/identityserver/store.
• Console rendering blank pages in outdated browsers due to missing or incomplete internationalization API.
• Error in edit user form (Console) when submitting without making any changes.
• description field not being fetched in edit user form (admin only) in the Console.
• Ignore invalid configuration when printing configuration with ttn-lw-cli config or ttn-lw-stack config.
• Emails about API key changes.
• Avoid rendering blank pages in the Console for certain errors.
• Blank page crashes in the Console for certain browsers that do not fully support Intl API.
• End device session keys handling in the Console.
• Byte input width in Safari in the Console.

Added

• New config option --as.packages.timeout to control the message processing timeout of application packages.
• Option to retain current Gateway ID while claiming.
• Option to view and copy error details in full view errors in the Console.
• Metrics for CUPS requests.
• Language chooser in the footer in the Console.
• Japanese language support in the Console.
• Metrics for gateway claiming.
• Option to disable primary email and name update via stack.

Changed

• Cache Root CA for client TLS configuration.
• Identity Server no longer allows removing the _ALL right from entity collaborators if that leaves the entity without any collaborator that has the _ALL right.
• The Network Server application uplink queue may now be skipped if the Application Server peer is available at enqueue time.
• The interval for updating gateways in Packet Broker is now 10 minutes (was 5 minutes) and the timeout is 5 seconds (was 2 seconds).

Removed

• Validation of the location field in GatewayAntenna.

Fixed

• Improved errors when ordering search requests by non-existent fields.
• LNS authentication key handling for gateways in the Console.

Added

• Gateway antenna placement; unknown, indoor or outdoor. This can now be specified with CLI, e.g. for the first antenna: ttn-lw-cli gateways set <gateway-id> --antenna.index 0 --antenna.placement OUTDOOR. The antenna placement will be reported to Packet Broker Mapper.
  • This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added columns.
• Payload formatter length validation in the Console.
• User session management (listing and deleting) in the Identity Server and the CLI.
• Improved logging for the OAuth server.
• LR-FHSS modulation
  • Additional fields for the Gateway and Rx Metadata API. This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added columns.
  • Support for LR-FHSS fields when translating uplink messages with the UDP protocol.
• Network Server now appends network identifiers in forwarded uplink messages. These are populated from the ns.net-id and the new ns.cluster-id configuration option.
  • See the new uplink_message.network_identifiers.net_id, uplink_message.network_identifiers.cluster_id and uplink_message.network_identifiers.tenant_id fields.
  • This can be useful for HTTP webhooks to determine the Network Server that received and forwarded an uplink message.
• GetDefaultMACSettings RPC for requesting the default and desired MAC settings for a Band (Frequency Plan) and LoRaWAN regional parameters version.
• Error handling for missing templates in device repository form in the Console.
• Cache gateway entity responses for requests made to the IS by the GS. See the new gs.entity-registry-cache.size and gs.entity-registry-cache.timeout configuration options.
• API for searching tenants.
• End device first activation timestamp in the Identity Server end device store.
  • This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added columns.
• Azure IoT Hub integration via an application package.
• Resilience to flakiness in upstream servers during gateway claiming.
• Opt out of Packet Broker for individual gateways, see the new disable_packet_broker_forwarding gateway option.
  • This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added columns.
  • This is only relevant when Packet Broker is enabled and configured by the network operator.
• Gateways are now disconnected when settings affecting the connection with Gateway Server change. Use the gs.fetch-gateway-interval and gs.fetch-interval-jitter to configure how often the gateway is fetched from the entity registry.
• Small UX improvements to the LoRaCloud DAS forms in the Console.
• End device first activation timestamp in the Identity Server end device store.
  • This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added columns.
• AppJs interface for applications to get the LoRaWAN AppSKey directly from the Join Server.
• Console support for DevEUI generation from the configured DevEUI address block.
  • This requires console.ui.dev-eui-issuing-enabled and console.ui.dev-eui-app-limit to be set with the same values as in the Identity Server configuration.

Changed

• When a gateway uplink message contains duplicate data uplinks, only the one with the highest RSSI are forwarded.
• The HTTP port now allows HTTP/2 connections over cleartext (h2c).
• ttn-lw-stack ns-db migrate command records the schema version and only performs migrations if on a newer version.
  • Use the --force flag to force perform migrations.
• Any authenticated user in the network can now list the collaborators of entities in the network.
• The search RPCs no longer require fields to be specified in the field mask when those fields are already specified as filters.
• When generating client configuration with the CLI use command, automatically set the correct Identity Server and OAuth Server addresses for The Things Stack Cloud and The Things Stack Community Edition.

Removed

• The old log format.

Fixed

• Network Server ADR algorithm data rate adjustment behavior on negative margin.
• CLI gateway set --antenna.remove command failing to remove gateway antennas in some cases.
• CLI gateway set --antenna.gain <gain> command crashing when no gateway antennas are present.
• Webhook template path variable expansion of query parameters.
• LBS LNS Auth Secret displays garbage value when updated.
• Transmit confirmation messages for LoRa Basics Station gateways.
• Instability and frequent crashes when internet connection is lost in the Console.
• Rate limiting for TenantRegistry.GetTenantIdentifiersForGatewayEUI is now applied per gateway EUI.
• Rate limiting for TenantRegistry.GetTenantIdentifiersForEndDeviceEUIs is now applied per end device EUI.
• Panic in GCS when CUPS rotation is set without a key.
• Rate limiting for GatewayRegistry.GetGatewayIdentifiersForEUI is now applied per gateway EUI.
• Network Server ensures that the Band ID in the end device version identifiers match the configured Frequency Plan of the device.

Added

• Support claiming of The Things Indoor Gateways.
• Configurable log formats with the log.format configuration option.
  • The console format that prints logs as more human-friendly text. This is the new default.
  • The json format that prints logs as JSON. This is the recommended format for production deployments.
  • The old format (deprecated). This can be used if you need to adapt your log analysis tooling before v3.14.
• ttn_lw_gs_ns_uplink_latency_seconds, ttn_lw_ns_as_uplink_latency_seconds and ttn_lw_gtw_as_uplink_latency_seconds metrics to track latency of uplink processing.
• Signing of releases.
• Hard delete option to delete applications, gateways and organizations in the Console.

Changed

• Relaxed the cookie policy for cross-origin requests from Strict to Lax.
• Changed the cookie policy for OAuth state to None.

Deprecated

• The old log format is deprecated and will be removed in v3.14.

Fixed

• Incorrect Storage Integration API URL in Console.
• Permissions issue for reading and writing gateway secrets in the Console.
• Current and future rights selection for organization collaborators in the Console.
• Current and future rights selection for user api keys in the Console.
• Low or no throughput of message handling from Packet Broker when the ingress is high when Packet Broker Agent starts.
• Unset ADR bit in downlink messages to multicast devices.

Added

• More contextual tooltips to end device and gateway form fields in the Console.
• Warnings in the Console when changing or revoking your own access to an entity.

Changed

• Enable LoRa Standard channel for Tabs Hubs gateways.
• UDP gateway traffic is limited by IP address and source port number.
• Do not print error line logs for rate limited gRPC and HTTP API requests.
• The ttn_lw_log_log_messages_total metric was renamed to ttn_lw_log_messages_total and has an additional error_name label.
• Authenticated users now have access to gateway status and location when those are set to public.
• Cookies are no longer allowed in cross-origin requests to the HTTP API. Applications must instead use Bearer tokens in the Authorization header.

Fixed

• Loading event data in (non-historical) event streams.
• Downlink queue eviction on FCnt mismatch.
• End device payload formatter view crashing in the Console.
• End device overview frequently crashing in the Console.
• Panic on empty downlink in zero indexed downlink token.

Added

• Publish Snapcraft snaps tti-lw-stack and tti-lw-cli

Changed

• Optimized storage of recent application uplinks in Application Server.

Fixed

• Validation of OAuth token exchange requests from the CLI.
• Validation of join-request types when using the Crypto Server backend.
• Application Server session recovery functionality for imported devices.
• Fetching AppSKey when the session is rebuilt but the identifier did not change.
• Brew artifacts are now in a publicly accessible location.

Added

• Contextual tooltips to form fields in the Console.
• C-Style uint32_t representation for end device address field.
• Gateway Configuration Server to the cluster package.
  • This introduces a new config option cluster.gateway-configuration-server that needs to be set in multi-instance deployments.
• Uplink storage for integrations in the Application Server. The number of uplinks stored per end device may be configured via the config option as.uplink-storage.limit.
• LoRaCloud GLS multi frame request support.
• LoRaCloud GNSS request support.
• LoRaCloud WiFi request support.

Changed

• Allow the LinkADRReq commands to lower the data rate used by the end devices.

Fixed

• Occasional crashes in the ratelimit middleware.
• Handling of zero EUI CUPS update-info requests.
• Backend validation messages for some forms.
• Gateway downlink message previews not displaying correctly in the event view of the Console.
• Importing end devices from the Console would occasionally ignore some device MAC settings fields.

Added

• Payload formatter testing functionality in the Console.
• Options in the Identity Server to reject passwords that contain the user ID (is.user-registration.password-requirements.reject-user-id) or common passwords (is.user-registration.password-requirements.reject-common).

Changed

• Network Server now takes uplink data rate index for ADR.
• Event streams are now closed when the callers rights are revoked.

Fixed

• It is no longer possible to accidentally create API keys without any rights.
• Application overview page crashing for collaborators without certain rights.
• A bug that crashed the Identity Server when calling certain Tenant RPCs.
• mac_settings.factory_preset_frequencies handling in US-like bands.

Security

• This release fixes an issue that prevented some request messages from being validated correctly, allowing invalid values in certain fields.

Added

• API and CLI commands for listing, searching for and restoring recently deleted applications, OAuth clients, gateways, organizations and users.
• State Description fields for adding context to the (rejected, flagged, suspended) state of Users, OAuth clients and tenants.
  • This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added columns.
• Configure retention policy for the Storage Integration database (only supported when using TimescaleDB).
  • You will need to run the ttn-lw-stack storage-db migrate command to apply the new configuration. See the --timescaledb.chunk-time-interval, --timescaledb.enable-retention-policy and --timescaledb.retention-days command-line flags.
• Searching for gateways by EUI.
• Searching for users and OAuth clients by state.
• Gateway Server forwards Tx Acknowlegdment packets to the Network Server for scheduled downlinks. These can be used by the Network Server to forward downlink_ack upstream messages to the Application Server.
• UDP connection error caching. The duration can be configured via the gs.udp.connection-error-expires configuration entry.
• Option to require individual gateways to use authenticated connections.
  • This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added columns.
• Login Tokens (magic login links) that can be used for password-less login.
  • This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added columns.
  • This feature is disabled by default; use the new is.login-tokens.enabled option to enable it.
• Packet Broker registration, configuration of routing policies, listing home networks and viewing routing policies set by forwarding networks. See ttn-lw-cli packetbroker --help for more information.
• Support LoRa 2.4 GHz with Packet Broker.
• Include gateway identifiers from Packet Broker in metadata.
• Support for claiming gateways through CLI. See ttn-lw-cli gateways claim --help for more information.
• Session and MAC state import functionality. This means that devices can be migrated without rejoin.
• Rate limiting for HTTP endpoints, gRPC endpoints, MQTT, UDP and WebSockets connections.
  • Rate limiting is disabled by default. Refer to the rate-limiting configuration entry to enable.
• Profile settings link to header dropdown menu.

Changed

• Changed the pub/sub channels that the Redis backend of the Events system uses.
• Changed the encoding of events transported by the Redis backend of the Events system.
• All external HTTP calls are now using TLS client configuration. This fixes issues where HTTP calls would fail if custom (e.g. self-signed) CAs were used.
• All external HTTP calls are now using a default timeout. This fixes issues where HTTP calls would stall for a long time.
• All value wrappers now are encoded and decoded as the value being wrapped in JSON. That means, that, e.g. format of mac_settings.rx1_delay is changed from {"value": 2} to just 2.
• Changed the error that is returned when attempting to validate already validated contact info.
  • This requires a database schema migration (ttn-lw-stack is-db migrate) because of the added column.
• Update Go to 1.16
• Network Server now performs more strict validation and will disallow creation and updates of invalid devices.
• DevEUI is not required for multicast devices anymore, regarding of LoRaWAN version.
• Pub/Sub provider visibility in console based on disabled status.

Removed

• AWS IoT Integration as pub/sub integration. The integration is only available as application package.
  • This requires a database migration (ttn-lw-stack as-db migrate).

Fixed

• Incorrect documentation url for event details data formats.
• Search functionality for applications, gateways and organizations in the Console.
• Error handling of end device template formats for the application overview page in the Console.
• Payload size limits for AU915 data rates 8 and 9, which are now consistent with Regional Parameters RP002-1.0.2.
• Payload size limit calculation in Network Server.
• Occasional panic in Network Server on downlink with corrupted device states.
• Occasional panic in Identity Server on extracting log fields from invalid requests.
• Print an error message stating that the Storage Integration is not available in the open source edition of The Things Stack when trying to execute ttn-lw-stack storage-db commands.
• Applications without AWS IoT integration displaying configuration of a previously inspected application.

Added

• sentry.environment configuration option to set the environment tag on Sentry reports.
• TR005 QR code format, with ID tr005.
• LoRa Cloud Geolocation Services support for TOA/RSSI end device geolocation.

Changed

• Device Claiming Server can now contact Join Servers outside of the cluster: the source end device’s registered Join Server address is dialed.
• Default value of gs.udp.addr-change-block is now 0, which disables the IP firewall for UDP traffic. Deployments that need to enforce the IP check should set a value greater than 0. Note that the new default value makes UDP connections less secure.
• Prevent flooding logs with “Packet Filtered” messages when UDP gateways exceed the maximum rate limit. Only one message per minute will be printed for each gateway.

Deprecated

• TR005 Draft 2 and 3 QR code formats. Use the final version of the technical recommendation, with ID tr005.

Fixed

• Downlink queue operations on ABP devices not working under specific circumstances.
• NwkKey handling for end devices in the Console.

Added

• Pagination flags for the users oauth authorizations list and users oauth access-tokens list CLI commands.
• End device ID generation based on DevEUI in The LoRaWAN Device Repository creation form in the Console.
• remote_ip and user_agent metadata on OAuth events.
• created_at and updated_at fields to API Keys.
• Telemetry for Packet Broker Agent.
• User rights check for managing API keys in the Console.

Changed

• When multiple base domains are configured, the middleware uses the longest match to extract the Tenant ID.
• temp field of the UDP stats message is now type float32 (pointer).

Fixed

• Ocassional race condition in uplink matching with replicated Network Server instances.
• Ocassional race condition when matching pending sessions.
• Application uplink queue handling in Network Server.
• Conflict error when registering an end device via the wizard in the Console.
• Pagination in the List and ListTokens RPCs of the OAuthAuthorizationRegistry.
• Event name on user login.
• Application uplink queue handling in Network Server.
• Application Server session desynchronization with the Network Server. The Application Server will now attempt to synchronize the end device session view on downlink queue operational errors. This fixes the f_cnt_too_low and unknown_session errors reported on downlink queue push and replace.
• Panic while generating SX1301 config for frequency plans without radio configuration.

Added

• Profile settings view to the Account App.
  • Functionality to change basic profile information, such as name, email address and profile picture.
  • Functionality to update the account password.
  • Functionality to delete the account.

Changed

• Improved logging.
• AWS IoT integration is now implemented as package instead of a pub/sub provider.

Deprecated

• Configuration of the AWS IoT Core integration as pub/sub provider. Please use the configuration with the package name aws-iot and update to the latest version of the AWS IoT Integration CloudFormation template. The configuration via pub/sub provider will be removed in later version.
  • To migrate existing pub/sub integrations to packages, migrate the Application Server database (ttn-lw-stack as-db migrate).
• AWS IoT Core publisher of all upstream messages that is configured with as.aws.iot.telemetry.
  • This only affects The Things Stack AWS Launcher customers that use the AWS Marketplace AMI listing.
  • All deployments are now advised to use the AWS IoT Core package that can be configured per application in Application Server.

Fixed

• Synchronization in Gateway Server scheduler that caused race conditions in scheduling downlink traffic.
• End device claiming when the source device uses derived root keys.
• Console link in Account App for multi-tenant deployments.

Fixed

• Packet Broker multi-tenancy with a single tenant The Things Stack configuration.

Fixed

• The inability to see individual end devices in the Console due to field mask errors.

Added

• Pub/Sub providers may now be disabled, or marked for deprecation via --as.pubsub.providers.
• Configuration option is.admin-rights.all to grant admins all rights, including _KEYS and _ALL.
• ttn-lw-stack CLI command for creating an API Key with full rights on a user.
• Storage integration panel in the Console.

Changed

• Branding (updated TTS Enterprise logo, colors, etc).
• Packet Broker API version to v3.2.0-tts and routing API to v1.0.2-tts.

Deprecated

• Packet Broker mutual TLS authentication: use OAuth 2.0 client credentials instead; set pba.authentication-mode to oauth2 and configure pba.oauth2.
• Packet Broker forwarder blacklist setting pba.home-network.blacklist-forwarder has become ineffective.

Added

• Support for sending end device uplinks using the CLI (see ttn-lw-cli simulate application-uplink command).
• Clients can now perform custom ADR by modifying ADR parameters in mac-state.desired-parameters of the device.

Changed

• Form field layouts in the Console (field labels are now dispayed above the field).
• Small structural changes to the custom webhook form in the Console.
• Renamed experimental command ttn-lw-cli simulate uplink to ttn-lw-cli simulate gateway-uplink.
• Renamed experimental command ttn-lw-cli simulate join-request to ttn-lw-cli simulate gateway-join-request.

Fixed

• Removed misleading warning message for missing package data when setting up the storage integration package association.

Added

• Configure application activation settings from the CLI (see ttn-lw-cli application activation-settings commands).
• User API keys management to the Console.
• Purge RPC and cli command for entity purge (hard-delete) from the database.
• More password validation rules in the user management form in the Console.
• Support for class B end devices in the Console.
• MAC settings configuration when creating and editing end devices in the Console.
• Support for the LR1110 LTV stream protocol.

Changed

• Branding (updated TTS Open Source logo, colors, etc).

Fixed

• Simulated uplinks visibility in webhook messages.
• Retransmission handling.
• RTT recording for LBS gateways. The maximum round trip delay for RTT calculation is configurable via --gs.basic-station.max-valid-round-trip-delay.
• Memory leak in GS scheduler.

Added

• Configure application activation settings from the CLI (see ttn-lw-cli application activation-settings commands).
• CLI support for federated authentication provider deletion (ttn-lw-stack is-db delete-auth-provider).

Security

• Fixed an issue with authentication on the /debug/pprof, /healthz and /metrics endpoints.

Added

• gRPC middleware to extract proxy headers from trusted proxies. This adds a configuration grpc.trusted-proxies that is similar to the existing http.trusted-proxies option.

Changed

• Log field consistency for HTTP and gRPC request logs.

Fixed

• Uplink frame counter reset handling.
• Uplink retransmission handling in Network Server.
• DevAddr generation for NetID Type 3 and 4, according to errata.
• Packet Broker downlink in multi-tenant clusters.
• HTTP header propagation (such as Request ID) to gRPC services.

Added

• More password validation rules in the user management form in the Console.

Changed

• Limitation of displayed and stored events in the Console to 2000.
• Application Server will unwrap the AppSKey if it can, even if skipping payload crypto is enabled. This is to avoid upstream applications to receive wrapped keys they cannot unwrap. For end-to-end encryption, configure Join Servers with wrap keys unknown to the Application Server.
• More precise payload labels for event previews in the Console.

Fixed

• Next button title in the end device wizard in the Console.
• Navigation to the user edit page after creation in the Console.
• The port number of the http.redirect-to-host option was ignored when http.redirect-to-tls was used. This could lead to situations where the HTTPS server would always redirect to port 443, even if a different one was specified.
  • If the HTTPS server is available on https://thethings.example.com:8443, the following flags (or equivalent environment variables or configuration options) are required: --http.redirect-to-tls --http.redirect-to-host=thethings.example.com:8443.
• Status display on the error view in the Console.
• Event views in the Console freezing after receiving thousands of events.
• Wrong FPort value displayed for downlink attempt events in the Console.
• Network Server sending duplicate application downlink NACKs.
• Network Server now sends downlink NACK when it assumes confirmed downlink is lost.
• Network Server application uplink drainage.