Ctrl+K

System logs

System logs#

Log records generated by any cluster node are collected and stored in the leader node. See Logs persistence for additional information.

Accessing logs#

The System Logs page allows users to efficiently search logs based on date, text query, or context. The available contexts include:

  • Cluster: Displays logs from all cluster nodes.

  • Node: Shows logs from a specific node.

  • Application: Presents logs from a particular application instance.

Users can select between two modes of operation:

  • Dump mode Retrieves a limited number of log records within a specified time period. The maximum number of lines can be adjusted using the Max lines field.

  • Follow mode Displays logs in real-time, providing live updates.

If the Search query field is set, only log records matching the given string are returned. Only exact substring matches are allowed.

For comparative analysis of logs from two nodes or applications, follow these steps:

  • Click on Add search button.

  • Optionally, switch to Vertical layout from the three-dots menu for a side-by-side comparison.

For example, comparing logs can be useful to see the Traefik log from one side, to see incoming HTTP requests, and Nextcloud logs on the other side to see that application activity.

Note

By default, log searches are directed to the active Loki instance. If there are inactive Loki instances within the cluster, it is possible to select them to search past log entries.

Command line interface#

In addition to accessing logs via the System Logs web page, users can utilize the api-server-logs command for log searching. Below are examples illustrating its usage:

  1. Basic invocation: Enters “follow mode” for the entire cluster:

    api-server-logs logs

  2. Follow mode for application instance: Enables follow mode for the specified application (module) instance, such as traefik1. The --entity flag selects the context:

    api-server-logs logs --entity module --name traefik1

  3. Dump mode for specific instance in a time period: Initiates dump mode for the same instance within a specific time period. Dates must adhere to the ISO8601 format:

    api-server-logs logs --mode dump --entity module --name traefik1 --from 2024-04-09T16:43:22Z --to 2024-04-09T16:55:31Z

  4. Changing output timezone: Modifies the output timezone while maintaining the same query. Refer to timedatectl list-timezones for a full list of options:

    api-server-logs logs --timezone America/New_York --mode dump --entity module --name traefik1 --from 2024-04-09T16:43:22Z --to 2024-04-09T16:55:31Z

Logs persistence#

Upon cluster creation, a Loki [1] core module instance is installed on the leader node and designated as the active instance. The leader node, like any other worker node, continuously streams its log data to this active Loki instance [2].

Adjusting Settings#

Navigate to the Settings page and click on the System logs card to modify log retention (select Edit retention) or assign a user-friendly name to the active Loki instance (choose the three-dots menu, then Edit label).

Understanding log retention#

Log retention refers to the maximum age of stored log records. Records older than the retention period are automatically purged. By default, System logs have a retention period of 365 days, but this can be customized to any desired duration. For compliance with common regulations and best practices, a recommended retention period is 200 days or longer.

Inactive Loki instances#

When a worker node is promoted to leader, a new Loki instance is installed on it and becomes the active instance, while the old instance is marked as inactive.

  • The new active instance inherits the retention setting from the old one.

  • An inactive instance retains logs based on its last retention setting.

  • Log searches can still be performed within an inactive instance.

  • Restoring a Loki instance from backup renders it inactive.

  • To remove an inactive instance, select the three-dots menu and choose the Uninstall action.

Footnotes

Contents