Use a centralized logging system by configuring tools like rsyslog, syslog-ng, or modern platforms such as ELK Stack or Graylog to aggregate logs from multiple servers, improving visibility and auditability. 2. Rotate logs regularly using logrotate with configurations in /etc/logrotate.conf or /etc/logrotate.d/ to prevent disk exhaustion, compress old logs, and retain logs for a defined period, testing configurations with logrotate -d before applying. 3. Monitor log sizes and disk usage via commands like journalctl --disk-usage and du -sh /var/log/*, set up alerts using monitoring tools when /var/log exceeds thresholds, and limit systemd journal growth by setting SystemMaxUse, SystemMaxFileSize, and MaxRetentionSec in journald.conf. 4. Secure log files by setting proper permissions (e.g., chmod 644 or 600), avoiding public directories, using ACLs or encryption at rest when required by compliance standards, and sending logs to remote write-only servers to prevent tampering. 5. Filter and structure log output by adopting machine-readable formats like JSON, using structured logging libraries, tagging logs by service or host, and filtering noise in rsyslog with conditional rules to improve parsing and searchability. 6. Set up alerts for critical events by leveraging tools like fail2ban, logwatch, or SIEM solutions such as Wazuh, writing custom scripts with grep and awk to detect anomalies like repeated failed logins, and automating checks via cron to send real-time notifications, ensuring proactive system monitoring and incident response.

Managing logs effectively on a Linux system is essential for troubleshooting, security monitoring, and maintaining system performance. Poor log management can lead to disk exhaustion, difficulty in diagnosing issues, or missing critical alerts. Here are key best practices to keep your logging under control.

1. Use a Centralized Logging System

Relying solely on local logs limits visibility, especially in multi-server environments. Centralized logging allows you to aggregate logs from multiple systems into a single location for easier analysis.

• Tools like rsyslog or syslog-ng can forward logs to a central server.
• Modern solutions include ELK Stack (Elasticsearch, Logstash, Kibana) or Graylog for advanced searching and visualization.
• For lightweight setups, consider Fluentd or Prometheus Loki.

Example: Configure rsyslog to send logs over TCP/UDP to a central server by editing /etc/rsyslog.conf:

*.* @central-logging-server:514

This improves auditability and simplifies monitoring across infrastructure.

2. Rotate Logs Regularly with logrotate

Uncontrolled log growth can fill up disks quickly. The logrotate utility automates the process of archiving, compressing, and removing old logs.

• Default configuration is in /etc/logrotate.conf, with app-specific rules in /etc/logrotate.d/.
• Set rotation frequency (daily, weekly), retention period, and compression.

Example configuration for a custom app:

/var/log/myapp/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 644 root root
}

This rotates logs daily, keeps 7 days of history, and compresses old files to save space.

? Tip: Test your logrotate config with logrotate -d /etc/logrotate.conf before applying.

3. Monitor Log Sizes and Disk Usage

Even with rotation, misconfigurations or sudden spikes in logging can consume disk space.

• Use tools like du, df, and journalctl to check log sizes:

  journalctl --disk-usage        # For systemd journals
  du -sh /var/log/*              # Check individual log directories

• Set up alerts using monitoring tools (e.g., Nagios, Zabbix, or Prometheus) when /var/log exceeds a threshold.

For systemd-based systems, limit journal size in /etc/systemd/journald.conf:

SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=1month

This prevents journals from growing indefinitely.

4. Secure and Protect Log Files

Logs often contain sensitive data (e.g., IP addresses, usernames, error details). Unauthorized access can expose system vulnerabilities.

• Set proper permissions:

  chmod 644 /var/log/*.log       # Readable by admin, not writable by others
  chmod 600 /var/log/sensitive.log  # Restrict access

• Use ACLs or group-based access if needed.
• Consider encrypting logs at rest if compliance (e.g., GDPR, HIPAA) requires it.
• Prevent log tampering by sending logs to a remote, write-only log server.

?? Never store logs in publicly accessible directories (e.g., web root).

5. Filter and Structure Log Output

Applications should log in a consistent, machine-readable format (like JSON) when possible. This makes parsing and alerting easier.

• Use structured logging in apps (e.g., via libraries like logrus or winston).
• Filter unnecessary noise in rsyslog/syslog-ng to avoid clutter.
• Tag logs by service, environment, or host for better filtering.

Example rsyslog filter to route specific logs:

if $programname == 'nginx' then /var/log/nginx/filtered.log
& stop

This reduces noise and improves searchability.

6. Set Up Alerts for Critical Events

Don’t wait for a failure to notice log entries. Proactively monitor for patterns like repeated authentication failures, service crashes, or disk errors.

• Use tools like fail2ban (for SSH brute-force detection) or logwatch (daily summaries).
• Integrate with SIEM tools (e.g., Wazuh, OSSEC) for real-time alerting.
• Write custom scripts with grep, awk, or jq to detect anomalies.

Example: Alert on repeated failed logins:

grep "Failed password" /var/log/auth.log | grep "$(date -d '1 hour ago' ' %b %d %H')"

Automate such checks via cron and email alerts.

Basically, good log management balances retention, performance, and security. Use rotation, centralization, monitoring, and access control together — it’s not complex, but it’s easy to overlook one piece.

The above is the detailed content of Best Practices for Managing Logs on a Linux System. For more information, please follow other related articles on the PHP Chinese website!