Manage system log files with Logrotate

Manage system log files with Logrotate

full manage logs

Tools

2020.10.27

0 #log #ubuntu #logrotate

Since I’m using Nginx to proxy my AdGuardHome web interface and DNS over HTTPS, the access log files increased rapidly in size. So I’m going to limit these logs with a tool called Logrotate.

Introduction

Log rotation is a process of removing/storing old logs while not affecting the latest logs. Almost every single application generates meaningful logs, and if not properly handled, logs can eat up all the space on your disk. Logrotate helps to manage logs, compress them, remove unnecessary/old logs or even email them after a certain time period.

Installation

Most linux distributions come with logrotate by default. If that’s not the case on your device, just install it with following commands (e.g. Debian/Ubuntu):

sudo apt install logrotate

Configure logrotate

Logrotate has a configuration file in which we can mention all the files we want to rotate. It needs a time period unit i.e., daily, weekly, monthly etc., and rotate count i.e., 3, 4, 5 for each rotation. Log files are rotated count times before being removed. If you set the rotate count to 0, it means old version of logs will be removed rather than rotated. Otherwise, if your logs are being saved in file named myapp.log then after a rotation a new file will be created with name myapp.1.log, and so on.

Configuration file

Logrotate is configured using the main configuration file, or the logrotate configuration folder. the configuration file usually holds the global configuration and the pointer to the configuration folder. In common Linux distributions such as Ubuntu and Debian, the logrotate configuration file can be found:

/etc/logrotate.conf

and the configuration folder can be found:

/etc/logrotate.d/

The configuration folder is usually where you would add new log file configuration which are unique for a particular log file, or set of log files. For example:

/var/lib/docker/containers/*/*.log {
    rotate 5
    copytruncate
    missingok
    notifempty
    compress
    maxsize 200M
    daily
}
  • It represents the path where logrotate will monitor the log files with .log file extension.
  • Store logs till 5 rotations, which means it will store maximum 5 files of old logs when rotation hits, and will discard oldest log when there are already 5 files.
  • Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its log file and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost.
  • missingok indicates if the log file is missing, do not generate an error, and move on the next file.
  • Then do not rotate the log if it is empty.
  • Old version of logs are compressed.
  • Rotate the log file if it exceeds 200Mb, regardless of the rotation time unit.
  • Rotation process should happen daily.

Below will explain some commonly used configurations in detail.

Create a new empty template

To create a new logrotate configuration file, you need to create a new file in /etc/logrotate.d/. You will then need to add a reference to the log files you wish to rotate. This can be directly to a single file or use pattern matching to match a group of log files. The below example matches all log files in /var/log/myapp/ which have a .log extension:

/var/log/myapp/*.log {

}

You will need to add further commands to this template before it becomes useful.

Rotate based on log file size

Use the size keyword to rotate the log file when it exceeded a given file size. The below example rotates a file when it reaches 10 KB:

/var/log/myapp/*.log {
  size 10k
}

Rotate based on time

You can rotate logs using the monthly, weekly, or daily keyword to create a new log based on duration. The keywords explain them selves, and they can be used in conjunction with the size keyword to rotate on which ever criteria is met first.

/var/log/myapp/*.log {
  size 10k
  weekly
}

Limit how many log files are kept after rotation by number

The rotate keyword allows us to specify how many old, rotated, log files are kept before logrotate deletes them. The rotate keyword requires an integer to specify the number of old log files.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
}

Limit how many files are kept after rotation by date

You can specify how long to keep rotated files using the maxage keyword. Any rotated log files which are older then the maxage will be deleted. The below example will keep rotated log files 49 days.

/var/log/myapp/*.log {
  size 10k
  weekly
  maxage 49
}

Compress rotated log files

Log files which have been rotated can be compressed to save disk space. Gzip is used by default.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
  compress
}

You can change the default gzip compression to another format by specifying the compresscmd command with a different executable to use. For example, change it to bzip2:

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
  compress
  compresscmd /bin/bzip2
}

Ignore missing log files

If a log file does not exist when logrotate is running then an error will be thrown. You can use the keyword missingok to avoid this scenario.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
  missingok
}

Continue writing to the same file after rotation

Usually when a log file is rotated, the log file is moved to a new location. Some applications may throw an error, and others may continue to write to the relocated file. The copytruncate keyword copies all the log in the file to a new file and then truncates the original file. This keeps the original log file in place and also allows rotation to continue.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
  copytruncate
}

For complete details about logrotate configuration, please refer to man logrotate.

Clean up all log files

If you intend to delete all the log file in /var/log/, you can use the following script:

# echo empty content to log files
logs=`find /var/log -type f`
for item in $logs
do
> $item
done

# delete logrotate files
find /var/log -name "*.[0-9]*" -type f -delete
Ads by Google

林宏

Frank Lin

Hey, there! This is Frank Lin (@flinhong), one of the 1.4 billion 🇨🇳. This 'inDev. Journal' site holds the exploration of my quirky thoughts and random adventures through life. Hope you enjoy reading and perusing my posts.

YOU MAY ALSO LIKE

Setup an IKEv2 server with StrongSwan

Tutorials

2020.01.09

Setup an IKEv2 server with StrongSwan

IKEv2, or Internet Key Exchange v2, is a protocol that allows for direct IPSec tunneling between two points. In IKEv2 implementations, IPSec provides encryption for the network traffic. IKEv2 is natively supported on some platforms (OS X 10.11+, iOS 9.1+, and Windows 10) with no additional applications necessary, and it handles client hiccups quite smoothly.

Using Liquid in Jekyll - Live with Demos

Web Notes

2016.08.20

Using Liquid in Jekyll - Live with Demos

Liquid is a simple templating language that Jekyll uses to process pages on your site. With Liquid you can output an modify variables, have logic statements inside your pages and loop over content.

Hands on IBM Cloud Functions with CLI

Tools

2020.10.20

Hands on IBM Cloud Functions with CLI

IBM Cloud CLI allows complete management of the Cloud Functions system. You can use the Cloud Functions CLI plugin-in to manage your code snippets in actions, create triggers, and rules to enable your actions to respond to events, and bundle actions into packages.