[olug] Log Retention

[Christopher Cashell]

On Sun, Dec 11, 2011 at 7:11 PM, T. J. Brumfield <enderandrew at gmail.com> wrote:
> I need to make sure I've got something in place to keep log files in check.
> I don't want to turn off logging, but I need to rotate logs at file size X
> and then delete logs when I get over Y number of logs so I keep my overall
> log storage at a fairly constant number.

logrotate is probably the tool you'll want to be using.  It's the
standard log rotation tool on most Linux systems.  It can be
configured to compress logs, to rotate on multiple schedules (daily,
weekly, etc), or based on log size.  Default compression is gzip,
although you can use an alternate compression commend if you really
need to (bzip2 tends to do very well with text).

If you have logs that will or could grow at a fast rate, you'll want
to rotate them more frequently.  Logs, being text, tend to compress
very well (usually down to 10% or smaller as compared to the original
size).  That means that your live uncompressed log file will be the
key that determines how much you can retain and how fast you need to
rotate.

> I need to do this for MySQL, Apache, PHP, Murmur and local system logs
> because I can't afford to let any logs run me out of disk space.

logrotate can handle logs for any of those, and is probably already
configured for many of them.  You might just need to turn on
compression and tweak some settings to match your requirements.

Another option you might want to consider is remote logging.  Send
your systems logs to a remote box with more disk space for long term
archiving, and only keep a few days worth of logs on the local system.
 This can be done as a batch process (cron'ed scp of the compressed
logs every night), or it can be done live, by configuring all your
log-worthy things to log to syslog, and then configure syslog to send
the logs to a remote destination.

The latter is actually what I do with my VPS.  It keeps 1 week of logs
local, and also sends all logs back to a separate loghost (home
server) for longer storage.  I have a VPN connection running between
the two for secure transmission (I use OpenVPN, initiated from the
home server, to ease firewall/dynamic IP issues on the home box side).

> -- T. J. Brumfield

-- 
Christopher