[olug] Splunk and log scraping

[Matt Goeres]

I usually use OSSEC for real time alerting and then manual awk grep and sed for most everything else.

--Matt

On Thu, Dec 17, 2009 at 07:48:50PM -0600, Kevin wrote:
> Possible solution, will involve a fair amount of setup and know-how:
> On each monitored machine, use cron to scp logs over to a destination
> log-gathering machine.
> On the log-gathering machine:
> alias mega-grep='grep -v "undesired pattern 1" machine01/*
> machine02/*... | grep -v "undesired pattern 2" | grep -v "undesired
> pattern 3"...'
> 
> Daisy chain aliases if need be.
> 
> Not the prettiest solution, and there's bound to be better ways, but I
> don't know of any offhand. Maybe webalizer? Depends on what your logs
> are intended to say.
> 
> On Thu, Dec 17, 2009 at 19:40, T. J. Brumfield <enderandrew at gmail.com> wrote:
> > I was looking at Splunk, and they were quoting us a price of over
> > $300,000 per year just for our team to use it. It looks useful, but I
> > just can't see justifying the price.
> >
> > We want a tool to filter through logs to help us get right down to the
> > most relevant data. Anyone can manually grep through logs from time to
> > time, but it would be nice to automate this process.
> >
> > We're currently looking a solution to start pointing about 3 gigs of
> > logs per day (for one group of users, from one app) to a central
> > place, to filter those logs and look for problems. Splunk was the
> > first thing we looked at, but I assume there are alternatives. I'm
> > trying to get my employer to start looking at and considering some
> > OSS, since we're almost entirely a Microsoft company (corporate wide)
> > even when vendors encourage otherwise. I was hoping there might be a
> > good OSS alternative.
> >
> > There are a lot of SysAdmins on this group. I can't be the first one
> > on this list who has needed a log scraping solution.
> >
> > -- T. J.
> >
> > On Thu, Dec 17, 2009 at 4:41 PM, Irish <irish.masms at gmail.com> wrote:
> >> On Thu, Dec 17, 2009 at 3:06 PM, Kevin <sharpestmarble at gmail.com> wrote:
> >>
> >>> From what I remember, Splunk does log mining. "Look at your logs, what
> >>> is there interesting?" I haven't used it, though, and all that is
> >>> coming just from a combination of the ads I saw and what does an app
> >>> like that do.
> >>>
> >>> I don't know what TJ's research has turned up, nor do I know what he's
> >>> trying to accomplish.
> >>>
> >>
> >> I've been using Slunk for about 1.5 years now - not a bad tool for log
> >> management IMHO. Point all your systems logs to the Splunk server, get a
> >> 'google like' interface to those logs. Good for giving access to those
> >> network, desktop, & server admins to help troubleshoot issues - and look for
> >> the miscreants on your network.
> >> _______________________________________________
> >> OLUG mailing list
> >> OLUG at olug.org
> >> https://lists.olug.org/mailman/listinfo/olug
> > _______________________________________________
> > OLUG mailing list
> > OLUG at olug.org
> > https://lists.olug.org/mailman/listinfo/olug
> >
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> https://lists.olug.org/mailman/listinfo/olug