[olug] Splunk and log scraping

[Kevin]

Possible solution, will involve a fair amount of setup and know-how:
On each monitored machine, use cron to scp logs over to a destination
log-gathering machine.
On the log-gathering machine:
alias mega-grep='grep -v "undesired pattern 1" machine01/*
machine02/*... | grep -v "undesired pattern 2" | grep -v "undesired
pattern 3"...'

Daisy chain aliases if need be.

Not the prettiest solution, and there's bound to be better ways, but I
don't know of any offhand. Maybe webalizer? Depends on what your logs
are intended to say.

On Thu, Dec 17, 2009 at 19:40, T. J. Brumfield <enderandrew at gmail.com> wrote:
> I was looking at Splunk, and they were quoting us a price of over
> $300,000 per year just for our team to use it. It looks useful, but I
> just can't see justifying the price.
>
> We want a tool to filter through logs to help us get right down to the
> most relevant data. Anyone can manually grep through logs from time to
> time, but it would be nice to automate this process.
>
> We're currently looking a solution to start pointing about 3 gigs of
> logs per day (for one group of users, from one app) to a central
> place, to filter those logs and look for problems. Splunk was the
> first thing we looked at, but I assume there are alternatives. I'm
> trying to get my employer to start looking at and considering some
> OSS, since we're almost entirely a Microsoft company (corporate wide)
> even when vendors encourage otherwise. I was hoping there might be a
> good OSS alternative.
>
> There are a lot of SysAdmins on this group. I can't be the first one
> on this list who has needed a log scraping solution.
>
> -- T. J.
>
> On Thu, Dec 17, 2009 at 4:41 PM, Irish <irish.masms at gmail.com> wrote:
>> On Thu, Dec 17, 2009 at 3:06 PM, Kevin <sharpestmarble at gmail.com> wrote:
>>
>>> From what I remember, Splunk does log mining. "Look at your logs, what
>>> is there interesting?" I haven't used it, though, and all that is
>>> coming just from a combination of the ads I saw and what does an app
>>> like that do.
>>>
>>> I don't know what TJ's research has turned up, nor do I know what he's
>>> trying to accomplish.
>>>
>>
>> I've been using Slunk for about 1.5 years now - not a bad tool for log
>> management IMHO. Point all your systems logs to the Splunk server, get a
>> 'google like' interface to those logs. Good for giving access to those
>> network, desktop, & server admins to help troubleshoot issues - and look for
>> the miscreants on your network.
>> _______________________________________________
>> OLUG mailing list
>> OLUG at olug.org
>> https://lists.olug.org/mailman/listinfo/olug
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> https://lists.olug.org/mailman/listinfo/olug
>