[olug] Splunk and log scraping

[Curtis LaMasters]

I have phplogcon running at a few locations.  That might be worth a
shot.  It's also prepackaged for VMWare at
http://www.syslogappliance.de/en/.

Curtis LaMasters
http://www.curtis-lamasters.com
http://www.builtnetworks.com



On Thu, Dec 17, 2009 at 11:30 PM, Matt Goeres <mgoeres at gmail.com> wrote:
> I usually use OSSEC for real time alerting and then manual awk grep and sed for most everything else.
>
> --Matt
>
> On Thu, Dec 17, 2009 at 07:48:50PM -0600, Kevin wrote:
>> Possible solution, will involve a fair amount of setup and know-how:
>> On each monitored machine, use cron to scp logs over to a destination
>> log-gathering machine.
>> On the log-gathering machine:
>> alias mega-grep='grep -v "undesired pattern 1" machine01/*
>> machine02/*... | grep -v "undesired pattern 2" | grep -v "undesired
>> pattern 3"...'
>>
>> Daisy chain aliases if need be.
>>
>> Not the prettiest solution, and there's bound to be better ways, but I
>> don't know of any offhand. Maybe webalizer? Depends on what your logs
>> are intended to say.
>>
>> On Thu, Dec 17, 2009 at 19:40, T. J. Brumfield <enderandrew at gmail.com> wrote:
>> > I was looking at Splunk, and they were quoting us a price of over
>> > $300,000 per year just for our team to use it. It looks useful, but I
>> > just can't see justifying the price.
>> >
>> > We want a tool to filter through logs to help us get right down to the
>> > most relevant data. Anyone can manually grep through logs from time to
>> > time, but it would be nice to automate this process.
>> >
>> > We're currently looking a solution to start pointing about 3 gigs of
>> > logs per day (for one group of users, from one app) to a central
>> > place, to filter those logs and look for problems. Splunk was the
>> > first thing we looked at, but I assume there are alternatives. I'm
>> > trying to get my employer to start looking at and considering some
>> > OSS, since we're almost entirely a Microsoft company (corporate wide)
>> > even when vendors encourage otherwise. I was hoping there might be a
>> > good OSS alternative.
>> >
>> > There are a lot of SysAdmins on this group. I can't be the first one
>> > on this list who has needed a log scraping solution.
>> >
>> > -- T. J.
>> >
>> > On Thu, Dec 17, 2009 at 4:41 PM, Irish <irish.masms at gmail.com> wrote:
>> >> On Thu, Dec 17, 2009 at 3:06 PM, Kevin <sharpestmarble at gmail.com> wrote:
>> >>
>> >>> From what I remember, Splunk does log mining. "Look at your logs, what
>> >>> is there interesting?" I haven't used it, though, and all that is
>> >>> coming just from a combination of the ads I saw and what does an app
>> >>> like that do.
>> >>>
>> >>> I don't know what TJ's research has turned up, nor do I know what he's
>> >>> trying to accomplish.
>> >>>
>> >>
>> >> I've been using Slunk for about 1.5 years now - not a bad tool for log
>> >> management IMHO. Point all your systems logs to the Splunk server, get a
>> >> 'google like' interface to those logs. Good for giving access to those
>> >> network, desktop, & server admins to help troubleshoot issues - and look for
>> >> the miscreants on your network.
>> >> _______________________________________________
>> >> OLUG mailing list
>> >> OLUG at olug.org
>> >> https://lists.olug.org/mailman/listinfo/olug
>> > _______________________________________________
>> > OLUG mailing list
>> > OLUG at olug.org
>> > https://lists.olug.org/mailman/listinfo/olug
>> >
>> _______________________________________________
>> OLUG mailing list
>> OLUG at olug.org
>> https://lists.olug.org/mailman/listinfo/olug
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> https://lists.olug.org/mailman/listinfo/olug
>