[olug] Splunk and log scraping

[T. J. Brumfield]

phpLogCon looks promising, except it says I need to specify specific
text files to monitor for logs. The documentation doesn't tell me if I
can monitor an entire directory. I need to monitor application logs
for thousands of users. The file names will be based off nt login, and
will change as employee turnover occurs.

It also says it can talk to external tools, but doesn't offer any
specifics there either. It would be nice if I could send an alert to
SCOM.

-- T. J.

On Thu, Dec 17, 2009 at 11:53 PM, Curtis LaMasters
<curtislamasters at gmail.com> wrote:
> I have phplogcon running at a few locations.  That might be worth a
> shot.  It's also prepackaged for VMWare at
> http://www.syslogappliance.de/en/.
>
> Curtis LaMasters
> http://www.curtis-lamasters.com
> http://www.builtnetworks.com
>
>
>
> On Thu, Dec 17, 2009 at 11:30 PM, Matt Goeres <mgoeres at gmail.com> wrote:
>> I usually use OSSEC for real time alerting and then manual awk grep and sed for most everything else.
>>
>> --Matt
>>
>> On Thu, Dec 17, 2009 at 07:48:50PM -0600, Kevin wrote:
>>> Possible solution, will involve a fair amount of setup and know-how:
>>> On each monitored machine, use cron to scp logs over to a destination
>>> log-gathering machine.
>>> On the log-gathering machine:
>>> alias mega-grep='grep -v "undesired pattern 1" machine01/*
>>> machine02/*... | grep -v "undesired pattern 2" | grep -v "undesired
>>> pattern 3"...'
>>>
>>> Daisy chain aliases if need be.
>>>
>>> Not the prettiest solution, and there's bound to be better ways, but I
>>> don't know of any offhand. Maybe webalizer? Depends on what your logs
>>> are intended to say.
>>>
>>> On Thu, Dec 17, 2009 at 19:40, T. J. Brumfield <enderandrew at gmail.com> wrote:
>>> > I was looking at Splunk, and they were quoting us a price of over
>>> > $300,000 per year just for our team to use it. It looks useful, but I
>>> > just can't see justifying the price.
>>> >
>>> > We want a tool to filter through logs to help us get right down to the
>>> > most relevant data. Anyone can manually grep through logs from time to
>>> > time, but it would be nice to automate this process.
>>> >
>>> > We're currently looking a solution to start pointing about 3 gigs of
>>> > logs per day (for one group of users, from one app) to a central
>>> > place, to filter those logs and look for problems. Splunk was the
>>> > first thing we looked at, but I assume there are alternatives. I'm
>>> > trying to get my employer to start looking at and considering some
>>> > OSS, since we're almost entirely a Microsoft company (corporate wide)
>>> > even when vendors encourage otherwise. I was hoping there might be a
>>> > good OSS alternative.
>>> >
>>> > There are a lot of SysAdmins on this group. I can't be the first one
>>> > on this list who has needed a log scraping solution.
>>> >
>>> > -- T. J.
>>> >
>>> > On Thu, Dec 17, 2009 at 4:41 PM, Irish <irish.masms at gmail.com> wrote:
>>> >> On Thu, Dec 17, 2009 at 3:06 PM, Kevin <sharpestmarble at gmail.com> wrote:
>>> >>
>>> >>> From what I remember, Splunk does log mining. "Look at your logs, what
>>> >>> is there interesting?" I haven't used it, though, and all that is
>>> >>> coming just from a combination of the ads I saw and what does an app
>>> >>> like that do.
>>> >>>
>>> >>> I don't know what TJ's research has turned up, nor do I know what he's
>>> >>> trying to accomplish.
>>> >>>
>>> >>
>>> >> I've been using Slunk for about 1.5 years now - not a bad tool for log
>>> >> management IMHO. Point all your systems logs to the Splunk server, get a
>>> >> 'google like' interface to those logs. Good for giving access to those
>>> >> network, desktop, & server admins to help troubleshoot issues - and look for
>>> >> the miscreants on your network.
>>> >> _______________________________________________
>>> >> OLUG mailing list
>>> >> OLUG at olug.org
>>> >> https://lists.olug.org/mailman/listinfo/olug
>>> > _______________________________________________
>>> > OLUG mailing list
>>> > OLUG at olug.org
>>> > https://lists.olug.org/mailman/listinfo/olug
>>> >
>>> _______________________________________________
>>> OLUG mailing list
>>> OLUG at olug.org
>>> https://lists.olug.org/mailman/listinfo/olug
>> _______________________________________________
>> OLUG mailing list
>> OLUG at olug.org
>> https://lists.olug.org/mailman/listinfo/olug
>>
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> https://lists.olug.org/mailman/listinfo/olug