Re: need basic information
Do logstash/graylog2 do log processing/searching in real time? Or can scale for real time need? I guess harshadmehta is looking for real-time indexing/search. Regards Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588p4004996.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: need basic information
Agreed. There are a lot of products that do this already. Writing it from scratch in Solr seems like a huge waste of time. You should also check out Graylog2: http://graylog2.org/ wunder On Aug 31, 2012, at 7:05 AM, Alexandre Rafalovitch wrote: > Have you tried looking at http://logstash.net/ first? Or Splunk > (http://www.splunk.com/) if you have money These might be a better > starting point than bare SOLR. > > Regards, > Alex > Personal blog: http://blog.outerthoughts.com/ > LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch > - Time is the quality of nature that keeps events from happening all > at once. Lately, it doesn't seem to be working. (Anonymous - via GTD > book) > > > On Thu, Aug 30, 2012 at 11:03 PM, harshadmehta wrote: >> I am trying to use solr for processing logs and searching them. >> >> But i dont see a clear sample anywhere for this type of scenario. >> >> i need to index plain text files , example content - >> >> -mm-dd : Account 123 created >> -mm-dd : Account 123 updated >> >> >> Account 123 being spread across multiple files. >> >> How do i index this so that i can search Account 123 activity over a date >> range. >> >> Using default solr config search, i will get each log file in full that has >> any entry for Account 123 plus all other Accounts in those files as well. >> >> Thanks >> >> >> >> >> >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588.html >> Sent from the Solr - User mailing list archive at Nabble.com. -- Walter Underwood wun...@wunderwood.org
Re: need basic information
Think of the log file as a flat database, each line/entry a "row". So, each log line/entry would need to be added to Solr as a separate document. Maybe you could do this using DIH and a LineEntityProcessor and RegexTransformer, DateFormatTransformer, etc. -- Jack Krupansky -Original Message- From: harshadmehta Sent: Thursday, August 30, 2012 11:03 PM To: solr-user@lucene.apache.org Subject: need basic information I am trying to use solr for processing logs and searching them. But i dont see a clear sample anywhere for this type of scenario. i need to index plain text files , example content - -mm-dd : Account 123 created -mm-dd : Account 123 updated Account 123 being spread across multiple files. How do i index this so that i can search Account 123 activity over a date range. Using default solr config search, i will get each log file in full that has any entry for Account 123 plus all other Accounts in those files as well. Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: need basic information
I have looked at splunk and logstash but want to explore solr to do the job. Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588p4004763.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: need basic information
Have you tried looking at http://logstash.net/ first? Or Splunk (http://www.splunk.com/) if you have money These might be a better starting point than bare SOLR. Regards, Alex Personal blog: http://blog.outerthoughts.com/ LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch - Time is the quality of nature that keeps events from happening all at once. Lately, it doesn't seem to be working. (Anonymous - via GTD book) On Thu, Aug 30, 2012 at 11:03 PM, harshadmehta wrote: > I am trying to use solr for processing logs and searching them. > > But i dont see a clear sample anywhere for this type of scenario. > > i need to index plain text files , example content - > > -mm-dd : Account 123 created > -mm-dd : Account 123 updated > > > Account 123 being spread across multiple files. > > How do i index this so that i can search Account 123 activity over a date > range. > > Using default solr config search, i will get each log file in full that has > any entry for Account 123 plus all other Accounts in those files as well. > > Thanks > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: need basic information
You got what i am looking for but indexing part is where i am not sure how it needs to be done. So to send these log files for indexing in CSV format, is it just a configuration change to pull these 3 fields from each line in text files or i need to write code for that. I simplified the lines in text file but they will contain lot more text, but i don't think that will matter. Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588p4004660.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: need basic information
One basic and trivial solution could be to have schema like; Date (of type date/string) --> this would store the '-mm-dd' format date Tag (of type string) --> the text/tag 'Account' goes into this account-id (of type sint/int) --> account id like '123' goes into this action (of type sting) --> values like 'created'/'updated' goes into this Then just push your logs into solr. http://wiki.apache.org/solr/UpdateCSV http://wiki.apache.org/solr/UpdateCSV Then to get log activity for account id '123', you could query like: http://localhost:/solr/select/?q=id:123&fq=Tag:Account&fq=Date:[d1 TO d2] then process the results for plotting/reporting OR you could ask for faceting on the 'action' field like; http://localhost:/solr/select/?q=id:123&fq=Tag:Account&fq=Date:[d1 TO d2]&facet=true&facet.field=action This way you have facet count for created/updated/deleted etc. Hope this is what u r looking for. Thanx Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588p4004637.html Sent from the Solr - User mailing list archive at Nabble.com.
need basic information
I am trying to use solr for processing logs and searching them. But i dont see a clear sample anywhere for this type of scenario. i need to index plain text files , example content - -mm-dd : Account 123 created -mm-dd : Account 123 updated Account 123 being spread across multiple files. How do i index this so that i can search Account 123 activity over a date range. Using default solr config search, i will get each log file in full that has any entry for Account 123 plus all other Accounts in those files as well. Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588.html Sent from the Solr - User mailing list archive at Nabble.com.