I'm so glad you found this info useful! Good luck with your tests and thanks for your nice feedback!
-- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Oct 4, 2014 at 7:44 PM, Carlos Manuel Trepeu Pupo < [email protected]> wrote: > *Radu Gheorghe wowwww ... just wowwww.I'm right now installing my lab to > begin testing all this stuff. Thanks a lot. I'm gonna keep you guys up to > date about everything. Thanks again ...* > > On Sat, Oct 4, 2014 at 12:29 PM, Radu Gheorghe <[email protected] > > > wrote: > > > Hi Carlos, > > > > Yes, you can do that in Elasticsearch, provided that you have the needed > > data in its own field. hich is why people talk about mmnormalize - logs > > typically come in free text format which has to be somehow parsed into a > > nicely formatted JSON for Elasticsearch to consume. > > > > You'd probably make heavy use of aggregations > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations.html > > > > > to meet your requirements. Aggregations give you all sorts of counts over > > sets of documents (called buckets). Specifically: > > 1. If you have timestamps for surfing activities in one field and > > enterprise names in another field, this should be doable with the > > significant > > terms aggregation > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-significantterms-aggregation.html > > >. > > Basically, you'd filter > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/query-dsl-filtered-query.html#query-dsl-filtered-query > > > > > on the timestamp field to get documents from the weekend, and run a > > significant terms aggregation on the enterprise names field. The > > aggregation results would be enterprise names sorted by the difference > > between the foreground set of documents (surfing logs during the weekend > > for that enterprise) and the background set (all surfing logs stored in > the > > indices you search on). In other words, enterprises that surf more on > > weekends compared to the average will come out on top. > > > > 2. If you have the enterprise name and the size of each Email in their > own > > fields, you could use the sum aggregation > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-metrics-sum-aggregation.html > > >. > > You'd filter on a specific enterprise and then run the sum aggregation on > > the size field. > > > > 3. I'm not 100% sure what you mean for this requirement. If you need the > > number of, say, surfing logs generated by an enterprise for each hour of > a > > time interval, you could use the date histogram aggregation > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-datehistogram-aggregation.html > > >. > > Provided that you have timestamps and enterprise names in their own > fields, > > you could filter on the specific enterprise and on the timeframe you're > > interested in, then the date histogram aggregation would give you the > > number of logs in each hour (if you set "interval" to "hour"). > > > > Not sure what you mean by "dates". For "sites" and "files" you'd probably > > use the terms aggregation > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html > > >. > > Provided that you have sites and files in their own field, you should be > > able to filter on a specific enterprise and get the top X unique sites or > > files, ordered by a configurable criterion (default to the number of > > occurrences, which will give you the "most popular" sites and files). > > > > Best regards, > > Radu > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > On Sat, Oct 4, 2014 at 4:59 PM, Carlos Manuel Trepeu Pupo < > > [email protected]> wrote: > > > > > Ok, maybe I don't explain myself as well as I guess. I read about the > log > > > analyzer of elasticsearch, but I understood that analyze is for > statistic > > > of incoming logs and more options, but here are a couple of case that I > > > need to report: > > > > > > 1- My boss ask me for a report for top 10 enterprises that have more > > > surfing in weekend. > > > 2- My principal specialist ask me for the total of outgoing MB of email > > of > > > an user or other Enterprise. > > > 3- There's a problem with an enterprise, so we need to make a report > with > > > the hours (out of work days), dates, sites and files for that > enterprise. > > > > > > Is possible to make this kind of analysis with elasticsearch and export > > it? > > > > > > On Fri, Oct 3, 2014 at 10:53 PM, David Lang <[email protected]> wrote: > > > > > > > What are you looking for when you say "analyze logs" > > > > > > > > There is real-time analysis of logs to look for specific entries or > > > > combinations of entries and generate alerts. Simple Event Correlator > > > (sec) > > > > is a very powerful tool for this sort of work > > > > > > > > There are periodic reports summarizing data into reports > > > > > > > > There is generating trending data (frequently for graphs) > > > > > > > > There is unplanned searches of logs (Elasticsearch is great for this) > > > > > > > > David Lang > > > > > > > > > > > > > > > > On Fri, 3 Oct 2014, Carlos Manuel Trepeu Pupo wrote: > > > > > > > > OK, thanks both of you to answer almost all my doubts. I have been > > > passed > > > >> all day reading and here come new problems. > > > >> > > > >> How can I analyze the LOGs ? I use WebSpy as log analyzer, but > anyone > > of > > > >> you guys tell me how can I analyze POSTFIX, SQUID, FREE RADIUS, and > > > others > > > >> if they are in database? > > > >> > > > >> In case that the databases are in mySQL there is no problem, but > when > > I > > > >> have elasticsearch, what software I can use? > > > >> > > > >> P.S: I read about elasticsearch and I love the way they solve > problems > > > and > > > >> show statistic, but without log analyzer, I can't do anything. > > > >> _______________________________________________ > > > >> rsyslog mailing list > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >> http://www.rsyslog.com/professional-services/ > > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > >> DON'T LIKE THAT. > > > >> > > > >> _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > > DON'T LIKE THAT. > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
