*Radu Gheorghe wowwww ... just wowwww.I'm right now installing my lab to begin testing all this stuff. Thanks a lot. I'm gonna keep you guys up to date about everything. Thanks again ...*
On Sat, Oct 4, 2014 at 12:29 PM, Radu Gheorghe <[email protected]> wrote: > Hi Carlos, > > Yes, you can do that in Elasticsearch, provided that you have the needed > data in its own field. hich is why people talk about mmnormalize - logs > typically come in free text format which has to be somehow parsed into a > nicely formatted JSON for Elasticsearch to consume. > > You'd probably make heavy use of aggregations > < > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations.html > > > to meet your requirements. Aggregations give you all sorts of counts over > sets of documents (called buckets). Specifically: > 1. If you have timestamps for surfing activities in one field and > enterprise names in another field, this should be doable with the > significant > terms aggregation > < > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-significantterms-aggregation.html > >. > Basically, you'd filter > < > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/query-dsl-filtered-query.html#query-dsl-filtered-query > > > on the timestamp field to get documents from the weekend, and run a > significant terms aggregation on the enterprise names field. The > aggregation results would be enterprise names sorted by the difference > between the foreground set of documents (surfing logs during the weekend > for that enterprise) and the background set (all surfing logs stored in the > indices you search on). In other words, enterprises that surf more on > weekends compared to the average will come out on top. > > 2. If you have the enterprise name and the size of each Email in their own > fields, you could use the sum aggregation > < > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-metrics-sum-aggregation.html > >. > You'd filter on a specific enterprise and then run the sum aggregation on > the size field. > > 3. I'm not 100% sure what you mean for this requirement. If you need the > number of, say, surfing logs generated by an enterprise for each hour of a > time interval, you could use the date histogram aggregation > < > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-datehistogram-aggregation.html > >. > Provided that you have timestamps and enterprise names in their own fields, > you could filter on the specific enterprise and on the timeframe you're > interested in, then the date histogram aggregation would give you the > number of logs in each hour (if you set "interval" to "hour"). > > Not sure what you mean by "dates". For "sites" and "files" you'd probably > use the terms aggregation > < > http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html > >. > Provided that you have sites and files in their own field, you should be > able to filter on a specific enterprise and get the top X unique sites or > files, ordered by a configurable criterion (default to the number of > occurrences, which will give you the "most popular" sites and files). > > Best regards, > Radu > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > On Sat, Oct 4, 2014 at 4:59 PM, Carlos Manuel Trepeu Pupo < > [email protected]> wrote: > > > Ok, maybe I don't explain myself as well as I guess. I read about the log > > analyzer of elasticsearch, but I understood that analyze is for statistic > > of incoming logs and more options, but here are a couple of case that I > > need to report: > > > > 1- My boss ask me for a report for top 10 enterprises that have more > > surfing in weekend. > > 2- My principal specialist ask me for the total of outgoing MB of email > of > > an user or other Enterprise. > > 3- There's a problem with an enterprise, so we need to make a report with > > the hours (out of work days), dates, sites and files for that enterprise. > > > > Is possible to make this kind of analysis with elasticsearch and export > it? > > > > On Fri, Oct 3, 2014 at 10:53 PM, David Lang <[email protected]> wrote: > > > > > What are you looking for when you say "analyze logs" > > > > > > There is real-time analysis of logs to look for specific entries or > > > combinations of entries and generate alerts. Simple Event Correlator > > (sec) > > > is a very powerful tool for this sort of work > > > > > > There are periodic reports summarizing data into reports > > > > > > There is generating trending data (frequently for graphs) > > > > > > There is unplanned searches of logs (Elasticsearch is great for this) > > > > > > David Lang > > > > > > > > > > > > On Fri, 3 Oct 2014, Carlos Manuel Trepeu Pupo wrote: > > > > > > OK, thanks both of you to answer almost all my doubts. I have been > > passed > > >> all day reading and here come new problems. > > >> > > >> How can I analyze the LOGs ? I use WebSpy as log analyzer, but anyone > of > > >> you guys tell me how can I analyze POSTFIX, SQUID, FREE RADIUS, and > > others > > >> if they are in database? > > >> > > >> In case that the databases are in mySQL there is no problem, but when > I > > >> have elasticsearch, what software I can use? > > >> > > >> P.S: I read about elasticsearch and I love the way they solve problems > > and > > >> show statistic, but without log analyzer, I can't do anything. > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com/professional-services/ > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >> DON'T LIKE THAT. > > >> > > >> _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
