Hi Carlos, Yes, you can do that in Elasticsearch, provided that you have the needed data in its own field. hich is why people talk about mmnormalize - logs typically come in free text format which has to be somehow parsed into a nicely formatted JSON for Elasticsearch to consume.
You'd probably make heavy use of aggregations <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations.html> to meet your requirements. Aggregations give you all sorts of counts over sets of documents (called buckets). Specifically: 1. If you have timestamps for surfing activities in one field and enterprise names in another field, this should be doable with the significant terms aggregation <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-significantterms-aggregation.html>. Basically, you'd filter <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/query-dsl-filtered-query.html#query-dsl-filtered-query> on the timestamp field to get documents from the weekend, and run a significant terms aggregation on the enterprise names field. The aggregation results would be enterprise names sorted by the difference between the foreground set of documents (surfing logs during the weekend for that enterprise) and the background set (all surfing logs stored in the indices you search on). In other words, enterprises that surf more on weekends compared to the average will come out on top. 2. If you have the enterprise name and the size of each Email in their own fields, you could use the sum aggregation <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-metrics-sum-aggregation.html>. You'd filter on a specific enterprise and then run the sum aggregation on the size field. 3. I'm not 100% sure what you mean for this requirement. If you need the number of, say, surfing logs generated by an enterprise for each hour of a time interval, you could use the date histogram aggregation <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-datehistogram-aggregation.html>. Provided that you have timestamps and enterprise names in their own fields, you could filter on the specific enterprise and on the timeframe you're interested in, then the date histogram aggregation would give you the number of logs in each hour (if you set "interval" to "hour"). Not sure what you mean by "dates". For "sites" and "files" you'd probably use the terms aggregation <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html>. Provided that you have sites and files in their own field, you should be able to filter on a specific enterprise and get the top X unique sites or files, ordered by a configurable criterion (default to the number of occurrences, which will give you the "most popular" sites and files). Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Oct 4, 2014 at 4:59 PM, Carlos Manuel Trepeu Pupo < [email protected]> wrote: > Ok, maybe I don't explain myself as well as I guess. I read about the log > analyzer of elasticsearch, but I understood that analyze is for statistic > of incoming logs and more options, but here are a couple of case that I > need to report: > > 1- My boss ask me for a report for top 10 enterprises that have more > surfing in weekend. > 2- My principal specialist ask me for the total of outgoing MB of email of > an user or other Enterprise. > 3- There's a problem with an enterprise, so we need to make a report with > the hours (out of work days), dates, sites and files for that enterprise. > > Is possible to make this kind of analysis with elasticsearch and export it? > > On Fri, Oct 3, 2014 at 10:53 PM, David Lang <[email protected]> wrote: > > > What are you looking for when you say "analyze logs" > > > > There is real-time analysis of logs to look for specific entries or > > combinations of entries and generate alerts. Simple Event Correlator > (sec) > > is a very powerful tool for this sort of work > > > > There are periodic reports summarizing data into reports > > > > There is generating trending data (frequently for graphs) > > > > There is unplanned searches of logs (Elasticsearch is great for this) > > > > David Lang > > > > > > > > On Fri, 3 Oct 2014, Carlos Manuel Trepeu Pupo wrote: > > > > OK, thanks both of you to answer almost all my doubts. I have been > passed > >> all day reading and here come new problems. > >> > >> How can I analyze the LOGs ? I use WebSpy as log analyzer, but anyone of > >> you guys tell me how can I analyze POSTFIX, SQUID, FREE RADIUS, and > others > >> if they are in database? > >> > >> In case that the databases are in mySQL there is no problem, but when I > >> have elasticsearch, what software I can use? > >> > >> P.S: I read about elasticsearch and I love the way they solve problems > and > >> show statistic, but without log analyzer, I can't do anything. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
