Hi Mat, I think you could also use spark SQL to query the logs. Hope the following link helps
https://databricks.com/blog/2014/09/23/databricks-reference-applications.html On May 23, 2016 10:59 AM, "Mat Schaffer" <m...@schaffer.me> wrote: > I'm curious about trying to use spark as a cheap/slow ELK > (ElasticSearch,Logstash,Kibana) system. Thinking something like: > > - instances rotate local logs > - copy rotated logs to s3 > (s3://logs/region/grouping/instance/service/*.logs) > - spark to convert from raw text logs to parquet > - maybe presto to query the parquet? > > I'm still new on Spark though, so thought I'd ask if anyone was familiar > with this sort of thing and if there are maybe some articles or documents I > should be looking at in order to learn how to build such a thing. Or if > such a thing even made sense. > > Thanks in advance, and apologies if this has already been asked and I > missed it! > > -Mat > > matschaffer.com >
