I'm curious about trying to use spark as a cheap/slow ELK (ElasticSearch,Logstash,Kibana) system. Thinking something like:
- instances rotate local logs - copy rotated logs to s3 (s3://logs/region/grouping/instance/service/*.logs) - spark to convert from raw text logs to parquet - maybe presto to query the parquet? I'm still new on Spark though, so thought I'd ask if anyone was familiar with this sort of thing and if there are maybe some articles or documents I should be looking at in order to learn how to build such a thing. Or if such a thing even made sense. Thanks in advance, and apologies if this has already been asked and I missed it! -Mat matschaffer.com
