Re: Spark on EMR: out-of-the-box solution for real-time application logs monitoring?

Thanks for your advice, Steve. I'm mainly talking about application logs. To be more clear, just for instance think about the "//hadoop/userlogs/application_blablabla/container_blablabla/stderr_or_stdout". So YARN's applications containers logs, stored (at least for EMR's hadoop 2.4) on DataNodes

Re: Spark on EMR: out-of-the-box solution for real-time application logs monitoring?

> On 10 Dec 2015, at 14:52, Roberto Coluccio wrote: > > Hello, > > I'm investigating on a solution to real-time monitor Spark logs produced by > my EMR cluster in order to collect statistics and trigger alarms. Being on > EMR, I found the CloudWatch Logs + Lambda

Spark on EMR: out-of-the-box solution for real-time application logs monitoring?

Hello, I'm investigating on a solution to real-time monitor Spark logs produced by my EMR cluster in order to collect statistics and trigger alarms. Being on EMR, I found the CloudWatch Logs + Lambda pretty straightforward and, since I'm on AWS, those service are pretty well integrated