Hi guys, curious how you deal with the logs. I feel difficulty in debugging with the logs: run spark-streaming in our yarn cluster using client-mode. So have two logs: yarn log and local log ( for client ). Whenever I have problem, the log is too big to read with "gedit" and "grep". (e.g. after running 10 hours, the local log is 1GB ). Do you use any tools to analyze/monitor/read the logs? such as logstash?
Thanks, Fang, Yan yanfang...@gmail.com +1 (206) 849-4108
