Currently there seems to be 3 places to check task level logs: 1) Using spark UI 2) `yarn application log` 3) log aggregation on hdfs (if enabled)
All above only give you log at executor(container) level. However one executor can have multiple threads and each might be running part of different stages(stg1, stg2)) and within that different tasks(tid1, tid2...) . It's hard to track particular task activities in Executor logs. It'd be nice if: 1) mark all log entries with stageId followed by taskId 2) have a separate log file for each task (taskId) 3) have a separated log file for of stage level logs If I missed something let me know Thanks, Nirav