Github user boy-uber commented on the issue:
https://github.com/apache/spark/pull/22429
Great to see the code here to capture more information from the plan and
save to file. We have similar needs with our Spark applications as well.
Adding my two cents:
1. Anyone feel the need to log the spark plan as a json or other structured
representation? Current text based logging for spark plan is good for human,
but hard to be processed by code/tool.
2. Could we generalize the solution a little bit? for example, writing the
logical/physical plan json representation (or similar tree structure) to Spark
listener, so people could write their own plugin to either write to file, or
write to database, or write to kafka, etc.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
