Github user LantaoJin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22353#discussion_r217229063
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala ---
@@ -59,6 +57,12 @@ private[execution] object SparkPlanInfo {
new SQLMetricInfo(metric.name.getOrElse(key), metric.id,
metric.metricType)
}
- new SparkPlanInfo(plan.nodeName, plan.simpleString,
children.map(fromSparkPlan), metrics)
+ // dump the file scan metadata (e.g file path) to event log
--- End diff --
Not yet. This field only removed from 2.3. The event log size should be
same with before 2.3. The main increasing is input path. The example above read
10000 parquet files will log 10000 paths in one SQLExecutionStart event. No
regression pathes. The size increased is foreseeable.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
