Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r217928334
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ---
@@ -250,5 +254,36 @@ class QueryExecution(val sparkSession: SparkSession,
val logical: LogicalPlan) {
def codegenToSeq(): Seq[(String, String)] = {
org.apache.spark.sql.execution.debug.codegenStringSeq(executedPlan)
}
+
+ /**
+ * Dumps debug information about query execution into the specified
file.
+ */
+ def toFile(path: String): Unit = {
+ val maxFields = SparkEnv.get.conf.getInt(Utils.MAX_TO_STRING_FIELDS,
+ Utils.DEFAULT_MAX_TO_STRING_FIELDS)
+ val filePath = new Path(path)
+ val fs = FileSystem.get(filePath.toUri,
sparkSession.sessionState.newHadoopConf())
+ val writer = new BufferedWriter(new
OutputStreamWriter(fs.create(filePath)))
+
+ try {
+ SparkEnv.get.conf.set(Utils.MAX_TO_STRING_FIELDS,
Int.MaxValue.toString)
--- End diff --
It is generally a bad idea to change this conf as people expect that it is
immutable. Also this change has some far reaching consequences, others will now
also be exposed to a different `Utils.MAX_TO_STRING_FIELDS` value when calling
`explain()`. Can you please just pass the parameter down the tree?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
