Repository: spark Updated Branches: refs/heads/master 20b4ca140 -> 52d4f6194
[SPARK-18909][SQL] The error messages in `ExpressionEncoder.toRow/fromRow` are too verbose ## What changes were proposed in this pull request? In `ExpressionEncoder.toRow` and `fromRow`, we catch the exception and output `treeString` of serializer/deserializer expressions in the error message. However, encoder can be very complex and the serializer/deserializer expressions can be very large trees and blow up the log files(e.g. generate over 500mb logs for this single error message.) As a first attempt, this PR try to use `simpleString` instead. **BEFORE** ```scala scala> :paste // Entering paste mode (ctrl-D to finish) case class TestCaseClass(value: Int) import spark.implicits._ Seq(TestCaseClass(1)).toDS().collect() // Exiting paste mode, now interpreting. java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException newInstance(class TestCaseClass) +- assertnotnull(input[0, int, false], - field (class: "scala.Int", name: "value"), - root class: "TestCaseClass") +- input[0, int, false] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:303) ... ``` **AFTER** ```scala ... // Exiting paste mode, now interpreting. java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException newInstance(class TestCaseClass) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:303) ... ``` ## How was this patch tested? Manual. Author: Dongjoon Hyun <dongj...@apache.org> Closes #16701 from dongjoon-hyun/SPARK-18909-EXPR-ERROR. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/52d4f619 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/52d4f619 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/52d4f619 Branch: refs/heads/master Commit: 52d4f61941b0ac8dcdca04aac042a492141c2068 Parents: 20b4ca1 Author: Dongjoon Hyun <dongj...@apache.org> Authored: Fri Feb 3 20:26:53 2017 +0800 Committer: Wenchen Fan <wenc...@databricks.com> Committed: Fri Feb 3 20:26:53 2017 +0800 ---------------------------------------------------------------------- .../apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/52d4f619/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala ---------------------------------------------------------------------- diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala index 3757ecc..0782143 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala @@ -288,7 +288,7 @@ case class ExpressionEncoder[T]( } catch { case e: Exception => throw new RuntimeException( - s"Error while encoding: $e\n${serializer.map(_.treeString).mkString("\n")}", e) + s"Error while encoding: $e\n${serializer.map(_.simpleString).mkString("\n")}", e) } /** @@ -300,7 +300,7 @@ case class ExpressionEncoder[T]( constructProjection(row).get(0, ObjectType(clsTag.runtimeClass)).asInstanceOf[T] } catch { case e: Exception => - throw new RuntimeException(s"Error while decoding: $e\n${deserializer.treeString}", e) + throw new RuntimeException(s"Error while decoding: $e\n${deserializer.simpleString}", e) } /** --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
