Repository: spark Updated Branches: refs/heads/master 2022afe57 -> 54f758b5f
[SPARK-15884][SPARKR][SQL] Overriding stringArgs in MapPartitionsInR ## What changes were proposed in this pull request? As discussed in https://github.com/apache/spark/pull/12836 we need to override stringArgs method in MapPartitionsInR in order to avoid too large strings generated by "stringArgs" method based on the input arguments. In this case exclude some of the input arguments: serialized R objects. ## How was this patch tested? Existing test cases Author: Narine Kokhlikyan <narine.kokhlik...@gmail.com> Closes #13610 from NarineK/dapply_MapPartitionsInR_stringArgs. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/54f758b5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/54f758b5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/54f758b5 Branch: refs/heads/master Commit: 54f758b5fc60ecb0da6b191939a72ef5829be38c Parents: 2022afe Author: Narine Kokhlikyan <narine.kokhlik...@gmail.com> Authored: Fri Jun 10 17:17:47 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Fri Jun 10 17:17:47 2016 -0700 ---------------------------------------------------------------------- .../org/apache/spark/sql/catalyst/plans/logical/object.scala | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/54f758b5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala ---------------------------------------------------------------------- diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala index 55d8adf..78e8822 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala @@ -141,6 +141,9 @@ case class MapPartitionsInR( outputObjAttr: Attribute, child: LogicalPlan) extends ObjectConsumer with ObjectProducer { override lazy val schema = outputSchema + + override protected def stringArgs: Iterator[Any] = Iterator(inputSchema, outputSchema, + outputObjAttr, child) } object MapElements { --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org