Repository: spark Updated Branches: refs/heads/branch-2.0 f41f433b1 -> 0a450cfff
[SPARK-15884][SPARKR][SQL] Overriding stringArgs in MapPartitionsInR ## What changes were proposed in this pull request? As discussed in https://github.com/apache/spark/pull/12836 we need to override stringArgs method in MapPartitionsInR in order to avoid too large strings generated by "stringArgs" method based on the input arguments. In this case exclude some of the input arguments: serialized R objects. ## How was this patch tested? Existing test cases Author: Narine Kokhlikyan <narine.kokhlik...@gmail.com> Closes #13610 from NarineK/dapply_MapPartitionsInR_stringArgs. (cherry picked from commit 54f758b5fc60ecb0da6b191939a72ef5829be38c) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0a450cff Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0a450cff Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0a450cff Branch: refs/heads/branch-2.0 Commit: 0a450cfffada67f841795a09af3bf6320343b358 Parents: f41f433 Author: Narine Kokhlikyan <narine.kokhlik...@gmail.com> Authored: Fri Jun 10 17:17:47 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Fri Jun 10 17:17:57 2016 -0700 ---------------------------------------------------------------------- .../org/apache/spark/sql/catalyst/plans/logical/object.scala | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/0a450cff/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala ---------------------------------------------------------------------- diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala index 55d8adf..78e8822 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala @@ -141,6 +141,9 @@ case class MapPartitionsInR( outputObjAttr: Attribute, child: LogicalPlan) extends ObjectConsumer with ObjectProducer { override lazy val schema = outputSchema + + override protected def stringArgs: Iterator[Any] = Iterator(inputSchema, outputSchema, + outputObjAttr, child) } object MapElements { --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
