spark git commit: [SPARK-15811][SQL] fix the Python UDF in Scala 2.10

[rxin]

Repository: spark
Updated Branches:
  refs/heads/branch-2.0 b82abde06 -> 3c3865d0b


[SPARK-15811][SQL] fix the Python UDF in Scala 2.10

## What changes were proposed in this pull request?

Iterator can't be serialized in Scala 2.10, we should force it into a array to 
make sure that .

## How was this patch tested?

Build with Scala 2.10 and ran all the Python unit tests manually (will be 
covered by a jenkins build).

Author: Davies Liu 

Closes #13717 from davies/fix_udf_210.

(cherry picked from commit ef43b4ed87894982678fcc6f2c61cf1487ee9e14)
Signed-off-by: Reynold Xin 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3c3865d0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3c3865d0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3c3865d0

Branch: refs/heads/branch-2.0
Commit: 3c3865d0b42413435f8cc9044da842cfb4458afc
Parents: b82abde
Author: Davies Liu 
Authored: Fri Jun 17 00:34:33 2016 -0700
Committer: Reynold Xin 
Committed: Fri Jun 17 00:34:38 2016 -0700

--
 .../org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/3c3865d0/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
--
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
index 668470e..87583c8 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
@@ -132,7 +132,7 @@ private[spark] object ExtractPythonUDFs extends 
Rule[SparkPlan] {
 val validUdfs = udfs.filter { case udf =>
   // Check to make sure that the UDF can be evaluated with only the 
input of this child.
   udf.references.subsetOf(child.outputSet)
-}
+}.toArray  // Turn it into an array since iterators cannot be 
serialized in Scala 2.10
 if (validUdfs.nonEmpty) {
   val resultAttrs = udfs.zipWithIndex.map { case (u, i) =>
 AttributeReference(s"pythonUDF$i", u.dataType)()


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-15811][SQL] fix the Python UDF in Scala 2.10

[rxin]

Repository: spark
Updated Branches:
  refs/heads/master e5d703bca -> ef43b4ed8


[SPARK-15811][SQL] fix the Python UDF in Scala 2.10

## What changes were proposed in this pull request?

Iterator can't be serialized in Scala 2.10, we should force it into a array to 
make sure that .

## How was this patch tested?

Build with Scala 2.10 and ran all the Python unit tests manually (will be 
covered by a jenkins build).

Author: Davies Liu 

Closes #13717 from davies/fix_udf_210.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ef43b4ed
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ef43b4ed
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ef43b4ed

Branch: refs/heads/master
Commit: ef43b4ed87894982678fcc6f2c61cf1487ee9e14
Parents: e5d703b
Author: Davies Liu 
Authored: Fri Jun 17 00:34:33 2016 -0700
Committer: Reynold Xin 
Committed: Fri Jun 17 00:34:33 2016 -0700

--
 .../org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/ef43b4ed/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
--
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
index 668470e..87583c8 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
@@ -132,7 +132,7 @@ private[spark] object ExtractPythonUDFs extends 
Rule[SparkPlan] {
 val validUdfs = udfs.filter { case udf =>
   // Check to make sure that the UDF can be evaluated with only the 
input of this child.
   udf.references.subsetOf(child.outputSet)
-}
+}.toArray  // Turn it into an array since iterators cannot be 
serialized in Scala 2.10
 if (validUdfs.nonEmpty) {
   val resultAttrs = udfs.zipWithIndex.map { case (u, i) =>
 AttributeReference(s"pythonUDF$i", u.dataType)()


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org