Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21977#discussion_r210274748
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/BatchEvalPythonExec.scala
---
@@ -69,7 +67,7 @@ case class BatchEvalPythonExec(udfs: Seq[PythonUDF],
output: Seq[Attribute], chi
// Output iterator for results from Python.
val outputIterator = new PythonUDFRunner(
- funcs, bufferSize, reuseWorker, PythonEvalType.SQL_BATCHED_UDF,
argOffsets)
+ funcs, PythonEvalType.SQL_BATCHED_UDF, argOffsets,
sparkContext.conf)
--- End diff --
I _think_ this code is executed on the workers, which means you can't
access the SparkContext here which could explain the local test failures.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
