[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 After collected the opinions so far, doing just document is the consensus. I will close this for now and submit a simple PR to document it later. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 ping @ueshin @BryanCutler @cloud-fan Would you mind to provide some insights? Should we add just a document for it or fix it in your opinions? Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83167/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83167 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83167/testReport)** for PR 19592 at commit [`138a366`](https://github.com/apache/spark/commit/138a366349c36b92a89d0b033baf28aedb380688). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BatchEvalPythonExecBase(` * `case class BatchEvalPythonExec(udfs: Seq[PythonUDF], output: Seq[Attribute], child: SparkPlan)` * `case class BatchOptEvalPythonExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83167 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83167/testReport)** for PR 19592 at commit [`138a366`](https://github.com/apache/spark/commit/138a366349c36b92a89d0b033baf28aedb380688). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83156/testReport)** for PR 19592 at commit [`138a366`](https://github.com/apache/spark/commit/138a366349c36b92a89d0b033baf28aedb380688). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BatchEvalPythonExecBase(` * `case class BatchEvalPythonExec(udfs: Seq[PythonUDF], output: Seq[Attribute], child: SparkPlan)` * `case class BatchOptEvalPythonExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83156/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 @HyukjinKwon Thanks for comment. Yeah, want to know if we have consensus that just to document it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19592 To me, I think I slightly more prefer documenting this limitation, given complexity vs gain for now. But want to know what others think. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83156/testReport)** for PR 19592 at commit [`138a366`](https://github.com/apache/spark/commit/138a366349c36b92a89d0b033baf28aedb380688). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83150/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83150 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83150/testReport)** for PR 19592 at commit [`3a5c4c8`](https://github.com/apache/spark/commit/3a5c4c8c8eb874532b3c203e0b290928495fa4a6). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BatchEvalPythonExecBase(` * `case class BatchEvalPythonExec(udfs: Seq[PythonUDF], output: Seq[Attribute], child: SparkPlan)` * `case class BatchOptEvalPythonExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83152/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83152/testReport)** for PR 19592 at commit [`138a366`](https://github.com/apache/spark/commit/138a366349c36b92a89d0b033baf28aedb380688). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BatchEvalPythonExecBase(` * `case class BatchEvalPythonExec(udfs: Seq[PythonUDF], output: Seq[Attribute], child: SparkPlan)` * `case class BatchOptEvalPythonExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83152 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83152/testReport)** for PR 19592 at commit [`138a366`](https://github.com/apache/spark/commit/138a366349c36b92a89d0b033baf28aedb380688). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 A large part of the change is refactoring. IMHO, if possibly, it is better to allow Python UDFs running with conditional expressions normally. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83150 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83150/testReport)** for PR 19592 at commit [`3a5c4c8`](https://github.com/apache/spark/commit/3a5c4c8c8eb874532b3c203e0b290928495fa4a6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 One question is this behavior isn't so much intuitive for end users without knowledge of Python UDFs internals. I'm just not sure if making it as it is is a best option, even with document for it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 Yeah, it is also an option. It is relatively easy to incorporate the conditional logic into Python UDFs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19592 Is this complexity worth it? Can we just document it as a behavior and users need to be careful with it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 Seems it fails on python3.4, let me check it locally. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83141/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83141/testReport)** for PR 19592 at commit [`9744e77`](https://github.com/apache/spark/commit/9744e77178363a813093dfe0dbe131a7d9f27176). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BatchEvalPythonExecBase(` * `case class BatchEvalPythonExec(udfs: Seq[PythonUDF], output: Seq[Attribute], child: SparkPlan)` * `case class BatchOptEvalPythonExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83140/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83140/testReport)** for PR 19592 at commit [`1c523d1`](https://github.com/apache/spark/commit/1c523d1fe0741376a714d5b3e96553159a1b1ce9). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BatchEvalPythonExecBase(` * `case class BatchEvalPythonExec(udfs: Seq[PythonUDF], output: Seq[Attribute], child: SparkPlan)` * `case class BatchOptEvalPythonExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83141/testReport)** for PR 19592 at commit [`9744e77`](https://github.com/apache/spark/commit/9744e77178363a813093dfe0dbe131a7d9f27176). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 cc @HyukjinKwon @ueshin @BryanCutler @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83140/testReport)** for PR 19592 at commit [`1c523d1`](https://github.com/apache/spark/commit/1c523d1fe0741376a714d5b3e96553159a1b1ce9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83139/testReport)** for PR 19592 at commit [`0515435`](https://github.com/apache/spark/commit/0515435ec463cd40f69a00be74ff1efddc07bde4). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BatchEvalPythonExecBase(` * `case class BatchEvalPythonExec(udfs: Seq[PythonUDF], output: Seq[Attribute], child: SparkPlan)` * `case class BatchOptEvalPythonExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19592 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83139/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19592 **[Test build #83139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83139/testReport)** for PR 19592 at commit [`0515435`](https://github.com/apache/spark/commit/0515435ec463cd40f69a00be74ff1efddc07bde4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
