Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/21650
Thanks @HyukjinKwon @BryanCutler for the review!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21650
LGTM.
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93686/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93686 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93686/testReport)**
for PR 21650 at commit
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/21650
retest please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93688/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93688 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93688/testReport)**
for PR 21650 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93688 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93688/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93686 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93686/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93667/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93667 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93667/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93668/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93668 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93668/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93668 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93668/testReport)**
for PR 21650 at commit
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/21650
@BryanCutler @HyukjinKwon I updated the PR based on Bryan's suggestion.
Please take a look and let me know if you have further comments.
Thanks!
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93667 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93667/testReport)**
for PR 21650 at commit
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/21650
@HyukjinKwon I think Bryan's imple looks promising. Please let me take a
look.
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21650
Hm, then how about giving a try in a followup @BryanCutler if you see some
values on it?
---
-
To unsubscribe, e-mail:
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21650
>ehh .. @BryanCutler, WDYT about just doing the previous one for now? The
approach you suggested sounds efficient of course but.. here's not a hot path
so I think the previous way is fine too
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21650
ehh .. @BryanCutler, WDYT about just doing the previous one for now? The
approach you suggested sounds efficient of course but.. here's not a hot path
so I think the previous way is fine too ..
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93546/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93546 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93546/testReport)**
for PR 21650 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93546 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93546/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21650
I'm okay with
https://github.com/apache/spark/pull/21650#issuecomment-407506043's way too but
should be really simplified. Either way LGTM.
---
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/21650
@BryanCutler Thanks for taking a look at this! Yeah I think this works too.
Let me update the code and try it. Thanks again!
---
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21650
I gave it a shot to extract the UDFs in one traversal, using the first
occurrence of either pandas or batch udf. I think it's much clearer
```scala
object ExtractPythonUDFs extends
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/21650
@BryanCutler I've address most of you comments and explained the ones that
I didn't change. Do you mind take another look? Thanks!
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93450/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93450 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93450/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93451/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93451 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93451/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93451 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93451/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #93450 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93450/testReport)**
for PR 21650 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21650
ping @BryanCutler Any update about this PR?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21650
I think the previous behavior was to not allow mixing pandas and regular
udfs, but you're probably right that there are some cases where data could be
handled differently. I'll try to look at
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/21650
@BryanCutler I think your suggestion would change the behavior. Using
ArrowEvalExec and BatchEvalExec are still different when it comes to corner
cases, for example, type coercion (ArrowEvalExec
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21650
I had an idea of a slightly different approach.. Would it be possible to
"promote" the regular `udf` to a `pandas_udf`? By this I mean wrap the
function using `apply()` so that it takes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92482/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #92482 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92482/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/589/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #92482 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92482/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92443/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #92443 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92443/testReport)**
for PR 21650 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/557/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21650
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21650
**[Test build #92443 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92443/testReport)**
for PR 21650 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21650
Would you mind changing cast (1) in your description? It threw me off a
little as they looked independent at first glance. Maybe something like:
```
df = spark.range(0, 1).toDF('v') \
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/21650
@viirya I have added the query plan output. @maropu I updated the PR title.
Thanks!
---
-
To unsubscribe, e-mail:
72 matches
Mail list logo
