Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
@cloud-fan Sounds good. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18732
Let's discuss more on the new PR. At least we should create different UDF
types in the implementation, the user-facing API can remain `@pandas_udf`.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18732
> Use different function name for different input/output type
Yea it's a bad idea as there are many combinations, and I just wanna use
different APIs for different scenarios, e,g,
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
@cloud-fan Thanks for your feedback.
I think it makes sense to define `pandas_udaf` as it's own function because
it is a multi-step udf and is very different from the existing
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18732
@icexelloss I think as an API, it's a little confusing that `@pandas_udf`
can define both `Series* -> Series` function and `DataFrame -> DataFrame`
function. Besides, to support `StructType` as
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
I am still not crazy about introducing a `pandas_grouped_udf` unless there
is strong reason to. @ueshin do you think this is just an issue of returnType
or is there some other reason?
---
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18732
I submitted a pr #19505 to introduce `@pandas_grouped_udf` instead of
reusing `@pandas_udf`.
---
-
To unsubscribe, e-mail:
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18732
Grouped UDFs, or Grouped Vectorized UDFs.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18732
How to name it? Group-By vectorized UDFs?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18732
I'm +0 for now.
I'm just wondering whether we can support struct types in vectorized UDF
when needed in the future.
As for adding pandas UDAF, I think we need another decorator or
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18732
@ueshin is working on pandas UDAF, let's wait for his feedback.
---
-
To unsubscribe, e-mail:
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
@cloud-fan, it's a good question, I thought quite a bit about it and
discussed with @viirya
-https://github.com/apache/spark/pull/18732#pullrequestreview-66106082
Just to recap, I think
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18732
I think @viirya raised this question too -
https://github.com/apache/spark/pull/18732#issuecomment-333065737 and I think I
also left few worries about thus here and there. To me, +0.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18732
A late question: shall we create another API for it instead of reusing
`pandas_udf`? cc @ueshin
---
-
To unsubscribe, e-mail:
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
@HyukjinKwon Thanks!
Thanks for everyone for reviewing this tirelessly.
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18732
Nice work ð
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18732
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82599/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82599 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82599/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82599 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82599/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82587/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82587 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82587/testReport)**
for PR 18732 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18732
I had some minor comments on the docs, otherwise LGTM!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82585/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82585 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82585/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82584/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82584 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82584/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82587 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82587/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82585 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82585/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82584 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82584/testReport)**
for PR 18732 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18732
add to whitelist
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18732
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82577/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82577 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82577/testReport)**
for PR 18732 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18732
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82576/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82576 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82576/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82577 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82577/testReport)**
for PR 18732 at commit
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
Merged some last minute changes from @BryanCutler to make the wrapping a
bit cleaner. Thanks @BryanCutler!
---
-
To
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82576 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82576/testReport)**
for PR 18732 at commit
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18732
I'm OK with the naming. We can change them later if needed before the
release.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82553/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82553 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82553/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82553 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82553/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82515/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82515 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82515/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82515 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82515/testReport)**
for PR 18732 at commit
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
Hi All, I think all comments should be addressed at this point, except for
the naming comment from @rxin.
If I missed something or if there is anything else you want me to address,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82501/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82501 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82501/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82501 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82501/testReport)**
for PR 18732 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18732
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82493/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82493 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82493/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82493 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82493/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82490/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82490 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82490/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82490 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82490/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82489 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82489/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82489/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82489 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82489/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82477/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82477 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82477/testReport)**
for PR 18732 at commit
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
@HyukjinKwon Thanks for the summarry!
* https://github.com/apache/spark/pull/18732#discussion_r142735696
`ArrowPandasSerialzer`I will spend some time address this today.
*
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18732
Ongoing discussions that (I think) might block this PR:
- https://github.com/apache/spark/pull/18732#discussion_r142735696 by
@BryanCutler: `ArrowPandasSerializer` able to serialize
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82477 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82477/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82469/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82469 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82469/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82466/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82466 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82466/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82463/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82463 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82463/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82469 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82469/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82466 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82466/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82463 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82463/testReport)**
for PR 18732 at commit
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18732
I pushed a new commit addressing the comments. Let me scan through the
comments again. I think there are some comments around worker.py not being
addressed yet.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82457/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82457 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82457/testReport)**
for PR 18732 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82457 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82457/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82440/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18732
**[Test build #82440 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82440/testReport)**
for PR 18732 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82438/
Test FAILed.
---
1 - 100 of 140 matches
Mail list logo
