Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84359/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #84359 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84359/testReport)**
for PR 18906 at commit
[`90e1684`](https://github.com/apache/spark/commit/9
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #84359 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84359/testReport)**
for PR 18906 at commit
[`90e1684`](https://github.com/apache/spark/commit/90
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/18906
Thanks for the background Bryan :) So it sounds like from an API
perspective it makes sense to support this in the future possibly on the Pandas
UDFs (but the code isn't there on the JVM side). I'd
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18906
I believe the equivalent API in Scala would only be in the following form
when registering a UDF
```
spark.udf.register("func", () => { 1 }).asNonNullable()
```
Would it be prefer
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/18906
So I think with the performance improvements coming into Python UDFs
considering annotating results as nullable or not could make sense (although I
imagine we'd need to do something differeent for t
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83846/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83846 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83846/testReport)**
for PR 18906 at commit
[`9856be6`](https://github.com/apache/spark/commit/9
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18906
I meant the actual equivalent endpoints and actual codes with usecases.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spa
Github user ptkool commented on the issue:
https://github.com/apache/spark/pull/18906
Here are the similar changes in the Scala API:
https://github.com/apache/spark/pull/17911
---
-
To unsubscribe, e-mail: reviews-u
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18906
> Satisfying a use case that I described in my initial comments.
I am actually not quite clear of the usecases. Providing actual codes and
elaborating it should be helpful.
> C
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83846 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83846/testReport)**
for PR 18906 at commit
[`9856be6`](https://github.com/apache/spark/commit/98
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/18906
CC @HyukjinKwon
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83814/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83814 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83814/testReport)**
for PR 18906 at commit
[`feefb3e`](https://github.com/apache/spark/commit/f
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83814 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83814/testReport)**
for PR 18906 at commit
[`feefb3e`](https://github.com/apache/spark/commit/fe
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83798/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83798 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83798/testReport)**
for PR 18906 at commit
[`402a814`](https://github.com/apache/spark/commit/4
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83798 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83798/testReport)**
for PR 18906 at commit
[`402a814`](https://github.com/apache/spark/commit/40
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83796 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83796/testReport)**
for PR 18906 at commit
[`0bee999`](https://github.com/apache/spark/commit/0
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83796/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83796 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83796/testReport)**
for PR 18906 at commit
[`0bee999`](https://github.com/apache/spark/commit/0b
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83786 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83786/testReport)**
for PR 18906 at commit
[`38dc32d`](https://github.com/apache/spark/commit/3
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83786/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83786 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83786/testReport)**
for PR 18906 at commit
[`38dc32d`](https://github.com/apache/spark/commit/38
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83784/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83752/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83752 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83752/testReport)**
for PR 18906 at commit
[`38dc32d`](https://github.com/apache/spark/commit/3
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18906
**[Test build #83752 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83752/testReport)**
for PR 18906 at commit
[`38dc32d`](https://github.com/apache/spark/commit/38
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/18906
Jenkins, ok to test.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18906
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ptkool commented on the issue:
https://github.com/apache/spark/pull/18906
@rxin
This PR isn't about performance at all. I realize Python UDFs do not
perform well and I also realize annotating Python UDFs with nullability is not
going to make any difference perfor
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18906
I understand why you are using Python. What I don't understand is why you'd
need to annotate nullability, because those are typically annotated for the
purpose of performance improvement, but Python UD
Github user ptkool commented on the issue:
https://github.com/apache/spark/pull/18906
@rxin We have several large systems with 100s of Spark jobs implemented in
Python and PySpark, and use Python UDFs due to lack of equivalent functionality
in Spark. I understand what your saying re P
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18906
@ptkool have you seen a real use case so far that you need this? I'm a bit
surprised since Python UDFs are already pretty slow, and you'd care about this.
Are there other cases you run into?
O
Github user ptkool commented on the issue:
https://github.com/apache/spark/pull/18906
@ueshin Thanks for commenting.
It's unfortunate that users find nullability confusing. If you're coming
from a SQL world, you should be quite familiar with nullability and null
values. Never
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18906
@ptkool Thank you for working on this!
I'd like to ask what your use-case is. Users have historically been
confused about what nullable means, and we don't think we should give them yet
another a
101 - 145 of 145 matches
Mail list logo
