Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
Thanks @holdenk
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16533
Thanks for working on this @zero323, merged to master :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72885 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72885/testReport)**
for PR 16533 at commit
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
@holdenk done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72885 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72885/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16533
Great :) So the sql python tests have changed a bit which means this needs
another update (sorry @zero323 ) give me a ping when its updated and I'll try
and come back to this quickly :)
---
If
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16533
Thanks for cc'ing me. I like this pythonic way. +1 and looks okay to me too
despite of the trick in argument checking.
---
If your project is set up for it, you can reply to this email and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72825/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72825 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72825/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72824/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72824 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72824/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72825 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72825/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72823/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72823 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72823/testReport)**
for PR 16533 at commit
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16533
Ok this looks pretty reasonable, but I'm going to give it a day incase
anyone (e.g. @rdblue or @HyukjinKwon ) have any last comments or suggestions.
Thanks so much for working on this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72821/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72821 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72821/testReport)**
for PR 16533 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72824 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72824/testReport)**
for PR 16533 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72823 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72823/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72822/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72822 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72822/testReport)**
for PR 16533 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72822 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72822/testReport)**
for PR 16533 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72821 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72821/testReport)**
for PR 16533 at commit
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16533
Great, I've merged that PR, can you update the instance checks here as
required and add a test using the decorator with a string?
---
If your project is set up for it, you can reply to this email
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
@holdenk We cannot support it without modifying `UserDefinedFunction`
first so I decided to open a separate PR for this:
https://github.com/apache/spark/pull/16769
---
If your project is set up
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16533
So one minor thing, I don't see support for taking the return type as
string - was that decided to be dropped?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16533
Ok back from the summit, let me take a look through - sorry for the
slowness with my travel.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72331/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72331 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72331/testReport)**
for PR 16533 at commit
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
@holdenk If believe this is a good as it gets at this moment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72331 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72331/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72328/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72328 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72328/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72328 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72328/testReport)**
for PR 16533 at commit
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
Handling data type strings proved to be much simpler than I thought (I
missed that PySpark already has its own parser):
[SPARK-19427](https://issues.apache.org/jira/browse/SPARK-19427) and #16769
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72241 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72241/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72241/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72241 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72241/testReport)**
for PR 16533 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/16533
I agree with @zero323. I don't think this method should change how users
interact with DataTypes. There is a precedent for using type strings instead,
as @rxin suggested, and that's a good idea that
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
@lins05 I am not convinced. This has fairly limited applications (we
cannot use this to define complex types) and could be confusing for users (why
we can use it here but not for `Column.cast`,
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
@rxin
We can do something like this:
```python
class UserDefinedFunction(object):
"""
User defined function in Python
.. versionadded:: 1.3
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/16533
What about also support using the type name without the parentheses, as a
syntax sugar? e.g.:
```python
@udf(returnType =IntegerType) # instead of IntegerType()
def f(...):
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
@rxin Sure. Give me a second. In a meantime would consider reviewing #16742
([SPARK-19403](https://issues.apache.org/jira/browse/SPARK-19403))?
---
If your project is set up for it, you can reply
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/16533
Can return type also take a string?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72074/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72074 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72074/testReport)**
for PR 16533 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72074 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72074/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72073/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72073 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72073/testReport)**
for PR 16533 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72073 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72073/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72072 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72072/testReport)**
for PR 16533 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72072/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #72072 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72072/testReport)**
for PR 16533 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/16533
+1
Looks good to me. Thanks, @zero323!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16533
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71721/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #71721 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71721/testReport)**
for PR 16533 at commit
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
@rdblue You're probably right and I am overthinking this. After all the
goal here is to make this as user friendly as possible, and while `isinstance`
is not pretty it does the trick. If we ever
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16533
**[Test build #71721 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71721/testReport)**
for PR 16533 at commit
73 matches
Mail list logo
