Github user davies commented on the issue:
https://github.com/apache/spark/pull/13778
LGTM, merging this into master and 2.0 branch, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @cloud-fan again, this is waiting for a while. Do you have time to
look at again? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @cloud-fan Do you miss this? Or you have other concern? Please let me
know. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @cloud-fan What do you think about this? Can we merge it now? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @cloud-fan Can you check if this is good for you now? It is for a
while. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @liancheng @yhuai Maybe you can review this too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @cloud-fan Can you review this? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @cloud-fan Please see if this is ok for you now. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62151/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #62151 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62151/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #62151 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62151/consoleFull)**
for PR 13778 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
@cloud-fan Updated. Please take a look. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
@cloud-fan I just checked the python UDT. In python side, we will serialize
the python UDT to binary. The python UDT passed to java includes the binary.
Then in python side, in the worker we will
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13778
yea, so whatever the data type is(python udt or normal sql type), at java
side there is no difference, the data is converted to corrected format by
pickler. That's why I think maybe it's possible
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
Oh, I mean they should be serialized/deserialized by pickler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13778
Can you point out where we catch `PythonUserDefinedType` and do special
serialization at java side? It looks to me that we just get its corresponding
sql type.
---
If your project is set up for
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @cloud-fan any more concern?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
Python UDT in python side only serializes the python data to sql type
defined in the Python UDT. The problem now is happened at the serialization to
row in java side on the serialized python data. I
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13778
From another point of view, is it necessary to propagate the python UDF
from python side to jvm side? IIUC the serialization of python UDT happens at
python side, and the jvm side can only see
Github user vlad17 commented on the issue:
https://github.com/apache/spark/pull/13778
LGTM +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @cloud-fan @vlad17 Any thing else?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61903/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61903 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61903/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61903 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61903/consoleFull)**
for PR 13778 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61836/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61836 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61836/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61836 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61836/consoleFull)**
for PR 13778 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61441/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61441 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61441/consoleFull)**
for PR 13778 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
@cloud-fan @vlad17 Is this change good for your now? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61435/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61435 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61435/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61441 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61441/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61435 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61435/consoleFull)**
for PR 13778 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61383/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61383 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61383/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61383 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61383/consoleFull)**
for PR 13778 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
also cc @yhuai @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
ping @vlad17 @davies @liancheng Any thing else?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61000/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61000 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61000/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #61000 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61000/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #60996 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60996/consoleFull)**
for PR 13778 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60996/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #60996 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60996/consoleFull)**
for PR 13778 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60916/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #60916 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60916/consoleFull)**
for PR 13778 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
@mengxr Although UDTs are private APIs, but as you see from the example,
the users can define user classes and corresponding UDTs in Python that will be
PythonUserDefinedType. The issues in this PR
Github user mengxr commented on the issue:
https://github.com/apache/spark/pull/13778
@viirya Do we need to fix this in Spark 2.0? UDTs are private APIs and the
only intended use case is Vector/Matrix UDTs for MLlib, which doesn't put
vectors or matrices inside an array inside a
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #60916 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60916/consoleFull)**
for PR 13778 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
@vlad17 Thanks! I will look into that issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user vlad17 commented on the issue:
https://github.com/apache/spark/pull/13778
Another update:
https://gist.github.com/vlad17/cfcd42f30ea2380df4fb0bfa30dda7ce unresolved
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user vlad17 commented on the issue:
https://github.com/apache/spark/pull/13778
Update: looks like the above is just an issue with the __str__ method of
udf-returned UDTs, which is a different bug (a bug that's also pretty harmless).
---
If your project is set up for it, you
Github user vlad17 commented on the issue:
https://github.com/apache/spark/pull/13778
Here's an unresolved example:
https://gist.github.com/vlad17/2db8e14972344c693e8a3f03d91c9c8d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60837/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #60837 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60837/consoleFull)**
for PR 13778 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/13778
cc @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60835/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13778
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #60835 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60835/consoleFull)**
for PR 13778 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13778
**[Test build #60837 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60837/consoleFull)**
for PR 13778 at commit
71 matches
Mail list logo
