Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15979
cc @yhuai @liancheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69000 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69000/consoleFull)**
for PR 15979 at commit
[`cb834b8`](https://github.com/apache/spark/commit/c
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69000 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69000/consoleFull)**
for PR 15979 at commit
[`cb834b8`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69000/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15979
What does "non-flat type" mean?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and w
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15979
"non-flat type" means "complex type", i.e. array, seq, map, product, etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15979
retest it please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, o
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/15979
looks good. @liancheng want to double check?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69326 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69326/consoleFull)**
for PR 15979 at commit
[`01b072d`](https://github.com/apache/spark/commit/0
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69326 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69326/consoleFull)**
for PR 15979 at commit
[`01b072d`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69326/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15979
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69328 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69328/consoleFull)**
for PR 15979 at commit
[`01b072d`](https://github.com/apache/spark/commit/0
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69328 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69328/consoleFull)**
for PR 15979 at commit
[`01b072d`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69328/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/15979
LGTM, merging to master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/15979
My only concern is that "non-flat" type is neither intuitive nor a
well-known term. In fact, this PR only prevents `Option[T <: Product]` to be
top-level Dataset types. How about just call them "P
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15979
FWIW I don't think we should call it nonflat.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69387 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69387/consoleFull)**
for PR 15979 at commit
[`70dd650`](https://github.com/apache/spark/commit/7
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69387/
Test FAILed.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15979
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69394 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69394/consoleFull)**
for PR 15979 at commit
[`70dd650`](https://github.com/apache/spark/commit/7
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/15979
Good to merge pending Jenkins. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
en
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69394 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69394/consoleFull)**
for PR 15979 at commit
[`70dd650`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69394/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15979
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69403 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69403/consoleFull)**
for PR 15979 at commit
[`70dd650`](https://github.com/apache/spark/commit/7
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69403 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69403/consoleFull)**
for PR 15979 at commit
[`70dd650`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69403/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69411 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69411/consoleFull)**
for PR 15979 at commit
[`876e5c7`](https://github.com/apache/spark/commit/8
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15979
**[Test build #69411 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69411/consoleFull)**
for PR 15979 at commit
[`876e5c7`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69411/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15979
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/15979
Merging to master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wi
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/15979
@rxin Shall we backport this to branch-2.1? I think it's relatively safe.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15979
Sounds good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/15979
Also backported to branch-2.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled an
Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/15979
this means anything that uses an encoder can no longer use Option[_ <:
Product].
encoders are not just used for the top level Dataset creation.
Dataset.groupByKey[K] requires an enc
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15979
```
val x: Dataset[String, Option[(String, String)]] = ...
x.groupByKey(_._1).mapValues(_._2).agg(someAgg)
```
Does it work before?
Please see the discussion in the JIRA:
h
Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/15979
Yes it worked before
On Dec 4, 2016 02:33, "Wenchen Fan" wrote:
> val x: Dataset[String, Option[(String, String)]] = ...
> x.groupByKey(_._1).mapValues(_._2).agg(someAgg)
Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/15979
spark 2.0.x does not have mapValues. but this works:
scala> Seq(("a", Some((1, 1))), ("a",
None)).toDS.groupByKey(_._2).count.show
+---++
|key|count(1
Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/15979
admittedly the result looks weird. it really should be:
+---++
|key|count(1)|
+---++
| null| 1|
| [1,1]| 1|
48 matches
Mail list logo