Github user YanTangZhai commented on the pull request:
https://github.com/apache/spark/pull/3137#issuecomment-67452153
@marmbrus Thanks. I'm also trying another approach to optimize this
operation. I want to discuss it with you later.
---
If your project is set up for it, you can rep
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3137
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3137#issuecomment-67377695
Thanks for working on this! Looks like there is still some discussion on
the correct approach here. To keep the PR queue small, I propose we close this
issue and revis
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/3137#issuecomment-62260611
Probably a better way is providing the Sort-Merge-Aggregation not just for
`DistinctCount`. I am working on a POC, hope we can discuss that soon.
---
If your proj
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/3137#issuecomment-62020596
Do you have a test for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have th
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3137#issuecomment-61996365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3137#issuecomment-61996358
[Test build #23003 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23003/consoleFull)
for PR 3137 at commit
[`eecb499`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3137#issuecomment-61980195
[Test build #23003 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23003/consoleFull)
for PR 3137 at commit
[`eecb499`](https://githu
GitHub user YanTangZhai opened a pull request:
https://github.com/apache/spark/pull/3137
[WIP] [SPARK-4273] [SQL] Providing ExternalSet to avoid OOM when
count(distinct)
Some task may OOM when count(distinct) if it needs to process many records.
CombineSetsAndCountFunction puts all
