Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142988703
Yes, it's not related. What you mention here is a bigger problem. The
current solution only ensures that we don't starve any operators. In the future
we can improve t
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142811363
Thank you @andrewor14 for the explanation. I believe you're talking about
to reserve the data page in advanced via the
`UnsafeFixedWidthAggregationMap.map` (Bytes
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142810091
[Test build #1809 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1809/console)
for PR at commit
[`2dc34c3`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142809622
[Test build #1810 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1810/console)
for PR at commit
[`2dc34c3`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142806924
[Test build #1811 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1811/console)
for PR at commit
[`2dc34c3`](https://github
Github user andrewor14 closed the pull request at:
https://github.com/apache/spark/pull/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142786491
[Test build #1808 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1808/console)
for PR at commit
[`2dc34c3`](https://github
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142786398
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142786420
OK it passed tests. I'm merging this into master 1.5.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142786397
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142786251
[Test build #42942 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42942/console)
for PR at commit
[`2dc34c3`](https://github.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142786202
@chenghao-intel but we do reserve the page in advance. See
`TungstenAggregate`:
https://github.com/apache/spark/blob/084e4e126211d74a79e8dbd2d0e604dd3c650822/sql/core
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142783827
@andrewor14 that's actually what I mean, if we didn't reserve the memory
when creating the down streaming operator, we probably never get the chance to
acquire th
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142781604
[Test build #1809 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1809/consoleFull)
for PR at commit
[`2dc34c3`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142781478
[Test build #1807 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1807/consoleFull)
for PR at commit
[`2dc34c3`](https://git
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142781447
> it's running core tests because you changed
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
Ah yes never mind.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142781505
[Test build #1811 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1811/consoleFull)
for PR at commit
[`2dc34c3`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142781407
[Test build #1810 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1810/consoleFull)
for PR at commit
[`2dc34c3`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142781271
[Test build #1808 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1808/consoleFull)
for PR at commit
[`2dc34c3`](https://git
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142780755
@chenghao-intel SPARK-10474 is caused by an aggregate falling back to
sort-based aggregation. In this case we don't acquire the page in the
constructor, but we do acq
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142780195
@andrewor14, it's running core tests because you changed
`core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java`
---
If your proje
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142779662
https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java#L138
Seems reserve the data
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142778874
By the way @JoshRosen this seems to run core tests like
`SecurityManagerSuite` even though it touches only SQL stuff. Just FYI.
---
If your project is set up for it,
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142767783
[Test build #1801 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1801/console)
for PR at commit
[`ed36351`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142767433
[Test build #1802 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1802/console)
for PR at commit
[`ed36351`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764726
[Test build #1800 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1800/console)
for PR at commit
[`ed36351`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764634
[Test build #1805 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1805/consoleFull)
for PR at commit
[`ed36351`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764595
[Test build #42942 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42942/consoleFull)
for PR at commit
[`2dc34c3`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764537
[Test build #1799 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1799/console)
for PR at commit
[`ed36351`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764549
[Test build #1806 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1806/consoleFull)
for PR at commit
[`ed36351`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764532
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764520
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764383
[Test build #1804 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1804/consoleFull)
for PR at commit
[`ed36351`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142764288
[Test build #1803 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1803/consoleFull)
for PR at commit
[`ed36351`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142763573
[Test build #42925 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42925/console)
for PR at commit
[`ed36351`](https://github.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142763627
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142763624
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142761176
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142761172
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142761034
[Test build #42924 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42924/console)
for PR at commit
[`c910d0b`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142740235
[Test build #1802 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1802/consoleFull)
for PR at commit
[`ed36351`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142740012
[Test build #1801 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1801/consoleFull)
for PR at commit
[`ed36351`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142739875
[Test build #1800 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1800/consoleFull)
for PR at commit
[`ed36351`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142739759
[Test build #1799 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1799/consoleFull)
for PR at commit
[`ed36351`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142739836
[Test build #42925 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42925/consoleFull)
for PR at commit
[`ed36351`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142738684
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142738489
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142738680
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142738545
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142737927
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled an
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142737065
[Test build #42924 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42924/consoleFull)
for PR at commit
[`c910d0b`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142736193
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142736252
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142734914
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/#issuecomment-142734890
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/
[SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array
(round 2)
This patch reverts most of the changes in a previous fix #8827.
The real cause of the issue is that i
Github user andrewor14 closed the pull request at:
https://github.com/apache/spark/pull/8827
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141631551
OK, I'm merging into master and 1.5. Thanks for the reviews.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141624490
@andrewor14 Since we will refactor all the memory management in 1.6, and
this PR does fix the problem, so LGTM.
---
If your project is set up for it, you can reply to th
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141592494
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141592497
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141592416
[Test build #42684 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42684/console)
for PR 8827 at commit
[`f7dd17f`](https://github.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141571421
also @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featu
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141561012
Looks good. @rxin or @JoshRosen can any of you take a final look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141560514
[Test build #42684 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42684/consoleFull)
for PR 8827 at commit
[`f7dd17f`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141556741
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141556675
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/8827
[SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array
When `TungstenAggregation` hits memory pressure, it switches from
hash-based to sort-based aggregation in-place. However
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/8827#issuecomment-141556218
@yhuai
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enab
69 matches
Mail list logo
