Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3527
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65033421
Merging in master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65033393
Yea as @aarondav pointed out, I don't think akka framesize is going to be a
problem anymore in 1.2+, regardless of the number of partitions. Still good to
have this check to
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65028820
[Test build #23974 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23974/consoleFull)
for PR 3527 at commit
[`0089c7a`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65028826
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65028666
> I believe it is only 1 bit, not byte, per block
Thank you for correcting me. Was not aware of `HighlyCompressedMapStatus`.
---
If your project is set up for it
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65028176
I believe it is only 1 bit, not byte, per block. Further I would estimate
compression on largely uniform data to be at least around 10x. So your
example would idea
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65027426
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65027423
[Test build #23973 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23973/consoleFull)
for PR 3527 at commit
[`0089c7a`](https://gith
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65027150
1 partitions doesn't sound that extreme to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your p
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65026993
> @zsxwing Note that the case you mentioned should no longer cause this
issue either, as we use an extra compressed data structure when dealing with
very large
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65025980
@zsxwing Note that the case you mentioned should no longer cause this issue
either, as we use an extra compressed data structure when dealing with very
large numbers of
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65024260
[Test build #23974 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23974/consoleFull)
for PR 3527 at commit
[`0089c7a`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65023903
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65023301
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65023051
A potential usage of `spark.akka.frameSize` is when the size of
`MapStatus`s exceeds `spark.akka.frameSize`, such as large number of mappers
and reducers.
A rele
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65022875
[Test build #23973 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23973/consoleFull)
for PR 3527 at commit
[`0089c7a`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3527#issuecomment-65022618
Nice catch. I don't think that it's very common to set
`spark.akka.frameSize` these days, since 1.1's task broadcasting should have
addressed the most common causes of
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/3527
[SPARK-4664][Core] Throw an exception when spark.akka.frameSize > 2047
If `spark.akka.frameSize` > 2047, it will overflow and become negative.
Should have some assertion in `maxFrameSizeBytes` to wa
19 matches
Mail list logo
