Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22511
BTW it can be argued that you don't need a dispose in that code path since
`serializerManager.dataSerializeWithExplicitClassTag` creates an on-heap buffer
that doesn't need to be disposed. But it
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22511
I think the deal with the dispose in TorrentBroadcast is that it's
definitely needed in the local read case, but may need adjustments in the
remote read case.
The local read case
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22511
> but I don't feel confident about making that change for 2.4
Makes sense. cc @vanzin for more context about
Github user squito commented on the issue:
https://github.com/apache/spark/pull/22511
> This PR is directly heading to branch-2.4 by bypassing master branch
YEs good point, sorry I opened this against 2.4 just for testing in case
the errors were more likely in 2.4 for some
Github user squito commented on the issue:
https://github.com/apache/spark/pull/22511
> The analysis makes sense to me. The thing I'm not sure is, how can we hit
it? The "fetch block to temp file" code path is only enabled for big blocks (>
2GB).
The failing tests cases
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22511
a possible approach: can we just not dispose the data in `TorrentBroadcast`?
---
-
To unsubscribe, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22511
The analysis makes sense to me. The thing I'm not sure is, how can we hit
it? The "fetch block to temp file" code path is only enabled for big blocks (>
2GB).
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96526/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96526 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96526/testReport)**
for PR 22511 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22511
@squito This PR is directly heading to `branch-2.4` by bypassing `master`
branch. Is there a reason to bypass `master` branch? If there is no reason, in
order to prevent future regression at
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22511
LGTM. I went back and took a look at the related changes, and agree with
Imran that this is basically the same thing that 2.3 did; so no perf
regression, just higher memory usage than in the mmap
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96526 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96526/testReport)**
for PR 22511 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22511
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96512/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96512 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96512/testReport)**
for PR 22511 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96512 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96512/testReport)**
for PR 22511 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22511
Retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96497/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96497 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96497/testReport)**
for PR 22511 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22511
Also cc @zsxwing @JoshRosen
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96497 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96497/testReport)**
for PR 22511 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22511
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #4346 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4346/testReport)**
for PR 22511 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/22511
> this seems like a big change, will we hit perf regression?
Not vs. 2.3. It only effects things when stream-to-disk is enabled, and
when it is enabled, for reading remote cached blocks,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96408/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96408 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96408/testReport)**
for PR 22511 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22511
is this a long-standing bug?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22511
this seems like a big change, will we hit perf regression?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96408 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96408/testReport)**
for PR 22511 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22511
cc @jiangxb1987
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22511
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96394/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96394 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96394/testReport)**
for PR 22511 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22511
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22511
**[Test build #96394 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96394/testReport)**
for PR 22511 at commit
49 matches
Mail list logo
