Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21390
Thanks @JoshRosen for very detailed and thoughtful reply.
Agreed TTL could be fragile, but I was very concern with what point
> There is a related issue where shuffle files can be
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/21390
Feel free to do the TTL in a followup. My feeling is that it won't be super
useful in practice, though:
1. Cleanup of non-shuffle disk block manager files following executor exit
only
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21390
sounds like we need to have some sort of lifetime management, TTL -like
design shuffle file - should we have a new JIRA on that?
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21390
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21390
Are there any other concerns over this PR?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/21390
Yeah, this is only concerned with non-shuffle files which are located in
the block manager temp directories (e.g. large sorter spill files).
There is a related issue where shuffle files
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21390
Actually since this specifically applies to _non_-shuffle files I think
Kubernetes will be fine here regardless.
---
-
To
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21390
For Kubernetes, without a story around the external shuffle service, all
the scratch space used by executors will be cleaned up by Kubernetes itself.
When we want shuffle data to persist across
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91128/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91128 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91128/testReport)**
for PR 21390 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3565/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21390
@jerryshao Agree it should be useful to add a `debug-delay-sec` config for
ease of developing, since this PR has already bring in a brunch of code
changes, maybe we can add the config in a
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91128 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91128/testReport)**
for PR 21390 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21390
YARN will clean container local dirs when container (executor) is exited,
so this may not be a problem in YARN.
YARN has a useful configuration "yarn.nodemanager.delete.debug-delay-sec"
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91083/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91083 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91083/testReport)**
for PR 21390 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3536/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91083 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91083/testReport)**
for PR 21390 at commit
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21390
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91078/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91078 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91078/testReport)**
for PR 21390 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3533/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91078 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91078/testReport)**
for PR 21390 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91064/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91064 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91064/testReport)**
for PR 21390 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3521/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91064 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91064/testReport)**
for PR 21390 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21390
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91006/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91006 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91006/testReport)**
for PR 21390 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3483/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #91006 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91006/testReport)**
for PR 21390 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90942/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #90942 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90942/testReport)**
for PR 21390 at commit
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/21390
Context for other reviewers: the issue addressed by this patch is actually
a real issue in practice, especially for long-lived Spark clusters; I have seen
this specific problem play a large
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21390
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3448/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21390
**[Test build #90942 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90942/testReport)**
for PR 21390 at commit
49 matches
Mail list logo
