Github user zenglian commented on the issue:
https://github.com/apache/spark/pull/20460
I suggest to introduce a new property `spark.kubernetes.coreUnit = cpu |
mcpu` (cpu by default), which would affect k8s only.
---
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
Agreed. This is a fundamental change to the way Spark handles task
scheduling, task parallelism, and dynamic resource allocation, etc., and it
impacts every scheduler backends. I'm closing this
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20460
I would suggest to bring out a discussion or even a design on dev mail list
before doing such ground changing. This may affect not only dynamic allocation,
but also scheduler. It is better to
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
Just realized that the update I did still won't work when dynamic resource
allocation is enabled. So please ignore the update. This is definitely much
more impactful than I thought.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86998 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86998/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86998/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/535/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86998 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86998/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86995/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86995 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86995/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/532/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86995 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86995/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86993/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86993 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86993/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/530/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86993 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86993/testReport)**
for PR 20460 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/527/
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86992/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86992 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86992/testReport)**
for PR 20460 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/527/
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/529/
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
@jiangxb1987 @felixcheung @jerryshao this PR has been updated to allow both
`spark.executor.cores` and `spark.task.cpus` to have fractional values. Please
take a look at the updated PR
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86992 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86992/testReport)**
for PR 20460 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20460
ah, sounds like this is more impactful than we thought
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
Agreed. It seems much more complicated than expected, considering the
interaction between `spark.executor.cores` and `spark.task.cpus` and the
implication for each scheduler backend. Will do a
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20460
This may require more changes than it appears to be, agree we may need more
tests to ensure it don't break anything, so +1 on target it to 2.4 instead of
2.3
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86949/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20460
> I'd target this 2.3 & master. Waiting for tests
@felixcheung is it too risky to target to 2.3, this is a fundamental
behavior change. We should make sure k8s could well use fractional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86949 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86949/testReport)**
for PR 20460 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20460
I think here
(https://github.com/apache/spark/blob/032c11b83f0d276bf8085992229b8c598f02798a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L117)
should also be fixed.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/488/
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/488/
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/490/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86949 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86949/testReport)**
for PR 20460 at commit
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
@jiangxb1987 fixed the check in
https://github.com/apache/spark/pull/20460/commits/44e489eca6c22254fa57ff6f4750e3af85c43e34.
---
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20460
Yea, we should either move the check to elsewhere or modify it, to avoid
potential failure.
---
-
To unsubscribe, e-mail:
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/20460
@jiangxb1987 it seems `spark.cores.max` only applies to standalone and
Mesos coarse-grained modes. In k8s mode, `spark.cores.max` is not applicable. I
think the check really should not be here,
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20460
Actually I think this may fail some check (though may not throw exceptions)
for instance this one:
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20460
Do we also want to update the comment of `SPARK_EXECUTOR_CORES` in
`spark-env.sh` ?
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #4088 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4088/testReport)**
for PR 20460 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #4088 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4088/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86915/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86915 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86915/testReport)**
for PR 20460 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20460
I'd target this 2.3 & master. Waiting for tests
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/462/
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/462/
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/464/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86915 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86915/testReport)**
for PR 20460 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20460
Jenkins, retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86895/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86895 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86895/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86890/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86890 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86890/testReport)**
for PR 20460 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/446/
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/446/
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/448/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86895 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86895/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86894/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86894 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86894/testReport)**
for PR 20460 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/447/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86894 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86894/testReport)**
for PR 20460 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test status success
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/442/
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
Kubernetes integration test starting
URL:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/442/
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/443/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20460
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20460
**[Test build #86890 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86890/testReport)**
for PR 20460 at commit
83 matches
Mail list logo
