[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user rvesse commented on the issue: https://github.com/apache/spark/pull/22805 @liyinan926 I will rebase and squash appropriately once PR #22820 is merged --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/22805 @rvesse I want to take back my comment on bumping the version of the fabric8 client in the same PR. Based on the discussion so far, it seems more appropriate to do that in a separate PR, like in #22820. Can you revert the changes to bump the version? Sorry about that. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user rvesse commented on the issue: https://github.com/apache/spark/pull/22805 Ran successfully against one of our dev K8S clusters today:  Will continue onto testing against one of our larger production clusters --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user rvesse commented on the issue: https://github.com/apache/spark/pull/22805 @ifilonenko I have done the generalisation today since it was fairly trivial and it actually resolves a number of concerns about the first pass implementation @skonto I have restored the cloud backend as requested, I ended up having two separate cloud backends. One using the original code that simply takes a master URL and another that takes a desired K8S context and auto-configures itself from the K8S config file and the context specified. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22805 @rvesse I use no vm: `sudo -E ./minikube start --vm-driver=none` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user rvesse commented on the issue: https://github.com/apache/spark/pull/22805 @skonto Yep, I plan to do that tomorrow At least for my `minikube` instance I found 4g insufficient and a couple of tests would fail because their pods didn't get scheduled. 8g is probably over allocating, somewhere in-between likely works also --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22805 @rvesse also the following memory limit works for me: `minikube config set memory 4096` so maybe g its too much. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22805 @rvesse this capability was in the old integration tests repo [here](https://github.com/apache-spark-on-k8s/spark-integration/blob/master/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/cloud/CloudTestBackend.scala). In our[ fork](https://github.com/lightbend/spark/tree/spark-k8s-2.4-snapshot/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend) we have merged that part back, can we add that too? I find it really useful to test real platforms like Openshift etc. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user rvesse commented on the issue: https://github.com/apache/spark/pull/22805 @srowen Yes there are a lot of assumptions made by the integration tests that are not documented anywhere and I figured out by digging in the code and POMs. Broadly speaking right now to run integration tests you need the following: - A runnable distribution of Spark built - `minikube` installed, configured with at least 4 CPUs and 8 GB memory and started Probably the next thing I will do tomorrow on this is to improve the documentation. There is a `README.md` in the integration testing module but it omits various details I have encountered working with these recently. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22805 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22805 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22805 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
