Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r183162778
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala
---
@@ -101,17 +109,29 @@ private[spark
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r183162555
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
---
@@ -54,7 +54,7
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r183162517
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
---
@@ -88,15 +94,22
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r183078482
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
---
@@ -88,15 +94,22
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r182567449
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
---
@@ -88,15 +94,22
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r182567225
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBuilder.scala
---
@@ -29,9 +30,11
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r182567081
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
---
@@ -71,7 +77,7
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r182566963
--- Diff:
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
---
@@ -0,0 +1,33 @@
+#
+# Licensed
GitHub user ifilonenko opened a pull request:
https://github.com/apache/spark/pull/21092
[SPARK-23984][K8S][WIP] Initial Python Bindings for PySpark on K8s
## What changes were proposed in this pull request?
Introducing Python Bindings for PySpark.
- [ ] Running
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r175262319
--- Diff:
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh ---
@@ -53,14 +53,10 @@ fi
case "$SPARK_K8
Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/20669
Results from integration testing:
```
Discovery starting.
Discovery completed in 123 milliseconds.
Run starting. Expected test count is: 8
KubernetesSuite:
- Run SparkPi
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174902563
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
---
@@ -110,30 +109,29
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174630216
--- Diff:
examples/src/main/scala/org/apache/spark/examples/SparkRemoteFileTest.scala ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174629033
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
---
@@ -89,26 +56,16 @@ private[spark
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174628986
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
---
@@ -89,26 +56,16 @@ private[spark
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174628963
--- Diff: bin/docker-image-tool.sh ---
@@ -63,9 +63,11 @@ function build {
error "Cannot find docker image. This script must be run
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174583140
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
---
@@ -89,26 +56,16 @@ private[spark
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174582921
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
---
@@ -110,30 +109,29
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174582150
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
---
@@ -33,7 +33,9
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174350660
--- Diff: bin/docker-image-tool.sh ---
@@ -63,9 +63,11 @@ function build {
error "Cannot find docker image. This script must be run
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174348342
--- Diff:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/ClientSuite.scala
---
@@ -108,62 +111,42 @@ class
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174345430
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
---
@@ -89,26 +56,16 @@ private[spark
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174345374
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
---
@@ -89,26 +56,16 @@ private[spark
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r174345265
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
---
@@ -180,6 +179,26
Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/20669
Newest push passes all tests (with this merged I will then merge in
[this](https://github.com/apache-spark-on-k8s/spark-integration/pull/42/files))
```
KubernetesSuite:
- Run SparkPi
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r173665102
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/DriverConfigPropertiesStep.scala
---
@@ -0,0 +1,85
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20669#discussion_r173663339
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/DriverConfigPropertiesStep.scala
---
@@ -0,0 +1,85
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r171426248
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/20669
@vanzin @mccheah @foxish This is ready for your review as it passes unit
tests and integration tests. At the moment, it is required that we modify the
integration tests to call `SparkFiles.get
Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/20669
@vanzin you are definitely right and that is what needs to happen. However,
one of the functions of the init container was resolving the file location
which atm isnât supported unless
GitHub user ifilonenko opened a pull request:
https://github.com/apache/spark/pull/20669
[SPARK-22839][K8S] Remove the use of init-container for downloading remote
dependencies
## What changes were proposed in this pull request?
Removal of the init-container
301 - 331 of 331 matches
Mail list logo
