[GitHub] spark issue #22087: [SPARK-25097][ML] Support prediction on single instance ...

2018-11-07 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22087 PredictionModel as a super-class of unsupervised and supervised seems sane to me. Returning a double to unify the signature also seems sane, although the thought of casting it might irk

[GitHub] spark issue #22087: [SPARK-25097][ML] Support prediction on single instance ...

2018-11-06 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22087 This LGTM, but it raises a more general question about the lack of single-sample prediction over the entire hierarchy. For example (IMO) there should be some kind of single-sample method

[GitHub] spark issue #22146: [SPARK-24434][K8S] pod template files

2018-10-29 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22146 @mccheah integration testing is passing with the latest container selection policy, good to merge? --- - To unsubscribe

[GitHub] spark issue #22146: [SPARK-24434][K8S] pod template files

2018-10-29 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22146 thanks @shaneknapp ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22146: [SPARK-24434][K8S] pod template files

2018-10-29 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22146 k8s integration test failure appears repeatable: ``` Downloading from central: https://repo.maven.apache.org/maven2/com/squareup/okhttp3/okhttp/3.9.1/okhttp-3.9.1.jar [WARNING

[GitHub] spark issue #22146: [SPARK-24434][K8S] pod template files

2018-10-29 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22146 Latest container selection w/ default to first LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22820: [SPARK-25828][K8S] Bumping Kubernetes-Client version to ...

2018-10-26 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22820 merging --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Clie...

2018-10-25 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22820#discussion_r228321586 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala --- @@ -157,7 +157,10 @@ private[spark

[GitHub] spark pull request #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Clie...

2018-10-25 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22820#discussion_r228321407 --- Diff: docs/running-on-kubernetes.md --- @@ -45,7 +45,8 @@ logs and remains in "completed" state in the Kubernetes API until it's

[GitHub] spark issue #22838: [SPARK-25835][K8s] fix issues with k8s tests

2018-10-25 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22838 LGTM - the k8s integration tests are off by default, can be run via `-P...`, and are passwing via Jenkins

[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...

2018-10-22 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22433 @suryag10, all things being equal, it is considered preferable to provide testing for new functionality on the same PR. Are there are logistical problems adding testing here

[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...

2018-10-22 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22433 @suryag10 you were probably encountering github server problems from yesterday: https://status.github.com/messages

[GitHub] spark issue #13440: [SPARK-15699] [ML] Implement a Chi-Squared test statisti...

2018-10-19 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/13440 update - I'm consulting with some teammates about what it might mean to also support Bayesian variations on split quality, since there has been a lot of interest in the last few years

[GitHub] spark pull request #22608: [SPARK-23257][K8S][TESTS] Kerberos Support Integr...

2018-10-16 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22608#discussion_r225625939 --- Diff: bin/docker-image-tool.sh --- @@ -71,18 +71,29 @@ function build { --build-arg base_img=$(image_ref spark

[GitHub] spark issue #22608: [SPARK-23257][K8S][TESTS] Kerberos Support Integration T...

2018-10-06 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22608 Although this is a large patch, its impact on existing code is small, and it is nearly all testing code. Unless the tests themselves are unstable, I'd consider this plausible to include

[GitHub] spark issue #22608: [SPARK-23257][K8S][TESTS] Kerberos Support Integration T...

2018-10-03 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22608 re: hadoop-2.7.3.tgz is that something Shane needs to install on the testing infra, to build the images you want

[GitHub] spark issue #22608: [SPARK-23257][K8S][TESTS] Kerberos Support Integration T...

2018-10-03 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22608 @ifilonenko can we work with the existing service-account-name config parameters for obtaining the resource permissions

[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...

2018-10-01 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22433 If possible, there should be some basic integration testing. Run a thrift server command against the minishift cluster used by the other testing

[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...

2018-10-01 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22433 In the scenario of a cluster-mode submission, what is the command-line behavior? Does the thrift-server script "block" until the thrift server pod is

[GitHub] spark issue #13440: [SPARK-15699] [ML] Implement a Chi-Squared test statisti...

2018-09-22 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/13440 I think targeting 3.0 with a refactor makes the most sense. There's no way to do this without making small breaking changes, but slightly larger changes could clean up the design

[GitHub] spark pull request #13440: [SPARK-15699] [ML] Implement a Chi-Squared test s...

2018-09-19 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/13440#discussion_r218997600 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala --- @@ -71,6 +71,23 @@ object Gini extends Impurity { @Since

[GitHub] spark pull request #13440: [SPARK-15699] [ML] Implement a Chi-Squared test s...

2018-09-17 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/13440#discussion_r218252461 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurity.scala --- @@ -52,6 +52,49 @@ trait Impurity extends Serializable

[GitHub] spark pull request #13440: [SPARK-15699] [ML] Implement a Chi-Squared test s...

2018-09-17 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/13440#discussion_r218252156 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala --- @@ -71,6 +71,23 @@ object Gini extends Impurity { @Since

[GitHub] spark pull request #13440: [SPARK-15699] [ML] Implement a Chi-Squared test s...

2018-09-17 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/13440#discussion_r218246415 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala --- @@ -71,6 +71,23 @@ object Gini extends Impurity { @Since

[GitHub] spark pull request #13440: [SPARK-15699] [ML] Implement a Chi-Squared test s...

2018-09-17 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/13440#discussion_r218245862 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/ChiSquared.scala --- @@ -0,0 +1,162 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #13440: [SPARK-15699] [ML] Implement a Chi-Squared test s...

2018-09-17 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/13440#discussion_r218245670 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala --- @@ -670,14 +670,32 @@ private[spark] object RandomForest extends

[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...

2018-09-16 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22433 I'm wondering, is there some reason this isn't supported in cluster mode for yarn & mesos? Or put another way, what is the rationale for k8s being added as an exception to this

[GitHub] spark issue #22381: [SPARK-25394][CORE] Add an application status metrics so...

2018-09-12 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22381 These new metrics seem useful. Is there a way to provide unit or integration testing for it? Do these have enable/disable via metrics.properties files

[GitHub] spark issue #22146: [SPARK-24434][K8S] pod template files

2018-08-30 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22146 +1 for @skonto recommendation about CI testing templates populated with larger numbers of fields, and template errors

[GitHub] spark issue #22294: [SPARK-25287][INFRA] Add up-front check for JIRA_USERNAM...

2018-08-30 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22294 cc @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22294: [SPARK-25287][INFRA] Add up-front check for JIRA_...

2018-08-30 Thread erikerlandson
GitHub user erikerlandson opened a pull request: https://github.com/apache/spark/pull/22294 [SPARK-25287][INFRA] Add up-front check for JIRA_USERNAME and JIRA_PASSWORD ## What changes were proposed in this pull request? Add an up-front check that `JIRA_USERNAME

[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-30 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22179 For maximum breaking-change forgiveness, the "safe" option would be to punt it to 3.0; which seems likely to be the next rel

[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-30 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22179 If it doesn't expose any breaking changes to users, that seems reasonable. I assume there would be no integration problems with scala 2.12

[GitHub] spark issue #22285: [SPARK-25275][K8S] require memberhip in wheel to run 'su...

2018-08-30 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22285 looks like alpine doesn't have PAM installed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22285: [SPARK-25275][K8S] require memberhip in wheel to run 'su...

2018-08-30 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22285 jenkins test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22285: [SPARK-25275][K8S] require memberhip in wheel to ...

2018-08-30 Thread erikerlandson
GitHub user erikerlandson opened a pull request: https://github.com/apache/spark/pull/22285 [SPARK-25275][K8S] require memberhip in wheel to run 'su' in dockerfiles ## What changes were proposed in this pull request? Add a PAM configuration in k8s dockerfile to require

[GitHub] spark pull request #21279: [SPARK-24219][k8s] Improve the docker building sc...

2018-08-28 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21279#discussion_r213317184 --- Diff: bin/docker-image-tool.sh --- @@ -44,15 +44,37 @@ function image_ref { function build { local BUILD_ARGS local IMG_PATH

[GitHub] spark pull request #21279: [SPARK-24219][k8s] Improve the docker building sc...

2018-08-28 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21279#discussion_r213316408 --- Diff: bin/docker-image-tool.sh --- @@ -44,15 +44,37 @@ function image_ref { function build { local BUILD_ARGS local IMG_PATH

[GitHub] spark pull request #21279: [SPARK-24219][k8s] Improve the docker building sc...

2018-08-28 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21279#discussion_r213314870 --- Diff: bin/docker-image-tool.sh --- @@ -44,15 +44,37 @@ function image_ref { function build { local BUILD_ARGS local IMG_PATH

[GitHub] spark issue #22212: [SPARK-25220] Seperate kubernetes node selector config b...

2018-08-27 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22212 I agree there's an argument for keeping this, but an alternative would be to leave the original for backward compatability, deprecate it, and recommend people make use of custom pod templates

[GitHub] spark pull request #22212: [SPARK-25220] Seperate kubernetes node selector c...

2018-08-27 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22212#discussion_r213127037 --- Diff: docs/running-on-kubernetes.md --- @@ -663,11 +663,21 @@ specific to Spark on Kubernetes

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-24 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212668742 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -225,6 +225,18 @@ private[spark] object

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-23 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212469389 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -225,6 +225,18 @@ private[spark] object

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-23 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212460665 --- Diff: docs/running-on-kubernetes.md --- @@ -775,4 +787,183 @@ specific to Spark on Kubernetes. This sets the major Python version

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-23 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212438850 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -225,6 +225,18 @@ private[spark] object

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-23 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212423082 --- Diff: docs/running-on-kubernetes.md --- @@ -775,4 +787,183 @@ specific to Spark on Kubernetes. This sets the major Python version

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-23 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212424515 --- Diff: docs/running-on-kubernetes.md --- @@ -775,4 +787,183 @@ specific to Spark on Kubernetes. This sets the major Python version

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-23 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212423941 --- Diff: docs/running-on-kubernetes.md --- @@ -775,4 +787,183 @@ specific to Spark on Kubernetes. This sets the major Python version

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-23 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212399808 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -225,6 +225,18 @@ private[spark] object

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-08-23 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r212364915 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -225,6 +225,18 @@ private[spark] object

[GitHub] spark issue #22146: [WIP][SPARK-24434][K8S] pod template files

2018-08-20 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22146 Thanks @onursatici! Can you please resolve merge conflicts (rebase?), so CI can build it? --- - To unsubscribe, e-mail

[GitHub] spark issue #22146: [WIP][SPARK-24434][K8S] pod template files

2018-08-20 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/22146 Jenkins, OK to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21555: [SPARK-24547][K8S] Allow for building spark on k8s docke...

2018-06-20 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21555 LGMT, I am OK to merge. Most of the automated image build tooling I've seen is custom, but I agree w/ Matt that being able to selectively build is worth supporting, via a followup PR

[GitHub] spark issue #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s commands

2018-06-19 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21572 testing environment is synced again and its passing, so I'm going to merge --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21551: [K8S] Fix issue in 'docker-image-tool.sh'

2018-06-19 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21551 @fabriziocucci can you close this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s commands

2018-06-19 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21572 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21551: [K8S] Fix issue in 'docker-image-tool.sh'

2018-06-18 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21551 OK, I see it referenced above as commit b8dbfcc. I'd close this, but that option is disabled for me. Probably assuming it happens from the merge-pr script

[GitHub] spark issue #21551: [K8S] Fix issue in 'docker-image-tool.sh'

2018-06-18 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21551 @mccheah this isn't showing as merged, you said you ran the merge script? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s commands

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21572 please test this --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s commands

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21572 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s comman...

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21572#discussion_r195866275 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -37,11 +37,17 @@ if [ -z "$uidentry" ] ; th

[GitHub] spark pull request #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s comman...

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21572#discussion_r195811228 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -38,10 +38,10 @@ fi SPARK_K8S_CMD=&qu

[GitHub] spark pull request #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s comman...

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21572#discussion_r195804610 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -38,10 +38,10 @@ fi SPARK_K8S_CMD=&qu

[GitHub] spark pull request #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s comman...

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21572#discussion_r195794051 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -38,10 +38,10 @@ fi SPARK_K8S_CMD=&qu

[GitHub] spark pull request #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s comman...

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21572#discussion_r195793358 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -110,8 +110,7 @@ case "$SPARK_K8

[GitHub] spark pull request #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s comman...

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21572#discussion_r195787809 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -38,10 +38,10 @@ fi SPARK_K8S_CMD=&qu

[GitHub] spark issue #21572: [SPARK-24534][K8S] Bypass non spark-on-k8s commands

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21572 @rimolive there are some significant differences between the head of master and your file. I'm wondering if you should get the head of master and re-edit from there, because rebasing didn't

[GitHub] spark issue #21572: Bypass non spark-on-k8s commands

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21572 @rimolive I think you may need to rebase this to latest head of master, to pick up the pyspark updates

[GitHub] spark issue #21572: Bypass non spark-on-k8s commands

2018-06-15 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21572 Thanks @rimolive ! Can you please prepend `[SPARK-24534][K8S]` to the title of this PR? --- - To unsubscribe, e-mail

[GitHub] spark pull request #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to...

2018-05-24 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r190755750 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/DeterministicExecutorPodsEventQueue.scala

[GitHub] spark pull request #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to...

2018-05-21 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r189735073 --- Diff: pom.xml --- @@ -150,6 +150,7 @@ 4.5.4 4.4.8 +3.0.1 --- End diff -- My take

[GitHub] spark pull request #21366: [SPARK-24248][K8S][WIP] Use the Kubernetes API to...

2018-05-19 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r189435162 --- Diff: pom.xml --- @@ -150,6 +150,7 @@ 4.5.4 4.4.8 +3.0.1 --- End diff -- it looks like this dep

[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...

2018-05-09 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21238 I agree with @mcheah that the potential code reuse is small. Keeping this as a separate pod construction step, decoupled from the user-exposed step, is cleaner

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186859389 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186855467 --- Diff: docs/running-on-kubernetes.md --- @@ -561,6 +561,13 @@ specific to Spark on Kubernetes. This is distinct from spark.executor.cores

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186846341 --- Diff: docs/running-on-kubernetes.md --- @@ -561,6 +561,13 @@ specific to Spark on Kubernetes. This is distinct from spark.executor.cores

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186839692 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-02 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20697 @ssuchter good point, they should. I thought they were set up to be invoked because kube has already been added to modules.py

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-05-02 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20697 Barring further feedback, resolving a two issues should have this ready to merge: 1. move cloud-backed testing to a new PR 1. remove the repository clone logic

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-05-02 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21092 @holdenk I think your [comment above](https://github.com/apache/spark/pull/21092#issuecomment-383211329) gets at a use-case "ambiguity" that containerization causes. There are no

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-20 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r183161979 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile --- @@ -0,0 +1,33 @@ +# +# Licensed

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182564020 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala --- @@ -71,7 +77,7

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182563748 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -62,6 +69,14 @@ case "$SPARK_K8

[GitHub] spark issue #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings for PySp...

2018-04-18 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21092 Thanks @ifilonenko ! I'm interested in figuring out what it means for the container images to be "python 2/3 generic" - does that imply being able to run either, based

[GitHub] spark issue #20669: [SPARK-22839][K8S] Remove the use of init-container for ...

2018-03-19 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20669 @mccheah workflow is to use `dev/merge_spark_pr.py` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #20669: [SPARK-22839][K8S] Remove the use of init-contain...

2018-03-19 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/20669#discussion_r175532438 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -53,14 +53,10 @@ fi case "$SPARK_K8

[GitHub] spark issue #20822: [SPARK-23680] Fix entrypoint.sh to properly support Arbi...

2018-03-16 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20822 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20822: [SPARK-23680] Fix entrypoint.sh to properly support Arbi...

2018-03-14 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20822 LGTM, pending tests --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20822: [SPARK-23680] Fix entrypoint.sh to properly suppo...

2018-03-14 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/20822#discussion_r174527459 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -17,13 +17,15 @@ # # echo commands

[GitHub] spark issue #20822: [SPARK-23680] Fix entrypoint.sh to properly support Arbi...

2018-03-14 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20822 jenkins please test this --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #20822: [SPARK-23680] Fix entrypoint.sh to properly support Arbi...

2018-03-14 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20822 @rimolive thanks for this! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

2018-02-14 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/19775 Although this is not kube-specific, kubernetes deployment is a major prometheus use case. Has it been tested in a kube environment

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

2018-02-14 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/19775 I agree w/ @jerryshao that adding new deps to core isn't ideal. (Also that having #11994 would be really nice) New deps on a sub-project seems more palatable, but interested in what

[GitHub] spark issue #20007: [SPARK-22777][Scheduler] Kubernetes mode dockerfile perm...

2017-12-18 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20007 If we wanted to lock down permissions a bit more, we might consider `750`, but I'd prefer to make permission changes with more time for testing, maybe for 2.4

[GitHub] spark issue #20007: [SPARK-22777][Scheduler] Kubernetes mode dockerfile perm...

2017-12-18 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20007 Yes, by default in openshift, containers run as an anonymous uid, and as group id 0. So there are a few things that need to be given access to gid 0. I asked some of our security people

[GitHub] spark issue #19995: [SPARK-22807] [Scheduler] Remove config that says docker...

2017-12-15 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/19995 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19995: [SPARK-22807] [Scheduler] Remove config that says docker...

2017-12-15 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/19995 I'm picking up a few stragglers from `git grep` ``` resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/BaseDriverConfigurationStep.scala

[GitHub] spark pull request #19946: [SPARK-22648] [Scheduler] Spark on Kubernetes - D...

2017-12-15 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/19946#discussion_r157274013 --- Diff: docs/running-on-kubernetes.md --- @@ -0,0 +1,502 @@ +--- +layout: global +title: Running Spark on Kubernetes

[GitHub] spark pull request #19946: [SPARK-22648] [Scheduler] Spark on Kubernetes - D...

2017-12-14 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/19946#discussion_r157076409 --- Diff: docs/running-on-kubernetes.md --- @@ -0,0 +1,502 @@ +--- +layout: global +title: Running Spark on Kubernetes

[GitHub] spark pull request #19946: [SPARK-22648] [Scheduler] Spark on Kubernetes - D...

2017-12-14 Thread erikerlandson
Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/19946#discussion_r157075744 --- Diff: docs/running-on-kubernetes.md --- @@ -0,0 +1,502 @@ +--- +layout: global +title: Running Spark on Kubernetes

  1   2   3   >