[jira] [Commented] (SPARK-26775) Update Jenkins nodes to support local volumes for K8s integration tests
[ https://issues.apache.org/jira/browse/SPARK-26775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784953#comment-16784953 ] shane knapp commented on SPARK-26775: - btw, once https://issues.apache.org/jira/browse/SPARK-26742 is taken care of, we can continue w/this. > Update Jenkins nodes to support local volumes for K8s integration tests > --- > > Key: SPARK-26775 > URL: https://issues.apache.org/jira/browse/SPARK-26775 > Project: Spark > Issue Type: Improvement > Components: jenkins, Kubernetes >Affects Versions: 3.0.0 >Reporter: Stavros Kontopoulos >Assignee: shane knapp >Priority: Major > > Current version of Minikube on test machines does not support properly the > local persistent volume feature required by this PR: > [https://github.com/apache/spark/pull/23514]. > We get his error: > "spec.local: Forbidden: Local volumes are disabled by feature-gate, > metadata.annotations: Required value: Local volume requires node affinity" > This is probably due to this: > [https://github.com/rancher/rancher/issues/13864] which implies that we need > to update to 1.10+ as described in > [https://kubernetes.io/docs/concepts/storage/volumes/#local]. Fabric8io > client is already updated in the PR mentioned at the beginning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26775) Update Jenkins nodes to support local volumes for K8s integration tests
[ https://issues.apache.org/jira/browse/SPARK-26775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756415#comment-16756415 ] shane knapp commented on SPARK-26775: - ill install the latest version on my test worker and run a couple of builds to check for compatibility... it should be fine... ...i hope. :) this is definitely out of scope for this issue btw and it won't be a gating factor for resolving this. also, i launched a test for the new k8s version: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7593/ looks good so far! > Update Jenkins nodes to support local volumes for K8s integration tests > --- > > Key: SPARK-26775 > URL: https://issues.apache.org/jira/browse/SPARK-26775 > Project: Spark > Issue Type: Improvement > Components: jenkins, Kubernetes >Affects Versions: 3.0.0 >Reporter: Stavros Kontopoulos >Assignee: shane knapp >Priority: Major > > Current version of Minikube on test machines does not support properly the > local persistent volume feature required by this PR: > [https://github.com/apache/spark/pull/23514]. > We get his error: > "spec.local: Forbidden: Local volumes are disabled by feature-gate, > metadata.annotations: Required value: Local volume requires node affinity" > This is probably due to this: > [https://github.com/rancher/rancher/issues/13864] which implies that we need > to update to 1.10+ as described in > [https://kubernetes.io/docs/concepts/storage/volumes/#local]. Fabric8io > client is already updated in the PR mentioned at the beginning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26775) Update Jenkins nodes to support local volumes for K8s integration tests
[
https://issues.apache.org/jira/browse/SPARK-26775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756388#comment-16756388
]
shane knapp commented on SPARK-26775:
-
all done!
{noformat}
-bash-4.1$ pssh -i -h ubuntu_workers.txt "minikube config set
kubernetes-version v1.10.0"
[1] 10:24:44 [SUCCESS] amp-jenkins-staging-worker-02
[2] 10:24:44 [SUCCESS] amp-jenkins-staging-worker-01
[3] 10:24:44 [SUCCESS] research-jenkins-worker-07
[4] 10:24:44 [SUCCESS] research-jenkins-worker-08
{noformat}
...and for testing:
{noformat}
jenkins@ubuntu-testing:~$ minikube config set kubernetes-version v1.10.0
jenkins@ubuntu-testing:~$ minikube --vm-driver=kvm2 start --memory 6000 --cpus 8
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Downloading localkube binary
173.54 MB / 173.54 MB [] 100.00% 0s
0 B / 65 B [--] 0.00%
65 B / 65 B [==] 100.00%
0sSetting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.
{noformat}
> Update Jenkins nodes to support local volumes for K8s integration tests
> ---
>
> Key: SPARK-26775
> URL: https://issues.apache.org/jira/browse/SPARK-26775
> Project: Spark
> Issue Type: Improvement
> Components: jenkins, Kubernetes
>Affects Versions: 3.0.0
>Reporter: Stavros Kontopoulos
>Assignee: shane knapp
>Priority: Major
>
> Current version of Minikube on test machines does not support properly the
> local persistent volume feature required by this PR:
> [https://github.com/apache/spark/pull/23514].
> We get his error:
> "spec.local: Forbidden: Local volumes are disabled by feature-gate,
> metadata.annotations: Required value: Local volume requires node affinity"
> This is probably due to this:
> [https://github.com/rancher/rancher/issues/13864] which implies that we need
> to update to 1.10+ as described in
> [https://kubernetes.io/docs/concepts/storage/volumes/#local]. Fabric8io
> client is already updated in the PR mentioned at the beginning.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26775) Update Jenkins nodes to support local volumes for K8s integration tests
[ https://issues.apache.org/jira/browse/SPARK-26775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756399#comment-16756399 ] Stavros Kontopoulos commented on SPARK-26775: - Yeah v0.25.0 is ancient history I guess :) > Update Jenkins nodes to support local volumes for K8s integration tests > --- > > Key: SPARK-26775 > URL: https://issues.apache.org/jira/browse/SPARK-26775 > Project: Spark > Issue Type: Improvement > Components: jenkins, Kubernetes >Affects Versions: 3.0.0 >Reporter: Stavros Kontopoulos >Assignee: shane knapp >Priority: Major > > Current version of Minikube on test machines does not support properly the > local persistent volume feature required by this PR: > [https://github.com/apache/spark/pull/23514]. > We get his error: > "spec.local: Forbidden: Local volumes are disabled by feature-gate, > metadata.annotations: Required value: Local volume requires node affinity" > This is probably due to this: > [https://github.com/rancher/rancher/issues/13864] which implies that we need > to update to 1.10+ as described in > [https://kubernetes.io/docs/concepts/storage/volumes/#local]. Fabric8io > client is already updated in the PR mentioned at the beginning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26775) Update Jenkins nodes to support local volumes for K8s integration tests
[ https://issues.apache.org/jira/browse/SPARK-26775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756390#comment-16756390 ] shane knapp commented on SPARK-26775: - we're quite behind w/the minikube version however: v0.25.0 most recent is v0.33.1 > Update Jenkins nodes to support local volumes for K8s integration tests > --- > > Key: SPARK-26775 > URL: https://issues.apache.org/jira/browse/SPARK-26775 > Project: Spark > Issue Type: Improvement > Components: jenkins, Kubernetes >Affects Versions: 3.0.0 >Reporter: Stavros Kontopoulos >Assignee: shane knapp >Priority: Major > > Current version of Minikube on test machines does not support properly the > local persistent volume feature required by this PR: > [https://github.com/apache/spark/pull/23514]. > We get his error: > "spec.local: Forbidden: Local volumes are disabled by feature-gate, > metadata.annotations: Required value: Local volume requires node affinity" > This is probably due to this: > [https://github.com/rancher/rancher/issues/13864] which implies that we need > to update to 1.10+ as described in > [https://kubernetes.io/docs/concepts/storage/volumes/#local]. Fabric8io > client is already updated in the PR mentioned at the beginning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26775) Update Jenkins nodes to support local volumes for K8s integration tests
[ https://issues.apache.org/jira/browse/SPARK-26775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756375#comment-16756375 ] shane knapp commented on SPARK-26775: - okie dokie, i should get this sorted today. > Update Jenkins nodes to support local volumes for K8s integration tests > --- > > Key: SPARK-26775 > URL: https://issues.apache.org/jira/browse/SPARK-26775 > Project: Spark > Issue Type: Improvement > Components: jenkins, Kubernetes >Affects Versions: 3.0.0 >Reporter: Stavros Kontopoulos >Assignee: shane knapp >Priority: Major > > Current version of Minikube on test machines does not support properly the > local persistent volume feature required by this PR: > [https://github.com/apache/spark/pull/23514]. > We get his error: > "spec.local: Forbidden: Local volumes are disabled by feature-gate, > metadata.annotations: Required value: Local volume requires node affinity" > This is probably due to this: > [https://github.com/rancher/rancher/issues/13864] which implies that we need > to update to 1.10+ as described in > [https://kubernetes.io/docs/concepts/storage/volumes/#local]. Fabric8io > client is already updated in the PR mentioned at the beginning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
