[
https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312517#comment-17312517
]
Shane Knapp edited comment on SPARK-34738 at 3/31/21, 4:43 PM:
---------------------------------------------------------------
alright, sometimes these things go smoothly, sometimes not.
this is firmly in the 'not' camp.
after upgrading minikube and k8s, i was unable to mount a persistent volume
when using the kvm2 driver. much debugging ensued. no progress was made and
the error reported was that the minikube pod was unable to connect to the
localhost and mount (Connection refused).
so, i decided to randomly try the docker minikube driver. voila! i'm now able
to happily mount persistent volumes.
however, when running the k8s integration test, everything passes *except* the
PVs w/local storage.
from [https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s-clone:]
{code:java}
- PVs with local storage *** FAILED ***
The code passed to eventually never returned normally. Attempted 179 times
over 3.002424470466666 minutes. Last failure message: container not found
("spark-kubernetes-driver"). (PVTestsSuite.scala:117){code}
i've never seen this error before, and apparently there aren't many things
here's how we launch minikube and create the mount:
{code:java}
minikube --vm-driver=docker start --memory 6000 --cpus 8
minikube mount ${PVC_TESTS_HOST_PATH}:${PVC_TESTS_VM_PATH}
--9p-version=9p2000.L &
{code}
we're using ZFS on the bare metal, and minikube is complaining:
{code:java}
! docker is currently using the zfs storage driver, consider switching to
overlay2 for better performance{code}
i'll continue to dig in to this today, but i'm currently blocked...
was (Author: shaneknapp):
alright, sometimes these things go smoothly, sometimes not.
this is firmly in the 'not' camp.
after upgrading minikube and k8s, i was unable to mount a persistent volume
when using the kvm2 driver. much debugging ensued. no progress was made.
so, i decided to randomly try the docker minikube driver. voila! i'm now able
to happily mount persistent volumes.
however, when running the k8s integration test, everything passes *except* the
PVs w/local storage.
from [https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s-clone:]
{code:java}
- PVs with local storage *** FAILED ***
The code passed to eventually never returned normally. Attempted 179 times
over 3.002424470466666 minutes. Last failure message: container not found
("spark-kubernetes-driver"). (PVTestsSuite.scala:117){code}
i've never seen this error before, and apparently there aren't many things
here's how we launch minikube and create the mount:
{code:java}
minikube --vm-driver=docker start --memory 6000 --cpus 8
minikube mount ${PVC_TESTS_HOST_PATH}:${PVC_TESTS_VM_PATH}
--9p-version=9p2000.L &
{code}
we're using ZFS on the bare metal, and minikube is complaining:
{code:java}
! docker is currently using the zfs storage driver, consider switching to
overlay2 for better performance{code}
i'll continue to dig in to this today, but i'm currently blocked...
> Upgrade Minikube and kubernetes cluster version on Jenkins
> ----------------------------------------------------------
>
> Key: SPARK-34738
> URL: https://issues.apache.org/jira/browse/SPARK-34738
> Project: Spark
> Issue Type: Task
> Components: jenkins, Kubernetes
> Affects Versions: 3.2.0
> Reporter: Attila Zsolt Piros
> Assignee: Shane Knapp
> Priority: Major
>
> [~shaneknapp] as we discussed [on the mailing
> list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
> Minikube can be upgraded to the latest (v1.18.1) and kubernetes version
> should be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
> [Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
> method to configure the kubernetes client. Thanks in advance to use it for
> testing on the Jenkins after the Minikube version is updated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
