[
https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324167#comment-17324167
]
Attila Zsolt Piros commented on SPARK-34738:
--------------------------------------------
My new commit:
https://github.com/apache/spark/pull/31829/commits/1f594aa40061da96545d29d28ca2b2a410655c34
So the new tag is "persistentVolume".
This PR on its own should switch off the "PVs with local storage" unit test,
but to be the safe side please run a test with it.
> Upgrade Minikube and kubernetes and move to docker 'virtualization' layer
> -------------------------------------------------------------------------
>
> Key: SPARK-34738
> URL: https://issues.apache.org/jira/browse/SPARK-34738
> Project: Spark
> Issue Type: Task
> Components: jenkins, Kubernetes
> Affects Versions: 3.2.0
> Reporter: Attila Zsolt Piros
> Assignee: Shane Knapp
> Priority: Major
> Attachments: integration-tests.log
>
>
> [~shaneknapp] as we discussed [on the mailing
> list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
> Minikube can be upgraded to the latest (v1.18.1) and kubernetes version
> should be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
> [Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
> method to configure the kubernetes client. Thanks in advance to use it for
> testing on the Jenkins after the Minikube version is updated.
>
> Added by Shane:
> we also need to move from the kvm2 virtualization layer to docker. docker is
> a recommended driver w/the latest versions of minikube, and this will allow
> devs to more easily recreate the minikube/k8s env on their local workstations
> and run the integration tests in an identical environment as jenkins.
> the TL;DR is that upgrading to docker works, except that the PV integration
> tests are failing due to a couple of possible reasons:
> 1) the 'spark-kubernetes-driver' isn't properly being loaded
> (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)
> 2) during the PV test run, the error message 'Given path
> (/opt/spark/pv-tests/tmp4595937990978494271.txt) does not exist' shows up in
> the logs. however, the mk cluster *does* mount successfully to the local
> bare-metal filesystem *and* if i 'minikube ssh' in to it, i can see the mount
> and read/write successfully to it
> (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312548&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312548)
> i could really use some help, and if it's useful, i can create some local
> accounts manually and allow ssh access for a couple of people to assist me.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
