[
https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Knapp updated SPARK-34738:
--------------------------------
Description:
[~shaneknapp] as we discussed [on the mailing
list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
Minikube can be upgraded to the latest (v1.18.1) and kubernetes version should
be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
[Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
method to configure the kubernetes client. Thanks in advance to use it for
testing on the Jenkins after the Minikube version is updated.
Added by Shane:
we also need to move from the kvm2 virtualization layer to docker. docker is a
recommended driver w/the latest versions of minikube, and this will allow devs
to more easily recreate the minikube/k8s env on their local workstations and
run the integration tests in an identical environment as jenkins.
the TL;DR is that upgrading to docker works, except that the PV integration
tests are failing due to a couple of possible reasons:
1) the 'spark-kubernetes-driver' isn't properly being loaded
(https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)
2) during the PV test run, the error message 'Given path
(/opt/spark/pv-tests/tmp4595937990978494271.txt) does not exist' shows up in
the logs. however, the mk cluster *does* mount successfully to the local
bare-metal filesystem *and* if i 'minikube ssh' in to it, i can see the mount
and read/write successfully to it (
was:
[~shaneknapp] as we discussed [on the mailing
list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
Minikube can be upgraded to the latest (v1.18.1) and kubernetes version should
be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
[Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
method to configure the kubernetes client. Thanks in advance to use it for
testing on the Jenkins after the Minikube version is updated.
Added by Shane:
we also need to move from the kvm2 virtualization layer to docker. docker is a
recommended driver w/the latest versions of minikube, and this will allow devs
to more easily recreate the minikube/k8s env on their local workstations and
run the integration tests in an identical environment as jenkins.
the TL;DR is that upgrading to docker works, except that the PV integration
tests are failing due to a couple of possible reasons:
1) the 'spark-kubernetes-driver' isn't properly being loaded
(https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)
2) /
> Upgrade Minikube and kubernetes and move to docker 'virtualization' layer
> -------------------------------------------------------------------------
>
> Key: SPARK-34738
> URL: https://issues.apache.org/jira/browse/SPARK-34738
> Project: Spark
> Issue Type: Task
> Components: jenkins, Kubernetes
> Affects Versions: 3.2.0
> Reporter: Attila Zsolt Piros
> Assignee: Shane Knapp
> Priority: Major
> Attachments: integration-tests.log
>
>
> [~shaneknapp] as we discussed [on the mailing
> list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
> Minikube can be upgraded to the latest (v1.18.1) and kubernetes version
> should be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
> [Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
> method to configure the kubernetes client. Thanks in advance to use it for
> testing on the Jenkins after the Minikube version is updated.
>
> Added by Shane:
> we also need to move from the kvm2 virtualization layer to docker. docker is
> a recommended driver w/the latest versions of minikube, and this will allow
> devs to more easily recreate the minikube/k8s env on their local workstations
> and run the integration tests in an identical environment as jenkins.
> the TL;DR is that upgrading to docker works, except that the PV integration
> tests are failing due to a couple of possible reasons:
> 1) the 'spark-kubernetes-driver' isn't properly being loaded
> (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)
> 2) during the PV test run, the error message 'Given path
> (/opt/spark/pv-tests/tmp4595937990978494271.txt) does not exist' shows up in
> the logs. however, the mk cluster *does* mount successfully to the local
> bare-metal filesystem *and* if i 'minikube ssh' in to it, i can see the mount
> and read/write successfully to it (
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
