[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785581#comment-16785581 ] Stavros Kontopoulos commented on SPARK-26997: - [~shaneknapp] are the test nodes upgraded so we can create a PR with the update? > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780802#comment-16780802 ] shane knapp commented on SPARK-26997: - ok... everything is staged and all that remains to be done (in this order): * deploy symlinks on the jenkins workers to point minikube and kvm2 drivers to v0.34.1, minikube delete + rm -rf stuff * uncomment kubectl clusterrolebinding setup line in the k8s prb * 4.1.2 client PR created * see if build is happy * merge PR we can do this monday morning, or some time on wednesday. definitely not today or tomorrow please. > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779881#comment-16779881 ] shane knapp commented on SPARK-26997: - great news, everyone! :) i was able to get everything upgraded on my staging box to the latest-n-greatest, and all of the integration tests pass w/the 4.1.2 client. here are the pertinent versions of all the things: {noformat} minikube + kvm2 driver: v0.34.1 k8s: 1.13.3 client: 4.1.2 {noformat} TODO for each jenkins worker: 1) download + install latest minikube version 2) dist out a home-rolled kvm2 driver 3) change symlinks for minikube + docker-machine-driver-kvm2 to point to latest binaries 4) minikube delete, rm -rf .kube .minikube TODO for the jenkins job config: change the minikube start sequence to be the following: {code:java} $ minikube --vm-driver=kvm2 start --memory 6000 --cpus 8 --extra-config=apiserver.authorization-mode=RBAC $ kubectl create clusterrolebinding serviceaccounts-cluster-admin \ --clusterrole=cluster-admin \ --group=system:serviceaccounts{code} only when these are done will we be able to re-merge [https://github.com/apache/spark/pull/23814] to master, as well as back-port to 2.4 TIMING: i can stage this stuff on the jenkins workers, and mid-next week (once the dust literally settles) i can coordinate with whomever wants to merge/backport and make this work. phew. at least we now have some breathing room WRT deciding which k8s version/etc to test against. > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779599#comment-16779599 ] shane knapp commented on SPARK-26997: - i just confirmed that k8s 1.10.0 is definitely incompatible w/the 4.1.2 client. testing against [~vanzin]'s recent (and absolutely trivial) PR, i found that the integration tests would fail w/his changes reverted (using 4.1.2), and passed with his changes testing against 4.1.0. this means we need a minimum viable version of k8s to test against. how do we choose? especially as it seems that backwards compatibility isn't "a thing". putting on my build engineer hat: i am extremely reluctant to tie spark testing infrastructure deps directly to some magical process that follows k8s releases. it also appears that slight variations in local testing infrastructure (i use kvm2, [~vanzin] uses virtualbox, and [~skonto] none at all) cause failures (or successes) in different, and spectacular ways. and i think we might be going about creating the pods and containers to run these tests. it might be better to use yaml instead of setting a bunch of scala vars and hitting an api. this is already proven to be an absolute nightmare to debug, and for all these exact reasons. discuss. :) > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778557#comment-16778557 ] Stavros Kontopoulos commented on SPARK-26997: - [~vanzin] I run the tests successfully with the latest minikube version (v0.34.1) and k8s 1.13.3. It is a compatibility issue AFAIK. I have seen the failures in our intern ci as well. [~shaneknapp] one problem though is that while in theory using something like ` --kubernetes-version=v1.11.7` should work for targeting any k8s version it didnt. It failed with: `Caused by: java.net.UnknownHostException: kubernetes.default.svc: Try again to make tests pass for https://github.com/apache/spark/pull/23514. > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778376#comment-16778376 ] Marcelo Vanzin commented on SPARK-26997: It's linked in the description. I can revert the revert that caused it to pass again. > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778411#comment-16778411 ] shane knapp commented on SPARK-26997: - that'd be super. :) -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778247#comment-16778247 ] shane knapp commented on SPARK-26997: - i bet it's a version incompatibility. is there an open PR w/your failing changes that i can test against? > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778240#comment-16778240 ] Marcelo Vanzin commented on SPARK-26997: I'm using virtualbox. > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778234#comment-16778234 ] shane knapp commented on SPARK-26997: - how exactly are you launching minikube when performing local testing? in particular, what VM driver are you using? > k8s integration tests failing after client upgraded to 4.1.2 > > > Key: SPARK-26997 > URL: https://issues.apache.org/jira/browse/SPARK-26997 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Marcelo Vanzin >Priority: Critical > > SPARK-26742 upgraded the client libs to version 4.1.2, and that doesn't seem > to agree well with the minikube we're using in jenkins. My PRs are failing > (minikube 0.25): > {noformat} > 19/02/25 17:46:52.599 ScalaTest-main-running-KubernetesSuite INFO > ProcessUtils: 19/02/25 17:46:52 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3007689c-e3ca-48f5-a673-f3bad5c4774a > 19/02/25 17:46:52.788 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:500. Message:container not found > ("spark-kubernetes-driver") > java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal > Server Error' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 19/02/25 17:46:52.999 OkHttp https://192.168.39.69:8443/... ERROR > ExecWebSocketListener: Exec Failure: HTTP:404. Message:404 page not found > java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found' > at > okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) > at > okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) > at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) > at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} > Tests pass on my local minikube (0.34). Reverting that change makes them pass > on jenkins (see https://github.com/apache/spark/pull/23893). > Not sure if this is a client bug or a compatibility issue. > [~shaneknapp] [~skonto] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org