[
https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Schweigert updated SPARK-28921:
------------------------------------
Description:
Spark jobs are failing on latest versions of Kubernetes when jobs attempt to
provision executor pods (jobs like Spark-Pi that do not launch executors run
without a problem):
Here's an example error message:
{code:java}
19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors from
Kubernetes.
19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors from
Kubernetes.19/08/30 01:29:09 WARN WatchConnectionManager: Exec Failure: HTTP
403, Status: 403 -
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Looks like the issue is caused by the internal master Kubernetes url not having
the port specified:
[https://github.com/apache/spark/blob/master//resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L82:7]
Using the master with the port (443) seems to fix the problem.
was:
Spark jobs are failing on latest versions of Kubernetes when jobs attempt to
provision executor pods (jobs like Spark-Pi that do not launch executors run
without a problem):
Here's an example error message:
{code:java}
19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors from
Kubernetes.19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2
executors from Kubernetes.19/08/30 01:29:09 WARN WatchConnectionManager: Exec
Failure: HTTP 403, Status: 403 - java.net.ProtocolException: Expected HTTP 101
response but was '403 Forbidden' at
okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216) at
okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183) at
okhttp3.RealCall$AsyncCall.execute(RealCall.java:141) at
okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Looks like the issue is caused by the internal master Kubernetes url not having
the port specified:
[https://github.com/apache/spark/blob/master//resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L82:7]
Using the master with the port (443) seems to fix the problem.
> Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10)
> -----------------------------------------------------------------------------
>
> Key: SPARK-28921
> URL: https://issues.apache.org/jira/browse/SPARK-28921
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.3
> Reporter: Paul Schweigert
> Priority: Minor
>
> Spark jobs are failing on latest versions of Kubernetes when jobs attempt to
> provision executor pods (jobs like Spark-Pi that do not launch executors run
> without a problem):
>
> Here's an example error message:
>
> {code:java}
> 19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors
> from Kubernetes.
> 19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors
> from Kubernetes.19/08/30 01:29:09 WARN WatchConnectionManager: Exec Failure:
> HTTP 403, Status: 403 -
> java.net.ProtocolException: Expected HTTP 101 response but was '403
> Forbidden'
> at
> okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
> at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> Looks like the issue is caused by the internal master Kubernetes url not
> having the port specified:
> [https://github.com/apache/spark/blob/master//resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L82:7]
>
> Using the master with the port (443) seems to fix the problem.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
