rami created SPARK-53775:
----------------------------
Summary: Spark 4 on k3s NoSuchMethodError in
io.netty.util.internal.PlatformDependent
Key: SPARK-53775
URL: https://issues.apache.org/jira/browse/SPARK-53775
Project: Spark
Issue Type: Bug
Components: k8s
Affects Versions: 4.0.1, 4.0.0
Environment: {noformat}
❯ k version
Client Version: v1.34.0
Kustomize Version: v5.7.1
Server Version: v1.33.4+k3s1{noformat}
The bug is reproducible with Spark 4.0.0 and Spark 4.0.1.
It is not reproducible with Spark 3.5.[67]
Reporter: rami
Hello,
I see the exception below on k3s but *not* on other Kubernetes flavors.
Looks like k3s sets up pods in such a way that triggers a dependency bug that
is otherwise not visible.
I can provide more details if required.
{noformat}
Exception in thread "main"
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:109)
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:102)
at
io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44)
at
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1155)
at
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:98)
at
org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:154)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$6(KubernetesClientApplication.scala:258)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$6$adapted(KubernetesClientApplication.scala:252)
at
org.apache.spark.util.SparkErrorUtils.tryWithResource(SparkErrorUtils.scala:48)
at
org.apache.spark.util.SparkErrorUtils.tryWithResource$(SparkErrorUtils.scala:46)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:99)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:252)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:225)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1027)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Failed to create SSL connection
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340)
at
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:754)
at
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:98)
at
io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42)
... 17 more
Caused by: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
...
Caused by: javax.net.ssl.SSLException: failure when writing TLS control frames
at
io.netty.handler.ssl.SslHandler.setHandshakeFailureTransportFailure(SslHandler.java:2050)
... 30 more
Caused by: java.lang.NoSuchMethodError: 'java.util.Queue
io.netty.util.internal.PlatformDependent.newFixedMpscUnpaddedQueue(int)'
at
io.netty.buffer.PoolThreadCache$MemoryRegionCache.<init>(PoolThreadCache.java:337)
at
io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache.<init>(PoolThreadCache.java:302)
at
io.netty.buffer.PoolThreadCache.createSubPageCaches(PoolThreadCache.java:113)
at io.netty.buffer.PoolThreadCache.<init>(PoolThreadCache.java:77)
at
io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.initialValue(PooledByteBufAllocator.java:541)
at
io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.initialValue(PooledByteBufAllocator.java:518)
at
io.netty.util.concurrent.FastThreadLocal.initialize(FastThreadLocal.java:177)
at io.netty.util.concurrent.FastThreadLocal.get(FastThreadLocal.java:142)
at
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:398)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
at
io.vertx.core.buffer.impl.PartialPooledByteBufAllocator.directBuffer(PartialPooledByteBufAllocator.java:84)
at
io.netty.channel.nio.AbstractNioChannel.newDirectBuffer(AbstractNioChannel.java:447)
at
io.netty.channel.nio.AbstractNioByteChannel.filterOutboundMessage(AbstractNioByteChannel.java:278)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:866)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1314)
at
io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:889)
at
io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875)
at
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984)
at
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868)
at
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:863)
... 21 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
