[
https://issues.apache.org/jira/browse/SPARK-53775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18074433#comment-18074433
]
Xiaoxuan Li commented on SPARK-53775:
-------------------------------------
hi [~razvanmihai], Could you please share: The full classpath of the Spark pod
on k3s, are there any Netty jars outside the Spark distribution? Are you using
the official Spark Docker image or a custom one?
The NoSuchMethodError suggests two different Netty versions at runtime, and we
want to confirm whether the conflict comes from Spark's build or the k3s
environment.
> Spark 4 on k3s NoSuchMethodError in io.netty.util.internal.PlatformDependent
> ----------------------------------------------------------------------------
>
> Key: SPARK-53775
> URL: https://issues.apache.org/jira/browse/SPARK-53775
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 4.0.0, 4.0.1
> Environment: {noformat}
> ❯ k version
> Client Version: v1.34.0
> Kustomize Version: v5.7.1
> Server Version: v1.33.4+k3s1{noformat}
> The bug is reproducible with Spark 4.0.0 and Spark 4.0.1.
> It is not reproducible with Spark 3.5.[67]
> Reporter: rami
> Priority: Major
>
> Hello,
> I see the exception below on k3s but *not* on other Kubernetes flavors.
> Looks like k3s sets up pods in such a way that triggers a dependency bug that
> is otherwise not visible.
> I can provide more details if required.
> {noformat}
> Exception in thread "main"
> io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:109)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:102)
> at
> io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44)
> at
> io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1155)
> at
> io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:98)
> at
> org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:154)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$6(KubernetesClientApplication.scala:258)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$6$adapted(KubernetesClientApplication.scala:252)
> at
> org.apache.spark.util.SparkErrorUtils.tryWithResource(SparkErrorUtils.scala:48)
> at
> org.apache.spark.util.SparkErrorUtils.tryWithResource$(SparkErrorUtils.scala:46)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:99)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:252)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:225)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1027)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.io.IOException: Failed to create SSL connection
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504)
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340)
> at
> io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:754)
> at
> io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:98)
> at
> io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42)
> ... 17 more
> Caused by: javax.net.ssl.SSLHandshakeException: Failed to create SSL
> connection
> ...
> Caused by: javax.net.ssl.SSLException: failure when writing TLS control frames
> at
> io.netty.handler.ssl.SslHandler.setHandshakeFailureTransportFailure(SslHandler.java:2050)
> ... 30 more
> Caused by: java.lang.NoSuchMethodError: 'java.util.Queue
> io.netty.util.internal.PlatformDependent.newFixedMpscUnpaddedQueue(int)'
> at
> io.netty.buffer.PoolThreadCache$MemoryRegionCache.<init>(PoolThreadCache.java:337)
> at
> io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache.<init>(PoolThreadCache.java:302)
> at
> io.netty.buffer.PoolThreadCache.createSubPageCaches(PoolThreadCache.java:113)
> at io.netty.buffer.PoolThreadCache.<init>(PoolThreadCache.java:77)
> at
> io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.initialValue(PooledByteBufAllocator.java:541)
> at
> io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.initialValue(PooledByteBufAllocator.java:518)
> at
> io.netty.util.concurrent.FastThreadLocal.initialize(FastThreadLocal.java:177)
> at io.netty.util.concurrent.FastThreadLocal.get(FastThreadLocal.java:142)
> at
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:398)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
> at
> io.vertx.core.buffer.impl.PartialPooledByteBufAllocator.directBuffer(PartialPooledByteBufAllocator.java:84)
> at
> io.netty.channel.nio.AbstractNioChannel.newDirectBuffer(AbstractNioChannel.java:447)
> at
> io.netty.channel.nio.AbstractNioByteChannel.filterOutboundMessage(AbstractNioByteChannel.java:278)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:866)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1314)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:889)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875)
> at
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984)
> at
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868)
> at
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:863)
> ... 21 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
