Iain Cardnell created SPARK-43175:
-------------------------------------
Summary: decom.sh can cause an UnsupportedOperationException
Key: SPARK-43175
URL: https://issues.apache.org/jira/browse/SPARK-43175
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 3.3.0
Reporter: Iain Cardnell
decom.sh can cause an UnsupportedOperationException which then causes the
Executor to die with a SparkUncaughtException and does not complete the
decommission properly.
*Problem:*
SignalUtils.scala line 124:
{code:java}
if (escalate) {
prevHandler.handle(sig)
}{code}
*Logs:*
{noformat}
failed - error: command '/opt/decom.sh' exited with 137: + echo 'Asked to
decommission' + date + tee -a ++ ps -o pid -C java ++ awk '{ sub(/^[ \t]+/,
""""); print }' ++ tail -n 1 + WORKER_PID=17 + echo 'Using worker pid 17' +
kill -s SIGPWR 17 + echo 'Waiting for worker pid to exit' + timeout 60 tail
--pid=17 -f /dev/null , message: ""Asked to decommission\nMon Apr 17 23:44:35
UTC 2023\nUsing worker pid 17\nWaiting for worker pid to exit\n+ echo 'Asked to
decommission'\n+ date\n+ tee -a\n++ ps -o pid -C java\n++ awk '{ sub(/^[
\\t]+/, \""\""); print }'\n++ tail -n 1\n+ WORKER_PID=17\n+ echo 'Using worker
pid 17'\n+ kill -s SIGPWR 17\n+ echo 'Waiting for worker pid to exit'\n+
timeout 60 tail --pid=17 -f /dev/null\n""",2023-04-17T23:44:39Z,
"java.lang.UnsupportedOperationException: invoking native signal handle not
supported at java.base/jdk.internal.misc.Signal$NativeHandler.handle(Unknown
Source) at jdk.unsupported/sun.misc.Signal$SunMiscHandler.handle(Unknown
Source) at
org.apache.spark.util.SignalUtils$ActionHandler.handle(SignalUtils.scala:124)
at jdk.unsupported/sun.misc.Signal$InternalMiscHandler.handle(Unknown Source)
at java.base/jdk.internal.misc.Signal$1.run(Unknown Source) at
java.base/java.lang.Thread.run(Unknown Source)",2023-04-17T23:44:35.407488217Z
"2023-04-17 23:44:35 [SIGPWR handler] ERROR
org.apache.spark.util.SparkUncaughtExceptionHandler - Uncaught exception in
thread Thread[SIGPWR handler,9,system] - {}",2023-04-17T23:44:35.407457859Z "
... 1 more",2023-04-17T23:44:35.405548994Z " at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)",2023-04-17T23:44:35.405542621Z
" at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)",2023-04-17T23:44:35.405536674Z
" at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)",2023-04-17T23:44:35.405516396Z
" at
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)",2023-04-17T23:44:35.405416352Z
" at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)",2023-04-17T23:44:35.405410491Z
" at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)",2023-04-17T23:44:35.405402143Z
" at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)",2023-04-17T23:44:35.405396413Z
" at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)",2023-04-17T23:44:35.405390525Z
" at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)",2023-04-17T23:44:35.405384806Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)",2023-04-17T23:44:35.405378755Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)",2023-04-17T23:44:35.405372709Z
" at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)",2023-04-17T23:44:35.405359325Z
" at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)",2023-04-17T23:44:35.405353609Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)",2023-04-17T23:44:35.405347958Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)",2023-04-17T23:44:35.405342114Z
" at
org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:102)",2023-04-17T23:44:35.405336302Z
" at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)",2023-04-17T23:44:35.405330321Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)",2023-04-17T23:44:35.405324741Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)",2023-04-17T23:44:35.405319173Z
" at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)",2023-04-17T23:44:35.405313526Z
" at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)",2023-04-17T23:44:35.405307743Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)",2023-04-17T23:44:35.405302059Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)",2023-04-17T23:44:35.405296299Z
" at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)",2023-04-17T23:44:35.405290583Z
" at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)",2023-04-17T23:44:35.405284780Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)",2023-04-17T23:44:35.405278938Z
" at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)",2023-04-17T23:44:35.405273359Z
" at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)",2023-04-17T23:44:35.405267848Z
" at
org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53)",2023-04-17T23:44:35.405262304Z
" at
org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:142)",2023-04-17T23:44:35.405256591Z
" at
org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:209)",2023-04-17T23:44:35.405250814Z{noformat}
In this case prevHandler is the NativeHandler (See
[https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/jdk/internal/misc/Signal.java#L280|Github
JDK Source]) and it throws the exception.
*Possible Solutions:* * Check if prevHandler is an instance of NativeHandler
and do not call it in that case.
* try catch around the invoke of the handler and log a warning/error on
exceptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
