[
https://issues.apache.org/jira/browse/HADOOP-20053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18136326#comment-18136326
]
Cheng Pan commented on HADOOP-20053:
------------------------------------
> align the node-side {{libhadoop.so}} with the Java code version (the root
> trigger in this incident).
[~yumwang] this seems to be a natural precondition for hadoop deployment; I
think `libhadoop.so` does not guarantee ABI/API forward/backward compatibility
> SIGSEGV in jni_NewObject after a failed NativeIO initialization poisons the
> shared FileDescriptor natives cache
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20053
> URL: https://issues.apache.org/jira/browse/HADOOP-20053
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.5.0
> Reporter: Yuming Wang
> Priority: Major
>
> h2. Symptom
> Spark executors with HDFS short-circuit reads enabled die with SIGSEGV. Two
> production hs_err files show the *identical* crash site:
> {code}
> # Problematic frame:
> # V [libjvm.so+0x8932b7] jni_NewObject+0x107
> siginfo: si_addr: 0x0 (SEGV_MAPERR)
> Registers: RSI=0x0, RDX=0x0 <- jclass and jmethodID arguments are NULL
> {code}
> Java stack of the crashing thread:
> {code}
> DomainSocket.receiveFileDescriptors0
> DomainSocket.recvFileInputStreams
> DfsClientShmManager$EndpointShmManager.requestNewShm
> ... ShortCircuitCache.allocShmSlot ... BlockReaderFactory.build
> ... ParquetFileReader.readFooter ... (Spark task reading HDFS)
> {code}
> Sibling executors that do not crash instead log:
> {code}
> java.lang.UnsupportedOperationException: NativeIO is not available.
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm.<init>(ShortCircuitShm.java:470)
> {code}
> h2. Root cause chain (all links verified against production logs)
> *1)* A version skew (newer Java code, older {{libhadoop.so}} on the nodes)
> makes {{NativeIO.POSIX}} initialization fail. The failure is only logged at
> DEBUG via PerformanceAdvisory, so production logs show nothing. With DEBUG
> enabled we captured both directions of the skew:
> {code}
> DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
> java.lang.UnsatisfiedLinkError: 'void
> ...NativeIO$POSIX.initNativePosix(boolean)'
> ...
> DEBUG PerformanceAdvisory: Unable to initialize NativeIO libraries
> java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
> {code}
> *2)* The native {{initNative*}} *error paths call {{fd_deinit(env)}}* ("these
> are all idempotent and safe to call even if the class wasn't inited yet").
> That comment is wrong for the FileDescriptor cache:
> {{fd_class}}/{{fd_constructor}} are *shared with DomainSocket*, which may
> have initialized them already through {{DomainSocket.anchorNative()}} (a
> once-per-JVM call). After the teardown they are NULL forever - {{fd_init()}}
> is never re-invoked.
> *3)* Whether a given JVM crashes or only warns depends on *class
> initialization order*: if {{DomainSocket.<clinit>}} ran before the NativeIO
> failure, the cache is poisoned and the next {{receiveFileDescriptors0}}
> executes {{fd_create()}} -> {{NewObject(env, NULL, NULL)}} -> SIGSEGV. If
> {{DomainSocket.<clinit>}} runs after the failure, its {{fd_init()}}
> re-populates the cache and the same request fails gracefully with "NativeIO
> is not available".
> *4)* {{fd_create()}}/{{fd_get()}} had no NULL guards, so any JVM crashes at
> the same PC deterministically. This is JVM-independent.
> Note: we initially suspected JVM-side GC bugs (G1 pinning) and ruled them out
> - the registers show clean NULLs, not heap corruption.
> h2. Proposed fix
> * {{fd_init()}}: resolve all members into locals first and publish the
> statics atomically; failures leave the statics NULL and are retry-friendly.
> * {{fd_get()}}/{{fd_create()}} (unix and windows): lazily re-initialize the
> cache, then throw IllegalStateException instead of calling JNI with NULL
> members. The existing {{receiveFileDescriptors0}} error path propagates it
> and the client falls back to a regular block reader.
> * {{NativeIO.c}}: remove {{fd_deinit(env)}} from the three init error paths
> (the FileDescriptor cache must not be torn down by unrelated NativeIO
> failures). {{fd_deinit()}} itself is retained for symmetry / explicit
> shutdown use.
> * {{NativeIO.java}}: log the init failure at WARN instead of DEBUG through a
> shared constant message (kept a constant expression so javac folds it into
> the inner classes whose static initializers can run re-entrantly).
> Known tradeoff (documented in source and commit message): {{fd_init()}} is
> intentionally not thread-safe; a cold-cache race can leak at most one global
> ref per racing thread and is caught by the NULL checks, so the worst outcome
> is a spurious IllegalStateException. fd_create() is only reached from
> short-circuit shm setup which is near-serialized (DfsClientShmManager loading
> lock).
> h2. Testing
> No unit test added: reproducing requires a version-skewed libhadoop.so, which
> cannot be constructed in a unit test. Failure is evidenced by the production
> hs_err files above; post-fix behavior (graceful IllegalStateException through
> the existing DomainSocket error path) is verified by inspection; end-to-end
> verification plan: deploy a rebuilt libhadoop.so to the skewed environment
> and observe the WARN plus read fallback instead of the SIGSEGV. Existing
> TestNativeIO / TestDomainSocket suites continue to pass.
> h2. Mitigation for affected deployments
> * {{dfs.client.read.shortcircuit=false}}, or
> * align the node-side {{libhadoop.so}} with the Java code version (the root
> trigger in this incident).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
