Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM
threads
......................................................................
IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
JVM threads are being reused across JNI invocations. This means
if the interrupt flag is being set for a thread when the JNI call
returns, the next JNI call might get the interrupt (if it checks for
it).
It can be reproduced manually by adding
Thread.currentThread().interrupt() calls at the end of our methods
(e.g. JniFrontend.convertTable(), JniCatalog.updateCatalog()).
Doing so we can trigger errors we can see in IMPALA-12621, IMPALA-10633,
IMPALA-10924, IMPALA-10540, IMPALA-12261. All these issues have
a stack trace similar to the following:
W20260221 07:53:31.855443 1324125 DataStreamer.java:832] DataStreamer Exception
Java exception follows:
java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:781)
I20260221 07:53:31.863165 1312138 jni-util.cc:321]
fa4015e85de1494e:e1586d1500000000]
org.apache.iceberg.exceptions.RuntimeIOException: Failed to write json to file:
hdfs://localhost:20500/test-warehouse/lineitem_sixblocks_iceberg/metadata/00000-93a5f622-d41e-4ccb-87a4-eb26d9bd7a5c.metadata.json
at
org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:133)
at
org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:115)
at
org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
at
org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
at
org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:173)
at
org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at
org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:75)
at
org.apache.impala.catalog.iceberg.IcebergHiveCatalog.createTable(IcebergHiveCatalog.java:74)
at
org.apache.impala.util.MigrateTableUtil.migrateToIcebergTable(MigrateTableUtil.java:99)
at org.apache.impala.service.Frontend.convertTable(Frontend.java:1004)
at
org.apache.impala.service.JniFrontend.convertTable(JniFrontend.java:243)
I.e., something interrupted the thread when it tried to create a new
JSON file for an Iceberg table. Looking at the HDFS logs, the file
was created successfully:
org.apache.hadoop.hdfs.StateChange: DIR* completeFile: ...metadata.json
There is also no thread interruption logic in the code path of
JniFrontend.convertTable() that could explain the interruption.
So probably the best explanation to the above is:
* JNI thread gets interrupted in a way its interrupt flag remains set
* New JNI call on thread runs Iceberg code that checks interruption
* Above exception being thrown
Testing:
* tested manually by adding Thread.currentThread().interrupt() calls
at the end of JniCatalog/JniFronted methods.
Generated-by: Gemini Pro
Generated-by: Claude Sonnet 4.5
Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Reviewed-on: http://gerrit.cloudera.org:8080/24029
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
M be/src/util/jni-util.cc
M be/src/util/jni-util.h
M fe/src/main/java/org/apache/impala/common/JniUtil.java
3 files changed, 51 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
