Hello Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24989
to look at the new patch set (#3).
Change subject: IMPALA-14778: Fix Close() crashes in Iceberg metadata and
Paimon scans
......................................................................
IMPALA-14778: Fix Close() crashes in Iceberg metadata and Paimon scans
IcebergMetadataScanNode::Close() crashed impalad when Open() did not
complete:
- If the scanner was never created, e.g. because a UNION closed an
unopened child after a cancellation or after reaching its LIMIT,
Close() dereferenced a null 'metadata_scanner_'.
- If the base table was dropped, failed to load or was no longer an
Iceberg table when Open() reloaded it, the Java scanner's
constructor threw, and Close() passed the uninitialized
'jmetadata_scanner_' to DeleteGlobalRef().
This patch initializes the pointer and checks the scanner in Close().
JniFrontend.getCatalogTable() throws TableNotFoundException instead of
returning null, and IcebergMetadataScanner checks the table, so these
cases fail with "Cannot scan metadata table ..." and the load error as
the cause. IcebergMetadataScanNode::GetNext() now calls
ExecDebugAction() like other scan nodes.
PaimonJniScanner and PaimonJniScanNode::Close() had the same two bugs
and are fixed the same way.
IcebergMetadataScanner also leaked the JNI global reference to the
base table, and the Java scanner never closed its Iceberg iterators,
leaving manifest readers open, e.g. for ENTRIES. Close() now deletes
the reference and calls the new Java close(), and the rows iterator
of each task is closed when the scan moves to the next task.
Testing:
- Added EE tests for the crashing cases, a metadata table query to
test_failpoints.py, and FE tests.
- Checked manually that 'JNI global refs' (jcmd Thread.print) no
longer grows with metadata table queries.
Assisted-by: Claude Opus 5.5 <[email protected]>
Change-Id: I0519035ecdca966bdc1a26147a157dc62bad2d8f
---
M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc
M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h
M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.cc
M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/paimon/paimon-jni-scanner.cc
M be/src/exec/paimon/paimon-jni-scanner.h
M be/src/service/frontend.h
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java
A fe/src/test/java/org/apache/impala/util/IcebergMetadataScannerTest.java
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test
M testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
M tests/failure/test_failpoints.py
M tests/query_test/test_iceberg.py
16 files changed, 398 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/24989/3
--
To view, visit http://gerrit.cloudera.org:8080/24989
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0519035ecdca966bdc1a26147a157dc62bad2d8f
Gerrit-Change-Number: 24989
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
