Senthil Kumar created HIVE-30135:
------------------------------------
Summary: Metastore OOM and empty ACID reads when TXNS loses its
gap boundary or Oracle issues TXN_ID out of order
Key: HIVE-30135
URL: https://issues.apache.org/jira/browse/HIVE-30135
Project: Hive
Issue Type: Bug
Reporter: Senthil Kumar
Metastore can crash with OutOfMemoryError, and ACID tables can read empty,
under two related conditions in the transaction-id machinery.
h2. Problem 1 — unbounded open-txn gap fill (OOM)
getOpenTxnsList() synthesises an "open" transaction for every txn id missing
from TXNS between the highest row older than TXN_OPENTXN_TIMEOUT (the low
boundary) and the high water mark. Normally TXNS keeps old rows so the gap is
bounded to ids allocated inside the timeout window. If TXNS loses its row older
than the window (aggressive cleaner, empty committed/aborted sweep, or manual
deletion), the low boundary collapses to 0 and every id from 1 up to the high
water mark is materialised as an OpenTxn. On a busy warehouse that is hundreds
of millions of objects; a 47 GB heap was exhausted in the field.
h2. Problem 2 — empty ACID reads (writeIdList ...:1:1:1:)
Two sub-causes:
* (a) If empty-TXNS cleanup removes TXNS rows still mapped in TXN_TO_WRITE_ID /
COMPLETED_TXN_COMPONENTS, max(TXN_ID) over TXNS falls below committed writers.
A reader's high water mark sits below those writers, their write ids are
reported invalid/open, and the table reads empty.
* (b) On Oracle RAC, TXN_ID comes from an identity column or sequence that
defaults to NOORDER: each instance caches its own block so ids stop increasing
with time. A later transaction can carry a *smaller* id than an earlier
committed writer, leaving the writer's id above the reader's HWM — the same
empty read.
h2. Fix
# Seed gap-fill from the highest TXNS row older than the timeout window, and
refuse to answer once synthesised gaps exceed metastore.txn.opentxn.gapfill.max
(default 100000), incrementing total_num_open_txn_gap_fill_aborted instead of
exhausting the heap.
# Skip gap-fill for ids still present in TXN_TO_WRITE_ID /
COMPLETED_TXN_COMPONENTS (total_num_open_txn_gap_fill_skipped).
# Raise open-txn and high-water-mark queries to MAX over TXNS, TXN_TO_WRITE_ID
and COMPLETED_TXN_COMPONENTS, so cleanup cannot drop HWM below a committed
writer.
# cleanEmptyAbortedAndCommittedTxns refuses to delete a TXNS row still mapped
in TXN_TO_WRITE_ID.
# Oracle: declare/create TXN_ID ORDER (identity column and sequence), and
restate ORDER when reseeding; remediate-txn-id-sequence-order.oracle.sql
detects which generator a schema uses and prints the correct ALTER for existing
installs.
h2. How to reproduce
h3. GetOpenTxns gap-fill OOM
# Run a Derby metastore (default for unit tests) or Postgres.
# Open and commit 3 transactions with allocateTableWriteIds so the txn table is
populated.
# Remove every TXNS row while leaving TXN_TO_WRITE_ID populated, then open a
new transaction inside TXN_OPENTXN_TIMEOUT.
# Call getOpenTxns(): pre-fix it gap-fills the missing ids as OPEN (no cap);
post-fix it reports total_num_open_txn_gap_fill_aborted and refuses the
snapshot.
h3. Empty read from HWM fall + gap-fill
Covered by TestTxnHandler#testOpenTxnHwmSurvivesEmptyCommittedCleanup and
#testOpenTxnGapFillSkipsAllocatedWriters.
h3. Oracle NOORDER
# Two Oracle RAC instances, default NOORDER identity.
# Writer commits on instance B with a cached id above a later reader's.
# SELECT returns no rows; NEXTVAL trace shows non-monotonic ids.
h2. Tests
Regression tests cover HWM survival across empty committed cleanup and gap-fill
skipping allocated writers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
