[
https://issues.apache.org/jira/browse/HIVE-30135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-30135:
----------------------------------
Labels: acid metastore pull-request-available transactions (was: acid
metastore transactions)
> Metastore OOM and empty ACID reads when TXNS loses its gap boundary or Oracle
> issues TXN_ID out of order
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30135
> URL: https://issues.apache.org/jira/browse/HIVE-30135
> Project: Hive
> Issue Type: Bug
> Reporter: Senthil Kumar
> Priority: Major
> Labels: acid, metastore, pull-request-available, transactions
>
> Metastore can crash with OutOfMemoryError, and ACID tables can read empty,
> under two related conditions in the transaction-id machinery.
> h2. Problem 1 — unbounded open-txn gap fill (OOM)
> getOpenTxnsList() synthesises an "open" transaction for every txn id missing
> from TXNS between the highest row older than TXN_OPENTXN_TIMEOUT (the low
> boundary) and the high water mark. Normally TXNS keeps old rows so the gap is
> bounded to ids allocated inside the timeout window. If TXNS loses its row
> older than the window (aggressive cleaner, empty committed/aborted sweep, or
> manual deletion), the low boundary collapses to 0 and every id from 1 up to
> the high water mark is materialised as an OpenTxn. On a busy warehouse that
> is hundreds of millions of objects; a 47 GB heap was exhausted in the field.
> h2. Problem 2 — empty ACID reads (writeIdList ...:1:1:1:)
> Two sub-causes:
> * (a) If empty-TXNS cleanup removes TXNS rows still mapped in TXN_TO_WRITE_ID
> / COMPLETED_TXN_COMPONENTS, max(TXN_ID) over TXNS falls below committed
> writers. A reader's high water mark sits below those writers, their write ids
> are reported invalid/open, and the table reads empty.
> * (b) On Oracle RAC, TXN_ID comes from an identity column or sequence that
> defaults to NOORDER: each instance caches its own block so ids stop
> increasing with time. A later transaction can carry a *smaller* id than an
> earlier committed writer, leaving the writer's id above the reader's HWM —
> the same empty read.
> h2. Fix
> # Seed gap-fill from the highest TXNS row older than the timeout window, and
> refuse to answer once synthesised gaps exceed
> metastore.txn.opentxn.gapfill.max (default 100000), incrementing
> total_num_open_txn_gap_fill_aborted instead of exhausting the heap.
> # Skip gap-fill for ids still present in TXN_TO_WRITE_ID /
> COMPLETED_TXN_COMPONENTS (total_num_open_txn_gap_fill_skipped).
> # Raise open-txn and high-water-mark queries to MAX over TXNS,
> TXN_TO_WRITE_ID and COMPLETED_TXN_COMPONENTS, so cleanup cannot drop HWM
> below a committed writer.
> # cleanEmptyAbortedAndCommittedTxns refuses to delete a TXNS row still mapped
> in TXN_TO_WRITE_ID.
> # Oracle: declare/create TXN_ID ORDER (identity column and sequence), and
> restate ORDER when reseeding; remediate-txn-id-sequence-order.oracle.sql
> detects which generator a schema uses and prints the correct ALTER for
> existing installs.
> h2. How to reproduce
> h3. GetOpenTxns gap-fill OOM
> # Run a Derby metastore (default for unit tests) or Postgres.
> # Open and commit 3 transactions with allocateTableWriteIds so the txn table
> is populated.
> # Remove every TXNS row while leaving TXN_TO_WRITE_ID populated, then open a
> new transaction inside TXN_OPENTXN_TIMEOUT.
> # Call getOpenTxns(): pre-fix it gap-fills the missing ids as OPEN (no cap);
> post-fix it reports total_num_open_txn_gap_fill_aborted and refuses the
> snapshot.
> h3. Empty read from HWM fall + gap-fill
> Covered by TestTxnHandler#testOpenTxnHwmSurvivesEmptyCommittedCleanup and
> #testOpenTxnGapFillSkipsAllocatedWriters.
> h3. Oracle NOORDER
> # Two Oracle RAC instances, default NOORDER identity.
> # Writer commits on instance B with a cached id above a later reader's.
> # SELECT returns no rows; NEXTVAL trace shows non-monotonic ids.
> h2. Tests
> Regression tests cover HWM survival across empty committed cleanup and
> gap-fill skipping allocated writers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
