Aleksander Nek created HDFS-18022:
-------------------------------------

             Summary: JournalNode serves discarded edits from RPC cache after 
rolling-upgrade rollback
                 Key: HDFS-18022
                 URL: https://issues.apache.org/jira/browse/HDFS-18022
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: journal-node
    Affects Versions: 3.6.0
            Reporter: Aleksander Nek


h2. Problem

Journal.discardSegments() removes on-disk segments and rewinds committedTxnId, 
but keeps the old RPC edits cache and highestWrittenTxId/FJM readable 
transaction state. With dfs.ha.tail-edits.in-progress=true, a restarted 
NameNode can replay intentionally discarded operations from warm JournalNodes 
after rolling-upgrade rollback.

h2. Minimal reproduction

# Start a journal epoch and finalize segment [1,3].
# Write transactions [4,6] to a second segment (either finalized or still in 
progress). Verify getJournaledEdits(4,10) returns those edits.
# Successfully discardSegments(4).
# Request getJournaledEdits(4,10) again.

Expected: no transactions at or after 4, and highest available transaction 3.
Actual: highestWrittenTxId remains 6 and the cache still contains the discarded 
edits. In an HA integration reproducer, namespace entries created after the 
rollback image reappear after rollback.

h2. Proposed fix and validation

Invalidate the RPC cache after successful discard, rewind highest/FJM readable 
TXID, and abort a discarded open segment. Snapshot the cache and readable 
boundary together for RPC readers. A rejected mid-segment discard must leave 
the existing cache unchanged.

Before-fix unit tests fail for both finalized and in-progress segments 
(expected highest 3, actual 6). The focused patch adds coverage for both, 
reused TXIDs in a new epoch, and rejected discard. All 19 TestJournal cases 
pass on trunk c815ec8e1e98f20e3f0ae2637665d4adb9196660 (JDK 17, macOS ARM64). A 
separate warm-JN HA integration test also confirms discarded paths stay absent 
without restarting JournalNodes.

This is distinct from HDFS-17710 (edits cached before a failed durable write) 
and HDFS-15485 (JNStorage properties after directory rollback). Here the edits 
were successfully written and are subsequently discarded deliberately. No 
wire/disk format change or new configuration is required.

Contains content generated by OpenAI Codex.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to