[
https://issues.apache.org/jira/browse/KAFKA-20753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Bejeck updated KAFKA-20753:
--------------------------------
Description:
~With {{processing.guarantee=exactly_once_v2}} and
{{enable.transactional.statestores=true,}} a rebalance that adds a StreamThread
can transiently co-locate the same task's active and standby on one
instance (different threads). The freshly-assigned active opens the
persistent RocksDB store while the live sibling standby still holds it open, so
the store-open status check rejects it and the task enters a {{TaskCorrupted →
re-init → corrupt}} loop.~
When {{{}enable.transactional.statestores=true{}}}, the KIP-1035 open/closed
status marker written to the RocksDB offsets column family on {{open()}}
and {{close()}} was being routed through {{{}TransactionalDBAccessor.put(){}}},
which stages writes in the in-memory {{{}RocksDBTransactionBuffer{}}}. The
buffer is discarded without committing on store close, so the CLOSED
marker was silently lost. The on-disk status remained permanently at
OPEN, causing every subsequent {{open()}} under EOS
({{{}ignoreInvalidState=false{}}}) to throw
{{{}ProcessorStateException("Invalid state during store open..."){}}}, which
escalated to
{{{}TaskCorruptedException{}}}. Because the wipe-and-restore cycle would
reproduce the same condition, this created an unrecoverable loop
was:
With {{processing.guarantee=exactly_once_v2}} and
{{enable.transactional.statestores=true,}} a rebalance that adds a StreamThread
can transiently co-locate the same task's active and standby on one
instance (different threads). The freshly-assigned active opens the
persistent RocksDB store while the live sibling standby still holds it open, so
the store-open status check rejects it and the task enters a {{TaskCorrupted →
re-init → corrupt}} loop.
> Transactional RocksDB store fails to reopen with transactional status marker
> in EOS
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-20753
> URL: https://issues.apache.org/jira/browse/KAFKA-20753
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.4.0
> Reporter: Bill Bejeck
> Assignee: Nicholas Telford
> Priority: Blocker
> Fix For: 4.4.0
>
>
> ~With {{processing.guarantee=exactly_once_v2}} and
> {{enable.transactional.statestores=true,}} a rebalance that adds a
> StreamThread can transiently co-locate the same task's active and standby on
> one
> instance (different threads). The freshly-assigned active opens the
> persistent RocksDB store while the live sibling standby still holds it open,
> so the store-open status check rejects it and the task enters a
> {{TaskCorrupted → re-init → corrupt}} loop.~
> When {{{}enable.transactional.statestores=true{}}}, the KIP-1035 open/closed
> status marker written to the RocksDB offsets column family on {{open()}}
> and {{close()}} was being routed through
> {{{}TransactionalDBAccessor.put(){}}},
> which stages writes in the in-memory {{{}RocksDBTransactionBuffer{}}}. The
> buffer is discarded without committing on store close, so the CLOSED
> marker was silently lost. The on-disk status remained permanently at
> OPEN, causing every subsequent {{open()}} under EOS
> ({{{}ignoreInvalidState=false{}}}) to throw
> {{{}ProcessorStateException("Invalid state during store open..."){}}}, which
> escalated to
> {{{}TaskCorruptedException{}}}. Because the wipe-and-restore cycle would
> reproduce the same condition, this created an unrecoverable loop
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
