Bill Bejeck created KAFKA-20543:
-----------------------------------
Summary: Drop RocksDB offsets column family on clean shutdown when
downgrading from 4.3
Key: KAFKA-20543
URL: https://issues.apache.org/jira/browse/KAFKA-20543
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 4.3.0
Reporter: Bill Bejeck
Assignee: Bill Bejeck
Fix For: 4.3.1, 4.4.0
KIP-1035 introduced a dedicated offsets column family in each RocksDB instance
to persist changelog offsets per-store. KAFKA-19710 added a downgrade path that
writes a legacy per-task .checkpoint file on
clean shutdown when upgrade.from is set to a pre-4.3 version, but it does not
remove the offsets column family from RocksDB. As a result, older Kafka Streams
versions (which do not declare this column
family at open time) fail with RocksDBException and the application crashes
on startup, making in-flight downgrade impossible.
This ticket covers extending the clean-shutdown downgrade path to also drop
the offsets column family (via db.dropColumnFamily(handle)) in RocksDBStore,
RocksDBTimestampedStore,
RocksDBTimestampedStoreWithHeaders, RocksDBMigratingWindowStoreWithHeaders,
and RocksDBMigratingSessionStoreWithHeaders before the underlying RocksDB
instance is closed, so that older versions can open the
store and resume from the legacy .checkpoint file written by KAFKA-19710.
Once this is in place, the downgrade documentation in
docs/streams/upgrade-guide.md should be updated to describe the supported
in-flight upgrade.from procedure instead of the current wipe-and-restore
workaround.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
