Kadir Ozdemir created PHOENIX-7382:
--------------------------------------
Summary: Eliminating index building and treating max lookback as
TTL for CDC Index
Key: PHOENIX-7382
URL: https://issues.apache.org/jira/browse/PHOENIX-7382
Project: Phoenix
Issue Type: Improvement
Reporter: Kadir Ozdemir
Phoenix-7001 introduced a Change Data Capture (CDC) feature by leveraging
Phoenix Max Lookback and Uncovered Global Index on PHOENIX_ROW_TIMESTAMP(). The
max lookback feature retains recent changes to a table and the uncovered index
allows efficient retrieval of the changes in the order of their arrival.
Since the changes are retained only within the max lookback window, a CDC index
does not need to have rows beyond the max lookback window. This means we can
treat the max lookback age as the TTL for CDC indexes.
When a CDC feature is enabled on a table, a CDC index is created with the
building state, and like any other index, it is built from the data table. This
means for every mutation on the data table, an index row mutation is built.
However, the mutations beyond max lookback are not required for CDC. Since
index built is an expensive background operation (it can take days for very
large tables), eliminating index built is desirable when the CDC is enabled on
a table. This means that the CDC feature will track changes that happen after
it is enabled, which is the expected behavior.
The code changes for this improvement will be for
* Creating a CDC index with the active state
* Treating the max lookback age as the TTL for CDC indexes during compaction,
and index verification, repair and rebuild.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
