Shubham Roy created PHOENIX-8000:
------------------------------------
Summary: Conditional TTL is not enforced at compaction for SYSTEM
tables — expired rows are never physically purged (e.g. SYSTEM.CDC_STREAM)
Key: PHOENIX-8000
URL: https://issues.apache.org/jira/browse/PHOENIX-8000
Project: Phoenix
Issue Type: Bug
Components: phoenix
Affects Versions: 5.3.2, 5.3.1, 5.3.0
Reporter: Shubham Roy
Assignee: Shubham Roy
Fix For: 5.3.3
Summary
A SYSTEM table configured with a conditional TTL never has that TTL applied
during compaction. Expired rows are hidden at read time (masking) but are
never physically removed from the HFiles, so they accumulate indefinitely.
SYSTEM.CDC_STREAM — which is created with a conditional TTL to expire
closed-partition rows after a minimum age — is the concrete instance where this
shows up.
Root cause
In CompactionScanner.NonPartitionedTableTTLTracker, SYSTEM tables are
unconditionally routed to the HBase column-family-descriptor TTL instead of the
table's compiled conditional TTL expression:
boolean isSystemTable = pTable.getType() == PTableType.SYSTEM;
...
if (isSystemTable
||
pTable.getTTLExpression().equals(TTL_EXPRESSION_DEFINED_IN_TABLE_DESCRIPTOR)) {
ColumnFamilyDescriptor cfd = store.getColumnFamilyDescriptor();
ttlExpr = TTLExpressionFactory.create(cfd.getTimeToLive()); // descriptor TTL,
NOT the conditional expr
} else {
ttlExpr = !pTable.getTTLExpression().equals(TTL_EXPRESSION_NOT_DEFINED)
? pTable.getCompiledTTLExpression(pConn) // conditional path — skipped for
SYSTEM
: TTL_EXPRESSION_FOREVER;
}
For a table whose TTL is a conditional expression, the CF-descriptor TTL is
FOREVER (Phoenix stores a numeric TTL in the descriptor only for literal
expressions; conditional TTLs are stored in SYSTEM.CATALOG and the descriptor
is left FOREVER). Consequently, for a SYSTEM table:
- ttlExprForRow resolves to a literal FOREVER, so
RowContext.hasConditionalTTL() is always false;
- therefore postProcessForConditionalTTL(...) — the only place the conditional
expression is evaluated and an expired row is dropped (result.clear()) — is
never reached;
- so expiry is enforced only as read-time masking (TTLRegionScanner), never at
compaction.
Minor compaction / flush also never purge (they use TTL_EXPRESSION_FOREVER by
design), so major compaction is the only place this should happen — and it
doesn't for SYSTEM tables.
Steps to reproduce
1. Create/use a SYSTEM table with a conditional TTL (e.g. SYSTEM.CDC_STREAM,
whose TTL is PARTITION_END_TIME IS NOT NULL AND TO_NUMBER(CURRENT_TIME()) -
TO_NUMBER(PHOENIX_ROW_TIMESTAMP()) >= <min-age>, default
PHOENIX_CDC_STREAM_PARTITION_EXPIRY_MIN_AGE_MS ≈ 30h).
2. Insert rows that satisfy the expiry condition and advance the clock past the
min age.
3. Trigger a major compaction on the table.
4. Observed: the rows are still physically present in the HFiles (a raw/setRaw
scan still returns them); they are only hidden from normal Phoenix reads.
5. Expected: the expired rows are physically removed by the major compaction.
Expected behavior
SYSTEM tables should honor their conditional TTL at (major) compaction —
evaluating the compiled conditional expression and physically purging matching
rows — the same way non-SYSTEM tables do. The isSystemTable short-circuit
should not force a conditional-TTL table onto the (FOREVER) descriptor TTL.
Proposed fix
In NonPartitionedTableTTLTracker, do not route a SYSTEM table to the descriptor
TTL when it has a conditional TTL expression; use the table's compiled
conditional TTL (pTable.getCompiledTTLExpression(...)) so hasConditionalTTL()
becomes true and postProcessForConditionalTTL(...) runs at major compaction.
(Keep the descriptor-TTL path only for tables that genuinely define TTL in the
descriptor / literal TTLs.)
Test
Add an integration test that:
- uses a SYSTEM table with a conditional TTL (SYSTEM.CDC_STREAM),
- writes rows satisfying the condition and ages them past the configured
min-age (~30h),
- runs a major compaction, and
- asserts the expired rows are physically removed (e.g. a raw scan /
all-versions scan no longer returns them), not merely masked.
Impact
Any SYSTEM table with a conditional TTL retains expired rows on disk forever,
causing unbounded growth of the system table and defeating the intent of the
conditional TTL (rows are hidden but never reclaimed).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
