shubham-roy opened a new pull request, #2613:
URL: https://github.com/apache/phoenix/pull/2613
## Problem
`SYSTEM` tables configured with a **conditional TTL** (e.g.
`SYSTEM.CDC_STREAM`) never
have that TTL applied at compaction. Expired rows are hidden at read time
(masking) but
are **never physically purged** from the HFiles, so they accumulate
indefinitely.
## Root cause
In `CompactionScanner.NonPartitionedTableTTLTracker`, `SYSTEM` tables are
unconditionally
routed to the HBase column-family **descriptor** TTL instead of the
compiled **conditional**
TTL expression:
boolean isSystemTable = pTable.getType() == PTableType.SYSTEM;
if (isSystemTable
||
pTable.getTTLExpression().equals(TTL_EXPRESSION_DEFINED_IN_TABLE_DESCRIPTOR)) {
ttlExpr = TTLExpressionFactory.create(cfd.getTimeToLive()); //
FOREVER for a conditional-TTL table
} else {
ttlExpr = ... pTable.getCompiledTTLExpression(pConn) ...
}
A conditional TTL is stored in `SYSTEM.CATALOG`, not the descriptor, so
the descriptor TTL
stays `FOREVER`. As a result `ttlExprForRow` is a literal `FOREVER`,
`RowContext.hasConditionalTTL()` is always `false`, and
`postProcessForConditionalTTL()`
(the only place expired rows are dropped) never runs. Expiry is enforced
only as read-time
masking, never at compaction.
## Fix
Route a `SYSTEM` table through the compiled conditional-TTL path when its
TTL is a
`ConditionalTTLExpression`; keep the descriptor path for literal /
descriptor-defined TTLs.
boolean isConditionalTTL = pTable.getTTLExpression() instanceof
ConditionalTTLExpression;
if ((isSystemTable && !isConditionalTTL)
||
pTable.getTTLExpression().equals(TTL_EXPRESSION_DEFINED_IN_TABLE_DESCRIPTOR)) {
No behavior change for literal/descriptor-defined TTLs or for
non-conditional SYSTEM tables.
## Testing
New integration test `CDCStreamCompactionTTLIT`
(`phoenix-core/src/it/java/.../end2end/`):
splits a CDC-enabled table to create a **closed** `SYSTEM.CDC_STREAM`
partition row
(`PARTITION_END_TIME IS NOT NULL`), advances a `ManualEnvironmentEdge`
past the partition
expiry window, major-compacts `SYSTEM.CDC_STREAM`, and asserts the
closed-partition row is
**physically removed** (raw HBase row count drops from `total` to `total -
closed`).
Written test-first: fails on current code (`expected:<2> but was:<3>`),
passes with the fix.
mvn -pl phoenix-core -am -Dit.test=CDCStreamCompactionTTLIT
-DfailIfNoTests=false verify
JIRA: https://issues.apache.org/jira/browse/PHOENIX-8000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
