cecemei commented on PR #19791:
URL: https://github.com/apache/druid/pull/19791#issuecomment-5135499596
While reviewing this PR I've some thoughts on catalog, partial load match,
etc... not feedback on the PR itself, just some future work thoughts:
1. Canonical naming layer. Right now, clustered segments and projections
each track their "grouping columns" differently (one uses a richer type-aware
structure, the other a plain list of names). We can give both a shared,
catalog-defined canonical name for these columns, so there's one consistent way
to refer to the column this rule cares about.
2. Fast-check for partial-load matching. Today, matching a rule against a
segment requires re-deriving whether the segment's virtual column actually
matches what the rule is asking for every single time. If a segment is already
known to match the catalog's current schema, we could skip that work and just
say "yes" immediately. If a segment doesn't match what the catalog currently
says, that alone doesn't tell us the segment is wrong, it just means we don't
know yet, so we still have to run the real check.
3. Segment to catalog conformance check. To make idea #2 possible, each
segment would need some way to know "does my schema still match what the
catalog says today?" e.g., a stored fingerprint it can compare, similar to
fingerprints rule matches.
4. Fold into reindex/compaction. If a segment doesn't match the catalog,
we could rewrite/reindex it in the background so it matches again over time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
