arunkumarucet opened a new pull request, #19797:
URL: https://github.com/apache/pinot/pull/19797
## Problem
Forward index defaults are configurable cluster-wide through
`pinot.forward.index.default.*` —
raw index writer version, target max chunk size, target docs per chunk — but
the **compression
codec is not**. `ForwardIndexType#getDefaultCompressionType` is hardcoded:
```java
if (fieldType == FieldSpec.FieldType.METRIC) {
return ChunkCompressionType.PASS_THROUGH;
} else {
return ChunkCompressionType.LZ4;
}
```
So an operator who wants a different codec across a cluster has to name it
in every table config,
column by column. That is not just verbose — it changes behaviour, because a
codec named in a table
config is treated as explicit, and
`ForwardIndexHandler#shouldChangeRawCompressionType` will then
rewrite forward indexes on reload. A default carries no such cost, by design:
```java
// Note that default compression type (PASS_THROUGH for metric and LZ4 for
dimension) is not considered if the
// compressionType is not explicitly provided in tableConfig. This is to
avoid incorrectly rewriting all the
// forward indexes during segmentReload when the default compressionType
changes.
```
## Change
Adds `pinot.forward.index.default.compression.codec`, applied at startup in
`ServiceStartableUtils#initForwardIndexConfig` next to the existing forward
index defaults, backed by
a settable default on `ForwardIndexConfig` like its neighbours.
- **Metrics keep `PASS_THROUGH`.** They are usually small fixed-width values
where compression costs
more than it saves, so a cluster-wide default should not silently start
compressing them. The new
key replaces the non-metric default only.
- **Table configs still win.** A column naming a codec is unaffected.
- **Existing segments are untouched.** Because this stays a default rather
than a table config value,
a reload will not rewrite forward indexes when it changes — the behaviour
the comment above
describes.
- **An unparseable value is logged and ignored** rather than failing
startup, matching how the
neighbouring forward index defaults behave.
## Testing
`ForwardIndexDefaultCompressionTypeTest` covers the LZ4 default for
non-metric fields, the
`PASS_THROUGH` default for metrics, that a configured default applies to
DIMENSION / DATE_TIME /
COMPLEX, and that it deliberately does **not** apply to METRIC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
