Jackie-Jiang commented on code in PR #19581:
URL: https://github.com/apache/pinot/pull/19581#discussion_r4067820176
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:
##########
@@ -378,65 +419,71 @@ protected void removeColumnIndices(String column) {
/// Helper method to create the V1 indices (dictionary and forward index)
for a column, returns `true` if the
/// creation succeeds, `false` otherwise.
+ ///
+ /// A derived column whose argument is itself created in this same run is
supported: [#updateDefaultColumns] visits
+ /// the columns in dependency order and [#getColumnMetadata] sees the
metadata written a moment ago. A source
+ /// column with its forward index disabled is supported too, by regenerating
that index for the duration of the
+ /// run. An argument that is genuinely absent from the segment still falls
back to the default value.
protected boolean createColumnV1Indices(String column)
throws Exception {
boolean errorOnFailure = _indexLoadingConfig.isErrorOnColumnBuildFailure();
- IngestionConfig ingestionConfig = _tableConfig.getIngestionConfig();
- if (ingestionConfig != null && ingestionConfig.getTransformConfigs() !=
null) {
- List<TransformConfig> transformConfigs =
ingestionConfig.getTransformConfigs();
- for (TransformConfig transformConfig : transformConfigs) {
- if (transformConfig.getColumnName().equals(column)) {
- String transformFunction = transformConfig.getTransformFunction();
- FunctionEvaluator functionEvaluator =
FunctionEvaluatorFactory.getExpressionEvaluator(transformFunction);
-
- // Check if all arguments exist in the segment
- // TODO: Support chained derived column
- List<String> arguments = functionEvaluator.getArguments();
- List<ColumnMetadata> argumentsMetadata = new
ArrayList<>(arguments.size());
- for (String argument : arguments) {
- ColumnMetadata columnMetadata =
_segmentMetadata.getColumnMetadataFor(argument);
- if (columnMetadata == null) {
- LOGGER.warn("Assigning default value to derived column: {}
because argument: {} does not exist in the "
- + "segment", column, argument);
- createDefaultValueColumnV1Indices(column);
- return true;
- }
- // TODO: Support creation of derived columns from forward index
disabled columns
- if (!_segmentWriter.hasIndexFor(argument,
StandardIndexes.forward())) {
- throw new UnsupportedOperationException(String.format("Operation
not supported! Cannot create a derived "
- + "column %s because argument: %s does not have a
forward index. Enable forward index and "
- + "refresh/backfill the segments to create a derived
column from source column", column,
- argument));
- }
- argumentsMetadata.add(columnMetadata);
- }
-
- // TODO: Support forward index disabled derived column
- if (isForwardIndexDisabled(column)) {
- LOGGER.warn("Skip creating forward index disabled derived column:
{}", column);
- if (errorOnFailure) {
- throw new UnsupportedOperationException(
- String.format("Failed to create forward index disabled
derived column: %s", column));
- }
- return false;
+ TransformConfig transformConfig = getTransformConfig(column);
+ if (transformConfig != null) {
+ String transformFunction = transformConfig.getTransformFunction();
+ FunctionEvaluator functionEvaluator = getFunctionEvaluator(column,
transformFunction);
+
+ // Check if all arguments can be read from the segment
+ List<String> arguments = functionEvaluator.getArguments();
+ List<ColumnMetadata> argumentsMetadata = new
ArrayList<>(arguments.size());
+ for (String argument : arguments) {
+ ColumnMetadata columnMetadata = getColumnMetadata(argument);
+ if (columnMetadata == null) {
+ LOGGER.warn("Assigning default value to derived column: {} because
argument: {} does not exist in the "
+ + "segment", column, argument);
+ createDefaultValueColumnV1Indices(column);
+ recordCreatedColumnMetadata(column);
+ return true;
+ }
+ // The values are read through the argument's forward index. When the
argument has it disabled, regenerate
+ // it from the dictionary and inverted index for the duration of this
run.
+ if (!materializeSourceForwardIndex(argument)) {
+ // Regeneration needs the argument's dictionary and inverted index;
without them the only way to get the
+ // values back is a refresh or back-fill. This used to be an
unconditional failure, so keep failing when
+ // the table asks for it, and otherwise degrade the way a missing
argument already does.
+ if (errorOnFailure) {
+ throw new UnsupportedOperationException(String.format("Operation
not supported! Cannot create a derived "
+ + "column %s because argument: %s does not have a forward
index and it could not be regenerated "
+ + "from its dictionary and inverted index. Enable forward
index and refresh/backfill the segments "
+ + "to create a derived column from source column", column,
argument));
}
+ LOGGER.warn("Assigning default value to derived column: {} because
argument: {} has no forward index and "
+ + "it could not be regenerated", column, argument);
+ createDefaultValueColumnV1Indices(column);
+ recordCreatedColumnMetadata(column);
+ return true;
+ }
+ argumentsMetadata.add(columnMetadata);
Review Comment:
**[P1] Refresh argument metadata after regenerating its forward index**
`columnMetadata` was obtained before `materializeSourceForwardIndex`, but
regeneration can change the metadata needed to read the index. For example,
rebuilding a disabled MV source containing duplicates reduces
`TOTAL_NUMBER_OF_ENTRIES`; the old count then makes
`FixedBitMVForwardIndexReader` calculate incorrect buffer offsets. A supported
reload that enables a formerly disabled source with RAW encoding similarly
opens the raw bytes with the old DICTIONARY reader. Have materialization return
the refreshed column metadata and use it for argument readers, as
`BaseIndexHandler.createForwardIndexIfNeeded` does.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:
##########
@@ -1219,6 +1266,218 @@ private void putDictionaryCompressionStats(String
column, long uncompressedValue
compressionMetadata.applyTo(_segmentProperties, column);
}
+ /// Reorders the actions so that a column is created after every column it
reads that is also being created in
+ /// this run -- the dependency that makes chained derived columns work, and
that also lets a derived column read a
+ /// plain default column added in the same reload. Non-derived columns keep
their original relative order.
+ ///
+ /// A dependency cycle cannot be satisfied in any order, so the columns in
it are emitted last, unordered: each
+ /// still gets created, falling back to its default value because its
argument is not in the segment yet. The
+ /// cycle is logged rather than failing the segment load.
+ private Map<String, DefaultColumnAction> orderByDependencies(Map<String,
DefaultColumnAction> actionMap) {
+ Map<String, List<String>> dependencies = new HashMap<>();
+ for (Map.Entry<String, DefaultColumnAction> entry : actionMap.entrySet()) {
+ if (entry.getValue().isAddAction()) {
+ List<String> arguments = derivedColumnArguments(entry.getKey());
+ if (arguments != null) {
+ List<String> pending = new ArrayList<>(arguments.size());
+ for (String argument : arguments) {
+ // Only columns created in this run constrain the order; anything
already in the segment is readable.
+ DefaultColumnAction argumentAction = actionMap.get(argument);
+ if (argumentAction != null && argumentAction.isAddAction() &&
!argument.equals(entry.getKey())) {
Review Comment:
**[P1] Include UPDATE actions in dependency ordering**
A source being updated is not safely readable before its action completes.
For example, let existing autogenerated INT `q` contain default `1`; change its
schema default to `2` while adding `r=plus(q,1)` and `p=plus(r,1)`. The action
map iterates `p,q,r`, but this traversal pulls `r` ahead of `q` because `q` has
an UPDATE action rather than ADD. It permanently writes `r=2,p=3` instead of
`r=3,p=4`. Before this change, `q` updated before `r`. Include ADD and UPDATE
actions when constructing dependency edges, with V1/V3 coverage combining a
source-default update and a new chain.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:
##########
@@ -1219,6 +1266,218 @@ private void putDictionaryCompressionStats(String
column, long uncompressedValue
compressionMetadata.applyTo(_segmentProperties, column);
}
+ /// Reorders the actions so that a column is created after every column it
reads that is also being created in
+ /// this run -- the dependency that makes chained derived columns work, and
that also lets a derived column read a
+ /// plain default column added in the same reload. Non-derived columns keep
their original relative order.
+ ///
+ /// A dependency cycle cannot be satisfied in any order, so the columns in
it are emitted last, unordered: each
+ /// still gets created, falling back to its default value because its
argument is not in the segment yet. The
+ /// cycle is logged rather than failing the segment load.
+ private Map<String, DefaultColumnAction> orderByDependencies(Map<String,
DefaultColumnAction> actionMap) {
+ Map<String, List<String>> dependencies = new HashMap<>();
+ for (Map.Entry<String, DefaultColumnAction> entry : actionMap.entrySet()) {
+ if (entry.getValue().isAddAction()) {
+ List<String> arguments = derivedColumnArguments(entry.getKey());
+ if (arguments != null) {
+ List<String> pending = new ArrayList<>(arguments.size());
+ for (String argument : arguments) {
+ // Only columns created in this run constrain the order; anything
already in the segment is readable.
+ DefaultColumnAction argumentAction = actionMap.get(argument);
+ if (argumentAction != null && argumentAction.isAddAction() &&
!argument.equals(entry.getKey())) {
+ pending.add(argument);
+ }
+ }
+ if (!pending.isEmpty()) {
+ dependencies.put(entry.getKey(), pending);
+ }
+ }
+ }
+ }
+ if (dependencies.isEmpty()) {
+ return actionMap;
+ }
+
+ Map<String, DefaultColumnAction> ordered = new
LinkedHashMap<>(actionMap.size());
+ Set<String> visiting = new LinkedHashSet<>();
+ for (String column : actionMap.keySet()) {
+ visitForOrdering(column, actionMap, dependencies, visiting, ordered);
+ }
+ // Columns dropped by a cycle still have to be created; they land last and
take the default-value path.
+ for (Map.Entry<String, DefaultColumnAction> entry : actionMap.entrySet()) {
+ ordered.putIfAbsent(entry.getKey(), entry.getValue());
+ }
+ return ordered;
+ }
+
+ private void visitForOrdering(String column, Map<String,
DefaultColumnAction> actionMap,
+ Map<String, List<String>> dependencies, Set<String> visiting,
Map<String, DefaultColumnAction> ordered) {
+ if (ordered.containsKey(column)) {
+ return;
+ }
+ if (!visiting.add(column)) {
+ LOGGER.warn("Derived columns form a dependency cycle: {}. They will be
assigned default values instead of "
+ + "being derived; break the cycle in the transform configs.",
visiting);
+ return;
+ }
+ try {
+ for (String dependency : dependencies.getOrDefault(column, List.of())) {
+ visitForOrdering(dependency, actionMap, dependencies, visiting,
ordered);
+ }
+ // A dependency stuck in a cycle is absent from `ordered`; emitting this
column anyway is safe because the
+ // missing argument makes it take the default-value path.
+ ordered.put(column, actionMap.get(column));
Review Comment:
**[P2] Explicitly apply the fallback to every detected cycle member**
Returning when a node is already in `visiting` does not stop the enclosing
calls from adding the cycle members here. For `a=plus(b,1), b=plus(a,1)`, the
traversal emits one first, which defaults, and the other then reads that newly
created value and evaluates its transform instead of receiving its own default.
This contradicts the documented fallback and differs from the previous
behavior. Track cycle members and explicitly default them, or reject the cycle.
The new cycle test only checks cardinality; asserting each column's actual
default value would expose this.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:
##########
@@ -378,65 +419,71 @@ protected void removeColumnIndices(String column) {
/// Helper method to create the V1 indices (dictionary and forward index)
for a column, returns `true` if the
/// creation succeeds, `false` otherwise.
+ ///
+ /// A derived column whose argument is itself created in this same run is
supported: [#updateDefaultColumns] visits
+ /// the columns in dependency order and [#getColumnMetadata] sees the
metadata written a moment ago. A source
+ /// column with its forward index disabled is supported too, by regenerating
that index for the duration of the
+ /// run. An argument that is genuinely absent from the segment still falls
back to the default value.
protected boolean createColumnV1Indices(String column)
throws Exception {
boolean errorOnFailure = _indexLoadingConfig.isErrorOnColumnBuildFailure();
- IngestionConfig ingestionConfig = _tableConfig.getIngestionConfig();
- if (ingestionConfig != null && ingestionConfig.getTransformConfigs() !=
null) {
- List<TransformConfig> transformConfigs =
ingestionConfig.getTransformConfigs();
- for (TransformConfig transformConfig : transformConfigs) {
- if (transformConfig.getColumnName().equals(column)) {
- String transformFunction = transformConfig.getTransformFunction();
- FunctionEvaluator functionEvaluator =
FunctionEvaluatorFactory.getExpressionEvaluator(transformFunction);
-
- // Check if all arguments exist in the segment
- // TODO: Support chained derived column
- List<String> arguments = functionEvaluator.getArguments();
- List<ColumnMetadata> argumentsMetadata = new
ArrayList<>(arguments.size());
- for (String argument : arguments) {
- ColumnMetadata columnMetadata =
_segmentMetadata.getColumnMetadataFor(argument);
- if (columnMetadata == null) {
- LOGGER.warn("Assigning default value to derived column: {}
because argument: {} does not exist in the "
- + "segment", column, argument);
- createDefaultValueColumnV1Indices(column);
- return true;
- }
- // TODO: Support creation of derived columns from forward index
disabled columns
- if (!_segmentWriter.hasIndexFor(argument,
StandardIndexes.forward())) {
- throw new UnsupportedOperationException(String.format("Operation
not supported! Cannot create a derived "
- + "column %s because argument: %s does not have a
forward index. Enable forward index and "
- + "refresh/backfill the segments to create a derived
column from source column", column,
- argument));
- }
- argumentsMetadata.add(columnMetadata);
- }
-
- // TODO: Support forward index disabled derived column
- if (isForwardIndexDisabled(column)) {
- LOGGER.warn("Skip creating forward index disabled derived column:
{}", column);
- if (errorOnFailure) {
- throw new UnsupportedOperationException(
- String.format("Failed to create forward index disabled
derived column: %s", column));
- }
- return false;
+ TransformConfig transformConfig = getTransformConfig(column);
+ if (transformConfig != null) {
+ String transformFunction = transformConfig.getTransformFunction();
+ FunctionEvaluator functionEvaluator = getFunctionEvaluator(column,
transformFunction);
+
+ // Check if all arguments can be read from the segment
+ List<String> arguments = functionEvaluator.getArguments();
+ List<ColumnMetadata> argumentsMetadata = new
ArrayList<>(arguments.size());
+ for (String argument : arguments) {
+ ColumnMetadata columnMetadata = getColumnMetadata(argument);
Review Comment:
**[P1] Preserve intermediate transform values when evaluating chains**
Reading a newly created dependency from its stored index changes the result
relative to normal ingestion when the intermediate value needs type conversion.
For example, with source `a=1`, INT `b=divide(a,2)`, and DOUBLE `c=plus(b,1)`,
ingestion evaluates both expressions before `DataTypeTransformer`, so it stores
`b=0,c=1.5`. This reload path first stores/coerces `b`, then reads `0` to
produce `c=1.0`. Evaluate the newly created chain using intermediate expression
results before storage conversion, and add a regression comparing ingestion and
reload for this case.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:
##########
@@ -1219,6 +1266,218 @@ private void putDictionaryCompressionStats(String
column, long uncompressedValue
compressionMetadata.applyTo(_segmentProperties, column);
}
+ /// Reorders the actions so that a column is created after every column it
reads that is also being created in
+ /// this run -- the dependency that makes chained derived columns work, and
that also lets a derived column read a
+ /// plain default column added in the same reload. Non-derived columns keep
their original relative order.
+ ///
+ /// A dependency cycle cannot be satisfied in any order, so the columns in
it are emitted last, unordered: each
+ /// still gets created, falling back to its default value because its
argument is not in the segment yet. The
+ /// cycle is logged rather than failing the segment load.
+ private Map<String, DefaultColumnAction> orderByDependencies(Map<String,
DefaultColumnAction> actionMap) {
+ Map<String, List<String>> dependencies = new HashMap<>();
+ for (Map.Entry<String, DefaultColumnAction> entry : actionMap.entrySet()) {
+ if (entry.getValue().isAddAction()) {
+ List<String> arguments = derivedColumnArguments(entry.getKey());
+ if (arguments != null) {
+ List<String> pending = new ArrayList<>(arguments.size());
+ for (String argument : arguments) {
+ // Only columns created in this run constrain the order; anything
already in the segment is readable.
+ DefaultColumnAction argumentAction = actionMap.get(argument);
+ if (argumentAction != null && argumentAction.isAddAction() &&
!argument.equals(entry.getKey())) {
+ pending.add(argument);
+ }
+ }
+ if (!pending.isEmpty()) {
+ dependencies.put(entry.getKey(), pending);
+ }
+ }
+ }
+ }
+ if (dependencies.isEmpty()) {
+ return actionMap;
+ }
+
+ Map<String, DefaultColumnAction> ordered = new
LinkedHashMap<>(actionMap.size());
+ Set<String> visiting = new LinkedHashSet<>();
+ for (String column : actionMap.keySet()) {
+ visitForOrdering(column, actionMap, dependencies, visiting, ordered);
+ }
+ // Columns dropped by a cycle still have to be created; they land last and
take the default-value path.
+ for (Map.Entry<String, DefaultColumnAction> entry : actionMap.entrySet()) {
+ ordered.putIfAbsent(entry.getKey(), entry.getValue());
+ }
+ return ordered;
+ }
+
+ private void visitForOrdering(String column, Map<String,
DefaultColumnAction> actionMap,
+ Map<String, List<String>> dependencies, Set<String> visiting,
Map<String, DefaultColumnAction> ordered) {
+ if (ordered.containsKey(column)) {
+ return;
+ }
+ if (!visiting.add(column)) {
+ LOGGER.warn("Derived columns form a dependency cycle: {}. They will be
assigned default values instead of "
+ + "being derived; break the cycle in the transform configs.",
visiting);
+ return;
+ }
+ try {
+ for (String dependency : dependencies.getOrDefault(column, List.of())) {
+ visitForOrdering(dependency, actionMap, dependencies, visiting,
ordered);
+ }
+ // A dependency stuck in a cycle is absent from `ordered`; emitting this
column anyway is safe because the
+ // missing argument makes it take the default-value path.
+ ordered.put(column, actionMap.get(column));
+ } finally {
+ visiting.remove(column);
+ }
+ }
+
+ /// The arguments of the transform function that derives this column, or
null when the column is not derived.
+ @Nullable
+ private List<String> derivedColumnArguments(String column) {
+ TransformConfig transformConfig = getTransformConfig(column);
+ if (transformConfig == null) {
+ return null;
+ }
+ try {
+ return getFunctionEvaluator(column,
transformConfig.getTransformFunction()).getArguments();
+ } catch (Exception e) {
+ // An unparseable transform function fails later in
createColumnV1Indices with the full context; ordering just
+ // treats the column as dependency-free.
+ return null;
+ }
+ }
+
+ @Nullable
+ private TransformConfig getTransformConfig(String column) {
+ if (_transformConfigsByColumn == null) {
+ buildTransformConfigIndex();
+ }
+ return _transformConfigsByColumn.get(column);
+ }
+
+ /// Indexes the transform configs by the column they produce, and collects
every column their expressions read.
+ /// An unparseable expression is left out of the argument set; it fails
later in [#createColumnV1Indices] with the
+ /// transform function in the message.
+ private void buildTransformConfigIndex() {
+ _transformConfigsByColumn = new HashMap<>();
+ _derivedColumnArgumentNames = new HashSet<>();
+ IngestionConfig ingestionConfig = _tableConfig.getIngestionConfig();
+ List<TransformConfig> transformConfigs =
+ ingestionConfig != null ? ingestionConfig.getTransformConfigs() : null;
+ if (transformConfigs == null) {
+ return;
+ }
+ for (TransformConfig transformConfig : transformConfigs) {
+ _transformConfigsByColumn.putIfAbsent(transformConfig.getColumnName(),
transformConfig);
+ try {
+ _derivedColumnArgumentNames.addAll(
+ getFunctionEvaluator(transformConfig.getColumnName(),
transformConfig.getTransformFunction())
+ .getArguments());
+ } catch (Exception e) {
+ LOGGER.debug("Could not parse the transform function of column: {}
while indexing the transform configs",
+ transformConfig.getColumnName(), e);
+ }
+ }
+ }
+
+ /// The parsed expression for a derived column. Cached because ordering the
columns and deriving them both need it.
+ /// A parse failure propagates and is not cached, so the caller still sees
it.
+ private FunctionEvaluator getFunctionEvaluator(String column, String
transformFunction) {
+ return _functionEvaluators.computeIfAbsent(column,
+ k ->
FunctionEvaluatorFactory.getExpressionEvaluator(transformFunction));
+ }
+
+ /// Metadata for a column, including one created earlier in this same run
(see [#_createdColumnMetadata]).
+ @Nullable
+ private ColumnMetadata getColumnMetadata(String column) {
+ ColumnMetadata created = _createdColumnMetadata.get(column);
+ return created != null ? created :
_segmentMetadata.getColumnMetadataFor(column);
+ }
+
+ /// Records the metadata just written for a created column so later columns
in this run can read it. Only columns
+ /// that a transform function actually reads are read back: a schema
evolution adding hundreds of plain default
+ /// columns would otherwise re-parse the metadata of every one of them for
nothing.
+ private void recordCreatedColumnMetadata(String column) {
+ if (_transformConfigsByColumn == null) {
+ buildTransformConfigIndex();
+ }
+ if (!_derivedColumnArgumentNames.contains(column)) {
+ return;
+ }
+ try {
+ _createdColumnMetadata.put(column,
+ ColumnMetadataImpl.fromPropertiesConfiguration(_segmentProperties,
_segmentMetadata.getTotalDocs(), column));
+ } catch (Exception e) {
+ // Only chained reads need this; a failure here must not fail the column
that was created successfully.
+ LOGGER.warn("Could not read back the metadata of newly created column:
{}; a derived column reading it in this "
+ + "same reload will fall back to its default value", column, e);
+ }
+ }
+
+ /// Makes a forward index available for a source column that has it
disabled, by regenerating it from the
+ /// dictionary and inverted index. The regenerated index is temporary:
[#removeTemporaryForwardIndexes] drops it
+ /// once every default column is created, so the column keeps the shape its
config asks for.
+ ///
+ /// Returns false when regeneration is not possible -- no segment directory
(the metadata-only path), or the
+ /// dictionary / inverted index the rebuild needs is missing. The caller
then falls back to the default value
+ /// rather than failing the load.
+ private boolean materializeSourceForwardIndex(String column) {
+ if (_segmentWriter.hasIndexFor(column, StandardIndexes.forward())) {
+ return true;
+ }
+ if (_segmentDirectory == null) {
+ return false;
+ }
+ if (!_segmentWriter.hasIndexFor(column, StandardIndexes.dictionary())
+ || !_segmentWriter.hasIndexFor(column, StandardIndexes.inverted())) {
+ LOGGER.warn("Cannot regenerate the forward index of column: {} to derive
from it: it needs both a dictionary "
+ + "({}) and an inverted index ({})", column,
+ _segmentWriter.hasIndexFor(column, StandardIndexes.dictionary()) ?
"present" : "missing",
+ _segmentWriter.hasIndexFor(column, StandardIndexes.inverted()) ?
"present" : "missing");
+ return false;
+ }
+ FieldIndexConfigs fieldIndexConfigs =
_indexLoadingConfig.getFieldIndexConfig(column);
+ if (fieldIndexConfigs == null) {
+ return false;
+ }
+ try {
+ LOGGER.info("Temporarily regenerating the forward index of
forward-index-disabled column: {} so a derived "
+ + "column can read it", column);
+ // The creator writes metadata.properties itself
(SegmentMetadataUtils#updateMetadataProperties). Our own
+ // in-memory copy was loaded before this point and is saved at the end
of the run, so flush the columns
+ // created so far to disk and re-read afterwards -- otherwise the save
would discard the creator's writes.
+ if (_segmentProperties != null) {
+ SegmentMetadataUtils.savePropertiesConfiguration(_segmentProperties,
_segmentMetadata.getIndexDir());
+ }
+ new
InvertedIndexAndDictionaryBasedForwardIndexCreator(_segmentDirectory,
_segmentWriter, _tableConfig, column,
Review Comment:
**[P1] Supply newly created source metadata to the regenerator**
When the same reload adds a plain MV default column with forward index
disabled and a derived column reading it, the source has dictionary/inverted
indexes and is visible through `_createdColumnMetadata`. However, this creator
looks it up through `_segmentDirectory.getSegmentMetadata()` and dereferences
null in its constructor because that metadata does not contain the new column.
The catch then makes the dependent column take its default value, or fails
loading when `errorOnColumnBuildFailure` is enabled. Flushing properties above
is insufficient: the directory metadata is stale and the segment's column lists
are only updated after this loop. Pass the current source metadata into
regeneration, or create a temporary forward index for the new default source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
