mcvsubbu commented on a change in pull request #5597:
URL: https://github.com/apache/incubator-pinot/pull/5597#discussion_r443831786
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableConfig.java
##########
@@ -79,6 +80,9 @@
@JsonPropertyDescription(value = "upsert related config")
private UpsertConfig _upsertConfig;
+ @JsonPropertyDescription(value = "Config related to table ingestion")
Review comment:
```suggestion
@JsonPropertyDescription(value = "Config related to ingesting data into
the table")
```
Should we consider moving append frequency into this?
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/GenericRow.java
##########
@@ -49,7 +49,17 @@
*/
public class GenericRow {
+ /**
+ * This key is used by a Decoder/RecordReader to handle 1 record to many
records flattening.
+ * If a Decoder/RecordReader produces multiple GenericRows from the given
record, they must be put into the destination GenericRow as a List<GenericRow>
with this key
+ * The segment generation drivers handle this key as a special case and
process the multiple records
+ */
public static final String MULTIPLE_RECORDS_KEY = "$MULTIPLE_RECORDS_KEY$";
+ /**
+ * This key is used by the FilterTransformer to handle filtering out of
records during ingestion
+ * The FilterTransformer puts this key into the GenericRow with value true,
if the record matches the filtering out criteria, based on FilterConfig
+ */
+ public static final String FILTER_RECORD_KEY = "$FILTER_RECORD_KEY$";
Review comment:
I would suggest renaming these as
"$PINOT_INTERNAL_MULTIPLE_RECORDS_KEY$" and
"$PINOT_INTERNAL_SHOULD_INGEST_ROW$". Of course, this will mean that the
decoders that put in the existing MULTIPLE_RECORDS_KEY should be handled right
(if we released this in 0.4.0 without documentation). I don't see any
documentation for it in our docs. Let me know if I am not looking at the right
place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
