npawar commented on issue #5268: Filter during ingestion URL: https://github.com/apache/incubator-pinot/issues/5268#issuecomment-615382688 Initial thoughts: We need to think about where filtering will be applied. a) On source columns - should we fix that filtering is only on source fields? b) On transformed columns - Should we allow a column to evaluate the transform expression, and then apply filtering? c) Do we allow both? The answer to the above will decide where the filtering gets applied- within RecordExtractor/RecordReader, after Expression Transformer, after all transformers (i.e. after getting default null values, data type transformation etc). The config for this also needs to be designed.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
