npawar opened a new pull request #5597: URL: https://github.com/apache/incubator-pinot/pull/5597
## Description Added support for filtering during ingestion Design doc: https://docs.google.com/document/d/1Cahnas3nh0XErETH0KHLaecN6xCnRVYWNKO3rDn7qcI/edit?usp=sharing Issue: https://github.com/apache/incubator-pinot/issues/5268 **TLDR** - New config added: ``` tableConfig: { tableName: ..., tableType: ..., ingestionConfig: { filterConfig: { filterFunction: “<expression>” } } } ``` The expressions allowed here will be within the scope of the transform functions support that we have in Pinot today i.e. Groovy expressions, or any inbuilt functions. - Filtering has been modeled as a RecordTransformer called `FilterTransformer`. - In order to filter the record, we will use a special key `$FILTER_RECORD_KEY$` Inside the `FilterTransformer`, if `filterFunction` evaluates to true, this key will be put into the `GenericRow`, with value `true`. ## Release Notes Added support for filtering during ingestion ## Documentation TBD ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
