alamb opened a new issue, #10546: URL: https://github.com/apache/datafusion/issues/10546
### Is your feature request related to a problem or challenge? It is common in databases and other analytic system to have additional external "indexes" (perhaps stored in the "metadata catalog", perhaps stored alongside the data files, perhaps embedded in the files, perhaps elsewhere) These indexes are used to speed up queries by "pruning" the files -- specifically evaluating a predicate on the index and then only reading files / portions of files that would pass the filters in the query Implementing such a index requires several three steps: 1. Creating / maintaining the actual index 2. A way to evaluate the index against the query predicate 3. A way to use the index to filter the files that need to be read DataFusion has code already to do 2 (PruningPredicate) and 3 (ParquetExec) but I am not sure how obvious it is to put togeher DataFusion actually also has basic support for 1 (e.g. the ListingTableProvider reads statistics and prunes based on their statistics) We have some version of this in InfluxDB today, and we are in the process of extending it. However, I think the usecase is much more general and could be useful for other systems as well (e.g. full text indexes across parquet, for example). I also think creating an example will help motivate the work from @NGA-TRAN in https://github.com/apache/datafusion/issues/10453 and myself in https://github.com/apache/datafusion/issues/9929 ### Describe the solution you'd like _No response_ ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
