[
https://issues.apache.org/jira/browse/DRILL-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059039#comment-17059039
]
ASF GitHub Bot commented on DRILL-7330:
---------------------------------------
vvysotskyi commented on pull request #2026: DRILL-7330: Implement metadata
usage for all format plugins
URL: https://github.com/apache/drill/pull/2026
# [DRILL-7330](https://issues.apache.org/jira/browse/DRILL-7330): Implement
metadata usage for all format plugins
## Description
This pull request adds support of all format plugins for Metastore.
It contains two parts:
- Making changes to collect metadata for easy group scan
- Allowing usage Metastore metadata for easy group scan during querying data
To reach the first goal was added AnalyzeFileInfoProviderImpl to handle
correctly EasyGroupScan tables in MetastoreAnalyzeTableHandler and specify
correct table type.
Accommodated EVF scan framework to incorporate Metastore-specific additional
implicit columns and handle the case of empty files when metadata should be
returned.
To reach the second goal, EasyGroupScan was updated to extend
AbstractGroupScanWithMetadata.
MetastoreFileTableMetadataProvider based on existing
MetastoreParquetTableMetadataProvider was introduced to provide metastore
metadata for file-based tables.
Accommodated ParquetPushDownFilter rule to work with easy format plugins.
Refactored some code and fixed additional issues.
## Documentation
After this PR is merged, Drill Web Site docs should be updated to reflect
this feature.
## Testing
Added tests to verify the correctness of collected metadata and ensure that
new optimizations work correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Implement metadata usage for text format plugin
> -----------------------------------------------
>
> Key: DRILL-7330
> URL: https://issues.apache.org/jira/browse/DRILL-7330
> Project: Apache Drill
> Issue Type: Sub-task
> Reporter: Arina Ielchiieva
> Assignee: Vova Vysotskyi
> Priority: Major
> Fix For: 1.18.0
>
>
> 1. Change the current group scan to leverage Schema from Metastore;
> 2. Use stats for enabling additional logical planning rules for text format
> plugin. It will enable such optimizations as limit, filter push and so on.
> + add possibility to pass schema through schema file (using path or table
> root), inline.
> + check for other enhancements in analyze command
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
