[ 
https://issues.apache.org/jira/browse/DRILL-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141583#comment-15141583
 ] 

ASF GitHub Bot commented on DRILL-4287:
---------------------------------------

Github user adeneche commented on a diff in the pull request:

    https://github.com/apache/drill/pull/345#discussion_r52519003
  
    --- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
 ---
    @@ -521,6 +543,25 @@ public void setEndpointByteMap(EndpointByteMap 
byteMap) {
         }
       }
     
    +  private FileSelection
    +  getSelectionFromMetadataCache(DrillFileSystem fs, FileSelection 
selection) throws IOException {
    +    FileStatus metaRootDir = selection.getFirstPath(fs);
    +    Path metaFilePath = new Path(metaRootDir.getPath(), 
Metadata.METADATA_FILENAME);
    +
    +    // get the metadata for the directory by reading the metadata file
    +    Metadata.ParquetTableMetadataBase metadata  = 
Metadata.readBlockMeta(fs, metaFilePath.toString());
    +    List<String> fileNames = Lists.newArrayList();
    +    for (Metadata.ParquetFileMetadata file : metadata.getFiles()) {
    +      fileNames.add(file.getPath());
    +    }
    +    // when creating the file selection, set the selection root in the 
form /a/b instead of
    +    // file:/a/b.  The reason is that the file names above have been 
created in the form
    +    // /a/b/c.parquet and the format of the selection root must match that 
of the file names
    +    // otherwise downstream operations such as partition pruning can break.
    +    final Path metaRootPath = 
Path.getPathWithoutSchemeAndAuthority(metaRootDir.getPath());
    +    return FileSelection.create(selection.getStatuses(fs), fileNames, 
metaRootPath.toString());
    --- End diff --
    
    `FileSelection.create()` expects either a list of statuses or a list of 
filenames, but not both.


> Do lazy reading of parquet metadata cache file
> ----------------------------------------------
>
>                 Key: DRILL-4287
>                 URL: https://issues.apache.org/jira/browse/DRILL-4287
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>    Affects Versions: 1.4.0
>            Reporter: Aman Sinha
>            Assignee: Jinfeng Ni
>
> Currently, the parquet metadata cache file is read eagerly during creation of 
> the DrillTable (as part of ParquetFormatMatcher.isReadable()).  This is not 
> desirable from performance standpoint since there are scenarios where we want 
> to do some up-front optimizations - e.g. directory-based partition pruning 
> (see DRILL-2517) or potential limit 0 optimization etc. - and in such 
> situations it is better to do lazy reading of the metadata cache file.   
> This is a placeholder to perform such delayed reading since it is needed for 
> the aforementioned optimizations.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to