[
https://issues.apache.org/jira/browse/DRILL-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059460#comment-17059460
]
ASF GitHub Bot commented on DRILL-7330:
---------------------------------------
paul-rogers commented on pull request #2026: DRILL-7330: Implement metadata
usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392609285
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ReaderSchemaOrchestrator.java
##########
@@ -118,9 +135,59 @@ public void endBatch() {
// Fill in the null and metadata columns.
populateNonDataColumns();
}
+ if (projected) {
+ setProjectMetadata(null);
+ }
rootTuple.setRowCount(tableContainer.getRecordCount());
}
+ /**
+ * Updates {@code PROJECT_METADATA} implicit column value to {@code "FALSE"}
to handle current batch as
+ * a batch with metadata information only for the case when this batch is
first and empty.
+ */
Review comment:
Why? As it turns out, this kind of logic is handled in the
`ScanOperatorExec` and `ReaderState`. See
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/ReaderState.java#L444
I have to apologize, that reader code is quite tricky. It was hard to handle
all the strange things that readers can do such as hitting EOF on open (a CSV
file declared with headers but which is empty), on the first batch (a CVS file
without headers which is empty), and so on.
Can we insert this logic there somehow? Even better, does the existing
handling provide the behavior you are trying to achieve? Maybe this logic was
needed for the old scan which was less graceful at handling odd cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Implement metadata usage for text format plugin
> -----------------------------------------------
>
> Key: DRILL-7330
> URL: https://issues.apache.org/jira/browse/DRILL-7330
> Project: Apache Drill
> Issue Type: Sub-task
> Reporter: Arina Ielchiieva
> Assignee: Vova Vysotskyi
> Priority: Major
> Fix For: 1.18.0
>
>
> 1. Change the current group scan to leverage Schema from Metastore;
> 2. Use stats for enabling additional logical planning rules for text format
> plugin. It will enable such optimizations as limit, filter push and so on.
> + add possibility to pass schema through schema file (using path or table
> root), inline.
> + check for other enhancements in analyze command
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
