[
https://issues.apache.org/jira/browse/IMPALA-10332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237075#comment-17237075
]
ASF subversion and git services commented on IMPALA-10332:
----------------------------------------------------------
Commit 301d7ebe75d2e74f361f048020181b51dce9f216 in impala's branch
refs/heads/master from Daniel Becker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=301d7eb ]
IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.
List all file formats that a HdfsScanNode needs to process in any
fragment instance. It is possible that some file formats will not be
needed in all fragment instances.
This is a step towards sharing codegen between different impala
backends. Using the file formats provided in the thrift file, a backend
can codegen code for file formats that are not needed in its own process
but are needed in other fragment instances running on other backends,
and the resulting binary can be shared between multiple backends.
Codegenning for file formats will be done based on the thrift message
and not on what is needed for the actual backend. This leads to some
extra work in case a file format is not needed for the current backend
and codegen sharing is not available (at this point it is not
implemented). However, the overall number of such cases is low.
Also adding the file formats to the node's explain string at level 3.
Testing:
- Added tests to verify that the file formats are present in the
explain string at level 3.
Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d
Reviewed-on: http://gerrit.cloudera.org:8080/16728
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Reviewed-by: Csaba Ringhofer <csringho...@cloudera.com>
> Add file formats to HdfsScanNode's thrift representation and codegen for those
> ------------------------------------------------------------------------------
>
> Key: IMPALA-10332
> URL: https://issues.apache.org/jira/browse/IMPALA-10332
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
> Reporter: Daniel Becker
> Assignee: Daniel Becker
> Priority: Major
>
> List all file formats that a HdfsScanNode needs to process in any fragment
> instance. It is possible that some file formats will not be needed in all
> fragment instances.
> This is a step towards sharing codegen between different impala backends.
> Using the file formats provided in the thrift file, a backend can codegen
> code for file formats that are not needed in its own process but are needed
> in other fragment instances running on other backends, and the resulting
> binary can be shared between multiple backends.
> Codegenning for file formats will be done based on the thrift message and not
> on what is needed for the actual backend. This leads to some extra work in
> case a file format is not needed for the current backend and codegen sharing
> is not available (at this point it is not implemented). However, the overall
> number of such cases is low.
> Also adding the file formats to the node's explain string.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
