[
https://issues.apache.org/jira/browse/BEAM-8343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Kozlov updated BEAM-8343:
--------------------------------
Description:
The objective is to create a universal way for Beam SQL IO APIs to support
predicate/project push-down.
A proposed way to achieve that is by introducing an interface responsible for
identifying what portion(s) of a Calc can be moved down to IO layer. Also,
adding following methods to a BeamSqlTable interface to pass necessary
parameters to IO APIs:
- BeamSqlTableFilter constructFilter(List<RexNode> filter)
- ProjectSupport supportsProjects()
- PCollection<Row> buildIOReader(PBegin begin, BeamSqlTableFilter filters,
List<String> fieldNames)
ProjectSupport is an enum with the following options:
* NONE
* WITHOUT_FIELD_REORDERING
* WITH_FIELD_REORDERING
Design doc
[link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing].
was:
The objective is to create a universal way for Beam SQL IO APIs to support
predicate/project push-down.
A proposed way to achieve that is by introducing an interface responsible for
identifying what portion(s) of a Calc can be moved down to IO layer. Also,
adding following methods to a BeamSqlTable interface to pass necessary
parameters to IO APIs:
- BeamSqlTableFilter supportsFilter(RexProgram program, RexNode filter)
- ProjectSupport supportsProjects()
- PCollection<Row> buildIOReader(PBegin begin, BeamSqlTableFilter filters,
List<String> fieldNames)
* ProjectSupport is an enum with the following options:
* NONE
* WITHOUT_FIELD_REORDERING
* WITH_FIELD_REORDERING
Design doc
[link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing].
> Add means for IO APIs to support predicate and/or project push-down when
> running SQL pipelines
> ----------------------------------------------------------------------------------------------
>
> Key: BEAM-8343
> URL: https://issues.apache.org/jira/browse/BEAM-8343
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
> Reporter: Kirill Kozlov
> Assignee: Kirill Kozlov
> Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> The objective is to create a universal way for Beam SQL IO APIs to support
> predicate/project push-down.
> A proposed way to achieve that is by introducing an interface responsible
> for identifying what portion(s) of a Calc can be moved down to IO layer.
> Also, adding following methods to a BeamSqlTable interface to pass necessary
> parameters to IO APIs:
> - BeamSqlTableFilter constructFilter(List<RexNode> filter)
> - ProjectSupport supportsProjects()
> - PCollection<Row> buildIOReader(PBegin begin, BeamSqlTableFilter filters,
> List<String> fieldNames)
>
> ProjectSupport is an enum with the following options:
> * NONE
> * WITHOUT_FIELD_REORDERING
> * WITH_FIELD_REORDERING
>
> Design doc
> [link|https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
