[
https://issues.apache.org/jira/browse/FLINK-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liya Fan updated FLINK-13053:
-----------------------------
Description:
Vectorization is a popular technique in SQL engines today. Compared with
traditional row-based approach, it has some distinct advantages, for example:
* Better use of CPU resources (e.g. SIMD)
* More compact memory layout
* More friendly to compressed data format.
Currently, Flink is based on a row-based SQL engine for both stream and batch
workloads. To enjoy the above benefits, we want to bring vectorization to
Flink. This involves substantial changes to the existing code base. Therefore,
we give a plan to carry out such changes in small, incremental steps, in order
not to affect existing features. We want to apply it to batch workload first.
The details can be found in our proposal
(https://docs.google.com/document/d/1cUHb-_Pbe4NMU3Igwt4tytEmI66jQxev00IL99e2wFY/edit#heading=h.50xdeg1htedb)
.
For the past months, we have developed an initial implementation of the above
ideas. Initial performance evaluations on TPC-H benchmarks show that
substantial performance improvements can be obtained by vectorization (see the
figure below). More details can be found in our proposal
(https://docs.google.com/document/d/1cUHb-_Pbe4NMU3Igwt4tytEmI66jQxev00IL99e2wFY/edit#heading=h.50xdeg1htedb).
!image-2019-07-02-15-26-39-550.png!
Special thanks to @Kurt Young’s team for all the kind help.
Special thanks to @Piotr Nowojski for all the valuable feedback and help
suggestions.
was:
Vectorization is a popular technique in SQL engines today. Compared with
traditional row-based approach, it has some distinct advantages, for example:
* Better use of CPU resources (e.g. SIMD)
* More compact memory layout
* More friendly to compressed data format.
Currently, Flink is based on a row-based SQL engine for both stream and batch
workloads. To enjoy the above benefits, we want to bring vectorization to
Flink. This involves substantial changes to the existing code base. Therefore,
we give a plan to carry out such changes in small, incremental steps, in order
not to affect existing features. We want to apply it to batch workload first.
The details can be found in our proposal.
For the past months, we have developed an initial implementation of the above
ideas. Initial performance evaluations on TPC-H benchmarks show that
substantial performance improvements can be obtained by vectorization (see the
figure below). More details can be found in our proposal.
!image-2019-07-02-15-26-39-550.png!
Special thanks to @Kurt Young’s team for all the kind help.
Special thanks to @Piotr Nowojski for all the valuable feedback and help
suggestions.
> Vectorization Support in Flink
> ------------------------------
>
> Key: FLINK-13053
> URL: https://issues.apache.org/jira/browse/FLINK-13053
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
> Reporter: Liya Fan
> Assignee: Liya Fan
> Priority: Critical
> Attachments: image-2019-07-02-15-26-39-550.png
>
>
> Vectorization is a popular technique in SQL engines today. Compared with
> traditional row-based approach, it has some distinct advantages, for example:
>
> * Better use of CPU resources (e.g. SIMD)
> * More compact memory layout
> * More friendly to compressed data format.
>
> Currently, Flink is based on a row-based SQL engine for both stream and batch
> workloads. To enjoy the above benefits, we want to bring vectorization to
> Flink. This involves substantial changes to the existing code base.
> Therefore, we give a plan to carry out such changes in small, incremental
> steps, in order not to affect existing features. We want to apply it to batch
> workload first. The details can be found in our proposal
> (https://docs.google.com/document/d/1cUHb-_Pbe4NMU3Igwt4tytEmI66jQxev00IL99e2wFY/edit#heading=h.50xdeg1htedb)
> .
>
> For the past months, we have developed an initial implementation of the above
> ideas. Initial performance evaluations on TPC-H benchmarks show that
> substantial performance improvements can be obtained by vectorization (see
> the figure below). More details can be found in our proposal
> (https://docs.google.com/document/d/1cUHb-_Pbe4NMU3Igwt4tytEmI66jQxev00IL99e2wFY/edit#heading=h.50xdeg1htedb).
> !image-2019-07-02-15-26-39-550.png!
> Special thanks to @Kurt Young’s team for all the kind help.
> Special thanks to @Piotr Nowojski for all the valuable feedback and help
> suggestions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
