[
https://issues.apache.org/jira/browse/FLINK-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17460615#comment-17460615
]
Piotr Nowojski commented on FLINK-25318:
----------------------------------------
Hi all. Thanks for taking up this interesting initiative. So far we - Flink
developers - were not paying much attention to short living jobs/queries, and
often that impacted our decisions in the past. It would be interesting to see
how much demand is there for such use cases in Flink and how much we can
improve Flink in this regard.
I would like to point out two things.
# If we care about something, it should be tested, otherwise feature might be
lost/accidentally removed. Here the feature is performance, and as such I think
ideally (whenever it's feasible) every change that you are doing should be
backed up by a visible benchmark improvement.
# For quite some time we are maintaining [various micro benchmarks
|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847].
Since they are micro benchmarks, some of them are submitting a job with small
bounded input and simply measuring how long does it take to process the bounded
input and those jobs are expected to finish under 1s. This accidentally was
testing OLAP use cases. That was not our purpose, just a side effect of trying
to make benchmarks run quickly. If we detected the performance regression that
was caused by slower startup/initialisation (for example FLINK-23593), most
often we were simply ignoring it or just extending the length of the test. If
OLAP support is something that we want to seriously tackle, it would be great
to have more support from the OLAP devs in investigating and policing this kind
of issues in the future. Help with that would be very much welcome by the
community.
> Improvement of scheduler and execution for Flink OLAP
> -----------------------------------------------------
>
> Key: FLINK-25318
> URL: https://issues.apache.org/jira/browse/FLINK-25318
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
> Affects Versions: 1.14.0, 1.12.5, 1.13.3
> Reporter: Shammon
> Priority: Major
> Labels: Umbrella
> Fix For: 1.15.0
>
>
> We use flink to perform OLAP queries. We launch flink session cluster, submit
> batch jobs to the cluster as OLAP queries, and fetch the jobs' results. OLAP
> jobs are generally small queries which will finish at the seconds or
> milliseconds, and users always submit multiple jobs to the session cluster
> concurrently. We found the qps and latency of jobs will be greatly affected
> when there're tens jobs are running, even when there's little data in each
> query. We will give the result of benchmark for the latest version later.
> After discussed with [~xtsong], and thanks for his advice, we create this
> issue to trace and manager Flink OLAP related improvements. More users and
> developers are welcome and feel free to create Flink OLAP related subtasks
> here, thanks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
