[
https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863947#comment-16863947
]
zhijiang edited comment on FLINK-12070 at 6/14/19 11:09 AM:
------------------------------------------------------------
In Yingjie's testing based on session mode, the container is started with large
memory by considering many slots per TM with less number of containers, instead
of less num slots with more num containers.
In common cases for per-job mode, the required container resource should be 1
cpu core with 4GB memory which might have 1GB network buffers then.
I agree with the above assumption Stephan mentioned. We do not expect the
previous SpillableSubpartition occupying network memory because it is difficult
to be consumed multiple times and has many other issues. If the previous
SpillableSubpartition is transformed to spilled mode because of limited network
memory, then the performance comparing with new BlockingPartition might be same
more or less. Even though comparing with pure memory mode in
SpillableSubpartition, as long as we could confirm the new BlockingPartition
implementation writes the data into OS cache and not waits for flushing onto
disk, then the performance should also in the same level.
I remembered the Berkeley DB provides three levels for writing as the tradeoff
in performance and safety. The first is writing into process memory as success.
The second is writing into OS cache as success. And the third is writing and
flushing into disk as success. I am not sure if it exists the similar mechanism
to take the second way in our case.
I would also take a look at the alternative ways Stephan provides.
was (Author: zjwang):
In Yingjie's testing based on session mode, the container is started with large
memory by considering many slots per TM with less number of containers, instead
of less num slots with more num containers.
In common cases for per-job mode, the required container resource should be 1
cpu core with 4GB memory which might have 1GB network buffers then.
I agree with the above assumption Stephan mentioned. We do not expect the
previous SpillableSubpartition occupying network memory because it is difficult
to be consumed multiple times and has many other issues. If the previous
SpillableSubpartition is transformed to spilled mode because of limited network
memory, then the performance comparing with new BlockingPartition might be same
more or less. Even though comparing with pure memory mode in
SpillableSubpartition, as long as we could confirm the new BlockingPartition
implementation writes the data into OS cache and not waits for flushing onto
disk, then the performance should also in the same level.
I remembered the Berkeley DB provides three levels for writing __ as the
tradeoff in performance and safety. The first is writing into process memory as
success. The second is writing into OS cache as success. And the third is
writing and flushing into disk as success. I am not sure if it exists the
similar mechanism to take the second way in our case.
I would also take a look at the alternative ways Stephan provides.
> Make blocking result partitions consumable multiple times
> ---------------------------------------------------------
>
> Key: FLINK-12070
> URL: https://issues.apache.org/jira/browse/FLINK-12070
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.9.0
> Reporter: Till Rohrmann
> Assignee: Stephan Ewen
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Attachments: image-2019-04-18-17-38-24-949.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In order to avoid writing produced results multiple times for multiple
> consumers and in order to speed up batch recoveries, we should make the
> blocking result partitions to be consumable multiple times. At the moment a
> blocking result partition will be released once the consumers has processed
> all data. Instead the result partition should be released once the next
> blocking result has been produced and all consumers of a blocking result
> partition have terminated. Moreover, blocking results should not hold on slot
> resources like network buffers or memory as it is currently the case with
> {{SpillableSubpartitions}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
