[
https://issues.apache.org/jira/browse/FLINK-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-10762:
--------------------------------
Description:
Every call to {{SingleInputGate#getNextBufferOrEvent()}} will also try to
request partitions via {{SingleInputGate#requestPartitions()}}. However, in the
current design, this must be under the {{requestLock}} which is costly and
unnecessary.
I propose a redesign to change that and make the
{{SingleInputGate#requestPartitions()}} a one-time only thing that should be
called *before* going into any loop asking for new records. In that case, we
could keep the lock and ignore any performance-related penalties.
Alternatively, if we had a separate {{unrequestedPartitions}} list aside to
{{inputChannels}} and only go through these when executing
{{SingleInputGate#requestPartitions()}}, we would change {{updateInputChannel}}
to always request the newly created partition and will not have the concurrency
issue between these two methods anymore. This {{unrequestedPartitions}} list
would basically also make {{requestedPartitionsFlag}} obsolete if we, for
example, set it to {{Optional.empty}} after finishing the requests.
was:
Every call to {{SingleInputGate#getNextBufferOrEvent()}} will also try to
request partitions via {{SingleInputGate#requestPartitions()}}. However, in the
current design, this must be under the {{requestLock}} which is costly and
unnecessary.
I propose a redesign to change that and make the
{{SingleInputGate#requestPartitions()}} a one-time only thing that should be
called *before* going into any loop asking for new records. In that case, we
could keep the lock and ignore any performance-related pernalties.
Alternatively, if we had a separate {{unrequestedPartitions}} list aside to
{{inputChannels}} and only go through these when executing
{{SingleInputGate#requestPartitions()}}, we would change {{updateInputChannel}}
to always request the newly created partition and will not have the concurrency
issue between these two methods anymore. This {{unrequestedPartitions}} list
would basically also make {{requestedPartitionsFlag}} obsolete if we, for
example, set it to {{Optional.empty}} after finishing the requests.
> Do not try to request partitions in SingleInputGate per getRecord
> -----------------------------------------------------------------
>
> Key: FLINK-10762
> URL: https://issues.apache.org/jira/browse/FLINK-10762
> Project: Flink
> Issue Type: Improvement
> Components: Network
> Affects Versions: 1.7.0
> Reporter: Nico Kruber
> Priority: Major
> Fix For: 1.8.0
>
>
> Every call to {{SingleInputGate#getNextBufferOrEvent()}} will also try to
> request partitions via {{SingleInputGate#requestPartitions()}}. However, in
> the current design, this must be under the {{requestLock}} which is costly
> and unnecessary.
> I propose a redesign to change that and make the
> {{SingleInputGate#requestPartitions()}} a one-time only thing that should be
> called *before* going into any loop asking for new records. In that case, we
> could keep the lock and ignore any performance-related penalties.
> Alternatively, if we had a separate {{unrequestedPartitions}} list aside to
> {{inputChannels}} and only go through these when executing
> {{SingleInputGate#requestPartitions()}}, we would change
> {{updateInputChannel}} to always request the newly created partition and will
> not have the concurrency issue between these two methods anymore. This
> {{unrequestedPartitions}} list would basically also make
> {{requestedPartitionsFlag}} obsolete if we, for example, set it to
> {{Optional.empty}} after finishing the requests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
