Jiayi-Liao commented on a change in pull request #11567: [FLINK-16645] Limit
the maximum backlogs in subpartitions
URL: https://github.com/apache/flink/pull/11567#discussion_r405233271
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java
##########
@@ -375,4 +397,29 @@ void onConsumedSubpartition(int subpartitionIndex) {
private void checkInProduceState() throws IllegalStateException {
checkState(!isFinished, "Partition already finished.");
}
+
+ /**
+ * Check whether all subpartitions' backlogs are less than the
limitation of max backlogs, and make this partition
+ * available again if yes.
+ */
+ public void notifyDecreaseBacklog(int buffersInBacklog) {
+ if (buffersInBacklog == maxBuffersPerChannel) {
+ if (--unavailableSubpartitionsCount == 0) {
+ CompletableFuture<?> toNotify =
availabilityHelper.getUnavailableToResetAvailable();
+ toNotify.complete(null);
+ }
+ }
+ }
+
+ /**
+ * Check whether any subpartition's backlog exceeds the limitation of
max backlogs, and make this partition
+ * unavailabe if yes.
+ */
+ public void notifyIncreaseBacklog(int buffersInBacklog) {
+ if (buffersInBacklog == maxBuffersPerChannel + 1) {
+ if (++unavailableSubpartitionsCount == 1) {
+ availabilityHelper.resetUnavailable();
+ }
+ }
+ }
Review comment:
@pnowojski Thanks for reminding :). A new idea occurs to me this morning,
maybe it'll help open our thoughts a little bit. Since each subpartition is
synchronized on buffers, what if we let each subpartition know its buffers'
limit and we create the availability future in every subpartition? A
subpartition will be unavailable when number of buffers are increasing and
exceeds the limit, will be available when number of buffers are decreasing. In
`ResultPartition`, there'll be something like this:
```
public boolean addBufferConsumer(BufferConsumer bufferConsumer, int
subpartitionIndex) {
...
...
subpartition.add(bufferConsumer);
if (!subpartition.isAvailabe()) {
subpartitionsFuture.combine(subpartition.availableFuture);
}
...
}
```
But the problem here is we may have hundreds of futures combined together in
the subpartitionsFuture if user sets a low value to the number of max buffers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
