Jiayi-Liao commented on a change in pull request #11567: [FLINK-16645] Limit
the maximum backlogs in subpartitions
URL: https://github.com/apache/flink/pull/11567#discussion_r403700830
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java
##########
@@ -375,4 +397,29 @@ void onConsumedSubpartition(int subpartitionIndex) {
private void checkInProduceState() throws IllegalStateException {
checkState(!isFinished, "Partition already finished.");
}
+
+ /**
+ * Check whether all subpartitions' backlogs are less than the
limitation of max backlogs, and make this partition
+ * available again if yes.
+ */
+ public void notifyDecreaseBacklog(int buffersInBacklog) {
+ if (buffersInBacklog == maxBuffersPerChannel) {
+ if (--unavailableSubpartitionsCount == 0) {
+ CompletableFuture<?> toNotify =
availabilityHelper.getUnavailableToResetAvailable();
+ toNotify.complete(null);
+ }
+ }
+ }
+
+ /**
+ * Check whether any subpartition's backlog exceeds the limitation of
max backlogs, and make this partition
+ * unavailabe if yes.
+ */
+ public void notifyIncreaseBacklog(int buffersInBacklog) {
+ if (buffersInBacklog == maxBuffersPerChannel + 1) {
+ if (++unavailableSubpartitionsCount == 1) {
+ availabilityHelper.resetUnavailable();
+ }
+ }
+ }
Review comment:
Mmm.. it seems to make sense, but let me make sure I understand this
correctly.
Firstly we maintain a `int[] subpartitionsBufferCount` in `LocalBufferPool`
to represent the count of each subpartition's buffers.
(1) About adding buffers. We add an new version of `requestMemorySegment()`,
maybe something like `requestMemorySegment(int subpartitionIndex)` and operate
the `subpartitionsBufferCount` inside the new function, which will only be
called by `ResultPartition` without affecting the calling stack of `InputGate`.
(2) About recycling. Create a custom `BufferRecyler` with channel
information like `new CustomBufferRecycler(subpartitionIndex)` to make sure we
know where the buffer comes from when recycling. And this means we need to
create a single `BufferRecycler` instance for every subpartition. If I
understand this part correctly, wouldn't it be adding too many (thousands of)
`BufferRecycler` instances here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
