Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

reswqa merged PR #23851: URL: https://github.com/apache/flink/pull/23851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1505305577 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java: ## @@ -132,15 +149,73 @@ private boolean shouldContinueReq

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1505305235 ## flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/table/LegacyTableSinkITCase.scala: ## @@ -277,6 +277,7 @@ class Lega

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1505241156 ## flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java: ## @@ -73,32 +69,32 @@ public static Tuple3 getMinMaxNetworkBuffersPerResul

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

reswqa commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1503610332 ## flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/table/LegacyTableSinkITCase.scala: ## @@ -277,6 +277,7 @@ class LegacyTab

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

xintongsong commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1501757320 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java: ## @@ -600,73 +605,138 @@ private void redistributeBuffers() {

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

TanYuxin-tyx commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1479461473 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java: ## @@ -224,6 +246,16 @@ private int tryRequestBuffers() {

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

TanYuxin-tyx commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1479461473 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java: ## @@ -224,6 +246,16 @@ private int tryRequestBuffers() {

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

TanYuxin-tyx commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1479461473 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java: ## @@ -224,6 +246,16 @@ private int tryRequestBuffers() {

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1473868814 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImpl.java: ## @@ -244,18 +246,8 @@ publ

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1473848488 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/common/TieredStorageConfiguration.java: ## @@ -37,8 +37,16 @@ public clas

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1473848488 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/common/TieredStorageConfiguration.java: ## @@ -37,8 +37,16 @@ public clas

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

TanYuxin-tyx commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1472472085 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java: ## @@ -132,16 +134,44 @@ private boolean shouldContinueRe

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

TanYuxin-tyx commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1472562350 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java: ## @@ -160,16 +159,14 @@ public SingleInputGate

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

TanYuxin-tyx commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1472434029 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java: ## @@ -184,6 +184,27 @@ public boolean append(ByteBuffer source, int

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

TanYuxin-tyx commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1472451043 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/common/TieredStorageConfiguration.java: ## @@ -37,8 +37,16 @@ public cla

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1466154957 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImpl.java: ## @@ -335,19 +338,21 @@ pri

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1466115976 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java: ## @@ -94,29 +99,58 @@ public class S

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1464734165 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/CreditBasedInputBuffersUsageGauge.java: ## @@ -28,27 +31,35 @@ */ public class CreditB

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

TanYuxin-tyx commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1462693982 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/common/TieredStorageConfiguration.java: ## @@ -133,10 +157,14 @@ public

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

xintongsong commented on code in PR #23851: URL: https://github.com/apache/flink/pull/23851#discussion_r1461707655 ## flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/CreditBasedInputBuffersUsageGauge.java: ## @@ -28,27 +31,35 @@ */ public class CreditB

Re: [PR] [FLINK-33668] Decoupling Shuffle network memory and job topology [flink]

jiangxin369 commented on PR #23851: URL: https://github.com/apache/flink/pull/23851#issuecomment-1894790270 The CI has some failures in the module `flink-table-planner`, but I cannot reproduce it locally. I suspect it was caused by some unexpected configuration. I'll keep troubleshooting it