shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1765124295
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,45 @@ protected void processTable
KKcorps merged PR #13584:
URL: https://github.com/apache/pinot/pull/13584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apac
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1763607852
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,45 @@ protected void processTable
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1763445783
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,45 @@ protected void processTable(Str
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1762202266
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,45 @@ protected void processTable
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1761941816
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,45 @@ protected void processTable(Str
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1759248976
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,36 @@ protected void processTable(Str
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1758116054
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,36 @@ protected void processTable
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1757335291
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,34 @@ protected void processTable(Str
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1757332293
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,36 @@ protected void processTable(Str
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1757275377
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,34 @@ protected void processTable
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1757102975
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +112,34 @@ protected void processTable(Str
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1756734820
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##
@@ -43,14 +44,19 @@ public class StorageQuotaChecker {
pri
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1755282957
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##
@@ -43,14 +44,19 @@ public class StorageQuotaChecker {
private
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1741836921
##
pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java:
##
@@ -160,7 +160,8 @@ public enum ControllerGauge implements
AbstractMetrics.
shounakmk219 commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2326191290
Hey @mcvsubbu @sajjad-moradi I have updated the PR with the changes from
PauseState PR and also removed the storage quota check from segment commit
process and now only have it as pa
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1707957034
##
pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java:
##
@@ -160,7 +160,8 @@ public enum ControllerGauge implements
AbstractMetrics.Gaug
shounakmk219 commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2274108292
Hey @mcvsubbu @sajjad-moradi , it is ok if the table storage spills over the
quota by few segments before stopping the consumption and that said we can keep
the quota validation + IS
mcvsubbu commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-227402
> On a separate note, @mcvsubbu thinks it's better to have all the logic for
hanlding storage quota (pausing/resuming) in one place, in "realtime segment
validation manager" periodic job
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2273986235
@Jackie-Jiang any thoughts here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2273985656
On a separate note, @mcvsubbu thinks it's better to have all the logic for
hanlding storage quota (pausing/resuming) in one place, in "realtime segment
validation manager" periodic
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2273949793
The priority you described makes sense. Basically storage quota should not
be even be checked if the table is paused by an admin.
--
This is an automated message from the Apache G
shounakmk219 commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2268114649
@sajjad-moradi @mcvsubbu , the `pauseStatus` field makes sense. Do you think
we need to maintain the set of different pause reasons active on a table as
well? Context will be lost du
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2266179376
I just had a discussion with @mcvsubbu about this. Here's what we think:
With the proposed solution in this PR, there will be two IS flags indicating
if the table is paused: "isTa
shounakmk219 commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2252520066
@sajjad-moradi can you please review the PR?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
U
swaminathanmanish commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1692536389
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##
@@ -1696,6 +1705,32 @@ private Ideal
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1692531490
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##
@@ -1696,6 +1705,32 @@ private IdealState
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1686745580
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##
@@ -1696,6 +1705,32 @@ private IdealState
swaminathanmanish commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1685973069
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##
@@ -1696,6 +1705,32 @@ private Ideal
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2240321105
> Mostly good. Can we add a test to verify this?
>
> @swaminathanmanish @sajjad-moradi please also take a look
Will review next week.
--
This is an automated message
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1682291845
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##
@@ -205,4 +213,48 @@ public QuotaCheckerResponse
isSegmentSt
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1682278381
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##
@@ -108,8 +110,20 @@ protected void processTable
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1682260924
##
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##
@@ -1696,6 +1705,20 @@ private IdealState
Jackie-Jiang commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1681811635
##
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##
@@ -75,9 +80,12 @@ public static QuotaCheckerResponse failure
codecov-commenter commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-200098
##
[Codecov](https://app.codecov.io/gh/apache/pinot/pull/13584?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u
shounakmk219 opened a new pull request, #13584:
URL: https://github.com/apache/pinot/pull/13584
# Description
This PR allows pinot to impose storage quota restrictions on realtime tables.
To block the ingestion on realtime tables below consideration are kept in
mind
1. Repli
36 matches
Mail list logo
