Re: [PR] Add offset based lag metrics [pinot]

jadami10 commented on PR #13298: URL: https://github.com/apache/pinot/pull/13298#issuecomment-2361386066 > the earlier metric was really noisy since it relies on time column value instead of ingestion time which lead to false positives. That sounds like a misuse of the `StreamMessageM

Re: [PR] Add offset based lag metrics [pinot]

KKcorps commented on PR #13298: URL: https://github.com/apache/pinot/pull/13298#issuecomment-236392 hi @jadami10 thanks for brining this up. the earlier metric was really noisy since it relies on time column value instead of ingestion time which lead to false positives. wou

Re: [PR] Add offset based lag metrics [pinot]

jadami10 commented on PR #13298: URL: https://github.com/apache/pinot/pull/13298#issuecomment-2353694120 the other big issue I see is if we fail to request the latest offset, we stop publishing both the time based and offset based ingestion lag. -- This is an automated message from the Ap

Re: [PR] Add offset based lag metrics [pinot]

jadami10 commented on PR #13298: URL: https://github.com/apache/pinot/pull/13298#issuecomment-2353497319 we're upgrading to 1.2 now, and this increased our kafka offset requests in our QA environment by 3 orders of magnitude. I think at minimum we need a way to turn this off (internally we'

Re: [PR] Add offset based lag metrics [pinot]

Jackie-Jiang commented on PR #13298: URL: https://github.com/apache/pinot/pull/13298#issuecomment-2322120993 We are making one extra metadata read per message batch. Have you verified the overhead? Should we add a config to turn this on/off? -- This is an automated message from the Apache

Re: [PR] Add offset based lag metrics [pinot]

KKcorps merged PR #13298: URL: https://github.com/apache/pinot/pull/13298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

Re: [PR] Add offset based lag metrics [pinot]

KKcorps commented on code in PR #13298: URL: https://github.com/apache/pinot/pull/13298#discussion_r1631045755 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java: ## @@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingest

Re: [PR] Add offset based lag metrics [pinot]

KKcorps commented on code in PR #13298: URL: https://github.com/apache/pinot/pull/13298#discussion_r1631036184 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java: ## @@ -88,6 +90,16 @@ private static class IngestionTimestamps {

Re: [PR] Add offset based lag metrics [pinot]

KKcorps commented on code in PR #13298: URL: https://github.com/apache/pinot/pull/13298#discussion_r1631035110 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java: ## @@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingest

Re: [PR] Add offset based lag metrics [pinot]

KKcorps commented on code in PR #13298: URL: https://github.com/apache/pinot/pull/13298#discussion_r1631034832 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java: ## @@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingest

Re: [PR] Add offset based lag metrics [pinot]

KKcorps commented on code in PR #13298: URL: https://github.com/apache/pinot/pull/13298#discussion_r1631032269 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java: ## @@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingest

Re: [PR] Add offset based lag metrics [pinot]

KKcorps commented on code in PR #13298: URL: https://github.com/apache/pinot/pull/13298#discussion_r1631032269 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java: ## @@ -174,6 +188,22 @@ private long getIngestionDelayMs(long ingest

Re: [PR] Add offset based lag metrics [pinot]

gortiz commented on code in PR #13298: URL: https://github.com/apache/pinot/pull/13298#discussion_r1630692381 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java: ## @@ -88,6 +90,16 @@ private static class IngestionTimestamps {

Re: [PR] Add offset based lag metrics [pinot]

swaminathanmanish commented on code in PR #13298: URL: https://github.com/apache/pinot/pull/13298#discussion_r1630651998 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java: ## @@ -174,6 +188,22 @@ private long getIngestionDelayMs(l

Re: [PR] Add offset based lag metrics [pinot]

codecov-commenter commented on PR #13298: URL: https://github.com/apache/pinot/pull/13298#issuecomment-2145015757 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13298?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u