[GitHub] [pinot] gortiz commented on issue #8762: Allow plugins to register and report metrics

2022-12-13 Thread GitBox
gortiz commented on issue #8762: URL: https://github.com/apache/pinot/issues/8762#issuecomment-1350577858 Do we actually need this? Metrics are not that different than logging. We do not have a logging api, we use slf4j. If some dependency we have uses log4j to log, then we can simply use t

[pinot] branch master updated: Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly (#9988)

2022-12-13 Thread nehapawar
This is an automated email from the ASF dual-hosted git repository. nehapawar pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 2d0cbc19de Ability to initialize S3PinotFs wit

[GitHub] [pinot] npawar merged pull request #9988: Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly

2022-12-13 Thread GitBox
npawar merged PR #9988: URL: https://github.com/apache/pinot/pull/9988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.

[GitHub] [pinot] navina commented on a diff in pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on code in PR #9977: URL: https://github.com/apache/pinot/pull/9977#discussion_r1048060753 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java: ## @@ -45,7 +45,10 @@ public enum ServerGauge implements AbstractMetrics.Gauge { // Dedu

[GitHub] [pinot] codecov-commenter commented on pull request #9988: Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly

2022-12-13 Thread GitBox
codecov-commenter commented on PR #9988: URL: https://github.com/apache/pinot/pull/9988#issuecomment-1350454180 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9988?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] klsince commented on a diff in pull request #9988: Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly

2022-12-13 Thread GitBox
klsince commented on code in PR #9988: URL: https://github.com/apache/pinot/pull/9988#discussion_r1048032072 ## pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java: ## @@ -149,10 +128,57 @@ public void init(PinotConfiguration

[GitHub] [pinot] npawar opened a new pull request, #9988: Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly

2022-12-13 Thread GitBox
npawar opened a new pull request, #9988: URL: https://github.com/apache/pinot/pull/9988 When using init method below, we cannot initialize serverSideEncryption properties. ``` public void init(S3Client s3Client) { _s3Client = s3Client; } ``` Adding a new init method,

[GitHub] [pinot] jadami10 opened a new issue, #9987: Zookeeper browser incorrectly shows "code" and "error" as children when a node is not found

2022-12-13 Thread GitBox
jadami10 opened a new issue, #9987: URL: https://github.com/apache/pinot/issues/9987 this is easiest to reproduce if you click one of the MESSAGES nodes under `//INSTANCES//MESSAGES`. Those are fairly ephemeral. the controller ZK api will return a 404, but the response has a "code" and "err

[GitHub] [pinot] 61yao opened a new issue, #9986: [multistage] [bug] Wrong result when both left join and inequi join exist and inequi condition filter out keys

2022-12-13 Thread GitBox
61yao opened a new issue, #9986: URL: https://github.com/apache/pinot/issues/9986 Example: tbl1 numCol, strCol 1, "a" 2, "b" tbl2 numCol, strCol 1, "a" 2, "c" Select * from tbl1 left join tbl2 on tbl1.numCol = tbl2.numCol and tbl1.strCol != tbl2.st

[pinot] branch master updated: Fix ser/de for StringLongPair (#9985)

2022-12-13 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new f743ff124e Fix ser/de for StringLongPair (#9985)

[GitHub] [pinot] Jackie-Jiang merged pull request #9985: Fix ser/de for StringLongPair

2022-12-13 Thread GitBox
Jackie-Jiang merged PR #9985: URL: https://github.com/apache/pinot/pull/9985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.a

[GitHub] [pinot] snleee commented on a diff in pull request #9985: Fix ser/de for StringLongPair

2022-12-13 Thread GitBox
snleee commented on code in PR #9985: URL: https://github.com/apache/pinot/pull/9985#discussion_r1047914173 ## pinot-core/src/test/java/org/apache/pinot/core/common/ObjectSerDeUtilsTest.java: ## @@ -189,9 +189,8 @@ public void testDoubleValueTimePair() { @Test public void

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #9985: Fix ser/de for StringLongPair

2022-12-13 Thread GitBox
Jackie-Jiang opened a new pull request, #9985: URL: https://github.com/apache/pinot/pull/9985 Fix the ser/de bug in `StringLongPair` which cannot handle special characters -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[pinot] branch master updated: [hotfix] github action script contains duplicate (#9984)

2022-12-13 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 7d20fbb312 [hotfix] github action script contains

[GitHub] [pinot] walterddr merged pull request #9984: [hotfix] remove github action script test duplicate

2022-12-13 Thread GitBox
walterddr merged PR #9984: URL: https://github.com/apache/pinot/pull/9984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

[GitHub] [pinot] mcvsubbu commented on a diff in pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
mcvsubbu commented on code in PR #9977: URL: https://github.com/apache/pinot/pull/9977#discussion_r1047906528 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerGauge.java: ## @@ -45,7 +45,10 @@ public enum ServerGauge implements AbstractMetrics.Gauge { // De

[GitHub] [pinot] navina commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1350201055 >The reason I was thinking of holding off is that some of the areas are common -- detecting partitions not served anymore, etc. Or, maybe we don't need that in the offset case ? If we do, th

[GitHub] [pinot] mcvsubbu commented on a diff in pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
mcvsubbu commented on code in PR #9977: URL: https://github.com/apache/pinot/pull/9977#discussion_r1047905553 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java: ## @@ -617,6 +624,40 @@ private boolean processStreamEvents(Me

[GitHub] [pinot] mcvsubbu commented on a diff in pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
mcvsubbu commented on code in PR #9977: URL: https://github.com/apache/pinot/pull/9977#discussion_r1047905063 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java: ## @@ -255,6 +255,11 @@ public void deleteSegmentFile() { p

[GitHub] [pinot] mcvsubbu commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
mcvsubbu commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1350198523 > @mcvsubbu I see it as a different solution where you are sampling to compute the worst ingestion lag at a table-level and emitting a metric based on some more signals from the realtime e

[GitHub] [pinot] codecov-commenter commented on pull request #9984: [hotfix] remove github action script test duplicate

2022-12-13 Thread GitBox
codecov-commenter commented on PR #9984: URL: https://github.com/apache/pinot/pull/9984#issuecomment-1350196821 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9984?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[pinot] branch master updated: [multistage][bugfix] fixing empty response unable to convert issue (#9983)

2022-12-13 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 1870f662b3 [multistage][bugfix] fixing empty respo

[GitHub] [pinot] walterddr merged pull request #9983: [multistage][bugfix] fixing empty response unable to convert issue

2022-12-13 Thread GitBox
walterddr merged PR #9983: URL: https://github.com/apache/pinot/pull/9983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

[GitHub] [pinot] walterddr closed issue #9980: [multistage][bug] multistage engine leaf stage returns incompatible data type.

2022-12-13 Thread GitBox
walterddr closed issue #9980: [multistage][bug] multistage engine leaf stage returns incompatible data type. URL: https://github.com/apache/pinot/issues/9980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [pinot] codecov-commenter commented on pull request #9983: [multistage][bugfix] fixing empty response unable to convert issue

2022-12-13 Thread GitBox
codecov-commenter commented on PR #9983: URL: https://github.com/apache/pinot/pull/9983#issuecomment-1350142204 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9983?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] navina commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1350063920 @mcvsubbu I see it as a different solution where you are sampling to compute the worst ingestion lag at a table-level and emitting a metric based on some more signals from the realtime engin

[GitHub] [pinot] mcvsubbu commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
mcvsubbu commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1350027454 > @mcvsubbu is this the work being done by linkedin - https://github.com/jugomezv/pinot/pull/3/files ? yes -- This is an automated message from the Apache Git Service. To respond

[GitHub] [pinot] walterddr commented on a diff in pull request #9983: [multistage][bugfix] fixing empty response unable to convert issue

2022-12-13 Thread GitBox
walterddr commented on code in PR #9983: URL: https://github.com/apache/pinot/pull/9983#discussion_r1047845778 ## pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/results/BaseResultsBlock.java: ## @@ -158,6 +158,11 @@ public void setNumServerThreads(int numServerTh

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9983: [multistage][bugfix] fixing empty response unable to convert issue

2022-12-13 Thread GitBox
Jackie-Jiang commented on code in PR #9983: URL: https://github.com/apache/pinot/pull/9983#discussion_r1047844145 ## pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/results/BaseResultsBlock.java: ## @@ -158,6 +158,11 @@ public void setNumServerThreads(int numServe

[GitHub] [pinot] navina commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1350004868 @mcvsubbu is this the work being done by linkedin - https://github.com/jugomezv/pinot/pull/3/files ? -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [pinot] walterddr opened a new pull request, #9983: [multistage][bugfix] fixing empty response unable to convert issue

2022-12-13 Thread GitBox
walterddr opened a new pull request, #9983: URL: https://github.com/apache/pinot/pull/9983 this fixes #9980. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

[GitHub] [pinot] navina commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1349993251 > Yes, Basically, the approach is to have a time-based lag as the diff between System.getCurrentMillis() and msgMetadata.getRecordIngestionTimeMs() and emit that as a metric. That's p

[GitHub] [pinot] mcvsubbu commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
mcvsubbu commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1349986645 Yes, Basically, the approach is to have a time-based lag as the diff between `System.getCurrentMillis()` and `msgMetadata.getRecordIngestionTimeMs()` and emit that as a metric. However

[pinot] branch master updated: [Clean up] Do not cache PropertiesConfiguration within SegmentMetadataImpl (#9952)

2022-12-13 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new bb12396406 [Clean up] Do not cache PropertiesConf

[GitHub] [pinot] Jackie-Jiang merged pull request #9952: [Clean up] Do not cache PropertiesConfiguration within SegmentMetadataImpl

2022-12-13 Thread GitBox
Jackie-Jiang merged PR #9952: URL: https://github.com/apache/pinot/pull/9952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.a

[GitHub] [pinot] walterddr commented on a diff in pull request #9967: [POC] Operator resource management

2022-12-13 Thread GitBox
walterddr commented on code in PR #9967: URL: https://github.com/apache/pinot/pull/9967#discussion_r1047811993 ## pinot-core/src/main/java/org/apache/pinot/core/common/Operator.java: ## @@ -41,6 +41,8 @@ */ T nextBlock(); + void close(); + Review Comment: let's don

[GitHub] [pinot] somandal opened a new pull request, #9982: Reload the SegmentMetadata even for temporary forward index

2022-12-13 Thread GitBox
somandal opened a new pull request, #9982: URL: https://github.com/apache/pinot/pull/9982 This PR fixes an issue where the `SegmentMetadata` isn't updated and reloaded for temporary forward index (which is created just to create other indexes off of and is then deleted once the other `Index

[GitHub] [pinot] navina commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1349856511 > We will be putting up a PR very soon on time based lag. Could we wait to take a look and see if we can come up with something that addresses both requirements? thanks. Can you or wh

[GitHub] [pinot] Jackie-Jiang commented on issue #9981: Support StarTree index with nullHandlingEnabled

2022-12-13 Thread GitBox
Jackie-Jiang commented on issue #9981: URL: https://github.com/apache/pinot/issues/9981#issuecomment-1349822308 Should be doable. Essentially we should: 1. Skip aggregating null values if it is configured to be pre-aggregated 2. Store null vector for null dimension values -- This is

[GitHub] [pinot] 61yao commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
61yao commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1047767657 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/RoundRobinScheduler.java: ## @@ -58,12 +64,30 @@ public class RoundRobinScheduler implements O

[GitHub] [pinot] mcvsubbu commented on pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
mcvsubbu commented on PR #9977: URL: https://github.com/apache/pinot/pull/9977#issuecomment-1349689401 We will be putting up a PR very soon on time based lag. Could we wait to take a look and see if we can come up with something that addresses both requirements? thanks. -- This is an

[GitHub] [pinot] evandialpad opened a new issue, #9981: Support StarTree index with nullHandlingEnabled

2022-12-13 Thread GitBox
evandialpad opened a new issue, #9981: URL: https://github.com/apache/pinot/issues/9981 Not sure if this is technically feasible but would be great to have this support. Perhaps during ST index creation, the null vector index could be used to skip adding specific values to the startree inde

[pinot] branch master updated (c56a023f7a -> e092140d5f)

2022-12-13 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from c56a023f7a coalesce literal (#9958) add e092140d5f [multistage] add defensive scheduling, enable for tests (#9962)

[GitHub] [pinot] walterddr merged pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
walterddr merged PR #9962: URL: https://github.com/apache/pinot/pull/9962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

[GitHub] [pinot] walterddr opened a new issue, #9980: [multistage][bug] multistage engine leaf stage returns incompatible data type.

2022-12-13 Thread GitBox
walterddr opened a new issue, #9980: URL: https://github.com/apache/pinot/issues/9980 Problematic GROUP BY V1 behavior returns STRING instead of INT as indicated in schema see: https://github.com/apache/pinot/actions/runs/3688265209/jobs/6242862869 ``` 2022-12-13T18:43:31.798528

[GitHub] [pinot] walterddr commented on a diff in pull request #9969: [multistage] [bugfix] Throw error when GrpcMailbox receiving buffer is full

2022-12-13 Thread GitBox
walterddr commented on code in PR #9969: URL: https://github.com/apache/pinot/pull/9969#discussion_r1047720806 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentStreamObserver.java: ## @@ -93,7 +105,19 @@ public void onNext(Mailbox.Mailbox

[GitHub] [pinot] walterddr commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
walterddr commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1047713887 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java: ## @@ -98,9 +108,12 @@ public void runJob() {

[GitHub] [pinot] 61yao commented on a diff in pull request #9974: [multistage] improve scheduling to be more mailbox-aware

2022-12-13 Thread GitBox
61yao commented on code in PR #9974: URL: https://github.com/apache/pinot/pull/9974#discussion_r1047682339 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/V2Operator.java: ## @@ -0,0 +1,45 @@ +/** + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [pinot] agavra commented on a diff in pull request #9969: [multistage] [bugfix] Throw error when GrpcMailbox receiving buffer is full

2022-12-13 Thread GitBox
agavra commented on code in PR #9969: URL: https://github.com/apache/pinot/pull/9969#discussion_r1047678758 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentStreamObserver.java: ## @@ -93,7 +105,19 @@ public void onNext(Mailbox.MailboxCon

[GitHub] [pinot] 61yao commented on a diff in pull request #9969: [multistage] [bugfix] Throw error when GrpcMailbox receiving buffer is full

2022-12-13 Thread GitBox
61yao commented on code in PR #9969: URL: https://github.com/apache/pinot/pull/9969#discussion_r1047663720 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentStreamObserver.java: ## @@ -116,11 +140,9 @@ public void onNext(Mailbox.MailboxCon

[GitHub] [pinot] 61yao closed issue #9931: [feature] Support coalesce for literals

2022-12-13 Thread GitBox
61yao closed issue #9931: [feature] Support coalesce for literals URL: https://github.com/apache/pinot/issues/9931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

[GitHub] [pinot] 61yao commented on issue #9931: [feature] Support coalesce for literals

2022-12-13 Thread GitBox
61yao commented on issue #9931: URL: https://github.com/apache/pinot/issues/9931#issuecomment-1349592509 https://github.com/apache/pinot/pull/9958 got merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [pinot] 61yao commented on a diff in pull request #9969: [multistage] [bugfix] Throw error when GrpcMailbox receiving buffer is full

2022-12-13 Thread GitBox
61yao commented on code in PR #9969: URL: https://github.com/apache/pinot/pull/9969#discussion_r1047651175 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentStreamObserver.java: ## @@ -93,7 +105,19 @@ public void onNext(Mailbox.MailboxCont

[GitHub] [pinot] Jackie-Jiang merged pull request #9958: [Feature] Support coalesce literal

2022-12-13 Thread GitBox
Jackie-Jiang merged PR #9958: URL: https://github.com/apache/pinot/pull/9958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.a

[pinot] branch master updated: coalesce literal (#9958)

2022-12-13 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c56a023f7a coalesce literal (#9958) c56a023f7a is

[GitHub] [pinot] agavra commented on a diff in pull request #9969: [multistage] [bugfix] Throw error when GrpcMailbox receiving buffer is full

2022-12-13 Thread GitBox
agavra commented on code in PR #9969: URL: https://github.com/apache/pinot/pull/9969#discussion_r1047585319 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentStreamObserver.java: ## @@ -93,7 +105,19 @@ public void onNext(Mailbox.MailboxCon

[GitHub] [pinot] navina commented on a diff in pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on code in PR #9977: URL: https://github.com/apache/pinot/pull/9977#discussion_r1047579926 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java: ## @@ -255,6 +255,11 @@ public void deleteSegmentFile() { pri

[GitHub] [pinot] agavra closed pull request #9974: [multistage] improve scheduling to be more mailbox-aware

2022-12-13 Thread GitBox
agavra closed pull request #9974: [multistage] improve scheduling to be more mailbox-aware URL: https://github.com/apache/pinot/pull/9974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [pinot] agavra commented on pull request #9974: [multistage] improve scheduling to be more mailbox-aware

2022-12-13 Thread GitBox
agavra commented on PR #9974: URL: https://github.com/apache/pinot/pull/9974#issuecomment-1349492534 Closing this PR since it's unlikely it'll solve the problem given https://github.com/apache/pinot/pull/9974#discussion_r1046628009 While this is still a general improvement, I don't th

[GitHub] [pinot] walterddr commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
walterddr commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1047576553 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/RoundRobinScheduler.java: ## @@ -58,12 +64,29 @@ public class RoundRobinScheduler implement

[GitHub] [pinot] agavra commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
agavra commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1047566110 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/RoundRobinScheduler.java: ## @@ -58,12 +64,29 @@ public class RoundRobinScheduler implements

[GitHub] [pinot] agavra commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
agavra commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1047557013 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java: ## @@ -98,9 +108,12 @@ public void runJob() { /

[GitHub] [pinot] agavra commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
agavra commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1047543621 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/QueryConfig.java: ## @@ -59,6 +62,12 @@ public class QueryConfig { public static final String KEY_OF

[GitHub] [pinot] navina commented on a diff in pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on code in PR #9977: URL: https://github.com/apache/pinot/pull/9977#discussion_r1047540796 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java: ## @@ -255,6 +255,11 @@ public void deleteSegmentFile() { pri

[GitHub] [pinot] agavra commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
agavra commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1047537391 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java: ## @@ -98,9 +108,12 @@ public void runJob() { /

[GitHub] [pinot] navina commented on a diff in pull request #9977: Emit consumer partition lag from the server

2022-12-13 Thread GitBox
navina commented on code in PR #9977: URL: https://github.com/apache/pinot/pull/9977#discussion_r1047533146 ## pinot-tools/src/main/resources/examples/stream/airlineStats/airlineStats_realtime_table_config.json: ## @@ -25,8 +25,7 @@ "stream.kafka.zk.broker.url": "localhos

[GitHub] [pinot] walterddr commented on a diff in pull request #9957: [multistage] [feature] Add a query option to pass some v1 limit

2022-12-13 Thread GitBox
walterddr commented on code in PR #9957: URL: https://github.com/apache/pinot/pull/9957#discussion_r1047532717 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineIntegrationTest.java: ## @@ -99,6 +99,14 @@ public void testGeneratedQueries

[GitHub] [pinot] walterddr commented on a diff in pull request #9957: [multistage] [feature] Add a query option to pass some v1 limit

2022-12-13 Thread GitBox
walterddr commented on code in PR #9957: URL: https://github.com/apache/pinot/pull/9957#discussion_r1047531456 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -291,6 +291,21 @@ public static class QueryOptionKey { public static final

[GitHub] [pinot] walterddr commented on issue #9970: [multistage] default group limit unable to override

2022-12-13 Thread GitBox
walterddr commented on issue #9970: URL: https://github.com/apache/pinot/issues/9970#issuecomment-1349303934 not really. we also want to configure this on the fly using QueryOptions. not just cluster config. -- This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [pinot] snleee commented on issue #9970: [multistage] default group limit unable to override

2022-12-13 Thread GitBox
snleee commented on issue #9970: URL: https://github.com/apache/pinot/issues/9970#issuecomment-1349287009 @walterddr Is https://github.com/apache/pinot/pull/9978 addressing this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [pinot] agavra commented on a diff in pull request #9974: [multistage] improve scheduling to be more mailbox-aware

2022-12-13 Thread GitBox
agavra commented on code in PR #9974: URL: https://github.com/apache/pinot/pull/9974#discussion_r1047479950 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/HashJoinOperator.java: ## @@ -121,6 +121,15 @@ protected TransferableBlock getNextBlock() {

[GitHub] [pinot] snleee commented on pull request #9978: Fix the executor config wiring issue for Multi-stage engine

2022-12-13 Thread GitBox
snleee commented on PR #9978: URL: https://github.com/apache/pinot/pull/9978#issuecomment-1349104080 @walterddr thanks for modifying the code for me 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [pinot] egalpin commented on issue #7519: FILTER Clause Support

2022-12-13 Thread GitBox
egalpin commented on issue #7519: URL: https://github.com/apache/pinot/issues/7519#issuecomment-1349081193 I wanted to mention that I'm currently exploring/investigating GROUP BY support with FILTER expressions. As best as I can tell so far, GroupByPlanNode will need to have similar modifi

[GitHub] [pinot] walterddr commented on a diff in pull request #9956: [multistage] [observability] Add debugging log for mailbox service

2022-12-13 Thread GitBox
walterddr commented on code in PR #9956: URL: https://github.com/apache/pinot/pull/9956#discussion_r1047433664 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcMailboxService.java: ## @@ -89,20 +92,27 @@ public int getMailboxPort() { * @param mailboxId

[GitHub] [pinot] walterddr commented on a diff in pull request #9956: [multistage] [observability] Add debugging log for mailbox service

2022-12-13 Thread GitBox
walterddr commented on code in PR #9956: URL: https://github.com/apache/pinot/pull/9956#discussion_r1047433664 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcMailboxService.java: ## @@ -89,20 +92,27 @@ public int getMailboxPort() { * @param mailboxId

[GitHub] [pinot] walterddr commented on a diff in pull request #9956: [multistage] [observability] Add debugging log for mailbox service

2022-12-13 Thread GitBox
walterddr commented on code in PR #9956: URL: https://github.com/apache/pinot/pull/9956#discussion_r1047433083 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcMailboxService.java: ## @@ -89,20 +92,27 @@ public int getMailboxPort() { * @param mailboxId

[GitHub] [pinot] walterddr commented on a diff in pull request #9956: [multistage] [observability] Add debugging log for mailbox service

2022-12-13 Thread GitBox
walterddr commented on code in PR #9956: URL: https://github.com/apache/pinot/pull/9956#discussion_r1047432457 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcMailboxService.java: ## @@ -89,20 +92,27 @@ public int getMailboxPort() { * @param mailboxId

[GitHub] [pinot] walterddr merged pull request #9978: Fix the executor config wiring issue for Multi-stage engine

2022-12-13 Thread GitBox
walterddr merged PR #9978: URL: https://github.com/apache/pinot/pull/9978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

[pinot] branch master updated: Fix the executor config wiring issue for Multi-stage engine (#9978)

2022-12-13 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 0338d5d681 Fix the executor config wiring issue fo

[GitHub] [pinot] walterddr commented on a diff in pull request #9873: [multistage] Add Pluggable Physical Optimizers + Greedy Shuffle Skip Optimizer

2022-12-13 Thread GitBox
walterddr commented on code in PR #9873: URL: https://github.com/apache/pinot/pull/9873#discussion_r1047408268 ## pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java: ## @@ -221,7 +225,7 @@ private RelNode optimize(RelRoot relRoot, PlannerContext plan

[GitHub] [pinot] walterddr commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
walterddr commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1046610518 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/QueryConfig.java: ## @@ -59,6 +62,12 @@ public class QueryConfig { public static final String KEY

[GitHub] [pinot] walterddr commented on a diff in pull request #9962: [multistage] add defensive scheduling, enable for tests

2022-12-13 Thread GitBox
walterddr commented on code in PR #9962: URL: https://github.com/apache/pinot/pull/9962#discussion_r1046610518 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/service/QueryConfig.java: ## @@ -59,6 +62,12 @@ public class QueryConfig { public static final String KEY

[GitHub] [pinot] walterddr commented on a diff in pull request #9978: Fix the executor config wiring issue for Multi-stage engine

2022-12-13 Thread GitBox
walterddr commented on code in PR #9978: URL: https://github.com/apache/pinot/pull/9978#discussion_r1047352757 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/QueryRunner.java: ## @@ -100,12 +103,11 @@ public void init(PinotConfiguration config, InstanceData

[GitHub] [pinot] walterddr commented on a diff in pull request #9978: Fix the executor config wiring issue for Multi-stage engine

2022-12-13 Thread GitBox
walterddr commented on code in PR #9978: URL: https://github.com/apache/pinot/pull/9978#discussion_r1047350226 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/QueryRunner.java: ## @@ -100,12 +103,16 @@ public void init(PinotConfiguration config, InstanceData

[pinot] branch release-0.12.0-rc created (now 7f643b1df8)

2022-12-13 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch release-0.12.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git at 7f643b1df8 Adding configs for zk client timeout (#9975) No new revisions were added by this update.

[pinot] branch master updated: Adding configs for zk client timeout (#9975)

2022-12-13 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 7f643b1df8 Adding configs for zk client timeout

[GitHub] [pinot] xiangfu0 merged pull request #9975: Adding configs for zk client timeout

2022-12-13 Thread GitBox
xiangfu0 merged PR #9975: URL: https://github.com/apache/pinot/pull/9975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[GitHub] [pinot] xiangfu0 commented on a diff in pull request #9950: [multistage][testing] Add test for null handling, round func and some other cases

2022-12-13 Thread GitBox
xiangfu0 commented on code in PR #9950: URL: https://github.com/apache/pinot/pull/9950#discussion_r1046875679 ## pinot-query-runtime/src/test/resources/queries/NullHanlding.json: ## @@ -0,0 +1,40 @@ +{ + "select_expression_test": { Review Comment: duplicated test name. -

[GitHub] [pinot] xiangfu0 commented on a diff in pull request #9950: [multistage][testing] Add test for null handling, round func and some other cases

2022-12-13 Thread GitBox
xiangfu0 commented on code in PR #9950: URL: https://github.com/apache/pinot/pull/9950#discussion_r1046874180 ## pinot-query-runtime/src/test/resources/queries/Cast.json: ## @@ -0,0 +1,20 @@ +{ + "select_expression_test": { Review Comment: Cast also has issue of: ```