[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11715: Reduce cpu usage by avoiding throwing an exception during query execution

2023-09-29 Thread via GitHub
Jackie-Jiang commented on code in PR #11715: URL: https://github.com/apache/pinot/pull/11715#discussion_r1341915681 ## pinot-core/src/main/java/org/apache/pinot/core/operator/combine/CombineOperatorUtils.java: ## @@ -48,15 +48,12 @@ public static void

[GitHub] [pinot] Jackie-Jiang commented on pull request #11711: Post build index creation

2023-09-29 Thread via GitHub
Jackie-Jiang commented on PR #11711: URL: https://github.com/apache/pinot/pull/11711#issuecomment-1741687994 cc @gortiz -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [pinot] Jackie-Jiang opened a new issue, #11718: [Flaky-test] StarTreeClusterIntegrationTest.testGeneratedQueries

2023-09-29 Thread via GitHub
Jackie-Jiang opened a new issue, #11718: URL: https://github.com/apache/pinot/issues/11718 ``` Error: Failures: Error: StarTreeClusterIntegrationTest.testGeneratedQueries:163->testStarQuery:185 Query comparison failed for: Star Query: SELECT avg(ArrDel15) FROM mytable WHERE

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11710: Reduce direct memory OOM chances on broker with a per server query response size budget

2023-09-29 Thread via GitHub
Jackie-Jiang commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341914079 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -362,6 +365,11 @@ public static class QueryOptionKey { public static

[pinot] branch master updated: Add more test for broker jersey bounded thread pool (#11705)

2023-09-29 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ae1681273c Add more test for broker jersey

[GitHub] [pinot] Jackie-Jiang merged pull request #11705: Add more test for broker jersey bounded thread pool

2023-09-29 Thread via GitHub
Jackie-Jiang merged PR #11705: URL: https://github.com/apache/pinot/pull/11705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11717: [Flaky-test] Fix TableCacheTest

2023-09-29 Thread via GitHub
Jackie-Jiang opened a new pull request, #11717: URL: https://github.com/apache/pinot/pull/11717 Fix #11435 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [pinot] codecov-commenter commented on pull request #11716: Use string to represent BigDecimal response

2023-09-29 Thread via GitHub
codecov-commenter commented on PR #11716: URL: https://github.com/apache/pinot/pull/11716#issuecomment-1741607104 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11716?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging

[pinot] branch master updated (82140f1177 -> dcffc6dea1)

2023-09-29 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 82140f1177 JSON index: Add support for ignoring values longer than a given length. (#11604) add dcffc6dea1

[GitHub] [pinot] Jackie-Jiang merged pull request #11714: Gapfill: Fix bug with SumAvgGapfillProcessor.

2023-09-29 Thread via GitHub
Jackie-Jiang merged PR #11714: URL: https://github.com/apache/pinot/pull/11714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [pinot] abhioncbr commented on a diff in pull request #11653: ScalarTransformFunctionWrapper NULL support

2023-09-29 Thread via GitHub
abhioncbr commented on code in PR #11653: URL: https://github.com/apache/pinot/pull/11653#discussion_r1341865991 ## pinot-common/src/main/java/org/apache/pinot/common/utils/DataSchema.java: ## @@ -297,6 +297,7 @@ public DataType toDataType() { case TIMESTAMP:

[GitHub] [pinot] Jackie-Jiang commented on pull request #11716: Use string to represent BigDecimal response

2023-09-29 Thread via GitHub
Jackie-Jiang commented on PR #11716: URL: https://github.com/apache/pinot/pull/11716#issuecomment-1741594165 This issue was introduced in #8503 where responses from aggregation are stored as string, responses from other queries are stored as `BigDecimal`; #11453 changed it to always store

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11716: Use string to represent BigDecimal response

2023-09-29 Thread via GitHub
Jackie-Jiang opened a new pull request, #11716: URL: https://github.com/apache/pinot/pull/11716 It is best practice to store `BigDecimal` as string in JSON to prevent losing precision. Currently we directly serialize `BigDecimal` into number (without quote) in the query response, but

[GitHub] [pinot] abhioncbr commented on a diff in pull request #11653: ScalarTransformFunctionWrapper NULL support

2023-09-29 Thread via GitHub
abhioncbr commented on code in PR #11653: URL: https://github.com/apache/pinot/pull/11653#discussion_r1341862027 ## pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapperTest.java: ## @@ -262,6 +262,27 @@ public void

[GitHub] [pinot] walterddr closed pull request #11686: [multistage] support legacy query function "lookup" in multi-stage

2023-09-29 Thread via GitHub
walterddr closed pull request #11686: [multistage] support legacy query function "lookup" in multi-stage URL: https://github.com/apache/pinot/pull/11686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [pinot] walterddr commented on pull request #11686: [multistage] support legacy query function "lookup" in multi-stage

2023-09-29 Thread via GitHub
walterddr commented on PR #11686: URL: https://github.com/apache/pinot/pull/11686#issuecomment-1741579551 closing as it is not a full support fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [pinot] codecov-commenter commented on pull request #11715: Reduce cpu usage by avoiding throwing an exception during query execution

2023-09-29 Thread via GitHub
codecov-commenter commented on PR #11715: URL: https://github.com/apache/pinot/pull/11715#issuecomment-1741574245 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11715?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging

[GitHub] [pinot] vvivekiyer opened a new pull request, #11715: Reduce cpu usage by avoiding throwing an exception during query execution

2023-09-29 Thread via GitHub
vvivekiyer opened a new pull request, #11715: URL: https://github.com/apache/pinot/pull/11715 In #9092, we added query execution stats to fetch setNumConsumingSegmentsProcessed and setNumConsumingSegmentsMatched In one of our production clusters, we observed that

[GitHub] [pinot] codecov-commenter commented on pull request #11714: Gapfill: Fix bug with SumAvgGapfillProcessor.

2023-09-29 Thread via GitHub
codecov-commenter commented on PR #11714: URL: https://github.com/apache/pinot/pull/11714#issuecomment-1741493424 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11714?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging

[GitHub] [pinot] ege-st commented on a diff in pull request #11710: Reduce direct memory OOM chances on broker with a per server query response size budget

2023-09-29 Thread via GitHub
ege-st commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341787880 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -170,15 +171,18 @@ public

[GitHub] [pinot] ege-st commented on a diff in pull request #11710: Reduce direct memory OOM chances on broker with a per server query response size budget

2023-09-29 Thread via GitHub
ege-st commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341781521 ## pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java: ## @@ -168,6 +169,19 @@ protected byte[]

[GitHub] [pinot] ege-st commented on a diff in pull request #11710: Reduce direct memory OOM chances on broker with a per server query response size budget

2023-09-29 Thread via GitHub
ege-st commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341779996 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -318,6 +318,9 @@ public static class Broker {

[GitHub] [pinot] weixiangsun opened a new pull request, #11714: Gapfill: Fix bug with SumAvgGapfillProcessor.

2023-09-29 Thread via GitHub
weixiangsun opened a new pull request, #11714: URL: https://github.com/apache/pinot/pull/11714 This is bugfix for Gapfill SumAvgGapfillProcessor. The bug is that SumAvgGapfillProcessor is not using the correct previous value before doing the gapfill. The value after the gapfill range

[GitHub] [pinot] zhtaoxiang commented on pull request #11711: Post build index creation

2023-09-29 Thread via GitHub
zhtaoxiang commented on PR #11711: URL: https://github.com/apache/pinot/pull/11711#issuecomment-1741368196 I approved the PR by accident  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [pinot] zhtaoxiang commented on a diff in pull request #11711: Post build index creation

2023-09-29 Thread via GitHub
zhtaoxiang commented on code in PR #11711: URL: https://github.com/apache/pinot/pull/11711#discussion_r1341714287 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java: ## @@ -313,6 +325,36 @@ private void

[GitHub] [pinot] Jackie-Jiang closed issue #11445: [multistage] infer query runner host from broker or server hostname

2023-09-29 Thread via GitHub
Jackie-Jiang closed issue #11445: [multistage] infer query runner host from broker or server hostname URL: https://github.com/apache/pinot/issues/11445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341681228 ## pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java: ## @@ -168,6 +169,19 @@ protected byte[]

[GitHub] [pinot] siddharthteotia commented on pull request #11575: Add metrics to export netty direct memory used and max

2023-09-29 Thread via GitHub
siddharthteotia commented on PR #11575: URL: https://github.com/apache/pinot/pull/11575#issuecomment-1741307162 I think @Jackie-Jiang has a comment. I am good with merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [pinot] siddharthteotia commented on pull request #11575: Add metrics to export netty direct memory used and max

2023-09-29 Thread via GitHub
siddharthteotia commented on PR #11575: URL: https://github.com/apache/pinot/pull/11575#issuecomment-1741305937 > But maybe what do you mean is that in some place in this PR I'm not calling the version with the supplier and therefore the value never change. It wouldn't be strange given

[GitHub] [pinot] jasperjiaguo commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
jasperjiaguo commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341666747 ## pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java: ## @@ -168,6 +169,19 @@ protected byte[]

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341664954 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -1654,6 +1669,24 @@ private long

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341663123 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -362,6 +365,11 @@ public static class QueryOptionKey { public static

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341663319 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -318,6 +318,9 @@ public static class Broker {

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
siddharthteotia commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1341663319 ## pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java: ## @@ -318,6 +318,9 @@ public static class Broker {

[GitHub] [pinot] vvivekiyer commented on pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
vvivekiyer commented on PR #11710: URL: https://github.com/apache/pinot/pull/11710#issuecomment-1741230828 > How will a user determine what the threshold should be? @ege-st Good question. We did think about this. The ideal answer is that the value is use case dependent.

[GitHub] [pinot] chenboat merged pull request #11604: JSON index: Add support for ignoring values longer than a given length.

2023-09-29 Thread via GitHub
chenboat merged PR #11604: URL: https://github.com/apache/pinot/pull/11604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[pinot] branch master updated: JSON index: Add support for ignoring values longer than a given length. (#11604)

2023-09-29 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 82140f1177 JSON index: Add support for

[GitHub] [pinot] ege-st commented on pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
ege-st commented on PR #11710: URL: https://github.com/apache/pinot/pull/11710#issuecomment-1740980503 Cool, thanks! I think reducing the likelyhood of DM OOMs is the logical next step. How will a user determine what the threshold should be? A critical requirement for a feature like

[GitHub] [pinot] codecov-commenter commented on pull request #11713: Adding a metric to measure total thread cpu time for a table

2023-09-29 Thread via GitHub
codecov-commenter commented on PR #11713: URL: https://github.com/apache/pinot/pull/11713#issuecomment-1740964085 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11713?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging

[GitHub] [pinot] swaminathanmanish commented on a diff in pull request #11712: [DO NOT MERGE] Add a test to prove how to use protobuf nullability

2023-09-29 Thread via GitHub
swaminathanmanish commented on code in PR #11712: URL: https://github.com/apache/pinot/pull/11712#discussion_r1341419664 ## pinot-plugins/pinot-input-format/pinot-protobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordExtractor.java: ## @@ -62,7 +62,7

[GitHub] [pinot] mcvsubbu opened a new pull request, #11713: Adding a metric to measure total thread cpu time for a table

2023-09-29 Thread via GitHub
mcvsubbu opened a new pull request, #11713: URL: https://github.com/apache/pinot/pull/11713 Allows us to compute the total thread cpu time each table uses for queries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [pinot] codecov-commenter commented on pull request #11712: [DO NOT MERGE] Add a test to prove how to use protobuf nullability

2023-09-29 Thread via GitHub
codecov-commenter commented on PR #11712: URL: https://github.com/apache/pinot/pull/11712#issuecomment-1740862007 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11712?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging

[GitHub] [pinot] gortiz opened a new pull request, #11712: [DO NOT MERGE] Add a test to prove how to use protobuf nullability

2023-09-29 Thread via GitHub
gortiz opened a new pull request, #11712: URL: https://github.com/apache/pinot/pull/11712 This PR adds a test to verify the correct nullability read on `ProtoBufRecordExtractor`. The PR is not yet ready to merge because for some reason the descriptor is using v2 instead of 3.

[GitHub] [pinot] codecov-commenter commented on pull request #11711: Post build index creation

2023-09-29 Thread via GitHub
codecov-commenter commented on PR #11711: URL: https://github.com/apache/pinot/pull/11711#issuecomment-1740701803 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11711?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging

[GitHub] [pinot] saurabhd336 opened a new pull request, #11711: Post build index creation

2023-09-29 Thread via GitHub
saurabhd336 opened a new pull request, #11711: URL: https://github.com/apache/pinot/pull/11711 It may be desirable for some index types to be build post segment file has been created. This PR allows for that. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [pinot] codecov-commenter commented on pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
codecov-commenter commented on PR #11710: URL: https://github.com/apache/pinot/pull/11710#issuecomment-1740388935 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11710?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging

[GitHub] [pinot] gortiz commented on pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
gortiz commented on PR #11710: URL: https://github.com/apache/pinot/pull/11710#issuecomment-1740385931 I've added some notes, but I think the PR is in good shape -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [pinot] gortiz commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
gortiz commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1340953692 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java: ## @@ -602,6 +602,16 @@ public void testTimeFunc(boolean

[GitHub] [pinot] gortiz commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
gortiz commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1340952925 ## pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java: ## @@ -168,6 +169,19 @@ protected byte[]

[GitHub] [pinot] gortiz commented on a diff in pull request #11710: Fix Direct Memory OOM on broker by limiting query response size

2023-09-29 Thread via GitHub
gortiz commented on code in PR #11710: URL: https://github.com/apache/pinot/pull/11710#discussion_r1340952382 ## pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java: ## @@ -168,6 +169,19 @@ protected byte[]

[GitHub] [pinot] codecov-commenter commented on pull request #11709: update map initial capacity

2023-09-29 Thread via GitHub
codecov-commenter commented on PR #11709: URL: https://github.com/apache/pinot/pull/11709#issuecomment-1740361099 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11709?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging

[GitHub] [pinot] xiangfu0 commented on a diff in pull request #11574: Fix schema name in table config during controller startup

2023-09-29 Thread via GitHub
xiangfu0 commented on code in PR #11574: URL: https://github.com/apache/pinot/pull/11574#discussion_r1340932616 ## pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java: ## @@ -549,6 +555,69 @@ protected void configure() {

[GitHub] [pinot] dang-stripe commented on pull request #11650: Support omitting time values in SimpleSegmentNameGenerator

2023-09-29 Thread via GitHub
dang-stripe commented on PR #11650: URL: https://github.com/apache/pinot/pull/11650#issuecomment-1740356183 Yes, we still want to use the global sequence ID which isn't supported by the `FixedSegmentNameGenerator`. Another option here is to update `FixedSegmentNameGenerator` to support

[GitHub] [pinot] vvivekiyer opened a new pull request, #11710: Throw exception if response size exceeds thresholds

2023-09-29 Thread via GitHub
vvivekiyer opened a new pull request, #11710: URL: https://github.com/apache/pinot/pull/11710 When server responds for a query with a large response, the broker can potentially crash with direct memory OOM. In PR https://github.com/apache/pinot/pull/11496 - a fix was added to