Re: [PR] Introduce retries while creating stream message decoder for more robustness [pinot]

2024-05-01 Thread via GitHub
swaminathanmanish commented on code in PR #13036: URL: https://github.com/apache/pinot/pull/13036#discussion_r1585479559 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -1504,14 +1510,24 @@ public

Re: [I] Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: DFM25B81QM9RZ2WX, Extended Request ID: 7vSTw22PgYLApWtVYc4blcMDFPRq1N

2024-05-01 Thread via GitHub
swaminathanmanish commented on issue #11446: URL: https://github.com/apache/pinot/issues/11446#issuecomment-2089592328 > @swaminathanmanish the above link is not working, can you please share the workaround here? @xiangfu0 had suggested the following in that slack thread to the

Re: [PR] bugfix: prevent background merges on the realtime lucene index [pinot]

2024-05-01 Thread via GitHub
codecov-commenter commented on PR #13050: URL: https://github.com/apache/pinot/pull/13050#issuecomment-2089491794 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13050?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

[PR] bugfix: prevent background merges on the realtime lucene index [pinot]

2024-05-01 Thread via GitHub
itschrispeck opened a new pull request, #13050: URL: https://github.com/apache/pinot/pull/13050 https://github.com/apache/pinot/pull/12744 changed the Lucene index build process to commit the realtime Lucene index, and reuse it in the immutable segment. `.commit()` is called at the

Re: [I] Dependency issue detected in AdminConsoleIntegrationTest [pinot]

2024-05-01 Thread via GitHub
abhioncbr commented on issue #13047: URL: https://github.com/apache/pinot/issues/13047#issuecomment-2089389461 Similar [issue](https://github.com/apache/pinot/issues/5306) ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] dynamic chunk sizing for v4 raw forward index [pinot]

2024-05-01 Thread via GitHub
itschrispeck commented on code in PR #12945: URL: https://github.com/apache/pinot/pull/12945#discussion_r1586948934 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/SingleValueVarByteRawIndexCreator.java: ## @@ -70,23 +69,32 @@ public

(pinot) branch master updated (5d1dc73cc6 -> 076cd404c4)

2024-05-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 5d1dc73cc6 log the log rate limiter rate for dropped broker logs (#13041) add 076cd404c4 Cleanup deprecated

Re: [PR] Cleanup deprecated query options [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang merged PR #13040: URL: https://github.com/apache/pinot/pull/13040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] #12635 Bug Fix createDictionaryForColumn does not take into account inverted index [pinot]

2024-05-01 Thread via GitHub
deepthi912 opened a new pull request, #13048: URL: https://github.com/apache/pinot/pull/13048 **Fix:** 1) Precheck condition would be required if the column has inverted index enabled. 2) When the column has inverted index enabled, we should require dictionary for the column and so

Re: [PR] Add protobuf codegen decoder [pinot]

2024-05-01 Thread via GitHub
rseetham commented on code in PR #12980: URL: https://github.com/apache/pinot/pull/12980#discussion_r1586863328 ## pinot-plugins/pinot-input-format/pinot-protobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufUtils.java: ## @@ -48,21 +51,156 @@ public

Re: [PR] Add protobuf codegen decoder [pinot]

2024-05-01 Thread via GitHub
rseetham commented on code in PR #12980: URL: https://github.com/apache/pinot/pull/12980#discussion_r1586859081 ## pinot-plugins/pinot-input-format/pinot-protobuf/src/main/java/com/google/protobuf/ProtobufInternalUtils.java: ## @@ -0,0 +1,28 @@ +/** + * Licensed to the Apache

Re: [PR] Multi stage stats [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1586820017 ## pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java: ## @@ -36,108 +39,129 @@ */ public class MetadataBlock extends

(pinot) branch master updated: log the log rate limiter rate for dropped broker logs (#13041)

2024-05-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 5d1dc73cc6 log the log rate limiter rate for

Re: [PR] log the log rate limiter rate for dropped broker logs [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang merged PR #13041: URL: https://github.com/apache/pinot/pull/13041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] dynamic chunk sizing for v4 raw forward index [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang commented on code in PR #12945: URL: https://github.com/apache/pinot/pull/12945#discussion_r1586813726 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/SingleValueVarByteRawIndexCreator.java: ## @@ -70,23 +69,32 @@ public

Re: [I] [feature] Support parallel combine and disk spill for groupBy execution [pinot]

2024-05-01 Thread via GitHub
wirybeaver commented on issue #12080: URL: https://github.com/apache/pinot/issues/12080#issuecomment-2089118697 https://www.databend.com/blog/2024-04-12-towards-efficient-distributed-group-aggregation.md/ dataend execution is incredibly fast. they follow duckdb -- This is an

Re: [PR] Multi stage stats [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1586803055 ## pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java: ## @@ -36,108 +39,129 @@ */ public class MetadataBlock extends

Re: [I] Support new data type: `TimestampWithTimeZone` [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang commented on issue #13004: URL: https://github.com/apache/pinot/issues/13004#issuecomment-2089094786 Each `DataType` has a `getStoredType()`. `BOOLEAN` is stored as `INT`, `TIMESTAMP` is stored as `LONG`, `JSON` is stored as `STRING`. Take a look at its usage -- This is

(pinot) branch dependabot/maven/lucene.version-9.10.0 deleted (was ee54cfcc8c)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/lucene.version-9.10.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was ee54cfcc8c Bump lucene.version from 9.8.0 to 9.10.0 The revisions that were on

Re: [PR] Bump lucene.version from 9.8.0 to 9.10.0 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] commented on PR #12844: URL: https://github.com/apache/pinot/pull/12844#issuecomment-2089068913 OK, I won't notify you again about this release, but will get in touch when a new version is available. You can also ignore all major, minor, or patch releases for a dependency

Re: [PR] Bump lucene.version from 9.8.0 to 9.10.0 [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang closed pull request #12844: Bump lucene.version from 9.8.0 to 9.10.0 URL: https://github.com/apache/pinot/pull/12844 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

(pinot) branch master updated: Upgrade lucene to 9.10.0 and compatibility changes to code. (#12866)

2024-05-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c8b223f1fc Upgrade lucene to 9.10.0 and

Re: [PR] Upgrade lucene to 9.10.0 and compatibility changes to code. [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang merged PR #12866: URL: https://github.com/apache/pinot/pull/12866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(pinot) branch dependabot/maven/aws.sdk.version-2.25.42 deleted (was 6c07f999d3)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/aws.sdk.version-2.25.42 in repository https://gitbox.apache.org/repos/asf/pinot.git was 6c07f999d3 Bump aws.sdk.version from 2.25.40 to 2.25.42 The revisions that were

Re: [PR] Bump aws.sdk.version from 2.25.40 to 2.25.42 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] commented on PR #13044: URL: https://github.com/apache/pinot/pull/13044#issuecomment-2089049653 OK, I won't notify you again about this release, but will get in touch when a new version is available. You can also ignore all major, minor, or patch releases for a dependency

Re: [PR] Bump aws.sdk.version from 2.25.40 to 2.25.42 [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang closed pull request #13044: Bump aws.sdk.version from 2.25.40 to 2.25.42 URL: https://github.com/apache/pinot/pull/13044 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

(pinot) branch dependabot/maven/org.scala-lang-scala-library-2.13.14 deleted (was f4457a7149)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.scala-lang-scala-library-2.13.14 in repository https://gitbox.apache.org/repos/asf/pinot.git was f4457a7149 Bump org.scala-lang:scala-library from 2.11.12 to

Re: [PR] Bump org.scala-lang:scala-library from 2.11.12 to 2.13.14 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] commented on PR #13043: URL: https://github.com/apache/pinot/pull/13043#issuecomment-2089048940 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] Bump org.scala-lang:scala-library from 2.11.12 to 2.13.14 [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang closed pull request #13043: Bump org.scala-lang:scala-library from 2.11.12 to 2.13.14 URL: https://github.com/apache/pinot/pull/13043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

(pinot) branch dependabot/maven/com.adobe.testing-s3mock-testcontainers-3.7.2 deleted (was 6576394c0c)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.adobe.testing-s3mock-testcontainers-3.7.2 in repository https://gitbox.apache.org/repos/asf/pinot.git was 6576394c0c Bump com.adobe.testing:s3mock-testcontainers

Re: [PR] Bump com.adobe.testing:s3mock-testcontainers from 2.17.0 to 3.7.2 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] commented on PR #13042: URL: https://github.com/apache/pinot/pull/13042#issuecomment-2089047078 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] Bump com.adobe.testing:s3mock-testcontainers from 2.17.0 to 3.7.2 [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang closed pull request #13042: Bump com.adobe.testing:s3mock-testcontainers from 2.17.0 to 3.7.2 URL: https://github.com/apache/pinot/pull/13042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] fix merging null multi value in partial upsert [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang merged PR #13031: URL: https://github.com/apache/pinot/pull/13031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(pinot) branch master updated: fix merging null multi value in partial upsert (#13031)

2024-05-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 0f28a5cc6f fix merging null multi value in

Re: [PR] Cleanup deprecated query options [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang commented on PR #13040: URL: https://github.com/apache/pinot/pull/13040#issuecomment-2088984901 > LGTM, but there are multiple tests failing, so I guess we would need to fix something else Find the reason. Currently we only set query option when it is not empty, but the

Re: [PR] Cleanup deprecated query options [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang commented on code in PR #13040: URL: https://github.com/apache/pinot/pull/13040#discussion_r1586682821 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java: ## @@ -85,24 +84,22 @@ public BrokerResponse

(pinot) branch master updated: Enhancement: Sketch value aggregator performance (#13020)

2024-05-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ad7068619a Enhancement: Sketch value aggregator

Re: [PR] Enhancement: Sketch value aggregator performance [pinot]

2024-05-01 Thread via GitHub
Jackie-Jiang merged PR #13020: URL: https://github.com/apache/pinot/pull/13020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [HELM] Allow statefulsets to automatically roll when updating configmaps [pinot]

2024-05-01 Thread via GitHub
abhioncbr commented on issue #13046: URL: https://github.com/apache/pinot/issues/13046#issuecomment-2088802430 Looking into it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[I] [HELM] Allow statefulsets to automatically roll when updating configmaps [pinot]

2024-05-01 Thread via GitHub
nilaprasad opened a new issue, #13046: URL: https://github.com/apache/pinot/issues/13046 ### Background Currently, the helm charts don't roll the corresponding pods for a given statefulset when an associated configmap is updated and it needs to be done manually. This can lead to issues

(pinot) branch dependabot/maven/aws.sdk.version-2.25.42 created (now 6c07f999d3)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/aws.sdk.version-2.25.42 in repository https://gitbox.apache.org/repos/asf/pinot.git at 6c07f999d3 Bump aws.sdk.version from 2.25.40 to 2.25.42 No new revisions were

(pinot) branch dependabot/maven/aws.sdk.version-2.25.41 deleted (was dc3024ec0c)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/aws.sdk.version-2.25.41 in repository https://gitbox.apache.org/repos/asf/pinot.git was dc3024ec0c Bump aws.sdk.version from 2.25.40 to 2.25.41 The revisions that were

Re: [PR] Bump aws.sdk.version from 2.25.40 to 2.25.41 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] closed pull request #13033: Bump aws.sdk.version from 2.25.40 to 2.25.41 URL: https://github.com/apache/pinot/pull/13033 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

(pinot) branch dependabot/maven/com.google.cloud-libraries-bom-26.38.0 created (now 871042aa33)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.google.cloud-libraries-bom-26.38.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 871042aa33 Bump com.google.cloud:libraries-bom from 26.37.0 to

[PR] Bump com.google.cloud:libraries-bom from 26.37.0 to 26.38.0 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] opened a new pull request, #13045: URL: https://github.com/apache/pinot/pull/13045 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.37.0 to 26.38.0. Release notes Sourced from

Re: [PR] Bump aws.sdk.version from 2.25.40 to 2.25.41 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] commented on PR #13033: URL: https://github.com/apache/pinot/pull/13033#issuecomment-2088359632 Superseded by #13044. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Bump aws.sdk.version from 2.25.40 to 2.25.42 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] opened a new pull request, #13044: URL: https://github.com/apache/pinot/pull/13044 Bumps `aws.sdk.version` from 2.25.40 to 2.25.42. Updates `software.amazon.awssdk:bom` from 2.25.40 to 2.25.42 Updates `software.amazon.awssdk:apache-client` from 2.25.40 to 2.25.42

(pinot) branch dependabot/maven/org.scala-lang-scala-library-2.13.14 created (now f4457a7149)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.scala-lang-scala-library-2.13.14 in repository https://gitbox.apache.org/repos/asf/pinot.git at f4457a7149 Bump org.scala-lang:scala-library from 2.11.12 to

[PR] Bump org.scala-lang:scala-library from 2.11.12 to 2.13.14 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] opened a new pull request, #13043: URL: https://github.com/apache/pinot/pull/13043 Bumps [org.scala-lang:scala-library](https://github.com/scala/scala) from 2.11.12 to 2.13.14. Release notes Sourced from

(pinot) branch dependabot/maven/com.adobe.testing-s3mock-testcontainers-3.7.2 created (now 6576394c0c)

2024-05-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.adobe.testing-s3mock-testcontainers-3.7.2 in repository https://gitbox.apache.org/repos/asf/pinot.git at 6576394c0c Bump com.adobe.testing:s3mock-testcontainers

[PR] Bump com.adobe.testing:s3mock-testcontainers from 2.17.0 to 3.7.2 [pinot]

2024-05-01 Thread via GitHub
dependabot[bot] opened a new pull request, #13042: URL: https://github.com/apache/pinot/pull/13042 Bumps com.adobe.testing:s3mock-testcontainers from 2.17.0 to 3.7.2. [![Dependabot compatibility

Re: [PR] Upgrade lucene to 9.10.0 and compatibility changes to code. [pinot]

2024-05-01 Thread via GitHub
abhioncbr commented on PR #12866: URL: https://github.com/apache/pinot/pull/12866#issuecomment-2088309617 @Jackie-Jiang / @xiangfu0, Can you please review it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Multi stage stats [pinot]

2024-05-01 Thread via GitHub
gortiz commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1586072327 ## pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java: ## @@ -36,108 +39,129 @@ */ public class MetadataBlock extends BaseDataBlock {

Re: [PR] Multi stage stats [pinot]

2024-05-01 Thread via GitHub
gortiz commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1583030085 ## pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java: ## @@ -36,108 +39,129 @@ */ public class MetadataBlock extends BaseDataBlock {

Re: [I] Support new data type: `TimestampWithTimeZone` [pinot]

2024-05-01 Thread via GitHub
mghildiy commented on issue #13004: URL: https://github.com/apache/pinot/issues/13004#issuecomment-2088146342 Which part of codebase takes care of internal storage format? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Enhancement: Sketch value aggregator performance [pinot]

2024-05-01 Thread via GitHub
davecromberge commented on PR #13020: URL: https://github.com/apache/pinot/pull/13020#issuecomment-2088112915 @gortiz as a starting point, here are some empirical measurements that I took from a run in our production environment. The build time refers to the StarTree build time for a

Re: [PR] Enhancement: Sketch value aggregator performance [pinot]

2024-05-01 Thread via GitHub
davecromberge commented on PR #13020: URL: https://github.com/apache/pinot/pull/13020#issuecomment-2088109017 > The PR looks fine to me, but it is a bit strange to approve a performance wise PR without an actual benchmark we can reproduce. Could you include a JMH benchmark as part of the

Re: [PR] Enhancement: Sketch value aggregator performance [pinot]

2024-05-01 Thread via GitHub
davecromberge commented on code in PR #13020: URL: https://github.com/apache/pinot/pull/13020#discussion_r1585986210 ## pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/DistinctCountCPCSketchValueAggregatorTest.java: ## @@ -34,19 +34,18 @@ public

Re: [PR] Cleanup deprecated query options [pinot]

2024-05-01 Thread via GitHub
gortiz commented on PR #13040: URL: https://github.com/apache/pinot/pull/13040#issuecomment-2088064750 LGTM, but there are multiple tests failing, so I guess we would need to fix something else -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Cleanup deprecated query options [pinot]

2024-05-01 Thread via GitHub
gortiz commented on code in PR #13040: URL: https://github.com/apache/pinot/pull/13040#discussion_r1585949962 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java: ## @@ -85,24 +84,22 @@ public BrokerResponse