Re: [I] MultiValue Boolean Data Type Failing on Realtime Ingestion [pinot]

2024-07-17 Thread via GitHub
yashmayya commented on issue #13604: URL: https://github.com/apache/pinot/issues/13604#issuecomment-2233298045 @Jayesh-Asrani what's the exact exception you're seeing during the real time ingestion? I tried ingesting some batch data with an MV boolean column, and even that fails with the

[PR] Bump com.google.errorprone:error_prone_annotations from 2.28.0 to 2.29.0 [pinot]

2024-07-17 Thread via GitHub
dependabot[bot] opened a new pull request, #13642: URL: https://github.com/apache/pinot/pull/13642 Bumps [com.google.errorprone:error_prone_annotations](https://github.com/google/error-prone) from 2.28.0 to 2.29.0. Release notes Sourced from

[PR] Bump org.roaringbitmap:RoaringBitmap from 1.2.0 to 1.2.1 [pinot]

2024-07-17 Thread via GitHub
dependabot[bot] opened a new pull request, #13641: URL: https://github.com/apache/pinot/pull/13641 Bumps [org.roaringbitmap:RoaringBitmap](https://github.com/RoaringBitmap/RoaringBitmap) from 1.2.0 to 1.2.1. Release notes Sourced from

[PR] Bump com.azure:azure-identity from 1.13.0 to 1.13.1 [pinot]

2024-07-17 Thread via GitHub
dependabot[bot] opened a new pull request, #13640: URL: https://github.com/apache/pinot/pull/13640 Bumps [com.azure:azure-identity](https://github.com/Azure/azure-sdk-for-java) from 1.13.0 to 1.13.1. Release notes Sourced from

Re: [PR] Always use Unsafe as the default PinotBufferFactory [pinot]

2024-07-17 Thread via GitHub
codecov-commenter commented on PR #13639: URL: https://github.com/apache/pinot/pull/13639#issuecomment-223317 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13639?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [PR] Remove larray [pinot]

2024-07-17 Thread via GitHub
gortiz commented on PR #13545: URL: https://github.com/apache/pinot/pull/13545#issuecomment-2233014899 I've just open #13639 to disable LArray by default as a first step to end up removing it completely -- This is an automated message from the Apache Git Service. To respond to the

[PR] Always use Unsafe as the default PinotBufferFactory [pinot]

2024-07-17 Thread via GitHub
gortiz opened a new pull request, #13639: URL: https://github.com/apache/pinot/pull/13639 We have been using UnsafePinotBufferFactory for a while with no regression detected. I think it is a good time to start working on the plan to remove LArray dependency (see #12810) This PR

Re: [PR] Improve performance of DataBlock serde [pinot]

2024-07-17 Thread via GitHub
gortiz commented on code in PR #13303: URL: https://github.com/apache/pinot/pull/13303#discussion_r1680820132 ## pinot-common/src/main/java/org/apache/pinot/common/datablock/BaseDataBlock.java: ## @@ -109,92 +114,27 @@ public BaseDataBlock(int numRows, @Nullable DataSchema

Re: [I] Flaky metric: llcPartitionConsuming [pinot]

2024-07-17 Thread via GitHub
tibrewalpratik17 commented on issue #13629: URL: https://github.com/apache/pinot/issues/13629#issuecomment-2232962841 I found this comment -- https://github.com/apache/pinot/pull/6887#issuecomment-837094283 where one of the concerns was that removing the metric while a new consuming

Re: [PR] Refactor function registry for multi-stage engine [pinot]

2024-07-17 Thread via GitHub
gortiz commented on code in PR #13573: URL: https://github.com/apache/pinot/pull/13573#discussion_r1680795734 ## pinot-query-runtime/src/test/resources/queries/UDFAggregates.json: ## @@ -150,15 +150,15 @@ }, { "sql": "SELECT PERCENTILE_TDIGEST(float_col,

Re: [PR] Refactor function registry for multi-stage engine [pinot]

2024-07-17 Thread via GitHub
gortiz commented on code in PR #13573: URL: https://github.com/apache/pinot/pull/13573#discussion_r1680792812 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/operands/FunctionOperand.java: ## @@ -47,9 +48,32 @@ public

Re: [I] Flaky metric: llcPartitionConsuming [pinot]

2024-07-17 Thread via GitHub
tibrewalpratik17 commented on issue #13629: URL: https://github.com/apache/pinot/issues/13629#issuecomment-2232917058 > Have you figured out thy the metric is not emitted? I'm not entirely sure, but the `cleanupMetrics` method might be the cause. It specifically removes the

[I] flaky test for S3PinotFSTest [pinot]

2024-07-17 Thread via GitHub
xiangfu0 opened a new issue, #13638: URL: https://github.com/apache/pinot/issues/13638 ``` Error: org.apache.pinot.plugin.filesystem.S3PinotFSTest.testMoveFile -- Time elapsed: 1.377 s <<< FAILURE! java.lang.AssertionError: expected [2024-07-17T08:16:14Z] but found

Re: [PR] Accept Request in controller AccessControl [pinot]

2024-07-17 Thread via GitHub
codecov-commenter commented on PR #13637: URL: https://github.com/apache/pinot/pull/13637#issuecomment-2232803602 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13637?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [I] Accessing Peer certificates in Access Control [pinot]

2024-07-17 Thread via GitHub
mgranderath commented on issue #13556: URL: https://github.com/apache/pinot/issues/13556#issuecomment-2232727805 Added a draft of our solution [here](https://github.com/apache/pinot/pull/13637/). I am also very open to entirely changing the approach if there are better ideas! -- This is

[PR] Accept Request in controller AccessControl [pinot]

2024-07-17 Thread via GitHub
mgranderath opened a new pull request, #13637: URL: https://github.com/apache/pinot/pull/13637 ### Context - https://github.com/apache/pinot/issues/13556 This allows for using properties of the `Request` object in the `AccessControl` to achieve things like using the peer TLS

Re: [PR] fixing TLS configuration for http clients [pinot]

2024-07-17 Thread via GitHub
mgranderath commented on PR #13477: URL: https://github.com/apache/pinot/pull/13477#issuecomment-2232662769 @Jackie-Jiang made the adjustment. Version 5 should fix the issue where the connection factory is not being picked up from the `PoolingHttpClientConnectionManager` so this has become

Re: [PR] Do not expand large SEARCH(input, Sarg) [pinot]

2024-07-17 Thread via GitHub
gortiz closed pull request #13605: Do not expand large SEARCH(input, Sarg) URL: https://github.com/apache/pinot/pull/13605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Do not expand large SEARCH(input, Sarg) [pinot]

2024-07-17 Thread via GitHub
gortiz commented on PR #13605: URL: https://github.com/apache/pinot/pull/13605#issuecomment-2232651228 Even we may need to apply something like this in the future, we are going to let it be in standby for now given we don't know the impact that join main have, specially in NOT IN case --

Re: [PR] Support more formats in TimestampUtils [pinot]

2024-07-17 Thread via GitHub
gortiz merged PR #13609: URL: https://github.com/apache/pinot/pull/13609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Support more formats in TimestampUtils [pinot]

2024-07-17 Thread via GitHub
gortiz commented on PR #13609: URL: https://github.com/apache/pinot/pull/13609#issuecomment-2232640284 > Should we also add the support for ISO format such as 2024-07-12T15:32:36? I think we should define the exact syntax we support. IMO we should include what we are accepting right

Re: [PR] Fix TextMatchFilterOptimizer `or not` behavior [pinot]

2024-07-17 Thread via GitHub
jasperjiaguo commented on PR #13533: URL: https://github.com/apache/pinot/pull/13533#issuecomment-2232605900 nit: I think `x OR NOT y` is equal to `x NOT y` (from my testing) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Fixing boolean[] indexing failure during ingestion [pinot]

2024-07-17 Thread via GitHub
xiangfu0 commented on PR #13630: URL: https://github.com/apache/pinot/pull/13630#issuecomment-2232548249 > `boolean[]` should not flow into the segment creation engine. Its value should already be replaced with internal storage format within `DataTypeTransformer` Seems that the

Re: [PR] add metadata ttl field to DedupConfig [pinot]

2024-07-17 Thread via GitHub
codecov-commenter commented on PR #13636: URL: https://github.com/apache/pinot/pull/13636#issuecomment-223253 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13636?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

[PR] add metadata ttl field to DedupConfig [pinot]

2024-07-16 Thread via GitHub
zhtaoxiang opened a new pull request, #13636: URL: https://github.com/apache/pinot/pull/13636 Similar to `UpsertConfig`, we want to add a `metadataTTL` to `DedupConfig`. With this config, we can clean up stale dedup metadata. This PR only adds this config. How to use this config to

Re: [PR] Automatically detect whether a v1 query could have run on the v2 query engine [pinot]

2024-07-16 Thread via GitHub
yashmayya commented on code in PR #13628: URL: https://github.com/apache/pinot/pull/13628#discussion_r1680430202 ## pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java: ## @@ -190,14 +194,32 @@ public List getTableNamesForQuery(String sqlQuery) {

Re: [PR] Reformatting files of Upsert-related code, not introducing any code changes [pinot]

2024-07-16 Thread via GitHub
codecov-commenter commented on PR #13635: URL: https://github.com/apache/pinot/pull/13635#issuecomment-2232430358 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13635?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [PR] Avoid collision in SegmentLocks [pinot]

2024-07-16 Thread via GitHub
swaminathanmanish commented on code in PR #13632: URL: https://github.com/apache/pinot/pull/13632#discussion_r1680401702 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SegmentLocks.java: ## @@ -18,30 +18,25 @@ */ package

[PR] Reformatting files of Upsert-related code, not introducing any code changes [pinot]

2024-07-16 Thread via GitHub
tibrewalpratik17 opened a new pull request, #13635: URL: https://github.com/apache/pinot/pull/13635 label: `refactor` Recently, while working on files affected by the PR, I attempted to use IntelliJ's reformat-file feature. However, it introduced many unrelated changes, so I

Re: [PR] Avoid collision in SegmentLocks [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang commented on code in PR #13632: URL: https://github.com/apache/pinot/pull/13632#discussion_r1680390383 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SegmentLocks.java: ## @@ -18,30 +18,25 @@ */ package

Re: [PR] Avoid collision in SegmentLocks [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13632: URL: https://github.com/apache/pinot/pull/13632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] fix intermittent failure in S3PinotFSTest [pinot]

2024-07-16 Thread via GitHub
codecov-commenter commented on PR #13634: URL: https://github.com/apache/pinot/pull/13634#issuecomment-2232294564 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13634?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

[PR] fix intermittent failure in S3PinotFSTest [pinot]

2024-07-16 Thread via GitHub
sullis opened a new pull request, #13634: URL: https://github.com/apache/pinot/pull/13634 Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incompat` 6.

Re: [PR] Added support to upload segments in batch mode with METADATA upload type [pinot]

2024-07-16 Thread via GitHub
swaminathanmanish commented on code in PR #13597: URL: https://github.com/apache/pinot/pull/13597#discussion_r1680289873 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java: ## @@ -555,6 +750,67 @@ public

Re: [PR] Cleanup of the method hasAccess. [pinot]

2024-07-16 Thread via GitHub
codecov-commenter commented on PR #13633: URL: https://github.com/apache/pinot/pull/13633#issuecomment-2232160351 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13633?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [PR] Added support to upload segments in batch mode with METADATA upload type [pinot]

2024-07-16 Thread via GitHub
swaminathanmanish commented on PR #13597: URL: https://github.com/apache/pinot/pull/13597#issuecomment-2232159425 > > I suppose the scenario you are trying to optimize is when each minion uploads more than one segment (presumably two, to split data over a time boundary?) and hundreds of

Re: [PR] Added support to upload segments in batch mode with METADATA upload type [pinot]

2024-07-16 Thread via GitHub
swaminathanmanish commented on code in PR #13597: URL: https://github.com/apache/pinot/pull/13597#discussion_r1680255251 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ## @@ -2327,6 +2327,82 @@ public void

Re: [PR] Avoid collision in SegmentLocks [pinot]

2024-07-16 Thread via GitHub
klsince commented on code in PR #13632: URL: https://github.com/apache/pinot/pull/13632#discussion_r1680227655 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SegmentLocks.java: ## @@ -18,30 +18,25 @@ */ package org.apache.pinot.segment.local.utils;

[PR] Cleanup of the method hasAccess. [pinot]

2024-07-16 Thread via GitHub
abhioncbr opened a new pull request, #13633: URL: https://github.com/apache/pinot/pull/13633 **Labels** - `cleanup` **Description** As per the[ discussion comment](https://github.com/apache/pinot/pull/13574#discussion_r1676402235) on the

Re: [PR] Avoid collision in SegmentLocks [pinot]

2024-07-16 Thread via GitHub
codecov-commenter commented on PR #13632: URL: https://github.com/apache/pinot/pull/13632#issuecomment-2232078122 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13632?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [PR] API to expose gauge metric values on minion [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang commented on code in PR #13550: URL: https://github.com/apache/pinot/pull/13550#discussion_r1680211463 ## pinot-minion/src/main/java/org/apache/pinot/minion/api/resources/PinotMinionMetricsResource.java: ## @@ -0,0 +1,68 @@ +/** + * Licensed to the Apache Software

Re: [PR] API to expose gauge metric values on minion [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang commented on code in PR #13550: URL: https://github.com/apache/pinot/pull/13550#discussion_r1680210207 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java: ## @@ -494,6 +494,16 @@ public void addValueToGlobalGauge(final G gauge, final

Re: [I] Flaky metric: llcPartitionConsuming [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang commented on issue #13629: URL: https://github.com/apache/pinot/issues/13629#issuecomment-2232072921 @tibrewalpratik17 Have you figured out thy the metric is not emitted? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Flaky metric: llcPartitionConsuming [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang commented on issue #13629: URL: https://github.com/apache/pinot/issues/13629#issuecomment-2232071857 cc @zhtaoxiang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Fix bug when evaluating resource status during service startup check [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang commented on PR #13541: URL: https://github.com/apache/pinot/pull/13541#issuecomment-2232071149 My bad, now I see the problem. Should we consider first looping over the ideal state and create a map from serving partition to state, then only loop over this map in the second

Re: [PR] add assertions for S3PinotFS move [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13631: URL: https://github.com/apache/pinot/pull/13631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Fix TextMatchFilterOptimizer `or not` behavior [pinot]

2024-07-16 Thread via GitHub
jasperjiaguo commented on code in PR #13533: URL: https://github.com/apache/pinot/pull/13533#discussion_r1680198511 ## pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/QueryOptimizerTest.java: ## @@ -293,6 +293,11 @@ public void testQueries() { + "AND

Re: [PR] Fix TextMatchFilterOptimizer `or not` behavior [pinot]

2024-07-16 Thread via GitHub
jasperjiaguo commented on code in PR #13533: URL: https://github.com/apache/pinot/pull/13533#discussion_r1680197805 ## pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/QueryOptimizerTest.java: ## @@ -293,6 +293,11 @@ public void testQueries() { + "AND

Re: [PR] Fix TextMatchFilterOptimizer `or not` behavior [pinot]

2024-07-16 Thread via GitHub
jasperjiaguo commented on code in PR #13533: URL: https://github.com/apache/pinot/pull/13533#discussion_r1680197805 ## pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/QueryOptimizerTest.java: ## @@ -293,6 +293,11 @@ public void testQueries() { + "AND

Re: [PR] add assertions for S3PinotFS move [pinot]

2024-07-16 Thread via GitHub
codecov-commenter commented on PR #13631: URL: https://github.com/apache/pinot/pull/13631#issuecomment-2231945461 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13631?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

[PR] add assertions for S3PinotFS move [pinot]

2024-07-16 Thread via GitHub
sullis opened a new pull request, #13631: URL: https://github.com/apache/pinot/pull/13631 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Fix bug when evaluating resource status during service startup check [pinot]

2024-07-16 Thread via GitHub
itschrispeck commented on PR #13541: URL: https://github.com/apache/pinot/pull/13541#issuecomment-2231869927 > I don't fully follow the change. When there is no partition assigned to the given instance, the loop should all be skipped, and it should return GOOD right? Not sure I

Re: [PR] Fixing boolean[] indexing failure during ingestion [pinot]

2024-07-16 Thread via GitHub
xiangfu0 commented on PR #13630: URL: https://github.com/apache/pinot/pull/13630#issuecomment-2231855672 > `boolean[]` should not flow into the segment creation engine. Its value should already be replaced with internal storage format within `DataTypeTransformer` good point! I

Re: [PR] Added support to upload segments in batch mode with METADATA upload type [pinot]

2024-07-16 Thread via GitHub
jtao15 commented on PR #13597: URL: https://github.com/apache/pinot/pull/13597#issuecomment-2231850381 @rajagopr Thank you for the interesting PR. I'd like to take a look soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Fixing boolean[] indexing failure during ingestion [pinot]

2024-07-16 Thread via GitHub
codecov-commenter commented on PR #13630: URL: https://github.com/apache/pinot/pull/13630#issuecomment-2231847982 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13630?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [I] Use Foreign Memory API, introduced in Java 22 [pinot]

2024-07-16 Thread via GitHub
aditya0811 commented on issue #12809: URL: https://github.com/apache/pinot/issues/12809#issuecomment-2231704910 Are these still open questions? I was unable to approach this as part of this issue. a) When we are adding src/main/java21, by configuring a different profile, how are we

Re: [PR] Bump com.azure:azure-core-http-netty from 1.15.1 to 1.15.2 [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13621: URL: https://github.com/apache/pinot/pull/13621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Bump confluent.version from 7.6.1 to 7.6.2 [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13620: URL: https://github.com/apache/pinot/pull/13620 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Bump com.azure:azure-core from 1.49.1 to 1.50.0 [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13624: URL: https://github.com/apache/pinot/pull/13624 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Bump com.h2database:h2 from 2.2.224 to 2.3.230 [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13622: URL: https://github.com/apache/pinot/pull/13622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Bump org.apache.kafka:kafka-clients from 2.8.1 to 7.6.2-ce [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang closed pull request #13625: Bump org.apache.kafka:kafka-clients from 2.8.1 to 7.6.2-ce URL: https://github.com/apache/pinot/pull/13625 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Bump commons-codec:commons-codec from 1.17.0 to 1.17.1 [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13623: URL: https://github.com/apache/pinot/pull/13623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Bump org.apache.kafka:kafka-clients from 2.8.1 to 7.6.2-ce [pinot]

2024-07-16 Thread via GitHub
dependabot[bot] commented on PR #13625: URL: https://github.com/apache/pinot/pull/13625#issuecomment-2231636486 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [I] Multi-stage engine might break MV filter semantic [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang closed issue #13595: Multi-stage engine might break MV filter semantic URL: https://github.com/apache/pinot/issues/13595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Make ARRAY_TO_MV a not deterministic function [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13618: URL: https://github.com/apache/pinot/pull/13618 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] UnsupportedOperationException when querying a converted dictionary -> noDictionary column [pinot]

2024-07-16 Thread via GitHub
jadami10 commented on issue #11893: URL: https://github.com/apache/pinot/issues/11893#issuecomment-2231604530 we only saw it on this 1 table. So I've assumed it's somewhat tied to JSON columns -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Apply SEARCH expand rule in the end [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang merged PR #13614: URL: https://github.com/apache/pinot/pull/13614 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Added support to upload segments in batch mode with METADATA upload type [pinot]

2024-07-16 Thread via GitHub
rajagopr commented on PR #13597: URL: https://github.com/apache/pinot/pull/13597#issuecomment-2231578183 > I suppose the scenario you are trying to optimize is when each minion uploads more than one segment (presumably two, to split data over a time boundary?) and hundreds of minions

Re: [PR] Apply SEARCH expand rule in the end [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang commented on PR #13614: URL: https://github.com/apache/pinot/pull/13614#issuecomment-2231579057 Discussed offline. We will merge this one to reduce the overhead, in the meanwhile trying to figure out a thorough solution. -- This is an automated message from the Apache Git

Re: [PR] Added support to upload segments in batch mode with METADATA upload type [pinot]

2024-07-16 Thread via GitHub
Jackie-Jiang commented on PR #13597: URL: https://github.com/apache/pinot/pull/13597#issuecomment-2231572794 @mcvsubbu We want to optimize the scenario where multiple clients trying to push segments, each with multiple segments to push, not necessary from minion. The bottleneck right now

Re: [I] UnsupportedOperationException when querying a converted dictionary -> noDictionary column [pinot]

2024-07-16 Thread via GitHub
shounakmk219 commented on issue #11893: URL: https://github.com/apache/pinot/issues/11893#issuecomment-2231519384 I see. I will try to test out few hypothesis to repro this with the old code as well. Also wanted to confirm if this issue was seen on just one table or do you remember any

Re: [I] UnsupportedOperationException when querying a converted dictionary -> noDictionary column [pinot]

2024-07-16 Thread via GitHub
jadami10 commented on issue #11893: URL: https://github.com/apache/pinot/issues/11893#issuecomment-2231445800 unfortunately this issue is all the context I have as the table has long since retentioned any data that did not have that column -- This is an automated message from the Apache

Re: [PR] Added support to upload segments in batch mode with METADATA upload type [pinot]

2024-07-16 Thread via GitHub
kishoreg commented on code in PR #13597: URL: https://github.com/apache/pinot/pull/13597#discussion_r1679749949 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java: ## @@ -555,6 +750,67 @@ public void

Re: [PR] Added support to upload segments in batch mode with METADATA upload type [pinot]

2024-07-16 Thread via GitHub
mcvsubbu commented on PR #13597: URL: https://github.com/apache/pinot/pull/13597#issuecomment-2231327374 I suppose the scenario you are trying to optimize is when each minion uploads more than one segment (presumably two, to split data over a time boundary?) and hundreds of minions working

[I] Flaky metric: llcPartitionConsuming [pinot]

2024-07-16 Thread via GitHub
tibrewalpratik17 opened a new issue, #13629: URL: https://github.com/apache/pinot/issues/13629 We use `llcPartitionConsuming` to monitor whether Pinot is ingesting data for every partition across all replicas. However, we have been experiencing numerous flaky alerts because Pinot sometimes

Re: [I] _enableSnapshot loads upsert segments from snapshot even if _enablePreload is disabled [pinot]

2024-07-16 Thread via GitHub
tibrewalpratik17 commented on issue #13466: URL: https://github.com/apache/pinot/issues/13466#issuecomment-2231278395 Closing as the behaviour is fixed in #13579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] _enableSnapshot loads upsert segments from snapshot even if _enablePreload is disabled [pinot]

2024-07-16 Thread via GitHub
tibrewalpratik17 closed issue #13466: _enableSnapshot loads upsert segments from snapshot even if _enablePreload is disabled URL: https://github.com/apache/pinot/issues/13466 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] UnsupportedOperationException when querying a converted dictionary -> noDictionary column [pinot]

2024-07-16 Thread via GitHub
shounakmk219 commented on issue #11893: URL: https://github.com/apache/pinot/issues/11893#issuecomment-2231122702 Hey @jadami10 , I tried to follow the mentioned steps with the latest master and was not able to reproduce the issue. Would like to get some more context on this. >

Re: [PR] Automatically detect whether a v1 query could have run on the v2 query engine [pinot]

2024-07-16 Thread via GitHub
codecov-commenter commented on PR #13628: URL: https://github.com/apache/pinot/pull/13628#issuecomment-2231068266 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13628?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [PR] Database query quota [pinot]

2024-07-16 Thread via GitHub
vrajat commented on code in PR #13544: URL: https://github.com/apache/pinot/pull/13544#discussion_r1679509981 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/QuotaConfig.java: ## @@ -35,7 +35,8 @@ public class QuotaConfig extends BaseJsonConfig { private static

Re: [PR] Automatically detect whether a v1 query could have run on the v2 query engine [pinot]

2024-07-16 Thread via GitHub
yashmayya commented on PR #13628: URL: https://github.com/apache/pinot/pull/13628#issuecomment-2230936245 Metric name and broker config are up for debate (naming is hard)  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Add logs indicating the star-tree config diff to understand the rebuild purpose [pinot]

2024-07-16 Thread via GitHub
codecov-commenter commented on PR #13627: URL: https://github.com/apache/pinot/pull/13627#issuecomment-2230935692 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13627?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

[PR] Automatically detect whether a v1 query could have run on the v2 query engine [pinot]

2024-07-16 Thread via GitHub
yashmayya opened a new pull request, #13628: URL: https://github.com/apache/pinot/pull/13628 - With the v2 / multi-stage query engine becoming more robust and feature complete (https://github.com/apache/pinot/pulls?q=is%3Apr+label%3Amulti-stage+is%3Aclosed), existing users might want to

[PR] Add logs indicating the star-tree config diff to understand the rebuild purpose [pinot]

2024-07-16 Thread via GitHub
shounakmk219 opened a new pull request, #13627: URL: https://github.com/apache/pinot/pull/13627 This PR prints the `StarTreeV2BuilderConfig` for existing star-trees on the segment and the incoming star-trees from the updated table config. This will help to understand why certain

[I] Realtime Ingestion Stalled : Consuming Segments Not converting to Completed [pinot]

2024-07-16 Thread via GitHub
Jayesh-Asrani opened a new issue, #13626: URL: https://github.com/apache/pinot/issues/13626 1. Steps to re-produce : Create a Realtime table with a sorted column id and few other Multi value columns 2. Ingest wrong data into pinot for the MV columns where the columns are represented

Re: [PR] Database query quota [pinot]

2024-07-16 Thread via GitHub
shounakmk219 commented on code in PR #13544: URL: https://github.com/apache/pinot/pull/13544#discussion_r1679336060 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/QuotaConfig.java: ## @@ -35,7 +35,8 @@ public class QuotaConfig extends BaseJsonConfig { private

Re: [PR] Database query quota [pinot]

2024-07-16 Thread via GitHub
shounakmk219 commented on code in PR #13544: URL: https://github.com/apache/pinot/pull/13544#discussion_r1679333442 ## pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java: ## @@ -27,6 +27,8 @@ import

Re: [PR] Database query quota [pinot]

2024-07-16 Thread via GitHub
shounakmk219 commented on code in PR #13544: URL: https://github.com/apache/pinot/pull/13544#discussion_r1679330670 ## pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java: ## @@ -46,21 +55,31 @@ import org.slf4j.Logger;

Re: [PR] Database query quota [pinot]

2024-07-16 Thread via GitHub
shounakmk219 commented on code in PR #13544: URL: https://github.com/apache/pinot/pull/13544#discussion_r1679329428 ## pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java: ## @@ -230,6 +248,72 @@ private void

Re: [PR] API to expose gauge metric values on minion [pinot]

2024-07-16 Thread via GitHub
shounakmk219 commented on PR #13550: URL: https://github.com/apache/pinot/pull/13550#issuecomment-2230801206 @Jackie-Jiang based on your inputs updated the code to expose just the gauge values for now. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Database query quota [pinot]

2024-07-16 Thread via GitHub
vrajat commented on code in PR #13544: URL: https://github.com/apache/pinot/pull/13544#discussion_r1679268404 ## pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java: ## @@ -46,21 +55,31 @@ import org.slf4j.Logger; import

[PR] Bump org.apache.kafka:kafka-clients from 2.8.1 to 7.6.2-ce [pinot]

2024-07-16 Thread via GitHub
dependabot[bot] opened a new pull request, #13625: URL: https://github.com/apache/pinot/pull/13625 Bumps org.apache.kafka:kafka-clients from 2.8.1 to 7.6.2-ce. [![Dependabot compatibility

[PR] Bump com.azure:azure-core from 1.49.1 to 1.50.0 [pinot]

2024-07-16 Thread via GitHub
dependabot[bot] opened a new pull request, #13624: URL: https://github.com/apache/pinot/pull/13624 Bumps [com.azure:azure-core](https://github.com/Azure/azure-sdk-for-java) from 1.49.1 to 1.50.0. Release notes Sourced from

[PR] Bump commons-codec:commons-codec from 1.17.0 to 1.17.1 [pinot]

2024-07-16 Thread via GitHub
dependabot[bot] opened a new pull request, #13623: URL: https://github.com/apache/pinot/pull/13623 Bumps [commons-codec:commons-codec](https://github.com/apache/commons-codec) from 1.17.0 to 1.17.1. Changelog Sourced from

[PR] Bump com.h2database:h2 from 2.2.224 to 2.3.230 [pinot]

2024-07-16 Thread via GitHub
dependabot[bot] opened a new pull request, #13622: URL: https://github.com/apache/pinot/pull/13622 Bumps [com.h2database:h2](https://github.com/h2database/h2database) from 2.2.224 to 2.3.230. Release notes Sourced from

[PR] Bump com.azure:azure-core-http-netty from 1.15.1 to 1.15.2 [pinot]

2024-07-16 Thread via GitHub
dependabot[bot] opened a new pull request, #13621: URL: https://github.com/apache/pinot/pull/13621 Bumps [com.azure:azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.15.1 to 1.15.2. Release notes Sourced from

[PR] Bump confluent.version from 7.6.1 to 7.6.2 [pinot]

2024-07-16 Thread via GitHub
dependabot[bot] opened a new pull request, #13620: URL: https://github.com/apache/pinot/pull/13620 Bumps `confluent.version` from 7.6.1 to 7.6.2. Updates `io.confluent:kafka-schema-registry-client` from 7.6.1 to 7.6.2 Commits See full diff in

Re: [I] Use Foreign Memory API, introduced in Java 22 [pinot]

2024-07-16 Thread via GitHub
gortiz commented on issue #12809: URL: https://github.com/apache/pinot/issues/12809#issuecomment-2230610695 > we should also care for this because of End of Live (EOL) for Red Hat builds of Openjdk v11 on 30 Oct 2024 see e.g. https://endoflife.date/redhat-build-of-openjdk Just to

Re: [PR] Make ARRAY_TO_MV a not deterministic function [pinot]

2024-07-16 Thread via GitHub
yashmayya commented on PR #13618: URL: https://github.com/apache/pinot/pull/13618#issuecomment-2230571414 That makes sense, thanks for the explanation! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Make ARRAY_TO_MV a not deterministic function [pinot]

2024-07-16 Thread via GitHub
gortiz commented on code in PR #13618: URL: https://github.com/apache/pinot/pull/13618#discussion_r1679161515 ## pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineIntegrationTest.java: ## @@ -704,6 +704,26 @@ public void

  1   2   3   4   5   6   7   8   9   10   >