Re: [I] Retention manager not running for all tables [pinot]

2024-04-01 Thread via GitHub
vineethvp commented on issue #12660: URL: https://github.com/apache/pinot/issues/12660#issuecomment-2031120572 @Jackie-Jiang I'm running the latest version. I have re created some of my tables and the issue seems to be slightly different now. I can see some of the segments are being purged

Re: [PR] Upgrade Jersey related libraries [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12772: URL: https://github.com/apache/pinot/pull/12772#issuecomment-2031073775 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12772?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [I] Add array UDFs to operate multiple MV columns [pinot]

2024-04-01 Thread via GitHub
wirybeaver closed issue #11442: Add array UDFs to operate multiple MV columns URL: https://github.com/apache/pinot/issues/11442 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Download offline segments from peers if the deep storage copy is not available [pinot]

2024-04-01 Thread via GitHub
wirybeaver closed issue #9709: Download offline segments from peers if the deep storage copy is not available URL: https://github.com/apache/pinot/issues/9709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Update metric rules to emit database as a label [pinot]

2024-04-01 Thread via GitHub
shounakmk219 commented on PR #12739: URL: https://github.com/apache/pinot/pull/12739#issuecomment-2031052011 @zhtaoxiang Updated the PR description with testing performed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Update metric rules to emit database as a label [pinot]

2024-04-01 Thread via GitHub
shounakmk219 commented on code in PR #12739: URL: https://github.com/apache/pinot/pull/12739#discussion_r1547114035 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ValidationMetrics.java: ## @@ -133,124 +134,126 @@ public ValidationMetrics(PinotMetricsRegistry

Re: [PR] Update metric rules to emit database as a label [pinot]

2024-04-01 Thread via GitHub
shounakmk219 commented on code in PR #12739: URL: https://github.com/apache/pinot/pull/12739#discussion_r1547113167 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java: ## @@ -796,6 +809,7 @@ private void removeGaugeFromMetricRegistry(String

Re: [PR] Update metric rules to emit database as a label [pinot]

2024-04-01 Thread via GitHub
zhtaoxiang commented on PR #12739: URL: https://github.com/apache/pinot/pull/12739#issuecomment-2031038813 Can you please also explain how we tested it in the description section? Otherwise, looks good to me. -- This is an automated message from the Apache Git Service. To respond to the

[I] Speed up immutable json index build time [pinot]

2024-04-01 Thread via GitHub
wirybeaver opened a new issue, #12774: URL: https://github.com/apache/pinot/issues/12774 previously have an internal discussion with the team on translating mutable json index onto the immutable json index to avoid repeatedly parsing and flatten json docs. But It was temporarily blocked

Re: [PR] Update metric rules to emit database as a label [pinot]

2024-04-01 Thread via GitHub
zhtaoxiang commented on code in PR #12739: URL: https://github.com/apache/pinot/pull/12739#discussion_r1547100308 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ValidationMetrics.java: ## @@ -133,124 +134,126 @@ public ValidationMetrics(PinotMetricsRegistry

[I] Segments become Unavailable and fail to recover upon restarting server pods. [pinot]

2024-04-01 Thread via GitHub
rajatgupta063 opened a new issue, #12773: URL: https://github.com/apache/pinot/issues/12773 I have a pinot cluster running on AWS EKS. My pinot cluster is continuously reading data from kinesis stream. The cluster has around 700GB of data with more than 12B records. Total number of

Re: [PR] Update metric rules to emit database as a label [pinot]

2024-04-01 Thread via GitHub
zhtaoxiang commented on code in PR #12739: URL: https://github.com/apache/pinot/pull/12739#discussion_r1547100308 ## pinot-common/src/main/java/org/apache/pinot/common/metrics/ValidationMetrics.java: ## @@ -133,124 +134,126 @@ public ValidationMetrics(PinotMetricsRegistry

Re: [PR] Bug fix. Allow passing null http headers object to translateTableName [pinot]

2024-04-01 Thread via GitHub
shounakmk219 commented on PR #12764: URL: https://github.com/apache/pinot/pull/12764#issuecomment-2031015956 Not sure if implementations of `BrokerRequestHandler. handleRequest` are fine with a null header. `FineGrainedAccessControl.hasAccess` interface which is invoked in

Re: [PR] validate swagger endpoints [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12750: URL: https://github.com/apache/pinot/pull/12750#issuecomment-2030951981 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12750?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

(pinot) branch master updated (4f0bc11b95 -> 08fc2c7d62)

2024-04-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 4f0bc11b95 Cleanup Javax and Jakarta dependencies (#12760) add 08fc2c7d62 upgrade lmax disruptor to 4.0.0

Re: [PR] upgrade lmax disruptor to 4.0.0 [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang merged PR #12769: URL: https://github.com/apache/pinot/pull/12769 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Remove unused jcabi-log dependency [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12771: URL: https://github.com/apache/pinot/pull/12771#issuecomment-2030800513 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12771?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [I] Add support for DATETIMECONVERT() with Timestamp index [pinot]

2024-04-01 Thread via GitHub
dd-willgan commented on issue #10591: URL: https://github.com/apache/pinot/issues/10591#issuecomment-2030771578 Ended up finding a workaround for this internally, likely won't be able to get to it for now  . -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Set column major segment builder to be on by default [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12770: URL: https://github.com/apache/pinot/pull/12770#issuecomment-2030723847 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12770?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

[PR] Upgrade Jersey related libraries [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang opened a new pull request, #12772: URL: https://github.com/apache/pinot/pull/12772 - Jersey: 2.39 -> 3.1.5 - HK2: 2.6.1 -> 3.1.0 - Javassist: 3.19.0-GA -> 3.30.2-GA - Swagger: 1.6.9 -> 1.6.14 -- This is an automated message from the Apache Git Service. To respond to

[PR] Remove unused jcabi-log dependency [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang opened a new pull request, #12771: URL: https://github.com/apache/pinot/pull/12771 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

(pinot) branch dependabot/maven/hk2.version-3.1.0 updated (23bd28f3e4 -> ee40f9aec5)

2024-04-01 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hk2.version-3.1.0 in repository https://gitbox.apache.org/repos/asf/pinot.git discard 23bd28f3e4 Bump hk2.version from 2.6.1 to 3.1.0 add 0c61bb67af Bugfix. Avoid

Re: [PR] Cleanup Javax and Jakarta dependencies [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang merged PR #12760: URL: https://github.com/apache/pinot/pull/12760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(pinot) branch master updated (e71c3dc023 -> 4f0bc11b95)

2024-04-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from e71c3dc023 upgrade to slf4j 2.0.12 (#12761) add 4f0bc11b95 Cleanup Javax and Jakarta dependencies (#12760) No

[PR] Set column major segment builder to be on by default [pinot]

2024-04-01 Thread via GitHub
ege-st opened a new pull request, #12770: URL: https://github.com/apache/pinot/pull/12770 This enables the faster Column Major segment builder to be on by default. This feature has been in the wild for several months and since it's purpose is to speed up segment building it makes sense to

Re: [I] Streaming data ingestion support in Pinot for Redpanda. [pinot]

2024-04-01 Thread via GitHub
dttung2905 commented on issue #12155: URL: https://github.com/apache/pinot/issues/12155#issuecomment-2030516261 Hi @abhioncbr, are you still working on this ? If not I can give it a try too @Jackie-Jiang my simple understanding on this is we just have to add some documentations on

Re: [I] non-compliant download page: links should use closer.lua script [pinot]

2024-04-01 Thread via GitHub
pjfanning commented on issue #12442: URL: https://github.com/apache/pinot/issues/12442#issuecomment-2030514459 @Jackie-Jiang I am not a Pinot contributor and don't know where your web site code is kept. Should I bring this up with Pinot dev list. For me, ASF project PMC members should be

Re: [I] DISTINCTCOUNTMV returns incorrect value when field is included in select/group by [pinot]

2024-04-01 Thread via GitHub
aw381246 commented on issue #12429: URL: https://github.com/apache/pinot/issues/12429#issuecomment-2030514242 @Jackie-Jiang, if I understand `VALUE_IN`, I could specify a specific "host" in the query above, and I'd only get the first row, and the distinct count would = 1 right? But that

Re: [I] Allow peerDownload config for minion-tasks [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang commented on issue #12458: URL: https://github.com/apache/pinot/issues/12458#issuecomment-2030495606 I'm good with the proposal. Maybe renaming to `allowDownloadFromServer` -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] non-compliant download page: links should use closer.lua script [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang commented on issue #12442: URL: https://github.com/apache/pinot/issues/12442#issuecomment-2030490083 Thanks for raising this. Do you want to help fix this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Table segment size is not showing up [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang commented on issue #12441: URL: https://github.com/apache/pinot/issues/12441#issuecomment-2030481126 cc @mayankshriv -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Supported readDictIds method for multivalue columns in forward index. [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang commented on PR #12438: URL: https://github.com/apache/pinot/pull/12438#issuecomment-2030461152 Thanks for taking this up. This is not the correct approach. For the issue in #12374, we'll need to make `equals()` and other `BinaryOperatorTransformFunction`s support MV. We

Re: [PR] Supported readDictIds method for multivalue columns in forward index. [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang closed pull request #12438: Supported readDictIds method for multivalue columns in forward index. URL: https://github.com/apache/pinot/pull/12438 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Too many segments in a table caused entire cluster ingestion hanging [pinot]

2024-04-01 Thread via GitHub
estebanz01 commented on issue #2088: URL: https://github.com/apache/pinot/issues/2088#issuecomment-2030441661 I'm seeing this behaviour in a table with 36 GB of data in roughly 368000 segments. It's not fun when it happens :grinning: -- This is an automated message from the Apache Git

Re: [PR] upgrade lmax disruptor to 4.0.0 [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12769: URL: https://github.com/apache/pinot/pull/12769#issuecomment-2030433658 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12769?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [PR] Upgrade Calcite to 1.36.0 [pinot]

2024-04-01 Thread via GitHub
xiangfu0 commented on PR #12754: URL: https://github.com/apache/pinot/pull/12754#issuecomment-2030422110 wow! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add minionInstanceTag config to minion-tasks for resource isolation [pinot]

2024-04-01 Thread via GitHub
tibrewalpratik17 commented on code in PR #12459: URL: https://github.com/apache/pinot/pull/12459#discussion_r1546746797 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java: ## @@ -531,78 +532,98 @@ private synchronized Map

Re: [PR] Add minionInstanceTag config to minion-tasks for resource isolation [pinot]

2024-04-01 Thread via GitHub
tibrewalpratik17 commented on code in PR #12459: URL: https://github.com/apache/pinot/pull/12459#discussion_r1546744519 ## pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java: ## @@ -615,12 +615,15 @@ public Map

Re: [I] CalciteSQLParser error even when query executes succesfully [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang commented on issue #11823: URL: https://github.com/apache/pinot/issues/11823#issuecomment-2030378352 Unfortunately we haven't got a chance to work on this yet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] pluggable partial upsert merger [pinot]

2024-04-01 Thread via GitHub
rohityadav1993 commented on PR #11983: URL: https://github.com/apache/pinot/pull/11983#issuecomment-2030364352 @Jackie-Jiang @deemoliu , please review the updated changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] upgrade lmax disruptor to 4.0.0 [pinot]

2024-04-01 Thread via GitHub
sullis opened a new pull request, #12769: URL: https://github.com/apache/pinot/pull/12769 lmax disruptor 4.0.0 was releated in September 2023 https://mvnrepository.com/artifact/com.lmax/disruptor -- This is an automated message from the Apache Git Service. To respond to the

(pinot) branch master updated (62b97ef3b4 -> e71c3dc023)

2024-04-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 62b97ef3b4 Add support for phrase search with wildcard and prefix matching for Lucene indexed tables (#12680)

Re: [PR] upgrade to slf4j 2.0.12 [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang merged PR #12761: URL: https://github.com/apache/pinot/pull/12761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[I] QUERY_QUOTA_CAPACITY_UTILIZATION_RATE broker metric remains stale if there is no incoming query for a while [pinot]

2024-04-01 Thread via GitHub
jackjlli opened a new issue, #12768: URL: https://github.com/apache/pinot/issues/12768 The `QUERY_QUOTA_CAPACITY_UTILIZATION_RATE` broker metric remains stale if there is no incoming query against the table for a while. The reason is that this broker metric is a gauge metric. If there is

[PR] Fetch the query quota capacity utilization rate broker metric in a callback function [pinot]

2024-04-01 Thread via GitHub
jackjlli opened a new pull request, #12767: URL: https://github.com/apache/pinot/pull/12767 This PR fetches the query quota capacity utilization rate broker metric in a callback function. The reason is that this broker metric is a gauge metric. If there is no incoming query, the

Re: [I] Ingest from Pulsar: relying on/ compatible to EOL version of Pulsar [pinot]

2024-04-01 Thread via GitHub
Jackie-Jiang commented on issue #12713: URL: https://github.com/apache/pinot/issues/12713#issuecomment-2030288345 I see. Interestingly they still release on non-active support version. We will try to upgrade it to `3.2.1` -- This is an automated message from the Apache Git Service. To

(pinot) branch fix-broker-metrics created (now ca7004a14b)

2024-04-01 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch fix-broker-metrics in repository https://gitbox.apache.org/repos/asf/pinot.git at ca7004a14b Fetch query quota capacity utilization rate metric in a callback function This branch includes

(pinot) 01/01: Fetch query quota capacity utilization rate metric in a callback function

2024-04-01 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a commit to branch fix-broker-metrics in repository https://gitbox.apache.org/repos/asf/pinot.git commit ca7004a14bf049bcd8bb58a7f52bd80bbfab778c Author: jlli_LinkedIn AuthorDate: Mon Apr 1 11:14:30 2024 -0700

(pinot) branch master updated (c15c3912cb -> 62b97ef3b4)

2024-04-01 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from c15c3912cb Percentile operations supporting null (#12271) add 62b97ef3b4 Add support for phrase search with

Re: [PR] Add support for phrase search with wildcard and prefix matching for Lucene indexed tables [pinot]

2024-04-01 Thread via GitHub
chenboat merged PR #12680: URL: https://github.com/apache/pinot/pull/12680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Make task manager APIs database aware [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12766: URL: https://github.com/apache/pinot/pull/12766#issuecomment-2030040090 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12766?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

Re: [PR] Delete database API [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12765: URL: https://github.com/apache/pinot/pull/12765#issuecomment-2029994316 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12765?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

[PR] Make task manager APIs database aware [pinot]

2024-04-01 Thread via GitHub
shounakmk219 opened a new pull request, #12766: URL: https://github.com/apache/pinot/pull/12766 # Description Make `PinotTaskManager` APIs database aware and deprecate the downstream APIs which do not accept database as param in favour of database aware APIs # Deprecated APIs

[PR] Delete database API [pinot]

2024-04-01 Thread via GitHub
shounakmk219 opened a new pull request, #12765: URL: https://github.com/apache/pinot/pull/12765 # Description API to delete the given database which effectively deletes all the tables under the given database. The API has a sync behaviour which means partial database deletion is

Re: [PR] Multi stage stats [pinot]

2024-04-01 Thread via GitHub
gortiz commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1546150898 ## pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java: ## @@ -36,108 +36,86 @@ */ public class MetadataBlock extends BaseDataBlock { -

Re: [PR] Multi stage stats [pinot]

2024-04-01 Thread via GitHub
gortiz commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1546149252 ## pinot-common/src/main/java/org/apache/pinot/common/datablock/BaseDataBlock.java: ## @@ -71,9 +71,6 @@ * +---+ * |

Re: [PR] Bug fix. Allow passing null http headers object to translateTableName [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12764: URL: https://github.com/apache/pinot/pull/12764#issuecomment-2029436994 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12764?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

(pinot) branch master updated: Percentile operations supporting null (#12271)

2024-04-01 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c15c3912cb Percentile operations supporting null

Re: [PR] Percentile operations supporting null [pinot]

2024-04-01 Thread via GitHub
gortiz merged PR #12271: URL: https://github.com/apache/pinot/pull/12271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(pinot) branch master updated: Improved null check for varargs (#12673)

2024-04-01 Thread gortiz
This is an automated email from the ASF dual-hosted git repository. gortiz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 3185e303f4 Improved null check for varargs

Re: [PR] Improved null check for varargs [pinot]

2024-04-01 Thread via GitHub
gortiz merged PR #12673: URL: https://github.com/apache/pinot/pull/12673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] NPE when broker header is null [pinot]

2024-04-01 Thread via GitHub
shounakmk219 commented on issue #12745: URL: https://github.com/apache/pinot/issues/12745#issuecomment-2029400572 @jackjlli regarding how the database name is being passed, let's use the discussion thread

[PR] Bug fix. Allow passing null http headers object to translateTableName [pinot]

2024-04-01 Thread via GitHub
shounakmk219 opened a new pull request, #12764: URL: https://github.com/apache/pinot/pull/12764 Addresses issue #12745 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [null support] Allow fill the derived column with null when transform function failed to execute [pinot]

2024-04-01 Thread via GitHub
codecov-commenter commented on PR #12763: URL: https://github.com/apache/pinot/pull/12763#issuecomment-2029220791 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12763?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report