Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]

2024-02-02 Thread via GitHub
kishoreg commented on issue #12355: URL: https://github.com/apache/pinot/issues/12355#issuecomment-1925038472 +1 to this feature. We have this for StarTree index. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [Multi-stage] Optimize query dispatch [pinot]

2024-02-02 Thread via GitHub
codecov-commenter commented on PR #12358: URL: https://github.com/apache/pinot/pull/12358#issuecomment-1925005610 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12358?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] log information about SSLFactory renewal [pinot]

2024-02-02 Thread via GitHub
codecov-commenter commented on PR #12357: URL: https://github.com/apache/pinot/pull/12357#issuecomment-1925001300 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12357?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] [Multi-stage] Optimize query dispatch [pinot]

2024-02-02 Thread via GitHub
Jackie-Jiang opened a new pull request, #12358: URL: https://github.com/apache/pinot/pull/12358 - Do the server independent serialization only once per stage - Send one plan per server instead of one plan per stage per server - Parallel execute the server dependent serialization -- T

(pinot) branch release-1.1-rc created (now 4823802886)

2024-02-02 Thread vvivekiyer
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to branch release-1.1-rc in repository https://gitbox.apache.org/repos/asf/pinot.git at 4823802886 Wire soft upsert delete for compaction task (#12330) No new revisions were added by this update

[PR] log information about SSLFactory renewal [pinot]

2024-02-02 Thread via GitHub
zhtaoxiang opened a new pull request, #12357: URL: https://github.com/apache/pinot/pull/12357 Add logs to changes made in https://github.com/apache/pinot/pull/12277 and https://github.com/apache/pinot/pull/12325 to make it easy to track operations and debug problems -- This is an automat

Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]

2024-02-02 Thread via GitHub
egalpin commented on issue #12355: URL: https://github.com/apache/pinot/issues/12355#issuecomment-1924941336 Thoughts on the list scheme? How should the index names be referenced? Re-using the names from the tableSpec looks a little bit odd as it would lead to things like `SET indexSkipList

Re: [PR] Fix file handle leaks in Pinot Driver (apache#12263) [pinot]

2024-02-02 Thread via GitHub
codecov-commenter commented on PR #12356: URL: https://github.com/apache/pinot/pull/12356#issuecomment-1924922931 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12356?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [I] CVEs detected in dependencies of pinot-java-client and pinot-common [pinot]

2024-02-02 Thread via GitHub
robertzych commented on issue #12341: URL: https://github.com/apache/pinot/issues/12341#issuecomment-1924921750 The only dependency that had to be whitelisted was avatica-core 1.24.0 as it couldn't be excluded without introducing a regression. It's CVE ([CVE-2022-39135](https://nvd.nist.gov

[PR] Fix file handle leaks in Pinot Driver (apache#12263) [pinot]

2024-02-02 Thread via GitHub
BrendanStans21 opened a new pull request, #12356: URL: https://github.com/apache/pinot/pull/12356 PinotDriver.connect() leaks the PinotClientTransport and PinotControllerTransport if an exception is thrown. I added unit tests to test both a bad URL and a good URL that doesn't connect

Re: [I] [multistage] attach default limit 10 to queries that doesn't have a limit [pinot]

2024-02-02 Thread via GitHub
dttung2905 commented on issue #12304: URL: https://github.com/apache/pinot/issues/12304#issuecomment-1924884957 Thanks @walterddr for the detailed pointer. I take a look at the code and I think `validate` stage might be a good place to implement that check, something like ```java

[I] Add QueryOption support for column-level index skipping at query time [pinot]

2024-02-02 Thread via GitHub
egalpin opened a new issue, #12355: URL: https://github.com/apache/pinot/issues/12355 In some cases it may be beneficial to skip the usage of an index despite its existence, such as utilizing many inverted indices over many columns, and combining the resulting bitmaps. It may be faster to

Re: [PR] Make segment download from Peer servers more robust by retrying both peer discovery and download. [pinot]

2024-02-02 Thread via GitHub
chenboat commented on code in PR #12317: URL: https://github.com/apache/pinot/pull/12317#discussion_r1476849074 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java: ## @@ -658,11 +656,10 @@ private void downloadSegmentFromPeer(St

Re: [PR] Make segment download from Peer servers more robust by retrying both peer discovery and download. [pinot]

2024-02-02 Thread via GitHub
chenboat commented on code in PR #12317: URL: https://github.com/apache/pinot/pull/12317#discussion_r1476843904 ## pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java: ## @@ -109,6 +112,38 @@ public File fetchUntarSegmentToLocalStreamed(URI u

Re: [PR] Add round-robin logic during downloadSegmentFromPeer [pinot]

2024-02-02 Thread via GitHub
tibrewalpratik17 commented on PR #12353: URL: https://github.com/apache/pinot/pull/12353#issuecomment-1924772595 cc @Jackie-Jiang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

(pinot) branch master updated: Cover the race condition for upsert compaction (#12346)

2024-02-02 Thread snlee
This is an automated email from the ASF dual-hosted git repository. snlee pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ab525ee929 Cover the race condition for upsert com

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-02 Thread via GitHub
snleee merged PR #12346: URL: https://github.com/apache/pinot/pull/12346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-02 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1476580694 ## pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/ValidDocIdsType.java: ## @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache Software Foundation (AS

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-02 Thread via GitHub
snleee commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1476580343 ## pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/ValidDocIdsType.java: ## @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache Software Foundation (AS

Re: [PR] Add metrics for no-HA segments monitoring [pinot]

2024-02-02 Thread via GitHub
lnbest0707-uber commented on PR #12336: URL: https://github.com/apache/pinot/pull/12336#issuecomment-1924507664 > > > the approach looks good. i was wondering if we should have some definitions. e.g. when nReplica equals the following, how should we call each status. HighAvailability is not

Re: [PR] [null-aggr] Add null handling support in `mode` aggregation [pinot]

2024-02-02 Thread via GitHub
gortiz commented on code in PR #12227: URL: https://github.com/apache/pinot/pull/12227#discussion_r1476505233 ## pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java: ## @@ -0,0 +1,368 @@ +/** + * Licensed to the A

[PR] [draft] Mode null benchmark [pinot]

2024-02-02 Thread via GitHub
gortiz opened a new pull request, #12354: URL: https://github.com/apache/pinot/pull/12354 There are some questions about the performance on the implementation proposed in #12227. This PR modifies the code to add several Mode implementations that can be selected by using `mode(col, MI

(pinot) branch master updated: Release all segments of a table in releaseAndRemoveAllSegments method (#12297)

2024-02-02 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 87c089f8eb Release all segments of a table in rele

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments deleted (was a796baa1b5)

2024-02-02 Thread jlli
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments in repository https://gitbox.apache.org/repos/asf/pinot.git was a796baa1b5 Release all segments of a table in releaseAndRemoveAllSegments meth

Re: [PR] Release all segments of a table in releaseAndRemoveAllSegments method [pinot]

2024-02-02 Thread via GitHub
jackjlli merged PR #12297: URL: https://github.com/apache/pinot/pull/12297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apa

Re: [PR] Cover the race condition for upsert compaction [pinot]

2024-02-02 Thread via GitHub
Jackie-Jiang commented on code in PR #12346: URL: https://github.com/apache/pinot/pull/12346#discussion_r1476487009 ## pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/ValidDocIdsType.java: ## @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache Software Foundati

Re: [PR] Make segment download from Peer servers more robust by retrying both peer discovery and download. [pinot]

2024-02-02 Thread via GitHub
tibrewalpratik17 commented on code in PR #12317: URL: https://github.com/apache/pinot/pull/12317#discussion_r1475797609 ## pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java: ## @@ -109,6 +112,38 @@ public File fetchUntarSegmentToLocalStream

Re: [PR] Fluent test framework [pinot]

2024-02-02 Thread via GitHub
gortiz commented on code in PR #12215: URL: https://github.com/apache/pinot/pull/12215#discussion_r1475802396 ## pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/CountAggregationFunctionTest.java: ## @@ -0,0 +1,184 @@ +/** + * Licensed to the Apache Soft

Re: [PR] Make segment download from Peer servers more robust by retrying both peer discovery and download. [pinot]

2024-02-02 Thread via GitHub
tibrewalpratik17 commented on code in PR #12317: URL: https://github.com/apache/pinot/pull/12317#discussion_r1475797609 ## pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java: ## @@ -109,6 +112,38 @@ public File fetchUntarSegmentToLocalStream

Re: [PR] Fluent test framework [pinot]

2024-02-02 Thread via GitHub
gortiz commented on code in PR #12215: URL: https://github.com/apache/pinot/pull/12215#discussion_r1475800914 ## pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/AbstractAggregationFunctionTest.java: ## @@ -0,0 +1,121 @@ +/** + * Licensed to the Apache S

Re: [PR] Add round-robin logic during downloadSegmentFromPeer [pinot]

2024-02-02 Thread via GitHub
codecov-commenter commented on PR #12353: URL: https://github.com/apache/pinot/pull/12353#issuecomment-1923391691 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12353?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] Add round-robin logic during downloadSegmentFromPeer [pinot]

2024-02-02 Thread via GitHub
tibrewalpratik17 opened a new pull request, #12353: URL: https://github.com/apache/pinot/pull/12353 label: `bugfix` During downloadSegmentFromPeer : https://github.com/apache/pinot/blob/041e04078f5a94fca92c805a8db8fdf1f904a985/pinot-core/src/main/java/org/apache/pinot/core/data/manage

Re: [PR] Cleaning replication delay metrics on cleanupMetrics [pinot]

2024-02-02 Thread via GitHub
gortiz closed pull request #12343: Cleaning replication delay metrics on cleanupMetrics URL: https://github.com/apache/pinot/pull/12343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Cleaning replication delay metrics on cleanupMetrics [pinot]

2024-02-02 Thread via GitHub
gortiz commented on PR #12343: URL: https://github.com/apache/pinot/pull/12343#issuecomment-1923282629 We weren't sure about the implications of this PR. Instead, we decided to merge #12351, which solves the issue in a better way. For the future, we will try to fix Helix so it can transitio

Re: [PR] Work around the problem of Helix sending 2 transitions for CONSUMING -> DROPPED [pinot]

2024-02-02 Thread via GitHub
gortiz commented on code in PR #12351: URL: https://github.com/apache/pinot/pull/12351#discussion_r1475698239 ## pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentOnlineOfflineStateModelFactory.java: ## @@ -215,6 +240,15 @@ public void onBecomeDroppedFromOf