Re: [PR] Support initializing broker tags from config [pinot]

2023-12-18 Thread via GitHub
codecov-commenter commented on PR #12175: URL: https://github.com/apache/pinot/pull/12175#issuecomment-1862111517 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12175?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] Support initializing broker tags from config [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang opened a new pull request, #12175: URL: https://github.com/apache/pinot/pull/12175 Address #11755 Support initializing broker tags from config and automatically updating broker resource when broker joins the cluster for the first time. ## Configuration `pinot.b

Re: [PR] Bug fix: reset primary key count to 0 when table is deleted [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang commented on code in PR #12169: URL: https://github.com/apache/pinot/pull/12169#discussion_r1430863933 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -772,6 +772,8 @@ public synchronized void

Re: [PR] Bug fix: reset primary key count to 0 when table is deleted [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang commented on code in PR #12169: URL: https://github.com/apache/pinot/pull/12169#discussion_r1430863535 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -772,6 +772,8 @@ public synchronized void

Re: [I] Streaming data ingestion support in Pinot for Redpanda. [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang commented on issue #12155: URL: https://github.com/apache/pinot/issues/12155#issuecomment-1862024619 Seems like RedPanda is Kafka compatible, so I'd suggest keeping it simple and just follow the same configs. +1 on adding an integration test on RedPanda if it doesn't pull in to

Re: [PR] Support array literal functions [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang commented on code in PR #12118: URL: https://github.com/apache/pinot/pull/12118#discussion_r1430849697 ## pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java: ## @@ -156,7 +156,12 @@ public enum ObjectType { FloatArrayList(44), Stri

Re: [I] PinotS3 : Connection Pool Shutdown after one hour on EKS on release-1.0.0 [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang commented on issue #11761: URL: https://github.com/apache/pinot/issues/11761#issuecomment-1862018568 @abhijeetkushe Can you try out #12063 and see if it works? We are waiting for a confirmation that it works before merging it -- This is an automated message from the Apache Gi

(pinot) branch dependabot/maven/org.apache.yetus-audience-annotations-0.15.0 deleted (was 9a579b5a25)

2023-12-18 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.apache.yetus-audience-annotations-0.15.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 9a579b5a25 Bump org.apache.yetus:audience-annotations from 0.13

(pinot) branch master updated: Bump org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0 (#12170)

2023-12-18 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new c085f0497d Bump org.apache.yetus:audience-annotat

Re: [PR] Bump org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0 [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang merged PR #12170: URL: https://github.com/apache/pinot/pull/12170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [PR] Bump com.gradle:common-custom-user-data-maven-extension from 1.12.4 to 1.12.5 [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang merged PR #12171: URL: https://github.com/apache/pinot/pull/12171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch dependabot/maven/com.gradle-common-custom-user-data-maven-extension-1.12.5 deleted (was b722175cb7)

2023-12-18 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/com.gradle-common-custom-user-data-maven-extension-1.12.5 in repository https://gitbox.apache.org/repos/asf/pinot.git was b722175cb7 Bump com.gradle:common-custom-user-data

(pinot) branch master updated (ccc5f34507 -> 3d3012fd20)

2023-12-18 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from ccc5f34507 Bump org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14 (#12172) add 3d3012fd20 Bump com.gradl

(pinot) branch dependabot/maven/org.apache.httpcomponents-httpclient-4.5.14 deleted (was d146614662)

2023-12-18 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.apache.httpcomponents-httpclient-4.5.14 in repository https://gitbox.apache.org/repos/asf/pinot.git was d146614662 Bump org.apache.httpcomponents:httpclient from 4.5.13

Re: [PR] Allow tenant level config override [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang commented on PR #12039: URL: https://github.com/apache/pinot/pull/12039#issuecomment-1862013778 Tenant to tag mapping is not one to one for servers (e.g `myTenant_OFFLINE` and `myTenant_REALTIME`), so using tenant name will lose the flexibility of putting different config for `

(pinot) branch master updated (333e17c765 -> ccc5f34507)

2023-12-18 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 333e17c765 Update LICENSE-binary for commons-configuration2 upgrade. (#12165) add ccc5f34507 Bump org.apache.http

Re: [PR] Bump org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14 [pinot]

2023-12-18 Thread via GitHub
Jackie-Jiang merged PR #12172: URL: https://github.com/apache/pinot/pull/12172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] Streaming data ingestion support in Pinot for Redpanda. [pinot]

2023-12-18 Thread via GitHub
abhioncbr commented on issue #12155: URL: https://github.com/apache/pinot/issues/12155#issuecomment-1861998701 @Jackie-Jiang / @xiangfu0, is it okay to follow the same config as Kafka or change something related to Redpanda, like `streamType` as `RedPanda`. Also do we want to add the tes

Re: [I] Does Pinot support string split function? [pinot]

2023-12-18 Thread via GitHub
xiangfu0 commented on issue #12089: URL: https://github.com/apache/pinot/issues/12089#issuecomment-1861888432 https://github.com/apache/pinot/pull/12118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Please don't hardcode zookeeper version [pinot]

2023-12-18 Thread via GitHub
xiangfu0 commented on issue #12174: URL: https://github.com/apache/pinot/issues/12174#issuecomment-1861885225 Where did you find it? We can definitely make it configurable. Feel free to raise a PR for this. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Support array literal functions [pinot]

2023-12-18 Thread via GitHub
xiangfu0 commented on code in PR #12118: URL: https://github.com/apache/pinot/pull/12118#discussion_r1430765003 ## pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java: ## @@ -156,7 +156,12 @@ public enum ObjectType { FloatArrayList(44), StringAr

[I] Please don't hardcode zookeeper version [pinot]

2023-12-18 Thread via GitHub
pospospos2007 opened a new issue, #12174: URL: https://github.com/apache/pinot/issues/12174 Hello, can you please not hardcode the zookeeper version of docker image: **image: docker.io/bitnami/zookeeper:3.8.0-debian-11-r5** When I use helm install pinot and

Re: [PR] Detect expired messages in Kafka. Log and set a guage. [pinot]

2023-12-18 Thread via GitHub
ege-st commented on PR #12157: URL: https://github.com/apache/pinot/pull/12157#issuecomment-1861814766 > > We should be able to emit metric within `PartitionGroupConsumer.fetchMessages()` when the start offset is not available (e.g. the asked offset range is not fully returned). > >

Re: [PR] Detect expired messages in Kafka. Log and set a guage. [pinot]

2023-12-18 Thread via GitHub
ege-st commented on PR #12157: URL: https://github.com/apache/pinot/pull/12157#issuecomment-1861811576 It is possible for messages to get expired in the middle of the lifetime of a consuming segment. One way this could happen is if the ingestion on the table is paused for a period longer th

Re: [PR] Detect expired messages in Kafka. Log and set a guage. [pinot]

2023-12-18 Thread via GitHub
swaminathanmanish commented on PR #12157: URL: https://github.com/apache/pinot/pull/12157#issuecomment-1861764515 > > We should be able to emit metric within `PartitionGroupConsumer.fetchMessages()` when the start offset is not available (e.g. the asked offset range is not fully returned).

Re: [PR] Bug fix: reset primary key count to 0 when table is deleted [pinot]

2023-12-18 Thread via GitHub
walterddr commented on code in PR #12169: URL: https://github.com/apache/pinot/pull/12169#discussion_r1430532128 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java: ## @@ -772,6 +772,8 @@ public synchronized void cl

(pinot) branch master updated: Update LICENSE-binary for commons-configuration2 upgrade. (#12165)

2023-12-18 Thread rongr
This is an automated email from the ASF dual-hosted git repository. rongr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 333e17c765 Update LICENSE-binary for commons-confi

Re: [PR] Update LICENSE-binary for commons-configuration2 upgrade. [pinot]

2023-12-18 Thread via GitHub
walterddr merged PR #12165: URL: https://github.com/apache/pinot/pull/12165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.ap

Re: [I] JDK 21 incompatible with Pinot [pinot]

2023-12-18 Thread via GitHub
walterddr commented on issue #12167: URL: https://github.com/apache/pinot/issues/12167#issuecomment-1861301224 CC @gortiz :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] test [pinot]

2023-12-18 Thread via GitHub
weiqing-li closed pull request #12173: test URL: https://github.com/apache/pinot/pull/12173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsu

[PR] test [pinot]

2023-12-18 Thread via GitHub
weiqing-li opened a new pull request, #12173: URL: https://github.com/apache/pinot/pull/12173 Instructions: 1. The PR has to be tagged with at least one of the following labels (*): 1. `feature` 2. `bugfix` 3. `performance` 4. `ui` 5. `backward-incompat`

Re: [PR] Maintain pool selection for the minimizeDataMovement instance partition assignment strategy [pinot]

2023-12-18 Thread via GitHub
Ferix9288 commented on PR #11953: URL: https://github.com/apache/pinot/pull/11953#issuecomment-1861257035 Hello~ I've been lurking for the past months or so haha. Any chance this can be merged in soon as our company really dislikes our current workaround for this problem? Thanks so much aga

Re: [PR] [POC] function registry unification [pinot]

2023-12-18 Thread via GitHub
walterddr commented on PR #12163: URL: https://github.com/apache/pinot/pull/12163#issuecomment-1861248100 - Follow-up POC for unified polymorphism: https://github.com/walterddr/pinot/pull/91 - Follow-up POC for parameter-type-based lookup: https://github.com/walterddr/pinot/pull/91 --

Re: [PR] [POC] function registry unification [pinot]

2023-12-18 Thread via GitHub
codecov-commenter commented on PR #12163: URL: https://github.com/apache/pinot/pull/12163#issuecomment-1861089527 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12163?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Add column name to JSONParsing exception message during index build [pinot]

2023-12-18 Thread via GitHub
vrajat commented on PR #12151: URL: https://github.com/apache/pinot/pull/12151#issuecomment-1861065440 I'll do the manual test. re: integration test, the current test builds a segment for real. I copied the idea from another test that tests segment build. So hopefully this test is en

Re: [PR] Add column name to JSONParsing exception message during index build [pinot]

2023-12-18 Thread via GitHub
ege-st commented on PR #12151: URL: https://github.com/apache/pinot/pull/12151#issuecomment-1860961248 > > By the way, besides unit tests, were you able to do a manual test and see what happens when Pinot is running and this error happens? > > No I did not. Do I have to implement an i

Re: [PR] Add column name to JSONParsing exception message during index build [pinot]

2023-12-18 Thread via GitHub
vrajat commented on PR #12151: URL: https://github.com/apache/pinot/pull/12151#issuecomment-1860951021 > By the way, besides unit tests, were you able to do a manual test and see what happens when Pinot is running and this error happens? No I did not. Do I have to implement an integra

Re: [PR] Make FunctionRegistry use Calcite NamedMultiMap [pinot]

2023-12-18 Thread via GitHub
walterddr closed pull request #12144: Make FunctionRegistry use Calcite NamedMultiMap URL: https://github.com/apache/pinot/pull/12144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] PinotS3 : Connection Pool Shutdown after one hour on EKS on release-1.0.0 [pinot]

2023-12-18 Thread via GitHub
abhijeetkushe commented on issue #11761: URL: https://github.com/apache/pinot/issues/11761#issuecomment-1860786974 Any updates on when this might be fixed.We are seeing a similar error while reading from AWS kinesis in a Realtime Table -- This is an automated message from the Apache Git S

Re: [PR] Add column name to JSONParsing exception message during index build [pinot]

2023-12-18 Thread via GitHub
ege-st commented on PR #12151: URL: https://github.com/apache/pinot/pull/12151#issuecomment-1860774295 By the way, besides unit tests, were you able to do a manual test and see what happens when Pinot is running and this error happens? -- This is an automated message from the Apache Git S

Re: [PR] Bump org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0 [pinot]

2023-12-18 Thread via GitHub
codecov-commenter commented on PR #12170: URL: https://github.com/apache/pinot/pull/12170#issuecomment-1860333252 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12170?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Bump com.gradle:common-custom-user-data-maven-extension from 1.12.4 to 1.12.5 [pinot]

2023-12-18 Thread via GitHub
codecov-commenter commented on PR #12171: URL: https://github.com/apache/pinot/pull/12171#issuecomment-1860327526 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12171?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Bump org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14 [pinot]

2023-12-18 Thread via GitHub
codecov-commenter commented on PR #12172: URL: https://github.com/apache/pinot/pull/12172#issuecomment-1860309424 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12172?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] Bump org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14 [pinot]

2023-12-18 Thread via GitHub
dependabot[bot] opened a new pull request, #12172: URL: https://github.com/apache/pinot/pull/12172 Bumps org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-nam

[PR] Bump com.gradle:common-custom-user-data-maven-extension from 1.12.4 to 1.12.5 [pinot]

2023-12-18 Thread via GitHub
dependabot[bot] opened a new pull request, #12171: URL: https://github.com/apache/pinot/pull/12171 Bumps [com.gradle:common-custom-user-data-maven-extension](https://github.com/gradle/common-custom-user-data-maven-extension) from 1.12.4 to 1.12.5. Release notes Sourced from https:

(pinot) branch dependabot/maven/org.apache.yetus-audience-annotations-0.15.0 created (now 9a579b5a25)

2023-12-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.yetus-audience-annotations-0.15.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 9a579b5a25 Bump org.apache.yetus:audience-annotations from

(pinot) branch dependabot/maven/org.apache.httpcomponents-httpclient-4.5.14 created (now d146614662)

2023-12-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.httpcomponents-httpclient-4.5.14 in repository https://gitbox.apache.org/repos/asf/pinot.git at d146614662 Bump org.apache.httpcomponents:httpclient from 4.

(pinot) branch dependabot/maven/com.gradle-common-custom-user-data-maven-extension-1.12.5 created (now b722175cb7)

2023-12-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/com.gradle-common-custom-user-data-maven-extension-1.12.5 in repository https://gitbox.apache.org/repos/asf/pinot.git at b722175cb7 Bump com.gradle:common-custom-user-

[PR] Bump org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0 [pinot]

2023-12-18 Thread via GitHub
dependabot[bot] opened a new pull request, #12170: URL: https://github.com/apache/pinot/pull/12170 Bumps org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-na

Re: [PR] [feature] use async methods to read from Pulsar topic [pinot]

2023-12-18 Thread via GitHub
KKcorps commented on PR #12061: URL: https://github.com/apache/pinot/pull/12061#issuecomment-1859958543 > This PR started from addressing a concurrency issue when interacting with the List used to hold messages pulled from Pulsar after the read timeout expired. It then evolved into a refact

Re: [PR] [feature] use async methods to read from Pulsar topic [pinot]

2023-12-18 Thread via GitHub
KKcorps commented on code in PR #12061: URL: https://github.com/apache/pinot/pull/12061#discussion_r1429804967 ## pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarPartitionLevelConsumer.java: ## @@ -75,81 +71,68 @@ publi

(pinot) branch master updated: [feature] add support for StreamNative OAuth2 authentication for pulsar. (#12068)

2023-12-18 Thread kharekartik
This is an automated email from the ASF dual-hosted git repository. kharekartik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new fd45c1bbd2 [feature] add support for StreamN

Re: [PR] [feature] add support for StreamNative OAuth2 authentication for pulsar. [pinot]

2023-12-18 Thread via GitHub
KKcorps merged PR #12068: URL: https://github.com/apache/pinot/pull/12068 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apac

Re: [I] [feature] Add support for oauth2 to Pulsar ingest plugin [pinot]

2023-12-18 Thread via GitHub
KKcorps closed issue #12067: [feature] Add support for oauth2 to Pulsar ingest plugin URL: https://github.com/apache/pinot/issues/12067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c