[GitHub] [pinot] saurabhd336 commented on issue #8726: Null data in derived columns after Table config update [Verification Pending]

saurabhd336 commented on issue #8726: URL: https://github.com/apache/pinot/issues/8726#issuecomment-1143195812 Confirming that issue is seen in master branch too. Looks like, _transformPipeline gets created once when LLRealtimeSegmentDataManager is created for the segment, and any changes t

[GitHub] [pinot] saurabhd336 commented on issue #8726: Null data in derived columns after Table config update

saurabhd336 commented on issue #8726: URL: https://github.com/apache/pinot/issues/8726#issuecomment-1143196790 > Yeah, you are right. This was provided by the person who got the issue. Just for future reference, JSONPATHSTRING(header_js, '$.nnTransId') is indeed the correct expression

[GitHub] [pinot] KKcorps commented on a diff in pull request #8787: Cleanup Kinesis Logs

KKcorps commented on code in PR #8787: URL: https://github.com/apache/pinot/pull/8787#discussion_r886354409 ## pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java: ## @@ -159,23 +159,23 @@ private KinesisRe

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r886538200 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java: ## @@ -505,34 +505,40 @@ static void validateTaskConfigs(TableConfig t

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r886539092 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java: ## @@ -505,34 +505,40 @@ static void validateTaskConfigs(TableConfig t

[GitHub] [pinot] lksvenoy-r7 commented on a diff in pull request #8802: Fix metadata push mode in IngestionUtils

lksvenoy-r7 commented on code in PR #8802: URL: https://github.com/apache/pinot/pull/8802#discussion_r886556315 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/IngestionUtils.java: ## @@ -240,7 +240,7 @@ public static void uploadSegment(String tableName

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r886561566 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/tablestate/TableState.java: ## @@ -0,0 +1,99 @@ +/** + * Licensed to the Apache Software Foun

[GitHub] [pinot] lksvenoy-r7 opened a new issue, #8811: Delete all segments API does not respect retention period

lksvenoy-r7 opened a new issue, #8811: URL: https://github.com/apache/pinot/issues/8811 I just tried to use the API for deleting segments `/segments/{tableName} ` through the swagger documentation, and found that it did not respect the retention policy I specified. https://user-image

[GitHub] [pinot] xiangfu0 merged pull request #8802: Fix metadata push mode in IngestionUtils

xiangfu0 merged PR #8802: URL: https://github.com/apache/pinot/pull/8802 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[pinot] branch master updated: Fix metadata push mode in IngestionUtils (#8802)

This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 9abf15fd6d Fix metadata push mode in IngestionUt

[GitHub] [pinot] stym06 commented on a diff in pull request #8779: Change DateTimeFormatSpec delimiter

stym06 commented on code in PR #8779: URL: https://github.com/apache/pinot/pull/8779#discussion_r886691428 ## pinot-spi/src/main/java/org/apache/pinot/spi/data/DateTimeFormatSpec.java: ## @@ -84,6 +127,23 @@ public DateTimeFormatSpec(int columnSize, String columnUnit, String co

[GitHub] [pinot] stym06 commented on a diff in pull request #8779: Change DateTimeFormatSpec delimiter

stym06 commented on code in PR #8779: URL: https://github.com/apache/pinot/pull/8779#discussion_r886691428 ## pinot-spi/src/main/java/org/apache/pinot/spi/data/DateTimeFormatSpec.java: ## @@ -84,6 +127,23 @@ public DateTimeFormatSpec(int columnSize, String columnUnit, String co

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r886704042 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableConfig.java: ## @@ -288,8 +303,8 @@ public void setTierConfigsList(List tierConfigsList) { }

[GitHub] [pinot] KKcorps closed issue #8717: Data not getting consumed from all Kafka partitions

KKcorps closed issue #8717: Data not getting consumed from all Kafka partitions URL: https://github.com/apache/pinot/issues/8717 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [pinot] KKcorps commented on issue #8717: Data not getting consumed from all Kafka partitions

KKcorps commented on issue #8717: URL: https://github.com/apache/pinot/issues/8717#issuecomment-1143653665 The issue for this error is still unclear. Most likely cause is the offset.retention period getting expired (which is different from topic retention period). Out of Range errors were n

[GitHub] [pinot] diogobaeder commented on issue #8805: includeFileNamePattern not working as expected

diogobaeder commented on issue #8805: URL: https://github.com/apache/pinot/issues/8805#issuecomment-1143670593 Ahhh, got it! I didn't know it was for the absolute path - I think the combination of `inputDirURI` and `includeFileNamePattern` makes this a bit confusing, as there's no mention t

[GitHub] [pinot] diogobaeder commented on issue #8805: includeFileNamePattern not working as expected

diogobaeder commented on issue #8805: URL: https://github.com/apache/pinot/issues/8805#issuecomment-1143674168 @kkrugler because it might have to traverse through all matching file paths according to the glob, and then filter in the ones that match `inputDirURI`, no? I mean, this is just sp

[GitHub] [pinot] mayankshriv commented on issue #8811: Delete all segments API does not respect retention period

mayankshriv commented on issue #8811: URL: https://github.com/apache/pinot/issues/8811#issuecomment-1143697233 Hi @lksvenoy-r7, this is expected behavior from the delete api. It is indeed expected to delete all (or specified) segments for a table. Retention is auto-managed (via periodic job

[GitHub] [pinot] Jackie-Jiang commented on issue #8809: S3 rate limiting on realtime server restart

Jackie-Jiang commented on issue #8809: URL: https://github.com/apache/pinot/issues/8809#issuecomment-1143890185 Thanks for reporting the issue. This warning log is from the controller side when server is trying to commit a segment file. It should be called once per segment commit, so not su

[GitHub] [pinot] Jackie-Jiang commented on issue #8800: Add a "Distinct" implementation that leverages index for low cardinality columns

Jackie-Jiang commented on issue #8800: URL: https://github.com/apache/pinot/issues/8800#issuecomment-1143908580 I think I get the general idea of using inverted index to solve distinct and group-by queries: - SELECT DISTINCT colA FROM myTable WHERE ... - SELECT COUNT(*) FROM myTable WH

[GitHub] [pinot] Jackie-Jiang merged pull request #8713: Allow table name with dots by a PinotConfiguration switch

Jackie-Jiang merged PR #8713: URL: https://github.com/apache/pinot/pull/8713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.a

[pinot] branch master updated (9abf15fd6d -> ae392432c4)

This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 9abf15fd6d Fix metadata push mode in IngestionUtils (#8802) add ae392432c4 Allow table name with dots by a PinotCo

[GitHub] [pinot] siddharthteotia merged pull request #8796: allow up to 4GB per bitmap index

siddharthteotia merged PR #8796: URL: https://github.com/apache/pinot/pull/8796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pino

[pinot] branch master updated: allow up to 4GB per bitmap index (#8796)

This is an automated email from the ASF dual-hosted git repository. siddteotia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 4269bfdeaa allow up to 4GB per bitmap index (

[pinot] branch master updated: Add segment replace api failure metrics. (#8784)

This is an automated email from the ASF dual-hosted git repository. jlli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 07b3ee639c Add segment replace api failure metrics.

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887162173 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/PartitionSelector.java: ## @@ -0,0 +1,46 @@ +/** + * Licensed to the A

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887163556 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/PartitionSelector.java: ## @@ -0,0 +1,46 @@ +/** + * Licensed to the A

[GitHub] [pinot] jackjlli merged pull request #8784: Add segment replace api failure metrics.

jackjlli merged PR #8784: URL: https://github.com/apache/pinot/pull/8784 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887171360 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/assignment/InstanceAssignmentConfig.java: ## @@ -28,29 +28,37 @@ public class InstanceAssignmentCon

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887172863 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/assignment/InstanceAssignmentConfig.java: ## @@ -28,29 +28,37 @@ public class InstanceAssignmentCon

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8787: Cleanup Kinesis Logs

Jackie-Jiang commented on code in PR #8787: URL: https://github.com/apache/pinot/pull/8787#discussion_r887192421 ## pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java: ## @@ -159,23 +159,23 @@ private Kine

[GitHub] [pinot] kkrugler opened a new pull request, #8812: Fix segment generation error handling

kkrugler opened a new pull request, #8812: URL: https://github.com/apache/pinot/pull/8812 Fix for issue #8641. Should be tagged with `bugfix`. Might be considered a backwards-incompatible change, since the stand-alone tool will now fail if there's a segment generation error, wh

[GitHub] [pinot] kkrugler commented on issue #8641: "Failed to generate Pinot segment for file" didn't throw an exception

kkrugler commented on issue #8641: URL: https://github.com/apache/pinot/issues/8641#issuecomment-1144012826 @npawar @Jackie-Jiang - just created https://github.com/apache/pinot/pull/8812 as first step ("Track all the failed segments into a list, but keep the task running. After all the segm

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887197719 ## pinot-common/src/main/java/org/apache/pinot/common/assignment/InstanceAssignmentConfigUtils.java: ## @@ -120,6 +120,6 @@ public static InstanceAssignmentConfig

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887197719 ## pinot-common/src/main/java/org/apache/pinot/common/assignment/InstanceAssignmentConfigUtils.java: ## @@ -120,6 +120,6 @@ public static InstanceAssignmentConfig

[GitHub] [pinot] codecov-commenter commented on pull request #8812: Fix segment generation error handling

codecov-commenter commented on PR #8812: URL: https://github.com/apache/pinot/pull/8812#issuecomment-1144040652 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8812?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8779: Change DateTimeFormatSpec delimiter

Jackie-Jiang commented on code in PR #8779: URL: https://github.com/apache/pinot/pull/8779#discussion_r887218963 ## pinot-spi/src/main/java/org/apache/pinot/spi/data/DateTimeFormatSpec.java: ## @@ -84,6 +127,23 @@ public DateTimeFormatSpec(int columnSize, String columnUnit, Str

[GitHub] [pinot] codecov-commenter commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation

codecov-commenter commented on PR #8781: URL: https://github.com/apache/pinot/pull/8781#issuecomment-1144043889 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] Jackie-Jiang commented on issue #8805: includeFileNamePattern not working as expected

Jackie-Jiang commented on issue #8805: URL: https://github.com/apache/pinot/issues/8805#issuecomment-1144051438 @diogobaeder Pinot will create a `PathMatcher` from the given file name pattern, then iterate over all the files under the provided data dir and match it with the `PathMatcher`. S

[GitHub] [pinot] stym06 commented on a diff in pull request #8779: Change DateTimeFormatSpec delimiter

stym06 commented on code in PR #8779: URL: https://github.com/apache/pinot/pull/8779#discussion_r887228904 ## pinot-spi/src/main/java/org/apache/pinot/spi/data/DateTimeFormatSpec.java: ## @@ -84,6 +127,23 @@ public DateTimeFormatSpec(int columnSize, String columnUnit, String co

[GitHub] [pinot] Jackie-Jiang commented on issue #8811: Delete all segments API does not respect retention period

Jackie-Jiang commented on issue #8811: URL: https://github.com/apache/pinot/issues/8811#issuecomment-1144054802 This retention is not the retention for the table, but the retention for the deleted segments. When you specify `3d` here, Pinot will keep the deleted segments under the `deletedS

[GitHub] [pinot] diogobaeder commented on issue #8805: includeFileNamePattern not working as expected

diogobaeder commented on issue #8805: URL: https://github.com/apache/pinot/issues/8805#issuecomment-1144061159 Ah, nice! Sure, I'll try to do that change ASAP, thanks guys! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8779: Change DateTimeFormatSpec delimiter

Jackie-Jiang commented on code in PR #8779: URL: https://github.com/apache/pinot/pull/8779#discussion_r887239953 ## pinot-spi/src/main/java/org/apache/pinot/spi/data/DateTimeFormatPatternSpec.java: ## @@ -65,6 +65,20 @@ public DateTimeFormatPatternSpec(String timeFormat, @Nulla

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887162173 ## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/PartitionSelector.java: ## @@ -0,0 +1,46 @@ +/** + * Licensed to the A

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887251035 ## pinot-spi/src/main/java/org/apache/pinot/spi/config/table/assignment/Constants.java: ## @@ -0,0 +1,28 @@ +/** + * Licensed to the Apache Software Foundation (AS

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8434: [WIP][Not Ready for Review]Add replica group based instance assigment algorithm with FD awareness

siddharthteotia commented on code in PR #8434: URL: https://github.com/apache/pinot/pull/8434#discussion_r887253393 ## pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java: ## @@ -349,7 +354,7 @@ public void testPo

[pinot] branch helix-1.0 updated (4c70ad51ac -> 55af559d7e)

This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch helix-1.0 in repository https://gitbox.apache.org/repos/asf/pinot.git omit 4c70ad51ac Upgrade Helix to 1.0.3 add a359c49787 Optimize ColumnValueSegmentPruner by caching value hashes (#876

[GitHub] [pinot] kkrugler commented on issue #8641: "Failed to generate Pinot segment for file" didn't throw an exception

kkrugler commented on issue #8641: URL: https://github.com/apache/pinot/issues/8641#issuecomment-1144092481 @Jackie-Jiang - I was thinking about the `terminateOnFailure` flag a bit. The other segment generating code (Spark & Hadoop) already terminate on failure implicitly, as the task (and

[GitHub] [pinot] Jackie-Jiang commented on issue #8641: "Failed to generate Pinot segment for file" didn't throw an exception

Jackie-Jiang commented on issue #8641: URL: https://github.com/apache/pinot/issues/8641#issuecomment-1144129654 @kkrugler Agree on having the consistent behavior. I think terminating on failure by default might be the desired behavior, and we can introduce a flag to `continueOnFailure` (may

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8812: Fix segment generation error handling

Jackie-Jiang commented on code in PR #8812: URL: https://github.com/apache/pinot/pull/8812#discussion_r887304419 ## pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/src/main/java/org/apache/pinot/plugin/ingestion/batch/standalone/SegmentGenerationJobRunner.java:

[GitHub] [pinot] kkrugler commented on issue #8641: "Failed to generate Pinot segment for file" didn't throw an exception

kkrugler commented on issue #8641: URL: https://github.com/apache/pinot/issues/8641#issuecomment-1144155700 Hi @Jackie-Jiang - I can keep support for reporting multiple segment failures, and change this PR to terminate on the first failure. I think support for `cleanupOnFailure` would be ne

[GitHub] [pinot] Jackie-Jiang commented on issue #8641: "Failed to generate Pinot segment for file" didn't throw an exception

Jackie-Jiang commented on issue #8641: URL: https://github.com/apache/pinot/issues/8641#issuecomment-1144163739 @kkrugler Sounds good. To simplify it, we can keep an AtomicReference for the failed file, and when an exception is encountered, we set the reference and drain the latch. In the m

[GitHub] [pinot] xiangfu0 opened a new pull request, #8813: Adding support for broker routing queries to other tenants

xiangfu0 opened a new pull request, #8813: URL: https://github.com/apache/pinot/pull/8813 Allow pinot brokers re-route queries to the right broker tenant to serve the queries. Adding a new broker config: `pinot.broker.route.requests.to.other.tenants` to turn on this feature. -- This i

[GitHub] [pinot] dependabot[bot] opened a new pull request, #8814: Bump eventsource from 1.1.0 to 1.1.1 in /pinot-controller/src/main/resources

dependabot[bot] opened a new pull request, #8814: URL: https://github.com/apache/pinot/pull/8814 Bumps [eventsource](https://github.com/EventSource/eventsource) from 1.1.0 to 1.1.1. Changelog Sourced from https://github.com/EventSource/eventsource/blob/master/HISTORY.md";>eventsour

[pinot] branch dependabot/npm_and_yarn/pinot-controller/src/main/resources/eventsource-1.1.1 created (now d20ec2594d)

This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/pinot-controller/src/main/resources/eventsource-1.1.1 in repository https://gitbox.apache.org/repos/asf/pinot.git at d20ec2594d Bump eventsource in /pinot-contr

[GitHub] [pinot] codecov-commenter commented on pull request #8813: Adding support for broker routing queries to other tenants

codecov-commenter commented on PR #8813: URL: https://github.com/apache/pinot/pull/8813#issuecomment-1144204865 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8813?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] kkrugler commented on issue #8641: "Failed to generate Pinot segment for file" didn't throw an exception

kkrugler commented on issue #8641: URL: https://github.com/apache/pinot/issues/8641#issuecomment-1144218263 @Jackie-Jiang OK, updated the PR accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [pinot] cbalci commented on a diff in pull request #8810: Spark connector pom fixes

cbalci commented on code in PR #8810: URL: https://github.com/apache/pinot/pull/8810#discussion_r887360216 ## pinot-connectors/pinot-spark-connector/pom.xml: ## @@ -199,19 +186,25 @@ -maven-assembly-plugin - -

[GitHub] [pinot] npawar opened a new pull request, #8815: Allow moveToFinalLocation in METADATA push based on config

npawar opened a new pull request, #8815: URL: https://github.com/apache/pinot/pull/8815 `METADATA` push didn't allow the option of `moveSegmentToFinalLocation`. This meant that if someone had generated segments in a location that was not the deep store, there was absolutely no way to move t

[GitHub] [pinot] cbalci commented on pull request #8810: Spark connector pom fixes

cbalci commented on PR #8810: URL: https://github.com/apache/pinot/pull/8810#issuecomment-1144223812 Thanks for the review @xiangfu0! Added a change per your suggestion. Feel free to merge if it looks good, otherwise please let me know. -- This is an automated message from the Apache G

[GitHub] [pinot] codecov-commenter commented on pull request #8814: Bump eventsource from 1.1.0 to 1.1.1 in /pinot-controller/src/main/resources

codecov-commenter commented on PR #8814: URL: https://github.com/apache/pinot/pull/8814#issuecomment-1144232612 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8814?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] xiangfu0 merged pull request #8810: Spark connector pom fixes

xiangfu0 merged PR #8810: URL: https://github.com/apache/pinot/pull/8810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

[pinot] branch master updated: Spark connector pom fixes (#8810)

This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 8b2b8f5762 Spark connector pom fixes (#8810) 8b2

[GitHub] [pinot] xiangfu0 opened a new pull request, #8816: Adding acl support for helm chart

xiangfu0 opened a new pull request, #8816: URL: https://github.com/apache/pinot/pull/8816 Adding ACL support in helmChart. helmChart values.yaml section: ``` pinotAuth: enabled: true controllerFactoryClass: org.apache.pinot.controller.api.access.BasicAuthAccessControlF

[GitHub] [pinot] codecov-commenter commented on pull request #8815: Allow moveToFinalLocation in METADATA push based on config

codecov-commenter commented on PR #8815: URL: https://github.com/apache/pinot/pull/8815#issuecomment-1144242363 # [Codecov](https://codecov.io/gh/apache/pinot/pull/8815?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Softwar

[GitHub] [pinot] xiangfu0 commented on issue #8439: helm chart and tpl for extras config.

xiangfu0 commented on issue #8439: URL: https://github.com/apache/pinot/issues/8439#issuecomment-1144249789 @fcomte Please take a look if there this PR what you are looking for: #8816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8708: Dedup config

Jackie-Jiang commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887385795 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java: ## @@ -0,0 +1,154 @@ +/** + * Licensed to the Apache Sof

[GitHub] [pinot] npawar merged pull request #8815: Allow moveToFinalLocation in METADATA push based on config

npawar merged PR #8815: URL: https://github.com/apache/pinot/pull/8815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.

[pinot] branch master updated (8b2b8f5762 -> 8788f1c0c3)

This is an automated email from the ASF dual-hosted git repository. nehapawar pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 8b2b8f5762 Spark connector pom fixes (#8810) add 8788f1c0c3 Allow moveToFinalLocation in METADATA push based on

[GitHub] [pinot] npawar commented on a diff in pull request #8321: Handle out of range in KafkaConsumer

npawar commented on code in PR #8321: URL: https://github.com/apache/pinot/pull/8321#discussion_r887407920 ## pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelConsumer.java: ## @@ -57,6 +59,14 @@ public

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887547896 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java: ## @@ -538,6 +539,7 @@ private void processStreamEvents(Mess

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887548470 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java: ## @@ -0,0 +1,154 @@ +/** + * Licensed to the Apache Soft

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887549081 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java: ## @@ -0,0 +1,154 @@ +/** + * Licensed to the Apache Soft

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887552243 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java: ## @@ -0,0 +1,154 @@ +/** + * Licensed to the Apache Soft

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887554522 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java: ## @@ -0,0 +1,154 @@ +/** + * Licensed to the Apache Soft

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887554795 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java: ## @@ -477,8 +482,22 @@ public boolean index(GenericR

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887554894 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java: ## @@ -477,8 +482,22 @@ public boolean index(GenericR

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887555107 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/tablestate/TableState.java: ## @@ -0,0 +1,99 @@ +/** + * Licensed to the Apache Software Foun

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887555360 ## pinot-segment-local/src/test/resources/data/test_dedup_data.json: ## @@ -0,0 +1,22 @@ +[ + { +"event_id": "aa", +"description" : "first", +"secondsSinc

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #8708: Dedup config

saurabhd336 commented on code in PR #8708: URL: https://github.com/apache/pinot/pull/8708#discussion_r887554522 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java: ## @@ -0,0 +1,154 @@ +/** + * Licensed to the Apache Soft

[GitHub] [pinot] stym06 commented on a diff in pull request #8779: Change DateTimeFormatSpec delimiter

stym06 commented on code in PR #8779: URL: https://github.com/apache/pinot/pull/8779#discussion_r887598445 ## pinot-spi/src/main/java/org/apache/pinot/spi/data/DateTimeFormatSpec.java: ## @@ -97,6 +157,19 @@ public DateTimeFormatSpec(int columnSize, String columnUnit, String co

[GitHub] [pinot] KKcorps opened a new pull request, #8817: Update ZK config in helm chart

KKcorps opened a new pull request, #8817: URL: https://github.com/apache/pinot/pull/8817 Currently we don't set a retention for zookeeper transaction logs and snapshots in helm chart. This causes storage issues for clusters which have too many metadata updates. -- This is an automated m

[GitHub] [pinot] KKcorps commented on a diff in pull request #8787: Cleanup Kinesis Logs

KKcorps commented on code in PR #8787: URL: https://github.com/apache/pinot/pull/8787#discussion_r887608914 ## pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java: ## @@ -159,23 +159,23 @@ private KinesisRe