[GitHub] [incubator-pinot] kishoreg commented on pull request #5394: add callback interface for upsert component

kishoreg commented on pull request #5394: URL: https://github.com/apache/incubator-pinot/pull/5394#issuecomment-652789093 @jamesyfshao is there anything pending here. Let's get this in. This is an automated message from the A

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5336: [Part 4] Deep-store bypass for LLC: Add a peer to peer segment fetcher.

chenboat commented on a change in pull request #5336: URL: https://github.com/apache/incubator-pinot/pull/5336#discussion_r448741283 ## File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java ## @@ -268,21 +276,79 @@ public

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5639: Refactor how Pinot controller stores segment download url in Zookeeper to deal with peer uri format

chenboat commented on a change in pull request #5639: URL: https://github.com/apache/incubator-pinot/pull/5639#discussion_r448738096 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java ## @@ -485,9 +4

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5639: Refactor how Pinot controller stores segment download url in Zookeeper to deal with peer uri format

chenboat commented on a change in pull request #5639: URL: https://github.com/apache/incubator-pinot/pull/5639#discussion_r448736274 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java ## @@ -355,10 +

[incubator-pinot] branch master updated: SQL Compilation Fixes (#5643)

This is an automated email from the ASF dual-hosted git repository. siddteotia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 9678035 SQL Compilation Fixes (#56

[GitHub] [incubator-pinot] siddharthteotia merged pull request #5643: SQL Compilation Fixes

siddharthteotia merged pull request #5643: URL: https://github.com/apache/incubator-pinot/pull/5643 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [incubator-pinot] kishoreg commented on pull request #5617: Add segment encryption on Controller based on table config

kishoreg commented on pull request #5617: URL: https://github.com/apache/incubator-pinot/pull/5617#issuecomment-652727702 > I would recommend putting this under the `SegmentsValidationAndRetentionConfig` instead of creating another top level config. Why is this the right place for t

[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #5617: Add segment encryption on Controller based on table config

kishoreg commented on a change in pull request #5617: URL: https://github.com/apache/incubator-pinot/pull/5617#discussion_r448697821 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java ## @@ -177

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5652: Use UUID instead of nanoTime to create unique temp file

Jackie-Jiang opened a new pull request #5652: URL: https://github.com/apache/incubator-pinot/pull/5652 Using nanoTime to create unique temp file can cause file name collision (especially in multi-threaded case) This is causing the flakiness of the integration tests when all segments are

[GitHub] [incubator-pinot] kishoreg commented on pull request #5617: Add segment encryption on Controller based on table config

kishoreg commented on pull request #5617: URL: https://github.com/apache/incubator-pinot/pull/5617#issuecomment-652725239 is Crypter part of the pinot-spi? Should we treat this similar to other plugins? This is an automated

[GitHub] [incubator-pinot] mayankshriv commented on a change in pull request #5617: Add segment encryption on Controller based on table config

mayankshriv commented on a change in pull request #5617: URL: https://github.com/apache/incubator-pinot/pull/5617#discussion_r448690902 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java ## @@ -

[GitHub] [incubator-pinot] mcvsubbu removed a comment on issue #5559: Table truncate command

mcvsubbu removed a comment on issue #5559: URL: https://github.com/apache/incubator-pinot/issues/5559#issuecomment-649671122 Good quet This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [incubator-pinot] Jackie-Jiang closed issue #4082: Add API for adding/removing a server to a specific tenant

Jackie-Jiang closed issue #4082: URL: https://github.com/apache/incubator-pinot/issues/4082 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[incubator-pinot] branch master updated: Adding new Controller API for and setting tag for an instance (#4952)

This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 47ff09b Adding new Controller API for

[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #4952: Adding new Controller API for updating the instance config in the cluster

Jackie-Jiang merged pull request #4952: URL: https://github.com/apache/incubator-pinot/pull/4952 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5625: Make DataSource independent of query execution (not extend Operator)

siddharthteotia commented on a change in pull request #5625: URL: https://github.com/apache/incubator-pinot/pull/5625#discussion_r448681287 ## File path: pinot-core/src/test/java/org/apache/pinot/index/readerwriter/FixedByteChunkSVForwardIndexReaderWriterTest.java ## @@ -276,5

[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5625: Make DataSource independent of query execution (not extend Operator)

siddharthteotia commented on a change in pull request #5625: URL: https://github.com/apache/incubator-pinot/pull/5625#discussion_r448681175 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/ForwardIndexCreator.java ## @@ -19,7 +19,105 @@ package org

[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5625: Make DataSource independent of query execution (not extend Operator)

siddharthteotia commented on a change in pull request #5625: URL: https://github.com/apache/incubator-pinot/pull/5625#discussion_r448681078 ## File path: pinot-core/src/main/java/org/apache/pinot/core/io/writer/ForwardIndexWriter.java ## @@ -0,0 +1,138 @@ +/** + * Licensed to

[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5625: Make DataSource independent of query execution (not extend Operator)

siddharthteotia commented on a change in pull request #5625: URL: https://github.com/apache/incubator-pinot/pull/5625#discussion_r448680574 ## File path: pinot-core/src/main/java/org/apache/pinot/core/common/ColumnValueReader.java ## @@ -0,0 +1,248 @@ +/** + * Licensed to the

[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5625: Make DataSource independent of query execution (not extend Operator)

siddharthteotia commented on a change in pull request #5625: URL: https://github.com/apache/incubator-pinot/pull/5625#discussion_r448680805 ## File path: pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/FixedByteChunkSVForwardIndexReader.java ## @@ -16,123 +16,117

[GitHub] [incubator-pinot] npawar opened a new issue #5651: Introduce PinotInputFormat and PinotOutputFormat

npawar opened a new issue #5651: URL: https://github.com/apache/incubator-pinot/issues/5651 to allow easy integration between different systems This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [incubator-pinot] npawar opened a new issue #5650: Map dataType

npawar opened a new issue #5650: URL: https://github.com/apache/incubator-pinot/issues/5650 Support Map type natively in Pinot. Current ways to handle map: 1. stores Map in Avro as 2 MV columns. 2. Store Map in Json as a json string Native Map dataType would be nice

[GitHub] [incubator-pinot] sajjad-moradi commented on a change in pull request #5514: Add integration test for theta sketches

sajjad-moradi commented on a change in pull request #5514: URL: https://github.com/apache/incubator-pinot/pull/5514#discussion_r448667440 ## File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ThetaSketchIntegrationTest.java ## @@ -0,0 +1,263 @@

[GitHub] [incubator-pinot] npawar opened a new issue #5649: Cluster Manager

npawar opened a new issue #5649: URL: https://github.com/apache/incubator-pinot/issues/5649 A cluster manager UI for Pinot operations This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [incubator-pinot] mcvsubbu commented on issue #5648: Kinesis integration

mcvsubbu commented on issue #5648: URL: https://github.com/apache/incubator-pinot/issues/5648#issuecomment-652694430 @KKcorps is working on this, right? For discussion, please join the channel #kinesis-integration One outstanding question here is how to tackle kinesis increase/red

[GitHub] [incubator-pinot] mcvsubbu commented on issue #5647: Ingest multiple streams into 1 Pinot table

mcvsubbu commented on issue #5647: URL: https://github.com/apache/incubator-pinot/issues/5647#issuecomment-652693651 Please add a design doc addressing some general ideas on this feature: 1. Do multiple topics consumed have the same schema or different schemas? 2. Are they expected to

[GitHub] [incubator-pinot] npawar opened a new issue #5648: Kinesis integration

npawar opened a new issue #5648: URL: https://github.com/apache/incubator-pinot/issues/5648 Add connector for ingesting from Amazon Kinesis This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [incubator-pinot] npawar opened a new issue #5647: Ingest multiple streams into 1 Pinot table

npawar opened a new issue #5647: URL: https://github.com/apache/incubator-pinot/issues/5647 A Pinot table can ingest from only 1 topic. The feature request is to be able to configure multiple topics for a single table. T

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5336: [Part 4] Deep-store bypass for LLC: Add a peer to peer segment fetcher.

mcvsubbu commented on a change in pull request #5336: URL: https://github.com/apache/incubator-pinot/pull/5336#discussion_r448664817 ## File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java ## @@ -268,21 +276,79 @@ public

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5639: Refactor how Pinot controller stores segment download url in Zookeeper to deal with peer uri format

mcvsubbu commented on a change in pull request #5639: URL: https://github.com/apache/incubator-pinot/pull/5639#discussion_r448660319 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java ## @@ -355,10 +

[GitHub] [incubator-pinot] cintoSunny opened a new issue #5646: Not creating segments on VPN

cintoSunny opened a new issue #5646: URL: https://github.com/apache/incubator-pinot/issues/5646 When I run the startup scripts for batch with VPN, the Pinot segments are not created.  Pinot version: Saw this error in 0.3.0 and 0.4.0 OS: Mac startup script: bin/quick-sta

[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5608: Pinot Configuration Refactoring - Phase 1

jackjlli commented on a change in pull request #5608: URL: https://github.com/apache/incubator-pinot/pull/5608#discussion_r448610265 ## File path: pinot-spi/src/main/java/org/apache/pinot/spi/env/PinotConfiguration.java ## @@ -0,0 +1,417 @@ +/** + * Licensed to the Apache Soft

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5336: [Part 4] Deep-store bypass for LLC: Add a peer to peer segment fetcher.

chenboat commented on a change in pull request #5336: URL: https://github.com/apache/incubator-pinot/pull/5336#discussion_r448601742 ## File path: pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java ## @@ -35,11 +35,11 @@ private Segment

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5639: Refactor how Pinot controller stores segment download url in Zookeeper to deal with peer uri format

chenboat commented on a change in pull request #5639: URL: https://github.com/apache/incubator-pinot/pull/5639#discussion_r448600944 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java ## @@ -355,8 +3

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5639: Refactor how Pinot controller stores segment download url in Zookeeper to deal with peer uri format

chenboat commented on a change in pull request #5639: URL: https://github.com/apache/incubator-pinot/pull/5639#discussion_r448601088 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java ## @@ -1068,16 +1069,

[incubator-pinot] branch master updated: Add segment encryption on Controller based on table config (#5617)

This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 40a3152 Add segment encryption on Co

[GitHub] [incubator-pinot] mcvsubbu merged pull request #5617: Add segment encryption on Controller based on table config

mcvsubbu merged pull request #5617: URL: https://github.com/apache/incubator-pinot/pull/5617 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-pinot] yupeng9 commented on pull request #5631: Add the message constraint on all instances in Helix

yupeng9 commented on pull request #5631: URL: https://github.com/apache/incubator-pinot/pull/5631#issuecomment-652626786 > Great work digging this out from the Helix code (I'm not able to find this from Helix documentation). > > LGTM in general. I suggest renaming all `maxMessages` t

[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #5631: Add the message constraint on all instances in Helix

yupeng9 commented on a change in pull request #5631: URL: https://github.com/apache/incubator-pinot/pull/5631#discussion_r448593606 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java ## @@ -350,6 +377,9 @@ private void setUpPinotCon

[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #5631: Add the message constraint on all instances in Helix

yupeng9 commented on a change in pull request #5631: URL: https://github.com/apache/incubator-pinot/pull/5631#discussion_r448581387 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java ## @@ -202,6 +209,18 @@ private void setupHelixSy

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5336: [Part 4] Deep-store bypass for LLC: Add a peer to peer segment fetcher.

chenboat commented on a change in pull request #5336: URL: https://github.com/apache/incubator-pinot/pull/5336#discussion_r448548971 ## File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java ## @@ -268,20 +273,81 @@ public

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5336: [Part 4] Deep-store bypass for LLC: Add a peer to peer segment fetcher.

chenboat commented on a change in pull request #5336: URL: https://github.com/apache/incubator-pinot/pull/5336#discussion_r448548586 ## File path: pinot-core/src/main/java/org/apache/pinot/core/util/PeerServerSegmentFinder.java ## @@ -0,0 +1,111 @@ +/** + * Licensed to the Apa

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5639: Refactor how Pinot controller stores segment download url in Zookeeper to deal with peer uri format

mcvsubbu commented on a change in pull request #5639: URL: https://github.com/apache/incubator-pinot/pull/5639#discussion_r448536743 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java ## @@ -1068,16 +1069,

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5336: [Part 4] Deep-store bypass for LLC: Add a peer to peer segment fetcher.

chenboat commented on a change in pull request #5336: URL: https://github.com/apache/incubator-pinot/pull/5336#discussion_r448535327 ## File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java ## @@ -268,20 +276,82 @@ public

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5617: Add segment encryption on Controller based on table config

mcvsubbu commented on a change in pull request #5617: URL: https://github.com/apache/incubator-pinot/pull/5617#discussion_r448527618 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java ## @@ -290

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5643: SQL Compilation Fixes

Jackie-Jiang commented on a change in pull request #5643: URL: https://github.com/apache/incubator-pinot/pull/5643#discussion_r448528028 ## File path: pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java ## @@ -272,7 +271,11 @@ private

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5645: Fix ORC Record reader to ignore extra fields

Jackie-Jiang commented on a change in pull request #5645: URL: https://github.com/apache/incubator-pinot/pull/5645#discussion_r448521328 ## File path: pinot-plugins/pinot-input-format/pinot-orc/src/main/java/org/apache/pinot/plugin/inputformat/orc/ORCRecordReader.java ## @@ -3

[GitHub] [incubator-pinot] mcvsubbu opened a new pull request #5645: Fix ORC Record reader to ignore extra fields

mcvsubbu opened a new pull request #5645: URL: https://github.com/apache/incubator-pinot/pull/5645 Fixing an issue introduced in PR #5267 We should not be validating the type of fields that we don't care about. Cleaned up the messages and exceptions thrown so that we know which fiel

[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5607: [TE] frontend - harleyjj/alert-details - Align time picker on Alert O…

akshayrai commented on a change in pull request #5607: URL: https://github.com/apache/incubator-pinot/pull/5607#discussion_r448514715 ## File path: thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js ## @@ -1449,32 +1449,13 @@ export default Component.ext

[GitHub] [incubator-pinot] akshayrai merged pull request #5607: [TE] frontend - harleyjj/alert-details - Align time picker on Alert O…

akshayrai merged pull request #5607: URL: https://github.com/apache/incubator-pinot/pull/5607 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[incubator-pinot] branch master updated: [TE] frontend - harleyjj/alert-details - Align time picker on Alert Overview and Preview with RCA (#5607)

This is an automated email from the ASF dual-hosted git repository. akshayrai09 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 7405f38 [TE] frontend - harleyjj/

[GitHub] [incubator-pinot] sajjad-moradi commented on a change in pull request #5617: Add segment encryption on Controller based on table config

sajjad-moradi commented on a change in pull request #5617: URL: https://github.com/apache/incubator-pinot/pull/5617#discussion_r448506657 ## File path: pinot-common/src/main/java/org/apache/pinot/common/utils/helix/TableCache.java ## @@ -98,7 +102,7 @@ public synchronized void

[GitHub] [incubator-pinot] sajjad-moradi commented on a change in pull request #5617: Add segment encryption on Controller based on table config

sajjad-moradi commented on a change in pull request #5617: URL: https://github.com/apache/incubator-pinot/pull/5617#discussion_r448505771 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java ## @@

[GitHub] [incubator-pinot] kishoreg commented on issue #5559: Table truncate command

kishoreg commented on issue #5559: URL: https://github.com/apache/incubator-pinot/issues/5559#issuecomment-652537334 @npawar can you point him to setup table code or the validation manager. This is an automated message from

[GitHub] [incubator-pinot] Sreemanth commented on issue #5559: Table truncate command

Sreemanth commented on issue #5559: URL: https://github.com/apache/incubator-pinot/issues/5559#issuecomment-652536385 @kishoreg Step 4 - Enable the table >> Done Step 5- call the setup table method >> Can you give some idea or share some resource about this step? ---

[GitHub] [incubator-pinot] daniellavoie commented on pull request #5608: Pinot Configuration Refactoring - Phase 1

daniellavoie commented on pull request #5608: URL: https://github.com/apache/incubator-pinot/pull/5608#issuecomment-652419549 @jackjlli I will add an extensive description about how configuration key sanitization works with examples. This PR was designed to involve no property key change.

[GitHub] [incubator-pinot] daniellavoie commented on a change in pull request #5608: Pinot Configuration Refactoring - Phase 1

daniellavoie commented on a change in pull request #5608: URL: https://github.com/apache/incubator-pinot/pull/5608#discussion_r448362557 ## File path: pinot-spi/src/main/java/org/apache/pinot/spi/env/PinotConfiguration.java ## @@ -0,0 +1,250 @@ +/** + * Licensed to the Apache

[GitHub] [incubator-pinot] daniellavoie opened a new pull request #5644: Added direnv config file to gitignore

daniellavoie opened a new pull request #5644: URL: https://github.com/apache/incubator-pinot/pull/5644 direnv is great tool that allows settings custom env variables for development without cluttering a ~/.profile script file. See https://direnv.net/ for more details --

[GitHub] [incubator-pinot] siddharthteotia opened a new pull request #5643: SQL compilation fixes

siddharthteotia opened a new pull request #5643: URL: https://github.com/apache/incubator-pinot/pull/5643 Two fixes: - Make in-built functions like TEXT_MATCH, REGEXP_LIKE case insensitive on SQL path. Should be converted to upper case since that is what is there in enum definitions