[incubator-pinot] 01/01: Add docs for record reader

[jackie]

This is an automated email from the ASF dual-hosted git repository.

jackie pushed a commit to branch record_reader_doc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git

commit 27dad760286ba25ecd60bea1ef04be7b4c1a6d8a
Author: Jackie (Xiaotian) Jiang 
AuthorDate: Fri Mar 22 17:35:22 2019 -0700

Add docs for record reader
---
 docs/extensions.rst|   3 +-
 docs/record_reader.rst | 133 +
 2 files changed, 135 insertions(+), 1 deletion(-)

diff --git a/docs/extensions.rst b/docs/extensions.rst
index 43872b8..678a924 100644
--- a/docs/extensions.rst
+++ b/docs/extensions.rst
@@ -26,4 +26,5 @@ This section provides an overview of options to extend Pinot 
code to make Pinot
 
pluggable_streams
segment_fetcher
-   pluggable_storage
\ No newline at end of file
+   record_reader
+   pluggable_storage
diff --git a/docs/record_reader.rst b/docs/record_reader.rst
new file mode 100644
index 000..bf061c5
--- /dev/null
+++ b/docs/record_reader.rst
@@ -0,0 +1,133 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Record Reader
+=
+
+Pinot supports indexing data from various file formats. To support reading 
from a file format, a record reader need to
+be provided to read the file and convert records into the general format which 
the indexing engine can understand. The
+record reader serves as the connector from each individual file format to 
Pinot record format.
+
+Pinot package provides the following record readers out of the box:
+
+- Avro record reader: record reader for Avro format files
+- CSV record reader: record reader for CSV format files
+- JSON record reader: record reader for JSON format files
+- Thrift record reader: record reader for Thrift format files
+- Pinot segment record reader: record reader for Pinot segment
+
+For other file formats, we provide a general interface for record reader - 
`RecordReader`. To index the file into Pinot
+segment, implement the interface and plug it into the index engine - 
`SegmentCreationDriverImpl`.
+
+Initialize Record Reader
+
+
+To initialize a record reader, the data file and table schema should be 
provided (for Pinot segment record reader, only
+need to provide the index directory because schema can be derived from the 
segment). The output record will follow the
+table schema provided.
+
+For Avro/JSON/Pinot segment record reader, no extra configuration is required 
as column names and multi-values are
+embedded in the data file.
+
+For CSV/Thrift record reader, extra configuration might be provided to 
determine the column names and multi-values for
+the data.
+
+CSV Record Reader Config
+
+
+The CSV record reader config contains the following settings:
+
+- Header: the header for the CSV file (column names)
+- Column delimiter: delimiter for each column
+- Multi-value delimiter: delimiter for each value for a multi-valued column
+
+If no config provided, use the default setting:
+
+- Use the first row in the data file as the header
+- Use ',' as the column delimiter
+- Use ';' as the multi-value delimiter
+
+Thrift Record Reader Config
+~~~
+
+The Thrift record reader config is mandatory. It contains the Thrift class 
name for the record reader to de-serialize
+the Thrift objects.
+
+Implement Your Own Record Reader
+
+
+You can implement your own record reader for file formats that is not 
supported natively. The following methods need to
+be implemented:
+
+.. code-block:: none
+
+  /**
+   * Return true if more records remain to be read.
+   */
+  boolean hasNext();
+
+  /**
+   * Get the next record.
+   */
+  GenericRow next()
+  throws IOException;
+
+  /**
+   * Get the next record. Re-use the given row if possible to reduce garbage.
+   * The passed in row should be returned by previous call to next().
+   */
+  GenericRow next(GenericRow reuse)
+  throws IOException;
+
+  /**
+   * Rewind the reader to start reading from the first record again.
+   */
+  void rewind()
+  throws IOException;
+
+  /**
+   * Get the Pinot schema.
+   */
+  Schema getSchema();
+
+
+Generic Row
+~~

[incubator-pinot] branch record_reader_doc created (now 27dad76)

[jackie]

This is an automated email from the ASF dual-hosted git repository.

jackie pushed a change to branch record_reader_doc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


  at 27dad76  Add docs for record reader

This branch includes the following new commits:

 new 27dad76  Add docs for record reader

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch orcMV updated (d800052 -> c48cd8e)

[jenniferdai]

This is an automated email from the ASF dual-hosted git repository.

jenniferdai pushed a change to branch orcMV
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


 discard d800052  Adding mv orc test
 new c48cd8e  Adding ORC Multivalue Support

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (d800052)
\
 N -- N -- N   refs/heads/orcMV (c48cd8e)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 5871 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 pinot-orc/pom.xml | 4 
 1 file changed, 4 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jenniferdai opened a new pull request #4009: Adding ORC Multi-value Column Support

[GitBox]

jenniferdai opened a new pull request #4009: Adding ORC Multi-value Column 
Support
URL: https://github.com/apache/incubator-pinot/pull/4009
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch orcMV updated (57f5fc0 -> d800052)

[jenniferdai]

This is an automated email from the ASF dual-hosted git repository.

jenniferdai pushed a change to branch orcMV
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


 discard 57f5fc0  Adding mv orc test
 new d800052  Adding mv orc test

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (57f5fc0)
\
 N -- N -- N   refs/heads/orcMV (d800052)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 5871 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/pinot/orc/data/readers/ORCRecordReader.java | 6 +-
 .../java/org/apache/pinot/orc/data/readers/ORCRecordReaderTest.java | 2 +-
 2 files changed, 2 insertions(+), 6 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch orcMV updated (5d9e806 -> 57f5fc0)

[jenniferdai]

This is an automated email from the ASF dual-hosted git repository.

jenniferdai pushed a change to branch orcMV
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


 discard 5d9e806  Add mv support
 discard 025159e  Fixing dependency issues
 discard 7f6263b  fixing unit test
 discard 91767a1  Adding ORC Reader Test
 discard d6c3a27  Addressing comments
 discard 2e6bf2f  Fixing dependencies and updating javadoc
 discard d620318  Editing reusable vectorized row batch to really be reusable
 discard ea75c00  Revert "Copying orc reader to pinot core for now so I don't 
have to edit the pinot script internally to publish jars" - giving up because 
so many dependency problems"
 discard d604449  Revert "Fixing maven enforcer dependency issuse"
 discard 63336cc  Fixing maven enforcer dependency issuse
 discard 41c6bf7  Copying orc reader to pinot core for now so I don't have to 
edit the pinot script internally to publish jars
 discard e51533f  Adding orc reader
 new 9e8e373  [TE] detection - preview a yaml with existing anomalies 
(#3983)
 new 565171f  add config to control kafka fetcher size and increase default 
(#3869)
 new f815e2e  [TE] detection - align metric slices (#3981)
 new df62374  [TE] frontend - harleyjj/preview - default preview to 2 days 
to accomodate daily metrics (#3980)
 new 2c5d42a  [TE] frontend - harleyjj/report-anomaly - adds back 
report-anomaly modal to alert overview (#3985)
 new d2a3d84  [TE] Fix for delayed anomalies due to watermark bug (#3984)
 new fe203b5  Pinot server side change to optimize LLC segment completion 
with direct metadata upload.  (#3941)
 new f26b2f3  [TE] Remove deprecated legacy logic in user dashboard (#3988)
 new 31f4fd0  [TE] frontend - harleyjj/edit-alert - update endpoint for 
preview when editing alert (#3987)
 new 59fd4aa  Add documentation (#3986)
 new 98dcebc  Add experiment section in getting started (#3989)
 new 205ec50  Update managing pinot doc (#3991)
 new d8061f3  [TE] frontend - harleyjj/edit-alert - fix subscription group 
put bug (#3995)
 new 6eb8e79  Fixing type casting issue for BYTES type values during 
realtime segment persistence (#3992)
 new d78a807  [TE] Clean up the yaml editor calls and messages (#3996)
 new eccf573  In TableConfig, add checks for mandatory fields (#3993)
 new d78160d  [TE] Aggregation function and double series aggregation 
mapping (#3999)
 new dd595cf  Updated realtime provisioning helper to take in max memory 
available in a host (#3997)
 new dfaaf11  [TE] yaml - anomaly filter validation (#4000)
 new a57caf0  Reserve version 0.2.0 for the next release (#3807)
 new 8074d0a  Adding ORC reader (#3994)
 new 21930fa  Multivalue column support
 new 57f5fc0  Adding mv orc test

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (5d9e806)
\
 N -- N -- N   refs/heads/orcMV (57f5fc0)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 5871 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 docs/getting_started.rst   | 156 ++
 docs/img/generate-segment.png  | Bin 218597 -> 0 bytes
 docs/img/list-schemas.png  | Bin 8952 -> 247946 bytes
 docs/img/pinot-console.png | Bin 0 -> 157310 bytes
 docs/img/query-table.png   | Bin 35914 -> 0 bytes
 docs/img/rebalance-table.png   | Bin 0 -> 164989 bytes
 docs/img/upload-segment.png| Bin 13944 -> 0 bytes
 docs/management_api.rst|  67 +++-
 pinot-api/pom.xml  |   2 +-
 pinot-azure-filesystem/pom.xml |   4 +-
 pinot-broker/pom.xml   |   2 +-
 .../queryquota/TableQueryQuotaManagerTest.java |  15 +-
 .../broker/routing/TimeBoundaryServiceTest.java|   3 +-
 .../HighLevelConsumerRoutingTableBuilderTest.java  |   4 +-
 .../LowLevelConsumerRoutingTableBuilderTest.java   |  12 +-
 pinot-common/pom.xml   |   2 +-
 .../apache/pinot/common/config/TableConfig.java| 235 --
 .../protocols/SegmentCompletionProtocol.java   |   6 +
 .../apache/pinot/com

[GitHub] [incubator-pinot] akshayrai merged pull request #4003: [TE] Fix ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret…

[GitBox]

akshayrai merged pull request #4003: [TE] Fix ArrayIndexOutOfBoundsException; 
Wrap replay endpoint and ret…
URL: https://github.com/apache/incubator-pinot/pull/4003
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: [TE] Fix ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret… (#4003)

[akshayrai09]

This is an automated email from the ASF dual-hosted git repository.

akshayrai09 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new d58f8bc  [TE] Fix ArrayIndexOutOfBoundsException; Wrap replay endpoint 
and ret… (#4003)
d58f8bc is described below

commit d58f8bce4b59de096b4ee9fee61c679482dd1d7d
Author: Akshay Rai 
AuthorDate: Fri Mar 22 16:03:14 2019 -0700

[TE] Fix ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret… 
(#4003)
---
 .../thirdeye/detection/DetectionResource.java  | 136 -
 .../PercentageChangeRuleAnomalyFilter.java |  24 +++-
 2 files changed, 97 insertions(+), 63 deletions(-)

diff --git 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
index 1504640..38fad77 100644
--- 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
+++ 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
@@ -29,6 +29,7 @@ import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.Date;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.LinkedHashMap;
 import java.util.List;
@@ -338,7 +339,6 @@ public class DetectionResource {
* @param windowSize (optional) override the default window size
* @param bucketSize (optional) override the default window size
* @return anomalies
-   * @throws Exception
*/
   @POST
   @Path("/legacy-replay/{id}")
@@ -350,47 +350,55 @@ public class DetectionResource {
   @QueryParam("end") long end,
   @QueryParam("deleteExistingAnomaly") @DefaultValue("false") boolean 
deleteExistingAnomaly,
   @QueryParam("windowSize") Long windowSize,
-  @QueryParam("bucketSize") Long bucketSize) throws Exception {
-
-DetectionConfigDTO config = this.configDAO.findById(configId);
-if (config == null) {
-  throw new IllegalArgumentException(String.format("Cannot find config 
%d", configId));
-}
-
-AnomalySlice slice = new AnomalySlice().withStart(start).withEnd(end);
-if (deleteExistingAnomaly) {
-  // clear existing anomalies
-  Collection existing =
-  this.provider.fetchAnomalies(Collections.singleton(slice), 
configId).get(slice);
-
-  List existingIds = new ArrayList<>();
-  for (MergedAnomalyResultDTO anomaly : existing) {
-existingIds.add(anomaly.getId());
+  @QueryParam("bucketSize") Long bucketSize) {
+Map responseMessage = new HashMap<>();
+Collection replayResult;
+try {
+  DetectionConfigDTO config = this.configDAO.findById(configId);
+  if (config == null) {
+throw new IllegalArgumentException(String.format("Cannot find config 
%d", configId));
   }
-  this.anomalyDAO.deleteByIds(existingIds);
-}
-
-// execute replay
-List monitoringWindows = getReplayMonitoringWindows(config, 
start, end, windowSize, bucketSize);
-for (Interval monitoringWindow : monitoringWindows){
-  DetectionPipeline pipeline = this.loader.from(this.provider, config, 
monitoringWindow.getStartMillis(), monitoringWindow.getEndMillis());
-  DetectionPipelineResult result = pipeline.run();
 
-  // Update
-  if (result.getLastTimestamp() > config.getLastTimestamp()) {
-config.setLastTimestamp(result.getLastTimestamp());
-this.configDAO.update(config);
+  AnomalySlice slice = new AnomalySlice().withStart(start).withEnd(end);
+  if (deleteExistingAnomaly) {
+// clear existing anomalies
+Collection existing =
+this.provider.fetchAnomalies(Collections.singleton(slice), 
configId).get(slice);
+
+List existingIds = new ArrayList<>();
+for (MergedAnomalyResultDTO anomaly : existing) {
+  existingIds.add(anomaly.getId());
+}
+this.anomalyDAO.deleteByIds(existingIds);
   }
 
-  for (MergedAnomalyResultDTO anomaly : result.getAnomalies()) {
-anomaly.setAnomalyResultSource(AnomalyResultSource.ANOMALY_REPLAY);
-this.anomalyDAO.save(anomaly);
+  // execute replay
+  List monitoringWindows = getReplayMonitoringWindows(config, 
start, end, windowSize, bucketSize);
+  for (Interval monitoringWindow : monitoringWindows){
+DetectionPipeline pipeline = this.loader.from(this.provider, config, 
monitoringWindow.getStartMillis(), monitoringWindow.getEndMillis());
+DetectionPipelineResult result = pipeline.run();
+
+// Update
+if (result.getLastTimestamp() > config.getLastTimestamp()) {
+  config.setLastTimestamp(result.getLastTimestamp());
+  this.configDAO.update(config);
+}
+
+for (MergedAnomalyResultDTO anomaly : result

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268340700
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,61 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+Pinot exposes several metrics to monitor the service and ensure that pinot 
users are not experiencing issues. In this section we discuss some of the key 
metrics that are useful to monitor. A full list of metrics is available in the 
`Metrics `_ section.
+
+Pinot Server
+
+
+* Missing Segments - `NUM_MISSING_SEGMENTS 
`_
+
+  * Number of missing segments that the broker queried for (expected to be on 
the server) but the server didn't have. This can be due to retention or stale 
routing table.
+
+* Query latency - `TOTAL_QUERY_TIME 
`_
+
+  * The number of exception which might have occurred during query execution
+
+* Query Execution Exceptions - `QUERY_EXECUTION_EXCEPTIONS 
`_
+
+  * The number of exception which might have occurred during query execution
+
+* Realtime Consumption Status - `LLC_PARTITION_CONSUMING 
`_
+
+  * This gives a binary value based on whether low-level consumption is 
healthy (1) or unhealthy (0). It's important to ensure at least a single 
replica of each partition is consuming
+
+* Realtime Highest Offset Consumed - `HIGHEST_STREAM_OFFSET_CONSUMED 
`_
+
+  * The highest offset which has been consumed so far.
+
+Pinot Broker
+
+
+* Incoming QPS (per broker) - `QUERIES 
`_
+
+  * The rate which an individual broker is receiving queries. Units are in QPS.
+
+* Dropped Requests - `REQUEST_DROPPED_DUE_TO_SEND_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_CONNECTION_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_ACCESS_ERROR 
`_
+
+  * These multiple metrics will indicate if a query is dropped, ie the 
processing of that query has been forfeited for some reason.
+
+* Partial Responses - `BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED 
`_
+
+  * Indicates a count of partial responses. A partial response is when at 
least 1 of the requested servers fails to respond to the query.
+
+* Table QPS quota exceeded - `QUERY_QUOTA_EXCEEDED 
`_
+
+  * Binary metric which will indicate when the configured QPS quota for a 
table is exceeded (1) or if there is capacity remaining (0).
+
+* Table QPS quota usage percent - `QUERY_QUOTA_CAPACITY_UTILIZATION_RATE 
`_
+
+  * Percentage of the configured QPS quota being utilized.
+
+Pinot Controller
 
 Review comment:
   See the ControllerGauge class.
   
   In general, we have classes BrokerGauge, BrokerMeter, ControllerGauge, 
ControllerMeter, ServerGauge. ServerMeter, MinionGauge, MinionMeter that define 
metric names. There are methiods in the base class that add table name if 
necessary, and  methods that add phase timings (e.g. query phase times)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr..

[GitHub] [incubator-pinot] codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table 
Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#issuecomment-473083001
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=h1) 
Report
   > Merging 
[#3975](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/5867b16f32145a80fa98de4d24ec6302b17bb95e?src=pr&el=desc)
 will **increase** coverage by `0.11%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/3975/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff @@
   ## master   #3975  +/-   ##
   ===
   + Coverage 66.99%   67.1%   +0.11% 
   - Complexity4  17  +13 
   ===
 Files  10321033   +1 
 Lines 51047   51244 +197 
 Branches   71267175  +49 
   ===
   + Hits  34200   34389 +189 
   + Misses14502   14478  -24 
   - Partials   23452377  +32
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...e/operator/dociditerators/MVScanDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9NVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=)
 | `48.48% <0%> (-12.13%)` | `0% <0%> (ø)` | |
   | 
[...impl/dictionary/DoubleOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==)
 | `68.88% <0%> (-6.67%)` | `0% <0%> (ø)` | |
   | 
[...impl/dictionary/FloatOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPZmZIZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==)
 | `87.27% <0%> (-5.46%)` | `0% <0%> (ø)` | |
   | 
[...mpl/dictionary/DoubleOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT2ZmSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=)
 | `72.72% <0%> (-3.64%)` | `0% <0%> (ø)` | |
   | 
[...api/resources/PinotTableConfigRestletResource.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFibGVDb25maWdSZXN0bGV0UmVzb3VyY2UuamF2YQ==)
 | `24.35% <0%> (-3.28%)` | `0% <0%> (ø)` | |
   | 
[...time/converter/stats/RealtimeColumnStatistics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9jb252ZXJ0ZXIvc3RhdHMvUmVhbHRpbWVDb2x1bW5TdGF0aXN0aWNzLmphdmE=)
 | `47.22% <0%> (-2.06%)` | `0% <0%> (ø)` | |
   | 
[...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=)
 | `41.17% <0%> (-1.48%)` | `0% <0%> (ø)` | |
   | 
[...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=)
 | `47.65% <0%> (-1.44%)` | `0% <0%> (ø)` | |
   | 
[...g/apache/pinot/common/metrics/AbstractMetrics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9BYnN0cmFjdE1ldHJpY3MuamF2YQ==)
 | `75.33% <0%> (-1.34%)` | `0% <0%> (ø)` | |
   | 
[.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==)
 | `92.68% <0%> (-1.15%)` | `0% <0%> (ø)` | |
   | ... and [41 
more](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree-more)
 | |
   
   --
   
   [Continue to review full report at 

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268339141
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -40,11 +40,14 @@ Recommended environment
 Deploying Pinot
 ---
 
-Direct deployment of Pinot
-~~
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended incase of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
 Review comment:
   Can you update the requirments section to be not as vague as it is now? A 
paragraph would do. Here is a suggestion. We can iterate and modify later, but 
let us have something in place now. Feel free to modify/add to as you please.
   
   You will need the following in order to run pinot in production:
   * Hardware for controller/broker/servers as per your load
   * Working installation of Zookeeper that Pinot can use. We recommend setting 
aside a path within zookpeer and including that path in pinot.controller.zkStr. 
Pinot will create its own cluster under this path (cluster name decided by 
pinot.controller.helixClusterName)
   * Shared storage mounted on controllers (if you plan to have multiple 
controllers for the same cluster). Alternatively, an implementation of PinotFS 
that the Pinot hosts have access to.
   * HTTP load balancers for spraying queries across brokers (or other 
mechanism to balance queries)
   * HTTP load balancers for spraying controller requests (e.g. segment push, 
or other controller APIs) or other mechanisms for distribution of these 
requests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: [TE] Update pinot version to 0.2.0-SNAPSHOT (#4008)

[akshayrai09]

This is an automated email from the ASF dual-hosted git repository.

akshayrai09 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new f636f97  [TE] Update pinot version to 0.2.0-SNAPSHOT (#4008)
f636f97 is described below

commit f636f9735acba74b096496a2bf4cfa7abf3f4515
Author: Akshay Rai 
AuthorDate: Fri Mar 22 14:00:20 2019 -0700

[TE] Update pinot version to 0.2.0-SNAPSHOT (#4008)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] akshayrai merged pull request #4008: [TE] Update pinot version to 0.2.0-SNAPSHOT

[GitBox]

akshayrai merged pull request #4008: [TE] Update pinot version to 0.2.0-SNAPSHOT
URL: https://github.com/apache/incubator-pinot/pull/4008
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch add-participant-role-in-controller deleted (was 4c77143)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch add-participant-role-in-controller
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


 was 4c77143  Address PR comments

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jackjlli merged pull request #3864: Add controller mode logic in ControllerStarter

[GitBox]

jackjlli merged pull request #3864: Add controller mode logic in 
ControllerStarter
URL: https://github.com/apache/incubator-pinot/pull/3864
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: Add controller mode logic in ControllerStarter (#3864)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new c9fbd6a  Add controller mode logic in ControllerStarter (#3864)
c9fbd6a is described below

commit c9fbd6a11ca18c6497462f4698130d2735addc77
Author: Jialiang Li 
AuthorDate: Fri Mar 22 13:56:13 2019 -0700

Add controller mode logic in ControllerStarter (#3864)

* Add controller mode logic

* Move controller mode out of resource manager
---
 .../broker/broker/HelixBrokerStarterTest.java  |  76 +++-
 .../apache/pinot/common/utils/CommonConstants.java |   1 +
 .../apache/pinot/controller/ControllerConf.java|  17 ++
 .../controller/ControllerLeadershipManager.java|   2 +
 .../apache/pinot/controller/ControllerStarter.java | 205 -
 .../helix/core/PinotHelixResourceManager.java  |  32 +++-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |   3 +
 .../core/realtime/SegmentCompletionManager.java|   5 +
 .../rebalance/RebalanceSegmentStrategyFactory.java |   5 +
 .../PinotInstanceRestletResourceTest.java  |  20 +-
 .../controller/api/resources/TableViewsTest.java   |   2 +
 .../helix/ControllerPeriodicTaskStarterTest.java   |   8 +-
 .../pinot/controller/helix/ControllerTest.java |  40 +++-
 .../controller/helix/PinotControllerModeTest.java  | 115 
 .../controller/helix/PinotResourceManagerTest.java |  56 +++---
 .../sharding/SegmentAssignmentStrategyTest.java|  88 -
 .../validation/ValidationManagerTest.java  |  57 +++---
 ...vertToRawIndexMinionClusterIntegrationTest.java |   4 +-
 .../tests/OfflineClusterIntegrationTest.java   |   3 +-
 .../admin/command/StartControllerCommand.java  |   7 +-
 20 files changed, 511 insertions(+), 235 deletions(-)

diff --git 
a/pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterTest.java
 
b/pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterTest.java
index a9a7568..08387a8 100644
--- 
a/pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterTest.java
+++ 
b/pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterTest.java
@@ -19,7 +19,6 @@
 package org.apache.pinot.broker.broker;
 
 import com.google.common.util.concurrent.Uninterruptibles;
-import com.yammer.metrics.core.MetricsRegistry;
 import java.io.IOException;
 import java.lang.reflect.Field;
 import java.util.ArrayList;
@@ -31,10 +30,8 @@ import java.util.concurrent.Callable;
 import java.util.concurrent.TimeUnit;
 import org.I0Itec.zkclient.ZkClient;
 import org.apache.commons.configuration.Configuration;
-import org.apache.helix.HelixAdmin;
 import org.apache.helix.model.ExternalView;
 import org.apache.helix.model.IdealState;
-import org.apache.helix.model.InstanceConfig;
 import org.apache.pinot.broker.broker.helix.DefaultHelixBrokerConfig;
 import org.apache.pinot.broker.broker.helix.HelixBrokerStarter;
 import org.apache.pinot.broker.routing.HelixExternalViewBasedRouting;
@@ -44,13 +41,10 @@ import org.apache.pinot.common.config.TableConfig;
 import org.apache.pinot.common.config.TableNameBuilder;
 import org.apache.pinot.common.data.Schema;
 import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
-import org.apache.pinot.common.metrics.ControllerMetrics;
 import org.apache.pinot.common.utils.CommonConstants;
 import org.apache.pinot.common.utils.ZkStarter;
-import org.apache.pinot.controller.ControllerConf;
 import org.apache.pinot.controller.helix.ControllerRequestBuilderUtil;
-import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
-import 
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager;
+import org.apache.pinot.controller.helix.ControllerTest;
 import org.apache.pinot.controller.utils.SegmentMetadataMockUtils;
 import org.testng.Assert;
 import org.testng.annotations.AfterTest;
@@ -58,17 +52,14 @@ import org.testng.annotations.BeforeTest;
 import org.testng.annotations.Test;
 
 
-public class HelixBrokerStarterTest {
+public class HelixBrokerStarterTest extends ControllerTest {
   private static final int SEGMENT_COUNT = 6;
-  private PinotHelixResourceManager _pinotResourceManager;
-  private static final String HELIX_CLUSTER_NAME = "TestHelixBrokerStarter";
   private static final String RAW_DINING_TABLE_NAME = "dining";
   private static final String DINING_TABLE_NAME = 
TableNameBuilder.OFFLINE.tableNameWithType(RAW_DINING_TABLE_NAME);
   private static final String COFFEE_TABLE_NAME = 
TableNameBuilder.OFFLINE.tableNameWithType("coffee");
   private final Configuration _pinotHelixBrokerProperties = 
DefaultHelixBrokerConfig.getDefaultBrokerConf();
 
   private ZkClient _zkClient;
-  private HelixAdmin _helixAdmin;
   private HelixBrokerStarter _helixBrokerStarter;
   private ZkStart

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268336400
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
+
+  * Allowed values - String
+
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+
+  * Allowed values - String, this must match the ``timeFieldSpec`` section in 
the schema
+
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "myPinotSchmea",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* ``invertedIndexColumns`` - Indicates a list of real column names as 
specified in the schema to create inv

[GitHub] [incubator-pinot] jihaozh commented on a change in pull request #4003: [TE] Fix ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret…

[GitBox]

jihaozh commented on a change in pull request #4003: [TE] Fix 
ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret…
URL: https://github.com/apache/incubator-pinot/pull/4003#discussion_r268335170
 
 

 ##
 File path: 
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleAnomalyFilter.java
 ##
 @@ -67,6 +70,11 @@ public boolean isQualified(MergedAnomalyResultDTO anomaly) {
 
 Map aggregates =
 this.dataFetcher.fetchData(new 
InputDataSpec().withAggregateSlices(slices)).getAggregates();
+if (aggregates.get(currentSlice).isEmpty() || 
aggregates.get(baselineSlice).isEmpty()) {
+  LOG.warn("Unable to fetch data for current or baseline slice for anomaly 
{}. start = {} end = {} filters = {}",
+  anomaly.getId(), anomaly.getStartTime(), anomaly.getEndTime(), 
me.getFilters());
+  return false;
 
 Review comment:
   right, now I think of it, in this case, should we use the value stored in 
the anomaly instead?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #4003: [TE] Fix ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret…

[GitBox]

akshayrai commented on a change in pull request #4003: [TE] Fix 
ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret…
URL: https://github.com/apache/incubator-pinot/pull/4003#discussion_r268333586
 
 

 ##
 File path: 
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleAnomalyFilter.java
 ##
 @@ -67,6 +70,11 @@ public boolean isQualified(MergedAnomalyResultDTO anomaly) {
 
 Map aggregates =
 this.dataFetcher.fetchData(new 
InputDataSpec().withAggregateSlices(slices)).getAggregates();
+if (aggregates.get(currentSlice).isEmpty() || 
aggregates.get(baselineSlice).isEmpty()) {
+  LOG.warn("Unable to fetch data for current or baseline slice for anomaly 
{}. start = {} end = {} filters = {}",
+  anomaly.getId(), anomaly.getStartTime(), anomaly.getEndTime(), 
me.getFilters());
+  return false;
 
 Review comment:
   Ideally, we shouldn't get this error. If we do get it, then displaying the 
anomaly can mislead the users. Returning true would mean that the anomaly 
passed the filter when in fact it didn't, right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jihaozh commented on a change in pull request #4003: [TE] Fix ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret…

[GitBox]

jihaozh commented on a change in pull request #4003: [TE] Fix 
ArrayIndexOutOfBoundsException; Wrap replay endpoint and ret…
URL: https://github.com/apache/incubator-pinot/pull/4003#discussion_r268332423
 
 

 ##
 File path: 
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleAnomalyFilter.java
 ##
 @@ -67,6 +70,11 @@ public boolean isQualified(MergedAnomalyResultDTO anomaly) {
 
 Map aggregates =
 this.dataFetcher.fetchData(new 
InputDataSpec().withAggregateSlices(slices)).getAggregates();
+if (aggregates.get(currentSlice).isEmpty() || 
aggregates.get(baselineSlice).isEmpty()) {
+  LOG.warn("Unable to fetch data for current or baseline slice for anomaly 
{}. start = {} end = {} filters = {}",
+  anomaly.getId(), anomaly.getStartTime(), anomaly.getEndTime(), 
me.getFilters());
+  return false;
 
 Review comment:
   should it return true? otherwise, the anomaly is suppressed. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot.wiki] branch master updated: also need to add replicasPerPartition for a valid realtime table config

[unknown]

This is an automated email from the ASF dual-hosted git repository.

unknown pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 2fc30f1  also need to add replicasPerPartition for a valid realtime 
table config
2fc30f1 is described below

commit 2fc30f1d2176a06d537dff4b66b3bc04af17427b
Author: Fred Weitendorf <37577626+fredxf...@users.noreply.github.com>
AuthorDate: Fri Mar 22 13:41:47 2019 -0700

also need to add replicasPerPartition for a valid realtime table config
---
 Pinot-Realtime-Workflow.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Pinot-Realtime-Workflow.md b/Pinot-Realtime-Workflow.md
index a9bf0a9..dab69f0 100644
--- a/Pinot-Realtime-Workflow.md
+++ b/Pinot-Realtime-Workflow.md
@@ -8,6 +8,7 @@ While creating a new resource you have to specify the 
resourceType as REALTIME a
 {
 "tableName":"RealtimeTable",
 "segmentsConfig": {
+"replicasPerPartition":"1",
 "retentionTimeUnit":"DAYS",
 "retentionTimeValue":"5",
 "segmentPushFrequency":"daily",


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] akshayrai opened a new pull request #4008: [TE] Update pinot version to 0.2.0-SNAPSHOT

[GitBox]

akshayrai opened a new pull request #4008: [TE] Update pinot version to 
0.2.0-SNAPSHOT
URL: https://github.com/apache/incubator-pinot/pull/4008
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] kishoreg closed pull request #4008: [TE] Update pinot version to 0.2.0-SNAPSHOT

[GitBox]

kishoreg closed pull request #4008: [TE] Update pinot version to 0.2.0-SNAPSHOT
URL: https://github.com/apache/incubator-pinot/pull/4008
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table 
Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#issuecomment-473083001
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=h1) 
Report
   > Merging 
[#3975](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/5867b16f32145a80fa98de4d24ec6302b17bb95e?src=pr&el=desc)
 will **increase** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/3975/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#3975  +/-   ##
   
   + Coverage 66.99%   67.02%   +0.02% 
   - Complexity4   17  +13 
   
 Files  1032 1033   +1 
 Lines 5104751244 +197 
 Branches   7126 7175  +49 
   
   + Hits  3420034344 +144 
   - Misses1450214531  +29 
   - Partials   2345 2369  +24
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==)
 | `50% <0%> (-50%)` | `0% <0%> (ø)` | |
   | 
[.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh)
 | `87.27% <0%> (-5.46%)` | `0% <0%> (ø)` | |
   | 
[...api/resources/PinotTableConfigRestletResource.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFibGVDb25maWdSZXN0bGV0UmVzb3VyY2UuamF2YQ==)
 | `24.35% <0%> (-3.28%)` | `0% <0%> (ø)` | |
   | 
[...manager/realtime/HLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh)
 | `78.35% <0%> (-2.58%)` | `0% <0%> (ø)` | |
   | 
[...time/converter/stats/RealtimeColumnStatistics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9jb252ZXJ0ZXIvc3RhdHMvUmVhbHRpbWVDb2x1bW5TdGF0aXN0aWNzLmphdmE=)
 | `47.22% <0%> (-2.06%)` | `0% <0%> (ø)` | |
   | 
[...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=)
 | `47.65% <0%> (-1.44%)` | `0% <0%> (ø)` | |
   | 
[...ot/common/protocols/SegmentCompletionProtocol.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcHJvdG9jb2xzL1NlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2wuamF2YQ==)
 | `92.48% <0%> (-1.09%)` | `0% <0%> (ø)` | |
   | 
[...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh)
 | `71.85% <0%> (-1.01%)` | `0% <0%> (ø)` | |
   | 
[...manager/realtime/LLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvTExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh)
 | `69.87% <0%> (-0.99%)` | `0% <0%> (ø)` | |
   | 
[...pby/NoDictionarySingleColumnGroupKeyGenerator.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9ncm91cGJ5L05vRGljdGlvbmFyeVNpbmdsZUNvbHVtbkdyb3VwS2V5R2VuZXJhdG9yLmphdmE=)
 | `87.5% <0%> (-0.97%)` | `0% <0%> (ø)` | |
   | ... and [35 
more](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tre

[GitHub] [incubator-pinot] akshayrai opened a new pull request #4008: [TE] Update pinot version to 0.2.0-SNAPSHOT

[GitBox]

akshayrai opened a new pull request #4008: [TE] Update pinot version to 
0.2.0-SNAPSHOT
URL: https://github.com/apache/incubator-pinot/pull/4008
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot.wiki] branch master updated: You need a metadata field defined to be able to upload the table config without errors

[unknown]

This is an automated email from the ASF dual-hosted git repository.

unknown pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 3f13697  You need a metadata field defined to be able to upload the 
table config without errors
3f13697 is described below

commit 3f1369732ed5f4766c69e6a83be38c53d5ea3b74
Author: Fred Weitendorf <37577626+fredxf...@users.noreply.github.com>
AuthorDate: Fri Mar 22 13:37:20 2019 -0700

You need a metadata field defined to be able to upload the table config 
without errors
---
 Pinot-Realtime-Workflow.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Pinot-Realtime-Workflow.md b/Pinot-Realtime-Workflow.md
index 358161c..a9bf0a9 100644
--- a/Pinot-Realtime-Workflow.md
+++ b/Pinot-Realtime-Workflow.md
@@ -37,6 +37,7 @@ While creating a new resource you have to specify the 
resourceType as REALTIME a
 "server":"DefaultTenant_SERVER"
 },
 "tableType":"REALTIME",
+"metadata":{}
 }
 ```
 


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table 
Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#issuecomment-473083001
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=h1) 
Report
   > Merging 
[#3975](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/5867b16f32145a80fa98de4d24ec6302b17bb95e?src=pr&el=desc)
 will **increase** coverage by `0.21%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/3975/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#3975  +/-   ##
   
   + Coverage 66.99%   67.21%   +0.21% 
   - Complexity4   17  +13 
   
 Files  1032 1033   +1 
 Lines 5104751244 +197 
 Branches   7126 7175  +49 
   
   + Hits  3420034443 +243 
   + Misses1450214417  -85 
   - Partials   2345 2384  +39
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==)
 | `75% <0%> (-25%)` | `0% <0%> (ø)` | |
   | 
[...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=)
 | `92.3% <0%> (-7.7%)` | `0% <0%> (ø)` | |
   | 
[.../apache/pinot/common/config/RealtimeTagConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1JlYWx0aW1lVGFnQ29uZmlnLmphdmE=)
 | `93.33% <0%> (-6.67%)` | `0% <0%> (ø)` | |
   | 
[...elix/core/relocation/RealtimeSegmentRelocator.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlbG9jYXRpb24vUmVhbHRpbWVTZWdtZW50UmVsb2NhdG9yLmphdmE=)
 | `74.72% <0%> (-6.6%)` | `0% <0%> (ø)` | |
   | 
[.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh)
 | `87.27% <0%> (-5.46%)` | `0% <0%> (ø)` | |
   | 
[...impl/dictionary/FloatOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPZmZIZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==)
 | `87.27% <0%> (-5.46%)` | `0% <0%> (ø)` | |
   | 
[...api/resources/PinotTableConfigRestletResource.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFibGVDb25maWdSZXN0bGV0UmVzb3VyY2UuamF2YQ==)
 | `24.35% <0%> (-3.28%)` | `0% <0%> (ø)` | |
   | 
[...manager/realtime/HLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh)
 | `78.35% <0%> (-2.58%)` | `0% <0%> (ø)` | |
   | 
[...e/impl/dictionary/LongOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09uSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=)
 | `86.66% <0%> (-2.23%)` | `0% <0%> (ø)` | |
   | 
[...time/converter/stats/RealtimeColumnStatistics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9jb252ZXJ0ZXIvc3RhdHMvUmVhbHRpbWVDb2x1bW5TdGF0aXN0aWNzLmphdmE=)
 | `47.22% <0%> (-2.06%)` | `0% <0%> (ø)` | |
   | ... and [46 
more](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree-more)
 | |
   
   --
   
   [Continue to review full rep

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268329156
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
 
 Review comment:
   @sunithabeeram Let me know if this commit fulfills what you are asking for - 
   
   
https://github.com/apache/incubator-pinot/pull/3975/commits/599bc90a29812e6ddeef7fca3f574519287b3ec5?short_path=192dc63#diff-192dc63b00c8f5f79817d3a1f931fdfd


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268325138
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
 
 Review comment:
   Reformatted and fixed. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268323366
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
 
 Review comment:
   Updated again


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268323095
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
 
 Review comment:
   xajiang [12:48 PM]
   You can check `TableConfig.validate()` method
   
   ```
   public void validate() {
   Preconditions.checkState(_tableName != null, "Table name is missing");
   Preconditions.checkState(_tableType != null, "Table type is missing");
   Preconditions.checkState(_validationConfig != null, "Validation config 
is missing");
   Preconditions.checkState(_tenantConfig != null, "Tenant config is 
missing");
   Preconditions.checkState(_indexingConfig != null, "Indexing config is 
missing");
   Preconditions.checkState(_customConfig != null, "Custom config is 
missing");
 }
   ```
   
   xajiang [12:51 PM]
   ```
 public static final String TABLE_NAME_KEY = "tableName";
 public static final String TABLE_TYPE_KEY = "tableType";
 public static final String VALIDATION_CONFIG_KEY = "segmentsConfig";
 public static final String TENANT_CONFIG_KEY = "tenants";
 public static final String INDEXING_CONFIG_KEY = "tableIndexConfig";
 public static final String CUSTOM_CONFIG_KEY = "metadata";
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268323095
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
 
 Review comment:
   xajiang [12:48 PM]
   You can check `TableConfig.validate()` method
   
 ```public void validate() {
   Preconditions.checkState(_tableName != null, "Table name is missing");
   Preconditions.checkState(_tableType != null, "Table type is missing");
   Preconditions.checkState(_validationConfig != null, "Validation config 
is missing");
   Preconditions.checkState(_tenantConfig != null, "Tenant config is 
missing");
   Preconditions.checkState(_indexingConfig != null, "Indexing config is 
missing");
   Preconditions.checkState(_customConfig != null, "Custom config is 
missing");
 }```
   
   xajiang [12:51 PM]
 ```public static final String TABLE_NAME_KEY = "tableName";
 public static final String TABLE_TYPE_KEY = "tableType";
 public static final String VALIDATION_CONFIG_KEY = "segmentsConfig";
 public static final String TENANT_CONFIG_KEY = "tenants";
 public static final String INDEXING_CONFIG_KEY = "tableIndexConfig";
 public static final String CUSTOM_CONFIG_KEY = "metadata";```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268322957
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
 
 Review comment:
   I've removed this sentence and reformatted the section. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268315366
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,32 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
+
+* Missing Segments - Number of missing segments - `NUM_MISSING_SEGMENTS 
`_
+* Query latency - Latency from the time a server receives a request to when it 
sends a response - `TOTAL_QUERY_TIME 
`_
+* Query Execution Exceptions - The number of exception which might have 
occurred during query execution - `QUERY_EXECUTION_EXCEPTIONS 
`_
+* Realtime Consumption Status - It's important to ensure at least a single 
replica of each partition is consuming - `LLC_PARTITION_CONSUMING 
`_
+* Realtime Highest Offset Consumed - `HIGHEST_STREAM_OFFSET_CONSUMED 
`_
+
+Pinot Broker
+
+* Incoming QPS (per broker) - `QUERIES 
`_
+* Dropped Requests - `REQUEST_DROPPED_DUE_TO_SEND_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_CONNECTION_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_ACCESS_ERROR 
`_
+* Partial Responses - `BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED 
`_
+* Table QPS quota exceeded - `QUERY_QUOTA_EXCEEDED 
`_
+* Table QPS quota usage percent - `QUERY_QUOTA_CAPACITY_UTILIZATION_RATE 
`_
+
+Pinot Controller
+
+* Missing Segment Count -
+* Segments in Error State -
 
 Review comment:
   I think this is outside of the scope of this section but I've noted it down 
and will file a jira ticket for this information to be added in a section later.
   
   I will leave this unresolved until we can get a 3rd opinion on whether to 
include this here or not. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268309823
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
 
 Review comment:
   Up to you. I'd try to present a consistent format throughout your 
documentation though.
   
   For example, rather than giving the paragraph style, you could include some 
information in your description of the possible options for `tableType` below.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268308656
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,61 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+Pinot exposes several metrics to monitor the service and ensure that pinot 
users are not experiencing issues. In this section we discuss some of the key 
metrics that are useful to monitor. A full list of metrics is available in the 
`Metrics `_ section.
+
+Pinot Server
+
+
+* Missing Segments - `NUM_MISSING_SEGMENTS 
`_
+
+  * Number of missing segments that the broker queried for (expected to be on 
the server) but the server didn't have. This can be due to retention or stale 
routing table.
+
+* Query latency - `TOTAL_QUERY_TIME 
`_
+
+  * The number of exception which might have occurred during query execution
+
+* Query Execution Exceptions - `QUERY_EXECUTION_EXCEPTIONS 
`_
+
+  * The number of exception which might have occurred during query execution
+
+* Realtime Consumption Status - `LLC_PARTITION_CONSUMING 
`_
+
+  * This gives a binary value based on whether low-level consumption is 
healthy (1) or unhealthy (0). It's important to ensure at least a single 
replica of each partition is consuming
+
+* Realtime Highest Offset Consumed - `HIGHEST_STREAM_OFFSET_CONSUMED 
`_
+
+  * The highest offset which has been consumed so far.
+
+Pinot Broker
+
+
+* Incoming QPS (per broker) - `QUERIES 
`_
+
+  * The rate which an individual broker is receiving queries. Units are in QPS.
+
+* Dropped Requests - `REQUEST_DROPPED_DUE_TO_SEND_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_CONNECTION_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_ACCESS_ERROR 
`_
+
+  * These multiple metrics will indicate if a query is dropped, ie the 
processing of that query has been forfeited for some reason.
+
+* Partial Responses - `BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED 
`_
+
+  * Indicates a count of partial responses. A partial response is when at 
least 1 of the requested servers fails to respond to the query.
+
+* Table QPS quota exceeded - `QUERY_QUOTA_EXCEEDED 
`_
+
+  * Binary metric which will indicate when the configured QPS quota for a 
table is exceeded (1) or if there is capacity remaining (0).
+
+* Table QPS quota usage percent - `QUERY_QUOTA_CAPACITY_UTILIZATION_RATE 
`_
+
+  * Percentage of the configured QPS quota being utilized.
+
+Pinot Controller
 
 Review comment:
   I am not sure how these controller metrics are created. I need someone to 
point me to that place in the code so that I can link to it from here. 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268305273
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
 
 Review comment:
   Sunitha has pointed me to this PR by @Jackie-Jiang 
   https://github.com/apache/incubator-pinot/pull/3993
   
   I am waiting for a response from Jackie to help me parse what exactly is 
required. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268304983
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
 
 Review comment:
   Here's some options that come to mind at the moment
   
   1. Keep the sentences I have and link to the section I mentioned above. 
   2. Remove my descriptions here and give a succinct 1-2 sentence introduction 
to what an offline table and what a realtime table is in the section I 
mentioned above and then simply get straight to business here with a link to 
the detailed section. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268304120
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
 
 Review comment:
   One concern I have here is that I don't want to go straight into 
offline/realtime tables without giving some preface of what we are discussing. 
   
   My thought is if I start it such as the following, I would expect a common 
question/thought to be "what is offline/realtime and how are they different"?
   
   > In this section we will describe how to configure offline and realtime 
tables. 
   
   There is a section in Introduction > Architecture > Pinot Tables, but this 
is a long description and not very succinct. I simply wanted to give a 1-2 
sentence intro to set the stage for what's be discussed. 
   
   I am not sure which route to go here to be honest.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268302113
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
 
 Review comment:
   I've confirmed this is only for realtime and updated the description. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268264550
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
 
 Review comment:
   Are these the only mandatory sections? I don't know what is mandatory or not 
in this case since we generally use all of them. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table 
Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#issuecomment-473083001
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=h1) 
Report
   > Merging 
[#3975](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/5867b16f32145a80fa98de4d24ec6302b17bb95e?src=pr&el=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/3975/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#3975  +/-   ##
   
   - Coverage 66.99%   66.96%   -0.03% 
   - Complexity4   17  +13 
   
 Files  1032 1033   +1 
 Lines 5104751244 +197 
 Branches   7126 7175  +49 
   
   + Hits  3420034317 +117 
   - Misses1450214544  +42 
   - Partials   2345 2383  +38
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr&el=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==)
 | `75% <0%> (-25%)` | `0% <0%> (ø)` | |
   | 
[...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=)
 | `43.33% <0%> (-11.67%)` | `0% <0%> (ø)` | |
   | 
[...esthandler/ConnectionPoolBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQ29ubmVjdGlvblBvb2xCcm9rZXJSZXF1ZXN0SGFuZGxlci5qYXZh)
 | `77.77% <0%> (-10.5%)` | `0% <0%> (ø)` | |
   | 
[...impl/dictionary/FloatOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPZmZIZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==)
 | `83.63% <0%> (-9.1%)` | `0% <0%> (ø)` | |
   | 
[...mpl/dictionary/DoubleOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT2ZmSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=)
 | `69.09% <0%> (-7.28%)` | `0% <0%> (ø)` | |
   | 
[...api/resources/PinotTableConfigRestletResource.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90VGFibGVDb25maWdSZXN0bGV0UmVzb3VyY2UuamF2YQ==)
 | `24.35% <0%> (-3.28%)` | `0% <0%> (ø)` | |
   | 
[...time/converter/stats/RealtimeColumnStatistics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9jb252ZXJ0ZXIvc3RhdHMvUmVhbHRpbWVDb2x1bW5TdGF0aXN0aWNzLmphdmE=)
 | `47.22% <0%> (-2.06%)` | `0% <0%> (ø)` | |
   | 
[.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh)
 | `90.9% <0%> (-1.82%)` | `0% <0%> (ø)` | |
   | 
[...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=)
 | `47.65% <0%> (-1.44%)` | `0% <0%> (ø)` | |
   | 
[...g/apache/pinot/common/response/ServerInstance.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzcG9uc2UvU2VydmVySW5zdGFuY2UuamF2YQ==)
 | `78.37% <0%> (-1.36%)` | `0% <0%> (ø)` | |
   | ... and [39 
more](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr&el=tree-more)
 | |
   
   

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268275802
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
 
 Review comment:
   Updated. Please review the new commit. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268275661
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
 
 Review comment:
   I've updated. Perhaps we don't need the sentence? I hear people refer to it 
in various ways so I wanted to be clear, but I am open to removing it if you 
think we don't need it. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268274286
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
+
+  * Allowed values - String
+
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+
+  * Allowed values - String, this must match the ``timeFieldSpec`` section in 
the schema
+
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "myPinotSchmea",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* ``invertedIndexColumns`` - Indicates a list of real column names as 
specified in the schema to create inv

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268273033
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,63 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
 
 Review comment:
   Re: monitoring wording
   
   I've removed my 2 paragraphs and replaced with your quoted text. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268272113
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
 
 Review comment:
   Also added 
   
   > Segment retention is ignored when set to ``REFRESH``.
   
   In a latter section. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268271486
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
 
 Review comment:
   Added 
   
   > Retention is only applicable to tables of type ``APPEND``


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268271020
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -40,11 +40,14 @@ Recommended environment
 Deploying Pinot
 ---
 
-Direct deployment of Pinot
-~~
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended incase of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
 Review comment:
   incase -> in case


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268270607
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
+
+  * Allowed values - String
+
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+
+  * Allowed values - String, this must match the ``timeFieldSpec`` section in 
the schema
+
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "myPinotSchmea",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* ``invertedIndexColumns`` - Indicates a list of real column names as 
specified in the schema to create inv

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268270425
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
+
+  * Allowed values - String
+
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+
+  * Allowed values - String, this must match the ``timeFieldSpec`` section in 
the schema
+
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "myPinotSchmea",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* ``invertedIndexColumns`` - Indicates a list of real column names as 
specified in the schema to create inv

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268269455
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
 
 Review comment:
   Removed. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268269066
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
 
 Review comment:
   Copy paste error, thanks for catching. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268266909
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -40,11 +40,14 @@ Recommended environment
 Deploying Pinot
 ---
 
-Direct deployment of Pinot
-~~
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended incase of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
 Review comment:
   I don't think that is the scope of this section. The way this was presented 
to me is that I need to describe how to deploy Pinot in terms of a version 
upgrade. 
   
   Everything you are describing sounds like it should be part of an 
architecture section for how to implement Pinot


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268266391
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,63 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
 
 Review comment:
   The empty section above are not assigned to me. I just filled in the outline 
as it was in the repo. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: Added documentation regarding design docs (#4002)

[mcvsubbu]

This is an automated email from the ASF dual-hosted git repository.

mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new ea2d621  Added documentation regarding design docs (#4002)
ea2d621 is described below

commit ea2d6215421cf422a1740bac97fdf44e001439b9
Author: Subbu Subramaniam 
AuthorDate: Fri Mar 22 10:18:19 2019 -0700

Added documentation regarding design docs (#4002)

Added documentation pointing to design docs.
Also fixed some warnings thrown during compilation
---
 docs/code_modules.rst|  5 +--
 docs/contribution_guidelines.rst | 75 
 docs/pinot_hadoop.rst|  3 +-
 3 files changed, 51 insertions(+), 32 deletions(-)

diff --git a/docs/code_modules.rst b/docs/code_modules.rst
index 33e5a20..cf439cb 100644
--- a/docs/code_modules.rst
+++ b/docs/code_modules.rst
@@ -141,13 +141,14 @@ Auxiliary modules
 In addition to the core modules described above, Pinot code provides the 
following modules:
 
 * ``pinot-tools``: This module is a collection of many tools useful for 
setting up Pinot cluster, creating/updating segments.
-It also houses the Pinot quick start guide code.
+   It also houses the Pinot quick start guide code.
 
 * ``pinot-perf``: This module has a collection of benchmark test code used to 
evaluate design options.
 
 * ``pinot-client-api``: This module houses the Java client API. See 
:ref:`java-client` for more info.
 
 * ``pinot-integration-tests``: This module holds integration tests that test 
functionality across multiple classes or components.
+
 These tests typically do not rely on mocking and provide more end to end 
coverage for code.
 
 .. _extension-modules:
@@ -157,4 +158,4 @@ Extension modules
 ``pinot-hadoop-filesystem`` and ``pinot-azure-filesystem`` are module added to 
support extensions to Pinot filesystem.
 The functionality is broken down into modules of their own to avoid polluting 
the common modules with additional large libraries.
 These libraries bring in transitive dependencies of their own that can cause 
classpath conflicts at runtime. We would like to
-avoid this for the common usage of Pinot as much as possible.
\ No newline at end of file
+avoid this for the common usage of Pinot as much as possible.
diff --git a/docs/contribution_guidelines.rst b/docs/contribution_guidelines.rst
index e980158..f4a4b47 100644
--- a/docs/contribution_guidelines.rst
+++ b/docs/contribution_guidelines.rst
@@ -25,12 +25,26 @@ Contribution Guidelines
 
 Before you begin to contribute, make sure you have reviewed :ref:`dev-setup` 
and :ref:`code-modules` sections and that you have created your own fork of the 
pinot source code.
 
+Create a design document
+
+If your change is relatively minor, you can skip this step. If you are
+adding new major feature, we suggest that you add a design document 
+and solicit comments from the community before submitting any code.
+
+`Here `_
+is a list of current design documents.
+
+
 Create an issue for the change
 --
 Create a Pinot issue for the change you would like to make. Provide 
information on why the change is needed and how you
 plan to address it. Use the conversations on the issue as a way to validate 
assumptions and the right way to proceed.
 Be sure to review sections on :ref:`compatibility-changes` and 
:ref:`external-libs`.
 
+If you have a design document, please refer to the design documents
+in your Issue. You may even want to create multiple
+issues depending on the extent of your change.
+
 Once you are clear about what you want to do, proceed with the next steps 
listed below.
 
 Create a branch for your change
@@ -61,8 +75,8 @@ Code documentation
 Please ensure your code is adequately documented. Some things to consider for 
documentation:
 
 * Always include class level java docs.
-At the top class level, we are looking for information about what 
functionality is provided by the class,
-what state is maintained by the class, whether there are 
concurrency/thread-safety concerns and any exceptional behavior that the class 
might exhibit.
+  At the top class level, we are looking for information about what 
functionality is provided by the class,
+  what state is maintained by the class, whether there are 
concurrency/thread-safety concerns and any exceptional behavior that the class 
might exhibit.
 
 * Document public methods and their parameters.
 
@@ -101,8 +115,8 @@ Be cautious about pulling in external dependencies. You 
will need to consider mu
 * Is the external library maintained by an active community of contributors?
 * What are the licensing terms for the library. For more information about 
handling licenses, see :ref:`handling-licenses`.
 * Are you adding the library to :r

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268265312
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
 
 Review comment:
   The intention here is that there was supposed to be a section for Schema so 
it would appear as 
   
   Table Config & Schema
   
   - Table Config
   - Schema
   
   I was waiting to hear back from Shraddha, but she wants to push off the 
Schema section


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu merged pull request #4002: Added documentation regarding design docs

[GitBox]

mcvsubbu merged pull request #4002: Added documentation regarding design docs
URL: https://github.com/apache/incubator-pinot/pull/4002
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

jgutmann commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268264550
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
 
 Review comment:
   Are these the only mandatory sections? I don't know what is mandatory or not 
in this case since generally use all of them. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: Set processingException when all queried segments cannot be acquired (#3942)

[sunithabeeram]

This is an automated email from the ASF dual-hosted git repository.

sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new d4f2ece  Set processingException when all queried segments cannot be 
acquired (#3942)
d4f2ece is described below

commit d4f2ecef660ab1d4efa9696a53b0623aac867c3f
Author: Sunitha Beeram 
AuthorDate: Fri Mar 22 09:33:09 2019 -0700

Set processingException when all queried segments cannot be acquired (#3942)

* Set processingException when all queried segments cannot be acquired

* Minor fixes

* Address review comments

* Disable setting processing exception until metrics are validated

* Address review comments

* Address review comments

* Address review comments

* Address review comments

* Address review comments: Make InstanceDataManager symmetric with respect 
to tracking segments added and deleted
---
 .../pinot/common/exception/QueryException.java |  2 +
 .../core/data/manager/BaseTableDataManager.java| 48 +
 .../core/data/manager/InstanceDataManager.java | 14 ++
 .../pinot/core/data/manager/TableDataManager.java  | 15 +++
 .../query/executor/ServerQueryExecutorV1Impl.java  | 33 +-
 .../data/manager/BaseTableDataManagerTest.java |  5 +++
 .../pinot/query/executor/QueryExecutorTest.java| 50 --
 .../starter/helix/HelixInstanceDataManager.java| 16 +++
 .../SegmentOnlineOfflineStateModelFactory.java |  5 +++
 9 files changed, 175 insertions(+), 13 deletions(-)

diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
index e0ed903..6c45897 100644
--- 
a/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
@@ -41,6 +41,7 @@ public class QueryException {
   public static final int SEGMENT_PLAN_EXECUTION_ERROR_CODE = 160;
   public static final int COMBINE_SEGMENT_PLAN_TIMEOUT_ERROR_CODE = 170;
   public static final int ACCESS_DENIED_ERROR_CODE = 180;
+  public static final int SEGMENTS_MISSING_ERROR_CODE = 190;
   public static final int QUERY_EXECUTION_ERROR_CODE = 200;
   // TODO: Handle these errors in broker
   public static final int SERVER_SHUTTING_DOWN_ERROR_CODE = 210;
@@ -96,6 +97,7 @@ public class QueryException {
   public static final ProcessingException QUERY_VALIDATION_ERROR = new 
ProcessingException(QUERY_VALIDATION_ERROR_CODE);
   public static final ProcessingException UNKNOWN_ERROR = new 
ProcessingException(UNKNOWN_ERROR_CODE);
   public static final ProcessingException QUOTA_EXCEEDED_ERROR = new 
ProcessingException(TOO_MANY_REQUESTS_ERROR_CODE);
+  public static final ProcessingException SEGMENTS_MISSING_ERROR = new 
ProcessingException(SEGMENTS_MISSING_ERROR_CODE);
 
   static {
 JSON_PARSING_ERROR.setMessage("JsonParsingError");
diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
index a456691..3f074a1 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
@@ -19,10 +19,13 @@
 package org.apache.pinot.core.data.manager;
 
 import com.google.common.base.Preconditions;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
 import java.io.File;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
 import javax.annotation.Nonnull;
 import javax.annotation.concurrent.ThreadSafe;
 import org.apache.helix.ZNRecord;
@@ -42,9 +45,13 @@ import org.slf4j.LoggerFactory;
 @ThreadSafe
 public abstract class BaseTableDataManager implements TableDataManager {
   private static final Logger LOGGER = 
LoggerFactory.getLogger(BaseTableDataManager.class);
+  // cache deleted segment names for utmost this duration
+  private static final int MAX_CACHE_DURATION_SEC = 6 * 3600; // 6 hours
 
   protected final ConcurrentHashMap 
_segmentDataManagerMap = new ConcurrentHashMap<>();
 
+  protected Cache _deletedSegmentsCache;
+
   protected TableDataManagerConfig _tableDataManagerConfig;
   protected String _instanceId;
   protected ZkHelixPropertyStore _propertyStore;
@@ -59,6 +66,7 @@ public abstract class BaseTableDataManager implements 
TableDataManager {
   @Nonnull ZkHelixPropertyStore propertyStore, @Nonnull 
ServerMetrics serverMetrics) {
 LOGGER.info("Initializing table data manager for table: {}", 
tableDataManagerConfig.getTableName());

[GitHub] [incubator-pinot] sunithabeeram merged pull request #3942: Set processingException when all queried segments cannot be acquired

[GitBox]

sunithabeeram merged pull request #3942: Set processingException when all 
queried segments cannot be acquired
URL: https://github.com/apache/incubator-pinot/pull/3942
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] sunithabeeram commented on issue #4007: Improve/Add completeness and freshness metrics for Pinot

[GitBox]

sunithabeeram commented on issue #4007: Improve/Add completeness and freshness 
metrics for Pinot
URL: 
https://github.com/apache/incubator-pinot/issues/4007#issuecomment-475688738
 
 
   https://github.com/apache/incubator-pinot/pull/3942 tracks Completeness (a) 
part of the issue. Note the following caveats:
   - There is a legitimate case where the servers may not have segments 
available - this happens during retention. We don't consider to be a serving 
error. The change makes a best effort attempt to track recently deleted 
segments using an in-memory cache. This does mean that on server restarts the 
segments retained out segments can be marked as missing.
   - When segments move or get refreshed with explicit state transitions (ie, 
an online->offline transition is triggered followed by a offline->online 
transition), and the query reaches a server after the online->offline 
transition), the server won't have the segment and we will count them as 
partial responses.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] sunithabeeram opened a new issue #4007: Improve/Add completeness and freshness metrics for Pinot

[GitBox]

sunithabeeram opened a new issue #4007: Improve/Add completeness and freshness 
metrics for Pinot
URL: https://github.com/apache/incubator-pinot/issues/4007
 
 
   This issue tracks the need to expose better metrics in Pinot to expose 
completeness and freshness in the query path. Today, a lot of individual 
metrics are available for tracking, but its typically hard to say which queries 
were actually affected when issues do happen. Allowing tracking/metadata to be 
sent in query response helps with this. This issue tracks efforts being made 
for this. 
   NOTE: the tracking is on a best effort basis - we would like to track the 
most common possibilities (without tying ourselves to specific individual 
scenarios) without compromising the execution time of the query and continuing 
to keep the design/implementation as simple/intuitive as possible. 
   
   Details:
   ==
   Completeness:
   ---
   Currently, we set a "partial-response" metadata flag when a broker is unable 
to get a connection to the server or response from a server within configured 
timeout. We would like to expand this to cover:
   (a) cases where servers don't have the segments the broker is requesting.
   (b) broker's routing table is incomplete - for ex, when all the replicas of 
a given segment are ofline or in error state.
   
   Freshness:
   ---
   In the current scope, we limit the freshness of the data to realtime tables. 
We would like to measure how "recent" the data available for querying in 
realtime is. 
   
   More details for each of these will follow.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] xiaohui-sun merged pull request #4004: [TE] frontend - harleyjj/yaml-editor - increases toast timeout from 4…

[GitBox]

xiaohui-sun merged pull request #4004: [TE] frontend - harleyjj/yaml-editor - 
increases toast timeout from 4…
URL: https://github.com/apache/incubator-pinot/pull/4004
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: [TE] frontend - harleyjj/yaml-editor - increases toast timeout from 4 seconds to 10 seconds (#4004)

[xhsun]

This is an automated email from the ASF dual-hosted git repository.

xhsun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 4b31ff6  [TE] frontend - harleyjj/yaml-editor - increases toast 
timeout from 4 seconds to 10 seconds (#4004)
4b31ff6 is described below

commit 4b31ff6e7fe09f69eb4b2e39f49009c191da65ea
Author: Harley Jackson 
AuthorDate: Fri Mar 22 07:36:36 2019 -0700

[TE] frontend - harleyjj/yaml-editor - increases toast timeout from 4 
seconds to 10 seconds (#4004)
---
 .../app/pods/components/alert-details/component.js  |  4 ++--
 .../app/pods/components/yaml-editor/component.js| 21 +++--
 .../app/pods/manage/explore/route.js| 12 +++-
 .../thirdeye-frontend/app/pods/manage/yaml/route.js |  9 +
 thirdeye/thirdeye-frontend/app/utils/constants.js   | 18 --
 5 files changed, 37 insertions(+), 27 deletions(-)

diff --git 
a/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js 
b/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
index 37433f4..978eb61 100644
--- a/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
+++ b/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
@@ -17,6 +17,7 @@ import Component from '@ember/component';
 import { computed, observer, set, get, getProperties } from '@ember/object';
 import { later } from '@ember/runloop';
 import { checkStatus, humanizeFloat, postProps } from 
'thirdeye-frontend/utils/utils';
+import { toastOptions } from 'thirdeye-frontend/utils/constants';
 import { colorMapping, toColor, makeTime, toMetricLabel, extractTail } from 
'thirdeye-frontend/utils/rca-utils';
 import { getYamlPreviewAnomalies,
   getAnomaliesByAlertId,
@@ -496,7 +497,6 @@ export default Component.extend({
   isPreviewMode,
   alertId
 } = this.getProperties('analysisRange', 'notifications', 'isPreviewMode', 
'alertId');
-
 //detection alert fetch
 const start = analysisRange[0];
 const end = analysisRange[1];
@@ -547,7 +547,7 @@ export default Component.extend({
 });
   }
 } catch (error) {
-  notifications.error(error.body.message);
+  notifications.error(error.body.message, toastOptions);
 }
 
 return {
diff --git 
a/thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/component.js 
b/thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/component.js
index 7f39068..1785d45 100644
--- a/thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/component.js
+++ b/thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/component.js
@@ -24,7 +24,7 @@
 import Component from '@ember/component';
 import {computed, set, get, getProperties, setProperties} from '@ember/object';
 import {checkStatus} from 'thirdeye-frontend/utils/utils';
-import {yamlAlertProps, yamlAlertSettings} from 
'thirdeye-frontend/utils/constants';
+import {yamlAlertProps, yamlAlertSettings, toastOptions} from 
'thirdeye-frontend/utils/constants';
 import yamljs from 'yamljs';
 import RSVP from "rsvp";
 import fetch from 'fetch';
@@ -141,7 +141,7 @@ export default Component.extend({
   const json = yield response.json();
   return json.filterBy('yaml');
 } catch (error) {
-  notifications.error('Failed to retrieve subscription groups.', 'Error');
+  notifications.error('Failed to retrieve subscription groups.', 'Error', 
toastOptions);
 }
   }).drop(),
 
@@ -404,12 +404,12 @@ export default Component.extend({
   set(this, 'subscriptionMsg', result.subscriptionMsg);
 }
 if (result.detectionAlertConfigId && result.detectionConfigId) {
-  notifications.success('Created alert successfully.', 'Created');
+  notifications.success('Created alert successfully.', 'Created', 
toastOptions);
 }
   }
 });
   }).catch((error) => {
-notifications.error('Create alert failed.', error);
+notifications.error('Create alert failed.', error, toastOptions);
   });
 },
 
@@ -425,6 +425,7 @@ export default Component.extend({
 alertId,
 subscriptionGroupId
   } = getProperties(this, 'detectionYaml', 'subscriptionYaml', 
'notifications', 'alertId', 'subscriptionGroupId');
+
   //PUT alert
   const alert_url = `/yaml/${alertId}`;
   const alertPostProps = {
@@ -438,12 +439,12 @@ export default Component.extend({
 const alert_json = await alert_result.json();
 if (alert_status !== 200) {
   set(this, 'errorMsg', get(alert_json, 'message'));
-  notifications.error('Failed to save the detection configuration.', 
'Error');
+  notifications.error('Failed to save the detection configuration.', 
'Error', toastOptions);
 } else {
-  notifications.success('Detection c

[GitHub] [incubator-pinot] ly923976094 opened a new pull request #4006: Controller web cannot create and update schemas

[GitBox]

ly923976094 opened a new pull request #4006: Controller web cannot create and 
update schemas 
URL: https://github.com/apache/incubator-pinot/pull/4006
 
 
   Controller web cannot create and update schemas in /schemas and 
/schemas/{schemaName}，Because these two interfaces need to upload a schema file 
to create and update the schema, but the web can only pass json strings, so 
they are deprecated and added services that can pass json strings to create and 
update the schema(/schemas and /schemas/v2/{schemaName}).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org