[GitHub] [incubator-druid] gocho1 commented on a change in pull request #7674: add s3 authentication method informations
gocho1 commented on a change in pull request #7674: add s3 authentication method informations URL: https://github.com/apache/incubator-druid/pull/7674#discussion_r285448467 ## File path: docs/content/development/extensions-core/s3.md ## @@ -65,6 +65,23 @@ As an example, to set the region to 'us-east-1' through system properties: `s3:GetObject` and `s3:PutObject` are basically required for pushing/loading segments to/from S3. If `druid.storage.disableAcl` is set to `false`, then `s3:GetBucketAcl` and `s3:PutObjectAcl` are additionally required to set ACL for objects. +### S3 authentication methods + +To connect to your S3 bucket (whether deep storage bucket or source bucket), Druid use the following credentials providers chain + +|order|type|details| +||---|---| +|1|Druid config file|Based on your runtime.properties if it contains values `druid.s3.accessKey` and `druid.s3.secretKey` | +|2|Custom properties file| Based on custom properties file where you can supply `sessionToken`, `accessKey` and `secretKey` values| Review comment: done. I also reported properties that were on one side but not on other (when relevant). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis opened a new pull request #7700: update lz4-java to 1.6.0 (lz4 1.9.1)
clintropolis opened a new pull request #7700: update lz4-java to 1.6.0 (lz4 1.9.1) URL: https://github.com/apache/incubator-druid/pull/7700 There are potential [decompression speed increases added in lz4 1.9.0](https://github.com/lz4/lz4/releases/tag/v1.9.0) [lz4-java changelog](https://github.com/lz4/lz4-java/blob/master/CHANGES.md#160) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis merged pull request #7682: Add checkstyle for "Prohibit @author tags in Javadoc"
clintropolis merged pull request #7682: Add checkstyle for "Prohibit @author tags in Javadoc" URL: https://github.com/apache/incubator-druid/pull/7682 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Add checkstyle for "Prohibit @author tags in Javadoc" (#7682)
This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 5f08b0b Add checkstyle for "Prohibit @author tags in Javadoc" (#7682)
5f08b0b is described below
commit 5f08b0b474a9d811f2a3b419d72fd3c6df157c9e
Author: Merlin Lee <39320777+lml2...@users.noreply.github.com>
AuthorDate: Mon May 20 15:09:51 2019 +0800
Add checkstyle for "Prohibit @author tags in Javadoc" (#7682)
* Add checkstyle for "Prohibit @author tags in Javadoc"
* Add "Do not use author tags/information in the code" back to
CONTRIBUTING.md
---
codestyle/checkstyle.xml | 9 +
.../main/java/org/apache/druid/java/util/common/io/Closer.java | 1 -
.../druid/examples/twitter/TwitterSpritzerFirehoseFactory.java | 2 --
.../java/org/apache/druid/extendedset/intset/AbstractIntSet.java | 1 -
.../java/org/apache/druid/extendedset/intset/ConciseSet.java | 4
.../main/java/org/apache/druid/extendedset/intset/IntSet.java| 1 -
.../druid/storage/cassandra/CassandraDataSegmentConfig.java | 2 --
.../druid/storage/cassandra/CassandraDataSegmentPusher.java | 2 --
.../aggregation/datasketches/hll/HllSketchAggregatorFactory.java | 1 -
.../aggregation/datasketches/hll/HllSketchBuildAggregator.java | 1 -
.../datasketches/hll/HllSketchBuildAggregatorFactory.java| 1 -
.../datasketches/hll/HllSketchBuildBufferAggregator.java | 1 -
.../aggregation/datasketches/hll/HllSketchMergeAggregator.java | 1 -
.../datasketches/hll/HllSketchMergeAggregatorFactory.java| 1 -
.../datasketches/hll/HllSketchMergeBufferAggregator.java | 1 -
.../query/aggregation/datasketches/hll/HllSketchModule.java | 1 -
.../hll/HllSketchToEstimateWithBoundsPostAggregator.java | 1 -
.../datasketches/hll/HllSketchToStringPostAggregator.java| 1 -
.../datasketches/hll/HllSketchUnionPostAggregator.java | 1 -
19 files changed, 9 insertions(+), 24 deletions(-)
diff --git a/codestyle/checkstyle.xml b/codestyle/checkstyle.xml
index 786b182..12e9532 100644
--- a/codestyle/checkstyle.xml
+++ b/codestyle/checkstyle.xml
@@ -32,6 +32,15 @@
+
+
+
+
+
+
+
+
+
diff --git
a/core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
b/core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
index 5bba0db..f5bfb2c 100644
--- a/core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
+++ b/core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
@@ -85,7 +85,6 @@ import java.util.Deque;
* careful to ensure that Closer is not used in code that might be executed by
Hadoop indexing service. We copy Closer
* to Druid codebase in order to remove this headache.
*
- * @author Colin Decker
* @since Guava 14.0
*/
// Coffee's for {@link Closer closers} only.
diff --git
a/examples/src/main/java/org/apache/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java
b/examples/src/main/java/org/apache/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java
index dc951e8..65bdacd 100644
---
a/examples/src/main/java/org/apache/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java
+++
b/examples/src/main/java/org/apache/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java
@@ -78,8 +78,6 @@ import java.util.regex.Pattern;
* can have number format exceptions), so it might be necessary to extract raw
json and process it
* separately. If so, set twitter4.jsonStoreEnabled=true and look at
DataObjectFactory#getRawJSON();
* com.fasterxml.jackson.databind.ObjectMapper should be used to parse.
- *
- * @author pbaclace
*/
@JsonTypeName("twitzer")
public class TwitterSpritzerFirehoseFactory implements
FirehoseFactory
diff --git
a/extendedset/src/main/java/org/apache/druid/extendedset/intset/AbstractIntSet.java
b/extendedset/src/main/java/org/apache/druid/extendedset/intset/AbstractIntSet.java
index aecee12..0f4ef59 100755
---
a/extendedset/src/main/java/org/apache/druid/extendedset/intset/AbstractIntSet.java
+++
b/extendedset/src/main/java/org/apache/druid/extendedset/intset/AbstractIntSet.java
@@ -23,7 +23,6 @@ package org.apache.druid.extendedset.intset;
* This class provides a skeletal implementation of the {@link IntSet}
* interface to minimize the effort required to implement this interface.
*
- * @author Alessandro Colantonio
* @version $Id: AbstractIntSet.java 156 2011-09-01 00:13:57Z cocciasik $
*/
public abstract class AbstractIntSet implements IntSet
diff --git
a/extendedset/src/main/java/org/apache/druid/extendedset/intset/ConciseSet.java
b/extendedset/src/main/java/org/apache/druid/extendedset/intset/ConciseSet.java
index a7db53b..5bce561 100755
---
a/extendedset/src/main/java/org/apache/druid/extended
[GitHub] [incubator-druid] clintropolis closed issue #7384: Prohibit @author tags in Javadocs
clintropolis closed issue #7384: Prohibit @author tags in Javadocs URL: https://github.com/apache/incubator-druid/issues/7384 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #7692: Fix case insensitive of ParserUtils.findDuplicates
clintropolis commented on a change in pull request #7692: Fix case insensitive of ParserUtils.findDuplicates URL: https://github.com/apache/incubator-druid/pull/7692#discussion_r285451454 ## File path: core/src/main/java/org/apache/druid/data/input/impl/DimensionsSpec.java ## @@ -199,9 +199,6 @@ private void verify(List spatialDimensions) "dimensions and dimensions exclusions cannot overlap" ); -ParserUtils.validateFields(dimNames); Review comment: Oh, i see, my bad :+1: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pzhdfy commented on issue #7582: [for 0.12.x] general exactly count distinct support in Druid
pzhdfy commented on issue #7582: [for 0.12.x] general exactly count distinct support in Druid URL: https://github.com/apache/incubator-druid/pull/7582#issuecomment-493971567 > Hi @pzhdfy thanks for the proposal and PR! For new features like this, we generally only will merge into the master branch, so I think we should close this PR to focus on #7594. I think so. I will close this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pzhdfy closed pull request #7582: [for 0.12.x] general exactly count distinct support in Druid
pzhdfy closed pull request #7582: [for 0.12.x] general exactly count distinct support in Druid URL: https://github.com/apache/incubator-druid/pull/7582 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] b-slim commented on issue #7676: Add OakIncrementalIndex to Druid
b-slim commented on issue #7676: Add OakIncrementalIndex to Druid URL: https://github.com/apache/incubator-druid/pull/7676#issuecomment-494027530 @sanastas that is simple not true, some of those locks are part of `allocate` [calls](https://github.com/yahoo/Oak/blob/master/core/src/main/java/com/oath/oak/NativeAllocator/OakNativeMemoryAllocator.java#L94-L100 ) and some are part of[ iterator loop](https://github.com/yahoo/Oak/blob/master/core/src/main/java/com/oath/oak/InternalOakMap.java#L272-L278) to do some stuff that i doubt it is static ... Looking at the allocator briefly i see bunch of red flags, like possible internal fragmentations up to factor 2 and not clear way to deal with external fragmentations ... Memory Management is a very hard thing in general and make it lock free is not going to be simple task Writing one form scratch is a common source of bugs, In fact most of the big data projects avoid this task and use the Netty allocator. The take our of this, current state is not lock free thus lets avoid confusion, (FYI even on the oak github page i do not see it called lock-free so not sure why you are calling it lock free). But the most important thing to add here is how OKA memory management will fit with how Druid manages Direct memory. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko opened a new pull request #7701: Bump Jackson to 2.9.9
Fokko opened a new pull request #7701: Bump Jackson to 2.9.9 URL: https://github.com/apache/incubator-druid/pull/7701 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #7701: Bump Jackson to 2.9.9
Fokko commented on a change in pull request #7701: Bump Jackson to 2.9.9 URL: https://github.com/apache/incubator-druid/pull/7701#discussion_r285645861 ## File path: extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/LegacyKafkaIndexTaskRunner.java ## @@ -307,8 +307,8 @@ private TaskStatus runInternal(TaskToolbox toolbox) throws Exception final SeekableStreamEndSequenceNumbers restoredNextPartitions = toolbox.getObjectMapper().convertValue( restoredMetadataMap.get(METADATA_NEXT_PARTITIONS), toolbox.getObjectMapper().getTypeFactory().constructParametrizedType( Review comment: Curious how this ever worked. It is assigned to a `SeekableStreamEndSequenceNumbers`, but Jackson expects a `SeekableStreamStartSequenceNumbers`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #7701: Bump Jackson to 2.9.9
Fokko commented on a change in pull request #7701: Bump Jackson to 2.9.9
URL: https://github.com/apache/incubator-druid/pull/7701#discussion_r285646031
##
File path:
extensions-core/kafka-indexing-service/src/test/java/org/apache/druid/indexing/kafka/KafkaIOConfigTest.java
##
@@ -51,7 +50,7 @@
public KafkaIOConfigTest()
{
mapper = new DefaultObjectMapper();
-mapper.registerModules((Iterable) new
KafkaIndexTaskModule().getJacksonModules());
+mapper.registerModules(new KafkaIndexTaskModule().getJacksonModules());
Review comment:
Unnecessary cast
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #7701: Bump Jackson to 2.9.9
Fokko commented on a change in pull request #7701: Bump Jackson to 2.9.9 URL: https://github.com/apache/incubator-druid/pull/7701#discussion_r285646917 ## File path: pom.xml ## @@ -84,8 +84,7 @@ 4.1.0 9.4.10.v20180503 1.19.3 - Review comment: We're already at Avatica `1.12.0`, which is at Jackson 2.9.x: https://mvnrepository.com/artifact/org.apache.calcite.avatica/avatica-core/1.12.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov removed a comment on issue #7066: Support assign tasks to run on different tiers of MiddleManagers
leventov removed a comment on issue #7066: Support assign tasks to run on different tiers of MiddleManagers URL: https://github.com/apache/incubator-druid/pull/7066#issuecomment-494080578 @QiuMM "category" sounds good. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #7066: Support assign tasks to run on different tiers of MiddleManagers
leventov commented on issue #7066: Support assign tasks to run on different tiers of MiddleManagers URL: https://github.com/apache/incubator-druid/pull/7066#issuecomment-494080578 @QiuMM "category" sounds good. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #7066: Support assign tasks to run on different tiers of MiddleManagers
leventov commented on issue #7066: Support assign tasks to run on different tiers of MiddleManagers URL: https://github.com/apache/incubator-druid/pull/7066#issuecomment-494080518 @QiuMM "category" sounds good. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #7681: Add checkstyle for "Local variable names shouldn't start with capital"
leventov commented on a change in pull request #7681: Add checkstyle for "Local
variable names shouldn't start with capital"
URL: https://github.com/apache/incubator-druid/pull/7681#discussion_r285696714
##
File path:
core/src/main/java/org/apache/druid/java/util/common/StringUtils.java
##
@@ -40,6 +40,7 @@
public class StringUtils
{
public static final byte[] EMPTY_BYTES = new byte[0];
+ public static final String LINE_SEPARATOR = System.lineSeparator();
Review comment:
Since there is a direct method in `System` (rather than getProperty(...))
there is no point in storing it in a constant. Please use directly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #7061: Fix concurrency in old Druid classes
leventov commented on issue #7061: Fix concurrency in old Druid classes URL: https://github.com/apache/incubator-druid/issues/7061#issuecomment-494083865 @umasankar-yedida great! You can take any class that has signs of concurrency (`synchronized` keyword, use of `ConcurrentHashMap`, ExecutorServices, Threads, etc.) and review it according to the [checklist](https://github.com/code-review-checklists/java-concurrency). If the class doesn't have bugs, at least it lacks comments and concurrent access documentation: see items in the [Documentation](https://github.com/code-review-checklists/java-concurrency#documentation) section of the checklist. So literally every old class with concurrency in Druid can be improved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #7682: Add checkstyle for "Prohibit @author tags in Javadoc"
leventov commented on a change in pull request #7682: Add checkstyle for "Prohibit @author tags in Javadoc" URL: https://github.com/apache/incubator-druid/pull/7682#discussion_r285702756 ## File path: CONTRIBUTING.md ## @@ -33,7 +33,6 @@ When submitting a pull request (PR), please use the following guidelines: using "Feature/Change" template. Non-trivial features should include unit tests covering the new functionality. Open a "Proposal" issue for large changes. - Bugfixes should include a unit test or integration test reproducing the issue. -- Do not use author tags/information in the code. Review comment: I think nobody ever adds this in non-Java files. It appears when people have default Java class templates in IDEs which insert this stuff. So IMO it's OK to remove this line from this file because it touches on a low-important topic, not in line with other, higher-level items in this list. Such item may be added to an extended PR checklist though. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #7197: Druid0.13.0 - 405 method not allow when I send post to coordinator url
vogievetsky commented on issue #7197: Druid0.13.0 - 405 method not allow when I send post to coordinator url URL: https://github.com/apache/incubator-druid/issues/7197#issuecomment-494087274 I think you need to post to `/druid/coordinator/v1/lookups/config` not `/druid/coordinator/v1/lookups/config/__default` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #6783: when I startup broker.
vogievetsky commented on issue #6783: when I startup broker. URL: https://github.com/apache/incubator-druid/issues/6783#issuecomment-494091220 Can you post more of the logs? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #6777: how to config bloom Filter?
vogievetsky commented on issue #6777: how to config bloom Filter? URL: https://github.com/apache/incubator-druid/issues/6777#issuecomment-494091399 http://druid.io/docs/latest/development/extensions-core/bloom-filter.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] AndriiSushko opened a new issue #7702: Kafka indexing of avro topic fails with java.lang.UnsupportedOperationException
AndriiSushko opened a new issue #7702: Kafka indexing of avro topic fails with
java.lang.UnsupportedOperationException
URL: https://github.com/apache/incubator-druid/issues/7702
### Affected Version
0.14.1, 0.13.0
### Description
- Druid is up and running:
_./apache-druid-0.14.1-incubating/bin/supervise -c
apache-druid-0.14.1-incubating/quickstart/tutorial/conf/tutorial-cluster.conf_
- druid-avro-extensions is included as an extension.
- Supervisor submitted:
_curl -X POST -H 'Content-Type: application/json' -d @configKafka.json
http://localhost:8090/druid/indexer/v1/supervisor_
with following configKafka.json:
`{
"type": "kafka",
"dataSchema": {
"dataSource": "teststream",
"parser": {
"type": "avro_stream",
"avroBytesDecoder": {
"type": "schema_inline",
"schema": {
"namespace": "io.druid.data",
"name": "User",
"type": "record",
"fields": [
{
"name": "f1",
"type": "string"
},
{
"name": "timestamp",
"type": "long"
},
{
"name": "f2",
"type": "int"
}
]
}
},
"parseSpec": {
"format": "timeAndDims",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [
"f1",
"f2"
]
}
}
},
"metricsSpec": [],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "HOUR",
"queryGranularity": "NONE"
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsPerSegment": 500
},
"ioConfig": {
"topic": "avroTest0",
"consumerProperties": {
"bootstrap.servers": "http://:9092";
},
"taskCount": 1,
"replicas": 1,
"taskDuration": "PT1M"
}
}`
**Expected:**
Kafka topic avtoTest0 being indexed to Druid table teststream.
**Actual:**
I can see the supervisor (http://:8090/console.html) and failed tasks
with following exceptions:
_2019-05-20T17:58:23,412 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner -
Encountered exception in run() before persisting.
java.lang.UnsupportedOperationException: not supported
at
org.apache.druid.data.input.impl.TimeAndDimsParseSpec$1.parseToMap(TimeAndDimsParseSpec.java:53)
~[druid-api-0.13.0-incubating.jar:0.13.0-incubating]
at
org.apache.druid.data.input.impl.StringInputRowParser.parseString(StringInputRowParser.java:155)
~[druid-api-0.13.0-incubating.jar:0.13.0-incubating]
at
org.apache.druid.data.input.impl.StringInputRowParser.buildStringKeyMap(StringInputRowParser.java:119)
~[druid-api-0.13.0-incubating.jar:0.13.0-incubating]
at
org.apache.druid.data.input.impl.StringInputRowParser.parseBatch(StringInputRowParser.java:80)
~[druid-api-0.13.0-incubating.jar:0.13.0-incubating]
at
org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:50)
~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at
org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:31)
~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
at
org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.runInternal(IncrementalPublishingKafkaIndexTaskRunner.java:493)
[druid
[GitHub] [incubator-druid] himanshug commented on issue #7674: add s3 authentication method informations
himanshug commented on issue #7674: add s3 authentication method informations URL: https://github.com/apache/incubator-druid/pull/7674#issuecomment-494095400 LGTM after the build. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #7618: Virtual column updates for exploiting base column internal structure
himanshug commented on issue #7618: Virtual column updates for exploiting base column internal structure URL: https://github.com/apache/incubator-druid/pull/7618#issuecomment-494098637 @clintropolis added comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #7066: Support assign tasks to run on different tiers of MiddleManagers
himanshug commented on issue #7066: Support assign tasks to run on different tiers of MiddleManagers URL: https://github.com/apache/incubator-druid/pull/7066#issuecomment-494100957 @QiuMM great, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7692: Fix case insensitive of ParserUtils.findDuplicates
jihoonson commented on issue #7692: Fix case insensitive of ParserUtils.findDuplicates URL: https://github.com/apache/incubator-druid/pull/7692#issuecomment-494129293 @fjy @drcrallen do you have more comments? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm merged pull request #7700: update lz4-java to 1.6.0 (lz4 1.9.1)
gianm merged pull request #7700: update lz4-java to 1.6.0 (lz4 1.9.1) URL: https://github.com/apache/incubator-druid/pull/7700 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: update lz4-java to 1.6.0 (lz4 1.9.1) (#7700)
This is an automated email from the ASF dual-hosted git repository. gian pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new ddda8b7 update lz4-java to 1.6.0 (lz4 1.9.1) (#7700) ddda8b7 is described below commit ddda8b74cbe095c047c179bc11f674241ad15e00 Author: Clint Wylie AuthorDate: Mon May 20 13:01:48 2019 -0700 update lz4-java to 1.6.0 (lz4 1.9.1) (#7700) --- LICENSE.BINARY | 4 ++-- pom.xml| 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/LICENSE.BINARY b/LICENSE.BINARY index d84bbbf..d73473f 100644 --- a/LICENSE.BINARY +++ b/LICENSE.BINARY @@ -502,8 +502,8 @@ BINARY/JAVA-CORE This product bundles JDBI version 2.63.1. * org.jdbi:jdbi -This product bundles LZ4 Java version 1.5.0. - * org.lz4:lz4-java +This product bundles LZ4 Java version 1.6.0. + * org.lz4:java This product bundles MapDB version 1.0.8. * org.mapdb:mapdb diff --git a/pom.xml b/pom.xml index ba9d713..eb295d0 100644 --- a/pom.xml +++ b/pom.xml @@ -601,7 +601,7 @@ org.lz4 lz4-java -1.5.0 +1.6.0 com.google.protobuf - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] mcbrewster opened a new pull request #7703: Tail
mcbrewster opened a new pull request #7703: Tail URL: https://github.com/apache/incubator-druid/pull/7703 Adds a check box to the task log view. When the check box is checked the get log function is called every 2 seconds and updates the log as well as position the scroll bar at the bottom. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson closed issue #7702: Kafka indexing of Avro topic fails with java.lang.UnsupportedOperationException
jihoonson closed issue #7702: Kafka indexing of Avro topic fails with java.lang.UnsupportedOperationException URL: https://github.com/apache/incubator-druid/issues/7702 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7702: Kafka indexing of Avro topic fails with java.lang.UnsupportedOperationException
jihoonson commented on issue #7702: Kafka indexing of Avro topic fails with java.lang.UnsupportedOperationException URL: https://github.com/apache/incubator-druid/issues/7702#issuecomment-494135118 Hi, it looks that your `parseSpec` is wrong. It should be `avro`. Please check http://druid.io/docs/latest/development/extensions-core/avro.html for details. I'm closing this since it looks not an issue. Please feel free to reopen if you still have this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #7703: Web console: add log tailing to task log view
vogievetsky commented on issue #7703: Web console: add log tailing to task log view URL: https://github.com/apache/incubator-druid/pull/7703#issuecomment-494136003 I think you should move  to be left aligned This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] xvrl commented on issue #7700: update lz4-java to 1.6.0 (lz4 1.9.1)
xvrl commented on issue #7700: update lz4-java to 1.6.0 (lz4 1.9.1) URL: https://github.com/apache/incubator-druid/pull/7700#issuecomment-494138291 nice, I was about to do it but you beat me to it :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vsharathchandra commented on issue #2320: [Proposal] support for setting javaOpts per a task
vsharathchandra commented on issue #2320: [Proposal] support for setting javaOpts per a task URL: https://github.com/apache/incubator-druid/issues/2320#issuecomment-494150627 Is there a way to do the same thing with kafka indexing service ? so that the real time ingestion peons of different datasources can be configured different memory. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] mcbrewster opened a new pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
mcbrewster opened a new pull request #7705: Web-Console: add go to editor button to tasks and supervisors view URL: https://github.com/apache/incubator-druid/pull/7705 https://user-images.githubusercontent.com/37322608/58051943-50f99700-7b08-11e9-9105-0dccf4682cf0.png";> https://user-images.githubusercontent.com/37322608/58051946-52c35a80-7b08-11e9-838f-0edc68cced46.png";> Under the wrench Icon of tasks for type index and index parallel, and kafka and kinesis for supervisors, the go to editor button will open the json spec in the json sepc editor from load-data-view. The button calls the goToLoadDataView and passes in the task/supervisor id, if the task id exists you will be taken directly to the editor view when getJsonValue() will be called and used to update the spec. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT.
gianm opened a new pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT. URL: https://github.com/apache/incubator-druid/pull/7704 They were inadvertently using UTC rather than the default timezone. Also, harmonize how time functions handle their parameters. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT.
gianm commented on issue #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT. URL: https://github.com/apache/incubator-druid/pull/7704#issuecomment-494155166 I don't think there's a reason to backport this to 0.15.0 @fjy This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] surekhasaharan opened a new pull request #7706: Get correct curr_size attribute value for historical servers
surekhasaharan opened a new pull request #7706: Get correct curr_size attribute value for historical servers URL: https://github.com/apache/incubator-druid/pull/7706 `sys.servers` is showing curr_size as 0 for all server types, it should get the correct size for historical server types. Fixed the curr_size, by getting it from `DruidServer` via `InventoryView` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new pull request #7707: SQL: Fix exception with OR of impossible filters.
gianm opened a new pull request #7707: SQL: Fix exception with OR of impossible filters. URL: https://github.com/apache/incubator-druid/pull/7707 Fixes #7671. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug opened a new pull request #7618: Virtual column updates for exploiting base column internal structure
himanshug opened a new pull request #7618: Virtual column updates for exploiting base column internal structure URL: https://github.com/apache/incubator-druid/pull/7618 Fixes #7574 This patch adds more methods to VirtualColumn interface to exploit base column's internal structure and few updates to interface users to use those methods. Unit tests are introduced to ensure expected use of VirtualColumn interface in Druid code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug closed pull request #7618: Virtual column updates for exploiting base column internal structure
himanshug closed pull request #7618: Virtual column updates for exploiting base column internal structure URL: https://github.com/apache/incubator-druid/pull/7618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7703: Web console: add log tailing to task log view
vogievetsky commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285784284
##
File path: web-console/src/components/show-log.tsx
##
@@ -71,12 +73,39 @@ export class ShowLog extends React.Component {
}
}
+ private tail() {
+ this.getLogInfo().then(() => {
Review comment:
don't use `.then` use `async` / `await` instead
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7703: Web console: add log tailing to task log view
vogievetsky commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285784062
##
File path: web-console/src/components/show-log.tsx
##
@@ -71,12 +73,39 @@ export class ShowLog extends React.Component {
}
}
+ private tail() {
+ this.getLogInfo().then(() => {
+if (this.state.tail) {
+ if (this.log.current) {
+this.log.current.scrollTo(0, this.log.current.scrollHeight); }
+ setTimeout(function() {
+this.tail();
+ }.bind(this), 2000);
+}
+ });
Review comment:
oh my the indentation in this function is all messed up also you do not need
to do `.bind(this)` you can just use `() => {}` syntax
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7703: Web console: add log tailing to task log view
vogievetsky commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285784080
##
File path: web-console/src/components/show-log.tsx
##
@@ -43,15 +43,17 @@ export interface ShowLogProps extends React.Props {
export interface ShowLogState {
logValue: string;
+ tail: boolean;
}
export class ShowLog extends React.Component {
+ log = React.createRef();
Review comment:
could you denote with with a `public` prefix and also add a empty line after
this one?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r285795335
##
File path: web-console/src/views/tasks-view.tsx
##
@@ -508,18 +518,24 @@ ORDER BY "rank" DESC, "created_time" DESC`);
;
}
- // --
-
- private getTaskActions(id: string, status: string): BasicAction[] {
-if (status !== 'RUNNING' && status !== 'WAITING' && status !== 'PENDING')
return [];
-return [
- {
-icon: IconNames.CROSS,
+ private getTaskActions(id: string, status: string, type: string):
BasicAction[] {
+const actions = [];
+if (type === 'index' || type === 'index_parallel') {
+ actions.push({
+title: 'Open in json spec editor',
+intent: Intent.DANGER,
Review comment:
Why is the intent danger?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis opened a new pull request #7708: fix result level cache issue with topN when ordering by post-aggregators
clintropolis opened a new pull request #7708: fix result level cache issue with topN when ordering by post-aggregators URL: https://github.com/apache/incubator-druid/pull/7708 This PR fixes an issue with TopN queries where the result level cache was recomputing post-aggregator values that are already available in the cache, causing issues especially when trying to re-compute post aggregators which used finalized aggregator results. The added test will fail with ``` java.lang.ClassCastException: java.lang.Double cannot be cast to org.apache.druid.hll.HyperLogLogCollector ``` if the fix is not in place. Beyond the scope of this fix is whether or not TopN should be computing those post aggregators inside of the cache retrieval method. This appears to be done because the post-aggs which are part of the ordering of the query are used as part of the cache key, but not stored in the cache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] mcbrewster commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
mcbrewster commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r285804403
##
File path: web-console/src/views/tasks-view.tsx
##
@@ -508,18 +518,24 @@ ORDER BY "rank" DESC, "created_time" DESC`);
;
}
- // --
-
- private getTaskActions(id: string, status: string): BasicAction[] {
-if (status !== 'RUNNING' && status !== 'WAITING' && status !== 'PENDING')
return [];
-return [
- {
-icon: IconNames.CROSS,
+ private getTaskActions(id: string, status: string, type: string):
BasicAction[] {
+const actions = [];
+if (type === 'index' || type === 'index_parallel') {
+ actions.push({
+title: 'Open in json spec editor',
+intent: Intent.DANGER,
Review comment:
it shouldn't, I've removed it now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new pull request #7710: SQL: TIME_EXTRACT should have 2 required operands.
gianm opened a new pull request #7710: SQL: TIME_EXTRACT should have 2 required operands. URL: https://github.com/apache/incubator-druid/pull/7710 Timestamp and time unit are both required. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new pull request #7709: SQL: Allow NULLs in place of optional arguments in many functions.
gianm opened a new pull request #7709: SQL: Allow NULLs in place of optional arguments in many functions. URL: https://github.com/apache/incubator-druid/pull/7709 Also adjust SQL docs to describe how to make time literals using TIME_PARSE (which is now possible in a nicer way). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: fix issue where result level cache was recomputing post aggs that were already cached, causing issues with finalizing aggregators (#7708)
This is an automated email from the ASF dual-hosted git repository.
gian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new c4a4223 fix issue where result level cache was recomputing post aggs
that were already cached, causing issues with finalizing aggregators (#7708)
c4a4223 is described below
commit c4a4223c9c2fd2d71958fd7d66ffdf682dfb8967
Author: Clint Wylie
AuthorDate: Mon May 20 16:51:51 2019 -0700
fix issue where result level cache was recomputing post aggs that were
already cached, causing issues with finalizing aggregators (#7708)
---
.../druid/query/topn/TopNQueryQueryToolChest.java | 10 +-
.../query/topn/TopNQueryQueryToolChestTest.java| 131 +
2 files changed, 136 insertions(+), 5 deletions(-)
diff --git
a/processing/src/main/java/org/apache/druid/query/topn/TopNQueryQueryToolChest.java
b/processing/src/main/java/org/apache/druid/query/topn/TopNQueryQueryToolChest.java
index d87a178..53af0908 100644
---
a/processing/src/main/java/org/apache/druid/query/topn/TopNQueryQueryToolChest.java
+++
b/processing/src/main/java/org/apache/druid/query/topn/TopNQueryQueryToolChest.java
@@ -294,8 +294,7 @@ public class TopNQueryQueryToolChest extends
QueryToolChest aggs =
Lists.newArrayList(query.getAggregatorSpecs());
private final List postAggs =
AggregatorUtil.pruneDependentPostAgg(
query.getPostAggregatorSpecs(),
- query.getTopNMetricSpec()
- .getMetricName(query.getDimensionSpec())
+ query.getTopNMetricSpec().getMetricName(query.getDimensionSpec())
);
@Override
@@ -419,14 +418,15 @@ public class TopNQueryQueryToolChest extends
QueryToolChest postItr =
query.getPostAggregatorSpecs().iterator();
while (postItr.hasNext() && resultIter.hasNext()) {
vals.put(postItr.next().getName(), resultIter.next());
}
+ } else {
+for (PostAggregator postAgg : postAggs) {
+ vals.put(postAgg.getName(), postAgg.compute(vals));
+}
}
retVal.add(vals);
}
diff --git
a/processing/src/test/java/org/apache/druid/query/topn/TopNQueryQueryToolChestTest.java
b/processing/src/test/java/org/apache/druid/query/topn/TopNQueryQueryToolChestTest.java
index 191cc55..f9080e7 100644
---
a/processing/src/test/java/org/apache/druid/query/topn/TopNQueryQueryToolChestTest.java
+++
b/processing/src/test/java/org/apache/druid/query/topn/TopNQueryQueryToolChestTest.java
@@ -24,6 +24,7 @@ import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import org.apache.druid.collections.CloseableStupidPool;
import org.apache.druid.collections.SerializablePair;
+import org.apache.druid.hll.HyperLogLogCollector;
import org.apache.druid.java.util.common.DateTimes;
import org.apache.druid.java.util.common.Intervals;
import org.apache.druid.java.util.common.granularity.Granularities;
@@ -40,6 +41,8 @@ import org.apache.druid.query.aggregation.AggregatorFactory;
import org.apache.druid.query.aggregation.CountAggregatorFactory;
import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
import org.apache.druid.query.aggregation.SerializablePairLongString;
+import org.apache.druid.query.aggregation.cardinality.CardinalityAggregator;
+import
org.apache.druid.query.aggregation.hyperloglog.HyperUniquesAggregatorFactory;
import org.apache.druid.query.aggregation.last.DoubleLastAggregatorFactory;
import org.apache.druid.query.aggregation.last.FloatLastAggregatorFactory;
import org.apache.druid.query.aggregation.last.LongLastAggregatorFactory;
@@ -47,6 +50,7 @@ import
org.apache.druid.query.aggregation.last.StringLastAggregatorFactory;
import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
import org.apache.druid.query.aggregation.post.ConstantPostAggregator;
import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
+import
org.apache.druid.query.aggregation.post.FinalizingFieldAccessPostAggregator;
import org.apache.druid.query.dimension.DefaultDimensionSpec;
import org.apache.druid.query.spec.MultipleIntervalSegmentSpec;
import org.apache.druid.segment.IncrementalIndexSegment;
@@ -80,6 +84,15 @@ public class TopNQueryQueryToolChestTest
}
@Test
+ public void testCacheStrategyOrderByPostAggs() throws Exception
+ {
+doTestCacheStrategyOrderByPost(ValueType.STRING, "val1");
+doTestCacheStrategyOrderByPost(ValueType.FLOAT, 2.1f);
+doTestCacheStrategyOrderByPost(ValueType.DOUBLE, 2.1d);
+doTestCacheStrategyOrderByPost(ValueType.LONG, 2L);
+ }
+
+ @Test
public void testComputeCacheKeyWithDifferentPostAgg()
{
final TopNQuery query1 = new TopNQuery(
@@ -306,6 +319,28 @@ public class TopNQueryQueryToolChestTest
}
}
+ private Hy
[GitHub] [incubator-druid] gianm merged pull request #7708: fix result level cache issue with topN when ordering by post-aggregators
gianm merged pull request #7708: fix result level cache issue with topN when ordering by post-aggregators URL: https://github.com/apache/incubator-druid/pull/7708 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT.
gianm commented on issue #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT. URL: https://github.com/apache/incubator-druid/pull/7704#issuecomment-494189759 Getting a weird error from TC when running the inspections: > Warning: No enabled compatible agents for this build configuration. Please register a build agent or tweak build configuration requirements. I'll try closing and reopening. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT.
gianm opened a new pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT. URL: https://github.com/apache/incubator-druid/pull/7704 They were inadvertently using UTC rather than the default timezone. Also, harmonize how time functions handle their parameters. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm closed pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT.
gianm closed pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT. URL: https://github.com/apache/incubator-druid/pull/7704 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug closed issue #7574: making VirtualColumn more powerful
himanshug closed issue #7574: making VirtualColumn more powerful URL: https://github.com/apache/incubator-druid/issues/7574 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug merged pull request #7618: Virtual column updates for exploiting base column internal structure
himanshug merged pull request #7618: Virtual column updates for exploiting base column internal structure URL: https://github.com/apache/incubator-druid/pull/7618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Virtual column updates for exploiting base column internal structure (#7618)
This is an automated email from the ASF dual-hosted git repository.
himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new fb0c846 Virtual column updates for exploiting base column internal
structure (#7618)
fb0c846 is described below
commit fb0c84694143cda3e66d59e3d706f38f3fd5d1cb
Author: Himanshu
AuthorDate: Mon May 20 17:04:35 2019 -0700
Virtual column updates for exploiting base column internal structure (#7618)
* VirtualColumn updates for exploiting base column internal structure
* unit tests for virtual column interface updates
* groupBy needs to use VirtualizedColumnSelectorFactory if outer query in
nested groupBy has virtual columns.
* fix strict compile checks
* fix teamcity build errors
* add comment explaining useVirtualizedColumnSelectorFactory flag in
RowBasedGrouperHelper.createGrouperAccumulatorPair(..)
---
.../epinephelinae/GroupByMergingQueryRunnerV2.java | 3 +-
.../groupby/epinephelinae/GroupByRowProcessor.java | 6 +-
.../epinephelinae/RowBasedGrouperHelper.java | 29 +-
.../query/groupby/strategy/GroupByStrategyV2.java | 4 +-
.../segment/ColumnSelectorBitmapIndexSelector.java | 55 ++-
.../QueryableIndexColumnSelectorFactory.java | 14 +-
.../org/apache/druid/segment/VirtualColumn.java| 55 ++-
.../org/apache/druid/segment/VirtualColumns.java | 32 ++
.../query/groupby/GroupByQueryRunnerTest.java | 6 +-
.../segment/virtual/DummyStringVirtualColumn.java | 289 +++
.../virtual/DummyStringVirtualColumnTest.java | 395 +
11 files changed, 865 insertions(+), 23 deletions(-)
diff --git
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
index b8cc735..ef8c304 100644
---
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
+++
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
@@ -201,7 +201,8 @@ public class GroupByMergingQueryRunnerV2 implements
QueryRunner
priority,
hasTimeout,
timeoutAt,
- mergeBufferSize
+ mergeBufferSize,
+ false
);
final Grouper grouper = pair.lhs;
final Accumulator accumulator = pair.rhs;
diff --git
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByRowProcessor.java
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByRowProcessor.java
index 171b8ad..2fa3b57 100644
---
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByRowProcessor.java
+++
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByRowProcessor.java
@@ -65,7 +65,8 @@ public class GroupByRowProcessor
final String processingTmpDir,
final int mergeBufferSize,
final List closeOnExit,
- final boolean wasQueryPushedDown
+ final boolean wasQueryPushedDown,
+ final boolean useVirtualizedColumnSelectorFactory
)
{
final GroupByQuery query = (GroupByQuery) queryParam;
@@ -112,7 +113,8 @@ public class GroupByRowProcessor
temporaryStorage,
spillMapper,
aggregatorFactories,
-mergeBufferSize
+mergeBufferSize,
+useVirtualizedColumnSelectorFactory
);
final Grouper grouper = pair.lhs;
final Accumulator accumulator = pair.rhs;
diff --git
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/RowBasedGrouperHelper.java
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/RowBasedGrouperHelper.java
index 0b82472..bd41d08 100644
---
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/RowBasedGrouperHelper.java
+++
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/RowBasedGrouperHelper.java
@@ -105,7 +105,8 @@ public class RowBasedGrouperHelper
final LimitedTemporaryStorage temporaryStorage,
final ObjectMapper spillMapper,
final AggregatorFactory[] aggregatorFactories,
- final int mergeBufferSize
+ final int mergeBufferSize,
+ final boolean useVirtualizedColumnSelectorFactory
)
{
return createGrouperAccumulatorPair(
@@ -123,7 +124,8 @@ public class RowBasedGrouperHelper
UNKNOWN_THREAD_PRIORITY,
false,
UNKNOWN_TIMEOUT,
-mergeBufferSize
+mergeBufferSize,
+useVirtualizedColumnSelectorFactory
);
}
@@ -147,7 +149,8 @@ public class RowBasedGrouperHelper
final int priority,
final boolean h
[GitHub] [incubator-druid] mcbrewster commented on a change in pull request #7703: Web console: add log tailing to task log view
mcbrewster commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285809509
##
File path: web-console/src/components/show-log.tsx
##
@@ -43,15 +43,17 @@ export interface ShowLogProps extends React.Props {
export interface ShowLogState {
logValue: string;
+ tail: boolean;
}
export class ShowLog extends React.Component {
+ log = React.createRef();
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] mcbrewster commented on a change in pull request #7703: Web console: add log tailing to task log view
mcbrewster commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285809537
##
File path: web-console/src/components/show-log.tsx
##
@@ -71,12 +73,39 @@ export class ShowLog extends React.Component {
}
}
+ private tail() {
+ this.getLogInfo().then(() => {
+if (this.state.tail) {
+ if (this.log.current) {
+this.log.current.scrollTo(0, this.log.current.scrollHeight); }
+ setTimeout(function() {
+this.tail();
+ }.bind(this), 2000);
+}
+ });
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] mcbrewster commented on a change in pull request #7703: Web console: add log tailing to task log view
mcbrewster commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285809467
##
File path: web-console/src/components/show-log.tsx
##
@@ -71,12 +73,39 @@ export class ShowLog extends React.Component {
}
}
+ private tail() {
+ this.getLogInfo().then(() => {
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] mcbrewster commented on a change in pull request #7703: Web console: add log tailing to task log view
mcbrewster commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285809509
##
File path: web-console/src/components/show-log.tsx
##
@@ -43,15 +43,17 @@ export interface ShowLogProps extends React.Props {
export interface ShowLogState {
logValue: string;
+ tail: boolean;
}
export class ShowLog extends React.Component {
+ log = React.createRef();
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] mcbrewster commented on a change in pull request #7703: Web console: add log tailing to task log view
mcbrewster commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285809899
##
File path: web-console/src/components/show-log.tsx
##
@@ -43,15 +43,17 @@ export interface ShowLogProps extends React.Props {
export interface ShowLogState {
logValue: string;
+ tail: boolean;
}
export class ShowLog extends React.Component {
+ log = React.createRef();
Review comment:
https://user-images.githubusercontent.com/37322608/58059475-eb1a0900-7b21-11e9-9687-de20c9e87510.png";>
Ts lint isn't onboard with that, should I do it anyway?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] mcbrewster commented on a change in pull request #7703: Web console: add log tailing to task log view
mcbrewster commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r285809899
##
File path: web-console/src/components/show-log.tsx
##
@@ -43,15 +43,17 @@ export interface ShowLogProps extends React.Props {
export interface ShowLogState {
logValue: string;
+ tail: boolean;
}
export class ShowLog extends React.Component {
+ log = React.createRef();
Review comment:
https://user-images.githubusercontent.com/37322608/58059475-eb1a0900-7b21-11e9-9687-de20c9e87510.png";>
Ts lint isn't onboard with that, should I do it anyway
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7710: SQL: TIME_EXTRACT should have 2 required operands.
jihoonson commented on issue #7710: SQL: TIME_EXTRACT should have 2 required operands. URL: https://github.com/apache/incubator-druid/pull/7710#issuecomment-494194629 Can we add a unit test that checks this function fails with a single operand? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT.
jihoonson commented on a change in pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT. URL: https://github.com/apache/incubator-druid/pull/7704#discussion_r285813126 ## File path: sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java ## @@ -5995,7 +5998,7 @@ public void testTimeseriesUsingTimeFloorWithTimeShift() throws Exception .setVirtualColumns( expressionVirtualColumn( "v0", - "timestamp_floor(timestamp_shift(\"__time\",'P1D',-1),'P1M',null,'UTC')", + "timestamp_floor(timestamp_shift(\"__time\",'P1D',-1,'UTC'),'P1M',null,'UTC')", Review comment: Should we add a test with non UTC timezone as well? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson opened a new pull request #7711: [Backport] fix result level cache issue with topN when ordering by post-aggregators
jihoonson opened a new pull request #7711: [Backport] fix result level cache issue with topN when ordering by post-aggregators URL: https://github.com/apache/incubator-druid/pull/7711 Backport of #7708 to 0.15.0-incubating. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7692: Fix case insensitive of ParserUtils.findDuplicates
jihoonson commented on issue #7692: Fix case insensitive of ParserUtils.findDuplicates URL: https://github.com/apache/incubator-druid/pull/7692#issuecomment-494198938 I'm merging this PR. @drcrallen Please let me know if you have more comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson merged pull request #7692: Fix case insensitive of ParserUtils.findDuplicates
jihoonson merged pull request #7692: Fix case insensitive of ParserUtils.findDuplicates URL: https://github.com/apache/incubator-druid/pull/7692 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Fix case insensitive of ParserUtils.findDuplicates (#7692)
This is an automated email from the ASF dual-hosted git repository.
jihoonson pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new d69aa6f Fix case insensitive of ParserUtils.findDuplicates (#7692)
d69aa6f is described below
commit d69aa6f7f660e3839dc9f829753bb31c0046a9ea
Author: Jihoon Son
AuthorDate: Mon May 20 17:49:15 2019 -0700
Fix case insensitive of ParserUtils.findDuplicates (#7692)
* Fix case insensitive ParserUtils.findDuplicates
* unused import
---
.../druid/data/input/impl/DimensionsSpec.java | 4 +-
.../java/util/common/parsers/ParserUtils.java | 12 +++---
.../java/util/common/parsers/ParserUtilsTest.java | 45 ++
3 files changed, 52 insertions(+), 9 deletions(-)
diff --git
a/core/src/main/java/org/apache/druid/data/input/impl/DimensionsSpec.java
b/core/src/main/java/org/apache/druid/data/input/impl/DimensionsSpec.java
index ab23b4a..54c90e0 100644
--- a/core/src/main/java/org/apache/druid/data/input/impl/DimensionsSpec.java
+++ b/core/src/main/java/org/apache/druid/data/input/impl/DimensionsSpec.java
@@ -199,9 +199,6 @@ public class DimensionsSpec
"dimensions and dimensions exclusions cannot overlap"
);
-ParserUtils.validateFields(dimNames);
-ParserUtils.validateFields(dimensionExclusions);
-
List spatialDimNames = Lists.transform(
spatialDimensions,
new Function()
@@ -216,6 +213,7 @@ public class DimensionsSpec
// Don't allow duplicates between main list and deprecated spatial list
ParserUtils.validateFields(Iterables.concat(dimNames, spatialDimNames));
+ParserUtils.validateFields(dimensionExclusions);
}
@Override
diff --git
a/core/src/main/java/org/apache/druid/java/util/common/parsers/ParserUtils.java
b/core/src/main/java/org/apache/druid/java/util/common/parsers/ParserUtils.java
index bfe732c..57f52c8 100644
---
a/core/src/main/java/org/apache/druid/java/util/common/parsers/ParserUtils.java
+++
b/core/src/main/java/org/apache/druid/java/util/common/parsers/ParserUtils.java
@@ -19,10 +19,10 @@
package org.apache.druid.java.util.common.parsers;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Function;
import com.google.common.base.Splitter;
import org.apache.druid.common.config.NullHandling;
-import org.apache.druid.java.util.common.StringUtils;
import org.joda.time.DateTimeZone;
import javax.annotation.Nullable;
@@ -77,17 +77,17 @@ public class ParserUtils
return names;
}
- public static Set findDuplicates(Iterable fieldNames)
+ @VisibleForTesting
+ static Set findDuplicates(Iterable fieldNames)
{
Set duplicates = new HashSet<>();
Set uniqueNames = new HashSet<>();
for (String fieldName : fieldNames) {
- String next = StringUtils.toLowerCase(fieldName);
- if (uniqueNames.contains(next)) {
-duplicates.add(next);
+ if (uniqueNames.contains(fieldName)) {
+duplicates.add(fieldName);
}
- uniqueNames.add(next);
+ uniqueNames.add(fieldName);
}
return duplicates;
diff --git
a/core/src/test/java/org/apache/druid/java/util/common/parsers/ParserUtilsTest.java
b/core/src/test/java/org/apache/druid/java/util/common/parsers/ParserUtilsTest.java
new file mode 100644
index 000..5645733
--- /dev/null
+++
b/core/src/test/java/org/apache/druid/java/util/common/parsers/ParserUtilsTest.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.parsers;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.List;
+
+public class ParserUtilsTest
+{
+ @Test
+ public void testFindDuplicatesMixedCases()
+ {
+final List fields = ImmutableList.of("f1", "f2", "F1", "F2", "f3");
+Assert.assertEquals(Collections.emptySet(),
ParserUtils.findDuplicates(fields));
+ }
+
+ @Test
+ public void testFindDuplicates()
+ {
+final List fi
[GitHub] [incubator-druid] jihoonson opened a new pull request #7712: [Backport] Fix case insensitive of ParserUtils.findDuplicates
jihoonson opened a new pull request #7712: [Backport] Fix case insensitive of ParserUtils.findDuplicates URL: https://github.com/apache/incubator-druid/pull/7712 Backport of #7692 to 0.15.0-incubating. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT.
gianm commented on a change in pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT. URL: https://github.com/apache/incubator-druid/pull/7704#discussion_r285817027 ## File path: sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java ## @@ -5995,7 +5998,7 @@ public void testTimeseriesUsingTimeFloorWithTimeShift() throws Exception .setVirtualColumns( expressionVirtualColumn( "v0", - "timestamp_floor(timestamp_shift(\"__time\",'P1D',-1),'P1M',null,'UTC')", + "timestamp_floor(timestamp_shift(\"__time\",'P1D',-1,'UTC'),'P1M',null,'UTC')", Review comment: I added one in ExpressionsTest. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #7710: SQL: TIME_EXTRACT should have 2 required operands.
gianm commented on issue #7710: SQL: TIME_EXTRACT should have 2 required operands. URL: https://github.com/apache/incubator-druid/pull/7710#issuecomment-494200891 I thought about doing that, but we don't have tests like that for any other functions, so I didn't. But since you asked, I just added one 😄 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new pull request #7713: SqlOperatorConversion Javadoc fix.
gianm opened a new pull request #7713: SqlOperatorConversion Javadoc fix. URL: https://github.com/apache/incubator-druid/pull/7713 Appears to be a copypasta error; the toDruidFilter method was referred to aggregations, but it's not handling aggregations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT.
jihoonson commented on a change in pull request #7704: SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT. URL: https://github.com/apache/incubator-druid/pull/7704#discussion_r285819812 ## File path: sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java ## @@ -5995,7 +5998,7 @@ public void testTimeseriesUsingTimeFloorWithTimeShift() throws Exception .setVirtualColumns( expressionVirtualColumn( "v0", - "timestamp_floor(timestamp_shift(\"__time\",'P1D',-1),'P1M',null,'UTC')", + "timestamp_floor(timestamp_shift(\"__time\",'P1D',-1,'UTC'),'P1M',null,'UTC')", Review comment: Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
URL: https://github.com/apache/incubator-druid/pull/7695#discussion_r285814113
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/TimeArithmeticOperatorConversion.java
##
@@ -70,44 +71,67 @@ public DruidExpression toDruidExpression(
throw new IAE("Expected 2 args, got %s", operands.size());
}
-final RexNode timeRexNode = operands.get(0);
-final RexNode shiftRexNode = operands.get(1);
+final RexNode leftRexNode = operands.get(0);
+final RexNode rightRexNode = operands.get(1);
-final DruidExpression timeExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, timeRexNode);
-final DruidExpression shiftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, shiftRexNode);
+final DruidExpression leftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, leftRexNode);
+final DruidExpression rightExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, rightRexNode);
-if (timeExpr == null || shiftExpr == null) {
+if (leftExpr == null || rightExpr == null) {
return null;
}
-if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
+if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
// timestamp_expr { + | - } (year-month interval)
// Period is a value in months.
return DruidExpression.fromExpression(
DruidExpression.functionCall(
"timestamp_shift",
- timeExpr,
- shiftExpr.map(
+ leftExpr,
+ rightExpr.map(
simpleExtraction -> null,
expression -> StringUtils.format("concat('P', %s, 'M')",
expression)
),
DruidExpression.fromExpression(DruidExpression.numberLiteral(direction > 0 ? 1
: -1))
)
);
-} else if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
+} else if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
// timestamp_expr { + | - } (day-time interval)
// Period is a value in milliseconds. Ignore time zone.
return DruidExpression.fromExpression(
StringUtils.format(
"(%s %s %s)",
- timeExpr.getExpression(),
+ leftExpr.getExpression(),
direction > 0 ? "+" : "-",
- shiftExpr.getExpression()
+ rightExpr.getExpression()
)
);
+} else if ((leftRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
+leftRexNode.getType().getFamily() == SqlTypeFamily.DATE) &&
+(rightRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
+rightRexNode.getType().getFamily() == SqlTypeFamily.DATE)) {
+ final String exprstr;
+ if (call.getType().getSqlTypeName() == SqlTypeName.INTERVAL_MONTH ||
Review comment:
I think this should include `INTERVAL_YEAR_MONTH` as well. Or, just check if
the type family is `SqlTypeFamily.INTERVAL_YEAR_MONTH`. It includes all three.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
URL: https://github.com/apache/incubator-druid/pull/7695#discussion_r285815332
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/TimeArithmeticOperatorConversion.java
##
@@ -70,44 +71,67 @@ public DruidExpression toDruidExpression(
throw new IAE("Expected 2 args, got %s", operands.size());
}
-final RexNode timeRexNode = operands.get(0);
-final RexNode shiftRexNode = operands.get(1);
+final RexNode leftRexNode = operands.get(0);
+final RexNode rightRexNode = operands.get(1);
-final DruidExpression timeExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, timeRexNode);
-final DruidExpression shiftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, shiftRexNode);
+final DruidExpression leftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, leftRexNode);
+final DruidExpression rightExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, rightRexNode);
-if (timeExpr == null || shiftExpr == null) {
+if (leftExpr == null || rightExpr == null) {
return null;
}
-if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
+if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
// timestamp_expr { + | - } (year-month interval)
// Period is a value in months.
return DruidExpression.fromExpression(
DruidExpression.functionCall(
"timestamp_shift",
- timeExpr,
- shiftExpr.map(
+ leftExpr,
+ rightExpr.map(
simpleExtraction -> null,
expression -> StringUtils.format("concat('P', %s, 'M')",
expression)
),
DruidExpression.fromExpression(DruidExpression.numberLiteral(direction > 0 ? 1
: -1))
)
);
-} else if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
+} else if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
// timestamp_expr { + | - } (day-time interval)
// Period is a value in milliseconds. Ignore time zone.
return DruidExpression.fromExpression(
StringUtils.format(
"(%s %s %s)",
- timeExpr.getExpression(),
+ leftExpr.getExpression(),
direction > 0 ? "+" : "-",
- shiftExpr.getExpression()
+ rightExpr.getExpression()
)
);
+} else if ((leftRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
+leftRexNode.getType().getFamily() == SqlTypeFamily.DATE) &&
+(rightRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
+rightRexNode.getType().getFamily() == SqlTypeFamily.DATE)) {
+ final String exprstr;
Review comment:
Somewhere in here, please verify the `direction` is minus. (We don't expect
to be here for plus, and the generated code wouldn't make sense.) Something
like:
```java
Preconditions.checkState(direction < 0, "direction < 0");
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
URL: https://github.com/apache/incubator-druid/pull/7695#discussion_r285818466
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/TimeArithmeticOperatorConversion.java
##
@@ -70,44 +71,67 @@ public DruidExpression toDruidExpression(
throw new IAE("Expected 2 args, got %s", operands.size());
}
-final RexNode timeRexNode = operands.get(0);
-final RexNode shiftRexNode = operands.get(1);
+final RexNode leftRexNode = operands.get(0);
+final RexNode rightRexNode = operands.get(1);
-final DruidExpression timeExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, timeRexNode);
-final DruidExpression shiftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, shiftRexNode);
+final DruidExpression leftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, leftRexNode);
+final DruidExpression rightExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, rightRexNode);
-if (timeExpr == null || shiftExpr == null) {
+if (leftExpr == null || rightExpr == null) {
return null;
}
-if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
+if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
// timestamp_expr { + | - } (year-month interval)
// Period is a value in months.
return DruidExpression.fromExpression(
DruidExpression.functionCall(
"timestamp_shift",
- timeExpr,
- shiftExpr.map(
+ leftExpr,
+ rightExpr.map(
simpleExtraction -> null,
expression -> StringUtils.format("concat('P', %s, 'M')",
expression)
),
DruidExpression.fromExpression(DruidExpression.numberLiteral(direction > 0 ? 1
: -1))
)
);
-} else if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
+} else if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
// timestamp_expr { + | - } (day-time interval)
// Period is a value in milliseconds. Ignore time zone.
return DruidExpression.fromExpression(
StringUtils.format(
"(%s %s %s)",
- timeExpr.getExpression(),
+ leftExpr.getExpression(),
direction > 0 ? "+" : "-",
- shiftExpr.getExpression()
+ rightExpr.getExpression()
)
);
+} else if ((leftRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
+leftRexNode.getType().getFamily() == SqlTypeFamily.DATE) &&
+(rightRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
+rightRexNode.getType().getFamily() == SqlTypeFamily.DATE)) {
+ final String exprstr;
+ if (call.getType().getSqlTypeName() == SqlTypeName.INTERVAL_MONTH ||
+ call.getType().getSqlTypeName() == SqlTypeName.INTERVAL_YEAR) {
+exprstr = StringUtils.format(
+"subtract_months(%s, %s)",
+leftExpr.getExpression(),
+rightExpr.getExpression()
+);
+ } else {
+exprstr = StringUtils.format(
+"(%s %s %s)",
+leftExpr.getExpression(),
+"-",
+rightExpr.getExpression()
+);
+ }
+
+ DruidExpression expr = DruidExpression.fromExpression(exprstr);
+ return expr;
Review comment:
Nit: might as well collapse these.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
URL: https://github.com/apache/incubator-druid/pull/7695#discussion_r285832035
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/TimeArithmeticOperatorConversion.java
##
@@ -70,44 +71,67 @@ public DruidExpression toDruidExpression(
throw new IAE("Expected 2 args, got %s", operands.size());
}
-final RexNode timeRexNode = operands.get(0);
-final RexNode shiftRexNode = operands.get(1);
+final RexNode leftRexNode = operands.get(0);
+final RexNode rightRexNode = operands.get(1);
-final DruidExpression timeExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, timeRexNode);
-final DruidExpression shiftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, shiftRexNode);
+final DruidExpression leftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, leftRexNode);
+final DruidExpression rightExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, rightRexNode);
-if (timeExpr == null || shiftExpr == null) {
+if (leftExpr == null || rightExpr == null) {
return null;
}
-if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
+if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
// timestamp_expr { + | - } (year-month interval)
// Period is a value in months.
return DruidExpression.fromExpression(
DruidExpression.functionCall(
"timestamp_shift",
- timeExpr,
- shiftExpr.map(
+ leftExpr,
+ rightExpr.map(
simpleExtraction -> null,
expression -> StringUtils.format("concat('P', %s, 'M')",
expression)
),
DruidExpression.fromExpression(DruidExpression.numberLiteral(direction > 0 ? 1
: -1))
)
);
-} else if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
+} else if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
// timestamp_expr { + | - } (day-time interval)
// Period is a value in milliseconds. Ignore time zone.
return DruidExpression.fromExpression(
StringUtils.format(
"(%s %s %s)",
- timeExpr.getExpression(),
+ leftExpr.getExpression(),
direction > 0 ? "+" : "-",
- shiftExpr.getExpression()
+ rightExpr.getExpression()
)
);
+} else if ((leftRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
Review comment:
Calcite forces us to do it this way, but I think it's pretty un-intuitive.
Could you please include a comment explaining what's going on? Maybe something
like:
```
// Calcite represents both TIMESTAMP - INTERVAL and TIMESTAMPDIFF (TIMESTAMP
- TIMESTAMP)
// with a DATE_MINUS operator, so we must tell which case we're in by
checking the type of
// the second argument.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
gianm commented on a change in pull request #7695: Add TIMESTAMPDIFF sql support
URL: https://github.com/apache/incubator-druid/pull/7695#discussion_r285812299
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/TimeArithmeticOperatorConversion.java
##
@@ -70,44 +71,67 @@ public DruidExpression toDruidExpression(
throw new IAE("Expected 2 args, got %s", operands.size());
}
-final RexNode timeRexNode = operands.get(0);
-final RexNode shiftRexNode = operands.get(1);
+final RexNode leftRexNode = operands.get(0);
+final RexNode rightRexNode = operands.get(1);
-final DruidExpression timeExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, timeRexNode);
-final DruidExpression shiftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, shiftRexNode);
+final DruidExpression leftExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, leftRexNode);
+final DruidExpression rightExpr =
Expressions.toDruidExpression(plannerContext, rowSignature, rightRexNode);
-if (timeExpr == null || shiftExpr == null) {
+if (leftExpr == null || rightExpr == null) {
return null;
}
-if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
+if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_YEAR_MONTH) {
// timestamp_expr { + | - } (year-month interval)
// Period is a value in months.
return DruidExpression.fromExpression(
DruidExpression.functionCall(
"timestamp_shift",
- timeExpr,
- shiftExpr.map(
+ leftExpr,
+ rightExpr.map(
simpleExtraction -> null,
expression -> StringUtils.format("concat('P', %s, 'M')",
expression)
),
DruidExpression.fromExpression(DruidExpression.numberLiteral(direction > 0 ? 1
: -1))
)
);
-} else if (shiftRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
+} else if (rightRexNode.getType().getFamily() ==
SqlTypeFamily.INTERVAL_DAY_TIME) {
// timestamp_expr { + | - } (day-time interval)
// Period is a value in milliseconds. Ignore time zone.
return DruidExpression.fromExpression(
StringUtils.format(
"(%s %s %s)",
- timeExpr.getExpression(),
+ leftExpr.getExpression(),
direction > 0 ? "+" : "-",
- shiftExpr.getExpression()
+ rightExpr.getExpression()
)
);
+} else if ((leftRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
+leftRexNode.getType().getFamily() == SqlTypeFamily.DATE) &&
+(rightRexNode.getType().getFamily() == SqlTypeFamily.TIMESTAMP ||
+rightRexNode.getType().getFamily() == SqlTypeFamily.DATE)) {
+ final String exprstr;
+ if (call.getType().getSqlTypeName() == SqlTypeName.INTERVAL_MONTH ||
+ call.getType().getSqlTypeName() == SqlTypeName.INTERVAL_YEAR) {
+exprstr = StringUtils.format(
+"subtract_months(%s, %s)",
Review comment:
`subtract_months` should also take a timezone parameter, and this code here
should pass down the context timezone. (See #7704 for a recent PR fixing this
behavior for TIME_SHIFT.) The idea is that the month that the two instants are
in, and therefore the expected result of `subtract_months`, could vary based on
the timezone.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #7713: SqlOperatorConversion Javadoc fix.
fjy merged pull request #7713: SqlOperatorConversion Javadoc fix. URL: https://github.com/apache/incubator-druid/pull/7713 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: SqlOperatorConversion Javadoc fix. (#7713)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new cb6ec2c SqlOperatorConversion Javadoc fix. (#7713)
cb6ec2c is described below
commit cb6ec2cab856dc0a36bc2ebd480a349ffaaece77
Author: Gian Merlino
AuthorDate: Mon May 20 21:21:21 2019 -0700
SqlOperatorConversion Javadoc fix. (#7713)
Appears to be a copypasta error; the toDruidFilter method was referred
to aggregations, but it's not handling aggregations.
---
.../apache/druid/sql/calcite/expression/SqlOperatorConversion.java| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/SqlOperatorConversion.java
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/SqlOperatorConversion.java
index dc85fdb..88d4947 100644
---
a/sql/src/main/java/org/apache/druid/sql/calcite/expression/SqlOperatorConversion.java
+++
b/sql/src/main/java/org/apache/druid/sql/calcite/expression/SqlOperatorConversion.java
@@ -52,10 +52,10 @@ public interface SqlOperatorConversion
DruidExpression toDruidExpression(PlannerContext plannerContext,
RowSignature rowSignature, RexNode rexNode);
/**
- * Returns a Druid Aggregation corresponding to a SQL {@link SqlOperator}
used to filter rows
+ * Returns a Druid filter corresponding to a Calcite {@code RexNode} used as
a filter condition.
*
* @param plannerContext SQL planner context
- * @param querySignature signature of the rows being aggregated and
expression column references
+ * @param querySignature signature of the rows being filtered, and any
expression column references
* @param rexNode filter expression rex node
*
* @return filter, or null if the call cannot be translated
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed pull request #7241: document query downtime behavior of upgrading overlords in 0.14
stale[bot] closed pull request #7241: document query downtime behavior of upgrading overlords in 0.14 URL: https://github.com/apache/incubator-druid/pull/7241 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7241: document query downtime behavior of upgrading overlords in 0.14
stale[bot] commented on issue #7241: document query downtime behavior of upgrading overlords in 0.14 URL: https://github.com/apache/incubator-druid/pull/7241#issuecomment-494238196 This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] prabakarankrishnan commented on issue #7623: How to Enable HistoricalMetricsMonitor、QueryCountStatsMonitor metric for Historical 、Middle Manager ；Because I started both Hist
prabakarankrishnan commented on issue #7623: How to Enable HistoricalMetricsMonitor、QueryCountStatsMonitor metric for Historical 、Middle Manager ；Because I started both Historical and Middle Manager processes on the same node URL: https://github.com/apache/incubator-druid/issues/7623#issuecomment-494255230 I am also getting same error in Druid 0.14.0 version... Any update This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
