[GitHub] [incubator-druid] AlexandreYang opened a new pull request #8238: [statsd-emitter] Add config to send Druid process/service as tag
AlexandreYang opened a new pull request #8238: [statsd-emitter] Add config to send Druid process/service as tag URL: https://github.com/apache/incubator-druid/pull/8238 ### Description When using `statsd-emitter` with Dogstatsd, the Druid process/service name (e.g. broker/coordinator) is included in the metric name e.g. `druid.broker.jvm.gc.cpu`. For the same metric e.g. `jvm.gc.cpu`, multiple metric names will be generated (`druid.broker.jvm.gc.cpu`, `druid.coordinator.jvm.gc.cpu`, `druid.historical.jvm.gc.cpu`, etc). **Since we have multiple metric names it's harder to get statistics across all Druid process types.** This PR intend to solve this issue by adding this new configuration: |property|description|required?|default| ||---|-|---| |`druid.emitter.statsd.dogstatsdServiceAsTag`|If `druid.emitter.statsd.dogstatsdServiceAsTag` is true, druid service (e.g. druid/broker, druid/coordinator, etc) is reported as a tag (e.g. `service:druid/broker`) instead of being included in metric name (e.g. `druid.broker.my_metric`).|no|false| So when this `druid.emitter.statsd.dogstatsdServiceAsTag` is true, the resulting metric send would be `jvm.gc.cpu` as name with a tag `service:druid/broker`. If needed, it can be combined with `druid.emitter.statsd.prefix` to add `druid` as prefix to get metric name like `druid.jvm.gc.cpu` instead of `jvm.gc.cpu`. This PR has: - [ ] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist above are strictly necessary, but it would be very helpful if you at least self-review the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs
stale[bot] commented on issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs URL: https://github.com/apache/incubator-druid/issues/6539#issuecomment-518185577 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #7562: Enable toggling request logging on/off for different query types
leventov commented on a change in pull request #7562: Enable toggling request logging on/off for different query types URL: https://github.com/apache/incubator-druid/pull/7562#discussion_r310683092 ## File path: server/src/main/java/org/apache/druid/server/log/FileRequestLoggerProvider.java ## @@ -58,8 +58,12 @@ @Override public RequestLogger get() { -FileRequestLogger logger = new FileRequestLogger(jsonMapper, factory.create(1, "RequestLogger-%s"), dir, -filePattern); +FileRequestLogger logger = new FileRequestLogger( +jsonMapper, Review comment: In Druid, the wrapping indentation is 4, not 8 spaces. Please also fix other places in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov opened a new issue #8239: Specify lineWrappingIndentation in Checkstyle's Indentation check
leventov opened a new issue #8239: Specify lineWrappingIndentation in Checkstyle's Indentation check URL: https://github.com/apache/incubator-druid/issues/8239 See https://checkstyle.org/config_misc.html#Indentation Currently, it's `lineWrappingIndentation` and `forceStrictCondition` are not set: https://github.com/apache/incubator-druid/blob/ac856fe4c179093c0a5e309a2e988c54a86e89c5/codestyle/checkstyle.xml#L112-L115 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310684935 ## File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java ## @@ -123,6 +123,17 @@ private Closer() return closeable; } + /** + * Registers a list of {@code closeable} to be closed when this {@code Closer} is Review comment: If `{@code closeable}` is a reference to the interface `Closeable`, then it should start with a capital letter. If this is a reference to the parameter, then it should be `closeables`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310686168 ## File path: processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java ## @@ -280,12 +280,17 @@ public AggregateResult call() return RowBasedGrouperHelper.makeGrouperIterator( grouper, query, - () -> Lists.reverse(resources).forEach(CloseQuietly::close) + resources ); } catch (Throwable e) { Review comment: Please rename to `t` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310683772 ## File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java ## @@ -123,6 +123,17 @@ private Closer() return closeable; } + /** + * Registers a list of {@code closeable} to be closed when this {@code Closer} is + * {@linkplain #close closed}. + */ + // close. this word no longer has any meaning to me. Review comment: Please remove this comment, tt doesn't seem to make any sense. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310684423 ## File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java ## @@ -123,6 +123,17 @@ private Closer() return closeable; } + /** + * Registers a list of {@code closeable} to be closed when this {@code Closer} is + * {@linkplain #close closed}. + */ + // close. this word no longer has any meaning to me. + public Iterable register(Iterable closeables) Review comment: Some class may implement both `Iterable` and `Closeable`, so ambiguity with the other `register()` method is possible. I suggest to call this method `registerAll()`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310686115 ## File path: processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java ## @@ -280,12 +280,17 @@ public AggregateResult call() return RowBasedGrouperHelper.makeGrouperIterator( grouper, query, - () -> Lists.reverse(resources).forEach(CloseQuietly::close) + resources ); } catch (Throwable e) { // Exception caught while setting up the iterator; release resources. - Lists.reverse(resources).forEach(CloseQuietly::close); + try { +resources.close(); + } + catch (IOException ex) { Review comment: I don't think it makes sense to catch anything weaker than `Throable` in another `catch (Throwable e) {` block. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed issue #6408: Selector filter for multi-value dimensions is not returning all values
stale[bot] closed issue #6408: Selector filter for multi-value dimensions is not returning all values URL: https://github.com/apache/incubator-druid/issues/6408 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6408: Selector filter for multi-value dimensions is not returning all values
stale[bot] commented on issue #6408: Selector filter for multi-value dimensions is not returning all values URL: https://github.com/apache/incubator-druid/issues/6408#issuecomment-518311283 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh opened a new pull request #8240: WIP Speedup Travis CI jobs
ccaominh opened a new pull request #8240: WIP Speedup Travis CI jobs URL: https://github.com/apache/incubator-druid/pull/8240 _**NOTE: This PR is a work-in-progress to see effects of the Travis CI changes.**_ ### Description Reorganize Travis CI jobs into smaller (and more) jobs. Add various maven options to skip unnecessary work and refactored Travis CI job definitions to follow DRY. Detailed changes: .travis.yml - Refactor build logic to get rid of copy-and-paste logic - Skip static checks and enable parallelism for maven install - Split static analysis into different jobs to ease triage - Use "name" attribute instead of NAME environment variable - Split "indexing" and "web console" out of "other modules test" - Split 2 integration test jobs into multiple smaller jobs build.sh - Enable parallelism - Disable more static checks travis_script_integration.sh travis_script_integration_part2.sh integration-tests/README.md - Use TestNG groups instead of shell scripts and move definition of jobs into Travis CI yaml integration-tests/pom.xml - Show elapsed time of individual tests to aid in rebalancing Travis CI integration test jobs run time TestNGGroup.java - Use TestNG groups to make it easy to have multiple Travis CI integration test jobs. TestNG groups also make it easier to have an "other" integration test group and makes it less likely a test will accidentally not be included in a CI job. ITHadoopIndexTest.java AbstractITBatchIndexTest.java AbstractKafkaIndexerTest.java ITAppenderatorDriverRealtimeIndexTaskTest.java ITCompactionTaskTest.java ITIndexerTest.java ITKafkaIndexingServiceTest.java ITKafkaIndexingServiceTransactionalTest.java ITNestedQueryPushDownTest.java ITParallelIndexTest.java ITRealtimeIndexTaskTest.java ITSystemTableBatchIndexTaskTest.java ITUnionQueryTest.java ITSystemTableQueryTest.java ITTwitterQueryTest.java ITWikipediaQueryTest.java ITBasicAuthConfigurationTest.java ITCoordinatorOverlordProxyAuthTest.java ITTLSTest.java - Add TestNG group - Fix various IntelliJ inspection warnings This PR has: - [x] been self-reviewed. - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov opened a new issue #8241: Prohibit creating collections from stream.collect(...) using Structural Search inspection
leventov opened a new issue #8241: Prohibit creating collections from stream.collect(...) using Structural Search inspection URL: https://github.com/apache/incubator-druid/issues/8241 1. ``` new $Coll$($stream$.collect($collector$)) ``` The whole match filter: type=java.util.Collection, check "Within type hierarchy". 2. The same about `Map` 3. ``` $Colls$.$factory$($stream$.collect($collector$)) ``` Filter on `Colls`: regex `Lists|Sets|Maps` - this is to catch Guava's factory methods like `Sets.newHashSet()`. There are violations of these rules in the codebase. See the instruction about how to add a rule [here](https://github.com/apache/incubator-druid/blob/ac856fe4c179093c0a5e309a2e988c54a86e89c5/dev/teamcity.md#creating-a-custom-inspection-from-a-structural-search-pattern). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310704426 ## File path: processing/src/main/java/org/apache/druid/query/filter/ColumnComparisonDimFilter.java ## @@ -114,10 +141,4 @@ public boolean equals(Object o) .collect(Collectors.toSet()) Review comment: Please fix `Sets.newHashSet(...collect(...))`. I've also opened #8241 about fixing this generically. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310704732 ## File path: processing/src/main/java/org/apache/druid/query/filter/ColumnComparisonDimFilter.java ## @@ -40,14 +43,25 @@ private static final Joiner COMMA_JOINER = Joiner.on(", "); private final List dimensions; + private final FilterTuning filterTuning; Review comment: Please annotate with `@Nullable` everything that is needed in this class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310693686 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/sql/BloomFilterOperatorConversion.java ## @@ -114,6 +115,7 @@ public DimFilter toDruidFilter( return new BloomDimFilter( virtualColumn.getOutputName(), holder, + null, Review comment: And here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310694567 ## File path: processing/src/main/java/org/apache/druid/query/filter/BoundDimFilter.java ## @@ -56,6 +58,7 @@ private final Supplier longPredicateSupplier; private final Supplier floatPredicateSupplier; private final Supplier doublePredicateSupplier; + private final FilterTuning filterTuning; Review comment: Please annotate with `@Nullable` everything that needs to be `@Nullable` in this class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310692390 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -174,6 +191,13 @@ public ExtractionFn getExtractionFn() return extractionFn; } + @JsonInclude(JsonInclude.Include.NON_NULL) + @JsonProperty + public FilterTuning getFilterTuning() Review comment: This method should be `@Nullable` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310707102 ## File path: processing/src/main/java/org/apache/druid/query/filter/ExpressionDimFilter.java ## @@ -39,23 +41,42 @@ { private final String expression; private final Supplier parsed; + private final FilterTuning filterTuning; Review comment: Please annotate with `@Nullable` everything that is needed in this class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310695880 ## File path: processing/src/main/java/org/apache/druid/query/filter/BoundDimFilter.java ## @@ -263,43 +290,30 @@ public boolean equals(Object o) if (o == null || getClass() != o.getClass()) { return false; } - BoundDimFilter that = (BoundDimFilter) o; - -if (isLowerStrict() != that.isLowerStrict()) { - return false; -} -if (isUpperStrict() != that.isUpperStrict()) { - return false; -} -if (!getDimension().equals(that.getDimension())) { - return false; -} -if (getUpper() != null ? !getUpper().equals(that.getUpper()) : that.getUpper() != null) { - return false; -} -if (getLower() != null ? !getLower().equals(that.getLower()) : that.getLower() != null) { - return false; -} -if (getExtractionFn() != null -? !getExtractionFn().equals(that.getExtractionFn()) -: that.getExtractionFn() != null) { - return false; -} -return getOrdering().equals(that.getOrdering()); +return lowerStrict == that.lowerStrict && + upperStrict == that.upperStrict && + dimension.equals(that.dimension) && + Objects.equals(upper, that.upper) && + Objects.equals(lower, that.lower) && + Objects.equals(extractionFn, that.extractionFn) && + Objects.equals(ordering, that.ordering) && + Objects.equals(filterTuning, that.filterTuning); } @Override public int hashCode() { -int result = getDimension().hashCode(); -result = 31 * result + (getUpper() != null ? getUpper().hashCode() : 0); -result = 31 * result + (getLower() != null ? getLower().hashCode() : 0); -result = 31 * result + (isLowerStrict() ? 1 : 0); -result = 31 * result + (isUpperStrict() ? 1 : 0); -result = 31 * result + (getExtractionFn() != null ? getExtractionFn().hashCode() : 0); -result = 31 * result + getOrdering().hashCode(); -return result; +return Objects.hash( +dimension, +upper, +lower, +lowerStrict, +upperStrict, +extractionFn, +ordering, +filterTuning +); } @Override Review comment: Please add `filterTuning` in the `toString()` form. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310705538 ## File path: processing/src/main/java/org/apache/druid/query/Druids.java ## @@ -204,7 +204,7 @@ public TimeseriesQueryBuilder filters(String dimensionName, String value) public TimeseriesQueryBuilder filters(String dimensionName, String value, String... values) { - dimFilter = new InDimFilter(dimensionName, Lists.asList(value, values), null); + dimFilter = new InDimFilter(dimensionName, Lists.asList(value, values), null, null); Review comment: Could call the old constructor This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310706626 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -56,6 +61,17 @@ public BloomDimFilter( this.bloomKFilter = bloomKFilterHolder.getFilter(); this.hash = bloomKFilterHolder.getFilterHash(); this.extractionFn = extractionFn; +this.filterTuning = filterTuning; + } + + @VisibleForTesting + public BloomDimFilter( + String dimension, Review comment: Unnecessary wrapping, also in many other newly added constructors below in this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310693642 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/sql/BloomFilterOperatorConversion.java ## @@ -100,7 +100,8 @@ public DimFilter toDruidFilter( return new BloomDimFilter( druidExpression.getSimpleExtraction().getColumn(), holder, - druidExpression.getSimpleExtraction().getExtractionFn() + druidExpression.getSimpleExtraction().getExtractionFn(), Review comment: Constructor annotated `@VisibleForTesting` doesn't mean it can't be used in production code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310691637 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -42,12 +45,14 @@ private final BloomKFilter bloomKFilter; private final HashCode hash; private final ExtractionFn extractionFn; + private final FilterTuning filterTuning; Review comment: Should be `@Nullable`, along with the corresponding constructor parameter This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310695665 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -174,6 +191,13 @@ public ExtractionFn getExtractionFn() return extractionFn; } + @JsonInclude(JsonInclude.Include.NON_NULL) + @JsonProperty + public FilterTuning getFilterTuning() + { +return filterTuning; + } + @Override public String toString() Review comment: Please add `filterTuning` in the `toString()` form. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310705663 ## File path: processing/src/main/java/org/apache/druid/query/Druids.java ## @@ -361,7 +361,7 @@ public SearchQueryBuilder dataSource(DataSource d) public SearchQueryBuilder filters(String dimensionName, String value) { - dimFilter = new SelectorDimFilter(dimensionName, value, null); + dimFilter = new SelectorDimFilter(dimensionName, value, null, null); Review comment: Could call the old constructor This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
leventov commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310693015 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -193,16 +217,17 @@ public boolean equals(Object o) if (o == null || getClass() != o.getClass()) { return false; } - BloomDimFilter that = (BloomDimFilter) o; +return dimension.equals(that.dimension) && + hash.equals(that.hash) && + Objects.equals(extractionFn, that.extractionFn) && Review comment: Everything about `extractionFn`: the field, constructor parameter, getter should be `@Nullable` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #8126: [Proposal] BufferAggregator support for growable sketches.
leventov commented on issue #8126: [Proposal] BufferAggregator support for growable sketches. URL: https://github.com/apache/incubator-druid/issues/8126#issuecomment-518326114 > Then presenting a solution to this problem using Oak to manage the off-heap index, handling growable sketches using Memory, and showing that it performs as good as or better than the existing implementation would be a win-win-win solution, correct? Yes. I just want to warn you about trying to bring all of that at once - the extent of the change may become unbearable. So even if it incurs more work, it's better to make those changes separately: e. g. first, the transition to `Memory` (in the given subsystem), then - Oak. Doing it in the opposite order may be harder because Oak is already based on `Memory`, despite there is a method [`unsafeByteBufferView`](https://datasketches.github.io/api/memory/snapshot/apidocs/org/apache/datasketches/memory/Memory.html#unsafeByteBufferView-long-int-). > Any documentation on how we should configure the cluster to allow it running in off-heap mode? Not sure what do you mean by that. It's not possible to use the off-heap incremental index for indexing even when all aggregations are constant-sized - it's not implemented. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #8126: [Proposal] BufferAggregator support for growable sketches.
leventov commented on issue #8126: [Proposal] BufferAggregator support for growable sketches. URL: https://github.com/apache/incubator-druid/issues/8126#issuecomment-518327693 Currently, the off-heap incremental index is used only in GroupBy V1, which is itself deprecated and may not work at all. So I don't really even see a point in running benchmarks against the current off-heap incremental index to prove that Oak is better. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks
leventov commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks URL: https://github.com/apache/incubator-druid/pull/8236#discussion_r310723222 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java ## @@ -146,10 +146,13 @@ private final RetryPolicyFactory retryPolicyFactory; @JsonIgnore - private List indexTaskSpecs; + private final AppenderatorsManager appenderatorsManager; @JsonIgnore - private AppenderatorsManager appenderatorsManager; + private List indexTaskSpecs; + + @Nullable + private volatile IndexTask currentRunningTaskSpec = null; Review comment: https://github.com/code-review-checklists/java-concurrency#justify-volatile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] capistrant commented on a change in pull request #7562: Enable toggling request logging on/off for different query types
capistrant commented on a change in pull request #7562: Enable toggling request logging on/off for different query types URL: https://github.com/apache/incubator-druid/pull/7562#discussion_r310739354 ## File path: server/src/test/java/org/apache/druid/server/log/FilteredRequestLoggerTest.java ## @@ -106,6 +123,9 @@ public void testNotFilterAboveThreshold() throws IOException EasyMock.expect(nativeRequestLogLine.getQueryStats()) .andReturn(new QueryStats(ImmutableMap.of("query/time", 1000))) .once(); +EasyMock.expect(nativeRequestLogLine.getQuery()) Review comment: @leventov Does that wrapping indentation of 4 standard include lines such as these 3? I see that the lines above and below are had been using 8. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #7562: Enable toggling request logging on/off for different query types
leventov commented on a change in pull request #7562: Enable toggling request logging on/off for different query types URL: https://github.com/apache/incubator-druid/pull/7562#discussion_r310741593 ## File path: server/src/test/java/org/apache/druid/server/log/FilteredRequestLoggerTest.java ## @@ -106,6 +123,9 @@ public void testNotFilterAboveThreshold() throws IOException EasyMock.expect(nativeRequestLogLine.getQueryStats()) .andReturn(new QueryStats(ImmutableMap.of("query/time", 1000))) .once(); +EasyMock.expect(nativeRequestLogLine.getQuery()) Review comment: No. Here, the alignment is on the dot. It's a coincidence that "EasyMock" is 8 characters long. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6541: compaction tasks run for much time
stale[bot] commented on issue #6541: compaction tasks run for much time URL: https://github.com/apache/incubator-druid/issues/6541#issuecomment-518353565 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Eshcar commented on issue #8126: [Proposal] BufferAggregator support for growable sketches.
Eshcar commented on issue #8126: [Proposal] BufferAggregator support for growable sketches. URL: https://github.com/apache/incubator-druid/issues/8126#issuecomment-518356531 Thanks, that was our impression - that off-heap incremental index is not operational (however it does exist in the code). So, indeed there is no way to compare to it. I also agree that doing the oak-sketches-druid integration in one step might be too complicated. We already have an open issue #5698 and a PR #7676 for getting Oak incremental index into Druid and we hope to get progress there soon. Oak is not based on Memory , yet :). The context of my suggestion is how to support growable sketches off-heap while ingesting data in druid - namely, in the context of Oak. So the order should be first integrating Oak, then having Oak support growable sketches. If I now understand correctly the purpose of the current proposal is to handle the queries aggregation problem. Oak might be a solution also for this problem but this is something we haven't looked at yet. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (ac856fe -> 93cf9d4)
This is an automated email from the ASF dual-hosted git repository. jonwei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from ac856fe Web-Console: Adds edit-context-dialog (#8228) add 93cf9d4 SQL support for t-digest based sketch aggregators (#8100) No new revisions were added by this update. Summary of changes: .../extensions-contrib/tdigestsketch-quantiles.md | 86 ++-- docs/content/querying/sql.md | 2 + extensions-contrib/tdigestsketch/pom.xml | 84 +++- .../TDigestMergeSketchAggregator.java | 83 .../TDigestMergeSketchAggregatorFactory.java | 64 --- .../TDigestMergeSketchBufferAggregator.java| 110 - ...ggregator.java => TDigestSketchAggregator.java} | 12 +- ...ry.java => TDigestSketchAggregatorFactory.java} | 37 +- ...tor.java => TDigestSketchBufferAggregator.java} | 8 +- .../TDigestSketchComplexMetricSerde.java | 2 +- .../tdigestsketch/TDigestSketchModule.java | 22 +- .../tdigestsketch/TDigestSketchObjectStrategy.java | 2 +- ... => TDigestSketchToQuantilePostAggregator.java} | 68 ++-- .../TDigestSketchToQuantilesPostAggregator.java| 6 +- .../tdigestsketch/TDigestSketchUtils.java | 33 ++ .../sql/TDigestGenerateSketchSqlAggregator.java| 183 + .../sql/TDigestSketchQuantileSqlAggregator.java| 112 ++ .../tdigestsketch/TDigestSketchAggregatorTest.java | 28 +- .../sql/TDigestSketchSqlAggregatorTest.java| 447 + .../druid/query/aggregation/AggregatorUtil.java| 2 - .../query/aggregation/post/PostAggregatorIds.java | 2 + 21 files changed, 907 insertions(+), 486 deletions(-) delete mode 100644 extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestMergeSketchAggregator.java delete mode 100644 extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestMergeSketchAggregatorFactory.java delete mode 100644 extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestMergeSketchBufferAggregator.java rename extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/{TDigestBuildSketchAggregator.java => TDigestSketchAggregator.java} (85%) rename extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/{TDigestBuildSketchAggregatorFactory.java => TDigestSketchAggregatorFactory.java} (79%) rename extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/{TDigestBuildSketchBufferAggregator.java => TDigestSketchBufferAggregator.java} (93%) copy extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/{TDigestSketchToQuantilesPostAggregator.java => TDigestSketchToQuantilePostAggregator.java} (67%) create mode 100644 extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestGenerateSketchSqlAggregator.java copy extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/sql/DoublesSketchSqlAggregator.java => extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestSketchQuantileSqlAggregator.java (61%) create mode 100644 extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestSketchSqlAggregatorTest.java - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei merged pull request #8100: SQL support for t-digest based sketch aggregators
jon-wei merged pull request #8100: SQL support for t-digest based sketch aggregators URL: https://github.com/apache/incubator-druid/pull/8100 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310748843 ## File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java ## @@ -123,6 +123,17 @@ private Closer() return closeable; } + /** + * Registers a list of {@code closeable} to be closed when this {@code Closer} is + * {@linkplain #close closed}. + */ + // close. this word no longer has any meaning to me. Review comment: This is copy-paste. It is already in current master :-) https://github.com/apache/incubator-druid/blob/master/core/src/main/java/org/apache/druid/java/util/common/io/Closer.java#L116 I'll remove it anyway 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310749285 ## File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java ## @@ -123,6 +123,17 @@ private Closer() return closeable; } + /** + * Registers a list of {@code closeable} to be closed when this {@code Closer} is + * {@linkplain #close closed}. + */ + // close. this word no longer has any meaning to me. + public Iterable register(Iterable closeables) Review comment: Good suggestion, done! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310749986 ## File path: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java ## @@ -123,6 +123,17 @@ private Closer() return closeable; } + /** + * Registers a list of {@code closeable} to be closed when this {@code Closer} is Review comment: Makes sense. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310750268 ## File path: processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java ## @@ -280,12 +280,17 @@ public AggregateResult call() return RowBasedGrouperHelper.makeGrouperIterator( grouper, query, - () -> Lists.reverse(resources).forEach(CloseQuietly::close) + resources ); } catch (Throwable e) { Review comment: Done 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310750551 ## File path: processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java ## @@ -280,12 +280,17 @@ public AggregateResult call() return RowBasedGrouperHelper.makeGrouperIterator( grouper, query, - () -> Lists.reverse(resources).forEach(CloseQuietly::close) + resources ); } catch (Throwable e) { // Exception caught while setting up the iterator; release resources. - Lists.reverse(resources).forEach(CloseQuietly::close); + try { +resources.close(); + } + catch (IOException ex) { Review comment: This can't be weaker since it is being thrown by the `.close()` method: https://github.com/apache/incubator-druid/blob/master/core/src/main/java/org/apache/druid/java/util/common/io/Closer.java#L155 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug opened a new issue #8242: should we deprecate extractionFn field in various Aggregators and Filters
himanshug opened a new issue #8242: should we deprecate extractionFn field in various Aggregators and Filters URL: https://github.com/apache/incubator-druid/issues/8242 they were added before `VirtualColumn` was introduced and I think a better alternative now could be to have a extractionFn based `VirtualColumn` that could be used. That way, all the filter and aggregator implementations don't need to handle them in different places. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm closed issue #7093: [PROPOSAL] Query vectorization
gianm closed issue #7093: [PROPOSAL] Query vectorization URL: https://github.com/apache/incubator-druid/issues/7093 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #7093: [PROPOSAL] Query vectorization
gianm commented on issue #7093: [PROPOSAL] Query vectorization URL: https://github.com/apache/incubator-druid/issues/7093#issuecomment-518383739 Closed by #6794. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug opened a new pull request #8243: make double sum/min/max agg work on string columns
himanshug opened a new pull request #8243: make double sum/min/max agg work on string columns URL: https://github.com/apache/incubator-druid/pull/8243 Partial work towards #8148 . There would be follow up PR to do same for other core aggregators once this is merged. ### Description This patch adds handling of single/multi value column handling by double sum/min/max aggregators to do a best effort parsing string as double. `StringColumnDoubleAggregatorWrapper` and `StringColumnDoubleBufferAggregatorWrapper` classes are introduced that can wrap existing double aggregators to handle string columns. Both of the classes are used by `SimpleDoubleAggregatorFactory` to be used when input column is known to be of String type. Currently it does not work if `extractionFn` instead of `fieldName` is provided. (Related #8242 ) This PR has: - [x] been self-reviewed. - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [x] added unit tests or modified existing tests to cover new code paths. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8109: fix merging of groupBy subtotal spec results
himanshug commented on issue #8109: fix merging of groupBy subtotal spec results URL: https://github.com/apache/incubator-druid/pull/8109#issuecomment-518385521 @gianm can you please review it again ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #8242: should we deprecate extractionFn field in various Aggregators and Filters
clintropolis commented on issue #8242: should we deprecate extractionFn field in various Aggregators and Filters URL: https://github.com/apache/incubator-druid/issues/8242#issuecomment-518386455 sgtm :+1: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8126: [Proposal] BufferAggregator support for growable sketches.
himanshug commented on issue #8126: [Proposal] BufferAggregator support for growable sketches. URL: https://github.com/apache/incubator-druid/issues/8126#issuecomment-518386793 > If I now understand correctly the purpose of the current proposal is to handle the queries aggregation problem. Oak might be a solution also for this problem but this is something we haven't looked at yet. yes, the context here applies to every place aggregation happens and also let it not be dependent on something specific but allow extensions to eventually add optimal behavior. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #8115: Add shuffleSegmentPusher for data shuffle
jihoonson commented on issue #8115: Add shuffleSegmentPusher for data shuffle URL: https://github.com/apache/incubator-druid/pull/8115#issuecomment-518389528 @himanshug do you have more comments? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug merged pull request #8115: Add shuffleSegmentPusher for data shuffle
himanshug merged pull request #8115: Add shuffleSegmentPusher for data shuffle URL: https://github.com/apache/incubator-druid/pull/8115 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Add shuffleSegmentPusher for data shuffle (#8115)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new ab5b3be Add shuffleSegmentPusher for data shuffle (#8115) ab5b3be is described below commit ab5b3be6c61d7be540f85385327a6a6489b363b1 Author: Jihoon Son AuthorDate: Mon Aug 5 13:38:35 2019 -0700 Add shuffleSegmentPusher for data shuffle (#8115) * Fix race between canHandle() and addSegment() in StorageLocation * add comment * Add shuffleSegmentPusher which is a dataSegmentPusher used for writing shuffle data in local storage. * add comments * unused import * add comments * fix test * address comments * remove tag from javadoc * address comments * comparingLong * Address comments * fix test --- .../druid/indexing/common/config/TaskConfig.java | 5 + .../indexing/worker/IntermediaryDataManager.java | 166 + .../indexing/worker/ShuffleDataSegmentPusher.java | 77 ++ .../IntermediaryDataManagerAutoCleanupTest.java| 15 +- ...ermediaryDataManagerManualAddAndDeleteTest.java | 47 +++--- .../worker/ShuffleDataSegmentPusherTest.java | 138 + .../druid/segment/loading/StorageLocation.java | 7 +- ...mentLoaderLocalCacheManagerConcurrencyTest.java | 2 - .../druid/segment/loading/StorageLocationTest.java | 14 +- 9 files changed, 366 insertions(+), 105 deletions(-) diff --git a/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java b/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java index 31405fe..52bf083 100644 --- a/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java +++ b/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java @@ -126,6 +126,11 @@ public class TaskConfig return new File(getTaskDir(taskId), "work"); } + public File getTaskTempDir(String taskId) + { +return new File(getTaskDir(taskId), "temp"); + } + public File getTaskLockFile(String taskId) { return new File(getTaskDir(taskId), "lock"); diff --git a/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java b/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java index c47425a..95acb22 100644 --- a/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java +++ b/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java @@ -33,11 +33,13 @@ import org.apache.druid.java.util.common.DateTimes; import org.apache.druid.java.util.common.IOE; import org.apache.druid.java.util.common.ISE; import org.apache.druid.java.util.common.concurrent.Execs; +import org.apache.druid.java.util.common.io.Closer; import org.apache.druid.java.util.common.lifecycle.LifecycleStart; import org.apache.druid.java.util.common.lifecycle.LifecycleStop; import org.apache.druid.java.util.common.logger.Logger; import org.apache.druid.segment.loading.StorageLocation; import org.apache.druid.timeline.DataSegment; +import org.apache.druid.utils.CompressionUtils; import org.joda.time.DateTime; import org.joda.time.Interval; import org.joda.time.Period; @@ -45,6 +47,7 @@ import org.joda.time.Period; import javax.annotation.Nullable; import java.io.File; import java.io.IOException; +import java.nio.file.Path; import java.nio.file.Paths; import java.util.Arrays; import java.util.Collections; @@ -57,12 +60,14 @@ import java.util.Map.Entry; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ThreadLocalRandom; import java.util.concurrent.TimeUnit; import java.util.stream.Collectors; +import java.util.stream.IntStream; /** * This class manages intermediary segments for data shuffle between native parallel index tasks. - * In native parallel indexing, phase 1 tasks store segment files in local storage of middleManagers + * In native parallel indexing, phase 1 tasks store segment files in local storage of middleManagers (or indexer) * and phase 2 tasks read those files via HTTP. * * The directory where segment files are placed is structured as @@ -75,11 +80,12 @@ import java.util.stream.Collectors; @ManageLifecycle public class IntermediaryDataManager { - private static final Logger log = new Logger(IntermediaryDataManager.class); + private static final Logger LOG = new Logger(IntermediaryDataManager.class); private final long intermediaryPartitionDiscoveryPeriodSec; private final long intermediaryPartitionCleanupPeriodSec; private final Period intermediaryPartitionTimeout; + priv
[GitHub] [incubator-druid] himanshug closed issue #8054: Issue while using Docker build
himanshug closed issue #8054: Issue while using Docker build URL: https://github.com/apache/incubator-druid/issues/8054 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug merged pull request #8237: Update docker build
himanshug merged pull request #8237: Update docker build URL: https://github.com/apache/incubator-druid/pull/8237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests
jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests URL: https://github.com/apache/incubator-druid/pull/8222#discussion_r310785662 ## File path: core/src/main/java/org/apache/druid/timeline/partition/OvershadowableManager.java ## @@ -189,89 +247,359 @@ private void transitAtomicUpdateGroupState(AtomicUpdateGroup atomicUpdateGrou } /** - * Find all atomicUpdateGroups of the given state overshadowed by the given rootPartitionRange and minorVersion. + * Find all atomicUpdateGroups of the given state overshadowed by the minorVersion in the given rootPartitionRange. * The atomicUpdateGroup of a higher minorVersion can have a wider RootPartitionRange. * To find all atomicUpdateGroups overshadowed by the given rootPartitionRange and minorVersion, * we first need to find the first key contained by the given rootPartitionRange. * Once we find such key, then we go through the entire map until we see an atomicUpdateGroup of which * rootRangePartition is not contained by the given rootPartitionRange. + * + * @param rangeOfAug the partition range to search for overshadowed groups. + * @param minorVersion the minor version to check overshadow relation. The found groups will have lower minor versions + * than this. + * @param fromStatethe state to search for overshadowed groups. + * + * @return a list of found atomicUpdateGroups. It could be empty if no groups are found. + */ + @VisibleForTesting + List> findOvershadowedBy(RootPartitionRange rangeOfAug, short minorVersion, State fromState) + { +final TreeMap>> stateMap = getStateMap(fromState); +Entry>> current = findLowestOverlappingEntry( +rangeOfAug, +stateMap, +true +); + +if (current == null) { + return Collections.emptyList(); +} + +// Going through the map to find all entries of the RootPartitionRange contained by the given rangeOfAug. +// Note that RootPartitionRange of entries are always consecutive. +final List> found = new ArrayList<>(); +while (current != null && rangeOfAug.overlaps(current.getKey())) { + if (rangeOfAug.contains(current.getKey())) { +// versionToGroup is sorted by minorVersion. +// versionToGroup.headMap(minorVersion) below returns a map containing all entries of lower minorVersions +// than the given minorVersion. +final Short2ObjectSortedMap> versionToGroup = current.getValue(); +// Short2ObjectRBTreeMap.SubMap.short2ObjectEntrySet() implementation, especially size(), is not optimized. +// Note that size() is indirectly called in ArrayList.addAll() when ObjectSortedSet.toArray() is called. +// See AbstractObjectCollection.toArray(). +// If you see performance degradation here, probably we need to improve the below line. +if (versionToGroup.firstShortKey() < minorVersion) { + found.addAll(versionToGroup.headMap(minorVersion).values()); +} + } + current = stateMap.higherEntry(current.getKey()); +} +return found; + } + + private List> findOvershadows(AtomicUpdateGroup aug, State fromState) + { +return findOvershadows(RootPartitionRange.of(aug), aug.getMinorVersion(), fromState); + } + + /** + * Find all atomicUpdateGroups which overshadow others of the given minorVersion in the given rootPartitionRange. + * Similar to {@link #findOvershadowedBy}. + * + * Note that one atommicUpdateGroup can overshadow multiple other groups. If you're finding overshadowing + * atomicUpdateGroups by calling this method in a loop, the results of this method can contain duplicate groups. + * + * @param rangeOfAug the partition range to search for overshadowing groups. + * @param minorVersion the minor version to check overshadow relation. The found groups will have higher minor + * versions than this. + * @param fromStatethe state to search for overshadowed groups. + * + * @return a list of found atomicUpdateGroups. It could be empty if no groups are found. */ - private List>> findOvershadowedBy( + @VisibleForTesting + List> findOvershadows(RootPartitionRange rangeOfAug, short minorVersion, State fromState) + { +final TreeMap>> stateMap = getStateMap(fromState); +Entry>> current = findLowestOverlappingEntry( +rangeOfAug, +stateMap, +false +); + +if (current == null) { + return Collections.emptyList(); +} + +// Going through the map to find all entries of the RootPartitionRange contains the given rangeOfAug. +// Note that RootPartitionRange of entries are always consecutive. +final List> found = new ArrayList<>(); +while (current != null && current.getKey().overlaps(rangeOfAug)) { + if (current.getKey().contains(rangeOfAug)) { +// versionToGroup is sorted by minorV
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests
jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests URL: https://github.com/apache/incubator-druid/pull/8222#discussion_r310785630 ## File path: core/src/main/java/org/apache/druid/timeline/partition/OvershadowableManager.java ## @@ -189,89 +247,359 @@ private void transitAtomicUpdateGroupState(AtomicUpdateGroup atomicUpdateGrou } /** - * Find all atomicUpdateGroups of the given state overshadowed by the given rootPartitionRange and minorVersion. + * Find all atomicUpdateGroups of the given state overshadowed by the minorVersion in the given rootPartitionRange. * The atomicUpdateGroup of a higher minorVersion can have a wider RootPartitionRange. * To find all atomicUpdateGroups overshadowed by the given rootPartitionRange and minorVersion, * we first need to find the first key contained by the given rootPartitionRange. * Once we find such key, then we go through the entire map until we see an atomicUpdateGroup of which * rootRangePartition is not contained by the given rootPartitionRange. + * + * @param rangeOfAug the partition range to search for overshadowed groups. + * @param minorVersion the minor version to check overshadow relation. The found groups will have lower minor versions + * than this. + * @param fromStatethe state to search for overshadowed groups. + * + * @return a list of found atomicUpdateGroups. It could be empty if no groups are found. + */ + @VisibleForTesting + List> findOvershadowedBy(RootPartitionRange rangeOfAug, short minorVersion, State fromState) + { +final TreeMap>> stateMap = getStateMap(fromState); +Entry>> current = findLowestOverlappingEntry( +rangeOfAug, +stateMap, +true +); + +if (current == null) { + return Collections.emptyList(); +} + +// Going through the map to find all entries of the RootPartitionRange contained by the given rangeOfAug. +// Note that RootPartitionRange of entries are always consecutive. +final List> found = new ArrayList<>(); +while (current != null && rangeOfAug.overlaps(current.getKey())) { + if (rangeOfAug.contains(current.getKey())) { +// versionToGroup is sorted by minorVersion. +// versionToGroup.headMap(minorVersion) below returns a map containing all entries of lower minorVersions +// than the given minorVersion. +final Short2ObjectSortedMap> versionToGroup = current.getValue(); +// Short2ObjectRBTreeMap.SubMap.short2ObjectEntrySet() implementation, especially size(), is not optimized. +// Note that size() is indirectly called in ArrayList.addAll() when ObjectSortedSet.toArray() is called. +// See AbstractObjectCollection.toArray(). +// If you see performance degradation here, probably we need to improve the below line. +if (versionToGroup.firstShortKey() < minorVersion) { + found.addAll(versionToGroup.headMap(minorVersion).values()); +} + } + current = stateMap.higherEntry(current.getKey()); +} +return found; + } + + private List> findOvershadows(AtomicUpdateGroup aug, State fromState) + { +return findOvershadows(RootPartitionRange.of(aug), aug.getMinorVersion(), fromState); + } + + /** + * Find all atomicUpdateGroups which overshadow others of the given minorVersion in the given rootPartitionRange. + * Similar to {@link #findOvershadowedBy}. + * + * Note that one atommicUpdateGroup can overshadow multiple other groups. If you're finding overshadowing + * atomicUpdateGroups by calling this method in a loop, the results of this method can contain duplicate groups. + * + * @param rangeOfAug the partition range to search for overshadowing groups. + * @param minorVersion the minor version to check overshadow relation. The found groups will have higher minor + * versions than this. + * @param fromStatethe state to search for overshadowed groups. + * + * @return a list of found atomicUpdateGroups. It could be empty if no groups are found. */ - private List>> findOvershadowedBy( + @VisibleForTesting + List> findOvershadows(RootPartitionRange rangeOfAug, short minorVersion, State fromState) + { +final TreeMap>> stateMap = getStateMap(fromState); +Entry>> current = findLowestOverlappingEntry( +rangeOfAug, +stateMap, +false +); + +if (current == null) { + return Collections.emptyList(); +} + +// Going through the map to find all entries of the RootPartitionRange contains the given rangeOfAug. +// Note that RootPartitionRange of entries are always consecutive. +final List> found = new ArrayList<>(); +while (current != null && current.getKey().overlaps(rangeOfAug)) { + if (current.getKey().contains(rangeOfAug)) { +// versionToGroup is sorted by minorV
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests
jihoonson commented on a change in pull request #8222: Fix bugs in overshadowableManager and add unit tests URL: https://github.com/apache/incubator-druid/pull/8222#discussion_r310785647 ## File path: core/src/main/java/org/apache/druid/timeline/partition/OvershadowableManager.java ## @@ -189,89 +247,359 @@ private void transitAtomicUpdateGroupState(AtomicUpdateGroup atomicUpdateGrou } /** - * Find all atomicUpdateGroups of the given state overshadowed by the given rootPartitionRange and minorVersion. + * Find all atomicUpdateGroups of the given state overshadowed by the minorVersion in the given rootPartitionRange. * The atomicUpdateGroup of a higher minorVersion can have a wider RootPartitionRange. * To find all atomicUpdateGroups overshadowed by the given rootPartitionRange and minorVersion, * we first need to find the first key contained by the given rootPartitionRange. * Once we find such key, then we go through the entire map until we see an atomicUpdateGroup of which * rootRangePartition is not contained by the given rootPartitionRange. + * + * @param rangeOfAug the partition range to search for overshadowed groups. + * @param minorVersion the minor version to check overshadow relation. The found groups will have lower minor versions + * than this. + * @param fromStatethe state to search for overshadowed groups. + * + * @return a list of found atomicUpdateGroups. It could be empty if no groups are found. + */ + @VisibleForTesting + List> findOvershadowedBy(RootPartitionRange rangeOfAug, short minorVersion, State fromState) + { +final TreeMap>> stateMap = getStateMap(fromState); +Entry>> current = findLowestOverlappingEntry( +rangeOfAug, +stateMap, +true +); + +if (current == null) { + return Collections.emptyList(); +} + +// Going through the map to find all entries of the RootPartitionRange contained by the given rangeOfAug. +// Note that RootPartitionRange of entries are always consecutive. +final List> found = new ArrayList<>(); +while (current != null && rangeOfAug.overlaps(current.getKey())) { + if (rangeOfAug.contains(current.getKey())) { +// versionToGroup is sorted by minorVersion. +// versionToGroup.headMap(minorVersion) below returns a map containing all entries of lower minorVersions +// than the given minorVersion. +final Short2ObjectSortedMap> versionToGroup = current.getValue(); +// Short2ObjectRBTreeMap.SubMap.short2ObjectEntrySet() implementation, especially size(), is not optimized. +// Note that size() is indirectly called in ArrayList.addAll() when ObjectSortedSet.toArray() is called. +// See AbstractObjectCollection.toArray(). +// If you see performance degradation here, probably we need to improve the below line. +if (versionToGroup.firstShortKey() < minorVersion) { + found.addAll(versionToGroup.headMap(minorVersion).values()); +} + } + current = stateMap.higherEntry(current.getKey()); +} +return found; + } + + private List> findOvershadows(AtomicUpdateGroup aug, State fromState) + { +return findOvershadows(RootPartitionRange.of(aug), aug.getMinorVersion(), fromState); + } + + /** + * Find all atomicUpdateGroups which overshadow others of the given minorVersion in the given rootPartitionRange. + * Similar to {@link #findOvershadowedBy}. + * + * Note that one atommicUpdateGroup can overshadow multiple other groups. If you're finding overshadowing + * atomicUpdateGroups by calling this method in a loop, the results of this method can contain duplicate groups. + * + * @param rangeOfAug the partition range to search for overshadowing groups. + * @param minorVersion the minor version to check overshadow relation. The found groups will have higher minor + * versions than this. + * @param fromStatethe state to search for overshadowed groups. + * + * @return a list of found atomicUpdateGroups. It could be empty if no groups are found. */ - private List>> findOvershadowedBy( + @VisibleForTesting + List> findOvershadows(RootPartitionRange rangeOfAug, short minorVersion, State fromState) + { +final TreeMap>> stateMap = getStateMap(fromState); +Entry>> current = findLowestOverlappingEntry( +rangeOfAug, +stateMap, +false +); + +if (current == null) { + return Collections.emptyList(); +} + +// Going through the map to find all entries of the RootPartitionRange contains the given rangeOfAug. +// Note that RootPartitionRange of entries are always consecutive. +final List> found = new ArrayList<>(); +while (current != null && current.getKey().overlaps(rangeOfAug)) { + if (current.getKey().contains(rangeOfAug)) { +// versionToGroup is sorted by minorV
[GitHub] [incubator-druid] jihoonson commented on issue #8115: Add shuffleSegmentPusher for data shuffle
jihoonson commented on issue #8115: Add shuffleSegmentPusher for data shuffle URL: https://github.com/apache/incubator-druid/pull/8115#issuecomment-518394084 @himanshug @clintropolis thank you for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #5872: add method getRequiredColumns for DimFilter
clintropolis commented on a change in pull request #5872: add method getRequiredColumns for DimFilter URL: https://github.com/apache/incubator-druid/pull/5872#discussion_r310786713 ## File path: processing/src/main/java/io/druid/query/filter/DimFilter.java ## @@ -75,4 +77,9 @@ * determine for this DimFilter. */ RangeSet getDimensionRangeSet(String dimension); + + /** + * @return a HashSet that represents all columns' name which the DimFilter required to do filter. + */ + HashSet getRequiredColumns(); Review comment: I'm making this change to use `Set` as part of #8209. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ilhanadiyaman opened a new pull request #8244: Update docker build
ilhanadiyaman opened a new pull request #8244: Update docker build URL: https://github.com/apache/incubator-druid/pull/8244 Sorry for opening again. My master branch was messed up. Fixes #8054 ### Description - [License generation scripts](https://github.com/apache/incubator-druid/blob/master/docs/_bin/generate-license.py) requires python3 and pip3. Python3 and pip3 installation added to docker build. - Docker entrypoint script changed in respond to the new directory structure #7590 - Dockerfile updated to allow dependency caching in multi-stage build This PR has: - [x] been self-reviewed. - [x] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ilhanadiyaman commented on issue #8244: Update docker build
ilhanadiyaman commented on issue #8244: Update docker build URL: https://github.com/apache/incubator-druid/pull/8244#issuecomment-518397727 @himanshug I am really sorry for opening this PR again. My forked master branch was messed up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8227: Web-Console: change go to sql button to more button
vogievetsky commented on issue #8227: Web-Console: change go to sql button to more button URL: https://github.com/apache/incubator-druid/pull/8227#issuecomment-518401428 Thank you for addressing my feedback, this is good to merge IMO 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #8235: Use Closer instead of List
leventov commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310793741 ## File path: processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java ## @@ -280,12 +280,17 @@ public AggregateResult call() return RowBasedGrouperHelper.makeGrouperIterator( grouper, query, - () -> Lists.reverse(resources).forEach(CloseQuietly::close) + resources ); } catch (Throwable e) { // Exception caught while setting up the iterator; release resources. - Lists.reverse(resources).forEach(CloseQuietly::close); + try { +resources.close(); + } + catch (IOException ex) { Review comment: Why it can't be weaker? You can catch a weaker Exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm merged pull request #8227: Web-Console: change go to sql button to more button
gianm merged pull request #8227: Web-Console: change go to sql button to more button URL: https://github.com/apache/incubator-druid/pull/8227 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Web-Console: change go to sql button to more button (#8227)
This is an automated email from the ASF dual-hosted git repository. gian pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new 5d0805d Web-Console: change go to sql button to more button (#8227) 5d0805d is described below commit 5d0805dd48890b2a22148ab009d9bdfbf92416d1 Author: mcbrewster <37322608+mcbrews...@users.noreply.github.com> AuthorDate: Mon Aug 5 14:12:16 2019 -0700 Web-Console: change go to sql button to more button (#8227) * change go to sql button * rename to See in SQL view * update snapshots --- .../__snapshots__/datasource-view.spec.tsx.snap| 42 +++-- .../src/views/datasource-view/datasource-view.tsx | 45 ++ .../__snapshots__/segments-view.spec.tsx.snap | 43 ++--- .../src/views/segments-view/segments-view.tsx | 55 -- .../__snapshots__/servers-view.spec.tsx.snap | 42 +++-- .../src/views/servers-view/servers-view.tsx| 44 + .../__snapshots__/tasks-view.spec.tsx.snap | 42 +++-- web-console/src/views/task-view/tasks-view.tsx | 34 + 8 files changed, 286 insertions(+), 61 deletions(-) diff --git a/web-console/src/views/datasource-view/__snapshots__/datasource-view.spec.tsx.snap b/web-console/src/views/datasource-view/__snapshots__/datasource-view.spec.tsx.snap index 9162516..0e4bc80 100755 --- a/web-console/src/views/datasource-view/__snapshots__/datasource-view.spec.tsx.snap +++ b/web-console/src/views/datasource-view/__snapshots__/datasource-view.spec.tsx.snap @@ -11,11 +11,43 @@ exports[`data source view matches snapshot 1`] = ` localStorageKey="datasources-refresh-rate" onRefresh={[Function]} /> - + + + + } + defaultIsOpen={false} + disabled={false} + fill={false} + hasBackdrop={false} + hoverCloseDelay={300} + hoverOpenDelay={150} + inheritDarkTheme={true} + interactionKind="click" + minimal={false} + modifiers={Object {}} + openOnTargetFocus={true} + position="bottom-left" + targetTagName="span" + transitionDuration={300} + usePortal={true} + wrapperTagName="span" +> + + +{!noSqlMode && ( + goToQuery(DatasourcesView.DATASOURCE_SQL)} + /> +)} + +); + +return ( + <> + + + + +); + } + private saveRules = async (datasource: string, rules: any[], comment: string) => { try { await axios.post(`/druid/coordinator/v1/rules/${datasource}`, rules, { @@ -901,7 +934,7 @@ GROUP BY 1`; } render(): JSX.Element { -const { goToQuery, noSqlMode } = this.props; +const { noSqlMode } = this.props; const { showDisabled, hiddenColumns, showChart, chartHeight, chartWidth } = this.state; return ( @@ -913,13 +946,7 @@ GROUP BY 1`; }} localStorageKey={LocalStorageKeys.DATASOURCES_REFRESH_RATE} /> - {!noSqlMode && ( - goToQuery(DatasourcesView.DATASOURCE_SQL)} -/> - )} + {this.renderBulkDatasourceActions()} - + + + +} +defaultIsOpen={false} +disabled={false} +fill={false} +hasBackdrop={false} +hoverCloseDelay={300} +hoverOpenDelay={150} +inheritDarkTheme={true} +interactionKind="click" +minimal={false} +modifiers={Object {}} +openOnTargetFocus={true} +position="bottom-left" +targetTagName="span" +transitionDuration={300} +usePortal={true} +wrapperTagName="span" + > + + Group by diff --git a/web-console/src/views/segments-view/segments-view.tsx b/web-console/src/views/segments-view/segments-view.tsx index 18583ff..6e10b68 100644 --- a/web-console/src/views/segments-view/segments-view.tsx +++ b/web-console/src/views/segments-view/segments-view.tsx @@ -16,7 +16,16 @@ * limitations under the License. */ -import { Button, ButtonGroup, Intent, Label } from '@blueprintjs/core'; +import { + Button, + ButtonGroup, + Intent, + Label, + Menu, + MenuItem, + Popover, + Position, +} from '@blueprintjs/core'; import { IconNames } from '@blueprintjs/icons'; import axios from 'axios'; import React from 'react'; @@ -607,6 +616,35 @@ export class SegmentsView extends React.PureComponent +{!noSqlMode && ( + { + if (!lastSegmentsQuery) return; + goToQuery(lastSegmentsQuery); +}} + /> +)} + +); + +return ( + <> + + + + +); + } + render(): JSX.Element { const { segme
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8235: Use Closer instead of List
Fokko commented on a change in pull request #8235: Use Closer instead of List URL: https://github.com/apache/incubator-druid/pull/8235#discussion_r310802823 ## File path: processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java ## @@ -280,12 +280,17 @@ public AggregateResult call() return RowBasedGrouperHelper.makeGrouperIterator( grouper, query, - () -> Lists.reverse(resources).forEach(CloseQuietly::close) + resources ); } catch (Throwable e) { // Exception caught while setting up the iterator; release resources. - Lists.reverse(resources).forEach(CloseQuietly::close); + try { +resources.close(); + } + catch (IOException ex) { Review comment: Ah like that, yes that's perfectly possible 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7648: Handle exceptions while fetching offsets
stale[bot] commented on issue #7648: Handle exceptions while fetching offsets URL: https://github.com/apache/incubator-druid/pull/7648#issuecomment-518413793 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310807921 ## File path: processing/src/main/java/org/apache/druid/query/filter/BoundDimFilter.java ## @@ -263,43 +290,30 @@ public boolean equals(Object o) if (o == null || getClass() != o.getClass()) { return false; } - BoundDimFilter that = (BoundDimFilter) o; - -if (isLowerStrict() != that.isLowerStrict()) { - return false; -} -if (isUpperStrict() != that.isUpperStrict()) { - return false; -} -if (!getDimension().equals(that.getDimension())) { - return false; -} -if (getUpper() != null ? !getUpper().equals(that.getUpper()) : that.getUpper() != null) { - return false; -} -if (getLower() != null ? !getLower().equals(that.getLower()) : that.getLower() != null) { - return false; -} -if (getExtractionFn() != null -? !getExtractionFn().equals(that.getExtractionFn()) -: that.getExtractionFn() != null) { - return false; -} -return getOrdering().equals(that.getOrdering()); +return lowerStrict == that.lowerStrict && + upperStrict == that.upperStrict && + dimension.equals(that.dimension) && + Objects.equals(upper, that.upper) && + Objects.equals(lower, that.lower) && + Objects.equals(extractionFn, that.extractionFn) && + Objects.equals(ordering, that.ordering) && + Objects.equals(filterTuning, that.filterTuning); } @Override public int hashCode() { -int result = getDimension().hashCode(); -result = 31 * result + (getUpper() != null ? getUpper().hashCode() : 0); -result = 31 * result + (getLower() != null ? getLower().hashCode() : 0); -result = 31 * result + (isLowerStrict() ? 1 : 0); -result = 31 * result + (isUpperStrict() ? 1 : 0); -result = 31 * result + (getExtractionFn() != null ? getExtractionFn().hashCode() : 0); -result = 31 * result + getOrdering().hashCode(); -return result; +return Objects.hash( +dimension, +upper, +lower, +lowerStrict, +upperStrict, +extractionFn, +ordering, +filterTuning +); } @Override Review comment: I'm not exactly sure the best way to add `filterTuning` to the `toString` output of these `DimFilter` with non-standard implementations, I guess a parenthesis maybe? e.g. `(filterTuning=...` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310810977 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/sql/BloomFilterOperatorConversion.java ## @@ -100,7 +100,8 @@ public DimFilter toDruidFilter( return new BloomDimFilter( druidExpression.getSimpleExtraction().getColumn(), holder, - druidExpression.getSimpleExtraction().getExtractionFn() + druidExpression.getSimpleExtraction().getExtractionFn(), Review comment: Yeah, this is true, but I was trying to avoid using these constructors in production code to minimize the chance of errors and make things more obvious. I initially had it more like you are suggesting and was using some of the 'test' constructors where we were always passing null, but [this comment](https://github.com/apache/incubator-druid/pull/8209#discussion_r310319858) suggested the way it is now and I agree, I think the production code explicitly passing in `null` for the `filterTuning` parameter makes it less ambiguous about if not having a tuning is intentional or not, without requiring a comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310811872 ## File path: processing/src/main/java/org/apache/druid/query/filter/ColumnComparisonDimFilter.java ## @@ -114,10 +141,4 @@ public boolean equals(Object o) .collect(Collectors.toSet()) Review comment: I've changed the `DimFilter` interface to use `Set` instead of `HashSet` and adjusted this to no longer dupe the set, along with changing single dimension filters to use `ImmutableSet.of` instead of `Sets.newHashSet` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310812077 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -42,12 +45,14 @@ private final BloomKFilter bloomKFilter; private final HashCode hash; private final ExtractionFn extractionFn; + private final FilterTuning filterTuning; Review comment: I think I've updated all `DimFilter` implementations to have the appropriate things annotated with `@Nullable` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8231: Web console: celebrate array based groupBy by supporting resultAsArray in the console
vogievetsky commented on issue #8231: Web console: celebrate array based groupBy by supporting resultAsArray in the console URL: https://github.com/apache/incubator-druid/pull/8231#issuecomment-518435704 Ok updated to address @fjy 's comments. Also updated the licenses that went out of date after the segment timeline PR was merged in. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
clintropolis commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310830104 ## File path: processing/src/main/java/org/apache/druid/query/filter/BoundDimFilter.java ## @@ -263,43 +290,30 @@ public boolean equals(Object o) if (o == null || getClass() != o.getClass()) { return false; } - BoundDimFilter that = (BoundDimFilter) o; - -if (isLowerStrict() != that.isLowerStrict()) { - return false; -} -if (isUpperStrict() != that.isUpperStrict()) { - return false; -} -if (!getDimension().equals(that.getDimension())) { - return false; -} -if (getUpper() != null ? !getUpper().equals(that.getUpper()) : that.getUpper() != null) { - return false; -} -if (getLower() != null ? !getLower().equals(that.getLower()) : that.getLower() != null) { - return false; -} -if (getExtractionFn() != null -? !getExtractionFn().equals(that.getExtractionFn()) -: that.getExtractionFn() != null) { - return false; -} -return getOrdering().equals(that.getOrdering()); +return lowerStrict == that.lowerStrict && + upperStrict == that.upperStrict && + dimension.equals(that.dimension) && + Objects.equals(upper, that.upper) && + Objects.equals(lower, that.lower) && + Objects.equals(extractionFn, that.extractionFn) && + Objects.equals(ordering, that.ordering) && + Objects.equals(filterTuning, that.filterTuning); } @Override public int hashCode() { -int result = getDimension().hashCode(); -result = 31 * result + (getUpper() != null ? getUpper().hashCode() : 0); -result = 31 * result + (getLower() != null ? getLower().hashCode() : 0); -result = 31 * result + (isLowerStrict() ? 1 : 0); -result = 31 * result + (isUpperStrict() ? 1 : 0); -result = 31 * result + (getExtractionFn() != null ? getExtractionFn().hashCode() : 0); -result = 31 * result + getOrdering().hashCode(); -return result; +return Objects.hash( +dimension, +upper, +lower, +lowerStrict, +upperStrict, +extractionFn, +ordering, +filterTuning +); } @Override Review comment: Refactored custom `DimFilter.toString` implementations, extracting common string builder stuff into a `DimFilterToStringBuilder` to clean this up a bit and help keep these custom `toString` implementations consistent with each other at least, and added `filterTuning` to them when it is not null. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8244: Update docker build
himanshug commented on issue #8244: Update docker build URL: https://github.com/apache/incubator-druid/pull/8244#issuecomment-518438602 > @himanshug I am really sorry for opening this PR again. My forked master branch was messed up. Its fine, everyone messes up including me. +1 for proactively fixing the issues. +1 after build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8242: should we deprecate extractionFn field in various Aggregators and Filters
gianm commented on issue #8242: should we deprecate extractionFn field in various Aggregators and Filters URL: https://github.com/apache/incubator-druid/issues/8242#issuecomment-518450278 I’m a bit nervous about removing something so widely used. I wonder if we can introduce a compat shim that translates these kinds of filters? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei opened a new pull request #8245: Add docs for CliIndexer as an experimental feature
jon-wei opened a new pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245 Adds a doc page for the new CliIndexer process (see #7900) as an experimental feature. This PR also adds a configuration option for the number of concurrent merge/persists allowed on the Indexer, `druid.worker.numConcurrentMerges` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis merged pull request #8231: Web console: celebrate array based groupBy by supporting resultAsArray in the console
clintropolis merged pull request #8231: Web console: celebrate array based groupBy by supporting resultAsArray in the console URL: https://github.com/apache/incubator-druid/pull/8231 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Web console: celebrate array based groupBy by supporting resultAsArray in the console (#8231)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new 0235b33 Web console: celebrate array based groupBy by supporting resultAsArray in the console (#8231) 0235b33 is described below commit 0235b338fc82cc5475011d7928006a493b8966b9 Author: Vadim Ogievetsky AuthorDate: Mon Aug 5 18:54:39 2019 -0700 Web console: celebrate array based groupBy by supporting resultAsArray in the console (#8231) * teach table about resultAsArray * use query result decoder * fix snapshot --- licenses.yaml | 349 + licenses/bin/commander.MIT | 22 ++ licenses/bin/d3-axis.BSD3 | 27 ++ licenses/bin/d3-brush.BSD3 | 27 ++ licenses/bin/d3-chord.BSD3 | 27 ++ licenses/bin/d3-collection.BSD3| 27 ++ licenses/bin/d3-color.BSD3 | 27 ++ licenses/bin/d3-contour.BSD3 | 27 ++ licenses/bin/d3-dispatch.BSD3 | 27 ++ licenses/bin/d3-drag.BSD3 | 27 ++ licenses/bin/d3-dsv.BSD3 | 27 ++ licenses/bin/d3-ease.BSD3 | 28 ++ licenses/bin/d3-fetch.BSD3 | 27 ++ licenses/bin/d3-force.BSD3 | 27 ++ licenses/bin/d3-format.BSD3| 27 ++ licenses/bin/d3-geo.BSD3 | 48 +++ licenses/bin/d3-hierarchy.BSD3 | 27 ++ licenses/bin/d3-interpolate.BSD3 | 27 ++ licenses/bin/d3-path.BSD3 | 27 ++ licenses/bin/d3-polygon.BSD3 | 27 ++ licenses/bin/d3-quadtree.BSD3 | 27 ++ licenses/bin/d3-random.BSD3| 27 ++ licenses/bin/d3-scale-chromatic.BSD3 | 44 +++ licenses/bin/d3-scale.BSD3 | 27 ++ licenses/bin/d3-selection.BSD3 | 26 ++ licenses/bin/d3-shape.BSD3 | 27 ++ licenses/bin/d3-time-format.BSD3 | 27 ++ licenses/bin/d3-time.BSD3 | 27 ++ licenses/bin/d3-timer.BSD3 | 27 ++ licenses/bin/d3-transition.BSD3| 58 licenses/bin/d3-voronoi.BSD3 | 50 +++ licenses/bin/d3-zoom.BSD3 | 27 ++ licenses/bin/d3.BSD3 | 27 ++ licenses/bin/rw.BSD3 | 26 ++ web-console/package-lock.json | 8 + web-console/package.json | 1 + web-console/src/setup-tests.ts | 1 + web-console/src/utils/general.tsx | 15 +- web-console/src/utils/index.tsx| 1 - web-console/src/utils/rune-decoder.tsx | 141 - .../__snapshots__/query-view.spec.tsx.snap | 2 +- .../views/query-view/query-output/query-output.tsx | 2 +- web-console/src/views/query-view/query-view.tsx| 82 ++--- .../query-view/run-button/run-button.spec.tsx | 2 +- .../src/views/query-view/run-button/run-button.tsx | 18 +- web-console/src/views/task-view/tasks-view.tsx | 6 +- 46 files changed, 1392 insertions(+), 213 deletions(-) diff --git a/licenses.yaml b/licenses.yaml index eb1d4a4..f6f8337 100644 --- a/licenses.yaml +++ b/licenses.yaml @@ -2378,6 +2378,16 @@ license_file_path: licenses/bin/classnames.MIT --- +name: "commander" +license_category: binary +module: web-console +license_name: MIT License +copyright: TJ Holowaychuk +version: 2.20.0 +license_file_path: licenses/bin/commander.MIT + +--- + name: "copy-to-clipboard" license_category: binary module: web-console @@ -2433,11 +2443,331 @@ license_category: binary module: web-console license_name: BSD-3-Clause License copyright: Mike Bostock +version: 1.2.4 +license_file_path: licenses/bin/d3-array.BSD3 + +--- + +name: "d3-array" +license_category: binary +module: web-console +license_name: BSD-3-Clause License +copyright: Mike Bostock version: 2.2.0 license_file_path: licenses/bin/d3-array.BSD3 --- +name: "d3-axis" +license_category: binary +module: web-console +license_name: BSD-3-Clause License +copyright: Mike Bostock +version: 1.0.12 +license_file_path: licenses/bin/d3-axis.BSD3 + +--- + +name: "d3-brush" +license_category: binary +module: web-console +license_name: BSD-3-Clause License +copyright: Mike Bostock +version: 1.0.6 +license_file_path: licenses/bin/d3-brush.BSD3 + +--- + +name: "d3-chord" +license_category: binary +module: web-console +license_name: BSD-3-Clause License +copyright: Mike Bostock +version:
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks
jihoonson commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks URL: https://github.com/apache/incubator-druid/pull/8236#discussion_r310870428 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java ## @@ -935,7 +900,7 @@ private TaskStatus generateAndPublishSegments( final BatchAppenderatorDriver driver = newDriver(appenderator, toolbox, segmentAllocator); final Firehose firehose = firehoseFactory.connect(dataSchema.getParser(), firehoseTempDir) ) { - this.appenderator = appenderator; + registerResourceCloserOnAbnormalExit(config -> appenderator.closeNow()); Review comment: Took `Closeable` out of `Appenderator`. Callers of `Appenderator` should explicitly call `close()` or `closeNow()`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks
jihoonson commented on a change in pull request #8236: Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks URL: https://github.com/apache/incubator-druid/pull/8236#discussion_r310870445 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java ## @@ -146,10 +146,13 @@ private final RetryPolicyFactory retryPolicyFactory; @JsonIgnore - private List indexTaskSpecs; + private final AppenderatorsManager appenderatorsManager; @JsonIgnore - private AppenderatorsManager appenderatorsManager; + private List indexTaskSpecs; + + @Nullable + private volatile IndexTask currentRunningTaskSpec = null; Review comment: Added javadoc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #8232: Web-Console: add dimensions to datasources icon
vogievetsky commented on issue #8232: Web-Console: add dimensions to datasources icon URL: https://github.com/apache/incubator-druid/pull/8232#issuecomment-518491012 Looks good. Thank you for fixing. g2g once travis passes. 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310861548 ## File path: extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java ## @@ -168,20 +183,40 @@ public BloomKFilter getBloomKFilter() return bloomKFilter; } + @Nullable @JsonProperty public ExtractionFn getExtractionFn() { return extractionFn; } + @Nullable + @JsonInclude(JsonInclude.Include.NON_NULL) + @JsonProperty + public FilterTuning getFilterTuning() + { +return filterTuning; + } + + @Override + public RangeSet getDimensionRangeSet(String dimension) + { +return null; + } + + @Override + public Set getRequiredColumns() + { +return ImmutableSet.of(dimension); + } + @Override public String toString() { -if (extractionFn != null) { - return StringUtils.format("%s(%s) = %s", extractionFn, dimension, hash.toString()); -} else { - return StringUtils.format("%s = %s", dimension, hash.toString()); -} +return new DimFilterToStringBuilder().appendDimension(dimension, extractionFn) Review comment: Whoa. This is cool. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310885038 ## File path: processing/src/main/java/org/apache/druid/segment/filter/Filters.java ## @@ -651,4 +652,17 @@ private static void generateAllCombinations( generateAllCombinations(result, andList.subList(1, andList.size()), nonAndList); } } + + public static boolean shouldUseIndex(Filter filter, BitmapIndexSelector indexSelector, FilterTuning filterTuning) Review comment: Could use a bit of javadocs as to its purpose ("standard" implementation of Filter.shouldUseIndex) and how it works (returns true if the filter supports indexes and if all underlying columns match the thresholds in `filterTuning`). Also that `filterTuning` is nullable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing
gianm commented on a change in pull request #8209: add mechanism to control filter optimization in historical query processing URL: https://github.com/apache/incubator-druid/pull/8209#discussion_r310864811 ## File path: processing/src/main/java/org/apache/druid/query/filter/Filter.java ## @@ -129,4 +130,19 @@ default boolean canVectorizeMatcher() { return false; } + + /** + * Set of columns used by a filter + */ + Set getRequiredColumns(); + + /** + * Determine if a filter *should* use a bitmap index based on information collected from the supplied + * {@link BitmapIndexSelector}. This method differs from {@link #supportsBitmapIndex(BitmapIndexSelector)} in that + * the former only indicates if a bitmap index is available and {@link #getBitmapIndex(BitmapIndexSelector)} may be + * used. This method, by default, will consider a {@link FilterTuning} to make decisions about when to use an + * available index. Override this method in a {@link Filter} implementation when {@link FilterTuning} alone is not + * adequate for making this decision. + */ Review comment: I'd reword slightly. This method doesn't do anything by default anymore (it did in an earlier version). Maybe cut the last two sentences, and replace with: > Implementations of this methods typically consider a {@link FilterTuning} to make decisions about when to use an available index. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888120 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. + +## Running + +``` +org.apache.druid.cli.Main server indexer +``` + +## Task Resource Sharing + +The following resources are shared across all tasks running inside an Indexer process: + +### Query resources + +The query processing threads and buffers are shared across all tasks. The Indexer will serve queries from a single endpoint shared by all tasks. + +### Server HTTP threads + +The Indexer maintains two equally sized pools of HTTP threads. + +One pool is exclusively used for task control messages between the Overlord and the Indexer ("chat handler threads"). The other pool is used for handling all other HTTP requests. + +The size of the pools are configured by the `druid.server.http.numThreads` configuration (e.g., if this is set to 10, there will be 10 chat handler threads and 10 non-chat handler threads). + +In addition to these two pools, 2 separate threads are allocated for lookup handling. If lookups are not used, these threads will not be used. + +### Memory Sharing + +The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes` configuration to impose a global heap limit across all of the tasks it is running. + +This global limit is evenly divided across the number of task slots configured by `druid.worker.capacity`. + +To apply the per-task heap limit, the Indexer will override `maxBytesInMemory` in task tuning configs (i.e., ignoring the default value or any user configured value). `maxRowsInMemory` will also be overridden to an essentially unlimited value: the Indexer does not support row limits. + +By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the available JVM heap. The remaining portion of the heap is reserved for query processing and segment persist/merge operations, and miscellaneous heap usage. + + Concurrent Segment Persist/Merge Limits + +To help reduce peak memory usage, the Indexer imposes a limit on the number of concurrent segment persist/merge operations across all running tasks. + +By default, the number of concurrent persist/merge operations is limited to (`druid.worker.capacity` / 2), rounded down. This limit can be configured with the `druid.worker.numConcurrentMerges` property. + +## Runtime Configuration + +In addition to the [common configurations](../configuration/index.html#common-configurations), the Indexer accepts the following configurations: + +|Property|Description|Default| +||---|---| +|`druid.worker.version`|Version identifier for the MiddleManager.|0| Review comment: This line, and a few other places, say "MiddleManager" when they should say "Indexer". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888497 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. + +## Running + +``` +org.apache.druid.cli.Main server indexer +``` + +## Task Resource Sharing + +The following resources are shared across all tasks running inside an Indexer process: + +### Query resources + +The query processing threads and buffers are shared across all tasks. The Indexer will serve queries from a single endpoint shared by all tasks. + +### Server HTTP threads + +The Indexer maintains two equally sized pools of HTTP threads. + +One pool is exclusively used for task control messages between the Overlord and the Indexer ("chat handler threads"). The other pool is used for handling all other HTTP requests. + +The size of the pools are configured by the `druid.server.http.numThreads` configuration (e.g., if this is set to 10, there will be 10 chat handler threads and 10 non-chat handler threads). + +In addition to these two pools, 2 separate threads are allocated for lookup handling. If lookups are not used, these threads will not be used. + +### Memory Sharing + +The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes` configuration to impose a global heap limit across all of the tasks it is running. + +This global limit is evenly divided across the number of task slots configured by `druid.worker.capacity`. + +To apply the per-task heap limit, the Indexer will override `maxBytesInMemory` in task tuning configs (i.e., ignoring the default value or any user configured value). `maxRowsInMemory` will also be overridden to an essentially unlimited value: the Indexer does not support row limits. + +By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the available JVM heap. The remaining portion of the heap is reserved for query processing and segment persist/merge operations, and miscellaneous heap usage. + + Concurrent Segment Persist/Merge Limits + +To help reduce peak memory usage, the Indexer imposes a limit on the number of concurrent segment persist/merge operations across all running tasks. + +By default, the number of concurrent persist/merge operations is limited to (`druid.worker.capacity` / 2), rounded down. This limit can be configured with the `druid.worker.numConcurrentMerges` property. + +## Runtime Configuration + +In addition to the [common configurations](../configuration/index.html#common-configurations), the Indexer accepts the following configurations: + +|Property|Description|Default| +||---|---| +|`druid.worker.version`|Version identifier for the MiddleManager.|0| +|`druid.worker.capacity`|Maximum number of tasks the MiddleManager can accept.|Number of available processors - 1| +|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available for ingestion processing. This is applied by automatically setting the `maxBytesInMemory` property on tasks.|60% of configured JVM heap| +|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge operations that can run concurrently across all tasks.|`druid.worker.capacity` / 2, rounded down| +|`druid.indexer.task.baseDir`|Base temporary working directory.|`System.getProperty("java.io.tmpdir")`| +|`druid.indexer.task.baseTaskDir`|Base temporary working directory for tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`| +|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client:2.8.3| +|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on middleManager restart for restorable tasks to gracefully exit.|PT5M| +|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop tasks.|`/tmp/druid-indexing`| +|`druid.indexer.task.restoreTasksOnRestart`|If true, middleManagers will attempt to stop tasks gracefully on shutdown and restore them on restart.|false| +|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to communicate with Overlord.|PT5S| +|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to communicate with Overlord.|PT1M| +|`druid.peon.taskActionClient.retry.maxRetryCount`|Th
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310887454 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. Review comment: "and better enable resource sharing between tasks"? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8245: Add docs for CliIndexer as an experimental feature
gianm commented on issue #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#issuecomment-518511216 I am so amped about the Indexer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310885372 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. + +## Running + +``` +org.apache.druid.cli.Main server indexer +``` + +## Task Resource Sharing + +The following resources are shared across all tasks running inside an Indexer process: + +### Query resources + +The query processing threads and buffers are shared across all tasks. The Indexer will serve queries from a single endpoint shared by all tasks. + +### Server HTTP threads + +The Indexer maintains two equally sized pools of HTTP threads. + +One pool is exclusively used for task control messages between the Overlord and the Indexer ("chat handler threads"). The other pool is used for handling all other HTTP requests. + +The size of the pools are configured by the `druid.server.http.numThreads` configuration (e.g., if this is set to 10, there will be 10 chat handler threads and 10 non-chat handler threads). + +In addition to these two pools, 2 separate threads are allocated for lookup handling. If lookups are not used, these threads will not be used. + +### Memory Sharing + +The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes` configuration to impose a global heap limit across all of the tasks it is running. + +This global limit is evenly divided across the number of task slots configured by `druid.worker.capacity`. + +To apply the per-task heap limit, the Indexer will override `maxBytesInMemory` in task tuning configs (i.e., ignoring the default value or any user configured value). `maxRowsInMemory` will also be overridden to an essentially unlimited value: the Indexer does not support row limits. + +By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the available JVM heap. The remaining portion of the heap is reserved for query processing and segment persist/merge operations, and miscellaneous heap usage. + + Concurrent Segment Persist/Merge Limits + +To help reduce peak memory usage, the Indexer imposes a limit on the number of concurrent segment persist/merge operations across all running tasks. + +By default, the number of concurrent persist/merge operations is limited to (`druid.worker.capacity` / 2), rounded down. This limit can be configured with the `druid.worker.numConcurrentMerges` property. + +## Runtime Configuration + +In addition to the [common configurations](../configuration/index.html#common-configurations), the Indexer accepts the following configurations: + +|Property|Description|Default| +||---|---| +|`druid.worker.version`|Version identifier for the MiddleManager.|0| +|`druid.worker.capacity`|Maximum number of tasks the MiddleManager can accept.|Number of available processors - 1| +|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available for ingestion processing. This is applied by automatically setting the `maxBytesInMemory` property on tasks.|60% of configured JVM heap| +|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge operations that can run concurrently across all tasks.|`druid.worker.capacity` / 2, rounded down| +|`druid.indexer.task.baseDir`|Base temporary working directory.|`System.getProperty("java.io.tmpdir")`| +|`druid.indexer.task.baseTaskDir`|Base temporary working directory for tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`| +|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client:2.8.3| +|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on middleManager restart for restorable tasks to gracefully exit.|PT5M| +|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop tasks.|`/tmp/druid-indexing`| +|`druid.indexer.task.restoreTasksOnRestart`|If true, middleManagers will attempt to stop tasks gracefully on shutdown and restore them on restart.|false| +|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to communicate with Overlord.|PT5S| +|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to communicate with Overlord.|PT1M| +|`druid.peon.taskActionClient.retry.maxRetryCount`|Th
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888041 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. + +## Running + +``` +org.apache.druid.cli.Main server indexer +``` + +## Task Resource Sharing + +The following resources are shared across all tasks running inside an Indexer process: + +### Query resources + +The query processing threads and buffers are shared across all tasks. The Indexer will serve queries from a single endpoint shared by all tasks. + +### Server HTTP threads + +The Indexer maintains two equally sized pools of HTTP threads. + +One pool is exclusively used for task control messages between the Overlord and the Indexer ("chat handler threads"). The other pool is used for handling all other HTTP requests. + +The size of the pools are configured by the `druid.server.http.numThreads` configuration (e.g., if this is set to 10, there will be 10 chat handler threads and 10 non-chat handler threads). + +In addition to these two pools, 2 separate threads are allocated for lookup handling. If lookups are not used, these threads will not be used. + +### Memory Sharing + +The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes` configuration to impose a global heap limit across all of the tasks it is running. + +This global limit is evenly divided across the number of task slots configured by `druid.worker.capacity`. + +To apply the per-task heap limit, the Indexer will override `maxBytesInMemory` in task tuning configs (i.e., ignoring the default value or any user configured value). `maxRowsInMemory` will also be overridden to an essentially unlimited value: the Indexer does not support row limits. + +By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the available JVM heap. The remaining portion of the heap is reserved for query processing and segment persist/merge operations, and miscellaneous heap usage. + + Concurrent Segment Persist/Merge Limits + +To help reduce peak memory usage, the Indexer imposes a limit on the number of concurrent segment persist/merge operations across all running tasks. + +By default, the number of concurrent persist/merge operations is limited to (`druid.worker.capacity` / 2), rounded down. This limit can be configured with the `druid.worker.numConcurrentMerges` property. + +## Runtime Configuration Review comment: How about adding these to `configuration/index.md`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310887504 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. + +## Running + +``` +org.apache.druid.cli.Main server indexer +``` + +## Task Resource Sharing + +The following resources are shared across all tasks running inside an Indexer process: Review comment: `.` would work better than `:` since the following sections are subheads (not a list). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky opened a new issue #8246: Cryptic error coming from Druid SQL in semi-join condition
vogievetsky opened a new issue #8246: Cryptic error coming from Druid SQL in semi-join condition URL: https://github.com/apache/incubator-druid/issues/8246 ### Affected Version 0.16.0-snapshot ### Description The query ```sql SELECT "segment_id", "datasource", "start", "end" FROM sys.segments WHERE "start" IN ( SELECT "start" FROM sys.segments WHERE "datasource" = 'wikiticker' GROUP BY 1 LIMIT 5 ) AND "datasource" = 'wikiticker' ORDER BY "start" DESC LIMIT 100 ``` Produces this cryptic error: `Unknown exception / Got a sink null to which there is no match source type! / java.lang.IllegalStateException` But strangely removing the `datasource` filter clause in the outer SQL makes it all work: ![image](https://user-images.githubusercontent.com/177816/62517102-775fd200-b7db-11e9-9904-6a0e0ec868da.png) As it stands right now this is a release blocker for 0.16.0 as this type of query is generated by a feature added to the web console. To unblock we could do one of: 1. Fix this issue 2. Remove/hide the 'group by interval' feature in the segments view 3. Change how that view fetches data (make it make two calls) 2 or 3 are very simple to do but wanted to get some discussion going on weather we should do 1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org