[GitHub] [druid] abhishekagarwal87 commented on a change in pull request #10680: integration test for coordinator and overlord leadership client
abhishekagarwal87 commented on a change in pull request #10680: URL: https://github.com/apache/druid/pull/10680#discussion_r543106571 ## File path: sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java ## @@ -586,7 +638,8 @@ public TableType getJdbcTableType() StringUtils.toLowerCase(discoveryDruidNode.getNodeRole().toString()), druidServerToUse.getTier(), currentSize, - druidServerToUse.getMaxSize() + druidServerToUse.getMaxSize(), + NullHandling.defaultLongValue() Review comment: hmm. shouldn't this always be zero? Data nodes don't have any leader. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated: Fix hadoop docker copy script (#10671)
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid.git The following commit(s) were added to refs/heads/master by this push: new 7a8e9bb Fix hadoop docker copy script (#10671) 7a8e9bb is described below commit 7a8e9bb156249ca042da7ed8ccb473cc7594afde Author: Abhishek Agarwal <1477457+abhishekagarwa...@users.noreply.github.com> AuthorDate: Tue Dec 15 12:38:50 2020 +0530 Fix hadoop docker copy script (#10671) --- integration-tests/script/copy_hadoop_resources.sh | 16 +--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/integration-tests/script/copy_hadoop_resources.sh b/integration-tests/script/copy_hadoop_resources.sh index 5f4c17e..ae378cf 100755 --- a/integration-tests/script/copy_hadoop_resources.sh +++ b/integration-tests/script/copy_hadoop_resources.sh @@ -14,17 +14,27 @@ # See the License for the specific language governing permissions and # limitations under the License. -set -e # wait for hadoop namenode to be up echo "Waiting for hadoop namenode to be up" +MAX_ITERATIONS=15 +i=1 docker exec -t druid-it-hadoop sh -c "./usr/local/hadoop/bin/hdfs dfs -mkdir -p /druid" -while [ $? -ne 0 ] +while [ $? -ne 0 ] && [ $i -lt $MAX_ITERATIONS ] do sleep 2 + i=$((i+1)) docker exec -t druid-it-hadoop sh -c "./usr/local/hadoop/bin/hdfs dfs -mkdir -p /druid" done -echo "Finished waiting for Hadoop namenode" + +if [ $i -lt $MAX_ITERATIONS ]; then + echo "Hadoop namenode is up after $i iterations" +else + echo "Exhausted all runs while waiting for namenode to be up. Exiting" + exit 1 +fi + +set -e # Setup hadoop druid dirs echo "Setting up druid hadoop dirs" - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #10671: Fix copy_hadoop_resources script in IT
jihoonson merged pull request #10671: URL: https://github.com/apache/druid/pull/10671 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 edited a comment on issue #10542: Create a travis build that runs Druid on a K8S Cluster and runs the integration tests
zhangyue19921010 edited a comment on issue #10542: URL: https://github.com/apache/druid/issues/10542#issuecomment-745067971 Hi @himanshug , I just make a PR https://github.com/apache/druid/pull/10669. This PR add a new IT Job `71`: 1. Deploy a K8s cluster based on minikube. 2. Deploy Druid cluster on K8s: single Druid, use local disk as Deep Storage and use derby as MetaStorage 3. Run single IT test ITNestedQueryPushDownTest which includes data ingestion, Historical loading and query action. Now job `71` is successful which means this PR is not WIP anymore! PTAL :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on pull request #10642: Correctly handle null values in time column results
abhishekagarwal87 commented on pull request #10642: URL: https://github.com/apache/druid/pull/10642#issuecomment-745087613 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 edited a comment on pull request #10680: integration test for coordinator and overlord leadership client
zhangyue19921010 edited a comment on pull request #10680: URL: https://github.com/apache/druid/pull/10680#issuecomment-745080942 Nice idea! And I have made a PR https://github.com/apache/druid/pull/10669 which run IT for druid on K8s. Maybe it can be helpful here :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 edited a comment on pull request #10680: integration test for coordinator and overlord leadership client
zhangyue19921010 edited a comment on pull request #10680: URL: https://github.com/apache/druid/pull/10680#issuecomment-745080942 Nice idea! And I have made a PR https://github.com/apache/druid/pull/10669 which run IT for druid on K8s. Maybe it can be helpful here. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10680: integration test for coordinator and overlord leadership client
zhangyue19921010 commented on pull request #10680: URL: https://github.com/apache/druid/pull/10680#issuecomment-745080942 Nice idea! And I have made a PR https://github.com/apache/druid/pull/10669 which run IT for druid on K8s. Maybe it can be helpful here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis opened a new pull request #10680: integration test for coordinator and overlord leadership client
clintropolis opened a new pull request #10680: URL: https://github.com/apache/druid/pull/10680 ### Description This PR adds integration tests that try to get some coverage of coordinator and overlord leadership changes, using queries against system tables and then cycling which containers are running and forcing leadership changes. The goal is to get some coverage for `DruidLeadershipClient` in the integration tests and avoid any sort of regressions if possible. Once kubernetes integration tests are in place, I imagine we could run these tests against k8s discovery instead of curator based as well since #10544 has now gone in. To aid in my sanity while testing this stuff, I have added `is_leader` to `sys.servers` which is a long column which returns `1` if the server is the leader and 0 if it is not (for coordinators and overlords), and for services which do not have the concept of leadership, will return the default long value (0 in default mode, null if `druid.generic.useDefaultValueForNull=false`). The integration tests add a new test group, 'high-availability', which has a special docker-compose file that brings up a cluster with 1 router, 1 broker, 2 overlords, and 2 coordinators (and zk/kafka and metadata store). The tests check which containers are the leader, issues some system tables queries which should flex the leadership clients to both the current overlord and coordinator leaders, and then restart the containers to force leadership change, repeating this process a few times. I removed many of the `links` sections of the docker-compose file for integration tests, which afaict is deprecated and not necessary, and also modified the base docker-compose file to specify the hostnames to the container names, and set `druid.host` to the same, so that the tests could refer to hosts by hostname instead of container IP address (which is what druid.host defaults to if not specified otherwise). Finally, I fixed a funny race condition that i think could really only happen when doing something like this in docker and starting multiple coordinators at the same time, which would have a race condition when trying to initialize the basic auth extension default auth stuffs, where both containers would detect that it had not been initialized, one would lose the race, and explode out of lifecycle start causing the service to die before starting. This probably isn't a big deal even in a real system because if the process gets started again it would succeed because it would be initialized on the 2nd pass, but our integration test configs do not auto-restart (which is noisy on purpose I think), so instead I just wrapped the initialization in a retry which will skip and continue startup if the duplicate initialization explosion is detected. This PR has: - [ ] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [x] added integration tests. - [ ] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10650: Historical load Segments enhancement
zhangyue19921010 commented on pull request #10650: URL: https://github.com/apache/druid/pull/10650#issuecomment-745071334 @himanshug Thanks for your review and merge! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 edited a comment on issue #10542: Create a travis build that runs Druid on a K8S Cluster and runs the integration tests
zhangyue19921010 edited a comment on issue #10542: URL: https://github.com/apache/druid/issues/10542#issuecomment-745067971 Hi @himanshug , I just make a PR linked before. This PR add a new IT Job `71`: 1. Deploy a K8s cluster based on minikube. 2. Deploy Druid cluster on K8s: single Druid, use local disk as Deep Storage and use derby as MetaStorage 3. Run single IT test ITNestedQueryPushDownTest which includes data ingestion, Historical loading and query action. Now job `71` is successful which means this PR is not WIP anymore! PTAL :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 edited a comment on issue #10542: Create a travis build that runs Druid on a K8S Cluster and runs the integration tests
zhangyue19921010 edited a comment on issue #10542: URL: https://github.com/apache/druid/issues/10542#issuecomment-745067971 Hi @himanshug , I just make a PR linked before. This PR add a new IT Job `71`: 1. Deploy a K8s cluster based on minikube. 2. Deploy Druid cluster on K8s: single Druid, use local disk as Deep Storage and use derby as MetaStorage 3. Run single IT test ITNestedQueryPushDownTest which includes data ingestion, Historical loading and query action. Now job `71` is successful which means this PR is not WIP anymore! PTAL :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on issue #10542: Create a travis build that runs Druid on a K8S Cluster and runs the integration tests
zhangyue19921010 commented on issue #10542: URL: https://github.com/apache/druid/issues/10542#issuecomment-745067971 Hi @himanshug , I just make a PR linked before. This PR add a new IT Job `71`: Deploy a K8s cluster based on minikube. Deploy Druid cluster on K8s: single Druid, use local disk as Deep Storage and use derby as MetaStorage Run single IT test ITNestedQueryPushDownTest which includes data ingestion, Historical loading and query action. Now job `71` is successful which means this PR is not WIP anymore! PTAL :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10669: [WIP] Do Integrate test base on K8s
zhangyue19921010 commented on pull request #10669: URL: https://github.com/apache/druid/pull/10669#issuecomment-745067417 This PR add a new IT Job `71`: 1. Deploy a K8s cluster based on minikube. 2. Deploy Druid cluster on K8s: single Druid, use local disk as Deep Storage and use derby as MetaStorage 3. Run single IT test `ITNestedQueryPushDownTest` which includes data ingestion, Historical loading and query action. Now job `71` is successful which means this PR is not `WIP` anymore! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (c2e26d2 -> ac1882b)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from c2e26d2 Add status/selfDiscovered endpoint to indexer for self discovery of indexer (#10679) add ac1882b kubernetes based discovery druid extension to run Druid on K8S without Zookeeper (#10544) No new revisions were added by this update. Summary of changes: .../druid/annotations/SuppressFBWarnings.java | 42 +++ distribution/bin/check-licenses.py | 1 + distribution/pom.xml | 2 + docs/development/extensions-core/kubernetes.md | 59 docs/operations/kubernetes.md | 2 + extensions-core/kubernetes-extensions/pom.xml | 152 + .../druid/k8s/discovery/DefaultK8sApiClient.java | 186 +++ .../discovery/DefaultK8sLeaderElectorFactory.java | 91 ++ .../DiscoveryDruidNodeAndResourceVersion.java | 44 +++ .../k8s/discovery/DiscoveryDruidNodeList.java | 52 +++ .../apache/druid/k8s/discovery/K8sApiClient.java | 37 +++ .../druid/k8s/discovery/K8sDiscoveryConfig.java| 205 .../druid/k8s/discovery/K8sDiscoveryModule.java| 152 + .../k8s/discovery/K8sDruidLeaderSelector.java | 152 + .../druid/k8s/discovery/K8sDruidNodeAnnouncer.java | 266 +++ .../discovery/K8sDruidNodeDiscoveryProvider.java | 363 + .../k8s/discovery/K8sLeaderElectorFactory.java | 41 +++ .../k8s/discovery/LeaderElectorAsyncWrapper.java | 119 +++ .../org/apache/druid/k8s/discovery/PodInfo.java| 59 .../apache/druid/k8s/discovery/WatchResult.java| 36 ++ .../org.apache.druid.initialization.DruidModule| 16 + .../discovery/K8sAnnouncerAndDiscoveryIntTest.java | 124 +++ .../k8s/discovery/K8sDiscoveryConfigTest.java | 79 + .../discovery/K8sDruidLeaderElectionIntTest.java | 166 ++ .../k8s/discovery/K8sDruidLeaderSelectorTest.java | 180 ++ .../k8s/discovery/K8sDruidNodeAnnouncerTest.java | 143 .../K8sDruidNodeDiscoveryProviderTest.java | 343 +++ .../druid/k8s/discovery/NoopServiceEmitter.java| 36 ++ licenses.yaml | 250 ++ pom.xml| 1 + .../CuratorDruidNodeDiscoveryProvider.java | 184 +-- .../druid/discovery/BaseNodeRoleWatcher.java | 301 + .../druid/discovery/DruidLeaderSelector.java | 1 + .../druid/discovery/BaseNodeRoleWatcherTest.java | 161 + .../main/java/org/apache/druid/cli/CliBroker.java | 15 +- .../main/java/org/apache/druid/cli/CliIndexer.java | 5 +- .../main/java/org/apache/druid/cli/CliPeon.java| 15 +- website/.spelling | 8 + 38 files changed, 3910 insertions(+), 179 deletions(-) create mode 100644 core/src/main/java/org/apache/druid/annotations/SuppressFBWarnings.java create mode 100644 docs/development/extensions-core/kubernetes.md create mode 100644 extensions-core/kubernetes-extensions/pom.xml create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sLeaderElectorFactory.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DiscoveryDruidNodeAndResourceVersion.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DiscoveryDruidNodeList.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sApiClient.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDiscoveryConfig.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDiscoveryModule.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidLeaderSelector.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeAnnouncer.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeDiscoveryProvider.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sLeaderElectorFactory.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/LeaderElectorAsyncWrapper.java create mode 100644 extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/PodInfo.java create mode 100644
[GitHub] [druid] himanshug merged pull request #10544: kubernetes based discovery druid extension to run Druid on K8S without Zookeeper
himanshug merged pull request #10544: URL: https://github.com/apache/druid/pull/10544 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug closed issue #9053: [Proposal] Druid discovery extension for Kubernetes
himanshug closed issue #9053: URL: https://github.com/apache/druid/issues/9053 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #10544: kubernetes based discovery druid extension to run Druid on K8S without Zookeeper
himanshug commented on a change in pull request #10544:
URL: https://github.com/apache/druid/pull/10544#discussion_r543050899
##
File path:
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DiscoveryDruidNodeList.java
##
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.k8s.discovery;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.discovery.DiscoveryDruidNode;
+
+import java.util.Collections;
+import java.util.Map;
+
+public class DiscoveryDruidNodeList
+{
+ private final String resourceVersion;
+ private final Map druidNodes;
+
+ public DiscoveryDruidNodeList(
+ String resourceVersion,
+ Map druidNodes
Review comment:
added, thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 closed pull request #10669: [WIP] Do Integrate test base on K8s
zhangyue19921010 closed pull request #10669: URL: https://github.com/apache/druid/pull/10669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10669: [WIP] Do Integrate test base on K8s
zhangyue19921010 commented on pull request #10669: URL: https://github.com/apache/druid/pull/10669#issuecomment-745041271 re-start to trigger CI again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated: Add status/selfDiscovered endpoint to indexer for self discovery of indexer (#10679)
This is an automated email from the ASF dual-hosted git repository. xvrl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid.git The following commit(s) were added to refs/heads/master by this push: new c2e26d2 Add status/selfDiscovered endpoint to indexer for self discovery of indexer (#10679) c2e26d2 is described below commit c2e26d2e1cc2218dbb1b25dc2794787fc43e24fa Author: Harini Rajendran AuthorDate: Mon Dec 14 21:04:14 2020 -0600 Add status/selfDiscovered endpoint to indexer for self discovery of indexer (#10679) Added the status/selfDiscovered endpoint to indexer. Per the api-reference doc, all services support status/selfDiscovered endpoint. So this change would fix that expected behavior. Also added example config files for indexer process that can be used to spin up the indexer process. --- .../conf/druid/cluster/data/indexer/jvm.config | 9 + .../conf/druid/cluster/data/indexer/main.config| 1 + .../druid/cluster/data/indexer/runtime.properties | 38 ++ .../main/java/org/apache/druid/cli/CliIndexer.java | 5 +++ 4 files changed, 53 insertions(+) diff --git a/examples/conf/druid/cluster/data/indexer/jvm.config b/examples/conf/druid/cluster/data/indexer/jvm.config new file mode 100644 index 000..4611a65 --- /dev/null +++ b/examples/conf/druid/cluster/data/indexer/jvm.config @@ -0,0 +1,9 @@ +-server +-Xms4g +-Xmx4g +-XX:MaxDirectMemorySize=4g +-XX:+ExitOnOutOfMemoryError +-Duser.timezone=UTC +-Dfile.encoding=UTF-8 +-Djava.io.tmpdir=var/tmp +-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager diff --git a/examples/conf/druid/cluster/data/indexer/main.config b/examples/conf/druid/cluster/data/indexer/main.config new file mode 100644 index 000..5183399 --- /dev/null +++ b/examples/conf/druid/cluster/data/indexer/main.config @@ -0,0 +1 @@ +org.apache.druid.cli.Main server indexer diff --git a/examples/conf/druid/cluster/data/indexer/runtime.properties b/examples/conf/druid/cluster/data/indexer/runtime.properties new file mode 100644 index 000..b36c9eb --- /dev/null +++ b/examples/conf/druid/cluster/data/indexer/runtime.properties @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +druid.service=druid/indexer +druid.plaintextPort=8091 + +# Number of tasks per indexer +druid.worker.capacity=4 + +# Task launch parameters +druid.indexer.task.baseTaskDir=var/druid/task + +# HTTP server threads +druid.server.http.numThreads=60 + +# Processing threads and buffers on Indexer +druid.processing.numMergeBuffers=2 +druid.processing.buffer.sizeBytes=100MiB +druid.processing.numThreads=4 + +# Hadoop indexing +druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp diff --git a/services/src/main/java/org/apache/druid/cli/CliIndexer.java b/services/src/main/java/org/apache/druid/cli/CliIndexer.java index 54208fc..4d8e887 100644 --- a/services/src/main/java/org/apache/druid/cli/CliIndexer.java +++ b/services/src/main/java/org/apache/druid/cli/CliIndexer.java @@ -22,6 +22,7 @@ package org.apache.druid.cli; import com.google.common.collect.ImmutableList; import com.google.inject.Binder; import com.google.inject.Inject; +import com.google.inject.Key; import com.google.inject.Module; import com.google.inject.Provides; import com.google.inject.name.Names; @@ -72,6 +73,7 @@ import org.apache.druid.server.coordination.ServerType; import org.apache.druid.server.coordination.ZkCoordinator; import org.apache.druid.server.http.HistoricalResource; import org.apache.druid.server.http.SegmentListerResource; +import org.apache.druid.server.http.SelfDiscoveryResource; import org.apache.druid.server.initialization.jetty.CliIndexerServerModule; import org.apache.druid.server.initialization.jetty.JettyServerInitializer; import org.eclipse.jetty.server.Server; @@ -168,6 +170,9 @@ public class CliIndexer extends ServerRunnable ) .build() ); + +Jerseys.addResource(binder, SelfDiscoveryResource.class); +LifecycleModule.registerKey(binder, Key.get(SelfDiscoveryResource.class)); } @Provides
[GitHub] [druid] xvrl commented on pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery
xvrl commented on pull request #10679: URL: https://github.com/apache/druid/pull/10679#issuecomment-745021418 merging since test failures appear unrelated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xvrl merged pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery
xvrl merged pull request #10679: URL: https://github.com/apache/druid/pull/10679 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other
lgtm-com[bot] commented on pull request #10676: URL: https://github.com/apache/druid/pull/10676#issuecomment-744870421 This pull request **introduces 1 alert** when merging 8b9d26d07527b8b2b6b1a8ed09dc0ac36e353797 into 0ad27c06da4e0bb195a67ed623cc8fbba5967e27 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-57a96773a0a587e3ec39d38f407caf73431abbc4) **new alerts:** * 1 for Dereferenced variable may be null This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10670: [Doc-Fix] Remove extra comma in `druid/integration-tests/README.md`
zhangyue19921010 commented on pull request #10670: URL: https://github.com/apache/druid/pull/10670#issuecomment-744850396 @a2l007 Thanks for your review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10661: [Minor Doc Fix] Correct the default value of `druid.server.http.gracefulShutdownTimeout`
zhangyue19921010 commented on pull request #10661: URL: https://github.com/apache/druid/pull/10661#issuecomment-744849382 @a2l007 Thanks for your review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on pull request #10524: Dynamic auto scale Kafka-Stream ingest tasks
zhangyue19921010 commented on pull request #10524: URL: https://github.com/apache/druid/pull/10524#issuecomment-744842307 Hi @capistrant , Hi @nishantmonu51 ! Sorry to bother you. Is there any updates? Looking forward to your response. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (64f97e7 -> 0ad27c0)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 64f97e7 fix DruidSchema incorrectly listing tables with no segments (#10660) add 0ad27c0 Historical load Segments enhancement (#10650) No new revisions were added by this update. Summary of changes: .../loading/SegmentLoaderLocalCacheManager.java| 36 +- .../coordination/SegmentLoadDropHandler.java | 5 ++- 2 files changed, 39 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug merged pull request #10650: Historical load Segments enhancement
himanshug merged pull request #10650: URL: https://github.com/apache/druid/pull/10650 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug closed issue #10649: Druid Historical may never load specific segment again which is failed loading before.
himanshug closed issue #10649: URL: https://github.com/apache/druid/issues/10649 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery
xvrl commented on a change in pull request #10679: URL: https://github.com/apache/druid/pull/10679#discussion_r542820582 ## File path: examples/conf/druid/cluster/data/indexer/runtime.properties ## @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +druid.service=druid/indexer +druid.plaintextPort=8091 + +# Number of tasks per indexer +druid.worker.capacity=4 + +# Task launch parameters +druid.indexer.task.baseTaskDir=var/druid/task + +# HTTP server threads +druid.server.http.numThreads=60 + +# Processing threads and buffers on Indexer +druid.processing.numMergeBuffers=2 +druid.processing.buffer.sizeBytes=500MiB Review comment: processing buffers are per thread, so we you can leave that to 100MB to match the middle manager This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] harinirajendran commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery
harinirajendran commented on a change in pull request #10679: URL: https://github.com/apache/druid/pull/10679#discussion_r542803426 ## File path: examples/conf/druid/cluster/data/indexer/jvm.config ## @@ -0,0 +1,9 @@ +-server +-Xms128m +-Xmx128m +-XX:MaxDirectMemorySize=1g Review comment: Made it 4g and also increased `druid.processing.buffer.sizeBytes` to 500MiB . Also increased -Xms and -Xmx setting to be 4x. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] harinirajendran commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery
harinirajendran commented on a change in pull request #10679: URL: https://github.com/apache/druid/pull/10679#discussion_r542796281 ## File path: examples/conf/druid/cluster/data/indexer/runtime.properties ## @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +druid.service=druid/indexer +druid.plaintextPort=8091 + +# Number of tasks per indexer +druid.worker.capacity=2 Review comment: Sure! I can update it to match with MM config. ## File path: examples/conf/druid/cluster/data/indexer/jvm.config ## @@ -0,0 +1,9 @@ +-server +-Xms128m +-Xmx128m +-XX:MaxDirectMemorySize=1g Review comment: Made it 4g and also increased `druid.processing.buffer.sizeBytes` to 500MiB This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer
xvrl commented on a change in pull request #10679: URL: https://github.com/apache/druid/pull/10679#discussion_r542786845 ## File path: examples/conf/druid/cluster/data/indexer/jvm.config ## @@ -0,0 +1,9 @@ +-server +-Xms128m +-Xmx128m +-XX:MaxDirectMemorySize=1g Review comment: we probably need more heap to match our middle-manager config, which had 1GB per task. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer
xvrl commented on a change in pull request #10679: URL: https://github.com/apache/druid/pull/10679#discussion_r542786016 ## File path: examples/conf/druid/cluster/data/indexer/runtime.properties ## @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +druid.service=druid/indexer +druid.plaintextPort=8091 + +# Number of tasks per indexer +druid.worker.capacity=2 + +# Task launch parameters +druid.indexer.task.baseTaskDir=var/druid/task + +# HTTP server threads +druid.server.http.numThreads=60 + +# Processing threads and buffers on Indexer +druid.processing.numMergeBuffers=2 +druid.processing.buffer.sizeBytes=100MiB +druid.processing.numThreads=1 Review comment: given we have 1 processing thread per task in the example middle-manager config, I would set this to equal the worker capacity to have a comparable config. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery.
xvrl commented on a change in pull request #10679: URL: https://github.com/apache/druid/pull/10679#discussion_r542784759 ## File path: examples/conf/druid/cluster/data/indexer/runtime.properties ## @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +druid.service=druid/indexer +druid.plaintextPort=8091 + +# Number of tasks per indexer +druid.worker.capacity=2 Review comment: the corresponding middleManager config has 4 tasks, should we match that for consistency? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] harinirajendran commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery.
harinirajendran commented on a change in pull request #10679: URL: https://github.com/apache/druid/pull/10679#discussion_r542741175 ## File path: examples/conf/druid/cluster/data/indexer/runtime.properties ## @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +druid.service=druid/indexer +druid.plaintextPort=8092 Review comment: Actually, I set it back to 8091 and tested the docker image. To avoid localhost port conflict I was using the following port mappings: 8091:8091 for MM and 8092:8092 for indexer. Now updated the indexer port mapping to 8092:8091. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on pull request #10653: Fix post-aggregator computation when used with subtotals
abhishekagarwal87 commented on pull request #10653: URL: https://github.com/apache/druid/pull/10653#issuecomment-744659147 > @abhishekagarwal87 Could you check the errors in https://travis-ci.com/github/apache/druid/jobs/456420636, including the one in GroupByQueryRunnerTest.testGroupByWithSubtotalsSpecWithLongDimensionColumn? It seems to be an existing bug. It can be reproduced in master by writing a subquery that generates null values for numeric dimensions. `RowBasedGrouperHelper#getValueSuppliersForDimensions` should handle the scenario when input can be a numeric null. The bug occurs only when there is a subtotal or subquery used. It was caught here as when I removed the renaming of dimensions, I also removed the type change that was happening earlier (the dummy dimensions were string). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] druid-matt opened a new pull request #198: remove 12/9 event
druid-matt opened a new pull request #198: URL: https://github.com/apache/druid-website-src/pull/198 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery.
xvrl commented on a change in pull request #10679: URL: https://github.com/apache/druid/pull/10679#discussion_r542563847 ## File path: examples/conf/druid/cluster/data/indexer/runtime.properties ## @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +druid.service=druid/indexer +druid.plaintextPort=8092 Review comment: any reason this is different than the default indexer port? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other
lgtm-com[bot] commented on pull request #10676: URL: https://github.com/apache/druid/pull/10676#issuecomment-744586871 This pull request **introduces 1 alert** when merging 027938ea80e385136b32c405f8046b9cef675fc1 into 64f97e7003ebc922e404ff49a31969993669747c - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-228dcd6147cb7770cb451d256ebc1948856ac909) **new alerts:** * 1 for Dereferenced variable may be null This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] harinirajendran opened a new pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery.
harinirajendran opened a new pull request #10679: URL: https://github.com/apache/druid/pull/10679 In this PR I have added example config files that can be used to spin up the indexer process. And have also added the `status/selfDiscovered` endpoint to indexer which we use as readiness probe for indexer in our indexer process's docker config. Testing Built my own docker image, added the following in `docker-compose.yml` file ``` + indexer: +image: apache/druid:harini +container_name: indexer +volumes: + - ./storage:/opt/data + - middle_var:/opt/druid/var +depends_on: + - zookeeper + - postgres + - coordinator +ports: + - "8092:8092" +command: + - indexer +env_file: + - environment + ``` Brought all the services up ``` hrajendran@Harini-Rajendran's-MBP16 docker % docker-compose up -d Creating network "docker_default" with the default driver Creating postgres ... done Creating zookeeper ... done Creating coordinator ... done Creating indexer ... done Creating router... done Creating broker... done Creating historical... done Creating middlemanager ... done hrajendran@Harini-Rajendran's-MBP16 docker % docker ps CONTAINER IDIMAGE COMMAND CREATED STATUS PORTSNAMES d4adafa2d20bapache/druid:harini "/druid.sh broker" 21 minutes ago Up 21 minutes 0.0.0.0:8082->8082/tcp broker 23d2d9dab52aapache/druid:harini "/druid.sh middleMan…" 21 minutes ago Up 21 minutes 0.0.0.0:8091->8091/tcp middlemanager 4af88a4f819bapache/druid:harini "/druid.sh historical" 21 minutes ago Up 21 minutes 0.0.0.0:8083->8083/tcp historical 8247c53aff87apache/druid:harini "/druid.sh router" 21 minutes ago Up 21 minutes 0.0.0.0:->/tcp router 664d268118ddapache/druid:harini "/druid.sh indexer" 21 minutes ago Up 21 minutes 0.0.0.0:8092->8092/tcp indexer 07ffca0bc593apache/druid:harini "/druid.sh coordinat…" 21 minutes ago Up 21 minutes 0.0.0.0:8081->8081/tcp coordinator b16704e3ee3dpostgres:latest "docker-entrypoint.s…" 22 minutes ago Up 21 minutes 5432/tcp postgres 9a61879c560ezookeeper:3.5 "/docker-entrypoint.…" 22 minutes ago Up 21 minutes 2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp zookeeper ``` Made `status/selfDiscovered` curl call to indexer running on port 8092 and got back 200 OK. ``` hrajendran@Harini-Rajendran's-MBP16 docker % curl -v http://localhost:8092/status/selfDiscovered * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 8092 (#0) > GET /status/selfDiscovered HTTP/1.1 > Host: localhost:8092 > User-Agent: curl/7.64.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 14 Dec 2020 17:09:49 GMT < Content-Type: application/json < Vary: Accept-Encoding, User-Agent < Content-Length: 0 < * Connection #0 to host localhost left intact * Closing connection 0 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] ankitnayan opened a new issue #10678: How to store and query array of key value pairs?
ankitnayan opened a new issue #10678:
URL: https://github.com/apache/druid/issues/10678
My data format is like:
```
{
timestamp string,
name string,
kind int,
status_code int,
...
tags: {
key1: value1,
key2: value2,
...
}
}
```
**The list of keys in tags is varying in size for each row and the keys are
also not defined (it can be anything string).**
I want to query dataset by tags, i.e., return all rows which have given key:
value pair.
In Pinot, we can achieve this by storing all keys in a multi-value dimension
TAGS__KEYS and all values in another multi-value dimension TAGS__VALUES and
then query using MAP_VALUE function.
Sample query:
```SELECT * FROM xyzView WHERE ( tags__KEYS = 'key1' AND tags__VALUES =
'value1' AND mapValue(tags__KEYS,'key1',tags__VALUES) = 'value1' )```
How can we achieve the same in Druid? Druid also has a multi-value dimension
but does not have mapValue functionality. Or can we achieve this using any
other way in Druid?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] nishantmonu51 opened a new pull request #10677: Add URI based allowPrefexList and denyPrefixList
nishantmonu51 opened a new pull request #10677: URL: https://github.com/apache/druid/pull/10677 Description Right now there is no support to restrict druid based ingestion to deny ingestion from specific paths. This PR adds a new config `druid.ingestion.uri.allowPrefexList` and `druid.ingestion.uri.denyPrefexList` which allows/deny configured URI prefixes from being accessed through druid ingestion. Default behavior is to allow ingestion from all paths. This PR has: [*] been self-reviewed. [*] added documentation for new or modified features or behaviors. [*] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] capistrant opened a new pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other
capistrant opened a new pull request #10676:
URL: https://github.com/apache/druid/pull/10676
### Description
High Level Description
Add configuration in `tuningConfig` for end user to tell Indexing Service to
wait for segments to become available for query before completing the indexing
task. The configuration is a timeout value in milliseconds to prevent waiting
forever. If the timeout expires, the task still succeeds, but the task reports
will indicate that Druid was not able to confirm that the segments became
available for query.
This new configuration stems from my experience operating a production
cluster with many tenants who often have the same complaint: "My indexing job
is complete but the latest data is not available right when the job finishes".
This addresses that by letting the client set a reasonable timeout. After the
job completes, they can parse the ingestion report and see if their segments
became available. More often than not, with a reasonable timeout, their
segments will indeed be available right when the job completes.
Implementation
A lot of the code is already written for realtime handoffs. I extracted that
code out of the realtime packages into a Java package so it is less confusing
as to why non-realtime tasks are using it. `org.apache.druid.segment.handoff`
is a new package in `druid-server` module.
`AbstractBatchIndexTask` gets a new method,
`waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec,
List segmentsToWaitFor, long waitTimeout)` that handles the
waiting. Batch Indexing implementations leverage this method at the end of
their ingestion task code if the client's tuningConfig has a non-zero wait time
for segment availability. Default is to not wait.
A new key:value pair is added to the IngestionStatsAndErrorsTaskReport
`segmentAvailabilityConfirmed`. This is a boolean that indicates if the job was
able to confirm query availability of the new segments before finishing.
Parallel index task supervisor did not previously have this report, so this PR
adds the report with the needed availability key:value pair so all of simple
native, parallel native, and hadoop native can implement this availability wait.
Alternatives
https://github.com/apache/druid/releases#19-datasource-loadstatus became
available in druid 0.20.0. However, I worry about giving ingestion clients the
green light to hit this API endpoint due to the possible expense of the calls
depending on the questions asked.
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
# Key changed/added classes in this PR
* AbstractBatchIndexTask
* HadoopIndexTask
* IndexTask
* ParallelIndexSupervisorTask
* HadoopTuningConfig
* ParallelIndexTuningConfig
* IngestionStatsAndErrorsTaskReportData
* AbstractITBatchIndexTest
* ITHadoopIndexTest
* ITBestEffortRollupParallelIndexTest
* ITIndexerTest
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] smit-aterlo commented on pull request #9116: WIP: gcloud pubsub indexing service
smit-aterlo commented on pull request #9116: URL: https://github.com/apache/druid/pull/9116#issuecomment-744539253 Hello @aditya-r-m, Any update on this branch. We would love to use druid if it comes with PubSub indexing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10650: Historical load Segments enhancement
zhangyue19921010 commented on a change in pull request #10650:
URL: https://github.com/apache/druid/pull/10650#discussion_r542465563
##
File path:
server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
##
@@ -157,7 +184,15 @@ public File getSegmentFiles(DataSegment segment) throws
SegmentLoadingException
try {
StorageLocation loc = findStorageLocationIfLoaded(segment);
String storageDir = DataSegmentPusher.getDefaultStorageDir(segment,
false);
-
+if (loc != null) {
+ File localStorageDir = new File(loc.getPath(), storageDir);
+ if (checkSegmentFilesIntact(localStorageDir)) {
Review comment:
Done. Thanks for your review!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10650: Historical load Segments enhancement
zhangyue19921010 commented on a change in pull request #10650:
URL: https://github.com/apache/druid/pull/10650#discussion_r542465563
##
File path:
server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
##
@@ -157,7 +184,15 @@ public File getSegmentFiles(DataSegment segment) throws
SegmentLoadingException
try {
StorageLocation loc = findStorageLocationIfLoaded(segment);
String storageDir = DataSegmentPusher.getDefaultStorageDir(segment,
false);
-
+if (loc != null) {
+ File localStorageDir = new File(loc.getPath(), storageDir);
+ if (checkSegmentFilesIntact(localStorageDir)) {
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] nishantmonu51 commented on a change in pull request #10544: kubernetes based discovery druid extension to run Druid on K8S without Zookeeper
nishantmonu51 commented on a change in pull request #10544:
URL: https://github.com/apache/druid/pull/10544#discussion_r542410520
##
File path:
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DiscoveryDruidNodeList.java
##
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.k8s.discovery;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.discovery.DiscoveryDruidNode;
+
+import java.util.Collections;
+import java.util.Map;
+
+public class DiscoveryDruidNodeList
+{
+ private final String resourceVersion;
+ private final Map druidNodes;
+
+ public DiscoveryDruidNodeList(
+ String resourceVersion,
+ Map druidNodes
Review comment:
nit: Add @Nullable annotations.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
