[GitHub] [incubator-druid] vogievetsky opened a new pull request #8454: Fix segment re-ingest
vogievetsky opened a new pull request #8454: Fix segment re-ingest URL: https://github.com/apache/incubator-druid/pull/8454 Fixing the data loader segment reingest flow to use segment metadata and preserve whatever aggregates were defined. Also going though the console and doing misc usability fixes: - Better example loader UI - Fixed bug where examples would sometimes not load - Fix removing of HAVING clause in Query view - Add more calcite keywords to highlighter - Fix casing - Remove `console.log` s - Expose `httpAuthenticationUsername` and `httpAuthenticationPassword` - Fix status dialog in home screen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] wsjz commented on issue #8453: druid 0.12.1 cant load gplcompression
wsjz commented on issue #8453: druid 0.12.1 cant load gplcompression URL: https://github.com/apache/incubator-druid/issues/8453#issuecomment-527307973 and this 2019-09-03T05:15:14,592 ERROR [task-runner-0-priority-0] com.hadoop.compression.lzo.LzoCodec - Cannot load native-lzo without native-hadoop This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] wsjz commented on issue #8453: druid 0.12.1 cant load gplcompression
wsjz commented on issue #8453: druid 0.12.1 cant load gplcompression URL: https://github.com/apache/incubator-druid/issues/8453#issuecomment-527292428 i have add all lib from /hadoop/lib to /druid/lib/, and make sure java.library.path have these libs I think it affected by this? WARN JNDI lookup class is not available because this **JRE does not support JNDI**. JNDI string lookups will not be available, continuing configuration. java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.lookup.JndiLookup to org.apache.logging.log4j.core.lookup.StrLookup This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] wsjz opened a new issue #8453: druid 0.12.1 cant load gplcompression
wsjz opened a new issue #8453: druid 0.12.1 cant load gplcompression URL: https://github.com/apache/incubator-druid/issues/8453 I have no idea for this problem in EMR clusters, I have nearly tried all method . 2019-09-03T03:21:46,819 INFO [task-runner-0-priority-0] org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at nge480390-master-instance-4qz0nkxz.novalocal/192.168.11.157:8050 2019-09-03T03:21:47,362 WARN [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobResourceUploader - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2019-09-03T03:21:47,435 WARN [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2019-09-03T03:21:49,144 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 2019-09-03T03:21:49,216 ERROR [task-runner-0-priority-0] **com.hadoop.compression.lzo.GPLNativeCodeLoader - Could not load native gpl library java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] sekingme opened a new pull request #8452: For exp
sekingme opened a new pull request #8452: For exp URL: https://github.com/apache/incubator-druid/pull/8452 Fix operator mistake of expression OR # Key changed/added classes in this PR * `math-expr.md` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] elloooooo commented on a change in pull request #8450: get active task by datasource when supervisor discover tasks
elloo commented on a change in pull request #8450: get active task by datasource when supervisor discover tasks URL: https://github.com/apache/incubator-druid/pull/8450#discussion_r320070936 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java ## @@ -1293,7 +1293,8 @@ private void discoverTasks() throws ExecutionException, InterruptedException, Ti int taskCount = 0; List futureTaskIds = new ArrayList<>(); List> futures = new ArrayList<>(); -List tasks = taskStorage.getActiveTasks(); +List tasks = taskStorage.getActiveTasksByDatasource(dataSource); Review comment: Yes, I will change it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed pull request #7648: Handle exceptions while fetching offsets
stale[bot] closed pull request #7648: Handle exceptions while fetching offsets URL: https://github.com/apache/incubator-druid/pull/7648 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7648: Handle exceptions while fetching offsets
stale[bot] commented on issue #7648: Handle exceptions while fetching offsets URL: https://github.com/apache/incubator-druid/pull/7648#issuecomment-527253379 This pull request/issue has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] RestfulBlue commented on issue #8186: SeekableStreamSupervisor failed to handle notice
RestfulBlue commented on issue #8186: SeekableStreamSupervisor failed to handle notice URL: https://github.com/apache/incubator-druid/issues/8186#issuecomment-527243418 @jihoonson Hi, sorry for late response. Yes, we use auto reset. But also we have a problem with #8215 . In some places where we have a lot of data, sometimes we have huge overload(that's okey in that case, we can loss some data here), and as a result we need to reset offsets, but they are not resetting automatically and task is just being stuck :( Will #8305 fix it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6541: compaction tasks run for much time
stale[bot] commented on issue #6541: compaction tasks run for much time URL: https://github.com/apache/incubator-druid/issues/6541#issuecomment-527231962 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed issue #6541: compaction tasks run for much time
stale[bot] closed issue #6541: compaction tasks run for much time URL: https://github.com/apache/incubator-druid/issues/6541 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] iamshreeram commented on issue #8430: Update index.md with latest zookeeper
iamshreeram commented on issue #8430: Update index.md with latest zookeeper URL: https://github.com/apache/incubator-druid/pull/8430#issuecomment-527224787 @Fokko / @clintropolis , Sure. Will look for it! If so, You may want to update `zookeeper-3.4.14.tar.gz` to `zookeeper-3.4.14.bin.tar.gz`. We are seeing #8429 issue. Note : I'm running druid cluster in pre-prod environments with `zk-3.5.5` and don't see any issues so far. We are planning to run prod cluster as well with same `zk`. Do you foresee any issues? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] coding4cc opened a new issue #8451: Kafka Indexing Service Cannot allocate segment for timestamp[2019-09-15T15:59:59.999Z], sequenceName[index_kafka_org_month_durations_e3cf
coding4cc opened a new issue #8451: Kafka Indexing Service Cannot allocate segment for timestamp[2019-09-15T15:59:59.999Z], sequenceName[index_kafka_org_month_durations_e3cfbd595efc56f_0] URL: https://github.com/apache/incubator-druid/issues/8451 Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes"). ### Affected Version Druid 0.15.0 ### Description Exception `2019-09-02T11:06:23,884 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29 2019-09-02T11:06:32,510 WARN [main] com.sun.jersey.spi.inject.Errors - The following warnings have been detected with resource and/or provider classes: WARNING: A HTTP GET method, public void org.apache.druid.server.http.SegmentListerResource.getSegments(long,long,long,javax.servlet.http.HttpServletRequest) throws java.io.IOException, MUST return a non-void type. 2019-09-02T11:06:38,980 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Cannot allocate segment for timestamp[2019-09-15T15:59:59.999Z], sequenceName[index_kafka_org_month_durations_e3cfbd595efc56f_0]. 2019-09-02T11:06:38,982 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting. org.apache.druid.java.util.common.ISE: Could not allocate segment for row with timestamp[2019-09-15T15:59:59.999Z] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:605) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:246) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:167) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] 2019-09-02T11:06:39,087 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task. org.apache.druid.java.util.common.ISE: Could not allocate segment for row with timestamp[2019-09-15T15:59:59.999Z] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:605) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:246) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:167) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] Finished peon task 2019-09-02 11:06:39,238 pool-4-thread-1 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started at org.apache.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:47) at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273) at
[GitHub] [incubator-druid] stale[bot] commented on issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs
stale[bot] commented on issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs URL: https://github.com/apache/incubator-druid/issues/6539#issuecomment-527108545 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs
stale[bot] closed issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs URL: https://github.com/apache/incubator-druid/issues/6539 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (31732f0 -> abd8646)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 31732f0 Add realization for updating version of derived segments in Materiali… (#8281) add abd8646 Bump ORC library to 1.5.6 (#8405) No new revisions were added by this update. Summary of changes: extensions-core/orc-extensions/pom.xml | 2 +- licenses.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis merged pull request #8405: Bump ORC library to 1.5.6
clintropolis merged pull request #8405: Bump ORC library to 1.5.6 URL: https://github.com/apache/incubator-druid/pull/8405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #8430: Update index.md with latest zookeeper
clintropolis commented on issue #8430: Update index.md with latest zookeeper URL: https://github.com/apache/incubator-druid/pull/8430#issuecomment-527073434 HI @iamshreeram, thanks for the contribution, but I agree with @Fokko and I don't think we should update the documentation to suggest this until we update our libraries and tests to test against this version and make sure everything still works as expected. I am not aware of any compatibility issues, but we never can be sure until that stuff is in place. 3.5.5 appears to be the first stable release of the 3.5.x branch, and was on released a few months ago, so I'm not sure how heavily Druid has been tested against it at this point. But, it is nice to see that the branch has become stable, it has been a long time coming, so it is probably worth looking into updating the client and tests to use this version at some point. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8404: Bump HttpClient to 4.5.9
clintropolis commented on a change in pull request #8404: Bump HttpClient to 4.5.9 URL: https://github.com/apache/incubator-druid/pull/8404#discussion_r319874659 ## File path: licenses.yaml ## @@ -2807,20 +2806,6 @@ notices: --- -name: Apache HttpClient Review comment: Hmm, it could be that the license checker doesn't pick it up because nothing directly references this which just gets packed into the hadoop client folder at distribution packaging time. I pulled your branch, and definitely still see version 4.5.2 in there though: ``` .../hadoop-dependencies/hadoop-client/2.8.3/$ ls -1 | grep -C2 httpclient hadoop-yarn-server-common-2.8.3.jar htrace-core4-4.0.1-incubating.jar httpclient-4.5.2.jar httpcore-4.4.4.jar jackson-core-asl-1.9.13.jar ``` which is what this entry in the `licenses.yaml` file is for, so I still think this entry should be added back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8404: Bump HttpClient to 4.5.9
clintropolis commented on a change in pull request #8404: Bump HttpClient to 4.5.9 URL: https://github.com/apache/incubator-druid/pull/8404#discussion_r319874659 ## File path: licenses.yaml ## @@ -2807,20 +2806,6 @@ notices: --- -name: Apache HttpClient Review comment: Hmm, it could be that the license checker doesn't pick it up because nothing directly references this which just gets packed into the hadoop client folder at distribution packaging time. I pulled your branch, and definitely still see version 4.5.2 in there though: ``` $ ls -1 | grep -C2 httpclient hadoop-yarn-server-common-2.8.3.jar htrace-core4-4.0.1-incubating.jar httpclient-4.5.2.jar httpcore-4.4.4.jar jackson-core-asl-1.9.13.jar ``` which is what this entry in the `licenses.yaml` file is for, so I still think this entry should be added back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8433: StringDictionaryEncodedColumn dimSelector to return CARDINALITY_UNKNOWN with extractionFn
clintropolis commented on a change in pull request #8433: StringDictionaryEncodedColumn dimSelector to return CARDINALITY_UNKNOWN with extractionFn URL: https://github.com/apache/incubator-druid/pull/8433#discussion_r319858158 ## File path: processing/src/main/java/org/apache/druid/segment/column/StringDictionaryEncodedColumn.java ## @@ -121,7 +121,11 @@ public HistoricalDimensionSelector makeDimensionSelector( @Override public int getValueCardinality() { -return getCardinality(); +if (extractionFn != null) { + return CARDINALITY_UNKNOWN; Review comment: Should this be based on `ExtractionType` since we still know the cardinality if it is `ONE_TO_ONE`, or is that not reliably set on all extractionFn implementations? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8450: get active task by datasource when supervisor discover tasks
clintropolis commented on a change in pull request #8450: get active task by datasource when supervisor discover tasks URL: https://github.com/apache/incubator-druid/pull/8450#discussion_r319855584 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java ## @@ -1293,7 +1293,8 @@ private void discoverTasks() throws ExecutionException, InterruptedException, Ti int taskCount = 0; List futureTaskIds = new ArrayList<>(); List> futures = new ArrayList<>(); -List tasks = taskStorage.getActiveTasks(); +List tasks = taskStorage.getActiveTasksByDatasource(dataSource); Review comment: I think you can also change line 1301, since it no longer needs to check that the datasource is equal This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis merged pull request #8281: Add realization for updating version of derived segments in Materiali…
clintropolis merged pull request #8281: Add realization for updating version of derived segments in Materiali… URL: https://github.com/apache/incubator-druid/pull/8281 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (58fbb69 -> 31732f0)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 58fbb69 Fix hump nomenclature (#8447) add 31732f0 Add realization for updating version of derived segments in Materiali… (#8281) No new revisions were added by this update. Summary of changes: .../MaterializedViewSupervisor.java| 15 ++ .../MaterializedViewSupervisorTest.java| 56 ++ 2 files changed, 71 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Fokko commented on a change in pull request #8404: Bump HttpClient to 4.5.9
Fokko commented on a change in pull request #8404: Bump HttpClient to 4.5.9 URL: https://github.com/apache/incubator-druid/pull/8404#discussion_r319848800 ## File path: licenses.yaml ## @@ -2807,20 +2806,6 @@ notices: --- -name: Apache HttpClient Review comment: @clintropolis any further thoughts on this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on issue #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on issue #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#issuecomment-527034655 @jihoonson @b-slim : Updated PR based on review comments, please check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833945 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterModule.java ## @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard; + +import com.fasterxml.jackson.databind.Module; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.inject.Binder; +import com.google.inject.Injector; +import com.google.inject.Key; +import com.google.inject.Provides; +import com.google.inject.name.Named; +import com.google.inject.name.Names; +import org.apache.druid.guice.JsonConfigProvider; +import org.apache.druid.guice.ManageLifecycle; +import org.apache.druid.initialization.DruidModule; +import org.apache.druid.java.util.emitter.core.Emitter; + +import java.util.Collections; +import java.util.List; +import java.util.stream.Collectors; + +public class DropwizardEmitterModule implements DruidModule +{ + private static final String EMITTER_TYPE = "dropwizard"; + + @Override + public List getJacksonModules() + { +return Collections.EMPTY_LIST; + } + + @Override + public void configure(Binder binder) + { +JsonConfigProvider.bind(binder, "druid.emitter." + EMITTER_TYPE, DropwizardEmitterConfig.class); + } + + @Provides + @ManageLifecycle Review comment: removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833894 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/reporters/DropwizardJMXReporter.java ## @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard.reporters; + +import com.codahale.metrics.JmxReporter; +import com.codahale.metrics.MetricRegistry; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonTypeName; +import org.apache.druid.emitter.dropwizard.DropwizardReporter; + +import java.util.Objects; + +@JsonTypeName("jmx") Review comment: removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833658 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterConfig.java ## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; + +import java.util.Collections; +import java.util.List; +import java.util.Objects; + + +public class DropwizardEmitterConfig +{ + private static int DEFAULT_MAX_GAUGE_COUNT = 100_000; + @JsonProperty + private final List reporters; + @JsonProperty + private final String prefix; + @JsonProperty + private final Boolean includeHost; + @JsonProperty + private final String dimensionMapPath; + @JsonProperty + private final List alertEmitters; + @JsonProperty + private final int maxGaugeCount; + + @JsonCreator + public DropwizardEmitterConfig( + @JsonProperty("reporters") List reporters, + @JsonProperty("prefix") String prefix, + @JsonProperty("includeHost") Boolean includeHost, + @JsonProperty("dimensionMapPath") String dimensionMapPath, + @JsonProperty("alertEmitters") List alertEmitters, + @JsonProperty("maxGaugeCount") Integer maxGaugeCount + ) + { +this.reporters = reporters; +this.prefix = prefix; +this.alertEmitters = alertEmitters == null ? Collections.emptyList() : alertEmitters; +this.includeHost = includeHost != null ? includeHost : true; +this.dimensionMapPath = dimensionMapPath; +this.maxGaugeCount = maxGaugeCount == null ? DEFAULT_MAX_GAUGE_COUNT : maxGaugeCount; + } + + @JsonProperty + public List getReporters() + { +return reporters; + } + + @JsonProperty + public String getPrefix() + { +return prefix; + } + + @JsonProperty + public Boolean getIncludeHost() + { +return includeHost; + } + + @JsonProperty + public String getDimensionMapPath() + { +return dimensionMapPath; + } + + @JsonProperty + public List getAlertEmitters() + { +return alertEmitters; + } + + @JsonProperty + public int getMaxGaugeCount() + { +return maxGaugeCount; + } + + @Override + public boolean equals(Object o) + { +if (this == o) { + return true; +} +if (o == null || getClass() != o.getClass()) { + return false; +} +DropwizardEmitterConfig that = (DropwizardEmitterConfig) o; +return maxGaugeCount == that.maxGaugeCount && + Objects.equals(reporters, that.reporters) && + Objects.equals(prefix, that.prefix) && + Objects.equals(includeHost, that.includeHost) && + Objects.equals(dimensionMapPath, that.dimensionMapPath) && + Objects.equals(alertEmitters, that.alertEmitters); + } + + @Override + public int hashCode() + { +return Objects.hash(reporters, prefix, includeHost, dimensionMapPath, alertEmitters, maxGaugeCount); + } + + @Override + public String toString() + { +return "DropwizardEmitterConfig{" + + "reporters=" + reporters + Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833694 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/reporters/DropwizardConsoleReporter.java ## @@ -0,0 +1,135 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard.reporters; + +import com.codahale.metrics.ConsoleReporter; +import com.codahale.metrics.MetricRegistry; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonTypeName; +import org.apache.druid.emitter.dropwizard.DropwizardReporter; + +import java.util.concurrent.TimeUnit; + +@JsonTypeName("console") Review comment: removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833086 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| Review comment: added This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833057 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitter.java ## @@ -0,0 +1,168 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard; + +import com.codahale.metrics.Gauge; +import com.codahale.metrics.MetricRegistry; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.common.base.Joiner; +import com.google.common.collect.ImmutableList; +import org.apache.druid.java.util.common.ISE; +import org.apache.druid.java.util.common.StringUtils; +import org.apache.druid.java.util.common.logger.Logger; +import org.apache.druid.java.util.emitter.core.Emitter; +import org.apache.druid.java.util.emitter.core.Event; +import org.apache.druid.java.util.emitter.service.AlertEvent; +import org.apache.druid.java.util.emitter.service.ServiceMetricEvent; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.atomic.AtomicBoolean; + +public class DropwizardEmitter implements Emitter +{ + private static final Logger log = new Logger(DropwizardEmitter.class); + private final MetricRegistry metricsRegistry = new MetricRegistry(); + private final AtomicBoolean started = new AtomicBoolean(false); + private final DropwizardConverter converter; + private final List alertEmitters; + private final List reporters; + private final DropwizardEmitterConfig config; + // Note: the gauges do not represent the actual instantaneous value for the metrics. + // Instead they have the last known value for it. + private final Map gagues; + + public DropwizardEmitter( + DropwizardEmitterConfig config, + ObjectMapper mapper, + List alertEmitters + ) + { +this.alertEmitters = alertEmitters; +this.config = config; +this.reporters = config.getReporters(); +this.converter = new DropwizardConverter(mapper, config.getDimensionMapPath()); +this.gagues = Collections.synchronizedMap(new GaugesCache<>(config.getMaxGaugeCount())); + } + + + @Override + public void start() + { +final boolean alreadyStarted = started.getAndSet(true); +if (!alreadyStarted) { + for (DropwizardReporter reporter : reporters) { +reporter.start(metricsRegistry); + } +} + } + + @Override + public void emit(Event event) + { +synchronized (started) { + if (!started.get()) { +throw new RejectedExecutionException("Dropwizard emitter Service not started."); + } +} +if (event instanceof ServiceMetricEvent) { + ServiceMetricEvent metricEvent = (ServiceMetricEvent) event; + String host = metricEvent.getHost(); + String service = metricEvent.getService(); + String metric = metricEvent.getMetric(); + Map userDims = metricEvent.getUserDims(); + Number value = metricEvent.getValue(); + ImmutableList.Builder nameBuilder = new ImmutableList.Builder<>(); + LinkedHashMap dims = new LinkedHashMap<>(); + final DropwizardMetricSpec metricSpec = converter.addFilteredUserDims(service, metric, userDims, dims); + + if (metricSpec != null) { +if (config.getPrefix() != null) { + nameBuilder.add(config.getPrefix()); +} +nameBuilder.add("metric=" + metric); +nameBuilder.add("service=" + service); +if (config.getIncludeHost()) { + nameBuilder.add("hostname=" + host); +} +dims.forEach((key, value1) -> nameBuilder.add(key + "=" + value1)); + +String fullName = StringUtils.replaceChar(Joiner.on(",").join(nameBuilder.build()), '/', "."); +updateMetric(fullName, value, metricSpec); + } else { +log.debug("Service=[%s], Metric=[%s] has no mapping", service, metric); + } +} else if (event instanceof AlertEvent) { + for (Emitter emitter : alertEmitters) { +emitter.emit(event); + } +} else {
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832908 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardConverter.java ## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard; + +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.common.base.Strings; +import org.apache.curator.shaded.com.google.common.io.Closeables; +import org.apache.druid.java.util.common.ISE; +import org.apache.druid.java.util.common.logger.Logger; + +import javax.annotation.Nullable; +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.util.Map; + +/** + * + */ +public class DropwizardConverter +{ + private static final Logger log = new Logger(DropwizardConverter.class); + private final Map metricMap; + + public DropwizardConverter(ObjectMapper mapper, String dimensionMapPath) + { +metricMap = readMap(mapper, dimensionMapPath); + } + + /** + * Filters user dimensions for given metric and adds them to filteredDimensions. + * Returns null if there is no mapping present for the given metric. + */ + @Nullable + public DropwizardMetricSpec addFilteredUserDims( + String service, + String metric, + Map userDims, + Map filteredDimensions + ) + { + /* +Find the metric in the map. If we cant find it try to look it up prefixed by the service name. Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832658 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| +|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the dropwizard metric type, and desired dimensions for every Druid metric|no|Default mapping provided. See below.| +|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)| +|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge metrics to be cached at any time. |no| 100K| + + +### Druid to Dropwizard Event Conversion + +Each metric emitted using Dropwizard must specify a type, one of `[timer, counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to +be provided as a JSON file. Additionally, this mapping specifies which dimensions should be included for each metric. +If the user does not specify their own JSON file, a [default mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json) is used. Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832671 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| +|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the dropwizard metric type, and desired dimensions for every Druid metric|no|Default mapping provided. See below.| +|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)| +|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge metrics to be cached at any time. |no| 100K| + + +### Druid to Dropwizard Event Conversion + +Each metric emitted using Dropwizard must specify a type, one of `[timer, counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to +be provided as a JSON file. Additionally, this mapping specifies which dimensions should be included for each metric. +If the user does not specify their own JSON file, a [default mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json) is used. +All metrics are expected to be mapped. Metrics which are not mapped will be ignored. +Dropwizard metric path is organized using the following schema: + +` : { "dimensions" : , "type" : , "timeUnit" : }` + +e.g. + +`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer", "timeUnit": "MILLISECONDS"}` Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832782 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| +|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the dropwizard metric type, and desired dimensions for every Druid metric|no|Default mapping provided. See below.| +|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)| +|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge metrics to be cached at any time. |no| 100K| + + +### Druid to Dropwizard Event Conversion + +Each metric emitted using Dropwizard must specify a type, one of `[timer, counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to +be provided as a JSON file. Additionally, this mapping specifies which dimensions should be included for each metric. +If the user does not specify their own JSON file, a [default mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json) is used. +All metrics are expected to be mapped. Metrics which are not mapped will be ignored. +Dropwizard metric path is organized using the following schema: + +` : { "dimensions" : , "type" : , "timeUnit" : }` + +e.g. + +`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer", "timeUnit": "MILLISECONDS"}` +`"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"}` + + +For most use-cases, the default mapping is sufficient. + +### Supported Dropwizard reporters + + JMX Reporter +Used to report druid metrics via JMX. +```json + +druid.emitter.dropwizard.reporter={"type":"jmx"} + +``` + + Console Reporter +Used to print Druid Metrics to console logs. + +```json Review comment: removed json. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832857 ## File path: extensions-contrib/dropwizard-emitter/pom.xml ## @@ -0,0 +1,79 @@ + + + +http://www.w3.org/2001/XMLSchema-instance; xmlns="http://maven.apache.org/POM/4.0.0; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd;> + 4.0.0 + + +org.apache.druid +druid +0.16.0-incubating-SNAPSHOT +../../pom.xml + + + org.apache.druid.extensions.contrib + dropwizard-emitter + dropwizard-emitter + Druid emitter extension to convert druid metric to Dropwizard metrics + + + + org.apache.druid + druid-core + ${project.parent.version} + provided + + + org.apache.druid + druid-server + ${project.parent.version} + provided + + + io.dropwizard.metrics + metrics-core + 3.1.0 Review comment: changed to point to the core dependency version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832726 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| +|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the dropwizard metric type, and desired dimensions for every Druid metric|no|Default mapping provided. See below.| +|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)| +|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge metrics to be cached at any time. |no| 100K| + + +### Druid to Dropwizard Event Conversion + +Each metric emitted using Dropwizard must specify a type, one of `[timer, counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to +be provided as a JSON file. Additionally, this mapping specifies which dimensions should be included for each metric. +If the user does not specify their own JSON file, a [default mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json) is used. +All metrics are expected to be mapped. Metrics which are not mapped will be ignored. +Dropwizard metric path is organized using the following schema: + +` : { "dimensions" : , "type" : , "timeUnit" : }` + +e.g. + +`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer", "timeUnit": "MILLISECONDS"}` +`"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"}` + + +For most use-cases, the default mapping is sufficient. + +### Supported Dropwizard reporters + + JMX Reporter +Used to report druid metrics via JMX. +```json Review comment: removed json. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319831612 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| +|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the dropwizard metric type, and desired dimensions for every Druid metric|no|Default mapping provided. See below.| +|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)| +|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge metrics to be cached at any time. |no| 100K| + + +### Druid to Dropwizard Event Conversion + +Each metric emitted using Dropwizard must specify a type, one of `[timer, counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319831410 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,86 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] lswyyy commented on issue #8438: druid-0.15.0运行后自动进程退出
lswyyy commented on issue #8438: druid-0.15.0运行后自动进程退出 URL: https://github.com/apache/incubator-druid/issues/8438#issuecomment-527031277 try to use "exit" command after "nohup bin/start-micro-quickstart &" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319822249 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitter.java ## @@ -0,0 +1,165 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard; + +import com.codahale.metrics.Gauge; +import com.codahale.metrics.MetricRegistry; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.common.base.Joiner; +import com.google.common.collect.ImmutableList; +import org.apache.druid.java.util.common.ISE; +import org.apache.druid.java.util.common.StringUtils; +import org.apache.druid.java.util.common.logger.Logger; +import org.apache.druid.java.util.emitter.core.Emitter; +import org.apache.druid.java.util.emitter.core.Event; +import org.apache.druid.java.util.emitter.service.AlertEvent; +import org.apache.druid.java.util.emitter.service.ServiceMetricEvent; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +public class DropwizardEmitter implements Emitter +{ + private static Logger log = new Logger(DropwizardEmitter.class); + private final MetricRegistry metricsRegistry = new MetricRegistry(); + private final AtomicBoolean started = new AtomicBoolean(false); + private final DropwizardConverter converter; + private final List alertEmitters; + private final List reporters; + private final DropwizardEmitterConfig config; + // Note: the gauges do not represent the actual instantaneous value for the metrics. + // Instead they have the last known value for it. + private final ConcurrentHashMap gagues = new ConcurrentHashMap<>(); + + public DropwizardEmitter( + DropwizardEmitterConfig config, + ObjectMapper mapper, + List alertEmitters + ) + { +this.alertEmitters = alertEmitters; +this.config = config; +this.reporters = config.getReporters(); +this.converter = new DropwizardConverter(mapper, config.getDimensionMapPath()); + } + + + @Override + public void start() Review comment: yeah, it should be called through lifecycle, It seems other emitter has guards too so i added a it here also. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319822306 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterModule.java ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard; + +import com.fasterxml.jackson.databind.Module; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.common.collect.Lists; +import com.google.inject.Binder; +import com.google.inject.Injector; +import com.google.inject.Key; +import com.google.inject.Provides; +import com.google.inject.name.Named; +import com.google.inject.name.Names; +import org.apache.druid.guice.JsonConfigProvider; +import org.apache.druid.guice.ManageLifecycle; +import org.apache.druid.initialization.DruidModule; +import org.apache.druid.java.util.emitter.core.Emitter; + +import java.util.Collections; +import java.util.List; + +public class DropwizardEmitterModule implements DruidModule +{ + private static final String EMITTER_TYPE = "dropwizard"; + + @Override + public List getJacksonModules() + { +return Collections.EMPTY_LIST; + } + + @Override + public void configure(Binder binder) + { +JsonConfigProvider.bind(binder, "druid.emitter." + EMITTER_TYPE, DropwizardEmitterConfig.class); + } + + @Provides + @ManageLifecycle + @Named(EMITTER_TYPE) + public Emitter getEmitter( + DropwizardEmitterConfig dropwizardEmitterConfig, + ObjectMapper mapper, + final Injector injector + ) + { +List alertEmitters = Lists.transform( Review comment: done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid
nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319821801 ## File path: extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardConverter.java ## @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.emitter.dropwizard; + +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.common.base.Strings; +import org.apache.druid.java.util.common.ISE; +import org.apache.druid.java.util.common.logger.Logger; + +import javax.annotation.Nullable; +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.util.Map; + +/** + * + */ +public class DropwizardConverter +{ + private static final Logger log = new Logger(DropwizardConverter.class); + private Map metricMap; + + public DropwizardConverter(ObjectMapper mapper, String dimensionMapPath) + { +metricMap = readMap(mapper, dimensionMapPath); + } + + @Nullable + public DropwizardMetricSpec addFilteredUserDims( + String service, + String metric, + Map userDims, + Map builder + ) + { + /* +Find the metric in the map. If we cant find it try to look it up prefixed by the service name. +This is because some metrics are reported differently, but with the same name, from different services. + */ +DropwizardMetricSpec metricSpec = null; +if (metricMap.containsKey(metric)) { + metricSpec = metricMap.get(metric); +} else if (metricMap.containsKey(service + "-" + metric)) { + metricSpec = metricMap.get(service + "-" + metric); +} +if (metricSpec != null) { + for (String dim : metricSpec.getDimensions()) { +if (userDims.containsKey(dim)) { + builder.put(dim, userDims.get(dim).toString()); +} + } + return metricSpec; +} else { + return null; +} + } + + private Map readMap(ObjectMapper mapper, String dimensionMapPath) + { +try { + InputStream is; + if (Strings.isNullOrEmpty(dimensionMapPath)) { +log.info("Using default metric dimension and types"); +is = this.getClass().getClassLoader().getResourceAsStream("defaultMetricDimensions.json"); + } else { +log.info("Using metric dimensions at types at [%s]", dimensionMapPath); +is = new FileInputStream(new File(dimensionMapPath)); + } + return mapper.readerFor(new TypeReference>() + { + }).readValue(is); +} +catch (IOException e) { Review comment: thought ObjectReader.readValue will close the stream in normal flow This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on issue #7537: Avoid file rename when pushing segments with HDFSDataPusher
nishantmonu51 commented on issue #7537: Avoid file rename when pushing segments with HDFSDataPusher URL: https://github.com/apache/incubator-druid/pull/7537#issuecomment-527019852 ping @gianm: Any more comments here ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7537: Avoid file rename when pushing segments with HDFSDataPusher
stale[bot] commented on issue #7537: Avoid file rename when pushing segments with HDFSDataPusher URL: https://github.com/apache/incubator-druid/pull/7537#issuecomment-527019749 This pull request/issue is no longer marked as stale. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on issue #8248: Add TrustedDomain Authenticator
nishantmonu51 commented on issue #8248: Add TrustedDomain Authenticator URL: https://github.com/apache/incubator-druid/pull/8248#issuecomment-527019385 @jon-wei : Yes, its possible to spoof X-forwarded header, that is one of the reasons its not allowed by default. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #8248: Add TrustedDomain Authenticator
nishantmonu51 commented on a change in pull request #8248: Add TrustedDomain Authenticator URL: https://github.com/apache/incubator-druid/pull/8248#discussion_r319819357 ## File path: server/src/main/java/org/apache/druid/server/security/TrustedDomainAuthenticator.java ## @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.security; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonTypeName; +import com.google.common.base.Preconditions; +import com.google.common.base.Strings; +import org.apache.druid.java.util.common.logger.Logger; + +import javax.annotation.Nullable; +import javax.servlet.DispatcherType; +import javax.servlet.Filter; +import javax.servlet.FilterChain; +import javax.servlet.FilterConfig; +import javax.servlet.ServletException; +import javax.servlet.ServletRequest; +import javax.servlet.ServletResponse; +import javax.servlet.http.HttpServletRequest; +import java.io.IOException; +import java.util.EnumSet; +import java.util.Map; + +/** + * Authenticates requests coming from a specific domain and directs them to an authorizer. + */ +@JsonTypeName(AuthConfig.TRUSTED_DOMAIN_NAME) +public class TrustedDomainAuthenticator implements Authenticator +{ + private static final Logger LOGGER = new Logger(TrustedDomainAuthenticator.class); + private static final String DEFAULT_IDENTITY = "defaultUser"; + private static final String X_FORWARDED_FOR = "X-Forwarded-For"; + + private static final boolean DEFAULT_USE_FORWARDED_HEADERS = false; + private final AuthenticationResult authenticationResult; + private final String domain; + private final boolean useForwardedHeaders; + + @JsonCreator + public TrustedDomainAuthenticator( + @JsonProperty("name") String name, + @JsonProperty("domain") String domain, + @JsonProperty("useForwardedHeaders") Boolean useForwardedHeaders, + @JsonProperty("authorizerName") String authorizerName, + @JsonProperty("identity") String identity + ) + { +Preconditions.checkArgument(!Strings.isNullOrEmpty(domain), "Invalid domain name %s", domain); +this.domain = domain; +this.useForwardedHeaders = useForwardedHeaders == null ? DEFAULT_USE_FORWARDED_HEADERS : useForwardedHeaders; +this.authenticationResult = new AuthenticationResult( +identity == null ? DEFAULT_IDENTITY : identity, +authorizerName, +name, +null +); + } + + @Override + public Class getFilterClass() + { +return null; + } + + @Override + public Map getInitParameters() + { +return null; + } + + @Override + public String getPath() + { +return "/*"; + } + + @Override + public EnumSet getDispatcherType() + { +return null; Review comment: added This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #8248: Add TrustedDomain Authenticator
nishantmonu51 commented on a change in pull request #8248: Add TrustedDomain Authenticator URL: https://github.com/apache/incubator-druid/pull/8248#discussion_r319819341 ## File path: server/src/main/java/org/apache/druid/server/security/TrustedDomainAuthenticator.java ## @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.server.security; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonTypeName; +import com.google.common.base.Preconditions; +import com.google.common.base.Strings; +import org.apache.druid.java.util.common.logger.Logger; + +import javax.annotation.Nullable; +import javax.servlet.DispatcherType; +import javax.servlet.Filter; +import javax.servlet.FilterChain; +import javax.servlet.FilterConfig; +import javax.servlet.ServletException; +import javax.servlet.ServletRequest; +import javax.servlet.ServletResponse; +import javax.servlet.http.HttpServletRequest; +import java.io.IOException; +import java.util.EnumSet; +import java.util.Map; + +/** + * Authenticates requests coming from a specific domain and directs them to an authorizer. + */ +@JsonTypeName(AuthConfig.TRUSTED_DOMAIN_NAME) +public class TrustedDomainAuthenticator implements Authenticator +{ + private static final Logger LOGGER = new Logger(TrustedDomainAuthenticator.class); + private static final String DEFAULT_IDENTITY = "defaultUser"; + private static final String X_FORWARDED_FOR = "X-Forwarded-For"; + + private static final boolean DEFAULT_USE_FORWARDED_HEADERS = false; + private final AuthenticationResult authenticationResult; + private final String domain; + private final boolean useForwardedHeaders; + + @JsonCreator + public TrustedDomainAuthenticator( + @JsonProperty("name") String name, + @JsonProperty("domain") String domain, + @JsonProperty("useForwardedHeaders") Boolean useForwardedHeaders, + @JsonProperty("authorizerName") String authorizerName, + @JsonProperty("identity") String identity + ) + { +Preconditions.checkArgument(!Strings.isNullOrEmpty(domain), "Invalid domain name %s", domain); +this.domain = domain; +this.useForwardedHeaders = useForwardedHeaders == null ? DEFAULT_USE_FORWARDED_HEADERS : useForwardedHeaders; +this.authenticationResult = new AuthenticationResult( +identity == null ? DEFAULT_IDENTITY : identity, +authorizerName, +name, +null +); + } + + @Override + public Class getFilterClass() + { +return null; + } + + @Override + public Map getInitParameters() + { +return null; Review comment: added This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org