[GitHub] [incubator-druid] vogievetsky opened a new pull request #8454: Fix segment re-ingest

2019-09-02 Thread GitBox
vogievetsky opened a new pull request #8454: Fix segment re-ingest
URL: https://github.com/apache/incubator-druid/pull/8454
 
 
   Fixing the data loader segment reingest flow to use segment metadata and 
preserve whatever aggregates were defined.
   
   Also going though the console and doing misc usability fixes:
   - Better example loader UI
   - Fixed bug where examples would sometimes not load
   - Fix removing of HAVING clause in Query view
   - Add more calcite keywords to highlighter 
   - Fix casing
   - Remove `console.log` s
   - Expose `httpAuthenticationUsername` and `httpAuthenticationPassword`
   - Fix status dialog in home screen


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] wsjz commented on issue #8453: druid 0.12.1 cant load gplcompression

2019-09-02 Thread GitBox
wsjz commented on issue #8453: druid 0.12.1 cant load gplcompression
URL: 
https://github.com/apache/incubator-druid/issues/8453#issuecomment-527307973
 
 
   and this
   
   2019-09-03T05:15:14,592 ERROR [task-runner-0-priority-0] 
com.hadoop.compression.lzo.LzoCodec - Cannot load native-lzo without 
native-hadoop


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] wsjz commented on issue #8453: druid 0.12.1 cant load gplcompression

2019-09-02 Thread GitBox
wsjz commented on issue #8453: druid 0.12.1 cant load gplcompression
URL: 
https://github.com/apache/incubator-druid/issues/8453#issuecomment-527292428
 
 
   i have add all lib from /hadoop/lib to /druid/lib/, and make sure 
java.library.path have these libs
   
   I think it affected by this? 
   
   WARN JNDI lookup class is not available because this **JRE does not support 
JNDI**. JNDI string lookups will not be available, continuing configuration. 
java.lang.ClassCastException: Cannot cast 
org.apache.logging.log4j.core.lookup.JndiLookup to 
org.apache.logging.log4j.core.lookup.StrLookup


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] wsjz opened a new issue #8453: druid 0.12.1 cant load gplcompression

2019-09-02 Thread GitBox
wsjz opened a new issue #8453: druid 0.12.1 cant load gplcompression
URL: https://github.com/apache/incubator-druid/issues/8453
 
 
   I have no idea for this problem in EMR clusters, I have nearly tried all 
method .
   
   2019-09-03T03:21:46,819 INFO [task-runner-0-priority-0] 
org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at 
nge480390-master-instance-4qz0nkxz.novalocal/192.168.11.157:8050
   2019-09-03T03:21:47,362 WARN [task-runner-0-priority-0] 
org.apache.hadoop.mapreduce.JobResourceUploader - Hadoop command-line option 
parsing not performed. Implement the Tool interface and execute your 
application with ToolRunner to remedy this.
   2019-09-03T03:21:47,435 WARN [task-runner-0-priority-0] 
org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set.  User 
classes may not be found. See Job or Job#setJar(String).
   2019-09-03T03:21:49,144 INFO [task-runner-0-priority-0] 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to 
process : 1
   2019-09-03T03:21:49,216 ERROR [task-runner-0-priority-0] 
   **com.hadoop.compression.lzo.GPLNativeCodeLoader - Could not load native gpl 
library
   java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path**


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] sekingme opened a new pull request #8452: For exp

2019-09-02 Thread GitBox
sekingme opened a new pull request #8452: For exp
URL: https://github.com/apache/incubator-druid/pull/8452
 
 
   Fix operator mistake of expression OR 
   
   
   # Key changed/added classes in this PR
* `math-expr.md`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] elloooooo commented on a change in pull request #8450: get active task by datasource when supervisor discover tasks

2019-09-02 Thread GitBox
elloo commented on a change in pull request #8450: get active task by 
datasource when supervisor discover tasks
URL: https://github.com/apache/incubator-druid/pull/8450#discussion_r320070936
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
 ##
 @@ -1293,7 +1293,8 @@ private void discoverTasks() throws ExecutionException, 
InterruptedException, Ti
 int taskCount = 0;
 List futureTaskIds = new ArrayList<>();
 List> futures = new ArrayList<>();
-List tasks = taskStorage.getActiveTasks();
+List tasks = taskStorage.getActiveTasksByDatasource(dataSource);
 
 Review comment:
   Yes, I will change it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] closed pull request #7648: Handle exceptions while fetching offsets

2019-09-02 Thread GitBox
stale[bot] closed pull request #7648: Handle exceptions while fetching offsets 
URL: https://github.com/apache/incubator-druid/pull/7648
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] commented on issue #7648: Handle exceptions while fetching offsets

2019-09-02 Thread GitBox
stale[bot] commented on issue #7648: Handle exceptions while fetching offsets 
URL: https://github.com/apache/incubator-druid/pull/7648#issuecomment-527253379
 
 
   This pull request/issue has been closed due to lack of activity. If you 
think that is incorrect, or the pull request requires review, you can revive 
the PR at any time.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] RestfulBlue commented on issue #8186: SeekableStreamSupervisor failed to handle notice

2019-09-02 Thread GitBox
RestfulBlue commented on issue #8186: SeekableStreamSupervisor failed to handle 
notice
URL: 
https://github.com/apache/incubator-druid/issues/8186#issuecomment-527243418
 
 
   @jihoonson Hi, sorry for late response. Yes, we use auto reset. But also we 
have a problem with #8215 . In some places where we have a lot of data, 
sometimes we have huge overload(that's okey in that case, we can loss some data 
here), and as a result we need to reset offsets, but they are not resetting 
automatically and task is just being stuck :( 
   Will #8305 fix it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] commented on issue #6541: compaction tasks run for much time

2019-09-02 Thread GitBox
stale[bot] commented on issue #6541: compaction tasks run for much time
URL: 
https://github.com/apache/incubator-druid/issues/6541#issuecomment-527231962
 
 
   This issue has been closed due to lack of activity. If you think that is 
incorrect, or the issue requires additional review, you can revive the issue at 
any time.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] closed issue #6541: compaction tasks run for much time

2019-09-02 Thread GitBox
stale[bot] closed issue #6541: compaction tasks run for much time
URL: https://github.com/apache/incubator-druid/issues/6541
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] iamshreeram commented on issue #8430: Update index.md with latest zookeeper

2019-09-02 Thread GitBox
iamshreeram commented on issue #8430: Update index.md with latest zookeeper
URL: https://github.com/apache/incubator-druid/pull/8430#issuecomment-527224787
 
 
   @Fokko / @clintropolis , Sure. Will look for it! 
   
   If so, You may want to update `zookeeper-3.4.14.tar.gz` to 
`zookeeper-3.4.14.bin.tar.gz`. We are seeing #8429 issue.
   
   Note : I'm running druid cluster in pre-prod environments with `zk-3.5.5` 
and don't see any issues so far. 
   We are planning to run prod cluster as well with same `zk`. Do you foresee 
any issues? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] coding4cc opened a new issue #8451: Kafka Indexing Service Cannot allocate segment for timestamp[2019-09-15T15:59:59.999Z], sequenceName[index_kafka_org_month_durations_e3cf

2019-09-02 Thread GitBox
coding4cc opened a new issue #8451: Kafka Indexing Service Cannot allocate 
segment for timestamp[2019-09-15T15:59:59.999Z], 
sequenceName[index_kafka_org_month_durations_e3cfbd595efc56f_0]
URL: https://github.com/apache/incubator-druid/issues/8451
 
 
   Please provide a detailed title (e.g. "Broker crashes when using TopN query 
with Bound filter" instead of just "Broker crashes").
   
   ### Affected Version
   
   Druid 0.15.0
   
   ### Description
   
   Exception
   `2019-09-02T11:06:23,884 WARN [main] 
org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). 
Pinning to 29
   2019-09-02T11:06:32,510 WARN [main] com.sun.jersey.spi.inject.Errors - The 
following warnings have been detected with resource and/or provider classes:
 WARNING: A HTTP GET method, public void 
org.apache.druid.server.http.SegmentListerResource.getSegments(long,long,long,javax.servlet.http.HttpServletRequest)
 throws java.io.IOException, MUST return a non-void type.
   2019-09-02T11:06:38,980 WARN [task-runner-0-priority-0] 
org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Cannot 
allocate segment for timestamp[2019-09-15T15:59:59.999Z], 
sequenceName[index_kafka_org_month_durations_e3cfbd595efc56f_0]. 
   2019-09-02T11:06:38,982 ERROR [task-runner-0-priority-0] 
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - 
Encountered exception in run() before persisting.
   org.apache.druid.java.util.common.ISE: Could not allocate segment for row 
with timestamp[2019-09-15T15:59:59.999Z]
at 
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:605)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at 
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:246)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at 
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:167)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at 
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at 
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_191]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_191]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   2019-09-02T11:06:39,087 ERROR [task-runner-0-priority-0] 
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - 
Encountered exception while running task.
   org.apache.druid.java.util.common.ISE: Could not allocate segment for row 
with timestamp[2019-09-15T15:59:59.999Z]
at 
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:605)
 ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at 
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:246)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at 
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:167)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at 
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at 
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391)
 [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_191]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_191]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   Finished peon task
   2019-09-02 11:06:39,238 pool-4-thread-1 ERROR Unable to register shutdown 
hook because JVM is shutting down. java.lang.IllegalStateException: Not started
at 
org.apache.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:47)
at 
org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273)
at 

[GitHub] [incubator-druid] stale[bot] commented on issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs

2019-09-02 Thread GitBox
stale[bot] commented on issue #6539: Task failure for kafka indexing service in 
druid 0.12.1 with no error logs
URL: 
https://github.com/apache/incubator-druid/issues/6539#issuecomment-527108545
 
 
   This issue has been closed due to lack of activity. If you think that is 
incorrect, or the issue requires additional review, you can revive the issue at 
any time.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] closed issue #6539: Task failure for kafka indexing service in druid 0.12.1 with no error logs

2019-09-02 Thread GitBox
stale[bot] closed issue #6539: Task failure for kafka indexing service in druid 
0.12.1 with no error logs
URL: https://github.com/apache/incubator-druid/issues/6539
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[incubator-druid] branch master updated (31732f0 -> abd8646)

2019-09-02 Thread cwylie
This is an automated email from the ASF dual-hosted git repository.

cwylie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 31732f0  Add realization for updating version of derived segments in 
Materiali… (#8281)
 add abd8646  Bump ORC library to 1.5.6 (#8405)

No new revisions were added by this update.

Summary of changes:
 extensions-core/orc-extensions/pom.xml | 2 +-
 licenses.yaml  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis merged pull request #8405: Bump ORC library to 1.5.6

2019-09-02 Thread GitBox
clintropolis merged pull request #8405: Bump ORC library to 1.5.6
URL: https://github.com/apache/incubator-druid/pull/8405
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on issue #8430: Update index.md with latest zookeeper

2019-09-02 Thread GitBox
clintropolis commented on issue #8430: Update index.md with latest zookeeper
URL: https://github.com/apache/incubator-druid/pull/8430#issuecomment-527073434
 
 
   HI @iamshreeram, thanks for the contribution, but I agree with @Fokko and I 
don't think we should update the documentation to suggest this until we update 
our libraries and tests to test against this version and make sure everything 
still works as expected. I am not aware of any compatibility issues, but we 
never can be sure until that stuff is in place.
   
   3.5.5 appears to be the first stable release of the 3.5.x branch, and was on 
released a few months ago, so I'm not sure how heavily Druid has been tested 
against it at this point. But, it is nice to see that the branch has become 
stable, it has been a long time coming, so it is probably worth looking into 
updating the client and tests to use this version at some point.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8404: Bump HttpClient to 4.5.9

2019-09-02 Thread GitBox
clintropolis commented on a change in pull request #8404: Bump HttpClient to 
4.5.9
URL: https://github.com/apache/incubator-druid/pull/8404#discussion_r319874659
 
 

 ##
 File path: licenses.yaml
 ##
 @@ -2807,20 +2806,6 @@ notices:
 
 ---
 
-name: Apache HttpClient
 
 Review comment:
   Hmm, it could be that the license checker doesn't pick it up because nothing 
directly references this which just gets packed into the hadoop client folder 
at distribution packaging time.
   
   I pulled your branch, and definitely still see version 4.5.2 in there though:
   ```
   .../hadoop-dependencies/hadoop-client/2.8.3/$ ls -1 | grep -C2 httpclient
   hadoop-yarn-server-common-2.8.3.jar
   htrace-core4-4.0.1-incubating.jar
   httpclient-4.5.2.jar
   httpcore-4.4.4.jar
   jackson-core-asl-1.9.13.jar
   ```
   which is what this entry in the `licenses.yaml` file is for, so I still 
think this entry should be added back.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8404: Bump HttpClient to 4.5.9

2019-09-02 Thread GitBox
clintropolis commented on a change in pull request #8404: Bump HttpClient to 
4.5.9
URL: https://github.com/apache/incubator-druid/pull/8404#discussion_r319874659
 
 

 ##
 File path: licenses.yaml
 ##
 @@ -2807,20 +2806,6 @@ notices:
 
 ---
 
-name: Apache HttpClient
 
 Review comment:
   Hmm, it could be that the license checker doesn't pick it up because nothing 
directly references this which just gets packed into the hadoop client folder 
at distribution packaging time.
   
   I pulled your branch, and definitely still see version 4.5.2 in there though:
   ```
   $ ls -1 | grep -C2 httpclient
   hadoop-yarn-server-common-2.8.3.jar
   htrace-core4-4.0.1-incubating.jar
   httpclient-4.5.2.jar
   httpcore-4.4.4.jar
   jackson-core-asl-1.9.13.jar
   ```
   which is what this entry in the `licenses.yaml` file is for, so I still 
think this entry should be added back.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8433: StringDictionaryEncodedColumn dimSelector to return CARDINALITY_UNKNOWN with extractionFn

2019-09-02 Thread GitBox
clintropolis commented on a change in pull request #8433: 
StringDictionaryEncodedColumn dimSelector to return CARDINALITY_UNKNOWN with 
extractionFn
URL: https://github.com/apache/incubator-druid/pull/8433#discussion_r319858158
 
 

 ##
 File path: 
processing/src/main/java/org/apache/druid/segment/column/StringDictionaryEncodedColumn.java
 ##
 @@ -121,7 +121,11 @@ public HistoricalDimensionSelector makeDimensionSelector(
   @Override
   public int getValueCardinality()
   {
-return getCardinality();
+if (extractionFn != null) {
+  return CARDINALITY_UNKNOWN;
 
 Review comment:
   Should this be based on `ExtractionType` since we still know the cardinality 
if it is `ONE_TO_ONE`, or is that not reliably set on all extractionFn 
implementations?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8450: get active task by datasource when supervisor discover tasks

2019-09-02 Thread GitBox
clintropolis commented on a change in pull request #8450: get active task by 
datasource when supervisor discover tasks
URL: https://github.com/apache/incubator-druid/pull/8450#discussion_r319855584
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
 ##
 @@ -1293,7 +1293,8 @@ private void discoverTasks() throws ExecutionException, 
InterruptedException, Ti
 int taskCount = 0;
 List futureTaskIds = new ArrayList<>();
 List> futures = new ArrayList<>();
-List tasks = taskStorage.getActiveTasks();
+List tasks = taskStorage.getActiveTasksByDatasource(dataSource);
 
 Review comment:
   I think you can also change line 1301, since it no longer needs to check 
that the datasource is equal


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis merged pull request #8281: Add realization for updating version of derived segments in Materiali…

2019-09-02 Thread GitBox
clintropolis merged pull request #8281: Add realization for updating version of 
derived segments in Materiali…
URL: https://github.com/apache/incubator-druid/pull/8281
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[incubator-druid] branch master updated (58fbb69 -> 31732f0)

2019-09-02 Thread cwylie
This is an automated email from the ASF dual-hosted git repository.

cwylie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 58fbb69  Fix hump nomenclature (#8447)
 add 31732f0  Add realization for updating version of derived segments in 
Materiali… (#8281)

No new revisions were added by this update.

Summary of changes:
 .../MaterializedViewSupervisor.java| 15 ++
 .../MaterializedViewSupervisorTest.java| 56 ++
 2 files changed, 71 insertions(+)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] Fokko commented on a change in pull request #8404: Bump HttpClient to 4.5.9

2019-09-02 Thread GitBox
Fokko commented on a change in pull request #8404: Bump HttpClient to 4.5.9
URL: https://github.com/apache/incubator-druid/pull/8404#discussion_r319848800
 
 

 ##
 File path: licenses.yaml
 ##
 @@ -2807,20 +2806,6 @@ notices:
 
 ---
 
-name: Apache HttpClient
 
 Review comment:
   @clintropolis any further thoughts on this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on issue #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on issue #7363: Implementing dropwizard emitter for 
druid
URL: https://github.com/apache/incubator-druid/pull/7363#issuecomment-527034655
 
 
   @jihoonson  @b-slim : Updated PR based on review comments, please check. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833945
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterModule.java
 ##
 @@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.inject.Binder;
+import com.google.inject.Injector;
+import com.google.inject.Key;
+import com.google.inject.Provides;
+import com.google.inject.name.Named;
+import com.google.inject.name.Names;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.initialization.DruidModule;
+import org.apache.druid.java.util.emitter.core.Emitter;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class DropwizardEmitterModule implements DruidModule
+{
+  private static final String EMITTER_TYPE = "dropwizard";
+
+  @Override
+  public List getJacksonModules()
+  {
+return Collections.EMPTY_LIST;
+  }
+
+  @Override
+  public void configure(Binder binder)
+  {
+JsonConfigProvider.bind(binder, "druid.emitter." + EMITTER_TYPE, 
DropwizardEmitterConfig.class);
+  }
+
+  @Provides
+  @ManageLifecycle
 
 Review comment:
   removed. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833894
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/reporters/DropwizardJMXReporter.java
 ##
 @@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard.reporters;
+
+import com.codahale.metrics.JmxReporter;
+import com.codahale.metrics.MetricRegistry;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.emitter.dropwizard.DropwizardReporter;
+
+import java.util.Objects;
+
+@JsonTypeName("jmx")
 
 Review comment:
   removed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833658
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterConfig.java
 ##
 @@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+
+public class DropwizardEmitterConfig
+{
+  private static int DEFAULT_MAX_GAUGE_COUNT = 100_000;
+  @JsonProperty
+  private final List reporters;
+  @JsonProperty
+  private final String prefix;
+  @JsonProperty
+  private final Boolean includeHost;
+  @JsonProperty
+  private final String dimensionMapPath;
+  @JsonProperty
+  private final List alertEmitters;
+  @JsonProperty
+  private final int maxGaugeCount;
+
+  @JsonCreator
+  public DropwizardEmitterConfig(
+  @JsonProperty("reporters") List reporters,
+  @JsonProperty("prefix") String prefix,
+  @JsonProperty("includeHost") Boolean includeHost,
+  @JsonProperty("dimensionMapPath") String dimensionMapPath,
+  @JsonProperty("alertEmitters") List alertEmitters,
+  @JsonProperty("maxGaugeCount") Integer maxGaugeCount
+  )
+  {
+this.reporters = reporters;
+this.prefix = prefix;
+this.alertEmitters = alertEmitters == null ? Collections.emptyList() : 
alertEmitters;
+this.includeHost = includeHost != null ? includeHost : true;
+this.dimensionMapPath = dimensionMapPath;
+this.maxGaugeCount = maxGaugeCount == null ? DEFAULT_MAX_GAUGE_COUNT : 
maxGaugeCount;
+  }
+
+  @JsonProperty
+  public List getReporters()
+  {
+return reporters;
+  }
+
+  @JsonProperty
+  public String getPrefix()
+  {
+return prefix;
+  }
+
+  @JsonProperty
+  public Boolean getIncludeHost()
+  {
+return includeHost;
+  }
+
+  @JsonProperty
+  public String getDimensionMapPath()
+  {
+return dimensionMapPath;
+  }
+
+  @JsonProperty
+  public List getAlertEmitters()
+  {
+return alertEmitters;
+  }
+
+  @JsonProperty
+  public int getMaxGaugeCount()
+  {
+return maxGaugeCount;
+  }
+
+  @Override
+  public boolean equals(Object o)
+  {
+if (this == o) {
+  return true;
+}
+if (o == null || getClass() != o.getClass()) {
+  return false;
+}
+DropwizardEmitterConfig that = (DropwizardEmitterConfig) o;
+return maxGaugeCount == that.maxGaugeCount &&
+   Objects.equals(reporters, that.reporters) &&
+   Objects.equals(prefix, that.prefix) &&
+   Objects.equals(includeHost, that.includeHost) &&
+   Objects.equals(dimensionMapPath, that.dimensionMapPath) &&
+   Objects.equals(alertEmitters, that.alertEmitters);
+  }
+
+  @Override
+  public int hashCode()
+  {
+return Objects.hash(reporters, prefix, includeHost, dimensionMapPath, 
alertEmitters, maxGaugeCount);
+  }
+
+  @Override
+  public String toString()
+  {
+return "DropwizardEmitterConfig{" +
+   "reporters=" + reporters +
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833694
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/reporters/DropwizardConsoleReporter.java
 ##
 @@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard.reporters;
+
+import com.codahale.metrics.ConsoleReporter;
+import com.codahale.metrics.MetricRegistry;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.emitter.dropwizard.DropwizardReporter;
+
+import java.util.concurrent.TimeUnit;
+
+@JsonTypeName("console")
 
 Review comment:
   removed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833086
 
 

 ##
 File path: docs/content/development/extensions-contrib/dropwizard.md
 ##
 @@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to 
[include](../../operations/including-extensions.html) `dropwizard-emitter` 
extension.
+
+## Introduction
+
+This extension integrates 
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics 
library with druid so that dropwizard users can easily absorb druid into their 
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various 
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and 
histogram. 
+These metrics can be emitted using either Console or JMX reporter. 
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under 
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be 
used.|yes|none|
 
 Review comment:
   added


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319833057
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitter.java
 ##
 @@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.core.Emitter;
+import org.apache.druid.java.util.emitter.core.Event;
+import org.apache.druid.java.util.emitter.service.AlertEvent;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class DropwizardEmitter implements Emitter
+{
+  private static final Logger log = new Logger(DropwizardEmitter.class);
+  private final MetricRegistry metricsRegistry = new MetricRegistry();
+  private final AtomicBoolean started = new AtomicBoolean(false);
+  private final DropwizardConverter converter;
+  private final List alertEmitters;
+  private final List reporters;
+  private final DropwizardEmitterConfig config;
+  // Note: the gauges do not represent the actual instantaneous value for the 
metrics.
+  // Instead they have the last known value for it.
+  private final Map gagues;
+
+  public DropwizardEmitter(
+  DropwizardEmitterConfig config,
+  ObjectMapper mapper,
+  List alertEmitters
+  )
+  {
+this.alertEmitters = alertEmitters;
+this.config = config;
+this.reporters = config.getReporters();
+this.converter = new DropwizardConverter(mapper, 
config.getDimensionMapPath());
+this.gagues = Collections.synchronizedMap(new 
GaugesCache<>(config.getMaxGaugeCount()));
+  }
+
+
+  @Override
+  public void start()
+  {
+final boolean alreadyStarted = started.getAndSet(true);
+if (!alreadyStarted) {
+  for (DropwizardReporter reporter : reporters) {
+reporter.start(metricsRegistry);
+  }
+}
+  }
+
+  @Override
+  public void emit(Event event)
+  {
+synchronized (started) {
+  if (!started.get()) {
+throw new RejectedExecutionException("Dropwizard emitter Service not 
started.");
+  }
+}
+if (event instanceof ServiceMetricEvent) {
+  ServiceMetricEvent metricEvent = (ServiceMetricEvent) event;
+  String host = metricEvent.getHost();
+  String service = metricEvent.getService();
+  String metric = metricEvent.getMetric();
+  Map userDims = metricEvent.getUserDims();
+  Number value = metricEvent.getValue();
+  ImmutableList.Builder nameBuilder = new 
ImmutableList.Builder<>();
+  LinkedHashMap dims = new LinkedHashMap<>();
+  final DropwizardMetricSpec metricSpec = 
converter.addFilteredUserDims(service, metric, userDims, dims);
+
+  if (metricSpec != null) {
+if (config.getPrefix() != null) {
+  nameBuilder.add(config.getPrefix());
+}
+nameBuilder.add("metric=" + metric);
+nameBuilder.add("service=" + service);
+if (config.getIncludeHost()) {
+  nameBuilder.add("hostname=" + host);
+}
+dims.forEach((key, value1) -> nameBuilder.add(key + "=" + value1));
+
+String fullName = 
StringUtils.replaceChar(Joiner.on(",").join(nameBuilder.build()), '/', ".");
+updateMetric(fullName, value, metricSpec);
+  } else {
+log.debug("Service=[%s], Metric=[%s] has no mapping", service, metric);
+  }
+} else if (event instanceof AlertEvent) {
+  for (Emitter emitter : alertEmitters) {
+emitter.emit(event);
+  }
+} else {

[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832908
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardConverter.java
 ##
 @@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Strings;
+import org.apache.curator.shaded.com.google.common.io.Closeables;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+/**
+ *
+ */
+public class DropwizardConverter
+{
+  private static final Logger log = new Logger(DropwizardConverter.class);
+  private final Map metricMap;
+
+  public DropwizardConverter(ObjectMapper mapper, String dimensionMapPath)
+  {
+metricMap = readMap(mapper, dimensionMapPath);
+  }
+
+  /**
+   * Filters user dimensions for given metric and adds them to 
filteredDimensions.
+   * Returns null if there is no mapping present for the given metric.
+   */
+  @Nullable
+  public DropwizardMetricSpec addFilteredUserDims(
+  String service,
+  String metric,
+  Map userDims,
+  Map filteredDimensions
+  )
+  {
+ /*
+Find the metric in the map. If we cant find it try to look it up 
prefixed by the service name.
 
 Review comment:
   done. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832658
 
 

 ##
 File path: docs/content/development/extensions-contrib/dropwizard.md
 ##
 @@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to 
[include](../../operations/including-extensions.html) `dropwizard-emitter` 
extension.
+
+## Introduction
+
+This extension integrates 
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics 
library with druid so that dropwizard users can easily absorb druid into their 
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various 
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and 
histogram. 
+These metrics can be emitted using either Console or JMX reporter. 
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under 
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be 
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics 
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part 
of the metric name.|no|yes|
+|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the 
dropwizard metric type, and desired dimensions for every Druid 
metric|no|Default mapping provided. See below.|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will 
be forwarded to. |no| empty list (no forwarding)|
+|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge 
metrics to be cached at any time. |no| 100K|
+
+
+### Druid to Dropwizard Event Conversion
+
+Each metric emitted using Dropwizard must specify a type, one of `[timer, 
counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to
+be provided as a JSON file.  Additionally, this mapping specifies which 
dimensions should be included for each metric.
+If the user does not specify their own JSON file, a [default 
mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json)
 is used.
 
 Review comment:
   done. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832671
 
 

 ##
 File path: docs/content/development/extensions-contrib/dropwizard.md
 ##
 @@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to 
[include](../../operations/including-extensions.html) `dropwizard-emitter` 
extension.
+
+## Introduction
+
+This extension integrates 
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics 
library with druid so that dropwizard users can easily absorb druid into their 
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various 
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and 
histogram. 
+These metrics can be emitted using either Console or JMX reporter. 
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under 
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be 
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics 
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part 
of the metric name.|no|yes|
+|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the 
dropwizard metric type, and desired dimensions for every Druid 
metric|no|Default mapping provided. See below.|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will 
be forwarded to. |no| empty list (no forwarding)|
+|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge 
metrics to be cached at any time. |no| 100K|
+
+
+### Druid to Dropwizard Event Conversion
+
+Each metric emitted using Dropwizard must specify a type, one of `[timer, 
counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to
+be provided as a JSON file.  Additionally, this mapping specifies which 
dimensions should be included for each metric.
+If the user does not specify their own JSON file, a [default 
mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json)
 is used.
+All metrics are expected to be mapped. Metrics which are not mapped will be 
ignored.
+Dropwizard metric path is organized using the following schema:
+
+` : { "dimensions" : , "type" : , "timeUnit" : }`
+
+e.g.
+
+`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer", 
"timeUnit": "MILLISECONDS"}`
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832782
 
 

 ##
 File path: docs/content/development/extensions-contrib/dropwizard.md
 ##
 @@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to 
[include](../../operations/including-extensions.html) `dropwizard-emitter` 
extension.
+
+## Introduction
+
+This extension integrates 
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics 
library with druid so that dropwizard users can easily absorb druid into their 
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various 
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and 
histogram. 
+These metrics can be emitted using either Console or JMX reporter. 
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under 
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be 
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics 
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part 
of the metric name.|no|yes|
+|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the 
dropwizard metric type, and desired dimensions for every Druid 
metric|no|Default mapping provided. See below.|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will 
be forwarded to. |no| empty list (no forwarding)|
+|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge 
metrics to be cached at any time. |no| 100K|
+
+
+### Druid to Dropwizard Event Conversion
+
+Each metric emitted using Dropwizard must specify a type, one of `[timer, 
counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to
+be provided as a JSON file.  Additionally, this mapping specifies which 
dimensions should be included for each metric.
+If the user does not specify their own JSON file, a [default 
mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json)
 is used.
+All metrics are expected to be mapped. Metrics which are not mapped will be 
ignored.
+Dropwizard metric path is organized using the following schema:
+
+` : { "dimensions" : , "type" : , "timeUnit" : }`
+
+e.g.
+
+`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer", 
"timeUnit": "MILLISECONDS"}`
+`"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"}`
+
+
+For most use-cases, the default mapping is sufficient.
+
+### Supported Dropwizard reporters
+
+ JMX Reporter
+Used to report druid metrics via JMX.
+```json
+
+druid.emitter.dropwizard.reporter={"type":"jmx"}
+
+```
+
+ Console Reporter
+Used to print Druid Metrics to console logs.
+
+```json
 
 Review comment:
   removed json.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832857
 
 

 ##
 File path: extensions-contrib/dropwizard-emitter/pom.xml
 ##
 @@ -0,0 +1,79 @@
+
+
+
+http://www.w3.org/2001/XMLSchema-instance; 
xmlns="http://maven.apache.org/POM/4.0.0;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+  4.0.0
+
+  
+org.apache.druid
+druid
+0.16.0-incubating-SNAPSHOT
+../../pom.xml
+  
+
+  org.apache.druid.extensions.contrib
+  dropwizard-emitter
+  dropwizard-emitter
+  Druid emitter extension to convert druid metric to Dropwizard 
metrics
+
+  
+
+  org.apache.druid
+  druid-core
+  ${project.parent.version}
+  provided
+
+
+  org.apache.druid
+  druid-server
+  ${project.parent.version}
+  provided
+
+
+  io.dropwizard.metrics
+  metrics-core
+  3.1.0
 
 Review comment:
   changed to point to the core dependency version. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319832726
 
 

 ##
 File path: docs/content/development/extensions-contrib/dropwizard.md
 ##
 @@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to 
[include](../../operations/including-extensions.html) `dropwizard-emitter` 
extension.
+
+## Introduction
+
+This extension integrates 
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics 
library with druid so that dropwizard users can easily absorb druid into their 
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various 
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and 
histogram. 
+These metrics can be emitted using either Console or JMX reporter. 
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under 
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be 
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics 
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part 
of the metric name.|no|yes|
+|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the 
dropwizard metric type, and desired dimensions for every Druid 
metric|no|Default mapping provided. See below.|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will 
be forwarded to. |no| empty list (no forwarding)|
+|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge 
metrics to be cached at any time. |no| 100K|
+
+
+### Druid to Dropwizard Event Conversion
+
+Each metric emitted using Dropwizard must specify a type, one of `[timer, 
counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to
+be provided as a JSON file.  Additionally, this mapping specifies which 
dimensions should be included for each metric.
+If the user does not specify their own JSON file, a [default 
mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json)
 is used.
+All metrics are expected to be mapped. Metrics which are not mapped will be 
ignored.
+Dropwizard metric path is organized using the following schema:
+
+` : { "dimensions" : , "type" : , "timeUnit" : }`
+
+e.g.
+
+`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer", 
"timeUnit": "MILLISECONDS"}`
+`"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"}`
+
+
+For most use-cases, the default mapping is sufficient.
+
+### Supported Dropwizard reporters
+
+ JMX Reporter
+Used to report druid metrics via JMX.
+```json
 
 Review comment:
   removed json. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319831612
 
 

 ##
 File path: docs/content/development/extensions-contrib/dropwizard.md
 ##
 @@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to 
[include](../../operations/including-extensions.html) `dropwizard-emitter` 
extension.
+
+## Introduction
+
+This extension integrates 
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics 
library with druid so that dropwizard users can easily absorb druid into their 
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various 
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and 
histogram. 
+These metrics can be emitted using either Console or JMX reporter. 
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under 
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be 
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics 
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part 
of the metric name.|no|yes|
+|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the 
dropwizard metric type, and desired dimensions for every Druid 
metric|no|Default mapping provided. See below.|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will 
be forwarded to. |no| empty list (no forwarding)|
+|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge 
metrics to be cached at any time. |no| 100K|
+
+
+### Druid to Dropwizard Event Conversion
+
+Each metric emitted using Dropwizard must specify a type, one of `[timer, 
counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319831410
 
 

 ##
 File path: docs/content/development/extensions-contrib/dropwizard.md
 ##
 @@ -0,0 +1,86 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to 
[include](../../operations/including-extensions.html) `dropwizard-emitter` 
extension.
+
+## Introduction
+
+This extension integrates 
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics 
library with druid so that dropwizard users can easily absorb druid into their 
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various 
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and 
histogram. 
+These metrics can be emitted using either Console or JMX reporter. 
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under 
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be 
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics 
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part 
of the metric name.|no|yes|
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] lswyyy commented on issue #8438: druid-0.15.0运行后自动进程退出

2019-09-02 Thread GitBox
lswyyy commented on issue #8438: druid-0.15.0运行后自动进程退出
URL: 
https://github.com/apache/incubator-druid/issues/8438#issuecomment-527031277
 
 
   try to use "exit" command after "nohup bin/start-micro-quickstart &"


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319822249
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitter.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.core.Emitter;
+import org.apache.druid.java.util.emitter.core.Event;
+import org.apache.druid.java.util.emitter.service.AlertEvent;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class DropwizardEmitter implements Emitter
+{
+  private static Logger log = new Logger(DropwizardEmitter.class);
+  private final MetricRegistry metricsRegistry = new MetricRegistry();
+  private final AtomicBoolean started = new AtomicBoolean(false);
+  private final DropwizardConverter converter;
+  private final List alertEmitters;
+  private final List reporters;
+  private final DropwizardEmitterConfig config;
+  // Note: the gauges do not represent the actual instantaneous value for the 
metrics.
+  // Instead they have the last known value for it.
+  private final ConcurrentHashMap gagues = new 
ConcurrentHashMap<>();
+
+  public DropwizardEmitter(
+  DropwizardEmitterConfig config,
+  ObjectMapper mapper,
+  List alertEmitters
+  )
+  {
+this.alertEmitters = alertEmitters;
+this.config = config;
+this.reporters = config.getReporters();
+this.converter = new DropwizardConverter(mapper, 
config.getDimensionMapPath());
+  }
+
+
+  @Override
+  public void start()
 
 Review comment:
   yeah, it should be called through lifecycle, It seems other emitter has 
guards too so i added a it here also. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319822306
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterModule.java
 ##
 @@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Lists;
+import com.google.inject.Binder;
+import com.google.inject.Injector;
+import com.google.inject.Key;
+import com.google.inject.Provides;
+import com.google.inject.name.Named;
+import com.google.inject.name.Names;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.initialization.DruidModule;
+import org.apache.druid.java.util.emitter.core.Emitter;
+
+import java.util.Collections;
+import java.util.List;
+
+public class DropwizardEmitterModule implements DruidModule
+{
+  private static final String EMITTER_TYPE = "dropwizard";
+
+  @Override
+  public List getJacksonModules()
+  {
+return Collections.EMPTY_LIST;
+  }
+
+  @Override
+  public void configure(Binder binder)
+  {
+JsonConfigProvider.bind(binder, "druid.emitter." + EMITTER_TYPE, 
DropwizardEmitterConfig.class);
+  }
+
+  @Provides
+  @ManageLifecycle
+  @Named(EMITTER_TYPE)
+  public Emitter getEmitter(
+  DropwizardEmitterConfig dropwizardEmitterConfig,
+  ObjectMapper mapper,
+  final Injector injector
+  )
+  {
+List alertEmitters = Lists.transform(
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #7363: Implementing dropwizard emitter for druid

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #7363: Implementing 
dropwizard emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r319821801
 
 

 ##
 File path: 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardConverter.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Strings;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+/**
+ *
+ */
+public class DropwizardConverter
+{
+  private static final Logger log = new Logger(DropwizardConverter.class);
+  private Map metricMap;
+
+  public DropwizardConverter(ObjectMapper mapper, String dimensionMapPath)
+  {
+metricMap = readMap(mapper, dimensionMapPath);
+  }
+
+  @Nullable
+  public DropwizardMetricSpec addFilteredUserDims(
+  String service,
+  String metric,
+  Map userDims,
+  Map builder
+  )
+  {
+ /*
+Find the metric in the map. If we cant find it try to look it up 
prefixed by the service name.
+This is because some metrics are reported differently, but with the 
same name, from different services.
+   */
+DropwizardMetricSpec metricSpec = null;
+if (metricMap.containsKey(metric)) {
+  metricSpec = metricMap.get(metric);
+} else if (metricMap.containsKey(service + "-" + metric)) {
+  metricSpec = metricMap.get(service + "-" + metric);
+}
+if (metricSpec != null) {
+  for (String dim : metricSpec.getDimensions()) {
+if (userDims.containsKey(dim)) {
+  builder.put(dim, userDims.get(dim).toString());
+}
+  }
+  return metricSpec;
+} else {
+  return null;
+}
+  }
+
+  private Map readMap(ObjectMapper mapper, 
String dimensionMapPath)
+  {
+try {
+  InputStream is;
+  if (Strings.isNullOrEmpty(dimensionMapPath)) {
+log.info("Using default metric dimension and types");
+is = 
this.getClass().getClassLoader().getResourceAsStream("defaultMetricDimensions.json");
+  } else {
+log.info("Using metric dimensions at types at [%s]", dimensionMapPath);
+is = new FileInputStream(new File(dimensionMapPath));
+  }
+  return mapper.readerFor(new TypeReference>()
+  {
+  }).readValue(is);
+}
+catch (IOException e) {
 
 Review comment:
   thought ObjectReader.readValue will close the stream in normal flow


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on issue #7537: Avoid file rename when pushing segments with HDFSDataPusher

2019-09-02 Thread GitBox
nishantmonu51 commented on issue #7537:  Avoid file rename when pushing 
segments with HDFSDataPusher
URL: https://github.com/apache/incubator-druid/pull/7537#issuecomment-527019852
 
 
   ping @gianm: Any more comments here ?  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] commented on issue #7537: Avoid file rename when pushing segments with HDFSDataPusher

2019-09-02 Thread GitBox
stale[bot] commented on issue #7537:  Avoid file rename when pushing segments 
with HDFSDataPusher
URL: https://github.com/apache/incubator-druid/pull/7537#issuecomment-527019749
 
 
   This pull request/issue is no longer marked as stale.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on issue #8248: Add TrustedDomain Authenticator

2019-09-02 Thread GitBox
nishantmonu51 commented on issue #8248: Add TrustedDomain Authenticator
URL: https://github.com/apache/incubator-druid/pull/8248#issuecomment-527019385
 
 
   @jon-wei : Yes, its possible to spoof X-forwarded header, that is one of the 
reasons its not allowed by default. 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #8248: Add TrustedDomain Authenticator

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #8248: Add TrustedDomain 
Authenticator
URL: https://github.com/apache/incubator-druid/pull/8248#discussion_r319819357
 
 

 ##
 File path: 
server/src/main/java/org/apache/druid/server/security/TrustedDomainAuthenticator.java
 ##
 @@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.security;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.annotation.Nullable;
+import javax.servlet.DispatcherType;
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServletRequest;
+import java.io.IOException;
+import java.util.EnumSet;
+import java.util.Map;
+
+/**
+ * Authenticates requests coming from a specific domain and directs them to an 
authorizer.
+ */
+@JsonTypeName(AuthConfig.TRUSTED_DOMAIN_NAME)
+public class TrustedDomainAuthenticator implements Authenticator
+{
+  private static final Logger LOGGER = new 
Logger(TrustedDomainAuthenticator.class);
+  private static final String DEFAULT_IDENTITY = "defaultUser";
+  private static final String X_FORWARDED_FOR = "X-Forwarded-For";
+
+  private static final boolean DEFAULT_USE_FORWARDED_HEADERS = false;
+  private final AuthenticationResult authenticationResult;
+  private final String domain;
+  private final boolean useForwardedHeaders;
+
+  @JsonCreator
+  public TrustedDomainAuthenticator(
+  @JsonProperty("name") String name,
+  @JsonProperty("domain") String domain,
+  @JsonProperty("useForwardedHeaders") Boolean useForwardedHeaders,
+  @JsonProperty("authorizerName") String authorizerName,
+  @JsonProperty("identity") String identity
+  )
+  {
+Preconditions.checkArgument(!Strings.isNullOrEmpty(domain), "Invalid 
domain name %s", domain);
+this.domain = domain;
+this.useForwardedHeaders = useForwardedHeaders == null ? 
DEFAULT_USE_FORWARDED_HEADERS : useForwardedHeaders;
+this.authenticationResult = new AuthenticationResult(
+identity == null ? DEFAULT_IDENTITY : identity,
+authorizerName,
+name,
+null
+);
+  }
+
+  @Override
+  public Class getFilterClass()
+  {
+return null;
+  }
+
+  @Override
+  public Map getInitParameters()
+  {
+return null;
+  }
+
+  @Override
+  public String getPath()
+  {
+return "/*";
+  }
+
+  @Override
+  public EnumSet getDispatcherType()
+  {
+return null;
 
 Review comment:
   added


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] nishantmonu51 commented on a change in pull request #8248: Add TrustedDomain Authenticator

2019-09-02 Thread GitBox
nishantmonu51 commented on a change in pull request #8248: Add TrustedDomain 
Authenticator
URL: https://github.com/apache/incubator-druid/pull/8248#discussion_r319819341
 
 

 ##
 File path: 
server/src/main/java/org/apache/druid/server/security/TrustedDomainAuthenticator.java
 ##
 @@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.security;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.annotation.Nullable;
+import javax.servlet.DispatcherType;
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServletRequest;
+import java.io.IOException;
+import java.util.EnumSet;
+import java.util.Map;
+
+/**
+ * Authenticates requests coming from a specific domain and directs them to an 
authorizer.
+ */
+@JsonTypeName(AuthConfig.TRUSTED_DOMAIN_NAME)
+public class TrustedDomainAuthenticator implements Authenticator
+{
+  private static final Logger LOGGER = new 
Logger(TrustedDomainAuthenticator.class);
+  private static final String DEFAULT_IDENTITY = "defaultUser";
+  private static final String X_FORWARDED_FOR = "X-Forwarded-For";
+
+  private static final boolean DEFAULT_USE_FORWARDED_HEADERS = false;
+  private final AuthenticationResult authenticationResult;
+  private final String domain;
+  private final boolean useForwardedHeaders;
+
+  @JsonCreator
+  public TrustedDomainAuthenticator(
+  @JsonProperty("name") String name,
+  @JsonProperty("domain") String domain,
+  @JsonProperty("useForwardedHeaders") Boolean useForwardedHeaders,
+  @JsonProperty("authorizerName") String authorizerName,
+  @JsonProperty("identity") String identity
+  )
+  {
+Preconditions.checkArgument(!Strings.isNullOrEmpty(domain), "Invalid 
domain name %s", domain);
+this.domain = domain;
+this.useForwardedHeaders = useForwardedHeaders == null ? 
DEFAULT_USE_FORWARDED_HEADERS : useForwardedHeaders;
+this.authenticationResult = new AuthenticationResult(
+identity == null ? DEFAULT_IDENTITY : identity,
+authorizerName,
+name,
+null
+);
+  }
+
+  @Override
+  public Class getFilterClass()
+  {
+return null;
+  }
+
+  @Override
+  public Map getInitParameters()
+  {
+return null;
 
 Review comment:
   added


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org