[GitHub] incubator-eagle issue #556: EAGLE-670: make kafka publisher configurable and...
Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/556 The configuration is ok. The primary concern here is the `async` implementation, kafka producer natively support `async` mode, so that you just need pass through it instead of handle `async` thread manually which will look not very clean. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[ https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601322#comment-15601322 ] ASF GitHub Bot commented on EAGLE-670: -- Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/556 The configuration is ok. The primary concern here is the `async` implementation, kafka producer natively support `async` mode, so that you just need pass through it instead of handle `async` thread manually which will look not very clean. > AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning > -- > > Key: EAGLE-670 > URL: https://issues.apache.org/jira/browse/EAGLE-670 > Project: Eagle > Issue Type: Task >Affects Versions: v0.5.0 >Reporter: Su Ralph >Assignee: Garrett Li > Fix For: v0.5.0 > > > Kafka send alert in sync would limit the throughput. Make this configurable, > and use async by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[ https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601327#comment-15601327 ] ASF GitHub Bot commented on EAGLE-670: -- Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/556 BTW: these selected config keys are what we have evaluated in real-case and impact kafka throughput mostly, instead of not requiring you guys to include additional many kafka configurations ~~~ producer.type = async batch.num.messages = 3000 queue.buffering.max.ms = 5000 queue.buffering.max.messages = 1 ~~~ > AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning > -- > > Key: EAGLE-670 > URL: https://issues.apache.org/jira/browse/EAGLE-670 > Project: Eagle > Issue Type: Task >Affects Versions: v0.5.0 >Reporter: Su Ralph >Assignee: Garrett Li > Fix For: v0.5.0 > > > Kafka send alert in sync would limit the throughput. Make this configurable, > and use async by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #556: EAGLE-670: make kafka publisher configurable and...
Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/556 BTW: these selected config keys are what we have evaluated in real-case and impact kafka throughput mostly, instead of not requiring you guys to include additional many kafka configurations ~~~ producer.type = async batch.num.messages = 3000 queue.buffering.max.ms = 5000 queue.buffering.max.messages = 1 ~~~ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request #556: EAGLE-670: make kafka publisher configura...
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/556#discussion_r84639212
--- Diff:
eagle-core/eagle-alert-parent/eagle-alert/alert-engine/src/main/java/org/apache/eagle/alert/engine/publisher/impl/AlertKafkaPublisher.java
---
@@ -96,44 +103,63 @@ public void close() {
}
@SuppressWarnings( {"rawtypes", "unchecked"})
-protected PublishStatus emit(String topic, List
outputEvents) {
+protected void emit(String topic, List outputEvents)
{
// we need to check producer here since the producer is invisable
to extended kafka publisher
if (producer == null) {
LOG.warn("KafkaProducer is null due to the incorrect
configurations");
-return null;
+return;
}
if (outputEvents == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Alert stream events list in publishment is
empty");
}
-return null;
+return;
}
this.status = new PublishStatus();
try {
for (AlertStreamEvent outputEvent : outputEvents) {
ProducerRecord record = createRecord(outputEvent, topic);
if (record == null) {
LOG.error("Alert serialize return null, ignored
message! ");
-return null;
+return;
+}
+if (mode == KafkaWriteMode.sync) {
+Future future = producer.send(record);
+future.get(MAX_TIMEOUT_MS, TimeUnit.MILLISECONDS);
+succeed(mode, "");
+} else {
+producer.send(record, new Callback() {
--- End diff --
This is not `async` mode in kafka language
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[
https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601333#comment-15601333
]
ASF GitHub Bot commented on EAGLE-670:
--
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/556#discussion_r84639212
--- Diff:
eagle-core/eagle-alert-parent/eagle-alert/alert-engine/src/main/java/org/apache/eagle/alert/engine/publisher/impl/AlertKafkaPublisher.java
---
@@ -96,44 +103,63 @@ public void close() {
}
@SuppressWarnings( {"rawtypes", "unchecked"})
-protected PublishStatus emit(String topic, List
outputEvents) {
+protected void emit(String topic, List outputEvents)
{
// we need to check producer here since the producer is invisable
to extended kafka publisher
if (producer == null) {
LOG.warn("KafkaProducer is null due to the incorrect
configurations");
-return null;
+return;
}
if (outputEvents == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Alert stream events list in publishment is
empty");
}
-return null;
+return;
}
this.status = new PublishStatus();
try {
for (AlertStreamEvent outputEvent : outputEvents) {
ProducerRecord record = createRecord(outputEvent, topic);
if (record == null) {
LOG.error("Alert serialize return null, ignored
message! ");
-return null;
+return;
+}
+if (mode == KafkaWriteMode.sync) {
+Future future = producer.send(record);
+future.get(MAX_TIMEOUT_MS, TimeUnit.MILLISECONDS);
+succeed(mode, "");
+} else {
+producer.send(record, new Callback() {
--- End diff --
This is not `async` mode in kafka language
> AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
> --
>
> Key: EAGLE-670
> URL: https://issues.apache.org/jira/browse/EAGLE-670
> Project: Eagle
> Issue Type: Task
>Affects Versions: v0.5.0
>Reporter: Su Ralph
>Assignee: Garrett Li
> Fix For: v0.5.0
>
>
> Kafka send alert in sync would limit the throughput. Make this configurable,
> and use async by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-eagle pull request #556: EAGLE-670: make kafka publisher configura...
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/556#discussion_r84639463
--- Diff:
eagle-core/eagle-alert-parent/eagle-alert/alert-engine/src/main/java/org/apache/eagle/alert/engine/publisher/impl/AlertKafkaPublisher.java
---
@@ -96,44 +103,63 @@ public void close() {
}
@SuppressWarnings( {"rawtypes", "unchecked"})
-protected PublishStatus emit(String topic, List
outputEvents) {
+protected void emit(String topic, List outputEvents)
{
// we need to check producer here since the producer is invisable
to extended kafka publisher
if (producer == null) {
LOG.warn("KafkaProducer is null due to the incorrect
configurations");
-return null;
+return;
}
if (outputEvents == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Alert stream events list in publishment is
empty");
}
-return null;
+return;
}
this.status = new PublishStatus();
try {
for (AlertStreamEvent outputEvent : outputEvents) {
ProducerRecord record = createRecord(outputEvent, topic);
if (record == null) {
LOG.error("Alert serialize return null, ignored
message! ");
-return null;
+return;
+}
+if (mode == KafkaWriteMode.sync) {
+Future future = producer.send(record);
+future.get(MAX_TIMEOUT_MS, TimeUnit.MILLISECONDS);
+succeed(mode, "");
+} else {
+producer.send(record, new Callback() {
--- End diff --
Oh ... I got what you did now, but what we talk about `async` means "kafka
send message with batch in async thread"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[
https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601336#comment-15601336
]
ASF GitHub Bot commented on EAGLE-670:
--
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/556#discussion_r84639463
--- Diff:
eagle-core/eagle-alert-parent/eagle-alert/alert-engine/src/main/java/org/apache/eagle/alert/engine/publisher/impl/AlertKafkaPublisher.java
---
@@ -96,44 +103,63 @@ public void close() {
}
@SuppressWarnings( {"rawtypes", "unchecked"})
-protected PublishStatus emit(String topic, List
outputEvents) {
+protected void emit(String topic, List outputEvents)
{
// we need to check producer here since the producer is invisable
to extended kafka publisher
if (producer == null) {
LOG.warn("KafkaProducer is null due to the incorrect
configurations");
-return null;
+return;
}
if (outputEvents == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Alert stream events list in publishment is
empty");
}
-return null;
+return;
}
this.status = new PublishStatus();
try {
for (AlertStreamEvent outputEvent : outputEvents) {
ProducerRecord record = createRecord(outputEvent, topic);
if (record == null) {
LOG.error("Alert serialize return null, ignored
message! ");
-return null;
+return;
+}
+if (mode == KafkaWriteMode.sync) {
+Future future = producer.send(record);
+future.get(MAX_TIMEOUT_MS, TimeUnit.MILLISECONDS);
+succeed(mode, "");
+} else {
+producer.send(record, new Callback() {
--- End diff --
Oh ... I got what you did now, but what we talk about `async` means "kafka
send message with batch in async thread"
> AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
> --
>
> Key: EAGLE-670
> URL: https://issues.apache.org/jira/browse/EAGLE-670
> Project: Eagle
> Issue Type: Task
>Affects Versions: v0.5.0
>Reporter: Su Ralph
>Assignee: Garrett Li
> Fix For: v0.5.0
>
>
> Kafka send alert in sync would limit the throughput. Make this configurable,
> and use async by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (EAGLE-674) fix grouping between AlertBolt and AlertPublisher
wujinhu created EAGLE-674: - Summary: fix grouping between AlertBolt and AlertPublisher Key: EAGLE-674 URL: https://issues.apache.org/jira/browse/EAGLE-674 Project: Eagle Issue Type: Bug Affects Versions: v0.5.0 Reporter: wujinhu Assignee: wujinhu Fix For: v0.5.0 Currently, it is fields grouping boltDeclarer.fieldsGrouping(alertBoltNamePrefix + i, new Fields(AlertConstants.FIELD_0)); Sometimes, traffic will go to only one task of alert publisher, we should change to shuffleGrouping -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-674) fix grouping between AlertBolt and AlertPublisher
[ https://issues.apache.org/jira/browse/EAGLE-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601379#comment-15601379 ] ASF GitHub Bot commented on EAGLE-674: -- GitHub user wujinhu opened a pull request: https://github.com/apache/incubator-eagle/pull/557 [EAGLE-674] fix grouping between AlertBolt and AlertPublisher You can merge this pull request into a Git repository by running: $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-673 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/557.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #557 commit c82fe370ef20db5437f072bf25ec90f4f2917bba Author: wujinhu Date: 2016-10-24T05:54:02Z add numOfPublishExecutors to alert engine topology commit d25bfae642115560ccfc87218c168b52aa389d92 Author: wujinhu Date: 2016-10-24T08:44:34Z fix alertPublish grouping bug > fix grouping between AlertBolt and AlertPublisher > - > > Key: EAGLE-674 > URL: https://issues.apache.org/jira/browse/EAGLE-674 > Project: Eagle > Issue Type: Bug >Affects Versions: v0.5.0 >Reporter: wujinhu >Assignee: wujinhu > Fix For: v0.5.0 > > > Currently, it is fields grouping > boltDeclarer.fieldsGrouping(alertBoltNamePrefix + i, new > Fields(AlertConstants.FIELD_0)); > Sometimes, traffic will go to only one task of alert publisher, we should > change to shuffleGrouping -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #557: [EAGLE-674] fix grouping between AlertBol...
GitHub user wujinhu opened a pull request: https://github.com/apache/incubator-eagle/pull/557 [EAGLE-674] fix grouping between AlertBolt and AlertPublisher You can merge this pull request into a Git repository by running: $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-673 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/557.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #557 commit c82fe370ef20db5437f072bf25ec90f4f2917bba Author: wujinhu Date: 2016-10-24T05:54:02Z add numOfPublishExecutors to alert engine topology commit d25bfae642115560ccfc87218c168b52aa389d92 Author: wujinhu Date: 2016-10-24T08:44:34Z fix alertPublish grouping bug --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (EAGLE-674) fix grouping between AlertBolt and AlertPublisher
[ https://issues.apache.org/jira/browse/EAGLE-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu closed EAGLE-674. - Resolution: Fixed > fix grouping between AlertBolt and AlertPublisher > - > > Key: EAGLE-674 > URL: https://issues.apache.org/jira/browse/EAGLE-674 > Project: Eagle > Issue Type: Bug >Affects Versions: v0.5.0 >Reporter: wujinhu >Assignee: wujinhu > Fix For: v0.5.0 > > > Currently, it is fields grouping > boltDeclarer.fieldsGrouping(alertBoltNamePrefix + i, new > Fields(AlertConstants.FIELD_0)); > Sometimes, traffic will go to only one task of alert publisher, we should > change to shuffleGrouping -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #557: [EAGLE-674] fix grouping between AlertBol...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/557 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-674) fix grouping between AlertBolt and AlertPublisher
[ https://issues.apache.org/jira/browse/EAGLE-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601436#comment-15601436 ] ASF GitHub Bot commented on EAGLE-674: -- Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/557 > fix grouping between AlertBolt and AlertPublisher > - > > Key: EAGLE-674 > URL: https://issues.apache.org/jira/browse/EAGLE-674 > Project: Eagle > Issue Type: Bug >Affects Versions: v0.5.0 >Reporter: wujinhu >Assignee: wujinhu > Fix For: v0.5.0 > > > Currently, it is fields grouping > boltDeclarer.fieldsGrouping(alertBoltNamePrefix + i, new > Fields(AlertConstants.FIELD_0)); > Sometimes, traffic will go to only one task of alert publisher, we should > change to shuffleGrouping -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[
https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601509#comment-15601509
]
ASF GitHub Bot commented on EAGLE-670:
--
Github user garrettlish commented on the issue:
https://github.com/apache/incubator-eagle/pull/556
Thx @haoch @RalphSu. I thought we set async for kafka producer and what we
need to change is using callback rather than future wait, it is wrong, thanks
for pointing out.
I have updated the code to add kafka_client_config (list of name/value map)
to specify kafka producer configurations.
By default, I only set producer.type=async.
For batch.num.messages, queue.buffering.max.ms and
queue.buffering.max.messages, I think we can use kafka producer default
values.
The only difference for default value is batch.num.messages, it is 200 if
not specified. Could u please share with us what is your reason to set it to
3000?
the kafka producer properties could be defined in publish properties as
follows:
{
"name": "***",
"properties" : {
"kafka_broker": "***",
"topics": "***",
"kafka_client_config" : [
{
"name" : "request.requrie.acks",
"value": 1
},
{
"name" : "producer.type",
"value": "async"
},
...
]
}
}
> AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
> --
>
> Key: EAGLE-670
> URL: https://issues.apache.org/jira/browse/EAGLE-670
> Project: Eagle
> Issue Type: Task
>Affects Versions: v0.5.0
>Reporter: Su Ralph
>Assignee: Garrett Li
> Fix For: v0.5.0
>
>
> Kafka send alert in sync would limit the throughput. Make this configurable,
> and use async by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-eagle issue #556: EAGLE-670: make kafka publisher configurable and...
Github user garrettlish commented on the issue:
https://github.com/apache/incubator-eagle/pull/556
Thx @haoch @RalphSu. I thought we set async for kafka producer and what we
need to change is using callback rather than future wait, it is wrong, thanks
for pointing out.
I have updated the code to add kafka_client_config (list of name/value map)
to specify kafka producer configurations.
By default, I only set producer.type=async.
For batch.num.messages, queue.buffering.max.ms and
queue.buffering.max.messages, I think we can use kafka producer default
values.
The only difference for default value is batch.num.messages, it is 200 if
not specified. Could u please share with us what is your reason to set it to
3000?
the kafka producer properties could be defined in publish properties as
follows:
{
"name": "***",
"properties" : {
"kafka_broker": "***",
"topics": "***",
"kafka_client_config" : [
{
"name" : "request.requrie.acks",
"value": 1
},
{
"name" : "producer.type",
"value": "async"
},
...
]
}
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Created] (EAGLE-675) AlertEngine: don't host long-live curator framework for schedule
Su Ralph created EAGLE-675: -- Summary: AlertEngine: don't host long-live curator framework for schedule Key: EAGLE-675 URL: https://issues.apache.org/jira/browse/EAGLE-675 Project: Eagle Issue Type: Bug Affects Versions: v0.5.0 Reporter: Su Ralph Assignee: Zeng, Bryant Fix For: v0.5.0 In Coordinator, the config bus producer is hold as member. This is try to reuse connection for continous operation. But this is buggy, since when zk server in network issue, curator framework would generate a lot of logs that flood the log file. Same for ExclusiveExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #556: EAGLE-670: make kafka publisher configurable and...
Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/556 The default value is just for example :-). And when using kafka producer `async` mode and throughput becomes extremely larger, batch size is one of the most important configurations for tuning. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[ https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601536#comment-15601536 ] ASF GitHub Bot commented on EAGLE-670: -- Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/556 The default value is just for example :-). And when using kafka producer `async` mode and throughput becomes extremely larger, batch size is one of the most important configurations for tuning. > AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning > -- > > Key: EAGLE-670 > URL: https://issues.apache.org/jira/browse/EAGLE-670 > Project: Eagle > Issue Type: Task >Affects Versions: v0.5.0 >Reporter: Su Ralph >Assignee: Garrett Li > Fix For: v0.5.0 > > > Kafka send alert in sync would limit the throughput. Make this configurable, > and use async by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #556: EAGLE-670: make kafka publisher configurable and...
Github user garrettlish commented on the issue: https://github.com/apache/incubator-eagle/pull/556 Yes, cool, then we can keep the current implementation and override the batch size configuration in publish properties kafka_client_config :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[ https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601547#comment-15601547 ] ASF GitHub Bot commented on EAGLE-670: -- Github user garrettlish commented on the issue: https://github.com/apache/incubator-eagle/pull/556 Yes, cool, then we can keep the current implementation and override the batch size configuration in publish properties kafka_client_config :-) > AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning > -- > > Key: EAGLE-670 > URL: https://issues.apache.org/jira/browse/EAGLE-670 > Project: Eagle > Issue Type: Task >Affects Versions: v0.5.0 >Reporter: Su Ralph >Assignee: Garrett Li > Fix For: v0.5.0 > > > Kafka send alert in sync would limit the throughput. Make this configurable, > and use async by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[ https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601677#comment-15601677 ] ASF GitHub Bot commented on EAGLE-670: -- Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/556 @garrettlish thanks. > AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning > -- > > Key: EAGLE-670 > URL: https://issues.apache.org/jira/browse/EAGLE-670 > Project: Eagle > Issue Type: Task >Affects Versions: v0.5.0 >Reporter: Su Ralph >Assignee: Garrett Li > Fix For: v0.5.0 > > > Kafka send alert in sync would limit the throughput. Make this configurable, > and use async by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #556: EAGLE-670: make kafka publisher configurable and...
Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/556 @garrettlish thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request #556: EAGLE-670: make kafka publisher configura...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/556 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-670) AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning
[ https://issues.apache.org/jira/browse/EAGLE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601690#comment-15601690 ] ASF GitHub Bot commented on EAGLE-670: -- Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/556 > AlertEngine: Make Kafka Publisher configurable to async, for throughput tuning > -- > > Key: EAGLE-670 > URL: https://issues.apache.org/jira/browse/EAGLE-670 > Project: Eagle > Issue Type: Task >Affects Versions: v0.5.0 >Reporter: Su Ralph >Assignee: Garrett Li > Fix For: v0.5.0 > > > Kafka send alert in sync would limit the throughput. Make this configurable, > and use async by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #552: [EAGLE-650] Hadoop JMX Metric Monitoring ...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/552 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-650) Hadoop JMX Metric Monitoring Application
[ https://issues.apache.org/jira/browse/EAGLE-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601719#comment-15601719 ] ASF GitHub Bot commented on EAGLE-650: -- Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/552 > Hadoop JMX Metric Monitoring Application > > > Key: EAGLE-650 > URL: https://issues.apache.org/jira/browse/EAGLE-650 > Project: Eagle > Issue Type: New Feature >Affects Versions: v0.5.0 >Reporter: Hao Chen >Assignee: Hao Chen > Fix For: v0.5.0 > > > Hadoop JMX Monitoring Application > * Hadoop Namenode JMX > * Hadoop Resource Manager JMX -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (EAGLE-650) Hadoop JMX Metric Monitoring Application
[ https://issues.apache.org/jira/browse/EAGLE-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen resolved EAGLE-650. Resolution: Fixed > Hadoop JMX Metric Monitoring Application > > > Key: EAGLE-650 > URL: https://issues.apache.org/jira/browse/EAGLE-650 > Project: Eagle > Issue Type: New Feature >Affects Versions: v0.5.0 >Reporter: Hao Chen >Assignee: Hao Chen > Fix For: v0.5.0 > > > Hadoop JMX Monitoring Application > * Hadoop Namenode JMX > * Hadoop Resource Manager JMX -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (EAGLE-676) Package hadoop jmx metric collector into eagle pacakge
Hao Chen created EAGLE-676: -- Summary: Package hadoop jmx metric collector into eagle pacakge Key: EAGLE-676 URL: https://issues.apache.org/jira/browse/EAGLE-676 Project: Eagle Issue Type: Improvement Affects Versions: v0.5.0 Reporter: Hao Chen Assignee: Hao Chen Fix For: v0.5.0 Package hadoop jmx metric collector into eagle pacakge -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (EAGLE-677) Hadoop JMX Collector Support HA switching
Hao Chen created EAGLE-677: -- Summary: Hadoop JMX Collector Support HA switching Key: EAGLE-677 URL: https://issues.apache.org/jira/browse/EAGLE-677 Project: Eagle Issue Type: Improvement Affects Versions: v0.5.0 Reporter: Hao Chen Assignee: Hao Chen Fix For: v0.5.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (EAGLE-678) Support to configure links for site
Hao Chen created EAGLE-678: -- Summary: Support to configure links for site Key: EAGLE-678 URL: https://issues.apache.org/jira/browse/EAGLE-678 Project: Eagle Issue Type: Improvement Affects Versions: v0.5.0 Reporter: Hao Chen Assignee: Hao Chen Fix For: v0.5.0 Support to configure links for site. As to Hadoop, support set: * Name Node URL * Resource Manager URL As to HBase, support: * HBase master URL Spark: * Spark History Server URL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Podling Report Reminder - November 2016
Dear podling, This email was sent by an automated system on behalf of the Apache Incubator PMC. It is an initial reminder to give you plenty of time to prepare your quarterly board report. The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT. The report for your podling will form a part of the Incubator PMC report. The Incubator PMC requires your report to be submitted 2 weeks before the board meeting, to allow sufficient time for review and submission (Wed, November 02). Please submit your report with sufficient time to allow the Incubator PMC, and subsequently board members to review and digest. Again, the very latest you should submit your report is 2 weeks prior to the board meeting. Thanks, The Apache Incubator PMC Submitting your Report -- Your report should contain the following: * Your project name * A brief description of your project, which assumes no knowledge of the project or necessarily of its field * A list of the three most important issues to address in the move towards graduation. * Any issues that the Incubator PMC or ASF Board might wish/need to be aware of * How has the community developed since the last report * How has the project developed since the last report. This should be appended to the Incubator Wiki page at: http://wiki.apache.org/incubator/November2016 Note: This is manually populated. You may need to wait a little before this page is created from a template. Mentors --- Mentors should review reports for their project(s) and sign them off on the Incubator wiki page. Signing off reports shows that you are following the project - projects that are not signed may raise alarms for the Incubator PMC. Incubator PMC
[jira] [Created] (EAGLE-679) Fix missing CoprocessorService and MR Running feeder task number limitation
wujinhu created EAGLE-679: - Summary: Fix missing CoprocessorService and MR Running feeder task number limitation Key: EAGLE-679 URL: https://issues.apache.org/jira/browse/EAGLE-679 Project: Eagle Issue Type: Bug Affects Versions: v0.5.0 Reporter: wujinhu Assignee: wujinhu Fix For: v0.5.0 1. fix missing CoprocessorService ERROR [2016-10-25 02:56:34,495] net.sf.extcos.internal.JavaResourceAccessor: Unable to read interface [org.apache.hadoop.hbase.coprocessor.CoprocessorService] ! java.io.IOException: Class not found 2. If task number of a job greater than 5000, then does not fetch task details -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #558: [EAGLE-679] Fix missing CoprocessorServic...
GitHub user wujinhu opened a pull request: https://github.com/apache/incubator-eagle/pull/558 [EAGLE-679] Fix missing CoprocessorService and MR Running feeder task number limi⦠You can merge this pull request into a Git repository by running: $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-679 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/558.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #558 commit e8504f369457d6cc301009b61545f294b8cbbf4e Author: wujinhu Date: 2016-10-25T03:22:45Z Fix missing CoprocessorService and MR Running feeder task number limitation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-679) Fix missing CoprocessorService and MR Running feeder task number limitation
[ https://issues.apache.org/jira/browse/EAGLE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604039#comment-15604039 ] ASF GitHub Bot commented on EAGLE-679: -- GitHub user wujinhu opened a pull request: https://github.com/apache/incubator-eagle/pull/558 [EAGLE-679] Fix missing CoprocessorService and MR Running feeder task number limi… You can merge this pull request into a Git repository by running: $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-679 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/558.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #558 commit e8504f369457d6cc301009b61545f294b8cbbf4e Author: wujinhu Date: 2016-10-25T03:22:45Z Fix missing CoprocessorService and MR Running feeder task number limitation > Fix missing CoprocessorService and MR Running feeder task number limitation > --- > > Key: EAGLE-679 > URL: https://issues.apache.org/jira/browse/EAGLE-679 > Project: Eagle > Issue Type: Bug >Affects Versions: v0.5.0 >Reporter: wujinhu >Assignee: wujinhu > Fix For: v0.5.0 > > > 1. fix missing CoprocessorService > ERROR [2016-10-25 02:56:34,495] net.sf.extcos.internal.JavaResourceAccessor: > Unable to read interface > [org.apache.hadoop.hbase.coprocessor.CoprocessorService] > ! java.io.IOException: Class not found > 2. If task number of a job greater than 5000, then does not fetch task details -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (EAGLE-680) Can't generate Html coverage report
Chang chen created EAGLE-680: Summary: Can't generate Html coverage report Key: EAGLE-680 URL: https://issues.apache.org/jira/browse/EAGLE-680 Project: Eagle Issue Type: Improvement Affects Versions: v0.5.0 Reporter: Chang chen Assignee: Lingang Deng Fix For: v0.5.0 The Current cobertura plugin just generate xml format by default which is inconvenient for local verification, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #558: [EAGLE-679] Fix missing CoprocessorServic...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/558 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-679) Fix missing CoprocessorService and MR Running feeder task number limitation
[ https://issues.apache.org/jira/browse/EAGLE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604107#comment-15604107 ] ASF GitHub Bot commented on EAGLE-679: -- Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/558 > Fix missing CoprocessorService and MR Running feeder task number limitation > --- > > Key: EAGLE-679 > URL: https://issues.apache.org/jira/browse/EAGLE-679 > Project: Eagle > Issue Type: Bug >Affects Versions: v0.5.0 >Reporter: wujinhu >Assignee: wujinhu > Fix For: v0.5.0 > > > 1. fix missing CoprocessorService > ERROR [2016-10-25 02:56:34,495] net.sf.extcos.internal.JavaResourceAccessor: > Unable to read interface > [org.apache.hadoop.hbase.coprocessor.CoprocessorService] > ! java.io.IOException: Class not found > 2. If task number of a job greater than 5000, then does not fetch task details -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #559: EAGLE-680 Can't genetate Html coverage re...
GitHub user chitin opened a pull request: https://github.com/apache/incubator-eagle/pull/559 EAGLE-680 Can't genetate Html coverage report EAGLE-680 Can't genetate Html coverage report - remove xml format in eagle-parent pom.xml You can merge this pull request into a Git repository by running: $ git pull https://github.com/chitin/incubator-eagle EAGLE680 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/559.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #559 commit c8b35f85224b03588eda47527e7d2cb136fce7eb Author: chitin Date: 2016-10-25T05:52:19Z EAGLE-680 Can't genetate Html coverage report - remove xml --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
