[jira] [Resolved] (EAGLE-515) convert JobHistoryZKStateManager to singleton
[ https://issues.apache.org/jira/browse/EAGLE-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu resolved EAGLE-515. --- Resolution: Resolved > convert JobHistoryZKStateManager to singleton > - > > Key: EAGLE-515 > URL: https://issues.apache.org/jira/browse/EAGLE-515 > Project: Eagle > Issue Type: Improvement >Reporter: wujinhu >Assignee: wujinhu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #411: [EAGLE-515] refactor zk manager to single...
GitHub user wujinhu opened a pull request: https://github.com/apache/incubator-eagle/pull/411 [EAGLE-515] refactor zk manager to singleton You can merge this pull request into a Git repository by running: $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-515 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/411.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #411 commit 11ab670c80229246ba0a63f4837a118c621c3e07 Author: wujinhu Date: 2016-08-31T06:35:15Z refactor zk manager to singleton --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-515) convert JobHistoryZKStateManager to singleton
[ https://issues.apache.org/jira/browse/EAGLE-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451334#comment-15451334 ] ASF GitHub Bot commented on EAGLE-515: -- GitHub user wujinhu opened a pull request: https://github.com/apache/incubator-eagle/pull/411 [EAGLE-515] refactor zk manager to singleton You can merge this pull request into a Git repository by running: $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-515 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/411.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #411 commit 11ab670c80229246ba0a63f4837a118c621c3e07 Author: wujinhu Date: 2016-08-31T06:35:15Z refactor zk manager to singleton > convert JobHistoryZKStateManager to singleton > - > > Key: EAGLE-515 > URL: https://issues.apache.org/jira/browse/EAGLE-515 > Project: Eagle > Issue Type: Improvement >Reporter: wujinhu >Assignee: wujinhu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #407: DataNode Usage Metric added in JMX Collector
Github user senthilec566 commented on the issue: https://github.com/apache/incubator-eagle/pull/407 Closing this PR ! Will open new one against develop branch! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request #407: DataNode Usage Metric added in JMX Collec...
Github user senthilec566 closed the pull request at: https://github.com/apache/incubator-eagle/pull/407 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-513) Add DataNodes usages Metrics in JMX Collector
[ https://issues.apache.org/jira/browse/EAGLE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451329#comment-15451329 ] ASF GitHub Bot commented on EAGLE-513: -- GitHub user senthilec566 opened a pull request: https://github.com/apache/incubator-eagle/pull/409 Add DataNodes usages Metrics in JMX Collector - EAGLE-513 https://issues.apache.org/jira/browse/EAGLE-513 You can merge this pull request into a Git repository by running: $ git pull https://github.com/senthilec566/incubator-eagle EAGLE-513 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit 9849303ebf8a63e4f54f0eb035c8060c43e50611 Author: Hao Chen Date: 2016-05-27T11:34:32Z [EAGLE-273][EAGLE-269] Fix exceeding the maximum row size of 65,535 bytes in mysql and comparisons between LONG_VARCHAR not supported https://issues.apache.org/jira/browse/EAGLE-273 https://issues.apache.org/jira/browse/EAGLE-269 The problem is cause the limitation of maximum row size in mysql innodb engine, so need to keep the column size relatively small, but it will cause it can't store large field, it's a little tricky, so may have following possible solutions: * Solution One: Fix mysql innodb size to resolve the limitation. * Solution Two: Keep the field relatively small to avoid exceeding maximum row size and make sure creating table successfully, and modify field size according to actual usage (Which is the approach we take for this problem) Author: Hao Chen Closes #158 from haoch/EAGLE-273-LATEST. commit df576452f174166407dfaa7a705b7f263f932f33 Author: hdendukuri Date: 2016-05-27T11:36:26Z [EAGLE-279] Create Documentation for Sandbox Quick starter and JMX monitring Create Documentation for Sandbox Quick starter and JMX monitring Author: hdendukuri Closes #159 from hdendukuri/master. commit 750b4c9a87732b9f69a83e908a819d174554745d Author: Zhao, Qingwen Date: 2016-05-30T07:49:48Z EAGLE 304: Enable Advanced dedup configuration in policy definition Author: Zhao, Qingwen & Jiang, jilin Reviewer: Chen, Hao Date: Thu May 26 20:12:12 2016 +0800 Closes #209 commit 5a5a17c3dff1e15bf72df267f8b2fc0739b95114 Author: Ralph, Su Date: 2016-05-25T10:07:20Z EAGLE-309: Add code formatter template Author: ralphsu Reviewer: ralphsu Committer: ralphsu Closes #199 commit 982a6b7430d349199091add264c9bef1bdc8ae35 Author: jiljiang Date: 2016-06-02T10:45:52Z [EAGLE-219] front end documentation Author: jiljiang Closes #171 from zombieJ/doc. commit f51474dee0c6ab40b4df965871b5d723c06f84a3 Author: Zhao, Qingwen Date: 2016-06-06T02:25:05Z EAGLE-327: Fix topologyOperation start exception: java.lang.ClassCastException https://issues.apache.org/jira/browse/EAGLE-327 Author: Zhao, Qingwen Reviewer: Wu, Michael Closes #218 commit cbf3c76ce148b73379698d9626df6e1baa5162ab Author: yonzhang Date: 2016-06-07T23:01:41Z EAGLE-330 Fix Hive ql.Parser Hive ql.Parser cant parser a hive query sql with keywords https://issues.apache.org/jira/browse/EAGLE-330 Author: @wgyumg Reviewer: @pkuwm Closes: #228 commit 8f12f82620b13d69f89ddc8057cd72337920ea31 Author: yonzhang Date: 2016-06-16T00:29:11Z EAGLE-301 fix the bug of breaking mysql row size when creating tables fix the bug of breaking mysql row size when creating tables Author: @anyway1021 Reviewer: @yonzhang Closes: #238 commit 36422ea81da6e4c158fcac1e02d3cb9798bf6486 Author: yonzhang Date: 2016-06-16T21:55:59Z EAGLE-340 refactor bin/eagle-sandbox-starter.sh to make it easier to use refactor bin/eagle-sandbox-starter.sh to make it easier to use Author: @qingwen220 Reviewer: @yonzhang Closes: #237 commit 4627eb0b576dec25c8c97d57410a044dcbd6af05 Author: yonzhang Date: 2016-06-16T22:06:36Z EAGLE-307 Update "logstash-kafka-conf.md" Update "logstash-kafka-conf.md" Author: @DadanielZ Reviewer: @yonzhang Closes: #234 commit 11d4be769b9cc1decd97f9d18f96236afbf153bf Author: yonzhang Date: 2016-06-16T22:17:51Z EAGLE-333 Fix hive.ql.Parser cant parse complex SQL fix hive parser issues Author: @wangyum Reviewer: @yonzhang Closes: #230 commit f5b3102319591f6aabfc74b072919a7e9a05dea1 Author: yonzhang Date: 2016-06-16T22:23:11Z EAGLE-326 fix a typo fix a typo in documentation for application management configuration Author: Michael Reviwer: Edward Closes: #220 commit 3553528990c7d
[GitHub] incubator-eagle pull request #409: Add DataNodes usages Metrics in JMX Colle...
GitHub user senthilec566 opened a pull request: https://github.com/apache/incubator-eagle/pull/409 Add DataNodes usages Metrics in JMX Collector - EAGLE-513 https://issues.apache.org/jira/browse/EAGLE-513 You can merge this pull request into a Git repository by running: $ git pull https://github.com/senthilec566/incubator-eagle EAGLE-513 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit 9849303ebf8a63e4f54f0eb035c8060c43e50611 Author: Hao Chen Date: 2016-05-27T11:34:32Z [EAGLE-273][EAGLE-269] Fix exceeding the maximum row size of 65,535 bytes in mysql and comparisons between LONG_VARCHAR not supported https://issues.apache.org/jira/browse/EAGLE-273 https://issues.apache.org/jira/browse/EAGLE-269 The problem is cause the limitation of maximum row size in mysql innodb engine, so need to keep the column size relatively small, but it will cause it can't store large field, it's a little tricky, so may have following possible solutions: * Solution One: Fix mysql innodb size to resolve the limitation. * Solution Two: Keep the field relatively small to avoid exceeding maximum row size and make sure creating table successfully, and modify field size according to actual usage (Which is the approach we take for this problem) Author: Hao Chen Closes #158 from haoch/EAGLE-273-LATEST. commit df576452f174166407dfaa7a705b7f263f932f33 Author: hdendukuri Date: 2016-05-27T11:36:26Z [EAGLE-279] Create Documentation for Sandbox Quick starter and JMX monitring Create Documentation for Sandbox Quick starter and JMX monitring Author: hdendukuri Closes #159 from hdendukuri/master. commit 750b4c9a87732b9f69a83e908a819d174554745d Author: Zhao, Qingwen Date: 2016-05-30T07:49:48Z EAGLE 304: Enable Advanced dedup configuration in policy definition Author: Zhao, Qingwen & Jiang, jilin Reviewer: Chen, Hao Date: Thu May 26 20:12:12 2016 +0800 Closes #209 commit 5a5a17c3dff1e15bf72df267f8b2fc0739b95114 Author: Ralph, Su Date: 2016-05-25T10:07:20Z EAGLE-309: Add code formatter template Author: ralphsu Reviewer: ralphsu Committer: ralphsu Closes #199 commit 982a6b7430d349199091add264c9bef1bdc8ae35 Author: jiljiang Date: 2016-06-02T10:45:52Z [EAGLE-219] front end documentation Author: jiljiang Closes #171 from zombieJ/doc. commit f51474dee0c6ab40b4df965871b5d723c06f84a3 Author: Zhao, Qingwen Date: 2016-06-06T02:25:05Z EAGLE-327: Fix topologyOperation start exception: java.lang.ClassCastException https://issues.apache.org/jira/browse/EAGLE-327 Author: Zhao, Qingwen Reviewer: Wu, Michael Closes #218 commit cbf3c76ce148b73379698d9626df6e1baa5162ab Author: yonzhang Date: 2016-06-07T23:01:41Z EAGLE-330 Fix Hive ql.Parser Hive ql.Parser cant parser a hive query sql with keywords https://issues.apache.org/jira/browse/EAGLE-330 Author: @wgyumg Reviewer: @pkuwm Closes: #228 commit 8f12f82620b13d69f89ddc8057cd72337920ea31 Author: yonzhang Date: 2016-06-16T00:29:11Z EAGLE-301 fix the bug of breaking mysql row size when creating tables fix the bug of breaking mysql row size when creating tables Author: @anyway1021 Reviewer: @yonzhang Closes: #238 commit 36422ea81da6e4c158fcac1e02d3cb9798bf6486 Author: yonzhang Date: 2016-06-16T21:55:59Z EAGLE-340 refactor bin/eagle-sandbox-starter.sh to make it easier to use refactor bin/eagle-sandbox-starter.sh to make it easier to use Author: @qingwen220 Reviewer: @yonzhang Closes: #237 commit 4627eb0b576dec25c8c97d57410a044dcbd6af05 Author: yonzhang Date: 2016-06-16T22:06:36Z EAGLE-307 Update "logstash-kafka-conf.md" Update "logstash-kafka-conf.md" Author: @DadanielZ Reviewer: @yonzhang Closes: #234 commit 11d4be769b9cc1decd97f9d18f96236afbf153bf Author: yonzhang Date: 2016-06-16T22:17:51Z EAGLE-333 Fix hive.ql.Parser cant parse complex SQL fix hive parser issues Author: @wangyum Reviewer: @yonzhang Closes: #230 commit f5b3102319591f6aabfc74b072919a7e9a05dea1 Author: yonzhang Date: 2016-06-16T22:23:11Z EAGLE-326 fix a typo fix a typo in documentation for application management configuration Author: Michael Reviwer: Edward Closes: #220 commit 3553528990c7df136ca1fc24806e58e081e66138 Author: yonzhang Date: 2016-06-16T22:24:26Z EAGLE-319 modify java to jdbc data type modify java to jdbc data type mapping to be compatible with derby DB Author: Michael Reviewer: Edward Close
[GitHub] incubator-eagle pull request #410: Eagle-447 convert eagle-maprfs app to use...
GitHub user DadanielZ opened a pull request: https://github.com/apache/incubator-eagle/pull/410 Eagle-447 convert eagle-maprfs app to use new app framework eagle-maprfs app should use new app framework to manage its lifecycle - in AbstractHdfsAuditLogApplication, instead of new a HdfsAuditLogParserBolt object directly, now it call "getParserBolt()" to get BaseRichBolt You can merge this pull request into a Git repository by running: $ git pull https://github.com/DadanielZ/incubator-eagle EAGLE-447 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/410.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #410 commit b880b37530d9f7e4e453acd541c2caa7600aa3fe Author: DadanielZ Date: 2016-08-29T21:12:27Z Eagle-447 convert eagle-maprfs app to use new app framework eagle-maprfs app should use new app framework to manage its lifecycle --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request #402: Eagle-447 convert eagle-maprfs app to use...
Github user DadanielZ closed the pull request at: https://github.com/apache/incubator-eagle/pull/402 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request #402: Eagle-447 convert eagle-maprfs app to use...
GitHub user DadanielZ reopened a pull request: https://github.com/apache/incubator-eagle/pull/402 Eagle-447 convert eagle-maprfs app to use new app framework eagle-maprfs app should use new app framework to manage its lifecycle You can merge this pull request into a Git repository by running: $ git pull https://github.com/DadanielZ/incubator-eagle EAGLE-447 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/402.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #402 commit b1bfbc211d41eecff826fa335a902a91ce1a2915 Author: DadanielZ Date: 2016-08-29T21:12:27Z Eagle-447 convert eagle-maprfs app to use new app framework eagle-maprfs app should use new app framework to manage its lifecycle --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request #407: DataNode Usage Metric added in JMX Collec...
Github user senthilec566 commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/407#discussion_r76929966
--- Diff: eagle-external/hadoop_jmx_collector/hadoop_jmx_kafka.py ---
@@ -64,6 +64,23 @@ def on_bean(self, bean):
capacityusage = round(float(bean['CapacityUsed']) /
float(bean['CapacityTotal']) * 100, 2)
self.collector.on_bean_kv(self.PREFIX, "capacityusage",
capacityusage)
+class NNNodeUsageMetric(JmxMetricListener):
+PREFIX = "hadoop.namenode.nodeusage"
+
+def on_bean(self, bean):
+if bean["name"] == "Hadoop:service=NameNode,name=NameNodeInfo":
+nodeusagedic = json.loads(bean["NodeUsage"]);
+nodeusage_detail_dic = nodeusagedic["nodeUsage"];
+min = round( float(nodeusage_detail_dic["min"].strip('%')) ,2
);
--- End diff --
@haoch , thanks for pointing it out! Will change !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #402: Eagle-447 convert eagle-maprfs app to use...
Github user DadanielZ closed the pull request at: https://github.com/apache/incubator-eagle/pull/402 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #408: EAGLE-514: Add two job count apis
Github user wujinhu commented on the issue: https://github.com/apache/incubator-eagle/pull/408 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-514) Add two job count apis
[ https://issues.apache.org/jira/browse/EAGLE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451274#comment-15451274 ] ASF GitHub Bot commented on EAGLE-514: -- Github user wujinhu commented on the issue: https://github.com/apache/incubator-eagle/pull/408 LGTM > Add two job count apis > -- > > Key: EAGLE-514 > URL: https://issues.apache.org/jira/browse/EAGLE-514 > Project: Eagle > Issue Type: New Feature >Reporter: Zhao, Qingwen >Assignee: Zhao, Qingwen > > *Running job counting in timeline* > Description: number of Running job (stacked by job type) snapshot during a > period of time > /rest/mrJobs/runningJobCounts?site=sampleSite&intervalInSecs=900&durationBegin=2016-08-22%2020:13:00&durationEnd=2016-08-30%2024:13:00 > *History job counting by duration* > Description: group the jobs by duration, duration distribution, stack by job > type (only for history) > /rest/mrJobs/jobCountsByDuration?site=sampleSite&timelineInSecs=0,900&jobStartTimeBegin=2016-08-22%2020:13:00&jobStartTimeEnd=2016-08-29%2024:13:00 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (EAGLE-516) Implement application entity service by jdbc
JiJun Tang created EAGLE-516: Summary: Implement application entity service by jdbc Key: EAGLE-516 URL: https://issues.apache.org/jira/browse/EAGLE-516 Project: Eagle Issue Type: Improvement Affects Versions: v0.5.0 Reporter: JiJun Tang Assignee: JiJun Tang Priority: Minor Fix For: v0.5.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (EAGLE-514) Add two job count apis
[ https://issues.apache.org/jira/browse/EAGLE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhao, Qingwen updated EAGLE-514: Description: *Running job counting in timeline* Description: number of Running job (stacked by job type) snapshot during a period of time /rest/mrJobs/runningJobCounts?site=sampleSite&intervalInSecs=900&durationBegin=2016-08-22%2020:13:00&durationEnd=2016-08-30%2024:13:00 *History job counting by duration* Description: group the jobs by duration, duration distribution, stack by job type (only for history) /rest/mrJobs/jobCountsByDuration?site=sampleSite&timelineInSecs=0,900&jobStartTimeBegin=2016-08-22%2020:13:00&jobStartTimeEnd=2016-08-29%2024:13:00 was: *Running job counting in timeline* /rest/mrJobs/runningJobCounts?site=sampleSite&intervalInSecs=900&durationBegin=2016-08-22%2020:13:00&durationEnd=2016-08-30%2024:13:00 *History job counting by duration* /rest/mrJobs/jobCountsByDuration?site=sampleSite&timelineInSecs=0,900&jobStartTimeBegin=2016-08-22%2020:13:00&jobStartTimeEnd=2016-08-29%2024:13:00 > Add two job count apis > -- > > Key: EAGLE-514 > URL: https://issues.apache.org/jira/browse/EAGLE-514 > Project: Eagle > Issue Type: New Feature >Reporter: Zhao, Qingwen >Assignee: Zhao, Qingwen > > *Running job counting in timeline* > Description: number of Running job (stacked by job type) snapshot during a > period of time > /rest/mrJobs/runningJobCounts?site=sampleSite&intervalInSecs=900&durationBegin=2016-08-22%2020:13:00&durationEnd=2016-08-30%2024:13:00 > *History job counting by duration* > Description: group the jobs by duration, duration distribution, stack by job > type (only for history) > /rest/mrJobs/jobCountsByDuration?site=sampleSite&timelineInSecs=0,900&jobStartTimeBegin=2016-08-22%2020:13:00&jobStartTimeEnd=2016-08-29%2024:13:00 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-514) Add two job count apis
[ https://issues.apache.org/jira/browse/EAGLE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451261#comment-15451261 ] ASF GitHub Bot commented on EAGLE-514: -- GitHub user qingwen220 opened a pull request: https://github.com/apache/incubator-eagle/pull/408 EAGLE-514: Add two job count apis https://issues.apache.org/jira/browse/EAGLE-514 1. adding two job counting apis 2. add tracking url in running/history job execution entity 3. unify the status presentation for job/task execution entity 4. unify the name of the common fields between running job entity & history job entity You can merge this pull request into a Git repository by running: $ git pull https://github.com/qingwen220/incubator-eagle jobStats Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit 698328c30b7981980b01fd2e11e1bc618913898b Author: Qingwen Zhao Date: 2016-08-23T07:44:18Z add jobCount APi commit 399988a5072de9382fba6c1bca94986812f14836 Author: Qingwen Zhao Date: 2016-08-26T09:27:30Z Fix conflicts with apache develop branch commit 0a0407450376f74fd90ac52a6cc149d6fd11a072 Author: Qingwen Zhao Date: 2016-08-26T09:32:25Z Fields rename commit 4a83f42ba8de29fa4c1bcb1c894d9cfd55208755 Author: Qingwen Zhao Date: 2016-08-26T10:43:06Z update jpm service commit e922e429ea8765181accd99b7a9abe7d18f6c720 Author: Qingwen Zhao Date: 2016-08-29T02:47:02Z code style update commit f66731f72a3d8732c90b4819b072c0b1a182a1fd Author: Qingwen Zhao Date: 2016-08-29T13:23:34Z add history job count api commit 4ad4a9df9bc9fe59edb34336b1a0dae17ffa1d96 Author: Qingwen Zhao Date: 2016-08-30T02:13:26Z add exception handling commit e34ba77c7e217fe33383a3b7e81ca766149d0836 Author: Qingwen Zhao Date: 2016-08-30T06:32:02Z unify status presentation for running/finished tasks commit 31f5e0bd711c29ca948e50cec109b8d260cc88a7 Author: Qingwen Zhao Date: 2016-08-30T08:28:31Z add job trackingUrl commit e040ab0757fbe979275db399e33384d64fc0970f Author: Qingwen Zhao Date: 2016-08-30T12:20:13Z change parameter name commit 519a2bc9640f4d36ef787bd9a544381fbe867a6d Author: Qingwen Zhao Date: 2016-08-30T12:25:07Z Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-eagle into jobStats commit e07c16dcb512be0d353775f00f3bdc807b482852 Author: Qingwen Zhao Date: 2016-08-31T02:28:06Z change jobStatus SUCCESS to SUCCEEDED commit c75d8dd6f1600042c906e70e7ed4a43ce7e6ef7f Author: Qingwen Zhao Date: 2016-08-31T02:46:23Z Add zookeeper client close commit b5c3b0bb7d2683b056b563dd458317d4d2a1dabb Author: Qingwen Zhao Date: 2016-08-31T04:02:31Z Fix two bugs in JPM service commit 8891b2a5a51819c49314de18ef7580879a1edf55 Author: Qingwen Zhao Date: 2016-08-31T05:43:19Z fix an url path combination error > Add two job count apis > -- > > Key: EAGLE-514 > URL: https://issues.apache.org/jira/browse/EAGLE-514 > Project: Eagle > Issue Type: New Feature >Reporter: Zhao, Qingwen >Assignee: Zhao, Qingwen > > *Running job counting in timeline* > /rest/mrJobs/runningJobCounts?site=sampleSite&intervalInSecs=900&durationBegin=2016-08-22%2020:13:00&durationEnd=2016-08-30%2024:13:00 > *History job counting by duration* > /rest/mrJobs/jobCountsByDuration?site=sampleSite&timelineInSecs=0,900&jobStartTimeBegin=2016-08-22%2020:13:00&jobStartTimeEnd=2016-08-29%2024:13:00 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #408: EAGLE-514: Add two job count apis
GitHub user qingwen220 opened a pull request: https://github.com/apache/incubator-eagle/pull/408 EAGLE-514: Add two job count apis https://issues.apache.org/jira/browse/EAGLE-514 1. adding two job counting apis 2. add tracking url in running/history job execution entity 3. unify the status presentation for job/task execution entity 4. unify the name of the common fields between running job entity & history job entity You can merge this pull request into a Git repository by running: $ git pull https://github.com/qingwen220/incubator-eagle jobStats Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit 698328c30b7981980b01fd2e11e1bc618913898b Author: Qingwen Zhao Date: 2016-08-23T07:44:18Z add jobCount APi commit 399988a5072de9382fba6c1bca94986812f14836 Author: Qingwen Zhao Date: 2016-08-26T09:27:30Z Fix conflicts with apache develop branch commit 0a0407450376f74fd90ac52a6cc149d6fd11a072 Author: Qingwen Zhao Date: 2016-08-26T09:32:25Z Fields rename commit 4a83f42ba8de29fa4c1bcb1c894d9cfd55208755 Author: Qingwen Zhao Date: 2016-08-26T10:43:06Z update jpm service commit e922e429ea8765181accd99b7a9abe7d18f6c720 Author: Qingwen Zhao Date: 2016-08-29T02:47:02Z code style update commit f66731f72a3d8732c90b4819b072c0b1a182a1fd Author: Qingwen Zhao Date: 2016-08-29T13:23:34Z add history job count api commit 4ad4a9df9bc9fe59edb34336b1a0dae17ffa1d96 Author: Qingwen Zhao Date: 2016-08-30T02:13:26Z add exception handling commit e34ba77c7e217fe33383a3b7e81ca766149d0836 Author: Qingwen Zhao Date: 2016-08-30T06:32:02Z unify status presentation for running/finished tasks commit 31f5e0bd711c29ca948e50cec109b8d260cc88a7 Author: Qingwen Zhao Date: 2016-08-30T08:28:31Z add job trackingUrl commit e040ab0757fbe979275db399e33384d64fc0970f Author: Qingwen Zhao Date: 2016-08-30T12:20:13Z change parameter name commit 519a2bc9640f4d36ef787bd9a544381fbe867a6d Author: Qingwen Zhao Date: 2016-08-30T12:25:07Z Merge branch 'develop' of https://git-wip-us.apache.org/repos/asf/incubator-eagle into jobStats commit e07c16dcb512be0d353775f00f3bdc807b482852 Author: Qingwen Zhao Date: 2016-08-31T02:28:06Z change jobStatus SUCCESS to SUCCEEDED commit c75d8dd6f1600042c906e70e7ed4a43ce7e6ef7f Author: Qingwen Zhao Date: 2016-08-31T02:46:23Z Add zookeeper client close commit b5c3b0bb7d2683b056b563dd458317d4d2a1dabb Author: Qingwen Zhao Date: 2016-08-31T04:02:31Z Fix two bugs in JPM service commit 8891b2a5a51819c49314de18ef7580879a1edf55 Author: Qingwen Zhao Date: 2016-08-31T05:43:19Z fix an url path combination error --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (EAGLE-515) convert JobHistoryZKStateManager to singleton
wujinhu created EAGLE-515: - Summary: convert JobHistoryZKStateManager to singleton Key: EAGLE-515 URL: https://issues.apache.org/jira/browse/EAGLE-515 Project: Eagle Issue Type: Improvement Reporter: wujinhu Assignee: wujinhu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (EAGLE-512) add job count statistics to mr history feeder
[ https://issues.apache.org/jira/browse/EAGLE-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu closed EAGLE-512. - > add job count statistics to mr history feeder > - > > Key: EAGLE-512 > URL: https://issues.apache.org/jira/browse/EAGLE-512 > Project: Eagle > Issue Type: New Feature >Reporter: wujinhu >Assignee: wujinhu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (EAGLE-514) Add two job count apis
Zhao, Qingwen created EAGLE-514: --- Summary: Add two job count apis Key: EAGLE-514 URL: https://issues.apache.org/jira/browse/EAGLE-514 Project: Eagle Issue Type: New Feature Reporter: Zhao, Qingwen Assignee: Zhao, Qingwen *Running job counting in timeline* /rest/mrJobs/runningJobCounts?site=sampleSite&intervalInSecs=900&durationBegin=2016-08-22%2020:13:00&durationEnd=2016-08-30%2024:13:00 *History job counting by duration* /rest/mrJobs/jobCountsByDuration?site=sampleSite&timelineInSecs=0,900&jobStartTimeBegin=2016-08-22%2020:13:00&jobStartTimeEnd=2016-08-29%2024:13:00 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-512) add job count statistics to mr history feeder
[ https://issues.apache.org/jira/browse/EAGLE-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450882#comment-15450882 ] ASF GitHub Bot commented on EAGLE-512: -- Github user wujinhu closed the pull request at: https://github.com/apache/incubator-eagle/pull/406 > add job count statistics to mr history feeder > - > > Key: EAGLE-512 > URL: https://issues.apache.org/jira/browse/EAGLE-512 > Project: Eagle > Issue Type: New Feature >Reporter: wujinhu >Assignee: wujinhu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #406: [EAGLE-512] add job count(total&fail) for...
Github user wujinhu closed the pull request at: https://github.com/apache/incubator-eagle/pull/406 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #402: Eagle-447 convert eagle-maprfs app to use new ap...
Github user DadanielZ commented on the issue:
https://github.com/apache/incubator-eagle/pull/402
@yonzhang, I see now. In â AbstractHdfsAuditLogApplication.javaâ, there
is an overrided function which use "HdfsAuditLogParserBolt" directly:
```
@Override
public StormTopology execute(Config config, StormEnvironment environment){
...
HdfsAuditLogParserBolt parserBolt = new HdfsAuditLogParserBolt();
BoltDeclarer boltDeclarer = builder.setBolt("parserBolt", parserBolt,
numOfParserTasks);
...
}
```
that's why I wrote "AbstractMapRFSAuditLogApplication" and call
"MapRFSAuditLogParserBolt".
I think I should refactor the code in "AbstractHdfsAuditLogApplication" and
let it call "getParserBolt" to get parserBolt, then extend this class for mapr
application:
```
@Override
public StormTopology execute(Config config, StormEnvironment environment){
...
BaseRichBolt parserBolt = getParserBolt();
BoltDeclarer boltDeclarer = builder.setBolt("parserBolt", parserBolt,
numOfParserTasks);
...
}
```
Will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Updated] (EAGLE-508) Fix NullPointerException when zookeeper is trying to shutdown in ZookeeperEmbedded.
[ https://issues.apache.org/jira/browse/EAGLE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huizhi Lu updated EAGLE-508: Summary: Fix NullPointerException when zookeeper is trying to shutdown in ZookeeperEmbedded. (was: Jenkins building failed on build #71) > Fix NullPointerException when zookeeper is trying to shutdown in > ZookeeperEmbedded. > --- > > Key: EAGLE-508 > URL: https://issues.apache.org/jira/browse/EAGLE-508 > Project: Eagle > Issue Type: Bug >Affects Versions: v0.5.0 >Reporter: Huizhi Lu >Assignee: Huizhi Lu > Fix For: v0.5.0 > > > Root cause: zookeeper.getState() may be null so zookeeper.getState().equals() > would throw NullPointerException. > Solution: CuratorFrameworkState.STOPPED.equals(zookeeper.getState()) > --- > T E S T S > --- > Running com.apache.eagle.service.app.TestServiceAppWithZk > java.lang.NullPointerException > at > org.apache.eagle.alert.utils.ZookeeperEmbedded.shutdown(ZookeeperEmbedded.java:55) > at > com.apache.eagle.service.app.TestServiceAppWithZk.tearDown(TestServiceAppWithZk.java:76) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59) > at > org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120) > at > org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103) > at org.apache.maven.surefire.Surefire.run(Surefire.java:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350) > at > org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021) > Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.231 sec <<< > FAILURE! > Results : > Tests in error: > testMain(com.apache.eagle.service.app.TestServiceAppWithZk) > testMain(com.apache.eagle.service.app.TestServiceAppWithZk) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #402: Eagle-447 convert eagle-maprfs app to use new ap...
Github user yonzhang commented on the issue: https://github.com/apache/incubator-eagle/pull/402 @DadanielZ that is why there is already AbstractHdfsAuditLogApplication there, and MapRApplication can extend this and implement getParserBolt method --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #402: Eagle-447 convert eagle-maprfs app to use new ap...
Github user DadanielZ commented on the issue: https://github.com/apache/incubator-eagle/pull/402 @haoch thanks for the review, will add them later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #402: Eagle-447 convert eagle-maprfs app to use new ap...
Github user DadanielZ commented on the issue: https://github.com/apache/incubator-eagle/pull/402 @yonzhang Their code are almost the same except that: "AbstractHdfsAuditLogApplication" use "HdfsAuditLogParserBolt", "AbstractMapRFSAuditLogApplication" should use "MapRFSAuditLogParserBolt". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-508) Jenkins building failed on build #71
[ https://issues.apache.org/jira/browse/EAGLE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449642#comment-15449642 ] ASF GitHub Bot commented on EAGLE-508: -- Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/401 Pls. change to title to meaningful information which will be part of commit history. > Jenkins building failed on build #71 > > > Key: EAGLE-508 > URL: https://issues.apache.org/jira/browse/EAGLE-508 > Project: Eagle > Issue Type: Bug >Affects Versions: v0.5.0 >Reporter: Huizhi Lu >Assignee: Huizhi Lu > Fix For: v0.5.0 > > > Root cause: zookeeper.getState() may be null so zookeeper.getState().equals() > would throw NullPointerException. > Solution: CuratorFrameworkState.STOPPED.equals(zookeeper.getState()) > --- > T E S T S > --- > Running com.apache.eagle.service.app.TestServiceAppWithZk > java.lang.NullPointerException > at > org.apache.eagle.alert.utils.ZookeeperEmbedded.shutdown(ZookeeperEmbedded.java:55) > at > com.apache.eagle.service.app.TestServiceAppWithZk.tearDown(TestServiceAppWithZk.java:76) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59) > at > org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120) > at > org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103) > at org.apache.maven.surefire.Surefire.run(Surefire.java:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350) > at > org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021) > Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.231 sec <<< > FAILURE! > Results : > Tests in error: > testMain(com.apache.eagle.service.app.TestServiceAppWithZk) > testMain(com.apache.eagle.service.app.TestServiceAppWithZk) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #401: [EAGLE-508] Fix Jenkins building failed on build...
Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/401 Pls. change to title to meaningful information which will be part of commit history. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #402: Eagle-447 convert eagle-maprfs app to use new ap...
Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/402 Pls. refer to : * Metadata descriptor and SPI: https://github.com/apache/incubator-eagle/tree/develop/eagle-security/eagle-security-hdfs-auditlog/src/main/resources/META-INF * ApplicationProvider: https://github.com/apache/incubator-eagle/blob/develop/eagle-security/eagle-security-hdfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/HdfsAuditLogAppProvider.java * Better to have Unit Test to make sure all set: https://github.com/apache/incubator-eagle/blob/develop/eagle-jpm/eagle-jpm-spark-running/src/test/java/SparkRunningJobAppProviderTest.java#L30 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-510) Refactor HDFS audit log parser
[ https://issues.apache.org/jira/browse/EAGLE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449634#comment-15449634 ] ASF GitHub Bot commented on EAGLE-510: -- Github user yonzhang commented on the issue: https://github.com/apache/incubator-eagle/pull/403 sure, we can remove that method > Refactor HDFS audit log parser > -- > > Key: EAGLE-510 > URL: https://issues.apache.org/jira/browse/EAGLE-510 > Project: Eagle > Issue Type: Improvement >Reporter: Peter Kim >Priority: Minor > > The current HDFS audit log parser can be refactored by leveraging regex > instead of the substring methods it uses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #403: EAGLE-510: Refactor HDFS audit log parser
Github user yonzhang commented on the issue: https://github.com/apache/incubator-eagle/pull/403 sure, we can remove that method --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-510) Refactor HDFS audit log parser
[ https://issues.apache.org/jira/browse/EAGLE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449628#comment-15449628 ] ASF GitHub Bot commented on EAGLE-510: -- Github user peterkim95 commented on the issue: https://github.com/apache/incubator-eagle/pull/403 I see, can we can still refactor by removing the unused method parseUser > Refactor HDFS audit log parser > -- > > Key: EAGLE-510 > URL: https://issues.apache.org/jira/browse/EAGLE-510 > Project: Eagle > Issue Type: Improvement >Reporter: Peter Kim >Priority: Minor > > The current HDFS audit log parser can be refactored by leveraging regex > instead of the substring methods it uses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #402: Eagle-447 convert eagle-maprfs app to use new ap...
Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/402 ApplicationProvider and XML metadata descriptor is missing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #403: EAGLE-510: Refactor HDFS audit log parser
Github user peterkim95 commented on the issue: https://github.com/apache/incubator-eagle/pull/403 I see, can we can still refactor by removing the unused method parseUser --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-489) maven Javadoc phrase fails on develop branch
[ https://issues.apache.org/jira/browse/EAGLE-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449621#comment-15449621 ] ASF GitHub Bot commented on EAGLE-489: -- Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/405 LGTM, pls. merge. > maven Javadoc phrase fails on develop branch > > > Key: EAGLE-489 > URL: https://issues.apache.org/jira/browse/EAGLE-489 > Project: Eagle > Issue Type: Bug >Reporter: Michael Wu >Assignee: Michael Wu > > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Eagle Parent ... SUCCESS [12.430s] > [INFO] eagle core SUCCESS [1.398s] > [INFO] eagle-data-process SUCCESS [47.032s] > [INFO] eagle-alert-parent SUCCESS [1.382s] > [INFO] eagle-alert ... SUCCESS [1.136s] > [INFO] alert-common .. FAILURE [17.049s] > [INFO] alert-engine .. SKIPPED > [INFO] alert-metadata-parent . SKIPPED > [INFO] alert-metadata SKIPPED > [INFO] alert-metadata-service SKIPPED > [INFO] alert-coordinator . SKIPPED > [INFO] alert-assembly SKIPPED > [INFO] alert-devtools SKIPPED > [INFO] alert-service . SKIPPED > [INFO] eagle-query-parent SKIPPED > [INFO] eagle-antlr ... SKIPPED > [INFO] eagle-common .. SKIPPED > [INFO] eagle-embed-parent SKIPPED > [INFO] eagle-embed-hbase . SKIPPED > [INFO] eagle-entity-base . SKIPPED > [INFO] eagle-query-base .. SKIPPED > [INFO] eagle-storage-base SKIPPED > [INFO] eagle-embed-server SKIPPED > [INFO] eagle-client-base . SKIPPED > [INFO] eagle-audit-base .. SKIPPED > [INFO] eagle-storage-hbase ... SKIPPED > [INFO] eagle-service-base SKIPPED > [INFO] eagle-metadata SKIPPED > [INFO] eagle-metadata-base ... SKIPPED > [INFO] eagle-alert-service ... SKIPPED > [INFO] eagle-storage-jdbc SKIPPED > [INFO] eagle-metric .. SKIPPED > [INFO] eagle-metadata-jdbc ... SKIPPED > [INFO] eagle-metadata-mongo .. SKIPPED > [INFO] eagle-app-parent .. SKIPPED > [INFO] eagle-app-base SKIPPED > [INFO] eagle-security-parent . SKIPPED > [INFO] eagle-security-common . SKIPPED > [INFO] eagle-security-hdfs-auditlog .. SKIPPED > [INFO] eagle-security-maprfs-auditlog SKIPPED > [INFO] eagle-jpm-parent .. SKIPPED > [INFO] eagle-jpm-util SKIPPED > [INFO] eagle-security-hive ... SKIPPED > [INFO] eagle-security-hive-web ... SKIPPED > [INFO] eagle-security-hdfs-web ... SKIPPED > [INFO] eagle-security-maprfs-web . SKIPPED > [INFO] eagle-security-hdfs-authlog ... SKIPPED > [INFO] eagle-security-hbase-auditlog . SKIPPED > [INFO] eagle-security-hbase-web .. SKIPPED > [INFO] eagle-metric-collection ... SKIPPED > [INFO] eagle-security-oozie-auditlog . SKIPPED > [INFO] eagle-security-oozie-web .. SKIPPED > [INFO] eagle-external-parent . SKIPPED > [INFO] eagle-log4jkafka .. SKIPPED > [INFO] eagle-kafka ... SKIPPED > [INFO] eagle-examples SKIPPED > [INFO] eagle-app-example . SKIPPED > [INFO] eagle-jpm-app . SKIPPED > [INFO] eagle-gc .. SKIPPED > [INFO] eagle-server
[GitHub] incubator-eagle issue #405: [EAGLE-489] loosen javadoc restriction to get ri...
Github user haoch commented on the issue: https://github.com/apache/incubator-eagle/pull/405 LGTM, pls. merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request #407: DataNode Usage Metric added in JMX Collec...
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/407#discussion_r76841756
--- Diff: eagle-external/hadoop_jmx_collector/hadoop_jmx_kafka.py ---
@@ -64,6 +64,23 @@ def on_bean(self, bean):
capacityusage = round(float(bean['CapacityUsed']) /
float(bean['CapacityTotal']) * 100, 2)
self.collector.on_bean_kv(self.PREFIX, "capacityusage",
capacityusage)
+class NNNodeUsageMetric(JmxMetricListener):
+PREFIX = "hadoop.namenode.nodeusage"
+
+def on_bean(self, bean):
+if bean["name"] == "Hadoop:service=NameNode,name=NameNodeInfo":
+nodeusagedic = json.loads(bean["NodeUsage"]);
+nodeusage_detail_dic = nodeusagedic["nodeUsage"];
+min = round( float(nodeusage_detail_dic["min"].strip('%')) ,2
);
--- End diff --
In order to avoid confusing, could you pls. rename local variable min/max
to min_val/max_val as min and max are Python names?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (EAGLE-510) Refactor HDFS audit log parser
[ https://issues.apache.org/jira/browse/EAGLE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449597#comment-15449597 ] ASF GitHub Bot commented on EAGLE-510: -- Github user yonzhang commented on the issue: https://github.com/apache/incubator-eagle/pull/403 We may not change the code to use regex matching as it is too slow. We have achieved more than 10 times performance gain by using string operations than using regex. This is critical important as in production we have the issue with large amount of audit log and regex significantly make performance worse > Refactor HDFS audit log parser > -- > > Key: EAGLE-510 > URL: https://issues.apache.org/jira/browse/EAGLE-510 > Project: Eagle > Issue Type: Improvement >Reporter: Peter Kim >Priority: Minor > > The current HDFS audit log parser can be refactored by leveraging regex > instead of the substring methods it uses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle issue #403: EAGLE-510: Refactor HDFS audit log parser
Github user yonzhang commented on the issue: https://github.com/apache/incubator-eagle/pull/403 We may not change the code to use regex matching as it is too slow. We have achieved more than 10 times performance gain by using string operations than using regex. This is critical important as in production we have the issue with large amount of audit log and regex significantly make performance worse --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #399: Eagle505 oozie auditlog parser don't deal with N...
Github user yonzhang commented on the issue: https://github.com/apache/incubator-eagle/pull/399 Thanks for the fix. We can remove the internal try/catch and only use outer try/catch to catch exception and finally it will do ack anyway --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #402: Eagle-447 convert eagle-maprfs app to use new ap...
Github user yonzhang commented on the issue: https://github.com/apache/incubator-eagle/pull/402 Thanks for the improvement. Why we create a new AbstractMapRFSAuditLogApplication, can't we use AbstractHdfsAuditLogApplication? They share the same code --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle issue #407: DataNode Usage Metric added in JMX Collector
Github user yonzhang commented on the issue: https://github.com/apache/incubator-eagle/pull/407 good fix, but Senthil can you do PR against develop branch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
ApacheCon Seville CFP closes September 9th
It's traditional. We wait for the last minute to get our talk proposals in for conferences. Well, the last minute has arrived. The CFP for ApacheCon Seville closes on September 9th, which is less than 2 weeks away. It's time to get your talks in, so that we can make this the best ApacheCon yet. It's also time to discuss with your developer and user community whether there's a track of talks that you might want to propose, so that you have more complete coverage of your project than a talk or two. For Apache Big Data, the relevant URLs are: Event details: http://events.linuxfoundation.org/events/apache-big-data-europe CFP: http://events.linuxfoundation.org/events/apache-big-data-europe/program/cfp For ApacheCon Europe, the relevant URLs are: Event details: http://events.linuxfoundation.org/events/apachecon-europe CFP: http://events.linuxfoundation.org/events/apachecon-europe/program/cfp This year, we'll be reviewing papers "blind" - that is, looking at the abstracts without knowing who the speaker is. This has been shown to eliminate the "me and my buddies" nature of many tech conferences, producing more diversity, and more new speakers. So make sure your abstracts clearly explain what you'll be talking about. For further updated about ApacheCon, follow us on Twitter, @ApacheCon, or drop by our IRC channel, #apachecon on the Freenode IRC network. -- Rich Bowen WWW: http://apachecon.com/ Twitter: @ApacheCon
[GitHub] incubator-eagle issue #407: DataNode Usage Metric added in JMX Collector
Github user RalphSu commented on the issue: https://github.com/apache/incubator-eagle/pull/407 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Comment Edited] (EAGLE-513) Add DataNodes usages Metrics in JMX Collector
[
https://issues.apache.org/jira/browse/EAGLE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449104#comment-15449104
]
Senthilkumar edited comment on EAGLE-513 at 8/30/16 2:09 PM:
-
Added code in hadoo_jmx_kafka.py file
Sample Kafka Message:
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.median",
"component": "namenode", "site": "site", "value": 99.985, "host":
"namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.min",
"component": "namenode", "site": "site", "value": 1.2, "host": "namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.stddev",
"component": "namenode", "site": "site", "value": 19.922, "host":
"namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.max",
"component": "namenode", "site": "site", "value": 100.040001, "host":
"namenode"}
was (Author: senthilec566):
Added code in hadoo_jmx_kafka.py file
Sample Kafka Message:
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.median",
"component": "namenode", "site": "apollo", "value": 99.985, "host":
"namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.min",
"component": "namenode", "site": "apollo", "value": 1.2, "host": "namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.stddev",
"component": "namenode", "site": "apollo", "value": 19.922, "host":
"namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.max",
"component": "namenode", "site": "apollo", "value": 100.040001, "host":
"namenode"}
> Add DataNodes usages Metrics in JMX Collector
> -
>
> Key: EAGLE-513
> URL: https://issues.apache.org/jira/browse/EAGLE-513
> Project: Eagle
> Issue Type: Improvement
>Affects Versions: v0.5.0
>Reporter: Senthilkumar
>Assignee: Senthilkumar
>
> Instead of Capacity check , Its better to track "DataNodes usages" Median
> Values. As we won't hit 90% DFS utilisation before that will have More Jobs
> failures because of DataNode Issues..
> So we should clear/Balance the Cluster effectively By tracking DataNode
> Median. 95% Median would be Nice to Start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (EAGLE-513) Add DataNodes usages Metrics in JMX Collector
[
https://issues.apache.org/jira/browse/EAGLE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449104#comment-15449104
]
Senthilkumar commented on EAGLE-513:
Added code in hadoo_jmx_kafka.py file
Sample Kafka Message:
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.median",
"component": "namenode", "site": "apollo", "value": 99.985, "host":
"namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.min",
"component": "namenode", "site": "apollo", "value": 1.2, "host": "namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.stddev",
"component": "namenode", "site": "apollo", "value": 19.922, "host":
"namenode"}
{"timestamp": 1472563981729, "metric": "hadoop.namenode.nodeusage.max",
"component": "namenode", "site": "apollo", "value": 100.040001, "host":
"namenode"}
> Add DataNodes usages Metrics in JMX Collector
> -
>
> Key: EAGLE-513
> URL: https://issues.apache.org/jira/browse/EAGLE-513
> Project: Eagle
> Issue Type: Improvement
>Affects Versions: v0.5.0
>Reporter: Senthilkumar
>Assignee: Senthilkumar
>
> Instead of Capacity check , Its better to track "DataNodes usages" Median
> Values. As we won't hit 90% DFS utilisation before that will have More Jobs
> failures because of DataNode Issues..
> So we should clear/Balance the Cluster effectively By tracking DataNode
> Median. 95% Median would be Nice to Start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-eagle pull request #407: DataNode Usage Metric added in JMX Collec...
GitHub user senthilec566 opened a pull request: https://github.com/apache/incubator-eagle/pull/407 DataNode Usage Metric added in JMX Collector Modified Hadoop JMX collector to Parse Data Node Usage. You can merge this pull request into a Git repository by running: $ git pull https://github.com/senthilec566/incubator-eagle EAGLE-513 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/407.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #407 commit 3e2b2c9222bb9a63aa5151998c55b7a99b811b42 Author: senthilkumar Date: 2016-08-30T13:28:29Z DataNode Usage Metric added in JMX Collector --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (EAGLE-513) Add DataNodes usages Metrics in JMX Collector
Senthilkumar created EAGLE-513: -- Summary: Add DataNodes usages Metrics in JMX Collector Key: EAGLE-513 URL: https://issues.apache.org/jira/browse/EAGLE-513 Project: Eagle Issue Type: Improvement Affects Versions: v0.5.0 Reporter: Senthilkumar Assignee: Senthilkumar Instead of Capacity check , Its better to track "DataNodes usages" Median Values. As we won't hit 90% DFS utilisation before that will have More Jobs failures because of DataNode Issues.. So we should clear/Balance the Cluster effectively By tracking DataNode Median. 95% Median would be Nice to Start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (EAGLE-512) add job count statistics to mr history feeder
[ https://issues.apache.org/jira/browse/EAGLE-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu resolved EAGLE-512. --- Resolution: Resolved > add job count statistics to mr history feeder > - > > Key: EAGLE-512 > URL: https://issues.apache.org/jira/browse/EAGLE-512 > Project: Eagle > Issue Type: New Feature >Reporter: wujinhu >Assignee: wujinhu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-512) add job count statistics to mr history feeder
[ https://issues.apache.org/jira/browse/EAGLE-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15448646#comment-15448646 ] ASF GitHub Bot commented on EAGLE-512: -- GitHub user wujinhu opened a pull request: https://github.com/apache/incubator-eagle/pull/406 [EAGLE-512] add job count(total&fail) for mr history feeder You can merge this pull request into a Git repository by running: $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-512 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/406.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #406 commit 3b305482b0252fc12546944b3e9577655ec9362e Author: wujinhu Date: 2016-08-30T10:02:45Z add job count(total&fail) for mr history feeder > add job count statistics to mr history feeder > - > > Key: EAGLE-512 > URL: https://issues.apache.org/jira/browse/EAGLE-512 > Project: Eagle > Issue Type: New Feature >Reporter: wujinhu >Assignee: wujinhu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request #406: [EAGLE-512] add job count(total&fail) for...
GitHub user wujinhu opened a pull request: https://github.com/apache/incubator-eagle/pull/406 [EAGLE-512] add job count(total&fail) for mr history feeder You can merge this pull request into a Git repository by running: $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-512 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/406.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #406 commit 3b305482b0252fc12546944b3e9577655ec9362e Author: wujinhu Date: 2016-08-30T10:02:45Z add job count(total&fail) for mr history feeder --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
