[jira] [Commented] (EAGLE-136) Setup hadoop metric application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110295#comment-15110295
 ] 

ASF GitHub Bot commented on EAGLE-136:
--

GitHub user RalphSu opened a pull request:

https://github.com/apache/incubator-eagle/pull/68

EAGLE-136 : Setup hadoop metric application

https://issues.apache.org/jira/browse/EAGLE-136

1. Add eagle-hadoopo-metrics project
2. Add NameNodeLagMonitor app
3. Fix eagle core to remove eagleAlertContext injection
4. Fix the failed JDBC unit test (by comment out)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/RalphSu/incubator-eagle hadoop-metric

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/68.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #68


commit 22b29565b087fb39d9f403ac86eeaea4debac6b7
Author: Ralph, Su 
Date:   2016-01-13T02:24:52Z

init commit for hadoop-metric

commit aa3a4cfa56365849a83b4fc6169b139219949b9e
Author: Ralph, Su 
Date:   2016-01-17T11:33:47Z

second metric commit

commit 6c7c75e8ff06b86f19a4550b81c0ebba7e06f9ae
Author: Ralph, Su 
Date:   2016-01-17T11:33:54Z

Merge branch 'master' of https://github.com/apache/incubator-eagle into 
hadoop-metric

commit d12e7629526174dc6f7d08fb08ecd6b48384d314
Author: Ralph, Su 
Date:   2016-01-17T14:51:02Z

add NameNodeLag sql case

commit 17a62a03788a2a4f3ffa27697bddd7c8c73c8a10
Author: Ralph, Su 
Date:   2016-01-17T14:55:40Z

use query call back that used in the alert executor

commit 16750d16907b79b7b8e4e4faabdc6cf4071f1eea
Author: Ralph, Su 
Date:   2016-01-18T08:48:30Z

add more case

commit 31a9e8c926efefd8646185e6ab31a0bb519b4589
Author: Ralph, Su 
Date:   2016-01-19T11:40:21Z

add topology init shell

commit 9ffecf064964e71dee7e06ddb24eaf87b2082a1d
Author: Ralph, Su 
Date:   2016-01-19T14:40:31Z

Make #1 case alert

commit a462cb40452406a73403881b9904be2e5e78ecb0
Author: Ralph, Su 
Date:   2016-01-20T07:40:32Z

add test for simple aggregate executor

commit 659ab4987528863e862b01b911716e43bc298f87
Author: Ralph, Su 
Date:   2016-01-21T08:09:20Z

Fix to avoid inject eagleAlertContext into cep ql

commit 7e1151b362edf53e7541049c3fde9f2be5ac463c
Author: Ralph, Su 
Date:   2016-01-21T08:44:28Z

 merge with latest master and pass all UTs

commit 911f493e9ffe5a5db6442a00ccb30eb41e409b32
Author: Ralph, Su 
Date:   2016-01-21T08:53:12Z

Fix to remove sensitive information

commit 13071e2a7671e829b58f65f186599a41133c6bbd
Author: Ralph, Su 
Date:   2016-01-21T08:55:56Z

correct mail host




> Setup hadoop metric application
> ---
>
> Key: EAGLE-136
> URL: https://issues.apache.org/jira/browse/EAGLE-136
> Project: Eagle
>  Issue Type: Task
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-136 : Setup hadoop metric appl...

[RalphSu]

Github user RalphSu commented on the pull request:

https://github.com/apache/incubator-eagle/pull/68#issuecomment-173504214
  
@sunlibin @haoch @yonzhang 

Please help review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: EAGLE-136 : Setup hadoop metric appl...

[RalphSu]

GitHub user RalphSu opened a pull request:

https://github.com/apache/incubator-eagle/pull/68

EAGLE-136 : Setup hadoop metric application

https://issues.apache.org/jira/browse/EAGLE-136

1. Add eagle-hadoopo-metrics project
2. Add NameNodeLagMonitor app
3. Fix eagle core to remove eagleAlertContext injection
4. Fix the failed JDBC unit test (by comment out)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/RalphSu/incubator-eagle hadoop-metric

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/68.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #68


commit 22b29565b087fb39d9f403ac86eeaea4debac6b7
Author: Ralph, Su 
Date:   2016-01-13T02:24:52Z

init commit for hadoop-metric

commit aa3a4cfa56365849a83b4fc6169b139219949b9e
Author: Ralph, Su 
Date:   2016-01-17T11:33:47Z

second metric commit

commit 6c7c75e8ff06b86f19a4550b81c0ebba7e06f9ae
Author: Ralph, Su 
Date:   2016-01-17T11:33:54Z

Merge branch 'master' of https://github.com/apache/incubator-eagle into 
hadoop-metric

commit d12e7629526174dc6f7d08fb08ecd6b48384d314
Author: Ralph, Su 
Date:   2016-01-17T14:51:02Z

add NameNodeLag sql case

commit 17a62a03788a2a4f3ffa27697bddd7c8c73c8a10
Author: Ralph, Su 
Date:   2016-01-17T14:55:40Z

use query call back that used in the alert executor

commit 16750d16907b79b7b8e4e4faabdc6cf4071f1eea
Author: Ralph, Su 
Date:   2016-01-18T08:48:30Z

add more case

commit 31a9e8c926efefd8646185e6ab31a0bb519b4589
Author: Ralph, Su 
Date:   2016-01-19T11:40:21Z

add topology init shell

commit 9ffecf064964e71dee7e06ddb24eaf87b2082a1d
Author: Ralph, Su 
Date:   2016-01-19T14:40:31Z

Make #1 case alert

commit a462cb40452406a73403881b9904be2e5e78ecb0
Author: Ralph, Su 
Date:   2016-01-20T07:40:32Z

add test for simple aggregate executor

commit 659ab4987528863e862b01b911716e43bc298f87
Author: Ralph, Su 
Date:   2016-01-21T08:09:20Z

Fix to avoid inject eagleAlertContext into cep ql

commit 7e1151b362edf53e7541049c3fde9f2be5ac463c
Author: Ralph, Su 
Date:   2016-01-21T08:44:28Z

 merge with latest master and pass all UTs

commit 911f493e9ffe5a5db6442a00ccb30eb41e409b32
Author: Ralph, Su 
Date:   2016-01-21T08:53:12Z

Fix to remove sensitive information

commit 13071e2a7671e829b58f65f186599a41133c6bbd
Author: Ralph, Su 
Date:   2016-01-21T08:55:56Z

correct mail host




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-136) Setup hadoop metric application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110296#comment-15110296
 ] 

ASF GitHub Bot commented on EAGLE-136:
--

Github user RalphSu commented on the pull request:

https://github.com/apache/incubator-eagle/pull/68#issuecomment-173504214
  
@sunlibin @haoch @yonzhang 

Please help review.


> Setup hadoop metric application
> ---
>
> Key: EAGLE-136
> URL: https://issues.apache.org/jira/browse/EAGLE-136
> Project: Eagle
>  Issue Type: Task
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-136) Setup hadoop metric application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110369#comment-15110369
 ] 

ASF GitHub Bot commented on EAGLE-136:
--

Github user haoch commented on a diff in the pull request:

https://github.com/apache/incubator-eagle/pull/68#discussion_r50380417
  
--- Diff: 
eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/HadoopJmxMetricDeserializer.java
 ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.hadoop.metric;
+
+import org.apache.eagle.dataproc.core.JsonSerDeserUtils;
+import 
org.apache.eagle.dataproc.impl.storm.kafka.SpoutKafkaMessageDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.Charset;
+import java.util.Map;
+import java.util.Properties;
+import java.util.SortedMap;
+
+/**
+ * Created on 1/19/16.
+ */
+public class HadoopJmxMetricDeserializer implements 
SpoutKafkaMessageDeserializer {
--- End diff --

Duplicated implementation of 
org.apache.eagle.datastream.storm.JsonMessageDeserializer


> Setup hadoop metric application
> ---
>
> Key: EAGLE-136
> URL: https://issues.apache.org/jira/browse/EAGLE-136
> Project: Eagle
>  Issue Type: Task
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-136 : Setup hadoop metric appl...

[haoch]

Github user haoch commented on a diff in the pull request:

https://github.com/apache/incubator-eagle/pull/68#discussion_r50380417
  
--- Diff: 
eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/HadoopJmxMetricDeserializer.java
 ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.hadoop.metric;
+
+import org.apache.eagle.dataproc.core.JsonSerDeserUtils;
+import 
org.apache.eagle.dataproc.impl.storm.kafka.SpoutKafkaMessageDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.Charset;
+import java.util.Map;
+import java.util.Properties;
+import java.util.SortedMap;
+
+/**
+ * Created on 1/19/16.
+ */
+public class HadoopJmxMetricDeserializer implements 
SpoutKafkaMessageDeserializer {
--- End diff --

Duplicated implementation of 
org.apache.eagle.datastream.storm.JsonMessageDeserializer


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: EAGLE-136 : Setup hadoop metric appl...

[haoch]

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/68#issuecomment-173529965
  
LGTM

The PR builds successfully and works fine for me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-136) Setup hadoop metric application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110395#comment-15110395
 ] 

ASF GitHub Bot commented on EAGLE-136:
--

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/68#issuecomment-173529965
  
LGTM

The PR builds successfully and works fine for me.


> Setup hadoop metric application
> ---
>
> Key: EAGLE-136
> URL: https://issues.apache.org/jira/browse/EAGLE-136
> Project: Eagle
>  Issue Type: Task
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Cannot compile unknown AlertExecutor to a Storm Spout

[Krishna, Murali(AWF)]

Hi Team,

Running into below error when I run HdfsAuditLogProcessorMain. Using the 
sandbox-hdfsAuditLog-application.conf from assembly module as input to this.
Can someone point me in the right direction.

Exception in thread "main" java.lang.IllegalArgumentException: Cannot compile 
unknown AlertExecutor to a Storm Spout
at 
org.apache.eagle.datastream.storm.StormSpoutFactory$.createSpout(StormSpoutFactory.scala:43)
at 
org.apache.eagle.datastream.storm.StormTopologyCompiler.buildTopology(StormTopologyCompiler.scala:47)
at 
org.apache.eagle.datastream.storm.StormExecutionEnvironment.execute(StormExecutionEnvironment.scala:29)
at 
org.apache.eagle.datastream.core.StreamContextAdapter$class.submit(ExecutionEnvironment.scala:25)
at 
org.apache.eagle.datastream.core.ExecutionEnvironment.submit(ExecutionEnvironment.scala:35)
at 
org.apache.eagle.datastream.core.StreamContext.submit(StreamBuilder.scala:67)
at 
org.apache.eagle.datastream.core.ExecutionEnvironment.execute(ExecutionEnvironment.scala:43)
at 
org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain.execWithBalancedPartition(HdfsAuditLogProcessorMain.java:100)
at 
org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain.main(HdfsAuditLogProcessorMain.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

Thanks,
Murali Krishna

Re: Cannot compile unknown AlertExecutor to a Storm Spout

[Chen, Hao]

Thanks for reporting it, I will investigate the root cause.

Thanks,
Hao




On 1/21/16, 6:48 PM, "Krishna, Murali(AWF)"  wrote:

>Hi Team,
>
>Running into below error when I run HdfsAuditLogProcessorMain. Using the 
>sandbox-hdfsAuditLog-application.conf from assembly module as input to this.
>Can someone point me in the right direction.
>
>Exception in thread "main" java.lang.IllegalArgumentException: Cannot compile 
>unknown AlertExecutor to a Storm Spout
>at 
> org.apache.eagle.datastream.storm.StormSpoutFactory$.createSpout(StormSpoutFactory.scala:43)
>at 
> org.apache.eagle.datastream.storm.StormTopologyCompiler.buildTopology(StormTopologyCompiler.scala:47)
>at 
> org.apache.eagle.datastream.storm.StormExecutionEnvironment.execute(StormExecutionEnvironment.scala:29)
>at 
> org.apache.eagle.datastream.core.StreamContextAdapter$class.submit(ExecutionEnvironment.scala:25)
>at 
> org.apache.eagle.datastream.core.ExecutionEnvironment.submit(ExecutionEnvironment.scala:35)
>at 
> org.apache.eagle.datastream.core.StreamContext.submit(StreamBuilder.scala:67)
>at 
> org.apache.eagle.datastream.core.ExecutionEnvironment.execute(ExecutionEnvironment.scala:43)
>at 
> org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain.execWithBalancedPartition(HdfsAuditLogProcessorMain.java:100)
>at 
> org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain.main(HdfsAuditLogProcessorMain.java:109)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.lang.reflect.Method.invoke(Method.java:606)
>at 
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>Thanks,
>Murali Krishna

[jira] [Commented] (EAGLE-130) Eagle Pipeline DSL: Parser, Compiler, Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110432#comment-15110432
 ] 

ASF GitHub Bot commented on EAGLE-130:
--

GitHub user haoch opened a pull request:

https://github.com/apache/incubator-eagle/pull/69

EAGLE-130 Fix bad expansion logic for FlatMapProducer in AlertExpansion

https://issues.apache.org/jira/browse/EAGLE-130

Fix bad expansion logic for FlatMapProducer in AlertExpansion

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/haoch/incubator-eagle EAGLE-130-4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/69.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #69


commit 8ffaa0f2666f0303a2d6857d21e8d44c814253f1
Author: Hao Chen 
Date:   2016-01-21T11:05:34Z

Fix bad expansion logic for FlatMapProducer in AlertExpansion




> Eagle Pipeline DSL: Parser, Compiler, Runner
> 
>
> Key: EAGLE-130
> URL: https://issues.apache.org/jira/browse/EAGLE-130
> Project: Eagle
>  Issue Type: Sub-task
>Reporter: Hao Chen
>Assignee: Hao Chen
>
> https://cwiki.apache.org/confluence/display/EAG/Dynamical+Monitoring+Framework#DynamicalMonitoringFramework-DynamicalPipelineBuilder



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-130 Fix bad expansion logic fo...

[haoch]

GitHub user haoch opened a pull request:

https://github.com/apache/incubator-eagle/pull/69

EAGLE-130 Fix bad expansion logic for FlatMapProducer in AlertExpansion

https://issues.apache.org/jira/browse/EAGLE-130

Fix bad expansion logic for FlatMapProducer in AlertExpansion

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/haoch/incubator-eagle EAGLE-130-4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/69.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #69


commit 8ffaa0f2666f0303a2d6857d21e8d44c814253f1
Author: Hao Chen 
Date:   2016-01-21T11:05:34Z

Fix bad expansion logic for FlatMapProducer in AlertExpansion




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: EAGLE-130 Fix bad expansion logic fo...

[haoch]

Github user haoch closed the pull request at:

https://github.com/apache/incubator-eagle/pull/69


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-130) Eagle Pipeline DSL: Parser, Compiler, Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110441#comment-15110441
 ] 

ASF GitHub Bot commented on EAGLE-130:
--

GitHub user haoch opened a pull request:

https://github.com/apache/incubator-eagle/pull/70

EAGLE-130 Fix bad expansion logic for FlatMapProducer in AlertExpansion

Fix bad expansion logic for FlatMapProducer in AlertExpansion

https://issues.apache.org/jira/browse/EAGLE-130

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/haoch/incubator-eagle EAGLE-130-5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/70.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #70


commit cf9dfa7ad8efcbd5ffce6acde00016a9568b
Author: Hao Chen 
Date:   2016-01-21T11:15:22Z

EAGLE-130 Fix bad expansion logic for FlatMapProducer in AlertExpansion




> Eagle Pipeline DSL: Parser, Compiler, Runner
> 
>
> Key: EAGLE-130
> URL: https://issues.apache.org/jira/browse/EAGLE-130
> Project: Eagle
>  Issue Type: Sub-task
>Reporter: Hao Chen
>Assignee: Hao Chen
>
> https://cwiki.apache.org/confluence/display/EAG/Dynamical+Monitoring+Framework#DynamicalMonitoringFramework-DynamicalPipelineBuilder



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-130 Fix bad expansion logic fo...

[haoch]

GitHub user haoch opened a pull request:

https://github.com/apache/incubator-eagle/pull/70

EAGLE-130 Fix bad expansion logic for FlatMapProducer in AlertExpansion

Fix bad expansion logic for FlatMapProducer in AlertExpansion

https://issues.apache.org/jira/browse/EAGLE-130

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/haoch/incubator-eagle EAGLE-130-5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/70.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #70


commit cf9dfa7ad8efcbd5ffce6acde00016a9568b
Author: Hao Chen 
Date:   2016-01-21T11:15:22Z

EAGLE-130 Fix bad expansion logic for FlatMapProducer in AlertExpansion




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-130) Eagle Pipeline DSL: Parser, Compiler, Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110439#comment-15110439
 ] 

ASF GitHub Bot commented on EAGLE-130:
--

Github user haoch closed the pull request at:

https://github.com/apache/incubator-eagle/pull/69


> Eagle Pipeline DSL: Parser, Compiler, Runner
> 
>
> Key: EAGLE-130
> URL: https://issues.apache.org/jira/browse/EAGLE-130
> Project: Eagle
>  Issue Type: Sub-task
>Reporter: Hao Chen
>Assignee: Hao Chen
>
> https://cwiki.apache.org/confluence/display/EAG/Dynamical+Monitoring+Framework#DynamicalMonitoringFramework-DynamicalPipelineBuilder



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-130 Fix bad expansion logic fo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-eagle/pull/70


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-130) Eagle Pipeline DSL: Parser, Compiler, Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110445#comment-15110445
 ] 

ASF GitHub Bot commented on EAGLE-130:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-eagle/pull/70


> Eagle Pipeline DSL: Parser, Compiler, Runner
> 
>
> Key: EAGLE-130
> URL: https://issues.apache.org/jira/browse/EAGLE-130
> Project: Eagle
>  Issue Type: Sub-task
>Reporter: Hao Chen
>Assignee: Hao Chen
>
> https://cwiki.apache.org/confluence/display/EAG/Dynamical+Monitoring+Framework#DynamicalMonitoringFramework-DynamicalPipelineBuilder



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Cannot compile unknown AlertExecutor to a Storm Spout

[Chen, Hao]

Resolved: https://github.com/apache/incubator-eagle/pull/70


On 1/21/16, 6:53 PM, "Chen, Hao"  wrote:

>Thanks for reporting it, I will investigate the root cause.
>
>Thanks,
>Hao
>
>
>
>
>On 1/21/16, 6:48 PM, "Krishna, Murali(AWF)"  wrote:
>
>>Hi Team,
>>
>>Running into below error when I run HdfsAuditLogProcessorMain. Using the 
>>sandbox-hdfsAuditLog-application.conf from assembly module as input to this.
>>Can someone point me in the right direction.
>>
>>Exception in thread "main" java.lang.IllegalArgumentException: Cannot compile 
>>unknown AlertExecutor to a Storm Spout
>>at 
>> org.apache.eagle.datastream.storm.StormSpoutFactory$.createSpout(StormSpoutFactory.scala:43)
>>at 
>> org.apache.eagle.datastream.storm.StormTopologyCompiler.buildTopology(StormTopologyCompiler.scala:47)
>>at 
>> org.apache.eagle.datastream.storm.StormExecutionEnvironment.execute(StormExecutionEnvironment.scala:29)
>>at 
>> org.apache.eagle.datastream.core.StreamContextAdapter$class.submit(ExecutionEnvironment.scala:25)
>>at 
>> org.apache.eagle.datastream.core.ExecutionEnvironment.submit(ExecutionEnvironment.scala:35)
>>at 
>> org.apache.eagle.datastream.core.StreamContext.submit(StreamBuilder.scala:67)
>>at 
>> org.apache.eagle.datastream.core.ExecutionEnvironment.execute(ExecutionEnvironment.scala:43)
>>at 
>> org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain.execWithBalancedPartition(HdfsAuditLogProcessorMain.java:100)
>>at 
>> org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain.main(HdfsAuditLogProcessorMain.java:109)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>at java.lang.reflect.Method.invoke(Method.java:606)
>>at 
>> com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>>
>>Thanks,
>>Murali Krishna

[EAGLE BUILD FAILED] incubator-eagle-main failed on build #20

[Apache Jenkins Server]

The Apache Jenkins build system has built incubator-eagle-main (build #20) 
Status: Still Failing Check console output at 
https://builds.apache.org/job/incubator-eagle-main/20/ to view the results. 
Test Report is at: Test Result

RE: Cannot compile unknown AlertExecutor to a Storm Spout

[Krishna, Murali(AWF)]

Thanks Hao.

- Murali Krishna

-Original Message-
From: Chen, Hao [mailto:hao.ch...@ebay.com] 
Sent: Thursday, January 21, 2016 4:51 PM
To: dev@eagle.incubator.apache.org
Subject: Re: Cannot compile unknown AlertExecutor to a Storm Spout

Resolved: https://github.com/apache/incubator-eagle/pull/70


On 1/21/16, 6:53 PM, "Chen, Hao"  wrote:

>Thanks for reporting it, I will investigate the root cause.
>
>Thanks,
>Hao
>
>
>
>
>On 1/21/16, 6:48 PM, "Krishna, Murali(AWF)"  wrote:
>
>>Hi Team,
>>
>>Running into below error when I run HdfsAuditLogProcessorMain. Using the 
>>sandbox-hdfsAuditLog-application.conf from assembly module as input to this.
>>Can someone point me in the right direction.
>>
>>Exception in thread "main" java.lang.IllegalArgumentException: Cannot compile 
>>unknown AlertExecutor to a Storm Spout
>>at 
>> org.apache.eagle.datastream.storm.StormSpoutFactory$.createSpout(StormSpoutFactory.scala:43)
>>at 
>> org.apache.eagle.datastream.storm.StormTopologyCompiler.buildTopology(StormTopologyCompiler.scala:47)
>>at 
>> org.apache.eagle.datastream.storm.StormExecutionEnvironment.execute(StormExecutionEnvironment.scala:29)
>>at 
>> org.apache.eagle.datastream.core.StreamContextAdapter$class.submit(ExecutionEnvironment.scala:25)
>>at 
>> org.apache.eagle.datastream.core.ExecutionEnvironment.submit(ExecutionEnvironment.scala:35)
>>at 
>> org.apache.eagle.datastream.core.StreamContext.submit(StreamBuilder.scala:67)
>>at 
>> org.apache.eagle.datastream.core.ExecutionEnvironment.execute(ExecutionEnvironment.scala:43)
>>at 
>> org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain.execWithBalancedPartition(HdfsAuditLogProcessorMain.java:100)
>>at 
>> org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain.main(HdfsAuditLogProcessorMain.java:109)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>at java.lang.reflect.Method.invoke(Method.java:606)
>>at 
>> com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>>
>>Thanks,
>>Murali Krishna

[GitHub] incubator-eagle pull request: EAGLE-136 : Setup hadoop metric appl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-eagle/pull/68


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: EAGLE-136 : Setup hadoop metric appl...

[haoch]

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/68#issuecomment-173617368
  
@RalphSu  Merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-136) Setup hadoop metric application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110828#comment-15110828
 ] 

ASF GitHub Bot commented on EAGLE-136:
--

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/68#issuecomment-173617368
  
@RalphSu  Merged.


> Setup hadoop metric application
> ---
>
> Key: EAGLE-136
> URL: https://issues.apache.org/jira/browse/EAGLE-136
> Project: Eagle
>  Issue Type: Task
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-136) Setup hadoop metric application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110826#comment-15110826
 ] 

ASF GitHub Bot commented on EAGLE-136:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-eagle/pull/68


> Setup hadoop metric application
> ---
>
> Key: EAGLE-136
> URL: https://issues.apache.org/jira/browse/EAGLE-136
> Project: Eagle
>  Issue Type: Task
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (EAGLE-136) Setup hadoop metric application

[Hao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Chen resolved EAGLE-136.

Resolution: Resolved

> Setup hadoop metric application
> ---
>
> Key: EAGLE-136
> URL: https://issues.apache.org/jira/browse/EAGLE-136
> Project: Eagle
>  Issue Type: Task
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-136) Setup hadoop metric application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111004#comment-15111004
 ] 

ASF GitHub Bot commented on EAGLE-136:
--

Github user yonzhang commented on the pull request:

https://github.com/apache/incubator-eagle/pull/68#issuecomment-173654111
  
Nice and complete feature


> Setup hadoop metric application
> ---
>
> Key: EAGLE-136
> URL: https://issues.apache.org/jira/browse/EAGLE-136
> Project: Eagle
>  Issue Type: Task
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-136 : Setup hadoop metric appl...

[yonzhang]

Github user yonzhang commented on the pull request:

https://github.com/apache/incubator-eagle/pull/68#issuecomment-173654111
  
Nice and complete feature


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: EAGLE-104 fix unit test bug in hdfs ...

[RalphSu]

Github user RalphSu commented on the pull request:

https://github.com/apache/incubator-eagle/pull/61#issuecomment-173768560
  
Per commit log, this one has been merged, but the commit message does form 
well. (Should be Closes #61, but it's Closes 61).

Could we manually close this one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-104) Fix the unit test TestHDFSSecuritylogParser

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111731#comment-15111731
 ] 

ASF GitHub Bot commented on EAGLE-104:
--

Github user RalphSu commented on the pull request:

https://github.com/apache/incubator-eagle/pull/61#issuecomment-173768560
  
Per commit log, this one has been merged, but the commit message does form 
well. (Should be Closes #61, but it's Closes 61).

Could we manually close this one?


> Fix the unit test TestHDFSSecuritylogParser  
> -
>
> Key: EAGLE-104
> URL: https://issues.apache.org/jira/browse/EAGLE-104
> Project: Eagle
>  Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.3.0
>
> Attachments: EAGLE-104.1.patch
>
>
> The test fails with following error
> {noformat}
> java.lang.StringIndexOutOfBoundsException">
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at
> java.lang.String.substring(String.java:1911) at
> org.apache.eagle.security.util.LogParseUtil.parseUserFromUGI(LogParseUtil.java:39)
> at
> org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parseAttr(HDFSSecurityLogParser.java:78)
> at
> org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parse(HDFSSecurityLogParser.java:52)
> at
> org.apache.eagle.security.TestHDFSSecuritylogParser.test(TestHDFSSecuritylogParser.java:36)
> {noformat}
> Looks like the problem is that the UGI parsing method is not handling the 
> simple authentication scenario where user name is not a kerberos principal. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-11111 This pr is for testing c...

[anyway1021]

Github user anyway1021 closed the pull request at:

https://github.com/apache/incubator-eagle/pull/48


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: EAGLE-11111 This pr is for testing c...

[anyway1021]

GitHub user anyway1021 reopened a pull request:

https://github.com/apache/incubator-eagle/pull/48

EAGLE-1 This pr is for testing ci builds, please don't review or 
comment for it, thanks.

This pr is for testing ci builds, please don't review or comment for it, 
thanks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anyway1021/incubator-eagle test_qpl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/48.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #48


commit 08376deaac1636b394211af48b00cfa2e5d389db
Author: anyway1021 
Date:   2015-12-30T06:51:12Z

eagle-test/pr_merge_flag.file




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (EAGLE-131) make eagle main CI build pass

[Michael Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Wu closed EAGLE-131.

Resolution: Resolved

The job succeeded with all unit test cases passing at: 
https://builds.apache.org/job/incubator-eagle-main/21/

> make eagle main CI build pass
> -
>
> Key: EAGLE-131
> URL: https://issues.apache.org/jira/browse/EAGLE-131
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
>Priority: Critical
>
> task complete measurement should be:
> 1. the whole project is completely and successfully built
> 2. all unit tests run pass



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (EAGLE-137) narrow unit test scope of pr reviewer ci job so that it only test the affections in the corresponding pr

[Michael Wu (JIRA)]

Michael Wu created EAGLE-137:


 Summary: narrow unit test scope of pr reviewer ci job so that it 
only test the affections in the corresponding pr
 Key: EAGLE-137
 URL: https://issues.apache.org/jira/browse/EAGLE-137
 Project: Eagle
  Issue Type: Improvement
Reporter: Michael Wu
Assignee: Michael Wu
Priority: Critical


Currently, the job checking PR is running all the unit test cases as a whole, 
we have to narrow down the scope to focus on the things affected only in the PR.

This may not be accomplishable based on what we now have, but we should try it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (EAGLE-134) publish javadoc of eagle

[Michael Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Wu updated EAGLE-134:
-
Issue Type: Improvement  (was: Bug)

> publish javadoc of eagle
> 
>
> Key: EAGLE-134
> URL: https://issues.apache.org/jira/browse/EAGLE-134
> Project: Eagle
>  Issue Type: Improvement
>Reporter: Michael Wu
>Assignee: Michael Wu
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (EAGLE-138) Refactor JMX Collector to Enhance extensibility

[Hao Chen (JIRA)]

Hao Chen created EAGLE-138:
--

 Summary: Refactor JMX Collector to Enhance extensibility
 Key: EAGLE-138
 URL: https://issues.apache.org/jira/browse/EAGLE-138
 Project: Eagle
  Issue Type: Improvement
Affects Versions: 0.3.0
Reporter: Hao Chen
Assignee: Hao Chen
 Fix For: 0.3.0


Refactor JMX Collector to Enhance extensibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-95 DAM HDFS topology fails to ...

[murkrishn]

GitHub user murkrishn opened a pull request:

https://github.com/apache/incubator-eagle/pull/71

EAGLE-95 DAM HDFS topology fails to start, if there is invalid policy

Code changes include resolution to the bug - unable to start a topology 
with has an invalid policy defined already by skipping SiddhiRuntime execution 
to start for that policy. Achieved by introducing a markdown feature every 
policy defined (before/during topology runtime). The markdown will dictate 
actions related to that policy. 

Design document is available 
[here](https://cwiki.apache.org/confluence/display/EAG/Policy+Markdown+Feature).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/murkrishn/incubator-eagle EAGLE-95

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/71.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #71


commit fc28debb0e59849f5e3a80770379e04e21375dad
Author: murkrishn 
Date:   2016-01-22T05:54:44Z

Initial commit for changes related to jira: EAGLE-95. Updating markdown 
details to resolve topology startup failure with an invalid policy and using 
the markdown details as a check before sending an incoming event against an 
invalid policy.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-95) DAM HDFS topology fails to start, if there is invalid policy.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-95?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111986#comment-15111986
 ] 

ASF GitHub Bot commented on EAGLE-95:
-

GitHub user murkrishn opened a pull request:

https://github.com/apache/incubator-eagle/pull/71

EAGLE-95 DAM HDFS topology fails to start, if there is invalid policy

Code changes include resolution to the bug - unable to start a topology 
with has an invalid policy defined already by skipping SiddhiRuntime execution 
to start for that policy. Achieved by introducing a markdown feature every 
policy defined (before/during topology runtime). The markdown will dictate 
actions related to that policy. 

Design document is available 
[here](https://cwiki.apache.org/confluence/display/EAG/Policy+Markdown+Feature).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/murkrishn/incubator-eagle EAGLE-95

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/71.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #71


commit fc28debb0e59849f5e3a80770379e04e21375dad
Author: murkrishn 
Date:   2016-01-22T05:54:44Z

Initial commit for changes related to jira: EAGLE-95. Updating markdown 
details to resolve topology startup failure with an invalid policy and using 
the markdown details as a check before sending an incoming event against an 
invalid policy.




> DAM HDFS topology fails to start, if there is invalid policy.
> -
>
> Key: EAGLE-95
> URL: https://issues.apache.org/jira/browse/EAGLE-95
> Project: Eagle
>  Issue Type: Bug
>Reporter: hemanth dendukuri
>Assignee: Senthilkumar
>Priority: Critical
>
> Step 1: Create a invalid policy (from hdfsAuditLogEventStream[(cmd == 'open') 
> and (str:contains(src,'/tmp/p')==true)] select * insert into outputStream;) 
> Step 2 : Deploy Dam HDFS topology. 
> Actual : The HDFS topology could not start because of the invalid policy. 
> 2015-12-16 23:51:17 b.s.d.executor [INFO] Shutting down executor 
> AlertEntityDeduplicationExecutor_9:[2 2] 2015-12-16 23:51:17 b.s.util [INFO] 
> Async loop interrupted! 
> Expected: Topology should start ignoring the invalid policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (EAGLE-96) Support activity monitoring for Knox

[Senthilkumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-96?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Senthilkumar reassigned EAGLE-96:
-

Assignee: Senthilkumar

> Support activity monitoring for Knox
> 
>
> Key: EAGLE-96
> URL: https://issues.apache.org/jira/browse/EAGLE-96
> Project: Eagle
>  Issue Type: Bug
>Reporter: Arun Manoharan
>Assignee: Senthilkumar
>
> The Knox Gateway provides a single access point for all REST interactions 
> with Hadoop clusters. It will be valuable to monitor the access events 
> happening in knox gateway and see if there is an anomaly and generate an 
> alert. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-96) Support activity monitoring for Knox

[Senthilkumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112027#comment-15112027
 ] 

Senthilkumar commented on EAGLE-96:
---

Sample gateway logs:

>> #1 CAN BE USED FOR MONITORING INBOUND REQUESTS TO KNOX
>> REQUEST TO KNOX GATEWAY

2015-11-24 18:56:59,985 DEBUG server.Server (Server.java:handle(365)) - REQUEST 
/gateway/sandbox/webhdfs/v1/tmp/knox on 
AsyncHttpConnection@267d7dc4,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-5,l=3,c=0},r=1
2015-11-24 18:56:59,985 DEBUG server.session (SessionHandler.java:doScope(187)) 
- sessionManager=org.eclipse.jetty.server.session.HashSessionManager@533824d3

>> #2 WHICH USER IS TRYING TO REQUEST THE SERVICE
>> AUTHENTICATING USER AGAINST LDAPS SERVER

2015-11-24 18:56:59,986 DEBUG authc.BasicHttpAuthenticationFilter 
(BasicHttpAuthenticationFilter.java:createToken(308)) - Attempting to execute 
login with headers [Basic bXVya3Jpc2huYTpNM2VpejAhJCM=]
2015-11-24 18:56:59,987 DEBUG ldap.JndiLdapRealm 
(JndiLdapRealm.java:queryForAuthenticationInfo(369)) - Authenticating user 
'murkrishna' through LDAP

>> #3 AUTHENTICATION DONE AGAINST WHICH SERVER AND WITH WHICH USER DOMAIN
>> userDN and LDAPS SERVER DETAILS WHERE AUTH TAKES PLACE

2015-11-24 18:56:59,987 INFO  hadoop.gateway 
(KnoxLdapRealm.java:getUserDn(513)) - Computed userDn: 
uid=murkrishna,ou=People,dc=ebay,dc=com using dnTemplate for principal: 
murkrishna
2015-11-24 18:56:59,987 DEBUG ldap.JndiLdapContextFactory 
(JndiLdapContextFactory.java:getLdapContext(488)) - Initializing LDAP context 
using URL [ldaps://phxldap06.phx.ebay.com:636] and principal 
[uid=murkrishna,ou=People,dc=ebay,dc=com] with pooling disabled
2015-11-24 18:57:00,022 DEBUG realm.AuthenticatingRealm 
(AuthenticatingRealm.java:getAuthenticationInfo(569)) - Looked up 
AuthenticationInfo [murkrishna] from doGetAuthenticationInfo
2015-11-24 18:57:00,022 DEBUG realm.AuthenticatingRealm 
(AuthenticatingRealm.java:cacheAuthenticationInfoIfPossible(507)) - 
AuthenticationInfo caching is disabled for info [murkrishna].  Submitted token: 
[org.apache.shiro.authc.UsernamePasswordToken - murkrishna, rememberMe=false 
(10.115.42.50)].
2015-11-24 18:57:00,023 DEBUG authc.AbstractAuthenticator

>> #4 AUTH RESULT

AbstractAuthenticator.java:authenticate(231)) - Authentication successful for 
token [org.apache.shiro.authc.UsernamePasswordToken - murkrishna, 
rememberMe=false (10.115.42.50)].  Returned account [murkrishna]

>> #5 HADOOP SERVICE CALL DETAILS
>> ESTABLISH CONNECTION TO THE NODE WHERE THE ACTUAL SERVICE CALL TAKES PLACE

2015-11-24 18:57:00,029 DEBUG conn.BasicClientConnectionManager 
(BasicClientConnectionManager.java:getConnection(159)) - Get connection for 
route {}->http://phx4b03c-378d.stratus.phx.ebay.com:50070
2015-11-24 18:57:00,029 DEBUG conn.DefaultClientConnectionOperator 
(DefaultClientConnectionOperator.java:openConnection(177)) - Connecting to 
phx4b03c-378d.stratus.phx.ebay.com:50070

>> AFTER SUCCESSFUL AUTHENTICATION, FORWARDING THE REQUEST TO THE ACTUAL 
>> SERVICE URL.. IN THIS CASE WebHDFS CALL.. PAYLOAD AND OTHER DETAILS OF THE 
>> REQUEST
>> CAN EXTRACT WHICH USER IS TRYING TO ACCESS WHICH FOLDER IN THIS CASE OF 
>> WebHDFS. SIMILAR DETAILS CAN BE GOT FROM USING HBase/Oozie SERVICES.

2015-11-24 18:57:00,031 DEBUG client.DefaultHttpClient 
(DefaultRequestDirector.java:tryExecute(713)) - Attempt 1 to execute request
2015-11-24 18:57:00,031 DEBUG conn.DefaultClientConnection 
(DefaultClientConnection.java:sendRequestHeader(269)) - Sending request: GET 
/webhdfs/v1/tmp/knox?user.name=murkrishna&op=LISTSTATUS HTTP/1.1
2015-11-24 18:57:00,031 DEBUG http.wire (Wire.java:wire(63)) - >> "GET 
/webhdfs/v1/tmp/knox?user.name=murkrishna&op=LISTSTATUS HTTP/1.1[\r][\n]"
2015-11-24 18:57:00,032 DEBUG http.wire (Wire.java:wire(63)) - >> "Accept: 
*/*[\r][\n]"
2015-11-24 18:57:00,032 DEBUG http.wire (Wire.java:wire(63)) - >> "User-Agent: 
curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 
libidn/1.18 libssh2/1.2.2[\r][\n]"
2015-11-24 18:57:00,032 DEBUG http.wire (Wire.java:wire(63)) - >> "Host: 
phx4b03c-378d.stratus.phx.ebay.com:50070[\r][\n]"
2015-11-24 18:57:00,032 DEBUG http.wire (Wire.java:wire(63)) - >> "Connection: 
Keep-Alive[\r][\n]"


We need to explore Knox audit/gateway logs to see all attributes like who 
requested ? URI etcc...

Above logs has all information but this requires combining three events 
together to grep user , uri , status etc... 

> Support activity monitoring for Knox
> 
>
> Key: EAGLE-96
> URL: https://issues.apache.org/jira/browse/EAGLE-96
> Project: Eagle
>  Issue Type: Bug
>Reporter: Arun Manoharan
>Assignee: Senthilkumar
>
> The Knox Gateway provides a single access point for all REST interactions 
> with Hadoop clusters. It will be valuable t