[jira] [Closed] (EAGLE-79) Provide analytic DSL support

[Hao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Chen closed EAGLE-79.
-

> Provide analytic DSL support 
> -
>
> Key: EAGLE-79
> URL: https://issues.apache.org/jira/browse/EAGLE-79
> Project: Eagle
>  Issue Type: New Feature
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Hao Chen
> Fix For: 0.3.0
>
>
> Eagle input comes with stream data (like security audit log), eagle provide 
> alerting computation based on CEP DSL.
> Similar to this process, eagle should be able to provide same DSL support to 
> expose the real-time monitoring feature, and furthermore could be integrated 
> with some storage backend to provide dashboard/presentation to user.
> This would require 
> 1. eagle programming API to support a new semantic of query(or aggregation), 
> using the similar alert DSL.
> 2. a clear definition of time series storage interface (so that different 
> storage engine could be adopted), currently we might start from the eagle 
> built-in hbase implementation
> 3. Metric API/Dashboard.
> Currently, it require a lot of user customization and CEP engine capability 
> could not be reused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (EAGLE-79) Provide analytic DSL support

[Hao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Chen resolved EAGLE-79.
---
Resolution: Done

> Provide analytic DSL support 
> -
>
> Key: EAGLE-79
> URL: https://issues.apache.org/jira/browse/EAGLE-79
> Project: Eagle
>  Issue Type: New Feature
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Hao Chen
> Fix For: 0.3.0
>
>
> Eagle input comes with stream data (like security audit log), eagle provide 
> alerting computation based on CEP DSL.
> Similar to this process, eagle should be able to provide same DSL support to 
> expose the real-time monitoring feature, and furthermore could be integrated 
> with some storage backend to provide dashboard/presentation to user.
> This would require 
> 1. eagle programming API to support a new semantic of query(or aggregation), 
> using the similar alert DSL.
> 2. a clear definition of time series storage interface (so that different 
> storage engine could be adopted), currently we might start from the eagle 
> built-in hbase implementation
> 3. Metric API/Dashboard.
> Currently, it require a lot of user customization and CEP engine capability 
> could not be reused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[EAGLE BUILD FAILED] incubator-eagle-main failed on build #9

[Apache Jenkins Server]

The Apache Jenkins build system has built incubator-eagle-main (build #9) 
Status: Still Failing Check console output at 
https://builds.apache.org/job/incubator-eagle-main/9/ to view the results. 
Test Report is at: Test Result

[jira] [Commented] (EAGLE-79) Provide analytic DSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093490#comment-15093490
 ] 

ASF GitHub Bot commented on EAGLE-79:
-

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/52#issuecomment-170828591
  
@RalphSu thanks for the contribution and pull request has been merged, 
please resolve the JIRA ticket https://issues.apache.org/jira/browse/EAGLE-79.


> Provide analytic DSL support 
> -
>
> Key: EAGLE-79
> URL: https://issues.apache.org/jira/browse/EAGLE-79
> Project: Eagle
>  Issue Type: New Feature
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
> Fix For: 0.3.0
>
>
> Eagle input comes with stream data (like security audit log), eagle provide 
> alerting computation based on CEP DSL.
> Similar to this process, eagle should be able to provide same DSL support to 
> expose the real-time monitoring feature, and furthermore could be integrated 
> with some storage backend to provide dashboard/presentation to user.
> This would require 
> 1. eagle programming API to support a new semantic of query(or aggregation), 
> using the similar alert DSL.
> 2. a clear definition of time series storage interface (so that different 
> storage engine could be adopted), currently we might start from the eagle 
> built-in hbase implementation
> 3. Metric API/Dashboard.
> Currently, it require a lot of user customization and CEP engine capability 
> could not be reused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-79 Provide basic aggregation

[haoch]

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/52#issuecomment-170828591
  
@RalphSu thanks for the contribution and pull request has been merged, 
please resolve the JIRA ticket https://issues.apache.org/jira/browse/EAGLE-79.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-79) Provide analytic DSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093489#comment-15093489
 ] 

ASF GitHub Bot commented on EAGLE-79:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-eagle/pull/52


> Provide analytic DSL support 
> -
>
> Key: EAGLE-79
> URL: https://issues.apache.org/jira/browse/EAGLE-79
> Project: Eagle
>  Issue Type: New Feature
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
> Fix For: 0.3.0
>
>
> Eagle input comes with stream data (like security audit log), eagle provide 
> alerting computation based on CEP DSL.
> Similar to this process, eagle should be able to provide same DSL support to 
> expose the real-time monitoring feature, and furthermore could be integrated 
> with some storage backend to provide dashboard/presentation to user.
> This would require 
> 1. eagle programming API to support a new semantic of query(or aggregation), 
> using the similar alert DSL.
> 2. a clear definition of time series storage interface (so that different 
> storage engine could be adopted), currently we might start from the eagle 
> built-in hbase implementation
> 3. Metric API/Dashboard.
> Currently, it require a lot of user customization and CEP engine capability 
> could not be reused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-79 Provide basic aggregation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-eagle/pull/52


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: EAGLE-79 Provide basic aggregation

[haoch]

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/52#issuecomment-170826317
  
Will merge this pull request into master branch today if no additional 
comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-79) Provide analytic DSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093483#comment-15093483
 ] 

ASF GitHub Bot commented on EAGLE-79:
-

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/52#issuecomment-170826317
  
Will merge this pull request into master branch today if no additional 
comments.


> Provide analytic DSL support 
> -
>
> Key: EAGLE-79
> URL: https://issues.apache.org/jira/browse/EAGLE-79
> Project: Eagle
>  Issue Type: New Feature
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
> Fix For: 0.3.0
>
>
> Eagle input comes with stream data (like security audit log), eagle provide 
> alerting computation based on CEP DSL.
> Similar to this process, eagle should be able to provide same DSL support to 
> expose the real-time monitoring feature, and furthermore could be integrated 
> with some storage backend to provide dashboard/presentation to user.
> This would require 
> 1. eagle programming API to support a new semantic of query(or aggregation), 
> using the similar alert DSL.
> 2. a clear definition of time series storage interface (so that different 
> storage engine could be adopted), currently we might start from the eagle 
> built-in hbase implementation
> 3. Metric API/Dashboard.
> Currently, it require a lot of user customization and CEP engine capability 
> could not be reused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-79) Provide analytic DSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093473#comment-15093473
 ] 

ASF GitHub Bot commented on EAGLE-79:
-

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/52#issuecomment-170824782
  
LGTM :+1: 



> Provide analytic DSL support 
> -
>
> Key: EAGLE-79
> URL: https://issues.apache.org/jira/browse/EAGLE-79
> Project: Eagle
>  Issue Type: New Feature
>Affects Versions: 0.3.0
>Reporter: Su Ralph
>Assignee: Su Ralph
> Fix For: 0.3.0
>
>
> Eagle input comes with stream data (like security audit log), eagle provide 
> alerting computation based on CEP DSL.
> Similar to this process, eagle should be able to provide same DSL support to 
> expose the real-time monitoring feature, and furthermore could be integrated 
> with some storage backend to provide dashboard/presentation to user.
> This would require 
> 1. eagle programming API to support a new semantic of query(or aggregation), 
> using the similar alert DSL.
> 2. a clear definition of time series storage interface (so that different 
> storage engine could be adopted), currently we might start from the eagle 
> built-in hbase implementation
> 3. Metric API/Dashboard.
> Currently, it require a lot of user customization and CEP engine capability 
> could not be reused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: EAGLE-79 Provide basic aggregation

[haoch]

Github user haoch commented on the pull request:

https://github.com/apache/incubator-eagle/pull/52#issuecomment-170824782
  
LGTM :+1: 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (EAGLE-84) Create scripts for merging PRs automatically

[Michael Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Wu resolved EAGLE-84.
-
Resolution: Done

It's now being tested.

After it's merged to master, I'll close this ticket.

> Create scripts for merging PRs automatically
> 
>
> Key: EAGLE-84
> URL: https://issues.apache.org/jira/browse/EAGLE-84
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Hao Chen
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> While a pull request gets passed the testing steps, and is verified as 
> good-to-merge, people are willing to merge it with scripts to go through 
> necessary steps automatically, and can form the message of final aggregated 
> commit in a unified format. To achieve this, we are attempted to create the 
> script.
> As discussed, the format of message should be like below (ignore the lines 
> consisted of asterisks):
> **
> EAGLE-${ticket_number} ${ticket_title}
> ${ticket_url}
> Author: ${commiter}
> Reviewer: ${reviewer}
> Closes #${pr_number} from ${do_person}
> **



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-104) Fix the unit test TestHDFSSecuritylogParser

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093431#comment-15093431
 ] 

ASF GitHub Bot commented on EAGLE-104:
--

Github user sunlibin commented on the pull request:

https://github.com/apache/incubator-eagle/pull/43#issuecomment-170817951
  
@pmujumdar, thanks for the fix, but please make parseUserFromUGI() without 
regular-expression, because it will affect our performance


> Fix the unit test TestHDFSSecuritylogParser  
> -
>
> Key: EAGLE-104
> URL: https://issues.apache.org/jira/browse/EAGLE-104
> Project: Eagle
>  Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Prasad Mujumdar
> Attachments: EAGLE-104.1.patch
>
>
> The test fails with following error
> {noformat}
> java.lang.StringIndexOutOfBoundsException">
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at
> java.lang.String.substring(String.java:1911) at
> org.apache.eagle.security.util.LogParseUtil.parseUserFromUGI(LogParseUtil.java:39)
> at
> org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parseAttr(HDFSSecurityLogParser.java:78)
> at
> org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parse(HDFSSecurityLogParser.java:52)
> at
> org.apache.eagle.security.TestHDFSSecuritylogParser.test(TestHDFSSecuritylogParser.java:36)
> {noformat}
> Looks like the problem is that the UGI parsing method is not handling the 
> simple authentication scenario where user name is not a kerberos principal. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: https://issues.apache.org/jira/brows...

[sunlibin]

Github user sunlibin commented on the pull request:

https://github.com/apache/incubator-eagle/pull/43#issuecomment-170817951
  
@pmujumdar, thanks for the fix, but please make parseUserFromUGI() without 
regular-expression, because it will affect our performance


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Fix Eagle Unit Test] please fix eagle's unit test

[Michael Wu]

Okay, thanks.

On Tue, Jan 12, 2016 at 12:22 PM, Zhang, Edward (GDI Hadoop) <
yonzh...@ebay.com> wrote:

> I have fixed
> org.apache.eagle.alert.state.TestSiddhiStateSnapshotAndRestore.persistenceT
> est7, will create pull request.
>
> For https://github.com/apache/incubator-eagle/pull/43, can you look at why
> it is not passing the checks. We need merge that pull request 43 soon.
>
> Thanks
> edward
>
> On 1/11/16, 19:05, "Michael Wu"  wrote:
>
> >Hi all,
> >
> >I've setup a CI job for building and running unit test against ealge's
> >master branch, the job is at:
> >https://builds.apache.org/job/incubator-eagle-main/, with unit test
> >failing
> >all the time. I think it's the time for us to reach out and fix the unit
> >tests.
> >
> >Each time the job fails, it's configured to send out an email to all of us
> >with subject starting with "[EAGLE BUILD FAILED]", please take a look when
> >you receive such emails to make sure if there is any failing case of your
> >own, or you can define a rule/label to classify them.
> >
> >For every CI build, there is a testing result page of it, on which we can
> >see overall status of the build.
> >
> >To get to the test result page, we can either click into a specific build
> >then click "Test Result" in the left-nav, or use below url template with
> >replacing build-number and then open it in browsers:
> >
> https://builds.apache.org/job/incubator-eagle-main/${build-number}/testRep
> >ort/
> >.
> >
> >E.g. the 7th build of the job, holds test result at
> >https://builds.apache.org/job/incubator-eagle-main/7/testReport/
> >
> >Currently, I summed the failing test case as below, please anyone who is
> >the owner try to fix them:
> >org.apache.eagle.alert.cep.TestSiddhiEvaluator.test
> ><
> https://builds.apache.org/job/incubator-eagle-main/7/testReport/org.apach
> >e.eagle.alert.cep/TestSiddhiEvaluator/test/>
> >org.apache.eagle.alert.dao.TestSiddhiStreamMetadataUtils.test
> ><
> https://builds.apache.org/job/incubator-eagle-main/7/testReport/org.apach
> >e.eagle.alert.dao/TestSiddhiStreamMetadataUtils/test/>
> >org.apache.eagle.alert.dao.TestStreamDefinitionDAOImpl.test
> ><
> https://builds.apache.org/job/incubator-eagle-main/7/testReport/org.apach
> >e.eagle.alert.dao/TestStreamDefinitionDAOImpl/test/>
> >org.apache.eagle.alert.state.TestSiddhiStateSnapshotAndRestore.persistence
> >Test7
> ><
> https://builds.apache.org/job/incubator-eagle-main/7/testReport/org.apach
> >e.eagle.alert.state/TestSiddhiStateSnapshotAndRestore/persistenceTest7/>
> >
> >Thanks.
> >Michael
>
>

[jira] [Resolved] (EAGLE-47) Ability to audit who made changes to Eagle policies

[Senthilkumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Senthilkumar resolved EAGLE-47.
---
Resolution: Fixed

Reviewer : Edward
Pull Request Status : Closed

Github user asfgit closed the pull request at:
https://github.com/apache/incubator-eagle/pull/39



> Ability to audit who made changes to Eagle policies
> ---
>
> Key: EAGLE-47
> URL: https://issues.apache.org/jira/browse/EAGLE-47
> Project: Eagle
>  Issue Type: New Feature
>Affects Versions: 0.3.0
>Reporter: Arun Manoharan
>Assignee: Senthilkumar
> Fix For: 0.3.0
>
> Attachments: EAGLE-47-v1-24122015.patch, eagle_47_design.docx, 
> eagle_47_generic_api_design.docx
>
>
> As an admin of Apache Eagle I would like to know who created policies, 
> modified policies, Deleted policies.
> Also who added source, site etc.
> This information is a must have for security related products to have an 
> audit trail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-104) Fix the unit test TestHDFSSecuritylogParser

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093339#comment-15093339
 ] 

ASF GitHub Bot commented on EAGLE-104:
--

Github user yonzhang commented on the pull request:

https://github.com/apache/incubator-eagle/pull/43#issuecomment-170804388
  
I run this unit test in my local and observed the following errors

---
Test set: org.apache.eagle.security.TestHDFSSecuritylogParser

---
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.404 sec 
<<< FAILURE!
testSimpleAuth(org.apache.eagle.security.TestHDFSSecuritylogParser)  Time 
elapsed: 0.002 sec  <<< ERROR!
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1911)
at 
org.apache.eagle.security.util.LogParseUtil.parseUserFromUGI(LogParseUtil.java:39)
at 
org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parseAttr(HDFSSecurityLogParser.java:78)
at 
org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parse(HDFSSecurityLogParser.java:52)
at 
org.apache.eagle.security.TestHDFSSecuritylogParser.verifyParserAttributes(TestHDFSSecuritylogParser.java:63)
at 
org.apache.eagle.security.TestHDFSSecuritylogParser.testSimpleAuth(TestHDFSSecuritylogParser.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at 
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120)
at 
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103)


> Fix the unit test TestHDFSSecuritylogParser  
> -
>
> Key: EAGLE-104
> URL: https://issues.apache.org/jira/browse/EAGLE-104
> Project: Eagle
>  Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Prasad Mujumdar
> Attachments: EAGLE-104.1.patch
>
>
> The test fails with following error
> {noformat}
> java.lang.StringIndexOutOfBoundsException">
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at
> java.lang.String.substring(String.java:1911) at
> org.apache.eagle.security.util.LogParseUtil.parseUserFromUGI(LogParseUtil.java:39)
> at
> org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parseAttr(HDFSSecurityLogParser.java:78)
> at
> org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parse(HDFSSecurityLogParser.java:52)
> at
> org.apache.eagle.security.TestHDFSSecuritylogParser.test(TestHDFSSecuritylogParser.java:36)
> {noformat}
> Looks like the problem is that the UGI parsing method is not handling the 
> simple authentication scenario where user name is not a kerberos principal. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: https://issues.apache.org/jira/brows...

[yonzhang]

Github user yonzhang commented on the pull request:

https://github.com/apache/incubator-eagle/pull/43#issuecomment-170804388
  
I run this unit test in my local and observed the following errors

---
Test set: org.apache.eagle.security.TestHDFSSecuritylogParser

---
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.404 sec 
<<< FAILURE!
testSimpleAuth(org.apache.eagle.security.TestHDFSSecuritylogParser)  Time 
elapsed: 0.002 sec  <<< ERROR!
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1911)
at 
org.apache.eagle.security.util.LogParseUtil.parseUserFromUGI(LogParseUtil.java:39)
at 
org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parseAttr(HDFSSecurityLogParser.java:78)
at 
org.apache.eagle.security.securitylog.parse.HDFSSecurityLogParser.parse(HDFSSecurityLogParser.java:52)
at 
org.apache.eagle.security.TestHDFSSecuritylogParser.verifyParserAttributes(TestHDFSSecuritylogParser.java:63)
at 
org.apache.eagle.security.TestHDFSSecuritylogParser.testSimpleAuth(TestHDFSSecuritylogParser.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at 
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120)
at 
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: EAGLE-119 fix unit test from siddhi

[yonzhang]

GitHub user yonzhang opened a pull request:

https://github.com/apache/incubator-eagle/pull/58

EAGLE-119 fix unit test from siddhi



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yonzhang/incubator-eagle EAGLE-119

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/58.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #58


commit 1935c9178ff005332efbe5f30faf9ce10cdf6a64
Author: yonzhang 
Date:   2016-01-12T04:53:28Z

fix unit test from siddhi




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-119) fix unit testing bugs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093310#comment-15093310
 ] 

ASF GitHub Bot commented on EAGLE-119:
--

GitHub user yonzhang opened a pull request:

https://github.com/apache/incubator-eagle/pull/58

EAGLE-119 fix unit test from siddhi



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yonzhang/incubator-eagle EAGLE-119

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/58.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #58


commit 1935c9178ff005332efbe5f30faf9ce10cdf6a64
Author: yonzhang 
Date:   2016-01-12T04:53:28Z

fix unit test from siddhi




> fix unit testing bugs
> -
>
> Key: EAGLE-119
> URL: https://issues.apache.org/jira/browse/EAGLE-119
> Project: Eagle
>  Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Edward Zhang
>Assignee: Edward Zhang
> Fix For: 0.3.0
>
>
> Need fix unit test bugs as soon as possible. Most of them are with siddhi 
> window based unit test
> Running org.apache.eagle.alert.dao.TestAlertDefinitionDAOImpl
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> Running org.apache.eagle.alert.cep.TestSiddhiEvaluator
> 2016-01-11 05:20:51,317 INFO [main] siddhi.StreamMetadataManager[91]: Already 
> initialized, skip
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.165 sec <<< 
> FAILURE!
> Results :
> Failed tests: 
>   
> persistenceTest7(org.apache.eagle.alert.state.TestSiddhiStateSnapshotAndRestore)
> Tests in error: 
>   test(org.apache.eagle.alert.dao.TestStreamDefinitionDAOImpl)
>   test(org.apache.eagle.alert.dao.TestSiddhiStreamMetadataUtils)
>   test(org.apache.eagle.alert.cep.TestSiddhiEvaluator)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle pull request: fix unit test from siddhi

[yonzhang]

Github user yonzhang commented on the pull request:

https://github.com/apache/incubator-eagle/pull/57#issuecomment-170798296
  
need normalized title


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: fix unit test from siddhi

[yonzhang]

GitHub user yonzhang opened a pull request:

https://github.com/apache/incubator-eagle/pull/57

fix unit test from siddhi



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yonzhang/incubator-eagle EAGLE-119

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/57.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #57


commit 1935c9178ff005332efbe5f30faf9ce10cdf6a64
Author: yonzhang 
Date:   2016-01-12T04:53:28Z

fix unit test from siddhi




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: fix unit test from siddhi

[yonzhang]

Github user yonzhang closed the pull request at:

https://github.com/apache/incubator-eagle/pull/57


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: Eagle 119 fixunittest

[yonzhang]

Github user yonzhang closed the pull request at:

https://github.com/apache/incubator-eagle/pull/56


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: Eagle 119 fixunittest

[yonzhang]

Github user yonzhang commented on the pull request:

https://github.com/apache/incubator-eagle/pull/56#issuecomment-170786490
  
only need include unit test fix


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request: Eagle 119 fixunittest

[yonzhang]

GitHub user yonzhang opened a pull request:

https://github.com/apache/incubator-eagle/pull/56

Eagle 119 fixunittest

Siddhi does not support external time based snapshot yet, need cancel this 
unit test

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yonzhang/incubator-eagle EAGLE-119_fixunittest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/56.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #56


commit 95cd0eb381e7ecf219deb32b9baf7da009dcdcc2
Author: yonzhang 
Date:   2015-12-22T22:57:47Z

test siddhi aggregation and snapshot and restore

commit 675d680dedc8d447c1b69d37673dad1d63f06bfa
Author: yonzhang 
Date:   2016-01-06T18:45:32Z

Merge branch 'master' of https://github.com/yonzhang/incubator-eagle

commit c671a0cbf5d7ff118899898ef0e508b096059095
Author: yonzhang 
Date:   2016-01-12T01:13:07Z

Merge remote-tracking branch 'upstream/master'

commit cdd4f2d8708be8010b3ebf35532b982b4ad65966
Author: yonzhang 
Date:   2016-01-12T04:27:17Z

fix unit test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Fix Eagle Unit Test] please fix eagle's unit test

[Zhang, Edward (GDI Hadoop)]

I have fixed 
org.apache.eagle.alert.state.TestSiddhiStateSnapshotAndRestore.persistenceT
est7, will create pull request.

For https://github.com/apache/incubator-eagle/pull/43, can you look at why
it is not passing the checks. We need merge that pull request 43 soon.

Thanks
edward

On 1/11/16, 19:05, "Michael Wu"  wrote:

>Hi all,
>
>I've setup a CI job for building and running unit test against ealge's
>master branch, the job is at:
>https://builds.apache.org/job/incubator-eagle-main/, with unit test
>failing
>all the time. I think it's the time for us to reach out and fix the unit
>tests.
>
>Each time the job fails, it's configured to send out an email to all of us
>with subject starting with "[EAGLE BUILD FAILED]", please take a look when
>you receive such emails to make sure if there is any failing case of your
>own, or you can define a rule/label to classify them.
>
>For every CI build, there is a testing result page of it, on which we can
>see overall status of the build.
>
>To get to the test result page, we can either click into a specific build
>then click "Test Result" in the left-nav, or use below url template with
>replacing build-number and then open it in browsers:
>https://builds.apache.org/job/incubator-eagle-main/${build-number}/testRep
>ort/
>.
>
>E.g. the 7th build of the job, holds test result at
>https://builds.apache.org/job/incubator-eagle-main/7/testReport/
>
>Currently, I summed the failing test case as below, please anyone who is
>the owner try to fix them:
>org.apache.eagle.alert.cep.TestSiddhiEvaluator.test
>e.eagle.alert.cep/TestSiddhiEvaluator/test/>
>org.apache.eagle.alert.dao.TestSiddhiStreamMetadataUtils.test
>e.eagle.alert.dao/TestSiddhiStreamMetadataUtils/test/>
>org.apache.eagle.alert.dao.TestStreamDefinitionDAOImpl.test
>e.eagle.alert.dao/TestStreamDefinitionDAOImpl/test/>
>org.apache.eagle.alert.state.TestSiddhiStateSnapshotAndRestore.persistence
>Test7
>e.eagle.alert.state/TestSiddhiStateSnapshotAndRestore/persistenceTest7/>
>
>Thanks.
>Michael

suggestion: add field "threshold" to current "fileSensitivity structure" in Eagle

[Daniel Zhou]

Hi all,

Recently I am working on a project to automatically fetch the metadata of 
sensitive info stored in DB and then create eagle policy. I am wondering if we 
can add a field called "threshold" to current "fileSensitivity structure" in 
eagle so that we can create a policy with more details.

Our company's product "DgSecure" can discover all the sensitive elements within 
every file in hadoop  automatically,  so we have many details of these 
sensitive information. With these information, we can make the policy more 
precisely.  For example, I want to create a policy based on two parameters, one 
is  "sensitivity type", the other is called "threshold".
Only when the total number of that particular sensitive type element  reaches 
or exceeds "threshold" can the alerts be triggered.

So the trigger condition could be something like this:
 if (sensitiveType == "MailAddress" && NumberOfSensData >=threshodl) 
.

I think this condition makes more sense than just tagging a file with a 
sensitive type.

Please let me know if you have any opinions or suggestions. :)

Thanks!
Daniel

[EAGLE BUILD FAILED] incubator-eagle-main failed on build #8

[Apache Jenkins Server]

The Apache Jenkins build system has built incubator-eagle-main (build #8) 
Status: Still Failing Check console output at 
https://builds.apache.org/job/incubator-eagle-main/8/ to view the results. 
Test Result is at: Test Result

[jira] [Commented] (EAGLE-119) fix unit testing bugs

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093207#comment-15093207
 ] 

Prasad Mujumdar commented on EAGLE-119:
---

There's also EAGLE-104 for TestHDFSSecuritylogParser failures. I don't see that 
failure in the pre-commit runs anymore, not sure if it's disabled.

> fix unit testing bugs
> -
>
> Key: EAGLE-119
> URL: https://issues.apache.org/jira/browse/EAGLE-119
> Project: Eagle
>  Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Edward Zhang
>Assignee: Edward Zhang
> Fix For: 0.3.0
>
>
> Need fix unit test bugs as soon as possible. Most of them are with siddhi 
> window based unit test
> Running org.apache.eagle.alert.dao.TestAlertDefinitionDAOImpl
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
> Running org.apache.eagle.alert.cep.TestSiddhiEvaluator
> 2016-01-11 05:20:51,317 INFO [main] siddhi.StreamMetadataManager[91]: Already 
> initialized, skip
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.165 sec <<< 
> FAILURE!
> Results :
> Failed tests: 
>   
> persistenceTest7(org.apache.eagle.alert.state.TestSiddhiStateSnapshotAndRestore)
> Tests in error: 
>   test(org.apache.eagle.alert.dao.TestStreamDefinitionDAOImpl)
>   test(org.apache.eagle.alert.dao.TestSiddhiStreamMetadataUtils)
>   test(org.apache.eagle.alert.cep.TestSiddhiEvaluator)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Fix Eagle Unit Test] please fix eagle's unit test

[Michael Wu]

Hi all,

I've setup a CI job for building and running unit test against ealge's
master branch, the job is at:
https://builds.apache.org/job/incubator-eagle-main/, with unit test failing
all the time. I think it's the time for us to reach out and fix the unit
tests.

Each time the job fails, it's configured to send out an email to all of us
with subject starting with "[EAGLE BUILD FAILED]", please take a look when
you receive such emails to make sure if there is any failing case of your
own, or you can define a rule/label to classify them.

For every CI build, there is a testing result page of it, on which we can
see overall status of the build.

To get to the test result page, we can either click into a specific build
then click "Test Result" in the left-nav, or use below url template with
replacing build-number and then open it in browsers:
https://builds.apache.org/job/incubator-eagle-main/${build-number}/testReport/
.

E.g. the 7th build of the job, holds test result at
https://builds.apache.org/job/incubator-eagle-main/7/testReport/

Currently, I summed the failing test case as below, please anyone who is
the owner try to fix them:
org.apache.eagle.alert.cep.TestSiddhiEvaluator.test

org.apache.eagle.alert.dao.TestSiddhiStreamMetadataUtils.test

org.apache.eagle.alert.dao.TestStreamDefinitionDAOImpl.test

org.apache.eagle.alert.state.TestSiddhiStateSnapshotAndRestore.persistenceTest7


Thanks.
Michael

[jira] [Created] (EAGLE-119) fix unit testing bugs

[Edward Zhang (JIRA)]

Edward Zhang created EAGLE-119:
--

 Summary: fix unit testing bugs
 Key: EAGLE-119
 URL: https://issues.apache.org/jira/browse/EAGLE-119
 Project: Eagle
  Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Edward Zhang
Assignee: Edward Zhang
 Fix For: 0.3.0


Need fix unit test bugs as soon as possible. Most of them are with siddhi 
window based unit test

Running org.apache.eagle.alert.dao.TestAlertDefinitionDAOImpl
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
Running org.apache.eagle.alert.cep.TestSiddhiEvaluator
2016-01-11 05:20:51,317 INFO [main] siddhi.StreamMetadataManager[91]: Already 
initialized, skip
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.165 sec <<< 
FAILURE!

Results :

Failed tests: 
  
persistenceTest7(org.apache.eagle.alert.state.TestSiddhiStateSnapshotAndRestore)

Tests in error: 
  test(org.apache.eagle.alert.dao.TestStreamDefinitionDAOImpl)
  test(org.apache.eagle.alert.dao.TestSiddhiStreamMetadataUtils)
  test(org.apache.eagle.alert.cep.TestSiddhiEvaluator)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-103) add comments to readme to tell users: currently, eagle is tested under jdk1.7.x, may have compile error with jdk1.8.x

[Edward Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092897#comment-15092897
 ] 

Edward Zhang commented on EAGLE-103:


[~haoch]could we have this information on 
http://eagle.incubator.apache.org/docs/quick-start.html and 
https://github.com/apache/incubator-eagle sooner, I saw some users report the 
same build issue. Thanks.

> add comments to readme to tell users: currently, eagle is tested under 
> jdk1.7.x, may have compile error with jdk1.8.x
> -
>
> Key: EAGLE-103
> URL: https://issues.apache.org/jira/browse/EAGLE-103
> Project: Eagle
>  Issue Type: Improvement
>Reporter: Michael Wu
>Assignee: Hao Chen
>Priority: Minor
>
> Compilation error found while building with JDK1.8.x (refer to 
> https://issues.apache.org/jira/browse/EAGLE-101).
> I think we should update the required version of jdk in readme file of the 
> project.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Proposal] Dynamical Topology Manager

[Edward Zhang]

Thanks for the valuable comments, you are right in the high-level
observations :-)

(I participated some offline discussion on this proposal) I think this
proposal is based on the requirements that Eagle not only monitors security
events from hadoop but also monitors security events from other data
source, for example cassandra and even more requirements are from hadoop
native metric monitoring. In last 2 months, when we want to onboard new
diverse datasources(for example mongo db metrics, hadoop native metrics
etc.), we find it is impossible for user(mostly operations team) to
understand storm topology or write code for very simple metrics
ingestion/alert rules. For monitoring perspective, user usually wants
metric/log onboarding is as simple as turn-key operation.

It is possible for Eagle developers to develop applications for each data
source, but it might be better for Eagle to support logs/metrics with some
general schema. User just needs some configurations to get new data source
flow into Eagle and create policy on-the-fly against the streaming data.
Maybe I am wrong, but Eagle looks is an application to Storm framework, but
Eagle would be a framework to monitoring applications e.g. security or
other data activity.

The point of Eagle to separate application and framework is very correct.
In Eagle source code, the application code and framework code are separated
from the beginning. For those features like policy restore after machine
fails, DSL, aggregation etc, I think Eagle team should look for
contributing back to Storm if people agree that is what stream framework
should have.

We also found that Eagle monitoring uses streaming framework but streaming
framework is not customized for monitoring. The gap between monitoring
platform and streaming framework has to be filled to make sure monitoring
is reliable. For example compared to real-time streaming analytics,
monitoring does not want to have any false alert or missing alert
especially for security event, which requires more processing semantics
than popular streaming framework provides. We will actively explore help
from streaming projects.

Thanks

Edward

On Mon, Jan 11, 2016 at 10:55 AM, Julian Hyde  wrote:

> Can I make a high-level observation? (And, although I’m a mentor of this
> project, I’m not speaking as a mentor, just someone who has built various
> database and streaming systems over the years.)
>
> I’ve noticed that Eagle is taking on several problems that —
> architecturally speaking — should be part of the underlying streaming
> system. This topology manager and also, a DSL declarative streaming
> queries, and making sure that streaming queries continue where they left
> off, even in the presence of failures of individual stream-processing nodes.
>
> These are very hard distributed systems problems, and they are horizontal
> problems that have nothing to do with Eagle’s problem domain (security). It
> would be analogous to an application that is selling concert tickets
> deciding to develop HBase as part of the application.
>
> If the Eagle community wants to solve these problems, that’s awesome, and
> you should go for it. Apache projects are great at pulling in people with
> diverse skills and when they gather momentum they can build some amazing
> technology. But I think Eagle should consider putting more architectural
> separation between your stream management technology and your application.
> You could do that by building separate modules (and testing them
> independently). Or you could contribute the functionality you need to the
> underlying system (e.g. Storm/Nimbus). Your project will run much more
> smoothly if you call out the hard problems you are trying to solve.
>
> And, as a side benefit, projects that have nothing to do with Hadoop
> security will be able to use (and test, and bug-fix) the technology you are
> developing.
>
> Julian
>
>
> > On Jan 11, 2016, at 8:57 AM, Hao Chen  wrote:
> >
> > Currently eagle is requiring user to manually manage topologies
> completely
> > independent of eagle components,  which is not very smooth for the user
> and
> > management experience end-to-end from on-boarding datasource, starting
> > topologies, defining policy and also monitoring policy and execution
> > status, so how do you think we manage everything in single place and
> > dynamically manage topology lifecycle like
> > starting/stopping/status/monitoring as well policy
> > creation/modification/monitoring all in eagle ui only? So that user don't
> > need to touch storm anymore except specifying where the nimbus is when
> > setting up eagle.
> >
> >
> > --
> >
> > Hao
>
>

Eagle with RDBMS as storage type

[Jaspaul Chahal]

Hello Guys,


My name is Jaspaul and am working as Senior Engineer at Dataguise Inc. We, 
here, are trying to get Eagle working with RDBMS as storage-type. So, we need 
help on the steps to be followed to get it working with Mysql.


I have the following questions regarding this:

  *   Is Eagle working with JDBC storage type already?

  *   What are the steps to be followed to setup storage type to JDBC?

 *   We know the following steps , so far:
*   modify eagle-service.conf file ,set storage type to JDBC and other 
JDBC parameters
*   Copy JDBC connector jar (MySql connector jar is already there) to 
~/eagle/lib/tomcat/webapps/eagle-service/WEB-INF/lib/
*   Copy eagle-storage-jdbc-0.3.0.jar to 
~/eagle/lib/tomcat/webapps/eagle-service/WEB-INF/lib/.

  *   How JDBC database and tables are created, where is the DDL file? How can 
we generate the DDL?


Thanks in Advance!


Regards,
Jaspaul Chahal

Re: [Proposal] Dynamical Topology Manager

[Julian Hyde]

Can I make a high-level observation? (And, although I’m a mentor of this 
project, I’m not speaking as a mentor, just someone who has built various 
database and streaming systems over the years.)

I’ve noticed that Eagle is taking on several problems that — architecturally 
speaking — should be part of the underlying streaming system. This topology 
manager and also, a DSL declarative streaming queries, and making sure that 
streaming queries continue where they left off, even in the presence of 
failures of individual stream-processing nodes.

These are very hard distributed systems problems, and they are horizontal 
problems that have nothing to do with Eagle’s problem domain (security). It 
would be analogous to an application that is selling concert tickets deciding 
to develop HBase as part of the application.

If the Eagle community wants to solve these problems, that’s awesome, and you 
should go for it. Apache projects are great at pulling in people with diverse 
skills and when they gather momentum they can build some amazing technology. 
But I think Eagle should consider putting more architectural separation between 
your stream management technology and your application. You could do that by 
building separate modules (and testing them independently). Or you could 
contribute the functionality you need to the underlying system (e.g. 
Storm/Nimbus). Your project will run much more smoothly if you call out the 
hard problems you are trying to solve.

And, as a side benefit, projects that have nothing to do with Hadoop security 
will be able to use (and test, and bug-fix) the technology you are developing.

Julian


> On Jan 11, 2016, at 8:57 AM, Hao Chen  wrote:
> 
> Currently eagle is requiring user to manually manage topologies completely
> independent of eagle components,  which is not very smooth for the user and
> management experience end-to-end from on-boarding datasource, starting
> topologies, defining policy and also monitoring policy and execution
> status, so how do you think we manage everything in single place and
> dynamically manage topology lifecycle like
> starting/stopping/status/monitoring as well policy
> creation/modification/monitoring all in eagle ui only? So that user don't
> need to touch storm anymore except specifying where the nimbus is when
> setting up eagle.
> 
> 
> --
> 
> Hao

[Proposal] Dynamical Topology Manager

[Hao Chen]

Currently eagle is requiring user to manually manage topologies completely
independent of eagle components,  which is not very smooth for the user and
management experience end-to-end from on-boarding datasource, starting
topologies, defining policy and also monitoring policy and execution
status, so how do you think we manage everything in single place and
dynamically manage topology lifecycle like
starting/stopping/status/monitoring as well policy
creation/modification/monitoring all in eagle ui only? So that user don't
need to touch storm anymore except specifying where the nimbus is when
setting up eagle.


--

Hao

[jira] [Created] (EAGLE-118) Siddhi contains is neither a function extension nor an aggregated attribute extension

[Hao Chen (JIRA)]

Hao Chen created EAGLE-118:
--

 Summary: Siddhi contains is neither a function extension nor an 
aggregated attribute extension 
 Key: EAGLE-118
 URL: https://issues.apache.org/jira/browse/EAGLE-118
 Project: Eagle
  Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Hao Chen
Assignee: Edward Zhang
 Fix For: 0.3.0



HdfsUserCommandReassembler failed to initialize with exception: 
**org.wso2.siddhi.core.exception.ExecutionPlanCreationException: contains is 
neither a function extension nor an aggregated attribute extension**

{code}
org.wso2.siddhi.query.api.exception.ExecutionPlanValidationException: contains 
is neither a function extension nor an aggregated attribute extension, when 
creating query copyFromLocal in execution plan 
"b38f8f76-c2cf-4288-a26a-cb2a30161b42"
at 
org.wso2.siddhi.core.util.parser.ExecutionPlanParser.parse(ExecutionPlanParser.java:127)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.SiddhiManager.createExecutionPlanRuntime(SiddhiManager.java:51)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.SiddhiManager.createExecutionPlanRuntime(SiddhiManager.java:59)
 ~[stormjar.jar:na]
at 
org.apache.eagle.security.auditlog.HdfsUserCommandReassembler.init(HdfsUserCommandReassembler.java:130)
 ~[stormjar.jar:na]
at 
org.apache.eagle.datastream.storm.JavaStormBoltWrapper.prepare(JavaStormBoltWrapper.scala:36)
 ~[stormjar.jar:na]
at 
backtype.storm.daemon.executor$fn__5017$fn__5030.invoke(executor.clj:732) 
~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:463) 
~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: org.wso2.siddhi.core.exception.ExecutionPlanCreationException: 
contains is neither a function extension nor an aggregated attribute extension, 
when creating query copyFromLocal
at 
org.wso2.siddhi.core.util.parser.QueryParser.parse(QueryParser.java:109) 
~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.ExecutionPlanParser.parse(ExecutionPlanParser.java:117)
 ~[stormjar.jar:na]
... 8 common frames omitted
Caused by: org.wso2.siddhi.core.exception.ExecutionPlanCreationException: 
contains is neither a function extension nor an aggregated attribute extension
at 
org.wso2.siddhi.core.util.parser.ExpressionParser.parseExpression(ExpressionParser.java:244)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.ExpressionParser.parseExpression(ExpressionParser.java:103)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.SingleInputStreamParser.generateProcessor(SingleInputStreamParser.java:129)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.SingleInputStreamParser.parseInputStream(SingleInputStreamParser.java:86)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.StateInputStreamParser.parse(StateInputStreamParser.java:92)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.StateInputStreamParser.parse(StateInputStreamParser.java:134)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.StateInputStreamParser.parse(StateInputStreamParser.java:134)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.StateInputStreamParser.parse(StateInputStreamParser.java:134)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.StateInputStreamParser.parse(StateInputStreamParser.java:134)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.StateInputStreamParser.parse(StateInputStreamParser.java:171)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.StateInputStreamParser.parseInputStream(StateInputStreamParser.java:72)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.InputStreamParser.parse(InputStreamParser.java:57)
 ~[stormjar.jar:na]
at 
org.wso2.siddhi.core.util.parser.QueryParser.parse(QueryParser.java:64) 
~[stormjar.jar:na]
... 9 common frames omitted
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-81) Notification Plugin Framework

[Senthilkumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-81?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15091763#comment-15091763
 ] 

Senthilkumar commented on EAGLE-81:
---

Email Notification API ready .. Setting up IntelliJ IDE to debug the changes.. 
Eclipse not detecting Scala even if we add Scala Plugin.

> Notification Plugin Framework
> -
>
> Key: EAGLE-81
> URL: https://issues.apache.org/jira/browse/EAGLE-81
> Project: Eagle
>  Issue Type: New Feature
>Reporter: Senthilkumar
>Assignee: Senthilkumar
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Today notification is only allowed for email .. We need more notification 
> methods to send alert to external system . 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)