[jira] [Closed] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Su Ralph closed EAGLE-85. - > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Su Ralph resolved EAGLE-85. --- Resolution: Fixed > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Provide analytic DSL support
We probably also need support declarative persist along with this feature,
I think.
Is UI design started?
Thanks
Edward
On 12/14/15, 23:34, "Liangfei.Su" wrote:
>q1. Yes, same mechanism like policy definition would be used. User would
>be
>able to define an analyze-policy, and when analyze executor would try to
>load and execute the policy. More, in programming, user could simply input
>their analyze sql directly through API for simplicity.
>
>q2. Sure, user would be able to define simple analyze.
>
>q3. No, but there are dependency since Hao's work impact the API a lot.
>pull 26 would quickly decouple this dependency.
>
>Ralph
>
>
>On Mon, Dec 14, 2015 at 12:44 PM, Zhang, Edward (GDI Hadoop) <
>yonzh...@ebay.com> wrote:
>
>> Thanks for updating.
>> Some questions:
>> 1. do we need aggregator declaration to downloaded from eagle service?
>>(I
>> believe it can be used in code directly) If that is true, can we use the
>> same mechanism for policy lifecycle management? and do we want this
>> declaration can be updated dynamically?
>> 2. because aggregator declaration can be expressed with limited syntax,
>> group by/max/top/avg/Š, is that possible future UI part can be more
>> intuitive than current policy UI? :-)
>> 3. How this design is aligned to general purpose monitoring design which
>> Hao/Chen is working on. I mean in terms of input/output and business
>> logic, will that be reused in the future?
>>
>> Those questions are not urgent request, but we can think of that while
>> implementing.
>>
>> Thanks
>> Edward
>>
>>
>> On 12/13/15, 18:58, "Liangfei.Su" wrote:
>>
>> >Had a draft spec at
>> >https://cwiki.apache.org/confluence/display/EAG/Stream+Analyze
>> >
>> >Please suggest.
>> >
>> >
>> >Thanks,
>> >Ralph
>> >
>> >
>> >On Mon, Dec 7, 2015 at 6:00 PM, Liangfei.Su
>>wrote:
>> >
>> >> For #1, the eagle programming API is mostly sit at the same place of
>> >> Trident. Besides the platform independence and type safe, the eagle
>>CEP
>> >> could be used to help reduce the code effort to submit a topology.
>>This
>> >> extend the current alerting define experience to more wise cases.
>> >>
>> >> Like
>> >> trident style of join
>> >>
>> >> topology.join(stream1, new Fields("key"), stream2, new Fields("x"),
>>new
>> >>Fields("key", "a", "b", "c"));
>> >>
>> >>
>> >> to sql like
>> >> from stream1=.., stream2=...
>> >> select stream1.key, stream2.a, stream2.b, stream3.c where
>> >> stream1.key=stream2.x
>> >>
>> >> from windowed join, things could be more complicated, and trident
>> >>require
>> >> user to do a couple of persiste/stateQuery by their code.
>> >>
>> >> Thanks,
>> >> Ralph
>> >>
>> >> On Mon, Dec 7, 2015 at 5:16 PM, Chen, Hao wrote:
>> >>
>> >>> 1. Are you guys re-implement part of Trident?
>> >>> >> 1) Trident is high-level API but field-based, eagle is
>> >>>type-oritended.
>> >>> >> 2) Eagle datastream is platform-indepent, not only on storm
>> >>> >> 3) Eagle datastream support CEP CQL except for programming API.
>> >>>
>> >>> 2. How can the type information kept during the data processing by
>> >>>Storm?
>> >>> >> Type information is provided by Scala TypeTag[T]
>> >>> >> Eagle could serialize valuable type information like type class,
>> >>>type
>> >>> fields and so on from TypeTag[T] before submitting to execution
>> >>>environment
>> >>> and then shared between processing element like spout/bolt.
>> >>>
>> >>>
>> >>> Thanks,
>> >>> Hao
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On 12/7/15, 5:02 PM, "Meng, Yiming" wrote:
>> >>>
>> >>> >
>> >>> >Quick questions:
>> >>> >
>> >>> >1. Are you guys re-implement part of Trident?
>> >>> >2. How can the type information kept during the data processing by
>> >>>Storm?
>> >>> >
>> >>> >Regards,
>> >>> >Yiming Meng
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >On 12/7/15, 4:58 PM, "Chen, Hao" wrote:
>> >>> >
>> >>> >>It¹s very good point and I¹m refactoring
>> >>> https://issues.apache.org/jira/browse/EAGLE-66, after the work, we
>> >>>could
>> >>> start with Analytics DSL using siddhi.
>> >>> >>
>> >>> >>After we finished:
>> >>> >>1. Typesafe DSL: EAGLE-66
>> >>> >>2. SQL CEP (siddhi): EAGLE-79
>> >>> >>3. DAG visualization/status/metric/dashboard
>> >>> >>
>> >>> >>We could even propose eagle-datastream as a general streaming
>> >>>framework
>> >>> independently for any streaming cases like real-time ETL.
>> >>> >>
>> >>> >>Thanks,
>> >>> >>Hao
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>On 12/7/15, 4:50 PM, "Liangfei.Su" wrote:
>> >>> >>
>> >>> >>>Eagle input comes with stream data (like security audit log),
>>eagle
>> >>> provide
>> >>> >>>alerting computation based on CEP DSL.
>> >>> >>>
>> >>> >>>Similar to this process, eagle should be able to provide same DSL
>> >>> support
>> >>> >>>to expose the real-time monitoring feature, and furthermore
>>could be
>> >>> >>>integrated with some storage backend (or as another streaming
>> >>>output)
>> >>> to
>> >>> >>>provide dashboard/presentation to
[GitHub] incubator-eagle pull request: [EAGLE-76] hide stream operation if ...
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/31#issuecomment-164676301 LGTM. 1. Please rename title as `EAGLE-76 hide stream operation if role is not admin` 2. Commit with following message EAGLE-76 hide stream operation if role is not admin https://issues.apache.org/jira/browse/EAGLE-76 Author: @zombieJ Reviewer: @haoch Closes #31. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-76) Disable creating newStream for common user role
[ https://issues.apache.org/jira/browse/EAGLE-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057520#comment-15057520 ] ASF GitHub Bot commented on EAGLE-76: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/31#issuecomment-164676301 LGTM. 1. Please rename title as `EAGLE-76 hide stream operation if role is not admin` 2. Commit with following message EAGLE-76 hide stream operation if role is not admin https://issues.apache.org/jira/browse/EAGLE-76 Author: @zombieJ Reviewer: @haoch Closes #31. > Disable creating newStream for common user role > --- > > Key: EAGLE-76 > URL: https://issues.apache.org/jira/browse/EAGLE-76 > Project: Eagle > Issue Type: Bug >Reporter: Zhao, Qingwen >Assignee: Jilin, Jiang >Priority: Minor > > for a regular user, he/she should not have a right to create a new stream. We > should hide this function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [Discuss] Hadoop metrics,job,GC monitoring
started some documentation on https://cwiki.apache.org/confluence/display/EAG/Hadoop+Native+Metrics+Monit oring Thanks Hao, Ralph etc. for offline review and suggestions, I would improve that. In terms of the question ³if user adds a new metric to monitor, how processing layer would change accordingly² I think if user adds a new metric, this metric should be added into metadata table, and data source layer and processing layer should see consistent list of metrics. But we still need bake this design, please comment whatever is your thoughts. Thanks Edward On 12/14/15, 11:04, "Arun Manoharan" wrote: >Thanks Edward for starting the thread. I think it is important to have the >job monitoring (MR/Spark) workloads for performance of the cluster and >availability. > >But it will be beneficial to have an extensible framework where users can >create business rules like "I want an alert when NN is in safemode or RM >is >flipping etc". > >Thanks, >Arun > >On Mon, Dec 14, 2015 at 10:58 AM, Zhang, Edward (GDI Hadoop) < >yonzh...@ebay.com> wrote: > >> Hi Eagle devs/users, >> >> As proposed in apache eagle incubator proposal, Eagle will start >> design/dev to support Hadoop system monitoring besides security >>monitoring >> which includes Hadoop native metrics, job, gclog etc. >> >> The community is also interested in Hadoop system monitoring by Eagle >>when >> we recently talked about Eagle product in public conferences, meet up >>etc. >> >> Take Hadoop native metrics as an example, first of all those metrics are >> pretty valuable in determining system health status, secondly collecting >> huge amount metrics, visualizing, and alerting is very challenging. We >> need think of declarative collection, dynamic aggregation, metric >>storage, >> metric query engine etc. >> >> Besides technical design, comprehensive policy/rule are also valuable to >> be shared in the community. Those policy/rule represent best practice in >> the world to manage large Hadoop clusters. >> >> Please suggest whatever is for engineering design or business >>policy/rules. >> >> Thanks >> Edward >> >>
[jira] [Commented] (EAGLE-78) eagle-lib.sh script not work for mac osx and windows
[ https://issues.apache.org/jira/browse/EAGLE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057513#comment-15057513 ] Zhaokun Qin commented on EAGLE-78: -- I have test in my laptop of window environment. And There are always some problems in installation, I think it the memory limited. I think device with higher memory size can run well? > eagle-lib.sh script not work for mac osx and windows > > > Key: EAGLE-78 > URL: https://issues.apache.org/jira/browse/EAGLE-78 > Project: Eagle > Issue Type: Sub-task >Affects Versions: 0.3.0 > Environment: mac osx, windows >Reporter: Hao Chen >Assignee: Hao Chen > Fix For: 0.3.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Provide analytic DSL support
q1. Yes, same mechanism like policy definition would be used. User would be
able to define an analyze-policy, and when analyze executor would try to
load and execute the policy. More, in programming, user could simply input
their analyze sql directly through API for simplicity.
q2. Sure, user would be able to define simple analyze.
q3. No, but there are dependency since Hao's work impact the API a lot.
pull 26 would quickly decouple this dependency.
Ralph
On Mon, Dec 14, 2015 at 12:44 PM, Zhang, Edward (GDI Hadoop) <
yonzh...@ebay.com> wrote:
> Thanks for updating.
> Some questions:
> 1. do we need aggregator declaration to downloaded from eagle service? (I
> believe it can be used in code directly) If that is true, can we use the
> same mechanism for policy lifecycle management? and do we want this
> declaration can be updated dynamically?
> 2. because aggregator declaration can be expressed with limited syntax,
> group by/max/top/avg/Š, is that possible future UI part can be more
> intuitive than current policy UI? :-)
> 3. How this design is aligned to general purpose monitoring design which
> Hao/Chen is working on. I mean in terms of input/output and business
> logic, will that be reused in the future?
>
> Those questions are not urgent request, but we can think of that while
> implementing.
>
> Thanks
> Edward
>
>
> On 12/13/15, 18:58, "Liangfei.Su" wrote:
>
> >Had a draft spec at
> >https://cwiki.apache.org/confluence/display/EAG/Stream+Analyze
> >
> >Please suggest.
> >
> >
> >Thanks,
> >Ralph
> >
> >
> >On Mon, Dec 7, 2015 at 6:00 PM, Liangfei.Su wrote:
> >
> >> For #1, the eagle programming API is mostly sit at the same place of
> >> Trident. Besides the platform independence and type safe, the eagle CEP
> >> could be used to help reduce the code effort to submit a topology. This
> >> extend the current alerting define experience to more wise cases.
> >>
> >> Like
> >> trident style of join
> >>
> >> topology.join(stream1, new Fields("key"), stream2, new Fields("x"), new
> >>Fields("key", "a", "b", "c"));
> >>
> >>
> >> to sql like
> >> from stream1=.., stream2=...
> >> select stream1.key, stream2.a, stream2.b, stream3.c where
> >> stream1.key=stream2.x
> >>
> >> from windowed join, things could be more complicated, and trident
> >>require
> >> user to do a couple of persiste/stateQuery by their code.
> >>
> >> Thanks,
> >> Ralph
> >>
> >> On Mon, Dec 7, 2015 at 5:16 PM, Chen, Hao wrote:
> >>
> >>> 1. Are you guys re-implement part of Trident?
> >>> >> 1) Trident is high-level API but field-based, eagle is
> >>>type-oritended.
> >>> >> 2) Eagle datastream is platform-indepent, not only on storm
> >>> >> 3) Eagle datastream support CEP CQL except for programming API.
> >>>
> >>> 2. How can the type information kept during the data processing by
> >>>Storm?
> >>> >> Type information is provided by Scala TypeTag[T]
> >>> >> Eagle could serialize valuable type information like type class,
> >>>type
> >>> fields and so on from TypeTag[T] before submitting to execution
> >>>environment
> >>> and then shared between processing element like spout/bolt.
> >>>
> >>>
> >>> Thanks,
> >>> Hao
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 12/7/15, 5:02 PM, "Meng, Yiming" wrote:
> >>>
> >>> >
> >>> >Quick questions:
> >>> >
> >>> >1. Are you guys re-implement part of Trident?
> >>> >2. How can the type information kept during the data processing by
> >>>Storm?
> >>> >
> >>> >Regards,
> >>> >Yiming Meng
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >On 12/7/15, 4:58 PM, "Chen, Hao" wrote:
> >>> >
> >>> >>It¹s very good point and I¹m refactoring
> >>> https://issues.apache.org/jira/browse/EAGLE-66, after the work, we
> >>>could
> >>> start with Analytics DSL using siddhi.
> >>> >>
> >>> >>After we finished:
> >>> >>1. Typesafe DSL: EAGLE-66
> >>> >>2. SQL CEP (siddhi): EAGLE-79
> >>> >>3. DAG visualization/status/metric/dashboard
> >>> >>
> >>> >>We could even propose eagle-datastream as a general streaming
> >>>framework
> >>> independently for any streaming cases like real-time ETL.
> >>> >>
> >>> >>Thanks,
> >>> >>Hao
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>On 12/7/15, 4:50 PM, "Liangfei.Su" wrote:
> >>> >>
> >>> >>>Eagle input comes with stream data (like security audit log), eagle
> >>> provide
> >>> >>>alerting computation based on CEP DSL.
> >>> >>>
> >>> >>>Similar to this process, eagle should be able to provide same DSL
> >>> support
> >>> >>>to expose the real-time monitoring feature, and furthermore could be
> >>> >>>integrated with some storage backend (or as another streaming
> >>>output)
> >>> to
> >>> >>>provide dashboard/presentation to user.
> >>> >>>
> >>> >>>This would require
> >>> >>>1. eagle programming API to support a new semantic of query(or
> >>> >>>aggregation), using the similar alert DSL.
> >>> >>>2. a clear definition of materialization interface, currently we
> >>>might
> >>> >>>start from the eagle built-in hbase storage implementation.
> >>> >>>3.
[jira] [Commented] (EAGLE-76) Disable creating newStream for common user role
[ https://issues.apache.org/jira/browse/EAGLE-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057497#comment-15057497 ] ASF GitHub Bot commented on EAGLE-76: - GitHub user zombieJ opened a pull request: https://github.com/apache/incubator-eagle/pull/31 [EAGLE-76] hide stream operation if role is not admin User don't have permission to add/update/delete metadata stream. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zombieJ/incubator-eagle userNoCreation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/31.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #31 commit e05d4f2b224c6b25e1deb19f7bddd5b88f5dbbed Author: jiljiang Date: 2015-12-15T07:26:04Z hide stream operation if role is not admin > Disable creating newStream for common user role > --- > > Key: EAGLE-76 > URL: https://issues.apache.org/jira/browse/EAGLE-76 > Project: Eagle > Issue Type: Bug >Reporter: Zhao, Qingwen >Assignee: Jilin, Jiang >Priority: Minor > > for a regular user, he/she should not have a right to create a new stream. We > should hide this function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: [EAGLE-76] hide stream operation if ...
GitHub user zombieJ opened a pull request: https://github.com/apache/incubator-eagle/pull/31 [EAGLE-76] hide stream operation if role is not admin User don't have permission to add/update/delete metadata stream. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zombieJ/incubator-eagle userNoCreation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/31.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #31 commit e05d4f2b224c6b25e1deb19f7bddd5b88f5dbbed Author: jiljiang Date: 2015-12-15T07:26:04Z hide stream operation if role is not admin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [jira] [Commented] (EAGLE-84) Create scripts for merging PRs automatically
Good, I am looking forward to the script. Please send pull request to eagle
repository.
--
Hao
On Tue, Dec 15, 2015 at 3:24 PM, Michael Wu wrote:
> Sure, Hao, i'll update the jira ticket once valuable steps are completed.
>
> On Tue, Dec 15, 2015 at 10:35 AM, Hao Chen (JIRA) wrote:
>
> >
> > [
> >
> https://issues.apache.org/jira/browse/EAGLE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057221#comment-15057221
> > ]
> >
> > Hao Chen commented on EAGLE-84:
> > ---
> >
> > And we have documented the contributing progress here:
> > https://cwiki.apache.org/confluence/display/EAG/Contributing+to+Eagle
> > please make sure keep consistent with it if no problem.
> >
> > > Create scripts for merging PRs automatically
> > >
> > >
> > > Key: EAGLE-84
> > > URL: https://issues.apache.org/jira/browse/EAGLE-84
> > > Project: Eagle
> > > Issue Type: Task
> > >Reporter: Michael Wu
> > >Assignee: Michael Wu
> > > Original Estimate: 168h
> > > Remaining Estimate: 168h
> > >
> > > While a pull request gets passed the testing steps, and is verified as
> > good-to-merge, people are willing to merge it with scripts to go through
> > necessary steps automatically, and can form the message of final
> aggregated
> > commit in a unified format. To achieve this, we are attempted to create
> the
> > script.
> > > As discussed, the format of message should be like below (ignore the
> > lines consisted of asterisks):
> > > **
> > > EAGLE-${ticket_number} ${ticket_title}
> > > ${ticket_url}
> > > Author: ${commiter}
> > > Reviewer: ${reviewer}
> > > Closes #${pr_number} from ${do_person}
> > > **
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>
Re: [jira] [Commented] (EAGLE-83) Integrate Jenkins with GitHub for checking pull requests
Cool, thanks. On Tue, Dec 15, 2015 at 3:25 PM, Michael Wu wrote: > Will fill udpates tomorrow. > > On Tue, Dec 15, 2015 at 10:36 AM, Hao Chen (JIRA) wrote: > > > > > [ > > > https://issues.apache.org/jira/browse/EAGLE-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057223#comment-15057223 > > ] > > > > Hao Chen commented on EAGLE-83: > > --- > > > > How about the progress for the ticket? > > > > > Integrate Jenkins with GitHub for checking pull requests > > > > > > > > > Key: EAGLE-83 > > > URL: https://issues.apache.org/jira/browse/EAGLE-83 > > > Project: Eagle > > > Issue Type: Task > > >Reporter: Michael Wu > > >Assignee: Michael Wu > > > Labels: test > > > Original Estimate: 168h > > > Remaining Estimate: 168h > > > > > > We expect that when a pull request comes up, a Jenkins job is triggered > > to verify the quality of new updates relevant to the PR, and > automatically > > provide advices for whether it should be applied, with the verification > > result posted to github PR page for reference. > > > > > > > > -- > > This message was sent by Atlassian JIRA > > (v6.3.4#6332) > > >
Re: [jira] [Commented] (EAGLE-83) Integrate Jenkins with GitHub for checking pull requests
Will fill udpates tomorrow. On Tue, Dec 15, 2015 at 10:36 AM, Hao Chen (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/EAGLE-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057223#comment-15057223 > ] > > Hao Chen commented on EAGLE-83: > --- > > How about the progress for the ticket? > > > Integrate Jenkins with GitHub for checking pull requests > > > > > > Key: EAGLE-83 > > URL: https://issues.apache.org/jira/browse/EAGLE-83 > > Project: Eagle > > Issue Type: Task > >Reporter: Michael Wu > >Assignee: Michael Wu > > Labels: test > > Original Estimate: 168h > > Remaining Estimate: 168h > > > > We expect that when a pull request comes up, a Jenkins job is triggered > to verify the quality of new updates relevant to the PR, and automatically > provide advices for whether it should be applied, with the verification > result posted to github PR page for reference. > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) >
Re: [jira] [Commented] (EAGLE-84) Create scripts for merging PRs automatically
Sure, Hao, i'll update the jira ticket once valuable steps are completed.
On Tue, Dec 15, 2015 at 10:35 AM, Hao Chen (JIRA) wrote:
>
> [
> https://issues.apache.org/jira/browse/EAGLE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057221#comment-15057221
> ]
>
> Hao Chen commented on EAGLE-84:
> ---
>
> And we have documented the contributing progress here:
> https://cwiki.apache.org/confluence/display/EAG/Contributing+to+Eagle
> please make sure keep consistent with it if no problem.
>
> > Create scripts for merging PRs automatically
> >
> >
> > Key: EAGLE-84
> > URL: https://issues.apache.org/jira/browse/EAGLE-84
> > Project: Eagle
> > Issue Type: Task
> >Reporter: Michael Wu
> >Assignee: Michael Wu
> > Original Estimate: 168h
> > Remaining Estimate: 168h
> >
> > While a pull request gets passed the testing steps, and is verified as
> good-to-merge, people are willing to merge it with scripts to go through
> necessary steps automatically, and can form the message of final aggregated
> commit in a unified format. To achieve this, we are attempted to create the
> script.
> > As discussed, the format of message should be like below (ignore the
> lines consisted of asterisks):
> > **
> > EAGLE-${ticket_number} ${ticket_title}
> > ${ticket_url}
> > Author: ${commiter}
> > Reviewer: ${reviewer}
> > Closes #${pr_number} from ${do_person}
> > **
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
[jira] [Updated] (EAGLE-88) Eagle web-service should have status url
[ https://issues.apache.org/jira/browse/EAGLE-88?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Su Ralph updated EAGLE-88: -- Issue Type: Improvement (was: Test) > Eagle web-service should have status url > > > Key: EAGLE-88 > URL: https://issues.apache.org/jira/browse/EAGLE-88 > Project: Eagle > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > Fix For: 0.3.0 > > > As a standalone service, eagle-service should have status url which expose > the system status. > This is the convention that for a service should make itself reachable and > monitor-able, and by have this status url service downgrade would be > possible. And this url is good fit of dropwizard metrics expose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhao, Qingwen updated EAGLE-74: --- Comment: was deleted (was: https://github.com/eaglemonitoring/eaglemonitoring.github.io/pull/38) > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (EAGLE-88) Eagle web-service should have status url
Su Ralph created EAGLE-88: - Summary: Eagle web-service should have status url Key: EAGLE-88 URL: https://issues.apache.org/jira/browse/EAGLE-88 Project: Eagle Issue Type: Test Affects Versions: 0.3.0 Reporter: Su Ralph Assignee: Su Ralph Fix For: 0.3.0 As a standalone service, eagle-service should have status url which expose the system status. This is the convention that for a service should make itself reachable and monitor-able, and by have this status url service downgrade would be possible. And this url is good fit of dropwizard metrics expose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057271#comment-15057271 ] Zhao, Qingwen commented on EAGLE-74: https://github.com/eaglemonitoring/eaglemonitoring.github.io/pull/38 > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057272#comment-15057272 ] Zhao, Qingwen commented on EAGLE-74: https://github.com/eaglemonitoring/eaglemonitoring.github.io/pull/38 > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhao, Qingwen updated EAGLE-74: --- Comment: was deleted (was: https://github.com/eaglemonitoring/eaglemonitoring.github.io/pull/38) > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057273#comment-15057273 ] Zhao, Qingwen commented on EAGLE-74: https://github.com/eaglemonitoring/eaglemonitoring.github.io/pull/38 > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: [EAGLE-74] Change package name from ...
Github user qingwen220 commented on the pull request: https://github.com/apache/incubator-eagle/pull/30#issuecomment-164632662 Sorry, duplicate pull request --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request: [EAGLE-74] Change package name from ...
Github user qingwen220 closed the pull request at: https://github.com/apache/incubator-eagle/pull/30 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057266#comment-15057266 ] ASF GitHub Bot commented on EAGLE-74: - Github user qingwen220 commented on the pull request: https://github.com/apache/incubator-eagle/pull/30#issuecomment-164632662 Sorry, duplicate pull request > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057267#comment-15057267 ] ASF GitHub Bot commented on EAGLE-74: - Github user qingwen220 closed the pull request at: https://github.com/apache/incubator-eagle/pull/30 > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057262#comment-15057262 ] ASF GitHub Bot commented on EAGLE-74: - GitHub user qingwen220 opened a pull request: https://github.com/apache/incubator-eagle/pull/30 [EAGLE-74] Change package name from eagle. to org.apache.eagle https://issues.apache.org/jira/browse/EAGLE-74 You can merge this pull request into a Git repository by running: $ git pull https://github.com/qingwen220/incubator-eagle EAGLE-74 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/30.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #30 commit 993690ca9d5ca8148543cb45b1458ee1eaedebff Author: Zhao, Qingwen Date: 2015-12-08T10:25:17Z delete unused doc > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: [EAGLE-74] Change package name from ...
GitHub user qingwen220 opened a pull request: https://github.com/apache/incubator-eagle/pull/30 [EAGLE-74] Change package name from eagle. to org.apache.eagle https://issues.apache.org/jira/browse/EAGLE-74 You can merge this pull request into a Git repository by running: $ git pull https://github.com/qingwen220/incubator-eagle EAGLE-74 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/30.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #30 commit 993690ca9d5ca8148543cb45b1458ee1eaedebff Author: Zhao, Qingwen Date: 2015-12-08T10:25:17Z delete unused doc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057254#comment-15057254 ] Hao Chen commented on EAGLE-74: --- What's the pull request for website change? > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057253#comment-15057253 ] Hao Chen commented on EAGLE-74: --- Thanks > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057232#comment-15057232 ] Zhao, Qingwen commented on EAGLE-74: Actually it's resolved. As this problem is about the website. Another commit has been submit to that repo. Best Regards, Qingwen Zhao | 赵晴雯 > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-83) Integrate Jenkins with GitHub for checking pull requests
[ https://issues.apache.org/jira/browse/EAGLE-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057223#comment-15057223 ] Hao Chen commented on EAGLE-83: --- How about the progress for the ticket? > Integrate Jenkins with GitHub for checking pull requests > > > Key: EAGLE-83 > URL: https://issues.apache.org/jira/browse/EAGLE-83 > Project: Eagle > Issue Type: Task >Reporter: Michael Wu >Assignee: Michael Wu > Labels: test > Original Estimate: 168h > Remaining Estimate: 168h > > We expect that when a pull request comes up, a Jenkins job is triggered to > verify the quality of new updates relevant to the PR, and automatically > provide advices for whether it should be applied, with the verification > result posted to github PR page for reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-84) Create scripts for merging PRs automatically
[
https://issues.apache.org/jira/browse/EAGLE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057221#comment-15057221
]
Hao Chen commented on EAGLE-84:
---
And we have documented the contributing progress here:
https://cwiki.apache.org/confluence/display/EAG/Contributing+to+Eagle please
make sure keep consistent with it if no problem.
> Create scripts for merging PRs automatically
>
>
> Key: EAGLE-84
> URL: https://issues.apache.org/jira/browse/EAGLE-84
> Project: Eagle
> Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> While a pull request gets passed the testing steps, and is verified as
> good-to-merge, people are willing to merge it with scripts to go through
> necessary steps automatically, and can form the message of final aggregated
> commit in a unified format. To achieve this, we are attempted to create the
> script.
> As discussed, the format of message should be like below (ignore the lines
> consisted of asterisks):
> **
> EAGLE-${ticket_number} ${ticket_title}
> ${ticket_url}
> Author: ${commiter}
> Reviewer: ${reviewer}
> Closes #${pr_number} from ${do_person}
> **
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (EAGLE-84) Create scripts for merging PRs automatically
[
https://issues.apache.org/jira/browse/EAGLE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057218#comment-15057218
]
Hao Chen commented on EAGLE-84:
---
Thanks for pseudocode, how about the development progress now?
> Create scripts for merging PRs automatically
>
>
> Key: EAGLE-84
> URL: https://issues.apache.org/jira/browse/EAGLE-84
> Project: Eagle
> Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> While a pull request gets passed the testing steps, and is verified as
> good-to-merge, people are willing to merge it with scripts to go through
> necessary steps automatically, and can form the message of final aggregated
> commit in a unified format. To achieve this, we are attempted to create the
> script.
> As discussed, the format of message should be like below (ignore the lines
> consisted of asterisks):
> **
> EAGLE-${ticket_number} ${ticket_title}
> ${ticket_url}
> Author: ${commiter}
> Reviewer: ${reviewer}
> Closes #${pr_number} from ${do_person}
> **
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (EAGLE-86) Rewrite HDFSAuditLogParser to a non-regular expression version
[ https://issues.apache.org/jira/browse/EAGLE-86?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057215#comment-15057215 ] Hao Chen commented on EAGLE-86: --- Please go through https://cwiki.apache.org/confluence/display/EAG/Contributing+to+Eagle before making contribution. > Rewrite HDFSAuditLogParser to a non-regular expression version > -- > > Key: EAGLE-86 > URL: https://issues.apache.org/jira/browse/EAGLE-86 > Project: Eagle > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Libin, Sun >Assignee: Libin, Sun >Priority: Minor > Fix For: 0.3.0 > > > HDFSAuditLogParser use many regular expressions to parse audit log, which is > a bit heavy, to improve topology's throughput, consider to rewrite it to a > non-regular expression version -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (EAGLE-86) Rewrite HDFSAuditLogParser to a non-regular expression version
[ https://issues.apache.org/jira/browse/EAGLE-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated EAGLE-86: -- Fix Version/s: 0.3.0 > Rewrite HDFSAuditLogParser to a non-regular expression version > -- > > Key: EAGLE-86 > URL: https://issues.apache.org/jira/browse/EAGLE-86 > Project: Eagle > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Libin, Sun >Assignee: Libin, Sun >Priority: Minor > Fix For: 0.3.0 > > > HDFSAuditLogParser use many regular expressions to parse audit log, which is > a bit heavy, to improve topology's throughput, consider to rewrite it to a > non-regular expression version -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (EAGLE-80) remove httpd directory
[ https://issues.apache.org/jira/browse/EAGLE-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen resolved EAGLE-80. --- Resolution: Fixed > remove httpd directory > -- > > Key: EAGLE-80 > URL: https://issues.apache.org/jira/browse/EAGLE-80 > Project: Eagle > Issue Type: Improvement >Reporter: Zhaokun Qin > Fix For: 0.3.0 > > > remove empty and useless httpd directory in docker image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-80: remove httpd directory
Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/21 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-80) remove httpd directory
[ https://issues.apache.org/jira/browse/EAGLE-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057177#comment-15057177 ] ASF GitHub Bot commented on EAGLE-80: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/21 > remove httpd directory > -- > > Key: EAGLE-80 > URL: https://issues.apache.org/jira/browse/EAGLE-80 > Project: Eagle > Issue Type: Improvement >Reporter: Zhaokun Qin > Fix For: 0.3.0 > > > remove empty and useless httpd directory in docker image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-80) remove httpd directory
[ https://issues.apache.org/jira/browse/EAGLE-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057174#comment-15057174 ] ASF GitHub Bot commented on EAGLE-80: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/21#issuecomment-164618975 LGTM, will merge. > remove httpd directory > -- > > Key: EAGLE-80 > URL: https://issues.apache.org/jira/browse/EAGLE-80 > Project: Eagle > Issue Type: Improvement >Reporter: Zhaokun Qin > Fix For: 0.3.0 > > > remove empty and useless httpd directory in docker image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-80: remove httpd directory
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/21#issuecomment-164618975 LGTM, will merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (EAGLE-87) Fail creating new policyEvaluator if the policy uses contains function.
hemanth dendukuri created EAGLE-87: -- Summary: Fail creating new policyEvaluator if the policy uses contains function. Key: EAGLE-87 URL: https://issues.apache.org/jira/browse/EAGLE-87 Project: Eagle Issue Type: Bug Environment: Hortonworks Sandbox 2.4 Reporter: hemanth dendukuri Priority: Minor Here is the policy : from hdfsAuditLogEventStream[(cmd == 'rename') and (str:contains(dst,'/tmp/pii')==true)] select * insert into outputStream; Error in Storm topology log. 2015-12-15 00:19:39 o.a.e.s.c.i.EagleServiceClientImpl [INFO] Going to query service: http://localhost:9099/eagle-service/rest/entities?query=AlertDefinitionService%5B%40site%3D%22sandbox%22+AND+%40dataSource%3D%22hdfsAuditLog%22%5D%7B*%7D&pageSize=2147483647&treeAgg=false 2015-12-15 00:19:39 o.a.e.a.n.AlertNotificationExecutor [INFO] alert notification config really changed prefix:alertdef, timestamp:0, humanReadableDate:1970-01-01 00:00:00,000, tags: site=sandbox,dataSource=hdfsAuditLog,policyId=ContainsTest,alertExecutorId=hdfsAuditLogAlertExecutor,policyType=siddhiCEPEngine,, encodedRowkey:YEktKX_62aP_6x97yoSv3B0ANd9Hby--xyCZKe2KKltxS9hcZXeJk1Je-7-Mrq0lGQ 2015-12-15 00:19:39 o.a.e.a.d.AlertDeduplicationExecutorBase [INFO] Alert dedup config really added prefix:alertdef, timestamp:0, humanReadableDate:1970-01-01 00:00:00,000, tags: site=sandbox,dataSource=hdfsAuditLog,policyId=ContainsTest,alertExecutorId=hdfsAuditLogAlertExecutor,policyType=siddhiCEPEngine,, encodedRowkey:YEktKX_62aP_6x97yoSv3B0ANd9Hby--xyCZKe2KKltxS9hcZXeJk1Je-7-Mrq0lGQ 2015-12-15 00:19:39 o.a.e.a.d.AlertDeduplicationExecutorBase [INFO] Alert dedup config really added prefix:alertdef, timestamp:0, humanReadableDate:1970-01-01 00:00:00,000, tags: site=sandbox,dataSource=hdfsAuditLog,policyId=ContainsTest,alertExecutorId=hdfsAuditLogAlertExecutor,policyType=siddhiCEPEngine,, encodedRowkey:YEktKX_62aP_6x97yoSv3B0ANd9Hby--xyCZKe2KKltxS9hcZXeJk1Je-7-Mrq0lGQ 2015-12-15 00:19:39 o.a.e.e.AlertExecutor [INFO] hdfsAuditLogAlertExecutor, partition 0 policy really added prefix:alertdef, timestamp:0, humanReadableDate:1970-01-01 00:00:00,000, tags: site=sandbox,dataSource=hdfsAuditLog,policyId=ContainsTest,alertExecutorId=hdfsAuditLogAlertExecutor,policyType=siddhiCEPEngine,, encodedRowkey:YEktKX_62aP_6x97yoSv3B0ANd9Hby--xyCZKe2KKltxS9hcZXeJk1Je-7-Mrq0lGQ 2015-12-15 00:19:40 o.a.e.a.s.SiddhiPolicyEvaluator [INFO] Siddhi stream definition : define stream hdfsAuditLogEventStream(eagleAlertContext object,allowed bool,cmd string,dst string,host string,securityZone string,sensitivityType string,src string,timestamp long,user string); 2015-12-15 00:19:40 o.a.e.e.AlertExecutor [ERROR] Fail creating new policyEvaluator java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_79] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_79] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_79] at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_79] at org.apache.eagle.executor.AlertExecutor.createPolicyEvaluator(AlertExecutor.java:217) [stormjar.jar:na] at org.apache.eagle.executor.AlertExecutor.onPolicyCreated(AlertExecutor.java:356) [stormjar.jar:na] at org.apache.eagle.alert.policy.DynamicPolicyLoader$1.handleEvent(DynamicPolicyLoader.java:119) [stormjar.jar:na] at com.netflix.config.AbstractPollingScheduler.fireEvent(AbstractPollingScheduler.java:184) [stormjar.jar:na] at com.netflix.config.AbstractPollingScheduler.access$200(AbstractPollingScheduler.java:41) [stormjar.jar:na] at com.netflix.config.AbstractPollingScheduler$1.run(AbstractPollingScheduler.java:165) [stormjar.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_79] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: org.wso2.siddhi.query.api.exception.ExecutionPlanValidationException: contains is neither a function extension nor an aggregated attribute extension, when creating quer
API test
Hi all,
I called the eagle API to create a policy but ended with "HTTP Status 403
Access Denied". I'm not sure if there is something wrong with the
authorization part?
Here is how I called the policy definition API:
URL:
http://centos.da2:9099/eagle-service/rest/entities?serviceName=AlertDefinitionService
Method: POST
Headers:
"Content-Type:application/json",
"Authorization: Basic YWRtaW46c2VjcmV0"( "admin:secret" is encoded to
Base64 format, which is "YWRtaW46c2VjcmV0")
Data:
[{
"tags": {
"site": "Demo",
"dataSource": "hdfsAuditLog",
"policyId": "testPolicy",
"alertExecutorId": "hdfsAuditLogAlertExecutor",
"policyType": "siddhiCEPEngine"
},
"desc": "test alert policy",
"policyDef": {
"type": "siddhiCEPEngine",
"expression": "from hdfsAuditLogEventStream[src
=='/test0'] select * insert into outputStream;"
},
"notificationDef": [{
"sender": "noreply-ea...@company.com",
"recipients": "u...@company.com",
"subject": "test alert policy",
"flavor": "email",
"id": "email_1"
}],
"enabled": true
}]
Thanks,
Daniel
RE: Cygwin do no support npm?
Thanks for your advice, I've fixed it now. Regards, Daniel -Original Message- From: Zhang, Edward (GDI Hadoop) [mailto:yonzh...@ebay.com] Sent: Sunday, December 13, 2015 8:00 PM To: dev@eagle.incubator.apache.org Subject: Re: Cygwin do no support npm? I don¹t think that is related to windows. In linux we should do the same thing npm, git etc. I tried 3 environment, linux, mac osx, and windows. I don¹t see any issue with build. Thanks Edward On 12/13/15, 18:07, "Jilin" wrote: >Hi Daniel, >If you want to use Cygwin to build eagle, you need do much more >additional work on it. It will cost your *much time* on the windows >environment configuration. I strongly suggest to use Linux instead. >Here is the windows install step: > > 1. Install windows version nodejs > 2. Add npm in path > 3. Install git for windows(https://git-for-windows.github.io/) > 4. Add git in path > 5. (If you installed git in Cygwin, you need remove Cygwin version) > >It's not good way for eagle building. If you don't have linux env, I >think you can try this. > >2015-12-12 6:04 GMT+08:00 Daniel Zhou : > >> Hi all, >> >> Just a short question: >> My system is windows 7, in Cygwin I'm using maven to build >> "eagle-incubator". When it comes to "eagle-webservice", it says: >> "Environment Check... >> npm not installed!". >> The official document from npm says "installing on Cygwin: No". Does >> it mean I cannot use Cygwin to develop eagle-incubator? >> >> Regards, >> Daniel >>
Re: [Discussion] Apache Eagle roadmap
That is valid concern. Looks Arun will materialize roadmap to wiki and collect feedback soon. In overall, the plan should be the same with that which was proposed in incubator proposal. But hopefully community can give more suggestions. Thanks Edward On 12/14/15, 11:16, "Balaji Ganesan" wrote: >As an outside observer, this would be really helpful to me. Right now I do >see a lot of activity, but I am not sure which direction the community is >headed. > >Having a simple roadmap page in the wiki would be very useful. > >On Mon, Dec 14, 2015 at 10:55 AM, Arun Manoharan > >wrote: > >> Hi Folks, >> >> Any advice on creating a roadmap, collaborating and publishing in Apache >> way ? >> >> We are working on different areas of the platform and we have new use >>cases >> to be onboarded wrt system monitoring. So i think it will be beneficial >>to >> have a quarterly roadmap. >> >> Thanks, >> Arun >>
Re: [Discussion] Apache Eagle roadmap
As an outside observer, this would be really helpful to me. Right now I do see a lot of activity, but I am not sure which direction the community is headed. Having a simple roadmap page in the wiki would be very useful. On Mon, Dec 14, 2015 at 10:55 AM, Arun Manoharan wrote: > Hi Folks, > > Any advice on creating a roadmap, collaborating and publishing in Apache > way ? > > We are working on different areas of the platform and we have new use cases > to be onboarded wrt system monitoring. So i think it will be beneficial to > have a quarterly roadmap. > > Thanks, > Arun >
Re: [Discuss] Hadoop metrics,job,GC monitoring
Thanks Edward for starting the thread. I think it is important to have the job monitoring (MR/Spark) workloads for performance of the cluster and availability. But it will be beneficial to have an extensible framework where users can create business rules like "I want an alert when NN is in safemode or RM is flipping etc". Thanks, Arun On Mon, Dec 14, 2015 at 10:58 AM, Zhang, Edward (GDI Hadoop) < yonzh...@ebay.com> wrote: > Hi Eagle devs/users, > > As proposed in apache eagle incubator proposal, Eagle will start > design/dev to support Hadoop system monitoring besides security monitoring > which includes Hadoop native metrics, job, gclog etc. > > The community is also interested in Hadoop system monitoring by Eagle when > we recently talked about Eagle product in public conferences, meet up etc. > > Take Hadoop native metrics as an example, first of all those metrics are > pretty valuable in determining system health status, secondly collecting > huge amount metrics, visualizing, and alerting is very challenging. We > need think of declarative collection, dynamic aggregation, metric storage, > metric query engine etc. > > Besides technical design, comprehensive policy/rule are also valuable to > be shared in the community. Those policy/rule represent best practice in > the world to manage large Hadoop clusters. > > Please suggest whatever is for engineering design or business policy/rules. > > Thanks > Edward > >
[Discuss] Hadoop metrics,job,GC monitoring
Hi Eagle devs/users, As proposed in apache eagle incubator proposal, Eagle will start design/dev to support Hadoop system monitoring besides security monitoring which includes Hadoop native metrics, job, gclog etc. The community is also interested in Hadoop system monitoring by Eagle when we recently talked about Eagle product in public conferences, meet up etc. Take Hadoop native metrics as an example, first of all those metrics are pretty valuable in determining system health status, secondly collecting huge amount metrics, visualizing, and alerting is very challenging. We need think of declarative collection, dynamic aggregation, metric storage, metric query engine etc. Besides technical design, comprehensive policy/rule are also valuable to be shared in the community. Those policy/rule represent best practice in the world to manage large Hadoop clusters. Please suggest whatever is for engineering design or business policy/rules. Thanks Edward
[Discussion] Apache Eagle roadmap
Hi Folks, Any advice on creating a roadmap, collaborating and publishing in Apache way ? We are working on different areas of the platform and we have new use cases to be onboarded wrt system monitoring. So i think it will be beneficial to have a quarterly roadmap. Thanks, Arun
[jira] [Updated] (EAGLE-81) Notification Plugin Framework
[ https://issues.apache.org/jira/browse/EAGLE-81?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manoharan updated EAGLE-81: Assignee: Senthilkumar > Notification Plugin Framework > - > > Key: EAGLE-81 > URL: https://issues.apache.org/jira/browse/EAGLE-81 > Project: Eagle > Issue Type: New Feature >Reporter: Senthilkumar >Assignee: Senthilkumar > Original Estimate: 336h > Remaining Estimate: 336h > > Today notification is only allowed for email .. We need more notification > methods to send alert to external system . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (EAGLE-82) Eagle - Remediation Connector with Apache Ranger
[ https://issues.apache.org/jira/browse/EAGLE-82?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manoharan updated EAGLE-82: Assignee: Senthilkumar > Eagle - Remediation Connector with Apache Ranger > > > Key: EAGLE-82 > URL: https://issues.apache.org/jira/browse/EAGLE-82 > Project: Eagle > Issue Type: New Feature >Reporter: Senthilkumar >Assignee: Senthilkumar > > Today there is no API to send Remediation Request to Ranger.. We need API > that connects Ranger and Request for Remediation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (EAGLE-47) Ability to audit who made changes to Eagle policies
[ https://issues.apache.org/jira/browse/EAGLE-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manoharan updated EAGLE-47: Assignee: Senthilkumar > Ability to audit who made changes to Eagle policies > --- > > Key: EAGLE-47 > URL: https://issues.apache.org/jira/browse/EAGLE-47 > Project: Eagle > Issue Type: New Feature >Affects Versions: 0.3.0 >Reporter: Arun Manoharan >Assignee: Senthilkumar > Fix For: 0.3.0 > > > As an admin of Apache Eagle I would like to know who created policies, > modified policies, Deleted policies. > Also who added source, site etc. > This information is a must have for security related products to have an > audit trail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (EAGLE-82) Eagle - Remediation Connector with Apache Ranger
[ https://issues.apache.org/jira/browse/EAGLE-82?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manoharan updated EAGLE-82: Assignee: Arun Manoharan > Eagle - Remediation Connector with Apache Ranger > > > Key: EAGLE-82 > URL: https://issues.apache.org/jira/browse/EAGLE-82 > Project: Eagle > Issue Type: New Feature >Reporter: Senthilkumar >Assignee: Arun Manoharan > > Today there is no API to send Remediation Request to Ranger.. We need API > that connects Ranger and Request for Remediation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (EAGLE-82) Eagle - Remediation Connector with Apache Ranger
[ https://issues.apache.org/jira/browse/EAGLE-82?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manoharan updated EAGLE-82: Assignee: (was: Arun Manoharan) > Eagle - Remediation Connector with Apache Ranger > > > Key: EAGLE-82 > URL: https://issues.apache.org/jira/browse/EAGLE-82 > Project: Eagle > Issue Type: New Feature >Reporter: Senthilkumar > > Today there is no API to send Remediation Request to Ranger.. We need API > that connects Ranger and Request for Remediation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056248#comment-15056248 ] ASF GitHub Bot commented on EAGLE-85: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/27 > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/27 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164467062 LGTM. Thanks very much for the contribution! @RalphSu --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056107#comment-15056107 ] ASF GitHub Bot commented on EAGLE-85: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164467062 LGTM. Thanks very much for the contribution! @RalphSu > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164463110 `shengli` should be someone's internal account name :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056086#comment-15056086 ] ASF GitHub Bot commented on EAGLE-85: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164463110 `shengli` should be someone's internal account name :-) > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056071#comment-15056071 ] ASF GitHub Bot commented on EAGLE-85: - Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164459877 Make them as more meaningless ones t1 t2 > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164459877 Make them as more meaningless ones t1 t2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056029#comment-15056029 ] ASF GitHub Bot commented on EAGLE-85: - Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164449409 i don't see why test_lateral_view_shengli/bm2_item4_5 are sensitive? > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164449409 i don't see why test_lateral_view_shengli/bm2_item4_5 are sensitive? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-66) Eagle TypeSafe Stream Processing DSL
[
https://issues.apache.org/jira/browse/EAGLE-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055973#comment-15055973
]
ASF GitHub Bot commented on EAGLE-66:
-
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/26#discussion_r47496505
--- Diff: test.txt ---
@@ -1 +0,0 @@
-test for keep file for user command reassembler
--- End diff --
I remove test.txt here, dose anyone know why should we keep this file under
code base root?
> Eagle TypeSafe Stream Processing DSL
>
>
> Key: EAGLE-66
> URL: https://issues.apache.org/jira/browse/EAGLE-66
> Project: Eagle
> Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Hao Chen
>Assignee: Hao Chen
> Fix For: 0.3.0
>
>
> h1. Main Features
> 1. Typesafe API: Currently the stream processing API is not type-safe (Type
> info are erased by stream framework), all programming interfaces for
> developer are faced to Object/AnyRef, which is not very friendly and
> extensible for framework user (application developer):
> {code}
> public void flatMap(java.util.List input, Collector AlertAPIEntity>> outputCollector)
> {code}
> So i propose the interface as (all type information are transparent for
> developer, not need additional parameters, supported by Scala implicit
> TypeTag)
> {code}class StreamProducer[+T <: Any] extends StreamInfo with
> StreamProtocol[T]{code}
> * StreamInfo: contains Stream core information including streamId,
> processing element id/name, entity type info (class/TypeTag)
> * StreamProtocol extends JavaStreamProtocol: contains basic Stream DSL API
> and java-compatible API
> {code}
> class StreamInfo extends Serializable{
> val id:Int = UniqueId.incrementAndGetId()
> var name: String = null
> var streamId:String=null
> var parallelismNum: Int = 1
> var inKeyed:Boolean = false
> var outKeyed:Boolean = false
> var keySelector:KeySelector = null
> var typeClass:Class[_] = classOf[AnyRef]
> @transient implicit var typeTag:ru.TypeTag[_] = ru.typeTag[AnyRef]
> }
> {code}
> And the StreamInfo can be shared through the runtime as implicit context for
> execution layer as well:
> {code}
> abstract class AbstractStreamBolt[T](val fieldsNum:Int=0, val ack:Boolean =
> true)(implicit streamInfo:StreamInfo) extends BaseRichBolt
> {code}
> 2. KeyValue Based Structure: currently framework user (developer) have to
> handle with field declaration again and again, and framework and business
> logic are highly coupled, according to the StreamProtocol description, user
> should not care about framework level detail like internal data structure for
> Storm using List with Fields which is not friendly for
> developer, we should make sure user focus on business logic only like:
> {code}
> env.from(tuples)
> .groupByKey(_.name)
> {code}
> 3. Spout grouping instead of overriding Schema: currently especially in use
> case like HdfsAuditLog Monitoring, if developer wants to groupby certain
> key, they are forced to override Schema (specific for storm) , which is not
> good and un-reusable.
> 4. Environment Decoupled: currently the stream (metadata) /dag (logic) /
> environment (execution) are coupled with storm internal implementation, which
> is not good for becoming a metadata-driven pipeline framework in future, so
> we should decouple it.
>
> 5. Compatible with Field-based Structure in old framework and application.
> 6. Configuration: enhanced config wrapper upon typesafe-config for supporting
> get/set/default and integrated with ExecutionEnvironment
> {code}
> val env = ExecutionEnvironments.get[StormExecutionEnvironment](args)
> val streamName = env.config.get[String]("eagle.stream.name","eventStream")
> val streamExecutorId =
> env.config.get[String]("eagle.stream.executor",s"${streamName}Executor")
>
> env.config.set("dataSourceConfig.deserializerClass",classOf[JsonMessageDeserializer].getCanonicalName)
> {code}
> h1. Sample Application
> {code}
> case class Entity(name:String,value:Double,var inc:Int=0)
> val tuples = Seq(
> Entity("a", 1),
> Entity("a", 2),
> Entity("a", 3),
> Entity("b", 2),
> Entity("c", 3),
> Entity("d", 3)
> )
> val env = ExecutionEnvironments.get[StormExecutionEnvironment](args)
> // DAG is fully automatically aware: Entity -> Tuple2 -> Tuple3
> env.from(tuples)
> .groupByKey(_.name)
> .map(o => {o.inc += 2;o})
> .filter(_.name != "b")
> .filter(_.name != "c")
> .groupByKey(o=>(o.name,o.value))
> .map(o => (o.name,o))
> .map(o => (o._1,o._2.value,o._2.inc))
> .foreach(println)
> env.execute()
> {code}
> Type is transparent for developer during both DAG compiling (programming) and
> runtime (me
[GitHub] incubator-eagle pull request: EAGLE-66. Typesafe Streaming DSL and...
Github user haoch commented on a diff in the pull request: https://github.com/apache/incubator-eagle/pull/26#discussion_r47496505 --- Diff: test.txt --- @@ -1 +0,0 @@ -test for keep file for user command reassembler --- End diff -- I remove test.txt here, dose anyone know why should we keep this file under code base root? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-66) Eagle TypeSafe Stream Processing DSL
[
https://issues.apache.org/jira/browse/EAGLE-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055969#comment-15055969
]
ASF GitHub Bot commented on EAGLE-66:
-
Github user haoch commented on the pull request:
https://github.com/apache/incubator-eagle/pull/17#issuecomment-164435279
Deprecated pull request, will close soon, please just ignore.
> Eagle TypeSafe Stream Processing DSL
>
>
> Key: EAGLE-66
> URL: https://issues.apache.org/jira/browse/EAGLE-66
> Project: Eagle
> Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Hao Chen
>Assignee: Hao Chen
> Fix For: 0.3.0
>
>
> h1. Main Features
> 1. Typesafe API: Currently the stream processing API is not type-safe (Type
> info are erased by stream framework), all programming interfaces for
> developer are faced to Object/AnyRef, which is not very friendly and
> extensible for framework user (application developer):
> {code}
> public void flatMap(java.util.List input, Collector AlertAPIEntity>> outputCollector)
> {code}
> So i propose the interface as (all type information are transparent for
> developer, not need additional parameters, supported by Scala implicit
> TypeTag)
> {code}class StreamProducer[+T <: Any] extends StreamInfo with
> StreamProtocol[T]{code}
> * StreamInfo: contains Stream core information including streamId,
> processing element id/name, entity type info (class/TypeTag)
> * StreamProtocol extends JavaStreamProtocol: contains basic Stream DSL API
> and java-compatible API
> {code}
> class StreamInfo extends Serializable{
> val id:Int = UniqueId.incrementAndGetId()
> var name: String = null
> var streamId:String=null
> var parallelismNum: Int = 1
> var inKeyed:Boolean = false
> var outKeyed:Boolean = false
> var keySelector:KeySelector = null
> var typeClass:Class[_] = classOf[AnyRef]
> @transient implicit var typeTag:ru.TypeTag[_] = ru.typeTag[AnyRef]
> }
> {code}
> And the StreamInfo can be shared through the runtime as implicit context for
> execution layer as well:
> {code}
> abstract class AbstractStreamBolt[T](val fieldsNum:Int=0, val ack:Boolean =
> true)(implicit streamInfo:StreamInfo) extends BaseRichBolt
> {code}
> 2. KeyValue Based Structure: currently framework user (developer) have to
> handle with field declaration again and again, and framework and business
> logic are highly coupled, according to the StreamProtocol description, user
> should not care about framework level detail like internal data structure for
> Storm using List with Fields which is not friendly for
> developer, we should make sure user focus on business logic only like:
> {code}
> env.from(tuples)
> .groupByKey(_.name)
> {code}
> 3. Spout grouping instead of overriding Schema: currently especially in use
> case like HdfsAuditLog Monitoring, if developer wants to groupby certain
> key, they are forced to override Schema (specific for storm) , which is not
> good and un-reusable.
> 4. Environment Decoupled: currently the stream (metadata) /dag (logic) /
> environment (execution) are coupled with storm internal implementation, which
> is not good for becoming a metadata-driven pipeline framework in future, so
> we should decouple it.
>
> 5. Compatible with Field-based Structure in old framework and application.
> 6. Configuration: enhanced config wrapper upon typesafe-config for supporting
> get/set/default and integrated with ExecutionEnvironment
> {code}
> val env = ExecutionEnvironments.get[StormExecutionEnvironment](args)
> val streamName = env.config.get[String]("eagle.stream.name","eventStream")
> val streamExecutorId =
> env.config.get[String]("eagle.stream.executor",s"${streamName}Executor")
>
> env.config.set("dataSourceConfig.deserializerClass",classOf[JsonMessageDeserializer].getCanonicalName)
> {code}
> h1. Sample Application
> {code}
> case class Entity(name:String,value:Double,var inc:Int=0)
> val tuples = Seq(
> Entity("a", 1),
> Entity("a", 2),
> Entity("a", 3),
> Entity("b", 2),
> Entity("c", 3),
> Entity("d", 3)
> )
> val env = ExecutionEnvironments.get[StormExecutionEnvironment](args)
> // DAG is fully automatically aware: Entity -> Tuple2 -> Tuple3
> env.from(tuples)
> .groupByKey(_.name)
> .map(o => {o.inc += 2;o})
> .filter(_.name != "b")
> .filter(_.name != "c")
> .groupByKey(o=>(o.name,o.value))
> .map(o => (o.name,o))
> .map(o => (o._1,o._2.value,o._2.inc))
> .foreach(println)
> env.execute()
> {code}
> Type is transparent for developer during both DAG compiling (programming) and
> runtime (metadata) phases
> {code}
> 2015-12-07 15:17:19,820 INFO [main] utils.GraphPrinter$[43]: Before expanded
> DAG
> {
> IterableStreamProducer[Entity]_1{1} ->
[GitHub] incubator-eagle pull request: [EAGLE-66] Typesafe Streaming DSL an...
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/17#issuecomment-164435279 Deprecated pull request, will close soon, please just ignore. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-66) Eagle TypeSafe Stream Processing DSL
[
https://issues.apache.org/jira/browse/EAGLE-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055966#comment-15055966
]
ASF GitHub Bot commented on EAGLE-66:
-
Github user haoch commented on the pull request:
https://github.com/apache/incubator-eagle/pull/26#issuecomment-164434239
@RalphSu thanks for reviewing. Could anyone else please help review as
well? (Before Dec 15th, 2015)
> Eagle TypeSafe Stream Processing DSL
>
>
> Key: EAGLE-66
> URL: https://issues.apache.org/jira/browse/EAGLE-66
> Project: Eagle
> Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Hao Chen
>Assignee: Hao Chen
> Fix For: 0.3.0
>
>
> h1. Main Features
> 1. Typesafe API: Currently the stream processing API is not type-safe (Type
> info are erased by stream framework), all programming interfaces for
> developer are faced to Object/AnyRef, which is not very friendly and
> extensible for framework user (application developer):
> {code}
> public void flatMap(java.util.List input, Collector AlertAPIEntity>> outputCollector)
> {code}
> So i propose the interface as (all type information are transparent for
> developer, not need additional parameters, supported by Scala implicit
> TypeTag)
> {code}class StreamProducer[+T <: Any] extends StreamInfo with
> StreamProtocol[T]{code}
> * StreamInfo: contains Stream core information including streamId,
> processing element id/name, entity type info (class/TypeTag)
> * StreamProtocol extends JavaStreamProtocol: contains basic Stream DSL API
> and java-compatible API
> {code}
> class StreamInfo extends Serializable{
> val id:Int = UniqueId.incrementAndGetId()
> var name: String = null
> var streamId:String=null
> var parallelismNum: Int = 1
> var inKeyed:Boolean = false
> var outKeyed:Boolean = false
> var keySelector:KeySelector = null
> var typeClass:Class[_] = classOf[AnyRef]
> @transient implicit var typeTag:ru.TypeTag[_] = ru.typeTag[AnyRef]
> }
> {code}
> And the StreamInfo can be shared through the runtime as implicit context for
> execution layer as well:
> {code}
> abstract class AbstractStreamBolt[T](val fieldsNum:Int=0, val ack:Boolean =
> true)(implicit streamInfo:StreamInfo) extends BaseRichBolt
> {code}
> 2. KeyValue Based Structure: currently framework user (developer) have to
> handle with field declaration again and again, and framework and business
> logic are highly coupled, according to the StreamProtocol description, user
> should not care about framework level detail like internal data structure for
> Storm using List with Fields which is not friendly for
> developer, we should make sure user focus on business logic only like:
> {code}
> env.from(tuples)
> .groupByKey(_.name)
> {code}
> 3. Spout grouping instead of overriding Schema: currently especially in use
> case like HdfsAuditLog Monitoring, if developer wants to groupby certain
> key, they are forced to override Schema (specific for storm) , which is not
> good and un-reusable.
> 4. Environment Decoupled: currently the stream (metadata) /dag (logic) /
> environment (execution) are coupled with storm internal implementation, which
> is not good for becoming a metadata-driven pipeline framework in future, so
> we should decouple it.
>
> 5. Compatible with Field-based Structure in old framework and application.
> 6. Configuration: enhanced config wrapper upon typesafe-config for supporting
> get/set/default and integrated with ExecutionEnvironment
> {code}
> val env = ExecutionEnvironments.get[StormExecutionEnvironment](args)
> val streamName = env.config.get[String]("eagle.stream.name","eventStream")
> val streamExecutorId =
> env.config.get[String]("eagle.stream.executor",s"${streamName}Executor")
>
> env.config.set("dataSourceConfig.deserializerClass",classOf[JsonMessageDeserializer].getCanonicalName)
> {code}
> h1. Sample Application
> {code}
> case class Entity(name:String,value:Double,var inc:Int=0)
> val tuples = Seq(
> Entity("a", 1),
> Entity("a", 2),
> Entity("a", 3),
> Entity("b", 2),
> Entity("c", 3),
> Entity("d", 3)
> )
> val env = ExecutionEnvironments.get[StormExecutionEnvironment](args)
> // DAG is fully automatically aware: Entity -> Tuple2 -> Tuple3
> env.from(tuples)
> .groupByKey(_.name)
> .map(o => {o.inc += 2;o})
> .filter(_.name != "b")
> .filter(_.name != "c")
> .groupByKey(o=>(o.name,o.value))
> .map(o => (o.name,o))
> .map(o => (o._1,o._2.value,o._2.inc))
> .foreach(println)
> env.execute()
> {code}
> Type is transparent for developer during both DAG compiling (programming) and
> runtime (metadata) phases
> {code}
> 2015-12-07 15:17:19,820 INFO [main] utils.GraphPrinter$[43]: Before expanded
> DAG
> {
>
[GitHub] incubator-eagle pull request: EAGLE-66. Typesafe Streaming DSL and...
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/26#issuecomment-164434239 @RalphSu thanks for reviewing. Could anyone else please help review as well? (Before Dec 15th, 2015) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-80) remove httpd directory
[ https://issues.apache.org/jira/browse/EAGLE-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055938#comment-15055938 ] Hao Chen commented on EAGLE-80: --- That's great, how do you fix it and where are the fixing changes? > remove httpd directory > -- > > Key: EAGLE-80 > URL: https://issues.apache.org/jira/browse/EAGLE-80 > Project: Eagle > Issue Type: Improvement >Reporter: Zhaokun Qin > Fix For: 0.3.0 > > > remove empty and useless httpd directory in docker image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE 86: Rewrite HDFSAuditLogParser...
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/29#issuecomment-164427043 We should make sure referring to ticket id `EAGLE-ID` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-86) Rewrite HDFSAuditLogParser to a non-regular expression version
[ https://issues.apache.org/jira/browse/EAGLE-86?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055935#comment-15055935 ] ASF GitHub Bot commented on EAGLE-86: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/29#issuecomment-164426803 LGTM. Only one problem: could you pls. change the pull request title as "EAGLE-86: Rewrite HDFSAuditLogParser to a non-regular expression version"? > Rewrite HDFSAuditLogParser to a non-regular expression version > -- > > Key: EAGLE-86 > URL: https://issues.apache.org/jira/browse/EAGLE-86 > Project: Eagle > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Libin, Sun >Assignee: Libin, Sun >Priority: Minor > > HDFSAuditLogParser use many regular expressions to parse audit log, which is > a bit heavy, to improve topology's throughput, consider to rewrite it to a > non-regular expression version -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE 86: Rewrite HDFSAuditLogParser...
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/29#issuecomment-164426803 LGTM. Only one problem: could you pls. change the pull request title as "EAGLE-86: Rewrite HDFSAuditLogParser to a non-regular expression version"? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164424948 https://github.com/RalphSu/incubator-eagle/blob/master/eagle-security/eagle-security-hive/src/test/java/org/apache/eagle/security/hive/ql/TestParser.java#L126 https://github.com/RalphSu/incubator-eagle/blob/master/eagle-security/eagle-security-hive/src/test/java/org/apache/eagle/security/hive/ql/TestParser.java#L153 https://github.com/RalphSu/incubator-eagle/blob/master/eagle-security/eagle-security-hive/src/test/java/org/apache/eagle/security/hive/ql/TestParser.java#L173 https://github.com/RalphSu/incubator-eagle/blob/master/eagle-security/eagle-security-hive/src/test/java/org/apache/eagle/security/hive/ql/TestParser.java#L262 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055917#comment-15055917 ] ASF GitHub Bot commented on EAGLE-85: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164424948 https://github.com/RalphSu/incubator-eagle/blob/master/eagle-security/eagle-security-hive/src/test/java/org/apache/eagle/security/hive/ql/TestParser.java#L126 https://github.com/RalphSu/incubator-eagle/blob/master/eagle-security/eagle-security-hive/src/test/java/org/apache/eagle/security/hive/ql/TestParser.java#L153 https://github.com/RalphSu/incubator-eagle/blob/master/eagle-security/eagle-security-hive/src/test/java/org/apache/eagle/security/hive/ql/TestParser.java#L173 https://github.com/RalphSu/incubator-eagle/blob/master/eagle-security/eagle-security-hive/src/test/java/org/apache/eagle/security/hive/ql/TestParser.java#L262 > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055912#comment-15055912 ] ASF GitHub Bot commented on EAGLE-85: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164424435 Thanks for the fix but seems not completely clean yet :-) > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164424435 Thanks for the fix but seems not completely clean yet :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055855#comment-15055855 ] ASF GitHub Bot commented on EAGLE-85: - Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164414066 Updated. > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164414066 Updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-80) remove httpd directory
[ https://issues.apache.org/jira/browse/EAGLE-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055854#comment-15055854 ] Zhaokun Qin commented on EAGLE-80: -- The httpd directory is empty and unused. We add it into /var/log directory as a temporary solution of ganglia installation's problem. But now, we can remove t because I find another way tackle this problem. > remove httpd directory > -- > > Key: EAGLE-80 > URL: https://issues.apache.org/jira/browse/EAGLE-80 > Project: Eagle > Issue Type: Improvement >Reporter: Zhaokun Qin > Fix For: 0.3.0 > > > remove empty and useless httpd directory in docker image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (EAGLE-80) remove httpd directory
[ https://issues.apache.org/jira/browse/EAGLE-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055854#comment-15055854 ] Zhaokun Qin edited comment on EAGLE-80 at 12/14/15 11:22 AM: - The httpd directory is empty and unused. We add it into /var/log directory as a temporary solution of ganglia installation's problem. But now, we can remove it because I find another way tackle this problem. was (Author: zqin): The httpd directory is empty and unused. We add it into /var/log directory as a temporary solution of ganglia installation's problem. But now, we can remove t because I find another way tackle this problem. > remove httpd directory > -- > > Key: EAGLE-80 > URL: https://issues.apache.org/jira/browse/EAGLE-80 > Project: Eagle > Issue Type: Improvement >Reporter: Zhaokun Qin > Fix For: 0.3.0 > > > remove empty and useless httpd directory in docker image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (EAGLE-66) Eagle TypeSafe Stream Processing DSL
[
https://issues.apache.org/jira/browse/EAGLE-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055840#comment-15055840
]
ASF GitHub Bot commented on EAGLE-66:
-
Github user RalphSu commented on the pull request:
https://github.com/apache/incubator-eagle/pull/26#issuecomment-164410901
Looks good
> Eagle TypeSafe Stream Processing DSL
>
>
> Key: EAGLE-66
> URL: https://issues.apache.org/jira/browse/EAGLE-66
> Project: Eagle
> Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Hao Chen
>Assignee: Hao Chen
> Fix For: 0.3.0
>
>
> h1. Main Features
> 1. Typesafe API: Currently the stream processing API is not type-safe (Type
> info are erased by stream framework), all programming interfaces for
> developer are faced to Object/AnyRef, which is not very friendly and
> extensible for framework user (application developer):
> {code}
> public void flatMap(java.util.List input, Collector AlertAPIEntity>> outputCollector)
> {code}
> So i propose the interface as (all type information are transparent for
> developer, not need additional parameters, supported by Scala implicit
> TypeTag)
> {code}class StreamProducer[+T <: Any] extends StreamInfo with
> StreamProtocol[T]{code}
> * StreamInfo: contains Stream core information including streamId,
> processing element id/name, entity type info (class/TypeTag)
> * StreamProtocol extends JavaStreamProtocol: contains basic Stream DSL API
> and java-compatible API
> {code}
> class StreamInfo extends Serializable{
> val id:Int = UniqueId.incrementAndGetId()
> var name: String = null
> var streamId:String=null
> var parallelismNum: Int = 1
> var inKeyed:Boolean = false
> var outKeyed:Boolean = false
> var keySelector:KeySelector = null
> var typeClass:Class[_] = classOf[AnyRef]
> @transient implicit var typeTag:ru.TypeTag[_] = ru.typeTag[AnyRef]
> }
> {code}
> And the StreamInfo can be shared through the runtime as implicit context for
> execution layer as well:
> {code}
> abstract class AbstractStreamBolt[T](val fieldsNum:Int=0, val ack:Boolean =
> true)(implicit streamInfo:StreamInfo) extends BaseRichBolt
> {code}
> 2. KeyValue Based Structure: currently framework user (developer) have to
> handle with field declaration again and again, and framework and business
> logic are highly coupled, according to the StreamProtocol description, user
> should not care about framework level detail like internal data structure for
> Storm using List with Fields which is not friendly for
> developer, we should make sure user focus on business logic only like:
> {code}
> env.from(tuples)
> .groupByKey(_.name)
> {code}
> 3. Spout grouping instead of overriding Schema: currently especially in use
> case like HdfsAuditLog Monitoring, if developer wants to groupby certain
> key, they are forced to override Schema (specific for storm) , which is not
> good and un-reusable.
> 4. Environment Decoupled: currently the stream (metadata) /dag (logic) /
> environment (execution) are coupled with storm internal implementation, which
> is not good for becoming a metadata-driven pipeline framework in future, so
> we should decouple it.
>
> 5. Compatible with Field-based Structure in old framework and application.
> 6. Configuration: enhanced config wrapper upon typesafe-config for supporting
> get/set/default and integrated with ExecutionEnvironment
> {code}
> val env = ExecutionEnvironments.get[StormExecutionEnvironment](args)
> val streamName = env.config.get[String]("eagle.stream.name","eventStream")
> val streamExecutorId =
> env.config.get[String]("eagle.stream.executor",s"${streamName}Executor")
>
> env.config.set("dataSourceConfig.deserializerClass",classOf[JsonMessageDeserializer].getCanonicalName)
> {code}
> h1. Sample Application
> {code}
> case class Entity(name:String,value:Double,var inc:Int=0)
> val tuples = Seq(
> Entity("a", 1),
> Entity("a", 2),
> Entity("a", 3),
> Entity("b", 2),
> Entity("c", 3),
> Entity("d", 3)
> )
> val env = ExecutionEnvironments.get[StormExecutionEnvironment](args)
> // DAG is fully automatically aware: Entity -> Tuple2 -> Tuple3
> env.from(tuples)
> .groupByKey(_.name)
> .map(o => {o.inc += 2;o})
> .filter(_.name != "b")
> .filter(_.name != "c")
> .groupByKey(o=>(o.name,o.value))
> .map(o => (o.name,o))
> .map(o => (o._1,o._2.value,o._2.inc))
> .foreach(println)
> env.execute()
> {code}
> Type is transparent for developer during both DAG compiling (programming) and
> runtime (metadata) phases
> {code}
> 2015-12-07 15:17:19,820 INFO [main] utils.GraphPrinter$[43]: Before expanded
> DAG
> {
> IterableStreamProducer[Entity]_1{1} ->
> GroupByKeyProducer[(Entity)]_2{1} in shuffle
[GitHub] incubator-eagle pull request: EAGLE-66. Typesafe Streaming DSL and...
Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/26#issuecomment-164410901 Looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request: EAGLE 86: Rewrite HDFSAuditLogParser...
Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/29#issuecomment-164410161 Looks good to me +2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055828#comment-15055828 ] Hao Chen commented on EAGLE-74: --- [~qingwzhao] is the issue resolved? > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE 86: Rewrite HDFSAuditLogParser...
GitHub user sunlibin opened a pull request: https://github.com/apache/incubator-eagle/pull/29 EAGLE 86: Rewrite HDFSAuditLogParser to a non-regular expression version You can merge this pull request into a Git repository by running: $ git pull https://github.com/sunlibin/incubator-eagle EAGLE-86 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-eagle/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit 60f9132f5c08db0e4c9f52df4b920b625ee26903 Author: sunlibin Date: 2015-12-14T10:56:52Z rewrite HDFSAuditLogParser to a non-regular expression version --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055826#comment-15055826 ] ASF GitHub Bot commented on EAGLE-74: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/20 > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: [EAGLE-74] Delete unused doc
Github user asfgit closed the pull request at: https://github.com/apache/incubator-eagle/pull/20 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (EAGLE-86) Rewrite HDFSAuditLogParser to a non-regular expression version
[ https://issues.apache.org/jira/browse/EAGLE-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Libin, Sun updated EAGLE-86: Affects Version/s: 0.3.0 > Rewrite HDFSAuditLogParser to a non-regular expression version > -- > > Key: EAGLE-86 > URL: https://issues.apache.org/jira/browse/EAGLE-86 > Project: Eagle > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Libin, Sun >Assignee: Libin, Sun >Priority: Minor > > HDFSAuditLogParser use many regular expressions to parse audit log, which is > a bit heavy, to improve topology's throughput, consider to rewrite it to a > non-regular expression version -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE 75: Integrate dropwizard metri...
Github user RalphSu commented on the pull request: https://github.com/apache/incubator-eagle/pull/28#issuecomment-164408458 Applied patch, runs ok for me. Would like to another reviewer. for the project name, 'eagle-security-metric-collection' seems better before we make it general. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-74) in sandbox setup, change package name from eagle. to org.apache.eagle
[ https://issues.apache.org/jira/browse/EAGLE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055822#comment-15055822 ] ASF GitHub Bot commented on EAGLE-74: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/20#issuecomment-164408273 LGTM. > in sandbox setup, change package name from eagle. to org.apache.eagle > - > > Key: EAGLE-74 > URL: https://issues.apache.org/jira/browse/EAGLE-74 > Project: Eagle > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Zhao, Qingwen > > http://eagle.incubator.apache.org/docs/deployment-in-sandbox.html > log4j.appender.KAFKA_HDFS_AUDIT=eagle.log4j.kafka.KafkaLog4jAppender should > be renamed package to prefixed by org.apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE-80: remove httpd directory
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/21#issuecomment-164408208 Why remove httpd directory? Didn't it cause ganglia related problem? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request: [EAGLE-74] Delete unused doc
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/20#issuecomment-164408273 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-80) remove httpd directory
[ https://issues.apache.org/jira/browse/EAGLE-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055821#comment-15055821 ] ASF GitHub Bot commented on EAGLE-80: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/21#issuecomment-164408208 Why remove httpd directory? Didn't it cause ganglia related problem? > remove httpd directory > -- > > Key: EAGLE-80 > URL: https://issues.apache.org/jira/browse/EAGLE-80 > Project: Eagle > Issue Type: Improvement >Reporter: Zhaokun Qin > Fix For: 0.3.0 > > > remove empty and useless httpd directory in docker image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (EAGLE-86) Rewrite HDFSAuditLogParser to a non-regular expression version
Libin, Sun created EAGLE-86: --- Summary: Rewrite HDFSAuditLogParser to a non-regular expression version Key: EAGLE-86 URL: https://issues.apache.org/jira/browse/EAGLE-86 Project: Eagle Issue Type: Improvement Reporter: Libin, Sun Assignee: Libin, Sun Priority: Minor HDFSAuditLogParser use many regular expressions to parse audit log, which is a bit heavy, to improve topology's throughput, consider to rewrite it to a non-regular expression version -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-eagle pull request: EAGLE 75: Integrate dropwizard metri...
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/28#issuecomment-164406314 The pull request title is bad. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request: EAGLE 75: Integrate dropwizard metri...
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/28#issuecomment-164405519 https://cwiki.apache.org/confluence/display/EAG --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request: EAGLE 75: Integrate dropwizard metri...
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/28#issuecomment-164405367 Could you please help list a metric list in eagle wiki before we merge this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-eagle pull request: EAGLE-85 : Fix unit test failure.
Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164404890 There are some sensitive table/field name in the unit test case sql, please rename/remove. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (EAGLE-85) Fix unit test failure.
[ https://issues.apache.org/jira/browse/EAGLE-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055793#comment-15055793 ] ASF GitHub Bot commented on EAGLE-85: - Github user haoch commented on the pull request: https://github.com/apache/incubator-eagle/pull/27#issuecomment-164404890 There are some sensitive table/field name in the unit test case sql, please rename/remove. > Fix unit test failure. > -- > > Key: EAGLE-85 > URL: https://issues.apache.org/jira/browse/EAGLE-85 > Project: Eagle > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Su Ralph >Assignee: Su Ralph > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
