Re: [DISCUSS] SystemML Incubator Proposal

[Henry Saputra]

Thanks Luciano, I got my answer but would probably helped to
distinguish option to run it as Apache Hadoop MapReduce or YARN
application, and with abstraction of Apache Spark.

Looking forward possibility of having it run with Apache Flink :)

- Henry

On Sat, Oct 24, 2015 at 12:32 PM, Luciano Resende  wrote:
> On Sat, Oct 24, 2015 at 11:31 AM, Henry Saputra 
> wrote:
>
>> I have one question about the proposal, it keep mentioning that it
>> could run on "Hadoop or Spark", but technically Spark can run on
>> Hadoop YARN.
>> Was it trying to say it could be run in Hadoop YARN (maybe via
>> MapReduce) or Spark?
>>
>>
> Exactly, if this is a point of confusion i can clarify it on the proposal.
>
>
>> I would love to see if the execution abstraction is well enough
>> defined to be able to run it on the others distributed framework like
>> Flink or Tez  (maybe via Crunch?)
>>
>>
> Yes, this is definitely a possibility, we have talked about Flink before as
> a possible next runtime.
>
>
>> Thanks,
>>
>> Henry
>>
>
>
> --
> Luciano Resende
> http://people.apache.org/~lresende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

request to join ipmc

[Reynold Xin]

Hi,

I am an Apache member and would like to join the IPMC. I'm a Spark
committer & PMC member, and have also contributed to various other projects
including Hive, Hadoop, etc. Let me know what else you need from me.

Thanks!

Re: Request to be added to Incubator PMC

[Julien Le Dem]

Thanks for the explanation.
I'd like to candidate as well.
I'm the Apache Parquet PMC chair and I'm on the Apache Pig PMC.
I've been through the incubator with Parquet recently and would like to
contribute.
Julien

On Mon, Oct 26, 2015 at 9:09 AM, Marvin Humphrey 
wrote:

> On Sun, Oct 25, 2015 at 4:11 PM, Patrick Wendell 
> wrote:
>
> > I would like to be added to the Incubator PMC to help mentor a new
> project.
> > I am an Apache Member. I am not sure the exact process to be added, so I
> am
> > emailing this list as a first step!
>
> Hello Patrick, and also Reynold and Jaques as they have made similar
> requests on other threads...
>
> It's great to have you and we all look forward to your participating
> in incubation projects in a new capacity!
>
> It's the standard procedure for adding a PMC member from here on out:
> the Incubator's PMC Chair sends a NOTICE to the Board, we wait for 72
> hours to ensure that no Board member objects to the addition, then the
> Chair finishes the clerical work. From the standpoint of the
> corporation, you officially become PMC members (with valid voting
> privileges, etc.) when committee-info.txt is updated.
>
>   http://www.apache.org/dev/pmc.html#newpmc
>
> The only difference in the process for the Incubator is that while
> non-ASF-Member candidates for the IPMC must be voted on as with all
> other PMCs, ASF Members may sidestep the vote. Member requests usually
> go to the private list because that's where personnel changes are
> traditionally discussed at Apache and there is a (very) remote
> possibility that the Board might reject a candidate. But of course
> it's fine to receive your requests on general@incubator.
>
> Finally, lest this explanation sow confusion with regards to adding
> new *PPMC* members, the process is slightly different -- the IPMC gets
> notified instead of the Board.
>   http://incubator.apache.org/guides/ppmc.html#Voting+in+a+new+PPMC+member
>
> Good luck and let us know if you have questions!
>
> Marvin Humphrey
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>


-- 
Julien

Re: [Result] Accept Eagle into Apache Incubation

[Ted Dunning]

As a clarification, all votes were +1



On Mon, Oct 26, 2015 at 10:59 AM, Manoharan, Arun 
wrote:

> Hello Everyone,
>
> Thanks for participating in the vote and discussions about Eagle.
>
> Binding votes = 10
> Non-binding votes = 14
> Total votes = 24
>
> Thanks,
>
> Arun
>
>
> On 10/25/15, 8:37 PM, "Don Bosco Durai"  wrote:
>
> >+1 non binding
> >Bosco
> >
> >
> >
> >_
> >From: Li Yang 
> >Sent: Sunday, October 25, 2015 8:13 PM
> >Subject: Re: [VOTE] Accept Eagle into Apache Incubation
> >To:  
> >
> >
> >+1 (non-binding)
> >
> >On Mon, Oct 26, 2015 at 10:50 AM, hongbin ma 
> wrote:
> >
> >> +1 (non binding)
> >>
> >> On Mon, Oct 26, 2015 at 12:20 AM, Ralph Goers
> >>
> >> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > Ralph
> >> >
> >> > > On Oct 23, 2015, at 7:11 AM, Manoharan, Arun 
> >> > wrote:
> >> > >
> >> > > Hello Everyone,
> >> > >
> >> > > Thanks for all the feedback on the Eagle Proposal.
> >> > >
> >> > > I would like to call for a [VOTE] on Eagle joining the ASF as an
> >> > incubation project.
> >> > >
> >> > > The vote is open for 72 hours:
> >> > >
> >> > > [ ] +1 accept Eagle in the Incubator
> >> > > [ ] ±0
> >> > > [ ] -1 (please give reason)
> >> > >
> >> > > Eagle is a Monitoring solution for Hadoop to instantly identify
> >>access
> >> > to sensitive data, recognize attacks, malicious activities and take
> >> actions
> >> > in real time. Eagle supports a wide variety of policies on HDFS data
> >>and
> >> > Hive. Eagle also provides machine learning models for detecting
> >>anomalous
> >> > user behavior in Hadoop.
> >> > >
> >> > > The proposal is available on the wiki here:
> >> > > https://wiki.apache.org/incubator/EagleProposal
> >> > >
> >> > > The text of the proposal is also available at the end of this email.
> >> > >
> >> > > Thanks for your time and help.
> >> > >
> >> > > Thanks,
> >> > > Arun
> >> > >
> >> > > 
> >> > >
> >> > > Eagle
> >> > >
> >> > > Abstract
> >> > > Eagle is an Open Source Monitoring solution for Hadoop to instantly
> >> > identify access to sensitive data, recognize attacks, malicious
> >> activities
> >> > in hadoop and take actions.
> >> > >
> >> > > Proposal
> >> > > Eagle audits access to HDFS files, Hive and HBase tables in real
> >>time,
> >> > enforces policies defined on sensitive data access and alerts or
> >>blocks
> >> > user’s access to that sensitive data in real time. Eagle also creates
> >> user
> >> > profiles based on the typical access behaviour for HDFS and Hive and
> >> sends
> >> > alerts when anomalous behaviour is detected. Eagle can also import
> >> > sensitive data information classified by external classification
> >>engines
> >> to
> >> > help define its policies.
> >> > >
> >> > > Overview of Eagle
> >> > > Eagle has 3 main parts.
> >> > > 1.Data collection and storage - Eagle collects data from various
> >>hadoop
> >> > logs in real time using Kafka/Yarn API and uses HDFS and HBase for
> >> storage.
> >> > > 2.Data processing and policy engine - Eagle allows users to create
> >> > policies based on various metadata properties on HDFS, Hive and HBase
> >> data.
> >> > > 3.Eagle services - Eagle services include policy manager, query
> >>service
> >> > and the visualization component. Eagle provides intuitive user
> >>interface
> >> to
> >> > administer Eagle and an alert dashboard to respond to real time
> >>alerts.
> >> > >
> >> > > Data Collection and Storage:
> >> > > Eagle provides programming API for extending Eagle to integrate any
> >> data
> >> > source into Eagle policy evaluation framework. For example, Eagle hdfs
> >> > audit monitoring collects data from Kafka which is populated from
> >> namenode
> >> > log4j appender or from logstash agent. Eagle hive monitoring collects
> >> hive
> >> > query logs from running job through YARN API, which is designed to be
> >> > scalable and fault-tolerant. Eagle uses HBase as storage for storing
> >> > metadata and metrics data, and also supports relational database
> >>through
> >> > configuration change.
> >> > >
> >> > > Data Processing and Policy Engine:
> >> > > Processing Engine: Eagle provides stream processing API which is an
> >> > abstraction of Apache Storm. It can also be extended to other
> >>streaming
> >> > engines. This abstraction allows developers to assemble data
> >> > transformation, filtering, external data join etc. without physically
> >> bound
> >> > to a specific streaming platform. Eagle streaming API allows
> >>developers
> >> to
> >> > easily integrate business logic with Eagle policy engine and
> >>internally
> >> > Eagle framework compiles business logic execution DAG into program
> >> > primitives of underlying stream infrastructure e.g. Apache Storm. For
> >> > example, Eagle HDFS monitoring transforms audit log from Namenode to
> >> object
> >> > 

[ANNOUNCE] Apache Kylin 1.1-incubating released

[ShaoFeng Shi]

The Apache Kylin team is pleased to announce the immediate availability
of the 1.1 release. The release note can be found here [1]; The source
code and binary package can be downloaded from Kylin's download page [2].

Some of the highlights of this release:

[KYLIN-740] - Fix the slowness with many IN() values [3]
[KYLIN-910] - Allow user to configure "retention range" for Cube [4]
[KYLIN-956] - Allow user to configure HBase compression algorithm [5]
[KYLIN-957] - Support HBase in a separate cluster [6]
[KYLIN-1047] - Upgrade to Calcite 1.4 [7]
[KYLIN-1048] - Fix the CPU and memory killer in Cuboid.findById() [8]

The Apache Kylin Team would like to hear from you and welcomes
your comments and contributions.

Thanks,
The Apache Kylin Team

[1] https://kylin.incubator.apache.org/docs/release_notes.html
[2] https://kylin.incubator.apache.org/download/
[3] https://issues.apache.org/jira/browse/KYLIN-740
[4] https://issues.apache.org/jira/browse/KYLIN-910
[5] https://issues.apache.org/jira/browse/KYLIN-956
[6] https://issues.apache.org/jira/browse/KYLIN-957
[7] https://issues.apache.org/jira/browse/KYLIN-1047
[8] https://issues.apache.org/jira/browse/KYLIN-1048

[VOTE] Release Apache Slider 0.81.1-incubating

[Jon Maron]

Hello,

This is a call for a vote for releasing Apache Slider 0.81.1-incubating.

This is a source release.

Summary of fixes: http://s.apache.org/sgG
Release Notes:  http://s.apache.org/ZZP
Vote thread: http://s.apache.org/Ejv
Results: http://s.apache.org/ufJ

Staged artifacts:
https://repository.apache.org/content/repositories/orgapacheslider-1008/org/apache/slider

Git source:
https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;a=commit;h=a9b1d659c642ab7eb7b21ecbae97f805259c9f9f
SHA1: a9b1d659c642ab7eb7b21ecbae97f805259c9f9f 
Tag: slider-0.81.1-incubating

PGP key:
http://pgp.mit.edu:11371/pks/lookup?op=vindex=jma...@apache.org

Basic build/test instructions:
http://slider.incubator.apache.org/developing/building.html

Please vote on releasing this package as Apache Slider 0.81.1-incubating.

This vote will be open for 72 hours.

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thank You,
The Apache Slider Team

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [VOTE] TinkerPop 3.0.2-incubating Release

[Rich Bowen]

+1, binding, mentor
On Oct 23, 2015 9:07 PM, "Stephen Mallette"  wrote:

> Hello,
> We are happy to announce that TinkerPop 3.0.2-incubating is ready for
> release.
>
> The release artifacts can be found at this location:
>
> https://dist.apache.org/repos/dist/dev/incubator/tinkerpop/3.0.2-incubating/
>
> The source distribution is provided by:
> apache-tinkerpop-3.0.2-incubating-source-release.zip
>
> Two binary distributions are provided for user convenience:
> apache-gremlin-console-3.0.2-incubating-distribution.zip
> apache-gremlin-server-3.0.2-incubating-distribution.zip
>
> The GPG key used to sign the release artifacts is available at:
> https://dist.apache.org/repos/dist/dev/incubator/tinkerpop/KEYS
>
> The online docs can be found here:
> http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/ (user
> docs)
>
> http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/upgrade.html#_tinkerpop_3_0_2
> (upgrade docs)
> http://tinkerpop.incubator.apache.org/javadocs/3.0.2-incubating/core/
> (core javadoc)
> http://tinkerpop.incubator.apache.org/javadocs/3.0.2-incubating/full/
> (full javadoc)
>
> The tag in Apache Git can be found here:
>
> https://git-wip-us.apache.org/repos/asf?p=incubator-tinkerpop.git;a=tag;h=8e9af13d6beb184a137067caa0445157351435ab
>
> The release notes are available here:
>
> https://github.com/apache/incubator-tinkerpop/blob/3.0.2-incubating/CHANGELOG.asciidoc#tinkerpop-302-release-date-october-19-2015
>
> Finally, the dev@tinkerpop [VOTE] thread can be found at this location:
>
>
> http://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/201510.mbox/%3CCAA-H439qBNzu1gO7P0m%2BeUQ4OZyPU9Ya2D1icTEftd3fpzNvrA%40mail.gmail.com%3E
>
> Result summary: +14 (4 binding, 10 non-binding), 0 (0), -1 (0)
>
> The [VOTE] will be open for the next 72 hours --- closing Monday
> (October 26, 2015) at 8am EST.
>
> Thanks,
>
> Stephen
>
> P.S. Hopefully we were able to get LICENSE/NOTICE solid on this
> release.  Justin Mclean I hope you get a chance to verify and vote as
> you usually seem to do.
>

??????[VOTE] Accept Eagle into Apache Incubation

[????]

+1 (no binding)


Jiang Xu

--  --
??: Li Yang 
: 2015??10??26?? 11:14
??: general 
: Re: [VOTE] Accept Eagle into Apache Incubation



+1 (non-binding)

On Mon, Oct 26, 2015 at 10:50 AM, hongbin ma  wrote:

> +1 (non binding)
>
> On Mon, Oct 26, 2015 at 12:20 AM, Ralph Goers 
> wrote:
>
> > +1 (binding)
> >
> > Ralph
> >
> > > On Oct 23, 2015, at 7:11 AM, Manoharan, Arun 
> > wrote:
> > >
> > > Hello Everyone,
> > >
> > > Thanks for all the feedback on the Eagle Proposal.
> > >
> > > I would like to call for a [VOTE] on Eagle joining the ASF as an
> > incubation project.
> > >
> > > The vote is open for 72 hours:
> > >
> > > [ ] +1 accept Eagle in the Incubator
> > > [ ] ??0
> > > [ ] -1 (please give reason)
> > >
> > > Eagle is a Monitoring solution for Hadoop to instantly identify access
> > to sensitive data, recognize attacks, malicious activities and take
> actions
> > in real time. Eagle supports a wide variety of policies on HDFS data and
> > Hive. Eagle also provides machine learning models for detecting anomalous
> > user behavior in Hadoop.
> > >
> > > The proposal is available on the wiki here:
> > > https://wiki.apache.org/incubator/EagleProposal
> > >
> > > The text of the proposal is also available at the end of this email.
> > >
> > > Thanks for your time and help.
> > >
> > > Thanks,
> > > Arun
> > >
> > > 
> > >
> > > Eagle
> > >
> > > Abstract
> > > Eagle is an Open Source Monitoring solution for Hadoop to instantly
> > identify access to sensitive data, recognize attacks, malicious
> activities
> > in hadoop and take actions.
> > >
> > > Proposal
> > > Eagle audits access to HDFS files, Hive and HBase tables in real time,
> > enforces policies defined on sensitive data access and alerts or blocks
> > user??s access to that sensitive data in real time. Eagle also creates
> user
> > profiles based on the typical access behaviour for HDFS and Hive and
> sends
> > alerts when anomalous behaviour is detected. Eagle can also import
> > sensitive data information classified by external classification engines
> to
> > help define its policies.
> > >
> > > Overview of Eagle
> > > Eagle has 3 main parts.
> > > 1.Data collection and storage - Eagle collects data from various hadoop
> > logs in real time using Kafka/Yarn API and uses HDFS and HBase for
> storage.
> > > 2.Data processing and policy engine - Eagle allows users to create
> > policies based on various metadata properties on HDFS, Hive and HBase
> data.
> > > 3.Eagle services - Eagle services include policy manager, query service
> > and the visualization component. Eagle provides intuitive user interface
> to
> > administer Eagle and an alert dashboard to respond to real time alerts.
> > >
> > > Data Collection and Storage:
> > > Eagle provides programming API for extending Eagle to integrate any
> data
> > source into Eagle policy evaluation framework. For example, Eagle hdfs
> > audit monitoring collects data from Kafka which is populated from
> namenode
> > log4j appender or from logstash agent. Eagle hive monitoring collects
> hive
> > query logs from running job through YARN API, which is designed to be
> > scalable and fault-tolerant. Eagle uses HBase as storage for storing
> > metadata and metrics data, and also supports relational database through
> > configuration change.
> > >
> > > Data Processing and Policy Engine:
> > > Processing Engine: Eagle provides stream processing API which is an
> > abstraction of Apache Storm. It can also be extended to other streaming
> > engines. This abstraction allows developers to assemble data
> > transformation, filtering, external data join etc. without physically
> bound
> > to a specific streaming platform. Eagle streaming API allows developers
> to
> > easily integrate business logic with Eagle policy engine and internally
> > Eagle framework compiles business logic execution DAG into program
> > primitives of underlying stream infrastructure e.g. Apache Storm. For
> > example, Eagle HDFS monitoring transforms audit log from Namenode to
> object
> > and joins sensitivity metadata, security zone metadata which are
> generated
> > from external programs or configured by user. Eagle hive monitoring
> filters
> > running jobs to get hive query string and parses query string into object
> > and then joins sensitivity metadata.
> > > Alerting Framework: Eagle Alert Framework includes stream metadata API,
> > scalable policy engine framework, extensible policy engine framework.
> > Stream metadata API allows developers to declare event schema including
> > what attributes constitute an event, what is the type for each attribute,
> > and how to dynamically resolve attribute value in runtime when user
> > configures policy. Scalable policy engine framework allows policies to be
> > executed on different physical 

Re: request to join ipmc

[Luke Han]

Welcome Reynold:)


Best Regards!
-

Luke Han

On Mon, Oct 26, 2015 at 4:36 PM, Reynold Xin  wrote:

> Hi,
>
> I am an Apache member and would like to join the IPMC. I'm a Spark
> committer & PMC member, and have also contributed to various other projects
> including Hive, Hadoop, etc. Let me know what else you need from me.
>
> Thanks!
>

Request to join IPMC

[Jacques Nadeau]

I would like to join the IPMC. I'm PMC Chair of Drill, a PMC member for
Calcite and have also contributed to Parquet & HBase. Let me know what you
need.

thanks,
Jacques

Re: Request to be added to Incubator PMC

[Marvin Humphrey]

On Sun, Oct 25, 2015 at 4:11 PM, Patrick Wendell  wrote:

> I would like to be added to the Incubator PMC to help mentor a new project.
> I am an Apache Member. I am not sure the exact process to be added, so I am
> emailing this list as a first step!

Hello Patrick, and also Reynold and Jaques as they have made similar
requests on other threads...

It's great to have you and we all look forward to your participating
in incubation projects in a new capacity!

It's the standard procedure for adding a PMC member from here on out:
the Incubator's PMC Chair sends a NOTICE to the Board, we wait for 72
hours to ensure that no Board member objects to the addition, then the
Chair finishes the clerical work. From the standpoint of the
corporation, you officially become PMC members (with valid voting
privileges, etc.) when committee-info.txt is updated.

  http://www.apache.org/dev/pmc.html#newpmc

The only difference in the process for the Incubator is that while
non-ASF-Member candidates for the IPMC must be voted on as with all
other PMCs, ASF Members may sidestep the vote. Member requests usually
go to the private list because that's where personnel changes are
traditionally discussed at Apache and there is a (very) remote
possibility that the Board might reject a candidate. But of course
it's fine to receive your requests on general@incubator.

Finally, lest this explanation sow confusion with regards to adding
new *PPMC* members, the process is slightly different -- the IPMC gets
notified instead of the Board.
  http://incubator.apache.org/guides/ppmc.html#Voting+in+a+new+PPMC+member

Good luck and let us know if you have questions!

Marvin Humphrey

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

[Result] Accept Eagle into Apache Incubation

[Manoharan, Arun]

Hello Everyone,

Thanks for participating in the vote and discussions about Eagle.

Binding votes = 10
Non-binding votes = 14
Total votes = 24

Thanks,

Arun 


On 10/25/15, 8:37 PM, "Don Bosco Durai"  wrote:

>+1 non binding 
>Bosco 
>
>
>
>_
>From: Li Yang 
>Sent: Sunday, October 25, 2015 8:13 PM
>Subject: Re: [VOTE] Accept Eagle into Apache Incubation
>To:  
>
>
>+1 (non-binding)
>
>On Mon, Oct 26, 2015 at 10:50 AM, hongbin ma  wrote:
>
>> +1 (non binding)
>>
>> On Mon, Oct 26, 2015 at 12:20 AM, Ralph Goers
>>
>> wrote:
>>
>> > +1 (binding)
>> >
>> > Ralph
>> >
>> > > On Oct 23, 2015, at 7:11 AM, Manoharan, Arun 
>> > wrote:
>> > >
>> > > Hello Everyone,
>> > >
>> > > Thanks for all the feedback on the Eagle Proposal.
>> > >
>> > > I would like to call for a [VOTE] on Eagle joining the ASF as an
>> > incubation project.
>> > >
>> > > The vote is open for 72 hours:
>> > >
>> > > [ ] +1 accept Eagle in the Incubator
>> > > [ ] ±0
>> > > [ ] -1 (please give reason)
>> > >
>> > > Eagle is a Monitoring solution for Hadoop to instantly identify
>>access
>> > to sensitive data, recognize attacks, malicious activities and take
>> actions
>> > in real time. Eagle supports a wide variety of policies on HDFS data
>>and
>> > Hive. Eagle also provides machine learning models for detecting
>>anomalous
>> > user behavior in Hadoop.
>> > >
>> > > The proposal is available on the wiki here:
>> > > https://wiki.apache.org/incubator/EagleProposal
>> > >
>> > > The text of the proposal is also available at the end of this email.
>> > >
>> > > Thanks for your time and help.
>> > >
>> > > Thanks,
>> > > Arun
>> > >
>> > > 
>> > >
>> > > Eagle
>> > >
>> > > Abstract
>> > > Eagle is an Open Source Monitoring solution for Hadoop to instantly
>> > identify access to sensitive data, recognize attacks, malicious
>> activities
>> > in hadoop and take actions.
>> > >
>> > > Proposal
>> > > Eagle audits access to HDFS files, Hive and HBase tables in real
>>time,
>> > enforces policies defined on sensitive data access and alerts or
>>blocks
>> > user’s access to that sensitive data in real time. Eagle also creates
>> user
>> > profiles based on the typical access behaviour for HDFS and Hive and
>> sends
>> > alerts when anomalous behaviour is detected. Eagle can also import
>> > sensitive data information classified by external classification
>>engines
>> to
>> > help define its policies.
>> > >
>> > > Overview of Eagle
>> > > Eagle has 3 main parts.
>> > > 1.Data collection and storage - Eagle collects data from various
>>hadoop
>> > logs in real time using Kafka/Yarn API and uses HDFS and HBase for
>> storage.
>> > > 2.Data processing and policy engine - Eagle allows users to create
>> > policies based on various metadata properties on HDFS, Hive and HBase
>> data.
>> > > 3.Eagle services - Eagle services include policy manager, query
>>service
>> > and the visualization component. Eagle provides intuitive user
>>interface
>> to
>> > administer Eagle and an alert dashboard to respond to real time
>>alerts.
>> > >
>> > > Data Collection and Storage:
>> > > Eagle provides programming API for extending Eagle to integrate any
>> data
>> > source into Eagle policy evaluation framework. For example, Eagle hdfs
>> > audit monitoring collects data from Kafka which is populated from
>> namenode
>> > log4j appender or from logstash agent. Eagle hive monitoring collects
>> hive
>> > query logs from running job through YARN API, which is designed to be
>> > scalable and fault-tolerant. Eagle uses HBase as storage for storing
>> > metadata and metrics data, and also supports relational database
>>through
>> > configuration change.
>> > >
>> > > Data Processing and Policy Engine:
>> > > Processing Engine: Eagle provides stream processing API which is an
>> > abstraction of Apache Storm. It can also be extended to other
>>streaming
>> > engines. This abstraction allows developers to assemble data
>> > transformation, filtering, external data join etc. without physically
>> bound
>> > to a specific streaming platform. Eagle streaming API allows
>>developers
>> to
>> > easily integrate business logic with Eagle policy engine and
>>internally
>> > Eagle framework compiles business logic execution DAG into program
>> > primitives of underlying stream infrastructure e.g. Apache Storm. For
>> > example, Eagle HDFS monitoring transforms audit log from Namenode to
>> object
>> > and joins sensitivity metadata, security zone metadata which are
>> generated
>> > from external programs or configured by user. Eagle hive monitoring
>> filters
>> > running jobs to get hive query string and parses query string into
>>object
>> > and then joins sensitivity metadata.
>> > > Alerting Framework: Eagle Alert Framework includes stream metadata
>>API,
>> > scalable policy engine framework, extensible