+1 (binding) --Chris Nauroth
On 12/3/15, 9:33 AM, "Owen O'Malley" <omal...@apache.org> wrote: >The [DISCUSS] thread has would down, so I'd like to start a VOTE on >whether >Apache Incubator should accept Metron as a podling. The proposal is pasted >below and is available on the wiki as well. > >https://wiki.apache.org/incubator/MetronProposal > >We've added a paragraph in the background section discussing how Apache >avoids hostile forks of projects, because we don't want to fork >communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip >Rhodes to the proposal. > >The vote will run until 12pm PST on Sunday. > >Thanks, > Owen > >= Apache Metron Proposal = > >---- >/!\ '''FINAL''' /!\ > >This proposal is now complete and has been submitted for a VOTE. >---- > >== Abstract == > >The Metron project is an open source project dedicated to providing an >extensible and scalable advanced security analytics tool. It has strong >foundations in the Apache Hadoop ecosystem. > >== Proposal == > >Metron integrates a variety of open source big data technologies in order >to offer a centralized tool for security monitoring and analysis. Metron >provides capabilities for log aggregation, full packet capture indexing, >storage, advanced behavioral analytics and data enrichment, while applying >the most current threat-intelligence information to security telemetry >within a single platform. > >Metron can be divided into 4 areas: > > 1. '''A mechanism to capture, store, and normalize any type of security >telemetry at extremely high rates.''' Because security telemetry is >constantly being generated, it requires a method for ingesting the data at >high speeds and pushing it to various processing units for advanced >computation and analytics. > 1. '''Real time processing and application of enrichments''' such as >threat intelligence, geolocation, and DNS information to telemetry being >collected. The immediate application of this information to incoming >telemetry provides the context and situational awareness, as well as the >³who² and ³where² information that is critical for investigation. > 1. '''Efficient information storage''' based on how the information will >be used: > a. Logs and telemetry are stored such that they can be efficiently >mined and analyzed for concise security visibility > a. The ability to extract and reconstruct full packets helps an >analyst >answer questions such as who the true attacker was, what data was leaked, >and where that data was sent > a. Long-term storage not only increases visibility over time, but also >enables advanced analytics such as machine learning techniques to be used >to create models on the information. Incoming data can then be scored >against these stored models for advanced anomaly detection. > 1. '''An interface that gives a security investigator a centralized view >of data and alerts passed through the system.''' Metron¹s interface >presents alert summaries with threat intelligence and enrichment data >specific to that alert on one single page. Furthermore, advanced search >capabilities and full packet extraction tools are presented to the analyst >for investigation without the need to pivot into additional tools. > >Big data is a natural fit for powerful security analytics. The Metron >framework integrates a number of elements from the Hadoop ecosystem to >provide a scalable platform for security analytics, incorporating such >functionality as full-packet capture, stream processing, batch processing, >real-time search, and telemetry aggregation. With Metron, our goal is to >tie big data into security analytics and drive towards an extensible >centralized platform to effectively enable rapid detection and rapid >response for advanced security threats. > >== Background == > >OpenSOC was developed by Cisco over the last two years and pushed out to >Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the >development was mostly closed and has largely stopped. As evidence of the >inactivity, users have complained that pull requests are not answered for >a >while >https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ. >Finally, no public releases of OpenSOC have been made. From an Apache >point >of view, the current community is not viable. > >However, some of the developers of the project have left Cisco and have >found interest from several others that would like to work together to >form >an active and open community at Apache starting from the current OpenSOC >code base. A message to the current support group proposing moving to >Apache got a single positive response. >https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ > >In general Apache accepts only voluntary contributions and avoids >hostile forks. In this case, given that the community is demonstrably >dead, it seems fair to fork the existing code at Apache to allow a new >community to work on it. Once incubation starts, we will send a >message pointing to the new home to the OpenSOC support group. > >Because Cisco is not currently interested in being involved, the project >expects to change their name. The project would like to use Metron, >although we will perform a podling name search to check for conflicts. >Metron, meaning measure, is half of the greek root for the word >'telemetry.' Metron is also a DC Comics character who ³... wanders in >search of greater knowledge beyond his own². > > >== Rationale == >Metron strives to move the state of the art in security analytics forward. >We want to move away from the proprietary nature of legacy security point >tools and develop an open platform where people can contribute and share >datasets, machine learning models, telemetry parsers, sources of telemetry >enrichment, and threat intelligence feeds. Cyber security is too large of >a problem for a single corporation to tackle on its own and the current >tooling is too fragmented and proprietary for us to be able to rally >around >a single tool or vendor. > >In addition to being open and facilitating advancement in security >analytics, Metron has several advantages over a conventional Security >Information Management System (SIEM). > > * Metron uses all open source stack under the hood and runs on commodity >hardware. This means Metron is much cheaper to run then the competition. >In security cost plays a major factor because the cost of your >countermeasure for monitoring and reacting to a threat should not exceed >the cost of what is being protected. By driving down the cost of security >the economics works for more assets to be monitored, which means more >secure data centers. > * Metron, being in the open, allows additional vetting and scrutiny by >the open source community for all of its components. This is a better >model for a security-oriented tool than doing it closed source. All the >problems should be flushed out and fixed in the open. The closed source >competition does not have this kind of rigor, is motivated by marketing >and >sales, and thus, does not inspire confidence when it comes to security. > * Being Hadoop-based, Metron can process unprecedented volumes of >streaming data via Apache Storm. When an organization is hit with malware >or malicious behavior most commonly this happens as a part of a global >malware campaign, signatures for which are known and are available from >third party threat intelligence feeds. Having the ability to take in all >the feeds and reference them against every telemetry message processed by >Metron in real time does not only facilitate detection of such campaigns, >it changes the economics for the ³bad guys². If you have to customize >your >malware for each of your targets these global attacks become a lot more >expensive and non viable for them. > * Metron strives to shift conventional SOC workflows away from being >rules-driven to a more data-driven approach that incorporates machine >learning and a higher degree of automation and autonomous detection. The >modern threat landscape is too dynamic to be manageable via static rules >alone, which is what conventional SIEMs rely on. Rule bases tend to >bloat, >and if improperly maintained turn themselves into sources of false >positive >alerts. > >The ability to analyze and model large volumes of data at rest and then >being able to push up the output of that into a stream processor is >essential in disrupting the > >== Current Status == > >As stated in the background section, the current community isn¹t healthy, >which is why we are proposing moving to Apache Incubator. In this section, >we will describe the current state of the OpenSOC project. > >=== Meritocracy === >The OpenSOC development is controlled by Cisco and pull requests are being >ignored. The development list is private and requests to join are rejected >because there is no activity on it. The goal of moving to Apache is to >form >a meritocracy where a variety of individuals, regardless of their current >employer, come together and work together. We understand that diversity, >open development, and open governance are critical to being a successful >Apache project. > >=== Community === >The OpenSOC project is not responding to pull requests or making releases. >The easiest solution would be to create a variety of forks of the project >on github, but that would further fracture the community and prevent it >from reaching critical mass. Our prefered solution is to build a single >large diverse and open community at Apache. > >=== Core Developers === >The core developers of Metron are James Sirota, Charles Porter, and Mark >Bittmann. None of them have experience running an open source project, but >they are eager to learn. > >=== Alignment === >The ASF is a natural host for Metron given that it is already the home of >Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data >projects. Metron leverages many of Apache open-source products. We are >very >interested in a place to develop our community and integrations with the >other Apache big data projects. > >== Known Risks == > >=== Orphaned Products === > >The current product developers are all salaried developers at a small >number of companies and thus there is a risk of becoming an orphaned >product. However, the companies view Metron as very important to their >product offering and plan to ramp up their work in the space. The project >is unique in the product space and thus has strong potential to become a >sustainable community. > >=== Inexperience with Open Source === >The vast majority of the developers are inexperienced with open source >development and the Apache Way. One of the major hurdles to graduation >from >the Apache Incubator will be demonstrating that they have learned the >Apache Way and are applying it to how the project is managed. Vinod Kumar >Vavilapalli is an Apache Member and plans on actively working as a >committer in the project. They also have the other mentors to help them >learn as they progress. > >=== Homogenous Developers === >The developers are employed by four diverse companies (B23, Hortonworks, >Mantech, and Rackspace), They are distributed across the United States. We >hope to attract additional diversity as an Apache project. > >=== Reliance on Salaried Developers === >Metron is currently being developed exclusively by salaried developers, >but >the goal of coming to Apache is to form a community of users and >developers >that is much more diverse including non-salaried developers. > >=== Relationships with Other Apache Products === >Metron has a strong relationship and dependency with Apache Flume, Hadoop, >HBase, Hive, Kafka, Spark, and Storm. Being part of Apache¹s Incubation >community could help with a closer collaboration among these projects and >as well as others. > >We note that although there is a superficial resemblance to Apache Eagle, >which does security analysis of Hadoop audit events, the projects are >significantly different. In particular, Metron is focused on analyzing >network packet traffic and thus has a very different scope and scale of >events than Eagle. > >=== An Excessive Fascination with the Apache Brand === > >While the Apache brand is important, we are much more interested in >finding >a home for the project that encourages open development and open >governance. We want to form the new community using the Apache Way with >its >strong focus on meritocracy, organizational independence, and open >development. > >== Documentation == >The current information on the OpenSOC project is here: >http://opensoc.github.io/ >A slide deck presenting background material is here: >http://www.slideshare.net/JamesSirota/cisco-opensoc > >== Initial Source == >The initial code is on github: http://opensoc.github.io/ > >== External Dependencies == >Metron has the following external dependencies: > * Apache Flume > * Apache Hadoop > * Apache HBase > * Apache Hive > * Apache Kafka > * Apache Spark > * Apache Storm > * ElasticSearch > * MySQL > >The project understands that it will need to support alternatives for >MySQL >that are licensed under a ALv2 compatible license. > >== Cryptography == >Metron will eventually support encryption on the wire, but this is not one >of the initial goals, and we do not expect Metron to be a controlled >export >item due to the use of encryption. Metron supports but does not require >the >Kerberos authentication mechanism to access secured Hadoop services. > >== Required Resources == > >=== Mailing List === > > * metron-private for private PMC discussions > * metron-dev for developers > * metron-commits for all commits > * metron-users for all users > >=== Version Control === >Git is the preferred source control system. > >=== Issue Tracking === > > * JIRA (METRON) > >=== Other Resources === >The existing code already has unit tests so we will make use of existing >Apache continuous testing infrastructure. The resulting load should not be >very large. > >== Initial Committers == > * Jim Baker < jim.baker at rackspace dot com > > * Mark Bittmann < mark at b23 dot io > > * Sheetal Dolas < sheetal at hortonworks dot com > > * Discovery Gerdes < discovery.gerdes at rackspace dot com > > * P. Taylor Goetz < ptgoetz at apache dot org > > * Andrew Hartnett < andrew.hartnett at rackspace dot com > > * Dave Hirko < dave at b23 dot io > > * Paul Kehrer < paul.kehrer at rackspace dot com > > * Brad Kolarov < brad at b23 dot io > > * Kiran Komaravolu <kkomaravolu at hortonworks dot com > > * Larry McCay < lmccay at appache.org > > * Ryan Merriman < rmerriman at hortonworks dot com > > * Michael Perez < michael.perez at hortonworks dot com> > * Charles Porter < Charles.Porter at mcs dot mantech dot com > > * Phillip Rhodes < motley.crue.fan at gmail dot com > > * Sean Schulte < sean.schulte at rackspace dot com > > * James Sirota < jsirota at hortonworks dot com > > * Casey Stella < cstella at hortonworks dot com > > * Bryan Taylor < bryan.taylor at rackspace dot com > > * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com > > * Vinod Kumar Vavilapalli < vinodkv at apache dot org > > * George Vetticaden < gvetticaden at hortonworks dot com > > * Oskar Zabik < oskar.zabik at rackspace dot com > > >== Affiliations == >The initial committers are employees of: > * Jim Baker - Rackspace > * Mark Bittmann - B23 > * Sheetal Dolas - Hortonworks > * Discovery Gerdes - Rackspace > * P. Taylor Goetz - Hortonworks > * Andrew Hartnett - Rackspace > * Dave Hirko - B23 > * Paul Kehrer - Rackspace > * Brad Kolarov - B23 > * Kiran Komaravolu - Hortonworks > * Larry McCay - Hortonworks > * Ryan Merriman - Hortonworks > * Michael Perez - Hortonworks > * Charles Porter - Mantech > * Phillip Rhodes - Fogbeam Labs > * Sean Schulte - Rackspace > * James Sirota - Hortonworks > * Casey Stella - Hortonworks > * Bryan Taylor - Rackspace > * Ray Urciuoli - Mantech > * Vinod Kumar Vavilapalli - Hortonworks > * George Vetticaden - Hortonworks > * Oskar Zabik - Rackspace > >== Sponsors == > >=== Champion === > * Owen O¹Malley - Apache IPMC member > >=== Nominated Mentors === > * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member, >Hortonworks > * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member, >NASA > * Owen O¹Malley < omalley at apache dot org > - Apache IPMC member, >Hortonworks > * Billie Rinaldi < billie at apache dot org > - Apache IPMC member, >Hortonworks > * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC >member, Hortonworks > >=== Sponsoring Entity === >We are requesting the Incubator to sponsor this project. --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
