We’re also replacing much of the functionality of Metron with a series of Apache Flink based streaming components in a number of installations. It makes for a composable approach, and borrows from elements of the Metron architecture, while using more efficient formats like Apache Avro instead of JSON to reduce Kafka consumption and increase performance. Using Flink also allows for more efficient aggregation and sql based rules.
It’s much more of a custom solution than the generic project Metron took on, but seems to be working well for many of the same log related use cases. Simon On Wed, 3 Feb 2021 at 00:08, Alex Scammon <a...@alexscammon.com> wrote: > Hey there Jack, > > We were also disappointed that Metron was shuttered. But we've seen it as > an opportunity to continue an internal project which builds on top of > Metron. Our goal is to make our project open source as a potential > successor to Metron. > > We're maybe a month or two away from making it public, but we'd love some > eyes on it before we take that step if you're interested in taking a look. > > Since it builds on Metron, a lot of the core architecture remains > familiar. Java, Kafka, Storm, etc.... Hopefully, that presents a familiar > ecosystem for folks who are currently using Metron. For improvements, we > focused on: > > - Ensuring that simple configuration mistakes don't bring down the > whole pipeline > - A git-based approval workflow for rules updates (approvals and an > audit trail are important for us) > - An improved, modern-looking UI in Angular > - Easier installation steps > > Let me know if you're interested in discussing more -- I'd be interested > to hear whether there are any particulars about the models you're running > that we should take into consideration. > > Cheers, > > Alex Scammon > Head of Open Source Development > G-Research > gresearch.co.uk > > > On Tue, Feb 2, 2021 at 1:18 AM Jack Roberts <jrobe...@turing.ac.uk> wrote: > >> Hi Metron community, >> >> >> >> I recently started to explore Metron as part of a research project I’m >> involved in, but I’ve just seen the unfortunate news that the project is >> being “moved to the Attic”. I’d be very grateful if anyone could help to >> clarify the following: >> >> >> >> - Is there likely to be any continued development of Metron outside >> of Apache? >> - Are there any alternatives to Metron that people in this community >> would recommend? In particular, we’re looking for something open source >> that we can deploy ourselves, and with the functionality to >> straightforwardly integrate our own machine learning models for anomaly >> detection/similar. >> >> >> >> Many thanks and best wishes, >> >> Jack >> The Alan Turing Institute is a limited liability company, registered in >> England with registered number 09512457 with registered offices at British >> Library, 96 Euston Road, London, England, NW1 2DB >> <https://www.google.com/maps/search/96+Euston+Road,+London,+England,+NW1+2DB?entry=gmail&source=g>. >> We are also a charity registered in England with charity number 1162533. >> DISCLAIMER: Although we have taken reasonable precautions to ensure the >> completeness and accuracy of this e-mail, transmission cannot be guaranteed >> to be secure or error-free as information could be intercepted, corrupted, >> lost, destroyed, arrive late or be incomplete. If you receive a suspicious >> or unexpected email from us, or purporting to have been sent on our behalf, >> particularly containing different bank details, please do not reply to the >> email, click on any links, open any attachments, or comply with any >> instructions contained within it. Our Transparency Notice found here - >> https://www.turing.ac.uk/transparency-notice sets out how and why we >> collect, store, use and share your personal data and it explains your >> rights and how to raise concerns with us. >> > -- -- simon elliston ball @sireb
