Re: Metron's Future / Alternatives

[Simon Elliston Ball]

We’re also replacing much of the functionality of Metron with a series of
Apache Flink based streaming components in a number of installations. It
makes for a composable approach, and borrows from elements of the Metron
architecture, while using more efficient formats like Apache Avro instead
of JSON to reduce Kafka consumption and increase performance. Using Flink
also allows for more efficient aggregation and sql based rules.

It’s much more of a custom solution than the generic project Metron took
on, but seems to be working well for many of the same log related use cases.

Simon

On Wed, 3 Feb 2021 at 00:08, Alex Scammon  wrote:

> Hey there Jack,
>
> We were also disappointed that Metron was shuttered.  But we've seen it as
> an opportunity to continue an internal project which builds on top of
> Metron.  Our goal is to make our project open source as a potential
> successor to Metron.
>
> We're maybe a month or two away from making it public, but we'd love some
> eyes on it before we take that step if you're interested in taking a look.
>
> Since it builds on Metron, a lot of the core architecture remains
> familiar.  Java, Kafka, Storm, etc  Hopefully, that presents a familiar
> ecosystem for folks who are currently using Metron.  For improvements, we
> focused on:
>
>- Ensuring that simple configuration mistakes don't bring down the
>whole pipeline
>- A git-based approval workflow for rules updates (approvals and an
>audit trail are important for us)
>- An improved, modern-looking UI in Angular
>- Easier installation steps
>
> Let me know if you're interested in discussing more -- I'd be interested
> to hear whether there are any particulars about the models you're running
> that we should take into consideration.
>
> Cheers,
>
> Alex Scammon
> Head of Open Source Development
> G-Research
> gresearch.co.uk
>
>
> On Tue, Feb 2, 2021 at 1:18 AM Jack Roberts  wrote:
>
>> Hi Metron community,
>>
>>
>>
>> I recently started to explore Metron as part of a research project I’m
>> involved in, but I’ve just seen the unfortunate news that the project is
>> being “moved to the Attic”. I’d be very grateful if anyone could help to
>> clarify the following:
>>
>>
>>
>>- Is there likely to be any continued development of Metron outside
>>of Apache?
>>- Are there any alternatives to Metron that people in this community
>>would recommend? In particular, we’re looking for something open source
>>that we can deploy ourselves, and with the functionality to
>>straightforwardly integrate our own machine learning models for anomaly
>>detection/similar.
>>
>>
>>
>> Many thanks and best wishes,
>>
>> Jack
>> The Alan Turing Institute is a limited liability company, registered in
>> England with registered number 09512457 with registered offices at British
>> Library, 96 Euston Road, London, England, NW1 2DB
>> .
>> We are also a charity registered in England with charity number 1162533.
>> DISCLAIMER: Although we have taken reasonable precautions to ensure the
>> completeness and accuracy of this e-mail, transmission cannot be guaranteed
>> to be secure or error-free as information could be intercepted, corrupted,
>> lost, destroyed, arrive late or be incomplete. If you receive a suspicious
>> or unexpected email from us, or purporting to have been sent on our behalf,
>> particularly containing different bank details, please do not reply to the
>> email, click on any links, open any attachments, or comply with any
>> instructions contained within it. Our Transparency Notice found here -
>> https://www.turing.ac.uk/transparency-notice sets out how and why we
>> collect, store, use and share your personal data and it explains your
>> rights and how to raise concerns with us.
>>
> --
--
simon elliston ball
@sireb

Re: Metron's Future / Alternatives

[Alex Scammon]

Hey there Jack,

We were also disappointed that Metron was shuttered.  But we've seen it as
an opportunity to continue an internal project which builds on top of
Metron.  Our goal is to make our project open source as a potential
successor to Metron.

We're maybe a month or two away from making it public, but we'd love some
eyes on it before we take that step if you're interested in taking a look.

Since it builds on Metron, a lot of the core architecture remains
familiar.  Java, Kafka, Storm, etc  Hopefully, that presents a familiar
ecosystem for folks who are currently using Metron.  For improvements, we
focused on:

   - Ensuring that simple configuration mistakes don't bring down the whole
   pipeline
   - A git-based approval workflow for rules updates (approvals and an
   audit trail are important for us)
   - An improved, modern-looking UI in Angular
   - Easier installation steps

Let me know if you're interested in discussing more -- I'd be interested to
hear whether there are any particulars about the models you're running that
we should take into consideration.

Cheers,

Alex Scammon
Head of Open Source Development
G-Research
gresearch.co.uk


On Tue, Feb 2, 2021 at 1:18 AM Jack Roberts  wrote:

> Hi Metron community,
>
>
>
> I recently started to explore Metron as part of a research project I’m
> involved in, but I’ve just seen the unfortunate news that the project is
> being “moved to the Attic”. I’d be very grateful if anyone could help to
> clarify the following:
>
>
>
>- Is there likely to be any continued development of Metron outside of
>Apache?
>- Are there any alternatives to Metron that people in this community
>would recommend? In particular, we’re looking for something open source
>that we can deploy ourselves, and with the functionality to
>straightforwardly integrate our own machine learning models for anomaly
>detection/similar.
>
>
>
> Many thanks and best wishes,
>
> Jack
> The Alan Turing Institute is a limited liability company, registered in
> England with registered number 09512457 with registered offices at British
> Library, 96 Euston Road, London, England, NW1 2DB. We are also a charity
> registered in England with charity number 1162533. DISCLAIMER: Although we
> have taken reasonable precautions to ensure the completeness and accuracy
> of this e-mail, transmission cannot be guaranteed to be secure or
> error-free as information could be intercepted, corrupted, lost, destroyed,
> arrive late or be incomplete. If you receive a suspicious or unexpected
> email from us, or purporting to have been sent on our behalf, particularly
> containing different bank details, please do not reply to the email, click
> on any links, open any attachments, or comply with any instructions
> contained within it. Our Transparency Notice found here -
> https://www.turing.ac.uk/transparency-notice sets out how and why we
> collect, store, use and share your personal data and it explains your
> rights and how to raise concerns with us.
>

Metron's Future / Alternatives

[Jack Roberts]

Hi Metron community,

I recently started to explore Metron as part of a research project I’m involved 
in, but I’ve just seen the unfortunate news that the project is being “moved to 
the Attic”. I’d be very grateful if anyone could help to clarify the following:


  *   Is there likely to be any continued development of Metron outside of 
Apache?
  *   Are there any alternatives to Metron that people in this community would 
recommend? In particular, we’re looking for something open source that we can 
deploy ourselves, and with the functionality to straightforwardly integrate our 
own machine learning models for anomaly detection/similar.

Many thanks and best wishes,
Jack
The Alan Turing Institute is a limited liability company, registered in England 
with registered number 09512457 with registered offices at British Library, 96 
Euston Road, London, England, NW1 2DB. We are also a charity registered in 
England with charity number 1162533. DISCLAIMER: Although we have taken 
reasonable precautions to ensure the completeness and accuracy of this e-mail, 
transmission cannot be guaranteed to be secure or error-free as information 
could be intercepted, corrupted, lost, destroyed, arrive late or be incomplete. 
If you receive a suspicious or unexpected email from us, or purporting to have 
been sent on our behalf, particularly containing different bank details, please 
do not reply to the email, click on any links, open any attachments, or comply 
with any instructions contained within it. Our Transparency Notice found here - 
https://www.turing.ac.uk/transparency-notice sets out how and why we collect, 
store, use and share your personal data and it explains your rights and how to 
raise concerns with us.