Re: How to delete the original message field once the message parsed?

2018-06-26 Thread Michel Sumbul
Its done: https://issues.apache.org/jira/browse/METRON-1640 I hope that I filled it correctly^^ 2018-06-26 22:17 GMT+01:00 Otto Fowler : > You are welcome to create a jira with your request and the reasoning > behind it. > https://issues.apache.org/jira/projects/METRON/issues/METRON-1588?filte

Flush profiler based on condition and not based time

2018-06-26 Thread Michel Sumbul
Hello Metron Guru, I would like to know if there's way to have a profiler that will flush the result to hbase not based on a time period but on some conditions. For example, if we are tracking a specific sequence over the time for an ip/user like (event A, event b and event C). If this entire seq

Re: How to delete the original message field once the message parsed?

2018-06-26 Thread Otto Fowler
You are welcome to create a jira with your request and the reasoning behind it. https://issues.apache.org/jira/projects/METRON/issues/METRON-1588?filter=allopenissues On June 26, 2018 at 17:01:17, Michel Sumbul (michelsum...@gmail.com) wrote: Hi, This make completely sense. I agree that in some

Re: How to delete the original message field once the message parsed?

2018-06-26 Thread Michel Sumbul
Hi, This make completely sense. I agree that in some case the original string need to be store for legal reason or future other analysis. In the same time keeping it increase a lot the cost of the needed infrastructure. which can slow down the adoption of Metron. Your idea to keep it in HDFS and

Re: How to delete the original message field once the message parsed?

2018-06-26 Thread Simon Elliston Ball
Agreed. I think of the hdfs batch store as the throw away nothing store, and the lucene real-time store as more of an index or cache which does not have to be quite so complete, where we could definitely optimise down some of the fields. Simon > On 26 Jun 2018, at 04:57, Otto Fowler wrote: >