At the parser level, there's some configuration you can use for filtering events. Specifically "filterClassName". Take a look at the documentation, you can either use a custom class, or use Stellar. The example is even for "exists(field)", which you could modify to fail for missing fields.
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html On Tue, May 5, 2020 at 7:53 PM Yerex, Tom <[email protected]> wrote: > Good afternoon, > > Our incoming data is not always perfect, in some cases events are simply > missing fields. We would like a way to drop events when particular fields > are empty (or have values we don't care about). > > One way we thought to do this might be to write a custom Stellar function. > Does anyone know of another solution? > > Thank you, > > Tom. >
