Hi Nick!
We, like Tom, plan to use Elastic Common Schema (ECS) to store events in Metron.
A feature of ECS is the nesting of JSON objects, and therefore the "system.id"
field implies storage in the form {"'system": {"id": "<value>"}}
Converting ECS to flat json where the fields take the form {"system.id":
"<value>"} is not a good option, because the very meaning of its use and the
convenience of the JSON format are lost.
Now, in order to work with nested JSON using Stellar, we are forced to use such
complex constructs using the MAP_GET and MAP_PUT functions, for example:
"fieldTransformations": [
{
"output": ["system"],
"transformation": "STELLAR",
"config": {
"system": "MAP_PUT('id',
MAP_GET('id',MAP_GET('os',host_id)), system)"
}
}
]
And with deep nesting, this generally turns into unreadable, hard-to-maintain
code.
And now the question: is there a way to easily work with nested JSON in
Stellar? Deep diving into the documentation and source code has not yet given
an answer.
Now this is a fundamentally important issue that affects the moments of
enrichment, TI, profiling and simply changing data when parsing.
On 2019/11/01 17:50:29, Nick Allen <[email protected]> wrote:
> Hi Tom -
>
> > In the case of Metron, should we be modifying the field names to replace
> dots? Can the Metron STELLAR language handle a field name with a dot in it,
> or are there any special steps required such as surrounding event fields
> with single or double-quotes in order to properly handle those field names?
>
> I cannot think of any facilities within Metron itself that would have
> difficulties with periods in field names.
>
>
> > I noticed that our fields names arrive with a period in the name, for
> example "client.ip" and "user.id"... Our internal naming convention is
> intended to align the data ingestion solution with the Elasticsearch Common
> Schema. From experience, working with those dots in Elasticsearch is a
> challenge
>
> You can use Metron to translate the field names however you like. For
> example, replace "client.ip" with "client_ip". There are some examples of
> this in the Parsers documentation here [1]
> <https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html>.
> Looks under the section "fieldTransformation configuration".
>
>
> ---
>
> [1]
> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html
>
>
>
>
>
> On Fri, Nov 1, 2019 at 1:21 PM Yerex, Tom <[email protected]> wrote:
>
> > Good day to everyone. I'm working on our own variation of the Geographic
> > Login Outliers use case (
> > https://metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html).
> > I noticed that our fields names arrive with a period in the name, for
> > example "client.ip" and "user.id".
> >
> >
> >
> > Our internal naming convention is intended to align the data ingestion
> > solution with the Elasticsearch Common Schema. From experience, working
> > with those dots in Elasticsearch is a challenge and it raises the question
> > if we need to handle field names with a dot in a different matter in Metron.
> >
> >
> >
> > In the case of Metron, should we be modifying the field names to replace
> > dots? Can the Metron STELLAR language handle a field name with a dot in it,
> > or are there any special steps required such as surrounding event fields
> > with single or double-quotes in order to properly handle those field names?
> >
> >
> >
> > Thank you,
> >
> >
> >
> > Tom.
> >
> >
> >
>
