I recently worked on a PR that involved changing the default behavior of
the ElasticsearchWriter to store data using field names with the default
Metron separator, dots. One of the unfortunate consequences of this is
that although dots are allowed in more recent versions of ES, it changes
how thes
Can you elaborate on what you mean by "convert to internal?" From your
description, it looks like the challenge is from our violations of DRY when
it comes to constants referencing those keys, which would be eliminated by
refactoring.
On Fri, Sep 7, 2018, 3:50 PM Ryan Merriman wrote:
> I recentl
Internal means it’s not configurable, doesn’t contain our default separator
(dots) and is namespaced with metron. We can definitely improve on DRY but
there’s more to it than that. For example, having 2 different versions of this
field name (ES and Solr) adds a significant amount of complexity
I propose that we just disallow having dots in the field name. Dots seem to
have a special meaning and as we keep adding data stores we may run into some
unintended behavior. We should have logic in our code to check for it and
either auto-correct it (replace with underscores?) or at least thr
Totally agree with replacing dot with something else. We have had so much
drama to use either dot or column with ORC either via Hive or Spark.
Although we have replaced it with an underscore, it may not be a good idea
as it can be confusing with underscores in the internal field names.
Cheers,
Ali