I recently worked on a PR that involved changing the default behavior of the ElasticsearchWriter to store data using field names with the default Metron separator, dots. One of the unfortunate consequences of this is that although dots are allowed in more recent versions of ES, it changes how these fields are stored. Having a dot in a field name causes ES to treat it as an object field type. We're not quite comfortable with this because it could introduce unforeseen side effects that may not be obvious. Here's the PR: https://github.com/apache/metron/pull/1181
As I worked through it I noticed there are a couple fields that include separators where it's not actually necessary. They are not nested by nature and are internal to Metron. The fact that they are internal means they show up in constants and are hardcoded in several different places. That made the work in the PR above much harder and tedious than it should have been. There are 2 in particular that I had to deal with: source:type and threat:triage:score in metaalerts. Is it worth considering converting these to internal Metron fields so that they stay constant and this isn't a problem in the future? I could see these fields following the same pattern as 'metron_alert'. However this would cause pain when upgrading because existing data would need to be updated with these new fields. Just an idea. Curious if other have an opinion on the subject.
