Just chiming in on a part of this: definitely we do not want to lose automatic config updates (at least, I'd be strongly, strongly STRONGLY against it).
I definitely agree that JSON files could easily get unwieldy. I don't know anything about JSON pointers, could you cover that briefly, Matt? Even a URL or two to get started would be great. Basic googling (while on vacation) yielded that it was something like xpath for json, but I probably just googled the wrong thing. Casey On Fri, Jul 14, 2017 at 6:27 PM, Matt Foley <[email protected]> wrote: > In the abstract, this is a good idea. I see it as related to METRON-987, > which was the first step in allowing sequences of Stellar statements (aka > "programs" :-) ) instead of just unrelated groups of single statements. > Your proposal lets us really work with programs as first-class entities. > > However, some concerns need to be resolved: > > 1. Syntax. > > Currently Stellar syntax and JSON fit neatly together. Where would be the > cut line for file substitutions? Referencing METRON-987, would you only > allow a file substitution where we currently allow square-bracketed Stellar > string sequences? What about Profile config syntax, where several chunks > of code are intimately related (hence want to be located in the same file), > but don't all get executed at the same time? (This is not a showstopper > question because Profile configs are usually simple and don't really need > file substitution. The need is much greater in Enrichment.) > > 2. Config Updates. > > Currently Metron configuration is stored in ZK, but managed through > Curator libraries. In return for considerable complexity, this gives > instant update whenever a config changes, without effort in the BI part of > the application. This differs sharply from file-based configuration, where > updates in response to config changes require either a restart, an explicit > reload command from the user, or frequent state-checking in the application. > > So currently people trying to develop a new enrichment can update the > config, and immediately test the result, without restarting and without any > explicit reload command. We probably want to not lose this. > > Rather than roll our own file pointer model, can we use JSON Pointers? > Will they work with Curator? Both of those get into some fairly obscure > features, that would need to be studied. It also actually relates to the > syntax question presented above. > > > On 7/14/17, 6:17 AM, "Otto Fowler" <[email protected]> wrote: > > https://issues.apache.org/jira/browse/METRON-1046 > > I was thinking this morning that managing stellar statements in the > config > json could become, and maybe is kind of unwieldy. > To that end, if in say a parser configuration I can refer to a ‘file’ > in > zookeeper as an alternative, we would add the capability to execute and > manage more complex statements, and even chain multiple statements > together. > > These files could be shared as well. > > This could be a Bad Idea™, so I thought I’d throw it out to the list. > > Please check out the jira, give some thought, and comment there or on > the > list or both. > > O > > >
