I think the ‘files’ should be stored in zk, and updated with the same mechanism.
On July 14, 2017 at 13:27:36, Matt Foley ([email protected]) wrote: In the abstract, this is a good idea. I see it as related to METRON-987, which was the first step in allowing sequences of Stellar statements (aka "programs" :-) ) instead of just unrelated groups of single statements. Your proposal lets us really work with programs as first-class entities. However, some concerns need to be resolved: 1. Syntax. Currently Stellar syntax and JSON fit neatly together. Where would be the cut line for file substitutions? Referencing METRON-987, would you only allow a file substitution where we currently allow square-bracketed Stellar string sequences? What about Profile config syntax, where several chunks of code are intimately related (hence want to be located in the same file), but don't all get executed at the same time? (This is not a showstopper question because Profile configs are usually simple and don't really need file substitution. The need is much greater in Enrichment.) 2. Config Updates. Currently Metron configuration is stored in ZK, but managed through Curator libraries. In return for considerable complexity, this gives instant update whenever a config changes, without effort in the BI part of the application. This differs sharply from file-based configuration, where updates in response to config changes require either a restart, an explicit reload command from the user, or frequent state-checking in the application. So currently people trying to develop a new enrichment can update the config, and immediately test the result, without restarting and without any explicit reload command. We probably want to not lose this. Rather than roll our own file pointer model, can we use JSON Pointers? Will they work with Curator? Both of those get into some fairly obscure features, that would need to be studied. It also actually relates to the syntax question presented above. On 7/14/17, 6:17 AM, "Otto Fowler" <[email protected]> wrote: https://issues.apache.org/jira/browse/METRON-1046 I was thinking this morning that managing stellar statements in the config json could become, and maybe is kind of unwieldy. To that end, if in say a parser configuration I can refer to a ‘file’ in zookeeper as an alternative, we would add the capability to execute and manage more complex statements, and even chain multiple statements together. These files could be shared as well. This could be a Bad Idea™, so I thought I’d throw it out to the list. Please check out the jira, give some thought, and comment there or on the list or both. O
