Re: [Freeipa-devel] Using JSON for tlog config files
On 15.6.2016 13:52, Nikolai Kondrashov wrote: > On 06/15/2016 02:41 PM, Martin Kosek wrote: >> Removing the secondary list from this discussion. >> >> On 06/15/2016 01:29 PM, Nikolai Kondrashov wrote: >>> Hi Simo, >>> >>> On 06/15/2016 12:25 AM, Simo Sorce wrote: On Tue, 2016-06-14 at 16:40 +0300, Nikolai Kondrashov wrote: > Although this was mentioned several times before, I'd like to bring > additional > attention to the idea of using config files written in JSON for tlog, > because > there were some concerns over that being appropriate. What was the reasoning behind this questioning ? >>> >>> The concern that JSON in those files might be inconvenient for >>> administrators. >>> In order to understand whether json is appropriate I need to ask who do you think will be the primary user of the configuration file. - Is it going to be mainly sysadmins manually setting things up ? >>> >>> At first, yes. Later minimally, because we plan to implement central control >>> of most (if not all) of the parameters. >> >> Yes, but AFAIK, tlog will still also exist as standalone component and could >> be >> configured without integration with FreeIPA. This means admins main >> configuration file would still be the config file. > > Yes, that's true. > > It wouldn't make much sense for small setups to use tlog, though, because it > implies ElasticSearch. At least Ansible or something like that will need to be > used for bigger setups, if not FreeIPA/SSSD, but yes, in that case it will > still be JSON. I would say that properly indented JSON with extensive comments and examples is good enough. -- Petr^2 Spacek -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Using JSON for tlog config files
On 06/15/2016 02:41 PM, Martin Kosek wrote: Removing the secondary list from this discussion. On 06/15/2016 01:29 PM, Nikolai Kondrashov wrote: Hi Simo, On 06/15/2016 12:25 AM, Simo Sorce wrote: On Tue, 2016-06-14 at 16:40 +0300, Nikolai Kondrashov wrote: Although this was mentioned several times before, I'd like to bring additional attention to the idea of using config files written in JSON for tlog, because there were some concerns over that being appropriate. What was the reasoning behind this questioning ? The concern that JSON in those files might be inconvenient for administrators. In order to understand whether json is appropriate I need to ask who do you think will be the primary user of the configuration file. - Is it going to be mainly sysadmins manually setting things up ? At first, yes. Later minimally, because we plan to implement central control of most (if not all) of the parameters. Yes, but AFAIK, tlog will still also exist as standalone component and could be configured without integration with FreeIPA. This means admins main configuration file would still be the config file. Yes, that's true. It wouldn't make much sense for small setups to use tlog, though, because it implies ElasticSearch. At least Ansible or something like that will need to be used for bigger setups, if not FreeIPA/SSSD, but yes, in that case it will still be JSON. Nick -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Using JSON for tlog config files
Removing the secondary list from this discussion. On 06/15/2016 01:29 PM, Nikolai Kondrashov wrote: > Hi Simo, > > On 06/15/2016 12:25 AM, Simo Sorce wrote: >> On Tue, 2016-06-14 at 16:40 +0300, Nikolai Kondrashov wrote: >>> Although this was mentioned several times before, I'd like to bring >>> additional >>> attention to the idea of using config files written in JSON for tlog, >>> because >>> there were some concerns over that being appropriate. >> >> What was the reasoning behind this questioning ? > > The concern that JSON in those files might be inconvenient for administrators. > >> In order to understand whether json is appropriate I need to ask who do >> you think will be the primary user of the configuration file. >> - Is it going to be mainly sysadmins manually setting things up ? > > At first, yes. Later minimally, because we plan to implement central control > of most (if not all) of the parameters. Yes, but AFAIK, tlog will still also exist as standalone component and could be configured without integration with FreeIPA. This means admins main configuration file would still be the config file. > The suggestions so far were to have > the same JSON simply stored in HBAC-bound LDAP entries verbatim. Although, I'm > not sure if that would be best. > >> - Is this going to be something that may need to be templated and >> delivered via things like puppet ansible ? > > I didn't plan specifically for that. I expect the chances of that are about > the same as for any other config file. > >> - Is it going to be a file generated by some other program (like sssd or >> similar) based on rules stored in a central system ? > > I'd like to avoid requiring any program controlling tlog to write any files. > That's why I implemented passing the whole or part of the configuration via an > environment variable value, overriding the global config. However, yes, that > value is expected to be the same JSON at the moment. > >> The answer to these questions will help understanding what format is >> better suited. > > Thanks, Simo! > > Nick > -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Using JSON for tlog config files
Hi Simo, On 06/15/2016 12:25 AM, Simo Sorce wrote: On Tue, 2016-06-14 at 16:40 +0300, Nikolai Kondrashov wrote: Although this was mentioned several times before, I'd like to bring additional attention to the idea of using config files written in JSON for tlog, because there were some concerns over that being appropriate. What was the reasoning behind this questioning ? The concern that JSON in those files might be inconvenient for administrators. In order to understand whether json is appropriate I need to ask who do you think will be the primary user of the configuration file. - Is it going to be mainly sysadmins manually setting things up ? At first, yes. Later minimally, because we plan to implement central control of most (if not all) of the parameters. The suggestions so far were to have the same JSON simply stored in HBAC-bound LDAP entries verbatim. Although, I'm not sure if that would be best. - Is this going to be something that may need to be templated and delivered via things like puppet ansible ? I didn't plan specifically for that. I expect the chances of that are about the same as for any other config file. - Is it going to be a file generated by some other program (like sssd or similar) based on rules stored in a central system ? I'd like to avoid requiring any program controlling tlog to write any files. That's why I implemented passing the whole or part of the configuration via an environment variable value, overriding the global config. However, yes, that value is expected to be the same JSON at the moment. The answer to these questions will help understanding what format is better suited. Thanks, Simo! Nick -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
[Freeipa-devel] Using JSON for tlog config files
Hi everyone, Although this was mentioned several times before, I'd like to bring additional attention to the idea of using config files written in JSON for tlog, because there were some concerns over that being appropriate. Tlog is a terminal I/O recording package [1], with primary purpose of sending the recordings as JSON-formatted log messages to ElasticSearch. For the purpose of reading what it wrote, it links with json-c. At this moment both of tlog programs (tlog-rec and tlog-play) expect their global configuration to be in JSON, with comments allowed. See the default configurations attached. Plus, tlog-rec accepts an environment variable, containing the whole, or a part of the configuration to (partially) override the global one. That is also in JSON. Tlog uses the same json-c to parse all of these. Internally, tlog uses json-c structures to pass around and merge the configurations. The question is, should the global tlog configuration, located in /etc/tlog/tlog-rec.conf and /etc/tlog/tlog-play.conf, be in JSON, or should it be something else? The cons I heard so far: * Administrators don't expect to find JSON in /etc. * JSON is a fragile format. The pros I'd like to present: * Administrators setting up tlog are expected to also be familiar with ElasticSearch, which tlog targets as the storage. ElasticSearch speaks JSON exclusively and is configured using either YAML or JSON. So, the administrators should be largely familiar with it. * Although JSON uses explicit and rigid syntax, such as quoting and prohibited trailing commas, it is still easy to read, and its specification is succint and easy to learn: http://json.org/ * Tlog is already linked with json-c, to read what it wrote, and reusing it for configuration reading avoids adding another dependency. Overall, I consider the present situation a good compromise between smaller/simpler code and reduced dependencies vs. familiarity and ease of editing and reading for administrators. The alternatives presented so far are YAML and INI. I'll list each of their pros and cons, as I see them. YAML Pros * Has a subset of syntax (sufficient for our purposes), which is easy to read and write, doesn't require quoting, not critical to commas and other separators. * Has official specification. Cons * Requires additional dependency to be used in tlog. * Only one implementation in C. * Uses significant whitespace, which is easier to overlook than explicit syntax. * *Sometimes* requires quoting to enforce value type, which is easy to overlook. E.g. an all-digits string requires quoting, otherwise it is considered a number. * Although well-defined, specification is long and complicated: http://www.yaml.org/spec/1.2/spec.html This makes it hard to fully understand the language and be proficient at it. * In an attempt to make the language as human-readable as possible, made it actually harder for humans to write, in some cases. * Has too many features, complicating parsers, leading to harder to use APIs, and more bugs. INI Pros * Familiar, already used by sister projects: SSSD, Kerberos, Samba, etc. * Light, simple syntax Cons * Requires additional dependency to be used in tlog. * No official specification, lots of variance in the field: https://en.wikipedia.org/wiki/INI_file#Varying_features This requires explicit description of the actually used syntax in the program manuals. I.e. it cannot simply link to a specification. Administrators have to discover which flavor to use. This will become worse if we'll implement storing (a subset of) tlog-rec configuration in LDAP verbatim, as suggested so far, because the documentation for the format will be less discoverable for the person editing the directory. * Cannot be written without newlines, in a single line. This will make overriding configuration with an environment variable in tlog-rec harder to use. I.e. the environment variable value will have to contain newlines, or instead refer to a file containing the configuration. * No escaping for special characters, multiline value support is patchy (not present at all in dinglibs). This will limit the ways to specify the recording notice presented to the users at the start of tlog-rec. Your own pros/cons, and suggestions for other formats to use are welcome! Thank you for your attention. Nick [1]: https://github.com/Scribery/tlog // // Tlog-play system-wide configuration. See tlog-play.conf(5) for details. // This file uses JSON format with both C and C++ comments allowed. // { // The type of "log reader" to use for retrieving log messages. The chosen // reader needs to be configured using its own dedicated parameters. // "reader" : "file", // File reader parameters "file": { //