On 06/06/2014 05:34 PM, Dmitri Pal wrote:
On 06/06/2014 06:46 AM, Petr Viktorin wrote:
On 06/06/2014 04:09 AM, Dmitri Pal wrote:
Hello,
Over the time multiple tools have been developed and used to address
different use cases around SSSD configuration.
We now have:
1) SSSD that needs to read a configuration. It uses ding-libs
libini_config for that purpose.
libini_config was historically developed to read configuration from C
services like SSSD.
It is currently used by SSSD and GSS proxy
It is not capable of modifying and saving the configuration. To be
fair
it is capable but the interfaces are not exposed at the high level
API.
It will take some (couple days or so to close this gap).
2) SSSD also needed to create sssd.conf at the installation time.
It is
ueasy to do it from Python this is why an SSSD config python
script have
been created. It is good at modifying the configuration and setting
specific values at the beginning of the sssd.conf's life. It is not
suited well (at least yet) for long term maintenance of the
configuration file. It is also SSSD specific.
3) Over the time there were RFEs that suggested that sssd
configuration
should be extensible or overwritable.
The following ticket cover that:
https://fedorahosted.org/sssd/ticket/2247
The design has been published here
https://fedorahosted.org/sssd/wiki/DesignDocs/ding-libs/INIConfigMerge
4) There is a desire to manage SSSD over OpenLMI a cockpit in an
object
oriented way. To do that (following the standard LMI model) the
following layring has been suggested and implemented:
Consumer (OpenLMI/Cockpit) -> D-Bus interface -> ifp service
(responder)
-> augeas c api
More details are here:
https://fedorahosted.org/sssd/wiki/DesignDocs/OpenLMIProvider
5) In the absence of the good configuration solution for the
integration
purposes, i.e. to integrate applications running on top of the system
with SSSD as a short term solution a puppet-augeas based way of
configuring sssd.conf was introduced the other day. The main
requirements are to be able to modify existing configuration and more
specifically change the values that represent a list of values.
6) From the very beginning we were hoping that there would be a
way to
be able to define the grammar of the configuration file and check
it to
be able to diagnose the config files and prevent their corruption.
The
idea was to add this capability to libini as it provides a set of
nice
convenience capabilities in comparison to Augeas. Also Augeas
until not
long ago was not included in some distors.
Could you elaborate on the convenience capabilities?
Last time I looked at Augeas (which was some time ago) it did not:
- allow multiple key instances like this
key = val1
key = val2
and ability to iterate a section and access them.
This feature is used by GSS proxy
- allow processing the config and handling the sections in different
ways detecting collisions, erroring on them or merging them on the fly
as one requests
- merging two configs the way you want
- support of BOMs - encodings at the beginning of the file
- adding and or modifying comments
- allowing to wrap values
- processing lists of values in a convenient way
Some of these are explicit non-goals of Augeas; some are features
they might have or want to have.
In other words, yes, we will need to do some work if we adopt Augeas.
Some of that work would need to be be on top of Augeas.
May be some of these feature already available and I am just not aware.
I guess some research is in order.
Also with libini there is a lot of control that we have to add features
on the pace we need.
Is it more work to complete libini, or add convenience capabilities
to/on top of Augeas?
libini has continence capabilities. It will be more work and AFAIU
feature degradation if we replace one with another but may be augeas is
more mature than I remember it to be.
Another factor is how many projects will benefit from SSSD devs
working on libini vs. on Augeas.
This is hard to say. We already did this evaluation once and said
Augest
is not good enough for us so we will go with libini.
But I started this discussion to make a conscious choice moving
forward.
Obviously the choice is yours, I'm not even a SSSD developer. I'm
just providing some arguments to weigh.
The list above presents a spectrum of use cases and solution to those
use cases.
The goal of this email is to try to enumerate the use cases and
constraints and understand how the solution for those use cases
can be
consolidated and unified and where it makes sense todo and where
not. It
is a goal to define a reasonable alignment between the
technologies and
provide a path forward to a reusable and maintanable stack.
So we have the needs:
1) Read config files at run time from C (daemons and D-BUS provider)
2) Write config files from C (configuration interface over LMI)
3) Create and modify files at installation and upgade using some
simple
mean/API which can be C, python or anything else
4) Be able to manage configuration files in a object oriented manner
with or without D-BUS running
5) Be able to augment the code file without touching it
I don't understand this sentence, could you clarify?
s/code/conf
6) Provide a way to validate the file
So I see the following layering
+------------------+
| High level tools |
+------------------+
|
\/
+---------------------------+
| Object oriented interface |
+---------------------------+
| |
\/ \/
+-----------------+ +------------------+
| D-BUS responder | | D-BUS shim layer |
+-----------------+ +------------------+
| --------------------|
| |
\/ \/
+-------+ +----------------+
| C API |<-------------| File validator |
+-------+ +----------------+
/\ \
| \ +--------------+
| -------------->| Config file |
| +--------------+
| /\
+----------------+ |
| SSSD/GSS proxy | |
+----------------+ |
|
+------------------+
| SSSD Coinfig API |
+------------------+
/\
|
+--------------+
| Simple tools |
+--------------+
This layout shows three distinct paths towards config file:
1) Old python API and tools on top (does authconfig use this path?)
2) Top level D-BUS api and advanced object oriented tools
3) C API
We need to make several decisions at this point:
a) How much we enhance-update-improve existing python API instead of
using new D-BUS interface.
IMO we should leave python API alone for foreseeable future but
any new
changes should go via the new D-BUS interface
b) How we make sure that the D-BUS interface is really usable. It
is not
that light weight.
Would be nice to be able to have a shim layer for the cases like
docker
where there might not be D-BUS or systemd. This however might not
be a
problem for us to solve. It might be a generic problem for all LMI
style
providers. A lot in D-BUS world is autogenerated so I suggest that we
ask to be able to generate a mini D-BUS interface that would go
directly
from the interface to the logic implemented in responder. I do not
know
what is best. But I think it is a discussion to have with D-BUS
guys and
should be more general.
For now we can assume that D-BUS is there and docker use case will be
solved in some generic way.
c) If we assume D-BUS and systemd it really makes sense to
separate the
responder in charge of configuration from other responders and
make it
started on demand when there is a real call over the D-BUS to make a
config change. IMO this work for splitting ifp into data ifp and
config
ifp should be filed as a ticket and done as a part of 1.13 release.
d) What is going to be the low level C library? We have two
candidates
libini and augeas.
I definitely have a preference towards libini but this does not
mean I
am right. I do not have a lot of time to maintain libini. I do not
see a
reason to rip it out and replace with augeas yet. But I also do
not see
a reason why we should rely on augeas. Right now IMO it would be more
work to rip libini and switch to augeas but over time if we do
more and
more augeas work it might be much less. I feel people are more
comfortable with augeas than using libini and adding to it as they
go. I
do not know why but suspect it is because augeas has appeal of a more
commonly used tool/interface. It is all about what we think is the
right
path. If we decide that augeas is the way to go then we should not
invest into enhancing libini and start building our improvements in
augeas or around augeas. If we think that libini is the right path
because it fill give us more features, flexibility and control
then let
us make it so that I am not the only contributor to it and people
would
not have problems throwing patches at it on as needed basis.
For me this is a very important decision. Right now I feel I am on
hook
to continue investing my scares time into libini. I think I am
holding
people back with this so I would rather come to terms and move
forward
whatever we think is best for SSSD, GSS proxy and potential other
daemons that need to read and deal with config files.
It seems the SSSD devs here in Red Hat's Brno office don't have the
courage to say this, so let me convey the general idea floating around
here: continuing with libini is a bad idea.
We're in the business of collaboration, not writing our own little
libraries where the main advantage is having control over the project.
I would agree with you if we were at the decision point several
years ago.
libini now is mature and can do what we need. It is stable.
Adding to it is simple and not much effort. I can prove it and turn
back
pretty much all the code we need for it (except grammar validation) to
address writes, merging etc. But I can't support it. It is not
sustainable in a long run.
That's the point.
Augeas is sustainable in the long run, because it's, AFAICS, a
healthy community project.
And anything useful built on top of Augeas has a better chance to get
a community. (Or be replaced with a more popular Augeas-based
implementation of the same feature, of course.) That's the way things
go.
The grammar validation IMO should be a generic feature and can be
potentially another layer that processes config and check if it is OK.
It can be independent of augeas/libini but making it generic would be
more work than really needed so it should rather pick one and target
it.
If you want a generic feature, you'll need to base it on a tool that
other people use, not on a tool that only caters to the needs of two
projects. Otherwise the feature will never be generic, because it
will also only cater to those two projects. And if the two projects
happen to not have many outside contributions, neither will the feature.
Of course, having a non-generic feature is possible, but you'll have
to write it and support the little tower of code forever. From the
ground up it'd be incompatible with what other people use, and
there'd be no integration at the higher levels because of the "making
things generic is more work" philosophy.
Now here is the situation - we have to pick one. I look at how much
work
there is for me to address everything that needs to be addressed in
libini to meet the requirements of the stack above and I say days
(though I do not have those days). I look at augeas and I say weeks or
months. But this is my take because I know libini and invested into it.
Well, you don't have the time, so ask the person who'll need to do
the changes for an opinion.
Or actually don't ask the developer; developers' estimates always
fall short. Especially when they know the code :)
We already have libini. We already support it and it has a lot of
useful
features. I am fine replacing it if we are sure that augeas would not
prevent us from doing what we are doing now. If that would require a
major refactoring and it would require us to contribute to augeas
significant chunks of code I do not think it is justified.
Contributing to Augeas will help many more people than playing in our
own sandbox.
True in general case. But we are not altruists here. Everything has
cost
and merit.
So far I have not seen any real technical argument why augeas is a
better choice. I see a generic statement about general value to open
source that is not measurable. Let us be practical. Give me the value
and the cost and convince me that this is the right cost to have.
Look, you're the manager here. Cost and benefit analysis is your
turf; I don't have the capability of measuring ideas by money.
I am, however, an employee of a company that prides itself on
collaboration. I am used to working with others to solve common
problems. Augeas clearly solves many of the same problems libini
does, it's fairly popular and has some people who clearly know what
they are doing.
We are not altruists. We are collaborators, because working with
others is better for us in the long run.
I can be convinced otherwise I would not have had this conversation but
not with the argument above.
And it'll also be better for us -- I don't think libini is getting
Ruby bindings or a big tool ecosystem any time soon.
Do we need ruby bindings? Is it really a use case we care?
Well, that's for the manager to decide. I can see two possible answers:
* No, it is not.
We don't care about anything that doesn't benefit us directly.
This is exactly why there will never be a community around libini.
* Yes, it is.
There's a module for Puppet that uses these Ruby bindings to
configure stuff.
AFAIK, neither Puppet nor Augeas authors needed to write the
integration. It came from the community.
By joining the Augeas community, we'd get Puppet integration for SSSD
configs, right now.
And later, we'd automatically get similar integration for whatever
tools are popular enough for someone to write a generic Augeas plugin
for.
If there is anything missing in Augeas then we should extend Augeas,
instead of trying to make another implementation.
We already have implementation this is the point and the question
is: is
it worth dropping it or not.
Convince me that it is worth.
On 06/06/2014 05:47 PM, Dmitri Pal wrote:
So let me summarize pros and cons:
Pros of libini
- already working and stable parser for INI files
- already integrated with SSSD and GSS proxy
- has advanced convenience features?
- can be turn around and delivered very fast because of control
Cons of libini
- zero community
- INI files only
- main developer does not have time to work on it
Pros of augeas
- has parser
- has community
- contributing to it will have general benefit
Cons of augeas
- not integrated into SSSD and GSS proxy
I'm sure you heard more than once that we need to integrate things,
rather than build separate stacks.
- does not have advanced features (to be verified)
- less control and thus slower turn around
Yes, unfortunately, collaboration means having to do politics. It's
the way it works in open source, except maybe in our little island of
open-source projects with no community.
Anyway, control is primarily based on the time you invest into the
project.
If push comes to shove, you can always fork it, or carry your own
stack of patches for absolute control.
So when I look at it for me it makes perfect sense to continue
investing into libini to address its cons, i.e. if we contribute to
it we will foster a community for it.
Umm, how good are we at fostering developer communities?
Can you create a sustainable community around libini? Is it actually
useful for use cases we don't care about?
If you want a *community* around the config parsing library we'll
use, I can tell you without doubt that porting to Augeas will be WAY
easier than attracting a community to libini.
A healthy community would also mean that *we* would not really be in
control. One of libini's big advantages would be lost.
Then we already leverage the functionality we have, work for greater
good, do not waste cycles and show leadership. It also allows us to
be focused on the use cases we really care about.
