I merged the new backward conversion feature and released it in netjsonconfig 0.6.1: https://github.com/openwisp/netjsonconfig/pull/83
In the near future we will be able to collect the whole device configuration and store it in openwisp2, which will make management easier. A bit more work and building blocks are needed to get there, but the hardest part is done! FINALLY. Federico On Wednesday, May 31, 2017 at 12:55:13 PM UTC+2, Federico Capoano wrote: > > Hi everyone, > > recently I started introducing some important changes in netjsonconfig > <https://github.com/openwisp/netjsonconfig>, the library used to generate > configuration of routers. > > This change should not affect openwisp2 users negatively, I'm already > testing the current master on 3 instances and already fixed a few bugs. > > Nevertheless, the change is affecting negatively the 2 GSoC students who > have started to work on the AirOS and Raspbian backends, so I will try to > explain why this change is needed. > > *Backward conversion* > > One of the features that is missing in OpenWISP 2 is the possibility to > easily store the full configuration of devices in the controller. > > Since OpenWRT/LEDE allows merging a remote configuration with the local > one, that's the default mode we are using now in OpenWISP 2 (it was also > the only mode available in OpenWISP 1). > > This method has its own disadvantages: > > - operators have to remember what is in their local configuration, in > order to write a remote configuration that works well with the local one > - some packages use anonymous UCI blocks by default, which are > impossible for openwisp to override in the merge mode, hence creating > duplicate configurations > > These disadvantages are holding down OpenWISP 2 from becoming easier to > use. OpenWISP 2 is very powerful, but hard to use properly. Which is a > shame, because it means many people will give up and use some kind of home > baked solution with its own class of bugs, reinventing the wheel: no good > framework and ecosystem of tools for low cost open source networks can > emerge for such a situation. > > That's why we need to add a way to store the full configuration of routers > in the controller and this needs to happen automatically: here where > backward conversion in netjsonconfig comes into play. > > Without the backward conversion, we cannot even use the AirOS or Raspbian > backends because those don't have a way to easily merge different > configurations like OpenWRT/LEDE (uci import --merge), users would need to > manually enter the full configuration of their devices in openwisp2: that > will never happen! > > *The new implementation* > > I've been designing and planning this feature since at least a year, but > never got the time to implement because there were more urgent matters. Now > is the right time to introduce it because it's very much needed. > > The new implementation adds a few new concepts in the codebase: > > - intermediate data structure > - converters > - parsers > > Everything revolves around the data structures: the *NetJSON > <http://netjson.org/> configuration dictionary* and the *intermediate > data structures*. > > Remember what *Linux Torvalds* said? > *"Bad programmers worry about the code. Good programmers worry about data > structures and their relationships."* > > This intermediate data structure represents the native configuration of > the backend in a python data structure (lists and dictionaries). > > This data structure is a convenient way to simplify conversion in both > direction, parsing and rendering, and will be different for each backend. > > Take a look at this diagram: > [image: netjsonconfig backward conversion.jpg] > *Converter* classes take care of converting between NetJSON and > intermediate data structure (and vice versa): > > - to_intermediate converts the NetJSON configuration dictionary to the > intermediate data structure > - to_netjson (not pushed to the repository yet) converts the > intermediate data structure to NetJSON configuration dictionary > > These two main methods can then call helper methods in order to split > complex logic in smaller methods that are easier for the human mind to > grasp. > > *Renderers* take care of rendering the intermediate data structure to the > native format. > > *Parsers* perform the opposite operation of Renderers: they take care of > parsing native format and build the intermediate data structure. > > In case of OpenWRT and AirOS, we can get away with just 1 renderer (with > just 1 jinja2 tempate) and 1 parser. In the case of Raspbian, we may need > to have a render and a template for each format that needs to be rendered > (eg: /etc/network/interfaces, ini files, dnsmasq, iptables, ecc). > > *Advantages* > > - Cleaner, well defined workflow - also easier to document and learn > - Less boilerplate code in those cases where the native configuration > of the router is centralized in one format (OpenWRT / AirOS / PfSense / > vyos), because most of the logic goes in converters while there can be > only > 1 renderer and 1 parser > - Last but not least, makes it possible to implement backward > conversion without going crazy with messy spaghetti code. Our sanity is > very important not only for our work but for our life! > > *Disadvantages* > > Developers using lower level features of netjsonconfig in their programs > and GSOC students working on the netjsonconfig backends will have to > rewrite part of their code. A bit of more work needed, but in my opinion a > worthy price to pay for a cleaner and more robust implementation. > > *How to upgrade your code* > > Proceed as follows: > > - the logic that before was contained in the renderers, should most > probably go into the converters > - design an intermediate data structure with the following features: > - closely represents your native configuration format (or formats) > - can be easily rendered with jinja2 > - can be easily built when parsing the native format (or formats) > - simplify your renderers so that you have 1 renderer and 1 template > for each format supported > > *Conclusion* > > That's it. Sorry for the length. I hope the explanation is clear enough. > > Federico > -- You received this message because you are subscribed to the Google Groups "OpenWISP" group. To unsubscribe from this group and stop receiving emails from it, send an email to openwisp+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
