Re: Massive libxo-zation that breaks everything
On 03/02/15 11:55, Julian Elischer wrote: > On 3/2/15 5:27 AM, Alfred Perlstein wrote: >> >> >> On 3/2/15 4:14 AM, Julian Elischer wrote: >>> On 3/1/15 10:49 AM, Harrison Grundy wrote: >>>> Thanks! >>>> >>>> That does seem useful, but I'm not sure I see the reasoning behind >>>> putting into base, over a port or package, since processing XML in base >>>> is a pain, and it can't serve up JSON or HTML without additional >>>> utilities anyway. >>>> >>>> (If I'm reviving a long-settled thing, let me know and I'll drop it. >>>> I'm >>>> trying to understand the use case for this.) >>> >>> To me it would almost seem more useful to have a programmable filter >>> for which you could produce >>> parse grammars to parse the output of various programs.. >>> thus >>> >>> ifconfig -a | xmlize -g ifconfig | your-favourite-xml-parser >>> with a set of grammars in /usr/share/xmlize/ >>> then we could use it for out-of-tree programs as well if we wrote >>> grammars for them.. >>> >>> The sentiment of machine-readable output is nice, but I think it's >>> slightly off target. >>> we shouldn't have to change all out utilities, and it isn't going to >>> help at all with 3rd party apps, >>> e.g. samba stuff. A generally easy to program output grammar parser >>> would be truely useful. >>> and not just for FreeBSD. >>> >>> I've been watching with an uncomfortable feeling, but it's taken me a >>> while to put my >>> finger on what it was.. >>> >>> >> Are you sure it's not the hairs on the back of your neck standing up >> due to NIH? >> >> Juniper has been doing this for years and it's very useful for them. > I'm not saying the ability to generate machine readable output is wrong, > but that the 'unix way' would be to make a filter for it. It seems that > the noisy people don't > agree with me so I will not stand in the way of progress.. > Julian and Alfred, Does the "lib-ification" idea address your differing concerns? The idea is to split base binaries into a library that does the "work", and a binary that does the UI bits. This way, "df" becomes a CLI interface connected to "libdf" that handles the CLI output. "xodf" (or whatever we'd like to name it) becomes a libxo interface on top of libdf. This allows the CLI code to remain largely unchanged, and permits the XML output code to link across multiple utilities at the same time. (So that things like, say, ifconfig information can be incorporated into route and netstat results by linking to "libifconfig" "libroute" and "libnetstat", respectively. If we're going to have to touch every base binary anyway, it seems like this might be a useful abstraction to get at the same time. --- Harrison ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Massive libxo-zation that breaks everything
On 03/02/15 01:23, David Chisnall wrote: > On 1 Mar 2015, at 21:29, Rui Paulo wrote: >> >> On Mar 1, 2015, at 11:11, David Chisnall >> wrote: >>> How would it be in a port? It involves modifying core >>> utilities (some of which, like ifconfig, rely on kernel APIs >>> that change between releases) to emit structured output. >>> Maintaining two copies of each utility, one in the base system >>> with plain-text output only and another in ports with XML/JSON >>> output would be very painful. >> >> It would work fine if we had *libraries* for >> ifconfig/netstat/route/etc. Obviously that's not the case and no >> one has stepped up to implement them. I've also seen FreeBSD >> committers expressing their distaste for libraries for "trivial" >> command line utilities, which implies they are unaware of another >> world beyond the CLI. :-) > > I am completely in favour of libraries for the underlying > functionality of these commands and would love to see all of the > system management commands become thin wrappers around a library, > though it's a lot of engineering work. In particular, these > libraries will need to have stable APIs that we can support across > multiple major releases, and getting those right is difficult. We > really don't want to be stuck in 10 years maintaining a hastily > designed API for a library. > > I see one use of the libxo output as helping to design those APIs. > People are going to wrap various tools in libraries for their > favourite scripting languages and this will give us a corpus for > experimenting. > > It's also worth noting that often invoking a tool and consuming its > output is the easiest way to get a stable API and ABI where > performance is not a primary concern (i.e. most management > interfaces). > > As to a world beyond the CLI, I saw a nice demo a few years ago of > a terminal emulator that used WebKit and came with a hacked-up set > of parsers for common tools. I'd love to have something simpler > (no need for a full WebKit - simple outline and table views would > be enough and could be done with curses for ssh) for FreeBSD where > I could type ls in the CLI and get a table view that I could then > sort and filter by selecting column headings. Those of us that > have used Lisp and Smalltalk environments know that a CLI doesn't > have to be a teletype emulator. > > David It would seem like the libxo stuff runs the risk of becoming this same API. --- Harrison ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Massive libxo-zation that breaks everything
On 03/01/15 13:25, Craig Rodrigues wrote: > On Sun, Mar 1, 2015 at 10:49 AM, Harrison Grundy < > harrison.gru...@astrodoggroup.com> wrote: > >> Thanks! >> >> That does seem useful, but I'm not sure I see the reasoning behind >> putting into base, over a port or package >> > > >> , since processing XML in base is a pain, and it can't serve up JSON or >> HTML without additional >> utilities anyway. >> > > > I think if you take another pass at reading the entire thread of responses > to see the discussion for the motivation behind libxo, you will get the > idea: > https://lists.freebsd.org/pipermail/freebsd-arch/2014-July/015633.html > > > There are many people who are building products on top of FreeBSD. > For these people, it is super useful for the base utilities in FreeBSD > to emit output in properly formatted XML or JSON. > > That way, they do not need to write scripts to take the output of > say, netstat, and use awk/sed/whatever scripts to take the human readable > netstat output and convert it to a form which can be used in a script. > > There are many, many parsers for XML and JSON not in the base system. > For people building products on top of FreeBSD, they don't care > if these parsers are not in the base, since they can add these parsers on > top of base FreeBSD. > > For example, languages like Python and Ruby have excellent parsers > for JSON and XML. Many people build products using these languages. > There are JSON and XML parsers in C, C++, and other languages as well. > > In addition to people building products, the other audience of people who > benefit from libxo are devops people. > These days, devops folks have no problem using Python, Ruby, Perl, > whatever to write scripts to interact with Unix boxes and pull information > off of it. Having the base utilities emit info in native JSON or XML > greatly facilitates this. > > I talked to one person who is improving FreeBSD support for Saltstack (a > devops framework). He told me > that if more base utilities such as sysctl, could use libxo to emit output > in JSON, that would greatly facilitate improving FreeBSD support for > these devops frameworks. That is because Saltstack would require > less FreeBSD-specific parsing code for getting info from base utilities. I like the idea behind this... where I'm running into difficulty is why these bits of functionality need to be combined. What someone does with ifconfig on the command line, versus what someone wants to know about their network interfaces in an XML dump may be very distinct bits of information. (For instance, an XML dump of network information may want to incorporate data from netstat, route, and ifconfig, while ifconfig is really only designed to deal with the interfaces themselves.) I know it involves some notable reworking, but would there be any interest in seeing what "libifying" a handful of binaries and splitting the libxo and CLI functionality looks like? It seems like this may give FreeBSD the best of both worlds, so the XML output isn't logically limited by how the CLI needs to work, and the CLI won't have to deal with the complexity XML output can require. I'd be happy to get this set up for a few binaries so people can see how it might work and what it could look like if there's any interest in going that route. --- Harrison ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Massive libxo-zation that breaks everything
On 03/01/15 11:11, David Chisnall wrote: > On 1 Mar 2015, at 18:49, Harrison Grundy > wrote: >> >> That does seem useful, but I'm not sure I see the reasoning >> behind putting into base, over a port or package, since >> processing XML in base is a pain, and it can't serve up JSON or >> HTML without additional utilities anyway. > > How would it be in a port? It involves modifying core utilities > (some of which, like ifconfig, rely on kernel APIs that change > between releases) to emit structured output. Maintaining two > copies of each utility, one in the base system with plain-text > output only and another in ports with XML/JSON output would be very > painful. > > The goal of having machine-readable output in the base system is > that people building systems atop FreeBSD will be able to expect a > stable, machine-parsable, extensible, output from these tools. If > you're building a web admin GUI or automated administration tool > for FreeBSD 11, or improving integration in your favourite DE, then > you should be able to rely on the output from base system tools, > without having to depend on anything external. > > My only concern with libxo at present is that many of the modified > tools are not emitting self-describing output (e.g. not specifying > units for things), but that's something that we have a year to > shake out before 11. > > David Thinking about the kernel interfaces, I can see the need for it in base... but then it boils down to why, say, ifconfig in particular needs to support this behavior. It seems easier/safer to implement this independently of the existent commands using the various APIs. By putting this in the actual base utilities, an API that I would assume needs to remain relatively static is being introduced that may have some fairly large consequences for how those utilities need to operate. It just seems odd to me to have the same command one would use to, say, join a wireless network and set an IP also spit out HTML reports on interfaces. It seems like the needs of those programs may diverge over time, or that functionality needed for some new ifconfig thing would then require picking (and being stuck with) an XML schema for that information at the same time. --- Harrison ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Massive libxo-zation that breaks everything
Thanks! That does seem useful, but I'm not sure I see the reasoning behind putting into base, over a port or package, since processing XML in base is a pain, and it can't serve up JSON or HTML without additional utilities anyway. (If I'm reviving a long-settled thing, let me know and I'll drop it. I'm trying to understand the use case for this.) --- Harrison On 03/01/15 10:31, Craig Rodrigues wrote: > On Sun, Mar 1, 2015 at 9:25 AM, Harrison Grundy < > harrison.gru...@astrodoggroup.com> wrote: > >> >> >> If someone could summarize what this is, I'd greatly appreciate it. >> > > https://lists.freebsd.org/pipermail/freebsd-arch/2014-July/015633.html > ___ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Massive libxo-zation that breaks everything
On 03/01/15 05:33, Sulev-Madis Silber (ketas) wrote: > Hello. > > First, I would be happy to have JSON and XML output about > filesystems, users, routes... but I don't like how it makes code of > df, w, netstat hard to read/maintain and often broken. > > I don't think it would be good to continue with this. Maybe the > effort should be put to creating new layer/library and then > something on top of it that actually outputs JSON and XML. > > Or, if that's too difficult... maybe just regular df/w/netstat > could be copied to somewhere else and made code libxo-output-only. > And original df/w/netstat changes reverted and left alone. > > Then, maybe later, df/w/netstat/... could be updated to this new > layer/library. Or maybe this should be just left as it is. > > That would mean having two netstat's in system, which could be both > good (separation) and bad (maintaining). > > Just some ideas... I don't know how to solve this issue fully. I'm > also not likely the one who would write code for all this. Hell, > those aren't even all my ideas here. I just worry that system > drop-in xo-zation is bad for overall health of base. > > Oh and, it makes rescue larger and more complex, too? On that, > there was suggestion to maybe create separate "first aid kit" and > "emergency room" types of system rescue utils/methods. > > > Thanks. ___ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current To > unsubscribe, send any mail to > "freebsd-current-unsubscr...@freebsd.org" > Forgive my ignorance about exactly what this is, and possibly reviving a long-settled debate, but this sounds like something that would be a great deal more useful as a port/package, rather than in base. Due to the lack of XML parsing code in -base, the difficulty in maintaining yet another interface, and the overhead involved in doing it, I don't quite see where one would really want XML output *and* be entirely opposed to ports or packages. If someone could summarize what this is, I'd greatly appreciate it. --- Harrison ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
