On Mon, Jan 20, 2014 at 11:27 AM, Daniel P. Berrange <berra...@redhat.com>wrote:
> On Fri, Dec 20, 2013 at 11:03:53PM -0500, Adam Walters wrote: > > This patchset adds a driver named 'config' that allows access to > configuration > > data, such as secret and storage definitions. This is a pre-requisite > for my > > next patchset which resolves the race condition on libvirtd startup and > the > > circular dependencies between QEMU and the storage driver. > > I vaguely recall something being mentioned in the past, but not the > details. Can you explain the details of the circular dependancy > problem that you're currently facing ? > > > The basic rationale behind this idea is that there exist circumstances > under > > which a driver may need to access things such as secrets during a time at > > which there is no active connection to a hypervisor. Without a > connection, > > the data can't be accessed currently. I felt that this was a much simpler > > solution to the problem that building new APIs that do not require a > connection > > to operate. > > We have a handful of places in our code where we call out to public > APIs to implement some piece of functionality. I've never been all > that happy about these scenarios. If we call the public API directly, > they cause havoc with error reporting because we end up dispatching > and resetting errors in the middle of an nested API call. Calling the > driver methods directly by doing conn->driver->invokeMethod() is > somewhat tedious & error prone because we're then skipping various > sanity tests that the public APIs do. With ACL checking now implemented > we also have the slightly odd situation that a public API check which > is documented as requiring permission 'xxxx' may also require permissions > 'yyyy' and 'zzzz' to deal with other public APIs we invoke secretly. > > I think there is a fairly strong argument that our internal implementations > could be decoupled from the public APIs, so we can call methods internally > without having to go via the public APIs at all. > > On the flip side though, there's also a long term desire to separate > the different drivers into separate daemons, eg so the secrets driver > might move into a libvirt-secretsd daemon, which might explicitly > require use to go via the public APIs. > Mulling this over last night, I think there may be an alternative implementation that would be sustainable long-term. You mentioned a desire to split the drivers into separate daemons eventually... It might seem silly today, but what if I went ahead and implemented a connection URI for each of the existing drivers? This would result in a 'network:///', 'secrets:///', 'storage:///', etc. Once complete, the existing code base could slowly be updated to utilize connections to those new URIs in preparation for splitting the code out into daemons. This would end up with the current libvirtd process eventually becoming a connection broker, with a compatibility shim to allow access to the API as it is currently defined for backwards compatibility. In effect, today nothing would change, other than the storage driver would use 'secrets:///' to open a connection for its backends. Eventually, though, all drivers would use the new URIs (the actual APIs implemented would not need to change), allowing an easier split of the daemons. The compatibility shim I mentioned would just be some code to open a connection to the proper URI, make the request, close the connection, and return the data. The one thing that would be needed at some point (which I'm not sure how to really implement today), would be a restriction on which API functions could be called. Basically, if you opened a connection to 'secrets:///', you should only be able to access secrets. If you tried to access network information, an error would need to be thrown in order to prevent all of the drivers having a connection to all of the other drivers, burning up sockets on the host system. I'm guessing that this could be implemented using the ACL framework that is already in place, but I haven't dug through the code to verify this yet. In this long-term desire to split libvirt into multiple daemons, is there a plan to also create a libvirt-domaind? Currently, the various hypervisors don't actually know how to read the domain XML fully, so a daemon like that would probably be desired. As a bonus, it could also allow a single connection to view all running domains, regardless of which hypervisor driver actually created it, though that isn't strictly needed. The reason I ask is that currently, it is nigh impossible to determine driver inter-dependencies automatically. If strict controls were in place to prevent cross-URI calls (in effect making libvirt as a single process act as though it were already split), it should become easier to determine a driver's dependencies at compile (or possibly even run) time. Having that data would allow for a true dependency-based driver load order. While I think implementation of a dependency-aware driver loader is probably beyond my personal programming skills, it would be the best method in the long run. Though, in a multi-process model, it may be simpler to define a new connection function that waits (optionally until a timeout expires) for the connection to succeed, performing automatic retries in that time. This would allow the various daemons to start in any order, yet automatically initialize in the proper dependency order. Any circular dependency would, of course, cause a long hang during startup, but a timeout could detect that and fail the startup of libvirt as a whole. The same timeout-based connection function could also be used in the single-process model, of course, removing the need for driver load order to be changed at all if all existing code used the proposed new URIs. In effect, the driver load order change would be needed today, but once all existing code is updated, the change could be reverted to the current free-for-all model, as there would be implied control over that by virtue of the fact that the drivers would wait for their dependencies to finish connecting. > > > This driver is technically what one may call a hypervisor driver, but it > > does not implement any domain operations. It simply exists to handle > requests > > by drivers for access to informatino that would otherwise require a > connection. > > The URI used for this driver is 'config:///' and has been tested working > on 4 > > different machines of mine, running three different distributions of > Linux > > (Archlinux, Gentoo, and CentOS). Being a very simple driver, I would > expect > > it to work pretty much anywhere. > > > > I would love to hear any comments and suggestions you may have about this > > driver. At the very least this plus my next patchset resolves the startup > > race condition on my machine. If a more robust setup (likely a new > internal > > API) is in the works, this driver could act as a band-aid to allow access > > to this type of data in the interim if a better resolution is a ways off. > > One big concern I have about this approach is the fact that once we add > this 'config:///' driver, it is very hard for us to ever remove it again, > since this concept leaks out into the public API. So we're going to want > to be very sure that this is something we wish to support long term, and > I don't really have such confidence myself. > > I'd like to understand a bit more about the dependancy issue you're facing > to see if there's something else we can do to address it. > > Daniel > -- > |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| > |: http://libvirt.org -o- http://virt-manager.org:| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:| >
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list