Re: [libvirt] [PATCH v3 00/48] Split the libvirtd daemon into per-driver daemons
On Tue, Jul 30, 2019 at 12:13:30PM +0200, Christophe de Dinechin wrote: > > > > On 30 Jul 2019, at 11:38, Daniel P. Berrangé wrote: > > > > On Tue, Jul 30, 2019 at 11:05:25AM +0200, Christophe de Dinechin wrote: > >> > >> Daniel P. Berrangé writes: > >> > >>> This is what all the driver refactoring I've done has been about > >>> enabling. > >>> > >>> We gain new daemons for each driver, for the primary virt drivers: > >>> > >>> virtlibxld > >> > >> virtxend? > >> > >>> virtlxcd > >>> virtqemud > >>> virtvboxd > >>> virtvzd > >>> > >>> And again for the secondary drivers > >>> > >>> virtinterfaced > >>> virtnetworkd > >>> virtnodedevd > >>> virtnwfilterd > >>> virtsecretd > >>> virtstoraged > >>> > >>> Finally to support IP connectivity, and also the legacy lbivirtd UNIX > >>> domain socket (for the old libvirt remote driver SSH tunnelling): > >>> > >>> virtproxyd > >>> > >>> The the sake of facilitating upgrades, the existing libvirtd still > >>> exists and works the same way it always has. > >>> > >>> You either run libvirtd, or you run the per-driver daemons, never both. > >> > >> What happens if you run both? > >> (I'll try to figure out by reviewing the rest of the code and/or testing) > > > > The drivers acquire an exclusive lock, causing the 2nd daemon to fail > > to startup > > > > $ ./src/libvirtd & > > > > $ ./src/virtqemud > > 2019-07-30 09:36:34.339+: 22809: info : libvirt version: 5.6.0 > > 2019-07-30 09:36:34.339+: 22809: info : hostname: > > dhcp-16-132.lcy.redhat.com > > 2019-07-30 09:36:34.339+: 22809: error : virPidFileAcquirePath:376 : > > Failed to acquire pid file '/run/user/501/libvirt/qemu/run/driver.pid': > > Resource temporarily unavailable > > 2019-07-30 09:36:34.339+: 22809: error : virStateInitialize:688 : > > Initialisation of QEMU state driver failed: Failed to acquire pid file > > '/run/user/501/libvirt/qemu/run/driver.pid': Resource temporarily > > unavailable > > 2019-07-30 09:36:34.339+: 22809: error : daemonRunStateInit:821 : > > Driver state initialisation failed > > > > > > The same works the other way around too > > > > $ ./src/virtqemud & > > > > $ ./src/libvirtd > > 2019-07-30 09:37:45.398+: 23109: info : libvirt version: 5.6.0 > > 2019-07-30 09:37:45.398+: 23109: info : hostname: > > dhcp-16-132.lcy.redhat.com > > 2019-07-30 09:37:45.398+: 23109: error : virPidFileAcquirePath:376 : > > Failed to acquire pid file '/run/user/501/libvirt/qemu/run/driver.pid': > > Resource temporarily unavailable > > 2019-07-30 09:37:45.398+: 23109: error : virStateInitialize:688 : > > Initialisation of QEMU state driver failed: Failed to acquire pid file > > '/run/user/501/libvirt/qemu/run/driver.pid': Resource temporarily > > unavailable > > 2019-07-30 09:37:45.398+: 23109: error : daemonRunStateInit:821 : > > Driver state initialisation failed > > > > > > > > the systemd unit files also have Conflicts rules which should prevent > > even getting that far > > Thanks for testing. But that can only work for one deamon which shares the > lock file > name with libvirtd. What about the other drivers? I guess they can’t all > share the > same lock file, or I missed something big in the design. Libvirt has many drivers (qemu, lxc, bhyve, libxl, storage, network). Each one of these acquires its own lock file under its own private directory - either /var/run/libvirt/$DRIVERNAME/driver.pid (as root) or /run/user/$UID/libvirt/$DRIVERNAME/run/driver.pid (as non-root). Libvirtd loads all the drivers, so will end up holding a lock for every driver it has loaded. Each new driver only loads a single driver and so will hold a single lock. The daemons themselves also hold their own locks to prevent the same daemon being started twice. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v3 00/48] Split the libvirtd daemon into per-driver daemons
> On 30 Jul 2019, at 11:38, Daniel P. Berrangé wrote: > > On Tue, Jul 30, 2019 at 11:05:25AM +0200, Christophe de Dinechin wrote: >> >> Daniel P. Berrangé writes: >> >>> This is what all the driver refactoring I've done has been about >>> enabling. >>> >>> We gain new daemons for each driver, for the primary virt drivers: >>> >>> virtlibxld >> >> virtxend? >> >>> virtlxcd >>> virtqemud >>> virtvboxd >>> virtvzd >>> >>> And again for the secondary drivers >>> >>> virtinterfaced >>> virtnetworkd >>> virtnodedevd >>> virtnwfilterd >>> virtsecretd >>> virtstoraged >>> >>> Finally to support IP connectivity, and also the legacy lbivirtd UNIX >>> domain socket (for the old libvirt remote driver SSH tunnelling): >>> >>> virtproxyd >>> >>> The the sake of facilitating upgrades, the existing libvirtd still >>> exists and works the same way it always has. >>> >>> You either run libvirtd, or you run the per-driver daemons, never both. >> >> What happens if you run both? >> (I'll try to figure out by reviewing the rest of the code and/or testing) > > The drivers acquire an exclusive lock, causing the 2nd daemon to fail > to startup > > $ ./src/libvirtd & > > $ ./src/virtqemud > 2019-07-30 09:36:34.339+: 22809: info : libvirt version: 5.6.0 > 2019-07-30 09:36:34.339+: 22809: info : hostname: > dhcp-16-132.lcy.redhat.com > 2019-07-30 09:36:34.339+: 22809: error : virPidFileAcquirePath:376 : > Failed to acquire pid file '/run/user/501/libvirt/qemu/run/driver.pid': > Resource temporarily unavailable > 2019-07-30 09:36:34.339+: 22809: error : virStateInitialize:688 : > Initialisation of QEMU state driver failed: Failed to acquire pid file > '/run/user/501/libvirt/qemu/run/driver.pid': Resource temporarily unavailable > 2019-07-30 09:36:34.339+: 22809: error : daemonRunStateInit:821 : Driver > state initialisation failed > > > The same works the other way around too > > $ ./src/virtqemud & > > $ ./src/libvirtd > 2019-07-30 09:37:45.398+: 23109: info : libvirt version: 5.6.0 > 2019-07-30 09:37:45.398+: 23109: info : hostname: > dhcp-16-132.lcy.redhat.com > 2019-07-30 09:37:45.398+: 23109: error : virPidFileAcquirePath:376 : > Failed to acquire pid file '/run/user/501/libvirt/qemu/run/driver.pid': > Resource temporarily unavailable > 2019-07-30 09:37:45.398+: 23109: error : virStateInitialize:688 : > Initialisation of QEMU state driver failed: Failed to acquire pid file > '/run/user/501/libvirt/qemu/run/driver.pid': Resource temporarily unavailable > 2019-07-30 09:37:45.398+: 23109: error : daemonRunStateInit:821 : Driver > state initialisation failed > > > > the systemd unit files also have Conflicts rules which should prevent > even getting that far Thanks for testing. But that can only work for one deamon which shares the lock file name with libvirtd. What about the other drivers? I guess they can’t all share the same lock file, or I missed something big in the design. (Sorry, still reviewing, so did not really have time to try it yet) > > > Regards, > Daniel > -- > |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o-https://fstop138.berrange.com :| > |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| > > -- > libvir-list mailing list > libvir-list@redhat.com > https://www.redhat.com/mailman/listinfo/libvir-list -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v3 00/48] Split the libvirtd daemon into per-driver daemons
On Tue, Jul 30, 2019 at 11:05:25AM +0200, Christophe de Dinechin wrote: > > Daniel P. Berrangé writes: > > > This is what all the driver refactoring I've done has been about > > enabling. > > > > We gain new daemons for each driver, for the primary virt drivers: > > > > virtlibxld > > virtxend? > > > virtlxcd > > virtqemud > > virtvboxd > > virtvzd > > > > And again for the secondary drivers > > > > virtinterfaced > > virtnetworkd > > virtnodedevd > > virtnwfilterd > > virtsecretd > > virtstoraged > > > > Finally to support IP connectivity, and also the legacy lbivirtd UNIX > > domain socket (for the old libvirt remote driver SSH tunnelling): > > > > virtproxyd > > > > The the sake of facilitating upgrades, the existing libvirtd still > > exists and works the same way it always has. > > > > You either run libvirtd, or you run the per-driver daemons, never both. > > What happens if you run both? > (I'll try to figure out by reviewing the rest of the code and/or testing) The drivers acquire an exclusive lock, causing the 2nd daemon to fail to startup $ ./src/libvirtd & $ ./src/virtqemud 2019-07-30 09:36:34.339+: 22809: info : libvirt version: 5.6.0 2019-07-30 09:36:34.339+: 22809: info : hostname: dhcp-16-132.lcy.redhat.com 2019-07-30 09:36:34.339+: 22809: error : virPidFileAcquirePath:376 : Failed to acquire pid file '/run/user/501/libvirt/qemu/run/driver.pid': Resource temporarily unavailable 2019-07-30 09:36:34.339+: 22809: error : virStateInitialize:688 : Initialisation of QEMU state driver failed: Failed to acquire pid file '/run/user/501/libvirt/qemu/run/driver.pid': Resource temporarily unavailable 2019-07-30 09:36:34.339+: 22809: error : daemonRunStateInit:821 : Driver state initialisation failed The same works the other way around too $ ./src/virtqemud & $ ./src/libvirtd 2019-07-30 09:37:45.398+: 23109: info : libvirt version: 5.6.0 2019-07-30 09:37:45.398+: 23109: info : hostname: dhcp-16-132.lcy.redhat.com 2019-07-30 09:37:45.398+: 23109: error : virPidFileAcquirePath:376 : Failed to acquire pid file '/run/user/501/libvirt/qemu/run/driver.pid': Resource temporarily unavailable 2019-07-30 09:37:45.398+: 23109: error : virStateInitialize:688 : Initialisation of QEMU state driver failed: Failed to acquire pid file '/run/user/501/libvirt/qemu/run/driver.pid': Resource temporarily unavailable 2019-07-30 09:37:45.398+: 23109: error : daemonRunStateInit:821 : Driver state initialisation failed the systemd unit files also have Conflicts rules which should prevent even getting that far Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v3 00/48] Split the libvirtd daemon into per-driver daemons
Daniel P. Berrangé writes: > This is what all the driver refactoring I've done has been about > enabling. > > We gain new daemons for each driver, for the primary virt drivers: > > virtlibxld virtxend? > virtlxcd > virtqemud > virtvboxd > virtvzd > > And again for the secondary drivers > > virtinterfaced > virtnetworkd > virtnodedevd > virtnwfilterd > virtsecretd > virtstoraged > > Finally to support IP connectivity, and also the legacy lbivirtd UNIX > domain socket (for the old libvirt remote driver SSH tunnelling): > > virtproxyd > > The the sake of facilitating upgrades, the existing libvirtd still > exists and works the same way it always has. > > You either run libvirtd, or you run the per-driver daemons, never both. What happens if you run both? (I'll try to figure out by reviewing the rest of the code and/or testing) > > The remote driver will look to see whether libvirtd is running to figure > out whether to connect to libvirtd or the new per-driver daemons. > > When auto-spawning daemons for nonroot users, we default to spawning the > per-driver daemons. > > This can be controlled with a UR parameter "?mode=direct|legacy|auto", > where 'direct' means per-driver and 'legacy' means libvirtd (or indirect > via virtproxyd if that's running). > > Changed in v3: > > - Add identity forwarding between daemons for polkit auth > - Make virtxend sockets conditional on Xen kernel > - Other misc review fixes > > Changed in v2: > > - Added systemd unit files for service & sockets, ensuring >conflicts with libvirtd > - Fixed proxy to actually handle probing of URIs (still not >quite perfect) > - Renamed virtlibxld to virtxend as "xen" is the user facing >name of the URI > - Lazy loading of secondary drivers, so connecting to virtqemud >does't auto-spawn all secondary driver daemons, until a >relevant API is actually invoked > - Actually generated config files / augeas files per daemon > - Abort daemon startup if driver fails to load > - Many other fixes > > A few nice to have things, but not merge blocking > > - MAYBE make it possible to disable build of libvirtd, or of the per-driver >daemons so downstream vendors can decide which to ship. Alternatively >they can just not include the binary in the package file list ? > - Tuning of the daemon defaults for worker threads to better suit >the fact that we have per-driver daemons > - More work on RPM packaging to allow install of per-driver daemosn "daemons" >without pulling in libvirtd too > > Daniel P. Berrangé (48): > build: make augeas-gentest.pl write to stdout > build: collapse rules adding augeas tests to CLEANFILES > build: create all augeas test files in same dir as their source > build: use a common rule for checking augeas test data files > build: centralize rule for handling generated config files > remote: stop trying to print help as giant blocks of text > remote: conditionalize socket names in libvirtd daemon > remote: conditionalize daemon name in libvirtd daemon > remote: conditionalize driver loading in libvirtd daemon > remote: conditionalize IP socket usage in libvirtd daemon > build: use @CONFIG@ instead of ::CONFIG:: in augeas tests > remote: conditionalize IP socket config in libvirtd.conf > remote: conditionalize IP socket config in augeas definitions > remote: refactor & rename variables for building libvirtd > build: don't hardcode /etc in the config related files > remote: reduce duplication in systemd unit file make rules into one > remote: conditionalize systemd socket unit files > remote: refactor how list of systemd unit files is built > remote: in per-driver daemons ensure that state initialize succeeds > remote: introduce virtproxyd daemon to handle IP connectivity > secret: introduce virtsecretd daemon > network: introduce virtnetworkd daemon > interface: introduce virtinterfaced daemon > storage: introduce virtstoraged daemon > nodedev: introduce virtnodedevd daemon > nwfilter: introduce virtnwfilterd daemon > libxl: introduce virtxend daemon > qemu: introduce virtqemud daemon > lxc: introduce virtlxcd daemon > vbox: introduce virtvboxd daemon > bhyve: introduce virtbhyved daemon > vz: introduce virtvzd daemon > admin: add ability to connect to the per-driver daemon sockets > remote: get rid of bogus ATTRIBUTE_UNUSED annotation client param > remote: change generated methods to not directly access connection > remote: fix lock ordering mistake in event registration > remote: change hand written methods to not directly access connection > remote: open secondary drivers via remote driver if needed > remote: handle autoprobing of driver within virtproxyd > remote: use enum helpers for parsing remote driver transport > remote: refactor the code for choosing the UNIX socket path > remote: enable connecting to the per-driver daemons > api: introduce virConnectSetIdentity fo
[libvirt] [PATCH v3 00/48] Split the libvirtd daemon into per-driver daemons
This is what all the driver refactoring I've done has been about enabling. We gain new daemons for each driver, for the primary virt drivers: virtlibxld virtlxcd virtqemud virtvboxd virtvzd And again for the secondary drivers virtinterfaced virtnetworkd virtnodedevd virtnwfilterd virtsecretd virtstoraged Finally to support IP connectivity, and also the legacy lbivirtd UNIX domain socket (for the old libvirt remote driver SSH tunnelling): virtproxyd The the sake of facilitating upgrades, the existing libvirtd still exists and works the same way it always has. You either run libvirtd, or you run the per-driver daemons, never both. The remote driver will look to see whether libvirtd is running to figure out whether to connect to libvirtd or the new per-driver daemons. When auto-spawning daemons for nonroot users, we default to spawning the per-driver daemons. This can be controlled with a UR parameter "?mode=direct|legacy|auto", where 'direct' means per-driver and 'legacy' means libvirtd (or indirect via virtproxyd if that's running). Changed in v3: - Add identity forwarding between daemons for polkit auth - Make virtxend sockets conditional on Xen kernel - Other misc review fixes Changed in v2: - Added systemd unit files for service & sockets, ensuring conflicts with libvirtd - Fixed proxy to actually handle probing of URIs (still not quite perfect) - Renamed virtlibxld to virtxend as "xen" is the user facing name of the URI - Lazy loading of secondary drivers, so connecting to virtqemud does't auto-spawn all secondary driver daemons, until a relevant API is actually invoked - Actually generated config files / augeas files per daemon - Abort daemon startup if driver fails to load - Many other fixes A few nice to have things, but not merge blocking - MAYBE make it possible to disable build of libvirtd, or of the per-driver daemons so downstream vendors can decide which to ship. Alternatively they can just not include the binary in the package file list ? - Tuning of the daemon defaults for worker threads to better suit the fact that we have per-driver daemons - More work on RPM packaging to allow install of per-driver daemosn without pulling in libvirtd too Daniel P. Berrangé (48): build: make augeas-gentest.pl write to stdout build: collapse rules adding augeas tests to CLEANFILES build: create all augeas test files in same dir as their source build: use a common rule for checking augeas test data files build: centralize rule for handling generated config files remote: stop trying to print help as giant blocks of text remote: conditionalize socket names in libvirtd daemon remote: conditionalize daemon name in libvirtd daemon remote: conditionalize driver loading in libvirtd daemon remote: conditionalize IP socket usage in libvirtd daemon build: use @CONFIG@ instead of ::CONFIG:: in augeas tests remote: conditionalize IP socket config in libvirtd.conf remote: conditionalize IP socket config in augeas definitions remote: refactor & rename variables for building libvirtd build: don't hardcode /etc in the config related files remote: reduce duplication in systemd unit file make rules into one remote: conditionalize systemd socket unit files remote: refactor how list of systemd unit files is built remote: in per-driver daemons ensure that state initialize succeeds remote: introduce virtproxyd daemon to handle IP connectivity secret: introduce virtsecretd daemon network: introduce virtnetworkd daemon interface: introduce virtinterfaced daemon storage: introduce virtstoraged daemon nodedev: introduce virtnodedevd daemon nwfilter: introduce virtnwfilterd daemon libxl: introduce virtxend daemon qemu: introduce virtqemud daemon lxc: introduce virtlxcd daemon vbox: introduce virtvboxd daemon bhyve: introduce virtbhyved daemon vz: introduce virtvzd daemon admin: add ability to connect to the per-driver daemon sockets remote: get rid of bogus ATTRIBUTE_UNUSED annotation client param remote: change generated methods to not directly access connection remote: fix lock ordering mistake in event registration remote: change hand written methods to not directly access connection remote: open secondary drivers via remote driver if needed remote: handle autoprobing of driver within virtproxyd remote: use enum helpers for parsing remote driver transport remote: refactor the code for choosing the UNIX socket path remote: enable connecting to the per-driver daemons api: introduce virConnectSetIdentity for pasing uid, gid, selinux info util: change identity class attribute names util: make generic identity accessors private util: storage identity attrs as virTypedParameter internally util: allow identity to be imported/exported as typed parameters remote: pass identity across to newly opened daemons .gitignore| 62 +- b