Re: [PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread David Gibson
On Thu, Feb 16, 2023 at 05:21:48PM +0100, Michal Prívozník wrote: > On 2/16/23 16:42, Jonathon Jongsma wrote: > > On 2/16/23 7:32 AM, Michal Privoznik wrote: > >> Passt has '--stderr' argument which makes it report error onto > >> stderr rather to system log. Unfortunately, it's currently > >> impo

Re: [libvirt PATCH v5 29/32] schema: add configuration for host verification of ssh disks

2023-02-16 Thread Jonathon Jongsma
On 2/16/23 10:45 AM, Peter Krempa wrote: On Tue, Feb 14, 2023 at 11:08:16 -0600, Jonathon Jongsma wrote: In order to make ssh disks usable, we need to be able to validate a remote host. To do this, add a xml element for ssh disks to allow the user to specify a location for a file that contains

Re: [libvirt PATCH v5 27/32] schema: add password configuration for ssh disk

2023-02-16 Thread Jonathon Jongsma
On 2/16/23 10:43 AM, Peter Krempa wrote: On Tue, Feb 14, 2023 at 11:08:14 -0600, Jonathon Jongsma wrote: Right now, ssh network disks are not usable. There is some basic support in libvirt that is meant to support disk chains that have backing disks located at ssh urls, but there is no real way

Re: [libvirt PATCH] docs: introduce a more interesting 404 error page

2023-02-16 Thread Daniel P . Berrangé
On Thu, Feb 16, 2023 at 04:21:33PM +0100, Peter Krempa wrote: > Additionally I've planned to convert also the 404 page to rST > eventually to go with the rest of the web. The only reason I didn't do > so yet is the use of which rST doesn't support and I didn't bother > to come up with a different

Re: [libvirt PATCH] docs: introduce a more interesting 404 error page

2023-02-16 Thread Andrea Bolognani
On Thu, Feb 16, 2023 at 02:53:13PM +, Daniel P. Berrangé wrote: > Our 404 error page is a little bit too boring. The solution is to > add more penguins ! > > This relies on MIT licensed javascript code imported from > > https://github.com/VincentGarreau/particles.js > > The image is extracted

Re: [PATCH v2 4/5] qemu_passt: Deduplicate passt killing code

2023-02-16 Thread Stefano Brivio
On Thu, 16 Feb 2023 17:38:47 +0100 Michal Prívozník wrote: > On 2/16/23 17:07, Stefano Brivio wrote: > > On Thu, 16 Feb 2023 14:32:51 +0100 > > Michal Privoznik wrote: > > > >> There are two places where we kill passt: > >> > >> 1) qemuPasstStop() - called transitively from qemuProcessStop(),

Re: [libvirt PATCH v5 32/32] qemu: implement keyfile auth for ssh disk with nbdkit

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:19 -0600, Jonathon Jongsma wrote: > For ssh disks that are served by nbdkit, we can support logging in with > an ssh key file. Pass the path to the configured key file and the > username to the nbdkit process. > > The key file may be password protected, and libvirt ca

Re: [libvirt PATCH v5 30/32] qemu: implement knownHosts for ssh disks with nbdkit

2023-02-16 Thread Peter Krempa
On Thu, Feb 16, 2023 at 17:50:50 +0100, Peter Krempa wrote: > On Tue, Feb 14, 2023 at 11:08:17 -0600, Jonathon Jongsma wrote: > > For ssh disks that are served by nbdkit, use the configured value for > > knownHosts and pass it to the nbdkit process. > > > > Signed-off-by: Jonathon Jongsma > > ---

Re: [libvirt PATCH v5 30/32] qemu: implement knownHosts for ssh disks with nbdkit

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:17 -0600, Jonathon Jongsma wrote: > For ssh disks that are served by nbdkit, use the configured value for > knownHosts and pass it to the nbdkit process. > > Signed-off-by: Jonathon Jongsma > --- > src/conf/domain_conf.c| 8 ++

Re: [PATCH v2 5/5] qemu_passt: Let passt write the PID file

2023-02-16 Thread Laine Stump
On 2/16/23 8:32 AM, Michal Privoznik wrote: The way we start passt currently is: we use virCommandSetPidFile() to use our virCommand machinery to acquire the PID file and leak opened FD into passt. Then, we use virPidFile*() APIs to read the PID file (which is needed when placing it into CGroups

Re: [PATCH v2 0/5] qemu_passt: Fix issues with PID file

2023-02-16 Thread Laine Stump
On 2/16/23 8:32 AM, Michal Privoznik wrote: This is a v2 of: https://listman.redhat.com/archives/libvir-list/2023-February/237731.html diff to v1: - Merged patches that were ACKed in v1, - Dropped 4/4 from the original series (the one that sets --foreground), and implemented a different appr

Re: [libvirt PATCH v5 29/32] schema: add configuration for host verification of ssh disks

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:16 -0600, Jonathon Jongsma wrote: > In order to make ssh disks usable, we need to be able to validate a > remote host. To do this, add a xml element for ssh disks to > allow the user to specify a location for a file that contains known host > keys. Implementation to f

Re: [libvirt PATCH v5 27/32] schema: add password configuration for ssh disk

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:14 -0600, Jonathon Jongsma wrote: > Right now, ssh network disks are not usable. There is some basic support > in libvirt that is meant to support disk chains that have backing disks > located at ssh urls, but there is no real way for a user to configure a > ssh-based

Re: [PATCH v2 4/5] qemu_passt: Deduplicate passt killing code

2023-02-16 Thread Laine Stump
On 2/16/23 8:32 AM, Michal Privoznik wrote: There are two places where we kill passt: 1) qemuPasstStop() - called transitively from qemuProcessStop(), 2) qemuPasstStart() - after failed start. Now, the code from 2) lack error preservation (so if there's another error during cleanup we might ove

Re: [PATCH v2 2/5] qemu_passt: Report passt's error on failed start

2023-02-16 Thread Laine Stump
On 2/16/23 8:32 AM, Michal Privoznik wrote: When starting passt, it may write something onto its stderr (convincing it to print even more is addressed later). Pass this string we read to user. Since we're not daemonizing passt anymore (see previous commit), we can let virCommand module do all th

Re: [PATCH v2 4/5] qemu_passt: Deduplicate passt killing code

2023-02-16 Thread Michal Prívozník
On 2/16/23 17:07, Stefano Brivio wrote: > On Thu, 16 Feb 2023 14:32:51 +0100 > Michal Privoznik wrote: > >> There are two places where we kill passt: >> >> 1) qemuPasstStop() - called transitively from qemuProcessStop(), >> 2) qemuPasstStart() - after failed start. >> >> Now, the code from 2) lac

Re: [PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread Laine Stump
On 2/16/23 11:27 AM, Michal Prívozník wrote: On 2/16/23 17:07, Stefano Brivio wrote: On Thu, 16 Feb 2023 14:32:50 +0100 Michal Privoznik wrote: Passt has '--stderr' argument which makes it report error onto stderr rather to system log. Unfortunately, it's currently impossible to use both '--l

Re: [PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread Stefano Brivio
On Thu, 16 Feb 2023 17:27:11 +0100 Michal Prívozník wrote: > On 2/16/23 17:07, Stefano Brivio wrote: > > On Thu, 16 Feb 2023 14:32:50 +0100 > > Michal Privoznik wrote: > > > >> Passt has '--stderr' argument which makes it report error onto > >> stderr rather to system log. Unfortunately, it's

Re: [PATCH v2 1/5] qemu_passt: Avoid double daemonizing passt

2023-02-16 Thread Laine Stump
On 2/16/23 8:32 AM, Michal Privoznik wrote: When passt is started, it daemonizes itself by default. There's no point in having our virCommand module daemonize it too. Signed-off-by: Michal Privoznik Reviewed-by: Laine Stump --- src/qemu/qemu_passt.c | 1 - 1 file changed, 1 deletion(-)

Re: [PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread Laine Stump
This is all unnecessary. The issue with error messages has been fixed upstream in passt (along with other logging issues), and just serves to unnecessarily complicate the code. On 2/16/23 8:32 AM, Michal Privoznik wrote: Passt has '--stderr' argument which makes it report error onto stderr rat

Re: [libvirt PATCH v5 26/32] qemu: try to connect to nbdkit early to detect errors

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:13 -0600, Jonathon Jongsma wrote: > When using nbdkit to serve a network disk source, the nbdkit process > will start and wait for an nbd connection before actually attempting to > connect to the (remote) disk location. Because of this, nbdkit will not > report an erro

Re: [libvirt PATCH v5 25/32] qemu: Monitor nbdkit process for exit

2023-02-16 Thread Peter Krempa
On Thu, Feb 16, 2023 at 17:23:19 +0100, Peter Krempa wrote: > On Tue, Feb 14, 2023 at 11:08:12 -0600, Jonathon Jongsma wrote: > > Adds the ability to monitor the nbdkit process so that we can take > > action in case the child exits unexpectedly. > > > > When the nbdkit process exits, we pause the

Re: [libvirt PATCH v5 21/32] qemu: use nbdkit to serve network disks if available

2023-02-16 Thread Peter Krempa
On Thu, Feb 16, 2023 at 10:13:27 -0600, Jonathon Jongsma wrote: > On 2/16/23 9:55 AM, Peter Krempa wrote: > > On Tue, Feb 14, 2023 at 11:08:08 -0600, Jonathon Jongsma wrote: > > > For virStorageSource objects that contain an nbdkitProcess, start that > > > nbdkit process to serve that network drive

Re: [PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread Michal Prívozník
On 2/16/23 17:07, Stefano Brivio wrote: > On Thu, 16 Feb 2023 14:32:50 +0100 > Michal Privoznik wrote: > >> Passt has '--stderr' argument which makes it report error onto >> stderr rather to system log. Unfortunately, it's currently >> impossible to use both '--log-file' and '--stderr', so pass t

Re: [libvirt PATCH v5 25/32] qemu: Monitor nbdkit process for exit

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:12 -0600, Jonathon Jongsma wrote: > Adds the ability to monitor the nbdkit process so that we can take > action in case the child exits unexpectedly. > > When the nbdkit process exits, we pause the vm, restart nbdkit, and then > resume the vm. This allows the vm to co

Re: [PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread Michal Prívozník
On 2/16/23 16:42, Jonathon Jongsma wrote: > On 2/16/23 7:32 AM, Michal Privoznik wrote: >> Passt has '--stderr' argument which makes it report error onto >> stderr rather to system log. Unfortunately, it's currently >> impossible to use both '--log-file' and '--stderr', so pass the >> latter only i

Re: [libvirt PATCH v5 21/32] qemu: use nbdkit to serve network disks if available

2023-02-16 Thread Jonathon Jongsma
On 2/16/23 9:55 AM, Peter Krempa wrote: On Tue, Feb 14, 2023 at 11:08:08 -0600, Jonathon Jongsma wrote: For virStorageSource objects that contain an nbdkitProcess, start that nbdkit process to serve that network drive and then pass the nbdkit socket to qemu rather than sending the network url to

Re: [PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread Stefano Brivio
On Thu, 16 Feb 2023 14:32:50 +0100 Michal Privoznik wrote: > Passt has '--stderr' argument which makes it report error onto > stderr rather to system log. Unfortunately, it's currently > impossible to use both '--log-file' and '--stderr', so pass the > latter only if the former isn't passed. Then

Re: [PATCH v2 2/5] qemu_passt: Report passt's error on failed start

2023-02-16 Thread Stefano Brivio
On Thu, 16 Feb 2023 14:32:49 +0100 Michal Privoznik wrote: > When starting passt, it may write something onto its stderr > (convincing it to print even more is addressed later). Pass this > string we read to user. > > Since we're not daemonizing passt anymore (see previous commit), > we can let

Re: [PATCH v2 1/5] qemu_passt: Avoid double daemonizing passt

2023-02-16 Thread Stefano Brivio
On Thu, 16 Feb 2023 14:32:48 +0100 Michal Privoznik wrote: > When passt is started, it daemonizes itself by default. There's > no point in having our virCommand module daemonize it too. > > Signed-off-by: Michal Privoznik > --- > src/qemu/qemu_passt.c | 1 - > 1 file changed, 1 deletion(-) >

Re: [PATCH v2 4/5] qemu_passt: Deduplicate passt killing code

2023-02-16 Thread Stefano Brivio
On Thu, 16 Feb 2023 14:32:51 +0100 Michal Privoznik wrote: > There are two places where we kill passt: > > 1) qemuPasstStop() - called transitively from qemuProcessStop(), > 2) qemuPasstStart() - after failed start. > > Now, the code from 2) lack error preservation (so if there's > another erro

Re: [libvirt PATCH v5 23/32] tests: add tests for nbdkit invocation

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:10 -0600, Jonathon Jongsma wrote: > We were testing the arguments that were being passed to qemu when a disk > was being served by nbdkit, but the arguments used to start nbdkit > itself were not testable. This adds a test to ensure that we're invoking > nbdkit correct

Re: [libvirt PATCH v5 21/32] qemu: use nbdkit to serve network disks if available

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:08 -0600, Jonathon Jongsma wrote: > For virStorageSource objects that contain an nbdkitProcess, start that > nbdkit process to serve that network drive and then pass the nbdkit > socket to qemu rather than sending the network url to qemu directly. > > Signed-off-by: J

Re: [libvirt PATCH] docs: introduce a more interesting 404 error page

2023-02-16 Thread Jiri Denemark
On Thu, Feb 16, 2023 at 14:53:13 +, Daniel P. Berrangé wrote: > Our 404 error page is a little bit too boring. The solution is to > add more penguins ! > > This relies on MIT licensed javascript code imported from > > https://github.com/VincentGarreau/particles.js > > The image is extracte

Re: [PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread Jonathon Jongsma
On 2/16/23 7:32 AM, Michal Privoznik wrote: Passt has '--stderr' argument which makes it report error onto stderr rather to system log. Unfortunately, it's currently impossible to use both '--log-file' and '--stderr', so pass the latter only if the former isn't passed. Then, use the stderr to pro

Re: [libvirt PATCH v5 14/32] qemu: move qemuProcessReadLog() to qemuLogContext

2023-02-16 Thread Jonathon Jongsma
On 2/16/23 8:56 AM, Peter Krempa wrote: On Tue, Feb 14, 2023 at 11:08:01 -0600, Jonathon Jongsma wrote: This code can be used by the nbdkit implementation for reading back filtered log data for error reporting. Move it to qemuLogContext so that it can be shared. Renamed to qemuLogContextReadFilt

Re: [libvirt PATCH] docs: introduce a more interesting 404 error page

2023-02-16 Thread Peter Krempa
On Thu, Feb 16, 2023 at 14:53:13 +, Daniel P. Berrangé wrote: > Our 404 error page is a little bit too boring. The solution is to > add more penguins ! > > This relies on MIT licensed javascript code imported from > > https://github.com/VincentGarreau/particles.js > > The image is extracte

Re: [libvirt PATCH v5 20/32] qemu: pass sensitive data to nbdkit via pipe

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:07 -0600, Jonathon Jongsma wrote: > Rather than passing passwords and cookies (which could contain > passwords) to nbdkit via commandline arguments, use the alternate format > that nbdkit supports where we can specify a file descriptor which nbdkit > will read to get t

Re: [libvirt PATCH v5 19/32] util: secure erase virCommand send buffers

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:06 -0600, Jonathon Jongsma wrote: > All users of virCommandSetSendBuffer() are using it to send sensitive > data to a child process. So, since these buffers contain sensitive > information, clear it with virSecureErase(). > > Signed-off-by: Jonathon Jongsma > --- >

Re: [libvirt PATCH v5 18/32] qemu: include nbdkit state in private xml

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:05 -0600, Jonathon Jongsma wrote: > Add xml to the private data for a disk source to represent the nbdkit > process so that the state can be re-created if the libvirt daemon is > restarted. Format: > > > /path/to/nbdkit.pid > /path/to/nbdkit.socket >

Re: [libvirt PATCH v5 15/32] qemu: log error output from nbdkit

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:02 -0600, Jonathon Jongsma wrote: > log stderr and stdout from nbdkit into its own log so that > nbdkit-related issues can be debugged more easily. > > Signed-off-by: Jonathon Jongsma > --- > src/qemu/qemu_nbdkit.c | 30 +- > 1 file chang

Re: [libvirt PATCH v5 14/32] qemu: move qemuProcessReadLog() to qemuLogContext

2023-02-16 Thread Peter Krempa
On Tue, Feb 14, 2023 at 11:08:01 -0600, Jonathon Jongsma wrote: > This code can be used by the nbdkit implementation for reading back > filtered log data for error reporting. Move it to qemuLogContext so that > it can be shared. Renamed to qemuLogContextReadFiltered(). > > Signed-off-by: Jonathon

[libvirt PATCH] docs: introduce a more interesting 404 error page

2023-02-16 Thread Daniel P . Berrangé
Our 404 error page is a little bit too boring. The solution is to add more penguins ! This relies on MIT licensed javascript code imported from https://github.com/VincentGarreau/particles.js The image is extracted from the existing libvirt logo SVG file. Signed-off-by: Daniel P. Berrangé ---

[PATCH v2 3/5] qemu_passt: Make passt report errors to stderr whenever possible

2023-02-16 Thread Michal Privoznik
Passt has '--stderr' argument which makes it report error onto stderr rather to system log. Unfortunately, it's currently impossible to use both '--log-file' and '--stderr', so pass the latter only if the former isn't passed. Then, use the stderr to produce more user friendly error message on faile

[PATCH v2 4/5] qemu_passt: Deduplicate passt killing code

2023-02-16 Thread Michal Privoznik
There are two places where we kill passt: 1) qemuPasstStop() - called transitively from qemuProcessStop(), 2) qemuPasstStart() - after failed start. Now, the code from 2) lack error preservation (so if there's another error during cleanup we might overwrite the original error). Therefore, move th

[PATCH v2 2/5] qemu_passt: Report passt's error on failed start

2023-02-16 Thread Michal Privoznik
When starting passt, it may write something onto its stderr (convincing it to print even more is addressed later). Pass this string we read to user. Since we're not daemonizing passt anymore (see previous commit), we can let virCommand module do all the heavy lifting and switch to virCommandSetErr

[PATCH v2 1/5] qemu_passt: Avoid double daemonizing passt

2023-02-16 Thread Michal Privoznik
When passt is started, it daemonizes itself by default. There's no point in having our virCommand module daemonize it too. Signed-off-by: Michal Privoznik --- src/qemu/qemu_passt.c | 1 - 1 file changed, 1 deletion(-) diff --git a/src/qemu/qemu_passt.c b/src/qemu/qemu_passt.c index 78830fdc26..

[PATCH v2 5/5] qemu_passt: Let passt write the PID file

2023-02-16 Thread Michal Privoznik
The way we start passt currently is: we use virCommandSetPidFile() to use our virCommand machinery to acquire the PID file and leak opened FD into passt. Then, we use virPidFile*() APIs to read the PID file (which is needed when placing it into CGroups or killing it). But this does not fly really b

[PATCH v2 0/5] qemu_passt: Fix issues with PID file

2023-02-16 Thread Michal Privoznik
This is a v2 of: https://listman.redhat.com/archives/libvir-list/2023-February/237731.html diff to v1: - Merged patches that were ACKed in v1, - Dropped 4/4 from the original series (the one that sets --foreground), and implemented a different approach Michal Prívozník (5): qemu_passt: Avoid

Re: [PULL 0/5] Misc next patches

2023-02-16 Thread Peter Maydell
On Wed, 15 Feb 2023 at 17:48, Daniel P. Berrangé wrote: > > The following changes since commit 6a50f64ca01d0a7b97f14f069762bfd88160f31e: > > Merge tag 'pull-request-2023-02-14' of https://gitlab.com/thuth/qemu into > staging (2023-02-14 14:46:10 +) > > are available in the Git repository at

Re: [PATCH 4/4] qemu_passt: Don't let passt fork off

2023-02-16 Thread Stefano Brivio
On Thu, 16 Feb 2023 09:52:27 +0100 Michal Prívozník wrote: > On 2/15/23 19:30, Stefano Brivio wrote: > > On Wed, 15 Feb 2023 18:04:56 +0100 > > Michal Prívozník wrote: > > > >> On 2/15/23 08:50, Laine Stump wrote: > >>> On 2/14/23 8:02 AM, Stefano Brivio wrote: > On Tue, 14 Feb 202

Re: [PATCH 4/4] qemu_passt: Don't let passt fork off

2023-02-16 Thread Peter Krempa
On Thu, Feb 16, 2023 at 09:52:27 +0100, Michal Prívozník wrote: > On 2/15/23 19:30, Stefano Brivio wrote: > > On Wed, 15 Feb 2023 18:04:56 +0100 > > Michal Prívozník wrote: > >> On 2/15/23 08:50, Laine Stump wrote: [...] > > *should* already cover all the cases where libvirt is interested in > >

Re: [PATCH 4/4] qemu_passt: Don't let passt fork off

2023-02-16 Thread Michal Prívozník
On 2/15/23 19:30, Stefano Brivio wrote: > On Wed, 15 Feb 2023 18:04:56 +0100 > Michal Prívozník wrote: > >> On 2/15/23 08:50, Laine Stump wrote: >>> On 2/14/23 8:02 AM, Stefano Brivio wrote: On Tue, 14 Feb 2023 12:51:22 +0100 Michal Privoznik wrote: > When passt starts it