Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

On Thu, 26 Apr 2018 09:55:40 -0500
Eric Blake  wrote:

> On 04/26/2018 09:39 AM, Igor Mammedov wrote:
> 
> >>> I'll try to come up with a text for qemu-doc.texi, not about
> >>> deprecating -S but about when --preconfig should be used vs -S
> >>> and where to get list of commands that could be used at preconfig state.  
> >>>   
> >>
> >> Sounds good to me.  Thanks!  
> > how about something like this:
> > 
> > diff --git a/qemu-tech.texi b/qemu-tech.texi
> > index 52a56ae..6951258 100644
> > --- a/qemu-tech.texi
> > +++ b/qemu-tech.texi
> > @@ -5,6 +5,7 @@
> >  * CPU emulation::
> >  * Translator Internals::
> >  * QEMU compared to other emulators::
> > +* Managed start up options::
> >  * Bibliography::
> >  @end menu
> >  
> > @@ -314,6 +315,44 @@ VirtualBox [9], Xen [10] and KVM [11] are based on 
> > QEMU. QEMU-SystemC
> >  [12] uses QEMU to simulate a system where some hardware devices are
> >  developed in SystemC.
> >  
> > +@node Managed start up options
> > +@section Managed start up options
> > +
> > +In system mode emulation, it's possible to create VM in paused state using
> > +-S command line option. In this state the machine is completely initialized
> > +according to command line options and ready to execute VM code but VCPU 
> > threads
> > +are not executing any code. VM state in this paused state depends on way 
> > QEMU
> > +was started. It could be in:
> > +@table @asis
> > +@item initial state (after reset/power on state)
> > +@item with direct kernel loading initial state could be ammended to 
> > execute  
> 
> s/ammended/amended/
> 
> > +code loaded by QEMU in VM's RAM and with incomming migration  
> 
> s/incomming/incoming/
> 
> > +@item with incomming migrartion, initial state will by ammended by the 
> > migrated  
> 
> and again, for both words
> s/migrartion/migration/
> 
> > +machine state after migration completes.
> > +@end table
> > +
> > +This paused state is typically used by users to query machine state and/or
> > +additionally configure machine (hotplug devices) in runtime before allowing
> > +VM code to run.
> > +
> > +However at -S pause point it's impossible to configure options that affect
> > +initial VM creation (like: -smp/-m/-numa ...) or cold plug devices. That's
> > +when -preconfig command line option should be used. It allows to pause  
> 
> s/to pause/pausing/
> 
> > +QEMU before initial VM creation in preconfig state, query being created
> > +VM at runtime and configure start up options depending on previous query  
> 
> Reads awkwardly.  Maybe:
> 
> It allows pausing QEMU before the initial VM creation, in a new
> preconfig state, where additional queries and configuration can be
> performed via QMP before moving on to the resulting configuration startup.
> 
> > +results. In preconfig state QEMU allows to configure VM only via QMP 
> > monitor
> > +with a limited command set which doesn't depend on completely initialized
> > +machine, which includes but not limited to:  
> 
> In the preconfig state, QEMU only allows a limited set of commands over
> the QMP monitor, where the commands do not depend on an initialized
> machine, including but not limited to:

thanks for review,
I amended patches as you suggested.
I plan to respin v6 today with these changes since exit-preconfig
affected several patches in series.

> 
> > +@table @asis
> > +@item qmp_capabilities
> > +@item query-qmp-schema
> > +@item query-commands
> > +@item query-status
> > +@end table
> > +The full list of commands is in QMP schema which could be queried with
> > +query-qmp-schema, where commands supported at preconfig state have option
> > +'allowed-in-preconfig' set to true.
> > +
> >  @node Bibliography
> >  @section Bibliography
> >   
> 
> >> A separate command would have less room for ambiguity.  
> > I've added following instead of reusing 'cont':
> > 
> > ##  
> >  
> > # @exit-preconfig:  
> >
> 
> This should definitely be mentioned in the docs section of commands
> permitted during preconfig.
done

> 
> > #   
> >  
> > # Exit from "preconfig" state   
> >  
> > #   
> >  
> > # Since 2.13
> >  
> > #   
> >  
> > # Returns: nothing  
> >  
> > #   
> >  
> > # Notes: Command makes QEMU exit from preconfig state and proceeds with 
> >  
> > # VM initialization using configuration data provided on command line   
> >  
> > # and via 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Eric Blake]

On 04/26/2018 09:39 AM, Igor Mammedov wrote:

>>> I'll try to come up with a text for qemu-doc.texi, not about
>>> deprecating -S but about when --preconfig should be used vs -S
>>> and where to get list of commands that could be used at preconfig state.  
>>
>> Sounds good to me.  Thanks!
> how about something like this:
> 
> diff --git a/qemu-tech.texi b/qemu-tech.texi
> index 52a56ae..6951258 100644
> --- a/qemu-tech.texi
> +++ b/qemu-tech.texi
> @@ -5,6 +5,7 @@
>  * CPU emulation::
>  * Translator Internals::
>  * QEMU compared to other emulators::
> +* Managed start up options::
>  * Bibliography::
>  @end menu
>  
> @@ -314,6 +315,44 @@ VirtualBox [9], Xen [10] and KVM [11] are based on QEMU. 
> QEMU-SystemC
>  [12] uses QEMU to simulate a system where some hardware devices are
>  developed in SystemC.
>  
> +@node Managed start up options
> +@section Managed start up options
> +
> +In system mode emulation, it's possible to create VM in paused state using
> +-S command line option. In this state the machine is completely initialized
> +according to command line options and ready to execute VM code but VCPU 
> threads
> +are not executing any code. VM state in this paused state depends on way QEMU
> +was started. It could be in:
> +@table @asis
> +@item initial state (after reset/power on state)
> +@item with direct kernel loading initial state could be ammended to execute

s/ammended/amended/

> +code loaded by QEMU in VM's RAM and with incomming migration

s/incomming/incoming/

> +@item with incomming migrartion, initial state will by ammended by the 
> migrated

and again, for both words
s/migrartion/migration/

> +machine state after migration completes.
> +@end table
> +
> +This paused state is typically used by users to query machine state and/or
> +additionally configure machine (hotplug devices) in runtime before allowing
> +VM code to run.
> +
> +However at -S pause point it's impossible to configure options that affect
> +initial VM creation (like: -smp/-m/-numa ...) or cold plug devices. That's
> +when -preconfig command line option should be used. It allows to pause

s/to pause/pausing/

> +QEMU before initial VM creation in preconfig state, query being created
> +VM at runtime and configure start up options depending on previous query

Reads awkwardly.  Maybe:

It allows pausing QEMU before the initial VM creation, in a new
preconfig state, where additional queries and configuration can be
performed via QMP before moving on to the resulting configuration startup.

> +results. In preconfig state QEMU allows to configure VM only via QMP monitor
> +with a limited command set which doesn't depend on completely initialized
> +machine, which includes but not limited to:

In the preconfig state, QEMU only allows a limited set of commands over
the QMP monitor, where the commands do not depend on an initialized
machine, including but not limited to:

> +@table @asis
> +@item qmp_capabilities
> +@item query-qmp-schema
> +@item query-commands
> +@item query-status
> +@end table
> +The full list of commands is in QMP schema which could be queried with
> +query-qmp-schema, where commands supported at preconfig state have option
> +'allowed-in-preconfig' set to true.
> +
>  @node Bibliography
>  @section Bibliography
> 

>> A separate command would have less room for ambiguity.
> I've added following instead of reusing 'cont':
> 
> ##
>
> # @exit-preconfig:
>

This should definitely be mentioned in the docs section of commands
permitted during preconfig.

> # 
>
> # Exit from "preconfig" state 
>
> # 
>
> # Since 2.13  
>
> # 
>
> # Returns: nothing
>
> # 
>
> # Notes: Command makes QEMU exit from preconfig state and proceeds with   
>
> # VM initialization using configuration data provided on command line 
>
> # and via QMP monitor at preconfig state. Command is available only at
>
> # preconfig state (i.e. if --preconfig command line option).  
>
> # 
>
> # Example:
>
> # 
>
> # -> { "execute": "exit-preconfig" }  
>
> # <- { "return": {} } 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

On Mon, 23 Apr 2018 17:45:31 -0300
Eduardo Habkost  wrote:

> On Mon, Apr 23, 2018 at 06:55:14PM +0200, Igor Mammedov wrote:
> > On Mon, 23 Apr 2018 10:05:54 -0300
> > Eduardo Habkost  wrote:
> >   
> > > On Mon, Apr 23, 2018 at 11:50:16AM +0200, Igor Mammedov wrote:  
> > > > On Fri, 20 Apr 2018 08:31:18 +0200
> > > > Markus Armbruster  wrote:
> > > > 
> > > > > Eduardo Habkost  writes:
> > > > > 
> > > > > > On Thu, Apr 19, 2018 at 10:00:04AM +0200, Igor Mammedov wrote:  
> > > > > >> On Wed, 18 Apr 2018 09:08:30 +0200
> > > > > >> Markus Armbruster  wrote:
> > > > > >>   
> > > > > >> > Eduardo Habkost  writes:
> > > > > >> >   
> > > > > >> > > On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote: 
> > > > > >> > >
> > > > > >> > >> On Tue, 17 Apr 2018 11:27:39 -0300
> > > > > >> > >> Eduardo Habkost  wrote:
> > > > > >> > >> 
> > > > > >> > >> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster 
> > > > > >> > >> > wrote:
> > > > > >> > >> > > Igor Mammedov  writes:
> > > > > >> > >> > > 
> > > > > >> > >> > > [...]  
> > > > > >> > >> > > > Series allows to configure NUMA mapping at runtime 
> > > > > >> > >> > > > using QMP
> > > > > >> > >> > > > interface. For that to happen it introduces a new 
> > > > > >> > >> > > > '-preconfig' CLI option
> > > > > >> > >> > > > which allows to pause QEMU before machine_init() is run 
> > > > > >> > >> > > > and
> > > > > >> > >> > > > adds new set-numa-node QMP command which in conjunction 
> > > > > >> > >> > > > with
> > > > > >> > >> > > > query-hotpluggable-cpus allows to configure NUMA 
> > > > > >> > >> > > > mapping for cpus.
> > > > > >> > >> > > >
> > > > > >> > >> > > > Later we can modify other commands to run early, for 
> > > > > >> > >> > > > example device_add.
> > > > > >> > >> > > > I recall SPAPR had problem when libvirt started QEMU 
> > > > > >> > >> > > > with -S and, while it's
> > > > > >> > >> > > > paused, added CPUs with device_add. Intent was to 
> > > > > >> > >> > > > coldplug CPUs (but at that
> > > > > >> > >> > > > stage it's considered hotplug already), so SPAPR had to 
> > > > > >> > >> > > > work around the issue.  
> > > > > >> > >> > > 
> > > > > >> > >> > > That instance is just stupidity / laziness, I think: we 
> > > > > >> > >> > > consider any
> > > > > >> > >> > > plug after machine creation a hot plug.  Real machines 
> > > > > >> > >> > > remain cold until
> > > > > >> > >> > > you press the power button.  Our virtual machines should 
> > > > > >> > >> > > remain cold
> > > > > >> > >> > > until they start running, i.e. with -S until the first 
> > > > > >> > >> > > "cont".
> > > > > >> > >> It probably would be too risky to change semantics of -S from 
> > > > > >> > >> hotplug to coldplug.
> > > > > >> > >> But even if we were easy it won't matter in case if dynamic 
> > > > > >> > >> configuration
> > > > > >> > >> done properly. More on it below.
> > > > > >> > >> 
> > > > > >> > >> > > I vaguely remember me asking this before, but your answer 
> > > > > >> > >> > > didn't make it
> > > > > >> > >> > > into this cover letter, which gives me a pretext to ask 
> > > > > >> > >> > > again instead of
> > > > > >> > >> > > looking it up in the archives: what exactly prevents us 
> > > > > >> > >> > > from keeping the
> > > > > >> > >> > > machine cold enough for numa configuration until the 
> > > > > >> > >> > > first "cont"?  
> > > > > >> > >> > 
> > > > > >> > >> > I also think this would be better, but it seems to be 
> > > > > >> > >> > difficult
> > > > > >> > >> > in practice, see:
> > > > > >> > >> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
> > > > > >> > >> > 
> > > > > >> > >> 
> > > > > >> > >> In addition to Eduardo's reply, here is what I've answered 
> > > > > >> > >> back
> > > > > >> > >> when you've asked question the 1st time (v2 late at -S pause 
> > > > > >> > >> point reconfig):
> > > > > >> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
> > > > > >> > >> 
> > > > > >> > >> In short:
> > > > > >> > >> I think it's wrong in general doing fixups after machine is 
> > > > > >> > >> build
> > > > > >> > >> instead of getting correct configuration before building 
> > > > > >> > >> machine.
> > > > > >> > >> That's going to be complex and fragile and might be hard to 
> > > > > >> > >> do at
> > > > > >> > >> all depending on what we are fixing up.
> > > > > >> > >
> > > > > >> > > What "building the machine" should mean, exactly, for external
> > > > > >> > > users?  
> > > > > >> under "building machine", I've meant machine_run_board_init()
> > > > > >> and all follow up steps to machine_done stage.
> > > > > >>   
> > > > > >> > > The main question I'd 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Eduardo Habkost]

On Mon, Apr 23, 2018 at 06:55:14PM +0200, Igor Mammedov wrote:
> On Mon, 23 Apr 2018 10:05:54 -0300
> Eduardo Habkost  wrote:
> 
> > On Mon, Apr 23, 2018 at 11:50:16AM +0200, Igor Mammedov wrote:
> > > On Fri, 20 Apr 2018 08:31:18 +0200
> > > Markus Armbruster  wrote:
> > >   
> > > > Eduardo Habkost  writes:
> > > >   
> > > > > On Thu, Apr 19, 2018 at 10:00:04AM +0200, Igor Mammedov wrote:
> > > > >> On Wed, 18 Apr 2018 09:08:30 +0200
> > > > >> Markus Armbruster  wrote:
> > > > >> 
> > > > >> > Eduardo Habkost  writes:
> > > > >> > 
> > > > >> > > On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote:   
> > > > >> > >
> > > > >> > >> On Tue, 17 Apr 2018 11:27:39 -0300
> > > > >> > >> Eduardo Habkost  wrote:
> > > > >> > >>   
> > > > >> > >> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster 
> > > > >> > >> > wrote:  
> > > > >> > >> > > Igor Mammedov  writes:
> > > > >> > >> > > 
> > > > >> > >> > > [...]
> > > > >> > >> > > > Series allows to configure NUMA mapping at runtime using 
> > > > >> > >> > > > QMP
> > > > >> > >> > > > interface. For that to happen it introduces a new 
> > > > >> > >> > > > '-preconfig' CLI option
> > > > >> > >> > > > which allows to pause QEMU before machine_init() is run 
> > > > >> > >> > > > and
> > > > >> > >> > > > adds new set-numa-node QMP command which in conjunction 
> > > > >> > >> > > > with
> > > > >> > >> > > > query-hotpluggable-cpus allows to configure NUMA mapping 
> > > > >> > >> > > > for cpus.
> > > > >> > >> > > >
> > > > >> > >> > > > Later we can modify other commands to run early, for 
> > > > >> > >> > > > example device_add.
> > > > >> > >> > > > I recall SPAPR had problem when libvirt started QEMU with 
> > > > >> > >> > > > -S and, while it's
> > > > >> > >> > > > paused, added CPUs with device_add. Intent was to 
> > > > >> > >> > > > coldplug CPUs (but at that
> > > > >> > >> > > > stage it's considered hotplug already), so SPAPR had to 
> > > > >> > >> > > > work around the issue.
> > > > >> > >> > > 
> > > > >> > >> > > That instance is just stupidity / laziness, I think: we 
> > > > >> > >> > > consider any
> > > > >> > >> > > plug after machine creation a hot plug.  Real machines 
> > > > >> > >> > > remain cold until
> > > > >> > >> > > you press the power button.  Our virtual machines should 
> > > > >> > >> > > remain cold
> > > > >> > >> > > until they start running, i.e. with -S until the first 
> > > > >> > >> > > "cont".  
> > > > >> > >> It probably would be too risky to change semantics of -S from 
> > > > >> > >> hotplug to coldplug.
> > > > >> > >> But even if we were easy it won't matter in case if dynamic 
> > > > >> > >> configuration
> > > > >> > >> done properly. More on it below.
> > > > >> > >>   
> > > > >> > >> > > I vaguely remember me asking this before, but your answer 
> > > > >> > >> > > didn't make it
> > > > >> > >> > > into this cover letter, which gives me a pretext to ask 
> > > > >> > >> > > again instead of
> > > > >> > >> > > looking it up in the archives: what exactly prevents us 
> > > > >> > >> > > from keeping the
> > > > >> > >> > > machine cold enough for numa configuration until the first 
> > > > >> > >> > > "cont"?
> > > > >> > >> > 
> > > > >> > >> > I also think this would be better, but it seems to be 
> > > > >> > >> > difficult
> > > > >> > >> > in practice, see:
> > > > >> > >> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
> > > > >> > >> >   
> > > > >> > >> 
> > > > >> > >> In addition to Eduardo's reply, here is what I've answered back
> > > > >> > >> when you've asked question the 1st time (v2 late at -S pause 
> > > > >> > >> point reconfig):
> > > > >> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
> > > > >> > >> 
> > > > >> > >> In short:
> > > > >> > >> I think it's wrong in general doing fixups after machine is 
> > > > >> > >> build
> > > > >> > >> instead of getting correct configuration before building 
> > > > >> > >> machine.
> > > > >> > >> That's going to be complex and fragile and might be hard to do 
> > > > >> > >> at
> > > > >> > >> all depending on what we are fixing up.  
> > > > >> > >
> > > > >> > > What "building the machine" should mean, exactly, for external
> > > > >> > > users?
> > > > >> under "building machine", I've meant machine_run_board_init()
> > > > >> and all follow up steps to machine_done stage.
> > > > >> 
> > > > >> > > The main question I'd like to see answered is: why exactly we
> > > > >> > > must "build" the machine before the first "cont" is issued when
> > > > >> > > using -S?  Why can't we delay everything to "cont" when using 
> > > > >> > > -S?  
> > > > >> Nor sure what question is about,
> > > > >> Did you mean if it were possible to postpone 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

On Mon, 23 Apr 2018 10:05:54 -0300
Eduardo Habkost  wrote:

> On Mon, Apr 23, 2018 at 11:50:16AM +0200, Igor Mammedov wrote:
> > On Fri, 20 Apr 2018 08:31:18 +0200
> > Markus Armbruster  wrote:
> >   
> > > Eduardo Habkost  writes:
> > >   
> > > > On Thu, Apr 19, 2018 at 10:00:04AM +0200, Igor Mammedov wrote:
> > > >> On Wed, 18 Apr 2018 09:08:30 +0200
> > > >> Markus Armbruster  wrote:
> > > >> 
> > > >> > Eduardo Habkost  writes:
> > > >> > 
> > > >> > > On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote: 
> > > >> > >  
> > > >> > >> On Tue, 17 Apr 2018 11:27:39 -0300
> > > >> > >> Eduardo Habkost  wrote:
> > > >> > >>   
> > > >> > >> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster 
> > > >> > >> > wrote:  
> > > >> > >> > > Igor Mammedov  writes:
> > > >> > >> > > 
> > > >> > >> > > [...]
> > > >> > >> > > > Series allows to configure NUMA mapping at runtime using QMP
> > > >> > >> > > > interface. For that to happen it introduces a new 
> > > >> > >> > > > '-preconfig' CLI option
> > > >> > >> > > > which allows to pause QEMU before machine_init() is run and
> > > >> > >> > > > adds new set-numa-node QMP command which in conjunction with
> > > >> > >> > > > query-hotpluggable-cpus allows to configure NUMA mapping 
> > > >> > >> > > > for cpus.
> > > >> > >> > > >
> > > >> > >> > > > Later we can modify other commands to run early, for 
> > > >> > >> > > > example device_add.
> > > >> > >> > > > I recall SPAPR had problem when libvirt started QEMU with 
> > > >> > >> > > > -S and, while it's
> > > >> > >> > > > paused, added CPUs with device_add. Intent was to coldplug 
> > > >> > >> > > > CPUs (but at that
> > > >> > >> > > > stage it's considered hotplug already), so SPAPR had to 
> > > >> > >> > > > work around the issue.
> > > >> > >> > > 
> > > >> > >> > > That instance is just stupidity / laziness, I think: we 
> > > >> > >> > > consider any
> > > >> > >> > > plug after machine creation a hot plug.  Real machines remain 
> > > >> > >> > > cold until
> > > >> > >> > > you press the power button.  Our virtual machines should 
> > > >> > >> > > remain cold
> > > >> > >> > > until they start running, i.e. with -S until the first 
> > > >> > >> > > "cont".  
> > > >> > >> It probably would be too risky to change semantics of -S from 
> > > >> > >> hotplug to coldplug.
> > > >> > >> But even if we were easy it won't matter in case if dynamic 
> > > >> > >> configuration
> > > >> > >> done properly. More on it below.
> > > >> > >>   
> > > >> > >> > > I vaguely remember me asking this before, but your answer 
> > > >> > >> > > didn't make it
> > > >> > >> > > into this cover letter, which gives me a pretext to ask again 
> > > >> > >> > > instead of
> > > >> > >> > > looking it up in the archives: what exactly prevents us from 
> > > >> > >> > > keeping the
> > > >> > >> > > machine cold enough for numa configuration until the first 
> > > >> > >> > > "cont"?
> > > >> > >> > 
> > > >> > >> > I also think this would be better, but it seems to be difficult
> > > >> > >> > in practice, see:
> > > >> > >> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
> > > >> > >> >   
> > > >> > >> 
> > > >> > >> In addition to Eduardo's reply, here is what I've answered back
> > > >> > >> when you've asked question the 1st time (v2 late at -S pause 
> > > >> > >> point reconfig):
> > > >> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
> > > >> > >> 
> > > >> > >> In short:
> > > >> > >> I think it's wrong in general doing fixups after machine is build
> > > >> > >> instead of getting correct configuration before building machine.
> > > >> > >> That's going to be complex and fragile and might be hard to do at
> > > >> > >> all depending on what we are fixing up.  
> > > >> > >
> > > >> > > What "building the machine" should mean, exactly, for external
> > > >> > > users?
> > > >> under "building machine", I've meant machine_run_board_init()
> > > >> and all follow up steps to machine_done stage.
> > > >> 
> > > >> > > The main question I'd like to see answered is: why exactly we
> > > >> > > must "build" the machine before the first "cont" is issued when
> > > >> > > using -S?  Why can't we delay everything to "cont" when using -S?  
> > > >> > > 
> > > >> Nor sure what question is about,
> > > >> Did you mean if it were possible to postpone machine_run_board_init()
> > > >> and all later steps to -S/cont time?
> > (1)
> > As David said -S pause point is practically breakpoint on some
> > instruction of built/existing machine and current monitor commands
> > expect it to be valid. Moving -S before machine_run_board_init()
> > will break semantics of current -S pause point (i.e. user expectation
> > on existing machine) as 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Eduardo Habkost]

On Mon, Apr 23, 2018 at 11:50:16AM +0200, Igor Mammedov wrote:
> On Fri, 20 Apr 2018 08:31:18 +0200
> Markus Armbruster  wrote:
> 
> > Eduardo Habkost  writes:
> > 
> > > On Thu, Apr 19, 2018 at 10:00:04AM +0200, Igor Mammedov wrote:  
> > >> On Wed, 18 Apr 2018 09:08:30 +0200
> > >> Markus Armbruster  wrote:
> > >>   
> > >> > Eduardo Habkost  writes:
> > >> >   
> > >> > > On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote:
> > >> > >> On Tue, 17 Apr 2018 11:27:39 -0300
> > >> > >> Eduardo Habkost  wrote:
> > >> > >> 
> > >> > >> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster 
> > >> > >> > wrote:
> > >> > >> > > Igor Mammedov  writes:
> > >> > >> > > 
> > >> > >> > > [...]  
> > >> > >> > > > Series allows to configure NUMA mapping at runtime using QMP
> > >> > >> > > > interface. For that to happen it introduces a new 
> > >> > >> > > > '-preconfig' CLI option
> > >> > >> > > > which allows to pause QEMU before machine_init() is run and
> > >> > >> > > > adds new set-numa-node QMP command which in conjunction with
> > >> > >> > > > query-hotpluggable-cpus allows to configure NUMA mapping for 
> > >> > >> > > > cpus.
> > >> > >> > > >
> > >> > >> > > > Later we can modify other commands to run early, for example 
> > >> > >> > > > device_add.
> > >> > >> > > > I recall SPAPR had problem when libvirt started QEMU with -S 
> > >> > >> > > > and, while it's
> > >> > >> > > > paused, added CPUs with device_add. Intent was to coldplug 
> > >> > >> > > > CPUs (but at that
> > >> > >> > > > stage it's considered hotplug already), so SPAPR had to work 
> > >> > >> > > > around the issue.  
> > >> > >> > > 
> > >> > >> > > That instance is just stupidity / laziness, I think: we 
> > >> > >> > > consider any
> > >> > >> > > plug after machine creation a hot plug.  Real machines remain 
> > >> > >> > > cold until
> > >> > >> > > you press the power button.  Our virtual machines should remain 
> > >> > >> > > cold
> > >> > >> > > until they start running, i.e. with -S until the first "cont".  
> > >> > >> > >   
> > >> > >> It probably would be too risky to change semantics of -S from 
> > >> > >> hotplug to coldplug.
> > >> > >> But even if we were easy it won't matter in case if dynamic 
> > >> > >> configuration
> > >> > >> done properly. More on it below.
> > >> > >> 
> > >> > >> > > I vaguely remember me asking this before, but your answer 
> > >> > >> > > didn't make it
> > >> > >> > > into this cover letter, which gives me a pretext to ask again 
> > >> > >> > > instead of
> > >> > >> > > looking it up in the archives: what exactly prevents us from 
> > >> > >> > > keeping the
> > >> > >> > > machine cold enough for numa configuration until the first 
> > >> > >> > > "cont"?  
> > >> > >> > 
> > >> > >> > I also think this would be better, but it seems to be difficult
> > >> > >> > in practice, see:
> > >> > >> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
> > >> > >> > 
> > >> > >> 
> > >> > >> In addition to Eduardo's reply, here is what I've answered back
> > >> > >> when you've asked question the 1st time (v2 late at -S pause point 
> > >> > >> reconfig):
> > >> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
> > >> > >> 
> > >> > >> In short:
> > >> > >> I think it's wrong in general doing fixups after machine is build
> > >> > >> instead of getting correct configuration before building machine.
> > >> > >> That's going to be complex and fragile and might be hard to do at
> > >> > >> all depending on what we are fixing up.
> > >> > >
> > >> > > What "building the machine" should mean, exactly, for external
> > >> > > users?  
> > >> under "building machine", I've meant machine_run_board_init()
> > >> and all follow up steps to machine_done stage.
> > >>   
> > >> > > The main question I'd like to see answered is: why exactly we
> > >> > > must "build" the machine before the first "cont" is issued when
> > >> > > using -S?  Why can't we delay everything to "cont" when using -S?
> > >> Nor sure what question is about,
> > >> Did you mean if it were possible to postpone machine_run_board_init()
> > >> and all later steps to -S/cont time?  
> (1)
> As David said -S pause point is practically breakpoint on some
> instruction of built/existing machine and current monitor commands
> expect it to be valid. Moving -S before machine_run_board_init()
> will break semantics of current -S pause point (i.e. user expectation
> on existing machine) as well as most of the commands that evolved
> in environment where machine already existed.

OK, so what's missing here is a clear description what the user
can expect on -S.

> 
> Hence a new -preconfig option and runstate to avoid breaking
> exiting users and being able to cleanly handle configuration that
> affects 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

On Fri, 20 Apr 2018 08:31:18 +0200
Markus Armbruster  wrote:

> Eduardo Habkost  writes:
> 
> > On Thu, Apr 19, 2018 at 10:00:04AM +0200, Igor Mammedov wrote:  
> >> On Wed, 18 Apr 2018 09:08:30 +0200
> >> Markus Armbruster  wrote:
> >>   
> >> > Eduardo Habkost  writes:
> >> >   
> >> > > On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote:
> >> > >> On Tue, 17 Apr 2018 11:27:39 -0300
> >> > >> Eduardo Habkost  wrote:
> >> > >> 
> >> > >> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:  
> >> > >> >   
> >> > >> > > Igor Mammedov  writes:
> >> > >> > > 
> >> > >> > > [...]  
> >> > >> > > > Series allows to configure NUMA mapping at runtime using QMP
> >> > >> > > > interface. For that to happen it introduces a new '-preconfig' 
> >> > >> > > > CLI option
> >> > >> > > > which allows to pause QEMU before machine_init() is run and
> >> > >> > > > adds new set-numa-node QMP command which in conjunction with
> >> > >> > > > query-hotpluggable-cpus allows to configure NUMA mapping for 
> >> > >> > > > cpus.
> >> > >> > > >
> >> > >> > > > Later we can modify other commands to run early, for example 
> >> > >> > > > device_add.
> >> > >> > > > I recall SPAPR had problem when libvirt started QEMU with -S 
> >> > >> > > > and, while it's
> >> > >> > > > paused, added CPUs with device_add. Intent was to coldplug CPUs 
> >> > >> > > > (but at that
> >> > >> > > > stage it's considered hotplug already), so SPAPR had to work 
> >> > >> > > > around the issue.  
> >> > >> > > 
> >> > >> > > That instance is just stupidity / laziness, I think: we consider 
> >> > >> > > any
> >> > >> > > plug after machine creation a hot plug.  Real machines remain 
> >> > >> > > cold until
> >> > >> > > you press the power button.  Our virtual machines should remain 
> >> > >> > > cold
> >> > >> > > until they start running, i.e. with -S until the first "cont".
> >> > >> It probably would be too risky to change semantics of -S from hotplug 
> >> > >> to coldplug.
> >> > >> But even if we were easy it won't matter in case if dynamic 
> >> > >> configuration
> >> > >> done properly. More on it below.
> >> > >> 
> >> > >> > > I vaguely remember me asking this before, but your answer didn't 
> >> > >> > > make it
> >> > >> > > into this cover letter, which gives me a pretext to ask again 
> >> > >> > > instead of
> >> > >> > > looking it up in the archives: what exactly prevents us from 
> >> > >> > > keeping the
> >> > >> > > machine cold enough for numa configuration until the first 
> >> > >> > > "cont"?  
> >> > >> > 
> >> > >> > I also think this would be better, but it seems to be difficult
> >> > >> > in practice, see:
> >> > >> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
> >> > >> > 
> >> > >> 
> >> > >> In addition to Eduardo's reply, here is what I've answered back
> >> > >> when you've asked question the 1st time (v2 late at -S pause point 
> >> > >> reconfig):
> >> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
> >> > >> 
> >> > >> In short:
> >> > >> I think it's wrong in general doing fixups after machine is build
> >> > >> instead of getting correct configuration before building machine.
> >> > >> That's going to be complex and fragile and might be hard to do at
> >> > >> all depending on what we are fixing up.
> >> > >
> >> > > What "building the machine" should mean, exactly, for external
> >> > > users?  
> >> under "building machine", I've meant machine_run_board_init()
> >> and all follow up steps to machine_done stage.
> >>   
> >> > > The main question I'd like to see answered is: why exactly we
> >> > > must "build" the machine before the first "cont" is issued when
> >> > > using -S?  Why can't we delay everything to "cont" when using -S?
> >> Nor sure what question is about,
> >> Did you mean if it were possible to postpone machine_run_board_init()
> >> and all later steps to -S/cont time?  
(1)
As David said -S pause point is practically breakpoint on some
instruction of built/existing machine and current monitor commands
expect it to be valid. Moving -S before machine_run_board_init()
will break semantics of current -S pause point (i.e. user expectation
on existing machine) as well as most of the commands that evolved
in environment where machine already existed.

Hence a new -preconfig option and runstate to avoid breaking
exiting users and being able to cleanly handle configuration that
affects machine_run_board_init().

> > Exactly.  In other words, what exactly must be done before the
> > monitor is available when using -S,
for MUST, it should be commands that affect machine_run_board_init()
like being added set-numa-node

> > and what exactly can be postponed after "cont" when using -S?
hotplug configuration and various runtime query commands that
expect built 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Markus Armbruster]

Eduardo Habkost  writes:

> On Thu, Apr 19, 2018 at 10:00:04AM +0200, Igor Mammedov wrote:
>> On Wed, 18 Apr 2018 09:08:30 +0200
>> Markus Armbruster  wrote:
>> 
>> > Eduardo Habkost  writes:
>> > 
>> > > On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote:  
>> > >> On Tue, 17 Apr 2018 11:27:39 -0300
>> > >> Eduardo Habkost  wrote:
>> > >>   
>> > >> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:  
>> > >> > > Igor Mammedov  writes:
>> > >> > > 
>> > >> > > [...]
>> > >> > > > Series allows to configure NUMA mapping at runtime using QMP
>> > >> > > > interface. For that to happen it introduces a new '-preconfig' 
>> > >> > > > CLI option
>> > >> > > > which allows to pause QEMU before machine_init() is run and
>> > >> > > > adds new set-numa-node QMP command which in conjunction with
>> > >> > > > query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
>> > >> > > >
>> > >> > > > Later we can modify other commands to run early, for example 
>> > >> > > > device_add.
>> > >> > > > I recall SPAPR had problem when libvirt started QEMU with -S and, 
>> > >> > > > while it's
>> > >> > > > paused, added CPUs with device_add. Intent was to coldplug CPUs 
>> > >> > > > (but at that
>> > >> > > > stage it's considered hotplug already), so SPAPR had to work 
>> > >> > > > around the issue.
>> > >> > > 
>> > >> > > That instance is just stupidity / laziness, I think: we consider any
>> > >> > > plug after machine creation a hot plug.  Real machines remain cold 
>> > >> > > until
>> > >> > > you press the power button.  Our virtual machines should remain cold
>> > >> > > until they start running, i.e. with -S until the first "cont".  
>> > >> It probably would be too risky to change semantics of -S from hotplug 
>> > >> to coldplug.
>> > >> But even if we were easy it won't matter in case if dynamic 
>> > >> configuration
>> > >> done properly. More on it below.
>> > >>   
>> > >> > > I vaguely remember me asking this before, but your answer didn't 
>> > >> > > make it
>> > >> > > into this cover letter, which gives me a pretext to ask again 
>> > >> > > instead of
>> > >> > > looking it up in the archives: what exactly prevents us from 
>> > >> > > keeping the
>> > >> > > machine cold enough for numa configuration until the first "cont"?  
>> > >> > >   
>> > >> > 
>> > >> > I also think this would be better, but it seems to be difficult
>> > >> > in practice, see:
>> > >> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
>> > >> >   
>> > >> 
>> > >> In addition to Eduardo's reply, here is what I've answered back
>> > >> when you've asked question the 1st time (v2 late at -S pause point 
>> > >> reconfig):
>> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
>> > >> 
>> > >> In short:
>> > >> I think it's wrong in general doing fixups after machine is build
>> > >> instead of getting correct configuration before building machine.
>> > >> That's going to be complex and fragile and might be hard to do at
>> > >> all depending on what we are fixing up.  
>> > >
>> > > What "building the machine" should mean, exactly, for external
>> > > users?
>> under "building machine", I've meant machine_run_board_init()
>> and all follow up steps to machine_done stage.
>> 
>> > > The main question I'd like to see answered is: why exactly we
>> > > must "build" the machine before the first "cont" is issued when
>> > > using -S?  Why can't we delay everything to "cont" when using -S?  
>> Nor sure what question is about,
>> Did you mean if it were possible to postpone machine_run_board_init()
>> and all later steps to -S/cont time?
>
> Exactly.  In other words, what exactly must be done before the
> monitor is available when using -S, and what exactly can be
> postponed after "cont" when using -S?
>
>>  
>> > > Is it just because it's a long and complex task?  Does that mean
>> > > we might still do that eventually, and eliminate the
>> > > prelaunch/preconfig distinction in the distant future?  
>> > 
>> > Why would anyone want to use -S going forward?  For reasons other "we've
>> > always used -S, and can't be bothered to change".
>> We should be able to deprecate/remove -S once we can do all
>> initial configuration that's possible to do there at
>> preconfig time.

This sounds like there are things we can do with -S but can't
--preconfig now.  Is that correct?

If yes, I got another question.  If you want to configure NUMA, you need
to use --preconfig.  If you also want to configure one of the things you
can't with --preconfig, you need to use -S.  In other words, you need to
use both.  How would that work?

> If the plan is to deprecate -S, what are the important
> user-visible differences between -S and -preconfig today?  Do we
> plan to eliminate all those differences before
> deprecating/removing -S?

Documentation (including 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[David Gibson]

On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:
> Igor Mammedov  writes:
> 
> [...]
> > Series allows to configure NUMA mapping at runtime using QMP
> > interface. For that to happen it introduces a new '-preconfig' CLI option
> > which allows to pause QEMU before machine_init() is run and
> > adds new set-numa-node QMP command which in conjunction with
> > query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
> >
> > Later we can modify other commands to run early, for example device_add.
> > I recall SPAPR had problem when libvirt started QEMU with -S and, while it's
> > paused, added CPUs with device_add. Intent was to coldplug CPUs (but at that
> > stage it's considered hotplug already), so SPAPR had to work around the 
> > issue.
> 
> That instance is just stupidity / laziness, I think: we consider any
> plug after machine creation a hot plug.  Real machines remain cold until
> you press the power button.  Our virtual machines should remain cold
> until they start running, i.e. with -S until the first "cont".

Makes sense to me.  As I recall, the chief problem I had here with
PAPR hotplug was that -S stopped *after* the machine_reset, which
meant the only mechanisms we really had to inform the guest of the
hardware were the hotplug mechanisms.  Sort of.  It got complicated
because of the feature negotiation system that PAPR guests have.

At present -S doesn't really operate like a "stop before hitting the
power", it instead acts more like a breakpoint on the first
instruction of the firmware.

If we were to move the -S stop to before the machine_reset, I believe
that might simplify several things in our hotplug handling code for
papr.

> I vaguely remember me asking this before, but your answer didn't make it
> into this cover letter, which gives me a pretext to ask again instead of
> looking it up in the archives: what exactly prevents us from keeping the
> machine cold enough for numa configuration until the first "cont"?
> 
> >
> > Example of configuration session:
> > $QEMU -smp 2 -preconfig ...
> >
> > QMP:
> > # get CPUs layout for current target/machine/CLI
> > -> {'execute': 'query-hotpluggable-cpus' }  
> > <- {'return': [
> >{'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 1}, ... },
> >{'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 0}, ... }
> >]}
> >
> > # configure 1st node
> > -> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 0 
> > } }  
> > <- {'return': {}}
> > -> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',   
> >'node-id': 0, 'core-id': 0, 'thread-id': 0, 'socket-id': 1, }
> >}
> > <- {'return': {}}
> >
> > # configure 2nd node
> > -> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 1 
> > } }
> > -> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',  
> >'node-id': 1, 'core-id': 0, 'thread-id': 0, 'socket-id': 0 }
> >}
> > <- {'return': {}}
> >
> > # [optional] verify configuration
> > -> {'execute': 'query-hotpluggable-cpus' }  
> > <- {'return': [
> >{'props': {'core-id': 0, 'thread-id': 0, 'node-id': 0, 'socket-id': 
> > 1}, ... },
> >{'props': {'core-id': 0, 'thread-id': 0, 'node-id': 1, 'socket-id': 
> > 0}, ... }
> >]}
> >
> >
> > Git tree:
> > https://github.com/imammedo/qemu.git qmp_preconfig_v3
> >
> > Ref to v1:
> > https://lists.gnu.org/archive/html/qemu-devel/2017-10/msg03583.html
> > Message-Id: <1508170976-96869-1-git-send-email-imamm...@redhat.com>
> 

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Eduardo Habkost]

On Thu, Apr 19, 2018 at 10:00:04AM +0200, Igor Mammedov wrote:
> On Wed, 18 Apr 2018 09:08:30 +0200
> Markus Armbruster  wrote:
> 
> > Eduardo Habkost  writes:
> > 
> > > On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote:  
> > >> On Tue, 17 Apr 2018 11:27:39 -0300
> > >> Eduardo Habkost  wrote:
> > >>   
> > >> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:  
> > >> > > Igor Mammedov  writes:
> > >> > > 
> > >> > > [...]
> > >> > > > Series allows to configure NUMA mapping at runtime using QMP
> > >> > > > interface. For that to happen it introduces a new '-preconfig' CLI 
> > >> > > > option
> > >> > > > which allows to pause QEMU before machine_init() is run and
> > >> > > > adds new set-numa-node QMP command which in conjunction with
> > >> > > > query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
> > >> > > >
> > >> > > > Later we can modify other commands to run early, for example 
> > >> > > > device_add.
> > >> > > > I recall SPAPR had problem when libvirt started QEMU with -S and, 
> > >> > > > while it's
> > >> > > > paused, added CPUs with device_add. Intent was to coldplug CPUs 
> > >> > > > (but at that
> > >> > > > stage it's considered hotplug already), so SPAPR had to work 
> > >> > > > around the issue.
> > >> > > 
> > >> > > That instance is just stupidity / laziness, I think: we consider any
> > >> > > plug after machine creation a hot plug.  Real machines remain cold 
> > >> > > until
> > >> > > you press the power button.  Our virtual machines should remain cold
> > >> > > until they start running, i.e. with -S until the first "cont".  
> > >> It probably would be too risky to change semantics of -S from hotplug to 
> > >> coldplug.
> > >> But even if we were easy it won't matter in case if dynamic configuration
> > >> done properly. More on it below.
> > >>   
> > >> > > I vaguely remember me asking this before, but your answer didn't 
> > >> > > make it
> > >> > > into this cover letter, which gives me a pretext to ask again 
> > >> > > instead of
> > >> > > looking it up in the archives: what exactly prevents us from keeping 
> > >> > > the
> > >> > > machine cold enough for numa configuration until the first "cont"?   
> > >> > >  
> > >> > 
> > >> > I also think this would be better, but it seems to be difficult
> > >> > in practice, see:
> > >> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
> > >> >   
> > >> 
> > >> In addition to Eduardo's reply, here is what I've answered back
> > >> when you've asked question the 1st time (v2 late at -S pause point 
> > >> reconfig):
> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
> > >> 
> > >> In short:
> > >> I think it's wrong in general doing fixups after machine is build
> > >> instead of getting correct configuration before building machine.
> > >> That's going to be complex and fragile and might be hard to do at
> > >> all depending on what we are fixing up.  
> > >
> > > What "building the machine" should mean, exactly, for external
> > > users?
> under "building machine", I've meant machine_run_board_init()
> and all follow up steps to machine_done stage.
> 
> > > The main question I'd like to see answered is: why exactly we
> > > must "build" the machine before the first "cont" is issued when
> > > using -S?  Why can't we delay everything to "cont" when using -S?  
> Nor sure what question is about,
> Did you mean if it were possible to postpone machine_run_board_init()
> and all later steps to -S/cont time?

Exactly.  In other words, what exactly must be done before the
monitor is available when using -S, and what exactly can be
postponed after "cont" when using -S?

>  
> > > Is it just because it's a long and complex task?  Does that mean
> > > we might still do that eventually, and eliminate the
> > > prelaunch/preconfig distinction in the distant future?  
> > 
> > Why would anyone want to use -S going forward?  For reasons other "we've
> > always used -S, and can't be bothered to change".
> We should be able to deprecate/remove -S once we can do all
> initial configuration that's possible to do there at
> preconfig time.

If the plan is to deprecate -S, what are the important
user-visible differences between -S and -preconfig today?  Do we
plan to eliminate all those differences before
deprecating/removing -S?

>  
> > > Even if we follow your approach, we need to answer these
> > > questions.  I'm sure we will try to reorder initialization steps
> > > between the preconfig/prelaunch states in the future, and we
> > > shouldn't break any expectations from external users when doing
> > > that.
> As minimum I expect -preconfig to be a runtime equivalent to CLI,
> with difference that it will be interactive and use QMP interface.
> As long as it sits between CLI parsing and the rest of initialization
> it shouldn't break that.

What 

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

On Wed, 18 Apr 2018 09:08:30 +0200
Markus Armbruster  wrote:

> Eduardo Habkost  writes:
> 
> > On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote:  
> >> On Tue, 17 Apr 2018 11:27:39 -0300
> >> Eduardo Habkost  wrote:
> >>   
> >> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:  
> >> > > Igor Mammedov  writes:
> >> > > 
> >> > > [...]
> >> > > > Series allows to configure NUMA mapping at runtime using QMP
> >> > > > interface. For that to happen it introduces a new '-preconfig' CLI 
> >> > > > option
> >> > > > which allows to pause QEMU before machine_init() is run and
> >> > > > adds new set-numa-node QMP command which in conjunction with
> >> > > > query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
> >> > > >
> >> > > > Later we can modify other commands to run early, for example 
> >> > > > device_add.
> >> > > > I recall SPAPR had problem when libvirt started QEMU with -S and, 
> >> > > > while it's
> >> > > > paused, added CPUs with device_add. Intent was to coldplug CPUs (but 
> >> > > > at that
> >> > > > stage it's considered hotplug already), so SPAPR had to work around 
> >> > > > the issue.
> >> > > 
> >> > > That instance is just stupidity / laziness, I think: we consider any
> >> > > plug after machine creation a hot plug.  Real machines remain cold 
> >> > > until
> >> > > you press the power button.  Our virtual machines should remain cold
> >> > > until they start running, i.e. with -S until the first "cont".  
> >> It probably would be too risky to change semantics of -S from hotplug to 
> >> coldplug.
> >> But even if we were easy it won't matter in case if dynamic configuration
> >> done properly. More on it below.
> >>   
> >> > > I vaguely remember me asking this before, but your answer didn't make 
> >> > > it
> >> > > into this cover letter, which gives me a pretext to ask again instead 
> >> > > of
> >> > > looking it up in the archives: what exactly prevents us from keeping 
> >> > > the
> >> > > machine cold enough for numa configuration until the first "cont"?
> >> > 
> >> > I also think this would be better, but it seems to be difficult
> >> > in practice, see:
> >> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain 
> >> >  
> >> 
> >> In addition to Eduardo's reply, here is what I've answered back
> >> when you've asked question the 1st time (v2 late at -S pause point 
> >> reconfig):
> >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
> >> 
> >> In short:
> >> I think it's wrong in general doing fixups after machine is build
> >> instead of getting correct configuration before building machine.
> >> That's going to be complex and fragile and might be hard to do at
> >> all depending on what we are fixing up.  
> >
> > What "building the machine" should mean, exactly, for external
> > users?
under "building machine", I've meant machine_run_board_init()
and all follow up steps to machine_done stage.

> > The main question I'd like to see answered is: why exactly we
> > must "build" the machine before the first "cont" is issued when
> > using -S?  Why can't we delay everything to "cont" when using -S?  
Nor sure what question is about,
Did you mean if it were possible to postpone machine_run_board_init()
and all later steps to -S/cont time?
 
> > Is it just because it's a long and complex task?  Does that mean
> > we might still do that eventually, and eliminate the
> > prelaunch/preconfig distinction in the distant future?  
> 
> Why would anyone want to use -S going forward?  For reasons other "we've
> always used -S, and can't be bothered to change".
We should be able to deprecate/remove -S once we can do all
initial configuration that's possible to do there at
preconfig time.
 
> > Even if we follow your approach, we need to answer these
> > questions.  I'm sure we will try to reorder initialization steps
> > between the preconfig/prelaunch states in the future, and we
> > shouldn't break any expectations from external users when doing
> > that.
As minimum I expect -preconfig to be a runtime equivalent to CLI,
with difference that it will be interactive and use QMP interface.
As long as it sits between CLI parsing and the rest of initialization
it shouldn't break that.

> Moreover, the questions need to be answered in Git.  Commit message,
> comments, docs/, use your judgement.
I've thought that commit messages/patches were describing introduced
changes sufficiently. But I've been sitting on these patches for
a long time and what's obvious to me might be not so clear to others.
I might just not see what's missing. Any suggestions to improve it
are welcome.

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Markus Armbruster]

Eduardo Habkost  writes:

> On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote:
>> On Tue, 17 Apr 2018 11:27:39 -0300
>> Eduardo Habkost  wrote:
>> 
>> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:
>> > > Igor Mammedov  writes:
>> > > 
>> > > [...]  
>> > > > Series allows to configure NUMA mapping at runtime using QMP
>> > > > interface. For that to happen it introduces a new '-preconfig' CLI 
>> > > > option
>> > > > which allows to pause QEMU before machine_init() is run and
>> > > > adds new set-numa-node QMP command which in conjunction with
>> > > > query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
>> > > >
>> > > > Later we can modify other commands to run early, for example 
>> > > > device_add.
>> > > > I recall SPAPR had problem when libvirt started QEMU with -S and, 
>> > > > while it's
>> > > > paused, added CPUs with device_add. Intent was to coldplug CPUs (but 
>> > > > at that
>> > > > stage it's considered hotplug already), so SPAPR had to work around 
>> > > > the issue.  
>> > > 
>> > > That instance is just stupidity / laziness, I think: we consider any
>> > > plug after machine creation a hot plug.  Real machines remain cold until
>> > > you press the power button.  Our virtual machines should remain cold
>> > > until they start running, i.e. with -S until the first "cont".
>> It probably would be too risky to change semantics of -S from hotplug to 
>> coldplug.
>> But even if we were easy it won't matter in case if dynamic configuration
>> done properly. More on it below.
>> 
>> > > I vaguely remember me asking this before, but your answer didn't make it
>> > > into this cover letter, which gives me a pretext to ask again instead of
>> > > looking it up in the archives: what exactly prevents us from keeping the
>> > > machine cold enough for numa configuration until the first "cont"?  
>> > 
>> > I also think this would be better, but it seems to be difficult
>> > in practice, see:
>> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
>> 
>> In addition to Eduardo's reply, here is what I've answered back
>> when you've asked question the 1st time (v2 late at -S pause point reconfig):
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
>> 
>> In short:
>> I think it's wrong in general doing fixups after machine is build
>> instead of getting correct configuration before building machine.
>> That's going to be complex and fragile and might be hard to do at
>> all depending on what we are fixing up.
>
> What "building the machine" should mean, exactly, for external
> users?
>
> The main question I'd like to see answered is: why exactly we
> must "build" the machine before the first "cont" is issued when
> using -S?  Why can't we delay everything to "cont" when using -S?

Exactly.

> Is it just because it's a long and complex task?  Does that mean
> we might still do that eventually, and eliminate the
> prelaunch/preconfig distinction in the distant future?

Why would anyone want to use -S going forward?  For reasons other "we've
always used -S, and can't be bothered to change".

> Even if we follow your approach, we need to answer these
> questions.  I'm sure we will try to reorder initialization steps
> between the preconfig/prelaunch states in the future, and we
> shouldn't break any expectations from external users when doing
> that.

Moreover, the questions need to be answered in Git.  Commit message,
comments, docs/, use your judgement.

>> BTW this is an outdated version of series and there is a newer one v5
>> https://patchwork.ozlabs.org/cover/895315/

Sorry about that.  I'm drowning in a sea of two months worth of patches.

>> so pleases review it.
>> 
>> Short diff vs 1:
>>  - only limited(minimum) set of commands is available at preconfig stage for 
>> now
>>  - use QAPI schema to mark commands as preconfig enabled,
>>so mgmt could see when it can use commands.
>>  - added preconfig runstate state-machine instead of adding more global 
>> variables
>>to cleanly keep track of where QEMU is paused and what it's allowed to do

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Eduardo Habkost]

On Tue, Apr 17, 2018 at 05:41:10PM +0200, Igor Mammedov wrote:
> On Tue, 17 Apr 2018 11:27:39 -0300
> Eduardo Habkost  wrote:
> 
> > On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:
> > > Igor Mammedov  writes:
> > > 
> > > [...]  
> > > > Series allows to configure NUMA mapping at runtime using QMP
> > > > interface. For that to happen it introduces a new '-preconfig' CLI 
> > > > option
> > > > which allows to pause QEMU before machine_init() is run and
> > > > adds new set-numa-node QMP command which in conjunction with
> > > > query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
> > > >
> > > > Later we can modify other commands to run early, for example device_add.
> > > > I recall SPAPR had problem when libvirt started QEMU with -S and, while 
> > > > it's
> > > > paused, added CPUs with device_add. Intent was to coldplug CPUs (but at 
> > > > that
> > > > stage it's considered hotplug already), so SPAPR had to work around the 
> > > > issue.  
> > > 
> > > That instance is just stupidity / laziness, I think: we consider any
> > > plug after machine creation a hot plug.  Real machines remain cold until
> > > you press the power button.  Our virtual machines should remain cold
> > > until they start running, i.e. with -S until the first "cont".
> It probably would be too risky to change semantics of -S from hotplug to 
> coldplug.
> But even if we were easy it won't matter in case if dynamic configuration
> done properly. More on it below.
> 
> > > I vaguely remember me asking this before, but your answer didn't make it
> > > into this cover letter, which gives me a pretext to ask again instead of
> > > looking it up in the archives: what exactly prevents us from keeping the
> > > machine cold enough for numa configuration until the first "cont"?  
> > 
> > I also think this would be better, but it seems to be difficult
> > in practice, see:
> > http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain
> 
> In addition to Eduardo's reply, here is what I've answered back
> when you've asked question the 1st time (v2 late at -S pause point reconfig):
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html
> 
> In short:
> I think it's wrong in general doing fixups after machine is build
> instead of getting correct configuration before building machine.
> That's going to be complex and fragile and might be hard to do at
> all depending on what we are fixing up.

What "building the machine" should mean, exactly, for external
users?

The main question I'd like to see answered is: why exactly we
must "build" the machine before the first "cont" is issued when
using -S?  Why can't we delay everything to "cont" when using -S?

Is it just because it's a long and complex task?  Does that mean
we might still do that eventually, and eliminate the
prelaunch/preconfig distinction in the distant future?

Even if we follow your approach, we need to answer these
questions.  I'm sure we will try to reorder initialization steps
between the preconfig/prelaunch states in the future, and we
shouldn't break any expectations from external users when doing
that.

> 
> BTW this is an outdated version of series and there is a newer one v5
> https://patchwork.ozlabs.org/cover/895315/
> so pleases review it.
> 
> Short diff vs 1:
>  - only limited(minimum) set of commands is available at preconfig stage for 
> now
>  - use QAPI schema to mark commands as preconfig enabled,
>so mgmt could see when it can use commands.
>  - added preconfig runstate state-machine instead of adding more global 
> variables
>to cleanly keep track of where QEMU is paused and what it's allowed to do

-- 
Eduardo

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

On Tue, 17 Apr 2018 11:27:39 -0300
Eduardo Habkost  wrote:

> On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:
> > Igor Mammedov  writes:
> > 
> > [...]  
> > > Series allows to configure NUMA mapping at runtime using QMP
> > > interface. For that to happen it introduces a new '-preconfig' CLI option
> > > which allows to pause QEMU before machine_init() is run and
> > > adds new set-numa-node QMP command which in conjunction with
> > > query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
> > >
> > > Later we can modify other commands to run early, for example device_add.
> > > I recall SPAPR had problem when libvirt started QEMU with -S and, while 
> > > it's
> > > paused, added CPUs with device_add. Intent was to coldplug CPUs (but at 
> > > that
> > > stage it's considered hotplug already), so SPAPR had to work around the 
> > > issue.  
> > 
> > That instance is just stupidity / laziness, I think: we consider any
> > plug after machine creation a hot plug.  Real machines remain cold until
> > you press the power button.  Our virtual machines should remain cold
> > until they start running, i.e. with -S until the first "cont".
It probably would be too risky to change semantics of -S from hotplug to 
coldplug.
But even if we were easy it won't matter in case if dynamic configuration
done properly. More on it below.

> > I vaguely remember me asking this before, but your answer didn't make it
> > into this cover letter, which gives me a pretext to ask again instead of
> > looking it up in the archives: what exactly prevents us from keeping the
> > machine cold enough for numa configuration until the first "cont"?  
> 
> I also think this would be better, but it seems to be difficult
> in practice, see:
> http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain

In addition to Eduardo's reply, here is what I've answered back
when you've asked question the 1st time (v2 late at -S pause point reconfig):
https://www.mail-archive.com/qemu-devel@nongnu.org/msg504140.html

In short:
I think it's wrong in general doing fixups after machine is build
instead of getting correct configuration before building machine.
That's going to be complex and fragile and might be hard to do at
all depending on what we are fixing up.

BTW this is an outdated version of series and there is a newer one v5
https://patchwork.ozlabs.org/cover/895315/
so pleases review it.

Short diff vs 1:
 - only limited(minimum) set of commands is available at preconfig stage for now
 - use QAPI schema to mark commands as preconfig enabled,
   so mgmt could see when it can use commands.
 - added preconfig runstate state-machine instead of adding more global 
variables
   to cleanly keep track of where QEMU is paused and what it's allowed to do

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Eduardo Habkost]

On Tue, Apr 17, 2018 at 04:13:34PM +0200, Markus Armbruster wrote:
> Igor Mammedov  writes:
> 
> [...]
> > Series allows to configure NUMA mapping at runtime using QMP
> > interface. For that to happen it introduces a new '-preconfig' CLI option
> > which allows to pause QEMU before machine_init() is run and
> > adds new set-numa-node QMP command which in conjunction with
> > query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
> >
> > Later we can modify other commands to run early, for example device_add.
> > I recall SPAPR had problem when libvirt started QEMU with -S and, while it's
> > paused, added CPUs with device_add. Intent was to coldplug CPUs (but at that
> > stage it's considered hotplug already), so SPAPR had to work around the 
> > issue.
> 
> That instance is just stupidity / laziness, I think: we consider any
> plug after machine creation a hot plug.  Real machines remain cold until
> you press the power button.  Our virtual machines should remain cold
> until they start running, i.e. with -S until the first "cont".
> 
> I vaguely remember me asking this before, but your answer didn't make it
> into this cover letter, which gives me a pretext to ask again instead of
> looking it up in the archives: what exactly prevents us from keeping the
> machine cold enough for numa configuration until the first "cont"?

I also think this would be better, but it seems to be difficult
in practice, see:
http://mid.mail-archive.com/20180323210532.GD28161@localhost.localdomain

-- 
Eduardo

Re: [Qemu-devel] [PATCH v4 0/9] enable numa configuration before machine_init() from QMP

[Markus Armbruster]

Igor Mammedov  writes:

[...]
> Series allows to configure NUMA mapping at runtime using QMP
> interface. For that to happen it introduces a new '-preconfig' CLI option
> which allows to pause QEMU before machine_init() is run and
> adds new set-numa-node QMP command which in conjunction with
> query-hotpluggable-cpus allows to configure NUMA mapping for cpus.
>
> Later we can modify other commands to run early, for example device_add.
> I recall SPAPR had problem when libvirt started QEMU with -S and, while it's
> paused, added CPUs with device_add. Intent was to coldplug CPUs (but at that
> stage it's considered hotplug already), so SPAPR had to work around the issue.

That instance is just stupidity / laziness, I think: we consider any
plug after machine creation a hot plug.  Real machines remain cold until
you press the power button.  Our virtual machines should remain cold
until they start running, i.e. with -S until the first "cont".

I vaguely remember me asking this before, but your answer didn't make it
into this cover letter, which gives me a pretext to ask again instead of
looking it up in the archives: what exactly prevents us from keeping the
machine cold enough for numa configuration until the first "cont"?

>
> Example of configuration session:
> $QEMU -smp 2 -preconfig ...
>
> QMP:
> # get CPUs layout for current target/machine/CLI
> -> {'execute': 'query-hotpluggable-cpus' }  
> <- {'return': [
>{'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 1}, ... },
>{'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 0}, ... }
>]}
>
> # configure 1st node
> -> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 0 } 
> }  
> <- {'return': {}}
> -> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',   
>'node-id': 0, 'core-id': 0, 'thread-id': 0, 'socket-id': 1, }
>}
> <- {'return': {}}
>
> # configure 2nd node
> -> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 1 } }
> -> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',  
>'node-id': 1, 'core-id': 0, 'thread-id': 0, 'socket-id': 0 }
>}
> <- {'return': {}}
>
> # [optional] verify configuration
> -> {'execute': 'query-hotpluggable-cpus' }  
> <- {'return': [
>{'props': {'core-id': 0, 'thread-id': 0, 'node-id': 0, 'socket-id': 
> 1}, ... },
>{'props': {'core-id': 0, 'thread-id': 0, 'node-id': 1, 'socket-id': 
> 0}, ... }
>]}
>
>
> Git tree:
> https://github.com/imammedo/qemu.git qmp_preconfig_v3
>
> Ref to v1:
> https://lists.gnu.org/archive/html/qemu-devel/2017-10/msg03583.html
> Message-Id: <1508170976-96869-1-git-send-email-imamm...@redhat.com>