Re: [PATCH] nvmm: Fix support for stable version

[Kamil Rytarowski]

Reviewed-by: Kamil Rytarowski 

Paolo, could you please merge it?

On 13.10.2021 15:54, nia wrote:
> NVMM user version 1 is the version being shipped with netbsd-9,
> which is the most recent stable branch of NetBSD. This makes it
> possible to use the NVMM accelerator on the most recent NetBSD
> release, 9.2, which lacks nvmm_cpu_stop.
> 
> (CC'ing maintainers)
> 
> Signed-off-by: Nia Alarie 
> ---
>  meson.build |  4 +---
>  target/i386/nvmm/nvmm-all.c | 10 ++
>  2 files changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/meson.build b/meson.build
> index 15ef4d3c41..6e4d9b919a 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -244,9 +244,7 @@ if not get_option('hax').disabled()
>endif
>  endif
>  if targetos == 'netbsd'
> -  if cc.has_header_symbol('nvmm.h', 'nvmm_cpu_stop', required: 
> get_option('nvmm'))
> -nvmm = cc.find_library('nvmm', required: get_option('nvmm'))
> -  endif
> +  nvmm = cc.find_library('nvmm', required: get_option('nvmm'))
>if nvmm.found()
>  accelerators += 'CONFIG_NVMM'
>endif
> diff --git a/target/i386/nvmm/nvmm-all.c b/target/i386/nvmm/nvmm-all.c
> index a488b00e90..4a10412427 100644
> --- a/target/i386/nvmm/nvmm-all.c
> +++ b/target/i386/nvmm/nvmm-all.c
> @@ -750,7 +750,11 @@ nvmm_vcpu_loop(CPUState *cpu)
>  nvmm_vcpu_pre_run(cpu);
>  
>  if (qatomic_read(&cpu->exit_request)) {
> +#if NVMM_USER_VERSION >= 2
>  nvmm_vcpu_stop(vcpu);
> +#else
> +qemu_cpu_kick_self();
> +#endif
>  }
>  
>  /* Read exit_request before the kernel reads the immediate exit flag 
> */
> @@ -767,6 +771,7 @@ nvmm_vcpu_loop(CPUState *cpu)
>  switch (exit->reason) {
>  case NVMM_VCPU_EXIT_NONE:
>  break;
> +#if NVMM_USER_VERSION >= 2
>  case NVMM_VCPU_EXIT_STOPPED:
>  /*
>   * The kernel cleared the immediate exit flag; cpu->exit_request
> @@ -775,6 +780,7 @@ nvmm_vcpu_loop(CPUState *cpu)
>  smp_wmb();
>  qcpu->stop = true;
>  break;
> +#endif
>  case NVMM_VCPU_EXIT_MEMORY:
>  ret = nvmm_handle_mem(mach, vcpu);
>  break;
> @@ -888,8 +894,12 @@ nvmm_ipi_signal(int sigcpu)
>  {
>  if (current_cpu) {
>  struct qemu_vcpu *qcpu = get_qemu_vcpu(current_cpu);
> +#if NVMM_USER_VERSION >= 2
>  struct nvmm_vcpu *vcpu = &qcpu->vcpu;
>  nvmm_vcpu_stop(vcpu);
> +#else
> +qcpu->stop = true;
> +#endif
>  }
>  }
>  
> 

Re: [PATCH v6 12/40] accel/nvmm: Implement AccelOpsClass::has_work()

[Kamil Rytarowski]

On 24.09.2021 11:38, Philippe Mathieu-Daudé wrote:
> Since there is no specific NVMM handling for cpu_has_work() in
> cpu_thread_is_idle(), implement NVMM has_work() handler as a
> simple 'return false' code.
>
> Signed-off-by: Philippe Mathieu-Daudé 
Reviewed-by: Kamil Rytarowski 
> ---
>  target/i386/nvmm/nvmm-accel-ops.c | 6 ++
>  1 file changed, 6 insertions(+)
>
> diff --git a/target/i386/nvmm/nvmm-accel-ops.c 
> b/target/i386/nvmm/nvmm-accel-ops.c
> index f788f75289f..36296f79ff8 100644
> --- a/target/i386/nvmm/nvmm-accel-ops.c
> +++ b/target/i386/nvmm/nvmm-accel-ops.c
> @@ -83,6 +83,11 @@ static void nvmm_kick_vcpu_thread(CPUState *cpu)
>  cpus_kick_thread(cpu);
>  }
>
> +static bool nvmm_cpu_has_work(CPUState *cpu)
> +{
> +return false;
> +}
> +
>  static void nvmm_accel_ops_class_init(ObjectClass *oc, void *data)
>  {
>  AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
> @@ -94,6 +99,7 @@ static void nvmm_accel_ops_class_init(ObjectClass *oc, void 
> *data)
>  ops->synchronize_post_init = nvmm_cpu_synchronize_post_init;
>  ops->synchronize_state = nvmm_cpu_synchronize_state;
>  ops->synchronize_pre_loadvm = nvmm_cpu_synchronize_pre_loadvm;
> +ops->has_work = nvmm_cpu_has_work;
>  }
>
>  static const TypeInfo nvmm_accel_ops_type = {
>

Re: [PATCH v2 7/8] qemu/bswap: Use compiler __builtin_bswap() on NetBSD

[Kamil Rytarowski]

On 29.09.2020 10:58, Peter Maydell wrote:
> On Mon, 28 Sep 2020 at 23:02, Kamil Rytarowski  wrote:
>>
>> Personally, I prefer using the system headers. but if you want to use
>> the GCC builtins, please go for it.
> 
> I'd agree if the system header approach was cross-platform
> or if this was a BSD-only program or if we were aiming for
> complete compiler-implementation independence, but since we
> rely on GCC/clang all over the place already it seems nicer to
> avoid all the machinery for identifying which of the multiple
> different system header implementations is present, and
> instead just have a single implementation that works on
> all the hosts we care about...
> 

This is already a part of POSIX:

https://www.austingroupbugs.net/view.php?id=162

We have got everything needed from the standard now to implement bswap
without relying on compiler builtins. Every modern enough POSIX-like OS
already ships with .

> thanks
> -- PMM
> 

Re: [PATCH v2 7/8] qemu/bswap: Use compiler __builtin_bswap() on NetBSD

[Kamil Rytarowski]

On 28.09.2020 15:19, Philippe Mathieu-Daudé wrote:
> Since commit efc6c070aca ("configure: Add a test for the minimum
> compiler version") the minimum compiler version required for GCC
> is 4.8, which supports __builtin_bswap().
> Remove the NetBSD specific ifdef'ry.
> 
> This reverts commit 1360677cfe3ca8f945fa1de77823df21a77e4500
> ("makes NetBSD use the native bswap functions").
> 

Personally, I prefer using the system headers. but if you want to use
the GCC builtins, please go for it.

> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  configure| 15 ---
>  include/qemu/bswap.h |  5 -
>  2 files changed, 20 deletions(-)
> 
> diff --git a/configure b/configure
> index bff787daea7..1b0a02a0af8 100755
> --- a/configure
> +++ b/configure
> @@ -4886,18 +4886,6 @@ if test "$docs" != "no" ; then
>fi
>  fi
>  
> -# Search for bswap32 function
> -bswap_h=no
> -cat > $TMPC << EOF
> -#include 
> -#include 
> -#include 
> -int main(void) { return bswap32(0); }
> -EOF
> -if compile_prog "" "" ; then
> -  bswap_h=yes
> -fi
> -
>  ##
>  # Do we have libiscsi >= 1.9.0
>  if test "$libiscsi" != "no" ; then
> @@ -6779,9 +6767,6 @@ fi
>  if test "$st_atim" = "yes" ; then
>echo "HAVE_STRUCT_STAT_ST_ATIM=y" >> $config_host_mak
>  fi
> -if test "$bswap_h" = "yes" ; then
> -  echo "CONFIG_MACHINE_BSWAP_H=y" >> $config_host_mak
> -fi
>  if test "$curl" = "yes" ; then
>echo "CONFIG_CURL=y" >> $config_host_mak
>echo "CURL_CFLAGS=$curl_cflags" >> $config_host_mak
> diff --git a/include/qemu/bswap.h b/include/qemu/bswap.h
> index 1a297bfec22..7e586531c09 100644
> --- a/include/qemu/bswap.h
> +++ b/include/qemu/bswap.h
> @@ -3,17 +3,12 @@
>  
>  #include "fpu/softfloat-types.h"
>  
> -#ifdef CONFIG_MACHINE_BSWAP_H
> -# include 
> -# include 
> -#else
>  #undef  bswap16
>  #define bswap16(_x) __builtin_bswap16(_x)
>  #undef  bswap32
>  #define bswap32(_x) __builtin_bswap32(_x)
>  #undef  bswap64
>  #define bswap64(_x) __builtin_bswap64(_x)
> -#endif /* ! CONFIG_MACHINE_BSWAP_H */
>  
>  static inline void bswap16s(uint16_t *s)
>  {
> 

Re: [PATCH v5 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

Ping?

On 11.08.2020 15:01, Kamil Rytarowski wrote:
> From: Maxime Villard 
> 
> Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
> introduces the nvmm.h sysemu API for managing the vcpu scheduling and
> management.
> 
> Signed-off-by: Maxime Villard 
> Signed-off-by: Kamil Rytarowski 
> Reviewed-by: Sergio Lopez 
> Reviewed-by: Philippe Mathieu-Daudé 
> Tested-by: Jared McNeill 
> ---
>  accel/stubs/Makefile.objs |  1 +
>  accel/stubs/nvmm-stub.c   | 43 +++
>  include/sysemu/nvmm.h | 35 +++
>  3 files changed, 79 insertions(+)
>  create mode 100644 accel/stubs/nvmm-stub.c
>  create mode 100644 include/sysemu/nvmm.h
> 
> diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
> index bbd14e71fb..38660a0b9b 100644
> --- a/accel/stubs/Makefile.objs
> +++ b/accel/stubs/Makefile.objs
> @@ -1,6 +1,7 @@
>  obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
>  obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
>  obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
> +obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
>  obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
>  obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
>  obj-$(call lnot,$(CONFIG_XEN))  += xen-stub.o
> diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
> new file mode 100644
> index 00..c2208b84a3
> --- /dev/null
> +++ b/accel/stubs/nvmm-stub.c
> @@ -0,0 +1,43 @@
> +/*
> + * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
> + *
> + * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu-common.h"
> +#include "cpu.h"
> +#include "sysemu/nvmm.h"
> +
> +int nvmm_init_vcpu(CPUState *cpu)
> +{
> +return -1;
> +}
> +
> +int nvmm_vcpu_exec(CPUState *cpu)
> +{
> +return -1;
> +}
> +
> +void nvmm_destroy_vcpu(CPUState *cpu)
> +{
> +}
> +
> +void nvmm_cpu_synchronize_state(CPUState *cpu)
> +{
> +}
> +
> +void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
> +{
> +}
> +
> +void nvmm_cpu_synchronize_post_init(CPUState *cpu)
> +{
> +}
> +
> +void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
> +{
> +}
> diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
> new file mode 100644
> index 00..10496f3980
> --- /dev/null
> +++ b/include/sysemu/nvmm.h
> @@ -0,0 +1,35 @@
> +/*
> + * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
> + *
> + * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#ifndef QEMU_NVMM_H
> +#define QEMU_NVMM_H
> +
> +#include "config-host.h"
> +#include "qemu-common.h"
> +
> +int nvmm_init_vcpu(CPUState *);
> +int nvmm_vcpu_exec(CPUState *);
> +void nvmm_destroy_vcpu(CPUState *);
> +
> +void nvmm_cpu_synchronize_state(CPUState *);
> +void nvmm_cpu_synchronize_post_reset(CPUState *);
> +void nvmm_cpu_synchronize_post_init(CPUState *);
> +void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
> +
> +#ifdef CONFIG_NVMM
> +
> +int nvmm_enabled(void);
> +
> +#else /* CONFIG_NVMM */
> +
> +#define nvmm_enabled() (0)
> +
> +#endif /* CONFIG_NVMM */
> +
> +#endif /* CONFIG_NVMM */
> --
> 2.28.0
> 

Re: [PATCH v5 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

Ping?

On 11.08.2020 16:10, Kamil Rytarowski wrote:
> From: Maxime Villard 
> 
> Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
> introduces the nvmm.h sysemu API for managing the vcpu scheduling and
> management.
> 
> Signed-off-by: Maxime Villard 
> Signed-off-by: Kamil Rytarowski 
> Reviewed-by: Sergio Lopez 
> Reviewed-by: Philippe Mathieu-Daudé 
> Tested-by: Jared McNeill 
> ---
>  accel/stubs/Makefile.objs |  1 +
>  accel/stubs/nvmm-stub.c   | 43 +++
>  include/sysemu/nvmm.h | 35 +++
>  3 files changed, 79 insertions(+)
>  create mode 100644 accel/stubs/nvmm-stub.c
>  create mode 100644 include/sysemu/nvmm.h
> 
> diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
> index bbd14e71fb..38660a0b9b 100644
> --- a/accel/stubs/Makefile.objs
> +++ b/accel/stubs/Makefile.objs
> @@ -1,6 +1,7 @@
>  obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
>  obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
>  obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
> +obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
>  obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
>  obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
>  obj-$(call lnot,$(CONFIG_XEN))  += xen-stub.o
> diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
> new file mode 100644
> index 00..c2208b84a3
> --- /dev/null
> +++ b/accel/stubs/nvmm-stub.c
> @@ -0,0 +1,43 @@
> +/*
> + * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
> + *
> + * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu-common.h"
> +#include "cpu.h"
> +#include "sysemu/nvmm.h"
> +
> +int nvmm_init_vcpu(CPUState *cpu)
> +{
> +return -1;
> +}
> +
> +int nvmm_vcpu_exec(CPUState *cpu)
> +{
> +return -1;
> +}
> +
> +void nvmm_destroy_vcpu(CPUState *cpu)
> +{
> +}
> +
> +void nvmm_cpu_synchronize_state(CPUState *cpu)
> +{
> +}
> +
> +void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
> +{
> +}
> +
> +void nvmm_cpu_synchronize_post_init(CPUState *cpu)
> +{
> +}
> +
> +void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
> +{
> +}
> diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
> new file mode 100644
> index 00..10496f3980
> --- /dev/null
> +++ b/include/sysemu/nvmm.h
> @@ -0,0 +1,35 @@
> +/*
> + * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
> + *
> + * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#ifndef QEMU_NVMM_H
> +#define QEMU_NVMM_H
> +
> +#include "config-host.h"
> +#include "qemu-common.h"
> +
> +int nvmm_init_vcpu(CPUState *);
> +int nvmm_vcpu_exec(CPUState *);
> +void nvmm_destroy_vcpu(CPUState *);
> +
> +void nvmm_cpu_synchronize_state(CPUState *);
> +void nvmm_cpu_synchronize_post_reset(CPUState *);
> +void nvmm_cpu_synchronize_post_init(CPUState *);
> +void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
> +
> +#ifdef CONFIG_NVMM
> +
> +int nvmm_enabled(void);
> +
> +#else /* CONFIG_NVMM */
> +
> +#define nvmm_enabled() (0)
> +
> +#endif /* CONFIG_NVMM */
> +
> +#endif /* CONFIG_NVMM */
> --
> 2.28.0
> 

[PATCH v5 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NVMM accelerator cpu enlightenments to actually use the nvmm-all
accelerator on NetBSD platforms.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 include/sysemu/hw_accel.h | 14 ++
 softmmu/cpus.c| 58 +++
 target/i386/helper.c  |  2 +-
 3 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/hw_accel.h b/include/sysemu/hw_accel.h
index e128f8b06b..9e19f5794c 100644
--- a/include/sysemu/hw_accel.h
+++ b/include/sysemu/hw_accel.h
@@ -16,6 +16,7 @@
 #include "sysemu/kvm.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"
 
 static inline void cpu_synchronize_state(CPUState *cpu)
 {
@@ -31,6 +32,9 @@ static inline void cpu_synchronize_state(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_state(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_state(cpu);
+}
 }
 
 static inline void cpu_synchronize_post_reset(CPUState *cpu)
@@ -47,6 +51,10 @@ static inline void cpu_synchronize_post_reset(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_reset(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_reset(cpu);
+}
+
 }
 
 static inline void cpu_synchronize_post_init(CPUState *cpu)
@@ -63,6 +71,9 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_init(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_init(cpu);
+}
 }
 
 static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
@@ -79,6 +90,9 @@ static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_pre_loadvm(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_pre_loadvm(cpu);
+}
 }
 
 #endif /* QEMU_HW_ACCEL_H */
diff --git a/softmmu/cpus.c b/softmmu/cpus.c
index a802e899ab..3b44b92830 100644
--- a/softmmu/cpus.c
+++ b/softmmu/cpus.c
@@ -43,6 +43,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"
 #include "exec/exec-all.h"
 
 #include "qemu/thread.h"
@@ -1621,6 +1622,48 @@ static void *qemu_whpx_cpu_thread_fn(void *arg)
 return NULL;
 }
 
+static void *qemu_nvmm_cpu_thread_fn(void *arg)
+{
+CPUState *cpu = arg;
+int r;
+
+assert(nvmm_enabled());
+
+rcu_register_thread();
+
+qemu_mutex_lock_iothread();
+qemu_thread_get_self(cpu->thread);
+cpu->thread_id = qemu_get_thread_id();
+current_cpu = cpu;
+
+r = nvmm_init_vcpu(cpu);
+if (r < 0) {
+fprintf(stderr, "nvmm_init_vcpu failed: %s\n", strerror(-r));
+exit(1);
+}
+
+/* signal CPU creation */
+cpu->created = true;
+qemu_cond_signal(&qemu_cpu_cond);
+
+do {
+if (cpu_can_run(cpu)) {
+r = nvmm_vcpu_exec(cpu);
+if (r == EXCP_DEBUG) {
+cpu_handle_guest_debug(cpu);
+}
+}
+qemu_wait_io_event(cpu);
+} while (!cpu->unplug || cpu_can_run(cpu));
+
+nvmm_destroy_vcpu(cpu);
+cpu->created = false;
+qemu_cond_signal(&qemu_cpu_cond);
+qemu_mutex_unlock_iothread();
+rcu_unregister_thread();
+return NULL;
+}
+
 #ifdef _WIN32
 static void CALLBACK dummy_apc_func(ULONG_PTR unused)
 {
@@ -1998,6 +2041,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
 #endif
 }
 
+static void qemu_nvmm_start_vcpu(CPUState *cpu)
+{
+char thread_name[VCPU_THREAD_NAME_SIZE];
+
+cpu->thread = g_malloc0(sizeof(QemuThread));
+cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+qemu_cond_init(cpu->halt_cond);
+snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/NVMM",
+ cpu->cpu_index);
+qemu_thread_create(cpu->thread, thread_name, qemu_nvmm_cpu_thread_fn,
+   cpu, QEMU_THREAD_JOINABLE);
+}
+
 static void qemu_dummy_start_vcpu(CPUState *cpu)
 {
 char thread_name[VCPU_THREAD_NAME_SIZE];
@@ -2038,6 +2094,8 @@ void qemu_init_vcpu(CPUState *cpu)
 qemu_tcg_init_vcpu(cpu);
 } else if (whpx_enabled()) {
 qemu_whpx_start_vcpu(cpu);
+} else if (nvmm_enabled()) {
+qemu_nvmm_start_vcpu(cpu);
 } else {
 qemu_dummy_start_vcpu(cpu);
 }
diff --git a/target/i386/helper.c b/target/i386/helper.c
index 70be53e2c3..c2f1aef65c 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -983,7 +983,7 @@ void cpu_report_tpr_access(CPUX86State *env, TPRAccess 
access)
 X86CPU *cpu = env_archcpu(env);
 CPUState *cs = env_cpu(env);
 
-if (kvm_enabled() || whpx_enabled()) {
+if (kvm_enabled() || whpx_enabled() || nvmm_enabled()) {
 env->tpr_access_type = access;
 
 cpu_interrupt(cs, CPU_INTERRUPT_TPR);
-- 
2.24.1

[PATCH v5 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

From: Maxime Villard 

Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
introduces the nvmm.h sysemu API for managing the vcpu scheduling and
management.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 accel/stubs/Makefile.objs |  1 +
 accel/stubs/nvmm-stub.c   | 43 +++
 include/sysemu/nvmm.h | 35 +++
 3 files changed, 79 insertions(+)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h

diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
index bbd14e71fb..38660a0b9b 100644
--- a/accel/stubs/Makefile.objs
+++ b/accel/stubs/Makefile.objs
@@ -1,6 +1,7 @@
 obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
 obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
 obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
+obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
 obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
 obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
 obj-$(call lnot,$(CONFIG_XEN))  += xen-stub.o
diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
new file mode 100644
index 00..c2208b84a3
--- /dev/null
+++ b/accel/stubs/nvmm-stub.c
@@ -0,0 +1,43 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "sysemu/nvmm.h"
+
+int nvmm_init_vcpu(CPUState *cpu)
+{
+return -1;
+}
+
+int nvmm_vcpu_exec(CPUState *cpu)
+{
+return -1;
+}
+
+void nvmm_destroy_vcpu(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_state(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_init(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
+{
+}
diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
new file mode 100644
index 00..10496f3980
--- /dev/null
+++ b/include/sysemu/nvmm.h
@@ -0,0 +1,35 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_NVMM_H
+#define QEMU_NVMM_H
+
+#include "config-host.h"
+#include "qemu-common.h"
+
+int nvmm_init_vcpu(CPUState *);
+int nvmm_vcpu_exec(CPUState *);
+void nvmm_destroy_vcpu(CPUState *);
+
+void nvmm_cpu_synchronize_state(CPUState *);
+void nvmm_cpu_synchronize_post_reset(CPUState *);
+void nvmm_cpu_synchronize_post_init(CPUState *);
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
+
+#ifdef CONFIG_NVMM
+
+int nvmm_enabled(void);
+
+#else /* CONFIG_NVMM */
+
+#define nvmm_enabled() (0)
+
+#endif /* CONFIG_NVMM */
+
+#endif /* CONFIG_NVMM */
-- 
2.24.1

[PATCH v5 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

From: Maxime Villard 

Introduces the configure support for the new NetBSD Virtual Machine Monitor that
allows for hypervisor acceleration from usermode components on the NetBSD
platform.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 configure   | 37 +
 qemu-options.hx | 10 +-
 2 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/configure b/configure
index 2acc4d1465..fb9ffba2bf 100755
--- a/configure
+++ b/configure
@@ -246,6 +246,17 @@ supported_whpx_target() {
 return 1
 }
 
+supported_nvmm_target() {
+test "$nvmm" = "yes" || return 1
+glob "$1" "*-softmmu" || return 1
+case "${1%-softmmu}" in
+i386|x86_64)
+return 0
+;;
+esac
+return 1
+}
+
 supported_target() {
 case "$1" in
 *-softmmu)
@@ -273,6 +284,7 @@ supported_target() {
 supported_hax_target "$1" && return 0
 supported_hvf_target "$1" && return 0
 supported_whpx_target "$1" && return 0
+supported_nvmm_target "$1" && return 0
 print_error "TCG disabled, but hardware accelerator not available for 
'$target'"
 return 1
 }
@@ -395,6 +407,7 @@ kvm="no"
 hax="no"
 hvf="no"
 whpx="no"
+nvmm="no"
 rdma=""
 pvrdma=""
 gprof="no"
@@ -847,6 +860,7 @@ DragonFly)
 NetBSD)
   bsd="yes"
   hax="yes"
+  nvmm="yes"
   make="${MAKE-gmake}"
   audio_drv_list="oss try-sdl"
   audio_possible_drivers="oss sdl"
@@ -1233,6 +1247,10 @@ for opt do
   ;;
   --enable-whpx) whpx="yes"
   ;;
+  --disable-nvmm) nvmm="no"
+  ;;
+  --enable-nvmm) nvmm="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1879,6 +1897,7 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   hax HAX acceleration support
   hvf Hypervisor.framework acceleration support
   whpxWindows Hypervisor Platform acceleration support
+  nvmmNetBSD Virtual Machine Monitor acceleration support
   rdmaEnable RDMA-based migration
   pvrdma  Enable PVRDMA support
   vde support for vde network
@@ -2965,6 +2984,20 @@ if test "$whpx" != "no" ; then
 fi
 fi
 
+##
+# NetBSD Virtual Machine Monitor (NVMM) accelerator check
+if test "$nvmm" != "no" ; then
+if check_include "nvmm.h" ; then
+nvmm="yes"
+   LIBS="-lnvmm $LIBS"
+else
+if test "$nvmm" = "yes"; then
+feature_not_found "NVMM" "NVMM is not available"
+fi
+nvmm="no"
+fi
+fi
+
 ##
 # Sparse probe
 if test "$sparse" != "no" ; then
@@ -6934,6 +6967,7 @@ echo "KVM support   $kvm"
 echo "HAX support   $hax"
 echo "HVF support   $hvf"
 echo "WHPX support  $whpx"
+echo "NVMM support  $nvmm"
 echo "TCG support   $tcg"
 if test "$tcg" = "yes" ; then
 echo "TCG debug enabled $debug_tcg"
@@ -8332,6 +8366,9 @@ fi
 if test "$target_aligned_only" = "yes" ; then
   echo "TARGET_ALIGNED_ONLY=y" >> $config_target_mak
 fi
+if supported_nvmm_target $target; then
+echo "CONFIG_NVMM=y" >> $config_target_mak
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi
diff --git a/qemu-options.hx b/qemu-options.hx
index 708583b4ce..697accaa7e 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -26,7 +26,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
 "-machine [type=]name[,prop[=value][,...]]\n"
 "selects emulated machine ('-machine help' for list)\n"
 "property accel=accel1[:accel2[:...]] selects 
accelerator\n"
-"supported accelerators are kvm, xen, hax, hvf, whpx or 
tcg (default: tcg)\n"
+"supported accelerators are kvm, xen, hax, hvf, nvmm, whpx 
or tcg (default: tcg)\n"
 "vmport=on|off|auto controls emulation of vmport (default: 
auto)\n"
 "dump-guest-core=on|off include guest memory in a core 
dump (default=on)\n"
 "mem-merge=on|off 

[PATCH v5 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NetBSD Virtual Machine Monitor (NVMM) target. Which
acts as a hypervisor accelerator for QEMU on the NetBSD platform. This enables
QEMU much greater speed over the emulated x86_64 path's that are taken on
NetBSD today.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Tested-by: Jared McNeill 
---
 target/i386/Makefile.objs |1 +
 target/i386/nvmm-all.c| 1226 +
 2 files changed, 1227 insertions(+)
 create mode 100644 target/i386/nvmm-all.c

diff --git a/target/i386/Makefile.objs b/target/i386/Makefile.objs
index 0b93143e27..ff0df68404 100644
--- a/target/i386/Makefile.objs
+++ b/target/i386/Makefile.objs
@@ -18,6 +18,7 @@ obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-posix.o
 endif
 obj-$(CONFIG_HVF) += hvf/
 obj-$(CONFIG_WHPX) += whpx-all.o
+obj-$(CONFIG_NVMM) += nvmm-all.o
 endif
 obj-$(CONFIG_SEV) += sev.o
 obj-$(call lnot,$(CONFIG_SEV)) += sev-stub.o
diff --git a/target/i386/nvmm-all.c b/target/i386/nvmm-all.c
new file mode 100644
index 00..408f7305b9
--- /dev/null
+++ b/target/i386/nvmm-all.c
@@ -0,0 +1,1226 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator for QEMU.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/address-spaces.h"
+#include "exec/ioport.h"
+#include "qemu-common.h"
+#include "strings.h"
+#include "sysemu/accel.h"
+#include "sysemu/nvmm.h"
+#include "sysemu/runstate.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+
+#include 
+
+struct qemu_vcpu {
+struct nvmm_vcpu vcpu;
+uint8_t tpr;
+bool stop;
+
+/* Window-exiting for INTs/NMIs. */
+bool int_window_exit;
+bool nmi_window_exit;
+
+/* The guest is in an interrupt shadow (POP SS, etc). */
+bool int_shadow;
+};
+
+struct qemu_machine {
+struct nvmm_capability cap;
+struct nvmm_machine mach;
+};
+
+/* -- 
*/
+
+static bool nvmm_allowed;
+static struct qemu_machine qemu_mach;
+
+static struct qemu_vcpu *
+get_qemu_vcpu(CPUState *cpu)
+{
+return (struct qemu_vcpu *)cpu->hax_vcpu;
+}
+
+static struct nvmm_machine *
+get_nvmm_mach(void)
+{
+return &qemu_mach.mach;
+}
+
+/* -- 
*/
+
+static void
+nvmm_set_segment(struct nvmm_x64_state_seg *nseg, const SegmentCache *qseg)
+{
+uint32_t attrib = qseg->flags;
+
+nseg->selector = qseg->selector;
+nseg->limit = qseg->limit;
+nseg->base = qseg->base;
+nseg->attrib.type = __SHIFTOUT(attrib, DESC_TYPE_MASK);
+nseg->attrib.s = __SHIFTOUT(attrib, DESC_S_MASK);
+nseg->attrib.dpl = __SHIFTOUT(attrib, DESC_DPL_MASK);
+nseg->attrib.p = __SHIFTOUT(attrib, DESC_P_MASK);
+nseg->attrib.avl = __SHIFTOUT(attrib, DESC_AVL_MASK);
+nseg->attrib.l = __SHIFTOUT(attrib, DESC_L_MASK);
+nseg->attrib.def = __SHIFTOUT(attrib, DESC_B_MASK);
+nseg->attrib.g = __SHIFTOUT(attrib, DESC_G_MASK);
+}
+
+static void
+nvmm_set_registers(CPUState *cpu)
+{
+struct CPUX86State *env = (CPUArchState *)cpu->env_ptr;
+struct nvmm_machine *mach = get_nvmm_mach();
+struct qemu_vcpu *qcpu = get_qemu_vcpu(cpu);
+struct nvmm_vcpu *vcpu = &qcpu->vcpu;
+struct nvmm_x64_state *state = vcpu->state;
+uint64_t bitmap;
+size_t i;
+int ret;
+
+assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
+
+/* GPRs. */
+state->gprs[NVMM_X64_GPR_RAX] = env->regs[R_EAX];
+state->gprs[NVMM_X64_GPR_RCX] = env->regs[R_ECX];
+state->gprs[NVMM_X64_GPR_RDX] = env->regs[R_EDX];
+state->gprs[NVMM_X64_GPR_RBX] = env->regs[R_EBX];
+state->gprs[NVMM_X64_GPR_RSP] = env->regs[R_ESP];
+state->gprs[NVMM_X64_GPR_RBP] = env->regs[R_EBP];
+state->gprs[NVMM_X64_GPR_RSI] = env->regs[R_ESI];
+state->gprs[NVMM_X64_GPR_RDI] = env->regs[R_EDI];
+#ifdef TARGET_X86_64
+state->gprs[NVMM_X64_GPR_R8]  = env->regs[R_R8];
+state->gprs[NVMM_X64_GPR_R9]  = env->regs[R_R9];
+state->gprs[NVMM_X64_GPR_R10] = env->regs[R_R10];
+state->gprs[NVMM_X64_GPR_R11] = env->regs[R_R11];
+state->gprs[NVMM_X64_GPR_R12] = env->regs[R_R12];
+state->gprs[NVMM_X64_GPR_R13] = env->regs[R_R13];
+state->gprs[NVMM_X64_GPR_R14] = env->regs[R_R14];
+state->gp

[PATCH v5 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NetBSD Virtual Machine Monitor (NVMM) target. Which
acts as a hypervisor accelerator for QEMU on the NetBSD platform. This enables
QEMU much greater speed over the emulated x86_64 path's that are taken on
NetBSD today.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Tested-by: Jared McNeill 
---
 target/i386/Makefile.objs |1 +
 target/i386/nvmm-all.c| 1226 +
 2 files changed, 1227 insertions(+)
 create mode 100644 target/i386/nvmm-all.c

diff --git a/target/i386/Makefile.objs b/target/i386/Makefile.objs
index 0b93143e27..ff0df68404 100644
--- a/target/i386/Makefile.objs
+++ b/target/i386/Makefile.objs
@@ -18,6 +18,7 @@ obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-posix.o
 endif
 obj-$(CONFIG_HVF) += hvf/
 obj-$(CONFIG_WHPX) += whpx-all.o
+obj-$(CONFIG_NVMM) += nvmm-all.o
 endif
 obj-$(CONFIG_SEV) += sev.o
 obj-$(call lnot,$(CONFIG_SEV)) += sev-stub.o
diff --git a/target/i386/nvmm-all.c b/target/i386/nvmm-all.c
new file mode 100644
index 00..408f7305b9
--- /dev/null
+++ b/target/i386/nvmm-all.c
@@ -0,0 +1,1226 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator for QEMU.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/address-spaces.h"
+#include "exec/ioport.h"
+#include "qemu-common.h"
+#include "strings.h"
+#include "sysemu/accel.h"
+#include "sysemu/nvmm.h"
+#include "sysemu/runstate.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+
+#include 
+
+struct qemu_vcpu {
+struct nvmm_vcpu vcpu;
+uint8_t tpr;
+bool stop;
+
+/* Window-exiting for INTs/NMIs. */
+bool int_window_exit;
+bool nmi_window_exit;
+
+/* The guest is in an interrupt shadow (POP SS, etc). */
+bool int_shadow;
+};
+
+struct qemu_machine {
+struct nvmm_capability cap;
+struct nvmm_machine mach;
+};
+
+/* -- 
*/
+
+static bool nvmm_allowed;
+static struct qemu_machine qemu_mach;
+
+static struct qemu_vcpu *
+get_qemu_vcpu(CPUState *cpu)
+{
+return (struct qemu_vcpu *)cpu->hax_vcpu;
+}
+
+static struct nvmm_machine *
+get_nvmm_mach(void)
+{
+return &qemu_mach.mach;
+}
+
+/* -- 
*/
+
+static void
+nvmm_set_segment(struct nvmm_x64_state_seg *nseg, const SegmentCache *qseg)
+{
+uint32_t attrib = qseg->flags;
+
+nseg->selector = qseg->selector;
+nseg->limit = qseg->limit;
+nseg->base = qseg->base;
+nseg->attrib.type = __SHIFTOUT(attrib, DESC_TYPE_MASK);
+nseg->attrib.s = __SHIFTOUT(attrib, DESC_S_MASK);
+nseg->attrib.dpl = __SHIFTOUT(attrib, DESC_DPL_MASK);
+nseg->attrib.p = __SHIFTOUT(attrib, DESC_P_MASK);
+nseg->attrib.avl = __SHIFTOUT(attrib, DESC_AVL_MASK);
+nseg->attrib.l = __SHIFTOUT(attrib, DESC_L_MASK);
+nseg->attrib.def = __SHIFTOUT(attrib, DESC_B_MASK);
+nseg->attrib.g = __SHIFTOUT(attrib, DESC_G_MASK);
+}
+
+static void
+nvmm_set_registers(CPUState *cpu)
+{
+struct CPUX86State *env = (CPUArchState *)cpu->env_ptr;
+struct nvmm_machine *mach = get_nvmm_mach();
+struct qemu_vcpu *qcpu = get_qemu_vcpu(cpu);
+struct nvmm_vcpu *vcpu = &qcpu->vcpu;
+struct nvmm_x64_state *state = vcpu->state;
+uint64_t bitmap;
+size_t i;
+int ret;
+
+assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
+
+/* GPRs. */
+state->gprs[NVMM_X64_GPR_RAX] = env->regs[R_EAX];
+state->gprs[NVMM_X64_GPR_RCX] = env->regs[R_ECX];
+state->gprs[NVMM_X64_GPR_RDX] = env->regs[R_EDX];
+state->gprs[NVMM_X64_GPR_RBX] = env->regs[R_EBX];
+state->gprs[NVMM_X64_GPR_RSP] = env->regs[R_ESP];
+state->gprs[NVMM_X64_GPR_RBP] = env->regs[R_EBP];
+state->gprs[NVMM_X64_GPR_RSI] = env->regs[R_ESI];
+state->gprs[NVMM_X64_GPR_RDI] = env->regs[R_EDI];
+#ifdef TARGET_X86_64
+state->gprs[NVMM_X64_GPR_R8]  = env->regs[R_R8];
+state->gprs[NVMM_X64_GPR_R9]  = env->regs[R_R9];
+state->gprs[NVMM_X64_GPR_R10] = env->regs[R_R10];
+state->gprs[NVMM_X64_GPR_R11] = env->regs[R_R11];
+state->gprs[NVMM_X64_GPR_R12] = env->regs[R_R12];
+state->gprs[NVMM_X64_GPR_R13] = env->regs[R_R13];
+state->gprs[NVMM_X64_GPR_R14] = env->regs[R_R14];
+state->gp

[PATCH v5 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NVMM accelerator cpu enlightenments to actually use the nvmm-all
accelerator on NetBSD platforms.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 include/sysemu/hw_accel.h | 14 ++
 softmmu/cpus.c| 58 +++
 target/i386/helper.c  |  2 +-
 3 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/hw_accel.h b/include/sysemu/hw_accel.h
index e128f8b06b..9e19f5794c 100644
--- a/include/sysemu/hw_accel.h
+++ b/include/sysemu/hw_accel.h
@@ -16,6 +16,7 @@
 #include "sysemu/kvm.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"

 static inline void cpu_synchronize_state(CPUState *cpu)
 {
@@ -31,6 +32,9 @@ static inline void cpu_synchronize_state(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_state(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_state(cpu);
+}
 }

 static inline void cpu_synchronize_post_reset(CPUState *cpu)
@@ -47,6 +51,10 @@ static inline void cpu_synchronize_post_reset(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_reset(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_reset(cpu);
+}
+
 }

 static inline void cpu_synchronize_post_init(CPUState *cpu)
@@ -63,6 +71,9 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_init(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_init(cpu);
+}
 }

 static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
@@ -79,6 +90,9 @@ static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_pre_loadvm(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_pre_loadvm(cpu);
+}
 }

 #endif /* QEMU_HW_ACCEL_H */
diff --git a/softmmu/cpus.c b/softmmu/cpus.c
index a802e899ab..3b44b92830 100644
--- a/softmmu/cpus.c
+++ b/softmmu/cpus.c
@@ -43,6 +43,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"
 #include "exec/exec-all.h"

 #include "qemu/thread.h"
@@ -1621,6 +1622,48 @@ static void *qemu_whpx_cpu_thread_fn(void *arg)
 return NULL;
 }

+static void *qemu_nvmm_cpu_thread_fn(void *arg)
+{
+CPUState *cpu = arg;
+int r;
+
+assert(nvmm_enabled());
+
+rcu_register_thread();
+
+qemu_mutex_lock_iothread();
+qemu_thread_get_self(cpu->thread);
+cpu->thread_id = qemu_get_thread_id();
+current_cpu = cpu;
+
+r = nvmm_init_vcpu(cpu);
+if (r < 0) {
+fprintf(stderr, "nvmm_init_vcpu failed: %s\n", strerror(-r));
+exit(1);
+}
+
+/* signal CPU creation */
+cpu->created = true;
+qemu_cond_signal(&qemu_cpu_cond);
+
+do {
+if (cpu_can_run(cpu)) {
+r = nvmm_vcpu_exec(cpu);
+if (r == EXCP_DEBUG) {
+cpu_handle_guest_debug(cpu);
+}
+}
+qemu_wait_io_event(cpu);
+} while (!cpu->unplug || cpu_can_run(cpu));
+
+nvmm_destroy_vcpu(cpu);
+cpu->created = false;
+qemu_cond_signal(&qemu_cpu_cond);
+qemu_mutex_unlock_iothread();
+rcu_unregister_thread();
+return NULL;
+}
+
 #ifdef _WIN32
 static void CALLBACK dummy_apc_func(ULONG_PTR unused)
 {
@@ -1998,6 +2041,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
 #endif
 }

+static void qemu_nvmm_start_vcpu(CPUState *cpu)
+{
+char thread_name[VCPU_THREAD_NAME_SIZE];
+
+cpu->thread = g_malloc0(sizeof(QemuThread));
+cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+qemu_cond_init(cpu->halt_cond);
+snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/NVMM",
+ cpu->cpu_index);
+qemu_thread_create(cpu->thread, thread_name, qemu_nvmm_cpu_thread_fn,
+   cpu, QEMU_THREAD_JOINABLE);
+}
+
 static void qemu_dummy_start_vcpu(CPUState *cpu)
 {
 char thread_name[VCPU_THREAD_NAME_SIZE];
@@ -2038,6 +2094,8 @@ void qemu_init_vcpu(CPUState *cpu)
 qemu_tcg_init_vcpu(cpu);
 } else if (whpx_enabled()) {
 qemu_whpx_start_vcpu(cpu);
+} else if (nvmm_enabled()) {
+qemu_nvmm_start_vcpu(cpu);
 } else {
 qemu_dummy_start_vcpu(cpu);
 }
diff --git a/target/i386/helper.c b/target/i386/helper.c
index 70be53e2c3..c2f1aef65c 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -983,7 +983,7 @@ void cpu_report_tpr_access(CPUX86State *env, TPRAccess 
access)
 X86CPU *cpu = env_archcpu(env);
 CPUState *cs = env_cpu(env);

-if (kvm_enabled() || whpx_enabled()) {
+if (kvm_enabled() || whpx_enabled() || nvmm_enabled()) {
 env->tpr_access_type = access;

 cpu_interrupt(cs, CPU_INTERRUPT_TPR);
--
2.28.0

[PATCH v5 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

From: Maxime Villard 

Introduces the configure support for the new NetBSD Virtual Machine Monitor that
allows for hypervisor acceleration from usermode components on the NetBSD
platform.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 configure   | 37 +
 qemu-options.hx | 10 +-
 2 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/configure b/configure
index 2acc4d1465..fb9ffba2bf 100755
--- a/configure
+++ b/configure
@@ -246,6 +246,17 @@ supported_whpx_target() {
 return 1
 }

+supported_nvmm_target() {
+test "$nvmm" = "yes" || return 1
+glob "$1" "*-softmmu" || return 1
+case "${1%-softmmu}" in
+i386|x86_64)
+return 0
+;;
+esac
+return 1
+}
+
 supported_target() {
 case "$1" in
 *-softmmu)
@@ -273,6 +284,7 @@ supported_target() {
 supported_hax_target "$1" && return 0
 supported_hvf_target "$1" && return 0
 supported_whpx_target "$1" && return 0
+supported_nvmm_target "$1" && return 0
 print_error "TCG disabled, but hardware accelerator not available for 
'$target'"
 return 1
 }
@@ -395,6 +407,7 @@ kvm="no"
 hax="no"
 hvf="no"
 whpx="no"
+nvmm="no"
 rdma=""
 pvrdma=""
 gprof="no"
@@ -847,6 +860,7 @@ DragonFly)
 NetBSD)
   bsd="yes"
   hax="yes"
+  nvmm="yes"
   make="${MAKE-gmake}"
   audio_drv_list="oss try-sdl"
   audio_possible_drivers="oss sdl"
@@ -1233,6 +1247,10 @@ for opt do
   ;;
   --enable-whpx) whpx="yes"
   ;;
+  --disable-nvmm) nvmm="no"
+  ;;
+  --enable-nvmm) nvmm="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1879,6 +1897,7 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   hax HAX acceleration support
   hvf Hypervisor.framework acceleration support
   whpxWindows Hypervisor Platform acceleration support
+  nvmmNetBSD Virtual Machine Monitor acceleration support
   rdmaEnable RDMA-based migration
   pvrdma  Enable PVRDMA support
   vde support for vde network
@@ -2965,6 +2984,20 @@ if test "$whpx" != "no" ; then
 fi
 fi

+##
+# NetBSD Virtual Machine Monitor (NVMM) accelerator check
+if test "$nvmm" != "no" ; then
+if check_include "nvmm.h" ; then
+nvmm="yes"
+   LIBS="-lnvmm $LIBS"
+else
+if test "$nvmm" = "yes"; then
+feature_not_found "NVMM" "NVMM is not available"
+fi
+nvmm="no"
+fi
+fi
+
 ##
 # Sparse probe
 if test "$sparse" != "no" ; then
@@ -6934,6 +6967,7 @@ echo "KVM support   $kvm"
 echo "HAX support   $hax"
 echo "HVF support   $hvf"
 echo "WHPX support  $whpx"
+echo "NVMM support  $nvmm"
 echo "TCG support   $tcg"
 if test "$tcg" = "yes" ; then
 echo "TCG debug enabled $debug_tcg"
@@ -8332,6 +8366,9 @@ fi
 if test "$target_aligned_only" = "yes" ; then
   echo "TARGET_ALIGNED_ONLY=y" >> $config_target_mak
 fi
+if supported_nvmm_target $target; then
+echo "CONFIG_NVMM=y" >> $config_target_mak
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi
diff --git a/qemu-options.hx b/qemu-options.hx
index 708583b4ce..697accaa7e 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -26,7 +26,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
 "-machine [type=]name[,prop[=value][,...]]\n"
 "selects emulated machine ('-machine help' for list)\n"
 "property accel=accel1[:accel2[:...]] selects 
accelerator\n"
-"supported accelerators are kvm, xen, hax, hvf, whpx or 
tcg (default: tcg)\n"
+"supported accelerators are kvm, xen, hax, hvf, nvmm, whpx 
or tcg (default: tcg)\n"
 "vmport=on|off|auto controls emulation of vmport (default: 
auto)\n"
 "dump-guest-core=on|off include guest memory in a core 
dump (default=on)\n"
 "mem-merge=on|off controls m

[PATCH v5 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

From: Maxime Villard 

Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
introduces the nvmm.h sysemu API for managing the vcpu scheduling and
management.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 accel/stubs/Makefile.objs |  1 +
 accel/stubs/nvmm-stub.c   | 43 +++
 include/sysemu/nvmm.h | 35 +++
 3 files changed, 79 insertions(+)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h

diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
index bbd14e71fb..38660a0b9b 100644
--- a/accel/stubs/Makefile.objs
+++ b/accel/stubs/Makefile.objs
@@ -1,6 +1,7 @@
 obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
 obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
 obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
+obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
 obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
 obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
 obj-$(call lnot,$(CONFIG_XEN))  += xen-stub.o
diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
new file mode 100644
index 00..c2208b84a3
--- /dev/null
+++ b/accel/stubs/nvmm-stub.c
@@ -0,0 +1,43 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "sysemu/nvmm.h"
+
+int nvmm_init_vcpu(CPUState *cpu)
+{
+return -1;
+}
+
+int nvmm_vcpu_exec(CPUState *cpu)
+{
+return -1;
+}
+
+void nvmm_destroy_vcpu(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_state(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_init(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
+{
+}
diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
new file mode 100644
index 00..10496f3980
--- /dev/null
+++ b/include/sysemu/nvmm.h
@@ -0,0 +1,35 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_NVMM_H
+#define QEMU_NVMM_H
+
+#include "config-host.h"
+#include "qemu-common.h"
+
+int nvmm_init_vcpu(CPUState *);
+int nvmm_vcpu_exec(CPUState *);
+void nvmm_destroy_vcpu(CPUState *);
+
+void nvmm_cpu_synchronize_state(CPUState *);
+void nvmm_cpu_synchronize_post_reset(CPUState *);
+void nvmm_cpu_synchronize_post_init(CPUState *);
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
+
+#ifdef CONFIG_NVMM
+
+int nvmm_enabled(void);
+
+#else /* CONFIG_NVMM */
+
+#define nvmm_enabled() (0)
+
+#endif /* CONFIG_NVMM */
+
+#endif /* CONFIG_NVMM */
--
2.28.0

[PATCH v5 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NVMM accelerator cpu enlightenments to actually use the nvmm-all
accelerator on NetBSD platforms.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 include/sysemu/hw_accel.h | 14 ++
 softmmu/cpus.c| 58 +++
 target/i386/helper.c  |  2 +-
 3 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/hw_accel.h b/include/sysemu/hw_accel.h
index e128f8b06b..9e19f5794c 100644
--- a/include/sysemu/hw_accel.h
+++ b/include/sysemu/hw_accel.h
@@ -16,6 +16,7 @@
 #include "sysemu/kvm.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"

 static inline void cpu_synchronize_state(CPUState *cpu)
 {
@@ -31,6 +32,9 @@ static inline void cpu_synchronize_state(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_state(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_state(cpu);
+}
 }

 static inline void cpu_synchronize_post_reset(CPUState *cpu)
@@ -47,6 +51,10 @@ static inline void cpu_synchronize_post_reset(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_reset(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_reset(cpu);
+}
+
 }

 static inline void cpu_synchronize_post_init(CPUState *cpu)
@@ -63,6 +71,9 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_init(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_init(cpu);
+}
 }

 static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
@@ -79,6 +90,9 @@ static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_pre_loadvm(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_pre_loadvm(cpu);
+}
 }

 #endif /* QEMU_HW_ACCEL_H */
diff --git a/softmmu/cpus.c b/softmmu/cpus.c
index a802e899ab..3b44b92830 100644
--- a/softmmu/cpus.c
+++ b/softmmu/cpus.c
@@ -43,6 +43,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"
 #include "exec/exec-all.h"

 #include "qemu/thread.h"
@@ -1621,6 +1622,48 @@ static void *qemu_whpx_cpu_thread_fn(void *arg)
 return NULL;
 }

+static void *qemu_nvmm_cpu_thread_fn(void *arg)
+{
+CPUState *cpu = arg;
+int r;
+
+assert(nvmm_enabled());
+
+rcu_register_thread();
+
+qemu_mutex_lock_iothread();
+qemu_thread_get_self(cpu->thread);
+cpu->thread_id = qemu_get_thread_id();
+current_cpu = cpu;
+
+r = nvmm_init_vcpu(cpu);
+if (r < 0) {
+fprintf(stderr, "nvmm_init_vcpu failed: %s\n", strerror(-r));
+exit(1);
+}
+
+/* signal CPU creation */
+cpu->created = true;
+qemu_cond_signal(&qemu_cpu_cond);
+
+do {
+if (cpu_can_run(cpu)) {
+r = nvmm_vcpu_exec(cpu);
+if (r == EXCP_DEBUG) {
+cpu_handle_guest_debug(cpu);
+}
+}
+qemu_wait_io_event(cpu);
+} while (!cpu->unplug || cpu_can_run(cpu));
+
+nvmm_destroy_vcpu(cpu);
+cpu->created = false;
+qemu_cond_signal(&qemu_cpu_cond);
+qemu_mutex_unlock_iothread();
+rcu_unregister_thread();
+return NULL;
+}
+
 #ifdef _WIN32
 static void CALLBACK dummy_apc_func(ULONG_PTR unused)
 {
@@ -1998,6 +2041,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
 #endif
 }

+static void qemu_nvmm_start_vcpu(CPUState *cpu)
+{
+char thread_name[VCPU_THREAD_NAME_SIZE];
+
+cpu->thread = g_malloc0(sizeof(QemuThread));
+cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+qemu_cond_init(cpu->halt_cond);
+snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/NVMM",
+ cpu->cpu_index);
+qemu_thread_create(cpu->thread, thread_name, qemu_nvmm_cpu_thread_fn,
+   cpu, QEMU_THREAD_JOINABLE);
+}
+
 static void qemu_dummy_start_vcpu(CPUState *cpu)
 {
 char thread_name[VCPU_THREAD_NAME_SIZE];
@@ -2038,6 +2094,8 @@ void qemu_init_vcpu(CPUState *cpu)
 qemu_tcg_init_vcpu(cpu);
 } else if (whpx_enabled()) {
 qemu_whpx_start_vcpu(cpu);
+} else if (nvmm_enabled()) {
+qemu_nvmm_start_vcpu(cpu);
 } else {
 qemu_dummy_start_vcpu(cpu);
 }
diff --git a/target/i386/helper.c b/target/i386/helper.c
index 70be53e2c3..c2f1aef65c 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -983,7 +983,7 @@ void cpu_report_tpr_access(CPUX86State *env, TPRAccess 
access)
 X86CPU *cpu = env_archcpu(env);
 CPUState *cs = env_cpu(env);

-if (kvm_enabled() || whpx_enabled()) {
+if (kvm_enabled() || whpx_enabled() || nvmm_enabled()) {
 env->tpr_access_type = access;

 cpu_interrupt(cs, CPU_INTERRUPT_TPR);
--
2.28.0

[PATCH v5 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

From: Maxime Villard 

Introduces the configure support for the new NetBSD Virtual Machine Monitor that
allows for hypervisor acceleration from usermode components on the NetBSD
platform.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 configure   | 37 +
 qemu-options.hx | 10 +-
 2 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/configure b/configure
index 2acc4d1465..fb9ffba2bf 100755
--- a/configure
+++ b/configure
@@ -246,6 +246,17 @@ supported_whpx_target() {
 return 1
 }

+supported_nvmm_target() {
+test "$nvmm" = "yes" || return 1
+glob "$1" "*-softmmu" || return 1
+case "${1%-softmmu}" in
+i386|x86_64)
+return 0
+;;
+esac
+return 1
+}
+
 supported_target() {
 case "$1" in
 *-softmmu)
@@ -273,6 +284,7 @@ supported_target() {
 supported_hax_target "$1" && return 0
 supported_hvf_target "$1" && return 0
 supported_whpx_target "$1" && return 0
+supported_nvmm_target "$1" && return 0
 print_error "TCG disabled, but hardware accelerator not available for 
'$target'"
 return 1
 }
@@ -395,6 +407,7 @@ kvm="no"
 hax="no"
 hvf="no"
 whpx="no"
+nvmm="no"
 rdma=""
 pvrdma=""
 gprof="no"
@@ -847,6 +860,7 @@ DragonFly)
 NetBSD)
   bsd="yes"
   hax="yes"
+  nvmm="yes"
   make="${MAKE-gmake}"
   audio_drv_list="oss try-sdl"
   audio_possible_drivers="oss sdl"
@@ -1233,6 +1247,10 @@ for opt do
   ;;
   --enable-whpx) whpx="yes"
   ;;
+  --disable-nvmm) nvmm="no"
+  ;;
+  --enable-nvmm) nvmm="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1879,6 +1897,7 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   hax HAX acceleration support
   hvf Hypervisor.framework acceleration support
   whpxWindows Hypervisor Platform acceleration support
+  nvmmNetBSD Virtual Machine Monitor acceleration support
   rdmaEnable RDMA-based migration
   pvrdma  Enable PVRDMA support
   vde support for vde network
@@ -2965,6 +2984,20 @@ if test "$whpx" != "no" ; then
 fi
 fi

+##
+# NetBSD Virtual Machine Monitor (NVMM) accelerator check
+if test "$nvmm" != "no" ; then
+if check_include "nvmm.h" ; then
+nvmm="yes"
+   LIBS="-lnvmm $LIBS"
+else
+if test "$nvmm" = "yes"; then
+feature_not_found "NVMM" "NVMM is not available"
+fi
+nvmm="no"
+fi
+fi
+
 ##
 # Sparse probe
 if test "$sparse" != "no" ; then
@@ -6934,6 +6967,7 @@ echo "KVM support   $kvm"
 echo "HAX support   $hax"
 echo "HVF support   $hvf"
 echo "WHPX support  $whpx"
+echo "NVMM support  $nvmm"
 echo "TCG support   $tcg"
 if test "$tcg" = "yes" ; then
 echo "TCG debug enabled $debug_tcg"
@@ -8332,6 +8366,9 @@ fi
 if test "$target_aligned_only" = "yes" ; then
   echo "TARGET_ALIGNED_ONLY=y" >> $config_target_mak
 fi
+if supported_nvmm_target $target; then
+echo "CONFIG_NVMM=y" >> $config_target_mak
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi
diff --git a/qemu-options.hx b/qemu-options.hx
index 708583b4ce..697accaa7e 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -26,7 +26,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
 "-machine [type=]name[,prop[=value][,...]]\n"
 "selects emulated machine ('-machine help' for list)\n"
 "property accel=accel1[:accel2[:...]] selects 
accelerator\n"
-"supported accelerators are kvm, xen, hax, hvf, whpx or 
tcg (default: tcg)\n"
+"supported accelerators are kvm, xen, hax, hvf, nvmm, whpx 
or tcg (default: tcg)\n"
 "vmport=on|off|auto controls emulation of vmport (default: 
auto)\n"
 "dump-guest-core=on|off include guest memory in a core 
dump (default=on)\n"
 "mem-merge=on|off controls m

[PATCH v5 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NetBSD Virtual Machine Monitor (NVMM) target. Which
acts as a hypervisor accelerator for QEMU on the NetBSD platform. This enables
QEMU much greater speed over the emulated x86_64 path's that are taken on
NetBSD today.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Tested-by: Jared McNeill 
---
 target/i386/Makefile.objs |1 +
 target/i386/nvmm-all.c| 1226 +
 2 files changed, 1227 insertions(+)
 create mode 100644 target/i386/nvmm-all.c

diff --git a/target/i386/Makefile.objs b/target/i386/Makefile.objs
index 0b93143e27..ff0df68404 100644
--- a/target/i386/Makefile.objs
+++ b/target/i386/Makefile.objs
@@ -18,6 +18,7 @@ obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-posix.o
 endif
 obj-$(CONFIG_HVF) += hvf/
 obj-$(CONFIG_WHPX) += whpx-all.o
+obj-$(CONFIG_NVMM) += nvmm-all.o
 endif
 obj-$(CONFIG_SEV) += sev.o
 obj-$(call lnot,$(CONFIG_SEV)) += sev-stub.o
diff --git a/target/i386/nvmm-all.c b/target/i386/nvmm-all.c
new file mode 100644
index 00..408f7305b9
--- /dev/null
+++ b/target/i386/nvmm-all.c
@@ -0,0 +1,1226 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator for QEMU.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/address-spaces.h"
+#include "exec/ioport.h"
+#include "qemu-common.h"
+#include "strings.h"
+#include "sysemu/accel.h"
+#include "sysemu/nvmm.h"
+#include "sysemu/runstate.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+
+#include 
+
+struct qemu_vcpu {
+struct nvmm_vcpu vcpu;
+uint8_t tpr;
+bool stop;
+
+/* Window-exiting for INTs/NMIs. */
+bool int_window_exit;
+bool nmi_window_exit;
+
+/* The guest is in an interrupt shadow (POP SS, etc). */
+bool int_shadow;
+};
+
+struct qemu_machine {
+struct nvmm_capability cap;
+struct nvmm_machine mach;
+};
+
+/* -- 
*/
+
+static bool nvmm_allowed;
+static struct qemu_machine qemu_mach;
+
+static struct qemu_vcpu *
+get_qemu_vcpu(CPUState *cpu)
+{
+return (struct qemu_vcpu *)cpu->hax_vcpu;
+}
+
+static struct nvmm_machine *
+get_nvmm_mach(void)
+{
+return &qemu_mach.mach;
+}
+
+/* -- 
*/
+
+static void
+nvmm_set_segment(struct nvmm_x64_state_seg *nseg, const SegmentCache *qseg)
+{
+uint32_t attrib = qseg->flags;
+
+nseg->selector = qseg->selector;
+nseg->limit = qseg->limit;
+nseg->base = qseg->base;
+nseg->attrib.type = __SHIFTOUT(attrib, DESC_TYPE_MASK);
+nseg->attrib.s = __SHIFTOUT(attrib, DESC_S_MASK);
+nseg->attrib.dpl = __SHIFTOUT(attrib, DESC_DPL_MASK);
+nseg->attrib.p = __SHIFTOUT(attrib, DESC_P_MASK);
+nseg->attrib.avl = __SHIFTOUT(attrib, DESC_AVL_MASK);
+nseg->attrib.l = __SHIFTOUT(attrib, DESC_L_MASK);
+nseg->attrib.def = __SHIFTOUT(attrib, DESC_B_MASK);
+nseg->attrib.g = __SHIFTOUT(attrib, DESC_G_MASK);
+}
+
+static void
+nvmm_set_registers(CPUState *cpu)
+{
+struct CPUX86State *env = (CPUArchState *)cpu->env_ptr;
+struct nvmm_machine *mach = get_nvmm_mach();
+struct qemu_vcpu *qcpu = get_qemu_vcpu(cpu);
+struct nvmm_vcpu *vcpu = &qcpu->vcpu;
+struct nvmm_x64_state *state = vcpu->state;
+uint64_t bitmap;
+size_t i;
+int ret;
+
+assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
+
+/* GPRs. */
+state->gprs[NVMM_X64_GPR_RAX] = env->regs[R_EAX];
+state->gprs[NVMM_X64_GPR_RCX] = env->regs[R_ECX];
+state->gprs[NVMM_X64_GPR_RDX] = env->regs[R_EDX];
+state->gprs[NVMM_X64_GPR_RBX] = env->regs[R_EBX];
+state->gprs[NVMM_X64_GPR_RSP] = env->regs[R_ESP];
+state->gprs[NVMM_X64_GPR_RBP] = env->regs[R_EBP];
+state->gprs[NVMM_X64_GPR_RSI] = env->regs[R_ESI];
+state->gprs[NVMM_X64_GPR_RDI] = env->regs[R_EDI];
+#ifdef TARGET_X86_64
+state->gprs[NVMM_X64_GPR_R8]  = env->regs[R_R8];
+state->gprs[NVMM_X64_GPR_R9]  = env->regs[R_R9];
+state->gprs[NVMM_X64_GPR_R10] = env->regs[R_R10];
+state->gprs[NVMM_X64_GPR_R11] = env->regs[R_R11];
+state->gprs[NVMM_X64_GPR_R12] = env->regs[R_R12];
+state->gprs[NVMM_X64_GPR_R13] = env->regs[R_R13];
+state->gprs[NVMM_X64_GPR_R14] = env->regs[R_R14];
+state->gp

[PATCH v5 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

From: Maxime Villard 

Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
introduces the nvmm.h sysemu API for managing the vcpu scheduling and
management.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 accel/stubs/Makefile.objs |  1 +
 accel/stubs/nvmm-stub.c   | 43 +++
 include/sysemu/nvmm.h | 35 +++
 3 files changed, 79 insertions(+)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h

diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
index bbd14e71fb..38660a0b9b 100644
--- a/accel/stubs/Makefile.objs
+++ b/accel/stubs/Makefile.objs
@@ -1,6 +1,7 @@
 obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
 obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
 obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
+obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
 obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
 obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
 obj-$(call lnot,$(CONFIG_XEN))  += xen-stub.o
diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
new file mode 100644
index 00..c2208b84a3
--- /dev/null
+++ b/accel/stubs/nvmm-stub.c
@@ -0,0 +1,43 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "sysemu/nvmm.h"
+
+int nvmm_init_vcpu(CPUState *cpu)
+{
+return -1;
+}
+
+int nvmm_vcpu_exec(CPUState *cpu)
+{
+return -1;
+}
+
+void nvmm_destroy_vcpu(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_state(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_init(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
+{
+}
diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
new file mode 100644
index 00..10496f3980
--- /dev/null
+++ b/include/sysemu/nvmm.h
@@ -0,0 +1,35 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_NVMM_H
+#define QEMU_NVMM_H
+
+#include "config-host.h"
+#include "qemu-common.h"
+
+int nvmm_init_vcpu(CPUState *);
+int nvmm_vcpu_exec(CPUState *);
+void nvmm_destroy_vcpu(CPUState *);
+
+void nvmm_cpu_synchronize_state(CPUState *);
+void nvmm_cpu_synchronize_post_reset(CPUState *);
+void nvmm_cpu_synchronize_post_init(CPUState *);
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
+
+#ifdef CONFIG_NVMM
+
+int nvmm_enabled(void);
+
+#else /* CONFIG_NVMM */
+
+#define nvmm_enabled() (0)
+
+#endif /* CONFIG_NVMM */
+
+#endif /* CONFIG_NVMM */
--
2.28.0

Re: [PATCH v4 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

On 10.03.2020 07:45, Maxime Villard wrote:
> Le 02/03/2020 à 20:35, Paolo Bonzini a écrit :
>>
>>
>> Il lun 2 mar 2020, 20:28 Maxime Villard > > ha scritto:
>>
>>
>> >> +        nvmm_vcpu_pre_run(cpu);
>> >> +
>> >> +        if (atomic_read(&cpu->exit_request)) {
>> >> +            qemu_cpu_kick_self();
>> >> +        }
>> >> +
>> >
>> > This is racy without something like KVM's immediate_exit mechanism.
>> > This should be fixed in NVMM.
>>
>> I don't immediately see how this is racy.
>>
>>
>> You can get an IPI signal immediately after reading cpu->exit_request.
>>
>> It reproduces the existing
>> logic found in whpx-all.c, and if there is a real problem it can be
>> fixed in a future commit along with WHPX.
>>
>>
>> It's buggy there too and it has to be fixed in the hypervisor so it can't be 
>> done at the same time I'm both. KVM does it right by having a flag 
>> ("immediate_exit") that is set by the signal handler and checked by the 
>> hypervisor.
>>
>> An earlier version of KVM instead atomically unblocked the signal while 
>> executing the guest, and then ate it with a sigwaitinfo after exiting back 
>> to userspace.
>>
>> You don't have to fix it immediately, but adding a FIXME would be a good 
>> idea.
>>
>> Paolo
> 
> Kamil, please add /* FIXME: possible race here */ before the atomic_read().
> 
> Thanks
> 

So, is this still considered as a possible race? Were there any other
changes to be introduced?

Can we see this patchset merged?

Re: [PATCH 1/1] util/oslib: Returns real thread identifier on FreeBSD and NetBSD

[Kamil Rytarowski]

Reviewed-by: Kamil Rytarowski 

On 26.05.2020 09:29, David CARLIER wrote:
> From 792fbcd9114f43bd80fd1ef5b25cd9935a536f9f Mon Sep 17 00:00:00 2001
> From: David Carlier 
> Date: Tue, 26 May 2020 08:25:26 +0100
> Subject: [PATCH] util/oslib: Returns the real thread identifier on FreeBSD and
>  NetBSD
> 
> getpid is good enough in a mono thread context, however
>  thr_self/_lwp_self reflects the real current thread identifier
>  from a given process.
> ---
>  util/oslib-posix.c | 9 +
>  1 file changed, 9 insertions(+)
> 
> diff --git a/util/oslib-posix.c b/util/oslib-posix.c
> index 062236a1ab..916f1be224 100644
> --- a/util/oslib-posix.c
> +++ b/util/oslib-posix.c
> @@ -48,11 +48,13 @@
>  #ifdef __FreeBSD__
>  #include 
>  #include 
> +#include 
>  #include 
>  #endif
> 
>  #ifdef __NetBSD__
>  #include 
> +#include 
>  #endif
> 
>  #include "qemu/mmap-alloc.h"
> @@ -84,6 +86,13 @@ int qemu_get_thread_id(void)
>  {
>  #if defined(__linux__)
>  return syscall(SYS_gettid);
> +#elif defined(__FreeBSD__)
> +/* thread id is up to INT_MAX */
> +long tid;
> +thr_self(&tid);
> +return (int)tid;
> +#elif defined(__NetBSD__)
> +return _lwp_self();
>  #else
>  return getpid();
>  #endif
> 

Re: [PATCH v4 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

On 11.03.2020 22:21, Maxime Villard wrote:
> Le 11/03/2020 à 21:42, Paolo Bonzini a écrit :
>> On 11/03/20 21:14, Maxime Villard wrote:
 The problem is that qcpu->stop is checked _before_ entering the
 hypervisor and not after, so there is a small race window.
>>> Ok. I don't understand what's supposed to be the race here. If we get an
>>> IPI between the check and the call to nvmm_vcpu_run() then we'll just do
>>> one run and stop in the next iteration, because the IPI will have set
>>> qcpu->stop. Is this extra iteration undesired?
>>
>> Yes, you don't know how long that run would take.  I don't know about
>> NVMM but for KVM it may even never leave if the guest is in HLT state.
>
> Ok, I see, thanks.
>
> In NVMM the runs are short, the syscalls are fast, and pending signals
> cause returns to userland. Therefore, in practice, it's not a big problem,
> because (1) the window is small and (2) if we have a miss it's not going
> to take long to come back to Qemu.
>
> I see a quick kernel change I can make to reduce 95% of the window
> already in the current state. The remaining 5% will need a new
> nvmm_vcpu_kick() function.
>
> For now this issue is unimportant and no Qemu change is required.
>
> Kamil, please also drop the XXX in
> /* XXX Needed, otherwise infinite loop. */
> It's not a bug.
>

OK. I will do it.

> Thanks,
> Maxime
>

Re: [PATCH v4 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

On 10.03.2020 07:45, Maxime Villard wrote:
> Le 02/03/2020 à 20:35, Paolo Bonzini a écrit :
>>
>>
>> Il lun 2 mar 2020, 20:28 Maxime Villard > > ha scritto:
>>
>>
>> >> +        nvmm_vcpu_pre_run(cpu);
>> >> +
>> >> +        if (atomic_read(&cpu->exit_request)) {
>> >> +            qemu_cpu_kick_self();
>> >> +        }
>> >> +
>> >
>> > This is racy without something like KVM's immediate_exit mechanism.
>> > This should be fixed in NVMM.
>>
>> I don't immediately see how this is racy.
>>
>>
>> You can get an IPI signal immediately after reading cpu->exit_request.
>>
>> It reproduces the existing
>> logic found in whpx-all.c, and if there is a real problem it can be
>> fixed in a future commit along with WHPX.
>>
>>
>> It's buggy there too and it has to be fixed in the hypervisor so it can't be 
>> done at the same time I'm both. KVM does it right by having a flag 
>> ("immediate_exit") that is set by the signal handler and checked by the 
>> hypervisor.
>>
>> An earlier version of KVM instead atomically unblocked the signal while 
>> executing the guest, and then ate it with a sigwaitinfo after exiting back 
>> to userspace.
>>
>> You don't have to fix it immediately, but adding a FIXME would be a good 
>> idea.
>>
>> Paolo
>
> Kamil, please add /* FIXME: possible race here */ before the atomic_read().
>
> Thanks
>

I will do it and submit a new patchset revision.

Re: [PATCH v2 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

On 02.03.2020 20:40, Paolo Bonzini wrote:
>
>
> Il lun 2 mar 2020, 20:14 Maxime Villard  <mailto:m...@m00nbsd.net>> ha scritto:
>
> Le 02/03/2020 à 19:05, Kamil Rytarowski a écrit :
> > On 02.03.2020 18:12, Paolo Bonzini wrote:
> >> On 03/02/20 12:56, Kamil Rytarowski wrote:
> >>> On 03.02.2020 12:41, Philippe Mathieu-Daudé wrote:
> >>>>> @@ -1768,6 +1785,7 @@ disabled with --disable-FEATURE, default is
> >>>>> enabled if available:
> >>>>>     hax HAX acceleration support
> >>>>>     hvf Hypervisor.framework acceleration support
> >>>>>     whpx    Windows Hypervisor Platform acceleration
> support
> >>>>> +  nvmm    NetBSD Virtual Machine Monitor acceleration
> support
> >>>>>     rdma    Enable RDMA-based migration
> >>>>>     pvrdma  Enable PVRDMA support
> >>>>>     vde support for vde network
> >>>>> @@ -2757,6 +2775,20 @@ if test "$whpx" != "no" ; then
> >>>>>   fi
> >>>>>   fi
> >>>>>
> >>>>
> >>>> Maybe you can add something like:
> >>>>
> >>>> if test "$targetos" = "NetBSD"; then
> >>>>     nvmm="check"
> >>>> fi
> >>>>
> >>>> to build by default with NVMM if available.
> >>>
> >>> I will add nvmm=yes to the NetBSD) targetos check section.
> >>
> >> No, nvmm=yes instead should fail the build if nvmm.h is not
> available.
> >> That is not a good default.
> >>
> >> Paolo
> >>
> >>
> >
> > Most users will get nvmm.h in place now and this is still a tunable.
> >
> > I have got no opinion what to put there, nvmm=check still works.
>
> I would keep "yes", for consistency with the other entries. Changing all
> entries to "check" should be done in a separate commit, unrelated to
> NVMM.
>
>
> The difference is that KVM for example does not need external includes
> or libraries.
>
> Paolo
>

We don't support this scenario and after a year there might be no
supported release without NVMM.

The only concern is about using qemu on !amd64, but we have there not
many users of qemu for understandable reasons.

For AArch64 we plan to implement a dedicated NVMM backend.

>
> > diff --git a/configure b/configure
> > index d4a837cf9d..b3560d88bb 100755
> > --- a/configure
> > +++ b/configure
> > @@ -836,7 +836,7 @@ DragonFly)
> >  NetBSD)
> >    bsd="yes"
> >    hax="yes"
> > -  nvmm="yes"
> > +  nvmm="check"
> >    make="${MAKE-gmake}"
> >    audio_drv_list="oss try-sdl"
> >    audio_possible_drivers="oss sdl"
> >
>

Re: [PATCH v2 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

On 02.03.2020 18:11, Paolo Bonzini wrote:
> On 03/02/20 12:41, Philippe Mathieu-Daudé wrote:
>>
>> Maybe you can add something like:
>>
>> if test "$targetos" = "NetBSD"; then
>>     nvmm="check"
>> fi
>
> You could do just nvmm="" and, below,
>
> if test "$nvmm" != "no" && test "$targetos" = "NetBSD"
>
> But maybe even testing NetBSD is not needed since nvmm.h will likely not
> be there.
>
> Paolo
>

I have got no opinion here.

I can just change on request nvmm="yes" to nvmm="check" and be done.

>> to build by default with NVMM if available.
>>
>>> +##
>>> +# NetBSD Virtual Machine Monitor (NVMM) accelerator check
>>> +if test "$nvmm" != "no" ; then
>>> +    if check_include "nvmm.h" ; then
>>> +    nvmm="yes"
>>> +    LIBS="-lnvmm $LIBS"
>>> +    else
>>> +    if test "$nvmm" = "yes"; then
>>> +    feature_not_found "NVMM" "NVMM is not available"
>>> +    fi
>>> +    nvmm="no"
>>> +    fi
>>> +fi
>

Re: [PATCH v2 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

On 02.03.2020 18:12, Paolo Bonzini wrote:
> On 03/02/20 12:56, Kamil Rytarowski wrote:
>> On 03.02.2020 12:41, Philippe Mathieu-Daudé wrote:
>>>> @@ -1768,6 +1785,7 @@ disabled with --disable-FEATURE, default is
>>>> enabled if available:
>>>>     hax HAX acceleration support
>>>>     hvf Hypervisor.framework acceleration support
>>>>     whpx    Windows Hypervisor Platform acceleration support
>>>> +  nvmm    NetBSD Virtual Machine Monitor acceleration support
>>>>     rdma    Enable RDMA-based migration
>>>>     pvrdma  Enable PVRDMA support
>>>>     vde support for vde network
>>>> @@ -2757,6 +2775,20 @@ if test "$whpx" != "no" ; then
>>>>   fi
>>>>   fi
>>>>
>>>
>>> Maybe you can add something like:
>>>
>>> if test "$targetos" = "NetBSD"; then
>>>     nvmm="check"
>>> fi
>>>
>>> to build by default with NVMM if available.
>>
>> I will add nvmm=yes to the NetBSD) targetos check section.
>
> No, nvmm=yes instead should fail the build if nvmm.h is not available.
> That is not a good default.
>
> Paolo
>
>

Most users will get nvmm.h in place now and this is still a tunable.

I have got no opinion what to put there, nvmm=check still works.

diff --git a/configure b/configure
index d4a837cf9d..b3560d88bb 100755
--- a/configure
+++ b/configure
@@ -836,7 +836,7 @@ DragonFly)
 NetBSD)
   bsd="yes"
   hax="yes"
-  nvmm="yes"
+  nvmm="check"
   make="${MAKE-gmake}"
   audio_drv_list="oss try-sdl"
   audio_possible_drivers="oss sdl"

Re: [PATCH v4 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

On 02.03.2020 18:10, Eduardo Habkost wrote:
> Hi Kamil, Maxime,
>
> I haven't managed to reserve time to review this, sorry for that.
> I hope others can chime in before I do.
>
> Would any of you be willing to be included as maintainer of the
> new code on MAINTAINERS?
>

I'm already mentioned as the NetBSD maintainer and NVMM is NetBSD-only
(at least today).

>
> On Mon, Mar 02, 2020 at 06:02:18PM +0100, Kamil Rytarowski wrote:
>> Ping?
>>
>> On 24.02.2020 16:17, Kamil Rytarowski wrote:
>>> Ping?
>>>
>>> On 17.02.2020 10:07, Kamil Rytarowski wrote:
>>>> Ping?
>>>>
>>>> On 06.02.2020 22:32, Kamil Rytarowski wrote:
>>>>> Hello QEMU Community!
>>>>>
>>>>> Over the past year the NetBSD team has been working hard on a new 
>>>>> user-mode API
>>>>> for our hypervisor that will be released as part of the upcoming NetBSD 
>>>>> 9.0.
>>>>> This new API adds user-mode capabilities to create and manage virtual 
>>>>> machines,
>>>>> configure memory mappings for guest machines, and create and control 
>>>>> execution
>>>>> of virtual processors.
>>>>>
>>>>> With this new API we are now able to bring our hypervisor to the QEMU
>>>>> community! The following patches implement the NetBSD Virtual Machine 
>>>>> Monitor
>>>>> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
>>>>>
>>>>> When compiling QEMU for x86_64 passing the --enable-nvmm flag will 
>>>>> compile the
>>>>> accelerator for use. At runtime using the '-accel nvmm' should see a
>>>>> significant performance improvement over emulation, much like when using 
>>>>> 'hax'
>>>>> on NetBSD.
>>>>>
>>>>> The documentation for this new API is visible at https://man.netbsd.org 
>>>>> under
>>>>> the libnvmm(3) and nvmm(4) pages.
>>>>>
>>>>> NVMM was designed and implemented by Maxime Villard.
>>>>>
>>>>> Thank you for your feedback.
>>>>>
>>>>> Refrences:
>>>>> https://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html
>>>>>
>>>>> Test plan:
>>>>>
>>>>> 1. Download a NetBSD 9.0 pre-release snapshot:
>>>>> http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/images/NetBSD-9.0_RC1-amd64.iso
>>>>>
>>>>> 2. Install it natively on a not too old x86_64 hardware (Intel or AMD).
>>>>>
>>>>> There is no support for nested virtualization in NVMM.
>>>>>
>>>>> 3. Setup the system.
>>>>>
>>>>>  export 
>>>>> PKG_PATH=http://www.ki.nu/pkgsrc/packages/current/NetBSD-9.0_RC1/All
>>>>>  pkg_add git gmake python37 glib2 bison pkgconf pixman
>>>>>
>>>>> Install mozilla-rootcerts and follow post-install instructions.
>>>>>
>>>>>  pkg_add mozilla-rootcerts
>>>>>
>>>>> More information: https://wiki.qemu.org/Hosts/BSD#NetBSD
>>>>>
>>>>> 4. Build qemu
>>>>>
>>>>>  mkdir build
>>>>>  cd build
>>>>>  ../configure --python=python3.7
>>>>>  gmake
>>>>>  gmake check
>>>>>
>>>>> 5. Test
>>>>>
>>>>>  qemu -accel nvmm ...
>>>>>
>>>>>
>>>>> History:
>>>>> v3 -> v4:
>>>>>  - Correct build warning by adding a missing include
>>>>>  - Do not set R8-R16 registers unless TARGET_X86_64
>>>>> v2 -> v3:
>>>>>  - Register nvmm in targetos NetBSD check
>>>>>  - Stop including hw/boards.h
>>>>>  - Rephrase old code comments (remove XXX)
>>>>> v1 -> v2:
>>>>>  - Included the testing plan as requested by Philippe Mathieu-Daude
>>>>>  - Formatting nit fix in qemu-options.hx
>>>>>  - Document NVMM in the accel section of qemu-options.hx
>>>>>
>>>>> Maxime Villard (4):
>>>>>   Add the NVMM vcpu API
>>>>>   Add the NetBSD Virtual Machine Monitor accelerator.
>>>>>   Introduce the NVMM impl
>>>>>   Add the NVMM acceleration enlightenments
>>>>>
>>>>>  accel/stubs/Makefile.objs |1 +
>>>>>  accel/stubs/nvmm-stub.c   |   43 ++
>>>>>  configure |   37 ++
>>>>>  cpus.c|   58 ++
>>>>>  include/sysemu/hw_accel.h |   14 +
>>>>>  include/sysemu/nvmm.h |   35 ++
>>>>>  qemu-options.hx   |   16 +-
>>>>>  target/i386/Makefile.objs |1 +
>>>>>  target/i386/helper.c  |2 +-
>>>>>  target/i386/nvmm-all.c| 1226 +
>>>>>  10 files changed, 1424 insertions(+), 9 deletions(-)
>>>>>  create mode 100644 accel/stubs/nvmm-stub.c
>>>>>  create mode 100644 include/sysemu/nvmm.h
>>>>>  create mode 100644 target/i386/nvmm-all.c
>>>>>
>>>>> --
>>>>> 2.25.0
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: [PATCH v4 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Ping?

On 24.02.2020 16:17, Kamil Rytarowski wrote:
> Ping?
>
> On 17.02.2020 10:07, Kamil Rytarowski wrote:
>> Ping?
>>
>> On 06.02.2020 22:32, Kamil Rytarowski wrote:
>>> Hello QEMU Community!
>>>
>>> Over the past year the NetBSD team has been working hard on a new user-mode 
>>> API
>>> for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
>>> This new API adds user-mode capabilities to create and manage virtual 
>>> machines,
>>> configure memory mappings for guest machines, and create and control 
>>> execution
>>> of virtual processors.
>>>
>>> With this new API we are now able to bring our hypervisor to the QEMU
>>> community! The following patches implement the NetBSD Virtual Machine 
>>> Monitor
>>> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
>>>
>>> When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile 
>>> the
>>> accelerator for use. At runtime using the '-accel nvmm' should see a
>>> significant performance improvement over emulation, much like when using 
>>> 'hax'
>>> on NetBSD.
>>>
>>> The documentation for this new API is visible at https://man.netbsd.org 
>>> under
>>> the libnvmm(3) and nvmm(4) pages.
>>>
>>> NVMM was designed and implemented by Maxime Villard.
>>>
>>> Thank you for your feedback.
>>>
>>> Refrences:
>>> https://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html
>>>
>>> Test plan:
>>>
>>> 1. Download a NetBSD 9.0 pre-release snapshot:
>>> http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/images/NetBSD-9.0_RC1-amd64.iso
>>>
>>> 2. Install it natively on a not too old x86_64 hardware (Intel or AMD).
>>>
>>> There is no support for nested virtualization in NVMM.
>>>
>>> 3. Setup the system.
>>>
>>>  export PKG_PATH=http://www.ki.nu/pkgsrc/packages/current/NetBSD-9.0_RC1/All
>>>  pkg_add git gmake python37 glib2 bison pkgconf pixman
>>>
>>> Install mozilla-rootcerts and follow post-install instructions.
>>>
>>>  pkg_add mozilla-rootcerts
>>>
>>> More information: https://wiki.qemu.org/Hosts/BSD#NetBSD
>>>
>>> 4. Build qemu
>>>
>>>  mkdir build
>>>  cd build
>>>  ../configure --python=python3.7
>>>  gmake
>>>  gmake check
>>>
>>> 5. Test
>>>
>>>  qemu -accel nvmm ...
>>>
>>>
>>> History:
>>> v3 -> v4:
>>>  - Correct build warning by adding a missing include
>>>  - Do not set R8-R16 registers unless TARGET_X86_64
>>> v2 -> v3:
>>>  - Register nvmm in targetos NetBSD check
>>>  - Stop including hw/boards.h
>>>  - Rephrase old code comments (remove XXX)
>>> v1 -> v2:
>>>  - Included the testing plan as requested by Philippe Mathieu-Daude
>>>  - Formatting nit fix in qemu-options.hx
>>>  - Document NVMM in the accel section of qemu-options.hx
>>>
>>> Maxime Villard (4):
>>>   Add the NVMM vcpu API
>>>   Add the NetBSD Virtual Machine Monitor accelerator.
>>>   Introduce the NVMM impl
>>>   Add the NVMM acceleration enlightenments
>>>
>>>  accel/stubs/Makefile.objs |1 +
>>>  accel/stubs/nvmm-stub.c   |   43 ++
>>>  configure |   37 ++
>>>  cpus.c|   58 ++
>>>  include/sysemu/hw_accel.h |   14 +
>>>  include/sysemu/nvmm.h |   35 ++
>>>  qemu-options.hx   |   16 +-
>>>  target/i386/Makefile.objs |1 +
>>>  target/i386/helper.c  |2 +-
>>>  target/i386/nvmm-all.c| 1226 +
>>>  10 files changed, 1424 insertions(+), 9 deletions(-)
>>>  create mode 100644 accel/stubs/nvmm-stub.c
>>>  create mode 100644 include/sysemu/nvmm.h
>>>  create mode 100644 target/i386/nvmm-all.c
>>>
>>> --
>>> 2.25.0
>>>
>>>
>>
>

Re: [PATCH v4 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Ping?

On 17.02.2020 10:07, Kamil Rytarowski wrote:
> Ping?
>
> On 06.02.2020 22:32, Kamil Rytarowski wrote:
>> Hello QEMU Community!
>>
>> Over the past year the NetBSD team has been working hard on a new user-mode 
>> API
>> for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
>> This new API adds user-mode capabilities to create and manage virtual 
>> machines,
>> configure memory mappings for guest machines, and create and control 
>> execution
>> of virtual processors.
>>
>> With this new API we are now able to bring our hypervisor to the QEMU
>> community! The following patches implement the NetBSD Virtual Machine Monitor
>> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
>>
>> When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile 
>> the
>> accelerator for use. At runtime using the '-accel nvmm' should see a
>> significant performance improvement over emulation, much like when using 
>> 'hax'
>> on NetBSD.
>>
>> The documentation for this new API is visible at https://man.netbsd.org under
>> the libnvmm(3) and nvmm(4) pages.
>>
>> NVMM was designed and implemented by Maxime Villard.
>>
>> Thank you for your feedback.
>>
>> Refrences:
>> https://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html
>>
>> Test plan:
>>
>> 1. Download a NetBSD 9.0 pre-release snapshot:
>> http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/images/NetBSD-9.0_RC1-amd64.iso
>>
>> 2. Install it natively on a not too old x86_64 hardware (Intel or AMD).
>>
>> There is no support for nested virtualization in NVMM.
>>
>> 3. Setup the system.
>>
>>  export PKG_PATH=http://www.ki.nu/pkgsrc/packages/current/NetBSD-9.0_RC1/All
>>  pkg_add git gmake python37 glib2 bison pkgconf pixman
>>
>> Install mozilla-rootcerts and follow post-install instructions.
>>
>>  pkg_add mozilla-rootcerts
>>
>> More information: https://wiki.qemu.org/Hosts/BSD#NetBSD
>>
>> 4. Build qemu
>>
>>  mkdir build
>>  cd build
>>  ../configure --python=python3.7
>>  gmake
>>  gmake check
>>
>> 5. Test
>>
>>  qemu -accel nvmm ...
>>
>>
>> History:
>> v3 -> v4:
>>  - Correct build warning by adding a missing include
>>  - Do not set R8-R16 registers unless TARGET_X86_64
>> v2 -> v3:
>>  - Register nvmm in targetos NetBSD check
>>  - Stop including hw/boards.h
>>  - Rephrase old code comments (remove XXX)
>> v1 -> v2:
>>  - Included the testing plan as requested by Philippe Mathieu-Daude
>>  - Formatting nit fix in qemu-options.hx
>>  - Document NVMM in the accel section of qemu-options.hx
>>
>> Maxime Villard (4):
>>   Add the NVMM vcpu API
>>   Add the NetBSD Virtual Machine Monitor accelerator.
>>   Introduce the NVMM impl
>>   Add the NVMM acceleration enlightenments
>>
>>  accel/stubs/Makefile.objs |1 +
>>  accel/stubs/nvmm-stub.c   |   43 ++
>>  configure |   37 ++
>>  cpus.c|   58 ++
>>  include/sysemu/hw_accel.h |   14 +
>>  include/sysemu/nvmm.h |   35 ++
>>  qemu-options.hx   |   16 +-
>>  target/i386/Makefile.objs |1 +
>>  target/i386/helper.c  |2 +-
>>  target/i386/nvmm-all.c| 1226 +
>>  10 files changed, 1424 insertions(+), 9 deletions(-)
>>  create mode 100644 accel/stubs/nvmm-stub.c
>>  create mode 100644 include/sysemu/nvmm.h
>>  create mode 100644 target/i386/nvmm-all.c
>>
>> --
>> 2.25.0
>>
>>
>

Re: [PATCH v4 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Ping?

On 06.02.2020 22:32, Kamil Rytarowski wrote:
> Hello QEMU Community!
>
> Over the past year the NetBSD team has been working hard on a new user-mode 
> API
> for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
> This new API adds user-mode capabilities to create and manage virtual 
> machines,
> configure memory mappings for guest machines, and create and control execution
> of virtual processors.
>
> With this new API we are now able to bring our hypervisor to the QEMU
> community! The following patches implement the NetBSD Virtual Machine Monitor
> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
>
> When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile the
> accelerator for use. At runtime using the '-accel nvmm' should see a
> significant performance improvement over emulation, much like when using 'hax'
> on NetBSD.
>
> The documentation for this new API is visible at https://man.netbsd.org under
> the libnvmm(3) and nvmm(4) pages.
>
> NVMM was designed and implemented by Maxime Villard.
>
> Thank you for your feedback.
>
> Refrences:
> https://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html
>
> Test plan:
>
> 1. Download a NetBSD 9.0 pre-release snapshot:
> http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/images/NetBSD-9.0_RC1-amd64.iso
>
> 2. Install it natively on a not too old x86_64 hardware (Intel or AMD).
>
> There is no support for nested virtualization in NVMM.
>
> 3. Setup the system.
>
>  export PKG_PATH=http://www.ki.nu/pkgsrc/packages/current/NetBSD-9.0_RC1/All
>  pkg_add git gmake python37 glib2 bison pkgconf pixman
>
> Install mozilla-rootcerts and follow post-install instructions.
>
>  pkg_add mozilla-rootcerts
>
> More information: https://wiki.qemu.org/Hosts/BSD#NetBSD
>
> 4. Build qemu
>
>  mkdir build
>  cd build
>  ../configure --python=python3.7
>  gmake
>  gmake check
>
> 5. Test
>
>  qemu -accel nvmm ...
>
>
> History:
> v3 -> v4:
>  - Correct build warning by adding a missing include
>  - Do not set R8-R16 registers unless TARGET_X86_64
> v2 -> v3:
>  - Register nvmm in targetos NetBSD check
>  - Stop including hw/boards.h
>  - Rephrase old code comments (remove XXX)
> v1 -> v2:
>  - Included the testing plan as requested by Philippe Mathieu-Daude
>  - Formatting nit fix in qemu-options.hx
>  - Document NVMM in the accel section of qemu-options.hx
>
> Maxime Villard (4):
>   Add the NVMM vcpu API
>   Add the NetBSD Virtual Machine Monitor accelerator.
>   Introduce the NVMM impl
>   Add the NVMM acceleration enlightenments
>
>  accel/stubs/Makefile.objs |1 +
>  accel/stubs/nvmm-stub.c   |   43 ++
>  configure |   37 ++
>  cpus.c|   58 ++
>  include/sysemu/hw_accel.h |   14 +
>  include/sysemu/nvmm.h |   35 ++
>  qemu-options.hx   |   16 +-
>  target/i386/Makefile.objs |1 +
>  target/i386/helper.c  |2 +-
>  target/i386/nvmm-all.c| 1226 +
>  10 files changed, 1424 insertions(+), 9 deletions(-)
>  create mode 100644 accel/stubs/nvmm-stub.c
>  create mode 100644 include/sysemu/nvmm.h
>  create mode 100644 target/i386/nvmm-all.c
>
> --
> 2.25.0
>
>

Re: [PATCH] tests/acceptance/ppc_prep_40p: Do not run NetBSD test by default

[Kamil Rytarowski]

Please use cdn.netbsd.org always.

On 11.02.2020 14:19, Philippe Mathieu-Daudé wrote:
> The ftp.netbsd.org server is slow and downloading the NetBSD ISO
> takes too long. Do not include this test in the default suite.
> 
> Similarly to commit 471c97a69b:
> 
>   Currently the Avocado framework does not distinct the time spent
>   downloading assets vs. the time spent running a test. With big
>   assets (like a full VM image) the tests likely fail.
> 
>   This is a limitation known by the Avocado team.
>   Until this issue get fixed, do not run this tests automatically.
> 
>   Tests can still be run setting the AVOCADO_TIMEOUT_EXPECTED
>   environment variable.
> 
> Reported-by: Alex Bennée 
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  tests/acceptance/ppc_prep_40p.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tests/acceptance/ppc_prep_40p.py 
> b/tests/acceptance/ppc_prep_40p.py
> index b27572f212..efe06037ba 100644
> --- a/tests/acceptance/ppc_prep_40p.py
> +++ b/tests/acceptance/ppc_prep_40p.py
> @@ -61,6 +61,7 @@ def test_openbios_192m(self):
>  wait_for_console_pattern(self, '>> CPU type PowerPC,604')
>  
>  @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
> +@skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>  def test_openbios_and_netbsd(self):
>  """
>  :avocado: tags=arch:ppc
> 




signature.asc
Description: OpenPGP digital signature

[PATCH v4 3/4 FIXUP] Introduce the NVMM impl

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NetBSD Virtual Machine Monitor (NVMM) target. Which
acts as a hypervisor accelerator for QEMU on the NetBSD platform. This enables
QEMU much greater speed over the emulated x86_64 path's that are taken on
NetBSD today.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Tested-by: Jared McNeill 
---
 target/i386/Makefile.objs |1 +
 target/i386/nvmm-all.c| 1226 +
 2 files changed, 1227 insertions(+)
 create mode 100644 target/i386/nvmm-all.c

diff --git a/target/i386/Makefile.objs b/target/i386/Makefile.objs
index 48e0c28434..bdcdb32e93 100644
--- a/target/i386/Makefile.objs
+++ b/target/i386/Makefile.objs
@@ -17,6 +17,7 @@ obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-posix.o
 endif
 obj-$(CONFIG_HVF) += hvf/
 obj-$(CONFIG_WHPX) += whpx-all.o
+obj-$(CONFIG_NVMM) += nvmm-all.o
 endif
 obj-$(CONFIG_SEV) += sev.o
 obj-$(call lnot,$(CONFIG_SEV)) += sev-stub.o
diff --git a/target/i386/nvmm-all.c b/target/i386/nvmm-all.c
new file mode 100644
index 00..b3f1c11984
--- /dev/null
+++ b/target/i386/nvmm-all.c
@@ -0,0 +1,1226 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator for QEMU.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/address-spaces.h"
+#include "exec/ioport.h"
+#include "qemu-common.h"
+#include "strings.h"
+#include "sysemu/accel.h"
+#include "sysemu/nvmm.h"
+#include "sysemu/runstate.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+
+#include 
+
+struct qemu_vcpu {
+struct nvmm_vcpu vcpu;
+uint8_t tpr;
+bool stop;
+
+/* Window-exiting for INTs/NMIs. */
+bool int_window_exit;
+bool nmi_window_exit;
+
+/* The guest is in an interrupt shadow (POP SS, etc). */
+bool int_shadow;
+};
+
+struct qemu_machine {
+struct nvmm_capability cap;
+struct nvmm_machine mach;
+};
+
+/* -- 
*/
+
+static bool nvmm_allowed;
+static struct qemu_machine qemu_mach;
+
+static struct qemu_vcpu *
+get_qemu_vcpu(CPUState *cpu)
+{
+return (struct qemu_vcpu *)cpu->hax_vcpu;
+}
+
+static struct nvmm_machine *
+get_nvmm_mach(void)
+{
+return &qemu_mach.mach;
+}
+
+/* -- 
*/
+
+static void
+nvmm_set_segment(struct nvmm_x64_state_seg *nseg, const SegmentCache *qseg)
+{
+uint32_t attrib = qseg->flags;
+
+nseg->selector = qseg->selector;
+nseg->limit = qseg->limit;
+nseg->base = qseg->base;
+nseg->attrib.type = __SHIFTOUT(attrib, DESC_TYPE_MASK);
+nseg->attrib.s = __SHIFTOUT(attrib, DESC_S_MASK);
+nseg->attrib.dpl = __SHIFTOUT(attrib, DESC_DPL_MASK);
+nseg->attrib.p = __SHIFTOUT(attrib, DESC_P_MASK);
+nseg->attrib.avl = __SHIFTOUT(attrib, DESC_AVL_MASK);
+nseg->attrib.l = __SHIFTOUT(attrib, DESC_L_MASK);
+nseg->attrib.def = __SHIFTOUT(attrib, DESC_B_MASK);
+nseg->attrib.g = __SHIFTOUT(attrib, DESC_G_MASK);
+}
+
+static void
+nvmm_set_registers(CPUState *cpu)
+{
+struct CPUX86State *env = (CPUArchState *)cpu->env_ptr;
+struct nvmm_machine *mach = get_nvmm_mach();
+struct qemu_vcpu *qcpu = get_qemu_vcpu(cpu);
+struct nvmm_vcpu *vcpu = &qcpu->vcpu;
+struct nvmm_x64_state *state = vcpu->state;
+uint64_t bitmap;
+size_t i;
+int ret;
+
+assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
+
+/* GPRs. */
+state->gprs[NVMM_X64_GPR_RAX] = env->regs[R_EAX];
+state->gprs[NVMM_X64_GPR_RCX] = env->regs[R_ECX];
+state->gprs[NVMM_X64_GPR_RDX] = env->regs[R_EDX];
+state->gprs[NVMM_X64_GPR_RBX] = env->regs[R_EBX];
+state->gprs[NVMM_X64_GPR_RSP] = env->regs[R_ESP];
+state->gprs[NVMM_X64_GPR_RBP] = env->regs[R_EBP];
+state->gprs[NVMM_X64_GPR_RSI] = env->regs[R_ESI];
+state->gprs[NVMM_X64_GPR_RDI] = env->regs[R_EDI];
+#ifdef TARGET_X86_64
+state->gprs[NVMM_X64_GPR_R8]  = env->regs[R_R8];
+state->gprs[NVMM_X64_GPR_R9]  = env->regs[R_R9];
+state->gprs[NVMM_X64_GPR_R10] = env->regs[R_R10];
+state->gprs[NVMM_X64_GPR_R11] = env->regs[R_R11];
+state->gprs[NVMM_X64_GPR_R12] = env->regs[R_R12];
+state->gprs[NVMM_X64_GPR_R13] = env->regs[R_R13];
+state->gprs[NVMM_X64_GPR_R14] = env->regs[R_R14];
+state->gp

[PATCH v4 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NetBSD Virtual Machine Monitor (NVMM) target. Which
acts as a hypervisor accelerator for QEMU on the NetBSD platform. This enables
QEMU much greater speed over the emulated x86_64 path's that are taken on
NetBSD today.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Tested-by: Jared McNeill 
---
 target/i386/Makefile.objs |1 +
 target/i386/nvmm-all.c| 1226 +
 2 files changed, 1227 insertions(+)
 create mode 100644 target/i386/nvmm-all.c

diff --git a/target/i386/Makefile.objs b/target/i386/Makefile.objs
index 48e0c28434..bdcdb32e93 100644
--- a/target/i386/Makefile.objs
+++ b/target/i386/Makefile.objs
@@ -17,6 +17,7 @@ obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-posix.o
 endif
 obj-$(CONFIG_HVF) += hvf/
 obj-$(CONFIG_WHPX) += whpx-all.o
+obj-$(CONFIG_NVMM) += nvmm-all.o
 endif
 obj-$(CONFIG_SEV) += sev.o
 obj-$(call lnot,$(CONFIG_SEV)) += sev-stub.o
diff --git a/target/i386/nvmm-all.c b/target/i386/nvmm-all.c
new file mode 100644
index 00..a21908f46a
--- /dev/null
+++ b/target/i386/nvmm-all.c
@@ -0,0 +1,1226 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator for QEMU.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/address-spaces.h"
+#include "exec/ioport.h"
+#include "qemu-common.h"
+#include "strings.h"
+#include "sysemu/accel.h"
+#include "sysemu/nvmm.h"
+#include "sysemu/runstate.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+
+#include 
+
+struct qemu_vcpu {
+struct nvmm_vcpu vcpu;
+uint8_t tpr;
+bool stop;
+
+/* Window-exiting for INTs/NMIs. */
+bool int_window_exit;
+bool nmi_window_exit;
+
+/* The guest is in an interrupt shadow (POP SS, etc). */
+bool int_shadow;
+};
+
+struct qemu_machine {
+struct nvmm_capability cap;
+struct nvmm_machine mach;
+};
+
+/* -- 
*/
+
+static bool nvmm_allowed;
+static struct qemu_machine qemu_mach;
+
+static struct qemu_vcpu *
+get_qemu_vcpu(CPUState *cpu)
+{
+return (struct qemu_vcpu *)cpu->hax_vcpu;
+}
+
+static struct nvmm_machine *
+get_nvmm_mach(void)
+{
+return &qemu_mach.mach;
+}
+
+/* -- 
*/
+
+static void
+nvmm_set_segment(struct nvmm_x64_state_seg *nseg, const SegmentCache *qseg)
+{
+uint32_t attrib = qseg->flags;
+
+nseg->selector = qseg->selector;
+nseg->limit = qseg->limit;
+nseg->base = qseg->base;
+nseg->attrib.type = __SHIFTOUT(attrib, DESC_TYPE_MASK);
+nseg->attrib.s = __SHIFTOUT(attrib, DESC_S_MASK);
+nseg->attrib.dpl = __SHIFTOUT(attrib, DESC_DPL_MASK);
+nseg->attrib.p = __SHIFTOUT(attrib, DESC_P_MASK);
+nseg->attrib.avl = __SHIFTOUT(attrib, DESC_AVL_MASK);
+nseg->attrib.l = __SHIFTOUT(attrib, DESC_L_MASK);
+nseg->attrib.def = __SHIFTOUT(attrib, DESC_B_MASK);
+nseg->attrib.g = __SHIFTOUT(attrib, DESC_G_MASK);
+}
+
+static void
+nvmm_set_registers(CPUState *cpu)
+{
+struct CPUX86State *env = (CPUArchState *)cpu->env_ptr;
+struct nvmm_machine *mach = get_nvmm_mach();
+struct qemu_vcpu *qcpu = get_qemu_vcpu(cpu);
+struct nvmm_vcpu *vcpu = &qcpu->vcpu;
+struct nvmm_x64_state *state = vcpu->state;
+uint64_t bitmap;
+size_t i;
+int ret;
+
+assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
+
+/* GPRs. */
+state->gprs[NVMM_X64_GPR_RAX] = env->regs[R_EAX];
+state->gprs[NVMM_X64_GPR_RCX] = env->regs[R_ECX];
+state->gprs[NVMM_X64_GPR_RDX] = env->regs[R_EDX];
+state->gprs[NVMM_X64_GPR_RBX] = env->regs[R_EBX];
+state->gprs[NVMM_X64_GPR_RSP] = env->regs[R_ESP];
+state->gprs[NVMM_X64_GPR_RBP] = env->regs[R_EBP];
+state->gprs[NVMM_X64_GPR_RSI] = env->regs[R_ESI];
+state->gprs[NVMM_X64_GPR_RDI] = env->regs[R_EDI];
+#ifdef TARGET_X86_64
+state->gprs[NVMM_X64_GPR_R8]  = env->regs[R_R8];
+state->gprs[NVMM_X64_GPR_R9]  = env->regs[R_R9];
+state->gprs[NVMM_X64_GPR_R10] = env->regs[R_R10];
+state->gprs[NVMM_X64_GPR_R11] = env->regs[R_R11];
+state->gprs[NVMM_X64_GPR_R12] = env->regs[R_R12];
+state->gprs[NVMM_X64_GPR_R13] = env->regs[R_R13];
+state->gprs[NVMM_X64_GPR_R14] = env->regs[R_R14];
+state->gp

[PATCH v4 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NVMM accelerator cpu enlightenments to actually use the nvmm-all
accelerator on NetBSD platforms.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 cpus.c| 58 +++
 include/sysemu/hw_accel.h | 14 ++
 target/i386/helper.c  |  2 +-
 3 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index b4f8b84b61..f833da4a60 100644
--- a/cpus.c
+++ b/cpus.c
@@ -42,6 +42,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"
 #include "exec/exec-all.h"

 #include "qemu/thread.h"
@@ -1670,6 +1671,48 @@ static void *qemu_whpx_cpu_thread_fn(void *arg)
 return NULL;
 }

+static void *qemu_nvmm_cpu_thread_fn(void *arg)
+{
+CPUState *cpu = arg;
+int r;
+
+assert(nvmm_enabled());
+
+rcu_register_thread();
+
+qemu_mutex_lock_iothread();
+qemu_thread_get_self(cpu->thread);
+cpu->thread_id = qemu_get_thread_id();
+current_cpu = cpu;
+
+r = nvmm_init_vcpu(cpu);
+if (r < 0) {
+fprintf(stderr, "nvmm_init_vcpu failed: %s\n", strerror(-r));
+exit(1);
+}
+
+/* signal CPU creation */
+cpu->created = true;
+qemu_cond_signal(&qemu_cpu_cond);
+
+do {
+if (cpu_can_run(cpu)) {
+r = nvmm_vcpu_exec(cpu);
+if (r == EXCP_DEBUG) {
+cpu_handle_guest_debug(cpu);
+}
+}
+qemu_wait_io_event(cpu);
+} while (!cpu->unplug || cpu_can_run(cpu));
+
+nvmm_destroy_vcpu(cpu);
+cpu->created = false;
+qemu_cond_signal(&qemu_cpu_cond);
+qemu_mutex_unlock_iothread();
+rcu_unregister_thread();
+return NULL;
+}
+
 #ifdef _WIN32
 static void CALLBACK dummy_apc_func(ULONG_PTR unused)
 {
@@ -2038,6 +2081,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
 #endif
 }

+static void qemu_nvmm_start_vcpu(CPUState *cpu)
+{
+char thread_name[VCPU_THREAD_NAME_SIZE];
+
+cpu->thread = g_malloc0(sizeof(QemuThread));
+cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+qemu_cond_init(cpu->halt_cond);
+snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/NVMM",
+ cpu->cpu_index);
+qemu_thread_create(cpu->thread, thread_name, qemu_nvmm_cpu_thread_fn,
+   cpu, QEMU_THREAD_JOINABLE);
+}
+
 static void qemu_dummy_start_vcpu(CPUState *cpu)
 {
 char thread_name[VCPU_THREAD_NAME_SIZE];
@@ -2078,6 +2134,8 @@ void qemu_init_vcpu(CPUState *cpu)
 qemu_tcg_init_vcpu(cpu);
 } else if (whpx_enabled()) {
 qemu_whpx_start_vcpu(cpu);
+} else if (nvmm_enabled()) {
+qemu_nvmm_start_vcpu(cpu);
 } else {
 qemu_dummy_start_vcpu(cpu);
 }
diff --git a/include/sysemu/hw_accel.h b/include/sysemu/hw_accel.h
index 0ec2372477..dbfa7a02f9 100644
--- a/include/sysemu/hw_accel.h
+++ b/include/sysemu/hw_accel.h
@@ -15,6 +15,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/kvm.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"

 static inline void cpu_synchronize_state(CPUState *cpu)
 {
@@ -27,6 +28,9 @@ static inline void cpu_synchronize_state(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_state(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_state(cpu);
+}
 }

 static inline void cpu_synchronize_post_reset(CPUState *cpu)
@@ -40,6 +44,10 @@ static inline void cpu_synchronize_post_reset(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_reset(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_reset(cpu);
+}
+
 }

 static inline void cpu_synchronize_post_init(CPUState *cpu)
@@ -53,6 +61,9 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_init(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_init(cpu);
+}
 }

 static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
@@ -66,6 +77,9 @@ static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_pre_loadvm(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_pre_loadvm(cpu);
+}
 }

 #endif /* QEMU_HW_ACCEL_H */
diff --git a/target/i386/helper.c b/target/i386/helper.c
index c3a6e4fabe..2e79d61329 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -981,7 +981,7 @@ void cpu_report_tpr_access(CPUX86State *env, TPRAccess 
access)
 X86CPU *cpu = env_archcpu(env);
 CPUState *cs = env_cpu(env);

-if (kvm_enabled() || whpx_enabled()) {
+if (kvm_enabled() || whpx_enabled() || nvmm_enabled()) {
 env->tpr_access_type = access;

 cpu_interrupt(cs, CPU_INTERRUPT_TPR);
--
2.25.0

[PATCH v4 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

From: Maxime Villard 

Introduces the configure support for the new NetBSD Virtual Machine Monitor that
allows for hypervisor acceleration from usermode components on the NetBSD
platform.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 configure   | 37 +
 qemu-options.hx | 16 
 2 files changed, 45 insertions(+), 8 deletions(-)

diff --git a/configure b/configure
index 115dc38085..d4a837cf9d 100755
--- a/configure
+++ b/configure
@@ -241,6 +241,17 @@ supported_whpx_target() {
 return 1
 }

+supported_nvmm_target() {
+test "$nvmm" = "yes" || return 1
+glob "$1" "*-softmmu" || return 1
+case "${1%-softmmu}" in
+i386|x86_64)
+return 0
+;;
+esac
+return 1
+}
+
 supported_target() {
 case "$1" in
 *-softmmu)
@@ -268,6 +279,7 @@ supported_target() {
 supported_hax_target "$1" && return 0
 supported_hvf_target "$1" && return 0
 supported_whpx_target "$1" && return 0
+supported_nvmm_target "$1" && return 0
 print_error "TCG disabled, but hardware accelerator not available for 
'$target'"
 return 1
 }
@@ -388,6 +400,7 @@ kvm="no"
 hax="no"
 hvf="no"
 whpx="no"
+nvmm="no"
 rdma=""
 pvrdma=""
 gprof="no"
@@ -823,6 +836,7 @@ DragonFly)
 NetBSD)
   bsd="yes"
   hax="yes"
+  nvmm="yes"
   make="${MAKE-gmake}"
   audio_drv_list="oss try-sdl"
   audio_possible_drivers="oss sdl"
@@ -1169,6 +1183,10 @@ for opt do
   ;;
   --enable-whpx) whpx="yes"
   ;;
+  --disable-nvmm) nvmm="no"
+  ;;
+  --enable-nvmm) nvmm="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1773,6 +1791,7 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   hax HAX acceleration support
   hvf Hypervisor.framework acceleration support
   whpxWindows Hypervisor Platform acceleration support
+  nvmmNetBSD Virtual Machine Monitor acceleration support
   rdmaEnable RDMA-based migration
   pvrdma  Enable PVRDMA support
   vde support for vde network
@@ -2764,6 +2783,20 @@ if test "$whpx" != "no" ; then
 fi
 fi

+##
+# NetBSD Virtual Machine Monitor (NVMM) accelerator check
+if test "$nvmm" != "no" ; then
+if check_include "nvmm.h" ; then
+nvmm="yes"
+   LIBS="-lnvmm $LIBS"
+else
+if test "$nvmm" = "yes"; then
+feature_not_found "NVMM" "NVMM is not available"
+fi
+nvmm="no"
+fi
+fi
+
 ##
 # Sparse probe
 if test "$sparse" != "no" ; then
@@ -6543,6 +6576,7 @@ echo "KVM support   $kvm"
 echo "HAX support   $hax"
 echo "HVF support   $hvf"
 echo "WHPX support  $whpx"
+echo "NVMM support  $nvmm"
 echo "TCG support   $tcg"
 if test "$tcg" = "yes" ; then
 echo "TCG debug enabled $debug_tcg"
@@ -7828,6 +7862,9 @@ fi
 if test "$target_aligned_only" = "yes" ; then
   echo "TARGET_ALIGNED_ONLY=y" >> $config_target_mak
 fi
+if supported_nvmm_target $target; then
+echo "CONFIG_NVMM=y" >> $config_target_mak
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi
diff --git a/qemu-options.hx b/qemu-options.hx
index 224a8e8712..10c046c916 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -31,7 +31,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
 "-machine [type=]name[,prop[=value][,...]]\n"
 "selects emulated machine ('-machine help' for list)\n"
 "property accel=accel1[:accel2[:...]] selects 
accelerator\n"
-"supported accelerators are kvm, xen, hax, hvf, whpx or 
tcg (default: tcg)\n"
+"supported accelerators are kvm, xen, hax, hvf, nvmm, whpx 
or tcg (default: tcg)\n"
 "vmport=on|off|auto controls emulation of vmport (default: 
auto)\n"
 "dump-guest-core=on|off include guest memory in a core 
dump (default=on)\n"
 "mem-merge=on|off contr

[PATCH v4 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Hello QEMU Community!

Over the past year the NetBSD team has been working hard on a new user-mode API
for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
This new API adds user-mode capabilities to create and manage virtual machines,
configure memory mappings for guest machines, and create and control execution
of virtual processors.

With this new API we are now able to bring our hypervisor to the QEMU
community! The following patches implement the NetBSD Virtual Machine Monitor
accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.

When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile the
accelerator for use. At runtime using the '-accel nvmm' should see a
significant performance improvement over emulation, much like when using 'hax'
on NetBSD.

The documentation for this new API is visible at https://man.netbsd.org under
the libnvmm(3) and nvmm(4) pages.

NVMM was designed and implemented by Maxime Villard.

Thank you for your feedback.

Refrences:
https://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html

Test plan:

1. Download a NetBSD 9.0 pre-release snapshot:
http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/images/NetBSD-9.0_RC1-amd64.iso

2. Install it natively on a not too old x86_64 hardware (Intel or AMD).

There is no support for nested virtualization in NVMM.

3. Setup the system.

 export PKG_PATH=http://www.ki.nu/pkgsrc/packages/current/NetBSD-9.0_RC1/All
 pkg_add git gmake python37 glib2 bison pkgconf pixman

Install mozilla-rootcerts and follow post-install instructions.

 pkg_add mozilla-rootcerts

More information: https://wiki.qemu.org/Hosts/BSD#NetBSD

4. Build qemu

 mkdir build
 cd build
 ../configure --python=python3.7
 gmake
 gmake check

5. Test

 qemu -accel nvmm ...


History:
v3 -> v4:
 - Correct build warning by adding a missing include
 - Do not set R8-R16 registers unless TARGET_X86_64
v2 -> v3:
 - Register nvmm in targetos NetBSD check
 - Stop including hw/boards.h
 - Rephrase old code comments (remove XXX)
v1 -> v2:
 - Included the testing plan as requested by Philippe Mathieu-Daude
 - Formatting nit fix in qemu-options.hx
 - Document NVMM in the accel section of qemu-options.hx

Maxime Villard (4):
  Add the NVMM vcpu API
  Add the NetBSD Virtual Machine Monitor accelerator.
  Introduce the NVMM impl
  Add the NVMM acceleration enlightenments

 accel/stubs/Makefile.objs |1 +
 accel/stubs/nvmm-stub.c   |   43 ++
 configure |   37 ++
 cpus.c|   58 ++
 include/sysemu/hw_accel.h |   14 +
 include/sysemu/nvmm.h |   35 ++
 qemu-options.hx   |   16 +-
 target/i386/Makefile.objs |1 +
 target/i386/helper.c  |2 +-
 target/i386/nvmm-all.c| 1226 +
 10 files changed, 1424 insertions(+), 9 deletions(-)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h
 create mode 100644 target/i386/nvmm-all.c

--
2.25.0

[PATCH v4 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

From: Maxime Villard 

Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
introduces the nvmm.h sysemu API for managing the vcpu scheduling and
management.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Jared McNeill 
---
 accel/stubs/Makefile.objs |  1 +
 accel/stubs/nvmm-stub.c   | 43 +++
 include/sysemu/nvmm.h | 35 +++
 3 files changed, 79 insertions(+)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h

diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
index 3894caf95d..09f2d3e1dd 100644
--- a/accel/stubs/Makefile.objs
+++ b/accel/stubs/Makefile.objs
@@ -1,5 +1,6 @@
 obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
 obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
 obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
+obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
 obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
 obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
new file mode 100644
index 00..c2208b84a3
--- /dev/null
+++ b/accel/stubs/nvmm-stub.c
@@ -0,0 +1,43 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "sysemu/nvmm.h"
+
+int nvmm_init_vcpu(CPUState *cpu)
+{
+return -1;
+}
+
+int nvmm_vcpu_exec(CPUState *cpu)
+{
+return -1;
+}
+
+void nvmm_destroy_vcpu(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_state(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_init(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
+{
+}
diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
new file mode 100644
index 00..10496f3980
--- /dev/null
+++ b/include/sysemu/nvmm.h
@@ -0,0 +1,35 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_NVMM_H
+#define QEMU_NVMM_H
+
+#include "config-host.h"
+#include "qemu-common.h"
+
+int nvmm_init_vcpu(CPUState *);
+int nvmm_vcpu_exec(CPUState *);
+void nvmm_destroy_vcpu(CPUState *);
+
+void nvmm_cpu_synchronize_state(CPUState *);
+void nvmm_cpu_synchronize_post_reset(CPUState *);
+void nvmm_cpu_synchronize_post_init(CPUState *);
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
+
+#ifdef CONFIG_NVMM
+
+int nvmm_enabled(void);
+
+#else /* CONFIG_NVMM */
+
+#define nvmm_enabled() (0)
+
+#endif /* CONFIG_NVMM */
+
+#endif /* CONFIG_NVMM */
--
2.25.0

Re: [PATCH v2 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

On 06.02.2020 17:07, Philippe Mathieu-Daudé wrote:
> On 2/6/20 4:38 PM, Kamil Rytarowski wrote:
>> On 06.02.2020 15:13, Markus Armbruster wrote:
>>> Kamil Rytarowski  writes:
>>>
>>>> On 06.02.2020 14:09, Philippe Mathieu-Daudé wrote:
>>>>> On Thu, Feb 6, 2020 at 2:06 PM Markus Armbruster
>>>>>  wrote:
>>>>>> Kamil Rytarowski  writes:
>>>>>>
>>>>>>> On 03.02.2020 12:54, Philippe Mathieu-Daudé wrote:
>>>>>>>>> @@ -2029,6 +2072,19 @@ static void
>>>>>>>>> qemu_whpx_start_vcpu(CPUState *cpu)
>>>>>>>>>    #endif
>>>>>>>>>    }
>>>>>>>>>
>>>>>>>>> +static void qemu_nvmm_start_vcpu(CPUState *cpu)
>>>>>>>>> +{
>>>>>>>>> +    char thread_name[VCPU_THREAD_NAME_SIZE];
>>>>>>>>> +
>>>>>>>>> +    cpu->thread = g_malloc0(sizeof(QemuThread));
>>>>>>>>> +    cpu->halt_cond = g_malloc0(sizeof(QemuCond));
>>>>>>>>
>>>>>>>> Nitpick, we prefer g_new0().
>>>>>>>
>>>>>>> In this file other qemu_*_start_vcpu() use  g_malloc0().
>>>>>>>
>>>>>>> I will leave this part unchanged and defer tor future style
>>>>>>> fixups if
>>>>>>> someone is interested.
>>>>>>
>>>>>> Time to re-run Coccinelle with the semantic patch from commit
>>>>>> b45c03f585e.
>>>>>
>>>>> I thought about it, but then noticed it would be clever to modify
>>>>> checkpatch to refuse 'g_malloc0?(.*sizeof.*);'
>>>>>
>>>>>
>>>>
>>>> As the patchset was reviewed, could we please merge it in the current
>>>> (v3) form (*) please?
>>>
>>> No objection.  If I wanted you to clean this up before we accept your
>>> work, I would've told you :)
>>>
>>> [...]
>>>
>>>
>>
>> I see. I don't own myself a merge queue so I depend on yours.
>
> As you said [*] you'd love to have this feature in NetBSD 9.0, no
> objection neither. You still need some X86 specialist to review patch 3.
> The usual reviewers Paolo/Eduardo/Richard are currently very busy.
>
> Also while I'd love to use this feature to be able to regularly run QEMU
> CI on NetBSD, I don't have time to test it on a bare metal hardware :|
> Maybe do you know someone from the NetBSD community who already did?
>
> [*] https://www.mail-archive.com/qemu-devel@nongnu.org/msg676199.html
>
>

I'm going to find a person to test it and submit "Tested-by:".

Re: [PATCH v2 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

On 06.02.2020 15:13, Markus Armbruster wrote:
> Kamil Rytarowski  writes:
>
>> On 06.02.2020 14:09, Philippe Mathieu-Daudé wrote:
>>> On Thu, Feb 6, 2020 at 2:06 PM Markus Armbruster  wrote:
>>>> Kamil Rytarowski  writes:
>>>>
>>>>> On 03.02.2020 12:54, Philippe Mathieu-Daudé wrote:
>>>>>>> @@ -2029,6 +2072,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
>>>>>>>   #endif
>>>>>>>   }
>>>>>>>
>>>>>>> +static void qemu_nvmm_start_vcpu(CPUState *cpu)
>>>>>>> +{
>>>>>>> +char thread_name[VCPU_THREAD_NAME_SIZE];
>>>>>>> +
>>>>>>> +cpu->thread = g_malloc0(sizeof(QemuThread));
>>>>>>> +cpu->halt_cond = g_malloc0(sizeof(QemuCond));
>>>>>>
>>>>>> Nitpick, we prefer g_new0().
>>>>>
>>>>> In this file other qemu_*_start_vcpu() use  g_malloc0().
>>>>>
>>>>> I will leave this part unchanged and defer tor future style fixups if
>>>>> someone is interested.
>>>>
>>>> Time to re-run Coccinelle with the semantic patch from commit
>>>> b45c03f585e.
>>>
>>> I thought about it, but then noticed it would be clever to modify
>>> checkpatch to refuse 'g_malloc0?(.*sizeof.*);'
>>>
>>>
>>
>> As the patchset was reviewed, could we please merge it in the current
>> (v3) form (*) please?
>
> No objection.  If I wanted you to clean this up before we accept your
> work, I would've told you :)
>
> [...]
>
>

I see. I don't own myself a merge queue so I depend on yours.

Thank you in advance!

Re: [PATCH v2 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

On 06.02.2020 14:09, Philippe Mathieu-Daudé wrote:
> On Thu, Feb 6, 2020 at 2:06 PM Markus Armbruster  wrote:
>> Kamil Rytarowski  writes:
>>
>>> On 03.02.2020 12:54, Philippe Mathieu-Daudé wrote:
>>>>> @@ -2029,6 +2072,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
>>>>>   #endif
>>>>>   }
>>>>>
>>>>> +static void qemu_nvmm_start_vcpu(CPUState *cpu)
>>>>> +{
>>>>> +char thread_name[VCPU_THREAD_NAME_SIZE];
>>>>> +
>>>>> +cpu->thread = g_malloc0(sizeof(QemuThread));
>>>>> +cpu->halt_cond = g_malloc0(sizeof(QemuCond));
>>>>
>>>> Nitpick, we prefer g_new0().
>>>
>>> In this file other qemu_*_start_vcpu() use  g_malloc0().
>>>
>>> I will leave this part unchanged and defer tor future style fixups if
>>> someone is interested.
>>
>> Time to re-run Coccinelle with the semantic patch from commit
>> b45c03f585e.
>
> I thought about it, but then noticed it would be clever to modify
> checkpatch to refuse 'g_malloc0?(.*sizeof.*);'
>
>

As the patchset was reviewed, could we please merge it in the current
(v3) form (*) please?

Feel free to fixup the style after that as you like.

We plan to release NetBSD 9.0 in 1-2 weeks unless there will be a delay.

https://blog.netbsd.org/tnf/entry/second_final_release_candidate_for

(*) https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg01405.html

Re: [PATCH v3 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Am I supposed to do something with this or is this an issue in a script?

On 06.02.2020 14:13, no-re...@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20200206115731.13552-1-...@gmx.com/
>
>
>
> Hi,
>
> This series seems to have some coding style problems. See output below for
> more information:
>
> Subject: [PATCH v3 0/4] Implements the NetBSD Virtual Machine Monitor 
> accelerator
> Message-id: 20200206115731.13552-1-...@gmx.com
> Type: series
>
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> git rev-parse base > /dev/null || exit 0
> git config --local diff.renamelimit 0
> git config --local diff.renames True
> git config --local diff.algorithm histogram
> ./scripts/checkpatch.pl --mailback base..
> === TEST SCRIPT END ===
>
> Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
> fatal: git fetch_pack: expected ACK/NAK, got 'ERR upload-pack: not our ref 
> 1c298dad3d820f7a2161054ff581cf2fa65ee1b4'
> fatal: The remote end hung up unexpectedly
> error: Could not fetch 3c8cf5a9c21ff8782164d1def7f44bd888713384
> Traceback (most recent call last):
>   File "patchew-tester/src/patchew-cli", line 521, in test_one
> git_clone_repo(clone, r["repo"], r["head"], logf, True)
>   File "patchew-tester/src/patchew-cli", line 48, in git_clone_repo
> stdout=logf, stderr=logf)
>   File "/opt/rh/rh-python36/root/usr/lib64/python3.6/subprocess.py", line 
> 291, in check_call
> raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['git', 'remote', 'add', '-f', 
> '--mirror=fetch', '3c8cf5a9c21ff8782164d1def7f44bd888713384', 
> 'https://github.com/patchew-project/qemu']' returned non-zero exit status 1.
>
>
>
> The full log is available at
> http://patchew.org/logs/20200206115731.13552-1-...@gmx.com/testing.checkpatch/?type=message.
> ---
> Email generated automatically by Patchew [https://patchew.org/].
> Please send your feedback to patchew-de...@redhat.com
>

[PATCH v3 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NetBSD Virtual Machine Monitor (NVMM) target. Which
acts as a hypervisor accelerator for QEMU on the NetBSD platform. This enables
QEMU much greater speed over the emulated x86_64 path's that are taken on
NetBSD today.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
---
 target/i386/Makefile.objs |1 +
 target/i386/nvmm-all.c| 1221 +
 2 files changed, 1222 insertions(+)
 create mode 100644 target/i386/nvmm-all.c

diff --git a/target/i386/Makefile.objs b/target/i386/Makefile.objs
index 48e0c28434..bdcdb32e93 100644
--- a/target/i386/Makefile.objs
+++ b/target/i386/Makefile.objs
@@ -17,6 +17,7 @@ obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-posix.o
 endif
 obj-$(CONFIG_HVF) += hvf/
 obj-$(CONFIG_WHPX) += whpx-all.o
+obj-$(CONFIG_NVMM) += nvmm-all.o
 endif
 obj-$(CONFIG_SEV) += sev.o
 obj-$(call lnot,$(CONFIG_SEV)) += sev-stub.o
diff --git a/target/i386/nvmm-all.c b/target/i386/nvmm-all.c
new file mode 100644
index 00..6988400f53
--- /dev/null
+++ b/target/i386/nvmm-all.c
@@ -0,0 +1,1221 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator for QEMU.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/address-spaces.h"
+#include "exec/ioport.h"
+#include "qemu-common.h"
+#include "strings.h"
+#include "sysemu/accel.h"
+#include "sysemu/nvmm.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+
+#include 
+
+struct qemu_vcpu {
+struct nvmm_vcpu vcpu;
+uint8_t tpr;
+bool stop;
+
+/* Window-exiting for INTs/NMIs. */
+bool int_window_exit;
+bool nmi_window_exit;
+
+/* The guest is in an interrupt shadow (POP SS, etc). */
+bool int_shadow;
+};
+
+struct qemu_machine {
+struct nvmm_capability cap;
+struct nvmm_machine mach;
+};
+
+/* -- 
*/
+
+static bool nvmm_allowed;
+static struct qemu_machine qemu_mach;
+
+static struct qemu_vcpu *
+get_qemu_vcpu(CPUState *cpu)
+{
+return (struct qemu_vcpu *)cpu->hax_vcpu;
+}
+
+static struct nvmm_machine *
+get_nvmm_mach(void)
+{
+return &qemu_mach.mach;
+}
+
+/* -- 
*/
+
+static void
+nvmm_set_segment(struct nvmm_x64_state_seg *nseg, const SegmentCache *qseg)
+{
+uint32_t attrib = qseg->flags;
+
+nseg->selector = qseg->selector;
+nseg->limit = qseg->limit;
+nseg->base = qseg->base;
+nseg->attrib.type = __SHIFTOUT(attrib, DESC_TYPE_MASK);
+nseg->attrib.s = __SHIFTOUT(attrib, DESC_S_MASK);
+nseg->attrib.dpl = __SHIFTOUT(attrib, DESC_DPL_MASK);
+nseg->attrib.p = __SHIFTOUT(attrib, DESC_P_MASK);
+nseg->attrib.avl = __SHIFTOUT(attrib, DESC_AVL_MASK);
+nseg->attrib.l = __SHIFTOUT(attrib, DESC_L_MASK);
+nseg->attrib.def = __SHIFTOUT(attrib, DESC_B_MASK);
+nseg->attrib.g = __SHIFTOUT(attrib, DESC_G_MASK);
+}
+
+static void
+nvmm_set_registers(CPUState *cpu)
+{
+struct CPUX86State *env = (CPUArchState *)cpu->env_ptr;
+struct nvmm_machine *mach = get_nvmm_mach();
+struct qemu_vcpu *qcpu = get_qemu_vcpu(cpu);
+struct nvmm_vcpu *vcpu = &qcpu->vcpu;
+struct nvmm_x64_state *state = vcpu->state;
+uint64_t bitmap;
+size_t i;
+int ret;
+
+assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
+
+/* GPRs. */
+state->gprs[NVMM_X64_GPR_RAX] = env->regs[R_EAX];
+state->gprs[NVMM_X64_GPR_RCX] = env->regs[R_ECX];
+state->gprs[NVMM_X64_GPR_RDX] = env->regs[R_EDX];
+state->gprs[NVMM_X64_GPR_RBX] = env->regs[R_EBX];
+state->gprs[NVMM_X64_GPR_RSP] = env->regs[R_ESP];
+state->gprs[NVMM_X64_GPR_RBP] = env->regs[R_EBP];
+state->gprs[NVMM_X64_GPR_RSI] = env->regs[R_ESI];
+state->gprs[NVMM_X64_GPR_RDI] = env->regs[R_EDI];
+state->gprs[NVMM_X64_GPR_R8]  = env->regs[R_R8];
+state->gprs[NVMM_X64_GPR_R9]  = env->regs[R_R9];
+state->gprs[NVMM_X64_GPR_R10] = env->regs[R_R10];
+state->gprs[NVMM_X64_GPR_R11] = env->regs[R_R11];
+state->gprs[NVMM_X64_GPR_R12] = env->regs[R_R12];
+state->gprs[NVMM_X64_GPR_R13] = env->regs[R_R13];
+state->gprs[NVMM_X64_GPR_R14] = env->regs[R_R14];
+state->gprs[NVMM_X64_GPR_R15] = env->regs[R_R15];
+
+/* RIP and RFLAGS. */
+state->

[PATCH v3 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NVMM accelerator cpu enlightenments to actually use the nvmm-all
accelerator on NetBSD platforms.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
---
 cpus.c| 58 +++
 include/sysemu/hw_accel.h | 14 ++
 target/i386/helper.c  |  2 +-
 3 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index b4f8b84b61..f833da4a60 100644
--- a/cpus.c
+++ b/cpus.c
@@ -42,6 +42,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"
 #include "exec/exec-all.h"

 #include "qemu/thread.h"
@@ -1670,6 +1671,48 @@ static void *qemu_whpx_cpu_thread_fn(void *arg)
 return NULL;
 }

+static void *qemu_nvmm_cpu_thread_fn(void *arg)
+{
+CPUState *cpu = arg;
+int r;
+
+assert(nvmm_enabled());
+
+rcu_register_thread();
+
+qemu_mutex_lock_iothread();
+qemu_thread_get_self(cpu->thread);
+cpu->thread_id = qemu_get_thread_id();
+current_cpu = cpu;
+
+r = nvmm_init_vcpu(cpu);
+if (r < 0) {
+fprintf(stderr, "nvmm_init_vcpu failed: %s\n", strerror(-r));
+exit(1);
+}
+
+/* signal CPU creation */
+cpu->created = true;
+qemu_cond_signal(&qemu_cpu_cond);
+
+do {
+if (cpu_can_run(cpu)) {
+r = nvmm_vcpu_exec(cpu);
+if (r == EXCP_DEBUG) {
+cpu_handle_guest_debug(cpu);
+}
+}
+qemu_wait_io_event(cpu);
+} while (!cpu->unplug || cpu_can_run(cpu));
+
+nvmm_destroy_vcpu(cpu);
+cpu->created = false;
+qemu_cond_signal(&qemu_cpu_cond);
+qemu_mutex_unlock_iothread();
+rcu_unregister_thread();
+return NULL;
+}
+
 #ifdef _WIN32
 static void CALLBACK dummy_apc_func(ULONG_PTR unused)
 {
@@ -2038,6 +2081,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
 #endif
 }

+static void qemu_nvmm_start_vcpu(CPUState *cpu)
+{
+char thread_name[VCPU_THREAD_NAME_SIZE];
+
+cpu->thread = g_malloc0(sizeof(QemuThread));
+cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+qemu_cond_init(cpu->halt_cond);
+snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/NVMM",
+ cpu->cpu_index);
+qemu_thread_create(cpu->thread, thread_name, qemu_nvmm_cpu_thread_fn,
+   cpu, QEMU_THREAD_JOINABLE);
+}
+
 static void qemu_dummy_start_vcpu(CPUState *cpu)
 {
 char thread_name[VCPU_THREAD_NAME_SIZE];
@@ -2078,6 +2134,8 @@ void qemu_init_vcpu(CPUState *cpu)
 qemu_tcg_init_vcpu(cpu);
 } else if (whpx_enabled()) {
 qemu_whpx_start_vcpu(cpu);
+} else if (nvmm_enabled()) {
+qemu_nvmm_start_vcpu(cpu);
 } else {
 qemu_dummy_start_vcpu(cpu);
 }
diff --git a/include/sysemu/hw_accel.h b/include/sysemu/hw_accel.h
index 0ec2372477..dbfa7a02f9 100644
--- a/include/sysemu/hw_accel.h
+++ b/include/sysemu/hw_accel.h
@@ -15,6 +15,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/kvm.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"

 static inline void cpu_synchronize_state(CPUState *cpu)
 {
@@ -27,6 +28,9 @@ static inline void cpu_synchronize_state(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_state(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_state(cpu);
+}
 }

 static inline void cpu_synchronize_post_reset(CPUState *cpu)
@@ -40,6 +44,10 @@ static inline void cpu_synchronize_post_reset(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_reset(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_reset(cpu);
+}
+
 }

 static inline void cpu_synchronize_post_init(CPUState *cpu)
@@ -53,6 +61,9 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_init(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_init(cpu);
+}
 }

 static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
@@ -66,6 +77,9 @@ static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_pre_loadvm(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_pre_loadvm(cpu);
+}
 }

 #endif /* QEMU_HW_ACCEL_H */
diff --git a/target/i386/helper.c b/target/i386/helper.c
index c3a6e4fabe..2e79d61329 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -981,7 +981,7 @@ void cpu_report_tpr_access(CPUX86State *env, TPRAccess 
access)
 X86CPU *cpu = env_archcpu(env);
 CPUState *cs = env_cpu(env);

-if (kvm_enabled() || whpx_enabled()) {
+if (kvm_enabled() || whpx_enabled() || nvmm_enabled()) {
 env->tpr_access_type = access;

 cpu_interrupt(cs, CPU_INTERRUPT_TPR);
--
2.25.0

[PATCH v3 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

From: Maxime Villard 

Introduces the configure support for the new NetBSD Virtual Machine Monitor that
allows for hypervisor acceleration from usermode components on the NetBSD
platform.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
---
 configure   | 37 +
 qemu-options.hx | 16 
 2 files changed, 45 insertions(+), 8 deletions(-)

diff --git a/configure b/configure
index 115dc38085..d4a837cf9d 100755
--- a/configure
+++ b/configure
@@ -241,6 +241,17 @@ supported_whpx_target() {
 return 1
 }

+supported_nvmm_target() {
+test "$nvmm" = "yes" || return 1
+glob "$1" "*-softmmu" || return 1
+case "${1%-softmmu}" in
+i386|x86_64)
+return 0
+;;
+esac
+return 1
+}
+
 supported_target() {
 case "$1" in
 *-softmmu)
@@ -268,6 +279,7 @@ supported_target() {
 supported_hax_target "$1" && return 0
 supported_hvf_target "$1" && return 0
 supported_whpx_target "$1" && return 0
+supported_nvmm_target "$1" && return 0
 print_error "TCG disabled, but hardware accelerator not available for 
'$target'"
 return 1
 }
@@ -388,6 +400,7 @@ kvm="no"
 hax="no"
 hvf="no"
 whpx="no"
+nvmm="no"
 rdma=""
 pvrdma=""
 gprof="no"
@@ -823,6 +836,7 @@ DragonFly)
 NetBSD)
   bsd="yes"
   hax="yes"
+  nvmm="yes"
   make="${MAKE-gmake}"
   audio_drv_list="oss try-sdl"
   audio_possible_drivers="oss sdl"
@@ -1169,6 +1183,10 @@ for opt do
   ;;
   --enable-whpx) whpx="yes"
   ;;
+  --disable-nvmm) nvmm="no"
+  ;;
+  --enable-nvmm) nvmm="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1773,6 +1791,7 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   hax HAX acceleration support
   hvf Hypervisor.framework acceleration support
   whpxWindows Hypervisor Platform acceleration support
+  nvmmNetBSD Virtual Machine Monitor acceleration support
   rdmaEnable RDMA-based migration
   pvrdma  Enable PVRDMA support
   vde support for vde network
@@ -2764,6 +2783,20 @@ if test "$whpx" != "no" ; then
 fi
 fi

+##
+# NetBSD Virtual Machine Monitor (NVMM) accelerator check
+if test "$nvmm" != "no" ; then
+if check_include "nvmm.h" ; then
+nvmm="yes"
+   LIBS="-lnvmm $LIBS"
+else
+if test "$nvmm" = "yes"; then
+feature_not_found "NVMM" "NVMM is not available"
+fi
+nvmm="no"
+fi
+fi
+
 ##
 # Sparse probe
 if test "$sparse" != "no" ; then
@@ -6543,6 +6576,7 @@ echo "KVM support   $kvm"
 echo "HAX support   $hax"
 echo "HVF support   $hvf"
 echo "WHPX support  $whpx"
+echo "NVMM support  $nvmm"
 echo "TCG support   $tcg"
 if test "$tcg" = "yes" ; then
 echo "TCG debug enabled $debug_tcg"
@@ -7828,6 +7862,9 @@ fi
 if test "$target_aligned_only" = "yes" ; then
   echo "TARGET_ALIGNED_ONLY=y" >> $config_target_mak
 fi
+if supported_nvmm_target $target; then
+echo "CONFIG_NVMM=y" >> $config_target_mak
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi
diff --git a/qemu-options.hx b/qemu-options.hx
index 224a8e8712..10c046c916 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -31,7 +31,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
 "-machine [type=]name[,prop[=value][,...]]\n"
 "selects emulated machine ('-machine help' for list)\n"
 "property accel=accel1[:accel2[:...]] selects 
accelerator\n"
-"supported accelerators are kvm, xen, hax, hvf, whpx or 
tcg (default: tcg)\n"
+"supported accelerators are kvm, xen, hax, hvf, nvmm, whpx 
or tcg (default: tcg)\n"
 "vmport=on|off|auto controls emulation of vmport (default: 
auto)\n"
 "dump-guest-core=on|off include guest memory in a core 
dump (default=on)\n"
 "mem-merge=on|off controls memory merge support 

[PATCH v3 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

From: Maxime Villard 

Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
introduces the nvmm.h sysemu API for managing the vcpu scheduling and
management.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
Reviewed-by: Philippe Mathieu-Daudé 
---
 accel/stubs/Makefile.objs |  1 +
 accel/stubs/nvmm-stub.c   | 43 +++
 include/sysemu/nvmm.h | 35 +++
 3 files changed, 79 insertions(+)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h

diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
index 3894caf95d..09f2d3e1dd 100644
--- a/accel/stubs/Makefile.objs
+++ b/accel/stubs/Makefile.objs
@@ -1,5 +1,6 @@
 obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
 obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
 obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
+obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
 obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
 obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
new file mode 100644
index 00..c2208b84a3
--- /dev/null
+++ b/accel/stubs/nvmm-stub.c
@@ -0,0 +1,43 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "sysemu/nvmm.h"
+
+int nvmm_init_vcpu(CPUState *cpu)
+{
+return -1;
+}
+
+int nvmm_vcpu_exec(CPUState *cpu)
+{
+return -1;
+}
+
+void nvmm_destroy_vcpu(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_state(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_init(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
+{
+}
diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
new file mode 100644
index 00..10496f3980
--- /dev/null
+++ b/include/sysemu/nvmm.h
@@ -0,0 +1,35 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_NVMM_H
+#define QEMU_NVMM_H
+
+#include "config-host.h"
+#include "qemu-common.h"
+
+int nvmm_init_vcpu(CPUState *);
+int nvmm_vcpu_exec(CPUState *);
+void nvmm_destroy_vcpu(CPUState *);
+
+void nvmm_cpu_synchronize_state(CPUState *);
+void nvmm_cpu_synchronize_post_reset(CPUState *);
+void nvmm_cpu_synchronize_post_init(CPUState *);
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
+
+#ifdef CONFIG_NVMM
+
+int nvmm_enabled(void);
+
+#else /* CONFIG_NVMM */
+
+#define nvmm_enabled() (0)
+
+#endif /* CONFIG_NVMM */
+
+#endif /* CONFIG_NVMM */
--
2.25.0

[PATCH v3 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Hello QEMU Community!

Over the past year the NetBSD team has been working hard on a new user-mode API
for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
This new API adds user-mode capabilities to create and manage virtual machines,
configure memory mappings for guest machines, and create and control execution
of virtual processors.

With this new API we are now able to bring our hypervisor to the QEMU
community! The following patches implement the NetBSD Virtual Machine Monitor
accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.

When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile the
accelerator for use. At runtime using the '-accel nvmm' should see a
significant performance improvement over emulation, much like when using 'hax'
on NetBSD.

The documentation for this new API is visible at https://man.netbsd.org under
the libnvmm(3) and nvmm(4) pages.

NVMM was designed and implemented by Maxime Villard.

Thank you for your feedback.

Refrences:
https://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html

Test plan:

1. Download a NetBSD 9.0 pre-release snapshot:
http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/images/NetBSD-9.0_RC1-amd64.iso

2. Install it natively on a not too old x86_64 hardware (Intel or AMD).

There is no support for nested virtualization in NVMM.

3. Setup the system.

 export PKG_PATH=http://www.ki.nu/pkgsrc/packages/current/NetBSD-9.0_RC1/All
 pkg_add git gmake python37 glib2 bison pkgconf pixman

Install mozilla-rootcerts and follow post-install instructions.

 pkg_add mozilla-rootcerts

More information: https://wiki.qemu.org/Hosts/BSD#NetBSD

4. Build qemu

 mkdir build
 cd build
 ../configure --python=python3.7
 gmake
 gmake check

5. Test

 qemu -accel nvmm ...


History:
v2 -> v3:
 - Register nvmm in targetos NetBSD check
 - Stop including hw/boards.h
 - Rephrase old code comments (remove XXX)
v1 -> v2:
 - Included the testing plan as requested by Philippe Mathieu-Daude
 - Formatting nit fix in qemu-options.hx
 - Document NVMM in the accel section of qemu-options.hx

Maxime Villard (4):
  Add the NVMM vcpu API
  Add the NetBSD Virtual Machine Monitor accelerator.
  Introduce the NVMM impl
  Add the NVMM acceleration enlightenments

 accel/stubs/Makefile.objs |1 +
 accel/stubs/nvmm-stub.c   |   43 ++
 configure |   37 ++
 cpus.c|   58 ++
 include/sysemu/hw_accel.h |   14 +
 include/sysemu/nvmm.h |   35 ++
 qemu-options.hx   |   16 +-
 target/i386/Makefile.objs |1 +
 target/i386/helper.c  |2 +-
 target/i386/nvmm-all.c| 1221 +
 10 files changed, 1419 insertions(+), 9 deletions(-)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h
 create mode 100644 target/i386/nvmm-all.c

--
2.25.0

Re: [PATCH v2 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

On 03.02.2020 12:54, Philippe Mathieu-Daudé wrote:
>> @@ -2029,6 +2072,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
>>   #endif
>>   }
>>
>> +static void qemu_nvmm_start_vcpu(CPUState *cpu)
>> +{
>> +    char thread_name[VCPU_THREAD_NAME_SIZE];
>> +
>> +    cpu->thread = g_malloc0(sizeof(QemuThread));
>> +    cpu->halt_cond = g_malloc0(sizeof(QemuCond));
>
> Nitpick, we prefer g_new0().

In this file other qemu_*_start_vcpu() use  g_malloc0().

I will leave this part unchanged and defer tor future style fixups if
someone is interested.

Re: [PATCH v2 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

On 03.02.2020 12:51, Philippe Mathieu-Daudé wrote:
> Except the XXX comments, LGTM but I'm not a X86 guy.
>
>

These comments were old and I will drop them and resubmit.

Re: [PATCH v2 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

On 03.02.2020 12:41, Philippe Mathieu-Daudé wrote:
>> @@ -1768,6 +1785,7 @@ disabled with --disable-FEATURE, default is
>> enabled if available:
>>     hax HAX acceleration support
>>     hvf Hypervisor.framework acceleration support
>>     whpx    Windows Hypervisor Platform acceleration support
>> +  nvmm    NetBSD Virtual Machine Monitor acceleration support
>>     rdma    Enable RDMA-based migration
>>     pvrdma  Enable PVRDMA support
>>     vde support for vde network
>> @@ -2757,6 +2775,20 @@ if test "$whpx" != "no" ; then
>>   fi
>>   fi
>>
> 
> Maybe you can add something like:
> 
> if test "$targetos" = "NetBSD"; then
>     nvmm="check"
> fi
> 
> to build by default with NVMM if available.

I will add nvmm=yes to the NetBSD) targetos check section.

Re: [PATCH v2 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Ping?

We plan to release NetBSD 9.0 in two weeks and we would love to have
this patchset merged.

"A second (and hopefulle final) release candidate for the upcoming
NetBSD 9.0 release is now available.
Please help testing it!

Tentative final 9.0 release date: February 14, 2020"

http://netbsd.org/

On 28.01.2020 15:09, Kamil Rytarowski wrote:
> Hello QEMU Community!
>
> Over the past year the NetBSD team has been working hard on a new user-mode 
> API
> for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
> This new API adds user-mode capabilities to create and manage virtual 
> machines,
> configure memory mappings for guest machines, and create and control execution
> of virtual processors.
>
> With this new API we are now able to bring our hypervisor to the QEMU
> community! The following patches implement the NetBSD Virtual Machine Monitor
> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
>
> When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile the
> accelerator for use. At runtime using the '-accel nvmm' should see a
> significant performance improvement over emulation, much like when using 'hax'
> on NetBSD.
>
> The documentation for this new API is visible at https://man.netbsd.org under
> the libnvmm(3) and nvmm(4) pages.
>
> NVMM was designed and implemented by Maxime Villard.
>
> Thank you for your feedback.
>
> Refrences:
> https://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html
>
> Test plan:
>
> 1. Download a NetBSD 9.0 pre-release snapshot:
> http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/images/NetBSD-9.0_RC1-amd64.iso
>
> 2. Install it natively on a not too old x86_64 hardware (Intel or AMD).
>
> There is no support for nested virtualization in NVMM.
>
> 3. Setup the system.
>
>  export PKG_PATH=http://www.ki.nu/pkgsrc/packages/current/NetBSD-9.0_RC1/All
>  pkg_add git gmake python37 glib2 bison pkgconf pixman
>
> Install mozilla-rootcerts and follow post-install instructions.
>
>  pkg_add mozilla-rootcerts
>
> More information: https://wiki.qemu.org/Hosts/BSD#NetBSD
>
> 4. Build qemu
>
>  mkdir build
>  cd build
>  ../configure --python=python3.7
>  gmake
>  gmake check
>
> 5. Test
>
>  qemu -accel nvmm ...
>
>
> History:
> v1 -> v2:
>  - Included the testing plan as requested by Philippe Mathieu-Daude
>  - Formatting nit fix in qemu-options.hx
>  - Document NVMM in the accel section of qemu-options.hx
>
> Maxime Villard (4):
>   Add the NVMM vcpu API
>   Add the NetBSD Virtual Machine Monitor accelerator.
>   Introduce the NVMM impl
>   Add the NVMM acceleration enlightenments
>
>  accel/stubs/Makefile.objs |1 +
>  accel/stubs/nvmm-stub.c   |   43 ++
>  configure |   36 ++
>  cpus.c|   58 ++
>  include/sysemu/hw_accel.h |   14 +
>  include/sysemu/nvmm.h |   35 ++
>  qemu-options.hx   |   16 +-
>  target/i386/Makefile.objs |1 +
>  target/i386/helper.c  |2 +-
>  target/i386/nvmm-all.c| 1222 +
>  10 files changed, 1419 insertions(+), 9 deletions(-)
>  create mode 100644 accel/stubs/nvmm-stub.c
>  create mode 100644 include/sysemu/nvmm.h
>  create mode 100644 target/i386/nvmm-all.c
>
> --
> 2.24.1
>

[PATCH v2 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NetBSD Virtual Machine Monitor (NVMM) target. Which
acts as a hypervisor accelerator for QEMU on the NetBSD platform. This enables
QEMU much greater speed over the emulated x86_64 path's that are taken on
NetBSD today.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
---
 target/i386/Makefile.objs |1 +
 target/i386/nvmm-all.c| 1222 +
 2 files changed, 1223 insertions(+)
 create mode 100644 target/i386/nvmm-all.c

diff --git a/target/i386/Makefile.objs b/target/i386/Makefile.objs
index 48e0c28434..bdcdb32e93 100644
--- a/target/i386/Makefile.objs
+++ b/target/i386/Makefile.objs
@@ -17,6 +17,7 @@ obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-posix.o
 endif
 obj-$(CONFIG_HVF) += hvf/
 obj-$(CONFIG_WHPX) += whpx-all.o
+obj-$(CONFIG_NVMM) += nvmm-all.o
 endif
 obj-$(CONFIG_SEV) += sev.o
 obj-$(call lnot,$(CONFIG_SEV)) += sev-stub.o
diff --git a/target/i386/nvmm-all.c b/target/i386/nvmm-all.c
new file mode 100644
index 00..66b08f4f66
--- /dev/null
+++ b/target/i386/nvmm-all.c
@@ -0,0 +1,1222 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator for QEMU.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/address-spaces.h"
+#include "exec/ioport.h"
+#include "qemu-common.h"
+#include "strings.h"
+#include "sysemu/accel.h"
+#include "sysemu/nvmm.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "qemu/main-loop.h"
+#include "hw/boards.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+
+#include 
+
+struct qemu_vcpu {
+struct nvmm_vcpu vcpu;
+uint8_t tpr;
+bool stop;
+
+/* Window-exiting for INTs/NMIs. */
+bool int_window_exit;
+bool nmi_window_exit;
+
+/* The guest is in an interrupt shadow (POP SS, etc). */
+bool int_shadow;
+};
+
+struct qemu_machine {
+struct nvmm_capability cap;
+struct nvmm_machine mach;
+};
+
+/* -- 
*/
+
+static bool nvmm_allowed;
+static struct qemu_machine qemu_mach;
+
+static struct qemu_vcpu *
+get_qemu_vcpu(CPUState *cpu)
+{
+return (struct qemu_vcpu *)cpu->hax_vcpu;
+}
+
+static struct nvmm_machine *
+get_nvmm_mach(void)
+{
+return &qemu_mach.mach;
+}
+
+/* -- 
*/
+
+static void
+nvmm_set_segment(struct nvmm_x64_state_seg *nseg, const SegmentCache *qseg)
+{
+uint32_t attrib = qseg->flags;
+
+nseg->selector = qseg->selector;
+nseg->limit = qseg->limit;
+nseg->base = qseg->base;
+nseg->attrib.type = __SHIFTOUT(attrib, DESC_TYPE_MASK);
+nseg->attrib.s = __SHIFTOUT(attrib, DESC_S_MASK);
+nseg->attrib.dpl = __SHIFTOUT(attrib, DESC_DPL_MASK);
+nseg->attrib.p = __SHIFTOUT(attrib, DESC_P_MASK);
+nseg->attrib.avl = __SHIFTOUT(attrib, DESC_AVL_MASK);
+nseg->attrib.l = __SHIFTOUT(attrib, DESC_L_MASK);
+nseg->attrib.def = __SHIFTOUT(attrib, DESC_B_MASK);
+nseg->attrib.g = __SHIFTOUT(attrib, DESC_G_MASK);
+}
+
+static void
+nvmm_set_registers(CPUState *cpu)
+{
+struct CPUX86State *env = (CPUArchState *)cpu->env_ptr;
+struct nvmm_machine *mach = get_nvmm_mach();
+struct qemu_vcpu *qcpu = get_qemu_vcpu(cpu);
+struct nvmm_vcpu *vcpu = &qcpu->vcpu;
+struct nvmm_x64_state *state = vcpu->state;
+uint64_t bitmap;
+size_t i;
+int ret;
+
+assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
+
+/* GPRs. */
+state->gprs[NVMM_X64_GPR_RAX] = env->regs[R_EAX];
+state->gprs[NVMM_X64_GPR_RCX] = env->regs[R_ECX];
+state->gprs[NVMM_X64_GPR_RDX] = env->regs[R_EDX];
+state->gprs[NVMM_X64_GPR_RBX] = env->regs[R_EBX];
+state->gprs[NVMM_X64_GPR_RSP] = env->regs[R_ESP];
+state->gprs[NVMM_X64_GPR_RBP] = env->regs[R_EBP];
+state->gprs[NVMM_X64_GPR_RSI] = env->regs[R_ESI];
+state->gprs[NVMM_X64_GPR_RDI] = env->regs[R_EDI];
+state->gprs[NVMM_X64_GPR_R8]  = env->regs[R_R8];
+state->gprs[NVMM_X64_GPR_R9]  = env->regs[R_R9];
+state->gprs[NVMM_X64_GPR_R10] = env->regs[R_R10];
+state->gprs[NVMM_X64_GPR_R11] = env->regs[R_R11];
+state->gprs[NVMM_X64_GPR_R12] = env->regs[R_R12];
+state->gprs[NVMM_X64_GPR_R13] = env->regs[R_R13];
+state->gprs[NVMM_X64_GPR_R14] = env->regs[R_R14];
+state->gprs[NVMM_X64_GPR_R15] = env->regs[R_R15];
+
+ 

[PATCH v2 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

From: Maxime Villard 

Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
introduces the nvmm.h sysemu API for managing the vcpu scheduling and
management.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
---
 accel/stubs/Makefile.objs |  1 +
 accel/stubs/nvmm-stub.c   | 43 +++
 include/sysemu/nvmm.h | 35 +++
 3 files changed, 79 insertions(+)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h

diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
index 3894caf95d..09f2d3e1dd 100644
--- a/accel/stubs/Makefile.objs
+++ b/accel/stubs/Makefile.objs
@@ -1,5 +1,6 @@
 obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
 obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
 obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
+obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
 obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
 obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
new file mode 100644
index 00..c2208b84a3
--- /dev/null
+++ b/accel/stubs/nvmm-stub.c
@@ -0,0 +1,43 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "sysemu/nvmm.h"
+
+int nvmm_init_vcpu(CPUState *cpu)
+{
+return -1;
+}
+
+int nvmm_vcpu_exec(CPUState *cpu)
+{
+return -1;
+}
+
+void nvmm_destroy_vcpu(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_state(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_init(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
+{
+}
diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
new file mode 100644
index 00..10496f3980
--- /dev/null
+++ b/include/sysemu/nvmm.h
@@ -0,0 +1,35 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_NVMM_H
+#define QEMU_NVMM_H
+
+#include "config-host.h"
+#include "qemu-common.h"
+
+int nvmm_init_vcpu(CPUState *);
+int nvmm_vcpu_exec(CPUState *);
+void nvmm_destroy_vcpu(CPUState *);
+
+void nvmm_cpu_synchronize_state(CPUState *);
+void nvmm_cpu_synchronize_post_reset(CPUState *);
+void nvmm_cpu_synchronize_post_init(CPUState *);
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
+
+#ifdef CONFIG_NVMM
+
+int nvmm_enabled(void);
+
+#else /* CONFIG_NVMM */
+
+#define nvmm_enabled() (0)
+
+#endif /* CONFIG_NVMM */
+
+#endif /* CONFIG_NVMM */
--
2.24.1

[PATCH v2 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NVMM accelerator cpu enlightenments to actually use the nvmm-all
accelerator on NetBSD platforms.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
---
 cpus.c| 58 +++
 include/sysemu/hw_accel.h | 14 ++
 target/i386/helper.c  |  2 +-
 3 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index b472378b70..3c3f63588c 100644
--- a/cpus.c
+++ b/cpus.c
@@ -42,6 +42,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"
 #include "exec/exec-all.h"

 #include "qemu/thread.h"
@@ -1666,6 +1667,48 @@ static void *qemu_whpx_cpu_thread_fn(void *arg)
 return NULL;
 }

+static void *qemu_nvmm_cpu_thread_fn(void *arg)
+{
+CPUState *cpu = arg;
+int r;
+
+assert(nvmm_enabled());
+
+rcu_register_thread();
+
+qemu_mutex_lock_iothread();
+qemu_thread_get_self(cpu->thread);
+cpu->thread_id = qemu_get_thread_id();
+current_cpu = cpu;
+
+r = nvmm_init_vcpu(cpu);
+if (r < 0) {
+fprintf(stderr, "nvmm_init_vcpu failed: %s\n", strerror(-r));
+exit(1);
+}
+
+/* signal CPU creation */
+cpu->created = true;
+qemu_cond_signal(&qemu_cpu_cond);
+
+do {
+if (cpu_can_run(cpu)) {
+r = nvmm_vcpu_exec(cpu);
+if (r == EXCP_DEBUG) {
+cpu_handle_guest_debug(cpu);
+}
+}
+qemu_wait_io_event(cpu);
+} while (!cpu->unplug || cpu_can_run(cpu));
+
+nvmm_destroy_vcpu(cpu);
+cpu->created = false;
+qemu_cond_signal(&qemu_cpu_cond);
+qemu_mutex_unlock_iothread();
+rcu_unregister_thread();
+return NULL;
+}
+
 #ifdef _WIN32
 static void CALLBACK dummy_apc_func(ULONG_PTR unused)
 {
@@ -2029,6 +2072,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
 #endif
 }

+static void qemu_nvmm_start_vcpu(CPUState *cpu)
+{
+char thread_name[VCPU_THREAD_NAME_SIZE];
+
+cpu->thread = g_malloc0(sizeof(QemuThread));
+cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+qemu_cond_init(cpu->halt_cond);
+snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/NVMM",
+ cpu->cpu_index);
+qemu_thread_create(cpu->thread, thread_name, qemu_nvmm_cpu_thread_fn,
+   cpu, QEMU_THREAD_JOINABLE);
+}
+
 static void qemu_dummy_start_vcpu(CPUState *cpu)
 {
 char thread_name[VCPU_THREAD_NAME_SIZE];
@@ -2069,6 +2125,8 @@ void qemu_init_vcpu(CPUState *cpu)
 qemu_tcg_init_vcpu(cpu);
 } else if (whpx_enabled()) {
 qemu_whpx_start_vcpu(cpu);
+} else if (nvmm_enabled()) {
+qemu_nvmm_start_vcpu(cpu);
 } else {
 qemu_dummy_start_vcpu(cpu);
 }
diff --git a/include/sysemu/hw_accel.h b/include/sysemu/hw_accel.h
index 0ec2372477..dbfa7a02f9 100644
--- a/include/sysemu/hw_accel.h
+++ b/include/sysemu/hw_accel.h
@@ -15,6 +15,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/kvm.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"

 static inline void cpu_synchronize_state(CPUState *cpu)
 {
@@ -27,6 +28,9 @@ static inline void cpu_synchronize_state(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_state(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_state(cpu);
+}
 }

 static inline void cpu_synchronize_post_reset(CPUState *cpu)
@@ -40,6 +44,10 @@ static inline void cpu_synchronize_post_reset(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_reset(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_reset(cpu);
+}
+
 }

 static inline void cpu_synchronize_post_init(CPUState *cpu)
@@ -53,6 +61,9 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_init(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_init(cpu);
+}
 }

 static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
@@ -66,6 +77,9 @@ static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_pre_loadvm(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_pre_loadvm(cpu);
+}
 }

 #endif /* QEMU_HW_ACCEL_H */
diff --git a/target/i386/helper.c b/target/i386/helper.c
index c3a6e4fabe..2e79d61329 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -981,7 +981,7 @@ void cpu_report_tpr_access(CPUX86State *env, TPRAccess 
access)
 X86CPU *cpu = env_archcpu(env);
 CPUState *cs = env_cpu(env);

-if (kvm_enabled() || whpx_enabled()) {
+if (kvm_enabled() || whpx_enabled() || nvmm_enabled()) {
 env->tpr_access_type = access;

 cpu_interrupt(cs, CPU_INTERRUPT_TPR);
--
2.24.1

[PATCH v2 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

From: Maxime Villard 

Introduces the configure support for the new NetBSD Virtual Machine Monitor that
allows for hypervisor acceleration from usermode components on the NetBSD
platform.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
Reviewed-by: Sergio Lopez 
---
 configure   | 36 
 qemu-options.hx | 16 
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/configure b/configure
index 0ce2c0354a..eb456a271e 100755
--- a/configure
+++ b/configure
@@ -241,6 +241,17 @@ supported_whpx_target() {
 return 1
 }

+supported_nvmm_target() {
+test "$nvmm" = "yes" || return 1
+glob "$1" "*-softmmu" || return 1
+case "${1%-softmmu}" in
+i386|x86_64)
+return 0
+;;
+esac
+return 1
+}
+
 supported_target() {
 case "$1" in
 *-softmmu)
@@ -268,6 +279,7 @@ supported_target() {
 supported_hax_target "$1" && return 0
 supported_hvf_target "$1" && return 0
 supported_whpx_target "$1" && return 0
+supported_nvmm_target "$1" && return 0
 print_error "TCG disabled, but hardware accelerator not available for 
'$target'"
 return 1
 }
@@ -387,6 +399,7 @@ kvm="no"
 hax="no"
 hvf="no"
 whpx="no"
+nvmm="no"
 rdma=""
 pvrdma=""
 gprof="no"
@@ -1168,6 +1181,10 @@ for opt do
   ;;
   --enable-whpx) whpx="yes"
   ;;
+  --disable-nvmm) nvmm="no"
+  ;;
+  --enable-nvmm) nvmm="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1768,6 +1785,7 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   hax HAX acceleration support
   hvf Hypervisor.framework acceleration support
   whpxWindows Hypervisor Platform acceleration support
+  nvmmNetBSD Virtual Machine Monitor acceleration support
   rdmaEnable RDMA-based migration
   pvrdma  Enable PVRDMA support
   vde support for vde network
@@ -2757,6 +2775,20 @@ if test "$whpx" != "no" ; then
 fi
 fi

+##
+# NetBSD Virtual Machine Monitor (NVMM) accelerator check
+if test "$nvmm" != "no" ; then
+if check_include "nvmm.h" ; then
+nvmm="yes"
+   LIBS="-lnvmm $LIBS"
+else
+if test "$nvmm" = "yes"; then
+feature_not_found "NVMM" "NVMM is not available"
+fi
+nvmm="no"
+fi
+fi
+
 ##
 # Sparse probe
 if test "$sparse" != "no" ; then
@@ -6495,6 +6527,7 @@ echo "KVM support   $kvm"
 echo "HAX support   $hax"
 echo "HVF support   $hvf"
 echo "WHPX support  $whpx"
+echo "NVMM support  $nvmm"
 echo "TCG support   $tcg"
 if test "$tcg" = "yes" ; then
 echo "TCG debug enabled $debug_tcg"
@@ -7771,6 +7804,9 @@ fi
 if test "$target_aligned_only" = "yes" ; then
   echo "TARGET_ALIGNED_ONLY=y" >> $config_target_mak
 fi
+if supported_nvmm_target $target; then
+echo "CONFIG_NVMM=y" >> $config_target_mak
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi
diff --git a/qemu-options.hx b/qemu-options.hx
index e9d6231438..4ddf7c91a0 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -31,7 +31,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
 "-machine [type=]name[,prop[=value][,...]]\n"
 "selects emulated machine ('-machine help' for list)\n"
 "property accel=accel1[:accel2[:...]] selects 
accelerator\n"
-"supported accelerators are kvm, xen, hax, hvf, whpx or 
tcg (default: tcg)\n"
+"supported accelerators are kvm, xen, hax, hvf, nvmm, whpx 
or tcg (default: tcg)\n"
 "vmport=on|off|auto controls emulation of vmport (default: 
auto)\n"
 "dump-guest-core=on|off include guest memory in a core 
dump (default=on)\n"
 "mem-merge=on|off controls memory merge support (default: 
on)\n"
@@ -63,9 +63,9 @@ Supported machine properties are:
 @table @option
 @item accel=@var{accels1}[:@var{accels2}[:...]]
 This is used to enable an accelerator. Depending on the target architecture,
-kvm, xen, hax, hvf, whpx or tcg can be available.

[PATCH v2 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Hello QEMU Community!

Over the past year the NetBSD team has been working hard on a new user-mode API
for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
This new API adds user-mode capabilities to create and manage virtual machines,
configure memory mappings for guest machines, and create and control execution
of virtual processors.

With this new API we are now able to bring our hypervisor to the QEMU
community! The following patches implement the NetBSD Virtual Machine Monitor
accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.

When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile the
accelerator for use. At runtime using the '-accel nvmm' should see a
significant performance improvement over emulation, much like when using 'hax'
on NetBSD.

The documentation for this new API is visible at https://man.netbsd.org under
the libnvmm(3) and nvmm(4) pages.

NVMM was designed and implemented by Maxime Villard.

Thank you for your feedback.

Refrences:
https://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html

Test plan:

1. Download a NetBSD 9.0 pre-release snapshot:
http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/images/NetBSD-9.0_RC1-amd64.iso

2. Install it natively on a not too old x86_64 hardware (Intel or AMD).

There is no support for nested virtualization in NVMM.

3. Setup the system.

 export PKG_PATH=http://www.ki.nu/pkgsrc/packages/current/NetBSD-9.0_RC1/All
 pkg_add git gmake python37 glib2 bison pkgconf pixman

Install mozilla-rootcerts and follow post-install instructions.

 pkg_add mozilla-rootcerts

More information: https://wiki.qemu.org/Hosts/BSD#NetBSD

4. Build qemu

 mkdir build
 cd build
 ../configure --python=python3.7
 gmake
 gmake check

5. Test

 qemu -accel nvmm ...


History:
v1 -> v2:
 - Included the testing plan as requested by Philippe Mathieu-Daude
 - Formatting nit fix in qemu-options.hx
 - Document NVMM in the accel section of qemu-options.hx

Maxime Villard (4):
  Add the NVMM vcpu API
  Add the NetBSD Virtual Machine Monitor accelerator.
  Introduce the NVMM impl
  Add the NVMM acceleration enlightenments

 accel/stubs/Makefile.objs |1 +
 accel/stubs/nvmm-stub.c   |   43 ++
 configure |   36 ++
 cpus.c|   58 ++
 include/sysemu/hw_accel.h |   14 +
 include/sysemu/nvmm.h |   35 ++
 qemu-options.hx   |   16 +-
 target/i386/Makefile.objs |1 +
 target/i386/helper.c  |2 +-
 target/i386/nvmm-all.c| 1222 +
 10 files changed, 1419 insertions(+), 9 deletions(-)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h
 create mode 100644 target/i386/nvmm-all.c

--
2.24.1

Re: [PATCH 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

On 28.01.2020 13:08, Sergio Lopez wrote:
> On Tue, Jan 28, 2020 at 12:11:51PM +0100, Kamil Rytarowski wrote:
>> On 28.01.2020 10:10, Sergio Lopez wrote:
>>> On Tue, Jan 07, 2020 at 01:53:57PM +0100, Kamil Rytarowski wrote:
>>>> Hello QEMU Community!
>>>>
>>>> Over the past year the NetBSD team has been working hard on a new 
>>>> user-mode API
>>>> for our hypervisor that will be released as part of the upcoming NetBSD 
>>>> 9.0.
>>>> This new API adds user-mode capabilities to create and manage virtual 
>>>> machines,
>>>> configure memory mappings for guest machines, and create and control 
>>>> execution
>>>> of virtual processors.
>>>>
>>>> With this new API we are now able to bring our hypervisor to the QEMU
>>>> community! The following patches implement the NetBSD Virtual Machine 
>>>> Monitor
>>>> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
>>>>
>>>> When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile 
>>>> the
>>>> accelerator for use. At runtime using the '-accel nvmm' should see a
>>>> significant performance improvement over emulation, much like when using 
>>>> 'hax'
>>>> on NetBSD.
>>>>
>>>> The documentation for this new API is visible at https://man.netbsd.org 
>>>> under
>>>> the libnvmm(3) and nvmm(4) pages.
>>>>
>>>> NVMM was designed and implemented by Maxime Villard.
>>>>
>>>> Thank you for your feedback.
>>>
>>> Thank you for working on this, it's nice to see that QEMU will gain
>>> HW-assisted acceleration on NetBSD 9.0. A couple things:
>>>
>>
>> Thank you for the review!
>>
>>>  - Are you and/or Maxime willing to step up as maintainers for the
>>>NVMM support in QEMU?
>>>
>>
>> NVMM (as of today) is part of the NetBSD support and I am the maintainer
>> for the NetBSD code (noted in MAINTAINERS).
>>
>> In case of non-trivial changes I will reach Maxime for his feedback.
> 
> OK, sounds good to me.
> 
>>>  - In the next version of the patch series, please use
>>>"scripts/get_maintainer.pl" to get the list of people you need to
>>>CC for the patch series.
>>>
>>
>> I've submitted a fixup patch '[PATCH v2 2/4] Add the NetBSD Virtual
>> Machine Monitor accelerator.' instead of the full series.
>>
>> I have rechecked the maintainers with the patch and I don't see anybody
>> else as a candidate for review.
> 
> $ ./scripts/get_maintainer.pl /tmp/0004.patch 
> Paolo Bonzini  (maintainer:Main loop)
> Richard Henderson  (maintainer:Overall TCG CPUs)
> Eduardo Habkost  (maintainer:X86 TCG CPUs)
> qemu-devel@nongnu.org (open list:All patches CC here)
> 
> $ ./scripts/get_maintainer.pl /tmp/0002.patch 
> get_maintainer.pl: No maintainers found, printing recent contributors.
> get_maintainer.pl: Do not blindly cc: them on patches!  Use common sense.
> 
> "Philippe Mathieu-Daudé"  (commit_signer:31/112=28%)
> Paolo Bonzini  (commit_signer:31/112=28%)
> Thomas Huth  
> (commit_signer:25/112=22%,commit_signer:14/53=26%)
> "Daniel P. Berrangé"  (commit_signer:24/112=21%)
> Laurent Vivier  
> (commit_signer:22/112=20%,commit_signer:10/53=19%)
> Gerd Hoffmann  (commit_signer:13/53=25%)
> "Michael S. Tsirkin"  (commit_signer:9/53=17%)
> Markus Armbruster  (commit_signer:9/53=17%)
> qemu-devel@nongnu.org (open list:All patches CC here)
> 
> I think you should resping the whole series as v2, adding Richard,
> Eduardo and Philippe to the CC list. That should help with the review
> process.
> 

OK. I will do it.

Please have a look at v2 2/4, submit reviewed-by and I will submit the
whole series again.

> Thanks,
> Sergio.
> 
>> I don't maintain a merge queue on my own. Please put the reviewed
>> patches on a merge queue yourself.
>>
>>
>> Thank you in advance,
>>
>>> Thanks,
>>> Sergio.
>>>
>>
>>
> 
> 
> 




signature.asc
Description: OpenPGP digital signature

Re: [PATCH 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

On 28.01.2020 10:10, Sergio Lopez wrote:
> On Tue, Jan 07, 2020 at 01:53:57PM +0100, Kamil Rytarowski wrote:
>> Hello QEMU Community!
>>
>> Over the past year the NetBSD team has been working hard on a new user-mode 
>> API
>> for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
>> This new API adds user-mode capabilities to create and manage virtual 
>> machines,
>> configure memory mappings for guest machines, and create and control 
>> execution
>> of virtual processors.
>>
>> With this new API we are now able to bring our hypervisor to the QEMU
>> community! The following patches implement the NetBSD Virtual Machine Monitor
>> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
>>
>> When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile 
>> the
>> accelerator for use. At runtime using the '-accel nvmm' should see a
>> significant performance improvement over emulation, much like when using 
>> 'hax'
>> on NetBSD.
>>
>> The documentation for this new API is visible at https://man.netbsd.org under
>> the libnvmm(3) and nvmm(4) pages.
>>
>> NVMM was designed and implemented by Maxime Villard.
>>
>> Thank you for your feedback.
> 
> Thank you for working on this, it's nice to see that QEMU will gain
> HW-assisted acceleration on NetBSD 9.0. A couple things:
> 

Thank you for the review!

>  - Are you and/or Maxime willing to step up as maintainers for the
>NVMM support in QEMU?
> 

NVMM (as of today) is part of the NetBSD support and I am the maintainer
for the NetBSD code (noted in MAINTAINERS).

In case of non-trivial changes I will reach Maxime for his feedback.

>  - In the next version of the patch series, please use
>"scripts/get_maintainer.pl" to get the list of people you need to
>CC for the patch series.
> 

I've submitted a fixup patch '[PATCH v2 2/4] Add the NetBSD Virtual
Machine Monitor accelerator.' instead of the full series.

I have rechecked the maintainers with the patch and I don't see anybody
else as a candidate for review.

I don't maintain a merge queue on my own. Please put the reviewed
patches on a merge queue yourself.


Thank you in advance,

> Thanks,
> Sergio.
> 




signature.asc
Description: OpenPGP digital signature

[PATCH v2 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

From: Maxime Villard 

Introduces the configure support for the new NetBSD Virtual Machine Monitor that
allows for hypervisor acceleration from usermode components on the NetBSD
platform.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
---
 configure   | 36 
 qemu-options.hx | 16 
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/configure b/configure
index 0ce2c0354a..eb456a271e 100755
--- a/configure
+++ b/configure
@@ -241,6 +241,17 @@ supported_whpx_target() {
 return 1
 }

+supported_nvmm_target() {
+test "$nvmm" = "yes" || return 1
+glob "$1" "*-softmmu" || return 1
+case "${1%-softmmu}" in
+i386|x86_64)
+return 0
+;;
+esac
+return 1
+}
+
 supported_target() {
 case "$1" in
 *-softmmu)
@@ -268,6 +279,7 @@ supported_target() {
 supported_hax_target "$1" && return 0
 supported_hvf_target "$1" && return 0
 supported_whpx_target "$1" && return 0
+supported_nvmm_target "$1" && return 0
 print_error "TCG disabled, but hardware accelerator not available for 
'$target'"
 return 1
 }
@@ -387,6 +399,7 @@ kvm="no"
 hax="no"
 hvf="no"
 whpx="no"
+nvmm="no"
 rdma=""
 pvrdma=""
 gprof="no"
@@ -1168,6 +1181,10 @@ for opt do
   ;;
   --enable-whpx) whpx="yes"
   ;;
+  --disable-nvmm) nvmm="no"
+  ;;
+  --enable-nvmm) nvmm="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1768,6 +1785,7 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   hax HAX acceleration support
   hvf Hypervisor.framework acceleration support
   whpxWindows Hypervisor Platform acceleration support
+  nvmmNetBSD Virtual Machine Monitor acceleration support
   rdmaEnable RDMA-based migration
   pvrdma  Enable PVRDMA support
   vde support for vde network
@@ -2757,6 +2775,20 @@ if test "$whpx" != "no" ; then
 fi
 fi

+##
+# NetBSD Virtual Machine Monitor (NVMM) accelerator check
+if test "$nvmm" != "no" ; then
+if check_include "nvmm.h" ; then
+nvmm="yes"
+   LIBS="-lnvmm $LIBS"
+else
+if test "$nvmm" = "yes"; then
+feature_not_found "NVMM" "NVMM is not available"
+fi
+nvmm="no"
+fi
+fi
+
 ##
 # Sparse probe
 if test "$sparse" != "no" ; then
@@ -6495,6 +6527,7 @@ echo "KVM support   $kvm"
 echo "HAX support   $hax"
 echo "HVF support   $hvf"
 echo "WHPX support  $whpx"
+echo "NVMM support  $nvmm"
 echo "TCG support   $tcg"
 if test "$tcg" = "yes" ; then
 echo "TCG debug enabled $debug_tcg"
@@ -7771,6 +7804,9 @@ fi
 if test "$target_aligned_only" = "yes" ; then
   echo "TARGET_ALIGNED_ONLY=y" >> $config_target_mak
 fi
+if supported_nvmm_target $target; then
+echo "CONFIG_NVMM=y" >> $config_target_mak
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi
diff --git a/qemu-options.hx b/qemu-options.hx
index e9d6231438..4ddf7c91a0 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -31,7 +31,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
 "-machine [type=]name[,prop[=value][,...]]\n"
 "selects emulated machine ('-machine help' for list)\n"
 "property accel=accel1[:accel2[:...]] selects 
accelerator\n"
-"supported accelerators are kvm, xen, hax, hvf, whpx or 
tcg (default: tcg)\n"
+"supported accelerators are kvm, xen, hax, hvf, nvmm, whpx 
or tcg (default: tcg)\n"
 "vmport=on|off|auto controls emulation of vmport (default: 
auto)\n"
 "dump-guest-core=on|off include guest memory in a core 
dump (default=on)\n"
 "mem-merge=on|off controls memory merge support (default: 
on)\n"
@@ -63,9 +63,9 @@ Supported machine properties are:
 @table @option
 @item accel=@var{accels1}[:@var{accels2}[:...]]
 This is used to enable an accelerator. Depending on the target architecture,
-kvm, xen, hax, hvf, whpx or tcg can be available. By default, tcg is used. If 

Re: [PATCH 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Ping? 2x

On 15.01.2020 14:14, Kamil Rytarowski wrote:
> Ping?
> 
> On 07.01.2020 13:53, Kamil Rytarowski wrote:
>> Hello QEMU Community!
>>
>> Over the past year the NetBSD team has been working hard on a new user-mode 
>> API
>> for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
>> This new API adds user-mode capabilities to create and manage virtual 
>> machines,
>> configure memory mappings for guest machines, and create and control 
>> execution
>> of virtual processors.
>>
>> With this new API we are now able to bring our hypervisor to the QEMU
>> community! The following patches implement the NetBSD Virtual Machine Monitor
>> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
>>
>> When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile 
>> the
>> accelerator for use. At runtime using the '-accel nvmm' should see a
>> significant performance improvement over emulation, much like when using 
>> 'hax'
>> on NetBSD.
>>
>> The documentation for this new API is visible at https://man.netbsd.org under
>> the libnvmm(3) and nvmm(4) pages.
>>
>> NVMM was designed and implemented by Maxime Villard.
>>
>> Thank you for your feedback.
>>
>> Maxime Villard (4):
>>   Add the NVMM vcpu API
>>   Add the NetBSD Virtual Machine Monitor accelerator.
>>   Introduce the NVMM impl
>>   Add the NVMM acceleration enlightenments
>>
>>  accel/stubs/Makefile.objs |1 +
>>  accel/stubs/nvmm-stub.c   |   43 ++
>>  configure |   36 ++
>>  cpus.c|   58 ++
>>  include/sysemu/hw_accel.h |   14 +
>>  include/sysemu/nvmm.h |   35 ++
>>  qemu-options.hx   |4 +-
>>  target/i386/Makefile.objs |1 +
>>  target/i386/helper.c  |2 +-
>>  target/i386/nvmm-all.c| 1222 +
>>  10 files changed, 1413 insertions(+), 3 deletions(-)
>>  create mode 100644 accel/stubs/nvmm-stub.c
>>  create mode 100644 include/sysemu/nvmm.h
>>  create mode 100644 target/i386/nvmm-all.c
>>
>> --
>> 2.24.0
>>
> 
> 




signature.asc
Description: OpenPGP digital signature

Re: [PATCH 0/4] tests/boot_linux_console: Test booting NetBSD via U-Boot on OrangePi PC

[Kamil Rytarowski]

Thank you for improving testing of the NetBSD target.

On 18.01.2020 20:16, Philippe Mathieu-Daudé wrote:
> This series add a test on the OrangePi PC for:
> - SD Card booting
> - U-boot & UART
> - NetBSD 9
> 
> I simply followed Niek description in docs/orangepi.rst:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg669347.html
> 
> The sdcard image is big, but the test runs very quick (1min),
> even on Travis CI: https://travis-ci.org/philmd/qemu/jobs/638823612#L3778
> 
> (11/48) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9:
>   PASS (65.17 s)
> 
>   $ TMPDIR=/tmp AVOCADO_ALLOW_LARGE_STORAGE=yes avocado --show=app,console 
> run -t machine:orangepi-pc tests/acceptance/boot_linux_console.py
>   JOB ID : 4e11f0d22d121fd766ab5f0956ec464cf9fce64b
>   JOB LOG: 
> /home/phil/avocado/job-results/job-2020-01-18T19.14-4e11f0d/job.log
>(1/1) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9:
>   console: U-Boot SPL 2020.01+dfsg-1 (Jan 08 2020 - 08:19:44 +)
>   console: DRAM: 1024 MiB
>   console: Failed to set core voltage! Can't set CPU frequency
>   console: Trying to boot from MMC1
>   console: U-Boot 2020.01+dfsg-1 (Jan 08 2020 - 08:19:44 +) Allwinner 
> Technology
>   console: CPU:   Allwinner H3 (SUN8I )
>   console: Model: Xunlong Orange Pi Plus / Plus 2
>   console: DRAM:  1 GiB
>   console: MMC:   Device 'mmc@1c11000': seq 1 is in use by 'mmc@1c1'
>   console: mmc@1c0f000: 0, mmc@1c1: 2, mmc@1c11000: 1
>   console: Loading Environment from FAT... Warning: HDMI PHY init timeout!
>   console: Warning: HDMI PHY init timeout!
>   console: In:serial
>   console: Out:   serial
>   console: Err:   serial
>   console: Net:   phy interface7
>   console: Could not get PHY for ethernet@1c3: addr 0
>   console: No ethernet found.
>   console: starting USB...
>   console: Bus usb@1c1b000: USB EHCI 1.00
>   console: Bus usb@1c1d000: USB EHCI 1.00
>   console: scanning bus usb@1c1b000 for devices... 1 USB Device(s) found
>   console: scanning bus usb@1c1d000 for devices... 1 USB Device(s) found
>   console: scanning usb for storage devices... 0 Storage Device(s) found
>   console: Hit any key to stop autoboot:  0
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: =>
>   console: => setenv bootargs root=ld0a
>   console: => setenv kernel netbsd-GENERIC.ub
>   console: => setenv fdtfile dtb/sun8i-h3-orangepi-pc.dtb
>   console: => setenv bootcmd 'fatload mmc 0:1 ${kernel_addr_r} ${kernel}; 
> fatload mmc 0:1 ${fdt_addr_r} ${fdtfile}; fdt addr ${fdt_addr_r}; bootm 
> ${kernel_addr_r} - ${fdt_addr_r}'
>   console: => boot
>   console: 8850008 bytes read in 2583 ms (3.3 MiB/s)
>   console: 28162 bytes read in 34 ms (808.6 KiB/s)
>   console: ## Booting kernel from Legacy Image at 4200 ...
>   console: Image Name:   NetBSD/earmv7hf 9.0_RC1
>   console: Image Type:   ARM Linux Kernel Image (no loading done) 
> (uncompressed)
>   console: Data Size:8849944 Bytes = 8.4 MiB
>   console: Load Address: 
>   console: Entry Point:  
>   console: Verifying Checksum ... OK
>   console: ## Flattened Device Tree blob at 4300
>   console: Booting using the fdt blob at 0x4300
>   console: XIP Kernel Image (no loading done)
>   console: Loading Device Tree to 49ff6000, end 49fffe01 ... OK
>   console: Starting kernel ...
>   console: [   1.000] NetBSD/evbarm (fdt) booting ...
>   console: [   1.000] [ Kernel symbol table missing! ]
>   console: [   1.000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 
> 2002, 2003, 2004, 2005,
>   console: [   1.000] 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 
> 2014, 2015, 2016, 2017,
>   console: [   1.000] 2018, 2019 The NetBSD Foundation, Inc.  All 
> rights reserved.
>   console: [   1.000] Copyright (c) 1982, 1986, 1989, 1991, 1993
>   console: [   1.000] The Regents of the University of California.  
> All rights reserved.
>   console: [   1.000] NetBSD 9.0_RC1 (GENERIC) #0: Wed Nov 27 16:14:52 
> UTC 2019
>   console: [   1.000] 
> mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/evbarm/compile/GENERIC
>   console: [   1.000] total memory = 1024 MB
>   console: [   1.000] avail memory = 1003 MB
>   console: [   1.000] armfdt0 (root)
>   console: [   1.000] simplebus0 at armfdt0: Xunlong Orange Pi PC
>   console: [   1.000] simplebus1 at simplebus0
>   console: [   1.000] simplebus2 at simplebus0
>   console: [   1.000] cpus0 at simplebus0
>   console: [   1.000] simplebus3 at simplebus0
>   console: [   1.000] cpu0 at cpus0: Cortex-A7 r0p5 (Cortex V7A core)
>   console: [   1.000] cpu0: DC enab

Re: [PATCH 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Ping?

On 07.01.2020 13:53, Kamil Rytarowski wrote:
> Hello QEMU Community!
> 
> Over the past year the NetBSD team has been working hard on a new user-mode 
> API
> for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
> This new API adds user-mode capabilities to create and manage virtual 
> machines,
> configure memory mappings for guest machines, and create and control execution
> of virtual processors.
> 
> With this new API we are now able to bring our hypervisor to the QEMU
> community! The following patches implement the NetBSD Virtual Machine Monitor
> accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.
> 
> When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile the
> accelerator for use. At runtime using the '-accel nvmm' should see a
> significant performance improvement over emulation, much like when using 'hax'
> on NetBSD.
> 
> The documentation for this new API is visible at https://man.netbsd.org under
> the libnvmm(3) and nvmm(4) pages.
> 
> NVMM was designed and implemented by Maxime Villard.
> 
> Thank you for your feedback.
> 
> Maxime Villard (4):
>   Add the NVMM vcpu API
>   Add the NetBSD Virtual Machine Monitor accelerator.
>   Introduce the NVMM impl
>   Add the NVMM acceleration enlightenments
> 
>  accel/stubs/Makefile.objs |1 +
>  accel/stubs/nvmm-stub.c   |   43 ++
>  configure |   36 ++
>  cpus.c|   58 ++
>  include/sysemu/hw_accel.h |   14 +
>  include/sysemu/nvmm.h |   35 ++
>  qemu-options.hx   |4 +-
>  target/i386/Makefile.objs |1 +
>  target/i386/helper.c  |2 +-
>  target/i386/nvmm-all.c| 1222 +
>  10 files changed, 1413 insertions(+), 3 deletions(-)
>  create mode 100644 accel/stubs/nvmm-stub.c
>  create mode 100644 include/sysemu/nvmm.h
>  create mode 100644 target/i386/nvmm-all.c
> 
> --
> 2.24.0
> 




signature.asc
Description: OpenPGP digital signature

[PATCH 3/4] Introduce the NVMM impl

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NetBSD Virtual Machine Monitor (NVMM) target. Which
acts as a hypervisor accelerator for QEMU on the NetBSD platform. This enables
QEMU much greater speed over the emulated x86_64 path's that are taken on
NetBSD today.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
---
 target/i386/Makefile.objs |1 +
 target/i386/nvmm-all.c| 1222 +
 2 files changed, 1223 insertions(+)
 create mode 100644 target/i386/nvmm-all.c

diff --git a/target/i386/Makefile.objs b/target/i386/Makefile.objs
index 48e0c28434..bdcdb32e93 100644
--- a/target/i386/Makefile.objs
+++ b/target/i386/Makefile.objs
@@ -17,6 +17,7 @@ obj-$(CONFIG_HAX) += hax-all.o hax-mem.o hax-posix.o
 endif
 obj-$(CONFIG_HVF) += hvf/
 obj-$(CONFIG_WHPX) += whpx-all.o
+obj-$(CONFIG_NVMM) += nvmm-all.o
 endif
 obj-$(CONFIG_SEV) += sev.o
 obj-$(call lnot,$(CONFIG_SEV)) += sev-stub.o
diff --git a/target/i386/nvmm-all.c b/target/i386/nvmm-all.c
new file mode 100644
index 00..66b08f4f66
--- /dev/null
+++ b/target/i386/nvmm-all.c
@@ -0,0 +1,1222 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator for QEMU.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/address-spaces.h"
+#include "exec/ioport.h"
+#include "qemu-common.h"
+#include "strings.h"
+#include "sysemu/accel.h"
+#include "sysemu/nvmm.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "qemu/main-loop.h"
+#include "hw/boards.h"
+#include "qemu/error-report.h"
+#include "qemu/queue.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+
+#include 
+
+struct qemu_vcpu {
+struct nvmm_vcpu vcpu;
+uint8_t tpr;
+bool stop;
+
+/* Window-exiting for INTs/NMIs. */
+bool int_window_exit;
+bool nmi_window_exit;
+
+/* The guest is in an interrupt shadow (POP SS, etc). */
+bool int_shadow;
+};
+
+struct qemu_machine {
+struct nvmm_capability cap;
+struct nvmm_machine mach;
+};
+
+/* -- 
*/
+
+static bool nvmm_allowed;
+static struct qemu_machine qemu_mach;
+
+static struct qemu_vcpu *
+get_qemu_vcpu(CPUState *cpu)
+{
+return (struct qemu_vcpu *)cpu->hax_vcpu;
+}
+
+static struct nvmm_machine *
+get_nvmm_mach(void)
+{
+return &qemu_mach.mach;
+}
+
+/* -- 
*/
+
+static void
+nvmm_set_segment(struct nvmm_x64_state_seg *nseg, const SegmentCache *qseg)
+{
+uint32_t attrib = qseg->flags;
+
+nseg->selector = qseg->selector;
+nseg->limit = qseg->limit;
+nseg->base = qseg->base;
+nseg->attrib.type = __SHIFTOUT(attrib, DESC_TYPE_MASK);
+nseg->attrib.s = __SHIFTOUT(attrib, DESC_S_MASK);
+nseg->attrib.dpl = __SHIFTOUT(attrib, DESC_DPL_MASK);
+nseg->attrib.p = __SHIFTOUT(attrib, DESC_P_MASK);
+nseg->attrib.avl = __SHIFTOUT(attrib, DESC_AVL_MASK);
+nseg->attrib.l = __SHIFTOUT(attrib, DESC_L_MASK);
+nseg->attrib.def = __SHIFTOUT(attrib, DESC_B_MASK);
+nseg->attrib.g = __SHIFTOUT(attrib, DESC_G_MASK);
+}
+
+static void
+nvmm_set_registers(CPUState *cpu)
+{
+struct CPUX86State *env = (CPUArchState *)cpu->env_ptr;
+struct nvmm_machine *mach = get_nvmm_mach();
+struct qemu_vcpu *qcpu = get_qemu_vcpu(cpu);
+struct nvmm_vcpu *vcpu = &qcpu->vcpu;
+struct nvmm_x64_state *state = vcpu->state;
+uint64_t bitmap;
+size_t i;
+int ret;
+
+assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
+
+/* GPRs. */
+state->gprs[NVMM_X64_GPR_RAX] = env->regs[R_EAX];
+state->gprs[NVMM_X64_GPR_RCX] = env->regs[R_ECX];
+state->gprs[NVMM_X64_GPR_RDX] = env->regs[R_EDX];
+state->gprs[NVMM_X64_GPR_RBX] = env->regs[R_EBX];
+state->gprs[NVMM_X64_GPR_RSP] = env->regs[R_ESP];
+state->gprs[NVMM_X64_GPR_RBP] = env->regs[R_EBP];
+state->gprs[NVMM_X64_GPR_RSI] = env->regs[R_ESI];
+state->gprs[NVMM_X64_GPR_RDI] = env->regs[R_EDI];
+state->gprs[NVMM_X64_GPR_R8]  = env->regs[R_R8];
+state->gprs[NVMM_X64_GPR_R9]  = env->regs[R_R9];
+state->gprs[NVMM_X64_GPR_R10] = env->regs[R_R10];
+state->gprs[NVMM_X64_GPR_R11] = env->regs[R_R11];
+state->gprs[NVMM_X64_GPR_R12] = env->regs[R_R12];
+state->gprs[NVMM_X64_GPR_R13] = env->regs[R_R13];
+state->gprs[NVMM_X64_GPR_R14] = env->regs[R_R14];
+state->gprs[NVMM_X64_GPR_R15] = env->regs[R_R15];
+
+/* RIP and RFLAGS. */
+   

[PATCH 1/4] Add the NVMM vcpu API

[Kamil Rytarowski]

From: Maxime Villard 

Adds support for the NetBSD Virtual Machine Monitor (NVMM) stubs and
introduces the nvmm.h sysemu API for managing the vcpu scheduling and
management.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
---
 accel/stubs/Makefile.objs |  1 +
 accel/stubs/nvmm-stub.c   | 43 +++
 include/sysemu/nvmm.h | 35 +++
 3 files changed, 79 insertions(+)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h

diff --git a/accel/stubs/Makefile.objs b/accel/stubs/Makefile.objs
index 3894caf95d..09f2d3e1dd 100644
--- a/accel/stubs/Makefile.objs
+++ b/accel/stubs/Makefile.objs
@@ -1,5 +1,6 @@
 obj-$(call lnot,$(CONFIG_HAX))  += hax-stub.o
 obj-$(call lnot,$(CONFIG_HVF))  += hvf-stub.o
 obj-$(call lnot,$(CONFIG_WHPX)) += whpx-stub.o
+obj-$(call lnot,$(CONFIG_NVMM)) += nvmm-stub.o
 obj-$(call lnot,$(CONFIG_KVM))  += kvm-stub.o
 obj-$(call lnot,$(CONFIG_TCG))  += tcg-stub.o
diff --git a/accel/stubs/nvmm-stub.c b/accel/stubs/nvmm-stub.c
new file mode 100644
index 00..c2208b84a3
--- /dev/null
+++ b/accel/stubs/nvmm-stub.c
@@ -0,0 +1,43 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator stub.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "cpu.h"
+#include "sysemu/nvmm.h"
+
+int nvmm_init_vcpu(CPUState *cpu)
+{
+return -1;
+}
+
+int nvmm_vcpu_exec(CPUState *cpu)
+{
+return -1;
+}
+
+void nvmm_destroy_vcpu(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_state(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_reset(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_post_init(CPUState *cpu)
+{
+}
+
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *cpu)
+{
+}
diff --git a/include/sysemu/nvmm.h b/include/sysemu/nvmm.h
new file mode 100644
index 00..10496f3980
--- /dev/null
+++ b/include/sysemu/nvmm.h
@@ -0,0 +1,35 @@
+/*
+ * Copyright (c) 2018-2019 Maxime Villard, All rights reserved.
+ *
+ * NetBSD Virtual Machine Monitor (NVMM) accelerator support.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_NVMM_H
+#define QEMU_NVMM_H
+
+#include "config-host.h"
+#include "qemu-common.h"
+
+int nvmm_init_vcpu(CPUState *);
+int nvmm_vcpu_exec(CPUState *);
+void nvmm_destroy_vcpu(CPUState *);
+
+void nvmm_cpu_synchronize_state(CPUState *);
+void nvmm_cpu_synchronize_post_reset(CPUState *);
+void nvmm_cpu_synchronize_post_init(CPUState *);
+void nvmm_cpu_synchronize_pre_loadvm(CPUState *);
+
+#ifdef CONFIG_NVMM
+
+int nvmm_enabled(void);
+
+#else /* CONFIG_NVMM */
+
+#define nvmm_enabled() (0)
+
+#endif /* CONFIG_NVMM */
+
+#endif /* CONFIG_NVMM */
--
2.24.0

[PATCH 2/4] Add the NetBSD Virtual Machine Monitor accelerator.

[Kamil Rytarowski]

From: Maxime Villard 

Introduces the configure support for the new NetBSD Virtual Machine Monitor that
allows for hypervisor acceleration from usermode components on the NetBSD
platform.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
---
 configure   | 36 
 qemu-options.hx |  4 ++--
 2 files changed, 38 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index 0ce2c0354a..eb456a271e 100755
--- a/configure
+++ b/configure
@@ -241,6 +241,17 @@ supported_whpx_target() {
 return 1
 }

+supported_nvmm_target() {
+test "$nvmm" = "yes" || return 1
+glob "$1" "*-softmmu" || return 1
+case "${1%-softmmu}" in
+i386|x86_64)
+return 0
+;;
+esac
+return 1
+}
+
 supported_target() {
 case "$1" in
 *-softmmu)
@@ -268,6 +279,7 @@ supported_target() {
 supported_hax_target "$1" && return 0
 supported_hvf_target "$1" && return 0
 supported_whpx_target "$1" && return 0
+supported_nvmm_target "$1" && return 0
 print_error "TCG disabled, but hardware accelerator not available for 
'$target'"
 return 1
 }
@@ -387,6 +399,7 @@ kvm="no"
 hax="no"
 hvf="no"
 whpx="no"
+nvmm="no"
 rdma=""
 pvrdma=""
 gprof="no"
@@ -1168,6 +1181,10 @@ for opt do
   ;;
   --enable-whpx) whpx="yes"
   ;;
+  --disable-nvmm) nvmm="no"
+  ;;
+  --enable-nvmm) nvmm="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1768,6 +1785,7 @@ disabled with --disable-FEATURE, default is enabled if 
available:
   hax HAX acceleration support
   hvf Hypervisor.framework acceleration support
   whpxWindows Hypervisor Platform acceleration support
+  nvmmNetBSD Virtual Machine Monitor acceleration support
   rdmaEnable RDMA-based migration
   pvrdma  Enable PVRDMA support
   vde support for vde network
@@ -2757,6 +2775,20 @@ if test "$whpx" != "no" ; then
 fi
 fi

+##
+# NetBSD Virtual Machine Monitor (NVMM) accelerator check
+if test "$nvmm" != "no" ; then
+if check_include "nvmm.h" ; then
+nvmm="yes"
+   LIBS="-lnvmm $LIBS"
+else
+if test "$nvmm" = "yes"; then
+feature_not_found "NVMM" "NVMM is not available"
+fi
+nvmm="no"
+fi
+fi
+
 ##
 # Sparse probe
 if test "$sparse" != "no" ; then
@@ -6495,6 +6527,7 @@ echo "KVM support   $kvm"
 echo "HAX support   $hax"
 echo "HVF support   $hvf"
 echo "WHPX support  $whpx"
+echo "NVMM support  $nvmm"
 echo "TCG support   $tcg"
 if test "$tcg" = "yes" ; then
 echo "TCG debug enabled $debug_tcg"
@@ -7771,6 +7804,9 @@ fi
 if test "$target_aligned_only" = "yes" ; then
   echo "TARGET_ALIGNED_ONLY=y" >> $config_target_mak
 fi
+if supported_nvmm_target $target; then
+echo "CONFIG_NVMM=y" >> $config_target_mak
+fi
 if test "$target_bigendian" = "yes" ; then
   echo "TARGET_WORDS_BIGENDIAN=y" >> $config_target_mak
 fi
diff --git a/qemu-options.hx b/qemu-options.hx
index e9d6231438..f2dbac68ef 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -31,7 +31,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \
 "-machine [type=]name[,prop[=value][,...]]\n"
 "selects emulated machine ('-machine help' for list)\n"
 "property accel=accel1[:accel2[:...]] selects 
accelerator\n"
-"supported accelerators are kvm, xen, hax, hvf, whpx or 
tcg (default: tcg)\n"
+"supported accelerators are kvm, xen, hax, hvf, nvmm, whpx 
or tcg (default: tcg)\n"
 "vmport=on|off|auto controls emulation of vmport (default: 
auto)\n"
 "dump-guest-core=on|off include guest memory in a core 
dump (default=on)\n"
 "mem-merge=on|off controls memory merge support (default: 
on)\n"
@@ -63,7 +63,7 @@ Supported machine properties are:
 @table @option
 @item accel=@var{accels1}[:@var{accels2}[:...]]
 This is used to enable an accelerator. Depending on the target architecture,
-kvm, xen, hax, hvf, whpx or tcg can be available. By default, tcg is used. If 
there is
+kvm, xen, hax, hvf, nvmm, whpx or tcg can be available. By default, tcg is 
used. If there is
 more than one accelerator specified, the next one is used if the previous one
 fails to initialize.
 @item vmport=on|off|auto
--
2.24.0

[PATCH 4/4] Add the NVMM acceleration enlightenments

[Kamil Rytarowski]

From: Maxime Villard 

Implements the NVMM accelerator cpu enlightenments to actually use the nvmm-all
accelerator on NetBSD platforms.

Signed-off-by: Maxime Villard 
Signed-off-by: Kamil Rytarowski 
---
 cpus.c| 58 +++
 include/sysemu/hw_accel.h | 14 ++
 target/i386/helper.c  |  2 +-
 3 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index b472378b70..3c3f63588c 100644
--- a/cpus.c
+++ b/cpus.c
@@ -42,6 +42,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/hvf.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"
 #include "exec/exec-all.h"

 #include "qemu/thread.h"
@@ -1666,6 +1667,48 @@ static void *qemu_whpx_cpu_thread_fn(void *arg)
 return NULL;
 }

+static void *qemu_nvmm_cpu_thread_fn(void *arg)
+{
+CPUState *cpu = arg;
+int r;
+
+assert(nvmm_enabled());
+
+rcu_register_thread();
+
+qemu_mutex_lock_iothread();
+qemu_thread_get_self(cpu->thread);
+cpu->thread_id = qemu_get_thread_id();
+current_cpu = cpu;
+
+r = nvmm_init_vcpu(cpu);
+if (r < 0) {
+fprintf(stderr, "nvmm_init_vcpu failed: %s\n", strerror(-r));
+exit(1);
+}
+
+/* signal CPU creation */
+cpu->created = true;
+qemu_cond_signal(&qemu_cpu_cond);
+
+do {
+if (cpu_can_run(cpu)) {
+r = nvmm_vcpu_exec(cpu);
+if (r == EXCP_DEBUG) {
+cpu_handle_guest_debug(cpu);
+}
+}
+qemu_wait_io_event(cpu);
+} while (!cpu->unplug || cpu_can_run(cpu));
+
+nvmm_destroy_vcpu(cpu);
+cpu->created = false;
+qemu_cond_signal(&qemu_cpu_cond);
+qemu_mutex_unlock_iothread();
+rcu_unregister_thread();
+return NULL;
+}
+
 #ifdef _WIN32
 static void CALLBACK dummy_apc_func(ULONG_PTR unused)
 {
@@ -2029,6 +2072,19 @@ static void qemu_whpx_start_vcpu(CPUState *cpu)
 #endif
 }

+static void qemu_nvmm_start_vcpu(CPUState *cpu)
+{
+char thread_name[VCPU_THREAD_NAME_SIZE];
+
+cpu->thread = g_malloc0(sizeof(QemuThread));
+cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+qemu_cond_init(cpu->halt_cond);
+snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/NVMM",
+ cpu->cpu_index);
+qemu_thread_create(cpu->thread, thread_name, qemu_nvmm_cpu_thread_fn,
+   cpu, QEMU_THREAD_JOINABLE);
+}
+
 static void qemu_dummy_start_vcpu(CPUState *cpu)
 {
 char thread_name[VCPU_THREAD_NAME_SIZE];
@@ -2069,6 +2125,8 @@ void qemu_init_vcpu(CPUState *cpu)
 qemu_tcg_init_vcpu(cpu);
 } else if (whpx_enabled()) {
 qemu_whpx_start_vcpu(cpu);
+} else if (nvmm_enabled()) {
+qemu_nvmm_start_vcpu(cpu);
 } else {
 qemu_dummy_start_vcpu(cpu);
 }
diff --git a/include/sysemu/hw_accel.h b/include/sysemu/hw_accel.h
index 0ec2372477..dbfa7a02f9 100644
--- a/include/sysemu/hw_accel.h
+++ b/include/sysemu/hw_accel.h
@@ -15,6 +15,7 @@
 #include "sysemu/hax.h"
 #include "sysemu/kvm.h"
 #include "sysemu/whpx.h"
+#include "sysemu/nvmm.h"

 static inline void cpu_synchronize_state(CPUState *cpu)
 {
@@ -27,6 +28,9 @@ static inline void cpu_synchronize_state(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_state(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_state(cpu);
+}
 }

 static inline void cpu_synchronize_post_reset(CPUState *cpu)
@@ -40,6 +44,10 @@ static inline void cpu_synchronize_post_reset(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_reset(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_reset(cpu);
+}
+
 }

 static inline void cpu_synchronize_post_init(CPUState *cpu)
@@ -53,6 +61,9 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_post_init(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_post_init(cpu);
+}
 }

 static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
@@ -66,6 +77,9 @@ static inline void cpu_synchronize_pre_loadvm(CPUState *cpu)
 if (whpx_enabled()) {
 whpx_cpu_synchronize_pre_loadvm(cpu);
 }
+if (nvmm_enabled()) {
+nvmm_cpu_synchronize_pre_loadvm(cpu);
+}
 }

 #endif /* QEMU_HW_ACCEL_H */
diff --git a/target/i386/helper.c b/target/i386/helper.c
index c3a6e4fabe..2e79d61329 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -981,7 +981,7 @@ void cpu_report_tpr_access(CPUX86State *env, TPRAccess 
access)
 X86CPU *cpu = env_archcpu(env);
 CPUState *cs = env_cpu(env);

-if (kvm_enabled() || whpx_enabled()) {
+if (kvm_enabled() || whpx_enabled() || nvmm_enabled()) {
 env->tpr_access_type = access;

 cpu_interrupt(cs, CPU_INTERRUPT_TPR);
--
2.24.0

[PATCH 0/4] Implements the NetBSD Virtual Machine Monitor accelerator

[Kamil Rytarowski]

Hello QEMU Community!

Over the past year the NetBSD team has been working hard on a new user-mode API
for our hypervisor that will be released as part of the upcoming NetBSD 9.0.
This new API adds user-mode capabilities to create and manage virtual machines,
configure memory mappings for guest machines, and create and control execution
of virtual processors.

With this new API we are now able to bring our hypervisor to the QEMU
community! The following patches implement the NetBSD Virtual Machine Monitor
accelerator (NVMM) for QEMU on NetBSD 9.0 and newer hosts.

When compiling QEMU for x86_64 passing the --enable-nvmm flag will compile the
accelerator for use. At runtime using the '-accel nvmm' should see a
significant performance improvement over emulation, much like when using 'hax'
on NetBSD.

The documentation for this new API is visible at https://man.netbsd.org under
the libnvmm(3) and nvmm(4) pages.

NVMM was designed and implemented by Maxime Villard.

Thank you for your feedback.

Maxime Villard (4):
  Add the NVMM vcpu API
  Add the NetBSD Virtual Machine Monitor accelerator.
  Introduce the NVMM impl
  Add the NVMM acceleration enlightenments

 accel/stubs/Makefile.objs |1 +
 accel/stubs/nvmm-stub.c   |   43 ++
 configure |   36 ++
 cpus.c|   58 ++
 include/sysemu/hw_accel.h |   14 +
 include/sysemu/nvmm.h |   35 ++
 qemu-options.hx   |4 +-
 target/i386/Makefile.objs |1 +
 target/i386/helper.c  |2 +-
 target/i386/nvmm-all.c| 1222 +
 10 files changed, 1413 insertions(+), 3 deletions(-)
 create mode 100644 accel/stubs/nvmm-stub.c
 create mode 100644 include/sysemu/nvmm.h
 create mode 100644 target/i386/nvmm-all.c

--
2.24.0

Re: [PATCH v1 0/6] testing/next (netbsd stuff)

[Kamil Rytarowski]

On 07.11.2019 18:46, Peter Maydell wrote:
> On Mon, 4 Nov 2019 at 17:39, Alex Bennée  wrote:
>>
>> Hi,
>>
>> As we approach hard-freeze I'm trying to temper what comes in through
>> the testing/next tree. However it would be nice to get the NetBSD upto
>> speed with the other NetBSDs. Although the serial install is working
>> well for me this has had a rocky road so if others could also give it
>> a good testing that would be great. I've also disabled one of the
>> regular failing tests for non-Linux targets. There are other tests
>> that still fail however including the tests/test-aio-multithread which
>> asserts in the async utils around about 20% of the time:
>>
>>   assertion "QSLIST_EMPTY(&ctx->scheduled_coroutines)" failed: file
>> "/home/qemu/qemu-test.nS1czd/src/util/async.c", line 279, function
>> "aio_ctx_finalize"
> 
> This is unrelated to your NetBSD update in this series -- it's
> one of the persistent intermittents I see on the BSDs:
> https://lore.kernel.org/qemu-devel/20190916153312.GD25552@stefanha-x1.localdomain/t/
> 
> (though the failure rate I see is I think <20%, but I haven't
> really carefully measured it.)
> 
> thanks
> -- PMM
> 

Does this patch rely on AIO API in the kernel? If so than this is
unreliable as of today on NetBSD. We plan to fix it, but there is no
expected time of accomplishment.



signature.asc
Description: OpenPGP digital signature

Re: [PATCH v6 3/4] tests/vm: use console_consume for netbsd

[Kamil Rytarowski]

Thank you for this work. I hope it will be fine now.

On 31.10.2019 09:53, Gerd Hoffmann wrote:
> Use new helper to read all pending console output,
> not just a single char.  Unblocks installer boot.
> 
> Signed-off-by: Gerd Hoffmann 
> ---
>  tests/vm/netbsd | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/vm/netbsd b/tests/vm/netbsd
> index 5e04dcd9b1..d1bfd0 100755
> --- a/tests/vm/netbsd
> +++ b/tests/vm/netbsd
> @@ -93,7 +93,7 @@ class NetBSDVM(basevm.BaseVM):
>  for char in list("5consdev com0\n"):
>  time.sleep(0.2)
>  self.console_send(char)
> -self.console_wait("")
> +self.console_consume()
>  self.console_wait_send("> ", "boot\n")
>  
>  self.console_wait_send("Terminal type","xterm\n")
> 




signature.asc
Description: OpenPGP digital signature

Re: Python 2 and test/vm/netbsd

[Kamil Rytarowski]

On 22.10.2019 15:16, Samuel Thibault wrote:
> Eduardo Habkost, le ven. 18 oct. 2019 13:41:43 -0300, a ecrit:
>> On Fri, Oct 18, 2019 at 06:00:19PM +0200, Samuel Thibault wrote:
>>> It was implemented at the time of introduction of IPv6 in SLIRP. Perhaps
>>> NetBSD has a slightly different behavior which makes the implementation
>>> fail to notice the error.
>>
>> If anybody is interested in investigating it, a network traffic
>> dump generated by `-object filter-dump` is attached.
>
> The dump does show the Destination Unreachable icmp message, so it seems
> it's the guest which does not notice it for some reason.
>
> Samuel
>

I will try to investigate it on the NetBSD kernel level.

Re: [PATCH v5 0/3] tests/vm: netbsd autoinstall, with IPv6 disabled

[Kamil Rytarowski]

On 21.10.2019 15:47, Alex Bennée wrote:
>
> Eduardo Habkost  writes:
>
>> I'm numbering this series v5 because it's a new version of the
>> patch sent by Gerd, at:
>>
>>   Date: Mon, 17 Jun 2019 06:38:56 +0200
>>   Message-Id: <20190617043858.8290-10-kra...@redhat.com>
>>   Subject: [PATCH v4 09/11] tests/vm: netbsd autoinstall, using serial
>>   console
>
> Queued to testing/next, thanks. I've made the changes Thomas suggested
> in his review.
>
>>
>> Changes v4 -> v5:
>> * Rebase to latest qemu.git master
>> * Disable IPv6 by default (see
>>   https://lore.kernel.org/qemu-devel/20191017225548.gl4...@habkost.net/ for 
>> context)
>>
>> Eduardo Habkost (2):
>>   tests/vm: Let subclasses disable IPv6
>>   tests/vm/netbsd: Disable IPv6
>>

I'm going to report this one to other NetBSD developers and hopefully
the problem will go away in future.

I have faced such problems in the past myself but didn't know what and
who is culprit.

>> Gerd Hoffmann (1):
>>   tests/vm: netbsd autoinstall, using serial console
>>
>>  tests/vm/basevm.py |   5 +-
>>  tests/vm/netbsd| 196 ++---
>>  2 files changed, 190 insertions(+), 11 deletions(-)
>
>
> --
> Alex Bennée
>

Re: [PATCH v2] Do not use %m in common code to print error messages

[Kamil Rytarowski]

On 18.10.2019 15:42, Stefano Garzarella wrote:
> On Fri, Oct 18, 2019 at 03:07:16PM +0200, Thomas Huth wrote:
>> The %m format specifier is an extension from glibc - and when compiling
>> QEMU for NetBSD, the compiler correctly complains, e.g.:
>>
>> /home/qemu/qemu-test.ELjfrQ/src/util/main-loop.c: In function 
>> 'sigfd_handler':
>> /home/qemu/qemu-test.ELjfrQ/src/util/main-loop.c:64:13: warning: %m is only
>>  allowed in syslog(3) like functions [-Wformat=]
>>  printf("read from sigfd returned %zd: %m\n", len);
>>  ^
>> Let's use g_strerror() here instead, which is an easy-to-use wrapper
>> around the thread-safe strerror_r() function.
>>
>> While we're at it, also convert the "printf()" in main-loop.c into
>> the preferred "error_report()".
>>
>> Signed-off-by: Thomas Huth 
>> ---
>>  v2: Do not try to g_free() the strings
>>
>>  hw/misc/tmp421.c | 4 ++--
>>  util/main-loop.c | 3 ++-
>>  util/systemd.c   | 4 ++--
>>  3 files changed, 6 insertions(+), 5 deletions(-)
>
> There are many uses of %m also in hw/vfio/ but that's Linux stuff.
> Should we change those too or it doesn't matter since it never really
> compiled on NetBSD?
>

It's a gnu (glibc) extension and linux can use alternative libc
implementations. Probably most of them capable to host qemu use %m.

> Anyway, this patch LGTM:
> Reviewed-by: Stefano Garzarella 
>
> Thanks,
> Stefano
>
>>
>> diff --git a/hw/misc/tmp421.c b/hw/misc/tmp421.c
>> index 9f044705fa..c0bc150bca 100644
>> --- a/hw/misc/tmp421.c
>> +++ b/hw/misc/tmp421.c
>> @@ -120,7 +120,7 @@ static void tmp421_get_temperature(Object *obj, Visitor 
>> *v, const char *name,
>>  int tempid;
>>
>>  if (sscanf(name, "temperature%d", &tempid) != 1) {
>> -error_setg(errp, "error reading %s: %m", name);
>> +error_setg(errp, "error reading %s: %s", name, g_strerror(errno));
>>  return;
>>  }
>>
>> @@ -160,7 +160,7 @@ static void tmp421_set_temperature(Object *obj, Visitor 
>> *v, const char *name,
>>  }
>>
>>  if (sscanf(name, "temperature%d", &tempid) != 1) {
>> -error_setg(errp, "error reading %s: %m", name);
>> +error_setg(errp, "error reading %s: %s", name, g_strerror(errno));
>>  return;
>>  }
>>
>> diff --git a/util/main-loop.c b/util/main-loop.c
>> index e3eaa55866..eda63fe4e0 100644
>> --- a/util/main-loop.c
>> +++ b/util/main-loop.c
>> @@ -61,7 +61,8 @@ static void sigfd_handler(void *opaque)
>>  }
>>
>>  if (len != sizeof(info)) {
>> -printf("read from sigfd returned %zd: %m\n", len);
>> +error_report("read from sigfd returned %zd: %s", len,
>> + g_strerror(errno));
>>  return;
>>  }
>>
>> diff --git a/util/systemd.c b/util/systemd.c
>> index d22e86c707..1dd0367d9a 100644
>> --- a/util/systemd.c
>> +++ b/util/systemd.c
>> @@ -60,8 +60,8 @@ unsigned int check_socket_activation(void)
>>   * and we should exit.
>>   */
>>  error_report("Socket activation failed: "
>> - "invalid file descriptor fd = %d: %m",
>> - fd);
>> + "invalid file descriptor fd = %d: %s",
>> + fd, g_strerror(errno));
>>  exit(EXIT_FAILURE);
>>  }
>>  }
>> --
>> 2.18.1
>>
>>
>

Re: Python 2 and test/vm/netbsd

[Kamil Rytarowski]

On 16.10.2019 08:11, Thomas Huth wrote:

On 16/10/2019 05.00, Eduardo Habkost wrote:

On Tue, Sep 17, 2019 at 08:31:40PM -0300, Eduardo Habkost wrote:

On Mon, Jul 01, 2019 at 07:25:27PM -0300, Eduardo Habkost wrote:

On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:

[...]

The configure check also spits out deprecation warnings for
the NetBSD/FreeBSD/OpenBSD tests/vm configurations. It would be nice
to get those updated.


CCing the test/vm maintainers.

Fam, Alex, are you able to fix this and create new BSD VM images
with Python 3 available?  I thought the VM image configurations
were stored in the source tree, but they are downloaded from
download.patchew.org.


Fam, Alex, can you help us on this?  Python 2 won't be supported
anymore, so we need the VM images to be updated.


Anyone?

I'm about to submit patches to remove Python 2 support, and this
will break tests/vm/netbsd.

I'm powerless to fix this issue, because the netbsd image is
hosted at download.patchew.org.


Gerd had a patch to convert the netbsd VM script to ad hoc image
creation, too:

https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg04459.html

But there was a regression with the serial port between QEMU v3.0 and
v4.x, so it was not included:

https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg06784.html

I guess someone™ needs to bisect that regression, so we can fix that bug
and finally include Gerd's patch...

  Thomas



Is this a regression in qemu? How to reproduce the problem? "make
vm-build-netbsd V=1" ?

I can have a look but I need to know exact specifics of the problem.

Re: [Qemu-devel] [PATCH 1/5] tests/acceptance: Add test that runs NetBSD installer on PRep/40p

[Kamil Rytarowski]

On 05.07.2019 20:00, Philippe Mathieu-Daudé wrote:
> Hi Kamil,
> 
> On 6/27/19 11:47 PM, Kamil Rytarowski wrote:
>> On 27.06.2019 13:01, Philippe Mathieu-Daudé wrote:
>>> +bios_url = ('ftp://ftp.boulder.ibm.com/rs6000/firmware/'
>>> +'7020-40p/P12H0456.IMG')
>>> +bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
>>> +bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
>>> +drive_url = ('https://ftp.netbsd.org/pub/NetBSD/NetBSD-archive/'
>>> + 'NetBSD-4.0/prep/installation/floppy/generic_com0.fs')
>>> +drive_hash = 'dbcfc09912e71bd5f0d82c7c1ee43082fb596ceb'
>>> +drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash)
>>> +
>>> +self.vm.set_machine('40p')
>>> +self.vm.set_console()
>>> +self.vm.add_args('-bios', bios_path,
>>> + '-fda', drive_path)
>>> +self.vm.launch()
>>> +os_banner = 'NetBSD 4.0 (GENERIC) #0: Sun Dec 16 00:49:40 PST 2007'
>>
>> Is there a specific reason to use NetBSD 4.0? It's a very old release,
>> the newest one is 8.1.
> 
> The goal of these integration tests is to verify a specific
> configuration that worked in the past still works with today codebase.
> 
> It is particularly useful for machines that have not a big quantity of
> users.
> 
> This test does not intent to test NetBSD, but that the PReP/40p machine
> is still working, as it used to work in the post saved in the commit
> description:
> 
> http://mail-index.netbsd.org/port-prep/2017/04/11/msg000112.html
> 
> So this test is useful to avoid the PReP machine code to bitrot.
> 
> I'd like to know what other from the QEMU community think/expect about
> these tests.
> 

This is understood and appreciated as NetBSD still formally supports the
prep port.

> FWIW I tested newer versions and they don't boot.
> 

This is what I wanted to know, whether this was an accident in using an
older version or not. I will try to report it to other developers and
maybe someone could debug and fix it.

> Regards,
> 
> Phil.
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v4 09/11] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

On 05.07.2019 12:47, Philippe Mathieu-Daudé wrote:
> On 7/5/19 12:43 PM, Kamil Rytarowski wrote:
>> On 05.07.2019 12:24, Alex Bennée wrote:
>>>
>>> Gerd Hoffmann  writes:
>>>
>>>> Instead of fetching the prebuilt image from patchew download the install
>>>> iso and prepare the image locally.  Install to disk, using the serial
>>>> console.  Create qemu user, configure ssh login.  Install packages
>>>> needed for qemu builds.
>>>
>>> I've had to drop this from my v3 PR as Peter was seeing a very
>>> slow/hanging install when running his merge tests. I've tried to
>>> reproduce and I see it stall while installing packages but nowhere near
>>> the delay Peter has seen.
>>>
>>> Any pointers on how to debug gratefully received.
>>>
>>
>> Does it use virtio? There were performance issues with virio disk device.
> 
> Yes, virtio-blk, you can see how the VM is started looking at
> tests/vm/basevm.py:
> 
>  "-drive",
>  "file=%s,if=none,id=%s,cache=writeback,format=raw" % (tarfile, name),
> 
>  "-device",
>  "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)
> 

Try to switch to a non-virtio option and retry.



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v4 09/11] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

On 05.07.2019 12:24, Alex Bennée wrote:
> 
> Gerd Hoffmann  writes:
> 
>> Instead of fetching the prebuilt image from patchew download the install
>> iso and prepare the image locally.  Install to disk, using the serial
>> console.  Create qemu user, configure ssh login.  Install packages
>> needed for qemu builds.
> 
> I've had to drop this from my v3 PR as Peter was seeing a very
> slow/hanging install when running his merge tests. I've tried to
> reproduce and I see it stall while installing packages but nowhere near
> the delay Peter has seen.
> 
> Any pointers on how to debug gratefully received.
> 

Does it use virtio? There were performance issues with virio disk device.

>>
>> Signed-off-by: Gerd Hoffmann 
>> Reviewed-by: Kamil Rytarowski 
>> Tested-by: Thomas Huth 
>> ---
>>  tests/vm/netbsd | 187 +---
>>  1 file changed, 177 insertions(+), 10 deletions(-)
>>
>> diff --git a/tests/vm/netbsd b/tests/vm/netbsd
>> index 4c6624ea5ed5..be59a2c1da1d 100755
>> --- a/tests/vm/netbsd
>> +++ b/tests/vm/netbsd
>> @@ -2,10 +2,11 @@
>>  #
>>  # NetBSD VM image
>>  #
>> -# Copyright 2017 Red Hat Inc.
>> +# Copyright 2017-2019 Red Hat Inc.
>>  #
>>  # Authors:
>>  #  Fam Zheng 
>> +#  Gerd Hoffmann 
>>  #
>>  # This code is licensed under the GPL version 2 or later.  See
>>  # the COPYING file in the top-level directory.
>> @@ -13,32 +14,198 @@
>>
>>  import os
>>  import sys
>> +import time
>>  import subprocess
>>  import basevm
>>
>>  class NetBSDVM(basevm.BaseVM):
>>  name = "netbsd"
>>  arch = "x86_64"
>> +
>> +link = 
>> "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso";
>> +size = "20G"
>> +pkgs = [
>> +# tools
>> +"git-base",
>> +"pkgconf",
>> +"xz",
>> +"python37",
>> +
>> +# gnu tools
>> +"bash",
>> +"gmake",
>> +"gsed",
>> +"flex", "bison",
>> +
>> +# libs: crypto
>> +"gnutls",
>> +
>> +# libs: images
>> +"jpeg",
>> +"png",
>> +
>> +# libs: ui
>> +"SDL2",
>> +"gtk3+",
>> +"libxkbcommon",
>> +]
>> +
>>  BUILD_SCRIPT = """
>>  set -e;
>> -rm -rf /var/tmp/qemu-test.*
>> -cd $(mktemp -d /var/tmp/qemu-test.XX);
>> +rm -rf /home/qemu/qemu-test.*
>> +cd $(mktemp -d /home/qemu/qemu-test.XX);
>> +mkdir src build; cd src;
>>  tar -xf /dev/rld1a;
>> -./configure --python=python2.7 {configure_opts};
>> +cd ../build
>> +../src/configure --python=python3.7 --disable-opengl 
>> {configure_opts};
>>  gmake --output-sync -j{jobs} {target} {verbose};
>>  """
>> +poweroff = "/sbin/poweroff"
>>
>>  def build_image(self, img):
>> -cimg = 
>> self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz";,
>> - 
>> sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
>> -img_tmp_xz = img + ".tmp.xz"
>> +cimg = self._download_with_cache(self.link)
>>  img_tmp = img + ".tmp"
>> -sys.stderr.write("Extracting the image...\n")
>> -subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
>> -subprocess.check_call(["xz", "-dvf", img_tmp_xz])
>> +iso = img + ".install.iso"
>> +
>> +self.print_step("Preparing iso and disk image")
>> +subprocess.check_call(["cp", "-f", cimg, iso])
>> +subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
>> +   img_tmp, self.size])
>> +
>> +self.print_step("Booting installer")
>> +self.boot(img_tmp, extra_args = [
>> +"-bios", "pc-bios/bios-256k.bin",
>> +"-machine", "graphics=off

Re: [Qemu-devel] [PATCH 1/5] tests/acceptance: Add test that runs NetBSD installer on PRep/40p

[Kamil Rytarowski]

On 27.06.2019 13:01, Philippe Mathieu-Daudé wrote:
> +bios_url = ('ftp://ftp.boulder.ibm.com/rs6000/firmware/'
> +'7020-40p/P12H0456.IMG')
> +bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
> +bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
> +drive_url = ('https://ftp.netbsd.org/pub/NetBSD/NetBSD-archive/'
> + 'NetBSD-4.0/prep/installation/floppy/generic_com0.fs')
> +drive_hash = 'dbcfc09912e71bd5f0d82c7c1ee43082fb596ceb'
> +drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash)
> +
> +self.vm.set_machine('40p')
> +self.vm.set_console()
> +self.vm.add_args('-bios', bios_path,
> + '-fda', drive_path)
> +self.vm.launch()
> +os_banner = 'NetBSD 4.0 (GENERIC) #0: Sun Dec 16 00:49:40 PST 2007'

Is there a specific reason to use NetBSD 4.0? It's a very old release,
the newest one is 8.1.



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v3 00/14] tests/vm: serial console autoinstall, misc fixes.

[Kamil Rytarowski]

On 12.06.2019 19:35, Alex Bennée wrote:
> 
> Alex Bennée  writes:
> 
>> Gerd Hoffmann  writes:
>>
>>> This patch series changes the way virtual machines for test builds are
>>> managed.  They are created locally on the developer machine now.  The
>>> installer is booted on the serial console and the scripts walks through
>>> the dialogs to install and configure the guest.
>>>
>>> That takes the download.patchew.org server out of the loop and makes it
>>> alot easier to tweak the guest images (adding build dependencies for
>>> example).
>>>
>>> The install scripts take care to apply host proxy settings (from *_proxy
>>> environment variables) to the guest, so any package downloads will be
>>> routed through the proxy and can be cached that way.  This also makes
>>> them work behind strict firewalls.
>>>
>>> There are also a bunch of smaller tweaks for tests/vm to fix issues I
>>> was struggling with.  See commit messages of individual patches for
>>> details.
>>
>> Queued to testing/next, thanks.
>>
>> One of the machines I'm testing on seems to have problems with getting
>> the installer working over the serial link but it works on my main dev
>> box and others have it working as well so I suspect it might be a local
>> problem.
> 
> 
> OK I had to drop this series due to persistent problems with the serial
> link on one of the boxes Peter uses to do his merge testing. Some of it
> seems to be QEMU specific (Ubuntu 18.04 system qemu just hangs) but I
> suspect there is also an issue with BSDs and our serial emulation
> generally.
> 
> --
> Alex Bennée
> 

What version of NetBSD did you test? There was recently a short lived
regression in curses(3) that affected sysinst(8) (installer) in HEAD,
but it was fixed.



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v2 11/13] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

On 10.05.2019 12:46, Gerd Hoffmann wrote:
> Instead of fetching the prebuilt image from patchew download the install
> iso and prepare the image locally.  Install to disk, using the serial
> console.  Create qemu user, configure ssh login.  Install packages
> needed for qemu builds.
> 
> Signed-off-by: Gerd Hoffmann 

Reviewed-by: Kamil Rytarowski 

> ---
>  tests/vm/netbsd | 187 +---
>  1 file changed, 177 insertions(+), 10 deletions(-)
> 
> diff --git a/tests/vm/netbsd b/tests/vm/netbsd
> index 4c6624ea5ed5..6dbfc1b0fbe3 100755
> --- a/tests/vm/netbsd
> +++ b/tests/vm/netbsd
> @@ -2,10 +2,11 @@
>  #
>  # NetBSD VM image
>  #
> -# Copyright 2017 Red Hat Inc.
> +# Copyright 2017-2019 Red Hat Inc.
>  #
>  # Authors:
>  #  Fam Zheng 
> +#  Gerd Hoffmann 
>  #
>  # This code is licensed under the GPL version 2 or later.  See
>  # the COPYING file in the top-level directory.
> @@ -13,32 +14,198 @@
>  
>  import os
>  import sys
> +import time
>  import subprocess
>  import basevm
>  
>  class NetBSDVM(basevm.BaseVM):
>  name = "netbsd"
>  arch = "x86_64"
> +
> +link = 
> "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso";
> +size = "20G"
> +pkgs = [
> +# tools
> +"git-base",
> +"pkgconf",
> +"xz",
> +"python37",
> +
> +# gnu tools
> +"bash",
> +"gmake",
> +"gsed",
> +"flex", "bison",
> +
> +# libs: crypto
> +"gnutls",
> +
> +# libs: images
> +"jpeg",
> +"png",
> +
> + # libs: ui
> +"SDL2",
> +"gtk3+",
> +"libxkbcommon",
> +]
> +
>  BUILD_SCRIPT = """
>  set -e;
> -rm -rf /var/tmp/qemu-test.*
> -cd $(mktemp -d /var/tmp/qemu-test.XX);
> +rm -rf /home/qemu/qemu-test.*
> +cd $(mktemp -d /home/qemu/qemu-test.XX);
> +mkdir src build; cd src;
>  tar -xf /dev/rld1a;
> -./configure --python=python2.7 {configure_opts};
> +cd ../build
> +../src/configure --python=python3.7 --disable-opengl 
> {configure_opts};
>  gmake --output-sync -j{jobs} {target} {verbose};
>  """
> +poweroff = "/sbin/poweroff"
>  
>  def build_image(self, img):
> -cimg = 
> self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz";,
> - 
> sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
> -img_tmp_xz = img + ".tmp.xz"
> +cimg = self._download_with_cache(self.link)
>  img_tmp = img + ".tmp"
> -sys.stderr.write("Extracting the image...\n")
> -subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
> -subprocess.check_call(["xz", "-dvf", img_tmp_xz])
> +iso = img + ".install.iso"
> +
> +self.print_step("Preparing iso and disk image")
> +subprocess.check_call(["cp", "-f", cimg, iso])
> +subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
> +   img_tmp, self.size])
> +
> +self.print_step("Booting installer")
> +self.boot(img_tmp, extra_args = [
> +"-device", "VGA",
> +"-machine", "graphics=off",
> +"-cdrom", iso
> +])
> +self.console_init()
> +self.console_wait("Primary Bootstrap")
> +
> +# serial console boot menu output doesn't work for some
> +# reason, so we have to fly blind ...
> +for char in list("5consdev com0\n"):
> +time.sleep(0.2)
> +self.console_send(char)
> +self.console_wait("")
> +self.console_wait_send("> ", "boot\n")
> +
> +self.console_wait_send("Terminal type","xterm\n")
> +self.console_wait_send("a: Installation messages", "a\n")
> +self.console_wait_send("b: US-English","b\n")
> +self.console_wait_send("a: Install NetBSD","a\

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Kamil Rytarowski]

On 09.05.2019 15:57, Thomas Huth wrote:
> On 09/05/2019 15.50, Gerd Hoffmann wrote:
>>   Hi,
>>
 Do we have accelerator support for the BSDs?  A "make check" for a full
 build takes ages, and I suspect tcg being used is part of the problem.
 I did my tests using "TARGET_LIST=x86_64-softmmu" because of that.
>>>
>>> I think they should be running with "--enable-kvm".
>>
>> The images themself yes, but the tests running *inside* (on make check) 
>> don't.
>
> No, we don't have accelerator support for *BSD, as far as I know.

As mentioned in the other mail, KVM-style?

NetBSD does support HAXM (--accel hax) and in a downstream copy NVMM
(-accel nvmm).

http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge

http://blog.netbsd.org/tnf/entry/from_zero_to_nvmm

Once NVMM will stabilize we intend to submit it upstream.

There is no support for hardware assisted acceleration in qemu for any
other BSD.

> But we
> also do not run that much TCG tests during "make check" that you should
> see such a big difference here. And for me, the compilation step is
> already way slower than on the host, so I think the problem is likely
> something else...
>
>  Thomas
>

Re: [Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

On 09.05.2019 18:39, Richard Henderson wrote:
> On 5/8/19 11:47 PM, Gerd Hoffmann wrote:
>> So, from looking at the patch it seems you need two mappings of the same
>> page, one writable and one executable.
>>
>> Or, maybe it is also possible with one mapping which is writable first
>> when you fill it with code, then gets flipped over to executable when
>> you are done with the initialization and want use it.
>>
>> Is that correct?
> 
> That's certainly the way I read that patch.
> 
>> I suspect supporting that in tcg isn't exactly trivial.
> 
> It shouldn't be too hard, if required.  All of the writing to the buffer is
> isolated to a couple of inline functions.
> 
> I do wonder if using paxctl -m as a part of the build process for affected 
> *BSD
> isn't just as easy?
> 

paxctl(8) is available only for NetBSD.

paxctl(8) is the last resort solution and shall be avoided due to a
security risk.

Only few exceptions shall be allowed to switch mapping protection or
request RWX mappings (mainly JIT and a process running under a debugger).

> 
> r~
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Kamil Rytarowski]

On 08.05.2019 10:56, Gerd Hoffmann wrote:
> This patch series changes the way virtual machines for test builds are
> managed.  They are created locally on the developer machine now.  The
> installer is booted on the serial console and the scripts walks through
> the dialogs to install and configure the guest.
> 
> That takes the download.patchew.org server out of the loop and makes it
> alot easier to tweak the guest images (adding build dependencies for
> example).
> 
> The install scripts take care to apply host proxy settings (from *_proxy
> environment variables) to the guest, so any package downloads will be
> routed through the proxy and can be cached that way.  This also makes
> them work behind strict firewalls.
> 
> There are also a bunch of smaller tweaks for tests/vm to fix issues I
> was struggling with.  See commit messages of individual patches for
> details.
> 
> Known issue:  NetBSD package install is not working for me right now.
> It did work a while ago.  Not sure what is going on here.
> 

Error log? What is the command? pkgin install?

> Do we have accelerator support for the BSDs?

KVM-style?

NetBSD does support HAXM (--accel hax) and in a downstream copy NVMM
(-accel nvmm).

http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge

http://blog.netbsd.org/tnf/entry/from_zero_to_nvmm

Once NVMM will stabilize we intend to submit it upstream.

There is no support for hardware assisted acceleration in qemu for any
other BSD.

>  A "make check" for a full
> build takes ages, and I suspect tcg being used is part of the problem.
> I did my tests using "TARGET_LIST=x86_64-softmmu" because of that.
> 
> Gerd Hoffmann (13):
>   scripts: use git archive in archive-source
>   tests/vm: send proxy environment variables over ssh
>   tests/vm: send locale environment variables over ssh
>   tests/vm: use ssh with pty unconditionally
>   tests/vm: run test builds on snapshot
>   tests/vm: add vm-boot-{ssh,serial}- targets
>   tests/vm: add DEBUG=1 to help text
>   tests/vm: serial console support helpers
>   tests/vm: openbsd autoinstall, using serial console
>   tests/vm: freebsd autoinstall, using serial console
>   tests/vm: netbsd autoinstall, using serial console
>   tests/vm: fedora autoinstall, using serial console
>   tests/vm: ubuntu.i386: apt proxy setup
> 
>  tests/vm/basevm.py| 125 ++---
>  scripts/archive-source.sh |  72 +++
>  tests/vm/Makefile.include |  25 -
>  tests/vm/fedora   | 187 ++
>  tests/vm/freebsd  | 172 +--
>  tests/vm/netbsd   | 178 ++--
>  tests/vm/openbsd  | 150 +++---
>  tests/vm/ubuntu.i386  |   4 +
>  8 files changed, 830 insertions(+), 83 deletions(-)
>  create mode 100755 tests/vm/fedora
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

On 09.05.2019 08:47, Gerd Hoffmann wrote:
>   Hi,
> 
>> I recommend to add one extra step into generated image:
>>
>> echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf
> 
> Done.
> 

Thanks! Once there will be PaX MPROTECT support in qemu, we can and
should drop it.

>> Alternatively (and preferably) enhance qemu to handle RWX allocation for
>> JIT on NetBSD.
>>
>> Example in libffi.
>>
>> https://github.com/libffi/libffi/commit/2bfcd29955c02b67fa10a68cc4200f6838181e0f
> 
> So, from looking at the patch it seems you need two mappings of the same
> page, one writable and one executable.
> > Or, maybe it is also possible with one mapping which is writable first
> when you fill it with code, then gets flipped over to executable when
> you are done with the initialization and want use it.
> 
> Is that correct?
> 

If we need RWX in the same time we need double mapping (RW and RX). This
has been done in libffi.

If we can switch between RW and RX it's sufficient to use a single
mapping, however there is need to allocate a memory region with mmap(2)
using the PROT_MPROTECT() macro.

Example from the LLVM code:

https://github.com/llvm-mirror/llvm/blob/90dd07f5c5946a3d9d6861effe3291620c88c06f/lib/Support/Unix/Memory.inc#L99

> I suspect supporting that in tcg isn't exactly trivial.
> Does OpenBSD work the same way btw?
> 

No. OpenBSD does not implement escape API.

There is need to pass "-z wxneeded" to the linker flags, but I'm not the
right person to test this or write a patch.

https://man.openbsd.org/ld

FreeBSD does not support W^X.

> cheers,
>   Gerd
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

On 08.05.2019 10:56, Gerd Hoffmann wrote:
> Instead of fetching the prebuilt image from patchew download the install
> iso and prepare the image locally.  Install to disk, using the serial
> console.  Create qemu user, configure ssh login.  Install packages
> needed for qemu builds.
> 

I recommend to add one extra step into generated image:

echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf

Alternatively (and preferably) enhance qemu to handle RWX allocation for
JIT on NetBSD.

Example in libffi.

https://github.com/libffi/libffi/commit/2bfcd29955c02b67fa10a68cc4200f6838181e0f

> Signed-off-by: Gerd Hoffmann 
> ---
>  tests/vm/netbsd | 178 +---
>  1 file changed, 169 insertions(+), 9 deletions(-)
> 
> diff --git a/tests/vm/netbsd b/tests/vm/netbsd
> index 4c6624ea5ed5..eaf0ae21db42 100755
> --- a/tests/vm/netbsd
> +++ b/tests/vm/netbsd
> @@ -13,32 +13,192 @@
>  
>  import os
>  import sys
> +import time
>  import subprocess
>  import basevm
>  
>  class NetBSDVM(basevm.BaseVM):
>  name = "netbsd"
>  arch = "x86_64"
> +
> +link = 
> "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso";
> +size = "20G"
> +pkgs = [
> +# tools
> +"git",
> +"pkgconf",
> +"bzip2", "xz",
> +
> +# gnu tools
> +"bash",
> +"gmake",
> +"gsed",
> +"flex", "bison",
> +
> +# libs: crypto
> +"gnutls",
> +
> +# libs: images
> +"jpeg",
> +"png",
> +
> + # libs: ui
> +"SDL2",
> +"gtk3+",
> +"libxkbcommon",
> +]
> +
>  BUILD_SCRIPT = """
>  set -e;
> -rm -rf /var/tmp/qemu-test.*
> -cd $(mktemp -d /var/tmp/qemu-test.XX);
> +rm -rf /home/qemu/qemu-test.*
> +cd $(mktemp -d /home/qemu/qemu-test.XX);
> +mkdir src build; cd src;
>  tar -xf /dev/rld1a;
> -./configure --python=python2.7 {configure_opts};
> +cd ../build
> +../src/configure --python=python2.7 --disable-opengl 
> {configure_opts};
>  gmake --output-sync -j{jobs} {target} {verbose};
>  """
>  
>  def build_image(self, img):
> -cimg = 
> self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz";,
> - 
> sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
> -img_tmp_xz = img + ".tmp.xz"
> +cimg = self._download_with_cache(self.link)
>  img_tmp = img + ".tmp"
> -sys.stderr.write("Extracting the image...\n")
> -subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
> -subprocess.check_call(["xz", "-dvf", img_tmp_xz])
> +iso = img + ".install.iso"
> +
> +self.print_step("Preparing iso and disk image")
> +subprocess.check_call(["cp", "-f", cimg, iso])
> +subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
> +   img_tmp, self.size])
> +
> +self.print_step("Booting installer")
> +self.boot(img_tmp, extra_args = [
> +"-device", "VGA",
> +"-machine", "graphics=off",
> +"-cdrom", iso
> +])
> +self.console_init()
> +self.console_wait("Primary Bootstrap")
> +
> +# serial console boot menu output doesn't work for some
> +# reason, so we have to fly blind ...
> +for char in list("5consdev com0\n"):
> +time.sleep(0.2)
> +self.console_send(char)
> +self.console_wait("")
> +self.console_wait_send("> ", "boot\n")
> +
> +self.console_wait_send("Terminal type","xterm\n")
> +self.console_wait_send("a: Installation messages", "a\n")
> +self.console_wait_send("b: US-English","b\n")
> +self.console_wait_send("a: Install NetBSD","a\n")
> +self.console_wait("Shall we continue?")
> +self.console_wait_send("b: Yes",   "b\n")
> +
> +self.console_wait_send("a: ld0",   "a\n")
> +self.console_wait_send("a: This is the correct",   "a\n")
> +self.console_wait_send("b: Use the entire disk",   "b\n")
> +self.console_wait("NetBSD bootcode")
> +self.console_wait_send("a: Yes",   "a\n")
> +self.console_wait_send("b: Use existing part", "b\n")
> +self.console_wait_send("x: Partition sizes ok","x\n")
> +self.console_wait_send("for your NetBSD disk", "\n")
> +self.console_wait("Shall we continue?")
> +self.console_wait_send("b: Yes",   "b\n")
> +
> +self.console_wait_send("b: Use serial port com0",  "b\n")
> +self.console_wait_send("f: Set serial baud rate",  "f\n")
> +self.console_wait_send("a: 9600",  "a\n")
> +self.console_wait_send("x: Exit",

Re: [Qemu-devel] Update *BSD images with gnu-sed and bash

[Kamil Rytarowski]

On 08.05.2019 10:07, Thomas Huth wrote:
> On 08/05/2019 09.06, Kamil Rytarowski wrote:
>> On 06.05.2019 12:12, Thomas Huth wrote:
> [...]
>>>  Kamil,
>>>
>>> could you maybe help with the NetBSD image and the tests/vm/netbsd script?
>>>
>>
>> Please be more specific what am I expected to do.
> 
> We have some VMs (including NetBSD) available that are used during
> Peter's regression tests when somebody sends him a PULL requests. You
> can run them also locally with:
> 
>  make BUILD_TARGET=check vm-build-netbsd
> 
> From time to time, we've got to update these images, either to a newer
> version or to add some missing packages (like bash and gnu-sed in this
> case).

The process has been documented on wiki.

https://wiki.qemu.org/Hosts/BSD#NetBSD

> However, many people (including me) don't have a clue about the various
> *BSD flavours, so also no clue about how to update these images easily.
> That's why I was hoping you could help here.
> 
> But looks like Gerd is already working on a way to generate these images
> in a more automated way, so let's hope that he'll find some spare time
> to finish that work soon.
> 

I recommend to upgrade to 8.0.

One extra step is to disable PaX MPROTECT (tcg violates W^X):

This should be addressed in qemu with an extension flag to
mmap(2)/mremap(2). I still have this on my TODO list.

sysctl -w security.pax.mprotect.enabled=1

NetBSD 9.0 will be released sooner than later and soon after that
NetBSD-7.x will be EOL.

>  Thomas
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] Update *BSD images with gnu-sed and bash

[Kamil Rytarowski]

On 06.05.2019 12:12, Thomas Huth wrote:
> On 26/04/2019 18.43, Laszlo Ersek wrote:
>> On 04/25/19 08:00, Thomas Huth wrote:
>>> On 24/04/2019 23.29, Wainer dos Santos Moschetta wrote:
 Hello Thomas,

 On 04/24/2019 07:37 AM, Thomas Huth wrote:
> We are going to enable the qemu-iotests during "make check" again,
> and for running the iotests, we need bash and gnu-sed.
 [...]
 It needs to get the Freebsd image [1] updated too, in order to `make
 BUILD_TARGET=check vm-build-freebsd` passes. Here it failed with:
 ---
 env: bash: No such file or directory
 gmake: *** [/var/tmp/qemu-test.6OlDFH/tests/Makefile.include:1101:
 check-tests/qemu-iotests-quick.sh] Error 1
 gmake: *** Waiting for unfinished jobs
 ---

 I'm not sure about the netbsd and openbsd images, they might need bash
 and gnu-sed as well.
>>>
>>> D'oh! Does anybody know what are the correct steps to update these images?
>>
>> (1) make the OPENBSD subsystem maintainer care
>>
>> (2) update 
>>
>> (3) download the image from download.patchew.org, boot it and update it
>>
>> (4) upload the image to download.patchew.org
>>
>> (5) update the "tests/vm/openbsd" script in the QEMU tree in sync
>> (checksums, commands etc)
> 
>  Ed, Li-Wen,
> 
> any chance you could help with updating the FreeBSD image and the
> tests/vm/freebsd script, so that we get more test covarage here?
> 
>  Kamil,
> 
> could you maybe help with the NetBSD image and the tests/vm/netbsd script?
> 

Please be more specific what am I expected to do.

>  Brad,
> 
> could you please help with the OpenBSD image and the tests/vm/openbsd
> script?
> 
> I think it would also be good to update the images to the latest
> released versions, too...
> 
> And IIRC, there was also a thread recently that it would also be good to
> have the "xz" tool in all of these images, too...
> 
>  Thanks,
>   Thomas
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCHv3 1/2] ui/curses: Do not assume wchar_t contains unicode

[Kamil Rytarowski]

On 27.04.2019 19:57, Samuel Thibault wrote:
> Kamil Rytarowski, le sam. 27 avril 2019 19:36:40 +0200, a ecrit:
>> On 27.04.2019 18:30, Samuel Thibault wrote:
>>> E.g. BSD and Solaris even use locale-specific encoding there.
>>>
>>> We thus have to go through the native multibyte representation and use
>>> mbtowc/wctomb to make a proper conversion.
>>>
>>> Signed-off-by: Samuel Thibault 
>>
>> Both patches work for me on NetBSD/amd64 8.99.37 for qemu-system-x86_64.
>> Borders are printed correctly.
>>
>> Regarding the patch I'm not sure whether to use MB_LEN_MAX or MB_CUR_MAX?
> 
> I don't know if qemu can afford VLA?
> 

It's better to avoid VLA and pick MB_LEN_MAX.

>> I'm also unsure whether to reset conversion state between a multibyte
>> character and wide character, with: `mbtowc(NULL, 0, 0);`. It's
>> recommended to use in code examples examples. I think it doesn't make
>> any harm to use it.
> 
> Mmm, better yet, we should actually use mbrtowc and wcrtomb. I have
> fixed this in my tree.
> 

This is even better.

>> I'm not sure if this is related, but "qemu-system-hppa -curses" is
>> broken for me. I didn't use it in the past as it just recently acquired
>> NetBSD guest support.
>>
>> (lldb) bt
>> libcurses.so.7`mvwadd_wchnstr(win=0x, y=,
>> x=, wchstr=0x7f7fe020, n=0) at add_wchstr.c:123
>>   * frame #2: 0x0078629e
>> qemu-system-hppa`curses_update(dcl=0x7f7ff7bd8bc0, x=0, y=0, w=79,
>> h=24) at curses.c:86:9
>> frame #3: 0x00753dae
>> qemu-system-hppa`dpy_text_update(con=0x7f7ff7bae580, x=0, y=0, w=79,
> 
>> (lldb) p screenpad
>> (WINDOW *) $2 = 0x
> 
> I don't think this is related at all, screenpad management is another
> matter.
> 

OK! I will treat it as an independent issue and try to address it.

> Samuel
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCHv3 1/2] ui/curses: Do not assume wchar_t contains unicode

[Kamil Rytarowski]

On 27.04.2019 18:30, Samuel Thibault wrote:
> E.g. BSD and Solaris even use locale-specific encoding there.
> 
> We thus have to go through the native multibyte representation and use
> mbtowc/wctomb to make a proper conversion.
> 
> Signed-off-by: Samuel Thibault 

Both patches work for me on NetBSD/amd64 8.99.37 for qemu-system-x86_64.
Borders are printed correctly.

Regarding the patch I'm not sure whether to use MB_LEN_MAX or MB_CUR_MAX?

I'm also unsure whether to reset conversion state between a multibyte
character and wide character, with: `mbtowc(NULL, 0, 0);`. It's
recommended to use in code examples examples. I think it doesn't make
any harm to use it.


I'm not sure if this is related, but "qemu-system-hppa -curses" is
broken for me. I didn't use it in the past as it just recently acquired
NetBSD guest support.

(lldb) bt
* thread #1, stop reason = signal SIGSEGV
frame #0: 0x7f7ff6c1fb98
libcurses.so.7`wmove(win=0x, y=0, x=0) at move.c:71
frame #1: 0x7f7ff6c0ca9b
libcurses.so.7`mvwadd_wchnstr(win=0x, y=,
x=, wchstr=0x7f7fe020, n=0) at add_wchstr.c:123
  * frame #2: 0x0078629e
qemu-system-hppa`curses_update(dcl=0x7f7ff7bd8bc0, x=0, y=0, w=79,
h=24) at curses.c:86:9
frame #3: 0x00753dae
qemu-system-hppa`dpy_text_update(con=0x7f7ff7bae580, x=0, y=0, w=79,
h=24) at console.c:1658:13
frame #4: 0x00758abe
qemu-system-hppa`text_console_update(opaque=0x7f7ff7bae580,
chardata=0x0118e490) at console.c:1264:9
frame #5: 0x00751ef8
qemu-system-hppa`graphic_hw_text_update(con=0x7f7ff7bae580,
chardata=0x0118c690) at console.c:389:9
frame #6: 0x00785bcb
qemu-system-hppa`curses_refresh(dcl=0x7f7ff7bd8bc0) at curses.c:273:5
frame #7: 0x00758248
qemu-system-hppa`dpy_refresh(s=0x7f7ff7bd8770) at console.c:1622:13
frame #8: 0x0075809d
qemu-system-hppa`gui_update(opaque=0x7f7ff7bd8770) at console.c:205:5
frame #9: 0x008d9f4d
qemu-system-hppa`timerlist_run_timers(timer_list=0x7f7ff7e57d20) at
qemu-timer.c:574:9
frame #10: 0x008da01d
qemu-system-hppa`qemu_clock_run_timers(type=QEMU_CLOCK_REALTIME) at
qemu-timer.c:588:12
frame #11: 0x008da4ea
qemu-system-hppa`qemu_clock_run_all_timers at qemu-timer.c:708:25
frame #12: 0x008da962
qemu-system-hppa`main_loop_wait(nonblocking=0) at main-loop.c:519:5
frame #13: 0x005570a4 qemu-system-hppa`main_loop at vl.c:1970:9
frame #14: 0x00551fa4 qemu-system-hppa`main(argc=2,
argv=0x7f7fe768, envp=0x7f7fe780) at vl.c:4604:5
frame #15: 0x0040e7ad qemu-system-hppa`___start + 280

(lldb) p screenpad
(WINDOW *) $2 = 0x

We pass NULL window argument to mvwadd_wchnstr(3) and crash. Can you
reproduce it locally?

I will try to investigate it.

> ---
>  ui/curses.c | 151 
>  1 file changed, 94 insertions(+), 57 deletions(-)
> 
> diff --git a/ui/curses.c b/ui/curses.c
> index fb63945188..395f9545e9 100644
> --- a/ui/curses.c
> +++ b/ui/curses.c
> @@ -400,65 +400,102 @@ static void curses_atexit(void)
>  endwin();
>  }
>  
> +/*
> + * In the following:
> + * - fch is the font glyph number
> + * - uch is the unicode value
> + * - wch is the wchar_t value (may not be unicode, e.g. on BSD/solaris)
> + * - mbch is the native local-dependent multibyte representation
> + */
> +
>  /* Setup wchar glyph for one UCS-2 char */
> -static void convert_ucs(int glyph, uint16_t ch, iconv_t conv)
> +static void convert_ucs(unsigned char fch, uint16_t uch, iconv_t conv)
>  {
> +char mbch[MB_CUR_MAX];
>  wchar_t wch;
> -char *pch, *pwch;
> -size_t sch, swch;
> -
> -pch = (char *) &ch;
> -pwch = (char *) &wch;
> -sch = sizeof(ch);
> -swch = sizeof(wch);
> +char *puch, *pmbch;
> +size_t such, smbch;
> +
> +puch = (char *) &uch;
> +pmbch = (char *) mbch;
> +such = sizeof(uch);
> +smbch = sizeof(mbch);
> +
> +if (iconv(conv, &puch, &such, &pmbch, &smbch) == (size_t) -1) {
> +fprintf(stderr, "Could not convert 0x%04x "
> +"from UCS-2 to a multibyte character: %s\n",
> +uch, strerror(errno));
> +return;
> +}
>  
> -if (iconv(conv, &pch, &sch, &pwch, &swch) == (size_t) -1) {
> -fprintf(stderr, "Could not convert 0x%04x from UCS-2 to WCHAR_T: 
> %s\n",
> -ch, strerror(errno));
> -} else {
> -vga_to_curses[glyph].chars[0] = wch;
> +if (mbtowc(&wch, mbch, sizeof(mbch) - smbch) == -1) {
> +fprintf(stderr, "Could not convert 0x%04x "
> +"from a multibyte character to wchar_t: %s\n",
> +uch, strerror(errno));
> +return;
>  }
> +vga_to_curses[fch].chars[0] = wch;
>  }
>  
>  /* Setup wchar glyph for one font character */
> -static 

Re: [Qemu-devel] [PATCH] target-i386: Enhance the stub for kvm_arch_get_supported_cpuid()

[Kamil Rytarowski]

On 25.02.2019 08:10, Paolo Bonzini wrote:
> I have replied already.
> 



Sorry, I don't have this mail in my mailbox for some reason.

The issue is gone with the following patch:

https://www.mail-archive.com/qemu-devel@nongnu.org/msg598417.html

Tested-by: Kamil Rytarowski 

Please merge with qemu. Thank you in advance!




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH] target-i386: Enhance the stub for kvm_arch_get_supported_cpuid()

[Kamil Rytarowski]

Ping.

On 21.02.2019 03:08, Kamil Rytarowski wrote:
> On 20.02.2019 18:29, Paolo Bonzini wrote:
>> On 20/02/19 12:59, Kamil Rytarowski wrote:
>>> Ping, still valid.
>>
>> Sorry, I missed your email.
>>
>>> On 15.02.2019 00:38, Kamil Rytarowski wrote:
>>>> I consider it as fragile hack and certainly not something to depend on.
>>>> Also in some circumstances of such code, especially "if (zero0)" we want
>>>> to enable disabled code under a debugger.
>>
>> That's a good objection, but certainly does not apply to KVM on NetBSD.
>>
> 
> There is KVM for Darwin (experimental and rather toy project) and it
> might be ported to NetBSD (I have actually forked it on GitHub
> recently), but I doubt that someone would enable KVM on any platform
> under a debugger this way and expect something to work.
> 
>>>> There were also kernel backdoors due to this optimization.
>>
>> Citation please?
>>
> 
> I saw an exploit for such case with a .txt writeup on ftp of grsecurity
> but that service seems to be gone (probably long time ago), so please
> defer discussion on it. If someone is interested to find it out, there
> are enough pointers to dig it (assuming that this is still possible).
> 
>>>> Requested cpu.i (hopefully correctly generated)
>>>>
>>>> http://netbsd.org/~kamil/qemu/cpu.i.bz2
>>
>> So, first thing first I can reproduce clang's behavior with this .i file
>> and also with this reduced test case.
>>
>> extern void f(void);
>> int i, j;
>> int main()
>> {
>> if (0  && i) f();
>> if (j  && 0) f();
>>}
>>
>> The first is eliminated but the second is not, just like in QEMU where
>> this works:
>>
>> if (kvm_enabled() && cpu->enable_pmu) {
>> KVMState *s = cs->kvm_state;
>>
>> *eax = kvm_arch_get_supported_cpuid(s, 0xA, count, R_EAX);
>> *ebx = kvm_arch_get_supported_cpuid(s, 0xA, count, R_EBX);
>> *ecx = kvm_arch_get_supported_cpuid(s, 0xA, count, R_ECX);
>> *edx = kvm_arch_get_supported_cpuid(s, 0xA, count, R_EDX);
>> } else if (hvf_enabled() && cpu->enable_pmu) {
>> *eax = hvf_get_supported_cpuid(0xA, count, R_EAX);
>> *ebx = hvf_get_supported_cpuid(0xA, count, R_EBX);
>> *ecx = hvf_get_supported_cpuid(0xA, count, R_ECX);
>> *edx = hvf_get_supported_cpuid(0xA, count, R_EDX);
>>
>> while this doesn't:
>>
>> if ((env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_INTEL_PT) &&
>> kvm_enabled()) {
>> KVMState *s = CPU(cpu)->kvm_state;
>> uint32_t eax_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_EAX);
>> uint32_t ebx_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_EBX);
>> uint32_t ecx_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_ECX);
>> uint32_t eax_1 = kvm_arch_get_supported_cpuid(s, 0x14, 1, R_EAX);
>> uint32_t ebx_1 = kvm_arch_get_supported_cpuid(s, 0x14, 1, R_EBX);
>>
>> But, that's okay, it's -O0 so we give clang a pass for that  Note that
>> clang does do the optimization even in more complex cases like
>>
>> extern _Bool f(void);
>> int main()
>> {
>> if (!0) return 0;
>> if (!f()) return 0;
>> }
>>
>> The problem is that there is a kvm-stub.c entry for that, and in fact
>> my compilation passes and the symbol is resolved correctly:
>>
>> $ nm target/i386/cpu.o |grep kvm_.*get_sup
>>  U kvm_arch_get_supported_cpuid
>> $ nm target/i386/kvm-stub.o|grep kvm_.*get_sup
>> 0030 T kvm_arch_get_supported_cpuid
>> $ nm qemu-system-x86_64 |grep kvm_.*get_sup
>> 0046eab0 T kvm_arch_get_supported_cpuid
>>
>> As expected, something much less obvious is going on for you, in
>> particular __OPTIMIZE__seems not to be working properly.  However,
>> that would also be very surprising.
>>
>> Please:
>>
>> 1) run the last two "nm" commands on your build (wthout grep).
>>
> 
> I cannot run nm(1) on qemu-system-x86_64 as it's not linkable.
> 
> I'm getting the same result for target/i386/cpu.o and
> target/i386/kvm-stub.o.
> 
> $ nm ./i386-softmmu/target/i386/kvm-stub.o
>  U abort
>  T kvm_allows_irq0_override
> 0030 T kvm_arch_get_supported_cpuid
> 0020 T k

Re: [Qemu-devel] [PATCH] target-i386: Enhance the stub for kvm_arch_get_supported_cpuid()

[Kamil Rytarowski]

On 20.02.2019 18:29, Paolo Bonzini wrote:
> On 20/02/19 12:59, Kamil Rytarowski wrote:
>> Ping, still valid.
> 
> Sorry, I missed your email.
> 
>> On 15.02.2019 00:38, Kamil Rytarowski wrote:
>>> I consider it as fragile hack and certainly not something to depend on.
>>> Also in some circumstances of such code, especially "if (zero0)" we want
>>> to enable disabled code under a debugger.
> 
> That's a good objection, but certainly does not apply to KVM on NetBSD.
>

There is KVM for Darwin (experimental and rather toy project) and it
might be ported to NetBSD (I have actually forked it on GitHub
recently), but I doubt that someone would enable KVM on any platform
under a debugger this way and expect something to work.

>>> There were also kernel backdoors due to this optimization.
> 
> Citation please?
> 

I saw an exploit for such case with a .txt writeup on ftp of grsecurity
but that service seems to be gone (probably long time ago), so please
defer discussion on it. If someone is interested to find it out, there
are enough pointers to dig it (assuming that this is still possible).

>>> Requested cpu.i (hopefully correctly generated)
>>>
>>> http://netbsd.org/~kamil/qemu/cpu.i.bz2
> 
> So, first thing first I can reproduce clang's behavior with this .i file
> and also with this reduced test case.
> 
> extern void f(void);
> int i, j;
> int main()
> {
> if (0  && i) f();
> if (j  && 0) f();
>}
> 
> The first is eliminated but the second is not, just like in QEMU where
> this works:
> 
> if (kvm_enabled() && cpu->enable_pmu) {
> KVMState *s = cs->kvm_state;
> 
> *eax = kvm_arch_get_supported_cpuid(s, 0xA, count, R_EAX);
> *ebx = kvm_arch_get_supported_cpuid(s, 0xA, count, R_EBX);
> *ecx = kvm_arch_get_supported_cpuid(s, 0xA, count, R_ECX);
> *edx = kvm_arch_get_supported_cpuid(s, 0xA, count, R_EDX);
> } else if (hvf_enabled() && cpu->enable_pmu) {
> *eax = hvf_get_supported_cpuid(0xA, count, R_EAX);
> *ebx = hvf_get_supported_cpuid(0xA, count, R_EBX);
> *ecx = hvf_get_supported_cpuid(0xA, count, R_ECX);
> *edx = hvf_get_supported_cpuid(0xA, count, R_EDX);
> 
> while this doesn't:
> 
> if ((env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_INTEL_PT) &&
> kvm_enabled()) {
> KVMState *s = CPU(cpu)->kvm_state;
> uint32_t eax_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_EAX);
> uint32_t ebx_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_EBX);
> uint32_t ecx_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_ECX);
> uint32_t eax_1 = kvm_arch_get_supported_cpuid(s, 0x14, 1, R_EAX);
> uint32_t ebx_1 = kvm_arch_get_supported_cpuid(s, 0x14, 1, R_EBX);
> 
> But, that's okay, it's -O0 so we give clang a pass for that  Note that
> clang does do the optimization even in more complex cases like
> 
> extern _Bool f(void);
> int main()
> {
> if (!0) return 0;
> if (!f()) return 0;
> }
> 
> The problem is that there is a kvm-stub.c entry for that, and in fact
> my compilation passes and the symbol is resolved correctly:
> 
> $ nm target/i386/cpu.o |grep kvm_.*get_sup
>  U kvm_arch_get_supported_cpuid
> $ nm target/i386/kvm-stub.o|grep kvm_.*get_sup
> 0030 T kvm_arch_get_supported_cpuid
> $ nm qemu-system-x86_64 |grep kvm_.*get_sup
> 0046eab0 T kvm_arch_get_supported_cpuid
> 
> As expected, something much less obvious is going on for you, in
> particular __OPTIMIZE__seems not to be working properly.  However,
> that would also be very surprising.
> 
> Please:
> 
> 1) run the last two "nm" commands on your build (wthout grep).
> 

I cannot run nm(1) on qemu-system-x86_64 as it's not linkable.

I'm getting the same result for target/i386/cpu.o and
target/i386/kvm-stub.o.

$ nm ./i386-softmmu/target/i386/kvm-stub.o
 U abort
 T kvm_allows_irq0_override
0030 T kvm_arch_get_supported_cpuid
0020 T kvm_enable_x2apic
0010 T kvm_has_smm
0050 T kvm_hv_vpindex_settable

grep(1) used, but otherwise I would need to upload results somewhere else.

$ nm ./i386-bsd-user/target/i386/cpu.o |grep kvm
 U kvm_arch_get_supported_cpuid
1290 d kvm_default_props
 U kvm_state
0240 T x86_cpu_change_kvm_default

Please note that there are 4 types of x86 build: i386, x86_64 and two
bsd-user (32-bit and 64-bit).

According to my o

Re: [Qemu-devel] [PATCH] target-i386: Enhance the stub for kvm_arch_get_supported_cpuid()

[Kamil Rytarowski]

Ping, still valid.

On 15.02.2019 00:38, Kamil Rytarowski wrote:
> On 14.02.2019 21:51, Paolo Bonzini wrote:
>> On 14/02/19 20:41, Kamil Rytarowski wrote:
>>> Please do replace the current kludge that is sensitive to:
>>>  - compiler behavior that can change with new versions
>>>  - compiler gcc/clang
>>>  - optimization options
>>
>> Not really, any half-decent compiler will optimize away "if (0)" and
>> QEMU is far from being the only software that relies on that.
>>
>> GCC has been doing that even at -O0 for like 15 years, at some point it
>> was basically the only optimization it did.  Just try it for yourself:
>>
>>  int f(void);
>>
>>  int main()
>>  {
>>  if (0)
>>  return f();
>>  else
>>  return 0;
>>  }
>>
>> Throw it at all compilers and optimization levels, and it *will* work.
>> If it doesn't then I'll consider again your patch.
>>
> 
> I consider it as fragile hack and certainly not something to depend on.
> Also in some circumstances of such code, especially "if (zero0)" we want
> to enable disabled code under a debugger.
> 
> There were also kernel backdoors due to this optimization.
> 
>>>  - linux(KVM) - non-linux (no-KVM) build
>>
>> That's the point.  We want your non-Linux non-KVM build to be as lean as
>> possible and not cause possible run-time failures due to people
>> forgetting about them.
>>
>>>  - community not actively testing non-linux no-kvm build with
>>> optimization on clang
>>
>> False, we test OS X and there are VM builds for the BSDs.
> 
> Unfortunately not in the same combination of options as nobody caught it
> in years. (Probably not many people actually develop it on these OSes
> with debug flags). I was keeping this patch locally for some time now.
> This hack was introduced several years ago.
> 
>>> My patch replaced it makes it work.
>>>
>>> Build error:
>>>
>>>   LINKi386-bsd-user/qemu-i386
>>
>> Ok, please use "make -C i386-bsd-user target/i386/cpu.o V=1" to get the
>> command line, invoke it again with "-save-temps" at the end, and send me
>> both the command line and the resulting "cpu.i" file.
>>
> 
> I'm building qemu with pkgsrc that provides all the dependencies and
> compiler settings. It also uses wrappers to translate original compiler
> options with transformed ones.
> 
> Log from pkgsrc with command lines:
> 
> http://netbsd.org/~kamil/qemu/qemu-build-2019-02-14.txt.bz2
> 
> Requested cpu.i (hopefully correctly generated)
> 
> http://netbsd.org/~kamil/qemu/cpu.i.bz2
> 
> I've generated it manually with this command.
> 
> /usr/local/bin/clang -iquote
> /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/target/i386 -iquote
> target/i386 -iquote /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/tcg
> -iquote /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/tcg/i386 -iquote .
> -iquote /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0 -iquote
> /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/accel/tcg -iquote
> /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/include
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.x11-buildlink/include/pixman-1
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/dtc/libfdt -pthread
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include/glib/glib-2.0
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/lib/glib-2.0/include
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include -m64 -mcx16
> -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
> -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings
> -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv
> -Wno-error=address-of-packed-member -Wno-string-plus-int
> -Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels
> -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body
> -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self
> -Wignored-qualifiers -Wold-style-definition -Wtype-limits
> -fstack-protector-strong
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include/libpng16
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/capstone/include -iquote
> .. -iquote /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/target/i386
> -DNEED_CPU_H -iquote
> /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/include -MMD -MP -MT
> target/i386/cpu.o -MF target/i386/cpu.d -O2 -g -O2 -O0 -g -ggdb
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include/SDL2
> -I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.x11-buildlink/include
> -I/tmp/p

Re: [Qemu-devel] [PATCH] target-i386: Enhance the stub for kvm_arch_get_supported_cpuid()

[Kamil Rytarowski]

On 14.02.2019 21:51, Paolo Bonzini wrote:
> On 14/02/19 20:41, Kamil Rytarowski wrote:
>> Please do replace the current kludge that is sensitive to:
>>  - compiler behavior that can change with new versions
>>  - compiler gcc/clang
>>  - optimization options
> 
> Not really, any half-decent compiler will optimize away "if (0)" and
> QEMU is far from being the only software that relies on that.
> 
> GCC has been doing that even at -O0 for like 15 years, at some point it
> was basically the only optimization it did.  Just try it for yourself:
> 
>   int f(void);
> 
>   int main()
>   {
>   if (0)
>   return f();
>   else
>   return 0;
>   }
> 
> Throw it at all compilers and optimization levels, and it *will* work.
> If it doesn't then I'll consider again your patch.
> 

I consider it as fragile hack and certainly not something to depend on.
Also in some circumstances of such code, especially "if (zero0)" we want
to enable disabled code under a debugger.

There were also kernel backdoors due to this optimization.

>>  - linux(KVM) - non-linux (no-KVM) build
> 
> That's the point.  We want your non-Linux non-KVM build to be as lean as
> possible and not cause possible run-time failures due to people
> forgetting about them.
> 
>>  - community not actively testing non-linux no-kvm build with
>> optimization on clang
> 
> False, we test OS X and there are VM builds for the BSDs.

Unfortunately not in the same combination of options as nobody caught it
in years. (Probably not many people actually develop it on these OSes
with debug flags). I was keeping this patch locally for some time now.
This hack was introduced several years ago.

>> My patch replaced it makes it work.
>>
>> Build error:
>>
>>   LINKi386-bsd-user/qemu-i386
> 
> Ok, please use "make -C i386-bsd-user target/i386/cpu.o V=1" to get the
> command line, invoke it again with "-save-temps" at the end, and send me
> both the command line and the resulting "cpu.i" file.
> 

I'm building qemu with pkgsrc that provides all the dependencies and
compiler settings. It also uses wrappers to translate original compiler
options with transformed ones.

Log from pkgsrc with command lines:

http://netbsd.org/~kamil/qemu/qemu-build-2019-02-14.txt.bz2

Requested cpu.i (hopefully correctly generated)

http://netbsd.org/~kamil/qemu/cpu.i.bz2

I've generated it manually with this command.

/usr/local/bin/clang -iquote
/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/target/i386 -iquote
target/i386 -iquote /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/tcg
-iquote /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/tcg/i386 -iquote .
-iquote /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0 -iquote
/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/accel/tcg -iquote
/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/include
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.x11-buildlink/include/pixman-1
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/dtc/libfdt -pthread
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include/glib/glib-2.0
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/lib/glib-2.0/include
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include -m64 -mcx16
-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
-Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings
-Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv
-Wno-error=address-of-packed-member -Wno-string-plus-int
-Wno-initializer-overrides -Wexpansion-to-defined -Wendif-labels
-Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body
-Wnested-externs -Wformat-security -Wformat-y2k -Winit-self
-Wignored-qualifiers -Wold-style-definition -Wtype-limits
-fstack-protector-strong
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include/libpng16
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/capstone/include -iquote
.. -iquote /tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/target/i386
-DNEED_CPU_H -iquote
/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/include -MMD -MP -MT
target/i386/cpu.o -MF target/i386/cpu.d -O2 -g -O2 -O0 -g -ggdb
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include/SDL2
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.x11-buildlink/include
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.x11-buildlink/include/libdrm
-I/tmp/pkgsrc-tmp/wip/qemu-haxm/work/.buildlink/include/glib/gio-unix-2.0
-I/usr/include/krb5 -c -o target/i386/cpu.o
/tmp/pkgsrc-tmp/wip/qemu-haxm/work/qemu-3.0.0/target/i386/cpu.c
-Qunused-arguments -fstack-protector -save-temps

> Paolo
> 
>> /usr/bin/ld: /usr/lib/libc.so and /usr/lib/crt0.o: warning: multiple
>> common of `environ'
>> /usr/bin/ld: target/i386/cpu.o: in function `x86_cpu_filter_features':
>> /tmp/pkgsrc-

Re: [Qemu-devel] [PATCH] target-i386: Enhance the stub for kvm_arch_get_supported_cpuid()

[Kamil Rytarowski]

On 14.02.2019 19:44, Paolo Bonzini wrote:
> On 02/02/19 15:45, Kamil Rytarowski wrote:
>>
>> Clang/LLVM on NetBSD with enabled optimization cannot link
>> correct qemu program because of a missing symbol of
>> kvm_arch_get_supported_cpuid() in kvm-stubs.o used by executables.
> 
> Can you please include the full error message?  Usually these things are
> a sign of a bug elsewhere.
> 
> Paolo
> 

Please do replace the current kludge that is sensitive to:
 - compiler behavior that can change with new versions
 - compiler gcc/clang
 - optimization options
 - linux(KVM) - non-linux (no-KVM) build
 - community not actively testing non-linux no-kvm build with
optimization on clang


My patch replaced it makes it work.

Build error:

  LINKi386-bsd-user/qemu-i386
/usr/bin/ld: /usr/lib/libc.so and /usr/lib/crt0.o: warning: multiple
common of `environ'
/usr/bin/ld: target/i386/cpu.o: in function `x86_cpu_filter_features':
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5047:
undefined reference to `kvm_arch_get_supported_cpuid'
/usr/bin/ld:
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5048:
undefined reference to `kvm_arch_get_supported_cpuid'
/usr/bin/ld:
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5049:
undefined reference to `kvm_arch_get_supported_cpuid'
/usr/bin/ld:
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5050:
undefined reference to `kvm_arch_get_supported_cpuid'
/usr/bin/ld:
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5051:
undefined reference to `kvm_arch_get_supported_cpuid'
clang-9: error: linker command failed with exit code 1 (use -v to see
invocation)
make[1]: *** [Makefile:199: qemu-i386] Error 1
gmake: *** [Makefile:483: subdir-i386-bsd-user] Error 2
gmake: *** Waiting for unfinished jobs
  LINKx86_64-bsd-user/qemu-x86_64
/usr/bin/ld: /usr/lib/libc.so and /usr/lib/crt0.o: warning: multiple
common of `environ'
/usr/bin/ld: target/i386/cpu.o: in function `x86_cpu_filter_features':
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5047:
undefined reference to `kvm_arch_get_supported_cpuid'
/usr/bin/ld:
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5048:
undefined reference to `kvm_arch_get_supported_cpuid'
/usr/bin/ld:
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5049:
undefined reference to `kvm_arch_get_supported_cpuid'
/usr/bin/ld:
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5050:
undefined reference to `kvm_arch_get_supported_cpuid'
/usr/bin/ld:
/tmp/pkgsrc-tmp/emulators/qemu/work/qemu-3.1.0/target/i386/cpu.c:5051:
undefined reference to `kvm_arch_get_supported_cpuid'
clang-9: error: linker command failed with exit code 1 (use -v to see
invocation)
make[1]: *** [Makefile:199: qemu-x86_64] Error 1
gmake: *** [Makefile:483: subdir-x86_64-bsd-user] Error 2
*** Error code 2



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH] target-i386: Enhance the stub for kvm_arch_get_supported_cpuid()

[Kamil Rytarowski]

Ping?

On 02.02.2019 15:45, Kamil Rytarowski wrote:
> This improves the commit:
> "target-i386: Fix build by providing stub kvm_arch_get_supported_cpuid()"
> r. 2140cfa51d59177815f5b82e94ac48fb24909aba
> 
> Clang/LLVM on NetBSD with enabled optimization cannot link
> correct qemu program because of a missing symbol of
> kvm_arch_get_supported_cpuid() in kvm-stubs.o used by executables.
> 
> There are more than a single one kvm-stub.c and several types
> of possible programs such as bsd-user ones. the previous workaround
> does not work reliably for all use-cases. Instead of reworking
> the stubs and linking rules, move the workaround from a code that
> depends on the __OPTIMIZE__ builtin compiler flag, build option (KVM),
> compiler and arrangement of linking rules to a simple macro in a
> shared header with all the users that defines fallback dummy
> implementation, ignoring whether it is optimized out or not.
> 
> Signed-off-by: Kamil Rytarowski 
> ---
>  include/sysemu/kvm.h   | 13 +
>  target/i386/kvm-stub.c | 10 --
>  2 files changed, 13 insertions(+), 10 deletions(-)
> 
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index a6d1cd190f..93d3c0f0b3 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -459,8 +459,21 @@ int kvm_vm_check_extension(KVMState *s, unsigned int 
> extension);
>  kvm_vcpu_ioctl(cpu, KVM_ENABLE_CAP, &cap);   \
>  })
>  
> +#ifdef CONFIG_KVM
>  uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
>uint32_t index, int reg);
> +#else
> +/*
> + * This function is only called inside conditionals which we
> + * rely on the compiler to optimize out when CONFIG_KVM is not
> + * defined.
> + */
> +#define kvm_arch_get_supported_cpuid(a, b, c, d) \
> +({   \
> +abort(); \
> +0;   \
> +})
> +#endif
>  uint32_t kvm_arch_get_supported_msr_feature(KVMState *s, uint32_t index);
>  
>  
> diff --git a/target/i386/kvm-stub.c b/target/i386/kvm-stub.c
> index e7a673e5db..9ce8566700 100644
> --- a/target/i386/kvm-stub.c
> +++ b/target/i386/kvm-stub.c
> @@ -29,16 +29,6 @@ bool kvm_enable_x2apic(void)
>  {
>  return false;
>  }
> -
> -/* This function is only called inside conditionals which we
> - * rely on the compiler to optimize out when CONFIG_KVM is not
> - * defined.
> - */
> -uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
> -  uint32_t index, int reg)
> -{
> -abort();
> -}
>  #endif
>  
>  bool kvm_hv_vpindex_settable(void)
> 




signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PATCH] configure: Add HAX support in NetBSD

[Kamil Rytarowski]

The NetBSD support in Intel HAXM has beem merged upstream and is functional.

Signed-off-by: Kamil Rytarowski 
---
 configure | 1 +
 1 file changed, 1 insertion(+)

diff --git a/configure b/configure
index b18281c61f..89b889072d 100755
--- a/configure
+++ b/configure
@@ -819,6 +819,7 @@ DragonFly)
 ;;
 NetBSD)
   bsd="yes"
+  hax="yes"
   make="${MAKE-gmake}"
   audio_drv_list="oss try-sdl"
   audio_possible_drivers="oss sdl"
-- 
2.20.1

Re: [Qemu-devel] [PATCH] hax: Support for Linux hosts

[Kamil Rytarowski]

On 25.11.2018 18:14, Paolo Bonzini wrote:
> On 25/11/18 00:50, Kamil Rytarowski wrote:
>> On 22.11.2018 08:24, Kamil Rytarowski wrote:
>>> On 16.11.2018 13:52, Paolo Bonzini wrote:
>>>> On 14/11/18 14:04, Alexandro Sanchez Bach wrote:
>>>>> Intel HAXM supports now 32-bit and 64-bit Linux hosts. This patch includes
>>>>> the corresponding userland changes.
>>>>>
>>>>> Since the Darwin userland backend is POSIX-compliant, the hax-darwin.{c,h}
>>>>> files have been renamed to hax-posix.{c,h}. This prefix is consistent with
>>>>> the naming used in the rest of QEMU.
>>>>
>>>> What's the advantage of HAXM when Linux hosts can just run KVM?  I guess
>>>> avoiding bitrot?
>>>>
>>>> Paolo
>>>>
>>>
>>> This patch is also useful for NetBSD, even if it's not a Linux host.
>>> There is a driver in progress again (thanks to the newly added Linux
>>> port, it's now much easier to get done).
>>>
>>> I recommend to merge this patch.
>>>
>>
>> For the record, I've a functional version of HAXM for NetBSD as host.
>> Once you will merge this patch, I will submit another one to configure
>> to enable haxm for NetBSD.
>>
>> I need to keep the patch by Alexandro in a local copy of qemu.
> 
> Sure, it will be accepted for the release after 3.1.
> 
> Paolo
> 
> 

I've pushed the haxm patch for NetBSD through qemu-trivial.

https://lists.gnu.org/archive/html/qemu-trivial/2019-01/msg00161.html

A proof that it is usable:

http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge



signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PATCH] target-i386: Enhance the stub for kvm_arch_get_supported_cpuid()

[Kamil Rytarowski]

This improves the commit:
"target-i386: Fix build by providing stub kvm_arch_get_supported_cpuid()"
r. 2140cfa51d59177815f5b82e94ac48fb24909aba

Clang/LLVM on NetBSD with enabled optimization cannot link
correct qemu program because of a missing symbol of
kvm_arch_get_supported_cpuid() in kvm-stubs.o used by executables.

There are more than a single one kvm-stub.c and several types
of possible programs such as bsd-user ones. the previous workaround
does not work reliably for all use-cases. Instead of reworking
the stubs and linking rules, move the workaround from a code that
depends on the __OPTIMIZE__ builtin compiler flag, build option (KVM),
compiler and arrangement of linking rules to a simple macro in a
shared header with all the users that defines fallback dummy
implementation, ignoring whether it is optimized out or not.

Signed-off-by: Kamil Rytarowski 
---
 include/sysemu/kvm.h   | 13 +
 target/i386/kvm-stub.c | 10 --
 2 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index a6d1cd190f..93d3c0f0b3 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -459,8 +459,21 @@ int kvm_vm_check_extension(KVMState *s, unsigned int 
extension);
 kvm_vcpu_ioctl(cpu, KVM_ENABLE_CAP, &cap);   \
 })
 
+#ifdef CONFIG_KVM
 uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
   uint32_t index, int reg);
+#else
+/*
+ * This function is only called inside conditionals which we
+ * rely on the compiler to optimize out when CONFIG_KVM is not
+ * defined.
+ */
+#define kvm_arch_get_supported_cpuid(a, b, c, d) \
+({   \
+abort(); \
+0;   \
+})
+#endif
 uint32_t kvm_arch_get_supported_msr_feature(KVMState *s, uint32_t index);
 
 
diff --git a/target/i386/kvm-stub.c b/target/i386/kvm-stub.c
index e7a673e5db..9ce8566700 100644
--- a/target/i386/kvm-stub.c
+++ b/target/i386/kvm-stub.c
@@ -29,16 +29,6 @@ bool kvm_enable_x2apic(void)
 {
 return false;
 }
-
-/* This function is only called inside conditionals which we
- * rely on the compiler to optimize out when CONFIG_KVM is not
- * defined.
- */
-uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
-  uint32_t index, int reg)
-{
-abort();
-}
 #endif
 
 bool kvm_hv_vpindex_settable(void)
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 08/18] tests/bios-tables: Improve portability by searching bash in the $PATH

[Kamil Rytarowski]

On 29.01.2019 18:53, Philippe Mathieu-Daudé wrote:
> Bash is not always installed as /bin/bash. In particular on OpenBSD,
> the package installs it in /usr/local/bin.
> Use the 'env' shebang to search bash in the $PATH.
> 
> Signed-off-by: Philippe Mathieu-Daudé 

Reviewed-by: Kamil Rytarowski 

> ---
>  tests/data/acpi/rebuild-expected-aml.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/data/acpi/rebuild-expected-aml.sh 
> b/tests/data/acpi/rebuild-expected-aml.sh
> index bf9ba242ad..ff77a751c8 100755
> --- a/tests/data/acpi/rebuild-expected-aml.sh
> +++ b/tests/data/acpi/rebuild-expected-aml.sh
> @@ -1,4 +1,4 @@
> -#! /bin/bash
> +#! /usr/bin/env bash
>  
>  #
>  # Rebuild expected AML files for acpi unit-test
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v2 07/18] tests/multiboot: Improve portability by searching bash in the $PATH

[Kamil Rytarowski]

On 29.01.2019 18:53, Philippe Mathieu-Daudé wrote:
> Bash is not always installed as /bin/bash. In particular on OpenBSD,
> the package installs it in /usr/local/bin.
> Use the 'env' shebang to search bash in the $PATH.
> 
> Signed-off-by: Philippe Mathieu-Daudé 

Reviewed-by: Kamil Rytarowski 

> ---
>  tests/multiboot/run_test.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/multiboot/run_test.sh b/tests/multiboot/run_test.sh
> index 6c33003e71..a88e423992 100755
> --- a/tests/multiboot/run_test.sh
> +++ b/tests/multiboot/run_test.sh
> @@ -1,4 +1,4 @@
> -#!/bin/bash
> +#! /usr/bin/env bash
>  
>  # Copyright (c) 2013 Kevin Wolf 
>  #
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PULL v2 07/49] util: check the return value of fcntl in qemu_set_{block, nonblock}

[Kamil Rytarowski]

On 25.01.2019 19:53, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> On 1/15/19 9:04 PM, Michael S. Tsirkin wrote:
>> From: Li Qiang 
>>
>> Assert that the return value is not an error. This is like commit
>> 7e6478e7d4f for qemu_set_cloexec.
>>
>> Signed-off-by: Li Qiang 
>> Reviewed-by: Thomas Huth 
>> Reviewed-by: Michael S. Tsirkin 
>> Signed-off-by: Michael S. Tsirkin 
>> ---
>>  util/oslib-posix.c | 8 ++--
>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/util/oslib-posix.c b/util/oslib-posix.c
>> index c1bee2a581..4ce1ba9ca4 100644
>> --- a/util/oslib-posix.c
>> +++ b/util/oslib-posix.c
>> @@ -233,14 +233,18 @@ void qemu_set_block(int fd)
>>  {
>>  int f;
>>  f = fcntl(fd, F_GETFL);
>> -fcntl(fd, F_SETFL, f & ~O_NONBLOCK);
>> +assert(f != -1);
>> +f = fcntl(fd, F_SETFL, f & ~O_NONBLOCK);
>> +assert(f != -1);
>>  }
>>  
>>  void qemu_set_nonblock(int fd)
>>  {
>>  int f;
>>  f = fcntl(fd, F_GETFL);
>> -fcntl(fd, F_SETFL, f | O_NONBLOCK);
>> +assert(f != -1);
>> +f = fcntl(fd, F_SETFL, f | O_NONBLOCK);
>> +assert(f != -1);
> 
> This commit breaks OpenBSD, when trying to start QEMU I get:
> assertion "f != -1" failed: file "util/oslib-posix.c", line 247,
> function "qemu_set_nonblock"
> 
> Having a quick look at gdb, the last device opened is /dev/null, and
> when fcntl() fails we have errno = ENODEV.
> 
> 19 ENODEV Operation not supported by device.
> An attempt was made to apply an inappropriate function to a device,
> for example, trying to read a write-only device such as a printer.
> 
> Digging further I found a recent commit which could fix this problem:
> https://github.com/openbsd/src/commit/c2a35b387f9d3c
> "fcntl(F_SETFL) invokes the FIONBIO and FIOASYNC ioctls internally, so
> the memory devices (/dev/null, /dev/zero, etc) need to permit them."
> 
> Brad: Do you think this might be the fix? If so, any idea what is the
> first release to contain this fix? I don't know OpenBSD and can't figure
> this out... Also, what would be the cleaner QEMU fix?
> 
> Thanks,
> 

I cannot speak for OpenBSD (never installed it myself), but if there is
a critical patch to test on NetBSD - please let me know.

> Phil.
> 
>>  }
>>  
>>  int socket_set_fast_reuse(int fd)
>>
> 




signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH RFC 8/9] tests: Add OpenBSD image

[Kamil Rytarowski]

On 24.01.2019 16:52, Philippe Mathieu-Daudé wrote:
> On 8/16/17 9:21 AM, Fam Zheng wrote:
>> The image is prepared following instructions as in:
>>
>> https://wiki.qemu.org/Hosts/BSD
>>
>> Signed-off-by: Fam Zheng 
>> ---
>>  tests/vm/openbsd | 45 +
>>  1 file changed, 45 insertions(+)
>>  create mode 100755 tests/vm/openbsd
>>
>> diff --git a/tests/vm/openbsd b/tests/vm/openbsd
>> new file mode 100755
>> index 00..d37ff83a59
>> --- /dev/null
>> +++ b/tests/vm/openbsd
>> @@ -0,0 +1,45 @@
>> +#!/usr/bin/env python
>> +#
>> +# OpenBSD VM image
>> +#
>> +# Copyright (C) 2017 Red Hat Inc.
>> +#
>> +# Authors:
>> +#  Fam Zheng 
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2.  See
>> +# the COPYING file in the top-level directory.
>> +#
>> +
>> +import os
>> +import sys
>> +import logging
>> +import subprocess
>> +import tempfile
>> +import time
>> +import basevm
>> +
>> +class OpenBSDVM(basevm.BaseVM):
>> +name = "openbsd"
>> +BUILD_SCRIPT = """
>> +set -e;
>> +cd $(mktemp -d /var/tmp/qemu-test.XX);
>> +tar -xf /dev/rsd1c;
>> +./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 
>> --python=python2.7 {configure_opts};
>> +gmake -j{jobs};
>> +# XXX: "gmake check" seems to always hang or fail
>> +#gmake check;
> 
> OK, Now it makes more sense...
> 
> After spending various hours trying to fix various issues on OpenBSD, I
> notice that we never ran tests on this OS.
> The only binary I can run is qemu-img, the rest seems useless.
> I'll summarize in a different thread.
> 

Is this W^X related?

>> +"""
>> +
>> +def build_image(self, img, rebuild=False):
>> +if os.path.exists(img) and not rebuild:
>> +return
>> +cimg = 
>> self._download_with_cache("http://download.patchew.org/openbsd.img.xz";)
>> +img_tmp_xz = img + ".tmp.xz"
>> +img_tmp = img + ".tmp"
>> +subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
>> +subprocess.check_call(["xz", "-df", img_tmp_xz])
>> +subprocess.check_call(["mv", img_tmp, img])
>> +
>> +if __name__ == "__main__":
>> +sys.exit(basevm.main(OpenBSDVM))
>>
> 




signature.asc
Description: OpenPGP digital signature