Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

[Harry Schmalzbauer]

Am 21.05.2021 um 14:50 schrieb Harry Schmalzbauer:

Am 19.05.2021 um 13:26 schrieb Harry Schmalzbauer:

...
I can just assure that I tracked it down to the single r332489 
committ which makes vmbus(4) failing on i386.

Verification done:
- built/installed world on i386 HyV guest with r332488
  reboot fine.
- built/installed kernel on the same vm after svnlite up -r 332489
  reboot fails with "vmbus0: cannot find free IDT vector"
- loaded kernel.old/kernel, did revert 332489, by 'svnlite up -r 
332488' and compiled/reinstalled kernel (again r332488)

  reboot fine.


While having hard time catching up with x86 basics, I found another 
oddity:
On Hyper-V gen1, the maximum ammount of RAM I can assign a FreeBSD 
guest beeing able to boot i386 is 3914MB, startring with r332489.
As soon as I configure the guest to have 3916MB or more, FreeBSD-i386 
r332489 booting leads to guest reset.  Loader starts kernel, machine 
reboots.

Either reverting to r332488 or reducing guest-RAM solves this.

The most interesting fact here is that I couldn't reproduce it on ESXi.
I ducplicated the Hyper-V HDD and started the exactly same code on 
ESXi(v6) guest, where it doesn't crash!  I can boot FreeBSD-i386 
r332489 with any ammount of guest-RAM on ESXi.

(speaking for GENERIC, non-PAE)


Now that I have a living r332489 i386 available, I checked some sysctl.
It seems on ESXi and i386, large page mappings aren't used anymore, or 
is it just wrong indicating?


These valueas are suspicious to my eyes:

kern.version: FreeBSD 12.0-CURRENT #0 r332489: Fri May 21 15:32:35 CEST 2021
    root@esxtest:/usr/obj/altroot/usr/src/i386.i386/sys/GENERIC
hw.pagesizes: 4096 0
hw.realmem: 0
vm.min_kernel_address: 0
vm.max_kernel_address: 4286578688
vm.kvm_free: -549453824
vm.kvm_size: -16777216
vm.pmap.pg_ps_enabled: 0

kern.version: FreeBSD 12.0-CURRENT #1 r332488: Fri May 21 17:35:58 CEST 2021
    root@esxtest:/usr/obj/altroot/usr/src/i386.i386/sys/GENERIC
hw.pagesizes: 4096 4194304
hw.realmem: 0
vm.min_kernel_address: 3220172800
vm.max_kernel_address: 4294963200
vm.kvm_free: 624947200
vm.kvm_size: 1073737728
vm.pmap.pg_ps_enabled: 1

Thanks for any hints,
-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

[Harry Schmalzbauer]

Am 19.05.2021 um 13:26 schrieb Harry Schmalzbauer:

...
I can just assure that I tracked it down to the single r332489 committ 
which makes vmbus(4) failing on i386.

Verification done:
- built/installed world on i386 HyV guest with r332488
  reboot fine.
- built/installed kernel on the same vm after svnlite up -r 332489
  reboot fails with "vmbus0: cannot find free IDT vector"
- loaded kernel.old/kernel, did revert 332489, by 'svnlite up -r 
332488' and compiled/reinstalled kernel (again r332488)

  reboot fine.


While having hard time catching up with x86 basics, I found another oddity:
On Hyper-V gen1, the maximum ammount of RAM I can assign a FreeBSD guest 
beeing able to boot i386 is 3914MB, startring with r332489.
As soon as I configure the guest to have 3916MB or more, FreeBSD-i386 
r332489 booting leads to guest reset.  Loader starts kernel, machine 
reboots.

Either reverting to r332488 or reducing guest-RAM solves this.

The most interesting fact here is that I couldn't reproduce it on ESXi.
I ducplicated the Hyper-V HDD and started the exactly same code on 
ESXi(v6) guest, where it doesn't crash!  I can boot FreeBSD-i386 r332489 
with any ammount of guest-RAM on ESXi.

(speaking for GENERIC, non-PAE)

Host CPU differs significantly though and I have no possibility to test 
this influence.  Ther HyperV is Dual-Socket E5-2630 while the ESXi host 
is a single socket X3470 Xeon (pre SandyBridge).


Maybe someone has an idea how this affects these symptoms.

Thanks,
-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

[Harry Schmalzbauer]

Am 17.05.2021 um 11:39 schrieb Harry Schmalzbauer:

Am 17.05.2021 um 08:47 schrieb Konstantin Belousov:

On Mon, May 17, 2021 at 06:33:19AM +, Wei Hu wrote:

-Original Message-
From: Konstantin Belousov 
Sent: Monday, May 17, 2021 1:47 PM
To: Wei Hu 
Cc: Harry Schmalzbauer ; Chris Knight
; freebsd-virtualization@freebsd.org
Subject: Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

On Mon, May 17, 2021 at 04:12:20AM +, Wei Hu wrote:

:
:
:

separately. The workaround for this on amd64 is either build these
Hyper-V modules into the kernel or completely disable mitigation by
setting vm.pmap.pti="0" in /boot/loader.conf in the affect FreeBSD 
guests.




Just to share some hopefully helpful pre-check's I've done so far:

Disabling PTI via vm.pmap.pti="0" in /boot/loader.conf doesn't make 
any difference for the i386 issue (at least x86/x86/local_apic.c 
always uses non-pti path; tried nontheless, with identical result).


As far as my analysis isn't completely wrong, the problem is about IDT 
pointer.
IDTVEC(rsvd) doesn't match in line 2180 of x86/x86/local_apic.c in 
native_lapic_ipi_alloc():

(ip->gd_hioffset << 16) | ip->gd_looffset;



I managed to setup an i386 vm with -current from 3 years ago (not the 
most straight forward task due to compiler infrastructure changes back 
then).

Up to r332488 vmbus(4) resp. hyperv(4) is working fine.
Updating r332488 to r332489 breaks like already described.

Since r332489 is a huge patch with lots of x86 art, the only chance I 
see, how this can be resolved, is with Konstantin's help.


I can just assure that I tracked it down to the single r332489 committ 
which makes vmbus(4) failing on i386.

Verification done:
- built/installed world on i386 HyV guest with r332488
  reboot fine.
- built/installed kernel on the same vm after svnlite up -r 332489
  reboot fails with "vmbus0: cannot find free IDT vector"
- loaded kernel.old/kernel, did revert 332489, by 'svnlite up -r 332488' 
and compiled/reinstalled kernel (again r332488)

  reboot fine.

I tried to revert r332489 on stable/13 but failed due to lots of 
conflicting changes (including CVE driven) affecting the same area.


To summarize: r332489 only breaks i386 native_lapic_ipi_alloc(), amd64 
is not affected (but haven't checked if PTI has any influence on amd64).

The effect is 100% reproducable.
Happy to test any suggestion or debug extension!

Thanks,
-harry




___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

[Harry Schmalzbauer]

Am 17.05.2021 um 08:47 schrieb Konstantin Belousov:

On Mon, May 17, 2021 at 06:33:19AM +, Wei Hu wrote:

-Original Message-
From: Konstantin Belousov 
Sent: Monday, May 17, 2021 1:47 PM
To: Wei Hu 
Cc: Harry Schmalzbauer ; Chris Knight
; freebsd-virtualization@freebsd.org
Subject: Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

On Mon, May 17, 2021 at 04:12:20AM +, Wei Hu wrote:

:
:
:

separately. The workaround for this on amd64 is either build these
Hyper-V modules into the kernel or completely disable mitigation by
setting vm.pmap.pti="0" in /boot/loader.conf in the affect FreeBSD guests.



Just to share some hopefully helpful pre-check's I've done so far:

Disabling PTI via vm.pmap.pti="0" in /boot/loader.conf doesn't make any 
difference for the i386 issue (at least x86/x86/local_apic.c always uses 
non-pti path; tried nontheless, with identical result).


As far as my analysis isn't completely wrong, the problem is about IDT 
pointer.
IDTVEC(rsvd) doesn't match in line 2180 of x86/x86/local_apic.c in 
native_lapic_ipi_alloc():

(ip->gd_hioffset << 16) | ip->gd_looffset;

Workign 11.4 shows these values:
11.4-i386 idx 251 (252-1 because no r363527, IPI_SWI) :
  hi =  (49417)1100 0001  1001
  lo =   (5472)0001 0101 0110 
func = 1100 0001  1001 0001 0101 0110 
IDTVEC(rsvd) = 1100 0001  1001 0001 0101 0110 
== rsvd_pti

stable/13:
i386 idx 252 Nok:
  hi =  (65472)  1100 
  lo =  (12456)0011  1010 1000
func =   1100  0011  1010 1000
IDTVEC(rsvd) = 0 1110 0100 1011 1011 1110 1000
== rsvd_pti

My knowledge about IDT and it's handling by OS/FreeBSD is by far not 
enough to get a solution/explanation for the above.  I couldn't find the 
relevant paths in our source code (starting with the lack of basic asm 
skills).
For sure I'm missing the important path here, maybe I'm totaly wrong... 
if not this hopefully saves you some time.


I noticed r332489 too as a candidate for the regression beteween 12(.0) 
and 11(.2+).
I prepared a diff which applies to 11.4 (sources from iso) but 
recompiling 11.4 fails currently, I would need to rework the 
i386/i386/pmap.c adoption.



:
:
:

know the link to the patch or the commit if you already have checked in? I will
need to test and inform the parties who got affected.


It was mailed for you after your query, then I pinged you several times.
Anyway, look at and try https://reviews.freebsd.org/D30310



I applied it to stable/13 and this doesn't make any difference, still 
the (ip->gd_hioffset << 16) | ip->gd_looffset; and 
(uintptr_t)IDTVEC(rsvd) differ with exactly the same values as shown above.


Thanks for your attention, best regards,
-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

[Harry Schmalzbauer]

Am 14.05.2021 um 18:44 schrieb Harry Schmalzbauer:

Am 16.10.2020 um 11:35 schrieb Wei Hu via freebsd-virtualization:

:


Did you try 12.0? Maybe something changed in 12.x causes failure.
I386 is not well tested on Hyper-V. I will try to reproduce and take 
a look.


The regression happened before 12.0.
Today I stumbled across this problem too and tried 11.4 (/May 8th, 
2020) /sucessfully, while 12.0 (/Oct 19th, 2018)/ is failing with i386 
releases.


One commit which is in 12 but not in 11.4 is r332489.


:
:
:

Unfortunately beyond my skills.


In sys/x86/x86/local_apic.c, I added this debug printf:
    for (idx = IPI_DYN_FIRST; idx <= IPI_DYN_LAST; idx++) {
printf("DEBUG: native_lapic_ipi_alloc(), idx = %i\n", idx);
    ip = [idx];
    func = (ip->gd_hioffset << 16) | ip->gd_looffset;
    if ((!pti && func == (uintptr_t)(rsvd)) ||
    (pti && func == (uintptr_t)(rsvd_pti))) {
    vector = idx;
    setidt(vector, ipifunc, SDT_APIC, SEL_KPL, 
GSEL_APIC);

    break;
    }
    }

This is my latest attempt to see if at least the origin of my fault 
tracing matches.

It does.

From #define IPI_DYN_FIRST   (APIC_IPI_INTS + 9) in 
sys/x86/include/apicvar.h

I expected to see 252, 253 and 254, which I sucessfully verified.

Now comes the part I cannot read: (uintptr_t)(rsvd)
IDTVEC(rsvd) seems to be a assembler makro from sys/i386/i386/exception.s

But now it doesn't make any sense for me to dig further, I simply do not 
understand it.


Unfortunately having broken i386 support on Hyper-V is not a cosmetic issue.
Removing "device hyperv" from kernel gives access to the v-hdd via 
legacy IDE/ada, but then I have no NIC.
Choosing between missing HDD and missing network support is hard 
nowadays ;-)


Any help getting i386 up and running back on Hyper-V highly 
appreciated!  11.4 is the latest working version, all i386-releases 
starting with 12.0 don't work!


Thanks,
-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

[Harry Schmalzbauer]

Am 16.10.2020 um 11:35 schrieb Wei Hu via freebsd-virtualization:

Howdy,

Just started the upgrade process on some FreeBSD 11.4 VMs to FreeBSD 12.1
using freebsd-update. VMs are running on multiple Hyper-V 2012 R2 and 2019
systems, all systems with 2020-09 Cumulative Updates installed.
The amd64 VMs are upgrading happily.
The i386 VMs die a horrible death - the kernel throws the following error:

vmbus0: cannot find free IDT vector

shortly followed by mountroot failure due to the lack of storage device
initialisation.
Checked the FreeBSD 12.1 i386 ISO in case of environment pollution - same
vmbus error, same lack of storage device initialisation failure. Ditto with
the 12.2-RC2 i386 ISO.


Did you try 12.0? Maybe something changed in 12.x causes failure.
I386 is not well tested on Hyper-V. I will try to reproduce and take a look.


The regression happened before 12.0.
Today I stumbled across this problem too and tried 11.4 (/May 8th, 2020) 
/sucessfully, while 12.0 (/Oct 19th, 2018)/ is failing with i386 releases.


One commit which is in 12 but not in 11.4 is r332489.
Code path is like
sys/dev/hyperv/vmbus/vmbus.c:
vmbus_intr_setup(struct vmbus_softc *sc)
{
    int cpu;

    CPU_FOREACH(cpu) {
    char buf[MAXCOMLEN + 1];
    cpuset_t cpu_mask;

    /* Allocate an interrupt counter for Hyper-V interrupt */
    snprintf(buf, sizeof(buf), "cpu%d:hyperv", cpu);
    intrcnt_add(buf, VMBUS_PCPU_PTR(sc, intr_cnt, cpu));

    /*
 * Setup taskqueue to handle events.  Task will be per-
 * channel.
 */
    VMBUS_PCPU_GET(sc, event_tq, cpu) = taskqueue_create_fast(
    "hyperv event", M_WAITOK, taskqueue_thread_enqueue,
    VMBUS_PCPU_PTR(sc, event_tq, cpu));
    if (vmbus_pin_evttask) {
    CPU_SETOF(cpu, _mask);
    taskqueue_start_threads_cpuset(
    VMBUS_PCPU_PTR(sc, event_tq, cpu), 1, PI_NET,
    _mask, "hvevent%d", cpu);
    } else {
    taskqueue_start_threads(
    VMBUS_PCPU_PTR(sc, event_tq, cpu), 1, PI_NET,
    "hvevent%d", cpu);
    }

    /*
 * Setup tasks and taskqueues to handle messages.
 */
    VMBUS_PCPU_GET(sc, message_tq, cpu) = 
taskqueue_create_fast(

    "hyperv msg", M_WAITOK, taskqueue_thread_enqueue,
    VMBUS_PCPU_PTR(sc, message_tq, cpu));
    CPU_SETOF(cpu, _mask);
    taskqueue_start_threads_cpuset(
    VMBUS_PCPU_PTR(sc, message_tq, cpu), 1, PI_NET, 
_mask,

    "hvmsg%d", cpu);
    TASK_INIT(VMBUS_PCPU_PTR(sc, message_task, cpu), 0,
    vmbus_msg_task, sc);
    }

    /*
 * All Hyper-V ISR required resources are setup, now let's find a
 * free IDT vector for Hyper-V ISR and set it up.
 */
    sc->vmbus_idtvec = lapic_ipi_alloc(pti ? IDTVEC(vmbus_isr_pti) :
    IDTVEC(vmbus_isr));
    if (sc->vmbus_idtvec < 0) {
    device_printf(sc->vmbus_dev, "cannot find free IDT 
vector\n");

    return ENXIO;
    }
    if (bootverbose) {
    device_printf(sc->vmbus_dev, "vmbus IDT vector %d\n",
    sc->vmbus_idtvec);
    }
    return 0;
}

In sys/dev/hyperv/vmbus/i386/vmbus_vector.S

    SUPERALIGN_TEXT
IDTVEC(vmbus_isr_pti)
IDTVEC(vmbus_isr)
    PUSH_FRAME
    SET_KERNEL_SREGS
    cld
    KENTER
    FAKE_MCOUNT(TF_EIP(%esp))
    pushl   %esp
    call    vmbus_handle_intr
    add $4, %esp
    MEXITCOUNT
    jmp doreti


amd64/i386 diff (problem doesn"t show up on amd64):
--- sys/dev/hyperv/vmbus/i386/vmbus_vector.S    2021-05-14 
18:04:58.527699000 +0200
+++ sys/dev/hyperv/vmbus/amd64/vmbus_vector.S   2021-01-23 
13:00:56.902717000 +0100

@@ -38,15 +38,9 @@
  */
    .text
    SUPERALIGN_TEXT
-IDTVEC(vmbus_isr_pti)
-IDTVEC(vmbus_isr)
-   PUSH_FRAME
-   SET_KERNEL_SREGS
-   cld
-   KENTER
-   FAKE_MCOUNT(TF_EIP(%esp))
-   pushl   %esp
+   INTR_HANDLER    vmbus_isr
+   FAKE_MCOUNT(TF_RIP(%rsp))
+   movq    %rsp, %rdi
    call    vmbus_handle_intr
-   add $4, %esp
    MEXITCOUNT
    jmp doreti

Unfortunately beyond my skills.

Any news/ideas?

Thanks,
-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ConfigRefactor-621b5090487d rises XHCI issues [Was: Re: stable/13 (PCI reset) yet to be identified regression]

[Harry Schmalzbauer]

Am 21.03.2021 um 16:35 schrieb Harry Schmalzbauer:

Am 20.03.2021 um 23:48 schrieb Harry Schmalzbauer:

:
:
:
but with stable/13 from yesterday the XHCI USB hub is marked 
non-functional in the Windows device manager with the hint "error 
during reset"
and the passthrough NIC doesn"t work anymore -> looks like D3D0 or 
the like initialization fails.  I'm not 100% sure for the XHCI hub 
issue, but the NIC worked without


Forgot to drop the note that ab899f8937c1 fixes the XHCI issue on 
stable/13 with 621b5090487d (ConfigRefactor, enabling vm.cfg 
support)(actually using 
621b5090487d+9f40a3be3d5d+4d5460a720c5+ab899f8937c1 in production on 
stable/13).


Thanks,
-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

ConfigRefactor-621b5090487d rises XHCI issues [Was: Re: stable/13 (PCI reset) yet to be identified regression]

[Harry Schmalzbauer]

Am 20.03.2021 um 23:48 schrieb Harry Schmalzbauer:

Hello,

I'm suffering from a late minute regression with stable/13.
4 weeks ago I did pre-production tests with Windows Server VMs 
migrating to bhyve.
Things went fine, a 82576 ppt Dual Ethernet card worked after 
installing a specific driver version in Server 2012R2.


VMconfig nad host hardware is exaclty the same,
but with stable/13 from yesterday the XHCI USB hub is marked 
non-functional in the Windows device manager with the hint "error 
during reset"
and the passthrough NIC doesn"t work anymore -> looks like D3D0 or the 
like initialization fails.  I'm not 100% sure for the XHCI hub issue, 
but the NIC worked without problems some weeks ago.


The NIC passthrough problem turned out to be a switch problem...
And any kind of ppt limitation, which makes it impossible to use the 
prosetdx (Inetl nEtwork Connetcion App) Teaming function creating a LACP 
(802.11ad) group (creation successful, but no way to "activate"- similar 
as after "deactivating" a NIC via Windows context: re-activating fails, 
but works after reboot;  group doesn't work after reboot).


But the XHCI problem is reproducable by applying jhb's 
ConfigRefactor-621b5090487d.
the difference at bhyve startup with legacy CLI arguments, but applied 
621b5090487d:

pci_xhci no USB devices configured
pci_xhci: portregs_write to unpowered port 1
pci_xhci: portregs_write to unpowered port 2
pci_xhci: portregs_write to unpowered port 3
pci_xhci: portregs_write to unpowered port 4
pci_xhci: portregs_write to unpowered port 5
:
:
:

No matter if I use -k configfile or old CLI arguments
Without the diff from main, same bhyve CLI startup leads to working XHCI 
in the guest (win2k12R2) and the messeges show above vanish.


Thanks,
-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

stable/13 (PCI reset) yet to be identified regression

[Harry Schmalzbauer]

Hello,

I'm suffering from a late minute regression with stable/13.
4 weeks ago I did pre-production tests with Windows Server VMs migrating 
to bhyve.
Things went fine, a 82576 ppt Dual Ethernet card worked after installing 
a specific driver version in Server 2012R2.


VMconfig nad host hardware is exaclty the same,
but with stable/13 from yesterday the XHCI USB hub is marked 
non-functional in the Windows device manager with the hint "error during 
reset"
and the passthrough NIC doesn"t work anymore -> looks like D3D0 or the 
like initialization fails.  I'm not 100% sure for the XHCI hub issue, 
but the NIC worked without problems some weeks ago.


Is anybody aware of any commit which could cause the PCI reset symptoms? 
(very long day, will try to go through after some sleep)



Thanks,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve win-guest benchmark comparing

[Harry Schmalzbauer]

Am 22.10.2018 um 13:26 schrieb Harry Schmalzbauer:
…


Test-Runs:
Each hypervisor had only the one bench-guest running, no other 
tasks/guests were running besides system's native standard processes.
Since the time between powering up the guest and finishing logon 
differed notably (~5s vs. ~20s) from one host to the other, I did a 
quick synthetic IO-Test beforehand.
I'm using IOmeter since heise.de published a great test pattern called 
IOmix – about 18 years ago I guess.  This access pattern has always 
perfectly reflected the system performance for human computer usage 
with non-caculation-centric applications, and still is my favourite, 
despite throughput and latency changed by some orders of manitudes 
during the last decade (and I had defined something for "fio" which 
mimics IOmix and shows reasonable relational results; but I'm still 
prefering IOmeter for homogenous IO benchmarking).


The results is about factor 7 :-(
~3800iops&69MB/s (CPU-guest-usage 42%IOmeter+12%irq)
                vs.
~29000iops&530MB/s (CPU-guest-usage 11%IOmeter+19%irq)


    [with debug kernel and debug-malloc, numbers are 3000iops&56MB/s,
 virtio-blk instead of ahci,hd: results in 5660iops&104MB/s with 
non-debug kernel

 – much better, but even higher CPU load and still factor 4 slower]

What I don't understand is, why the IOmeter process differs that much 
in CPU utilization!?!  It's the same binary on the same OS (guest) 
with the same OS-driver and the same underlying hardware – "just" the 
AHCI emulation and the vmm differ...


I repeated this test with a slightly different device backend (Samsung 
850pro SSD on mps(4) instead of mfid(4)).
After applying r358848 to stable/12, the numbers changed dramatically.0 
on the same haswell based Xeon E3 platform.


With the single SSD, the IOmeter numbers for ESXi as host drop
from ~29000iops&530MB/s to ~11000/205MB/s.
    But the numbers for bhyve as host
raise from ~3800iops&69MB/s to ~8800/160MB/s at the same time!!!

So there's still a penalty of ~20% for ahci-bhyve vs. ahci-esx, but this 
is a enourmous improvement.

Please don't skip the MFC for r358848!

Thanks a lot for all the work!

-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve 11.1->stable/12 ZVOL-WinGuest slowdown – r358848 far beyond imagination

[Harry Schmalzbauer]

Hello,

after upgrading my ancient ivy-bridge (Xeon E3, no vAPIC CPU support) 
from 11.1 to stable/12 I noteced a quiet heavy slowdown of my windows 
guest (Server 2012R2).


The guest-disk is a ZVOL backed virtio-blk device, which I suspected 
suffers from performance regressions – especially since I use them in 
volmode=dev.
But then I found the guest spending almost one full vCPU cycles in 
interrupt handling during light SQL load; something I rarely see on 
windows.  No idea what caused the slowdown, but I remembered r358848 – 
which isn't MFCd yet, but I gave it a try.
Word's can't describe the surprise.  Never imagined any Win-Guest on 
bhyve can feel like that…

Thanks a lot to Sir Burmeister et. al.

Hope this will land in stable/12 soon.

Thanks,

-Harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: BSD11 vs BSD12 KVM performance

[Harry Schmalzbauer]

Am 02.05.2019 um 20:51 schrieb Bapat, Udayan:

Hi All,

I work in performance team with NetApp, Inc.

During qualification process of BSD11 as a guest, we observed that the cost of 
rdtsc call has magnified by 10x compared to BSD10. We are using qemu-2.10 that 
came as a default package in RH7.4. It appears that BSD11 was released before 
qemu-2.10 so I am not sure if we have any supportability issue and I would need 
to upgrade QEMU version.  In what way, I can do more debugging around this? We 
are also experimenting with ESX and we didn’t see this issue there. I also 
could not find any performance benchmarking around BSD+QEMU/KVM.


Hello, please clarify whether it is about FreeBSD-10 vs. FreeBSD-11, 
like the body tells, or FreeBSD-11 vs. FreeBSD-12, like the topic tells.

Can't help in any case regading the QMEU question, unfortunately.

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: NVMe and Bhyve

[Harry Schmalzbauer]

Am 16.02.2019 um 05:11 schrieb Victor Sudakov:

Stefan Bethke wrote:

I was just worried that using ahci-hd instead of paravirtualized disks
causes suboptimal performance of guests. Do you think there is a hope to get
paravirtualized disks support for Windows guests in bhyve?

I was under the impression that the Windows VirtIO disk driver works
with the Bhyve virtio-blk device. You need to provide these drivers to
Windows somehow (for example, as a second CD image during
installation):
https://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers

I use NetKVM (VirtIO network) driver from there all the time, because
e1000 is bad.

However, I've read a number of bhyve resources including
https://wiki.freebsd.org/bhyve/Windows and I've never found a success
story with paravirtualized Windows disk drivers.

If anyone has been successful with viostor(?), please share your experience.


You'll need
https://github.com/freenas/os/commit/0e4d6e1826f8aa7041cbeeb4365c797eeec5c5f4
if I remember correctly.
I asked for commit, since I'm using this in production for a long time, 
but once upon a time, there has been a linux-guest specific side effect.

Don't know much about it and haven't had a chance to reproduce it.
In my bhyve world, any guest runs fine with the higher BLOCKIF_IOV limit.

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve win-guest benchmark comparing

[Harry Schmalzbauer]

Am 30.10.2018 um 02:09 schrieb Dustin Marquess:
It would be interesting to test running it under Xen with FreeBSD as the 
dom0.


ACK, so do I think. This was on my to-check list, but had to be 
postponed until the next HV project legitimates another test setup ;-)
Unfortunately I don't know much about Xen – next to nothing usage wise. 
The machine must go into production this weekend, so I replaced one of 
it's 2308(IT/IR) by a 2208(MegaRAID) and FreeBSD will run as guest, 
utilizing ESXi's pciPassthrough driver for the LynxPoint AHCI, the 
LSI2308 and the ql2464 (or 82576) – like I'm doing for almost a decade 
now.  This setup provides by far the highest resource efficiency for 
such single socket systems (along with much higher single thread guest 
performance compared to linear priced dual socket servers – since 
sc-Xeons gold-6x you can achive the same guest single thread performance 
also with a dual socket system, but price is exponential...).  My boss 
doesn't give me money and/or time to do all the nice stuff possible (I 
simply don't have it) ;-)
SR-IOV should improve resource efficiency a lot for bhyve.  Also curious 
how it competes in that scenario with ESXi!  If I have accees to SR-IOV 
capabale hardware package I'll repeat the current test prior to the Xen 
comparison.  Hopefully both will happen sooner than later.

Perhaps somebody else has Xen test Seup running?
I could provide the benchmark tools/configs – or even the image, since 
the OS-SSDs I used are not erased yet.


But much more interesting was if somebody could describe/explain the 
results!
It's not about optimization here, but I'll see what dtrace can tell me. 
Actually, the whole code is much too complex for me to understand why it 
is what it is and I won't be able to contribute :-(  But I still have 
one bhyve setup running in my own office (where all important services 
are running on the host itself, in jails). Only one (win) guest produces 
minimal load and performance isn't of any importance.  This is with 
stable/11, and I never saw "stuttering" RDP sessions, So my feeling is 
that virtio-net suffers from additional problems in 12... But I won't be 
able to compare, I just confirmed myself that bhyve:virtio-net in 12 
isn't really usable in it's current state.
I'm wondering if somebody else has 11 _and_ 12 running with byhve guests 
which use virtio-net.  Can't imagine it's working for anybody, since 
even transfer rates via DSL lines lead to very high CPU load.


-harry



___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve win-guest benchmark comparing

[Harry Schmalzbauer]

Am 22.10.2018 um 13:26 schrieb Harry Schmalzbauer:
…

Test-Runs:
Each hypervisor had only the one bench-guest running, no other 
tasks/guests were running besides system's native standard processes.
Since the time between powering up the guest and finishing logon 
differed notably (~5s vs. ~20s) from one host to the other, I did a 
quick synthetic IO-Test beforehand.
I'm using IOmeter since heise.de published a great test pattern called 
IOmix – about 18 years ago I guess.  This access pattern has always 
perfectly reflected the system performance for human computer usage 
with non-caculation-centric applications, and still is my favourite, 
despite throughput and latency changed by some orders of manitudes 
during the last decade (and I had defined something for "fio" which 
mimics IOmix and shows reasonable relational results; but I'm still 
prefering IOmeter for homogenous IO benchmarking).


The results is about factor 7 :-(
~3800iops&69MB/s (CPU-guest-usage 42%IOmeter+12%irq)
                vs.
~29000iops&530MB/s (CPU-guest-usage 11%IOmeter+19%irq)


    [with debug kernel and debug-malloc, numbers are 3000iops&56MB/s,
 virtio-blk instead of ahci,hd: results in 5660iops&104MB/s with 
non-debug kernel

 – much better, but even higher CPU load and still factor 4 slower]

What I don't understand is, why the IOmeter process differs that much 
in CPU utilization!?!  It's the same binary on the same OS (guest) 
with the same OS-driver and the same underlying hardware – "just" the 
AHCI emulation and the vmm differ...


Unfortunately, the picture for virtio-net vs. vmxnet3 is similar sad.
Copying a single 5GB file from CIFS share to DB-ssd results in 100% 
guest-CPU usage, where 40% are irqs and the throughput max out at 
~40MB/s.
When copying the same file from the same source with the same guest on 
the same host but host booted ESXi, there's 20% guest-CPU usage while 
transfering 111MB/s – the uplink GbE limit.


These synthetic benchmark very well explain the "feelable" difference 
when using a guest between the two hypervisors, but 

…

To add an additional and rather surprinsing result, at least for me:

Virtualbox provides
'VBoxManage internalcommands createrawvmdk -filename 
"testbench_da0.vmdk" -rawdisk /dev/da0'


So I could use the exactly same test setup as for ESXi and bhyve.
FreeBSD-Virtualbox (running on the same host installation like bhyve) 
performed quiet well, although it doesn't survive IOmix benchmark run 
when the "testbench_da0.vmdk" (the "raw" SSD-R0-array) is hooked up to 
the SATA controller.
But connected to the emulated SAS controller(LSI1068), it runs without 
problems and results in 9600iops@185MB/s with 1%IOmeter+7%irq CPU 
utilization (yes, 1% vs. 42% for IOmeter load).
Still far away from what ESXi provides, but almost double performance of 
virtio-blk with bhyve, and most important, much less load (host and 
guest show exactly the same low values as opposed to the very high loads 
which are shown on host and guest with bhyve:virtio-blk).
The HDtune random access benchmark also shows the factor 2, linear over 
all block sizes.


Virtualbox's virtio-net setup gives ~100MB/s with peaks at 111 and ~40% 
CPU load.
Guest uses the same driver like with bhyve:virtio-blk, while backend of 
virtualbox:virtio-net is vboxnetflt utilizing netgraph and vboxnetadp.ko 
vs. tap(4).
So not only the IO efficiency (lower throughput but also much lower CPU 
utilization) is remarbably better, but also the network performance.  
Even low-bandwidth RDP sessions via GbE-LAN suffer from micro hangs 
under bhyve and virtio-net.  And 40MB/s transfers cause 100% CPU load on 
bhyve – both runs had exactly the same WIndows virtio-net driver in use 
(RedHat 141).


Conclusion: Virtualbox vs. ESXi shows a 0.5% efficiency factor, while 
bhyve vs. ESXi shows 0.25% overall efficiency factor.
I tried to provide a test environment with shortest hardware paths 
possible.  At least the benchmarks were run 100% reproducable with the 
same binaries.


So I'm really interested if
…
Are these (emulation(only?) related, I guess) performace issues well 
known?  I mean, does somebody know what needs to be done in what area, 
in order to catch up with the other results? So it's just a matter of 
time/resources?
Or are these results surprising and extensive analysis must be done 
before anybody can tell how to fix the IO limitations?


Is the root cause for the problematic low virtio-net throughput 
probably the same as for the disk IO limits?  Both really hurt in my 
use case and the host is not idling in relation, but even showing 
higher load with lower results.  So even if the lower 
user-experience-performance would be considered as toleratable, the 
guests/host ratio was only half dense.


Thanks,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freeb

Re: Network identification problem with Windows 10 on bhyve

[Harry Schmalzbauer]

Am 26.10.2018 um 09:16 schrieb Andrea Venturoli:

Hello.

I've got a FreeBSD 11.2/amd64 + vm-bhyve + Windows 10 pro.
I'm building the network bridge statically, i.e., I have the following 
in /etc/rc.conf:

cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="up addm igb0"


I'm using virtio network drivers and I'm also fixing the MAC address; 
i.e. I have the following in /zroot/vm/win10/win10.conf:

network0_type="virtio-net"
network0_switch="public"
network0_mac="58:9c:fc:00:11:65"
network0_device="tap0"


The Windows machine is connected to an AD domain.



Every time I reboot the server, Windows will think it is in a new 
network: connecting to the box via RDP, a message will pop up asking 
me whether I want the machine to be discoverable on "Network 1".
Next time I reboot and connect via RDP, it will ask about "Network 2"; 
then "Network 3" and so on.


Have you checked inside the guest that it really uses the defined MAC 
with the interface?
Other than that I don't have an idea.  Gateway MAC is also taken into 
account by Windows NetworkListManager...


Good luck,
-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve win-guest benchmark comparing

[Harry Schmalzbauer]

Hello,

I started using bhyve for some of my local setups about one or two years 
ago.  I'm utilizing rc.local along with some homebrew start_if.NG 
scripts to connect tap(4) and ng_bridge(4) with a single vlan(4) uplink 
child, so I know bhyve well enough to know that it isn't comparable in 
many ways with ESXi as a "product", and I'm completely fine with the 
extra jobs I have to do for using bhyve!


But I've always felt that there are significant performance penalties, 
wich hasn't been a big issue for my own guests (tinker and WSUS windows).
Since I wanted to evaluate the possibility to replace ESXi instances 
elsewhere, I decided to run some hopefully meaningful benchmark tests.  
Unfortunately, the performance penalty is much too high.  I'd like to 
share my measurings here.


Host-Config:

    database-
 |                                         |
  da0    da1 <- Windows Server 2012R2, SQLExpress2017
 /  \  |
    | r0 |    |
    S    S    S
    S    S    S
    D    D    D
mps0/vmhba0
|
|         ,- bhyve-ssd (ufs 12.0-beta1)
|    ahci0/vmhba32 –-:
|    |    `- esxi-ssd (6.7)
|    |
32GXeonE34x3.6G(hyperthreading enabled)

So the guest is booting from it's own physical disk (single SSD via mps).

Guest-Config:
When the host was running FreeBSD, the relevant bhyve disk setup reads 
"-s 3,ahci,hd:/dev/da1,hd:/dev/da0"
Likewise, when the host was running ESXi, the corresponding 
disks/vml were attached to the ESXi "SATA Controller" (via RDM).
So in both cases the built-in generic guest-OS (Win2k12R2) AHCI driver 
was in use, for both, the OS-system disk and the db/bechmark disk.

Both hypervisors assign 2 CPU cores (in one package) and 4GB RAM.
The guest operating system of choice is Windows Server 2012R2.  As 
real-world application I chose MS-SQLServerExpress 2017. Simply because 
I looked for a "industry" benchmark tool and found a trial version which 
was easy to setup, bringing test-data along with several workload templates.
After OS-setup was done, all (G)UI-actions were done through RDP session 
in both cases.


Test-Runs:
Each hypervisor had only the one bench-guest running, no other 
tasks/guests were running besides system's native standard processes.
Since the time between powering up the guest and finishing logon 
differed notably (~5s vs. ~20s) from one host to the other, I did a 
quick synthetic IO-Test beforehand.
I'm using IOmeter since heise.de published a great test pattern called 
IOmix – about 18 years ago I guess.  This access pattern has always 
perfectly reflected the system performance for human computer usage with 
non-caculation-centric applications, and still is my favourite, despite 
throughput and latency changed by some orders of manitudes during the 
last decade (and I had defined something for "fio" which mimics IOmix 
and shows reasonable relational results; but I'm still prefering IOmeter 
for homogenous IO benchmarking).


The results is about factor 7 :-(
~3800iops&69MB/s (CPU-guest-usage 42%IOmeter+12%irq)
                vs.
~29000iops&530MB/s (CPU-guest-usage 11%IOmeter+19%irq)


    [with debug kernel and debug-malloc, numbers are 3000iops&56MB/s,
 virtio-blk instead of ahci,hd: results in 5660iops&104MB/s with 
non-debug kernel

 – much better, but even higher CPU load and still factor 4 slower]

What I don't understand is, why the IOmeter process differs that much in 
CPU utilization!?!  It's the same binary on the same OS (guest) with the 
same OS-driver and the same underlying hardware – "just" the AHCI 
emulation and the vmm differ...


Unfortunately, the picture for virtio-net vs. vmxnet3 is similar sad.
Copying a single 5GB file from CIFS share to DB-ssd results in 100% 
guest-CPU usage, where 40% are irqs and the throughput max out at ~40MB/s.
When copying the same file from the same source with the same guest on 
the same host but host booted ESXi, there's 20% guest-CPU usage while 
transfering 111MB/s – the uplink GbE limit.


These synthetic benchmark very well explain the "feelable" difference 
when using a guest between the two hypervisors, but fortunately not by 
that factor most times.  So I continued with the initially aimed 
database test.


Disclaimer: I'm no database expert and it's not about achieving maximum 
performance from DB workload.  It's just about generating reporducable 
CPU-bound load together with IO load to illustrate overall performance 
_differences_.
So I combined two "industry standard" benchmarks from "Benchmark 
Factory" and scaled them (TPC-C by 75 and TCP-H by 3) to generate a 
database with 10GB size.


Interestingly, the difference is by far not as big as expected after the 
previous results.

There's clearly a difference, but the worst case isn't even factor 2.
I did two consecutive runs for each hypervisor. Run4 and Run5 were on 
bhyve, Run6 and Run7 on ESXi.

Please see the graph here:

kern.hz – Windows aequivalent, high CPU usage on bhyve copmared to ESXi

[Harry Schmalzbauer]

Hello,

has anybody found out if Windows does something similar as FreeBSD with 
kern.hz when running on a hypervisor?


I haven't done meaningful measuring, but on bhyve I see 2-digit CPU load 
with idle windows (2012R2) guest.
I never saw that on ESXi, it's much lower there, whil FreeBSD as guest 
on ESXi is even lower when idle.


Can somebody confirm this difference?
Is there a way to spoof hypervisor vendor string (from userland)? Resp. 
has anyone done that to see if windows behaviuor changes?


Thanks,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: pci_virtio_block.c Assertion failed: line 216

[Harry Schmalzbauer]

Am 15.11.2017 um 15:19 schrieb Harry Schmalzbauer:

Bezüglich Peter Grehan's Nachricht vom 03.01.2017 20:36 (localtime):

Hi Harry,


trying to use bhyve(8) with virtio-blk and Windows guest results in core
dump:
Assertion failed: (n >= 2 && n <= BLOCKIF_IOV_MAX + 2), function
pci_vtblk_proc, file usr.sbin/bhyve/pci_virtio_block.c, line
216.
Abort trap (core dumped)

Unfortunately this is on a production-test machine which lacks gdb etc.
Will try to reproduce on antoher machine, but maybe someone already
knows that problem?

  virtio-blk isn't currently supported with Windows guests. You'll need
to use ahci-hd for now.

  However, I do have a fix that can hopefully be committed shortly.

…

Mising in another reply:


Wiadomość napisana przez Harry Schmalzbauer mailto:free...@omnilan.de>> w dniu 03.01.2017, o godz. 20:33:

…

Will try to reproduce on antoher machine, but maybe someone already
knows that problem?

I've seen that problem and fixed it, will upstream the patch later today.

JFYI, fixing
commit:

https://github.com/freenas/os/commit/0e4d6e1826f8aa7041cbeeb4365c797eeec5c5f4

Thanks Jakub for the info.

I can confirm that increasing BLOCKIF_IOV_MAX from 33 to 128, like the
diff shows, solved the problem for me.
I've successfully done some performace tests on Windows7 (virtio-blk vs.
ahci,hd:) and also migrated one Server 2012R2 to bhyve using virtio-blk.

Peter, is your mentioned fix different from just increasing BLOCKIF_IOV_MAX?
If not, would you commit that please?


I hope that I don't bug people knowing better, but this simple diff 
makes virtio-blk usable for Windows 7.

Is there any reason not to commit?

Index: src/usr.sbin/bhyve/block_if.h
===
--- usr.sbin/bhyve/block_if.h   (Revision 325745)
+++ usr.sbin/bhyve/block_if.h   (Arbeitskopie)
@@ -39,7 +39,7 @@
 #include 
 #include 

-#define BLOCKIF_IOV_MAX    33  /* not practical to be 
IOV_MAX */
+#define BLOCKIF_IOV_MAX    128 /* not practical to be 
IOV_MAX */


 struct blockif_req {
    struct iovec    br_iov[BLOCKIF_IOV_MAX];

Thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve networking

[Harry Schmalzbauer]

Bezüglich Paul Esson's Nachricht vom 25.04.2018 23:15 (localtime):
> Hi Rod,
> Can you share a command line for that?  I also tried presenting an
> access port from my switch on a specific VLAN - not trimmed.  Would I
> still have to tag the interface on the guest in that scenario?

Hmm, I lost the overview – I'm not familar with 'vm'.
To filter a specific id (tag/untag frames) inside the guest:
'ifconfig vlan[N] create vlandev vtnet0 vlan '
'ifconfig vlan[N] create vlandev vtnet0 vlan nnnm'

At boot time by rc(8):
vlans_vtnet0="vtnet_dmz vtnet_dmz2"
create_args_vtnet_dmz="vlan "
create_args_vtnet_dmz2="vlan nnnm"

[To optionally also rename the vlan interfaces after manually creating
cloned vlan interfaces, which is what the rc.conf(5) example does:
ifconfig rename vlan0 vtnet_dmz; ifconfig rename vlan0 vtnet_dmz2; ]

Hth,

-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve networking

[Harry Schmalzbauer]

 Bezüglich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime):
> Hi Folks,
>
> I'm struggling with networking when using vm-bhyve on FreeBSD 11.1-RELEASE. I 
> have two NICs and have configured the first (igb0) on a management network 
> and want to use the second (igb1) for VMs.  However, I can't get any VM to 
> communicate through the virtual switch if I have igb1 added to it.  If I take 
> the NIC out of the switch and configure an ipv4 address on it I can reach 
> other hosts on the relevant subnet so I believe the poet set-up is valid.  If 
> I replace igb1 in the switch with igb0, I can then configure VMs on my 
> management network and they have network connectivity.  Can I only use

Hello,

a example of the command you used was nice.
I guess you're using if_bridge(4) – the example would clarify.
But there's ng_bridge(4) and vale(4) also, and others are using
OpenVSwitch...

>  an interface that has an IP address configured at the host level before 
> adding it to the switch?   I've used other 

No.
But the interface has to be in promisc mode. And some offloading
functions must be disabled, but in case of if_bridge(4), it's done
automagically (and reverted if you remove the interface again).

Hth,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve and arp problem

[Harry Schmalzbauer]

Bezüglich Daniel Braniss's Nachricht vom 04.04.2018 15:08 (localtime):
> 
> 
>> On 2 Apr 2018, at 19:19, Harry Schmalzbauer <free...@omnilan.de> wrote:
>>
>> Bezüglich Rodney W. Grimes's Nachricht vom 02.04.2018 18:11 (localtime):
>>>> Bez?glich Daniel Braniss's Nachricht vom 02.04.2018 16:58 (localtime):
>>>>
>>>> ?
>>>>>>
>>>>>> Are you trying to use the HOSTS ip address in the GUEST?
>>>>>
>>>>> the client is using the server?s /usr/local, which is mounted via nfs.
>>>>
>>>> I guess you have reasons to prefer nfs over nullfs.
>>>
>>> You can not nullfs mount from inside a bhyve guest the host file
>>> system would be his reason.
>>
>> Doh, of course – I mind-mixed with jails…  Almost all my FreeBSD
>> "guests" on a bhyve(8) host are jails... Please simply ignore ;-)
>>
>>
>> …
>>>> your results... Else I can't explain what you see other than the above
>>>> scenario.
>>>
>>> Is it still true that if you have a bridge hooked to an interface
>>> you must move the IP assignement from the interace device to the
>>> bridge?
>>
>> I have no idea, I haven't used if_bridge(4) for some years.  Perferably
>> I use vale(4) (if I don't need .11q) or ng_bridge(4) for bhyve(8) setups.
>>
>> -harry
> 
> maybe this could be the cause:
> on the server, every time a client is started iI see:
> kernel: bridge0: error setting interface capabilities on mlxen0

Members added to the bridge will automatically get some (TX) offloadings
disabled, which seems to fail in your case.

man 4 if_bridge:
The TOE, TSO, TXCSUM and TXCSUM6 capabilities on all interfaces added to
the bridge are disabled if any of the interfaces doesn't support/enable
them.  The LRO capability is always disabled.  All the capabilities are
restored when the interface is removed from bridge.  Changing
capabilities in run time may cause NIC reinit and the link flap.

I don't think the kernel-bridge0-logs are related to your ARP issue –
txcsum offloading enabled on if_bridge(4) uplinks cause other errors
(dropped packets – one layer up).

-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve and arp problem

[Harry Schmalzbauer]

Bezüglich Rodney W. Grimes's Nachricht vom 02.04.2018 18:11 (localtime):
>> Bez?glich Daniel Braniss's Nachricht vom 02.04.2018 16:58 (localtime):
>>
>> ?

 Are you trying to use the HOSTS ip address in the GUEST?
>>>
>>> the client is using the server?s /usr/local, which is mounted via nfs.
>>
>> I guess you have reasons to prefer nfs over nullfs.
> 
> You can not nullfs mount from inside a bhyve guest the host file
> system would be his reason.

Doh, of course – I mind-mixed with jails…  Almost all my FreeBSD
"guests" on a bhyve(8) host are jails... Please simply ignore ;-)


…
>> your results... Else I can't explain what you see other than the above
>> scenario.
> 
> Is it still true that if you have a bridge hooked to an interface
> you must move the IP assignement from the interace device to the
> bridge?

I have no idea, I haven't used if_bridge(4) for some years.  Perferably
I use vale(4) (if I don't need .11q) or ng_bridge(4) for bhyve(8) setups.

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve and arp problem

[Harry Schmalzbauer]

Bezüglich Daniel Braniss's Nachricht vom 02.04.2018 16:58 (localtime):

…
>>
>> Are you trying to use the HOSTS ip address in the GUEST?
> 
> the client is using the server’s /usr/local, which is mounted via nfs.

I guess you have reasons to prefer nfs over nullfs.


> so I guess the answer is yes.

I don't see how /usr/local is related to IP address assigning...


…
> the guest has only vtnet0 and lo0
> the ip of the client is obtained via dhcp
> on the server, there is a bridge, bridge0 and it bridges between the taps and 
> the mxlen0

Hmm, do you assing an IP address to your if_tap(4) interfaces at your
bhyve-host?

if_tap(4) is used for interconnecting bhyve(8)'s virtio-net "backend"
with the host's host-IP stack via the character device /dev/tapX.
The if_tap(4) interfaces on the host mustn't get the guest's (clients)
IP address!  The address must be assigned to the if_vtnet(4) interface
_inside_ the guest (I call it the front-end of bhyve's virtio-net) –
done usually in /etc/rc.conf – no relation to /usr/local…

But chances are good I misunderstood something, either your setup or
your results... Else I can't explain what you see other than the above
scenario.

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve and arp problem

[Harry Schmalzbauer]

 Bezüglich Daniel Braniss's Nachricht vom 30.03.2018 13:16 (localtime):
> hi,
> this is my first attempt at bhyve, and so far all seems ok, except
> in my guest, the mac address of the hosting keeps flipping, ie, every 20 
> minutes
> i see a message :
>   … arp: nnn (the hosting ip)  moved from  to 
> on both the host and guest I’m running a very resent -stable.
> the  is the mac of the host nic, while the  is the tap0
>
> i know this looks harmless, but it’s annoying

You can calm it with
'sysctl net.link.ether.inet.log_arp_movements=0'

There's also "net.link.ether.inet.log_arp_wrong_iface" and
"net.inet.ip.check_interface" which influence related behaviour.

You also posted (documentationized IP-addresses):
> I think the problem starts with the host seeing the client/guest on 2 
> interfaces, the nic (mlnxen0) and the tap(tap0)
> on the host:
>
> arp -a
> ...
> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on tap0 expires in 1001 
> seconds [ethernet]
> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on mlxen0 expires in 644 
> seconds [ethernet]

Initially, you reference two MAC-addresses with  and .
The recent post indicates non-different MAC-addresses.

If  and  - resp. xx.xx.xx.xx.xx - are equal (but seen on
different interfaces), this wouldn't get logged I think.
But it was the only harmless case for straight forward setups.
Even with STP/LACP/CARP/etc. in place, "arp: IP-address moved" always
indicates a misconfiguration and I don't know any example where the two
different MAC-Addresses for one IP-address were harmless.
While using a single (locally administrated?) MAC address more than once
sitewide _can_ make sense, having two interfaces on one host which both
are on the same ethernet segment like the two interfaces with the same
MAC address, looks like an unintended setup.

So I strongly suggest to analyze your setup before altering the
mentioned sysctl!!!

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve - windows7 installation

[Harry Schmalzbauer]

Bezüglich Ludovit Koren's Nachricht vom 01.03.2018 19:29 (localtime):
>>>>>> Rodney W Grimes <freebsd-...@pdx.rh.cn85.dnsmgr.net> writes:
> 
>     >> >>>>> Harry Schmalzbauer <free...@omnilan.de> writes:

…
> >> bhyve: passthru device 0/26/0 BAR 0: base 0xd4739000 or size 0x400 not 
> page aligned
> >^
> >> 
> >> bhyve: failed to initialize BARs for PCI 0/26/0
> >> 
> >> 
> >> 
> >> pciconf -vlb ppt0@pci0:0:26:0
> >> ppt0@pci0:0:26:0:   class=0x0c0320 card=0x17df103c chip=0x1e2d8086 
> rev=0x04 hdr=0x00
> >> vendor = 'Intel Corporation'
> >> device = '7 Series/C216 Chipset Family USB Enhanced Host 
> Controller'
> >> class  = serial bus
> >> subclass   = USB
> >> bar   [10] = type Memory, range 32, base 0xd4739000, size 1024, enabled
> >^^
> 
> > Due to the design of the IOMMU you can only manage IO space in page
> > (4096 on x86) granually sizes.  The device your trying to pass in
> > has a 1024 byte memory region that is part of a 4096 byte page that
> > may have other things in it.
> 
> > At this time bhyve does not have any way to deal with this, though some
> > other hypervisors have techniques that make this work.
> 
> I used VirtualBox. I gave a try to bhyve. I am using FreeBSD for all work
> I need. I need windows for some tasks: some VPN (I am not able to
> connect with vpnc, openconnect, racoon, strongswan), Iphone backup,
> etc. I am not able to use a single hypervisor for all of that. Could you
> recommend one?


I know your situation – I tried virtualbox-ose-3.2.8 (~8 yeras ago) and
it's still running on my ancient workstation. Simply works for the one
XP I need to have arround for some of the named reasons.

Regarding USB-related requirements, I always found a device which made
trouble – even with ESXi's generally very well working USB passthrough.
So I use a USB-Device-IP-bridge.  The one I import from USA to Europe,
because of it's yet problemfree design/hardware/(Win-)software, can be
found for ~US$55.- if you search for ID-DS0611-S1 – might be out of
budget for private use though...

I'm very curious about the workaround avg@ posted.
For all server virtualization tasks I switched to bhyve(8) half a year
ago for my local office.  It simply works (for FreeBSD + Win2012R2
guests) and I'm happily finding workarounds for current limitations in
favour of having a bare metal ZFS storage subsystem in parallel to the
hypervisor :-)

Thanks to all coders for this nice OS extention!

-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve - windows7 installation

[Harry Schmalzbauer]

Bezüglich Ludovit Koren's Nachricht vom 01.03.2018 14:33 (localtime):
>>>>>> Harry Schmalzbauer <free...@omnilan.de> writes:

…
> > I'd suggest the following mass storage config (adding a second ODD with
> > the virtio-141 drivers):
> > -s
> > 
> 3,ahci,hd:/dev/zvol/zroot/msw7,sectorsize=512/8192,cd:./en_windows_7_professional_with_sp1_vl_build_x64_dvd_u_677791.iso,cd:./virtio-win-0.1.141.iso
> 
> > After connecting via VNC the vm boots from DVD and you see the setup
> > program. You won't have mouse support, but keyboard's been working, 
> right?
> > So you have been able to finsih setup and login via VNC, still no mouse.
> > Use your existing setup or do a new one to make the last assumptions 
> true.
> > If you booted with the two ISO images, use the keyboard to start
> > "control panel/Device Manager", where you should find the unattached
> > virtio NIC. Installt/Update the driver from there, you can tell the
> > assistant to use e:\ as source (your 2nd. ODD).
> 
> Thank you very much. It is up and running.
> 
> I have an additional question. When I try to run:
> 
> bhyve -c 1 -s 0,hostbridge -s 
> 3,ahci-hd,/dev/zvol/zroot/msw7,sectorsize=512/8192 -s 10,virtio-net,tap1 -s 
> 31,lpc -s 29,fbuf,tcp=0.0.0.0:5900,w=1366,h=768,wait -s 26,passthru,0/26/0 -S 
> -l com1,/dev/nmdm0A -l com2,/dev/nmdm1A -l 
> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -m 4G -H -w windows7
> bhyve: passthru device 0/26/0 BAR 0: base 0xd4739000 or size 0x400 not page 
> aligned
> 
> bhyve: failed to initialize BARs for PCI 0/26/0
> 
> 
> 
> pciconf -vlb ppt0@pci0:0:26:0
> ppt0@pci0:0:26:0:   class=0x0c0320 card=0x17df103c chip=0x1e2d8086 
> rev=0x04 hdr=0x00
> vendor = 'Intel Corporation'
> device = '7 Series/C216 Chipset Family USB Enhanced Host Controller'
> class  = serial bus
> subclass   = USB
> bar   [10] = type Memory, range 32, base 0xd4739000, size 1024, enabled
> 
> 
> The intention is to have usb device in the windows.
> 
> Any suggestions?

Unfortunately I can only tell that this is a known limitation/bug.
For some background, you can see
https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-December/004000.html
and
https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-December/005012.html
and a quote from mav@:
Bezüglich Alexander Motin's Nachricht vom 29.12.2016 11:32 (localtime):
>> ppt0@pci0:6:0:0:class=0x010601 card=0x10601b21 chip=0x06121b21
>> > rev=0x01 hdr=0x00
>> >   bar [10] = type I/O Port, range 32, base 0x5050, size 8, enabled
>> >   bar [14] = type I/O Port, range 32, base 0x5040, size 4, enabled
>> >   bar [18] = type I/O Port, range 32, base 0x5030, size 8, enabled
>> >   bar [1c] = type I/O Port, range 32, base 0x5020, size 4, enabled
>> >   bar [20] = type I/O Port, range 32, base 0x5000, size 32, enabled
>> >   bar [24] = type Memory, range 32, base 0xc3e1, size 512, enabled
> I believe it is bhyve bug, since these values are just what hardware
> reports.  BAR size of 512 bytes indeed does not align to 4K, but this
> not our problem. :)

As long as your controller supports MSI (message signaling interrupts),
you correctly assigned ppt(4) and there's nothing non-hackers can do.
To play the same old song: I'm short in skills resp. time to investigate
further.

Mabye you want to search/file a PR at https://bugs.freebsd.org/bugzilla/

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve - windows7 installation

[Harry Schmalzbauer]

 Bezüglich Ludovit Koren's Nachricht vom 01.03.2018 08:22 (localtime):
>
> Hi,
>
> on FreeBSD 11.1-STABLE #0 r329013, I was trying to install windows7
> using bhyve. I run the following commands:
>
> mkdir -p iso-remaster/virtio
> /usr/local/bin/7z x 
> en_windows_7_professional_with_sp1_vl_build_x64_dvd_u_677791.iso
> -oiso-remaster
> tar xf virtio-win-0.1.141.iso  -C iso-remaster/virtio/
> mkisofs -b boot/etfsboot.com -no-emul-boot -c BOOT.CAT -iso-level 4 -J -l -D 
> -N -joliet-long -relaxed-filenames -v -V "Custom" -udf -boot-info-table 
> -eltorito-alt-boot -eltorito-platform 0xEF -eltorito-boot 
> efi/microsoft/boot/efisys_noprompt.bin -no-emul-boot -o 
> install_windows_w7_professional_x64.iso iso-remaster
>
> ifconfig bridge0 create
> ifconfig bridge0 up
> ifconfig tap1 create
> ifconfig bridge0 addm em0 addm tap1
>
> and I was able to install windows successfully:
>
> bhyve -S -c 1 -s 0,hostbridge -s 
> 3,ahci-hd,/dev/zvol/zroot/msw7,sectorsize=512 -s 
> 4,ahci-cd,install_windows_w7_professional_x64.iso -s 2:0,virtio-net,tap1 -s 
> 31,lpc -s 29,fbuf,tcp=0.0.0.0:5900,w=1366,h=768,wait -l com1,/dev/nmdm0A -l 
> com2,/dev/nmdm1A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -m 
> 4G -H -w windows7
>
>
> The windows seems to be working fine. The problem is, I have no working
> network interface, nor can I install one.
>
> I do not know, what I am doing wrong. Any help appreciated.

If you want to make windows Setup aware of 3rd-party drivers, you need
to do that with some unattended answer file.
Simply adding driver files to the same setup source won't work (while
there is/was a standard path for such things – $$\$OEM$ or the like –
but I don't remember out of mind and I'm even unsure if this was NT only...)

I'm either attaching a second ODD or another HDD, wich provides driver
files on FAT32 – until I find some time to improve C skills and write
floppy support for bhyve(8) ;-) (which would be a very good practicing
project imho).
I'd suggest the following mass storage config (adding a second ODD with
the virtio-141 drivers):
-s
3,ahci,hd:/dev/zvol/zroot/msw7,sectorsize=512/8192,cd:./en_windows_7_professional_with_sp1_vl_build_x64_dvd_u_677791.iso,cd:./virtio-win-0.1.141.iso

After connecting via VNC the vm boots from DVD and you see the setup
program. You won't have mouse support, but keyboard's been working, right?
So you have been able to finsih setup and login via VNC, still no mouse.
Use your existing setup or do a new one to make the last assumptions true.
If you booted with the two ISO images, use the keyboard to start
"control panel/Device Manager", where you should find the unattached
virtio NIC. Installt/Update the driver from there, you can tell the
assistant to use e:\ as source (your 2nd. ODD).

Hth,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: superfluous host interfaces

[Harry Schmalzbauer]

 Bezüglich Vincent Hoffman-Kazlauskas's Nachricht vom 27.02.2018 21:09
(localtime):
>
> On 27/02/2018 08:17, Harry Schmalzbauer wrote:
>> Bezüglich Paul Vixie's Nachricht vom 27.02.2018 07:14 (localtime):
>>>
>>> Harry Schmalzbauer wrote:
>>>> Bezüglich Ruben's Nachricht vom 26.02.2018 11:34 (localtime):

…

>> If you have only one "LAN" sharing all VMs, the one additional interface
>> is neglectable.
>> But my setups are different.
>> I have almost as many different 802.11q separated ethernet collsion
>> domains (VLANs) as VMs.
>> That's what ESXi's portgroup is used for.  I need a separate switch for
>> each VLAN (guests mustn't be able to sniff traffic etc.).
>>
> Untested by me with bhyve but it looks like net/openvswitch
> (http://www.openvswitch.org/) could be useful to you. As I say though
> untested by me so cant speak for performance etc.

I made a local OVS port which incorporated netmap support, but I was
told that OVS lost attraction for the netmap team, when I tried to solve
some problems, since netmap patches from upstream were highly linux
specific and considered obsolete.
Intel contributed some DPDK resources, which look interesting (and seem
to perform great on linux).
But OVS itself would need more resources to get better support on
FreeBSD and additionally a huge ammount of resources to get DPDK|netmap
enabled.
In 10GE days, OVS without netmap|DPDK doesn't make much sense imho.

We have netmap/vale(4), which could be extenden to cover a small subset
of OVS feature, with porbably moderate ammount of resources.
So in my opinion, for those using bhyve(4) as slim hypervisor, the not
very slim OVS doesn't fit well overall, especially due to perfomance
constraints.

A portgroup/vlan filter+manager on top of vale(4) was a much better
companion for bhyve(4).  I'll start immediately when retireing (learing
to use whatever debugger will be arround then...)  ;-)

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: superfluous host interfaces

[Harry Schmalzbauer]

Bezüglich Paul Vixie's Nachricht vom 27.02.2018 07:14 (localtime):
> 
> 
> Harry Schmalzbauer wrote:
>> Bezüglich Ruben's Nachricht vom 26.02.2018 11:34 (localtime):
>>> On 26/02/2018 10:56, Harry Schmalzbauer wrote:
>> …
>>>> Another, personally very significant, reason is that you'll get a
>>>> superfluous host interface for each if_bridge(4), which makes the
>>>> output
>>>> of plain ifconfig(8) kind of unreadable.
>> …
>>> By superflous host interfaces, do you mean the tap interfaces configured
>>> for each vm together with the bridge interfaces they are "bundled" in?
>>
>> Additionally to the if_tap(4) ethernet host interfaces, you also get
>> if_bridge(4) ethernet interfaces, named bridgeX if I remember correctly.
> 
> you do not remember correctly.

Please see next para.


>> [mm1.redbarn:amd64] egrep 'bridge|tap' /etc/rc.conf
>> autobridge_interfaces="bridge0"
>> autobridge_bridge0="tap* igb1"
>> cloned_interfaces="bridge0 tap0 tap1 tap2 tap3 tap4 tap5 tap6 tap7"
>> ifconfig_bridge0="inet 24.104.150.210/27"
>> ifconfig_bridge0_ipv6="inet6 2001:559:8000:cd::2/64 auto_linklocal up"
>> ifconfig_tap0="up"
>> ifconfig_tap1="up"
>> ifconfig_tap2="up"
>> ifconfig_tap3="up"
>> ifconfig_tap4="up"
>> ifconfig_tap5="up"
>> ifconfig_tap6="up"
>> ifconfig_tap7="up"
>> [mm1.redbarn:amd64] ifconfig | egrep '^(bridge|tap)'
>> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>> mtu 1500

That's what I mean; and it's named bridgeX, so memeory works in that
case ;-)
If you have only one "LAN" sharing all VMs, the one additional interface
is neglectable.
But my setups are different.
I have almost as many different 802.11q separated ethernet collsion
domains (VLANs) as VMs.
That's what ESXi's portgroup is used for.  I need a separate switch for
each VLAN (guests mustn't be able to sniff traffic etc.).


…

>> And using ng_bridge(4) instead of if_bridge(4) doesn't change the need
>> for if_tap(4).  Only with vale(4) switches, bhyve(8) was able to provide
>> virtio-net connection wihtout "spamming" the host's ethernet interface
>> list (no tapX, no bridgeX).
> 
> how did you get bhyve to use the netmap API rather than the tap
> character special device?

Not me, peter commited the following:
https://svnweb.freebsd.org/base/head/usr.sbin/bhyve/pci_virtio_net.c?r1=288470=293459

>From that an, you don't need to add any host ethernet
devices/interfaces, simply start your VM with e.g. "-s
1:0,virtio-net,vale0:1,mac=02:03:04:05:06:07".

As long as you created vale0, port [:]1 will be created dynamically and
also destroyed after shutdown.

Again, to mimic ESXi's portgroups, you need one vale(4) switch for each
VLAN.  And to uplink, you need to utilize if_valn(4), which forces
netmap emulated mode.
If you don't have/need VLANs, you can uplink a supported NIC via native
netmap support, and additionally gain significatn efficiency improvements.
Unfortunately, at least if_vlan(4) uplinks, don't work reliably.  After
some short time, the complete network netmap(4) subsystem locks up.
I talked with Vincenco Maffione (a member of Liugi Rizzo's netmap(4)
team of University of Pisa) and fixing emulated netpmap mode on FreeBSD
doesn't have really high priority there, since such a setup is
considered as weak design.  For sure, it's a hack/workarround, but we
don't have VLAN/portgroup support in vale(4) nor in byhve(8), and
writing my own userland filter is beyond my scope.

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: superfluous host interfaces

[Harry Schmalzbauer]

Bezüglich Ruben's Nachricht vom 26.02.2018 11:34 (localtime):
> On 26/02/2018 10:56, Harry Schmalzbauer wrote:
…
>> Another, personally very significant, reason is that you'll get a
>> superfluous host interface for each if_bridge(4), which makes the output
>> of plain ifconfig(8) kind of unreadable.
…
> 
> By superflous host interfaces, do you mean the tap interfaces configured
> for each vm together with the bridge interfaces they are "bundled" in?

Additionally to the if_tap(4) ethernet host interfaces, you also get
if_bridge(4) ethernet interfaces, named bridgeX if I remember correctly.
The if_bridge(4) host interface is for control purposes only on a VM-SDN
host – at least with my setups.  I never needed to make use of IP
numbered bridges.  And I don't need to utilize any if_bridge(4) features
like STP, so I consider the bridgeX host interfaces as superfluous in
the VM-SDN use case.

I'd call the if_tap(4) host interfaces likewise superfluous – you would
only need the corresponding character devices – but that's been
implemented long before the need for SDN setups, so it is like it is.
And using ng_bridge(4) instead of if_bridge(4) doesn't change the need
for if_tap(4).  Only with vale(4) switches, bhyve(8) was able to provide
virtio-net connection wihtout "spamming" the host's ethernet interface
list (no tapX, no bridgeX).


> Overall I'm very happy with my bhyve setups atm. If there are any
> speed-/administrative-advantages that come with bridge_ng however, I'm
> very interested in switching to such a setup (or at least play with it).
> I'm running my vm's without any helper project so I'm flexible enough to
> do some fiddling :P
> 
> Do you know of any documentation on using bridge_ng together with bhyve?
> My search-engines don't turn up much Im affraid and I haven't stumbled
> on it before.

Unfortunately it's not too easy to get started with netgraph.
Besides numerous man pages for the different nodes (ng_bridge(4) e.g.),
I only know the following source for a good overview:
http://www.netbsd.org/gallery/presentations/ast/2012_AsiaBSDCon/Tutorial_NETGRAPH.pdf

One convenience disadvantage with ng_bridge(4) is that you have to
assign MACs manually, while if_bridge(4) does that itself (adjustable by
sysctl net.link.bridge.inherit_mac).
And you need to script all setups yourself.  Almost all of my setups
seem to be awkward enough that I always had to do some local scripting,
so that wasn't really a disadvantage for me.

If you're happy with your setup, I don't think you gain anything from
switching to ng_bridge(4), besides learning to control netgraph(4)
(which is very desirable imho).
I haven't had time left to do useful benchmarking regarding ng_bridge(4)
vs. if_bridge(4). I even don't know if netgraph nodes are still limited
to single threads.  But rough load comparings on a IvyBride machine
showed similar resource usage for both bridges, both easy capable of
1GbE saturation with small frames (while I remember one run with
ng_bridge(4) and if_vmnet(4), which couldn't deliver 1GbE speed, and I
wanted to falsify for vmnet/tap difference... just ran out of time :-( ).

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve manager

[Harry Schmalzbauer]

Bezüglich Ruben's Nachricht vom 26.02.2018 10:15 (localtime):

…
> On 26/02/2018 10:09, Harry Schmalzbauer wrote:
>>  Bezüglich tech-lists's Nachricht vom 25.02.2018 14:14 (localtime):
>>> Hi,
>>>
>>> What do folks use for their bhyve guest management?
>>>
…
>> Copy'n'pasting relevant lines from 'cat ~/vm-launches.txt'
>>
>> Reason:
>> I very much dislike utilizing if_bridge(4), instead would prefere
>> vale(4), but that's reproducibly locking up in real world setups.
>> So my choice is ng_bridge(4), which isn't covered by any byhve(4) helper
> Hi Harry,
> 
> 
> What are your reasons for preferring ng_bridge over the "normal" bridge?

Two very different main reasons:
if_bridge(4) is very standards compliant (e.g. that different reserved
MAC addresses won't get forwarded – don't know any explicit examples out
of mind), which was problematic for some SDN setups (software defined
networking, in means of sharing a PHY for multiple VMs and
simultaniously interconnect VMs to VMs)

Another, personally very significant, reason is that you'll get a
superfluous host interface for each if_bridge(4), which makes the output
of plain ifconfig(8) kind of unreadable.
For VM SDN, I don't need/want those host interfaces, despite they don't
do any harm.

vale(4) was extremely convinient.  Simply create a switch, then each VM
attaches on the fly :-)
Unfortunately, I'm unable to debug the lockups and my setups was kind of
hacky, since I haven't used NIC's native netmap(4) support, but used
emulated netmap(4) for if_vlan(4).  This leads to loss of almost all
performance advantages, but left convinience advantages.  Unfortunately,
emulated netmap(4) is supposed to have some unresolved problems on
FreeBSD and upstream hackers consider my hacky setup as wrong by nature
– which it is technically speaking.  For real-world usagen, one would
need to code a VLAN filter between bhyve(4) and vale(4). Skillwise, I'm
not the one :-(

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve manager

[Harry Schmalzbauer]

 Bezüglich tech-lists's Nachricht vom 25.02.2018 14:14 (localtime):
> Hi,
>
> What do folks use for their bhyve guest management?
>
> I have always spun bhyve guests up by hand but now I'm considering
> streamlining the process. What do you use?

Just to feed the poll:
Copy'n'pasting relevant lines from 'cat ~/vm-launches.txt'

Reason:
I very much dislike utilizing if_bridge(4), instead would prefere
vale(4), but that's reproducibly locking up in real world setups.
So my choice is ng_bridge(4), which isn't covered by any byhve(4) helper
project afaik.
2nd reason:
People are working on native config file support.  There's
https://reviews.freebsd.org/D2448 and https://reviews.freebsd.org/D2505
e.g., which seems desireable to me and I liked it very much at the time
of evaluating.
Unfortunately, neither time nor skills allow me to seriously
contribute.  But that's the way to go imho, so I dropped hacking
vm-bhyve/chyves in favour of coming up with small, _bhyve.conf
compatible_ shell parser.  Unfortunately, even for that, I couldn't find
any time.

So still 'cat ~/vm-launches.txt', where my MAC database resides ;-)

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve passthrough problems

[Harry Schmalzbauer]

 Bezüglich Peter Grehan's Nachricht vom 18.01.2018 09:58 (localtime):
> Hi Mario,
>
>> I'm trying to passthrough a firewire pci card to a windows 7 64 guest.
>
>  First, you'll need to verify if that card supports MSI or MSI-x since
> that is required by bhyve for passthru support. This can be seen with
> 'pciconf -lbc' on the FreeBSD host.
>
>> VM exits with:
>>
>> Assertion failed: (mr->name == memp->name), function
>> unregister_mem, file /usr/src/usr.sbin/bhyve/mem.c, line 265.
>> Abort trap
>
>  This is bug in bhyve when running Windows with PCI devices that have
> more than 1 memory BAR (e.g. Intel igb ethernet devices), due to the
> order that Windows determines BAR sizes. I've been meaning to fix this
> for a long time so will try and get to it.

Thanks in advance, Peter!

Inderictly related, but also a big improvement for bhyve, could you have
a minute on that too:
https://lists.freebsd.org/pipermail/freebsd-virtualization/2017-November/005875.html

Thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Question about bhyve

[Harry Schmalzbauer]

 Bezüglich Marcelo Araujo's Nachricht vom 12.01.2018 03:20 (localtime):
> Hi,
>
> You can give a try using devel/libhyve-remote, it is a VNC server based on
> libvncserver, clipboard works there.

Can someone please give a on-line explanation how this port is to be
used – not for end users as far as I understand the description.
The sentence above confuses me even more, because of the "libvncserver"!

I haven't found time to look into it, but always wondered if it replaces 
bhyve-vnc-_server_ (so one needs to re-compile bhyve(8) from base after 
installing that port) or if it is a VNC client, lending X naming convensions 
where the client - which is really the server - is correctly named "server".

AFAIK VNC is different, it's just a X11 client – correct me if I'm wrong.

But since you suggest this as a solution to a end-user-task problem, a
developer library isn't probably what it is!?!

Thanks,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Unable to use renamed tap device

[Harry Schmalzbauer]

 Bezüglich Shawn Webb's Nachricht vom 30.12.2017 07:10 (localtime):
> Hey All,
>
> I'm in the process of reorganizing my bhyve setup on my development
> laptop. I'd like to have rename the tap devices to match the name of
> the VM so that it's easier to keep track of. Otherwise, I have to keep
> a spreadsheet of (tap3 -> win10-vm, tap4 -> fbsd-vm).
>
> It appears bhyve doesn't attach renamed tap devices. Here's the steps
> I used:
>
> ifconfig bridge0 create
> ifconfig tap0 create name fbsd-01
> ifconfig bridge0 addm em0 addm fbsd-01 up
> sh /usr/share/examples/bhyve/vmrun.sh -t fbsd-01 [normal vmrun.sh arguments 
> here]
>
> (In this example, em0 is the physical network device connected to the
> LAN. I want to share em0 with the host and the guest via the bridge.)
>
> The net.link.tap.up_on_open sysctl node is set to 1. Normally, when
> bhyve starts up (with tap0 instead of fbsd-01), it opens the tap
> device and UPs it. I'm not seeing that same behavior with a renamed
> tap interface:
>
> $ ifconfig ld-03_01
> ld-03_01: flags=8903 metric 0 mtu 1500
> options=8
> ether 00:bd:df:e9:f6:04
> groups: tap
> media: Ethernet autoselect
> status: no carrier
> nd6 options=29
>
> So, it seems to me that bhyve doesn't like it when tap devices are
> renamed. Can anyone shed some light on this?

Unfortunately not too much, besides this report:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219746

As long as the control device name isn't also renamed, tap/vmnet
renaming should be blocked in rc(8), since it can't work the way it is
at the moment.
Unfortunately my C skills don't allow me to check if renaming the
control device could/should be implemented in whatever functions are
responsible for IF renaming (meaning to make it a kernel task).  Or if
it would be better to utilize devd(8)!?

-harry 
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve. Unable install Windows 7/Server 2008R2

[Harry Schmalzbauer]

 Bezüglich Lars Engels's Nachricht vom 05.12.2017 17:57 (localtime):
> On Tue, Dec 05, 2017 at 04:49:45AM -0800, Rodney W. Grimes wrote:
>>> Hi,
>>>
   bhyve -c 2 -s 0,hostbridge -s 
 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0
>>>   For win7/2k8*, the sector size presented to the guest has to be forced 
>>> to 512 bytes i.e. for the ahci-cd slot, the config would look like:
>>>
>>>   -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0,sectorsize=512
>> Should we start a wiki page on "Guest Quirks"?
> Please do.

I'd vote for making ,"sectorsize=512/4096" a artificial default.
I'm not entire sure about the different Windows versions, but since
BHYVE_UEFI.fd isn't able to access drives other than 512b sector size, I
think it doesn't make sense to provide 8k drives by default in case of
default ZFS volume.
If I remember correctly Win6.3 (8.1/Server2k12) did support 4k and 8k
native drives, with the limitation that "windows internal database -
mssql" might not be usable, but I couldn't UEFI boot unless I changed
sectrosize to 512 and stripezesize to 4k/8k.
I guess not may other operating systems support bigger sector sizes as
well as FreeBSD does today.
So this artifical format might be what most times makes most sense.

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: pci_virtio_block.c Assertion failed: line 216

[Harry Schmalzbauer]

Bezüglich Peter Grehan's Nachricht vom 03.01.2017 20:36 (localtime):
> Hi Harry,
> 
>> trying to use bhyve(8) with virtio-blk and Windows guest results in core
>> dump:
>> Assertion failed: (n >= 2 && n <= BLOCKIF_IOV_MAX + 2), function
>> pci_vtblk_proc, file usr.sbin/bhyve/pci_virtio_block.c, line
>> 216.
>> Abort trap (core dumped)
>>
>> Unfortunately this is on a production-test machine which lacks gdb etc.
>> Will try to reproduce on antoher machine, but maybe someone already
>> knows that problem?
> 
>  virtio-blk isn't currently supported with Windows guests. You'll need
> to use ahci-hd for now.
> 
>  However, I do have a fix that can hopefully be committed shortly.
…

Mising in another reply:

>> Wiadomość napisana przez Harry Schmalzbauer <free...@omnilan.de
>> <mailto:free...@omnilan.de>> w dniu 03.01.2017, o godz. 20:33:

…
>> Will try to reproduce on antoher machine, but maybe someone already
>> knows that problem?
>
> I've seen that problem and fixed it, will upstream the patch later today.
>
> JFYI, fixing
> commit:
https://github.com/freenas/os/commit/0e4d6e1826f8aa7041cbeeb4365c797eeec5c5f4

Thanks Jakub for the info.

I can confirm that increasing BLOCKIF_IOV_MAX from 33 to 128, like the
diff shows, solved the problem for me.
I've successfully done some performace tests on Windows7 (virtio-blk vs.
ahci,hd:) and also migrated one Server 2012R2 to bhyve using virtio-blk.

Peter, is your mentioned fix different from just increasing BLOCKIF_IOV_MAX?
If not, would you commit that please?

Thanks,

-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve and windows 2012/2016

[Harry Schmalzbauer]

 Bezüglich Александр Поволоцкий's Nachricht vom 13.10.2017 12:12
(localtime):
> Hello
>
> I'm trying to install Windows 2012 evaluation in bhyve. It looks like
> neither HDD (ahci-hd) nor CD (ahci-cd) are available.
>
> Command is
>
>  /usr/sbin/bhyve  -c 2 -m 17179869184 -H -A -w -s 0:0,hostbridge-s
> 3:0,ahci-hd,/usr/jails/jails-data/serv1c-data/dsk1.vhd,sectorsize=512
> -s 2:0,ahci-cd,/usr/jails/src/iso/windows2012.iso  -s
> 4:0,e1000,tap6,mac=00:a0:98:cf:ab:82 -s 5:0,virtio-rnd  -s
> 6,fbuf,tcp=0.0.0.0:5902,w=1024,h=768,wait -s 30,xhci,tablet -s 31,lpc
> -l com1,stdio -l bootrom,/usr/local/cbsd/upgrade/patch/efi.fd serv1c
>
> Only X: drive is available, seems to be some memory image
>
> What should I check? What could have I done wrong?

Hmm, nut sure about vhd image struture, but I doubt you can use that as
block backend.
You can attach the HDD and ODD to the same controller.
If you use md(4) as hdd backend, does taht change anything (mdconfig -a
-t swap -s 10G and "-s
3:0,ahci,hd:/dev/mdX,cd:usr/jails/src/iso/windows2012.iso")?
Or a zvol or sparse file (truncate -s 10g
/your/data/path/bhyve-testdisk.file) insetad of md, just to rule out the
vhd image file.

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve ppt usage can cause severe RAM corruption [Was: Re: panic: Memory modified after free in zio_create, passthru in use]

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 11.06.2017 12:37 (localtime):
>  Bezüglich Harry Schmalzbauer's Nachricht vom 06.06.2017 14:03 (localtime):
>>  Hello,
>>
>> suddenly, I'm getting this error:
>> /lib/libc.so.7: Undefined symbol "xdr_accepted_reply"
>>
>> Very mysterious: It showed up on a running system, which worked
>> flawlessly for some hours. And that host has root-fs (/) mounted
>> readonly from a memorydisk. So to my understanding, it's completely
>> impossible that /lib/libc.so.7 is corrupted since last boot.
>>
>> I'm completely out of ideas what could cause this strange error during
>> "normal" operation.
>>
>> Normal operation in this case is serving as a bhyve test machine.
>> I first noticed that error after one guest - with passthru device
>> attached - was shut down.
>>
>> My suspicion is some undiscovered passthru interference... Since I
>> noticed one other _very_ strange passthru-effect:
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215740
> Hello,
>
> this time I caught a panic with a debuging kernel under 11.1-BETA1,
> which again occured after shuting down a VM which had ppt in use:
>
…
> Please, can anybody of the xperts add a comment?

It turned out that it's a problem with PCIe cards which don't support
FLR or cards, which are not PCIe, even if they have FLR capabilitiy.

jhb@ helped me to diagnose this.

Unfortunately I once forgot to manually bring down the passthrough-nics
in question, which resulted in a completely destroyed ZFS pool.
That hurted, so I won't rely on manual intervention before shutting down
(I had to recreate the complete (system) pool).
Unfortunately my skills don't allow me to help fixing the root cause, so
I created a little rc(8) script, which should protect reliably.
Please see also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222937

Since it's quite small overhead, I'll also attach it here (to be copied
to /etc/rc.d).

-harry

#!/bin/sh
#

# PROVIDE: pciptdetach
# REQUIRE: swap
# BEFORE: devd
# KEYWORD: shutdown

. /etc/rc.subr

name=pciptdetach
rcvar=pciptdetach_enable

load_rc_config ${name}

: ${pciptdetach_enable:="YES"}

start_cmd="true"
stop_cmd="${name}"

pciptdetach()
{
sysctl -n hw.hv_vendor | grep -q bhyve || return 0

echo "Disabling passthrough adapters:"

pptcandidate=`pciconf -l | grep -v -E \
  "^([[:blank:]]|hostb|virtio|isab)[^@]+" | sed -n -E \
 's/^[[:blank:]]*(^[[:alnum:]]+)@([^[:blank:]]+)(:[[:blank:]]).*$/\2/p'`

for pcidev in ${pptcandidate}; do

drv_class=`pciconf -lv | grep -A 3 "@${pcidev}" | sed -n -E -e \
 's/^[[:blank:]]*class[[:blank:]]+=[[:blank:]]+([^[:blank:]].*)$/\1/p' \
-e 's/^([[:alnum:]]+)@.*$/\1/p' | tr '\n' ' '`

# Don't disable mass storage devices, might be busy for shutdown
[ X"${drv_class}" = X"${drv_class%mass storage*}" ] || continue

# Make sure network adapters don't have active vlan(4) clones.
if [ -z "${netstoped}" ] &&
[ X"${drv_class}" != X"${drv_class%network*}" ]
then
/etc/rc.d/netif stop >/dev/null 2>&1 && netstoped=y
fi

# Non-PCIe devices and PCIe devices without FLR support are
# known to cause RAM corruption.
if ! pciconf -lc ${pcidev} | grep -A 20 PCI-Express |
grep -q "[[:blank:]]FLR"
then
devctl disable ${pcidev} >/dev/null 2>&1 ||
echo " ${drv_class%% *}:FAILED"
fi

done
}

run_rc_command "$1"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Any support creating a Windows Server 2012 unattended install

[Harry Schmalzbauer]

 Bezüglich Frank Leonhardt's Nachricht vom 13.07.2017 10:40 (localtime):
> On 13/07/2017 09:35, Harry Schmalzbauer wrote:
>> Bezüglich Paul Webster's Nachricht vom 13.07.2017 10:22 (localtime):
>>> Ah ha we can now see installs, perfect thank you harry! just what I
>>> needed I thought we still had no way of seeing the install process
>> You can run anything that provides a UEFIx64 loader with VNC-graphics,
>> due to the ongoing effort of many bhyve developers.
>> Thank goes to them!
> I'll second that! I wish I'd known this 18 months ago (but then it
> wasn't available then).
>
> Does you know if it works with OS/2 and Windows XP/98?

You can use the CSM-UEFI Firmware. This emulates some/all rudementary
(PC-)BIOS functions to boot legacy MBR systems.
But you won't have VGA support. So I'd estimate you have no chance to
install win98 and a good chance to run XP installer in EMS/unattended
mode (only, no graphical installer/login!). Unfortunately I never had
the pleasure to play with OS/2. If it depends on VGA or doesn't provide
any form of serial console, you'll be out of luck currently.

bhyve(8) aimed to be a modern design, tolerating limited versatile
usage, as far as I can tell.

People are working on VGA-passthrough support, so one day you could
probably be able to install _one_ legacy OS, if UEFI-CSM doesn't need
additional modification (which I could imagine might be needed due to
ACPI dependency of the guest OS? I can't tell since I have a only very
brief overview of the BIOS boot process, maybe the epxerts here can
answer this).
But it will always follow different goals than virtualbox and/or QEMU
do. The latter are designed to provide versitale guest support!

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Any support creating a Windows Server 2012 unattended install

[Harry Schmalzbauer]

Bezüglich Paul Webster's Nachricht vom 13.07.2017 10:22 (localtime):
> Ah ha we can now see installs, perfect thank you harry! just what I
> needed I thought we still had no way of seeing the install process

You can run anything that provides a UEFIx64 loader with VNC-graphics,
due to the ongoing effort of many bhyve developers.
Thank goes to them!

-Harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Any support creating a Windows Server 2012 unattended install

[Harry Schmalzbauer]

 Bezüglich Paul Webster via freebsd-virtualization's Nachricht vom
13.07.2017 10:07 (localtime):
> Or alternatively any of you guys have one that I can use?
>
> I was following https://wiki.freebsd.org/bhyve/Windows but I can not seem
> to get it to work; though it may be because I had to convert the
> 'install.esd' file to 'install.wim' as the disk does not have one.
>
> If I just the ISO I ended up creating in virtualbox it just seems to go
> into a standard install, but that might be because it detected a graphics
> adapter.

Is it about unattended itself, or do you just want to install
WinServer2012R2 in a bhyve guest? For the latter you can use GOP/fbuf.
See https://wiki.freebsd.org/bhyve/UEFI – no need for unattended detour
any more.

Regarding unattended setups, I don't have any reusable xmls, but answers
to common pitfalls. Feel free to contact me if you identified a specific
one.

-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve.8 referencing non-existing vmm.4 [Was: Re: VNC stuff]

[Harry Schmalzbauer]

Bezüglich Rodney W. Grimes's Nachricht vom 14.06.2017 00:40 (localtime):

…
>> I can confirm that the 302504 MFC (r304424) works with BSD guests.
>> Also the man page correctly was included in the MFC ? my fault.
>>
>> But I just checked these, which needs MFC:
>> 307517,314342,316357,317545 and the already mentioned
>>
>>> r31 should be MFC'ed asap.
> 
> These 5 are now merged to 11-stable.

Great, thanks.
Unfortunately I forgot about the reference to vmm(4) in bhyve.8.
There is no vmm.4, is/was there?

Thanks,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: PCIe passthrough really that expensive?

[Harry Schmalzbauer]

Bezüglich Anish's Nachricht vom 13.06.2017 06:13 (localtime):
> Hi Harry,
>>Any hints highly appreciated!
…
> 
> Now use cpuset to route IRQ#265 to say core 0
> 
> $cpuset -l 0 -x 265
> 
> Again use cpuset to force VM[PID 1222] to run on all core except #0  
> 
> root@svmhost:~ # ps
> 
>  PID TT  STATTIME COMMAND
> 
> 
> 
> 1222  1  I+   5:59.45 bhyve: vm1 (bhyve)
> 
> 
> VM can run on all cores except #0.
> 
> $ cpuset -l 1-3 -p 1222
> 
> 
> You can monitor guest due to interrupts using
> 
> $root@svmhost:~ # bhyvectl --get-stats --vm= --cpu= |
> grep external
> 
> vm exits due to external interrupt  27273
> 
> root@svmhost:~ # 

Thank you very much for that detailed explanation.  I didn't thought
that cpuset(1) could also pin IRQ (handlers?) to specific cpus.

In my case, I couldn't get a noticable difference.
Since I have hyperthreading enabled on a single-socket quad core, I
pinned cpu2+3 to the bhyve pid (2vCPUs) and for testing copied two times
the same 8GB file ofer NFS, while having ppt's IRQ handler pinned to
CPU1, CPU4, CPU3 and CPU2.  So two times different hostCPUs than guest
uses and two times the same.  I couldn't see any load nor performance
difference and similar 'vm exits due to external interrupt' count groth.

To name numbers: 1st CPU had about 40k "vm exits due to external
interrupt" per 8GP transfer, the other vCPU ~160k "vm exits due to
external interrupt".

Like mentioned, different host-CPU-pinning didn't influence that noticable.

Thanks four this lesson!

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: VNC stuff

[Harry Schmalzbauer]

 Bezüglich Peter Grehan's Nachricht vom 13.06.2017 06:39 (localtime):
> Hi Rod,
>
>> r302504 is not likely to be MFC'ed at this time, this is the e1000
>> driver
>> and man page.  This code has known issues at this time and is an action
>> item list on the bhyve developers work list.
>
>  The e1000 driver was MFC'd 11 months ago. The issues are currently
> only with Windows - it works fine with Linux/*BSD/Solaris. I'm hoping
> that the Windows issue can be fixed before 11.1.

While we're at issues:

virtio-net as one unresolved:
[bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215737

Most likely this won't be able to fix in time for 11.1, but I wanted to
point out, since users depending on jumbo frame to/from the guest, are
forced to use the e1000, and if there are problems with different
guests, there's no more option left for them.

Thanks,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: VNC stuff

[Harry Schmalzbauer]

Bezüglich Rodney W. Grimes's Nachricht vom 13.06.2017 03:02 (localtime):
> [ Charset ISO-8859-1 unsupported, converting... ]
>>  Bez?glich Zane C. B-H.'s Nachricht vom 12.06.2017 19:30 (localtime):
>>> On 2017-06-12 09:39, Peter Grehan wrote:
 Hi Zane,

> I assume that means the VNC/fbuf stuff as well? Not seeing that that
> currently in 11-STABLE.

  Sure you don't mean 10-STABLE ? The uefi graphics code was committed
 11 months ago:


 https://svnweb.freebsd.org/base/stable/11/usr.sbin/bhyve/pci_fbuf.c?view=log

>>>
>>> Hmm... interesting. Not seeing that at all mentioned in the man for
>>> bhyve.
>>
>> True.
>> MFCing r302504-31 for usr.sbin/bhyve/bhyve.8 would document the
>> precious bhyve(8) development progress for upcoming 11.1 users.
> 
> I think you meant r302504,r31.
> 
> r302504 is not likely to be MFC'ed at this time, this is the e1000 driver
> and man page.  This code has known issues at this time and is an action
> item list on the bhyve developers work list.

I can confirm that the 302504 MFC (r304424) works with BSD guests.
Also the man page correctly was included in the MFC – my fault.

But I just checked these, which needs MFC:
307517,314342,316357,317545 and the already mentioned

> r31 should be MFC'ed asap.


Thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: VNC stuff

[Harry Schmalzbauer]

 Bezüglich Zane C. B-H.'s Nachricht vom 12.06.2017 19:30 (localtime):
> On 2017-06-12 09:39, Peter Grehan wrote:
>> Hi Zane,
>>
>>> I assume that means the VNC/fbuf stuff as well? Not seeing that that
>>> currently in 11-STABLE.
>>
>>  Sure you don't mean 10-STABLE ? The uefi graphics code was committed
>> 11 months ago:
>>
>>
>> https://svnweb.freebsd.org/base/stable/11/usr.sbin/bhyve/pci_fbuf.c?view=log
>>
>
> Hmm... interesting. Not seeing that at all mentioned in the man for
> bhyve.

True.
MFCing r302504-31 for usr.sbin/bhyve/bhyve.8 would document the
precious bhyve(8) development progress for upcoming 11.1 users.

-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

panic: Memory modified after free in zio_create, passthru in use [Was: 11.1-pre runtime Undefined symbol "xdr_accepted_reply" /lib/libc.so.7]

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 06.06.2017 14:03 (localtime):
>  Hello,
>
> suddenly, I'm getting this error:
> /lib/libc.so.7: Undefined symbol "xdr_accepted_reply"
>
> Very mysterious: It showed up on a running system, which worked
> flawlessly for some hours. And that host has root-fs (/) mounted
> readonly from a memorydisk. So to my understanding, it's completely
> impossible that /lib/libc.so.7 is corrupted since last boot.
>
> I'm completely out of ideas what could cause this strange error during
> "normal" operation.
>
> Normal operation in this case is serving as a bhyve test machine.
> I first noticed that error after one guest - with passthru device
> attached - was shut down.
>
> My suspicion is some undiscovered passthru interference... Since I
> noticed one other _very_ strange passthru-effect:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215740

Hello,

this time I caught a panic with a debuging kernel under 11.1-BETA1,
which again occured after shuting down a VM which had ppt in use:
cpuid = 5
KDB: stack backtrace:
#0 0x805bf327 at kdb_backtrace+0x67
#1 0x8057f266 at vpanic+0x186
#2 0x8057f2e3 at panic+0x43
#3 0x8082eaeb at trash_ctor+0x4b
#4 0x8082aaec at uma_zalloc_arg+0x52c
#5 0x813b54a6 at zio_add_child+0x26
#6 0x813b5a05 at zio_create+0x385
#7 0x813b6de2 at zio_vdev_child_io+0x232
#8 0x81396be0 at vdev_mirror_io_start+0x370
#9 0x813bc629 at zio_vdev_io_start+0x4a9
#10 0x813b76bc at zio_execute+0x36c
#11 0x813b6868 at zio_nowait+0xb8
#12 0x81396bec at vdev_mirror_io_start+0x37c
#13 0x813bc383 at zio_vdev_io_start+0x203
#14 0x813b76bc at zio_execute+0x36c
#15 0x805d10dd at taskqueue_run_locked+0x13d
#16 0x805d1e78 at taskqueue_thread_loop+0x88
#17 0x80543844 at fork_exit+0x84

#0  doadump (textdump=) at pcpu.h:222
#1  0x8057ece0 in kern_reboot (howto=260) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:366
#2  0x8057f2a0 in vpanic (fmt=, ap=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:759
#3  0x8057f2e3 in panic (fmt=) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:690
#4  0x8082eaeb in trash_ctor (mem=,
size=, arg=, flags=)
at /usr/local/share/deploy-tools/RELENG_11/src/sys/vm/uma_dbg.c:80
#5  0x8082aaec in uma_zalloc_arg (zone=0xf8001febc680,
udata=0xf8001ad5f340, flags=)
at /usr/local/share/deploy-tools/RELENG_11/src/sys/vm/uma_core.c:2152
#6  0x813b54a6 in zio_add_child (pio=0xf8026f350b88,
cio=0xf8002478b7b0)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:460
#7  0x813b5a05 in zio_create (pio=0xf8026f350b88, spa=, txg=433989, bp=,
data=0xfe0058afa000,
size=1024, type=,
priority=ZIO_PRIORITY_ASYNC_WRITE, flags=,
vd=,
offset=, zb=,
pipeline=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:690
#8  0x813b6de2 in zio_vdev_child_io (pio=0xf8026f350b88,
bp=, vd=, offset=325398016,
data=, size=1024, type=,
flags=1048704, done=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1141
#9  0x81396be0 in vdev_mirror_io_start (zio=0xf8026f350b88)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:488
#10 0x813bc629 in zio_vdev_io_start (zio=0xf8026f350b88)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3143
#11 0x813b76bc in zio_execute (zio=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1681
#12 0x813b6868 in zio_nowait (zio=0xf8026f350b88)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1739
#13 0x81396bec in vdev_mirror_io_start (zio=0xf8026f7a7b88)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:488
#14 0x813bc383 in zio_vdev_io_start (zio=0xf8026f7a7b88)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3021
#15 0x813b76bc in zio_execute (zio=)
at
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1681
#16 0x805d10dd in taskqueue_run_locked
(queue=0xf8001ab5a700) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/subr_taskqueue.c:454
#17 0x805d1e78 in taskqueue_thread_loop (arg=) at
/usr/local/share/deploy-tools/RELENG_11/src/sys/kern/subr_taskqueue.c:741
#18 0x80543844 in fork_exit (callout=0x805d1df0
, arg=0xf8001aa90720, frame=0xfe043f609ac0)
at 

Re: PCIe passthrough really that expensive?

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 09.06.2017 10:22 (localtime):
> Bezüglich Anish's Nachricht vom 08.06.2017 14:35 (localtime):
>> Hi Harry,
>>> I thought I'd save these expensive VM_Exits by using the passthru path.
>> Completely wrong, is it?
>>
>> It depends on which processor you are using. For example APICv was
>> introduced in IvyBridge which enabled h/w assisted localAPIC rather than
>> using s/w emulated, bhyve supports it on Intel processors. 
…
> I'm still usign IvyBridge (E3v2) with this "new" machine, but haven't
> ever heard/thought about APCIv!

It seems APICv is available on IvyBridge-EP (Xeon E5/E7v2) only, not for
E3v2 :-(
Furthermore, if I didn't miss anything in the datasheets, no currently
available E3 Xeon offers local APIC virtualization. Can somebody of the
xperts confirm that?


…
>> Can you run a simple experiment, assign pptdev interrupts to core that's
>> not running guest/vcpu? This will reduce #VMEXIT on vcpu which we know
>> is expensive.
> Interesting approach.  But I have no idea how I should assign a PCIe
> specific core to a PCIe dev.  Is it pptdev specific? The tunables in
> device.hints(5) can't be used for that, can they?

I wasn't able to find out how to do that.
Any hints highly appreciated!

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: PCIe passthrough really that expensive?

[Harry Schmalzbauer]

Bezüglich Anish's Nachricht vom 08.06.2017 14:35 (localtime):
> Hi Harry,
>>I thought I'd save these expensive VM_Exits by using the passthru path.
> Completely wrong, is it?
> 
> It depends on which processor you are using. For example APICv was
> introduced in IvyBridge which enabled h/w assisted localAPIC rather than
> using s/w emulated, bhyve supports it on Intel processors. 
> 
> Intel Broadwell introduced PostedInterrupt which enabled interrupt to
> delivered to guest directly, bypassing hypervisor[2] for
> passthrough devices. Emulated devices interrupt will still go through
> hypervisor. 

That's very interesting, thanks so much!
I wasn't ware that there were post VT-c improvements, guess I'll have to
refresh my very basic knowledge urgently.
I'm still usign IvyBridge (E3v2) with this "new" machine, but haven't
ever heard/thought about APCIv!


> You can verify capability using sysctl hw.vmm.vmx. What processor you
> are using for these performance benchmarking?

hw.vmm.vmx.vpid_alloc_failed: 0
hw.vmm.vmx.posted_interrupt_vector: -1
hw.vmm.vmx.cap.posted_interrupts: 0
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0
hw.vmm.vmx.cap.invpcid: 0
hw.vmm.vmx.cap.monitor_trap: 1
hw.vmm.vmx.cap.unrestricted_guest: 1
hw.vmm.vmx.cap.pause_exit: 1
hw.vmm.vmx.cap.halt_exit: 1
hw.vmm.vmx.initialized: 1
hw.vmm.vmx.cr4_zeros_mask: 18446744073708017664
hw.vmm.vmx.cr4_ones_mask: 8192
hw.vmm.vmx.cr0_zeros_mask: 18446744071025197056
hw.vmm.vmx.cr0_ones_mask: 3

I did very simply 'time cp' with 8GB files over NFSv4, which come from
ZFS-cache on the remote side and locally watching host+guest vmstat.


> Can you run a simple experiment, assign pptdev interrupts to core that's
> not running guest/vcpu? This will reduce #VMEXIT on vcpu which we know
> is expensive.

Interesting approach.  But I have no idea how I should assign a PCIe
specific core to a PCIe dev.  Is it pptdev specific? The tunables in
device.hints(5) can't be used for that, can they?
It seems pptdev(0) couldn't get a man page yet, but I'll have a look at
the sources, maybe I can find hints until earth has done it's job and
present you the same nice sunshine I'm enjoying today :-)

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

11.1-pre runtime Undefined symbol "xdr_accepted_reply" /lib/libc.so.7

[Harry Schmalzbauer]

 Hello,

suddenly, I'm getting this error:
/lib/libc.so.7: Undefined symbol "xdr_accepted_reply"

Very mysterious: It showed up on a running system, which worked
flawlessly for some hours. And that host has root-fs (/) mounted
readonly from a memorydisk. So to my understanding, it's completely
impossible that /lib/libc.so.7 is corrupted since last boot.

I'm completely out of ideas what could cause this strange error during
"normal" operation.

Normal operation in this case is serving as a bhyve test machine.
I first noticed that error after one guest - with passthru device
attached - was shut down.

My suspicion is some undiscovered passthru interference... Since I
noticed one other _very_ strange passthru-effect:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215740

Thanks for any hints,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: What is the recommended version of virtio-win for bhyve

[Harry Schmalzbauer]

 Bezüglich Christian Stærk's Nachricht vom 07.04.2017 13:15 (localtime):
> Hello,
>
> I am trying to get Windows 10 up and running on bhyve on a FreeBSD
> 11.0 host.
>
> The installation went smooth and I have Windows running on the
> framebuffer.
>
> I installed the NetKVM driver from virtio-win-0.1.126.iso, but it
> looks like windows is onlu receiving packets but are unable to send.
>
> I remeber some talk earlier about not all versions og virtio-win
> working correctly.
>
> Can anoyone here recommend a version, that works with bhyve and
> windows 10?

According to my directory content
(ftp://ftp.omnilan.de/pub/Driver/Windows/), the latest version working
with win 10 was 0.1.118 by the end of 12/2016 (bisected).

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: tap on lagg ?

[Harry Schmalzbauer]

Bezüglich Vincent Olivier's Nachricht vom 21.03.2017 16:51 (localtime):
> Hi, I can confirm that ping works but ssh and (I haven’t tried anything else, 
> but I assume you are right) TCP as a whole doesn’t work.
> 
> From there, I guess that, since I haven’t changed the MTU on the 1G 
> interfaces (only on the 10G ones which are isolated from the 1G network). 
> this leaves offloading.
> 
> Should I disable it (which ones)? On all the physical interfaces or also on 
> the lagg and maybe bridge?

You seem to have the following problem:
if_bridge(4) tries to disable TXCSUM on all members added.
But you add if_lagg(4), which doesn't pass those requests to it's
members, but simply ignores the request.
So you need to manually -txcsum (-txcsum6), e.g. in rc.conf when you set
them "up".

Unofrtunately I don't know how offloading is implemented generally, nor
how it works for if_igb(4), so I haven't thought about the reason yet,
why you need to disable TXCSUM.
Much more important, does it also affect TSO?
I can't tell, maybe someone with more knowledge can jump in.

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: tap on lagg ?

[Harry Schmalzbauer]

 Bezüglich Vincent Olivier's Nachricht vom 20.03.2017 23:32 (localtime):
> Hello,
>
> Sorry for waiting so long. I don’t know if i’m doing it right but I tried «  
> -vlanhwtag » all the interfaces and I’m still having problems. Namely (as I 
> didn’t have this information before) that all participating interfaces in the 
> bridge itself are in promiscuous mode (and, if that is related) I cannot ssh 
> into the host machine from any bhyve virtual machine. My goal is to be able 
> to ssh and mount host nfs exports onto the VMs. Doing a « -promisc » on all 
> the interfaces won’t change anything. Can someone help? Pleas find below a 
> ifconfig dump.

I'd go for tcpdump.
First, check that routing is no issue.  In your constellation I guess
VMs Ips are in the 192.168.1.0/24 network, correct?
Else make sure your default gateway does
routing/deflection/icmp-redirection.

Then watch 'tcpdump -n -e - s 150 -i bridge0' on the host and the like
inside your VM (vtnet?)
Start with ping and check if ARP is working.
Also 'arp -a' on host and VM provides fundamentally information to find
the problem.
If ARP and icmp (ping) work but TCP (ssh) not, it's PMTU or offloading
related most likely.

-harry


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [CFT] emulators/open-vm-tools{,-nox11} update to 10.1.0

[Harry Schmalzbauer]

 Bezüglich Steve Wills's Nachricht vom 01.03.2017 06:57 (localtime):
> Hi All,
>
> Thanks to everyone who tested.
>
> I've updated the patch and packages in the same location and fixed a
> number of issues. Please re-test if you can.
>
> The only remaining issue that I know of is building with libunwind
> installed. If you run into an issue buidling, uninstall libunwind.
> Upstream is looking at that.
>
> This is likely the final change before I commit this, which currently I
> plan to do on 3/15. So if you use it, now is your chance to test and
> report issues.

Thanks for your work!

I'm new to open-vm-tools, until now I always took the source from the
ESXi provided iso which contains vmware-freebsd-tools.tar.gz and
compiled modules under vmware-tools-distrib/lib/modules/source/
For stable/11, there's some adoption to do, so I thought it's a good
time to try open-vm-tools.

All my guests don't have X11, so I'm only testing the no-x11 version.

vmmemctl.ko seems to work and vmtoolsd also seems to do it's job, I can
shutdown the machine and viclient reports guest's IP adderss.

I don't use vnx, I just can report that it attaches.
But there are no drivers for vmci nor pvscsi in open-vm-tools also, right?

What's vmblock.ko providing?

Thanks,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Providing serial consoles to multiple bhyve guests

[Harry Schmalzbauer]

 Bezüglich C. L. Martinez's Nachricht vom 17.02.2017 15:29 (localtime):
> On Fri, Feb 17, 2017 at 01:36:52PM +0100, Harald Schmalzbauer wrote:
>>  Bezüglich C. L. Martinez's Nachricht vom 17.02.2017 09:50 (localtime):
…
>>>  I have tried to add vnc channel, but it doesn't works ..:(
>> I haven't had VNC problems besides keymapping. It's done twice and I
>> haven't found a way or a client to disable keymapping locally on a
>> session basis.
>> Have you used the "wait" option for fbuf? Did the vm start to run after
>> VNC connection attempt?
>>
> Yes, but I don't see anything in vncviewer console. Can be a problem with 
> graphics controller on my laptop (my laptop is the bhyve host)?

Just to be sure: There was a UEFI loader in use? fbuf is only for GOP,
so if you boot with CSM, you won't see anything since there's no VGA
emulation. Only UEFI-GOP.

Best,

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Zvol, Bhyve/Pfsense and config restore

[Harry Schmalzbauer]

Bezüglich Rajil Saraswat's Nachricht vom 09.02.2017 15:51 (localtime):
>  How can i access the vm filesystem on the host?
>>
>> Which volmode is in use for vmpool/os5 ('zfs get volmode vmpool/os5')?
>>
>> I don't know what patritioning scheme pfsens uses, but 'gpart show
>> /dev/zvol/vmpool/os5' would be of interest.
>>
>> You need volmode "geom" and something like /dev/zvol/vmpool/os5p2 or
>> /dev/zvol/vmpool/os5a for your mount special I guess.
>>
>> -harry
>>
> 
> The volmode was set as default. After changing it to geom, i was able
> to access the zvol and copy the file.
> 
> Thanks for the tip!

Glad to hear you got it working.
'default' references "vfs.zfs.vol.mode":
sysctl -d vfs.zfs.vol.mode
vfs.zfs.vol.mode: Expose as GEOM providers (1), device files (2) or neither

It's probably modified unintentionally on your host. On the other hand I
remember someone documented that under unclear circumstances, it needs
to be set explicitly to 'geom' even with the sysctl beeing still the
default "1" (=geom).

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Zvol, Bhyve/Pfsense and config restore

[Harry Schmalzbauer]

 Bezüglich Rajil Saraswat's Nachricht vom 09.02.2017 14:04 (localtime):
> Hello,
>
> I am trying to setup a pfsense instance under bhyve. Until now i had a
> standalone machine for pfsense but it is going away for RMA so i am
> trying to replace it with a bhyve instance. The /cf/conf/config.xml file
> needs to be copied over to the bhyve instance.
>
> I created the zvol (zfs create -sV 8G "vmpool/os5") and used this zvol
> in the bhyve vm. To copy the config.xml, i mounted the zvol on the host
> which worked. But I am unable to list the contents of the zvol.
>
> # mount /dev/zvol/vmpool/os5 /mnt/tst
>
> #df
>
> /dev/zvol/vmpool/os5  
> 8106716   69927451188 0%/mnt/tst
>
> #ls -la /mnt/tst
>
> ls: /mnt/tst: Bad file descriptor
>
> How can i access the vm filesystem on the host?

Which volmode is in use for vmpool/os5 ('zfs get volmode vmpool/os5')?

I don't know what patritioning scheme pfsens uses, but 'gpart show
/dev/zvol/vmpool/os5' would be of interest.

You need volmode "geom" and something like /dev/zvol/vmpool/os5p2 or
/dev/zvol/vmpool/os5a for your mount special I guess.

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [Bug 215740] [bhyve] utilizing passthru breaks raw device usage with virtio-blk | ahci-hd

[Harry Schmalzbauer]

 Bezüglich bugzilla-nore...@freebsd.org's Nachricht vom 04.01.2017 21:37
(localtime):
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215740
>
…
> --- Comment #1 from Peter Grehan  ---
> Would you be able to post a verbose dmesg (boot -v) ?

Due to lack of knowledge I experimented with BIOS VT-d settings "Remap
Interrupt enable" and "Remap DMA enable".
No difference in any combination.
Is there anything else I can provide to help narrowing down the problem?
Has anybody else ever tried such a VM (using character special file as
virtio-blk backend together with a passthru device)? With success?

Thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

pci_virtio_block.c Assertion failed: line 216

[Harry Schmalzbauer]

 Dear all,

trying to use bhyve(8) with virtio-blk and Windows guest results in core
dump:
Assertion failed: (n >= 2 && n <= BLOCKIF_IOV_MAX + 2), function
pci_vtblk_proc, file usr.sbin/bhyve/pci_virtio_block.c, line
216.  
Abort trap (core dumped)

Unfortunately this is on a production-test machine which lacks gdb etc.
Will try to reproduce on antoher machine, but maybe someone already
knows that problem?

Thanks,
-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Jumbo frames truncated at 4084 bytes by virtio-net? (using if_bridge(4) and vtnet(4) with mtu 9000)

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 21.12.2016 17:40 (localtime):
>  Bezüglich Harry Schmalzbauer's Nachricht vom 21.12.2016 17:08 (localtime):
>
> …
>> Guest flow:
>> 16:57:06.642073 00:a0:98:73:9f:42 > 96:07:e9:78:c6:ac, ethertype IPv4
>> (0x0800), length 4085: 172.21.35.1 > 172.21.35.32: ICMP echo request, id
>> 27401, seq 0, length 4051
>> 16:57:06.642233 96:07:e9:78:c6:ac > 00:a0:98:73:9f:42, ethertype IPv4
>> (0x0800), length 4084: truncated-ip - 1 bytes missing! 172.21.35.32 >
>> 172.21.35.1: ICMP echo reply, id 27401, seq 0, length 405
>>
>> Now my problem is that I can't simply keep guest's mtu at 1500, since
>> the host will send jumbo frames as answer wich never get through virtio-net.
>>
>> Does anybody have an idea how to fix/work arround?
> For the records, replacing virtio-net with e1000 (as bhyve
> PCI-slot-resident) solves the jumbo frame issue!
> It seems 82545EM is fully emulated (not 82545GM which doesn't support
> jumbo frames) :-)
>
> Thank you very much for this nice work!
>
> mav's commit (r302504) mentions heavy performance penalties (factor 2).

Real world usage shows slightly heavier penalty.
Simply doing 'make package-depends-list' of x11/kde4 takes 1:22 with
virtio-net and 3:25 with e1000 (nfsv4 mounted ports tree).

I've filed a bug report:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215737

Hopefully fixing that isn't too complicated...

thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Unhandled ps2 keyboard keysym with UEFI-edk2-bootrom and bhyve(8)

[Harry Schmalzbauer]

Bezüglich Peter Grehan's Nachricht vom 13.12.2016 06:57 (localtime):
> Hi Harry,
> 
>> Unfortunately the VNC part seems to have keyscan problems:
>> Unhandled ps2 keyboard keysym 0xc4
>> Unhandled ps2 keyboard keysym 0xf6
>> Unhandled ps2 keyboard keysym 0xff7f
>> Just to list a view...
>>
>> The really unfortunate problem is that some characters are completely
>> missing, while others are just shifted (involving even meta keys).
>> I haven't found any rule yet, it looks like a mixture of en-US and de-DE
>> layout.
>> Most unfortunate, I can't login because of the missing characters :-( So
>> close yet so far...
> 
>  Would you be able to list the key mappings that don't work (and also
> the VNC client you are using) ?

Sorry for the long delay,

I tried a view VNC clients for Windows and FreeBSD, but all show exactly
the same disfunctional/shifted keys (german maps here).

It seems that keymapping is done twice?!?
If I use en_US layout on the machine running VNC Client, I get correct
_german_ keystrokes inside the guest (since the guest was configured for
german layout)

Windows clients "VNC viewer plus" and "RealVNC" most times can't connect
because »RFB protocol error: unknown rect encoding 255«
Early connection during guest's boot process allows these to connect!?!

I have absolutely no clue about VNC, so I guess it's simply my fault. If
anybody knows how to feed VNC-client with unmapped keystrokes please
drop me a note! Or what these RFB protocol errors mean...

Here's a list I created before trying en_US layout...

keycode 11+s (2): 0x22 becomes 0xc4 (keycode 48+s)
keycode 12+s (3): 0xa7 becomes '' (Unhandled ps2 keyboard keysym 0xa7)
keycode 15+s (6): 0x26 becomes 0x2f (keycode 16+s)
keycode 16+s (7): 0x2f becomes 0x5f (keycode 61+s)
keycode 17+s (8): 0x28 becomes 0x29 (keycode 18+s)
keycode 18+s (9): 0x29 becomes 0x3d (keycode 19+s)
keycode 19+s (0): 0x3d becomes 0xfe50 (keycode 21+s)

keycode 20+s: 0x3f becomes 0x5f (keycode 61+s, like kc16+s)
keycode 20: 0xdf becomes '' (Unhandled ps2 keyboard keysym 0xdf)

keycode 21+s: 0xfe50 become '' (Unhandled ps2 keyboard keysym 0xfe50)
keycode 21: 0xfe51 become '' (Unhandled ps2 keyboard keysym 0xfe51)

keycode 34+s: 0xdc becomes '' (Unhandled ps2 keyboard keysym 0xdc)
keycode 34: 0xfc becomes '' (Unhandled ps2 keyboard keysym 0xfc)

keycode 35+s: 0x2a becomes 0x28 (keycode 17+s)
keycode 35: 0x2b becomes 0xfe51 (keycode 21)

keycode 47+s: 0xd6 becomes '' (Unhandled ps2 keyboard keysym 0xd6)
keycode 47: 0xf6 becomes '' (Unhandled ps2 keyboard keysym 0xf6)

keycode 48+s: 0xc4 becomes '' (Unhandled ps2 keyboard keysym 0xc4)
keycode 48: 0xe4 becomes '' (Unhandled ps2 keyboard keysym 0xe4)

keycode 49+s: Unhandled ps2 keyboard keysym 0xb0
keycode 49: Unhandled ps2 keyboard keysym 0xfe52

keycode 51+s: 0x27 becomes 0xc4 (keycode 48+s)
keycode 51:  0x23 becomes 0x33 (keycode 12)

keycode 59+s: 0x3b becomes 0xd6 (keycode 47+s)
keycode 60+s: 0x3a becomes 0xd6 (keycode 47+s, like kc59+s)

keycode 61+s: 0x5f becomes 0x3f (keycode 20+s)
keycode 61: 0x2d becomes 0xdf (keycode 20)

keycode 94: 0x3c becomes 0x2c (keycode 59)
keycode 94+s: 0x3e becomes 0x3a (keycode 60+s)

-harry

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ASM1062 AHCI timeouts, ppt(4) BAR aligning [Was: Re: svn commit: r309251 - head/sys/dev/ahci]

[Harry Schmalzbauer]

Bezüglich Alexander Motin's Nachricht vom 29.12.2016 11:32 (localtime):
> On 29.12.2016 10:35, Harry Schmalzbauer wrote:
>> I'd like to report that this doesn't fix timeouts for me (applied to
>> 11-stable).
>>
>> For example my REV120 works without problems on Intel-AHCI but not on
>> ASM1062-AHCI.
>> Even attaching gives different output. Both look fine at first:
>> #cd0 at ahcich0 bus 0 scbus5 target 0 lun 0
>> #cd0:  Removable CD-ROM SCSI device
>> #cd0: Serial Number 0C1E4D046E5DFF18
>> #cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO
>> 8192bytes)
>>
>> When attached to the Intel-AHCI, it's followed by
>> +cd0: Attempt to query device size failed: NOT READY, Medium not present
>> while attaching to ASM1062 it reads (!?)
>> -cd0: 0MB (1 0 byte sectors)
>>
>> Then these timeouts occur:
>> ahcich7: Timeout on slot 11 port 0
>> ahcich7: is  cs 0c00 ss  rs 0c00 tfd 6051 serr
>>  cmd 0004cb17
>> ahcich7: Timeout on slot 24 port 0
>> ahcich7: is  cs 0180 ss  rs 0180 tfd 2051 serr
>>  cmd 0004d817
>> ahcich7: Timeout on slot 6 port 0
>> ahcich7: is  cs 0060 ss  rs 0060 tfd 2051 serr
>>  cmd 0004c617
>> ahcich7: Timeout on slot 20 port 0
>> ahcich7: is  cs 0018 ss  rs 0018 tfd 2051 serr
>>  cmd 0004d417
>>
>> Also IDENT (via camcontrol) "hangs" for 20 seconds, but finally succeeds.
> 
> I think problem may be different in your case.  The HBA still reports
> that command is not completed by the device.  Unfortunately I don't have
> those fancy drives to try, but I'll try to reproduce it with regular CD
> drive when I get back home after short New Year holidays.

Oic, then I need to test the patch with regular SSD usage!
I have had problems with the asm1062 when I used it for a "roaming" SSD,
backing virtio-blk/ahci,hd:/dev/ada4. After some mentionable I/O there
were _always_ outages which I don't remember exactly, but moving the SSD
from asm-ahci to intel-ahci made them vanish.
I'll see if these are now solved for the ASM1062. Otherwise I'll report.


>> Btw: I already found out that extending ppt(4) to support unaligned base
>> address register wouldn't be too easy.
>> Initially I added that ASM1062 card to use it for byhve(8) passthrough.
>> Unfortunately that doesn't work:
>> bhyve: passthru device 6/0/0 BAR 5: base 0xc3e1 or size 0x200 not
…
> 
> I believe it is bhyve bug, since these values are just what hardware
> reports.  BAR size of 512 bytes indeed does not align to 4K, but that is
> not our problem. :)
> 
>> Are there any recommendations for AHCI (SATA-PCIe) controller
>> cards/chips that do work (both, for byhve passthrough and also as plain
>> AHCI provider)?
> 
> Please don't mix multiple unrelated questions in one email.

Yes, sorry, I should have sent that in two different mails. Took the
wrong route because I thought others who are possibly searching/trying
low-power SATA passthrough controllers could find this (useful)...


> There is very little reasonable external AHCI controllers on the market
> now.  I am not sure anything other then Marvell and ASmedia were
> released at all in last years since 6Gbps SATA came out.  Marvell and
> ASmedia probably worth each other, while later Marvell may be slightly
> better on functionality (number of ports and FBS PMP support), but they
> are both desktop products.  If you need this in server environment --
> think about about SAS adapter like LSI.  Or just use on-board Intel
> AHCI, since they are probably the best om reliability you may get out of
> SATA.


Thanks for your hints!
Usually I go with LSI2008 for such cases, but this time the additional
7W power consumption for only _one_ roaming SATA-SSD seemd
inappropriate. Furthermore, I only have one spare slot, so I got a card
with the ASM1062 and an Etron EJ168 USB 3.0 combined.

That was exactly what I'd want to have as passthrough for my guest :-)
Hopefully bhyve(8)/ppt will make that possible in the future.

Thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

ASM1062 AHCI timeouts, ppt(4) BAR aligning [Was: Re: svn commit: r309251 - head/sys/dev/ahci]

[Harry Schmalzbauer]

 Bezüglich Alexander Motin's Nachricht vom 28.11.2016 17:23 (localtime):
> Author: mav
> Date: Mon Nov 28 16:23:32 2016
> New Revision: 309251
> URL: https://svnweb.freebsd.org/changeset/base/309251
>
> Log:
>   Process port interrupt even is PxIS register is zero.
>   
>   ASMedia ASM1062 AHCI chips with some fancy firmware handling PMP inside
>   seems sometimes forgeting to set bits in PxIS, causing command timeouts.
>   Removal of this check fixes the issue by the theoretical cost of slightly
>   higher CPU usage in some odd cases, but this is what Linux does too.
>   
>   MFC after:  1 month
>
> Modified:
>   head/sys/dev/ahci/ahci.c
>
> Modified: head/sys/dev/ahci/ahci.c
> ==
> --- head/sys/dev/ahci/ahci.c  Mon Nov 28 15:14:31 2016(r309250)
> +++ head/sys/dev/ahci/ahci.c  Mon Nov 28 16:23:32 2016(r309251)
> @@ -1169,8 +1169,6 @@ ahci_ch_intr(void *arg)
>  
>   /* Read interrupt statuses. */
>   istatus = ATA_INL(ch->r_mem, AHCI_P_IS);
> - if (istatus == 0)
> - return;
>  
>   mtx_lock(>mtx);
>   ahci_ch_intr_main(ch, istatus);
> @@ -1187,8 +1185,6 @@ ahci_ch_intr_direct(void *arg)
>  
>   /* Read interrupt statuses. */
>   istatus = ATA_INL(ch->r_mem, AHCI_P_IS);
> - if (istatus == 0)
> - return;
>  
>   mtx_lock(>mtx);
>   ch->batch = 1;

Hello,

I'd like to report that this doesn't fix timeouts for me (applied to
11-stable).

For example my REV120 works without problems on Intel-AHCI but not on
ASM1062-AHCI.
Even attaching gives different output. Both look fine at first:
#cd0 at ahcich0 bus 0 scbus5 target 0 lun 0
#cd0:  Removable CD-ROM SCSI device
#cd0: Serial Number 0C1E4D046E5DFF18
#cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO
8192bytes)

When attached to the Intel-AHCI, it's followed by
+cd0: Attempt to query device size failed: NOT READY, Medium not present
while attaching to ASM1062 it reads (!?)
-cd0: 0MB (1 0 byte sectors)

Then these timeouts occur:
ahcich7: Timeout on slot 11 port 0
ahcich7: is  cs 0c00 ss  rs 0c00 tfd 6051 serr
 cmd 0004cb17
ahcich7: Timeout on slot 24 port 0
ahcich7: is  cs 0180 ss  rs 0180 tfd 2051 serr
 cmd 0004d817
ahcich7: Timeout on slot 6 port 0
ahcich7: is  cs 0060 ss  rs 0060 tfd 2051 serr
 cmd 0004c617
ahcich7: Timeout on slot 20 port 0
ahcich7: is  cs 0018 ss  rs 0018 tfd 2051 serr
 cmd 0004d417

Also IDENT (via camcontrol) "hangs" for 20 seconds, but finally succeeds.

Btw: I already found out that extending ppt(4) to support unaligned base
address register wouldn't be too easy.
Initially I added that ASM1062 card to use it for byhve(8) passthrough.
Unfortunately that doesn't work:
bhyve: passthru device 6/0/0 BAR 5: base 0xc3e1 or size 0x200 not
page aligned
That's the ASM1062:
ppt0@pci0:6:0:0:class=0x010601 card=0x10601b21 chip=0x06121b21
rev=0x01 hdr=0x00
bar   [10] = type I/O Port, range 32, base 0x5050, size 8, enabled
bar   [14] = type I/O Port, range 32, base 0x5040, size 4, enabled
bar   [18] = type I/O Port, range 32, base 0x5030, size 8, enabled
bar   [1c] = type I/O Port, range 32, base 0x5020, size 4, enabled
bar   [20] = type I/O Port, range 32, base 0x5000, size 32, enabled
bar   [24] = type Memory, range 32, base 0xc3e1, size 512, enabled

Are there any recommendations for AHCI (SATA-PCIe) controller
cards/chips that do work (both, for byhve passthrough and also as plain
AHCI provider)?

Thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Jumbo frames truncated at 4084 bytes by virtio-net? (using if_bridge(4) and vtnet(4) with mtu 9000)

[Harry Schmalzbauer]

Bezüglich Andrey V. Elsukov's Nachricht vom 22.12.2016 11:13 (localtime):
> On 21.12.2016 19:08, Harry Schmalzbauer wrote:
>>  Hello,
>>
>> I'm having guest connectivity problems with jumbo frames.
>>
>> Since if_bridge(4) requires all interfaces to have the same MTU (and the
>> uplink interface also handles iscsi traffic, which greatly benefits from
>> jumbo frames), I
>> 'ifconfig create vmnet0 mtu 9000'
>>
>> Now my problem is that I can't simply keep guest's mtu at 1500, since
>> the host will send jumbo frames as answer wich never get through
>> virtio-net.
>>
>> Does anybody have an idea how to fix/work arround?
> 
> This looks like the problem with mbufs bigger than PAGE_SIZE.
> Do you see some denied requests in the `netstat -m` output?


Thanks for your attention!

Nope, there are no denied mbuf requests after sending icmp echo-request
through virtio-net with all participants' MTU set to 9000:
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied

The icmp echo-reply just doesn't make it through virtio-net (the
echo-request reaches the destination host, and the echo-reply reaches
vmnet0!).
Keeping everything untouched, just
replacing '-s 5,virtio-net,vmnet0'
with '-s 5,e1000,vmnet0', I get the reply into the guest:
ping -D -s 8972 mirashare
PING mirashare.egn.mo1.omnilan.net (172.21.34.11): 8972 data bytes
8980 bytes from 172.21.34.11: icmp_seq=0 ttl=64 time=1.590 ms

Thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Jumbo frames truncated at 4084 bytes by virtio-net? (using if_bridge(4) and vtnet(4) with mtu 9000)

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 21.12.2016 17:08 (localtime):

…
> Guest flow:
> 16:57:06.642073 00:a0:98:73:9f:42 > 96:07:e9:78:c6:ac, ethertype IPv4
> (0x0800), length 4085: 172.21.35.1 > 172.21.35.32: ICMP echo request, id
> 27401, seq 0, length 4051
> 16:57:06.642233 96:07:e9:78:c6:ac > 00:a0:98:73:9f:42, ethertype IPv4
> (0x0800), length 4084: truncated-ip - 1 bytes missing! 172.21.35.32 >
> 172.21.35.1: ICMP echo reply, id 27401, seq 0, length 405
>
> Now my problem is that I can't simply keep guest's mtu at 1500, since
> the host will send jumbo frames as answer wich never get through virtio-net.
>
> Does anybody have an idea how to fix/work arround?

For the records, replacing virtio-net with e1000 (as bhyve
PCI-slot-resident) solves the jumbo frame issue!
It seems 82545EM is fully emulated (not 82545GM which doesn't support
jumbo frames) :-)

Thank you very much for this nice work!

mav's commit (r302504) mentions heavy performance penalties (factor 2).
Has anyone tried the influence of disabling/keeping offload functions in
the guest?

Thanks,

-Harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Jumbo frames truncated at 4084 bytes by virtio-net? (using if_bridge(4) and vtnet(4) with mtu 9000)

[Harry Schmalzbauer]

 Hello,

I'm having guest connectivity problems with jumbo frames.

Since if_bridge(4) requires all interfaces to have the same MTU (and the
uplink interface also handles iscsi traffic, which greatly benefits from
jumbo frames), I
'ifconfig create vmnet0 mtu 9000'

Afert adding it to the bridge 'ifconfig bregn addm vmnet0' and setting
guest mtu to 9000 (which vtnet(4) claims to support), I can transceive
frames larger than 1500,
which this flow from the guest's vtnet(4) interface demonstrates:
16:54:36.672709 00:a0:98:73:9f:42 > 96:07:e9:78:c6:ac, ethertype IPv4
(0x0800), length 4084: 172.21.35.1 > 172.21.35.32: ICMP echo request, id
56840, seq 0, length 4050
16:54:36.672791 96:07:e9:78:c6:ac > 00:a0:98:73:9f:42, ethertype IPv4
(0x0800), length 4084: 172.21.35.32 > 172.21.35.1: ICMP echo reply, id
56840, seq 0, length 4050
On the host this looks similar.

Now with a payload size of 4043 instead of 4042 bytes, the reply never
makes it through virtio-net:
Host flow:
16:57:06.641382 00:a0:98:73:9f:42 > 96:07:e9:78:c6:ac, ethertype IPv4
(0x0800), length 4085: 172.21.35.1 > 172.21.35.32: ICMP echo request, id
27401, seq 0, length 4051
16:57:06.641399 96:07:e9:78:c6:ac > 00:a0:98:73:9f:42, ethertype IPv4
(0x0800), length 4085: 172.21.35.32 > 172.21.35.1: ICMP echo reply, id
27401, seq 0, length 4051
Guest flow:
16:57:06.642073 00:a0:98:73:9f:42 > 96:07:e9:78:c6:ac, ethertype IPv4
(0x0800), length 4085: 172.21.35.1 > 172.21.35.32: ICMP echo request, id
27401, seq 0, length 4051
16:57:06.642233 96:07:e9:78:c6:ac > 00:a0:98:73:9f:42, ethertype IPv4
(0x0800), length 4084: truncated-ip - 1 bytes missing! 172.21.35.32 >
172.21.35.1: ICMP echo reply, id 27401, seq 0, length 405

Now my problem is that I can't simply keep guest's mtu at 1500, since
the host will send jumbo frames as answer wich never get through virtio-net.

Does anybody have an idea how to fix/work arround?

Thanks,

-harry
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Unhandled ps2 keyboard keysym with UEFI-edk2-bootrom and bhyve(8)

[Harry Schmalzbauer]

 Dear all,

booting guests with uefi-edk2 bootrom works great, thanks for that great
support!

Unfortunately the VNC part seems to have keyscan problems:
Unhandled ps2 keyboard keysym 0xc4
Unhandled ps2 keyboard keysym 0xf6
Unhandled ps2 keyboard keysym 0xff7f
Just to list a view...

The really unfortunate problem is that some characters are completely
missing, while others are just shifted (involving even meta keys).
I haven't found any rule yet, it looks like a mixture of en-US and de-DE
layout.
Most unfortunate, I can't login because of the missing characters :-( So
close yet so far...

I haven't really looked into EDKII but saw that BhyvePkg/ (of
sysutils/uefi-edk2-bhyve) doesn't include filesystem support.

But what I'm really wondering about is how to influence boot order e.g.

If I boot into the EFI shell (edk2-bootrom guest) and return with exit
into the uefi firmware setup (still edk2-bootrom), I can select what to
boot with the boot manager (also MBR booting via CSM works great, thanks
thanks thanks!), and I can also save the settings, but they are lost if
bhyve(8) terminates.
There's /dev/vmm.io/host-bootrom, which I don't understand yet, but
probably that's a way to feed the firmware?

Thanks for any hints,

-harry

P.S.: Please set me CC, I'm not subscribed to virtualization
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

BAR and pci passthrough

[Harry Schmalzbauer]

 In reply to
https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-December/004000.html

>/>> The problem probably that the length of the bar is smaller and not
/>/>> aligned with the pagesize. Could the length of the BAR be modified in
/>/>> order to perform the pci passthrough?
/>/>
/>/> Yes, that's correct - the size of the BAR is not a multiple of the
/>/> page size which leads to the error. If this BAR is mapped into the
/>/> guest's address space then it will "leak" an additional 3K into the
/>/> guest (since the minimum nested mapping is 4KB in size).

/> The proper fix is to modify the ppt(4) driver so that
> it asks the PCI bus to allocate a full page for BARs that are smaller than a
> page.  Unfortunately the PCI bus driver doesn't currently provide a way to do
> that.  Even if it did it would not necessarily always work since the device
> may be behind a PCI-PCI bridge whose windows can't grow due to existing
> allocations of adjacent addresses.  Alternatively the hypervisor could trap
> all accesses to this page and only permit accesses to the range that contains
> the BAR, but that would be quite slow.

My stupid idea was to set in device.hints:
hint.ppt.0.msize="0x1000"

Since John (jhb@) already posted that the PCI bus driver deosn't provide the 
necessary capability to increase allocation, this clearly doesn't work.
But since this hint was one year ago, I wanted to ask if there are news 
regarding PCI passthru (ppt's) page size handling or other workarrounds.
My specific problem reads:
sh -c 'while true ; do (bhyve -u -A -H -P -s 0,hostbridge -s 
6,virtio-net,vmnet0 -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait -s 31,lpc -s 
1,passthru,6/0/0 -l com1,stdio -l 
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd -S -m 4G -c 4 preed); 
vmm_exit=$?; if [ ${vmm_exit} -ne 0 ]; then break; fi; done ; 
/usr/sbin/bhyvectl --destroy --vm=preed'
bhyve: passthru device 6/0/0 BAR 5: base 0xc3e1 or size 0x200 not page 
aligned

Please make sure im on CC, I'm not subscribed.

Thanks,

-Harry

P.S.: Thanks to all byhve hackers, really awsome work! I'm evaluating ESXi 
reliever... Perhaps this obstacle is interesting for some people here (and one 
reason I need to passthru SATA controller)
https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086157.html

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"