on bhyve statistics

[Fabian Freyer]

Hi list,

I'm currently looking at getting the libvirt prometheus exporter[1] to
work with libvirt+bhyve. In its current state this doesn't work because
at least one of the API calls exposed by libvirt isn't implemented by
the libvirt bhyve driver - so I started looking at implementing it.

The first API call in question is virDomainBlockStats[2], which returns
statistics (number of read and written bytes and ops, respectively).

Currently, bhyve does not expose any of these statistics. All the stats
available through bhyvectl --get-stats seem to be coming from the VMM,
not from the userspace emulation. OTOH, I did see that there are *some*
stats being collected in bhyverun.c (see struct bhyvestats {...}
stats;). I can't see how these are exposed though -  a grep of /usr/src
turned up no other uses. Which brings me to the following questions:

- are the stats in struct bhyvestats {...} stats exposed or used in any
  non-obvious way?

- architecturally, what would be the best ways to get stats out of the
  user-space emulations? Off of the top of my head, I could think of the
  following possibilities:
  - prometheus exporter
  - having some socket or pipe to request them
  - DTrace probes

I wouldn't mind implementing any of the above, and so would like to know
which of these (or other options) would be the most acceptable, and
would appreciate some guidance.

CC'ing novel@ for the libvirt side, and grehan@ for the architectural
bhyve questions.

Fabian

[1] https://github.com/kumina/libvirt_exporter
[2] https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Marcelo Araujo]

Hi Fabian,

2018-08-27 22:14 GMT+08:00 Fabian Freyer 
:

> Hi list,
>
> I'm currently looking at getting the libvirt prometheus exporter[1] to
> work with libvirt+bhyve. In its current state this doesn't work because
> at least one of the API calls exposed by libvirt isn't implemented by
> the libvirt bhyve driver - so I started looking at implementing it.
>
> The first API call in question is virDomainBlockStats[2], which returns
> statistics (number of read and written bytes and ops, respectively).
>
> Currently, bhyve does not expose any of these statistics. All the stats
> available through bhyvectl --get-stats seem to be coming from the VMM,
> not from the userspace emulation. OTOH, I did see that there are *some*
> stats being collected in bhyverun.c (see struct bhyvestats {...}
> stats;). I can't see how these are exposed though -  a grep of /usr/src
> turned up no other uses. Which brings me to the following questions:
>
> - are the stats in struct bhyvestats {...} stats exposed or used in any
>   non-obvious way?
>

They are most used inside bhyverun by the name "stats" and the purpose is
not really to collect statistics about the guest vm.


>
> - architecturally, what would be the best ways to get stats out of the
>   user-space emulations? Off of the top of my head, I could think of the
>   following possibilities:
>   - prometheus exporter
>   - having some socket or pipe to request them
>   - DTrace probes
>

I don't know what kind of stats do you need to collect, but based on the
assumptions you listed above, maybe you can take a look at this project:
https://github.com/freenas/bhyve-vm-goagent

bhyve-vm-goagent can easily be extended to collect other information from
guest. Although I'm not sure if it will be useful for your case, looks like
you are looking for something less intrusive than bhyve-vm-goagent.



>
> I wouldn't mind implementing any of the above, and so would like to know
> which of these (or other options) would be the most acceptable, and
> would appreciate some guidance.
>
> CC'ing novel@ for the libvirt side, and grehan@ for the architectural
> bhyve questions.
>
> Fabian
>
> [1] https://github.com/kumina/libvirt_exporter
> [2] https://libvirt.org/html/libvirt-libvirt-domain.html#
> virDomainBlockStats
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-
> unsubscr...@freebsd.org"
>



-- 

-- 
Marcelo Araujo(__)ara...@freebsd.org
\\\'',)http://www.FreeBSD.org    \/  \ ^
Power To Server. .\. /_)
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Fabian Freyer]

On 27.08.2018 16:34, Marcelo Araujo wrote:
> Hi Fabian,
> 
> 2018-08-27 22:14 GMT+08:00 Fabian Freyer
>  >:
[...]
> - are the stats in struct bhyvestats {...} stats exposed or used in any
>   non-obvious way?
> 
> 
> They are most used inside bhyverun by the name "stats" and the purpose
> is not really to collect statistics about the guest vm.

What are they used for? I only see them get increased, never actually
checked:

$ rg stats /usr/src/usr.sbin/bhyve
/usr/src/usr.sbin/bhyve/bhyverun.c
117:struct bhyvestats {
126:} stats;
510:stats.vmexit_bogus++;
521:stats.vmexit_reqidle++;
530:stats.vmexit_hlt++;
544:stats.vmexit_pause++;
555:stats.vmexit_mtrap++;
566:stats.vmexit_inst_emul++;


> - architecturally, what would be the best ways to get stats out of the
>   user-space emulations? Off of the top of my head, I could think of the
>   following possibilities:
>   - prometheus exporter
>   - having some socket or pipe to request them
>   - DTrace probes
> 
> 
> I don't know what kind of stats do you need to collect, but based on the
> assumptions you listed above, maybe you can take a look at this project:
> https://github.com/freenas/bhyve-vm-goagent
> 
> bhyve-vm-goagent can easily be extended to collect other information
> from guest. Although I'm not sure if it will be useful for your case,
> looks like you are looking for something less intrusive than
> bhyve-vm-goagent.

Thanks for the pointer - but you're right, I'm looking for
guest-agnostic monitoring on the host.

Fabian
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Query regarding tutorials (Please have a quick read of me!)

[Paul Webster via freebsd-virtualization]

Heya all part 2 'windows server 2016' is ready:

https://www.youtube.com/watch?v=tHIWdoNXGec

On Sat, 25 Aug 2018 at 09:55, Paul Webster 
wrote:

> Hey all,
>
> A while ago I created a youtube channel to start doing video example
> versions of the FreeBSD documentation, I have decided that this weekend I
> would take a swing at showing howto setup a Bhyve VM, I personally have:
>
> Windows Server 2012
> CentOS 6
> Debian
> FreeBSD
> OpenBSD
> and a few other more strange linuxs I believe (got a zpool full of the
> zvols (not named very well ;))) anyhow IRC tells me that a lot of users
> have trouble getting windows server working. I know obviously that 2012
> works but I am willing to take a swing at 2016 if people would want that.
>
> Or would the community benefit more of an example of 2012 (which is much
> lighter)?
>
> Or perhaps something entirely different, any replies would be more than
> welcome :)
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Rodney W. Grimes]

> Hi list,
> 
> I'm currently looking at getting the libvirt prometheus exporter[1] to
> work with libvirt+bhyve. In its current state this doesn't work because
> at least one of the API calls exposed by libvirt isn't implemented by
> the libvirt bhyve driver - so I started looking at implementing it.
> 
> The first API call in question is virDomainBlockStats[2], which returns
> statistics (number of read and written bytes and ops, respectively).
> 
> Currently, bhyve does not expose any of these statistics. All the stats
> available through bhyvectl --get-stats seem to be coming from the VMM,
> not from the userspace emulation.

That is correct, byhvectl is a diagnostics tool for getting
information from the kernel/vmm module.

> OTOH, I did see that there are *some*
> stats being collected in bhyverun.c (see struct bhyvestats {...}
> stats;). I can't see how these are exposed though -  a grep of /usr/src
> turned up no other uses. Which brings me to the following questions:
> 
> - are the stats in struct bhyvestats {...} stats exposed or used in any
>   non-obvious way?

Not that I am aware of.

> - architecturally, what would be the best ways to get stats out of the
>   user-space emulations? Off of the top of my head, I could think of the
>   following possibilities:
>   - prometheus exporter
>   - having some socket or pipe to request them
>   - DTrace probes
> 
> I wouldn't mind implementing any of the above, and so would like to know
> which of these (or other options) would be the most acceptable, and
> would appreciate some guidance.

I differ to others on what may be the best way to do this.

> CC'ing novel@ for the libvirt side, and grehan@ for the architectural
> bhyve questions.

You should replace @grehan with @jhb,@tychon as Peter has moved on,
and John and Tycho are now the bhyve maintainers.  I was going to
add them, and remove Peter, but I see no cc: anyway, so I am sure
that they are on the virtualization list though.

> Fabian
> 
> [1] https://github.com/kumina/libvirt_exporter
> [2] https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Query regarding tutorials (Please have a quick read of me!)

[Tommi Pernila]

Hi Paul,

thanks for the videos, they are looking great.

some feedback:
Do keep the small mistakes like typoes or similar in the video. It's always
interesting to see how people correct the errors.

In the first video the left side of the terminal was slightly cut out. Not
too much though.

In the second video your webcam footage is partly blocking the view. Maybe
split the right side vertically with tmux?

Looking forward to more of your videos.


Br,

Tommi

On Mon, 27 Aug 2018 at 18.01, Paul Webster via freebsd-virtualization <
freebsd-virtualization@freebsd.org> wrote:

> Heya all part 2 'windows server 2016' is ready:
>
> https://www.youtube.com/watch?v=tHIWdoNXGec
>
> On Sat, 25 Aug 2018 at 09:55, Paul Webster 
> wrote:
>
> > Hey all,
> >
> > A while ago I created a youtube channel to start doing video example
> > versions of the FreeBSD documentation, I have decided that this weekend I
> > would take a swing at showing howto setup a Bhyve VM, I personally have:
> >
> > Windows Server 2012
> > CentOS 6
> > Debian
> > FreeBSD
> > OpenBSD
> > and a few other more strange linuxs I believe (got a zpool full of the
> > zvols (not named very well ;))) anyhow IRC tells me that a lot of users
> > have trouble getting windows server working. I know obviously that 2012
> > works but I am willing to take a swing at 2016 if people would want that.
> >
> > Or would the community benefit more of an example of 2012 (which is much
> > lighter)?
> >
> > Or perhaps something entirely different, any replies would be more than
> > welcome :)
> >
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Query regarding tutorials (Please have a quick read of me!)

[Paul Webster via freebsd-virtualization]

Thanks for the feedback!

Indeed I quite like the errors too when I see other people, its nice when
its not to 'rehearsed' feels more natural :)

On Mon, 27 Aug 2018 at 16:14, Tommi Pernila  wrote:

> Hi Paul,
>
> thanks for the videos, they are looking great.
>
> some feedback:
> Do keep the small mistakes like typoes or similar in the video. It's
> always interesting to see how people correct the errors.
>
> In the first video the left side of the terminal was slightly cut out. Not
> too much though.
>
> In the second video your webcam footage is partly blocking the view. Maybe
> split the right side vertically with tmux?
>
> Looking forward to more of your videos.
>
>
> Br,
>
> Tommi
>
> On Mon, 27 Aug 2018 at 18.01, Paul Webster via freebsd-virtualization <
> freebsd-virtualization@freebsd.org> wrote:
>
>> Heya all part 2 'windows server 2016' is ready:
>>
>> https://www.youtube.com/watch?v=tHIWdoNXGec
>>
>> On Sat, 25 Aug 2018 at 09:55, Paul Webster > >
>> wrote:
>>
>> > Hey all,
>> >
>> > A while ago I created a youtube channel to start doing video example
>> > versions of the FreeBSD documentation, I have decided that this weekend
>> I
>> > would take a swing at showing howto setup a Bhyve VM, I personally have:
>> >
>> > Windows Server 2012
>> > CentOS 6
>> > Debian
>> > FreeBSD
>> > OpenBSD
>> > and a few other more strange linuxs I believe (got a zpool full of the
>> > zvols (not named very well ;))) anyhow IRC tells me that a lot of users
>> > have trouble getting windows server working. I know obviously that 2012
>> > works but I am willing to take a swing at 2016 if people would want
>> that.
>> >
>> > Or would the community benefit more of an example of 2012 (which is much
>> > lighter)?
>> >
>> > Or perhaps something entirely different, any replies would be more than
>> > welcome :)
>> >
>> ___
>> freebsd-virtualization@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>> To unsubscribe, send any mail to "
>> freebsd-virtualization-unsubscr...@freebsd.org"
>>
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve guest boot error, Buffer I/O error on dev vda1, logical block 0, lost async page write

[Pete Wright]

On 8/26/18 10:07 PM, Sami Halabi wrote:

Hi,
IIRC pool shouldn't be above 90% used otherwise bad things happen.. but at
69%
Can someone explain thid please?


it's possible the filesystem was heavily fragmented?  seems suspicious - 
did scrubbing your pool find any errors?


-pete

--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Bug 229167] [Hyper-V] [Jun 19, 2018] Recently FreeBSD VM panics during boot-up, especially with Mellanox VF configured

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229167

--- Comment #32 from Dexuan Cui  ---
(In reply to Andriy Gapon from comment #31)

Unluckily I still can't reproduce the issue recently. :-(

I created the Marketplace 11.1 VM on Azure, and can't reproduce the issue:

FreeBSD decui-bsd111 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0
r313908+f9756700e2c(releng/11.1)-dirty: Thu Jan 11 23:23:54 CST 2018
root@bsd:/usr/obj/usr/xhx/freebsd/sys/GENERIC  amd64

Next, I upgraded it to the latest 11.1, and still no luck:

FreeBSD decui-bsd111 11.1-RELEASE-p13 FreeBSD 11.1-RELEASE-p13 #0: Tue Aug 14
19:34:21 UTC 2018
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Finally, I upgraded it to the latest 11.2, and still no luck:

FreeBSD decui-bsd111 11.2-RELEASE-p2 FreeBSD 11.2-RELEASE-p2 #0: Tue Aug 14
21:45:40 UTC 2018
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Please let us know if you can still reproduce the issue, especially with the
same latest 11.1 and 11.2 releases.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve guest boot error, Buffer I/O error on dev vda1, logical block 0, lost async page write

[Rajil Saraswat]

On 08/27/2018 11:50 AM, Pete Wright wrote:
>
>
> On 8/26/18 10:07 PM, Sami Halabi wrote:
>> Hi,
>> IIRC pool shouldn't be above 90% used otherwise bad things happen..
>> but at
>> 69%
>> Can someone explain thid please?
>
> it's possible the filesystem was heavily fragmented?  seems suspicious
> - did scrubbing your pool find any errors?
>
> -pete
>

This is how the pool looked like,

|# zpool list vmpool NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP
HEALTH ALTROOT vmpool 372G 257G 115G - - 64% 69% 1.00x ONLINE - |

Scrub did not show the errors. After deleting unused zvols, the guests
started to work and pool looks like this,
|
|

|# zpool list vmpool
NAME SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP 
HEALTH  ALTROOT
vmpool   372G   197G   175G    - -    56%    53%  1.00x 
ONLINE  -
|

|
|
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Anish]

>> Currently, bhyve does not expose any of these statistics. All the stats
> available through bhyvectl --get-stats seem to be coming from the VMM,
> not from the userspace emulation.

>That is correct, byhvectl is a diagnostics tool for getting
information from the kernel/vmm module.

bhyvectl provide stats related to processor vmx/svm from vmm.ko and is the
first thing you want to run for performance regression. It will be nice to
include it as part of bhyve perf tool/dashboard that you are intended to
build.

-Anish

On Mon, Aug 27, 2018 at 8:20 AM Rodney W. Grimes <
freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:

> > Hi list,
> >
> > I'm currently looking at getting the libvirt prometheus exporter[1] to
> > work with libvirt+bhyve. In its current state this doesn't work because
> > at least one of the API calls exposed by libvirt isn't implemented by
> > the libvirt bhyve driver - so I started looking at implementing it.
> >
> > The first API call in question is virDomainBlockStats[2], which returns
> > statistics (number of read and written bytes and ops, respectively).
> >
> > Currently, bhyve does not expose any of these statistics. All the stats
> > available through bhyvectl --get-stats seem to be coming from the VMM,
> > not from the userspace emulation.
>
> That is correct, byhvectl is a diagnostics tool for getting
> information from the kernel/vmm module.
>
> > OTOH, I did see that there are *some*
> > stats being collected in bhyverun.c (see struct bhyvestats {...}
> > stats;). I can't see how these are exposed though -  a grep of /usr/src
> > turned up no other uses. Which brings me to the following questions:
> >
> > - are the stats in struct bhyvestats {...} stats exposed or used in any
> >   non-obvious way?
>
> Not that I am aware of.
>
> > - architecturally, what would be the best ways to get stats out of the
> >   user-space emulations? Off of the top of my head, I could think of the
> >   following possibilities:
> >   - prometheus exporter
> >   - having some socket or pipe to request them
> >   - DTrace probes
> >
> > I wouldn't mind implementing any of the above, and so would like to know
> > which of these (or other options) would be the most acceptable, and
> > would appreciate some guidance.
>
> I differ to others on what may be the best way to do this.
>
> > CC'ing novel@ for the libvirt side, and grehan@ for the architectural
> > bhyve questions.
>
> You should replace @grehan with @jhb,@tychon as Peter has moved on,
> and John and Tycho are now the bhyve maintainers.  I was going to
> add them, and remove Peter, but I see no cc: anyway, so I am sure
> that they are on the virtualization list though.
>
> > Fabian
> >
> > [1] https://github.com/kumina/libvirt_exporter
> > [2]
> https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats
> > ___
> > freebsd-virtualization@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
> --
> Rod Grimes
> rgri...@freebsd.org
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve guest boot error, Buffer I/O error on dev vda1, logical block 0, lost async page write

[John-Mark Gurney]

Rajil Saraswat wrote this message on Mon, Aug 27, 2018 at 17:42 -0500:
> On 08/27/2018 11:50 AM, Pete Wright wrote:
> >
> >
> > On 8/26/18 10:07 PM, Sami Halabi wrote:
> >> Hi,
> >> IIRC pool shouldn't be above 90% used otherwise bad things happen..
> >> but at
> >> 69%
> >> Can someone explain thid please?
> >
> > it's possible the filesystem was heavily fragmented?  seems suspicious
> > - did scrubbing your pool find any errors?
> >
> > -pete
> >
> 
> This is how the pool looked like,
> 
> |# zpool list vmpool NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP
> HEALTH ALTROOT vmpool 372G 257G 115G - - 64% 69% 1.00x ONLINE - |
> 
> Scrub did not show the errors. After deleting unused zvols, the guests
> started to work and pool looks like this,
> |
> |
> 
> |# zpool list vmpool
> NAME SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP 
> HEALTH  ALTROOT
> vmpool   372G   197G   175G    - -    56%    53%  1.00x 
> ONLINE  -

There are a couple things that can happen.. you could have a quota
or refquota that prevents a dataset/zdev, or it's descendants from growing
larger, or you could have refreservation or reservation set which limits
the space that other datasets can consume..

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"