Re: [RFC] Adding a Rados block driver to bhyve

[Willem Jan Withagen]

On 10-3-2020 16:15, Alan Somers wrote:
On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen > wrote:


On 9-3-2020 14:46, Alan Somers wrote:

On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen
mailto:w...@digiware.nl>> wrote:

Hi all,

And sorry for crosspoing three groups, but the answer
can/could be a mix
of things to do in these three areas.

I have a prototype of bhyve running on Rados/Ceph working:
https://github.com/freebsd/freebsd/pull/426


..


4) Create a bhyve-blockrbd port.
 This is much like 3) but instead of building a bhyve-rbd
executable,
 it delivers a libblockrbd.so that is dynamically
loadable by the
 standaard bhyve that comes with base.




> Great work!  I also agree that option 4 sounds like the best. 
There's precedent for ports that
> require the FreeBSD Sources.  For example, see devel/py-libzfs
or emulators/virtualbox-ose.
> You just need to define the SRC_BASE variable.
Hi Alan,

Thanx for the hint, and it made me check what is actually
available within the poudriere jail
And that does have full source, so the Makefile code is mainly for
those that build in a different way.

I've got a proto version working when compiling stuff with `make
buildworld`, but run in the
problem that libblock_rbd.so is stripped in such a way that the
symbol I need is removed.
Using the unstripped version does work.

Is there an incantation for the SRC Makefiles that builds a
dynamical loadable lib??
And I'm still looking for a PORTS example of building a dynamical
loadable lib.
Or is there no generic code for that in the PORTS Mk files?

--WjW

BTW: Still haven't worked in your AIO code :(


There are plenty of dynamic libraries built with the SRC makefiles.  
For example, 
https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup 
.


That looks dangerously close to what I have for libblock_rbd.
===
> cat Makefile-librbd
#
# $FreeBSD$
#

PACKAGE=lib${LIB}

.include 

LIB=    block_rbd
SHLIB_MAJOR=    1

SRCS=   block_rbd.c

CFLAGS+=-I${SRCTOP}/sys
CFLAGS+=-g -O0 -fPIC -rdynamic
LDFLAGS+=-Wl,-export-dynamic,-Bdynamic
CFLAGS+=-DWITHOUT_CAPSICUM

LOCALBASE?= /usr/local
CFLAGS+=    -I${LOCALBASE}/include
LDFLAGS+=   -L${LOCALBASE}/lib -lrados -lrbd

WARNS?= 2

===

This is the code that mk.lib.bsd runs:
objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug
objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug 
libblock_rbd.so.1.full libblock_rbd.so.1


So still I get a stripped lib in /usr/lib. And then the one and only 
symbol I need to load

is not found. Copying libblock_rbd.so.1.full actually works for me.

So either I'm doing it the wrong way, like special options on the 
symbols oid.

Or mk.lib.bsd cannot deliver dlopen/dlsym-able files?

And there are plenty of ports that build shared libraries too, just look 
at /usr/local/lib/*.so.  However, the ports framework doesn't have much 
special code just to support building libraries.  Instead the hard work 
is always done by the ports themselves.  Some use autotools, some cmake, 
etc etc.  The simplest port I can find that uses both SRC_BASE and 
INSTALL_LIB is this one: 
https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ .


Oke thanx, I'll have a look at it, and given that I can see most of the 
compile build stuff

in the SRC_BASE version I'll get it to work.

--WjW

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: [RFC] Adding a Rados block driver to bhyve

[Willem Jan Withagen]

On 9-3-2020 14:46, Alan Somers wrote:
On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen > wrote:


Hi all,

And sorry for crosspoing three groups, but the answer can/could be
a mix
of things to do in these three areas.

I have a prototype of bhyve running on Rados/Ceph working:
https://github.com/freebsd/freebsd/pull/426


..


4) Create a bhyve-blockrbd port.
 This is much like 3) but instead of building a bhyve-rbd
executable,
 it delivers a libblockrbd.so that is dynamically loadable by the
 standaard bhyve that comes with base.




> Great work!  I also agree that option 4 sounds like the best. There's 
precedent for ports that
> require the FreeBSD Sources.  For example, see devel/py-libzfs or 
emulators/virtualbox-ose.

> You just need to define the SRC_BASE variable.
Hi Alan,

Thanx for the hint, and it made me check what is actually available 
within the poudriere jail
And that does have full source, so the Makefile code is mainly for those 
that build in a different way.


I've got a proto version working when compiling stuff with `make 
buildworld`, but run in the
problem that libblock_rbd.so is stripped in such a way that the symbol I 
need is removed.

Using the unstripped version does work.

Is there an incantation for the SRC Makefiles that builds a dynamical 
loadable lib??
And I'm still looking for a PORTS example of building a dynamical 
loadable lib.

Or is there no generic code for that in the PORTS Mk files?

--WjW

BTW: Still haven't worked in your AIO code :(


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: [RFC] Adding a Rados block driver to bhyve

[Dave Horsfall]

On Mon, 9 Mar 2020, Miroslav Lachman wrote:

There are some ports (for example sysutils/lsof) which need kernel 
sources to build.  [...]


Kernel sources, or just the headers?

-- Dave
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: [RFC] Adding a Rados block driver to bhyve

[Willem Jan Withagen]

On 9-3-2020 12:38, Miroslav Lachman wrote:

Willem Jan Withagen wrote on 2020/03/09 11:31:

[...]


3) Create a bhyve-rbd port.
 Problem with that is that it will require the FreeBSD source tree 
for the

 bhyve sources, but there is no Ports option for that?
     Or bhyve sources are manually copied into the port. And then
 try to keep these sources up to date.
 Then compile and install a bhyve-rbd into /usr/local/sbin


There are some ports (for example sysutils/lsof) which need kernel 
sources to build. So this can be a way too. I cannot say if this is the 
best way or not.


Yes, there seems a flag for kernel sources...
But not for world-sources.??

--WjW
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: [RFC] Adding a Rados block driver to bhyve

[Miroslav Lachman]

Willem Jan Withagen wrote on 2020/03/09 11:31:

[...]


3) Create a bhyve-rbd port.
     Problem with that is that it will require the FreeBSD source tree 
for the

     bhyve sources, but there is no Ports option for that?
     Or bhyve sources are manually copied into the port. And then
     try to keep these sources up to date.
     Then compile and install a bhyve-rbd into /usr/local/sbin


There are some ports (for example sysutils/lsof) which need kernel 
sources to build. So this can be a way too. I cannot say if this is the 
best way or not.


[...]

Kind regards
Miroslav Lachman
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

[RFC] Adding a Rados block driver to bhyve

[Willem Jan Withagen]

Hi all,

And sorry for crosspoing three groups, but the answer can/could be a mix
of things to do in these three areas.

I have a prototype of bhyve running on Rados/Ceph working:
    https://github.com/freebsd/freebsd/pull/426

But there are a few catches on how to get it in the FreeBSd sources...

1) Easiest would be to just compile it in with the code of the current 
bhyve.

    That will require librados/librbd libraries...
    Ceph of this purpose is LGPL2/3 and could go into contrib.
    In this case bhyve will hold the rbd-driver by default and a user 
does not

    need to do anything by himself
    But I have the feeling that this is the most unwanted scenario

2) User first installs a Ceph package and FreeBSD sources, and then 
recompiles

    bhyve with the option BHYVE_RBD.
    And then reinstalls this new version as bhyve or bhyve-rbd in /usr/sbin

3) Create a bhyve-rbd port.
    Problem with that is that it will require the FreeBSD source tree 
for the

    bhyve sources, but there is no Ports option for that?
    Or bhyve sources are manually copied into the port. And then
    try to keep these sources up to date.
    Then compile and install a bhyve-rbd into /usr/local/sbin

4) Create a bhyve-blockrbd port.
    This is much like 3) but instead of building a bhyve-rbd executable,
    it delivers a libblockrbd.so that is dynamically loadable by the
    standaard bhyve that comes with base.

    For this bhyve needs to be extended with dynamic loadable driver 
modules.
    This is reasonably doable, but is this acceptable for the bhyve 
maintainers?


    For building the port, the bhyve-blockrbd code will only need a 
limited set
    of files from /usr/src/usr.bin/bhyve thus limiting the chance of 
running out

    sequence with the bhyve from base.

Looking over these 4 options, I think that 4 is the most desirable one?
But 2 would parhaps be workable for users as well, but the project might 
think

otherwise.

Are there other options?
And/or is 4 the best way to go, with 2 as a nice intermediate?

Thanx,
--WjW
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"