Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
于 2012-7-24 2:15, Blue Swirl 写道:
On Wed, Jul 18, 2012 at 8:51 AM, Wenchao Xia wrote:
Hi, following is API draft, prototypes were taken from qemu/block.h,
and the API prefix is changed frpm bdrv to qbdrvs, to declare related
object is BlockDriverState, not BlockDriver. One issue here is it may
require include block_int.h, which is not LGPL2 licensed yet.
API format is kept mostly the same with qemu generic block layer, to
make it easier for implement, and easier to make qemu migrate on it if
possible.
/* structure init and uninit */
BlockDriverState *qbdrvs_new(const char *device_name);
void qbdrvs_delete(BlockDriverState *bs);
/* file open and close */
int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags,
BlockDriver *drv);
How are the errors passed?
Returning negative value should indicate errors. I plan to insert
a qblock-err.h to define all number there, and then a function of
const char *qberr2str(int err) to translate error.
Alternative version with file descriptor or struct FILE instead of
filename might become useful but can be added later.
Agree.
void qbdrvs_close(BlockDriverState *bs);
int qbdrvs_img_create(const char *filename, const char *fmt,
const char *base_filename, const char *base_fmt,
char *options, uint64_t img_size, int flags);
'const char *options'
/* sync access */
int qbdrvs_read(BlockDriverState *bs, int64_t sector_num,
uint8_t *buf, int nb_sectors);
int qbdrvs_write(BlockDriverState *bs, int64_t sector_num,
const uint8_t *buf, int nb_sectors);
Do we want to use sectors here? How about just raw byte offsets and
number of bytes? I'd leave any hardware details out and just provide
file semantics (open/read/write/close). Future QEMU refactorings could
make supporting HW info inconvenient. If HW details (geometry etc., VM
state) are needed, there should be a separate API.
not sure why original API used sector instead of raw address,
if no special requirement for sector I think raw address is better.
Vectored I/O might be useful too.
/* info retrieve */
//sector, size and geometry info
int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi);
Currently BlockDriverInfo does not look very useful for outside users,
with the exception of dirty state. How about accessors instead:
bool qbdrvs_is_dirty(BlockDriverState *bs);
Also the format (QCOW) is needed so that the user can use backing file
functions:
const char *qbdrvs_get_format(BlockDriverState *bs);
I hope to package the info into a structure, such as
#define QBLOCK_INFO_SIZE (128)
struct QblockInfo {
int dirty;
char *format;
int reserved[RESERVE_SPACE];
};
assert sizeof(QblockInfo) == QBLOCK_INFO_SIZE in every version. This
will make caller's code more clean.
int64_t qbdrvs_getlength(BlockDriverState *bs);
int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs);
void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr);
Also this function shouldn't be needed if we don't give HW details
with this API.
OK.
//image type
const char *qbdrvs_get_format_name(BlockDriverState *bs);
//backing file info
void qbdrvs_get_backing_filename(BlockDriverState *bs,
char *filename, int filename_size);
void qbdrvs_get_full_backing_filename(BlockDriverState *bs,
char *dest, size_t sz);
These are specific to QCOW etc., so
bool qbdrvs_has_backing_files(BlockDriverState *bs)?
How about using return value to indicate it?
int qbdrvs_get_backing_filename(BlockDriverState *bs, char *filename,
int filename_size);
/* advanced image content access */
int qbdrvs_is_allocated(BlockDriverState *bs, int64_t sector_num, int
nb_sectors,
int *pnum);
int qbdrvs_discard(BlockDriverState *bs, int64_t sector_num, int
nb_sectors);
Again, some files do not have a concept of allocation.
Maybe return value could indicate it.
int qbdrvs_has_zero_init(BlockDriverState *bs);
Il 16/07/2012 10:16, Wenchao Xia ha scritto:
Really thanks for the investigation, I paid quite sometime to dig out
which license is compatible to LGPL, this have sorted it out.
The coroutine and structure inside is quite a challenge.
Coroutines are really just a small complication in the program flow if
all you support is synchronous access to files (i.e. no HTTP etc.).
Their usage should be completely transparent.
What about
provide the library first in nbd + sync access, and waiting for the
library employer response? If it is good to use, then replace implement
code to native qemu block layer code, change code's license, while keep
API unchanged.
You can start by proposing the API.
Paolo
--
Best Regards
Wenchao Xia
--
Best Regards
Wenchao Xia
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Paolo Bonzini writes: > Il 23/07/2012 13:55, Lluís Vilanova ha scritto: >> I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later". >>> > What about LGPLv2+? (Note the "L".) >> I'd prefer to keep it non-lesser. Is it absolutely necessary? > This is about making the block layer part of a library. The block layer > is 90% under a BSD license, but the LGPLv2 is a good compromise. I understand. I'm not much of an expert on licensing, but if it's necessary to make the tracing control API LGPLv2+ or LGPLv3+ in order to provide a non-viric API to the users of the block layer library, I have no problem with it. Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Wed, Jul 18, 2012 at 8:51 AM, Wenchao Xia wrote: > Hi, following is API draft, prototypes were taken from qemu/block.h, > and the API prefix is changed frpm bdrv to qbdrvs, to declare related > object is BlockDriverState, not BlockDriver. One issue here is it may > require include block_int.h, which is not LGPL2 licensed yet. > API format is kept mostly the same with qemu generic block layer, to > make it easier for implement, and easier to make qemu migrate on it if > possible. > > > /* structure init and uninit */ > BlockDriverState *qbdrvs_new(const char *device_name); > void qbdrvs_delete(BlockDriverState *bs); > > > /* file open and close */ > int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags, > BlockDriver *drv); How are the errors passed? Alternative version with file descriptor or struct FILE instead of filename might become useful but can be added later. > void qbdrvs_close(BlockDriverState *bs); > int qbdrvs_img_create(const char *filename, const char *fmt, > const char *base_filename, const char *base_fmt, > char *options, uint64_t img_size, int flags); 'const char *options' > > > /* sync access */ > int qbdrvs_read(BlockDriverState *bs, int64_t sector_num, > uint8_t *buf, int nb_sectors); > int qbdrvs_write(BlockDriverState *bs, int64_t sector_num, >const uint8_t *buf, int nb_sectors); Do we want to use sectors here? How about just raw byte offsets and number of bytes? I'd leave any hardware details out and just provide file semantics (open/read/write/close). Future QEMU refactorings could make supporting HW info inconvenient. If HW details (geometry etc., VM state) are needed, there should be a separate API. Vectored I/O might be useful too. > > > /* info retrieve */ > //sector, size and geometry info > int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi); Currently BlockDriverInfo does not look very useful for outside users, with the exception of dirty state. How about accessors instead: bool qbdrvs_is_dirty(BlockDriverState *bs); Also the format (QCOW) is needed so that the user can use backing file functions: const char *qbdrvs_get_format(BlockDriverState *bs); > int64_t qbdrvs_getlength(BlockDriverState *bs); > int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs); > void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr); Also this function shouldn't be needed if we don't give HW details with this API. > //image type > const char *qbdrvs_get_format_name(BlockDriverState *bs); > //backing file info > void qbdrvs_get_backing_filename(BlockDriverState *bs, >char *filename, int filename_size); > void qbdrvs_get_full_backing_filename(BlockDriverState *bs, > char *dest, size_t sz); These are specific to QCOW etc., so bool qbdrvs_has_backing_files(BlockDriverState *bs)? > > > /* advanced image content access */ > int qbdrvs_is_allocated(BlockDriverState *bs, int64_t sector_num, int > nb_sectors, > int *pnum); > int qbdrvs_discard(BlockDriverState *bs, int64_t sector_num, int > nb_sectors); Again, some files do not have a concept of allocation. > int qbdrvs_has_zero_init(BlockDriverState *bs); > > >> Il 16/07/2012 10:16, Wenchao Xia ha scritto: >>>Really thanks for the investigation, I paid quite sometime to dig out >>> which license is compatible to LGPL, this have sorted it out. >>>The coroutine and structure inside is quite a challenge. >> >> >> Coroutines are really just a small complication in the program flow if >> all you support is synchronous access to files (i.e. no HTTP etc.). >> Their usage should be completely transparent. >> >>> What about >>> provide the library first in nbd + sync access, and waiting for the >>> library employer response? If it is good to use, then replace implement >>> code to native qemu block layer code, change code's license, while keep >>> API unchanged. >> >> >> You can start by proposing the API. >> >> Paolo >> > > > -- > Best Regards > > Wenchao Xia > > >
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 23/07/2012 13:55, Lluís Vilanova ha scritto: >>> >> I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later". >> > What about LGPLv2+? (Note the "L".) > I'd prefer to keep it non-lesser. Is it absolutely necessary? This is about making the block layer part of a library. The block layer is 90% under a BSD license, but the LGPLv2 is a good compromise. Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Stefan Hajnoczi writes: [...] >> > So I tried trimming down the list of files needed to compile >> > qemu tools, and here is a list: >> >> > Easy to relicense to LGPLv2+: >> > block/raw.c none (GPLv2+: Red Hat, IBM) >> > error.c LGPLv2 (Red Hat, IBM, Stefan Weil) >> > iov.c GPLv2 (Red Hat, SuSE/Hannes Reinecke, >> > Michael Tokarev) >> > module.cGPLv2 (Red Hat, IBM, Blue Swirl) >> > qemu-error.cGPLv2+ (Red Hat, Blue Swirl, IBM) >> > trace/control.c GPLv2 (Lluis Vilanova) >> > trace/default.c GPLv2 (Lluis Vilanova) >> >> > (I added some people to Cc. Lluis and Michael, can you also look at >> > http://wiki.qemu.org/Relicensing if you're willing to relicense >> > your past contributions from GPLv2 to GPLv2+?. Blue Swirl said >> > he'd accept any other GPLv2 or GPLv3 compatible license, which >> > should include LGPLv2+). >> >> I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later". > What about LGPLv2+? (Note the "L".) I'd prefer to keep it non-lesser. Is it absolutely necessary? PS: I suppose the "+" stands for "or later" Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 20/07/2012 13:38, Daniel P. Berrange ha scritto: > There's two aspects to it. First we forbid use of malloc/free/realloc > in favour of an alternative set of APIs designed such that we can get > compile time errors when people don't check the result for NULL. > > > http://berrange.com/posts/2008/05/23/better-living-through-api-design-low-level-memory-allocation/ > > This has been very successful in ensuring code checks for OOM correctly. > Second there is a mode in our test suites where you can run under > simulated OOM - it runs once to count the total set of mallocs in the > test, and then re-run it 'n' times failing each malloc in turn and > verifying that we correctly report the OOM error condition. Does it fail each malloc in turn, or does it fail each malloc "above X bytes" after the Y-th? Because the latter is what you actually get in practice, and it seriously hampers your recovery capabilities. Right now, we always use g_malloc and abort on failure, but we could replace it with g_try_malloc in specific large allocation cases, where large could even be not that large, for example around 64K. Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Thu, Jul 19, 2012 at 01:37:11PM +0200, Paolo Bonzini wrote: > Il 18/07/2012 17:35, Daniel P. Berrange ha scritto: > > Oh, and will this library depend on glib > > Yes, in all likelihood. > > , and will it have the > > abort-on-oom behaviour QEMU has ? From a libvirt POV, we won't > > use any library that aborts-on-oom. > > Out of curiosity, how do you test OOM? There's two aspects to it. First we forbid use of malloc/free/realloc in favour of an alternative set of APIs designed such that we can get compile time errors when people don't check the result for NULL. http://berrange.com/posts/2008/05/23/better-living-through-api-design-low-level-memory-allocation/ This has been very successful in ensuring code checks for OOM correctly. Second there is a mode in our test suites where you can run under simulated OOM - it runs once to count the total set of mallocs in the test, and then re-run it 'n' times failing each malloc in turn and verifying that we correctly report the OOM error condition. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 18/07/2012 17:35, Daniel P. Berrange ha scritto: > Oh, and will this library depend on glib Yes, in all likelihood. , and will it have the > abort-on-oom behaviour QEMU has ? From a libvirt POV, we won't > use any library that aborts-on-oom. Out of curiosity, how do you test OOM? Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Wed, Jul 18, 2012 at 02:58:46PM +0100, Daniel P. Berrange wrote: > On Wed, Jul 18, 2012 at 04:51:03PM +0800, Wenchao Xia wrote: > > Hi, following is API draft, prototypes were taken from qemu/block.h, > > and the API prefix is changed frpm bdrv to qbdrvs, to declare related > > object is BlockDriverState, not BlockDriver. One issue here is it may > > require include block_int.h, which is not LGPL2 licensed yet. > > API format is kept mostly the same with qemu generic block layer, to > > make it easier for implement, and easier to make qemu migrate on it if > > possible. > > > How is error reporting dealt with, and what is the intent around > thread safety of the APIs ? I'd like to see a fully thread safe > API - multiple threads can use the same 'BlockDriverState *' > concurrently, and thread-local error reporting. Oh, and will this library depend on glib, and will it have the abort-on-oom behaviour QEMU has ? From a libvirt POV, we won't use any library that aborts-on-oom. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Am 18.07.2012 11:03, schrieb Paolo Bonzini: > Let's get things right, and only have open/close: > > int qbdrvs_open(BlockDriverState **bs, const char *filename, int flags, > const char *format_name); > void qbdrvs_close(BlockDriverState *bs); What is flags? Are we really happy with a function that can't provide the features that -blockdev will give us? >> int qbdrvs_img_create(const char *filename, const char *fmt, >> const char *base_filename, const char *base_fmt, >> char *options, uint64_t img_size, int flags); This prototype is totally wrong. It's already not nice that the signature of some block layer internal function looks like this. Basically all of the options need to be replaced by something like a single QDict. >> /* sync access */ >> int qbdrvs_read(BlockDriverState *bs, int64_t sector_num, >> uint8_t *buf, int nb_sectors); >> int qbdrvs_write(BlockDriverState *bs, int64_t sector_num, >>const uint8_t *buf, int nb_sectors); > > I would like to have also a scatter gather API (qbdrvs_readv and > qbdrvs_writev) taking a "struct iovec *iov, int niov" instead of > "uint8_t *buf, int nb_sectors". > > flush is missing. Yes, both very important. Kevin
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Am 18.07.2012 16:12, schrieb Daniel P. Berrange: > On Wed, Jul 18, 2012 at 04:02:15PM +0200, Paolo Bonzini wrote: >> Il 18/07/2012 15:58, Daniel P. Berrange ha scritto: >>> How is error reporting dealt with >> >> These APIs just return errno values. > > Which has led to somewhat unhelpful error reporting in the past. If we're > designing a library API it'd be nice to improve on this. But in most cases, errno is what we get from the OS, so we can't do much more than passing it on. Maybe we can do a bit better with bdrv_open(), which is relatively likely to fail in qemu rather than in the kernel because something's wrong with the content of the image. >>> , and what is the intent around >>> thread safety of the APIs ? I'd like to see a fully thread safe >>> API - multiple threads can use the same 'BlockDriverState *' >>> concurrently, and thread-local error reporting. >> >> This is a bit difficult to provide, since the QEMU block layer itself is >> not thread-safe. > > Yep, I'd expect that this is something we'd need to fix when turning the > code into a library. NB, I don't mean to say QEMU should protect against > an app doing stupid things like letting 2 threads write to the same area > of the file at once. That's upto the application. I simply mean that the > BlockDriverState shouldn't corrupt itself if 2 separate APIs are called > concurrently on the same instance. I think it makes sense to make the library thread-safe - and if it only means taking the global mutex like we do in qemu. I think threading is a good interface to allow clients to do AIO (even though possibly not the only one we want to provide), and eventually it will match what qemu is doing internally. Kevin
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Wed, Jul 18, 2012 at 04:02:15PM +0200, Paolo Bonzini wrote: > Il 18/07/2012 15:58, Daniel P. Berrange ha scritto: > > How is error reporting dealt with > > These APIs just return errno values. Which has led to somewhat unhelpful error reporting in the past. If we're designing a library API it'd be nice to improve on this. > > > , and what is the intent around > > thread safety of the APIs ? I'd like to see a fully thread safe > > API - multiple threads can use the same 'BlockDriverState *' > > concurrently, and thread-local error reporting. > > This is a bit difficult to provide, since the QEMU block layer itself is > not thread-safe. Yep, I'd expect that this is something we'd need to fix when turning the code into a library. NB, I don't mean to say QEMU should protect against an app doing stupid things like letting 2 threads write to the same area of the file at once. That's upto the application. I simply mean that the BlockDriverState shouldn't corrupt itself if 2 separate APIs are called concurrently on the same instance. > Another missing feature is passwords. Oh yes. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 18/07/2012 15:58, Daniel P. Berrange ha scritto: > How is error reporting dealt with These APIs just return errno values. > , and what is the intent around > thread safety of the APIs ? I'd like to see a fully thread safe > API - multiple threads can use the same 'BlockDriverState *' > concurrently, and thread-local error reporting. This is a bit difficult to provide, since the QEMU block layer itself is not thread-safe. Another missing feature is passwords. Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Wed, Jul 18, 2012 at 04:51:03PM +0800, Wenchao Xia wrote: > Hi, following is API draft, prototypes were taken from qemu/block.h, > and the API prefix is changed frpm bdrv to qbdrvs, to declare related > object is BlockDriverState, not BlockDriver. One issue here is it may > require include block_int.h, which is not LGPL2 licensed yet. > API format is kept mostly the same with qemu generic block layer, to > make it easier for implement, and easier to make qemu migrate on it if > possible. How is error reporting dealt with, and what is the intent around thread safety of the APIs ? I'd like to see a fully thread safe API - multiple threads can use the same 'BlockDriverState *' concurrently, and thread-local error reporting. > > > /* structure init and uninit */ > BlockDriverState *qbdrvs_new(const char *device_name); > void qbdrvs_delete(BlockDriverState *bs); > > > /* file open and close */ > int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags, > BlockDriver *drv); > void qbdrvs_close(BlockDriverState *bs); > int qbdrvs_img_create(const char *filename, const char *fmt, > const char *base_filename, const char *base_fmt, > char *options, uint64_t img_size, int flags); s/img_create/create/ Can this return an actual BlockDriverState struct too. > > > /* sync access */ > int qbdrvs_read(BlockDriverState *bs, int64_t sector_num, > uint8_t *buf, int nb_sectors); > int qbdrvs_write(BlockDriverState *bs, int64_t sector_num, >const uint8_t *buf, int nb_sectors); > > > /* info retrieve */ > //sector, size and geometry info > int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi); What is in BlockDriverInfo and what is the intended ABI stability policy for it ? > int64_t qbdrvs_getlength(BlockDriverState *bs); > int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs); > void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr); Could this data all just be part of BlockDriverInfo data ? > //image type > const char *qbdrvs_get_format_name(BlockDriverState *bs); > //backing file info > void qbdrvs_get_backing_filename(BlockDriverState *bs, >char *filename, int filename_size); You need to include the backing file format here too. > void qbdrvs_get_full_backing_filename(BlockDriverState *bs, > char *dest, size_t sz); Not sure I see why we need this in addition to the above ? Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Am 18.07.2012 15:51, schrieb Andreas Färber: > Am 18.07.2012 10:51, schrieb Wenchao Xia: >> Hi, following is API draft, prototypes were taken from qemu/block.h, >> and the API prefix is changed frpm bdrv to qbdrvs, to declare related >> object is BlockDriverState, not BlockDriver. [...] After the refactoring that Markus is working on it won't refer to a BlockDriverState, but to a BlockBackend. (And changes like this make quite clear why the internals of the current block layer are not suitable as a public API. The API needs to be defined in a separate layer than can abstract such changes away.) > So let the bikeshedding begin: ;) > > What about qbds_ prefix rather than qbdrvs_? I find the proposed mixture > of acronym (q for QEMU, b for Block, s for State) and abbreviation (drv > as in Driver) a bit ugly. > > Or just simply go for qblock - might be better memorable. :) Yes, something like qblk that isn't tied to internals sounds better. Kevin
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Am 18.07.2012 10:51, schrieb Wenchao Xia: > Hi, following is API draft, prototypes were taken from qemu/block.h, > and the API prefix is changed frpm bdrv to qbdrvs, to declare related > object is BlockDriverState, not BlockDriver. [...] So let the bikeshedding begin: ;) What about qbds_ prefix rather than qbdrvs_? I find the proposed mixture of acronym (q for QEMU, b for Block, s for State) and abbreviation (drv as in Driver) a bit ugly. Or just simply go for qblock - might be better memorable. :) Just my 2¢ (on something I don't use much), Andreas -- SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Wed, Jul 18, 2012 at 06:42:24AM -0400, Paolo Bonzini wrote: > > Synchronous APIs are great for writing dedicated tools like dd, cp, > > convert, etc. > > > > Asynchronous APIs are essential for integrating image file I/O into > > event-driven programs like libvirt. Here, the ability to do other > > things while image file I/O is in progress is a requirement. It may > > also be necessary to cancel or timeout if an operation is not making > > progress or the user decides to stop it. > > > > I think we need to provide both sync and async. Libraries like > > libssh2 and libcurl already do this so their APIs can be used as a > > starting point for async I/O. > > If we want to provide an asynchronous API, the easiest thing would > be to provide a GSource and that's it. That would even make sense for > QEMU itself, in fact. > > What I'm worried about, is how to support _both_ synchronous and > asynchronous access. I'd like the library to be clean of things like > qemu_aio_wait() and qemu_aio_flush(), at least in the beginning. > That's why I think async can come later, once we actually get > applications needing it. Right now, libvirt's requirements are > simple (e.g. probing the backing file chain) and would be synchronous > anyway. Yes, qemu_aio_wait() and qemu_aio_flush() are ugly. Starting with sync makes sense, it's a convenient API to have even if we add async later. Stefan
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
> Whether to provide sync and/or async access is a key question. Indeed. > Synchronous APIs are great for writing dedicated tools like dd, cp, > convert, etc. > > Asynchronous APIs are essential for integrating image file I/O into > event-driven programs like libvirt. Here, the ability to do other > things while image file I/O is in progress is a requirement. It may > also be necessary to cancel or timeout if an operation is not making > progress or the user decides to stop it. > > I think we need to provide both sync and async. Libraries like > libssh2 and libcurl already do this so their APIs can be used as a > starting point for async I/O. If we want to provide an asynchronous API, the easiest thing would be to provide a GSource and that's it. That would even make sense for QEMU itself, in fact. What I'm worried about, is how to support _both_ synchronous and asynchronous access. I'd like the library to be clean of things like qemu_aio_wait() and qemu_aio_flush(), at least in the beginning. That's why I think async can come later, once we actually get applications needing it. Right now, libvirt's requirements are simple (e.g. probing the backing file chain) and would be synchronous anyway. Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Wed, Jul 18, 2012 at 9:51 AM, Wenchao Xia wrote: > /* sync access */ > int qbdrvs_read(BlockDriverState *bs, int64_t sector_num, > uint8_t *buf, int nb_sectors); > int qbdrvs_write(BlockDriverState *bs, int64_t sector_num, >const uint8_t *buf, int nb_sectors); Whether to provide sync and/or async access is a key question. Synchronous APIs are great for writing dedicated tools like dd, cp, convert, etc. Asynchronous APIs are essential for integrating image file I/O into event-driven programs like libvirt. Here, the ability to do other things while image file I/O is in progress is a requirement. It may also be necessary to cancel or timeout if an operation is not making progress or the user decides to stop it. I think we need to provide both sync and async. Libraries like libssh2 and libcurl already do this so their APIs can be used as a starting point for async I/O. Stefan
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 18/07/2012 10:51, Wenchao Xia ha scritto: > Hi, following is API draft, prototypes were taken from qemu/block.h, > and the API prefix is changed frpm bdrv to qbdrvs, to declare related > object is BlockDriverState, not BlockDriver. One issue here is it may > require include block_int.h, which is not LGPL2 licensed yet. block_int.h is BSD-licensed. It's compatible. > API format is kept mostly the same with qemu generic block layer, to > make it easier for implement, and easier to make qemu migrate on it if > possible. I'll note that img_create, and almost all retrieval functions cannot be implemented on top of NBD. > /* structure init and uninit */ > BlockDriverState *qbdrvs_new(const char *device_name); device_name is not needed in the library. > void qbdrvs_delete(BlockDriverState *bs); Being able to reuse the same BDS with close/open is one of the worst parts of the QEMU block layer. It is only needed to implement eject in QEMU. Let's get things right, and only have open/close: int qbdrvs_open(BlockDriverState **bs, const char *filename, int flags, const char *format_name); void qbdrvs_close(BlockDriverState *bs); > > > /* file open and close */ > int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags, > BlockDriver *drv); You didn't have an API to find the BlockDriver *. Let's just pass a format name, or NULL for probing (see above). This is consistent with qbdrvs_img_create. > void qbdrvs_close(BlockDriverState *bs); > int qbdrvs_img_create(const char *filename, const char *fmt, > const char *base_filename, const char *base_fmt, > char *options, uint64_t img_size, int flags); > > > /* sync access */ > int qbdrvs_read(BlockDriverState *bs, int64_t sector_num, > uint8_t *buf, int nb_sectors); > int qbdrvs_write(BlockDriverState *bs, int64_t sector_num, >const uint8_t *buf, int nb_sectors); I would like to have also a scatter gather API (qbdrvs_readv and qbdrvs_writev) taking a "struct iovec *iov, int niov" instead of "uint8_t *buf, int nb_sectors". flush is missing. > > /* info retrieve */ > //sector, size and geometry info > int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi); One problem here is that BlockDriverInfo may grow in the future. This is a problem for the ABI of the library. I don't have any particular ideas here, except adding a separate API to the library for each member of BlockDriverInfo. > int64_t qbdrvs_getlength(BlockDriverState *bs); qbdrvs_get_length. > int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs); > void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr); Not needed, it's the same as bdrv_getlength. > //image type > const char *qbdrvs_get_format_name(BlockDriverState *bs); > //backing file info > void qbdrvs_get_backing_filename(BlockDriverState *bs, >char *filename, int filename_size); > void qbdrvs_get_full_backing_filename(BlockDriverState *bs, > char *dest, size_t sz); > > > /* advanced image content access */ > int qbdrvs_is_allocated(BlockDriverState *bs, int64_t sector_num, int > nb_sectors, > int *pnum); > int qbdrvs_discard(BlockDriverState *bs, int64_t sector_num, int > nb_sectors); > int qbdrvs_has_zero_init(BlockDriverState *bs); Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Hi, following is API draft, prototypes were taken from qemu/block.h, and the API prefix is changed frpm bdrv to qbdrvs, to declare related object is BlockDriverState, not BlockDriver. One issue here is it may require include block_int.h, which is not LGPL2 licensed yet. API format is kept mostly the same with qemu generic block layer, to make it easier for implement, and easier to make qemu migrate on it if possible. /* structure init and uninit */ BlockDriverState *qbdrvs_new(const char *device_name); void qbdrvs_delete(BlockDriverState *bs); /* file open and close */ int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags, BlockDriver *drv); void qbdrvs_close(BlockDriverState *bs); int qbdrvs_img_create(const char *filename, const char *fmt, const char *base_filename, const char *base_fmt, char *options, uint64_t img_size, int flags); /* sync access */ int qbdrvs_read(BlockDriverState *bs, int64_t sector_num, uint8_t *buf, int nb_sectors); int qbdrvs_write(BlockDriverState *bs, int64_t sector_num, const uint8_t *buf, int nb_sectors); /* info retrieve */ //sector, size and geometry info int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi); int64_t qbdrvs_getlength(BlockDriverState *bs); int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs); void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr); //image type const char *qbdrvs_get_format_name(BlockDriverState *bs); //backing file info void qbdrvs_get_backing_filename(BlockDriverState *bs, char *filename, int filename_size); void qbdrvs_get_full_backing_filename(BlockDriverState *bs, char *dest, size_t sz); /* advanced image content access */ int qbdrvs_is_allocated(BlockDriverState *bs, int64_t sector_num, int nb_sectors, int *pnum); int qbdrvs_discard(BlockDriverState *bs, int64_t sector_num, int nb_sectors); int qbdrvs_has_zero_init(BlockDriverState *bs); Il 16/07/2012 10:16, Wenchao Xia ha scritto: Really thanks for the investigation, I paid quite sometime to dig out which license is compatible to LGPL, this have sorted it out. The coroutine and structure inside is quite a challenge. Coroutines are really just a small complication in the program flow if all you support is synchronous access to files (i.e. no HTTP etc.). Their usage should be completely transparent. What about provide the library first in nbd + sync access, and waiting for the library employer response? If it is good to use, then replace implement code to native qemu block layer code, change code's license, while keep API unchanged. You can start by proposing the API. Paolo -- Best Regards Wenchao Xia
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Sat, Jul 14, 2012 at 01:55:07AM +0300, Lluís Vilanova wrote: > Paolo Bonzini writes: > > > Il 13/07/2012 11:51, Paolo Bonzini ha scritto: > >> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto: > "Working around the QEMU block layer license" is not a goal per se, > especially because you haven't a) assessed _what_ is the GPL code that > the library would use; b) told us why the library should not be under > the GPL. > > Please design first according to the functionality you want to > implement, then think about the implementation. > >>> > >>> Licensing is one headache but the real challenge is that the QEMU block > >>> layer relies on the QEMU main loop and a bunch of other architecture. > >> > >> It doesn't really, not on Windows which has no AIO for example. That's > >> why I suggested: > >> > >> - assessing what code is GPL and what are the dependencies on it > > > So I tried trimming down the list of files needed to compile > > qemu tools, and here is a list: > > > Easy to relicense to LGPLv2+: > > block/raw.c none (GPLv2+: Red Hat, IBM) > > error.c LGPLv2 (Red Hat, IBM, Stefan Weil) > > iov.c GPLv2 (Red Hat, SuSE/Hannes Reinecke, > > Michael Tokarev) > > module.cGPLv2 (Red Hat, IBM, Blue Swirl) > > qemu-error.cGPLv2+ (Red Hat, Blue Swirl, IBM) > > trace/control.c GPLv2 (Lluis Vilanova) > > trace/default.c GPLv2 (Lluis Vilanova) > > > (I added some people to Cc. Lluis and Michael, can you also look at > > http://wiki.qemu.org/Relicensing if you're willing to relicense > > your past contributions from GPLv2 to GPLv2+?. Blue Swirl said > > he'd accept any other GPLv2 or GPLv3 compatible license, which > > should include LGPLv2+). > > I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later". What about LGPLv2+? (Note the "L".) Stefan
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 16/07/2012 10:16, Wenchao Xia ha scritto: >> > Really thanks for the investigation, I paid quite sometime to dig out > which license is compatible to LGPL, this have sorted it out. > The coroutine and structure inside is quite a challenge. Coroutines are really just a small complication in the program flow if all you support is synchronous access to files (i.e. no HTTP etc.). Their usage should be completely transparent. > What about > provide the library first in nbd + sync access, and waiting for the > library employer response? If it is good to use, then replace implement > code to native qemu block layer code, change code's license, while keep > API unchanged. You can start by proposing the API. Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
于 2012-7-13 19:33, Paolo Bonzini 写道: Il 13/07/2012 11:51, Paolo Bonzini ha scritto: Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto: "Working around the QEMU block layer license" is not a goal per se, especially because you haven't a) assessed _what_ is the GPL code that the library would use; b) told us why the library should not be under the GPL. Please design first according to the functionality you want to implement, then think about the implementation. Licensing is one headache but the real challenge is that the QEMU block layer relies on the QEMU main loop and a bunch of other architecture. It doesn't really, not on Windows which has no AIO for example. That's why I suggested: - assessing what code is GPL and what are the dependencies on it So I tried trimming down the list of files needed to compile qemu tools, and here is a list: Really thanks for the investigation, I paid quite sometime to dig out which license is compatible to LGPL, this have sorted it out. The coroutine and structure inside is quite a challenge. What about provide the library first in nbd + sync access, and waiting for the library employer response? If it is good to use, then replace implement code to native qemu block layer code, change code's license, while keep API unchanged. Easy to relicense to LGPLv2+: block/raw.c none (GPLv2+: Red Hat, IBM) error.c LGPLv2 (Red Hat, IBM, Stefan Weil) iov.c GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev) module.cGPLv2 (Red Hat, IBM, Blue Swirl) qemu-error.cGPLv2+ (Red Hat, Blue Swirl, IBM) trace/control.c GPLv2 (Lluis Vilanova) trace/default.c GPLv2 (Lluis Vilanova) (I added some people to Cc. Lluis and Michael, can you also look at http://wiki.qemu.org/Relicensing if you're willing to relicense your past contributions from GPLv2 to GPLv2+?. Blue Swirl said he'd accept any other GPLv2 or GPLv3 compatible license, which should include LGPLv2+). Harder to relicense to LGPLv2+: block/vdi.c GPLv2+ "Good" license: aes.c BSD async.c BSD block.c BSD block/bochs.c BSD block/cloop.c BSD block/cow.c BSD block/dmg.c BSD block/parallels.c BSD block/qcow.cBSD block/qcow2-cache.c BSD block/qcow2-cluster.c BSD block/qcow2-refcount.c BSD block/qcow2-snapshot.c BSD block/qcow2.c BSD block/qed-check.c BSD block/qed-cluster.c BSD block/qed-gencb.c BSD block/qed-l2-cache.cBSD block/qed-table.c BSD block/qed.c BSD block/vmdk.cBSD block/vpc.c BSD block/vvfat.c BSD cutils.cBSD osdep.c BSD oslib-posix.c BSD qemu-coroutine-io.c BSD qemu-coroutine-lock.c BSD qemu-option.c BSD qemu-progress.c BSD coroutine-ucontext.cLGPLv2+ json-lexer.cLGPLv2+ json-parser.c LGPLv2+ json-streamer.c LGPLv2+ qbool.c LGPLv2+ qdict.c LGPLv2+ qemu-coroutine.cLGPLv2+ qerror.cLGPLv2+ qfloat.cLGPLv2+ qint.c LGPLv2+ qjson.c LGPLv2+ qlist.c LGPLv2+ qstring.c LGPLv2+ Doesn't need to be included in a library: qemu-tool.c GPLv2 Autogenerated: trace.c Remaining undefined symbols: qemu_aio_flush qemu_aio_wait qemu_free_timer qemu_new_timer qemu_mod_timer qemu_del_timer qemu_get_clock_ns vm_clock + those defined in qemu-tool.c Paolo -- Best Regards Wenchao Xia
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
于 2012-7-13 17:27, Christoph Hellwig 写道: On Fri, Jul 13, 2012 at 10:13:15AM +0100, Stefan Hajnoczi wrote: How is that different from all the qemu-io commands? qemu-io has no modes to just dumb the output without additional information / statistics or for the write case just take user input instead of a pattern. I actually tried to add raw arguments to qemu-io, which still worked ou ok for reads but started to get fairly ugly for the write. What I use in production right now is a trivial qemu-cat tool that just does the raw reads and writes, but I think adding it as a new sub command to qemu-img instead of another tool seems a bit cleaner. If you and Kevin or Anthony disagree and want the qemu-cat tool I can submit a patch for that instead. That seems one user case the library should support. The library is supposed to do more to expose generic block library which qemu device emulator have used, the ideal expect is that other program linked with the library could operate image the same way as qemu core. -- Best Regards Wenchao Xia
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
> On Fri, Jul 13, 2012 at 10:12:47AM +0100, Stefan Hajnoczi wrote: On Tue, Jul 10, 2012 at 09:18:01AM +0200, Paolo Bonzini wrote: Il 10/07/2012 07:37, Wenchao Xia ha scritto: For getting the other metadata about the disk image you mention, another possibility to is just make 'qemu-img info' return the data in a machine parseable format, ie JSON& make a client API for extracting data from this JSON document. Thank u for the idea. The .so is introduced to let program access the image more directly, parsing string is not so fast and it depends on another program's stdout output, I hope to get a faster way. I doubt you actually have profiled it. I think speed is not the issue, instead it's just providing an API that external programs can use. Management tools, backup software, custom administration tools, etc. It's convenient to have an API. Actually I think speed could well be quite relevant. In large deployments it would not be surpising to see 1000's of images in a directory. If you want to be able to query metadata about all of them at once, then being able to open()+read(4k)+close() 1000 times is going to be dramatically faster than doing fork()+execve(qemu-img) 1000 times. I tried to avoid fork() and execv() in the library as much as possible. Dealing with another program's string output will cost quite some codes about its timeout or exception error, more "native" code seems better to me. NB, I still think qemu-img info should be able to return a JSON parsable data format, regardless of what any block library does. An option to qemu-img to specify output format may be a good way. Daniel -- Best Regards Wenchao Xia
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Paolo Bonzini writes: > Il 13/07/2012 11:51, Paolo Bonzini ha scritto: >> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto: "Working around the QEMU block layer license" is not a goal per se, especially because you haven't a) assessed _what_ is the GPL code that the library would use; b) told us why the library should not be under the GPL. Please design first according to the functionality you want to implement, then think about the implementation. >>> >>> Licensing is one headache but the real challenge is that the QEMU block >>> layer relies on the QEMU main loop and a bunch of other architecture. >> >> It doesn't really, not on Windows which has no AIO for example. That's >> why I suggested: >> >> - assessing what code is GPL and what are the dependencies on it > So I tried trimming down the list of files needed to compile > qemu tools, and here is a list: > Easy to relicense to LGPLv2+: > block/raw.c none (GPLv2+: Red Hat, IBM) > error.c LGPLv2 (Red Hat, IBM, Stefan Weil) > iov.c GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael > Tokarev) > module.cGPLv2 (Red Hat, IBM, Blue Swirl) > qemu-error.cGPLv2+ (Red Hat, Blue Swirl, IBM) > trace/control.c GPLv2 (Lluis Vilanova) > trace/default.c GPLv2 (Lluis Vilanova) > (I added some people to Cc. Lluis and Michael, can you also look at > http://wiki.qemu.org/Relicensing if you're willing to relicense > your past contributions from GPLv2 to GPLv2+?. Blue Swirl said > he'd accept any other GPLv2 or GPLv3 compatible license, which > should include LGPLv2+). I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later". Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Am 13.07.2012 13:33, schrieb Paolo Bonzini: Il 13/07/2012 11:51, Paolo Bonzini ha scritto: Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto: "Working around the QEMU block layer license" is not a goal per se, especially because you haven't a) assessed _what_ is the GPL code that the library would use; b) told us why the library should not be under the GPL. Please design first according to the functionality you want to implement, then think about the implementation. Licensing is one headache but the real challenge is that the QEMU block layer relies on the QEMU main loop and a bunch of other architecture. It doesn't really, not on Windows which has no AIO for example. That's why I suggested: - assessing what code is GPL and what are the dependencies on it So I tried trimming down the list of files needed to compile qemu tools, and here is a list: Easy to relicense to LGPLv2+: block/raw.c none (GPLv2+: Red Hat, IBM) error.c LGPLv2 (Red Hat, IBM, Stefan Weil) I only added an include statement and don't mind changing the license for error.c to LGPLv2+. iov.c GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev) module.cGPLv2 (Red Hat, IBM, Blue Swirl) qemu-error.cGPLv2+ (Red Hat, Blue Swirl, IBM) trace/control.c GPLv2 (Lluis Vilanova) trace/default.c GPLv2 (Lluis Vilanova) (I added some people to Cc. Lluis and Michael, can you also look at http://wiki.qemu.org/Relicensing if you're willing to relicense your past contributions from GPLv2 to GPLv2+?. Blue Swirl said he'd accept any other GPLv2 or GPLv3 compatible license, which should include LGPLv2+). Harder to relicense to LGPLv2+: block/vdi.c GPLv2+ Indeed, that one is harder. Most of the code is from me, and I need a good reason why the license should be changed. Of course the dynamic library can also be compiled without VDI support. Regards, Stefan W.
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Fri, Jul 13, 2012 at 11:33 AM, Paolo Bonzini wrote: > Il 13/07/2012 11:51, Paolo Bonzini ha scritto: >> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto: "Working around the QEMU block layer license" is not a goal per se, especially because you haven't a) assessed _what_ is the GPL code that the library would use; b) told us why the library should not be under the GPL. Please design first according to the functionality you want to implement, then think about the implementation. >>> >>> Licensing is one headache but the real challenge is that the QEMU block >>> layer relies on the QEMU main loop and a bunch of other architecture. >> >> It doesn't really, not on Windows which has no AIO for example. That's >> why I suggested: >> >> - assessing what code is GPL and what are the dependencies on it > > So I tried trimming down the list of files needed to compile > qemu tools, and here is a list: > > Easy to relicense to LGPLv2+: > block/raw.c none (GPLv2+: Red Hat, IBM) > error.c LGPLv2 (Red Hat, IBM, Stefan Weil) > iov.c GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael > Tokarev) > module.cGPLv2 (Red Hat, IBM, Blue Swirl) > qemu-error.cGPLv2+ (Red Hat, Blue Swirl, IBM) > trace/control.c GPLv2 (Lluis Vilanova) > trace/default.c GPLv2 (Lluis Vilanova) > > (I added some people to Cc. Lluis and Michael, can you also look at > http://wiki.qemu.org/Relicensing if you're willing to relicense > your past contributions from GPLv2 to GPLv2+?. Blue Swirl said > he'd accept any other GPLv2 or GPLv3 compatible license, which > should include LGPLv2+). I'm fine with LGPLv2+ too. I have some reservations to supporting a block device library API/ABI by external users, but if this is OK with block maintainer and we allow some decent API/ABI changes in the future if needed, this should be OK. > > Harder to relicense to LGPLv2+: > block/vdi.c GPLv2+ > > "Good" license: > aes.c BSD > async.c BSD > block.c BSD > block/bochs.c BSD > block/cloop.c BSD > block/cow.c BSD > block/dmg.c BSD > block/parallels.c BSD > block/qcow.cBSD > block/qcow2-cache.c BSD > block/qcow2-cluster.c BSD > block/qcow2-refcount.c BSD > block/qcow2-snapshot.c BSD > block/qcow2.c BSD > block/qed-check.c BSD > block/qed-cluster.c BSD > block/qed-gencb.c BSD > block/qed-l2-cache.cBSD > block/qed-table.c BSD > block/qed.c BSD > block/vmdk.cBSD > block/vpc.c BSD > block/vvfat.c BSD > cutils.cBSD > osdep.c BSD > oslib-posix.c BSD > qemu-coroutine-io.c BSD > qemu-coroutine-lock.c BSD > qemu-option.c BSD > qemu-progress.c BSD > coroutine-ucontext.cLGPLv2+ > json-lexer.cLGPLv2+ > json-parser.c LGPLv2+ > json-streamer.c LGPLv2+ > qbool.c LGPLv2+ > qdict.c LGPLv2+ > qemu-coroutine.cLGPLv2+ > qerror.cLGPLv2+ > qfloat.cLGPLv2+ > qint.c LGPLv2+ > qjson.c LGPLv2+ > qlist.c LGPLv2+ > qstring.c LGPLv2+ > > Doesn't need to be included in a library: > qemu-tool.c GPLv2 > > Autogenerated: > trace.c > > Remaining undefined symbols: > qemu_aio_flush > qemu_aio_wait > qemu_free_timer > qemu_new_timer > qemu_mod_timer > qemu_del_timer > qemu_get_clock_ns > vm_clock > + those defined in qemu-tool.c > > Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On 13.07.2012 15:33, Paolo Bonzini wrote: [] > So I tried trimming down the list of files needed to compile > qemu tools, and here is a list: > > Easy to relicense to LGPLv2+: > block/raw.c none (GPLv2+: Red Hat, IBM) > error.c LGPLv2 (Red Hat, IBM, Stefan Weil) > iov.c GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael > Tokarev) > module.cGPLv2 (Red Hat, IBM, Blue Swirl) > qemu-error.cGPLv2+ (Red Hat, Blue Swirl, IBM) > trace/control.c GPLv2 (Lluis Vilanova) > trace/default.c GPLv2 (Lluis Vilanova) > > (I added some people to Cc. Lluis and Michael, can you also look at > http://wiki.qemu.org/Relicensing if you're willing to relicense > your past contributions from GPLv2 to GPLv2+?. Blue Swirl said > he'd accept any other GPLv2 or GPLv3 compatible license, which > should include LGPLv2+). I'm fine with relicensing any my contributions to qemu project under any version of GPL or LGPL, BSD, or any other open license, as the project see fit. Thank you! /mjt
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 13/07/2012 11:51, Paolo Bonzini ha scritto: > Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto: >>> "Working around the QEMU block layer license" is not a goal per se, >>> especially because you haven't a) assessed _what_ is the GPL code that >>> the library would use; b) told us why the library should not be under >>> the GPL. >>> >>> Please design first according to the functionality you want to >>> implement, then think about the implementation. >> >> Licensing is one headache but the real challenge is that the QEMU block >> layer relies on the QEMU main loop and a bunch of other architecture. > > It doesn't really, not on Windows which has no AIO for example. That's > why I suggested: > > - assessing what code is GPL and what are the dependencies on it So I tried trimming down the list of files needed to compile qemu tools, and here is a list: Easy to relicense to LGPLv2+: block/raw.c none (GPLv2+: Red Hat, IBM) error.c LGPLv2 (Red Hat, IBM, Stefan Weil) iov.c GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev) module.cGPLv2 (Red Hat, IBM, Blue Swirl) qemu-error.cGPLv2+ (Red Hat, Blue Swirl, IBM) trace/control.c GPLv2 (Lluis Vilanova) trace/default.c GPLv2 (Lluis Vilanova) (I added some people to Cc. Lluis and Michael, can you also look at http://wiki.qemu.org/Relicensing if you're willing to relicense your past contributions from GPLv2 to GPLv2+?. Blue Swirl said he'd accept any other GPLv2 or GPLv3 compatible license, which should include LGPLv2+). Harder to relicense to LGPLv2+: block/vdi.c GPLv2+ "Good" license: aes.c BSD async.c BSD block.c BSD block/bochs.c BSD block/cloop.c BSD block/cow.c BSD block/dmg.c BSD block/parallels.c BSD block/qcow.cBSD block/qcow2-cache.c BSD block/qcow2-cluster.c BSD block/qcow2-refcount.c BSD block/qcow2-snapshot.c BSD block/qcow2.c BSD block/qed-check.c BSD block/qed-cluster.c BSD block/qed-gencb.c BSD block/qed-l2-cache.cBSD block/qed-table.c BSD block/qed.c BSD block/vmdk.cBSD block/vpc.c BSD block/vvfat.c BSD cutils.cBSD osdep.c BSD oslib-posix.c BSD qemu-coroutine-io.c BSD qemu-coroutine-lock.c BSD qemu-option.c BSD qemu-progress.c BSD coroutine-ucontext.cLGPLv2+ json-lexer.cLGPLv2+ json-parser.c LGPLv2+ json-streamer.c LGPLv2+ qbool.c LGPLv2+ qdict.c LGPLv2+ qemu-coroutine.cLGPLv2+ qerror.cLGPLv2+ qfloat.cLGPLv2+ qint.c LGPLv2+ qjson.c LGPLv2+ qlist.c LGPLv2+ qstring.c LGPLv2+ Doesn't need to be included in a library: qemu-tool.c GPLv2 Autogenerated: trace.c Remaining undefined symbols: qemu_aio_flush qemu_aio_wait qemu_free_timer qemu_new_timer qemu_mod_timer qemu_del_timer qemu_get_clock_ns vm_clock + those defined in qemu-tool.c Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Am 13.07.2012 12:55, schrieb Christoph Hellwig: > On Fri, Jul 13, 2012 at 12:42:41PM +0200, Kevin Wolf wrote: >> It still feels a bit more like qemu-io-style operations. Not sure what >> your use case looks like exactly, but adding a qemu-io command that >> reads data from a file and writes it at a given offset into the images >> (or vice versa) should be easy. This would be more or less a qemu-dd. > > We already have that if you read/write the whole image, it's called > qemu-img convert. Hm. Fair point, I guess. Though qemu-img tends to have commands that deal with complete disks instead of just areas. >> If you need to get data from stdin or output it to stdout, then it might >> not be the right solution. > > That is the use case. We could appromite it by writing a temp file and > using qemu-img convert, but that's not very efficient. Yeah, it's not what you want then. So the question is whether to have it integrated in qemu-img or standalone. I'm undecided: Having everything in one well-known tool has its advantages. But then, a qemu-dd that feels like a real dd, just that it opens image formats with the right driver instead of always using raw, certainly sounds attractive, too. Let's wait a bit for more opinions. If there aren't any - you write the code, you decide. Kevin
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Fri, Jul 13, 2012 at 12:42:41PM +0200, Kevin Wolf wrote: > It still feels a bit more like qemu-io-style operations. Not sure what > your use case looks like exactly, but adding a qemu-io command that > reads data from a file and writes it at a given offset into the images > (or vice versa) should be easy. This would be more or less a qemu-dd. We already have that if you read/write the whole image, it's called qemu-img convert. > If you need to get data from stdin or output it to stdout, then it might > not be the right solution. That is the use case. We could appromite it by writing a temp file and using qemu-img convert, but that's not very efficient.
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Am 13.07.2012 11:43, schrieb Stefan Hajnoczi: > On Fri, Jul 13, 2012 at 11:27:55AM +0200, Christoph Hellwig wrote: >> On Fri, Jul 13, 2012 at 10:13:15AM +0100, Stefan Hajnoczi wrote: >>> How is that different from all the qemu-io commands? >> >> qemu-io has no modes to just dumb the output without additional >> information / statistics or for the write case just take user input >> instead of a pattern. I actually tried to add raw arguments to >> qemu-io, which still worked ou ok for reads but started to get >> fairly ugly for the write. >> >> What I use in production right now is a trivial qemu-cat tool that >> just does the raw reads and writes, but I think adding it as a new >> sub command to qemu-img instead of another tool seems a bit cleaner. >> >> If you and Kevin or Anthony disagree and want the qemu-cat tool I can >> submit a patch for that instead. > > Okay, I see what you mean. I have used the hex output mode (when you > use the verbose option) but it's not raw. > > Sounds like you want a qemu-dd :). I think adding that to qemu-img is > fine though since it's already the tool that users are familiar with for > image file manipulation and that gets shipped. It still feels a bit more like qemu-io-style operations. Not sure what your use case looks like exactly, but adding a qemu-io command that reads data from a file and writes it at a given offset into the images (or vice versa) should be easy. This would be more or less a qemu-dd. If you need to get data from stdin or output it to stdout, then it might not be the right solution. Kevin
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto: >> "Working around the QEMU block layer license" is not a goal per se, >> especially because you haven't a) assessed _what_ is the GPL code that >> the library would use; b) told us why the library should not be under >> the GPL. >> >> Please design first according to the functionality you want to >> implement, then think about the implementation. > > Licensing is one headache but the real challenge is that the QEMU block > layer relies on the QEMU main loop and a bunch of other architecture. It doesn't really, not on Windows which has no AIO for example. That's why I suggested: - assessing what code is GPL and what are the dependencies on it - only including the formats in the library + a synchronous file protocol. No NBD, no libiscsi, no hdev, etc. My suspicion is that with this strategy the only complicated dependency is on coroutines. Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Fri, Jul 13, 2012 at 10:16:40AM +0100, Daniel P. Berrange wrote: > On Fri, Jul 13, 2012 at 10:12:47AM +0100, Stefan Hajnoczi wrote: > > On Tue, Jul 10, 2012 at 09:18:01AM +0200, Paolo Bonzini wrote: > > > Il 10/07/2012 07:37, Wenchao Xia ha scritto: > > > >> > > > >> For getting the other metadata about the disk image you mention, > > > >> another > > > >> possibility to is just make 'qemu-img info' return the data in a > > > >> machine > > > >> parseable format, ie JSON& make a client API for extracting data from > > > >> this JSON document. > > > >> > > > > Thank u for the idea. The .so is introduced to let program access the > > > > image more directly, parsing string is not so fast and it depends on > > > > another program's stdout output, I hope to get a faster way. > > > > > > I doubt you actually have profiled it. > > > > I think speed is not the issue, instead it's just providing an API that > > external programs can use. Management tools, backup software, custom > > administration tools, etc. It's convenient to have an API. > > Actually I think speed could well be quite relevant. In large deployments > it would not be surpising to see 1000's of images in a directory. If you > want to be able to query metadata about all of them at once, then being > able to open()+read(4k)+close() 1000 times is going to be dramatically > faster than doing fork()+execve(qemu-img) 1000 times. Yes, that's true. For situations like querying a whole repository of image files speed matters. I still think that we should focus on the API first. Whether the .so forks qemu-nbd/qemu-img or includes the actual block layer code only matters once we are satisfied that this library is useful and works. And for applications that use qemu-img today there won't be a speed decrease, at least. Stefan
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Fri, Jul 13, 2012 at 11:27:55AM +0200, Christoph Hellwig wrote: > On Fri, Jul 13, 2012 at 10:13:15AM +0100, Stefan Hajnoczi wrote: > > How is that different from all the qemu-io commands? > > qemu-io has no modes to just dumb the output without additional > information / statistics or for the write case just take user input > instead of a pattern. I actually tried to add raw arguments to > qemu-io, which still worked ou ok for reads but started to get > fairly ugly for the write. > > What I use in production right now is a trivial qemu-cat tool that > just does the raw reads and writes, but I think adding it as a new > sub command to qemu-img instead of another tool seems a bit cleaner. > > If you and Kevin or Anthony disagree and want the qemu-cat tool I can > submit a patch for that instead. Okay, I see what you mean. I have used the hex output mode (when you use the verbose option) but it's not raw. Sounds like you want a qemu-dd :). I think adding that to qemu-img is fine though since it's already the tool that users are familiar with for image file manipulation and that gets shipped. Stefan
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Fri, Jul 13, 2012 at 10:13:15AM +0100, Stefan Hajnoczi wrote: > How is that different from all the qemu-io commands? qemu-io has no modes to just dumb the output without additional information / statistics or for the write case just take user input instead of a pattern. I actually tried to add raw arguments to qemu-io, which still worked ou ok for reads but started to get fairly ugly for the write. What I use in production right now is a trivial qemu-cat tool that just does the raw reads and writes, but I think adding it as a new sub command to qemu-img instead of another tool seems a bit cleaner. If you and Kevin or Anthony disagree and want the qemu-cat tool I can submit a patch for that instead.
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Fri, Jul 13, 2012 at 10:12:47AM +0100, Stefan Hajnoczi wrote: > On Tue, Jul 10, 2012 at 09:18:01AM +0200, Paolo Bonzini wrote: > > Il 10/07/2012 07:37, Wenchao Xia ha scritto: > > >> > > >> For getting the other metadata about the disk image you mention, another > > >> possibility to is just make 'qemu-img info' return the data in a machine > > >> parseable format, ie JSON& make a client API for extracting data from > > >> this JSON document. > > >> > > > Thank u for the idea. The .so is introduced to let program access the > > > image more directly, parsing string is not so fast and it depends on > > > another program's stdout output, I hope to get a faster way. > > > > I doubt you actually have profiled it. > > I think speed is not the issue, instead it's just providing an API that > external programs can use. Management tools, backup software, custom > administration tools, etc. It's convenient to have an API. Actually I think speed could well be quite relevant. In large deployments it would not be surpising to see 1000's of images in a directory. If you want to be able to query metadata about all of them at once, then being able to open()+read(4k)+close() 1000 times is going to be dramatically faster than doing fork()+execve(qemu-img) 1000 times. NB, I still think qemu-img info should be able to return a JSON parsable data format, regardless of what any block library does. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Tue, Jul 10, 2012 at 09:17:05AM +0200, Paolo Bonzini wrote: > Il 10/07/2012 07:04, Wenchao Xia ha scritto: > > 于 2012-7-9 17:13, Paolo Bonzini 写道: > >> Il 09/07/2012 10:54, Wenchao Xia ha scritto: > >>> Following is my implementing plan draft: > >>>1 introduce libqblock.so in sub directory in qemu. > >>>2 write a nbd client in libqblock, similar to qemu nbd client. Then > >>> use it to talk with nbd server, by default is qemu-nbd, to get access > >>> to images. In this way, libqblock.so could be friendly LGPL licensed. > >> > >> Did you actually assess the license situation of the block layer? > >> block.c and large parts of block/* are under a BSD license, for example. > >> If the library only has to support raw files, it might do so using > >> synchronous I/O only. This would remove a large body of GPL-licensed code. > >> > > If the library was built as nbd-client communicating with nbd-server, > > which then employ the BSO licensed code, could the library ignore the > > server side's license problem? > > Yes, but if your first worry is the legal problem you are doomed to > design an awful library. > > > The reason using nbd-client approach are: > > work around qemu block layer license issue and easy to implement > > "Working around the QEMU block layer license" is not a goal per se, > especially because you haven't a) assessed _what_ is the GPL code that > the library would use; b) told us why the library should not be under > the GPL. > > Please design first according to the functionality you want to > implement, then think about the implementation. Licensing is one headache but the real challenge is that the QEMU block layer relies on the QEMU main loop and a bunch of other architecture. Using NBD allows us to focus on the library API instead worrying about untangling the block layer. If the library becomes useful it may be worth fully moving the block layer code over into the library. I think it makes sense to hammer out the library first before going down a long road of internal QEMU refactoring before we know whether the library will take off. Stefan
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Mon, Jul 09, 2012 at 04:36:07PM +0200, Christoph Hellwig wrote: > On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote: > > Hi, Paolo and folks, > > qemu have good capabilities to access different virtual disks, I want > > to expose its block layer API to let 3rd party program linked in, such > > as management stack or block tools, to access images data directly. > > > > Following is the objects: > > (1) API to write/read block device at offset. > > (2) Determine the image type,qcow2/qed/raw > > (3) Determine which blocks are allocated. > > (4) Determine backing file. > > Sounds like you want a procedural interface for that. At least for (1) > I have patches I'll submit soon to add qemu img read/write commands. How is that different from all the qemu-io commands? Stefan
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Tue, Jul 10, 2012 at 09:18:01AM +0200, Paolo Bonzini wrote: > Il 10/07/2012 07:37, Wenchao Xia ha scritto: > >> > >> For getting the other metadata about the disk image you mention, another > >> possibility to is just make 'qemu-img info' return the data in a machine > >> parseable format, ie JSON& make a client API for extracting data from > >> this JSON document. > >> > > Thank u for the idea. The .so is introduced to let program access the > > image more directly, parsing string is not so fast and it depends on > > another program's stdout output, I hope to get a faster way. > > I doubt you actually have profiled it. I think speed is not the issue, instead it's just providing an API that external programs can use. Management tools, backup software, custom administration tools, etc. It's convenient to have an API. Stefan
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 10/07/2012 07:37, Wenchao Xia ha scritto: >> >> For getting the other metadata about the disk image you mention, another >> possibility to is just make 'qemu-img info' return the data in a machine >> parseable format, ie JSON& make a client API for extracting data from >> this JSON document. >> > Thank u for the idea. The .so is introduced to let program access the > image more directly, parsing string is not so fast and it depends on > another program's stdout output, I hope to get a faster way. I doubt you actually have profiled it. Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 10/07/2012 07:04, Wenchao Xia ha scritto: > 于 2012-7-9 17:13, Paolo Bonzini 写道: >> Il 09/07/2012 10:54, Wenchao Xia ha scritto: >>> Following is my implementing plan draft: >>>1 introduce libqblock.so in sub directory in qemu. >>>2 write a nbd client in libqblock, similar to qemu nbd client. Then >>> use it to talk with nbd server, by default is qemu-nbd, to get access >>> to images. In this way, libqblock.so could be friendly LGPL licensed. >> >> Did you actually assess the license situation of the block layer? >> block.c and large parts of block/* are under a BSD license, for example. >> If the library only has to support raw files, it might do so using >> synchronous I/O only. This would remove a large body of GPL-licensed code. >> > If the library was built as nbd-client communicating with nbd-server, > which then employ the BSO licensed code, could the library ignore the > server side's license problem? Yes, but if your first worry is the legal problem you are doomed to design an awful library. > The reason using nbd-client approach are: > work around qemu block layer license issue and easy to implement "Working around the QEMU block layer license" is not a goal per se, especially because you haven't a) assessed _what_ is the GPL code that the library would use; b) told us why the library should not be under the GPL. Please design first according to the functionality you want to implement, then think about the implementation. > , if > other tool found this labrary useful then considering about directly > employ the qemu block code. Again, I find this to be quite backwards. Writing a replacement for the QEMU block layer just for licensing reasons is going to be a waste of resources, since 90% of it is already BSD-licensed. Perhaps you can produce two variants of the library, one using GPLed backends and one entirely under the BSD license. That would be good. However we cannot help much in finding the best way to reach your goals, again because you haven't reported on what actually is the GPL code that you're worried about and why. >>>3 still not got a good way to get additional info in (2)(3)(4), >>> currently in my head is patch qemu-nbd to add an additional nbd command, >>> "image-info", in which returns related info. >> >> On the Linux kernel mailing list I would have no qualms labeling such >> command as "crap". However, since the social standards on qemu-devel >> are a bit higher, I'll ask instead: what information would the command >> provide beyond the size? >> > The API need to report the image format it is using, such as > "qcow2". And also API should report if a block at offset have been > allocated or it is a hole. qemu-nbd is designed to always provide the image format as raw, so its client has no business knowing whether the image is originally stored as qcow2 or something else. Paolo
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
于 2012-7-9 22:36, Christoph Hellwig 写道: On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote: Hi, Paolo and folks, qemu have good capabilities to access different virtual disks, I want to expose its block layer API to let 3rd party program linked in, such as management stack or block tools, to access images data directly. Following is the objects: (1) API to write/read block device at offset. (2) Determine the image type,qcow2/qed/raw (3) Determine which blocks are allocated. (4) Determine backing file. Sounds like you want a procedural interface for that. At least for (1) I have patches I'll submit soon to add qemu img read/write commands. Yes, the purpose is introduce API interface about block data, the operation was supposed to happen frequently, a linked-in library may be better than process output string parsing in performance. -- Best Regards Wenchao Xia
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
于 2012-7-9 17:27, Daniel P. Berrange 写道: On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote: Hi, Paolo and folks, qemu have good capabilities to access different virtual disks, I want to expose its block layer API to let 3rd party program linked in, such as management stack or block tools, to access images data directly. Following is the objects: (1) API to write/read block device at offset. (2) Determine the image type,qcow2/qed/raw (3) Determine which blocks are allocated. (4) Determine backing file. Following is my implementing plan draft: 1 introduce libqblock.so in sub directory in qemu. 2 write a nbd client in libqblock, similar to qemu nbd client. Then use it to talk with nbd server, by default is qemu-nbd, to get access to images. In this way, libqblock.so could be friendly LGPL licensed. 3 still not got a good way to get additional info in (2)(3)(4), currently in my head is patch qemu-nbd to add an additional nbd command, "image-info", in which returns related info. What do you think about it? For arbirary read/write access to disk images, I can see a little value in having a standalone libnbd client API, that is able to just talk to any NBD server. Arguably such a thing does not need to be part of the QEMU source tree - eg see the recently written libiscsi.so client. For getting the other metadata about the disk image you mention, another possibility to is just make 'qemu-img info' return the data in a machine parseable format, ie JSON& make a client API for extracting data from this JSON document. Thank u for the idea. The .so is introduced to let program access the image more directly, parsing string is not so fast and it depends on another program's stdout output, I hope to get a faster way. For a full-blown RPC API for doing arbitrary tasks related to block devices, then many apps will tend towards libguestfs, since it provides such a wide range of functionality for manipulating disk images. I used libguestfs to make my image too, but the target of the .so is a bit different: it expose block data in a lower level, expose everything qemu main code(except block code) can see. Potential purpose is as first step to make qemu block layer independent, then qemu and other tool would be just employers of the library. But now it is out of plan because license issue, and the library acts as client now lowering the performance. Regards, Daniel -- Best Regards Wenchao Xia
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
于 2012-7-9 17:13, Paolo Bonzini 写道: > Il 09/07/2012 10:54, Wenchao Xia ha scritto: >> Following is my implementing plan draft: >>1 introduce libqblock.so in sub directory in qemu. >>2 write a nbd client in libqblock, similar to qemu nbd client. Then >> use it to talk with nbd server, by default is qemu-nbd, to get access >> to images. In this way, libqblock.so could be friendly LGPL licensed. > > Did you actually assess the license situation of the block layer? > block.c and large parts of block/* are under a BSD license, for example. > If the library only has to support raw files, it might do so using > synchronous I/O only. This would remove a large body of GPL-licensed code. > If the library was built as nbd-client communicating with nbd-server, which then employ the BSO licensed code, could the library ignore the server side's license problem? The reason using nbd-client approach are: work around qemu block layer license issue and easy to implement, if other tool found this labrary useful then considering about directly employ the qemu block code. >>3 still not got a good way to get additional info in (2)(3)(4), >> currently in my head is patch qemu-nbd to add an additional nbd command, >> "image-info", in which returns related info. > > On the Linux kernel mailing list I would have no qualms labeling such > command as "crap". However, since the social standards on qemu-devel > are a bit higher, I'll ask instead: what information would the command > provide beyond the size? > The API need to report the image format it is using, such as "qcow2". And also API should report if a block at offset have been allocated or it is a hole. > Paolo > -- Best Regards Wenchao Xia
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote: > Hi, Paolo and folks, > qemu have good capabilities to access different virtual disks, I want > to expose its block layer API to let 3rd party program linked in, such > as management stack or block tools, to access images data directly. > > Following is the objects: > (1) API to write/read block device at offset. > (2) Determine the image type,qcow2/qed/raw > (3) Determine which blocks are allocated. > (4) Determine backing file. Sounds like you want a procedural interface for that. At least for (1) I have patches I'll submit soon to add qemu img read/write commands.
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote: > Hi, Paolo and folks, > qemu have good capabilities to access different virtual disks, I want > to expose its block layer API to let 3rd party program linked in, such > as management stack or block tools, to access images data directly. > > Following is the objects: > (1) API to write/read block device at offset. > (2) Determine the image type,qcow2/qed/raw > (3) Determine which blocks are allocated. > (4) Determine backing file. > > Following is my implementing plan draft: > 1 introduce libqblock.so in sub directory in qemu. > 2 write a nbd client in libqblock, similar to qemu nbd client. Then > use it to talk with nbd server, by default is qemu-nbd, to get access > to images. In this way, libqblock.so could be friendly LGPL licensed. > 3 still not got a good way to get additional info in (2)(3)(4), > currently in my head is patch qemu-nbd to add an additional nbd command, > "image-info", in which returns related info. > > What do you think about it? For arbirary read/write access to disk images, I can see a little value in having a standalone libnbd client API, that is able to just talk to any NBD server. Arguably such a thing does not need to be part of the QEMU source tree - eg see the recently written libiscsi.so client. For getting the other metadata about the disk image you mention, another possibility to is just make 'qemu-img info' return the data in a machine parseable format, ie JSON & make a client API for extracting data from this JSON document. For a full-blown RPC API for doing arbitrary tasks related to block devices, then many apps will tend towards libguestfs, since it provides such a wide range of functionality for manipulating disk images. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Il 09/07/2012 10:54, Wenchao Xia ha scritto: > Following is my implementing plan draft: > 1 introduce libqblock.so in sub directory in qemu. > 2 write a nbd client in libqblock, similar to qemu nbd client. Then > use it to talk with nbd server, by default is qemu-nbd, to get access > to images. In this way, libqblock.so could be friendly LGPL licensed. Did you actually assess the license situation of the block layer? block.c and large parts of block/* are under a BSD license, for example. If the library only has to support raw files, it might do so using synchronous I/O only. This would remove a large body of GPL-licensed code. > 3 still not got a good way to get additional info in (2)(3)(4), > currently in my head is patch qemu-nbd to add an additional nbd command, > "image-info", in which returns related info. On the Linux kernel mailing list I would have no qualms labeling such command as "crap". However, since the social standards on qemu-devel are a bit higher, I'll ask instead: what information would the command provide beyond the size? Paolo
[Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API
Hi, Paolo and folks, qemu have good capabilities to access different virtual disks, I want to expose its block layer API to let 3rd party program linked in, such as management stack or block tools, to access images data directly. Following is the objects: (1) API to write/read block device at offset. (2) Determine the image type,qcow2/qed/raw (3) Determine which blocks are allocated. (4) Determine backing file. Following is my implementing plan draft: 1 introduce libqblock.so in sub directory in qemu. 2 write a nbd client in libqblock, similar to qemu nbd client. Then use it to talk with nbd server, by default is qemu-nbd, to get access to images. In this way, libqblock.so could be friendly LGPL licensed. 3 still not got a good way to get additional info in (2)(3)(4), currently in my head is patch qemu-nbd to add an additional nbd command, "image-info", in which returns related info. What do you think about it? -- Best Regards Wenchao Xia mail:xiaw...@linux.vnet.ibm.com tel:86-010-82450803
