[PATCH 00/26] btrfs-progs: introduce libbtrfsutil, "btrfs-progs as a library"
From: Omar Sandoval Hello, One of the features requests I get most often is a library to do the sorts of operations that we do with btrfs-progs. We can shell out to btrfs-progs, but the output format isn't always easily parsasble, and shelling out isn't always ideal. There's libbtrfs, but it's very incomplete, doesn't have a well thought out API, and is licensed under the GPL, making it hard to use for many projects. libbtrfsutil is a new library written from scratch to address these issues. The implementation is completely independent of the existing btrfs-progs code, including kerncompat.h, and has a clean API and naming scheme. It is licensed under the LGPL. It also includes Python bindings by default. I will maintain the library code. Patch 1 is a preparation cleanup which can go in independently. Patch 2 adds the build system stuff for the library, and patch 3 does the same for the Python bindings. Patches 4-14 implement the library helpers, currently subvolume helpers and the sync ioctls. Patches 15-26 replace the btrfs-progs and libbtrfs code to use libbtrfsutil instead. I took care to preserve backwards-compatibility. `btrfs subvol list` in particular had some buggy behaviors for -o and -a that I emulated in the new code, see the comments in the code. These patches are also available on my GitHub: https://github.com/osandov/btrfs-progs/tree/libbtrfsutil. That branch will rebase as I update this series. Please share feedback regarding the API, implementation, or anything else. Thanks! Omar Sandoval (26): btrfs-progs: get rid of undocumented qgroup inheritance options Add libbtrfsutil libbtrfsutil: add Python bindings libbtrfsutil: add btrfs_util_is_subvolume() and btrfs_util_subvolume_id() libbtrfsutil: add qgroup inheritance helpers libbtrfsutil: add btrfs_util_create_subvolume() libbtrfsutil: add btrfs_util_subvolume_info() libbtrfsutil: add btrfs_util_[gs]et_read_only() libbtrfsutil: add btrfs_util_[gs]et_default_subvolume() libbtrfsutil: add subvolume iterator helpers libbtrfsutil: add btrfs_util_create_snapshot() libbtrfsutil: add btrfs_util_delete_subvolume() libbtrfsutil: add btrfs_util_deleted_subvolumes() libbtrfsutil: add filesystem sync helpers btrfs-progs: use libbtrfsutil for read-only property btrfs-progs: use libbtrfsutil for sync ioctls btrfs-progs: use libbtrfsutil for set-default btrfs-progs: use libbtrfsutil for get-default btrfs-progs: use libbtrfsutil for subvol create and snapshot btrfs-progs: use libbtrfsutil for subvol delete btrfs-progs: use libbtrfsutil for subvol show btrfs-progs: use libbtrfsutil for subvol sync btrfs-progs: replace test_issubvolume() with btrfs_util_is_subvolume() btrfs-progs: add recursive snapshot/delete using libbtrfsutil btrfs-progs: deprecate libbtrfs helpers with libbtrfsutil equivalents btrfs-progs: use libbtrfsutil for subvolume list .gitignore |2 + Documentation/btrfs-subvolume.asciidoc | 14 +- INSTALL |4 + Makefile | 84 +- Makefile.inc.in |2 + btrfs-list.c | 1201 +++--- btrfs-list.h | 22 +- cmds-filesystem.c| 20 +- cmds-inspect.c |9 +- cmds-qgroup.c| 20 +- cmds-receive.c | 12 +- cmds-subvolume.c | 807 +-- configure.ac | 15 + libbtrfsutil/COPYING | 674 + libbtrfsutil/COPYING.LESSER | 165 +++ libbtrfsutil/README.md | 35 + libbtrfsutil/btrfsutil.h | 621 libbtrfsutil/errors.c| 55 + libbtrfsutil/filesystem.c| 103 ++ libbtrfsutil/internal.h | 36 + libbtrfsutil/python/.gitignore |7 + libbtrfsutil/python/btrfsutilpy.h| 84 ++ libbtrfsutil/python/error.c | 202 libbtrfsutil/python/filesystem.c | 94 ++ libbtrfsutil/python/module.c | 321 ++ libbtrfsutil/python/qgroup.c | 141 +++ libbtrfsutil/python/setup.py | 103 ++ libbtrfsutil/python/subvolume.c | 665 + libbtrfsutil/python/tests/__init__.py| 66 ++ libbtrfsutil/python/tests/test_filesystem.py | 73 ++ libbtrfsutil/python/tests/test_qgroup.py | 57 ++ libbtrfsutil/python/tests/test_subvolume.py | 383 +++ libbtrfsutil/qgroup.c| 86 ++ libbtrfsutil/subvolume.c | 1383 ++ messages.h | 14 + props.c
Re: [PATCH 00/26] btrfs-progs: introduce libbtrfsutil, "btrfs-progs as a library"
On Fri, Jan 26, 2018 at 10:40:48AM -0800, Omar Sandoval wrote: > From: Omar Sandoval > > Hello, > > One of the features requests I get most often is a library to do the > sorts of operations that we do with btrfs-progs. We can shell out to > btrfs-progs, but the output format isn't always easily parsasble, and > shelling out isn't always ideal. There's libbtrfs, but it's very > incomplete, doesn't have a well thought out API, and is licensed under > the GPL, making it hard to use for many projects. > > libbtrfsutil is a new library written from scratch to address these > issues. The implementation is completely independent of the existing > btrfs-progs code, including kerncompat.h, and has a clean API and naming > scheme. It is licensed under the LGPL. It also includes Python bindings > by default. I will maintain the library code. *speechless* That's awesome, Omar (although with the python bindings, you've probably just ruined Hans' day ;) ). Hugo. > Patch 1 is a preparation cleanup which can go in independently. Patch 2 > adds the build system stuff for the library, and patch 3 does the same > for the Python bindings. Patches 4-14 implement the library helpers, > currently subvolume helpers and the sync ioctls. Patches 15-26 replace > the btrfs-progs and libbtrfs code to use libbtrfsutil instead. I took > care to preserve backwards-compatibility. `btrfs subvol list` in > particular had some buggy behaviors for -o and -a that I emulated in the > new code, see the comments in the code. > > These patches are also available on my GitHub: > https://github.com/osandov/btrfs-progs/tree/libbtrfsutil. That branch > will rebase as I update this series. > > Please share feedback regarding the API, implementation, or anything > else. > > Thanks! > > Omar Sandoval (26): > btrfs-progs: get rid of undocumented qgroup inheritance options > Add libbtrfsutil > libbtrfsutil: add Python bindings > libbtrfsutil: add btrfs_util_is_subvolume() and > btrfs_util_subvolume_id() > libbtrfsutil: add qgroup inheritance helpers > libbtrfsutil: add btrfs_util_create_subvolume() > libbtrfsutil: add btrfs_util_subvolume_info() > libbtrfsutil: add btrfs_util_[gs]et_read_only() > libbtrfsutil: add btrfs_util_[gs]et_default_subvolume() > libbtrfsutil: add subvolume iterator helpers > libbtrfsutil: add btrfs_util_create_snapshot() > libbtrfsutil: add btrfs_util_delete_subvolume() > libbtrfsutil: add btrfs_util_deleted_subvolumes() > libbtrfsutil: add filesystem sync helpers > btrfs-progs: use libbtrfsutil for read-only property > btrfs-progs: use libbtrfsutil for sync ioctls > btrfs-progs: use libbtrfsutil for set-default > btrfs-progs: use libbtrfsutil for get-default > btrfs-progs: use libbtrfsutil for subvol create and snapshot > btrfs-progs: use libbtrfsutil for subvol delete > btrfs-progs: use libbtrfsutil for subvol show > btrfs-progs: use libbtrfsutil for subvol sync > btrfs-progs: replace test_issubvolume() with btrfs_util_is_subvolume() > btrfs-progs: add recursive snapshot/delete using libbtrfsutil > btrfs-progs: deprecate libbtrfs helpers with libbtrfsutil equivalents > btrfs-progs: use libbtrfsutil for subvolume list > > .gitignore |2 + > Documentation/btrfs-subvolume.asciidoc | 14 +- > INSTALL |4 + > Makefile | 84 +- > Makefile.inc.in |2 + > btrfs-list.c | 1201 +++--- > btrfs-list.h | 22 +- > cmds-filesystem.c| 20 +- > cmds-inspect.c |9 +- > cmds-qgroup.c| 20 +- > cmds-receive.c | 12 +- > cmds-subvolume.c | 807 +-- > configure.ac | 15 + > libbtrfsutil/COPYING | 674 + > libbtrfsutil/COPYING.LESSER | 165 +++ > libbtrfsutil/README.md | 35 + > libbtrfsutil/btrfsutil.h | 621 > libbtrfsutil/errors.c| 55 + > libbtrfsutil/filesystem.c| 103 ++ > libbtrfsutil/internal.h | 36 + > libbtrfsutil/python/.gitignore |7 + > libbtrfsutil/python/btrfsutilpy.h| 84 ++ > libbtrfsutil/python/error.c | 202 > libbtrfsutil/python/filesystem.c | 94 ++ > libbtrfsutil/python/module.c | 321 ++ > libbtrfsutil/python/qgroup.c | 141 +++ > libbtrfsutil/python/setup.py | 103 ++ > libbtrfsutil/python/subvolume.c | 665 + > libbtrfsutil/python/tests/__init__.py| 66 ++ > libbtrfsutil/pyth
Re: [PATCH 00/26] btrfs-progs: introduce libbtrfsutil, "btrfs-progs as a library"
Hi, On 01/26/2018 07:51 PM, Hugo Mills wrote: > On Fri, Jan 26, 2018 at 10:40:48AM -0800, Omar Sandoval wrote: >> From: Omar Sandoval >> >> One of the features requests I get most often is a library to do the >> sorts of operations that we do with btrfs-progs. We can shell out to >> btrfs-progs, but the output format isn't always easily parsasble, and >> shelling out isn't always ideal. There's libbtrfs, but it's very >> incomplete, doesn't have a well thought out API, and is licensed under >> the GPL, making it hard to use for many projects. >> >> libbtrfsutil is a new library written from scratch to address these >> issues. The implementation is completely independent of the existing >> btrfs-progs code, including kerncompat.h, and has a clean API and naming >> scheme. It is licensed under the LGPL. It also includes Python bindings >> by default. I will maintain the library code. Insert mandatory picture here: https://memegenerator.net/img/instances/12076366/python-all-the-things.jpg >*speechless* > >That's awesome, Omar (although with the python bindings, you've > probably just ruined Hans' day ;) ). Not quite, or should I say, quite the opposite, actually. I can think of a few different python libs related to btrfs right now... 1. This one, which works on the same level as the btrfs-progs command line utils and provides the same set of functionality, with the same input and output, only way easier to handle when automating things. The audience is the same audience as who would else try to 'shell out' to btrfs-progs and then do painful parsing of the output. 2. A lib for advanced btrfs users (more of a niche audience) who want to get a peek behind the curtains of what is going on in their online filesystem. Directly do searches in metadata, directly call an ioctl etc, either for educational purposes, or to e.g. do things like write a small special purpose dedupe program, visualize things etc. This is the python3-btrfs lib that I've been working on for some time now. 3. A lib for btrfs developers who want to script working on offline fileystems. This one does not exist yet. Well, it's a bit of an unborn baby now, my first brain dump: https://github.com/knorrie/btrfs-progs/blob/python/python3/README.md This would be the lib that exposes the (to be revised and improved) internal libbtrfs (I guess). All of these are not competitors. They serve different purpose and a different audience. So I'm happy to see number 1 also happening. :-) >> Patch 1 is a preparation cleanup which can go in independently. Patch 2 >> adds the build system stuff for the library, and patch 3 does the same >> for the Python bindings. Patches 4-14 implement the library helpers, >> currently subvolume helpers and the sync ioctls. Patches 15-26 replace >> the btrfs-progs and libbtrfs code to use libbtrfsutil instead. I took >> care to preserve backwards-compatibility. `btrfs subvol list` in >> particular had some buggy behaviors for -o and -a that I emulated in the >> new code, see the comments in the code. +1 for starting to clean up that code. Especially the whole subvolume list part looks like a great improvement. >> These patches are also available on my GitHub: >> https://github.com/osandov/btrfs-progs/tree/libbtrfsutil. That branch >> will rebase as I update this series. >> >> Please share feedback regarding the API, implementation, or anything >> else. Great work, and I'm of course also interested in what you think about the lib nr.3 idea, since I didn't throw it around on the list before yet. :-) -- Hans van Kranenburg -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
