[PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while without the need of root privilege. However the code is quite old and doesn't get much review or test. This makes some strange behavior, from customized chunk allocation (which uses the reserved 0~1M device space) to lack of special file handler (Fixed in previous 2 patches). The reworked --rootdir will be based on traditional mkfs, everything is processed after traditional mkfs, so nothing is customized. The result will be an equivalent of mkfs, mount, cp, umount. (If btrfs-progs chunk/extent allocator acts as the same as kernel) And, add extra explanation for --rootdir, since the old implement introduced a confusing behavior to limit the filesystem size. The 1st patch fixes a bug that causes any write after cleanup_temp_chunks() to trigger a NULL pointer dereference. The 2nd patch changes the work flow of --rootdir, and slightly changed the behavior, which will not shrink filesystem size by its own. The 3rd patch will shrink the fs, to keep the behavior the same as old --rootdir. If anyone doesn't like the behavior, it can be removed easily. (Which is the main part of the new code, I'd like to get rid of this) The 4th patch enhance the verbose output to reflect the real chunk allocation of --rootdir. The final patch add extra documentary explanation. Qu Wenruo (5): btrfs-progs: Fix one-byte overlap bug in free_block_group_cache btrfs-progs: mkfs: Rework rootdir option to avoid custom chunk layout btrfs-progs: mkfs: Shrink the image for rootdir to minimal size btrfs-progs: mkfs: Update allocation info before verbose output btrfs-progs: Doc/mkfs: Add explanation for rootdir parameter Documentation/mkfs.btrfs.asciidoc | 3 + extent-tree.c | 2 +- mkfs/main.c | 342 ++ 3 files changed, 164 insertions(+), 183 deletions(-) -- 2.14.1 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: > mkfs.btrfs --rootdir provides user a method to generate btrfs with > pre-written content while without the need of root privilege. > > However the code is quite old and doesn't get much review or test. > This makes some strange behavior, from customized chunk allocation > (which uses the reserved 0~1M device space) to lack of special file > handler (Fixed in previous 2 patches). The cleanup in this area is most welcome. The patches look good after a quick look, I'll do another review round. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 2017年09月05日 02:08, David Sterba wrote: On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while without the need of root privilege. However the code is quite old and doesn't get much review or test. This makes some strange behavior, from customized chunk allocation (which uses the reserved 0~1M device space) to lack of special file handler (Fixed in previous 2 patches). The cleanup in this area is most welcome. The patches look good after a quick look, I'll do another review round. To save you some time, I found that my rework can't create new image which old --rootdir can do. So it's still not completely the same behavior. I can fix it by creating a large sparse file first and then truncate it using current method easily. But this really concerns me, do we need to shrink the fs? I had a discussion with Austin about this, thread named "[btrfs-progs] Bug in mkfs.btrfs -r". The only equivalent I found is "mkfs.ext4 -d", which can only create new file if size is given and will not shrink fs. (Genext2fs shrinks the fs, but is no longer in e2fsprogs) If we follow that behavior, the 3rd and 5th patches are not needed, which I'm pretty happy with. Functionally, both behavior can be implemented with current method, but I hope to make sure which is the designed behavior so I can stick to it. I hope you could make the final decision on this so I can update the patchset. Thanks, Qu -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 09/05/2017 10:19 AM, Qu Wenruo wrote: > > > On 2017年09月05日 02:08, David Sterba wrote: >> On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: >>> mkfs.btrfs --rootdir provides user a method to generate btrfs with >>> pre-written content while without the need of root privilege. >>> >>> However the code is quite old and doesn't get much review or test. >>> This makes some strange behavior, from customized chunk allocation >>> (which uses the reserved 0~1M device space) to lack of special file >>> handler (Fixed in previous 2 patches). >> >> The cleanup in this area is most welcome. The patches look good after a >> quick look, I'll do another review round. > > To save you some time, I found that my rework can't create new image which > old --rootdir can do. So it's still not completely the same behavior. > I can fix it by creating a large sparse file first and then truncate it using > current method easily. > > But this really concerns me, do we need to shrink the fs? I still fatigue to understand in what "mkfs.btrfs --rootdir" would be better than a "simple tar"; in the first case I have to do a1) mkfs.btrfs --root-dir (create the archive) a2) dd (copy and truncate the image and store it in the archive) a3) dd (take the archived image, and restore it) a4) btrfs fi resize (expand the image) in the second case I have to b1) tar cf ... (create the image an store it in the archive, this is a1+a2) b2) mkfs,btrfs (create the filesystem with the final size) b3) tar xf ... (take the archived image and restore it) However the code is already written (and it seems simple enough), so a possible compromise could be to have the "shrinking" only if another option is passed; eg. mkfs.btrfs --root ...--> populate the filesystem mkfs.btrfs --shrink --root --> populate and shrink the filesystem however I find this useful only if it is possible to creating the filesystem in a file; ie. mkfs.btrfs --shrink --root where doesn't have to exists before mkfs.btrfs, and after a) contains the image b) is the smallest possible size. Definitely I don't like the truncate done by the operator by hand after the mkfs.btrfs (current behavior). BTW I compiled successfully the patches, and these seems to work. PS: I tried to cross-compile mkfs.btrfs ton arm, but mkfs.btrfs was unable to work: $ uname -a Linux bananapi 4.4.66-bananian #2 SMP Sat May 6 19:26:50 UTC 2017 armv7l GNU/Linux $ sudo ./mkfs.btrfs /dev/loop0 btrfs-progs v4.12.1-5-g3c9451cd See http://btrfs.wiki.kernel.org for more information. ERROR: superblock magic doesn't match Performing full device TRIM /dev/loop0 (10.00GiB) ... ERROR: open ctree failed However this problem exists even with a plain v4.12.1. The first error seems to suggest that there is some endian-ness issue BR G.Baroncelli > > I had a discussion with Austin about this, thread named "[btrfs-progs] Bug in > mkfs.btrfs -r". > The only equivalent I found is "mkfs.ext4 -d", which can only create new file > if size is given and will not shrink fs. > (Genext2fs shrinks the fs, but is no longer in e2fsprogs) > > If we follow that behavior, the 3rd and 5th patches are not needed, which I'm > pretty happy with. > > Functionally, both behavior can be implemented with current method, but I > hope to make sure which is the designed behavior so I can stick to it. > > I hope you could make the final decision on this so I can update the patchset. > > Thanks, > Qu > >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- gpg @keyserver.linux.it: Goffredo Baroncelli Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 2017年09月06日 03:05, Goffredo Baroncelli wrote: On 09/05/2017 10:19 AM, Qu Wenruo wrote: On 2017年09月05日 02:08, David Sterba wrote: On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while without the need of root privilege. However the code is quite old and doesn't get much review or test. This makes some strange behavior, from customized chunk allocation (which uses the reserved 0~1M device space) to lack of special file handler (Fixed in previous 2 patches). The cleanup in this area is most welcome. The patches look good after a quick look, I'll do another review round. To save you some time, I found that my rework can't create new image which old --rootdir can do. So it's still not completely the same behavior. I can fix it by creating a large sparse file first and then truncate it using current method easily. But this really concerns me, do we need to shrink the fs? I still fatigue to understand in what "mkfs.btrfs --rootdir" would be better than a "simple tar"; in the first case I have to do a1) mkfs.btrfs --root-dir (create the archive) a2) dd (copy and truncate the image and store it in the archive) a3) dd (take the archived image, and restore it) a4) btrfs fi resize (expand the image) in the second case I have to b1) tar cf ... (create the image an store it in the archive, this is a1+a2) b2) mkfs,btrfs (create the filesystem with the final size) b3) tar xf ... (take the archived image and restore it) However the code is already written (and it seems simple enough), so a possible compromise could be to have the "shrinking" only if another option is passed; eg. mkfs.btrfs --root ...--> populate the filesystem mkfs.btrfs --shrink --root --> populate and shrink the filesystem however I find this useful only if it is possible to creating the filesystem in a file; ie. mkfs.btrfs --shrink --root where doesn't have to exists before mkfs.btrfs, and after a) contains the image b) is the smallest possible size. Yes, that's the original behavior. And what my rework can't do yet. It can't determine the size of the device, so it can't continue. If we decide to follow the original behavior, then I have to create sparse file first and truncate the file at the end. But still quite easy to do. And if we decide to follow mkfs.ext -d behavior, then I just need to remove 2 patches from the patchset (shrink patch and doc patch, which adds about 100 lines), and slightly modify the rework patch to remove the O_CREATE open flag. Definitely I don't like the truncate done by the operator by hand after the mkfs.btrfs (current behavior). BTW I compiled successfully the patches, and these seems to work. PS: I tried to cross-compile mkfs.btrfs ton arm, but mkfs.btrfs was unable to work: $ uname -a Linux bananapi 4.4.66-bananian #2 SMP Sat May 6 19:26:50 UTC 2017 armv7l GNU/Linux $ sudo ./mkfs.btrfs /dev/loop0 btrfs-progs v4.12.1-5-g3c9451cd See http://btrfs.wiki.kernel.org for more information. ERROR: superblock magic doesn't match Performing full device TRIM /dev/loop0 (10.00GiB) ... ERROR: open ctree failed However this problem exists even with a plain v4.12.1. The first error seems to suggest that there is some endian-ness issue I'd better get one cheap ARM board if I want to do native debug. BTW, what's the output of dump-super here? Which may gives us some clue to fix it. Thanks, Qu BR G.Baroncelli I had a discussion with Austin about this, thread named "[btrfs-progs] Bug in mkfs.btrfs -r". The only equivalent I found is "mkfs.ext4 -d", which can only create new file if size is given and will not shrink fs. (Genext2fs shrinks the fs, but is no longer in e2fsprogs) If we follow that behavior, the 3rd and 5th patches are not needed, which I'm pretty happy with. Functionally, both behavior can be implemented with current method, but I hope to make sure which is the designed behavior so I can stick to it. I hope you could make the final decision on this so I can update the patchset. Thanks, Qu -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 2017-09-05 15:05, Goffredo Baroncelli wrote: On 09/05/2017 10:19 AM, Qu Wenruo wrote: On 2017年09月05日 02:08, David Sterba wrote: On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while without the need of root privilege. However the code is quite old and doesn't get much review or test. This makes some strange behavior, from customized chunk allocation (which uses the reserved 0~1M device space) to lack of special file handler (Fixed in previous 2 patches). The cleanup in this area is most welcome. The patches look good after a quick look, I'll do another review round. To save you some time, I found that my rework can't create new image which old --rootdir can do. So it's still not completely the same behavior. I can fix it by creating a large sparse file first and then truncate it using current method easily. But this really concerns me, do we need to shrink the fs? I still fatigue to understand in what "mkfs.btrfs --rootdir" would be better than a "simple tar"; in the first case I have to do a1) mkfs.btrfs --root-dir (create the archive) a2) dd (copy and truncate the image and store it in the archive) a3) dd (take the archived image, and restore it) a4) btrfs fi resize (expand the image) The primary use case for this is to generate installation images. Using this method removes the need for tar in the installation environment, and if you defer step a4 until the first boot of the system, it also removes the need to have btrfs-progs in the installation environment. Taken together, that's a pretty significant space savings. It's also somewhat useful for creating minimalistic seed device images, which have a couple of interesting uses, namely: * Base system images for 'factory reset'. The general principal is simple, your base system is a seed device, plus a storage device associated with it. When you want to do a factory reset, you wipe the storage partition, and recreate an empty one associated with the seed image. This usage pretty much requires a minimally sized filesystem, as anything more wastes space that would be otherwise usable by the end user. * Seed-device based install images. The general concept for this has been tossed around a couple of times before. You start with a minimal seed device, boot to a live system using that and a temporary in-memory device for root, set up the persistent storage, re-balance everything to persistent storage, then remove the seed device and in-memory device so the user can keep using the system without needing to reboot. This also needs a minimalistic image, for the same reason any install disc needs to have a minimal base image. Note that without resize being able to shrink chunks (and ideally completely shrink them so there is no slack space in the FS), you have to use a hex editor to get a truly minimalistic filesystem image. in the second case I have to b1) tar cf ... (create the image an store it in the archive, this is a1+a2) b2) mkfs,btrfs (create the filesystem with the final size) b3) tar xf ... (take the archived image and restore it) However the code is already written (and it seems simple enough), so a possible compromise could be to have the "shrinking" only if another option is passed; eg. mkfs.btrfs --root ...--> populate the filesystem mkfs.btrfs --shrink --root --> populate and shrink the filesystem however I find this useful only if it is possible to creating the filesystem in a file; ie. mkfs.btrfs --shrink --root where doesn't have to exists before mkfs.btrfs, and after a) contains the image b) is the smallest possible size. Definitely I don't like the truncate done by the operator by hand after the mkfs.btrfs (current behavior). FWIW, the current release behavior doesn't require the truncate, and properly generates the file for the filesystem. It also does some odd things with chunk placement (including putting data in the 0-1M range which is supposed to be reserved), and that odd behavior is primarily what prompted this patch set. BTW I compiled successfully the patches, and these seems to work. PS: I tried to cross-compile mkfs.btrfs ton arm, but mkfs.btrfs was unable to work: $ uname -a Linux bananapi 4.4.66-bananian #2 SMP Sat May 6 19:26:50 UTC 2017 armv7l GNU/Linux $ sudo ./mkfs.btrfs /dev/loop0 btrfs-progs v4.12.1-5-g3c9451cd See http://btrfs.wiki.kernel.org for more information. ERROR: superblock magic doesn't match Performing full device TRIM /dev/loop0 (10.00GiB) ... ERROR: open ctree failed However this problem exists even with a plain v4.12.1. The first error seems to suggest that there is some endian-ness issue BR G.Baroncelli I had a discussion with Austin about this, thread named "[btrfs-progs] Bug in mkfs.btrfs -r". The only equivalent I found is "mkfs.ext4 -d", which can only create new file if size is giv
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 09/06/2017 01:31 PM, Austin S. Hemmelgarn wrote: > On 2017-09-05 15:05, Goffredo Baroncelli wrote: >> On 09/05/2017 10:19 AM, Qu Wenruo wrote: >>> >>> >>> On 2017年09月05日 02:08, David Sterba wrote: On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: > mkfs.btrfs --rootdir provides user a method to generate btrfs with > pre-written content while without the need of root privilege. > > However the code is quite old and doesn't get much review or test. > This makes some strange behavior, from customized chunk allocation > (which uses the reserved 0~1M device space) to lack of special file > handler (Fixed in previous 2 patches). The cleanup in this area is most welcome. The patches look good after a quick look, I'll do another review round. >>> >>> To save you some time, I found that my rework can't create new image which >>> old --rootdir can do. So it's still not completely the same behavior. >>> I can fix it by creating a large sparse file first and then truncate it >>> using current method easily. >>> >>> But this really concerns me, do we need to shrink the fs? >> >> I still fatigue to understand in what "mkfs.btrfs --rootdir" would be better >> than a "simple tar"; >> >> in the first case I have to do >> a1) mkfs.btrfs --root-dir (create the archive) >> a2) dd (copy and truncate the image and store it in the archive) >> a3) dd (take the archived image, and restore it) >> a4) btrfs fi resize (expand the image) > The primary use case for this is to generate installation images. Using this > method removes the need for tar in the installation environment, and if you > defer step a4 until the first boot of the system, it also removes the need to > have btrfs-progs in the installation environment. Taken together, that's a > pretty significant space savings. Sorry but I don't understand. If you reach the step a3; you have: - the final disk, and an environment fully working. So I am still inclined to think that using "mkfs.btrfs --root-dir" is more complicated in *this case*. > > It's also somewhat useful for creating minimalistic seed device images, which > have a couple of interesting uses, namely: > > * Base system images for 'factory reset'. The general principal is simple, > your base system is a seed device, plus a storage device associated with it. > When you want to do a factory reset, you wipe the storage partition, and > recreate an empty one associated with the seed image. This usage pretty much > requires a minimally sized filesystem, as anything more wastes space that > would be otherwise usable by the end user. > > * Seed-device based install images. The general concept for this has been > tossed around a couple of times before. You start with a minimal seed > device, boot to a live system using that and a temporary in-memory device for > root, set up the persistent storage, re-balance everything to persistent > storage, then remove the seed device and in-memory device so the user can > keep using the system without needing to reboot. This also needs a > minimalistic image, for the same reason any install disc needs to have a > minimal base image. For the above cases I agree that this could be useful to have "--rootdir" > > Note that without resize being able to shrink chunks (and ideally completely > shrink them so there is no slack space in the FS), you have to use a hex > editor to get a truly minimalistic filesystem image. >> >> in the second case I have to >> b1) tar cf ... (create the image an store it in the archive, this is a1+a2) >> b2) mkfs,btrfs (create the filesystem with the final size) >> b3) tar xf ... (take the archived image and restore it) >> >> >> However the code is already written (and it seems simple enough), so a >> possible compromise could be to have the "shrinking" only if another option >> is passed; eg. >> >> mkfs.btrfs --root ... --> populate the filesystem >> mkfs.btrfs --shrink --root --> populate and shrink the filesystem >> >> however I find this useful only if it is possible to creating the filesystem >> in a file; ie. >> >> mkfs.btrfs --shrink --root >> >> where doesn't have to exists before mkfs.btrfs, and >> after >> a) contains the image >> b) is the smallest possible size. >> >> Definitely I don't like the truncate done by the operator by hand after the >> mkfs.btrfs (current behavior). > FWIW, the current release behavior doesn't require the truncate, and properly > generates the file for the filesystem. If you don't do truncate, you have the fully partition... Or there is something that I miss ? [...] -- gpg @keyserver.linux.it: Goffredo Baroncelli Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 2017-09-06 12:43, Goffredo Baroncelli wrote: On 09/06/2017 01:31 PM, Austin S. Hemmelgarn wrote: On 2017-09-05 15:05, Goffredo Baroncelli wrote: On 09/05/2017 10:19 AM, Qu Wenruo wrote: On 2017年09月05日 02:08, David Sterba wrote: On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while without the need of root privilege. However the code is quite old and doesn't get much review or test. This makes some strange behavior, from customized chunk allocation (which uses the reserved 0~1M device space) to lack of special file handler (Fixed in previous 2 patches). The cleanup in this area is most welcome. The patches look good after a quick look, I'll do another review round. To save you some time, I found that my rework can't create new image which old --rootdir can do. So it's still not completely the same behavior. I can fix it by creating a large sparse file first and then truncate it using current method easily. But this really concerns me, do we need to shrink the fs? I still fatigue to understand in what "mkfs.btrfs --rootdir" would be better than a "simple tar"; in the first case I have to do a1) mkfs.btrfs --root-dir (create the archive) a2) dd (copy and truncate the image and store it in the archive) a3) dd (take the archived image, and restore it) a4) btrfs fi resize (expand the image) The primary use case for this is to generate installation images. Using this method removes the need for tar in the installation environment, and if you defer step a4 until the first boot of the system, it also removes the need to have btrfs-progs in the installation environment. Taken together, that's a pretty significant space savings. Sorry but I don't understand. If you reach the step a3; you have: - the final disk, and an environment fully working. So I am still inclined to think that using "mkfs.btrfs --root-dir" is more complicated in *this case*. With the current released code (without these patches), `-r` can be used to generate a filesystem image that has zero free space. In that case, step a3 does not give you a fully working environment, it gives you a basic environment that can probably be booted to a minimal degree and run just enough to be able to resize the filesystem to take up the whole disk. IOW, the use case I'm referring to here is more akin to that of genext2fs -d than mkfs.ext4 -d, although mkfs.btrfs -r generates a minimally sized FS right now, while neither of the other two do so by default (although I could have sworn that genext2fs could). It's also somewhat useful for creating minimalistic seed device images, which have a couple of interesting uses, namely: * Base system images for 'factory reset'. The general principal is simple, your base system is a seed device, plus a storage device associated with it. When you want to do a factory reset, you wipe the storage partition, and recreate an empty one associated with the seed image. This usage pretty much requires a minimally sized filesystem, as anything more wastes space that would be otherwise usable by the end user. * Seed-device based install images. The general concept for this has been tossed around a couple of times before. You start with a minimal seed device, boot to a live system using that and a temporary in-memory device for root, set up the persistent storage, re-balance everything to persistent storage, then remove the seed device and in-memory device so the user can keep using the system without needing to reboot. This also needs a minimalistic image, for the same reason any install disc needs to have a minimal base image. For the above cases I agree that this could be useful to have "--rootdir" Note that without resize being able to shrink chunks (and ideally completely shrink them so there is no slack space in the FS), you have to use a hex editor to get a truly minimalistic filesystem image. in the second case I have to b1) tar cf ... (create the image an store it in the archive, this is a1+a2) b2) mkfs,btrfs (create the filesystem with the final size) b3) tar xf ... (take the archived image and restore it) However the code is already written (and it seems simple enough), so a possible compromise could be to have the "shrinking" only if another option is passed; eg. mkfs.btrfs --root ... --> populate the filesystem mkfs.btrfs --shrink --root --> populate and shrink the filesystem however I find this useful only if it is possible to creating the filesystem in a file; ie. mkfs.btrfs --shrink --root where doesn't have to exists before mkfs.btrfs, and after a) contains the image b) is the smallest possible size. Definitely I don't like the truncate done by the operator by hand after the mkfs.btrfs (current behavior). FWIW, the current release behavior doesn't require the truncate, and properly generates the file f
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 09/06/2017 07:16 PM, Austin S. Hemmelgarn wrote: > On 2017-09-06 12:43, Goffredo Baroncelli wrote: >> On 09/06/2017 01:31 PM, Austin S. Hemmelgarn wrote: >>> On 2017-09-05 15:05, Goffredo Baroncelli wrote: On 09/05/2017 10:19 AM, Qu Wenruo wrote: > > > On 2017年09月05日 02:08, David Sterba wrote: >> On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: >>> mkfs.btrfs --rootdir provides user a method to generate btrfs with >>> pre-written content while without the need of root privilege. >>> >>> However the code is quite old and doesn't get much review or test. >>> This makes some strange behavior, from customized chunk allocation >>> (which uses the reserved 0~1M device space) to lack of special file >>> handler (Fixed in previous 2 patches). >> >> The cleanup in this area is most welcome. The patches look good after a >> quick look, I'll do another review round. > > To save you some time, I found that my rework can't create new image > which old --rootdir can do. So it's still not completely the same > behavior. > I can fix it by creating a large sparse file first and then truncate it > using current method easily. > > But this really concerns me, do we need to shrink the fs? I still fatigue to understand in what "mkfs.btrfs --rootdir" would be better than a "simple tar"; in the first case I have to do a1) mkfs.btrfs --root-dir (create the archive) a2) dd (copy and truncate the image and store it in the archive) a3) dd (take the archived image, and restore it) a4) btrfs fi resize (expand the image) >>> The primary use case for this is to generate installation images. Using >>> this method removes the need for tar in the installation environment, and >>> if you defer step a4 until the first boot of the system, it also removes >>> the need to have btrfs-progs in the installation environment. Taken >>> together, that's a pretty significant space savings. >> >> Sorry but I don't understand. If you reach the step a3; you have: >> - the final disk, and an environment fully working. So I am still inclined >> to think that using "mkfs.btrfs --root-dir" is more complicated in *this >> case*. > With the current released code (without these patches), `-r` can be used to > generate a filesystem image that has zero free space. In that case, step a3 > does not give you a fully working environment, True, this doesn't *give* you a filly working environment, _but_ you perform the step a3 in a "fully working environment", an you have at hand the target disk.. What I am saying is that both step a1..a3 and b1..b3 are done in a "fully working environment". The only difference is that you will need a further step a4 > it gives you a basic environment that can probably be booted to a minimal > degree and run just enough to be able to resize the filesystem to take up the > whole disk. > > IOW, the use case I'm referring to here is more akin to that of genext2fs -d > than mkfs.ext4 -d, although mkfs.btrfs -r generates a minimally sized FS > right now, while neither of the other two do so by default (although I could > have sworn that genext2fs could). >>> >>> It's also somewhat useful for creating minimalistic seed device images, >>> which have a couple of interesting uses, namely: >>> >>> * Base system images for 'factory reset'. The general principal is simple, >>> your base system is a seed device, plus a storage device associated with >>> it. When you want to do a factory reset, you wipe the storage partition, >>> and recreate an empty one associated with the seed image. This usage >>> pretty much requires a minimally sized filesystem, as anything more wastes >>> space that would be otherwise usable by the end user. >>> >>> * Seed-device based install images. The general concept for this has been >>> tossed around a couple of times before. You start with a minimal seed >>> device, boot to a live system using that and a temporary in-memory device >>> for root, set up the persistent storage, re-balance everything to >>> persistent storage, then remove the seed device and in-memory device so the >>> user can keep using the system without needing to reboot. This also needs >>> a minimalistic image, for the same reason any install disc needs to have a >>> minimal base image. >> >> For the above cases I agree that this could be useful to have "--rootdir" >>> >>> Note that without resize being able to shrink chunks (and ideally >>> completely shrink them so there is no slack space in the FS), you have to >>> use a hex editor to get a truly minimalistic filesystem image. in the second case I have to b1) tar cf ... (create the image an store it in the archive, this is a1+a2) b2) mkfs,btrfs (create the filesystem with the final size) b3) tar xf ... (take the archived image and restore it) However t
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 2017-09-06 13:48, Goffredo Baroncelli wrote: On 09/06/2017 07:16 PM, Austin S. Hemmelgarn wrote: On 2017-09-06 12:43, Goffredo Baroncelli wrote: On 09/06/2017 01:31 PM, Austin S. Hemmelgarn wrote: On 2017-09-05 15:05, Goffredo Baroncelli wrote: On 09/05/2017 10:19 AM, Qu Wenruo wrote: On 2017年09月05日 02:08, David Sterba wrote: On Mon, Sep 04, 2017 at 03:41:05PM +0900, Qu Wenruo wrote: mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while without the need of root privilege. However the code is quite old and doesn't get much review or test. This makes some strange behavior, from customized chunk allocation (which uses the reserved 0~1M device space) to lack of special file handler (Fixed in previous 2 patches). The cleanup in this area is most welcome. The patches look good after a quick look, I'll do another review round. To save you some time, I found that my rework can't create new image which old --rootdir can do. So it's still not completely the same behavior. I can fix it by creating a large sparse file first and then truncate it using current method easily. But this really concerns me, do we need to shrink the fs? I still fatigue to understand in what "mkfs.btrfs --rootdir" would be better than a "simple tar"; in the first case I have to do a1) mkfs.btrfs --root-dir (create the archive) a2) dd (copy and truncate the image and store it in the archive) a3) dd (take the archived image, and restore it) a4) btrfs fi resize (expand the image) The primary use case for this is to generate installation images. Using this method removes the need for tar in the installation environment, and if you defer step a4 until the first boot of the system, it also removes the need to have btrfs-progs in the installation environment. Taken together, that's a pretty significant space savings. Sorry but I don't understand. If you reach the step a3; you have: - the final disk, and an environment fully working. So I am still inclined to think that using "mkfs.btrfs --root-dir" is more complicated in *this case*. With the current released code (without these patches), `-r` can be used to generate a filesystem image that has zero free space. In that case, step a3 does not give you a fully working environment, True, this doesn't *give* you a filly working environment, _but_ you perform the step a3 in a "fully working environment", an you have at hand the target disk.. You could just as easily be booted into a minimalistic install environment, and if you netbooted that, then it's pretty likely that you want it as small as possible, and not needing tar or btrfs-progs for the actual install will save a lot of space (multiple MB doesn't sound like much, but when you're dealing with a tiny system to begin with, it can be very significant). What I am saying is that both step a1..a3 and b1..b3 are done in a "fully working environment". The only difference is that you will need a further step a4 it gives you a basic environment that can probably be booted to a minimal degree and run just enough to be able to resize the filesystem to take up the whole disk. IOW, the use case I'm referring to here is more akin to that of genext2fs -d than mkfs.ext4 -d, although mkfs.btrfs -r generates a minimally sized FS right now, while neither of the other two do so by default (although I could have sworn that genext2fs could). It's also somewhat useful for creating minimalistic seed device images, which have a couple of interesting uses, namely: * Base system images for 'factory reset'. The general principal is simple, your base system is a seed device, plus a storage device associated with it. When you want to do a factory reset, you wipe the storage partition, and recreate an empty one associated with the seed image. This usage pretty much requires a minimally sized filesystem, as anything more wastes space that would be otherwise usable by the end user. * Seed-device based install images. The general concept for this has been tossed around a couple of times before. You start with a minimal seed device, boot to a live system using that and a temporary in-memory device for root, set up the persistent storage, re-balance everything to persistent storage, then remove the seed device and in-memory device so the user can keep using the system without needing to reboot. This also needs a minimalistic image, for the same reason any install disc needs to have a minimal base image. For the above cases I agree that this could be useful to have "--rootdir" Note that without resize being able to shrink chunks (and ideally completely shrink them so there is no slack space in the FS), you have to use a hex editor to get a truly minimalistic filesystem image. in the second case I have to b1) tar cf ... (create the image an store it in the archive, this is a1+a2) b2) mkfs,btrfs (create the filesystem with the final size) b3) tar xf
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 09/06/2017 08:02 PM, Austin S. Hemmelgarn wrote: > On 2017-09-06 13:48, Goffredo Baroncelli wrote: >> On 09/06/2017 07:16 PM, Austin S. Hemmelgarn wrote: [...] Sorry but I don't understand. If you reach the step a3; you have: - the final disk, and an environment fully working. So I am still inclined to think that using "mkfs.btrfs --root-dir" is more complicated in *this case*. >>> With the current released code (without these patches), `-r` can be used to >>> generate a filesystem image that has zero free space. In that case, step >>> a3 does not give you a fully working environment, >> >> True, this doesn't *give* you a filly working environment, _but_ you perform >> the step a3 in a "fully working environment", an you have at hand the target >> disk.. > You could just as easily be booted into a minimalistic install environment, > and if you netbooted that, then it's pretty likely that you want it as small > as possible, and not needing tar or btrfs-progs for the actual install will > save a lot of space (multiple MB doesn't sound like much, but when you're > dealing with a tiny system to begin with, it can be very significant). Step a3 need to have access to the raw disk image build at step1; this is quite incompatible with a "minimalistic install environment"; and even if you have access it via net, in the same way you can have access to a fully working environment [...] >> >> where doesn't have to exists before mkfs.btrfs, and >> after >> a) contains the image >> b) is the smallest possible size. >> >> Definitely I don't like the truncate done by the operator by hand after >> the mkfs.btrfs (current behavior). > FWIW, the current release behavior doesn't require the truncate, and > properly generates the file for the filesystem. If you don't do truncate, you have the fully partition... Or there is something that I miss ? >>> The current release, without these patches, run using a non-existent file >>> and the `-r` option, will produce a filesystem image of the exact size >>> needed to hold everything in the directory passed to `-r`. It doesn't >>> require truncation unless used on a file that already exists. >> >> Of course the truncate is not needed, because you are using a sparse file. >> But if you use a sparse file. it is not even needed the shrinking! >> Because the file will consume the same space on the disk ! > Unless you want to use the file elsewhere. It's a pretty rare occurrence > outside of testing that you generate a filesystem image and use it as-is > without transferring it somewhere (usually onto an actual storage device). > Once you're talking about moving it, whether or not the file itself is sparse > usually doesn't matter, especially if the file is leaving the local system by > some means other than NFS or rsync. If you don't truncate you have the full-image in any case > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- gpg @keyserver.linux.it: Goffredo Baroncelli Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 09/06/2017 05:20 AM, Qu Wenruo wrote: >> >> BTW I compiled successfully the patches, and these seems to work. >> >> PS: I tried to cross-compile mkfs.btrfs ton arm, but mkfs.btrfs was unable >> to work: >> >> $ uname -a >> Linux bananapi 4.4.66-bananian #2 SMP Sat May 6 19:26:50 UTC 2017 armv7l >> GNU/Linux >> $ sudo ./mkfs.btrfs /dev/loop0 >> btrfs-progs v4.12.1-5-g3c9451cd >> See http://btrfs.wiki.kernel.org for more information. >> >> ERROR: superblock magic doesn't match >> Performing full device TRIM /dev/loop0 (10.00GiB) ... >> ERROR: open ctree failed >> >> However this problem exists even with a plain v4.12.1. The first error seems >> to suggest that there is some endian-ness issue > > I'd better get one cheap ARM board if I want to do native debug. > > BTW, what's the output of dump-super here? Further tests seems to highlight that it was a my setup problem. Before I build mkfs.btrfs cross-compiling from an amd64, and I got the error; retrying on the native environment I was unable to reproduce the problem. So please ignore the previous email. > Which may gives us some clue to fix it. > > Thanks, > Qu -- gpg @keyserver.linux.it: Goffredo Baroncelli Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior
On 2017-09-06 14:31, Goffredo Baroncelli wrote: On 09/06/2017 08:02 PM, Austin S. Hemmelgarn wrote: On 2017-09-06 13:48, Goffredo Baroncelli wrote: On 09/06/2017 07:16 PM, Austin S. Hemmelgarn wrote: [...] Sorry but I don't understand. If you reach the step a3; you have: - the final disk, and an environment fully working. So I am still inclined to think that using "mkfs.btrfs --root-dir" is more complicated in *this case*. With the current released code (without these patches), `-r` can be used to generate a filesystem image that has zero free space. In that case, step a3 does not give you a fully working environment, True, this doesn't *give* you a filly working environment, _but_ you perform the step a3 in a "fully working environment", an you have at hand the target disk.. You could just as easily be booted into a minimalistic install environment, and if you netbooted that, then it's pretty likely that you want it as small as possible, and not needing tar or btrfs-progs for the actual install will save a lot of space (multiple MB doesn't sound like much, but when you're dealing with a tiny system to begin with, it can be very significant). Step a3 need to have access to the raw disk image build at step1; this is quite incompatible with a "minimalistic install environment"; and even if you have access it via net, in the same way you can have access to a fully working environment The pretty standard case for a netboot install environment for Linux is to have a small netboot image (usually less than 50MB) that runs entirely from RAM, and then downloads the system image or packages to install. [...] where doesn't have to exists before mkfs.btrfs, and after a) contains the image b) is the smallest possible size. Definitely I don't like the truncate done by the operator by hand after the mkfs.btrfs (current behavior). FWIW, the current release behavior doesn't require the truncate, and properly generates the file for the filesystem. If you don't do truncate, you have the fully partition... Or there is something that I miss ? The current release, without these patches, run using a non-existent file and the `-r` option, will produce a filesystem image of the exact size needed to hold everything in the directory passed to `-r`. It doesn't require truncation unless used on a file that already exists. Of course the truncate is not needed, because you are using a sparse file. But if you use a sparse file. it is not even needed the shrinking! Because the file will consume the same space on the disk ! Unless you want to use the file elsewhere. It's a pretty rare occurrence outside of testing that you generate a filesystem image and use it as-is without transferring it somewhere (usually onto an actual storage device). Once you're talking about moving it, whether or not the file itself is sparse usually doesn't matter, especially if the file is leaving the local system by some means other than NFS or rsync. If you don't truncate you have the full-image in any case How about a more concrete example. Say you're creating a filesystem image on your local system for a VM you'll run on some server. Unless you use rsync, NFS, or some custom software, you will probably need enough free space on the server to hold the entirety of the filesystem image including all the free space that's sparse on the local system, because almost any other software doesn't properly do sparse writes, and you'll need to call `fallocate -d` on the file on the server to make it sparse again. Even working locally, there are quite a few tools that won't copy the file sparsely. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
