Re: moving disks to new case
On Thu, Apr 15, 2021 at 02:36:37PM -0500, Charles Zeitler wrote: > disks are raid5, i don't know which are /dev/sdb /dev/sdc etc. > is this going to be an issue? Nope. Hugo. Oh, all right, I'll explain... The superblock on each device contains the UUID of the filesystem and a device ID (you can see both of these in the output of btrfs fi show). When btrfs dev scan is run (for example, by udev as it enumerates the devices), it attempts to read the superblock on each device. Any superblocks that are found are read, and the UUID and devid of that device node are passed to the kernel. The kernel holds a lookup table of information for every device known of every filesystem, and uses that table to work out which device nodes it needs to use for any given FS. Hugo. -- Hugo Mills | Guards! Help! We're being rescued! hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | The Stainless Steel Rat Forever
Re: Machine died with OOM after (re)moving lots of data and snapshot
On 2017年11月01日 18:22, Lars Noschinski wrote: > Hi everyone, > > I have a machine which had an almost full 4TB btrfs partition on a > SATA harddisk. I added another 4TB harddisk, put a btrfs partition on > it and moved 1TB of data to the new partition. > > After that, I moved another part of the data (probably around 500GB) > to the new partition and deleted all the snapshots taken during the > last year -- probably around 300. > > The machine was slow during this (which did not surprise me due to > heavy disk I/O). I left the machine working for the rest of the day > and found it totally dead the next morning (no ping, not even the VGA > console coming up after pressing a key). Did you enabled quota? It's known that quota will consume a lot of memory for backref walk. (With the heaviest overhead, compared to the remaining factors) If not, did you use a lot of reflink or offline dedupe? Which both will cause a lot of stress for backref walk, and may consume all your memory. If still not, then the number of snapshots is the cause. By design, btrfs is fast doing snapshot, with all the overhead moved to backref walk. It's OK if you only do normal read/write or creating snapshots, but when you do the following things, the problem will begin to show: 1) Snapshot remove 2) Balance 3) Qgroup And, unless we have backgroup thread to solve shared (using parent) backref to unshared backref, such problem can't really be solved. (Or even more, change how btrfs create snapshot at all). So in conclusion, btrfs snapshot is fast, but don't abuse it. Keeping snapshot and subvolumes to a reasonable number (<20 seems good enough) is a good idea. Thanks, Qu > > > After a restart, I found the following messages in the kernel log. For > a few minutes, there where a few messages like: > > Oct 31 15:48:44 wuerfelzucker kernel: [107426.623818] INFO: task > rm:4456 blocked for more than 120 seconds. > Oct 31 15:48:44 wuerfelzucker kernel: [107426.623952] Not > tainted 4.9.0-4-amd64 #1 Debian 4.9.51-1 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624058] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624200] rm > D0 4456 4453 0x > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624209] > 94060f2ee800 94060e8ed080 94061fc98240 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624216] > 9406102d3140 b807818bfd90 82c038e3 00020b0f4000 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624222] > 00ffc03c17ee 94061fc98240 940611e36288 94060e8ed080 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624228] Call Trace: > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624242] > [] ? __schedule+0x233/0x6d0 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624248] > [] ? schedule+0x32/0x80 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624315] > [] ? wait_current_trans.isra.20+0xc1/0x110 [btrfs] > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624323] > [] ? prepare_to_wait_event+0xf0/0xf0 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624374] > [] ? start_transaction+0x25b/0x480 [btrfs] > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624425] > [] ? btrfs_evict_inode+0x45f/0x5d0 [btrfs] > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624432] > [] ? evict+0xb6/0x180 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624439] > [] ? do_unlinkat+0x148/0x330 > Oct 31 15:48:44 wuerfelzucker kernel: [107426.624447] > [] ? system_call_fast_compare_end+0xc/0x9b > > 9h later, the OOM killer kicked in and kept killing processes for > another 30 minutes, before the logs end. The full log file is found at > > https://bugzilla.kernel.org/attachment.cgi?id=260455 > > The last few invocations of the OOM killer where by a process kalled > "btrfs-transacti". > > As far as I can tell from the logs, moving the data was finished by > then (at least, I don't see an instance of "mv" in the process list > from the OOM killer). Otherwise, the machine was idle. > > If you need any more data for investigating this problem, I might be > able to provide that; I haven't touched the affected disks yet. > > The machine is a HP Microserver N54L with 10GB of ECC ram running a > current Debian stretch: > * kernel: Linux version 4.9.0-4-amd64 (debian-ker...@lists.debian.org) > (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.51-1 > (2017-09-28) > * btrfs-tools: 4.7.3-1 > > I also reported this as #197627 in the kernel bugzilla. > > Best regards, Lars Noschinski > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > signature.asc Description: OpenPGP digital signature
Machine died with OOM after (re)moving lots of data and snapshot
Hi everyone, I have a machine which had an almost full 4TB btrfs partition on a SATA harddisk. I added another 4TB harddisk, put a btrfs partition on it and moved 1TB of data to the new partition. After that, I moved another part of the data (probably around 500GB) to the new partition and deleted all the snapshots taken during the last year -- probably around 300. The machine was slow during this (which did not surprise me due to heavy disk I/O). I left the machine working for the rest of the day and found it totally dead the next morning (no ping, not even the VGA console coming up after pressing a key). After a restart, I found the following messages in the kernel log. For a few minutes, there where a few messages like: Oct 31 15:48:44 wuerfelzucker kernel: [107426.623818] INFO: task rm:4456 blocked for more than 120 seconds. Oct 31 15:48:44 wuerfelzucker kernel: [107426.623952] Not tainted 4.9.0-4-amd64 #1 Debian 4.9.51-1 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624058] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 31 15:48:44 wuerfelzucker kernel: [107426.624200] rm D0 4456 4453 0x Oct 31 15:48:44 wuerfelzucker kernel: [107426.624209] 94060f2ee800 94060e8ed080 94061fc98240 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624216] 9406102d3140 b807818bfd90 82c038e3 00020b0f4000 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624222] 00ffc03c17ee 94061fc98240 940611e36288 94060e8ed080 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624228] Call Trace: Oct 31 15:48:44 wuerfelzucker kernel: [107426.624242] [] ? __schedule+0x233/0x6d0 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624248] [] ? schedule+0x32/0x80 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624315] [] ? wait_current_trans.isra.20+0xc1/0x110 [btrfs] Oct 31 15:48:44 wuerfelzucker kernel: [107426.624323] [] ? prepare_to_wait_event+0xf0/0xf0 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624374] [] ? start_transaction+0x25b/0x480 [btrfs] Oct 31 15:48:44 wuerfelzucker kernel: [107426.624425] [] ? btrfs_evict_inode+0x45f/0x5d0 [btrfs] Oct 31 15:48:44 wuerfelzucker kernel: [107426.624432] [] ? evict+0xb6/0x180 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624439] [] ? do_unlinkat+0x148/0x330 Oct 31 15:48:44 wuerfelzucker kernel: [107426.624447] [] ? system_call_fast_compare_end+0xc/0x9b 9h later, the OOM killer kicked in and kept killing processes for another 30 minutes, before the logs end. The full log file is found at https://bugzilla.kernel.org/attachment.cgi?id=260455 The last few invocations of the OOM killer where by a process kalled "btrfs-transacti". As far as I can tell from the logs, moving the data was finished by then (at least, I don't see an instance of "mv" in the process list from the OOM killer). Otherwise, the machine was idle. If you need any more data for investigating this problem, I might be able to provide that; I haven't touched the affected disks yet. The machine is a HP Microserver N54L with 10GB of ECC ram running a current Debian stretch: * kernel: Linux version 4.9.0-4-amd64 (debian-ker...@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.51-1 (2017-09-28) * btrfs-tools: 4.7.3-1 I also reported this as #197627 in the kernel bugzilla. Best regards, Lars Noschinski -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving btrfs fs disk between computers
Adam Borowski posted on Sat, 19 Aug 2017 04:35:26 +0200 as excerpted: > On Fri, Aug 18, 2017 at 11:09:22PM -0300, Hérikz Nawarro wrote: >> Hello everyone, >> >> Can i create a btrfs fs in a machine and move the disk to another >> machine like a ext4 or ntfs? > > Yeah, no problem whatsoever, even for multi-device filesystems. > Btrfs doesn't care about what devices it is on. > > There has been no new incompat features in quite a while, so old kernels > shouldn't be an issue either. > > I think the only way for valid btrfs to be unmountable on another > machine with a modern kernel is to mess with page size, which is AFAIK > not even possible on x86. IOW, as long as the two machines are the same arch, or more precisely, have the same page size if different archs, it should be fine as long as you're not running a really old kernel on one of them. But don't try mounting a filesystem created on x86_64 on something exotic like sparc or ppc or something. If the two archs' page sizes are the same it should work, but I'd still be sure I had an extra set of backups before I tried something like that. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving btrfs fs disk between computers
On Fri, Aug 18, 2017 at 11:09:22PM -0300, Hérikz Nawarro wrote: > Hello everyone, > > Can i create a btrfs fs in a machine and move the disk to another > machine like a ext4 or ntfs? Yeah, no problem whatsoever, even for multi-device filesystems. Btrfs doesn't care about what devices it is on. There has been no new incompat features in quite a while, so old kernels shouldn't be an issue either. I think the only way for valid btrfs to be unmountable on another machine with a modern kernel is to mess with page size, which is AFAIK not even possible on x86. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!? ⢿⡄⠘⠷⠚⠋⠀-- Genghis Ht'rok'din ⠈⠳⣄ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
On Wed, Mar 23, 2016 at 8:08 PM, Ryan Erato wrote: > Success! Using the same ISO you previously linked to, I ran 'btrfs > check --repair', did another check which actually revealed many new > issues, ran a repair again and after that successive checks showed no > signs of other issues. I was able to successfully send my 'home' > subvolume to the SSD and wow, huge difference with otherwise little > effort. > > Thanks to you and everybody else for helping me resolve this issue. Yay, good news! -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
Success! Using the same ISO you previously linked to, I ran 'btrfs check --repair', did another check which actually revealed many new issues, ran a repair again and after that successive checks showed no signs of other issues. I was able to successfully send my 'home' subvolume to the SSD and wow, huge difference with otherwise little effort. Thanks to you and everybody else for helping me resolve this issue. On Tue, Mar 22, 2016 at 8:34 PM, Chris Murphy wrote: > On Tue, Mar 22, 2016 at 7:40 PM, Ryan Erato wrote: >> Finally got around to running the suggested commands. Same error with >> the send, but not much output to help. The check operation did seem >> to reveal some potential issues. Here's the play-by-play along with >> the file output from check: >> >> [liveuser@localhost /]$ sudo btrfs check /dev/sda6 > >> /home/liveuser/btrfscheck.txt >> checking extents >> checking free space cache >> checking fs roots >> root 257 inode 13324701 errors 200, dir isize wrong >> root 258 inode 226392 errors 200, dir isize wrong >> root 258 inode 236055 errors 2000, link count wrong >> unresolved ref dir 226392 index 35 namelen 7 name LOG.old filetype 0 >> errors 3, no dir item, no dir index >> root 258 inode 236273 errors 2000, link count wrong >> root 258 inode 236276 errors 2000, link count wrong >> unresolved ref dir 226392 index 39 namelen 15 name MANIFEST-15 >> filetype 0 errors 3, no dir item, no dir index >> root 258 inode 236277 errors 2000, link count wrong >> unresolved ref dir 226392 index 41 namelen 7 name CURRENT filetype 0 >> errors 3, no dir item, no dir index >> root 258 inode 240618 errors 2000, link count wrong >> unresolved ref dir 226392 index 115 namelen 10 name 89.log >> filetype 0 errors 3, no dir item, no dir index >> root 487 inode 13324701 errors 200, dir isize wrong >> root 488 inode 226392 errors 200, dir isize wrong >> root 488 inode 236055 errors 2000, link count wrong >> unresolved ref dir 226392 index 35 namelen 7 name LOG.old filetype 0 >> errors 3, no dir item, no dir index >> root 488 inode 236273 errors 2000, link count wrong >> root 488 inode 236276 errors 2000, link count wrong >> unresolved ref dir 226392 index 39 namelen 15 name MANIFEST-15 >> filetype 0 errors 3, no dir item, no dir index >> root 488 inode 236277 errors 2000, link count wrong >> unresolved ref dir 226392 index 41 namelen 7 name CURRENT filetype 0 >> errors 3, no dir item, no dir index >> root 488 inode 240618 errors 2000, link count wrong >> unresolved ref dir 226392 index 115 namelen 10 name 89.log >> filetype 0 errors 3, no dir item, no dir index > > OK so now the question is if 'btrfs check --repair' can fix this, and > what version to use? 4.4.1 or 4.5.0? Based on the changelog, you can > probably use either version. And I think it should be safe. But, you > should still have backups seeing as you can mount the volume. > > >> >> [liveuser@localhost /]$ sudo mount /dev/sda6 /mnt/hdd >> >> [liveuser@localhost /]$ sudo btrfs send -vvv --no-data -f homesnap.btr >> /mnt/hdd/home/home.snap/ >> Mode NO_FILE_DATA enabled >> At subvol /mnt/hdd/home/home.snap/ >> ERROR: send ioctl failed with -2: No such file or directory > > OK so there's something off with the metadata and it's not going to do > a send as a result is what this sounds like to me. > > > -- > Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
On Tue, Mar 22, 2016 at 7:40 PM, Ryan Erato wrote: > Finally got around to running the suggested commands. Same error with > the send, but not much output to help. The check operation did seem > to reveal some potential issues. Here's the play-by-play along with > the file output from check: > > [liveuser@localhost /]$ sudo btrfs check /dev/sda6 > > /home/liveuser/btrfscheck.txt > checking extents > checking free space cache > checking fs roots > root 257 inode 13324701 errors 200, dir isize wrong > root 258 inode 226392 errors 200, dir isize wrong > root 258 inode 236055 errors 2000, link count wrong > unresolved ref dir 226392 index 35 namelen 7 name LOG.old filetype 0 > errors 3, no dir item, no dir index > root 258 inode 236273 errors 2000, link count wrong > root 258 inode 236276 errors 2000, link count wrong > unresolved ref dir 226392 index 39 namelen 15 name MANIFEST-15 > filetype 0 errors 3, no dir item, no dir index > root 258 inode 236277 errors 2000, link count wrong > unresolved ref dir 226392 index 41 namelen 7 name CURRENT filetype 0 > errors 3, no dir item, no dir index > root 258 inode 240618 errors 2000, link count wrong > unresolved ref dir 226392 index 115 namelen 10 name 89.log > filetype 0 errors 3, no dir item, no dir index > root 487 inode 13324701 errors 200, dir isize wrong > root 488 inode 226392 errors 200, dir isize wrong > root 488 inode 236055 errors 2000, link count wrong > unresolved ref dir 226392 index 35 namelen 7 name LOG.old filetype 0 > errors 3, no dir item, no dir index > root 488 inode 236273 errors 2000, link count wrong > root 488 inode 236276 errors 2000, link count wrong > unresolved ref dir 226392 index 39 namelen 15 name MANIFEST-15 > filetype 0 errors 3, no dir item, no dir index > root 488 inode 236277 errors 2000, link count wrong > unresolved ref dir 226392 index 41 namelen 7 name CURRENT filetype 0 > errors 3, no dir item, no dir index > root 488 inode 240618 errors 2000, link count wrong > unresolved ref dir 226392 index 115 namelen 10 name 89.log > filetype 0 errors 3, no dir item, no dir index OK so now the question is if 'btrfs check --repair' can fix this, and what version to use? 4.4.1 or 4.5.0? Based on the changelog, you can probably use either version. And I think it should be safe. But, you should still have backups seeing as you can mount the volume. > > [liveuser@localhost /]$ sudo mount /dev/sda6 /mnt/hdd > > [liveuser@localhost /]$ sudo btrfs send -vvv --no-data -f homesnap.btr > /mnt/hdd/home/home.snap/ > Mode NO_FILE_DATA enabled > At subvol /mnt/hdd/home/home.snap/ > ERROR: send ioctl failed with -2: No such file or directory OK so there's something off with the metadata and it's not going to do a send as a result is what this sounds like to me. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
Finally got around to running the suggested commands. Same error with the send, but not much output to help. The check operation did seem to reveal some potential issues. Here's the play-by-play along with the file output from check: [liveuser@localhost /]$ sudo btrfs check /dev/sda6 > /home/liveuser/btrfscheck.txt checking extents checking free space cache checking fs roots root 257 inode 13324701 errors 200, dir isize wrong root 258 inode 226392 errors 200, dir isize wrong root 258 inode 236055 errors 2000, link count wrong unresolved ref dir 226392 index 35 namelen 7 name LOG.old filetype 0 errors 3, no dir item, no dir index root 258 inode 236273 errors 2000, link count wrong root 258 inode 236276 errors 2000, link count wrong unresolved ref dir 226392 index 39 namelen 15 name MANIFEST-15 filetype 0 errors 3, no dir item, no dir index root 258 inode 236277 errors 2000, link count wrong unresolved ref dir 226392 index 41 namelen 7 name CURRENT filetype 0 errors 3, no dir item, no dir index root 258 inode 240618 errors 2000, link count wrong unresolved ref dir 226392 index 115 namelen 10 name 89.log filetype 0 errors 3, no dir item, no dir index root 487 inode 13324701 errors 200, dir isize wrong root 488 inode 226392 errors 200, dir isize wrong root 488 inode 236055 errors 2000, link count wrong unresolved ref dir 226392 index 35 namelen 7 name LOG.old filetype 0 errors 3, no dir item, no dir index root 488 inode 236273 errors 2000, link count wrong root 488 inode 236276 errors 2000, link count wrong unresolved ref dir 226392 index 39 namelen 15 name MANIFEST-15 filetype 0 errors 3, no dir item, no dir index root 488 inode 236277 errors 2000, link count wrong unresolved ref dir 226392 index 41 namelen 7 name CURRENT filetype 0 errors 3, no dir item, no dir index root 488 inode 240618 errors 2000, link count wrong unresolved ref dir 226392 index 115 namelen 10 name 89.log filetype 0 errors 3, no dir item, no dir index " btrfscheck.txt Checking filesystem on /dev/sda6 UUID: 6bb38bce-d824-4b9c-8b03-adad460c0f97 found 79333650514 bytes used err is 1 total csum bytes: 60906940 total tree bytes: 1530494976 total fs tree bytes: 1392934912 total extent tree bytes: 59506688 btree space waste bytes: 343045471 file data blocks allocated: 97137004544 referenced 85008084992 [liveuser@localhost /]$ sudo mount /dev/sda6 /mnt/hdd [liveuser@localhost /]$ sudo btrfs send -vvv --no-data -f homesnap.btr /mnt/hdd/home/home.snap/ Mode NO_FILE_DATA enabled At subvol /mnt/hdd/home/home.snap/ ERROR: send ioctl failed with -2: No such file or directory On Sun, Mar 20, 2016 at 10:42 PM, Chris Murphy wrote: > On Sun, Mar 20, 2016 at 10:34 PM, Ryan Erato wrote: > . >> >> Sending "home.snap" to "/mnt/ssd" results in the -2 error. What is >> peculiar, or possibly a red herring, is that it seems to fail at the >> same point each time, at 4.39GB in to the transfer. > > > > That's pretty suspicious. I didn't realize from the first description > that the command is doing something for a while before failing. I > thought it was failing immediately. > > Try this: > > btrfs send -vvv --no-data -f homesnap.btr home.snapshot > > That will write out metadata only to a file, no receive. See if the > error still happens and if the extra v gives more info. > > If it still fails with no more useful information then what I'd try > next is a btrfs check with the most recent btrfs-progs you can find. > If you're in need of a suggestion, this has btrfs-progs 4.4.1, I've > tested that it boots, it's got a published sha256 hash, and is served > over https. Yes, it's not even an alpha, but all you're doing is a > check, not a --repair, and no need to mount it (although that's > probably safe also, I've been doing it most of the weekend). > https://dl.fedoraproject.org/pub/alt/stage/24_Alpha-1.6/Workstation/x86_64/iso/ > dd that iso file to a USB stick, it will destroy all data on the > stick, and then boot the computer, and switch to tty2 (control-alt-f2) > to get to a shell. > > I think 'btrfs check > btrfscheck.txt' will output most of the results > to a text file. Often it misses the first few lines for whatever > reason. You can either 'fpaste ' and then note the URL and > post it here, or you can scp the file elsewhere, if you have wired > ethernet connected. > > > -- > Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
On Sun, Mar 20, 2016 at 10:34 PM, Ryan Erato wrote: . > > Sending "home.snap" to "/mnt/ssd" results in the -2 error. What is > peculiar, or possibly a red herring, is that it seems to fail at the > same point each time, at 4.39GB in to the transfer. That's pretty suspicious. I didn't realize from the first description that the command is doing something for a while before failing. I thought it was failing immediately. Try this: btrfs send -vvv --no-data -f homesnap.btr home.snapshot That will write out metadata only to a file, no receive. See if the error still happens and if the extra v gives more info. If it still fails with no more useful information then what I'd try next is a btrfs check with the most recent btrfs-progs you can find. If you're in need of a suggestion, this has btrfs-progs 4.4.1, I've tested that it boots, it's got a published sha256 hash, and is served over https. Yes, it's not even an alpha, but all you're doing is a check, not a --repair, and no need to mount it (although that's probably safe also, I've been doing it most of the weekend). https://dl.fedoraproject.org/pub/alt/stage/24_Alpha-1.6/Workstation/x86_64/iso/ dd that iso file to a USB stick, it will destroy all data on the stick, and then boot the computer, and switch to tty2 (control-alt-f2) to get to a shell. I think 'btrfs check > btrfscheck.txt' will output most of the results to a text file. Often it misses the first few lines for whatever reason. You can either 'fpaste ' and then note the URL and post it here, or you can scp the file elsewhere, if you have wired ethernet connected. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
Here's an example of what I've been trying: " mount new ssd root / # mount /dev/sdb6 /mnt/ssd/ " snapshot ROOT sub-volume mounted at / root / # btrfs subvol snapshot -r / /ROOT.snap Create a readonly snapshot of '/' in '//ROOT.snap' root / # btrfs filesystem sync / FSSync '/' root / # btrfs subvol list / ID 257 gen 747906 top level 5 path ROOT ID 258 gen 747906 top level 257 path home ID 487 gen 747893 top level 257 path ROOT.snap root / # btrfs send /ROOT.snap/ | btrfs receive /mnt/ssd/ At subvol /ROOT.snap/ At subvol ROOT.snap " snapshot home subvolume at /home root / # btrfs subvol snapshot -r /home/ /home/home.snap Create a readonly snapshot of '/home/' in '/home/home.snap' root / # btrfs filesystem sync /home FSSync '/home' root / # btrfs subvol list / ID 257 gen 747944 top level 5 path ROOT ID 258 gen 747944 top level 257 path home ID 487 gen 747893 top level 257 path ROOT.snap ID 488 gen 747942 top level 258 path home/home.snap root / # btrfs send /home/home.snap/ | btrfs receive /mnt/ssd/ At subvol /home/home.snap/ At subvol home.snap ERROR: send ioctl failed with -2: No such file or directory ERROR: unexpected EOF in stream. root / # btrfs subvol list /mnt/ssd/ ID 257 gen 57 top level 5 path ROOT.snap ID 285 gen 62 top level 5 path home.snap I have also attempted to mount /dev/sdb6 with mount option "subvol=ROOT" like I do in my system fstab for the current drive. I have also tried saving the snapshot in "/" and "/home", with the same results. Sending "home.snap" to "/mnt/ssd" results in the -2 error. What is peculiar, or possibly a red herring, is that it seems to fail at the same point each time, at 4.39GB in to the transfer. Using verbose output on receive it seems to process the files in the same order as again, I see the same set of files just before it fails each and every time. I setup my system according to gentoo documentation, so I'm not sure if I did something wrong with the initial btrfs partition setup that is making this more difficult. On Sun, Mar 20, 2016 at 12:31 PM, Chris Murphy wrote: > On Sat, Mar 19, 2016 at 4:58 PM, Ryan Erato wrote: >> I'm having quite the time trying to move my current Gentoo install to >> an SSD. I first attempted Clonezilla, but that failed while cloning >> the btrfs partition. I then realized I could use btrfs send/receive. >> >> The partition has 2 subvolumes ROOT and ROOT/home. ROOT snapshots and >> sends without hitch, but /home consistently errors at the same point >> (from -vv output on btrfs send). btrfs send returns -2, no such file >> or directory, unexpected EOF in stream. > > What's the exact command you're using for btrfs send receive for ROOT and > home? > > >> >> # cat /etc/fstab >> /dev/sda4 /boot vfat noauto,noatime 1 2 >> /dev/sda6 / btrfs defaults,noatime,compress=lzo,autodefrag,subvol=ROOT 0 0 >> /dev/sda5 none swap sw 0 0 >> >> # btrfs su l / >> ID 257 gen 745235 top level 5 path ROOT >> ID 258 gen 745235 top level 257 path home >> ID 498 gen 744743 top level 5 path ROOT.snap >> ID 501 gen 745043 top level 258 path home/home.snap > > Nested subvolumes makes this more complicated in my opinion. You might > have better luck mounting subvolid=5 to /mnt and using a fully > qualified pathname for send. > > > > -- > Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
I'm not an expert by any means, but I did a migration like this a few weeks ago. The most consistent recommendation on this mailing list is to use the newest kernels and btrfs-progs feasible. I did my migration using Fedora 24 live media, which at the time was kernel ~4.3. I see your btrfs-progs is a little old. Maybe the kernel is as well and has a bug. I know you tried a bunch of variations, but it would be helpful if you posted the commands you've tried. Perhaps the most obvious problem that could be occurring is trying to send a RW subvol, which is illegal. I can't tell if you made a RO snapshot first, but your success with ROOT indicates you're aware of this limitation. I'm guessing that there's some problem with the destination path that you're providing. The path should be the containing subvol where you want it to go, not the destination name (i.e. btrfs receive /var/media/backups/ not btrfs receive /var/media/backups/HOME). Here's some send/receive commands that I've successfully used recently. Maybe they'll point you in the right direction: pull a RO snapshot from a remote system: cd ~/ksp/backups ssh 192.168.0.122 "sudo btrfs send /home/justin/ksp/backups/2016-03-15-17:24" | sudo btrfs receive . mv 2016-03-15-17:24 ../current Note how I received into the backups directory and have no control over the initial subvol name. I mv it to the proper location and name afterwards. move a RO snapshot between local file system: cd /tmp/old-btrfs/ sudo btrfs subvol snap -r home home-snap cd /tmp/new-btrfs/ sudo btrfs send /tmp/old-btrfs/home-snap | sudo btrfs receive . sudo btrfs subvol snapshot home-snap home sudo btrfs subvol delete home-snap Fedora names it's /home subvol "home." You *shouldn't* need to mess around with advanced mounting options to get this to work. On Sun, Mar 20, 2016 at 11:52 AM, Ryan Erato wrote: > I do plan on physically replacing the current drive with the new one > and my fstab/boot comands use device. I never could get UUID or labels > to work, but that's another project. > > However, this still leaves me unable to take advantage of btrfs > features for implementing an incremental backup solution to another > disk. > Ryan Erato > C. 414.630.5295 > rer...@gmail.com > http://www.linkedin.com/in/ryanerato > > > On Sun, Mar 20, 2016 at 1:55 AM, Phantom Guy wrote: >> You tried to hard. >> >> Build the new media If you need encryption or meets devices. >> >> >> Then add the new media to the whole filesystem. >> >> >> Then remove the old media from the filesystem. >> >> >> # btrfs device add /dev/sdb1 /mountpoint >> >> # btrfs device del /dev/sda1 /mountpoint >> >> Then sync the filesystem and wait. >> >> Once the old device ID no longer part of the filesystem it's done. >> >> When the sync finishes, the /dev/sda1 device is nill. >> >> If you've designed your fstsb and boot commands correctly the label or UUID >> has migrated so as long as you've prepped the disk with grub of whatever >> then you can move the disks with impunity. >> >> You don't even need to be in any sort of maintenance mode. >> >> >>> >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
On Sat, Mar 19, 2016 at 4:58 PM, Ryan Erato wrote: > I'm having quite the time trying to move my current Gentoo install to > an SSD. I first attempted Clonezilla, but that failed while cloning > the btrfs partition. I then realized I could use btrfs send/receive. > > The partition has 2 subvolumes ROOT and ROOT/home. ROOT snapshots and > sends without hitch, but /home consistently errors at the same point > (from -vv output on btrfs send). btrfs send returns -2, no such file > or directory, unexpected EOF in stream. What's the exact command you're using for btrfs send receive for ROOT and home? > > # cat /etc/fstab > /dev/sda4 /boot vfat noauto,noatime 1 2 > /dev/sda6 / btrfs defaults,noatime,compress=lzo,autodefrag,subvol=ROOT 0 0 > /dev/sda5 none swap sw 0 0 > > # btrfs su l / > ID 257 gen 745235 top level 5 path ROOT > ID 258 gen 745235 top level 257 path home > ID 498 gen 744743 top level 5 path ROOT.snap > ID 501 gen 745043 top level 258 path home/home.snap Nested subvolumes makes this more complicated in my opinion. You might have better luck mounting subvolid=5 to /mnt and using a fully qualified pathname for send. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving btrfs subvolumes to new disk
I do plan on physically replacing the current drive with the new one and my fstab/boot comands use device. I never could get UUID or labels to work, but that's another project. However, this still leaves me unable to take advantage of btrfs features for implementing an incremental backup solution to another disk. Ryan Erato C. 414.630.5295 rer...@gmail.com http://www.linkedin.com/in/ryanerato On Sun, Mar 20, 2016 at 1:55 AM, Phantom Guy wrote: > You tried to hard. > > Build the new media If you need encryption or meets devices. > > > Then add the new media to the whole filesystem. > > > Then remove the old media from the filesystem. > > > # btrfs device add /dev/sdb1 /mountpoint > > # btrfs device del /dev/sda1 /mountpoint > > Then sync the filesystem and wait. > > Once the old device ID no longer part of the filesystem it's done. > > When the sync finishes, the /dev/sda1 device is nill. > > If you've designed your fstsb and boot commands correctly the label or UUID > has migrated so as long as you've prepped the disk with grub of whatever > then you can move the disks with impunity. > > You don't even need to be in any sort of maintenance mode. > > >> > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving snapshots
On Fri, Jan 23, 2015 at 5:05 AM, Matthias Urlichs wrote: > Hello, > > how do I move a (read-only) snapshot? > If you want to move a read-only snapshot to a different directory, '..' changes, and therefore is not a read-only operation. > Simply creating another read-only snap from the first one and then deleting > the source works, except that it destroy's the snapshot's identity -- which > means that it can't be used as a parent for "btrfs receive" any more. :-( # btrfs property set -ts /path/to/snapshot ro false # mv /path/to/snapshot /new/path # btrfs property set -ts /new/path ro true worked for me > > -- > -- Matthias Urlichs -- Noah -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Tue, Dec 02, 2014 at 08:41:35PM +0530, Shriramana Sharma wrote: > That makes sense. Is there anywhere that the "official" SuSE > recommended subvol layout is mentioned that I can refer to without > having to start up an installer? https://www.suse.com/documentation/sles-12/singlehtml/book_sle_admin/book_sle_admin.html#sec.snapper.setup "Directories that are Excluded from Snapshots" > I am now reading a SuSECon 2013 presentation by Nyers and Schnell but > they are very generic about the recommendations. There are some recommended defaults that should be ok, the configuration is flexible and the user can tune the settings later according to the usage pattern (expected amount of data changes between snapshots, frequency of changes). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Wed, Dec 03, 2014 at 08:02:31AM +0530, Shriramana Sharma wrote: > On Tue, Dec 2, 2014 at 2:04 PM, Hugo Mills wrote: > > > >> Is that correct: what btr sub list shows as "top level" is indeed the > >> parent subvolume? > > > >No, it's the top-level subvolume. (See my earlier mail about > > nomenclature). "Parent subvolume" has a number of meanings, none of > > which should be "the subvolume with subvolid 5". > > Um I searched my inbox but didn't find a specific definition from you > for "top-level". You only said it's better to avoid calling it "root" > to avoid confounding it with the subvol that may be mounted at root > i.e. /. It was the first line I wrote in my first reply to your thread about subvol 5 vs subvol 0. I had hoped to be both definitive and comprehensive. > IIUC the "top-level subvolume" can only be subvolid 5 which accords > with your later comment: > > > that putting files in the top-level subvol can't do what most people > > want to do with it. Hence the recommended subvol management layout at > > [1] https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes > > ... which means that I am not able to understand the output of btr sub > list which gives the subvolid of whichever subvol is currently the > "parent" (as in outer nesting) subvol. Observe: > > $ btr sub list . > ID 257 gen 10 top level 5 path test1 > ID 258 gen 10 top level 5 path test2 > ID 259 gen 9 top level 258 path test2/foo > $ sudo mv test2/foo test1/ > $ btr sub list . > ID 257 gen 10 top level 5 path test1 > ID 258 gen 10 top level 5 path test2 > ID 259 gen 9 top level 257 path test1/foo > $ > > So now what is the meaning of "top level"? Urgh. I haven't seriously looked at that piece of output in a while. That's broken, in my opinion. Here, "top level" means "containing subvolume ID". Hugo. -- Hugo Mills | "I don't like the look of it, I tell you." "Well, hugo@... carfax.org.uk | stop looking at it, then." http://carfax.org.uk/ | PGP: 65E74AC0 | The Goons signature.asc Description: Digital signature
Re: Moving an entire subvol?
On Wed, Dec 3, 2014 at 2:43 AM, Austin S Hemmelgarn wrote: > Here's my approach to things: Wow, thanks a lot people! I'm really benefiting from your experience here. -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा
Re: Moving an entire subvol?
On Tue, Dec 2, 2014 at 2:04 PM, Hugo Mills wrote: > >> Is that correct: what btr sub list shows as "top level" is indeed the >> parent subvolume? > >No, it's the top-level subvolume. (See my earlier mail about > nomenclature). "Parent subvolume" has a number of meanings, none of > which should be "the subvolume with subvolid 5". Um I searched my inbox but didn't find a specific definition from you for "top-level". You only said it's better to avoid calling it "root" to avoid confounding it with the subvol that may be mounted at root i.e. /. IIUC the "top-level subvolume" can only be subvolid 5 which accords with your later comment: > that putting files in the top-level subvol can't do what most people > want to do with it. Hence the recommended subvol management layout at > [1] https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes ... which means that I am not able to understand the output of btr sub list which gives the subvolid of whichever subvol is currently the "parent" (as in outer nesting) subvol. Observe: $ btr sub list . ID 257 gen 10 top level 5 path test1 ID 258 gen 10 top level 5 path test2 ID 259 gen 9 top level 258 path test2/foo $ sudo mv test2/foo test1/ $ btr sub list . ID 257 gen 10 top level 5 path test1 ID 258 gen 10 top level 5 path test2 ID 259 gen 9 top level 257 path test1/foo $ So now what is the meaning of "top level"? -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On 2014-12-02 10:11, Shriramana Sharma wrote: On Tue, Dec 2, 2014 at 6:58 PM, David Sterba wrote: A subvolume is also a snapshotting barrier, so it's convenient to create subvolumes in well-known paths that contain data that should not be rolled back (/var/log, /srv, bootloader). Hi David -- a real honour to meet one of the core Btrfs/SuSE (heh, when that was the spelling!) guys! That makes sense. Is there anywhere that the "official" SuSE recommended subvol layout is mentioned that I can refer to without having to start up an installer? (I currently chose ext4 for / for other reasons so I can't refer to my layout.) I am now reading a SuSECon 2013 presentation by Nyers and Schnell but they are very generic about the recommendations. Here's my approach to things: In the top level of the btrfs filesystem I use for /, I have a subvolume called /root, This is what get's mount on /. I also have a separate subvolume called /home for the home directories when I have those on the same FS. I place /boot on an entirely separate filesystem because I use a bunch of mount options there that would break or slow down other filesystems (most notably, noexec, nosuid, nodev, and sync). Within both /home and /root, I use a handful of subvolumes to control what gets saved in a snapshot, the most notable examples being /var//log, /usr/portage, and /home/austin/dropbox. As far as snapshots go, I take a snapshot of /root every time I boot, and keep the past 2 days worth, take a snapshot of /home hourly, and keep a weeks worth, and do a snapshot of both when I generate a system backup. I generally don't do snapshots of /boot, as I keep around the previous few kernel versions anyway, and mark things there as immutable so that I can't accidentally mess them up. smime.p7s Description: S/MIME Cryptographic Signature
Re: Moving an entire subvol?
On 12/02/2014 07:11 AM, Shriramana Sharma wrote:
On Tue, Dec 2, 2014 at 6:58 PM, David Sterba wrote:
A subvolume is also a snapshotting barrier, so it's convenient to create
subvolumes in well-known paths that contain data that should not be
rolled back (/var/log, /srv, bootloader).
That makes sense. Is there anywhere that the "official" SuSE
recommended subvol layout is mentioned that I can refer to without
having to start up an installer? (I currently chose ext4 for / for
other reasons so I can't refer to my layout.)
There are lots of ways to arrange your system
My preference is to create the snapshots in a super-volume outside of
the normally mounted hierarchy. This simplifies the normal operation of
tools like "locate" which don't understand that the duplicate files from
the snapshot are not interesting. It also means that live operations
(e.g. anything using "find") naturally will not traverse the snapshots
unless the supervolume is mounted explicitly.
I typically call the active root of the system /__System and set it as
the default subvolume to make booting easier.
As in...
mkfs.btrfs /dev/whatever
mount /dev/whatever /mnt
btrfs sub create /mnt/__System
btrfs sub create /mnt/__System/home
btrfs sub set-default (number) /mnt
umount /mnt
mount /mnt
(create OS layout in /mnt)
Then when the snapshotting goes on...
mount -o subvol=/ /dev/whatever /maintenance
SUFFIX=$(date "+_BACKUP_%Y-%M-%d")
cd /maintenance
btrfs sub snap -r __System __System${SUFFIX}
btrfs sub snap -r __System/home __System_home${SIFFIX}
# etc
cd /
umount /maintenance
---
The Real Way™ to think about your active subvolume layouts is to think
about what you really need to preserve and how often.
/home is an obvious candidate for frequent snapshots as it is the place
individual users are most likely to mess up, and it has the most
"irreplaceable" data.
/ (e.g. the semantic system root) [in my example /__System] [not
counting its various subvolumes really only needs backing up before
system software modifications via apt/yumm/portage/wahtever your distro
uses. Or right before you start doing aything "tricky" in /etc
/etc might rate its own subvolume if you are a tinker or your
system-wide configuration needs a lot of motility.
/var tends to have per system configuration stuff
But the real question is "how much complexity of maintenance does the
system _really_ need, and how much of it are you _really_ going to maintain.
The desire to use a feature "just 'cause it's there" should be resisted.
If you are not going to be using the snapshots feature. If you are just
dropping the box in and you are going to ignore it, then don't subvolume
it at all.
You are looking for a balance between the theoretical ideal and the
practical outcome. If you don't know exactly why you are putting the
subvolume in place then it will likely just end up annoying you without
giving any value.
Same for taking and positioning the snapshots.
This is a corollary of the rule that states "A backup script that you've
never done a restore from, should be assumed to be an _unsafe_ or
complete backup, or no backup at all."
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Tue, Dec 2, 2014 at 6:58 PM, David Sterba wrote: > > A subvolume is also a snapshotting barrier, so it's convenient to create > subvolumes in well-known paths that contain data that should not be > rolled back (/var/log, /srv, bootloader). Hi David -- a real honour to meet one of the core Btrfs/SuSE (heh, when that was the spelling!) guys! That makes sense. Is there anywhere that the "official" SuSE recommended subvol layout is mentioned that I can refer to without having to start up an installer? (I currently chose ext4 for / for other reasons so I can't refer to my layout.) I am now reading a SuSECon 2013 presentation by Nyers and Schnell but they are very generic about the recommendations. -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Tue, Dec 02, 2014 at 08:51:40AM +0530, Shriramana Sharma wrote: > > openSUSE uses subvol id 5 for installing the OS to, and some > > directories are made subvolumes such as home var and maybe usr. > > Therefore when subvolid 5 is snapshot, those are exempt, and have to > > be individually snapshot. > > Yes I also noticed that openSUSE creates such separate subvols, but is > there any particular benefit to making it so? A subvolume is also a snapshotting barrier, so it's convenient to create subvolumes in well-known paths that contain data that should not be rolled back (/var/log, /srv, bootloader). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
Shriramana Sharma posted on Tue, 02 Dec 2014 08:51:40 +0530 as excerpted: > On Mon, Dec 1, 2014 at 6:24 AM, Chris Murphy > wrote: >>> But isn't it just possible to move i.e. reparent a subvol so I can >>> move these two under another subvol and have that as default? >> >> You can move subvolumes. > > OK so I just found out that just mv test1/foo test2/ where test1, > test2 and foo are all subvolumes is sufficient to reparent foo to test2, > if what btr sub list shows as "top level" is indeed the parent > subvolume. > > Is that correct: what btr sub list shows as "top level" is indeed the > parent subvolume? [Noting that my use-case doesn't involve subvolumes so while I've played with them a bit my direct knowledge is limited...] It should be correct, yes. Subvolumes are in many ways "super-directories", so it's little surprise simply directory manipulation such as moves would do what you might expect. They just happen to be directly mountable too, and to have various btrfs-specific effects such as snapshots stopping at subvolume boundaries, usage for btrfs send/receive, etc. >> My suggestion is subvolumes containing binaries shouldn't be located >> within another subvolume that ends up being mounted, that way old >> binaries with possible vulnerabilities aren't exposed in the normal >> search path. > > I'm not sure what you mean. Are you saying that for example /usr/bin > should be: > > 1) a separate subvolume than / or /usr, > 2) not a child subvolume of / or /usr? What I believe he's referencing is the potential security issue if for example you have older snapshots of /usr (which would include /usr/bin and /usr/lib(64)) accessible under normal operating conditions. These snapshots would contain older versions of binaries (and libraries) that have been security-updated on the main system, but the snapshots would of course contain the still vulnerable versions. A user trying to do a root- escalation, for instance, could then access and run one of these old and vulnerable versions by specifying the full path instead of just the name, thus getting access to a known root-escalation vuln long since patched in the main path but still vulnerable in the snapshot path. If for instance the master id=5 subvolume is still the default and routinely mounted, it will have all snapshots appearing as directories somewhere beneath its mountpoint in the tree. If those snapshots contain bin or lib dirs, the above security scenario is a real possibility, since they'll be accessible in the tree. So making something other than the master id=5 subvolume the default, mounting id=5 only when doing subvolume maintenance not routinely, and making such bin/lib-containing snapshots direct children of id=5 instead of children of the / subvolume or the like, will keep the snapshots containing the possibly vulnerable bins/libs out of normal accessibility as they'll only be visible in the tree when id=5 is mounted for snapshot maintenance, etc. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Tue, Dec 02, 2014 at 08:51:40AM +0530, Shriramana Sharma wrote: > On Mon, Dec 1, 2014 at 6:24 AM, Chris Murphy wrote: > >> But isn't it just possible to move i.e. reparent a > >> subvol so I can move these two under another subvol and have that as > >> default? > > > > You can move subvolumes. > > OK so I just found out that just mv test1/foo test2/ where test1, > test2 and foo are all subvolumes is sufficient to reparent foo to > test2, if what btr sub list shows as "top level" is indeed the parent > subvolume. > > Is that correct: what btr sub list shows as "top level" is indeed the > parent subvolume? No, it's the top-level subvolume. (See my earlier mail about nomenclature). "Parent subvolume" has a number of meanings, none of which should be "the subvolume with subvolid 5". > > My suggestion is subvolumes containing > > binaries shouldn't be located within another subvolume that ends up > > being mounted, that way old binaries with possible vulnerabilities > > aren't exposed in the normal search path. > > I'm not sure what you mean. Are you saying that for example /usr/bin should > be: > > 1) a separate subvolume than / or /usr, > 2) not a child subvolume of / or /usr? > > > openSUSE uses subvol id 5 for installing the OS to, and some > > directories are made subvolumes such as home var and maybe usr. > > Therefore when subvolid 5 is snapshot, those are exempt, and have to > > be individually snapshot. > > Yes I also noticed that openSUSE creates such separate subvols, but is > there any particular benefit to making it so? In the sense of allowing independent snapshotting, yes. I might want to back up / (with usr, var, and so forth) only when I do a system upgrade, but /home every night. Making /home a separate subvol gives me the ability to snapshot those two areas independently. > > Fedora uses subvolumes root and home by default, and fstab uses > > subvol=root and subvol=home to mount them at / and /home respectively. > > This seems similar to Ubuntu's @ and @home setup. > > Is there any advantage to either? That is, one model installs root to > the topmost subvol and makes usr, home etc nested subvols, whereas > another makes root a nested subvol under the topmost just like usr > home etc, and then mounts it to /... > > In general it seems people (or at least distros) prefer avoiding > nesting subvolumes. Is there any particular reason for this? Esp in > regard to /usr etc it would seem that if they are nested within the > subvol for /, then just mounting that subvol would automatically mount > all nested subvolumes, right? So the extra effort needed to mount the > nested subvols would not be necessary, no? Nested subvols tend to get messy in practice. It's harder to replace a "higher level" one, because you've got to move the "lower level" ones around. It's also much harder to make a send/receive backup of the subvols in their original relationships, because of the read-only requirement. Whilst the theory came first, several years of practice has shown both that nesting subvolumes is generally more awkward to manage, and that putting files in the top-level subvol can't do what most people want to do with it. Hence the recommended subvol management layout at [1]. Hugo. [1] https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes -- Hugo Mills | We teach people management skills by examining hugo@... carfax.org.uk | characters in Shakespeare. You could look at http://carfax.org.uk/ | Claudius's crisis management techniques, for PGP: 65E74AC0 | example. Richard Smith-Jones, Slings and Arrows signature.asc Description: Digital signature
Re: Moving an entire subvol?
On Mon, Dec 1, 2014 at 6:24 AM, Chris Murphy wrote: >> But isn't it just possible to move i.e. reparent a >> subvol so I can move these two under another subvol and have that as >> default? > > You can move subvolumes. OK so I just found out that just mv test1/foo test2/ where test1, test2 and foo are all subvolumes is sufficient to reparent foo to test2, if what btr sub list shows as "top level" is indeed the parent subvolume. Is that correct: what btr sub list shows as "top level" is indeed the parent subvolume? > My suggestion is subvolumes containing > binaries shouldn't be located within another subvolume that ends up > being mounted, that way old binaries with possible vulnerabilities > aren't exposed in the normal search path. I'm not sure what you mean. Are you saying that for example /usr/bin should be: 1) a separate subvolume than / or /usr, 2) not a child subvolume of / or /usr? > openSUSE uses subvol id 5 for installing the OS to, and some > directories are made subvolumes such as home var and maybe usr. > Therefore when subvolid 5 is snapshot, those are exempt, and have to > be individually snapshot. Yes I also noticed that openSUSE creates such separate subvols, but is there any particular benefit to making it so? > Fedora uses subvolumes root and home by default, and fstab uses > subvol=root and subvol=home to mount them at / and /home respectively. This seems similar to Ubuntu's @ and @home setup. Is there any advantage to either? That is, one model installs root to the topmost subvol and makes usr, home etc nested subvols, whereas another makes root a nested subvol under the topmost just like usr home etc, and then mounts it to /... In general it seems people (or at least distros) prefer avoiding nesting subvolumes. Is there any particular reason for this? Esp in regard to /usr etc it would seem that if they are nested within the subvol for /, then just mounting that subvol would automatically mount all nested subvolumes, right? So the extra effort needed to mount the nested subvols would not be necessary, no? Shriramana. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Sat, Nov 29, 2014 at 8:31 PM, Shriramana Sharma wrote: > So the Ubuntu Wiki BtrFS entry advises against using subvol > set-default because it boots its kernel using root=subvol=@ and home > as subvol=@home, and these two subvols are only present under the > subvol with ID 5. The advice may have had to do with GRUB behavior prior to 2.02. Previously GRUB attempted to honor the btrfs default subvolume, and therefore treated any path in grub.cfg relative to the default subvolume. Now, GRUB behaves the same as the subvol= mount option, it is always treated as an absolute path from subvol id 5, hence the default subvolume is ignored. Since the default subvolume is set by a user space program I think it's a domain violation for anything to subvert this; it really should remain a shortcut for the user's benefit only, so they can use mount without -o subvol=. Everything else should explicitly pass subvol= > But isn't it just possible to move i.e. reparent a > subvol so I can move these two under another subvol and have that as > default? You can move subvolumes. My suggestion is subvolumes containing binaries shouldn't be located within another subvolume that ends up being mounted, that way old binaries with possible vulnerabilities aren't exposed in the normal search path. > > Possibly this is a hypothetical question as I'm not sure whether it > would be actually practically required but looking at the specific > Ubuntu advice on this I thought I should ask. > > I'm also not sure what openSUSE (or other distros) do about this... Do > they mount root using subvolid, or subvol name or such? openSUSE uses subvol id 5 for installing the OS to, and some directories are made subvolumes such as home var and maybe usr. Therefore when subvolid 5 is snapshot, those are exempt, and have to be individually snapshot. The snapshots are found in the same root directory everything else is, in a . directory (I think .snapshots ?) Fedora uses subvolumes root and home by default, and fstab uses subvol=root and subvol=home to mount them at / and /home respectively. I don't know any distro using subvolid right now but that might be prudent as it's far less user domain than subvolume names. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Sun, Nov 30, 2014 at 03:57:06PM +0530, Shriramana Sharma wrote: > On Sun, Nov 30, 2014 at 9:51 AM, Marc MERLIN wrote: > > > >> So the Ubuntu Wiki BtrFS entry advises against using subvol > >> set-default because it boots its kernel using root=subvol=@ and home > >> as subvol=@home, and these two subvols are only present under the > >> subvol with ID 5. But isn't it just possible to move i.e. reparent a > >> subvol so I can move these two under another subvol and have that as > >> default? > > > > Make a new subvolume called /root and just mount subvol=root > > Sorry if my question wasn't clear: I wanted to know how to move a > subvol to appear under another subvol other than its original parent. > Turns out that sudo mv @ @home target/ is quite sufficient. If so why > would the Ubuntu wiki require that set-default not be used? Just @ > @home need to be moved to the new place, no? I've never done that. If I had to move them, I'd just change the mountpoint. > > Note that you can't mount subvols recursively in one mount AFAIK. > > I'm not sure what you mean. I have a few subvols in my external HDD > which is entirely formatted as BtrFS and if I just mount the external > HDD /dev/sdc1 I am able to access all the subvols' contents as well. Yes, if you mount the root, it works of course. If you mount a subvol, you cannot have it automatically have it mount other subvols. Subvols don't really know or care where they are mounted compared to one another, and who is under whom. It's just mount setup. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving contents from one subvol to another
On Sun, Nov 30, 2014 at 9:23 AM, Shriramana Sharma wrote: > > > Why should noCoW affect cp --reflink anyhow? I just created a 500 MiB > file from /dev/urandom under a chattr +C-ed dir, and copied to another > subvol using cp --reflink, and fi df still shows 500 MiB, not 1 GiB. Looks like I might have spoken too soon (because I've read that some changes aren't visible until the next FS commit) so right now it actually says 1 GiB used, which I can't grok because why should a nocow file be physically copied (to new blocks) just because it's nocow? Is it because it is possible that the two copies are overwritten separately at the same time? But still, it seems to me that mv should make it so that the nocow attr is temporarily (atomically?) suspended/ignored just for the duration of the relocation, since there aren't going to be any two copies to be overwritten at the same time. Comments? -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Sun, Nov 30, 2014 at 9:51 AM, Marc MERLIN wrote: > >> So the Ubuntu Wiki BtrFS entry advises against using subvol >> set-default because it boots its kernel using root=subvol=@ and home >> as subvol=@home, and these two subvols are only present under the >> subvol with ID 5. But isn't it just possible to move i.e. reparent a >> subvol so I can move these two under another subvol and have that as >> default? > > Make a new subvolume called /root and just mount subvol=root Sorry if my question wasn't clear: I wanted to know how to move a subvol to appear under another subvol other than its original parent. Turns out that sudo mv @ @home target/ is quite sufficient. If so why would the Ubuntu wiki require that set-default not be used? Just @ @home need to be moved to the new place, no? > Note that you can't mount subvols recursively in one mount AFAIK. I'm not sure what you mean. I have a few subvols in my external HDD which is entirely formatted as BtrFS and if I just mount the external HDD /dev/sdc1 I am able to access all the subvols' contents as well. -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving an entire subvol?
On Sun, Nov 30, 2014 at 09:01:42AM +0530, Shriramana Sharma wrote: > So the Ubuntu Wiki BtrFS entry advises against using subvol > set-default because it boots its kernel using root=subvol=@ and home > as subvol=@home, and these two subvols are only present under the > subvol with ID 5. But isn't it just possible to move i.e. reparent a > subvol so I can move these two under another subvol and have that as > default? Make a new subvolume called /root and just mount subvol=root Note that you can't mount subvols recursively in one mount AFAIK. This is what my system looks like: LABEL=btrfs_pool1 / btrfs subvol=root,defaults,compress=lzo,discard,skip_balance,noatime 0 0 LABEL=btrfs_pool1 /usrbtrfs subvol=usr,defaults,compress=lzo,discard,skip_balance,noatime 0 0 LABEL=btrfs_pool1 /varbtrfs subvol=var,defaults,compress=lzo,discard,skip_balance,noatime 0 0 LABEL=btrfs_pool1 /home btrfs subvol=home,defaults,compress=lzo,discard,skip_balance,noatime 0 0 LABEL=btrfs_pool1 /tmpbtrfs subvol=tmp,defaults,compress=lzo,discard,skip_balance,noatime,noexec 0 0 LABEL=btrfs_pool1 /mnt/btrfs_pool1 btrfs defaults,compress=lzo,discard,skip_balance,noatime,subvolid=0 0 0 Hope this helps. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving contents from one subvol to another
On Sat, Nov 29, 2014 at 10:37 PM, Robert White wrote: > > One thing to keep in mind is that mv, when crossing any of these boundaries > degenerates to a copy-and-remove operation and _none_ of the source files > will be removed until _all_ of the files have been copied. If any of the > copy operations fail the removes will not take place at all. It would only > take a couple large NOCOW files to put you over a limit somewhere. Hmm... So you're saying like because the copy routine that mv calls will see the nocow attribute (and it doesn't know it's being called as part of a move operation) and so do a full copy rather than reflink? Correct me if I'm wrong but it seems that mv should actually ignore the nocow attribute as far as moving it to a new location is concerned, no, because I'm moving, not copying? Of course it should retain the attribute of the original files *after* the move is done. Why should noCoW affect cp --reflink anyhow? I just created a 500 MiB file from /dev/urandom under a chattr +C-ed dir, and copied to another subvol using cp --reflink, and fi df still shows 500 MiB, not 1 GiB. > If you are consolidating sub-volumes (as per the original question) on a > "nearly full" drive you may want to do it all long-hand with a script moving > various chunks or something instead of just trying a move/copy of "cp > --reflinks /vol1/* /vol2/" (same for mv when you get that --reflinks > revision). As I said, there doesn't actually seem to be a --reflink command line option. -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving contents from one subvol to another
On 11/29/2014 07:15 AM, Shriramana Sharma wrote: On Sat, Nov 29, 2014 at 7:58 PM, Hugo Mills wrote: The latest version of mv should be able to use CoW copies to make it more efficient. It has a --reflink option, the same as cp. Note that you can't make reflinks crossing a mount boundary, but you can do so crossing a subvolume boundary (as you're doing here). One thing to keep in mind is that mv, when crossing any of these boundaries degenerates to a copy-and-remove operation and _none_ of the source files will be removed until _all_ of the files have been copied. If any of the copy operations fail the removes will not take place at all. It would only take a couple large NOCOW files to put you over a limit somewhere. So if you get to an out-of-space condition mid-move you are going to have to disentangle your stuff by hand anyway. If you are consolidating sub-volumes (as per the original question) on a "nearly full" drive you may want to do it all long-hand with a script moving various chunks or something instead of just trying a move/copy of "cp --reflinks /vol1/* /vol2/" (same for mv when you get that --reflinks revision). ASIDE: Also be aware that such a moment would be the perfect time to consider compression and so on. A regular copy (non reflinks) will apply the currently selected compress= regime (and reconsider sparsity etc) in a way that the move will not. e.g. once you decide to do intrusive maintenance you might be well served by taking the extra time to restructure your storage. 8-) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving contents from one subvol to another
On Sat, Nov 29, 2014 at 7:58 PM, Hugo Mills wrote: >The latest version of mv should be able to use CoW copies to make > it more efficient. It has a --reflink option, the same as cp. Note > that you can't make reflinks crossing a mount boundary, but you can do > so crossing a subvolume boundary (as you're doing here). Hi thanks for this. I suppose you are referring to the commit: http://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=b47231be6813e6cb5305266e391b4bb745f27f13 >From http://git.savannah.gnu.org/cgit/coreutils.git/log/?qt=grep&q=mv%3A, http://git.savannah.gnu.org/cgit/coreutils.git/plain/NEWS?id=b47231be6813e6cb5305266e391b4bb745f27f13 and finally http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/mv.c?id=b47231be6813e6cb5305266e391b4bb745f27f13 it doesn't seem as if there was any earlier commit actually adding a --reflink option so it seems the improvement is in-built. That's nice to know! Any idea when the next coreutils point release with this will be out? -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving contents from one subvol to another
On Sat, Nov 29, 2014 at 07:51:07PM +0530, Shriramana Sharma wrote:
> Hello. I am now taking the first steps to making my backup external
> HDD in BtrFS. From
> http://askubuntu.com/questions/119014/btrfs-subvolumes-vs-folders I
> understand that the only difference between subvolumes and ordinary
> folders is that the former can be snapshotted and independently
> mounted.
>
> But I have a question. I have two subvols test1, test2.
>
> $ cd test1
> $ dd if=/dev/urandom of=file bs=1M count=500
> 500+0 records in
> 500+0 records out
> 524288000 bytes (524 MB) copied, 36.2291 s, 14.5 MB/s
> $ time mv file ../test2/
> real0m2.061s
> user0m0.013s
> sys 0m0.459s
> $ time { cp --reflink ../test2/file . && rm ../test2/file ; }
> real0m0.677s
> user0m0.022s
> sys 0m0.086s
> $ mkdir foo
> $ time mv file foo/
> real0m0.096s
> user0m0.008s
> sys 0m0.013s
>
> It seems that mv is not CoW aware and hence is not able to create
> reflinks so it is actually processing the entire file because it
> thinks test2 is a different device/filesystem/partition or such. Is
> this understanding correct?
The latest version of mv should be able to use CoW copies to make
it more efficient. It has a --reflink option, the same as cp. Note
that you can't make reflinks crossing a mount boundary, but you can do
so crossing a subvolume boundary (as you're doing here).
> So doing cp --reflink with rm is much faster. But it is still slower
> than doing mv within the same subvol. Is it because of the
> housekeeping with updating the metadata of the two subvols?
I should think so, yes.
> Methinks --reflink option should be added to mv for the above usecase.
> Do people think this is useful? Why or why not?
See above: it already has been. :)
> My concern is that if somebody wants to consolidate two subvols into
> one, though really only the metadata needs to be processed using
> ordinary mv isn't aware of this and using cp --reflink with rm is
> unnecessarily complicated, especially if it will involve multiple
> files.
>
> And it's not clear to me what it would entail to cp --reflink + rm an
> entire directory tree because IIUC I'd have to handle each file
> separately. Perhaps something (unnecessarily convoluted) like:
>
> find . | while read f
> do
> [ -d "$f" ] && mkdir target/"$f" && touch target/"$f" -r "$f"
> [ -f "$f" ] && cp -a --reflink "$f" target/ && rm "$f"
> done
>
> Again, what would happen to files which are not regular directories or files?
Probably just the same thing that would happen without the
--reflink=always.
> And why isn't --reflink given a single letter alias for cp?
I don't know about that; you'll have to ask the coreutils
developers. They're probably expecting it to be largely set to a
single value by default (e.g. through a shall alias).
Hugo.
--
Hugo Mills | "I will not be pushed, filed, stamped, indexed,
hugo@... carfax.org.uk | briefed, debriefed or numbered.
http://carfax.org.uk/ | My life is my own."
PGP: 65E74AC0 |Number 6, The Prisoner
signature.asc
Description: Digital signature
Re: moving a subvol
On Fri, Dec 13, 2013 at 11:31:47AM -0500, Gene Czarcinski wrote: > On 12/13/2013 11:07 AM, Gene Czarcinski wrote: > >I have hunted high and low for the information but, if it is out there, > >it is well hidden. > > > >What I want to do: I want to relocate (move) a subvolume from one BTRFS > >volume to a different BTRFS volume. > > > >I have seen the writeup for incremental backup which takes a snapshot, > >and then using btrfs-send and btrfs-receive to create a copy on a > >different volume. That works fine except the btrfs-receive set the new > >copy to be readonly. > > > >What I want to do is to use that new subvolume read/write. The only way > >I have figured out to do this is to create another snapshot on the new > >volume which is NOT readonly. I believe that means I need to keep both > >of these subvolumes around. > > As soon as I sent this I believe I found the answer. > > I was sort of right about creating the new read-write (NOT readonly) > snapshot. After it is created simply delete the one which was created by > btrfs-receive. ATM, this is the way to do it. There's an ioctl to toggle the RO/RW status of a subvolume, so it could save you one snapshot, but there's not tool support to do that. I hope this will be possible with the upcomming 'per-object properties' feature. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving a subvol
On Dec 15, 2013, at 10:40 AM, Gene Czarcinski wrote: > On 12/14/2013 01:43 PM, Chris Murphy wrote: >> On Dec 14, 2013, at 2:57 AM, Gene Czarcinski wrote: >>> Since I run Fedora with anaconda I use kickstart installs and can easily >>> repeat an install since it included almost everything I want installed. >>> And then I have a post-install script I run to pickup additional stuff. >> This is a bit of a hijack question, but does kickstart offer a way to mount >> Btrfs with compression option? >> >> >> Chris Murphy > Interesting; it does not "currently" but that sounds like it should be (RFE > time) an added option so that it would be in fstab from the start. It may > not be that simple since if you specify compression for a subvolume, that > means that everything anaconda installs on the subvolume should be subjected > to compression and not just stuff written post-install. It needs to be mounted with the option prior to installation, and also needs to be added to the fstab. > > I do not currently use compression but I should give it a try. > > Chris, since I know kickstart and you do not, I will BZ the RFE. BTW, is this > an option on the regular GUI install? No, and I'm not finding it in any of the code for kickstart or anaconda GUI. My recollection is that if it was brought into the GUI it wasn't going to offer any choice of compression algorithm, it would use the Btrfs default. Currently that's zlib, which I think maybe isn't the best default choice because while it offers higher compression ratio over lzo, it's slower to compress and decompress. HDD space is cheap, the benefit of compression isn't so much saving space as it is to make r/w faster. In all of my tests LZO does this better. I *think* Snappy is better still than LZO, but that's not yet an option. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving a subvol
On 12/14/2013 01:43 PM, Chris Murphy wrote: On Dec 14, 2013, at 2:57 AM, Gene Czarcinski wrote: Since I run Fedora with anaconda I use kickstart installs and can easily repeat an install since it included almost everything I want installed. And then I have a post-install script I run to pickup additional stuff. This is a bit of a hijack question, but does kickstart offer a way to mount Btrfs with compression option? Chris Murphy Interesting; it does not "currently" but that sounds like it should be (RFE time) an added option so that it would be in fstab from the start. It may not be that simple since if you specify compression for a subvolume, that means that everything anaconda installs on the subvolume should be subjected to compression and not just stuff written post-install. I do not currently use compression but I should give it a try. Chris, since I know kickstart and you do not, I will BZ the RFE. BTW, is this an option on the regular GUI install? Gene -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving a subvol
On Dec 14, 2013, at 2:57 AM, Gene Czarcinski wrote: > Since I run Fedora with anaconda I use kickstart installs and can easily > repeat an install since it included almost everything I want installed. And > then I have a post-install script I run to pickup additional stuff. This is a bit of a hijack question, but does kickstart offer a way to mount Btrfs with compression option? Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving a subvol
On 12/14/2013 02:07 AM, Chris Samuel wrote: On Fri, 13 Dec 2013 02:25:34 PM Gene Czarcinski wrote: Next it is to delete the old BTRFS volume, use fdisk to increase the size of the partition and then attempt to increate the size of this new volume. I hope I don't get bit in the rear end with this. Just make sure you have good (tested) backups.. Data subvolumes such as for /home have separate backups but the rootfs subvolumes do not. If somethings gets screwed up, it is a matter of reinstalling. Since I run Fedora with anaconda I use kickstart installs and can easily repeat an install since it included almost everything I want installed. And then I have a post-install script I run to pickup additional stuff. After successfully resizing, I then repeated with a two-device BTRFS volume. Here the resize needs to be done for each device. Question: What is did involved two BTRFS volumes on /dev/sdb1 and /dev/sdb2. I deleted /dev/sdb2 and then expanded/resized /dev/sdb1. This worked but I assume that doing the opposite would not work. That is, deleteing the "lower" /dev/sdb1 and then expanding the "upper" /dev/sdb2 because after rebooting the filesystem would not be at the beginning of the partition. Gene -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving a subvol
On Fri, 13 Dec 2013 02:25:34 PM Gene Czarcinski wrote: > Next it is to delete the old BTRFS volume, use fdisk to increase the > size of the partition and then attempt to increate the size of this new > volume. I hope I don't get bit in the rear end with this. Just make sure you have good (tested) backups.. -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP signature.asc Description: This is a digitally signed message part.
Re: moving a subvol
On 12/13/2013 01:02 PM, Chris Murphy wrote: On Dec 13, 2013, at 10:37 AM, Cobin Bluth wrote: That seems a little less intuitive than one would think. I wonder if there would an argument to specify read-write for btrfs-receive. I'm going to guess it's on purpose to prevent writes from happening in the subvolume while it's still being received. The send file is a stream so my interpretation of what's going on is that subvolume's file system isn't really in a state to be modified until it's completely done being written. And for that matter, it's the same thing with sending. The wiki says send requires subvolumes to be ro, and all of the examples there are ro subvolumes. I don't know that this is an apt analogy, but think of dd'ing a partition that contains a live mounted volume. Oops - not a good idea right? Because as you dd the first part, the uncopied portion is being modified and is now inconsistent. Seems to have worked fine. What more could I ask for. With Fedora 20 a go for gold, I decided to update to Fedora 20 on almost all of my systems. At the same time I decided to do some consolidation and restructuring of my disk layouts to put almost all of it under some BTRFS volumes. I really did not want to install Fedora 19 again and just want to move the F19 rootfs from one volume to a newer other volume. It required some editing of /etc/fstab to use the new UUIDs and also some manual editing (ugh) of grub.cfg so that it had the correct UUIDs for the volume with subvol=root4. Next it is to delete the old BTRFS volume, use fdisk to increase the size of the partition and then attempt to increate the size of this new volume. I hope I don't get bit in the rear end with this. Gene -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving a subvol
On Dec 13, 2013, at 10:37 AM, Cobin Bluth wrote: > That seems a little less intuitive than one would think. I wonder if there > would an argument to specify read-write for btrfs-receive. I'm going to guess it's on purpose to prevent writes from happening in the subvolume while it's still being received. The send file is a stream so my interpretation of what's going on is that subvolume's file system isn't really in a state to be modified until it's completely done being written. And for that matter, it's the same thing with sending. The wiki says send requires subvolumes to be ro, and all of the examples there are ro subvolumes. I don't know that this is an apt analogy, but think of dd'ing a partition that contains a live mounted volume. Oops - not a good idea right? Because as you dd the first part, the uncopied portion is being modified and is now inconsistent. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving a subvol
On Dec 13, 2013, at 9:31 AM, Gene Czarcinski wrote: > On 12/13/2013 11:07 AM, Gene Czarcinski wrote: >> I have hunted high and low for the information but, if it is out there, >> it is well hidden. >> >> What I want to do: I want to relocate (move) a subvolume from one BTRFS >> volume to a different BTRFS volume. >> >> I have seen the writeup for incremental backup which takes a snapshot, >> and then using btrfs-send and btrfs-receive to create a copy on a >> different volume. That works fine except the btrfs-receive set the new >> copy to be readonly. >> >> What I want to do is to use that new subvolume read/write. The only way >> I have figured out to do this is to create another snapshot on the new >> volume which is NOT readonly. I believe that means I need to keep both >> of these subvolumes around. > > As soon as I sent this I believe I found the answer. > > I was sort of right about creating the new read-write (NOT readonly) > snapshot. After it is created simply delete the one which was created by > btrfs-receive. > > I also found out that if you want to rename a subvolume, siple do it with > "mv" like you would for a directory or a file. > > OK folks, do I have it correct? Yes. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: moving a subvol
On 12/13/2013 11:07 AM, Gene Czarcinski wrote: I have hunted high and low for the information but, if it is out there, it is well hidden. What I want to do: I want to relocate (move) a subvolume from one BTRFS volume to a different BTRFS volume. I have seen the writeup for incremental backup which takes a snapshot, and then using btrfs-send and btrfs-receive to create a copy on a different volume. That works fine except the btrfs-receive set the new copy to be readonly. What I want to do is to use that new subvolume read/write. The only way I have figured out to do this is to create another snapshot on the new volume which is NOT readonly. I believe that means I need to keep both of these subvolumes around. As soon as I sent this I believe I found the answer. I was sort of right about creating the new read-write (NOT readonly) snapshot. After it is created simply delete the one which was created by btrfs-receive. I also found out that if you want to rename a subvolume, siple do it with "mv" like you would for a directory or a file. OK folks, do I have it correct? Gene -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving over to use BTRFS - first time, needs encryption
btrfs.nrb posted on Tue, 29 Oct 2013 15:19:19 + as excerpted: > Hi, I would be pleased to get some help after I have looked and not > figured out how this should work: > > Summary === > btrfs device add > Returns - Inappropriate ioctl for device > # btrfs --version > Btrfs Btrfs v0.19 > # uname -a > Linux 874-deb 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux [I've no particular personal knowledge on encrypted filesystem usage, but this is a general reply...] First, are you aware of the btrfs wiki and have you read its encryption discussion (among other things)? https://btrfs.wiki.kernel.org Second, if you've read the wiki you know this already, but it's worth repeating... btrfs is still under heavy development and still labeled experimental. As such, if you're running btrfs you are by definition testing an experimental filesystem and should consider data reliability in that regard. IOW, KEEP TESTED BACKUPS AND BE PREPARED TO USE THEM should your btrfs tests go badly! Further, every new kernel brings important fixes to known issues, and anything older than two kernel series old is ANCIENT. Ideally, you're on the latest stable kernel release at the OLDEST, and may well be running mainline kernel rcs if not even the btrfs-next patch-set. (Personally, I run a mainline git kernel, but don't normally update to the development kernel until after rc1, but try to get to it before rc3, so if I notice any regressions I can file bugs with time to have them fixed before release.) Run anything older and not only are you very literally needlessly risking your data on known bugs that are already fixed, but if you do catch a bug and report it, your reports will be less useful because your kernel is so old. For btrfs testers, kernel 3.2 is "ancient as the hills!" There are reasons people might want to be conservative and run older kernels, but they really aren't compatible with the reasons people would want to test new and potentially unstable filesystems such as btrfs. Please choose one or the other, and if you're going to test btrfs, do so with a current kernel, currently meaning a post-3.11.5 stable series kernel at the oldest as I believe it had some critical btrfs patches, or 3.12 release cadidates or live-git, since 3.12 is very close to release now. Similarly but not /quite/ as critical, btrfs-progs v0.19 is /old/. Even v0.20-rc1 is from late last year, IIRC, and according to the version string I have here[1], 358 commits behind current. btrfs-progs development policy is to keep the git master branch "release ready" and do actual development in other branches which are only merged when they're considered ready, so running "git-master" of btrfs-progs, updated weekly or no less than monthly, should be your best and safest option. --- [1] Btrfs-progs here, live-git updated less than a week ago, tho from memory the latest commit was back in July or so: btrfs --version Btrfs v0.20-rc1-358-g194aa4a -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving over to use BTRFS - first time, needs encryption (btrfs: message 1 of 20)
Update: I found this: .. by upgrading kernel ...according to the BTRFS wiki, changing raid levels after the filesystem has been created isn't supported for 3.2 series and older kernels. Need to move from stable stock., > Ubuntu 12.04LTS and Debian 7.2. http://serverfault.com/questions/496503/converting-btrfs-filesystem-to-raid1-fails-with-inappropriate-ioctl-for-device -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving...
On 04/02/2013 07:14 PM, Swâmi Petaramesh wrote: > Le 02/04/2013 19:04, Roman Mamedov a écrit : >> but at this point I trust my data to BTRFS more, than I would trust ZFS. > > My experience with ZFS on Linux is still somewhat limited, the only > thing that I can say is that I've used it for about 2 years and a half > on my (quite loaded) home server without ever encoutering a single issue > (besides some tricky Ubuntu version upgrades, only due to the fact that > ZFS is "not in the standard kernel" by opposition to BTRFS). > > On the BTRFS side, I've used it on 5 machine for about a year on a daily > basis ; I've completely lost a filesystem, beyond repair, twice ; I > found myself with a machine that wouldn't boot anymore and would > necessitate serious and tricky FS maintenance, 3 times, I had to > completely reinstall the FS because of extremely degraded performance, 4 > times. > > Add to this that the FS management tools and properly working FS > features are an order of magnitude better in ZFS than in BTRFS... > > That's enough for me... With these experiences, I am not surprise that you changed the filesystem. However I would like to report my experience which is quite positive regarding btrfs. I used BTRFS as root filesystem for about two-three years. And I encountered two filesystem corruption. The first one was due to a bug in BTRFS. I don't remember the details, it was due to a problem with the linking a file between two subvolumes. However the filesystem worked well, the only problem which I had was the in-ability to remove a file (the link). The second problem was due a hardware fault: the power supply was insufficient, so sometime the hard disk returned errors. With scrub I was able to find the files corrupted and to recreate the metadata (which in BTRFS is DUP-ed). This problem happened several time before I replaced the power supply; and the filesystem worked well (with the exception of the corrupted file of course). May be that I was very lucky or you was very un-lukcy. I hope that with ZFS you will be able to solve all your problems. > > I've already converted 2 machines from BTRFS to ZFS since yesterday. > > My netbook (now ZFS) boots to GDM in less than 30 seconds. My son's same > machine (except for the FS) takes 75 MORE seconds. The BTRFS performance are not the best, but good enough. May be that BTRFS require more memory than other filesystem, so it is not good for old computer. For my old laptop I had to revert to ext4. However I have to point out that am not a fan of the test "time to login": I prefer the responsiveness of the system, and with the latest kernel (IIRC from 3.6/3.7 onwards) this was a lot improved. > > You get it. > > Kind regards. > -- gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [zfs-discuss] Re: Moving...
Le 03/04/2013 10:47, Uncle Stoatwarbler a écrit : > my (5 year old) laptop with a relatively slow SSD in it boots to GDM > (Mint LXDE) in about 5 seconds with ext4. A 9 year old laptop fitted > with a very slow PATA SSD takes about 9 seconds to do the same. I've been using Linux daily since 1996, all distros, on every imaginable hardware (but not on modern SSDs). I've never seen such performance anywhere. Would be extremely curious about the distro, filesystem and setup of the "9 year old laptop fitted with a very slow PATA SSD" that boots to GDM in 9 seconds... -- Swâmi Petaramesh http://petaramesh.org PGP 9076E32E Ne cherchez pas : Je ne suis pas sur Facebook. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [zfs-discuss] Re: Moving...
On 02/04/13 18:14, Swâmi Petaramesh wrote: My netbook (now ZFS) boots to GDM in less than 30 seconds. My son's same machine (except for the FS) takes 75 MORE seconds. my (5 year old) laptop with a relatively slow SSD in it boots to GDM (Mint LXDE) in about 5 seconds with ext4. A 9 year old laptop fitted with a very slow PATA SSD takes about 9 seconds to do the same. I'm not so sure that ZFS is a good fit for single drive machines ((Nor is ext4 for SSDs for that matter), but I'm happy to be corrected. What I do know from banging hard on both is that I'd far sooner trust ZFS in an enterprise environment than BTRFS (or XFS), because I'm not really that happy about the idea of having to periodically restore 500Tb+ of data from backups. Speed is a secondary issue, but ZFS is definitely faster at large scale. There's a lot of wheel reinvention going on and I feel there must be some way of merging the good parts of both sets of code, then moving on. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving...
On Tue, Apr 2, 2013 at 7:14 PM, Swâmi Petaramesh wrote: > Le 02/04/2013 19:04, Roman Mamedov a écrit : >> but at this point I trust my data to BTRFS more, than I would trust ZFS. > > My experience with ZFS on Linux is still somewhat limited, the only > thing that I can say is that I've used it for about 2 years and a half > on my (quite loaded) home server without ever encoutering a single issue > (besides some tricky Ubuntu version upgrades, only due to the fact that > ZFS is "not in the standard kernel" by opposition to BTRFS). > > On the BTRFS side, I've used it on 5 machine for about a year on a daily > basis ; I've completely lost a filesystem, beyond repair, twice ; I > found myself with a machine that wouldn't boot anymore and would > necessitate serious and tricky FS maintenance, 3 times, I had to > completely reinstall the FS because of extremely degraded performance, 4 > times. > > Add to this that the FS management tools and properly working FS > features are an order of magnitude better in ZFS than in BTRFS... > > That's enough for me... > > I've already converted 2 machines from BTRFS to ZFS since yesterday. > > My netbook (now ZFS) boots to GDM in less than 30 seconds. My son's same > machine (except for the FS) takes 75 MORE seconds. > > You get it. > > Kind regards. > > -- > Swâmi Petaramesh http://petaramesh.org PGP 9076E32E > Ne cherchez pas : Je ne suis pas sur Facebook. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html I don't find it too suprising that you have more problems with a filesystem that is not final than with one that has seen several years of post-final improvements. I would be shocked if ZFS were in worse shape than btrfs right now. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving...
Le 02/04/2013 19:04, Roman Mamedov a écrit : > but at this point I trust my data to BTRFS more, than I would trust ZFS. My experience with ZFS on Linux is still somewhat limited, the only thing that I can say is that I've used it for about 2 years and a half on my (quite loaded) home server without ever encoutering a single issue (besides some tricky Ubuntu version upgrades, only due to the fact that ZFS is "not in the standard kernel" by opposition to BTRFS). On the BTRFS side, I've used it on 5 machine for about a year on a daily basis ; I've completely lost a filesystem, beyond repair, twice ; I found myself with a machine that wouldn't boot anymore and would necessitate serious and tricky FS maintenance, 3 times, I had to completely reinstall the FS because of extremely degraded performance, 4 times. Add to this that the FS management tools and properly working FS features are an order of magnitude better in ZFS than in BTRFS... That's enough for me... I've already converted 2 machines from BTRFS to ZFS since yesterday. My netbook (now ZFS) boots to GDM in less than 30 seconds. My son's same machine (except for the FS) takes 75 MORE seconds. You get it. Kind regards. -- Swâmi Petaramesh http://petaramesh.org PGP 9076E32E Ne cherchez pas : Je ne suis pas sur Facebook. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving...
On Tue, 02 Apr 2013 10:24:39 +0200 Swâmi Petaramesh wrote: > Goodbye BTRFS, hello ZFS :-) > > I'm finally making the move, I couldn't stand the terrible BTRFS > performance anymore, and spending 2 long minutes waiting for the HD LED > to come off everytime I clicked anywhere. > > Did what I could, got the latest kernels, defragged, removed BTRFS > snapshots, to no avail... > > So I'm in process of moving the 3 main laptops in here from BTRFS to ZFS... Whatever works for you -- but at this point I trust my data to BTRFS more, than I would trust ZFS. (of course having current backups etc). The following may not sound like a compliment to BTRFS; but on my filesystems with heavy snapshotting, recently I am trialling a practice to run major apps like a mail client and a browser via "eatmydata", which overrides and NOOPs fsync. Since in any case I have "sync; btrfs sub snap" on those machines set to every 30 minutes in crontab, the potential of data loss due to "eatmydata" is never too great. I noticed the general experience is a lot snappier this way. You could try at least to verify whether or not your HD LED issues (:D) are indeed caused by slow fsyncs, or by something else. I should note that I currently use a 3.7 series kernel, and haven't upgraded to 3.8 and 3.9 yet, AFAIK those had some fsync improvements which might render this trick useless. But this still doesn't change the fact that apps in general seem to over-use fsync. -- With respect, Roman signature.asc Description: PGP signature
Re: Moving top level to a subvolume
Fajar A. Nugraha posted on Wed, 13 Jun 2012 16:08:40 +0700 as excerpted: >> My system's old and has a bit of a problem with overheating in the >> Phoenix summer, so has been suffering SATA resets > >> it's exactly this sort of corner-case that filesystems need to be able >> to deal with > > IIRC XFS had corruption problems when used on top of LVM (or other block > device that doesn't support barriers correctly), while using ext2/3/4 on > the same block device will be "fine". Yet XFS doesn't have the mark of > "unstable, highly experimental, do not use". People simply use the right > (for them) fs for the right job. It /does/ have a reputation of "don't use for a normal home system or other location without a suitably dependable UPS on fully stable hardware", however. (From what I've read however, much like reiserfs has been for me here after data=ordered, xfs is vastly improved now, and said reputation likely no longer applies.) If btrfs is to replace ext3/4, then that sort of reputation isn't what its devs will be shooting for. We have at least the two filesystems (reiserfs and xfs) with serious reputation problems from their earlier life, and my big concern is that if enough people fail to consider btrfs' current development state and end up with data loss as a result, btrfs will end up with a very similar reputation, which will similarly live many years longer than the reality that created it. But the other two weren't shooting for the ext* mantle, and btrfs is. If its reputation is damaged to that extent and it becomes the assumed Linux default as ext3/4 is now, that will be the Linux reputation. > My point is yes, btrfs is new. And it's being developed at much faster > rate than any other more-mature fs out there. And there are known cases > of data loss on certain configuration of corner cases/"buggy" hardware > and/or old version of kernel. But when used in the correct environment, > btrfs can be a good choice, even for critical data. Of course, critical data is backed up, or by definition you don't REALLY consider it so critical after all. (There was a time when drives were small enough and expensive enough, as were the alternatives, that wasn't the case. That time is long gone, for first and second world usage, anyway. Even a home user with a single drive can split it into multiple partitions, with backup data on separate partitions, at least, with the real critical data on a thumb-drive too.) But while people can and do unfortunately go without backups and are often lucky, doing so on btrfs at this stage in its development is "doubly insane", to channel Linus (actually being nice that day =:^) describing a recent commit, in his revert. But there's obviously "doubly insane" people out there! =:^\ > Of course IF the data were REALLY critical, and I REALLY need btrfs' > features, and it were on an enterprise environment, I would've bought > support from oracle linux (or SLES 12, when it's out, or whatever > enterprise distro supporting btrfs which sells support contract) so I > can have someone to turn to in case of problems, and (in some cases) > transfer the risk/blame :D That's a good point. Of course, as many people find out, such "support" often /does/ really come down to "someone else to blame". Yes, they'll help with recovery if it comes to it, but if your $100K+ an hour business is down in the mean time... and you didn't have those backups and at /least/ "cold" failover... they'll be finding someone to blame as well! -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On 06/13/2012 09:21 AM, Arne Jansen wrote:
> On 13.06.2012 09:04, C Anthony Risinger wrote:
>> On Fri, Jun 8, 2012 at 2:40 PM, Arne Jansen wrote:
>>> On 06/08/2012 09:24 PM, Matthew Hawn wrote:
I just converted my root filesystem to btrfs with btrfs-convert. However,
since I am running Ubuntu, I would like to have the same subvolume
structure as a default install,. How do I move the top-level subvolume
(where all my files currently are) to another subvolume?
>>>
>>> Just snapshot the root subvol and continue working in the snapshot.
>>
>> ... yeah but that solution totally sucks when you:
>>
>> a) have a lot of data
>> b) need to do this via script
>> c) ???
>>
>> ... because in a), data will *copied* the slow way, and in b) you
>> leave a bunch of junk laying around in the old root that will rot
>> unless you `rm -rf` it ... and idk about you, but issuing what is very
>> near to that command on someone else's machine -- via script -- makes
>> me REALLY uneasy ;-)
>
> well, don't put data in the top level in the first place. Yes, you have
> to remove the content of the subvol / by rm -rf, but I don't really see
> the problem with it.
It is slow. You have to change a lot of metadata (each shared metadata
block have to be unshared, and then one copy will be deleted ).
> What I don't understand is why you think data will be copied.
>
>>
>> i have asked this exact question at least 4 times specifically, and
>> referenced it probably 8-10, in the last 3 years or more. i needed it
>> then. i still need it now. but since i never got an answer up/down
>> or around, i gave up and told people to `rm -rf`themselves ...
>>
>> http://markmail.org/message/7hj5ioqrztkeerqv
>>
>> ... that's from May of 2010, but i don't think it's the first.
>>
>> so, would it possible to implement this, or could someone kindly (and
>> briefly!) explain why it cannot be done?
>
> The default subvol ('/') has the special number 5 and is expected to
> always be around. All other subvols get numbers starting with 256.
> Creating a new 5 and internally renumbering the old 5 isn't easy, because
> each tree block has an owner recorded in it. Also, all backreferences
> have the root number in them. If you have to touch each tree block, you
> can as well choose the snapshot/rm -rf approach.
I don't know very well the internal of btrfs. Do you think that It is
possible to move/swap the root subvolume ?
>
>>
[...]
> Or you could hack mkfs.btrfs to always create an additional subvol.
Which can be the default one: so nobody should complain. I
> Even making / readonly except for creating mountpoint could be possible.
> Just some random ideas...
>
> -Arne
>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> .
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On Wed, Jun 13, 2012 at 4:44 PM, C Anthony Risinger wrote: > On Wed, Jun 13, 2012 at 2:21 AM, Arne Jansen wrote: >> On 13.06.2012 09:04, C Anthony Risinger wrote: >>> ... because in a), data will *copied* the slow way >> What I don't understand is why you think data will be copied. > at one point i tried to create a new subvol and `mv` files there, and > it took quite some time to complete > (cross-link-device-what-have-you?), but maybe things changed ... will > try it out. IIRC it hasn't. Not in upstream anyway. Some distros (e.g. opensuse) carry their own patch which allows cross-subvolume links (cp --reflink ...). But it shouldn't matter anyway, since you can SNAPSHOT the old subvol (even root subvol), instead of creating a new subvol. Which means nothing needs to be copied. You'd still have to do "rm" manually though. -- Fajar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On Wed, Jun 13, 2012 at 2:21 AM, Arne Jansen wrote:
> On 13.06.2012 09:04, C Anthony Risinger wrote:
>>
>> a) have a lot of data
>> b) need to do this via script
>> c) ???
>>
>> ... because in a), data will *copied* the slow way, and in b) you
>> leave a bunch of junk laying around in the old root that will rot
>> unless you `rm -rf` it ... [...]
>
> What I don't understand is why you think data will be copied.
at one point i tried to create a new subvol and `mv` files there, and
it took quite some time to complete
(cross-link-device-what-have-you?), but maybe things changed ... will
try it out.
>> [...]
>>
>> so, would it possible to implement this, or could someone kindly (and
>> briefly!) explain why it cannot be done?
>
> The default subvol ('/') has the special number 5 and is expected to
> always be around. All other subvols get numbers starting with 256.
> Creating a new 5 and internally renumbering the old 5 isn't easy, because
> each tree block has an owner recorded in it. Also, all backreferences
> have the root number in them. If you have to touch each tree block, you
> can as well choose the snapshot/rm -rf approach.
ok this makes sense thanks, the last sentence especially ... top-level
volume is different. it's identical to other subvols in 99% of ways
save one-gotcha-little-1%. couldn't we shield ourselves a little
better?
>> 1. people install stuff to the top-level
>> 2. top-level is unmanageable
>> 3. problem
>> [...]
>
> Can't instead add code to the installer that warns a user if he wants
> to install into the default subvol?
> Just some random ideas...
i would like to see #5 cut off from natural access: accessible by an
_explicit_ manual mount only, cannot be made default, and cannot be
removed. maybe btrfs manages a proxy/facade subvol, say, #10, settable
by `--flag-origin` or `{insert-here}` option -- a symlink to subvol?
if, at absolutely any time or whatever reason, #10 pointer should not
exist, immediately snapshot #5 and update.
#5 -> #10 -> #256+ ?
... this might allow the root to be "replaced". default is set to #10
proxy volume when FS is initialized.
> [...]
> Or you could hack mkfs.btrfs to always create an additional subvol.
> Even making / readonly except for creating mountpoint could be possible.
^ yeah, this sounds like exactly what i'm thinking, differing
mainly on who does the work... i just want a guaranteed way of
replacing the "logical root", at #10. the "physical root" at #5 it's
more-or-less indestructible and off limits, and never available except
as a template.
... i am new to postgresql, but their template0/template1 feels
related to solving problems like this.
--
C Anthony
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On Wed, Jun 13, 2012 at 2:23 PM, Duncan <1i5t5.dun...@cox.net> wrote: > Fajar A. Nugraha posted on Wed, 13 Jun 2012 08:49:47 +0700 as excerpted: > >> As for "lose their filesystems", are there recent ones that uses one of >> the three distros above, and is purely btrfs "fault"? The ones I can >> remember (from the post to this list) were broken on earlier kernels, or >> caused by bad disks. > My system's old and has a bit of a problem with overheating in the > Phoenix summer, so has been suffering SATA resets > it's exactly this sort of > corner-case that filesystems need to be able to deal with IIRC XFS had corruption problems when used on top of LVM (or other block device that doesn't support barriers correctly), while using ext2/3/4 on the same block device will be "fine". Yet XFS doesn't have the mark of "unstable, highly experimental, do not use". People simply use the right (for them) fs for the right job. My point is yes, btrfs is new. And it's being developed at much faster rate than any other more-mature fs out there. And there are known cases of data loss on certain configuration of corner cases/"buggy" hardware and/or old version of kernel. But when used in the correct environment, btrfs can be a good choice, even for critical data. Of course IF the data were REALLY critical, and I REALLY need btrfs' features, and it were on an enterprise environment, I would've bought support from oracle linux (or SLES 12, when it's out, or whatever enterprise distro supporting btrfs which sells support contract) so I can have someone to turn to in case of problems, and (in some cases) transfer the risk/blame :D -- Fajar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
Fajar A. Nugraha posted on Wed, 13 Jun 2012 08:49:47 +0700 as excerpted: > As for "lose their filesystems", are there recent ones that uses one of > the three distros above, and is purely btrfs "fault"? The ones I can > remember (from the post to this list) were broken on earlier kernels, or > caused by bad disks. I tried btrfs during the 3.4 cycle for a bit, and didn't lose the whole filesystem, but definitely found it not upto my usual standard of robustness, my previous and back to now filesystem, Chris's former project, reiserfs. My system's old and has a bit of a problem with overheating in the Phoenix summer, so has been suffering SATA resets (not the disk, the sata chipset most likely, and/or issues with the graphics overheating since I'm using an AMD 8xxx chipset with AGPGART split between IOMMU for storage I/O and graphics) and full system freezes. Not only did I have way more stuff disappearing or being zeroed out than on reiserfs (in default data=ordered mode), but in one case I had a segment disappear out of the middle of a file, and in another, I had firefox's crash-resume-file /content/ show up as what SHOULD have been an entirely unrelated configuration file. Naturally I had backups to restore from, and if it wasn't for the freezes, it would have likely been fine, but it's exactly this sort of corner-case that filesystems need to be able to deal with, and what bothered me wasn't disappearing or zeroed out last few seconds of work with well documented explanations, but having random segments of files that I hadn't changed (whether the app was rewriting them with the same data's another question) in some time disappear, and having one file's content show up with an entirely unrelated name. I thought that's the sort of thing btrfs checksums were supposed to detect and effectively zero out, but... I decided that's /too/ experimental for me ATM, especially with not-quite- stable hardware (it's worth noting that I survived bad memory and the related crashes on reiserfs, without /that/ sort of damage, at least not since data=ordered mode!), so am back on reiserfs for now, anyway. I'll likely try again next year sometime... -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On 13.06.2012 09:04, C Anthony Risinger wrote:
> On Fri, Jun 8, 2012 at 2:40 PM, Arne Jansen wrote:
>> On 06/08/2012 09:24 PM, Matthew Hawn wrote:
>>> I just converted my root filesystem to btrfs with btrfs-convert. However,
>>> since I am running Ubuntu, I would like to have the same subvolume
>>> structure as a default install,. How do I move the top-level subvolume
>>> (where all my files currently are) to another subvolume?
>>
>> Just snapshot the root subvol and continue working in the snapshot.
>
> ... yeah but that solution totally sucks when you:
>
> a) have a lot of data
> b) need to do this via script
> c) ???
>
> ... because in a), data will *copied* the slow way, and in b) you
> leave a bunch of junk laying around in the old root that will rot
> unless you `rm -rf` it ... and idk about you, but issuing what is very
> near to that command on someone else's machine -- via script -- makes
> me REALLY uneasy ;-)
well, don't put data in the top level in the first place. Yes, you have
to remove the content of the subvol / by rm -rf, but I don't really see
the problem with it.
What I don't understand is why you think data will be copied.
>
> i have asked this exact question at least 4 times specifically, and
> referenced it probably 8-10, in the last 3 years or more. i needed it
> then. i still need it now. but since i never got an answer up/down
> or around, i gave up and told people to `rm -rf`themselves ...
>
> http://markmail.org/message/7hj5ioqrztkeerqv
>
> ... that's from May of 2010, but i don't think it's the first.
>
> so, would it possible to implement this, or could someone kindly (and
> briefly!) explain why it cannot be done?
The default subvol ('/') has the special number 5 and is expected to
always be around. All other subvols get numbers starting with 256.
Creating a new 5 and internally renumbering the old 5 isn't easy, because
each tree block has an owner recorded in it. Also, all backreferences
have the root number in them. If you have to touch each tree block, you
can as well choose the snapshot/rm -rf approach.
>
> 1. people install stuff to the top-level
> 2. top-level is unmanageable
> 3. problem
>
> in my case i wrote an initramfs hook that implemented rollback
> functionality, but there was not way for me to cleanly -- and safely
> -- "rotate" the user's setup to one that DOES NOT have user items in
> the top-level volume.
Can't instead add code to the installer that warns a user if he wants
to install into the default subvol?
Or you could hack mkfs.btrfs to always create an additional subvol.
Even making / readonly except for creating mountpoint could be possible.
Just some random ideas...
-Arne
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On Fri, Jun 8, 2012 at 2:40 PM, Arne Jansen wrote: > On 06/08/2012 09:24 PM, Matthew Hawn wrote: >> I just converted my root filesystem to btrfs with btrfs-convert. However, >> since I am running Ubuntu, I would like to have the same subvolume structure >> as a default install,. How do I move the top-level subvolume (where all my >> files currently are) to another subvolume? > > Just snapshot the root subvol and continue working in the snapshot. ... yeah but that solution totally sucks when you: a) have a lot of data b) need to do this via script c) ??? ... because in a), data will *copied* the slow way, and in b) you leave a bunch of junk laying around in the old root that will rot unless you `rm -rf` it ... and idk about you, but issuing what is very near to that command on someone else's machine -- via script -- makes me REALLY uneasy ;-) i have asked this exact question at least 4 times specifically, and referenced it probably 8-10, in the last 3 years or more. i needed it then. i still need it now. but since i never got an answer up/down or around, i gave up and told people to `rm -rf`themselves ... http://markmail.org/message/7hj5ioqrztkeerqv ... that's from May of 2010, but i don't think it's the first. so, would it possible to implement this, or could someone kindly (and briefly!) explain why it cannot be done? 1. people install stuff to the top-level 2. top-level is unmanageable 3. problem in my case i wrote an initramfs hook that implemented rollback functionality, but there was not way for me to cleanly -- and safely -- "rotate" the user's setup to one that DOES NOT have user items in the top-level volume. -- C Anthony -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On Tue, Jun 12, 2012 at 9:52 PM, Randy Barlow wrote: > I personally run Gentoo, but I've been told by some coworkers that the Ubuntu > installer offers btrfs as an option to the users without marking it as > experimental, unstable, or under development. I wonder if that is why we see > so many people surprised when they lose their filesystems. Can anyone verify > whether that is true of Ubuntu, or of any other Linux distributions? Oracle linux (when used with UEK2) officially supports btrfs. Opensuse also supports btrfs, and use its functionality for snapper. I haven't found any updated (i.e. released post 12.04) official support status statement from Ubuntu, but they do offer btrfs as installation option. As for "lose their filesystems", are there recent ones that uses one of the three distros above, and is purely btrfs "fault"? The ones I can remember (from the post to this list) were broken on earlier kernels, or caused by bad disks. -- Fajar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
The Ubuntu wiki does not(in a straight-forward way) say BTRFS is experimental and unstable. It looks like they copied and pasted from the BTRFS official Wiki. Link: https://help.ubuntu.com/community/btrfs I do not have an account on the BTRFS wiki, but I believe changing the first paragraphs to something like what I included at the bottom. The big change is that we need to specifically and in big bold letters say that BTRFS is experimental and unstable as well as outline that BTRFS shouldn't be used on data where loss would cause headaches. I don't think that as is we draw enough attention to these facts. We are assuming that by saying(on the Wiki) "Btrfs is under heavy development" people will realize that it is experimental and unstable. Right after that we say that "every effort is being made to keep the filesystem stable and fast" which reads more like, "we are making every effort to KEEP it stable and fast". This ambiguity would lead users to believe that BTRFS is fast and stable(instead of that "we are attempting to have it be fast and stable") Btrfs is under heavy development, but while every effort is being made to keep the filesystem stable and fast, it is still marked as [bold][bold]experimental and unstable[/bold][/bold]. Btrfs should not yet be used on data where loss would cause headaches. Because of the speed of development, you should run the latest kernel you can (either the latest release kernel from kernel.org, or the latest -rc kernel. Please email the Btrfs mailing list if you have any problems or questions while using Btrfs. On Tue, Jun 12, 2012 at 9:52 AM, Randy Barlow wrote: > > On Tuesday, June 12, 2012 01:53:23 AM Duncan wrote: > > We get a lot of folks on this list who somehow miss the kernel warning, > > and the wiki warning, and the general community knowledge, that btrfs is > > still marked experimental and is still under heavy development. If > > something goes wrong, as it often has for these folks when they post > > here... > > I personally run Gentoo, but I've been told by some coworkers that the Ubuntu > installer offers btrfs as an option to the users without marking it as > experimental, unstable, or under development. I wonder if that is why we see > so many people surprised when they lose their filesystems. Can anyone verify > whether that is true of Ubuntu, or of any other Linux distributions? > > -- > R > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On Tuesday, June 12, 2012 01:53:23 AM Duncan wrote: > We get a lot of folks on this list who somehow miss the kernel warning, > and the wiki warning, and the general community knowledge, that btrfs is > still marked experimental and is still under heavy development. If > something goes wrong, as it often has for these folks when they post > here... I personally run Gentoo, but I've been told by some coworkers that the Ubuntu installer offers btrfs as an option to the users without marking it as experimental, unstable, or under development. I wonder if that is why we see so many people surprised when they lose their filesystems. Can anyone verify whether that is true of Ubuntu, or of any other Linux distributions? -- R -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
Matthew Hawn posted on Fri, 08 Jun 2012 12:24:26 -0700 as excerpted: > I just converted my root filesystem to btrfs with btrfs-convert. > However, > since I am running Ubuntu, I would like to have the same subvolume > structure as a default install,. How do I move the top-level subvolume > (where all my files currently are) to another subvolume? (Replied to list and to you, in case you're not subscribed.) Arne answered your question, but just let me add a caution. We get a lot of folks on this list who somehow miss the kernel warning, and the wiki warning, and the general community knowledge, that btrfs is still marked experimental and is still under heavy development. If something goes wrong, as it often has for these folks when they post here... So just be aware of that and be sure that if you're going to run it, you keep backups on something other than btrfs just in case, and that you keep them relatively current (depending on how much data from your btrfs systems you're willing to lose if it dies). Also, it's worthwhile to keep up with this list to see current developments and bugs, to run a current kernel, generally meaning latest Linus release either stable or rc, and if you've not read up on the wiki, read up on btrfs there, too. https://btrfs.wiki.kernel.org/ -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Moving top level to a subvolume
On 06/08/2012 09:24 PM, Matthew Hawn wrote: > I just converted my root filesystem to btrfs with btrfs-convert. However, > since I am running Ubuntu, I would like to have the same subvolume structure > as a default install,. How do I move the top-level subvolume (where all my > files currently are) to another subvolume? > Just snapshot the root subvol and continue working in the snapshot. > Matt > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
