Re: Task Hang
Not the same problem, but I do have a lockup with another situation.
I tried adding some new devices... but accidentally screwed up the
syntax (not sure if this had anything to do with the lockup)
btrfs device add / /dev/sdb
probe of / failed, cannot detect existing filesystem.
Use the -f option to force overwrite.
office-backup {~} root# btrfs device add /dev/sdb /dev/sdd
ERROR: error adding the device '/dev/sdb' - Inappropriate ioctl for device
D 1772 [btrfs-transacti]
D 6530 [btrfs-submit-2]
D 7301 /usr/bin/perl /usr/share/backuppc/bin/BackupPC_link demo3
D 7471 fdisk /dev/sdb
D 7523 [btrfs-submit-2]
D 7526 -bash
D 7527 -bash
D 28611 /usr/bin/perl /usr/share/backuppc/bin/BackupPC_trashClean
Attached is output from w to sysrq
On 03/04/2014 09:58 AM, Josef Bacik wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 03/04/2014 09:19 AM, Mark Murawski wrote:
I have btrfs as the fs for a backuppc box. updatedb was running at
the same time as a massive rsync.
Mar 4 08:31:00 office-backup kernel: INFO: task
updatedb.mlocat:903 blocked for more than 120 seconds. Mar 4
08:31:00 office-backup kernel: Not tainted 3.13.2 #3 Mar 4
08:31:00 office-backup kernel: echo 0
/proc/sys/kernel/hung_task_timeout_secs disables this message. Mar
4 08:31:00 office-backup kernel: updatedb.mlocat D
0 903899 0x Mar 4 08:31:00
office-backup kernel: 88007cad6270 0086
88007c0ee900 4000 Mar 4 08:31:00 office-backup
kernel: 88005a705fd8 88007cad6270 00541a6be000
88007b9ac000 Mar 4 08:31:00 office-backup kernel:
88001ad2ef68 880001f97000 8800452d4480
812c3e5c Mar 4 08:31:00 office-backup kernel: Call Trace:
Mar 4 08:31:00 office-backup kernel: [812c3e5c] ?
btrfs_map_bio+0x4ac/0x5a0 Mar 4 08:31:00 office-backup kernel:
[812b79c0] ? repair_io_failure+0x210/0x210 Mar 4
08:31:00 office-backup kernel: [810d9a40] ?
__lock_page+0x70/0x70 Mar 4 08:31:00 office-backup kernel:
[817b8177] ? io_schedule+0x87/0xd0 Mar 4 08:31:00
office-backup kernel: [810d9a49] ?
sleep_on_page+0x9/0x10 Mar 4 08:31:00 office-backup kernel:
[817b8742] ? __wait_on_bit+0x52/0x80 Mar 4 08:31:00
office-backup kernel: [81294d21] ?
btree_submit_bio_hook+0xe1/0x110 Mar 4 08:31:00 office-backup
kernel: [810d9cc3] ? wait_on_page_bit+0x73/0x80 Mar 4
08:31:00 office-backup kernel: [8109bdc0] ?
wake_atomic_t_function+0x30/0x30 Mar 4 08:31:00 office-backup
kernel: [812bb2ba] ?
read_extent_buffer_pages+0x2aa/0x2e0 Mar 4 08:31:00 office-backup
kernel: [810da355] ? add_to_page_cache_lru+0x25/0x40 Mar
4 08:31:00 office-backup kernel: [81344b21] ?
radix_tree_insert+0x91/0x250 Mar 4 08:31:00 office-backup kernel:
[81292a10] ? verify_parent_transid+0x170/0x170 Mar 4
08:31:00 office-backup kernel: [812949a9] ?
btree_read_extent_buffer_pages.constprop.126+0xa9/0x110 Mar 4
08:31:00 office-backup kernel: [81294f23] ?
read_tree_block+0x33/0x60 Mar 4 08:31:00 office-backup kernel:
[812775dc] ? read_block_for_search.isra.45+0x18c/0x3b0
Mar 4 08:31:00 office-backup kernel: [81276d47] ?
comp_keys+0x27/0x30 Mar 4 08:31:00 office-backup kernel:
[81279c8c] ? btrfs_search_slot+0x41c/0x920 Mar 4
08:31:00 office-backup kernel: [812b1641] ?
btrfs_get_token_16+0x61/0xf0 Mar 4 08:31:00 office-backup kernel:
[812919f5] ? btrfs_lookup_inode+0x25/0xa0 Mar 4 08:31:00
office-backup kernel: [81113ef0] ?
kmem_cache_alloc+0xc0/0xe0 Mar 4 08:31:00 office-backup kernel:
[812a6313] ? btrfs_iget+0x103/0x520 Mar 4 08:31:00
office-backup kernel: [8128eb1f] ?
btrfs_lookup_dir_item+0x9f/0xd0 Mar 4 08:31:00 office-backup
kernel: [812a843b] ? btrfs_lookup_dentry+0x3db/0x4c0 Mar
4 08:31:00 office-backup kernel: [812a8529] ?
btrfs_lookup+0x9/0x20 Mar 4 08:31:00 office-backup kernel:
[81120624] ? lookup_real+0x14/0x50 Mar 4 08:31:00
office-backup kernel: [811211f2] ?
__lookup_hash+0x32/0x50 Mar 4 08:31:00 office-backup kernel:
[81121b38] ? lookup_slow+0x48/0xc0 Mar 4 08:31:00
office-backup kernel: [81123a91] ?
path_lookupat+0x711/0x760 Mar 4 08:31:00 office-backup kernel:
[81123b0f] ? filename_lookup+0x2f/0xd0 Mar 4 08:31:00
office-backup kernel: [81121d77] ?
getname_flags+0xb7/0x190 Mar 4 08:31:00 office-backup kernel:
[811269ee] ? user_path_at_empty+0x5e/0xb0 Mar 4 08:31:00
office-backup kernel: [8111bd6d] ?
cp_new_stat+0x10d/0x120 Mar 4 08:31:00 office-backup kernel:
[8111bef1] ? vfs_fstatat+0x41/0x90 Mar 4 08:31:00
office-backup kernel: [8111c0c2] ?
SyS_newlstat+0x12/0x30 Mar 4 08:31:00 office-backup kernel:
[817bb7a2] ? system_call_fastpath+0x16/0x1b Mar 4
08:55:01 office-backup kernel: INFO: task updatedb.mlocat:903
blocked for more than 120 seconds. Mar 4 08:55:01
Re: ENOSPC errors during raid1 rebalance
On Mar 5, 2014, at 3:13 PM, Michael Russo m...@papersolve.com wrote: Chris Murphy lists at colorremedies.com writes: Did you do a defrag and balance after ext4btrfs conversion, but before data/metadata profile conversion? No I didn't, as I thought it was only optional and didn't realize it might later affect my ability to change profiles. It's possible it's more necessary in certain situations than others. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Understanding btrfs and backups
apologies if this is a resend - it appeared to me that it was rejected because of something in how Gmail was formatting the message. I can't find it in the Gmane archives which leads me to believe it was never delivered. I was hoping to gain some clarification on btrfs snapshops and how they function as backups. I did a bit of Googling and found lots of examples of bash commands, but no one seemed to explain what was going on to a level that would satisfy me for my data needs. I read this Ars Technica article today http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ First of all, the btrfs-raid1 sounds awesome. Because it helps protect against one of RAID1's failings - bit rot issues. But raid1 is not backup, it's just redundancy. Second, the article mentions using snapshots as a backup method. Page 3 section: Using the features. He makes a snapshot and sends that. Then he sends what changed the second time. He mentions that because btrfs knows what's changed it's a quick process. Right now on my Linux computer I use Back in Time which, I think, is just an rsync frontend. It takes a long time to complete the backup for my 1 TB /home drive. The copy part is nice and quick, but the comparison part takes a long time and hammers the CPU. I have it setup to run at night because if it runs while I'm using the computer, things can crawl. So I was wondering if btrfs snapshots are a substitute for this. Right now if I realize I deleted a file 5 days ago, I can go into Back in Time (the gui) or just navigate to it on the backup drive and restore that one file. From what I've read about btrfs, I'd have to restore the entire home drive, right? Which means I'd lose all the changes from the past five days. If that's the case, it wouldn't really solve my problem - although maybe I'm just not thinking creatively. Also, if I first do the big snapshot backup and then the increments, how do I delete the older snapshots? In other words, the way I'm picturing things working is that I have the main snapshot and every snapshot after that is just a description of what's changed since then. So wouldn't the entire chain be necessary to reconstruct where I'm at now? On a somewhat separate note, I have noticed that many people/utilities for btrfs mention making snapshots every hour. Are the snapshots generally that small that such a think wouldn't quickly fill a hard drive? Thanks for reading my questions, I appreciate the help. When all is said and done I'd certainly like to publish a how-to from my point of undertanding. -- Eric Mesa -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Understanding btrfs and backups
Brian Wong wrote: a snapshot is different than a backup, with a snapshot you're still accessing a read-only version of the live filesystem. i don't know the specifics of btrfs but if you take daily snapshots, you should be able to restore a single file from the five-days-ago snapshot by browsing that snapshot's directory tree and then copying the file to the live version of the filesystem, if that makes sense. in the snapshot case the live filesystem serves the same function as the full backup would if you did full backups then incrementals. the snapshots are the incrementals of the live filesystem, only going backwards in time whereas with backup you would take a full backup then go forward in time with incrementals. the filesystem takes care of making sure every snapshot is complete. in the snapshot case redundancy is then more important because you may not have a bunch of full backups (i.e. full copies) lying around. so full backups still are useful. -- OK, I THINK I understand things a bit better. So from the point of view of restoring a single file, that functionality is there. Excellent. And I guess you're saying that because the snapshots are diffs off the live system, that I'd need a backup of the live system - ie snapshots wouldn't be enough. But what if my first snapshot was a clone of the system at that point (as it seems from the article) And I back that up to a separate drive. Let me illustrate with what I plan to do exactly. Three hard drives: A, B, and C. Hard drives A and B - btrfs RAID-1 so that if one drive dies I can keep using my system until the replacement for the raid arrives. Hard drive C - gets (hourly/daily/weekly/or some combination of the above) snapshots from the RAID. (Starting with the initial state snapshot) Each timepoint another snapshot is copied to hard drive C. So in the case of a file disappearing on me or being over-written or w/e - I reach into the directory of the snapshot that contains the file just as I would now with the backup. So if that's what I'm doing, do snapshots become a way to do backups? Thanks -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Understanding btrfs and backups
Brian Wong wrote: a snapshot is different than a backup, with a snapshot you're still accessing a read-only version of the live filesystem. i don't know the specifics of btrfs but if you take daily snapshots, you should be able to restore a single file from the five-days-ago snapshot by browsing that snapshot's directory tree and then copying the file to the live version of the filesystem, if that makes sense. in the snapshot case the live filesystem serves the same function as the full backup would if you did full backups then incrementals. the snapshots are the incrementals of the live filesystem, only going backwards in time whereas with backup you would take a full backup then go forward in time with incrementals. the filesystem takes care of making sure every snapshot is complete. in the snapshot case redundancy is then more important because you may not have a bunch of full backups (i.e. full copies) lying around. so full backups still are useful. -- OK, I THINK I understand things a bit better. So from the point of view of restoring a single file, that functionality is there. Excellent. And I guess you're saying that because the snapshots are diffs off the live system, that I'd need a backup of the live system - ie snapshots wouldn't be enough. But what if my first snapshot was a clone of the system at that point (as it seems from the article) And I back that up to a separate drive. Let me illustrate with what I plan to do exactly. Three hard drives: A, B, and C. Hard drives A and B - btrfs RAID-1 so that if one drive dies I can keep using my system until the replacement for the raid arrives. Hard drive C - gets (hourly/daily/weekly/or some combination of the above) snapshots from the RAID. (Starting with the initial state snapshot) Each timepoint another snapshot is copied to hard drive C. So in the case of a file disappearing on me or being over-written or w/e - I reach into the directory of the snapshot that contains the file just as I would now with the backup. So if that's what I'm doing, do snapshots become a way to do backups? Thanks -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Understanding btrfs and backups
On 2014/03/06 09:27 PM, Eric Mesa wrote: Brian Wong wrote: a snapshot is different than a backup [snip] ... Three hard drives: A, B, and C. Hard drives A and B - btrfs RAID-1 so that if one drive dies I can keep using my system until the replacement for the raid arrives. Hard drive C - gets (hourly/daily/weekly/or some combination of the above) snapshots from the RAID. (Starting with the initial state snapshot) Each timepoint another snapshot is copied to hard drive C. [snip]... So if that's what I'm doing, do snapshots become a way to do backups? An important distinction for anyone joining the conversation is that snapshots are *not* backups, in a similar way that you mentioned that RAID is not a backup. If a hard drive implodes, its snapshots go with it. Snapshots can (and should) be used as part of a backup methodology - and your example is almost exactly the same as previous good backup examples. I think most of the time there's mention of an external backup server keeping the backups, which is the only major difference compared to the process you're looking at. Btrfs send/receive with snapshots can make the process far more efficient compared to rsync. Rsync doesn't have any record as to what information has changed so it has to compare all the data (causing heavy I/O). Btrfs keeps a record and can skip to the part of sending the data. I do something similar to what you have described on my Archlinux desktop - however I haven't updated my (very old) backup script to take advantage of btrfs' send/receive functionality. I'm still using rsync. :-/ / and /home are on btrfs-raid1 on two smallish disks /mnt/btrfs-backup is on btrfs single/dup on a single larger disk See https://btrfs.wiki.kernel.org/index.php/Incremental_Backup for a basic incremental methodology using btrfs send/receive -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Understanding btrfs and backups
Eric Mesa posted on Thu, 06 Mar 2014 18:18:15 + as excerpted: apologies if this is a resend - it appeared to me that it was rejected because of something in how Gmail was formatting the message. I can't find it in the Gmane archives which leads me to believe it was never delivered. Probably HTML-formatted. AFAIK vger.kernel.org (the list-serv for many kernel lists) is set to reject that. Too bad more list-servs don't do likewise. =:^( I was hoping to gain some clarification on btrfs snapshops and how they function as backups. Looking at the below it does indeed appear you are confused, but this is the place to post the questions necessary to get unconfused. =:^) I did a bit of Googling and found lots of examples of bash commands, but no one seemed to explain what was going on to a level that would satisfy me for my data needs. You don't mention whether you've seen/read the btrfs wiki or not. That's the most direct and authoritative place to look... and to bookmark. =:^) https://btrfs.wiki.kernel.org I read this Ars Technica article today http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic- cows-inside-next-gen-filesystems/ First of all, the btrfs-raid1 sounds awesome. Because it helps protect against one of RAID1's failings - bit rot issues. But raid1 is not backup, it's just redundancy. Second, the article mentions using snapshots as a backup method. Well, this is where you start to be confused. Snapshots are not backups either, altho they're sort of opposite raid in that while raid is redundancy-only, snapshots are rollback-only, without the redundancy (I'll explain...). Page 3 section: Using the features. He makes a snapshot and sends that. Then he sends what changed the second time. He mentions that because btrfs knows what's changed it's a quick process. OK, what that is discussing is btrfs send/receive, with snapshots simply part of the process of doing that. Think rsync in effect, but btrfs- specific and much more efficient. Btrfs send/receive does use snapshots but only as part of making the send/receive process more reliable and efficient. I'll discuss snapshots (and COW) first, below, then bring in btrfs send/receive at the end. Right now on my Linux computer I use Back in Time which, I think, is just an rsync frontend. It takes a long time to complete the backup for my 1 TB /home drive. The copy part is nice and quick, but the comparison part takes a long time and hammers the CPU. I have it setup to run at night because if it runs while I'm using the computer, things can crawl. So I was wondering if btrfs snapshots are a substitute for this. Right now if I realize I deleted a file 5 days ago, I can go into Back in Time (the gui) or just navigate to it on the backup drive and restore that one file. From what I've read about btrfs, I'd have to restore the entire home drive, right? Which means I'd lose all the changes from the past five days. If that's the case, it wouldn't really solve my problem - although maybe I'm just not thinking creatively. No, in snapshot terms you don't restore the entire drive. Rather, the snapshots are taken on the local filesystem, storing (like one still frame in a series that makes a movie, thus the term snapshot) the state of the filesystem at the point the snapshot was taken. Files can be created/deleted/moved/altered after the snapshot, and only the differences between snapshots and between the last snapshot and the current state are changed. The fact that btrfs is a copy-on-write (COW) filesystem makes snapshotting very easy... trivial... since it's a byproduct of the COW nature of the filesystem and thus comes very nearly for free, with only hooking up some way to access specific bits of functionality that's already there necessary in ordered to get snapshotting. A copy-on-write illustration (please view with a monospace font for proper alignment): Suppose each letter of the following string represents a block of a particular size (say 4KiB) of a file, with the corresponding block addresses noted as well: 0111 1234567890123456 abcdefgxijklmnop It's the first bit of the alphabet, but notice the x where h belongs. Now someone notices and edits the file, correcting the problem: abcdefghijklmnop Except when they save the file, a COW-based filesystem will make the change like this: 00050111 1234567390123456 ||| abcdefg ijklmnop | h The unchanged blocks of the file all remain in place. The only change is to the one block, which unlike normal filesystems, isn't edited in-place, but rather, is written into a new location, and the filesystem simply notes that the new location (53) should be used to read that file block now, instead of the old location (08). Of course as illustrated here, the addresses each take up two characters while the data block only takes up one, but each
[PATCH] Btrfs: fix deadlock with nested trans handles
Zach found this deadlock that would happen like this
btrfs_end_transaction - reduce trans-use_count to 0
btrfs_run_delayed_refs
btrfs_cow_block
find_free_extent
btrfs_start_transaction - increase trans-use_count to 1
allocate chunk
btrfs_end_transaction - decrease trans-use_count to 0
btrfs_run_delayed_refs
lock tree block we are cowing above ^^
We need to only decrease trans-use_count if it is above 1, otherwise leave it
alone. This will make nested trans be the only ones who decrease their added
ref, and will let us get rid of the trans-use_count++ hack if we have to commit
the transaction. Thanks,
cc: sta...@vger.kernel.org
Reported-by: Zach Brown z...@redhat.com
Signed-off-by: Josef Bacik jba...@fb.com
---
fs/btrfs/transaction.c | 14 --
1 file changed, 4 insertions(+), 10 deletions(-)
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 34cd831..b05bf58 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -683,7 +683,8 @@ static int __btrfs_end_transaction(struct
btrfs_trans_handle *trans,
int lock = (trans-type != TRANS_JOIN_NOLOCK);
int err = 0;
- if (--trans-use_count) {
+ if (trans-use_count 1) {
+ trans-use_count--;
trans-block_rsv = trans-orig_rsv;
return 0;
}
@@ -731,17 +732,10 @@ static int __btrfs_end_transaction(struct
btrfs_trans_handle *trans,
}
if (lock ACCESS_ONCE(cur_trans-state) == TRANS_STATE_BLOCKED) {
- if (throttle) {
- /*
-* We may race with somebody else here so end up having
-* to call end_transaction on ourselves again, so inc
-* our use_count.
-*/
- trans-use_count++;
+ if (throttle)
return btrfs_commit_transaction(trans, root);
- } else {
+ else
wake_up_process(info-transaction_kthread);
- }
}
if (trans-type __TRANS_FREEZABLE)
--
1.8.3.1
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: fix deadlock with nested trans handles
On Thu, Mar 06, 2014 at 07:01:07PM -0500, Josef Bacik wrote: Zach found this deadlock that would happen like this btrfs_end_transaction - reduce trans-use_count to 0 btrfs_run_delayed_refs btrfs_cow_block find_free_extent btrfs_start_transaction - increase trans-use_count to 1 allocate chunk btrfs_end_transaction - decrease trans-use_count to 0 btrfs_run_delayed_refs lock tree block we are cowing above ^^ Indeed, I stumbled across this while trying to reproduce reported problems with iozone. This deadlock would consistently hit during random 1k reads in a 2gig file. We need to only decrease trans-use_count if it is above 1, otherwise leave it alone. This will make nested trans be the only ones who decrease their added ref, and will let us get rid of the trans-use_count++ hack if we have to commit the transaction. Thanks, And this fixes it. It's run through a few times successfully. cc: sta...@vger.kernel.org Reported-by: Zach Brown z...@redhat.com Signed-off-by: Josef Bacik jba...@fb.com Tested-by: Zach Brown z...@redhat.com - z -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ENOSPC errors during raid1 rebalance
Duncan 1i5t5.duncan at cox.net writes: But if you're not using compression, /that/ can't explain it... Ha! Well while that was an interesting discussion of fragmentation, I am only using the default mount options here and so no compression. The only reason I'm really even looking at the fragmentation issue is because running the defragment (with varying sizes of -t which I'm not sure why that's even necessary) seemed to force btrfs to move segments around and let me rebalance more block groups. I don't even care if the files are defragged, I just want them all in the RAID1 profile. Hopefully if I move each file out to some other FS like /dev/shm and then back it will work, I just gotta hack a script together to do so. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
