3.15 btrfs free space cache oops

2014-08-11 Thread Daniel J Blueman
When running MonetDB against a BTRFS RAID-0 set over 4 SSDs [1] on 3.15.5, we see io_ctl have a bad address of 0x20, causing a fatal pagefault in memcpy(): (gdb) list *(__btrfs_write_out_cache+0x3e4) 0x81365984 is in __btrfs_write_out_cache (fs/btrfs/free-space-cache.c:521). 516

Re: [BUG] btrfs send/receive, page allocation failure

2014-08-11 Thread Filipe David Manana
On Mon, Aug 11, 2014 at 4:07 AM, Chris Murphy li...@colorremedies.com wrote: Can you try the following patch and confirm if it helps? https://patchwork.kernel.org/patch/4705171/ This one applies without problems, I didn't build it because I saw v4. The v4 patch I get: + patch -p1 -F1 -s

[PATCH 1/3] btrfs-progs: random fixes of btrfs-filesystem documentation

2014-08-11 Thread Satoru Takeuchi
From: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com - Simplify and unify the description of both man and usage. - Fix to show -m and -d is not exclusive with path|uuid|device|label. - Add the description about short options for --mounted and --all-devices, -m and -d respectively. -

[PATCH 2/3] btrfs-progs: avoid to use numeric literal for the size of uuid buffer

2014-08-11 Thread Satoru Takeuchi
From: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com Replace a numeric literal to more descriptive macro for the size of uuid buffer. Signed-of-by: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com --- cmds-filesystem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git

[PATCH 3/3] btrfs-progs: Show error message if btrfs filesystem show failed to find any btrfs filesystem

2014-08-11 Thread Satoru Takeuchi
From: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com Current btrfs doesn't display any error message if this command failed to find any btrfs filesystem corresponding to path|uuid|device|label which user specified. Signed-off-by: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com ---

[PATCH 4/4] btrfs: update sprout seed pointer when seed fs is relinquished

2014-08-11 Thread Anand Jain
We are not updating sprout fs seed pointer when all seed device is replaced. This patch will check if all seed device has been replaced and then update the sprout pointer accordingly. Same reproducer as in the previous patch would apply here. And notice that btrfs_close_device will check if seed

[PATCH 3/4] btrfs: fix rw_devices miss match after seed replace

2014-08-11 Thread Anand Jain
reproducer: reproducer: mount /dev/sdb /btrfs btrfs dev add /dev/sdc /btrfs btrfs rep start -B /dev/sdb /dev/sdd /btrfs umount /btrfs WARNING: CPU: 0 PID: 3882 at fs/btrfs/volumes.c:892 __btrfs_close_devices+0x1c8/0x200 [btrfs]() which is

[PATCH 1/4] btrfs: preparatory to make btrfs_rm_dev_replace_srcdev() seed aware

2014-08-11 Thread Anand Jain
There is no logical change in this patch, just a preparatory patch, so that changes can be easily reasoned. Signed-off-by: Anand Jain anand.j...@oracle.com --- fs/btrfs/volumes.c | 14 +- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/volumes.c

[PATCH 2/4] btrfs: replace seed device followed by unmount causes kernel WARNING

2014-08-11 Thread Anand Jain
reproducer: mount /dev/sdb /btrfs btrfs dev add /dev/sdc /btrfs btrfs rep start -B /dev/sdb /dev/sdd /btrfs umount /btrfs WARNING: CPU: 0 PID: 12661 at fs/btrfs/volumes.c:891 __btrfs_close_devices+0x1b0/0x200 [btrfs]() :: __btrfs_close_devices() :: WARN_ON(fs_devices-open_devices);

Re: [PATCH] btrfs: replace seed device followed by unmount causes kernel WARNING

2014-08-11 Thread Anand Jain
I have sent out the patch-set [PATCH 1/4] btrfs: preparatory to make btrfs_rm_dev_replace_srcdev() seed aware in replacement for this patch. Kindly use/review the above patch set. Thanks. Anand On 31/07/2014 16:45, Anand Jain wrote: On 30/07/2014 15:42, Miao Xie wrote: On Fri, 25

Re: [PATCH] Btrfs: don't monopolize a core when evicting inode

2014-08-11 Thread Satoru Takeuchi
(2014/08/08 10:47), Filipe Manana wrote: If an inode has a very large number of extent maps, we can spend a lot of time freeing them, which triggers a soft lockup warning. Therefore reschedule if we need to when freeing the extent maps while evicting the inode. I could trigger this all the

Re: [PATCH] Btrfs: clone, don't create invalid hole extent map

2014-08-11 Thread Satoru Takeuchi
(2014/08/08 10:47), Filipe Manana wrote: When cloning a file that consists of an inline extent, we were creating an extent map that represents a non-existing trailing hole starting at a file offset that isn't a multiple of the sector size. This happened because when processing an inline extent

File system stuck in scrub

2014-08-11 Thread Nikolaus Rath
Hello, I started a scrub of one of my btrfs filesystem and then had to restart the system. `systemctl restart` seemed to terminate all processes, but then got stuck at the end. The disk activity led was still flashing rapidly at that point, so I assume that the active scrub was preventing the

Re: File system stuck in scrub

2014-08-11 Thread Hugo Mills
On Mon, Aug 11, 2014 at 08:12:46AM -0700, Nikolaus Rath wrote: I started a scrub of one of my btrfs filesystem and then had to restart the system. `systemctl restart` seemed to terminate all processes, but then got stuck at the end. The disk activity led was still flashing rapidly at that

Re: File system stuck in scrub

2014-08-11 Thread Calvin Walton
Hi, On Mon, 2014-08-11 at 08:12 -0700, Nikolaus Rath wrote: Hello, I started a scrub of one of my btrfs filesystem and then had to restart the system. `systemctl restart` seemed to terminate all processes, but then got stuck at the end. The disk activity led was still flashing rapidly

Re: File system stuck in scrub

2014-08-11 Thread Marc MERLIN
On Mon, Aug 11, 2014 at 11:45:45AM -0400, Calvin Walton wrote: $ sudo btrfs scrub start /home/nikratio/ ERROR: scrub is already running. To cancel use 'btrfs scrub cancel /home/nikratio/'. To see the status use 'btrfs scrub status [-d] /home/nikratio/'. My guess is that this is a mismatch

Re: [BUG] btrfs send/receive, page allocation failure

2014-08-11 Thread Chris Murphy
On Aug 11, 2014, at 2:46 AM, Filipe David Manana fdman...@gmail.com wrote: On Mon, Aug 11, 2014 at 4:07 AM, Chris Murphy li...@colorremedies.com wrote: Can you try the following patch and confirm if it helps? https://patchwork.kernel.org/patch/4705171/ This one applies without

Re: [BUG] btrfs send/receive, page allocation failure

2014-08-11 Thread Filipe David Manana
On Mon, Aug 11, 2014 at 4:55 PM, Chris Murphy li...@colorremedies.com wrote: On Aug 11, 2014, at 2:46 AM, Filipe David Manana fdman...@gmail.com wrote: On Mon, Aug 11, 2014 at 4:07 AM, Chris Murphy li...@colorremedies.com wrote: Can you try the following patch and confirm if it helps?

Re: [BUG] btrfs send/receive, page allocation failure

2014-08-11 Thread Chris Murphy
On Aug 11, 2014, at 10:04 AM, Filipe David Manana fdman...@gmail.com wrote: On Mon, Aug 11, 2014 at 4:55 PM, Chris Murphy li...@colorremedies.com wrote: On Aug 11, 2014, at 2:46 AM, Filipe David Manana fdman...@gmail.com wrote: On Mon, Aug 11, 2014 at 4:07 AM, Chris Murphy

Re: [BUG] btrfs send/receive, page allocation failure

2014-08-11 Thread lists
On 2014-08-11 10:25, Chris Murphy wrote: On Aug 11, 2014, at 10:04 AM, Filipe David Manana fdman...@gmail.com wrote: On Mon, Aug 11, 2014 at 4:55 PM, Chris Murphy li...@colorremedies.com wrote: On Aug 11, 2014, at 2:46 AM, Filipe David Manana fdman...@gmail.com wrote: On Mon, Aug 11,

Re: [PATCH 1/3] btrfs-progs: random fixes of btrfs-filesystem documentation

2014-08-11 Thread Eric Sandeen
On 8/11/14, 2:11 AM, Satoru Takeuchi wrote: From: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com - Simplify and unify the description of both man and usage. - Fix to show -m and -d is not exclusive with path|uuid|device|label. - Add the description about short options for --mounted

Re: [PATCH 1/3] btrfs-progs: random fixes of btrfs-filesystem documentation

2014-08-11 Thread Eric Sandeen
On 8/11/14, 10:05 AM, Eric Sandeen wrote: On 8/11/14, 2:11 AM, Satoru Takeuchi wrote: From: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com - Simplify and unify the description of both man and usage. - Fix to show -m and -d is not exclusive with path|uuid|device|label. - Add the

Announcement: buttersink - like rsync for btrfs snapshots

2014-08-11 Thread Ames Cornish
I've written a utility to help me with using btrfs send and receive for backups or other synchronization, and I'd love to get feedback on it. As of this release, buttersink will synchronize a set of read-only snapshots in a btrfs filesystem to an Amazon S3 bucket, and vice-versa. It

Re: Announcement: buttersink - like rsync for btrfs snapshots

2014-08-11 Thread Jim Salter
How has it been for reliability? I wrote a btrsync app a while back, and the app /itself/ worked fine, but the btrfs send / btrfs receive itself proved problematic. Since btrfs would keep a partial receive - with no easy way to tell whether a receive WAS partial or full - I would inevitably

Re: Announcement: buttersink - like rsync for btrfs snapshots

2014-08-11 Thread Jim Salter
To any core btrfs devs who are listening and care - the unreliability of btrfs send/receive is IMO the single biggest roadblock to adoption of btrfs as a serious next-gen FS. I can live with occasional corner-case performance issues, I can even live with (very) occasional filesystem

Re: Announcement: buttersink - like rsync for btrfs snapshots

2014-08-11 Thread Ames Cornish
Jim, btrfs send reliability has been an issue, though I've been able to successfully use it for my backups. buttersink usually detects the errors and will either move the destination snapshot to mark it as partial/failed (for btrfs), or cancel and delete the partial upload (for S3). I've also

Large files, nodatacow and fragmentation

2014-08-11 Thread G. Richard Bellamy
I've been playing with btrfs as a backing store for my KVM images. I've used 'chattr +C' on the directory where those images are stored. You can see my recipe below [1]. I've read the gotchas found here [2] I'm having continuing performance issues inside the Guest VM that is created inside the

Re: Large files, nodatacow and fragmentation

2014-08-11 Thread Roman Mamedov
On Mon, 11 Aug 2014 11:36:46 -0700 G. Richard Bellamy rbell...@pteradigm.com wrote: I've been playing with btrfs as a backing store for my KVM images. I've used 'chattr +C' on the directory where those images are stored. You can see my recipe below [1]. I've read the gotchas found here [2]

Re: Ideas for a feature implementation

2014-08-11 Thread Chris Murphy
On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn ahferro...@gmail.com wrote: Another thing that isn't listed there, that I would personally love to see is support for secure file deletion. To be truly secure though, this would need to hook into the COW logic so that files marked for

Re: [PATCH] Btrfs: fix compressed write corruption on enospc

2014-08-11 Thread Chris Mason
On 08/10/2014 10:55 AM, Liu Bo wrote: On Thu, Aug 07, 2014 at 10:02:15AM -0400, Chris Mason wrote: On 08/07/2014 04:20 AM, Miao Xie wrote: On Thu, 7 Aug 2014 15:50:30 +0800, Liu Bo wrote: [90496.156016] kworker/u8:14 D 880044e38540 0 21050 2 0x [90496.157683]

Re: [BUG] btrfs send/receive, page allocation failure

2014-08-11 Thread Chris Murphy
On Aug 11, 2014, at 10:41 AM, li...@colorremedies.com wrote: https://friendpaste.com/Bgwdjk31P3pZHtArr341G OK that friendpaste is completely different than the [PATCH v4] email. # from above URL diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 6528aa6..95891c0 100644 I can confirm

Re: [BUG] btrfs send/receive, page allocation failure

2014-08-11 Thread Filipe David Manana
On Mon, Aug 11, 2014 at 9:30 PM, Chris Murphy li...@colorremedies.com wrote: On Aug 11, 2014, at 10:41 AM, li...@colorremedies.com wrote: https://friendpaste.com/Bgwdjk31P3pZHtArr341G OK that friendpaste is completely different than the [PATCH v4] email. # from above URL diff --git

Re: 40TB volume taking over 16 hours to mount, any ideas?

2014-08-11 Thread Jose Ildefonso Camargo Tolosa
As I hate when a thread is left hanging, you deserve to know what happened in the end, you likely already guessed, but anyway: I nuked the filesystem, and started over. After some internal discussion in the company, we decided to move to ZFS for now. However, we will keep an eye on btrfs, and

Re: Large files, nodatacow and fragmentation

2014-08-11 Thread G. Richard Bellamy
On Mon, Aug 11, 2014 at 12:14 PM, Roman Mamedov r...@romanrm.net wrote: First of all, why do you require a COW filesystem in the first place... if all you do is just use it in a NoCOW mode? Second, why qcow2? It can also have internal fragmentation which is unlikely to do anything good for

Re: [PATCH] Btrfs: fix regression of btrfs device replace

2014-08-11 Thread Chris Murphy
On Jul 29, 2014, at 5:09 AM, Liu Bo bo.li@oracle.com wrote: Commit 49c6f736f34f901117c20960ebd7d5e60f12fcac( btrfs: dev replace should replace the sysfs entry) added the missing sysfs entry in the process of device replace, but didn't take missing devices into account, so now we have

Re: Large files, nodatacow and fragmentation

2014-08-11 Thread Chris Murphy
On Aug 11, 2014, at 1:14 PM, Roman Mamedov r...@romanrm.net wrote: Second, why qcow2? It can also have internal fragmentation which is unlikely to do anything good for performance. It really depends on what version of libvirt and qemu-image you've got. I did some testing during Fedora

Re: [PATCH 1/3] btrfs-progs: random fixes of btrfs-filesystem documentation

2014-08-11 Thread Satoru Takeuchi
Hi Eric, (2014/08/12 2:05), Eric Sandeen wrote: On 8/11/14, 2:11 AM, Satoru Takeuchi wrote: From: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com - Simplify and unify the description of both man and usage. - Fix to show -m and -d is not exclusive with path|uuid|device|label. - Add

Re: [PATCH 1/3] btrfs-progs: random fixes of btrfs-filesystem documentation

2014-08-11 Thread Satoru Takeuchi
Hi Eric, (2014/08/12 2:14), Eric Sandeen wrote: On 8/11/14, 10:05 AM, Eric Sandeen wrote: On 8/11/14, 2:11 AM, Satoru Takeuchi wrote: From: Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com - Simplify and unify the description of both man and usage. - Fix to show -m and -d is not exclusive

Re: [PATCH 1/3] btrfs-progs: random fixes of btrfs-filesystem documentation

2014-08-11 Thread Eric Sandeen
On Aug 11, 2014, at 4:51 PM, Satoru Takeuchi takeuchi_sat...@jp.fujitsu.com wrote: Hi Eric, ... OK, I'll fix my patch based on your comment. # Of course, I'll replace (btrfs?) with something proper words. I assumed it only scans btrfs but didn't know for sure :) Can I add your

Re: [PATCH] Btrfs: fix regression of btrfs device replace

2014-08-11 Thread Satoru Takeuchi
Hi Liu, (2014/08/12 6:41), Chris Murphy wrote: On Jul 29, 2014, at 5:09 AM, Liu Bo bo.li@oracle.com wrote: Commit 49c6f736f34f901117c20960ebd7d5e60f12fcac( btrfs: dev replace should replace the sysfs entry) added the missing sysfs entry in the process of device replace, but didn't take

Re: Ideas for a feature implementation

2014-08-11 Thread Austin S Hemmelgarn
On 08/11/2014 04:27 PM, Chris Murphy wrote: On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn ahferro...@gmail.com wrote: Another thing that isn't listed there, that I would personally love to see is support for secure file deletion. To be truly secure though, this would need to hook

Re: Blocked tasks on 3.15.1

2014-08-11 Thread Charles Cazabon
The blocked tasks issue that got significantly worse in 3.15 -- did anything go into 3.16 related to this? I didn't see a single btrfs in Linus' 3.16 announcement, so I don't know whether it should be better, the same, or worse in this respect... I haven't seen a definite statement about this on

Re: [PATCH] Btrfs: fix compressed write corruption on enospc

2014-08-11 Thread Miao Xie
On Sun, 10 Aug 2014 22:55:44 +0800, Liu Bo wrote: This part of the trace is relatively new because Liu Bo's patch made us redirty the pages, making it more likely that we'd try to write them during commit. But, at the end of the day we have a fundamental deadlock with committing a

Re: Blocked tasks on 3.15.1

2014-08-11 Thread Liu Bo
On Mon, Aug 11, 2014 at 08:55:21PM -0600, Charles Cazabon wrote: The blocked tasks issue that got significantly worse in 3.15 -- did anything go into 3.16 related to this? I didn't see a single btrfs in Linus' 3.16 announcement, so I don't know whether it should be better, the same, or worse

Re: [PATCH] Btrfs: fix regression of btrfs device replace

2014-08-11 Thread Liu Bo
On Tue, Aug 12, 2014 at 11:25:00AM +0900, Satoru Takeuchi wrote: Hi Liu, (2014/08/12 6:41), Chris Murphy wrote: On Jul 29, 2014, at 5:09 AM, Liu Bo bo.li@oracle.com wrote: Commit 49c6f736f34f901117c20960ebd7d5e60f12fcac( btrfs: dev replace should replace the sysfs entry) added the

Re: Ideas for a feature implementation

2014-08-11 Thread Chris Murphy
On Aug 11, 2014, at 8:27 PM, Austin S Hemmelgarn ahferro...@gmail.com wrote: On 08/11/2014 04:27 PM, Chris Murphy wrote: On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn ahferro...@gmail.com wrote: Another thing that isn't listed there, that I would personally love to see is support

Re: 40TB volume taking over 16 hours to mount, any ideas?

2014-08-11 Thread Duncan
Jose Ildefonso Camargo Tolosa posted on Mon, 11 Aug 2014 16:33:36 -0500 as excerpted: As I hate when a thread is left hanging, you deserve to know what happened in the end, you likely already guessed, but anyway: I nuked the filesystem, and started over. After some internal discussion in

Re: Blocked tasks on 3.15.1

2014-08-11 Thread Duncan
Liu Bo posted on Tue, 12 Aug 2014 10:56:42 +0800 as excerpted: On Mon, Aug 11, 2014 at 08:55:21PM -0600, Charles Cazabon wrote: The blocked tasks issue that got significantly worse in 3.15 -- did anything go into 3.16 related to this? I didn't see a single btrfs in Linus' 3.16 announcement,

Re: Blocked tasks on 3.15.1

2014-08-11 Thread Marc MERLIN
On Mon, Aug 11, 2014 at 08:55:21PM -0600, Charles Cazabon wrote: The blocked tasks issue that got significantly worse in 3.15 -- did anything go into 3.16 related to this? I didn't see a single btrfs in Linus' 3.16 announcement, so I don't know whether it should be better, the same, or worse

3.15 btrfs free space cache oops

2014-08-11 Thread Daniel J Blueman
When running MonetDB over a BTRFS RAID-0 set over 4 SSDs [1] on 3.15.5, we see io_ctl have a bad address of 0x20, causing a fatal pagefault in memcpy(): (gdb) list *(__btrfs_write_out_cache+0x3e4) 0x81365984 is in __btrfs_write_out_cache (fs/btrfs/free-space-cache.c:521). 516