Re: Applications using fsync cause hangs for several seconds every few minutes
Are these processes principally btrfs-submit and btrfs-transacti in particular? Then it may be related to my very similar issue reported earlier. On 08/18/2011 08:47 AM, Chris Samuel wrote: > On 18/08/11 00:29, Michael Cronenworth wrote: > >> I'm running kernel 3.0 (Fedora 15's 2.6.40) on two boxes >> and I have not seen slow downs or hangs. I use Firefox. > > I've got btrfs on an external USB drive with the 3.0.1 kernel and > I see that sync seems to take an age, according to iotop it seems > that the btrfs processes are hitting it quite hard, IIRC. > > cheers, > Chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Applications using fsync cause hangs for several seconds every few minutes
On 18/08/11 00:29, Michael Cronenworth wrote: > I'm running kernel 3.0 (Fedora 15's 2.6.40) on two boxes > and I have not seen slow downs or hangs. I use Firefox. I've got btrfs on an external USB drive with the 3.0.1 kernel and I see that sync seems to take an age, according to iotop it seems that the btrfs processes are hitting it quite hard, IIRC. cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Applications using fsync cause hangs for several seconds every few minutes
This is most probably related to the same regression seen after 2.6.38, my blocked comment on 3 August included an indication to that the behavior was present in my distro 2.6.38 kernel too, it just was appearing after a considerably longer uptime (on my desktop system using btrfs as rootfs on an Intel ICH10 driven SATA HDD). I have reverted my / to ext4 since, and I'm okay with it, although I would be very happy to see some improvement on this serious-for-me issue. Btrfs slowdown news://news.gmane.org:119/cao47_-9blkwugdeuzalqhsq9tzkauao8fmqey1ppk9a2hb+...@mail.gmail.com Also, a patch by Josef Bacik was an attempt for fixing this, but no one reported about testing it on an affected system, it did not eliminate the slowdowns for me: PLEASE TEST: Everybody who is seeing weird and long hangs news://news.gmane.org:119/4e36c47e.70...@redhat.com My comment was going as an aswer to Mck's post in "Btrfs slowdown" thread, where I reported about this in a little more detail - but it never appeared on the list. I try including it now: I'm confirming this too. Following advices given on #btrfs irc, I have applied Josef's second patch for fs/btrfs/extent_io.c and I'm reporting that it did NOT make the slowdowns disappear on 3.0 kernels (even with some rather different configs). The HDD thrashing appeared on all other kernel versions I tried, higher than 2.6.37. Initially, I had been into looking for a latest known good kernel (to prepare a proper git bisect as cwillu advised) and at first I also felt like 2.6.38 does not show this miserable behaviour. But later it turned out this was only for approximately 2 days of uptime. Given enough time, the lock-ups appeared on 2.6.38 too. Although they were not that apparent than on later kernel versions, and the individual lockups took much less time with 2.6.38 running for 2 days (binary Sabayon Linux repository kernel). My HDD, with btrfs as / on it emits very distinct (and loud enough) noises with a slightly different character for reads and writes - and I can actually hear the disk's repetitive seek pattern during a such thrashing period. Based on that, I guess it must be the exact same thing happening with 2.6.38 as with later kernels because they sound very similar. They last much shorter but they have a similarly repetitive seeking nature with other I/O severely throttled and I believe it is write what is mostly what's happening during a lockup. So I concluded that I failed to identify a known good version so far. I didn't have time to get into earlier kernels than .38. (Tried .37, but for too brief of uptime to claim they did not appear when I was on .37) Similar with my current kernel. It started happening after about 12 hours of running the machine using # uname -a Linux insula 3.0.0-git15genseed #2 SMP PREEMPT Tue Aug 2 20:10:05 CEST 2011 x86_64 Intel(R) Core(TM)2 Duo CPU E4500 @ 2.20GHz GenuineIntel GNU/Linux As appended string reflects, it is a custom kernel, it has Josef's patch applied with the config attached.Tried to patch my distro's 3.0 kernel, no change was experienced with regards to the issue (iirc it was even a lot worse). Let me know if I can contribute with anything that would be valuable for the developers towards elimination of this very nasty bug. Now, after 23 hours of uptime, my PC has become almost unusable. Currently there's about 8 seconds thrashing, 10 seconds not thrashing, and during thrashing, all other (disk) I/O is practically blocked. SysRq+W under thrashing (dunno how informative it is, but here's one): [62279.779382] SysRq : Show Blocked State [62279.779389] taskPC stack pid father [62279.779404] btrfs-submit-0 D 5616 4678 2 0x [62279.779413] 88012b1370d0 0046 8801 8182c020 [62279.779422] 880128d39fd8 00010480 4000 880128d38000 [62279.779429] 880128d39fd8 00010480 88012b1370d0 00010480 [62279.779437] Call Trace: [62279.779449] [] ? cfq_set_request+0x33e/0x37e [62279.779456] [] ? cfq_cic_lookup+0x35/0x139 [62279.779462] [] ? cfq_may_queue+0x51/0x6e [62279.779470] [] ? io_schedule+0x4e/0x63 [62279.779477] [] ? get_request_wait+0xaa/0x10e [62279.779484] [] ? wake_up_bit+0x23/0x23 [62279.779490] [] ? __make_request+0x175/0x26b [62279.779496] [] ? generic_make_request+0x224/0x289 [62279.779502] [] ? submit_bio+0xb3/0xbc [62279.779509] [] ? dm_any_congested+0x4f/0x57 [62279.779516] [] ? run_scheduled_bios+0x246/0x3b1 [62279.779523] [] ? worker_loop+0x180/0x4bb [62279.779529] [] ? btrfs_queue_worker+0x24e/0x24e [62279.779535] [] ? kthread+0x7a/0x82 [62279.779542] [] ? kernel_thread_helper+0x4/0x10 [62279.779548] [] ? kthread_worker_fn+0x149/0x149 [62279.779554] [] ? gs_change+0xb/0xb [62279.779560] btrfs-transacti D 0001 3856 4689 2 0x [62279.779568] 88
Re: Applications using fsync cause hangs for several seconds every few minutes
Dave, good to have a test case on the 3.0 kernel. do you have btrfs as root fs ? and can you show how are you using the btrfs mainly I would need 'btrfs fi show' let me try if I can reproduce. Thanks, Anand I've been simply living with this issue. I can reproduce it by rsyncing very large files to a btrfs volume. My entire desktop will freeze for up to three minutes and no amount of nice/ionice can temper this. Once I've finished the rsync certain apps will periodically hang (Firefox in particular). This behavior goes away after a reboot. I'm running kernel version 3.0. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs: fix d_off in the first dirent
(2011/08/18 11:12), Li Zefan wrote:
>> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
>> index 15fceef..9c1297b 100644
>> --- a/fs/btrfs/inode.c
>> +++ b/fs/btrfs/inode.c
>> @@ -4125,7 +4125,8 @@ static int btrfs_real_readdir(struct file *filp, void
>> *dirent,
>>
>> /* special case for "." */
>> if (filp->f_pos == 0) {
>> -over = filldir(dirent, ".", 1, 1, btrfs_ino(inode), DT_DIR);
>> +over = filldir(dirent, ".", 1,
>> + filp->f_pos, inode->i_ino, DT_DIR);
>
> please stick to btrfs_ino().
Wow, I made a misstep on rebase...
Here is the updated one.
Thanks,
H.Seto
=
Since the d_off in the first dirent for "." (that originates from
the 4th argument "offset" of filldir() for the 2nd dirent for "..")
is wrongly assigned in btrfs_real_readdir(), telldir returns same
offset for different locations.
| # mkfs.btrfs /dev/sdb1
| # mount /dev/sdb1 fs0
| # cd fs0
| # touch file0 file1
| # ../test
| telldir: 0
| readdir: d_off = 2, d_name = "."
| telldir: 2
| readdir: d_off = 2, d_name = ".."
| telldir: 2
| readdir: d_off = 3, d_name = "file0"
| telldir: 3
| readdir: d_off = 2147483647, d_name = "file1"
| telldir: 2147483647
To fix this problem, pass filp->f_pos (which is loff_t) instead.
| # ../test
| telldir: 0
| readdir: d_off = 1, d_name = "."
| telldir: 1
| readdir: d_off = 2, d_name = ".."
| telldir: 2
| readdir: d_off = 3, d_name = "file0"
:
At the moment the "offset" for "." is unused because there is no
preceding dirent, however it is better to pass filp->f_pos to follow
grammatical usage.
Signed-off-by: Hidetoshi Seto
---
fs/btrfs/inode.c |5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 15fceef..addf025 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -4125,7 +4125,8 @@ static int btrfs_real_readdir(struct file *filp, void
*dirent,
/* special case for "." */
if (filp->f_pos == 0) {
- over = filldir(dirent, ".", 1, 1, btrfs_ino(inode), DT_DIR);
+ over = filldir(dirent, ".", 1,
+ filp->f_pos, btrfs_ino(inode), DT_DIR);
if (over)
return 0;
filp->f_pos = 1;
@@ -4134,7 +4135,7 @@ static int btrfs_real_readdir(struct file *filp, void
*dirent,
if (filp->f_pos == 1) {
u64 pino = parent_ino(filp->f_path.dentry);
over = filldir(dirent, "..", 2,
- 2, pino, DT_DIR);
+ filp->f_pos, pino, DT_DIR);
if (over)
return 0;
filp->f_pos = 2;
--
1.7.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs: fix d_off in the first dirent
> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index 15fceef..9c1297b 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -4125,7 +4125,8 @@ static int btrfs_real_readdir(struct file *filp, void
> *dirent,
>
> /* special case for "." */
> if (filp->f_pos == 0) {
> - over = filldir(dirent, ".", 1, 1, btrfs_ino(inode), DT_DIR);
> + over = filldir(dirent, ".", 1,
> +filp->f_pos, inode->i_ino, DT_DIR);
please stick to btrfs_ino().
> if (over)
> return 0;
> filp->f_pos = 1;
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Honest timeline for btrfsck
Chris Mason oracle.com> writes: > > Aside from making sure the kernel code is stable, btrfsck is all I'm > working on right now. I do expect a release in the next two weeks that > can recover your data (and many others). > > Thanks, > Chris > -- Chris, We're all on the edge of our seats. Can you provide an updated ETA on the release of the first functional btrfsck tool? No pressure or anything ;) -Yalonda -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] btrfs auto snapshot
Hello Ken and all, On 2011-08-17 T 19:38 +0200 Matthias G. Eckermann wrote: > P.S.: I also added "snapper" itself there. I am not sure > though, if it will build out of the box. ... Stay tuned. A dinner later, the packages (.rpm/.src.rpm) for blocxx and also snapper are available in the openSUSE Buildservice at: http://download.opensuse.org/repositories/home:/mge1512:/snapper/ for Fedora_15 RHEL-6 SLE_11_SP1 openSUSE_Factory The download directory is a YUM repository and should easily work with either of the distributions. Enjoy! so long - MgE P.S.: Again, if someone sends the packaging files for Debian/Ubuntu, I can also add those. Unfortunately my .deb experience is limited, ... -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX Products GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] btrfs auto snapshot
Hi Anand, On Wed, Aug 17, 2011 at 11:24, Anand Jain wrote: > And a rough implementation design is here below. (As of now this does > not include the GNOME integration since I have no idea how to do that). Very cool idea! With regards to the Gnome integration, you might want to take a look at what the Solaris folks did for Nautilus - I really liked the UI and the integration with ZFS: http://src.opensolaris.org/source/xref/jds/spec-files/branches/gnome-2-30/patches/nautilus-14-zfs-snapshot.diff Bye, LenZ -- Lenz Grimmer - http://www.lenzg.net/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs: fix warning in iput for bad-inode
iput() shouldn't be called for inodes in I_NEW state,
lets call __destroy_inode() and btrfs_destroy_inode() instead
[1.871723] WARNING: at fs/inode.c:1309 iput+0x1d9/0x200()
[1.873722] Modules linked in:
[1.873722] Pid: 1, comm: swapper Tainted: GW 3.1.0-rc2-zurg #58
[1.875722] Call Trace:
[1.875722] [] ? iput+0x1d9/0x200
[1.876722] [] warn_slowpath_common+0x7a/0xb0
[1.877722] [] warn_slowpath_null+0x15/0x20
[1.879722] [] iput+0x1d9/0x200
[1.879722] [] btrfs_iget+0x1c4/0x450
[1.881721] [] ? btrfs_tree_read_unlock_blocking+0x3b/0x60
[1.882721] [] ? kmem_cache_free+0x2a/0x160
[1.883721] [] btrfs_lookup_dentry+0x413/0x490
[1.885721] [] ? get_parent_ip+0x11/0x50
[1.886720] [] btrfs_lookup+0x11/0x30
[1.887720] [] d_alloc_and_lookup+0x40/0x80
[1.888720] [] ? d_lookup+0x30/0x50
[1.889720] [] do_lookup+0x288/0x370
[1.890720] [] ? get_parent_ip+0x11/0x50
[1.891720] [] do_last+0xe0/0x910
[1.892720] [] path_openat+0xcd/0x3a0
[1.893719] [] ? wait_for_xmitr+0x3b/0xa0
[1.895719] [] ? put_dec_full+0x5a/0xb0
[1.896719] [] ? serial8250_console_putchar+0x2b/0x40
[1.897719] [] do_filp_open+0x3d/0xa0
[1.898719] [] ? get_parent_ip+0x11/0x50
[1.899718] [] ? get_parent_ip+0x11/0x50
[1.900718] [] ? sub_preempt_count+0x9d/0xd0
[1.902718] [] open_exec+0x2d/0xf0
[1.903718] [] do_execve_common.isra.32+0x12f/0x340
[1.906717] [] do_execve+0x16/0x20
[1.907717] [] sys_execve+0x42/0x70
[1.908717] [] kernel_execve+0x68/0xd0
[1.909717] [] ? run_init_process+0x1e/0x20
[1.911717] [] init_post+0x8e/0xc0
[1.912716] [] kernel_init+0x13d/0x13d
[1.913716] [] kernel_thread_helper+0x4/0x10
[1.914716] [] ? start_kernel+0x33f/0x33f
[1.915716] [] ? gs_change+0xb/0xb
Signed-off-by: Konstantin Khlebnikov
---
fs/btrfs/inode.c | 10 +++---
1 files changed, 3 insertions(+), 7 deletions(-)
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 15fceef..3e949bd 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -3952,7 +3952,6 @@ struct inode *btrfs_iget(struct super_block *s, struct
btrfs_key *location,
struct btrfs_root *root, int *new)
{
struct inode *inode;
- int bad_inode = 0;
inode = btrfs_iget_locked(s, location->objectid, root);
if (!inode)
@@ -3968,15 +3967,12 @@ struct inode *btrfs_iget(struct super_block *s, struct
btrfs_key *location,
if (new)
*new = 1;
} else {
- bad_inode = 1;
+ __destroy_inode(inode);
+ btrfs_destroy_inode(inode);
+ inode = ERR_PTR(-ESTALE);
}
}
- if (bad_inode) {
- iput(inode);
- inode = ERR_PTR(-ESTALE);
- }
-
return inode;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] btrfs auto snapshot
On 2011-08-17 T 09:50 -0500 Ken A wrote: > and much easier but less powerful:-) > than trying to get snapper to compile on fedora libblocxx > ? :-) Ah, sure. Sorry. Packages for "blocxx" for: Fedora_14 Fedora_15 RHEL-5 RHEL-6 SLE_11_SP1 openSUSE_11.4 openSUSE_Factory are available in the openSUSE buildservice at: http://download.opensuse.org/repositories/home:/mge1512:/snapper/ See also the build project at: https://build.opensuse.org/project/show?project=home%3Amge1512%3Asnapper If someone sends the packaging files for Debian/Ubuntu, I can also add those. so long - MgE P.S.: I also added "snapper" itself there. I am not sure though, if it will build out of the box. ... Stay tuned. -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX Products GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: snapshot ctime // Re: [RFC] btrfs auto snapshot
On Wed, Aug 17, 2011 at 11:13 AM, Roman Mamedov wrote: > So until someone cares about snapshot ctime enough to fix this, btrfs will > not be a convenient FS to work with timed snapshotting/cleanup. Isn't the ctime the creation date of the original folder? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Honest timeline for btrfsck
On Wed, Aug 03, 2011 at 04:53:26PM -0400, Chris Mason wrote: > I do expect a release in the next two weeks that can recover your data (and > many others). I actually set an appointment reminder in my Blackberry for the two week anniversary of this email. I expect today will be a milestone in the btrfs ecosystem (lack of fsck being the most frequent reason for not trying btrfs). I know I've got two existing instances that I can test this tool on. -- -=[dave]=- Entropy isn't what it used to be. pgpwZXDAxQRKk.pgp Description: PGP signature
snapshot ctime // Re: [RFC] btrfs auto snapshot
On Wed, 17 Aug 2011 10:04:33 -0400 Dave wrote: > I've already done something similar. I take hourly, daily, weekly, and > monthly > snapshots of my /home subvolume. Here's the script I've created for this: On one machine I make hourly snapshots of my /home and of the root FS as well. The tricky part is actually not the snapshotting, but the deletion of outdated snapshots. That's due to the unfortunate fact (bug?), that snapshot-directories do not have their ctime set correctly at all, they have some totally bogus ctime instead. .../snaps/$ ls -la --time=ctime | tail dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-05@02-31-51 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-06@02-49-29 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-07@00-17-40 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-08@01-53-29 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-10@03-09-32 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-12@00-26-54 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-13@01-40-19 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-14@04-22-07 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-15@02-29-13 dr-xr-xr-x 1 root root 102 2011-06-28 12:29 2011-08-16@10-27-57 As you can see I have to store creation date/time in the snapshot name, and then parse it out to delete snapshots e.g. older than 3 months. So until someone cares about snapshot ctime enough to fix this, btrfs will not be a convenient FS to work with timed snapshotting/cleanup. -- With respect, Roman signature.asc Description: PGP signature
Re: [RFC] btrfs auto snapshot
and much easier than trying to get snapper to compile on fedora libblocxx ? :-) Ken On 8/17/2011 9:04 AM, Dave wrote: I've already done something similar. I take hourly, daily, weekly, and monthly snapshots of my /home subvolume. Here's the script I've created for this: #! /bin/bash if [ "$#" -ne 2 ]; then echo Usage $0 SNAPSHOT_PREFIX NUM_SNAPSHOTS exit 1 fi SNAPS=/var/lib/btrfs-root/__snapshot/home btrfs subvolume snapshot -r /home $SNAPS/$1_$(date +%F_%T.%N) num_snaps=$(ls -1d $SNAPS/$1_* | sort | wc -l) if [ "$num_snaps" -gt "$2" ]; then let over=$num_snaps-$2 ls -1d $SNAPS/$1_* | head -n $over | while read s; do btrfs subvolume delete $s done fi Here's my crontab: 0 * * * * /usr/local/bin/snapshot hourly 6 0 0 * * * /usr/local/bin/snapshot daily 7 0 0 * * 0 /usr/local/bin/snapshot weekly 4 -- Ken Anderson -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Applications using fsync cause hangs for several seconds every few minutes
On Wed, Aug 17, 2011 at 10:38:42AM -0400, Andrew Guertin wrote: > Well I'd expect it to be somewhat uncommon, or it wouldn't survive 3 > kernel versions :) But at least 3 people have reported it, and for me at > least it's reliably reproducible enough to bisect, so I'm quite certain > there's something going on. I've been simply living with this issue. I can reproduce it by rsyncing very large files to a btrfs volume. My entire desktop will freeze for up to three minutes and no amount of nice/ionice can temper this. Once I've finished the rsync certain apps will periodically hang (Firefox in particular). This behavior goes away after a reboot. I'm running kernel version 3.0. -- -=[dave]=- Entropy isn't what it used to be. pgpGtgOCW3Zae.pgp Description: PGP signature
Re: Applications using fsync cause hangs for several seconds every few minutes
On 08/17/2011 10:29 AM, Michael Cronenworth wrote: > Andrew Guertin on 08/17/2011 09:24 AM wrote: >> I (and presumably others) haven't >> been able to upgrade my kernel past 2.6.38 because of this. > > I'm running kernel 3.0 (Fedora 15's 2.6.40) on two boxes and I have not > seen slow downs or hangs. I use Firefox. Well I'd expect it to be somewhat uncommon, or it wouldn't survive 3 kernel versions :) But at least 3 people have reported it, and for me at least it's reliably reproducible enough to bisect, so I'm quite certain there's something going on. --Andrew -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Applications using fsync cause hangs for several seconds every few minutes
Andrew Guertin on 08/17/2011 09:24 AM wrote: I (and presumably others) haven't been able to upgrade my kernel past 2.6.38 because of this. I'm running kernel 3.0 (Fedora 15's 2.6.40) on two boxes and I have not seen slow downs or hangs. I use Firefox. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Applications using fsync cause hangs for several seconds every few minutes
On 08/09/2011 05:29 PM, Andrew Guertin wrote: > On 06/21/2011 01:15 PM, Jan Stilow wrote: >> Hello, >> >> Nirbheek Chauhan gentoo.org> writes: >>> [...] >>> >>> Every few minutes, (I guess) when applications do fsync (firefox, >>> xchat, vim, etc), all applications that use fsync() hang for several >>> seconds, and applications that use general IO suffer extreme >>> slowdowns. iotop shows various combinations of the processes listed >>> below doing writes, and the total write as 2-3MB/s. >>> >>> [btrfs-dealloc-] >>> [btrfs-submit-0] >>> [btrfs-transacti] >>> [btrfs-endio-wri] >>> [flush-btrfs-1] >> >> I'm using btrfs under a 2.6.39-ARCH kernel and run into the same issue. >> >> In my case the [btrfs-submit-0] and [btrfs-transacti] shows up in iotop >> and produce 99% of IO at the time a application is frozen. For something >> like 10 to 30 seconds. >> >> [...] > > I see the same issue. I have bisected it to > > 4e69b598f6cfb0940b75abf7e179d6020e94ad1e is the first bad commit > commit 4e69b598f6cfb0940b75abf7e179d6020e94ad1e > Author: Josef Bacik > Date: Mon Mar 21 10:11:24 2011 -0400 > > Btrfs: cleanup how we setup free space clusters > > ...which came in between 2.6.38 and 2.6.39. Any chance of someone looking at this? I (and presumably others) haven't been able to upgrade my kernel past 2.6.38 because of this. --Andrew -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] btrfs auto snapshot
I've already done something similar. I take hourly, daily, weekly, and monthly snapshots of my /home subvolume. Here's the script I've created for this: #! /bin/bash if [ "$#" -ne 2 ]; then echo Usage $0 SNAPSHOT_PREFIX NUM_SNAPSHOTS exit 1 fi SNAPS=/var/lib/btrfs-root/__snapshot/home btrfs subvolume snapshot -r /home $SNAPS/$1_$(date +%F_%T.%N) num_snaps=$(ls -1d $SNAPS/$1_* | sort | wc -l) if [ "$num_snaps" -gt "$2" ]; then let over=$num_snaps-$2 ls -1d $SNAPS/$1_* | head -n $over | while read s; do btrfs subvolume delete $s done fi Here's my crontab: 0 * * * * /usr/local/bin/snapshot hourly 6 0 0 * * * /usr/local/bin/snapshot daily 7 0 0 * * 0 /usr/local/bin/snapshot weekly 4 -- -=[dave]=- Entropy isn't what it used to be. pgp09202hq1TV.pgp Description: PGP signature
Re: [RFC] btrfs auto snapshot
Hello Anand and all, On 2011-08-17 T 10:15 +0800 Anand Jain wrote: > Appears that no one is working on the auto-snapshot feature for > btrfs, so here I am implementing the same. thanks for bringing this up! The group of features you are listing is indeed of high interest for people using btrfs. That said, not only have other people though about this, but a lot of your question already have been implemented in "snapper", and open source infrastructure developed as part of openSUSE and SUSE Linux Enterprise. Please see: http://en.opensuse.org/Portal:Snapper http://en.opensuse.org/openSUSE:Snapper_install http://lizards.opensuse.org/2011/04/01/introducing-snapper/ Source code is here: http://gitorious.org/opensuse/snapper "snapper" will be part of openSUSE 12.1 and SUSE Linux Enterprise 11 Service Pack 2, and is available as part of the respective Beta releases and Milestones already. snapper's concept in short: - shared library to make the functionality available to other tools as well - libsnapper is implemented on top of the btrfsprogs - cmdline tool "snapper" - global configuration file /etc/sysconfig/snapper - one configuration file per subvolume to be snapshotted /etc/snapper/configs/ I call this a "single configuration" going forward. Here also policies for time based snapshotting and cleanup are to be configured. - Integration into SUSE's management framework (YaST2/zypper), however, "snapper" should work independent of those, i.e. usable on other distributions easily. > Below is a draft on the feature list. Any comments / questions / > suggestions are welcome, please do let me know. Let me go through the single features quickly and list the matching snapper functionality. > btrfs auto snapshot feature will include: > Initially: > - configurable timely snapshots Yes. Configured per single configuration > - uses services and crontab to schedule Yes. > - Gnome integration I more see a need for integration into systems management frameworks. > - snapshot rollback and cleanups Yes. Rules for cleanups (time based, number of snapshots) per single configuration. > - snapshot trashing based on available space // not yet done. > - snapshot destination will be subvol/.btrfs/snapshot@ and snapshot destination is "/.snapshots//", >snapshot/.btrfs/snapshot@ for subvolume and snapshot >respectively Timestamp and Description of a snapshot are stored in a small XML file /.snapshots//info.xml". One small file per snapshot. [...] > Challenges: >- rollback per file or dir instead of entire snapshot-rollback ? snapper implements "rollback" on a FILE level only. To differentiate this way of "rolling back" from jumping into another snapshot, we call it "undochange" for now. This keeps the option to also manage a full per snapshot-rollback in a later point int time. [...] > modify the snapshot - do we need to implement a kind of read-only > snapshot ? snapper treats snapshots as read only snapshots, i.e. when doing a rollback - aehem, I should say "undochange" - only the "master" volume will be changed, not the single snapshots. We are aware that this has pros and cons. But that's another discussion. I hope that this is a starting point for you. Enjoy "snapper". so long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX Products GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] btrfs auto snapshot
> And a rough implementation design is here below. (As of now this does > not include the GNOME integration since I have no idea how to do that). > > Further, implementation will contain 2 new files > /etc/init.d/btrfs and //btrfs-auto-snapshot, > > any idea where does a file like btrfs-auto-snapshot should be placed > (with its purpose as mentioned below). [...] > 2. > Crontab: Upon start, crontab will be updated with something like the > following. > every 15min `/btrfs-auto-snapshot cleanup>` > every 15min `//btrfs-auto-snapshot 15min` > every hr `//btrfs-auto-snapshot hr` > every day `//btrfs-auto-snapshot day` > every month `//btrfs-auto-snapshot month` > every year `//btrfs-auto-snapshot year` I think that you need to be careful not to impose your idea of when to take snapshots and how long to keep them onto the design. For example why take snapshots every 15 minutes? Why not every 10 or every hour? Why treat monthly snapshots as special when it does not fit into most working weeks? would weekly be more logical? What about 2 weekly (When I worked at Nokia, internal releases where done on Tuesday of each even numbered week, so we would have wanted the snapshot taken on that day to be retained longer than snapshots taken on other days, or Tuesdays in odd numbered weeks.) I think a more flexible design would be to allow the user to specify (via a config file for each subvolume) a label for each type of snapshot and how long to keep snapshots depending on when they are taken. This can be done using syntax similar to crontab: Example # Format: # Keep 15 min snapshots for 6 hours. 15_minute */15 * * * *6 hour # Keep hourly snapshots for 5 day hourly 0 * * * * 5 day # Keep daily snapshots for 21 days. Use the noon snapshot daily 0 12 * * * 21 day # Keep weekly snapshots for 6 months, Use the Tuesday snapshot # TODO: we would need a week number field to specify even # numbered weeks only. weekly 0 12 * * 2 6 mon The btrfs-auto-snapshot script would need to make sure that the crontab entry that takes snapshots runs frequently enough. -- David Pottage Error compiling committee.c To many arguments to function. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs: fix d_off in the first dirent
Since the d_off in the first dirent for "." (that originates from
the 4th argument "offset" of filldir() for the 2nd dirent for "..")
is wrongly assigned in btrfs_real_readdir(), telldir returns same
offset for different locations.
| # mkfs.btrfs /dev/sdb1
| # mount /dev/sdb1 fs0
| # cd fs0
| # touch file0 file1
| # ../test
| telldir: 0
| readdir: d_off = 2, d_name = "."
| telldir: 2
| readdir: d_off = 2, d_name = ".."
| telldir: 2
| readdir: d_off = 3, d_name = "file0"
| telldir: 3
| readdir: d_off = 2147483647, d_name = "file1"
| telldir: 2147483647
To fix this problem, pass filp->f_pos (which is loff_t) instead of
the wrong constant value.
| # ../test
| telldir: 0
| readdir: d_off = 1, d_name = "."
| telldir: 1
| readdir: d_off = 2, d_name = ".."
| telldir: 2
| readdir: d_off = 3, d_name = "file0"
:
At the moment the "offset" for "." is unused because there is no
preceding dirent, however it is better to pass filp->f_pos to follow
grammatical usage.
Signed-off-by: Hidetoshi Seto
---
fs/btrfs/inode.c |5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 15fceef..9c1297b 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -4125,7 +4125,8 @@ static int btrfs_real_readdir(struct file *filp, void
*dirent,
/* special case for "." */
if (filp->f_pos == 0) {
- over = filldir(dirent, ".", 1, 1, btrfs_ino(inode), DT_DIR);
+ over = filldir(dirent, ".", 1,
+ filp->f_pos, inode->i_ino, DT_DIR);
if (over)
return 0;
filp->f_pos = 1;
@@ -4134,7 +4135,7 @@ static int btrfs_real_readdir(struct file *filp, void
*dirent,
if (filp->f_pos == 1) {
u64 pino = parent_ino(filp->f_path.dentry);
over = filldir(dirent, "..", 2,
- 2, pino, DT_DIR);
+ filp->f_pos, pino, DT_DIR);
if (over)
return 0;
filp->f_pos = 2;
--
1.7.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] btrfs auto snapshot
And a rough implementation design is here below. (As of now this does not include the GNOME integration since I have no idea how to do that). Further, implementation will contain 2 new files /etc/init.d/btrfs and //btrfs-auto-snapshot, any idea where does a file like btrfs-auto-snapshot should be placed (with its purpose as mentioned below). --- 1. File: /etc/init.d/btrfs : This will contain the global start and stop knob (something like `service btrfs start`) 2. Crontab: Upon start, crontab will be updated with something like the following. every 15min `/btrfs-auto-snapshot cleanup>` every 15min `//btrfs-auto-snapshot 15min` every hr `//btrfs-auto-snapshot hr` every day `//btrfs-auto-snapshot day` every month `//btrfs-auto-snapshot month` every year `//btrfs-auto-snapshot year` 3. File: //btrfs-auto-snapshot : to process the call from cronjob. - check the config and check the target fs for the snapshot - check the space in the target FS - call btrfs cli to take snapshot (provide src and destination) - check if snapshot cleanup is required to process the calls from '/etc/init.d/btrfs' configuration setting. - to set which btrfs fs will participate in the auto-snapshot - to indicate if auto snapshot has to stop when target FS is 80% full - Thanks, Anand On 08/17/2011 10:15 AM, Anand Jain wrote: sorry forgot to follow the protocol, now included RFC in the subject. Hi, Appears that no one is working on the auto-snapshot feature for btrfs, so here I am implementing the same. Below is a draft on the feature list. Any comments / questions / suggestions are welcome, please do let me know. btrfs auto snapshot feature will include: Initially: - configurable timely snapshots - uses services and crontab to schedule - Gnome integration - snapshot rollback and cleanups - snapshot trashing based on available space - snapshot destination will be subvol/.btrfs/snapshot@ and snapshot/.btrfs/snapshot@ for subvolume and snapshot respectively Later: -integration with (a ?) backup software - snapshot trashing per backup confirmation - rollback from the backup server Challenges: - rollback per file or dir instead of entire snapshot-rollback ? - integrating into the btrfs gui ? - some FS (samfs) support continues backup - do we need a new role for the snapshot management ? zfs does. - we don't need a snapshot if the master file-system didn't change at all - snapshots are writable by default, hope that sudo-er doesn't modify the snapshot - do we need to implement a kind of read-only snapshot ? Thanks, -Anand -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
