Abysmal performance when doing a rm -r and rsync backup at the same time
Hi! Today I started my backup script which rsync´s my system to an external 3,5 inch 2 TB harddisk with the wrong destination dir which I notices more than 150 GiB of data has been copied twice instead of diffing with an existing subvolume. Thus I rm -rf the misplaced backup and started the backup script at the same time before taking a bath. After the bath both the backup script and the rm - rf were still running. Disk was 100% utilizized and it didn´t seem to go anywhere. Source of the backup was /home on an Intel SSD 320 which easily outperforms the harddisk. I tried to look how much rm -rf already has deleted with du -sch but that du didn´t really like to complete as well. rm, rsync and du were often in D process state. Then I stopped the backup script and the rm and let the disk settle for a few moments. After it has settled down I did the du -sch and about 150 GiB of data were still undeleted. There could only have been less than 239 GiB of data in there cause the home directory isn´t bigger than that and the rsync backup has not yet been completed. So most of the rm work was not yet done. Well I run the rm command again and it completed rather quickly, say 10 seconds or so. Then I started the backup script and it already completed /home. Also quite quickly. Is such a performance of a rsync versus rm -r issue known? I have no exact measurements, but it virtually took ages. Harddisk was fully utilized, but I didn´t look closely at any throughput or IOPS numbers. Kernel in use: martin@merkaba:~ cat /proc/version Linux version 3.13.0-rc4-tp520 (martin@merkaba) (gcc version 4.8.2 (Debian 4.8.2-10) ) #39 SMP PREEMPT Tue Dec 17 13:57:12 CET 2013 Characteristics of backup data: About 239 GiB, lzo compressed with lots of small mail files (easily a million), but also larger music files. martin@merkaba:~ find -type d | wc -l 36359 martin@merkaba:~ find -type f | wc -l 1090049 martin@merkaba:~ find -type l | wc -l 1337 Mount info: martin@merkaba:~ egrep (home |steigerwald).*btrfs /proc/mounts /dev/dm-1 /home btrfs rw,noatime,compress=lzo,ssd,space_cache 0 0 /dev/sdc1 /mnt/steigerwald btrfs rw,relatime,compress=lzo,space_cache,autodefrag 0 0 Subvolume amount: General, most of them are snapshots: merkaba:/mnt/steigerwald btrfs subvol list . | wc -l 32 Of which are snapshots: merkaba:/mnt/steigerwald btrfs subvol list . | grep -- -20 | wc -l 23 Subvolume merkaba where the backup went into after fixing the path and its snapshots: merkaba:/mnt/steigerwald btrfs subvol list . | grep merkaba | wc -l 13 This is the situation after the rm and most of the backup (just some small remote server left) has been completed: # ./btrfs filesystem disk-usage -t /mnt/steigerwald Data Metadata Metadata System System Single Single DUP Single DUP Unallocated /dev/sdc1 1.43TB 8.00MB 76.00GB 4.00MB 16.00MB322.98GB == == === Total 1.43TB 8.00MB 38.00GB 4.00MB 8.00MB322.98GB Used 1.25TB 0.00 12.45GB 0.00 168.00KB # ./btrfs device disk-usage /mnt/steigerwald /dev/sdc1 1.82TB Data,Single: 1.43TB Metadata,Single: 8.00MB Metadata,DUP:76.00GB System,Single:4.00MB System,DUP: 16.00MB Unallocated:322.98GB # ./btrfs filesystem df /mnt/steigerwald Disk size: 1.82TB Disk allocated:1.50TB Disk unallocated:322.98GB Used: 1.26TB Free (Estimated):521.48GB (Max: 529.46GB, min: 367.96GB) Data to disk ratio: 98 % (yeah I still love the patches by Goffredo regarding disk-usage output :) # btrfs fi show Label: 'steigerwald' uuid: … Total devices 1 FS bytes used 1.26TB devid1 size 1.82TB used 1.50TB path /dev/sdc1 Ciao, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On Tue, 21 Jun 2011 11:24:11 -0400, Calvin Walton wrote: On Mon, 2011-06-20 at 23:51 +0200, Henning Rohlfs wrote: Hello, I've migrated my system to btrfs (raid1) a few months ago. Since then the performance has been pretty bad, but recently it's gotten unbearable: a simple sync called while the system is idle can take 20 up to 60 seconds. Creating or deleting files often has several seconds latency, too. I think I’ve been seeing a fairly similar, or possibly the same? issue as well. It looks like it’s actually a regression introduced in 2.6.39 - if I switch back to a 2.6.38 kernel, my latency issues magically go away! (I'm curious: does using the older 2.6.38.x kernel help with anyone else that's seeing the issue?) Some hardware/configuration details: btrfs on a single disc (Seagate Momentus XT hybrid), lzo compression and space cache enabled. Some snapshots in use. I notice that in latencytop I'm seeing a lot of lines with (cropped) traces like sleep_on_page wait_on_page_bit read_extent_buffer_ 13.3 msec 0.5 % showing up that I didn't see with the 2.6.38 kernel. I occasionally see latencies as bad as 20-30 seconds on operations like fsync or synchronous writes. I think I can reproduce the issue well enough to bisect it, so I might give that a try. It'll be slow going, though. You are right. This seems to be a regression in the .39 kernel. I tested with 2.6.38.2 just now and the performance is back to normal. Thanks, Henning server ~ # uname -a Linux server 2.6.38.2 #1 SMP Thu Apr 14 13:05:35 CEST 2011 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ AuthenticAMD GNU/Linux server ~ # sync; time sync real0m0.144s user0m0.000s sys 0m0.020s server ~ # bonnie++ -d tmp -u 0:0 Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP server 16G 147 97 279933 56 73245 34 1258 78 102379 23 177.3 50 Latency 423ms 103ms 645ms 163ms 404ms 264ms Version 1.96 --Sequential Create-- Random Create server -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 3784 28 + +++ 8519 60 13694 59 + +++ 11710 76 Latency 127ms1024us 18718us 15958us 119us 2459us 1.96,1.96,server,1,1308745595,16G,,147,97,279933,56,73245,34,1258,78,102379,23,177.3,50,16,3784,28,+,+++,8519,60,13694,59,+,+++,11710,76,423ms,103ms,645ms,163ms,404ms,264ms,127ms,1024us,18718us,15958us,119us,2459us -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On 06/22/2011 10:15 AM, Henning Rohlfs wrote: On Tue, 21 Jun 2011 11:24:11 -0400, Calvin Walton wrote: On Mon, 2011-06-20 at 23:51 +0200, Henning Rohlfs wrote: Hello, I've migrated my system to btrfs (raid1) a few months ago. Since then the performance has been pretty bad, but recently it's gotten unbearable: a simple sync called while the system is idle can take 20 up to 60 seconds. Creating or deleting files often has several seconds latency, too. I think I’ve been seeing a fairly similar, or possibly the same? issue as well. It looks like it’s actually a regression introduced in 2.6.39 - if I switch back to a 2.6.38 kernel, my latency issues magically go away! (I'm curious: does using the older 2.6.38.x kernel help with anyone else that's seeing the issue?) Some hardware/configuration details: btrfs on a single disc (Seagate Momentus XT hybrid), lzo compression and space cache enabled. Some snapshots in use. I notice that in latencytop I'm seeing a lot of lines with (cropped) traces like sleep_on_page wait_on_page_bit read_extent_buffer_ 13.3 msec 0.5 % showing up that I didn't see with the 2.6.38 kernel. I occasionally see latencies as bad as 20-30 seconds on operations like fsync or synchronous writes. I think I can reproduce the issue well enough to bisect it, so I might give that a try. It'll be slow going, though. You are right. This seems to be a regression in the .39 kernel. I tested with 2.6.38.2 just now and the performance is back to normal. Would you mind bisecting? You can make it go faster by doing git bisect start fs/ that way only changes in fs are used. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On Wed, 2011-06-22 at 11:39 -0400, Josef Bacik wrote: On 06/22/2011 10:15 AM, Henning Rohlfs wrote: On Tue, 21 Jun 2011 11:24:11 -0400, Calvin Walton wrote: On Mon, 2011-06-20 at 23:51 +0200, Henning Rohlfs wrote: Hello, I've migrated my system to btrfs (raid1) a few months ago. Since then the performance has been pretty bad, but recently it's gotten unbearable: a simple sync called while the system is idle can take 20 up to 60 seconds. Creating or deleting files often has several seconds latency, too. I think I’ve been seeing a fairly similar, or possibly the same? issue as well. It looks like it’s actually a regression introduced in 2.6.39 - if I switch back to a 2.6.38 kernel, my latency issues magically go away! (I'm curious: does using the older 2.6.38.x kernel help with anyone else that's seeing the issue?) I think I can reproduce the issue well enough to bisect it, so I might give that a try. It'll be slow going, though. You are right. This seems to be a regression in the .39 kernel. I tested with 2.6.38.2 just now and the performance is back to normal. Would you mind bisecting? Just before I was going to try bisecting, I tried the 3.0-rc4 kernel out of curiosity. And it seems to be quite a bit better; at the very least, I’m not seeing gui applications stalling for ~10 seconds when doing things like opening or writing files. Latencytop is reporting fsync() latencies staying pretty steady in the range of under 300ms, with occasional outliers at up to 2s, and it's not getting worse with time. I'll still look into doing a bisect between 2.6.38 and 2.6.39, I'm curious what went wrong. -- Calvin Walton calvin.wal...@kepstin.ca -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On 06/22/2011 11:57 AM, Calvin Walton wrote: On Wed, 2011-06-22 at 11:39 -0400, Josef Bacik wrote: On 06/22/2011 10:15 AM, Henning Rohlfs wrote: On Tue, 21 Jun 2011 11:24:11 -0400, Calvin Walton wrote: On Mon, 2011-06-20 at 23:51 +0200, Henning Rohlfs wrote: Hello, I've migrated my system to btrfs (raid1) a few months ago. Since then the performance has been pretty bad, but recently it's gotten unbearable: a simple sync called while the system is idle can take 20 up to 60 seconds. Creating or deleting files often has several seconds latency, too. I think I’ve been seeing a fairly similar, or possibly the same? issue as well. It looks like it’s actually a regression introduced in 2.6.39 - if I switch back to a 2.6.38 kernel, my latency issues magically go away! (I'm curious: does using the older 2.6.38.x kernel help with anyone else that's seeing the issue?) I think I can reproduce the issue well enough to bisect it, so I might give that a try. It'll be slow going, though. You are right. This seems to be a regression in the .39 kernel. I tested with 2.6.38.2 just now and the performance is back to normal. Would you mind bisecting? Just before I was going to try bisecting, I tried the 3.0-rc4 kernel out of curiosity. And it seems to be quite a bit better; at the very least, I’m not seeing gui applications stalling for ~10 seconds when doing things like opening or writing files. Latencytop is reporting fsync() latencies staying pretty steady in the range of under 300ms, with occasional outliers at up to 2s, and it's not getting worse with time. I'll still look into doing a bisect between 2.6.38 and 2.6.39, I'm curious what went wrong. Yeah that makes two of us :). There were some other plugging changes that went into to 38, so maybe bisect all of the kernel, not just fs/ just in case it was those and not us. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On Mon, 20 Jun 2011 20:12:16 -0400, Josef Bacik wrote: On 06/20/2011 05:51 PM, Henning Rohlfs wrote: Hello, I've migrated my system to btrfs (raid1) a few months ago. Since then the performance has been pretty bad, but recently it's gotten unbearable: a simple sync called while the system is idle can take 20 up to 60 seconds. Creating or deleting files often has several seconds latency, too. One curious - but maybe unrelated - observation is that even though I'm using a raid1 btrfs setup, the hdds are often being written to sequentially. One hard-drive sees some write activity and after it subsides, the other drive sees some activity. (See attached sequential-writes.txt.) Can you do sysrq+w while this is happening so we can see who is doing the writing? Thanks, Josef When I call sync, it starts with several seconds of 100% (one core) cpu usage by sync itself. Afterwards btrfs-submit-0 and sync are blocked. sysrq+w output is attached.SysRq : Show Blocked State taskPC stack pid father btrfs-submit-0 D 88021c8bbf00 0 851 2 0x 880245b07b20 0046 0048 880245b06010 880246ce2b80 00011340 880245b07fd8 4000 880245b07fd8 00011340 88021f2d95c0 880246ce2b80 Call Trace: [81333f7a] ? put_device+0x12/0x14 [81343af6] ? scsi_request_fn+0x341/0x40d [812a4643] ? __blk_run_queue+0x16/0x18 [814fdc2f] io_schedule+0x51/0x66 [812a6bd9] get_request_wait+0xa1/0x12f [81050c85] ? wake_up_bit+0x25/0x25 [812a3585] ? elv_merge+0x99/0xa9 [812a6dee] __make_request+0x187/0x27f [812a540f] generic_make_request+0x229/0x2a4 [812a5553] submit_bio+0xc9/0xe8 [81267b24] run_scheduled_bios+0x296/0x415 [81044692] ? del_timer+0x83/0x83 [81267cb3] pending_bios_fn+0x10/0x12 [8126d591] worker_loop+0x189/0x4b7 [8126d408] ? btrfs_queue_worker+0x263/0x263 [8126d408] ? btrfs_queue_worker+0x263/0x263 [810508bc] kthread+0x7d/0x85 [81500c94] kernel_thread_helper+0x4/0x10 [8105083f] ? kthread_worker_fn+0x13a/0x13a [81500c90] ? gs_change+0xb/0xb syncD 000102884001 0 22091 19401 0x 88019f0edc98 0086 8801 88019f0ec010 880227808000 00011340 88019f0edfd8 4000 88019f0edfd8 00011340 8181f020 880227808000 Call Trace: [810b189b] ? add_partial+0x1b/0x64 [810b36d6] ? kmem_cache_free+0x8e/0x93 [812623ad] ? free_extent_state+0x43/0x47 [81086c2b] ? __lock_page+0x68/0x68 [814fdc2f] io_schedule+0x51/0x66 [81086c34] sleep_on_page+0x9/0xd [814fe32c] __wait_on_bit+0x43/0x76 [81086df3] wait_on_page_bit+0x6d/0x74 [81050cb9] ? autoremove_wake_function+0x34/0x34 [81086b0e] ? find_get_page+0x19/0x66 [81247d60] btrfs_wait_marked_extents+0xeb/0x124 [81247ee6] btrfs_write_and_wait_marked_extents+0x2a/0x3c [81247f3a] btrfs_write_and_wait_transaction+0x42/0x44 [81248676] btrfs_commit_transaction+0x53c/0x650 [81050c85] ? wake_up_bit+0x25/0x25 [810db775] ? __sync_filesystem+0x75/0x75 [8122a33a] btrfs_sync_fs+0x66/0x6b [810db761] __sync_filesystem+0x61/0x75 [810db786] sync_one_sb+0x11/0x13 [810bc89c] iterate_supers+0x67/0xbd [810db7c8] sys_sync+0x40/0x57 [814fff7b] system_call_fastpath+0x16/0x1b Sched Debug Version: v0.10, 2.6.39 #3 ktime : 141912370.788052 sched_clk : 141831001.710073 cpu_clk : 141912370.788875 jiffies : 4337451011 sched_clock_stable : 0 sysctl_sched .sysctl_sched_latency: 12.00 .sysctl_sched_min_granularity: 1.50 .sysctl_sched_wakeup_granularity : 2.00 .sysctl_sched_child_runs_first : 0 .sysctl_sched_features : 7279 .sysctl_sched_tunable_scaling: 1 (logaritmic) cpu#0, 2009.138 MHz .nr_running: 0 .load : 0 .nr_switches : 145331671 .nr_load_updates : 14210439 .nr_uninterruptible: 0 .next_balance : 4337.451012 .curr-pid : 0 .clock : 141912370.762325 .cpu_load[0] : 0 .cpu_load[1] : 0 .cpu_load[2] : 14 .cpu_load[3] : 87 .cpu_load[4] : 139 .yld_count : 20406719 .sched_switch : 0 .sched_count : 167876926 .sched_goidle : 51210495 .avg_idle
Re: Abysmal Performance
Henning Rohlfs wrote (ao): - space_cache was enabled, but it seemed to make the problem worse. It's no longer in the mount options. space_cache is a one time mount option which enabled space_cache. Not supplying it anymore as a mount option has no effect (dmesg | grep btrfs). Sander -- Humilis IT Services and Solutions http://www.humilis.net -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On Tue, 21 Jun 2011 10:00:59 +0200, Sander wrote: Henning Rohlfs wrote (ao): - space_cache was enabled, but it seemed to make the problem worse. It's no longer in the mount options. space_cache is a one time mount option which enabled space_cache. Not supplying it anymore as a mount option has no effect (dmesg | grep btrfs). I'm sure that after the first reboot after removing the flag from the mount options, the system was faster for a while. That must have been a coincidence (or just an error on my part). Anyway, I rebooted with clear_cache as mount option and there was no improvement either. Thanks for pointing this out, Henning -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On 06/21/2011 05:26 AM, Henning Rohlfs wrote: On Tue, 21 Jun 2011 10:00:59 +0200, Sander wrote: Henning Rohlfs wrote (ao): - space_cache was enabled, but it seemed to make the problem worse. It's no longer in the mount options. space_cache is a one time mount option which enabled space_cache. Not supplying it anymore as a mount option has no effect (dmesg | grep btrfs). I'm sure that after the first reboot after removing the flag from the mount options, the system was faster for a while. That must have been a coincidence (or just an error on my part). No, the space cache will make your system faster _after_ having been enabled once. The reason for this is because we have to build the cache the slow way at first, and then after that we can do it the fast way. What is probably happening is your box is slowing down trying to build this cache. Don't mount with clear_cache unless there is a bug in your cache. Let it do it's thing and stuff will get faster. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On Mon, 2011-06-20 at 23:51 +0200, Henning Rohlfs wrote: Hello, I've migrated my system to btrfs (raid1) a few months ago. Since then the performance has been pretty bad, but recently it's gotten unbearable: a simple sync called while the system is idle can take 20 up to 60 seconds. Creating or deleting files often has several seconds latency, too. I think I’ve been seeing a fairly similar, or possibly the same? issue as well. It looks like it’s actually a regression introduced in 2.6.39 - if I switch back to a 2.6.38 kernel, my latency issues magically go away! (I'm curious: does using the older 2.6.38.x kernel help with anyone else that's seeing the issue?) Some hardware/configuration details: btrfs on a single disc (Seagate Momentus XT hybrid), lzo compression and space cache enabled. Some snapshots in use. I notice that in latencytop I'm seeing a lot of lines with (cropped) traces like sleep_on_page wait_on_page_bit read_extent_buffer_ 13.3 msec 0.5 % showing up that I didn't see with the 2.6.38 kernel. I occasionally see latencies as bad as 20-30 seconds on operations like fsync or synchronous writes. I think I can reproduce the issue well enough to bisect it, so I might give that a try. It'll be slow going, though. -- Calvin Walton calvin.wal...@kepstin.ca -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Abysmal Performance
On Tue, 21 Jun 2011 11:18:30 -0400, Josef Bacik wrote: On 06/21/2011 05:26 AM, Henning Rohlfs wrote: On Tue, 21 Jun 2011 10:00:59 +0200, Sander wrote: Henning Rohlfs wrote (ao): - space_cache was enabled, but it seemed to make the problem worse. It's no longer in the mount options. space_cache is a one time mount option which enabled space_cache. Not supplying it anymore as a mount option has no effect (dmesg | grep btrfs). I'm sure that after the first reboot after removing the flag from the mount options, the system was faster for a while. That must have been a coincidence (or just an error on my part). No, the space cache will make your system faster _after_ having been enabled once. The reason for this is because we have to build the cache the slow way at first, and then after that we can do it the fast way. What is probably happening is your box is slowing down trying to build this cache. Don't mount with clear_cache unless there is a bug in your cache. Let it do it's thing and stuff will get faster. I'm just reporting what I experienced. I had space_cache in the mount options while the problem developed and removed it when the system got too slow. After the next reboot the system was responsive for a short time (an hour maybe - which seems to have been unrelated to the mount option though from what you described). Now there's no difference whatsoever between no options, space_cache and clear_cache. To sum it up: I only played with the clear_cache option because the system got too slow in the first place. I don't see how the problem can be related to this option if changing it it makes no difference. Thanks, Henning -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Abysmal Performance
Hello, I've migrated my system to btrfs (raid1) a few months ago. Since then the performance has been pretty bad, but recently it's gotten unbearable: a simple sync called while the system is idle can take 20 up to 60 seconds. Creating or deleting files often has several seconds latency, too. One curious - but maybe unrelated - observation is that even though I'm using a raid1 btrfs setup, the hdds are often being written to sequentially. One hard-drive sees some write activity and after it subsides, the other drive sees some activity. (See attached sequential-writes.txt.) - 64bit gentoo with vanilla 2.6.39 kernel - lzo compression enabled - 2x WD1000FYPS (1TB WD hdds) - Athlon x2 2.2GHz with 8GB RAM - space_cache was enabled, but it seemed to make the problem worse. It's no longer in the mount options. Any help is appreciated. Thanks, Henning server ~ # sync; time sync real0m28.869s user0m0.000s sys 0m5.750s server ~ # uname -a Linux server 2.6.39 #3 SMP Sat May 28 17:25:31 CEST 2011 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ AuthenticAMD GNU/Linux server ~ # mount | grep btrfs /dev/sdb2 on / type btrfs (rw,noatime,compress=lzo,noacl) /dev/sda2 on /mnt/pool type btrfs (rw,noatime,subvolid=0,compress=lzo) /dev/sda2 on /usr/portage type btrfs (rw,noatime,subvol=newportage,compress=lzo) /dev/sda2 on /home type btrfs (rw,noatime,subvol=home,compress=lzo) /dev/sda2 on /home/mythtv type btrfs (rw,noatime,subvol=mythtv,compress=lzo) server ~ # btrfs fi show Label: none uuid: 7676eb78-e411-4505-ac51-ccd12aa5a6b6 Total devices 2 FS bytes used 281.58GB devid1 size 931.28GB used 898.26GB path /dev/sda2 devid3 size 931.27GB used 898.26GB path /dev/sdb2 Btrfs v0.19-35-g1b444cd-dirty server ~ # btrfs fi df / Data, RAID1: total=875.00GB, used=279.30GB System, RAID1: total=8.00MB, used=140.00KB System: total=4.00MB, used=0.00 Metadata, RAID1: total=23.25GB, used=2.28GB bonnie++ Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP server 16G 147 90 76321 18 31787 16 1370 71 64812 14 27.0 66 Latency 66485us7581ms4455ms 25011us 695ms 959ms Version 1.96 --Sequential Create-- Random Create server -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 238 51 + +++ 219 51 284 52 + +++ 390 57 Latency 1914ms 524us3461ms1141ms 39us 1308ms 1.96,1.96,server,1,1308618030,16G,,147,90,76321,18,31787,16,1370,71,64812,14,27.0,66,16,238,51,+,+++,219,51,284,52,+,+++,390,57,66485us,7581ms,4455ms,25011us,695ms,959ms,1914ms,524us,3461ms,1141ms,39us,1308ms server ~ # iostat -m 5 avg-cpu: %user %nice %system %iowait %steal %idle 3.000.00 45.205.200.00 46.60 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 0.00 0.00 0.00 0 0 sdb 15.20 0.06 0.00 0 0 md0 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 4.300.00 37.46 42.360.00 15.88 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 45.00 0.00 0.38 0 1 sdb 467.60 0.02 2.06 0 10 md0 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 4.310.00 19.34 58.820.00 17.54 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 8.80 0.00 0.04 0 0 sdb 649.80 0.02 2.67 0 13 md0 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 3.200.00 63.24 31.970.001.60 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 585.80 0.00 2.36 0 11 sdb 20.80 0.08 0.16 0 0 md0 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 3.300.00 42.60 39.300.00 14.80 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 514.20 0.00 2.29 0 11 sdb 59.20 0.10 0.17 0 0 md0 0.00
Re: Abysmal Performance
On 06/20/2011 05:51 PM, Henning Rohlfs wrote: Hello, I've migrated my system to btrfs (raid1) a few months ago. Since then the performance has been pretty bad, but recently it's gotten unbearable: a simple sync called while the system is idle can take 20 up to 60 seconds. Creating or deleting files often has several seconds latency, too. One curious - but maybe unrelated - observation is that even though I'm using a raid1 btrfs setup, the hdds are often being written to sequentially. One hard-drive sees some write activity and after it subsides, the other drive sees some activity. (See attached sequential-writes.txt.) Can you do sysrq+w while this is happening so we can see who is doing the writing? Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Peter Stuge pe...@stuge.se wrote: Hey, defragging btrfs does not seem to work for me. I have run the filefrag command over the whole fs and (manually) tried to defrag a few heavily fragmented files, but I don't get it to work (it still has the same number of extends and they are horrently uncorrelated) root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found root@schleppi:~# btrfs filesystem defrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found I'm using Ubuntu Natty (2.6.38.4) and tried both btrfs-tools from Natty (201006xx) and from Debian experimental (git from 20101101). Both show the same symptoms. I don't think fragmentation is bad on this box (due to having an SSD), but my system at home is getting dog slow and I'd like to try that when I come home end of the week. Best Regards, Bernhard -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
On Tue, May 3, 2011 at 4:33 AM, Bernhard Schmidt be...@birkenwald.de wrote: Peter Stuge pe...@stuge.se wrote: Hey, defragging btrfs does not seem to work for me. I have run the filefrag command over the whole fs and (manually) tried to defrag a few heavily fragmented files, but I don't get it to work (it still has the same number of extends and they are horrently uncorrelated) root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found root@schleppi:~# btrfs filesystem defrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found I'm using Ubuntu Natty (2.6.38.4) and tried both btrfs-tools from Natty (201006xx) and from Debian experimental (git from 20101101). Both show the same symptoms. I don't think fragmentation is bad on this box (due to having an SSD), but my system at home is getting dog slow and I'd like to try that when I come home end of the week. You're not using compression on that filesystem are you? If so, be aware that the number of extents isn't going to change after defragmentation, although you should find that the _locations_ of those extents is contiguous. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Excerpts from John Wyzer's message of 2011-04-30 18:33:20 -0400: Excerpts from Mitch Harder's message of Sun May 01 00:16:53 +0200 2011: Hmm. Tried it and it gives me about 50 lines of FIBMAP: Invalid argument and then: large_file: 1 extent found Is that the way it is supposed to work? Just asking because this was part of a vmware disk image. Both the virtual machine and the rest of the host system are almost unusable once the VM ist started (even more unusable than without vmware :-D ) No. It sounds like the filefrag command is getting confused in the virtual environment. Misunderstanding :-) I tried filefrag _on_ a vmware disk image, not inside a virtual machine. The whole btrfs story here is on a real machine. The most important files to defrag are going to be your internal firefox files (I think in .mozilla), your sup database (in .sup) and your vmware images. I would definitely suggest that you try to narrow down if vmware is making the machine seem much slower after the defrag is done. It might make sense to run the nocow ioctl on your vmware images, they are probably triggering lots of seeks. You'll notice the machine is much slower after a reboot, this is because we have to do a lot of IO to recache the extent allocation tree. If you pull from the btrfs-unstable tree, you can use mount -o space_cache, which cuts down on the reading after a reboot dramatically. If none of this works, we'll look at the files that you're fsyncing. That seems to be the bulk of your latencies. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Excerpts from John Wyzer's message of 2011-04-30 18:33:20 -0400: Excerpts from Mitch Harder's message of Sun May 01 00:16:53 +0200 2011: Hmm. Tried it and it gives me about 50 lines of FIBMAP: Invalid argument and then: large_file: 1 extent found Is that the way it is supposed to work? Just asking because this was part of a vmware disk image. Both the virtual machine and the rest of the host system are almost unusable once the VM ist started (even more unusable than without vmware :-D ) No. It sounds like the filefrag command is getting confused in the virtual environment. Misunderstanding :-) I tried filefrag _on_ a vmware disk image, not inside a virtual machine. The whole btrfs story here is on a real machine. Older filefrag uses fibmap, which we don't support (we use fiemap instead). If you update your e2fsprogs you should get a newer filefrag. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Excerpts from Bernhard Schmidt's message of 2011-05-03 06:33:25 -0400: Peter Stuge pe...@stuge.se wrote: Hey, defragging btrfs does not seem to work for me. I have run the filefrag command over the whole fs and (manually) tried to defrag a few heavily fragmented files, but I don't get it to work (it still has the same number of extends and they are horrently uncorrelated) root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found root@schleppi:~# btrfs filesystem defrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found I'm using Ubuntu Natty (2.6.38.4) and tried both btrfs-tools from Natty (201006xx) and from Debian experimental (git from 20101101). Both show the same symptoms. I don't think fragmentation is bad on this box (due to having an SSD), but my system at home is getting dog slow and I'd like to try that when I come home end of the week. Do you have compression on? The file the defrag ioctl works is that it schedules things for defrag but doesn't force out the IO immediately unless you use -f. So, to test the result of the defrag, you need to either wait a bit or run sync. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Am 03.05.2011 13:08, schrieb Chris Mason: defragging btrfs does not seem to work for me. I have run the filefrag command over the whole fs and (manually) tried to defrag a few heavily fragmented files, but I don't get it to work (it still has the same number of extends and they are horrently uncorrelated) root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found root@schleppi:~# btrfs filesystem defrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found I'm using Ubuntu Natty (2.6.38.4) and tried both btrfs-tools from Natty (201006xx) and from Debian experimental (git from 20101101). Both show the same symptoms. I don't think fragmentation is bad on this box (due to having an SSD), but my system at home is getting dog slow and I'd like to try that when I come home end of the week. Do you have compression on? Yes. lzo to be exact. The file the defrag ioctl works is that it schedules things for defrag but doesn't force out the IO immediately unless you use -f. So, to test the result of the defrag, you need to either wait a bit or run sync. Did so, no change. See my reply to cwillu for the data. I usually mount my / without any compression option and did mount -o remount,compress=lzo / before. I cannot reboot at the moment and I did not find any option to disable compression again. There does not seem to be a nocompress or compress=[none|off] option. Is this correct? Bernhard -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Excerpts from Bernhard Schmidt's message of 2011-05-03 07:30:36 -0400: Am 03.05.2011 13:08, schrieb Chris Mason: defragging btrfs does not seem to work for me. I have run the filefrag command over the whole fs and (manually) tried to defrag a few heavily fragmented files, but I don't get it to work (it still has the same number of extends and they are horrently uncorrelated) root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found root@schleppi:~# btrfs filesystem defrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found I'm using Ubuntu Natty (2.6.38.4) and tried both btrfs-tools from Natty (201006xx) and from Debian experimental (git from 20101101). Both show the same symptoms. I don't think fragmentation is bad on this box (due to having an SSD), but my system at home is getting dog slow and I'd like to try that when I come home end of the week. Do you have compression on? Yes. lzo to be exact. The file the defrag ioctl works is that it schedules things for defrag but doesn't force out the IO immediately unless you use -f. So, to test the result of the defrag, you need to either wait a bit or run sync. Did so, no change. See my reply to cwillu for the data. I usually mount my / without any compression option and did mount -o remount,compress=lzo / before. I cannot reboot at the moment and I did not find any option to disable compression again. There does not seem to be a nocompress or compress=[none|off] option. Is this correct? Using compression is not a problem, but in order to reduce the maximum amount of ram we need to uncompress an extent, we enforce a max size on the extent. So you'll tend to have more extents, but they should be close together on disk. Could you please do a filefrag -v on the file? Lets see how bad it really is. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Am 03.05.2011 13:00, schrieb cwillu: Hi, defragging btrfs does not seem to work for me. I have run the filefrag command over the whole fs and (manually) tried to defrag a few heavily fragmented files, but I don't get it to work (it still has the same number of extends and they are horrently uncorrelated) root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found root@schleppi:~# btrfs filesystem defrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 root@schleppi:~# filefrag /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found I'm using Ubuntu Natty (2.6.38.4) and tried both btrfs-tools from Natty (201006xx) and from Debian experimental (git from 20101101). Both show the same symptoms. I don't think fragmentation is bad on this box (due to having an SSD), but my system at home is getting dog slow and I'd like to try that when I come home end of the week. You're not using compression on that filesystem are you? If so, be aware that the number of extents isn't going to change after defragmentation, although you should find that the _locations_ of those extents is contiguous. Actually I do run compression, but the location is not continguous afterwards (even after btrfs filesystem sync / or using -f): root@schleppi:~# btrfs filesystem defrag -f /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 root@schleppi:~# btrfs filesystem sync /FSSync '/' root@schleppi:~# filefrag -v /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 Filesystem type is: 9123683e File size of /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 is 9361528 (2286 blocks, blocksize 4096) ext logical physical expected length flags 0 0 4542111 32 1 32 4542134 4542142 32 2 64 4573263 4542165 32 3 96 4573285 4573294 32 4 128 4579639 4573316 32 5 160 4579664 4579670 32 6 192 4581178 4579695 32 7 224 4579811 4581209 32 8 256 4579836 4579842 32 9 288 4579861 4579867 32 10 320 4579884 4579892 32 11 352 4580698 4579915 32 12 384 4580720 4580729 32 13 416 4580746 4580751 32 14 448 4580768 4580777 32 15 480 4580793 4580799 32 16 512 4580819 4580824 32 17 544 4581238 4580850 32 18 576 4600396 4581269 32 19 608 4600422 4600427 32 20 640 4600447 4600453 32 21 672 4600472 4600478 32 22 704 4600498 4600503 32 23 736 4600523 4600529 32 24 768 4601483 4600554 32 25 800 4601509 4601514 32 26 832 4601534 4601540 32 27 864 4601558 4601565 32 28 896 4601583 4601589 32 29 928 4601608 4601614 32 30 960 4618420 4601639 32 31 992 4618443 4618451 32 321024 4541221 4618474 32 331056 4618463 4541252 32 341088 4618485 4618494 32 351120 4618505 4618516 32 361152 4579536 4618536 32 371184 4579688 4579567 32 381216 4579740 4579719 32 391248 4618526 4579771 32 401280 4618544 4618557 32 411312 4618563 4618575 32 421344 4618583 4618594 32 431376 4618605 4618614 32 441408 4618626 4618636 32 451440 4618652 4618657 32 461472 4618677 4618683 32 471504 4618703 4618708 32 481536 4618728 4618734 32 491568 4618754 4618759 32 501600 4618774 4618785 32 511632 4618782 4618805 32 521664 4561195 4618813 32 531696 4600548 4561226 32 541728 4618793 4600579 32 551760 4618807 4618824 32 561792 4539912 4618838 32 571824 4542619 4539943 32 581856 4556887 4542650 32 591888 4601632 4556918 32 601920 4558150 4601663 32 611952 4561224 4558181 32 621984 4618816 4561255 32 632016 4618835 4618847 32 642048 4618861 4618866 32 652080 4618881 4618892 32 662112 4618901 4618912 32 672144 4618917 4618932 32 682176 4539241 4618948 32 692208 4539915 4539272 32 702240 4539985 4539946 32 712272 4540096 4540016 14 eof /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found Best Regards, Bernhard -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Hi, Using compression is not a problem, but in order to reduce the maximum amount of ram we need to uncompress an extent, we enforce a max size on the extent. So you'll tend to have more extents, but they should be close together on disk. Could you please do a filefrag -v on the file? Lets see how bad it really is. root@schleppi:~# filefrag -v /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 Filesystem type is: 9123683e File size of /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 is 9361528 (2286 blocks, blocksize 4096) ext logical physical expected length flags 0 0 4542111 32 1 32 4542134 4542142 32 2 64 4573263 4542165 32 3 96 4573285 4573294 32 4 128 4579639 4573316 32 5 160 4579664 4579670 32 6 192 4581178 4579695 32 7 224 4579811 4581209 32 8 256 4579836 4579842 32 9 288 4579861 4579867 32 10 320 4579884 4579892 32 11 352 4580698 4579915 32 12 384 4580720 4580729 32 13 416 4580746 4580751 32 14 448 4580768 4580777 32 15 480 4580793 4580799 32 16 512 4580819 4580824 32 17 544 4581238 4580850 32 18 576 4600396 4581269 32 19 608 4600422 4600427 32 20 640 4600447 4600453 32 21 672 4600472 4600478 32 22 704 4600498 4600503 32 23 736 4600523 4600529 32 24 768 4601483 4600554 32 25 800 4601509 4601514 32 26 832 4601534 4601540 32 27 864 4601558 4601565 32 28 896 4601583 4601589 32 29 928 4601608 4601614 32 30 960 4618420 4601639 32 31 992 4618443 4618451 32 321024 4541221 4618474 32 331056 4618463 4541252 32 341088 4618485 4618494 32 351120 4618505 4618516 32 361152 4579536 4618536 32 371184 4579688 4579567 32 381216 4579740 4579719 32 391248 4618526 4579771 32 401280 4618544 4618557 32 411312 4618563 4618575 32 421344 4618583 4618594 32 431376 4618605 4618614 32 441408 4618626 4618636 32 451440 4618652 4618657 32 461472 4618677 4618683 32 471504 4618703 4618708 32 481536 4618728 4618734 32 491568 4618754 4618759 32 501600 4618774 4618785 32 511632 4618782 4618805 32 521664 4561195 4618813 32 531696 4600548 4561226 32 541728 4618793 4600579 32 551760 4618807 4618824 32 561792 4539912 4618838 32 571824 4542619 4539943 32 581856 4556887 4542650 32 591888 4601632 4556918 32 601920 4558150 4601663 32 611952 4561224 4558181 32 621984 4618816 4561255 32 632016 4618835 4618847 32 642048 4618861 4618866 32 652080 4618881 4618892 32 662112 4618901 4618912 32 672144 4618917 4618932 32 682176 4539241 4618948 32 692208 4539915 4539272 32 702240 4539985 4539946 32 712272 4540096 4540016 14 eof /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found Bernhard -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Excerpts from Bernhard Schmidt's message of 2011-05-03 07:43:04 -0400: Hi, Using compression is not a problem, but in order to reduce the maximum amount of ram we need to uncompress an extent, we enforce a max size on the extent. So you'll tend to have more extents, but they should be close together on disk. Could you please do a filefrag -v on the file? Lets see how bad it really is. root@schleppi:~# filefrag -v /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 Filesystem type is: 9123683e File size of /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 is 9361528 (2286 blocks, blocksize 4096) ext logical physical expected length flags 0 0 4542111 32 1 32 4542134 4542142 32 2 64 4573263 4542165 32 Ok, looks like we could be doing a little better job when compression is on to build out a bigger extent. This shouldn't be causing trouble on an ssd at all but on your rotating disk it'll be slightly slower. Still most of these extents are somewhat close together, this is roughly what mount -o ssd (which is enabled automatically when we detect a non-rotating drive) would try for. The problematic files are going to have thousands of extents, this file should be fine. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Hi, Ok, looks like we could be doing a little better job when compression is on to build out a bigger extent. This shouldn't be causing trouble on an ssd at all but on your rotating disk it'll be slightly slower. Still most of these extents are somewhat close together, this is roughly what mount -o ssd (which is enabled automatically when we detect a non-rotating drive) would try for. The problematic files are going to have thousands of extents, this file should be fine. Thanks, I'll check on my system with rotating disks at home when I get back. Bernhard -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
On Tue, May 3, 2011 at 8:03 AM, Bernhard Schmidt be...@birkenwald.de wrote: Hi, Ok, looks like we could be doing a little better job when compression is on to build out a bigger extent. This shouldn't be causing trouble on an ssd at all but on your rotating disk it'll be slightly slower. Still most of these extents are somewhat close together, this is roughly what mount -o ssd (which is enabled automatically when we detect a non-rotating drive) would try for. The problematic files are going to have thousands of extents, this file should be fine. Thanks, I'll check on my system with rotating disks at home when I get back. I'd also be curious to see if mounting with -o compress-force=lzo affects anything. As I recall, the compress-force option was added because performance could be affected negatively when trying to optimize compression with -o compress. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
On 3 May 2011 19:30, Bernhard Schmidt be...@birkenwald.de wrote: [] The file the defrag ioctl works is that it schedules things for defrag but doesn't force out the IO immediately unless you use -f. So, to test the result of the defrag, you need to either wait a bit or run sync. Did so, no change. See my reply to cwillu for the data. Can you try with the compression option enabled? Eg: # filefrag foo.dat foo.dat: 11 extents found # find . -xdev -type f -print0 | xargs -0 btrfs filesystem defragment -c # filefrag foo.dat foo.dat: 1 extent found Seems to work fine on 2.6.39-rc5; I mounted with '-o compress,clear_cache' though. Daniel -- Daniel J Blueman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Am 03.05.2011 16:54, schrieb Daniel J Blueman: Hi, The file the defrag ioctl works is that it schedules things for defrag but doesn't force out the IO immediately unless you use -f. So, to test the result of the defrag, you need to either wait a bit or run sync. Did so, no change. See my reply to cwillu for the data. Can you try with the compression option enabled? Eg: # filefrag foo.dat foo.dat: 11 extents found # find . -xdev -type f -print0 | xargs -0 btrfs filesystem defragment -c # filefrag foo.dat foo.dat: 1 extent found Seems to work fine on 2.6.39-rc5; I mounted with '-o compress,clear_cache' though. Maybe I was expecting too much. I tried it on a file with 72 extends and was expecting for the count to go down to 1 (or very very few). This does not seem to happen with this particular file. I just tested another file (with 193 extends) and it was reduced to 5. defrag with -f, but without -c. Still mounted with compress=lzo. However, the 72 frags file is not getting any better, no matter which flags I tried. No big problem at the moment, I've found an older (Ubuntu Maverick) based system with a rotating disk that had like 5 extends for a single file. I expect defragging that will increase performance quite nicely :-) Bernhard -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
On Tue, May 3, 2011 at 9:41 AM, Daniel J Blueman daniel.blue...@gmail.com wrote: It does seem the case generally; on 2.6.39-rc5, writing to a fresh filesystem using rsync with BTRFS compression enabled, 128KB extents seem very common [1] (filefrag inconsistency noted). Defragmenting with compression gives a nice linear extent [2]. It looks like it'll be a good win to prevent extents being split at writeout for the read case on rotational media. Yes, 128KB extents are hardcoded in Btrfs right now. There are two reasons cited in the comments for this: (1) Ease the RAM required when spreading compression across several CPUs. (2) Make sure the amount of IO required to do a random read is reasonably small. For about 4 months, I've been playing locally with 2 patches to increase the extent size to 512KB. I haven't noticed any issues running with these patches. However, I only have a Core2duo with 2 CPUs, so I'm probably not running into issues that someone with more CPUs might encounter. I'll submit these patches to the list as an RFC so more people can at least see where this is done. But with my limited hardware, I can't assert this change is the best for everyone. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Excerpts from Mitch Harder's message of 2011-05-03 11:42:56 -0400: On Tue, May 3, 2011 at 9:41 AM, Daniel J Blueman daniel.blue...@gmail.com wrote: It does seem the case generally; on 2.6.39-rc5, writing to a fresh filesystem using rsync with BTRFS compression enabled, 128KB extents seem very common [1] (filefrag inconsistency noted). Defragmenting with compression gives a nice linear extent [2]. It looks like it'll be a good win to prevent extents being split at writeout for the read case on rotational media. Yes, 128KB extents are hardcoded in Btrfs right now. There are two reasons cited in the comments for this: (1) Ease the RAM required when spreading compression across several CPUs. (2) Make sure the amount of IO required to do a random read is reasonably small. For about 4 months, I've been playing locally with 2 patches to increase the extent size to 512KB. I haven't noticed any issues running with these patches. However, I only have a Core2duo with 2 CPUs, so I'm probably not running into issues that someone with more CPUs might encounter. I'll submit these patches to the list as an RFC so more people can at least see where this is done. But with my limited hardware, I can't assert this change is the best for everyone. The problem is just that any random read into the file will require reading the full 512KB in order to decompress the extent. And you need to make sure you have enough room in ram to represent the decompressed bytes in order to find the pages you care about. The alternative is to keep the smaller compressed extent size and make the allocator work harder to find contiguous 128KB extents to store all of the file bytes. This will work out much better ;) -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
On Fri, Apr 29, 2011 at 10:01 AM, Chris Mason chris.ma...@oracle.com wrote:
Excerpts from John Wyzer's message of 2011-04-29 10:46:08 -0400:
Currently on
commit 7cf96da3ec7ca225acf4f284b0e904a1f5f98821
Author: Tsutomu Itoh t-i...@jp.fujitsu.com
Date: Mon Apr 25 19:43:53 2011 -0400
Btrfs: cleanup error handling in inode.c
merged into 2.6.38.4
I'm on a btrfs filesystem that has been used for some time. Let's say nine
months. Very recently I noticed performance getting worse and worse.
Most of the time it feels as if the system is just busy with iowait.
Write and read performance during random access is mostly around 2MB/s,
sometimes 1MB/s or slower. It's better for big files which can be read with
about
6-9MB/s. The disk is a reasonably recent SATA disk (WDC_WD3200BEVT) so 30MB/s
or 40MB/s linear reading should not be a problem.
rootfs 291G 242G 35G 88% /
I tried btrfs filesystem defragment -v / but did not notice any improvement
after that.
Is this a known phenomenon? :-)
Sounds like you're hitting fragmentation, which we can confirm with
latencytop. Please run latencytop while you're seeing poor performance
and take a look at where you're spending most of your time.
-chris
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Also, please note that 'btrfs filesystem defragment -v /' will
defragment the directory structure, but not the files.
See:
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#Defragmenting_a_directory_doesn.27t_work
To defragment your entire volume, you'll need a command like:
# for file in $(find PATH/TO/BTRFS/VOL/ -type f); do btrfs
filesystem defragment ${file}; done
There's also a similar command in the FAQ referenced above.
If you just want to see your fragmentation you can use the 'filefrag'
program from e2fsprogs:
# for file in $(find PATH/TO/BTRFS/VOL/ -type f); do filefrag
${file}; done | sort -n -k 2 | less
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Excerpts from Mitch Harder's message of Sat Apr 30 19:33:16 +0200 2011:
Also, please note that 'btrfs filesystem defragment -v /' will
defragment the directory structure, but not the files.
[...]
To defragment your entire volume, you'll need a command like:
# for file in $(find PATH/TO/BTRFS/VOL/ -type f); do btrfs
filesystem defragment ${file}; done
Thanks, I'm doing something like that at the moment (sorted the whole system
according to atimes and mtimes and started defragmenting in order of recent
access...)
However at this speed this will never end.
I'm willing to let it run some more nights however to see whether there will be
an effect in the end.
By the way: does it make a difference to run defrag on one file at a time or on
more?
At the moment I'm doing 100 files/directories per btrfs call...
If you just want to see your fragmentation you can use the 'filefrag'
program from e2fsprogs:
# for file in $(find PATH/TO/BTRFS/VOL/ -type f); do filefrag
${file}; done | sort -n -k 2 | less
Hmm.
Tried it and it gives me about 50 lines of
FIBMAP: Invalid argument
and then:
large_file: 1 extent found
Is that the way it is supposed to work?
Just asking because this was part of a vmware disk image. Both the virtual
machine and the rest of the host system are almost unusable once the VM ist
started (even more unusable than without vmware :-D )
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Resending this to the list since I did not notice that I just responded to Chris Mason for the last emails... Excerpts from John Wyzer's message of Sat Apr 30 08:57:06 +0200 2011: Excerpts from Chris Mason's message of Fri Apr 29 22:48:58 +0200 2011: http://bayimg.com/NahClAadn http://bayimg.com/NahcnaADn http://bayimg.com/NAhCoAAdN http://bayimg.com/PahCaaAdN Ok, you have three processes that may be causing trouble. We probably just need to defragment the files related to these three and life will be good again. 1) Firefox. firefox has a bunch of little databases that are going to fragment badly as we cow. 2) sup. Is this the sup email client? 3) vmware. Are you hosting vmware virtual images on btrfs too? @2: yes, the sup mail client, polling for messages. @3: yes, from some tasks I use vmware But those were just the active applications at the time of taking the screenshot. I have the same thing with opera or during snapshot deletion or practically anything that involves some disk access. I'll probably get all atimes for files on my system, sort and defragment the files in order of importance... We'll see how btrfs behaves afterwards... -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
On Sat, Apr 30, 2011 at 3:40 PM, John Wyzer john.wy...@gmx.de wrote:
If you just want to see your fragmentation you can use the 'filefrag'
program from e2fsprogs:
# for file in $(find PATH/TO/BTRFS/VOL/ -type f); do filefrag
${file}; done | sort -n -k 2 | less
Hmm.
Tried it and it gives me about 50 lines of
FIBMAP: Invalid argument
and then:
large_file: 1 extent found
Is that the way it is supposed to work?
Just asking because this was part of a vmware disk image. Both the virtual
machine and the rest of the host system are almost unusable once the VM ist
started (even more unusable than without vmware :-D )
No. It sounds like the filefrag command is getting confused in the
virtual environment.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Mitch Harder wrote:
To defragment your entire volume, you'll need a command like:
# for file in $(find PATH/TO/BTRFS/VOL/ -type f); do btrfs
filesystem defragment ${file}; done
Suggest:
find /path/to/btrfs/vol -type f -exec btrfs filesystem defragment '{}' ';'
If you just want to see your fragmentation you can use the 'filefrag'
program from e2fsprogs:
# for file in $(find PATH/TO/BTRFS/VOL/ -type f); do filefrag
${file}; done | sort -n -k 2 | less
find /path/to/btrfs/vol -type f -exec filefrag '{}' ';'
If either command can take multiple filenames as parameters, then e.g.:
find /path/to/btrfs/vol -type f -execdir filefrag '+'
(significantly better because not a fork per file)
//Peter
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
abysmal performance
Currently on commit 7cf96da3ec7ca225acf4f284b0e904a1f5f98821 Author: Tsutomu Itoh t-i...@jp.fujitsu.com Date: Mon Apr 25 19:43:53 2011 -0400 Btrfs: cleanup error handling in inode.c merged into 2.6.38.4 I'm on a btrfs filesystem that has been used for some time. Let's say nine months. Very recently I noticed performance getting worse and worse. Most of the time it feels as if the system is just busy with iowait. Write and read performance during random access is mostly around 2MB/s, sometimes 1MB/s or slower. It's better for big files which can be read with about 6-9MB/s. The disk is a reasonably recent SATA disk (WDC_WD3200BEVT) so 30MB/s or 40MB/s linear reading should not be a problem. rootfs291G 242G 35G 88% / I tried btrfs filesystem defragment -v / but did not notice any improvement after that. Is this a known phenomenon? :-) -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: abysmal performance
Excerpts from John Wyzer's message of 2011-04-29 10:46:08 -0400: Currently on commit 7cf96da3ec7ca225acf4f284b0e904a1f5f98821 Author: Tsutomu Itoh t-i...@jp.fujitsu.com Date: Mon Apr 25 19:43:53 2011 -0400 Btrfs: cleanup error handling in inode.c merged into 2.6.38.4 I'm on a btrfs filesystem that has been used for some time. Let's say nine months. Very recently I noticed performance getting worse and worse. Most of the time it feels as if the system is just busy with iowait. Write and read performance during random access is mostly around 2MB/s, sometimes 1MB/s or slower. It's better for big files which can be read with about 6-9MB/s. The disk is a reasonably recent SATA disk (WDC_WD3200BEVT) so 30MB/s or 40MB/s linear reading should not be a problem. rootfs291G 242G 35G 88% / I tried btrfs filesystem defragment -v / but did not notice any improvement after that. Is this a known phenomenon? :-) Sounds like you're hitting fragmentation, which we can confirm with latencytop. Please run latencytop while you're seeing poor performance and take a look at where you're spending most of your time. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
