Re: Bad fs performance, IO freezes

[cheater00 .]

No, sadly the intervals don't seem to be regular like that.

On Tue, Oct 27, 2015 at 7:39 AM, Duncan <1i5t5.dun...@cox.net> wrote:
> cheater00 . posted on Tue, 27 Oct 2015 03:00:05 +0100 as excerpted:
>
>> currently my computer freezes every several seconds for half a second or
>> so. Using it feels like I'm playing musical chairs with the kernel.
>> I have just one download happening on utorrent right now - this is what
>> the graph looks like:
>> http://i.imgur.com/LqhMtrJ.png and every time a new spike happens, a
>> freeze happens just before that... that's the only time those freezes
>> happen, too.
>
> Is it perchance every 30 seconds?  (The graph seems to indicate more like
> 50 second cycles, but...)
>
> Because that's btrfs' normal commit time.  If it is, there's a mount
> option to change the commit time (the wiki says commit=N, N=30 by
> default, since kernel 3.12).  You could try fiddling with that, say
> setting it to 15 or 60, not necessarily to fix the problem (tho 60
> seconds might increase the problem while making it less frequent, while
> 15 might make it more frequent but less of a problem), but to see if the
> cycle changes with the commit time option, or stays @ 30 seconds.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 06/21] btrfs: delayed_ref: Add new function to record reserved space into delayed ref

[Qu Wenruo]

Chris Mason wrote on 2015/10/27 02:12 -0400:

On Tue, Oct 27, 2015 at 01:48:34PM +0800, Qu Wenruo wrote:

Are you testing integration-4.4 from Chris repo?
Or 4.3-rc from mainline repo with my qgroup reserve patchset applied?

Although integration-4.4 already merged qgroup reserve patchset, but it's
causing some strange bug like over decrease data sinfo->bytes_may_use,
mainly in generic/127 testcase.

But if qgroup reserve patchset is rebased to integration-4.3 (I did all my
old tests based on that), no generic/127 problem at all.


Did I mismerge things?

-chris


Not sure yet.

But at least some patches in 4.3 is not in integration-4.4, like the
following patch:
btrfs: Avoid truncate tailing page if fallocate range doesn't exceed inode
size


Have you tried testing integration-4.4 merged with current Linus git?

-chris


Nice advice, compiling now.

But even rebasing my local old branch(based on 7d35199e15b82a4d1a200, 
with 21 patches) to 4.3-rc7, the result got screwed up...


If things get crazy, I'm afraid it would need to revert all my qgroup 
patchset from integration-4.4. :(


Thanks,
Qu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 06/21] btrfs: delayed_ref: Add new function to record reserved space into delayed ref

[Qu Wenruo]

Chris Mason wrote on 2015/10/27 02:12 -0400:

On Tue, Oct 27, 2015 at 01:48:34PM +0800, Qu Wenruo wrote:

Are you testing integration-4.4 from Chris repo?
Or 4.3-rc from mainline repo with my qgroup reserve patchset applied?

Although integration-4.4 already merged qgroup reserve patchset, but it's
causing some strange bug like over decrease data sinfo->bytes_may_use,
mainly in generic/127 testcase.

But if qgroup reserve patchset is rebased to integration-4.3 (I did all my
old tests based on that), no generic/127 problem at all.


Did I mismerge things?

-chris


Not sure yet.

But at least some patches in 4.3 is not in integration-4.4, like the
following patch:
btrfs: Avoid truncate tailing page if fallocate range doesn't exceed inode
size


Have you tried testing integration-4.4 merged with current Linus git?

-chris


Integration-4.4 merged with Linus' master also fails. :(

Current known working branches are all based on 4.3-integration(4.2-rc5):
https://github.com/adam900710/linux/tree/qgroup_reserve_good

Tried 4.3-rc5 and 4.3-rc7, all fails with kernel warning in generic/137.

And due to the huge difference, I'm afraid it won't take a short time to 
find the root cause...


Thanks,
Qu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Duncan]

cheater00 . posted on Tue, 27 Oct 2015 03:00:05 +0100 as excerpted:

> currently my computer freezes every several seconds for half a second or
> so. Using it feels like I'm playing musical chairs with the kernel.
> I have just one download happening on utorrent right now - this is what
> the graph looks like:
> http://i.imgur.com/LqhMtrJ.png and every time a new spike happens, a
> freeze happens just before that... that's the only time those freezes
> happen, too.

Is it perchance every 30 seconds?  (The graph seems to indicate more like 
50 second cycles, but...)

Because that's btrfs' normal commit time.  If it is, there's a mount 
option to change the commit time (the wiki says commit=N, N=30 by 
default, since kernel 3.12).  You could try fiddling with that, say 
setting it to 15 or 60, not necessarily to fix the problem (tho 60 
seconds might increase the problem while making it less frequent, while 
15 might make it more frequent but less of a problem), but to see if the 
cycle changes with the commit time option, or stays @ 30 seconds.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: random i/o error without error in dmesg

[Duncan]

Marc Joliet posted on Mon, 26 Oct 2015 15:23:39 +0100 as excerpted:

> Occasionally they go away by themselves, but usually I have to reboot to
> make them go away.  This happens when getmail attempts to fetch mail,
> which fails due to the above error.  After the reboot getmail succeeds
> again.

Just out of curiosity, does a remount,ro, followed by a remount,rw, solve 
the problem?

The ro/rw cycle should force anything in memory to device, so if that 
eliminates the problem, it could well be some sort of sync issue.  If it 
doesn't, then it's more likely an in-memory filesystem state issue, 
that's cleared by the reboot.

And if the ro/rw cycle doesn't clear the problem, what about a full 
unmount/mount cycle, at least of that subvolume?

If you're running multiple subvolumes with root being one of them, you 
can't of course unmount the entire filesystem, but you could go down to 
emergency mode (systemctl emergency), try unmounting everything but /, 
mounting / ro, and then switching back to normal mode (from emergency 
mode, simply exiting should return you to normal multi-user or gui 
target, remounting filesystems as necessary, etc).

IOW, does it take a full reboot to clear the problem, or is a simple ro/rw 
mount cycle enough, or an unmount/remount?

Finally, assuming root itself isn't btrfs, if you have btrfs configured 
as a module, you could try unmounting all btrfs and then unloading the 
module, then reloading and remounting.  That should entirely clear all in-
memory btrfs state, so if that doesn't solve the problem, while rebooting 
does, then the problem's very possibly outside of btrfs scope.  Of course 
if root is btrfs, you can't really check that.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: random i/o error without error in dmesg

[Marc Joliet]

On Tuesday 27 October 2015 06:23:06 Duncan wrote:
>Marc Joliet posted on Mon, 26 Oct 2015 15:23:39 +0100 as excerpted:
>> Occasionally they go away by themselves, but usually I have to reboot to
>> make them go away.  This happens when getmail attempts to fetch mail,
>> which fails due to the above error.  After the reboot getmail succeeds
>> again.
>
>Just out of curiosity, does a remount,ro, followed by a remount,rw, solve
>the problem?
>
>The ro/rw cycle should force anything in memory to device, so if that
>eliminates the problem, it could well be some sort of sync issue.  If it
>doesn't, then it's more likely an in-memory filesystem state issue,
>that's cleared by the reboot.
>
>And if the ro/rw cycle doesn't clear the problem, what about a full
>unmount/mount cycle, at least of that subvolume?
>
>If you're running multiple subvolumes with root being one of them, you
>can't of course unmount the entire filesystem, but you could go down to
>emergency mode (systemctl emergency), try unmounting everything but /,
>mounting / ro, and then switching back to normal mode (from emergency
>mode, simply exiting should return you to normal multi-user or gui
>target, remounting filesystems as necessary, etc).
>
>IOW, does it take a full reboot to clear the problem, or is a simple ro/rw
>mount cycle enough, or an unmount/remount?
>
>Finally, assuming root itself isn't btrfs, if you have btrfs configured
>as a module, you could try unmounting all btrfs and then unloading the
>module, then reloading and remounting.  That should entirely clear all in-
>memory btrfs state, so if that doesn't solve the problem, while rebooting
>does, then the problem's very possibly outside of btrfs scope.  Of course
>if root is btrfs, you can't really check that.

Thanks for the hints.  I just upgraded to gentoo-sources 4.1.11 and will see 
if that changes anything.  If not, I'll have to try unmounting /home from 
emergency mode (it's a subvolume mount).

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup


signature.asc
Description: This is a digitally signed message part.

Re: [PATCH v3 06/21] btrfs: delayed_ref: Add new function to record reserved space into delayed ref

[Filipe Manana]

On Tue, Oct 27, 2015 at 4:13 AM, Qu Wenruo  wrote:
>
>
> Filipe Manana wrote on 2015/10/25 14:39 +:
>>
>> On Tue, Oct 13, 2015 at 3:20 AM, Qu Wenruo 
>> wrote:
>>>
>>> Add new function btrfs_add_delayed_qgroup_reserve() function to record
>>> how much space is reserved for that extent.
>>>
>>> As btrfs only accounts qgroup at run_delayed_refs() time, so newly
>>> allocated extent should keep the reserved space until then.
>>>
>>> So add needed function with related members to do it.
>>>
>>> Signed-off-by: Qu Wenruo 
>>> ---
>>> v2:
>>>None
>>> v3:
>>>None
>>> ---
>>>   fs/btrfs/delayed-ref.c | 29 +
>>>   fs/btrfs/delayed-ref.h | 14 ++
>>>   2 files changed, 43 insertions(+)
>>>
>>> diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
>>> index ac3e81d..bd9b63b 100644
>>> --- a/fs/btrfs/delayed-ref.c
>>> +++ b/fs/btrfs/delayed-ref.c
>>> @@ -476,6 +476,8 @@ add_delayed_ref_head(struct btrfs_fs_info *fs_info,
>>>  INIT_LIST_HEAD(_ref->ref_list);
>>>  head_ref->processing = 0;
>>>  head_ref->total_ref_mod = count_mod;
>>> +   head_ref->qgroup_reserved = 0;
>>> +   head_ref->qgroup_ref_root = 0;
>>>
>>>  /* Record qgroup extent info if provided */
>>>  if (qrecord) {
>>> @@ -746,6 +748,33 @@ int btrfs_add_delayed_data_ref(struct btrfs_fs_info
>>> *fs_info,
>>>  return 0;
>>>   }
>>>
>>> +int btrfs_add_delayed_qgroup_reserve(struct btrfs_fs_info *fs_info,
>>> +struct btrfs_trans_handle *trans,
>>> +u64 ref_root, u64 bytenr, u64
>>> num_bytes)
>>> +{
>>> +   struct btrfs_delayed_ref_root *delayed_refs;
>>> +   struct btrfs_delayed_ref_head *ref_head;
>>> +   int ret = 0;
>>> +
>>> +   if (!fs_info->quota_enabled || !is_fstree(ref_root))
>>> +   return 0;
>>> +
>>> +   delayed_refs = >transaction->delayed_refs;
>>> +
>>> +   spin_lock(_refs->lock);
>>> +   ref_head = find_ref_head(_refs->href_root, bytenr, 0);
>>> +   if (!ref_head) {
>>> +   ret = -ENOENT;
>>> +   goto out;
>>> +   }
>>
>>
>> Hi Qu,
>>
>> So while running btrfs/063, with qgroups enabled (I modified the test
>> to enable qgroups), ran into this 2 times:
>>
>> [169125.246506] BTRFS info (device sdc): disk space caching is enabled
>> [169125.363164] [ cut here ]
>> [169125.365236] WARNING: CPU: 10 PID: 2827 at fs/btrfs/inode.c:2929
>> btrfs_finish_ordered_io+0x347/0x4eb [btrfs]()
>> [169125.367702] BTRFS: Transaction aborted (error -2)
>> [169125.368830] Modules linked in: btrfs dm_flakey dm_mod
>> crc32c_generic xor raid6_pq nfsd auth_rpcgss oid_registry nfs_acl nfs
>> lockd grace fscache sunrpc loop fuse parport_pc parport i2c_piix4
>> psmouse acpi_cpufreq microcode pcspkr processor evdev i2c_core
>> serio_raw button ext4 crc16 jbd2 mbcache sd_mod sg sr_mod cdrom
>> ata_generic virtio_scsi ata_piix libata floppy virtio_pci virtio_ring
>> scsi_mod e1000 virtio [last unloaded: btrfs]
>> [169125.376755] CPU: 10 PID: 2827 Comm: kworker/u32:14 Tainted: G
>>W   4.3.0-rc5-btrfs-next-17+ #1
>
>
> Hi Filipe,

Hi Qu,

>
> Although not related to the bug report, I'm a little interested in your
> testing kernel.
>
> Are you testing integration-4.4 from Chris repo?

Yes, I got that from Chris' integration-4.4 branch.

> Or 4.3-rc from mainline repo with my qgroup reserve patchset applied?
>
> Although integration-4.4 already merged qgroup reserve patchset, but it's
> causing some strange bug like over decrease data sinfo->bytes_may_use,
> mainly in generic/127 testcase.

Haven't hit that one yet.

>
> But if qgroup reserve patchset is rebased to integration-4.3 (I did all my
> old tests based on that), no generic/127 problem at all.
>
> Thanks,
> Qu
>
>
>> [169125.378522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>> BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org
>> 04/01/2014
>> [169125.380916] Workqueue: btrfs-endio-write btrfs_endio_write_helper
>> [btrfs]
>> [169125.382167]   88007ef2bc28 812566f4
>> 88007ef2bc70
>> [169125.383643]  88007ef2bc60 8104d0a6 a03cac33
>> 8801f5ca6db0
>> [169125.385197]  8802c6c7ee98 880122bc1000 fffe
>> 88007ef2bcc8
>> [169125.386691] Call Trace:
>> [169125.387194]  [] dump_stack+0x4e/0x79
>> [169125.388205]  [] warn_slowpath_common+0x9f/0xb8
>> [169125.389386]  [] ?
>> btrfs_finish_ordered_io+0x347/0x4eb [btrfs]
>> [169125.390837]  [] warn_slowpath_fmt+0x48/0x50
>> [169125.391839]  [] ? unpin_extent_cache+0xbe/0xcc
>> [btrfs]
>> [169125.392973]  []
>> btrfs_finish_ordered_io+0x347/0x4eb [btrfs]
>> [169125.395714]  [] ?
>> _raw_spin_unlock_irqrestore+0x38/0x60
>> [169125.396888]  [] ?
>> trace_hardirqs_off_caller+0x1f/0xb9
>> [169125.397986]  [] 

ENOSPC regression in integration-4.4 branch

[Filipe Manana]

Hi Josef,

Not sure if you are aware or got my report on IRC, but one of the
allocator fixes/improvements (Chris' integration-4.4 branch) is
causing new ENOSPC failures:

http://git.kernel.org/cgit/linux/kernel/git/mason/linux-btrfs.git/commit/?h=integration-4.4=a5e681d9bd641c4f0677e87d3a0c92a8f4f16293

(couldn't find the patch in the mailing list, so starting this thread)

A few xfstests (generic ones) are now failing, like generic/077 which
constantly fails for me with:

244045.147965] run fstests generic/077 at 2015-10-27 00:24:27
[244045.945396] BTRFS: device fsid
cf433807-c8dd-4f89-96d1-73dcce55ee38 devid 1 transid 3 /dev/sdc
[244046.106326] BTRFS info (device sdc): disk space caching is enabled
[244046.107825] BTRFS: has skinny extents
[244046.113251] BTRFS: creating UUID tree
[244055.206756] BTRFS info (device sdc): disk space caching is enabled
[244055.208472] BTRFS: has skinny extents
[244055.240473] BTRFS: error (device sdc) in
btrfs_free_dev_extent:1454: errno=-28 No space left (Slot search
failed)
[244055.243138] BTRFS info (device sdc): forced readonly
[244055.244528] [ cut here ]
[244055.245426] WARNING: CPU: 1 PID: 32239 at fs/btrfs/volumes.c:2771
btrfs_remove_chunk+0x2ff/0x795 [btrfs]()
[244055.247216] BTRFS: Transaction aborted (error -28)
[244055.248271] Modules linked in: btrfs dm_snapshot dm_bufio
dm_flakey dm_mod crc32c_generic xor raid6_pq nfsd auth_rpcgss
oid_registry nfs_acl nfs lockd grace fscache sunrpc loop fuse
parport_pc parport i2c_piix4 psmouse acpi_cpufreq microcode pcspkr
processor evdev i2c_core serio_raw button ext4 crc16 jbd2 mbcache
sd_mod sg sr_mod cdrom ata_generic virtio_scsi ata_piix libata floppy
virtio_pci virtio_ring scsi_mod e1000 virtio [last unloaded: btrfs]
[244055.258189] CPU: 1 PID: 32239 Comm: umount Tainted: GW
  4.3.0-rc5-btrfs-next-17+ #1
[244055.259874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org
04/01/2014
[244055.262228]   88011987fbd8 812566f4
88011987fc20
[244055.263688]  88011987fc10 8104d0a6 a03e4fc7
ffe4
[244055.265174]  880045a9dc00 ffe4 8800acd6ad28
88011987fc78
[244055.266541] Call Trace:
[244055.267023]  [] dump_stack+0x4e/0x79
[244055.267889]  [] warn_slowpath_common+0x9f/0xb8
[244055.268966]  [] ? btrfs_remove_chunk+0x2ff/0x795 [btrfs]
[244055.270326]  [] warn_slowpath_fmt+0x48/0x50
[244055.271405]  [] btrfs_remove_chunk+0x2ff/0x795 [btrfs]
[244055.272563]  []
btrfs_delete_unused_bgs+0x29a/0x365 [btrfs]
[244055.273857]  [] close_ctree+0xf3/0x33e [btrfs]
[244055.274837]  [] ? evict_inodes+0x13b/0x14a
[244055.275780]  [] btrfs_put_super+0x19/0x1b [btrfs]
[244055.276864]  [] generic_shutdown_super+0x6a/0xea
[244055.277983]  [] kill_anon_super+0x12/0x1c
[244055.278948]  [] btrfs_kill_super+0x16/0x21 [btrfs]
[244055.280042]  [] deactivate_locked_super+0x3b/0x68
[244055.281165]  [] deactivate_super+0x36/0x39
[244055.282111]  [] cleanup_mnt+0x58/0x76
[244055.282995]  [] __cleanup_mnt+0x12/0x14
[244055.283899]  [] task_work_run+0x6a/0x93
[244055.284848]  [] prepare_exit_to_usermode+0x91/0xac
[244055.287301]  [] syscall_return_slowpath+0x152/0x1ab
[244055.288506]  [] int_ret_from_sys_call+0x25/0x9f
[244055.289437] ---[ end trace 6ee4342a5722b12b ]---

Other tests like generic/103 also intermittently failed with ENOSPC here.
Without that patch, they all pass again.
There's plenty of unallocated space left (100G test devices) so these
tests shouldn't fail with ENOSPC.

Rings a bell?

thanks



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 01/13] Btrfs: __btrfs_buffered_write: Reserve/release extents aligned to block size

[Chandan Rajendra]

Currently, the code reserves/releases extents in multiples of PAGE_CACHE_SIZE
units. Fix this by doing reservation/releases in block size units.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/ctree.h |  3 +++
 fs/btrfs/file.c  | 43 ++-
 2 files changed, 33 insertions(+), 13 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index bc3c711..c9963f6 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2260,6 +2260,9 @@ struct btrfs_map_token {
unsigned long offset;
 };
 
+#define BTRFS_BYTES_TO_BLKS(fs_info, bytes) \
+   ((bytes) >> (fs_info)->sb->s_blocksize_bits)
+
 static inline void btrfs_init_map_token (struct btrfs_map_token *token)
 {
token->kaddr = NULL;
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 1243205..e31e120 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -499,7 +499,7 @@ int btrfs_dirty_pages(struct btrfs_root *root, struct inode 
*inode,
loff_t isize = i_size_read(inode);
 
start_pos = pos & ~((u64)root->sectorsize - 1);
-   num_bytes = ALIGN(write_bytes + pos - start_pos, root->sectorsize);
+   num_bytes = round_up(write_bytes + pos - start_pos, root->sectorsize);
 
end_of_last_block = start_pos + num_bytes - 1;
err = btrfs_set_extent_delalloc(inode, start_pos, end_of_last_block,
@@ -1362,16 +1362,19 @@ fail:
 static noinline int
 lock_and_cleanup_extent_if_need(struct inode *inode, struct page **pages,
size_t num_pages, loff_t pos,
+   size_t write_bytes,
u64 *lockstart, u64 *lockend,
struct extent_state **cached_state)
 {
+   struct btrfs_root *root = BTRFS_I(inode)->root;
u64 start_pos;
u64 last_pos;
int i;
int ret = 0;
 
-   start_pos = pos & ~((u64)PAGE_CACHE_SIZE - 1);
-   last_pos = start_pos + ((u64)num_pages << PAGE_CACHE_SHIFT) - 1;
+   start_pos = round_down(pos, root->sectorsize);
+   last_pos = start_pos
+   + round_up(pos + write_bytes - start_pos, root->sectorsize) - 1;
 
if (start_pos < inode->i_size) {
struct btrfs_ordered_extent *ordered;
@@ -1486,6 +1489,7 @@ static noinline ssize_t __btrfs_buffered_write(struct 
file *file,
 
while (iov_iter_count(i) > 0) {
size_t offset = pos & (PAGE_CACHE_SIZE - 1);
+   size_t sector_offset;
size_t write_bytes = min(iov_iter_count(i),
 nrptrs * (size_t)PAGE_CACHE_SIZE -
 offset);
@@ -1494,6 +1498,8 @@ static noinline ssize_t __btrfs_buffered_write(struct 
file *file,
size_t reserve_bytes;
size_t dirty_pages;
size_t copied;
+   size_t dirty_sectors;
+   size_t num_sectors;
 
WARN_ON(num_pages > nrptrs);
 
@@ -1506,7 +1512,9 @@ static noinline ssize_t __btrfs_buffered_write(struct 
file *file,
break;
}
 
-   reserve_bytes = num_pages << PAGE_CACHE_SHIFT;
+   sector_offset = pos & (root->sectorsize - 1);
+   reserve_bytes = round_up(write_bytes + sector_offset,
+   root->sectorsize);
 
if (BTRFS_I(inode)->flags & (BTRFS_INODE_NODATACOW |
 BTRFS_INODE_PREALLOC)) {
@@ -1525,7 +1533,9 @@ static noinline ssize_t __btrfs_buffered_write(struct 
file *file,
 */
num_pages = DIV_ROUND_UP(write_bytes + offset,
 PAGE_CACHE_SIZE);
-   reserve_bytes = num_pages << PAGE_CACHE_SHIFT;
+   reserve_bytes = round_up(write_bytes
+   + sector_offset,
+   root->sectorsize);
goto reserve_metadata;
}
}
@@ -1559,8 +1569,8 @@ again:
break;
 
ret = lock_and_cleanup_extent_if_need(inode, pages, num_pages,
- pos, , ,
- _state);
+   pos, write_bytes, ,
+   , _state);
if (ret < 0) {
if (ret == -EAGAIN)
goto again;
@@ -1596,9 +1606,16 @@ again:
 * we still have an outstanding extent for the chunk we actually
 * managed to copy.
 */
-   if (num_pages > dirty_pages) {
-   release_bytes 

[PATCH V7 00/13] Btrfs: Pre subpagesize-blocksize cleanups

[Chandan Rajendra]

The patches posted along with this cover letter are cleanups made
during the development of subpagesize-blocksize patchset. I believe
that they can be integrated with the mainline kernel. Hence I have
posted them separately from the subpagesize-blocksize patchset.

I have tested the patchset by running xfstests on ppc64 and
x86_64. On ppc64, some of the Btrfs specific tests and generic/255
fail because they assume 4K as the filesystem's block size. I have
fixed some of the test cases. I will fix the rest and mail them to the
fstests mailing list in the near future.

Changes from V6:
1. Rebased on linux-btrfs/integration-4.4 branch. As a result the
   following patches have been trivially modified.
`  - Btrfs: __btrfs_buffered_write: Reserve/release extents aligned
 to block size.
   - Btrfs: fallocate: Work with sectorsized blocks.
   - Btrfs: btrfs_page_mkwrite: Reserve space in sectorsized units.

Changes from V5:
1. Introduced BTRFS_BYTES_TO_BLKS() helper to compute the number of
   filesystem blocks spanning across a range of bytes. A call to this
   macro replaces code such as "nr_blks = bytes >> inode->i_blkbits".

Changes from V4:
1. Removed the RFC tag.

Changes from V3:
Two new issues have been been fixed by the patches,
1. Btrfs: prepare_pages: Retry adding a page to the page cache.
2. Btrfs: Return valid delalloc range when the page does not have
   PG_Dirty flag set or has been invalidated.
IMHO, The above issues are also applicable to the "page size == block
size" scenario but for reasons unknown to me they aren't seen even
when the tests are run for a long time.

Changes from V2:
1. For detecting logical errors, Use ASSERT() calls instead of calls to
   BUG_ON().
2. In the patch "Btrfs: Compute and look up csums based on sectorsized
   blocks", fix usage of kmap_atomic/kunmap_atomic such that between the
   kmap_atomic() and kunmap_atomic() calls we do not invoke any function
   that might cause the current task to sleep.
   
Changes from V1:
1. Call round_[down,up]() functions instead of doing hard coded alignment.

Chandan Rajendra (13):
  Btrfs: __btrfs_buffered_write: Reserve/release extents aligned to
block size
  Btrfs: Compute and look up csums based on sectorsized blocks
  Btrfs: Direct I/O read: Work on sectorsized blocks
  Btrfs: fallocate: Work with sectorsized blocks
  Btrfs: btrfs_page_mkwrite: Reserve space in sectorsized units
  Btrfs: Search for all ordered extents that could span across a page
  Btrfs: Use (eb->start, seq) as search key for tree modification log
  Btrfs: btrfs_submit_direct_hook: Handle map_length < bio vector length
  Btrfs: Limit inline extents to root->sectorsize
  Btrfs: Fix block size returned to user space
  Btrfs: Clean pte corresponding to page straddling i_size
  Btrfs: prepare_pages: Retry adding a page to the page cache
  Btrfs: Return valid delalloc range when the page does not have
PG_Dirty flag set or has been invalidated

 fs/btrfs/ctree.c |  34 +++
 fs/btrfs/ctree.h |   5 +-
 fs/btrfs/extent_io.c |   5 +-
 fs/btrfs/file-item.c |  92 ---
 fs/btrfs/file.c  | 115 
 fs/btrfs/inode.c | 248 ---
 6 files changed, 336 insertions(+), 163 deletions(-)

-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 11/13] Btrfs: Clean pte corresponding to page straddling i_size

[Chandan Rajendra]

When extending a file by either "truncate up" or by writing beyond i_size, the
page which had i_size needs to be marked "read only" so that future writes to
the page via mmap interface causes btrfs_page_mkwrite() to be invoked. If not,
a write performed after extending the file via the mmap interface will find
the page to be writaeable and continue writing to the page without invoking
btrfs_page_mkwrite() i.e. we end up writing to a file without reserving disk
space.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/file.c  | 12 ++--
 fs/btrfs/inode.c |  2 +-
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 81df4fa..5a7551c 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -1757,6 +1757,8 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
ssize_t err;
loff_t pos;
size_t count;
+   loff_t oldsize;
+   int clean_page = 0;
 
mutex_lock(>i_mutex);
err = generic_write_checks(iocb, from);
@@ -1795,14 +1797,17 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
pos = iocb->ki_pos;
count = iov_iter_count(from);
start_pos = round_down(pos, root->sectorsize);
-   if (start_pos > i_size_read(inode)) {
+   oldsize = i_size_read(inode);
+   if (start_pos > oldsize) {
/* Expand hole size to cover write data, preventing empty gap */
end_pos = round_up(pos + count, root->sectorsize);
-   err = btrfs_cont_expand(inode, i_size_read(inode), end_pos);
+   err = btrfs_cont_expand(inode, oldsize, end_pos);
if (err) {
mutex_unlock(>i_mutex);
goto out;
}
+   if (start_pos > round_up(oldsize, root->sectorsize))
+   clean_page = 1;
}
 
if (sync)
@@ -1814,6 +1819,9 @@ static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
num_written = __btrfs_buffered_write(file, from, pos);
if (num_written > 0)
iocb->ki_pos = pos + num_written;
+   if (clean_page)
+   pagecache_isize_extended(inode, oldsize,
+   i_size_read(inode));
}
 
mutex_unlock(>i_mutex);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 52880a4..9b1d7ba 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -4943,7 +4943,6 @@ static int btrfs_setsize(struct inode *inode, struct 
iattr *attr)
}
 
if (newsize > oldsize) {
-   truncate_pagecache(inode, newsize);
/*
 * Don't do an expanding truncate while snapshoting is ongoing.
 * This is to ensure the snapshot captures a fully consistent
@@ -4966,6 +4965,7 @@ static int btrfs_setsize(struct inode *inode, struct 
iattr *attr)
 
i_size_write(inode, newsize);
btrfs_ordered_update_i_size(inode, i_size_read(inode), NULL);
+   pagecache_isize_extended(inode, oldsize, newsize);
ret = btrfs_update_inode(trans, root, inode);
btrfs_end_write_no_snapshoting(root);
btrfs_end_transaction(trans, root);
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 13/13] Btrfs: Return valid delalloc range when the page does not have PG_Dirty flag set or has been invalidated

[Chandan Rajendra]

The following issue was observed when running generic/095 test on
subpagesize-blocksize patchset.

Assume that we are trying to write a dirty page that is mapping file offset
range [159744, 163839].

writepage_delalloc()
  find_lock_delalloc_range(*start = 159744, *end = 0)
find_delalloc_range()
  Returns range [X, Y] where (X > 163839)
lock_delalloc_pages()
  One of the pages in range [X, Y] has dirty flag cleared;
  Loop once more restricting the delalloc range to span only
  PAGE_CACHE_SIZE bytes;
find_delalloc_range()
  Returns range [356352, 360447];
lock_delalloc_pages()
  The page [356352, 360447] has dirty flag cleared;
Returns with *start = 159744 and *end = 0;
  *start = *end + 1;
  find_lock_delalloc_range(*start = 1, *end = 0)
Finds and returns delalloc range [1, 12288];
  cow_file_range()
Clears delalloc range [1, 12288]
Create ordered extent for range [1, 12288]

The ordered extent thus created above breaks the rule that extents have to be
aligned to the filesystem's block size.

In cases where lock_delalloc_pages() fails (either due to PG_dirty flag being
cleared or the page no longer being a member of the inode's page cache), this
patch sets and returns the delalloc range that was found by
find_delalloc_range().

Reviewed-by: Josef Bacik 
Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index d093643..9c5891a 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -1794,6 +1794,8 @@ again:
goto again;
} else {
found = 0;
+   *start = delalloc_start;
+   *end = delalloc_end;
goto out_failed;
}
}
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 12/13] Btrfs: prepare_pages: Retry adding a page to the page cache

[Chandan Rajendra]

When reading the page from the disk, we can race with Direct I/O which can get
the page lock (before prepare_uptodate_page() gets it) and can go ahead and
invalidate the page. Hence if the page is not found in the inode's address
space, retry the operation of getting a page.

Reported-by: Jakub Palider 
Reviewed-by: Josef Bacik 
Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/file.c | 16 
 1 file changed, 16 insertions(+)

diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 5a7551c..0e54a53 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -1316,6 +1316,7 @@ static noinline int prepare_pages(struct inode *inode, 
struct page **pages,
int faili;
 
for (i = 0; i < num_pages; i++) {
+again:
pages[i] = find_or_create_page(inode->i_mapping, index + i,
   mask | __GFP_WRITE);
if (!pages[i]) {
@@ -1330,6 +1331,21 @@ static noinline int prepare_pages(struct inode *inode, 
struct page **pages,
if (i == num_pages - 1)
err = prepare_uptodate_page(pages[i],
pos + write_bytes, false);
+
+   /*
+* When reading the page from the disk, we can race
+* with direct i/o which can get the page lock (before
+* prepare_uptodate_page() gets it) and can go ahead
+* and invalidate the page. Hence if the page is found
+* to be not belonging to the inode's address space,
+* retry the operation of getting a page.
+*/
+   if (unlikely(pages[i]->mapping != inode->i_mapping)) {
+   unlock_page(pages[i]);
+   page_cache_release(pages[i]);
+   goto again;
+   }
+
if (err) {
page_cache_release(pages[i]);
faili = i - 1;
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 05/13] Btrfs: btrfs_page_mkwrite: Reserve space in sectorsized units

[Chandan Rajendra]

In subpagesize-blocksize scenario, if i_size occurs in a block which is not
the last block in the page, then the space to be reserved should be calculated
appropriately.

Reviewed-by: Liu Bo 
Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/inode.c | 35 ++-
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index a54dad6..8c5b33a 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -8772,15 +8772,28 @@ int btrfs_page_mkwrite(struct vm_area_struct *vma, 
struct vm_fault *vmf)
loff_t size;
int ret;
int reserved = 0;
+   u64 reserved_space;
u64 page_start;
u64 page_end;
+   u64 end;
+
+   reserved_space = PAGE_CACHE_SIZE;
 
sb_start_pagefault(inode->i_sb);
page_start = page_offset(page);
page_end = page_start + PAGE_CACHE_SIZE - 1;
+   end = page_end;
 
+   /*
+* Reserving delalloc space after obtaining the page lock can lead to
+* deadlock. For example, if a dirty page is locked by this function
+* and the call to btrfs_delalloc_reserve_space() ends up triggering
+* dirty page write out, then the btrfs_writepage() function could
+* end up waiting indefinitely to get a lock on the page currently
+* being processed by btrfs_page_mkwrite() function.
+*/
ret = btrfs_delalloc_reserve_space(inode, page_start,
-  PAGE_CACHE_SIZE);
+  reserved_space);
if (!ret) {
ret = file_update_time(vma->vm_file);
reserved = 1;
@@ -8814,7 +8827,7 @@ again:
 * we can't set the delalloc bits if there are pending ordered
 * extents.  Drop our locks and wait for them to finish
 */
-   ordered = btrfs_lookup_ordered_extent(inode, page_start);
+   ordered = btrfs_lookup_ordered_range(inode, page_start, page_end);
if (ordered) {
unlock_extent_cached(io_tree, page_start, page_end,
 _state, GFP_NOFS);
@@ -8824,6 +8837,18 @@ again:
goto again;
}
 
+   if (page->index == ((size - 1) >> PAGE_CACHE_SHIFT)) {
+   reserved_space = round_up(size - page_start, root->sectorsize);
+   if (reserved_space < PAGE_CACHE_SIZE) {
+   end = page_start + reserved_space - 1;
+   spin_lock(_I(inode)->lock);
+   BTRFS_I(inode)->outstanding_extents++;
+   spin_unlock(_I(inode)->lock);
+   btrfs_delalloc_release_space(inode, size,
+   PAGE_CACHE_SIZE - 
reserved_space);
+   }
+   }
+
/*
 * XXX - page_mkwrite gets called every time the page is dirtied, even
 * if it was already dirty, so for space accounting reasons we need to
@@ -8831,12 +8856,12 @@ again:
 * is probably a better way to do this, but for now keep consistent with
 * prepare_pages in the normal write path.
 */
-   clear_extent_bit(_I(inode)->io_tree, page_start, page_end,
+   clear_extent_bit(_I(inode)->io_tree, page_start, end,
  EXTENT_DIRTY | EXTENT_DELALLOC |
  EXTENT_DO_ACCOUNTING | EXTENT_DEFRAG,
  0, 0, _state, GFP_NOFS);
 
-   ret = btrfs_set_extent_delalloc(inode, page_start, page_end,
+   ret = btrfs_set_extent_delalloc(inode, page_start, end,
_state);
if (ret) {
unlock_extent_cached(io_tree, page_start, page_end,
@@ -8875,7 +8900,7 @@ out_unlock:
}
unlock_page(page);
 out:
-   btrfs_delalloc_release_space(inode, page_start, PAGE_CACHE_SIZE);
+   btrfs_delalloc_release_space(inode, page_start, reserved_space);
 out_noreserve:
sb_end_pagefault(inode->i_sb);
return ret;
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 07/13] Btrfs: Use (eb->start, seq) as search key for tree modification log

[Chandan Rajendra]

In subpagesize-blocksize a page can map multiple extent buffers and hence
using (page index, seq) as the search key is incorrect. For example, searching
through tree modification log tree can return an entry associated with the
first extent buffer mapped by the page (if such an entry exists), when we are
actually searching for entries associated with extent buffers that are mapped
at position 2 or more in the page.

Reviewed-by: Liu Bo 
Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/ctree.c | 34 +-
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index 5b8e235..51ef032 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -311,7 +311,7 @@ struct tree_mod_root {
 
 struct tree_mod_elem {
struct rb_node node;
-   u64 index;  /* shifted logical */
+   u64 logical;
u64 seq;
enum mod_log_op op;
 
@@ -435,11 +435,11 @@ void btrfs_put_tree_mod_seq(struct btrfs_fs_info *fs_info,
 
 /*
  * key order of the log:
- *   index -> sequence
+ *   node/leaf start address -> sequence
  *
- * the index is the shifted logical of the *new* root node for root replace
- * operations, or the shifted logical of the affected block for all other
- * operations.
+ * The 'start address' is the logical address of the *new* root node
+ * for root replace operations, or the logical address of the affected
+ * block for all other operations.
  *
  * Note: must be called with write lock (tree_mod_log_write_lock).
  */
@@ -460,9 +460,9 @@ __tree_mod_log_insert(struct btrfs_fs_info *fs_info, struct 
tree_mod_elem *tm)
while (*new) {
cur = container_of(*new, struct tree_mod_elem, node);
parent = *new;
-   if (cur->index < tm->index)
+   if (cur->logical < tm->logical)
new = &((*new)->rb_left);
-   else if (cur->index > tm->index)
+   else if (cur->logical > tm->logical)
new = &((*new)->rb_right);
else if (cur->seq < tm->seq)
new = &((*new)->rb_left);
@@ -523,7 +523,7 @@ alloc_tree_mod_elem(struct extent_buffer *eb, int slot,
if (!tm)
return NULL;
 
-   tm->index = eb->start >> PAGE_CACHE_SHIFT;
+   tm->logical = eb->start;
if (op != MOD_LOG_KEY_ADD) {
btrfs_node_key(eb, >key, slot);
tm->blockptr = btrfs_node_blockptr(eb, slot);
@@ -588,7 +588,7 @@ tree_mod_log_insert_move(struct btrfs_fs_info *fs_info,
goto free_tms;
}
 
-   tm->index = eb->start >> PAGE_CACHE_SHIFT;
+   tm->logical = eb->start;
tm->slot = src_slot;
tm->move.dst_slot = dst_slot;
tm->move.nr_items = nr_items;
@@ -699,7 +699,7 @@ tree_mod_log_insert_root(struct btrfs_fs_info *fs_info,
goto free_tms;
}
 
-   tm->index = new_root->start >> PAGE_CACHE_SHIFT;
+   tm->logical = new_root->start;
tm->old_root.logical = old_root->start;
tm->old_root.level = btrfs_header_level(old_root);
tm->generation = btrfs_header_generation(old_root);
@@ -739,16 +739,15 @@ __tree_mod_log_search(struct btrfs_fs_info *fs_info, u64 
start, u64 min_seq,
struct rb_node *node;
struct tree_mod_elem *cur = NULL;
struct tree_mod_elem *found = NULL;
-   u64 index = start >> PAGE_CACHE_SHIFT;
 
tree_mod_log_read_lock(fs_info);
tm_root = _info->tree_mod_log;
node = tm_root->rb_node;
while (node) {
cur = container_of(node, struct tree_mod_elem, node);
-   if (cur->index < index) {
+   if (cur->logical < start) {
node = node->rb_left;
-   } else if (cur->index > index) {
+   } else if (cur->logical > start) {
node = node->rb_right;
} else if (cur->seq < min_seq) {
node = node->rb_left;
@@ -1230,9 +1229,10 @@ __tree_mod_log_oldest_root(struct btrfs_fs_info *fs_info,
return NULL;
 
/*
-* the very last operation that's logged for a root is the replacement
-* operation (if it is replaced at all). this has the index of the *new*
-* root, making it the very first operation that's logged for this root.
+* the very last operation that's logged for a root is the
+* replacement operation (if it is replaced at all). this has
+* the logical address of the *new* root, making it the very
+* first operation that's logged for this root.
 */
while (1) {
tm = tree_mod_log_search_oldest(fs_info, root_logical,
@@ -1336,7 +1336,7 @@ __tree_mod_log_rewind(struct btrfs_fs_info *fs_info, 
struct extent_buffer *eb,
if (!next)
break;
 

[PATCH V7 10/13] Btrfs: Fix block size returned to user space

[Chandan Rajendra]

btrfs_getattr() returns PAGE_CACHE_SIZE as the block size. Since
generic_fillattr() already does the right thing (by obtaining block size
from inode->i_blkbits), just remove the statement from btrfs_getattr.

Reviewed-by: Josef Bacik 
Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/inode.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 3a527af..52880a4 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -9312,7 +9312,6 @@ static int btrfs_getattr(struct vfsmount *mnt,
 
generic_fillattr(inode, stat);
stat->dev = BTRFS_I(inode)->root->anon_dev;
-   stat->blksize = PAGE_CACHE_SIZE;
 
spin_lock(_I(inode)->lock);
delalloc_bytes = BTRFS_I(inode)->delalloc_bytes;
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 04/13] Btrfs: fallocate: Work with sectorsized blocks

[Chandan Rajendra]

While at it, this commit changes btrfs_truncate_page() to truncate sectorsized
blocks instead of pages. Hence the function has been renamed to
btrfs_truncate_block().

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/ctree.h |  2 +-
 fs/btrfs/file.c  | 44 -
 fs/btrfs/inode.c | 60 ++--
 3 files changed, 55 insertions(+), 51 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index c9963f6..4469254 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -3923,7 +3923,7 @@ int btrfs_unlink_subvol(struct btrfs_trans_handle *trans,
struct btrfs_root *root,
struct inode *dir, u64 objectid,
const char *name, int name_len);
-int btrfs_truncate_page(struct inode *inode, loff_t from, loff_t len,
+int btrfs_truncate_block(struct inode *inode, loff_t from, loff_t len,
int front);
 int btrfs_truncate_inode_items(struct btrfs_trans_handle *trans,
   struct btrfs_root *root,
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index e31e120..81df4fa 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -2285,10 +2285,10 @@ static int btrfs_punch_hole(struct inode *inode, loff_t 
offset, loff_t len)
int ret = 0;
int err = 0;
unsigned int rsv_count;
-   bool same_page;
+   bool same_block;
bool no_holes = btrfs_fs_incompat(root->fs_info, NO_HOLES);
u64 ino_size;
-   bool truncated_page = false;
+   bool truncated_block = false;
bool updated_inode = false;
 
ret = btrfs_wait_ordered_range(inode, offset, len);
@@ -2296,7 +2296,7 @@ static int btrfs_punch_hole(struct inode *inode, loff_t 
offset, loff_t len)
return ret;
 
mutex_lock(>i_mutex);
-   ino_size = round_up(inode->i_size, PAGE_CACHE_SIZE);
+   ino_size = round_up(inode->i_size, root->sectorsize);
ret = find_first_non_hole(inode, , );
if (ret < 0)
goto out_only_mutex;
@@ -2309,31 +2309,30 @@ static int btrfs_punch_hole(struct inode *inode, loff_t 
offset, loff_t len)
lockstart = round_up(offset, BTRFS_I(inode)->root->sectorsize);
lockend = round_down(offset + len,
 BTRFS_I(inode)->root->sectorsize) - 1;
-   same_page = ((offset >> PAGE_CACHE_SHIFT) ==
-   ((offset + len - 1) >> PAGE_CACHE_SHIFT));
-
+   same_block = (BTRFS_BYTES_TO_BLKS(root->fs_info, offset))
+   == (BTRFS_BYTES_TO_BLKS(root->fs_info, offset + len - 1));
/*
-* We needn't truncate any page which is beyond the end of the file
+* We needn't truncate any block which is beyond the end of the file
 * because we are sure there is no data there.
 */
/*
-* Only do this if we are in the same page and we aren't doing the
-* entire page.
+* Only do this if we are in the same block and we aren't doing the
+* entire block.
 */
-   if (same_page && len < PAGE_CACHE_SIZE) {
+   if (same_block && len < root->sectorsize) {
if (offset < ino_size) {
-   truncated_page = true;
-   ret = btrfs_truncate_page(inode, offset, len, 0);
+   truncated_block = true;
+   ret = btrfs_truncate_block(inode, offset, len, 0);
} else {
ret = 0;
}
goto out_only_mutex;
}
 
-   /* zero back part of the first page */
+   /* zero back part of the first block */
if (offset < ino_size) {
-   truncated_page = true;
-   ret = btrfs_truncate_page(inode, offset, 0, 0);
+   truncated_block = true;
+   ret = btrfs_truncate_block(inode, offset, 0, 0);
if (ret) {
mutex_unlock(>i_mutex);
return ret;
@@ -2368,9 +2367,10 @@ static int btrfs_punch_hole(struct inode *inode, loff_t 
offset, loff_t len)
if (!ret) {
/* zero the front end of the last page */
if (tail_start + tail_len < ino_size) {
-   truncated_page = true;
-   ret = btrfs_truncate_page(inode,
-   tail_start + tail_len, 0, 1);
+   truncated_block = true;
+   ret = btrfs_truncate_block(inode,
+   tail_start + tail_len,
+   0, 1);
if (ret)
goto out_only_mutex;
}
@@ -2537,7 +2537,7 @@ out:

[PATCH V7 06/13] Btrfs: Search for all ordered extents that could span across a page

[Chandan Rajendra]

In subpagesize-blocksize scenario it is not sufficient to search using the
first byte of the page to make sure that there are no ordered extents
present across the page. Fix this.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c |  3 ++-
 fs/btrfs/inode.c | 25 ++---
 2 files changed, 20 insertions(+), 8 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 33a01ea..d093643 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -3281,7 +3281,8 @@ static int __extent_read_full_page(struct extent_io_tree 
*tree,
 
while (1) {
lock_extent(tree, start, end);
-   ordered = btrfs_lookup_ordered_extent(inode, start);
+   ordered = btrfs_lookup_ordered_range(inode, start,
+   PAGE_CACHE_SIZE);
if (!ordered)
break;
unlock_extent(tree, start, end);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 8c5b33a..c883ae8 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -1990,7 +1990,8 @@ again:
if (PagePrivate2(page))
goto out;
 
-   ordered = btrfs_lookup_ordered_extent(inode, page_start);
+   ordered = btrfs_lookup_ordered_range(inode, page_start,
+   PAGE_CACHE_SIZE);
if (ordered) {
unlock_extent_cached(_I(inode)->io_tree, page_start,
 page_end, _state, GFP_NOFS);
@@ -8653,6 +8654,8 @@ static void btrfs_invalidatepage(struct page *page, 
unsigned int offset,
struct extent_state *cached_state = NULL;
u64 page_start = page_offset(page);
u64 page_end = page_start + PAGE_CACHE_SIZE - 1;
+   u64 start;
+   u64 end;
int inode_evicting = inode->i_state & I_FREEING;
 
/*
@@ -8672,14 +8675,18 @@ static void btrfs_invalidatepage(struct page *page, 
unsigned int offset,
 
if (!inode_evicting)
lock_extent_bits(tree, page_start, page_end, 0, _state);
-   ordered = btrfs_lookup_ordered_extent(inode, page_start);
+again:
+   start = page_start;
+   ordered = btrfs_lookup_ordered_range(inode, start,
+   page_end - start + 1);
if (ordered) {
+   end = min(page_end, ordered->file_offset + ordered->len - 1);
/*
 * IO on this page will never be started, so we need
 * to account for any ordered extents now
 */
if (!inode_evicting)
-   clear_extent_bit(tree, page_start, page_end,
+   clear_extent_bit(tree, start, end,
 EXTENT_DIRTY | EXTENT_DELALLOC |
 EXTENT_LOCKED | EXTENT_DO_ACCOUNTING |
 EXTENT_DEFRAG, 1, 0, _state,
@@ -8696,22 +8703,26 @@ static void btrfs_invalidatepage(struct page *page, 
unsigned int offset,
 
spin_lock_irq(>lock);
set_bit(BTRFS_ORDERED_TRUNCATED, >flags);
-   new_len = page_start - ordered->file_offset;
+   new_len = start - ordered->file_offset;
if (new_len < ordered->truncated_len)
ordered->truncated_len = new_len;
spin_unlock_irq(>lock);
 
if (btrfs_dec_test_ordered_pending(inode, ,
-  page_start,
-  PAGE_CACHE_SIZE, 1))
+  start,
+  end - start + 1, 1))
btrfs_finish_ordered_io(ordered);
}
btrfs_put_ordered_extent(ordered);
if (!inode_evicting) {
cached_state = NULL;
-   lock_extent_bits(tree, page_start, page_end, 0,
+   lock_extent_bits(tree, start, end, 0,
 _state);
}
+
+   start = end + 1;
+   if (start < page_end)
+   goto again;
}
 
/*
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 02/13] Btrfs: Compute and look up csums based on sectorsized blocks

[Chandan Rajendra]

Checksums are applicable to sectorsize units. The current code uses
bio->bv_len units to compute and look up checksums. This works on machines
where sectorsize == PAGE_SIZE. This patch makes the checksum computation and
look up code to work with sectorsize units.

Reviewed-by: Liu Bo 
Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/file-item.c | 92 +---
 1 file changed, 59 insertions(+), 33 deletions(-)

diff --git a/fs/btrfs/file-item.c b/fs/btrfs/file-item.c
index 58ece65..e2a1cad 100644
--- a/fs/btrfs/file-item.c
+++ b/fs/btrfs/file-item.c
@@ -172,6 +172,7 @@ static int __btrfs_lookup_bio_sums(struct btrfs_root *root,
u64 item_start_offset = 0;
u64 item_last_offset = 0;
u64 disk_bytenr;
+   u64 page_bytes_left;
u32 diff;
int nblocks;
int bio_index = 0;
@@ -220,6 +221,8 @@ static int __btrfs_lookup_bio_sums(struct btrfs_root *root,
disk_bytenr = (u64)bio->bi_iter.bi_sector << 9;
if (dio)
offset = logical_offset;
+
+   page_bytes_left = bvec->bv_len;
while (bio_index < bio->bi_vcnt) {
if (!dio)
offset = page_offset(bvec->bv_page) + bvec->bv_offset;
@@ -243,7 +246,7 @@ static int __btrfs_lookup_bio_sums(struct btrfs_root *root,
if (BTRFS_I(inode)->root->root_key.objectid ==
BTRFS_DATA_RELOC_TREE_OBJECTID) {
set_extent_bits(io_tree, offset,
-   offset + bvec->bv_len - 1,
+   offset + root->sectorsize - 1,
EXTENT_NODATASUM, GFP_NOFS);
} else {

btrfs_info(BTRFS_I(inode)->root->fs_info,
@@ -281,11 +284,17 @@ static int __btrfs_lookup_bio_sums(struct btrfs_root 
*root,
 found:
csum += count * csum_size;
nblocks -= count;
-   bio_index += count;
+
while (count--) {
-   disk_bytenr += bvec->bv_len;
-   offset += bvec->bv_len;
-   bvec++;
+   disk_bytenr += root->sectorsize;
+   offset += root->sectorsize;
+   page_bytes_left -= root->sectorsize;
+   if (!page_bytes_left) {
+   bio_index++;
+   bvec++;
+   page_bytes_left = bvec->bv_len;
+   }
+
}
}
btrfs_free_path(path);
@@ -432,6 +441,8 @@ int btrfs_csum_one_bio(struct btrfs_root *root, struct 
inode *inode,
struct bio_vec *bvec = bio->bi_io_vec;
int bio_index = 0;
int index;
+   int nr_sectors;
+   int i;
unsigned long total_bytes = 0;
unsigned long this_sum_bytes = 0;
u64 offset;
@@ -459,41 +470,56 @@ int btrfs_csum_one_bio(struct btrfs_root *root, struct 
inode *inode,
if (!contig)
offset = page_offset(bvec->bv_page) + bvec->bv_offset;
 
-   if (offset >= ordered->file_offset + ordered->len ||
-   offset < ordered->file_offset) {
-   unsigned long bytes_left;
-   sums->len = this_sum_bytes;
-   this_sum_bytes = 0;
-   btrfs_add_ordered_sum(inode, ordered, sums);
-   btrfs_put_ordered_extent(ordered);
+   data = kmap_atomic(bvec->bv_page);
 
-   bytes_left = bio->bi_iter.bi_size - total_bytes;
+   nr_sectors = BTRFS_BYTES_TO_BLKS(root->fs_info,
+   bvec->bv_len + root->sectorsize
+   - 1);
+
+   for (i = 0; i < nr_sectors; i++) {
+   if (offset >= ordered->file_offset + ordered->len ||
+   offset < ordered->file_offset) {
+   unsigned long bytes_left;
+
+   kunmap_atomic(data);
+   sums->len = this_sum_bytes;
+   this_sum_bytes = 0;
+   btrfs_add_ordered_sum(inode, ordered, sums);
+   btrfs_put_ordered_extent(ordered);
+
+   bytes_left = bio->bi_iter.bi_size - total_bytes;
+
+   sums = kzalloc(btrfs_ordered_sum_size(root, 
bytes_left),
+   GFP_NOFS);
+   BUG_ON(!sums); /* -ENOMEM */
+   sums->len = bytes_left;
+   ordered = 

[PATCH V7 03/13] Btrfs: Direct I/O read: Work on sectorsized blocks

[Chandan Rajendra]

The direct I/O read's endio and corresponding repair functions work on
page sized blocks. This commit adds the ability for direct I/O read to work on
subpagesized blocks.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/inode.c | 98 +++-
 1 file changed, 75 insertions(+), 23 deletions(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index a018e47..98d901e 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -7764,9 +7764,9 @@ static int btrfs_check_dio_repairable(struct inode *inode,
 }
 
 static int dio_read_error(struct inode *inode, struct bio *failed_bio,
- struct page *page, u64 start, u64 end,
- int failed_mirror, bio_end_io_t *repair_endio,
- void *repair_arg)
+   struct page *page, unsigned int pgoff,
+   u64 start, u64 end, int failed_mirror,
+   bio_end_io_t *repair_endio, void *repair_arg)
 {
struct io_failure_record *failrec;
struct bio *bio;
@@ -7787,7 +7787,9 @@ static int dio_read_error(struct inode *inode, struct bio 
*failed_bio,
return -EIO;
}
 
-   if (failed_bio->bi_vcnt > 1)
+   if ((failed_bio->bi_vcnt > 1)
+   || (failed_bio->bi_io_vec->bv_len
+   > BTRFS_I(inode)->root->sectorsize))
read_mode = READ_SYNC | REQ_FAILFAST_DEV;
else
read_mode = READ_SYNC;
@@ -7795,7 +7797,7 @@ static int dio_read_error(struct inode *inode, struct bio 
*failed_bio,
isector = start - btrfs_io_bio(failed_bio)->logical;
isector >>= inode->i_sb->s_blocksize_bits;
bio = btrfs_create_repair_bio(inode, failed_bio, failrec, page,
- 0, isector, repair_endio, repair_arg);
+   pgoff, isector, repair_endio, repair_arg);
if (!bio) {
free_io_failure(inode, failrec);
return -EIO;
@@ -7825,12 +7827,17 @@ struct btrfs_retry_complete {
 static void btrfs_retry_endio_nocsum(struct bio *bio)
 {
struct btrfs_retry_complete *done = bio->bi_private;
+   struct inode *inode;
struct bio_vec *bvec;
int i;
 
if (bio->bi_error)
goto end;
 
+   ASSERT(bio->bi_vcnt == 1);
+   inode = bio->bi_io_vec->bv_page->mapping->host;
+   ASSERT(bio->bi_io_vec->bv_len == BTRFS_I(inode)->root->sectorsize);
+
done->uptodate = 1;
bio_for_each_segment_all(bvec, bio, i)
clean_io_failure(done->inode, done->start, bvec->bv_page, 0);
@@ -7842,25 +7849,35 @@ end:
 static int __btrfs_correct_data_nocsum(struct inode *inode,
   struct btrfs_io_bio *io_bio)
 {
+   struct btrfs_fs_info *fs_info;
struct bio_vec *bvec;
struct btrfs_retry_complete done;
u64 start;
+   unsigned int pgoff;
+   u32 sectorsize;
+   int nr_sectors;
int i;
int ret;
 
+   fs_info = BTRFS_I(inode)->root->fs_info;
+   sectorsize = BTRFS_I(inode)->root->sectorsize;
+
start = io_bio->logical;
done.inode = inode;
 
bio_for_each_segment_all(bvec, _bio->bio, i) {
-try_again:
+   nr_sectors = BTRFS_BYTES_TO_BLKS(fs_info, bvec->bv_len);
+   pgoff = bvec->bv_offset;
+
+next_block_or_try_again:
done.uptodate = 0;
done.start = start;
init_completion();
 
-   ret = dio_read_error(inode, _bio->bio, bvec->bv_page, start,
-start + bvec->bv_len - 1,
-io_bio->mirror_num,
-btrfs_retry_endio_nocsum, );
+   ret = dio_read_error(inode, _bio->bio, bvec->bv_page,
+   pgoff, start, start + sectorsize - 1,
+   io_bio->mirror_num,
+   btrfs_retry_endio_nocsum, );
if (ret)
return ret;
 
@@ -7868,10 +7885,15 @@ try_again:
 
if (!done.uptodate) {
/* We might have another mirror, so try again */
-   goto try_again;
+   goto next_block_or_try_again;
}
 
-   start += bvec->bv_len;
+   start += sectorsize;
+
+   if (nr_sectors--) {
+   pgoff += sectorsize;
+   goto next_block_or_try_again;
+   }
}
 
return 0;
@@ -7881,7 +7903,9 @@ static void btrfs_retry_endio(struct bio *bio)
 {
struct btrfs_retry_complete *done = bio->bi_private;
struct btrfs_io_bio *io_bio = btrfs_io_bio(bio);
+   struct inode *inode;
struct bio_vec *bvec;
+   u64 start;
int uptodate;
int ret;
int i;

[PATCH V7 08/13] Btrfs: btrfs_submit_direct_hook: Handle map_length < bio vector length

[Chandan Rajendra]

In subpagesize-blocksize scenario, map_length can be less than the length of a
bio vector. Such a condition may cause btrfs_submit_direct_hook() to submit a
zero length bio. Fix this by comparing map_length against block size rather
than with bv_len.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/inode.c | 25 +
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index c883ae8..6d66558 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -8248,9 +8248,11 @@ static int btrfs_submit_direct_hook(int rw, struct 
btrfs_dio_private *dip,
u64 file_offset = dip->logical_offset;
u64 submit_len = 0;
u64 map_length;
-   int nr_pages = 0;
-   int ret;
+   u32 blocksize = root->sectorsize;
int async_submit = 0;
+   int nr_sectors;
+   int ret;
+   int i;
 
map_length = orig_bio->bi_iter.bi_size;
ret = btrfs_map_block(root->fs_info, rw, start_sector << 9,
@@ -8280,9 +8282,12 @@ static int btrfs_submit_direct_hook(int rw, struct 
btrfs_dio_private *dip,
atomic_inc(>pending_bios);
 
while (bvec <= (orig_bio->bi_io_vec + orig_bio->bi_vcnt - 1)) {
-   if (map_length < submit_len + bvec->bv_len ||
-   bio_add_page(bio, bvec->bv_page, bvec->bv_len,
-bvec->bv_offset) < bvec->bv_len) {
+   nr_sectors = BTRFS_BYTES_TO_BLKS(root->fs_info, bvec->bv_len);
+   i = 0;
+next_block:
+   if (unlikely(map_length < submit_len + blocksize ||
+   bio_add_page(bio, bvec->bv_page, blocksize,
+   bvec->bv_offset + (i * blocksize)) < blocksize)) {
/*
 * inc the count before we submit the bio so
 * we know the end IO handler won't happen before
@@ -8303,7 +8308,6 @@ static int btrfs_submit_direct_hook(int rw, struct 
btrfs_dio_private *dip,
file_offset += submit_len;
 
submit_len = 0;
-   nr_pages = 0;
 
bio = btrfs_dio_bio_alloc(orig_bio->bi_bdev,
  start_sector, GFP_NOFS);
@@ -8321,9 +8325,14 @@ static int btrfs_submit_direct_hook(int rw, struct 
btrfs_dio_private *dip,
bio_put(bio);
goto out_err;
}
+
+   goto next_block;
} else {
-   submit_len += bvec->bv_len;
-   nr_pages++;
+   submit_len += blocksize;
+   if (--nr_sectors) {
+   i++;
+   goto next_block;
+   }
bvec++;
}
}
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V7 09/13] Btrfs: Limit inline extents to root->sectorsize

[Chandan Rajendra]

cow_file_range_inline() limits the size of an inline extent to
PAGE_CACHE_SIZE. This breaks in subpagesize-blocksize scenarios. Fix this by
comparing against root->sectorsize.

Reviewed-by: Josef Bacik 
Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/inode.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 6d66558..3a527af 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -257,7 +257,7 @@ static noinline int cow_file_range_inline(struct btrfs_root 
*root,
data_len = compressed_size;
 
if (start > 0 ||
-   actual_end > PAGE_CACHE_SIZE ||
+   actual_end > root->sectorsize ||
data_len > BTRFS_MAX_INLINE_DATA_SIZE(root) ||
(!compressed_size &&
(actual_end & (root->sectorsize - 1)) == 0) ||
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: pass proper enum type to start_transaction()

[David Sterba]

On Sun, Oct 25, 2015 at 07:35:44PM +, Alexandru Moise wrote:
> Signed-off-by: Alexandru Moise <00moses.alexande...@gmail.com>

Reviewed-by: David Sterba 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 06/21] btrfs: delayed_ref: Add new function to record reserved space into delayed ref

[Chris Mason]

On Tue, Oct 27, 2015 at 01:48:34PM +0800, Qu Wenruo wrote:
> >>Are you testing integration-4.4 from Chris repo?
> >>Or 4.3-rc from mainline repo with my qgroup reserve patchset applied?
> >>
> >>Although integration-4.4 already merged qgroup reserve patchset, but it's
> >>causing some strange bug like over decrease data sinfo->bytes_may_use,
> >>mainly in generic/127 testcase.
> >>
> >>But if qgroup reserve patchset is rebased to integration-4.3 (I did all my
> >>old tests based on that), no generic/127 problem at all.
> >
> >Did I mismerge things?
> >
> >-chris
> >
> Not sure yet.
> 
> But at least some patches in 4.3 is not in integration-4.4, like the
> following patch:
> btrfs: Avoid truncate tailing page if fallocate range doesn't exceed inode
> size

Have you tried testing integration-4.4 merged with current Linus git?

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Henk Slager]

I don't have a lot experience with autodefrag, but as indicated by
Austin, expect a lot of full rewrites of files that are relatively
slowly filled up by a torrent client, starting with a sparse file. So
1st advice would be to remove this option and run it as crontask at
particular times.

What SATA-USB bridge is between the harddisk and the PC motherboard ?
Also what USB host chipset is on the PC motherboard ?
Why don't you run 64-bit Ubuntu on this core i7 ?


On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
 wrote:
> On 2015-10-26 22:00, cheater00 . wrote:
>>
>> Hello,
>> currently my computer freezes every several seconds for half a second
>> or so. Using it feels like I'm playing musical chairs with the kernel.
>> I have just one download happening on utorrent right now - this is
>> what the graph looks like:
>> http://i.imgur.com/LqhMtrJ.png
>> and every time a new spike happens, a freeze happens just before
>> that... that's the only time those freezes happen, too.
>>
> Do you have the 'autodefrag' mount option enabled?  If it is turned on, then
> that may be the problem.  Most bittorrent clients pre-allocate the space for
> a download, then write each block directly into the location it's supposed
> to be in the resultant download, which means depending on how it's
> pre-allocating the space, that you end up with a large number of randomly
> ordered writes into a single file, which in turn will trigger the autodefrag
> code, which can cause latency spikes when you're also hitting the disk at
> the same time.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Austin S Hemmelgarn]

On 2015-10-27 09:00, Henk Slager wrote:

I don't have a lot experience with autodefrag, but as indicated by
Austin, expect a lot of full rewrites of files that are relatively
slowly filled up by a torrent client, starting with a sparse file. So
1st advice would be to remove this option and run it as crontask at
particular times.

What SATA-USB bridge is between the harddisk and the PC motherboard ?
I hadn't thought of this, but the specific adapter being used for the 
disk can have a lot of impact on how it preforms.  I've personally had 
lots of issues with JMicron chipsets (ranging from latency issues like 
what you are seeing to sever data corruption), but have found that 
ASMedia ones tend to be pretty much rock solid reliable and have good 
performance (although I think they only do USB 3.0 adapters).

Also what USB host chipset is on the PC motherboard ?
If it's a Intel motherboard, the USB 2.0 ports are probably routed 
through on-board hubs to the ports provided by whatever Intel calls 
their equivalent of the south bridge these days, and the USB 3.0 ports 
are probably a mix of Intel and ASMedia XHCI controllers (ASMedia was 
one of the first companies to do an inexpensive standalone XHCI chip, so 
they're relatively popular for extra USB 3.0 ports).  FWIW, the first 
generation of Intel XHCI chips had some issues with older Linux kernels, 
although IIRC those issues were along the lines of a port just 
disappearing after disconnecting whatever was connected to it.

Why don't you run 64-bit Ubuntu on this core i7 ?
64 versus 32 bit shouldn't cause anything like this to happen (although, 
if it can be proven that it does, then that is a serious bug that needs 
to be fixed).  That said, unless you have some desperate need to be 
running 32-bit only, you should seriously look into updating to a 64-bit 
version, your whole system should run faster, and Ubuntu has really good 
32-bit compatibility in the 64-bit version (which is part of why it's 
popular as a support target for third party software like Steam).



On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
 wrote:

On 2015-10-26 22:00, cheater00 . wrote:


Hello,
currently my computer freezes every several seconds for half a second
or so. Using it feels like I'm playing musical chairs with the kernel.
I have just one download happening on utorrent right now - this is
what the graph looks like:
http://i.imgur.com/LqhMtrJ.png
and every time a new spike happens, a freeze happens just before
that... that's the only time those freezes happen, too.


Do you have the 'autodefrag' mount option enabled?  If it is turned on, then
that may be the problem.  Most bittorrent clients pre-allocate the space for
a download, then write each block directly into the location it's supposed
to be in the resultant download, which means depending on how it's
pre-allocating the space, that you end up with a large number of randomly
ordered writes into a single file, which in turn will trigger the autodefrag
code, which can cause latency spikes when you're also hitting the disk at
the same time.





smime.p7s
Description: S/MIME Cryptographic Signature

Re: ENOSPC regression in integration-4.4 branch

[Chris Mason]

On Tue, Oct 27, 2015 at 09:34:13AM +, Filipe Manana wrote:
> Hi Josef,
> 
> Not sure if you are aware or got my report on IRC, but one of the
> allocator fixes/improvements (Chris' integration-4.4 branch) is
> causing new ENOSPC failures:
> 
> http://git.kernel.org/cgit/linux/kernel/git/mason/linux-btrfs.git/commit/?h=integration-4.4=a5e681d9bd641c4f0677e87d3a0c92a8f4f16293
> 
> (couldn't find the patch in the mailing list, so starting this thread)
> 
> A few xfstests (generic ones) are now failing, like generic/077 which
> constantly fails for me with:

I was having trouble hitting this consistently, but I did see it in the
integration branch just before heading out to the kernel summit.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 06/21] btrfs: delayed_ref: Add new function to record reserved space into delayed ref

[Chris Mason]

On Tue, Oct 27, 2015 at 05:05:56PM +0800, Qu Wenruo wrote:
> 
> 
> Chris Mason wrote on 2015/10/27 02:12 -0400:
> >On Tue, Oct 27, 2015 at 01:48:34PM +0800, Qu Wenruo wrote:
> Are you testing integration-4.4 from Chris repo?
> Or 4.3-rc from mainline repo with my qgroup reserve patchset applied?
> 
> Although integration-4.4 already merged qgroup reserve patchset, but it's
> causing some strange bug like over decrease data sinfo->bytes_may_use,
> mainly in generic/127 testcase.
> 
> But if qgroup reserve patchset is rebased to integration-4.3 (I did all my
> old tests based on that), no generic/127 problem at all.
> >>>
> >>>Did I mismerge things?
> >>>
> >>>-chris
> >>>
> >>Not sure yet.
> >>
> >>But at least some patches in 4.3 is not in integration-4.4, like the
> >>following patch:
> >>btrfs: Avoid truncate tailing page if fallocate range doesn't exceed inode
> >>size
> >
> >Have you tried testing integration-4.4 merged with current Linus git?
> >
> >-chris
> >
> Integration-4.4 merged with Linus' master also fails. :(
> 
> Current known working branches are all based on 4.3-integration(4.2-rc5):
> https://github.com/adam900710/linux/tree/qgroup_reserve_good
> 
> Tried 4.3-rc5 and 4.3-rc7, all fails with kernel warning in generic/137.
> 
> And due to the huge difference, I'm afraid it won't take a short time to
> find the root cause...

Ok, this is the top merge commit in integration:

commit a9e6d153563d2ed69c6cd7fb4fa5ce4ca7c712eb
Merge: 56fa9d0 0584f71
Author: Chris Mason 
Date:   Wed Oct 21 19:00:38 2015 -0700

Merge branch 'allocator-fixes' into for-linus-4.4

Please try commit 56fa9d0, which doesn't have Josef's allocator fixes.
It's possible there is a conflict with your changes in there.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-balance causes system-freeze on full disk

[Jakob Schürz]

Found a new habit...

I have a lot Snapshots on my drive. (Take every 10 Min a new one, every
houry one, every day one, every system-update, every plugin after
external HD...)

If there are to much snapshots (didn't find out the count how much
exactly) balancing fails with a system-freeze.
if i delete some of my snapshots, balancing is working...

Maybe this is a hint for bugfixing!!

Do you think?

greez

Jakob

Am 2015-10-21 um 22:51 schrieb Kyle Manna:
> I had a number of similar btrfs balance crashes in the past few days,
> but the disk wasn't full.  You should try tailing the system logs from
> a remote machine when it happens. You'll likely see some bug info
> before the system dies and becomes unusable.
> 
> The issue I encountered is described @
> https://bugzilla.kernel.org/show_bug.cgi?id=105681
> ᐧ
> 
> On Wed, Oct 21, 2015 at 12:38 PM, Jakob Schürz
>  wrote:
>> Hi there!
>>
>> Is it possible, what i've recognized now. My system (debian) runs on
>> btrfs, and i have a lot of snapshots on my hard-disk.
>> Since some days my system freezes totally. I recognized, it always
>> happens during btrfs-balance.
>>
>> So i deleted some of the old snapshots and tried another balance-run.
>> Nothing happened... No system-freeze.
>>
>> System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
>> is frozen, no magic-sysreq, no ssh-login.
>>
>> Can btrfs cause such a freeze??
>>
>> greez
>>
>> jakob
>> --
>> http://xundeenergie.at
>> http://verkehrsloesungen.wordpress.com/
>> http://cogitationum.wordpress.com/
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
http://xundeenergie.at
http://verkehrsloesungen.wordpress.com/
http://cogitationum.wordpress.com/



signature.asc
Description: OpenPGP digital signature

Re: Bad fs performance, IO freezes

[cheater00 .]

I'll just add nodatacow to the fstab, because the whole disk is for downloads.
There actually is compression happening?

On Tue, Oct 27, 2015 at 4:01 PM, Holger Hoffstätte
 wrote:
> On 10/27/15 15:43, cheater00 . wrote:
>> I have remounted without autodefrag and the issue keeps on happening.
>
> At the risk of stating the obvious, the simplest thing is to stop using
> COW with compression for torrents. It's fundamentally not useful to have
> many small in-place writes, irregular compression and not encounter insane
> levels of fragmentation, which - at least in the current state of btrfs -
> is known to cause long stalls. Some fixes for that will be in 4.4.
>
> Until then just do yourself a favor, stop doing the wrong thing and put
> torrent downloads into a directory where they won't get COWed.
>
> See the wiki at:
> https://btrfs.wiki.kernel.org/index.php/FAQ#Can_copy-on-write_be_turned_off_for_data_blocks.3F
> for more.
>
> -h
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Holger Hoffstätte]

On 10/27/15 15:43, cheater00 . wrote:
> I have remounted without autodefrag and the issue keeps on happening.

At the risk of stating the obvious, the simplest thing is to stop using
COW with compression for torrents. It's fundamentally not useful to have
many small in-place writes, irregular compression and not encounter insane
levels of fragmentation, which - at least in the current state of btrfs -
is known to cause long stalls. Some fixes for that will be in 4.4.

Until then just do yourself a favor, stop doing the wrong thing and put
torrent downloads into a directory where they won't get COWed.

See the wiki at:
https://btrfs.wiki.kernel.org/index.php/FAQ#Can_copy-on-write_be_turned_off_for_data_blocks.3F
for more.

-h

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v7 1/4] vfs: add copy_file_range syscall and vfs helper

[Steve French]

The patch set looks good so far, and copy offload is important for
CIFS/SMB3 (not just Windows/Mac/Samba but most servers support one of
the various methods available to do this over the network via CIFS and
also SMB2/SMB3).  I have only implemented two ways of copy offload in
cifs.ko so far but will look at adding the remaining mechanisms.

Currently cifs.ko does have a similar restriction to the below one
that you have in your patch1, but I was planning to relax it (as long
as source and target are on the same server) since some of the
important use cases are having the server do copy offload from one
share to another.   So we should support the case where the files are
on the same file system type (nfs, cifs etc.) but not necessarily on
the same superblock (for the case of cifs they will be on the same
server, but with some forms of copy offload that would not necessarily
have to be on the same server).

The following check should be removed (an alternative would be to
check that source and target are the same filesystem type ie nfs, or
cifs or xfs etc. or simply let the file systems check file_in and
file_out's validity)

> +   /* this could be relaxed once a method supports cross-fs copies */
> +   if (inode_in->i_sb != inode_out->i_sb ||
> +   file_in->f_path.mnt != file_out->f_path.mnt)
> +   return -EXDEV;




-- 
Thanks,

Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Austin S Hemmelgarn]

On 2015-10-27 10:43, cheater00 . wrote:

I have remounted without autodefrag and the issue keeps on happening.
OK, that at least narrows things down further.  My guess is the spikes 
are utorrent getting a bunch of blocks at once from one place, and then 
trying to write all of them at the same time, which could theoretically 
cause a latency spike on any filesystem, and BTRFS may just be making it 
worse.


On Tue, Oct 27, 2015 at 3:30 PM, cheater00 .  wrote:

Feel free to suggest a good 1.5m USB3 cable, too. Let's get rid of all
the unknowns.
When it comes to external cables, I've had really good success with 
Amazon's 'Amazon Basics' branded stuff.  It's usually some of the best 
quality you can find for the price.  The 'Cable Matters' and 'Pluggable' 
brands also tend to be really good quality for the price.

On Tue, Oct 27, 2015 at 3:26 PM, cheater00 .  wrote:

If you can suggest a dual (or better yet quad) USB3 bay that can be
bought on Amazon, I'll buy it now, and once that arrives, we can be
sure it's not the JMicron chipset.
I don't really have any suggestions here.  Usually when I hook up an 
external drive, it's to recover data from a friends computer, so I don't 
typically use a enclosure, but just use a simple adapter cable.  I would 
suggest looking for one advertising 'UAS' or 'UASP' support, as that's a 
relatively new standard for USB storage devices, and newer hardware 
should be more reliable.  It's also notoriously hard to determine what 
chipset a given model of external drive bay has (there are people I know 
who bought multiples of the same model and each one had a different 
chipset internally), and to complicate matters, quite often the exact 
same hardware gets marketed under half a dozen different names.  JMicron 
is popular because their chips are comparatively inexpensive, and while 
I've not had good results with them, that doesn't mean that they are all 
bad (especially considering that they are highly configurable based on 
how they are wired into the device, and not everyone who designs 
hardware around them properly understands the implications of some of 
the features).


On Tue, Oct 27, 2015 at 3:22 PM, cheater00 .  wrote:

The (dual) HDD bay and the chipset are, according to lsusb:
Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
USA Technology Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Not sure how to find out specific model numbers? I could open up the
bay. OK I'll open up the bay.
Good thing I have just the right screwdriver. It's a JMS551, and just
for records sake, here's the manufacture info:

JMS551
1120 LGAA2 A
572QV0024

The laptop manual says it's either "Intel HM65 Express chipset with
NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
Here are technical documents for my model:
Manual: http://docdro.id/hG627JM
"Intel chipset datasheet": http://docdro.id/yKRupYO
Service guide: http://docdro.id/AuDgUdE
Service guide, alt. ver.: http://docdro.id/WwQRpsH
From what I can tell, you've got the one with the NEC USB 3.0 chip, I'm 
pretty cure that the HM65 doesn't have USB 3.0 itself.  FWIW, I've never 
personally had issues with NEC's USB 3.0 chips, but I've not had much 
experience using systems with them either.


FWIW I'm using one of the USB3 ports on the left. The ones on the
right are USB2.

I've never used docdro.id so if it's not good let me know where to
upload the PDFs to.

autodefrag is on, yes. But I have been having issues before turning it
on - I turned it on as a measure towards fixing the issues. I will
turn it off and remount, then report. But I don't think that should be
it. As you see the transfer speeds are minimal. They're *all* that's
happening on the disk. Right now that's under 100 KB/sec and I'm still
getting freezes albeit less. Also why would I be getting freezes when
the transfer speeds jump up - just for them to drop again? Hmm, maybe
utorrent has some sort of scheduler that gets preempted while the
spike is happening, and some algorithm in it gets the wrong idea and
turns some sort of flow control on, because it thinks it's hit some
sort of physical transfer speed barrier. Also notice the upload and
download both scale together, but that just might be how torrent
works, maybe it just tries to be fair i.e. only uploads as much as it
downloaded (scaled by a constant).
Yeah, utorrent defaults to trying for a 1:1 ratio of uploads to 
downloads (so in terms of viewing the group of clients downloading a 
torrent as a network, it defaults to contributing as much bandwidth as 
it uses).  This is pretty typical behavior for most torrent clients, and 
in fact downloading more than you upload for a torrent is generally 
considered bad etiquette.


The system is 32 bit because I installed ubuntu 6 one day and just
kept on upgrading. I keep on telling myself I'll update to 64 bits,
one of these days. But this laptop only has 

Re: Bad fs performance, IO freezes

[Holger Hoffstätte]

On 10/27/15 16:07, cheater00 . wrote:
> Can I have nodatacow but still have checksumming?

No, see "nodatacow":
https://btrfs.wiki.kernel.org/index.php/Mount_options

Torrents are typically rehashed both on a per-block basis and after
completion. That won't protect you from bitflips or fs corruption, but
then again you don't have data redundancy anyway, so it doesn't really
matter.

Also any silently bitflipped blocks will be rejected by peers since
they don't match they block hash, so even in that case you won't be
distributing corrupt data.

-h

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[cheater00 .]

Can I have nodatacow but still have checksumming?

On Tue, Oct 27, 2015 at 4:05 PM, cheater00 .  wrote:
> I'll just add nodatacow to the fstab, because the whole disk is for downloads.
> There actually is compression happening?
>
> On Tue, Oct 27, 2015 at 4:01 PM, Holger Hoffstätte
>  wrote:
>> On 10/27/15 15:43, cheater00 . wrote:
>>> I have remounted without autodefrag and the issue keeps on happening.
>>
>> At the risk of stating the obvious, the simplest thing is to stop using
>> COW with compression for torrents. It's fundamentally not useful to have
>> many small in-place writes, irregular compression and not encounter insane
>> levels of fragmentation, which - at least in the current state of btrfs -
>> is known to cause long stalls. Some fixes for that will be in 4.4.
>>
>> Until then just do yourself a favor, stop doing the wrong thing and put
>> torrent downloads into a directory where they won't get COWed.
>>
>> See the wiki at:
>> https://btrfs.wiki.kernel.org/index.php/FAQ#Can_copy-on-write_be_turned_off_for_data_blocks.3F
>> for more.
>>
>> -h
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[cheater00 .]

I have remounted without autodefrag and the issue keeps on happening.

On Tue, Oct 27, 2015 at 3:30 PM, cheater00 .  wrote:
> Feel free to suggest a good 1.5m USB3 cable, too. Let's get rid of all
> the unknowns.
>
> On Tue, Oct 27, 2015 at 3:26 PM, cheater00 .  wrote:
>> If you can suggest a dual (or better yet quad) USB3 bay that can be
>> bought on Amazon, I'll buy it now, and once that arrives, we can be
>> sure it's not the JMicron chipset.
>>
>> On Tue, Oct 27, 2015 at 3:22 PM, cheater00 .  wrote:
>>> The (dual) HDD bay and the chipset are, according to lsusb:
>>> Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
>>> USA Technology Corp.
>>> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
>>>
>>> Not sure how to find out specific model numbers? I could open up the
>>> bay. OK I'll open up the bay.
>>> Good thing I have just the right screwdriver. It's a JMS551, and just
>>> for records sake, here's the manufacture info:
>>>
>>> JMS551
>>> 1120 LGAA2 A
>>> 572QV0024
>>>
>>> The laptop manual says it's either "Intel HM65 Express chipset with
>>> NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
>>> Here are technical documents for my model:
>>> Manual: http://docdro.id/hG627JM
>>> "Intel chipset datasheet": http://docdro.id/yKRupYO
>>> Service guide: http://docdro.id/AuDgUdE
>>> Service guide, alt. ver.: http://docdro.id/WwQRpsH
>>>
>>> FWIW I'm using one of the USB3 ports on the left. The ones on the
>>> right are USB2.
>>>
>>> I've never used docdro.id so if it's not good let me know where to
>>> upload the PDFs to.
>>>
>>> autodefrag is on, yes. But I have been having issues before turning it
>>> on - I turned it on as a measure towards fixing the issues. I will
>>> turn it off and remount, then report. But I don't think that should be
>>> it. As you see the transfer speeds are minimal. They're *all* that's
>>> happening on the disk. Right now that's under 100 KB/sec and I'm still
>>> getting freezes albeit less. Also why would I be getting freezes when
>>> the transfer speeds jump up - just for them to drop again? Hmm, maybe
>>> utorrent has some sort of scheduler that gets preempted while the
>>> spike is happening, and some algorithm in it gets the wrong idea and
>>> turns some sort of flow control on, because it thinks it's hit some
>>> sort of physical transfer speed barrier. Also notice the upload and
>>> download both scale together, but that just might be how torrent
>>> works, maybe it just tries to be fair i.e. only uploads as much as it
>>> downloaded (scaled by a constant).
>>>
>>> The system is 32 bit because I installed ubuntu 6 one day and just
>>> kept on upgrading. I keep on telling myself I'll update to 64 bits,
>>> one of these days. But this laptop only has 8 gigs of ram, so no real
>>> reason to upgrade to 64 bit anyways. It's not like I need firefox to
>>> be able to eat 8 gb of ram whereas right now it can only eat 4. There
>>> is no simple upgrade path that I know of so it's either a fresh
>>> install or doing something like this: http://www.ewan.cc/?q=node/132
>>> -- I keep telling myself /one of these days/...
>>>
>>> On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
>>>  wrote:
 On 2015-10-27 09:00, Henk Slager wrote:
>
> I don't have a lot experience with autodefrag, but as indicated by
> Austin, expect a lot of full rewrites of files that are relatively
> slowly filled up by a torrent client, starting with a sparse file. So
> 1st advice would be to remove this option and run it as crontask at
> particular times.
>
> What SATA-USB bridge is between the harddisk and the PC motherboard ?

 I hadn't thought of this, but the specific adapter being used for the disk
 can have a lot of impact on how it preforms.  I've personally had lots of
 issues with JMicron chipsets (ranging from latency issues like what you are
 seeing to sever data corruption), but have found that ASMedia ones tend to
 be pretty much rock solid reliable and have good performance (although I
 think they only do USB 3.0 adapters).
>
> Also what USB host chipset is on the PC motherboard ?

 If it's a Intel motherboard, the USB 2.0 ports are probably routed through
 on-board hubs to the ports provided by whatever Intel calls their 
 equivalent
 of the south bridge these days, and the USB 3.0 ports are probably a mix of
 Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
 to do an inexpensive standalone XHCI chip, so they're relatively popular 
 for
 extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
 some issues with older Linux kernels, although IIRC those issues were along
 the lines of a port just disappearing after disconnecting whatever was
 connected to it.
>
> Why don't you run 64-bit Ubuntu on 

Re: random i/o error without error in dmesg

[Szalma László]

I wait for the problem to occur, i will try the remount,ro remount,rw. 
But I know for sure, in my case umount / mount solved the problem every 
time!


Reboot was not needed. (I think reboot is simpler because the fs is 
used, and the service has to be stopped etc.)


László Szalma

2015-10-27 07:23 keltezéssel, Duncan írta:

Marc Joliet posted on Mon, 26 Oct 2015 15:23:39 +0100 as excerpted:


Occasionally they go away by themselves, but usually I have to reboot to
make them go away.  This happens when getmail attempts to fetch mail,
which fails due to the above error.  After the reboot getmail succeeds
again.

Just out of curiosity, does a remount,ro, followed by a remount,rw, solve
the problem?

The ro/rw cycle should force anything in memory to device, so if that
eliminates the problem, it could well be some sort of sync issue.  If it
doesn't, then it's more likely an in-memory filesystem state issue,
that's cleared by the reboot.

And if the ro/rw cycle doesn't clear the problem, what about a full
unmount/mount cycle, at least of that subvolume?

If you're running multiple subvolumes with root being one of them, you
can't of course unmount the entire filesystem, but you could go down to
emergency mode (systemctl emergency), try unmounting everything but /,
mounting / ro, and then switching back to normal mode (from emergency
mode, simply exiting should return you to normal multi-user or gui
target, remounting filesystems as necessary, etc).

IOW, does it take a full reboot to clear the problem, or is a simple ro/rw
mount cycle enough, or an unmount/remount?

Finally, assuming root itself isn't btrfs, if you have btrfs configured
as a module, you could try unmounting all btrfs and then unloading the
module, then reloading and remounting.  That should entirely clear all in-
memory btrfs state, so if that doesn't solve the problem, while rebooting
does, then the problem's very possibly outside of btrfs scope.  Of course
if root is btrfs, you can't really check that.



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-balance causes system-freeze on full disk

[Jakob Schürz]

Am 2015-10-27 um 18:09 schrieb Hugo Mills:
> On Tue, Oct 27, 2015 at 05:05:55PM +0100, Jakob Schürz wrote:
>> Found a new habit...
>>
>> I have a lot Snapshots on my drive. (Take every 10 Min a new one, every
>> houry one, every day one, every system-update, every plugin after
>> external HD...)
>>
>> If there are to much snapshots (didn't find out the count how much
>> exactly) balancing fails with a system-freeze.
>> if i delete some of my snapshots, balancing is working...
> 
>The whole machine comes to a halt, or just the balancing? 

The whole machine. Even magic-sysreq-keys are not reacting... I have to
reboot my machine using the power-button...

> If you
> have lots of snapshots, balance can take *insane* amounts of time --
> my big storage array, for example, takes about 60 seconds to balance a
> data block group, and something like 4+ hours to balance some of the
> metadata block groups. While it's "stuck" like that, it is actually
> making progress, but it doesn't look like it.

I know, balancint takes a huge amount of time. So my command is:

/bin/btrfs balance start -dusage=55 -v 

When balancing is running, the machine reactes a little slower than
normal... but in my case, the machine is freezing. completly and
totally... :(

greez jakob





signature.asc
Description: OpenPGP digital signature

Re: [PATCH 2/9] cifs: add .copy_file_range file operation

[Steve French]

Patch looks fine, but it points out a Kconfig problem with cifs.ko.
Need to remove the

 #ifdef CONFIG_CIFS_POSIX

around the statements like:

 .unlocked_ioctl = cifs_ioctl,
 .copy_file_range = cifs_file_copy_range,

Copy offload does not require support for the old CIFS POSIX
extensions (or even cifs at all) - it works fine over SMB3.
Alternatively you can move the copy_file_range to the end of the
struct file_operations in your patch, and I can remove the
CONFIG_CIFS_POSIX around cifs_ioctl in a different patch that I put in
my tree - whichever you prefer.

Originally the first ioctl that cifs supported did require the CIFS
POSIX extensions because it used an info level defined in the CIFS
UNIX/POSIX extensions.  With later additions to cifs_ioctl that
restriction is now obsolete.  Later ioctls have been added that work
over SMB3, and do not require the CIFS POSIX extensions (such as copy
offload).   The patch that originally introduced the CIFS_POSIX ifdef
around this ioctl was

commit c67593a03129967eae8939c4899767182eb6d6cd
Author: Steve French 
Date:   Thu Apr 28 22:41:04 2005 -0700

[PATCH] cifs: Enable ioctl support in POSIX extensions to handle lsattr

but obviously now we don't need the CONFIG_CIFS_POSIX ifdef around the
ioctl functions.

On Sat, Oct 24, 2015 at 6:17 PM, Peng Tao  wrote:
> Signed-off-by: Peng Tao 
> ---
>  fs/cifs/cifsfs.c |  22 
>  fs/cifs/cifsfs.h |   4 ++-
>  fs/cifs/ioctl.c  | 100 
> +++
>  3 files changed, 82 insertions(+), 44 deletions(-)
>
> diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
> index e739950..6ef7c3c 100644
> --- a/fs/cifs/cifsfs.c
> +++ b/fs/cifs/cifsfs.c
> @@ -910,6 +910,22 @@ const struct inode_operations cifs_symlink_inode_ops = {
>  #endif
>  };
>
> +#ifdef CONFIG_CIFS_POSIX
> +ssize_t cifs_file_copy_range(struct file *file_in, loff_t pos_in,
> +struct file *file_out, loff_t pos_out,
> +size_t len, unsigned int flags)
> +{
> +   unsigned int xid;
> +   int rc;
> +
> +   xid = get_xid();
> +   rc = cifs_file_clone_range(xid, file_in, file_out, pos_in,
> +  len, pos_out, true);
> +   free_xid(xid);
> +   return rc < 0 ? rc : len;
> +}
> +#endif
> +
>  const struct file_operations cifs_file_ops = {
> .read_iter = cifs_loose_read_iter,
> .write_iter = cifs_file_write_iter,
> @@ -923,6 +939,7 @@ const struct file_operations cifs_file_ops = {
> .llseek = cifs_llseek,
>  #ifdef CONFIG_CIFS_POSIX
> .unlocked_ioctl = cifs_ioctl,
> +   .copy_file_range = cifs_file_copy_range,
>  #endif /* CONFIG_CIFS_POSIX */
> .setlease = cifs_setlease,
> .fallocate = cifs_fallocate,
> @@ -941,6 +958,7 @@ const struct file_operations cifs_file_strict_ops = {
> .llseek = cifs_llseek,
>  #ifdef CONFIG_CIFS_POSIX
> .unlocked_ioctl = cifs_ioctl,
> +   .copy_file_range = cifs_file_copy_range,
>  #endif /* CONFIG_CIFS_POSIX */
> .setlease = cifs_setlease,
> .fallocate = cifs_fallocate,
> @@ -959,6 +977,7 @@ const struct file_operations cifs_file_direct_ops = {
> .splice_read = generic_file_splice_read,
>  #ifdef CONFIG_CIFS_POSIX
> .unlocked_ioctl  = cifs_ioctl,
> +   .copy_file_range = cifs_file_copy_range,
>  #endif /* CONFIG_CIFS_POSIX */
> .llseek = cifs_llseek,
> .setlease = cifs_setlease,
> @@ -977,6 +996,7 @@ const struct file_operations cifs_file_nobrl_ops = {
> .llseek = cifs_llseek,
>  #ifdef CONFIG_CIFS_POSIX
> .unlocked_ioctl = cifs_ioctl,
> +   .copy_file_range = cifs_file_copy_range,
>  #endif /* CONFIG_CIFS_POSIX */
> .setlease = cifs_setlease,
> .fallocate = cifs_fallocate,
> @@ -994,6 +1014,7 @@ const struct file_operations cifs_file_strict_nobrl_ops 
> = {
> .llseek = cifs_llseek,
>  #ifdef CONFIG_CIFS_POSIX
> .unlocked_ioctl = cifs_ioctl,
> +   .copy_file_range = cifs_file_copy_range,
>  #endif /* CONFIG_CIFS_POSIX */
> .setlease = cifs_setlease,
> .fallocate = cifs_fallocate,
> @@ -1011,6 +1032,7 @@ const struct file_operations cifs_file_direct_nobrl_ops 
> = {
> .splice_read = generic_file_splice_read,
>  #ifdef CONFIG_CIFS_POSIX
> .unlocked_ioctl  = cifs_ioctl,
> +   .copy_file_range = cifs_file_copy_range,
>  #endif /* CONFIG_CIFS_POSIX */
> .llseek = cifs_llseek,
> .setlease = cifs_setlease,


Patch looks fine, but it points out a Kconfig problem with cifs.ko.
Need to remove the

#ifdef CONFIG_CIFS_POSIX

around the

 .unlocked_ioctl = cifs_ioctl,
 .copy_file_range = cifs_file_copy_range,

statements.  Alternatively you can move the copy_file_range to the end
of the struct file_operations in your patch, and I can remove the
CONFIG_CIFS_POSIX 

Re: btrfs-balance causes system-freeze on full disk

[Filipe Manana]

On Tue, Oct 27, 2015 at 4:05 PM, Jakob Schürz  wrote:
> Found a new habit...
>
> I have a lot Snapshots on my drive. (Take every 10 Min a new one, every
> houry one, every day one, every system-update, every plugin after
> external HD...)
>
> If there are to much snapshots (didn't find out the count how much
> exactly) balancing fails with a system-freeze.
> if i delete some of my snapshots, balancing is working...
>
> Maybe this is a hint for bugfixing!!

A patch set to fix this issue was already sent yesterday and tested
over the weekend by Stéphane:

http://thread.gmane.org/gmane.comp.file-systems.btrfs/49630

cheers

>
> Do you think?
>
> greez
>
> Jakob
>
> Am 2015-10-21 um 22:51 schrieb Kyle Manna:
>> I had a number of similar btrfs balance crashes in the past few days,
>> but the disk wasn't full.  You should try tailing the system logs from
>> a remote machine when it happens. You'll likely see some bug info
>> before the system dies and becomes unusable.
>>
>> The issue I encountered is described @
>> https://bugzilla.kernel.org/show_bug.cgi?id=105681
>> ᐧ
>>
>> On Wed, Oct 21, 2015 at 12:38 PM, Jakob Schürz
>>  wrote:
>>> Hi there!
>>>
>>> Is it possible, what i've recognized now. My system (debian) runs on
>>> btrfs, and i have a lot of snapshots on my hard-disk.
>>> Since some days my system freezes totally. I recognized, it always
>>> happens during btrfs-balance.
>>>
>>> So i deleted some of the old snapshots and tried another balance-run.
>>> Nothing happened... No system-freeze.
>>>
>>> System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
>>> is frozen, no magic-sysreq, no ssh-login.
>>>
>>> Can btrfs cause such a freeze??
>>>
>>> greez
>>>
>>> jakob
>>> --
>>> http://xundeenergie.at
>>> http://verkehrsloesungen.wordpress.com/
>>> http://cogitationum.wordpress.com/
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majord...@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
> --
> http://xundeenergie.at
> http://verkehrsloesungen.wordpress.com/
> http://cogitationum.wordpress.com/
>



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-balance causes system-freeze on full disk

[Hugo Mills]

On Tue, Oct 27, 2015 at 05:05:55PM +0100, Jakob Schürz wrote:
> Found a new habit...
> 
> I have a lot Snapshots on my drive. (Take every 10 Min a new one, every
> houry one, every day one, every system-update, every plugin after
> external HD...)
> 
> If there are to much snapshots (didn't find out the count how much
> exactly) balancing fails with a system-freeze.
> if i delete some of my snapshots, balancing is working...

   The whole machine comes to a halt, or just the balancing? If you
have lots of snapshots, balance can take *insane* amounts of time --
my big storage array, for example, takes about 60 seconds to balance a
data block group, and something like 4+ hours to balance some of the
metadata block groups. While it's "stuck" like that, it is actually
making progress, but it doesn't look like it.

   Hugo.

> Maybe this is a hint for bugfixing!!
> 
> Do you think?
> 
> greez
> 
> Jakob
> 
> Am 2015-10-21 um 22:51 schrieb Kyle Manna:
> > I had a number of similar btrfs balance crashes in the past few days,
> > but the disk wasn't full.  You should try tailing the system logs from
> > a remote machine when it happens. You'll likely see some bug info
> > before the system dies and becomes unusable.
> > 
> > The issue I encountered is described @
> > https://bugzilla.kernel.org/show_bug.cgi?id=105681
> > ᐧ
> > 
> > On Wed, Oct 21, 2015 at 12:38 PM, Jakob Schürz
> >  wrote:
> >> Hi there!
> >>
> >> Is it possible, what i've recognized now. My system (debian) runs on
> >> btrfs, and i have a lot of snapshots on my hard-disk.
> >> Since some days my system freezes totally. I recognized, it always
> >> happens during btrfs-balance.
> >>
> >> So i deleted some of the old snapshots and tried another balance-run.
> >> Nothing happened... No system-freeze.
> >>
> >> System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
> >> is frozen, no magic-sysreq, no ssh-login.
> >>
> >> Can btrfs cause such a freeze??
> >>
> >> greez
> >>
> >> jakob

-- 
Hugo Mills | Darkling's First Law of Filesystems:
hugo@... carfax.org.uk | The user hates their data
http://carfax.org.uk/  |
PGP: E2AB1DE4  |


signature.asc
Description: Digital signature

Re: random i/o error without error in dmesg

[Duncan]

Marc Joliet posted on Tue, 27 Oct 2015 21:54:40 +0100 as excerpted:

>>IOW, does it take a full reboot to clear the problem, or is a simple
>>ro/rw mount cycle enough, or an unmount/remount?
> 
> Seems that a full reboot is needed, but I would expect that it would
> have the same effect if I were to pivot back into the initramfs, unmount
> / from there,
> then boot back into the system.  Because quite frankly, I can't think of
> any reason why a power cycle to the SSD should make a difference here. 
> I vaguely remember that systemd can do that, so I'll see if I can find
> out how.

Agree with both the systemd returning to the initr* point (which I 
actually had in mind while writing the above but don't remember the 
details either, so chose to omit in the interest of limiting the size of 
the reply and research necessary to generate it), and the ssd power-cycle 
point.

>>Finally, assuming root itself isn't btrfs, if you have btrfs configured
>>as a module, you could try unmounting all btrfs and then unloading the
>>module, then reloading and remounting.  That should entirely clear all
>>in-memory btrfs state, so if that doesn't solve the problem, while
>>rebooting does, then the problem's very possibly outside of btrfs scope.
>> Of course if root is btrfs, you can't really check that.
> 
> Nope, btrfs is built-in (though it doesn't have to be, what with me
> using an initramfs).

Same here, also gentoo as I guess you know from previous exchanges.  But 
unfortunately, if your initr* is anything like mine, and your kernel 
monolithic as mine, making btrfs a module with a btrfs root isn't the 
easy thing it might seem to those who run ordinary distro supplied binary 
kernels with pretty much everything modularized, as doing so involves a 
whole new set of research on how to get that module properly included in 
the initr* and loaded there, as well as installing and building the whole 
module-handling infrastructure (modprobe and friends) again, as it's not 
actually installed on the system at all at this point, because with the 
kernel entirely monolithic, module-handling tools are unnecessary and 
thus just another unnecessary package to have to keep building updates 
for, if they remain installed.

So I definitely sympathize with the feeling that such a stone is better 
left unturned, if overturning it is at all a possibility that can be 
avoided, as it is here, this whole exercise being simply one of better 
pinning the bug down, not yet actually trying to solve it.  And given 
that unturned stone, there are certainly easier ways.

And one of those easier ways is investigating that whole systemd return 
to initr* idea, since we both remember reading something about it, but 
aren't familiar with the details.  In addition to addressing the problem 
headon if anyone offers a way to do so, that's the path I'd be looking at 
right now.


Meanwhile, addressing something in the snipped content, if a mount 
refuses to remount read-only, it's normally due to deleted but still open 
files.  Often, in many cases the overwhelming majority of the time, 
that's due to updates that have been done, removing for instance old 
library files that are still loaded into running executables.  The 
generalized solution is to quit or restart the affected executables, thus 
releasing the last open references to the otherwise already deleted 
library and other files, so the filesystem can finish deleting them.  
Once this is done, the filesystem can normally be remounted ro (tho bugs 
like the one of this thread may prevent that).

The problem lies in actually figuring out what still running executables 
are still holding open file references to otherwise deleted files.  /proc/ 
actually makes this sort of data available in its collection of files for 
a running process, as there's one (as the saying goes, finding which one 
is an exercise left for the reader, as I could look it up or check, but 
so can you if you're /that/ interested) that tells which files the 
process has open, and deleted files are clearly marked.

But it's still a hassle to go thru the appropriate /proc/ files for each 
process, at least if it's done manually, so various helper tools have 
been developed to automate the process.  The one I use here is a python-
based script, packaged on gentoo as...

app-admin/lib_users

Run as root, it'll tell you which executables are running that still hold 
references to deleted files, so you can restart them.  Of course when a 
library was updated for security reasons doing this restart is a good 
idea from that perspective as well.  Some tools actually have a table of 
executables and the services they belong to, and will automate the 
restart, but of course these sorts of tools tend to be somewhat complex 
and distro specific, while lib_users simply tells you the executables and 
lets you decide what to do with that information, that being both simpler 
and without the distro-specifics of the lists that the fully 

Re: Bad fs performance, IO freezes

[cheater00 .]

The (dual) HDD bay and the chipset are, according to lsusb:
Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
USA Technology Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Not sure how to find out specific model numbers? I could open up the
bay. OK I'll open up the bay.
Good thing I have just the right screwdriver. It's a JMS551, and just
for records sake, here's the manufacture info:

JMS551
1120 LGAA2 A
572QV0024

The laptop manual says it's either "Intel HM65 Express chipset with
NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
Here are technical documents for my model:
Manual: http://docdro.id/hG627JM
"Intel chipset datasheet": http://docdro.id/yKRupYO
Service guide: http://docdro.id/AuDgUdE
Service guide, alt. ver.: http://docdro.id/WwQRpsH

FWIW I'm using one of the USB3 ports on the left. The ones on the
right are USB2.

I've never used docdro.id so if it's not good let me know where to
upload the PDFs to.

autodefrag is on, yes. But I have been having issues before turning it
on - I turned it on as a measure towards fixing the issues. I will
turn it off and remount, then report. But I don't think that should be
it. As you see the transfer speeds are minimal. They're *all* that's
happening on the disk. Right now that's under 100 KB/sec and I'm still
getting freezes albeit less. Also why would I be getting freezes when
the transfer speeds jump up - just for them to drop again? Hmm, maybe
utorrent has some sort of scheduler that gets preempted while the
spike is happening, and some algorithm in it gets the wrong idea and
turns some sort of flow control on, because it thinks it's hit some
sort of physical transfer speed barrier. Also notice the upload and
download both scale together, but that just might be how torrent
works, maybe it just tries to be fair i.e. only uploads as much as it
downloaded (scaled by a constant).

The system is 32 bit because I installed ubuntu 6 one day and just
kept on upgrading. I keep on telling myself I'll update to 64 bits,
one of these days. But this laptop only has 8 gigs of ram, so no real
reason to upgrade to 64 bit anyways. It's not like I need firefox to
be able to eat 8 gb of ram whereas right now it can only eat 4. There
is no simple upgrade path that I know of so it's either a fresh
install or doing something like this: http://www.ewan.cc/?q=node/132
-- I keep telling myself /one of these days/...

On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
 wrote:
> On 2015-10-27 09:00, Henk Slager wrote:
>>
>> I don't have a lot experience with autodefrag, but as indicated by
>> Austin, expect a lot of full rewrites of files that are relatively
>> slowly filled up by a torrent client, starting with a sparse file. So
>> 1st advice would be to remove this option and run it as crontask at
>> particular times.
>>
>> What SATA-USB bridge is between the harddisk and the PC motherboard ?
>
> I hadn't thought of this, but the specific adapter being used for the disk
> can have a lot of impact on how it preforms.  I've personally had lots of
> issues with JMicron chipsets (ranging from latency issues like what you are
> seeing to sever data corruption), but have found that ASMedia ones tend to
> be pretty much rock solid reliable and have good performance (although I
> think they only do USB 3.0 adapters).
>>
>> Also what USB host chipset is on the PC motherboard ?
>
> If it's a Intel motherboard, the USB 2.0 ports are probably routed through
> on-board hubs to the ports provided by whatever Intel calls their equivalent
> of the south bridge these days, and the USB 3.0 ports are probably a mix of
> Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
> to do an inexpensive standalone XHCI chip, so they're relatively popular for
> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
> some issues with older Linux kernels, although IIRC those issues were along
> the lines of a port just disappearing after disconnecting whatever was
> connected to it.
>>
>> Why don't you run 64-bit Ubuntu on this core i7 ?
>
> 64 versus 32 bit shouldn't cause anything like this to happen (although, if
> it can be proven that it does, then that is a serious bug that needs to be
> fixed).  That said, unless you have some desperate need to be running 32-bit
> only, you should seriously look into updating to a 64-bit version, your
> whole system should run faster, and Ubuntu has really good 32-bit
> compatibility in the 64-bit version (which is part of why it's popular as a
> support target for third party software like Steam).
>
>>
>>
>> On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
>>  wrote:
>>>
>>> On 2015-10-26 22:00, cheater00 . wrote:


 Hello,
 currently my computer freezes every several seconds for half a second
 or so. Using it feels like I'm playing musical chairs with the kernel.
 I have just one download 

Re: Bad fs performance, IO freezes

[cheater00 .]

If you can suggest a dual (or better yet quad) USB3 bay that can be
bought on Amazon, I'll buy it now, and once that arrives, we can be
sure it's not the JMicron chipset.

On Tue, Oct 27, 2015 at 3:22 PM, cheater00 .  wrote:
> The (dual) HDD bay and the chipset are, according to lsusb:
> Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
> USA Technology Corp.
> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
>
> Not sure how to find out specific model numbers? I could open up the
> bay. OK I'll open up the bay.
> Good thing I have just the right screwdriver. It's a JMS551, and just
> for records sake, here's the manufacture info:
>
> JMS551
> 1120 LGAA2 A
> 572QV0024
>
> The laptop manual says it's either "Intel HM65 Express chipset with
> NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
> Here are technical documents for my model:
> Manual: http://docdro.id/hG627JM
> "Intel chipset datasheet": http://docdro.id/yKRupYO
> Service guide: http://docdro.id/AuDgUdE
> Service guide, alt. ver.: http://docdro.id/WwQRpsH
>
> FWIW I'm using one of the USB3 ports on the left. The ones on the
> right are USB2.
>
> I've never used docdro.id so if it's not good let me know where to
> upload the PDFs to.
>
> autodefrag is on, yes. But I have been having issues before turning it
> on - I turned it on as a measure towards fixing the issues. I will
> turn it off and remount, then report. But I don't think that should be
> it. As you see the transfer speeds are minimal. They're *all* that's
> happening on the disk. Right now that's under 100 KB/sec and I'm still
> getting freezes albeit less. Also why would I be getting freezes when
> the transfer speeds jump up - just for them to drop again? Hmm, maybe
> utorrent has some sort of scheduler that gets preempted while the
> spike is happening, and some algorithm in it gets the wrong idea and
> turns some sort of flow control on, because it thinks it's hit some
> sort of physical transfer speed barrier. Also notice the upload and
> download both scale together, but that just might be how torrent
> works, maybe it just tries to be fair i.e. only uploads as much as it
> downloaded (scaled by a constant).
>
> The system is 32 bit because I installed ubuntu 6 one day and just
> kept on upgrading. I keep on telling myself I'll update to 64 bits,
> one of these days. But this laptop only has 8 gigs of ram, so no real
> reason to upgrade to 64 bit anyways. It's not like I need firefox to
> be able to eat 8 gb of ram whereas right now it can only eat 4. There
> is no simple upgrade path that I know of so it's either a fresh
> install or doing something like this: http://www.ewan.cc/?q=node/132
> -- I keep telling myself /one of these days/...
>
> On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
>  wrote:
>> On 2015-10-27 09:00, Henk Slager wrote:
>>>
>>> I don't have a lot experience with autodefrag, but as indicated by
>>> Austin, expect a lot of full rewrites of files that are relatively
>>> slowly filled up by a torrent client, starting with a sparse file. So
>>> 1st advice would be to remove this option and run it as crontask at
>>> particular times.
>>>
>>> What SATA-USB bridge is between the harddisk and the PC motherboard ?
>>
>> I hadn't thought of this, but the specific adapter being used for the disk
>> can have a lot of impact on how it preforms.  I've personally had lots of
>> issues with JMicron chipsets (ranging from latency issues like what you are
>> seeing to sever data corruption), but have found that ASMedia ones tend to
>> be pretty much rock solid reliable and have good performance (although I
>> think they only do USB 3.0 adapters).
>>>
>>> Also what USB host chipset is on the PC motherboard ?
>>
>> If it's a Intel motherboard, the USB 2.0 ports are probably routed through
>> on-board hubs to the ports provided by whatever Intel calls their equivalent
>> of the south bridge these days, and the USB 3.0 ports are probably a mix of
>> Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
>> to do an inexpensive standalone XHCI chip, so they're relatively popular for
>> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
>> some issues with older Linux kernels, although IIRC those issues were along
>> the lines of a port just disappearing after disconnecting whatever was
>> connected to it.
>>>
>>> Why don't you run 64-bit Ubuntu on this core i7 ?
>>
>> 64 versus 32 bit shouldn't cause anything like this to happen (although, if
>> it can be proven that it does, then that is a serious bug that needs to be
>> fixed).  That said, unless you have some desperate need to be running 32-bit
>> only, you should seriously look into updating to a 64-bit version, your
>> whole system should run faster, and Ubuntu has really good 32-bit
>> compatibility in the 64-bit version (which is part of why it's popular as a
>> support target for third 

Re: Bad fs performance, IO freezes

[cheater00 .]

Feel free to suggest a good 1.5m USB3 cable, too. Let's get rid of all
the unknowns.

On Tue, Oct 27, 2015 at 3:26 PM, cheater00 .  wrote:
> If you can suggest a dual (or better yet quad) USB3 bay that can be
> bought on Amazon, I'll buy it now, and once that arrives, we can be
> sure it's not the JMicron chipset.
>
> On Tue, Oct 27, 2015 at 3:22 PM, cheater00 .  wrote:
>> The (dual) HDD bay and the chipset are, according to lsusb:
>> Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
>> USA Technology Corp.
>> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
>>
>> Not sure how to find out specific model numbers? I could open up the
>> bay. OK I'll open up the bay.
>> Good thing I have just the right screwdriver. It's a JMS551, and just
>> for records sake, here's the manufacture info:
>>
>> JMS551
>> 1120 LGAA2 A
>> 572QV0024
>>
>> The laptop manual says it's either "Intel HM65 Express chipset with
>> NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
>> Here are technical documents for my model:
>> Manual: http://docdro.id/hG627JM
>> "Intel chipset datasheet": http://docdro.id/yKRupYO
>> Service guide: http://docdro.id/AuDgUdE
>> Service guide, alt. ver.: http://docdro.id/WwQRpsH
>>
>> FWIW I'm using one of the USB3 ports on the left. The ones on the
>> right are USB2.
>>
>> I've never used docdro.id so if it's not good let me know where to
>> upload the PDFs to.
>>
>> autodefrag is on, yes. But I have been having issues before turning it
>> on - I turned it on as a measure towards fixing the issues. I will
>> turn it off and remount, then report. But I don't think that should be
>> it. As you see the transfer speeds are minimal. They're *all* that's
>> happening on the disk. Right now that's under 100 KB/sec and I'm still
>> getting freezes albeit less. Also why would I be getting freezes when
>> the transfer speeds jump up - just for them to drop again? Hmm, maybe
>> utorrent has some sort of scheduler that gets preempted while the
>> spike is happening, and some algorithm in it gets the wrong idea and
>> turns some sort of flow control on, because it thinks it's hit some
>> sort of physical transfer speed barrier. Also notice the upload and
>> download both scale together, but that just might be how torrent
>> works, maybe it just tries to be fair i.e. only uploads as much as it
>> downloaded (scaled by a constant).
>>
>> The system is 32 bit because I installed ubuntu 6 one day and just
>> kept on upgrading. I keep on telling myself I'll update to 64 bits,
>> one of these days. But this laptop only has 8 gigs of ram, so no real
>> reason to upgrade to 64 bit anyways. It's not like I need firefox to
>> be able to eat 8 gb of ram whereas right now it can only eat 4. There
>> is no simple upgrade path that I know of so it's either a fresh
>> install or doing something like this: http://www.ewan.cc/?q=node/132
>> -- I keep telling myself /one of these days/...
>>
>> On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
>>  wrote:
>>> On 2015-10-27 09:00, Henk Slager wrote:

 I don't have a lot experience with autodefrag, but as indicated by
 Austin, expect a lot of full rewrites of files that are relatively
 slowly filled up by a torrent client, starting with a sparse file. So
 1st advice would be to remove this option and run it as crontask at
 particular times.

 What SATA-USB bridge is between the harddisk and the PC motherboard ?
>>>
>>> I hadn't thought of this, but the specific adapter being used for the disk
>>> can have a lot of impact on how it preforms.  I've personally had lots of
>>> issues with JMicron chipsets (ranging from latency issues like what you are
>>> seeing to sever data corruption), but have found that ASMedia ones tend to
>>> be pretty much rock solid reliable and have good performance (although I
>>> think they only do USB 3.0 adapters).

 Also what USB host chipset is on the PC motherboard ?
>>>
>>> If it's a Intel motherboard, the USB 2.0 ports are probably routed through
>>> on-board hubs to the ports provided by whatever Intel calls their equivalent
>>> of the south bridge these days, and the USB 3.0 ports are probably a mix of
>>> Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
>>> to do an inexpensive standalone XHCI chip, so they're relatively popular for
>>> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
>>> some issues with older Linux kernels, although IIRC those issues were along
>>> the lines of a port just disappearing after disconnecting whatever was
>>> connected to it.

 Why don't you run 64-bit Ubuntu on this core i7 ?
>>>
>>> 64 versus 32 bit shouldn't cause anything like this to happen (although, if
>>> it can be proven that it does, then that is a serious bug that needs to be
>>> fixed).  That said, unless you have some desperate need to be running