Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-17 Thread Christoph Anton Mitterer
[I'm combining the messages again, since I feel a bit bad, when I write so many mails to the list ;) ] But from my side, feel free to split up as much as you want (perhaps not single characters or so ;) ) On Thu, 2015-12-17 at 04:06 +, Duncan wrote: > Just to mention here, that I said

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Christoph Anton Mitterer
On Wed, 2015-12-09 at 16:36 +, Duncan wrote: > But... as I've pointed out in other replies, in many cases including > this > specific one (bittorrent), applications have already had to develop > their > own integrity management features Well let's move discussion upon that into the "dear

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Christoph Anton Mitterer
On Mon, 2015-12-14 at 10:51 +, Duncan wrote: > > AFAIU, the one the get's fragmented then is the snapshot, right, > > and the > > "original" will stay in place where it was? (Which is of course > > good, > > because one probably marked it nodatacow, to avoid that > > fragmentation > > problem

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Kai Krakow
Am Wed, 9 Dec 2015 13:36:01 + (UTC) schrieb Duncan <1i5t5.dun...@cox.net>: > >> > 4) Duncan mentioned that defrag (and I guess that's also for > >> > auto- defrag) isn't ref-link aware... > >> > Isn't that somehow a complete showstopper? > > >> It is, but the one attempt at dealing with it

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Duncan
Christoph Anton Mitterer posted on Wed, 16 Dec 2015 22:59:01 +0100 as excerpted: >> And there very well might be such a tool... five or ten years down the >> road when btrfs is much more mature and generally stabilized, well >> beyond the "still maturing and stabilizing" status of the moment. >

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Duncan
Christoph Anton Mitterer posted on Wed, 16 Dec 2015 22:59:01 +0100 as excerpted: > In kinda curios, what free space fragmentation actually means here. > > Ist simply like this: > +--+-+---++ > |     F    |  D  | F |    D   | > +--+-+---++ > Where D is data

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Duncan
Christoph Anton Mitterer posted on Wed, 16 Dec 2015 22:59:01 +0100 as excerpted: > I'm a bit unsure how to read filefrag's output... (even in the > uncompressed case). > What would it show me if there was fragmentation /path/to/file: 18 extents found It tells you the number of extents found.

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Duncan
Christoph Anton Mitterer posted on Wed, 16 Dec 2015 22:59:01 +0100 as excerpted: >> he obviously didn't think thru the fact that compression MUST be a >> rewrite, thereby breaking snapshot reflinks, even were normal >> non-compression defrag to be snapshot aware, because compression >>

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Duncan
Christoph Anton Mitterer posted on Wed, 16 Dec 2015 22:59:01 +0100 as excerpted: >> It's certainly in quite a few on-list posts over the years > okay,.. in other words: no ;-) > scatter over the years list posts don't count as documentation :P =:^) -- Duncan - List replies preferred. No

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-16 Thread Duncan
Christoph Anton Mitterer posted on Wed, 16 Dec 2015 22:59:01 +0100 as excerpted: > On Wed, 2015-12-09 at 16:36 +, Duncan wrote: >> But... as I've pointed out in other replies, in many cases including >> this specific one (bittorrent), applications have already had to >> develop their own

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-14 Thread Duncan
Christoph Anton Mitterer posted on Mon, 14 Dec 2015 02:44:55 +0100 as excerpted: > Two more on these: > > On Thu, 2015-11-26 at 00:33 +, Hugo Mills wrote: >> 3) When I would actually disable datacow for e.g. a subvolume that >> > holds VMs or DBs... what are all the implications? >> After

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-14 Thread Duncan
Christoph Anton Mitterer posted on Mon, 14 Dec 2015 03:46:01 +0100 as excerpted: >> Same here.  In fact, my most anticipated feature is N-way-mirroring, > Hmm ... not totally sure about that... > AFAIU, N-way-mirroring is what currently the currently wrongly called > RAID1 is in btrfs, i.e.

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-13 Thread Christoph Anton Mitterer
On Wed, 2015-12-09 at 13:36 +, Duncan wrote: > Answering the BTW first, not to my knowledge, and I'd be > skeptical.  In > general, btrfs is cowed, and that's the focus.  To the extent that > nocow > is necessary for fragmentation/performance reasons, etc, the idea is > to > try to make cow

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-13 Thread Christoph Anton Mitterer
Two more on these: On Thu, 2015-11-26 at 00:33 +, Hugo Mills wrote: > 3) When I would actually disable datacow for e.g. a subvolume that > > holds VMs or DBs... what are all the implications? > > Obviously no checksumming, but what happens if I snapshot such a > > subvolume or if I

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-09 Thread Duncan
Christoph Anton Mitterer posted on Wed, 09 Dec 2015 06:43:01 +0100 as excerpted: > Hey Hugo, > > > On Thu, 2015-11-26 at 00:33 +, Hugo Mills wrote: > >> The issue is that nodatacow bypasses the transactional nature of >> the FS, making changes to live data immediately. This then means that

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-09 Thread Duncan
Christoph Anton Mitterer posted on Wed, 09 Dec 2015 06:45:47 +0100 as excerpted: > On 2015-11-27 00:08, Duncan wrote: >> Christoph Anton Mitterer posted on Thu, 26 Nov 2015 01:23:59 +0100 as >> excerpted: >>> 1) AFAIU, the fragmentation problem exists especially for those files >>> that see many

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-08 Thread Christoph Anton Mitterer
Hey Hugo, On Thu, 2015-11-26 at 00:33 +, Hugo Mills wrote: >    Answering the second part first, no, it can't. Thanks so far :) >    The issue is that nodatacow bypasses the transactional nature of > the FS, making changes to live data immediately. This then means that > if you modify a

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-12-08 Thread Christoph Anton Mitterer
On 2015-11-27 00:08, Duncan wrote: > Christoph Anton Mitterer posted on Thu, 26 Nov 2015 01:23:59 +0100 as > excerpted: >> 1) AFAIU, the fragmentation problem exists especially for those files >> that see many random writes, especially, but not limited to, big files. >> Now that databases and VMs

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-11-26 Thread Duncan
Christoph Anton Mitterer posted on Thu, 26 Nov 2015 01:23:59 +0100 as excerpted: > Hey. > > I've worried before about the topics Mitch has raised. > Some questions. > > 1) AFAIU, the fragmentation problem exists especially for those files > that see many random writes, especially, but not

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-11-25 Thread Hugo Mills
On Thu, Nov 26, 2015 at 01:23:59AM +0100, Christoph Anton Mitterer wrote: > 2) Why does notdatacow imply nodatasum and can that ever be decoupled? Answering the second part first, no, it can't. The issue is that nodatacow bypasses the transactional nature of the FS, making changes to live

Re: [auto-]defrag, nodatacow - general suggestions?(was: btrfs: poor performance on deleting many large files?)

2015-11-25 Thread Christoph Anton Mitterer
Hey. I've worried before about the topics Mitch has raised. Some questions. 1) AFAIU, the fragmentation problem exists especially for those files that see many random writes, especially, but not limited to, big files. Now that databases and VMs are affected by this, is probably broadly known in