Re: [dm-devel] I/O block when removing thin device on the same pool

2016-01-31 Thread Dennis Yang 
Hi,

2016-01-30 0:05 GMT+08:00 Joe Thornber :

> On Fri, Jan 29, 2016 at 07:01:44PM +0800, Dennis Yang wrote:
> > I had tried to define MAX_DECS as 1, 16, and 8192, and here is the
> > throughput I got.
> > When #define MAX_DECS 1, throughput drops from 3.2GB/s to around 800 ~
> 950
> > MB/s.
> > When #define MAX_DECS 16, throughput drops from 3.2GB/s to around 150 ~
> 400
> > MB/s
> > When #define MAX_DECS 8192, the I/O blocks until deletion is done.
> >
> > These throughput is gathered by writing to a newly created thin device
> > which means lots of provisioning take place. So it seems that the more
> fine
> > grained lock we use here results in the higher throughput. Is there any
> > concern if I set MAX_DECS to 1 for production?
>
> Does the time taken to remove the thin device change as you drop it to one?
>
> - Joe
>

Not that I am aware of, but I redo the experiment and the results are
listed below.

#define MAX_DECS 1
Delete a fully-mapped 10TB device without concurrent I/O takes 49 secs.
Delete a fully-mapped 10TB device with concurrent I/O to pool takes 44 secs.

#define MAX_DECS 16
Delete a fully-mapped 10TB device without concurrent I/O takes 47 secs.
Delete a fully-mapped 10TB device with concurrent I/O to pool takes 46 secs.

#define MAX_DECS 8192
Delete a fully-mapped 10TB device without concurrent I/O takes 47 secs.
Delete a fully-mapped 10TB device with concurrent I/O to pool takes 50 secs.

Thanks,
Dennis

-- 
Dennis Yang
QNAP Systems, Inc.
Skype: qnap.dennis.yang
Email: dennisy...@qnap.com
Tel: (+886)-2-2393-5152 ext. 15018
Address: 13F., No.56, Sec. 1, Xinsheng S. Rd., Zhongzheng Dist., Taipei
City, Taiwan
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] I/O block when removing thin device on the same pool

2016-01-29 Thread Joe Thornber 
On Fri, Jan 29, 2016 at 03:50:31PM +0100, Lars Ellenberg wrote:
> On Fri, Jan 22, 2016 at 04:43:46PM +, Joe Thornber wrote:
> > On Fri, Jan 22, 2016 at 02:38:28PM +0100, Lars Ellenberg wrote:
> > > We have seen lvremove of thin snapshots sometimes minutes,
> > > even ~20 minutes before.
> > 
> > I did some work on speeding up thin removal in autumn '14, in
> > particular agressively prefetching metadata pages sped up the tree
> > traversal hugely.  Could you confirm you're seeing pauses of this
> > duration with currently kernels please?
> 
> There is 
> https://bugzilla.redhat.com/show_bug.cgi?id=990583
> Bug 990583 - lvremove of thin snapshots takes 5 to 20 minutes (single
> core cpu bound?) 
> 
> >From August 2013, closed by you in October 2015,
> as "not a bug", also pointing to meta data prefetch.
> 
> Now, you tell me, how prefetching meta data (doing disk IO
> more efficiently) helps with something that is clearly CPU bound
> (eating 100% single core CPU traversing whatever)...
> 
> Reason I mention this bug again here is:
> there should be a lvm thin meta data dump in there,
> which you could use for benchmarking improvements yourself.

There is no metadata dump attached to that bug.  I do benchmark stuff
myself, and I found prefetching to make a big difference (obviously
I'm not cpu bound like you).  We all have different hardware, which is
why I ask people with more real world scenarios to test stuff separately.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] I/O block when removing thin device on the same pool

2016-01-26 Thread Dennis Yang 
Hi Joe,

I have applied this patch to kernel 4.4 and get the following result.

To delete a fully-mapped 10TB thin devices,
with this patch takes 48 sec.
without this patch takes 48 sec.

To read an empty thin device while deleting a fully-mapped 10TB thin
devices,
with this patch I/O throughput drops from 4.6TB/s to 4.3TB/s
without this patch, I/O blocks.

To write an empty thin device while deleting a fully-mapped 10TB thin
devices,
with this patch I/O throughput drops from 3.2TB/s to below 4MB/s
without this patch, I/O blocks

Since it looks like the write performance still suffer from the lock
contention, I make it to sleep 100 msec between lock release and reacquire
in commit_decs().

To write an empty thin device while deleting a fully-mapped 10TB thin
devices
With sleep in commit_decs(), I/O throughput drops from 3.2TB/s to 2.2TB/s,
but the deletion time grows from 48sec to 2m54sec.

The one thing I am curious about is what data structures are dm-thin tries
to protect by holding the pool lock during all those btree operations. At
first, I think the lock is held to protect the btree itself. But based on
the comments in the source code, I believe that it has already been
protected by the read/writes lock in transaction manager
(dm_tm_read/write_lock). Does this mean that the pool lock is held only to
protect the reference count bitmap/btree?

Thanks,
Dennis

2016-01-27 0:19 GMT+08:00 Joe Thornber :

> Hi Dennis,
>
> This is indeed useful.  Is there any chance you could re-run your test
> with this patch applied please?
>
>
> https://github.com/jthornber/linux-2.6/commit/64197a3802320c7a7359ff4a3e592e2bc5bb73dc
>
> - Joe
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>



-- 
Dennis Yang
QNAP Systems, Inc.
Skype: qnap.dennis.yang
Email: dennisy...@qnap.com
Tel: (+886)-2-2393-5152 ext. 15018
Address: 13F., No.56, Sec. 1, Xinsheng S. Rd., Zhongzheng Dist., Taipei
City, Taiwan
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] I/O block when removing thin device on the same pool

2016-01-25 Thread Dennis Yang 
Hi,

I had done some experiments with kernel 4.2.8 that I am using for
production right now and kernel 4.4 with commit 3d5f6733 ("dm thin
metadata: speed up discard of partially mapped volumes") for comparison.

All the experiments below are performed with a dm-thin pool (512KB block
size) which is built with a RAID 0 composed by two Intel 480GB SSDs as
metadata device and a zero-target DM device as the data device. The machine
is equipped with an Intel E3-1246 v3 CPU and 16GB ram.

To discard all the mappings of a fully-mapped 10TB thin device with 512KB
block size,
kernel 4.4 takes 6m57s
kernel 4.2.8 takes 6m49s

To delete a fully-mapped 10TB thin device,
kernel 4.4 takes 48s
kernel 4.2.8 takes 47s

In another experiment, I create an empty thin device and a fully-mapped
10TB thin device. Then, I start writing to the empty thin device
sequentially with fio before deleting the fully-mapped thin device. It can
be observed that the write requests get blocked for couple of seconds
(47~48sec) until the deletion process finishes on both kernel 4.2.8 and
kernel 4.4. If we discard all the mappings in parallel with fio instead of
deleting the fully-mapped thin device, write requests will still be blocked
until all discard requests finished. I think this is because that pool's
deferred list is full of all those discard requests and thus having no
spare computation resource for new write requests to the other thin device.
The kworker thread of thinp cause 100% CPU utilisation while processing the
discard requests.

Hope this information helps.

Thanks,
Dennis


2016-01-23 0:43 GMT+08:00 Joe Thornber :

> On Fri, Jan 22, 2016 at 02:38:28PM +0100, Lars Ellenberg wrote:
> > We have seen lvremove of thin snapshots sometimes minutes,
> > even ~20 minutes before.
>
> I did some work on speeding up thin removal in autumn '14, in
> particular agressively prefetching metadata pages sped up the tree
> traversal hugely.  Could you confirm you're seeing pauses of this
> duration with currently kernels please?
>
> Obviously any pause, even a few seconds is unacceptable.  Having a
> background kernel worker thread doing the delete, as you describe, is
> the way to go.  But there are complications to do with
> transactionality and crash protection that have prevented me
> implementing it.  I'll think on it some more now I know it's such a
> problem for you.
>
> - Joe
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>



-- 
Dennis Yang
QNAP Systems, Inc.
Skype: qnap.dennis.yang
Email: dennisy...@qnap.com
Tel: (+886)-2-2393-5152 ext. 15018
Address: 13F., No.56, Sec. 1, Xinsheng S. Rd., Zhongzheng Dist., Taipei
City, Taiwan
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel