I was freeing about 1TB per day before. I kept adding more time and
checking with 'zpool get -p freeing hcpdr03;sleep 1m;zpool get -p freeing
hcpdr03' to see if it was changing.
This is OmniOSce 030 so no extended iostat:
# zpool iostat -yl 5
invalid option 'y'
usage:
iostat [-v] [-T d|u] [pool] ... [interval [count]]
On Wed, Jun 3, 2020 at 7:09 AM Vitaliy Gusev <[email protected]>
wrote:
> Chip,
>
> On 3 Jun 2020, at 14:07, Schweiss, Chip <[email protected]> wrote:
>
> I've continued to move zfs_free_min_time_ms exponentially larger with
> some progress. At 1,000,000 its is making some progress on this large
> delete queue.
>
>
> I assume setting zfs_free_min_time_ms more than (zfs_txg_timeout * 1000)
> should
> no have effects.
>
>
> # zpool get freeing hcpdr03
> NAME PROPERTY VALUE SOURCE
> hcpdr03 freeing 103T default
>
>
> Previously you had 125T, so 22TB is freed during one day - good progress.
> Is resilvering completed ?
>
>
> The I/O load is still relatively low on the pool
>
>
> Does "zpool iostat -yl 5” show large values ? And what HDDs do you have
> (IOPs, latency, etc) ?
>
> ———
> Vitaliy Gusev
>
>
> At least it will now complete before Christmas.
>
> -Chip
>
> On Tue, Jun 2, 2020 at 8:58 AM Vitaliy Gusev <[email protected]>
> wrote:
>
>> Don’t you think that freeing can be hampered by random intensive I/O of
>> HDDs ? If you have resilvering at the same time during large freeing, they
>> can effect to each other.
>>
>>
>> 1 scanned out of 639T at 1/s, (scan is slow, no estimated time)
>>
>>
>> So resilvering is also stuck ?
>>
>> # iostat -xn |head
>> extended device statistics
>> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
>> 24.6 8910.7 115.4 56272.5 10.9 13.2 1.2 1.5 9 40 hcpdr01
>> 28.9 1823.5 125.7 14127.5 3.0 2.1 1.6 1.1 4 34 hcpdr02
>> 160.1 2279.3 687.9 21067.8 8.7 8.3 3.5 3.4 3 22 hcpdr03
>>
>>
>> Could you find most busy %b and “wait”, %w and others ? Does it have
>> high values ?
>>
>> Also it would be helpful to look at output:
>>
>> "zpool iostat -vyl $pool 10”
>>
>> and
>>
>> "zpool iostat -vyq $pool 10”
>>
>>
>> ———
>> Vitaliy Gusev
>>
>>
>>
>>
>> On 2 Jun 2020, at 14:14, Schweiss, Chip <[email protected]> wrote:
>>
>> Vitaliy,
>>
>> Thanks for the details. I wasn't aware of the 'freeing' property. That
>> is very useful to see progress.
>>
>> There's plenty of space on the pool both now and when the delete
>> started. No checkpoint, no dedup. This is a raidz3 pool of 90 12TB disks.
>>
>> I've been bumping zfs_free_min_time_ms but it has only has minor
>> influence. It currently set to 100000. Should I keep bumping this by
>> orders of magnitude? I'd rather hobble the pool temporarily to work
>> through this crippling problem.
>>
>> # zpool list hcpdr03
>> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP
>> HEALTH ALTROOT
>> hcpdr03 1.02P 631T 416T - - 10% 60% 1.00x
>> DEGRADED -
>>
>> # zpool get freeing hcpdr03
>> NAME PROPERTY VALUE SOURCE
>> hcpdr03 freeing 125T default
>>
>> # zpool status hcpdr03|head
>> pool: hcpdr03
>> state: DEGRADED
>> status: One or more devices is currently being resilvered. The pool will
>> continue to function, possibly in a degraded state.
>> action: Wait for the resilver to complete.
>> scan: resilver in progress since Mon Jun 1 21:52:38 2020
>> 1 scanned out of 639T at 1/s, (scan is slow, no estimated time)
>> 0 resilvered, 0.00% done
>>
>> It dropped a disk about two weeks ago and progress is almost
>> non-existant. It was rebooted yesterday. It was about 5% complete before
>> the reboot. Previously, this pool would resilver in 5-7 days.
>>
>> I/O is relatively low for this pool:
>> # zpool iostat hcpdr03
>> capacity operations bandwidth
>> pool alloc free read write read write
>> ---------- ----- ----- ----- ----- ----- -----
>> hcpdr03 631T 416T 118 553 507K 10.2M
>>
>> # iostat -xn |head
>> extended device statistics
>> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
>> 24.6 8910.7 115.4 56272.5 10.9 13.2 1.2 1.5 9 40 hcpdr01
>> 28.9 1823.5 125.7 14127.5 3.0 2.1 1.6 1.1 4 34 hcpdr02
>> 160.1 2279.3 687.9 21067.8 8.7 8.3 3.5 3.4 3 22 hcpdr03
>>
>> -Chip
>>
>>
>>
>>
>>
>>
>> On Mon, Jun 1, 2020 at 11:03 PM Vitaliy Gusev <[email protected]>
>> wrote:
>>
>>> 1. Can you play with zfs_free_min_time_ms ? Default value is 1/5 of the
>>> txg sync time (zfs_txg_timeout).
>>>
>>> unsigned int zfs_free_min_time_ms = 1000; /* min millisecs to free per
>>> txg */
>>>
>>> Also It could be that reading metadata for freeing is slow (due to ARC
>>> constraints or heavy I/O or fragmented pool on HDD) and this also could
>>> lead to side effect then metadata cannot be read effectively enough to be
>>> ready within zfs_txg_timeout seconds and blocks’ freeing is postponed to
>>> the next spa-sync. Look at dsl_scan_async_block_should_pause() for
>>> details.
>>>
>>> 2. Don't you have set checkpoint on the pool ? It can break reclaiming
>>> if there was no enough space, look at spa_suspend_async_destroy() for more
>>> details.
>>>
>>> 3. Don’t you have enabled dedup ? Data blocks can be referenced in this
>>> case and will not be freed.
>>>
>>> BTW, Do you see "zpool get freeing $pool” shows 150TB ?
>>>
>>> ———
>>> Vitaliy Gusev
>>>
>>>
------------------------------------------
illumos: illumos-discuss
Permalink:
https://illumos.topicbox.com/groups/discuss/T51c43cca03b19c45-M063c88f59606196209fc63c5
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
