Re: Slow zfs destroy
02.12.2019 20:35, Andriy Gapon wrote: > On 02/12/2019 11:27, Scott Bennett wrote: >> The vast majority of my "destroy" operations are for snapshots, but what >> I have seen is that, without the "-d", the command does not return until the >> disk activity of the "destroy" finishes, but with the "-d", it returns within >> a couple of seconds,--i.e., just long enough to get the operation going--and >> the disk I/Os continue until the work is done and free space in the pool >> increases >> until the I/Os stop. >> Perhaps the man pages for zfs(8) and zpool-features(7) need some >> modification/ >> clarification on this matter. > > I don't know how to explain what you see, but the manual pages are correct. > -d has nothing to do with asynchronous destroy which is automatic (when > enabled). It seems that "zfs destroy" writes and/or removes large enough amount of data to spend lots of time before return when TRIM performance is bad. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
Eugene Grosbein wrote: > 30.11.2019 0:57, Scott Bennett wrote: > > > On Thu, 28 Nov 2019 23:18:37 +0700 Eugene Grosbein > > wrote: > > > >> 28.11.2019 20:34, Steven Hartland wrote: > >> > >>> It may well depend on the extent of the deletes occurring. > >>> > >>> Have you tried disabling TRIM to see if it eliminates the delay? > >> > >> This system used mfi(4) first and mfi(4) does not support TRIM at all. > >> Performance was abysmal. > >> Now it uses mrsas(4) and after switch I ran trim(8) for all SSDs > >> one-by-one then re-added them to RAID1. > >> Disabling TRIM is not an option. > >> > >> Almost a year has passed since then and I suspect SSDs have no or a few > >> spare trimmed cells for some reason. > >> Is there documented way to check this out? Maybe some SMART attribute? > >> > > You neglected to state whether you used "zfs destroy datasetname" or > > "zfs destroy -d datasetname". If you used the former, then ZFS did what > > you told it to do. If you want the data set destroyed in the background, > > you will need to include the "-d" option in the command. (See the zfs(1) > > man page at defer_destroy under "Native Properties".) > > The manual says "zfs destroy -d" is not for "background" but for "deferred". > The "zfs destroy" without -d would return EBUSY for a snapshot on hold (zfs > hold) > or bound with a clone, but "zfs destroy -d" would mark the snapshot for later > destruction > in a moment the clone is deleted or user lock (hold) is lifted. > Until then the snapshot still usable and destruction does not happen. > > All my snapshots are free from holds or clones and can be deleted, > so "zfs destroy -d" is equal to "zfs destroy" for them. > What you say is true, and I have seen it accept a "zfs destroy -d" for a held snapshot but do nothing until the hold is released, whereupon the "destroy" begins. However, that cannot be the whole story because... The vast majority of my "destroy" operations are for snapshots, but what I have seen is that, without the "-d", the command does not return until the disk activity of the "destroy" finishes, but with the "-d", it returns within a couple of seconds,--i.e., just long enough to get the operation going--and the disk I/Os continue until the work is done and free space in the pool increases until the I/Os stop. Perhaps the man pages for zfs(8) and zpool-features(7) need some modification/ clarification on this matter. Scott Bennett, Comm. ASMELG, CFIAG ** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * ** * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * *-- Gov. John Hancock, New York Journal, 28 January 1790 * ** ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
30.11.2019 0:57, Scott Bennett wrote: > On Thu, 28 Nov 2019 23:18:37 +0700 Eugene Grosbein > wrote: > >> 28.11.2019 20:34, Steven Hartland wrote: >> >>> It may well depend on the extent of the deletes occurring. >>> >>> Have you tried disabling TRIM to see if it eliminates the delay? >> >> This system used mfi(4) first and mfi(4) does not support TRIM at all. >> Performance was abysmal. >> Now it uses mrsas(4) and after switch I ran trim(8) for all SSDs one-by-one >> then re-added them to RAID1. >> Disabling TRIM is not an option. >> >> Almost a year has passed since then and I suspect SSDs have no or a few >> spare trimmed cells for some reason. >> Is there documented way to check this out? Maybe some SMART attribute? >> > You neglected to state whether you used "zfs destroy datasetname" or > "zfs destroy -d datasetname". If you used the former, then ZFS did what > you told it to do. If you want the data set destroyed in the background, > you will need to include the "-d" option in the command. (See the zfs(1) > man page at defer_destroy under "Native Properties".) The manual says "zfs destroy -d" is not for "background" but for "deferred". The "zfs destroy" without -d would return EBUSY for a snapshot on hold (zfs hold) or bound with a clone, but "zfs destroy -d" would mark the snapshot for later destruction in a moment the clone is deleted or user lock (hold) is lifted. Until then the snapshot still usable and destruction does not happen. All my snapshots are free from holds or clones and can be deleted, so "zfs destroy -d" is equal to "zfs destroy" for them. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
On Thu, 28 Nov 2019 23:18:37 +0700 Eugene Grosbein wrote: >28.11.2019 20:34, Steven Hartland wrote: > >> It may well depend on the extent of the deletes occurring. >> >> Have you tried disabling TRIM to see if it eliminates the delay? > >This system used mfi(4) first and mfi(4) does not support TRIM at all. >Performance was abysmal. >Now it uses mrsas(4) and after switch I ran trim(8) for all SSDs one-by-one >then re-added them to RAID1. >Disabling TRIM is not an option. > >Almost a year has passed since then and I suspect SSDs have no or a few spare >trimmed cells for some reason. >Is there documented way to check this out? Maybe some SMART attribute? > You neglected to state whether you used "zfs destroy datasetname" or "zfs destroy -d datasetname". If you used the former, then ZFS did what you told it to do. If you want the data set destroyed in the background, you will need to include the "-d" option in the command. (See the zfs(1) man page at defer_destroy under "Native Properties".) Scott Bennett, Comm. ASMELG, CFIAG ** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * ** * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * *-- Gov. John Hancock, New York Journal, 28 January 1790 * ** ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
28.11.2019 20:34, Steven Hartland wrote: > It may well depend on the extent of the deletes occurring. > > Have you tried disabling TRIM to see if it eliminates the delay? This system used mfi(4) first and mfi(4) does not support TRIM at all. Performance was abysmal. Now it uses mrsas(4) and after switch I ran trim(8) for all SSDs one-by-one then re-added them to RAID1. Disabling TRIM is not an option. Almost a year has passed since then and I suspect SSDs have no or a few spare trimmed cells for some reason. Is there documented way to check this out? Maybe some SMART attribute? ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
It may well depend on the extent of the deletes occurring. Have you tried disabling TRIM to see if it eliminates the delay? Regards Steve On 28/11/2019 09:59, Eugene Grosbein wrote: 28.11.2019 14:26, Steven Hartland wrote: As you mentioned it’s on SSD you could be suffering from poor TRIM performance from your devices if you run gstat -pd you’ll be able to get an indication if this is the case. Yes, this box does have problems with poor TRIM performance. But isn't "zfs destroy" supposed to perform actual removal in background? "feature@async_destroy" is "enabled" for this pool. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
28.11.2019 14:26, Steven Hartland wrote: > As you mentioned it’s on SSD you could be suffering from poor TRIM > performance from your devices > if you run gstat -pd you’ll be able to get an indication if this is the case. Yes, this box does have problems with poor TRIM performance. But isn't "zfs destroy" supposed to perform actual removal in background? "feature@async_destroy" is "enabled" for this pool. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
28.11.2019 16:38, Pete French wrote: > On 28/Nov/2019 07:03, Eugene Grosbein wrote: >> 28.11.2019 13:46, Eugene Grosbein wrote: >> >>> Hi! >>> >>> Is it normal that "zfs destroy" for one ZVOL with attribute "used" equal to >>> 2112939808 bytes (~2GB) >>> takes over two minutes waiting on "tx_sync_done_cv"? The pool is RAID1 over >>> five SSDs encrypted with GELI >> >> The pool is *RAIDZ1* > > Not normal, no. Having said that I have seen zfs destory take a very lng time > on occasion, > but for a 2GB volume then thats surprisng. Is there actual disc activity > during this time ? Yes, a lot. Several rsync instances writing to another ZFS (mounted file systems) in parallel: copying data over ssh from remote hosts (backups). ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
On 28/Nov/2019 07:03, Eugene Grosbein wrote: 28.11.2019 13:46, Eugene Grosbein wrote: Hi! Is it normal that "zfs destroy" for one ZVOL with attribute "used" equal to 2112939808 bytes (~2GB) takes over two minutes waiting on "tx_sync_done_cv"? The pool is RAID1 over five SSDs encrypted with GELI The pool is *RAIDZ1* Not normal, no. Having said that I have seen zfs destory take a very lng time on occasion, but for a 2GB volume then thats surprisng. Is there actual disc activity during this time ? I've been using 12 a while now, and I don't recall seeing this recently, so maybe its soemthing which has been improved on over 11. -pete. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
As you mentioned it’s on SSD you could be suffering from poor TRIM performance from your devices if you run gstat -pd you’ll be able to get an indication if this is the case. On Thu, 28 Nov 2019 at 06:50, Eugene Grosbein wrote: > Hi! > > Is it normal that "zfs destroy" for one ZVOL with attribute "used" equal > to 2112939808 bytes (~2GB) > takes over two minutes waiting on "tx_sync_done_cv"? The pool is RAID1 > over five SSDs encrypted with GELI > having ZIL and Cache on distinct unencrypted SSD. > > 11.3-STABLE/amd64 r354667. System has 360G RAM and vfs.zfs.arc_max=160g. > ___ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" > ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Slow zfs destroy
28.11.2019 13:46, Eugene Grosbein wrote: > Hi! > > Is it normal that "zfs destroy" for one ZVOL with attribute "used" equal to > 2112939808 bytes (~2GB) > takes over two minutes waiting on "tx_sync_done_cv"? The pool is RAID1 over > five SSDs encrypted with GELI The pool is *RAIDZ1* > having ZIL and Cache on distinct unencrypted SSD. > > 11.3-STABLE/amd64 r354667. System has 360G RAM and vfs.zfs.arc_max=160g. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Slow zfs destroy
Hi! Is it normal that "zfs destroy" for one ZVOL with attribute "used" equal to 2112939808 bytes (~2GB) takes over two minutes waiting on "tx_sync_done_cv"? The pool is RAID1 over five SSDs encrypted with GELI having ZIL and Cache on distinct unencrypted SSD. 11.3-STABLE/amd64 r354667. System has 360G RAM and vfs.zfs.arc_max=160g. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
