Re: another crash and going forward with zfs
On 4/18/23, Pawel Jakub Dawidek wrote: > On 4/18/23 05:14, Mateusz Guzik wrote: >> On 4/17/23, Pawel Jakub Dawidek wrote: >>> Correct me if I'm wrong, but from my understanding there were zero >>> problems with block cloning when it wasn't in use or now disabled. >>> >>> The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to exactly >>> avoid mess like this and give us more time to sort all the problems out >>> while making it easy for people to try it. >>> >>> If there is no plan to revert the whole import, I don't see what value >>> removing just block cloning will bring if it is now disabled by default >>> and didn't cause any problems when disabled. >>> >> >> The feature definitely was not properly stress tested and what not and >> trying to do it keeps running into panics. Given the complexity of the >> feature I would expect there are many bug lurking, some of which >> possibly related to the on disk format. Not having to deal with any of >> this is can be arranged as described above and is imo the most >> sensible route given the timeline for 14.0 > > Block cloning doesn't create, remove or modify any on-disk data until it > is in use. > > Again, if we are not going to revert the whole merge, I see no point in > reverting block cloning as until it is enabled, its code is not > executed. This allow people who upgraded the pools to do nothing special > and it will allow people to test it easily. > Some people will zpool upgrade out of habit or whatever after moving to 14.0, which will then make them unable to go back to 13.x if woes show up. Woes don't even have to be zfs-related. This is a major release, one has to suspect there will be some breakage and it maybe the best way forward for some of the users will be to downgrade (e.g., with boot envinronments). As is they wont be able to do it if they zpool upgrade. If someone *does* zpool upgrade and there is further data corruption due to block cloning (which you really can't rule out given that the feature so far did not survive under load), telephone game is going to turn this into "14.0 corrupts data" and no amount of clarifying about an optional feature is going to help the press. If anything the real question is how come the feature got merged upstream, when: 1. FreeBSD CI for the project is offline 2. There is no Linux support 3. ... basic usage showed numerous bugs Should the feature get whipped into shape, it can be a 14.1 candidate. -- Mateusz Guzik
Re: another crash and going forward with zfs
> On 18 Apr 2023, at 09:46, Martin Matuska wrote: > > Btw. I am open for setting up a pre-merge stress testing > > I will check out if I can use the hourly-billed amd64 and arm64 cloud boxes > at Hetzner with FreeBSD. > Otherwise there are monthly-billed as well. I can provide a bhyve VM on some of my hosts. We can discuss it off-list. jl — Juraj Lutter o...@freebsd.org
Re: another crash and going forward with zfs
Btw. I am open for setting up a pre-merge stress testing I will check out if I can use the hourly-billed amd64 and arm64 cloud boxes at Hetzner with FreeBSD. Otherwise there are monthly-billed as well. Cheers, mm On 17. 4. 2023 22:14, Mateusz Guzik wrote: On 4/17/23, Pawel Jakub Dawidek wrote: On 4/18/23 03:51, Mateusz Guzik wrote: After bugfixes got committed I decided to zpool upgrade and sysctl vfs.zfs.bclone_enabled=1 vs poudriere for testing purposes. I very quickly got a new crash: panic: VERIFY(arc_released(db->db_buf)) failed cpuid = 9 time = 1681755046 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe0a90b8e5f0 vpanic() at vpanic+0x152/frame 0xfe0a90b8e640 spl_panic() at spl_panic+0x3a/frame 0xfe0a90b8e6a0 dbuf_redirty() at dbuf_redirty+0xbd/frame 0xfe0a90b8e6c0 dmu_buf_will_dirty_impl() at dmu_buf_will_dirty_impl+0xa2/frame 0xfe0a90b8e700 dmu_write_uio_dnode() at dmu_write_uio_dnode+0xe9/frame 0xfe0a90b8e780 dmu_write_uio_dbuf() at dmu_write_uio_dbuf+0x42/frame 0xfe0a90b8e7b0 zfs_write() at zfs_write+0x672/frame 0xfe0a90b8e960 zfs_freebsd_write() at zfs_freebsd_write+0x39/frame 0xfe0a90b8e980 VOP_WRITE_APV() at VOP_WRITE_APV+0xdb/frame 0xfe0a90b8ea90 vn_write() at vn_write+0x325/frame 0xfe0a90b8eb20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfe0a90b8eb80 vn_io_fault1() at vn_io_fault1+0x161/frame 0xfe0a90b8ecc0 vn_io_fault() at vn_io_fault+0x1b5/frame 0xfe0a90b8ed40 dofilewrite() at dofilewrite+0x81/frame 0xfe0a90b8ed90 sys_write() at sys_write+0xc0/frame 0xfe0a90b8ee00 amd64_syscall() at amd64_syscall+0x157/frame 0xfe0a90b8ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfe0a90b8ef30 --- syscall (4, FreeBSD ELF64, write), rip = 0x103cddf7949a, rsp = 0x103cdc85dd48, rbp = 0x103cdc85dd80 --- KDB: enter: panic [ thread pid 95000 tid 135035 ] Stopped at kdb_enter+0x32: movq$0,0x9e4153(%rip) The posted 14.0 schedule which plans to branch stable/14 on May 12 and one cannot bet on the feature getting beaten up into production shape by that time. Given whatever non-block_clonning and not even zfs bugs which are likely to come out I think this makes the feature a non-starter for said release. I note: 1. the current problems did not make it into stable branches. 2. there was block_cloning-related data corruption (fixed) and there may be more 3. there was unrelated data corruption (see https://github.com/openzfs/zfs/issues/14753), sorted out by reverting the problematic commit in FreeBSD, not yet sorted out upstream As such people's data may be partially hosed as is. Consequently the proposed plan is as follows: 1. whack the block cloning feature for the time being, but make sure pools which upgraded to it can be mounted read-only 2. run ztest and whatever other stress testing on FreeBSD, along with restoring openzfs CI -- I can do the first part, I'm sure pho will not mind to run some tests of his own 3. recommend people create new pools and restore data from backup. if restoring from backup is not an option, tar or cp (not zfs send) from the read-only mount block cloning beaten into shape would use block_cloning_v2 or whatever else, key point that the current feature name would be considered bogus (not blocking RO import though) to prevent RW usage of the current pools with it enabled. Comments? Correct me if I'm wrong, but from my understanding there were zero problems with block cloning when it wasn't in use or now disabled. The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to exactly avoid mess like this and give us more time to sort all the problems out while making it easy for people to try it. If there is no plan to revert the whole import, I don't see what value removing just block cloning will bring if it is now disabled by default and didn't cause any problems when disabled. The feature definitely was not properly stress tested and what not and trying to do it keeps running into panics. Given the complexity of the feature I would expect there are many bug lurking, some of which possibly related to the on disk format. Not having to deal with any of this is can be arranged as described above and is imo the most sensible route given the timeline for 14.0
Re: another crash and going forward with zfs
On 18. 4. 2023 3:16, Warner Losh wrote: Related question: what zfs branch is stable/14 going to track? With 13 it was whatever the next stable branch was. Warner FreeBSD 14.0 is about to track soon-to-be-branched OpenZFS 2.2
Re: another crash and going forward with zfs
On Mon, Apr 17, 2023, 5:37 PM Rick Macklem wrote: > On Mon, Apr 17, 2023 at 4:29 PM Cy Schubert > wrote: > > > > In message , Pawel > Jakub > > Dawi > > dek writes: > > > On 4/18/23 05:14, Mateusz Guzik wrote: > > > > On 4/17/23, Pawel Jakub Dawidek wrote: > > > >> Correct me if I'm wrong, but from my understanding there were zero > > > >> problems with block cloning when it wasn't in use or now disabled. > > > >> > > > >> The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to > exactly > > > >> avoid mess like this and give us more time to sort all the problems > out > > > >> while making it easy for people to try it. > > > >> > > > >> If there is no plan to revert the whole import, I don't see what > value > > > >> removing just block cloning will bring if it is now disabled by > default > > > >> and didn't cause any problems when disabled. > > > >> > > > > > > > > The feature definitely was not properly stress tested and what not > and > > > > trying to do it keeps running into panics. Given the complexity of > the > > > > feature I would expect there are many bug lurking, some of which > > > > possibly related to the on disk format. Not having to deal with any > of > > > > this is can be arranged as described above and is imo the most > > > > sensible route given the timeline for 14.0 > > > > > > Block cloning doesn't create, remove or modify any on-disk data until > it > > > is in use. > > > > > > Again, if we are not going to revert the whole merge, I see no point in > > > reverting block cloning as until it is enabled, its code is not > > > executed. This allow people who upgraded the pools to do nothing > special > > > and it will allow people to test it easily. > > > > In this case zpool upgrade and zpool status should return no feature > > upgrades are available instead of enticing users to zpool upgrade. The > > userland zpool command should test for this sysctl and print nothing > > regarding block_cloning. I can see a scenario when a user zpool upgrades > > their pools, notices the sysctl and does the unthinkable. Not only would > > this fill the mailing lists with angry chatter but it would spawn a > number > > of PRs plus give us a lot of bad press for data loss. > > > > Should we keep the new ZFS in 14, we should: > > > > 1. Make sure that zpool(8) does not mention or offer block_cloning in any > > way if the sysctl is disabled. > > > > 2. Print a cautionary note in release notes advising people not to enable > > this experimental sysctl. Maybe even have it print "(experimental)" to > warn > > users that it will hurt. > > > > 3. Update the man pages to caution that block_cloning is experimental and > > unstable. > I would suggest going a step further and making the sysctl RO for > FreeBSD14. > (This could be changed for FreeBSD14.n if/when block_cloning is believed to > be debugged.) > > I would apply all 3 of the above to "main", since some that install "main" > will not know how "bleeding edge" this is unless the above is done. > (Yes, I know "main" is "bleeding edge", but some still expect a stable > test system will result from installing it.) > > Thanks go to all that tracked this problem down, rick > Related question: what zfs branch is stable/14 going to track? With 13 it was whatever the next stable branch was. Warner > > > It's not enough to have a sysctl without hiding block_cloning completely > > from view. Only expose it in zpool(8) when the sysctl is enabled. Let's > > avoid people mistakenly enabling it. > > > > > > -- > > Cheers, > > Cy Schubert > > FreeBSD UNIX: Web: https://FreeBSD.org > > NTP: Web: https://nwtime.org > > > > e^(i*pi)+1=0 > > > > > > > >
Re: another crash and going forward with zfs
On Mon, Apr 17, 2023 at 4:29 PM Cy Schubert wrote: > > In message , Pawel Jakub > Dawi > dek writes: > > On 4/18/23 05:14, Mateusz Guzik wrote: > > > On 4/17/23, Pawel Jakub Dawidek wrote: > > >> Correct me if I'm wrong, but from my understanding there were zero > > >> problems with block cloning when it wasn't in use or now disabled. > > >> > > >> The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to exactly > > >> avoid mess like this and give us more time to sort all the problems out > > >> while making it easy for people to try it. > > >> > > >> If there is no plan to revert the whole import, I don't see what value > > >> removing just block cloning will bring if it is now disabled by default > > >> and didn't cause any problems when disabled. > > >> > > > > > > The feature definitely was not properly stress tested and what not and > > > trying to do it keeps running into panics. Given the complexity of the > > > feature I would expect there are many bug lurking, some of which > > > possibly related to the on disk format. Not having to deal with any of > > > this is can be arranged as described above and is imo the most > > > sensible route given the timeline for 14.0 > > > > Block cloning doesn't create, remove or modify any on-disk data until it > > is in use. > > > > Again, if we are not going to revert the whole merge, I see no point in > > reverting block cloning as until it is enabled, its code is not > > executed. This allow people who upgraded the pools to do nothing special > > and it will allow people to test it easily. > > In this case zpool upgrade and zpool status should return no feature > upgrades are available instead of enticing users to zpool upgrade. The > userland zpool command should test for this sysctl and print nothing > regarding block_cloning. I can see a scenario when a user zpool upgrades > their pools, notices the sysctl and does the unthinkable. Not only would > this fill the mailing lists with angry chatter but it would spawn a number > of PRs plus give us a lot of bad press for data loss. > > Should we keep the new ZFS in 14, we should: > > 1. Make sure that zpool(8) does not mention or offer block_cloning in any > way if the sysctl is disabled. > > 2. Print a cautionary note in release notes advising people not to enable > this experimental sysctl. Maybe even have it print "(experimental)" to warn > users that it will hurt. > > 3. Update the man pages to caution that block_cloning is experimental and > unstable. I would suggest going a step further and making the sysctl RO for FreeBSD14. (This could be changed for FreeBSD14.n if/when block_cloning is believed to be debugged.) I would apply all 3 of the above to "main", since some that install "main" will not know how "bleeding edge" this is unless the above is done. (Yes, I know "main" is "bleeding edge", but some still expect a stable test system will result from installing it.) Thanks go to all that tracked this problem down, rick > > It's not enough to have a sysctl without hiding block_cloning completely > from view. Only expose it in zpool(8) when the sysctl is enabled. Let's > avoid people mistakenly enabling it. > > > -- > Cheers, > Cy Schubert > FreeBSD UNIX: Web: https://FreeBSD.org > NTP: Web: https://nwtime.org > > e^(i*pi)+1=0 > > >
Re: another crash and going forward with zfs
In message , Pawel Jakub Dawi dek writes: > On 4/18/23 05:14, Mateusz Guzik wrote: > > On 4/17/23, Pawel Jakub Dawidek wrote: > >> Correct me if I'm wrong, but from my understanding there were zero > >> problems with block cloning when it wasn't in use or now disabled. > >> > >> The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to exactly > >> avoid mess like this and give us more time to sort all the problems out > >> while making it easy for people to try it. > >> > >> If there is no plan to revert the whole import, I don't see what value > >> removing just block cloning will bring if it is now disabled by default > >> and didn't cause any problems when disabled. > >> > > > > The feature definitely was not properly stress tested and what not and > > trying to do it keeps running into panics. Given the complexity of the > > feature I would expect there are many bug lurking, some of which > > possibly related to the on disk format. Not having to deal with any of > > this is can be arranged as described above and is imo the most > > sensible route given the timeline for 14.0 > > Block cloning doesn't create, remove or modify any on-disk data until it > is in use. > > Again, if we are not going to revert the whole merge, I see no point in > reverting block cloning as until it is enabled, its code is not > executed. This allow people who upgraded the pools to do nothing special > and it will allow people to test it easily. In this case zpool upgrade and zpool status should return no feature upgrades are available instead of enticing users to zpool upgrade. The userland zpool command should test for this sysctl and print nothing regarding block_cloning. I can see a scenario when a user zpool upgrades their pools, notices the sysctl and does the unthinkable. Not only would this fill the mailing lists with angry chatter but it would spawn a number of PRs plus give us a lot of bad press for data loss. Should we keep the new ZFS in 14, we should: 1. Make sure that zpool(8) does not mention or offer block_cloning in any way if the sysctl is disabled. 2. Print a cautionary note in release notes advising people not to enable this experimental sysctl. Maybe even have it print "(experimental)" to warn users that it will hurt. 3. Update the man pages to caution that block_cloning is experimental and unstable. It's not enough to have a sysctl without hiding block_cloning completely from view. Only expose it in zpool(8) when the sysctl is enabled. Let's avoid people mistakenly enabling it. -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0
Re: another crash and going forward with zfs
On 4/18/23 05:14, Mateusz Guzik wrote: On 4/17/23, Pawel Jakub Dawidek wrote: Correct me if I'm wrong, but from my understanding there were zero problems with block cloning when it wasn't in use or now disabled. The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to exactly avoid mess like this and give us more time to sort all the problems out while making it easy for people to try it. If there is no plan to revert the whole import, I don't see what value removing just block cloning will bring if it is now disabled by default and didn't cause any problems when disabled. The feature definitely was not properly stress tested and what not and trying to do it keeps running into panics. Given the complexity of the feature I would expect there are many bug lurking, some of which possibly related to the on disk format. Not having to deal with any of this is can be arranged as described above and is imo the most sensible route given the timeline for 14.0 Block cloning doesn't create, remove or modify any on-disk data until it is in use. Again, if we are not going to revert the whole merge, I see no point in reverting block cloning as until it is enabled, its code is not executed. This allow people who upgraded the pools to do nothing special and it will allow people to test it easily. -- Pawel Jakub Dawidek
Re: another crash and going forward with zfs
On 4/17/23, Pawel Jakub Dawidek wrote: > On 4/18/23 03:51, Mateusz Guzik wrote: >> After bugfixes got committed I decided to zpool upgrade and sysctl >> vfs.zfs.bclone_enabled=1 vs poudriere for testing purposes. I very >> quickly got a new crash: >> >> panic: VERIFY(arc_released(db->db_buf)) failed >> >> cpuid = 9 >> time = 1681755046 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfe0a90b8e5f0 >> vpanic() at vpanic+0x152/frame 0xfe0a90b8e640 >> spl_panic() at spl_panic+0x3a/frame 0xfe0a90b8e6a0 >> dbuf_redirty() at dbuf_redirty+0xbd/frame 0xfe0a90b8e6c0 >> dmu_buf_will_dirty_impl() at dmu_buf_will_dirty_impl+0xa2/frame >> 0xfe0a90b8e700 >> dmu_write_uio_dnode() at dmu_write_uio_dnode+0xe9/frame >> 0xfe0a90b8e780 >> dmu_write_uio_dbuf() at dmu_write_uio_dbuf+0x42/frame 0xfe0a90b8e7b0 >> zfs_write() at zfs_write+0x672/frame 0xfe0a90b8e960 >> zfs_freebsd_write() at zfs_freebsd_write+0x39/frame 0xfe0a90b8e980 >> VOP_WRITE_APV() at VOP_WRITE_APV+0xdb/frame 0xfe0a90b8ea90 >> vn_write() at vn_write+0x325/frame 0xfe0a90b8eb20 >> vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfe0a90b8eb80 >> vn_io_fault1() at vn_io_fault1+0x161/frame 0xfe0a90b8ecc0 >> vn_io_fault() at vn_io_fault+0x1b5/frame 0xfe0a90b8ed40 >> dofilewrite() at dofilewrite+0x81/frame 0xfe0a90b8ed90 >> sys_write() at sys_write+0xc0/frame 0xfe0a90b8ee00 >> amd64_syscall() at amd64_syscall+0x157/frame 0xfe0a90b8ef30 >> fast_syscall_common() at fast_syscall_common+0xf8/frame >> 0xfe0a90b8ef30 >> --- syscall (4, FreeBSD ELF64, write), rip = 0x103cddf7949a, rsp = >> 0x103cdc85dd48, rbp = 0x103cdc85dd80 --- >> KDB: enter: panic >> [ thread pid 95000 tid 135035 ] >> Stopped at kdb_enter+0x32: movq$0,0x9e4153(%rip) >> >> The posted 14.0 schedule which plans to branch stable/14 on May 12 and >> one cannot bet on the feature getting beaten up into production shape >> by that time. Given whatever non-block_clonning and not even zfs bugs >> which are likely to come out I think this makes the feature a >> non-starter for said release. >> >> I note: >> 1. the current problems did not make it into stable branches. >> 2. there was block_cloning-related data corruption (fixed) and there may >> be more >> 3. there was unrelated data corruption (see >> https://github.com/openzfs/zfs/issues/14753), sorted out by reverting >> the problematic commit in FreeBSD, not yet sorted out upstream >> >> As such people's data may be partially hosed as is. >> >> Consequently the proposed plan is as follows: >> 1. whack the block cloning feature for the time being, but make sure >> pools which upgraded to it can be mounted read-only >> 2. run ztest and whatever other stress testing on FreeBSD, along with >> restoring openzfs CI -- I can do the first part, I'm sure pho will not >> mind to run some tests of his own >> 3. recommend people create new pools and restore data from backup. if >> restoring from backup is not an option, tar or cp (not zfs send) from >> the read-only mount >> >> block cloning beaten into shape would use block_cloning_v2 or whatever >> else, key point that the current feature name would be considered >> bogus (not blocking RO import though) to prevent RW usage of the >> current pools with it enabled. >> >> Comments? > > Correct me if I'm wrong, but from my understanding there were zero > problems with block cloning when it wasn't in use or now disabled. > > The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to exactly > avoid mess like this and give us more time to sort all the problems out > while making it easy for people to try it. > > If there is no plan to revert the whole import, I don't see what value > removing just block cloning will bring if it is now disabled by default > and didn't cause any problems when disabled. > The feature definitely was not properly stress tested and what not and trying to do it keeps running into panics. Given the complexity of the feature I would expect there are many bug lurking, some of which possibly related to the on disk format. Not having to deal with any of this is can be arranged as described above and is imo the most sensible route given the timeline for 14.0 -- Mateusz Guzik
Re: another crash and going forward with zfs
On 4/18/23 03:51, Mateusz Guzik wrote: After bugfixes got committed I decided to zpool upgrade and sysctl vfs.zfs.bclone_enabled=1 vs poudriere for testing purposes. I very quickly got a new crash: panic: VERIFY(arc_released(db->db_buf)) failed cpuid = 9 time = 1681755046 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe0a90b8e5f0 vpanic() at vpanic+0x152/frame 0xfe0a90b8e640 spl_panic() at spl_panic+0x3a/frame 0xfe0a90b8e6a0 dbuf_redirty() at dbuf_redirty+0xbd/frame 0xfe0a90b8e6c0 dmu_buf_will_dirty_impl() at dmu_buf_will_dirty_impl+0xa2/frame 0xfe0a90b8e700 dmu_write_uio_dnode() at dmu_write_uio_dnode+0xe9/frame 0xfe0a90b8e780 dmu_write_uio_dbuf() at dmu_write_uio_dbuf+0x42/frame 0xfe0a90b8e7b0 zfs_write() at zfs_write+0x672/frame 0xfe0a90b8e960 zfs_freebsd_write() at zfs_freebsd_write+0x39/frame 0xfe0a90b8e980 VOP_WRITE_APV() at VOP_WRITE_APV+0xdb/frame 0xfe0a90b8ea90 vn_write() at vn_write+0x325/frame 0xfe0a90b8eb20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfe0a90b8eb80 vn_io_fault1() at vn_io_fault1+0x161/frame 0xfe0a90b8ecc0 vn_io_fault() at vn_io_fault+0x1b5/frame 0xfe0a90b8ed40 dofilewrite() at dofilewrite+0x81/frame 0xfe0a90b8ed90 sys_write() at sys_write+0xc0/frame 0xfe0a90b8ee00 amd64_syscall() at amd64_syscall+0x157/frame 0xfe0a90b8ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfe0a90b8ef30 --- syscall (4, FreeBSD ELF64, write), rip = 0x103cddf7949a, rsp = 0x103cdc85dd48, rbp = 0x103cdc85dd80 --- KDB: enter: panic [ thread pid 95000 tid 135035 ] Stopped at kdb_enter+0x32: movq$0,0x9e4153(%rip) The posted 14.0 schedule which plans to branch stable/14 on May 12 and one cannot bet on the feature getting beaten up into production shape by that time. Given whatever non-block_clonning and not even zfs bugs which are likely to come out I think this makes the feature a non-starter for said release. I note: 1. the current problems did not make it into stable branches. 2. there was block_cloning-related data corruption (fixed) and there may be more 3. there was unrelated data corruption (see https://github.com/openzfs/zfs/issues/14753), sorted out by reverting the problematic commit in FreeBSD, not yet sorted out upstream As such people's data may be partially hosed as is. Consequently the proposed plan is as follows: 1. whack the block cloning feature for the time being, but make sure pools which upgraded to it can be mounted read-only 2. run ztest and whatever other stress testing on FreeBSD, along with restoring openzfs CI -- I can do the first part, I'm sure pho will not mind to run some tests of his own 3. recommend people create new pools and restore data from backup. if restoring from backup is not an option, tar or cp (not zfs send) from the read-only mount block cloning beaten into shape would use block_cloning_v2 or whatever else, key point that the current feature name would be considered bogus (not blocking RO import though) to prevent RW usage of the current pools with it enabled. Comments? Correct me if I'm wrong, but from my understanding there were zero problems with block cloning when it wasn't in use or now disabled. The reason I've introduced vfs.zfs.bclone_enabled sysctl, was to exactly avoid mess like this and give us more time to sort all the problems out while making it easy for people to try it. If there is no plan to revert the whole import, I don't see what value removing just block cloning will bring if it is now disabled by default and didn't cause any problems when disabled. -- Pawel Jakub Dawidek
another crash and going forward with zfs
After bugfixes got committed I decided to zpool upgrade and sysctl vfs.zfs.bclone_enabled=1 vs poudriere for testing purposes. I very quickly got a new crash: panic: VERIFY(arc_released(db->db_buf)) failed cpuid = 9 time = 1681755046 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe0a90b8e5f0 vpanic() at vpanic+0x152/frame 0xfe0a90b8e640 spl_panic() at spl_panic+0x3a/frame 0xfe0a90b8e6a0 dbuf_redirty() at dbuf_redirty+0xbd/frame 0xfe0a90b8e6c0 dmu_buf_will_dirty_impl() at dmu_buf_will_dirty_impl+0xa2/frame 0xfe0a90b8e700 dmu_write_uio_dnode() at dmu_write_uio_dnode+0xe9/frame 0xfe0a90b8e780 dmu_write_uio_dbuf() at dmu_write_uio_dbuf+0x42/frame 0xfe0a90b8e7b0 zfs_write() at zfs_write+0x672/frame 0xfe0a90b8e960 zfs_freebsd_write() at zfs_freebsd_write+0x39/frame 0xfe0a90b8e980 VOP_WRITE_APV() at VOP_WRITE_APV+0xdb/frame 0xfe0a90b8ea90 vn_write() at vn_write+0x325/frame 0xfe0a90b8eb20 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfe0a90b8eb80 vn_io_fault1() at vn_io_fault1+0x161/frame 0xfe0a90b8ecc0 vn_io_fault() at vn_io_fault+0x1b5/frame 0xfe0a90b8ed40 dofilewrite() at dofilewrite+0x81/frame 0xfe0a90b8ed90 sys_write() at sys_write+0xc0/frame 0xfe0a90b8ee00 amd64_syscall() at amd64_syscall+0x157/frame 0xfe0a90b8ef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfe0a90b8ef30 --- syscall (4, FreeBSD ELF64, write), rip = 0x103cddf7949a, rsp = 0x103cdc85dd48, rbp = 0x103cdc85dd80 --- KDB: enter: panic [ thread pid 95000 tid 135035 ] Stopped at kdb_enter+0x32: movq$0,0x9e4153(%rip) The posted 14.0 schedule which plans to branch stable/14 on May 12 and one cannot bet on the feature getting beaten up into production shape by that time. Given whatever non-block_clonning and not even zfs bugs which are likely to come out I think this makes the feature a non-starter for said release. I note: 1. the current problems did not make it into stable branches. 2. there was block_cloning-related data corruption (fixed) and there may be more 3. there was unrelated data corruption (see https://github.com/openzfs/zfs/issues/14753), sorted out by reverting the problematic commit in FreeBSD, not yet sorted out upstream As such people's data may be partially hosed as is. Consequently the proposed plan is as follows: 1. whack the block cloning feature for the time being, but make sure pools which upgraded to it can be mounted read-only 2. run ztest and whatever other stress testing on FreeBSD, along with restoring openzfs CI -- I can do the first part, I'm sure pho will not mind to run some tests of his own 3. recommend people create new pools and restore data from backup. if restoring from backup is not an option, tar or cp (not zfs send) from the read-only mount block cloning beaten into shape would use block_cloning_v2 or whatever else, key point that the current feature name would be considered bogus (not blocking RO import though) to prevent RW usage of the current pools with it enabled. Comments? -- Mateusz Guzik
