On 17/05/2017 11:59, Andriy Gapon wrote:
>
> A FreeBSD user reported a problem with doing a force receive of a full stream
> (non-incremental) while using a debug build:
>
> panic: solaris assert: dbuf_is_metadata(db) == arc_is_metadata(buf) (0x0 ==
> 0x1)
> assfail3()
> dbuf_assign_arcbuf()
> dmu_assign_arcbuf()
> receive_writer_thread()
>
> Here is a corresponding write record header:
> header = {
> drr_type = DRR_WRITE,
> drr_payloadlen = 0,
> drr_u = {
> ...
> drr_write = {
> drr_object = 357,
> drr_type = DMU_OT_DIRECTORY_CONTENTS,
> drr_pad = 0,
> drr_offset = 0,
> drr_logical_size = 512,
> drr_toguid = 6461692233939371501,
> drr_checksumtype = 2 '\002',
> drr_checksumflags = 0 '\000',
> drr_compressiontype = 0 '\000',
> drr_pad2 = "\000\000\000\000",
> drr_key = {
> ddk_cksum = {
> zc_word = {0, 0, 0, 0}
> },
> ddk_prop = 0
> },
> drr_compressed_size = 0
> },
>
>
> Here is the ARC header:
> $6 = {
> b_dva = {
> dva_word = {0, 0}
> },
> b_birth = 0,
> b_type = ARC_BUFC_METADATA,
> b_hash_next = 0x0 <avl_balance2child>,
> b_flags = (ARC_FLAG_BUFC_METADATA | ARC_FLAG_HAS_L1HDR |
> ARC_FLAG_COMPRESSED_ARC | ARC_FLAG_SHARED_DATA | ARC_FLAG_COMPRESS_1),
> b_psize = 1,
> b_lsize = 1,
> b_spa = 7546455437792279428,
> ...
>
> Here is the dnode:
> dn_objset = 0xfffff800815f1800,
> dn_object = 357,
> dn_dbuf = 0xfffff800826da6b8,
> dn_handle = 0xfffff800815ca208,
> dn_phys = 0xfffffe0002d1aa00,
> dn_type = DMU_OT_PLAIN_FILE_CONTENTS,
> dn_bonuslen = 204,
> dn_bonustype = 44 ',',
> dn_nblkptr = 1 '\001',
> dn_checksum = 0 '\000',
> dn_compress = 0 '\000',
> dn_nlevels = 1 '\001',
> dn_indblkshift = 17 '\021',
> dn_datablkshift = 9 '\t',
> dn_moved = 0 '\000',
> dn_datablkszsec = 1,
> dn_datablksz = 512,
> dn_maxblkid = 0,
> dn_next_type = "\000\000\000",
> dn_next_nblkptr = "\000\000\000",
> dn_next_nlevels = "\000\000\000",
> dn_next_indblkshift = "\000\000\000\021",
> dn_next_bonustype = "\000\000\000,",
> dn_rm_spillblk = "\000\000\000",
> dn_next_bonuslen = {0, 0, 0, 204},
> dn_next_blksz = {0, 0, 0, 512},
> dn_dbufs_count = 2,
> dn_allocated_txg = 199719,
> dn_free_txg = 0,
> dn_assigned_txg = 199719,
> dn_dirtyctx = DN_DIRTY_OPEN,
>
> tx_txg = 199719 as well.
>
> The problem started happening after the compressed send / receive feature was
> introduced. But the feature was not used when the crash happened.
>
> I think that that being a force receive we are probably changing the object
> type
> to a different one (from a plain file to a directory), but something goes
> wrong
> with that. Alternatively, maybe there is an issue with setting
> drr_write.drr_type on the sender side.
I got more information and the problem seems to be with on-disk data on the
sender.
Relevant records from the stream:
OBJECT object = 357 type = 19 bonustype = 44 blksz = 512 bonuslen = 204
FREE object = 357 offset = 512 length = -1
WRITE object = 357 type = 20 checksum type = 2 compression type = 0
I wonder how this could be...
According to zdb the object is a plain file carrying a symbolic link and it has
some content in an embedded block pointer:
Object lvl iblk dblk dsize lsize %full type
357 1 16K 512 0 512 100.00 ZFS plain file
204 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /cache/pkg/libxcb-1.10_2.txz
...
mode 120755
size 28
parent 149
links 1
pflags 40800000104
Indirect blocks:
0 L0 EMBEDDED et=0 200L/1cP B=1158
I think that it's strange that the object has any data at all as the link target
is in the system attributes.
Here is a hex dump of the dnode physical data:
00000a00 13 0e 01 01 2c 00 00 03 01 00 cc 00 00 00 00 00 |....,...........|
00000a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000a20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000a30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000a40 00 00 00 18 21 03 00 01 00 51 80 cb a6 c7 e2 0a |....!....Q......|
00000a50 00 0f 02 00 ff d8 50 00 00 00 00 00 00 00 00 00 |......P.........|
00000a60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000a70 ff 01 00 36 8f 00 14 80 00 00 00 00 00 00 00 00 |...6............|
00000a80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000a90 86 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000aa0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000ab0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000ac0 5a 50 2f 00 03 08 18 00 1c 00 00 00 00 00 00 00 |ZP/.............|
00000ad0 ed a1 00 00 00 00 00 00 1c 00 00 00 00 00 00 00 |................|
00000ae0 12 ad 23 00 00 00 00 00 00 00 00 00 00 00 00 00 |..#.............|
00000af0 00 00 00 00 00 00 00 00 95 00 00 00 00 00 00 00 |................|
00000b00 04 01 00 00 08 04 00 00 b1 f2 f4 53 00 00 00 00 |...........S....|
00000b10 bb 57 87 3a 00 00 00 00 b1 f2 f4 53 00 00 00 00 |.W.:.......S....|
00000b20 bb 57 87 3a 00 00 00 00 b1 f2 f4 53 00 00 00 00 |.W.:.......S....|
00000b30 bb 57 87 3a 00 00 00 00 b1 f2 f4 53 00 00 00 00 |.W.:.......S....|
00000b40 bb 57 87 3a 00 00 00 00 01 00 00 00 00 00 00 00 |.W.:............|
00000b50 03 00 00 00 00 00 00 00 00 00 00 10 bf 01 1e 00 |................|
00000b60 00 00 40 20 a9 00 12 00 00 00 00 40 a9 00 12 00 |..@ .......@....|
00000b70 6c 69 62 78 63 62 2d 31 2e 31 30 5f 32 2d 36 39 |libxcb-1.10_2-69|
00000b80 32 63 38 62 30 34 32 36 2e 74 78 7a 00 00 00 00 |2c8b0426.txz....|
00000b90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000ba0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000bb0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000bc0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000bd0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000be0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000bf0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
If I am not mistaken the block pointer is in bytes 0xa40 - 0xabf.
So, the BP properties must be ff 01 00 36 8f 00 14 80 or 0x8014008f360001ff
They look sane, except that the BP type is 0x14 (20) instead of the expected
0x13 (19) which is the dnode type.
Puzzling...
--
Andriy Gapon
------------------------------------------
openzfs-developer
Archives: https://openzfs.topicbox.com/groups/developer/
Powered by Topicbox: https://topicbox.com
