[
https://issues.apache.org/jira/browse/HDDS-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-6256:
------------------------------------
Description:
I write a tiny key into a 3-2 bucket. Container group is still open - this
looks good 3 containers as expected:
{code:java}
Replicas: [State: OPEN; ReplicaIndex: 4; Origin:
3fc2c737-83b1-4485-91ac-b771e41fa1dd; Location:
3fc2c737-83b1-4485-91ac-b771e41fa1dd/ozone_datanode_1.ozone_default,
State: OPEN; ReplicaIndex: 5; Origin: 1be82529-d5f2-4c6b-8573-0bb75863ded1;
Location: 1be82529-d5f2-4c6b-8573-0bb75863ded1/ozone_datanode_4.ozone_default,
State: OPEN; ReplicaIndex: 1; Origin: e1d6ca08-d5eb-4e8a-b42a-bf1257e9f673;
Location:
e1d6ca08-d5eb-4e8a-b42a-bf1257e9f673/ozone_datanode_2.ozone_default]{code}
In the JSON:
{code:java}
"bytesUsed" : 177,
"replicaIndex" : 1
...
"bytesUsed" : 1048576,
"replicaIndex" : 5
...
"bytesUsed" : 1048576,
"replicaIndex" : 4 {code}
So the parity should only have 177 bytes too, but it appears to have 1MB.
Checking the block file on disk, it is indeed 1MB:
{code:java}
bash-4.2$ cd
/data/hdds/hdds/CID-137befd9-f6ec-430b-b0e0-ee683ac1d987/current/containerDir0/3/
bash-4.2$ ls
chunks metadata
bash-4.2$ cd chunks/
bash-4.2$ ls
109611004723200004.block
bash-4.2$ ls -al
total 1032
drwxr-xr-x 2 hadoop users 4096 Feb 2 22:08 .
drwxr-xr-x 4 hadoop users 4096 Feb 2 22:08 ..
rw-rr- 1 hadoop users 1048576 Feb 2 22:08 109611004723200004.block{code}
So something is not correct with the write path here. Checking the block
contents, its all padded with zeros:
{code:java}
bash-4.2$ hexdump -C 109611004723200004.block
00000000 e4 e5 e6 1a 10 1a 10 1a e4 07 24 25 21 d4 24 d3 |..........$%!.$.|
00000010 25 da 2c 06 16 16 e4 07 24 25 21 d4 24 d3 25 da |%.,.....$%!.$.%.|
00000020 2c eb 27 db 12 1b 24 25 21 d4 24 d3 25 da 2c eb |,.'...$%!.$.%.,.|
00000030 27 db 12 1b 24 25 25 db d5 d4 21 d2 06 22 23 10 |'...$%%...!.."#.|
00000040 10 16 16 10 07 27 db 12 1b 24 25 21 d4 24 d1 23 |.....'...$%!.$.#|
00000050 2c 06 22 22 10 10 16 16 10 07 27 db 12 1b d0 21 |,.""......'....!|
00000060 d4 da 2c db 2e 23 22 27 28 06 22 22 10 e5 16 16 |..,..#"'(.""....|
00000070 e4 07 27 db 12 1b d4 24 24 d1 25 d7 23 da 06 22 |..'....$$.%.#.."|
00000080 22 10 e5 16 16 e5 07 27 db 12 1b d4 24 24 2e 25 |"......'....$$.%|
00000090 d8 2c 23 2e da 06 e4 17 e5 1a e4 12 e3 1a e4 12 |.,#.............|
000000a0 10 1a e5 07 e6 e4 e7 11 d5 13 13 10 10 e3 d7 e3 |................|
000000b0 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00100000{code}
was:
I write a tiny key into a 3-2 bucket. Container group is still open - this
looks good 3 containers as expected:
Replicas: [State: OPEN; ReplicaIndex: 4; Origin:
3fc2c737-83b1-4485-91ac-b771e41fa1dd; Location:
3fc2c737-83b1-4485-91ac-b771e41fa1dd/ozone_datanode_1.ozone_default,
State: OPEN; ReplicaIndex: 5; Origin: 1be82529-d5f2-4c6b-8573-0bb75863ded1;
Location: 1be82529-d5f2-4c6b-8573-0bb75863ded1/ozone_datanode_4.ozone_default,
State: OPEN; ReplicaIndex: 1; Origin: e1d6ca08-d5eb-4e8a-b42a-bf1257e9f673;
Location: e1d6ca08-d5eb-4e8a-b42a-bf1257e9f673/ozone_datanode_2.ozone_default]
In the JSON:
"bytesUsed" : 177,
"replicaIndex" : 1
...
"bytesUsed" : 1048576,
"replicaIndex" : 5
...
"bytesUsed" : 1048576,
"replicaIndex" : 4
So the parity should only have 177 bytes too, but it appears to have 1MB.
Checking the block file on disk, it is indeed 1MB:
bash-4.2$ cd
/data/hdds/hdds/CID-137befd9-f6ec-430b-b0e0-ee683ac1d987/current/containerDir0/3/
bash-4.2$ ls
chunks metadata
bash-4.2$ cd chunks/
bash-4.2$ ls
109611004723200004.block
bash-4.2$ ls -al
total 1032
drwxr-xr-x 2 hadoop users 4096 Feb 2 22:08 .
drwxr-xr-x 4 hadoop users 4096 Feb 2 22:08 ..
-rw-r--r-- 1 hadoop users 1048576 Feb 2 22:08 109611004723200004.block
So something is not correct with the write path here. Checking the block
contents, its all padded with zeros:
bash-4.2$ hexdump -C 109611004723200004.block
00000000 e4 e5 e6 1a 10 1a 10 1a e4 07 24 25 21 d4 24 d3 |..........$%!.$.|
00000010 25 da 2c 06 16 16 e4 07 24 25 21 d4 24 d3 25 da |%.,.....$%!.$.%.|
00000020 2c eb 27 db 12 1b 24 25 21 d4 24 d3 25 da 2c eb |,.'...$%!.$.%.,.|
00000030 27 db 12 1b 24 25 25 db d5 d4 21 d2 06 22 23 10 |'...$%%...!.."#.|
00000040 10 16 16 10 07 27 db 12 1b 24 25 21 d4 24 d1 23 |.....'...$%!.$.#|
00000050 2c 06 22 22 10 10 16 16 10 07 27 db 12 1b d0 21 |,.""......'....!|
00000060 d4 da 2c db 2e 23 22 27 28 06 22 22 10 e5 16 16 |..,..#"'(.""....|
00000070 e4 07 27 db 12 1b d4 24 24 d1 25 d7 23 da 06 22 |..'....$$.%.#.."|
00000080 22 10 e5 16 16 e5 07 27 db 12 1b d4 24 24 2e 25 |"......'....$$.%|
00000090 d8 2c 23 2e da 06 e4 17 e5 1a e4 12 e3 1a e4 12 |.,#.............|
000000a0 10 1a e5 07 e6 e4 e7 11 d5 13 13 10 10 e3 d7 e3 |................|
000000b0 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00100000
> EC: Parity blocks are padded incorrectly with zeros to the chunk size
> ---------------------------------------------------------------------
>
> Key: HDDS-6256
> URL: https://issues.apache.org/jira/browse/HDDS-6256
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Stephen O'Donnell
> Priority: Major
>
> I write a tiny key into a 3-2 bucket. Container group is still open - this
> looks good 3 containers as expected:
> {code:java}
> Replicas: [State: OPEN; ReplicaIndex: 4; Origin:
> 3fc2c737-83b1-4485-91ac-b771e41fa1dd; Location:
> 3fc2c737-83b1-4485-91ac-b771e41fa1dd/ozone_datanode_1.ozone_default,
> State: OPEN; ReplicaIndex: 5; Origin: 1be82529-d5f2-4c6b-8573-0bb75863ded1;
> Location: 1be82529-d5f2-4c6b-8573-0bb75863ded1/ozone_datanode_4.ozone_default,
> State: OPEN; ReplicaIndex: 1; Origin: e1d6ca08-d5eb-4e8a-b42a-bf1257e9f673;
> Location:
> e1d6ca08-d5eb-4e8a-b42a-bf1257e9f673/ozone_datanode_2.ozone_default]{code}
> In the JSON:
>
> {code:java}
> "bytesUsed" : 177,
> "replicaIndex" : 1
> ...
> "bytesUsed" : 1048576,
> "replicaIndex" : 5
> ...
> "bytesUsed" : 1048576,
> "replicaIndex" : 4 {code}
> So the parity should only have 177 bytes too, but it appears to have 1MB.
> Checking the block file on disk, it is indeed 1MB:
> {code:java}
> bash-4.2$ cd
> /data/hdds/hdds/CID-137befd9-f6ec-430b-b0e0-ee683ac1d987/current/containerDir0/3/
> bash-4.2$ ls
> chunks metadata
> bash-4.2$ cd chunks/
> bash-4.2$ ls
> 109611004723200004.block
> bash-4.2$ ls -al
> total 1032
> drwxr-xr-x 2 hadoop users 4096 Feb 2 22:08 .
> drwxr-xr-x 4 hadoop users 4096 Feb 2 22:08 ..
> rw-rr- 1 hadoop users 1048576 Feb 2 22:08 109611004723200004.block{code}
> So something is not correct with the write path here. Checking the block
> contents, its all padded with zeros:
> {code:java}
> bash-4.2$ hexdump -C 109611004723200004.block
> 00000000 e4 e5 e6 1a 10 1a 10 1a e4 07 24 25 21 d4 24 d3 |..........$%!.$.|
> 00000010 25 da 2c 06 16 16 e4 07 24 25 21 d4 24 d3 25 da |%.,.....$%!.$.%.|
> 00000020 2c eb 27 db 12 1b 24 25 21 d4 24 d3 25 da 2c eb |,.'...$%!.$.%.,.|
> 00000030 27 db 12 1b 24 25 25 db d5 d4 21 d2 06 22 23 10 |'...$%%...!.."#.|
> 00000040 10 16 16 10 07 27 db 12 1b 24 25 21 d4 24 d1 23 |.....'...$%!.$.#|
> 00000050 2c 06 22 22 10 10 16 16 10 07 27 db 12 1b d0 21 |,.""......'....!|
> 00000060 d4 da 2c db 2e 23 22 27 28 06 22 22 10 e5 16 16 |..,..#"'(.""....|
> 00000070 e4 07 27 db 12 1b d4 24 24 d1 25 d7 23 da 06 22 |..'....$$.%.#.."|
> 00000080 22 10 e5 16 16 e5 07 27 db 12 1b d4 24 24 2e 25 |"......'....$$.%|
> 00000090 d8 2c 23 2e da 06 e4 17 e5 1a e4 12 e3 1a e4 12 |.,#.............|
> 000000a0 10 1a e5 07 e6 e4 e7 11 d5 13 13 10 10 e3 d7 e3 |................|
> 000000b0 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> 000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00100000{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
