[heketi-devel] Create Volume Build Failed: No space
Hello, I'm trying to create a large volume but it fails shortly showing "No space", though there is plenty of space. The debug logs doesn't say much: [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #122 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #123 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #124 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #125 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #126 [heketi] ERROR 2021/05/19 08:11:17 heketi/apps/glusterfs/operations_manage.go:220:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No space I'm using the following heketi-cli command: heketi-cli --user admin --secret xZbphMQHQ volume create --name=cache --size=95545 --gluster-volume-options="performance.write-behind off","performance.open-behind off" --replica=3 If I change the volume size to 94800, it works. It's a 3 cluster nodes, each node have 9 disks 10Tb each and I have set max_bricks_per_volume to 600 and brick_max_size_gb 10240. My topology info is the end of the mail. volume size / (max_bricks_per_volume/ 3 nodes) = 95545/200 = 477.725 brick size device size = 11175, so 11175/477.725 = 23 bricks per device 9 devices * 23 bricks = 207 So the free space on the devices allow the creation of 207 bricks (477.725 size each) on every node, I need 200 bricks per node to create the volume, then why it fails for "No space" ? Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 File: true Block: true Volumes: Name: heketidbstorage Size: 2 Id: cc07e3205650842e31324cab359bca43 Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 Mount: 10.50.50.51:heketidbstorage Mount Options: backup-volfile-servers=10.50.50.52,10.50.50.53 Durability Type: replicate Replica: 3 Snapshot: Disabled Bricks: Id: 7a4004f395cce5fe89f4a0fa5af61549 Path: /var/lib/heketi/mounts/vg_a81b43597c607f9a3ba54e2552b821e8/brick_7a4004f395cce5fe89f4a0fa5af61549/brick Size (GiB): 2 Node: e18b193a75bc688ea86659bd1aa0a96c Device: a81b43597c607f9a3ba54e2552b821e8 Id: 84879c1d761d54788285a8728fe7100f Path: /var/lib/heketi/mounts/vg_d25b7a27273da7d25d1e1d8ca60ec922/brick_84879c1d761d54788285a8728fe7100f/brick Size (GiB): 2 Node: fc13ff5b8a48893e31b1c3da2815ac16 Device: d25b7a27273da7d25d1e1d8ca60ec922 Id: 9d7dd2cf5ae65b87111f4481f32fb7d7 Path: /var/lib/heketi/mounts/vg_bbd7550f3fdeccf291397337375b0cd2/brick_9d7dd2cf5ae65b87111f4481f32fb7d7/brick Size (GiB): 2 Node: 7ad303638bf94f974ebae8aaf11ac127 Device: bbd7550f3fdeccf291397337375b0cd2 Nodes: Node Id: 7ad303638bf94f974ebae8aaf11ac127 State: online Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 Zone: 1 Management Hostnames: appliance1 Storage Hostnames: 10.50.50.51 Devices: Id:00599faa322c0ef3cd5de1f29c6fb38e State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdg Bricks: Id:35c678931dd899477289034ca9c4be87 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdc Bricks: Id:4b21b903a642a65557fee7f38745a393 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdh Bricks: Id:718ef6478d1209909e38567a81ee3dc5 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdj Bricks: Id:7569be6559a050ae87c18096a94f7ee1 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdk Bricks: Id:88e42232a848d4dce05fbc9e6ab5fe24 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdf Bricks: Id:a7129a57851de3c8f74f059486ef801d State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sde Bricks: Id:bbd7550f3fdeccf291397337375b0cd2 State:online Size (GiB):11175 Used (GiB):2 Free (GiB):11173 Known Paths: /dev/sdd Bricks: Id:9d7dd2cf5ae65b87111f4481f32fb7d7 Size (GiB):2 Path:
[heketi-devel] Create Volume Build Failed: No space
Hello, I'm trying to create a large volume but it fails shortly showing "No space", though there is plenty of space. The debug logs doesn't say much: [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #122 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #123 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #124 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #125 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #126 [heketi] ERROR 2021/05/19 08:11:17 heketi/apps/glusterfs/operations_manage.go:220:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No space I'm using the following heketi-cli command: heketi-cli --user admin --secret xZbphMQHQ volume create --name=cache --size=95545 --gluster-volume-options="performance.write-behind off","performance.open-behind off" --replica=3 If I change the volume size to 94800, it works. It's a 3 cluster nodes, each node have 9 disks 10Tb each and I have set max_bricks_per_volume to 600 and brick_max_size_gb 10240. My topology info is the end of the mail. volume size / (max_bricks_per_volume/ 3 nodes) = 95545/200 = 477.725 brick size device size = 11175, so 11175/477.725 = 23 bricks per device 9 devices * 23 bricks = 207 So the free space on the devices allow the creation of 207 bricks (477.725 size each) on every node, I need 200 bricks per node to create the volume, then why it fails for "No space" ? Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 File: true Block: true Volumes: Name: heketidbstorage Size: 2 Id: cc07e3205650842e31324cab359bca43 Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 Mount: 10.50.50.51:heketidbstorage Mount Options: backup-volfile-servers=10.50.50.52,10.50.50.53 Durability Type: replicate Replica: 3 Snapshot: Disabled Bricks: Id: 7a4004f395cce5fe89f4a0fa5af61549 Path: /var/lib/heketi/mounts/vg_a81b43597c607f9a3ba54e2552b821e8/brick_7a4004f395cce5fe89f4a0fa5af61549/brick Size (GiB): 2 Node: e18b193a75bc688ea86659bd1aa0a96c Device: a81b43597c607f9a3ba54e2552b821e8 Id: 84879c1d761d54788285a8728fe7100f Path: /var/lib/heketi/mounts/vg_d25b7a27273da7d25d1e1d8ca60ec922/brick_84879c1d761d54788285a8728fe7100f/brick Size (GiB): 2 Node: fc13ff5b8a48893e31b1c3da2815ac16 Device: d25b7a27273da7d25d1e1d8ca60ec922 Id: 9d7dd2cf5ae65b87111f4481f32fb7d7 Path: /var/lib/heketi/mounts/vg_bbd7550f3fdeccf291397337375b0cd2/brick_9d7dd2cf5ae65b87111f4481f32fb7d7/brick Size (GiB): 2 Node: 7ad303638bf94f974ebae8aaf11ac127 Device: bbd7550f3fdeccf291397337375b0cd2 Nodes: Node Id: 7ad303638bf94f974ebae8aaf11ac127 State: online Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 Zone: 1 Management Hostnames: appliance1 Storage Hostnames: 10.50.50.51 Devices: Id:00599faa322c0ef3cd5de1f29c6fb38e State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdg Bricks: Id:35c678931dd899477289034ca9c4be87 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdc Bricks: Id:4b21b903a642a65557fee7f38745a393 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdh Bricks: Id:718ef6478d1209909e38567a81ee3dc5 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdj Bricks: Id:7569be6559a050ae87c18096a94f7ee1 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdk Bricks: Id:88e42232a848d4dce05fbc9e6ab5fe24 State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sdf Bricks: Id:a7129a57851de3c8f74f059486ef801d State:online Size (GiB):11175 Used (GiB):0 Free (GiB):11175 Known Paths: /dev/sde Bricks: Id:bbd7550f3fdeccf291397337375b0cd2 State:online Size (GiB):11175 Used (GiB):2 Free (GiB):11173 Known Paths: /dev/sdd Bricks: Id:9d7dd2cf5ae65b87111f4481f32fb7d7 Size (GiB):2 Path: /var/lib/heketi/mounts/vg_bbd7550f3fdeccf291397337375b0cd2/brick_9d7dd2cf5ae65b87111f4481f32fb7d7/brick
