Hi Robert, We had a similar issue and it turned out that heketi stores and counts free/used space in its topology separately to underlying LVM. This means it can run out of sync. I found this article which is a nice walkthrough how to sync them again: https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.0/troubleshoot/heketi_disk.html
Mit freundlichen Grüßen / Best regards i.A. Dipl.-Ing.(FH) Johannes Grumböck Infrastructure Architect, Computing & Platform Services Infrastructure & Common Platforms Telefon +43 (0)662 4670-6323 Telefax +43 (0)662 4670-16323 johannes.grumbo...@porscheinformatik.at Porsche Informatik Gesellschaft m.b.H. | A - 5020 Salzburg | Louise-Piëch-Straße 9 Sitz: Salzburg | FN 72830 d / Landesgericht Salzburg | DVR 88439 | UID ATU 36773309 http://www.porscheinformatik.at/ Internal Von: heketi-devel-boun...@gluster.org <heketi-devel-boun...@gluster.org> Im Auftrag von Polasek, Robert Gesendet: Donnerstag, 25. Oktober 2018 22:30 An: heketi-devel@gluster.org Betreff: [EXT] [heketi-devel] volume provisioning fails with "No space" Hi everybody, I am testing heketi autoprovisioning of gluster volumes and I am hitting a problem where heketi reports "No space" in the log, even there should be a space. Would somebody be able to point me what I need to look into or where my problem is? Here are the details of my problem: I am using a curl to simulate my failing request from kubernetes. As a side note, our kubernetes is old, 1.4 and therefore I cannot robert@workstation:~/src/ifds/helm/tapremium$ curl -v -X POST -H "Authorization: bearer ****" -H "Content-Type: application/json" POST http://localhost:8080/volumes -d@/tmp/volume.json<mailto:-d@/tmp/volume.json> ================================================================================================================= This is a content of /tmp/volume.json file { "size": 10, "name": "", "durability": { "type": "replicate", "replicate": { "replica": 3 }, "disperse": {} }, "gid": 2000, "snapshot": { "enable": false, "factor": 0 } } This is my topology ================================================================================================================= Cluster Id: 7396c8b03593f9bfa1288cdebd9412ac Volumes: Name: vol_4708719dddaacda301e3b71763cb73fa Size: 10 Id: 4708719dddaacda301e3b71763cb73fa Cluster Id: 7396c8b03593f9bfa1288cdebd9412ac Mount: 192.168.48.9:vol_4708719dddaacda301e3b71763cb73fa Mount Options: backup-volfile-servers= Durability Type: none Snapshot: Disabled Bricks: Id: 99b33f7f23347c1691e4a93c7220af04 Path: /var/lib/heketi/mounts/vg_05df77bacca68b941f8d3d20d64a1aa2/brick_99b33f7f23347c1691e4a93c7220af04/brick Size (GiB): 5 Node: 31e1ac3014029ff48398a989c10952b6 Device: 05df77bacca68b941f8d3d20d64a1aa2 Id: acebc8961ffb304fa2b97c201c7af8da Path: /var/lib/heketi/mounts/vg_05df77bacca68b941f8d3d20d64a1aa2/brick_acebc8961ffb304fa2b97c201c7af8da/brick Size (GiB): 5 Node: 31e1ac3014029ff48398a989c10952b6 Device: 05df77bacca68b941f8d3d20d64a1aa2 Nodes: Node Id: 31e1ac3014029ff48398a989c10952b6 State: online Cluster Id: 7396c8b03593f9bfa1288cdebd9412ac Zone: 1 Management Hostname: 192.168.48.9 Storage Hostname: 192.168.48.9 Devices: Id:05df77bacca68b941f8d3d20d64a1aa2 Name:/dev/gluster_vg/heketi_volumeState:online Size (GiB):99 Used (GiB):10 Free (GiB):89 Bricks: Id:99b33f7f23347c1691e4a93c7220af04 Size (GiB):5 Path: /var/lib/heketi/mounts/vg_05df77bacca68b941f8d3d20d64a1aa2/brick_99b33f7f23347c1691e4a93c7220af04/brick Id:acebc8961ffb304fa2b97c201c7af8da Size (GiB):5 Path: /var/lib/heketi/mounts/vg_05df77bacca68b941f8d3d20d64a1aa2/brick_acebc8961ffb304fa2b97c201c7af8da/brick Id:6c0325d31e1b84a4f3aa088ba352e2ec Name:/dev/gluster_vg/heketi_volume2State:online Size (GiB):24 Used (GiB):0 Free (GiB):24 Bricks: Id:bac78b5e4d558d387218d12fdfb7bfa0 Name:/dev/gluster_vg/heketi_volume1State:online Size (GiB):24 Used (GiB):0 Free (GiB):24 Bricks: ================================================================================================================= and this is the log output [negroni] Started POST /volumes [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:101: [e8cb84a8b977a00d77e137a05a5b2530] Replica 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:580: Using the following clusters: [7396c8b03593f9bfa1288cdebd9412ac] [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: brick_size = 10485760 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets = 1 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: num_bricks = 3 [heketi] INFO 2018/10/25 14:08:28 Allocating brick set #0 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected space needed for amount=10485760 snapFactor=1 : 10539008 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device 6c0325d31e1b84a4f3aa088ba352e2ec[26079232] > required size [10539008] ? [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 1 / 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:455: Error detected. Cleaning up volume e8cb84a8b977a00d77e137a05a5b2530: Len(0) [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:56: No space, re-trying with smaller brick size [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: brick_size = 5242880 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets = 2 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: num_bricks = 6 [heketi] INFO 2018/10/25 14:08:28 Allocating brick set #0 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected space needed for amount=5242880 snapFactor=1 : 5271552 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device bac78b5e4d558d387218d12fdfb7bfa0[26079232] > required size [5271552] ? [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 1 / 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:455: Error detected. Cleaning up volume e8cb84a8b977a00d77e137a05a5b2530: Len(0) [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:56: No space, re-trying with smaller brick size [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: brick_size = 2621440 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets = 4 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: num_bricks = 12 [heketi] INFO 2018/10/25 14:08:28 Allocating brick set #0 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected space needed for amount=2621440 snapFactor=1 : 2637824 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device 05df77bacca68b941f8d3d20d64a1aa2[94179328] > required size [2637824] ? [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 1 / 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:455: Error detected. Cleaning up volume e8cb84a8b977a00d77e137a05a5b2530: Len(0) [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:56: No space, re-trying with smaller brick size [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: brick_size = 1310720 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets = 8 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: num_bricks = 24 [heketi] INFO 2018/10/25 14:08:28 Allocating brick set #0 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected space needed for amount=1310720 snapFactor=1 : 1318912 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device 05df77bacca68b941f8d3d20d64a1aa2[94179328] > required size [1318912] ? [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 1 / 3 [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:455: Error detected. Cleaning up volume e8cb84a8b977a00d77e137a05a5b2530: Len(0) [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:56: No space, re-trying with smaller brick size [heketi] ERROR 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:37: Minimum brick size limit reached. Out of space. [heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:445: Cluster 7396c8b03593f9bfa1288cdebd9412ac can not accommodate volume (Minimum brick size limit reached. Out of space.), trying next cluster [heketi] ERROR 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Volume Build Failed: No space [negroni] Completed 500 Internal Server Error in 3.397009ms This message is marked Public ________________________________ Please consider the environment before printing this email and any attachments. This e-mail and any attachments are intended only for the individual or company to which it is addressed and may contain information which is privileged, confidential and prohibited from disclosure or unauthorized use under applicable law. If you are not the intended recipient of this e-mail, you are hereby notified that any use, dissemination, or copying of this e-mail or the information contained in this e-mail is strictly prohibited by the sender. If you have received this transmission in error, please return the material received to the sender and delete all copies from your system.
_______________________________________________ heketi-devel mailing list heketi-devel@gluster.org https://lists.gluster.org/mailman/listinfo/heketi-devel
