Hi Dan, The Gluster Block volumes works with the concept of block-hosting volume, and these ones are created with 100GB by default.
To clarify, the block volumes will be provisioned over the block hosting volumes. Let's say you need a 10GB block volume, it will create a block hosting volume with 100GB and then the 10GB block volume over it, as the next block volumes requested until it reaches the 100GB. After that a new block hosting volume will be created and so on. So, if you have just 26GB available in each server, it's not enough to create the block hosting volume. You may need to add more devices to your CNS Cluster to grow your free space. Kind regards, Rodrigo Bersa Cloud Consultant, RHCVA, RHCE Red Hat Brasil <https://www.redhat.com> rbe...@redhat.com M: +55-11-99557-5841 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil pelo *Great Place to Work*. On Wed, May 16, 2018 at 10:35 PM, Dan Pungă <dan.pu...@gmail.com> wrote: > Hello all! > > I have setup a cluster with 3 glusterFS nodes for disk persistence just as > specified in the docs. I have configured the inventory file to install the > containerized version to be used by Openshift's integrated registry. This > works fine. > > Now I wanted to install the metrics component and I followed the procedure > described here: https://docs.openshift.org/latest/install_config/ > persistent_storage/persistent_storage_glusterfs.html#install-example-infra > > I end up with openshift-infra project set up, but with 3 pods failing to > start and I think this has to do with the PVC for cassandra that fails to > create. > > oc get pvc metrics-cassandra-1 -o yaml > > apiVersion: v1 > kind: PersistentVolumeClaim > metadata: > annotations: > control-plane.alpha.kubernetes.io/leader: > '{"holderIdentity":"8ef584d1-5923-11e8-8730-0a580a830040"," > leaseDurationSeconds":15,"acquireTime":"2018-05-17T00: > 38:34Z","renewTime":"2018-05-17T00:55:33Z","leaderTransitions":0}' > kubectl.kubernetes.io/last-applied-configuration: | > {"apiVersion":"v1","kind":"PersistentVolumeClaim"," > metadata":{"annotations":{"volume.beta.kubernetes.io/storage-provisioner > ":"gluster.org/glusterblock"},"labels":{"metrics-infra":"hawkular- > cassandra"},"name":"metrics-cassandra-1","namespace":" > openshift-infra"},"spec":{"accessModes":["ReadWriteOnce"] > ,"resources":{"requests":{"storage":"6Gi"}},"storageClassName":"glusterfs- > registry-block"}} > volume.beta.kubernetes.io/storage-provisioner: > gluster.org/glusterblock > creationTimestamp: 2018-05-17T00:38:34Z > labels: > metrics-infra: hawkular-cassandra > name: metrics-cassandra-1 > namespace: openshift-infra > resourceVersion: "1204482" > selfLink: /api/v1/namespaces/openshift-infra/persistentvolumeclaims/ > metrics-cassandra-1 > uid: a18b8c20-596a-11e8-8a63-fa163ed601cb > spec: > accessModes: > - ReadWriteOnce > resources: > requests: > storage: 6Gi > storageClassName: glusterfs-registry-block > status: > phase: Pending > > oc describe pvc metrics-cassandra-1 shows these warnings: > > 36m 23m 13 gluster.org/glusterblock > glusterblock-registry-provisioner-dc-1-tljbb > 8ef584d1-5923-11e8-8730-0a580a830040 > Warning ProvisioningFailed Failed to provision volume > with StorageClass "glusterfs-registry-block": failed to create volume: > [heketi] failed to create volume: Failed to allocate new block volume: No > space > 36m 21m 14 gluster.org/glusterblock > glusterblock-registry-provisioner-dc-1-tljbb > 8ef584d1-5923-11e8-8730-0a580a830040 > Normal Provisioning External provisioner is > provisioning volume for claim "openshift-infra/metrics-cassandra-1" > 21m 21m 1 gluster.org/glusterblock > glusterblock-registry-provisioner-dc-1-tljbb > 8ef584d1-5923-11e8-8730-0a580a830040 > Warning ProvisioningFailed Failed to provision volume > with StorageClass "glusterfs-registry-block": failed to create volume: > [heketi] failed to create volume: Post http://heketi-registry- > default.apps.my.net/blockvolumes: dial tcp: lookup > heketi-registry-default.apps.my.net on 192.168.150.16:53: no such host > > In the default project, if I check the logs for heketi-registry, I get a > lot of > > [heketi] ERROR 2018/05/17 00:46:47 /src/github.com/heketi/heketi/ > apps/glusterfs/operations.go:909: Create Block Volume Build Failed: No > space > [negroni] Started POST /blockvolumes > [heketi] INFO 2018/05/17 00:49:02 Loaded simple allocator > [heketi] INFO 2018/05/17 00:49:02 brick_num: 0 > [heketi] INFO 2018/05/17 00:49:02 brick_num: 0 > [heketi] INFO 2018/05/17 00:49:02 brick_num: 0 > [heketi] INFO 2018/05/17 00:49:02 brick_num: 0 > [heketi] INFO 2018/05/17 00:49:02 brick_num: 1 > [negroni] Completed 500 Internal Server Error in 7.091238ms > > For the other glusterFS-related pod, I see the same errors reported by the > pvc creation > > oc logs -f glusterblock-registry-provisioner-dc-1-tljbb -n default > > I0516 22:38:49.136388 1 controller.go:1167] scheduleOperation[lock- > provision-openshift-infra/metrics-cassandra-1[1191fb8d- > 5959-11e8-94c9-fa163e1cba7f]] > I0516 22:38:49.166658 1 leaderelection.go:156] attempting to acquire > leader lease... > I0516 22:38:49.197051 1 leaderelection.go:178] successfully acquired > lease to provision for pvc openshift-infra/metrics-cassandra-1 > I0516 22:38:49.197122 1 controller.go:1167] > scheduleOperation[provision-openshift-infra/metrics- > cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]] > E0516 22:38:49.207257 1 glusterblock-provisioner.go:441] BLOCK > VOLUME NAME I RECEIEVED: > E0516 22:38:49.207288 1 glusterblock-provisioner.go:449] BLOCK > VOLUME CREATE REQUEST: &{Size:6 Clusters:[] Name: Hacount:3 Auth:true} > E0516 22:38:49.355122 1 glusterblock-provisioner.go:451] BLOCK > VOLUME RESPONSE: <nil> > E0516 22:38:49.355204 1 glusterblock-provisioner.go:453] [heketi] > failed to create volume: Failed to allocate new block volume: No space > E0516 22:38:49.355262 1 controller.go:895] Failed to provision > volume for claim "openshift-infra/metrics-cassandra-1" with StorageClass > "glusterfs-registry-block": failed to create volume: [heketi] failed to > create volume: Failed to allocate new block volume: No space > E0516 22:38:49.355365 1 goroutinemap.go:165] Operation for > "provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]" > failed. No retries permitted until 2018-05-16 22:40:51.355301022 +0000 UTC > m=+23465.283195247 (durationBeforeRetry 2m2s). Error: "failed to create > volume: [heketi] failed to create volume: Failed to allocate new block > volume: No space" > I0516 22:38:51.241605 1 leaderelection.go:198] stopped trying to > renew lease to provision for pvc openshift-infra/metrics-cassandra-1, > task failed > > Regarding the no space message, I am certain that there is space on the > device (if there isn't some glusterFS config that's done on the servers > which prevents them to extend/create the volumes). All disks have the same > 26GB capacity and lvs on one of the machines shows: > > LV VG > Attr LSize Pool Origin Data% > Meta% Move Log Cpy%Sync Convert > docker-pool rootvg > twi-aot--- <4,16g 52,37 > 2,62 > home rootvg > -wi-ao---- 1,00g > > root rootvg > -wi-ao---- 2,00g > > swap rootvg > -wi-a----- 2,00g > > tmp rootvg > -wi-ao---- 1,17g > > usr rootvg > -wi-ao---- 4,00g > > var rootvg > -wi-ao---- 4,00g > > brick_7aa3a789badd1ae620a2bbefe51b8c73 vg_bd61a1e6f317bb9decade964449c12e8 > Vwi-aotz-- 2,00g tp_7aa3a789badd1ae620a2bbefe51b8c73 > 0,71 > brick_8818ffee7ab2244ca721b7d15ea1e514 vg_bd61a1e6f317bb9decade964449c12e8 > Vwi-aotz-- 5,00g tp_8818ffee7ab2244ca721b7d15ea1e514 > 7,57 > tp_7aa3a789badd1ae620a2bbefe51b8c73 vg_bd61a1e6f317bb9decade964449c12e8 > twi-aotz-- 2,00g 0,71 > 0,33 > tp_8818ffee7ab2244ca721b7d15ea1e514 vg_bd61a1e6f317bb9decade964449c12e8 > twi-aotz-- 5,00g 7,57 0,29 > > Any ideas where to look for misconfigurations? > > Thank you, > Dan Pungă > > _______________________________________________ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > >
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
