Hello Folks, I am new to heketi and was following the documentation word to work.
I successfully created a 3 node topolgy and the cluster was successfully created. ./heketi-cli --server http://localhost:8085 topology load --json=topology.json Creating cluster ... ID: 1233163d3b76a6538a34d9b27a855ead Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node glust-0 ... ID: 82af3c3335b7d24d0e5fc7559d44f8d7 Adding device /dev/sdb ... OK Adding device /dev/sdc ... OK Creating node glust-1 ... ID: ec106227ea3580571a252b28ad223109 Adding device /dev/sdb ... OK Adding device /dev/sdc ... OK Creating node glust-2 ... ID: fc6535a147b39d65e7037015dcc0cc20 Adding device /dev/sdb ... OK Adding device /dev/sdc ... OK And on all 3 nodes, I can see that I have successfully peered. glust-1:~# gluster volume info No volumes present glust-1:~# gluster peer status Number of Peers: 2 Hostname: glust-0 Uuid: 22891efe-28ce-4c12-b97a-42ee78d50995 State: Peer in Cluster (Connected) Hostname: 10.27.239.240 Uuid: fc5834ec-2b7e-402b-a412-6783bae0db66 State: Peer in Cluster (Connected) But when I create a volume, I get ./heketi-cli --server http://localhost:8085 volume create --size=10 > Error: Get http://localhost:8085/queue/729862db18ef8d968eb7d0017c09b04d: > dial tcp [::1]:8085: getsockopt: connection refused And heketi crashes after spitting out HUGE amounts of logs. My client binaries are the latest Darwin ones. My heketi.json looks like: > > ╰─ cat heketi.json { "_port_comment": "Heketi Server Port Number", "port": "8085", "_enable_tls_comment": "Enable TLS in Heketi Server", "enable_tls": false, "_cert_file_comment": "Path to a valid certificate file", "cert_file": "", "_key_file_comment": "Path to a valid private key file", "key_file": "", > "_use_auth": "Enable JWT authorization. Please enable for deployment", "use_auth": false, "_jwt": "Private keys for access", "jwt": { "_admin": "Admin has access to all APIs", "admin": { "key": "My Secret" }, "_user": "User only has access to /volumes endpoint", "user": { "key": "My Secret" } }, "_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes > secret when running in Kubernetes. Default is off.", "backup_db_to_kube_secret": false, "_glusterfs_comment": "GlusterFS Configuration", "glusterfs": { "_executor_comment": [ "Execute plugin. Possible choices: mock, ssh", "mock: This setting is used for testing and development.", " It will not send commands to any node.", "ssh: This setting will notify Heketi to ssh to the nodes.", " It will need the values in sshexec to be configured.", "kubernetes: Communicate with GlusterFS containers over", " Kubernetes exec api." > ], > "executor": "ssh", > "_sshexec_comment": "SSH username and private key file information", > "sshexec": { > "keyfile": "id_rsa", > "user": "root", > "port": "22", > "fstab": "/etc/fstab", > "backup_lvm_metadata": false }, > "_db_comment": "Database file name", "db": "heketi.db", "_refresh_time_monitor_gluster_nodes": "Refresh time in seconds to > monitor Gluster nodes", "refresh_time_monitor_gluster_nodes": 120, "_start_time_monitor_gluster_nodes": "Start time in seconds to monitor > Gluster nodes when the heketi comes up", "start_time_monitor_gluster_nodes": 10, "_loglevel_comment": [ "Set log level. Choices are:", " none, critical, error, warning, info, debug", "Default is warning" ], "loglevel" : "debug", "_auto_create_block_hosting_volume": "Creates Block Hosting volumes > automatically if not found or exsisting volume exhausted", "auto_create_block_hosting_volume": true, "_block_hosting_volume_size": "New block hosting volume will be created > in size mentioned, This is considered only if auto-create is enabled.", "block_hosting_volume_size": 500, "_block_hosting_volume_options": "New block hosting volume will be > created with the following set of options. Removing the group gluster-block > option is NOT recommended. Additional options can be added next to it > separated by a comma.", "block_hosting_volume_options": "group gluster-block" } } [cmdexec] ERROR 2018/11/09 09:44:33 /src/ > github.com/heketi/heketi/executors/cmdexec/volume.go:145: Unable to stop > volume vol_41062d80d73fc9065d6f38344a5033b8: volume stop: > vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume > vol_41062d80d73fc9065d6f38344a5033b8 does not exist [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [negroni] Completed 200 OK in 55.106µs > [cmdexec] ERROR 2018/11/09 09:44:35 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'gluster --mode=script volume delete > vol_41062d80d73fc9065d6f38344a5033b8'] on glust-0:22: Err[Process exited > with status 1]: Stdout []: Stderr [volume delete: > vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume > vol_41062d80d73fc9065d6f38344a5033b8 does not exist > ] > [cmdexec] ERROR 2018/11/09 09:44:35 /src/ > github.com/heketi/heketi/executors/cmdexec/volume.go:154: Unable to > delete volume vol_41062d80d73fc9065d6f38344a5033b8: volume delete: > vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume > vol_41062d80d73fc9065d6f38344a5033b8 does not exist > [heketi] INFO 2018/11/09 09:44:35 Deleting brick > 140fa407e1dd1bbb5c342eaad1081a3e > [heketi] INFO 2018/11/09 09:44:35 Deleting brick > b498b48679d9cbd3f380f865b95faa76 > [heketi] INFO 2018/11/09 09:44:35 Deleting brick > 4652460cce68f2d8ff6c4c34cb57e047 > [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [negroni] Completed 200 OK in 43.818µs > [cmdexec] ERROR 2018/11/09 09:44:36 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'umount > /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e'] > on glust-2:22: Err[Process exited with status 32]: Stdout []: Stderr > [umount: > /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e: > mountpoint not found > ] > [cmdexec] ERROR 2018/11/09 09:44:36 /src/ > github.com/heketi/heketi/executors/cmdexec/brick.go:198: umount: > /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e: > mountpoint not found > [cmdexec] ERROR 2018/11/09 09:44:36 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'umount > /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76'] > on glust-1:22: Err[Process exited with status 32]: Stdout []: Stderr > [umount: > /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76: > mountpoint not found > ] > [cmdexec] ERROR 2018/11/09 09:44:36 /src/ > github.com/heketi/heketi/executors/cmdexec/brick.go:198: umount: > /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76: > mountpoint not found > [cmdexec] ERROR 2018/11/09 09:44:36 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'umount > /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047'] > on glust-0:22: Err[Process exited with status 32]: Stdout []: Stderr > [umount: > /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047: > mountpoint not found > ] > [cmdexec] ERROR 2018/11/09 09:44:36 /src/ > github.com/heketi/heketi/executors/cmdexec/brick.go:198: umount: > /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047: > mountpoint not found > [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [negroni] Completed 200 OK in 50.819µs > [cmdexec] DEBUG 2018/11/09 09:44:37 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-2:22 > Command: /bin/bash -c 'mount' > Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) > proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) > udev on /dev type devtmpfs > (rw,nosuid,relatime,size=8206656k,nr_inodes=2051664,mode=755) > devpts on /dev/pts type devpts > (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) > tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755) > /dev/mapper/system-root on / type ext4 > (rw,relatime,discard,errors=remount-ro,data=ordered) > securityfs on /sys/kernel/security type securityfs > (rw,nosuid,nodev,noexec,relatime) > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) > tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) > tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) > cgroup on /sys/fs/cgroup/systemd type cgroup > (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd) > pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) > cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup > (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) > cgroup on /sys/fs/cgroup/cpuset type cgroup > (rw,nosuid,nodev,noexec,relatime,cpuset) > cgroup on /sys/fs/cgroup/perf_event type cgroup > (rw,nosuid,nodev,noexec,relatime,perf_event) > cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup > (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) > cgroup on /sys/fs/cgroup/freezer type cgroup > (rw,nosuid,nodev,noexec,relatime,freezer) > cgroup on /sys/fs/cgroup/blkio type cgroup > (rw,nosuid,nodev,noexec,relatime,blkio) > cgroup on /sys/fs/cgroup/memory type cgroup > (rw,nosuid,nodev,noexec,relatime,memory) > cgroup on /sys/fs/cgroup/pids type cgroup > (rw,nosuid,nodev,noexec,relatime,pids) > cgroup on /sys/fs/cgroup/devices type cgroup > (rw,nosuid,nodev,noexec,relatime,devices) > systemd-1 on /proc/sys/fs/binfmt_misc type autofs > (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11743) > debugfs on /sys/kernel/debug type debugfs (rw,relatime) > mqueue on /dev/mqueue type mqueue (rw,relatime) > hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) > /dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered) > fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) > [cmdexec] WARNING 2018/11/09 09:44:37 brick path > [/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e] > not mounted, assuming deleted > [cmdexec] DEBUG 2018/11/09 09:44:37 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-1:22 > Command: /bin/bash -c 'mount' > Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) > proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) > udev on /dev type devtmpfs > (rw,nosuid,relatime,size=8206664k,nr_inodes=2051666,mode=755) > devpts on /dev/pts type devpts > (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) > tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755) > /dev/mapper/system-root on / type ext4 > (rw,relatime,discard,errors=remount-ro,data=ordered) > securityfs on /sys/kernel/security type securityfs > (rw,nosuid,nodev,noexec,relatime) > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) > tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) > tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) > cgroup on /sys/fs/cgroup/systemd type cgroup > (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd) > pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) > cgroup on /sys/fs/cgroup/cpuset type cgroup > (rw,nosuid,nodev,noexec,relatime,cpuset) > cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup > (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) > cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup > (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) > cgroup on /sys/fs/cgroup/memory type cgroup > (rw,nosuid,nodev,noexec,relatime,memory) > cgroup on /sys/fs/cgroup/perf_event type cgroup > (rw,nosuid,nodev,noexec,relatime,perf_event) > cgroup on /sys/fs/cgroup/freezer type cgroup > (rw,nosuid,nodev,noexec,relatime,freezer) > cgroup on /sys/fs/cgroup/blkio type cgroup > (rw,nosuid,nodev,noexec,relatime,blkio) > cgroup on /sys/fs/cgroup/pids type cgroup > (rw,nosuid,nodev,noexec,relatime,pids) > cgroup on /sys/fs/cgroup/devices type cgroup > (rw,nosuid,nodev,noexec,relatime,devices) > systemd-1 on /proc/sys/fs/binfmt_misc type autofs > (rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=861) > debugfs on /sys/kernel/debug type debugfs (rw,relatime) > mqueue on /dev/mqueue type mqueue (rw,relatime) > hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) > /dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered) > fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) > [cmdexec] WARNING 2018/11/09 09:44:37 brick path > [/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76] > not mounted, assuming deleted > [cmdexec] DEBUG 2018/11/09 09:44:37 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-0:22 > Command: /bin/bash -c 'mount' > Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) > proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) > udev on /dev type devtmpfs > (rw,nosuid,relatime,size=8206664k,nr_inodes=2051666,mode=755) > devpts on /dev/pts type devpts > (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) > tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755) > /dev/mapper/system-root on / type ext4 > (rw,relatime,discard,errors=remount-ro,data=ordered) > securityfs on /sys/kernel/security type securityfs > (rw,nosuid,nodev,noexec,relatime) > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) > tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) > tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) > cgroup on /sys/fs/cgroup/systemd type cgroup > (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd) > pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) > cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup > (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) > cgroup on /sys/fs/cgroup/cpuset type cgroup > (rw,nosuid,nodev,noexec,relatime,cpuset) > cgroup on /sys/fs/cgroup/perf_event type cgroup > (rw,nosuid,nodev,noexec,relatime,perf_event) > cgroup on /sys/fs/cgroup/memory type cgroup > (rw,nosuid,nodev,noexec,relatime,memory) > cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup > (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) > cgroup on /sys/fs/cgroup/pids type cgroup > (rw,nosuid,nodev,noexec,relatime,pids) > cgroup on /sys/fs/cgroup/blkio type cgroup > (rw,nosuid,nodev,noexec,relatime,blkio) > cgroup on /sys/fs/cgroup/devices type cgroup > (rw,nosuid,nodev,noexec,relatime,devices) > cgroup on /sys/fs/cgroup/freezer type cgroup > (rw,nosuid,nodev,noexec,relatime,freezer) > systemd-1 on /proc/sys/fs/binfmt_misc type autofs > (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1553) > mqueue on /dev/mqueue type mqueue (rw,relatime) > hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) > debugfs on /sys/kernel/debug type debugfs (rw,relatime) > /dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered) > fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) > [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [cmdexec] WARNING 2018/11/09 09:44:37 brick path > [/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047] > not mounted, assuming deleted > [negroni] Completed 200 OK in 16.726183ms > [cmdexec] DEBUG 2018/11/09 09:44:38 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-2:22 > Command: /bin/bash -c 'sed -i.save > "/brick_140fa407e1dd1bbb5c342eaad1081a3e/d" /etc/fstab' > Result: > [cmdexec] DEBUG 2018/11/09 09:44:38 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-1:22 > Command: /bin/bash -c 'sed -i.save > "/brick_b498b48679d9cbd3f380f865b95faa76/d" /etc/fstab' > Result: > [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [negroni] Completed 200 OK in 50.054µs > [cmdexec] DEBUG 2018/11/09 09:44:38 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-0:22 > Command: /bin/bash -c 'sed -i.save > "/brick_4652460cce68f2d8ff6c4c34cb57e047/d" /etc/fstab' > Result: > [cmdexec] ERROR 2018/11/09 09:44:39 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvremove --autobackup=n -f > vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76'] > on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76" > ] > [cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv: > vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76 > [cmdexec] ERROR 2018/11/09 09:44:39 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvremove --autobackup=n -f > vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e'] > on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e" > ] > [cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv: > vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e > [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [negroni] Completed 200 OK in 47.799µs > [cmdexec] ERROR 2018/11/09 09:44:39 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvremove --autobackup=n -f > vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047'] > on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047" > ] > [cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv: > vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047 > [cmdexec] ERROR 2018/11/09 09:44:40 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvs --noheadings --options=thin_count > vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76'] > on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76" > ] > [cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin > pool: > vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76 > [cmdexec] ERROR 2018/11/09 09:44:40 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvs --noheadings --options=thin_count > vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e'] > on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e" > ] > [cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin > pool: > vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e > [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [negroni] Completed 200 OK in 57.345µs > [cmdexec] ERROR 2018/11/09 09:44:40 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvs --noheadings --options=thin_count > vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73'] > on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73" > ] > [cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin > pool: > vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73 > [cmdexec] ERROR 2018/11/09 09:44:41 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvremove --autobackup=n -f > vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76'] > on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76" > ] > [cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool: > vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76 > [cmdexec] ERROR 2018/11/09 09:44:41 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvremove --autobackup=n -f > vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e'] > on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e" > ] > [cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool: > vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e > [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [negroni] Completed 200 OK in 67.474µs > [cmdexec] ERROR 2018/11/09 09:44:41 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'lvremove --autobackup=n -f > vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73'] > on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [ > Failed to find logical volume > "vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73" > ] > [cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool: > vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73 > [cmdexec] ERROR 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'rmdir > /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76'] > on glust-1:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir: > failed to remove > '/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76': > No such file or directory > ] > [cmdexec] ERROR 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/executors/cmdexec/brick.go:279: rmdir: failed to > remove > '/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76': > No such file or directory > [cmdexec] ERROR 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'rmdir > /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e'] > on glust-2:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir: > failed to remove > '/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e': > No such file or directory > ] > [cmdexec] ERROR 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/executors/cmdexec/brick.go:279: rmdir: failed to > remove > '/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e': > No such file or directory > [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d > [negroni] Completed 200 OK in 48.399µs > [cmdexec] ERROR 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command > [/bin/bash -c 'rmdir > /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047'] > on glust-0:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir: > failed to remove > '/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047': > No such file or directory > ] > [cmdexec] ERROR 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/executors/cmdexec/brick.go:279: rmdir: failed to > remove > '/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047': > No such file or directory > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected > space needed for amount=10485760 snapFactor=1 : 10539008 > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected > space needed for amount=10485760 snapFactor=1 : 10539008 > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected > space needed for amount=10485760 snapFactor=1 : 10539008 > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry.go:580: Using the > following clusters: [1233163d3b76a6538a34d9b27a855ead] > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: > brick_size = 10485760 > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets > = 1 > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: > num_bricks = 3 > [heketi] INFO 2018/11/09 09:44:42 Allocating brick set #0 > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3 > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected > space needed for amount=10485760 snapFactor=1 : 10539008 > [heketi] DEBUG 2018/11/09 09:44:42 /src/ > github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device > d4b4ff32a56f249dcff9b4a88eef85f2[1073606656] > required size [10539008] ? > panic: CHECK: > func (github.com/heketi/heketi/pkg/utils.IdSource.ReadUUID) 0x19e0c2b > File /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/pkg/utils/uuid.go:34 > Info: [13 16] > goroutine 863 [running]: > github.com/heketi/heketi/vendor/github.com/lpabon/godbc.dbc_panic(0x24aeb24, > 0x5, 0x0, 0xc4207fd7a0, 0x2, 0x2) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/vendor/github.com/lpabon/godbc/godbc.go:85 +0x40c > github.com/heketi/heketi/vendor/github.com/lpabon/godbc.Check(0x224d900, > 0xc4207fd7a0, 0x2, 0x2) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/vendor/github.com/lpabon/godbc/godbc.go:123 +0x5d > github.com/heketi/heketi/pkg/utils.IdSource.ReadUUID(0x2f3d1e0, > 0xc4200741b0, 0x1, 0xc4207e3760) > /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/utils/uuid.go:34 > +0x19c > github.com/heketi/heketi/pkg/utils.GenUUID(0x23fab80, 0xc4207e3760) > /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/utils/uuid.go:42 > +0x39 > github.com/heketi/heketi/apps/glusterfs.NewBrickEntry(0xa00000, 0xa00000, > 0xd000, 0xc4207bdd00, 0x20, 0xc4207bdd20, 0x20, 0x0, 0xc42028c760, 0x20, > ...) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/brick_entry.go:64 +0x119 > > github.com/heketi/heketi/apps/glusterfs.(*DeviceEntry).NewBrickEntry(0xc4203ffd00, > 0xa00000, 0x3ff0000000000000, 0x0, 0xc42028c760, 0x20, 0xc4207a9100) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/device_entry.go:379 +0x239 > github.com/heketi/heketi/apps/glusterfs.tryAllocateBrickOnDevice(0x2f59a60, > 0xc42079ef40, 0x0, 0xc4203ffd00, 0xc4207fd760, 0x7) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:201 +0x1d2 > github.com/heketi/heketi/apps/glusterfs.findDeviceAndBrickForSet(0x2f59a60, > 0xc42079ef40, 0xc4207a9350, 0x0, 0xc4205b25a0, 0xc4207fd760, 0xc42001c000, > 0x6, 0x6, 0xc4205b2600) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:228 +0xd4 > github.com/heketi/heketi/apps/glusterfs.populateBrickSet(0x2f59a60, > 0xc42079ef40, 0xc4207a9350, 0x0, 0xc4205b25a0, 0xc4207bd180, 0x20, 0x0, > 0x0, 0x0, ...) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:253 +0x2ce > > github.com/heketi/heketi/apps/glusterfs.(*StandardBrickPlacer).PlaceAll(0x2ff8400, > 0x2f4bfe0, 0xc420821650, 0x2f59a60, 0xc42079ef40, 0x0, 0x0, 0x0, 0x0) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:463 +0x336 > github.com/heketi/heketi/apps/glusterfs.allocateBricks.func1(0xc4208447e0, > 0x1, 0xc420821620) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:286 +0x1a5 > github.com/heketi/heketi/pkg/db.(*TxWrap).View(0xc420304e80, > 0xc420821620, 0xc4207a94f8, 0x11f6a4a) > /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/db/wrap.go:78 +0x33 > github.com/heketi/heketi/apps/glusterfs.allocateBricks(0x2f3d7e0, > 0xc420304e80, 0xc4207bcdc0, 0x20, 0xc4202d0580, 0x1, 0xa00000, 0x2fd4b40, > 0x58, 0xa) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:282 +0x108 > > github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricks.func2(0xc4208447e0, > 0x23c56e0, 0xc4207a9501) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:468 +0xf0 > github.com/heketi/heketi/pkg/db.(*TxWrap).Update(0xc420304e10, > 0xc4206a4fc0, 0xc4207a96a0, 0xc4202d0580) > /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/db/wrap.go:87 +0x3f > > github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricks(0xc4202d0580, > 0x2f48420, 0xc420304e10, 0xc4207bcdc0, 0x20, 0x1, 0xa00000, 0x3440000, 0x0, > 0x2337d20, ...) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:466 > +0x14e > > github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricksInCluster(0xc4202d0580, > 0x2f48420, 0xc420304e10, 0xc4207bcdc0, 0x20, 0xa, 0x140, 0xa6, > 0xc4207a97e8, 0x1029a36, ...) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:54 +0x378 > > github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).tryAllocateBricks(0xc4202d0580, > 0x2f48420, 0xc420304e10, 0xc420304d90, 0x1, 0x1, 0x28, 0x38, 0xc4206a4f80, > 0x36a8288, ...) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry.go:436 +0x183 > > github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).saveCreateVolume.func1(0xc4208447e0, > 0x23b00c0, 0x2f20401) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry.go:605 +0xd1 > github.com/heketi/heketi/pkg/db.(*TxWrap).Update(0xc420304af0, > 0xc4206a4f80, 0xc4207a99e0, 0x19dfe05) > /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/db/wrap.go:87 +0x3f > > github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).saveCreateVolume(0xc4202d0580, > 0x2f48420, 0xc420304af0, 0xc420304d90, 0x1, 0x1, 0x1, 0x1, 0xc420304d90, > 0x1, ...) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry.go:602 +0xfc > > github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).createVolumeComponents(0xc4202d0580, > 0x2f48420, 0xc420304af0, 0xfffffffffffffffe, 0xc420302ae0, 0xc4207a9b40, > 0x1029099, 0xc400000008) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/volume_entry.go:582 +0x301 > > github.com/heketi/heketi/apps/glusterfs.(*VolumeCreateOperation).Build.func1(0xc4208447e0, > 0x2528490, 0xc4208447e0) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/operations.go:146 +0x7f > > github.com/heketi/heketi/vendor/github.com/boltdb/bolt.(*DB).Update(0xc420302960, > 0xc420304ae0, 0x0, 0x0) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/vendor/github.com/boltdb/bolt/db.go:595 +0x9a > > github.com/heketi/heketi/apps/glusterfs.(*VolumeCreateOperation).Build(0xc4208067b0, > 0x2f5e620, 0xc420257800) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/operations.go:144 +0x6f > github.com/heketi/heketi/apps/glusterfs.retryOperation(0x2f5a780, > 0xc4208067b0, 0x2f5e620, 0xc420257800, 0x1, 0x1) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/operations_manage.go:175 +0x217 > github.com/heketi/heketi/apps/glusterfs.AsyncHttpOperation.func1(0x0, > 0x0, 0x0, 0x0) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/apps/glusterfs/operations_manage.go:104 +0x552 > > github.com/heketi/heketi/vendor/github.com/heketi/rest.(*AsyncHttpHandler).handle.func1(0xc420818580, > 0xc42084ac30) > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/vendor/github.com/heketi/rest/asynchttp.go:291 > +0xf6 > created by > github.com/heketi/heketi/vendor/github.com/heketi/rest.(*AsyncHttpHandler).handle > /var/tmp/gtg-wrapper/src/ > github.com/heketi/heketi/vendor/github.com/heketi/rest/asynchttp.go:287 > +0x49
_______________________________________________ heketi-devel mailing list heketi-devel@gluster.org https://lists.gluster.org/mailman/listinfo/heketi-devel