Do you see any errors in /var/log/glusterfs/etc-glusterfs-glusterd.vol.log or vdsm.log when the service is trying to start automatically after the reboot?

Thanks,
Kanagaraj

On 11/24/2014 08:13 PM, Punit Dambiwal wrote:
Hi Kanagaraj,

Yes...once i will start the gluster service and then vdsmd ...the host can connect to cluster...but the question is why it's not started even it has chkconfig enabled...

I have tested it in two host cluster environment...(Centos 6.6 and centos 7.0) on both hypervisior cluster..it's failed to reconnect in to cluster after reboot....

In both the environment glusterd enabled for next boot....but it's failed with the same error....seems it's bug in either gluster or Ovirt ??

Please help me to find the workaround here if can not resolve it...as without this the Host machine can not connect after reboot....that means engine will consider it as down and every time need to manually start the gluster service and vdsmd... ??

Thanks,
Punit

On Mon, Nov 24, 2014 at 10:20 PM, Kanagaraj <kmayi...@redhat.com <mailto:kmayi...@redhat.com>> wrote:

    From vdsm.log "error: Connection failed. Please check if gluster
    daemon is operational."

    Starting glusterd service should fix this issue. 'service glusterd
    start'
    But i am wondering why the glusterd was not started automatically
    after the reboot.

    Thanks,
    Kanagaraj



    On 11/24/2014 07:18 PM, Punit Dambiwal wrote:
    Hi Kanagaraj,

    Please find the attached VDSM logs :-

    ----------------
    Thread-13::DEBUG::2014-11-24
    
21:41:17,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
    Owner.cancelAll requests {}
    Thread-13::DEBUG::2014-11-24
    21:41:17,182::task::993::Storage.TaskManager.Task::(_decref)
    Task=`1691d409-9b27-4585-8281-5ec26154367a`::ref 0 aborting False
    Thread-13::DEBUG::2014-11-24
    21:41:32,393::task::595::Storage.TaskManager.Task::(_updateState)
    Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state
    init -> state preparing
    Thread-13::INFO::2014-11-24
    21:41:32,393::logUtils::44::dispatcher::(wrapper) Run and
    protect: repoStats(options=None)
    Thread-13::INFO::2014-11-24
    21:41:32,393::logUtils::47::dispatcher::(wrapper) Run and
    protect: repoStats, Return response: {}
    Thread-13::DEBUG::2014-11-24
    21:41:32,393::task::1191::Storage.TaskManager.Task::(prepare)
    Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::finished: {}
    Thread-13::DEBUG::2014-11-24
    21:41:32,394::task::595::Storage.TaskManager.Task::(_updateState)
    Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state
    preparing -> state finished
    Thread-13::DEBUG::2014-11-24
    
21:41:32,394::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
    Owner.releaseAll requests {} resources {}
    Thread-13::DEBUG::2014-11-24
    
21:41:32,394::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
    Owner.cancelAll requests {}
    Thread-13::DEBUG::2014-11-24
    21:41:32,394::task::993::Storage.TaskManager.Task::(_decref)
    Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::ref 0 aborting False
    Thread-13::DEBUG::2014-11-24
    21:41:41,550::BindingXMLRPC::1132::vds::(wrapper) client
    [10.10.10.2]::call getCapabilities with () {}
    Thread-13::DEBUG::2014-11-24
    21:41:41,553::utils::738::root::(execCmd) /sbin/ip route show to
    0.0.0.0/0 <http://0.0.0.0/0> table all (cwd None)
    Thread-13::DEBUG::2014-11-24
    21:41:41,560::utils::758::root::(execCmd) SUCCESS: <err> = '';
    <rc> = 0
    Thread-13::DEBUG::2014-11-24
    21:41:41,588::caps::728::root::(_getKeyPackages) rpm package
    ('gluster-swift',) not found
    Thread-13::DEBUG::2014-11-24
    21:41:41,592::caps::728::root::(_getKeyPackages) rpm package
    ('gluster-swift-object',) not found
    Thread-13::DEBUG::2014-11-24
    21:41:41,593::caps::728::root::(_getKeyPackages) rpm package
    ('gluster-swift-plugin',) not found
    Thread-13::DEBUG::2014-11-24
    21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
    ('gluster-swift-account',) not found
    Thread-13::DEBUG::2014-11-24
    21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
    ('gluster-swift-proxy',) not found
    Thread-13::DEBUG::2014-11-24
    21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
    ('gluster-swift-doc',) not found
    Thread-13::DEBUG::2014-11-24
    21:41:41,599::caps::728::root::(_getKeyPackages) rpm package
    ('gluster-swift-container',) not found
    Thread-13::DEBUG::2014-11-24
    21:41:41,599::caps::728::root::(_getKeyPackages) rpm package
    ('glusterfs-geo-replication',) not found
    Thread-13::DEBUG::2014-11-24 21:41:41,600::caps::646::root::(get)
    VirtioRNG DISABLED: libvirt version 0.10.2-29.el6_5.9 required >=
    0.10.2-31
    Thread-13::DEBUG::2014-11-24
    21:41:41,603::BindingXMLRPC::1139::vds::(wrapper) return
    getCapabilities with {'status': {'message': 'Done', 'code': 0},
    'info': {'HBAInventory': {'iSCSI': [{'InitiatorName':
    'iqn.1994-05.com.redhat:32151ce183c8'}], 'FC': []}, 'packages2':
    {'kernel': {'release': '431.el6.x86_64', 'buildtime':
    1385061309.0, 'version': '2.6.32'}, 'glusterfs-rdma': {'release':
    '1.el6', 'buildtime': 1403622628L, 'version': '3.5.1'},
    'glusterfs-fuse': {'release': '1.el6', 'buildtime': 1403622628L,
    'version': '3.5.1'}, 'spice-server': {'release': '6.el6_5.2',
    'buildtime': 1402324637L, 'version': '0.12.4'}, 'vdsm':
    {'release': '1.gitdb83943.el6', 'buildtime': 1412784567L,
    'version': '4.16.7'}, 'qemu-kvm': {'release': '2.415.el6_5.10',
    'buildtime': 1402435700L, 'version': '0.12.1.2'}, 'qemu-img':
    {'release': '2.415.el6_5.10', 'buildtime': 1402435700L,
    'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.9',
    'buildtime': 1402404612L, 'version': '0.10.2'}, 'glusterfs':
    {'release': '1.el6', 'buildtime': 1403622628L, 'version':
    '3.5.1'}, 'mom': {'release': '2.el6', 'buildtime': 1403794344L,
    'version': '0.4.1'}, 'glusterfs-server': {'release': '1.el6',
    'buildtime': 1403622628L, 'version': '3.5.1'}},
    'numaNodeDistance': {'1': [20, 10], '0': [10, 20]}, 'cpuModel':
    'Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz', 'liveMerge':
    'false', 'hooks': {}, 'cpuSockets': '2', 'vmTypes': ['kvm'],
    'selinux': {'mode': '1'}, 'kdumpStatus': 0, 'supportedProtocols':
    ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0.10',
    'addr': '43.252.176.16', 'bridged': False, 'ipv6addrs':
    ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu': '1500', 'bootproto4':
    'none', 'netmask': '255.255.255.0', 'ipv4addrs':
    ['43.252.176.16/24' <http://43.252.176.16/24%27>], 'interface':
    u'bond0.10', 'ipv6gateway': '::', 'gateway': '43.25.17.1'},
    'Internal': {'iface': 'Internal', 'addr': '', 'cfg': {'DEFROUTE':
    'no', 'HOTPLUG': 'no', 'MTU': '9000', 'DELAY': '0',
    'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off',
    'DEVICE': 'Internal', 'TYPE': 'Bridge', 'ONBOOT': 'no'},
    'bridged': True, 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'],
    'gateway': '', 'bootproto4': 'none', 'netmask': '', 'stp': 'off',
    'ipv4addrs': [], 'mtu': '9000', 'ipv6gateway': '::', 'ports':
    ['bond1.100']}, 'storage': {'iface': u'bond1', 'addr':
    '10.10.10.6', 'bridged': False, 'ipv6addrs':
    ['fe80::210:18ff:fecd:daac/64'], 'mtu': '9000', 'bootproto4':
    'none', 'netmask': '255.255.255.0', 'ipv4addrs': ['10.10.10.6/24'
    <http://10.10.10.6/24%27>], 'interface': u'bond1', 'ipv6gateway':
    '::', 'gateway': ''}, 'VMNetwork': {'iface': 'VMNetwork', 'addr':
    '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU': '1500',
    'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP':
    'off', 'DEVICE': 'VMNetwork', 'TYPE': 'Bridge', 'ONBOOT': 'no'},
    'bridged': True, 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'],
    'gateway': '', 'bootproto4': 'none', 'netmask': '', 'stp': 'off',
    'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': '::', 'ports':
    ['bond0.36']}}, 'bridges': {'Internal': {'addr': '', 'cfg':
    {'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU': '9000', 'DELAY': '0',
    'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off',
    'DEVICE': 'Internal', 'TYPE': 'Bridge', 'ONBOOT': 'no'},
    'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'], 'mtu': '9000',
    'netmask': '', 'stp': 'off', 'ipv4addrs': [], 'ipv6gateway':
    '::', 'gateway': '', 'opts': {'topology_change_detected': '0',
    'multicast_last_member_count': '2', 'hash_elasticity': '4',
    'multicast_query_response_interval': '999', 'multicast_snooping':
    '1', 'multicast_startup_query_interval': '3124', 'hello_timer':
    '31', 'multicast_querier_interval': '25496', 'max_age': '1999',
    'hash_max': '512', 'stp_state': '0', 'root_id':
    '8000.001018cddaac', 'priority': '32768',
    'multicast_membership_interval': '25996', 'root_path_cost': '0',
    'root_port': '0', 'multicast_querier': '0',
    'multicast_startup_query_count': '2', 'hello_time': '199',
    'topology_change': '0', 'bridge_id': '8000.001018cddaac',
    'topology_change_timer': '0', 'ageing_time': '29995', 'gc_timer':
    '31', 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0',
    'multicast_query_interval': '12498',
    'multicast_last_member_interval': '99', 'multicast_router': '1',
    'forward_delay': '0'}, 'ports': ['bond1.100']}, 'VMNetwork':
    {'addr': '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU':
    '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
    'STP': 'off', 'DEVICE': 'VMNetwork', 'TYPE': 'Bridge', 'ONBOOT':
    'no'}, 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu':
    '1500', 'netmask': '', 'stp': 'off', 'ipv4addrs': [],
    'ipv6gateway': '::', 'gateway': '', 'opts':
    {'topology_change_detected': '0', 'multicast_last_member_count':
    '2', 'hash_elasticity': '4', 'multicast_query_response_interval':
    '999', 'multicast_snooping': '1',
    'multicast_startup_query_interval': '3124', 'hello_timer': '131',
    'multicast_querier_interval': '25496', 'max_age': '1999',
    'hash_max': '512', 'stp_state': '0', 'root_id':
    '8000.60eb6920b46c', 'priority': '32768',
    'multicast_membership_interval': '25996', 'root_path_cost': '0',
    'root_port': '0', 'multicast_querier': '0',
    'multicast_startup_query_count': '2', 'hello_time': '199',
    'topology_change': '0', 'bridge_id': '8000.60eb6920b46c',
    'topology_change_timer': '0', 'ageing_time': '29995', 'gc_timer':
    '31', 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0',
    'multicast_query_interval': '12498',
    'multicast_last_member_interval': '99', 'multicast_router': '1',
    'forward_delay': '0'}, 'ports': ['bond0.36']}}, 'uuid':
    '44454C4C-4C00-1057-8053-B7C04F504E31', 'lastClientIface':
    'bond1', 'nics': {'eth3': {'permhwaddr': '00:10:18:cd:da:ae',
    'addr': '', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU':
    '9000', 'HWADDR': '00:10:18:cd:da:ae', 'MASTER': 'bond1',
    'DEVICE': 'eth3', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'mtu':
    '9000', 'netmask': '', 'ipv4addrs': [], 'hwaddr':
    '00:10:18:cd:da:ac', 'speed': 1000}, 'eth2': {'permhwaddr':
    '00:10:18:cd:da:ac', 'addr': '', 'cfg': {'SLAVE': 'yes',
    'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR':
    '00:10:18:cd:da:ac', 'MASTER': 'bond1', 'DEVICE': 'eth2',
    'ONBOOT': 'no'}, 'ipv6addrs': [], 'mtu': '9000', 'netmask': '',
    'ipv4addrs': [], 'hwaddr': '00:10:18:cd:da:ac', 'speed': 1000},
    'eth1': {'permhwaddr': '60:eb:69:20:b4:6d', 'addr': '', 'cfg':
    {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR':
    '60:eb:69:20:b4:6d', 'MASTER': 'bond0', 'DEVICE': 'eth1',
    'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '',
    'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c', 'speed': 1000},
    'eth0': {'permhwaddr': '60:eb:69:20:b4:6c', 'addr': '', 'cfg':
    {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR':
    '60:eb:69:20:b4:6c', 'MASTER': 'bond0', 'DEVICE': 'eth0',
    'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '',
    'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c', 'speed': 1000}},
    'software_revision': '1', 'clusterLevels': ['3.0', '3.1', '3.2',
    '3.3', '3.4', '3.5'], 'cpuFlags':
    
u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270',
    'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:32151ce183c8',
    'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1',
    '3.2', '3.3', '3.4', '3.5'], 'autoNumaBalancing': 2,
    'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg':
    {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr':
    '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {'HOTPLUG':
    'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BONDING_OPTS':
    'mode=4 miimon=100', 'DEVICE': 'bond0', 'ONBOOT': 'yes'},
    'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu': '1500',
    'netmask': '', 'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c',
    'slaves': ['eth0', 'eth1'], 'opts': {'miimon': '100', 'mode':
    '4'}}, 'bond1': {'addr': '10.10.10.6', 'cfg': {'DEFROUTE': 'no',
    'IPADDR': '10.10.10.6', 'HOTPLUG': 'no', 'MTU': '9000',
    'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO':
    'none', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond1',
    'ONBOOT': 'no'}, 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'],
    'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4addrs':
    ['10.10.10.6/24' <http://10.10.10.6/24%27>], 'hwaddr':
    '00:10:18:cd:da:ac', 'slaves': ['eth2', 'eth3'], 'opts':
    {'miimon': '100', 'mode': '4'}}, 'bond2': {'addr': '', 'cfg': {},
    'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr':
    '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu':
    '1500', 'netmask': '', 'slaves': [], 'hwaddr':
    '00:00:00:00:00:00'}}, 'software_version': '4.16', 'memSize':
    '24019', 'cpuSpeed': '2667.000', 'numaNodes': {u'1':
    {'totalMemory': '12288', 'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20,
    21, 22, 23]}, u'0': {'totalMemory': '12278', 'cpus': [0, 1, 2, 3,
    4, 5, 12, 13, 14, 15, 16, 17]}}, 'version_name': 'Snow Man',
    'vlans': {'bond0.10': {'iface': 'bond0', 'addr': '43.25.17.16',
    'cfg': {'DEFROUTE': 'yes', 'VLAN': 'yes', 'IPADDR':
    '43.25.17.16', 'HOTPLUG': 'no', 'GATEWAY': '43.25.17.1',
    'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO':
    'none', 'DEVICE': 'bond0.10', 'MTU': '1500', 'ONBOOT': 'yes'},
    'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 10,
    'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4addrs':
    ['43.25.17.16/24'] <http://43.25.17.16/24%27%5D>}, 'bond0.36':
    {'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'VMNetwork',
    'VLAN': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED':
    'no', 'DEVICE': 'bond0.36', 'ONBOOT': 'no'}, 'ipv6addrs':
    ['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 36, 'mtu': '1500',
    'netmask': '', 'ipv4addrs': []}, 'bond1.100': {'iface': 'bond1',
    'addr': '', 'cfg': {'BRIDGE': 'Internal', 'VLAN': 'yes',
    'HOTPLUG': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'DEVICE':
    'bond1.100', 'ONBOOT': 'no'}, 'ipv6addrs':
    ['fe80::210:18ff:fecd:daac/64'], 'vlanid': 100, 'mtu': '9000',
    'netmask': '', 'ipv4addrs': []}}, 'cpuCores': '12', 'kvmEnabled':
    'true', 'guestOverhead': '65', 'cpuThreads': '24',
    'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0',
    u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0',
    u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem':
    {'release': '5.el6.centos.11.1', 'version': '6', 'name': 'RHEL'},
    'lastClient': '10.10.10.2'}}
    Thread-13::DEBUG::2014-11-24
    21:41:41,620::BindingXMLRPC::1132::vds::(wrapper) client
    [10.10.10.2]::call getHardwareInfo with () {}
    Thread-13::DEBUG::2014-11-24
    21:41:41,621::BindingXMLRPC::1139::vds::(wrapper) return
    getHardwareInfo with {'status': {'message': 'Done', 'code': 0},
    'info': {'systemProductName': 'CS24-TY', 'systemSerialNumber':
    '7LWSPN1', 'systemFamily': 'Server', 'systemVersion': 'A00',
    'systemUUID': '44454c4c-4c00-1057-8053-b7c04f504e31',
    'systemManufacturer': 'Dell'}}
    Thread-13::DEBUG::2014-11-24
    21:41:41,733::BindingXMLRPC::1132::vds::(wrapper) client
    [10.10.10.2]::call hostsList with () {} flowID [222e8036]
    Thread-13::ERROR::2014-11-24
    21:41:44,753::BindingXMLRPC::1148::vds::(wrapper) vdsm exception
    occured
    Traceback (most recent call last):
      File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1135, in wrapper
        res = f(*args, **kwargs)
      File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
        rv = func(*args, **kwargs)
      File "/usr/share/vdsm/gluster/api.py", line 251, in hostsList
        return {'hosts': self.svdsmProxy.glusterPeerStatus()}
      File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
        return callMethod()
      File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
        **kwargs)
      File "<string>", line 2, in glusterPeerStatus
      File "/usr/lib64/python2.6/multiprocessing/managers.py", line
    740, in _callmethod
        raise convert_to_error(kind, result)
    GlusterCmdExecFailedException: Command execution failed
    error: Connection failed. Please check if gluster daemon is
    operational.
    return code: 1
    Thread-13::DEBUG::2014-11-24
    21:41:50,949::task::595::Storage.TaskManager.Task::(_updateState)
    Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state
    init -> state preparing
    Thread-13::INFO::2014-11-24
    21:41:50,950::logUtils::44::dispatcher::(wrapper) Run and
    protect: repoStats(options=None)
    Thread-13::INFO::2014-11-24
    21:41:50,950::logUtils::47::dispatcher::(wrapper) Run and
    protect: repoStats, Return response: {}
    Thread-13::DEBUG::2014-11-24
    21:41:50,950::task::1191::Storage.TaskManager.Task::(prepare)
    Task=`c9042986-c978-4b08-adb2-616f5299e115`::finished: {}
    Thread-13::DEBUG::2014-11-24
    21:41:50,950::task::595::Storage.TaskManager.Task::(_updateState)
    Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state
    preparing -> state finished
    Thread-13::DEBUG::2014-11-24
    
21:41:50,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
    Owner.releaseAll requests {} resources {}
    Thread-13::DEBUG::2014-11-24
    
21:41:50,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
    Owner.cancelAll requests {}
    Thread-13::DEBUG::2014-11-24
    21:41:50,951::task::993::Storage.TaskManager.Task::(_decref)
    Task=`c9042986-c978-4b08-adb2-616f5299e115`::ref 0 aborting False
    -------------------------------

    [root@compute4 ~]# service glusterd status
    glusterd is stopped
    [root@compute4 ~]# chkconfig --list | grep glusterd
    glusterd        0:off   1:off   2:on  3:on    4:on    5:on    6:off
    [root@compute4 ~]#

    Thanks,
    Punit

    On Mon, Nov 24, 2014 at 6:36 PM, Kanagaraj <kmayi...@redhat.com
    <mailto:kmayi...@redhat.com>> wrote:

        Can you send the corresponding error in vdsm.log from the host?

        Also check if glusterd service is running.

        Thanks,
        Kanagaraj


        On 11/24/2014 03:39 PM, Punit Dambiwal wrote:
        Hi,

        After reboot my Hypervisior host can not activate again in
        the cluster and failed with the following error :-

        Gluster command [<UNKNOWN>] failed on server...

        Engine logs :-

        2014-11-24 18:05:28,397 INFO
         [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
        (DefaultQuartzScheduler_Worker-64) START,
        GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
        33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 5f251c90
        2014-11-24 18:05:30,609 INFO
         [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
        (DefaultQuartzScheduler_Worker-64) FINISH,
        GlusterVolumesListVDSCommand, return:
        
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d95203e0},
        log id: 5f251c90
        2014-11-24 18:05:33,768 INFO
         [org.ovirt.engine.core.bll.ActivateVdsCommand]
        (ajp--127.0.0.1-8702-8) [287d570d] Lock Acquired to object
        EngineLock [exclusiveLocks= key:
        0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
        , sharedLocks= ]
        2014-11-24 18:05:33,795 INFO
         [org.ovirt.engine.core.bll.ActivateVdsCommand]
        (org.ovirt.thread.pool-8-thread-45) [287d570d] Running
        command: ActivateVdsCommand internal: false. Entities
        affected :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type:
        VDSAction group MANIPULATE_HOST with role type ADMIN
        2014-11-24 18:05:33,796 INFO
         [org.ovirt.engine.core.bll.ActivateVdsCommand]
        (org.ovirt.thread.pool-8-thread-45) [287d570d] Before
        acquiring lock in order to prevent monitoring for host
        Compute5 from data-center SV_WTC
        2014-11-24 18:05:33,797 INFO
         [org.ovirt.engine.core.bll.ActivateVdsCommand]
        (org.ovirt.thread.pool-8-thread-45) [287d570d] Lock
        acquired, from now a monitoring of host will be skipped for
        host Compute5 from data-center SV_WTC
        2014-11-24 18:05:33,817 INFO
         [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
        (org.ovirt.thread.pool-8-thread-45) [287d570d] START,
        SetVdsStatusVDSCommand(HostName = Compute5, HostId =
        0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=Unassigned,
        nonOperationalReason=NONE, stopSpmFailureLogged=false), log
        id: 1cbc7311
        2014-11-24 18:05:33,820 INFO
         [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
        (org.ovirt.thread.pool-8-thread-45) [287d570d] FINISH,
        SetVdsStatusVDSCommand, log id: 1cbc7311
        2014-11-24 18:05:34,086 INFO
         [org.ovirt.engine.core.bll.ActivateVdsCommand]
        (org.ovirt.thread.pool-8-thread-45) Activate finished. Lock
        released. Monitoring can run now for host Compute5 from
        data-center SV_WTC
        2014-11-24 18:05:34,088 INFO
         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
        (org.ovirt.thread.pool-8-thread-45) Correlation ID:
        287d570d, Job ID: 5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a, Call
        Stack: null, Custom Event ID: -1, Message: Host Compute5 was
        activated by admin.
        2014-11-24 18:05:34,090 INFO
         [org.ovirt.engine.core.bll.ActivateVdsCommand]
        (org.ovirt.thread.pool-8-thread-45) Lock freed to object
        EngineLock [exclusiveLocks= key:
        0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
        , sharedLocks= ]
        2014-11-24 18:05:35,792 INFO
         [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
        (DefaultQuartzScheduler_Worker-55) [3706e836] START,
        GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
        33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 48a0c832
        2014-11-24 18:05:37,064 INFO
         [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
        (DefaultQuartzScheduler_Worker-69) START,
        GetHardwareInfoVDSCommand(HostName = Compute5, HostId =
        0bf6b00f-7947-4411-b55a-cc5eea2b381a,
        vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]),
        log id: 6d560cc2
        2014-11-24 18:05:37,074 INFO
         [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
        (DefaultQuartzScheduler_Worker-69) FINISH,
        GetHardwareInfoVDSCommand, log id: 6d560cc2
        2014-11-24 18:05:37,093 WARN
         [org.ovirt.engine.core.vdsbroker.VdsManager]
        (DefaultQuartzScheduler_Worker-69) Host Compute5 is running
        with disabled SELinux.
        2014-11-24 18:05:37,127 INFO
         [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
        (DefaultQuartzScheduler_Worker-69) [2b4a51cf] Running
        command: HandleVdsCpuFlagsOrClusterChangedCommand internal:
        true. Entities affected :  ID:
        0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
        2014-11-24 18:05:37,147 INFO
         [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
        (DefaultQuartzScheduler_Worker-69) [2b4a51cf] START,
        GlusterServersListVDSCommand(HostName = Compute5, HostId =
        0bf6b00f-7947-4411-b55a-cc5eea2b381a), log id: 4faed87
        2014-11-24 18:05:37,164 INFO
         [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
        (DefaultQuartzScheduler_Worker-69) [2b4a51cf] FINISH,
        GlusterServersListVDSCommand, log id: 4faed87
        2014-11-24 18:05:37,189 INFO
         [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
        (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Running
        command: SetNonOperationalVdsCommand internal: true.
        Entities affected :  ID:
        0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
        2014-11-24 18:05:37,206 INFO
         [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
        (DefaultQuartzScheduler_Worker-69) [4a84c4e5] START,
        SetVdsStatusVDSCommand(HostName = Compute5, HostId =
        0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=NonOperational,
        nonOperationalReason=GLUSTER_COMMAND_FAILED,
        stopSpmFailureLogged=false), log id: fed5617
        2014-11-24 18:05:37,209 INFO
         [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
        (DefaultQuartzScheduler_Worker-69) [4a84c4e5] FINISH,
        SetVdsStatusVDSCommand, log id: fed5617
        2014-11-24 18:05:37,223 ERROR
        [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
        (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation
        ID: 4a84c4e5, Job ID: 4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b,
        Call Stack: null, Custom Event ID: -1, Message: Gluster
        command [<UNKNOWN>] failed on server Compute5.
        2014-11-24 18:05:37,243 INFO
         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
        (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation
        ID: null, Call Stack: null, Custom Event ID: -1, Message:
        Status of host Compute5 was set to NonOperational.
        2014-11-24 18:05:37,272 INFO
         [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
        (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Running
        command: HandleVdsVersionCommand internal: true. Entities
        affected :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
        2014-11-24 18:05:37,274 INFO
         [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
        (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Host
        0bf6b00f-7947-4411-b55a-cc5eea2b381a : Compute5 is already
        in NonOperational status for reason GLUSTER_COMMAND_FAILED.
        SetNonOperationalVds command is skipped.
        2014-11-24 18:05:38,065 INFO
         [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
        (DefaultQuartzScheduler_Worker-55) [3706e836] FINISH,
        GlusterVolumesListVDSCommand, return:
        
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4e72a1b1},
        log id: 48a0c832
        2014-11-24 18:05:43,243 INFO
         [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
        (DefaultQuartzScheduler_Worker-35) START,
        GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
        33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 3ce13ebc
        ^C
        [root@ccr01 ~]#

        Thanks,
        Punit


        _______________________________________________
        Users mailing list
        Users@ovirt.org  <mailto:Users@ovirt.org>
        http://lists.ovirt.org/mailman/listinfo/users





_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Reply via email to