Hi Harish, In your setup, the cassandra and zk db are out of sync. The fq_name_to_uuid table in cassandra doesn’t have entry for “default” system created objects where as the zk db has it. (See the create time in zk db. ctime = Tue Apr 07 04:03:54 UTC 2015)
Can you check whether: 1. reset of cassandra db was done 2. Zk db was migrated from other/old setup 3. Cassandra server ip in contrail-api.conf(cassandra_server_list) Thanks, Prakash From: Harish Kumar <[email protected]<mailto:[email protected]>> Date: Tuesday, April 7, 2015 at 11:35 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [opencontrail-dev] intermittent initialization issues on opencontrail Hello, Sometimes while initializing (first time startup) contrail-api, I see below issues. It seems contrail is trying to create the object which are already exists in zk. I see this is happening for multiple objects one after other. I have single node contrail node using opencontrail 1.2. ## contrail-api trying to create an object in zk, and throw nodeexists error. INFO:api-0:Sending request(xid=1): Create(path=u'/fq-name-to-uuid/global_system_config:default-global-system-config', data='19f9ae74-f233-4ff8-b030-e2582ffab468', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) INFO:api-0:Received error(xid=1) NodeExistsError((), {}) ## The object api is trying to create is already exist in zk. [zk: localhost:2181(CONNECTED) 1] get /fq-name-to-uuid/global_system_config:default-global-system-config 12cf3860-d944-4fbf-8604-6fe6545a12a8 cZxid = 0x7 ctime = Tue Apr 07 04:03:54 UTC 2015 mZxid = 0x7 mtime = Tue Apr 07 04:03:54 UTC 2015 pZxid = 0x7 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 36 numChildren = 0 ## I see below errors in ifmap server, I dont know if that might be the issue 2015-04-07 05:17:03,600 [main] INFO - ChannelAcceptor: Listening on port 8444 for incoming certificate-based authentication connections 2015-04-07 05:17:03,600 [main] INFO - irond is running :-) 2015-04-07 05:17:07,076 [pool-6-thread-2] ERROR - ChannelThread: Receiving request failed 2015-04-07 05:17:07,076 [pool-6-thread-2] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?) 2015-04-07 05:17:07,076 [pool-6-thread-2] ERROR - ChannelThread: Setting channel 127.0.0.1:40616:1 into state 'BROKEN' 2015-04-07 05:17:10,093 [pool-6-thread-2] ERROR - ChannelThread: Receiving request failed 2015-04-07 05:17:10,093 [pool-6-thread-2] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?) 2015-04-07 05:17:10,093 [pool-6-thread-2] ERROR - ChannelThread: Setting channel 127.0.0.1:40745:5 into state 'BROKEN' 2015-04-07 05:17:13,102 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed 2015-04-07 05:17:13,102 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?) 2015-04-07 05:17:13,102 [pool-6-thread-1] ERROR - ChannelThread: Setting channel 127.0.0.1:40898:8 into state 'BROKEN' 2015-04-07 05:17:16,111 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed 2015-04-07 05:17:16,111 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?) 2015-04-07 05:17:16,111 [pool-6-thread-1] ERROR - ChannelThread: Setting channel 127.0.0.1:41064:12 into state 'BROKEN' ## Here is the trace from contrail-api - the type error in the bottom because of INFO:api-0:Connecting to 10.0.0.15:2181<http://10.0.0.15:2181> INFO:api-0:Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=20000, session_id=0, passwd='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) INFO:api-0:Zookeeper connection established, state: CONNECTED INFO:api-0:Connected to ZooKeeper! INFO:api-0:Sending request(xid=1): Create(path=u'/fq-name-to-uuid/global_system_config:default-global-system-config', data='19f9ae74-f233-4ff8-b030-e2582ffab468', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) INFO:api-0:Received error(xid=1) NodeExistsError((), {}) INFO:api-0:Sending request(xid=2): GetData(path=u'/fq-name-to-uuid/global_system_config:default-global-system-config', watcher=None) INFO:api-0:Received response(xid=2): ('12cf3860-d944-4fbf-8604-6fe6545a12a8', ZnodeStat(czxid=7, mzxid=7, ctime=1428379434385, mtime=1428379434385, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=36, numChildren=0, pzxid=7)) <type 'exceptions.TypeError'> Python 2.7.6: /usr/bin/python Tue Apr 7 04:39:31 2015 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/bin/contrail-api in <module>() 6 7 if __name__ == '__main__': 8 sys.exit( 9 load_entry_point('vnc-cfg-api-server==0.1dev', 'console_scripts', 'contrail-api')() 10 ) load_entry_point = <function load_entry_point> /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in server_main(args_str=None) 1570 cgitb.enable(format='text') 1571 1572 main() 1573 #server_main 1574 global main = <function main> /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in main(args_str=None) 1529 server = None 1530 def main(args_str=None): 1531 vnc_api_server = VncApiServer(args_str) 1532 # set module var for uses with import e.g unit test 1533 global server vnc_api_server undefined global VncApiServer = <class 'vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer'> args_str = None /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in __init__(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, args_str='--conf_file /etc/contrail/contrail-api.conf') 420 else: 421 self._db_connect(self._args.reset_config) 422 self._db_init_entries() 423 424 # Cpuinfo interface self = <vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object> self._db_init_entries = <bound method VncApiServer._db_init_entries of <...i_server.vnc_cfg_api_server.VncApiServer object>> /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in _db_init_entries(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>) 1072 glb_sys_cfg = self._create_singleton_entry( 1073 GlobalSystemConfig(autonomous_system=64512, 1074 config_version=CONFIG_VERSION)) 1075 def_domain = self._create_singleton_entry(Domain()) 1076 ip_fab_vn = self._create_singleton_entry( config_version undefined global CONFIG_VERSION = '1.0' /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in _create_singleton_entry(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, singleton_obj=<vnc_cfg_api_server.gen.resource_common.GlobalSystemConfig object>) 1123 (ok, result) = self._db_conn.dbe_alloc(obj_type, obj_dict) 1124 obj_ids = result 1125 self._db_conn.dbe_create(obj_type, obj_ids, obj_dict) 1126 method = '_%s_create_default_children' % (method_name) 1127 def_children_method = getattr(self, method) self = <vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object> self._db_conn = <vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object> self._db_conn.dbe_create = <bound method VncDbClient.wrapper2 of <vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>> obj_type = 'global-system-config' obj_ids = 'FQ Name: /fq-name-to-uuid/global_system_config:d...ady with ID: 12cf3860-d944-4fbf-8604-6fe6545a12a8' obj_dict = {'autonomous_system': 64512, 'config_version': '1.0', 'fq_name': [u'default-global-system-config'], 'id_perms': {u'created': 0, u'description': None, u'enable': True, u'last_modified': 0, u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': None}, 'uuid': None} /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in wrapper2(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>, obj_type='global-system-config', obj_ids='FQ Name: /fq-name-to-uuid/global_system_config:d...ady with ID: 12cf3860-d944-4fbf-8604-6fe6545a12a8', obj_dict={'autonomous_system': 64512, 'config_version': '1.0', 'fq_name': [u'default-global-system-config'], 'id_perms': {u'created': 0, u'description': None, u'enable': True, u'last_modified': 0, u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': None}, 'uuid': None}) 1699 obj_ids, obj_dict) 1700 try: 1701 ret = func(self, obj_type, obj_ids, obj_dict) 1702 trace_msg(trace, 'DBRequestTraceBuf', 1703 self._sandesh) ret undefined func = <function dbe_create> self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object> obj_type = 'global-system-config' obj_ids = 'FQ Name: /fq-name-to-uuid/global_system_config:d...ady with ID: 12cf3860-d944-4fbf-8604-6fe6545a12a8' obj_dict = {'autonomous_system': 64512, 'config_version': '1.0', 'fq_name': [u'default-global-system-config'], 'id_perms': {u'created': 0, u'description': None, u'enable': True, u'last_modified': 0, u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': None}, 'uuid': None} /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py in dbe_create(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>, obj_type='global-system-config', obj_ids='FQ Name: /fq-name-to-uuid/global_system_config:d...ady with ID: 12cf3860-d944-4fbf-8604-6fe6545a12a8', obj_dict={'autonomous_system': 64512, 'config_version': '1.0', 'fq_name': [u'default-global-system-config'], 'id_perms': {u'created': 0, u'description': None, u'enable': True, u'last_modified': 0, u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': None}, 'uuid': None}) 1717 method = getattr( 1718 self._cassandra_db, "_cassandra_%s_create" % (method_name)) 1719 (ok, result) = method(obj_ids, obj_dict) 1720 1721 # publish to ifmap via msgbus ok undefined result undefined method = <bound method VncCassandraClient._cassandra_glob..._server.vnc_cfg_ifmap.VncCassandraClient object>> obj_ids = 'FQ Name: /fq-name-to-uuid/global_system_config:d...ady with ID: 12cf3860-d944-4fbf-8604-6fe6545a12a8' obj_dict = {'autonomous_system': 64512, 'config_version': '1.0', 'fq_name': [u'default-global-system-config'], 'id_perms': {u'created': 0, u'description': None, u'enable': True, u'last_modified': 0, u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': None}, 'uuid': None} /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_cassandra_client_gen.py in _cassandra_global_system_config_create(self=<vnc_cfg_api_server.vnc_cfg_ifmap.VncCassandraClient object>, obj_ids='FQ Name: /fq-name-to-uuid/global_system_config:d...ady with ID: 12cf3860-d944-4fbf-8604-6fe6545a12a8', obj_dict={'autonomous_system': 64512, 'config_version': '1.0', 'fq_name': [u'default-global-system-config'], 'id_perms': {u'created': 0, u'description': None, u'enable': True, u'last_modified': 0, u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': None}, 'uuid': None}) 4031 field = obj_dict.get('autonomous_system', None) 4032 if field is not None: 4033 self._create_prop(bch, obj_ids['uuid'], 'autonomous_system', field) 4034 4035 field = obj_dict.get('config_version', None) self = <vnc_cfg_api_server.vnc_cfg_ifmap.VncCassandraClient object> self._create_prop = <bound method VncCassandraClient._create_prop of..._server.vnc_cfg_ifmap.VncCassandraClient object>> bch = <pycassa.batch.CfMutator object> obj_ids = 'FQ Name: /fq-name-to-uuid/global_system_config:d...ady with ID: 12cf3860-d944-4fbf-8604-6fe6545a12a8' field = 64512 <type 'exceptions.TypeError'>: string indices must be integers, not str __class__ = <type 'exceptions.TypeError'> __delattr__ = <method-wrapper '__delattr__' of exceptions.TypeError object> __dict__ = {} __doc__ = 'Inappropriate argument type.' __format__ = <built-in method __format__ of exceptions.TypeError object> __getattribute__ = <method-wrapper '__getattribute__' of exceptions.TypeError object> __getitem__ = <method-wrapper '__getitem__' of exceptions.TypeError object> __getslice__ = <method-wrapper '__getslice__' of exceptions.TypeError object> __hash__ = <method-wrapper '__hash__' of exceptions.TypeError object> __init__ = <method-wrapper '__init__' of exceptions.TypeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of exceptions.TypeError object> __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.TypeError object> __repr__ = <method-wrapper '__repr__' of exceptions.TypeError object> __setattr__ = <method-wrapper '__setattr__' of exceptions.TypeError object> __setstate__ = <built-in method __setstate__ of exceptions.TypeError object> __sizeof__ = <built-in method __sizeof__ of exceptions.TypeError object> __str__ = <method-wrapper '__str__' of exceptions.TypeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __unicode__ = <built-in method __unicode__ of exceptions.TypeError object> args = ('string indices must be integers, not str',) message = 'string indices must be integers, not str' The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/bin/contrail-api", line 9, in <module> load_entry_point('vnc-cfg-api-server==0.1dev', 'console_scripts', 'contrail-api')() File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1572, in server_main main() File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1531, in main vnc_api_server = VncApiServer(args_str) File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 422, in __init__ self._db_init_entries() File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1074, in _db_init_entries config_version=CONFIG_VERSION)) File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1125, in _create_singleton_entry self._db_conn.dbe_create(obj_type, obj_ids, obj_dict) File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1701, in wrapper2 ret = func(self, obj_type, obj_ids, obj_dict) File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1719, in dbe_create (ok, result) = method(obj_ids, obj_dict) File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_cassandra_client_gen.py", line 4033, in _cassandra_global_system_config_create self._create_prop(bch, obj_ids['uuid'], 'autonomous_system', field) TypeError: string indices must be integers, not str 04/07/2015 04:39:31 AM [ct1-testjenkins-puppet-rjil-gate-1019:ApiServer:Config:0]: Failed to import package "sandesh" ERROR:ct1-testjenkins-puppet-rjil-gate-1019:ApiServer:Config:0:Failed to import package "sandesh" 04/07/2015 04:39:31 AM [ct1-testjenkins-puppet-rjil-gate-1019:ApiServer:Config:0]: Failed to import package "sandesh" ERROR:ct1-testjenkins-puppet-rjil-gate-1019:ApiServer:Config:0:Failed to import package "sandesh" 04/07/2015 04:39:31 AM [ct1-testjenkins-puppet-rjil-gate-1019:ApiServer:Config:0]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG] INFO:ct1-testjenkins-puppet-rjil-gate-1019:ApiServer:Config:0:SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG] 04/07/2015 04:39:31 AM [ct1-testjenkins-puppet-rjil-gate-1019:ApiServer:Config:0]: SANDESH: Logging: FILE: [<stdout>] -> [/var/log/contrail/api.log] INFO:ct1-testjenkins-puppet-rjil-gate-1019:ApiServer:Config:0:SANDESH: Logging: FILE: [<stdout>] -> [/var/log/contrail/api.log] Bottle v0.12.0 server starting up (using GeventServer())... Listening on http://0.0.0.0:8084/ Hit Ctrl-C to quit. Thanks, Harish
_______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
