Thanks Ilya for the quick and useful response!
We’ll dig into monitor/db_change_aware logic.
Regards,
Vladislav Odintsov
> On 24 Jan 2023, at 17:00, Ilya Maximets <i.maxim...@ovn.org> wrote:
>
> On 1/24/23 14:12, Vladislav Odintsov wrote:
>> Hi Ilya,
>>
>> could you please take a look on this?
>> Maybe you can advice any direction how to investigate this issue?
>>
>> Thanks in advance.
>>
>> Regards,
>> Vladislav Odintsov
>>
>>> On 24 Nov 2022, at 21:10, Anton Vazhnetsov <dragen15...@gmail.com> wrote:
>>>
>>> Hi, Terry!
>>>
>>> In continuation to our yesterday’s conversation [0], we were able to
>>> reproduce the issue with KeyError. We found that the problem is not
>>> connected with ovsdb-server load but it appears when the ovsdb-server
>>> schema is converted online (it even doesn’t matter whether the real ovs
>>> schema is changed) while the active connection persists.
>>> Please use next commands to reproduce it:
>>>
>>> # in terminal 1
>>>
>>> ovsdb-tool create ./ovs.db /usr/share/ovn/ovn-nb.ovsschema
>>> ovsdb-server --remote punix://$(pwd)/ovs.sock <punix://$(pwd)/ovs.sock>
>>> $(pwd)/ovs.db -vconsole:dbg
>>>
>>>
>>> # in terminal 2. run python shell
>>> python3
>>> # setup connection
>>> import ovsdbapp.schema.ovn_northbound.impl_idl as nb_idl
>>> from ovsdbapp.backend.ovs_idl import connection
>>>
>>> remote = "unix:///<path to ovs.sock>"
>>>
>>> def get_idl():
>>> """Connection getter."""
>>>
>>> idl = connection.OvsdbIdl.from_server(remote, "OVN_Northbound",
>>> leader_only=False)
>>> return nb_idl.OvnNbApiIdlImpl(connection.Connection(idl, 100))
>>>
>>> idl = get_idl()
>>>
>>>
>>> # in terminal 1
>>> ovsdb-client convert unix:$(pwd)/ovs.sock /usr/share/ovn/ovn-nb.ovsschema
>>>
>>> # in terminal 2 python shell:
>>> idl.ls_add("test").execute()
>>>
>>>
>>> We get following traceback:
>>>
>>> Traceback (most recent call last):
>>> File
>>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
>>> line 131, in run
>>> txn.results.put(txn.do_commit())
>>> File
>>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
>>> line 143, in do_commit
>>> self.post_commit(txn)
>>> File
>>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
>>> line 73, in post_commit
>>> command.post_commit(txn)
>>> File
>>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py",
>>> line 94, in post_commit
>>> row = self.api.tables[self.table_name].rows[real_uuid]
>>> File "/usr/lib64/python3.6/collections/__init__.py", line 991, in
>>> __getitem__
>>> raise KeyError(key)
>>> KeyError: UUID('0256afa4-6dd0-4c2c-b6a2-686a360ab331')
>>>
>>> In ovsdb-server debug logs we see that update2 or update3 messages are not
>>> sent from server in response to client’s transaction, just reply with
>>> result UUID:
>>> 2022-11-24T17:42:36Z|00306|poll_loop|DBG|wakeup due to [POLLIN] on fd 18
>>> (///root/ovsdb-problem/ovs.sock<->) at lib/stream-fd.c:157
>>> 2022-11-24T17:42:36Z|00307|jsonrpc|DBG|unix#5: received request,
>>> method="transact",
>>> params=["OVN_Northbound",{"uuid-name":"row03ef28d6_93f1_43bc_b07a_eae58d4bd1c5","table":"Logical_Switch","op":"insert","row":{"name”:"test"}}],
>>> id=5
>>> 2022-11-24T17:42:36Z|00308|jsonrpc|DBG|unix#5: send reply,
>>> result=[{"uuid":["uuid","4eb7c407-beec-46ca-b816-19f942e57721"]}], id=5
>>>
>>> We checked same with ovn-nbctl running in daemon mode and found that the
>>> problem is not reproduced (ovsdb-server after database conversion sends out
>>> update3 message to ovn-nbctl daemon process in response to transaction, for
>>> example ovs-appctl -t <nbctl daemon socket> run ls-add test-ls):
>
> If the update3 is not sent, it means that the client doesn't monitor
> this table. You need to look at monitor requests sent and replied
> to figure out the difference.
>
> Since the database conversion is involved, server will close all
> monitors on schema update and notify all clients that are db_change_aware.
> Clients should re-send monitor requests at this point. If clients
> are not db_change_aware, server will just disconnect them, so they
> can re-connect and see the new schema and send new monitor requests.
>
> On a quick glance I don't see python idl handling 'monitor_canceled'
> notification. That is most likely a root cause. Python IDL claims
> to be db_change_aware, but it doesn't seem to be.
>
> Best regards, Ilya Maximets.
>
>>> 2022-11-24T17:54:51Z|00623|jsonrpc|DBG|unix#7: received request,
>>> method="transact",
>>> params=["OVN_Northbound",{"uuid-name":"rowcdb152ce_a9af_4761_b965_708ad300fcb7","table":"Logical_Switch","op":"insert","row":{"name":"test-ls"}},{"comment":"ovn-nbctl:
>>> run ls-add test-ls","op":"comment"}], id=5
>>> 2022-11-24T17:54:51Z|00624|jsonrpc|DBG|unix#7: send notification,
>>> method="update3",
>>> params=[["monid","OVN_Northbound"],"00000000-0000-0000-0000-000000000000",{"Logical_Switch":{"0b147f2c-248d-496a-b718-a5328d3c2995":{"insert":{"name":"test-ls"}}}}]
>>> 2022-11-24T17:54:51Z|00625|jsonrpc|DBG|unix#7: send reply,
>>> result=[{"uuid":["uuid","0b147f2c-248d-496a-b718-a5328d3c2995"]},{}], id=5
>>>
>>> So it seems that the problem is in python-ovs, not in ovsdb-server.
>>> We tested with ovsdb-server running version 2.17.3 and python-ovs 2.13.5
>>> and also python-ovs 2.17.3, the behaviour is the same.
>>>
>>> Do you have any ideas what can be a reason for such behaviour?
>>>
>>> 0:
>>> https://review.opendev.org/c/openstack/ovsdbapp/+/865454/comments/674c57e6_3849591b<https://review.opendev.org/c/openstack/ovsdbapp/+/865454/comments/674c57e6_3849591b>
>>>
>>> Regards, Anton.
>>
>
> _______________________________________________
> dev mailing list
> d...@openvswitch.org <mailto:d...@openvswitch.org>
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
