On Tue, Jan 24, 2023 at 8:00 AM Ilya Maximets <i.maxim...@ovn.org> wrote:
>
> On 1/24/23 14:12, Vladislav Odintsov wrote:
> > Hi Ilya,
> >
> > could you please take a look on this?
> > Maybe you can advice any direction how to investigate this issue?
> >
> > Thanks in advance.
> >
> > Regards,
> > Vladislav Odintsov
> >
> >> On 24 Nov 2022, at 21:10, Anton Vazhnetsov <dragen15...@gmail.com> wrote:
> >>
> >> Hi, Terry!
> >>
> >> In continuation to our yesterday’s conversation [0], we were able to
> >> reproduce the issue with KeyError. We found that the problem is not
> >> connected with ovsdb-server load but it appears when the ovsdb-server
> >> schema is converted online (it even doesn’t matter whether the real ovs
> >> schema is changed) while the active connection persists.
> >> Please use next commands to reproduce it:
> >>
> >> # in terminal 1
> >>
> >> ovsdb-tool create ./ovs.db /usr/share/ovn/ovn-nb.ovsschema
> >> ovsdb-server --remote punix://$(pwd)/ovs.sock <punix://$(pwd)/ovs.sock>
> >> $(pwd)/ovs.db -vconsole:dbg
> >>
> >>
> >> # in terminal 2. run python shell
> >> python3
> >> # setup connection
> >> import ovsdbapp.schema.ovn_northbound.impl_idl as nb_idl
> >> from ovsdbapp.backend.ovs_idl import connection
> >>
> >> remote = "unix:///<path to ovs.sock>"
> >>
> >> def get_idl():
> >> """Connection getter."""
> >>
> >> idl = connection.OvsdbIdl.from_server(remote, "OVN_Northbound",
> >> leader_only=False)
> >> return nb_idl.OvnNbApiIdlImpl(connection.Connection(idl, 100))
> >>
> >> idl = get_idl()
> >>
> >>
> >> # in terminal 1
> >> ovsdb-client convert unix:$(pwd)/ovs.sock /usr/share/ovn/ovn-nb.ovsschema
> >>
> >> # in terminal 2 python shell:
> >> idl.ls_add("test").execute()
> >>
> >>
> >> We get following traceback:
> >>
> >> Traceback (most recent call last):
> >> File
> >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
> >> line 131, in run
> >> txn.results.put(txn.do_commit())
> >> File
> >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
> >> line 143, in do_commit
> >> self.post_commit(txn)
> >> File
> >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
> >> line 73, in post_commit
> >> command.post_commit(txn)
> >> File
> >> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py",
> >> line 94, in post_commit
> >> row = self.api.tables[self.table_name].rows[real_uuid]
> >> File "/usr/lib64/python3.6/collections/__init__.py", line 991, in
> >> __getitem__
> >> raise KeyError(key)
> >> KeyError: UUID('0256afa4-6dd0-4c2c-b6a2-686a360ab331')
> >>
> >> In ovsdb-server debug logs we see that update2 or update3 messages are not
> >> sent from server in response to client’s transaction, just reply with
> >> result UUID:
> >> 2022-11-24T17:42:36Z|00306|poll_loop|DBG|wakeup due to [POLLIN] on fd 18
> >> (///root/ovsdb-problem/ovs.sock<->) at lib/stream-fd.c:157
> >> 2022-11-24T17:42:36Z|00307|jsonrpc|DBG|unix#5: received request,
> >> method="transact",
> >> params=["OVN_Northbound",{"uuid-name":"row03ef28d6_93f1_43bc_b07a_eae58d4bd1c5","table":"Logical_Switch","op":"insert","row":{"name”:"test"}}],
> >> id=5
> >> 2022-11-24T17:42:36Z|00308|jsonrpc|DBG|unix#5: send reply,
> >> result=[{"uuid":["uuid","4eb7c407-beec-46ca-b816-19f942e57721"]}], id=5
> >>
> >> We checked same with ovn-nbctl running in daemon mode and found that the
> >> problem is not reproduced (ovsdb-server after database conversion sends
> >> out update3 message to ovn-nbctl daemon process in response to
> >> transaction, for example ovs-appctl -t <nbctl daemon socket> run ls-add
> >> test-ls):
>
> If the update3 is not sent, it means that the client doesn't monitor
> this table. You need to look at monitor requests sent and replied
> to figure out the difference.
>
> Since the database conversion is involved, server will close all
> monitors on schema update and notify all clients that are db_change_aware.
> Clients should re-send monitor requests at this point. If clients
> are not db_change_aware, server will just disconnect them, so they
> can re-connect and see the new schema and send new monitor requests.
>
> On a quick glance I don't see python idl handling 'monitor_canceled'
> notification. That is most likely a root cause. Python IDL claims
> to be db_change_aware, but it doesn't seem to be.
>
Reviving a very old thread, but yes this is definitely the issue. I
just tested and see that we get the monitor_canceled and do not do
anything with it. Parsing it and calling self.force_reconnect() seems
to fix the issue. I can send a patch shortly.
Terry
> Best regards, Ilya Maximets.
>
> >> 2022-11-24T17:54:51Z|00623|jsonrpc|DBG|unix#7: received request,
> >> method="transact",
> >> params=["OVN_Northbound",{"uuid-name":"rowcdb152ce_a9af_4761_b965_708ad300fcb7","table":"Logical_Switch","op":"insert","row":{"name":"test-ls"}},{"comment":"ovn-nbctl:
> >> run ls-add test-ls","op":"comment"}], id=5
> >> 2022-11-24T17:54:51Z|00624|jsonrpc|DBG|unix#7: send notification,
> >> method="update3",
> >> params=[["monid","OVN_Northbound"],"00000000-0000-0000-0000-000000000000",{"Logical_Switch":{"0b147f2c-248d-496a-b718-a5328d3c2995":{"insert":{"name":"test-ls"}}}}]
> >> 2022-11-24T17:54:51Z|00625|jsonrpc|DBG|unix#7: send reply,
> >> result=[{"uuid":["uuid","0b147f2c-248d-496a-b718-a5328d3c2995"]},{}], id=5
> >>
> >> So it seems that the problem is in python-ovs, not in ovsdb-server.
> >> We tested with ovsdb-server running version 2.17.3 and python-ovs 2.13.5
> >> and also python-ovs 2.17.3, the behaviour is the same.
> >>
> >> Do you have any ideas what can be a reason for such behaviour?
> >>
> >> 0:
> >> https://review.opendev.org/c/openstack/ovsdbapp/+/865454/comments/674c57e6_3849591b
> >>
> >> <https://review.opendev.org/c/openstack/ovsdbapp/+/865454/comments/674c57e6_3849591b>
> >>
> >> Regards, Anton.
> >
>
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
