[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release ** Changed in: openvswitch (Ubuntu Groovy) Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
MPs for B/F/G SRUs: https://code.launchpad.net/~paelzer/ubuntu/+source/openvswitch/+git/openvswitch/+merge/396076 https://code.launchpad.net/~paelzer/ubuntu/+source/openvswitch/+git/openvswitch/+merge/396075 https://code.launchpad.net/~paelzer/ubuntu/+source/openvswitch/+git/openvswitch/+merge/396074 Also I updated the bug description. It should be ready for SRU upload by James if he agrees. ** Description changed: + [Impact] + + * The current systemd profile (only active in Debian/Ubuntu) in that form +has a runtime directory. But in the default that means the runtime +dir is removed on service stop or restart. + + * In the past dpdhvhostuser connections used to use paths under that run dir +which was no problem as they were dead on restart anyway. But more modern +dpdkvhostuserclient connections might (out of habit) use the same path +and the dir removal kills that and effectively prevents to keep guest +networking alive. + + * The fix ensures the directory is kept around via the proper systemd +statement + + [Test Case] + + * start the service and touch any new file in there e.g. +$ touch /var/run/openvswitch/foo +After a restart this should still be there +$ systemctl restart openvswitch-switch +$ ls -laF /var/run/openvswitch/foo + + [Where problems could occur] + + * In our discussions we didn't find a reason that requires to clean that +directory. But if there are any setup scenarios we have forgotten that need +it then on restart they will have to deal with that "old content". +Therefore on service restart is the place to watch out for regressions. + + [Other Info] + + * n/a + + + --- + TL;DR: - stoping/restarting OVS clears /var/run/openvswitch - out of the "vhostuser" connection times a common socket path used - was at /var/run/openvswitch + was at /var/run/openvswitch - if that path used with "vhostuserclient" that removes the sockets - on OVS stop/restart + on OVS stop/restart - Since qemu in server mode only creates this sockets once (as by - the client/server design makes sense) that breaks the guests until - restarted which is what the tech of vhostuserclient wanted to avoid. - + Workaround: do use a different path like e.g. - "/var/run/vhostuserclient/vhost-user-client-1" + the client/server design makes sense) that breaks the guests until + restarted which is what the tech of vhostuserclient wanted to avoid. + + Workaround: do use a different path like e.g. + "/var/run/vhostuserclient/vhost-user-client-1" + Solution: let us think if we could keep the path around on stop/restart --- vv original report vv --- My system is Ubuntu 18.04, I installed ovs DPDK by apt-get and used ovs- vswitchd DPDK version, but when I stop openvswitch-switch (sudo systemctl stop openvswitch-switch), /var/run/openvswitch is removed, so the exisitng VMs can't be accessed any more. I don't know why it is removed and who removed it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/openvswitch/+git/openvswitch/+merge/396074 ** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/openvswitch/+git/openvswitch/+merge/396075 ** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/openvswitch/+git/openvswitch/+merge/396076 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
This bug was fixed in the package openvswitch - 2.15.0~git20210104 .def6eb1ea-0ubuntu3 --- openvswitch (2.15.0~git20210104.def6eb1ea-0ubuntu3) hirsute; urgency=medium * d/openvswitch-switch.ovsdb-server.service: avoid removing the state dir on restart (LP: #1910209) -- Christian Ehrhardt Thu, 07 Jan 2021 12:14:35 + ** Changed in: openvswitch (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
This is uploaded to Ubuntu 21.04 already. Just needs some builds and tests to complete. I'm slightly unsure if this should be SRUed or if - for older releases we'd ask users to adapt the configuration if affected. I'm slightly leaning to an SRU but look at James/Frode for a decsion. ** Also affects: openvswitch (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: dpdk (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: openvswitch (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: dpdk (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: openvswitch (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: dpdk (Ubuntu Focal) Importance: Undecided Status: New ** No longer affects: dpdk (Ubuntu Bionic) ** No longer affects: dpdk (Ubuntu Focal) ** No longer affects: dpdk (Ubuntu Groovy) ** Changed in: openvswitch (Ubuntu Bionic) Status: New => Confirmed ** Changed in: openvswitch (Ubuntu Focal) Status: New => Confirmed ** Changed in: openvswitch (Ubuntu Groovy) Status: New => Confirmed ** Changed in: openvswitch (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
BTW, /var/run/openvswitch doesn't keep ovsdb data there, so preserving it is safe. It just saved some unix socket files and pid files, they will be cleaned by ovs deamons when daenons are stopped. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
Christian, thank you so much, the below solution works for me. $ sudo systemctl edit ovsdb-server.service [Service] RuntimeDirectoryPreserve=yes BTW, only one way to change vhostuser path is using ovs other_config :vhost-sock-dir, but it is a subdir under /var/run/openvswitch, so it can't fix this. https://github.com/openvswitch/ovs/blob/master/lib/dpdk.c#L360. If you create dpdkvhostuserclient port by yourself and add-port it by yourself, you can specify full path, but for me, these are handled by openstack, so I can't control this, other_config:vhost-sock-dir is only one way to change this, but it is relative path, it can't be beyond /var/run/openvswitch. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
openvswitch-switch consists of three services actually: openvswitch-switch.service (top level) -> ovsdb-server.service -> ovs-vswitchd.service Of those the start/stop/restart of ovsdb-server.service is the one creating/removing the directory of /var/run/openvswitch/. The latter ovs-vswitchd.service then re-uses that directory. The path is created due to /lib/systemd/system/ovsdb-server.service: RuntimeDirectory=openvswitch RuntimeDirectoryMode=0755 Acoording to [1] that will be "In case of RuntimeDirectory= the innermost subdirectories are removed when the unit is stopped. It is possible to preserve the specified directories in this case if RuntimeDirectoryPreserve= is configured to restart or yes (see below)" So the behavior atm is exactly as configured. The service files are "ours" (= packaging). Upstream only has rhel service files and those do not dynamically add/remove the path. Instead I've even seen discussions [2] saying about RuntimeDirectoryPreserve "We need to have this either as 'yes', or 'restart' - OVN daemons depend on this directory persisting even when the OVS daemons go away." We can add a change via: $ sudo systemctl edit ovsdb-server.service [Service] RuntimeDirectoryPreserve=yes That will keep the directory alive and avoid the issue. @Yi - for now that could be your solution as it would avoid having you to reconfigure (potentially many) other places. Packaging wise that would be as easy as [3] But I'm unsure if there is intention in removing this directory - e.g. to get the DB state cleaned and re-initialized for sure. I'd leave that up to James/Frode who have looked more at OVS. Based on their decision we can try to make this statement part of the default config (or not). Subscribing James/Frode here on the bug and on the MP to carry on from here. [1]: https://www.freedesktop.org/software/systemd/man/systemd.exec.html [2]: https://github.com/openvswitch/ovs/pull/247/files#r208172722 [3]: https://code.launchpad.net/~paelzer/ubuntu/+source/openvswitch/+git/openvswitch/+merge/395882 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/openvswitch/+git/openvswitch/+merge/395882 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
While the behavior has "more impact" when running with openvswitch-dpdk as one common path to put the sockets is under /var/run/openvswitch the problem is only in openvswitch. I updated the bug tasks accordingly. @Yi: I've checked my setups why I haven't seen this before and indeed I usually use a path like "/var/run/vhostuserclient/vhost-user-client-1" and it works fine - so using something like that might be more than just a workaround for you. Never the less I happen to know that /var/run/openvswitch was in some guides in the past. I even fixed one stray reference in our own docs [2] now to fully use "/var/run/vhostuserclient/vhost-user-client-1" as shown above (its examples had a mix of paths up to now). So thanks for the report for that alone ! The upstream OVS docs [1] usually seem to use a path like "/tmp/dpdkvhostclient0". But even in there are examples of "/usr/local/var/run/openvswitch/vhost-user-1". Reading that has made me realize that those paths with the pattern "/var/run/openvswitch" are all from the pre vhostuserclient time, and back then restarting OVS killed the connection anyway. So it wasn't a problem back then, but if users keep using those paths with vhostuserclient - then it is a problem. Adding a TL;DR to the bug description. [1]: https://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/ [2]: https://ubuntu.com/server/docs/openvswitch-dpdk ** Description changed: + TL;DR: + - stoping/restarting OVS clears /var/run/openvswitch + - out of the "vhostuser" connection times a common socket path used + was at /var/run/openvswitch + - if that path used with "vhostuserclient" that removes the sockets + on OVS stop/restart + - Since qemu in server mode only creates this sockets once (as by + the client/server design makes sense) that breaks the guests until + restarted which is what the tech of vhostuserclient wanted to avoid. + + Workaround: do use a different path like e.g. + "/var/run/vhostuserclient/vhost-user-client-1" + + Solution: let us think if we could keep the path around on stop/restart + + --- vv original report vv --- + My system is Ubuntu 18.04, I installed ovs DPDK by apt-get and used ovs- vswitchd DPDK version, but when I stop openvswitch-switch (sudo systemctl stop openvswitch-switch), /var/run/openvswitch is removed, so the exisitng VMs can't be accessed any more. I don't know why it is removed and who removed it. ** Changed in: openvswitch (Ubuntu) Importance: Undecided => Medium ** Also affects: serverguide Importance: Undecided Status: New ** Changed in: serverguide Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/serverguide/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
For the 16.04 case there was no dpdkvhostuserclient yet, so there it had to stay around anyway as OVS was the server. But I agree that it is in issue in regard to "/var/run/openvswitch/vhostuser/vhu* is created by qemu once" that matches what I expect and indeed in that case clearing the var/run directory is bad indeed. For the time being a temporary workaround could be to try using a different path. I'm unsure if - without further tweaks - OVS and the Qemu rocessed are allowed to access it (apparmor is in place), but you could try putting the socket paths at e.g. /var/run/ovs-qemu-sockets/ and see if this works as an interim solution. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
FYI: - systemctl reload works fine (as that calls /usr/share/openvswitch/scripts/ovs-systemd-reload) - systemctl restart also clears the /var/run/openvswitch directory ** Changed in: dpdk (Ubuntu) Status: Incomplete => Confirmed ** Also affects: openvswitch (Ubuntu) Importance: Undecided Status: New ** Changed in: openvswitch (Ubuntu) Status: New => Confirmed ** Changed in: dpdk (Ubuntu) Status: Confirmed => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
Christian, thank you so much for quickly replying, but in Ubuntu 16.04, /var/run/openvswitch wasn't removed after "systemctl stop openvswitch- switch", in Ubuntu 18.04, ovs deamon also doesn't remove /var/run/openvswitch, can you tell me who created /var/run/openvswitch and who removed it? I just want to let it work normally. For dpdkvhostuserclient case, this unix socket /var/run/openvswitch/vhostuser/vhu* is created by qemu once, ovs DPDK won't create it when it is restarted, this isn't ovs DPDK's mistake, it should be so. Right now, it is removed by "systemctl stop openvswitch- switch", this is root cause it can't work. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
Hi Yi Yang, TBH that isn't surprising. This is like saying "I have a server here, and after I pulled the power plug no one can connect to it anymore". The important bit for this to read into IMHO is vhost-user vs. vhost-user-client => https://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/#vhost-user-vs-vhost-user-client >From there you'll see: " ... if OVS dies, all VMs must be restarted." that is effectively the scenario you are in when you stop the service. But Recent versions fully support vhostuserclient in DPDK and OVS. Using that allows you to restart the OVS without loosing the guests. They might see a short network outage thou (which is reasonable and ok). Now your config already states type: dpdkvhostuserclient which means you started to do the right thing. But you haven't mentioned it at all. While OVS is down it is ok/expected that nothing works. But when OVS is back up, then the VMs should re-connect to the sockets and networking should get up again. Note I'm not 100% certain if the DPDK/OVS in 18.04 already had all it needed - you might consider using https://launchpad.net/~canonical- server/+archive/ubuntu/server-backports to try if that version suddenly makes you happy. If it does consider upgrading to 20.04 for a long term setup. If I missed some detail in your report please explain and let me know. Kind Regards Christian ** Changed in: dpdk (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
By the way, VM's vhost-server-path is
Port "vhu8dcfe028-32"
tag: 4095
Interface "vhu8dcfe028-32"
type: dpdkvhostuserclient
options:
{vhost-server-path="/var/run/openvswitch/vhostuser/vhu8dcfe028-32"}
It can't be removed.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1910209
Title:
"systemctl stop openvswitch-switch" will remove /var/run/openvswitch
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1910209/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
# dpkg -l | grep openvswitch ii openvswitch-common 2.9.5-0ubuntu0.18.04.1 amd64Open vSwitch common components ii openvswitch-switch 2.9.5-0ubuntu0.18.04.1 amd64Open vSwitch switch implementations ii openvswitch-switch-dpdk2.9.5-0ubuntu0.18.04.1 amd64DPDK enabled Open vSwitch switch implementation ii python-openvswitch 2.9.5-0ubuntu0.18.04.1 all Python bindings for Open vSwitch # cat /etc/*-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS" NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic # -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1910209] Re: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch
# ps aux | grep ovs-vswitchd root 75975 200 0.3 135321916 40 ?Shttps://bugs.launchpad.net/bugs/1910209 Title: "systemctl stop openvswitch-switch" will remove /var/run/openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1910209/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
