[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
I also experienced this on 2 of 3 units of designate running 21.01 charms upgrading from cloud:bionic-stein to cloud:bionic-train with action-managed-upgrade=false. Interestingly, the unit that did not exhibit this race was not the leader. I've got an SOSreport available if interested. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Hi, As we are upgrading our Cloud from Stein to Train, we experienced the exact same problem, one of our 3 HA designate unit reports "Services not running that should be : designate-producer", I tried to "pause" then "resume" the unit, nothing changed, neither after a reboot, and when I try to restart memcached.service and designate-producer.service, this last was still "active/dead". After checking the upgrade process, this blocked unit don't have any reference to "train repository" in the sources.list and seems to have not been able to perform the upgrade ... no designate-* packages found in version 9.0.1 (for train)... So I did the upgrade procedure once again, pausing hacluster-designate on the unit and designate unit itself, then launched again 'juju run- action designate/3 --wait openstack-upgrade' and then it did the job ! When I resumed the unit and his subordinated charm, juju reported it as "active/idle" and on the unit the "designate-producer.service" is "Active: active (running)" and no problem remains, all packages are up to date... I hope it will help anybody ! Best regards. Frédéric MOSSER. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Changed in: charm-deployment-guide Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Changed in: charm-designate Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Changed in: charm-designate Milestone: None => 20.08 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
released as cs:~llama-charmers-next/memcached-4 ** Changed in: charm-memcached Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Reviewed: https://review.opendev.org/733121 Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=5c929b3ba950257cd612c1ba94d57c145ec03526 Submitter: Zuul Branch:master commit 5c929b3ba950257cd612c1ba94d57c145ec03526 Author: Chris MacNaughton Date: Wed Jun 3 14:02:57 2020 +0200 Request a restart of memcached after a service upgrade after queens Change-Id: I54b235de947e63e3d7b86ccfdba9f1b968b75650 Closes-Bug: #1828534 ** Changed in: charm-designate Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Merge proposal linked: https://code.launchpad.net/~chris.macnaughton/charm-memcached/+git/charm-memcached/+merge/384961 ** Changed in: charm-memcached Status: Triaged => Fix Committed ** Changed in: charm-memcached Milestone: None => 20.08 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Changed in: charm-memcached Importance: Undecided => High ** Changed in: charm-memcached Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Changed in: charm-designate Assignee: (unassigned) => Chris MacNaughton (chris.macnaughton) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
This issue can be resolved by restarting the memcached service that designate is using for coordination. After this, the designate-producer service seems to run normally. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Chris' doc update has been merged: https://review.opendev.org/#/c/729300 ** Changed in: charm-deployment-guide Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
adding project charm-memcached per @chris.macnaughton comment #24. That charm could grow a hook for cache-relation-changed to respond to requests for encoding updates/service recycle from other charms. ** Also affects: charm-memcached Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Yes let's update the charm deployment guide, specifically this section should call this upgrade issue out: https://docs.openstack.org/project- deploy-guide/charm-deployment-guide/latest/app-upgrade-openstack.html #known-openstack-upgrade-issues ** Also affects: charm-deployment-guide Importance: Undecided Status: New ** Changed in: charm-deployment-guide Importance: Undecided => High ** Changed in: charm-deployment-guide Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Also we could consider cherry-picking the fixes back to rocky and stein packages. They've just not landed upstream. It would fix py2->py3 upgrades, which this bug reported. But I believe the fix would break py3->py3 upgrades. The charm deploys py3 as of rocky. The package was py2 by default in rocky (with py3 available) and py3 by default in stein. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-deployment-guide/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Hi Corey, if we can't fix this, we need to at least document it as a known issue in the charm OpenStack upgrade documentation, surely? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
@Dincer, that doesn't look the same -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
@Corey Is http://paste.openstack.org/show/787404/ related with this issue? Using stein with ubuntu. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
I don't think we'll be fixing this in stable/rocky or stable/stein as it would break existing py3 deployments in a stable release. The trade off is that it would fix py2 deployments that upgrade to py3. For now I'm going to mark those releases as "won't fix". ** Changed in: designate (Ubuntu Disco) Status: Triaged => Won't Fix ** Changed in: cloud-archive/stein Status: Triaged => Won't Fix ** Changed in: cloud-archive/rocky Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Also affects: designate (Ubuntu Eoan) Importance: Undecided Status: New ** Also affects: designate (Ubuntu Disco) Importance: Undecided Status: New ** Changed in: designate (Ubuntu Eoan) Status: New => Fix Released ** Changed in: designate (Ubuntu Eoan) Importance: Undecided => High ** Changed in: designate (Ubuntu Disco) Importance: Undecided => High ** Changed in: designate (Ubuntu Disco) Status: New => Triaged ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/stein Importance: Undecided Status: New ** Also affects: cloud-archive/rocky Importance: Undecided Status: New ** Also affects: cloud-archive/train Importance: Undecided Status: New ** Changed in: designate (Ubuntu) Status: Triaged => Fix Committed ** Changed in: designate (Ubuntu) Status: Fix Committed => Fix Released ** Changed in: cloud-archive/train Status: New => Fix Released ** Changed in: cloud-archive/train Importance: Undecided => High ** Changed in: cloud-archive/rocky Importance: Undecided => High ** Changed in: cloud-archive/stein Importance: Undecided => High ** Changed in: cloud-archive/stein Status: New => Triaged ** Changed in: cloud-archive/rocky Status: New => Triaged ** Changed in: cloud-archive Status: New => Fix Released ** Changed in: cloud-archive Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Reviewed: https://review.opendev.org/673360 Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=556a27e4e9c9c4e21fb0ea46d8d8832d28c85314 Submitter: Zuul Branch:master commit 556a27e4e9c9c4e21fb0ea46d8d8832d28c85314 Author: Corey Bryant Date: Mon Jul 29 15:44:48 2019 -0400 Ensure coordination IDs are encoded Ensure coordination IDs are encoded when working with coordination backend. This fixes an issue when upgrading to Python 3 (where bytes and str are different types) and _update_partitions() attempts to sort types of 'str' and 'bytes', causing designate-producer to crash. Change-Id: Id8206ee5285d3a73e00ef21b7d3961a29c23ab4b Closes-Bug: #1828534 ** Changed in: designate Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Changed in: charm-designate Milestone: 19.07 => None -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Fix proposed to branch: master Review: https://review.opendev.org/673360 ** Changed in: designate Status: New => In Progress ** Changed in: designate Assignee: (unassigned) => Corey Bryant (corey.bryant) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Tags added: py3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
I have a fix coming for this. It is very similar to the example at: https://docs.openstack.org/tooz/latest/user/tutorial/group_membership.html , where host-1 is binary encoded. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
** Also affects: designate Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
>From /usr/lib/python3/dist-packages/designate/coordination.py:
from oslo_config import cfg
CONF = cfg.CONF
class CoordinationMixin(object):
def __init__(self, *args, **kwargs):
super(CoordinationMixin, self).__init__(*args, **kwargs)
self._coordinator = None
def start(self):
self._coordination_id = ":".join([CONF.host, generate_uuid()]) #
<-- this line seems to match up with the logs above where CONF.host is
bytes and str type
It looks like that might be defined in one of these places:
ubuntu@juju-3d28eb-coreycb2-7:/usr/lib/python3/dist-packages/designate$ grep -r
StrOpt | grep \'host\'
__init__.py:cfg.StrOpt('host', default=socket.gethostname(),
pool_manager/__init__.py:cfg.StrOpt('host'),
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1828534
Title:
[19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not
running that should be: designate-producer"
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Reproducing with designate, the problem appears after we have one
designate unit that is at bionic-rocky (py3) and one that is at bionic-
queens (py2). It is resolved once all units are at bionic-rocky. More
details on that:
# upgrade one unit to bionic-rocky
juju config designate openstack-origin=cloud:bionic-rocky
juju run-action designate/0 openstack-upgrade
juju status
Unit Workload Agent Machine Public address Ports
Message
designate/0 error idle 710.5.0.198 9001/tcp
Services not running that should be: designate-producer
hacluster-designate/0 activeidle10.5.0.198
Unit is ready and clustered
designate/1* activeidle 810.5.0.69001/tcp
Unit is ready
hacluster-designate/1* activeidle10.5.0.6
Unit is ready and clustered
juju ssh designate/0
2019-07-25 18:04:27.452 24807 DEBUG designate.coordination [-] CCB:
list(self._get_members(self._group_id))=[b'juju-3d28eb-
coreycb2-8:86c49114-a37a-4e1a-8654-72bf1ccbde1f', 'juju-3d28eb-
coreycb2-7:d64cde68-d9f0-4910-9f1a-6d9724ea77b9'] _update_partitions
/usr/lib/python3/dist-packages/designate/coordination.py:202
where:
- machine 8 (b'juju-3d28eb-coreycb2-8:86c49114-a37a-4e1a-8654-72bf1ccbde1f')
hasn't been upgraded yet and is still bionic-queens
- machine 7 ('juju-3d28eb-coreycb2-7:d64cde68-d9f0-4910-9f1a-6d9724ea77b9') is
the machine that was upgraded to bionic-rocky
Note: designate coordinator backend gets the juju strings by calling
get_members() from the tooz memcached backend. /usr/lib/python3/dist-
packages/tooz/drivers/memcached.py
# now try to upgrade the other unit to bionic-rocky
juju run-action designate/1 openstack-upgrade
# restart memcache and designate-producer on designate/0 and resolve
juju resolved designate/0
# all better
Unit Workload Agent Machine Public address Ports
Message
designate/0 activeidle 710.5.0.198 9001/tcp
Unit is ready
hacluster-designate/0 activeidle10.5.0.198
Unit is ready and clustered
designate/1* activeidle 810.5.0.69001/tcp
Unit is ready
hacluster-designate/1* activeidle10.5.0.6
Unit is ready and clustered
juju ssh designate/0
2019-07-25 19:31:11.365 12217 DEBUG designate.coordination [-] CCB:
list(self._get_members(self._group_id))=['juju-3d28eb-
coreycb2-7:0ceca5d8-9eee-492d-bead-f99ee990a21b', 'juju-3d28eb-
coreycb2-8:134e6142-4364-43a5-96fc-b9f125ba87a0'] _update_partitions
/usr/lib/python3/dist-packages/designate/coordination.py:202
Both strings are str type now and able to be sorted.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1828534
Title:
[19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not
running that should be: designate-producer"
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
This is reproducible in a python3 shell. The problem is that sorted() is called on a list that includes both bytes and str types: $ python3 Python 3.7.4 (default, Jul 11 2019, 10:43:21) [GCC 9.1.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> blist=['1'.encode(), '3', '2'.encode()] >>> sorted(blist) Traceback (most recent call last): File "", line 1, in TypeError: '<' not supported between instances of 'str' and 'bytes' >>> -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
I was able to reproduce this. It's specific to coordination backend code that is used for HA. For charms, the following bundle config was used (snipped for brevity to just show designate bits): hacluster-designate: charm: cs:hacluster options: cluster_count: 2 designate: charm: cs:~openstack-charmers-next/designate constraints: mem=1G num_units: 2 options: action-managed-upgrade: true debug: true nameservers: 'ns1.ubuntu.com' nova-domain: 'serverstack.ubuntu.com.' neutron-domain: 'serverstack.ubuntu.com.' nova-domain-email: 'b...@serverstack.ubuntu.com' neutron-domain-email: 'b...@serverstack.ubuntu.com' vip: 10.5.20.1 designate-bind: charm: cs:~openstack-charmers-next/designate-bind relations: - [ designate, keystone ] - [ designate, mysql ] - [ designate, rabbitmq-server ] - [ designate, designate-bind ] - [ designate, memcached ] - [ designate, hacluster-designate ] - - designate:dnsaas - neutron-api:external-dns ** Changed in: designate (Ubuntu) Status: New => Triaged ** Changed in: designate (Ubuntu) Importance: Undecided => High ** Changed in: charm-designate Importance: Critical => Undecided -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
The traceback looks like an upstream designate issue with py3 support: [-] Error starting thread.: TypeError: '<' not supported between instances of 'str' and 'bytes' Traceback (most recent call last): File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 794, in run_service service.start() File "/usr/lib/python3/dist-packages/designate/producer/service.py", line 72, in start self._partitioner.start() File "/usr/lib/python3/dist-packages/designate/coordination.py", line 223, in start self._my_partitions = self._update_partitions()[1] File "/usr/lib/python3/dist-packages/designate/coordination.py", line 200, in _update_partitions members = sorted(list(self._get_members(self._group_id))) TypeError: '<' not supported between instances of 'str' and 'bytes' -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Attached logs from a healthy unit as well. ** Attachment added: "/var/log/ from designate/0 (healthy)" https://bugs.launchpad.net/charm-designate/+bug/1828534/+attachment/5275082/+files/var-log-designate-0.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Managed to reproduce it (only on one unit though), this time with debug=true - looks like it is a py2 - py3 issue: designate/0 activeidle 2/lxd/2 10.232.46.209 9001/tcpUnit is ready hacluster-designate/0* activeidle10.232.46.209 Unit is ready and clustered designate/1* blocked idle 3/lxd/1 10.232.46.208 9001/tcpServices not running that should be: designate-producer hacluster-designate/1 activeidle10.232.46.208 Unit is ready and clustered /var/log/designate/designate-producer.log https://paste.ubuntu.com/p/pMbt2q6KTF/ 2019-07-03 23:43:48.226 37919 DEBUG designate.service [-] Starting RPC server on topic 'producer' start /usr/lib/python3/dist-packages/designate/service.py:171 2019-07-03 23:43:48.252 37919 DEBUG designate.coordination [-] Starting partitioner start /usr/lib/python3/dist-packages/designate/coordination.py:213 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service [-] Error starting thread.: TypeError: '<' not supported between instances of 'str' and 'bytes' 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service Traceback (most recent call last): 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 794, in run_service 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service service.start() 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/producer/service.py", line 72, in start 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service self._partitioner.start() 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/coordination.py", line 223, in start 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service self._my_partitions = self._update_partitions()[1] 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/coordination.py", line 200, in _update_partitions 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service members = sorted(list(self._get_members(self._group_id))) 2019-07-03 23:43:48.257 37919 ERROR oslo_service.service TypeError: '<' not supported between instances of 'str' and 'bytes' systemctl status designate-producer ● designate-producer.service - OpenStack Designate DNSaaS producer Loaded: loaded (/lib/systemd/system/designate-producer.service; enabled; vendor preset: enabled) Active: inactive (dead) since Wed 2019-07-03 23:43:49 UTC; 16h ago Main PID: 37919 (code=exited, status=0/SUCCESS) ** Attachment added: "/var/log/ from designate/1 (broken)" https://bugs.launchpad.net/charm-designate/+bug/1828534/+attachment/5275081/+files/var-log-designate-1.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
I'm going to do another upgrade attempt. https://pastebin.canonical.com/p/72XWqsjMdr/ (bundle) https://paste.ubuntu.com/p/JSSFZvp3bB/ (upgrade script: os-upgrade.py with a `juju --wait` and service list modification) ~/bundles/os-upgrade-queens-rocky.py -p Let's see if I can reproduce it again. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Any tips on how to reproduce this? I just upgrade designate from queens to rocky and didn't hit it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828534] Re: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
I think this is a packaging bug ** Also affects: designate (Ubuntu) Importance: Undecided Status: New ** Changed in: charm-designate Status: Triaged => Invalid ** Changed in: charm-designate Assignee: Liam Young (gnuoy) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828534 Title: [19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer" To manage notifications about this bug go to: https://bugs.launchpad.net/charm-designate/+bug/1828534/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
