Re: failure upgrading Juju from 1.21.1 to 1.22.0 on trusty
I'd want to hear from others on this matter, as my experience with repairing upgrades is limited. I recommend you don't take any action on the following immediately, until others have weighed in. I will hold off doing anything else for now, as I don’t want to risk losing our running environment (everything that Juju set up still seems to be running fine at the moment, we just can’t make any changes). John figured that the error you've got could be related to having containers in the environment. Unfortunately it appears that the upgrade step doesn't work if there are any containers, as it mistakenly thinks they're MAAS machines (and is rightly told that they don't exist in MAAS). It'd be great to get confirmation that you do have containers in your environment. Yes! We are using containers, and they are present on machine 0. Actually, one of the reasons we are upgrading Juju is in the hopes to resolve another issue which seems related to having containers on machine 0, which was that the cloud-init user-data that Juju has started sending to MAAS was using the IP address of the LXC bridge for the state server instead of the real IP address, so a machine we were trying to enroll into Juju hasn’t been able to finish because it can’t contact the Juju state server. I saw some bug fix changes that went into the 1.22 version that I thought might fix that problem, which is why I was trying the upgrade. Anyway, it sounds like what you are saying is that the upgrade state database migration step is iterating through a list that contains all of the MAAS machines as well as the containers when it should just be iterating through the MAAS machines themselves? Now, this could be repaired without *too* much trouble I think. That upgrade step is the last one to upgrade to 1.22 completely. So, options are: 1. Hand-edit the agent's config file so it thinks it's finished upgrading. 2. Fix Mongo so the upgrade step thinks it has nothing to do. 3. Wait for a fix to Juju. I'm not sure how long that'll take, as we're all about to travel. I've just filed: https://bugs.launchpad.net/juju-core/+bug/1441478 Thanks for filing the bug! I’ve subscribed to it and am happy to move this discussion there. I’ll monitor both places. #1 will work, but will mean that you'll have no availability-zone information in your instances. I'm not sure if that'd cause any immediate problems (Eric?), but it will mean that charms won't be able to take advantage of that information. #2 would be preferable; this would involve opening the Juju state database with the mongo shell, and updating all documents in the instancedata collection so that they have availzone fields. I’ll take a look at the Juju state mongodb, and take a backup in preparation for making changes, but I’ll hold off on making them for now. Should there be a default value entered for “availzone” or is an empty value fine? My reading of the code (I think the relevant bit is: https://github.com/juju/juju/blob/1.22/state/upgrades.go#L559-582) is that just having the availzone key should be fine to convince the upgrade it is done, but I’m not sure whether the availzone value might be used during operations (will it safely be ignored for containers and/or is the empty string a safe value to set it to?). Also, it seems likely that the upgrade will have bailed out as soon as the interator hit the first container, meaning it may not have properly set the availzone for all of our MAAS machines. I suppose I could hope that it at least did machine 0 and use that value for the other MAAS machines (in any case, we only have one availability zone). Cheers, Josh. -- This email is sent on behalf of Genomics Limited, a limited company registered in England and Wales with registered number 8839972, VAT registered number 189 2635 65 and registered office at 24 Cornhill, London, EC3V 3ND, United Kingdom. The contents of this e-mail and any attachments are confidential to the intended recipient. If you are not the intended recipient please do not use or publish its contents, contact Genomics Limited immediately at i...@genomicsltd.com then delete. You may not copy, forward, use or disclose the contents of this email to anybody else if you are not the intended recipient. Emails are not secure and may contain viruses. -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: failure upgrading Juju from 1.21.1 to 1.22.0 on trusty
On Wed, Apr 8, 2015 at 3:24 PM, John Meinel j...@arbash-meinel.com wrote:
upgrade step set AvailZone in instanceData failed: no instances found
seems suspicious.
What version of MAAS are you running?
It is possible that it would come before MAAS availability zones?
Certainly we couldn't really find 0 instances in MAAS because the machine
this is running on has to be at least one of them.
Unfortunately a few people are on vacation this week in preparation for
our sprint next week, but hopefully we can still get ahold of the people
who were working on this upgrade step.
John
=:-
On Wed, Apr 8, 2015 at 4:49 AM, Joshua Randall
josh.rand...@genomicsltd.com wrote:
I ran `juju upgrade-juju` today to upgrade a MAAS environment to Juju
version 1.22.0 and now `juju status` says that the upgrade has failed (and
thus we have limited access to the state server since an upgrade is in
progress). What can I do to manually complete the upgrade?
`juju status` shows the following error:
environment: maas
machines:
0:
agent-state: error
agent-state-info: 'upgrade to 1.22.0 failed (giving up): set
AvailZone in instanceData:
no instances found'
agent-version: 1.22.0
...
While '/var/log/juju/machine-0.log’ on machine 0 has:
2015-04-08 00:03:16 DEBUG juju.provider.maas environprovider.go:32
opening environment maas.
2015-04-08 00:03:16 ERROR juju.upgrade upgrade.go:134 upgrade step set
AvailZone in instanceData failed: no instances found
2015-04-08 00:03:16 ERROR juju.cmd.jujud upgrade.go:360 upgrade from
1.21.1 to 1.22.0 for machine-0 failed (will retry): set AvailZone in
instanceData: no instances found
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:265 - [3]
machine-0
{RequestId:22,Type:Machiner,Request:Life,Params:{Entities:[{Tag:machine-0}]}}
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:272 - [3]
machine-0 410.699us
{RequestId:22,Response:{Results:[{Life:alive,Error:null}]}}
Machiner[].Life
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:265 - [3]
machine-0
{RequestId:23,Type:Machiner,Request:SetStatus,Params:{Entities:[{Tag:machine-0,Status:
error,Info:upgrade to 1.22.0 failed (will retry): set AvailZone in
instanceData: no instances found,Data:null}]}}
I would very much appreciate any pointers in the right direction on how
to go about resolving this… any ideas?
I'd want to hear from others on this matter, as my experience with
repairing upgrades is limited.
I recommend you don't take any action on the following immediately, until
others have weighed in.
John figured that the error you've got could be related to having
containers in the environment.
Unfortunately it appears that the upgrade step doesn't work if there are
any containers, as it
mistakenly thinks they're MAAS machines (and is rightly told that they
don't exist in MAAS). It'd
be great to get confirmation that you do have containers in your
environment.
Now, this could be repaired without *too* much trouble I think. That
upgrade step is the last one
to upgrade to 1.22 completely. So, options are:
1. Hand-edit the agent's config file so it thinks it's finished upgrading.
2. Fix Mongo so the upgrade step thinks it has nothing to do.
3. Wait for a fix to Juju. I'm not sure how long that'll take, as we're
all about to travel. I've just filed:
https://bugs.launchpad.net/juju-core/+bug/1441478
#1 will work, but will mean that you'll have no availability-zone
information in your instances.
I'm not sure if that'd cause any immediate problems (Eric?), but it will
mean that charms won't
be able to take advantage of that information.
#2 would be preferable; this would involve opening the Juju state database
with the mongo
shell, and updating all documents in the instancedata collection so that
they have availzone
fields.
Cheers,
Andrew
--
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
Re: failure upgrading Juju from 1.21.1 to 1.22.0 on trusty
MAAS version is 1.5.4 (Ubuntu package 1.5.4+bzr2294-0ubuntu1.2).
Cheers,
Josh.
On 8 Apr 2015, at 08:24, John Meinel j...@arbash-meinel.com wrote:
upgrade step set AvailZone in instanceData failed: no instances found seems
suspicious.
What version of MAAS are you running?
It is possible that it would come before MAAS availability zones?
Certainly we couldn't really find 0 instances in MAAS because the machine
this is running on has to be at least one of them.
Unfortunately a few people are on vacation this week in preparation for our
sprint next week, but hopefully we can still get ahold of the people who were
working on this upgrade step.
John
=:-
On Wed, Apr 8, 2015 at 4:49 AM, Joshua Randall josh.rand...@genomicsltd.com
wrote:
I ran `juju upgrade-juju` today to upgrade a MAAS environment to Juju version
1.22.0 and now `juju status` says that the upgrade has failed (and thus we
have limited access to the state server since an upgrade is in progress).
What can I do to manually complete the upgrade?
`juju status` shows the following error:
environment: maas
machines:
0:
agent-state: error
agent-state-info: 'upgrade to 1.22.0 failed (giving up): set AvailZone
in instanceData:
no instances found'
agent-version: 1.22.0
...
While '/var/log/juju/machine-0.log’ on machine 0 has:
2015-04-08 00:03:16 DEBUG juju.provider.maas environprovider.go:32 opening
environment maas.
2015-04-08 00:03:16 ERROR juju.upgrade upgrade.go:134 upgrade step set
AvailZone in instanceData failed: no instances found
2015-04-08 00:03:16 ERROR juju.cmd.jujud upgrade.go:360 upgrade from 1.21.1
to 1.22.0 for machine-0 failed (will retry): set AvailZone in
instanceData: no instances found
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:265 - [3] machine-0
{RequestId:22,Type:Machiner,Request:Life,Params:{Entities:[{Tag:machine-0}]}}
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:272 - [3] machine-0
410.699us
{RequestId:22,Response:{Results:[{Life:alive,Error:null}]}}
Machiner[].Life
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:265 - [3] machine-0
{RequestId:23,Type:Machiner,Request:SetStatus,Params:{Entities:[{Tag:machine-0,Status:
error,Info:upgrade to 1.22.0 failed (will retry): set AvailZone in
instanceData: no instances found,Data:null}]}}
I would very much appreciate any pointers in the right direction on how to go
about resolving this… any ideas?
Cheers,
Josh.
--
This email is sent on behalf of Genomics Limited, a limited company
registered in England and Wales with registered number 8839972, VAT
registered number 189 2635 65 and registered office at 24 Cornhill, London,
EC3V 3ND, United Kingdom.
The contents of this e-mail and any attachments are confidential to the
intended recipient. If you are not the intended recipient please do not use
or publish its contents, contact Genomics Limited immediately at
i...@genomicsltd.com then delete. You may not copy, forward, use or
disclose the contents of this email to anybody else if you are not the
intended recipient. Emails are not secure and may contain viruses.
--
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
--
This email is sent on behalf of Genomics Limited, a limited company
registered in England and Wales with registered number 8839972, VAT
registered number 189 2635 65 and registered office at 24 Cornhill, London,
EC3V 3ND, United Kingdom.
The contents of this e-mail and any attachments are confidential to the
intended recipient. If you are not the intended recipient please do not use
or publish its contents, contact Genomics Limited immediately at
i...@genomicsltd.com then delete. You may not copy, forward, use or
disclose the contents of this email to anybody else if you are not the
intended recipient. Emails are not secure and may contain viruses.
--
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
Re: failure upgrading Juju from 1.21.1 to 1.22.0 on trusty
upgrade step set AvailZone in instanceData failed: no instances found
seems suspicious.
What version of MAAS are you running?
It is possible that it would come before MAAS availability zones?
Certainly we couldn't really find 0 instances in MAAS because the machine
this is running on has to be at least one of them.
Unfortunately a few people are on vacation this week in preparation for our
sprint next week, but hopefully we can still get ahold of the people who
were working on this upgrade step.
John
=:-
On Wed, Apr 8, 2015 at 4:49 AM, Joshua Randall josh.rand...@genomicsltd.com
wrote:
I ran `juju upgrade-juju` today to upgrade a MAAS environment to Juju
version 1.22.0 and now `juju status` says that the upgrade has failed (and
thus we have limited access to the state server since an upgrade is in
progress). What can I do to manually complete the upgrade?
`juju status` shows the following error:
environment: maas
machines:
0:
agent-state: error
agent-state-info: 'upgrade to 1.22.0 failed (giving up): set
AvailZone in instanceData:
no instances found'
agent-version: 1.22.0
...
While '/var/log/juju/machine-0.log’ on machine 0 has:
2015-04-08 00:03:16 DEBUG juju.provider.maas environprovider.go:32
opening environment maas.
2015-04-08 00:03:16 ERROR juju.upgrade upgrade.go:134 upgrade step set
AvailZone in instanceData failed: no instances found
2015-04-08 00:03:16 ERROR juju.cmd.jujud upgrade.go:360 upgrade from
1.21.1 to 1.22.0 for machine-0 failed (will retry): set AvailZone in
instanceData: no instances found
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:265 - [3]
machine-0
{RequestId:22,Type:Machiner,Request:Life,Params:{Entities:[{Tag:machine-0}]}}
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:272 - [3]
machine-0 410.699us
{RequestId:22,Response:{Results:[{Life:alive,Error:null}]}}
Machiner[].Life
2015-04-08 00:03:16 DEBUG juju.apiserver apiserver.go:265 - [3]
machine-0
{RequestId:23,Type:Machiner,Request:SetStatus,Params:{Entities:[{Tag:machine-0,Status:
error,Info:upgrade to 1.22.0 failed (will retry): set AvailZone in
instanceData: no instances found,Data:null}]}}
I would very much appreciate any pointers in the right direction on how to
go about resolving this… any ideas?
Cheers,
Josh.
--
This email is sent on behalf of Genomics Limited, a limited company
registered in England and Wales with registered number 8839972, VAT
registered number 189 2635 65 and registered office at 24 Cornhill, London,
EC3V 3ND, United Kingdom.
The contents of this e-mail and any attachments are confidential to the
intended recipient. If you are not the intended recipient please do not use
or publish its contents, contact Genomics Limited immediately at
i...@genomicsltd.com then delete. You may not copy, forward, use or
disclose the contents of this email to anybody else if you are not the
intended recipient. Emails are not secure and may contain viruses.
--
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
--
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
