Re: Openstack User Feedback

[Ryan Beisner]

Touched base on IRC out of band with this, just circling back here as well
to confirm that we're working to reproduce and advise.

Thanks again,

Ryan

On Wed, Apr 4, 2018 at 11:04 AM, James Beedy  wrote:

> forgot one: https://bugs.launchpad.net/charms.ceph/+bug/1761230
>
> On Wed, Apr 4, 2018 at 8:19 AM, James Beedy  wrote:
>
>> Here are the bugs:
>>
>> 1) https://bugs.launchpad.net/charms.ceph/+bug/1761214
>> 2) https://bugs.launchpad.net/charms.ceph/+bug/1761208
>> 3) https://bugs.launchpad.net/charm-ceph-fs/+bug/1753640
>>
>>
>> On Wed, Apr 4, 2018 at 5:52 AM, James Beedy  wrote:
>>
>>> Hello,
>>>
>>> @chrismacnaughton @jamespage @beisner @openstack-charmers ceph <->
>>> openstack such a broken bone right now.
>>>
>>> Please fix.
>>>
>>> Here is my bundle https://pastebin.com/gEUYaiFh
>>>
>>>
>>> (From IRC)
>>> 05:29  heres my deploy, looks great
>>> 05:29  https://pastebin.com/a2iNrBxv
>>> 
>>> jamesbeedy - Pastebin.com
>>> 
>>> 05:30  following a (what appears to be successful) deploy, I only
>>> see a 'glance' pool in ceph, and its in warning due to pg_num > pgp_num
>>> 05:31  increasing pgp_num gets me a healthy status
>>> 05:32  follow by another health warning
>>> https://paste.ubuntu.com/p/zD7HxMz6s6/
>>> Ubuntu Pastebin
>>> 
>>> 05:32  running `sudo ceph osd pool application enable glance rbd`
>>> got me back to healthy on my 'glance' pool
>>> 05:33  these are the two first papercuts
>>> 05:33  next - Manager for service cinder-volume cinder@cinder-ceph
>>> is reporting problems, not sending heartbeat. Service will appear "down".
>>> 05:34   the only pool I see following a deploy is 'glance'
>>> https://paste.ubuntu.com/p/Gg5G3Bmhq3/
>>> Ubuntu Pastebin
>>> 
>>> 05:36  when I try to create an instance in openstack I get the
>>> good ol' - Error: Failed to perform requested operation on instance "aset",
>>> the instance has an error status: Please try again later [Error: Build of
>>> instance 13e170cd-6aea-43ed-9ca1-de2ae62c3118 aborted: Volume
>>> 72e36907-81df-446c-b580-78bd96134ce0 did not finish being created even
>>> after we waited 0 seconds or 1 attempts. And its status is error.].
>>> 05:36  possibly there is a bunch of post deploy configuration that
>>> need be done here?
>>>
>>>
>>> Thanks
>>>
>>
>>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: More juju upgrade-juju failings

[Sandor Zeestraten]

Glad to help. I reported http://pad.lv/1761288 regarding the ability to
abort/rollback broken upgrades in order to get out of sticky upgrade
situations.
Also hope to see a clean up of the UX and the addition of actual dry-run
capabilities in http://pad.lv/1638714

We're looking forward to upgrading from 2.3 to 2.4 with ease and confidence!

Regards
--
Sandor Zeestraten

On Wed, Apr 4, 2018 at 8:13 PM, Nicholas Skaggs <
nicholas.ska...@canonical.com> wrote:

> Sandor, thanks for this perspective! It was really helpful to see how
> upgrades went for you in real life, and more importantly, that 2.3.x seems
> to have gone smoothly. We'll be carefully watching and monitoring 2.3->2.4
> upgrades as the release draws nearer.
>
> Nicholas
>
> On 04/01/2018 04:04 AM, Sandor Zeestraten wrote:
>
>> Hi Nicholas,
>>
>> Thanks for the input. I wrote up a short blog post about our experiences
>> going from 2.1 to 2.3. Hopefully it provides some feedback and can be
>> helpful for others in the same position.
>>
>> http://zeestrataca.com/posts/upgrading-juju/ <
>> http://zeestrataca.com/posts/upgrading-juju/>
>>
>> Regards
>> --
>> Sandor Zeestraten
>>
>>
>> On Thu, Mar 22, 2018 at 4:39 PM, Nicholas Skaggs <
>> nicholas.ska...@canonical.com >
>> wrote:
>>
>> Sandor, re:sharing experiences, if you want to frame some
>> scenarios you've had trouble with, please feel free to share
>> those. Distilling it down into a repeatable deployment -> upgrade
>> will help us understand and account for it. As Tim mentioned,
>> tools like juju-upgrader are generally candidates for
>> incorporation into juju itself, provided they prove valuable to
>> the community at large. We always welcome any PR's, but especially
>> so for improvements that add this functionality.
>>
>> Nicholas
>>
>> On 03/21/2018 08:43 PM, Tim Penhey wrote:
>>
>> Hi Sandor,
>>
>> Streamlining upgrades is definitely on the team's road-map. We
>> are aware
>> of the juju-upgrader plugin, and will be looking to
>> incorporate some of
>> that functionality into the core codebase.
>>
>> The core team has worked on better upgrade testing as part of
>> our CI,
>> which enables more test scenarios to help discover issues
>> between more
>> versions.
>>
>> Cheers,
>> Tim
>>
>> On 22/03/18 05:32, Sandor Zeestraten wrote:
>>
>> Is this shared google doc open for external contributors?
>>
>> After spending a while upgrading our 2.1.x environments to
>> 2.3.x and
>> hitting some bugs along the way in staging (see below),
>> I'd agree that
>> the process needs a bit of love and wouldn't mind sharing
>> some experiences.
>>
>> Rick mentioned https://launchpad.net/juju-upgrader
>>  on the Juju show a
>> couple of episodes back, but I haven't seen it mentioned
>> anywhere else
>> yet. Are those tools supposed to deal with some of these
>> issues and
>> eventually be rolled into juju-core?
>>
>> https://bugs.launchpad.net/juju/+bug/1746265
>> 
>> https://bugs.launchpad.net/juju/+bug/1748294
>> 
>> https://bugs.launchpad.net/juju/+bug/1697936
>> 
>>
>> Regards
>> --
>> Sandor Zeestraten
>>
>> On Wed, Feb 28, 2018 at 8:30 AM, Mark Shuttleworth
>> mailto:m...@ubuntu.com>
>> >> wrote:
>>
>>
>>  I think this UX on the upgrade process has obvious
>> problems. Nobody
>>  should have to guess at what to do, in which sequence.
>>
>>  Can I suggest that we have a shared Google doc to
>> mock up a nice
>>  experience starting with the simple command 'juju
>> upgrade' which then
>>  walks people through the process, including the
>> distinction between
>>  upgrading charms, agents and controllers, as well as
>> the ability to do
>>  aerospace-grade upgrades through live migration to
>> newer controllers?
>>
>>  Mark
>>
>>  On 02/27/2018 11:26 PM, Tim Penhey wrote:
>>  > Hi Daniel,
>>  >
>>  > The issue here is that you are upgrading the
>> default model, not the
>>  > controller model itself.
>>  >
>>  > I think we could make the error messaging more
>> cle

Re: More juju upgrade-juju failings

[Nicholas Skaggs]

Sandor, thanks for this perspective! It was really helpful to see how 
upgrades went for you in real life, and more importantly, that 2.3.x 
seems to have gone smoothly. We'll be carefully watching and monitoring 
2.3->2.4 upgrades as the release draws nearer.


Nicholas

On 04/01/2018 04:04 AM, Sandor Zeestraten wrote:

Hi Nicholas,

Thanks for the input. I wrote up a short blog post about our 
experiences going from 2.1 to 2.3. Hopefully it provides some feedback 
and can be helpful for others in the same position.


http://zeestrataca.com/posts/upgrading-juju/ 



Regards
--
Sandor Zeestraten

On Thu, Mar 22, 2018 at 4:39 PM, Nicholas Skaggs 
mailto:nicholas.ska...@canonical.com>> 
wrote:


Sandor, re:sharing experiences, if you want to frame some
scenarios you've had trouble with, please feel free to share
those. Distilling it down into a repeatable deployment -> upgrade
will help us understand and account for it. As Tim mentioned,
tools like juju-upgrader are generally candidates for
incorporation into juju itself, provided they prove valuable to
the community at large. We always welcome any PR's, but especially
so for improvements that add this functionality.

Nicholas

On 03/21/2018 08:43 PM, Tim Penhey wrote:

Hi Sandor,

Streamlining upgrades is definitely on the team's road-map. We
are aware
of the juju-upgrader plugin, and will be looking to
incorporate some of
that functionality into the core codebase.

The core team has worked on better upgrade testing as part of
our CI,
which enables more test scenarios to help discover issues
between more
versions.

Cheers,
Tim

On 22/03/18 05:32, Sandor Zeestraten wrote:

Is this shared google doc open for external contributors?

After spending a while upgrading our 2.1.x environments to
2.3.x and
hitting some bugs along the way in staging (see below),
I'd agree that
the process needs a bit of love and wouldn't mind sharing
some experiences.

Rick mentioned https://launchpad.net/juju-upgrader
 on the Juju show a
couple of episodes back, but I haven't seen it mentioned
anywhere else
yet. Are those tools supposed to deal with some of these
issues and
eventually be rolled into juju-core?

https://bugs.launchpad.net/juju/+bug/1746265

https://bugs.launchpad.net/juju/+bug/1748294

https://bugs.launchpad.net/juju/+bug/1697936


Regards
--
Sandor Zeestraten

On Wed, Feb 28, 2018 at 8:30 AM, Mark Shuttleworth
mailto:m...@ubuntu.com>
>> wrote:


     I think this UX on the upgrade process has obvious
problems. Nobody
     should have to guess at what to do, in which sequence.

     Can I suggest that we have a shared Google doc to
mock up a nice
     experience starting with the simple command 'juju
upgrade' which then
     walks people through the process, including the
distinction between
     upgrading charms, agents and controllers, as well as
the ability to do
     aerospace-grade upgrades through live migration to
newer controllers?

     Mark

     On 02/27/2018 11:26 PM, Tim Penhey wrote:
     > Hi Daniel,
     >
     > The issue here is that you are upgrading the
default model, not the
     > controller model itself.
     >
     > I think we could make the error messaging more
clear, and also
     have the
     > command also check the controller version before
showing a lot of
     > baffling output.
     >
     > What you need to do is upgrade the controller model
first, so either
     > switch to it or run:
     >
     >   juju upgrade-juju -m controller --agent-version 2.3.3
     >
     > Cheers,
     > Tim
     >
     > On 28/02/18 11:19, Daniel Bidwell wrote:
     >> I am running on juju 2.3.3-xenial-amd64 and my
controllers are
     running
     >> in VMware Vsphere with version 2.3.2.  I thought
that it would be a
  

Re: Openstack User Feedback

[James Beedy]

forgot one: https://bugs.launchpad.net/charms.ceph/+bug/1761230

On Wed, Apr 4, 2018 at 8:19 AM, James Beedy  wrote:

> Here are the bugs:
>
> 1) https://bugs.launchpad.net/charms.ceph/+bug/1761214
> 2) https://bugs.launchpad.net/charms.ceph/+bug/1761208
> 3) https://bugs.launchpad.net/charm-ceph-fs/+bug/1753640
>
>
> On Wed, Apr 4, 2018 at 5:52 AM, James Beedy  wrote:
>
>> Hello,
>>
>> @chrismacnaughton @jamespage @beisner @openstack-charmers ceph <->
>> openstack such a broken bone right now.
>>
>> Please fix.
>>
>> Here is my bundle https://pastebin.com/gEUYaiFh
>>
>>
>> (From IRC)
>> 05:29  heres my deploy, looks great
>> 05:29  https://pastebin.com/a2iNrBxv
>> 
>> jamesbeedy - Pastebin.com
>> 
>> 05:30  following a (what appears to be successful) deploy, I only
>> see a 'glance' pool in ceph, and its in warning due to pg_num > pgp_num
>> 05:31  increasing pgp_num gets me a healthy status
>> 05:32  follow by another health warning
>> https://paste.ubuntu.com/p/zD7HxMz6s6/
>> Ubuntu Pastebin
>> 
>> 05:32  running `sudo ceph osd pool application enable glance rbd`
>> got me back to healthy on my 'glance' pool
>> 05:33  these are the two first papercuts
>> 05:33  next - Manager for service cinder-volume cinder@cinder-ceph
>> is reporting problems, not sending heartbeat. Service will appear "down".
>> 05:34   the only pool I see following a deploy is 'glance'
>> https://paste.ubuntu.com/p/Gg5G3Bmhq3/
>> Ubuntu Pastebin
>> 
>> 05:36  when I try to create an instance in openstack I get the good
>> ol' - Error: Failed to perform requested operation on instance "aset", the
>> instance has an error status: Please try again later [Error: Build of
>> instance 13e170cd-6aea-43ed-9ca1-de2ae62c3118 aborted: Volume
>> 72e36907-81df-446c-b580-78bd96134ce0 did not finish being created even
>> after we waited 0 seconds or 1 attempts. And its status is error.].
>> 05:36  possibly there is a bunch of post deploy configuration that
>> need be done here?
>>
>>
>> Thanks
>>
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Openstack User Feedback

[James Beedy]

Here are the bugs:

1) https://bugs.launchpad.net/charms.ceph/+bug/1761214
2) https://bugs.launchpad.net/charms.ceph/+bug/1761208
3) https://bugs.launchpad.net/charm-ceph-fs/+bug/1753640


On Wed, Apr 4, 2018 at 5:52 AM, James Beedy  wrote:

> Hello,
>
> @chrismacnaughton @jamespage @beisner @openstack-charmers ceph <->
> openstack such a broken bone right now.
>
> Please fix.
>
> Here is my bundle https://pastebin.com/gEUYaiFh
>
>
> (From IRC)
> 05:29  heres my deploy, looks great
> 05:29  https://pastebin.com/a2iNrBxv
> 
> jamesbeedy - Pastebin.com
> 
> 05:30  following a (what appears to be successful) deploy, I only
> see a 'glance' pool in ceph, and its in warning due to pg_num > pgp_num
> 05:31  increasing pgp_num gets me a healthy status
> 05:32  follow by another health warning https://paste.ubuntu.com/p/
> zD7HxMz6s6/
> Ubuntu Pastebin
> 
> 05:32  running `sudo ceph osd pool application enable glance rbd`
> got me back to healthy on my 'glance' pool
> 05:33  these are the two first papercuts
> 05:33  next - Manager for service cinder-volume cinder@cinder-ceph
> is reporting problems, not sending heartbeat. Service will appear "down".
> 05:34   the only pool I see following a deploy is 'glance'
> https://paste.ubuntu.com/p/Gg5G3Bmhq3/
> Ubuntu Pastebin
> 
> 05:36  when I try to create an instance in openstack I get the good
> ol' - Error: Failed to perform requested operation on instance "aset", the
> instance has an error status: Please try again later [Error: Build of
> instance 13e170cd-6aea-43ed-9ca1-de2ae62c3118 aborted: Volume
> 72e36907-81df-446c-b580-78bd96134ce0 did not finish being created even
> after we waited 0 seconds or 1 attempts. And its status is error.].
> 05:36  possibly there is a bunch of post deploy configuration that
> need be done here?
>
>
> Thanks
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Openstack User Feedback

[James Beedy]

Hello,

@chrismacnaughton @jamespage @beisner @openstack-charmers ceph <->
openstack such a broken bone right now.

Please fix.

Here is my bundle https://pastebin.com/gEUYaiFh


(From IRC)
05:29  heres my deploy, looks great
05:29  https://pastebin.com/a2iNrBxv

jamesbeedy - Pastebin.com

05:30  following a (what appears to be successful) deploy, I only see
a 'glance' pool in ceph, and its in warning due to pg_num > pgp_num
05:31  increasing pgp_num gets me a healthy status
05:32  follow by another health warning
https://paste.ubuntu.com/p/zD7HxMz6s6/
Ubuntu Pastebin

05:32  running `sudo ceph osd pool application enable glance rbd` got
me back to healthy on my 'glance' pool
05:33  these are the two first papercuts
05:33  next - Manager for service cinder-volume cinder@cinder-ceph is
reporting problems, not sending heartbeat. Service will appear "down".
05:34   the only pool I see following a deploy is 'glance'
https://paste.ubuntu.com/p/Gg5G3Bmhq3/
Ubuntu Pastebin

05:36  when I try to create an instance in openstack I get the good
ol' - Error: Failed to perform requested operation on instance "aset", the
instance has an error status: Please try again later [Error: Build of
instance 13e170cd-6aea-43ed-9ca1-de2ae62c3118 aborted: Volume
72e36907-81df-446c-b580-78bd96134ce0 did not finish being created even
after we waited 0 seconds or 1 attempts. And its status is error.].
05:36  possibly there is a bunch of post deploy configuration that
need be done here?


Thanks
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju