[Openstack-operators] leaving Openstack mailing lists

[Saverio Proto]

Hello,

I will be leaving this mailing list in a few days.

I am going to a new job and I will not be involved with Openstack at
least in the short term future.
Still, it was great working with the Openstack community in the past few years.

If you need to reach me about any bug/patch/review that I submitted in
the past, just write directly to my email. I will try to give answers.

Cheers

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Routing deployments + Storage networks

[Saverio Proto]

Hello,

we route the Ceph storage network in the same fabric. We did not have
problems with that so far.

Cheers

Saverio

Il giorno gio 16 ago 2018 alle ore 10:43 Paul Browne 
ha scritto:
>
> Hi operators,
>
> I had a quick question for those operators who use a routed topology for 
> their OpenStack deployments, whether routed spine-leaf or routed underlay 
> providing L2 connectivity in tunnels;
>
> Where using one, would the storage network (e.g. Ceph public network) also be 
> routed on the same fabric, or would separate fabric be employed here to 
> reduce hops?
>
> Many thanks,
> Paul Browne
>
> --
> ***
> Paul Browne
> Research Computing Platforms
> University Information Services
> Roger Needham Building
> JJ Thompson Avenue
> University of Cambridge
> Cambridge
> United Kingdom
> E-Mail: pf...@cam.ac.uk
> Tel: 0044-1223-746548
> ***
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova] ask deployment question

[Saverio Proto]

Hello Rambo,

you can find information about other deployments reading the User Survey:
https://www.openstack.org/user-survey/survey-2018/landing

For blog posts with experiences from other operators check out:
https://superuser.openstack.org/ and http://planet.openstack.org/

Cheers

Saverio

Il giorno gio 16 ago 2018 alle ore 11:59 Rambo 
ha scritto:
>
> Hi,all
>I have some questions about deploy the large scale openstack 
> cloud.Such as
>1.Only in one region situation,How many physical machines are the 
> biggest deployment scale in our community?
>Can you tell me more about these combined with own practice? Would you 
> give me some methods to learn it?Such as the website,blog and so on. Thank 
> you very much!Looking forward to hearing from you.
>
>
>
>
>
>
>
>
> Best Regards
> Rambo
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack Version discovery with the cli client.

[Saverio Proto]

Thanks !

I think the command I was looking for is:
"openstack versions show"

But for example for Neutron I get just version v2.0 from Newton to
Pike, that tells me very little.

The use case is when testing Kubernetes on Openstack, a lot of
kubernetes users cannot tell easily the version of Openstack they are
testing on. Because things like the LBaaS are so different between
Openstack releases that version v2.0 tells too little.
Often it is good to know what is the version of the Openstack cloud to
identify bugs on launchpad.

Cheers,

Saverio




Il giorno mar 7 ago 2018 alle ore 15:48 George Mihaiescu
 ha scritto:
>
> Hi Saverio,
>
> I think only the API versions supported by some of the endpoint are 
> discoverable, as described here: 
> https://wiki.openstack.org/wiki/VersionDiscovery
>
> curl https://x.x.x.x:9292/image
> curl https://x.x.x.x:8774/compute
>
>
> Cheers,
> George
>
> On Tue, Aug 7, 2018 at 9:30 AM, Saverio Proto  wrote:
>>
>> Hello Jimmy,
>>
>> thanks for your help. If I understand correctly the answer you linked,
>> that helps if you operate the cloud and you have access to the
>> servers. Then of course you can call nova-manage.
>>
>> But being a user of a public cloud without having access the the
>> infrastructure servers ... how do you do that ?
>>
>> thanks
>>
>> Saverio
>>
>>
>>
>> Il giorno mar 7 ago 2018 alle ore 15:09 Jimmy McArthur
>>  ha scritto:
>> >
>> > Hey Saverio,
>> >
>> > This answer from ask.openstack.org should have what you're looking for:
>> > https://ask.openstack.org/en/question/45513/how-to-find-out-which-version-of-openstack-is-installed/at
>> >
>> > Once you get the release number, you have to look it up here to match
>> > the release date: https://releases.openstack.org/
>> >
>> > I had to use this the other day when taking the COA.
>> >
>> > Cheers,
>> > Jimmy
>> >
>> > Saverio Proto wrote:
>> > > Hello,
>> > >
>> > > This is maybe a super trivial question bit I have to admit I could not
>> > > figure it out.
>> > >
>> > > Can the user with the openstack cli client discover the version of
>> > > Openstack that is running ?
>> > >
>> > > For example in kubernetes the kubectl version command returns the
>> > > version of the client and the version of the cluster.
>> > >
>> > > For Openstack I never managed to discover the backend version, and
>> > > this could be useful when using public clouds.
>> > >
>> > > Anyone knows how to do that ?
>> > >
>> > > thanks
>> > >
>> > > Saverio
>> > >
>> > > ___
>> > > OpenStack-operators mailing list
>> > > OpenStack-operators@lists.openstack.org
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Openstack Version discovery with the cli client.

[Saverio Proto]

Hello Jimmy,

thanks for your help. If I understand correctly the answer you linked,
that helps if you operate the cloud and you have access to the
servers. Then of course you can call nova-manage.

But being a user of a public cloud without having access the the
infrastructure servers ... how do you do that ?

thanks

Saverio



Il giorno mar 7 ago 2018 alle ore 15:09 Jimmy McArthur
 ha scritto:
>
> Hey Saverio,
>
> This answer from ask.openstack.org should have what you're looking for:
> https://ask.openstack.org/en/question/45513/how-to-find-out-which-version-of-openstack-is-installed/at
>
> Once you get the release number, you have to look it up here to match
> the release date: https://releases.openstack.org/
>
> I had to use this the other day when taking the COA.
>
> Cheers,
> Jimmy
>
> Saverio Proto wrote:
> > Hello,
> >
> > This is maybe a super trivial question bit I have to admit I could not
> > figure it out.
> >
> > Can the user with the openstack cli client discover the version of
> > Openstack that is running ?
> >
> > For example in kubernetes the kubectl version command returns the
> > version of the client and the version of the cluster.
> >
> > For Openstack I never managed to discover the backend version, and
> > this could be useful when using public clouds.
> >
> > Anyone knows how to do that ?
> >
> > thanks
> >
> > Saverio
> >
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Openstack Version discovery with the cli client.

[Saverio Proto]

Hello,

This is maybe a super trivial question bit I have to admit I could not
figure it out.

Can the user with the openstack cli client discover the version of
Openstack that is running ?

For example in kubernetes the kubectl version command returns the
version of the client and the version of the cluster.

For Openstack I never managed to discover the backend version, and
this could be useful when using public clouds.

Anyone knows how to do that ?

thanks

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Neutron not adding iptables rules for metadata agent

[Saverio Proto]

Hello,

I would suggest to open a bug on launchpad to track this issue.

thank you

Saverio

2018-06-18 12:19 GMT+02:00 Radu Popescu | eMAG, Technology
:
> Hi,
>
> We're using Openstack Ocata, deployed using Openstack Ansible v15.1.7.
> Neutron server is v10.0.3.
> I can see enable_isolated_metadata and enable_metadata_network only used for
> isolated networks that don't have a router which is not our case.
> Also, I checked all namespaces on all our novas and only affected 6 out of
> 66 ..and only 1 namespace / nova. Seems like isolated case that doesn't
> happen very often.
>
> Can it be RabbitMQ? I'm not sure where to check.
>
> Thanks,
> Radu
>
> On Fri, 2018-06-15 at 17:11 +0200, Saverio Proto wrote:
>
> Hello Radu,
>
>
> yours look more or less like a bug report. This you check existing
>
> open bugs for neutron ? Also what version of openstack are you running
>
> ?
>
>
> how did you configure enable_isolated_metadata and
>
> enable_metadata_network options ?
>
>
> Saverio
>
>
> 2018-06-13 12:45 GMT+02:00 Radu Popescu | eMAG, Technology
>
> :
>
> Hi all,
>
>
> So, I'm having the following issue. I'm creating a VM with floating IP.
>
> Everything is fine, namespace is there, postrouting and prerouting from the
>
> internal IP to the floating IP are there. The only rules missing are the
>
> rules to access metadata service:
>
>
> -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp
>
> --dport 80 -j REDIRECT --to-ports 9697
>
> -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp
>
> --dport 80 -j MARK --set-xmark 0x1/0x
>
>
> (this is taken from another working namespace with iptables-save)
>
>
> Forgot to mention, VM is booting ok, I have both the default route and the
>
> one for the metadata service (cloud-init is running at boot time):
>
> [   57.150766] cloud-init[892]: ci-info:
>
> ++--+--+---+---+---+
>
> [   57.150997] cloud-init[892]: ci-info: | Device |  Up  |   Address|
>
> Mask | Scope | Hw-Address|
>
> [   57.151219] cloud-init[892]: ci-info:
>
> ++--+--+---+---+---+
>
> [   57.151431] cloud-init[892]: ci-info: |  lo:   | True |  127.0.0.1   |
>
> 255.0.0.0   |   .   | . |
>
> [   57.151627] cloud-init[892]: ci-info: | eth0:  | True | 10.240.9.186 |
>
> 255.255.252.0 |   .   | fa:16:3e:43:d1:c2 |
>
> [   57.151815] cloud-init[892]: ci-info:
>
> ++--+--+---+---+---+
>
> [   57.152018] cloud-init[892]: ci-info:
>
> +++Route IPv4
>
> info
>
> [   57.152225] cloud-init[892]: ci-info:
>
> +---+-++-+---+---+
>
> [   57.152426] cloud-init[892]: ci-info: | Route |   Destination   |
>
> Gateway   | Genmask | Interface | Flags |
>
> [   57.152621] cloud-init[892]: ci-info:
>
> +---+-++-+---+---+
>
> [   57.152813] cloud-init[892]: ci-info: |   0   | 0.0.0.0 |
>
> 10.240.8.1 | 0.0.0.0 |eth0   |   UG  |
>
> [   57.153013] cloud-init[892]: ci-info: |   1   |10.240.1.0   |
>
> 0.0.0.0   |  255.255.255.0  |eth0   |   U   |
>
> [   57.153202] cloud-init[892]: ci-info: |   2   |10.240.8.0   |
>
> 0.0.0.0   |  255.255.252.0  |eth0   |   U   |
>
> [   57.153397] cloud-init[892]: ci-info: |   3   | 169.254.169.254 |
>
> 10.240.8.1 | 255.255.255.255 |eth0   |  UGH  |
>
> [   57.153579] cloud-init[892]: ci-info:
>
> +---+-++-+---+---+
>
>
> The extra route is there because the tenant has 2 subnets.
>
>
> Before adding those 2 rules manually, I had this coming from cloud-init:
>
>
> [  192.451801] cloud-init[892]: 2018-06-13 12:29:26,179 -
>
> url_helper.py[WARNING]: Calling
>
> 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]:
>
> request error [('Connection aborted.', error(113, 'No route to host'))]
>
> [  193.456805] cloud-init[892]: 2018-06-13 12:29:27,184 -
>
> url_helper.py[WARNING]: Calling
>
> 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]:
>
> request error [('Connection aborted.', error(113, 'No route to host'))]
>
> [  194.461592] cloud-init[892]: 2018-06-13 12:29:28,189 -
>
&

Re: [Openstack-operators] Neutron not adding iptables rules for metadata agent

[Saverio Proto]

Hello Radu,

yours look more or less like a bug report. This you check existing
open bugs for neutron ? Also what version of openstack are you running
?

how did you configure enable_isolated_metadata and
enable_metadata_network options ?

Saverio

2018-06-13 12:45 GMT+02:00 Radu Popescu | eMAG, Technology
:
> Hi all,
>
> So, I'm having the following issue. I'm creating a VM with floating IP.
> Everything is fine, namespace is there, postrouting and prerouting from the
> internal IP to the floating IP are there. The only rules missing are the
> rules to access metadata service:
>
> -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp
> --dport 80 -j REDIRECT --to-ports 9697
> -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp
> --dport 80 -j MARK --set-xmark 0x1/0x
>
> (this is taken from another working namespace with iptables-save)
>
> Forgot to mention, VM is booting ok, I have both the default route and the
> one for the metadata service (cloud-init is running at boot time):
> [   57.150766] cloud-init[892]: ci-info:
> ++--+--+---+---+---+
> [   57.150997] cloud-init[892]: ci-info: | Device |  Up  |   Address|
> Mask | Scope | Hw-Address|
> [   57.151219] cloud-init[892]: ci-info:
> ++--+--+---+---+---+
> [   57.151431] cloud-init[892]: ci-info: |  lo:   | True |  127.0.0.1   |
> 255.0.0.0   |   .   | . |
> [   57.151627] cloud-init[892]: ci-info: | eth0:  | True | 10.240.9.186 |
> 255.255.252.0 |   .   | fa:16:3e:43:d1:c2 |
> [   57.151815] cloud-init[892]: ci-info:
> ++--+--+---+---+---+
> [   57.152018] cloud-init[892]: ci-info:
> +++Route IPv4
> info
> [   57.152225] cloud-init[892]: ci-info:
> +---+-++-+---+---+
> [   57.152426] cloud-init[892]: ci-info: | Route |   Destination   |
> Gateway   | Genmask | Interface | Flags |
> [   57.152621] cloud-init[892]: ci-info:
> +---+-++-+---+---+
> [   57.152813] cloud-init[892]: ci-info: |   0   | 0.0.0.0 |
> 10.240.8.1 | 0.0.0.0 |eth0   |   UG  |
> [   57.153013] cloud-init[892]: ci-info: |   1   |10.240.1.0   |
> 0.0.0.0   |  255.255.255.0  |eth0   |   U   |
> [   57.153202] cloud-init[892]: ci-info: |   2   |10.240.8.0   |
> 0.0.0.0   |  255.255.252.0  |eth0   |   U   |
> [   57.153397] cloud-init[892]: ci-info: |   3   | 169.254.169.254 |
> 10.240.8.1 | 255.255.255.255 |eth0   |  UGH  |
> [   57.153579] cloud-init[892]: ci-info:
> +---+-++-+---+---+
>
> The extra route is there because the tenant has 2 subnets.
>
> Before adding those 2 rules manually, I had this coming from cloud-init:
>
> [  192.451801] cloud-init[892]: 2018-06-13 12:29:26,179 -
> url_helper.py[WARNING]: Calling
> 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]:
> request error [('Connection aborted.', error(113, 'No route to host'))]
> [  193.456805] cloud-init[892]: 2018-06-13 12:29:27,184 -
> url_helper.py[WARNING]: Calling
> 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]:
> request error [('Connection aborted.', error(113, 'No route to host'))]
> [  194.461592] cloud-init[892]: 2018-06-13 12:29:28,189 -
> url_helper.py[WARNING]: Calling
> 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]:
> request error [('Connection aborted.', error(113, 'No route to host'))]
> [  195.466441] cloud-init[892]: 2018-06-13 12:29:29,194 -
> url_helper.py[WARNING]: Calling
> 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [3/120s]:
> request error [('Connection aborted.', error(113, 'No route to host'))]
>
> I can see no errors in neither nova or neutron services.
> In the mean time, I've searched all our nova servers for this kind of
> behavior and we have 1 random namespace missing those rules on 6 of our 66
> novas.
>
> Any ideas would be greatly appreciated.
>
> Thanks,
> Radu
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev][publiccloud-wg][k8s][octavia] OpenStack Load Balancer APIs and K8s

[Saverio Proto]

Hello Flint,

what version of Kubernetes are you deploying on top of Openstack ?

are you using the external Openstack cloud controller ? I tested it an
it works only if you have at least v.1.10.3

Look at this page:
https://github.com/kubernetes/cloud-provider-openstack/tree/master/examples/loadbalancers

Please test that you can make a SSL termination on the loadbalancer,
describing it with Kubernetes yaml files. That is important for
production operation. Test also if you have downtime when you have to
renew SSL certificates.

You will also want to check that traffic that hits your pods has the
HTTP header X-Forwarded-For, or even better the IP packets you receive
at the Pods have the source IP address of the original client.

If needed test everything also with IPv6

I personally decided not to use Octavia, but to go for the Kubernetes
ingress-nginx
https://github.com/kubernetes/ingress-nginx

The key idea is that instead of Openstack controlling the LoadBalancer
having Octavia spinning up a VM running nginx, you have Kubernetes
controlling the LoadBalancer, running a nginx-container.
At the end you need a nginx to reverse proxy, you have to decided if
this resource is managed by Openstack or Kubernetes.

Keep in mind that if you go for a kubernetes ingress controller you
can avoid using nginx. There is already an alternative ha-proxy
implementation:
https://www.haproxy.com/blog/haproxy_ingress_controller_for_kubernetes/

Cheers,

Saverio

2018-05-28 19:09 GMT+02:00 Flint WALRUS :
> Hi everyone, I’m currently deploying Octavia as our global LBaaS for a lot
> of various workload such as Kubernetes ingress LB.
>
> We use Queens and plan to upgrade to rocky as soon as it reach the stable
> release and we use the native Octavia APIv2 (Not a neutron redirect etc).
>
> What do you need to know?
>
> Le lun. 28 mai 2018 à 14:50, Saverio Proto  a écrit :
>>
>> Hello Chris,
>>
>> I finally had the time to write about my deployment:
>>
>> https://cloudblog.switch.ch/2018/05/22/openstack-horizon-runs-on-kubernetes-in-production-at-switch/
>>
>> in this blog post I explain why I use the kubernetes nginx-ingress
>> instead of Openstack LBaaS.
>>
>> Cheers,
>>
>> Saverio
>>
>>
>> 2018-03-15 23:55 GMT+01:00 Chris Hoge :
>> > Hi everyone,
>> >
>> > I wanted to notify you of a thread I started in openstack-dev about the
>> > state
>> > of the OpenStack load balancer APIs and the difficulty in integrating
>> > them
>> > with Kubernetes. This in part directly relates to current public and
>> > private
>> > deployments, and any feedback you have would be appreciated. Especially
>> > feedback on which version of the load balancer APIs you deploy, and if
>> > you
>> > haven't moved on to Octavia, why.
>> >
>> >
>> > http://lists.openstack.org/pipermail/openstack-dev/2018-March/128399.html
>> > <http://lists.openstack.org/pipermail/openstack-dev/2018-March/128399.html>
>> >
>> > Thanks in advance,
>> > Chris
>> > ___
>> > OpenStack-operators mailing list
>> > OpenStack-operators@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev][publiccloud-wg][k8s][octavia] OpenStack Load Balancer APIs and K8s

[Saverio Proto]

Hello Chris,

I finally had the time to write about my deployment:
https://cloudblog.switch.ch/2018/05/22/openstack-horizon-runs-on-kubernetes-in-production-at-switch/

in this blog post I explain why I use the kubernetes nginx-ingress
instead of Openstack LBaaS.

Cheers,

Saverio


2018-03-15 23:55 GMT+01:00 Chris Hoge :
> Hi everyone,
>
> I wanted to notify you of a thread I started in openstack-dev about the state
> of the OpenStack load balancer APIs and the difficulty in integrating them
> with Kubernetes. This in part directly relates to current public and private
> deployments, and any feedback you have would be appreciated. Especially
> feedback on which version of the load balancer APIs you deploy, and if you
> haven't moved on to Octavia, why.
>
> http://lists.openstack.org/pipermail/openstack-dev/2018-March/128399.html 
> 
>
> Thanks in advance,
> Chris
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] attaching network cards to VMs taking a very long time

[Saverio Proto]

Glad to hear it!
Always monitor rabbitmq queues to identify bottlenecks !! :)

Cheers

Saverio

Il gio 24 mag 2018, 11:07 Radu Popescu | eMAG, Technology <
radu.pope...@emag.ro> ha scritto:

> Hi,
>
> did the change yesterday. Had no issue this morning with neutron not being
> able to move fast enough. Still, we had some storage issues, but that's
> another thing.
> Anyway, I'll leave it like this for the next few days and report back in
> case I get the same slow neutron errors.
>
> Thanks a lot!
> Radu
>
> On Wed, 2018-05-23 at 10:08 +, Radu Popescu | eMAG, Technology wrote:
>
> Hi,
>
> actually, I didn't know about that option. I'll enable it right now.
> Testing is done every morning at about 4:00AM ..so I'll know tomorrow
> morning if it changed anything.
>
> Thanks,
> Radu
>
> On Tue, 2018-05-22 at 15:30 +0200, Saverio Proto wrote:
>
> Sorry email went out incomplete.
>
> Read this:
>
> https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/
>
>
> make sure that Openstack rootwrap configured to work in daemon mode
>
>
> Thank you
>
>
> Saverio
>
>
>
> 2018-05-22 15:29 GMT+02:00 Saverio Proto :
>
> Hello Radu,
>
>
> do you have the Openstack rootwrap configured to work in daemon mode ?
>
>
> please read this article:
>
>
> 2018-05-18 10:21 GMT+02:00 Radu Popescu | eMAG, Technology
>
> :
>
> Hi,
>
>
> so, nova says the VM is ACTIVE and actually boots with no network. We are
>
> setting some metadata that we use later on and have cloud-init for different
>
> tasks.
>
> So, VM is up, OS is running, but network is working after a random amount of
>
> time, that can get to around 45 minutes. Thing is, is not happening to all
>
> VMs in that test (around 300), but it's happening to a fair amount - around
>
> 25%.
>
>
> I can see the callback coming few seconds after neutron openvswitch agent
>
> says it's completed the setup. My question is, why is it taking so long for
>
> nova openvswitch agent to configure the port? I can see the port up in both
>
> host OS and openvswitch. I would assume it's doing the whole namespace and
>
> iptables setup. But still, 30 minutes? Seems a lot!
>
>
> Thanks,
>
> Radu
>
>
> On Thu, 2018-05-17 at 11:50 -0400, George Mihaiescu wrote:
>
>
> We have other scheduled tests that perform end-to-end (assign floating IP,
>
> ssh, ping outside) and never had an issue.
>
> I think we turned it off because the callback code was initially buggy and
>
> nova would wait forever while things were in fact ok, but I'll  change
>
> "vif_plugging_is_fatal = True" and "vif_plugging_timeout = 300" and run
>
> another large test, just to confirm.
>
>
> We usually run these large tests after a version upgrade to test the APIs
>
> under load.
>
>
>
>
> On Thu, May 17, 2018 at 11:42 AM, Matt Riedemann 
>
> wrote:
>
>
> On 5/17/2018 9:46 AM, George Mihaiescu wrote:
>
>
> and large rally tests of 500 instances complete with no issues.
>
>
>
> Sure, except you can't ssh into the guests.
>
>
> The whole reason the vif plugging is fatal and timeout and callback code was
>
> because the upstream CI was unstable without it. The server would report as
>
> ACTIVE but the ports weren't wired up so ssh would fail. Having an ACTIVE
>
> guest that you can't actually do anything with is kind of pointless.
>
>
> ___
>
>
> OpenStack-operators mailing list
>
>
> OpenStack-operators@lists.openstack.org
>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
>
> ___
>
> OpenStack-operators mailing list
>
> OpenStack-operators@lists.openstack.org
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
> ___
>
> OpenStack-operators mailing list
>
> OpenStack-operators@lists.openstack.org
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] attaching network cards to VMs taking a very long time

[Saverio Proto]

Sorry email went out incomplete.
Read this:
https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/

make sure that Openstack rootwrap configured to work in daemon mode

Thank you

Saverio


2018-05-22 15:29 GMT+02:00 Saverio Proto :
> Hello Radu,
>
> do you have the Openstack rootwrap configured to work in daemon mode ?
>
> please read this article:
>
> 2018-05-18 10:21 GMT+02:00 Radu Popescu | eMAG, Technology
> :
>> Hi,
>>
>> so, nova says the VM is ACTIVE and actually boots with no network. We are
>> setting some metadata that we use later on and have cloud-init for different
>> tasks.
>> So, VM is up, OS is running, but network is working after a random amount of
>> time, that can get to around 45 minutes. Thing is, is not happening to all
>> VMs in that test (around 300), but it's happening to a fair amount - around
>> 25%.
>>
>> I can see the callback coming few seconds after neutron openvswitch agent
>> says it's completed the setup. My question is, why is it taking so long for
>> nova openvswitch agent to configure the port? I can see the port up in both
>> host OS and openvswitch. I would assume it's doing the whole namespace and
>> iptables setup. But still, 30 minutes? Seems a lot!
>>
>> Thanks,
>> Radu
>>
>> On Thu, 2018-05-17 at 11:50 -0400, George Mihaiescu wrote:
>>
>> We have other scheduled tests that perform end-to-end (assign floating IP,
>> ssh, ping outside) and never had an issue.
>> I think we turned it off because the callback code was initially buggy and
>> nova would wait forever while things were in fact ok, but I'll  change
>> "vif_plugging_is_fatal = True" and "vif_plugging_timeout = 300" and run
>> another large test, just to confirm.
>>
>> We usually run these large tests after a version upgrade to test the APIs
>> under load.
>>
>>
>>
>> On Thu, May 17, 2018 at 11:42 AM, Matt Riedemann 
>> wrote:
>>
>> On 5/17/2018 9:46 AM, George Mihaiescu wrote:
>>
>> and large rally tests of 500 instances complete with no issues.
>>
>>
>> Sure, except you can't ssh into the guests.
>>
>> The whole reason the vif plugging is fatal and timeout and callback code was
>> because the upstream CI was unstable without it. The server would report as
>> ACTIVE but the ports weren't wired up so ssh would fail. Having an ACTIVE
>> guest that you can't actually do anything with is kind of pointless.
>>
>> ___
>>
>> OpenStack-operators mailing list
>>
>> OpenStack-operators@lists.openstack.org
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] attaching network cards to VMs taking a very long time

[Saverio Proto]

Hello Radu,

do you have the Openstack rootwrap configured to work in daemon mode ?

please read this article:

2018-05-18 10:21 GMT+02:00 Radu Popescu | eMAG, Technology
:
> Hi,
>
> so, nova says the VM is ACTIVE and actually boots with no network. We are
> setting some metadata that we use later on and have cloud-init for different
> tasks.
> So, VM is up, OS is running, but network is working after a random amount of
> time, that can get to around 45 minutes. Thing is, is not happening to all
> VMs in that test (around 300), but it's happening to a fair amount - around
> 25%.
>
> I can see the callback coming few seconds after neutron openvswitch agent
> says it's completed the setup. My question is, why is it taking so long for
> nova openvswitch agent to configure the port? I can see the port up in both
> host OS and openvswitch. I would assume it's doing the whole namespace and
> iptables setup. But still, 30 minutes? Seems a lot!
>
> Thanks,
> Radu
>
> On Thu, 2018-05-17 at 11:50 -0400, George Mihaiescu wrote:
>
> We have other scheduled tests that perform end-to-end (assign floating IP,
> ssh, ping outside) and never had an issue.
> I think we turned it off because the callback code was initially buggy and
> nova would wait forever while things were in fact ok, but I'll  change
> "vif_plugging_is_fatal = True" and "vif_plugging_timeout = 300" and run
> another large test, just to confirm.
>
> We usually run these large tests after a version upgrade to test the APIs
> under load.
>
>
>
> On Thu, May 17, 2018 at 11:42 AM, Matt Riedemann 
> wrote:
>
> On 5/17/2018 9:46 AM, George Mihaiescu wrote:
>
> and large rally tests of 500 instances complete with no issues.
>
>
> Sure, except you can't ssh into the guests.
>
> The whole reason the vif plugging is fatal and timeout and callback code was
> because the upstream CI was unstable without it. The server would report as
> ACTIVE but the ports weren't wired up so ssh would fail. Having an ACTIVE
> guest that you can't actually do anything with is kind of pointless.
>
> ___
>
> OpenStack-operators mailing list
>
> OpenStack-operators@lists.openstack.org
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Receipt to transfer the ownership of an instance

[Saverio Proto]

Hello Massimo,

what we suggest to our users, is to migrate a volume, and to create a
new VM from that volume.
https://help.switch.ch/engines/documentation/migrating-resources/

the bad thing is that the new VM has a new IP address, so eventually
DNS records have to be updated by the users.

Cheers,

Saverio


2018-04-23 10:17 GMT+02:00 Massimo Sgaravatto :
> As far as I understand there is not a clean way to transfer the ownership of
> an instance from a user to another one (the implementation of the blueprint
> https://blueprints.launchpad.net/nova/+spec/transfer-instance-ownership was
> abandoned).
>
>
> Is there at least a receipt (i.e. what needs to be changed in the database)
> that operators can follow to implement such use case ?
>
> Thanks, Massimo
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Nova resources are out of sync in ocata version

[Saverio Proto]

It works for me in Newton.
Try it at your own risk :)

Cheers,

Saverio

2018-04-09 13:23 GMT+02:00 Anwar Durrani :
> No this is different one. should i try this one ? if it works ?
>
> On Mon, Apr 9, 2018 at 4:11 PM, Saverio Proto  wrote:
>>
>> Hello Anwar,
>>
>> are you talking about this script ?
>>
>> https://github.com/openstack/osops-tools-contrib/blob/master/nova/nova-libvirt-compare.py
>>
>> it does not work for you ?
>>
>> Saverio
>>
>> 2018-04-09 11:53 GMT+02:00 Anwar Durrani :
>> > Hi All,
>> >
>> > Nova resources are out of sync in ocata version, what values are showing
>> > on
>> > dashboard are mismatch of actual running instances, i do remember i had
>> > script for auto sync resources but this script is getting fail in this
>> > case,
>> > Kindly help here.
>> >
>> > --
>> > Thanks & regards,
>> > Anwar M. Durrani
>> > +91-9923205011
>> >
>> >
>> >
>> > ___
>> > OpenStack-operators mailing list
>> > OpenStack-operators@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>
>
>
>
> --
> Thanks & regards,
> Anwar M. Durrani
> +91-9923205011
>
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Nova resources are out of sync in ocata version

[Saverio Proto]

Hello Anwar,

are you talking about this script ?
https://github.com/openstack/osops-tools-contrib/blob/master/nova/nova-libvirt-compare.py

it does not work for you ?

Saverio

2018-04-09 11:53 GMT+02:00 Anwar Durrani :
> Hi All,
>
> Nova resources are out of sync in ocata version, what values are showing on
> dashboard are mismatch of actual running instances, i do remember i had
> script for auto sync resources but this script is getting fail in this case,
> Kindly help here.
>
> --
> Thanks & regards,
> Anwar M. Durrani
> +91-9923205011
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Meetup, Co-Location options, and User Feedback

[Saverio Proto]

I am also for +1
Thanks

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ocata security groups don't work with LBaaS v2 ports

[Saverio Proto]

Hello Ignazio,

it would interesting to know how this works. For instances ports,
those ports are created by openvswitch on the compute nodes, where the
neutron-agent will take care of the security groups enforcement (via
iptables or openvswitch rules).

the LBaaS is a namespace that lives where the neutron-lbaasv2-agent is running.

The question is if the neutron-lbaasv2-agent is capable for setting
iptables rules. I would start to read the code there.

Cheers,

Saverio


2018-03-23 13:51 GMT+01:00 Ignazio Cassano :
> Hi all,
> following the ocata documentation, I am trying to apply security group to a
> lbaas v2 port but
> it seems not working because any filter is applyed.
> The Port Security Enabled is True on lbaas port, so I expect applying
> security group should work.
> Is this a bug ?
> Regards
> Ignazio
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova] about rebuild instance booted from volume

[Saverio Proto]

My idea is that if delete_on_termination flag is set to False the
Volume should never be deleted by Nova.

my 2 cents

Saverio

2018-03-14 15:10 GMT+01:00 Tim Bell :
> Matt,
>
> To add another scenario and make things even more difficult (sorry (), if the 
> original volume has snapshots, I don't think you can delete it.
>
> Tim
>
>
> -Original Message-
> From: Matt Riedemann 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: Wednesday, 14 March 2018 at 14:55
> To: "openstack-...@lists.openstack.org" , 
> openstack-operators 
> Subject: Re: [openstack-dev] [nova] about rebuild instance booted from volume
>
> On 3/14/2018 3:42 AM, 李杰 wrote:
> >
> >  This is the spec about  rebuild a instance booted from
> > volume.In the spec,there is a
> >question about if we should delete the old root_volume.Anyone who
> > is interested in
> >booted from volume can help to review this. Any suggestion is
> > welcome.Thank you!
> >The link is here.
> >Re:the rebuild spec:https://review.openstack.org/#/c/532407/
>
> Copying the operators list and giving some more context.
>
> This spec is proposing to add support for rebuild with a new image for
> volume-backed servers, which today is just a 400 failure in the API
> since the compute doesn't support that scenario.
>
> With the proposed solution, the backing root volume would be deleted and
> a new volume would be created from the new image, similar to how boot
> from volume works.
>
> The question raised in the spec is whether or not nova should delete the
> root volume even if its delete_on_termination flag is set to False. The
> semantics get a bit weird here since that flag was not meant for this
> scenario, it's meant to be used when deleting the server to which the
> volume is attached. Rebuilding a server is not deleting it, but we would
> need to replace the root volume, so what do we do with the volume we're
> replacing?
>
> Do we say that delete_on_termination only applies to deleting a server
> and not rebuild and therefore nova can delete the root volume during a
> rebuild?
>
> If we don't delete the volume during rebuild, we could end up leaving a
> lot of volumes lying around that the user then has to clean up,
> otherwise they'll eventually go over quota.
>
> We need user (and operator) feedback on this issue and what they would
> expect to happen.
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] AggregateMultiTenancyIsolation with multiple (many) projects

[Saverio Proto]

> If you’re willing to, I could share with you a way to get a FrankeinCloud
> using a Docker method with kolla to get a pike/queens/whatever cloud at the
> same time that your Ocata one.

I am interested in knowing more about this. If you have any link /
blog post please share them :)

thank you

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Snapshot from Cinder to Glance doing a ceph rbd clone

[Saverio Proto]

Hello ops,

we have this working for Nova ephemeral images already, but Cinder did
not implement this spec:
https://specs.openstack.org/openstack/cinder-specs/specs/liberty/optimze-rbd-copy-volume-to-image.html

Is anyone carrying an unmerged patch that implements this spec ?
I could not believe my eyes this morning when I figured out this was
working only for nova.

thanks

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] oslo.log JSON logs are missing the request ID

[Saverio Proto]

Hello,

probably someone here is using stuff like Kibana to look at Openstack
logs. We are trying here to use the json logging, and we are surprised
that the request-id is not printed in the json output.

I wrote this email to the devs:
http://lists.openstack.org/pipermail/openstack-dev/2018-January/126144.html

Does anyone have this working, or the field is really missing in the
code of the oslo.log json formatter ?

thanks a lot

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Mixed service version CI testing

[Saverio Proto]

it makes sense and it is very valuable !
thanks

Saverio

Il 19 dic 2017 4:59 PM, "Matt Riedemann"  ha scritto:

> During discussion in the TC channel today [1], we got talking about how
> there is a perception that you must upgrade all of the services together
> for anything to work, at least the 'core' services like
> keystone/nova/cinder/neutron/glance - although maybe that's really just
> nova/cinder/neutron?
>
> Anyway, I posit that the services are not as tightly coupled as some
> people assume they are, at least not since kilo era when microversions
> started happening in nova.
>
> However, with the way we do CI testing, and release everything together,
> the perception is there that all things must go together to work.
>
> In our current upgrade job, we upgrade everything to N except the
> nova-compute service, that remains at N-1 to test rolling upgrades of your
> computes and to make sure guests are unaffected by the upgrade of the
> control plane.
>
> I asked if it would be valuable to our users (mostly ops for this right?)
> if we had an upgrade job where everything *except* nova were upgraded. If
> that's how the majority of people are doing upgrades anyway it seems we
> should make sure that works.
>
> I figure leaving nova at N-1 makes more sense because nova depends on the
> other services (keystone/glance/cinder/neutron) and is likely the harder
> / slower upgrade if you're going to do rolling upgrades of your compute
> nodes.
>
> This type of job would not run on nova changes on the master branch, since
> those changes would not be exercised in this type of environment. So we'd
> run this on master branch changes to keystone/cinder/glance/neutron
> /trove/designate/etc.
>
> Does that make sense? Would this be valuable at all? Or should the
> opposite be tested where we upgrade nova to N and leave all of the
> dependent services at N-1?
>
> Really looking for operator community feedback here.
>
> [1] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23op
> enstack-tc.2017-12-19.log.html#t2017-12-19T15:14:15
>
> --
>
> Thanks,
>
> Matt
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Security Groups and Metadata Service

[Saverio Proto]

Hello,

we have this recurring problem with our users.

An advanced user deletes all the default security groups to create his
own. This user will define only ingress rules.

Because there is no egress rule, the cloud-init will fail to open a
connection to the metadata service.

The user will open a ticket that he cant login into the VM, because of
corse the SSH key was not injected.

Does anyone has a good solution to prevent the user from setting the
system in a such a way that does not work ??

thank you

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] LTS pragmatic example

[Saverio Proto]

Hello,

here an example of a trivial patch that is important for people that
do operations, and they have to troubleshoot stuff.

with the old Stable Release thinking this patch would not be accepted
on old stable branches.

Let's see if this gets accepted back to stable/newton

https://review.openstack.org/#/q/If525313c63c4553abe8bea6f2bfaf75431ed18ea

Please note that a developers/operators that make the effort of fixing
this in master, should do also all the cherry-pickes back. We dont
have any automatic procudure for this.

thank you

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Upstream LTS Releases

[Saverio Proto]

The 1 year release cycle makes a lot of sense to me too. +1

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [cinder] volume in-use attached to None?

[Saverio Proto]

Hello Christopher,

check out this:
https://ask.openstack.org/en/question/66918/how-to-delete-volume-with-available-status-and-attached-to/

Saverio

2017-10-16 20:45 GMT+02:00 Christopher Hull :
> Running Liberty.
> I'd like to be able to create new volumes from old ones.  Launching
> instances from volumes results in the volume being "root attached", and
> therefore, it seems, forever wed to the instance.  It can not be copied.
> So I tried deleting the instance.  Still no good.  The volume is now in-use
> by None.   So now the volume is completely useless.
> How do I force Cinder to detach from either a root mounted , or of all
> things, None??
>
> -Chris
>
>
>
>
> - Christopher T. Hull
>
> http://faq.chrishull.com
> Sunnyvale CA. 94085
> (415) 385 4865
> chrishul...@gmail.com
> http://chrishull.com
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Guest crash and KVM unhandled rdmsr

[Saverio Proto]

Hello Blair,

I found this link in my browser history:
https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/1583819

Is it the same messages that you are seeing in Xenial ?

Saverio



2017-10-12 23:26 GMT+02:00 Blair Bethwaite :
> Hi all,
>
> Has anyone seen guest crashes/freezes associated with KVM unhandled rdmsr
> messages in dmesg on the hypervisor?
>
> We have seen these messages before but never with a strong correlation to
> guest problems. However over the past couple of weeks this is happening
> almost daily with consistent correlation for a set of hosts dedicated to a
> particular HPC workload. So far as I know the workload has not changed, but
> we have just recently moved the hypervisors to Ubuntu Xenial (though they
> were already on the Xenial kernel previously) and done minor guest (CentOS7)
> updates. CPU mode is host-passthrough. Currently trying to figure out if the
> CPU flags in the guest have changed since the host upgrade...
>
> Cheers,
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cinder quota per volume types in the dashboard

[Saverio Proto]

So we had a bad feedback on the bug:
https://bugs.launchpad.net/horizon/+bug/1717342

Do we have anything pushed to Gerrit that we can at least use carrying
a local patch ?

thanks

Saverio


2017-09-26 20:21 GMT+02:00 Massimo Sgaravatto :
> ok, I will
>
> 2017-09-26 14:43 GMT+02:00 Arne Wiebalck :
>>
>> Massimo,
>>
>> Following Rob’s comment on
>> https://bugs.launchpad.net/horizon/+bug/1717342, would you
>> be willing to write up a blueprint? Mateusz would then prepare our code
>> and submit it to
>> gerrit as a partial implementation (as we only have the user facing part,
>> not the admin panel).
>>
>> Cheers,
>>  Arne
>>
>>
>> On 25 Sep 2017, at 10:46, Arne Wiebalck  wrote:
>>
>> Ah, nice, wasn’t aware. Mateusz is one of the Horizon experts here at CERN
>> I was referring to :)
>>
>> On 25 Sep 2017, at 10:41, Massimo Sgaravatto
>>  wrote:
>>
>> Just found that there is already this one:
>>
>> https://bugs.launchpad.net/horizon/+bug/1717342
>>
>> 2017-09-25 10:28 GMT+02:00 Saverio Proto :
>>>
>>> Yes I am interested. Are you going to push them to gerrit ?
>>> Should we open a bug to track this change into Horizon ?
>>>
>>> massimo do you want to open the bug on Launchpad ? So if Arne pushed
>>> the patches on gerrit we can link them to the bug. I pointed
>>> robcresswell to this thread, he is reading us.
>>>
>>> thanks !
>>>
>>> Saverio
>>>
>>> 2017-09-25 10:13 GMT+02:00 Arne Wiebalck :
>>> > Massimo, Saverio,
>>> >
>>> > We faced the same issue and have created patches for Horizon to display
>>> > - the per volume quota in the volume request panel, and also
>>> > - additional information about the volume type (like IOPS and
>>> > throughput limits, intended usage etc.)
>>> >
>>> > The patches will need some polishing before being sent upstream (I’ll
>>> > need
>>> > need to cross-check with our Horizon experts), but we use them in prod
>>> > since
>>> > quite a while and are happy to already share patch files if you’re
>>> > interested.
>>> >
>>> > Cheers,
>>> >  Arne
>>> >
>>> >
>>> >
>>> >> On 25 Sep 2017, at 09:58, Saverio Proto  wrote:
>>> >>
>>> >> I am pinging on IRC robcresswell from the Horizon project. He is still
>>> >> PTL I think.
>>> >> If you are on IRC please join #openstack-horizon.
>>> >>
>>> >> We should ask the Horizon PTL how to get this feature request into
>>> >> implementation.
>>> >>
>>> >> With the command line interface, can you already see the two different
>>> >> quotas for the two different volume types ? Can you paste an example
>>> >> output from the CLI ?
>>> >>
>>> >> thank you
>>> >>
>>> >> Saverio
>>> >>
>>> >>
>>> >> 2017-09-25 9:55 GMT+02:00 Massimo Sgaravatto
>>> >> :
>>> >>> We are currently running Mitaka (preparing to update to Ocata). I see
>>> >>> the
>>> >>> same behavior on an Ocata based testbed
>>> >>>
>>> >>> Thanks, Massimo
>>> >>>
>>> >>> 2017-09-25 9:50 GMT+02:00 Saverio Proto :
>>> >>>>
>>> >>>> Hello Massimo,
>>> >>>>
>>> >>>> what is your version of Openstack ??
>>> >>>>
>>> >>>> thank you
>>> >>>>
>>> >>>> Saverio
>>> >>>>
>>> >>>> 2017-09-25 9:13 GMT+02:00 Massimo Sgaravatto
>>> >>>> :
>>> >>>>> Hi
>>> >>>>>
>>> >>>>>
>>> >>>>> In our OpenStack cloud we have two backends for Cinder (exposed
>>> >>>>> using
>>> >>>>> two
>>> >>>>> volume types), and we set different quotas for these two volume
>>> >>>>> types.
>>> >>>>>
>>> >>>>> The problem happens when a user, using the dashboard, tries to
>>> >>>>> create a
>>> >>>>> volume using a volume type for which the project quota is over:
>>> >&

Re: [Openstack-operators] [nova] Looking for feedback on a spec to limit max_count in multi-create requests

[Saverio Proto]

Hello Matt,

starting 1000 instances in production works for me already. We are on
Openstack Newton.
I described my configuration here:
https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/

If things blow up for you with hundreds, probably there is a
regression somewhere.

Thanks

Saverio


2017-10-06 23:43 GMT+02:00 Matt Riedemann :
> I've been chasing something weird I was seeing in devstack when creating
> hundreds of instances in a single request where at some limit, things blow
> up in an unexpected way during scheduling and all instances were put into
> ERROR state. Given the environment I was running in, this shouldn't have
> been happening, and today we figured out what was actually happening. To
> summarize, we retry scheduling requests on RPC timeout so you can have
> scheduler_max_attempts greenthreads running concurrently trying to schedule
> 1000 instances and melt your scheduler.
>
> I've started a spec which goes into the details of the actual issue:
>
> https://review.openstack.org/#/c/510235/
>
> It also proposes a solution, but I don't feel it's the greatest solution, so
> there are also some alternatives in there.
>
> I'm really interested in operator feedback on this because I assume that
> people are dealing with stuff like this in production already, and have had
> to come up with ways to solve it.
>
> --
>
> Thanks,
>
> Matt
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova] reset key pair during rebuilding

[Saverio Proto]

Hello,

I agree this feature of injecting a new keypair is something of great
use. We are always dealing with users that cant access their VMs
anymore.

But AFAIU here we are talking about injecting a new key at REBUILD. So
it does not fit the scenario of a staff member that leaves !

We hardly never use the rebuild feature in our workflow. Our users
just use create and delete.

I think it would be more useful a feature where you can reinject a new
keypair in the VM at any time. Ahhh it makes the users happy but of
course it is a security nightmare :D

Cheers

Saverio



2017-09-27 11:15 GMT+02:00 Marcus Furlong :
> On 27 September 2017 at 09:23, Michael Still  wrote:
>>
>> Operationally, why would I want to inject a new keypair? The scenario I can
>> think of is that there's data in that instance that I want, and I've lost
>> the keypair somehow. Unless that data is on an ephemeral, its gone if we do
>> a rebuild.
>
> This is quite a common scenario - staff member who started the
> instance leaves, and you want to access data on the instance, or
> maintain/debug the service running on the instance.
>
> Hitherto, I have used direct db calls to update the key, so it would
> be nice if there was an API call to do so.
>
> Cheers,
> Marcus.
> --
> Marcus Furlong
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cinder quota per volume types in the dashboard

[Saverio Proto]

Yes I am interested. Are you going to push them to gerrit ?
Should we open a bug to track this change into Horizon ?

massimo do you want to open the bug on Launchpad ? So if Arne pushed
the patches on gerrit we can link them to the bug. I pointed
robcresswell to this thread, he is reading us.

thanks !

Saverio

2017-09-25 10:13 GMT+02:00 Arne Wiebalck :
> Massimo, Saverio,
>
> We faced the same issue and have created patches for Horizon to display
> - the per volume quota in the volume request panel, and also
> - additional information about the volume type (like IOPS and throughput 
> limits, intended usage etc.)
>
> The patches will need some polishing before being sent upstream (I’ll need
> need to cross-check with our Horizon experts), but we use them in prod since
> quite a while and are happy to already share patch files if you’re interested.
>
> Cheers,
>  Arne
>
>
>
>> On 25 Sep 2017, at 09:58, Saverio Proto  wrote:
>>
>> I am pinging on IRC robcresswell from the Horizon project. He is still
>> PTL I think.
>> If you are on IRC please join #openstack-horizon.
>>
>> We should ask the Horizon PTL how to get this feature request into
>> implementation.
>>
>> With the command line interface, can you already see the two different
>> quotas for the two different volume types ? Can you paste an example
>> output from the CLI ?
>>
>> thank you
>>
>> Saverio
>>
>>
>> 2017-09-25 9:55 GMT+02:00 Massimo Sgaravatto :
>>> We are currently running Mitaka (preparing to update to Ocata). I see the
>>> same behavior on an Ocata based testbed
>>>
>>> Thanks, Massimo
>>>
>>> 2017-09-25 9:50 GMT+02:00 Saverio Proto :
>>>>
>>>> Hello Massimo,
>>>>
>>>> what is your version of Openstack ??
>>>>
>>>> thank you
>>>>
>>>> Saverio
>>>>
>>>> 2017-09-25 9:13 GMT+02:00 Massimo Sgaravatto
>>>> :
>>>>> Hi
>>>>>
>>>>>
>>>>> In our OpenStack cloud we have two backends for Cinder (exposed using
>>>>> two
>>>>> volume types), and we set different quotas for these two volume types.
>>>>>
>>>>> The problem happens when a user, using the dashboard, tries to create a
>>>>> volume using a volume type for which the project quota is over:
>>>>>
>>>>> - the reported error message simply reports "unable to create volume",
>>>>> without mentioning that the problem is with quota
>>>>>
>>>>> - (by default) the dashboard only shows the overall Cinder quota (and
>>>>> not
>>>>> the quota per volume)
>>>>>
>>>>>
>>>>> Do you know if it possible in some to expose on the dashboard the cinder
>>>>> quota per volume type ?
>>>>>
>>>>>
>>>>> Thanks, Massimo
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ___
>>>>> OpenStack-operators mailing list
>>>>> OpenStack-operators@lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>>
>>>
>>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
> --
> Arne Wiebalck
> CERN IT
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cinder quota per volume types in the dashboard

[Saverio Proto]

I am pinging on IRC robcresswell from the Horizon project. He is still
PTL I think.
 If you are on IRC please join #openstack-horizon.

We should ask the Horizon PTL how to get this feature request into
implementation.

With the command line interface, can you already see the two different
quotas for the two different volume types ? Can you paste an example
output from the CLI ?

thank you

Saverio


2017-09-25 9:55 GMT+02:00 Massimo Sgaravatto :
> We are currently running Mitaka (preparing to update to Ocata). I see the
> same behavior on an Ocata based testbed
>
> Thanks, Massimo
>
> 2017-09-25 9:50 GMT+02:00 Saverio Proto :
>>
>> Hello Massimo,
>>
>> what is your version of Openstack ??
>>
>> thank you
>>
>> Saverio
>>
>> 2017-09-25 9:13 GMT+02:00 Massimo Sgaravatto
>> :
>> > Hi
>> >
>> >
>> > In our OpenStack cloud we have two backends for Cinder (exposed using
>> > two
>> > volume types), and we set different quotas for these two volume types.
>> >
>> > The problem happens when a user, using the dashboard, tries to create a
>> > volume using a volume type for which the project quota is over:
>> >
>> > - the reported error message simply reports "unable to create volume",
>> > without mentioning that the problem is with quota
>> >
>> > - (by default) the dashboard only shows the overall Cinder quota (and
>> > not
>> > the quota per volume)
>> >
>> >
>> > Do you know if it possible in some to expose on the dashboard the cinder
>> > quota per volume type ?
>> >
>> >
>> > Thanks, Massimo
>> >
>> >
>> >
>> >
>> >
>> >
>> > ___
>> > OpenStack-operators mailing list
>> > OpenStack-operators@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cinder quota per volume types in the dashboard

[Saverio Proto]

Hello Massimo,

what is your version of Openstack ??

thank you

Saverio

2017-09-25 9:13 GMT+02:00 Massimo Sgaravatto :
> Hi
>
>
> In our OpenStack cloud we have two backends for Cinder (exposed using two
> volume types), and we set different quotas for these two volume types.
>
> The problem happens when a user, using the dashboard, tries to create a
> volume using a volume type for which the project quota is over:
>
> - the reported error message simply reports "unable to create volume",
> without mentioning that the problem is with quota
>
> - (by default) the dashboard only shows the overall Cinder quota (and not
> the quota per volume)
>
>
> Do you know if it possible in some to expose on the dashboard the cinder
> quota per volume type ?
>
>
> Thanks, Massimo
>
>
>
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Is there any reason to exclude originally failed build hosts during live migration?

[Saverio Proto]

> The actual fix for this is trivial:
>
> https://review.openstack.org/#/c/505771/

Why the change is called:
Ignore original retried hosts when live migrating
?

Isn't it implementing the opposite ? Dont Ignore ?

thanks

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Instances not getting DHCP Lease

[Saverio Proto]

> checking http://169.254.169.254/2009-04-04/instance-id
> failed 1/20: up 188.93. request failed
> failed 2/20: up 191.21. request failed
> failed 3/20: up 193.36. request failed
> failed 4/20: up 195.54. request failed
> failed 5/20: up 197.68. request failed
> failed 6/20: up 199.83. request failed
> failed 7/20: up 201.97. request failed
> failed 8/20: up 204.13. request failed
> failed 9/20: up 206.31. request failed
> failed 10/20: up 208.48. request failed
> failed 11/20: up 210.63. request failed
> failed 12/20: up 212.89. request failed
> failed 13/20: up 215.06. request failed
> failed 14/20: up 217.21. request failed

Hello,

this is the metadata service failing, it is not DHCP. Does the
instance get an ip address from DHCP or not ?

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Instances not getting DHCP Lease

[Saverio Proto]

Hello,

using:

openstack console log show 

you can check if it is really the dhclient failing to have an address.

it is usually also good to have a look at the nova-compute.log file in
the compute node where the instance is scheduled.

Saverio



2017-08-31 19:55 GMT+02:00 Divneet Singh :
> I realize this is a common question , and I've looked through more than a
> couple of links , s, but I'm afraid I'm new enough to OpenStack that I
> wasn't able to make sense of many of the replies and was unable to solve my
> problem after searching.
> I have a multi-node deployment with one controller/network node and one
> compute node. When I create a cirros image on the Compute node, I see in the
> Horizon interface that a DHCP address is assigned, but when I run ifconfig
> in the VNC console on the Horizon dashboard, I see the instance has no IP
> address
>
> Can anyone suggest possible fixes? As I said, I'm pretty new to OpenStack.
> I'm sure that I haven't provided all of the information needed to solve this
> problem; if someone could tell me what further information would be useful.
>
> Thanks !
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Experience with Cinder volumes as root disks?

[Saverio Proto]

Hello Conrad,

I jump late on the conversation because I was away from the mailing
lists last week.

We run Openstack with both nova ephemeral root disks and cinder volume
boot disks. Both are with ceph rbd backend. It is the user that flags
"boot from volume" in Horizon when starting an instance.

Everything works in both cases, and there are pros and cons as you see
from the many answers you had.

But, if I have to give you a suggestion, I would choose only 1 way to
go and stick with it.

Having all this flexibility is great for us, operators that understand
Openstack internals. But it is a nightmare for the openstack users.

First of all looking at Horizon it is very difficult to understand the
kind of root volume being used.
It is also difficult to understand that you have a snapshot of the
nova instance, and a snapshot of the cinder volume.
We have different snapshot procedures depending on the type for root
disk, and users always get confused.
When the root disk is cinder, if you snapshot from the instance page,
you will get a 0 byte glance image connected to a cinder snapshot.

A user that has a instance with a disk, should not have to understand
if the disk is managed by nova or cinder to finally backup his data
with a snapshot.
Looking at Cloud usability, I would say that mixing the two solutions
is not the best. Probably this explains the Amazon and Azure choices
you described earlier.

Cheers,

Saverio



2017-08-01 16:50 GMT+02:00 Kimball, Conrad :
> In our process of standing up an OpenStack internal cloud we are facing the
> question of ephemeral storage vs. Cinder volumes for instance root disks.
>
>
>
> As I look at public clouds such as AWS and Azure, the norm is to use
> persistent volumes for the root disk.  AWS started out with images booting
> onto ephemeral disk, but soon after they released Elastic Block Storage and
> ever since the clear trend has been to EBS-backed instances, and now when I
> look at their quick-start list of 33 AMIs, all of them are EBS-backed.  And
> I’m not even sure one can have anything except persistent root disks in
> Azure VMs.
>
>
>
> Based on this and a number of other factors I think we want our user normal
> / default behavior to boot onto Cinder-backed volumes instead of onto
> ephemeral storage.  But then I look at OpenStack and its design point
> appears to be booting images onto ephemeral storage, and while it is
> possible to boot an image onto a new volume this is clumsy (haven’t found a
> way to make this the default behavior) and we are experiencing performance
> problems (that admittedly we have not yet run to ground).
>
>
>
> So …
>
> · Are other operators routinely booting onto Cinder volumes instead
> of ephemeral storage?
>
> · What has been your experience with this; any advice?
>
>
>
> Conrad Kimball
>
> Associate Technical Fellow
>
> Chief Architect, Enterprise Cloud Services
>
> Application Infrastructure Services / Global IT Infrastructure / Information
> Technology & Data Analytics
>
> conrad.kimb...@boeing.com
>
> P.O. Box 3707, Mail Code 7M-TE
>
> Seattle, WA  98124-2207
>
> Bellevue 33-11 bldg, office 3A6-3.9
>
> Mobile:  425-591-7802
>
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] UDP Buffer Filling

[Saverio Proto]

It is merged in Mitaka but your glance images must be decorated with:

hw_vif_multiqueue_enabled='true'

when you do "openstack image show uuid"

in the property you should see this, and then you will have multiqueue

Saverio



2017-07-28 14:50 GMT+02:00 John Petrini :

> Hi Saverio,
>
> Thanks for the info. The parameter is missing completely:
>
> 
>   
>   
>   
>   
>function='0x0'/>
> 
>
> I've came across the blueprint for adding the image property
> hw_vif_multiqueue_enabled. Do you know if this feature is available in
> Mitaka?
>
> John Petrini
>
> Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
> Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
> <http://www.linkedin.com/company/99631>   [image: Google Plus]
> <https://plus.google.com/104062177220750809525/posts>   [image: Blog]
> <http://success.coredial.com/blog>
> 751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
> *P:* 215.297.4400 x232 <(215)%20297-4400>   //   *F: *215.297.4401
> <(215)%20297-4401>   //   *E: *jpetr...@coredial.com
>
> On Fri, Jul 28, 2017 at 3:59 AM, Saverio Proto  wrote:
>
>> Hello John,
>>
>> a common problem is packets being dropped when they pass from the
>> hypervisor to the instance. There is bottleneck there.
>>
>> check the 'virsh dumpxml' of one of the instances that is dropping
>> packets. Check for the interface section, should look like:
>>
>> 
>>   
>>   
>>   
>>   
>>   
>>   
>>   > function='0x0'/>
>> 
>>
>> how many queues you have ??? Usually if you have only 1 or if the
>> parameter is missing completely is not good.
>>
>> in Mitaka nova should use 1 queue for every instance CPU core you
>> have. It is worth to check if this is set correctly in your setup.
>>
>> Cheers,
>>
>> Saverio
>>
>>
>>
>> 2017-07-27 17:49 GMT+02:00 John Petrini :
>> > Hi List,
>> >
>> > We are running Mitaka with VLAN provider networking. We've recently
>> > encountered a problem where the UDP receive queue on instances is
>> filling up
>> > and we begin dropping packets. Moving instances out of OpenStack onto
>> bare
>> > metal resolves the issue completely.
>> >
>> > These instances are running asterisk which should be pulling these
>> packets
>> > off the queue but it appears to be falling behind no matter the
>> resources we
>> > give it.
>> >
>> > We can't seem to pin down a reason why we would see this behavior in
>> KVM but
>> > not on metal. I'm hoping someone on the list might have some insight or
>> > ideas.
>> >
>> > Thank You,
>> >
>> > John
>> >
>> > ___
>> > OpenStack-operators mailing list
>> > OpenStack-operators@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>>
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] UDP Buffer Filling

[Saverio Proto]

Hello John,

a common problem is packets being dropped when they pass from the
hypervisor to the instance. There is bottleneck there.

check the 'virsh dumpxml' of one of the instances that is dropping
packets. Check for the interface section, should look like:


  
  
  
  
  
  
  


how many queues you have ??? Usually if you have only 1 or if the
parameter is missing completely is not good.

in Mitaka nova should use 1 queue for every instance CPU core you
have. It is worth to check if this is set correctly in your setup.

Cheers,

Saverio



2017-07-27 17:49 GMT+02:00 John Petrini :
> Hi List,
>
> We are running Mitaka with VLAN provider networking. We've recently
> encountered a problem where the UDP receive queue on instances is filling up
> and we begin dropping packets. Moving instances out of OpenStack onto bare
> metal resolves the issue completely.
>
> These instances are running asterisk which should be pulling these packets
> off the queue but it appears to be falling behind no matter the resources we
> give it.
>
> We can't seem to pin down a reason why we would see this behavior in KVM but
> not on metal. I'm hoping someone on the list might have some insight or
> ideas.
>
> Thank You,
>
> John
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state

[Saverio Proto]

Hello,

as far as I know from the Openstack survey there is a very small
number of production environments with more than 100 servers running
already Ocata.

you probably will have to find your self that information and report
back to the list.

thank you

Saverio


2017-07-10 10:27 GMT+02:00 Kekane, Abhishek :
> Hi Saverio,
>
> Thank you for reply.
>
> Currently we are using Ocata release for Openstack.
>
> Please let me know if you get any update.
>
> Thank you,
>
> Abhishek
>
> -----Original Message-
> From: Saverio Proto [mailto:ziopr...@gmail.com]
> Sent: Monday, July 10, 2017 12:52 PM
> To: Kekane, Abhishek
> Cc: OpenStack Development Mailing List (not for usage questions); 
> openstack-operators@lists.openstack.org
> Subject: Re: [Openstack-operators] [openstack-dev] [masakari][nova] Allow 
> evacuation of instances in resized state
>
> Hello Abhishek,
>
> I am sorry I dont have an answer for your question. I would have to try my 
> self everything to give answer because I never experienced this use case you 
> describe.
>
> I would suggest also to specify what version of Openstack you are working 
> with. Because the behaviour can change a lot in different versions.
>
> Cheers,
>
> Saverio
>
>
> 2017-07-10 9:00 GMT+02:00 Kekane, Abhishek :
>> Hi Operators,
>>
>> Could you please let me know your opinion on below scenario? It will help me 
>> to proceed my work.
>>
>> Thank you,
>>
>> Abhishek
>>
>> -Original Message-
>> From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com]
>> Sent: Tuesday, July 04, 2017 2:52 PM
>> To: OpenStack Development Mailing List (not for usage questions);
>> openstack-operators@lists.openstack.org
>> Subject: Re: [Openstack-operators] [openstack-dev] [masakari][nova]
>> Allow evacuation of instances in resized state
>>
>> Hi operators,
>>
>> I want to know how evacuation of resized instances is handled in real 
>> environment.
>> For example if the vm is in resized state and if the compute host on which 
>> the vm is resized goes down, then how will operator evacuate the vm.
>>
>> One possible way is to reset that vm state to error and then evacuate it to 
>> new compute host.
>> Please refer below scenario for reference:
>>
>> Scenario:
>> =
>>
>> Pre-conditions:
>> 
>> 1. Config option allow_resize_to_same_host is False.
>> 2. Instance path is not mounted on shared storage.
>> 3. Three compute nodes: "compute node A", "compute node B" and "compute node 
>> C"
>>
>> Steps:
>> 
>> 1. Boot an instance on "compute node A".
>> 2. User tries to resize the newly created instance and nova-scheduler 
>> selects "compute node B" as a destination node for resize.
>>In this case nova creates a instance directory on destination "compute 
>> node B" and mark the instance directory which is present on the source 
>> "compute node A" as "*_resize".
>>
>> Note that the resize operation is yet not confirmed and "compute node B" 
>> goes down.
>>
>> 3. Reset instance state to ERROR as nova allows evacuation only if instance 
>> state is 'ACTIVE', 'STOPPED' or 'ERROR'.
>> 4. Evacuate the instance to "compute node C" using target_host option.
>>As a result, instance files which were on "compute node B" will be 
>> cleaned up after compute service on it is up again, but instance files which 
>> were on "compute node A" marked as "*_resize" will never be cleaned up. As 
>> of now there is no periodic task in nova to perform cleanup of these kinds 
>> of scenarios.
>>
>>
>> Questions:
>> 1. is this the only possible way of evacuating the resized instances in real 
>> world scenario?
>> 2. If yes is there any way to cleanup unused (*_resize) instance files from 
>> the source compute node other than cleaning up it manually?
>> 3. Should we add support of evacuating of resized instances in nova?
>>
>> Please let me know your opinions about the same.
>>
>>
>> Thank you,
>>
>> Abhishek Kekane
>>
>>
>>
>> -Original Message-
>> From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com]
>> Sent: Thursday, June 29, 2017 5:57 PM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of
>> instances in resiz

Re: [Openstack-operators] [openstack-dev] [masakari][nova] Allow evacuation of instances in resized state

[Saverio Proto]

Hello Abhishek,

I am sorry I dont have an answer for your question. I would have to
try my self everything to give answer because I never experienced this
use case you describe.

I would suggest also to specify what version of Openstack you are
working with. Because the behaviour can change a lot in different
versions.

Cheers,

Saverio


2017-07-10 9:00 GMT+02:00 Kekane, Abhishek :
> Hi Operators,
>
> Could you please let me know your opinion on below scenario? It will help me 
> to proceed my work.
>
> Thank you,
>
> Abhishek
>
> -Original Message-
> From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com]
> Sent: Tuesday, July 04, 2017 2:52 PM
> To: OpenStack Development Mailing List (not for usage questions); 
> openstack-operators@lists.openstack.org
> Subject: Re: [Openstack-operators] [openstack-dev] [masakari][nova] Allow 
> evacuation of instances in resized state
>
> Hi operators,
>
> I want to know how evacuation of resized instances is handled in real 
> environment.
> For example if the vm is in resized state and if the compute host on which 
> the vm is resized goes down, then how will operator evacuate the vm.
>
> One possible way is to reset that vm state to error and then evacuate it to 
> new compute host.
> Please refer below scenario for reference:
>
> Scenario:
> =
>
> Pre-conditions:
> 
> 1. Config option allow_resize_to_same_host is False.
> 2. Instance path is not mounted on shared storage.
> 3. Three compute nodes: "compute node A", "compute node B" and "compute node 
> C"
>
> Steps:
> 
> 1. Boot an instance on "compute node A".
> 2. User tries to resize the newly created instance and nova-scheduler selects 
> "compute node B" as a destination node for resize.
>In this case nova creates a instance directory on destination "compute 
> node B" and mark the instance directory which is present on the source 
> "compute node A" as "*_resize".
>
> Note that the resize operation is yet not confirmed and "compute node B" goes 
> down.
>
> 3. Reset instance state to ERROR as nova allows evacuation only if instance 
> state is 'ACTIVE', 'STOPPED' or 'ERROR'.
> 4. Evacuate the instance to "compute node C" using target_host option.
>As a result, instance files which were on "compute node B" will be cleaned 
> up after compute service on it is up again, but instance files which were on 
> "compute node A" marked as "*_resize" will never be cleaned up. As of now 
> there is no periodic task in nova to perform cleanup of these kinds of 
> scenarios.
>
>
> Questions:
> 1. is this the only possible way of evacuating the resized instances in real 
> world scenario?
> 2. If yes is there any way to cleanup unused (*_resize) instance files from 
> the source compute node other than cleaning up it manually?
> 3. Should we add support of evacuating of resized instances in nova?
>
> Please let me know your opinions about the same.
>
>
> Thank you,
>
> Abhishek Kekane
>
>
>
> -Original Message-
> From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com]
> Sent: Thursday, June 29, 2017 5:57 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances 
> in resized state
>
> Hi Chris,
>
> IMO we cannot perform auto-confirm as confirming or reverting is user's 
> choice, whereas reverting is not possible as the node where the instance is 
> resized is down.
> As suggested by you allowing this in nova require additional work. It is 
> possible if we take power-state into consideration for evacuation operation, 
> i.e. while evacuation if instance vmstate is resized and power-state is 
> shutoff then we can stop that instance after evacuation.
>
> Thank you,
>
> Abhishek Kekane
>
> -Original Message-
> From: Chris Friesen [mailto:chris.frie...@windriver.com]
> Sent: Wednesday, June 28, 2017 8:54 PM
> To: openstack-...@lists.openstack.org
> Subject: Re: [openstack-dev] [masakari][nova] Allow evacuation of instances 
> in resized state
>
> On 06/28/2017 05:50 AM, Kekane, Abhishek wrote:
>
>> In masakari, we are setting an instance to an error state if the
>> vmstate is resized before evacuating it to a new host.
>
> Arguably the instance should be set to an error state as soon as you notice 
> that the compute node is down.
>
>> Once an instance (which was in
>> resized state) is evacuated then it becomes active on the new host.
>> The main problem with this implementation from user’s point of view is
>> the instance goes into active state after evacuation, it should be in
>> stopped state if the prior action on the instance before resizing was
>> stop. In masakari, It’s possible to set the vm state to stopped state
>> after evacuation but for a short period the instance will go into the active 
>> state which is unacceptable.
>
> That's a valid point, I think.
>
>> *Proposing changes to Nova:*
>>
>> In the current nova code, if the instance is in stopped

Re: [Openstack-operators] OpenStack User Survey Now Open

[Saverio Proto]

Allison I clicked on "Add Deployment" and I got a 404 page (with a cat)

The URL I was redirected to is:
https://www.openstack.org/%7B$Controller.Link%7D%7B$CurrentStep.Template.Title%7D/add-entity

Saverio

2017-06-26 23:44 GMT+02:00 Allison Price :
> Hi everyone,
>
> If you’re running OpenStack, please participate in the OpenStack User
> Survey. If you have already completed the survey before, you can simply
> login to update your deployment details. Please note that if your survey
> response has not been updated in 12 months, it will expire, so we encourage
> you to take this time to update your existing profile so your deployment can
> be included in the upcoming analysis.
>
> As a member of our community, please help us spread the word. We're trying
> to gather as much real-world deployment data as possible to share back with
> you. We have made it easier to complete, and the survey is now available in
> 7 languages—English, German, Indonesian, Japanese, Korean, traditional
> Chinese and simplified Chinese.
>
> The information provided is confidential and will only be presented in
> aggregate unless you consent to making it public.
>
> The deadline to complete the survey and be part of the next report is
> Friday, August 11 at 23:59 UTC.
>
> You can login and complete the OpenStack User Survey here:
> http://www.openstack.org/user-survey
> If you’re interested in joining the OpenStack User Survey Working Group to
> help with the survey analysis, please complete this form:
> https://openstackfoundation.formstack.com/forms/user_survey_working_group
> Help us promote the User Survey:
> https://twitter.com/OpenStack/status/879434563134652416
>
>
> Please let me know if you have any questions.
>
> Cheers,
> Allison
>
>
> Allison Price
> OpenStack Foundation
> alli...@openstack.org
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ocata instance snapshot

[Saverio Proto]

Hello Ignazio,

do you mean the instance has booted from volume ?

when is booted from volume this 0 bytes glance image is created
together with a cinder snapshot.

I wrote about it for my openstack users here:
https://help.switch.ch/engines/documentation/backup-with-snapshots/

I think that is by design. Also my opinion is that the current design
with booting from volumes or images and different snapshotting paths
is very confusing for the users. That is why I had to write that page.

2017-06-23 10:29 GMT+02:00 Ignazio Cassano :
> I created the instance from an image and the instance has an attached volume
> (it is NOT ephemeral).
> I checked that in this case it makes a volume snapshot (probably because the
> istance is attached to a volume) but also an empty  image (why ?)
> In the case the instance is ethemeral (no volumes attached) , a not empty
> image is created.
> Regards
> Ignazio
>
> 2017-06-23 10:10 GMT+02:00 Ahmed Mostafa :
>>
>> What do you mean by the instance has an attached volume ?
>>
>> Have you started the instance from a volume ?
>>
>> On Fri, Jun 23, 2017 at 8:06 AM, Ignazio Cassano
>>  wrote:
>>>
>>> Hi All,
>>> using ocata dashboard I am trying to create instance snapshots.
>>> If the instance has an attached volume, the image created is empty.
>>> If the instance does not have an attached volume, the image created is
>>> not empty.
>>> This is by design or I must change my configuration ?
>>>
>>> Regards
>>> Ignazio
>>>
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova][cinder] Is there interest in an admin-api to refresh volume connection info?

[Saverio Proto]

Hello Matt,

It is true that we are refreshing something that rarely changes. But
if you deliver a cloud service for several years, at one point you
might have to do these parameters changes.

Something that should not change rarely are the secrets of the ceph
users to talk to the ceph cluster. Good security would suggest
periodic secret rotation, but today this is not really feasible.

I know the problem is also that you cannot change stuff in libvirt
while the VMs are running. Maybe is time for a discussion with libvirt
developers to make our voice louder about required features ?

The goal would be to change on the fly the ceph/rbd secret that a VM
uses to access a volume, while the VM is running. I think this is very
important.

thank you

Saverio


2017-06-09 6:15 GMT+02:00 Matt Riedemann :
> On 6/8/2017 1:39 PM, melanie witt wrote:
>>
>> On Thu, 8 Jun 2017 08:58:20 -0500, Matt Riedemann wrote:
>>>
>>> Nova stores the output of the Cinder os-initialize_connection info API in
>>> the Nova block_device_mappings table, and uses that later for making volume
>>> connections.
>>>
>>> This data can get out of whack or need to be refreshed, like if your ceph
>>> server IP changes, or you need to recycle some secret uuid for your ceph
>>> cluster.
>>>
>>> I think the only ways to do this on the nova side today are via volume
>>> detach/re-attach, reboot, migrations, etc - all of which, except live
>>> migration, are disruptive to the running guest.
>>
>>
>> I believe the only way to work around this currently is by doing a 'nova
>> shelve' followed by a 'nova unshelve'. That will end up querying the
>> connection_info from Cinder and update the block device mapping record for
>> the instance. Maybe detach/re-attach would work too but I can't remember
>> trying it.
>
>
> Shelve has it's own fun set of problems like the fact it doesn't terminate
> the connection to the volume backend on shelve. Maybe that's not a problem
> for Ceph, I don't know. You do end up on another host though potentially,
> and it's a full delete and spawn of the guest on that other host. Definitely
> disruptive.
>
>>
>>> I've kicked around the idea of adding some sort of admin API interface
>>> for refreshing the BDM.connection_info on-demand if needed by an operator.
>>> Does anyone see value in this? Are operators doing stuff like this already,
>>> but maybe via direct DB updates?
>>>
>>> We could have something in the compute API which calls down to the
>>> compute for an instance and has it refresh the connection_info from Cinder
>>> and updates the BDM table in the nova DB. It could be an admin action API,
>>> or part of the os-server-external-events API, like what we have for the
>>> 'network-changed' event sent from Neutron which nova uses to refresh the
>>> network info cache.
>>>
>>> Other ideas or feedback here?
>>
>>
>> We've discussed this a few times before and we were thinking it might be
>> best to handle this transparently and just do a connection_info refresh +
>> record update inline with the request flows that will end up reading
>> connection_info from the block device mapping records. That way, operators
>> won't have to intervene when connection_info changes.
>
>
> The thing that sucks about this is if we're going to be refreshing something
> that maybe rarely changes for every volume-related operation on the
> instance. That seems like a lot of overhead to me (nova/cinder API
> interactions, Cinder interactions to the volume backend, nova-compute round
> trips to conductor and the DB to update the BDM table, etc).
>
>>
>> At least in the case of Ceph, as long as a guest is running, it will
>> continue to work fine if the monitor IPs or secrets change because it will
>> continue to use its existing connection to the Ceph cluster. Things go wrong
>> when an instance action such as resize, stop/start, or reboot is done
>> because when the instance is taken offline and being brought back up, the
>> stale connection_info is read from the block_device_mapping table and
>> injected into the instance, and so it loses contact with the cluster. If we
>> query Cinder and update the block_device_mapping record at the beginning of
>> those actions, the instance will get the new connection_info.
>>
>> -melanie
>>
>>
>
>
> --
>
> Thanks,
>
> Matt
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Neutron not clearing up deleted routers

[Saverio Proto]

I patched this back in liberty.
What version of Openstack are you using ?

Saverio


2017-06-08 19:36 GMT+02:00 Grant Morley :

> Ignore that now all,
>
> Managed to fix it by restarting the l3-agent. Looks like it must have been
> cached in memory.
>
> Thanks,
>
> On 08/06/17 18:07, Grant Morley wrote:
>
> Hi All,
>
> We have  noticed in our neutron-l3-agent logs that there are a number of
> routers that neutron seems to think exist ( but they no longer exist in the
> data base )  and it is constantly trying to delete them. However because
> they don't appear to exist,  so neutron seems to get stuck in a loop of
> trying to delete  routers that are no longer there.
>
> We originally noticed that neutron was trying to delete the
> "qrouter--xxx-" and that didn't exist. So we added the router
> manually with "ip netns add qrouter-xxx-xxx-xxx-xxx" which then deletes the
> router however we then get the following error messages:
>
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent [-] Error while
> deleting router da7b633a-233b-46d1-ba3d-315b3bee6a61
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent Traceback (most
> recent call last):
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/agent.py", line 359, in _safe_router_removed
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> self._router_removed(router_id)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/agent.py", line 377, in _router_removed
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> ri.delete(self)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/ha_router.py", line 380, in delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> super(HaRouter, self).delete(agent)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/router_info.py", line 361, in delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> self.process_delete(agent)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/common/utils.py", line 385, in call
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent self.logger(e)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/oslo_utils/excutils.py", line 220, in __exit__
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> self.force_reraise()
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> six.reraise(self.type_, self.value, self.tb)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/common/utils.py", line 382, in call
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent return
> func(*args, **kwargs)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/router_info.py", line 972, in process_delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> self._process_external_on_delete(agent)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/router_info.py", line 794, in
> _process_external_on_delete
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> self._process_external_gateway(ex_gw_port, agent.pd)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/router_info.py", line 693, in
> _process_external_gateway
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> self.external_gateway_removed(self.ex_gw_port, interface_name)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/ha_router.py", line 371, in
> external_gateway_removed
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> interface_name)
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-
> packages/neutron/agent/l3/router_info.py", line 668, in
> external_gateway_removed
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent
> ip_addr['prefixlen']))
> 2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File
> "/openstack/venvs/neutron-13.3.8/lib/python2.7/site-

Re: [Openstack-operators] Ceph recovery going unusually slow

[Saverio Proto]

I would start troubleshooting restarting those OSDs that dont respond
to ceph pg  query.

Check the log files of those OSDs when you restart.

Saverio

2017-06-02 14:42 GMT+02:00 Grant Morley :
> We was just putting a security patch on that got released.
>
> After that on one of the nodes we ran a service ceph-all restart and the
> cluster went into error status.
>
> What we have been doing to query the pgs so far is:
>
> We get a list of the PGs from the output of ceph health detail
>
> or we have used: ceph pg dump_stuck inactive
>
> we then have a list of about 200 PGs and they are either peering or
> remapped+peering or inactive.
>
> When we query those we use: ceph pg PGNUM query
>
> Sometimes this replies with something like:
>
>
> https://pastebin.com/m3bH34RB
>
>
> Or we get a huge output that seems to be a sequence of peering and then
> remapping , then end of which looks like this:
>
> https://pastebin.com/EwW5rftq
>
> Or we get no reply at all and it just hangs forever.
>
> So, we have OSDs that hang forever when contacted directly using: ceph pg
>  query.  If we look at the OSDs then we also have OSDs that are not
> responding to queries such as: ceph tell osd.24 version - which also hangs
> forever.  If we restart the OSD service it can reply and then hang again
> forever.
>
> We may have more than 1 problem.  1 OSDs hanging on queries such as a
> simple: ceph tell osd.XX version
>
> What causes that?
>
> The other is the PGs that are not peering correctly, the NICs are all
> correct, we tested the network connection and it is working and the ports
> are open, but the peering process is not working between the OSDs for some
> PGs and we have been unable to unstick it.
>
> Thanks,
>
>
> On Fri, Jun 2, 2017 at 1:18 PM, Tyanko Aleksiev 
> wrote:
>>
>> Additionally, it could be useful to know what did you do during the
>> maintenance.
>>
>> Cheers,
>> Tyanko
>>
>> On 2 June 2017 at 14:08, Saverio Proto  wrote:
>>>
>>> To give you some help you need to tell us the ceph version you are
>>> using and from ceph.conf in the section [osd] what values you have for
>>> the following ?
>>>
>>> [osd]
>>> osd max backfills
>>> osd recovery max active
>>> osd recovery op priority
>>>
>>> these three settings can influence the recovery speed.
>>>
>>> Also, do you have big enough limits ?
>>>
>>> Check on any host the content of: /proc/`pid_of_the_osd`/limits
>>>
>>>
>>> Saverio
>>>
>>> 2017-06-02 14:00 GMT+02:00 Grant Morley :
>>> > HEALTH_ERR 210 pgs are stuck inactive for more than 300 seconds; 296
>>> > pgs
>>> > backfill_wait; 3 pgs backfilling; 1 pgs degraded; 202 pgs peering; 1
>>> > pgs
>>> > recovery_wait; 1 pgs stuck degraded; 210 pgs stuck inactive; 510 pgs
>>> > stuck
>>> > unclean; 3308 requests are blocked > 32 sec; 41 osds have slow
>>> > requests;
>>> > recovery 2/11091408 objects degraded (0.000%); recovery
>>> > 1778127/11091408
>>> > objects misplaced (16.032%); nodown,noout,noscrub,nodeep-scrub flag(s)
>>> > set
>>> >
>>> > pg 3.235 is stuck inactive for 138232.508429, current state peering,
>>> > last
>>> > acting [11,26,1]
>>> > pg 1.237 is stuck inactive for 138260.482588, current state peering,
>>> > last
>>> > acting [8,41,34]
>>> > pg 2.231 is stuck inactive for 138258.316031, current state peering,
>>> > last
>>> > acting [24,53,8]
>>> > pg 2.22e is stuck inactive for 194033.321591, current state
>>> > remapped+peering, last acting [0,29,1]
>>> > pg 1.22c is stuck inactive for 102514.200154, current state peering,
>>> > last
>>> > acting [51,7,20]
>>> > pg 2.228 is stuck inactive for 138258.317797, current state peering,
>>> > last
>>> > acting [53,4,34]
>>> > pg 1.227 is stuck inactive for 138258.244681, current state
>>> > remapped+peering, last acting [48,35,11]
>>> > pg 2.220 is stuck inactive for 193940.066322, current state
>>> > remapped+peering, last acting [9,39,8]
>>> > pg 1.222 is stuck inactive for 101474.087688, current state peering,
>>> > last
>>> > acting [23,11,35]
>>> > pg 3.130 is stuck inactive for 99735.451290, current state peering,
>>> > last
>>> > acting [27,37,17]
>>> > pg 3.136 is stuck inac

Re: [Openstack-operators] Ceph recovery going unusually slow

[Saverio Proto]

With

ceph health detail

get a list of problematic pgs

with
ceph pg  query

check why the pgs are stuck.

Check the log files of all OSDs on that restart where the restart
triggered the problem.

Saverio





2017-06-02 14:16 GMT+02:00 Grant Morley :
> We are using Ceph Jewel (10.2.7) running on Ubuntu 14.04LTS
>
> osd_recovery_max_active": "1"
> osd_max_backfills": "1"
> osd_recovery_op_priority": "3"
>
> Limit Soft Limit   Hard Limit   Units
> Max cpu time  unlimitedunlimitedseconds
> Max file size unlimitedunlimitedbytes
> Max data size unlimitedunlimitedbytes
> Max stack size8388608  unlimitedbytes
> Max core file size0unlimitedbytes
> Max resident set  unlimitedunlimitedbytes
> Max processes 256369   256369
> processes
> Max open files327680   327680   files
> Max locked memory 6553665536bytes
> Max address space unlimitedunlimitedbytes
> Max file locksunlimitedunlimitedlocks
> Max pending signals   256369   256369   signals
> Max msgqueue size 819200   819200   bytes
> Max nice priority 00
> Max realtime priority 00
> Max realtime timeout  unlimitedunlimitedus
>
> We did try changing the osd_recovery_max_active to "3" but that seemed tlo
> make things run slower
>
> Thanks,
>
> On Fri, Jun 2, 2017 at 1:08 PM, Saverio Proto  wrote:
>>
>> To give you some help you need to tell us the ceph version you are
>> using and from ceph.conf in the section [osd] what values you have for
>> the following ?
>>
>> [osd]
>> osd max backfills
>> osd recovery max active
>> osd recovery op priority
>>
>> these three settings can influence the recovery speed.
>>
>> Also, do you have big enough limits ?
>>
>> Check on any host the content of: /proc/`pid_of_the_osd`/limits
>>
>>
>> Saverio
>>
>> 2017-06-02 14:00 GMT+02:00 Grant Morley :
>> > HEALTH_ERR 210 pgs are stuck inactive for more than 300 seconds; 296 pgs
>> > backfill_wait; 3 pgs backfilling; 1 pgs degraded; 202 pgs peering; 1 pgs
>> > recovery_wait; 1 pgs stuck degraded; 210 pgs stuck inactive; 510 pgs
>> > stuck
>> > unclean; 3308 requests are blocked > 32 sec; 41 osds have slow requests;
>> > recovery 2/11091408 objects degraded (0.000%); recovery 1778127/11091408
>> > objects misplaced (16.032%); nodown,noout,noscrub,nodeep-scrub flag(s)
>> > set
>> >
>> > pg 3.235 is stuck inactive for 138232.508429, current state peering,
>> > last
>> > acting [11,26,1]
>> > pg 1.237 is stuck inactive for 138260.482588, current state peering,
>> > last
>> > acting [8,41,34]
>> > pg 2.231 is stuck inactive for 138258.316031, current state peering,
>> > last
>> > acting [24,53,8]
>> > pg 2.22e is stuck inactive for 194033.321591, current state
>> > remapped+peering, last acting [0,29,1]
>> > pg 1.22c is stuck inactive for 102514.200154, current state peering,
>> > last
>> > acting [51,7,20]
>> > pg 2.228 is stuck inactive for 138258.317797, current state peering,
>> > last
>> > acting [53,4,34]
>> > pg 1.227 is stuck inactive for 138258.244681, current state
>> > remapped+peering, last acting [48,35,11]
>> > pg 2.220 is stuck inactive for 193940.066322, current state
>> > remapped+peering, last acting [9,39,8]
>> > pg 1.222 is stuck inactive for 101474.087688, current state peering,
>> > last
>> > acting [23,11,35]
>> > pg 3.130 is stuck inactive for 99735.451290, current state peering, last
>> > acting [27,37,17]
>> > pg 3.136 is stuck inactive for 138221.552865, current state peering,
>> > last
>> > acting [26,49,10]
>> > pg 3.13c is stuck inactive for 137563.906503, current state peering,
>> > last
>> > acting [51,53,7]
>> > pg 2.142 is stuck inactive for 99962.462932, current state peering, last
>> > acting [37,16,34]
>> > pg 1.141 is stuck inactive for 138257.572476, current state
>> > remapped+peering, last acting [5,17,49]
>> > pg 2.141 is s

Re: [Openstack-operators] Ceph recovery going unusually slow

[Saverio Proto]

To give you some help you need to tell us the ceph version you are
using and from ceph.conf in the section [osd] what values you have for
the following ?

[osd]
osd max backfills
osd recovery max active
osd recovery op priority

these three settings can influence the recovery speed.

Also, do you have big enough limits ?

Check on any host the content of: /proc/`pid_of_the_osd`/limits


Saverio

2017-06-02 14:00 GMT+02:00 Grant Morley :
> HEALTH_ERR 210 pgs are stuck inactive for more than 300 seconds; 296 pgs
> backfill_wait; 3 pgs backfilling; 1 pgs degraded; 202 pgs peering; 1 pgs
> recovery_wait; 1 pgs stuck degraded; 210 pgs stuck inactive; 510 pgs stuck
> unclean; 3308 requests are blocked > 32 sec; 41 osds have slow requests;
> recovery 2/11091408 objects degraded (0.000%); recovery 1778127/11091408
> objects misplaced (16.032%); nodown,noout,noscrub,nodeep-scrub flag(s) set
>
> pg 3.235 is stuck inactive for 138232.508429, current state peering, last
> acting [11,26,1]
> pg 1.237 is stuck inactive for 138260.482588, current state peering, last
> acting [8,41,34]
> pg 2.231 is stuck inactive for 138258.316031, current state peering, last
> acting [24,53,8]
> pg 2.22e is stuck inactive for 194033.321591, current state
> remapped+peering, last acting [0,29,1]
> pg 1.22c is stuck inactive for 102514.200154, current state peering, last
> acting [51,7,20]
> pg 2.228 is stuck inactive for 138258.317797, current state peering, last
> acting [53,4,34]
> pg 1.227 is stuck inactive for 138258.244681, current state
> remapped+peering, last acting [48,35,11]
> pg 2.220 is stuck inactive for 193940.066322, current state
> remapped+peering, last acting [9,39,8]
> pg 1.222 is stuck inactive for 101474.087688, current state peering, last
> acting [23,11,35]
> pg 3.130 is stuck inactive for 99735.451290, current state peering, last
> acting [27,37,17]
> pg 3.136 is stuck inactive for 138221.552865, current state peering, last
> acting [26,49,10]
> pg 3.13c is stuck inactive for 137563.906503, current state peering, last
> acting [51,53,7]
> pg 2.142 is stuck inactive for 99962.462932, current state peering, last
> acting [37,16,34]
> pg 1.141 is stuck inactive for 138257.572476, current state
> remapped+peering, last acting [5,17,49]
> pg 2.141 is stuck inactive for 102567.745720, current state peering, last
> acting [36,7,15]
> pg 3.144 is stuck inactive for 138218.289585, current state
> remapped+peering, last acting [18,28,16]
> pg 1.14d is stuck inactive for 138260.030530, current state peering, last
> acting [46,43,17]
> pg 3.155 is stuck inactive for 138227.368541, current state
> remapped+peering, last acting [33,20,52]
> pg 2.8d is stuck inactive for 100251.802576, current state peering, last
> acting [6,39,27]
> pg 2.15c is stuck inactive for 102567.512279, current state
> remapped+peering, last acting [7,35,49]
> pg 2.167 is stuck inactive for 138260.093367, current state peering, last
> acting [35,23,17]
> pg 3.9d is stuck inactive for 117050.294600, current state peering, last
> acting [12,51,23]
> pg 2.16e is stuck inactive for 99846.214239, current state peering, last
> acting [25,5,8]
> pg 2.17b is stuck inactive for 99733.504794, current state peering, last
> acting [49,27,14]
> pg 3.178 is stuck inactive for 99973.600671, current state peering, last
> acting [29,16,40]
> pg 3.240 is stuck inactive for 28768.488851, current state remapped+peering,
> last acting [33,8,32]
> pg 3.b6 is stuck inactive for 138222.461160, current state peering, last
> acting [26,29,34]
> pg 2.17e is stuck inactive for 159229.154401, current state peering, last
> acting [13,42,48]
> pg 2.17c is stuck inactive for 104921.767401, current state
> remapped+peering, last acting [23,12,24]
> pg 3.17d is stuck inactive for 137563.979966, current state
> remapped+peering, last acting [43,24,29]
> pg 1.24b is stuck inactive for 93144.933177, current state peering, last
> acting [43,20,37]
> pg 1.bd is stuck inactive for 102616.793475, current state peering, last
> acting [16,30,35]
> pg 3.1d6 is stuck inactive for 99974.485247, current state peering, last
> acting [16,38,29]
> pg 2.172 is stuck inactive for 193919.627310, current state inactive, last
> acting [39,21,10]
> pg 1.171 is stuck inactive for 104947.558748, current state peering, last
> acting [49,9,25]
> pg 1.243 is stuck inactive for 208452.393430, current state peering, last
> acting [45,32,24]
> pg 3.aa is stuck inactive for 104958.230601, current state remapped+peering,
> last acting [51,12,13]
>
> 41 osds have slow requests
> recovery 2/11091408 objects degraded (0.000%)
> recovery 1778127/11091408 objects misplaced (16.032%)
> nodown,noout,noscrub,nodeep-scrub flag(s) set
>
> That is what we seem t

Re: [Openstack-operators] Ceph recovery going unusually slow

[Saverio Proto]

Usually 'ceph health detail' gives better info on what is making
everything stuck.

Saverio

2017-06-02 13:51 GMT+02:00 Grant Morley :
> Hi All,
>
> I wonder if anyone could help at all.
>
> We were doing some routine maintenance on our ceph cluster and after running
> a "service ceph-all restart" on one of our nodes we noticed that something
> wasn't quite right. The cluster has gone into an error mode and we have
> multiple stuck PGs and the object replacement recovery is taking a strangely
> long time. At first there was about 46% objects misplaced and we now have
> roughly 16%.
>
> However it has taken about 36 hours to do the recovery so far and with a
> possible 16 to go we are looking at a fairly major issue. As a lot of the
> system is now blocked for read / writes, customers cannot access their VMs.
>
> I think the main issue at the moment is that we have 210pgs stuck inactive
> and nothing we seem to do can get them to peer.
>
> Below is an ouptut of the ceph status. Can anyone help or have any ideas on
> how to speed up the recover process? We have tried turning down logging on
> the OSD's but some are going so slow they wont allow us to injectargs into
> them.
>
> health HEALTH_ERR
> 210 pgs are stuck inactive for more than 300 seconds
> 298 pgs backfill_wait
> 3 pgs backfilling
> 1 pgs degraded
> 200 pgs peering
> 1 pgs recovery_wait
> 1 pgs stuck degraded
> 210 pgs stuck inactive
> 512 pgs stuck unclean
> 3310 requests are blocked > 32 sec
> recovery 2/11094405 objects degraded (0.000%)
> recovery 1785063/11094405 objects misplaced (16.090%)
> nodown,noout,noscrub,nodeep-scrub flag(s) set
>
> election epoch 16314, quorum 0,1,2,3,4,5,6,7,8
> storage-1,storage-2,storage-3,storage-4,storage-5,storage-6,storage-7,storage-8,storage-9
>  osdmap e213164: 54 osds: 54 up, 54 in; 329 remapped pgs
> flags nodown,noout,noscrub,nodeep-scrub
>   pgmap v41030942: 2036 pgs, 14 pools, 14183 GB data, 3309 kobjects
> 43356 GB used, 47141 GB / 90498 GB avail
> 2/11094405 objects degraded (0.000%)
> 1785063/11094405 objects misplaced (16.090%)
> 1524 active+clean
>  298 active+remapped+wait_backfill
>  153 peering
>   47 remapped+peering
>   10 inactive
>3 active+remapped+backfilling
>1 active+recovery_wait+degraded+remapped
>
> Many thanks,
>
> Grant
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Routed provider networks...

[Saverio Proto]

> We use provider networks to essentially take neutron-l3 out of the equation. 
> Generally they are shared on all compute hosts, but usually there aren't huge 
> numbers of computes.

Hello,

we have a datacenter completely L3, routing to the host.

to implement the provider networks we are using the neutron-l2gw plugin.

basically there is a Switch where we bridge a physical port attached
to the provider network, with a virtual VXLAN tenant network. The
config of the switch is managed by neutron using the l2gw-plugin.

Cheers,

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Neutron Issues

[Saverio Proto]

is the instance scheduled to an hypervisor ? check this with openstack
server show uuid
(admin credential)

if yes check nova-compute.log on the hypervisor. maybe you find some good
information to debug

Saverio

Il 03 mag 2017 2:16 AM, "Steve Powell"  ha
scritto:

[ml2]

type_drivers = flat,vlan,vxlan

tenant_network_types = vxlan

mechanism_drivers = linuxbridge,l2population

extension_drivers = port_security



[ml2_type_flat]

flat_networks = provider









*From:* Neil Jerram [mailto:n...@tigera.io]
*Sent:* Tuesday, May 2, 2017 7:58 PM
*To:* Steve Powell ; Chris Sarginson <
csarg...@gmail.com>; openstack-operators@lists.openstack.org

*Subject:* Re: [Openstack-operators] Neutron Issues



I think you probably need to say what ML2 mechanism driver(s) you are using.





On Tue, May 2, 2017 at 10:29 PM Steve Powell 
wrote:

No OVS here. Thanks though! This one has me stumped!



*From:* Chris Sarginson [mailto:csarg...@gmail.com]
*Sent:* Tuesday, May 2, 2017 5:24 PM
*To:* Steve Powell ; openstack-operators@lists.
openstack.org
*Subject:* Re: [Openstack-operators] Neutron Issues



If you're using openvswitch, with Newton there was a change to the default
agent for configuring openvswitch to be the python ryu library, I think
it's been mentioned on here recently, so probably worth having a poke
through the archives for more information.  I'd check your neutron
openvswitch agent logs for errors pertaining to openflow configuration
specifically, and if you see anything, it's probably worth applying the
following config to your ml2 ini file under the [OVS] section:



of_interface = ovs-ofctl



https://docs.openstack.org/mitaka/config-reference/
networking/networking_options_reference.html



Then restart the neutron openvswitch agent, watch the logs, hopefully this
is of some use to you.



On Tue, 2 May 2017 at 21:30 Steve Powell  wrote:

I forgot to mention I’m running Newton and my neutron.conf file is below
and I’m running haproxy.



[DEFAULT]

core_plugin = ml2

service_plugins = router

allow_overlapping_ips = True

notify_nova_on_port_status_changes = True

notify_nova_on_port_data_changes = True

transport_url = rabbit://openstack:#@x.x.x.x

auth_strategy = keystone



[agent]

root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf



[cors]



[cors.subdomain]



[database]

connection = mysql+pymysql://neutron:###@
10.10.6.220/neutron



[keystone_authtoken]

auth_url = http://x.x.x.x:35357/v3

auth_uri = https://xxx..xxx:5000/v3

memcached_servers = x.x.x.x:11211

auth_type = password

project_domain_name = Default

user_domain_name = Default

project_name = service

username = neutron

password = ##





[matchmaker_redis]



[nova]



auth_url = http://x.x.x.x:35357/v3

auth_type = password

project_domain_name = Default

user_domain_name = Default

region_name = RegionOne

project_name = service

username = nova

password = ###



[oslo_concurrency]



[oslo_messaging_amqp]



[oslo_messaging_notifications]



[oslo_messaging_rabbit]



[oslo_messaging_zmq]



[oslo_middleware]

enable_proxy_headers_parsing = True

enable_http_proxy_to_wsgi = True



[oslo_policy]



[qos]



[quotas]



[ssl]



*From:* Steve Powell [mailto:spow...@silotechgroup.com]
*Sent:* Tuesday, May 2, 2017 4:16 PM
*To:* openstack-operators@lists.openstack.org
*Subject:* [Openstack-operators] Neutron Issues



This sender failed our fraud detection checks and may not
be who they appear to be. Learn about spoofing


Feedback 

Hello Ops!



I have a major issue slapping me in the face and seek any assistance
possible. When trying to spin up and instance whether from the command
line, manually in Horizon, or with a HEAT template I receive the following
error in nova and, where applicable, heat logs:



Failed to allocate the network(s), not rescheduling.



I see in the neutron logs where the request make it through to completion
but that info is obviously not making it back to nova.



INFO neutron.notifiers.nova [-] Nova event response: {u'status':
u'completed', u'code': 200, u'name': u'network-changed', u'server_uuid':
u'6892bb9e-4256-4fc9-a313-331f0c576a03'}



What am I missing? Why would the response from neutron not make it back to
nova?







___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac

Re: [Openstack-operators] [Mitaka - Heat] HTTPInternalServerError: ERROR: Remote error: BadRequest Expecting to find domain in user

[Saverio Proto]

Edgar,

as admin, and using V3, do the following command:

openstack user show heat

in the field domain_id do you see default or Default with capital D ?

in the heat config you pasted you have capital D, is that correct ?

ciao,

Saverio



2017-05-02 21:47 GMT+02:00 Edgar Magana :
> Hi,
>
> Yes, all that is in place. This is why #openstack stack list works.
> The openstack client is:
>
> # yum list installed |grep client
> python-openstackclient.noarch  3.2.0-2.el7  @3rdParty7
> python2-heatclient.noarch  1.5.0-1.el7  @3rdParty7
>
> Edgar
>
> On 5/2/17, 12:39 PM, "Saverio Proto"  wrote:
>
> Hello Edgar,
>
> what is the version of the openstack client ?
>
> did you export this?
>
> export OS_IDENTITY_API_VERSION=3
>
> also the auth URL should have the v3, like;
>
> export 
> OS_AUTH_URL=https://urldefense.proofpoint.com/v2/url?u=https-3A__hostname-3A5000_v3&d=DwIBaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=bwlIMYJQM6k26RrwtU0-iHi3-63kP3vUZLLHvFNSWZ0&s=aS1zOIR-pRfqhzkrTpilZrUBGwKku7fn_O5OjaYOV3g&e=
>
> check the client configuration and give us feedback,
>
> ciao,
>
> Saverio
>
> 2017-05-02 21:15 GMT+02:00 Edgar Magana :
> > Hello Ops,
> >
> >
> >
> > In our Mitaka Deployment with SSL and Keystone v3, we are having the 
> below
> > errors while trying to create a stack but they do not show while listing
> > them:
> >
> >
> >
> > # openstack stack list
> >
> > WARNING: openstackclient.common.utils is deprecated and will be removed
> > after Jun 2017. Please use osc_lib.utils
> >
> >
> >
> > [root@wpc-controller ~]# openstack stack create --template 
> instance-hot.yaml
> > hot-instance
> >
> > ERROR: Remote error: BadRequest Expecting to find domain in user - the
> > server could not comply with the request since it is either malformed or
> > otherwise incorrect. The client is assumed to be in error. (HTTP 400)
> > (Request-ID: req-e281f8bd-c7cd-486f-87be-dd6401b25cd3)
> >
> > [u'
> >
> >
> >
> > Our Heat Conf:
> >
> > [DEFAULT]
> >
> > log_dir = /var/log/heat
> >
> > heat_metadata_server_url =
> > 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__wpc-2Dcontroller.cedev1.d002.eng.pdx.wd-3A8000&d=DwIBaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=bwlIMYJQM6k26RrwtU0-iHi3-63kP3vUZLLHvFNSWZ0&s=JENHqqh-HFBKqgqYOoBuhULMNKV1XIV3lrsM2GKWfio&e=
> >
> > heat_waitcondition_server_url =
> > 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__wpc-2Dcontroller.cedev1.d002.eng.pdx.wd-3A8000_v1_waitcondition&d=DwIBaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=bwlIMYJQM6k26RrwtU0-iHi3-63kP3vUZLLHvFNSWZ0&s=3ja1hV81lm9YdBA3c9asFSLij9cYr9toU8oZ4giTHAQ&e=
> >
> > heat_watch_server_url = 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__wpc-2Dcontroller.cedev1.d002.eng.pdx.wd-3A8003&d=DwIBaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=bwlIMYJQM6k26RrwtU0-iHi3-63kP3vUZLLHvFNSWZ0&s=X_TRvJLmYhjJXyOPwbfGan5uYINzsBwkyc6VfmIlZNE&e=
> >
> > region_name_for_services = RegionOne
> >
> > num_engine_workers = 5
> >
> > auth_encryption_key = orchestration_auth_encryption_32
> >
> >
> >
> > # Domain
> >
> > stack_domain_admin_password = service_pass
> >
> > stack_domain_admin = heat_domain_admin
> >
> > stack_user_domain = d41933e4c1314a9e8f589db66cc9f844
> >
> > deferred_auth_method=trusts
> >
> >
> >
> >
> >
> > [oslo_messaging_notifications]
> >
> > driver = heat.openstack.common.notifier.rpc_notifier
> >
> >
> >
> > [keystone_authtoken]
> >
> > auth_type = v3password
> >
> > username = heat
> >
> > project_name = service
> >
> > project_domain_name = Default
> >
> > user_domain_name = Default
> >
> > auth_url = 
> https://urldefense.proofpoint.com/v2/url?u=http

Re: [Openstack-operators] [Mitaka - Heat] HTTPInternalServerError: ERROR: Remote error: BadRequest Expecting to find domain in user

[Saverio Proto]

Hello Edgar,

what is the version of the openstack client ?

did you export this?

export OS_IDENTITY_API_VERSION=3

also the auth URL should have the v3, like;

export OS_AUTH_URL=https://hostname:5000/v3

check the client configuration and give us feedback,

ciao,

Saverio

2017-05-02 21:15 GMT+02:00 Edgar Magana :
> Hello Ops,
>
>
>
> In our Mitaka Deployment with SSL and Keystone v3, we are having the below
> errors while trying to create a stack but they do not show while listing
> them:
>
>
>
> # openstack stack list
>
> WARNING: openstackclient.common.utils is deprecated and will be removed
> after Jun 2017. Please use osc_lib.utils
>
>
>
> [root@wpc-controller ~]# openstack stack create --template instance-hot.yaml
> hot-instance
>
> ERROR: Remote error: BadRequest Expecting to find domain in user - the
> server could not comply with the request since it is either malformed or
> otherwise incorrect. The client is assumed to be in error. (HTTP 400)
> (Request-ID: req-e281f8bd-c7cd-486f-87be-dd6401b25cd3)
>
> [u'
>
>
>
> Our Heat Conf:
>
> [DEFAULT]
>
> log_dir = /var/log/heat
>
> heat_metadata_server_url =
> https://wpc-controller.cedev1.d002.eng.pdx.wd:8000
>
> heat_waitcondition_server_url =
> https://wpc-controller.cedev1.d002.eng.pdx.wd:8000/v1/waitcondition
>
> heat_watch_server_url = https://wpc-controller.cedev1.d002.eng.pdx.wd:8003
>
> region_name_for_services = RegionOne
>
> num_engine_workers = 5
>
> auth_encryption_key = orchestration_auth_encryption_32
>
>
>
> # Domain
>
> stack_domain_admin_password = service_pass
>
> stack_domain_admin = heat_domain_admin
>
> stack_user_domain = d41933e4c1314a9e8f589db66cc9f844
>
> deferred_auth_method=trusts
>
>
>
>
>
> [oslo_messaging_notifications]
>
> driver = heat.openstack.common.notifier.rpc_notifier
>
>
>
> [keystone_authtoken]
>
> auth_type = v3password
>
> username = heat
>
> project_name = service
>
> project_domain_name = Default
>
> user_domain_name = Default
>
> auth_url = https://wpc-controller.cedev1.d002.eng.pdx.wd:5000/v3
>
> password = $PASSWORD
>
>
>
> [trustee]
>
> auth_plugin = v3password
>
> username = heat
>
> auth_url = https://wpc-controller.cedev1.d002.eng.pdx.wd:35357/v3
>
> password = mypass
>
>
>
> [oslo_messaging_rabbit]
>
> rabbit_userid = admin
>
> rabbit_password = mypass
>
> [clients_keystone]
>
> auth_uri = https://wpc-controller.cedev1.d002.eng.pdx.wd:5000/
>
> [ec2authtoken]
>
> auth_uri = https://wpc-controller.cedev1.d002.eng.pdx.wd:5000/v3
>
> [heat_api]
>
> bind_host = 0.0.0.0
>
> bind_port = 8004
>
> workers = 5
>
> cert_file = /etc/heat/ssl/certs/sslcert.pem
>
> key_file = /etc/heat/ssl/private/sslkey.pem
>
>
>
>
>
>
>
> Anyone has faced this problem? I tried any things and I got stuck!
>
>
>
> Edgar
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] RabbitMQ 3.6.x experience?

[Saverio Proto]

Hello there,

we tried an upgrade from 3.6.6 to 3.6.9 and it just did not work out.
But because of multiple things to tune during my newton upgrade I am
not 100% sure that it is the version of rabbitmq the problem. The fact
is that it was crashing every minute with 3.6.9.

can anyone confirm problems on 3.6.9 or I did something wrong ?

thank you

Saverio


2017-01-11 2:51 GMT+01:00 Matt Fischer :
> On Tue, Jan 10, 2017 at 4:08 PM, Sam Morrison  wrote:
>>
>>
>> > On 10 Jan 2017, at 11:04 pm, Tomáš Vondra  wrote:
>> >
>> > The version is 3.6.2, but the issue that I believe is relevant is still
>> > not fixed:
>> > https://github.com/rabbitmq/rabbitmq-management/issues/41
>> > Tomas
>> >
>>
>> Yeah we found this version unusable, 3.6.5 hasn’t had any problems for us.
>>
>> Sam
>
>
> This whole thread of bugs was my main worry with 3.6.x. It's clear they
> tried a new design and I didn't want to beta test it.
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Openstack Heat https links problems upgrading to Newton

[Saverio Proto]

Hello ops,

if anyone is interested I have problems with Heat and the Newton upgrade.

I sent an email about this here:
http://lists.openstack.org/pipermail/openstack-dev/2017-April/115412.html

If anyone already faced this issue any help would be appreciated !

thank you

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] about the R release name

[Saverio Proto]

Hello Ops,

I got the mail about the Poll for OpenStack R Release Naming

I am shocked that there are proposed names like Raspberry or Root !

Think about troubleshooting and searching on google:

Openstack Raspberry "string of log file"

The Raspberry or Root words are anti-google words that will pollute
the search with a lot of results from other context domain 

Please be smart when voting ! We need names that can identify easily
the releases when using search engines !!!

thank you

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] backup to object store - tool recommendations

[Saverio Proto]

Hello all,

we use rclone a lot, and we are happy with it.

the real problem I would say is that a lot of these tools use the
latest AWS4 signature.

AFAIK the radosgw with Ceph Jewel and Openstack keystone integration
supports only AWS2 signature because of this bug:
http://tracker.ceph.com/issues/19056

is anyone else hitting this ?

Saverio

2017-03-27 22:11 GMT+02:00 John Dickinson :
>
>
> On 27 Mar 2017, at 4:39, Blair Bethwaite wrote:
>
>> Hi all,
>>
>> Does anyone have any recommendations for good tools to perform
>> file-system/tree backups and restores to/from a (Ceph RGW-based)
>> object store (Swift or S3 APIs)? Happy to hear about both FOSS and
>> commercial options please.
>>
>> I'm interested in:
>> 1) tools known to work or not work at all for a basic file-based data backup
>
> There's a bunch of backup tools that will work with the Swift API and/or the 
> S3 API.
>
> Veritas, Commvault, Trilio, and CloudBerry all work. There's other companies 
> too that can back specific stuff up to Swift (e.g. Percona with MySQL).
>
> (The above list taken from https://www.swiftstack.com/solutions/backup [my 
> employer] because it's the first linkable place I knew of to answer your 
> question.)
>
>
> --John
>
>
>
>
>>
>> Plus these extras:
>> 2) preserves/restores correct file metadata (e.g. owner, group, acls etc)
>> 3) preserves/restores xattrs
>> 4) backs up empty directories and files
>> 5) supports some sort of snapshot/versioning/differential
>> functionality, i.e., will keep a copy or diff or last N versions of a
>> file or whole backup set, e.g., so that one can restore yesterday's
>> file/s or last week's but not have to keep two full copies to achieve
>> it
>> 6) is readily able to restore individual files
>> 7) can encrypt/decrypt client side
>> 8) anything else I should be considering?
>>
>> --
>> Cheers,
>> ~Blairo
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [neutron] Modify Default Quotas

[Saverio Proto]

Hello,

floating IPs is the real issue.

When using horizon it is very easy for users to allocate floating ips
but it is also very difficult to release them.

In our production cloud we had to change the default from 50 to 2. We
have to be very conservative with floatingips quota because our
experience is that the user will never release a floating IP.

A good starting point is to set the quota for the floatingips at the
the same quota for nova instances.

Saverio


2017-03-22 16:46 GMT+01:00 Morales, Victor :
> Hey there,
>
>
>
> I noticed that Ihar started working on a change to increase the default
> quotas values in Neutron[1].  Personally, I think that makes sense to change
> it but I’d like to complement it.  So, based on your experience, what should
> be the most common quota value for networks, subnets, ports, security
> groups, security rules, routers and Floating IPs per tenant?
>
>
>
> Regards/Saludos
>
> Victor Morales
>
> irc: electrocucaracha
>
>
>
> [1] https://review.openstack.org/#/c/444030
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] each project could only allocate floating ip from particular pool(external) ?

[Saverio Proto]

Hello,
check if this thread helps you:
http://lists.openstack.org/pipermail/openstack-operators/2016-September/011674.html
http://lists.openstack.org/pipermail/openstack-operators/2016-October/011699.html

Saverio

2017-03-21 18:12 GMT+01:00 Juanjuan Li :
> Hi,everyone! I have been working on this issue for a long time,and I really
> can not find a solution to it. I am trying to set up two external network
> (two network pools),such that project 1 could only allocate floating ip from
> pool 1 and project 2 could only allocate floating ip from pool 2,while
> project 1 can not allocate any floating ip from pool 2 and project 2 can not
> allocate any floating ip from pool 1.Are there any ideas?
>
> I tried all the combinations of external and shared on network creation,
> none of them works for me.
> I also tried the Role-Based Access Control, it does not work either.
> I also tried the quota per project,it does not work for me, because it set
> the quota from all the floating ip address and does not specify different
> pools.
>
> Thanks a lot.
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] ceph rbd root disk unexpected deletion

[Saverio Proto]

Hello Mike,

what version of openstack ?
is the instance booting from ephemeral disk or booting from cinder volume ?

When you boot from volume, that will be the root disk of your
instance. The user could have clicked on "Delete Volume on Instance
Delete". It can be selected when creating a new instance.

Saverio

2017-03-13 15:47 GMT+01:00 Mike Lowe :
> Over the weekend a user reported that his instance was in a stopped state and 
> could not be started, on further examination it appears that the vm had 
> crashed and the strange thing is that the root disk is now gone.  Has anybody 
> come across anything like this before?
>
> And why on earth is it attempting deletion of the rbd device without deletion 
> of the instance?
>
> 2017-03-12 10:59:07.591 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] 
> rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms 
> failed
> 2017-03-12 10:59:17.613 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] 
> rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms 
> failed
> 2017-03-12 10:59:26.143 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] 
> rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms 
> failed
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Thank you for the ops-midcycle in Milano

[Saverio Proto]

Hello !

thank you for the great event. Mariano & all the people from Milano
made an excellent work.

thanks all to all of you that helped moderating the sessions and
contributing to the etherpads.

it was really a great event and I am looking forward for the next ones.

Cheers,

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Operators Milan - Neutron Session

[Saverio Proto]

Hello Edgar,

there was a discussion about this here:
http://eavesdrop.openstack.org/meetings/ops_meetup_team/2017/ops_meetup_team.2017-03-07-20.16.log.txt

we had several sponsor requests to make presentations. We agreed that as
long as the sponsor presentations are strongly technical everybody will be
happy.

ciao,

Saverio



2017-03-13 18:21 GMT+01:00 Edgar Magana :

> WOW! Something has changed from previous Ops Meet-ups. I remember clearly
> that we (Operators), wanted to have a free-from-vendors environment and it
> seems that now we want them to come and provide updates on their offerings.
>
> I think vendors have plenty opportunities to show their features and
> demos, for instance the Summits and OpenStack Days around the world.
>
>
>
> Thanks,
>
>
>
> Edgar
>
>
>
> *From: *Christoph Andreas Torlinsky 
> *Date: *Monday, March 13, 2017 at 10:11 AM
> *To: *"randy.perry...@dell.com" 
> *Cc: *"OpenStack-operators@lists.openstack.org" <
> OpenStack-operators@lists.openstack.org>
> *Subject: *Re: [Openstack-operators] OpenStack Operators Milan - Neutron
> Session
>
>
>
> I'll be there from Nuage Networks, as far as I know.
>
>
>
>
>
>
> Christoph Andreas Torlinski
>
>
>
> christ...@nuagenetworks.net
>
>
>
> On 9 March 2017 at 13:36,  wrote:
>
> Here is the etherpad for the Neutron Session in Milan
>
> https://etherpad.openstack.org/p/MIL-ops-neutron
> 
>
>
>
> Please add to it anything you would like to cover.
>
>
>
> Also will anyone from SDN Vendors be there?  Please come and provide a 5
> minute update to your offering.
>
>
>
> Thank You
>
>
>
> *Randy Perryman*
>
> Technical Staff, Cloud Architect
> Certified OpenStack Administrator
>
>
> * Dell* *EMC* | OpenSource Solutions Group
>
> *office* + 01 603 249 7710
>
> *mobile *+01 603 321 5611
>
> email randy.perry...@dell.com
>
> http://www.dell.com/cloud
> 
>
>
>
> Please consider the environment before printing this email.
>
>
>
> Confidentiality Notice | This e-mail message, including any attachments,
> is for the sole use of the intended recipient(s) and may contain
> confidential or proprietary information. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the intended
> recipient, immediately contact the sender by reply e-mail and destroy all
> copies of the original message.
>
>
>
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Operators Milan

[Saverio Proto]

> Can someone please put together a list of urls to all the sessions, there are 
> so many urls flying around.

Hi,
you can find the list here:
https://etherpad.openstack.org/p/MIL-ops-meetup

Cheers

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] need input on log translations

[Saverio Proto]

Hello there,
I am Italian speaking.
Does not make any sense to have the log messages translated. I think
everything has already being said.

Saverio


2017-03-11 9:20 GMT+01:00 George Shuklin :
> Whole idea with log translation is half-backed anyway. About the half of
> important log messages contain output of things outside openstack. Libvirt,
> ip, sudo, kernel, etc. In any i18n installation there going to be some
> amount of untranslated messages. This kills whole idea of localization.
>
> Modern operator ought to know English at 'technical reading' level anyway.
> Therefore, localization does not achieve it goal, but cause pain instead:
> search segmentation, slightly misleading translation (f.e. 'stream' and
> 'thread' both translate into Russian 'поток', which brings ambiguity),
> different system may use slightly different translation, causing even more
> mess.
>
> As Russian speaker and openstack operator I definitely don't want to have
> logs translation.
>
> On Mar 10, 2017 4:42 PM, "Doug Hellmann"  wrote:
>
> There is a discussion on the -dev mailing list about the i18n team
> decision to stop translating log messages [1]. The policy change means
> that we may be able to clean up quite a lot of "clutter" throughout the
> service code, because without anyone actually translating the messages
> there is no need for the markup code used to tag those strings.
>
> If we do remove the markup from log messages, we will be effectively
> removing "multilingual logs" as a feature. Given the amount of work
> and code churn involved in the first roll out, I would not expect
> us to restore that feature later.
>
> Therefore, before we take what would almost certainly be an
> irreversible action, we would like some input about whether log
> message translations are useful to anyone. Please let us know if
> you or your customers use them.
>
> Thanks,
> Doug
>
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Milano session on Upgrades Patches and Packaging

[Saverio Proto]

Hello,

I prepared the skeleton for the session about Upgrades Patches and Packaging

https://etherpad.openstack.org/p/MIL-ops-upgrades-patches-packaging

Please contribute to the etherpad with stuff like:

 * patches that you are carrying in production
 * patches that you dont manage to get merged upstream
 * upgrade problems
 * regression bugs once you land on the new release

This is a session that is very useful if you have to do soon an
openstack upgrade :)

ciao,

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] nova mitaka->newton online_data_migrations

[Saverio Proto]

Hello there,

before doing the upgrade, when still in Mitaka (tag 13.1.3), I am
running the folllwing command:

nova-manage db online_data_migrations

I get this output:

Option "verbose" from group "DEFAULT" is deprecated for removal.  Its
value may be silently ignored in the future.
Option "notification_topics" from group "DEFAULT" is deprecated. Use
option "topics" from group "oslo_messaging_notifications".
Running batches of 50 until complete

I always get the same output.
 I also tried :
nova-manage db online_data_migrations --max-count 5000

In this case I only get the warnings about the option without 'Running
batches of 50 until complete'.

When I do not use max-count I will always get 'Running batches of 50
until complete'.

How do I understand when all the online_data_migrations are done ?

I never get anything printed like 'all done'.

thank you

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Migration to StoryBoard

[Saverio Proto]

Hello Kendall Nelson,

thanks for you email about Story Board. I did not understand if as
Openstack operators, are we supposed to open bugs on storyboard
instead of Launchpad ? When this starts ?

thank you

Saverio


2017-03-03 20:49 GMT+01:00 Kendall Nelson :
> Hello!
>
> As you may or may not know, OpenStack is in the process of migrating from
> using Launchpad as our task tracker to StoryBoard. This has been a long time
> effort and we are finally getting projects lined up to make the switch.
>
> To keep you all in the loop about the new tool, I wanted to take a moment to
> share with you the information we have been sharing with the projects
> themselves. We have a variety of blogposts we have been writing to help
> educate OpenStack users and developers about the differences between
> Launchpad[1][2] and Storyboard and posts about why we are making the
> switch[3].
>
> As storyboard is being worked on by OpenStack community members we always
> welcome feedback on the tool so if you are interested in playing around with
> the sandbox environment[4], want to share thoughts in one of our weekly
> meetings[5], are interested in the documentation[6], or just want to see the
> real thing and how its being used [7] please do!
>
> - Kendall Nelson (diablo_rojo) & the Storyboard team
>
> [1] https://storyboard-blog.io/things-that-storyboard-does-differently.html
> [2] https://storyboard-blog.io/mapping-launchpad-to-storyboard.html
> [3] https://storyboard-blog.io/why-storyboard-for-openstack.html
> [4] https://storyboard-dev.openstack.org/
> [5] https://wiki.openstack.org/wiki/StoryBoard
> [6] http://docs.openstack.org/infra/storyboard/
> [7] https://storyboard.openstack.org/
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Unable to launch instances because of glance errors

[Saverio Proto]

Everything LTGM

but if you do:

openstack image show Ubuntu-16.04

then you get again the error about the multiple names ?

if you go to the glance database and you do:

select * from images where name=''Ubuntu-16.04";

how many rows you have ?

after we collect this information we can try to understand if this is nova
or glance bug.

thank you

Saverio


2017-03-02 17:01 GMT+01:00 Grant Morley :

> sure no problem:
>
> openstack image show d5d43ba0-82e9-43de-8883-5ebaf07bf3e3
> +--+
> --+
> | Field| Value
>
> |
> +--+
> --+
> | checksum | 6008d3fdf650658965b132b547416d
> 83
> |
> | container_format | bare
>
> |
> | created_at   | 2016-12-07T16:00:54Z
>
> |
> | disk_format  | raw
>
> |
> | file | /v2/images/d5d43ba0-82e9-43de-
> 8883-5ebaf07bf3e3/file
> |
> | id   | d5d43ba0-82e9-43de-8883-
> 5ebaf07bf3e3
> |
> | min_disk | 0
>
> |
> | min_ram  | 0
>
> |
> | name | Ubuntu-16.04
>
> |
> | owner| 4a6213a64312482896130efc304719
> 5c
> |
> | properties   | direct_url='rbd://742cd163-
> 32ac-4121-8060-d440aa7345d6/images/d5d43ba0-82e9-43de-8883-5ebaf07bf3e3/snap'
> |
> | protected| False
>
> |
> | schema   | /v2/schemas/image
>
> |
> | size | 2361393152
>
> |
> | status   | active
>
> |
> | tags |
>
> |
> | updated_at   | 2016-12-07T16:01:45Z
>
> |
> | virtual_size | None
>
> |
> | visibility   | public
>
> |
> +--+
> --+
>
> On 02/03/17 15:56, Saverio Proto wrote:
>
> Can you share with us the output of:
>
> openstack image show 
>
> for that image ?
>
> Saverio
>
>
> 2017-03-02 13:54 GMT+01:00 Grant Morley :
>
>> Unfortunately not, I still get the same error.
>>
>> Grant
>>
>> On 02/03/17 12:54, Saverio Proto wrote:
>>
>> If you pass the uuid of the image does it work ?
>>
>> Saverio
>>
>> 2017-03-02 13:49 GMT+01:00 Grant Morley :
>>
>>> Hi Saverio,
>>>
>>> We are running Mitaka - sorry forgot to mention that.
>>>
>>> Grant
>>>
>>> On 02/03/17 12:45, Saverio Proto wrote:
>>>
>>> What version of Openstack are we talking about ?
>>>
>>> Saverio
>>>
>>> 2017-03-02 12:11 GMT+01:00 Grant Morley :
>>>
>>>> Hi All,
>>>>
>>>> Not sure if anyone can help, but as of today we are unable to launch
>>>> any instances and I have traced back the error to glance. Whenever I try
>>>> and launch an instance I get the below returned:
>>>>
>>>> openstack server create --flavor g1.small --image "Ubuntu-16.04"
>>>> --key-name ib-desktop --security-group default --nic
>>>> net-id=e2d925f0-63c1-442f-9158-6a138e5847ce gmtest
>>>> Could not find resource Ubuntu-16.04
>>>>
>>>> The image exists:
>>>>
>>>> openstack image list | grep Ubuntu-16.04
>>>>
>>>> | d5d43ba0-82e9-43de-8883-5ebaf07bf3e3 | Ubuntu-16.04   | active |
>>>>
>>>> The traceback gives a really odd message about Multiple choices ( All
>>>> images have unique names )
>>>>
>>>> Below is the stack trace - not sure if anyone has come across this
>>>> before? I am stumped at the moment.
>>>>
>>>> 2017-03-02 11:10:16.842 2107 INFO eventlet.wsgi.server [-]
>>>> 10.6.2.255,10.6.0.39 - - [02/Mar/2017 11:10:16] "GET
>>>> /images/detail?is_public=none&limit=20 HTTP/1.1" 300 787 0.000769
>>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>>> [req-3b498a7e-2d6a-4337-a314-12abfbac0117
>>>> 4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c - -
>>>> -] Registry client request GET /images/detail raised MultipleChoices
>>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>>> Traceback (most recent call last):
>>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>>> File "/openstack/venvs/glance-13.3.8/lib/pyth

Re: [Openstack-operators] Unable to launch instances because of glance errors

[Saverio Proto]

Can you share with us the output of:

openstack image show 

for that image ?

Saverio


2017-03-02 13:54 GMT+01:00 Grant Morley :

> Unfortunately not, I still get the same error.
>
> Grant
>
> On 02/03/17 12:54, Saverio Proto wrote:
>
> If you pass the uuid of the image does it work ?
>
> Saverio
>
> 2017-03-02 13:49 GMT+01:00 Grant Morley :
>
>> Hi Saverio,
>>
>> We are running Mitaka - sorry forgot to mention that.
>>
>> Grant
>>
>> On 02/03/17 12:45, Saverio Proto wrote:
>>
>> What version of Openstack are we talking about ?
>>
>> Saverio
>>
>> 2017-03-02 12:11 GMT+01:00 Grant Morley :
>>
>>> Hi All,
>>>
>>> Not sure if anyone can help, but as of today we are unable to launch any
>>> instances and I have traced back the error to glance. Whenever I try and
>>> launch an instance I get the below returned:
>>>
>>> openstack server create --flavor g1.small --image "Ubuntu-16.04"
>>> --key-name ib-desktop --security-group default --nic
>>> net-id=e2d925f0-63c1-442f-9158-6a138e5847ce gmtest
>>> Could not find resource Ubuntu-16.04
>>>
>>> The image exists:
>>>
>>> openstack image list | grep Ubuntu-16.04
>>>
>>> | d5d43ba0-82e9-43de-8883-5ebaf07bf3e3 | Ubuntu-16.04   | active |
>>>
>>> The traceback gives a really odd message about Multiple choices ( All
>>> images have unique names )
>>>
>>> Below is the stack trace - not sure if anyone has come across this
>>> before? I am stumped at the moment.
>>>
>>> 2017-03-02 11:10:16.842 2107 INFO eventlet.wsgi.server [-]
>>> 10.6.2.255,10.6.0.39 - - [02/Mar/2017 11:10:16] "GET
>>> /images/detail?is_public=none&limit=20 HTTP/1.1" 300 787 0.000769
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> [req-3b498a7e-2d6a-4337-a314-12abfbac0117 4c91f07132454a97b21fff35402b7825
>>> 4a6213a64312482896130efc3047195c - - -] Registry client request GET
>>> /images/detail raised MultipleChoices
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> Traceback (most recent call last):
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> File "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/
>>> glance/registry/client/v1/client.py", line 123, in do_request
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> **kwargs)
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> File 
>>> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
>>> line 70, in wrapped
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> return func(self, *args, **kwargs)
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> File 
>>> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
>>> line 373, in do_request
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> headers=copy.deepcopy(headers))
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> File 
>>> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
>>> line 87, in wrapped
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> return func(self, method, url, body, headers)
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> File 
>>> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
>>> line 534, in _do_request
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> raise exception.MultipleChoices(body=read_body(res))
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> MultipleChoices: The request returned a 302 Multiple Choices. This
>>> generally means that you have not included a version indicator in a request
>>> URI.
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client The
>>> body of response returned:
>>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>>> {"versions": [{"status": "CURRENT", "id": "v2.3", "links": [{"href&q

Re: [Openstack-operators] Unable to launch instances because of glance errors

[Saverio Proto]

If you pass the uuid of the image does it work ?

Saverio

2017-03-02 13:49 GMT+01:00 Grant Morley :

> Hi Saverio,
>
> We are running Mitaka - sorry forgot to mention that.
>
> Grant
>
> On 02/03/17 12:45, Saverio Proto wrote:
>
> What version of Openstack are we talking about ?
>
> Saverio
>
> 2017-03-02 12:11 GMT+01:00 Grant Morley :
>
>> Hi All,
>>
>> Not sure if anyone can help, but as of today we are unable to launch any
>> instances and I have traced back the error to glance. Whenever I try and
>> launch an instance I get the below returned:
>>
>> openstack server create --flavor g1.small --image "Ubuntu-16.04"
>> --key-name ib-desktop --security-group default --nic
>> net-id=e2d925f0-63c1-442f-9158-6a138e5847ce gmtest
>> Could not find resource Ubuntu-16.04
>>
>> The image exists:
>>
>> openstack image list | grep Ubuntu-16.04
>>
>> | d5d43ba0-82e9-43de-8883-5ebaf07bf3e3 | Ubuntu-16.04   | active |
>>
>> The traceback gives a really odd message about Multiple choices ( All
>> images have unique names )
>>
>> Below is the stack trace - not sure if anyone has come across this
>> before? I am stumped at the moment.
>>
>> 2017-03-02 11:10:16.842 2107 INFO eventlet.wsgi.server [-]
>> 10.6.2.255,10.6.0.39 - - [02/Mar/2017 11:10:16] "GET
>> /images/detail?is_public=none&limit=20 HTTP/1.1" 300 787 0.000769
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> [req-3b498a7e-2d6a-4337-a314-12abfbac0117 4c91f07132454a97b21fff35402b7825
>> 4a6213a64312482896130efc3047195c - - -] Registry client request GET
>> /images/detail raised MultipleChoices
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> Traceback (most recent call last):
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> File "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/
>> glance/registry/client/v1/client.py", line 123, in do_request
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> **kwargs)
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> File 
>> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
>> line 70, in wrapped
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> return func(self, *args, **kwargs)
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> File 
>> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
>> line 373, in do_request
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> headers=copy.deepcopy(headers))
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> File 
>> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
>> line 87, in wrapped
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> return func(self, method, url, body, headers)
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> File 
>> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
>> line 534, in _do_request
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> raise exception.MultipleChoices(body=read_body(res))
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> MultipleChoices: The request returned a 302 Multiple Choices. This
>> generally means that you have not included a version indicator in a request
>> URI.
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client The
>> body of response returned:
>> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
>> {"versions": [{"status": "CURRENT", "id": "v2.3", "links": [{"href":
>> "http://10.6.0.3:9191/v2/"; <http://10.6.0.3:9191/v2/>, "rel": "self"}]},
>> {"status": "SUPPORTED", "id": "v2.2", "links": [{"href":
>> "http://10.6.0.3:9191/v2/"; <http://10.6.0.3:9191/v2/>, "rel": "self"}]},
>> {"status": "SUPPORTED", "id": "v2.1", "links": [{"href":
>> "http://10.6.0.3:9191/v2/"; <http://10.6.0.3:9191/v2/>, "rel": "self"}]},
>>

Re: [Openstack-operators] Unable to launch instances because of glance errors

[Saverio Proto]

What version of Openstack are we talking about ?

Saverio

2017-03-02 12:11 GMT+01:00 Grant Morley :

> Hi All,
>
> Not sure if anyone can help, but as of today we are unable to launch any
> instances and I have traced back the error to glance. Whenever I try and
> launch an instance I get the below returned:
>
> openstack server create --flavor g1.small --image "Ubuntu-16.04"
> --key-name ib-desktop --security-group default --nic
> net-id=e2d925f0-63c1-442f-9158-6a138e5847ce gmtest
> Could not find resource Ubuntu-16.04
>
> The image exists:
>
> openstack image list | grep Ubuntu-16.04
>
> | d5d43ba0-82e9-43de-8883-5ebaf07bf3e3 | Ubuntu-16.04   | active |
>
> The traceback gives a really odd message about Multiple choices ( All
> images have unique names )
>
> Below is the stack trace - not sure if anyone has come across this before?
> I am stumped at the moment.
>
> 2017-03-02 11:10:16.842 2107 INFO eventlet.wsgi.server [-]
> 10.6.2.255,10.6.0.39 - - [02/Mar/2017 11:10:16] "GET
> /images/detail?is_public=none&limit=20 HTTP/1.1" 300 787 0.000769
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> [req-3b498a7e-2d6a-4337-a314-12abfbac0117 4c91f07132454a97b21fff35402b7825
> 4a6213a64312482896130efc3047195c - - -] Registry client request GET
> /images/detail raised MultipleChoices
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> Traceback (most recent call last):
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> File "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/
> glance/registry/client/v1/client.py", line 123, in do_request
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> **kwargs)
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> File 
> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
> line 70, in wrapped
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> return func(self, *args, **kwargs)
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> File 
> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
> line 373, in do_request
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> headers=copy.deepcopy(headers))
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> File 
> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
> line 87, in wrapped
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> return func(self, method, url, body, headers)
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> File 
> "/openstack/venvs/glance-13.3.8/lib/python2.7/site-packages/glance/common/client.py",
> line 534, in _do_request
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> raise exception.MultipleChoices(body=read_body(res))
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> MultipleChoices: The request returned a 302 Multiple Choices. This
> generally means that you have not included a version indicator in a request
> URI.
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client The
> body of response returned:
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> {"versions": [{"status": "CURRENT", "id": "v2.3", "links": [{"href":
> "http://10.6.0.3:9191/v2/"; , "rel": "self"}]},
> {"status": "SUPPORTED", "id": "v2.2", "links": [{"href":
> "http://10.6.0.3:9191/v2/"; , "rel": "self"}]},
> {"status": "SUPPORTED", "id": "v2.1", "links": [{"href":
> "http://10.6.0.3:9191/v2/"; , "rel": "self"}]},
> {"status": "SUPPORTED", "id": "v2.0", "links": [{"href":
> "http://10.6.0.3:9191/v2/"; , "rel": "self"}]},
> {"status": "SUPPORTED", "id": "v1.1", "links": [{"href":
> "http://10.6.0.3:9191/v1/"; , "rel": "self"}]},
> {"status": "SUPPORTED", "id": "v1.0", "links": [{"href":
> "http://10.6.0.3:9191/v1/"; , "rel": "self"}]}]}
> 2017-03-02 11:10:16.843 2104 ERROR glance.registry.client.v1.client
> 2017-03-02 11:10:16.844 2104 ERROR glance.common.wsgi
> [req-3b498a7e-2d6a-4337-a314-12abfbac0117 4c91f07132454a97b21fff35402b7825
> 4a6213a64312482896130efc3047195c - - -] Caught error: The request
> returned a 302 Multiple Choices. This generally means that you have not
> included a version indicator in a request URI.
>
> Any help would be great.
>
> Regards,
> --
> Grant Morley
> Cloud Lead
> Absolute DevOps Ltd
> Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
> www.absolutedevops.io gr...@absolutedevops.io  0845
> 874 0580
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.or

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

[Saverio Proto]

Hello,

I would use at least the stable/ocata branch. If you just use master
that is not supposed to be stable, and also I am not sure if you can
fill a bug against a specific commit in master.

Saverio

2017-02-21 21:12 GMT+01:00 Satyanarayana Patibandla
:
> Hi Saverio,
>
> We have tried to create 20 VMs each time using heat template. There is 1 sec
> time gap between each VM creation request. When we reached 114 VMs we got
> the error mentioned in the below mail.Heat template will boot instance from
> volume and it assigns floating IP to the instance.
>
> Except neutron-server container we restarted all the neutron agent
> containers which are present on all network and compute nodes. We are using
> kolla to deploy openstack services.
>
> We are using 1 month old master branch openstack code to deploy our
> services.
>
> Please find the error logs in the below link.
> http://paste.openstack.org/show/599892/
>
> Thanks,
> Satya.P
>
> On Wed, Feb 22, 2017 at 12:21 AM, Saverio Proto  wrote:
>>
>> Hello Satya,
>>
>> I would fill a bug on launchpad for this issue.
>> 114 VMs is not much. Can you identify how to trigger the issue to
>> reproduce it ? or it just happens randomly ?
>>
>> When you say rebooting the network node, do you mean the server
>> running the neutron-server process ?
>>
>> what version and distribution of openstack are you using ?
>>
>> thank you
>>
>> Saverio
>>
>>
>> 2017-02-21 13:54 GMT+01:00 Satyanarayana Patibandla
>> :
>> > Hi All,
>> >
>> > We are trying to deploy Openstack in our production environment. For
>> > networking we are using DVR with out L3 HA. We are able to create 114
>> > VMs
>> > with out any issue. After creating 114 VMs we are getting the below
>> > error.
>> >
>> > Error: 504 Gateway Time-out The server didn't
>> > respond
>> > in time. 
>> >
>> > Neutron services are getting freezed up due to a persistent lock on the
>> > agents table. it seems one of the network node is holding the lock on
>> > the
>> > table. After rebooting the network node, the Neutron CLI was responsive
>> > again.
>> >
>> > Neutron agent and neutron server is throwing below errors.
>> >
>> > Neutron-server errors:
>> > ERROR oslo_db.sqlalchemy.exc_filters "Can't reconnect until invalid
>> > "
>> > ERROR oslo_db.sqlalchemy.exc_filters InvalidRequestError: Can't
>> > reconnect
>> > until invalid transaction is rolled back
>> > ERROR neutron.api.v2.resource [req-24fa6eaa-a9e0-4f55-97e0-59db203e72c6
>> > 3eb776587c9c40569731ebe5c3557bc7 f43e8699cd5a46e89ffe39e3cac75341 - - -]
>> > index failed: No details.
>> > ERROR neutron.api.v2.resource DBError: Can't reconnect until invalid
>> > transaction is rolled back
>> >
>> >
>> > Neutron agents errors:
>> > MessagingTimeout: Timed out waiting for a reply to message ID
>> > 40638b6bf12c44cd9a404ecaa14a9909
>> >
>> > Could you please provide us your valuable inputs or suggestions for
>> > above
>> > errors.
>> >
>> > Thanks,
>> > Satya.P
>> >
>> > ___
>> > OpenStack-operators mailing list
>> > OpenStack-operators@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

[Saverio Proto]

Hello Satya,

I would fill a bug on launchpad for this issue.
114 VMs is not much. Can you identify how to trigger the issue to
reproduce it ? or it just happens randomly ?

When you say rebooting the network node, do you mean the server
running the neutron-server process ?

what version and distribution of openstack are you using ?

thank you

Saverio


2017-02-21 13:54 GMT+01:00 Satyanarayana Patibandla
:
> Hi All,
>
> We are trying to deploy Openstack in our production environment. For
> networking we are using DVR with out L3 HA. We are able to create 114 VMs
> with out any issue. After creating 114 VMs we are getting the below error.
>
> Error: 504 Gateway Time-out The server didn't respond
> in time. 
>
> Neutron services are getting freezed up due to a persistent lock on the
> agents table. it seems one of the network node is holding the lock on the
> table. After rebooting the network node, the Neutron CLI was responsive
> again.
>
> Neutron agent and neutron server is throwing below errors.
>
> Neutron-server errors:
> ERROR oslo_db.sqlalchemy.exc_filters "Can't reconnect until invalid "
> ERROR oslo_db.sqlalchemy.exc_filters InvalidRequestError: Can't reconnect
> until invalid transaction is rolled back
> ERROR neutron.api.v2.resource [req-24fa6eaa-a9e0-4f55-97e0-59db203e72c6
> 3eb776587c9c40569731ebe5c3557bc7 f43e8699cd5a46e89ffe39e3cac75341 - - -]
> index failed: No details.
> ERROR neutron.api.v2.resource DBError: Can't reconnect until invalid
> transaction is rolled back
>
>
> Neutron agents errors:
> MessagingTimeout: Timed out waiting for a reply to message ID
> 40638b6bf12c44cd9a404ecaa14a9909
>
> Could you please provide us your valuable inputs or suggestions for above
> errors.
>
> Thanks,
> Satya.P
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Instances are not creating after adding 3 additional nova nodes

[Saverio Proto]

Well,
I have no idea from this log file. Trying to make nova-compute more
verbose if you dont find anything in the logs

Saverio

2017-02-20 7:50 GMT+01:00 Anwar Durrani :
>
> On Thu, Feb 16, 2017 at 1:44 PM, Saverio Proto  wrote:
>>
>> openstack server show uuid
>
>
> Hi Saverio,
>
> I have investigated and progressed the case as per your saying, i got to
> know that instance was supposed to be launched on one of the nova node,
> where i dig and tried find out log as you mentioned, following output i have
> seen as below :
>
> tail -f /var/log/nova/nova-compute.log
>
> 2017-02-20 10:40:19.318 5365 WARNING nova.compute.manager
> [req-34fa4448-061d-44ad-b6e9-6ff0d1fd072f - - - - -] While synchronizing
> instance power states, found 4 instances in the database and 0 instances on
> the hypervisor.
>
> and
>
> other log where i have found following :
>
> tail -f /var/log/nova/nova-manage.log
>
> 2017-02-15 16:42:42.896 115003 TRACE nova   File
> "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 34, in
> sleep
>
> 2017-02-15 16:42:42.896 115003 TRACE nova hub.switch()
>
> 2017-02-15 16:42:42.896 115003 TRACE nova   File
> "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
>
> 2017-02-15 16:42:42.896 115003 TRACE nova return self.greenlet.switch()
>
> 2017-02-15 16:42:42.896 115003 TRACE nova   File
> "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
>
> 2017-02-15 16:42:42.896 115003 TRACE nova self.wait(sleep_time)
>
>
> Thanks
>
>
>
> --
> Thanks & regards,
> Anwar M. Durrani
> +91-9923205011
>
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Instances are not creating after adding 3 additional nova nodes

[Saverio Proto]

The three new compute nodes that you added are empty, so most likely
the new instances are scheduled to those three (3 attempts) and
something goes wrong.

with admin rights do:
openstack server show uuid

this should give you the info about the compute node where the
instance was scheduled. Check the nova-compute.log of that one, you
should find some debug info.

Saverio


2017-02-16 8:40 GMT+01:00 Anwar Durrani :
> Hi team,
>
> I am running Kilo setup with 1 Controller and 7 nova nodes, i have added 3
> additional nova nodes, where i can see, in system information under compute
> nodes that nodes are up and running, but when i am trying to launch instance
> then it is prompting below error :
>
> Error: Failed to perform requested operation on instance "test", the
> instance has an error status: Please try again later [Error: No valid host
> was found. Exceeded max scheduling attempts 3 for instance
> 969e71d4-845a-40da-be2a-d4f2619cbc68. Last exception: [u'Traceback (most
> recent call last):\n', u' File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2248, in
> _do].
>
>
> Thanks in advance.
>
>
> --
> Thanks & regards,
> Anwar M. Durrani
> +91-9923205011
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] about VISA to Italy for Ops Midcycle in Milano

[Saverio Proto]

Hello Ops,

the ops midcycle is in a about 1 month:
https://etherpad.openstack.org/p/MIL-ops-meetup

If you need a invitation letter for VISA to Italy please send an email
to egiannu...@enter.eu and Cc: ziopr...@gmail.com .

We will help you will the Italian forms to fill.
ENTER that is hosting the event will provide the invitation letter.

Thanks !

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Kolla in production (in near the future)

[Saverio Proto]

So far I am just learning. I wrote down some notes based on my
experience of this week.

https://github.com/zioproto/kolla-on-SWITCHengines/

I am looking forward to the Milano Ops Mid-cycle to chat face to face
with more operators interested in Kolla.

The key point is how to grow the Kolla inventory gradually to have a
migration path from the current infrastructure (puppet) to a full
Kolla deployment.

Have a good weekend.

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Neutron] Prefer local Neutron nodes?

[Saverio Proto]

Hello Adam,

this feature you describe makes sense only if the tenant VMs are also
aggregated in the same rack. Is this your scenario ?

Given that you could have 3 network nodes running the L3-agent, then
you can make some scripting to stick the specific router (linux
namespace) to the specific l3-agent.

To see an example of how to migrate a router to a specific l3-agent
please look at:
https://github.com/openstack/osops-tools-contrib/blob/master/neutron/l3-agent-evacuate.py

Saverio


2017-01-25 20:00 GMT+01:00 Adam Lawson :
> Hey everyone,
>
> If we want to implement multiple dedicated neutron routers in multiple racks
> (same cloud), is there a means of which I'm unaware that forces instances to
> route cross-tenant and public traffic to a neutron node within the local
> rack as opposed to load balancing L3 across all neutron nodes in all racks?
>
> Example:
> 6 racks,3 dedicated Neutron nodes per rack
>
> //adam
>
> Adam Lawson
>
> Principal Architect, CEO
> Office: +1-916-794-5706
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Newton consoleauth HA tokens

[Saverio Proto]

Did you try to restart memcached after chaning the configuration to HA ?

there are two sections where you can configure, memcached_servers
[DEFAULT]
 [keystone_authtoken]

how your config looks like ?

Saverio


2017-01-24 6:48 GMT+01:00 Chris Apsey :
> All,
>
> I attempted to deploy the nova service in HA, but when users attempt to
> connect via the console, it doesn't work about 30% of the time and they get
> the 1006 error.  The nova-consoleauth service is reporting their token as
> invalid.  I am running memcached, and have tried referencing it using both
> the legacy memcached_servers directive and in the new [cache] configuration
> section.  No dice.  If I disable the nova-consoleauth service on one of the
> nodes, everything works fine.  I see lots of bug reports floating around
> about this, but I can't quite get the solutions I have found reliably
> working.  I'm on Ubuntu 16.04 LTS+Newton from UCA.
>
> Ideas?
>
> --
> v/r
>
> Chris Apsey
> bitskr...@bitskrieg.net
> https://www.bitskrieg.net
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Mitaka to Newton Provider network issues

[Saverio Proto]

Hello,
in the upgrade did the version of ovs change ?

what Openstack distribution are you using ?

thanks

Saverio


2017-01-10 16:30 GMT+01:00 Telmo Morais :
>
> Hi All,
>
> We are currently on the process of upgrading from Mitaka to Newton, and on
> the upgraded compute nodes we lost all connectivity on provider networks.
>
> After digging through the ovs and ovs agent, we found that if we have only
> ONE provider network configured everything works as expected. But as soon as
> we add more providers networks in the config files we lose connectivity in
> them all. We also noticed that some flows in ovs are created, probably not
> all of them, as it doesn't work.
>
> Anyone has seen this behavior before?
>
> PS: no changes to the config were made during the upgrade.
>
> Thanks.
>
> Telmo Morais
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Question on coding style on osops-tools-contrib repo

[Saverio Proto]

Hello,

in the osops-tools-contrib repo so far I proposed always python
scripts that are contained in a single file.

Now I have this file:
openstackapi.py

that I reuse in many scripts, look at this:
https://review.openstack.org/#/c/401409/

but maybe is not the best idea to commit this generic file in the
neutron folder.

any advice how to handle this ? what is the accepted python import strategy ?

thanks

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Multiple floating IPs mapped to multiple vNICs (multi-homing)

[Saverio Proto]

Your policy routing looks good.
The problem must be somewhere else, where you do the nat maybe ?

Go in the network namespace where there is the neutron router with
address 10.0.16.1

If you tcpdump there what do you see ?

to be 100% sure about the policy routing just go in the network node
where you do the nat.

ip netns exec qrouter- wget -O /dev/null http://10.0.16.11/

uuid is the uuid of the neutron router where you are natting

I guess this will work.

Oh, did you double check the security groups ?

Saverio

2016-12-01 15:18 GMT+01:00 Paul Browne :
> Hello Saverio,
>
> Many thanks for the reply, I'll answer your queries below;
>
> On 01/12/16 12:49, Saverio Proto wrote:
>>
>> Hello,
>>
>> while the problem is in place, you should share the output of
>>
>> ip rule show
>> ip route show table 1
>>
>> It could be just a problem in your ruleset
>
>
> Of course, these are those outputs ;
>
> root@test1:~# ip rule show
> 0:  from all lookup local
> 32764:  from all to 10.0.16.11 lookup rt2
> 32765:  from 10.0.16.11 lookup rt2
> 32766:  from all lookup main
> 32767:  from all lookup default
>
> root@test1:~# ip route show table 1
> default via 10.0.16.1 dev eth1
> 10.0.16.0/24 dev eth1  scope link  src 10.0.16.11
>
>
>>
>> and, which one is your webserver ? can you tcpdump to make sure reply
>> packets get out on the NIC with src address 10.0.16.11 ?
>>
>> Saverio
>
>
> The instance has its two vNICs with source addresses 10.0.0.11 & 10.0.16.11,
> and the web-server is listening on both.
>
> The HTTP packets do seem to be getting out from 10.0.16.11 as source, but
> are stopped elsewhere upstream.
>
> I've attached two pcaps showing HTTP reply packets, one from 10.0.0.11
> (first vNIC; HTTP request and reply works to a remote client) and one from
> 10.0.16.11 (second vNIC; HTTP request is sent, reply not received by remote
> client). In the latter case, the server starts to make retransmissions to
> the remote client.
>
> Kind regards,
> Paul Browne
>
>
>
>
>>
>>
>> 2016-12-01 13:08 GMT+01:00 Paul Browne :
>>>
>>> Hello Operators,
>>>
>>> For reasons not yet amenable to persuasion otherwise, a customer of our
>>> ML2+OVS classic implemented OpenStack would like to map two floating IPs
>>> pulled from two separate external network floating IP pools, to two
>>> different vNICs on his instances.
>>>
>>> The floating IP pools correspond to one pool routable from the external
>>> Internet and another, RFC1918 pool routable from internal University
>>> networks.
>>>
>>> The tenant private networks are arranged as two RFC1918 VXLANs, each with
>>> a
>>> router to one of the two external networks.
>>>
>>> 10.0.0.0/24 -> route to -> 128.232.226.0/23
>>>
>>> 10.0.16.0/24 -> route to -> 172.24.46.0/23
>>>
>>>
>>> Mapping two floating IPs to instances isn't possible in Horizon, but is
>>> possible from command-line. This doesn't immediately work, however, as
>>> the
>>> return traffic from the instance needs to be sent back through the
>>> correct
>>> router gateway interface and not the instance default gateway.
>>>
>>> I'd initially thought this would be possible by placing a second routing
>>> table on the instances to handle the return traffic;
>>>
>>> debian@test1:/etc/iproute2$ less rt_tables
>>> #
>>> # reserved values
>>> #
>>> 255 local
>>> 254 main
>>> 253 default
>>> 0   unspec
>>> #
>>> # local
>>> #
>>> #1  inr.ruhep
>>> 1 rt2
>>>
>>> debian@test1:/etc/network$ less interfaces
>>> # The loopback network interface
>>> auto lo
>>> iface lo inet loopback
>>>
>>> # The first vNIC, eth0
>>> auto eth0
>>> iface eth0 inet dhcp
>>>
>>> # The second vNIC, eth1
>>> auto eth1
>>> iface eth1 inet static
>>>  address 10.0.16.11
>>>  netmask 255.255.255.0
>>>  post-up ip route add 10.0.16.0/24 dev eth1 src 10.0.16.11 table
>>> rt2
>>>  post-up ip route add default via 10.0.16.1 dev eth1 table rt2
>>>  post-up ip rule add from 10.0.16.11/32 table rt2
>>>  post-up ip rule add to 10.0.16.11/32 table rt2
>>>
>>> And this works well for SSH and ICMP, but curiously not for HTTP traffic.
>>>
>>

Re: [Openstack-operators] Multiple floating IPs mapped to multiple vNICs (multi-homing)

[Saverio Proto]

Hello,

while the problem is in place, you should share the output of

ip rule show
ip route show table 1

It could be just a problem in your ruleset

and, which one is your webserver ? can you tcpdump to make sure reply
packets get out on the NIC with src address 10.0.16.11 ?

Saverio


2016-12-01 13:08 GMT+01:00 Paul Browne :
> Hello Operators,
>
> For reasons not yet amenable to persuasion otherwise, a customer of our
> ML2+OVS classic implemented OpenStack would like to map two floating IPs
> pulled from two separate external network floating IP pools, to two
> different vNICs on his instances.
>
> The floating IP pools correspond to one pool routable from the external
> Internet and another, RFC1918 pool routable from internal University
> networks.
>
> The tenant private networks are arranged as two RFC1918 VXLANs, each with a
> router to one of the two external networks.
>
> 10.0.0.0/24 -> route to -> 128.232.226.0/23
>
> 10.0.16.0/24 -> route to -> 172.24.46.0/23
>
>
> Mapping two floating IPs to instances isn't possible in Horizon, but is
> possible from command-line. This doesn't immediately work, however, as the
> return traffic from the instance needs to be sent back through the correct
> router gateway interface and not the instance default gateway.
>
> I'd initially thought this would be possible by placing a second routing
> table on the instances to handle the return traffic;
>
> debian@test1:/etc/iproute2$ less rt_tables
> #
> # reserved values
> #
> 255 local
> 254 main
> 253 default
> 0   unspec
> #
> # local
> #
> #1  inr.ruhep
> 1 rt2
>
> debian@test1:/etc/network$ less interfaces
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> # The first vNIC, eth0
> auto eth0
> iface eth0 inet dhcp
>
> # The second vNIC, eth1
> auto eth1
> iface eth1 inet static
> address 10.0.16.11
> netmask 255.255.255.0
> post-up ip route add 10.0.16.0/24 dev eth1 src 10.0.16.11 table rt2
> post-up ip route add default via 10.0.16.1 dev eth1 table rt2
> post-up ip rule add from 10.0.16.11/32 table rt2
> post-up ip rule add to 10.0.16.11/32 table rt2
>
> And this works well for SSH and ICMP, but curiously not for HTTP traffic.
>
>
> Requests to a web-server listening on all vNICs are sent but replies not
> received when the requests are sent to the second mapped floating IP (HTTP
> requests and replies work as expected when sent to the first mapped floating
> IP). The requests are logged in both cases however, so traffic is making it
> to the instance in both cases.
>
> I'd say this is clearly an unusual (and possibly un-natural) arrangement,
> but I was wondering whether anyone else on Operators had come across a
> similar situation in trying to map floating IPs from two different external
> networks to an instance?
>
> Kind regards,
>
> Paul Browne
>
> --
> ***
> Paul Browne
> Research Computing Platforms
> University Information Services
> Roger Needham Building
> JJ Thompson Avenue
> University of Cambridge
> Cambridge
> United Kingdom
> E-Mail: pf...@cam.ac.uk
> Tel: 0044-1223-46548
> ***
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Snapshot residual data cleaning

[Saverio Proto]

Hello,

snapshot are meant to be backups.
I would not install software on a machine, and then use the snapshot
as a glance image to start many new instances.

you could just create machines with a clean glance image and install
the necessary software with ansible/puppet

if the installation time at boot is too long, you could create glance
images with the software you need with the diskimage-builder creating
a new dib.

Look at this example of a ubuntu glance image with zeppelin and spark
preinstalled:
https://github.com/switch-ch/diskimage-builder/tree/SWITCHengines/elements/zeppelin

Saverio


2016-11-22 7:44 GMT+01:00 Jack Gao :
> Hi everyone，
>
> I am using a virtual machine to create a snapshot, and then use this
> snapshot to create other virtual machines.Will encounter residual data
> didn't clean the problem, this will lead to network card information is
> inaccurate, puppet certificate conflict etc.
>
> The current practice is to use the sys-virtprep tools manual cleaning,
> within the opesntack system, whether to have a better solution?
>
> Regards.
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] unable to delete security group

[Saverio Proto]

Hello Suresh,
look at the log files !

Saverio

2016-11-09 17:54 GMT+01:00 suresh kumar :
> Hi All,
>
> I am unable to delete a security group in my kilo environment, this security
> group is not associated to any instances or LB
>
> nova secgroup-delete Test
>
> ERROR (BadRequest): Security Group 3s4gds-0304-5467-a63e-n5f94kdjg in use.
> (HTTP 400) (Request-ID: req-689effad-fd64-4178-86ea-50c0f96777f4)
>
>
> Thanks,
> Suresh
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] next Operators Mid-Cycle meetup, vote for the location on March 15-16th 2017

[Saverio Proto]

Hello Everyone,

we just had the Ops Meetup Team IRC meeting.
http://eavesdrop.openstack.org/meetings/ops_meetups_team/2016/ops_meetups_team.2016-11-08-14.03.html

There are two possible venues for the next OpenStack Operators
Mid-Cycle meetup: Milano and Tokyo.

To select the venue we created this poll (check log of the IRC meeting
for details about the decision making process)

http://doodle.com/poll/e7fcfhsf4s8cupyi

The poll is not binding :)

The Ops Meetups Team is hoping to see a good number of responses
within the next SEVEN DAYS.

Thanks for your attention.

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Meetups Team - Meeting Reminder

[Saverio Proto]

Hello Chris,

Thanks for clarify the meeting is Tuesday :)

I have a doubt about the time.
reading here:
https://wiki.openstack.org/wiki/Ops_Meetups_Team

Tuesdays at 14:00 UTC

On November the 8th:
It is 15:00 Zurich/Rome/Paris time
9:00 AM NYC time

because UTC never changes in summer and winter, so it used to be 10:00
in NYC and 16:00 in Europe but now it all shifted 1 hour earlier.

makes sense ? I dont know why you wrote 13:00 UTC, was it a mistake ?

thank you

Saverio



2016-11-04 19:25 GMT+01:00 Chris Morgan :
> Hello All,
>
>   When we met last week in Barcelona (some of us) I think there was
> consensus around picking up our normal Tuesday IRC meetings as of next
> Tuesday, which I see now is the 8th. Someone mentioned the 7th during that
> meeting, though, so I'd like to clarify. Are we all in agreement for Tuesday
> 8th? What time? It is normally 10AM my time which will be 13:00 UTC on the
> 8th.
>
> By the way, the Barcelona hosting offer has been updated with possible week
> day slots, see
> https://etherpad.openstack.org/p/ops-meetup-venue-discuss-spring-2017
>
> Regards
>
> Chris
>
> On Tue, Oct 4, 2016 at 10:45 AM, Saverio Proto  wrote:
>>
>> sorry I will not make it today
>> saverio
>>
>>
>> Il 04 ott 2016 4:42 PM, "Melvin Hillsman"  ha
>> scritto:
>>>
>>> #openstack-operators
>>>
>>> --
>>> Melvin Hillsman
>>> Ops Technical Lead
>>> OpenStack Innovation Center
>>> mrhills...@gmail.com <mailto:centermrhills...@gmail.com>
>>> phone: (210) 312-1267
>>> mobile: (210) 413-1659
>>> Learner | Ideation | Belief | Responsibility | Command
>>> http://osic.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 10/4/16, 9:39 AM, "Edgar Magana"  wrote:
>>>
>>> >Hello All,
>>> >
>>> >I am trying to join the meeting but I am not sure which IRC channel you
>>> > use.
>>> >
>>> >Edgar
>>> >
>>> >On 10/4/16, 5:30 AM, "Matt Van Winkle"  wrote:
>>> >
>>> >Hey Ops Meetup folks,
>>> >
>>> >Gentle reminder that the meeting is in about 1.5 hours at 14:00 UTC.
>>> > I won’t be able to make it because of an all-day planning meeting.
>>> > Hopefully, Tom will be around to help manage the chaos.  Edgar should be
>>> > joining to talk about the UC sessions and how we can help with those.  
>>> > Also,
>>> > we need to further refine the schedule.  I’ve made some notes in the
>>> > ehterpad [1]
>>> >
>>> >
>>> >
>>> >Thanks!
>>> >
>>> >VW
>>> >
>>> >
>>> >
>>> >[1]
>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__etherpad.openstack.org_p_ops-2Dmeetups-2Dteam&d=DQIGaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=5OOWM-nMlcl9Vc6QqIS3VJhGcj2VdyYOmTqhyMNIK8c&s=SYB5RddFgQ2KUjQ0-ZY3Ajq2uhuWMN6-Nq-ZK3N48Jg&e=
>>> >
>>> >
>>> >
>>> >___
>>> >OpenStack-operators mailing list
>>> >OpenStack-operators@lists.openstack.org
>>> >
>>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators&d=DQIGaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=5OOWM-nMlcl9Vc6QqIS3VJhGcj2VdyYOmTqhyMNIK8c&s=hu8MgvT0ZRfyCbRxSG-uzV10I5fLvMTfrP7YP8Cy6aA&e=
>>> >
>>> >
>>> >___
>>> >OpenStack-operators mailing list
>>> >OpenStack-operators@lists.openstack.org
>>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
>
> --
> Chris Morgan 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Nova snapshot using image-create not working

[Saverio Proto]

Hello,
if the instance is booted from Volume, when you "nova snapshot" in the
glance repo you find a image of 0 bytes, that just contains metadata.

you need to make cinder volume snapshots.

This is a recurring FAQ for the users of our cloud, I wrote something
about it here:
https://help.switch.ch/engines/documentation/backup-with-snapshots/

Saverio

2016-11-04 9:58 GMT+01:00 William Josefsson :
> Hi list, I have problems creating snapshots with nova image-create on
> Liberty/CentOS72. I have small ubuntu instances provisioned booted
> from a cinder volume, I stop the instance, then try to create a
> snapshot by issuing:
>
> As documented here:
> http://docs.openstack.org/user-guide/cli-use-snapshots-to-migrate-instances.html
>
> 'nova image-create --poll instance mySnapshot'
> This command is successful, and if I do glance image-list I can see
> the snapshot there.
>
> My storage backend is CEPH, and I can in neither of my pools, 'vms',
> 'volumes', or 'images' see this snapshot imaged stored. I also
> couldn't find the file in /var/lib/glance.
>
> ==> registry.log <==
> 2016-11-04 15:56:38.413 3525 INFO glance.registry.api.v1.images
> [req-b759199e-5180-4fcc-8ef4-a24faae81f3b
> 4919cce6c5d8455994447aa2bc9e6510 7308a926916848f98f2a6
>  28b36a091ad - - -] Successfully created image
> 6fd092d9-eab2-4176-94bc-d98bfb7a180f
>
> 2016-11-04 15:56:38.414 3525 INFO eventlet.wsgi.server
> [req-b759199e-5180-4fcc-8ef4-a24faae81f3b
> 4919cce6c5d8455994447aa2bc9e6510 7308a926916848f98f2a628b36a091
>   ad - - -] 127.0.0.1 - - [04/Nov/2016
> 15:56:38] "POST /images HTTP/1.1" 200 1184 0.117521
>
> If I try to dump the image using 'glance image-download --file
> snapshot.raw UUID', I get:
> NoneType object is not an interator
>
> Can anybody advice how to get snapshots working? thx will
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Neutron fails to notify nova on events network-vif-plugged

[Saverio Proto]

> I’m trying to use the puppet modules for liberty, I’ve had some issues there
> where the modules do not seem to exactly represent the configuration
> intended for liberty

Is this installation an upgrade from Kilo or it is a fresh Liberty
installation ?

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Tenant/Project naming restrictions

[Saverio Proto]

Is the '@' character allowed in the tenant/project names ?

Saverio

2016-10-05 23:36 GMT+02:00 Steve Martinelli :
> There are some restrictions.
>
> 1. The project name cannot be longer than 64 characters.
> 2. Within a domain, the project name is unique. So you can have project
> "foo" in the "default" domain, and in any other domain.
>
> On Wed, Oct 5, 2016 at 5:16 PM, Vigil, David Gabriel 
> wrote:
>>
>> What, if any, are the official tenant/project naming
>> requirements/restrictions? I can’t find any documentation that speaks to any
>> limitations. Is this documented somewhere?
>>
>>
>>
>>
>>
>>
>>
>> Dave G Vigil Sr
>>
>> Systems Integration Analyst Sr/SAIC Lead 09321
>>
>> Common Engineering Environment
>>
>> dgv...@sandia.gov
>>
>> 505-284-0157 (office)
>>
>> SAIC
>>
>>
>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Reserve an external network for 1 tenant

[Saverio Proto]

> Alternatively, you could drop the 'external' attribute and attach your
> instances directly to the provider network (no routers or private networks).

I can't. Because in my network design I do not have all the compute
nodes on a common L2 segment.
I have a l3 fabric between the compute nodes. So I cant just bridge
the provider network to the physical interface of any compute node.

I need the traffic to get to the network node, and there I can access
the provider network.

For a complete L2 setup I am investigating the L2-gw plugin

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [scientific][scientific-wg][Massively Distributed] - Workshop on Openstack-Federated Identity integration

[Saverio Proto]

Hello Adrien,

yes I am aware of the Massively Distributed Working Group. I will be
in Barcelona, and thanks for giving me input about WGs that could be
interested in the output of this workshop.
At the moment I thought of filling a User Story for the Openstack
Product Working group.

I hope to see your colleagues from Renater in Rome to start an early
discussion about this.
Mario Reale in Cc: is the organizer of the workshop, make sure he is
in the right loop of emails.

thanks

Saverio



2016-10-04 23:38 GMT+02:00  :
> Hi Saverio,
>
> I'm wondering whether you are aware of the massively distributed Working 
> Group [1].
> We are currently defining the agenda for our working sessions [2]. According 
> to your email (in particular the use-case you raised), the topic is 
> interesting and can be discussed during one of the scheduled sessions (if you 
> are available obviously ;))
>
> Actually, Renater (the French NREN) is taking part to the Beyond the clouds 
> initiative.
> This initiative targets the deployment of an OpenStack throughout several 
> points of presence of a network operator (basically one pop = one rack) [3]
> Several members of this initiative are strongly involved in this new WG.
>
> I just sent an  email to my colleagues from Renater to see whether they plan 
> to attend the meeting in Rome.
> I would be pleased to get informations and major results of your discussions.
>
> Will you attend the Barcelona summit ? if yes, do you think you can come and 
> present major results of your exchanges in one of the massively distributed 
> WG sessions.
>
> Thanks,
> Adrien
>
> [1] https://wiki.openstack.org/wiki/Massively_Distributed_Clouds
> [2] 
> https://etherpad.openstack.org/p/massively_distribute-barcelona_working_sessions
> [3] http://beyondtheclouds.github.io
>
> - Mail original -
>> De: "Saverio Proto" 
>> À: "stig openstack" 
>> Cc: "OpenStack Operators" 
>> Envoyé: Lundi 26 Septembre 2016 16:10:58
>> Objet: [Openstack-operators] [scientific][scientific-wg] - Workshop on 
>> Openstack-Federated Identity integration
>>
>> Hello operators,
>>
>> At GARR in Rome there will be an event shortly before Barcelona about
>> Openstack and Identity Federation.
>>
>> https://eventr.geant.org/events/2527
>>
>> This is a use case that is very important for NREN running public
>> cloud for Universities, where a Identity Federation is already
>> deployed for other services.
>>
>> At the meetup in Manchester when we talked about Federation we had an
>> interesting session:
>> https://etherpad.openstack.org/p/MAN-ops-Keystone-and-Federation
>>
>> I tagged the mail with scientific-wg because this looks like a
>> use-case that is of big interest of the academic institutions. Look
>> the etherpad of Manchester the section ''
>> Who do you want to federate with?''
>>
>> Thank you.
>>
>> Saverio
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Meetups Team - Meeting Reminder

[Saverio Proto]

sorry I will not make it today
saverio

Il 04 ott 2016 4:42 PM, "Melvin Hillsman"  ha scritto:

> #openstack-operators
>
> --
> Melvin Hillsman
> Ops Technical Lead
> OpenStack Innovation Center
> mrhills...@gmail.com 
> phone: (210) 312-1267
> mobile: (210) 413-1659
> Learner | Ideation | Belief | Responsibility | Command
> http://osic.org
>
>
>
>
>
>
>
>
> On 10/4/16, 9:39 AM, "Edgar Magana"  wrote:
>
> >Hello All,
> >
> >I am trying to join the meeting but I am not sure which IRC channel you
> use.
> >
> >Edgar
> >
> >On 10/4/16, 5:30 AM, "Matt Van Winkle"  wrote:
> >
> >Hey Ops Meetup folks,
> >
> >Gentle reminder that the meeting is in about 1.5 hours at 14:00 UTC.
> I won’t be able to make it because of an all-day planning meeting.
> Hopefully, Tom will be around to help manage the chaos.  Edgar should be
> joining to talk about the UC sessions and how we can help with those.
> Also, we need to further refine the schedule.  I’ve made some notes in the
> ehterpad [1]
> >
> >
> >
> >Thanks!
> >
> >VW
> >
> >
> >
> >[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__
> etherpad.openstack.org_p_ops-2Dmeetups-2Dteam&d=DQIGaQ&c=DS6PUFBBr_
> KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_
> wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=5OOWM-nMlcl9Vc6QqIS3VJhGcj2VdyYOmTqh
> yMNIK8c&s=SYB5RddFgQ2KUjQ0-ZY3Ajq2uhuWMN6-Nq-ZK3N48Jg&e=
> >
> >
> >
> >___
> >OpenStack-operators mailing list
> >OpenStack-operators@lists.openstack.org
> >https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
> openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators&d=DQIGaQ&c=
> DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_
> wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=5OOWM-nMlcl9Vc6QqIS3VJhGcj2VdyYOmTqh
> yMNIK8c&s=hu8MgvT0ZRfyCbRxSG-uzV10I5fLvMTfrP7YP8Cy6aA&e=
> >
> >
> >___
> >OpenStack-operators mailing list
> >OpenStack-operators@lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Reserve an external network for 1 tenant

[Saverio Proto]

Sorry I missed the Mailing List in the Cc:
Saverio

2016-10-03 9:15 GMT+02:00 Saverio Proto :
> Hello Kevin,
>
> thanks for your answer.
>
> so far I managed to make the network not shared just by making it not
> external. Because I dont need NAT and floatingips this will match my
> use case.
>
> As an admin I create the network like:
> openstack network create --no-share --project user_project_uuid
> --provider-physical-network physnet2 --provider-network-type flat
> NETWORKNAME
>
> In this way only the users that belong to user_project_uuid see the
> network with 'list' and 'show' operations.
>
> I still have to test carefully if Openstack will allow isolation to
> brake in case a user or admin tries to create more networks mapped to
> physnet2
>
> I hope I will upgrade to Mitaka as soon as possible.
>
> thank you
>
> Saverio
>
>
>
>
>
> 2016-10-03 7:00 GMT+02:00 Kevin Benton :
>> You will need mitaka to get an external network that is only available to
>> specific tenants. That is what the 'access_as_external' you identified does.
>>
>> Search for the section "Allowing a network to be used as an external
>> network" in
>> http://docs.openstack.org/mitaka/networking-guide/config-rbac.html.
>>
>> On Thu, Sep 29, 2016 at 5:01 AM, Saverio Proto  wrote:
>>>
>>> Hello,
>>>
>>> Context:
>>> - openstack liberty
>>> - ubuntu trusty
>>> - neutron networking with vxlan tunnels
>>>
>>> we have been running Openstack with a single external network so far.
>>>
>>> Now we have a specific VLAN in our datacenter with some hardware boxes
>>> that need a connection to a specific tenant network.
>>>
>>> To make this possible I changed the configuration of the network node
>>> to support multiple external networks. I am able to create a router
>>> and set as external network the new physnet where the boxes are.
>>>
>>> Everything looks nice except that all the projects can benefit from
>>> this new external network. In any tenant I can create a router, and
>>> set the external network and connect to the boxes. I cannot restrict
>>> it to a specific tenant.
>>>
>>> I found this piece of documentation:
>>>
>>>
>>> https://wiki.openstack.org/wiki/Neutron/sharing-model-for-external-networks
>>>
>>> So it looks like it is impossible to have a flat external network
>>> reserved for 1 specific tenant.
>>>
>>> I also tried to follow this documentation:
>>>
>>> http://docs.openstack.org/liberty/networking-guide/adv-config-network-rbac.html
>>>
>>> But it does not specify if it is possible to specify a policy for an
>>> external network to limit the sharing.
>>>
>>> It did not work for me so I guess this does not work when the secret
>>> network I want to create is external.
>>>
>>> There is an action --action access_as_external that is not clear to me.
>>>
>>> Also look like this feature is evolving in Newton:
>>> http://docs.openstack.org/draft/networking-guide/config-rbac.html
>>>
>>> Anyone has tried similar setups ? What is the minimum openstack
>>> version to get this done ?
>>>
>>> thank you
>>>
>>> Saverio
>>>
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Reserve an external network for 1 tenant

[Saverio Proto]

Hello Matt,

first of all in the file : plugins/ml2/openvswitch_agent.ini

you need to have bridge mappings, in my case for example:
bridge_mappings = physnet1:br-eth3,physnet2:br-eth4

this will define what physnet1 means in the openstack context. To
create the external network I do:

openstack network create --no-share --project uuid
--provider-physical-network physnet2 --provider-network-type flat
--external NETWORKNAME

Of course the --no-share is useless because being the network external
it will be shared by default.

Saverio


2016-10-03 6:15 GMT+02:00 Matt Kassawara :
> How are you creating the provider (external) network?
>
> On Thu, Sep 29, 2016 at 6:01 AM, Saverio Proto  wrote:
>>
>> Hello,
>>
>> Context:
>> - openstack liberty
>> - ubuntu trusty
>> - neutron networking with vxlan tunnels
>>
>> we have been running Openstack with a single external network so far.
>>
>> Now we have a specific VLAN in our datacenter with some hardware boxes
>> that need a connection to a specific tenant network.
>>
>> To make this possible I changed the configuration of the network node
>> to support multiple external networks. I am able to create a router
>> and set as external network the new physnet where the boxes are.
>>
>> Everything looks nice except that all the projects can benefit from
>> this new external network. In any tenant I can create a router, and
>> set the external network and connect to the boxes. I cannot restrict
>> it to a specific tenant.
>>
>> I found this piece of documentation:
>>
>>
>> https://wiki.openstack.org/wiki/Neutron/sharing-model-for-external-networks
>>
>> So it looks like it is impossible to have a flat external network
>> reserved for 1 specific tenant.
>>
>> I also tried to follow this documentation:
>>
>> http://docs.openstack.org/liberty/networking-guide/adv-config-network-rbac.html
>>
>> But it does not specify if it is possible to specify a policy for an
>> external network to limit the sharing.
>>
>> It did not work for me so I guess this does not work when the secret
>> network I want to create is external.
>>
>> There is an action --action access_as_external that is not clear to me.
>>
>> Also look like this feature is evolving in Newton:
>> http://docs.openstack.org/draft/networking-guide/config-rbac.html
>>
>> Anyone has tried similar setups ? What is the minimum openstack
>> version to get this done ?
>>
>> thank you
>>
>> Saverio
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Connecting 2 OpenStack clouds

[Saverio Proto]

Hello,

If your setup has a single user database, so all users are under the
same administrative domain, what you describe is like having different
Openstack Regions, or different Nova Cells.
I would suggest to look into Multi Region that is the easier to implement.
http://docs.openstack.org/arch-design/multi-site-architecture.html

If your setup has users in the two clouds that are managed under two
different administrative domains, then what you are talking about is
Cloud Federation.

We had a discussion about it in Machester:
https://etherpad.openstack.org/p/MAN-ops-Keystone-and-Federation

If your university is interested in Federation you should be aware of
this workshop:
https://eventr.geant.org/events/2527

Cheers,

Saverio



2016-09-30 8:06 GMT+02:00 Michael Stang :
> Hello all,
>
> I have a question if it is possible to connect 2 Openstack clouds to use the
> ressources together?
>
> We have a Mitaka installation at our site and we have colleagues who also
> have a mitaka installaton at their site which are both independent at the
> moment. We want to create a site-to-site vpn tunnel between our 2 management
> networks (with openvpn) so that both installations can see each other, and
> we searching now for a possibility to connect both together.
>
> Is there already some way to connect both controllers so the users of the
> one site can also use the resources of the other site and start instances
> there from their controller on the other controller?
>
> How is this done in  large installations when 2 clouds should be connected
> to each other? Is this even possible?
>
> Thank you and kind regards,
> Michael
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Logging and Monitoring NYC session. What warning do we ignore in Nagios ?

[Saverio Proto]

Hello,

In NYC we had this session:
https://etherpad.openstack.org/p/NYC-ops-Logging-and-monitoring

It came out that most of us configure Nagios to be less noisy, and
there are Warning strings that most people just ignore, because these
warnings are harmless and you dont want an email for each warning.

This is a lot of configuration effort grepping all these lines. Also
at each major upgrade the list of ignored warnings must be reviewed.

It would be so much better to report this upstream, so that useless
warnings can go to DEBUG or somewhere else.

I started a list:

https://etherpad.openstack.org/p/ops-ignored-warnings

Please +1 what your are also ignoring, or add more lines.

Thank you for the cooperation !

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Reserve an external network for 1 tenant

[Saverio Proto]

Hello,

Context:
- openstack liberty
- ubuntu trusty
- neutron networking with vxlan tunnels

we have been running Openstack with a single external network so far.

Now we have a specific VLAN in our datacenter with some hardware boxes
that need a connection to a specific tenant network.

To make this possible I changed the configuration of the network node
to support multiple external networks. I am able to create a router
and set as external network the new physnet where the boxes are.

Everything looks nice except that all the projects can benefit from
this new external network. In any tenant I can create a router, and
set the external network and connect to the boxes. I cannot restrict
it to a specific tenant.

I found this piece of documentation:

https://wiki.openstack.org/wiki/Neutron/sharing-model-for-external-networks

So it looks like it is impossible to have a flat external network
reserved for 1 specific tenant.

I also tried to follow this documentation:
http://docs.openstack.org/liberty/networking-guide/adv-config-network-rbac.html

But it does not specify if it is possible to specify a policy for an
external network to limit the sharing.

It did not work for me so I guess this does not work when the secret
network I want to create is external.

There is an action --action access_as_external that is not clear to me.

Also look like this feature is evolving in Newton:
http://docs.openstack.org/draft/networking-guide/config-rbac.html

Anyone has tried similar setups ? What is the minimum openstack
version to get this done ?

thank you

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific][scientific-wg] - Workshop on Openstack-Federated Identity integration

[Saverio Proto]

Hello operators,

At GARR in Rome there will be an event shortly before Barcelona about
Openstack and Identity Federation.

https://eventr.geant.org/events/2527

This is a use case that is very important for NREN running public
cloud for Universities, where a Identity Federation is already
deployed for other services.

At the meetup in Manchester when we talked about Federation we had an
interesting session:
https://etherpad.openstack.org/p/MAN-ops-Keystone-and-Federation

I tagged the mail with scientific-wg because this looks like a
use-case that is of big interest of the academic institutions. Look
the etherpad of Manchester the section ''
Who do you want to federate with?''

Thank you.

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators