[openstack-dev] [ironic] The scenary to rolling upgrade Ironic

[Tan, Lin]

Hi guys,

I am looking at https://bugs.launchpad.net/ironic/+bug/1502903 which is related 
to rolling upgrade and here is Jim's patch 
https://review.openstack.org/#/c/234450
I really have a concern or question about how to do Ironic doing rolling 
upgrades. It might be my mistake, but I would like to discuss here and get some 
feedback.

I manually did a rolling upgrade for a private OpenStack Cloud before. There 
are three main tasks for upgrade:
1. upgrade the code of service.
2. change configuration. 
3. the upgrade of DB Schema in DB, which is the most difficult and 
time-consuming part.

The current rolling upgrade solution or live upgrade are highly depends on 
upgrade different services in place one-by-one while make new service A can 
still communicate with old service B.
The ideal case is after we upgrade one of the services, others can still work 
without break.
This is can be done by using versionedobject and RPC version. For example, new 
Nova-API and new Nova-conductor can talk to old Nova-compute.
In the case of Nova services, it was suggests to follow below steps:
1. expand DB schema
2. pin RPC versions and object version at current
3. upgrade all nova-conductor servers because it will talk with DB
4. upgrade all nova services on controller nodes like nova-api
5. upgrade all nova-compute nodes
6. unpin RPC versions
7. shrink DB schema.
This is perfect for Nova. Because it has many nova-compute nodes, and few 
nova-conductor nodes and nova-api nodes. It's not necessary to upgrade 
nova-compute services at one time, which is time consuming.

For Ironic, we only have ir-conductor and ir-api. So the question is should we 
upgrade ir-conductor first or ir-api?
In my opinion, the ideal case is that we can have old ir-conductor and new 
ir-conductors coexist, which means we should upgrade ir-api to latest at first. 
But it's impossible at the moment, because ir-conductor will talk to DB 
directly and we only have one DB schema. That's a large difference between 
Ironic and Nova. We are missing a layer like nova-conductor.
The second case is upgrade ir-conductors first. That means if we upgrade the DB 
Schema, we have to upgrade all ir-conductors at once. During the upgrade, we 
could not provide Ironic service at all.

So I would suggest to stop all Ironic service, and upgrade ir-api first, and 
then upgrade ir-conductor one by one. Only enable the ir-conductor which has 
done the upgrade. Or upgrade ir-api and ir-conductors at once, although it 
sounds stupid a little bit.

What do you guys think?


Best Regards,

Tan


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [api][ironic] some questions about tags

[Tan, Lin]

Hi guys,

Ironic is implementing the tags stuff:
https://review.openstack.org/#/q/status:open+project:openstack/ironic+branch:master+topic:bp/nodes-tagging,n,z

And this work is follow the guidelines of API Workgroup:
http://specs.openstack.org/openstack/api-wg/guidelines/tags.html

But I have two doubts about the guideline:
1. Can we support partial update the Tag List using PATCH. I see there is an 
option to add/delete individual tag, but it is still using PUT. What's the 
disadvantage here for PATCH?

2. Can we update the tag as well? For example, in Gmail we can rename the label 
if necessary, which is much more friendly to me. But currently, this is not 
support from the guide. The only way to support this is cached the tags's 
entities and retag them in python client, I don't think it's a good way.
  
Best Regards,

Tan


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][glance][cinder][neutron]How to make use of x-openstack-request-id

[Tan, Lin]

Hi guys
I recently play around with 'x-openstack-request-id' header but have a 
dump question about how it works. At beginning, I thought an action across 
different services should use a same request-id but it looks like this is not 
the true. 

First I read the spec: 
https://blueprints.launchpad.net/nova/+spec/cross-service-request-id which said 
"This ID and the request ID of the other service will be logged at service 
boundaries". and I see cinder/neutron/glance will attach its context's 
request-id as the value of "x-openstack-request-id" header to its response 
while nova use X-Compute-Request-Id. This is easy to understand. So It looks 
like each service should generate its own request-id and attach to its 
response, that's all.

But then I see glance read 'X-Openstack-Request-ID' to generate the request-id 
while cinder/neutron/nova read 'openstack.request_id' when using with keystone. 
It is try to reuse the request-id from keystone.

This totally confused me. It would be great if you can correct me or point me 
some reference. Thanks a lot

Best Regards,

Tan


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Ironic] deadline for specs

[Tan, Lin]

Hi,

A quick question,
do we have a SpecProposalDeadline for Ironic, 18th Dec or ?

Thanks

Best Regards,

Tan


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic] deadline for specs

[Tan, Lin]

Hi Devananda,

It’s good news for me.
Many thanks for your quick response.

B.R
Tan

From: Devananda van der Veen [mailto:devananda@gmail.com]
Sent: Sunday, December 14, 2014 3:13 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ironic] deadline for specs


Hi, Tan,

No, ironic is not having an early feature proposal freeze for this cycle. Dec 
18 is the kilo-1 milestone, and that is all.

Please see the release schedule here:

https://wiki.openstack.org/wiki/Kilo_Release_Schedule

That being said, the earlier you can propose a spec, the better your chances 
for it landing in any given cycle.

Regards,
Devananda





On Sat, Dec 13, 2014, 10:10 PM Tan, Lin 
mailto:lin@intel.com>> wrote:

Hi,

A quick question,
do we have a SpecProposalDeadline for Ironic, 18th Dec or ?

Thanks

Best Regards,

Tan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project]How to make use of x-openstack-request-id

[Tan, Lin]

Thanks Kebane, I test glance/neutron/keystone with ``x-openstack-request-id`` 
and find something interesting.

I am able to pass ``x-openstack-request-id``  to glance and it will use the 
UUID as its request-id. But it failed with neutron and keystone.
Here is my test:
http://paste.openstack.org/show/484644/

It looks like because keystone and neutron are using 
oslo_middleware:RequestId.factory and in this part:
https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/request_id.py#L35
It will always generate an UUID and append to response as 
``x-openstack-request-id`` header.

My question is should we accept an external passed request-id as the project's 
own request-id or having its unique request-id?
In other words, which one is correct way, glance or neutron/keystone? There 
must be something wrong with one of them.

Thanks

B.R

Tan

From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com]
Sent: Wednesday, December 2, 2015 2:24 PM
To: OpenStack Development Mailing List (openstack-dev@lists.openstack.org)
Subject: Re: [openstack-dev] [nova][glance][cinder][neutron]How to make use of 
x-openstack-request-id


Hi Tan,



Most of the OpenStack RESTful API returns `X-Openstack-Request-Id` in the API 
response header but this request id is not available to the caller from the 
python client.

When you use --debug option from command from the command prompt using client, 
you can see `X-Openstack-Request-Id` on the console but it is not logged 
anywhere.



Currently a cross-project specs [1] is submitted and approved for returning 
X-Openstack-Request-Id to the caller and the implementation for the same is in 
progress.

Please go through the specs for detail information which will help you to 
understand more about request-ids and current work about the same.



Please feel free to revert back anytime for your doubts.



[1] 
https://github.com/openstack/openstack-specs/blob/master/specs/return-request-id.rst



Thanks,



Abhishek Kekane









Hi guys

I recently play around with 'x-openstack-request-id' header but have a 
dump question about how it works. At beginning, I thought an action across 
different services should use a same request-id but it looks like this is not 
the true.



First I read the spec: 
https://blueprints.launchpad.net/nova/+spec/cross-service-request-id which said 
"This ID and the request ID of the other service will be logged at service 
boundaries". and I see cinder/neutron/glance will attach its context's 
request-id as the value of "x-openstack-request-id" header to its response 
while nova use X-Compute-Request-Id. This is easy to understand. So It looks 
like each service should generate its own request-id and attach to its 
response, that's all.



But then I see glance read 'X-Openstack-Request-ID' to generate the request-id 
while cinder/neutron/nova read 'openstack.request_id' when using with keystone. 
It is try to reuse the request-id from keystone.



This totally confused me. It would be great if you can correct me or point me 
some reference. Thanks a lot



Best Regards,



Tan


__
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][glance][cinder][neutron]How to make use of x-openstack-request-id

[Tan, Lin]

Thanks Kebane, I test glance/neutron/keystone with ``x-openstack-request-id`` 
and find something interesting.

I am able to pass ``x-openstack-request-id``  to glance and it will use the 
UUID as its request-id. But it failed with neutron and keystone.
Here is my test:
http://paste.openstack.org/show/484644/

It looks like because keystone and neutron are using 
oslo_middleware:RequestId.factory and in this part:
https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/request_id.py#L35
It will always generate an UUID and append to response as 
``x-openstack-request-id`` header.

My question is should we accept an external passed request-id as the project's 
own request-id or having its unique request-id?
In other words, which one is correct way, glance or neutron/keystone? There 
must be something wrong with one of them.

Thanks

B.R

Tan


From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com]
Sent: Wednesday, December 2, 2015 2:24 PM
To: OpenStack Development Mailing List (openstack-dev@lists.openstack.org)
Subject: Re: [openstack-dev] [nova][glance][cinder][neutron]How to make use of 
x-openstack-request-id


Hi Tan,



Most of the OpenStack RESTful API returns `X-Openstack-Request-Id` in the API 
response header but this request id is not available to the caller from the 
python client.

When you use --debug option from command from the command prompt using client, 
you can see `X-Openstack-Request-Id` on the console but it is not logged 
anywhere.



Currently a cross-project specs [1] is submitted and approved for returning 
X-Openstack-Request-Id to the caller and the implementation for the same is in 
progress.

Please go through the specs for detail information which will help you to 
understand more about request-ids and current work about the same.



Please feel free to revert back anytime for your doubts.



[1] 
https://github.com/openstack/openstack-specs/blob/master/specs/return-request-id.rst



Thanks,



Abhishek Kekane









Hi guys

I recently play around with 'x-openstack-request-id' header but have a 
dump question about how it works. At beginning, I thought an action across 
different services should use a same request-id but it looks like this is not 
the true.



First I read the spec: 
https://blueprints.launchpad.net/nova/+spec/cross-service-request-id which said 
"This ID and the request ID of the other service will be logged at service 
boundaries". and I see cinder/neutron/glance will attach its context's 
request-id as the value of "x-openstack-request-id" header to its response 
while nova use X-Compute-Request-Id. This is easy to understand. So It looks 
like each service should generate its own request-id and attach to its 
response, that's all.



But then I see glance read 'X-Openstack-Request-ID' to generate the request-id 
while cinder/neutron/nova read 'openstack.request_id' when using with keystone. 
It is try to reuse the request-id from keystone.



This totally confused me. It would be great if you can correct me or point me 
some reference. Thanks a lot



Best Regards,



Tan


__
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][glance][cinder][neutron]How to make use of x-openstack-request-id

[Tan, Lin]

Thank you so much. Eron. This really helps me a lot!!

Tan

From: Kuvaja, Erno [mailto:kuv...@hpe.com]
Sent: Tuesday, January 26, 2016 8:34 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova][glance][cinder][neutron]How to make use of 
x-openstack-request-id

Hi Tan,

While the cross project spec was discussed Glance already had implementation of 
request ids in place. At the time of the Glance implementation we assumed that 
one request id is desired through the chain of services and we implemented the 
req id to be accepted as part of the request. This was mainly driven to have 
same request id through the chain between glance-api and glance-registry but as 
the same code was used in both api and registry services we got this 
functionality across glance.

The cross project discussion turned this approach down and decided that only 
new req id will be returned. We did not want to utilize 2 different code bases 
to handle req ids in glance-api and glance-registry, nor we wanted to remove 
the functionality to allow the req ids being passed to the service as that was 
already merged to our API. Thus is requests are passed without req id defined 
to the services they behave (apart from nova having different header name) same 
way, but with glance the request maker has the liberty to specify request id 
they want to use (within configured length limits).

Hopefully that clarifies it for you.


-  Erno

From: Tan, Lin [mailto:lin@intel.com]
Sent: 26 January 2016 01:26
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova][glance][cinder][neutron]How to make use of 
x-openstack-request-id

Thanks Kebane, I test glance/neutron/keystone with ``x-openstack-request-id`` 
and find something interesting.

I am able to pass ``x-openstack-request-id``  to glance and it will use the 
UUID as its request-id. But it failed with neutron and keystone.
Here is my test:
http://paste.openstack.org/show/484644/

It looks like because keystone and neutron are using 
oslo_middleware:RequestId.factory and in this part:
https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/request_id.py#L35
It will always generate an UUID and append to response as 
``x-openstack-request-id`` header.

My question is should we accept an external passed request-id as the project's 
own request-id or having its unique request-id?
In other words, which one is correct way, glance or neutron/keystone? There 
must be something wrong with one of them.

Thanks

B.R

Tan


From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com]
Sent: Wednesday, December 2, 2015 2:24 PM
To: OpenStack Development Mailing List 
(openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>)
Subject: Re: [openstack-dev] [nova][glance][cinder][neutron]How to make use of 
x-openstack-request-id


Hi Tan,



Most of the OpenStack RESTful API returns `X-Openstack-Request-Id` in the API 
response header but this request id is not available to the caller from the 
python client.

When you use --debug option from command from the command prompt using client, 
you can see `X-Openstack-Request-Id` on the console but it is not logged 
anywhere.



Currently a cross-project specs [1] is submitted and approved for returning 
X-Openstack-Request-Id to the caller and the implementation for the same is in 
progress.

Please go through the specs for detail information which will help you to 
understand more about request-ids and current work about the same.



Please feel free to revert back anytime for your doubts.



[1] 
https://github.com/openstack/openstack-specs/blob/master/specs/return-request-id.rst



Thanks,



Abhishek Kekane









Hi guys

I recently play around with 'x-openstack-request-id' header but have a 
dump question about how it works. At beginning, I thought an action across 
different services should use a same request-id but it looks like this is not 
the true.



First I read the spec: 
https://blueprints.launchpad.net/nova/+spec/cross-service-request-id which said 
"This ID and the request ID of the other service will be logged at service 
boundaries". and I see cinder/neutron/glance will attach its context's 
request-id as the value of "x-openstack-request-id" header to its response 
while nova use X-Compute-Request-Id. This is easy to understand. So It looks 
like each service should generate its own request-id and attach to its 
response, that's all.



But then I see glance read 'X-Openstack-Request-ID' to generate the request-id 
while cinder/neutron/nova read 'openstack.request_id' when using with keystone. 
It is try to reuse the request-id from keystone.



This totally confused me. It would be great if you can correct me or point me 
some reference. Thanks a lot



Best Regards,



Tan


__

Re: [openstack-dev] [Ironic] Stepping down from IPA core

[Tan, Lin]

It’s great to work with you, Josh. Thanks for your valuable comments and 
suggestions.
Wish you luck in your new Job.

Tan
From: Josh Gachnang [mailto:j...@pcsforeducation.com]
Sent: Monday, September 21, 2015 11:49 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [Ironic] Stepping down from IPA core

Hey y'all, it's with a heavy heart I have to announce I'll be stepping down 
from the IPA core team on Thurs, 9/24. I'm leaving Rackspace for a healthcare 
startup (Triggr Health) and won't have the time to dedicate to being an 
effective OpenStack reviewer.

Ever since the OnMetal team proposed IPA all the way back in the Icehouse 
midcycle, this community has been welcoming, helpful, and all around great. 
You've all helped me grow as a developer with your in depth and patient 
reviews, for which I am eternally grateful. I'm really sad I won't get to see 
everyone in Tokyo.

I'll still be on IRC after leaving, so feel free to ping me for any reason :)

- JoshNang
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] no more expand/contract for live upgrade?

[Tan, Lin]

Hi,

I noticed that expand/migrate/contract was revert in 
https://review.openstack.org/#/c/239922/
There is a new CMD 'online_data_migrations' was introduced to Nova and some 
data-migration scripts have been added.
So I wonder will Nova keep expand the DB schema at beginning of live upgrade 
like before Or Nova have some new ways to handle DB Schema change?
The upgrade doc was not update for a long time 
http://docs.openstack.org/developer/nova/upgrade.html

Thanks a lot.

Best Regards,

Tan



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Nominating Julia Kreger for core reviewer

[Tan, Lin]

+1! Always grateful for the invaluable comments from Julia

From: Vladyslav Drok [mailto:vd...@mirantis.com]
Sent: Friday, March 25, 2016 5:16 PM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [ironic] Nominating Julia Kreger for core reviewer

+1!

On Fri, Mar 25, 2016 at 4:03 AM, Zhenguo Niu 
mailto:niu.zgli...@gmail.com>> wrote:
+1, thanks for all the hard work Julia!

On Fri, Mar 25, 2016 at 8:39 AM, 守屋哲 / MORIYA，SATORU 
mailto:satoru.moriya...@hitachi.com>> wrote:
+1 for me. She has given lots of valuable reviews to boot from volume effort.
Thanks Julia!

Satoru

> -Original Message-
> From: Jim Rollenhagen 
> [mailto:j...@jimrollenhagen.com]
> Sent: Friday, March 25, 2016 4:09 AM
> To: 
> openstack-dev@lists.openstack.org
> Subject: [!][openstack-dev] [ironic] Nominating Julia Kreger for core reviewer
>
> Hey all,
>
> I'm nominating Julia Kreger (TheJulia in IRC) for ironic-core. She runs
> the Bifrost project, gives super valuable reviews, is beginning to lead
> the boot from volume efforts, and is clearly an expert in this space.
>
> All in favor say +1 :)
>
> // jim
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Best Regards,
Zhenguo Niu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] upgrade support between which versions of ironic?

[Tan, Lin]

I agree this is reasonable to support all these cases in “cold upgrades” but in 
supports-rolling-upgrade (live upgrade in another word) case it is different 
and complicated and not necessary,

During rolling upgrade, we will have old/new services co-existed, and we need 
to make services compatible which need some extra code work and this is the 
main purpose of spec [1]. And as far as I can see, we are  not allowed to skip 
over releases when rolling upgrading.  So my point is support name release is 
enough.

1. Because even if we want to support major number release, admins have to 
upgrade from 5.0 -> 6.0 then 6.0 -> 7.0 in Ruby’s case of 5.0.0, 5.1.0 == 
Mitaka, 5.2.0, 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton. And we might have a 
higher release frequency in the future. So it’s too much work for upgrade a 
service every six months.

2. As we usually rolling upgrade the whole cloud, not for ironic only. For 
example, other projects will upgrade from Mitaka to Netwon, there is not much 
sense to upgrade Ironic from 5.0 -> 6.0 only.

At last, we should add a multi-node grenade CI to test the rolling upgrade 
mechanism was not broken. Here is my suggestion, we will have two nodes, Node A 
and Node B. Node A will run both of last named release of ironic-api and 
ironic-conductor and node B will run last named release ironic-conductor only. 
Multi-node grenade CI will only upgrade Node A and then we can test the 
interaction between new SHA of ironic-api and new SHA/last named release of 
ironic-conductor still works.  This should also apply to stable branch.


B.R

Tan

[1] https://review.openstack.org/299245

From: Devananda van der Veen [mailto:devananda@gmail.com]
Sent: Wednesday, April 20, 2016 5:12 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [ironic] upgrade support between which versions of 
ironic?

Thanks for starting the thread, Ruby.


We need to first establish a grenade job to test "cold upgrades" and assert the 
supports-upgrade tag. I believe Ironic meets all the criteria for that tag 
except:
- having a job that tests it (so, you know, it might be broken and I might be 
wrong)
- having operator documentation describing the process (it should be here: 
http://docs.openstack.org/developer/ironic/deploy/upgrade-guide.html ) but all 
we have are release-specific upgrade notes.

I think all of the scenario you outline are valid upgrade paths for an 
operator, and we should try to allow all of them to work. However, some of them 
can be covered by one test case, and you also missed some things I think need 
to be covered. Also, I'm interpreting the word "master" in your scenarios to 
indicate the proposed change to our master branch, since we do pre-merge 
testing

So, here are the test cases I think we need to cover:

=== run on proposed changes to master ===

F. current master to new SHA

We need to ensure that we can upgrade master to the code being proposed. You 
listed this last, but I think it's actually the most important one.

D. last named release to new SHA
E. last numbered release to new SHA

Because we cut new releases from master, this is the basis of testing the 
upgrade between sequential (named or numbered) releases before we cut a new 
(named or numbered) release, and is our most important test to ensure that we 
don't break most operators. Based on the user survey, most operators are using 
named releases, so if we are resource constrained, I would prefer to cover (D) 
before (E)

=== run on proposed changes to a stable branch ===

A. stable/N-1 -> new SHA -> [ stable/N+1 or current master]

We don't need to test upgrades between two named releases (eg. Liberty -> 
Mitaka) every time we land a new patch on the master branch, but we do need to 
test any time we land a change on a stable branch. Changes to the most recent 
stable branch should be upgrade-tested to current master, whereas changes to 
any stable branch prior to that should get tested to the subsequent sequential 
release.

Eg, a backport to stable/liberty should trigger an upgrade test for both 
(stable/kilo -> newpatch) and (newpatch  -> stable/mitaka), whereas a backport 
to stable/mitaka should trigger a test for (stable/liberty -> newpatch) and 
(newpatch -> master)


Once we've done that, then yes, I agree we should also work towards asserting 
supports-rolling-upgrade. That will require a partial upgrade job, eg. where we 
run >1 instance of the API and Conductor services and upgrade some of them.

Regards,
Devananda



On Tue, Apr 19, 2016 at 12:52 PM, Ruby Loo 
mailto:opensr...@gmail.com>> wrote:
Hi,
Currently, ironic doesn't support ("live", "online", "rolling", or 
minimal-downtime) upgrades between named versions of ironic. (Where "named 
version" is the final release or stable release that is associated with a 
development cycle). So for example, Liberty -> Mitaka release.
We've been working towards that, and have a spec [1] and a design 

Re: [openstack-dev] [Ironic] Thinking about our python client UX

[Tan, Lin]

I agree, we need a discussion asap. 
I think below things can mitigate the unexpected behavior of python client and 
this is borrow from object model.
1. Python client should have its own version and attributes like supported api 
version.
2. It should be able to recognize the server's version and show corresponding 
fields instead of all. 

B.R

Tan
-Original Message-
From: Devananda van der Veen [mailto:devananda@gmail.com] 
Sent: Sunday, March 8, 2015 8:12 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Ironic] Thinking about our python client UX

Hi folks,

Recently, I've been thinking more of how users of our python client will 
interact with the service, and in particular, how they might expect different 
instances of Ironic to behave.

We added several extensions to the API this cycle, and along with that, also 
landed microversion support (I'll say more on that in another thread). However, 
I don't feel like we've collectively given nearly enough thought to the python 
client. It seems to work well enough for our CI testing, but is that really 
enough? What about user experience?

In my own testing of the client versioning patch that landed on Friday, I 
noticed some pretty appalling errors (some unrelated to that
patch) when pointing the current client at a server running the stable/juno 
code...

http://paste.openstack.org/show/u91DtCf0fwRyv0auQWpx/


I haven't filed specific bugs from yet this because I think the issue is large 
enough that we should talk about a plan first. I think that starts by agreeing 
on who the intended audience is and what level of forward-and-backward 
compatibility we are going to commit to [*], documenting that agreement, and 
then come up with a plan to deliver that during the L cycle. I'd like to start 
the discussion now, so I have put it on the agenda for Monday, but I also 
expect it will be a topic at the Vancouver summit.

-Devananda


[*] full disclosure

I believe we have to commit to building a client that works well with every 
release since Icehouse, and the changes we've introduced in the client in this 
cycle do not.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic] proposing rameshg87 to ironic-core

[Tan, Lin]

Congratulations Ramesh

B.R

Tan

From: Devananda van der Veen [mailto:devananda@gmail.com]
Sent: Friday, March 13, 2015 12:06 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Ironic] proposing rameshg87 to ironic-core

Without further ado, and since everyone (even though some haven't replied here) 
has +1'd this, and since we could really use ramesh's +2's in the run up to 
Kilo-3 and feature freeze, even without the customary waiting/voting period 
being completely satisfied (after all, when we all agree, why wait a week?), 
I'd like to officially welcome him to the core team.

-Devananda

On Tue, Mar 10, 2015 at 10:03 AM David Shrewsbury 
mailto:shrewsbury.d...@gmail.com>> wrote:
+1

On Mar 9, 2015, at 6:03 PM, Devananda van der Veen 
mailto:devananda@gmail.com>> wrote:

Hi all,

I'd like to propose adding Ramakrishnan (rameshg87) to ironic-core.

He's been consistently providing good code reviews, and been in the top five 
active reviewers for the last 90 days and top 10 for the last 180 days. Two 
cores have recently approached me to let me know that they, too, find his 
reviews valuable.

Furthermore, Ramakrishnan has made significant code contributions to Ironic 
over the last year. While working primarily on the iLO driver, he has also done 
a lot of refactoring of the core code, touched on several other drivers, and 
maintains the proliantutils library on stackforge. All in all, I feel this 
demonstrates a good and growing knowledge of the codebase and architecture of 
our project, and feel he'd be a valuable member of the core team.

Stats, for those that want them, are below the break.

Best Regards,
Devananda



http://stackalytics.com/?release=all&module=ironic-group&user_id=rameshg87

http://russellbryant.net/openstack-stats/ironic-reviewers-90.txt
http://russellbryant.net/openstack-stats/ironic-reviewers-180.txt
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison

[Tan, Lin]

Thank you for your effort and wish you luck, Ghe

B.R

Tan

From: Ghe Rivero [mailto:g...@debian.org]
Sent: Tuesday, May 26, 2015 12:46 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison

My focus on the Ironic project has been decreasing in the last cycles, so it's 
about time to relinquish my position as a oslo-ironic liaison so new 
contributors can take over it and help ironic to be the vibrant project it is.

So long, and thanks for all the fish,

Ghe Rivero
--
Pinky: "Gee, Brain, what do you want to do tonight?"
The Brain: "The same thing we do every night, Pinky—try to take over the world!"

 .''`.  Pienso, Luego Incordio
: :' :
`. `'
  `-www.debian.org
www.openstack.com

GPG Key: 26F020F7
GPG fingerprint: 4986 39DA D152 050B 4699  9A71 66DB 5A36 26F0 20F7
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison

[Tan, Lin]

Hi Doug and guys,

I would like to work as oslo-ironic liasison to sync Ironic with Oslo.
I will attend the regular Oslo meeting for sure. My IRC name is lintan, and 
Launchpad id is tan-lin-good

Thanks

Tan

-Original Message-
From: Doug Hellmann [mailto:d...@doughellmann.com] 
Sent: Tuesday, May 26, 2015 9:17 PM
To: openstack-dev
Subject: Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic 
liaison

Excerpts from Ghe Rivero's message of 2015-05-25 09:45:47 -0700:
> My focus on the Ironic project has been decreasing in the last cycles, 
> so it's about time to relinquish my position as a oslo-ironic liaison 
> so new contributors can take over it and help ironic to be the vibrant 
> project it is.
> 
> So long, and thanks for all the fish,
> 
> Ghe Rivero

Thanks for your help as liaison, Ghe, the Oslo team appreciates your effort!

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] [neutron] Replacement of SecurityGroup object model API

[Tan, Lin]

Hi all,

I have struggled with SecurityGroup and SecurityGroupRule object model for a 
while. And I have some questions about how to handle with SecurityGroup from 
Neutron.
Here is the work flow for my solution now:
[cid:image001.png@01CF00E7.12B37570]
Before Object Model, SecurityGroup is a dict and contains SecurityGroupRule 
like this:
{'id': 1, 'ip_protocol': 'tcp', 'from_port': 22, 'to_port': 22, rules: []}
With Object Model, SecurityGroup and SecurityGroupRule will be two individual 
identities. Only SecurityGroupRule knows who its parent_group is.  But 
sometimes we only have SecurityGroup and need to know its rules. So we need to 
search in dB to find its rules. This works fine with nova SecurityGroup. But it 
doesn't work with Neutron SecurityGroup, because they are not in Nova dB.
Right now, I pass neutron securityGroup to EC2 as dict, and EC2 will behavior 
varies depends on the SecurityGroup they received.

So the problem is do we have to convert the neutron group into object model as 
well? If so, then where to store the group and rule information, like nova dB? 
(I tried to save them in API request or store in neutron_driver, but they can't 
work) Otherwise we need to modify the design of Security Group Object, like 
combine them into one object?

You can see more details here: https://review.openstack.org/#/c/59655/


Best Regards,

Tan

<>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Infrastructure ]Support for PCI Passthrough

[Tan, Lin]

Hi,



We add the new feature of PCI passthrough in Nova enables assigning a PCI 
device to an instance in openstack.

Now we want to add integration tests in Temptest to cover our module. We come 
up with several ideas like fade device or QEMU IOMMU emulator, but they are not 
possible. We talk to people in Nova and they suggest us to contact you to find 
a better solution.

Our module only works on the compute node that enables VT-d and contains 
special PCIs which support the SR-IOV.

So is it possible to

1. setup compute node which enables pci passthrough.

2. modify the testing schedule logic allow the pci testing case be scheduled to 
that machine



It would also be great if you have some other suggestions.

Thanks a lot!



Tan



Here is the wiki of our work: 
https://wiki.openstack.org/wiki/Pci_passthrough#PCI_passthrough_use_notes

Here is the wiki of our possible solution: 
https://wiki.openstack.org/wiki/PCI_passthrough_testing_frame_work_considration




Best Regards,

Tan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO][DiskImage-builder]How to set up network during the deploy

[Tan, Lin]

Hi,

I am working on setting up Baremetal for a while. I notice that in the process 
of deployment in TripleO, I mean in the first round of PXE boot, it will append 
the IP info in the configuration and pass to the target machine as kernel 
parameters. Then init script will read the parameters and set the network like 
eth0 and turn it up. why do we have to do like this?
Because in my case, I want to avoid to pass the IP info as kernel parameters, 
but I still need to set up the network of target machine. I have two ideas now, 
but I am not sure:

1.   Get the IP info from the PXE client in target Machine

2.   Add some new elements in the deploy_ramdisk in order to request the IP 
from the DHCP again.

My question is which way is more reasonable?

Thanks in advanced

Best Regards,

Tan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic] Tooling for recovering nodes

[Tan, Lin]

Hi,

Recently, I am working on a spec[1] in order to recover nodes which get stuck 
in deploying state, so I really expect some feedback from you guys.

Ironic nodes can be stuck in 
deploying/deploywait/cleaning/cleanwait/inspecting/deleting if the node is 
reserved by a dead conductor (the exclusive lock was not released).
Any further requests will be denied by ironic because it thinks the node 
resource is under control of another conductor.

To be more clear, let's narrow the scope and focus on the deploying state 
first. Currently, people do have several choices to clear the reserved lock:
1. restart the dead conductor
2. wait up to 2 or 3 minutes and _check_deploying_states() will clear the lock.
3. The operator touches the DB to manually recover these nodes.

Option two looks very promising but there are some weakness:
2.1 It won't work if the dead conductor was renamed or deleted.
2.2 It won't work if the node's specific driver was not enabled on live 
conductors.
2.3 It won't work if the node is in maintenance. (only a corner case).

Definitely we should improve the option 2, but there are could be more issues I 
didn't know in a more complicated environment.
So my question is do we still need a new command to recover these node easier 
without accessing DB, like this PoC [2]:
  ironic-noderecover --node_uuids=UUID1,UUID2  
--config-file=/etc/ironic/ironic.conf

Best Regards,

Tan


[1] https://review.openstack.org/#/c/319812
[2] https://review.openstack.org/#/c/311273/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] Tooling for recovering nodes

[Tan, Lin]

Thanks Devananda for your suggestions. I opened a new bug for it.

But I am asking this is because this is a task from newton summit to create a 
new command "for getting nodes out of stuck *ing states"
https://etherpad.openstack.org/p/ironic-newton-summit-ops
And we have a RFE bug already for this[1]

But as Dmitry said, there is a big risk to remove the lock of nodes and mark it 
as deploy failed state. But if the tool didn't remove the lock of nodes, then 
users still cannot manipulate the node resource. So I want to involve more 
people to discuss the spec[2].

Considering ironic already have _check_deploying_states() to recover deploying 
state, should I focus on improving it?
Or
There is still a need to create a new command.

B.R

Tan

[1]https://bugs.launchpad.net/ironic/+bug/1580931
[2]https://review.openstack.org/#/c/319812
-Original Message-
From: Devananda van der Veen [mailto:devananda@gmail.com] 
Sent: Wednesday, June 1, 2016 3:26 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [ironic] Tooling for recovering nodes

On 05/31/2016 01:35 AM, Dmitry Tantsur wrote:
> On 05/31/2016 10:25 AM, Tan, Lin wrote:
>> Hi,
>>
>> Recently, I am working on a spec[1] in order to recover nodes which 
>> get stuck in deploying state, so I really expect some feedback from you guys.
>>
>> Ironic nodes can be stuck in
>> deploying/deploywait/cleaning/cleanwait/inspecting/deleting if the 
>> node is reserved by a dead conductor (the exclusive lock was not released).
>> Any further requests will be denied by ironic because it thinks the 
>> node resource is under control of another conductor.
>>
>> To be more clear, let's narrow the scope and focus on the deploying 
>> state first. Currently, people do have several choices to clear the reserved 
>> lock:
>> 1. restart the dead conductor
>> 2. wait up to 2 or 3 minutes and _check_deploying_states() will clear the 
>> lock.
>> 3. The operator touches the DB to manually recover these nodes.
>>
>> Option two looks very promising but there are some weakness:
>> 2.1 It won't work if the dead conductor was renamed or deleted.
>> 2.2 It won't work if the node's specific driver was not enabled on 
>> live conductors.
>> 2.3 It won't work if the node is in maintenance. (only a corner case).
> 
> We can and should fix all three cases.

2.1 and 2.2 appear to be a bug in the behavior of _check_deploying_status().

The method claims to do exactly what you suggest in 2.1 and 2.2 -- it gathers a 
list of Nodes reserved by *any* offline conductor and tries to release the lock.
However, it will always fail to update them, because objects.Node.release() 
raises a NodeLocked exception when called on a Node locked by a different 
conductor.

Here's the relevant code path:

ironic/conductor/manager.py:
1259 def _check_deploying_status(self, context):
...
1269 offline_conductors = self.dbapi.get_offline_conductors()
...
1273 node_iter = self.iter_nodes(
1274 fields=['id', 'reservation'],
1275 filters={'provision_state': states.DEPLOYING,
1276  'maintenance': False,
1277  'reserved_by_any_of': offline_conductors})
...
1281 for node_uuid, driver, node_id, conductor_hostname in node_iter:
1285 try:
1286 objects.Node.release(context, conductor_hostname, node_id)
...
1292 except exception.NodeLocked:
1293 LOG.warning(...)
1297 continue


As far as 2.3, I think we should change the query string at the start of this 
method so that it includes nodes in maintenance mode. I think it's both safe 
and reasonable (and, frankly, what an operator will expect) that a node which 
is in maintenance mode, and in DEPLOYING state, whose conductor is offline, 
should have that reservation cleared and be set to DEPLOYFAILED state.

--devananda

>>
>> Definitely we should improve the option 2, but there are could be 
>> more issues I didn't know in a more complicated environment.
>> So my question is do we still need a new command to recover these 
>> node easier without accessing DB, like this PoC [2]:
>>   ironic-noderecover --node_uuids=UUID1,UUID2 
>> --config-file=/etc/ironic/ironic.conf
> 
> I'm -1 to anything silently removing the lock until I see a clear use 
> case which is impossible to improve within Ironic itself. Such utility may 
> and will be abused.
> 
> I'm fine with anything that does not forcibly remove the lock by default.
> 
>>
>> Best Regards,
>>
>> Tan
>>
>>
>>

Re: [openstack-dev] [ironic] Tooling for recovering nodes

[Tan, Lin]

Hi Jay, Dmitry and guys

I submit two patches to try recover nodes which stuck in deploying state:

1. fix the issue of the ironic-conductor was brought up with a different 
hostname.
 https://review.openstack.org/325026
2. clear the lock of nodes in maintenance states
 https://review.openstack.org/#/c/324269/

If above solutions are promising, then we don't need a new tool to recover 
nodes in deploying state.

B.R

Tan



-Original Message-
From: Jay Faulkner [mailto:j...@jvf.cc] 
Sent: Thursday, June 2, 2016 7:45 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [ironic] Tooling for recovering nodes

Some comments inline.


On 5/31/16 12:26 PM, Devananda van der Veen wrote:
> On 05/31/2016 01:35 AM, Dmitry Tantsur wrote:
>> On 05/31/2016 10:25 AM, Tan, Lin wrote:
>>> Hi,
>>>
>>> Recently, I am working on a spec[1] in order to recover nodes which 
>>> get stuck in deploying state, so I really expect some feedback from you 
>>> guys.
>>>
>>> Ironic nodes can be stuck in
>>> deploying/deploywait/cleaning/cleanwait/inspecting/deleting if the 
>>> node is reserved by a dead conductor (the exclusive lock was not released).
>>> Any further requests will be denied by ironic because it thinks the 
>>> node resource is under control of another conductor.
>>>
>>> To be more clear, let's narrow the scope and focus on the deploying 
>>> state first. Currently, people do have several choices to clear the 
>>> reserved lock:
>>> 1. restart the dead conductor
>>> 2. wait up to 2 or 3 minutes and _check_deploying_states() will clear the 
>>> lock.
>>> 3. The operator touches the DB to manually recover these nodes.
>>>
>>> Option two looks very promising but there are some weakness:
>>> 2.1 It won't work if the dead conductor was renamed or deleted.
>>> 2.2 It won't work if the node's specific driver was not enabled on 
>>> live conductors.
>>> 2.3 It won't work if the node is in maintenance. (only a corner case).
>> We can and should fix all three cases.
> 2.1 and 2.2 appear to be a bug in the behavior of _check_deploying_status().
>
> The method claims to do exactly what you suggest in 2.1 and 2.2 -- it 
> gathers a list of Nodes reserved by *any* offline conductor and tries to 
> release the lock.
> However, it will always fail to update them, because 
> objects.Node.release() raises a NodeLocked exception when called on a Node 
> locked by a different conductor.
>
> Here's the relevant code path:
>
> ironic/conductor/manager.py:
> 1259 def _check_deploying_status(self, context):
> ...
> 1269 offline_conductors = self.dbapi.get_offline_conductors()
> ...
> 1273 node_iter = self.iter_nodes(
> 1274 fields=['id', 'reservation'],
> 1275 filters={'provision_state': states.DEPLOYING,
> 1276  'maintenance': False,
> 1277  'reserved_by_any_of': offline_conductors})
> ...
> 1281 for node_uuid, driver, node_id, conductor_hostname in node_iter:
> 1285 try:
> 1286 objects.Node.release(context, conductor_hostname, 
> node_id)
> ...
> 1292 except exception.NodeLocked:
> 1293 LOG.warning(...)
> 1297 continue
>
>
> As far as 2.3, I think we should change the query string at the start 
> of this method so that it includes nodes in maintenance mode. I think 
> it's both safe and reasonable (and, frankly, what an operator will 
> expect) that a node which is in maintenance mode, and in DEPLOYING 
> state, whose conductor is offline, should have that reservation cleared and 
> be set to DEPLOYFAILED state.

This is an excellent idea -- and I'm going to extend it further. If I have any 
nodes in a *ING state, and they are put into maintenance, it should force a 
failure. This is potentially a more API-friendly way of cleaning up nodes in 
bad states -- an operator would need to maintenance the node, and once it 
enters the *FAIL state, troubleshoot why it failed, unmaintenance, and return 
to production.

I obviously strongly desire an "override command" as an operator, but I really 
think this could handle a large percentage of the use cases that made me desire 
it in the first place.

> --devananda
>
>>> Definitely we should improve the option 2, but there are could be 
>>> more issues I didn't know in a more complicated environment.
>>> So my question is do we still need a new command to recover these 
>>> node easier without accessing DB,