Re: [openstack-dev] [telemetry][aodh][panko][oslo][performance] OSprofiler in Aodh & Panko

[vin...@vn.fujitsu.com]

Hello folks,

Just a reminder.

I have some patches related to OSProfiler that ready to review in Panko and 
Aodh.
Hope that you guys can leave a comment.

These patches are:
1. Aodh: https://review.openstack.org/#/c/483268/
2. Aodh client: https://review.openstack.org/#/c/484295/
3. Panko: https://review.openstack.org/#/c/483848/
4. Panko client: https://review.openstack.org/#/c/484294/

Thank you!

Best regards,

Vinh Nguyen Trong
PODC – Fujitsu Vietnam Ltd. 


> -Original Message-
> From: Nguyen, Trong Vinh
> Sent: Monday, 14 August, 2017 08:24
> To: OpenStack Development Mailing List (not for usage questions)  d...@lists.openstack.org>; Trong Vinh Nguyen (vin...@vn.fujitsu.com)
> <vin...@vn.fujitsu.com>
> Subject: [openstack-dev][telemetry][aodh][panko][oslo][performance] 
> OSprofiler in
> Aodh & Panko
> 
> Hello,
> 
> I’m sending this email for asking about the work in integrating OSprofiler 
> into Aodh &
> Panko.
> Currently, there are some patches related to this work, and they are waiting 
> for review:
> 1. Aodh: https://review.openstack.org/#/c/483268/
> 2. Aodh client: https://review.openstack.org/#/c/484295/
> 3. Panko: https://review.openstack.org/#/c/483848/
> 4. Panko client: https://review.openstack.org/#/c/484294/
> 
> FYI, OSprofiler provides functionality to generate a trace per request, that 
> goes
> through all involve services.
> This trace can visualize flow of a request [1] [2].
> A trace from OSprofiler can help us know these things:
> - Performance bottle-neck of a service
> - Trouble-shooting issue in a service
> - Understanding flow of a request (from cli client or other client)
> - Trace can be store in persistent storage
> - Visualization trace flow in many OpenTracing compatible tracer [2] 
> (will be done
> soon)
> - Head, tail-based sampling for reducing overhead [3]
> - Asynchronous tracing [4]
> 
> OSprofiler has already been in most of main OpenStack services such as: Nova,
> Neutron, Keystone, Glance, and Cinder...
> 
> Hope that it will receive reviews from you all.
> 
> Thanks!
> 
> [1] Demo with current OSprofiler patch set in Swift:
> https://tovin07.github.io/swift/swift-object-create.html
> [2] A demo with OpenTracing compatible (using Uber Jaeger):
> https://tovin07.github.io/opentracing/jaeger-openstack-image-list.png
> [3] Tail-based coherent sampling: 
> https://blueprints.launchpad.net/osprofiler/+spec/tail-
> based-coherent-sampling
> [4] Asynchronous tracing:
> https://blueprints.launchpad.net/osprofiler/+spec/asynchronous-trace-collection
> [5] OSprofiler documentation: https://docs.openstack.org/osprofiler/latest/
> 
> Best regards,
> 
> Vinh Nguyen Trong
> PODC – Fujitsu Vietnam Ltd.
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [telemetry][aodh][panko][oslo][performance] OSprofiler in Aodh & Panko

[vin...@vn.fujitsu.com]

Hello,

I'm sending this email for asking about the work in integrating OSprofiler into 
Aodh & Panko.
Currently, there are some patches related to this work, and they are waiting 
for review:
1. Aodh: https://review.openstack.org/#/c/483268/
2. Aodh client: https://review.openstack.org/#/c/484295/
3. Panko: https://review.openstack.org/#/c/483848/
4. Panko client: https://review.openstack.org/#/c/484294/

FYI, OSprofiler provides functionality to generate a trace per request, that 
goes through all involve services.
This trace can visualize flow of a request [1] [2].
A trace from OSprofiler can help us know these things:
- Performance bottle-neck of a service
- Trouble-shooting issue in a service
- Understanding flow of a request (from cli client or other client)
- Trace can be store in persistent storage
- Visualization trace flow in many OpenTracing compatible tracer [2] (will 
be done soon)
- Head, tail-based sampling for reducing overhead [3]
- Asynchronous tracing [4]

OSprofiler has already been in most of main OpenStack services such as: Nova, 
Neutron, Keystone, Glance, and Cinder...

Hope that it will receive reviews from you all.

Thanks!

[1] Demo with current OSprofiler patch set in Swift: 
https://tovin07.github.io/swift/swift-object-create.html
[2] A demo with OpenTracing compatible (using Uber Jaeger): 
https://tovin07.github.io/opentracing/jaeger-openstack-image-list.png
[3] Tail-based coherent sampling: 
https://blueprints.launchpad.net/osprofiler/+spec/tail-based-coherent-sampling
[4] Asynchronous tracing: 
https://blueprints.launchpad.net/osprofiler/+spec/asynchronous-trace-collection
[5] OSprofiler documentation: https://docs.openstack.org/osprofiler/latest/

Best regards,

Vinh Nguyen Trong
PODC - Fujitsu Vietnam Ltd. 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Tracing (all the places)

[vin...@vn.fujitsu.com]

> -Original Message-
> From: Joshua Harlow [mailto:harlo...@fastmail.com]
> Sent: Friday, 04 August, 2017 11:12
> To: OpenStack Development Mailing List (not for usage questions)  d...@lists.openstack.org>
> Subject: Re: [openstack-dev] Tracing (all the places)
> 
> vin...@vn.fujitsu.com wrote:
> > Hello harlowja,
> >
> > I'm really happy to see that you are back in this `tracing` topic [and 
> > @boris-42
> (too)].
> 
> We never left, haha, but ya, I can say (and probably boris would agree)
> that trying to get OSprofiler started and integrated somewhat 'burned'
> both of us (it involved a ton of convincing people of the value of it,
> when I had more hoped that the value of it was obvious). But I'm glad
> that people are starting to realize its value (even if they have to be
> told and educated by google or other companies that have been doing this
> for a long time).
> 
> >
> > Last week, we saw that Rajul proposed 02 new blueprint in OSprofiler [1] 
> > and [2].
> > Besides, some other blueprints are being implemented in OSprofiler
> > such as overhead control [3] and OpenTracing compatible [4] [5]
> > (Uber Jaeger [6] is one of OpenTracing compatible tracer out there).
> >
> > For OpenTracing part, I have a PoC to make OSprofiler compatible with
> > OpenTracing specification at [5]. You can take a look at it this time too.
> > However, this time, I focus on reporting span/trace to other destinations
> > (rather than current drivers for OSprofiler[7]).
> >
> > OpenTracing API is changing a little bit fast for now, therefore, some APIs 
> > will be
> deprecated soon.
> > I had some discussions with OpenTracing community about some trouble when
> making OSprofiler
> > compatible with OpenTracing API.
> 
> Ya I expected this, opentracing also I think has a python
> client/wrapper(?), have you looked at what this offers (last time I
> checked most of opentracing was just a bunch of wrappers actually, and
> not much actually code that did anything unique)?
> 

Wrapper? Yes and No :D
If you look to the code, OpenTracing is just an interface and other new tracers 
should
follow it to implement. However, with OpenTracing, user can use any tracers that
`OpenTracing compatible` without modify a bunch of instrumented code.
Just change the name of tracer that you want to use.

For example, change tracer from Jaeger to LightStep.

With Jaeger:
opentracing.tracer = jeager_client.Config().config.initialize_tracer()

With lightstep:
opentracing.tracer = lightstep.Tracer(
component_name='your_microservice_name',
access_token='{your_access_token}')

Very simple (but not enough) :D


Jaeger client: https://github.com/uber/jaeger-client-python
LightStep: https://github.com/lightstep/lightstep-tracer-python

> >
> > For OpenStack part, last cycle, Performance team and other OpenStack 
> > developers
> added
> > OSprofiler support for many other projects (Nova, Magnum, Ironic, Zun ...)
> > and Panko, Aodh, Swift are on the way.
> 
> Yippe, now the bigger questions is where are all the UIs visualizing the
> traces (I know boris had https://boris-42.github.io/ngk.html but there
> has to be something nicer that perhaps the OpenTracing community has for
> a UI, ideally not a java monster like Zipkin, ha). Any thoughts there?
> 

With the UI, you can see the example with OSprofiler ouput that I redirect to
Jaeger here: 
https://tovin07.github.io/opentracing/jaeger-openstack-image-list.png

> >
> > At last, hope you will join us (again) in OpenStack `tracing` things.
> 
> We shall see :-P
> 
> >
> > [1] 
> > https://blueprints.launchpad.net/osprofiler/+spec/asynchronous-trace-collection
> > [2] 
> > https://blueprints.launchpad.net/osprofiler/+spec/tail-based-coherent-sampling
> > [3] 
> > https://blueprints.launchpad.net/osprofiler/+spec/osprofiler-overhead-control
> > [4] https://blueprints.launchpad.net/osprofiler/+spec/opentracing-compatible
> > [5] https://review.openstack.org/#/c/480018/
> > [6] http://jaeger.readthedocs.io/en/latest/architecture/
> > [7] https://github.com/openstack/osprofiler/tree/master/osprofiler/drivers
> >
> > Best regards,
> >
> > Vinh Nguyen Trong
> > PODC – Fujitsu Vietnam Ltd.
> 
> ___
> ___
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][performance] Proposing tail-based sampling in OSProfiler

[vin...@vn.fujitsu.com]

Hi Rajul,

For the `agent idea`, I think it is very good.
However, in OpenStack, that idea may be really hard for us.
The reason is the same with what Boris think.

For the sampling part, head-based sampling can be implemented in OSprofiler.
For tail-based and adaptive sampling, it is another story.
However, in naïve way, we can use sampling abilities from other OpenTracing 
compatible tracers
such as Uber Jaeger, Appdash, Zipkin (has an open pull request), LighStep … by 
making OSprofiler
compatible with OpenTracing API.

ICYMI, Boris is father of OSprofiler in OpenStack [1]

[1] 
https://specs.openstack.org/openstack/oslo-specs/specs/mitaka/osprofiler-cross-service-project-profiling.html

Best regards,

Vinh Nguyen Trong
PODC – Fujitsu Vietnam Ltd.

From: Rajul Kumar [mailto:kumar.r...@husky.neu.edu]
Sent: Friday, 04 August, 2017 03:49
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [oslo][performance] Proposing tail-based sampling 
in OSProfiler

Hi Boris

That is a point of concern.
Can you please direct to any of those?

Anyways, we don't have anything in place for OpenStack yet.
Now, either we pick another tracing solution like Zipkin, Jaeger etc. which 
have their own limitations OR enhance OSProfiler.
We pick the later as it's most native and better coupled with OpenStack as of 
now.
I understand that we may be blocked by these issues. However, I feel it'll be 
better to fight with OSProfiler than anything else till we come up with 
something better :)

Thanks
Rajul



On Thu, Aug 3, 2017 at 4:01 PM, Boris Pavlovic 
> wrote:
Rajul,

May I ask why you think so?

Exposed by OSprofiler issues are going to be really hard to fix in current 
OpenStack architecture.

Best regards,
Boris Pavlovic

On Thu, Aug 3, 2017 at 12:56 PM, Rajul Kumar 
> wrote:
Hi Boris

Good to hear from you.
May I ask why you think so?

We do see some potential with OSProfiler for this and further objectives.

Thanks
Rajul

On Thu, Aug 3, 2017 at 3:48 PM, Boris Pavlovic 
> wrote:
Rajul,

It makes sense! However, maybe it's a bit too late... ;)

Best regards,
Boris Pavlovic

On Thu, Aug 3, 2017 at 12:16 PM, Rajul Kumar 
> wrote:
Hello everyone

I have added a blueprint on having tail-based sampling as a sampling option for 
continuous tracing in OSProfiler. It would be really helpful to have some 
thoughts, ideas, comments on this from the community.

Continuous tracing provides a good insight on how various transactions behave 
across in a distributed system. Currently, OpenStack doesn't have a defined 
solution for continuous tracing. Though, it has OSProfiler that does generates 
selective traces, it may not capture the occurrence. Even if we have OSProfiler 
running continuously [1], we need to sample the traces so as to cut down the 
data generated and still keep the useful info.

Head based sampling can be applied that decides initially whether a trace 
should be saved or not. However, it may miss out on some useful traces. I 
propose to have tail-based sampling [2] mechanism that makes the decision at 
the end of the transaction and tends to keep all the useful traces. This may 
require a lot of changes depending on what all type of info is required and the 
solution that we pick to implement it [2]. This may not affect the current 
working of any of the services on OpenStack as it will be off the critical path 
[3].

Please share your thoughts on this and what solution should be preferred in a 
broader OpenStack's perspective.
This is a step in the process of having an automated diagnostic solution for 
OpenStack cluster.

[1] 
https://blueprints.launchpad.net/osprofiler/+spec/osprofiler-overhead-control
[2] 
https://blueprints.launchpad.net/osprofiler/+spec/tail-based-coherent-sampling
[3] 
https://blueprints.launchpad.net/osprofiler/+spec/asynchronous-trace-collection

Thanks
Rajul Kumar


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] Tracing (all the places)

[vin...@vn.fujitsu.com]

Hello harlowja,

I'm really happy to see that you are back in this `tracing` topic [and 
@boris-42 (too)].

Last week, we saw that Rajul proposed 02 new blueprint in OSprofiler [1] and 
[2].
Besides, some other blueprints are being implemented in OSprofiler
such as overhead control [3] and OpenTracing compatible [4] [5]
(Uber Jaeger [6] is one of OpenTracing compatible tracer out there).

For OpenTracing part, I have a PoC to make OSprofiler compatible with
OpenTracing specification at [5]. You can take a look at it this time too.
However, this time, I focus on reporting span/trace to other destinations
(rather than current drivers for OSprofiler[7]).

OpenTracing API is changing a little bit fast for now, therefore, some APIs 
will be deprecated soon.
I had some discussions with OpenTracing community about some trouble when 
making OSprofiler
compatible with OpenTracing API.

For OpenStack part, last cycle, Performance team and other OpenStack developers 
added
OSprofiler support for many other projects (Nova, Magnum, Ironic, Zun ...)
and Panko, Aodh, Swift are on the way.

At last, hope you will join us (again) in OpenStack `tracing` things.

[1] 
https://blueprints.launchpad.net/osprofiler/+spec/asynchronous-trace-collection
[2] 
https://blueprints.launchpad.net/osprofiler/+spec/tail-based-coherent-sampling
[3] 
https://blueprints.launchpad.net/osprofiler/+spec/osprofiler-overhead-control
[4] https://blueprints.launchpad.net/osprofiler/+spec/opentracing-compatible
[5] https://review.openstack.org/#/c/480018/
[6] http://jaeger.readthedocs.io/en/latest/architecture/
[7] https://github.com/openstack/osprofiler/tree/master/osprofiler/drivers

Best regards,

Vinh Nguyen Trong
PODC – Fujitsu Vietnam Ltd. 

> -Original Message-
> From: Joshua Harlow [mailto:harlo...@fastmail.com]
> Sent: Friday, 04 August, 2017 02:52
> To: OpenStack Development Mailing List (not for usage questions)  d...@lists.openstack.org>
> Subject: [openstack-dev] Tracing (all the places)
> 
> Since I think there was another thread out there around tracing I'd
> thought I'd send out a few others for folks that show tracing being
> added to multiple other popular project (interesting to read over the
> proposals and such).
> 
> -
> https://github.com/grpc/grpc/blob/master/src/core/ext/census/README.md#census---a-
> resource-measurement-and-tracing-system
> 
> - https://github.com/kubernetes/kubernetes/issues/26507 (k8s tracing
> addition/proposal)
> 
> - http://jaeger.readthedocs.io/en/latest/architecture/
> 
> It'd be real nice to finally get some kind of tracing support integrated
> into openstack; osprofiler was started a long time ago and I think it's
> due time for it to actually be used and integrated :)
> 
> -Josh
> 
> ___
> ___
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [swift] [oslo] [osprofiler] [performance] OSprofiler in Swift

[vin...@vn.fujitsu.com]

Hello folks,

I'm sending this email for asking about the works in integrating OSprofiler 
into Swift.
Currently I'm working on this and the patches in Swift is waiting for review 
here: https://review.openstack.org/#/c/468316/

We knew that Swift has xprofile module [3] to do profiling works. However, 
OSprofiler is different from xprofile in many aspects [6].
xprofile is a profiling tool, OSprofiler is a distributed tracing tool for 
OpenStack services. We need OSProfiler in Swift for tracing across other 
OpenStack services.

FYI, OSprofiler provides functionality to generate a trace per request, that 
goes through all involve services.
This trace can visualize flow of a request [4] [5].
A trace from OSprofiler can help us know these things:
- Performance bottle-neck of a service
- Trouble-shooting issue in a service
- Understanding flow of a request (from cli client or other client)
- Trace can be store in persistent storage
- Visualization trace flow in many OpenTracing compatible tracer [5] (will 
be done soon)

Hope that it will receive reviews from you all.

Some related references:
[1] OSprofiler documentation: https://docs.openstack.org/osprofiler/latest/
[2] Swift documentation: https://docs.openstack.org/swift/latest/ 
[3] Swift xprofile: 
https://docs.openstack.org/swift/latest/middleware.html#module-swift.common.middleware.xprofile
 
[4] Demo with current OSprofiler patch set in Swift: 
https://tovin07.github.io/swift/swift-object-create.html
[5] A demo with OpenTracing compatible (using Uber Jaeger): 
https://tovin07.github.io/opentracing/jaeger-openstack-image-list.png
[6] Why not cProfile and others? 
https://docs.openstack.org/osprofiler/latest/user/background.html#why-not-cprofile-and-etc
[7] xprofile can use cProfile to profile internal python call: 
https://github.com/openstack/swift/search?utf8=%E2%9C%93=cprofile=
[8] Some concerns from notmyname: 
http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2017-05-03.log.html#t2017-05-03T15:33:25
[9] Discussions from IRC meeting log: 
http://eavesdrop.openstack.org/meetings/swift/2017/swift.2017-06-14-07.00.log.html#l-152

Best regards,

Vinh Nguyen Trong
PODC - Fujitsu Vietnam Ltd.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova][oslo][osprofiler] Next step of OSprofiler

[vin...@vn.fujitsu.com]

Hi Rui Wang,

We also recognize the benefit of OSProfiler and are on the way to improve it.

> 1. The integration with neutronclient,
> nova and novaclient has not been done yet.

Yes, the works in Nova are in reviewing but Nova team seems very busy
in this cycle and they want to have more tests before accepting it [1][2].
Progress on support in Neutron client is slow and temporary paused [3].

> 2. The blueprint "osprofiler-overhead-control" is not in progress.
> The spec[1] was workflow -1.
> Does it mean this feature would not been done in Ocata ?

Yes (too), this spec is in progress to finish itself, not the code.
So, I mark it with Workflow -1 to have time to finish it.
Besides, this cycle (Ocata) is very short, I don't think that we have chance
to finish the code for this in O-cycle.

> 3. We need a server/tool to collect and analysis the trace data.
> Should this implemented by OSprofiler,
> or this could be implemented by another project of OSprofiler?

OSProfiler is just a small library that support tracing and profiling requests.
I think OSProfiler should do its job (tracing) only.
Collecting and analyzing should be in other project(s) such as Monasca.
However, this is the next step after completely integrating OSProfiler
in OpenStack.

Thank you for your attention. Hope that we can collaborate in this work! :D

[1] https://review.openstack.org/#/c/254703/
[2] https://review.openstack.org/#/c/254699/
[3] https://review.openstack.org/#/c/281508/


Best regards,

Vinh Nguyen Trong (IRC, Gerrit: tovin07)
PODC – Fujitsu Vietnam Ltd.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][osprofiler] Sampling rate in OSProfiler

[vin...@vn.fujitsu.com]

Hey guys,

cc: harlowja, DinaBelova, boris-42

I saw a lot of great work in OSProfiler.

I see that "Trace every N request configuration (not done yet)". [1]
Both Zipkin [2] and OpenTracing [3] have this feature.
OSProfiler aim to be a small library --> May this feature make any overhead?

Do you guys have any idea how to implement this?

[1] Mitaka spec: 
https://specs.openstack.org/openstack/oslo-specs/specs/mitaka/osprofiler-cross-service-project-profiling.html#performance-impact
[2] Zipkin: http://zipkin.io/pages/instrumenting.html
[3] OpenTracing: 
http://opentracing.io/documentation/pages/api/data-conventions.html

Best regards,

Vinh Nguyen Trong (tovin07/Tovin Seven)
PODC - Fujitsu Vietnam Ltd. 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Vitrage] Problem at our gate in gerrit

[vin...@vn.fujitsu.com]

Thank you for your work!

Best regards,

Vinh Nguyen Trong
PODC - Fujitsu Vietnam Ltd.
Mobile: +84-164-953-8507 - Email: vin...@vn.fujitsu.com

> -Original Message-
> From: Weyl, Alexey (Nokia - IL) [mailto:alexey.w...@nokia.com]
> Sent: Tuesday, November 22, 2016 4:34 PM
> To: OpenStack Development Mailing List (not for usage questions)  d...@lists.openstack.org>
> Subject: [openstack-dev] [Vitrage] Problem at our gate in gerrit
> 
> Hi,
> 
> We have some problems at our gate in gerrit.
> 
> We have fixed one problem by changing the image that the tempests run on to 
> xenial
> image.
> 
> But that has created some other problem, and we are checking it.
> 
> I will send an email to update when the problem is resolved.
> 
> Alexey
> 
> ___
> ___
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev