Re: [openstack-dev] [MagnetoDB] Core developer nomination

[Illia Khudoshyn]

Congrats, Charles! Great job!

On Thu, Sep 18, 2014 at 12:05 AM, Ilya Sviridov 
wrote:

> Hello magnetodb contributors,
>
> I'm glad to nominate Charles Wang to core developers of MagnetoDB.
>
> He is top non-core reviewer [1], implemented notifications [2] in mdb and
> made a great progress with performance, stability and scalability testing
>  of MagnetoDB
>
> [1] http://stackalytics.com/report/contribution/magnetodb/90
> [2]
> https://blueprints.launchpad.net/magnetodb/+spec/magnetodb-notifications
>
> Welcome to team, Charles!
> Looking forward for your contribution
>
> --
> Ilya Sviridov
> isviridov @ FreeNode
>



-- 

Best regards,

Illia Khudoshyn,
Software Engineer, Mirantis, Inc.



38, Lenina ave. Kharkov, Ukraine

www.mirantis.com 

www.mirantis.ru



Skype: gluke_work

ikhudos...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] DVR Tunnel Design Question

[Kevin Benton]

This sounds like a bug in Openvswitch. The unique constraint should be
VLAN+MAC instead of just VLAN, so observing the same MAC on a new VLAN
should just result in a new entry without the old one being ejected.

Without this behavior, it will also break transparent L2 firewalls.
For example (diagram below), if you divide a switch into two VLANs and
then connect the sides together with a firewall in the middle that
passes frames without modifying the MAC, the switch will see the same
MAC on different VLANs. Based on the behavior you described, this
wouldn't work. Is that correct?


-
|  x  x  x  x   |   x  x  x  x  |
|---|-
 VLAN 1 | |  VLAN2
  
  |   Firewall  |
  

On Wed, Sep 17, 2014 at 11:19 PM, Narasimhan, Vivekanandan
 wrote:
> Hi Kevin,
>
>
>
> Typically we noticed that the underlay switches maintained a table like
> this:
>
>
>
> VLAN IDMAC Address  Learned-Interface
>
>
>
> In the physical underlay, with the current architecture if we enable VLAN,
> the same DVR Unique
>
> MAC will appear  on different VLANs as the packets get DVR Routed.
>
>
>
> This will result in the rows of the above tables in the switch to be updated
> very frequently with new
>
> VLANs noted in incoming packets for the same DVR MAC Address, even though
> they are from the
>
> same physical port.
>
>
>
> We are not sure if all the switches maintained the tables this way , but
> atleast we
>
> saw Openvswitch implementations did.  So we consciously did not promote
> VLANs for
>
> initial phase of DVR.
>
>
>
> --
>
> Thanks,
>
>
>
> Vivek
>
>
>
>
>
> From: Kevin Benton [mailto:blak...@gmail.com]
> Sent: Thursday, September 18, 2014 3:01 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question
>
>
>
> Can you clarify what you mean with the thrashing condition? MAC addresses
> only need to be unique per-VLAN so I don't see how the same MAC on multiple
> VLANs from the same physical port would lead to any issues.
>
>
>
> On Wed, Sep 17, 2014 at 12:41 PM, Armando M.  wrote:
>
> VLAN is on the radar, vxlan/gre was done to start with.
>
> I believe Vivek mentioned the rationale in some other thread. The gist
> of it below:
>
> In the current architecture, we use a unique DVR MAC per compute node
> to forward DVR Routed traffic directly to destination compute node.
> The DVR routed traffic from the source compute node will carry
> 'destination VMs underlay VLAN' in the frame, but the Source Mac in
> that same frame will be the DVR Unique MAC. So, same DVR Unique Mac is
> used for potentially a number of overlay network VMs that would exist
> on that same source compute node.
>
> The underlay infrastructure switches will see the same DVR Unique MAC
> being associated with different VLANs on incoming frames, and so this
> would result in VLAN Thrashing on the switches in the physical cloud
> infrastructure. Since tunneling protocols carry the entire DVR routed
> inner frames as tunnel payloads, there is no thrashing effect on
> underlay switches.
>
> There will still be thrashing effect on endpoints on CNs themselves,
> when they try to learn that association between inner frame source MAC
> and the TEP port on which the tunneled frame is received. But that we
> have addressed in L2 Agent by having a 'DVR Learning Blocker' table,
> which ensures that learning for DVR routed packets alone is
> side-stepped.
>
> As a result, VLAN was not promoted as a supported underlay for the
> initial DVR architecture.
>
> Cheers,
> Armando
>
>
> On 16 September 2014 20:35, 龚永生  wrote:
>> I think the VLAN should also be supported later.  The tunnel should not be
>> the prerequisite for the DVR feature.
>>
>>
>> -- Original --
>> From:  "Steve Wormley";
>> Date:  Wed, Sep 17, 2014 10:29 AM
>> To:  "openstack-dev";
>> Subject:  [openstack-dev] [neutron] DVR Tunnel Design Question
>>
>> In our environment using VXLAN/GRE would make it difficult to keep some of
>> the features we currently offer our customers. So for a while now I've
>> been
>> looking at the DVR code, blueprints and Google drive docs and other than
>> it
>> being the way the code was written I can't find anything indicating why a
>> Tunnel/Overlay network is required for DVR or what problem it was solving.
>>
>> Basically I'm just trying to see if I missed anything as I look into doing
>> a
>> VLAN/OVS implementation.
>>
>> Thanks,
>> -Steve Wormley
>>
>>
>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
> --
>
> Kevin Be

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[mar...@redhat.com]

On 17/09/14 16:40, Charles Crouch wrote:
> 
> 
> - Original Message -
>> Hi,
>>
>> as part of general housekeeping on our reviews, it was discussed at last
>> week's meeting [1] that we should set workflow -1 for stale reviews
>> (like gerrit used to do when I were a lad).
>>
>> The specific criteria discussed was 'items that have a -1 from a core
>> but no response from author for 14 days'. This topic came up again
>> during today's meeting and it wasn't clear if the intention was for
>> cores to start enforcing this? So:
>>
>> Do we start setting WIP/workflow -1 for those reviews that have a -1
>> from a core but no response from author for 14 days
>>
>> thanks, marios
>>
>> [1]
>> http://eavesdrop.openstack.org/meetings/tripleo/2014/tripleo.2014-09-09-19.04.log.html
> 
> So it looks like this has already started..
> 
> https://review.openstack.org/#/c/105275/
> 
> I think we need to document on the wiki *precisely* the criteria for setting 
> WIP/workflow -1. For example that review above has a Jenkins failure but no
> core reviews at all.

+1 on being precise - another case in point:

https://review.openstack.org/#/c/102304/

this review has a *-2* from core, which seems to 'stick' for future
revisions; the last revision is from > 14 days ago, so does this fulfill
the criteria (I'd say it doesn't)?

Not sure if you were also suggesting that -1 from Jenkins also counts,
which imo makes sense...

'items that have a -1 from core or jenkins, or a -2 from core on the
latest revision, and no response from author for 14 days'

Really this needs to be put as a motion and voted on in the next meeting,

thanks, marios


> 
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[mar...@redhat.com]

On 18/09/14 00:29, James Polley wrote:
> 
> 
> On Wed, Sep 17, 2014 at 6:26 PM, mar...@redhat.com
>   > wrote:
> 
> Hi,
> 
> as part of general housekeeping on our reviews, it was discussed at last
> week's meeting [1] that we should set workflow -1 for stale reviews
> (like gerrit used to do when I were a lad).
> 
> The specific criteria discussed was 'items that have a -1 from a core
> but no response from author for 14 days'. This topic came up again
> during today's meeting and it wasn't clear if the intention was for
> cores to start enforcing this? So:
> 
> Do we start setting WIP/workflow -1 for those reviews that have a -1
> from a core but no response from author for 14 days
> 
> 
> I'm in favour of doing this; as long as we make it clear that we're
> doing it to help us focus review effort on things that are under active
> development - it doesn't mean we think the patch shouldn't land, it just
> means we know it's not ready yet so we don't want reviewers to be
> looking at it until it moves forward.
> 
> For the sake of making sure new developers don't get put off, I'd like
> to see us leaving a comment explaining why we're WIPing the change and
> noting that uploading a new revision will remove the WIP automatically
> 

+1 - indeed, I'd say as part of this discussion, or if/when it comes up
as a motion for a vote in the weekly meeting, we should also put out and
agree on the 'standard' text to be used for this and stick it on the
wiki (regardless of whether this is to be implemented manually at first
and perhaps automated later),

thanks, marios

"setting workflow -1 as this review has been inactive for two weeks
following a negative review. Please see the wiki @ foo for more
information. Note that once you upload a new revision the workflow is
expected to be reset (feel free to shout on freenode/#tripleo if it isn't)."



> 
> thanks, marios
> 
> [1]
> 
> http://eavesdrop.openstack.org/meetings/tripleo/2014/tripleo.2014-09-09-19.04.log.html
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Chmouel Boudjnah]

Ian Cordasco  writes:

> urllib3 do that automatically. I haven’t started to investigate exactly
> why they do this. Likewise, glance client has custom certificate
> verification in glanceclient.common.https. Why? I’m not exactly certain

this probably come from pre-requests port uses when it was using httplib
which didn't have SSL verification. There is a old wiki page about it
here https://wiki.openstack.org/wiki/SecureClientConnections

Slightly off-topic, speaking about requests and OpenStack client, it
would be nice to have Expect/100-Continue feature tackled for
glanceclient and swiftclient :

https://github.com/kennethreitz/requests/issues/713

Chmouel

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Convergence - persistence desired and observed state

[Murugan, Visnusaran]

Yeah. Unmesh, we need to have ResourcePropertiesObserved. The columns too need 
to be as mentioned in blueprint. Having all properties under a single json will 
cause concurrency issues.

-Vishnu

-Original Message-
From: Qiming Teng [mailto:teng...@linux.vnet.ibm.com] 
Sent: Wednesday, September 17, 2014 6:52 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [heat] Convergence - persistence desired and 
observed state

On Wed, Sep 17, 2014 at 12:27:34PM +, Gurjar, Unmesh wrote:
> Hi All,
> 
> The convergence blueprint (https://review.openstack.org/#/c/95907/) 
> introduces two new database tables (resource_observed and 
> resource_properties_observed ) for storing the observed state of a resource 
> (currently under review: https://review.openstack.org/#/c/109012/).
> 
> However, it can be simplified by storing the desired and observed state of a 
> resource in the resource table itself (two columns in the form of a blob 
> storing a JSON). Please let me know your concerns or suggestions about this 
> approach.
> 
> Thanks,
> Unmesh G.

It doesn't sounds like a good idea to me unless we have some plans to handle 
potential concurrency and compatibility issues.

Regards,
Qiming


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[James Page]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 17/09/14 15:34, Doug Hellmann wrote:
> This thread [1] has turned more “future focused", so I’m moving the
> conversation to the -dev list where we usually have those sorts of
> discussions.
[...]
>> I'd also like to document the current design of the ZMQ driver -
>> Doug - where is the best place todo this? I thought in the source
>> tree somewhere.
> 
> The documentation in the oslo.messaging repository [2] would be a
> good place to start for that. If we decide deployers/operators need
> the information we can either refer to it from the guides managed
> by the documentation team or we can move/copy the information. How
> about if you start a new drivers subdirectory there, and add
> information about zmq. We can have other driver authors provide
> similar detail about their drivers in the same directory.
> 
> [2]
> http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source

OK
> 
- - I'll raise a review with some appropriate design documentation
sometime in the next few weeks.

>> 2) a list of the critical bugs that need to be fixed + any
>> existing patches associated with those bugs, so they can be
>> reviewed early in kilo
[...]
> That’s a good list, and shorter than I expected. I have added these
> bugs to the next-kilo milestone.

Thanks

>> 3) an analysis of what it would take to be able to run
>> functional tests for zeromq on our CI infrastructure, not
>> necessarily the full tempest run or devstack-gate job, probably
>> functional tests we place in the tree with the driver (we will be
>> doing this for all of the drivers) + besides writing new
>> functional tests, we need to bring the unit tests for zeromq into
>> the oslo.messaging repository
>> 
>> Kapil Thangavelu started work on both functional tests for the
>> ZMQ driver last week; the output from the sprint is here:
>> 
>> https://github.com/ostack-musketeers/oslo.messaging
>> 
>> it covers the ZMQ driver (including messaging through the
>> zmq-receiver proxy) and the associated MatchMakers (local, ring,
>> redis) at a varying levels of coverage, but I feel it moves
>> things in the right direction - Kapil's going to raise a review
>> for this in the next couple of days.
>> 
>> Doug - has any structure been agreed within the oslo.messaging
>> tree for unit/functional test splits? Right now we have them all
>> in one place.
> 
> I think we will want them split up, but we don’t have an agreed
> existing structure for that. I would like to see a test framework
> of some sort that defines the tests in a way that can be used to
> run the same functional for all of the drivers as separate jobs
> (with appropriate hooks for ensuring the needed services are
> running, etc.). Setting that up warrants its own spec, because
> there are going to be quite a few details to work out. We will also
> need to participate in the larger conversation about how to set up
> those functional test jobs to be consistent with the other
> projects.

I guess we can put then up for review as is and then refactor to
support any future framework changes that come along.

>> 4) and some improvements that we would like to make longer term
>> 
>> a) Connection re-use on outbound messaging avoiding the current
>> tcp setup overhead for every sent message.  This may also bring
>> further performance benefits due to underlying messaging batching
>> in ZMQ.
> 
> This sounds like it would be a good thing to do, but making what we
> have work correctly and testing it feels more important for now.

Agreed - these are all futures.

>> b) Moving from tcp PUSH/PULL sockets between servers to
>> DEALER/DEALER (or something similar) to allow for heartbeating
>> and more immediate failure detection
> 
> I would need to understand how much of a rewrite that represents
> before commenting further.

Sorry - that item was a little hand-wavey - it could be minimal in
terms of impact but I would probably see it tied to a) so it might
involve the zmq-receiver moving towards zmq-proxy for both inbound and
outbound tcp connections.

>> 
>> c) Crypto support
> 
> There are some other discussions about adding crypto to messaging,
> and I hope we can do that without having to touch each driver, if
> possible.

That would be even better IMHO.

- -- 
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUGo8jAAoJEL/srsug59jDAMMP/0RCc3ElFC/E/UqZJrY+X2oE
YMB87TM06AX26wS2uKe7F7xEAsv+vMTcSK/FC13xGfd+xBTT13pHquBQjeHwUx2C
+2Fbp6QoLwbvZF69IWL0E/8M/KueGth+JrHRpPqQqK5OcuwmDbDOeq2eychDMg1w
pkpXF0PPsa0tBRo3VQUHc43po9fAqT9Vpf8nG1M5vNTuOcCnrrmGg3DOFCzjVlG1
u2Mfo3HNIL48ZdkrSLHMk+7V1j4qHdU8ADxKaE3aPbxAaSieTlsKtf4pxJSM9Ikg
6WBsCChIVZA62Ox9xRZntidWdHTKcc4Lsv/K9ZhnFzST6OoWvEkpNxol1g/YX9lV
30TUdVBhNGm+ZU+2J8vxJ5suyVD0oW/J2hZGwhomi3tpKk9euKb5HMe5Ln0Fy1k9
63//XE8IzGIkNzwnBd4qnEubo1hg7Jfuw30uij9jJ85IdWRrpXiiei4XKZNJsAgD
/RsKzQt+BwlcztcWoQ0RX3OuOonQXb+5EgfqBce8DDHs+xi

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Flavio Percoco]

On 09/18/2014 04:43 AM, Donald Stufft wrote:
> 
>> On Sep 17, 2014, at 10:24 PM, Thomas Goirand > > wrote:
>>
>> On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
>>> I think that all of the conversation to this point has been valuable,
>>> the general consensus is vendoring a library is not as desirable as
>>> using it strictly as a dependency. It would be nice in a perfect
>>> world if vendoring wasn’t and issue, but in this case I think the
>>> root of the matter is that Debian un-vendors urllib3 and we have
>>> referenced the vendored urllib3 instead of installing and utilizing
>>> urllib3 directly.
>>>
>>> This poses at least one problem for us: we are not able to guarantee
>>> we’re using the same urllib3 library as requests is. I am unsure how
>>> big of a deal this ends up being, but it is a concern and has brought
>>> up a question of how to handle this in the most appropriate and
>>> consistent way across all of the distributions we as OpenStack support.
>>>
>>> Does this make requests a bad library we should toss aside for
>>> something else? Instead of being concerned with the reasons for
>>> vendoring urllib3 (or un-vendoring it) we should shift the conversation
>>> towards two questions:
>>>
>>> 1. Is it a real issue if the version of urllib3 is mismatched between
>>> our client libraries and requests?
>>> 2. If it is a real issue how are we solving it?
>>
>> The main issue is that urllib3 in requests, as other pointed out, is not
>> up-to-date, and will not be updated. In fact, that's the main reason why
>> the upstream authors of requests are vendorizing: it's because they
>> don't want to carry the burden of staying up-to-date.
> 
> I don’t think this is remotely true, often times requests updates itself
> to versions of urllib3 which aren’t even released yet. Sometimes urllib3
> might make commits and do a release that happens between requests
> versions though. I mean technically they might be not up to date until
> their next version release though.
> 
>>
>> And then, there's incompatibilities and divergences that appear, leading
>> to all sorts of unexpected issues, like one thing working with pip, but
>> not with the packages. This kind of issues are very hard to understand
>> and debug. Distributions may report the issue upstream, then upstream
>> will say "but it's working for me", and then we may loose a lot of time.
>> This happened already, and may happen again if we don't care enough.
> 
> I think this is bound to happen anytime you have downstream modifying
> things. It happens in pip (pip vendors things too) and yea it’s quite
> annoying
> but part of PEP 440 is providing ways for downstream to signal they’ve
> modified things so that instead of “foo 1.0” you have “foo 1.0+ubuntu1” or
> whatever.
> 
>>
>>> Obviously we can work with the requests team to figure out the best
>>> approach.
>>
>> There's only a single approach that works: have the requests upstream
>> authors to stop embedding foreign code, and use the dependency instead.
> 
> There are legitimate reasons for a project to vendor things. Linux
> distributions
> are not the end be all of distribution models and they don’t get to
> dictate to
> upstream.
> 
> Generally I agree that requests should not vendor urllib3, but it’s not
> a clear
> cut thing where there is one right way to do it.
> 
>>
>>> We should focus on the solution here rather than continuing
>>> down the path of whether requests should/shouldn’t be vendoring it’s
>>> dependencies since it is clear that the team has their reasons and
>>> does not want to switch to the dependency model again.
>>
>> I'm sure they have tons of wrong reasons. If they don't want to change
>> anything, then we can only try to work around the issue, and never use
>> the embedded version.
> 
> Generally you either work with the embedded versions or you don’t
> use requests. You’re going to get very strange incompatibility problems
> if you try to mis requests.packages.urllib3 and urllib3 in one codebase
> and if you’re using requests at all it’s going to be expecting to use
> the embedded copy of urllib3.


After having gone through the whole thread and read all the concerns,
problems and reasonings, I think we should stick to requests as-is for
now and deal with this particular issue.

Regardless of the vendorized urllib3 package, I believe requests is a
good library with an easy-to-consume API and it has solved several
problems throughout OpenStack. Not to mention it's also helpped with
making OpenStack more consistent. We've put a lot of effort to get to
this point and I don't think we should revert all that because of the
vendorized `urllib3` package.

Cheers,
Flavio


-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Design Summit planning

[Thierry Carrez]

Maish Saidel-Keesing wrote:
> On 17/09/2014 23:12, Anita Kuno wrote:
>> On 09/17/2014 04:01 PM, Maish Saidel-Keesing wrote:
>>> This looks great - but I am afraid that something might be missing.
>>>
>>> As part of the Design summit in Atlanta there was an Ops Meetup track.
>>> [1] I do not see where this fits into the current planning process that
>>> has been posted.
>>> I would like to assume that part of the purpose of the summit is to also
>>> collect feedback from Enterprise Operators and also from smaller ones as
>>> well.
>>>
>>> If that is so then I would kindly request that there be some other way
>>> of allowing that part of the community to voice their concerns, and
>>> provide feedback.
>>>
>>> Perhaps a track that is not only Operator centric - but also an End-user
>>> focused one as well (mixing the two would be fine as well)
>>>
>>> Most of them are not on the openstack-dev list and they do not
>>> participate in the IRC team meetings, simply because they have no idea
>>> that these exist or maybe do not feel comfortable there. So they will
>>> not have any exposure to the process.
>>>
>>> My 0.02 Shekels.
>>>
>>> [1] - http://junodesignsummit.sched.org/overview/type/ops+meetup
>>>
>> Hi Maish:
>>
>> This thread is about the Design Summit, the Operators Track is a
>> different thing.
>>
>> In Atlanta the Operators Track was organized by Tom Fifield and I have
>> every confidence he is working hard to ensure the operators have a voice
>> in Paris and that those interested can participate.
>>
>> Last summit the Operators Track ran on the Monday and the Friday giving
>> folks who usually spend most of the time at the Design summit to
>> participate and hear the operator's voices. I know I did and I found it
>> highly educational.
>>
>> Thanks,
>> Anita.
> Thanks for the clarification Anita :)

I think the plan is to have the Ops summit run on Monday, with an extra
day on Thursday to continue the discussions. I CCed Tom for more input
on that.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[James Polley]

On Thu, Sep 18, 2014 at 5:21 PM, mar...@redhat.com 
wrote:

> On 17/09/14 16:40, Charles Crouch wrote:
> >
> >
> > - Original Message -
> >> Hi,
> >>
> >> as part of general housekeeping on our reviews, it was discussed at last
> >> week's meeting [1] that we should set workflow -1 for stale reviews
> >> (like gerrit used to do when I were a lad).
> >>
> >> The specific criteria discussed was 'items that have a -1 from a core
> >> but no response from author for 14 days'. This topic came up again
> >> during today's meeting and it wasn't clear if the intention was for
> >> cores to start enforcing this? So:
> >>
> >> Do we start setting WIP/workflow -1 for those reviews that have a -1
> >> from a core but no response from author for 14 days
> >>
> >> thanks, marios
> >>
> >> [1]
> >>
> http://eavesdrop.openstack.org/meetings/tripleo/2014/tripleo.2014-09-09-19.04.log.html
> >
> > So it looks like this has already started..
> >
> > https://review.openstack.org/#/c/105275/
> >
> > I think we need to document on the wiki *precisely* the criteria for
> setting
> > WIP/workflow -1. For example that review above has a Jenkins failure but
> no
> > core reviews at all.
>
> +1 on being precise - another case in point:
>
> https://review.openstack.org/#/c/102304/
>
> this review has a *-2* from core, which seems to 'stick' for future
> revisions; the last revision is from > 14 days ago, so does this fulfill
> the criteria (I'd say it doesn't)?
>
> Not sure if you were also suggesting that -1 from Jenkins also counts,
> which imo makes sense...
>
> 'items that have a -1 from core or jenkins, or a -2 from core on the
> latest revision, and no response from author for 14 days'
>
> Really this needs to be put as a motion and voted on in the next meeting,
>

My understanding has always been that we don't make decisions based on
votes on IRC meetings, because it's hard to get a time for the meeting that
allows the whole team to be easily present. I wouldn't feel comfortable
doing this at the US-timezone meeting as it excludes most of APAC; nor at
the EU-timezone meeting as it excludes most of the US.

If we're looking for consensus, I'd suggest we use Gerrit to collect votes.
We could model this is a change to the CONTRIBUTING.rst[1], or perhaps we
could draft a spec around the expected workflow (perhaps formalising some
of our expectations around cores averaging 3 reviews/workday + 1 spec
review/workday at the same time?

But perhaps I'm overthinking and making this far more formal than it has to
be. We've had a fair bit of discussion on the list and we seem to be
broadly in agreement; perhaps we just need someone to propose some details
and see if we can get consensus here.

[1] What's that you say? We don't have a CONTRIBUTING.rst? One second, let
me fix that *tap tap tap* We will as soon as
https://review.openstack.org/#/c/122350/ lands!


> thanks, marios
>
>
> >
> >>
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[Sullivan, Jon Paul]

> -Original Message-
> From: James E. Blair [mailto:cor...@inaugust.com]
> Sent: 17 September 2014 23:24
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] Set WIP for stale patches?
> 
> "Sullivan, Jon Paul"  writes:
> 
> > I think this highlights exactly why this should be an automated
> > process.  No errors in application, and no errors in interpretation of
> > what has happened.
> >
> > So the -1 from Jenkins was a reaction to the comment created by adding
> > the workflow -1.  This is going to happen on all of the patches that
> > have their workflow value altered (tests will run, result would be
> > whatever the result of the test was, of course).
> 
> Jenkins only runs tests in reaction to comments if they say "recheck".

This is not my experience.

In my experience if the check results are not fresh enough the recheck is 
automatically run.  I am not on the infra team, so without looking up code I am 
just guessing, but my guess is that the workflow score change triggers the 
check on the presumption that it has been approved, not accounting for the 
recent(ish) update that move wip to the workflow score.

> 
> > But I also agree that the Jenkins vote should not be included in the
> > determination of marking a patch WIP, but a human review should (So
> > Code-Review and not Verified column).
> >
> > And in fact, for the specific example to hand, the last Jenkins vote
> > was actually a +1, so as I understand it should not have been marked
> > WIP.
> 
> I'd like to help you see the reviews you want to see without
> externalizing your individual workflow onto contributors.  What tool do
> you use to find reviews?
> 
> If it's gerrit's webui, have you tried using the Review Inbox dashboard?
> Here it is for the tripleo-image-elements project:
> 
>   https://review.openstack.org/#/projects/openstack/tripleo-image-
> elements,dashboards/important-changes:review-inbox-dashboard
> 
> If you would prefer something else, we can customize those dashboards to
> do whatever you want, including ignoring changes that have not been
> updated in 2 weeks.

This is not solely about finding reviews.  It is about pruning stale reviews.  
I think the auto-abandon code was excellent at doing this, but alas, it is no 
more. 

> 
> -Jim
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Thanks, 
Jon-Paul Sullivan ☺ Cloud Services - @hpcloud

Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, Galway.
Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's 
Quay, Dublin 2. 
Registered Number: 361933
 
The contents of this message and any attachments to it are confidential and may 
be legally privileged. If you have received this message in error you should 
delete it from your system immediately and advise the sender.

To any recipient of this message within HP, unless otherwise stated, you should 
consider this message and attachments as "HP CONFIDENTIAL".
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] Persist graph for convergence

[Murugan, Visnusaran]

Hi All,

Below is the model proposal for persisting ResourceGraph.

Column name

Type

Constraint

resource_id

varchar(36)

NOT NULL ForeignKey resource.id

needed_by

varchar(36)

NULL ForeignKey resource.id

stack_id

varchar(36)

NOT NULL ForeignKey stack.id

retry_count

Integer

Default (0)

observed_ready

Boolean

Default False



Create cycle:

1.   Persist graph

2.   Mark desired states for all resources in the stack. (table: resources)

3.   Initiate stack create by calculating leaf nodes (All 
observed_ready=False nodes in resource_id column but not in needed_by column)

4.   Post create resource jobs to engine worker

5.   Engine worker upon task initiation posts observe task to 
convergence-observer and complete tack execution (return true)

6.   Mark observed_ready upon receiving resource completion notification 
from observer. Repeat step 3 through 6 till all nodes (resources) are marked 
observed_ready.

7.   If not more nodes, then mark stack create complete and return happy

Update cycle:

1.   Persist new graph for new stack created

2.   Mark desired state for all resources in both old and new stacks.

3.   Perform steps 3 to 6

4.   Delete old stack, garbage collect unwanted resources

5.   Mark stack update complete and return happy.


Clint had in fact proposed to have a "version Integer default(0)". To achieve 
this I feel Table:Stack should also have versioning instead of creating a 
backup_stack.

Currently backup and nested stack relation is achieved using owner_id. In case 
if update, versioning makes more sense instead on creating a backup stack and 
setting owner_id as current stack.

-Vishnu





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Flavio Percoco]

On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> This thread [1] has turned more “future focused", so I’m moving the 
> conversation to the -dev list where we usually have those sorts of 
> discussions.
> 
> [1] http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> 


I saw this mentioned in the `openstack` thread but I'd like us to
reconsider it.

Since we've gone through the "hey please don't deprecate it, I'll help"
process a couple of times with the zmq driver, I'm thinking we should
pull it out of the code base anyway.

Here's a rough plan of what I think we should do until the zmq is
updated and has a proper gate working:

1. Pull it out the code base into its own repo (stackforge?)
2. Setup the basic `oslo.messaging` gates for it
3. Release the driver on pypi as: `oslo.messaging.zmq`
4. Make lots of noise and make sure people using zmq knows that they
have to install a separate package now. (I know this means creating a
new package for many distros).

If there are folks interested in maintaining this driver, they can still
do it with the driver outside the code base. If it ever gets enough
attention and maturity, it could be re-proposed for inclusion into the
code base.

I know there are folks interested in a broker-less solution and we now
have the not-tested-in-production-yet amqp1.0 driver which I think is a
very good solution and also shares most of the semantics we rely on
protocol-wise.

I didn't go through the whole thread so, I'm sorry if I'm repeating
something that has already been said/proposed/discussed.

Flavio

> On Sep 17, 2014, at 7:54 AM, James Page  wrote:
> 
>> Signed PGP part
>> Hi Li
>>
>> On 17/09/14 11:58, Li Ma wrote:
 The scale potential is very appealing and is something I want to
 test - - hopefully in the next month or so.

 Canonical are interested in helping to maintain this driver and
 hopefully we help any critical issues prior to Juno release.


>>> That sounds good. I just went through all the bugs reported in the
>>> community.
>>>
>>> The only critical bug which makes ZeroMQ malfunction is
>>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
>>> corresponding review is under way:
>>> https://review.openstack.org/#/c/84938/
>>
>> Agreed
>>
>>> Others are tagged to 'zmq' in
>>> https://bugs.launchpad.net/oslo.messaging
>>
>> Looking through Doug's suggested list of information and collating
>> what I know of from our work last week:
>>
>> 1) documentation for how to configure and use zeromq with
>> oslo.messaging (note, not the version in oslo-incubator, the version
>> in the messaging library repository)
>>
>> As part of our sprint, I worked on automating deployment of OpenStack
>> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
>> that work into some general documentation on how best to configure
>> ZeroMQ with OpenStack - the current documentation is a bit raw and
>> does not talk about how to configure the oslo-messaging-zmq-receiver
>> at all.
>>
>> I also plan some packaging updates for Debian/Ubuntu in our next dev
>> cycle to make this a little easier to configure and digest - for
>> example, right now no systemd unit/upstart configuration/sysv init
>> script is provided to manage the zmq-receiver.
>>
>> I'd also like to document the current design of the ZMQ driver - Doug
>> - where is the best place todo this? I thought in the source tree
>> somewhere.
> 
> The documentation in the oslo.messaging repository [2] would be a good place 
> to start for that. If we decide deployers/operators need the information we 
> can either refer to it from the guides managed by the documentation team or 
> we can move/copy the information. How about if you start a new drivers 
> subdirectory there, and add information about zmq. We can have other driver 
> authors provide similar detail about their drivers in the same directory.
> 
> [2] http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
> 
>>
>> 2) a list of the critical bugs that need to be fixed + any existing
>> patches associated with those bugs, so they can be reviewed early in kilo
>>
>> This blocks operation of nova+neutron environements:
>>
>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723
>>  Summary: Message was sent to wrong node with zmq as rpc_backend
>>  Patch: https://review.openstack.org/84938
>>
>> Also notifcations are effectively unimplemented which prevents use
>> with Ceilometer so I'd also add:
>>
>> https://bugs.launchpad.net/oslo.messaging/+bug/1368154
>>  Summary: https://bugs.launchpad.net/oslo.messaging/+bug/
>>  Patch: https://review.openstack.org/120745
> 
> That’s a good list, and shorter than I expected. I have added these bugs to 
> the next-kilo milestone.
> 
>>
>> 3) an analysis of what it would take to be able to run functional
>> tests for zeromq on our CI infrastructure, not necessarily the full
>> tempest run or devstack-gate job, probably functional tests we

Re: [openstack-dev] [Congress] Nominating Aaron Rosen for congress-core

[Rajdeep Dua]

+1

On Fri, Sep 12, 2014 at 12:14 PM, Tim Hinrichs  wrote:

> +1
>
> Tim
>
> On Sep 12, 2014, at 11:47 AM, Sean Roberts 
> wrote:
>
> > +1
> >
> > ~sean
> >
> >> On Sep 12, 2014, at 11:28 AM, Peter Balland 
> wrote:
> >>
> >> Hello,
> >>
> >> I would like to nominate Aaron Rosen for the congress-core team.
> >>
> >> Aaron has been involved in congress for the last few months providing
> >> valuable reviews as well as leading the keystone integration and
> >> congressclient work.
> >>
> >> References:
> >>
> >>
> https://review.openstack.org/#/q/owner:arosen+project:+stackforge/congress
> ,
> >> n,z
> >>
> >>
> https://review.openstack.org/#/q/owner:arosen+project:+stackforge/python-co
> >> ngressclient,n,z
> >>
> >>
> https://review.openstack.org/#/q/reviewer:arosen+project:+stackforge/congre
> >> ss,n,z
> >>
> >> Please respond with +1's or any concerns.
> >>
> >> Thanks,
> >>
> >> Peter
> >>
> >>
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Persist graph for convergence

[Gurjar, Unmesh]

Additions/comments inline below.

Thanks,
Unmesh G.
From: Murugan, Visnusaran
Sent: Thursday, September 18, 2014 1:46 PM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [heat] Persist graph for convergence

Hi All,

Below is the model proposal for persisting ResourceGraph.

Column name

Type

Constraint

resource_id

varchar(36)

NOT NULL ForeignKey resource.id

needed_by

varchar(36)

NULL ForeignKey resource.id

stack_id

varchar(36)

NOT NULL ForeignKey stack.id

retry_count

Integer

Default (0)

observed_ready

Boolean

Default False



Create cycle:

1.   Persist graph
[Unmesh] - So, it is a pre-requisite to persist all the resources of a stack 
(in the resource table) to be able to persist a graph (i.e step #0).

2.   Mark desired states for all resources in the stack. (table: resources)
[Unmesh] - This I believe should be done as a part of persisting a stack's 
resources (in step #0).

3.   Initiate stack create by calculating leaf nodes (All 
observed_ready=False nodes in resource_id column but not in needed_by column)

4.   Post create resource jobs to engine worker

5.   Engine worker upon task initiation posts observe task to 
convergence-observer and complete tack execution (return true)

6.   Mark observed_ready upon receiving resource completion notification 
from observer. Repeat step 3 through 6 till all nodes (resources) are marked 
observed_ready.
[Unmesh] - In case resource creation fails, the worker will increment 
retry_count and try creating the resource for a maximum of 'action_retry_limit' 
configured value.

7.   If not more nodes, then mark stack create complete and return happy

Update cycle:

1.   Persist new graph for new stack created

2.   Mark desired state for all resources in both old and new stacks.

3.   Perform steps 3 to 6

4.   Delete old stack, garbage collect unwanted resources

5.   Mark stack update complete and return happy.


Clint had in fact proposed to have a "version Integer default(0)". To achieve 
this I feel Table:Stack should also have versioning instead of creating a 
backup_stack.

Currently backup and nested stack relation is achieved using owner_id. In case 
if update, versioning makes more sense instead on creating a backup stack and 
setting owner_id as current stack.

-Vishnu





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][Architecture]Suggestions for the third vendors' plugin and driver

[Germy Lure]

Hi Trinath,

I think the vendor company has many experts to review their codes. They can
do it well.

But I still have some comments inline.

Germy

On Thu, Sep 18, 2014 at 1:42 PM, trinath.soman...@freescale.com <
trinath.soman...@freescale.com> wrote:

>  Though Code reviews for vendor code takes more time, I feel it must go
> through Core reviews.
>
>
>
> Since, Vendors might submit the code that is working fine within their
> third party CI environment but the Code review make it more efficient with
> respect to the coding standards followed in the community.
>
>
>
> Also, for all the vendor plugins/drivers the code reviews (+1s and +2s)
> give a feedback on the quality they must be in to be with Neutron.
>
I think the quality of a software mainly lies on developers, otherwise
reviewers will be very very busy.
We suppose that all core members reviewed your plugin and gave feedback
many +, so can you guarantee the plugin high quality? even no BUGs?
I think only the vendor, cooperating with customer and providing plugin and
driver, can and must guarantee the quality. But those *private* releases
only exist in vendor's disk and running in customer's machine. It cannot be
updated to community because of approving waiting, because of not efficient
enough, because of the coding standards, 

>
>
> But one suggestion I want to put forward, when an -1 or -2 is given to the
> code, Reviewers might give a brief comment on why this was given, what
> might be preferred solution and Is there any reference implementation that
> can be considered for the code in review to move away from these errors.
> This can help the developers.
>
If core members prefer Cisco's implementation, all the other vendors follow
it? Why different plugins? Only one is enough.
Of course, this is a very extreme assumption. We just discuss a problem.

>
>
>
>
>
>
>
>
> --
>
> Trinath Somanchi - B39208
>
> trinath.soman...@freescale.com | extn: 4048
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] DVR Tunnel Design Question

[Narasimhan, Vivekanandan]

Yes... If I recollect, we were using 1.10.x version during that time (wherein 
discovered this 
as output of ovsapp-ctl fdb-show). After that I didn’t get time to re-verify on 
later
versions of openvswitch.

BTW, if this is not intended behaviour, then I donot see any particular reason 
why VLANs
need not be enabled in the current DVR architecture.  

To enable dvr, I need to post a minor patch in L2 agent to bring-in 
DVR rules into Phys bridges (as VLANs are driven by phys bridges 
in OVS L2 Agent).

--
Thanks,

Vivek

-Original Message-
From: Kevin Benton [mailto:blak...@gmail.com] 
Sent: Thursday, September 18, 2014 12:44 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question

This sounds like a bug in Openvswitch. The unique constraint should be
VLAN+MAC instead of just VLAN, so observing the same MAC on a new VLAN
should just result in a new entry without the old one being ejected.

Without this behavior, it will also break transparent L2 firewalls.
For example (diagram below), if you divide a switch into two VLANs and then 
connect the sides together with a firewall in the middle that passes frames 
without modifying the MAC, the switch will see the same MAC on different VLANs. 
Based on the behavior you described, this wouldn't work. Is that correct?


-
|  x  x  x  x   |   x  x  x  x  |
|---|-
 VLAN 1 | |  VLAN2
  
  |   Firewall  |
  

On Wed, Sep 17, 2014 at 11:19 PM, Narasimhan, Vivekanandan 
 wrote:
> Hi Kevin,
>
>
>
> Typically we noticed that the underlay switches maintained a table 
> like
> this:
>
>
>
> VLAN IDMAC Address  Learned-Interface
>
>
>
> In the physical underlay, with the current architecture if we enable 
> VLAN, the same DVR Unique
>
> MAC will appear  on different VLANs as the packets get DVR Routed.
>
>
>
> This will result in the rows of the above tables in the switch to be 
> updated very frequently with new
>
> VLANs noted in incoming packets for the same DVR MAC Address, even 
> though they are from the
>
> same physical port.
>
>
>
> We are not sure if all the switches maintained the tables this way , 
> but atleast we
>
> saw Openvswitch implementations did.  So we consciously did not 
> promote VLANs for
>
> initial phase of DVR.
>
>
>
> --
>
> Thanks,
>
>
>
> Vivek
>
>
>
>
>
> From: Kevin Benton [mailto:blak...@gmail.com]
> Sent: Thursday, September 18, 2014 3:01 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question
>
>
>
> Can you clarify what you mean with the thrashing condition? MAC 
> addresses only need to be unique per-VLAN so I don't see how the same 
> MAC on multiple VLANs from the same physical port would lead to any issues.
>
>
>
> On Wed, Sep 17, 2014 at 12:41 PM, Armando M.  wrote:
>
> VLAN is on the radar, vxlan/gre was done to start with.
>
> I believe Vivek mentioned the rationale in some other thread. The gist 
> of it below:
>
> In the current architecture, we use a unique DVR MAC per compute node 
> to forward DVR Routed traffic directly to destination compute node.
> The DVR routed traffic from the source compute node will carry 
> 'destination VMs underlay VLAN' in the frame, but the Source Mac in 
> that same frame will be the DVR Unique MAC. So, same DVR Unique Mac is 
> used for potentially a number of overlay network VMs that would exist 
> on that same source compute node.
>
> The underlay infrastructure switches will see the same DVR Unique MAC 
> being associated with different VLANs on incoming frames, and so this 
> would result in VLAN Thrashing on the switches in the physical cloud 
> infrastructure. Since tunneling protocols carry the entire DVR routed 
> inner frames as tunnel payloads, there is no thrashing effect on 
> underlay switches.
>
> There will still be thrashing effect on endpoints on CNs themselves, 
> when they try to learn that association between inner frame source MAC 
> and the TEP port on which the tunneled frame is received. But that we 
> have addressed in L2 Agent by having a 'DVR Learning Blocker' table, 
> which ensures that learning for DVR routed packets alone is 
> side-stepped.
>
> As a result, VLAN was not promoted as a supported underlay for the 
> initial DVR architecture.
>
> Cheers,
> Armando
>
>
> On 16 September 2014 20:35, 龚永生  wrote:
>> I think the VLAN should also be supported later.  The tunnel should 
>> not be the prerequisite for the DVR feature.
>>
>>
>> -- Original --
>> From:  "Steve Wormley";
>> Date:  Wed, Sep 17, 2014 10:29 AM
>> To:  "openstack-dev";
>> Subject:  [openstack-dev] [neutron] DVR Tunnel Design Question
>>
>> In our environment using VXLAN/GRE would make it difficult to keep 
>> some of the features we currently offer our customers. So for a while

[openstack-dev] [nova] Nova API meeting

[Christopher Yeoh]

Hi,

Just a reminder that the weekly Nova API meeting is being held tomorrow
Friday UTC . 

We encourage cloud operators and those who use the REST API such as
SDK developers and others who and are interested in the future of the
API to participate.

In other timezones the meeting is at:

EST 20:00 (Thu)
Japan 09:00 (Fri)
China 08:00 (Fri)
ACDT 9:30 (Fri)

The proposed agenda and meeting details are here: 

https://wiki.openstack.org/wiki/Meetings/NovaAPI

Please feel free to add items to the agenda. 

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat][Zaqar] Integration plan moving forward

[Flavio Percoco]

Greetings,

If I recall correctly, Heat was planning to adopt Zaqar regardless of
the result of the graduation attempt (please correct me if I'm wrong).
Based on this assumption, I'd like to start working on a plan forward to
make this integration happen.

So far, these are the use cases I've collected from past discussions:

* Notify  heat user before an action is taken, and after - Heat may want
to wait  for a response before proceeding - notifications not
necessarily needed  and signed read-only queues might help, but not
necessary
* For integrating with user's tools
* Monitoring
* Control surface
* Config management tools
* Does not require notifications and/or read-only/signed queue endpoints
*[These may be helpful, but were not brought up in the discussion]
* Subscribe to an aggregate feed of interesting events from other
open-stack components (such as Nova)
* Heat is often deployed in a different place than other
components and doesn't have access to the AMQP bus
* Large  deployments consist of multiple AMQP brokers, and there
doesn't seem to  be a nice way to aggregate all those events [need to
confirm]
* Push metadata updates to os-collect-config agent running in
servers, instead of having them poll Heat


Few questions that I think we should start from:

- Does the above list cover Heat's needs?
- Which of the use cases listed above should be addressed first?
- Can we split the above into milestones w/ due dates?


Thanks,
Flavio

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [MagnetoDB] Core developer nomination

[Dmitriy Ukhlov]

+1 from me, Charles is very active contributor and I guess if we have such
developer in core team
MagnetoDB project will become much better that it is now.

On Thu, Sep 18, 2014 at 10:09 AM, Illia Khudoshyn 
wrote:

> Congrats, Charles! Great job!
>
> On Thu, Sep 18, 2014 at 12:05 AM, Ilya Sviridov 
> wrote:
>
>> Hello magnetodb contributors,
>>
>> I'm glad to nominate Charles Wang to core developers of MagnetoDB.
>>
>> He is top non-core reviewer [1], implemented notifications [2] in mdb and
>> made a great progress with performance, stability and scalability testing
>>  of MagnetoDB
>>
>> [1] http://stackalytics.com/report/contribution/magnetodb/90
>> [2]
>> https://blueprints.launchpad.net/magnetodb/+spec/magnetodb-notifications
>>
>> Welcome to team, Charles!
>> Looking forward for your contribution
>>
>> --
>> Ilya Sviridov
>> isviridov @ FreeNode
>>
>
>
>
> --
>
> Best regards,
>
> Illia Khudoshyn,
> Software Engineer, Mirantis, Inc.
>
>
>
> 38, Lenina ave. Kharkov, Ukraine
>
> www.mirantis.com 
>
> www.mirantis.ru
>
>
>
> Skype: gluke_work
>
> ikhudos...@mirantis.com
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Best regards,
Dmitriy Ukhlov
Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] DVR Tunnel Design Question

[Kevin Benton]

>To enable dvr, I need to post a minor patch in L2 agent to bring-in
DVR rules into Phys bridges (as VLANs are driven by phys bridges
in OVS L2 Agent).

Great! You read my mind and answered my follow-up question. :-)
Let me know if there is anything I can help with.

On Thu, Sep 18, 2014 at 1:51 AM, Narasimhan, Vivekanandan
 wrote:
> Yes... If I recollect, we were using 1.10.x version during that time (wherein 
> discovered this
> as output of ovsapp-ctl fdb-show). After that I didn’t get time to re-verify 
> on later
> versions of openvswitch.
>
> BTW, if this is not intended behaviour, then I donot see any particular 
> reason why VLANs
> need not be enabled in the current DVR architecture.
>
> To enable dvr, I need to post a minor patch in L2 agent to bring-in
> DVR rules into Phys bridges (as VLANs are driven by phys bridges
> in OVS L2 Agent).
>
> --
> Thanks,
>
> Vivek
>
> -Original Message-
> From: Kevin Benton [mailto:blak...@gmail.com]
> Sent: Thursday, September 18, 2014 12:44 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question
>
> This sounds like a bug in Openvswitch. The unique constraint should be
> VLAN+MAC instead of just VLAN, so observing the same MAC on a new VLAN
> should just result in a new entry without the old one being ejected.
>
> Without this behavior, it will also break transparent L2 firewalls.
> For example (diagram below), if you divide a switch into two VLANs and then 
> connect the sides together with a firewall in the middle that passes frames 
> without modifying the MAC, the switch will see the same MAC on different 
> VLANs. Based on the behavior you described, this wouldn't work. Is that 
> correct?
>
>
> -
> |  x  x  x  x   |   x  x  x  x  |
> |---|-
>  VLAN 1 | |  VLAN2
>   
>   |   Firewall  |
>   
>
> On Wed, Sep 17, 2014 at 11:19 PM, Narasimhan, Vivekanandan 
>  wrote:
>> Hi Kevin,
>>
>>
>>
>> Typically we noticed that the underlay switches maintained a table
>> like
>> this:
>>
>>
>>
>> VLAN IDMAC Address  Learned-Interface
>>
>>
>>
>> In the physical underlay, with the current architecture if we enable
>> VLAN, the same DVR Unique
>>
>> MAC will appear  on different VLANs as the packets get DVR Routed.
>>
>>
>>
>> This will result in the rows of the above tables in the switch to be
>> updated very frequently with new
>>
>> VLANs noted in incoming packets for the same DVR MAC Address, even
>> though they are from the
>>
>> same physical port.
>>
>>
>>
>> We are not sure if all the switches maintained the tables this way ,
>> but atleast we
>>
>> saw Openvswitch implementations did.  So we consciously did not
>> promote VLANs for
>>
>> initial phase of DVR.
>>
>>
>>
>> --
>>
>> Thanks,
>>
>>
>>
>> Vivek
>>
>>
>>
>>
>>
>> From: Kevin Benton [mailto:blak...@gmail.com]
>> Sent: Thursday, September 18, 2014 3:01 AM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [neutron] DVR Tunnel Design Question
>>
>>
>>
>> Can you clarify what you mean with the thrashing condition? MAC
>> addresses only need to be unique per-VLAN so I don't see how the same
>> MAC on multiple VLANs from the same physical port would lead to any issues.
>>
>>
>>
>> On Wed, Sep 17, 2014 at 12:41 PM, Armando M.  wrote:
>>
>> VLAN is on the radar, vxlan/gre was done to start with.
>>
>> I believe Vivek mentioned the rationale in some other thread. The gist
>> of it below:
>>
>> In the current architecture, we use a unique DVR MAC per compute node
>> to forward DVR Routed traffic directly to destination compute node.
>> The DVR routed traffic from the source compute node will carry
>> 'destination VMs underlay VLAN' in the frame, but the Source Mac in
>> that same frame will be the DVR Unique MAC. So, same DVR Unique Mac is
>> used for potentially a number of overlay network VMs that would exist
>> on that same source compute node.
>>
>> The underlay infrastructure switches will see the same DVR Unique MAC
>> being associated with different VLANs on incoming frames, and so this
>> would result in VLAN Thrashing on the switches in the physical cloud
>> infrastructure. Since tunneling protocols carry the entire DVR routed
>> inner frames as tunnel payloads, there is no thrashing effect on
>> underlay switches.
>>
>> There will still be thrashing effect on endpoints on CNs themselves,
>> when they try to learn that association between inner frame source MAC
>> and the TEP port on which the tunneled frame is received. But that we
>> have addressed in L2 Agent by having a 'DVR Learning Blocker' table,
>> which ensures that learning for DVR routed packets alone is
>> side-stepped.
>>
>> As a result, VLAN was not promoted as a supported underlay for the
>> initial DVR architecture.
>>
>> Cheers,
>> Armando
>>
>>
>> On 16 September 2014 20:35

[openstack-dev] [Neutron] Gate bugs for RC-1

[Salvatore Orlando]

Bug 1357055 [1] and 1323658 [2] affect neutron jobs and are among the top
gate offenders.
With this kind of bugs, it's hard to tell whether the root cause lies with
neutron, nova, tempest, or even cirros.
However, it is not ok that these bugs are not assigned in neutron. We need
to have some neutron developers' eyes on them and root cause them as soon
as possible.

If you wish to volunteer, please assign one of these bugs (or both of them)
to yourself. Please note that this might be a rather time consuming
activity.

Salvatore

[1] https://bugs.launchpad.net/neutron/+bug/1357055
[2] https://bugs.launchpad.net/neutron/+bug/1323658
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][db] Ensuring db isolation between tests

[Maru Newby]

For legacy reasons, the Neutron test suite creates and destroys a db for each 
test.  There is a patch proposed to create the tables once and then ensure the 
tables are wiped at the end of each test [1], providing a performance 
improvement of ~10%.  I was wondering if this is the best way to provide 
isolation, since I’ve heard that isolation via per-test transactions should 
also work.  The patch author reported problems with this approach - apparently 
nested commits were not being rolled back.  Is there some trick to isolating 
with transactions that wouldn’t be immediately obvious?

Thanks,


Maru

1: https://review.openstack.org/#/c/122028/
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][Zaqar] Integration plan moving forward

[Angus Salkeld]

On 18/09/2014 7:11 PM, "Flavio Percoco"  wrote:
>
> Greetings,
>
> If I recall correctly, Heat was planning to adopt Zaqar regardless of
> the result of the graduation attempt (please correct me if I'm wrong).
> Based on this assumption, I'd like to start working on a plan forward to
> make this integration happen.
>
> So far, these are the use cases I've collected from past discussions:
>
> * Notify  heat user before an action is taken, and after - Heat may want
> to wait  for a response before proceeding - notifications not
> necessarily needed  and signed read-only queues might help, but not
> necessary
> * For integrating with user's tools
> * Monitoring
> * Control surface
> * Config management tools
> * Does not require notifications and/or read-only/signed queue
endpoints
> *[These may be helpful, but were not brought up in the discussion]
> * Subscribe to an aggregate feed of interesting events from other
> open-stack components (such as Nova)
> * Heat is often deployed in a different place than other
> components and doesn't have access to the AMQP bus
> * Large  deployments consist of multiple AMQP brokers, and there
> doesn't seem to  be a nice way to aggregate all those events [need to
> confirm]
> * Push metadata updates to os-collect-config agent running in
> servers, instead of having them poll Heat
>
>
> Few questions that I think we should start from:
>
> - Does the above list cover Heat's needs?
> - Which of the use cases listed above should be addressed first?

IMHO it would be great to simply replace the event store we have currently,
so that the user can get a stream of progress messages during the
deployment.

-Angus

> - Can we split the above into milestones w/ due dates?
>
>
> Thanks,
> Flavio
>
> --
> @flaper87
> Flavio Percoco
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][db] Ensuring db isolation between tests

[Salvatore Orlando]

Nested commits in sqlalchemy should be seen as a single transaction on the
backend, shouldn't they?
I don't know anything about this specific problem, but the fact that unit
tests use sqlite might be a reason, since it's not really a full DBMS...

I think that wrapping tests in transaction also will require some changes
in the architecture of the tests themselves, as many tests call the API
router or the plugin which then gets a db session and open a new
transaction. Furthermore, it will change the test behaviour possibly hiding
errors; some operations indeed perform several distinct transactions, which
in this case will be seen a single transaction.

What Kevin is doing here I think was the original way we used to do that in
Neutron (Folsom). Then at some point we realised that due to plugin schema
differences we were laving tables behind and switched to drop_all and
rebuilding the schema using autogeneration at each test.

I think it should be ok to merge this patch. I will hold off the +A to give
other core reviewers a chance to look at it.

Salvatore


On 18 September 2014 11:44, Maru Newby  wrote:

> For legacy reasons, the Neutron test suite creates and destroys a db for
> each test.  There is a patch proposed to create the tables once and then
> ensure the tables are wiped at the end of each test [1], providing a
> performance improvement of ~10%.  I was wondering if this is the best way
> to provide isolation, since I’ve heard that isolation via per-test
> transactions should also work.  The patch author reported problems with
> this approach - apparently nested commits were not being rolled back.  Is
> there some trick to isolating with transactions that wouldn’t be
> immediately obvious?
>
> Thanks,
>
>
> Maru
>
> 1: https://review.openstack.org/#/c/122028/
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Alan Pevec]

2014-09-17 17:15 GMT+02:00 Thomas Goirand :
>   File "/tests/test_ssl.py", line 19, in 
> from requests.packages.urllib3 import poolmanager
> ImportError: No module named packages.urllib3

This is in tests only, in runtime code there is conditional import of
vendorized urllib3 falling back to system library.
So what about https://review.openstack.org/122379

Cheers,
Alan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Weekly IRC: let's cancel the meeting tomorrow

[Vladimir Kozhukalov]

+1 for canceling on 9/18/2014.

As for 9/25/2014, let's make a decision later.

Vladimir Kozhukalov

On Thu, Sep 18, 2014 at 9:33 AM, Mike Scherbakov 
wrote:

> Hi Fuelers,
> as we have mini-summit in CA and majority of leads is busy with it, let's
> cancel the meeting tomorrow.
>
> We might want to cancel the meeting on 25th as well.
> Thanks,
> --
> Mike Scherbakov
> #mihgen
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] are we going to remove the novaclient v3 shell or what?

[Day, Phil]

> -Original Message-
> From: Kenichi Oomichi [mailto:oomi...@mxs.nes.nec.co.jp]
> Sent: 18 September 2014 02:44
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [nova] are we going to remove the novaclient
> v3 shell or what?
> 
> 
> > -Original Message-
> > From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> > Sent: Wednesday, September 17, 2014 11:59 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: [openstack-dev] [nova] are we going to remove the novaclient v3
> shell or what?
> >
> > This has come up a couple of times in IRC now but the people that
> > probably know the answer aren't available.
> >
> > There are python-novaclient patches that are adding new CLIs to the v2
> > (v1_1) and v3 shells, but now that we have the v2.1 API (v2 on v3) why
> > do we still have a v3 shell in the client?  Are there plans to remove that?
> >
> > I don't really care either way, but need to know for code reviews.
> >
> > One example: [1]
> >
> > [1] https://review.openstack.org/#/c/108942/
> 
> Sorry for a little late response,
> I think we don't need new features of v3 into novaclient anymore.
> For example, the v3 part of the above[1] was not necessary because a new
> feature server-group quota is provided as v2 and v2.1, not v3.

That would be true if there was a version of the client that supported v2.1 
today, but while the V2.1 API is still presented as V3 and doesn't include the 
tenant_id - making the V3 client the only simple way to test new V2.1 features 
in devstack as far as I can see.


How about this as a plan:

1) We add support to the client for "--os-compute-api-version=v2.1"   which 
maps into the client with the URL set to include v2.1(this won't be usable 
until we do step 2)

2) We change the Nova  to present the v2.1 API  as 
'http://X.X.X.X:8774/v2.1//
 - At this point we will have a working client for all of the stuff that's been 
moved back from V3 to V2.1, but will lose access to any V3 stuff not yet moved 
(which is the opposite of the current state where the v3 client can only be 
used for things that haven't been refactored to V2.1)

3) We remove V3 from the client.


Until we get 1 & 2 done, to me it still makes sense to allow small changes into 
the v3 client, so that we keep it usable with the V2.1 API



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Christopher Yeoh]

On Sat, 13 Sep 2014 06:48:19 -0400
Sean Dague  wrote:

> On 09/13/2014 02:28 AM, Kenichi Oomichi wrote:
> > 
> > Hi Chris,
> > 
> > Thanks for bring it up here,
> > 
> >> -Original Message-
> >> From: Chris St. Pierre [mailto:stpie...@metacloud.com]
> >> Sent: Saturday, September 13, 2014 2:53 AM
> >> To: openstack-dev@lists.openstack.org
> >> Subject: [openstack-dev] [nova] Expand resource name allowed
> >> characters
> >>
> >> We have proposed that the allowed characters for all resource
> >> names in Nova (flavors, aggregates, etc.) be expanded to all
> >> printable unicode characters and horizontal spaces:
> >> https://review.openstack.org/#/c/119741
> >>
> >> Currently, the only allowed characters in most resource names are
> >> alphanumeric, space, and [.-_].
> >>
> >>
> >> We have proposed this change for two principal reasons:
> >>
> >> 1. We have customers who have migrated data forward since Essex,
> >> when no restrictions were in place, and thus have characters in
> >> resource names that are disallowed in the current version of
> >> OpenStack. This is only likely to be useful to people migrating
> >> from Essex or earlier, since the current restrictions were added
> >> in Folsom.
> >>
> >> 2. It's pretty much always a bad idea to add unnecessary
> >> restrictions without a good reason. While we don't have an
> >> immediate need to use, for example, the ever-useful
> >> http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
> >> up with a reason people *shouldn't* be allowed to use it.
> >>
> >> That said, apparently people have had a need to not be allowed to
> >> use some characters, but it's not clear why:
> >> https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
> >> knows any reason why these printable characters should not be
> >> joined in holy resource naming, speak now or forever hold your
> >> peace.
> > 
> > I also could not find the reason of current restriction on the bug
> > report, and I'd like to know it as the history.
> > On v2 API(not v2.1), each resource name contains the following
> > restriction for its name:
> > 
> >   Resource  | Length  | Pattern
> >  ---+-+--
> >   aggregate | 1-255   | nothing
> >   backup| nothing | nothing
> >   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
> > | |   [a-zA-Z0-9. _-]*$'
> >   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
> >   server| 1-255   | nothing
> >   cell  | 1-255   | don't contain "." and "!"
> > 
> > On v2.1 API, we have applied the same restriction rule[1] for whole
> > resource names for API consistency, so maybe we need to consider
> > this topic for whole names.
> > 
> > [1]:
> > https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44
> 
> Honestly, I bet this had to do with how the database used to be set
> up.
> 

So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
multibyte characters (only 1-3 bytes). For example if you do a create
image call with an image name to glance with a 4 byte multibyte
character in the name it will 500. I'd guess we do something
similar in places with the Nova API where we have inadequate input
validation. If you want 4 byte support you need to use utf8mb4 instead.

I don't know if postgresql has any restrictions (I don't think it
does) or if db2 does too. But I don't think we can/should make it a
complete free for all. It should at most be what most databases support.

I think its a big enough change that this late in the cycle we should
push it off to Kilo. It's always much easier to loosen input validation
than tighten it (or have to have an "oops" revert on an officially
released Nova). Perhaps some tests to verify that everything we allow
past the input validation checks we can actually store.


> That being said, i'm pro letting names be 'utf8'. The id fields that
> are strings (like flavor_id) I think we should keep constrained, as
> we do actually do joins on them. (And as jay said, the current utf8
> schema is actually highly inefficient and bloaty.)

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] PostgreSQL jobs slow in the gate

[Sean Dague]

On 09/17/2014 11:50 PM, Clark Boylan wrote:
> On Wed, Sep 17, 2014, at 06:48 PM, Clark Boylan wrote:
>> On Wed, Sep 17, 2014, at 06:37 PM, Matt Riedemann wrote:
>>>
>>>
>>> On 9/17/2014 7:59 PM, Ian Wienand wrote:
 On 09/18/2014 09:49 AM, Clark Boylan wrote:
> Recent sampling of test run times shows that our tempest jobs run
> against clouds using PostgreSQL are significantly slower than jobs run
> against clouds using MySQL.

 FYI There is a possibly relevant review out for max_connections limits
 [1], although it seems to have some issues with shmem usage

 -i

 [1] https://review.openstack.org/#/c/121952/

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

>>>
>>> That's a backport of a fix from master where we were hitting fatal 
>>> errors due to too many DB connections which was brought on by the 
>>> changes to cinder and glance to run as many workers as there were CPUs 
>>> available.  So I don't think it probably plays here...
>>>
>>> The errors pointed out in another part of the thread have been around 
>>> for awhile, I think they are due to negative tests where we're hitting 
>>> unique constraints because of the negative tests, so they are expected.
>>>
>>> We should also note that the postgresql jobs run with the nova metadata 
>>> API service, I'm not sure how much of a factor that would have here.
>>>
>>> Is there anything else unique about those jobs from the MySQL ones?
>>>
>> Good question. There are apparently other differences. The postgres job
>> runs Keystone under eventlet instead of via apache mod_wsgi. It also
>> sets FORCE_CONFIGDRIVE=False instead of always. And the final difference
>> I can find is the one you point out, nova api metadata service is run as
>> an independent thing.
>>
>> Could these things be related? It would be relatively simple to push a
>> change or two to devstack-gate to test this but there are enough options
>> here that I probably won't do that until we think at least one of these
>> options is at fault.
> I am starting to feel bad that I picked on PostgreSQL and completely
> forgot that there were other items in play here. I went ahead and
> uploaded [0] to run all devstack jobs without keystone wsgi services
> (eventlet) and [1] to run all devstack job with keystone wsgi services
> and the initial results are pretty telling.
> 
> It appears that keystone eventlet is the source of the slowness in this
> job. With keystone eventlet all of the devstack jobs are slower and with
> keystone wsgi all of the jobs are quicker. Probably need to collect a
> bit more data but this doesn't look good for keystone eventlet.
> 
> Thank you Matt for pointing me in this direction.
> 
> [0] https://review.openstack.org/#/c/122299/
> [1] https://review.openstack.org/#/c/122300/

Don't feel bad. :)

The point that Clark highlights here is a good one. There is an
assumption that once someone creates a job in infra, the magic elves are
responsible for it.

But there are no magic elves. So jobs like this need sponsors.

Maybe the right thing to do is not conflate this configuration and put
an eventlet version of the keystone job only on keystone (because the
keystone team was the one that proposed having a config like that, but
it's so far away from their project they aren't ever noticing when it's
regressing).

Same issue with the metadata server split. That's really only a thing
Nova cares about. It shouldn't impact anyone else.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Infra][Cinder] Coraid CI system

[Mykola Grygoriev]

Dear community members,

Please have a look to Coraid CI system -
http://38.111.159.9:8080/job/CoraidCI/

We have done all requirements for Third Party CI system, provided here -
http://ci.openstack.org/third_party.html#requirements

Please look at Coraid third-party system one more time and, please, show us
what we have to add or improve in order to get voting rights for gerrit
user coraid-ci.

On Fri, Sep 12, 2014 at 3:15 PM, Roman Bogorodskiy <
rbogorods...@mirantis.com> wrote:

> Hi,
>
> Mykola has some problems sending emails to the list, so he asked me to
> post a response, here it goes:
>
> ---
> Remy, I have improved Coraid CI system and added logs of all components of
> devstack. Please have a look:
>
> http://38.111.159.9:8080/job/Coraid_CI/164/
>
> According to the requirements from
> http://ci.openstack.org/third_party.html#requesting-a-service-account ,
> Gerrit plugin from Jenkins should be given the following options:
>
> Successful: gerrit approve , --message 'Build
> Successful ' --verified  --code-review
> 
> Failed: gerrit approve , --message 'Build Failed
> ' --verified  --code-review 
> Unstable: gerrit approve , --message 'Build Unstable
> ' --verified  --code-review 
>
> I configured gerrit plugin this way, so it sends the following comment
> after checking patchset or comment with "recheck". For example,
> https://review.openstack.org/#/c/120907/
>
> Patch Set 1:
>
> Build Successful
>
> http://38.111.159.9:8080/job/Coraid_CI/164/ : SUCCESS
>
>
> All logs are on this page. They are there as artifacts.
>
> > I took a quick look and I don’t see which test cases are being run?
> We test Coraid Cinder driver with standard tempest tests using
> ./driver_certs/cinder_driver_cert.sh script. Test cases are in the log of
> job.
>
> Please look at Coraid third-party system one more time and, please, show us
> what we have to add or improve in order to get voting rights for gerrit
> user coraid-ci.
>
> Also I have set gerrit plugin on our Jenkins to the silent mode as you
> suggested.
>
> Thank you in advance.
> 
>
> On Fri, Sep 5, 2014 at 7:34 PM, Asselin, Ramy  wrote:
>
>> -1 from me (non-cinder core)
>>
>> It very nice to see you're making progress. I, personally, was very
>> confused about voting.
>> Here's my understanding: "Voting": it is the ability to provide an
>> official +1 -1 vote in the gerrit system.
>>
>> I don't see a "stable history" [1]. Before requesting voting, you should
>> enable your system on the cinder project itself.
>> Initially, you should disable ALL gerrit comments, i.e. run in silent
>> mode, per request from cinder PTL [2]. Once stable there, you can enable
>> gerrit comments. At this point, everyone can see pass/fail comments with a
>> vote=0.
>> Once stable there on real patches, you can request voting again, where
>> the pass/fail would vote +1/-1.
>>
>> Ramy
>> [1] http://38.111.159.9:8080/job/Coraid_CI/35/console
>> [2]
>> http://lists.openstack.org/pipermail/openstack-dev/2014-August/043876.html
>>
>>
>> -Original Message-
>> From: Duncan Thomas [mailto:duncan.tho...@gmail.com]
>> Sent: Friday, September 05, 2014 7:55 AM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [Infra][Cinder] Coraid CI system
>>
>> +1 from me (Cinder core)
>>
>>
>> On 5 September 2014 15:09, Mykola Grygoriev 
>> wrote:
>> > Hi,
>> >
>> > My name is Mykola Grygoriev and I'm engineer who currently working on
>> > deploying 3d party CI for Сoraid Сinder driver.
>> >
>> > Following instructions on
>> >
>> > http://ci.openstack.org/third_party.html#requesting-a-service-account
>> >
>> > asking for adding gerrit CI account (coraid-ci) to the Voting
>> > Third-Party CI Gerrit group.
>> >
>> >
>> >
>> > We have already added description of Coraid CI system to wiki page -
>> > https://wiki.openstack.org/wiki/ThirdPartySystems/Coraid_CI
>> >
>> > We used openstack-dev/sandbox project to test current CI
>> > infrastructure with OpenStack Gerrit system. Please find our history
>> there.
>> >
>> > Please have a look to results of Coraid CI system. it currently takes
>> > updates from openstack/cinder project:
>> > http://38.111.159.9:8080/job/Coraid_CI/32/
>> > http://38.111.159.9:8080/job/Coraid_CI/33/
>> >
>> > Thank you in advance.
>> >
>> > --
>> > Best regards,
>> > Mykola Grygoriev
>> >
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>>
>>
>> --
>> Duncan Thomas
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
___

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Sean Dague]

On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
> On Sat, 13 Sep 2014 06:48:19 -0400
> Sean Dague  wrote:
> 
>> On 09/13/2014 02:28 AM, Kenichi Oomichi wrote:
>>>
>>> Hi Chris,
>>>
>>> Thanks for bring it up here,
>>>
 -Original Message-
 From: Chris St. Pierre [mailto:stpie...@metacloud.com]
 Sent: Saturday, September 13, 2014 2:53 AM
 To: openstack-dev@lists.openstack.org
 Subject: [openstack-dev] [nova] Expand resource name allowed
 characters

 We have proposed that the allowed characters for all resource
 names in Nova (flavors, aggregates, etc.) be expanded to all
 printable unicode characters and horizontal spaces:
 https://review.openstack.org/#/c/119741

 Currently, the only allowed characters in most resource names are
 alphanumeric, space, and [.-_].


 We have proposed this change for two principal reasons:

 1. We have customers who have migrated data forward since Essex,
 when no restrictions were in place, and thus have characters in
 resource names that are disallowed in the current version of
 OpenStack. This is only likely to be useful to people migrating
 from Essex or earlier, since the current restrictions were added
 in Folsom.

 2. It's pretty much always a bad idea to add unnecessary
 restrictions without a good reason. While we don't have an
 immediate need to use, for example, the ever-useful
 http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
 up with a reason people *shouldn't* be allowed to use it.

 That said, apparently people have had a need to not be allowed to
 use some characters, but it's not clear why:
 https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
 knows any reason why these printable characters should not be
 joined in holy resource naming, speak now or forever hold your
 peace.
>>>
>>> I also could not find the reason of current restriction on the bug
>>> report, and I'd like to know it as the history.
>>> On v2 API(not v2.1), each resource name contains the following
>>> restriction for its name:
>>>
>>>   Resource  | Length  | Pattern
>>>  ---+-+--
>>>   aggregate | 1-255   | nothing
>>>   backup| nothing | nothing
>>>   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
>>> | |   [a-zA-Z0-9. _-]*$'
>>>   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
>>>   server| 1-255   | nothing
>>>   cell  | 1-255   | don't contain "." and "!"
>>>
>>> On v2.1 API, we have applied the same restriction rule[1] for whole
>>> resource names for API consistency, so maybe we need to consider
>>> this topic for whole names.
>>>
>>> [1]:
>>> https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44
>>
>> Honestly, I bet this had to do with how the database used to be set
>> up.
>>
> 
> So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
> multibyte characters (only 1-3 bytes). For example if you do a create
> image call with an image name to glance with a 4 byte multibyte
> character in the name it will 500. I'd guess we do something
> similar in places with the Nova API where we have inadequate input
> validation. If you want 4 byte support you need to use utf8mb4 instead.

Oh... fun. :(

> I don't know if postgresql has any restrictions (I don't think it
> does) or if db2 does too. But I don't think we can/should make it a
> complete free for all. It should at most be what most databases support.
> 
> I think its a big enough change that this late in the cycle we should
> push it off to Kilo. It's always much easier to loosen input validation
> than tighten it (or have to have an "oops" revert on an officially
> released Nova). Perhaps some tests to verify that everything we allow
> past the input validation checks we can actually store.

So, honestly, that seems like a pendulum swing in an odd way.

Havana "use anything you want!"
Icehouse ?
Juno "strict asci!"
Kilo "utf8"

Can't we just catch the db exception correctly in glance and not have it
explode? And then allow it. Exploding with a 500 on a bad name seems the
wrong thing to do anyway.

That would also mean that if the user changed their database to support
utf8mb4 (which they might want to do if it was important to them) it
would all work.

I think some release notes would be fine to explain the current
situation and limitations.

Basically, lets skate towards the puck here, realizing some corner cases
exist, but that we're moving in the direction we want to be, not back
tracking.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [zaqar] Signing Off from Core

[Malini Kamalambal]

Thank You Alej – not just for the tons of code you wrote for Zaqar, but also in 
making Zaqar a fun, open and welcoming part of the community!
Good Luck for everything ahead!

From: Victoria Martínez de la Cruz 
mailto:victo...@vmartinezdelacruz.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, September 17, 2014 6:29 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [zaqar] Signing Off from Core

Thanks for everything Alej!

Besides your contributions to the Zaqar team you also shown to be a great 
person.

I'm truly grateful that I could have you as a mentor during GSoC.

All the best :)

2014-09-17 19:14 GMT-03:00 Flavio Percoco 
mailto:fla...@redhat.com>>:
On 09/17/2014 11:50 PM, Alejandro Cabrera wrote:
> Hey all,
>
> I am officially removing myself as a core member of the Zaqar project.
>
> Thanks for all the good times, friends, and I wish you the best for the 
> future!

Alejandro,

I think I speak for everyone when I say the project is where it is also
because of your contributions and support. You played a key role in the
project's growth and we all thank you a lot for that.

I'm sad to see you go and I wish you luck on your new adventures :)
Cheers,
Flavio


--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Ken'ichi Ohmichi]

2014-09-18 19:57 GMT+09:00 Sean Dague :
> On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
>> On Sat, 13 Sep 2014 06:48:19 -0400
>> Sean Dague  wrote:
>
> We have proposed that the allowed characters for all resource
> names in Nova (flavors, aggregates, etc.) be expanded to all
> printable unicode characters and horizontal spaces:
> https://review.openstack.org/#/c/119741
>
> Currently, the only allowed characters in most resource names are
> alphanumeric, space, and [.-_].
>
>
> We have proposed this change for two principal reasons:
>
> 1. We have customers who have migrated data forward since Essex,
> when no restrictions were in place, and thus have characters in
> resource names that are disallowed in the current version of
> OpenStack. This is only likely to be useful to people migrating
> from Essex or earlier, since the current restrictions were added
> in Folsom.
>
> 2. It's pretty much always a bad idea to add unnecessary
> restrictions without a good reason. While we don't have an
> immediate need to use, for example, the ever-useful
> http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
> up with a reason people *shouldn't* be allowed to use it.
>
> That said, apparently people have had a need to not be allowed to
> use some characters, but it's not clear why:
> https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
> knows any reason why these printable characters should not be
> joined in holy resource naming, speak now or forever hold your
> peace.

 I also could not find the reason of current restriction on the bug
 report, and I'd like to know it as the history.
 On v2 API(not v2.1), each resource name contains the following
 restriction for its name:

   Resource  | Length  | Pattern
  ---+-+--
   aggregate | 1-255   | nothing
   backup| nothing | nothing
   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
 | |   [a-zA-Z0-9. _-]*$'
   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
   server| 1-255   | nothing
   cell  | 1-255   | don't contain "." and "!"

 On v2.1 API, we have applied the same restriction rule[1] for whole
 resource names for API consistency, so maybe we need to consider
 this topic for whole names.

 [1]:
 https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44
>>>
>>> Honestly, I bet this had to do with how the database used to be set
>>> up.
>>>
>>
>> So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
>> multibyte characters (only 1-3 bytes). For example if you do a create
>> image call with an image name to glance with a 4 byte multibyte
>> character in the name it will 500. I'd guess we do something
>> similar in places with the Nova API where we have inadequate input
>> validation. If you want 4 byte support you need to use utf8mb4 instead.
>
> Oh... fun. :(
>
>> I don't know if postgresql has any restrictions (I don't think it
>> does) or if db2 does too. But I don't think we can/should make it a
>> complete free for all. It should at most be what most databases support.
>>
>> I think its a big enough change that this late in the cycle we should
>> push it off to Kilo. It's always much easier to loosen input validation
>> than tighten it (or have to have an "oops" revert on an officially
>> released Nova). Perhaps some tests to verify that everything we allow
>> past the input validation checks we can actually store.
>
> So, honestly, that seems like a pendulum swing in an odd way.
>
> Havana "use anything you want!"
> Icehouse ?
> Juno "strict asci!"
> Kilo "utf8"

Correct validation history is

Essex: "use anything you want!"
Folsom: "strict asci!"
[..]
Juno: "strict asci!"

So I don't think we should make the input validation loose right now
to avoid a pendulum swing.


> Can't we just catch the db exception correctly in glance and not have it
> explode? And then allow it. Exploding with a 500 on a bad name seems the
> wrong thing to do anyway.
>
> That would also mean that if the user changed their database to support
> utf8mb4 (which they might want to do if it was important to them) it
> would all work.
>
> I think some release notes would be fine to explain the current
> situation and limitations.
>
> Basically, lets skate towards the puck here, realizing some corner cases
> exist, but that we're moving in the direction we want to be, not back
> tracking.

One idea is that: How about using base64 encoding/decoding if non-ascii
chars come? REST API layer would encode resource names if necessary
before passing it to DB layer, and we don't need to consider backend DB
features. The disadvantage is that available name length becomes short if
non-ascii chars. But maybe that would be acceptable..

Th

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Sean Dague]

On 09/18/2014 07:19 AM, Ken'ichi Ohmichi wrote:
> 2014-09-18 19:57 GMT+09:00 Sean Dague :
>> On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
>>> On Sat, 13 Sep 2014 06:48:19 -0400
>>> Sean Dague  wrote:
>>
>> We have proposed that the allowed characters for all resource
>> names in Nova (flavors, aggregates, etc.) be expanded to all
>> printable unicode characters and horizontal spaces:
>> https://review.openstack.org/#/c/119741
>>
>> Currently, the only allowed characters in most resource names are
>> alphanumeric, space, and [.-_].
>>
>>
>> We have proposed this change for two principal reasons:
>>
>> 1. We have customers who have migrated data forward since Essex,
>> when no restrictions were in place, and thus have characters in
>> resource names that are disallowed in the current version of
>> OpenStack. This is only likely to be useful to people migrating
>> from Essex or earlier, since the current restrictions were added
>> in Folsom.
>>
>> 2. It's pretty much always a bad idea to add unnecessary
>> restrictions without a good reason. While we don't have an
>> immediate need to use, for example, the ever-useful
>> http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
>> up with a reason people *shouldn't* be allowed to use it.
>>
>> That said, apparently people have had a need to not be allowed to
>> use some characters, but it's not clear why:
>> https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
>> knows any reason why these printable characters should not be
>> joined in holy resource naming, speak now or forever hold your
>> peace.
>
> I also could not find the reason of current restriction on the bug
> report, and I'd like to know it as the history.
> On v2 API(not v2.1), each resource name contains the following
> restriction for its name:
>
>   Resource  | Length  | Pattern
>  ---+-+--
>   aggregate | 1-255   | nothing
>   backup| nothing | nothing
>   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
> | |   [a-zA-Z0-9. _-]*$'
>   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
>   server| 1-255   | nothing
>   cell  | 1-255   | don't contain "." and "!"
>
> On v2.1 API, we have applied the same restriction rule[1] for whole
> resource names for API consistency, so maybe we need to consider
> this topic for whole names.
>
> [1]:
> https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

 Honestly, I bet this had to do with how the database used to be set
 up.

>>>
>>> So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
>>> multibyte characters (only 1-3 bytes). For example if you do a create
>>> image call with an image name to glance with a 4 byte multibyte
>>> character in the name it will 500. I'd guess we do something
>>> similar in places with the Nova API where we have inadequate input
>>> validation. If you want 4 byte support you need to use utf8mb4 instead.
>>
>> Oh... fun. :(
>>
>>> I don't know if postgresql has any restrictions (I don't think it
>>> does) or if db2 does too. But I don't think we can/should make it a
>>> complete free for all. It should at most be what most databases support.
>>>
>>> I think its a big enough change that this late in the cycle we should
>>> push it off to Kilo. It's always much easier to loosen input validation
>>> than tighten it (or have to have an "oops" revert on an officially
>>> released Nova). Perhaps some tests to verify that everything we allow
>>> past the input validation checks we can actually store.
>>
>> So, honestly, that seems like a pendulum swing in an odd way.
>>
>> Havana "use anything you want!"
>> Icehouse ?
>> Juno "strict asci!"
>> Kilo "utf8"
> 
> Correct validation history is
> 
> Essex: "use anything you want!"
> Folsom: "strict asci!"
> [..]
> Juno: "strict asci!"
>
> So I don't think we should make the input validation loose right now
> to avoid a pendulum swing.

Ok, great. That history makes me ok with status quo. I didn't realize it
went back so far.

>> Can't we just catch the db exception correctly in glance and not have it
>> explode? And then allow it. Exploding with a 500 on a bad name seems the
>> wrong thing to do anyway.
>>
>> That would also mean that if the user changed their database to support
>> utf8mb4 (which they might want to do if it was important to them) it
>> would all work.
>>
>> I think some release notes would be fine to explain the current
>> situation and limitations.
>>
>> Basically, lets skate towards the puck here, realizing some corner cases
>> exist, but that we're moving in the direction we want to be, not back
>> tracking.
> 
> One idea is that: How about using base64 encoding/decoding if non-ascii
> chars

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Flavio Percoco]

On 09/17/2014 10:36 PM, Joe Gordon wrote:
> Hi All,
> 
> My understanding of Zaqar is that it's like SQS. SQS uses distributed
> queues, which have a few unusual properties [0]:
> 
> 
> Message Order
> 
> Amazon SQS makes a best effort to preserve order in messages, but due to
> the distributed nature of the queue, we cannot guarantee you will
> receive messages in the exact order you sent them. If your system
> requires that order be preserved, we recommend you place sequencing
> information in each message so you can reorder the messages upon receipt.
> 

Zaqar guarantees FIFO. To be more precise, it does that relying on the
storage backend ability to do so as well. Depending on the storage used,
guaranteeing FIFO may have some performance penalties.

We have discussed on adding a way to enable/disable some features - like
FIFO - in a per-queue bases and now that we have flavors, we may work on
allowing things like enabling/disabling FIFO on a per-flavor basis.


> At-Least-Once Delivery
> 
> Amazon SQS stores copies of your messages on multiple servers for
> redundancy and high availability. On rare occasions, one of the servers
> storing a copy of a message might be unavailable when you receive or
> delete the message. If that occurs, the copy of the message will not be
> deleted on that unavailable server, and you might get that message copy
> again when you receive messages. Because of this, you must design your
> application to be idempotent (i.e., it must not be adversely affected if
> it processes the same message more than once).

As of now, Zaqar fully relies on the storage replication/clustering
capabilities to provide data consistency, availability and fault
tolerance. However, as far as consuming messages is concerned, it can
guarantee once-and-only-once and/or at-least-once delivery depending on
the message pattern used to consume messages. Using pop or claims
guarantees the former whereas streaming messages out of Zaqar guarantees
the later.


> Message Sample
> 
> The behavior of retrieving messages from the queue depends whether you
> are using short (standard) polling, the default behavior, or long
> polling. For more information about long polling, see Amazon SQS Long
> Polling
> .
> 
> With short polling, when you retrieve messages from the queue, Amazon
> SQS samples a subset of the servers (based on a weighted random
> distribution) and returns messages from just those servers. This means
> that a particular receive request might not return all your messages.
> Or, if you have a small number of messages in your queue (less than
> 1000), it means a particular request might not return any of your
> messages, whereas a subsequent request will. If you keep retrieving from
> your queues, Amazon SQS will sample all of the servers, and you will
> receive all of your messages.
> 
> The following figure shows short polling behavior of messages being
> returned after one of your system components makes a receive request.
> Amazon SQS samples several of the servers (in gray) and returns the
> messages from those servers (Message A, C, D, and B). Message E is not
> returned to this particular request, but it would be returned to a
> subsequent request.

Image here:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/images/ArchOverview_Receive.png

Currently the best and only way to pull messages from Zaqar is by using
short polling. We'll work on long-polling and websocket support.
Requests are guaranteed to return available messages if there are any.

> Presumably SQS has these properties because it makes the
> system scalable, if so does Zaqar have the same properties (not just
> making these same guarantees in the API, but actually having these
> properties in the backends)? And if not, why?

Based on our short conversation on IRC last night, I understand you're
concerned that FIFO may result in performance issues. That's a valid
concern and I think the right answer is that it depends on the storage.
If the storage has a built-in FIFO guarantee then there's nothing Zaqar
needs to do there. In the other hand, if the storage does not have a
built-in support for FIFO, Zaqar will cover it in the driver
implementation. In the mongodb driver, each message has a marker that is
used to guarantee FIFO.


> I looked on the wiki [1]
> for information on this, but couldn't find anything.
> 


We have this[0] section in the FAQ but it definitely doesn't answer your
questions. I'm sure we had this documented way better but I can't find
the link. :(


[0]
https://wiki.openstack.org/wiki/Zaqar/Frequently_asked_questions#How_does_Zaqar_compare_to_AWS_.28SQS.2FSNS.29.3F

Hope the above helps,
Flavio

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/l

Re: [openstack-dev] [MagnetoDB] Core developer nomination

[Ajaya Agrawal]

+1

Cheers,
Ajaya

On Thu, Sep 18, 2014 at 2:38 PM, Dmitriy Ukhlov 
wrote:

> +1 from me, Charles is very active contributor and I guess if we have such
> developer in core team
> MagnetoDB project will become much better that it is now.
>
> On Thu, Sep 18, 2014 at 10:09 AM, Illia Khudoshyn  > wrote:
>
>> Congrats, Charles! Great job!
>>
>> On Thu, Sep 18, 2014 at 12:05 AM, Ilya Sviridov 
>> wrote:
>>
>>> Hello magnetodb contributors,
>>>
>>> I'm glad to nominate Charles Wang to core developers of MagnetoDB.
>>>
>>> He is top non-core reviewer [1], implemented notifications [2] in mdb
>>> and made a great progress with performance, stability and scalability
>>> testing  of MagnetoDB
>>>
>>> [1] http://stackalytics.com/report/contribution/magnetodb/90
>>> [2]
>>> https://blueprints.launchpad.net/magnetodb/+spec/magnetodb-notifications
>>>
>>> Welcome to team, Charles!
>>> Looking forward for your contribution
>>>
>>> --
>>> Ilya Sviridov
>>> isviridov @ FreeNode
>>>
>>
>>
>>
>> --
>>
>> Best regards,
>>
>> Illia Khudoshyn,
>> Software Engineer, Mirantis, Inc.
>>
>>
>>
>> 38, Lenina ave. Kharkov, Ukraine
>>
>> www.mirantis.com 
>>
>> www.mirantis.ru
>>
>>
>>
>> Skype: gluke_work
>>
>> ikhudos...@mirantis.com
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Best regards,
> Dmitriy Ukhlov
> Mirantis Inc.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Thomas Goirand]

On 09/18/2014 04:01 PM, Flavio Percoco wrote:
> After having gone through the whole thread and read all the concerns,
> problems and reasonings, I think we should stick to requests as-is for
> now and deal with this particular issue.
> 
> Regardless of the vendorized urllib3 package, I believe requests is a
> good library with an easy-to-consume API and it has solved several
> problems throughout OpenStack. Not to mention it's also helpped with
> making OpenStack more consistent. We've put a lot of effort to get to
> this point and I don't think we should revert all that because of the
> vendorized `urllib3` package.
> 
> Cheers,
> Flavio

I, at least, haven't suggested that we stop using requests. Just that we
don't internally use any of the requests.packages.* stuff.

The rest of the debate about the good or bad things with vendorizing,
even if it is my view that it's a really bad thing, is IMO not
interesting for the OpenStack project.

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Flavio Percoco]

On 09/18/2014 01:28 PM, Sean Dague wrote:
> On 09/18/2014 07:19 AM, Ken'ichi Ohmichi wrote:
>> 2014-09-18 19:57 GMT+09:00 Sean Dague :
>>> On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
 On Sat, 13 Sep 2014 06:48:19 -0400
 Sean Dague  wrote:
>>>
>>> We have proposed that the allowed characters for all resource
>>> names in Nova (flavors, aggregates, etc.) be expanded to all
>>> printable unicode characters and horizontal spaces:
>>> https://review.openstack.org/#/c/119741
>>>
>>> Currently, the only allowed characters in most resource names are
>>> alphanumeric, space, and [.-_].
>>>
>>>
>>> We have proposed this change for two principal reasons:
>>>
>>> 1. We have customers who have migrated data forward since Essex,
>>> when no restrictions were in place, and thus have characters in
>>> resource names that are disallowed in the current version of
>>> OpenStack. This is only likely to be useful to people migrating
>>> from Essex or earlier, since the current restrictions were added
>>> in Folsom.
>>>
>>> 2. It's pretty much always a bad idea to add unnecessary
>>> restrictions without a good reason. While we don't have an
>>> immediate need to use, for example, the ever-useful
>>> http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
>>> up with a reason people *shouldn't* be allowed to use it.
>>>
>>> That said, apparently people have had a need to not be allowed to
>>> use some characters, but it's not clear why:
>>> https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
>>> knows any reason why these printable characters should not be
>>> joined in holy resource naming, speak now or forever hold your
>>> peace.
>>
>> I also could not find the reason of current restriction on the bug
>> report, and I'd like to know it as the history.
>> On v2 API(not v2.1), each resource name contains the following
>> restriction for its name:
>>
>>   Resource  | Length  | Pattern
>>  ---+-+--
>>   aggregate | 1-255   | nothing
>>   backup| nothing | nothing
>>   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
>> | |   [a-zA-Z0-9. _-]*$'
>>   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
>>   server| 1-255   | nothing
>>   cell  | 1-255   | don't contain "." and "!"
>>
>> On v2.1 API, we have applied the same restriction rule[1] for whole
>> resource names for API consistency, so maybe we need to consider
>> this topic for whole names.
>>
>> [1]:
>> https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44
>
> Honestly, I bet this had to do with how the database used to be set
> up.
>

 So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
 multibyte characters (only 1-3 bytes). For example if you do a create
 image call with an image name to glance with a 4 byte multibyte
 character in the name it will 500. I'd guess we do something
 similar in places with the Nova API where we have inadequate input
 validation. If you want 4 byte support you need to use utf8mb4 instead.
>>>
>>> Oh... fun. :(
>>>
 I don't know if postgresql has any restrictions (I don't think it
 does) or if db2 does too. But I don't think we can/should make it a
 complete free for all. It should at most be what most databases support.

 I think its a big enough change that this late in the cycle we should
 push it off to Kilo. It's always much easier to loosen input validation
 than tighten it (or have to have an "oops" revert on an officially
 released Nova). Perhaps some tests to verify that everything we allow
 past the input validation checks we can actually store.
>>>
>>> So, honestly, that seems like a pendulum swing in an odd way.
>>>
>>> Havana "use anything you want!"
>>> Icehouse ?
>>> Juno "strict asci!"
>>> Kilo "utf8"
>>
>> Correct validation history is
>>
>> Essex: "use anything you want!"
>> Folsom: "strict asci!"
>> [..]
>> Juno: "strict asci!"
>>
>> So I don't think we should make the input validation loose right now
>> to avoid a pendulum swing.
> 
> Ok, great. That history makes me ok with status quo. I didn't realize it
> went back so far.
> 
>>> Can't we just catch the db exception correctly in glance and not have it
>>> explode? And then allow it. Exploding with a 500 on a bad name seems the
>>> wrong thing to do anyway.
>>>
>>> That would also mean that if the user changed their database to support
>>> utf8mb4 (which they might want to do if it was important to them) it
>>> would all work.
>>>
>>> I think some release notes would be fine to explain the current
>>> situation and limitations.
>>>
>>> Basically, lets skate towards the puck here, realizing some corner cases
>>> exist, 

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 7:43 AM, Thomas Goirand  wrote:
> 
> On 09/18/2014 04:01 PM, Flavio Percoco wrote:
>> After having gone through the whole thread and read all the concerns,
>> problems and reasonings, I think we should stick to requests as-is for
>> now and deal with this particular issue.
>> 
>> Regardless of the vendorized urllib3 package, I believe requests is a
>> good library with an easy-to-consume API and it has solved several
>> problems throughout OpenStack. Not to mention it's also helpped with
>> making OpenStack more consistent. We've put a lot of effort to get to
>> this point and I don't think we should revert all that because of the
>> vendorized `urllib3` package.
>> 
>> Cheers,
>> Flavio
> 
> I, at least, haven't suggested that we stop using requests. Just that we
> don't internally use any of the requests.packages.* stuff.
> 
> The rest of the debate about the good or bad things with vendorizing,
> even if it is my view that it's a really bad thing, is IMO not
> interesting for the OpenStack project.

I don’t believe that’s a good idea. If you’re wanting to use urllib3 in order
to interact with requests than you *should* be using requests.packages.urllib3,
to do anything else risks having two different versions of urllib3 primitives at
play in one subsystem.

It’s not even completely possible in the case that prompted this thread 
originally
since the reason requests.packages.urllib3 was being imported from was so that
there could be an is instance() check against one of the classes. If that wasn’t
imported from requests.packages.urllib3 but instead from just urllib3 than the
isinstance check would always fail.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Thomas Goirand]

On 09/18/2014 10:43 AM, Donald Stufft wrote:
>>> Obviously we can work with the requests team to figure out the best
>>> approach.
>>
>> There's only a single approach that works: have the requests upstream
>> authors to stop embedding foreign code, and use the dependency instead.
> 
> There are legitimate reasons for a project to vendor things.

Yes, there's lot of reasons. But so fare, I haven't read about any valid
one.

> Linux distributions are not the end be all of distribution models and
> they don’t get to dictate to upstream.

Well, distributions is where the final user is, and where software gets
consumed. Our priority should be the end users.

> Generally I agree that requests should not vendor urllib3

Unfortunately, it doesn't seem requests upstream agree, so we can only
deal with the issue. This means not using requests.packages.*.

>   You’re going to get very strange incompatibility problems
> if you try to mis requests.packages.urllib3 and urllib3 in one codebase
> and if you’re using requests at all it’s going to be expecting to use
> the embedded copy of urllib3.

I'm well aware of this. As I wrote, I already had to deal with issues
like that, and I'm expecting even more in the future.

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> 
>> 
>> Linux distributions are not the end be all of distribution models and
>> they don’t get to dictate to upstream.
> 
> Well, distributions is where the final user is, and where software gets
> consumed. Our priority should be the end users.


Distributions are not the only place that people get their software from,
unless you think that the ~3 million downloads requests has received
on PyPI in the last 30 days are distributions downloading requests to
package in their OSs.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[Jeremy Stanley]

On 2014-09-18 08:06:11 + (+), Sullivan, Jon Paul wrote:
> In my experience if the check results are not fresh enough the
> recheck is automatically run.  I am not on the infra team, so
> without looking up code I am just guessing, but my guess is that
> the workflow score change triggers the check on the presumption
> that it has been approved, not accounting for the recent(ish)
> update that move wip to the workflow score.

We turned off that behavior a couple months ago when the merge-check
pseudo-job was implemented to automatically -1 any open changes with
merge conflicts each time a new change merges to their target
branch. This covered the majority of the problems identified by the
freshness check, but without using any of our worker pool.

> This is not solely about finding reviews.  It is about pruning
> stale reviews.  I think the auto-abandon code was excellent at
> doing this, but alas, it is no more. 

I think it was excellent at arbitrarily abandoning open changes
which happened to meet a poorly-thought-out set of criteria. I'm
personally quite glad it broke and we didn't waste time
reimplementing something similar for new Gerrit versions.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] are we going to remove the novaclient v3 shell or what?

[Alex Xu]

On 2014年09月18日 18:14, Day, Phil wrote:

-Original Message-
From: Kenichi Oomichi [mailto:oomi...@mxs.nes.nec.co.jp]
Sent: 18 September 2014 02:44
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova] are we going to remove the novaclient
v3 shell or what?



-Original Message-
From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
Sent: Wednesday, September 17, 2014 11:59 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [nova] are we going to remove the novaclient v3

shell or what?

This has come up a couple of times in IRC now but the people that
probably know the answer aren't available.

There are python-novaclient patches that are adding new CLIs to the v2
(v1_1) and v3 shells, but now that we have the v2.1 API (v2 on v3) why
do we still have a v3 shell in the client?  Are there plans to remove that?

I don't really care either way, but need to know for code reviews.

One example: [1]

[1] https://review.openstack.org/#/c/108942/

Sorry for a little late response,
I think we don't need new features of v3 into novaclient anymore.
For example, the v3 part of the above[1] was not necessary because a new
feature server-group quota is provided as v2 and v2.1, not v3.

That would be true if there was a version of the client that supported v2.1 
today, but while the V2.1 API is still presented as V3 and doesn't include the 
tenant_id - making the V3 client the only simple way to test new V2.1 features 
in devstack as far as I can see.


How about this as a plan:

1) We add support to the client for "--os-compute-api-version=v2.1"   which 
maps into the client with the URL set to include v2.1(this won't be usable until we 
do step 2)

+1


2) We change the Nova  to present the v2.1 API  as 
'http://X.X.X.X:8774/v2.1//
  - At this point we will have a working client for all of the stuff that's 
been moved back from V3 to V2.1, but will lose access to any V3 stuff not yet 
moved (which is the opposite of the current state where the v3 client can only 
be used for things that haven't been refactored to V2.1)


Actually we already can access v2.1 API as 
''http://X.X.X.X:8774/v2.1//..'

https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini#L64



3) We remove V3 from the client.


Until we get 1 & 2 done, to me it still makes sense to allow small changes into 
the v3 client, so that we keep it usable with the V2.1 API



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Rally] Load testing of dependent resources

[Ilya Shakhat]

Ralliers,

I'd like to share with you an idea on how to test dependent resources.

Let's consider the following example: there is a network and ports
belonging to it and we would like to test port creation process. Currently
I see 2 options of writing scenario:
 a) The scenario creates network and then creates a specified number of
ports. Runner executes the scenario specified number of times. -- In this
case ports belonging to different networks are created in parallel, but all
ports belonging to the same network are created serially.
 b) The network already exists and the scenario just creates ports, -- This
case allows to test concurrent creation of ports belonging to the same
network but not between multiple networks.
It would be really cool to get benefits from both cases, i.e. to test
parallel port creation but without limiting to parent network resource.

One of possible approaches may be to construct chain of scenarios where
preceding scenario prepares context for the further one. From the above
example the execution would be:
 1) The task contains sequence of 2 scenarios:
* network creation - it creates network and returns it
* port creation - it picks the network from incoming params and creates
port on it
 2) The runner has execution pipeline (which currently is represented as
generator producing identical scenario runners). The pipeline starts with
sequence of network creation scenario runners. Once one of runner finishes,
it then puts the further scenario and the result into pipeline. The
pipeline ends once no further scenarios left.

>From the above example execution flow would be like:
 * pipeline is "filled" with N network scenarios (not actually filled, but
contains a generator that is able to do this)
 * R runners pick scenarios from pipeline
 * pipeline contains N-R network scenarios
 * one of runners finishes the scenario and updates pipeline with P port
scenarios
 * pipeline contains N-R-1 network scenarios followed by P port scenarios
 * work-work-work until pipeline is empty

This execution covers case from a) and b) but there still will be sequences
of child resources belonging to the same parent. The improvement may be to
pick scenario from pipeline randomly.

How does it sound?

Thanks,
Ilya
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Chmouel Boudjnah]

On Thu, Sep 18, 2014 at 1:58 PM, Donald Stufft  wrote:

> Distributions are not the only place that people get their software from,
> unless you think that the ~3 million downloads requests has received
> on PyPI in the last 30 days are distributions downloading requests to
> package in their OSs.
>


I think Thomas was speaking in the context of how OpenStack is used by the
end user and that probably the point of debate here, requests ships
libraries inside to make it easy for users that doen't use Linux distro
packages, when in OpenStack (or at least in prod) packagers are something
we generally very much care about.

Chmouel
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 9:00 AM, Chmouel Boudjnah  wrote:
> 
> 
> On Thu, Sep 18, 2014 at 1:58 PM, Donald Stufft  > wrote:
> Distributions are not the only place that people get their software from,
> unless you think that the ~3 million downloads requests has received
> on PyPI in the last 30 days are distributions downloading requests to
> package in their OSs.
> 
> 
> I think Thomas was speaking in the context of how OpenStack is used by the 
> end user and that probably the point of debate here, requests ships libraries 
> inside to make it easy for users that doen't use Linux distro packages, when 
> in OpenStack (or at least in prod) packagers are something we generally very 
> much care about.

Even then, my statement holds true, just with different numbers.

Every distribution modifies upstream in different ways, I think it's insane to
do contortions which will break things for people *not* getting things through
those channels. If distributions are going to modify one upstream project they
should expect to need to modify things that depend on that project in ways that
are sensitive to what they've modified.

The only real sane thing IMO is for openstack to consider requests as it is on
PyPI. If openstack wants to make it easier for downstream to de-vendor urllib3
from requests then when openstack wants to import from requests.packages.* it
can instead do:

try:
from requests.packages import urllib3
except ImportError:
import urllib3

This will cause it to work correctly when requests is installed in a pristine
state, and will fallback to handling the modifications that some downstream
redistributors make.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] are we going to remove the novaclient v3 shell or what?

[Ken'ichi Ohmichi]

2014-09-18 21:31 GMT+09:00 Alex Xu :
> On 2014年09月18日 18:14, Day, Phil wrote:
>>>
>>> -Original Message-
>>> From: Kenichi Oomichi [mailto:oomi...@mxs.nes.nec.co.jp]
>>> Sent: 18 September 2014 02:44
>>> To: OpenStack Development Mailing List (not for usage questions)
>>> Subject: Re: [openstack-dev] [nova] are we going to remove the novaclient
>>> v3 shell or what?
>>>
>>>
 -Original Message-
 From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
 Sent: Wednesday, September 17, 2014 11:59 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: [openstack-dev] [nova] are we going to remove the novaclient v3
>>>
>>> shell or what?

 This has come up a couple of times in IRC now but the people that
 probably know the answer aren't available.

 There are python-novaclient patches that are adding new CLIs to the v2
 (v1_1) and v3 shells, but now that we have the v2.1 API (v2 on v3) why
 do we still have a v3 shell in the client?  Are there plans to remove
 that?

 I don't really care either way, but need to know for code reviews.

 One example: [1]

 [1] https://review.openstack.org/#/c/108942/
>>>
>>> Sorry for a little late response,
>>> I think we don't need new features of v3 into novaclient anymore.
>>> For example, the v3 part of the above[1] was not necessary because a new
>>> feature server-group quota is provided as v2 and v2.1, not v3.
>>
>> That would be true if there was a version of the client that supported
>> v2.1 today, but while the V2.1 API is still presented as V3 and doesn't
>> include the tenant_id - making the V3 client the only simple way to test new
>> V2.1 features in devstack as far as I can see.

v2.1 URL contains tenant_id already, and we can use /v2.1 endpoint like:

$ curl -i 
'http://192.168.11.62:8774/v2.1/05d0f72534e449a0a3e70720c5d7c206/servers/detail'
-X GET -H "Accept: applica
tion/json" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token:
f41776736ef34e56a7773e2b2d18d2e3"
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 1749
X-Openstack-Request-Id: req-96768644-be2a-43c4-8757-ad1802736970
Date: Thu, 18 Sep 2014 13:03:54 GMT

{"servers": [{"status": "ACTIVE", "updated": "2014-09-18T11:57:12Z",
"hostId": "8c63113abf9b1e13f1b11dd6a2dbdf511f7aa2199cff31e847582414",
"OS-EXT-SRV-ATTR:host": "oomichi-trusty", "addresses": {"private":
[{"version": 4, "type": "fixed", "addr": "10.0.0.14", "mac_addr":
"fa:16:3e:65:7e:ab"}]}, "links": [{"href":
"http://192.168.11.62:8774/v2.1/05d0f72534e449a0a3e70720c5d7c206/servers/bca4e77b-9763-4cf4-8a84-193df620eeb7";,
"rel": "self"}, {"href":
"http://192.168.11.62:8774/05d0f72534e449a0a3e70720c5d7c206/servers/bca4e77b-9763-4cf4-8a84-193df620eeb7";,
"rel": "bookmark"}], "key_name": "key-temporary", "image": {"id":
"2ac73d2e-f0d8-4576-a62c-3b53f70d071b", "links": [{"href":
"http://192.168.11.62:8774/05d0f72534e449a0a3e70720c5d7c206/images/2ac73d2e-f0d8-4576-a62c-3b53f70d071b";,
"rel": "bookmark"}]}, "os-security-groups:security_groups": [{"name":
"sg-temporary"}], "os-pci:pci_devices": [], "OS-EXT-STS:task_state":
null, "os-extended-availability-zone:availability_zone": "nova",
"OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name":
"instance-000e", "OS-SRV-USG:launched_at":
"2014-09-18T11:57:12.00", "OS-EXT-SRV-ATTR:hypervisor_hostname":
"oomichi-trusty", "flavor": {"id": "84", "links": [{"href":
"http://192.168.11.62:8774/05d0f72534e449a0a3e70720c5d7c206/flavors/84";,
"rel": "bookmark"}]}, "id": "bca4e77b-9763-4cf4-8a84-193df620eeb7",
"OS-SRV-USG:terminated_at": null, "user_id":
"30bd25b4555d4664b1069d8d7e682eef", "name": "vm01", "created":
"2014-09-18T11:57:03Z", "tenant_id":
"05d0f72534e449a0a3e70720c5d7c206",
"os-extended-volumes:volumes_attached": [], "accessIPv4": "",
"accessIPv6": "", "OS-EXT-STS:locked_by": null, "progress": 0,
"OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {}}]}
$

DevStack doesn't register v2.1 endpoint to keytone now, but we can use
it with calling it directly.
It is true that it is difficult to use v2.1 API now and we can check
its behavior via v3 API instead.

>> How about this as a plan:
>>
>> 1) We add support to the client for "--os-compute-api-version=v2.1"
>> which maps into the client with the URL set to include v2.1(this won't
>> be usable until we do step 2)

v2.1 behavior(API URLs except the endpoint, request/response bodies,
status codes) should
be the same as v2. So if devstack registers v2.1 endpoint as the type
"computev21", we can
use it by specifying "computev21" as --service-type of current nova command.


>> 2) We change the Nova  to present the v2.1 API  as
>> 'http://X.X.X.X:8774/v2.1//
>>   - At this point we will have a working client for all of the stuff
>> that's been moved back from V3 to V2.1, but will lose access to any V3 stuff
>> not yet moved (which is the opposite of the current state where the v3
>> client can only be used fo

Re: [openstack-dev] [zaqar] Signing Off from Core

[Sriram Madapusi Vasudevan]

Thank You Alej!
Had an awesome time working with you.
Wish you luck in your endeavors ahead!

Best,
Sriram Madapusi Vasudevan



On Sep 18, 2014, at 6:58 AM, Malini Kamalambal 
mailto:malini.kamalam...@rackspace.com>> wrote:

Thank You Alej – not just for the tons of code you wrote for Zaqar, but also in 
making Zaqar a fun, open and welcoming part of the community!
Good Luck for everything ahead!

From: Victoria Martínez de la Cruz 
mailto:victo...@vmartinezdelacruz.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, September 17, 2014 6:29 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [zaqar] Signing Off from Core

Thanks for everything Alej!

Besides your contributions to the Zaqar team you also shown to be a great 
person.

I'm truly grateful that I could have you as a mentor during GSoC.

All the best :)

2014-09-17 19:14 GMT-03:00 Flavio Percoco 
mailto:fla...@redhat.com>>:
On 09/17/2014 11:50 PM, Alejandro Cabrera wrote:
> Hey all,
>
> I am officially removing myself as a core member of the Zaqar project.
>
> Thanks for all the good times, friends, and I wish you the best for the 
> future!

Alejandro,

I think I speak for everyone when I say the project is where it is also
because of your contributions and support. You played a key role in the
project's growth and we all thank you a lot for that.

I'm sad to see you go and I wish you luck on your new adventures :)
Cheers,
Flavio


--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [blazar]: proposal for a new lease type

[Lisa]

Hi all,

my name is Lisa Zangrando and I work at the Italian National Institute 
for Nuclear Physics (INFN). In particular I am leading a team which is 
addressing the issue concerning the efficiency in the resource usage in 
OpenStack.
Currently OpenStack allows just a static partitioning model where the 
resource allocation to the user teams (i.e. the projects) can be done 
only by considering fixed quotas which cannot be exceeded even if there 
are unused resources (but) assigned to different projects.
We studied the available BLAZAR's documentation and, in agreement with 
Tim Bell (who is responsible the OpenStack cloud project at CERN), we 
think this issue could be addressed within your framework.
Please find attached a document that describes our use cases (actually 
we think that many other environments have to deal with the same 
problems) and how they could be managed in BLAZAR, by defining a new 
lease type (i.e. fairShare lease) to be considered as extension of the 
list of the already supported lease types.

I would then be happy to discuss these ideas with you.

Thanks in advance,
Lisa



Use cases for a new lease type in BLAZAR.pdf
Description: Adobe PDF document
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [requirements][horizon] Dependency freeze exceptions

[Radomir Dopieralski]

Hello,

I would like to request dependency freeze exceptions for the following
patches for Horizon:

https://review.openstack.org/#/c/121509/
https://review.openstack.org/#/c/122410/

and

https://review.openstack.org/#/c/113184/

They are all fixing high priority bugs. The first two are related to
bugs with parsing Bootstrap 3.2.0 scss files that have been fixed
upstream. The third one makes the life of packagers a little eaiser,
by using versions that both Debian and Fedora, and possibly many
other distros, can ship.

I am not sure what the formal process for such an exception should be,
so I'm writing here. Please let me know if I should have done it
differently.
-- 
Radomir Dopieralski

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][L3] Sub team meeting cancelled

[Carl Baldwin]

I have a conflict today.  Keep working on RC1.

Carl
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [blazar]: proposal for a new lease type

[Sylvain Bauza]

Le 18/09/2014 15:27, Lisa a écrit :

Hi all,

my name is Lisa Zangrando and I work at the Italian National Institute 
for Nuclear Physics (INFN). In particular I am leading a team which is 
addressing the issue concerning the efficiency in the resource usage 
in OpenStack.
Currently OpenStack allows just a static partitioning model where the 
resource allocation to the user teams (i.e. the projects) can be done 
only by considering fixed quotas which cannot be exceeded even if 
there are unused resources (but) assigned to different projects.
We studied the available BLAZAR's documentation and, in agreement with 
Tim Bell (who is responsible the OpenStack cloud project at CERN), we 
think this issue could be addressed within your framework.
Please find attached a document that describes our use cases (actually 
we think that many other environments have to deal with the same 
problems) and how they could be managed in BLAZAR, by defining a new 
lease type (i.e. fairShare lease) to be considered as extension of the 
list of the already supported lease types.

I would then be happy to discuss these ideas with you.

Thanks in advance,
Lisa



Hi Lisa,

Glad to see you're interested in Blazar.

I tried to go thru your proposal, but could you please post the main 
concepts of what you plan to add into an etherpad and create a blueprint 
[1] mapped to it so we could discuss on the implementation ?
Of course, don't hesitate to ping me or the blazar community in 
#openstack-blazar if you need help with the process or the current 
Blazar design.


Thanks,
-Sylvain

[1] https://blueprints.launchpad.net/blazar/




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Oslo final releases ready

[Doug Hellmann]

All of the final releases for the Oslo libraries for the Juno cycle are 
available on PyPI. I’m working on a couple of patches to the global 
requirements list to update the baseline in the applications. In all cases, the 
final release is a second tag on a previously released version.

- oslo.config - 1.4.0 (same as 1.4.0.0a5)
- oslo.db - 1.0.0 (same as 0.5.0)
- oslo.i18n - 1.0.0 (same as 0.4.0)
- oslo.messaging - 1.4.0 (same as 1.4.0.0a5)
- oslo.rootwrap - 1.3.0 (same as 1.3.0.0a3)
- oslo.serialization - 1.0.0 (same as 0.3.0)
- oslosphinx - 2.2.0 (same as 2.2.0.0a3)
- oslotest - 1.1.0 (same as 1.1.0.0a2)
- oslo.utils - 1.0.0 (same as 0.3.0)
- cliff - 1.7.0 (previously tagged, so not a new release)
- stevedore - 1.0.0 (same as 1.0.0.0a2)

Congratulations and *Thank You* to the Oslo team for doing an amazing job with 
graduations this cycle!

Doug


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Gordon Sim]

On 09/18/2014 12:31 PM, Flavio Percoco wrote:

On 09/17/2014 10:36 PM, Joe Gordon wrote:

My understanding of Zaqar is that it's like SQS. SQS uses distributed
queues, which have a few unusual properties [0]:


 Message Order

Amazon SQS makes a best effort to preserve order in messages, but due to
the distributed nature of the queue, we cannot guarantee you will
receive messages in the exact order you sent them. If your system
requires that order be preserved, we recommend you place sequencing
information in each message so you can reorder the messages upon receipt.



Zaqar guarantees FIFO. To be more precise, it does that relying on the
storage backend ability to do so as well. Depending on the storage used,
guaranteeing FIFO may have some performance penalties.


Would it be accurate to say that at present Zaqar does not use 
distributed queues, but holds all queue data in a storage mechanism of 
some form which may internally distribute that data among servers but 
provides Zaqar with a consistent data model of some form?


[...]

As of now, Zaqar fully relies on the storage replication/clustering
capabilities to provide data consistency, availability and fault
tolerance.


Is the replication synchronous or asynchronous with respect to client 
calls? E.g. will the response to a post of messages be returned only 
once the replication of those messages is confirmed? Likewise when 
deleting a message, is the response only returned when replicas of the 
message are deleted?



However, as far as consuming messages is concerned, it can
guarantee once-and-only-once and/or at-least-once delivery depending on
the message pattern used to consume messages. Using pop or claims
guarantees the former whereas streaming messages out of Zaqar guarantees
the later.


From what I can see, pop provides unreliable delivery (i.e. its similar 
to no-ack). If the delete call using pop fails while sending back the 
response, the messages are removed but didn't get to the client.


What do you mean by 'streaming messages'?

[...]

Based on our short conversation on IRC last night, I understand you're
concerned that FIFO may result in performance issues. That's a valid
concern and I think the right answer is that it depends on the storage.
If the storage has a built-in FIFO guarantee then there's nothing Zaqar
needs to do there. In the other hand, if the storage does not have a
built-in support for FIFO, Zaqar will cover it in the driver
implementation. In the mongodb driver, each message has a marker that is
used to guarantee FIFO.


That marker is a sequence number of some kind that is used to provide 
ordering to queries? Is it generated by the database itself?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] PostgreSQL jobs slow in the gate

[Matt Riedemann]

On 9/18/2014 5:49 AM, Sean Dague wrote:

On 09/17/2014 11:50 PM, Clark Boylan wrote:

On Wed, Sep 17, 2014, at 06:48 PM, Clark Boylan wrote:

On Wed, Sep 17, 2014, at 06:37 PM, Matt Riedemann wrote:



On 9/17/2014 7:59 PM, Ian Wienand wrote:

On 09/18/2014 09:49 AM, Clark Boylan wrote:

Recent sampling of test run times shows that our tempest jobs run
against clouds using PostgreSQL are significantly slower than jobs run
against clouds using MySQL.


FYI There is a possibly relevant review out for max_connections limits
[1], although it seems to have some issues with shmem usage

-i

[1] https://review.openstack.org/#/c/121952/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



That's a backport of a fix from master where we were hitting fatal
errors due to too many DB connections which was brought on by the
changes to cinder and glance to run as many workers as there were CPUs
available.  So I don't think it probably plays here...

The errors pointed out in another part of the thread have been around
for awhile, I think they are due to negative tests where we're hitting
unique constraints because of the negative tests, so they are expected.

We should also note that the postgresql jobs run with the nova metadata
API service, I'm not sure how much of a factor that would have here.

Is there anything else unique about those jobs from the MySQL ones?


Good question. There are apparently other differences. The postgres job
runs Keystone under eventlet instead of via apache mod_wsgi. It also
sets FORCE_CONFIGDRIVE=False instead of always. And the final difference
I can find is the one you point out, nova api metadata service is run as
an independent thing.

Could these things be related? It would be relatively simple to push a
change or two to devstack-gate to test this but there are enough options
here that I probably won't do that until we think at least one of these
options is at fault.

I am starting to feel bad that I picked on PostgreSQL and completely
forgot that there were other items in play here. I went ahead and
uploaded [0] to run all devstack jobs without keystone wsgi services
(eventlet) and [1] to run all devstack job with keystone wsgi services
and the initial results are pretty telling.

It appears that keystone eventlet is the source of the slowness in this
job. With keystone eventlet all of the devstack jobs are slower and with
keystone wsgi all of the jobs are quicker. Probably need to collect a
bit more data but this doesn't look good for keystone eventlet.

Thank you Matt for pointing me in this direction.

[0] https://review.openstack.org/#/c/122299/
[1] https://review.openstack.org/#/c/122300/


Don't feel bad. :)

The point that Clark highlights here is a good one. There is an
assumption that once someone creates a job in infra, the magic elves are
responsible for it.

But there are no magic elves. So jobs like this need sponsors.

Maybe the right thing to do is not conflate this configuration and put
an eventlet version of the keystone job only on keystone (because the
keystone team was the one that proposed having a config like that, but
it's so far away from their project they aren't ever noticing when it's
regressing).

Same issue with the metadata server split. That's really only a thing
Nova cares about. It shouldn't impact anyone else.

-Sean



Neutron cares about the nova metadata API service right?

--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] PostgreSQL jobs slow in the gate

[Matt Riedemann]

On 9/18/2014 12:35 AM, Morgan Fainberg wrote:

-Original Message-
From: Dean Troyer 
Reply: OpenStack Development Mailing List (not for usage questions) 
>
Date: September 17, 2014 at 21:21:47
To: OpenStack Development Mailing List (not for usage questions) 
>
Subject:  Re: [openstack-dev] PostgreSQL jobs slow in the gate



Clark Boylan Wrotr :


It appears that keystone eventlet is the source of the slowness in this
job. With keystone eventlet all of the devstack jobs are slower and with
keystone wsgi all of the jobs are quicker. Probably need to collect a
bit more data but this doesn't look good for keystone eventlet.




On Wed, Sep 17, 2014 at 11:02 PM, Morgan Fainberg <
morgan.fainb...@gmail.com> wrote:


I've kicked off a test[1] as well to check into some tunable options
(eventlet workers) for improving keystone eventlet performance. I'll circle
back with the infra team once we have a little more data on both fronts.
The Keystone team will use this data to figure out the best way to approach
this issue.



Brant submitted https://review.openstack.org/#/c/121384/ to up the Keystone
workers when API_WORKERS is set.

I submitted https://review.openstack.org/#/c/122013/ to set a scaling
default for API_WORKERS based on the available CPUs ((nproc+1)/2). There
is a summary in that commit message of the current reviews addressing the
workers in various projects.

I think it has become clear that DevStack needs to set a default for most
services that are currently either too big (nproc) or too small (Keystone
at 1). Of course, moving things to mod_wsgi moots all of that, but it'll
be a while before everything moves.

dt

--

Dean Troyer
dtro...@gmail.com


Dean,

We should probably look at increasing the default number of workers as well in 
the Keystone configuration (rather than just devstack). It looks like, with 
limited datasets, we are seeing a real improvement with Keystone and 4 workers 
(from my previously mentioned quick test). Thanks for the extra data points. 
This helps to confirm the issue is Keystone under eventlet.

Cheers,
Morgan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



I pointed this out in Brant's devstack patch but we had a product team 
internally bring up this same point in Icehouse, they were really 
limited due to the eventlet workers issue in Keystone and once we 
provided the option (backported) it increased their throughput by 20%. 
We've been running with that in our internal Tempest runs (setting 
workers equal to number of CPUs / 2) and so far so good.


--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Chris St. Pierre]

On Thu, Sep 18, 2014 at 4:19 AM, Ken'ichi Ohmichi 
wrote:

> Correct validation history is
>
> Essex: "use anything you want!"
> Folsom: "strict asci!"
> [..]
> Juno: "strict asci!"
>

I'm not sure that's quite right. My patch doesn't actually add Unicode
support; that was already added in
825499fffc7a320466e976d2842e175c2d158c0e, which appears to have gone in for
Icehouse.  So:

Essex: Use anything you want
Folsom: Strict ASCII, inconsistent restrictions
Grizzly: Strict ASCII, inconsistent restrictions
Icehouse: Unicode, inconsistent restrictions
Juno: Unicode, consistent restrictions
Kilo (?): Use anything you want

At any rate, if accepting Unicode is an issue, then it's been an issue for
a while.

-- 
Chris St. Pierre
Senior Software Engineer
metacloud.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Kapil Thangavelu]

On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco  wrote:

> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> > This thread [1] has turned more “future focused", so I’m moving the
> conversation to the -dev list where we usually have those sorts of
> discussions.
> >
> > [1]
> http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> >
>
>
> I saw this mentioned in the `openstack` thread but I'd like us to
> reconsider it.
>
> Since we've gone through the "hey please don't deprecate it, I'll help"
> process a couple of times with the zmq driver, I'm thinking we should
> pull it out of the code base anyway.
>

I think the primary issue has been two fold, one is the lack of
reviews/merges on extant patches that fix critical items that have been
outstanding for months. I think that is compounded by the other issue which
was the lack of tests. We've sprinted last week on adding in both unit
tests, the extant patches and functionally verifying them by automating
cloud deployments with zmq for the messaging driver. We're also committing
as a company to supporting it on an ongoing basis. If we get the gates
going for kilo, i don't see any reason for the churn below, if the gates
don't get going we can yank to external in kilo anyway.

cheers,

Kapil


> Here's a rough plan of what I think we should do until the zmq is
> updated and has a proper gate working:
>
> 1. Pull it out the code base into its own repo (stackforge?)
> 2. Setup the basic `oslo.messaging` gates for it
> 3. Release the driver on pypi as: `oslo.messaging.zmq`
> 4. Make lots of noise and make sure people using zmq knows that they
> have to install a separate package now. (I know this means creating a
> new package for many distros).
>
> If there are folks interested in maintaining this driver, they can still
> do it with the driver outside the code base. If it ever gets enough
> attention and maturity, it could be re-proposed for inclusion into the
> code base.
>
> I know there are folks interested in a broker-less solution and we now
> have the not-tested-in-production-yet amqp1.0 driver which I think is a
> very good solution and also shares most of the semantics we rely on
> protocol-wise.
>
> I didn't go through the whole thread so, I'm sorry if I'm repeating
> something that has already been said/proposed/discussed.
>
> Flavio
>
> > On Sep 17, 2014, at 7:54 AM, James Page  wrote:
> >
> >> Signed PGP part
> >> Hi Li
> >>
> >> On 17/09/14 11:58, Li Ma wrote:
>  The scale potential is very appealing and is something I want to
>  test - - hopefully in the next month or so.
> 
>  Canonical are interested in helping to maintain this driver and
>  hopefully we help any critical issues prior to Juno release.
> 
> 
> >>> That sounds good. I just went through all the bugs reported in the
> >>> community.
> >>>
> >>> The only critical bug which makes ZeroMQ malfunction is
> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
> >>> corresponding review is under way:
> >>> https://review.openstack.org/#/c/84938/
> >>
> >> Agreed
> >>
> >>> Others are tagged to 'zmq' in
> >>> https://bugs.launchpad.net/oslo.messaging
> >>
> >> Looking through Doug's suggested list of information and collating
> >> what I know of from our work last week:
> >>
> >> 1) documentation for how to configure and use zeromq with
> >> oslo.messaging (note, not the version in oslo-incubator, the version
> >> in the messaging library repository)
> >>
> >> As part of our sprint, I worked on automating deployment of OpenStack
> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
> >> that work into some general documentation on how best to configure
> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
> >> does not talk about how to configure the oslo-messaging-zmq-receiver
> >> at all.
> >>
> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
> >> cycle to make this a little easier to configure and digest - for
> >> example, right now no systemd unit/upstart configuration/sysv init
> >> script is provided to manage the zmq-receiver.
> >>
> >> I'd also like to document the current design of the ZMQ driver - Doug
> >> - where is the best place todo this? I thought in the source tree
> >> somewhere.
> >
> > The documentation in the oslo.messaging repository [2] would be a good
> place to start for that. If we decide deployers/operators need the
> information we can either refer to it from the guides managed by the
> documentation team or we can move/copy the information. How about if you
> start a new drivers subdirectory there, and add information about zmq. We
> can have other driver authors provide similar detail about their drivers in
> the same directory.
> >
> > [2]
> http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
> >
> >>
> >> 2) a list of the critical bugs that need to be fixed + any existing
> >> patches associated with those bugs, so they

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
> 
> > On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> > 
> >> 
> >> Linux distributions are not the end be all of distribution models and
> >> they don’t get to dictate to upstream.
> > 
> > Well, distributions is where the final user is, and where software gets
> > consumed. Our priority should be the end users.
> 
> 
> Distributions are not the only place that people get their software from,
> unless you think that the ~3 million downloads requests has received
> on PyPI in the last 30 days are distributions downloading requests to
> package in their OSs.
> 

Do pypi users not also need to be able to detect and fix any versions
of libraries they might have? If one has some virtualenvs with various
libraries and apps installed and no --system-site-packages, one would
probably still want to run 'pip freeze' in all of them and find out what
libraries are there and need to be fixed.

Anyway, generally security updates require a comprehensive strategy.
One common comprehensive strategy is version assertion.

Vendoring complicates that immensely.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/17/14, 9:24 PM, "Thomas Goirand"  wrote:

>On 09/18/2014 08:22 AM, Morgan Fainberg wrote:
>> I think that all of the conversation to this point has been valuable,
>> the general consensus is vendoring a library is not as desirable as
>> using it strictly as a dependency. It would be nice in a perfect
>> world if vendoring wasn’t and issue, but in this case I think the
>> root of the matter is that Debian un-vendors urllib3 and we have
>> referenced the vendored urllib3 instead of installing and utilizing
>> urllib3 directly.
>> 
>> This poses at least one problem for us: we are not able to guarantee
>> we’re using the same urllib3 library as requests is. I am unsure how
>> big of a deal this ends up being, but it is a concern and has brought
>> up a question of how to handle this in the most appropriate and
>> consistent way across all of the distributions we as OpenStack support.
>> 
>> Does this make requests a bad library we should toss aside for
>> something else? Instead of being concerned with the reasons for
>> vendoring urllib3 (or un-vendoring it) we should shift the conversation
>> towards two questions:
>> 
>> 1. Is it a real issue if the version of urllib3 is mismatched between
>> our client libraries and requests?
>> 2. If it is a real issue how are we solving it?
>
>The main issue is that urllib3 in requests, as other pointed out, is not
>up-to-date, and will not be updated. In fact, that's the main reason why
>the upstream authors of requests are vendorizing: it's because they
>don't want to carry the burden of staying up-to-date.

How involved are you with requests’ development process? You must not be
very involved because this is the exact opposite reason of why we vendor.
More often that not we pull in urllib3 to get unreleased features that we
build upon for our newest release. If what you said was true, 2.4.{0,1}
would not have the ability to pass socket level options that nova client
decides it wants to set. If we were pinning to an old version of urllib3,
this feature would never be possible. We vendor, because in order to
provide these features we don’t want to have to support the ancient
versions of urllib3 that used to cause us headaches on platforms like
Debian.

>And then, there's incompatibilities and divergences that appear, leading
>to all sorts of unexpected issues, like one thing working with pip, but
>not with the packages. This kind of issues are very hard to understand
>and debug. Distributions may report the issue upstream, then upstream
>will say "but it's working for me", and then we may loose a lot of time.
>This happened already, and may happen again if we don't care enough.
>
>> Obviously we can work with the requests team to figure out the best
>> approach.
>
>There's only a single approach that works: have the requests upstream
>authors to stop embedding foreign code, and use the dependency instead.
>
>> We should focus on the solution here rather than continuing
>> down the path of whether requests should/shouldn’t be vendoring it’s
>> dependencies since it is clear that the team has their reasons and
>> does not want to switch to the dependency model again.
>
>I'm sure they have tons of wrong reasons. If they don't want to change
>anything, then we can only try to work around the issue, and never use
>the embedded version.

The fact is, anyone who doesn’t run their tests in Devstack runs them in a
virtual environment. The line you’re complaining about is actually correct
in that context (the context where pip installs requests). That said, the
test should instead use a conditional import and fallback to the vendored
copy.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/18/14, 2:27 AM, "Chmouel Boudjnah"  wrote:

>Ian Cordasco  writes:
>
>> urllib3 do that automatically. I haven’t started to investigate exactly
>> why they do this. Likewise, glance client has custom certificate
>> verification in glanceclient.common.https. Why? I’m not exactly certain
>
>this probably come from pre-requests port uses when it was using httplib
>which didn't have SSL verification. There is a old wiki page about it
>here https://wiki.openstack.org/wiki/SecureClientConnections
>
>Slightly off-topic, speaking about requests and OpenStack client, it
>would be nice to have Expect/100-Continue feature tackled for
>glanceclient and swiftclient :
>
>https://github.com/kennethreitz/requests/issues/713

I’m glad you linked to that issue Chmouel because it has all of the
information you need to realize that it’s an entirely impossible feature
while still relying on httplib. You can still send that header, but
requests has no meaningful way of special-casing it. The standard library
does not return the header we need to know that we can continue with the
upload and the RFC is wonderfully obscure enough to make requests’ current
behavior correct. It says a client needs to wait for the server to respond
or some amount of time before starting the upload. Granted we don’t call
“sleep” and start the upload immediately, but we have no way of
determining if the server has responded thanks to httplib.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Clint Byrum]

Great job highlighting what our friends over at Amazon are doing.

It's clear from these snippets, and a few other pieces of documentation
for SQS I've read, that the Amazon team approached SQS from a _massive_
scaling perspective. I think what may be forcing a lot of this frustration
with Zaqar is that it was designed with a much smaller scale in mind.

I think as long as that is the case, the design will remain in question.
I'd be comfortable saying that the use cases I've been thinking about
are entirely fine with the limitations SQS has.

Excerpts from Joe Gordon's message of 2014-09-17 13:36:18 -0700:
> Hi All,
> 
> My understanding of Zaqar is that it's like SQS. SQS uses distributed
> queues, which have a few unusual properties [0]:
> Message Order
> 
> Amazon SQS makes a best effort to preserve order in messages, but due to
> the distributed nature of the queue, we cannot guarantee you will receive
> messages in the exact order you sent them. If your system requires that
> order be preserved, we recommend you place sequencing information in each
> message so you can reorder the messages upon receipt.
> At-Least-Once Delivery
> 
> Amazon SQS stores copies of your messages on multiple servers for
> redundancy and high availability. On rare occasions, one of the servers
> storing a copy of a message might be unavailable when you receive or delete
> the message. If that occurs, the copy of the message will not be deleted on
> that unavailable server, and you might get that message copy again when you
> receive messages. Because of this, you must design your application to be
> idempotent (i.e., it must not be adversely affected if it processes the
> same message more than once).
> Message Sample
> 
> The behavior of retrieving messages from the queue depends whether you are
> using short (standard) polling, the default behavior, or long polling. For
> more information about long polling, see Amazon SQS Long Polling
> 
> .
> 
> With short polling, when you retrieve messages from the queue, Amazon SQS
> samples a subset of the servers (based on a weighted random distribution)
> and returns messages from just those servers. This means that a particular
> receive request might not return all your messages. Or, if you have a small
> number of messages in your queue (less than 1000), it means a particular
> request might not return any of your messages, whereas a subsequent request
> will. If you keep retrieving from your queues, Amazon SQS will sample all
> of the servers, and you will receive all of your messages.
> 
> The following figure shows short polling behavior of messages being
> returned after one of your system components makes a receive request.
> Amazon SQS samples several of the servers (in gray) and returns the
> messages from those servers (Message A, C, D, and B). Message E is not
> returned to this particular request, but it would be returned to a
> subsequent request.
> 
> 
> 
> Presumably SQS has these properties because it makes the system scalable,
> if so does Zaqar have the same properties (not just making these same
> guarantees in the API, but actually having these properties in the
> backends)? And if not, why? I looked on the wiki [1] for information on
> this, but couldn't find anything.
> 
> 
> 
> 
> 
> [0]
> http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html
> [1] https://wiki.openstack.org/wiki/Zaqar

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
> 
> Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
>> 
>>> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
>>> 
 
 Linux distributions are not the end be all of distribution models and
 they don’t get to dictate to upstream.
>>> 
>>> Well, distributions is where the final user is, and where software gets
>>> consumed. Our priority should be the end users.
>> 
>> 
>> Distributions are not the only place that people get their software from,
>> unless you think that the ~3 million downloads requests has received
>> on PyPI in the last 30 days are distributions downloading requests to
>> package in their OSs.
>> 
> 
> Do pypi users not also need to be able to detect and fix any versions
> of libraries they might have? If one has some virtualenvs with various
> libraries and apps installed and no --system-site-packages, one would
> probably still want to run 'pip freeze' in all of them and find out what
> libraries are there and need to be fixed.
> 
> Anyway, generally security updates require a comprehensive strategy.
> One common comprehensive strategy is version assertion.
> 
> Vendoring complicates that immensely.

It doesn’t really matter. PyPI doesn’t dictate to projects who host there what
that project is allowed to do except in some very broad circumstances. Whether
or not requests *should* do this doesn't really have any bearing on what
Openstack should do to cope with it. The facts are that requests does it, and
that people pulling things from PyPI is an actual platform that needs thought
about.

This leaves Openstack with a few reasonable/sane options:

1) Decide that vendoring in requests is unacceptable to what Openstack as a
   project is willing to support, and cease the use of requests.
2) Decide that what requests offers is good enough that it outweighs the fact
   that it vendors urllib3 and continue using it.

If the 2nd option is chosen, then doing anything but supporting the fact that
requests vendors urllib3 within the code that openstack writes is hurting the
users who fetch these projects from PyPI because you don't agree with one of
the choices that requests makes. By all means do conditional imports to lessen
the impact that the choice requests has made (and the one that Openstack has
made to use requests) on downstream distributors, but unconditionally importing
from the top level urllib3 for use within requests is flat out wrong.

Obviously neither of these options excludes the choice to lean on requests to
reverse this decision as well. However that is best done elsewhere as the
person making that decision isn't a member of these mailing lists as far as
I am aware.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][db] Ensuring db isolation between tests

[Mike Bayer]

I’ve done a lot of work on this issue and from my perspective, the code is 
mostly ready to go, however we’re in an extended phase of getting folks to sign 
off as well as that I’m waiting for some last-minute fixup from Robert Collins. 
  Patch: [1]  Blueprint, which is to be moved to Kilo: [2]

A “nested” transaction can actually mean one of two things, either a real 
SAVEPOINT, or a logical transaction block.  However, either kind of block will 
be removed if a ROLLBACK is emitted, which also rolls back the actual 
transactional state all the way to the end.The patch above makes this work 
as the result of two fixes.  One is that I replaced the system by which we do 
transactions with the pysqlite driver so that SAVEPOINT actually works [3].  
The other is that I created a comprehensive fixture in [1] that will maintain a 
transaction + savepoint block at all times, working smoothly with tests that 
call commit or rollback any number of times.

From an isolation perspective, we create on-demand databases per process, so 
that each serial test process uses a distinct database per backend.   The 
entire process is managed by testresources which will organize the order of 
tests to most effectively leave a single schema in place with minimal 
teardown/setup.

I’m hoping that my patches can go right in at the top of Kilo and we can begin 
rolling it out in projects that are participating in oslo.db, with the hopes 
that consuming projects will be able to remove a lot of boilerplate 
setup/teardown code. 


1: https://review.openstack.org/#/c/120870/  
2: https://review.openstack.org/#/c/117335/ 
3: https://review.openstack.org/#/c/113152/


On Sep 18, 2014, at 5:59 AM, Salvatore Orlando  wrote:

> Nested commits in sqlalchemy should be seen as a single transaction on the 
> backend, shouldn't they?
> I don't know anything about this specific problem, but the fact that unit 
> tests use sqlite might be a reason, since it's not really a full DBMS...
> 
> I think that wrapping tests in transaction also will require some changes in 
> the architecture of the tests themselves, as many tests call the API router 
> or the plugin which then gets a db session and open a new transaction. 
> Furthermore, it will change the test behaviour possibly hiding errors; some 
> operations indeed perform several distinct transactions, which in this case 
> will be seen a single transaction.
> 
> What Kevin is doing here I think was the original way we used to do that in 
> Neutron (Folsom). Then at some point we realised that due to plugin schema 
> differences we were laving tables behind and switched to drop_all and 
> rebuilding the schema using autogeneration at each test.
> 
> I think it should be ok to merge this patch. I will hold off the +A to give 
> other core reviewers a chance to look at it.
> 
> Salvatore
> 
> 
> On 18 September 2014 11:44, Maru Newby  wrote:
> For legacy reasons, the Neutron test suite creates and destroys a db for each 
> test.  There is a patch proposed to create the tables once and then ensure 
> the tables are wiped at the end of each test [1], providing a performance 
> improvement of ~10%.  I was wondering if this is the best way to provide 
> isolation, since I’ve heard that isolation via per-test transactions should 
> also work.  The patch author reported problems with this approach - 
> apparently nested commits were not being rolled back.  Is there some trick to 
> isolating with transactions that wouldn’t be immediately obvious?
> 
> Thanks,
> 
> 
> Maru
> 
> 1: https://review.openstack.org/#/c/122028/
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/18/14, 9:18 AM, "Clint Byrum"  wrote:

>Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
>> 
>> > On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
>> > 
>> >> 
>> >> Linux distributions are not the end be all of distribution models and
>> >> they don’t get to dictate to upstream.
>> > 
>> > Well, distributions is where the final user is, and where software
>>gets
>> > consumed. Our priority should be the end users.
>> 
>> 
>> Distributions are not the only place that people get their software
>>from,
>> unless you think that the ~3 million downloads requests has received
>> on PyPI in the last 30 days are distributions downloading requests to
>> package in their OSs.
>> 
>
>Do pypi users not also need to be able to detect and fix any versions
>of libraries they might have? If one has some virtualenvs with various
>libraries and apps installed and no --system-site-packages, one would
>probably still want to run 'pip freeze' in all of them and find out what
>libraries are there and need to be fixed.
>
>Anyway, generally security updates require a comprehensive strategy.
>One common comprehensive strategy is version assertion.
>
>Vendoring complicates that immensely.

Except that even OpenStack doesn’t pin requests because of how
extraordinarily stable our API is. While you can argue that Kenneth has
non-standard opinions about his library, Cory and I take backwards
compatibility and stability very seriously. This means anyone can upgrade
to a newer version of requests without worrying that it will be backwards
incompatible. 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[James Polley]

On 18 Sep 2014, at 6:06 pm, Sullivan, Jon Paul  wrote:

>> -Original Message-
>> From: James E. Blair [mailto:cor...@inaugust.com]
>> Sent: 17 September 2014 23:24
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [TripleO] Set WIP for stale patches?
>> 
>> "Sullivan, Jon Paul"  writes:
>> 
>>> I think this highlights exactly why this should be an automated
>>> process.  No errors in application, and no errors in interpretation of
>>> what has happened.
>>> 
>>> So the -1 from Jenkins was a reaction to the comment created by adding
>>> the workflow -1.  This is going to happen on all of the patches that
>>> have their workflow value altered (tests will run, result would be
>>> whatever the result of the test was, of course).
>> 
>> Jenkins only runs tests in reaction to comments if they say "recheck".
> 
> This is not my experience.
> 
> In my experience if the check results are not fresh enough the recheck is 
> automatically run.  I am not on the infra team, so without looking up code I 
> am just guessing, but my guess is that the workflow score change triggers the 
> check on the presumption that it has been approved, not accounting for the 
> recent(ish) update that move wip to the workflow score.
>
>> 
>>> But I also agree that the Jenkins vote should not be included in the
>>> determination of marking a patch WIP, but a human review should (So
>>> Code-Review and not Verified column).
>>> 
>>> And in fact, for the specific example to hand, the last Jenkins vote
>>> was actually a +1, so as I understand it should not have been marked
>>> WIP.
>> 
>> I'd like to help you see the reviews you want to see without
>> externalizing your individual workflow onto contributors.  What tool do
>> you use to find reviews?
>> 
>> If it's gerrit's webui, have you tried using the Review Inbox dashboard?
>> Here it is for the tripleo-image-elements project:
>> 
>>  https://review.openstack.org/#/projects/openstack/tripleo-image-
>> elements,dashboards/important-changes:review-inbox-dashboard
>> 
>> If you would prefer something else, we can customize those dashboards to
>> do whatever you want, including ignoring changes that have not been
>> updated in 2 weeks.
> 
> This is not solely about finding reviews.  It is about pruning stale reviews. 
>  I think the auto-abandon code was excellent at doing this, but alas, it is 
> no more. 

The only reason i'm aware of for pruning apparently stale reviews is to make it 
easier to find and review things that are being actively worked on.

Do you have other reasons for wanting stale reviews pruned?
> 
>> 
>> -Jim
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> Thanks, 
> Jon-Paul Sullivan ☺ Cloud Services - @hpcloud
> 
> Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, 
> Galway.
> Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's 
> Quay, Dublin 2. 
> Registered Number: 361933
> 
> The contents of this message and any attachments to it are confidential and 
> may be legally privileged. If you have received this message in error you 
> should delete it from your system immediately and advise the sender.
> 
> To any recipient of this message within HP, unless otherwise stated, you 
> should consider this message and attachments as "HP CONFIDENTIAL".
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Flavio Percoco]

On 09/18/2014 04:09 PM, Gordon Sim wrote:
> On 09/18/2014 12:31 PM, Flavio Percoco wrote:
>> On 09/17/2014 10:36 PM, Joe Gordon wrote:
>>> My understanding of Zaqar is that it's like SQS. SQS uses distributed
>>> queues, which have a few unusual properties [0]:
>>>
>>>
>>>  Message Order
>>>
>>> Amazon SQS makes a best effort to preserve order in messages, but due to
>>> the distributed nature of the queue, we cannot guarantee you will
>>> receive messages in the exact order you sent them. If your system
>>> requires that order be preserved, we recommend you place sequencing
>>> information in each message so you can reorder the messages upon
>>> receipt.
>>>
>>
>> Zaqar guarantees FIFO. To be more precise, it does that relying on the
>> storage backend ability to do so as well. Depending on the storage used,
>> guaranteeing FIFO may have some performance penalties.
> 
> Would it be accurate to say that at present Zaqar does not use
> distributed queues, but holds all queue data in a storage mechanism of
> some form which may internally distribute that data among servers but
> provides Zaqar with a consistent data model of some form?

I think this is accurate. The queue's distribution depends on the
storage ability to do so and deployers will be able to choose what
storage works best for them based on this as well. I'm not sure how
useful this separation is from a user perspective but I do see the
relevance when it comes to implementation details and deployments.


> [...]
>> As of now, Zaqar fully relies on the storage replication/clustering
>> capabilities to provide data consistency, availability and fault
>> tolerance.
> 
> Is the replication synchronous or asynchronous with respect to client
> calls? E.g. will the response to a post of messages be returned only
> once the replication of those messages is confirmed? Likewise when
> deleting a message, is the response only returned when replicas of the
> message are deleted?

It depends on the driver implementation and/or storage configuration.
For example, in the mongodb driver, we use the default write concern
called "acknowledged". This means that as soon as the message gets to
the master node (note it's not written on disk yet nor replicated) zaqar
will receive a confirmation and then send the response back to the
client. This is also configurable by the deployer by changing the
default write concern in the mongodb uri using `?w=SOME_WRITE_CONCERN`[0].

[0] http://docs.mongodb.org/manual/reference/connection-string/#uri.w

> 
>> However, as far as consuming messages is concerned, it can
>> guarantee once-and-only-once and/or at-least-once delivery depending on
>> the message pattern used to consume messages. Using pop or claims
>> guarantees the former whereas streaming messages out of Zaqar guarantees
>> the later.
> 
> From what I can see, pop provides unreliable delivery (i.e. its similar
> to no-ack). If the delete call using pop fails while sending back the
> response, the messages are removed but didn't get to the client.

Correct, pop works like no-ack. If you want to have pop+ack, it is
possible to claim just 1 message and then delete it.

> 
> What do you mean by 'streaming messages'?

I'm sorry, that went out wrong. I had the "browsability" term in my head
and went with something even worse. By streaming messages I meant
polling messages without claiming them. In other words, at-least-once is
guaranteed by default, whereas once-and-only-once is guaranteed just if
claims are used.

> 
> [...]
>> Based on our short conversation on IRC last night, I understand you're
>> concerned that FIFO may result in performance issues. That's a valid
>> concern and I think the right answer is that it depends on the storage.
>> If the storage has a built-in FIFO guarantee then there's nothing Zaqar
>> needs to do there. In the other hand, if the storage does not have a
>> built-in support for FIFO, Zaqar will cover it in the driver
>> implementation. In the mongodb driver, each message has a marker that is
>> used to guarantee FIFO.
> 
> That marker is a sequence number of some kind that is used to provide
> ordering to queries? Is it generated by the database itself?

It's a sequence number to provide ordering to queries, correct.
Depending on the driver, it may be generated by Zaqar or the database.
In mongodb's case it's generated by Zaqar[0].

[0]
https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/queues.py#L103-L185

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[James E. Blair]

"Sullivan, Jon Paul"  writes:

> This is not solely about finding reviews.  It is about pruning stale
> reviews.  I think the auto-abandon code was excellent at doing this,
> but alas, it is no more.

What's the purpose of pruning stale reviews?  I've read the IRC log of
the meeting you mentioned.  It's becoming apparent to me that this is
about making some numbers that reviewstats produces look good.

I am rather disappointed in that.

If reviewstats is not measuring what you want it to measure (eg, how
well you keep up with incoming reviews) then you should change how it
measures it.  If you want the number of open reviews to be low, then
change the definition of open reviews to what you think it should be --
don't create automated processes to WIP changes just to get your numbers
down.

The reason that we made it so that any core team member could WIP or
abandon a change was so that you could make a judgement call and say
"this change needs more work" or "this change is defunct".  You might
even use a tool to help you find those reviews and make those judgement
calls.  But no automated tool can make those decisions.  Luckily, it
does not need to.

If you want to organize your review queue, use a tool like
gerrit-dash-creator:

  https://github.com/stackforge/gerrit-dash-creator

If you want to change how stats are generated, patch reviewstats.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [MagnetoDB] Async create/delete table

[Illia Khudoshyn]

Hi stackers,

We're about to introduce asynchronous table creation and deletion in
MagnetoDB via oslo.messaging.
The draft specification is availiable by the link below.
https://wiki.openstack.org/wiki/MagnetoDB/specs/async-schema-operations

Please share your thoughts

-- 

Best regards,

Illia Khudoshyn,
Software Engineer, Mirantis, Inc.



38, Lenina ave. Kharkov, Ukraine

www.mirantis.com 

www.mirantis.ru



Skype: gluke_work

ikhudos...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Flavio Percoco]

On 09/18/2014 04:24 PM, Clint Byrum wrote:
> Great job highlighting what our friends over at Amazon are doing.
> 
> It's clear from these snippets, and a few other pieces of documentation
> for SQS I've read, that the Amazon team approached SQS from a _massive_
> scaling perspective. I think what may be forcing a lot of this frustration
> with Zaqar is that it was designed with a much smaller scale in mind.
> 
> I think as long as that is the case, the design will remain in question.
> I'd be comfortable saying that the use cases I've been thinking about
> are entirely fine with the limitations SQS has.

I think these are pretty strong comments with not enough arguments to
defend them.

Saying that Zaqar was designed with a smaller scale in mid without
actually saying why you think so is not fair besides not being true. So
please, do share why you think Zaqar was not designed for big scales and
provide comments that will help the project to grow and improve.

- Is it because the storage technologies that have been chosen?
- Is it because of the API?
- Is it because of the programing language/framework ?

So far, we've just discussed the API semantics and not zaqar's
scalability, which makes your comments even more surprising.

Flavio

> 
> Excerpts from Joe Gordon's message of 2014-09-17 13:36:18 -0700:
>> Hi All,
>>
>> My understanding of Zaqar is that it's like SQS. SQS uses distributed
>> queues, which have a few unusual properties [0]:
>> Message Order
>>
>> Amazon SQS makes a best effort to preserve order in messages, but due to
>> the distributed nature of the queue, we cannot guarantee you will receive
>> messages in the exact order you sent them. If your system requires that
>> order be preserved, we recommend you place sequencing information in each
>> message so you can reorder the messages upon receipt.
>> At-Least-Once Delivery
>>
>> Amazon SQS stores copies of your messages on multiple servers for
>> redundancy and high availability. On rare occasions, one of the servers
>> storing a copy of a message might be unavailable when you receive or delete
>> the message. If that occurs, the copy of the message will not be deleted on
>> that unavailable server, and you might get that message copy again when you
>> receive messages. Because of this, you must design your application to be
>> idempotent (i.e., it must not be adversely affected if it processes the
>> same message more than once).
>> Message Sample
>>
>> The behavior of retrieving messages from the queue depends whether you are
>> using short (standard) polling, the default behavior, or long polling. For
>> more information about long polling, see Amazon SQS Long Polling
>> 
>> .
>>
>> With short polling, when you retrieve messages from the queue, Amazon SQS
>> samples a subset of the servers (based on a weighted random distribution)
>> and returns messages from just those servers. This means that a particular
>> receive request might not return all your messages. Or, if you have a small
>> number of messages in your queue (less than 1000), it means a particular
>> request might not return any of your messages, whereas a subsequent request
>> will. If you keep retrieving from your queues, Amazon SQS will sample all
>> of the servers, and you will receive all of your messages.
>>
>> The following figure shows short polling behavior of messages being
>> returned after one of your system components makes a receive request.
>> Amazon SQS samples several of the servers (in gray) and returns the
>> messages from those servers (Message A, C, D, and B). Message E is not
>> returned to this particular request, but it would be returned to a
>> subsequent request.
>>
>>
>>
>> Presumably SQS has these properties because it makes the system scalable,
>> if so does Zaqar have the same properties (not just making these same
>> guarantees in the API, but actually having these properties in the
>> backends)? And if not, why? I looked on the wiki [1] for information on
>> this, but couldn't find anything.
>>
>>
>>
>>
>>
>> [0]
>> http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html
>> [1] https://wiki.openstack.org/wiki/Zaqar
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Flavio Percoco]

On 09/18/2014 04:16 PM, Kapil Thangavelu wrote:
> 
> 
> On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco  > wrote:
> 
> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> > This thread [1] has turned more “future focused", so I’m moving the 
> conversation to the -dev list where we usually have those sorts of 
> discussions.
> >
> > [1] 
> http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> >
> 
> 
> I saw this mentioned in the `openstack` thread but I'd like us to
> reconsider it.
> 
> Since we've gone through the "hey please don't deprecate it, I'll help"
> process a couple of times with the zmq driver, I'm thinking we should
> pull it out of the code base anyway.
> 
> 
> I think the primary issue has been two fold, one is the lack of
> reviews/merges on extant patches that fix critical items that have been
> outstanding for months. I think that is compounded by the other issue
> which was the lack of tests. We've sprinted last week on adding in both
> unit tests, the extant patches and functionally verifying them by
> automating cloud deployments with zmq for the messaging driver. We're
> also committing as a company to supporting it on an ongoing basis. If we
> get the gates going for kilo, i don't see any reason for the churn
> below, if the gates don't get going we can yank to external in kilo anyway.

The above sounds good to me. Thanks for sharing.

Can we have a deadline for all this? Last time we discussed zmq driver's
faith, we said that k-1 was a good deadline for it to get back in shape.
If the above plan sounds good to other folks, then I'd prefer us to
stick to k-1 for all the above to happen.

Thoughts?

Thanks again, Kapil
Flavio

>  
> cheers,
> 
> Kapil
> 
> 
> Here's a rough plan of what I think we should do until the zmq is
> updated and has a proper gate working:
> 
> 1. Pull it out the code base into its own repo (stackforge?)
> 2. Setup the basic `oslo.messaging` gates for it
> 3. Release the driver on pypi as: `oslo.messaging.zmq`
> 4. Make lots of noise and make sure people using zmq knows that they
> have to install a separate package now. (I know this means creating a
> new package for many distros).
> 
> If there are folks interested in maintaining this driver, they can still
> do it with the driver outside the code base. If it ever gets enough
> attention and maturity, it could be re-proposed for inclusion into the
> code base.
> 
> I know there are folks interested in a broker-less solution and we now
> have the not-tested-in-production-yet amqp1.0 driver which I think is a
> very good solution and also shares most of the semantics we rely on
> protocol-wise.
> 
> I didn't go through the whole thread so, I'm sorry if I'm repeating
> something that has already been said/proposed/discussed.
> 
> Flavio
> 
> > On Sep 17, 2014, at 7:54 AM, James Page  > wrote:
> >
> >> Signed PGP part
> >> Hi Li
> >>
> >> On 17/09/14 11:58, Li Ma wrote:
>  The scale potential is very appealing and is something I want to
>  test - - hopefully in the next month or so.
> 
>  Canonical are interested in helping to maintain this driver and
>  hopefully we help any critical issues prior to Juno release.
> 
> 
> >>> That sounds good. I just went through all the bugs reported in the
> >>> community.
> >>>
> >>> The only critical bug which makes ZeroMQ malfunction is
> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
> >>> corresponding review is under way:
> >>> https://review.openstack.org/#/c/84938/
> >>
> >> Agreed
> >>
> >>> Others are tagged to 'zmq' in
> >>> https://bugs.launchpad.net/oslo.messaging
> >>
> >> Looking through Doug's suggested list of information and collating
> >> what I know of from our work last week:
> >>
> >> 1) documentation for how to configure and use zeromq with
> >> oslo.messaging (note, not the version in oslo-incubator, the version
> >> in the messaging library repository)
> >>
> >> As part of our sprint, I worked on automating deployment of OpenStack
> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
> >> that work into some general documentation on how best to configure
> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
> >> does not talk about how to configure the oslo-messaging-zmq-receiver
> >> at all.
> >>
> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
> >> cycle to make this a little easier to configure and digest - for
> >> example, right now no systemd unit/upstart configuration/sysv init
> >> script is provided to manage the zmq-receiver.
> >>
> >> I'd also like to document t

Re: [openstack-dev] [blazar]: proposal for a new lease type

[Lisa]

Hi Sylvain,

thanks a lot for your answer.

I'd like to extend (if possible) the BLAZAR's list of the supported 
lease types by adding a new one which covers a specific use case which, 
it seems, it is not yet supported by BLAZAR. You know, In OpenStack 
every project has a granted fixed quota which cannot be extended 
dynamically. This implies that a project cannot allocate unused 
resources assigned to different projects. The idea is to lease such 
unused resources just for a limited duration time (e.g. from few hours 
to 1 day). Suppose the project "A" has consumed all own assigned 
resources while the project "B", in the current time, has several 
available resources. "B" may offer its own resources to "A" just for a 
limited and well known time duration (i.e. not forever) so that in the 
future it can claim, if needed, the available resources belonging to "A" 
(or other projects).
This is a fair-share mechanism which guarantees the resources usage is 
equally distributed among users and projects by considering the portion 
of the resources allocated to them (i.e. share) and the resources 
already consumed. Please remark the "share" is a different concept than 
"quota" in the cloud terminology. You can consider this proposal as a 
way to update dynamically the quotas by considering the historical usage 
of each project. This approach should make OpenStack very efficient in 
terms of resource utilization.


About the blueprint I will try to create a new one.

Thanks again.
Cheers,
Lisa

On 18/09/2014 16:00, Sylvain Bauza wrote:


Le 18/09/2014 15:27, Lisa a écrit :

Hi all,

my name is Lisa Zangrando and I work at the Italian National 
Institute for Nuclear Physics (INFN). In particular I am leading a 
team which is addressing the issue concerning the efficiency in the 
resource usage in OpenStack.
Currently OpenStack allows just a static partitioning model where the 
resource allocation to the user teams (i.e. the projects) can be done 
only by considering fixed quotas which cannot be exceeded even if 
there are unused resources (but) assigned to different projects.
We studied the available BLAZAR's documentation and, in agreement 
with Tim Bell (who is responsible the OpenStack cloud project at 
CERN), we think this issue could be addressed within your framework.
Please find attached a document that describes our use cases 
(actually we think that many other environments have to deal with the 
same problems) and how they could be managed in BLAZAR, by defining a 
new lease type (i.e. fairShare lease) to be considered as extension 
of the list of the already supported lease types.

I would then be happy to discuss these ideas with you.

Thanks in advance,
Lisa



Hi Lisa,

Glad to see you're interested in Blazar.

I tried to go thru your proposal, but could you please post the main 
concepts of what you plan to add into an etherpad and create a 
blueprint [1] mapped to it so we could discuss on the implementation ?
Of course, don't hesitate to ping me or the blazar community in 
#openstack-blazar if you need help with the process or the current 
Blazar design.


Thanks,
-Sylvain

[1] https://blueprints.launchpad.net/blazar/




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Davanum Srinivas]

Kapil,

I see just 2 relevant reviews for zmq in the oslo.messaging queue:
https://review.openstack.org/#/q/project:openstack/oslo.messaging+status:open+file:%255E.*zmq.*,n,z

Are there others i am missing? ("fix critical items", "tests" from your email)

thanks,
dims

On Thu, Sep 18, 2014 at 10:16 AM, Kapil Thangavelu
 wrote:
>
>
> On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco  wrote:
>>
>> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
>> > This thread [1] has turned more “future focused", so I’m moving the
>> > conversation to the -dev list where we usually have those sorts of
>> > discussions.
>> >
>> > [1]
>> > http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
>> >
>>
>>
>> I saw this mentioned in the `openstack` thread but I'd like us to
>> reconsider it.
>>
>> Since we've gone through the "hey please don't deprecate it, I'll help"
>> process a couple of times with the zmq driver, I'm thinking we should
>> pull it out of the code base anyway.
>
>
> I think the primary issue has been two fold, one is the lack of
> reviews/merges on extant patches that fix critical items that have been
> outstanding for months. I think that is compounded by the other issue which
> was the lack of tests. We've sprinted last week on adding in both unit
> tests, the extant patches and functionally verifying them by automating
> cloud deployments with zmq for the messaging driver. We're also committing
> as a company to supporting it on an ongoing basis. If we get the gates going
> for kilo, i don't see any reason for the churn below, if the gates don't get
> going we can yank to external in kilo anyway.
>
> cheers,
>
> Kapil
>
>>
>> Here's a rough plan of what I think we should do until the zmq is
>> updated and has a proper gate working:
>>
>> 1. Pull it out the code base into its own repo (stackforge?)
>> 2. Setup the basic `oslo.messaging` gates for it
>> 3. Release the driver on pypi as: `oslo.messaging.zmq`
>> 4. Make lots of noise and make sure people using zmq knows that they
>> have to install a separate package now. (I know this means creating a
>> new package for many distros).
>>
>> If there are folks interested in maintaining this driver, they can still
>> do it with the driver outside the code base. If it ever gets enough
>> attention and maturity, it could be re-proposed for inclusion into the
>> code base.
>>
>> I know there are folks interested in a broker-less solution and we now
>> have the not-tested-in-production-yet amqp1.0 driver which I think is a
>> very good solution and also shares most of the semantics we rely on
>> protocol-wise.
>>
>> I didn't go through the whole thread so, I'm sorry if I'm repeating
>> something that has already been said/proposed/discussed.
>>
>> Flavio
>>
>> > On Sep 17, 2014, at 7:54 AM, James Page  wrote:
>> >
>> >> Signed PGP part
>> >> Hi Li
>> >>
>> >> On 17/09/14 11:58, Li Ma wrote:
>>  The scale potential is very appealing and is something I want to
>>  test - - hopefully in the next month or so.
>> 
>>  Canonical are interested in helping to maintain this driver and
>>  hopefully we help any critical issues prior to Juno release.
>> 
>> 
>> >>> That sounds good. I just went through all the bugs reported in the
>> >>> community.
>> >>>
>> >>> The only critical bug which makes ZeroMQ malfunction is
>> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
>> >>> corresponding review is under way:
>> >>> https://review.openstack.org/#/c/84938/
>> >>
>> >> Agreed
>> >>
>> >>> Others are tagged to 'zmq' in
>> >>> https://bugs.launchpad.net/oslo.messaging
>> >>
>> >> Looking through Doug's suggested list of information and collating
>> >> what I know of from our work last week:
>> >>
>> >> 1) documentation for how to configure and use zeromq with
>> >> oslo.messaging (note, not the version in oslo-incubator, the version
>> >> in the messaging library repository)
>> >>
>> >> As part of our sprint, I worked on automating deployment of OpenStack
>> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
>> >> that work into some general documentation on how best to configure
>> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
>> >> does not talk about how to configure the oslo-messaging-zmq-receiver
>> >> at all.
>> >>
>> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
>> >> cycle to make this a little easier to configure and digest - for
>> >> example, right now no systemd unit/upstart configuration/sysv init
>> >> script is provided to manage the zmq-receiver.
>> >>
>> >> I'd also like to document the current design of the ZMQ driver - Doug
>> >> - where is the best place todo this? I thought in the source tree
>> >> somewhere.
>> >
>> > The documentation in the oslo.messaging repository [2] would be a good
>> > place to start for that. If we decide deployers/operators need the
>> > information we can either refer to it from the guides managed by th

Re: [openstack-dev] Oslo final releases ready

[Davanum Srinivas]

w00t!

-- dims

On Thu, Sep 18, 2014 at 10:04 AM, Doug Hellmann  wrote:
> All of the final releases for the Oslo libraries for the Juno cycle are 
> available on PyPI. I’m working on a couple of patches to the global 
> requirements list to update the baseline in the applications. In all cases, 
> the final release is a second tag on a previously released version.
>
> - oslo.config - 1.4.0 (same as 1.4.0.0a5)
> - oslo.db - 1.0.0 (same as 0.5.0)
> - oslo.i18n - 1.0.0 (same as 0.4.0)
> - oslo.messaging - 1.4.0 (same as 1.4.0.0a5)
> - oslo.rootwrap - 1.3.0 (same as 1.3.0.0a3)
> - oslo.serialization - 1.0.0 (same as 0.3.0)
> - oslosphinx - 2.2.0 (same as 2.2.0.0a3)
> - oslotest - 1.1.0 (same as 1.1.0.0a2)
> - oslo.utils - 1.0.0 (same as 0.3.0)
> - cliff - 1.7.0 (previously tagged, so not a new release)
> - stevedore - 1.0.0 (same as 1.0.0.0a2)
>
> Congratulations and *Thank You* to the Oslo team for doing an amazing job 
> with graduations this cycle!
>
> Doug
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-dev[[Congress]

[Tim Hinrichs]

You are right that we don't have those built ins hooked up yet. But there are 
patches in review that do that. If you would like to help let us know!

Tim

P. S. Pardon the brevity. Sent from my mobile. 

> On Sep 17, 2014, at 10:52 PM, "Madhu Mohan"  wrote:
> 
> Hi Congress Team,
> 
> Has anyone tried writing policies with Builtin operators like gt(), lt().
> 
> I see the class CongressBuiltinCategoryMap() to add/update the start map 
> defined in the same file.
> 
> However, these classes are never referred in any part of the code. So 
> clearly, I can assume Congress does not support such operators at this point
> 
> Is there a plan to implement this feature or someone looking into it already ?
> 
> Thanks and Regards,
> Madhu Mohan
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Steve Lewis]

On September 18, 2014 7:45 AM, Flavio Percoco wrote:

> On 09/18/2014 04:09 PM, Gordon Sim wrote:
>
>> However, as far as consuming messages is concerned, it can
>> guarantee once-and-only-once and/or at-least-once delivery depending on
>> the message pattern used to consume messages. Using pop or claims
>> guarantees the former whereas streaming messages out of Zaqar guarantees
>> the later.
>>
>> From what I can see, pop provides unreliable delivery (i.e. its similar
>> to no-ack). If the delete call using pop fails while sending back the
>> response, the messages are removed but didn't get to the client.
> 
> Correct, pop works like no-ack. If you want to have pop+ack, it is
> possible to claim just 1 message and then delete it.

Having some experience using SQS I would expect that there would be a mode
something like what SQS provides where a message gets picked up by one 
consumer (let's say by short-polling) and is hidden for a timeout duration from 
other consumers, so that the consumer has time to ack. Is that how you are 
using the term 'claim' in this case?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Flavio Percoco]

On 09/18/2014 05:42 PM, Steve Lewis wrote:
> On September 18, 2014 7:45 AM, Flavio Percoco wrote:
> 
>> On 09/18/2014 04:09 PM, Gordon Sim wrote:
>>
>>> However, as far as consuming messages is concerned, it can
>>> guarantee once-and-only-once and/or at-least-once delivery depending on
>>> the message pattern used to consume messages. Using pop or claims
>>> guarantees the former whereas streaming messages out of Zaqar guarantees
>>> the later.
>>>
>>> From what I can see, pop provides unreliable delivery (i.e. its similar
>>> to no-ack). If the delete call using pop fails while sending back the
>>> response, the messages are removed but didn't get to the client.
>>
>> Correct, pop works like no-ack. If you want to have pop+ack, it is
>> possible to claim just 1 message and then delete it.
> 
> Having some experience using SQS I would expect that there would be a mode
> something like what SQS provides where a message gets picked up by one 
> consumer (let's say by short-polling) and is hidden for a timeout duration 
> from 
> other consumers, so that the consumer has time to ack. Is that how you are 
> using the term 'claim' in this case?

Correct, that's the name of the feature in Zaqar. You can find a bit of
extra information here[0]. Hope this helps.

[0] https://wiki.openstack.org/wiki/Zaqar/specs/api/v1.1#Claims

Cheers,
Flavio

-- 
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Devananda van der Veen]

On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco  wrote:
> On 09/18/2014 04:09 PM, Gordon Sim wrote:
>> On 09/18/2014 12:31 PM, Flavio Percoco wrote:
>>> Zaqar guarantees FIFO. To be more precise, it does that relying on the
>>> storage backend ability to do so as well. Depending on the storage used,
>>> guaranteeing FIFO may have some performance penalties.
>>
>> Would it be accurate to say that at present Zaqar does not use
>> distributed queues, but holds all queue data in a storage mechanism of
>> some form which may internally distribute that data among servers but
>> provides Zaqar with a consistent data model of some form?
>
> I think this is accurate. The queue's distribution depends on the
> storage ability to do so and deployers will be able to choose what
> storage works best for them based on this as well. I'm not sure how
> useful this separation is from a user perspective but I do see the
> relevance when it comes to implementation details and deployments.

Guaranteeing FIFO and not using a distributed queue architecture
*above* the storage backend are both scale-limiting design choices.
That Zaqar's scalability depends on the storage back end is not a
desirable thing in a cloud-scale messaging system in my opinion,
because this will prevent use at scales which can not be accommodated
by a single storage back end.

And based on my experience consulting for companies whose needs grew
beyond the capabilities of a single storage backend, moving to
application-aware sharding required a significant amount of
rearchitecture in most cases.

-Devananda

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceilometer] Performance degradation (docs) for sampling rate less than 10 mins

[Srikanta Patanjali]

Hi Team,

I was considering to change the default sampling rate of the Ceilometer
from 10 mins to less than that. I foresee an adverse impact on its
performance due to increase in the data inside the MondoDB.

I was wondering if the QA team (or anyone else) has done any of the load
tests with these parameters ? It would be helpful to have access to these
results (if any).

If not, I would like to know the views on increasing the sample rate value
(in the pipleline.yaml file)

Cheers,
Srikanta
InIT ¦ ICCLab
ZHAW
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Devananda van der Veen]

On Thu, Sep 18, 2014 at 7:55 AM, Flavio Percoco  wrote:
> On 09/18/2014 04:24 PM, Clint Byrum wrote:
>> Great job highlighting what our friends over at Amazon are doing.
>>
>> It's clear from these snippets, and a few other pieces of documentation
>> for SQS I've read, that the Amazon team approached SQS from a _massive_
>> scaling perspective. I think what may be forcing a lot of this frustration
>> with Zaqar is that it was designed with a much smaller scale in mind.
>>
>> I think as long as that is the case, the design will remain in question.
>> I'd be comfortable saying that the use cases I've been thinking about
>> are entirely fine with the limitations SQS has.
>
> I think these are pretty strong comments with not enough arguments to
> defend them.
>

Please see my prior email. I agree with Clint's assertions here.

> Saying that Zaqar was designed with a smaller scale in mid without
> actually saying why you think so is not fair besides not being true. So
> please, do share why you think Zaqar was not designed for big scales and
> provide comments that will help the project to grow and improve.
>
> - Is it because the storage technologies that have been chosen?
> - Is it because of the API?
> - Is it because of the programing language/framework ?

It is not because of the storage technology or because of the
programming language.

> So far, we've just discussed the API semantics and not zaqar's
> scalability, which makes your comments even more surprising.

- guaranteed message order
- not distributing work across a configurable number of back ends

These are scale-limiting design choices which are reflected in the
API's characteristics.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] OpenStack bootstrapping hour - Friday Sept 19th - 3pm EST

[Sean Dague]

On 09/15/2014 06:56 PM, Sean Dague wrote:
> A few of us have decided to pull together a regular (cadence to be
> determined) video series taking on deep dives inside of OpenStack,
> looking at code, explaining why things work that way, and fielding
> questions from anyone interested.
> 
> For lack of a better title, I've declared it OpenStack Bootstrapping Hour.
> 
> Episode 0 - Mock best practices will kick off this Friday, Sept 19th,
> from 3pm - 4pm EST. Our experts for this will be Jay Pipes and Dan
> Smith. It will be done as a Google Hangout on Air, which means there
> will be a live youtube stream while it's on, and a recorded youtube
> video that's publicly accessible once we're done.
> 
> We'll be using an etherpad during the broadcast to provide links to the
> content people are looking at, as well as capture questions. That will
> be our backchannel, and audience participation forum, with the advantage
> that it creates a nice concise document at the end of the broadcast that
> pairs well with the video. (Also: the tech test showed that while code
> examples are perfectly viewable during in the final video, during the
> live stream they are a little hard to read, etherpad links will help
> people follow along at home).
> 
> Assuming this turns out to be useful, we're thinking about lots of other
> deep dives. The intent is that these are indepth dives. We as a
> community have learned so many things over the last 4 years, but as
> OpenStack has gotten so large, being familiar with more than a narrow
> slice is hard. This is hopefully a part of the solution to address that.
> As I've told others, if nothing else, I'm looking forward to learning a
> ton in the process.
> 
> Final links for the hangout + etherpad will be posted a little later in
> the week. Mostly wanted to make people aware it was coming.
> 
>   -Sean

Links for the event are now up in the wiki:
https://wiki.openstack.org/wiki/BootstrappingHour#Next_Episode -
including the youtube stream url and the etherpad.

Look forward to seeing folks tomorrow.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Alex Xu]

On 2014年09月18日 18:57, Sean Dague wrote:

On 09/18/2014 06:38 AM, Christopher Yeoh wrote:

On Sat, 13 Sep 2014 06:48:19 -0400
Sean Dague  wrote:


On 09/13/2014 02:28 AM, Kenichi Oomichi wrote:

Hi Chris,

Thanks for bring it up here,


-Original Message-
From: Chris St. Pierre [mailto:stpie...@metacloud.com]
Sent: Saturday, September 13, 2014 2:53 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [nova] Expand resource name allowed
characters

We have proposed that the allowed characters for all resource
names in Nova (flavors, aggregates, etc.) be expanded to all
printable unicode characters and horizontal spaces:
https://review.openstack.org/#/c/119741

Currently, the only allowed characters in most resource names are
alphanumeric, space, and [.-_].


We have proposed this change for two principal reasons:

1. We have customers who have migrated data forward since Essex,
when no restrictions were in place, and thus have characters in
resource names that are disallowed in the current version of
OpenStack. This is only likely to be useful to people migrating
from Essex or earlier, since the current restrictions were added
in Folsom.

2. It's pretty much always a bad idea to add unnecessary
restrictions without a good reason. While we don't have an
immediate need to use, for example, the ever-useful
http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
up with a reason people *shouldn't* be allowed to use it.

That said, apparently people have had a need to not be allowed to
use some characters, but it's not clear why:
https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
knows any reason why these printable characters should not be
joined in holy resource naming, speak now or forever hold your
peace.

I also could not find the reason of current restriction on the bug
report, and I'd like to know it as the history.
On v2 API(not v2.1), each resource name contains the following
restriction for its name:

   Resource  | Length  | Pattern
  ---+-+--
   aggregate | 1-255   | nothing
   backup| nothing | nothing
   flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
 | |   [a-zA-Z0-9. _-]*$'
   keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
   server| 1-255   | nothing
   cell  | 1-255   | don't contain "." and "!"

On v2.1 API, we have applied the same restriction rule[1] for whole
resource names for API consistency, so maybe we need to consider
this topic for whole names.

[1]:
https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44

Honestly, I bet this had to do with how the database used to be set
up.


So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
multibyte characters (only 1-3 bytes). For example if you do a create
image call with an image name to glance with a 4 byte multibyte
character in the name it will 500. I'd guess we do something
similar in places with the Nova API where we have inadequate input
validation. If you want 4 byte support you need to use utf8mb4 instead.

Oh... fun. :(


I don't know if postgresql has any restrictions (I don't think it
does) or if db2 does too. But I don't think we can/should make it a
complete free for all. It should at most be what most databases support.

I think its a big enough change that this late in the cycle we should
push it off to Kilo. It's always much easier to loosen input validation
than tighten it (or have to have an "oops" revert on an officially
released Nova). Perhaps some tests to verify that everything we allow
past the input validation checks we can actually store.

So, honestly, that seems like a pendulum swing in an odd way.

Havana "use anything you want!"
Icehouse ?
Juno "strict asci!"
Kilo "utf8"

Can't we just catch the db exception correctly in glance and not have it
explode? And then allow it. Exploding with a 500 on a bad name seems the
wrong thing to do anyway.

That would also mean that if the user changed their database to support
utf8mb4 (which they might want to do if it was important to them) it
would all work.

I think some release notes would be fine to explain the current
situation and limitations.

Basically, lets skate towards the puck here, realizing some corner cases
exist, but that we're moving in the direction we want to be, not back
tracking.

When we can return the json-schema to user in the future, can we say 
that means API accepting utf8 or utf8mb4 is discoverable? If it is 
discoverable, then we needn't limit anything in our python code.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Rally] Load testing of dependent resources

[Boris Pavlovic]

Ilya,

It sounds good, moreover few other devs asked about similar feature=)
So I am working on proposal how to implement this=)


Best regards,



On Thu, Sep 18, 2014 at 4:35 PM, Ilya Shakhat  wrote:

> Ralliers,
>
> I'd like to share with you an idea on how to test dependent resources.
>
> Let's consider the following example: there is a network and ports
> belonging to it and we would like to test port creation process. Currently
> I see 2 options of writing scenario:
>  a) The scenario creates network and then creates a specified number of
> ports. Runner executes the scenario specified number of times. -- In this
> case ports belonging to different networks are created in parallel, but all
> ports belonging to the same network are created serially.
>  b) The network already exists and the scenario just creates ports, --
> This case allows to test concurrent creation of ports belonging to the same
> network but not between multiple networks.
> It would be really cool to get benefits from both cases, i.e. to test
> parallel port creation but without limiting to parent network resource.
>
> One of possible approaches may be to construct chain of scenarios where
> preceding scenario prepares context for the further one. From the above
> example the execution would be:
>  1) The task contains sequence of 2 scenarios:
> * network creation - it creates network and returns it
> * port creation - it picks the network from incoming params and
> creates port on it
>  2) The runner has execution pipeline (which currently is represented as
> generator producing identical scenario runners). The pipeline starts with
> sequence of network creation scenario runners. Once one of runner finishes,
> it then puts the further scenario and the result into pipeline. The
> pipeline ends once no further scenarios left.
>
> From the above example execution flow would be like:
>  * pipeline is "filled" with N network scenarios (not actually filled, but
> contains a generator that is able to do this)
>  * R runners pick scenarios from pipeline
>  * pipeline contains N-R network scenarios
>  * one of runners finishes the scenario and updates pipeline with P port
> scenarios
>  * pipeline contains N-R-1 network scenarios followed by P port scenarios
>  * work-work-work until pipeline is empty
>
> This execution covers case from a) and b) but there still will be
> sequences of child resources belonging to the same parent. The improvement
> may be to pick scenario from pipeline randomly.
>
> How does it sound?
>
> Thanks,
> Ilya
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Expand resource name allowed characters

[Sean Dague]

On 09/18/2014 12:08 PM, Alex Xu wrote:
> On 2014年09月18日 18:57, Sean Dague wrote:
>> On 09/18/2014 06:38 AM, Christopher Yeoh wrote:
>>> On Sat, 13 Sep 2014 06:48:19 -0400
>>> Sean Dague  wrote:
>>>
 On 09/13/2014 02:28 AM, Kenichi Oomichi wrote:
> Hi Chris,
>
> Thanks for bring it up here,
>
>> -Original Message-
>> From: Chris St. Pierre [mailto:stpie...@metacloud.com]
>> Sent: Saturday, September 13, 2014 2:53 AM
>> To: openstack-dev@lists.openstack.org
>> Subject: [openstack-dev] [nova] Expand resource name allowed
>> characters
>>
>> We have proposed that the allowed characters for all resource
>> names in Nova (flavors, aggregates, etc.) be expanded to all
>> printable unicode characters and horizontal spaces:
>> https://review.openstack.org/#/c/119741
>>
>> Currently, the only allowed characters in most resource names are
>> alphanumeric, space, and [.-_].
>>
>>
>> We have proposed this change for two principal reasons:
>>
>> 1. We have customers who have migrated data forward since Essex,
>> when no restrictions were in place, and thus have characters in
>> resource names that are disallowed in the current version of
>> OpenStack. This is only likely to be useful to people migrating
>> from Essex or earlier, since the current restrictions were added
>> in Folsom.
>>
>> 2. It's pretty much always a bad idea to add unnecessary
>> restrictions without a good reason. While we don't have an
>> immediate need to use, for example, the ever-useful
>> http://codepoints.net/U+1F4A9 in a flavor name, it's hard to come
>> up with a reason people *shouldn't* be allowed to use it.
>>
>> That said, apparently people have had a need to not be allowed to
>> use some characters, but it's not clear why:
>> https://bugs.launchpad.net/nova/+bug/977187 So I guess if anyone
>> knows any reason why these printable characters should not be
>> joined in holy resource naming, speak now or forever hold your
>> peace.
> I also could not find the reason of current restriction on the bug
> report, and I'd like to know it as the history.
> On v2 API(not v2.1), each resource name contains the following
> restriction for its name:
>
>Resource  | Length  | Pattern
>   ---+-+--
>aggregate | 1-255   | nothing
>backup| nothing | nothing
>flavor| 1-255   | '^[a-zA-Z0-9. _-]*[a-zA-Z0-9_-]+
>  | |   [a-zA-Z0-9. _-]*$'
>keypair   | 1-255   | '^[a-zA-Z0-9 _-]+$'
>server| 1-255   | nothing
>cell  | 1-255   | don't contain "." and "!"
>
> On v2.1 API, we have applied the same restriction rule[1] for whole
> resource names for API consistency, so maybe we need to consider
> this topic for whole names.
>
> [1]:
> https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L44
>
 Honestly, I bet this had to do with how the database used to be set
 up.

>>> So it turns out that utf8 support in MySQL does not support UTF-8 4 byte
>>> multibyte characters (only 1-3 bytes). For example if you do a create
>>> image call with an image name to glance with a 4 byte multibyte
>>> character in the name it will 500. I'd guess we do something
>>> similar in places with the Nova API where we have inadequate input
>>> validation. If you want 4 byte support you need to use utf8mb4 instead.
>> Oh... fun. :(
>>
>>> I don't know if postgresql has any restrictions (I don't think it
>>> does) or if db2 does too. But I don't think we can/should make it a
>>> complete free for all. It should at most be what most databases support.
>>>
>>> I think its a big enough change that this late in the cycle we should
>>> push it off to Kilo. It's always much easier to loosen input validation
>>> than tighten it (or have to have an "oops" revert on an officially
>>> released Nova). Perhaps some tests to verify that everything we allow
>>> past the input validation checks we can actually store.
>> So, honestly, that seems like a pendulum swing in an odd way.
>>
>> Havana "use anything you want!"
>> Icehouse ?
>> Juno "strict asci!"
>> Kilo "utf8"
>>
>> Can't we just catch the db exception correctly in glance and not have it
>> explode? And then allow it. Exploding with a 500 on a bad name seems the
>> wrong thing to do anyway.
>>
>> That would also mean that if the user changed their database to support
>> utf8mb4 (which they might want to do if it was important to them) it
>> would all work.
>>
>> I think some release notes would be fine to explain the current
>> situation and limitations.
>>
>> Basically, lets skate towards the puck here, realizing some corner cases
>> exist, but that we're moving in the direction we want to be, not back
>> tracking.
>>
> When we can retur

Re: [openstack-dev] [Keystone][Horizon] CORS and Federation

[Adam Young]

On 09/17/2014 11:53 AM, Marek Denis wrote:

Hi,

First of all, we should clarify whether your JS client wants to 
implement ECP or WebSSO workflow. They are slightly different.


ECP seems to be poorly supported in live deployments, and we cannot 
count on it. WebSSO is the first round.  We should do  ECP as a second step.




I feel JS is smart enough to implement the ECP flow and then and it 
could simply implement what we already have in the keystoneclient [0]. 
This + some "discovery service" described by Adam would be required. 
However, some problems may arise when it comes to ADFS  support (we 
needed separate plugin in keystoneclient) and other protocols which 
should work by default from browsers.


[0] 
https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/auth/v3/saml2.py#L29


Rest of the comments inlined.

On 17.09.2014 17:00, Adam Young wrote:

On 09/17/2014 10:35 AM, David Chadwick wrote:
this would work as well, but wouldn't it require two different API 
calls?


I think it would be 2 calls no matter what.

OK,  lets talk this through:

1. Configure Horizon to return a generic login page, with a button that
says "Or do Federated"
2.  Use clicks that button and gets the Federated UI.  No new HTTP
request needed for this, still just static Javascript.  Form has an edit
field for the user to enter the SAML IdP, anda button to fetch the list
of the public IdPs from Keystone.
3.  Assume user clicks the list of public IdPs, those fill out a
drop-down box.  If the user clicks on one, it populates the textbox with
the URL for the IdP.
3a.  However, if the users IdP is not in the list, they go back to the
email they got from their IT guy that has "Paste this URL into the IdP
edit box"


Well, we don't keep any IdPs' URL in Keystone backend at the moment.
However, this can be easily fixed.

Right.  That is a feature request.





4. User clicks OK.


OK


Window pops up with the WebUI for the user to visit the SAML IdP URL.
This will be of the form
|
GET
htps://keystone/main/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth| 



Which will perform the necessary redirects to get the user the SAML
assertion, and return it to Keystone.


Let's assume this step would work for now. I am interested how would 
the SP-IDP-SP workflow look like from the JS perspective. In classic 
WebSSO, where user uses only his browser:


1) Go to the protected resource 
(/v3/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth 
in our case)


2) (skipping the DS step for now) get redirected to the IdP
3) Authenticate with the IdP
4) Get redirected to the SP
5) Read your protected content, because you have been authenticated 
and authorized to do so (get an unscoped, federated token issued by 
Keystone in our case)


Now, I can imagine, if that's the websso approach can we somehow make 
JS mimic the browser with all its blessing? So the user would actualy 
see the real HTML website? If so, that would be enough to make it work.
I think that we need to show the user the real website in a Popup window 
until we have a guarantee of ECP support.





5.  Javascript picks up the Federated unscoped token from the response
at the end of step 4 and use that to get a scoped token.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone][Horizon] CORS and Federation

[Adam Young]

On 09/17/2014 11:56 AM, Matthieu Huin wrote:

Hi,

- Original Message -

From: "Adam Young" 
To: openstack-dev@lists.openstack.org
Sent: Wednesday, September 17, 2014 5:00:16 PM
Subject: Re: [openstack-dev] [Keystone][Horizon] CORS and Federation

On 09/17/2014 10:35 AM, David Chadwick wrote:



this would work as well, but wouldn't it require two different API calls?

I think it would be 2 calls no matter what.

OK, lets talk this through:

1. Configure Horizon to return a generic login page, with a button that says
"Or do Federated"
2. Use clicks that button and gets the Federated UI. No new HTTP request
needed for this, still just static Javascript. Form has an edit field for
the user to enter the SAML IdP, anda button to fetch the list of the public
IdPs from Keystone.
3. Assume user clicks the list of public IdPs, those fill out a drop-down
box. If the user clicks on one, it populates the textbox with the URL for
the IdP.
3a. However, if the users IdP is not in the list, they go back to the email
they got from their IT guy that has "Paste this URL into the IdP edit box"

4. User clicks OK.

Window pops up with the WebUI for the user to visit the SAML IdP URL. This
will be of the form

GET
htps://keystone/main/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth

Which will perform the necessary redirects to get the user the SAML
assertion, and return it to Keystone.

5. Javascript picks up the Federated unscoped token from the response at the
end of step 4 and use that to get a scoped token.


I think the jump to step 6 isn't unnecessary: logging in Horizon requires only 
a username and a password,
unless I am wrong scoping is done afterwards by selecting a project in a list. 
Besides, should we expect
a federated user to necessarily know beforehand to which domain/project she was 
mapped ?


Think of it this way;  today, Horizon uses the userid and password to 
get a token.  In the case where that password is in LDAP, Horizon should 
never, ever see it.  Even  Keystone should never see it.  So the Token 
abstracts that away, and says  instead "I've checked their password.  
its good."





6. Javascript submites the scoped token to Horizon and user is logged in.

Horizon will also need to keep track of the fact that a federated auth was used:
Once Horizon has a token, it can do all this.  Federation is kindof 
irrelevant.


* to list projects and domains available to the user
* to scope the unscoped saml token

As these are done through the federation API rather than the usual one.


Whew.

Whew indeed !






On 17/09/2014 15:17, Adam Young wrote:



On 09/17/2014 10:07 AM, David Chadwick wrote:



On 17/09/2014 14:55, Marek Denis wrote:



On 17.09.2014 15:45, Steve Martinelli wrote:



++ to your suggestion David, I think making the list of trusted IdPs
publicly available makes sense.
I think this might be useful in an academic/science world but on the
other hand most cloud providers from the 'business' world might be very
reluctant to expose list of their clients for free.
It is interesting that this latter comment came from the
academic/science world, whereas the supportive one came from the
business world :-)

So maybe there could be a config parameter in keystone to determine
which option is installed?
My thought was that there would be a public list, which is a subset of
the overall list.

For non-publicized IdPs, the end users would get an URL out of  band and
enter that in when prompted;  if they enter an invalid URL, they would
get an warning message.

It wouldn't hide the fact that a customer was registered with a given
cloud provider, but wouldn't advertise it, either.



regards

David



cheers,

Marek.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Oslo final releases ready

[Thomas Goirand]

On 09/18/2014 10:04 PM, Doug Hellmann wrote:
> All of the final releases for the Oslo libraries for the Juno cycle are 
> available on PyPI. I’m working on a couple of patches to the global 
> requirements list to update the baseline in the applications. In all cases, 
> the final release is a second tag on a previously released version.
> 
> - oslo.config - 1.4.0 (same as 1.4.0.0a5)
> - oslo.db - 1.0.0 (same as 0.5.0)
> - oslo.i18n - 1.0.0 (same as 0.4.0)
> - oslo.messaging - 1.4.0 (same as 1.4.0.0a5)
> - oslo.rootwrap - 1.3.0 (same as 1.3.0.0a3)
> - oslo.serialization - 1.0.0 (same as 0.3.0)
> - oslosphinx - 2.2.0 (same as 2.2.0.0a3)
> - oslotest - 1.1.0 (same as 1.1.0.0a2)
> - oslo.utils - 1.0.0 (same as 0.3.0)
> - cliff - 1.7.0 (previously tagged, so not a new release)
> - stevedore - 1.0.0 (same as 1.0.0.0a2)
> 
> Congratulations and *Thank You* to the Oslo team for doing an amazing job 
> with graduations this cycle!
> 
> Doug

Great! Thanks for doing this early in the freeze.

FYI, I have already uploaded all Juno dependencies to Debian, either in
Sid, or in Experimental.

However, I made a small mistake. I used 1.4.0.0~a5 instead of 1.4.0~a5.
As a consequence, I may upload 1.4.0.0 instead of 1.4.0, so that it is
greater than 1.4.0.0~a5 (to avoid adding an EPOC, which is ugly and
annoying to maintain). It's going to be the same version though, I just
need to add a new tag, which isn't much of a problem for me (since I use
a git based workflow).

The 1.4.0.0a5 is confusing... :(

Thomas Goirand (zigo)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone][Horizon] CORS and Federation

[David Chadwick]

Adam
I agree with you
David

On 18/09/2014 17:17, Adam Young wrote:
> On 09/17/2014 11:53 AM, Marek Denis wrote:
>> Hi,
>>
>> First of all, we should clarify whether your JS client wants to
>> implement ECP or WebSSO workflow. They are slightly different.
> 
> ECP seems to be poorly supported in live deployments, and we cannot
> count on it. WebSSO is the first round.  We should do  ECP as a second
> step.
> 
>>
>> I feel JS is smart enough to implement the ECP flow and then and it
>> could simply implement what we already have in the keystoneclient [0].
>> This + some "discovery service" described by Adam would be required.
>> However, some problems may arise when it comes to ADFS  support (we
>> needed separate plugin in keystoneclient) and other protocols which
>> should work by default from browsers.
>>
>> [0]
>> https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/auth/v3/saml2.py#L29
>>
>>
>> Rest of the comments inlined.
>>
>> On 17.09.2014 17:00, Adam Young wrote:
>>> On 09/17/2014 10:35 AM, David Chadwick wrote:
 this would work as well, but wouldn't it require two different API
 calls?
>>>
>>> I think it would be 2 calls no matter what.
>>>
>>> OK,  lets talk this through:
>>>
>>> 1. Configure Horizon to return a generic login page, with a button that
>>> says "Or do Federated"
>>> 2.  Use clicks that button and gets the Federated UI.  No new HTTP
>>> request needed for this, still just static Javascript.  Form has an edit
>>> field for the user to enter the SAML IdP, anda button to fetch the list
>>> of the public IdPs from Keystone.
>>> 3.  Assume user clicks the list of public IdPs, those fill out a
>>> drop-down box.  If the user clicks on one, it populates the textbox with
>>> the URL for the IdP.
>>> 3a.  However, if the users IdP is not in the list, they go back to the
>>> email they got from their IT guy that has "Paste this URL into the IdP
>>> edit box"
>>
>> Well, we don't keep any IdPs' URL in Keystone backend at the moment.
>> However, this can be easily fixed.
> Right.  That is a feature request.
> 
> 
>>
>>> 4. User clicks OK.
>>
>> OK
>>
>>> Window pops up with the WebUI for the user to visit the SAML IdP URL.
>>> This will be of the form
>>> |
>>> GET
>>> htps://keystone/main/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth|
>>>
>>>
>>> Which will perform the necessary redirects to get the user the SAML
>>> assertion, and return it to Keystone.
>>
>> Let's assume this step would work for now. I am interested how would
>> the SP-IDP-SP workflow look like from the JS perspective. In classic
>> WebSSO, where user uses only his browser:
>>
>> 1) Go to the protected resource
>> (/v3/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth
>> in our case)
>>
>> 2) (skipping the DS step for now) get redirected to the IdP
>> 3) Authenticate with the IdP
>> 4) Get redirected to the SP
>> 5) Read your protected content, because you have been authenticated
>> and authorized to do so (get an unscoped, federated token issued by
>> Keystone in our case)
>>
>> Now, I can imagine, if that's the websso approach can we somehow make
>> JS mimic the browser with all its blessing? So the user would actualy
>> see the real HTML website? If so, that would be enough to make it work.
> I think that we need to show the user the real website in a Popup window
> until we have a guarantee of ECP support.
> 
>>
>>> 5.  Javascript picks up the Federated unscoped token from the response
>>> at the end of step 4 and use that to get a scoped token.
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
> 
> > On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
> > 
> > Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
> >> 
> >>> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> >>> 
>  
>  Linux distributions are not the end be all of distribution models and
>  they don’t get to dictate to upstream.
> >>> 
> >>> Well, distributions is where the final user is, and where software gets
> >>> consumed. Our priority should be the end users.
> >> 
> >> 
> >> Distributions are not the only place that people get their software from,
> >> unless you think that the ~3 million downloads requests has received
> >> on PyPI in the last 30 days are distributions downloading requests to
> >> package in their OSs.
> >> 
> > 
> > Do pypi users not also need to be able to detect and fix any versions
> > of libraries they might have? If one has some virtualenvs with various
> > libraries and apps installed and no --system-site-packages, one would
> > probably still want to run 'pip freeze' in all of them and find out what
> > libraries are there and need to be fixed.
> > 
> > Anyway, generally security updates require a comprehensive strategy.
> > One common comprehensive strategy is version assertion.
> > 
> > Vendoring complicates that immensely.
> 
> It doesn’t really matter. PyPI doesn’t dictate to projects who host there what
> that project is allowed to do except in some very broad circumstances. Whether
> or not requests *should* do this doesn't really have any bearing on what
> Openstack should do to cope with it. The facts are that requests does it, and
> that people pulling things from PyPI is an actual platform that needs thought
> about.
> 
> This leaves Openstack with a few reasonable/sane options:
> 
> 1) Decide that vendoring in requests is unacceptable to what Openstack as a
>project is willing to support, and cease the use of requests.
> 2) Decide that what requests offers is good enough that it outweighs the fact
>that it vendors urllib3 and continue using it.
> 

There's also 3) fork requests, which is the democratic way to vote out
an upstream that isn't supporting the needs of the masses.

I don't think we're anywhere near there, but I wanted to make it clear
there _is_ a more extreme option.

> If the 2nd option is chosen, then doing anything but supporting the fact that
> requests vendors urllib3 within the code that openstack writes is hurting the
> users who fetch these projects from PyPI because you don't agree with one of
> the choices that requests makes. By all means do conditional imports to lessen
> the impact that the choice requests has made (and the one that Openstack has
> made to use requests) on downstream distributors, but unconditionally 
> importing
> from the top level urllib3 for use within requests is flat out wrong.
> 
> Obviously neither of these options excludes the choice to lean on requests to
> reverse this decision as well. However that is best done elsewhere as the
> person making that decision isn't a member of these mailing lists as far as
> I am aware.
> 

To be clear, I think we should keep using requests. But we should lend
our influence upstream and explain that our users are required to deal
with this in a way that perhaps hasn't been considered or given the
appropriate priority.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Keystone][Oslo] Federation and Policy

[David Chadwick]

Our recent work on federation suggests we need an improvement to the way
the policy engine works. My understanding is that most functions are
protected by the policy engine, but some are not. The latter functions
are publicly accessible. But there is no way in the policy engine to
specify public access to a function and there ought to be. This will
allow an administrator to configure the policy for a function to range
from very lax (publicly accessible) to very strict (admin only). A
policy of "" means that any authenticated user can access the function.
But there is no way in the policy to specify that an unauthenticated
user (i.e. public) has access to a function.

We have already identified one function (get trusted IdPs
"identity:list_identity_providers") that needs to be publicly accessible
in order for users to choose which IdP to use for federated login.
However some organisations may not wish to make this API call publicly
accessible, whilst others may wish to restrict it to Horizon only etc.
This indicates that that the policy needs to be set by the
administrator, and not by changes to the code (i.e. to either call the
policy engine or not, or to have two different API calls).

If we can invent some policy syntax that indicates public access, e.g.
reserved keyword of public, then Keystone can always call the policy
file for every function and there would be no need to differentiate
between protected APIs and non-protected APIs as all would be protected
to a greater or lesser extent according to the administrator's policy.

Comments please

regards

David






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Donald Stufft]

> On Sep 18, 2014, at 12:29 PM, Clint Byrum  wrote:
> 
> Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
>> 
>>> On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
>>> 
>>> Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
 
> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> 
>> 
>> Linux distributions are not the end be all of distribution models and
>> they don’t get to dictate to upstream.
> 
> Well, distributions is where the final user is, and where software gets
> consumed. Our priority should be the end users.
 
 
 Distributions are not the only place that people get their software from,
 unless you think that the ~3 million downloads requests has received
 on PyPI in the last 30 days are distributions downloading requests to
 package in their OSs.
 
>>> 
>>> Do pypi users not also need to be able to detect and fix any versions
>>> of libraries they might have? If one has some virtualenvs with various
>>> libraries and apps installed and no --system-site-packages, one would
>>> probably still want to run 'pip freeze' in all of them and find out what
>>> libraries are there and need to be fixed.
>>> 
>>> Anyway, generally security updates require a comprehensive strategy.
>>> One common comprehensive strategy is version assertion.
>>> 
>>> Vendoring complicates that immensely.
>> 
>> It doesn’t really matter. PyPI doesn’t dictate to projects who host there 
>> what
>> that project is allowed to do except in some very broad circumstances. 
>> Whether
>> or not requests *should* do this doesn't really have any bearing on what
>> Openstack should do to cope with it. The facts are that requests does it, and
>> that people pulling things from PyPI is an actual platform that needs thought
>> about.
>> 
>> This leaves Openstack with a few reasonable/sane options:
>> 
>> 1) Decide that vendoring in requests is unacceptable to what Openstack as a
>>   project is willing to support, and cease the use of requests.
>> 2) Decide that what requests offers is good enough that it outweighs the fact
>>   that it vendors urllib3 and continue using it.
>> 
> 
> There's also 3) fork requests, which is the democratic way to vote out
> an upstream that isn't supporting the needs of the masses.
> 
> I don't think we're anywhere near there, but I wanted to make it clear
> there _is_ a more extreme option.

Technically that’s just a specific case of option 1) ;)

But yes that’s a thing Openstack can do.

> 
>> If the 2nd option is chosen, then doing anything but supporting the fact that
>> requests vendors urllib3 within the code that openstack writes is hurting the
>> users who fetch these projects from PyPI because you don't agree with one of
>> the choices that requests makes. By all means do conditional imports to 
>> lessen
>> the impact that the choice requests has made (and the one that Openstack has
>> made to use requests) on downstream distributors, but unconditionally 
>> importing
>> from the top level urllib3 for use within requests is flat out wrong.
>> 
>> Obviously neither of these options excludes the choice to lean on requests to
>> reverse this decision as well. However that is best done elsewhere as the
>> person making that decision isn't a member of these mailing lists as far as
>> I am aware.
>> 
> 
> To be clear, I think we should keep using requests. But we should lend
> our influence upstream and explain that our users are required to deal
> with this in a way that perhaps hasn't been considered or given the
> appropriate priority.

I think that’s completely reasonable. I don’t think there’s going to be much 
movement,
I’ve had this argument with Kenneth on more than one occasion and he’s very 
happy
with his decision to vendor urllib3, but hey maybe Openstack would have better 
luck.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Clint Byrum]

Excerpts from Ian Cordasco's message of 2014-09-18 07:35:10 -0700:
> On 9/18/14, 9:18 AM, "Clint Byrum"  wrote:
> 
> >Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
> >> 
> >> > On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
> >> > 
> >> >> 
> >> >> Linux distributions are not the end be all of distribution models and
> >> >> they don’t get to dictate to upstream.
> >> > 
> >> > Well, distributions is where the final user is, and where software
> >>gets
> >> > consumed. Our priority should be the end users.
> >> 
> >> 
> >> Distributions are not the only place that people get their software
> >>from,
> >> unless you think that the ~3 million downloads requests has received
> >> on PyPI in the last 30 days are distributions downloading requests to
> >> package in their OSs.
> >> 
> >
> >Do pypi users not also need to be able to detect and fix any versions
> >of libraries they might have? If one has some virtualenvs with various
> >libraries and apps installed and no --system-site-packages, one would
> >probably still want to run 'pip freeze' in all of them and find out what
> >libraries are there and need to be fixed.
> >
> >Anyway, generally security updates require a comprehensive strategy.
> >One common comprehensive strategy is version assertion.
> >
> >Vendoring complicates that immensely.
> 
> Except that even OpenStack doesn’t pin requests because of how
> extraordinarily stable our API is. While you can argue that Kenneth has
> non-standard opinions about his library, Cory and I take backwards
> compatibility and stability very seriously. This means anyone can upgrade
> to a newer version of requests without worrying that it will be backwards
> incompatible. 
> 

All of your hard work is very much appreciated. I don't understand what
your assertion means though. We don't pin things. However, our users end
up "pinning" when they install via pip, and our distros end up "pinning"
when they deliver a version. Without any indication that urllib3 is in
the system, they will fail at any cursory version audit that looks for it.

I'm not saying either way is right or wrong either.. I'm suggesting
that this is a valid, proven method for large scale risk assessment,
and it is complicated quite a bit by vendored libraries.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Joe Gordon]

On Thu, Sep 18, 2014 at 9:02 AM, Devananda van der Veen <
devananda@gmail.com> wrote:

> On Thu, Sep 18, 2014 at 7:55 AM, Flavio Percoco  wrote:
> > On 09/18/2014 04:24 PM, Clint Byrum wrote:
> >> Great job highlighting what our friends over at Amazon are doing.
> >>
> >> It's clear from these snippets, and a few other pieces of documentation
> >> for SQS I've read, that the Amazon team approached SQS from a _massive_
> >> scaling perspective. I think what may be forcing a lot of this
> frustration
> >> with Zaqar is that it was designed with a much smaller scale in mind.
> >>
> >> I think as long as that is the case, the design will remain in question.
> >> I'd be comfortable saying that the use cases I've been thinking about
> >> are entirely fine with the limitations SQS has.
> >
> > I think these are pretty strong comments with not enough arguments to
> > defend them.
> >
>
> Please see my prior email. I agree with Clint's assertions here.
>
> > Saying that Zaqar was designed with a smaller scale in mid without
> > actually saying why you think so is not fair besides not being true. So
> > please, do share why you think Zaqar was not designed for big scales and
> > provide comments that will help the project to grow and improve.
> >
> > - Is it because the storage technologies that have been chosen?
> > - Is it because of the API?
> > - Is it because of the programing language/framework ?
>
> It is not because of the storage technology or because of the
> programming language.
>
> > So far, we've just discussed the API semantics and not zaqar's
> > scalability, which makes your comments even more surprising.
>
> - guaranteed message order
> - not distributing work across a configurable number of back ends
>
> These are scale-limiting design choices which are reflected in the
> API's characteristics.
>

I agree with Clint and Devananda


>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Devananda van der Veen]

On Thu, Sep 18, 2014 at 8:54 AM, Devananda van der Veen
 wrote:
> On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco  wrote:
>> On 09/18/2014 04:09 PM, Gordon Sim wrote:
>>> On 09/18/2014 12:31 PM, Flavio Percoco wrote:
 Zaqar guarantees FIFO. To be more precise, it does that relying on the
 storage backend ability to do so as well. Depending on the storage used,
 guaranteeing FIFO may have some performance penalties.
>>>
>>> Would it be accurate to say that at present Zaqar does not use
>>> distributed queues, but holds all queue data in a storage mechanism of
>>> some form which may internally distribute that data among servers but
>>> provides Zaqar with a consistent data model of some form?
>>
>> I think this is accurate. The queue's distribution depends on the
>> storage ability to do so and deployers will be able to choose what
>> storage works best for them based on this as well. I'm not sure how
>> useful this separation is from a user perspective but I do see the
>> relevance when it comes to implementation details and deployments.
>
> Guaranteeing FIFO and not using a distributed queue architecture
> *above* the storage backend are both scale-limiting design choices.
> That Zaqar's scalability depends on the storage back end is not a
> desirable thing in a cloud-scale messaging system in my opinion,
> because this will prevent use at scales which can not be accommodated
> by a single storage back end.
>

It may be worth qualifying this a bit more.

While no single instance of any storage back-end is infinitely
scalable, some of them are really darn fast. That may be enough for
the majority of use cases. It's not outside the realm of possibility
that the inflection point [0] where these design choices result in
performance limitations is at the very high end of scale-out, eg.
public cloud providers who have the resources to invest further in
improving zaqar.

As an example of what I mean, let me refer to the 99th percentile
response time graphs in Kurt's benchmarks [1]... increasing the number
of clients with write-heavy workloads was enough to drive latency from
<10ms to >200 ms with a single service. That latency significantly
improved as storage and application instances were added, which is
good, and what I would expect. These benchmarks do not (and were not
intended to) show the maximal performance of a public-cloud-scale
deployment -- but they do show that performance under different
workloads improves as additional services are started.

While I have no basis for comparing the configuration of the
deployment he used in those tests to what a public cloud operator
might choose to deploy, and presumably such an operator would put
significant work into tuning storage and running more instances of
each service and thus shift that inflection point "to the right", my
point is that, by depending on a single storage instance, Zaqar has
pushed the *ability* to scale out down into the storage
implementation. Given my experience scaling SQL and NoSQL data stores
(in my past life, before working on OpenStack) I have a knee-jerk
reaction to believing that this approach will result in a
public-cloud-scale messaging system.

-Devananda

[0] http://en.wikipedia.org/wiki/Inflection_point -- in this context,
I mean the point on the graph of throughput vs latency where the
derivative goes from near-zero (linear growth) to non-zero
(exponential growth)

[1] https://wiki.openstack.org/wiki/Zaqar/Performance/PubSub/Redis

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Joe Gordon]

On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco  wrote:

> On 09/18/2014 04:09 PM, Gordon Sim wrote:
> > On 09/18/2014 12:31 PM, Flavio Percoco wrote:
> >> On 09/17/2014 10:36 PM, Joe Gordon wrote:
> >>> My understanding of Zaqar is that it's like SQS. SQS uses distributed
> >>> queues, which have a few unusual properties [0]:
> >>>
> >>>
> >>>  Message Order
> >>>
> >>> Amazon SQS makes a best effort to preserve order in messages, but due
> to
> >>> the distributed nature of the queue, we cannot guarantee you will
> >>> receive messages in the exact order you sent them. If your system
> >>> requires that order be preserved, we recommend you place sequencing
> >>> information in each message so you can reorder the messages upon
> >>> receipt.
> >>>
> >>
> >> Zaqar guarantees FIFO. To be more precise, it does that relying on the
> >> storage backend ability to do so as well. Depending on the storage used,
> >> guaranteeing FIFO may have some performance penalties.
> >
> > Would it be accurate to say that at present Zaqar does not use
> > distributed queues, but holds all queue data in a storage mechanism of
> > some form which may internally distribute that data among servers but
> > provides Zaqar with a consistent data model of some form?
>
> I think this is accurate. The queue's distribution depends on the
> storage ability to do so and deployers will be able to choose what
> storage works best for them based on this as well. I'm not sure how
> useful this separation is from a user perspective but I do see the
> relevance when it comes to implementation details and deployments.
>
>
> > [...]
> >> As of now, Zaqar fully relies on the storage replication/clustering
> >> capabilities to provide data consistency, availability and fault
> >> tolerance.
> >
> > Is the replication synchronous or asynchronous with respect to client
> > calls? E.g. will the response to a post of messages be returned only
> > once the replication of those messages is confirmed? Likewise when
> > deleting a message, is the response only returned when replicas of the
> > message are deleted?
>
> It depends on the driver implementation and/or storage configuration.
> For example, in the mongodb driver, we use the default write concern
> called "acknowledged". This means that as soon as the message gets to
> the master node (note it's not written on disk yet nor replicated) zaqar
> will receive a confirmation and then send the response back to the
> client. This is also configurable by the deployer by changing the
> default write concern in the mongodb uri using `?w=SOME_WRITE_CONCERN`[0].
>

This means that by default Zaqar cannot guarantee a message will be
delivered at all. A message can be acknowledged and then the 'master node'
crashes and the message is lost. Zaqar's ability to guarantee delivery is
limited by the reliability of a single node, while something like swift can
only loose a piece of data if 3 machines crash at the same time.


>
> [0] http://docs.mongodb.org/manual/reference/connection-string/#uri.w
>
> >
> >> However, as far as consuming messages is concerned, it can
> >> guarantee once-and-only-once and/or at-least-once delivery depending on
> >> the message pattern used to consume messages. Using pop or claims
> >> guarantees the former whereas streaming messages out of Zaqar guarantees
> >> the later.
> >
> > From what I can see, pop provides unreliable delivery (i.e. its similar
> > to no-ack). If the delete call using pop fails while sending back the
> > response, the messages are removed but didn't get to the client.
>
> Correct, pop works like no-ack. If you want to have pop+ack, it is
> possible to claim just 1 message and then delete it.
>
> >
> > What do you mean by 'streaming messages'?
>
> I'm sorry, that went out wrong. I had the "browsability" term in my head
> and went with something even worse. By streaming messages I meant
> polling messages without claiming them. In other words, at-least-once is
> guaranteed by default, whereas once-and-only-once is guaranteed just if
> claims are used.
>
> >
> > [...]
> >> Based on our short conversation on IRC last night, I understand you're
> >> concerned that FIFO may result in performance issues. That's a valid
> >> concern and I think the right answer is that it depends on the storage.
> >> If the storage has a built-in FIFO guarantee then there's nothing Zaqar
> >> needs to do there. In the other hand, if the storage does not have a
> >> built-in support for FIFO, Zaqar will cover it in the driver
> >> implementation. In the mongodb driver, each message has a marker that is
> >> used to guarantee FIFO.
> >
> > That marker is a sequence number of some kind that is used to provide
> > ordering to queries? Is it generated by the database itself?
>
> It's a sequence number to provide ordering to queries, correct.
> Depending on the driver, it may be generated by Zaqar or the database.
> In mongodb's case it's generated by Zaqar[0].
>
> [0]
>
> https://githu

Re: [openstack-dev] Please do *NOT* use "vendorized" versions of anything (here: glanceclient using requests.packages.urllib3)

[Ian Cordasco]

On 9/18/14, 11:29 AM, "Clint Byrum"  wrote:

>Excerpts from Donald Stufft's message of 2014-09-18 07:30:27 -0700:
>> 
>> > On Sep 18, 2014, at 10:18 AM, Clint Byrum  wrote:
>> > 
>> > Excerpts from Donald Stufft's message of 2014-09-18 04:58:06 -0700:
>> >> 
>> >>> On Sep 18, 2014, at 7:54 AM, Thomas Goirand  wrote:
>> >>> 
>>  
>>  Linux distributions are not the end be all of distribution models
>>and
>>  they don’t get to dictate to upstream.
>> >>> 
>> >>> Well, distributions is where the final user is, and where software
>>gets
>> >>> consumed. Our priority should be the end users.
>> >> 
>> >> 
>> >> Distributions are not the only place that people get their software
>>from,
>> >> unless you think that the ~3 million downloads requests has received
>> >> on PyPI in the last 30 days are distributions downloading requests to
>> >> package in their OSs.
>> >> 
>> > 
>> > Do pypi users not also need to be able to detect and fix any versions
>> > of libraries they might have? If one has some virtualenvs with various
>> > libraries and apps installed and no --system-site-packages, one would
>> > probably still want to run 'pip freeze' in all of them and find out
>>what
>> > libraries are there and need to be fixed.
>> > 
>> > Anyway, generally security updates require a comprehensive strategy.
>> > One common comprehensive strategy is version assertion.
>> > 
>> > Vendoring complicates that immensely.
>> 
>> It doesn’t really matter. PyPI doesn’t dictate to projects who host
>>there what
>> that project is allowed to do except in some very broad circumstances.
>>Whether
>> or not requests *should* do this doesn't really have any bearing on what
>> Openstack should do to cope with it. The facts are that requests does
>>it, and
>> that people pulling things from PyPI is an actual platform that needs
>>thought
>> about.
>> 
>> This leaves Openstack with a few reasonable/sane options:
>> 
>> 1) Decide that vendoring in requests is unacceptable to what Openstack
>>as a
>>project is willing to support, and cease the use of requests.
>> 2) Decide that what requests offers is good enough that it outweighs
>>the fact
>>that it vendors urllib3 and continue using it.
>> 
>
>There's also 3) fork requests, which is the democratic way to vote out
>an upstream that isn't supporting the needs of the masses.

Given requests’ download count, I have to doubt that OpenStack users
constitute the masses in this case.

>I don't think we're anywhere near there, but I wanted to make it clear
>there _is_ a more extreme option.
>
>> If the 2nd option is chosen, then doing anything but supporting the
>>fact that
>> requests vendors urllib3 within the code that openstack writes is
>>hurting the
>> users who fetch these projects from PyPI because you don't agree with
>>one of
>> the choices that requests makes. By all means do conditional imports to
>>lessen
>> the impact that the choice requests has made (and the one that
>>Openstack has
>> made to use requests) on downstream distributors, but unconditionally
>>importing
>> from the top level urllib3 for use within requests is flat out wrong.
>> 
>> Obviously neither of these options excludes the choice to lean on
>>requests to
>> reverse this decision as well. However that is best done elsewhere as
>>the
>> person making that decision isn't a member of these mailing lists as
>>far as
>> I am aware.
>> 
>
>To be clear, I think we should keep using requests. But we should lend
>our influence upstream and explain that our users are required to deal
>with this in a way that perhaps hasn't been considered or given the
>appropriate priority.

It’s been considered several times. There have been multiple issues.
There’s more than just the one you linked to. The decision is highly
unlikely to change whether it’s coming from a group of people in OpenStack
or another distribution package maintainer.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Ceilometer] Work at bug "Ceilometer HBase driver cannot handle non-ascii characters"

[Ilya Tyaptin]

Hi, folks!

I want to engage this bug
 [1].

But fixing of this bug has raised question.
Issue is that happybase does not support unicode symbols and approaches to
processing unicode symbols is needed to discuss.

I suggest next variants:
1. Anyway to check containing unicode symbols in string, if contains then
screen theys.
2. Transform unicode symbols like a html encoding (/xa0 -> %2Fxa0, for
example)

But in these approaches we can't be sure that we haven't false positives at
getting data from HBase.

Thanks for all, wait for your response!

[1] https://bugs.launchpad.net/ceilometer/+bug/1350826

-- 
Best regards,

Tyaptin Ilia,

Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] Performance degradation (docs) for sampling rate less than 10 mins

[Ilya Tyaptin]

Hi Patanjali!

We have inspected this question and got a document with results of this
testing.
Tests are running on physical lab and includes 2000 VMs ran by Nova and 60
second polling interval.

Expanded information in doc
https://docs.google.com/a/mirantis.com/document/d/1jvqIy6fWQBvTZEfdnk37FvF5xtA3l7Cx09VNoe6bBRU

If you have any question I'll be happy to answer.

On Thu, Sep 18, 2014 at 7:56 PM, Srikanta Patanjali 
wrote:

> Hi Team,
>
> I was considering to change the default sampling rate of the Ceilometer
> from 10 mins to less than that. I foresee an adverse impact on its
> performance due to increase in the data inside the MondoDB.
>
> I was wondering if the QA team (or anyone else) has done any of the load
> tests with these parameters ? It would be helpful to have access to these
> results (if any).
>
> If not, I would like to know the views on increasing the sample rate value
> (in the pipleline.yaml file)
>
> Cheers,
> Srikanta
> InIT ¦ ICCLab
> ZHAW
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

Best regards,

Tyaptin Ilia,

Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Thoughts on OpenStack Layers and a Big Tent model

[Monty Taylor]

Hey all,

I've recently been thinking a lot about Sean's Layers stuff. So I wrote
a blog post which Jim Blair and Devananda were kind enough to help me edit.

http://inaugust.com/post/108

Enjoy.

Monty

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev