Re: [Openstack] Canonical AWSOME
So if what u are talking about is anything RPC/MQ based, then I would say those are not internal API's. Once a RPC/MQ mechanism is introduced they don't really become internal API's anymore (if we are talking about the same API's, haha). Since other stuff can be reading those messages on the MQ its very useful to have a schema to know exactly what to read :-) On 4/24/12 11:46 AM, Russell Bryant rbry...@redhat.com wrote: On 04/24/2012 01:25 PM, Joshua Harlow wrote: I'm more in favor of just having a schema, I don't care if that compiles to protocol buffers, json, NEWAWESOMEhipsterMSGFORMAT. That schema will force people to think a little more when they add messages, and it will automatically document the messages that are being sent around. That's a big win I think and is a step to getting those schemas versioned... I'm not sure a schema is really necessary aside from the Python classes themselves. They're internal APIs, so they shouldn't be used from outside of Nova. A schema would be useful if we had to define the interfaces in some language neutral-format, but I don't think that really matters here. I'm going to work on a proposal / prototype for how we can handle versioning, though. The big goal here is making sure that we can maintain compatibility with the previous release. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
I've heard a few people mention pulling messages off the queue, or communicating via RPC outside of the project, or outside of Python. In theory, this sounds nice, but the RPC implementations are strictly making sure that A can execute calls on target B and that responses get back to A. This has little to do with message queues, other than that message queues are optionally supported. You shouldn't be peeking behind that curtain. This is specific to each RPC mechanism and enforcing something this early might be more problematic than you expect. I think that in the Folsom time frame, if not beyond, we should suggest a best practice of using the OpenStack RPC modules if you intend to communicate with Nova services. They're moving into OpenStack common, so this will become more friendly to use. It will require Python, but I don't see a reasonable alternative at the moment. Additionally, each RPC driver can provide a guide to complying with their protocol, which extends beyond simply the transport (i.e. AMQP or ZeroMQ). This might be harder than it sounds and might vary between, or even within, releases. -- Eric Windisch On Wednesday, April 25, 2012 at 1:24 PM, Joshua Harlow wrote: Re: [Openstack] Canonical AWSOME So if what u are talking about is anything RPC/MQ based, then I would say those are not internal API’s. Once a RPC/MQ mechanism is introduced they don’t really become internal API’s anymore (if we are talking about the same API’s, haha). Since other stuff can be reading those messages on the MQ its very useful to have a schema to know exactly what to read :-) On 4/24/12 11:46 AM, Russell Bryant rbry...@redhat.com wrote: On 04/24/2012 01:25 PM, Joshua Harlow wrote: I’m more in favor of just having a schema, I don’t care if that compiles to protocol buffers, json, NEWAWESOMEhipsterMSGFORMAT. That schema will force people to think a little more when they add messages, and it will automatically document the messages that are being sent around. That’s a big win I think and is a step to getting those schemas versioned... I'm not sure a schema is really necessary aside from the Python classes themselves. They're internal APIs, so they shouldn't be used from outside of Nova. A schema would be useful if we had to define the interfaces in some language neutral-format, but I don't think that really matters here. I'm going to work on a proposal / prototype for how we can handle versioning, though. The big goal here is making sure that we can maintain compatibility with the previous release. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net (mailto:openstack@lists.launchpad.net) Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
23. apr. 2012 17.15 skrev Justin Santa Barbara jus...@fathomdb.com: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? Although I wasn't able to attend the design summit session, right now we have two native APIs, which means we have two paths into the system. Yes. This is good. It keeps the layers of separation honest. Multiple consumers of internal API's helps in making it more obvious at which layer functionality belongs. Just like having multiple hypervisor drivers (at least supposedly) makes it more obvious what sort of stuff belongs in the drivers and what sort of stuff belongs in the compute manager. That is poor software engineering, because we must code and debug everything twice. I beg to differ. If we need to fix things twice, it's because we've violated the layers of separation somewhere. For the record, I'm fully prepared to believe that we've done so. I also fully believe that the fact that we haven't done so *more* is because we have two API's to consider. Eric Day went through all the calls from the frontends into the various backend managers a long time ago, ensuring this separation was clear. I'm convinced that the result was a massive improvement. Some developers will naturally favor one API over the other, and so disparities happen. Today, both APIs are effectively using an undocumented private API, which is problematic. Yes. This is a problem. However, as I understand it, there was a session at the summit about versioning the internal API's. I'm not sure how we can usefully version the API's without also documenting them, so that problem should be adressed in the relatively near future. We also can't really extend the EC2 API, so it is holding us back as we extend OpenStack's capabilities past those of the legacy clouds. If EC2 API is limiting what we can do, that's not going to change just because you move the EC2 API implementation into a proxy in front of the OpenStack API. The only difference is that it's suddenly the AWSOME development team's problem. With one native API, we can focus all our energies on making sure that API works. Then, knowing that the native API works, we can build other APIs on top through simple translation layers, and they will work also. Other APIs can be built on top in the same way (e.g. OCCI) Sorry, I'm having trouble here.. Are you suggesting that having two sibling frontend API's talking to a shared backend API is poor software engineering, but layering similar purposed API's on top of each other like this is good software engineering? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
On Tue, 2012-04-24 at 13:26 +0200, Soren Hansen wrote: 23. apr. 2012 17.15 skrev Justin Santa Barbara jus...@fathomdb.com: With one native API, we can focus all our energies on making sure that API works. Then, knowing that the native API works, we can build other APIs on top through simple translation layers, and they will work also. Other APIs can be built on top in the same way (e.g. OCCI) Sorry, I'm having trouble here.. Are you suggesting that having two sibling frontend API's talking to a shared backend API is poor software engineering, but layering similar purposed API's on top of each other like this is good software engineering? Yeah, I'm with you here. I think the frustration about the native EC2 support boils down to: a) we haven't (yet) got the clean separation of layers that we'd like b) not enough people are working on EC2 support c) there's a concern that OpenStack providers may not enable it And my reaction to those is: a) we can fix that b) starting a new project doesn't help c) adding an EC2 frontend to deltacloud is a much more sane way of addressing this Cheers, Mark. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
If EC2 API is limiting what we can do, that's not going to change just because you move the EC2 API implementation into a proxy in front of the OpenStack API. The only difference is that it's suddenly the AWSOME development team's problem. I think it's actually the EC2 API caller's problem. It's easy to map a subset to a superset, but you can't cover the superset with the subset. Suppose Quantum defines e.g. Layer 2 security groups, which I think has no parallel in EC2. If you want to use L2 security groups, you'd have to use the OpenStack API. A nice AWSOME developer might treat it as their problem, but it's not really. With one native API, we can focus all our energies on making sure that API works. Then, knowing that the native API works, we can build other APIs on top through simple translation layers, and they will work also. Other APIs can be built on top in the same way (e.g. OCCI) Sorry, I'm having trouble here.. Are you suggesting that having two sibling frontend API's talking to a shared backend API is poor software engineering, but layering similar purposed API's on top of each other like this is good software engineering? Obviously not when you put it like that :-) The difference that I see is whether the 'service' layer is documented and kept stable (i.e. subject to API evolution rules). Right now, we do that work for the OpenStack API only. I think it's great if someone is going to document and stabilize our internal APIs. However, unless you're willing to do it personally, I don't think anyone should assume it'll be done. So my viewpoint is that, while it would be great to have interface contracts on all our internal interfaces, I'll believe it when I see it. Until then, we have one stable interface, and so using that stable interface as the service layer seems like a sensible approach. I can see us using ProtocolBuffers or JSON schema for messages that go over the message bus, maybe even in Folsom. I can't see us locking down the other code interfaces before we give up on this Python thing and port to Java, i.e. it's not going to happen. Actually, I think JSON schema for our message-bus messages might be a really good idea (tm). Violations could just be warnings until we get things locked down... maybe I should propose a blueprint? (Although I have enough of a blueprint backlog as it is...) Justin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
Actually, I think JSON schema for our message-bus messages might be a really good idea (tm). Violations could just be warnings until we get things locked down... maybe I should propose a blueprint? (Although I have enough of a blueprint backlog as it is...) ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
Actually, I think JSON schema for our message-bus messages might be a really good idea (tm). Violations could just be warnings until we get things locked down... maybe I should propose a blueprint? (Although I have enough of a blueprint backlog as it is...) This was discussed at the summit in (I believe) the API versioning talk. There is a strong bias toward JSON inside RPC messages. However, this is actually transparent to Nova as the RPC implementations do the decoding and encoding. It is also hard to test and trigger such warnings as, as far as Nova knows, all the interfaces pass python data types, not JSON. Some RPC mechanisms could transparently serialize. As long as there is an abstraction layer, it should be oblivious to the serialization and we should not care too strongly. There was a patch a few weeks ago that has enforced using a common RPC exception serializer using JSON, which I'm not sure is, or is not, a good thing. I haven't yet updated the ZeroMQ driver to use this, but am in the process of making these changes as I update for Folsom. All that said, I do intend that the ZeroMQ driver will use JSON when it lands in Folsom. (it currently Pickles, but only because there was a bug in Essex at one point, breaking JSON serialization) -- Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
On 04/24/2012 11:39 AM, Eric Windisch wrote: Actually, I think JSON schema for our message-bus messages might be a really good idea (tm). Violations could just be warnings until we get things locked down... maybe I should propose a blueprint? (Although I have enough of a blueprint backlog as it is...) This was discussed at the summit in (I believe) the API versioning talk. There is a strong bias toward JSON inside RPC messages. However, this is actually transparent to Nova as the RPC implementations do the decoding and encoding. It is also hard to test and trigger such warnings as, as far as Nova knows, all the interfaces pass python data types, not JSON. Some RPC mechanisms could transparently serialize. As long as there is an abstraction layer, it should be oblivious to the serialization and we should not care too strongly. There was a patch a few weeks ago that has enforced using a common RPC exception serializer using JSON, which I'm not sure is, or is not, a good thing. I haven't yet updated the ZeroMQ driver to use this, but am in the process of making these changes as I update for Folsom. The change was just to the fake rpc backend to help catch more errors in unit tests where non-primitive types are getting passed into rpc. All that said, I do intend that the ZeroMQ driver will use JSON when it lands in Folsom. (it currently Pickles, but only because there was a bug in Essex at one point, breaking JSON serialization) There may still be some datetime related problems lurking in there, but we need to fix them as they come up. If it causes a problem for you, it's likely a problem for the other drivers, as well. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
Thanks for the pointer. I found the etherpad: http://etherpad.openstack.org/VersioningNovaRPCAPIs Is there a blueprint that came / is coming out of that? I think the data representation is orthogonal e.g. in theory, we could even use XML schemas: PyDict -- XML -- XML Schema Validation -- Warn / Throw \- JSON - Rabbit As it sounds like we are using JSON, it makes most sense to use JSON schemas. But doing so doesn't preclude us from using e.g. a binary format in future. I'd imagine that the RPC mechanism would simply select the relevant schema based on a few attributes of the dict (likely the queue method name). So this would remain transparent to the caller. A basic RPC mechanisms might validate and warn against a JSON schema, another might use the schema to build a compact binary representation. But I think we can achieve this without changing nova. On Tue, Apr 24, 2012 at 8:39 AM, Eric Windisch e...@cloudscaling.comwrote: Actually, I think JSON schema for our message-bus messages might be a really good idea (tm). Violations could just be warnings until we get things locked down... maybe I should propose a blueprint? (Although I have enough of a blueprint backlog as it is...) This was discussed at the summit in (I believe) the API versioning talk. There is a strong bias toward JSON inside RPC messages. However, this is actually transparent to Nova as the RPC implementations do the decoding and encoding. It is also hard to test and trigger such warnings as, as far as Nova knows, all the interfaces pass python data types, not JSON. Some RPC mechanisms could transparently serialize. As long as there is an abstraction layer, it should be oblivious to the serialization and we should not care too strongly. There was a patch a few weeks ago that has enforced using a common RPC exception serializer using JSON, which I'm not sure is, or is not, a good thing. I haven't yet updated the ZeroMQ driver to use this, but am in the process of making these changes as I update for Folsom. All that said, I do intend that the ZeroMQ driver will use JSON when it lands in Folsom. (it currently Pickles, but only because there was a bug in Essex at one point, breaking JSON serialization) -- Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
I'm more in favor of just having a schema, I don't care if that compiles to protocol buffers, json, NEWAWESOMEhipsterMSGFORMAT. That schema will force people to think a little more when they add messages, and it will automatically document the messages that are being sent around. That's a big win I think and is a step to getting those schemas versioned... Just don't do pickling, pleasee (for other language users). On 4/24/12 8:39 AM, Eric Windisch e...@cloudscaling.com wrote: Actually, I think JSON schema for our message-bus messages might be a really good idea (tm). Violations could just be warnings until we get things locked down... maybe I should propose a blueprint? (Although I have enough of a blueprint backlog as it is...) This was discussed at the summit in (I believe) the API versioning talk. There is a strong bias toward JSON inside RPC messages. However, this is actually transparent to Nova as the RPC implementations do the decoding and encoding. It is also hard to test and trigger such warnings as, as far as Nova knows, all the interfaces pass python data types, not JSON. Some RPC mechanisms could transparently serialize. As long as there is an abstraction layer, it should be oblivious to the serialization and we should not care too strongly. There was a patch a few weeks ago that has enforced using a common RPC exception serializer using JSON, which I'm not sure is, or is not, a good thing. I haven't yet updated the ZeroMQ driver to use this, but am in the process of making these changes as I update for Folsom. All that said, I do intend that the ZeroMQ driver will use JSON when it lands in Folsom. (it currently Pickles, but only because there was a bug in Essex at one point, breaking JSON serialization) ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
The change was just to the fake rpc backend to help catch more errors in unit tests where non-primitive types are getting passed into rpc. My current code should still work, but the tests seem to have eliminated the more generic exception handling case with the assumption that testing the exceptions and the exception serialization/deserialization is sufficient. That said, I'd rather utilize *more* stuff from rpc.common, so won't complain too badly. I know that the ZeroMQ driver could be thinned a bit, by better using some of the primitives available in rpc.common and we might be able to refactor some of the stuff in ampq.py to be more generically useful (maybe). Right now, none of that is a huge concern to me, we can get it integrated and do the DRY later. -- Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
On 04/24/2012 01:25 PM, Joshua Harlow wrote: I’m more in favor of just having a schema, I don’t care if that compiles to protocol buffers, json, NEWAWESOMEhipsterMSGFORMAT. That schema will force people to think a little more when they add messages, and it will automatically document the messages that are being sent around. That’s a big win I think and is a step to getting those schemas versioned... I'm not sure a schema is really necessary aside from the Python classes themselves. They're internal APIs, so they shouldn't be used from outside of Nova. A schema would be useful if we had to define the interfaces in some language neutral-format, but I don't think that really matters here. I'm going to work on a proposal / prototype for how we can handle versioning, though. The big goal here is making sure that we can maintain compatibility with the previous release. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
Philipp Wollermann wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? One thing that was mentioned is that the proxy could be run on top of a public cloud that chose to only deploy OpenStack API support. -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
On Mon, Apr 23, 2012 at 8:39 AM, Thierry Carrez thie...@openstack.orgwrote: Philipp Wollermann wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? One thing that was mentioned is that the proxy could be run on top of a public cloud that chose to only deploy OpenStack API support. It also makes project management easier, as the people interested in maintaining it can focus on the separate repository. Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
On 04/23/2012 02:39 PM, Thierry Carrez wrote: Philipp Wollermann wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? One thing that was mentioned is that the proxy could be run on top of a public cloud that chose to only deploy OpenStack API support. Another reason why we started this project is that we believe this architecture would be easier to use and maintain than what currently exists in Nova. This is why we are proposing this work as a longer term solution for EC2 compatibility. Nick signature.asc Description: OpenPGP digital signature ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? Although I wasn't able to attend the design summit session, right now we have two native APIs, which means we have two paths into the system. That is poor software engineering, because we must code and debug everything twice. Some developers will naturally favor one API over the other, and so disparities happen. Today, both APIs are effectively using an undocumented private API, which is problematic. We also can't really extend the EC2 API, so it is holding us back as we extend OpenStack's capabilities past those of the legacy clouds. With one native API, we can focus all our energies on making sure that API works. Then, knowing that the native API works, we can build other APIs on top through simple translation layers, and they will work also. Other APIs can be built on top in the same way (e.g. OCCI) Which is a long way of saying the external approach will result in _all_ APIs (OpenStack, EC2, OCCI etc) becoming more reliable, more secure and just more AWSOME. Justin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
We just have to be careful here to make sure that lots and lots of tests are done to ensure that the EC2 response is valid as well as to ensure the performance is not de-graded by having this proxy layer. If it does get de-graded then there needs to be some work done to ensure that this does not happen (ie new features in the openstack-api). Hopefully this proxy layer can also remain stateless, for as long as possible (for everyone's sanity). On 4/23/12 7:57 AM, Nick Barcet nick.bar...@canonical.com wrote: On 04/23/2012 02:39 PM, Thierry Carrez wrote: Philipp Wollermann wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? One thing that was mentioned is that the proxy could be run on top of a public cloud that chose to only deploy OpenStack API support. Another reason why we started this project is that we believe this architecture would be easier to use and maintain than what currently exists in Nova. This is why we are proposing this work as a longer term solution for EC2 compatibility. Nick ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
Creating a contract on the private API will allow the external APIs to be created and tested without needing a translation layer, even if contributory APIs were developed outside of the project (such as in AWSOME). It is clearly better, architecturally, if the EC2/OCCI apis can access the internal apis directly. The RPC and database can be made to scale in Nova, but a REST endpoint won't be as reliable or scale as well. -- Eric Windisch On Monday, April 23, 2012 at 11:15 AM, Justin Santa Barbara wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? Although I wasn't able to attend the design summit session, right now we have two native APIs, which means we have two paths into the system. That is poor software engineering, because we must code and debug everything twice. Some developers will naturally favor one API over the other, and so disparities happen. Today, both APIs are effectively using an undocumented private API, which is problematic. We also can't really extend the EC2 API, so it is holding us back as we extend OpenStack's capabilities past those of the legacy clouds. With one native API, we can focus all our energies on making sure that API works. Then, knowing that the native API works, we can build other APIs on top through simple translation layers, and they will work also. Other APIs can be built on top in the same way (e.g. OCCI) Which is a long way of saying the external approach will result in _all_ APIs (OpenStack, EC2, OCCI etc) becoming more reliable, more secure and just more AWSOME. Justin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net (mailto:openstack@lists.launchpad.net) Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
On 04/23/2012 10:42 AM, Doug Hellmann wrote: On Mon, Apr 23, 2012 at 8:39 AM, Thierry Carrez thie...@openstack.org mailto:thie...@openstack.org wrote: Philipp Wollermann wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? One thing that was mentioned is that the proxy could be run on top of a public cloud that chose to only deploy OpenStack API support. This makes sense. It also makes project management easier, as the people interested in maintaining it can focus on the separate repository. I'm not sure I buy into this, though. Why is it harder for people that are interested in EC2 support to work in the existing nova repo? If people want to collaborate before pushing into mainline, that can be done via feature branches, too. It risks making EC2 development harder, as well. Pulling it out of nova completely risks allowing the people that don't care about EC2 to care even less. It could make it easier for people to make changes that make EC2 compatibility harder to maintain. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
Martin Packman mentioned that there are some other groups interested in creating a separate AWS validation test suite, and that AWSOME would probably be using that. I think Joshua Harlow from Yahoo! was working in that area, too. On Mon, Apr 23, 2012 at 1:38 PM, Joshua Harlow harlo...@yahoo-inc.comwrote: We just have to be careful here to make sure that lots and lots of tests are done to ensure that the EC2 response is valid as well as to ensure the performance is not de-graded by having this proxy layer. If it does get de-graded then there needs to be some work done to ensure that this does not happen (ie new features in the openstack-api). Hopefully this proxy layer can also remain stateless, for as long as possible (for everyone's sanity). On 4/23/12 7:57 AM, Nick Barcet nick.bar...@canonical.com wrote: On 04/23/2012 02:39 PM, Thierry Carrez wrote: Philipp Wollermann wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? One thing that was mentioned is that the proxy could be run on top of a public cloud that chose to only deploy OpenStack API support. Another reason why we started this project is that we believe this architecture would be easier to use and maintain than what currently exists in Nova. This is why we are proposing this work as a longer term solution for EC2 compatibility. Nick ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
Ya, we should figure out how to organize to make this happen. It seems to provide many benefits (not just for openstack). Maybe setup a little working group around this... On 4/23/12 12:01 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Martin Packman mentioned that there are some other groups interested in creating a separate AWS validation test suite, and that AWSOME would probably be using that. I think Joshua Harlow from Yahoo! was working in that area, too. On Mon, Apr 23, 2012 at 1:38 PM, Joshua Harlow harlo...@yahoo-inc.com wrote: We just have to be careful here to make sure that lots and lots of tests are done to ensure that the EC2 response is valid as well as to ensure the performance is not de-graded by having this proxy layer. If it does get de-graded then there needs to be some work done to ensure that this does not happen (ie new features in the openstack-api). Hopefully this proxy layer can also remain stateless, for as long as possible (for everyone's sanity). On 4/23/12 7:57 AM, Nick Barcet nick.bar...@canonical.com http://nick.bar...@canonical.com wrote: On 04/23/2012 02:39 PM, Thierry Carrez wrote: Philipp Wollermann wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? One thing that was mentioned is that the proxy could be run on top of a public cloud that chose to only deploy OpenStack API support. Another reason why we started this project is that we believe this architecture would be easier to use and maintain than what currently exists in Nova. This is why we are proposing this work as a longer term solution for EC2 compatibility. Nick ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
On Monday, April 23, 2012 at 3:42 PM, Justin Santa Barbara wrote: I didn't realize people were willing to do so. Ah yes, well, that problem might still remain. There are certainly seem to be volunteers to work on the versioning code, but defining, tagging, and adhering to API contracts are another matter... -- Eric Windisch___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
How are REST endpoints not reliable or scalable ;-) I'd like to know, seeing as the web is built on them :-) It is clearly better to access the internal API's if performance is an issue, from an abstraction point of view if the EC2 apis are a view on top of the OS apis then that seems to make sense (if it can be done). Although a counter-point I will accept is that if u have 2 apis that openstack supports internally, ie the EC2/OS apis this does force u to think more about the APIs that u are building (to ensure it will work with the 2 external apis), which does help force better thinking as to how your API's will be designed. When u have 2+ entrypoints using an API, then u know that 3+ will be pretty easy, but if u only have 1 entrypoint using your API, then u might just have screwed yourself. On 4/23/12 11:01 AM, Eric Windisch e...@cloudscaling.com wrote: Creating a contract on the private API will allow the external APIs to be created and tested without needing a translation layer, even if contributory APIs were developed outside of the project (such as in AWSOME). It is clearly better, architecturally, if the EC2/OCCI apis can access the internal apis directly. The RPC and database can be made to scale in Nova, but a REST endpoint won't be as reliable or scale as well. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
+1 if you want people to care about something then it should be part of the main repo and part of the regular regression testing. thanks -Doug __ STSM | Standards Architect | IBM Software Group (919) 254-6905 | IBM 444-6905 | d...@us.ibm.com The more I'm around some people, the more I like my dog. Russell Bryant rbry...@redhat.com Sent by: openstack-bounces+dug=us.ibm@lists.launchpad.net 04/23/2012 02:16 PM To openstack@lists.launchpad.net cc Subject Re: [Openstack] Canonical AWSOME On 04/23/2012 10:42 AM, Doug Hellmann wrote: On Mon, Apr 23, 2012 at 8:39 AM, Thierry Carrez thie...@openstack.org mailto:thie...@openstack.org wrote: Philipp Wollermann wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? One thing that was mentioned is that the proxy could be run on top of a public cloud that chose to only deploy OpenStack API support. This makes sense. It also makes project management easier, as the people interested in maintaining it can focus on the separate repository. I'm not sure I buy into this, though. Why is it harder for people that are interested in EC2 support to work in the existing nova repo? If people want to collaborate before pushing into mainline, that can be done via feature branches, too. It risks making EC2 development harder, as well. Pulling it out of nova completely risks allowing the people that don't care about EC2 to care even less. It could make it easier for people to make changes that make EC2 compatibility harder to maintain. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
On Monday, April 23, 2012 at 4:00 PM, Joshua Harlow wrote: Re: [Openstack] Canonical AWSOME How are REST endpoints not reliable or scalable ;-) I’d like to know, seeing as the web is built on them :-) The resiliency of the internet is actually built on BGP. REST endpoints fall over constantly. Look no further than Google 500 errors, the fail-whale, etc. Even the EC2 API has been known to fall-over. Making HTTP services reliable is not as trivial as it should be. The reason is because they are single points. It is possible, through running many services and doing intelligent load balancing and failover, to make REST reasonably reliable. However, I'd rather not broker my requests through a questionably reliable REST broker, and send messages directly to their destinations to RPC consumers which are already running (and required) on those machines. If the destination is offline, it doesn't need my message. If the REST broker is offline, the recipient on the other end of that broker should still be guaranteed delivery… The problem can be simplified as: * How many REST endpoints do you need to service 100 compute machines? How many REST endpoints do you need to service 100 compute machines? How many points of failure exist? * How many compute machines do you need to service 100 compute machines? How many compute machines do you need to service 100 compute machines? How many points of failure exist? It is unclear how many REST endpoints you'll need. The compute machines scale as they scale, they're not dependent on a REST broker. Every compute machine itself can fail, although this failure is likely trivial (messages to a dead machine are generally vain). Meanwhile, the REST service has to deal with dead compute machines *and* the death of REST services supporting the architecture. -- Eric Windisch___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
+1 on this. Implementing adapter style code may also introduce additional state maintenance within the proxy/adapter to the eventual target API and if not at a minimum more complexity. Also care would be needed to avoid leaky abstractions [1]. However, I don't necessarily agree that a REST endpoint (esp. stateless ones like OSAPI) can't be scaled easily. Justin makes a very interesting point with regard to the non-extensible nature of the EC2 API (unlike that of OSAPI or OCCI). Perhaps it is an 'API parity' challenge ( i.e. where API X is functionally equivalent to OSAPI) and solution (having the ability and maintaining parity) that is key to the discussion? .2c Andy andy.edmonds.be [1] http://www.joelonsoftware.com/articles/LeakyAbstractions.html On Mon, Apr 23, 2012 at 19:01, Eric Windisch e...@cloudscaling.com wrote: Creating a contract on the private API will allow the external APIs to be created and tested without needing a translation layer, even if contributory APIs were developed outside of the project (such as in AWSOME). It is clearly better, architecturally, if the EC2/OCCI apis can access the internal apis directly. The RPC and database can be made to scale in Nova, but a REST endpoint won't be as reliable or scale as well. -- Eric Windisch On Monday, April 23, 2012 at 11:15 AM, Justin Santa Barbara wrote: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? Although I wasn't able to attend the design summit session, right now we have two native APIs, which means we have two paths into the system. That is poor software engineering, because we must code and debug everything twice. Some developers will naturally favor one API over the other, and so disparities happen. Today, both APIs are effectively using an undocumented private API, which is problematic. We also can't really extend the EC2 API, so it is holding us back as we extend OpenStack's capabilities past those of the legacy clouds. With one native API, we can focus all our energies on making sure that API works. Then, knowing that the native API works, we can build other APIs on top through simple translation layers, and they will work also. Other APIs can be built on top in the same way (e.g. OCCI) Which is a long way of saying the external approach will result in _all_ APIs (OpenStack, EC2, OCCI etc) becoming more reliable, more secure and just more AWSOME. Justin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
+1 - this is important for both incubated projects and vendor implementations. I seem to recall one of the sessions talking about gating commits to passing an upgrade test from the previous stable release and these interfaces are an obvious candidate. Identifying the specific interfaces will take some work, but the plugin points (e.g. scheduler, rpc, etc) are a good starting point. Michael - Michael Fork Cloud Architect, Emerging Solutions IBM Systems Technology Group From: Justin Santa Barbara jus...@fathomdb.com To: Eric Windisch e...@cloudscaling.com, Cc: openstack openstack@lists.launchpad.net Date: 04/23/2012 05:43 PM Subject:Re: [Openstack] Canonical AWSOME Sent by:openstack-bounces+mjfork=us.ibm@lists.launchpad.net On Mon, Apr 23, 2012 at 12:31 PM, Eric Windisch e...@cloudscaling.com wrote: There seemed to be a strong agreement at the summit regarding the need for contracts on those private apis. This is because those APIs are no longer really private, they're shared amongst incubated projects. Furthermore, it seems this may be required to support version heterogeneity during upgrades with versioned RPC calls. The incubated projects could use REST APIs, but we're talking of introducing artificial scaling and reliability constraints to do that. It seems far better to me, if we can have contracts on those internal APIs and use them between the incubated projects. There is a strong enough push to maintain these versions *anyway*. Ah - thanks for explaining that. I'm all for locking down these internal interfaces. I didn't realize people were willing to do so. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp inline: graycol.gif___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
I have the same doubt as you Philipp. By the way, some days ago I read Citrix takes cloudstack to apache, abandons openstack [ http://www.pcworld.com/businesscenter/article/253084/citrix_takes_cloudstack_to_apache_abandons_openstack.html ] They said that one of the reasons was that they believe supporting Amazon's API is crucial to the success of cloud platforms and While OpenStack supports some Amazon APIs, it appears more interested in backing its own APIs. I could not understand, since I can use euca2ools without problem with openstack. +1 2012/4/17 Philipp Wollermann wollermann_phil...@cyberagent.co.jp Hi, I just saw the announcement of Canonical's AWSOME ( https://launchpad.net/awsome), providing a translation proxy from the EC2 API to the OpenStack API. I'm definitely not an expert regarding OpenStack, but… doesn't OpenStack already support an EC2 compatibility API? At least, I can use the euca2ools without problems on OpenStack Nova Diablo. I haven't tried Swift yet. What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? Thanks :) Philipp -- Alisson Pontes ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
Please see: http://etherpad.openstack.org/FolsomEC2Compatibility Euca2ools is a a start, but there is more than meets the eye that is missing/lacking/incorrect in the EC2 layer. And there are many ways of solving it, but I think one of the key take-aways from that session is that we need a validation suite in order to figure out even where to begin (ie how can u fix something if u do not know how broken it is in the first place). On 4/19/12 10:58 AM, Alisson Soares Limeira Pontes apon...@cpqd.com.br wrote: I have the same doubt as you Philipp. By the way, some days ago I read Citrix takes cloudstack to apache, abandons openstack [http://www.pcworld.com/businesscenter/article/253084/citrix_takes_cloudstack_to_apache_abandons_openstack.html] They said that one of the reasons was that they believe supporting Amazon's API is crucial to the success of cloud platforms and While OpenStack supports some Amazon APIs, it appears more interested in backing its own APIs. I could not understand, since I can use euca2ools without problem with openstack. +1 2012/4/17 Philipp Wollermann wollermann_phil...@cyberagent.co.jp Hi, I just saw the announcement of Canonical's AWSOME (https://launchpad.net/awsome), providing a translation proxy from the EC2 API to the OpenStack API. I'm definitely not an expert regarding OpenStack, but... doesn't OpenStack already support an EC2 compatibility API? At least, I can use the euca2ools without problems on OpenStack Nova Diablo. I haven't tried Swift yet. What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? Thanks :) Philipp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Canonical AWSOME
Hi, I just saw the announcement of Canonical's AWSOME (https://launchpad.net/awsome), providing a translation proxy from the EC2 API to the OpenStack API. I'm definitely not an expert regarding OpenStack, but… doesn't OpenStack already support an EC2 compatibility API? At least, I can use the euca2ools without problems on OpenStack Nova Diablo. I haven't tried Swift yet. What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? Thanks :) Philipp -- Philipp Wollermann Infrastructure Engineer CyberAgent, Inc. (Tokyo) https://github.com/philwo ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
