Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-13 Thread Flavio Percoco
On 12/06/14 16:27 +, Janczuk, Tomasz wrote: What exactly is the core set of functionalities Marconi expects all implementations to support? (I understand it is a subset of the HTTP APIs Marconi exposes?) Correct. What the exact calls are, we still don't know. We've talked about them a bit a

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-12 Thread Janczuk, Tomasz
What exactly is the core set of functionalities Marconi expects all implementations to support? (I understand it is a subset of the HTTP APIs Marconi exposes?) On 6/12/14, 4:56 AM, "Flavio Percoco" wrote: >On 11/06/14 16:26 -0700, Devananda van der Veen wrote: >>On Tue, Jun 10, 2014 at 1:23 AM,

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-12 Thread Flavio Percoco
On 11/06/14 16:26 -0700, Devananda van der Veen wrote: On Tue, Jun 10, 2014 at 1:23 AM, Flavio Percoco wrote: Against: • Makes it hard for users to create applications that work across multiple clouds, since critical functionality may or may not be available in a given deployment. (coun

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Clint Byrum
Excerpts from Janczuk, Tomasz's message of 2014-06-11 10:05:54 -0700: > On 6/11/14, 2:43 AM, "Gordon Sim" wrote: > > >On 06/10/2014 09:57 PM, Janczuk, Tomasz wrote: > >> Using processes to isolate tenants is certainly possible. There is a > >>range > >> of isolation mechanisms that can be used, f

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Devananda van der Veen
On Tue, Jun 10, 2014 at 1:23 AM, Flavio Percoco wrote: >> Against: >> >> • Makes it hard for users to create applications that work across >> multiple >>clouds, since critical functionality may or may not be available in a >> given >>deployment. (counter: how many users need cross-cloud c

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Gordon Sim
On 06/11/2014 06:05 PM, Janczuk, Tomasz wrote: Process level isolation is more costly than sub-process level isolation primarily due to larger memory consumption. For example, CGI has worse cost characteristics than FastCGI when scaled out. But the example closer to Marconi¹s use case is database

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Janczuk, Tomasz
On 6/11/14, 2:43 AM, "Gordon Sim" wrote: >On 06/10/2014 09:57 PM, Janczuk, Tomasz wrote: >> Using processes to isolate tenants is certainly possible. There is a >>range >> of isolation mechanisms that can be used, from VM level isolation >> (basically a separate deployment of the broker per-tenan

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Duncan Thomas
On 10 June 2014 09:23, Flavio Percoco wrote: > On 09/06/14 19:31 +, Kurt Griffiths wrote: >> Against: >> >> • Makes it hard for users to create applications that work across >> multiple >>clouds, since critical functionality may or may not be available in a >> given >>deployment. (co

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Flavio Percoco
On 11/06/14 13:42 +0100, Gordon Sim wrote: On 06/11/2014 01:28 PM, Flavio Percoco wrote: There are three things that I consider really valuable: 1. Sharding support: It allows admins to have separate clusters/nodes of stores and configure marconi to use them all based on pre-configured rules. I

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Gordon Sim
On 06/11/2014 01:28 PM, Flavio Percoco wrote: There are three things that I consider really valuable: 1. Sharding support: It allows admins to have separate clusters/nodes of stores and configure marconi to use them all based on pre-configured rules. It's similar to what qpid-dispatch does but t

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Flavio Percoco
On 11/06/14 13:01 +0100, Gordon Sim wrote: On 06/11/2014 12:31 PM, Flavio Percoco wrote: On 06/10/2014 09:59 PM, Mark McLoughlin wrote: Now why is there a desire to implement these requirements using traditional message brokers? [...] There are 2 main reasons that I don't believe are strong

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Gordon Sim
On 06/11/2014 12:31 PM, Flavio Percoco wrote: On 06/10/2014 09:59 PM, Mark McLoughlin wrote: Now why is there a desire to implement these requirements using traditional message brokers? [...] There are 2 main reasons that I don't believe are strong enough to change the way Marconi works right

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Flavio Percoco
On 11/06/14 10:19 +0100, Gordon Sim wrote: On 06/10/2014 09:59 PM, Mark McLoughlin wrote: Now why is there a desire to implement these requirements using traditional message brokers? I would speculate that any desire to utilise a message broker as opposed to a database would be to achieve dif

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Gordon Sim
On 06/10/2014 09:57 PM, Janczuk, Tomasz wrote: > Using processes to isolate tenants is certainly possible. There is a range > of isolation mechanisms that can be used, from VM level isolation > (basically a separate deployment of the broker per-tenant), to process > level isolation, to sub-process

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-11 Thread Gordon Sim
On 06/10/2014 09:59 PM, Mark McLoughlin wrote: Now why is there a desire to implement these requirements using traditional message brokers? I would speculate that any desire to utilise a message broker as opposed to a database would be to achieve different performance characteristics for the

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Janczuk, Tomasz
> Either those semantics are fundamental requirements for this API, or the requirement to have support for traditional message brokers is the fundamental requirement. We can't have it both ways. This captures the key trade off well. I would generalize it a bit to say that the larger the scope of t

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Mark McLoughlin
On Tue, 2014-06-10 at 21:59 +0100, Mark McLoughlin wrote: > On Tue, 2014-06-10 at 17:33 +, Janczuk, Tomasz wrote: > > From my perspective the key promise of Marconi is to provide a > > *multi-tenant*, *HTTP* based queuing system. Think an OpenStack equivalent > > of SQS or Azure Storage Queues.

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Mark McLoughlin
On Tue, 2014-06-10 at 17:33 +, Janczuk, Tomasz wrote: > From my perspective the key promise of Marconi is to provide a > *multi-tenant*, *HTTP* based queuing system. Think an OpenStack equivalent > of SQS or Azure Storage Queues. > > As far as I know there are no off-the-shelve message brokers

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Janczuk, Tomasz
Using processes to isolate tenants is certainly possible. There is a range of isolation mechanisms that can be used, from VM level isolation (basically a separate deployment of the broker per-tenant), to process level isolation, to sub-process isolation. The higher the density the lower the overall

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Gordon Sim
On 06/10/2014 06:33 PM, Janczuk, Tomasz wrote: From my perspective the key promise of Marconi is to provide a *multi-tenant*,*HTTP* based queuing system. Think an OpenStack equivalent of SQS or Azure Storage Queues. As far as I know there are no off-the-shelve message brokers out these that fi

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Janczuk, Tomasz
>From my perspective the key promise of Marconi is to provide a *multi-tenant*, *HTTP* based queuing system. Think an OpenStack equivalent of SQS or Azure Storage Queues. As far as I know there are no off-the-shelve message brokers out these that fit that bill. Note that when I say ³multi-tenant²

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Kurt Griffiths
> Will Marconi only support HTTP as a transport, or will it add other >protocols as well? We are focusing on HTTP for Juno, but are considering adding a lower-level, persistent transport (perhaps based on WebSocket) in the K cycle. > Can anyone describe what is unique about the Marconi design wit

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Gordon Sim
On 06/10/2014 05:27 PM, Kurt Griffiths wrote: I think the crux of the issue is that Marconi follows the REST architectural style. As such, the client must track the state of where it is in the queue it is consuming (to keep the server stateless). So, it must be given some kind of marker, allowing

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Kurt Griffiths
> What are 'message feeds' in the Marconi context, in more detail? And >what aspect of them is it that message brokers don't support? Great question. When I say “feeds” I mean a “feed” in the sense of RSS or Atom. People do, in fact, use Atom to implement certain messaging patterns. You can think

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Gordon Sim
On 06/09/2014 08:31 PM, Kurt Griffiths wrote: Lately we have been talking about writing drivers for traditional message brokers that will not be able to support the message feeds part of the API. Could you elaborate a little on this point? In some sense of the term at least, handling message f

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Gordon Sim
On 06/10/2014 09:48 AM, Mark McLoughlin wrote: Perhaps the first point to get super clear on is why drivers for traditional message brokers are needed. What problems would such drivers address? Who would the drivers help? Would the Marconi team recommend using any of those drivers for a productio

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Julien Danjou
On Mon, Jun 09 2014, Doug Hellmann wrote: > We went with a single large storage API in ceilometer initially, but > we had some discussions at the Juno summit about it being a bad > decision because it resulted in storing some data like alarm > definitions in database formats that just didn't make

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Flavio Percoco
On 10/06/14 09:48 +0100, Mark McLoughlin wrote: On Mon, 2014-06-09 at 19:31 +, Kurt Griffiths wrote: Lately we have been talking about writing drivers for traditional message brokers that will not be able to support the message feeds part of the API. I’ve started to think that having a huge

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Mark McLoughlin
On Mon, 2014-06-09 at 19:31 +, Kurt Griffiths wrote: > Lately we have been talking about writing drivers for traditional > message brokers that will not be able to support the message feeds > part of the API. I’ve started to think that having a huge part of the > API that may or may not “work”,

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-10 Thread Flavio Percoco
On 09/06/14 19:31 +, Kurt Griffiths wrote: Folks, this may be a bit of a bombshell, but I think we have been dancing around the issue for a while now and we need to address it head on. Let me start with some background. Back when we started designing the Marconi API, we knew that we wanted t

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-09 Thread Janczuk, Tomasz
ev@lists.openstack.org>> Subject: [openstack-dev] [marconi] Reconsidering the unified API model Folks, this may be a bit of a bombshell, but I think we have been dancing around the issue for a while now and we need to address it head on. Let me start with some background. Back when we started d

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-09 Thread Doug Hellmann
On Mon, Jun 9, 2014 at 3:31 PM, Kurt Griffiths wrote: > Folks, this may be a bit of a bombshell, but I think we have been dancing > around the issue for a while now and we need to address it head on. Let me > start with some background. > > Back when we started designing the Marconi API, we knew t

Re: [openstack-dev] [marconi] Reconsidering the unified API model

2014-06-09 Thread Sam Harwell
, 2014 2:31 PM To: OpenStack Dev Subject: [openstack-dev] [marconi] Reconsidering the unified API model Folks, this may be a bit of a bombshell, but I think we have been dancing around the issue for a while now and we need to address it head on. Let me start with some background. Back when we

[openstack-dev] [marconi] Reconsidering the unified API model

2014-06-09 Thread Kurt Griffiths
Folks, this may be a bit of a bombshell, but I think we have been dancing around the issue for a while now and we need to address it head on. Let me start with some background. Back when we started designing the Marconi API, we knew that we wanted to support several messaging patterns. We could