Kurt Griffiths, Thanks for detailed explanation. Is there a comparison between Marconi and existing message brokers anywhere that you can point me out? I can see how your examples can be implemented using other brokers like RabbitMQ. So why there is a need another broker? And what is wrong with currently deployed RabbitMQ that most of OpenStack services are using (typically via oslo.messaging RPC)?
On Wed, Mar 19, 2014 at 4:00 AM, Kurt Griffiths < kurt.griffi...@rackspace.com> wrote: > I think we can agree that a data-plane API only makes sense if it is > useful to a large number of web and mobile developers deploying their apps > on OpenStack. Also, it only makes sense if it is cost-effective and > scalable for operators who wish to deploy such a service. > > Marconi was born of practical experience and direct interaction with > prospective users. When Marconi was kicked off a few summits ago, the > community was looking for a multi-tenant messaging service to round out > the OpenStack portfolio. Users were asking operators for something easier > to work with and more web-friendly than established options such as AMQP. > > To that end, we started drafting an HTTP-based API specification that > would afford several different messaging patterns, in order to support the > use cases that users were bringing to the table. We did this completely in > the open, and received lots of input from prospective users familiar with > a variety of message broker solutions, including more "cloudy" ones like > SQS and Iron.io. > > The resulting design was a hybrid that supported what you might call > "claim-based" semantics ala SQS and feed-based semantics ala RSS. > Application developers liked the idea of being able to use one or the > other, or combine them to come up with new patterns according to their > needs. For example: > > 1. A video app can use Marconi to feed a worker pool of transcoders. When > a video is uploaded, it is stored in Swift and a job message is posted to > Marconi. Then, a worker claims the job and begins work on it. If the > worker crashes, the claim expires and the message becomes available to be > claimed by a different worker. Once the worker is finished with the job, > it deletes the message so that another worker will not process it, and > claims another message. Note that workers never "list" messages in this > use case; those endpoints in the API are simply ignored. > > 2. A backup service can use Marconi to communicate with hundreds of > thousands of backup agents running on customers' machines. Since Marconi > queues are extremely light-weight, the service can create a different > queue for each agent, and additional queues to broadcast messages to all > the agents associated with a single customer. In this last scenario, the > service would post a message to a single queue and the agents would simply > list the messages on that queue, and everyone would get the same message. > This messaging pattern is emergent, and requires no special routing setup > in advance from one queue to another. > > 3. A metering service for an Internet application can use Marconi to > aggregate usage data from a number of web heads. Each web head collects > several minutes of data, then posts it to Marconi. A worker periodically > claims the messages off the queue, performs the final aggregation and > processing, and stores the results in a DB. So far, this messaging pattern > is very much like example #1, above. However, since Marconi's API also > affords the observer pattern via listing semantics, the metering service > could run an auditor that logs the messages as they go through the queue > in order to provide extremely valuable data for diagnosing problems in the > aggregated data. > > Users are excited about what Marconi offers today, and we are continuing > to evolve the API based on their feedback. > > Of course, app developers aren't the only audience Marconi needs to serve. > Operators want something that is cost-effective, scales, and is > customizable for the unique needs of their target market. > > While Marconi has plenty of room to improve (who doesn't?), here is where > the project currently stands in these areas: > > 1. Customizable. Marconi transport and storage drivers can be swapped out, > and messages can be manipulated in-flight with custom filter drivers. > Currently we have MongoDB and SQLAlchemy drivers, and are exploring Redis > and AMQP brokers. Now, the v1.0 API does impose some constraints on the > backend in order to support the use cases mentioned earlier. For example, > an AMQP backend would only be able to support a subset of the current API. > Operators occasionally ask about AMQP broker support, in particular, and > we are exploring ways to evolve the API in order to support that. > > 2. Scalable. Operators can use Marconi's HTTP transport to leverage their > existing infrastructure and expertise in scaling out web heads. When it > comes to the backend, for small deployments with minimal throughput needs, > we are providing a SQLAlchemy driver as a non-AGPL alternative to MongoDB. > For large-scale production deployments, we currently provide the MongoDB > driver and will likely add Redis as another option (there is already a POC > driver). And, of course, operators can provide drivers for NewSQL > databases, such as VelocityDB, that are very fast and scale extremely > well. In Marconi, every queue can be associated with a different backend > cluster. This allows operators to scale both up and out, according to what > is most cost-effective for them. Marconi's app-level sharding is currently > done using a lookup table to provide for maximum operator control over > placement, but I personally think it would be great to see this opened up > so that we can swap in other types of drivers, such as one based on hash > rings (TBD). > > 3. Cost-effective. The Marconi team has done a lot of work to (1) provide > several dimensions for scaling deployments that can be used according to > what is most cost-effective for a given use case, and (2) make the Marconi > service as efficient as possible, including time spent optimizing the > transport layer (using Falcon in lieu of Pecan, reducing the work that the > request handlers do, etc.), and tuning the MongoDB storage driver (the > SQLAlchemy driver is newer and we haven't had the chance to tune it yet, > but are planning to do so during Juno). Turnaround on requests is in the > low ms range (including dealing with HTTP), not the usec range, but that > works perfectly well for a large class of applications. We've been > benchmarking with Tsung for quite a while now, and we are working on > making the raw data more accessible to folks outside our team. I'll try to > get some of the latest data up on the wiki this week. > > Marconi was originally incubated because the community believed developers > building their apps on top of OpenStack were looking for this kind of > service, and it was a big missing gap in our portfolio. Since that time, > the team has worked hard to fill that gap. > > Kurt > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
