Re: [Openstack] Queue Service, next steps
I have been meaning to reply with the exact same thing. On Feb 18, 2011, at 10:12 AM, Jay Pipes wrote: > Personally, I'd prefer C++ since that's what I'm used to, but I'd be > open to learning Erlang, too. Been wanting to learn it for a while > now. > > -jay > > On Fri, Feb 18, 2011 at 1:11 AM, Eric Day wrote: >> Duck farm? :) >> >> Are you two concerned about building a developer community around a >> project in Erlang? I'm all for going that route if other folks are >> comfortable with it. >> >> I also have some concern around the speed of Erlang. It's great, >> especially if you know what primitives can be expensive and how to >> tame beam, but it will always be slower (sometimes significantly) >> that C/C++ for some tasks. >> >> -Eric >> >> On Thu, Feb 17, 2011 at 03:26:09PM -0800, Joshua McKenty wrote: >>>+1 for Erlang, as long as we have a duck farm. >>> >>>On Thu, Feb 17, 2011 at 2:25 PM, Devin Carlen >>>wrote: >>> >>> I'll put in a +1 for Erlang as an OpenStack supported platform. We'd >>> be >>> able to write a stable queue with a much more concise code base, and >>> this is project would be a great fit for Erlang. >>> Devin >>> On Feb 17, 2011, at 2:21 PM, Eric Day wrote: >>> >>> > Thanks to everyone who gave great feedback on the first queue service >>> > thread. I've updated the wiki page to include the suggestions. >>> > >>> > http://wiki.openstack.org/QueueService >>> > >>> > With a decent vision of what we want to build, the next step is >>> > figuring out how. In a previous thread it was suggested that the >>> > preferred languages for OpenStack projects are Python, C, and >>> > C++. Since there is an emphasis on speed and efficiency for the >>> > queue service, I suggest we use C++. I expect this service to be >>> > CPU bound and would benefit being able to leverage multiple cores >>> > efficiently (within the same process), so I don't think Python >>> > is a good fit. I think C++ is a better fit than C due to the need >>> > for modular interfaces. While this can obviously be done in C, C++ >>> > APIs are more concise and much less error prone. The OO style will >>> > also make it easier for Python developers who also want to learn and >>> > assist with C++ projects. >>> > >>> > Erlang is not on the preferred lists, but I would also put it out >>> > there as an option. While it may be a great fit for a project like >>> > this, I worry it won't attract the developer resources since Erlang >>> > isn't really a first-class language yet. >>> > >>> > If we decide to take the C++ path, I propose using a modular >>> > application framework I've been working on over the past year (mostly >>> > in my spare time). It provides a simple module programming interface >>> > with dependency tracking (kind of like Linux kernel modules). It >>> > already provides a multi-threaded event module (currently based on >>> > libevent, but this is pluggable) with simple networking abstractions >>> > built on top of it. We should be able to dive in an start writing the >>> > HTTP protocol module and queue processing modules. You can check out >>> > the current project at: >>> > >>> > https://launchpad.net/scalestack >>> > http://scalestack.org/ >>> > >>> > The intention of using a framework like this is so we can easily >>> reuse >>> > the other modules (auth, HTTP, logging, ...) for other OpenStack >>> > services in the future. Much like we use Eventlet, WSGI, etc. (and >>> > eventually openstack-common) for Python, we could prefer using the >>> > modules in Scale Stack for lower level projects. >>> > >>> > Thoughts? >>> > -Eric >>> > >>> > ___ >>> > Mailing list: https://launchpad.net/~openstack >>> > Post to : openstack@lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~openstack >>> > More help : https://help.launchpad.net/ListHelp >>> >>> ___ >>> Mailing list: https://launchpad.net/~openstack >>> Post to : openstack@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~openstack >>> More help : https://help.launchpad.net/ListHelp >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp Confidentiality Notice: This e-mail messa
Re: [Openstack] Queue Service, next steps
Hi Mike, On Sat, Feb 19, 2011 at 07:42:12PM -0600, Michael Barton wrote: > On Fri, Feb 18, 2011 at 9:38 PM, Eric Day wrote: > > Hi Mike, > > > > You make a good point, I apologize for not documenting some of the > > ideas sooner. The architecture I had in mind borrows from other > > queue systems I've worker on, along with feedback I've gotten from > > various conversations while getting things started. For the hashing, > > we could possibly reuse the distribution ring algorithm swift uses for > > selecting server sets from accounts. I've added some sample deployment > > descriptions to the wiki page: > > > > http://wiki.openstack.org/QueueService#Deployment_Configurations > > > That's cool, thanks for humoring me on that a bit. I was worried that > I hadn't seen anything suggesting this queue could meet some of our > expected needs. FWIW, I think the basic architecture is probably > good. It'll be interesting to see how different queue features can > interact with CAP theorem. The goal is AP, but of course do our best with C when possible to keep the UX acceptable. For this service it comes down to workers being idempotent, which we'll need to state clearly for users. Of course, all work being pushed through the queue doesn't need to be idempotent, there are various application-specific ways workers can deal with duplicates when needed. -Eric ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Queue Service, next steps
On Fri, Feb 18, 2011 at 9:38 PM, Eric Day wrote: > Hi Mike, > > You make a good point, I apologize for not documenting some of the > ideas sooner. The architecture I had in mind borrows from other > queue systems I've worker on, along with feedback I've gotten from > various conversations while getting things started. For the hashing, > we could possibly reuse the distribution ring algorithm swift uses for > selecting server sets from accounts. I've added some sample deployment > descriptions to the wiki page: > > http://wiki.openstack.org/QueueService#Deployment_Configurations That's cool, thanks for humoring me on that a bit. I was worried that I hadn't seen anything suggesting this queue could meet some of our expected needs. FWIW, I think the basic architecture is probably good. It'll be interesting to see how different queue features can interact with CAP theorem. -- Mike (re-sent to list, forgot to "reply all") ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Queue Service, next steps
Hi Todd, That's the multicast example, for a normal 1-1 queue, look at the first example: Worker: POST /account/queue?wait=60&hide=60&detail=all (long-polling worker, request blocks until a message is ready) Client: PUT /account/queue (message inserted, returns unique id that was created) Worker: Return from blocking POST request with new message and process it. The message is returned to only one worker when the hide option is given, as this is an atomic operation. Worker: DELETE /account/queue/id You could have more workers waiting with the same POST request, and the server will return the message to only one of them. There are options we can look at later in regards worker distribution, affinity, etc. I currently have it using POST instead of GET since it is really a modification operation, not a read-only operation. -Eric > How do you manage the case where you want many workers to service a > queue, and use long-polling to wait for a message, but you don't want > the message multicast to every worker? It looks like from the wiki > examples: > > Worker1: GET /account/queue?wait=60 > Worker2: GET /account/queue?wait=60 > Client: Put /account/queue/id1?ttl=60 > Worker1: Return from blocking GET request with message id1 > Worker2: Return from blocking GET request with message id1 > Worker1: GET /account/queue?wait=60&last=id1 > Worker2: GET /account/queue?wait=60&last=id1 > > i'd like to have worker1 receive message with id1, and worker2 receive > the next message (id2), even though they are both long polling when > id1 comes in. Will this be supported, or does long polling on the > same queue always yield multicast? > > -todd[1] ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Queue Service, next steps
When you want to modify messages in a queue and also return those that were modified. This is the atomic get/set needed by workers. POST /account/queue?hide=60 HTTP/1.1 This will return all messages currently in /account/queue, but also mark them as hidden for 60 seconds. This ensures only one worker was able to see them, at least until the hide value expires. -Eric On Sat, Feb 19, 2011 at 02:13:49PM -0500, Mark Washenberger wrote: > Can you give me some examples of the kinds of "update all messages in a > queue" requests you're thinking of? > > Thanks, > > Mark > > "Eric Day" said: > > > > I was wondering if someone would point out this discrepancy with > > traditional REST semantics. :) The reason for the change is the following: > > > > We need both an insert and update method for the queue and message > > level. Some map well, but the "update all messages in a queue" (or the > > set matching some criteria) is the oddball. A PUT .../queue doesn't > > quite fit because we're not inserting or replacing a queue, we're > > modifying all messages in it. I wanted to keep GET/HEAD read-only and > > side-effect free, so I didn't want to overload those. Another option > > was to use reserved words, possibly those prefixed with an underscore > > (such as POST ../queue/_all), but this seemed unnecessary. > > > > Instead, I propose using POST ../queue to mean modify the > > queue (and possibly all messages in it). POST implies more of a > > modify/append/process operation that PUT, which should strictly be > > insert/replace. Using this form meant not using POST ../queue for > > the traditional "insert but auto-generate ID" semantics, which is > > why I used PUT instead. In some ways this is more consistent too, > > since you are still inserting a message like PUT ../queue/message, > > just without the id. > > > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Queue Service, next steps
Hi Mark, On Sat, Feb 19, 2011 at 11:18:26AM -0500, Mark Washenberger wrote: > It seems like put and post on ../queue[/id] are reversed from the usual > sense. I am probably just not familiar with the idioms for previous queue > services, but I imagine the following scheme. > > POST .../queue # create a message without specifying the id > PUT .../queue/id # create a message with a specific id, > # or modify existing message's metadata > PUT .../queue # create a queue or modify existing queue's metadata > POST .../queue/id # drop this command (?) I was wondering if someone would point out this discrepancy with traditional REST semantics. :) The reason for the change is the following: We need both an insert and update method for the queue and message level. Some map well, but the "update all messages in a queue" (or the set matching some criteria) is the oddball. A PUT .../queue doesn't quite fit because we're not inserting or replacing a queue, we're modifying all messages in it. I wanted to keep GET/HEAD read-only and side-effect free, so I didn't want to overload those. Another option was to use reserved words, possibly those prefixed with an underscore (such as POST ../queue/_all), but this seemed unnecessary. Instead, I propose using POST ../queue to mean modify the queue (and possibly all messages in it). POST implies more of a modify/append/process operation that PUT, which should strictly be insert/replace. Using this form meant not using POST ../queue for the traditional "insert but auto-generate ID" semantics, which is why I used PUT instead. In some ways this is more consistent too, since you are still inserting a message like PUT ../queue/message, just without the id. If this divergence from traditional REST semantics is not welcome, we can certainly change it. We do need to find the best way to represent an atomic modify/get operation at the queue level though. > Is the FIFO constraint on GET .../queue a strong constraint? or more of an > approximation or eventual consistency type of weaker guarantee? For a single queue server, yes. When using an aggregate of all queue servers from a proxy or multi-connection worker, you'll need to handle this in the worker code since messages may be interleaved from multiple queue servers. > Looking at the public cloud service deployment diagram and notes, I started > to wonder about the performance demands of different use cases. I imagine > that a given account or a given queue could vary widely through time in the > number of messages produced and messages consumed. So an individual queue > might have a very high variance in the distribution of cpu and disk resources > consumed. I am worried about sharding hardware resources on the basis of the > account id alone. It seems like this would prevent the queue service from > scaling out a single queue to cluster level and would create hot-spots. Do we > need to worry about this type of use case? Am I missing how this would be > handled in your deployment example? I suppose there is no reason you couldn't perform the first level hash on account/queue instead of just account. In fact I think the plugin point for controlling distribution should do just this, pass in the full URI and server set, and return some set of servers. It can hash on whatever it likes. A distribution plugin could also use consistent hashing so we could optionally grow/shrink the queue server set if we notice hotspots for a particular set. > Going way back to a previous discussion on this thread--you had rejected IPC > because it would limit performance. It seems to me if we used an IPC-based > model of coordination across cores, it would be analogous to the model used > for scaling across multiple nodes in a zone. This approach would help us > scale an individual queue beyond the limitations of a single machine or HA > triplet. If I recall, that is sort of like the way erlang scales across > multiple machines. Not that I would want to deal with erlang's syntactic salt. Erlang doesn't use IPC across machines (well, not traditional System V IPC mechanisms), it's a socket daemon called epmd (Erlang Port Mapper Daemon) that forwards messages between different nodes. Within a single machine Erlang uses a multi-threaded event loop that maps all Erlang processes on top of it. This lets you efficiently use all cores and provide very lightweight message passing between the Erlang processes running on those cores (much lighter weight than any IPC mechanism). Having said all that, I suppose we could run multiple, independent queue servers per machine (one per core), and have each running on their own port. Each queue server process would represent a different node in the distribution set. The arguments against this are that it limits resource sharing for a machine since the processes would not know of one another. For example, if we have local persistent storage, this means multiple pro
Re: [Openstack] Nova-core membership request
+1 One more. On 2/18/11 9:15 PM, "Ed Leafe" wrote: >On Feb 18, 2011, at 6:07 PM, Rick Harris wrote: > >> Throwing my hat into the ring for nova-core membership. Eager to help >>knock down that merge-prop backlog we have at the moment :-) > >+1 > > >-- Ed Leafe > > > > >___ >Mailing list: https://launchpad.net/~openstack >Post to : openstack@lists.launchpad.net >Unsubscribe : https://launchpad.net/~openstack >More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
