Hi Kurt, On Fri, Feb 18, 2011 at 12:11:28PM -0500, Kurt Griffiths wrote: > Personally, I've been interested in learning Erlang as well. Eric, could you > explain more about why you anticipate that the queuing service will > CPU-bound? In either case, I've heard that Erlang can actually make better > use of multi-procs than C++, but I don't have any experience to speak from.
I wrote the majority of another queue server (Gearman C server) and in doing the performance testing and analysis there I found that for high speed network links, the CPU could certainly become the bottleneck before the network. Obviously the disk would bottleneck before either if we were using only persistent messages, but for in-memory only queues it's CPU bound. You can see this directly by increasing the number of threads (and therefore cores) for Gearman to use, with one core pushing about 18k messages/second, and four cores pushing close to 50k messages/second. Efficient multi-core use and protocol buffering/parsing/routing will make a difference. The buffering/parsing/routing code is where we'll see the higher level languages peg the CPU before C++. Having said all that, it may still be a worthwhile trade-off given the speed of development and ease of use with Erlang. Regarding Erlang using threads more efficiently than C++, this all depends on how you use threads in C++. :) Since Erlang itself is written in C, you can be just as efficient (and more) than Erlang if you do it the correct way. What Erlang does give you is the ease of using multiple threads/cores without really having to think about it. There is no shared state between 'processes' in Erlang (think lightweight cooperative threads), and therefore no need for locks, so concurrent programming is much less error prone. -Eric > > >Personally, I'd prefer C++ since that's what I'm used to, but I'd be open to > >learning Erlang, too. Been wanting to learn it for a while now. > > >-jay > > On Fri, Feb 18, 2011 at 1:11 AM, Eric Day <e...@oddments.org> wrote: > > Duck farm? :) > > > > Are you two concerned about building a developer community around a > > project in Erlang? I'm all for going that route if other folks are > > comfortable with it. > > > > I also have some concern around the speed of Erlang. It's great, > > especially if you know what primitives can be expensive and how to > > tame beam, but it will always be slower (sometimes significantly) that > > C/C++ for some tasks. > > > > -Eric > > > > On Thu, Feb 17, 2011 at 03:26:09PM -0800, Joshua McKenty wrote: > >> +1 for Erlang, as long as we have a duck farm. > >> > >> On Thu, Feb 17, 2011 at 2:25 PM, Devin Carlen > >> <devin.car...@gmail.com> > >> wrote: > >> > >> I'll put in a +1 for Erlang as an OpenStack supported platform. > >> We'd be > >> able to write a stable queue with a much more concise code base, > >> and > >> this is project would be a great fit for Erlang. > >> Devin > >> On Feb 17, 2011, at 2:21 PM, Eric Day wrote: > >> > >> > Thanks to everyone who gave great feedback on the first queue > >> service > >> > thread. I've updated the wiki page to include the suggestions. > >> > > >> > http://wiki.openstack.org/QueueService > >> > > >> > With a decent vision of what we want to build, the next step > >> is > >> > figuring out how. In a previous thread it was suggested that > >> the > >> > preferred languages for OpenStack projects are Python, C, and > >> > C++. Since there is an emphasis on speed and efficiency for > >> the > >> > queue service, I suggest we use C++. I expect this service to > >> be > >> > CPU bound and would benefit being able to leverage multiple > >> cores > >> > efficiently (within the same process), so I don't think Python > >> > is a good fit. I think C++ is a better fit than C due to the > >> need > >> > for modular interfaces. While this can obviously be done in C, > >> C++ > >> > APIs are more concise and much less error prone. The OO style > >> will > >> > also make it easier for Python developers who also want to > >> learn and > >> > assist with C++ projects. > >> > > >> > Erlang is not on the preferred lists, but I would also put it > >> out > >> > there as an option. While it may be a great fit for a project > >> like > >> > this, I worry it won't attract the developer resources since > >> Erlang > >> > isn't really a first-class language yet. > >> > > >> > If we decide to take the C++ path, I propose using a modular > >> > application framework I've been working on over the past year > >> (mostly > >> > in my spare time). It provides a simple module programming > >> interface > >> > with dependency tracking (kind of like Linux kernel modules). > >> It > >> > already provides a multi-threaded event module (currently > >> based on > >> > libevent, but this is pluggable) with simple networking > >> abstractions > >> > built on top of it. We should be able to dive in an start > >> writing the > >> > HTTP protocol module and queue processing modules. You can > >> check out > >> > the current project at: > >> > > >> > https://launchpad.net/scalestack > >> > http://scalestack.org/ > >> > > >> > The intention of using a framework like this is so we can > >> easily reuse > >> > the other modules (auth, HTTP, logging, ...) for other > >> OpenStack > >> > services in the future. Much like we use Eventlet, WSGI, etc. > >> (and > >> > eventually openstack-common) for Python, we could prefer using > >> the > >> > modules in Scale Stack for lower level projects. > >> > > >> > Thoughts? > >> > -Eric > >> > > >> > _______________________________________________ > >> > Mailing list: https://launchpad.net/~openstack > >> > Post to : openstack@lists.launchpad.net > >> > Unsubscribe : https://launchpad.net/~openstack > >> > More help : https://help.launchpad.net/ListHelp > >> > >> _______________________________________________ > >> Mailing list: https://launchpad.net/~openstack > >> Post to : openstack@lists.launchpad.net > >> Unsubscribe : https://launchpad.net/~openstack > >> More help : https://help.launchpad.net/ListHelp > > > > _______________________________________________ > > Mailing list: https://launchpad.net/~openstack Post to : > > openstack@lists.launchpad.net Unsubscribe : > > https://launchpad.net/~openstack More help : > > https://help.launchpad.net/ListHelp > > > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
