> From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net
> [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf
> of Eric Day [e...@oddments.org]
> Sent: Tuesday, March 08, 2011 5:58 PM
> To: Todd Willey
> Cc: openstack@lists.launch
ric Day [e...@oddments.org]
Sent: Tuesday, March 08, 2011 5:58 PM
To: Todd Willey
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Queue Service Implementation Thoughts
On Tue, Mar 08, 2011 at 04:47:38PM -0500, Todd Willey wrote:
> With this switch to python, does it make sense to revisi
;>
> >> -Eric
> >>
> >> On Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote:
> >> > Eric,
> >> > Thanks for your experimentation and analysis. Somewhat illustrates
> >> > the
> >> > poi
> > Thanks for your experimentation and analysis. Somewhat illustrates
>> > the
>> > point about premature optimization. First cut, have to agree with you
>> > and
>> > conclude that python implementation is effective, overall. As you
>> >
if we require any complex processing at the http server
> > layer)
> >we can optimize specific areas.
> >Just for sure, might be better for someone else to recheck. That way
> >we have done our due diligence.
> >Cheers
> >
>
n Sat, Mar 05, 2011 at 10:39:18PM -0700, ksan...@doubleclix.net wrote:
>>>> Eric,
>>>> Thanks for your experimentation and analysis. Somewhat illustrates the
>>>> point about premature optimization. First cut, have to agree with you and
>>>> conclude that python imple
the
> >> point about premature optimization. First cut, have to agree with you and
> >> conclude that python implementation is effective, overall. As you said,if
> >> we find performance bottlenecks, especially as the payload size increases
> >> (as well
e find performance bottlenecks, especially as the payload size increases
>> (as well as if we require any complex processing at the http server layer)
>> we can optimize specific areas.
>> Just for sure, might be better for someone else to recheck. That way
>> we have done
Agree with Eric and Thierry on this.
John
On Mar 8, 2011, at 1:09 AM, Thierry Carrez wrote:
> Eric Day wrote:
>> If we hit performance issues with this type of application, we'll
>> probably hit them around the same time with both Erlang and Python
>> (then we'll look to C/C++). Since most Open
Eric Day wrote:
> If we hit performance issues with this type of application, we'll
> probably hit them around the same time with both Erlang and Python
> (then we'll look to C/C++). Since most OpenStack developers are a lot
> more comfortable with Python, I suggest we make the switch. Please
> res
l as if we require any complex processing at the http server layer)
>we can optimize specific areas.
>Just for sure, might be better for someone else to recheck. That way
>we have done our due diligence.
>Cheers
>
>
> ---- Original Message -
increases (as well as if we require any complex processing at the http server layer) we can optimize specific areas. Just for sure, might be better for someone else to recheck. That way we have done our due diligence.Cheers
Original Message
Subject: [Openstack] Queue Service
Hi everyone,
When deciding to move forward with Erlang, I first tried out the Erlang
REST framework webmachine (it is built on top of mochiweb and used
by projects like Riak). After some performance testing, I decided to
write a simple wrapper over the HTTP packet parsing built into Erlang
(also u
ric Day
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Queue Service, next steps
Eric,
Thank you.
You raise lots of interest points. In no particular order:-
AMQP Observations
Your comments about AMQP seem to mostly be appropriate for one of the older
versions, eg 0-8, and I don&#x
ld easily route
50k fully-synchronous messages/second on a 4-core machine. This
was also without any form of batching optimizations, which will be
part of the OpenStack queue service. The pull operations need to be
designed correctly so they are optimized for high throughput, which
as a result makes
- and still gather 10,000s of items of data a second.
> The
> >AMQP protocol, particularly 1.0, make it's extremely clear how and
> when to
> >recover from failure. Indeed, AMQP's approach is failure happens - so
> deal
> >with it. HTTP on the othe
he other hand, has no such level of transactionality.
For the complexity concern, my main point is that in order to use
a queue, you need a channel, exchange, queue, and a binding between
an exchange/queue. This can be made fairly trivial by libraries you
mentioned, but there are a lot of objects and r
Eric & Team,
OpenStack's QueueService seems very interesting. As we have an existing
message queue implementation, we'd be happy to help you guys out. We're
about making messaging cloud-scale, so that everyone benefits.
However, it worries us that you're planning to implement a REST API for
messa
On Feb 22, 2011, at 8:58 AM, Eric Day wrote:
> On Tue, Feb 22, 2011 at 12:55:32AM -0800, Vishvananda Ishaya wrote:
>> Awesome stuff!
>> I had to do the following patch to get it to run (on python 2.6.1 on my mac)
>
> Yeah, the version in my junk repo was broken for a couple minutes
> last night,
gt;>
>>> -Eric
>>>
>>> On Mon, Feb 21, 2011 at 02:47:21PM -0600, John Purrier wrote:
>>>> I agree with this. Unless there are significant, obvious advantages to
>>>> Erlang I would suggest we stick with C/C++.
>>>>
>>>>
On Tue, Feb 22, 2011 at 12:55:32AM -0800, Vishvananda Ishaya wrote:
> Awesome stuff!
> I had to do the following patch to get it to run (on python 2.6.1 on my mac)
Yeah, the version in my junk repo was broken for a couple minutes
last night, you must have pulled at just the right moment. :)
> I f
gt;
>>> John
>>>
>>> -Original Message-
>>> From: openstack-bounces+john=openstack@lists.launchpad.net
>>> [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf
>>> Of Tim Bell
>>> Sent: Monday, Febru
Awesome stuff!
I had to do the following patch to get it to run (on python 2.6.1 on my mac)
=== modified file 'osq.py'
--- osq.py 2011-02-22 08:24:10 +
+++ osq.py 2011-02-22 08:52:02 +
@@ -118,8 +118,9 @@
help='Match messages that are hidden')
self.parser.ad
I decided to implement the current queue service specification
in Python this weekend as it is described on the wiki page. It is
functionally complete minus persistent queues and account verification
(plugin points for the real implementation). You can find the links
and description here (as well a
Message-
>> From: openstack-bounces+john=openstack@lists.launchpad.net
>> [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf
>> Of Tim Bell
>> Sent: Monday, February 21, 2011 2:08 PM
>> To: Eric Day; openstack@lists.launchpad.net
>> Subj
To: Eric Day; openstack@lists.launchpad.net
> Subject: Re: [Openstack] Queue Service, next steps
>
>
> Please bear in mind the long term maintainability of the openstack package.
> One of the attractive features at the moment is that there are not
> significant pre-reqs to se
Bell
Sent: Monday, February 21, 2011 2:08 PM
To: Eric Day; openstack@lists.launchpad.net
Subject: Re: [Openstack] Queue Service, next steps
Please bear in mind the long term maintainability of the openstack package.
One of the attractive features at the moment is that there are not
significant pre
Please bear in mind the long term maintainability of the openstack package.
One of the attractive features at the moment is that there are not significant
pre-reqs to set up the environment and most mass market environments can
support it.
Using C++ would not significantly change this situati
We would without a doubt need people that have developed/deployed OTP
applications. I consider myself proficient at Erlang, but have not written a
proper OTP application. I do have enough experience to know that we will want
to use OTP libs everywhere appropriate.
For those wanting to learn.
On Mon, Feb 21, 2011 at 06:46:58PM +, Chris Behrens wrote:
> Do we have enough Erlang knowledge right now? I would not want to see Eric
> get burned out because he can't get help. I've seen it happen before when
> Erlang was chosen and the project ended up a complete failure.
Knowing the i
I guess that's what I'm trying to say for this project. My opinion would be to
stick with what we know. :)
Do we have enough Erlang knowledge right now? I would not want to see Eric get
burned out because he can't get help. I've seen it happen before when Erlang
was chosen and the project e
For the record, I said I'd prefer C++ because that's what I know, but
I wouldn't *mind* learning Erlang ;)
-jay
On Mon, Feb 21, 2011 at 8:34 AM, Chris Behrens
wrote:
> I am very concerned about seeing a lot of "I'd love to learn Erlang" on
> something we probably don't want to screw up. At the
I am very concerned about seeing a lot of "I'd love to learn Erlang" on
something we probably don't want to screw up. At the same time, I know there's
a number of us very experienced with C/C++ (even with respect to
concurrency/scalability).
- Chris
On Feb 21, 2011, at 2:40 AM, Thierry Carre
Quick comment on the languages from a dev community perspective:
The Nova/Swift community is Python-based, so you won't directly reuse
it, whether you use C++ or Erlang. For new contributors, you either
target very experienced C++ developers (to get concurrency/scalability
right), or moderately-ex
I have been meaning to reply with the exact same thing.
On Feb 18, 2011, at 10:12 AM, Jay Pipes wrote:
> Personally, I'd prefer C++ since that's what I'm used to, but I'd be
> open to learning Erlang, too. Been wanting to learn it for a while
> now.
>
> -jay
>
> On Fri, Feb 18, 2011 at 1:11 AM
Hi Mike,
On Sat, Feb 19, 2011 at 07:42:12PM -0600, Michael Barton wrote:
> On Fri, Feb 18, 2011 at 9:38 PM, Eric Day wrote:
> > Hi Mike,
> >
> > You make a good point, I apologize for not documenting some of the
> > ideas sooner. The architecture I had in mind borrows from other
> > queue systems
On Fri, Feb 18, 2011 at 9:38 PM, Eric Day wrote:
> Hi Mike,
>
> You make a good point, I apologize for not documenting some of the
> ideas sooner. The architecture I had in mind borrows from other
> queue systems I've worker on, along with feedback I've gotten from
> various conversations while ge
Hi Todd,
That's the multicast example, for a normal 1-1 queue, look at the
first example:
Worker: POST /account/queue?wait=60&hide=60&detail=all
(long-polling worker, request blocks until a message is ready)
Client: PUT /account/queue
(message inserted, returns unique id that was created)
Wor
When you want to modify messages in a queue and also return those
that were modified. This is the atomic get/set needed by workers.
POST /account/queue?hide=60 HTTP/1.1
This will return all messages currently in /account/queue, but also
mark them as hidden for 60 seconds. This ensures only one wo
Hi Mark,
On Sat, Feb 19, 2011 at 11:18:26AM -0500, Mark Washenberger wrote:
> It seems like put and post on ../queue[/id] are reversed from the usual
> sense. I am probably just not familiar with the idioms for previous queue
> services, but I imagine the following scheme.
>
> POST .../queue
Hi Mike,
You make a good point, I apologize for not documenting some of the
ideas sooner. The architecture I had in mind borrows from other
queue systems I've worker on, along with feedback I've gotten from
various conversations while getting things started. For the hashing,
we could possibly reus
On Fri, Feb 18, 2011 at 6:23 PM, Eric Day wrote:
> Perhaps I've been assuming some things, but I thought everyone
> understood that is what we are looking to build (fault-tolerant,
> horizontally scalable, ...). We're certainly not looking to build
> a clustered queue (like RabbitMQ) with a REST A
...CheersP.S: Have done enough of the queue stuff and will pitch in as time allows
Original Message
Subject: Re: [Openstack] Queue Service, next steps
From: Eric Day <e...@oddments.org>
Date: Fri, February 18, 2011 4:22 pm
To: Curtis Carter <curtis.car...@rackspace.com>;
C
Hi Mike,
On Fri, Feb 18, 2011 at 06:02:52PM -0600, Michael Barton wrote:
> Maybe I'm describing a separate project, but a fault tolerant and
> scalable queue would be more interesting to me than something like
> RabbitMQ with a REST interface. There don't seem to be any reasonable
> open-source i
On Fri, Feb 18, 2011 at 5:22 PM, Eric Day wrote:
> The main question right now is where to land on the spectrum of service
> efficiency vs ease of development (C/C++ on one end and Python on
> the other). It seems we're landing in the middle with Erlang. :)
Maybe I'm describing a separate project
Original Message-
> > From: openstack-bounces+philip.schwartz=lexisnexis@lists.launchpad.net
> > [mailto:openstack-bounces+philip.schwartz=lexisnexis@lists.launchpad.net]
> > On Behalf Of Eric Day
> > Sent: Friday, February 18, 2011 1:54 PM
> > To: k
rface, if all think this
>> is a good idea
>> * And on the topic of Erlang, it is viable. I have done some work with
>> it. If we also use WebMachine, REST interface would be easier.
>> Cheers
>>
>>
>> Original Message
>&g
use LDAP, and so on.
-Eric
> -Original Message-
> From: "Eric Day"
> Sent: Thursday, February 17, 2011 5:21pm
> To: openstack@lists.launchpad.net
> Subject: [Openstack] Queue Service, next steps
>
> Thanks to everyone who gave great feedback on the first queu
openstack@lists.launchpad.net
Subject: Re: [Openstack] Queue Service, next steps
I checked out the CDMI spec, and while it covers the basics, it doesn't
seem to include features for visibility, concurrency control (multiple
workers grabbing messages via atomic operations), or long-polling. We
can stil
On Fri, Feb 18, 2011 at 11:20:23AM -0600, Monsyne Dragon wrote:
> Secondly, could we leverage existing opensource codebases to
> accomplish this? I know AMQP was having issues over WAN links, and
> likely isn't suited for disconnected operations (I've seen one
> multitenant hosted queueing servic
MI REST interface, if all think this
>is a good idea
> * And on the topic of Erlang, it is viable. I have done some work with
>it. If we also use WebMachine, REST interface would be easier.
>Cheers
>
>
> ---- Original Message
>
Hi Kurt,
On Fri, Feb 18, 2011 at 12:11:28PM -0500, Kurt Griffiths wrote:
> Personally, I've been interested in learning Erlang as well. Eric, could you
> explain more about why you anticipate that the queuing service will
> CPU-bound? In either case, I've heard that Erlang can actually make bett
uld be researched
to provide a scalable back-end for the queuing system.
Maybe I don't quite understand which problem Scale Stack is solving? HTTP API +
-Original Message-
From: "Eric Day"
Sent: Thursday, February 17, 2011 5:21pm
To: openstack@lists.launchpad.net
Subjec
On 2/17/11 4:21 PM, Eric Day wrote:
Thanks to everyone who gave great feedback on the first queue service
thread. I've updated the wiki page to include the suggestions.
http://wiki.openstack.org/QueueService
Pondering this, I have a few thoughts:
I'll note that Erlang seems a realistic conten
Personally, I've been interested in learning Erlang as well. Eric, could you
explain more about why you anticipate that the queuing service will CPU-bound?
In either case, I've heard that Erlang can actually make better use of
multi-procs than C++, but I don't have any experience to speak from.
easier.Cheers
Original Message
Subject: Re: [Openstack] Queue Service, next steps
From: Daniel Salinas <imsplit...@gmail.com>
Date: Fri, February 18, 2011 8:20 am
To: "openstack@lists.launchpad.net" <openstack@lists.launchpad.net>
I would totally be i
Personally, I'd prefer C++ since that's what I'm used to, but I'd be
open to learning Erlang, too. Been wanting to learn it for a while
now.
-jay
On Fri, Feb 18, 2011 at 1:11 AM, Eric Day wrote:
> Duck farm? :)
>
> Are you two concerned about building a developer community around a
> project in
I would totally be interested in an Erlang bootstrap. In SAT. Lets see if
we can get a group together.
On 2/18/11 4:03 AM, "Curtis Carter" wrote:
>I'm in if it's done in Erlang.
>I am willing to give talks or help anyone wanting to learn Erlang in the
>Austin/San Antonio area as well.
>
>On Feb
I'm in if it's done in Erlang.
I am willing to give talks or help anyone wanting to learn Erlang in the
Austin/San Antonio area as well.
On Feb 18, 2011, at 12:11 AM, Eric Day wrote:
> Duck farm? :)
>
> Are you two concerned about building a developer community around a
> project in Erlang? I'
Duck farm? :)
Are you two concerned about building a developer community around a
project in Erlang? I'm all for going that route if other folks are
comfortable with it.
I also have some concern around the speed of Erlang. It's great,
especially if you know what primitives can be expensive and ho
I'll put in a +1 for Erlang as an OpenStack supported platform. We'd be able
to write a stable queue with a much more concise code base, and this is project
would be a great fit for Erlang.
Devin
On Feb 17, 2011, at 2:21 PM, Eric Day wrote:
> Thanks to everyone who gave great feedback on the
Thanks to everyone who gave great feedback on the first queue service
thread. I've updated the wiki page to include the suggestions.
http://wiki.openstack.org/QueueService
With a decent vision of what we want to build, the next step is
figuring out how. In a previous thread it was suggested that
On Tue, Feb 15, 2011 at 01:18:46PM -0800, David Strauss wrote:
> Regardless of whether the system uses AMQP, I am a fan of AMQP's
> flexible routing model. Please don't build a service that lacks fanout
> and other non-1:1 distribution patterns. Non-1:1 routing is useful for
> web cache invalidatio
On Mon, 2011-02-14 at 09:51 -0800, Eric Day wrote:
> There are possible solutions to build an AMQP based service,
> but AMQP brings complexity and a protocol not optimized for high
> latency and intermittent connectivity.
Regardless of whether the system uses AMQP, I am a fan of AMQP's
flexible ro
Sounds like unanimous agreement that account should always be part
of the URI. I've also added the version as the first path element to
match other OpenStack REST APIs. I updated the wiki to include this.
Thanks!
-Eric
On Tue, Feb 15, 2011 at 09:47:23AM -0800, Adrian Otto wrote:
> I think we shou
Hi Muriel,
On Tue, Feb 15, 2011 at 10:04:20AM +0100, Muriel wrote:
> i'm completely new to openstack but i started working on something
> similar to eucalyptus. In the coming months we will begin a project
> to bring our work on openstack and share it with you.
Cool, looking forward to it!
> The
I think we should include the account string regardless if the auth headers are
present, just like we do in the swift API.
You can still cache entities without the account string in the url, but it
means that you need to add the account string (or some unique derivative) from
the auth headers i
Having the in the URL makes the resource cacheable (via HTTP, I
mean; you can always manually cache it, but that's not RESTful); without
it, you would need to require unique s to uniquely reference a
cached copy.
On 2/14/11 7:57 PM, "Paul Voccio" wrote:
>Looking at the swift docs, they referenc
Il 14/02/2011 18:51, Eric Day ha scritto:
Hi everyone,
When looking at other services to include as part of OpenStack, the
first that comes to mind for many is a queue. A queue service can
not only provide a useful public cloud service, but can also provide
one of the building blocks for other s
On Mon, Feb 14, 2011 at 7:57 PM, Paul Voccio wrote:
> Looking at the swift docs, they reference a container like so:
>
> METHOD /v1// HTTP/1.1
Yeah, this has worked out well for us. Delegated access,
authentication methods that don't provide the account name otherwise,
anonymous access...
And
Looking at the swift docs, they reference a container like so:
METHOD /v1// HTTP/1.1
http://docs.rackspacecloud.com/files/api/v1/cf-devguide-20110112.pdf
For the openstack api, it also includes the account id in the request:
POST /v1.1/214412/images HTTP/1.1
Host: servers.api.openstack.org
Con
On Tue, Feb 15, 2011 at 12:49:01AM +, Paul Voccio wrote:
> Dropping the account_id, would this also assume that there can be more
> than one queue per account?
Yeah. Think of the queue name as an extra namespace layer much like
a swift container, except you don't create or delete them, they ju
Maybe not. I thought more on the way home.
Dropping the account_id, would this also assume that there can be more
than one queue per account?
On 2/14/11 5:54 PM, "Eric Day" wrote:
>Yeah, for anonymous access that would be the case. Those are not needed
>when the request comes in with OpenStack
Yeah, for anonymous access that would be the case. Those are not needed
when the request comes in with OpenStack Auth headers (like nova).
Do you think we should be repeating the account id in the URI when
the auth headers are present?
-Eric
On Mon, Feb 14, 2011 at 11:44:58PM +, Paul Voccio
Eric,
Just looking at the http operations. Shouldn't the calls be around the
account then the queue?
GET /$account_id/queue/id to list all the queues
GET /$account_id/queue/id/message/id
Am I off here? Thoughts?
pvo
On 2/14/11 5:07 PM, "Eric Day" wrote:
>I've summarized the original emai
I've summarized the original email and added more sections for review
and discussion here:
http://wiki.openstack.org/QueueService
In particular there are details on the various components of the
queue service, a draft at what the REST operations will look like,
and a couple brief examples.
Pleas
Hi Adrian,
Yes, you've covered some of what my soon-to-send "API implementation"
email already covers. :)
On Mon, Feb 14, 2011 at 06:32:40PM +, Adrian Otto wrote:
> Ordered delivery of messages is a feature that I recognize is central to the
> design of a queue service. A queue must be durab
Hi Pete,
Two initial use cases are a comparable service to what other public
cloud providers already have (similar HTTP/REST API, but eventually
with more features) and a flexible, high-performance queue/event system
for backend log and event processing for other OpenStack services (see
the messag
Pete,
Queues are essential for highly concurrent use cases, and batch processing.
It's the core of what you need to build a cloud-specific application. Reference:
http://www.eecs.harvard.edu/~mdw/proj/seda/
Adrian
On Feb 14, 2011, at 10:20 AM, Pete Zaitcev wrote:
> On Mon, 14 Feb 2011 09:51:4
Eric,
Ordered delivery of messages is a feature that I recognize is central to the
design of a queue service. A queue must be durable (persistent) in order to
provide reliable ordering capability. Having this feature simplifies
implementation of applications that consume the queue. I agree this
On Mon, 14 Feb 2011 09:51:42 -0800
Eric Day wrote:
> [] A queue service can
> not only provide a useful public cloud service, but can also provide
> one of the building blocks for other services.
> I'm looking forward to your feedback. Thanks!
Do you have a specific application and/or testbed i
Hi everyone,
When looking at other services to include as part of OpenStack, the
first that comes to mind for many is a queue. A queue service can
not only provide a useful public cloud service, but can also provide
one of the building blocks for other services. I've been leading an
effort to rese
82 matches
Mail list logo
