Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

Apologies in advance for possible repetition and pedantry... On 09/24/2014 02:48 AM, Devananda van der Veen wrote: 2. Single Delivery - each message must be processed *exactly* once Example: Using a queue to process votes. Every vote must be counted only once. It is also important to consi

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 23/09/14 19:29, Devananda van der Veen wrote: On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter wrote: On 22/09/14 17:06, Joe Gordon wrote: If 50,000 messages per second doesn't count as small-to-moderate then Zaqar does not fulfill a major SQS use case. It's not a drop-in replacement, but a

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 23/09/14 17:59, Joe Gordon wrote: >> Zaqar is aiming for low latency per message, SQS doesn't appear to be. >> > >I've seen no evidence that Zaqar is actually aiming for that. There are >waaay lower-latency ways to implement messaging if you don't care about >durability (you wouldn't do sto

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/24/2014 12:06 AM, Joe Gordon wrote: > > > On Tue, Sep 23, 2014 at 2:40 AM, Flavio Percoco > wrote: > > On 09/23/2014 05:13 AM, Clint Byrum wrote: > > Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: > > [snip] > > >> > >> To

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/24/2014 03:48 AM, Devananda van der Veen wrote: > I've taken a bit of time out of this thread, and I'd like to jump back > in now and attempt to summarize what I've learned and hopefully frame > it in such a way that it helps us to answer the question Thierry > asked: > I *loved* it! Thank

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/23/2014 11:59 PM, Joe Gordon wrote: > > > On Tue, Sep 23, 2014 at 9:13 AM, Zane Bitter > wrote: > > On 22/09/14 22:04, Joe Gordon wrote: > > To me this is less about valid or invalid choices. The Zaqar team is > comparing Zaqar to SQS, but a

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

I've taken a bit of time out of this thread, and I'd like to jump back in now and attempt to summarize what I've learned and hopefully frame it in such a way that it helps us to answer the question Thierry asked: On Fri, Sep 19, 2014 at 2:00 AM, Thierry Carrez wrote: > > The underlying question b

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter wrote: > On 22/09/14 17:06, Joe Gordon wrote: >> >> If 50,000 messages per second doesn't count as small-to-moderate then >> Zaqar >> does not fulfill a major SQS use case. > > > It's not a drop-in replacement, but as I mentioned you can recreate the SQ

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

Excerpts from Joe Gordon's message of 2014-09-23 14:59:33 -0700: > On Tue, Sep 23, 2014 at 9:13 AM, Zane Bitter wrote: > > > On 22/09/14 22:04, Joe Gordon wrote: > > > >> To me this is less about valid or invalid choices. The Zaqar team is > >> comparing Zaqar to SQS, but after digging into the t

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Tue, Sep 23, 2014 at 2:40 AM, Flavio Percoco wrote: > On 09/23/2014 05:13 AM, Clint Byrum wrote: > > Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: > > [snip] > > >> > >> To me this is less about valid or invalid choices. The Zaqar team is > >> comparing Zaqar to SQS, but aft

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Tue, Sep 23, 2014 at 9:13 AM, Zane Bitter wrote: > On 22/09/14 22:04, Joe Gordon wrote: > >> To me this is less about valid or invalid choices. The Zaqar team is >> comparing Zaqar to SQS, but after digging into the two of them, zaqar >> barely looks like SQS. Zaqar doesn't guarantee what IMHO

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

ges, it'll likely be towards the option (1). Thanks for your feedback, Flavio > > Thanks, > Kevin > > > ____________ > From: Flavio Percoco [fla...@redhat.com] > Sent: Tuesday, September 23, 2014 5:58 AM > To: openstack-dev@lists.openst

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

ay, September 23, 2014 5:58 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues On 09/23/2014 10:58 AM, Gordon Sim wrote: > On 09/22/2014 05:58 PM, Zane Bitter wrote: >> On 22/09/14 10:11, Gordon Sim wrote:

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 22/09/14 22:04, Joe Gordon wrote: To me this is less about valid or invalid choices. The Zaqar team is comparing Zaqar to SQS, but after digging into the two of them, zaqar barely looks like SQS. Zaqar doesn't guarantee what IMHO is the most important parts of SQS: the message will be delivere

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 23/09/14 08:58, Flavio Percoco wrote: I believe the guarantee is still useful and it currently does not represent an issue for the service nor the user. 2 things could happen to FIFO in the future: 1. It's made optional and we allow users to opt-in in a per flavor basis. (I personally don't l

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/23/2014 10:58 AM, Gordon Sim wrote: > On 09/22/2014 05:58 PM, Zane Bitter wrote: >> On 22/09/14 10:11, Gordon Sim wrote: >>> As I understand it, pools don't help scaling a given queue since all the >>> messages for that queue must be in the same pool. At present traffic >>> through different

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/23/2014 05:13 AM, Clint Byrum wrote: > Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: [snip] >> >> To me this is less about valid or invalid choices. The Zaqar team is >> comparing Zaqar to SQS, but after digging into the two of them, zaqar >> barely looks like SQS. Zaqar

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/22/2014 05:58 PM, Zane Bitter wrote: On 22/09/14 10:11, Gordon Sim wrote: As I understand it, pools don't help scaling a given queue since all the messages for that queue must be in the same pool. At present traffic through different Zaqar queues are essentially entirely orthogonal streams

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Mon, Sep 22, 2014 at 8:13 PM, Clint Byrum wrote: > Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: > > On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter wrote: > > > > > On 22/09/14 17:06, Joe Gordon wrote: > > > > > >> On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter > wrote: > > >>

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: > On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter wrote: > > > On 22/09/14 17:06, Joe Gordon wrote: > > > >> On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter wrote: > >> > >> On 22/09/14 10:11, Gordon Sim wrote: > >>> > >>> On 09/19/201

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Mon, Sep 22, 2014 at 7:04 PM, Joe Gordon wrote: > > > On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter wrote: > >> On 22/09/14 17:06, Joe Gordon wrote: >> >>> On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter wrote: >>> >>> On 22/09/14 10:11, Gordon Sim wrote: On 09/19/2014 09:13 PM, Zane

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter wrote: > On 22/09/14 17:06, Joe Gordon wrote: > >> On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter wrote: >> >> On 22/09/14 10:11, Gordon Sim wrote: >>> >>> On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very limited guarantees,

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 22/09/14 17:06, Joe Gordon wrote: On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter wrote: On 22/09/14 10:11, Gordon Sim wrote: On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very limited guarantees, and it's clear that the reason for that is to make it massively, massively scal

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter wrote: > On 22/09/14 10:11, Gordon Sim wrote: > >> On 09/19/2014 09:13 PM, Zane Bitter wrote: >> >>> SQS offers very, very limited guarantees, and it's clear that the reason >>> for that is to make it massively, massively scalable in the way that >>> e

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 22/09/14 10:11, Gordon Sim wrote: On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very limited guarantees, and it's clear that the reason for that is to make it massively, massively scalable in the way that e.g. S3 is scalable while also remaining comparably durable (S3 is suppose

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very limited guarantees, and it's clear that the reason for that is to make it massively, massively scalable in the way that e.g. S3 is scalable while also remaining comparably durable (S3 is supposedly designed for 11 nines, BTW). Zaqa

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 18/09/14 10:55, Flavio Percoco wrote: On 09/18/2014 04:24 PM, Clint Byrum wrote: Great job highlighting what our friends over at Amazon are doing. It's clear from these snippets, and a few other pieces of documentation for SQS I've read, that the Amazon team approached SQS from a _massive_ s

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Fri, Sep 19, 2014 at 4:23 AM, Eoghan Glynn wrote: > > > > Hi All, > > > > My understanding of Zaqar is that it's like SQS. SQS uses distributed > queues, > > which have a few unusual properties [0]: > > Message Order > > > > > > Amazon SQS makes a best effort to preserve order in messages, but

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/19/2014 08:53 AM, Flavio Percoco wrote: On 09/18/2014 09:25 PM, Gordon Sim wrote: I don't see that the claim mechanism brings any stronger guarantee, it just offers a competing consumer behaviour where browsing is non-competing (non-destructive). In both cases you require the client to be

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

Excerpts from Eoghan Glynn's message of 2014-09-19 04:23:55 -0700: > > > Hi All, > > > > My understanding of Zaqar is that it's like SQS. SQS uses distributed > > queues, > > which have a few unusual properties [0]: > > Message Order > > > > > > Amazon SQS makes a best effort to preserve order

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

> Hi All, > > My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, > which have a few unusual properties [0]: > Message Order > > > Amazon SQS makes a best effort to preserve order in messages, but due to the > distributed nature of the queue, we cannot guarantee you w

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/19/2014 11:00 AM, Thierry Carrez wrote: > Joe Gordon wrote: >> On Thu, Sep 18, 2014 at 9:02 AM, Devananda van der Veen >> mailto:devananda@gmail.com>> wrote: >>> - guaranteed message order >>> - not distributing work across a configurable number of back ends >>> >>> These are

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

Joe Gordon wrote: > On Thu, Sep 18, 2014 at 9:02 AM, Devananda van der Veen > mailto:devananda@gmail.com>> wrote: >> - guaranteed message order >> - not distributing work across a configurable number of back ends >> >> These are scale-limiting design choices which are reflected in

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/18/2014 07:16 PM, Devananda van der Veen wrote: > On Thu, Sep 18, 2014 at 8:54 AM, Devananda van der Veen > wrote: >> On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco wrote: >>> On 09/18/2014 04:09 PM, Gordon Sim wrote: On 09/18/2014 12:31 PM, Flavio Percoco wrote: > Zaqar guarantee

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/18/2014 07:19 PM, Joe Gordon wrote: > > > On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco > wrote: > > On 09/18/2014 04:09 PM, Gordon Sim wrote: > > On 09/18/2014 12:31 PM, Flavio Percoco wrote: > >> On 09/17/2014 10:36 PM, Joe Gordon wrote: > >>>

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/18/2014 09:25 PM, Gordon Sim wrote: > On 09/18/2014 03:45 PM, Flavio Percoco wrote: >> On 09/18/2014 04:09 PM, Gordon Sim wrote: >>> Is the replication synchronous or asynchronous with respect to client >>> calls? E.g. will the response to a post of messages be returned only >>> once the repl

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/18/2014 03:45 PM, Flavio Percoco wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: Is the replication synchronous or asynchronous with respect to client calls? E.g. will the response to a post of messages be returned only once the replication of those messages is confirmed? Likewise when de

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco wrote: > On 09/18/2014 04:09 PM, Gordon Sim wrote: > > On 09/18/2014 12:31 PM, Flavio Percoco wrote: > >> On 09/17/2014 10:36 PM, Joe Gordon wrote: > >>> My understanding of Zaqar is that it's like SQS. SQS uses distributed > >>> queues, which have

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Thu, Sep 18, 2014 at 8:54 AM, Devananda van der Veen wrote: > On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco wrote: >> On 09/18/2014 04:09 PM, Gordon Sim wrote: >>> On 09/18/2014 12:31 PM, Flavio Percoco wrote: Zaqar guarantees FIFO. To be more precise, it does that relying on the s

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Thu, Sep 18, 2014 at 9:02 AM, Devananda van der Veen < devananda@gmail.com> wrote: > On Thu, Sep 18, 2014 at 7:55 AM, Flavio Percoco wrote: > > On 09/18/2014 04:24 PM, Clint Byrum wrote: > >> Great job highlighting what our friends over at Amazon are doing. > >> > >> It's clear from these

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Thu, Sep 18, 2014 at 7:55 AM, Flavio Percoco wrote: > On 09/18/2014 04:24 PM, Clint Byrum wrote: >> Great job highlighting what our friends over at Amazon are doing. >> >> It's clear from these snippets, and a few other pieces of documentation >> for SQS I've read, that the Amazon team approach

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco wrote: > On 09/18/2014 04:09 PM, Gordon Sim wrote: >> On 09/18/2014 12:31 PM, Flavio Percoco wrote: >>> Zaqar guarantees FIFO. To be more precise, it does that relying on the >>> storage backend ability to do so as well. Depending on the storage used

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/18/2014 05:42 PM, Steve Lewis wrote: > On September 18, 2014 7:45 AM, Flavio Percoco wrote: > >> On 09/18/2014 04:09 PM, Gordon Sim wrote: >> >>> However, as far as consuming messages is concerned, it can >>> guarantee once-and-only-once and/or at-least-once delivery depending on >>> the mes

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On September 18, 2014 7:45 AM, Flavio Percoco wrote: > On 09/18/2014 04:09 PM, Gordon Sim wrote: > >> However, as far as consuming messages is concerned, it can >> guarantee once-and-only-once and/or at-least-once delivery depending on >> the message pattern used to consume messages. Using pop or

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/18/2014 04:24 PM, Clint Byrum wrote: > Great job highlighting what our friends over at Amazon are doing. > > It's clear from these snippets, and a few other pieces of documentation > for SQS I've read, that the Amazon team approached SQS from a _massive_ > scaling perspective. I think what m

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/18/2014 04:09 PM, Gordon Sim wrote: > On 09/18/2014 12:31 PM, Flavio Percoco wrote: >> On 09/17/2014 10:36 PM, Joe Gordon wrote: >>> My understanding of Zaqar is that it's like SQS. SQS uses distributed >>> queues, which have a few unusual properties [0]: >>> >>> >>> Message Order >>> >>

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

Great job highlighting what our friends over at Amazon are doing. It's clear from these snippets, and a few other pieces of documentation for SQS I've read, that the Amazon team approached SQS from a _massive_ scaling perspective. I think what may be forcing a lot of this frustration with Zaqar is

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/18/2014 12:31 PM, Flavio Percoco wrote: On 09/17/2014 10:36 PM, Joe Gordon wrote: My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which have a few unusual properties [0]: Message Order Amazon SQS makes a best effort to preserve order in messages, but d

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

On 09/17/2014 10:36 PM, Joe Gordon wrote: > Hi All, > > My understanding of Zaqar is that it's like SQS. SQS uses distributed > queues, which have a few unusual properties [0]: > > > Message Order > > Amazon SQS makes a best effort to preserve order in messages, but due to > the distributed

[openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

Hi All, My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which have a few unusual properties [0]: Message Order Amazon SQS makes a best effort to preserve order in messages, but due to the distributed nature of the queue, we cannot guarantee you will receive messages