Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-16 Thread Pieter Hintjens
On Thu, Oct 16, 2014 at 4:04 PM, Holger Joukl  wrote:

> Ok. Still more or less the same to me: A receiver subscribes to a stream
> or a subset of the stream.

A publisher produces one stream; a receiver can get N streams.

> I guess it then depends on what you'd describe as
> a pub-sub "channel",

That is the stream, yes.

> So technically "streams" would then be discriminated by ports or you'd need
> to have some kind or root topic/subject.

Client specifies them at connection time (see the protocol wireframe)

https://github.com/hintjens/malamute-core/blob/master/src/mlm_msg.bnf

> I think my wording was a bit vague here: Suppose a topology of one malamute
> broker per host and each client connecting to its local broker. Sender
> and receivers are all on different hosts.

> Will the sender's broker or the (multiple) receivers' broker(s) store
> the messages?

I assume we'll store the messages redundantly in that case, across all
brokers that share a stream. I think also the unit of sharing will be
an entire stream, not part of it, so that readers can re-scan streams
later.

> Isn't it entirely possibly that the sending client works just fine but
> receiving and/or storing the data fails?

In the broker? That would be signaled as a critical system failure,
there is nothing the sender can do to recover or buffer (imagine you
are reading NASDAQ).

> I think e.g. Apache Kafka uses an optional acknowledgement for producers.

Perhaps, I'm not against it as such, it's just not a failure I've seen
in practice. I am happy to be proven wrong.

> I take it that this would then be functionality in a malamute client
> library that reveivers link/import/include.

Yes.

>> As an optional output socket for any given stream. Configurable, I guess.
>
> I still don't get it. Would the clients listen to a multicast address
> where a broker sends out data? Or would the multicast communication be
> inter-brokers, the clients then getting the data from their brokers
> over IPC or TCP?

The first, I assume. Honestly, we're too early for this detailed
question/answer. I suggest you try the software, which already works
in basic form, and ask questions based on that rather than theoretical
reading of the whitepaper.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-16 Thread Holger Joukl
Pieter,

thanks for your clarifications. Some follow-up coments/questions:

> Von: Pieter Hintjens
>
> A stream is a convenient top-level isolation. Typically one stream
> serves one application. Within streams, readers choose subjects they
> are interested in, using pattern matching.

Ok. Still more or less the same to me: A receiver subscribes to a stream
or a subset of the stream.

I quite like the analogon of tuning in to a radio frequency for
connection-less
broadcast/multicast pub-sub. I guess it then depends on what you'd describe
as
a pub-sub "channel", i.e. (protocol, port) or (protocol, port,
topic/subject).

So technically "streams" would then be discriminated by ports or you'd need
to have some kind or root topic/subject.

> > Would each malamute broker record the data all its connected consumer
> > clients are interested in or would the publisher's malamute broker
> record the data?
>
> Yes, the broker. Writers do not see readers.

I think my wording was a bit vague here: Suppose a topology of one malamute
broker per host and each client connecting to its local broker. Sender
and receivers are all on different hosts.

Will the sender's broker or the (multiple) receivers' broker(s) store
the messages?

> > How will be made sure that every published message will indeed get
> > persisted?
>
> There's no guarantee of this. We assume (hah!) that the broker is
> reliable, and that the failures we expect and have to solve are in
> clients. In practice we may use redundant brokers, for instance. To be
> seen.

Isn't it entirely possibly that the sending client works just fine but
receiving
and/or storing the data fails? Depends in part on how the client
communicates
with its broker I suppose but still sounds like some kind of acknowledge
would be necessary. I don't think redundant brokers would help here.

I think e.g. Apache Kafka uses an optional acknowledgement for producers.

> > When a consumer demands delivery of missed (by him) or historic
messages,
> > is the consumer responsible for its own delivery state and for sorting
out
> > duplicates?
>
> Yes. This would be hidden in the client stack.

I take it that this would then be functionality in a malamute client
library
that reveivers link/import/include.

> > - Where will the (PGM or NORM) multicast communication come into play?
>
> As an optional output socket for any given stream. Configurable, I guess.

I still don't get it. Would the clients listen to a multicast address
where a broker sends out data? Or would the multicast communication be
inter-brokers, the clients then getting the data from their brokers
over IPC or TCP?

Holger

Landesbank Baden-Wuerttemberg
Anstalt des oeffentlichen Rechts
Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz
HRA 12704
Amtsgericht Stuttgart

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-16 Thread Pieter Hintjens
On Thu, Oct 16, 2014 at 1:49 PM, Holger Joukl  wrote:

> - I don't quite follow the distinction between live stream data
> and publish-subscribe. IIUC pub-sub is described as a "segmentation" of
> the live stream, i.e. when consuming a live stream I'd receive all
> data on a certain "channel" (for lack of better words) but when
> subscribing to a publication I'd only receive messages belonging
> to/carrying a certain topic/subject.

A stream is a convenient top-level isolation. Typically one stream
serves one application. Within streams, readers choose subjects they
are interested in, using pattern matching.

> - Is the concept of on-demand streamed data persistence meant to be/do
> s.th. along the lines of Apache Kafka (or https://github.com/miniway/zper), 
> i.e.
> a per-topic/subject recorded journal of messages with a certain expiry
> period?

Not sure. It's meant to solve the late joiner and reconnection issues,
so a reader can switch between historical and live data invisibly
(except perhaps for timestamps in the message).

> Would each malamute broker record the data all its connected consumer
> clients are interested in or would the publisher's malamute broker record the 
> data?

Yes, the broker. Writers do not see readers.

> How will be made sure that every published message will indeed get
> persisted?

There's no guarantee of this. We assume (hah!) that the broker is
reliable, and that the failures we expect and have to solve are in
clients. In practice we may use redundant brokers, for instance. To be
seen.

> When a consumer demands delivery of missed (by him) or historic messages,
> is the consumer responsible for its own delivery state and for sorting out
> duplicates?

Yes. This would be hidden in the client stack.

> - Where will the (PGM or NORM) multicast communication come into play?

As an optional output socket for any given stream. Configurable, I guess.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-16 Thread Holger Joukl
Hi,

zeromq-dev-boun...@lists.zeromq.org schrieb am 13.10.2014 15:36:38:

> Von: Pieter Hintjens
>
> Mohit,
>
> I've started on a new broker project,
> https://github.com/Malamute/malamute, which aims to do what you're
> asking for.
>
> If you would like to read the description and tell me how closely it
> matches, that would be helpful.

I've some questions wrt the interesting malamute whitepaper:

- I don't quite follow the distinction between live stream data
and publish-subscribe. IIUC pub-sub is described as a "segmentation" of
the live stream, i.e. when consuming a live stream I'd receive all
data on a certain "channel" (for lack of better words) but when
subscribing to a publication I'd only receive messages belonging
to/carrying
a certain topic/subject.

The channel probably being an address, i.e. basically a network protocol
+ port combo.

But isn't receiving live stream data just pub-sub as well, basically using
a wildcard topic/subject?

- Is the concept of on-demand streamed data persistence meant to be/do
s.th.
along the lines of Apache Kafka (or https://github.com/miniway/zper), i.e.
a
per-topic/subject recorded journal of messages with a certain expiry
period?

Would each malamute broker record the data all its connected consumer
clients
are interested in or would the publisher's malamute broker record the data?

How will be made sure that every published message will indeed get
persisted?

When a consumer demands delivery of missed (by him) or historic messages,
is the
consumer responsible for its own delivery state and for sorting out
duplicates?

- Where will the (PGM or NORM) multicast communication come into play?
I assume
 * (sending) client connects to a malamute broker (IPC to a local or TCP to
a remote broker)
 * (sending) client publishes messages (sends data to a stream), or rather
hands
   the messages to its broker
 * sender's broker distributes the messages to other malamute brokers
 * those deliver the messages to their connected consumer clients.

The whitepaper states that in a distributed topology with one malamute
broker
per host those will communicate via TCP.

Best regards,
Holger

Landesbank Baden-Wuerttemberg
Anstalt des oeffentlichen Rechts
Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz
HRA 12704
Amtsgericht Stuttgart

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-13 Thread Pieter Hintjens
That depends on how packaged you need the code. Making the full
persistence part will take a while. The idea is to build the product
gradually, driven by real use and by people paying for specific
features. It already does basic transient pub-sub, with a client stack
in C.

On Mon, Oct 13, 2014 at 8:59 PM, Mohit Anchlia  wrote:
> Thanks Pieter, This is what I am looking for. Do you know when it would be
> ready for use?
>
> On Mon, Oct 13, 2014 at 6:36 AM, Pieter Hintjens  wrote:
>>
>> Mohit,
>>
>> I've started on a new broker project,
>> https://github.com/Malamute/malamute, which aims to do what you're
>> asking for.
>>
>> If you would like to read the description and tell me how closely it
>> matches, that would be helpful.
>>
>> -Pieter
>>
>> On Wed, Oct 1, 2014 at 9:19 PM, Trevor Bernard 
>> wrote:
>> >> Thanks for the pointer. Could you please describe high level
>> >> architecture of
>> >> how you are using it with zeromq?
>> >
>> > At a high level, my services flowed data in a pipeline. So for each
>> > process, there would be an upstream and downstream zeromq socket each
>> > running on their own thread. I choose push/pull but it could very
>> > easily work with pub/sub and use xpub,xsub to extend it.
>> >
>> > So on the upstream side, I would loop forever reading messages from
>> > the socket and publishing onto a disruptor.
>> >
>> > The first disruptor handler would serialize the message onto disk via
>> > Chronicle queue. The second handler would do the business logic,
>> > whatever that might be.. Then after it was done processing, it would
>> > publish on the downstream socket to whoever was listening.
>> >
>> > This worked pretty well and could achieve very high throughput with low
>> > latency.
>> >
>> > -Trev
>> > ___
>> > zeromq-dev mailing list
>> > zeromq-dev@lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-13 Thread Mohit Anchlia
Thanks Pieter, This is what I am looking for. Do you know when it would be
ready for use?
On Mon, Oct 13, 2014 at 6:36 AM, Pieter Hintjens  wrote:

> Mohit,
>
> I've started on a new broker project,
> https://github.com/Malamute/malamute, which aims to do what you're
> asking for.
>
> If you would like to read the description and tell me how closely it
> matches, that would be helpful.
>
> -Pieter
>
> On Wed, Oct 1, 2014 at 9:19 PM, Trevor Bernard 
> wrote:
> >> Thanks for the pointer. Could you please describe high level
> architecture of
> >> how you are using it with zeromq?
> >
> > At a high level, my services flowed data in a pipeline. So for each
> > process, there would be an upstream and downstream zeromq socket each
> > running on their own thread. I choose push/pull but it could very
> > easily work with pub/sub and use xpub,xsub to extend it.
> >
> > So on the upstream side, I would loop forever reading messages from
> > the socket and publishing onto a disruptor.
> >
> > The first disruptor handler would serialize the message onto disk via
> > Chronicle queue. The second handler would do the business logic,
> > whatever that might be.. Then after it was done processing, it would
> > publish on the downstream socket to whoever was listening.
> >
> > This worked pretty well and could achieve very high throughput with low
> latency.
> >
> > -Trev
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-13 Thread Pieter Hintjens
Mohit,

I've started on a new broker project,
https://github.com/Malamute/malamute, which aims to do what you're
asking for.

If you would like to read the description and tell me how closely it
matches, that would be helpful.

-Pieter

On Wed, Oct 1, 2014 at 9:19 PM, Trevor Bernard  wrote:
>> Thanks for the pointer. Could you please describe high level architecture of
>> how you are using it with zeromq?
>
> At a high level, my services flowed data in a pipeline. So for each
> process, there would be an upstream and downstream zeromq socket each
> running on their own thread. I choose push/pull but it could very
> easily work with pub/sub and use xpub,xsub to extend it.
>
> So on the upstream side, I would loop forever reading messages from
> the socket and publishing onto a disruptor.
>
> The first disruptor handler would serialize the message onto disk via
> Chronicle queue. The second handler would do the business logic,
> whatever that might be.. Then after it was done processing, it would
> publish on the downstream socket to whoever was listening.
>
> This worked pretty well and could achieve very high throughput with low 
> latency.
>
> -Trev
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Trevor Bernard
> Thanks for the pointer. Could you please describe high level architecture of
> how you are using it with zeromq?

At a high level, my services flowed data in a pipeline. So for each
process, there would be an upstream and downstream zeromq socket each
running on their own thread. I choose push/pull but it could very
easily work with pub/sub and use xpub,xsub to extend it.

So on the upstream side, I would loop forever reading messages from
the socket and publishing onto a disruptor.

The first disruptor handler would serialize the message onto disk via
Chronicle queue. The second handler would do the business logic,
whatever that might be.. Then after it was done processing, it would
publish on the downstream socket to whoever was listening.

This worked pretty well and could achieve very high throughput with low latency.

-Trev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Trevor Bernard
> Since you have experience with this library can you share your experience
> with using openhft ? Some of the pros and cons of using this library?

Pros: are it's high performance, generates almost no garbage and
fairly easy to use.

Cons: when I used it, it didn't have rolling logs quite right and I
had to devise my own compaction strategy otherwise t would infinitely
append to the log until you ran out of disk space. It since has
matured and the cons might not be relevant anymore.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Gregg Irwin
Hi Mohit,

MA> Are there any standard projects that users of ZeroMQ use today
MA> that provides persistence functionality as well? I am looking for
MA> a scalable ZeroMQ persistence layer that is able to provide
MA> pub/sub and is tunable to be non-persistent for async request/response.

https://github.com/miniway/zper was mentioned on the list recently.

-- Gregg

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Mohit Anchlia
On Wed, Oct 1, 2014 at 10:04 AM, Trevor Bernard 
wrote:

> I've had a lot of success persisting to an append only memory mapped
> file. A throughput of 300k/s 1kb size messages shouldn't be a problem.
> If you are on Java, I would suggest checking out the Chronicle queue
> from OpenHFT. I've used this in production in conjunction with ZeroMQ
> with fantastic results.
>
> http://openhft.net/products/chronicle-queue/
>
>
It looks like openhft is a persistence framework that seems like a log
based persistence. So I guess one could persist it using this library over
zeromq's pub/sub pattern.

Since you have experience with this library can you share your experience
with using openhft ? Some of the pros and cons of using this library?


> -Trev
>
> On Wed, Oct 1, 2014 at 1:53 PM, Mohit Anchlia 
> wrote:
> > Our average message size is < 1k and we are looking at 300K per second.
> If
> > the solution is scalable then does it really matter? In other words by
> > adding more machines we should be able to scale brokers and workers.
> >
> > On Tue, Sep 30, 2014 at 10:55 PM, Pieter Hintjens  wrote:
> >>
> >> What kinds of throughputs and persistence are you looking for?
> >>
> >> On Wed, Oct 1, 2014 at 12:44 AM, Mohit Anchlia 
> >> wrote:
> >> > Are there any standard projects that users of ZeroMQ use today that
> >> > provides
> >> > persistence functionality as well? I am looking for a scalable ZeroMQ
> >> > persistence layer that is able to provide pub/sub and is tunable to be
> >> > non-persistent for async request/response.
> >> > ___
> >> > zeromq-dev mailing list
> >> > zeromq-dev@lists.zeromq.org
> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >> >
> >> ___
> >> zeromq-dev mailing list
> >> zeromq-dev@lists.zeromq.org
> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
> >
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Erick Pagarigan Pagarigan
Aw
On Oct 2, 2014 1:12 AM, "Mohit Anchlia"  wrote:

> Thanks for the pointer. Could you please describe high level architecture
> of how you are using it with zeromq?
> On Wed, Oct 1, 2014 at 10:04 AM, Trevor Bernard 
> wrote:
>
>> I've had a lot of success persisting to an append only memory mapped
>> file. A throughput of 300k/s 1kb size messages shouldn't be a problem.
>> If you are on Java, I would suggest checking out the Chronicle queue
>> from OpenHFT. I've used this in production in conjunction with ZeroMQ
>> with fantastic results.
>>
>> http://openhft.net/products/chronicle-queue/
>>
>> -Trev
>>
>> On Wed, Oct 1, 2014 at 1:53 PM, Mohit Anchlia 
>> wrote:
>> > Our average message size is < 1k and we are looking at 300K per second.
>> If
>> > the solution is scalable then does it really matter? In other words by
>> > adding more machines we should be able to scale brokers and workers.
>> >
>> > On Tue, Sep 30, 2014 at 10:55 PM, Pieter Hintjens 
>> wrote:
>> >>
>> >> What kinds of throughputs and persistence are you looking for?
>> >>
>> >> On Wed, Oct 1, 2014 at 12:44 AM, Mohit Anchlia > >
>> >> wrote:
>> >> > Are there any standard projects that users of ZeroMQ use today that
>> >> > provides
>> >> > persistence functionality as well? I am looking for a scalable ZeroMQ
>> >> > persistence layer that is able to provide pub/sub and is tunable to
>> be
>> >> > non-persistent for async request/response.
>> >> > ___
>> >> > zeromq-dev mailing list
>> >> > zeromq-dev@lists.zeromq.org
>> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >> >
>> >> ___
>> >> zeromq-dev mailing list
>> >> zeromq-dev@lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> >
>> >
>> > ___
>> > zeromq-dev mailing list
>> > zeromq-dev@lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Mohit Anchlia
Thanks for the pointer. Could you please describe high level architecture
of how you are using it with zeromq?
On Wed, Oct 1, 2014 at 10:04 AM, Trevor Bernard 
wrote:

> I've had a lot of success persisting to an append only memory mapped
> file. A throughput of 300k/s 1kb size messages shouldn't be a problem.
> If you are on Java, I would suggest checking out the Chronicle queue
> from OpenHFT. I've used this in production in conjunction with ZeroMQ
> with fantastic results.
>
> http://openhft.net/products/chronicle-queue/
>
> -Trev
>
> On Wed, Oct 1, 2014 at 1:53 PM, Mohit Anchlia 
> wrote:
> > Our average message size is < 1k and we are looking at 300K per second.
> If
> > the solution is scalable then does it really matter? In other words by
> > adding more machines we should be able to scale brokers and workers.
> >
> > On Tue, Sep 30, 2014 at 10:55 PM, Pieter Hintjens  wrote:
> >>
> >> What kinds of throughputs and persistence are you looking for?
> >>
> >> On Wed, Oct 1, 2014 at 12:44 AM, Mohit Anchlia 
> >> wrote:
> >> > Are there any standard projects that users of ZeroMQ use today that
> >> > provides
> >> > persistence functionality as well? I am looking for a scalable ZeroMQ
> >> > persistence layer that is able to provide pub/sub and is tunable to be
> >> > non-persistent for async request/response.
> >> > ___
> >> > zeromq-dev mailing list
> >> > zeromq-dev@lists.zeromq.org
> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >> >
> >> ___
> >> zeromq-dev mailing list
> >> zeromq-dev@lists.zeromq.org
> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
> >
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Trevor Bernard
I've had a lot of success persisting to an append only memory mapped
file. A throughput of 300k/s 1kb size messages shouldn't be a problem.
If you are on Java, I would suggest checking out the Chronicle queue
from OpenHFT. I've used this in production in conjunction with ZeroMQ
with fantastic results.

http://openhft.net/products/chronicle-queue/

-Trev

On Wed, Oct 1, 2014 at 1:53 PM, Mohit Anchlia  wrote:
> Our average message size is < 1k and we are looking at 300K per second. If
> the solution is scalable then does it really matter? In other words by
> adding more machines we should be able to scale brokers and workers.
>
> On Tue, Sep 30, 2014 at 10:55 PM, Pieter Hintjens  wrote:
>>
>> What kinds of throughputs and persistence are you looking for?
>>
>> On Wed, Oct 1, 2014 at 12:44 AM, Mohit Anchlia 
>> wrote:
>> > Are there any standard projects that users of ZeroMQ use today that
>> > provides
>> > persistence functionality as well? I am looking for a scalable ZeroMQ
>> > persistence layer that is able to provide pub/sub and is tunable to be
>> > non-persistent for async request/response.
>> > ___
>> > zeromq-dev mailing list
>> > zeromq-dev@lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Mohit Anchlia
Our average message size is < 1k and we are looking at 300K per second. If
the solution is scalable then does it really matter? In other words by
adding more machines we should be able to scale brokers and workers.
On Tue, Sep 30, 2014 at 10:55 PM, Pieter Hintjens  wrote:

> What kinds of throughputs and persistence are you looking for?
>
> On Wed, Oct 1, 2014 at 12:44 AM, Mohit Anchlia 
> wrote:
> > Are there any standard projects that users of ZeroMQ use today that
> provides
> > persistence functionality as well? I am looking for a scalable ZeroMQ
> > persistence layer that is able to provide pub/sub and is tunable to be
> > non-persistent for async request/response.
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


Re: [zeromq-dev] Persistence with ZeroMQ

2014-09-30 Thread Pieter Hintjens
What kinds of throughputs and persistence are you looking for?

On Wed, Oct 1, 2014 at 12:44 AM, Mohit Anchlia  wrote:
> Are there any standard projects that users of ZeroMQ use today that provides
> persistence functionality as well? I am looking for a scalable ZeroMQ
> persistence layer that is able to provide pub/sub and is tunable to be
> non-persistent for async request/response.
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev