Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-16 Thread Pieter Hintjens
On Thu, Oct 16, 2014 at 4:04 PM, Holger Joukl wrote: > Ok. Still more or less the same to me: A receiver subscribes to a stream > or a subset of the stream. A publisher produces one stream; a receiver can get N streams. > I guess it then depends on what you'd describe as > a pub-sub "channel",

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-16 Thread Holger Joukl
Pieter, thanks for your clarifications. Some follow-up coments/questions: > Von: Pieter Hintjens > > A stream is a convenient top-level isolation. Typically one stream > serves one application. Within streams, readers choose subjects they > are interested in, using pattern matching. Ok. Still mo

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-16 Thread Pieter Hintjens
On Thu, Oct 16, 2014 at 1:49 PM, Holger Joukl wrote: > - I don't quite follow the distinction between live stream data > and publish-subscribe. IIUC pub-sub is described as a "segmentation" of > the live stream, i.e. when consuming a live stream I'd receive all > data on a certain "channel" (for

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-16 Thread Holger Joukl
Hi, zeromq-dev-boun...@lists.zeromq.org schrieb am 13.10.2014 15:36:38: > Von: Pieter Hintjens > > Mohit, > > I've started on a new broker project, > https://github.com/Malamute/malamute, which aims to do what you're > asking for. > > If you would like to read the description and tell me how clos

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-13 Thread Pieter Hintjens
That depends on how packaged you need the code. Making the full persistence part will take a while. The idea is to build the product gradually, driven by real use and by people paying for specific features. It already does basic transient pub-sub, with a client stack in C. On Mon, Oct 13, 2014 at

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-13 Thread Mohit Anchlia
Thanks Pieter, This is what I am looking for. Do you know when it would be ready for use? On Mon, Oct 13, 2014 at 6:36 AM, Pieter Hintjens wrote: > Mohit, > > I've started on a new broker project, > https://github.com/Malamute/malamute, which aims to do what you're > asking for. > > If you would

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-13 Thread Pieter Hintjens
Mohit, I've started on a new broker project, https://github.com/Malamute/malamute, which aims to do what you're asking for. If you would like to read the description and tell me how closely it matches, that would be helpful. -Pieter On Wed, Oct 1, 2014 at 9:19 PM, Trevor Bernard wrote: >> Than

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Trevor Bernard
> Thanks for the pointer. Could you please describe high level architecture of > how you are using it with zeromq? At a high level, my services flowed data in a pipeline. So for each process, there would be an upstream and downstream zeromq socket each running on their own thread. I choose push/pu

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Trevor Bernard
> Since you have experience with this library can you share your experience > with using openhft ? Some of the pros and cons of using this library? Pros: are it's high performance, generates almost no garbage and fairly easy to use. Cons: when I used it, it didn't have rolling logs quite right an

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Gregg Irwin
Hi Mohit, MA> Are there any standard projects that users of ZeroMQ use today MA> that provides persistence functionality as well? I am looking for MA> a scalable ZeroMQ persistence layer that is able to provide MA> pub/sub and is tunable to be non-persistent for async request/response. https://gi

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Mohit Anchlia
On Wed, Oct 1, 2014 at 10:04 AM, Trevor Bernard wrote: > I've had a lot of success persisting to an append only memory mapped > file. A throughput of 300k/s 1kb size messages shouldn't be a problem. > If you are on Java, I would suggest checking out the Chronicle queue > from OpenHFT. I've used t

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Erick Pagarigan Pagarigan
Aw On Oct 2, 2014 1:12 AM, "Mohit Anchlia" wrote: > Thanks for the pointer. Could you please describe high level architecture > of how you are using it with zeromq? > On Wed, Oct 1, 2014 at 10:04 AM, Trevor Bernard > wrote: > >> I've had a lot of success persisting to an append only memory mappe

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Mohit Anchlia
Thanks for the pointer. Could you please describe high level architecture of how you are using it with zeromq? On Wed, Oct 1, 2014 at 10:04 AM, Trevor Bernard wrote: > I've had a lot of success persisting to an append only memory mapped > file. A throughput of 300k/s 1kb size messages shouldn't b

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Trevor Bernard
I've had a lot of success persisting to an append only memory mapped file. A throughput of 300k/s 1kb size messages shouldn't be a problem. If you are on Java, I would suggest checking out the Chronicle queue from OpenHFT. I've used this in production in conjunction with ZeroMQ with fantastic resul

Re: [zeromq-dev] Persistence with ZeroMQ

2014-10-01 Thread Mohit Anchlia
Our average message size is < 1k and we are looking at 300K per second. If the solution is scalable then does it really matter? In other words by adding more machines we should be able to scale brokers and workers. On Tue, Sep 30, 2014 at 10:55 PM, Pieter Hintjens wrote: > What kinds of throughpu

Re: [zeromq-dev] Persistence with ZeroMQ

2014-09-30 Thread Pieter Hintjens
What kinds of throughputs and persistence are you looking for? On Wed, Oct 1, 2014 at 12:44 AM, Mohit Anchlia wrote: > Are there any standard projects that users of ZeroMQ use today that provides > persistence functionality as well? I am looking for a scalable ZeroMQ > persistence layer that is a

[zeromq-dev] Persistence with ZeroMQ

2014-09-30 Thread Mohit Anchlia
Are there any standard projects that users of ZeroMQ use today that provides persistence functionality as well? I am looking for a scalable ZeroMQ persistence layer that is able to provide pub/sub and is tunable to be non-persistent for async request/response. __