Re: [zeromq-dev] Help with pattern design

[Alex Massover]

Hello,

Thank you very much Michel! Router-to-Dealer is exactly what I need.
I should be reading the guide more carefully.


On 2 באוג 2012, at 19:17, "Michel Pelletier"  wrote:

> You want to read up on the ROUTER socket.  It's the goto when you need
> a custom routing setup other than the built in load balanced
> functionality in PUSH and DEALER.  In your server, you would maintain
> a mapping of session ids to worker ids.  The LRU pattern in the guide
> defines a nice example for this kind of setup.
> 
> -Michel
> 
> On Thu, Aug 2, 2012 at 8:34 AM, Alex Massover  wrote:
>> Hello,
>> 
>> 
>> 
>> I'm trying to design a pattern that will fit my application but can't find a
>> proper solution. I'd appreciate if someone could enlighten me.
>> 
>> 
>> 
>> I have a process that deals with sessions (voip calls). This server produce
>> events for every session, e.g.:
>> 
>> Session1: start
>> 
>> Session1: msgA
>> 
>> Session1: msgB
>> 
>> Session1: msgC
>> 
>> …
>> 
>> Session1: msgX
>> 
>> Session1: stop
>> 
>> 
>> 
>> Each event has a session id. Many sessions exist concurrently.
>> 
>> 
>> 
>> And there're multiple workers that consume these events. Single event should
>> go to single worker. (PUSH/PULL like pattern).
>> 
>> 
>> 
>> My initial though was single PUSH (server that produce events) and multiple
>> PULL (workers that consume).
>> 
>> 
>> 
>> My challenge is avoid race condition in processing events, i.e. workers
>> supposed to consume events in the same order as they produced. And this
>> should be done in realtime, I can't serialize at worker side. And single
>> worker is not enough from performance point of view.
>> 
>> 
>> 
>> Is there any ZMQ magic (pattern) that can force all events from single
>> session (based on session id?) to be sent to same worker? This way I'll
>> ensure that worker process events in the same order they appeared.
>> 
>> 
>> 
>> --
>> 
>> Best Regards,
>> 
>> Alex Massover
>> 
>> 
>> 
>> 
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> 
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Designing a new architecture

[Michel Pelletier]

Specifically, check out cliff's cool interactive mode:

http://cliff.readthedocs.org/en/latest/interactive_mode.html

On Thu, Aug 2, 2012 at 11:07 AM, Michel Pelletier
 wrote:
> On Thu, Aug 2, 2012 at 9:50 AM, andrea crotti  
> wrote:
>> I have to restructure a complex application and I would like to do this
>> with zeromq, I'm reading the doc and trying out things but I'm a bit
>> lost now, so some hints might be very useful..
>>
>> The application I'm writing is very parallel, there are many
>> long-running processes that do things, managed by a central daemon.
>>
>> I would like to be able to communicate with these processes to check the
>> current status, and interact with it, and these processes should write
>> the results on a common sink.
>>
>> So it looks like the multiple worker with a sink might be the right
>> choice, but I also want to be able to communicate with every single
>> process separately.
>
> I would suggest looking at the ROUTER socket in detail and
> specifically the LRU pattern in the guide.  You may only need the
> "back-half" of the LRU pattern to do what you need.  This allows you
> to have bidirectional communication with a bunch of workers.
>
>> Another thing which I'd like to do is to use Python Cmd class, and I
>> thought I could create a zmq class that behaves like a file, and thus be
>> passed in the Cmd object.
>
> I would avoid stretching the file abstraction over a zmq socket like
> that.   Maybe another library that is more flexible for command line
> and simple interpreters like cliff
> (http://pypi.python.org/pypi/cliff/) would be easier for you to
> integrate with zmq.  Cmd after all, is pretty archaic and file
> oriented.
>
> -Michel
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Designing a new architecture

[Michel Pelletier]

On Thu, Aug 2, 2012 at 9:50 AM, andrea crotti  wrote:
> I have to restructure a complex application and I would like to do this
> with zeromq, I'm reading the doc and trying out things but I'm a bit
> lost now, so some hints might be very useful..
>
> The application I'm writing is very parallel, there are many
> long-running processes that do things, managed by a central daemon.
>
> I would like to be able to communicate with these processes to check the
> current status, and interact with it, and these processes should write
> the results on a common sink.
>
> So it looks like the multiple worker with a sink might be the right
> choice, but I also want to be able to communicate with every single
> process separately.

I would suggest looking at the ROUTER socket in detail and
specifically the LRU pattern in the guide.  You may only need the
"back-half" of the LRU pattern to do what you need.  This allows you
to have bidirectional communication with a bunch of workers.

> Another thing which I'd like to do is to use Python Cmd class, and I
> thought I could create a zmq class that behaves like a file, and thus be
> passed in the Cmd object.

I would avoid stretching the file abstraction over a zmq socket like
that.   Maybe another library that is more flexible for command line
and simple interpreters like cliff
(http://pypi.python.org/pypi/cliff/) would be easier for you to
integrate with zmq.  Cmd after all, is pretty archaic and file
oriented.

-Michel
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] Designing a new architecture

[andrea crotti]

I have to restructure a complex application and I would like to do this
with zeromq, I'm reading the doc and trying out things but I'm a bit
lost now, so some hints might be very useful..

The application I'm writing is very parallel, there are many
long-running processes that do things, managed by a central daemon.

I would like to be able to communicate with these processes to check the
current status, and interact with it, and these processes should write
the results on a common sink.

So it looks like the multiple worker with a sink might be the right
choice, but I also want to be able to communicate with every single
process separately.

Another thing which I'd like to do is to use Python Cmd class, and I
thought I could create a zmq class that behaves like a file, and thus be
passed in the Cmd object.

So I did the following, but apparently I just can't get it working,
because every time I receive I should acknoledge with a write and
vice-versa..
Do you think it's possible/makes sense to do something like this?

__metaclass__ = type

import zmq

ADDR = "tcp://127.0.0.1:5556"
context = zmq.Context()


class AsFile:
def __init__(self):
self.sock = None

def _read(self):
return self.sock.recv()

read = _read
readline =  _read

def write(self, msg):
self.sock.send(msg)


class AsFileServer(AsFile):
def __init__(self):
super(AsFileServer, self).__init__()
self.sock = context.socket(zmq.REP)
self.sock.bind(ADDR)


class AsFileClient(AsFile):
def __init__(self):
super(AsFileClient, self).__init__()
self.sock = context.socket(zmq.REQ)
self.sock.connect(ADDR)
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] [ANNOUNCE] zerogw mailing list

[Thomas S Hatch]

On Thu, Aug 2, 2012 at 10:25 AM, Allan Wind wrote:

> On 2012-08-03 01:16:19, Pieter Hintjens wrote:
> > Nice...! Where's the git repo?
>
> Google says: 
>
> This looks really cool, is it production ready? I might use it with Salt
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] [ANNOUNCE] zerogw mailing list

[Allan Wind]

On 2012-08-03 01:16:19, Pieter Hintjens wrote:
> Nice...! Where's the git repo?

Google says: 


/Allan
-- 
Allan Wind
Life Integrity, LLC

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Help with pattern design

[Michel Pelletier]

You want to read up on the ROUTER socket.  It's the goto when you need
a custom routing setup other than the built in load balanced
functionality in PUSH and DEALER.  In your server, you would maintain
a mapping of session ids to worker ids.  The LRU pattern in the guide
defines a nice example for this kind of setup.

-Michel

On Thu, Aug 2, 2012 at 8:34 AM, Alex Massover  wrote:
> Hello,
>
>
>
> I'm trying to design a pattern that will fit my application but can't find a
> proper solution. I'd appreciate if someone could enlighten me.
>
>
>
> I have a process that deals with sessions (voip calls). This server produce
> events for every session, e.g.:
>
> Session1: start
>
> Session1: msgA
>
> Session1: msgB
>
> Session1: msgC
>
> …
>
> Session1: msgX
>
> Session1: stop
>
>
>
> Each event has a session id. Many sessions exist concurrently.
>
>
>
> And there're multiple workers that consume these events. Single event should
> go to single worker. (PUSH/PULL like pattern).
>
>
>
> My initial though was single PUSH (server that produce events) and multiple
> PULL (workers that consume).
>
>
>
> My challenge is avoid race condition in processing events, i.e. workers
> supposed to consume events in the same order as they produced. And this
> should be done in realtime, I can't serialize at worker side. And single
> worker is not enough from performance point of view.
>
>
>
> Is there any ZMQ magic (pattern) that can force all events from single
> session (based on session id?) to be sent to same worker? This way I'll
> ensure that worker process events in the same order they appeared.
>
>
>
> --
>
> Best Regards,
>
> Alex Massover
>
>
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] [ANNOUNCE] zerogw mailing list

[Pieter Hintjens]

On Thu, Aug 2, 2012 at 3:51 AM, Paul Colomiets  wrote:
> Hi,
>
> Due to recently grown interest in zerogw, I've setup the mailing list.
>
> zer...@googlegroups.com
>
> Feel free to join. For thouse who don't know what zerogw is, here is a
> short description:
>
> Zerogw is an fast HTTP server with backend protocol that uses zeromq
> as transport. Zerogw grown from web game industry, so it supports
> websockets with fallback to longpolling, and a few very powerful
> websocket-related features which help to write very efficient chat,
> gaming and other near real-time applications.

Nice...! Where's the git repo?

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] Help with pattern design

[Alex Massover]

Hello,

I'm trying to design a pattern that will fit my application but can't find a 
proper solution. I'd appreciate if someone could enlighten me.

I have a process that deals with sessions (voip calls). This server produce 
events for every session, e.g.:
Session1: start
Session1: msgA
Session1: msgB
Session1: msgC
...
Session1: msgX
Session1: stop

Each event has a session id. Many sessions exist concurrently.

And there're multiple workers that consume these events. Single event should go 
to single worker. (PUSH/PULL like pattern).

My initial though was single PUSH (server that produce events) and multiple 
PULL (workers that consume).

My challenge is avoid race condition in processing events, i.e. workers 
supposed to consume events in the same order as they produced. And this should 
be done in realtime, I can't serialize at worker side. And single worker is not 
enough from performance point of view.

Is there any ZMQ magic (pattern) that can force all events from single session 
(based on session id?) to be sent to same worker? This way I'll ensure that 
worker process events in the same order they appeared.

--
Best Regards,
Alex Massover

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] ZMQ_MCAST_LOOP with PGM

[Pierre Ynard]

> The OS only forwards packets to the first open socket.

> Check the RFC's on PGM and PGMCC for details of the protocols in
> question,
>
> http://www.ietf.org/rfc/rfc3208.txt
> http://www.ietf.org/proceedings/57/I-D/draft-ietf-rmt-bb-pgmcc-02.txt

To be clear, if I understand correctly, OpenPGM binds a UDP socket
to the port given with the multicast address, using SO_REUSEADDR.
Then it uses this socket to both receive the multicast stream (hence
SO_REUSEADDR) and the unicast NAKs. This is why if a sender and a
receiver run on the same host, the socket of only one of them will
receive unicast NAKs, breaking it for the other one. Right?

So NAKs are sent back upstream with the same destination port as packets
flowing downstream (the multicasting port). To an outsider like me, this
feels confusing and counter-intuitive, and looks like a weird quirk.
I tried to gather the reason from the RFC but I got very confused by
§8.3: is this really what is described there? Keeping in mind that the
UDP encapsulation mode is not described by the RFC? Anyway, I don't
grasp the whole implications of this in the protocol, but it would make
more sense to me if the NAKs were sent back to the source address and
port of the sender, back from an address they came from. Also it might
avoid this problem in the first place.

But more importantly for ZeroMQ; if that's what it is, the problem
is not really related to multicast loopback, just to running several
instances on the same host. Then, running several senders (to the same
group-port) on the same host is a problem too as it breaks reliability
for all but one of them. And that wasn't solved by removing the
ZMQ_MCAST_LOOP option. To be thorough, there would also be some kind of
security issue on multi-user hosts: a malicious user could open a UDP
socket on that same port, and would at best DoS the ZeroMQ application
without it noticing, or at worst spy on leaked information about the PGM
network topology.

Isn't the problem a bit more complicated than just removing the loopback
option because it shouldn't be used? This option fits my use case so of
course I'd welcome its return :) But I'm worried by the rest too because
regardless of receivers and loopback, I plan on running several senders
in parallel on the same host.

Best regards,

-- 
Pierre Ynard
"Une âme dans un corps, c'est comme un dessin sur une feuille de papier."
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev