[zeromq-dev] zmq @ datasift

2011-11-30 Thread Dhammika Pathirana
They send billions of messages a day through 0mq http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-12-tweets-p.html and they're running alpha build in production o_O Dhammika ___ zeromq-dev mailing list

Re: [zeromq-dev] protocol design question

2011-11-16 Thread Dhammika Pathirana
Hi Chuck, On Wed, Nov 16, 2011 at 7:11 AM, Chuck Remes cremes.devl...@mac.com wrote: On Nov 16, 2011, at 1:19 AM, Martin Sustrik wrote: Hi Chuck, The UUID would uniquely identify each client. Give some thought to what do you want to uniquely identify. Is it any client ever? If so, UUID

[zeromq-dev] freebsd netmap

2011-06-24 Thread Dhammika Pathirana
Hi, This is similar to pf_ring on linux, but more usable from user space. http://info.iet.unipi.it/~luigi/netmap Apparently they are baking a linux port, should be interesting... Dhammika ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org

Re: [zeromq-dev] Windows 7 performance comparisons

2011-06-24 Thread Dhammika Pathirana
Hi Martin, On Wed, Jun 22, 2011 at 2:12 PM, Martin Sustrik sust...@250bpm.com wrote: On 06/22/2011 11:07 PM, Steven McCoy wrote: Following on from within a VM, here is native Windows results. Pretty bad. Nobody ever done much optimisation on Win platform :| Btw, I recall there's the poll

Re: [zeromq-dev] freebsd netmap

2011-06-24 Thread Dhammika Pathirana
On Fri, Jun 24, 2011 at 1:33 AM, Martin Sustrik sust...@250bpm.com wrote: On 06/24/2011 10:29 AM, Dhammika Pathirana wrote: This is similar to pf_ring on linux, but more usable from user space. http://info.iet.unipi.it/~luigi/netmap Apparently they are baking a linux port, should

Re: [zeromq-dev] Meetup in Seat^H^H SF/ Bay Area

2011-05-29 Thread Dhammika Pathirana
Hi Seb, Apparently backtype is using zmq in their stream processing, http://tech.backtype.com/preview-of-storm-the-hadoop-of-realtime-proce Would be great if we can get some info on their experience. Dhammika On Sat, May 28, 2011 at 12:51 PM, Sebastien Pahl s...@dotcloud.com wrote: I just

Re: [zeromq-dev] Meetup in Seat^H^H SF/ Bay Area

2011-05-22 Thread Dhammika Pathirana
On Fri, May 20, 2011 at 4:40 PM, Pieter Hintjens p...@imatix.com wrote: On Fri, May 20, 2011 at 8:10 PM, Jon Gifford j...@loggly.com wrote: plenty of places nearby: 83 Proof, Louies, Kate O'Briens, The Irish Bank, Harringtons, Tadich, Salt House, Rickhouse, ... I've tentatively put up 7pm on

Re: [zeromq-dev] Problem building application in Suse 11.3 (uuid lib issues)

2011-03-06 Thread Dhammika Pathirana
Hi PJ, On Sun, Mar 6, 2011 at 10:34 PM, PJ Durai pjdtech2...@gmail.com wrote: Greetings, I am having some issues in building my application in Suse 11.3 instalation. Zero MQ library itself has no problems building. But when I build my application, I end up with the following error

Re: [zeromq-dev] Meetup in San Francisco

2011-02-27 Thread Dhammika Pathirana
On Fri, Feb 25, 2011 at 10:54 AM, Martin Sustrik sust...@250bpm.com wrote: On 02/25/2011 07:50 PM, Martin Sustrik wrote: Thougts anyone? As for the schedule, I would propose Tuesday, March 8th, after the working hours. If there's an interest in technical session we could schedule that say

Re: [zeromq-dev] Discuss: 0MQ releases and stability

2011-02-16 Thread Dhammika Pathirana
On Wed, Feb 16, 2011 at 12:44 AM, Pieter Hintjens p...@imatix.com wrote: On Wed, Feb 16, 2011 at 7:32 AM, Dhammika Pathirana dhamm...@gmail.com wrote: 1. Stop using topic branches for new development, use the master for this But we need a more robust testing framework. (a) if new work

Re: [zeromq-dev] need help with linux platform

2011-02-16 Thread Dhammika Pathirana
On Tue, Feb 15, 2011 at 11:39 PM, Martin Sustrik sust...@250bpm.com wrote: On 02/16/2011 07:01 AM, Dhammika Pathirana wrote: We have to either block on send() or return EAGAIN, but these are not trivial changes :-( There's no easy solution. Blocking leads to deadlocks. EAGAIN is just

Re: [zeromq-dev] need help with linux platform

2011-02-15 Thread Dhammika Pathirana
Hi Chuck, On Tue, Feb 15, 2011 at 4:52 PM, Chuck Remes cremes.devl...@mac.com wrote: On Feb 15, 2011, at 6:00 PM, Chuck Remes wrote: Due to some ongoing issues with 0mq on OSX, I switched over to using my linux box as the main dev and test server. I am running a very recent master from

Re: [zeromq-dev] need help with linux platform

2011-02-15 Thread Dhammika Pathirana
On Tue, Feb 15, 2011 at 6:30 PM, Chuck Remes cremes.devl...@mac.com wrote: On Feb 15, 2011, at 7:54 PM, Chuck Remes wrote: On Feb 15, 2011, at 7:41 PM, Dhammika Pathirana wrote: Assertion failed: new_sndbuf old_sndbuf (mailbox.cpp:183) new_sndbuf = 10485760, old_sndbuf = 10485760

Re: [zeromq-dev] Bad File Descriptor / nbytes != -1 error (mailbox.cpp:241)

2011-02-15 Thread Dhammika Pathirana
Hi Andrew, On Mon, Feb 14, 2011 at 6:29 PM, Andrew Cholakian and...@andrewvc.com wrote: It looks like what's happening is the following sequence inside of ZeroMQ 1. zmq tries to recvfrom FD 22, on FD in an internal zmq socketpair, and gets some data. 2. Periodic attempts after that (always in

Re: [zeromq-dev] assertion at mailbox.cpp:182

2011-01-31 Thread Dhammika Pathirana
Hi Pavel, On Sun, Jan 30, 2011 at 6:35 AM, Pavel Gushcha pavi...@gmail.com wrote: Hi, All! It seems, that I have the same problem as Zoufal Andreas: http://lists.zeromq.org/pipermail/zeromq-dev/2011-January/008914.html From some timepoint 2 applications from my system start to crash

Re: [zeromq-dev] [PATCH] Custom zmq_msg_t allocator

2011-01-20 Thread Dhammika Pathirana
Hi Douglas, On Thu, Jan 20, 2011 at 7:23 AM, Douglas Creager dcrea...@dcreager.net wrote: I like the tcmalloc suggestion, it sounds like an easier way around the problem.  That wouldn't require any changes at all to zeromq, right?  I'd only do the -ltcmalloc when linking the final

Re: [zeromq-dev] [PATCH] Fix handle connection reset during session init

2011-01-19 Thread Dhammika Pathirana
Hi Martin, On Wed, Jan 19, 2011 at 6:22 AM, Martin Sustrik sust...@250bpm.com wrote: Dhammika, +            //  Check if we had an error in previous attempt. +            if (unlikely (!(static_castT*  (this)-next))) +                return (size_t) -1; + I think this won't work. The next

Re: [zeromq-dev] nmap patch for github issues 147+149

2011-01-18 Thread Dhammika Pathirana
Hi Thijs, On Tue, Jan 18, 2011 at 1:41 AM, Thijs Terlouw thijsterl...@gmail.com wrote: From: Dhammika Pathirana dhamm...@gmail.com Subject: Re: [zeromq-dev] nmap patch for github issues 147+149 Do you have a coredump? Could you post a backtrace, run thread apply all bt in gdb. Hi, so far

[zeromq-dev] [PATCH] Fix handle connection reset during session init

2011-01-18 Thread Dhammika Pathirana
Patch to handle nmap version probes. Signed-off-by: Dhammika Pathirana dhamm...@gmail.com --- src/decoder.hpp|4 src/zmq_engine.cpp |2 +- 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/src/decoder.hpp b/src/decoder.hpp index 9da6f72..80987a8 100644 --- a/src

Re: [zeromq-dev] nmap patch for github issues 147+149

2011-01-17 Thread Dhammika Pathirana
Hi Thijs, On Mon, Jan 17, 2011 at 4:51 AM, Thijs Terlouw thijsterl...@gmail.com wrote: From: Pieter Hintjens p...@imatix.com This is really great, thanks! Could you [...] resubmit the patch as a signed-off patch following those instructions? in the first attempt i didn't use the correct

Re: [zeromq-dev] Segfault

2011-01-04 Thread Dhammika Pathirana
Hi Martin, On Tue, Jan 4, 2011 at 1:53 AM, Martin Sustrik sust...@250bpm.com wrote: Hi Dhammika, It's actually in zmq. Terminated pipe writer gets a activate command. But load balancer doesn't decrement active pipe count in terminate() call.  55 void zmq::lb_t::terminate ()  56 {  57    

Re: [zeromq-dev] Segfault

2011-01-01 Thread Dhammika Pathirana
this? There's a off by one indexing error in load balancer. Shutdown code is bit gnarly. I've attached a patch below, but haven't tested it. Subject: [PATCH] Fix pipe writer termination Signed-off-by: Dhammika Pathirana dhamm...@gmail.com --- src/pipe.cpp |5 +++-- 1 files changed, 3 insertions(+), 2

Re: [zeromq-dev] Q; running multiple REP listeners on an IPC socket, only the last one connected gets any messages

2010-12-18 Thread Dhammika Pathirana
On Sat, Dec 18, 2010 at 12:30 AM, Martin Sustrik sust...@250bpm.com wrote: Dhammika, 1) Using one zmq context, I bind four separate zmq sockets (in java) to the same ipc port; e.g. ipc://tmp/test Hm, you should be able to bind at most one socket to a particular endpoint. Are you sure all

Re: [zeromq-dev] Q; running multiple REP listeners on an IPC socket, only the last one connected gets any messages

2010-12-17 Thread Dhammika Pathirana
Martin, On Fri, Dec 17, 2010 at 3:31 AM, Martin Sustrik sust...@250bpm.com wrote: Hi, 1) Using one zmq context, I bind four separate zmq sockets (in java) to the same ipc port; e.g. ipc://tmp/test Hm, you should be able to bind at most one socket to a particular endpoint. Are you sure all

Re: [zeromq-dev] Question Regarding XREP

2010-12-15 Thread Dhammika Pathirana
Praveen, On Tue, Dec 14, 2010 at 1:06 PM, Praveen Baratam praveen.baratam+...@gmail.com wrote: Dhammika, Any progress with this issue? I was trying to recreate this, but no luck :-( https://gist.github.com/743150/2e72d18c7f2c1274a21c7fd06f1aafc47ec9764e Are you using language bindings?

Re: [zeromq-dev] Question Regarding XREP

2010-12-14 Thread Dhammika Pathirana
Hi Praveen, On Tue, Dec 14, 2010 at 1:06 PM, Praveen Baratam praveen.baratam+...@gmail.com wrote: Dhammika, Any progress with this issue? Sorry for the delay, I'll take a look. XREQ/XREP are widely used, we haven't had any other issues reported on socket shutdown. On duplicate identities, I

Re: [zeromq-dev] zmq and abstract unix domain sockets

2010-12-12 Thread Dhammika Pathirana
Hi Martin, On Sun, Dec 12, 2010 at 3:08 AM, Martin Sustrik sust...@250bpm.com wrote: Hi Dhammika, I've checked this patch. See my comments inlines. Sorry for the delay :( I still have your shutdown patch in the review queue. I've checked it once and it seemed to be OK, but the shutdown code

[zeromq-dev] [PATCH] fix overwriting errno on bind failure

2010-12-12 Thread Dhammika Pathirana
From 8dd8d580beb9fd84aec99aeb08ead34e3206a080 Mon Sep 17 00:00:00 2001 From: Dhammika Pathirana dhamm...@gmail.com Date: Sun, 12 Dec 2010 22:55:55 -0800 Subject: [PATCH 1/2] fix overwriting errno on bind failure Signed-off-by: Dhammika Pathirana dhamm...@gmail.com --- src/tcp_listener.cpp

[zeromq-dev] [PATCH] add uri validations

2010-12-12 Thread Dhammika Pathirana
From b6571910583e4d9dad094d6ec9fe5d430fe7db4c Mon Sep 17 00:00:00 2001 From: Dhammika Pathirana dhamm...@gmail.com Date: Sun, 12 Dec 2010 22:56:28 -0800 Subject: [PATCH 2/2] add basic uri validations Signed-off-by: Dhammika Pathirana dhamm...@gmail.com --- src/socket_base.cpp | 50

Re: [zeromq-dev] segfault in zmq_connect

2010-12-09 Thread Dhammika Pathirana
Hi Mahadevan, On Wed, Dec 8, 2010 at 2:14 AM, Mahadevan R mdeva...@gmail.com wrote: Hi guys, I'm getting a strange segfault with the following stack: #0  0x7f623e64edd0 in _int_malloc () from /lib/libc.so.6 #1  0x7f623e650ad8 in malloc () from /lib/libc.so.6 #2  0x7f623c375d4f in

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-12-07 Thread Dhammika Pathirana
Hi Martin, On Mon, Dec 6, 2010 at 4:38 AM, Martin Sustrik sust...@250bpm.com wrote: Hi Dhammika, Patch attached. I've checked the patch. Shouldn't the 'dispatch' be called when the engine is actually being moved to another thread (inside finalise_initialisation function) rather than in

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-12-05 Thread Dhammika Pathirana
On Fri, Dec 3, 2010 at 2:02 AM, Martin Sustrik sust...@250bpm.com wrote: Hi Dhammika, Patch attached. I've checked the patch. Shouldn't the 'dispatch' be called when the engine is actually being moved to another thread (inside finalise_initialisation function) rather than in 'read'

Re: [zeromq-dev] Question Regarding XREP

2010-12-02 Thread Dhammika Pathirana
On Thu, Dec 2, 2010 at 1:22 AM, Praveen Baratam praveen.baratam+...@gmail.com wrote: Dear Dhammika and Chuck, I am trying to use an XREP socket as a PUB socket with publisher side filtering. In our case we dont need multiple subscriptions from a single sub-socket, so we are trying to use  XREQ

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-12-02 Thread Dhammika Pathirana
Hi Martin, On Tue, Nov 23, 2010 at 5:49 AM, Martin Sustrik sust...@250bpm.com wrote: Dhammika, I donno, may be we should simplify this. Why don't we add a refcount? As a quick workaround -- yes. Do you have a patch for that kind of solution? However, thinking about it conceptually, the

Re: [zeromq-dev] Question Regarding XREP

2010-12-01 Thread Dhammika Pathirana
Hi Praveen, On Wed, Dec 1, 2010 at 2:19 PM, Praveen Baratam praveen.baratam+...@gmail.com wrote: Dear Chuck, Thanks for the quick reply. Well if ever an Identity is reused, I guess the earlier socket connected with that identity will be orphaned and all the messages will be routed to the new

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-23 Thread Dhammika Pathirana
On Tue, Nov 23, 2010 at 5:49 AM, Martin Sustrik sust...@250bpm.com wrote: Dhammika, I donno, may be we should simplify this. Why don't we add a refcount? As a quick workaround -- yes. Do you have a patch for that kind of solution? However, thinking about it conceptually, the problem is

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-22 Thread Dhammika Pathirana
On Sun, Nov 21, 2010 at 7:32 AM, Martin Sustrik sust...@250bpm.com wrote: On 11/21/2010 03:56 PM, Dhammika Pathirana wrote: if (!plugged) is dereferencing this-plugged, but we've deleted this object (engine). So it's going to access deleted memory. Right. Different option is to move

[zeromq-dev] Article on zdevices

2010-11-22 Thread Dhammika Pathirana
http://www.igvita.com/2010/11/17/routing-with-ruby-zeromq-device ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-21 Thread Dhammika Pathirana
On Fri, Nov 19, 2010 at 9:21 AM, Martin Sustrik sust...@250bpm.com wrote: On 11/19/2010 05:48 PM, Dhammika Pathirana wrote: -1 for auto buffer resizing. This is bound to go gaga anytime, and possibly has a remote trigger. Different option is to send cmd_t struct pointers. It'll give about

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-19 Thread Dhammika Pathirana
On Fri, Nov 19, 2010 at 4:28 AM, Martin Sustrik sust...@250bpm.com wrote: On 11/19/2010 01:08 PM, Martin Sustrik wrote: Hi Dhammika, I am trying to make sense from this, but I still don't see how it can happen. Can you possibly run the same test with printing unplugged in

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-19 Thread Dhammika Pathirana
On Tue, Nov 16, 2010 at 5:43 AM, Martin Lucina m...@kotelna.sk wrote: sust...@250bpm.com said: On 11/12/2010 11:34 AM, Christian Gudrian wrote: Hello, again! test_shutdown_stress proves stressing under Windows, too: The send call at mailbox.cpp:76 might return SOCKET_ERROR with

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-19 Thread Dhammika Pathirana
On Fri, Nov 19, 2010 at 8:53 AM, Martin Sustrik sust...@250bpm.com wrote: On 11/19/2010 05:33 PM, Dhammika Pathirana wrote: I think it's unplugged and passed to another thread, and second thread is deleting it. We do have two instances where we delete engines in process_attach(). Right

Re: [zeromq-dev] Strange client application fail

2010-11-17 Thread Dhammika Pathirana
Hi Олег, On Wed, Nov 17, 2010 at 2:35 AM, Олег Севостьянов oleg@gmail.com wrote: Maybe is not a bug, but I have got strange error while debuging with MSVC 2005. I have test client application (http://pastebin.com/emer9049) and test server application (http://pastebin.com/YTW732Xb). I

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-16 Thread Dhammika Pathirana
Hi Martin, On Sat, Nov 13, 2010 at 5:47 PM, Dhammika Pathirana dhamm...@gmail.com wrote: Hi Martin, On Sat, Nov 13, 2010 at 2:18 AM, Martin Sustrik sust...@250bpm.com wrote: On 11/13/2010 11:13 AM, Dhammika Pathirana wrote: In finalize_initialization(), we detach the engine and pass

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-13 Thread Dhammika Pathirana
On Fri, Nov 12, 2010 at 11:10 PM, Martin Sustrik sust...@250bpm.com wrote: Hi Dhammika, 0  zmq::signaler_t::send() 1  zmq::ctx_t::send_command() 2  zmq::object_t::send_command() 3  zmq::object_t::send_term_req() 4  zmq::own_t::terminate() 5  zmq::zmq_init_t::finalise_initialisation()    

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-13 Thread Dhammika Pathirana
Hi Martin, On Sat, Nov 13, 2010 at 2:18 AM, Martin Sustrik sust...@250bpm.com wrote: On 11/13/2010 11:13 AM, Dhammika Pathirana wrote: In finalize_initialization(), we detach the engine and pass it over to app thread. Actually, it's passed to another I/O thread rather than to the app thread

Re: [zeromq-dev] [PATCH] Re: assertion failure with master

2010-11-09 Thread Dhammika Pathirana
Hi, On Mon, Nov 8, 2010 at 4:39 AM, Martin Lucina m...@kotelna.sk wrote: Chuck, can you try the preliminary patch below against master and let me know if it fixes your problem? Martin, please review.  The code is getting decidedly ugly, so much for trying to emulate atomic message queueing

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-09 Thread Dhammika Pathirana
On Mon, Nov 8, 2010 at 3:01 AM, Martin Lucina m...@kotelna.sk wrote: dhamm...@gmail.com said: On Wed, Nov 3, 2010 at 1:12 PM, Christian Gudrian christ...@gudrian.org wrote: Am 03.11.2010 um 16:04 schrieb Dhammika Pathirana: ok, can you post mac crash log? Attached is a crash log

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-07 Thread Dhammika Pathirana
On Wed, Nov 3, 2010 at 1:12 PM, Christian Gudrian christ...@gudrian.org wrote: Am 03.11.2010 um 16:04 schrieb Dhammika Pathirana: ok, can you post mac crash log? Attached is a crash log for the Invalid argument error. Thanks for having a look. Christian This is in zmq master branch

Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails

2010-11-03 Thread Dhammika Pathirana
On Wed, Nov 3, 2010 at 12:49 AM, Christian Gudrian christ...@gudrian.org wrote: Am 03.11.2010 07:05, schrieb Dhammika Pathirana: size=-2 doesn't look right, Actually it's 444 -- I've used %d instead of %u.   could you post a test to recreate this Unfortunately no. As I've pointed out

Re: [zeromq-dev] EncryptedSocket added to pyzmq in branch

2010-11-03 Thread Dhammika Pathirana
Hi Min, On Tue, Nov 2, 2010 at 2:40 PM, MinRK benjami...@gmail.com wrote: On Tue, Nov 2, 2010 at 13:57, Burak Arslan burak.ars...@arskom.com.tr wrote:  On 11/02/10 21:27, MinRK wrote: Is there a better model for hiding message data using an unmodified current release version of zeromq,

Re: [zeromq-dev] Website -- separating newbies from the community

2010-10-29 Thread Dhammika Pathirana
Hi Pieter, On Thu, Oct 28, 2010 at 9:18 AM, Pieter Hintjens p...@imatix.com wrote: On Sat, Oct 23, 2010 at 7:18 PM, Martin Sustrik sust...@250bpm.com wrote: So let's do the zeromq.com-zeromq.org and zeromq.org-zeromq.org/community transition today and solve the remaining problems in

[zeromq-dev] [PATCH] drop connections with duplicate peer IDs

2010-10-26 Thread Dhammika Pathirana
Hi, Patches for issue #30 From f77b8e1ed020786c36216c819fb38d78d5a57d46 Mon Sep 17 00:00:00 2001 From: dhammika dhamm...@gmail.com Date: Mon, 25 Oct 2010 22:19:43 -0700 Subject: [PATCH] drop connection requests with duplicate peer identity Signed-off-by: dhammika dhamm...@gmail.com ---

Re: [zeromq-dev] [PATCH] drop connections with duplicate peer IDs

2010-10-26 Thread Dhammika Pathirana
Hi Martin, On Tue, Oct 26, 2010 at 7:04 AM, Martin Sustrik sust...@250bpm.com wrote: Dhammika, Patches for issue #30 Applied. You missed this, From 2f1de7f95aeae3b40fc1c851f052c7c0c0b9dee7 Mon Sep 17 00:00:00 2001 From: dhammika dhamm...@gmail.com Date: Mon, 25 Oct 2010 23:12:56 -0700

Re: [zeromq-dev] my day for oddities

2010-10-26 Thread Dhammika Pathirana
Hi Andrew, On Tue, Oct 26, 2010 at 7:06 AM, Andrew Hume and...@research.att.com wrote: i am getting sporadic core dumps with my zmq programs. no pattern detectable yet, except that whenever it happens, the immediate cause is a zero zmq_msg_t. e.g. (gdb) bt #0  zmq_msg_close(._0 *)

[zeromq-dev] [PATCH] decoder asserts

2010-10-21 Thread Dhammika Pathirana
Patch to handle decoding errors. From 0a179e4c16e390e091629ab83f08b813ae5ad4ed Mon Sep 17 00:00:00 2001 From: dhammika dhamm...@gmail.com Date: Wed, 20 Oct 2010 19:27:00 -0700 Subject: [PATCH] handle decoding malformed messages Signed-off-by: dhammika dhamm...@gmail.com --- src/decoder.cpp

Re: [zeromq-dev] [PATCH] Scalability improvements for large amounts of connections

2010-10-14 Thread Dhammika Pathirana
Hi Mato, +int zmq::zmq_connecter_t::get_reconnect_period () +{ +#if defined ZMQ_HAVE_WINDOWS +    return (reconnect_period + (((int)GetCurrentProcessId () * 13) +        % reconnect_period)); +#else +    return (reconnect_period + (((int)getpid () * 13) % reconnect_period)); +#endif +}

Re: [zeromq-dev] [PATCH] Crash if several TCP connections are closed simultaneously

2010-09-27 Thread Dhammika Pathirana
Hi, On Mon, Sep 27, 2010 at 5:14 AM, Martin Sustrik sust...@250bpm.com wrote: Dhammika, Good work! Can you submit this patch under MIT/X11 so that I can apply it? Patch attached, submitting under MIT/X11 license. Dhammika From 46f0018eee7e5cb792faed4ab9157fe03a7c7316 Mon Sep 17 00:00:00

Re: [zeromq-dev] Crash if several TCP connections are closed simultaneously

2010-09-19 Thread Dhammika Pathirana
On Thu, Sep 16, 2010 at 2:54 AM, Ilja Golshtein ilej...@narod.ru wrote: Hello! I've logged issue http://github.com/zeromq/zeromq2/issues/issue/79 Basically server application crashes if number of incoming TCP (under zmq control) connections are terminated. Looks like it does not matter what

Re: [zeromq-dev] Crash if inproc intensively used

2010-09-07 Thread Dhammika Pathirana
:-( The initial issue http://github.com/zeromq/zeromq2/issues#issue/64 is very important for me and I do appreciate any assistance. Thanks. 07.09.10, 08:36, Dhammika Pathirana dhamm...@gmail.com: Hi Ilja,  Works ok with master branch, but there's a bug in initializing  zmq::context_t

Re: [zeromq-dev] Crash if inproc intensively used

2010-09-06 Thread Dhammika Pathirana
Hi Ilja, Works ok with master branch, but there's a bug in initializing zmq::context_t with 0 io threads. I've attached following patch, diff --git a/src/ctx.cpp b/src/ctx.cpp index 65c5316..ec8a045 100644 --- a/src/ctx.cpp +++ b/src/ctx.cpp @@ -64,7 +64,7 @@ zmq::ctx_t::ctx_t (uint32_t

[zeromq-dev] [PATCH] assert on 0 length messages

2010-08-27 Thread Dhammika Pathirana
Hi, On receiving a new message, decoder inits a msg with size (*tmpbuf - 1). But a sender can craft a message such that *tmpbuf is 0 (ie. zmq::message_t msg((size_t)-1)). This creates a remote memory corruption in the receiver. Patch is a temporary fix, we need a better way to handle malformed

Re: [zeromq-dev] [PATCH] assert on 0 length messages

2010-08-27 Thread Dhammika Pathirana
, Aug 27, 2010 at 10:22 AM, Dhammika Pathirana dhamm...@gmail.com wrote: Hi, On receiving a new message, decoder inits a msg with size (*tmpbuf - 1). But a sender can craft a message such that *tmpbuf is 0 (ie. zmq::message_t msg((size_t)-1)). This creates a remote memory corruption

[zeromq-dev] Receive packet/flow steering

2010-08-06 Thread Dhammika Pathirana
Hi, Number of TCP receiver improvements in latest kernel, http://lwn.net/Articles/362339 http://lwn.net/Articles/382428 Concept is similar to Receive Side Scaling in high-end NICs, but this is done in software without any driver modifications. Impressive benchmarks. Dhammika

Re: [zeromq-dev] PATCH: zmq::signaler_t should not assert on EINTR

2010-08-06 Thread Dhammika Pathirana
Hi, -ssize_t nbytes = send (w, cmd_, sizeof (command_t), 0); +ssize_t nbytes; +do { +::send (w, cmd_, sizeof (command_t), 0); +} while (nbytes == -1 errno == EINTR); small typo, nbytes uninitialized. ___ zeromq-dev

[zeromq-dev] linux PF_RING

2010-08-01 Thread Dhammika Pathirana
Hi, Anyone following linux PF_RING developments? It's a new packet capturing API for Gbit wire speeds, a very different approach from current PF_PACKET based filtering. http://www.ntop.org/PF_RING.html http://luca.ntop.org/Ring.pdf http://luca.ntop.org/Blooms.pdf Dhammika

Re: [zeromq-dev] Creating a pyzmq organization on github

2010-07-22 Thread Dhammika Pathirana
On 7/22/10, Serge Aleynikov se...@aleynikov.org wrote: My work was based on foking Dhammika's initial implementation. I made many enhancements, added documentation, autotools-based config, cross-compilation for MacOS, but the two repos were not merged, so my repository can be thought of a

Re: [zeromq-dev] [patch] handle SIGPIPE

2010-07-07 Thread Dhammika Pathirana
I don't have the code handy right now, look for the call to pthread_sigmask(). The signal mask is cleared for all 0MQ I/O threads, so they will not get any signals (excepting SIGSEGV, etc.) Hence the SIGPIPE issue should be moot, unless on some broken OS a SIGPIPE correctly ignored by a

Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages

2010-07-07 Thread Dhammika Pathirana
On 7/7/10, Martin Lucina m...@kotelna.sk wrote: On 7/7/2010, Dhammika Pathirana dhamm...@gmail.com wrote: But how is this different from network or remote host queuing/dropping messages? Sending queued messages doesn't really guarantee delivery of messages. Of course this change

[zeromq-dev] [patch] handle SIGPIPE

2010-07-06 Thread Dhammika Pathirana
Patch to handle SIGPIPE in send(). SIGPIPE behavior is not consistent even across *ix platforms. Linux has MSG_NOSIGNAL and Mac supports SO_NOSIGPIPE. Best option is to set SIG_IGN, but it's more of an application setting. We should document this. diff --git a/src/tcp_connecter.cpp

Re: [zeromq-dev] [patch] handle SIGPIPE

2010-07-06 Thread Dhammika Pathirana
On 7/6/10, Martin Lucina m...@kotelna.sk wrote: dhamm...@gmail.com said: Patch to handle SIGPIPE in send(). SIGPIPE behavior is not consistent even across *ix platforms. Linux has MSG_NOSIGNAL and Mac supports SO_NOSIGPIPE. Best option is to set SIG_IGN, but it's more of an

Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages

2010-07-06 Thread Dhammika Pathirana
But how is this different from network or remote host queuing/dropping messages? Sending queued messages doesn't really guarantee delivery of messages. This gets even worse as TCP sends RST (ECONNRESET) on receiving data to a closed socket. In http world they work around this by sender doing a

Re: [zeromq-dev] Updated home page for zeromq.org

2010-05-28 Thread Dhammika Pathirana
Hi Pieter, Can we add a link to rfc.zeromq.org? Even ~search~ can't find it. On 5/28/10, Pieter Hintjens p...@imatix.com wrote: Hi folks, Here's a draft of a new homepage for the site, aiming to make better use of the space: http://www.zeromq.org/sandbox:start Feedback welcome,

Re: [zeromq-dev] Erlang 0MQ bindings

2010-05-26 Thread Dhammika Pathirana
Hi Serge, Awesome, let's publish this in the web page. On 5/25/10, Serge Aleynikov se...@aleynikov.org wrote: Here's the latest version of Erlang 0MQ bindings which is a fork of Dhammika's work. http://github.com/saleyn/erlzmq make docs generates documentation in HTML format, which is

Re: [zeromq-dev] zeromq Erlang client

2010-05-17 Thread Dhammika Pathirana
Hi Pieter, Thanks for posting this in the wiki, but it's not quite ready yet. Is it ok to discuss erlang related stuff (not necessarily on zmq) on this list? Dhammika On 5/15/10, Pieter Hintjens p...@imatix.com wrote: Jon, Thanks for that, I'd forgotten Dhammika's email. I've added a

Re: [zeromq-dev] zmq_poll

2010-05-13 Thread Dhammika Pathirana
AM, Dhammika Pathirana wrote: Hi Serge, I'm working on an erlang driver, if you're interested check it out. http://github.com/dhammika/erlzmq This is experimental/half-baked code, but most of the basic functionality is there. Dhammika On 5/11/10, Serge Aleynikovse

Re: [zeromq-dev] zmq_poll

2010-05-12 Thread Dhammika Pathirana
Hi Serge, I'm working on an erlang driver, if you're interested check it out. http://github.com/dhammika/erlzmq This is experimental/half-baked code, but most of the basic functionality is there. Dhammika On 5/11/10, Serge Aleynikov se...@aleynikov.org wrote: Hi, The current design of

Re: [zeromq-dev] Proposal for 2.0.7: Cleaning up the ZMQ_* constants

2010-04-15 Thread Dhammika Pathirana
On 4/15/10, Martin Lucina m...@kotelna.sk wrote: All, with the introduction of more send/recv flags (ZMQ_SNDMORE), socket options (ZMQ_RCVMORE) and the experimental device api (ZMQ_STREAMER et al) the ZMQ_ namespace for constants is starting to look rather ad-hoc and crowded. There are

Re: [zeromq-dev] 0MQ craching while polling in I/O thread

2010-03-28 Thread Dhammika Pathirana
Hi, Think we're casting zmq::socket_t to zmq::socket_base_t in zmq_poll(). Patch attached. On 3/25/10, Martin Sustrik sust...@250bpm.com wrote: Nelson, Brian, You've both reported an error where polling in I/O thread crashed. Thanks for the backtraces. I've spent some time staring at

Re: [zeromq-dev] Non-contiguous message thoughts

2010-03-06 Thread Dhammika Pathirana
Hi Martin, We can add scatter/gather IO API to zmq_msg. Proper readv/writev on sockets will need some changes in encoder/decoder, but we can use current encoder/decoder and read/write each iov entry separately. typedef struct { void *content; unsigned char shared; unsigned char

Re: [zeromq-dev] Non-contiguous message thoughts

2010-03-06 Thread Dhammika Pathirana
I like msg groups idea, but it sounds more like an application feature. I was thinking about application level framing, adding a msg header, body and checksum. If we can setup different buffers for each msg segment, then sender won't have to create a contiguous block and receiver can avoid having