date:20110908

Re: [zeromq-dev] "too many open files" error in zmq/jzmq

2011-09-08 Thread Mikko Koppanen

2011/9/8 Igor 'Lo' (И.L.) :
> Hi all.
>
> What can cause an REQ/REP pair to crash (after doing multiple requests
> from Java to C++, crashing both sides at random choice) with following
> message:
>
> Too many open files
> rc == 0 (mailbox.cpp:375)
> (END)
>
> C++ side:
>
>        zmq::context_t context (1);
>        zmq::socket_t socket (context, ZMQ_REP);
>        socket.bind ("tcp:://localhost:1234);
>
>        while (true) {
>                void *context = zmq_init(1);
>                zmq::message_t *request = new zmq::message_t;
>                socket.recv (request);
> ... // parse request, prepare reply: unsigned char data with length dsize
>                zmq::message_t reply (dsize+1);
>                memcpy((void *)reply.data(), &data, dsize);
>                memset((void *)reply.data()+dsize, '\0', 1);
>                socket.send(reply);
>       }

Hi,

I notice that you are creating a new context in while loop in the C++
code. Is this intentional? I think each context creates at least one
socket for the internal logging, which would cause it to run out of
handles eventually as these contexts are not closed in that piece of
code. The Java side could run out of file handles if you are rapidly
opening and closing large amounts of sockets. It might be better to
create one socket per thread and use it for a longer period.

-- 
Mikko Koppanen
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] "too many open files" error in zmq/jzmq

2011-09-08 Thread И.L.

Hi all.

What can cause an REQ/REP pair to crash (after doing multiple requests
from Java to C++, crashing both sides at random choice) with following
message:

Too many open files
rc == 0 (mailbox.cpp:375)
(END)

C++ side:

zmq::context_t context (1);
zmq::socket_t socket (context, ZMQ_REP);
socket.bind ("tcp:://localhost:1234);

while (true) {
void *context = zmq_init(1);
zmq::message_t *request = new zmq::message_t;
socket.recv (request);
... // parse request, prepare reply: unsigned char data with length dsize
zmq::message_t reply (dsize+1);
memcpy((void *)reply.data(), &data, dsize);
memset((void *)reply.data()+dsize, '\0', 1);
socket.send(reply);
   }

Java side:

ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket jsocket = context.socket(ZMQ.REQ);
 jsocket.connect ("tcp://localhost:1234");
 jsocket.send(req, 0);
ZMQ.Poller it = context.poller(1);
it.setTimeout(2*100);
it.register(jsocket, ZMQ.Poller.POLLIN);
if (it.pollin(0)) {
 byte[] reply = jsocket.recv(0);
... // parse reply
}
jsocket.close();

Java can call the above code from multiple threads.
I suspect the handbook didn't told me how to close used resources in a
correct way.. shouldn't something be done upon zmq::message_t or with
zmq::context? I thought that scope exit at loop should've been
destroying them..

--
cheers,
Igor
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Can a Trading Platform rely on ØMQ?

2011-09-08 Thread Anatoly

Thanks Martin,
/Anatoly

On Wed, Sep 7, 2011 at 2:51 AM, Martin Sustrik  wrote:

> Hi Anatoly,
>
>
>   1. Is there a built in mechanism to e.g. start off loading
>> messages to disk, in case a queue is close to be overflowed? Or to just
>> STOP instead of discard messages silently?
>>
>
> There's ZMQ_SWAP option in 2.x versions.
>
> Applying backpressure ("STOP") works fro REQ/REP and PUSH/PULL patterns.
>
> With PUB/SUB, applying backpressure combined with slow/dead subscriber can
> lead to unbounded latencies, even to deadlock of the whole message
> distribution system.
>
>
>   2. Mad Black Box is something that looks the closest to what we
>> need, however we already have publishers themselves sharded. Would
>> sharding them further into different Subscriber's "PUSH"ers be necessary
>> to avoid a "subscriber's overflow"? ( e.g. processing side would be
>> slower, as it deserializes [probably google protobufs] and persists
>> messages to disk )
>>
>
> I guess you are speaking of market data here. If the publisher is
> overloaded, think of creating a more complex topology with devices in the
> middle to distribute the load.
>
>
>   3. In case the answer to 3 is YES, does not an additional internal
>> sharding introduce a new point of failure ( e.g. a thread dies, etc.. ),
>> in which case is there a some kind of built in recovery / retry
>> mechanism ( similar to PGM, but a bit of a higher level, since we are
>> dealing with ZMTP messages )?
>>
>
> If the point that stores the message dies, the message is lost. That
> applies to PGM or any other mechanism. The only option is to store the
> messages on a disk, with obvious performance penalty. Even then, if the disk
> dies, the messages are lost. To prevent that you have to store them on RAID,
> SAN or somesuch. If the whole RAID is destroyed etc.
>
> Martin
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

2011-09-08 Thread Ilja Golshtein

HWM stands for High Water Mark. I suggest you should understand what is going on. Do you actually see any output ("std::cout << update.size();") ?Any changes if do sleep instead of any receiving?Did you try to move setsockopt before connect?08.09.2011, 18:36, "Omer Bacharach" :what is hwm?On Thu, Sep 8, 2011 at 4:58 PM, Ilja Golshtein  wrote:Hello. Your code looks correct, although I am not sure if is Ok to do setsockopt after connect. I guess it is zmq internal thread what eats all memory.To check if it is wrong just do some wait instead of recv. The only way to solve comes to my mind - use HWM.08.09.2011, 17:38, "Omer Bacharach" : HiI'm using zmq on windows.I'm using a subscriber with a filter to receive only "12345" a million times and I have a publisher sending "12345" repeatedly  When running both modules the subscriber shouts (after a while): "FATAL ERROR: OUT OF MEMORY".Any idea?Here is the relevant snippet from the subscriber's code:    zmq::context_t context (1);    //  Socket to talk to server    std::cout << "Collecting updates from weather server…\n" << std::endl;    zmq::socket_t subscriber (context, ZMQ_SUB);    subscriber.connect("tcp://localhost:5556");    subscriber.setsockopt(ZMQ_SUBSCRIBE, "12345", strlen ("12345"));    int update_nbr;    zmq::message_t update;    for (update_nbr = 0; update_nbr < 100; update_nbr++) {             subscriber.recv(&update);        std::cout << update.size(); }Regards,Omer.-- Omer BacharachDirector of Software Development Toot Trading Ltd. o...@toot-trading.comMobile +972-544-317959Office +972-3-7449466 -- Omer BacharachDirector of Software Development Toot Trading Ltd.o...@toot-trading.com Mobile +972-544-317959Office +972-3-7449466___zeromq-dev mailing listzeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev-- Best regards,Ilja Golshtein.___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Omer BacharachDirector of Software Development Toot Trading Ltd.o...@toot-trading.comMobile +972-544-317959Office +972-3-7449466___zeromq-dev mailing listzeromq-dev@lists.zeromq.orghttp://lists.zeromq.org/mailman/listinfo/zeromq-dev-- Best regards,Ilja Golshtein.___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

2011-09-08 Thread Gary Davidson

What version of windows?

Try this and see if you get the same error

subscriber.setsockopt(ZMQ_SUBSCRIBE, "", 0);

I have run several tests on windows with 5 pubs and 1 sub and do 4+ million 
message all the time.

I am setting up a test to run several hundred million messages in a 24 hour 
test . My SLA is 1 billion in 24 hours.

Will let you know how it goes.



[Description: 
C:\Users\gary.davidson\AppData\Roaming\Microsoft\AD_interclick\logo.png]

Gary Davidson, Chief Software Architect
gary.david...@interclick.com
P 561 300 2764
C 954 790 0715
F 561 300 2765

4800 T-Rex Avenue Suite 120
Boca Raton, FL 33431

www.interclick.com
NASDAQ: ICLK



From: zeromq-dev-boun...@lists.zeromq.org 
[mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Ilja Golshtein
Sent: Thursday, September 08, 2011 9:59 AM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF 
MEMORY"

Hello.

Your code looks correct, although I am not sure if is Ok to do setsockopt after 
connect.

I guess it is zmq internal thread what eats all memory.
To check if it is wrong just do some wait instead of recv.

The only way to solve comes to my mind - use HWM.
08.09.2011, 17:38, "Omer Bacharach" 
mailto:o...@toot-trading.com>>:

Hi
I'm using zmq on windows.
I'm using a subscriber with a filter to receive only "12345" a million times 
and I have a publisher sending "12345" repeatedly
When running both modules the subscriber shouts (after a while): "FATAL ERROR: 
OUT OF MEMORY".
Any idea?
Here is the relevant snippet from the subscriber's code:
zmq::context_t context (1);
//  Socket to talk to server
std::cout << "Collecting updates from weather server…\n" << std::endl;
zmq::socket_t subscriber (context, ZMQ_SUB);
subscriber.connect("tcp://localhost:5556");
subscriber.setsockopt(ZMQ_SUBSCRIBE, "12345", strlen ("12345"));
int update_nbr;
zmq::message_t update;
for (update_nbr = 0; update_nbr < 100; update_nbr++)
 {

subscriber.recv(&update);
std::cout << update.size();
 }
Regards,
Omer.
--
Omer Bacharach
Director of Software Development
Toot Trading Ltd.
o...@toot-trading.com
Mobile +972-544-317959
Office +972-3-7449466



--
Omer Bacharach
Director of Software Development
Toot Trading Ltd.
o...@toot-trading.com
Mobile +972-544-317959
Office +972-3-7449466

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
--
Best regards,
Ilja Golshtein.
<>___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

2011-09-08 Thread Omer Bacharach

what is hwm?

On Thu, Sep 8, 2011 at 4:58 PM, Ilja Golshtein  wrote:

> Hello.
>
> Your code looks correct, although I am not sure if is Ok to do setsockopt
> after connect.
>
> I guess it is zmq internal thread what eats all memory.
> To check if it is wrong just do some wait instead of recv.
>
> The only way to solve comes to my mind - use HWM.
> 08.09.2011, 17:38, "Omer Bacharach" :
>
>
> Hi
> I'm using zmq on windows.
> I'm using a subscriber with a filter to receive only "12345" a million
> times and I have a publisher sending "12345" repeatedly
> When running both modules the subscriber shouts (after a while): "FATAL
> ERROR: OUT OF MEMORY".
> Any idea?
> Here is the relevant snippet from the subscriber's code:
> zmq::context_t context (1);
> //  Socket to talk to server
> std::cout << "Collecting updates from weather server…\n" << std::endl;
> zmq::socket_t subscriber (context, ZMQ_SUB);
> subscriber.connect("tcp://localhost:5556");
> subscriber.setsockopt(ZMQ_SUBSCRIBE, "12345", strlen ("12345"));
> int update_nbr;
> zmq::message_t update;
> for (update_nbr = 0; update_nbr < 100; update_nbr++)
>  {
>
> subscriber.recv(&update);
> std::cout << update.size();
>  }
> Regards,
> Omer.
> --
> Omer Bacharach
> Director of Software Development
> Toot Trading Ltd.
> o...@toot-trading.com
> Mobile +972-544-317959
> Office +972-3-7449466
>
>
>
> --
> Omer Bacharach
> Director of Software Development
> Toot Trading Ltd.
> o...@toot-trading.com
> Mobile +972-544-317959
> Office +972-3-7449466
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> --
> Best regards,
> Ilja Golshtein.
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>


-- 
Omer Bacharach
Director of Software Development
Toot Trading Ltd.
o...@toot-trading.com
Mobile +972-544-317959
Office +972-3-7449466
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq C++ wrapper

2011-09-08 Thread andrew S

Hello, I am working for the same company as Ilja, and we finally
published our C++ library for multipart messages.
It is accessible at http://zmqmessage.org/  and it has documentation,
tutorial, examples, etc.
Note that this is a narrowly specialized tool which extensively deals
with multipart messages and nothing more (that's why it's called
zmqmessage). It uses standard C++ mapping (zmq.hpp in 0mq 2.x)  and
does not try to replace it or "cover" and hide completely.
The advantages of such approach, as opposed to single monolithic C++
binding mixing high and low level abstractions are:
- the C++ binding remains simple, elementary and it's API is the same
as in other languages,
- higher-level tools (another such tool could be zmq reactor) are
completely independent from each other, can be developed separately,
and users may use them only if they need them.

Anyway, if the community decides to create such thick binding out of
cppzmq, the code and concepts from zmqmessage could be used.

Andrew

On Wed, Sep 7, 2011 at 3:10 PM, Pieter Hintjens  wrote:
> On Wed, Sep 7, 2011 at 6:07 AM, Pieter Hintjens  wrote:
>
>> The lack of any improvement to cppzmq since it was created as a
>> project is a significant sign. I've no idea who will make cppzmq work
>> with 0MQ/3.0 or 4.0, and there are no abstractions in the binding that
>> would allow it to be multi-version, as other bindings are.
>
> Sorry, my confusion. In 2.x, this is still zmq.hpp. In 3.0 and on,
> it's a separate cppzmq project. My advice is still to invest in a
> single C++ binding that mixes high and low level semantics.
>
> -Pieter
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

2011-09-08 Thread Ilja Golshtein

Hello. Your code looks correct, although I am not sure if is Ok to do setsockopt after connect. I guess it is zmq internal thread what eats all memory.To check if it is wrong just do some wait instead of recv. The only way to solve comes to my mind - use HWM.08.09.2011, 17:38, "Omer Bacharach" :HiI'm using zmq on windows.I'm using a subscriber with a filter to receive only "12345" a million times and I have a publisher sending "12345" repeatedly  When running both modules the subscriber shouts (after a while): "FATAL ERROR: OUT OF MEMORY".Any idea?Here is the relevant snippet from the subscriber's code:    zmq::context_t context (1);    //  Socket to talk to server    std::cout << "Collecting updates from weather server…\n" << std::endl;    zmq::socket_t subscriber (context, ZMQ_SUB);    subscriber.connect("tcp://localhost:5556");    subscriber.setsockopt(ZMQ_SUBSCRIBE, "12345", strlen ("12345"));    int update_nbr;    zmq::message_t update;    for (update_nbr = 0; update_nbr < 100; update_nbr++) {             subscriber.recv(&update);        std::cout << update.size(); }Regards,Omer.-- Omer BacharachDirector of Software Development Toot Trading Ltd.o...@toot-trading.comMobile +972-544-317959Office +972-3-7449466 -- Omer BacharachDirector of Software Development Toot Trading Ltd.o...@toot-trading.comMobile +972-544-317959Office +972-3-7449466___zeromq-dev mailing listzeromq-dev@lists.zeromq.orghttp://lists.zeromq.org/mailman/listinfo/zeromq-dev-- Best regards,Ilja Golshtein.___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

2011-09-08 Thread Omer Bacharach

> Hi
> I'm using zmq on windows.
>
> I'm using a subscriber with a filter to receive only "12345" a million
> times and I have a publisher sending "12345" repeatedly
> When running both modules the subscriber shouts (after a while): "FATAL
> ERROR: OUT OF MEMORY".
> Any idea?
>
> Here is the relevant snippet from the subscriber's code:
>
>
> zmq::context_t context (1);
>
> //  Socket to talk to server
> std::cout << "Collecting updates from weather server…\n" << std::endl;
> zmq::socket_t subscriber (context, ZMQ_SUB);
> subscriber.connect("tcp://localhost:5556");
>
> subscriber.setsockopt(ZMQ_SUBSCRIBE, "12345", strlen ("12345"));
>
> int update_nbr;
> zmq::message_t update;
>
> for (update_nbr = 0; update_nbr < 100; update_nbr++)
>  {
>
> subscriber.recv(&update);
> std::cout << update.size();
>
>  }
>
>
> Regards,
> Omer.
> --
> Omer Bacharach
> Director of Software Development
> Toot Trading Ltd.
> o...@toot-trading.com
> Mobile +972-544-317959
> Office +972-3-7449466
>
>
>


-- 
Omer Bacharach
Director of Software Development
Toot Trading Ltd.
o...@toot-trading.com
Mobile +972-544-317959
Office +972-3-7449466
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] "too many open files" error in zmq/jzmq

[zeromq-dev] "too many open files" error in zmq/jzmq

Re: [zeromq-dev] Can a Trading Platform rely on ØMQ?

Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

Re: [zeromq-dev] zmq C++ wrapper

Re: [zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

[zeromq-dev] subscriber module prompting "FATAL ERROR: OUT OF MEMORY"

9 matches

Site Navigation

Mail list logo

Footer information