Re: [zeromq-dev] HWM default

2011-05-15 Thread Martin Sustrik
On 05/14/2011 04:16 PM, Paul Colomiets wrote: > What else would you do if you run out of memory? Any recovery > mechanism in user space is likely to fail because there's no memory > available :) Killing (and possibly restarting) the app seems like a > reasonable option to me. > > O

Re: [zeromq-dev] HWM default

2011-05-14 Thread Paul Colomiets
On Thu, May 12, 2011 at 9:56 AM, Martin Sustrik wrote: > On 05/09/2011 02:41 PM, Paul Colomiets wrote: > > Well, I have no practical experience, but a lot of alloc_assert's in the >> code says that it will be aborted. BTW, it will be so not only if >> overcommit_memory is set right, but also if

Re: [zeromq-dev] HWM default

2011-05-12 Thread Blair Bethwaite
On 12 May 2011 16:56, Martin Sustrik wrote: > On 05/09/2011 02:41 PM, Paul Colomiets wrote: > >> Well, I have no practical experience, but a lot of alloc_assert's in the >> code says that it will be aborted. BTW, it will be so not only if >> overcommit_memory is set right, but also if ulimit is se

Re: [zeromq-dev] HWM default

2011-05-11 Thread Martin Sustrik
On 05/09/2011 02:41 PM, Paul Colomiets wrote: > Well, I have no practical experience, but a lot of alloc_assert's in the > code says that it will be aborted. BTW, it will be so not only if > overcommit_memory is set right, but also if ulimit is set for the > application's memory. Which is awfully

Re: [zeromq-dev] HWM default

2011-05-09 Thread Paul Colomiets
On Mon, May 9, 2011 at 9:17 AM, Blair Bethwaite wrote: > > > As an aside, any idea of the 0MQ behaviour under flooding with no HWM > but sane kernel virtual memory settings (i.e., > /proc/sys/vm/overcommit_memory=2)...? > > Well, I have no practical experience, but a lot of alloc_assert's in the co

Re: [zeromq-dev] HWM default

2011-05-09 Thread Pieter Hintjens
On Mon, May 9, 2011 at 8:26 AM, Martin Sustrik wrote: > The context options are a different issue. They are completely > orthogonal to what the default for HWM should be. Let's discuss them in > a separate thread. Not to hijack the thread, but the first step would be to fix the context API to ma

Re: [zeromq-dev] HWM default

2011-05-08 Thread Andrew Hume
again, this would be easy to diagnose if only there were a plausible (if not completely accurate) way to determine the number of queued messages. On May 8, 2011, at 11:26 PM, Martin Sustrik wrote: > Hi Blair, > >> On 9 May 2011 15:40, Martin Sustrik wrote: >>> It should be acknowledged though t

Re: [zeromq-dev] HWM default

2011-05-08 Thread Tobias Hintze
> +1 for Pieter's context config option. I think I agree having a > default HWM would be good, however, only if HWM events are easily > exposed. 0MQ could be very verbose if the HWM (set by "context default") is reached. that should make it obvious what is going on. th -- Tobias Hintze ___

Re: [zeromq-dev] HWM default

2011-05-08 Thread Martin Sustrik
Hi Blair, > On 9 May 2011 15:40, Martin Sustrik wrote: >> It should be acknowledged though that OOMs are easier to debug than >> deadlocks. However, debugging is not exactly newbie activity. It's >> likely to be performed by more experienced users which should be aware >> of an option of setting

Re: [zeromq-dev] HWM default

2011-05-08 Thread Blair Bethwaite
On 9 May 2011 15:40, Martin Sustrik wrote: > It should be acknowledged though that OOMs are easier to debug than > deadlocks. However, debugging is not exactly newbie activity. It's > likely to be performed by more experienced users which should be aware > of an option of setting HWM to infinity t

Re: [zeromq-dev] HWM default

2011-05-08 Thread Martin Sustrik
On 05/09/2011 03:49 AM, Paul Colomiets wrote: > I should second Andrew Hume. The HWM is really easy way to get deadlock > for novice. And if you are proficient with zeromq, you know when to set > HWM and which value suites your usage. > > Also should remind that deadlock probability was the reason

Re: [zeromq-dev] HWM default

2011-05-08 Thread Paul Colomiets
I should second Andrew Hume. The HWM is really easy way to get deadlock for novice. And if you are proficient with zeromq, you know when to set HWM and which value suites your usage. Also should remind that deadlock probability was the reason why my "shutdown" proposal was rejected. On Sun, May 8

Re: [zeromq-dev] HWM default

2011-05-08 Thread Joshua Foster
I would prefer that the HWM stay as the default. The application needs to be built with the HWM in mind. If they don't explicitly set the value, they are probably not considering it. There is also the challenge of knowing the proper size that the HWM needs to be set based on the size of the mess

Re: [zeromq-dev] HWM default

2011-05-08 Thread Andrew Hume
having fixed HWM also consititutes a stability threat, namely that of deadlocks. and these are often far harder to diagnose than gradually increasing memory footprints. if dos is your threat model, then i think you might need more stuff anyway. (such as autoshed when the HWM is reached) or somesu

Re: [zeromq-dev] HWM default

2011-05-08 Thread Pieter Hintjens
On Sun, May 8, 2011 at 10:34 AM, Tobias Hintze wrote: >> It's a good idea but IMO it should be done as a context configuration >> option. > > do you mean as a _default_ when no limit for the socket is specified? Yes. -Pieter ___ zeromq-dev mailing lis

Re: [zeromq-dev] HWM default

2011-05-08 Thread Tobias Hintze
> It's a good idea but IMO it should be done as a context configuration > option. > do you mean as a _default_ when no limit for the socket is specified? th -- Tobias Hintze ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.

Re: [zeromq-dev] HWM default

2011-05-08 Thread Pieter Hintjens
On Sun, May 8, 2011 at 8:15 AM, Martin Sustrik wrote: > I though of changing the default value for HWM to say 1000 in 0MQ/3.0. It's a good idea but IMO it should be done as a context configuration option. In czmq I've added the ability to configure a context (e.g. to specify the number of i/o t

Re: [zeromq-dev] HWM default

2011-05-08 Thread Martin Sustrik
On 05/08/2011 10:15 AM, Tobias Hintze wrote: >> I though of changing the default value for HWM to say 1000 in 0MQ/3.0. >> >> The rationale being that unlimited buffer is a stability threat. In >> other words, default socket when overloaded or DoS'ed will run out of >> memory and crash the applicati

Re: [zeromq-dev] HWM default

2011-05-08 Thread Tobias Hintze
> I though of changing the default value for HWM to say 1000 in 0MQ/3.0. > > The rationale being that unlimited buffer is a stability threat. In > other words, default socket when overloaded or DoS'ed will run out of > memory and crash the application rather than behave decently. a HWM measured i

[zeromq-dev] HWM default

2011-05-07 Thread Martin Sustrik
Hi all, I though of changing the default value for HWM to say 1000 in 0MQ/3.0. The rationale being that unlimited buffer is a stability threat. In other words, default socket when overloaded or DoS'ed will run out of memory and crash the application rather than behave decently. So, I think, we