Re: [zeromq-dev] A design question
That's why I said "reserve" not "commit" :)~ I vaguely recall this to also be the case with Windows. The problem with 32bit arch hosts is not the amount of memory commit, but finding enough 1-2MB contiguous holes in the address space below 2GB (Windows) or 3GB (Linux) to reserve. You could also potentially have additional issues if you made heavy use of TLS. On Thu, Dec 18, 2014 at 10:28 AM, Michel Pelletier < pelletier.mic...@gmail.com> wrote: > > Modern Linux doesn't actually map stack space into real memory unless the > thread accesses the virtual memory addresses on demand. I presume it's the > same for Windows. Garrett Smith pointed this out to me when I made the > same claim at the Chicago meetup a couple years ago. > > -Michel > > On Thu, Dec 18, 2014 at 8:16 AM, Thomas Rodgers > wrote: >> >> The biggest problem with 280, mostly idle, threads would be the amount of >> stack space the OS is required to reserve (1MB for Windows/2MB for Linux). >> On a 64bit host this is unlikely to be much of an issue. >> >> On Mon, Dec 15, 2014 at 6:11 AM, Pieter Hintjens wrote: >>> >>> 280 threads sounds fine unless you have data showing that it's a >>> problem. That's easy to test on your target platform. >>> >>> On Sun, Dec 14, 2014 at 7:54 AM, Bob Clarke >>> wrote: >>> > Platform: 0MQ 4.0.4 on Windows 7/Windows Server 2008 >>> > >>> > I am writing a server monitoring program to replace an ancient (1999) >>> > program that is almost impossible to maintain. >>> > >>> > All of our application servers host proprietary Windows services >>> written in >>> > C++ (except for a couple .Net apps) that use a proprietary synchronous >>> > request/response system (it works very well, has been for many years, >>> and >>> > isn't being replaced any time soon). >>> > >>> > The monitoring program sends test scripts to the server apps and waits >>> for a >>> > reply. In most cases, the reply is almost immediate, but in some cases, >>> > there could be a delay of up to 30 seconds. So, to avoid one script >>> causing >>> > delays with others, each test script runs in its own thread. This >>> thread >>> > must send its results back to a listener, so it creates and uses a 0MQ >>> > socket to do so. Each test script has a repeat interval; some are run >>> every >>> > five seconds, but most are run every 45 seconds, 90 seconds, or less >>> > frequently. >>> > >>> > There are about 280 test scripts (we have thousands of servers across >>> three >>> > data centers), so I seem to have two unfortunate choices: >>> > 1) Keep the script threads around so I'm not creating and destroying a >>> > socket each time a test runs; >>> > 2) Let the script threads terminate so I am not keeping 280 threads >>> open at >>> > once. Since the script repeat intervals are quite long in computer >>> terms, >>> > these threads will spend most of their time do nothing. >>> > >>> > I lean towards the first option, but opening and closing sockets that >>> often >>> > doesn't sound good. Then again, keeping 280+ threads around doesn't >>> sound so >>> > great, either. >>> > >>> > So before I go too far down this development road, does anyone have >>> some >>> > experience they could share? >>> > >>> > Thanks. >>> > >>> > Bob >>> > >>> > >>> > ___ >>> > zeromq-dev mailing list >>> > zeromq-dev@lists.zeromq.org >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> > >>> ___ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] A design question
Modern Linux doesn't actually map stack space into real memory unless the thread accesses the virtual memory addresses on demand. I presume it's the same for Windows. Garrett Smith pointed this out to me when I made the same claim at the Chicago meetup a couple years ago. -Michel On Thu, Dec 18, 2014 at 8:16 AM, Thomas Rodgers wrote: > > The biggest problem with 280, mostly idle, threads would be the amount of > stack space the OS is required to reserve (1MB for Windows/2MB for Linux). > On a 64bit host this is unlikely to be much of an issue. > > On Mon, Dec 15, 2014 at 6:11 AM, Pieter Hintjens wrote: >> >> 280 threads sounds fine unless you have data showing that it's a >> problem. That's easy to test on your target platform. >> >> On Sun, Dec 14, 2014 at 7:54 AM, Bob Clarke >> wrote: >> > Platform: 0MQ 4.0.4 on Windows 7/Windows Server 2008 >> > >> > I am writing a server monitoring program to replace an ancient (1999) >> > program that is almost impossible to maintain. >> > >> > All of our application servers host proprietary Windows services >> written in >> > C++ (except for a couple .Net apps) that use a proprietary synchronous >> > request/response system (it works very well, has been for many years, >> and >> > isn't being replaced any time soon). >> > >> > The monitoring program sends test scripts to the server apps and waits >> for a >> > reply. In most cases, the reply is almost immediate, but in some cases, >> > there could be a delay of up to 30 seconds. So, to avoid one script >> causing >> > delays with others, each test script runs in its own thread. This thread >> > must send its results back to a listener, so it creates and uses a 0MQ >> > socket to do so. Each test script has a repeat interval; some are run >> every >> > five seconds, but most are run every 45 seconds, 90 seconds, or less >> > frequently. >> > >> > There are about 280 test scripts (we have thousands of servers across >> three >> > data centers), so I seem to have two unfortunate choices: >> > 1) Keep the script threads around so I'm not creating and destroying a >> > socket each time a test runs; >> > 2) Let the script threads terminate so I am not keeping 280 threads >> open at >> > once. Since the script repeat intervals are quite long in computer >> terms, >> > these threads will spend most of their time do nothing. >> > >> > I lean towards the first option, but opening and closing sockets that >> often >> > doesn't sound good. Then again, keeping 280+ threads around doesn't >> sound so >> > great, either. >> > >> > So before I go too far down this development road, does anyone have some >> > experience they could share? >> > >> > Thanks. >> > >> > Bob >> > >> > >> > ___ >> > zeromq-dev mailing list >> > zeromq-dev@lists.zeromq.org >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] A design question
The biggest problem with 280, mostly idle, threads would be the amount of stack space the OS is required to reserve (1MB for Windows/2MB for Linux). On a 64bit host this is unlikely to be much of an issue. On Mon, Dec 15, 2014 at 6:11 AM, Pieter Hintjens wrote: > > 280 threads sounds fine unless you have data showing that it's a > problem. That's easy to test on your target platform. > > On Sun, Dec 14, 2014 at 7:54 AM, Bob Clarke wrote: > > Platform: 0MQ 4.0.4 on Windows 7/Windows Server 2008 > > > > I am writing a server monitoring program to replace an ancient (1999) > > program that is almost impossible to maintain. > > > > All of our application servers host proprietary Windows services written > in > > C++ (except for a couple .Net apps) that use a proprietary synchronous > > request/response system (it works very well, has been for many years, and > > isn't being replaced any time soon). > > > > The monitoring program sends test scripts to the server apps and waits > for a > > reply. In most cases, the reply is almost immediate, but in some cases, > > there could be a delay of up to 30 seconds. So, to avoid one script > causing > > delays with others, each test script runs in its own thread. This thread > > must send its results back to a listener, so it creates and uses a 0MQ > > socket to do so. Each test script has a repeat interval; some are run > every > > five seconds, but most are run every 45 seconds, 90 seconds, or less > > frequently. > > > > There are about 280 test scripts (we have thousands of servers across > three > > data centers), so I seem to have two unfortunate choices: > > 1) Keep the script threads around so I'm not creating and destroying a > > socket each time a test runs; > > 2) Let the script threads terminate so I am not keeping 280 threads open > at > > once. Since the script repeat intervals are quite long in computer terms, > > these threads will spend most of their time do nothing. > > > > I lean towards the first option, but opening and closing sockets that > often > > doesn't sound good. Then again, keeping 280+ threads around doesn't > sound so > > great, either. > > > > So before I go too far down this development road, does anyone have some > > experience they could share? > > > > Thanks. > > > > Bob > > > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] A design question
280 threads sounds fine unless you have data showing that it's a problem. That's easy to test on your target platform. On Sun, Dec 14, 2014 at 7:54 AM, Bob Clarke wrote: > Platform: 0MQ 4.0.4 on Windows 7/Windows Server 2008 > > I am writing a server monitoring program to replace an ancient (1999) > program that is almost impossible to maintain. > > All of our application servers host proprietary Windows services written in > C++ (except for a couple .Net apps) that use a proprietary synchronous > request/response system (it works very well, has been for many years, and > isn't being replaced any time soon). > > The monitoring program sends test scripts to the server apps and waits for a > reply. In most cases, the reply is almost immediate, but in some cases, > there could be a delay of up to 30 seconds. So, to avoid one script causing > delays with others, each test script runs in its own thread. This thread > must send its results back to a listener, so it creates and uses a 0MQ > socket to do so. Each test script has a repeat interval; some are run every > five seconds, but most are run every 45 seconds, 90 seconds, or less > frequently. > > There are about 280 test scripts (we have thousands of servers across three > data centers), so I seem to have two unfortunate choices: > 1) Keep the script threads around so I'm not creating and destroying a > socket each time a test runs; > 2) Let the script threads terminate so I am not keeping 280 threads open at > once. Since the script repeat intervals are quite long in computer terms, > these threads will spend most of their time do nothing. > > I lean towards the first option, but opening and closing sockets that often > doesn't sound good. Then again, keeping 280+ threads around doesn't sound so > great, either. > > So before I go too far down this development road, does anyone have some > experience they could share? > > Thanks. > > Bob > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] A design question
Platform: 0MQ 4.0.4 on Windows 7/Windows Server 2008 I am writing a server monitoring program to replace an ancient (1999) program that is almost impossible to maintain. All of our application servers host proprietary Windows services written in C++ (except for a couple .Net apps) that use a proprietary synchronous request/response system (it works very well, has been for many years, and isn't being replaced any time soon). The monitoring program sends test scripts to the server apps and waits for a reply. In most cases, the reply is almost immediate, but in some cases, there could be a delay of up to 30 seconds. So, to avoid one script causing delays with others, each test script runs in its own thread. This thread must send its results back to a listener, so it creates and uses a 0MQ socket to do so. Each test script has a repeat interval; some are run every five seconds, but most are run every 45 seconds, 90 seconds, or less frequently. There are about 280 test scripts (we have thousands of servers across three data centers), so I seem to have two unfortunate choices: 1) Keep the script threads around so I'm not creating and destroying a socket each time a test runs; 2) Let the script threads terminate so I am not keeping 280 threads open at once. Since the script repeat intervals are quite long in computer terms, these threads will spend most of their time do nothing. I lean towards the first option, but opening and closing sockets that often doesn't sound good. Then again, keeping 280+ threads around doesn't sound so great, either. So before I go too far down this development road, does anyone have some experience they could share? Thanks. Bob ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
