[nodejs] Problem with REST api call
I m using https://github.com/danwrong/restler for pastebin REST client.
Here is my code :
*var sys = require('util'),*
*rest = require('restler');*
*var key = my key;*
*var paste = test from nodejs;*
*var apipas = paste;*
*var name = hirenTest ;*
*var hiren = new Object();*
* hiren.api_dev_key =key,*
* hiren.api_option = apipas,*
* hiren.api_paste_code = paste,*
*rest.post('http://pastebin.com/api/api_post.php', {*
* // method : POST ,*
* query : hiren*
*}).on('complete', function(data) {*
* console.log(data);*
*});*
Everytime I run this code it shows Bad API request, invalid api_option .
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[nodejs] Memory Usage and disparity between RSS and Heap size
Hi, I'm trying to debug and understand an issue on several of my Haraka servers after a recent update. I'm seeing large amounts of memory being used and never returned back to the operating system causing the machine to swap and eventually the Linux OOM killer shooting the processes. I thought that there must be a memory leak somewhere - but I've used node-heapdump and taken multiple snapshots and compared them and whilst I'm seeing a small growth I'm not seeing anything major.However the heapdump reports the size at ~97Mb whereas the RSS of the process at the time of the dump was 1.1Gb which is what I really don't understand at all. Would such a disparity between heap and RSS indicate that an external C++ module that is being used is leaking memory?? If so, what is the best way of finding out which? I've tried the mtrace module - but it caused the process to completely hang after only a few minutes of running so I had to kill it. The output that I got from it is attached - I'm not really sure it tells me anything though. Any suggestions as to how to work out what is going on here and how to fix it would be gratefully received. Kind regards, Steve. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. unrecognized type on line 156: antEiNS0_4Code4KindEiNS0_6HandleINS0_6StringEEENS4_INS0_6ObjectEEENS4_INS0_8JSObjectEEENS4_INS0_10JSFunctionEEE+0x26d)[0x89f6bd] - 0x1f7bccc0 regex fail on line 1375478: @ /lib64/libc.so.6:(__nss_lookup_function+0x parsed 1375478 lines AddrSize Count Traffic Module Symbol -- - - --- 0x30220f455a0 0 2/lib64/libc.so.6 getifaddrs 0x3022097b060 0 2/lib64/libc.so.6 getgrnam 0x30220dfdf20 0 2/lib64/libc.so.6 argp_parse 0x3022026ef80 0 2/lib64/libc.so.6 new_composite_name 0x302202853d0 0 2/lib64/libc.so.6 _nl_load_locale_from_archive 0x302202d97e0 0 2/lib64/libc.so.6 _nl_normalize_codeset 0x5e87a70 0 2/usr/bin/node uv_cpu_info 0x5e85f10 0 2/usr/bin/node uv_interface_addresses 0x2b52db4c81e7 0 0 2/usr/lib/node_modules/Haraka/... sdsnewlen 0x5d8dff0 0 4/usr/bin/node eio_fstat 0x2b52da5d8dac 0 0 4/usr/lib/node_modules/Haraka/... (anonymous namespace)::Iconv::Convert(char*, ... 0x5d99e30 0 4/usr/bin/node eio_open 0x5d90cf0 0 4/usr/bin/node eio_close 0x5d8f640 0 4/usr/bin/node eio_read 0x5d81e20 0 4/usr/bin/node No symbol 0x30220617be0 0 4/lib64/libc.so.6 getdelim 0x30220bc16b0 0 4/lib64/libc.so.6 gaih_inet 0x2b52da5d884d 0 0 4/usr/lib/node_modules/Haraka/... libiconv_open 0x5d20c50 0 6Haraka (worker) cn=1203 cc=10... ares_parse_ptr_reply 0x8dae6b0 0 6Haraka (worker) cn=5 cc=5 cps... v8::internal::Deoptimizer::DoComputeOsrOutput...
[nodejs] Re: Problem with REST api call
you are posting, your hiren object should go as data not as query. just an
asumption.
Am Freitag, 22. November 2013 11:35:07 UTC+1 schrieb Ashutosh Das:
I m using
https://github.com/danwrong/restlerhttps://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fdanwrong%2Frestlersa=Dsntz=1usg=AFQjCNFRbeZm0P47APp-N9NWD7hT6eIisw
for pastebin REST client.
Here is my code :
*var sys = require('util'),*
*rest = require('restler');*
*var key = my key;*
*var paste = test from nodejs;*
*var apipas = paste;*
*var name = hirenTest ;*
*var hiren = new Object();*
* hiren.api_dev_key =key,*
* hiren.api_option = apipas,*
* hiren.api_paste_code = paste,*
*rest.post('http://pastebin.com/api/api_post.php
http://www.google.com/url?q=http%3A%2F%2Fpastebin.com%2Fapi%2Fapi_post.phpsa=Dsntz=1usg=AFQjCNFl2mIaeDgFAbaWRWZzezhlVX82Sw',
{*
* // method : POST ,*
* query : hiren*
*}).on('complete', function(data) {*
* console.log(data);*
*});*
Everytime I run this code it shows Bad API request, invalid api_option .
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[nodejs] Re: Problem with REST api call
se also the examples in the readme on github. the post example uses data
Am Freitag, 22. November 2013 11:35:07 UTC+1 schrieb Ashutosh Das:
I m using
https://github.com/danwrong/restlerhttps://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fdanwrong%2Frestlersa=Dsntz=1usg=AFQjCNFRbeZm0P47APp-N9NWD7hT6eIisw
for pastebin REST client.
Here is my code :
*var sys = require('util'),*
*rest = require('restler');*
*var key = my key;*
*var paste = test from nodejs;*
*var apipas = paste;*
*var name = hirenTest ;*
*var hiren = new Object();*
* hiren.api_dev_key =key,*
* hiren.api_option = apipas,*
* hiren.api_paste_code = paste,*
*rest.post('http://pastebin.com/api/api_post.php
http://www.google.com/url?q=http%3A%2F%2Fpastebin.com%2Fapi%2Fapi_post.phpsa=Dsntz=1usg=AFQjCNFl2mIaeDgFAbaWRWZzezhlVX82Sw',
{*
* // method : POST ,*
* query : hiren*
*}).on('complete', function(data) {*
* console.log(data);*
*});*
Everytime I run this code it shows Bad API request, invalid api_option .
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Re: Problem with REST api call
So how can i pass parameter ?
On Fri, Nov 22, 2013 at 7:18 PM, greelgorke greelgo...@gmail.com wrote:
se also the examples in the readme on github. the post example uses data
Am Freitag, 22. November 2013 11:35:07 UTC+1 schrieb Ashutosh Das:
I m using
https://github.com/danwrong/restlerhttps://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fdanwrong%2Frestlersa=Dsntz=1usg=AFQjCNFRbeZm0P47APp-N9NWD7hT6eIisw
for pastebin REST client.
Here is my code :
*var sys = require('util'),*
*rest = require('restler');*
*var key = my key;*
*var paste = test from nodejs;*
*var apipas = paste;*
*var name = hirenTest ;*
*var hiren = new Object();*
* hiren.api_dev_key =key,*
* hiren.api_option = apipas,*
* hiren.api_paste_code = paste,*
*rest.post('http://pastebin.com/api/api_post.php
http://www.google.com/url?q=http%3A%2F%2Fpastebin.com%2Fapi%2Fapi_post.phpsa=Dsntz=1usg=AFQjCNFl2mIaeDgFAbaWRWZzezhlVX82Sw',
{*
* // method : POST ,*
* query : hiren*
*}).on('complete', function(data) {*
* console.log(data);*
*});*
Everytime I run this code it shows Bad API request, invalid api_option
.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to a topic in the
Google Groups nodejs group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/nodejs/sqRGIThNud4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Re: Problem with REST api call
thanks its working now :)
On Fri, Nov 22, 2013 at 7:21 PM, Ashutosh Das areos...@gmail.com wrote:
So how can i pass parameter ?
On Fri, Nov 22, 2013 at 7:18 PM, greelgorke greelgo...@gmail.com wrote:
se also the examples in the readme on github. the post example uses data
Am Freitag, 22. November 2013 11:35:07 UTC+1 schrieb Ashutosh Das:
I m using
https://github.com/danwrong/restlerhttps://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fdanwrong%2Frestlersa=Dsntz=1usg=AFQjCNFRbeZm0P47APp-N9NWD7hT6eIisw
for pastebin REST client.
Here is my code :
*var sys = require('util'),*
*rest = require('restler');*
*var key = my key;*
*var paste = test from nodejs;*
*var apipas = paste;*
*var name = hirenTest ;*
*var hiren = new Object();*
* hiren.api_dev_key =key,*
* hiren.api_option = apipas,*
* hiren.api_paste_code = paste,*
*rest.post('http://pastebin.com/api/api_post.php
http://www.google.com/url?q=http%3A%2F%2Fpastebin.com%2Fapi%2Fapi_post.phpsa=Dsntz=1usg=AFQjCNFl2mIaeDgFAbaWRWZzezhlVX82Sw',
{*
* // method : POST ,*
* query : hiren*
*}).on('complete', function(data) {*
* console.log(data);*
*});*
Everytime I run this code it shows Bad API request, invalid
api_option .
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to a topic in the
Google Groups nodejs group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/nodejs/sqRGIThNud4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Node.JS for sysadmin scripting or non-web stuff - books, guides, examples?
On Thu, Nov 21, 2013 at 11:39 PM, // ravi ravi-li...@g8o.net wrote: Is Node.JS suitable for these things? Sure, JavaScript is a capable and feature-rich *high-level* language. NodeJS provides decent access to OS facilities, and NPM is a reliable source of additional functionality (even if it is no CPAN, or not yet CPAN). But I’d say, stepping strictly into personal opinion or preference territory, JavaScript (or Python for that matter) will never be as comfortable for sys-admin tasks as Perl. If having to carry the cognitive load of thorough-going async for inherently synchronous activities does not tax you, then the lack of Perl’s terse and intuitive (for a Unix-head) syntax for things like file test (-f, -l, etc), pipes (“|”), regular expressions, program execution (``), etc, might. Even if they do, it might be a small price to pay *if* the rest of your code is JavaScript/NodeJS. I absolutely 100% concur. I've written things that are very sysadmin scripting like in Node and realistically they would have been far neater in Perl. It's just better designed for that kind of thing. Having said that, if you don't know Perl, the sigils ($/@/% in front of all the variables) and special variables will look really annoying. Matt. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[nodejs][ANN] UserDown - A way to stepdown user privilege for node apps
Hi, Sometimes we need to bind node apps to port 80. Some apps(tools) does not implement proper stepdown mechanism via *process.setuid*. In these cases, we need use port forwarding or some other work around. *Enter UserDown* *userdown *takes care of this issue and it can stepdown any Node app. See usage. userdown USERDOWN_UID=nobody USERDOWN_GID=nobody examples/port80server.js See for more Info: UserDown on GitHubhttps://github.com/arunoda/node-userdown Hope this helps. -- Arunoda Susiripala @arunoda http://twitter.com/arunoda http://gplus.to/arunodahttps://github.com/arunoda http://www.linkedin.com/in/arunoda -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[nodejs] Re: How does round-robin clustering work in Node?
I have a question that is not immediately obvious after reading these two links. How well does Node's round robin work with requests that produce significantly varied workloads on the server? I don't mean, how well does the round robin algorithm work with varies workloads, since I already know the answer to that... what I mean is, is Node smart enough to skip workers that are still busy from the last round robin request sent to them? In other words, if a worker is busy on a CPU bound function, will it be skipped until it gets freed up again? We have a development platform that is similar to Meteor ( http://www.meteor.com/ ) in that code transparently runs on the client or server. All processing is done in JavaScript, which means some functions are way more CPU bound than others - and the creative developers using this platform use it for all sorts of things (data processing, image processing, DB access, etc). What we like about the current clustering is that a request is not sent to a worker that is spinning away on some CPU bound function. Yes, the current approach does tend to send requests to the same workers when they are not busy, so round robin would be awesome to fix that issue... but is the current round robin smart enough to skip workers that are busy? Thanks! On Wednesday, November 20, 2013 12:37:47 PM UTC-6, Jimmy Guerrero wrote: In addition to the official docs here: http://nodejs.org/api/all.html#all_clusterhttp://www.google.com/url?q=http%3A%2F%2Fnodejs.org%2Fapi%2Fall.html%23all_clustersa=Dsntz=1usg=AFQjCNEXWleL_J373P9tUWeKISPRVeDuLQ Ben Noordhuis has written up a recap of how clustering in Node works along with an intro to the new round-robin clustering available starting in v0.11.2. http://strongloop.com/strongblog/whats-new-in-node-js-v0-12-cluster-round-robin-load-balancing/http://www.google.com/url?q=http%3A%2F%2Fstrongloop.com%2Fstrongblog%2Fwhats-new-in-node-js-v0-12-cluster-round-robin-load-balancing%2Fsa=Dsntz=1usg=AFQjCNFYQzi4MdPUsoS2jyh3MIARWisvqg -- Jimmy -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[nodejs] Re: Node.JS for sysadmin scripting or non-web stuff - books, guides, examples?
I'm curious on people's experiences with using Node.JS for sys-admin scripts or non-Web glue stuff - i.e. the sorts of things you might use Perl or Python for (e.g. log parsing, manipulating config files etc.) Is Node.JS suitable for these things? The main hurdle is node's asynchronous paradigm that doesn't exactly lend itself to script logic. E.g. running external commands and reuse their output isn't as easy as it should be. Most of the time there are npm packages to make life easier (for the moment) but on the long run you'd better learn to write your scripts in an async manner -- if you want to take that road. So far, I like the async scripts I've written in node, which I would have otherwise written in ruby. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Re: How does round-robin clustering work in Node?
On Fri, Nov 22, 2013 at 3:30 PM, Andrew D depu...@gmail.com wrote: I have a question that is not immediately obvious after reading these two links. How well does Node's round robin work with requests that produce significantly varied workloads on the server? I don't mean, how well does the round robin algorithm work with varies workloads, since I already know the answer to that... what I mean is, is Node smart enough to skip workers that are still busy from the last round robin request sent to them? In other words, if a worker is busy on a CPU bound function, will it be skipped until it gets freed up again? We have a development platform that is similar to Meteor ( http://www.meteor.com/ ) in that code transparently runs on the client or server. All processing is done in JavaScript, which means some functions are way more CPU bound than others - and the creative developers using this platform use it for all sorts of things (data processing, image processing, DB access, etc). What we like about the current clustering is that a request is not sent to a worker that is spinning away on some CPU bound function. Yes, the current approach does tend to send requests to the same workers when they are not busy, so round robin would be awesome to fix that issue... but is the current round robin smart enough to skip workers that are busy? The answer to that question is mostly 'no'. The master won't send new connections to workers that haven't acknowledged receipt of the last one yet but, apart from that, it doesn't really know whether a worker is busy or not. We'll undoubtedly address that in due time but that requires considerable engineering effort. Naively sending I'm busy / I'm idle messages back and forth whenever user code is executed leads to serious performance degradation because you quickly hit a point where you spend all your CPU time processing messages and not doing anything useful. There are solutions to that issue (mostly involving tricks with shared memory) but none are easy to implement in a cross-platform manner. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Memory Usage and disparity between RSS and Heap size
On Fri, Nov 22, 2013 at 12:37 PM, Steve Freegard st...@stevefreegard.com wrote: Hi, I'm trying to debug and understand an issue on several of my Haraka servers after a recent update. I'm seeing large amounts of memory being used and never returned back to the operating system causing the machine to swap and eventually the Linux OOM killer shooting the processes. I thought that there must be a memory leak somewhere - but I've used node-heapdump and taken multiple snapshots and compared them and whilst I'm seeing a small growth I'm not seeing anything major.However the heapdump reports the size at ~97Mb whereas the RSS of the process at the time of the dump was 1.1Gb which is what I really don't understand at all. Would such a disparity between heap and RSS indicate that an external C++ module that is being used is leaking memory?? If so, what is the best way of finding out which? I've tried the mtrace module - but it caused the process to completely hang after only a few minutes of running so I had to kill it. The output that I got from it is attached - I'm not really sure it tells me anything though. Any suggestions as to how to work out what is going on here and how to fix it would be gratefully received. Kind regards, Steve. Tracking down memory leaks in garbage-collected environments is one of the great unsolved problems of our generation. Okay, maybe not really but there is no one-size-fits-all solution. Here are some things you can try: * The mtrace log suggests you're using node.js v0.8. Consider upgrading. * Use Valgrind. Drawback: it's really, really slow and it uses a lot of memory itself. * SystemTap. Write a small stap script that collects malloc/free/etc. call graphs and aggregates them into histograms. Strong disparities between calls to malloc() and free() generally show up quickly that way. Apropos that heap snapshot, did you look at the self size or the retained size? Buffer objects tend to have small self sizes (usually on the order of a few hundred bytes) but can have massive retained sizes. Note that Buffers in snapshots taken with node.js = v0.8.11 don't report the true retained size. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Re: how to create a singleton
I'm not 100% sure of your requirements, but this is what I assume you're
after:
-
file : soapClient.js
var memoizedClient;
var soap = require('soap');
var create = function (cb){
if (memoizedClient){
return cb(null, memoizedClient);
}
var url = http://172.31.19.39/MgmtServer.wsdl;;
var endPoint = https://172.31.19.39:9088;;
var options = {};
options.endpoint = endPoint;
soap.createClient(url, options, function(err, result) {
if (err) {
return cb(null, memoizedClient); // pass errors properly! :)
}
setSecurity(new soap.BasicAuthSecurity('admin-priv', 'password'));
memoizedClient = result;
cb(null, result);
});
exports.create = create;
exports.client = memoizedClient; // I don't do this, but I think it's what
you're looking for
-
file: someClientCode.js
var client;
require('soapClient').create(function(err, client){
// do stuff here
});
-
ALSO:
If you're somehow sure that you've already called create() previously, you
could just do:
var client = require('soapClient').client;
// do stuff here
I know that's a hell of a lot prettier, but I usually don't do that because
I like to not
have to worry about whether i've called create() or not. I just call
create() everytime, and
suck up the fact that there's a callback. In the cases that I'd use this
pattern like you are
using it, it's ONLY because I'd like to not have to incur the network cost
of the asynch call
everywhere I use it, and not because I want fewer indentations in my code.
I do this with
database connection pools as well, for example, because those shouldn't be
created over
and over for every use. So in short, ALL of this hinges on my assumption
that
soap.createClient involves a network call that you don't want to repeat
everytime you
want to make a soap request. Otherwise, there's no good reason to memoize
(cache state).
Often I put a setter on the memoizedClient as well because it works nicely
for dependency
injection in unit tests.
S... I think I have to admit in retrospect that this is indeed a
singleton. At least this
way you can be more idiomatic than any general javascript singleton code
I've seen.
(I never ran any of this code, so it's probably broken in 14 ways)
Hope that helps,
G
On Thu, Nov 21, 2013 at 9:03 AM, Reza Razavipour
reza.razavip...@gmail.comwrote:
Ok so this is the way i have it constructed... SoapClient is my module and
I want it to be used as a singleton, i.e. if it is initialized, it just
returns the connection, or SoapClient.instance...
var SoapClient = function() {
var soap = require('soap');
this.init = function (){
var url = http://172.31.19.39/MgmtServer.wsdl;;
var endPoint = https://172.31.19.39:9088;;
var options = {};
options.endpoint = endPoint;
soap.createClient(url, options, function(err, result) {
if (err) {
console.log('soap client create failed ' + err);
return;
}
console.log('init is called ready');
SoapClient.instance = result;
SoapClient.instance.setSecurity(new soap.BasicAuthSecurity(
'admin-priv', 'password'));
});
};
};
SoapClient.getInstance = function () {
if (SoapClient.instance) {
return SoapClient.instance;
}
new SoapClient().init();
return SoapClient.instance;
};
SoapClient.instance = null;
module.exports = SoapClient;
Thoughts
On Thursday, November 21, 2013 8:34:14 AM UTC-8, Reza Razavipour wrote:
this I get the jist of the conversation but my Javascript skills, less
than 2 months, does not allow me to be able to code this up.
Also, as a starter in the new language the last thing I want to do is to
miss out on the language correct way of things and force lets say the Java
way of doing things in JS.
Can you show me a skeleton of such function and a tiny consumer of that?
The reason I say that is the fact that all of the examples are all done in
the same JS file and not setup as module and when I try to change to a
module and a consumer, I run into syntax problems and
I do not understand the difference between a class exporting an object as
opposed to a constructor
Thank you so much for showing me the correct way
On Thursday, November 21, 2013 7:37:32 AM UTC-8, Kamil Leszczuk wrote:
Aah, nevermind then, I misunerstood ;)
21 lis 2013 16:36 Gregg Caines gr...@caines.ca napisał(a):
Yeah... that's what I'm saying :)
G
On Thu, Nov 21, 2013 at 7:33 AM, Kamil Leszczuk kami...@gmail.comwrote:
For example, if I have a module for emailing with a send() method on
it, I don't have it export a constructor; I have it export an object.
That
object might maintain some state or it might not. When the module is
subsequently require()'d, it will have any state that it has accumulated
since.
For.most of the time, that's unnecessary - multiple require() calls
for the same module return same, cached module, so you can store state
Re: [nodejs] Re: How does round-robin clustering work in Node?
I think that works for us at the moment, as it seems comparable with the behavior of the current implementation, which, while it doesn't really know how busy a worker is, also won't receive requests until the worker has a chance to hit the event loop again. On Friday, November 22, 2013 10:12:41 AM UTC-6, Ben Noordhuis wrote: On Fri, Nov 22, 2013 at 3:30 PM, Andrew D dep...@gmail.com javascript: wrote: I have a question that is not immediately obvious after reading these two links. How well does Node's round robin work with requests that produce significantly varied workloads on the server? I don't mean, how well does the round robin algorithm work with varies workloads, since I already know the answer to that... what I mean is, is Node smart enough to skip workers that are still busy from the last round robin request sent to them? In other words, if a worker is busy on a CPU bound function, will it be skipped until it gets freed up again? We have a development platform that is similar to Meteor ( http://www.meteor.com/http://www.google.com/url?q=http%3A%2F%2Fwww.meteor.com%2Fsa=Dsntz=1usg=AFQjCNFsYVXZEstozl04pf0AN_kdnHh4dA) in that code transparently runs on the client or server. All processing is done in JavaScript, which means some functions are way more CPU bound than others - and the creative developers using this platform use it for all sorts of things (data processing, image processing, DB access, etc). What we like about the current clustering is that a request is not sent to a worker that is spinning away on some CPU bound function. Yes, the current approach does tend to send requests to the same workers when they are not busy, so round robin would be awesome to fix that issue... but is the current round robin smart enough to skip workers that are busy? The answer to that question is mostly 'no'. The master won't send new connections to workers that haven't acknowledged receipt of the last one yet but, apart from that, it doesn't really know whether a worker is busy or not. We'll undoubtedly address that in due time but that requires considerable engineering effort. Naively sending I'm busy / I'm idle messages back and forth whenever user code is executed leads to serious performance degradation because you quickly hit a point where you spend all your CPU time processing messages and not doing anything useful. There are solutions to that issue (mostly involving tricks with shared memory) but none are easy to implement in a cross-platform manner. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[nodejs] Re: how to create a singleton
thanks for the response. I will have to 'digest' it but I get the idea you are laying out. My original intent is for the consumer of this object is to use the client synchronously on demand, as opposed to async. On Monday, November 18, 2013 3:38:09 PM UTC-8, Reza Razavipour wrote: A newbie question... I have an app that connects and reuses the same connection to a remote database and a connection to a remote soap server. I want to implement a singleton pattern for each of these. I am used to doing that in C++ and Java but want to know what the standard implementation for a Singleton pattern is in node.js. Any recommendations or references. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[nodejs] difference between a module and a class
I am not clear what the difference is referring to a module and a generic class/function that i create and use. Can someone care to explain -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Re: Node.JS for sysadmin scripting or non-web stuff - books, guides, examples?
running external commands and reuse their output isn't as easy as it
should be.
How hard is it to do exec rm -rf /
or if you are coffeescript-impaired ...
exec('rm -rf /');
On Fri, Nov 22, 2013 at 6:33 AM, lith minil...@gmail.com wrote:
I'm curious on people's experiences with using Node.JS for sys-admin
scripts or non-Web glue stuff - i.e. the sorts of things you might use Perl
or Python for (e.g. log parsing, manipulating config files etc.)
Is Node.JS suitable for these things?
The main hurdle is node's asynchronous paradigm that doesn't exactly lend
itself to script logic. E.g. running external commands and reuse their
output isn't as easy as it should be. Most of the time there are npm
packages to make life easier (for the moment) but on the long run you'd
better learn to write your scripts in an async manner -- if you want to
take that road. So far, I like the async scripts I've written in node,
which I would have otherwise written in ruby.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an
email to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Re: Node.JS for sysadmin scripting or non-web stuff - books, guides, examples?
On Fri, Nov 22, 2013 at 1:45 PM, Mark Hahn m...@reevuit.com wrote:
How hard is it to do exec rm -rf /
or if you are coffeescript-impaired ...
exec('rm -rf /');
The problem is that's executed asynchronously, so you have to implement a
callback to catch errors and output. This gets hairy fast when just writing
a simple sysadmin script.
The node guys have promised us an execSync soon, but I'm not sure if it's
showing up in 0.12 or not.
Matt.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Memory Usage and disparity between RSS and Heap size
For what it's worth, I just went through an arduous process of tracking
down a memory leak for walmart, you can read about it
http://www.joyent.com/blog/walmart-node-js-memory-leak you could use the
techniques described there to help narrow down your issues. There are other
pieces of information available at
http://www.joyent.com/developers/node/debug#mdb
I will try and do a quick analysis for you, but if you have issues you can
hit me and others up on irc #node.js on freenode
On Fri, Nov 22, 2013 at 9:27 AM, Steve Freegard st...@stevefreegard.comwrote:
Hi Ben,
I managed to find the issue in Haraka - we were calling
fs.unwatchFile/fs.watchFile instead of consulting our configuration
cache which we should have been doing.
However - this has pointed us to a rather large memory leak in the
fs.unwatchFile and fs.watchFile functions. I've tested it on both 0.8
and 0.10. This is on 0.10:
[root@mta41 ~]# node --expose_gc watchfile_leak.js
{ rss: 8478720, heapTotal: 6163968, heapUsed: 1918824 }
{ rss: 827813888, heapTotal: 13387520, heapUsed: 4990520 }
Testcase is attached.
Kind regards,
Steve.
On 22/11/13 16:25, Ben Noordhuis wrote:
On Fri, Nov 22, 2013 at 12:37 PM, Steve Freegard
st...@stevefreegard.com wrote:
Hi,
I'm trying to debug and understand an issue on several of my Haraka
servers
after a recent update.
I'm seeing large amounts of memory being used and never returned back
to the
operating system causing the machine to swap and eventually the Linux
OOM
killer shooting the processes.
I thought that there must be a memory leak somewhere - but I've used
node-heapdump and taken multiple snapshots and compared them and whilst
I'm
seeing a small growth I'm not seeing anything major.However the
heapdump
reports the size at ~97Mb whereas the RSS of the process at the time of
the
dump was 1.1Gb which is what I really don't understand at all.
Would such a disparity between heap and RSS indicate that an external
C++
module that is being used is leaking memory?? If so, what is the best
way
of finding out which?
I've tried the mtrace module - but it caused the process to completely
hang
after only a few minutes of running so I had to kill it. The output
that I
got from it is attached - I'm not really sure it tells me anything
though.
Any suggestions as to how to work out what is going on here and how to
fix
it would be gratefully received.
Kind regards,
Steve.
Tracking down memory leaks in garbage-collected environments is one of
the great unsolved problems of our generation. Okay, maybe not really
but there is no one-size-fits-all solution. Here are some things you
can try:
* The mtrace log suggests you're using node.js v0.8. Consider upgrading.
* Use Valgrind. Drawback: it's really, really slow and it uses a lot
of memory itself.
* SystemTap. Write a small stap script that collects malloc/free/etc.
call graphs and aggregates them into histograms. Strong disparities
between calls to malloc() and free() generally show up quickly that
way.
Apropos that heap snapshot, did you look at the self size or the
retained size? Buffer objects tend to have small self sizes (usually
on the order of a few hundred bytes) but can have massive retained
sizes. Note that Buffers in snapshots taken with node.js = v0.8.11
don't report the true retained size.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an
email to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Memory Usage and disparity between RSS and Heap size
so you're creating a bunch of stat watchers, those require gc to be cleaned
up, but they're still too young by the time you run gc outside the loop, if
instead you move the gc into the loop or call it multiple times after the
loop is done it can effectively clean them up.
On Fri, Nov 22, 2013 at 11:58 AM, Timothy J Fontaine
tjfonta...@gmail.comwrote:
For what it's worth, I just went through an arduous process of tracking
down a memory leak for walmart, you can read about it
http://www.joyent.com/blog/walmart-node-js-memory-leak you could use the
techniques described there to help narrow down your issues. There are other
pieces of information available at
http://www.joyent.com/developers/node/debug#mdb
I will try and do a quick analysis for you, but if you have issues you can
hit me and others up on irc #node.js on freenode
On Fri, Nov 22, 2013 at 9:27 AM, Steve Freegard
st...@stevefreegard.comwrote:
Hi Ben,
I managed to find the issue in Haraka - we were calling
fs.unwatchFile/fs.watchFile instead of consulting our configuration
cache which we should have been doing.
However - this has pointed us to a rather large memory leak in the
fs.unwatchFile and fs.watchFile functions. I've tested it on both 0.8
and 0.10. This is on 0.10:
[root@mta41 ~]# node --expose_gc watchfile_leak.js
{ rss: 8478720, heapTotal: 6163968, heapUsed: 1918824 }
{ rss: 827813888, heapTotal: 13387520, heapUsed: 4990520 }
Testcase is attached.
Kind regards,
Steve.
On 22/11/13 16:25, Ben Noordhuis wrote:
On Fri, Nov 22, 2013 at 12:37 PM, Steve Freegard
st...@stevefreegard.com wrote:
Hi,
I'm trying to debug and understand an issue on several of my Haraka
servers
after a recent update.
I'm seeing large amounts of memory being used and never returned back
to the
operating system causing the machine to swap and eventually the Linux
OOM
killer shooting the processes.
I thought that there must be a memory leak somewhere - but I've used
node-heapdump and taken multiple snapshots and compared them and
whilst I'm
seeing a small growth I'm not seeing anything major.However the
heapdump
reports the size at ~97Mb whereas the RSS of the process at the time
of the
dump was 1.1Gb which is what I really don't understand at all.
Would such a disparity between heap and RSS indicate that an external
C++
module that is being used is leaking memory?? If so, what is the best
way
of finding out which?
I've tried the mtrace module - but it caused the process to completely
hang
after only a few minutes of running so I had to kill it. The output
that I
got from it is attached - I'm not really sure it tells me anything
though.
Any suggestions as to how to work out what is going on here and how to
fix
it would be gratefully received.
Kind regards,
Steve.
Tracking down memory leaks in garbage-collected environments is one of
the great unsolved problems of our generation. Okay, maybe not really
but there is no one-size-fits-all solution. Here are some things you
can try:
* The mtrace log suggests you're using node.js v0.8. Consider
upgrading.
* Use Valgrind. Drawback: it's really, really slow and it uses a lot
of memory itself.
* SystemTap. Write a small stap script that collects malloc/free/etc.
call graphs and aggregates them into histograms. Strong disparities
between calls to malloc() and free() generally show up quickly that
way.
Apropos that heap snapshot, did you look at the self size or the
retained size? Buffer objects tend to have small self sizes (usually
on the order of a few hundred bytes) but can have massive retained
sizes. Note that Buffers in snapshots taken with node.js = v0.8.11
don't report the true retained size.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an
email to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
Re: [nodejs] Memory Usage and disparity between RSS and Heap size
On Fri, Nov 22, 2013 at 4:14 PM, Timothy J Fontaine tjfonta...@gmail.comwrote: so you're creating a bunch of stat watchers, those require gc to be cleaned up, but they're still too young by the time you run gc outside the loop, if instead you move the gc into the loop or call it multiple times after the loop is done it can effectively clean them up. Doesn't necessarily explain the problem in Haraka though (albeit that was with the deprecated watchFile/unwatchFile API), which was running long term and leaking. Not saying it couldn't be some other cause in Haraka though, and we've solved that issue now, but I think this could probably use some more thorough testing. Matt. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Memory Usage and disparity between RSS and Heap size
On Fri, Nov 22, 2013 at 11:00 PM, Steve Freegard
st...@stevefreegard.com wrote:
On 22/11/13 21:14, Timothy J Fontaine wrote:
so you're creating a bunch of stat watchers, those require gc to be
cleaned up, but they're still too young by the time you run gc outside
the loop, if instead you move the gc into the loop or call it multiple
times after the loop is done it can effectively clean them up.
Ok - I tried that and moved the gc() call to the end of the loop and for
good measure added a couple outside the loop before outputting the
memory usage. I had to reduce the number of loops to 10,000 so that it
didn't take forever to complete; but it still shows a leak:
[root@mta41 ~]# node --expose_gc watchfile_leak.js
{ rss: 8478720, heapTotal: 6163968, heapUsed: 1919048 }
{ rss: 17838080, heapTotal: 20844032, heapUsed: 1928208 }
I also switched from fs.watchFile to fs.watch and I see a leak with that
too:
[root@mta41 ~]# node --expose_gc watchfile_leak.js
{ rss: 8478720, heapTotal: 6163968, heapUsed: 1918824 }
{ rss: 18067456, heapTotal: 18647040, heapUsed: 8692304 }
Code for this:
var fs = require('fs');
gc();
console.log(process.memoryUsage());
var name = '/etc/passwd';
for (var i=0; i1; i++) {
var foo = fs.watch(name, function () {});
foo.close();
gc();
}
gc();
console.log(process.memoryUsage());
This is on Linux 2.6.18 x86_64 running Node 0.10.22.
Kind regards,
Steve.
That doesn't necessarily imply a memory leak, the garbage collector
may just be growing the heap. What happens when you set
--max_old_space_size=128?
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Memory Usage and disparity between RSS and Heap size
Hi Ben,
On 23/11/13 00:40, Ben Noordhuis wrote:
That doesn't necessarily imply a memory leak, the garbage collector
may just be growing the heap. What happens when you set
--max_old_space_size=128?
Same result (this is with fs.watch):
[root@mta41 ~]# node --expose_gc --max_old_space_size=128 watchfile_leak.js
{ rss: 8499200, heapTotal: 6163968, heapUsed: 1918832 }
{ rss: 18079744, heapTotal: 18647040, heapUsed: 8688392 }
I tried it again with an 8M old space size with the same result:
[root@mta41 ~]# node --expose_gc --max_old_space_size=8 watchfile_leak.js
{ rss: 8491008, heapTotal: 6163968, heapUsed: 1918832 }
{ rss: 18071552, heapTotal: 18647040, heapUsed: 8688392 }
Regards,
Steve.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Memory Usage and disparity between RSS and Heap size
On Sat, Nov 23, 2013 at 1:54 AM, Steve Freegard st...@stevefreegard.com wrote:
Hi Ben,
On 23/11/13 00:40, Ben Noordhuis wrote:
That doesn't necessarily imply a memory leak, the garbage collector
may just be growing the heap. What happens when you set
--max_old_space_size=128?
Same result (this is with fs.watch):
[root@mta41 ~]# node --expose_gc --max_old_space_size=128 watchfile_leak.js
{ rss: 8499200, heapTotal: 6163968, heapUsed: 1918832 }
{ rss: 18079744, heapTotal: 18647040, heapUsed: 8688392 }
I tried it again with an 8M old space size with the same result:
[root@mta41 ~]# node --expose_gc --max_old_space_size=8 watchfile_leak.js
{ rss: 8491008, heapTotal: 6163968, heapUsed: 1918832 }
{ rss: 18071552, heapTotal: 18647040, heapUsed: 8688392 }
Regards,
Steve.
Right. I see that your test case creates 1M file watchers in a loop.
Those won't be released right away and calling gc() won't change that.
Disposing a file watcher takes two event loop ticks or more but in
your test case, everything is done in a single tick.
If you smear out the loop iterations with setImmediate(), you should
see more consistent memory usage.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Memory Usage and disparity between RSS and Heap size
On 23/11/13 01:02, Ben Noordhuis wrote:
Right. I see that your test case creates 1M file watchers in a loop.
Those won't be released right away and calling gc() won't change that.
Disposing a file watcher takes two event loop ticks or more but in
your test case, everything is done in a single tick. If you smear out
the loop iterations with setImmediate(), you should see more
consistent memory usage.
Ok - I tried that. I also added a gc() call after the loop, followed by
a setTimeout() of 60 seconds with a further call to gc() and then set an
process.on('exit') handler to output process.memoryUsage() so I would
only get the output once all of the watchers were closed.
I'm still seeing memory leaking:
using fs.watch:
[root@mta41 ~]# node --expose_gc --max_old_space_size=8 watchfile_leak.js
{ rss: 8491008, heapTotal: 6163968, heapUsed: 1919544 }
{ rss: 13017088, heapTotal: 13487360, heapUsed: 1966256 }
using fs.watchFile:
[root@mta41 ~]# node --expose_gc --max_old_space_size=8 watchfile_leak.js
{ rss: 8495104, heapTotal: 6163968, heapUsed: 1919632 }
{ rss: 13275136, heapTotal: 21875968, heapUsed: 1974984 }
This is with 10,000 watchers being added and immediately removed, so the
process would not exit until the watchers have been terminated.
Based on what we saw in Haraka - there is absolutely no doubt in my mind
that there is something amiss here. We were incorrectly calling
fs.unwatchFile() followed by fs.watchFile() thousands of times per
connection and the memory usage grew massively over a short space of
time. The moment I changed the code to avoid the calls to
fs.watchFile/unwatchFile we went from having an RSS of 1Gb in about an
hour to around 100Mb.
Cheers,
Steve.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
nodejs group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Re: How does round-robin clustering work in Node?
I have a naive question here. If cluster doesn't know if the worker is busy, what happens when a new connection comes in and it is still busy? Do requests queue at each worker? What behavior should be expected if you're using round robin with lots of busy workers? I'm thinking of the Heroku problem, which I feel like I read enough about to understand conceptually. This isn't my area, so when you answer you can assume I don't know what the hell I'm talking about :) :Marco On Friday, November 22, 2013 8:12:41 AM UTC-8, Ben Noordhuis wrote: On Fri, Nov 22, 2013 at 3:30 PM, Andrew D dep...@gmail.com javascript: wrote: I have a question that is not immediately obvious after reading these two links. How well does Node's round robin work with requests that produce significantly varied workloads on the server? I don't mean, how well does the round robin algorithm work with varies workloads, since I already know the answer to that... what I mean is, is Node smart enough to skip workers that are still busy from the last round robin request sent to them? In other words, if a worker is busy on a CPU bound function, will it be skipped until it gets freed up again? We have a development platform that is similar to Meteor ( http://www.meteor.com/http://www.google.com/url?q=http%3A%2F%2Fwww.meteor.com%2Fsa=Dsntz=1usg=AFQjCNFsYVXZEstozl04pf0AN_kdnHh4dA) in that code transparently runs on the client or server. All processing is done in JavaScript, which means some functions are way more CPU bound than others - and the creative developers using this platform use it for all sorts of things (data processing, image processing, DB access, etc). What we like about the current clustering is that a request is not sent to a worker that is spinning away on some CPU bound function. Yes, the current approach does tend to send requests to the same workers when they are not busy, so round robin would be awesome to fix that issue... but is the current round robin smart enough to skip workers that are busy? The answer to that question is mostly 'no'. The master won't send new connections to workers that haven't acknowledged receipt of the last one yet but, apart from that, it doesn't really know whether a worker is busy or not. We'll undoubtedly address that in due time but that requires considerable engineering effort. Naively sending I'm busy / I'm idle messages back and forth whenever user code is executed leads to serious performance degradation because you quickly hit a point where you spend all your CPU time processing messages and not doing anything useful. There are solutions to that issue (mostly involving tricks with shared memory) but none are easy to implement in a cross-platform manner. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [nodejs] Node.JS for sysadmin scripting or non-web stuff - books, guides, examples?
On Nov 22, 2013, at 9:53 AM, Matt hel...@gmail.com wrote: Having said that, if you don't know Perl, the sigils ($/@/% in front of all the variables) and special variables will look really annoying. Especially once the dereferencing gets complicated. —ravi -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups nodejs group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
