Re: 4.3: netstat question
Hi, On Sat, 14.06.2008 at 01:39:29 +0200, Claudio Jeker [EMAIL PROTECTED] wrote: Nope. That is not the problem. The main issues is that a full view will need a lot of memory for the sysctl. This memory needs to be available as real memory because it is wired into the kernel. If you run bgpd with full views on a box with less then 512MB of RAM you're most probably run out of memory. Theo and I had a look at this and bailing out in this situation is the right thing to do. thanks for the explanation! The right fix is to just spend 50 bucks on 1-2GB of additional RAM. I'll look into finding appropriate RAM and/or putting that card into a different box. c) work around (ugly but works) netstat -rnfinet -M /dev/mem Nice! d) the route sysctl needs to be rewritten to be fully restartable and so small chunks of the table can be fetched one after the other. This is a massive change and it will not happen for the upcomming release. I'm not sure that I understand the need to copy the table, or parts thereof, correctly. Sure, the table changes all the time. So, the routes viewed when running 'netstat -r' are only a snapshot and may have changed by the time the user views them, anyway. Would it be possible to walk along the live table, without copying the table, or would the continuous stream of route inserts and deletes lead to a corrupted view and/or access to the wrong parts of the system's memory (which must to be prevented), or would this be such a performance hit that this is unfeasible? Kind regards, --Toni++
Re: 4.3: netstat question
* Toni Mueller [EMAIL PROTECTED] [2008-06-14 11:29]: Would it be possible to walk along the live table, without copying the table, or would the continuous stream of route inserts and deletes lead to a corrupted view and/or access to the wrong parts of the system's memory (which must to be prevented), or would this be such a performance hit that this is unfeasible? userland can walk a kernel table since when exactly? (leave dirty /dev/mem style hacks aside) -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg Amsterdam
Re: 4.3: netstat question
On Sat, Jun 14, 2008 at 1:11 PM, Henning Brauer [EMAIL PROTECTED] wrote: * Toni Mueller [EMAIL PROTECTED] [2008-06-14 11:29]: Would it be possible to walk along the live table, without copying the table, or would the continuous stream of route inserts and deletes lead to a corrupted view and/or access to the wrong parts of the system's memory (which must to be prevented), or would this be such a performance hit that this is unfeasible? userland can walk a kernel table since when exactly? (leave dirty /dev/mem style hacks aside) If the kernel table is kept in an ordered state, userland could provide a starting value or key. The kernel can then return the requested chunk (up to the size requested) starting at the next table item that comes after the key. Also depends if you're willing to let netstat display routes that are may appear inconsistent. Just thinking off the top of my head for ways to avoid allocating the whole table at once. Apologies if it's too gross an API change or has other, worse repercussions. --david
Re: 4.3: netstat question
* David Higgs [EMAIL PROTECTED] [2008-06-15 01:59]: On Sat, Jun 14, 2008 at 1:11 PM, Henning Brauer [EMAIL PROTECTED] wrote: * Toni Mueller [EMAIL PROTECTED] [2008-06-14 11:29]: Would it be possible to walk along the live table, without copying the table, or would the continuous stream of route inserts and deletes lead to a corrupted view and/or access to the wrong parts of the system's memory (which must to be prevented), or would this be such a performance hit that this is unfeasible? userland can walk a kernel table since when exactly? (leave dirty /dev/mem style hacks aside) If the kernel table is kept in an ordered state, userland could provide a starting value or key. The kernel can then return the requested chunk (up to the size requested) starting at the next table item that comes after the key. wow. you completely miss the point. userland cannot poke in kernel memory. (footnote: ok, it can, but assuming it can't is better) -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg Amsterdam
Re: 4.3: netstat question
On Sat, Jun 14, 2008 at 9:16 PM, Henning Brauer [EMAIL PROTECTED] wrote: * David Higgs [EMAIL PROTECTED] [2008-06-15 01:59]: On Sat, Jun 14, 2008 at 1:11 PM, Henning Brauer [EMAIL PROTECTED] wrote: * Toni Mueller [EMAIL PROTECTED] [2008-06-14 11:29]: Would it be possible to walk along the live table, without copying the table, or would the continuous stream of route inserts and deletes lead to a corrupted view and/or access to the wrong parts of the system's memory (which must to be prevented), or would this be such a performance hit that this is unfeasible? userland can walk a kernel table since when exactly? (leave dirty /dev/mem style hacks aside) If the kernel table is kept in an ordered state, userland could provide a starting value or key. The kernel can then return the requested chunk (up to the size requested) starting at the next table item that comes after the key. wow. you completely miss the point. userland cannot poke in kernel memory. (footnote: ok, it can, but assuming it can't is better) I knew that, but I explained myself poorly. I was thinking something along the lines of making a different route sysctl (other than NET_RT_DUMP) that can copy out smaller portions of the routing table at a time. Userland programs could then iterate their way through the routing table. Depending on the structures being copied out, this might be completely unworkable. On top of that, you'd at best just push back the limits on available real memory. Best to wait for a restartable route sysctl. Apologies for the noise and my out-loud musings. --david
Re: 4.3: netstat question
On Sat, Jun 14, 2008 at 10:55:52PM -0400, David Higgs wrote: On Sat, Jun 14, 2008 at 9:16 PM, Henning Brauer [EMAIL PROTECTED] wrote: * David Higgs [EMAIL PROTECTED] [2008-06-15 01:59]: On Sat, Jun 14, 2008 at 1:11 PM, Henning Brauer [EMAIL PROTECTED] wrote: * Toni Mueller [EMAIL PROTECTED] [2008-06-14 11:29]: Would it be possible to walk along the live table, without copying the table, or would the continuous stream of route inserts and deletes lead to a corrupted view and/or access to the wrong parts of the system's memory (which must to be prevented), or would this be such a performance hit that this is unfeasible? userland can walk a kernel table since when exactly? (leave dirty /dev/mem style hacks aside) If the kernel table is kept in an ordered state, userland could provide a starting value or key. The kernel can then return the requested chunk (up to the size requested) starting at the next table item that comes after the key. wow. you completely miss the point. userland cannot poke in kernel memory. (footnote: ok, it can, but assuming it can't is better) I knew that, but I explained myself poorly. I was thinking something along the lines of making a different route sysctl (other than NET_RT_DUMP) that can copy out smaller portions of the routing table at a time. Userland programs could then iterate their way through the routing table. Depending on the structures being copied out, this might be completely unworkable. On top of that, you'd at best just push back the limits on available real memory. Best to wait for a restartable route sysctl. Apologies for the noise and my out-loud musings. Yes that's more or less what needs to be done. I'm willing to look at diffs and help working out the evil guts of this. -- :wq Claudio
Re: 4.3: netstat question
On Thu, Jun 12, 2008 at 3:47 PM, Toni Mueller [EMAIL PROTECTED] wrote: as of today (I didn't notice it earlier), I see this problem on one of my machines: # netstat -rnf inet netstat: sysctl of routing table: Cannot allocate memory netstat -r dumps the routing table by calling sysctl() twice, once to get the size of the table so that it can allocate enough memory to hold it, and then a second call to actually fill it in. That error means the routing table grew between the two calls, so the second call could return the entire table. Any idea about how to combat this, please? The code will need to be changed to a) add a fudge factor to the size that was returned, and b) retry the pair of calls if the second returns ENOMEM I'll try to send you a patch this weekend. Philip Guenther
Re: 4.3: netstat question
On Fri, Jun 13, 2008 at 04:20:45PM -0700, Philip Guenther wrote: On Thu, Jun 12, 2008 at 3:47 PM, Toni Mueller [EMAIL PROTECTED] wrote: as of today (I didn't notice it earlier), I see this problem on one of my machines: # netstat -rnf inet netstat: sysctl of routing table: Cannot allocate memory netstat -r dumps the routing table by calling sysctl() twice, once to get the size of the table so that it can allocate enough memory to hold it, and then a second call to actually fill it in. That error means the routing table grew between the two calls, so the second call could return the entire table. Nope. That is not the problem. The main issues is that a full view will need a lot of memory for the sysctl. This memory needs to be available as real memory because it is wired into the kernel. If you run bgpd with full views on a box with less then 512MB of RAM you're most probably run out of memory. Theo and I had a look at this and bailing out in this situation is the right thing to do. The right fix is to just spend 50 bucks on 1-2GB of additional RAM. Any idea about how to combat this, please? The code will need to be changed to a) add a fudge factor to the size that was returned, and There is already enough fudge in the sysctl itself. The estimate done by the sysctl in the first run is 10% over the needed memory. b) retry the pair of calls if the second returns ENOMEM Will not help either. c) work around (ugly but works) netstat -rnfinet -M /dev/mem d) the route sysctl needs to be rewritten to be fully restartable and so small chunks of the table can be fetched one after the other. This is a massive change and it will not happen for the upcomming release. -- :wq Claudio
Re: 4.3: netstat question
On Fri, Jun 13, 2008 at 5:39 PM, Claudio Jeker [EMAIL PROTECTED] wrote: ... Nope. That is not the problem. The main issues is that a full view will need a lot of memory for the sysctl. This memory needs to be available as real memory because it is wired into the kernel. If you run bgpd with full views on a box with less then 512MB of RAM you're most probably run out of memory. Theo and I had a look at this and bailing out in this situation is the right thing to do. The right fix is to just spend 50 bucks on 1-2GB of additional RAM. Yuck. For now, how about the following patch? Index: sysctl.3 === RCS file: /cvs/src/lib/libc/gen/sysctl.3,v retrieving revision 1.181 diff -u -r1.181 sysctl.3 --- sysctl.330 May 2008 19:09:42 - 1.181 +++ sysctl.314 Jun 2008 03:26:26 - @@ -2176,6 +2176,12 @@ The length pointed to by .Fa oldlenp is too short to hold the requested value. +.It Bq Er ENOMEM +There isn't enough real memory available to pin the buffers specified by +.Fa oldp +and +.Fa newp +in the kernel. .It Bq Er ENOTDIR The .Fa name Philip Guenther
4.3: netstat question
Hi, as of today (I didn't notice it earlier), I see this problem on one of my machines: # netstat -rnf inet netstat: sysctl of routing table: Cannot allocate memory This machine receives two full feeds @ ~255k routes each. Any idea about how to combat this, please? Kind regards, --Toni++
Re: 4.3: netstat question
On Fri, Jun 13, 2008 at 12:47:25AM +0200, Toni Mueller wrote: Hi, as of today (I didn't notice it earlier), I see this problem on one of my machines: # netstat -rnf inet netstat: sysctl of routing table: Cannot allocate memory This machine receives two full feeds @ ~255k routes each. Any idea about how to combat this, please? By sending a dmesg with your message? Kind regards, --Toni++
