Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Thu, Aug 28, 2008 at 09:32:13AM +0900, Paul Mundt wrote: On Wed, Aug 27, 2008 at 08:35:44PM +0300, Adrian Bunk wrote: On Thu, Aug 28, 2008 at 01:00:52AM +0900, Paul Mundt wrote: On Wed, Aug 27, 2008 at 02:58:30PM +0300, Adrian Bunk wrote: In addition to that, debugging the runaway stack users on 4k tends to be easier anyways since you end up blowing the stack a lot sooner. On sh we've had pretty good luck with it, though most of our users are using fairly deterministic workloads and continually profiling the footprint. Anything that runs away or uses an insane amount of stack space needs to be fixed well before that anyways, so catching it sooner is always preferable. I imagine the same case is true for m68knommu (even sans IRQ stacks). CONFIG_DEBUG_STACKOVERFLOW should give you the same information, and if wanted with an arbitrary limit. In some cases, yes. In the CONFIG_DEBUG_STACKOVERFLOW case the check is only performed from do_IRQ(), which is sporadic at best, especially on tickless. While it catches some things, it's not a complete solution in and of iteslf. In addition to this, there are even fewer platforms that support it than there are platforms that do 4k stacks. At first glance, it looks like it's only m32r, powerpc, sh, x86, and xtensa. ... As far as I can see the only architectures that optionally offer 4kB stacks today are m68knommu, s390, sh and x86. Did I miss some architectures or is 5 4 ;) ? Others support the Kconfig option, but don't seem to realize that it's not an option that the kernel does anything with by itself, and so don't actually do anything (ie, FRV). Unless I miss anything these others include only FRV. IMHO there seems to currently be a mismatch between it's maintainance cost and the actual number of users. That's in my opinion the main problem with it, no matter in which direction it gets resolved. Perhaps that's true on x86, but in general I take issue with that. On sh we've had to do very little maintenance for it and most shipping products are using it today (at least on MMU-Linux, we don't bother with it on nommu). Most of the problems we ran in to with 4k stacks tended to be stuff that we wanted to fix for 8k anyways. I suspect that this case is true for the other embedded platforms also. ... Most stack issues are not platform or architecture specific. The maintainance effort therefore mostly depends on whether a non-zero number of architectures uses 4kB stacks. And if something is considered to be important for small embedded systems, but not supported on ARM, MIPS or PowerPC, then that's a bit strange. cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Linus Torvalds writes: 4kB used to be the _only_ choice. And no, there weren't even irq stacks. So that 4kB was not just the whole kernel call-chain, it was also all the irq nesting above it. I think your memory is failing you. In 2.4 and earlier, the kernel stack was 8kB minus the size of the task_struct, which sat at the start of the 8kB. For instance, from include/asm-i386/processor.h for 2.4.29: #define THREAD_SIZE (2*PAGE_SIZE) #define alloc_task_struct() ((struct task_struct *) __get_free_pages(GFP_KERNEL,1)) #define free_task_struct(p) free_pages((unsigned long) (p), 1) Paul. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, 2008-08-26 at 18:54 -0400, Parag Warudkar wrote: On Tue, Aug 26, 2008 at 5:04 PM, Linus Torvalds [EMAIL PROTECTED] wrote: And embedded people (the ones that might care about 1% code size) are the ones that would also want smaller stacks even more! This is something I never understood - embedded devices are not going to run more than a few processes and 4K*(Few Processes) IMHO is not worth a saving now a days even in embedded world given falling memory prices. Or do I misunderstand? Falling prices are no reason to increase the amount of available RAM (or other hardware). Especially if you (intend to) build 1E5 devices - where every Euro counts. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, 2008-08-26 at 22:16 -0400, Parag Warudkar wrote: [...] Well, sure - but the industry as a whole seems to have gone the other The industry as a whole doesn't exist on that low level. You can't compare the laptop and/or desktop computer market (where one may buy today hardware that runs in 3 years with the next generation/release of the OS and applications) with the e.g. WLAN router market where - from the commercial point of view - every Euro counts (and where the requirements for the lifetime of the device are long frozen before the thing gets in a shop). way - do more with more at the similar or lower price points! By that definition of less is better we should try and make the kernel memory pageable (or has someone already done that?) - Windows does it, That doesn't help as in really small devices (like WLAN routers, cable modems, etc.) you run without any means of paging/swapping. And even binaries/read-only files are not necessarily executable in place (but must be loaded into RAM). So you can't flush these pages. And pageable kernel memory doesn't come for free - even if one only counts the increased code and it's complexity. by default ;) Which is more a sign that it is probably a very bad idea. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
What about deep call chains? The problem with the uptake of 4K stacks seems to be that is not reliably provable that it will work under all circumstances. On x86-32 with 8K stacks your IRQ paths share them so that is even harder to prove (not that you can prove any of them) and the bugs are more obscure and random. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, 2008-08-26 at 20:58 -0400, Parag Warudkar wrote: [...] The savings part -financial ones- are not always realizable with the way memory is priced/sized/fitted. Savings in few Mb of Kernel stack are not necessarily going to allow getting rid of a single memory chip of 64M or so. No, but you can put an additional service(s) on it and sales people have one (or two or ) line more for their sales brochures. Either that or embedded manufacturing/configurations are different than the desktop world. They are different. Think of running the complete system acting as a bridge, router and/or firewall (Kernel early 2.4 though) from 4MB flash in 32MB RAM and - listing the outside visible services - having a command-line interface, web-GUI (implying a http server) and and a (net-)SNMP agent on it. Running a glibc without thread support is win there (implying that there is no thread support available on that device). (If my device has 2 memory slots and my user space requires 100Mb including kernel memory - I anyways have to put in 64Mx2 there to take advantage of mass manufactured, general purpose memory - so no big deal if I saved 1.2Mb in Kernel stack or not. And savings of 64Mb Kernel memory are not feasible anyways to allow user space to work with 64Mb.) As soon as product management realizes that there is space left on the device, they get new ideas and/or customer requirements to run more services on that device. On the other hand reducing user space memory usage on those devices (not counting savings from kernel stack size) is a way more attractive option. There is no question if save space here or there. You save it - sooner or later - on all fronts. Period. And although you said in your later reply that Linux x86 with 4K stacks should be more than usable - my experiences running a untainted desktop/file server with 4K stack have been always disastrous XFS or not. It _might_ work for some well defined workloads but you would not want to risk 4K stacks otherwise. The embedded world of really small devices usually doesn't run XFS (or ext? or reiser* of jfs or NFS or ...) or stacks block devices on files or . I understand the having 4K stack option as a non-default for very specific workloads is a good idea but apart from that I think no one else seems to bother with reducing stack sizes (by no one I mean other OSes.) They probably gave the idea pretty soon because you need to rework/improve large parts of the kernel + drivers (and that has two major problems - it consumes a lot of man power for no new features and everything must be completely tested again[0] and it adds new risks). And that is practically impossible if one sells stable driver APIs for 3rd party (commercial) drivers because these must be changed too. Bernd [0]: Let alone if you (or your customers) need certificates from some governmental agencys. -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 05:28:37PM -0700, Linus Torvalds wrote: On Wed, 27 Aug 2008, Adrian Bunk wrote: When did we get callpaths like like nfs+xfs+md+scsi reliably working with 4kB stacks on x86-32? XFS may never have been usable, but the rest, sure. And you seem to be making this whole argument an excuse to SUCK, adn an excuse to let gcc crap even more on our stack space. Why? Why aren't you saying that we should be able to do better? Instead, you seem to asking us to do even worse than we do now? My main point is: - getting 4kB stacks working reliably is a hard task - having an eye on gcc increasing the stack usage, and fixing it if required, is relatively easy If we should be able to do better at getting (and keeping) 4kB stacks working, then coping with possible inlining problems caused by gcc should not be a big problem for us. Linus cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 06:49:19PM -0700, Linus Torvalds wrote: ... But part of it is definitely gcc. Some versions of gcc used to be absolutely _horrid_ when it came to stack usage, especially with some flags, and especially with the crazy inlining that module-at-a-time caused. ... That was gcc 3.4. And due to that we disable unit-at-a-time for gcc 3.4 on 32bit x86. Linus cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Wed, Aug 27, 2008 at 5:00 AM, Bernd Petrovitsch [EMAIL PROTECTED] wrote: They probably gave the idea pretty soon because you need to rework/improve large parts of the kernel + drivers (and that has two major problems - it consumes a lot of man power for no new features and everything must be completely tested again[0] and it adds new risks). And that is practically impossible if one sells stable driver APIs for 3rd party (commercial) drivers because these must be changed too. But not many embedded Linux arches support 4K stacks like Adrian pointed out earlier. So the same (lot of man power requirement) would apply to Linux. Sure it will be good - but how reasonable it is to attempt it and how reliably it will work under all conceived loads - those are the questions. Thanks Parag -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
By your logic though, XFS on x86 should work fine with 4K stacks - many will attest that it does not and blows up due to stack issues. I have first hand experiences of things blowing up with deep call chains when using 4K stacks where 8K worked just fine on same workload. So there is definitely some other problem with 4K stacks. Nothing of the sort. If it blows up with a 4K stack it will almost certainly blow up with an 8K stack *eventually* - when a heavy stack usage coincides with a heavy stack using IRQ handler. You won't catch it in simple testing, you won't catch it in trivial simulation and it'll be incredibly hard to reproduce. Not the kind of bug you want in a production system really. IRQ stacks make things much more predictable. Alan -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Wed, 27 Aug 2008, Paul Mackerras wrote: I think your memory is failing you. In 2.4 and earlier, the kernel stack was 8kB minus the size of the task_struct, which sat at the start of the 8kB. Yup, you're right. Linus -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Bernd Petrovitsch wrote: If you develop an embedded system (which is partly system integration of existing apps) to be installed in the field, you don't have that many conceivable work loads compared to a desktop/server system. And you have a fixed list of drivers and applications. Hah! Not in my line of embedded device. 32MB no-MMU ARM boards which people run new things and attach new devices to rather often - without making new hardware. Volume's too low per individual application to get new hardware designed and made. I'm seriously thinking of forwarding porting the 4 year old firmware from 2.4.26 to 2.6.current, just to get new drivers and capabilities. Backporting is tedious, so's feeling wretchedly far from the mainline world. A usual approach is to run stress tests on several (or all) subsystems/services/... in parallel and if the device survives it functioning correctly, it is at least good enough. Per application. Some little devices run hundreds of different applications and customers expect to customise, script themselves, and attach different devices (over USB). The next customer in the chain expects the bits you supplied to work in a variety of unexpected situations, even when you advise that it probably won't do that. Much like desktop/server Linux, but on a small device where silly little things like 'create a process' are a stress for the dear little thing. (My biggest lesson: insist on an MMU next time!) -- Jamie -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Wed, Aug 27, 2008 at 02:58:30PM +0300, Adrian Bunk wrote: On Tue, Aug 26, 2008 at 05:28:37PM -0700, Linus Torvalds wrote: On Wed, 27 Aug 2008, Adrian Bunk wrote: When did we get callpaths like like nfs+xfs+md+scsi reliably working with 4kB stacks on x86-32? XFS may never have been usable, but the rest, sure. And you seem to be making this whole argument an excuse to SUCK, adn an excuse to let gcc crap even more on our stack space. Why? Why aren't you saying that we should be able to do better? Instead, you seem to asking us to do even worse than we do now? My main point is: - getting 4kB stacks working reliably is a hard task - having an eye on gcc increasing the stack usage, and fixing it if required, is relatively easy If we should be able to do better at getting (and keeping) 4kB stacks working, then coping with possible inlining problems caused by gcc should not be a big problem for us. Out of the architectures you've mentioned for 4k stacks, they also tend to do IRQ stacks, which is something you seem to have overlooked. In addition to that, debugging the runaway stack users on 4k tends to be easier anyways since you end up blowing the stack a lot sooner. On sh we've had pretty good luck with it, though most of our users are using fairly deterministic workloads and continually profiling the footprint. Anything that runs away or uses an insane amount of stack space needs to be fixed well before that anyways, so catching it sooner is always preferable. I imagine the same case is true for m68knommu (even sans IRQ stacks). Things might be more sensitive on x86, but it's certainly not something that's a huge problem for the various embedded platforms to wire up, whether they want to go the IRQ stack route or not. In any event, lack of support for something on embedded architectures in the kernel is more often due to apathy/utter indifference on the part of the architecture maintainer rather than being indicative of any intrinsic difficulty in supporting the thing in question. Most new features on the lesser maintained architectures tend to end up there either out of peer pressure or copying-and-pasting accidents rather than any sort of design. ;-) -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Linus Torvalds wrote: Most LOCs of the kernel are not written by people like you or Al Viro or David Miller, and the average kernel developer is unlikely to do it as good as gcc. Sure. But we do have tools. We do have checkstack.pl, it's just that it hasn't been an issue in a long time, so I suspect many people didn't even _realize_ we have it, and I certainly can attest to the fact that even people who remember it - like me - don't actually tend to run it all that often. Sounds like what's really desired here isn't more worry and unpredictability, but for GCC+Binutils to gain the ability to calculate the stack depth over all callchains (doesn't have to be exact, just an upper bound; annotate recursions) in a way that's good enough to do on every compile, complain if a depth is exceeded statically (or it can't be proven), and to gain the architecture-independent option optimise to reduce stack usage. BTW: I just ran checkstack on a (roughly) allyesconfig kernel, and we have a new driver that allocates unsigned char recvbuf[1500]; on the stack... Yeah, it's _way_ too easy to do bad things. In my userspace code, I have macros tmp_alloc and tmp_free. They must be matched in the same function: unsigned char * recvbuf = tmp_alloc(1500); tmp_free(recvbuf); When stack is plentiful, it maps to alloca() which is roughly equivalent to using a stack variable. When stack is constrained (as it is on my little devices), that maps to xmalloc/free. The kernel equivalent would be kmalloc GFP_ATOMIC (perhaps). With different macros to mine, it may be possible to map small fixed-size requests exactly onto local variables, and large ones to kmalloc(). A stab at it (not tested): #define LOCAL_ALLOC_THRESHOLD 128 #define LOCAL_ALLOC(type, ptr)\ __typeof__(type) __attribute__((__unused__)) ptr##_local_struct; \ __typeof__(type) * ptr = \ ((__builtin_constant_p(sizeof(type))\ sizeof(type) = LOCAL_ALLOC_THRESHOLD) \ ? ptr##_local_struct : kmalloc(sizeof(type), GFP_ATOMIC)) #define LOCAL_FREE(ptr) \ ((__builtin_constant_p(sizeof (*(ptr))) \ sizeof(*(ptr)) = LOCAL_ALLOC_THRESHOLD) \ ? (void) 0 : kfree(ptr)) Would that be useful in the kernel? I'm thinking if it were a commonly used pattern for temporary buffers, unknown structures and arrays of macro-determined size, the new driver author would be less likely to accidentally drop a big object on the stack. Obviously it would be nicer for GCC to code such a thing automatically, but that really is wishful thinking. -- Jamie -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Wed, 2008-08-27 at 16:48 +0100, Jamie Lokier wrote: Bernd Petrovitsch wrote: If you develop an embedded system (which is partly system integration of existing apps) to be installed in the field, you don't have that many conceivable work loads compared to a desktop/server system. And you have a fixed list of drivers and applications. Hah! Not in my line of embedded device. 32MB no-MMU ARM boards which people run new things and attach new devices to rather often - without making new hardware. Volume's too low per individual application to get new hardware designed and made. Yes, you may have several products on the same hardware with somewhat differing requirements (or not). But that is much less than a general purpose system IMHO. I'm seriously thinking of forwarding porting the 4 year old firmware from 2.4.26 to 2.6.current, just to get new drivers and capabilities. That sounds reasonable (and I never meant maintaining the old system infinitely. Actually once the thing is shipped it usually enters deep maintenance mode and the next is more a fork from the old). Backporting is tedious, so's feeling wretchedly far from the mainline world. ACK. But that also depends on amount local changes (and sorry, but not all locally necessary patches would be accepted in mainline in any way). A usual approach is to run stress tests on several (or all) subsystems/services/... in parallel and if the device survives it functioning correctly, it is at least good enough. Per application. Some little devices run hundreds of different applications and customers expect to customise, script themselves, and attach different devices (over USB). The next customer in the chain expects the bits you supplied to work in a variety of unexpected situations, even when you advise that it probably won't do that. Basically their problem. Yes, they actually think they get a Linux system where they can do everything and it simply works. Oh, that's obviously not a usual WLAN-router style of product (where you are not expected to actually login on a console or per ssh). Much like desktop/server Linux, but on a small device where silly little things like 'create a process' are a stress for the dear little thing. (My biggest lesson: insist on an MMU next time!) ACK. We avoid MMU-less hardware too - especially since there is enough hardware with a MMU around. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Thu, Aug 28, 2008 at 01:00:52AM +0900, Paul Mundt wrote: On Wed, Aug 27, 2008 at 02:58:30PM +0300, Adrian Bunk wrote: On Tue, Aug 26, 2008 at 05:28:37PM -0700, Linus Torvalds wrote: On Wed, 27 Aug 2008, Adrian Bunk wrote: When did we get callpaths like like nfs+xfs+md+scsi reliably working with 4kB stacks on x86-32? XFS may never have been usable, but the rest, sure. And you seem to be making this whole argument an excuse to SUCK, adn an excuse to let gcc crap even more on our stack space. Why? Why aren't you saying that we should be able to do better? Instead, you seem to asking us to do even worse than we do now? My main point is: - getting 4kB stacks working reliably is a hard task - having an eye on gcc increasing the stack usage, and fixing it if required, is relatively easy If we should be able to do better at getting (and keeping) 4kB stacks working, then coping with possible inlining problems caused by gcc should not be a big problem for us. Out of the architectures you've mentioned for 4k stacks, they also tend to do IRQ stacks, which is something you seem to have overlooked. No, I am aware of that, and on i386 IRQ stacks are only used with 4kB stacks. On i386 it is effectively a step from 6kB to 4kB. In addition to that, debugging the runaway stack users on 4k tends to be easier anyways since you end up blowing the stack a lot sooner. On sh we've had pretty good luck with it, though most of our users are using fairly deterministic workloads and continually profiling the footprint. Anything that runs away or uses an insane amount of stack space needs to be fixed well before that anyways, so catching it sooner is always preferable. I imagine the same case is true for m68knommu (even sans IRQ stacks). CONFIG_DEBUG_STACKOVERFLOW should give you the same information, and if wanted with an arbitrary limit. Things might be more sensitive on x86, but it's certainly not something that's a huge problem for the various embedded platforms to wire up, whether they want to go the IRQ stack route or not. How many platforms use 4kB stacks on sh? Only 1 out of 34 defconfigs uses it. Are there any numbers for real life usage. In any event, lack of support for something on embedded architectures in the kernel is more often due to apathy/utter indifference on the part of the architecture maintainer rather than being indicative of any intrinsic difficulty in supporting the thing in question. Most new features on the lesser maintained architectures tend to end up there either out of peer pressure or copying-and-pasting accidents rather than any sort of design. ;-) arm or powerpc aren't exactly lesser maintained architectures. 4kB has shown to be a hard to achieve limit. After more than 4 years in mainline being available on i386 there are still cases where 4kB are not enough. IMHO there seems to currently be a mismatch between it's maintainance cost and the actual number of users. That's in my opinion the main problem with it, no matter in which direction it gets resolved. cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Bernd Petrovitsch wrote:
32MB no-MMU ARM boards which people run new things and attach new
devices to rather often - without making new hardware. Volume's too
low per individual application to get new hardware designed and made.
Yes, you may have several products on the same hardware with somewhat
differing requirements (or not). But that is much less than a general
purpose system IMHO.
It is, but the idea that small embedded systems go through a 'all
components are known, drivers are known, test and if it passes it's
shippable' does not always apply.
I'm seriously thinking of forwarding porting the 4 year old firmware
from 2.4.26 to 2.6.current, just to get new drivers and capabilities.
That sounds reasonable (and I never meant maintaining the old system
infinitely.
Sounds reasonable, but it's vetoed for anticipated time and cost,
compared with backporting on demand. Fair enough, since 2.6.current
doesn't support ARM no-MMU last I heard ('soon'?).
On the other hand, the 2.6 anti-fragmentation patches, including
latest SLUB stuff, ironically meant to help big machines, sound really
appealing for my current problem and totally unrealistic to
backport...
ACK. We avoid MMU-less hardware too - especially since there is enough
hardware with a MMU around.
I can't emphasise enough how much difference MMU makes to Linux userspace.
It's practically: MMU = standard Linux (with less RAM), have everything.
No-MMU = lots of familiar 'Linux' things not available or break.
-- Jamie
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Mit, 2008-08-27 at 18:51 +0100, Jamie Lokier wrote: Bernd Petrovitsch wrote: [...] It is, but the idea that small embedded systems go through a 'all components are known, drivers are known, test and if it passes it's shippable' does not always apply. Not always but often enough. And yes, there is ARM-based embedded hardware with 1GB Flash-RAM and 128MB RAM. I'm seriously thinking of forwarding porting the 4 year old firmware from 2.4.26 to 2.6.current, just to get new drivers and capabilities. That sounds reasonable (and I never meant maintaining the old system infinitely. Sounds reasonable, but it's vetoed for anticipated time and cost, That is to be expected;-) [] ACK. We avoid MMU-less hardware too - especially since there is enough hardware with a MMU around. I can't emphasise enough how much difference MMU makes to Linux userspace. It's practically: MMU = standard Linux (with less RAM), have everything. No-MMU = lots of familiar 'Linux' things not available or break. ACK. And tell that a customer that everything is more effort and more risk and not just simply cross-compile it as it runs on my desktop too. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Jamie Lokier wrote:
Bernd Petrovitsch wrote:
32MB no-MMU ARM boards which people run new things and attach new
devices to rather often - without making new hardware. Volume's too
low per individual application to get new hardware designed and made.
Yes, you may have several products on the same hardware with somewhat
differing requirements (or not). But that is much less than a general
purpose system IMHO.
It is, but the idea that small embedded systems go through a 'all
components are known, drivers are known, test and if it passes it's
shippable' does not always apply.
I'm seriously thinking of forwarding porting the 4 year old firmware
from 2.4.26 to 2.6.current, just to get new drivers and capabilities.
That sounds reasonable (and I never meant maintaining the old system
infinitely.
Sounds reasonable, but it's vetoed for anticipated time and cost,
compared with backporting on demand. Fair enough, since 2.6.current
doesn't support ARM no-MMU last I heard ('soon'?).
On the other hand, the 2.6 anti-fragmentation patches, including
latest SLUB stuff, ironically meant to help big machines, sound really
appealing for my current problem and totally unrealistic to
backport...
ACK. We avoid MMU-less hardware too - especially since there is enough
hardware with a MMU around.
I can't emphasise enough how much difference MMU makes to Linux userspace.
It's practically: MMU = standard Linux (with less RAM), have everything.
No-MMU = lots of familiar 'Linux' things not available or break.
And lots of things work in the usual way...
Of course the flip side is that for people who have platforms
without MMU they can run something more than the mostly toy
like operating systems typically available. There are plenty of
problem domains that the non-MMU limitations are not a problem for.
(Yours doesn't sound like one of them :-)
Regards
Greg
Greg Ungerer -- Chief Software Dude EMAIL: [EMAIL PROTECTED]
Secure Computing CorporationPHONE: +61 7 3435 2888
825 Stanley St, FAX: +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Jamie Lokier wrote: Bernd Petrovitsch wrote: If you develop an embedded system (which is partly system integration of existing apps) to be installed in the field, you don't have that many conceivable work loads compared to a desktop/server system. And you have a fixed list of drivers and applications. Hah! Not in my line of embedded device. 32MB no-MMU ARM boards which people run new things and attach new devices to rather often - without making new hardware. Volume's too low per individual application to get new hardware designed and made. I'm seriously thinking of forwarding porting the 4 year old firmware from 2.4.26 to 2.6.current, just to get new drivers and capabilities. Backporting is tedious, so's feeling wretchedly far from the mainline world. A usual approach is to run stress tests on several (or all) subsystems/services/... in parallel and if the device survives it functioning correctly, it is at least good enough. Per application. Some little devices run hundreds of different applications and customers expect to customise, script themselves, and attach different devices (over USB). The next customer in the chain expects the bits you supplied to work in a variety of unexpected situations, even when you advise that it probably won't do that. Much like desktop/server Linux, but on a small device where silly little things like 'create a process' are a stress for the dear little thing. (My biggest lesson: insist on an MMU next time!) But given you have hardware you can't change would you choose to not run Linux, even with the limitations of non-MMU? Hell no :-) Regards Greg Greg Ungerer -- Chief Software Dude EMAIL: [EMAIL PROTECTED] Secure Computing CorporationPHONE: +61 7 3435 2888 825 Stanley St, FAX: +61 7 3891 3630 Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Wed, Aug 27, 2008 at 08:35:44PM +0300, Adrian Bunk wrote: On Thu, Aug 28, 2008 at 01:00:52AM +0900, Paul Mundt wrote: On Wed, Aug 27, 2008 at 02:58:30PM +0300, Adrian Bunk wrote: In addition to that, debugging the runaway stack users on 4k tends to be easier anyways since you end up blowing the stack a lot sooner. On sh we've had pretty good luck with it, though most of our users are using fairly deterministic workloads and continually profiling the footprint. Anything that runs away or uses an insane amount of stack space needs to be fixed well before that anyways, so catching it sooner is always preferable. I imagine the same case is true for m68knommu (even sans IRQ stacks). CONFIG_DEBUG_STACKOVERFLOW should give you the same information, and if wanted with an arbitrary limit. In some cases, yes. In the CONFIG_DEBUG_STACKOVERFLOW case the check is only performed from do_IRQ(), which is sporadic at best, especially on tickless. While it catches some things, it's not a complete solution in and of iteslf. In addition to this, there are even fewer platforms that support it than there are platforms that do 4k stacks. At first glance, it looks like it's only m32r, powerpc, sh, x86, and xtensa. Others support the Kconfig option, but don't seem to realize that it's not an option that the kernel does anything with by itself, and so don't actually do anything (ie, FRV). Things might be more sensitive on x86, but it's certainly not something that's a huge problem for the various embedded platforms to wire up, whether they want to go the IRQ stack route or not. How many platforms use 4kB stacks on sh? Only 1 out of 34 defconfigs uses it. The defconfigs tend to enable as much random stuff as people are interested in for development and testing purposes. Most of these end up being reference boards and are the basis for products, rather than shipping products themselves. In the latter case, everything is gradually tightened down, and 4k stack utilization in that case is the norm, rather than the exception. In any event, lack of support for something on embedded architectures in the kernel is more often due to apathy/utter indifference on the part of the architecture maintainer rather than being indicative of any intrinsic difficulty in supporting the thing in question. Most new features on the lesser maintained architectures tend to end up there either out of peer pressure or copying-and-pasting accidents rather than any sort of design. ;-) arm or powerpc aren't exactly lesser maintained architectures. Indeed, which is why I find it bizarre that you would even bother applying what was said to those platforms. Specifically I was referring to the embedded platforms that don't do 4k stacks today. The fact they don't support them today has much less to do with 4k being an unattainable limit as it does with people simply not bothering to implement it on their platform. IMHO there seems to currently be a mismatch between it's maintainance cost and the actual number of users. That's in my opinion the main problem with it, no matter in which direction it gets resolved. Perhaps that's true on x86, but in general I take issue with that. On sh we've had to do very little maintenance for it and most shipping products are using it today (at least on MMU-Linux, we don't bother with it on nommu). Most of the problems we ran in to with 4k stacks tended to be stuff that we wanted to fix for 8k anyways. I suspect that this case is true for the other embedded platforms also. Note that on sh we also conditionalize IRQ stacks separately, so while they are often used together, it's possible to use 4k stacks without resorting to IRQ stacks (as m68knommu also seems to do). -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Paul Mundt [EMAIL PROTECTED] Date: Thu, 28 Aug 2008 09:32:13 +0900 On Wed, Aug 27, 2008 at 08:35:44PM +0300, Adrian Bunk wrote: CONFIG_DEBUG_STACKOVERFLOW should give you the same information, and if wanted with an arbitrary limit. In some cases, yes. In the CONFIG_DEBUG_STACKOVERFLOW case the check is only performed from do_IRQ(), which is sporadic at best, especially on tickless. While it catches some things, it's not a complete solution in and of iteslf. BTW, on sparc64 we have a stack overflow checker that runs via the profiling _mcount hook. So every function call we check if the stack is getting overused. If so, we jump onto a special static debugging stack and print the stack overflow message. And yes it works with IRQ stacks which is all that sparc64 uses nowadays. Perhaps this is useful enough to make generic. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Wed, Aug 27, 2008 at 05:46:05PM -0700, David Miller wrote: From: Paul Mundt [EMAIL PROTECTED] Date: Thu, 28 Aug 2008 09:32:13 +0900 On Wed, Aug 27, 2008 at 08:35:44PM +0300, Adrian Bunk wrote: CONFIG_DEBUG_STACKOVERFLOW should give you the same information, and if wanted with an arbitrary limit. In some cases, yes. In the CONFIG_DEBUG_STACKOVERFLOW case the check is only performed from do_IRQ(), which is sporadic at best, especially on tickless. While it catches some things, it's not a complete solution in and of iteslf. BTW, on sparc64 we have a stack overflow checker that runs via the profiling _mcount hook. So every function call we check if the stack is getting overused. If so, we jump onto a special static debugging stack and print the stack overflow message. And yes it works with IRQ stacks which is all that sparc64 uses nowadays. Perhaps this is useful enough to make generic. Thanks for the pointer, I'll take a look at it! -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Paul Mundt wrote: On Wed, Aug 27, 2008 at 02:58:30PM +0300, Adrian Bunk wrote: On Tue, Aug 26, 2008 at 05:28:37PM -0700, Linus Torvalds wrote: On Wed, 27 Aug 2008, Adrian Bunk wrote: When did we get callpaths like like nfs+xfs+md+scsi reliably working with 4kB stacks on x86-32? XFS may never have been usable, but the rest, sure. And you seem to be making this whole argument an excuse to SUCK, adn an excuse to let gcc crap even more on our stack space. Why? Why aren't you saying that we should be able to do better? Instead, you seem to asking us to do even worse than we do now? My main point is: - getting 4kB stacks working reliably is a hard task - having an eye on gcc increasing the stack usage, and fixing it if required, is relatively easy If we should be able to do better at getting (and keeping) 4kB stacks working, then coping with possible inlining problems caused by gcc should not be a big problem for us. Out of the architectures you've mentioned for 4k stacks, they also tend to do IRQ stacks, which is something you seem to have overlooked. In addition to that, debugging the runaway stack users on 4k tends to be easier anyways since you end up blowing the stack a lot sooner. On sh we've had pretty good luck with it, though most of our users are using fairly deterministic workloads and continually profiling the footprint. Anything that runs away or uses an insane amount of stack space needs to be fixed well before that anyways, so catching it sooner is always preferable. I imagine the same case is true for m68knommu (even sans IRQ stacks). Yep, definitely true for m68knommu in my experience. I haven't had any problems with 4k stacks recently. But yes the workloads do tend to be constrained - and almost never use any of the more exotic filesystems or drivers. Things might be more sensitive on x86, but it's certainly not something that's a huge problem for the various embedded platforms to wire up, whether they want to go the IRQ stack route or not. In any event, lack of support for something on embedded architectures in the kernel is more often due to apathy/utter indifference on the part of the architecture maintainer rather than being indicative of any intrinsic difficulty in supporting the thing in question. Most new features on the lesser maintained architectures tend to end up there either out of peer pressure or copying-and-pasting accidents rather than any sort of design. ;-) Indeed :-) Regards Greg Greg Ungerer -- Chief Software Dude EMAIL: [EMAIL PROTECTED] Secure Computing CorporationPHONE: +61 7 3435 2888 825 Stanley St, FAX: +61 7 3891 3630 Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 10:35:05AM -0700, Linus Torvalds wrote: On Tue, 26 Aug 2008, Rusty Russell wrote: Your workaround is very random, and that scares me. I think a huge number of CPUs needs a real solution (an actual cpumask allocator, then do something clever if we come across an actual fastpath). The thing is, the inlining thing is a separate issue. Yes, the cpumasks were what made stack pressure so critical to begin with, but no, a 400-byte stack frame in a deep callchain isn't acceptable _regardless_ of any cpumask_t issues. Gcc inlining is a total and utter pile of shit. And _that_ is the problem. I seriously think we shouldn't allow gcc to inline anything at all unless we tell it to. That's how it used to work, and quite frankly, that's how it _should_ work. The downsides of inlining are big enough from both a debugging and a real code generation angle (eg stack usage like this), that the upsides (_somesimes_ smaller kernel, possibly slightly faster code) simply aren't relevant. ... It would be interesting to see what -fno-default-inline does to the kernel. It really would get rid of a _lot_ of gcc version issues too. Inlining behavior of gcc has long been a problem for us. I added -fno-inline-functions-called-once -fno-early-inlining to KBUILD_CFLAGS, and (with gcc 4.3) that increased the size of my kernel image by 2%. And when David's -fwhole-program --combine will become ready the cost of disallowing gcc to inline functions will most likely increase. A debugging option (for better traces) to disallow gcc some inlining might make sense (and might even make sense for distributions to enable in their kernels), but when you go to use cases that require really small kernels the cost is too high. But if you don't trust gcc's inlining you should revert commit 3f9b5cc018566ad9562df0648395649aebdbc5e0 that increases gcc's freedom regarding what to inline in 2.6.27 - what gcc 4.2 does in the case of the regression tracked as Bugzilla #11276 is really not funny (two callers - function not inlined; gcc seems to emit the function although both callers later get removed (or at least should be removed) by dead code elimination). Linus cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 11:40:10AM -0700, Linus Torvalds wrote: On Tue, 26 Aug 2008, Adrian Bunk wrote: A debugging option (for better traces) to disallow gcc some inlining might make sense (and might even make sense for distributions to enable in their kernels), but when you go to use cases that require really small kernels the cost is too high. You ignore the fact that it's really not just about debugging. I had in mind that we anyway have to support it for tiny kernels. I simply don't see that we add kconfig options for 5kB of code for tiny kernels but remove something like this that can cause size increases 1%. Inlining really isn't the great tool some people think it is. Especially not since gcc stack allocation is so horrid that it won't re-use stack slots etc (which I don't disagree with per se - it's _hard_ to re-use stack slots while still allowing code scheduling). gcc's stack allocation has become better (that's why we disable unit-at-a-time only for gcc 3.4 on i386). NOTE! I also would never claim that _our_ choices of inline are all that great, and we've often inlined too much or not inlined things that really could be inlined. But at least when a developer says inline (or forgets to say it), we have somebody to blame. When the compiler does insane things that doesn't suit us, we're just screwed. Most LOCs of the kernel are not written by people like you or Al Viro or David Miller, and the average kernel developer is unlikely to do it as good as gcc. For the average driver the choice is realistically between inline's randomly sprinkled across the driver and no inline's, leave it to gcc. And code evolves during the years from tiny with 1 caller to huge with many callers. BTW: I just ran checkstack on a (roughly) allyesconfig kernel, and we have a new driver that allocates unsigned char recvbuf[1500]; on the stack... But if you don't trust gcc's inlining you should revert commit 3f9b5cc018566ad9562df0648395649aebdbc5e0 that increases gcc's freedom regarding what to inline in 2.6.27 Actually, that just allows gcc to _not_ inline. Which is probably ok. (Well, it would be ok if gcc did it well enough, it obviously has some problems at times). With the gcc inline's static functions you complain about we have 4-5 years of experience. Suddenly allowing 4 release series of gcc to ignore any inline's is a completely new area for us. I'd generally agree with giving gcc more freedom here, but I'd rather do it right by removing tons of wrong inline's than doing one global change hoping that it will make things better. And whether the optimized inlining actually makes the kernel bigger or smaller depends in my experience on the .config and the gcc version. Linus cu Adrian [1] there are some rare exceptions -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 11:47:01AM -0700, Linus Torvalds wrote: On Tue, 26 Aug 2008, Adrian Bunk wrote: I added -fno-inline-functions-called-once -fno-early-inlining to KBUILD_CFLAGS, and (with gcc 4.3) that increased the size of my kernel image by 2%. Btw, did you check with just -fno-inline-functions-called-once? The -fearly-inlining decisions _should_ be mostly right. If gcc sees early that a function is so small (even without any constant propagation etc) that it can be inlined, it's probably right. The inline-functions-called-once thing is what causes even big functions to be inlined, and that's where you find the big downsides too (eg the stack usage). -fno-inline-functions-called-once alone costs me nearly 1% in code size. And I'd expect it to become more with -fwhole-program --combine. If you think we have too many stacksize problems I'd suggest to consider removing the choice of 4k stacks on i386, sh and m68knommu instead of using -fno-inline-functions-called-once: Now that 32bit x86 is no longer used for extreme highend configurations the only serious usecase for 4k stacks are AFAIK space savings on embedded archs. 4k stacks have caused us much pain [1], and the cases where gcc inlined too much were the easy ones. I'm not saying that I'd like removing the choice of 4k stacks, but if we want to reduce the number of stack related problems that's IMHO the better alternative. Linus cu Adrian [1] AFAIR some callpaths in the kernel are still too big BTW: In case anyone wonders about why I suggest removing 4k stacks: My position is that 4k stacks should either be enabled unconditionally or no longer offered at all. And if we remove 4k stacks from 32bit x86 it's no longer realistically maintainable for other architectures. -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, 26 Aug 2008, Adrian Bunk wrote: If you think we have too many stacksize problems I'd suggest to consider removing the choice of 4k stacks on i386, sh and m68knommu instead of using -fno-inline-functions-called-once: Don't be silly. That makes the problem _worse_. We're much better off with a 1% code-size reduction than forcing big stacks on people. The 4kB stack option is also a good way of saying if it works with this, then 8kB is certainly safe. And embedded people (the ones that might care about 1% code size) are the ones that would also want smaller stacks even more! Linus -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Parag Warudkar wrote: On Tue, Aug 26, 2008 at 5:04 PM, Linus Torvalds [EMAIL PROTECTED] wrote: And embedded people (the ones that might care about 1% code size) are the ones that would also want smaller stacks even more! This is something I never understood - embedded devices are not going to run more than a few processes and 4K*(Few Processes) IMHO is not worth a saving now a days even in embedded world given falling memory prices. Or do I misunderstand? Embedded applications span a huge range of sizes, from the very small devices to which you refer, to quite complex devices. The cable settop boxes we develop have over a hundred interrupt sources, typically run 250-300 threads, and have 192+ MiB of memory. For all that, we are very cost sensitive and are under constant pressure to come up with reliable ways to save memory. Parag -- David VomLehn -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 02:04:57PM -0700, Linus Torvalds wrote: On Tue, 26 Aug 2008, Adrian Bunk wrote: If you think we have too many stacksize problems I'd suggest to consider removing the choice of 4k stacks on i386, sh and m68knommu instead of using -fno-inline-functions-called-once: Don't be silly. That makes the problem _worse_. We're much better off with a 1% code-size reduction than forcing big stacks on people. The 4kB stack option is also a good way of saying if it works with this, then 8kB is certainly safe. ... You implicitely assume both would solve the same problem. While 4kB stacks are something we anyway never got 100% working, the cases where gcc inlining functions causes a critical increase in stack usage are usually not that hard to find, and once found the fix is trivial. We should anyway monitor stack usages better since we have frequent programming errors in this area, and problems caused by gcc can this way be detected en passant. You have a good point that aiming at 4kB makes 8kB a very safe choice. But I do not think the problem you'd solve with -fno-inline-functions-called-once is big enough to warrant the size increase it causes. Linus cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 04:00:33PM -0700, David VomLehn wrote: Parag Warudkar wrote: On Tue, Aug 26, 2008 at 5:04 PM, Linus Torvalds [EMAIL PROTECTED] wrote: And embedded people (the ones that might care about 1% code size) are the ones that would also want smaller stacks even more! This is something I never understood - embedded devices are not going to run more than a few processes and 4K*(Few Processes) IMHO is not worth a saving now a days even in embedded world given falling memory prices. Or do I misunderstand? Embedded applications span a huge range of sizes, from the very small devices to which you refer, to quite complex devices. The cable settop boxes we develop have over a hundred interrupt sources, typically run 250-300 threads, and have 192+ MiB of memory. For all that, we are very cost sensitive and are under constant pressure to come up with reliable ways to save memory. As you say correctly the term embedded gets used for many different devices. And if you have 192+ MiB of memory you have so much that all these kernel size discussions don't really matter. David VomLehn cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, 26 Aug 2008, Parag Warudkar wrote: This is something I never understood - embedded devices are not going to run more than a few processes and 4K*(Few Processes) IMHO is not worth a saving now a days even in embedded world given falling memory prices. Or do I misunderstand? Well, by that argument, 1% of kernel size doesn't matter either.. 1% of a kernel for an embedded device is roughly 10-30kB or so depending on how small you make the configuration. If that matters, then so should the difference of 3-8 processes' kernel stack usage when you have a 4k/8k stack choice. And they _all_ will have at least 3-8 processes on them. Even the simplest ones will tend to have many more. Linus -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Wed, 27 Aug 2008, Adrian Bunk wrote: We're much better off with a 1% code-size reduction than forcing big stacks on people. The 4kB stack option is also a good way of saying if it works with this, then 8kB is certainly safe. You implicitely assume both would solve the same problem. I'm just saying that your logic doesn't hold water. If we can save kernel stack usage, then a 1% increase in kernel size is more than worth it. While 4kB stacks are something we anyway never got 100% working What? Don't be silly. Linux _historically_ always used 4kB stacks. No, they are likely not usable on x86-64, but dammit, they should be more than usable on x86-32 still. But I do not think the problem you'd solve with -fno-inline-functions-called-once is big enough to warrant the size increase it causes. You continually try to see the inlining as a single solution to one problem (debuggability, stack, whatever). The biggest problem with gcc inlining has always been that it has been _unpredictable_. It causes problems in many different ways. It has caused stability issues due to gcc versions doing random things. It causes the stack expansion. It makes stack traces harder for debugging, etc. If it was any one thing, I wouldn't care. But it's exactly the fact that it causes all these problems in different areas. Linus -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 04:51:52PM -0700, Linus Torvalds wrote: On Wed, 27 Aug 2008, Adrian Bunk wrote: We're much better off with a 1% code-size reduction than forcing big stacks on people. The 4kB stack option is also a good way of saying if it works with this, then 8kB is certainly safe. You implicitely assume both would solve the same problem. I'm just saying that your logic doesn't hold water. If we can save kernel stack usage, then a 1% increase in kernel size is more than worth it. From some tests the size increase seems to become bigger for smaller kernels, but I don't have any really good data. An interesting question is why most of our architectures for embedded devices only offer bigger stacks: The only architectures offering a 4kB stacks option are: - m68knommu - sh - 32bit x86 The following architectures that are used in embedded devices always use 8kB stacks (or bigger) in your tree: - arm - avr32 - blackfin - cris - frv - h8300 - m32r - m68k - mips - mn10300 (has an #ifdef CONFIG_4KSTACKS but no kconfig option) - powerpc - xtensa While 4kB stacks are something we anyway never got 100% working What? Don't be silly. Linux _historically_ always used 4kB stacks. No, they are likely not usable on x86-64, but dammit, they should be more than usable on x86-32 still. When did we get callpaths like like nfs+xfs+md+scsi reliably working with 4kB stacks on x86-32? ... Linus cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
Linus Torvalds wrote: On Tue, 26 Aug 2008, Parag Warudkar wrote: This is something I never understood - embedded devices are not going to run more than a few processes and 4K*(Few Processes) IMHO is not worth a saving now a days even in embedded world given falling memory prices. Or do I misunderstand? Well, by that argument, 1% of kernel size doesn't matter either.. 1% of a kernel for an embedded device is roughly 10-30kB or so depending on how small you make the configuration. If that matters, then so should the difference of 3-8 processes' kernel stack usage when you have a 4k/8k stack choice. And they _all_ will have at least 3-8 processes on them. Even the simplest ones will tend to have many more. I have some simple devices (network access/routers) with 8MB of RAM, at power up not really being configured to do anything running 25 processes. (Heck there is over 10 kernel processes running!). Configure some interfaces and services and that will easily push past 40. I'd be happy with a 160k saving :-) The init memory being freed at the end of the kernel boot is 88k, 4k stacks could save more than that. Regards Greg Greg Ungerer -- Chief Software Dude EMAIL: [EMAIL PROTECTED] Secure Computing CorporationPHONE: +61 7 3435 2888 825 Stanley St, FAX: +61 7 3891 3630 Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 7:47 PM, Linus Torvalds [EMAIL PROTECTED] wrote: If that matters, then so should the difference of 3-8 processes' kernel stack usage when you have a 4k/8k stack choice. The savings part -financial ones- are not always realizable with the way memory is priced/sized/fitted. Savings in few Mb of Kernel stack are not necessarily going to allow getting rid of a single memory chip of 64M or so. Either that or embedded manufacturing/configurations are different than the desktop world. (If my device has 2 memory slots and my user space requires 100Mb including kernel memory - I anyways have to put in 64Mx2 there to take advantage of mass manufactured, general purpose memory - so no big deal if I saved 1.2Mb in Kernel stack or not. And savings of 64Mb Kernel memory are not feasible anyways to allow user space to work with 64Mb.) On the other hand reducing user space memory usage on those devices (not counting savings from kernel stack size) is a way more attractive option. And although you said in your later reply that Linux x86 with 4K stacks should be more than usable - my experiences running a untainted desktop/file server with 4K stack have been always disastrous XFS or not. It _might_ work for some well defined workloads but you would not want to risk 4K stacks otherwise. I understand the having 4K stack option as a non-default for very specific workloads is a good idea but apart from that I think no one else seems to bother with reducing stack sizes (by no one I mean other OSes.) Parag -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 8:53 PM, Greg Ungerer [EMAIL PROTECTED] wrote: I have some simple devices (network access/routers) with 8MB of RAM, at power up not really being configured to do anything running 25 processes. (Heck there is over 10 kernel processes running!). Configure some interfaces and services and that will easily push past 40. I'd be happy with a 160k saving :-) So you really need to run all 25 processes on that 8Mb box? (For reference even the NGW100 development board comes with 16Mb RAM). Even if you do need those all 25 processes on the 8Mb box, fixing the memory usage of those user space hogs is lot better than trying to save 160Kb in kernel stacks. Last I looked, user space wasn't particularly frugal with memory usage. Parag -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, 26 Aug 2008, Parag Warudkar wrote: And although you said in your later reply that Linux x86 with 4K stacks should be more than usable - my experiences running a untainted desktop/file server with 4K stack have been always disastrous XFS or not. It _might_ work for some well defined workloads but you would not want to risk 4K stacks otherwise. Umm. How long? 4kB used to be the _only_ choice. And no, there weren't even irq stacks. So that 4kB was not just the whole kernel call-chain, it was also all the irq nesting above it. And yes, we've gotten much worse over time, and no, I can't really suggest going back to that in general. The code bloat has certainly been accompanied by a stack bloat too. But part of it is definitely gcc. Some versions of gcc used to be absolutely _horrid_ when it came to stack usage, especially with some flags, and especially with the crazy inlining that module-at-a-time caused. But I'd be really happy if some embedded people tried to take some of that bloat back, and aim for 4kB stacks. Because it's definitely not unrealistic. At least it _shouldn't_ be. And a lot of the cases of us having structures on the stack is actually not worth it, and tends to be about being lazy rather than anything else. Linus -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 9:31 PM, Greg Ungerer [EMAIL PROTECTED] wrote: And the pressure will still be on in _real_ products to reduce the RAM footprint as much as possible. There are exceptions but generally less is cheaper. Simple economics really. Well, sure - but the industry as a whole seems to have gone the other way - do more with more at the similar or lower price points! By that definition of less is better we should try and make the kernel memory pageable (or has someone already done that?) - Windows does it, by default ;) Parag -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, Aug 26, 2008 at 9:49 PM, Linus Torvalds [EMAIL PROTECTED] wrote: On Tue, 26 Aug 2008, Parag Warudkar wrote: And although you said in your later reply that Linux x86 with 4K stacks should be more than usable - my experiences running a untainted desktop/file server with 4K stack have been always disastrous XFS or not. It _might_ work for some well defined workloads but you would not want to risk 4K stacks otherwise. Umm. How long? IIRC the last I tried 4K stacks with x86 was on 2.6.21 - Fedora 7 kernel, around June 07 time frame. The oops included a ugly and long call trace that I still remember. And a lot of the cases of us having structures on the stack is actually not worth it, and tends to be about being lazy rather than anything else. What about deep call chains? The problem with the uptake of 4K stacks seems to be that is not reliably provable that it will work under all circumstances. Parag -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
On Tue, 26 Aug 2008, Parag Warudkar wrote: What about deep call chains? The problem with the uptake of 4K stacks seems to be that is not reliably provable that it will work under all circumstances. Umm. Neither is 8k stacks. Nobody proved anything. But yes, some subsystems have insanely deep call chains. And yes, things like the VFS recursion (for symlinks) makes that deeper yet for filesystems, although only on the lookup path. And that is exactly the kind of thing that can exacerbate the problem of the compiler artificially making for a bigger stack footprint of a function (*). For things like the VFS layer, right now we allow a nesting level of 8, I think. If I remember correctly, it was 5 historically. Part of raising that depth, though, was that we actually moved the recursive part into fs/namei.c, and the nesting stack-depth was something pretty damn small when the filesystem used follow_link properly and let the VFS do it for it (ie the callchain to actually look up the link could be deep, but it would not recurse back, and instead just return a pointer, so that the actual _recursive_ part was just __do_follow_link() and is just a few words on the stack). So yes, we do have some deep callchains, but they tend to be pretty well managed for _good_ code. The problems tend to be the areas with lots of indirection layers, and yeah, XFS, MD and ACPI all have those kinds of things. In an embdedded world, many of those should be a non-issue, though. Linus (*) ie the function that _is_ on the deep chain doesn't actually need much of a stack footprint at all itself, but it may call a helper function that is _not_ in the deep chain, and if it gets inlined it may give its excessive stack footprint to the deep chain - and this is _exactly_ the problem that happened with inlining load_module(). -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
