Following test case runs out of stack space when gcc tries to compute factorial (1599999). I have a patch which does two simple things. 1) it rewrites tree_fold_factorial function into its non-recursive version, and 2) it sets a limit before deciding to call chrec_evaluate. This limit is arbitrary in this patch. Author of the algorithm may want to decide when to stop evaluating feasibility of this optimization. Note that even with this limit, the computed factorial overflows. So, even a much smaller limit is needed if this value is significant.
/* bad.c */ static unsigned int *buffer; void FUNC (void) { unsigned int *base; int i, j; for (i = 0; i < 4; i++) for (j = 0; j < 1600000; j++) *base++ = buffer[j]; } % mygccm5 -c -O1 bad.c Out of stack space. Try running 'limit stacksize unlimited' in the shell to raise its limit. -- Summary: Simple loop runs out of stack at -O1 Product: gcc Version: 4.0.0 Status: UNCONFIRMED Severity: normal Priority: P2 Component: tree-optimization AssignedTo: unassigned at gcc dot gnu dot org ReportedBy: fjahanian at apple dot com CC: gcc-bugs at gcc dot gnu dot org GCC build triplet: powerpc-apple-darwin GCC host triplet: powerpc-apple-darwin GCC target triplet: powerpc-apple-darwin http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20216