Ping. Teresa
On Wed, Jul 18, 2012 at 8:48 AM, Teresa Johnson <tejohn...@google.com> wrote: > Ping (retrying ping in plain text mode so that it goes through properly). > > Thanks, > Teresa > > On Wed, Jul 11, 2012 at 10:42 AM, Teresa Johnson <tejohn...@google.com> wrote: >> Ports some patches related to improving FDO program summary information >> and using it to guide loop unrolling from google branches to mainline. >> The patch is enhanced to add additional summary information to aid >> in determining hot/cold decisions. >> >> The original patch description is at: >> http://gcc.gnu.org/ml/gcc-patches/2012-06/msg00437.html >> and further discussion about incorporating onto mainline is at: >> http://gcc.gnu.org/ml/gcc-patches/2012-06/threads.html#00414 >> >> Honza, can you take a look to see if this patch meets your needs? >> >> Full description: >> >> This patch adds new program summary information to the gcov >> profile files that indicate how many profiled counts compose >> the majority of the program's execution time. This is used to >> provide an indication of the overall code size of the program. >> >> The new profile summary information is then used to guide >> codesize based unroll and peel decisions, to prevent those >> optimizations from increasing code size too much when the >> program may be sensitive to icache effects. >> >> This patch also pulls in dependent portions of google/main r187660 that cache >> additional loop analysis results in the niter_desc auxiliary information >> hanging off the loop structure (the optimization portions of that >> change are not included here, and have an outstanding review request >> for mainline). >> >> Bootstrapped and tested on x86_64-unknown-linux-gnu. Ok for trunk? >> >> Thanks, >> Teresa >> >> 2012-07-11 Teresa Johnson <tejohn...@google.com> >> >> * libgcc/libgcov.c (sort_by_reverse_gcov_value): New function. >> (gcov_compute_cutoff_values): Ditto. >> (gcov_exit): Call gcov_compute_cutoff_values and merge new summary >> information. >> * gcc/doc/invoke.texi (roll much): Document new options >> -fpeel-codesize-limit and -funroll-codesize-limit, and new params >> codesize-hotness-threshold and unrollpeel-hotness-threshold. >> * gcc/gcov-io.c (gcov_write_summary): Write new summary info. >> (gcov_read_summary): Read new summary info. >> * gcc/gcov-io.h (GCOV_TAG_SUMMARY_LENGTH): Update for new summary >> info. >> (struct gcov_ctr_summary): Add new summary info: num_hot_counters and >> hot_cutoff_value. >> * gcc/loop-unroll.c (code_size_limit_factor): New function. >> (decide_unroll_runtime_iterations): Call code_size_limit_factor >> to control the unroll factor, and retrieve number of branches from >> niter_desc instead of via function that walks loop. >> (decide_peel_simple, decide_unroll_stupid): Ditto. >> * gcc/coverage.c (read_counts_file): Propagate new summary info. >> * gcc/loop-iv.c (get_simple_loop_desc): Invoke new analyze_loop_insns >> function, and add guards to enable this function to work for the >> outermost loop. >> * gcc/common.opt: Add -fpeel-codesize-limit and >> -funroll-codesize-limit. >> * gcc/cfgloop.c (insn_has_fp_set, analyze_loop_insns): New functions. >> (num_loop_branches): Remove. >> * gcc/cfgloop.h (struct niter_desc): Added new fields to cache >> additional loop analysis information. >> (num_loop_branches): Remove. >> (analyze_loop_insns): Declare. >> * gcc/params.def (PARAM_UNROLLPEEL_CODESIZE_THRESHOLD): Add. >> (PARAM_UNROLLPEEL_HOTNESS_THRESHOLD): Ditto. >> * gcc/gcov-dump.c (tag_summary): Dump new summary info. >> >> Index: libgcc/libgcov.c >> =================================================================== >> --- libgcc/libgcov.c (revision 189413) >> +++ libgcc/libgcov.c (working copy) >> @@ -276,6 +276,120 @@ gcov_version (struct gcov_info *ptr, gcov_unsigned >> return 1; >> } >> >> +/* Used by qsort to sort gcov values in descending order. */ >> + >> +static int >> +sort_by_reverse_gcov_value (const void *pa, const void *pb) >> +{ >> + const gcov_type a = *(gcov_type const *)pa; >> + const gcov_type b = *(gcov_type const *)pb; >> + >> + if (b > a) >> + return 1; >> + else if (b == a) >> + return 0; >> + else >> + return -1; >> +} >> + >> +/* Determines the number of counters required to cover a given percentage >> + of the total sum of execution counts in the summary, which is then also >> + recorded in SUM. */ >> + >> +static void >> +gcov_compute_cutoff_values (struct gcov_summary *sum) >> +{ >> + struct gcov_info *gi_ptr; >> + const struct gcov_fn_info *gfi_ptr; >> + const struct gcov_ctr_info *ci_ptr; >> + struct gcov_ctr_summary *cs_ptr; >> + unsigned t_ix, f_ix, i, ctr_info_ix, index; >> + gcov_unsigned_t c_num; >> + gcov_type *value_array; >> + gcov_type cum, cum_cutoff; >> + char *cutoff_str; >> + unsigned cutoff_perc; >> + >> +#define CUM_CUTOFF_PERCENT_TIMES_10 999 >> + cutoff_str = getenv ("GCOV_HOTCODE_CUTOFF_TIMES_10"); >> + if (cutoff_str && strlen (cutoff_str)) >> + cutoff_perc = atoi (cutoff_str); >> + else >> + cutoff_perc = CUM_CUTOFF_PERCENT_TIMES_10; >> + >> + /* This currently only applies to arc counters. */ >> + t_ix = GCOV_COUNTER_ARCS; >> + >> + /* First check if there are any counts recorded for this counter. */ >> + cs_ptr = &(sum->ctrs[t_ix]); >> + if (!cs_ptr->num) >> + return; >> + >> + /* Determine the cumulative counter value at the specified cutoff >> + percentage and record the percentage for use by gcov consumers. >> + Check for overflow when sum_all is multiplied by the cutoff_perc, >> + and if so, do the divide first. */ >> + if (cs_ptr->sum_all*cutoff_perc < cs_ptr->sum_all) >> + /* Overflow, do the divide first. */ >> + cum_cutoff = cs_ptr->sum_all / 1000 * cutoff_perc; >> + else >> + /* Otherwise multiply first to get the correct value for small >> + values of sum_all. */ >> + cum_cutoff = (cs_ptr->sum_all * cutoff_perc) / 1000; >> + >> + /* Next, walk through all the per-object structures and save each of >> + the count values in value_array. */ >> + index = 0; >> + value_array = (gcov_type *) malloc (sizeof (gcov_type) * cs_ptr->num); >> + for (gi_ptr = gcov_list; gi_ptr; gi_ptr = gi_ptr->next) >> + { >> + if (!gi_ptr->merge[t_ix]) >> + continue; >> + >> + /* Find the appropriate index into the gcov_ctr_info array >> + for the counter we are currently working on based on the >> + existence of the merge function pointer for this object. */ >> + for (i = 0, ctr_info_ix = 0; i < t_ix; i++) >> + { >> + if (gi_ptr->merge[i]) >> + ctr_info_ix++; >> + } >> + for (f_ix = 0; f_ix != gi_ptr->n_functions; f_ix++) >> + { >> + gfi_ptr = gi_ptr->functions[f_ix]; >> + >> + if (!gfi_ptr || gfi_ptr->key != gi_ptr) >> + continue; >> + >> + ci_ptr = &gfi_ptr->ctrs[ctr_info_ix]; >> + /* Sanity check that there are enough entries in value_arry >> + for this function's counters. Gracefully handle the case when >> + there are not, in case something in the profile info is >> + corrupted. */ >> + c_num = ci_ptr->num; >> + if (index + c_num > cs_ptr->num) >> + c_num = cs_ptr->num - index; >> + /* Copy over this function's counter values. */ >> + memcpy (&value_array[index], ci_ptr->values, >> + sizeof (gcov_type) * c_num); >> + index += c_num; >> + } >> + } >> + >> + /* Sort all the counter values by descending value and finally >> + accumulate the values from hottest on down until reaching >> + the cutoff value computed earlier. */ >> + qsort (value_array, cs_ptr->num, sizeof (gcov_type), >> + sort_by_reverse_gcov_value); >> + for (cum = 0, c_num = 0; c_num < cs_ptr->num && cum < cum_cutoff; c_num++) >> + cum += value_array[c_num]; >> + /* Record the number of counters required to reach the cutoff value, >> + as well as the counter value that caused cum to reach the cutoff. */ >> + cs_ptr->num_hot_counters = c_num; >> + cs_ptr->hot_cutoff_value = c_num > 0 ? value_array[c_num-1] : 0; >> + free (value_array); >> +} >> + >> /* Dump the coverage counts. We merge with existing counts when >> possible, to avoid growing the .da files ad infinitum. We use this >> program's checksum to make sure we only accumulate whole program >> @@ -347,6 +461,7 @@ gcov_exit (void) >> } >> } >> } >> + gcov_compute_cutoff_values (&this_prg); >> >> { >> /* Check if the level of dirs to strip off specified. */ >> @@ -598,7 +713,14 @@ gcov_exit (void) >> if (gi_ptr->merge[t_ix]) >> { >> if (!cs_prg->runs++) >> - cs_prg->num = cs_tprg->num; >> + { >> + cs_prg->num = cs_tprg->num; >> + if (cs_tprg->num_hot_counters > cs_prg->num_hot_counters) >> + { >> + cs_prg->num_hot_counters = cs_tprg->num_hot_counters; >> + cs_prg->hot_cutoff_value = cs_tprg->hot_cutoff_value; >> + } >> + } >> cs_prg->sum_all += cs_tprg->sum_all; >> if (cs_prg->run_max < cs_tprg->run_max) >> cs_prg->run_max = cs_tprg->run_max; >> Index: gcc/doc/invoke.texi >> =================================================================== >> --- gcc/doc/invoke.texi (revision 189413) >> +++ gcc/doc/invoke.texi (working copy) >> @@ -384,7 +384,7 @@ Objective-C and Objective-C++ Dialects}. >> -fno-sched-interblock -fno-sched-spec -fno-signed-zeros @gol >> -fno-toplevel-reorder -fno-trapping-math -fno-zero-initialized-in-bss @gol >> -fomit-frame-pointer -foptimize-register-move -foptimize-sibling-calls @gol >> --fpartial-inlining -fpeel-loops -fpredictive-commoning @gol >> +-fpartial-inlining -fpeel-codesize-limit -fpeel-loops >> -fpredictive-commoning @gol >> -fprefetch-loop-arrays @gol >> -fprofile-correction -fprofile-dir=@var{path} -fprofile-generate @gol >> -fprofile-generate=@var{path} @gol >> @@ -416,7 +416,7 @@ Objective-C and Objective-C++ Dialects}. >> -ftree-reassoc @gol >> -ftree-sink -ftree-sra -ftree-switch-conversion -ftree-tail-merge @gol >> -ftree-ter -ftree-vect-loop-version -ftree-vectorize -ftree-vrp @gol >> --funit-at-a-time -funroll-all-loops -funroll-loops @gol >> +-funit-at-a-time -funroll-all-loops -funroll-loops -funroll-codesize-limit >> @gol >> -funsafe-loop-optimizations -funsafe-math-optimizations -funswitch-loops >> @gol >> -fvariable-expansion-in-unroller -fvect-cost-model -fvpt -fweb @gol >> -fwhole-program -fwpa -fuse-linker-plugin @gol >> @@ -8505,6 +8505,14 @@ the loop is entered. This usually makes programs >> @option{-funroll-all-loops} implies the same options as >> @option{-funroll-loops}. >> >> +@item -funroll-codesize-limit >> +@opindex funroll-codesize-limit >> +Limit loop unrolling of non-const non-FP loops in a profile feedback >> compilation >> +under estimates of a large code footprint. Enabled by default with >> +@option{-fprofile-use}. Code size and execution weight thresholds are >> controlled >> +by the @option{unrollpeel-codesize-threshold} and >> +@option{unrollpeel-hotness-threshold} parameters. >> + >> @item -fpeel-loops >> @opindex fpeel-loops >> Peels loops for which there is enough information that they do not >> @@ -8513,6 +8521,14 @@ roll much (from profile feedback). It also turns >> >> Enabled with @option{-fprofile-use}. >> >> +@item -fpeel-codesize-limit >> +@opindex fpeel-codesize-limit >> +Limit loop peeling of non-const non-FP loops in a profile feedback >> compilation >> +under estimates of a large code footprint. Enabled by default with >> +@option{-fprofile-use}. Code size and execution weight thresholds are >> controlled >> +by the @option{unrollpeel-codesize-threshold} and >> +@option{unrollpeel-hotness-threshold} parameters. >> + >> @item -fmove-loop-invariants >> @opindex fmove-loop-invariants >> Enables the loop invariant motion pass in the RTL loop optimizer. Enabled >> @@ -8864,6 +8880,15 @@ The maximum number of iterations of a loop to be s >> @item max-completely-peel-loop-nest-depth >> The maximum depth of a loop nest suitable for complete peeling. >> >> +@item unrollpeel-codesize-threshold >> +Maximum profile-based code size footprint estimate for loop unrolling and >> +peeling. >> + >> +@item unrollpeel-hotness-threshold >> +Maximum ratio of total execution count to loop entry block count under which >> +most profile-based code size estimates will be ignored for loop unrolling >> and >> +peeling. >> + >> @item max-unswitch-insns >> The maximum number of insns of an unswitched loop. >> >> Index: gcc/gcov-io.c >> =================================================================== >> --- gcc/gcov-io.c (revision 189413) >> +++ gcc/gcov-io.c (working copy) >> @@ -376,6 +376,8 @@ gcov_write_summary (gcov_unsigned_t tag, const str >> for (csum = summary->ctrs, ix = GCOV_COUNTERS_SUMMABLE; ix--; csum++) >> { >> gcov_write_unsigned (csum->num); >> + gcov_write_unsigned (csum->num_hot_counters); >> + gcov_write_counter (csum->hot_cutoff_value); >> gcov_write_unsigned (csum->runs); >> gcov_write_counter (csum->sum_all); >> gcov_write_counter (csum->run_max); >> @@ -495,6 +497,8 @@ gcov_read_summary (struct gcov_summary *summary) >> for (csum = summary->ctrs, ix = GCOV_COUNTERS_SUMMABLE; ix--; csum++) >> { >> csum->num = gcov_read_unsigned (); >> + csum->num_hot_counters = gcov_read_unsigned (); >> + csum->hot_cutoff_value = gcov_read_counter (); >> csum->runs = gcov_read_unsigned (); >> csum->sum_all = gcov_read_counter (); >> csum->run_max = gcov_read_counter (); >> Index: gcc/gcov-io.h >> =================================================================== >> --- gcc/gcov-io.h (revision 189413) >> +++ gcc/gcov-io.h (working copy) >> @@ -310,7 +310,7 @@ typedef HOST_WIDEST_INT gcov_type; >> #define GCOV_TAG_OBJECT_SUMMARY ((gcov_unsigned_t)0xa1000000) /* Obsolete >> */ >> #define GCOV_TAG_PROGRAM_SUMMARY ((gcov_unsigned_t)0xa3000000) >> #define GCOV_TAG_SUMMARY_LENGTH \ >> - (1 + GCOV_COUNTERS_SUMMABLE * (2 + 3 * 2)) >> + (1 + GCOV_COUNTERS_SUMMABLE * (3 + 4 * 2)) >> >> /* Counters that are collected. */ >> #define GCOV_COUNTER_ARCS 0 /* Arc transitions. */ >> @@ -393,6 +393,10 @@ typedef HOST_WIDEST_INT gcov_type; >> struct gcov_ctr_summary >> { >> gcov_unsigned_t num; /* number of counters. */ >> + gcov_unsigned_t num_hot_counters;/* number of counters to reach a given >> + percent of sum_all. */ >> + gcov_type hot_cutoff_value; /* value of smallest counter included in >> + num_hot_counters. */ >> gcov_unsigned_t runs; /* number of program runs */ >> gcov_type sum_all; /* sum of all counters accumulated. */ >> gcov_type run_max; /* maximum value on a single run. */ >> Index: gcc/loop-unroll.c >> =================================================================== >> --- gcc/loop-unroll.c (revision 189413) >> +++ gcc/loop-unroll.c (working copy) >> @@ -32,6 +32,7 @@ along with GCC; see the file COPYING3. If not see >> #include "hashtab.h" >> #include "recog.h" >> #include "target.h" >> +#include "gcov-io.h" >> >> /* This pass performs loop unrolling and peeling. We only perform these >> optimizations on innermost loops (with single exception) because >> @@ -150,6 +151,75 @@ static void combine_var_copies_in_loop_exit (struc >> basic_block); >> static rtx get_expansion (struct var_to_expand *); >> >> +/* Determine whether and how much LOOP unrolling/peeling should be >> constrained >> + based on code footprint estimates. Returns the codesize-based factor to >> be >> + divided into the max instructions in an unrolled or peeled loop: >> + 1) For size <= threshold, do not limit (by returning 1). >> + 2) For threshold < size < 2*threshold, reduce maximum allowed peeled or >> + unrolled instructions according to loop hotness. >> + 3) For threshold >= 2*threshold, disable unrolling/peeling (by returning >> + INT_MAX). */ >> + >> +static int >> +code_size_limit_factor(struct loop *loop) >> +{ >> + unsigned size_threshold; >> + struct niter_desc *desc = get_simple_loop_desc (loop); >> + gcov_type sum_to_header_ratio; >> + int hotness_ratio_threshold; >> + int limit_factor; >> + >> + /* First check if the application has a large codesize footprint. >> + This is estimated from FDO profile summary information for the >> + program, where the num_hot_counters indicates the number of hottest >> + counters (blocks) that compose most of the execution time of >> + the program. A large value would indicate a large flat execution >> + profile where icache misses may be a concern. */ >> + size_threshold = PARAM_VALUE (PARAM_UNROLLPEEL_CODESIZE_THRESHOLD); >> + if (!profile_info >> + || profile_info->num_hot_counters <= size_threshold >> + || !profile_info->sum_all) >> + return 1; >> + >> + /* Next, exclude some loops where unrolling/peeling may be more >> + important to overall performance. */ >> + >> + /* Ignore FP loops, which are more likely to benefit heavily from >> + unrolling. */ >> + if (desc->has_fp) >> + return 1; >> + >> + /* Next, set the value of the codesize-based unroll factor divisor which >> in >> + most loops will need to be set to a value that will reduce or eliminate >> + unrolling/peeling. */ >> + if (profile_info->num_hot_counters < size_threshold * 2) >> + { >> + /* For applications that are less than twice the codesize limit, allow >> + limited unrolling for very hot loops. */ >> + sum_to_header_ratio = profile_info->sum_all / loop->header->count; >> + hotness_ratio_threshold = PARAM_VALUE >> (PARAM_UNROLLPEEL_HOTNESS_THRESHOLD); >> + /* When the profile count sum to loop entry header ratio is smaller >> than >> + the threshold (i.e. the loop entry is hot enough, the divisor is >> set >> + to 1 so the unroll/peel factor is not reduced. When it is bigger >> + than the ratio, increase the divisor by the amount this ratio >> + is over the threshold, which will quickly reduce the unroll/peel >> + factor to zero as the loop's hotness reduces. */ >> + if (sum_to_header_ratio > hotness_ratio_threshold) >> + { >> + limit_factor = sum_to_header_ratio / hotness_ratio_threshold; >> + gcc_assert (limit_factor >= 1); >> + } >> + else >> + limit_factor = 1; >> + } >> + else >> + /* For appliations that are at least twice the codesize limit, set >> + the divisor to a large value that will force the unroll factor to 0. >> */ >> + limit_factor = INT_MAX; >> + >> + return limit_factor; >> +} >> + >> /* Unroll and/or peel (depending on FLAGS) LOOPS. */ >> void >> unroll_and_peel_loops (int flags) >> @@ -802,6 +872,7 @@ decide_unroll_runtime_iterations (struct loop *loo >> { >> unsigned nunroll, nunroll_by_av, i; >> struct niter_desc *desc; >> + int limit_factor = 1; >> >> if (!(flags & UAP_UNROLL)) >> { >> @@ -814,10 +885,26 @@ decide_unroll_runtime_iterations (struct loop *loo >> "\n;; Considering unrolling loop with runtime " >> "computable number of iterations\n"); >> >> + if (flag_unroll_codesize_limit) >> + { >> + /* Determine whether to limit code size growth from unrolling, >> + using FDO profile summary information that gives an >> + estimated number of executed blocks. */ >> + limit_factor = code_size_limit_factor (loop); >> + if (dump_file && limit_factor > 1) >> + { >> + fprintf (dump_file, >> + ";; Due to large code size footprint estimate, limit " >> + "max unrolled insns by divisor %d\n", limit_factor); >> + } >> + } >> + >> /* nunroll = total number of copies of the original loop body in >> unrolled loop (i.e. if it is 2, we have to duplicate loop body once. >> */ >> - nunroll = PARAM_VALUE (PARAM_MAX_UNROLLED_INSNS) / loop->ninsns; >> - nunroll_by_av = PARAM_VALUE (PARAM_MAX_AVERAGE_UNROLLED_INSNS) / >> loop->av_ninsns; >> + nunroll = PARAM_VALUE (PARAM_MAX_UNROLLED_INSNS) / limit_factor >> + / loop->ninsns; >> + nunroll_by_av = PARAM_VALUE (PARAM_MAX_AVERAGE_UNROLLED_INSNS) >> + / limit_factor / loop->av_ninsns; >> if (nunroll > nunroll_by_av) >> nunroll = nunroll_by_av; >> if (nunroll > (unsigned) PARAM_VALUE (PARAM_MAX_UNROLL_TIMES)) >> @@ -1195,6 +1282,7 @@ decide_peel_simple (struct loop *loop, int flags) >> { >> unsigned npeel; >> struct niter_desc *desc; >> + int limit_factor = 1; >> >> if (!(flags & UAP_PEEL)) >> { >> @@ -1205,8 +1293,23 @@ decide_peel_simple (struct loop *loop, int flags) >> if (dump_file) >> fprintf (dump_file, "\n;; Considering simply peeling loop\n"); >> >> + if (flag_peel_codesize_limit) >> + { >> + /* Determine whether to limit code size growth from peeling, >> + using FDO profile summary information that gives an >> + estimated number of executed blocks. */ >> + limit_factor = code_size_limit_factor (loop); >> + if (dump_file && limit_factor > 1) >> + { >> + fprintf (dump_file, >> + ";; Due to large code size footprint estimate, limit " >> + "max peeled insns by divisor %d\n", limit_factor); >> + } >> + } >> + >> /* npeel = number of iterations to peel. */ >> - npeel = PARAM_VALUE (PARAM_MAX_PEELED_INSNS) / loop->ninsns; >> + npeel = PARAM_VALUE (PARAM_MAX_PEELED_INSNS) / limit_factor >> + / loop->ninsns; >> if (npeel > (unsigned) PARAM_VALUE (PARAM_MAX_PEEL_TIMES)) >> npeel = PARAM_VALUE (PARAM_MAX_PEEL_TIMES); >> >> @@ -1231,7 +1334,7 @@ decide_peel_simple (struct loop *loop, int flags) >> >> /* Do not simply peel loops with branches inside -- it increases number >> of mispredicts. */ >> - if (num_loop_branches (loop) > 1) >> + if (desc->num_branches > 1) >> { >> if (dump_file) >> fprintf (dump_file, ";; Not peeling, contains branches\n"); >> @@ -1348,6 +1451,7 @@ decide_unroll_stupid (struct loop *loop, int flags >> { >> unsigned nunroll, nunroll_by_av, i; >> struct niter_desc *desc; >> + int limit_factor = 1; >> >> if (!(flags & UAP_UNROLL_ALL)) >> { >> @@ -1358,11 +1462,26 @@ decide_unroll_stupid (struct loop *loop, int flags >> if (dump_file) >> fprintf (dump_file, "\n;; Considering unrolling loop stupidly\n"); >> >> + if (flag_unroll_codesize_limit) >> + { >> + /* Determine whether to limit code size growth from unrolling, >> + using FDO profile summary information that gives an >> + estimated number of executed blocks. */ >> + limit_factor = code_size_limit_factor (loop); >> + if (dump_file && limit_factor > 1) >> + { >> + fprintf (dump_file, >> + ";; Due to large code size footprint estimate, limit " >> + "max unrolled insns by divisor %d\n", limit_factor); >> + } >> + } >> + >> /* nunroll = total number of copies of the original loop body in >> unrolled loop (i.e. if it is 2, we have to duplicate loop body once. >> */ >> - nunroll = PARAM_VALUE (PARAM_MAX_UNROLLED_INSNS) / loop->ninsns; >> - nunroll_by_av >> - = PARAM_VALUE (PARAM_MAX_AVERAGE_UNROLLED_INSNS) / loop->av_ninsns; >> + nunroll = PARAM_VALUE (PARAM_MAX_UNROLLED_INSNS) / limit_factor >> + / loop->ninsns; >> + nunroll_by_av = PARAM_VALUE (PARAM_MAX_AVERAGE_UNROLLED_INSNS) >> + / limit_factor / loop->av_ninsns; >> if (nunroll > nunroll_by_av) >> nunroll = nunroll_by_av; >> if (nunroll > (unsigned) PARAM_VALUE (PARAM_MAX_UNROLL_TIMES)) >> @@ -1392,7 +1511,7 @@ decide_unroll_stupid (struct loop *loop, int flags >> >> /* Do not unroll loops with branches inside -- it increases number >> of mispredicts. */ >> - if (num_loop_branches (loop) > 1) >> + if (desc->num_branches > 1) >> { >> if (dump_file) >> fprintf (dump_file, ";; Not unrolling, contains branches\n"); >> Index: gcc/coverage.c >> =================================================================== >> --- gcc/coverage.c (revision 189413) >> +++ gcc/coverage.c (working copy) >> @@ -245,6 +245,14 @@ read_counts_file (void) >> gcov_read_summary (&sum); >> for (ix = 0; ix != GCOV_COUNTERS_SUMMABLE; ix++) >> { >> + if (summary.ctrs[ix].num_hot_counters >> + < sum.ctrs[ix].num_hot_counters) >> + { >> + summary.ctrs[ix].num_hot_counters >> + = sum.ctrs[ix].num_hot_counters; >> + summary.ctrs[ix].hot_cutoff_value >> + = sum.ctrs[ix].hot_cutoff_value; >> + } >> summary.ctrs[ix].runs += sum.ctrs[ix].runs; >> summary.ctrs[ix].sum_all += sum.ctrs[ix].sum_all; >> if (summary.ctrs[ix].run_max < sum.ctrs[ix].run_max) >> Index: gcc/loop-iv.c >> =================================================================== >> --- gcc/loop-iv.c (revision 189413) >> +++ gcc/loop-iv.c (working copy) >> @@ -2969,8 +2969,12 @@ get_simple_loop_desc (struct loop *loop) >> /* At least desc->infinite is not always initialized by >> find_simple_loop_exit. */ >> desc = XCNEW (struct niter_desc); >> - iv_analysis_loop_init (loop); >> - find_simple_exit (loop, desc); >> + if (loop->latch != EXIT_BLOCK_PTR) >> + { >> + iv_analysis_loop_init (loop); >> + find_simple_exit (loop, desc); >> + } >> + analyze_loop_insns (loop, desc); >> loop->aux = desc; >> >> if (desc->simple_p && (desc->assumptions || desc->infinite)) >> Index: gcc/common.opt >> =================================================================== >> --- gcc/common.opt (revision 189413) >> +++ gcc/common.opt (working copy) >> @@ -1551,6 +1551,10 @@ fpcc-struct-return >> Common Report Var(flag_pcc_struct_return,1) Init(DEFAULT_PCC_STRUCT_RETURN) >> Return small aggregates in memory, not registers >> >> +fpeel-codesize-limit >> +Common Report Var(flag_peel_codesize_limit) Init(1) Optimization >> +Limit non-const non-FP loop peeling under profile estimates of large code >> footprint >> + >> fpeel-loops >> Common Report Var(flag_peel_loops) Optimization >> Perform loop peeling >> @@ -2108,6 +2112,10 @@ funroll-all-loops >> Common Report Var(flag_unroll_all_loops) Optimization >> Perform loop unrolling for all loops >> >> +funroll-codesize-limit >> +Common Report Var(flag_unroll_codesize_limit) Init(1) Optimization >> +Limit non-const non-FP loop unrolling under profile estimates of large code >> footprint >> + >> ; Nonzero means that loop optimizer may assume that the induction variables >> ; that control loops do not overflow and that the loops with nontrivial >> ; exit condition are not infinite >> Index: gcc/cfgloop.c >> =================================================================== >> --- gcc/cfgloop.c (revision 189413) >> +++ gcc/cfgloop.c (working copy) >> @@ -1155,24 +1155,98 @@ get_loop_exit_edges (const struct loop *loop) >> return edges; >> } >> >> -/* Counts the number of conditional branches inside LOOP. */ >> +/* Determine if INSN is a floating point set. */ >> >> -unsigned >> -num_loop_branches (const struct loop *loop) >> +static bool >> +insn_has_fp_set(rtx insn) >> { >> - unsigned i, n; >> - basic_block * body; >> + int i; >> + rtx pat = PATTERN(insn); >> + if (GET_CODE (pat) == SET) >> + return (FLOAT_MODE_P (GET_MODE (SET_DEST (pat)))); >> + else if (GET_CODE (pat) == PARALLEL) >> + { >> + for (i = 0; i < XVECLEN (pat, 0); i++) >> + { >> + rtx sub = XVECEXP (pat, 0, i); >> + if (GET_CODE (sub) == SET) >> + return (FLOAT_MODE_P (GET_MODE (SET_DEST (sub)))); >> + } >> + } >> + return false; >> +} >> >> - gcc_assert (loop->latch != EXIT_BLOCK_PTR); >> +/* Analyzes the instructions inside LOOP, updating the DESC. Currently >> counts >> + the number of conditional branch instructions, calls and fp instructions, >> + as well as the average number of branches executed per iteration. */ >> >> +void >> +analyze_loop_insns (const struct loop *loop, struct niter_desc *desc) >> +{ >> + unsigned i, nbranch; >> + gcov_type weighted_nbranch; >> + bool has_call, has_fp; >> + basic_block * body, bb; >> + rtx insn; >> + gcov_type header_count = loop->header->count; >> + >> + nbranch = weighted_nbranch = 0; >> + has_call = has_fp = false; >> + >> body = get_loop_body (loop); >> - n = 0; >> for (i = 0; i < loop->num_nodes; i++) >> - if (EDGE_COUNT (body[i]->succs) >= 2) >> - n++; >> + { >> + bb = body[i]; >> + >> + if (EDGE_COUNT (bb->succs) >= 2) >> + { >> + nbranch++; >> + >> + /* If this block is executed less frequently than the header (loop >> + entry), then it is weighted based on its execution count, which >> + will be turned into a ratio compared to the loop header below. >> */ >> + if (bb->count < header_count) >> + weighted_nbranch += bb->count; >> + >> + /* When it is executed more frequently than the header (i.e. it is >> + in a nested inner loop), simply weight the branch the same as >> the >> + header execution count, so that it will contribute 1 branch to >> + the ratio computed below. */ >> + else >> + weighted_nbranch += header_count; >> + } >> + >> + /* No need to iterate through the instructions below if >> + both flags have already been set. */ >> + if (has_call && has_fp) >> + continue; >> + >> + FOR_BB_INSNS (bb, insn) >> + { >> + if (!INSN_P (insn)) >> + continue; >> + >> + if (!has_call) >> + has_call = CALL_P (insn); >> + >> + if (!has_fp) >> + has_fp = insn_has_fp_set (insn); >> + } >> + } >> free (body); >> >> - return n; >> + desc->num_branches = nbranch; >> + /* Now divide the weights computed above by the loop header execution >> count, >> + to compute the average number of branches through the loop. By adding >> + header_count/2 to the numerator we round to nearest with integer >> + division. */ >> + if (header_count != 0) >> + desc->av_num_branches >> + = (weighted_nbranch + header_count/2) / header_count; >> + else >> + desc->av_num_branches = 0; >> + desc->has_call = has_call; >> + desc->has_fp = has_fp; >> } >> >> /* Adds basic block BB to LOOP. */ >> Index: gcc/cfgloop.h >> =================================================================== >> --- gcc/cfgloop.h (revision 189413) >> +++ gcc/cfgloop.h (working copy) >> @@ -255,7 +255,6 @@ extern basic_block *get_loop_body_in_custom_order >> >> extern VEC (edge, heap) *get_loop_exit_edges (const struct loop *); >> edge single_exit (const struct loop *); >> -extern unsigned num_loop_branches (const struct loop *); >> >> extern edge loop_preheader_edge (const struct loop *); >> extern edge loop_latch_edge (const struct loop *); >> @@ -366,7 +365,8 @@ struct rtx_iv >> }; >> >> /* The description of an exit from the loop and of the number of iterations >> - till we take the exit. */ >> + till we take the exit. Also includes other information used primarily >> + by the loop unroller. */ >> >> struct niter_desc >> { >> @@ -407,6 +407,18 @@ struct niter_desc >> >> /* The number of iterations of the loop. */ >> rtx niter_expr; >> + >> + /* The number of branches in the loop. */ >> + unsigned num_branches; >> + >> + /* The number of executed branches per iteration. */ >> + unsigned av_num_branches; >> + >> + /* Whether the loop contains a call instruction. */ >> + bool has_call; >> + >> + /* Whether the loop contains fp instructions. */ >> + bool has_fp; >> }; >> >> extern void iv_analysis_loop_init (struct loop *); >> @@ -420,6 +432,7 @@ extern void iv_analysis_done (void); >> >> extern struct niter_desc *get_simple_loop_desc (struct loop *loop); >> extern void free_simple_loop_desc (struct loop *loop); >> +void analyze_loop_insns (const struct loop *, struct niter_desc *desc); >> >> static inline struct niter_desc * >> simple_loop_desc (struct loop *loop) >> Index: gcc/params.def >> =================================================================== >> --- gcc/params.def (revision 189413) >> +++ gcc/params.def (working copy) >> @@ -311,6 +311,23 @@ DEFPARAM(PARAM_MAX_UNROLL_ITERATIONS, >> "max-completely-peel-loop-nest-depth", >> "The maximum depth of a loop nest we completely peel", >> 8, 0, 0) >> +/* The maximum code size estimate under which loop unrolling and peeling >> + * is allowed in a profile feedback compile. This currently applies to loops >> + * with non-constant iteration counts and no floating point computations. >> */ >> +DEFPARAM(PARAM_UNROLLPEEL_CODESIZE_THRESHOLD, >> + "unrollpeel-codesize-threshold", >> + "Maximum profile-based code size footprint estimate for loop >> unrolling " >> + "and peeling", >> + 15000, 0, 0) >> +/* The maximum ratio of total profiled execution counts to loop entry block >> + count that must be exceeded to ignore most code size limits when >> unrolling >> + and peeling. */ >> +DEFPARAM(PARAM_UNROLLPEEL_HOTNESS_THRESHOLD, >> + "unrollpeel-hotness-threshold", >> + "Maximum ratio of total profiled execution count to loop entry " >> + "block count under which most codesize limits for unrolling and " >> + "peeling will be ignored", >> + 100, 1, 0) >> >> /* The maximum number of insns of an unswitched loop. */ >> DEFPARAM(PARAM_MAX_UNSWITCH_INSNS, >> Index: gcc/gcov-dump.c >> =================================================================== >> --- gcc/gcov-dump.c (revision 189413) >> +++ gcc/gcov-dump.c (working copy) >> @@ -456,8 +456,12 @@ tag_summary (const char *filename ATTRIBUTE_UNUSED >> { >> printf ("\n"); >> print_prefix (filename, 0, 0); >> - printf ("\t\tcounts=%u, runs=%u", >> - summary.ctrs[ix].num, summary.ctrs[ix].runs); >> + printf ("\t\tcounts=%u (num hot counts=%u, hot cutoff count=" >> + HOST_WIDEST_INT_PRINT_DEC "), runs=%u", >> + summary.ctrs[ix].num, >> + summary.ctrs[ix].num_hot_counters, >> + (HOST_WIDEST_INT)summary.ctrs[ix].hot_cutoff_value, >> + summary.ctrs[ix].runs); >> >> printf (", sum_all=" HOST_WIDEST_INT_PRINT_DEC, >> (HOST_WIDEST_INT)summary.ctrs[ix].sum_all); >> >> -- >> This patch is available for review at http://codereview.appspot.com/6351086 > > > > -- > Teresa Johnson | Software Engineer | tejohn...@google.com | 408-460-2413 -- Teresa Johnson | Software Engineer | tejohn...@google.com | 408-460-2413
