Re: [PATCH] Avoid ggc_collect () after WPA forking

[Martin Liška]

On 03/19/2014 03:55 PM, Richard Biener wrote:
On Wed, 19 Mar 2014, Martin Liška wrote:
There are stats for Firefox with LTO and -O2. According to graphs it
looks that memory consumption for parallel WPA phase is similar.
When I disable parallel WPA, wpa footprint is ~4GB, but ltrans memory
footprint is similar to parallel WPA that reduces libxul.so linking by ~10%.
Ok, so I suppose this tracks RSS, not virtual memory use (what is
"used" and what is "active")?

Data are given by vmstat, according to: 
http://stackoverflow.com/questions/18529723/what-is-active-memory-and-inactive-memory

*Active memory*is memory that is being used by a particular process.
*Inactive memory*is memory that was allocated to a process that is no 
longer running.

So please follow just 'blue' line that displays really used memory. 
According to man, vmstat tracks virtual memory statistics.

And it is WPA plus LTRANS stages, WPA ends where memory use first goes
down to zero?
I wonder if you can identify the point where parallel streaming
starts and where it ends ... ;)

Exactly, WPA ends when it goes to zero.

Btw, I have another patch in my local tree, limiting the
exponential growth of blocks we allocate when outputting sections.
But it shouldn't be _that_ bad ... maybe you can try if it has
any effect?

I can apply it.

Martin


Thanks,
Richard.

Index: gcc/lto-section-out.c
===================================================================
--- gcc/lto-section-out.c       (revision 208642)
+++ gcc/lto-section-out.c       (working copy)
@@ -99,13 +99,19 @@ lto_end_section (void)
  }
  
  
+/* We exponentially grow the size of the blocks as we need to make
+   room for more data to be written.  Start with a single page and go up
+   to 2MB pages for this.  */
+#define FIRST_BLOCK_SIZE 4096
+#define MAX_BLOCK_SIZE (2 * 1024 * 1024)
+
  /* Write all of the chars in OBS to the assembler.  Recycle the blocks
     in obs as this is being done.  */
  
  void
  lto_write_stream (struct lto_output_stream *obs)
  {
-  unsigned int block_size = 1024;
+  unsigned int block_size = FIRST_BLOCK_SIZE;
    struct lto_char_ptr_base *block;
    struct lto_char_ptr_base *next_block;
    if (!obs->first_block)
@@ -135,6 +141,7 @@ lto_write_stream (struct lto_output_stre
        else
        lang_hooks.lto.append_data (base, num_chars, block);
        block_size *= 2;
+      block_size = MIN (MAX_BLOCK_SIZE, block_size);
      }
  }
  
@@ -152,7 +159,7 @@ lto_append_block (struct lto_output_stre
      {
        /* This is the first time the stream has been written
         into.  */
-      obs->block_size = 1024;
+      obs->block_size = FIRST_BLOCK_SIZE;
        new_block = (struct lto_char_ptr_base*) xmalloc (obs->block_size);
        obs->first_block = new_block;
      }
@@ -162,6 +169,7 @@ lto_append_block (struct lto_output_stre
        /* Get a new block that is twice as big as the last block
         and link it into the list.  */
        obs->block_size *= 2;
+      obs->block_size = MIN (MAX_BLOCK_SIZE, obs->block_size);
        new_block = (struct lto_char_ptr_base*) xmalloc (obs->block_size);
        /* The first bytes of the block are reserved as a pointer to
         the next block.  Set the chain of the full block to the