https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Richard Biener changed:
What|Removed |Added
Target Milestone|11.2|---
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Jakub Jelinek changed:
What|Removed |Added
Target Milestone|11.0|11.2
--- Comment #44 from Jakub Jelinek
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Martin Liška changed:
What|Removed |Added
Status|ASSIGNED|NEW
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Martin Liška changed:
What|Removed |Added
Assignee|marxin at gcc dot gnu.org |unassigned at gcc dot
gnu.org
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #42 from CVS Commits ---
The master branch has been updated by Martin Liska :
https://gcc.gnu.org/g:ece21ff6ea9d969d3b6aae82136622a7126eefc1
commit r11-1778-gece21ff6ea9d969d3b6aae82136622a7126eefc1
Author: Martin Liska
Date:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #41 from Martin Liška ---
The patch survives PGO bootstrap of GCC and it shrinks size of gcda
file from 17MB to 12MB.
And compression can achieve the following:
zstd: 3.3 MB
$ time zstd *
real0m0.082s
user0m0.068s
sys
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Martin Liška changed:
What|Removed |Added
Status|WAITING |ASSIGNED
--- Comment #40 from Martin
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #39 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #38)
> Created attachment 48738 [details]
> Patch candidate v2
I have added this patch to my private gcc 8 with some change, works fine with
the small
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Martin Liška changed:
What|Removed |Added
Attachment #48660|0 |1
is obsolete|
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #37 from qinzhao at gcc dot gnu.org ---
So, the previous prof data size for the real application might not be correct.
After this bug is fixed, we might need to collect the new real code size
reduction.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #36 from qinzhao at gcc dot gnu.org ---
I found a bug with this proposed patch:
when doing automatic merging, the following error message is emitted:
Merge mismatch for function 1.
the bug can be repeated with the small testing case
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #35 from Martin Liška ---
(In reply to Qing Zhao from comment #34)
> >
> >> Though still bigger than what ICC generated.
> >
> > Yep, but we should be only 2x bigger right now?
> Yes, around 2-3 times bigger, much better now.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #34 from Qing Zhao ---
>
>> Though still bigger than what ICC generated.
>
> Yep, but we should be only 2x bigger right now?
Yes, around 2-3 times bigger, much better now.
>
> Can you please test the parallel merging script? I can
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #33 from Martin Liška ---
>
> The profile directory generated by the new executable compiled with this
> patch was 112G in size, a lot smaller than previous 1TB.
That's quite a promising reduction.
> Though still bigger than
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #32 from Qing Zhao ---
> I would be more interested in overall statistics for your training scenario.
> How much can you get from ~1TB of data?
The profile directory generated by the new executable compiled with this patch
was 112G
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #31 from Qing Zhao ---
> The explanation is not sufficient.
You mean the following explanation: (in comment 18)
we tried the scheme that all the processes generate profiling feedback data to
the single directory,
but looks like a
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #30 from Martin Liška ---
(In reply to Qing Zhao from comment #29)
> >
> > And you still haven't replied to my essential question: Why can't you merge
> > profiles into one directory during run? Or at least merge to a reasonable
> >
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #29 from Qing Zhao ---
>
> And you still haven't replied to my essential question: Why can't you merge
> profiles into one directory during run? Or at least merge to a reasonable
> number of folders that you'll merge later?
Comment
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #28 from Martin Liška ---
(In reply to Qing Zhao from comment #26)
> > --- Comment #25 from Martin Liška ---
> >> I will try to get more data on our real application.
> >>
> >> one question: why not just delete the entire records
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #27 from Martin Liška ---
(In reply to qinzhao from comment #24)
> with the patch added to gcc11, I tested it with the small testing case, and
> got the following data:
>
I would be more interested in overall statistics for your
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #26 from Qing Zhao ---
> --- Comment #25 from Martin Liška ---
>> I will try to get more data on our real application.
>>
>> one question: why not just delete the entire records whose counter is zero
>> and delete the entire file
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #25 from Martin Liška ---
> I will try to get more data on our real application.
>
> one question: why not just delete the entire records whose counter is zero
> and delete the entire file whose counter is zero?
Because we need to
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #24 from qinzhao at gcc dot gnu.org ---
with the patch added to gcc11, I tested it with the small testing case, and got
the following data:
**without the patch:
qinzhao@gcc14:~/Bugs/profile/small_gcc/gcc_prof_dir/13248$ ls -l
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #23 from Martin Liška ---
Created attachment 48660
--> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48660=edit
work-in-progress patch
There's patch that does not stream all zero counters for a function. The patch
only supports
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #19 from Qing Zhao ---
Hi, Martin,
I attached 3 profiling data files with this email (among over 5000 files under
one typical directory),
Hope this is helpful.
Thanks.
Qing
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #20 from Qing Zhao ---
Created attachment 48653
--> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48653=edit
A.data
--- Comment #21 from Qing Zhao ---
Created attachment 48654
-->
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #18 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #16)
> > For our application, all processes generating profiling feedback data to a
> > single directory seems is not a choice.
>
> Why is it problem?
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #17 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #6)
more details:
>
> Which means one run takes 100MB is size, right? As you mentioned, having
> 1000 .gcda files, it means that one takes 0.1MB?
>
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #16 from Martin Liška ---
> For our application, all processes generating profiling feedback data to a
> single directory seems is not a choice.
Why is it problem? You need to provide reasoning for that.
> We chose
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #15 from qinzhao at gcc dot gnu.org ---
please refer to
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #14 from Qing Zhao ---
>
> -fprofile-dir=gcc_prof_dir/%p"
>
> So my recommendation would be not to use it and let GCOV run-time library
> merge
> the profiles. Of course, I'll be interested in `perf report` of such a
>
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #13 from Qing Zhao ---
> The only exception is a cross-profiling:
> https://gcc.gnu.org/onlinedocs/gcc/Cross-profiling.html
>
> where one can use GCOV_PREFIX environment variable to save .gcda files to a
> separate location.
>
> Do
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #12 from Martin Liška ---
> Do you use it? Or do you use any of -fprofile-dir options?
Ah, ok, you use it. Based on the report:
-fprofile-dir=gcc_prof_dir/%p"
So my recommendation would be not to use it and let GCOV run-time
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #11 from Martin Liška ---
(In reply to qinzhao from comment #9)
> (In reply to Martin Liška from comment #7)
> > 1) You should not generate profile data for each process to a different
> > folder, but rather merge it.
>
> not sure
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #10 from Martin Liška ---
>
> around 14000 processes, they are not the same executable, so not all the run
>
Both I guess they share the majority of compiled object files, right?
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #9 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #7)
> 1) You should not generate profile data for each process to a different
> folder, but rather merge it.
not sure how to do this? can you provide more
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #8 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #6
> Which means one run takes 100MB is size, right? As you mentioned, having
> 1000 .gcda files, it means that one takes 0.1MB?
around 14000 processes,
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Martin Liška changed:
What|Removed |Added
Status|ASSIGNED|WAITING
--- Comment #7 from Martin Liška
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #6 from Martin Liška ---
(In reply to qinzhao from comment #5)
> (In reply to Martin Liška from comment #4)>
> > Can you please share some statistics how big are the files and how many
> > runs do you merge?
>
> There were on
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #5 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #4)>
> Can you please share some statistics how big are the files and how many runs
> do you merge?
There were on the order of 10,000 processes. Source
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #4 from Martin Liška ---
(In reply to qinzhao from comment #3)
> (In reply to Martin Liška from comment #2)
> > Thank you for the report. It's a known limitation Honza noticed me about.
> > Is the size problematic from size
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
--- Comment #3 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #2)
> Thank you for the report. It's a known limitation Honza noticed me about.
> Is the size problematic from size perspective or speed perspective?
I
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Martin Liška changed:
What|Removed |Added
CC||hubicka at gcc dot gnu.org
Target
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348
Richard Biener changed:
What|Removed |Added
Last reconfirmed||2020-05-27
44 matches
Mail list logo
