Re: Good news, bad news on the repository conversion

[Eric S. Raymond]

Mark Atwood :
> ESR, how much for the memory expansion?  It sounds like we have some
> volunteers to solve this problem with some money.

That's now rthe second problem out.  There's a malformation that has
turned up in the repo that may sink the conversion entirely.  I want to be
reasonably sure I can solve that before I go asking for money.
-- 
http://www.catb.org/~esr/";>Eric S. Raymond

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: Good news, bad news on the repository conversion

[Mark Atwood]

ESR, how much for the memory expansion?  It sounds like we have some
volunteers to solve this problem with some money.

..m

On Tue, Jul 10, 2018 at 2:14 AM Aldy Hernandez  wrote:

> Wait, there's a pot of money for making SVN go away?  Sign me up!
> While we're at it, let's start one for TCL and dejagnu!
> On Mon, Jul 9, 2018 at 6:58 PM Jeff Law  wrote:
> >
> > On 07/09/2018 10:53 AM, Janus Weil wrote:
> > > 2018-07-09 18:35 GMT+02:00 Eric S. Raymond :
> > >> David Edelsohn :
> >  The truth is we're near the bleeding edge of what conventional tools
> >  and hardware can handle gracefully.  Most jobs with working sets as
> >  big as this one's do only comparatively dumb operations that can be
> >  parallellized and thrown on a GPU or supercomputer.  Most jobs with
> >  the algorithmic complexity of repository surgery have *much* smaller
> >  working sets.  The combination of both extrema is hard.
> > >>>
> > >>> If you come to the conclusion that the GCC Community could help with
> > >>> resources, such as the GNU Compile Farm or paying for more RAM, let
> us
> > >>> know.
> > >>
> > >> 128GB of DDR4 registered RAM would allow me to run conversions with my
> > >> browser up, but be eye-wateringly expensive.  Thanks, but I'm not
> > >> going to yell for that help
> > >
> > > I for one would certainly be happy to donate some spare bucks towards
> > > beastie RAM if it helps to get the GCC repo converted to git in a
> > > timely manner, and I'm sure there are other GCC
> > > developers/users/sympathizers who'd be willing to join in. So, where
> > > do we throw those bucks?
> > I'd be willing to throw some $$$ at this as well.
> > Jeff
> >
>
-- 

Mark Atwood
http://about.me/markatwood
+1-206-604-2198

Re: -Wclass-memaccess warning should be in -Wextra, not -Wall

[Soul Studios]

I guess the phrasing is a bit weak, "some users" obviously has to refer 
to a significant proportion of users, "easy to avoid" cannot have too 
many drawbacks (in particular, generated code should be of equivalent 
quality), etc.


-Wclass-memaccess fits the "easy to avoid" quite well, since a simple 
cast disables it. -Wmaybe-uninitialized is much worse: it produces many 
false positives, that change with every release and are super hard to 
avoid. And even in the "easy to avoid" category where we don't want to 
litter the code with casts to quiet the warnings, I find -Wsign-compare 
way worse in practice than -Wclass-memaccess.


I personally get annoyed with the amount of hand-holding compilers seem 
to need now in order to do what I ask them to do - and this isn't 
leveled at GCC, but also clang and MSVC.
Some of it makes sense in order to check the programmer meant to make 
that conversion, but you end up with a hell of a lot of static/other 
casts, and it's not great for readability.
As a side-note, the workaround for memset/cpy/move as mentioned isn't 
visible in the warning, which I think it probably should be, rather than 
just the GCC documentation.
Still, as you all have mentioned, the workaround is pretty 
straight-forward in this case, so that's good.

M@

Re: -Wclass-memaccess warning should be in -Wextra, not -Wall

[Soul Studios]

Not sure how kosher it is to address several replies in one email, but 
I'm going to attempt it as there are overlapping topics:



Martin:


Simply because a struct has a constructor does not mean it isn't a
viable target/source for use with memcpy/memmove/memset.


As the documentation that Segher quoted explains, it does
mean exactly that.


I just thought I'd go back and address this, because it doesn't; you can 
have a struct with a basic constructor that is just a convenience 
function (or an optimization to avoid default type assignment), but all 
the types are POD. This is an obvious viable target for memcpy/memmove.




Please open bugs with small test cases showing
the inefficiencies so the optimizers can be improved.


I've done one today (86471), I may not have time to do more. In 
particular I've found GCC to generate less-optimal code for Core2 
processors as opposed to the i3 and upwards range, when avoiding memcpy.




No, programmers don't always know that.  In fact, it's easy even
for an expert programmer to make the mistake that what looks like
a POD struct can safely be cleared by memset or copied by memcpy
when doing so is undefined because one of the struct members is
of a non-trivial type (such a container like string).


I'm not sure that that would be 'easy' for an expert to make that 
mistake (Marc's example aside), but okay.
I'm coming from the understanding that the orthodox approach (and the 
one currently being taught in schools/universities) is to use the C++ 
templated functions as opposed to the older C-style functions, and this 
is the avenue that the majority of novice C++ programmers will be coming 
from.




Quite a lot of thought and discussion went into the design and
implementation of the warning, so venting your frustrations or
insulting those of us involved in the process is unlikely to
help you effect a change.  To make a compelling argument you


Where did I insult you? I didn't. And expressing (not venting) 
frustration is fine if I feel people are not accurately reading what 
I've written I think. Don't take too much offence.




Jonathan:

> It was clear in your first post, but that doesn't make it correct. The
> statement "any programmer [invoking undefined behaviour] is going to
> know what they're getting into" is laughable.

I didn't say [invoking undefined behaviour]. I said using 
memset/memcpy/memmove on the structs/classes-with-constructors-etc where 
it is viable to do so. And that may not be invoking undefined behaviour, 
as above-



> I've seen far more cases of assignment operators implemented with
> memcpy that were wrong and broken and due to ignorance or incompetence
> than I have seen them done by programmers who knew what they were
> doing, or knew what they were getting into. There are programmers who
> come from C, and don't realise that a std::string shouldn't be copied
> with memcpy. There are programmers who are too lazy to write out
> memberwise assignment for each member, so just want to save a few
> lines of code by copying everything in one go with memcpy. There are
> lots of other ways to do it wrong. Your statement is simply not based
> in fact, it's more likely based on your limited experience, and
> assumption that everybody is probably doing what you're doing.

Fair enough, if that's your experience, it's fine. My assumption was 
that the orthodox (as above) approach is currently 
pretty-well-propagated on the internet-and-academia, and that anyone not 
doing that would have their reasons. But, thank you for actually 
addressing my post's point and not simply dancing around the edge of it.
There are other non-lazy performance reasons to use these, even for 
assignment, but it depends on the situation. I don't say this not having 
benchmarked it extensively, and correctly, in my own code.


M@

Re: [GSOC] LTO dump tool project

[Hrishikesh Kulkarni]

Hi,

Thanks for suggestions. I would start working on these points and will
try to complete as early as possible.

Regards,

Hrishikesh

On Mon, Jul 9, 2018 at 2:04 PM, Martin Liška  wrote:
> On 07/09/2018 09:50 AM, Hrishikesh Kulkarni wrote:
>> Hi,
>>
>> The command line option -gimple-stats will dump the statistics of
>> gimple statements.
>>
>> For example:
>>
>> $ ../stage1-build/gcc/lto-dump test_hello.o -gimple-stats
>>
>> will dump:
>>
>> GIMPLE statements
>> Kind   Stmts  Bytes
>> ---
>> assignments3264
>> phi nodes  1248
>> conditionals   1 80
>> everything else   12704
>> ---
>> Total 17   1296
>> ---
>>
>> I have pushed the changes to the repo. Please find the diff file
>> attached herewith.
>>
>> Regards,
>>
>> Hrishikesh
>
> Hi.
>
> Thanks for the work. I briefly took a look at the code and I would
> focus now directly on refactoring:
>
> - please make a new branch, first copy lto-dump.c file and all the
> Makefile needed stuff and commit that.
> - next please rebase (squash) all your patches which you have on top
> of it; you did some formatting corrections and it's very hard to read
> - please fix coding style, it's mentioned here: 
> https://gcc.gnu.org/codingconventions.html
> the most problematic is usage of horizontal white spaces. We use 2 spaces
> for indentation level, and 8 spaces are replaced with a tabular; without that 
> reading your
> code is very hard for me
> - then please start refactoring functionality that is copied from lto.c and 
> put shared
> stuff into a header file that will be used by lto.c and lto-dump.c.
>
> Other observations:
> - you use "\t\t%s\t\t%s\t\t%s" formats for prints; I think it would be better 
> to
> use fixed strings with spaces, try %20s. It's probably also used by tools 
> like nm or objdump
> - come up with more specific name for 'entry' and 'compare'
> - 'entry' should have functions that will print names, ... according to 
> options
> (flag_lto_dump_demangle, ...); you can have overrides for functions and 
> variables
> - I would first put all symbols into the vector and then I would print it on 
> a single place
> - consider using vec from vec.h, hash_map from hash-map.h instead of std:: 
> variants
> - exit after functions like dump_list, dump_symbol,...
> - remove dummy 'dump' function
>
> Martin
>
>>
>> On Thu, Jul 5, 2018 at 10:41 PM, Hrishikesh Kulkarni
>>  wrote:
>>> Hi,
>>>
>>> I have added new command line option:
>>> -objects
>>> which will dump size, offset and name of each section for all lto objects
>>>
>>> for example:
>>> $ ../stage1-build/gcc/lto-dump test_hello.o test.o -objects
>>>
>>> gives output:
>>> LTO object name: test_hello.o
>>>
>>> NO.OFFSETSIZESECTION NAME
>>>
>>> 16415.gnu.lto_.profile.a7add72ac123628
>>> 27955.gnu.lto_.icf.a7add72ac123628
>>> 3134134.gnu.lto_.jmpfuncs.a7add72ac123628
>>> 4268116.gnu.lto_.inline.a7add72ac123628
>>> 538424.gnu.lto_.pureconst.a7add72ac123628
>>> 6408306.gnu.lto_foo.a7add72ac123628
>>> 7714469.gnu.lto_bar.a7add72ac123628
>>> 81183345.gnu.lto_main.a7add72ac123628
>>> 9152888.gnu.lto_.symbol_nodes.a7add72ac123628
>>> 10161615.gnu.lto_.refs.a7add72ac123628
>>> 1116311205.gnu.lto_.decls.a7add72ac123628
>>> 122836109.gnu.lto_.symtab.a7add72ac123628
>>> 13294576.gnu.lto_.opts
>>>
>>> LTO object name: test.o
>>>
>>> NO.OFFSETSIZESECTION NAME
>>>
>>> 16415.gnu.lto_.profile.ffab9cb8eb84fc03
>>> 27930.gnu.lto_.icf.ffab9cb8eb84fc03
>>> 3109108.gnu.lto_.jmpfuncs.ffab9cb8eb84fc03
>>> 421762.gnu.lto_.inline.ffab9cb8eb84fc03
>>> 527921.gnu.lto_.pureconst.ffab9cb8eb84fc03
>>> 6300194.gnu.lto_koo.ffab9cb8eb84fc03
>>> 7494389.gnu.lto_gain.ffab9cb8eb84fc03
>>> 888377.gnu.lto_.symbol_nodes.ffab9cb8eb84fc03
>>> 996015.gnu.lto_.refs.ffab9cb8eb84fc03
>>> 10975966.gnu.lto_.decls.ffab9cb8eb84fc03
>>> 11194158.gnu.lto_.symtab.ffab9cb8eb84fc03
>>> 12199976.gnu.lto_.opts
>>>
>>>
>>> I have pushed the changes to the repo. Please find the diff file
>>> attached herewith.
>>>
>>> Regards,
>>>
>>> Hrishikesh
>>>
>>> On Thu, Jul 5, 2018 at 12:24 AM, Hrishikesh Kulkarni
>>>  wrote:
 Hi,

 I have:
 tried to do all 

Re: ChangeLog's: do we have to?

[Paul Koning]

> On Jul 10, 2018, at 2:18 PM, NightStrike  wrote:
> 
> On Fri, Jul 6, 2018 at 1:17 AM, Siddhesh Poyarekar  
> wrote:
>> ...
>> 
>> We had discussed making addition of ChangeLog entries into the commit
>> message mandatory but the issue there is that commit logs cannot be (or more
>> precisely, should not be) modified after they're pushed so errors in
>> ChangeLog entries will remain.  Carlos had suggested git notes, but I don't
>> know off the top of my head how they work; I vaguely remember them being
>> editable.
> 
> You could do this now under svn, I believe, as svn allows --rev-prop
> changes to modify the log.

Yes, and possibly other rev properties as well, provided the server hook script 
is changed to enable whatever you want to allow.  By default all revprop 
modifications are blocked.

paul

Re: ChangeLog's: do we have to?

[NightStrike]

On Fri, Jul 6, 2018 at 1:17 AM, Siddhesh Poyarekar  wrote:
> On 07/05/2018 05:02 PM, Richard Biener wrote:
>>
>> I assumed you just want to remove the ChangeLog files, not change
>> contents.
>> Thus I assumed the commit message would simply contain the ChangeLog
>> entry as we requie it today?  In that case git log --grep would still
>> provide
>> everything grepping ChangeLogs does - maybe apart from reducing noise
>> because you can automatically grep specific ChangeLogs only (like only in
>> cp/).
>
>
> We had discussed making addition of ChangeLog entries into the commit
> message mandatory but the issue there is that commit logs cannot be (or more
> precisely, should not be) modified after they're pushed so errors in
> ChangeLog entries will remain.  Carlos had suggested git notes, but I don't
> know off the top of my head how they work; I vaguely remember them being
> editable.

You could do this now under svn, I believe, as svn allows --rev-prop
changes to modify the log.

Re: Question on -fopt-info-inline

[Qing Zhao]

> On Jul 10, 2018, at 11:32 AM, Richard Biener  
> wrote:
> 
> On July 10, 2018 5:42:40 PM GMT+02:00, Qing Zhao  wrote:
>> Hi, David,
>> 
>> thanks a lot for your information. very helpful.
>> 
>> specifically, I am mostly interested in the inline report part of the
>> opt-info:
>> 
>> 1. what’s the current status of inlining report through opt-info? 
>> (with the upstream GCC last week,
>> the -fopt-info-inline report nothing)
>> 2. what’s the plan for opt-info-inline? when will it be available?
> 
> Just implement it yourself? There is already Winline you can look at. 
Yes, I can definitely do that. Just want to be consistent with the current work 
David is doing, any specific requirement?
and also want to make sure no duplicated effort spent on this work. :-)

> 
>> 3. is there any design available for the messages from opt-info-inline?
>> will the call-chain info, profiling 
>> feedback info be available in the inlining report?
> 
> What do you need the report for? With C++ too much inlining happens to be 
> interesting. You can also always look at the ipa-inline and einline dumps. 

we are mostly interested in C inlining report. mainly to find all the callers 
of a callee that is inlined. 

I will check the dump files of ipa-inline and einline to see whether they can 
be filtered to get enough info we want first.

thanks.

Qing
> 
> Richard. 

Re: Question on -fopt-info-inline

[Richard Biener]

On July 10, 2018 5:42:40 PM GMT+02:00, Qing Zhao  wrote:
>Hi, David,
>
>thanks a lot for your information. very helpful.
>
>specifically, I am mostly interested in the inline report part of the
>opt-info:
>
>1. what’s the current status of inlining report through opt-info? 
>(with the upstream GCC last week,
>the -fopt-info-inline report nothing)
>2. what’s the plan for opt-info-inline? when will it be available?

Just implement it yourself? There is already Winline you can look at. 

>3. is there any design available for the messages from opt-info-inline?
>will the call-chain info, profiling 
>feedback info be available in the inlining report?

What do you need the report for? With C++ too much inlining happens to be 
interesting. You can also always look at the ipa-inline and einline dumps. 

Richard. 

>
>thanks.
>
>Qing
>
>> On Jul 5, 2018, at 3:55 PM, David Malcolm 
>wrote:
 
 where can I find more details of David’s work?
>>> 
>>> I don't have pointers to all the discussion handy.  BUt here's one
>of
>>> the early messages:
>>> 
>>> https://gcc.gnu.org/ml/gcc-patches/2018-05/msg01675.html
>> 
>> I'm currently attacking the problem of "better optimization
>> information" from two directions:
>> 
>> (a) More destinations for the existing optimization reports: being
>able
>> to send them through the diagnostics subsystem, and to be able to
>save
>> them in a machine-readable format from which reports can be generated
>> (e.g. prioritized by code hotness). The initial patch kit Jeff linked
>> to above introduced a new API for doing that, but I'm no longer doing
>> that, instead working on using the existing "dump_*" API in
>dumpfile.h.
>> Some of this work is now in trunk: dump messages are now tagged with
>> metadata about the hotness of the code being optimized, and where in
>> GCC's own code the messages was emitted from ...but this new metadata
>> is being dropped on the floor in dumpfile.c right now.  The latest
>> version of the patch kit for (a) is awaiting review at:
>>  "[PATCH 0/2] v4: optinfo framework and remarks"
>> https://gcc.gnu.org/ml/gcc-patches/2018-07/msg00066.html
>
>> 
>> (b) I'm looking at new, improved optimization reports for
>> vectorization, by capturing higher-level information about why a loop
>> can't be vectorized, in a form that hopefully is useful to an
>end-user. 
>> See a (very rough) prototype here:
>> 
>>  * "[PATCH] [RFC] Higher-level reporting of vectorization problems"
>>* https://gcc.gnu.org/ml/gcc-patches/2018-06/msg01462.html
>
>> 
>> I'm working on a less rough version of (b) and hope to post it to
>gcc-
>> patches soon.
>> 
>> Hope this sounds sane
>> Dave

Re: Question on -fopt-info-inline

[Qing Zhao]

Hi, David,

thanks a lot for your information. very helpful.

specifically, I am mostly interested in the inline report part of the opt-info:

1. what’s the current status of inlining report through opt-info?  (with the 
upstream GCC last week,
the -fopt-info-inline report nothing)
2. what’s the plan for opt-info-inline? when will it be available?
3. is there any design available for the messages from opt-info-inline? will 
the call-chain info, profiling 
feedback info be available in the inlining report?

thanks.

Qing

> On Jul 5, 2018, at 3:55 PM, David Malcolm  wrote:
>>> 
>>> where can I find more details of David’s work?
>> 
>> I don't have pointers to all the discussion handy.  BUt here's one of
>> the early messages:
>> 
>> https://gcc.gnu.org/ml/gcc-patches/2018-05/msg01675.html
> 
> I'm currently attacking the problem of "better optimization
> information" from two directions:
> 
> (a) More destinations for the existing optimization reports: being able
> to send them through the diagnostics subsystem, and to be able to save
> them in a machine-readable format from which reports can be generated
> (e.g. prioritized by code hotness). The initial patch kit Jeff linked
> to above introduced a new API for doing that, but I'm no longer doing
> that, instead working on using the existing "dump_*" API in dumpfile.h.
> Some of this work is now in trunk: dump messages are now tagged with
> metadata about the hotness of the code being optimized, and where in
> GCC's own code the messages was emitted from ...but this new metadata
> is being dropped on the floor in dumpfile.c right now.  The latest
> version of the patch kit for (a) is awaiting review at:
>  "[PATCH 0/2] v4: optinfo framework and remarks"
> https://gcc.gnu.org/ml/gcc-patches/2018-07/msg00066.html 
> 
> 
> (b) I'm looking at new, improved optimization reports for
> vectorization, by capturing higher-level information about why a loop
> can't be vectorized, in a form that hopefully is useful to an end-user. 
> See a (very rough) prototype here:
> 
>  * "[PATCH] [RFC] Higher-level reporting of vectorization problems"
>* https://gcc.gnu.org/ml/gcc-patches/2018-06/msg01462.html 
> 
> 
> I'm working on a less rough version of (b) and hope to post it to gcc-
> patches soon.
> 
> Hope this sounds sane
> Dave

Re: ChangeLog's: do we have to?

[Siddhesh Poyarekar]

On 07/10/2018 08:19 PM, Frank Ch. Eigler wrote:


siddhesh wrote:


[...]  We had discussed making addition of ChangeLog entries into the
commit message mandatory but the issue there is that commit logs
cannot be (or more precisely, should not be) modified after they're
pushed so errors in ChangeLog entries will remain.  [...]


In such a rare case, it would be possible to add a subsequent commit
that carries the corrected text, but no other changes (--allow-empty).
Or even revert the previous patch, then re-commit with a fixed text.
Watch out for "perfect is the enemy of the good" syndrome here (as in
the svn-git conversion).


I agree.

Siddhesh

Re: ChangeLog's: do we have to?

[Frank Ch. Eigler]

siddhesh wrote:

> [...]  We had discussed making addition of ChangeLog entries into the
> commit message mandatory but the issue there is that commit logs
> cannot be (or more precisely, should not be) modified after they're
> pushed so errors in ChangeLog entries will remain.  [...]

In such a rare case, it would be possible to add a subsequent commit
that carries the corrected text, but no other changes (--allow-empty).
Or even revert the previous patch, then re-commit with a fixed text.
Watch out for "perfect is the enemy of the good" syndrome here (as in
the svn-git conversion).

- FChE

Re: Are the extended algorithms in the header file going to be supported by gcc ?

[Marco Ippolito]

Thanks Thomas for the info.
As we can see in https://en.cppreference.com/w/cpp/compiler_support,  the
support for C++17 parallel algorithms is actually  the last major gcc
feature for C++17 that needs to be shipped.
>From the comments below the report of the last ISO C++ Meeting in
Rapperswill :
https://www.reddit.com/r/cpp/comments/8prqzm/2018_rapperswil_iso_c_committee_trip_report/
we just have a glimpse of how many C++ developers, as myself, that are
willing develop real-world modern (not-legacy) C++ software for real-world
applications.
So, I take this external Intel library as a temporary bridge.

Thanks for your important gcc development work.
Kind regards.
Marco




Il giorno mar 10 lug 2018 alle ore 15:35 Thomas Rodgers 
ha scritto:

>
> Major GCC releases ship once per year, roughly in May. You can however,
> today, use the Intel free standing implementation until libstdc++ formally
> ships with support. See -
>
> https://github.com/intel/parallelstl
>
> - Tom
>
> Marco Ippolito writes:
>
> > Hi Thomas,
> > since simplied and efficient parallelism is actually super-needed in a
> > world where fast, simple ed efficient software is paramount, when do you
> > reasonably foresee GCC9 shipping containing the C++17 parallel
> algorithms?
> >
> > Marco
> >
> > Il giorno lun 21 mag 2018 alle ore 14:32 Thomas Rodgers <
> trodg...@redhat.com>
> > ha scritto:
> >
> >>
> >> Marco Ippolito writes:
> >>
> >> >   Which gcc release will include the support for the C++17 parallel
> >> > algorithms?
> >>
> >> The expectation is they will ship as part of GCC9.
> >>
> >> - Tom
> >>
> >>
>
>

Re: Are the extended algorithms in the header file going to be supported by gcc ?

[Thomas Rodgers]

Major GCC releases ship once per year, roughly in May. You can however,
today, use the Intel free standing implementation until libstdc++ formally
ships with support. See -

https://github.com/intel/parallelstl

- Tom

Marco Ippolito writes:

> Hi Thomas,
> since simplied and efficient parallelism is actually super-needed in a
> world where fast, simple ed efficient software is paramount, when do you
> reasonably foresee GCC9 shipping containing the C++17 parallel algorithms?
>
> Marco
>
> Il giorno lun 21 mag 2018 alle ore 14:32 Thomas Rodgers 
> ha scritto:
>
>>
>> Marco Ippolito writes:
>>
>> >   Which gcc release will include the support for the C++17 parallel
>> > algorithms?
>>
>> The expectation is they will ship as part of GCC9.
>>
>> - Tom
>>
>>

Re: Are the extended algorithms in the header file going to be supported by gcc ?

[Marco Ippolito]

Hi Thomas,
since simplied and efficient parallelism is actually super-needed in a
world where fast, simple ed efficient software is paramount, when do you
reasonably foresee GCC9 shipping containing the C++17 parallel algorithms?

Marco

Il giorno lun 21 mag 2018 alle ore 14:32 Thomas Rodgers 
ha scritto:

>
> Marco Ippolito writes:
>
> >   Which gcc release will include the support for the C++17 parallel
> > algorithms?
>
> The expectation is they will ship as part of GCC9.
>
> - Tom
>
>

Re: Repo conversion troubles.

[Philip Martin]

"Eric S. Raymond"  writes:

> I'm saying I see rsync plus local checkout take 10-12 hours.

The rsync is a one-off cost.  Once you have the repository locally you
can checkout any individual revision much more quickly.  I have a local
copy of the gcc repository and a checkout of gcc trunk from localhost
takes about 40 seconds.  I'm not using fancy hardware.  I can even check
it out across my very average WiFi in just over 60 seconds.

-- 
Philip

Making GNU GCC choose_multiplier in expmed.c significantly faster

[colinb2 .]

Feel free to copy this email and attachment to anyone who might be interested.
I'm very happy to answer any questions anyone has.
The program can be compiled and run like this on Linux with GNU GCC:
gcc -O2 -o expmed2.exe expmed2.c
./expmed2.exe

This email deals with making part of the GNU GCC compiler - integer division
by a constant divisor - faster. (The calculation of the parameters for the
machine code will be faster; compiled programs won't run faster.)
Further down I mention inequality (1) which can be used to make the LLVM
compiler somewhat faster, because that currently uses code based on (2).
I don't know what - if anything - the Java JVM uses for this, or how other
compilers do this, but these ideas may be useful there.

By significantly faster I mean I have benchmarked alternative versions of
choose_multiplier which on my low specification netbook can take maybe less than
half the time the current version takes. Time saved in compilation is much less
important than time saved in running compiled programs, but the code for the
faster versions is about the same length as the code for the current version,
and is only a bit more complicated, so is worth considering?

A short summary of the following is that choose_multiplier currently uses an
elegant algorithm due to Granlund & Montgomery, but which as implemented seems
rather slow. We can make it faster while retaining the basic structure, and
using a different, mostly equivalent, algorithm, may be a bit faster than that.

Licencing: in Fractint people's words "Don't want money, want recognition!"
The version choose_multiplier_v2 is based - but improves - on what's in
the GCC choose_multiplier function in file expmed.c, so the GCC licence.
The version choose_multiplier_v4 is based - but improves - on magicu2 in
"Hacker's Delight", so the licence is you needn't acknowledge the source
but it would be nice to credit the code as from
magicu2 in "Hacker's Delight" by Henry S Warren http://hackersdelight.org
with improvements by Colin Bartlett .
This latter also applies to choose_multiplier_power2_divrem because that
is also an obvious (?) idea from magicu2 in "Hacker's Delight".  */

The idea is using "wide int" seems slow compared to "unsigned HOST_WIDE_INT",
so it makes sense to avoid using "wide int" as much as possible. We can easily
rewrite choose_multiplier to only use "wide int" to calculate the initial mlow;
this is choose_multiplier_v2. An alternative for choose_multiplier_v2 completely
avoids using "wide int" by iterating upwards for the initial mlow, but if that
benchmarks as faster than using "wide int" even once (I suspect it might) then
just iterating upwards may even be a bit faster; this is choose_multiplier_v4.

The attachment is self-contained, and I have checked that the values produced
agree with a "Hacker's Delight" table of M/2^s for small d and n=precision=32.

What follows is a short summary of the theory, applying it to choose_multiplier.

Problem: find M/2^s so that for 0<=i<=iMax>=d we have floor(i/d)=floor(i*M/2^s).
Let qc=floor((iMax+1)/d); nc=qc*d-1; lgup=ceiling(log2(d)).
Given s let M=floor(2^s/d)+1; delta=M*d-2^s.

For GCC choose_multiplier:

* equivalent necessary & sufficient conditions:
(1) 0
#include 
#include 

#define HOST_WIDE_INT int
#define HOST_BITS_PER_WIDE_INT 32
#define HOST_BITS_PER_DOUBLE_INT (2 * HOST_BITS_PER_WIDE_INT)

/* Making GNU GCC choose_multiplier in expmed.c significantly faster.
   By which we mean up to 50% or more faster for the compiler to calculate
   parameters for the machine code for integer division by a constant divisor.
   (Compiled programs won't run faster.)
   Licencing: in Fractint people's words "Don't want money, want recognition!"
   The version choose_multiplier_v2 is based - but improves - on what's in
   the GCC choose_multiplier function in file expmed.c, so the GCC licence.
   The version choose_multiplier_v4 is based - but improves - on magicu2 in
   "Hacker's Delight", so the licence is you needn't acknowledge the source
   but it would be nice to credit the code as from
   magicu2 in "Hacker's Delight" by Henry S Warren http://hackersdelight.org
   with improvements by Colin Bartlett .
   This latter also applies to choose_multiplier_power2_divrem because that
   is also an obvious (?) idea from magicu2 in "Hacker's Delight".  */

int
ceil_log2 (unsigned long long int iv)
{
  /* for now do it the long way */
  int s;
  if (iv == 0) return -1;
  iv -= 1;
  s = 0;
  while (iv)
{
  s += 1;
  iv = iv >> 1;
}
  return s;
}

void
gcc_assert (int iv)
{
  if (iv) return;
  printf ("gcc_assert error\n");
  exit (1);
}



/* Choose a minimal N + 1 bit approximation to 1/D that can be used to
   replace division by D, and put the least significant N bits of the result
   in *MULTIPLIER_PTR and return the most significant bit.

   The width of operations is N (should be <= HOST_BITS_PER_WIDE_INT), the
   needed precision is in PRECISION (should be <= N).

   PREC

Re: Repo conversion troubles.

[Eric S. Raymond]

Jonathan Wakely :
> On Tue, 10 Jul 2018 at 09:19, Jonathan Wakely  wrote:
> >
> > On Mon, 9 Jul 2018 at 21:00, Eric S. Raymond  wrote:
> > >
> > > Bernd Schmidt :
> > > > On 07/09/2018 09:19 PM, Eric S. Raymond wrote:
> > > > > Last time I did a comparison between SVN head and the git conversion
> > > > > tip they matched exactly.  This time I have mismatches in the 
> > > > > following
> > > > > files.
> > > >
> > > > So what are the diffs? Are we talking about small differences (like one
> > > > change missing) or large-scale mismatches?
> > >
> > > Large-scale, I'm afraid.  The context diff is about a GLOC.
> >
> > I don't see how that's possible. Most of those files are tiny, or
> > change very rarely, so I don't see how that large a diff can happen.
> >
> > Take zlib/configure.ac and zlib/configure, there's only been one
> > change in the past 18 months: https://gcc.gnu.org/r261739
> > That change didn't touch the other files in the list.
> >
> > libtool.m4 has one change in the past 2 years (just a few days ago):
> > https://gcc.gnu.org/r262451
> > That was also tiny, and didn't touch the other files.
> >
> > maintainer-scripts/crontab only has one change in the past 6 months:
> > https://gcc.gnu.org/r259637
> > That was a tiny change, and didn't touch any other files.
> >
> > None of those were merges from any other branch.
> 
> libtool.m4
> ltmain.sh
> 
> Changed by https://gcc.gnu.org/r262451
> 
> libvtv/ChangeLog
> libvtv/configure
> libvtv/testsuite/lib/libvtv.exp
> 
> Changed by https://gcc.gnu.org/r257809 https://gcc.gnu.org/r259462
> https://gcc.gnu.org/r259487 https://gcc.gnu.org/r259837
> https://gcc.gnu.org/r259838 (but mostly one line changes).
> 
> lto-plugin/ChangeLog
> lto-plugin/configure
> lto-plugin/lto-plugin.c
> 
> Changed by https://gcc.gnu.org/r259462 and https://gcc.gnu.org/r260960
> 
> MAINTAINERS
> 
> This file sees a air bit of churn, but all one line changes.
> https://gcc.gnu.org/viewcvs/gcc/trunk/MAINTAINERS?view=log
> 
> maintainer-scripts/ChangeLog
> maintainer-scripts/crontab
> maintainer-scripts/gcc_release
> 
> Changed by https://gcc.gnu.org/r257045 and https://gcc.gnu.org/r259637
> and https://gcc.gnu.org/r259881
> 
> Makefile.def
> Makefile.in
> Makefile.tpl
> 
> Changed by https://gcc.gnu.org/r261717 (which didn't touch any other
> files) but also by some large changes, which might have been merges:
> https://gcc.gnu.org/r255195 (large removal of feature)
> https://gcc.gnu.org/r259669 https://gcc.gnu.org/r259755
> https://gcc.gnu.org/r261304 (another large feature removal)
> https://gcc.gnu.org/r262267
> 
> zlib/configure
> zlib/configure.ac
> 
> Changed by https://gcc.gnu.org/r261739
> 
> There's no single change that touched all of them. Not even two or
> three changes that seem seem to have anything in common, except for
> autoconf regeneration, which happens frequently throughout GCC's
> history.

I don't know what's going on either, yet.  I'm trying to idenify the
earliest point of content mismatch now.

Thanks for all this data.  It may help a lot.
-- 
http://www.catb.org/~esr/";>Eric S. Raymond

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: Good news, bad news on the repository conversion

[Aldy Hernandez]

Wait, there's a pot of money for making SVN go away?  Sign me up!
While we're at it, let's start one for TCL and dejagnu!
On Mon, Jul 9, 2018 at 6:58 PM Jeff Law  wrote:
>
> On 07/09/2018 10:53 AM, Janus Weil wrote:
> > 2018-07-09 18:35 GMT+02:00 Eric S. Raymond :
> >> David Edelsohn :
>  The truth is we're near the bleeding edge of what conventional tools
>  and hardware can handle gracefully.  Most jobs with working sets as
>  big as this one's do only comparatively dumb operations that can be
>  parallellized and thrown on a GPU or supercomputer.  Most jobs with
>  the algorithmic complexity of repository surgery have *much* smaller
>  working sets.  The combination of both extrema is hard.
> >>>
> >>> If you come to the conclusion that the GCC Community could help with
> >>> resources, such as the GNU Compile Farm or paying for more RAM, let us
> >>> know.
> >>
> >> 128GB of DDR4 registered RAM would allow me to run conversions with my
> >> browser up, but be eye-wateringly expensive.  Thanks, but I'm not
> >> going to yell for that help
> >
> > I for one would certainly be happy to donate some spare bucks towards
> > beastie RAM if it helps to get the GCC repo converted to git in a
> > timely manner, and I'm sure there are other GCC
> > developers/users/sympathizers who'd be willing to join in. So, where
> > do we throw those bucks?
> I'd be willing to throw some $$$ at this as well.
> Jeff
>

Re: Repo conversion troubles.

[Jonathan Wakely]

On Tue, 10 Jul 2018 at 09:19, Jonathan Wakely  wrote:
>
> On Mon, 9 Jul 2018 at 21:00, Eric S. Raymond  wrote:
> >
> > Bernd Schmidt :
> > > On 07/09/2018 09:19 PM, Eric S. Raymond wrote:
> > > > Last time I did a comparison between SVN head and the git conversion
> > > > tip they matched exactly.  This time I have mismatches in the following
> > > > files.
> > >
> > > So what are the diffs? Are we talking about small differences (like one
> > > change missing) or large-scale mismatches?
> >
> > Large-scale, I'm afraid.  The context diff is about a GLOC.
>
> I don't see how that's possible. Most of those files are tiny, or
> change very rarely, so I don't see how that large a diff can happen.
>
> Take zlib/configure.ac and zlib/configure, there's only been one
> change in the past 18 months: https://gcc.gnu.org/r261739
> That change didn't touch the other files in the list.
>
> libtool.m4 has one change in the past 2 years (just a few days ago):
> https://gcc.gnu.org/r262451
> That was also tiny, and didn't touch the other files.
>
> maintainer-scripts/crontab only has one change in the past 6 months:
> https://gcc.gnu.org/r259637
> That was a tiny change, and didn't touch any other files.
>
> None of those were merges from any other branch.

libtool.m4
ltmain.sh

Changed by https://gcc.gnu.org/r262451

libvtv/ChangeLog
libvtv/configure
libvtv/testsuite/lib/libvtv.exp

Changed by https://gcc.gnu.org/r257809 https://gcc.gnu.org/r259462
https://gcc.gnu.org/r259487 https://gcc.gnu.org/r259837
https://gcc.gnu.org/r259838 (but mostly one line changes).

lto-plugin/ChangeLog
lto-plugin/configure
lto-plugin/lto-plugin.c

Changed by https://gcc.gnu.org/r259462 and https://gcc.gnu.org/r260960

MAINTAINERS

This file sees a air bit of churn, but all one line changes.
https://gcc.gnu.org/viewcvs/gcc/trunk/MAINTAINERS?view=log

maintainer-scripts/ChangeLog
maintainer-scripts/crontab
maintainer-scripts/gcc_release

Changed by https://gcc.gnu.org/r257045 and https://gcc.gnu.org/r259637
and https://gcc.gnu.org/r259881

Makefile.def
Makefile.in
Makefile.tpl

Changed by https://gcc.gnu.org/r261717 (which didn't touch any other
files) but also by some large changes, which might have been merges:
https://gcc.gnu.org/r255195 (large removal of feature)
https://gcc.gnu.org/r259669 https://gcc.gnu.org/r259755
https://gcc.gnu.org/r261304 (another large feature removal)
https://gcc.gnu.org/r262267

zlib/configure
zlib/configure.ac

Changed by https://gcc.gnu.org/r261739

There's no single change that touched all of them. Not even two or
three changes that seem seem to have anything in common, except for
autoconf regeneration, which happens frequently throughout GCC's
history.

Re: Repo conversion troubles.

[Jonathan Wakely]

On Mon, 9 Jul 2018 at 21:00, Eric S. Raymond  wrote:
>
> Bernd Schmidt :
> > On 07/09/2018 09:19 PM, Eric S. Raymond wrote:
> > > Last time I did a comparison between SVN head and the git conversion
> > > tip they matched exactly.  This time I have mismatches in the following
> > > files.
> >
> > So what are the diffs? Are we talking about small differences (like one
> > change missing) or large-scale mismatches?
>
> Large-scale, I'm afraid.  The context diff is about a GLOC.

I don't see how that's possible. Most of those files are tiny, or
change very rarely, so I don't see how that large a diff can happen.

Take zlib/configure.ac and zlib/configure, there's only been one
change in the past 18 months: https://gcc.gnu.org/r261739
That change didn't touch the other files in the list.

libtool.m4 has one change in the past 2 years (just a few days ago):
https://gcc.gnu.org/r262451
That was also tiny, and didn't touch the other files.

maintainer-scripts/crontab only has one change in the past 6 months:
https://gcc.gnu.org/r259637
That was a tiny change, and didn't touch any other files.

None of those were merges from any other branch.

Re: Good news, bad news on the repository conversion

[Jonathan Wakely]

On Tue, 10 Jul 2018 at 09:10, Alec Teal wrote:
> PS: Migrating what to what?

Git.

> Wasn't the git migration done years ago?

No.

Re: Good news, bad news on the repository conversion

[Jonathan Wakely]

On Mon, 9 Jul 2018, 19:05 Paul Smith,  wrote:
>
> On Mon, 2018-07-09 at 10:57 -0600, Jeff Law wrote:
> > On 07/09/2018 10:53 AM, Janus Weil wrote:
> > > 2018-07-09 18:35 GMT+02:00 Eric S. Raymond :
> > > > David Edelsohn :
> > > > > > The truth is we're near the bleeding edge of what conventional tools
> > > > > > and hardware can handle gracefully.  Most jobs with working sets as
> > > > > > big as this one's do only comparatively dumb operations that can be
> > > > > > parallellized and thrown on a GPU or supercomputer.  Most jobs with
> > > > > > the algorithmic complexity of repository surgery have *much* smaller
> > > > > > working sets.  The combination of both extrema is hard.
> > > > >
> > > > > If you come to the conclusion that the GCC Community could help with
> > > > > resources, such as the GNU Compile Farm or paying for more RAM, let us
> > > > > know.
> > > >
> > > > 128GB of DDR4 registered RAM would allow me to run conversions with my
> > > > browser up, but be eye-wateringly expensive.  Thanks, but I'm not
> > > > going to yell for that help
> > >
> > > I for one would certainly be happy to donate some spare bucks towards
> > > beastie RAM if it helps to get the GCC repo converted to git in a
> > > timely manner, and I'm sure there are other GCC
> > > developers/users/sympathizers who'd be willing to join in. So, where
> > > do we throw those bucks?
> >
> > I'd be willing to throw some $$$ at this as well.
>
> I may be misreading between the lines but I suspect Eric is more hoping
> to get everyone to focus on moving this through before the GCC commit
> count gets even more out of control, than he is asking for a hardware
> handout :).
>
> Maybe the question should rather be, what does the dev community need
> to do to help push this conversion through soonest?


Apart from making the repository read-only (so the commit count
doesn't grow), I don't see what the dev community can do here. Eric is
not waiting on us.

Re: -Wclass-memaccess warning should be in -Wextra, not -Wall

[Marc Glisse]

On Mon, 9 Jul 2018, Martin Sebor wrote:


My point to all of this (and I'm annoyed that I'm having to repeat it
again, as it my first post wasn't clear enough - which it was) was that
any programmer using memcpy/memmove/memset is going to know what they're
getting into.


No, programmers don't always know that.  In fact, it's easy even
for an expert programmer to make the mistake that what looks like
a POD struct can safely be cleared by memset or copied by memcpy
when doing so is undefined because one of the struct members is
of a non-trivial type (such a container like string).


Indeed, especially since some other compilers have implemented string in a 
way that is safe (even if theoretically UB) to memset/memcpy.



Therefore it makes no sense to penalize them by getting them to write
ugly, needless code - regardless of the surrounding politics/codewars.


Quite a lot of thought and discussion went into the design and
implementation of the warning, so venting your frustrations or
insulting those of us involved in the process is unlikely to
help you effect a change.  To make a compelling argument you
need to provide convincing evidence that we have missed
an important use case.  The best way to do that in this forum
is with test cases and/or real world designs that are hampered
by our choice.  That's a high bar to meet for warnings whose
documented purpose is to diagnose "constructions that some
users consider questionable, and that are easy to avoid (or
modify to prevent the warning)."


I guess the phrasing is a bit weak, "some users" obviously has to refer to 
a significant proportion of users, "easy to avoid" cannot have too many 
drawbacks (in particular, generated code should be of equivalent quality), 
etc.


-Wclass-memaccess fits the "easy to avoid" quite well, since a simple cast 
disables it. -Wmaybe-uninitialized is much worse: it produces many false 
positives, that change with every release and are super hard to avoid. And 
even in the "easy to avoid" category where we don't want to litter the 
code with casts to quiet the warnings, I find -Wsign-compare way worse in 
practice than -Wclass-memaccess.


--
Marc Glisse

Re: Good news, bad news on the repository conversion

[Alec Teal]

Is this still an issue? (I missed the convo due to an overzealous spam 
filter; this is the only message I have)



I often use AWS Spot instances (bidding on instances other people 
previsioned but put up for auction as it's not always needed) to get 
results extremely quickly without hearing a fan or to test changes on a 
"large" system.


What do you need and how long (roughly, eg days, hours...)?

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-purpose-instances.html 



https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/memory-optimized-instances.html

Take your pick. m4.16xlarge is 64 cores and 256Gib of RAM, x1e.16xlarge 
64 cores, just shy of 2 Tb of ram, x1e.32xlarge is 128 cores and 3.9 Tb 
of Ram


Alec


PS: Migrating what to what? Wasn't the git migration done years ago? 
Remember I only have the quoted message!



On 09/07/18 21:03, Eric S. Raymond wrote:

Florian Weimer :

* Eric S. Raymond:


The bad news is that my last test run overran the memnory capacity of
the 64GB Great Beast.  I shall have to find some way of reducing the
working set, as 128GB DD4 memory is hideously expensive.

Do you need interactive access to the machine, or can we run the job
for you?

If your application is not NUMA-aware, we probably need something that
has 128 GiB per NUMA node, which might be bit harder to find, but I'm
sure many of us have suitable lab machines which could be temporarily
allocated for that purpose.

I would need interactive access.

But that's now one level way from the principal problem; there is
somme kind of recent metadata damage - or maybe some "correct" but
weird and undocumented stream semantics that reposurgeon doesn't know
how to emulate - that is blocking correct conversion.

Re: -Wclass-memaccess warning should be in -Wextra, not -Wall

[Jonathan Wakely]

On Tue, 10 Jul 2018, 02:22 Soul Studios wrote:
> My point to all of this (and I'm annoyed that I'm having to repeat it
> again, as it my first post wasn't clear enough - which it was) was that
> any programmer using memcpy/memmove/memset is going to know what they're
> getting into.


It was clear in your first post, but that doesn't make it correct. The
statement "any programmer [invoking undefined behaviour] is going to
know what they're getting into" is laughable.

A programmer doing something that causes undefined behaviour probably
doesn't know what they're doing, even if they think they do. And
that's when warnings are useful.

I've seen far more cases of assignment operators implemented with
memcpy that were wrong and broken and due to ignorance or incompetence
than I have seen them done by programmers who knew what they were
doing, or knew what they were getting into. There are programmers who
come from C, and don't realise that a std::string shouldn't be copied
with memcpy. There are programmers who are too lazy to write out
memberwise assignment for each member, so just want to save a few
lines of code by copying everything in one go with memcpy. There are
lots of other ways to do it wrong. Your statement is simply not based
in fact, it's more likely based on your limited experience, and
assumption that everybody is probably doing what you're doing.

If you want to rely on memcpy (and don't want to report
missed-optimization bugs so the compiler can do a better job of
optimizing the non-memcpy code) then you can usually create a
trivially-copyable struct hoding your data members, and then
encapsulate that in a higher-level abstraction which establishes and
enforces the class invariants. The higher-level type has non-trivial
constructors/destructor/whatever but can still just use memcpy on the
trivially-copyable subobject.