Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-08-12 Thread Michael Catanzaro 
On Thu, Aug 12 2021 at 11:41:12 AM -0600, Jeff Law 
 wrote:

I don't think anyone is actively working on reducing that set.   But
that to me would be quite low priority.


Actually Marek Polacek has been reporting bugs! See e.g. 
https://bugzilla.redhat.com/show_bug.cgi?id=1990111 for a recent 
success case where I was able to get LTO working for WebKitGTK thanks 
to some investigation by Jakub Jelinek.


Michael

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-08-12 Thread Jeff Law 



On 8/12/2021 10:09 AM, Vít Ondruch wrote:
Just wonder, have somebody inherited this feature after Jeff? And is 
anybody actually working on the LTO?



~~~

$ grep -R lto | grep -E 'global|define' | grep nil | wc -l
190
I don't think anyone is actively working on reducing that set.   But 
that to me would be quite low priority.


Addressing the -ffat-lto-objects would seem to be more important as it 
would significantly reduce build times.  I did a test run a while back 
and identified all the packages that would need to get fixed, but 
changed jobs before I could start fixing them.


We're doing some light Fedora work here, which I expect to ramp up again 
in several months.  Reducing build times is something I can make a case 
to mgt for as it'll affect some of the work we're doing rather 
significantly.


Jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-08-12 Thread Vít Ondruch 
Just wonder, have somebody inherited this feature after Jeff? And is 
anybody actually working on the LTO?



~~~

$ grep -R lto | grep -E 'global|define' | grep nil | wc -l
190

~~~


Vít


Dne 30. 12. 20 v 20:52 Ben Cotton napsal(a):

https://fedoraproject.org/wiki/Changes/LTOBuildImprovements


== Summary ==
Currently all packages that are not opted out of LTO include
-ffat-lto-objects in their build flags.  This proposal would remove
-ffat-lto-objects from the default LTO flags and only use it for
packages that actually need it.

== Owner ==
* Name: [[User:law | Jeff Law]]
* Email: l...@redhat.com


== Detailed Description ==
-ffat-lto-objects was added to the default LTO flags to ensure that
any installed .o/.a files included actual compiled code rather than
just LTO bytecodes (which are stripped after the install phase).
However, that is wasteful from a compile-time standpoint as few
packages actually install any .o/.a files.

This proposal would remove -ffat-lto-objects from the default LTO
flags and packages that actually need the option would have to opt-in
via an RPM macro in their .spec file.  This should significantly
improve build times for most packages in Fedora.

To ensure that we can identify packages that need the opt-in now and
in the future, the plan is to pass to brp-strip-lto a flag indicating
whether or not the package has opted into -ffat-lto-objects.  If
brp-strip-lto finds .o/.a files, but the package has not opted into
-ffat-lto-objects, then brp-strip-lto would signal an error.


== Benefit to Fedora ==
The key benefit to Fedora is improved package build times and lower
load on the builders.

== Scope ==
* Proposal owners: The feature owner (Jeff Law) will need to settle on
a suitable RPM macro to indicate an opt-in to -ffat-lto-objects,
implement the necessary tests in brp-strip-lto and opt-in the initial
set of packages.  This will be accomplished by doing the prototype
implementation locally, building all the Fedora packages to generate
the opt-in set.  Committing the necessary opt-ins, then committing the
necessary changes to the RPM macros.

* Other developers:  There should be minimal work for other
developers.  The most likely scenarios where other developers will
need to get involved would include:
# Packages which are excluded from x86_64 builds and which need the
opt-in will need the appropriate package owners to add the opt-in.
# Packages which are FTBFS when the builds are run to find the set of
packages that need to opt-in and which need to opt-in will need
packager attention.
# It is possible that the faster builds may trigger build failures in
packages that have missing dependencies in their Makefiles.  We saw a
few of these during the initial LTO work and those packages were
either fixed or -j parallelism removed.  This work may expose more of
these problems.

I expect these all to be relatively rare occurences, but with 9000+
packages in Fedora I wouldn't be surprised if we see a few of each of
these issues.

* Release engineering: [https://pagure.io/releng/issues #Releng issue
number] (a check of an impact with Release Engineering is needed) This
should have no release engineering impacts.
* Policies and guidelines: The packaging guidelines will need to be
updated to document the new macro.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: This proposal does not align with any
current Fedora Objectives.

== Upgrade/compatibility impact ==
This change should have zero impact on upgrade/compatibility.  In
fact, the change should have no user visible impacts.


== How To Test ==
No special testing is needed.  Any issues with this proposal will show
up as FTBFS issues.


== User Experience ==
Users should see no changes to the user experience.

== Dependencies ==
Packages which need to opt-in to -ffat-lto-objects will need their
.spec files updated to include the
new macro.


== Contingency Plan ==
If this can not be completed by final development freeze, then the RPM
macro changes would not be installed and the change could defer to
Fedora 35.
* Contingency mechanism: Proposal owner will only commit the RPM macro
changes once the opt-in package set has been identified and opt-ins
added to those package's spec files.  So no special contingency
mechanism should be needed
* Contingency deadline:  It is most beneficial to have this completed
before the mass rebuild; however, the drop dead date should be beta
freeze.
* Blocks release? No
* Blocks product? No

== Documentation ==
No upstream documentation.  Packaging guidelines will need a minor update.

== Release Notes ==
I do not expect this change to require any release notes.



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-02-08 Thread Jeff Law 


On 2/2/21 5:03 PM, Ian McInerney wrote:
> On Mon, Jan 4, 2021 at 7:28 PM Jeff Law  > wrote:
>
>
> snip...
> >
> > What is this macro going to be called?  I would like to get an early
> > start on updating my packages.
> No idea.  I'm open to suggestions.
>
>
> Any update on the name of this macro/its implementation? I am working
> on revising a spec file that has a static library included and if this
> macro is available to use already I would like to add it as long as I
> am working in the spec.
No movement yet.  I had to push this (and other) stuff down to deal with
a variety of F34 firedrills.  I expect to come back to it during the F35
cycle.

jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-02-02 Thread Ian McInerney 
On Mon, Jan 4, 2021 at 7:28 PM Jeff Law  wrote:

>
> snip...
> >
> > What is this macro going to be called?  I would like to get an early
> > start on updating my packages.
> No idea.  I'm open to suggestions.
>
>
Any update on the name of this macro/its implementation? I am working on
revising a spec file that has a static library included and if this macro
is available to use already I would like to add it as long as I am working
in the spec.

-Ian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-04 Thread Tom Stellard 

On 1/4/21 11:28 AM, Jeff Law wrote:



On 1/4/21 12:22 PM, Tom Stellard wrote:

On 12/30/20 11:52 AM, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/LTOBuildImprovements


== Summary ==
Currently all packages that are not opted out of LTO include
-ffat-lto-objects in their build flags.  This proposal would remove
-ffat-lto-objects from the default LTO flags and only use it for
packages that actually need it.

== Owner ==
* Name: [[User:law | Jeff Law]]
* Email: l...@redhat.com


== Detailed Description ==
-ffat-lto-objects was added to the default LTO flags to ensure that
any installed .o/.a files included actual compiled code rather than
just LTO bytecodes (which are stripped after the install phase).
However, that is wasteful from a compile-time standpoint as few
packages actually install any .o/.a files.

This proposal would remove -ffat-lto-objects from the default LTO
flags and packages that actually need the option would have to opt-in
via an RPM macro in their .spec file.  This should significantly
improve build times for most packages in Fedora.



What is this macro going to be called?  I would like to get an early
start on updating my packages.

No idea.  I'm open to suggestions.

IIRC we had kicked around the idea of having the clang/llvm path
instantiate the LTO bytecodes in .o/.a files, which would avoid these
issues for packages using clang/llvm.  Is that something we still want
to pursue?  I vaguely recall discussing this with David B. and he came
up with a reason why that needed to happen before brp-strip-lto, but I
can't remember any of the details at this point.



Yes, David is still working on a solution for clang/llvm.  I'm not sure, 
though, at what point in the RPM build process the LTO bytecode will get

codegen'd.

-Tom


Jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-04 Thread Jeff Law 


On 1/2/21 10:39 AM, Ian McInerney wrote:
>
>
> On Sat, Jan 2, 2021 at 5:33 PM Jeff Law  > wrote:
>
>
>
> On 12/30/20 3:48 PM, Ian McInerney wrote:
> >
> >
> > On Wed, Dec 30, 2020 at 7:54 PM Ben Cotton  
> > >> wrote:
> >
> >     https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
> 
> >      >
> >
> >
> >     == Summary ==
> >     Currently all packages that are not opted out of LTO include
> >     -ffat-lto-objects in their build flags.  This proposal would
> remove
> >     -ffat-lto-objects from the default LTO flags and only use it for
> >     packages that actually need it.
> >
> >     == Owner ==
> >     * Name: [[User:law | Jeff Law]]
> >     * Email: l...@redhat.com 
> >
> >
> >
> >     == Detailed Description ==
> >     -ffat-lto-objects was added to the default LTO flags to
> ensure that
> >     any installed .o/.a files included actual compiled code
> rather than
> >     just LTO bytecodes (which are stripped after the install phase).
> >     However, that is wasteful from a compile-time standpoint as few
> >     packages actually install any .o/.a files.
> >
> >     This proposal would remove -ffat-lto-objects from the
> default LTO
> >     flags and packages that actually need the option would have
> to opt-in
> >     via an RPM macro in their .spec file.  This should significantly
> >     improve build times for most packages in Fedora.
> >
> >
> > Does this mean that packages that are explicitly shipping a static
> > library to the end user need to enable this macro to allow the
> > installed static library to be usable by an end-user's compiler? If
> > this is the case, then the packaging guidelines should be updated to
> > reflect this.
> Yes and the change request reflects that an update to the packaging
> guidelines is necessary.
>
>
> The proposal only says the macro will be documented, but doesn't go
> into any more detail. I think it would be beneficial if the proposal
> mentioned the macro would become required for static library packages
> and that policy needs to be added - since that is adding a new MUST
> section to the packaging guidelines.
ACK.  I'll add some additional text to the proposal.

jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-04 Thread Jeff Law 


On 1/4/21 12:22 PM, Tom Stellard wrote:
> On 12/30/20 11:52 AM, Ben Cotton wrote:
>> https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
>>
>>
>> == Summary ==
>> Currently all packages that are not opted out of LTO include
>> -ffat-lto-objects in their build flags.  This proposal would remove
>> -ffat-lto-objects from the default LTO flags and only use it for
>> packages that actually need it.
>>
>> == Owner ==
>> * Name: [[User:law | Jeff Law]]
>> * Email: l...@redhat.com
>>
>>
>> == Detailed Description ==
>> -ffat-lto-objects was added to the default LTO flags to ensure that
>> any installed .o/.a files included actual compiled code rather than
>> just LTO bytecodes (which are stripped after the install phase).
>> However, that is wasteful from a compile-time standpoint as few
>> packages actually install any .o/.a files.
>>
>> This proposal would remove -ffat-lto-objects from the default LTO
>> flags and packages that actually need the option would have to opt-in
>> via an RPM macro in their .spec file.  This should significantly
>> improve build times for most packages in Fedora.
>>
>
> What is this macro going to be called?  I would like to get an early
> start on updating my packages.
No idea.  I'm open to suggestions.

IIRC we had kicked around the idea of having the clang/llvm path
instantiate the LTO bytecodes in .o/.a files, which would avoid these
issues for packages using clang/llvm.  Is that something we still want
to pursue?  I vaguely recall discussing this with David B. and he came
up with a reason why that needed to happen before brp-strip-lto, but I
can't remember any of the details at this point.

Jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-04 Thread Tom Stellard 

On 12/30/20 11:52 AM, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/LTOBuildImprovements


== Summary ==
Currently all packages that are not opted out of LTO include
-ffat-lto-objects in their build flags.  This proposal would remove
-ffat-lto-objects from the default LTO flags and only use it for
packages that actually need it.

== Owner ==
* Name: [[User:law | Jeff Law]]
* Email: l...@redhat.com


== Detailed Description ==
-ffat-lto-objects was added to the default LTO flags to ensure that
any installed .o/.a files included actual compiled code rather than
just LTO bytecodes (which are stripped after the install phase).
However, that is wasteful from a compile-time standpoint as few
packages actually install any .o/.a files.

This proposal would remove -ffat-lto-objects from the default LTO
flags and packages that actually need the option would have to opt-in
via an RPM macro in their .spec file.  This should significantly
improve build times for most packages in Fedora.



What is this macro going to be called?  I would like to get an early 
start on updating my packages.


-Tom


To ensure that we can identify packages that need the opt-in now and
in the future, the plan is to pass to brp-strip-lto a flag indicating
whether or not the package has opted into -ffat-lto-objects.  If
brp-strip-lto finds .o/.a files, but the package has not opted into
-ffat-lto-objects, then brp-strip-lto would signal an error.


== Benefit to Fedora ==
The key benefit to Fedora is improved package build times and lower
load on the builders.

== Scope ==
* Proposal owners: The feature owner (Jeff Law) will need to settle on
a suitable RPM macro to indicate an opt-in to -ffat-lto-objects,
implement the necessary tests in brp-strip-lto and opt-in the initial
set of packages.  This will be accomplished by doing the prototype
implementation locally, building all the Fedora packages to generate
the opt-in set.  Committing the necessary opt-ins, then committing the
necessary changes to the RPM macros.

* Other developers:  There should be minimal work for other
developers.  The most likely scenarios where other developers will
need to get involved would include:
# Packages which are excluded from x86_64 builds and which need the
opt-in will need the appropriate package owners to add the opt-in.
# Packages which are FTBFS when the builds are run to find the set of
packages that need to opt-in and which need to opt-in will need
packager attention.
# It is possible that the faster builds may trigger build failures in
packages that have missing dependencies in their Makefiles.  We saw a
few of these during the initial LTO work and those packages were
either fixed or -j parallelism removed.  This work may expose more of
these problems.

I expect these all to be relatively rare occurences, but with 9000+
packages in Fedora I wouldn't be surprised if we see a few of each of
these issues.

* Release engineering: [https://pagure.io/releng/issues #Releng issue
number] (a check of an impact with Release Engineering is needed) This
should have no release engineering impacts.
* Policies and guidelines: The packaging guidelines will need to be
updated to document the new macro.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: This proposal does not align with any
current Fedora Objectives.

== Upgrade/compatibility impact ==
This change should have zero impact on upgrade/compatibility.  In
fact, the change should have no user visible impacts.


== How To Test ==
No special testing is needed.  Any issues with this proposal will show
up as FTBFS issues.


== User Experience ==
Users should see no changes to the user experience.

== Dependencies ==
Packages which need to opt-in to -ffat-lto-objects will need their
.spec files updated to include the
new macro.


== Contingency Plan ==
If this can not be completed by final development freeze, then the RPM
macro changes would not be installed and the change could defer to
Fedora 35.
* Contingency mechanism: Proposal owner will only commit the RPM macro
changes once the opt-in package set has been identified and opt-ins
added to those package's spec files.  So no special contingency
mechanism should be needed
* Contingency deadline:  It is most beneficial to have this completed
before the mass rebuild; however, the drop dead date should be beta
freeze.
* Blocks release? No
* Blocks product? No

== Documentation ==
No upstream documentation.  Packaging guidelines will need a minor update.

== Release Notes ==
I do not expect this change to require any release notes.



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-04 Thread Jeff Law 


On 1/4/21 2:27 AM, Florian Weimer wrote:
>
>>> A lot of the existing RPM post-processing steps detect, report, and
>>> ignore errors because the generated RPM package might still be partially
>>> useful.
>> True, but ignoring the error in this case runs the very real risk that a
>> package could install a .o/.a file with no code/symbols.  That in turn
>> can cause downstream FTBFS errors in other packages and all kinds of
>> headaches on developer systems.  Failing the build here seems much safer.
>>
>> Contrast to ignoring a dwz error.  The resulting binary RPMs are still
>> very much usable, the debuginfo packages are just larger than is
>> strictly necessary.
> The downside is that toolchain bugs tend to go unnoticed and aren't
> fixed as a result.
Right.  And I think that's been a real problem.  I really dislike
ignoring errors :(

jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-04 Thread Florian Weimer 
* Jeff Law:

> On 1/2/21 3:13 AM, Florian Weimer wrote:
>> * Ben Cotton:
>>
>>> To ensure that we can identify packages that need the opt-in now and
>>> in the future, the plan is to pass to brp-strip-lto a flag indicating
>>> whether or not the package has opted into -ffat-lto-objects.  If
>>> brp-strip-lto finds .o/.a files, but the package has not opted into
>>> -ffat-lto-objects, then brp-strip-lto would signal an error.
>> And presumably fail the build?

> Yes.  The point here is to ensure that if the build installs .o/.a files
> that it must have opted into -ffat-lto-objects so that we don't end up
> with useless .o/.a files in our binary RPM packages.

Makes sense.

>> A lot of the existing RPM post-processing steps detect, report, and
>> ignore errors because the generated RPM package might still be partially
>> useful.
>
> True, but ignoring the error in this case runs the very real risk that a
> package could install a .o/.a file with no code/symbols.  That in turn
> can cause downstream FTBFS errors in other packages and all kinds of
> headaches on developer systems.  Failing the build here seems much safer.
>
> Contrast to ignoring a dwz error.  The resulting binary RPMs are still
> very much usable, the debuginfo packages are just larger than is
> strictly necessary.

The downside is that toolchain bugs tend to go unnoticed and aren't
fixed as a result.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-02 Thread Ian McInerney 
On Sat, Jan 2, 2021 at 5:33 PM Jeff Law  wrote:

>
>
> On 12/30/20 3:48 PM, Ian McInerney wrote:
> >
> >
> > On Wed, Dec 30, 2020 at 7:54 PM Ben Cotton  > > wrote:
> >
> > https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
> > 
> >
> >
> > == Summary ==
> > Currently all packages that are not opted out of LTO include
> > -ffat-lto-objects in their build flags.  This proposal would remove
> > -ffat-lto-objects from the default LTO flags and only use it for
> > packages that actually need it.
> >
> > == Owner ==
> > * Name: [[User:law | Jeff Law]]
> > * Email: l...@redhat.com 
> >
> >
> > == Detailed Description ==
> > -ffat-lto-objects was added to the default LTO flags to ensure that
> > any installed .o/.a files included actual compiled code rather than
> > just LTO bytecodes (which are stripped after the install phase).
> > However, that is wasteful from a compile-time standpoint as few
> > packages actually install any .o/.a files.
> >
> > This proposal would remove -ffat-lto-objects from the default LTO
> > flags and packages that actually need the option would have to opt-in
> > via an RPM macro in their .spec file.  This should significantly
> > improve build times for most packages in Fedora.
> >
> >
> > Does this mean that packages that are explicitly shipping a static
> > library to the end user need to enable this macro to allow the
> > installed static library to be usable by an end-user's compiler? If
> > this is the case, then the packaging guidelines should be updated to
> > reflect this.
> Yes and the change request reflects that an update to the packaging
> guidelines is necessary.
>
>
The proposal only says the macro will be documented, but doesn't go into
any more detail. I think it would be beneficial if the proposal mentioned
the macro would become required for static library packages and that policy
needs to be added - since that is adding a new MUST section to the
packaging guidelines.

-Ian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-02 Thread Jeff Law 


On 12/30/20 3:48 PM, Ian McInerney wrote:
>
>
> On Wed, Dec 30, 2020 at 7:54 PM Ben Cotton  > wrote:
>
> https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
> 
>
>
> == Summary ==
> Currently all packages that are not opted out of LTO include
> -ffat-lto-objects in their build flags.  This proposal would remove
> -ffat-lto-objects from the default LTO flags and only use it for
> packages that actually need it.
>
> == Owner ==
> * Name: [[User:law | Jeff Law]]
> * Email: l...@redhat.com 
>
>
> == Detailed Description ==
> -ffat-lto-objects was added to the default LTO flags to ensure that
> any installed .o/.a files included actual compiled code rather than
> just LTO bytecodes (which are stripped after the install phase).
> However, that is wasteful from a compile-time standpoint as few
> packages actually install any .o/.a files.
>
> This proposal would remove -ffat-lto-objects from the default LTO
> flags and packages that actually need the option would have to opt-in
> via an RPM macro in their .spec file.  This should significantly
> improve build times for most packages in Fedora.
>
>
> Does this mean that packages that are explicitly shipping a static
> library to the end user need to enable this macro to allow the
> installed static library to be usable by an end-user's compiler? If
> this is the case, then the packaging guidelines should be updated to
> reflect this.
Yes and the change request reflects that an update to the packaging
guidelines is necessary.


Jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-02 Thread Jeff Law 


On 1/2/21 3:13 AM, Florian Weimer wrote:
> * Ben Cotton:
>
>> To ensure that we can identify packages that need the opt-in now and
>> in the future, the plan is to pass to brp-strip-lto a flag indicating
>> whether or not the package has opted into -ffat-lto-objects.  If
>> brp-strip-lto finds .o/.a files, but the package has not opted into
>> -ffat-lto-objects, then brp-strip-lto would signal an error.
> And presumably fail the build?
Yes.  The point here is to ensure that if the build installs .o/.a files
that it must have opted into -ffat-lto-objects so that we don't end up
with useless .o/.a files in our binary RPM packages.

>
> A lot of the existing RPM post-processing steps detect, report, and
> ignore errors because the generated RPM package might still be partially
> useful.
True, but ignoring the error in this case runs the very real risk that a
package could install a .o/.a file with no code/symbols.  That in turn
can cause downstream FTBFS errors in other packages and all kinds of
headaches on developer systems.  Failing the build here seems much safer.

Contrast to ignoring a dwz error.  The resulting binary RPMs are still
very much usable, the debuginfo packages are just larger than is
strictly necessary.

jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2021-01-02 Thread Florian Weimer 
* Ben Cotton:

> To ensure that we can identify packages that need the opt-in now and
> in the future, the plan is to pass to brp-strip-lto a flag indicating
> whether or not the package has opted into -ffat-lto-objects.  If
> brp-strip-lto finds .o/.a files, but the package has not opted into
> -ffat-lto-objects, then brp-strip-lto would signal an error.

And presumably fail the build?

A lot of the existing RPM post-processing steps detect, report, and
ignore errors because the generated RPM package might still be partially
useful.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 34 Change: LTO Build Improvements (System-Wide Change proposal)

2020-12-30 Thread Ian McInerney 
On Wed, Dec 30, 2020 at 7:54 PM Ben Cotton  wrote:

> https://fedoraproject.org/wiki/Changes/LTOBuildImprovements
>
>
> == Summary ==
> Currently all packages that are not opted out of LTO include
> -ffat-lto-objects in their build flags.  This proposal would remove
> -ffat-lto-objects from the default LTO flags and only use it for
> packages that actually need it.
>
> == Owner ==
> * Name: [[User:law | Jeff Law]]
> * Email: l...@redhat.com
>
>
> == Detailed Description ==
> -ffat-lto-objects was added to the default LTO flags to ensure that
> any installed .o/.a files included actual compiled code rather than
> just LTO bytecodes (which are stripped after the install phase).
> However, that is wasteful from a compile-time standpoint as few
> packages actually install any .o/.a files.
>
> This proposal would remove -ffat-lto-objects from the default LTO
> flags and packages that actually need the option would have to opt-in
> via an RPM macro in their .spec file.  This should significantly
> improve build times for most packages in Fedora.
>

Does this mean that packages that are explicitly shipping a static library
to the end user need to enable this macro to allow the installed static
library to be usable by an end-user's compiler? If this is the case, then
the packaging guidelines should be updated to reflect this.


>
> To ensure that we can identify packages that need the opt-in now and
> in the future, the plan is to pass to brp-strip-lto a flag indicating
> whether or not the package has opted into -ffat-lto-objects.  If
> brp-strip-lto finds .o/.a files, but the package has not opted into
> -ffat-lto-objects, then brp-strip-lto would signal an error.
>
>
> == Benefit to Fedora ==
> The key benefit to Fedora is improved package build times and lower
> load on the builders.
>
> == Scope ==
> * Proposal owners: The feature owner (Jeff Law) will need to settle on
> a suitable RPM macro to indicate an opt-in to -ffat-lto-objects,
> implement the necessary tests in brp-strip-lto and opt-in the initial
> set of packages.  This will be accomplished by doing the prototype
> implementation locally, building all the Fedora packages to generate
> the opt-in set.  Committing the necessary opt-ins, then committing the
> necessary changes to the RPM macros.
>
> * Other developers:  There should be minimal work for other
> developers.  The most likely scenarios where other developers will
> need to get involved would include:
> # Packages which are excluded from x86_64 builds and which need the
> opt-in will need the appropriate package owners to add the opt-in.
> # Packages which are FTBFS when the builds are run to find the set of
> packages that need to opt-in and which need to opt-in will need
> packager attention.
> # It is possible that the faster builds may trigger build failures in
> packages that have missing dependencies in their Makefiles.  We saw a
> few of these during the initial LTO work and those packages were
> either fixed or -j parallelism removed.  This work may expose more of
> these problems.
>
> I expect these all to be relatively rare occurences, but with 9000+
> packages in Fedora I wouldn't be surprised if we see a few of each of
> these issues.
>
> * Release engineering: [https://pagure.io/releng/issues #Releng issue
> number] (a check of an impact with Release Engineering is needed) This
> should have no release engineering impacts.
> * Policies and guidelines: The packaging guidelines will need to be
> updated to document the new macro.
> * Trademark approval: N/A (not needed for this Change)
> * Alignment with Objectives: This proposal does not align with any
> current Fedora Objectives.
>
> == Upgrade/compatibility impact ==
> This change should have zero impact on upgrade/compatibility.  In
> fact, the change should have no user visible impacts.
>
>
> == How To Test ==
> No special testing is needed.  Any issues with this proposal will show
> up as FTBFS issues.
>
>
> == User Experience ==
> Users should see no changes to the user experience.
>
> == Dependencies ==
> Packages which need to opt-in to -ffat-lto-objects will need their
> .spec files updated to include the
> new macro.
>
>
> == Contingency Plan ==
> If this can not be completed by final development freeze, then the RPM
> macro changes would not be installed and the change could defer to
> Fedora 35.
> * Contingency mechanism: Proposal owner will only commit the RPM macro
> changes once the opt-in package set has been identified and opt-ins
> added to those package's spec files.  So no special contingency
> mechanism should be needed
> * Contingency deadline:  It is most beneficial to have this completed
> before the mass rebuild; however, the drop dead date should be beta
> freeze.
> * Blocks release? No
> * Blocks product? No
>
> == Documentation ==
> No upstream documentation.  Packaging guidelines will need a minor update.
>
> == Release Notes ==
> I do not expect this change to require any release notes.
>
>
> --
>