Re: "ld -r" on mixed IR/non-IR objects (

2010-12-15 Thread H.J. Lu
On Wed, Dec 15, 2010 at 6:27 PM, Ian Lance Taylor  wrote:
> "H.J. Lu"  writes:
>
>> On Wed, Dec 15, 2010 at 5:23 PM, Ian Lance Taylor  wrote:
>>> On Thu, Dec 9, 2010 at 6:29 PM, H.J. Lu  wrote:
>>>
 The initial implementation of my proposal is available on hjl/lto-mixed
 branch at

 http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
>>>
>>> I don't know how to separate this idea from the other work on that branch.
>>
>> It is implemented on top of 2 stage linking infrastructure.
>
> Yes, which makes it hard to separate.  The idea is distinct from 2 stage
> linking, as far as I can tell.  I'm sure you saw my alternative proposal

That is true.  It happens to need similar book keeping as 2 stage linking.

> to 2 stage linking.

I saw it. Gold and ld can use different approaches to support plugin
as long as they work with the same GCC driver binary.

>
>> Linker knows nothing about the magic names.
>>
>> * Linker action:
>>   o Classify each input object file:
>>     * If there is a ".gnu_object_only" section, it is a mixed object file.
>>     * If there is a IR section, it is an IR object file.
>>
>> Linker checks if an object is claimed by the plugin. If yes,
>> it has an IR section.  Otherwise, it has no IR.
>>
>>     * Otherwise, it is a non-IR object file.
>
> Thanks for the clarification.  I will try again to understand the
> proposal.
>

I will update my proposal. Any feedbacks/comments are welcom/

Thanks.

-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-15 Thread Ian Lance Taylor
"H.J. Lu"  writes:

> On Wed, Dec 15, 2010 at 5:23 PM, Ian Lance Taylor  wrote:
>> On Thu, Dec 9, 2010 at 6:29 PM, H.J. Lu  wrote:
>>
>>> The initial implementation of my proposal is available on hjl/lto-mixed
>>> branch at
>>>
>>> http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
>>
>> I don't know how to separate this idea from the other work on that branch.
>
> It is implemented on top of 2 stage linking infrastructure.

Yes, which makes it hard to separate.  The idea is distinct from 2 stage
linking, as far as I can tell.  I'm sure you saw my alternative proposal
to 2 stage linking.


> Linker knows nothing about the magic names.
>
> * Linker action:
>   o Classify each input object file:
> * If there is a ".gnu_object_only" section, it is a mixed object file.
> * If there is a IR section, it is an IR object file.
>
> Linker checks if an object is claimed by the plugin. If yes,
> it has an IR section.  Otherwise, it has no IR.
>
> * Otherwise, it is a non-IR object file.

Thanks for the clarification.  I will try again to understand the
proposal.

Ian


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-15 Thread H.J. Lu
On Wed, Dec 15, 2010 at 5:23 PM, Ian Lance Taylor  wrote:
> On Thu, Dec 9, 2010 at 6:29 PM, H.J. Lu  wrote:
>
>> The initial implementation of my proposal is available on hjl/lto-mixed
>> branch at
>>
>> http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
>
> I don't know how to separate this idea from the other work on that branch.

It is implemented on top of 2 stage linking infrastructure.

> I'm concerned that this idea appears to deeply embeds knowledge about
> LTO into the linker
> proper.  One of the goals of the plugin approach was to have a clean
> separation, to give us
> more flexibility going forward.  I say "appears to" because I'm not
> sure I entirely understand
> the proposal.  The text is rather sketchy and I haven't seen the patch.

If you have specific questions, please just ask.

> For example, I think it would be a mistake for the linker to know the
> magic names that
> we currently use for LTO sections, and it would definitely be a
> mistake for the linker to
> know anything about the format of the data they contain.

Linker knows nothing about the magic names.

* Linker action:
  o Classify each input object file:
* If there is a ".gnu_object_only" section, it is a mixed object file.
* If there is a IR section, it is an IR object file.

Linker checks if an object is claimed by the plugin. If yes,
it has an IR section.  Otherwise, it has no IR.

* Otherwise, it is a non-IR object file.



-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-15 Thread Ian Lance Taylor
On Thu, Dec 9, 2010 at 6:29 PM, H.J. Lu  wrote:

> The initial implementation of my proposal is available on hjl/lto-mixed
> branch at
>
> http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary

I don't know how to separate this idea from the other work on that branch.

I'm concerned that this idea appears to deeply embeds knowledge about
LTO into the linker
proper.  One of the goals of the plugin approach was to have a clean
separation, to give us
more flexibility going forward.  I say "appears to" because I'm not
sure I entirely understand
the proposal.  The text is rather sketchy and I haven't seen the patch.

For example, I think it would be a mistake for the linker to know the
magic names that
we currently use for LTO sections, and it would definitely be a
mistake for the linker to
know anything about the format of the data they contain.

Ian


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-10 Thread H.J. Lu
On Fri, Dec 10, 2010 at 4:39 PM, H.J. Lu  wrote:
> On Fri, Dec 10, 2010 at 7:13 AM, H.J. Lu  wrote:
>> On Thu, Dec 9, 2010 at 8:55 PM, H.J. Lu  wrote:
>>> On Thu, Dec 9, 2010 at 6:29 PM, H.J. Lu  wrote:
 On Wed, Dec 8, 2010 at 9:36 AM, H.J. Lu  wrote:
> On Wed, Dec 8, 2010 at 5:54 AM, H.J. Lu  wrote:
>> On Wed, Dec 8, 2010 at 1:19 AM, Andi Kleen  wrote:
 On 12/07/2010 04:20 PM, Andi Kleen wrote:
>
> The only problem left is mixing of lto and non lto objects. this right
> now is not handled. IMHO still the best way to handle it is to use
> slim lto and then simply separate link the "left overs" after deleting
> the LTO objects. This can be actually done with objcopy (with some
> limitations), doesn't even need linker support.
>

 Quite possibly a better way to deal with that is to provide a mechanism
 for encapsulating arbitrary binary code objects inside the LTO IR.
>>>
>>> Then you would need to teach your assembler and everything
>>
>> The magic section is generated by linker directly. No changes to
>> assembler is required.
>>
>>> else that may generate ELF objects to generate this magic object. But 
>>> why
>>> not just ELF directly? that is what it is after all.
>>
>> My proposal isn't specific to ELF.
>>
>>>
>>> To be honest I don't really see the point of all this complexity you
>>> guys are proposing just to save fat LTO. Fat LTO is always a bad idea
>>> because it's slow and  does lots of redundant work. If LTO is to become
>>> a more wide spread mode it has to go simply because of the poor
>>> performance.
>>>
>>> With slim LTO passthrough is  very straight-forward: simple pass
>>> through every section that is not LTO and generate code for the LTO
>>> sections. No new magic sections needed at all.
>>>
>>
>> My proposal works on both fat and slim LTO objects.  The idea is
>> you can use "ld -r" on any combination of inputs and its output
>> still works as before "ld -r".
>>
>
> Here is the revised proposal.
>

 The initial implementation of my proposal is available on hjl/lto-mixed
 branch at

 http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary

 Simple case works.  More cleanups are needed.  Feedbacks
 are welcome.

>>>
>>> I checked in patches to remove temporary files.
>>>
>>>
>>
>> More fixes are checked in.  I will try Linux kernel next.
>>
>
> I checked in new fixes. "ld -r" works on Linux kernel build.
> But the final kernel link failed due to unrelated errors.
>

LTO work in BFD linker is done. I will submit a patch in the next
few days, which enables transparent LTO support in BFD linker.
No GCC changes are required.

-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-10 Thread H.J. Lu
On Fri, Dec 10, 2010 at 7:13 AM, H.J. Lu  wrote:
> On Thu, Dec 9, 2010 at 8:55 PM, H.J. Lu  wrote:
>> On Thu, Dec 9, 2010 at 6:29 PM, H.J. Lu  wrote:
>>> On Wed, Dec 8, 2010 at 9:36 AM, H.J. Lu  wrote:
 On Wed, Dec 8, 2010 at 5:54 AM, H.J. Lu  wrote:
> On Wed, Dec 8, 2010 at 1:19 AM, Andi Kleen  wrote:
>>> On 12/07/2010 04:20 PM, Andi Kleen wrote:

 The only problem left is mixing of lto and non lto objects. this right
 now is not handled. IMHO still the best way to handle it is to use
 slim lto and then simply separate link the "left overs" after deleting
 the LTO objects. This can be actually done with objcopy (with some
 limitations), doesn't even need linker support.

>>>
>>> Quite possibly a better way to deal with that is to provide a mechanism
>>> for encapsulating arbitrary binary code objects inside the LTO IR.
>>
>> Then you would need to teach your assembler and everything
>
> The magic section is generated by linker directly. No changes to
> assembler is required.
>
>> else that may generate ELF objects to generate this magic object. But why
>> not just ELF directly? that is what it is after all.
>
> My proposal isn't specific to ELF.
>
>>
>> To be honest I don't really see the point of all this complexity you
>> guys are proposing just to save fat LTO. Fat LTO is always a bad idea
>> because it's slow and  does lots of redundant work. If LTO is to become
>> a more wide spread mode it has to go simply because of the poor
>> performance.
>>
>> With slim LTO passthrough is  very straight-forward: simple pass
>> through every section that is not LTO and generate code for the LTO
>> sections. No new magic sections needed at all.
>>
>
> My proposal works on both fat and slim LTO objects.  The idea is
> you can use "ld -r" on any combination of inputs and its output
> still works as before "ld -r".
>

 Here is the revised proposal.

>>>
>>> The initial implementation of my proposal is available on hjl/lto-mixed
>>> branch at
>>>
>>> http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
>>>
>>> Simple case works.  More cleanups are needed.  Feedbacks
>>> are welcome.
>>>
>>
>> I checked in patches to remove temporary files.
>>
>>
>
> More fixes are checked in.  I will try Linux kernel next.
>

I checked in new fixes. "ld -r" works on Linux kernel build.
But the final kernel link failed due to unrelated errors.


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-10 Thread H.J. Lu
On Thu, Dec 9, 2010 at 8:55 PM, H.J. Lu  wrote:
> On Thu, Dec 9, 2010 at 6:29 PM, H.J. Lu  wrote:
>> On Wed, Dec 8, 2010 at 9:36 AM, H.J. Lu  wrote:
>>> On Wed, Dec 8, 2010 at 5:54 AM, H.J. Lu  wrote:
 On Wed, Dec 8, 2010 at 1:19 AM, Andi Kleen  wrote:
>> On 12/07/2010 04:20 PM, Andi Kleen wrote:
>>>
>>> The only problem left is mixing of lto and non lto objects. this right
>>> now is not handled. IMHO still the best way to handle it is to use
>>> slim lto and then simply separate link the "left overs" after deleting
>>> the LTO objects. This can be actually done with objcopy (with some
>>> limitations), doesn't even need linker support.
>>>
>>
>> Quite possibly a better way to deal with that is to provide a mechanism
>> for encapsulating arbitrary binary code objects inside the LTO IR.
>
> Then you would need to teach your assembler and everything

 The magic section is generated by linker directly. No changes to
 assembler is required.

> else that may generate ELF objects to generate this magic object. But why
> not just ELF directly? that is what it is after all.

 My proposal isn't specific to ELF.

>
> To be honest I don't really see the point of all this complexity you
> guys are proposing just to save fat LTO. Fat LTO is always a bad idea
> because it's slow and  does lots of redundant work. If LTO is to become
> a more wide spread mode it has to go simply because of the poor
> performance.
>
> With slim LTO passthrough is  very straight-forward: simple pass
> through every section that is not LTO and generate code for the LTO
> sections. No new magic sections needed at all.
>

 My proposal works on both fat and slim LTO objects.  The idea is
 you can use "ld -r" on any combination of inputs and its output
 still works as before "ld -r".

>>>
>>> Here is the revised proposal.
>>>
>>
>> The initial implementation of my proposal is available on hjl/lto-mixed
>> branch at
>>
>> http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
>>
>> Simple case works.  More cleanups are needed.  Feedbacks
>> are welcome.
>>
>
> I checked in patches to remove temporary files.
>
>

More fixes are checked in.  I will try Linux kernel next.


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-09 Thread H.J. Lu
On Thu, Dec 9, 2010 at 6:29 PM, H.J. Lu  wrote:
> On Wed, Dec 8, 2010 at 9:36 AM, H.J. Lu  wrote:
>> On Wed, Dec 8, 2010 at 5:54 AM, H.J. Lu  wrote:
>>> On Wed, Dec 8, 2010 at 1:19 AM, Andi Kleen  wrote:
> On 12/07/2010 04:20 PM, Andi Kleen wrote:
>>
>> The only problem left is mixing of lto and non lto objects. this right
>> now is not handled. IMHO still the best way to handle it is to use
>> slim lto and then simply separate link the "left overs" after deleting
>> the LTO objects. This can be actually done with objcopy (with some
>> limitations), doesn't even need linker support.
>>
>
> Quite possibly a better way to deal with that is to provide a mechanism
> for encapsulating arbitrary binary code objects inside the LTO IR.

 Then you would need to teach your assembler and everything
>>>
>>> The magic section is generated by linker directly. No changes to
>>> assembler is required.
>>>
 else that may generate ELF objects to generate this magic object. But why
 not just ELF directly? that is what it is after all.
>>>
>>> My proposal isn't specific to ELF.
>>>

 To be honest I don't really see the point of all this complexity you
 guys are proposing just to save fat LTO. Fat LTO is always a bad idea
 because it's slow and  does lots of redundant work. If LTO is to become
 a more wide spread mode it has to go simply because of the poor
 performance.

 With slim LTO passthrough is  very straight-forward: simple pass
 through every section that is not LTO and generate code for the LTO
 sections. No new magic sections needed at all.

>>>
>>> My proposal works on both fat and slim LTO objects.  The idea is
>>> you can use "ld -r" on any combination of inputs and its output
>>> still works as before "ld -r".
>>>
>>
>> Here is the revised proposal.
>>
>
> The initial implementation of my proposal is available on hjl/lto-mixed
> branch at
>
> http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
>
> Simple case works.  More cleanups are needed.  Feedbacks
> are welcome.
>

I checked in patches to remove temporary files.


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-09 Thread H.J. Lu
On Wed, Dec 8, 2010 at 9:36 AM, H.J. Lu  wrote:
> On Wed, Dec 8, 2010 at 5:54 AM, H.J. Lu  wrote:
>> On Wed, Dec 8, 2010 at 1:19 AM, Andi Kleen  wrote:
 On 12/07/2010 04:20 PM, Andi Kleen wrote:
>
> The only problem left is mixing of lto and non lto objects. this right
> now is not handled. IMHO still the best way to handle it is to use
> slim lto and then simply separate link the "left overs" after deleting
> the LTO objects. This can be actually done with objcopy (with some
> limitations), doesn't even need linker support.
>

 Quite possibly a better way to deal with that is to provide a mechanism
 for encapsulating arbitrary binary code objects inside the LTO IR.
>>>
>>> Then you would need to teach your assembler and everything
>>
>> The magic section is generated by linker directly. No changes to
>> assembler is required.
>>
>>> else that may generate ELF objects to generate this magic object. But why
>>> not just ELF directly? that is what it is after all.
>>
>> My proposal isn't specific to ELF.
>>
>>>
>>> To be honest I don't really see the point of all this complexity you
>>> guys are proposing just to save fat LTO. Fat LTO is always a bad idea
>>> because it's slow and  does lots of redundant work. If LTO is to become
>>> a more wide spread mode it has to go simply because of the poor
>>> performance.
>>>
>>> With slim LTO passthrough is  very straight-forward: simple pass
>>> through every section that is not LTO and generate code for the LTO
>>> sections. No new magic sections needed at all.
>>>
>>
>> My proposal works on both fat and slim LTO objects.  The idea is
>> you can use "ld -r" on any combination of inputs and its output
>> still works as before "ld -r".
>>
>
> Here is the revised proposal.
>

The initial implementation of my proposal is available on hjl/lto-mixed
branch at

http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary

Simple case works.  More cleanups are needed.  Feedbacks
are welcome.

Thanks.


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-09 Thread Andi Kleen
On Thu, Dec 09, 2010 at 02:34:27AM +, Dave Korn wrote:
> On 08/12/2010 18:40, Andi Kleen wrote:
> 
> > Fat LTO is just too slow. I suspect with that kind of performance
> > penalty most people simply would not use it at all.
> 
>   How slow is "too" slow?  How many people out of a hundred won't use it?  Got
> numbers, or just a gut feeling?

I didn't do a survey if that's what you're asking.

It depends on what tree you use. Before Honza's rebalancing fixes
it was really bad, especially with the tendency to run out of memory
on a 3-4GB system and swap a lot (or generate too many temporary files 
and when you use tmpfs in ram that amounts to the same)

There were also some problems that I had to limit parallelism
below what my machine could do to run out of memory, but that also got 
better in mainline.

Anyways with a recent 4.6 saw slowdowns in compilation of ~2.5x
which is imho too slow. If it was < 1.5x or so things would be
more acceptable I think, especially if the resulting code
is significantly better.

I believe Honza's gcc summit paper has more numbers:

There seem to be also some other bottlenecks currently,
like too much redundant information in the LTO files,
which causes too much IO time.

I haven't done detailed numbers again, but I expect slim LTO
and when the other problems are fixed to be significantly 
faster and have a chance to be below 1.5. There's still
some work to be done in all of these areas.

The memory consumption issue probably also
still needs some work, right now it's still too easy
to start swapping.

With slim LTO there's also some other avenues of speeding
up available, for example it would be pretty simple to not
call the assembler for the LTO generating stages but
just write object files directly. I'm not fully sure
how much this would buy.

-Andi
-- 
a...@linux.intel.com -- Speaking for myself only.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread Dave Korn
On 08/12/2010 18:40, Andi Kleen wrote:

> Fat LTO is just too slow. I suspect with that kind of performance
> penalty most people simply would not use it at all.

  How slow is "too" slow?  How many people out of a hundred won't use it?  Got
numbers, or just a gut feeling?

cheers,
  DaveK


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread Joseph S. Myers
On Wed, 8 Dec 2010, Andrew Pinski wrote:

> On Wed, Dec 8, 2010 at 10:40 AM, Andi Kleen  wrote:
> > The gcc maintainers unfortunately didn't want to integrate the
> > wrapper scripts to make it easy, but they can be always downloaded
> > separately and I assume distributions will eventually ship
> > them anyways.
> 
> No we do just not as scripts.  We want real programs rather shell
> based scripts so it is more portable.

And programs that take proper account of the transformed name under which 
the compiler driver they are running will be installed, rather than 
hardcoding "gcc" (and bad assumptions that the presence of "xgcc" means a 
GCC build directory), to mention another piece of my feedback that was 
ignored in the resubmission.

-- 
Joseph S. Myers
jos...@codesourcery.com


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread Andrew Pinski
On Wed, Dec 8, 2010 at 10:40 AM, Andi Kleen  wrote:
> The gcc maintainers unfortunately didn't want to integrate the
> wrapper scripts to make it easy, but they can be always downloaded
> separately and I assume distributions will eventually ship
> them anyways.

No we do just not as scripts.  We want real programs rather shell
based scripts so it is more portable.

-- Pinski


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread Andi Kleen
> As someone who encountered slim LTO on Unix 17 years ago (on MIPS) I can
> promise you that unless fat LTO is supported, there will never be a

Fat LTO is just too slow. I suspect with that kind of performance
penalty most people simply would not use it at all.

> successful transition.  The amount of work to deal with the make
> environment every time simply made it not worth it.

It's not too hard in my experience. I did it in a few cases for gcc. 

The gcc maintainers unfortunately didn't want to integrate the
wrapper scripts to make it easy, but they can be always downloaded
separately and I assume distributions will eventually ship
them anyways.

-Andi

-- 
a...@linux.intel.com -- Speaking for myself only.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread Xinliang David Li
On Wed, Dec 8, 2010 at 1:19 AM, Andi Kleen  wrote:
>> On 12/07/2010 04:20 PM, Andi Kleen wrote:
>>>
>>> The only problem left is mixing of lto and non lto objects. this right
>>> now is not handled. IMHO still the best way to handle it is to use
>>> slim lto and then simply separate link the "left overs" after deleting
>>> the LTO objects. This can be actually done with objcopy (with some
>>> limitations), doesn't even need linker support.


I agree that FAT lto objects are not necessary to make the everything
work and the integration of LTO with  existing  build environment
'transparent' --- there is compiler out in the world that does just
that -- produces IL only objects (wrapped in elf format); works with
archives with mixed objects; works with ld -r with mixed objects;
builds unix kernel successfully with LTO  ...


David



>>>
>>
>> Quite possibly a better way to deal with that is to provide a mechanism
>> for encapsulating arbitrary binary code objects inside the LTO IR.
>
> Then you would need to teach your assembler and everything
> else that may generate ELF objects to generate this magic object. But why
> not just ELF directly? that is what it is after all.
>
> To be honest I don't really see the point of all this complexity you
> guys are proposing just to save fat LTO. Fat LTO is always a bad idea
> because it's slow and  does lots of redundant work. If LTO is to become
> a more wide spread mode it has to go simply because of the poor
> performance.
>
> With slim LTO passthrough is  very straight-forward: simple pass
> through every section that is not LTO and generate code for the LTO
> sections. No new magic sections needed at all.
>
> -Andi
>
>


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread H. Peter Anvin
On 12/08/2010 01:19 AM, Andi Kleen wrote:
> 
> To be honest I don't really see the point of all this complexity you
> guys are proposing just to save fat LTO. Fat LTO is always a bad idea
> because it's slow and  does lots of redundant work. If LTO is to become
> a more wide spread mode it has to go simply because of the poor
> performance.
> 

As someone who encountered slim LTO on Unix 17 years ago (on MIPS) I can
promise you that unless fat LTO is supported, there will never be a
successful transition.  The amount of work to deal with the make
environment every time simply made it not worth it.

-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.



Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread H. Peter Anvin
On 12/08/2010 01:19 AM, Andi Kleen wrote:
>>
>> Quite possibly a better way to deal with that is to provide a mechanism
>> for encapsulating arbitrary binary code objects inside the LTO IR.
> 
> Then you would need to teach your assembler and everything
> else that may generate ELF objects to generate this magic object. But why
> not just ELF directly? that is what it is after all.
> 

No.  You just need to teach the linker to generate it when you're doing
a ld -r on mixed objects.

-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.



Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread H.J. Lu
On Wed, Dec 8, 2010 at 5:54 AM, H.J. Lu  wrote:
> On Wed, Dec 8, 2010 at 1:19 AM, Andi Kleen  wrote:
>>> On 12/07/2010 04:20 PM, Andi Kleen wrote:

 The only problem left is mixing of lto and non lto objects. this right
 now is not handled. IMHO still the best way to handle it is to use
 slim lto and then simply separate link the "left overs" after deleting
 the LTO objects. This can be actually done with objcopy (with some
 limitations), doesn't even need linker support.

>>>
>>> Quite possibly a better way to deal with that is to provide a mechanism
>>> for encapsulating arbitrary binary code objects inside the LTO IR.
>>
>> Then you would need to teach your assembler and everything
>
> The magic section is generated by linker directly. No changes to
> assembler is required.
>
>> else that may generate ELF objects to generate this magic object. But why
>> not just ELF directly? that is what it is after all.
>
> My proposal isn't specific to ELF.
>
>>
>> To be honest I don't really see the point of all this complexity you
>> guys are proposing just to save fat LTO. Fat LTO is always a bad idea
>> because it's slow and  does lots of redundant work. If LTO is to become
>> a more wide spread mode it has to go simply because of the poor
>> performance.
>>
>> With slim LTO passthrough is  very straight-forward: simple pass
>> through every section that is not LTO and generate code for the LTO
>> sections. No new magic sections needed at all.
>>
>
> My proposal works on both fat and slim LTO objects.  The idea is
> you can use "ld -r" on any combination of inputs and its output
> still works as before "ld -r".
>

Here is the revised proposal.


-- 
H.J.
Link with mixed IR/non-IR objects

* 2 kinds of object files
  o non-IR object file has
* non-IR sections
  o IR object file has
* IR sections
* non-IR sections
* The output of "ld -r" with mixed IR/non-IR objects should work with:
o Compilers/linkers with IR support.
o Compilers/linkers without IR support.
* Add the mixed object file which has
  o IR sections
  o non-IR sections:
* Object codes from IR sections.
* Object codes from non-IR object files.
  o Object-only section:
* With section name ".gnu_object_only" and SHT_GNU_OBJECT_ONLY
(0x6ff8) type on ELF. 
* Contain non-IR object file.
* Input is discarded after link.
* Linker action:
  o Classify each input object file:
* If there is a ".gnu_object_only" section, it is a mixed object file.
* If there is a IR section, it is an IR object file.
* Otherwise, it is a non-IR object file.
  o Relocatable non-IR link:
* Prepare for an object-only output.
* Prepare for a regular output.
* For each mixed object file:
  * Add IR and non-IR sections to the regular output.
  * For object-only section:
  * Extract object only file.
  * Add it to the object-only output.
  * Discard object-only section.
* For each IR object file:
  * Add IR and non-IR sections to the regular output.
* For each non-IR object file:
  * Add non-IR sections to the regular output.
  * Add non-IR sections to the object-only output.
* Final output:
  * If there are IR objects, non-IR objects and the object-only
  output isn't empty:
* Put the object-only output into the object-only section.
* Add the object-only section to the regular output.
* Remove the object-only output.
  o Normal link and relocatable IR link:
* Prepare for output.
* IR link:
  * For each mixed object file:
* Compile and add IR sections to the output.
* Discard non-IR sections.
* Object-only section:
  * Extract object only file.
  * Add it to the output.
  * Discard object-only section.
  * For each IR object file:
* Compile and add IR sections to the output.
* Discard non-IR sections.
  * For each non-IR object file:
* Add non-IR sections to the output.
* Non-IR link:
  * For each mixed object file:
* Add non-IR sections to the output.
* Discard IR sections and object-only section.
  * For each IR object file:
* Add non-IR sections to the output.
* Discard IR sections.
  * For each non-IR object file:
* Add non-IR sections to the output.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread H.J. Lu
On Wed, Dec 8, 2010 at 1:19 AM, Andi Kleen  wrote:
>> On 12/07/2010 04:20 PM, Andi Kleen wrote:
>>>
>>> The only problem left is mixing of lto and non lto objects. this right
>>> now is not handled. IMHO still the best way to handle it is to use
>>> slim lto and then simply separate link the "left overs" after deleting
>>> the LTO objects. This can be actually done with objcopy (with some
>>> limitations), doesn't even need linker support.
>>>
>>
>> Quite possibly a better way to deal with that is to provide a mechanism
>> for encapsulating arbitrary binary code objects inside the LTO IR.
>
> Then you would need to teach your assembler and everything

The magic section is generated by linker directly. No changes to
assembler is required.

> else that may generate ELF objects to generate this magic object. But why
> not just ELF directly? that is what it is after all.

My proposal isn't specific to ELF.

>
> To be honest I don't really see the point of all this complexity you
> guys are proposing just to save fat LTO. Fat LTO is always a bad idea
> because it's slow and  does lots of redundant work. If LTO is to become
> a more wide spread mode it has to go simply because of the poor
> performance.
>
> With slim LTO passthrough is  very straight-forward: simple pass
> through every section that is not LTO and generate code for the LTO
> sections. No new magic sections needed at all.
>

My proposal works on both fat and slim LTO objects.  The idea is
you can use "ld -r" on any combination of inputs and its output
still works as before "ld -r".

-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-08 Thread Andi Kleen
> On 12/07/2010 04:20 PM, Andi Kleen wrote:
>>
>> The only problem left is mixing of lto and non lto objects. this right
>> now is not handled. IMHO still the best way to handle it is to use
>> slim lto and then simply separate link the "left overs" after deleting
>> the LTO objects. This can be actually done with objcopy (with some
>> limitations), doesn't even need linker support.
>>
>
> Quite possibly a better way to deal with that is to provide a mechanism
> for encapsulating arbitrary binary code objects inside the LTO IR.

Then you would need to teach your assembler and everything
else that may generate ELF objects to generate this magic object. But why
not just ELF directly? that is what it is after all.

To be honest I don't really see the point of all this complexity you
guys are proposing just to save fat LTO. Fat LTO is always a bad idea
because it's slow and  does lots of redundant work. If LTO is to become
a more wide spread mode it has to go simply because of the poor
performance.

With slim LTO passthrough is  very straight-forward: simple pass
through every section that is not LTO and generate code for the LTO
sections. No new magic sections needed at all.

-Andi



Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H.J. Lu
On Tue, Dec 7, 2010 at 5:39 PM, Ian Lance Taylor  wrote:
> "H. Peter Anvin"  writes:
>
>> On 12/07/2010 04:20 PM, Andi Kleen wrote:
>>>
>>> The only problem left is mixing of lto and non lto objects. this right
>>> now is not handled. IMHO still the best way to handle it is to use
>>> slim lto and then simply separate link the "left overs" after deleting
>>> the LTO objects. This can be actually done with objcopy (with some
>>> limitations), doesn't even need linker support.
>>>
>>
>> Quite possibly a better way to deal with that is to provide a mechanism
>> for encapsulating arbitrary binary code objects inside the LTO IR.
>
> And when we use a special section name, using an unprintable name is
> needlessly painful and will make it hard to play convenient games with
> objcopy.  Just use a printable name starting with .gnu.
>
> (This assumes that we do need a special section, rather than, say, a
> note.)

Section works for non-ELF system. How about .gnu_object_only section
and with SHT_GNU_OBJECT_ONLY type on ELF?


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread Ian Lance Taylor
"H. Peter Anvin"  writes:

> On 12/07/2010 04:20 PM, Andi Kleen wrote:
>> 
>> The only problem left is mixing of lto and non lto objects. this right
>> now is not handled. IMHO still the best way to handle it is to use
>> slim lto and then simply separate link the "left overs" after deleting
>> the LTO objects. This can be actually done with objcopy (with some
>> limitations), doesn't even need linker support.
>> 
>
> Quite possibly a better way to deal with that is to provide a mechanism
> for encapsulating arbitrary binary code objects inside the LTO IR.

And when we use a special section name, using an unprintable name is
needlessly painful and will make it hard to play convenient games with
objcopy.  Just use a printable name starting with .gnu.

(This assumes that we do need a special section, rather than, say, a
note.)

Ian


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H. Peter Anvin
On 12/07/2010 03:58 PM, Dave Korn wrote:
> On 07/12/2010 23:15, Cary Coutant wrote:
> 
>>>   ○ Object-only section:
>>>   § Section name won't be generated by any tools, something like
>>> ".objectonly\004".
>>>   § Contains non-IR object file.
>>>   § Input is discarded after link.
>>
>> Please -- use a special section type, not a magic name.
> 
>   We're still gonna have to use a magic name on non-ELF platforms.
> 

Yes, but it probably should still be a special section type on ELF.

-hpa


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H.J. Lu
On Tue, Dec 7, 2010 at 4:24 PM, H. Peter Anvin  wrote:
> On 12/07/2010 04:20 PM, Andi Kleen wrote:
>>
>> The only problem left is mixing of lto and non lto objects. this right
>> now is not handled. IMHO still the best way to handle it is to use
>> slim lto and then simply separate link the "left overs" after deleting
>> the LTO objects. This can be actually done with objcopy (with some
>> limitations), doesn't even need linker support.
>>
>
> Quite possibly a better way to deal with that is to provide a mechanism
> for encapsulating arbitrary binary code objects inside the LTO IR.
>

If IR supports it, we can use it instead of magic section name.

-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H.J. Lu
On Tue, Dec 7, 2010 at 4:20 PM, Andi Kleen  wrote:
>>> Here is my proposal.  Any comments?
>>
>> We talked about ld -r a while back during the WHOPR project, and the
>> two ways that the linker could work: (1) combine all the .o files and
>> use the plugin to run LTRANS on the IR files, producing a pure,
>> optimized, object file; and (2) combine the non-IR object files as ld
>> -r normally would, and combine that result somehow with the IR from
>> the other files, for later optimization. If I remember correctly,
>> there was support for both modes of operation. The first mode is
>> easily handled with the current design (untested as far as I know --
>> there are probably bugs, and I'm not sure if we get the symbol
>> visibility correct in those cases).
>
> the first mode is imho useless because you'll never get whole program
> optimizations this way. I tested it some time ago and it worked in a
> limited way
> (there were some problems that gcc crashed if you didn't specify
> -fwhole-program which would be obviously a lie, but those might be fixed
> now)
> But it won't give you the LTO advantages in any case.
>
> I implemented (2) by giving the sections appropiate names
> so they don't get messed up. this works today with gcc mainline, as
> long as all objects in the combined object are LTO.
>
> The only problem left is mixing of lto and non lto objects. this right
> now is not handled. IMHO still the best way to handle it is to use
> slim lto and then simply separate link the "left overs" after deleting
> the LTO objects. This can be actually done with objcopy (with some
> limitations), doesn't even need linker support.
>

My proposal should address mixing of lto and non lto objects.


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H. Peter Anvin
On 12/07/2010 04:20 PM, Andi Kleen wrote:
> 
> The only problem left is mixing of lto and non lto objects. this right
> now is not handled. IMHO still the best way to handle it is to use
> slim lto and then simply separate link the "left overs" after deleting
> the LTO objects. This can be actually done with objcopy (with some
> limitations), doesn't even need linker support.
> 

Quite possibly a better way to deal with that is to provide a mechanism
for encapsulating arbitrary binary code objects inside the LTO IR.

-hpa


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread Andi Kleen
>> Here is my proposal.  Any comments?
>
> We talked about ld -r a while back during the WHOPR project, and the
> two ways that the linker could work: (1) combine all the .o files and
> use the plugin to run LTRANS on the IR files, producing a pure,
> optimized, object file; and (2) combine the non-IR object files as ld
> -r normally would, and combine that result somehow with the IR from
> the other files, for later optimization. If I remember correctly,
> there was support for both modes of operation. The first mode is
> easily handled with the current design (untested as far as I know --
> there are probably bugs, and I'm not sure if we get the symbol
> visibility correct in those cases).

the first mode is imho useless because you'll never get whole program
optimizations this way. I tested it some time ago and it worked in a
limited way
(there were some problems that gcc crashed if you didn't specify
-fwhole-program which would be obviously a lie, but those might be fixed
now)
But it won't give you the LTO advantages in any case.

I implemented (2) by giving the sections appropiate names
so they don't get messed up. this works today with gcc mainline, as
long as all objects in the combined object are LTO.

The only problem left is mixing of lto and non lto objects. this right
now is not handled. IMHO still the best way to handle it is to use
slim lto and then simply separate link the "left overs" after deleting
the LTO objects. This can be actually done with objcopy (with some
limitations), doesn't even need linker support.

-Andi





Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H.J. Lu
On Tue, Dec 7, 2010 at 3:58 PM, H.J. Lu  wrote:
> On Tue, Dec 7, 2010 at 3:57 PM, Andrew Pinski  wrote:
>> On Tue, Dec 7, 2010 at 3:53 PM, H.J. Lu  wrote:
>>> On Tue, Dec 7, 2010 at 3:15 PM, Cary Coutant  wrote:
> Here is my proposal.  Any comments?

 We talked about ld -r a while back during the WHOPR project, and the
 two ways that the linker could work: (1) combine all the .o files and
 use the plugin to run LTRANS on the IR files, producing a pure,
 optimized, object file; and (2) combine the non-IR object files as ld
 -r normally would, and combine that result somehow with the IR from
 the other files, for later optimization. If I remember correctly,
 there was support for both modes of operation. The first mode is
 easily handled with the current design (untested as far as I know --
 there are probably bugs, and I'm not sure if we get the symbol
 visibility correct in those cases).
>>>
>>> We considered it.  The problem is LTO performs the best when
>>> generating the final executable/DSO.  That is we want the full IR in the
>>> output of "ld -r".
>>
>> What happens when ld -r is the final link?  Think loadable linux
>> kernel modules and some other stuff that abuse elf relocatable
>> objects?
>>
>
> "ld -plugin ... -r" will be treated as final link.
>

Here is the updated proposal.


-- 
H.J.
---
• 2 kinds of object files
○ non-IR object file has
§ non-IR sections
○ IR object file has
§ IR sections
§ non-IR sections
• The output of "ld -r" with mixed IR/non-IR objects should work with:
○ Compilers/linkers with IR support.
○ Compilers/linkers without IR support.
• Add the mixed object file which has
○ IR sections
○ non-IR sections:
§ Object codes from IR sections.
§ Object codes from non-IR object files.
○ Object-only section:
§ Section name won't be generated by any tools, something like
".objectonly\004".
§ Contains non-IR object file.
§ Input is discarded after link.
• Linker action:
○ Classify each input object file:
□ If there is a ".objectonly\004" section, it is a mixed object 
file.
□ If there is a IR section, it is an IR object file.
□ Otherwise, it is a non-IR object file.
○ Relocatable non-IR link:
§ Prepare for an object-only output.
§ Prepare for a regular output.
§ For each mixed object file:
□ Add IR and non-IR sections to the regular output.
□ For object-only section:
® Extract object only file.
® Add it to the object-only output.
® Discard object-only section.
§ For each IR object file:
□ Add IR and non-IR sections to the regular output.
§ For each non-IR object file:
□ Add non-IR sections to the regular output.
□ Add non-IR sections to the object-only output.
§ Final output:
□ If there are IR objects, non-IR objects and the 
object-only
output isn't empty:
® Put the object-only output into the 
object-only section.
® Add the object-only section to the regular 
output.
□ Remove the object-only output.
○ Normal link and relocatable IR link:
§ Prepare for output.
§ IR link:
□ For each mixed object file:
® Compile and add IR sections to the output.
® Discard non-IR sections.
® Object-only section:
◊ Extract object only file.
◊ Add it to the output.
◊ Discard object-only section.
□ For each IR object file:
® Compile and add IR sections to the output.
® Discard non-IR sections.
□ For each non-IR object file,
® Add non-IR sections to the output.
§ Non-IR link:
□ For each mixed object file:
® Add non-IR sections to the output.
® Discard IR sections and object-only section.
□ For each IR object file:
® Add non-IR sections to the output.
® Discard IR sections .
□ For each non-IR object file:
® Add non-IR sections to the output.

Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H.J. Lu
On Tue, Dec 7, 2010 at 3:57 PM, Andrew Pinski  wrote:
> On Tue, Dec 7, 2010 at 3:53 PM, H.J. Lu  wrote:
>> On Tue, Dec 7, 2010 at 3:15 PM, Cary Coutant  wrote:
 Here is my proposal.  Any comments?
>>>
>>> We talked about ld -r a while back during the WHOPR project, and the
>>> two ways that the linker could work: (1) combine all the .o files and
>>> use the plugin to run LTRANS on the IR files, producing a pure,
>>> optimized, object file; and (2) combine the non-IR object files as ld
>>> -r normally would, and combine that result somehow with the IR from
>>> the other files, for later optimization. If I remember correctly,
>>> there was support for both modes of operation. The first mode is
>>> easily handled with the current design (untested as far as I know --
>>> there are probably bugs, and I'm not sure if we get the symbol
>>> visibility correct in those cases).
>>
>> We considered it.  The problem is LTO performs the best when
>> generating the final executable/DSO.  That is we want the full IR in the
>> output of "ld -r".
>
> What happens when ld -r is the final link?  Think loadable linux
> kernel modules and some other stuff that abuse elf relocatable
> objects?
>

"ld -plugin ... -r" will be treated as final link.


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread Andrew Pinski
On Tue, Dec 7, 2010 at 3:53 PM, H.J. Lu  wrote:
> On Tue, Dec 7, 2010 at 3:15 PM, Cary Coutant  wrote:
>>> Here is my proposal.  Any comments?
>>
>> We talked about ld -r a while back during the WHOPR project, and the
>> two ways that the linker could work: (1) combine all the .o files and
>> use the plugin to run LTRANS on the IR files, producing a pure,
>> optimized, object file; and (2) combine the non-IR object files as ld
>> -r normally would, and combine that result somehow with the IR from
>> the other files, for later optimization. If I remember correctly,
>> there was support for both modes of operation. The first mode is
>> easily handled with the current design (untested as far as I know --
>> there are probably bugs, and I'm not sure if we get the symbol
>> visibility correct in those cases).
>
> We considered it.  The problem is LTO performs the best when
> generating the final executable/DSO.  That is we want the full IR in the
> output of "ld -r".

What happens when ld -r is the final link?  Think loadable linux
kernel modules and some other stuff that abuse elf relocatable
objects?

Thanks,
Andrew Pinski


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H.J. Lu
On Tue, Dec 7, 2010 at 3:15 PM, Cary Coutant  wrote:
>> Here is my proposal.  Any comments?
>
> We talked about ld -r a while back during the WHOPR project, and the
> two ways that the linker could work: (1) combine all the .o files and
> use the plugin to run LTRANS on the IR files, producing a pure,
> optimized, object file; and (2) combine the non-IR object files as ld
> -r normally would, and combine that result somehow with the IR from
> the other files, for later optimization. If I remember correctly,
> there was support for both modes of operation. The first mode is
> easily handled with the current design (untested as far as I know --
> there are probably bugs, and I'm not sure if we get the symbol
> visibility correct in those cases).

We considered it.  The problem is LTO performs the best when
generating the final executable/DSO.  That is we want the full IR in the
output of "ld -r".

> The second mode corresponds with your proposal here. It's complicated
> by the fact that it's difficult to tell, once the objects are
> combined, which compiled code came without corresponding IR. For this,
> I've got a suggestion that seems a bit simpler than your
> ".objectonly\004" section, based on an idea for something completely
> unrelated[1] that I've been pondering over for a while. Instead of
> embedding the non-IR objects into the mixed object file, let's instead
> produce an archive file with several members: one that contains the
> result of running ld -r on the non-IR objects in the link, and one
> member for each of the IR files (alternatively, exactly one member
> that contains the result of running ld -r on all of the IR objects).
> In order to make the archive such that a subsequent link loads all of
> the members unconditionally, I propose to add a special symbol
> ".FORCE" into the archive symbol table for each member; when the
> linker sees that symbol in the archive symbol table, it will load the
> corresponding member unconditionally.
>
>>       ○ Object-only section:
>>               § Section name won't be generated by any tools, something like
>>".objectonly\004".
>>               § Contains non-IR object file.
>>               § Input is discarded after link.
>
> Please -- use a special section type, not a magic name.
>

As Dave pointed out, we need the magic section name for non-ELF
platform.  One main feature of my proposal is transparent:

# ld -r -o foo.o foo1.o foo2.o foo3.o ...
# ld -r o -bar.o bar1.o bar2.o bar3.o ...
...
# ld  -r o new.o foo.o bar.o ...

where foo.o, bar.o ... are mixed object files.  That is more user-friendly.
Projects like Linux kernel can take advantage of LTO with simple changes
to their Makefiles.


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread Dave Korn
On 07/12/2010 23:15, Cary Coutant wrote:

>>   ○ Object-only section:
>>   § Section name won't be generated by any tools, something like
>> ".objectonly\004".
>>   § Contains non-IR object file.
>>   § Input is discarded after link.
> 
> Please -- use a special section type, not a magic name.

  We're still gonna have to use a magic name on non-ELF platforms.

cheers,
  DaveK


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread Cary Coutant
> Here is my proposal.  Any comments?

We talked about ld -r a while back during the WHOPR project, and the
two ways that the linker could work: (1) combine all the .o files and
use the plugin to run LTRANS on the IR files, producing a pure,
optimized, object file; and (2) combine the non-IR object files as ld
-r normally would, and combine that result somehow with the IR from
the other files, for later optimization. If I remember correctly,
there was support for both modes of operation. The first mode is
easily handled with the current design (untested as far as I know --
there are probably bugs, and I'm not sure if we get the symbol
visibility correct in those cases).

The second mode corresponds with your proposal here. It's complicated
by the fact that it's difficult to tell, once the objects are
combined, which compiled code came without corresponding IR. For this,
I've got a suggestion that seems a bit simpler than your
".objectonly\004" section, based on an idea for something completely
unrelated[1] that I've been pondering over for a while. Instead of
embedding the non-IR objects into the mixed object file, let's instead
produce an archive file with several members: one that contains the
result of running ld -r on the non-IR objects in the link, and one
member for each of the IR files (alternatively, exactly one member
that contains the result of running ld -r on all of the IR objects).
In order to make the archive such that a subsequent link loads all of
the members unconditionally, I propose to add a special symbol
".FORCE" into the archive symbol table for each member; when the
linker sees that symbol in the archive symbol table, it will load the
corresponding member unconditionally.

>   ○ Object-only section:
>   § Section name won't be generated by any tools, something like
>".objectonly\004".
>   § Contains non-IR object file.
>   § Input is discarded after link.

Please -- use a special section type, not a magic name.

-cary


[1] My unrelated idea is about "__attribute__ (( used ))" -- when a
symbol is marked as used, it should not only suppress unused warnings
in the compiler, but it should also force the resulting object module
to be linked from an archive library. I've been thinking about a
proposal to mark any object file that contains a used symbol, have ar
recognize that mark and add the ".FORCE" symbol to the archive symbol
table for that object, then have the linker recognize the ".FORCE"
symbol and load the member unconditionally.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-07 Thread H.J. Lu
On Mon, Dec 6, 2010 at 4:05 PM, H.J. Lu  wrote:
>>
>> Without slim lto you never know if a duplicate symbol is a mistake
>> of the programmer or just the "fat lto" copy. Also ELF semantics
>> like weak are hard if you have multiple copies.
>>
>
> It isn't easy, but doable.
>

Here is my proposal.  Any comments?

Thanks.


-- 
H.J.
---
• 2 kinds of object files
○ non-IR object file has
§ non-IR sections
○ IR object file has
§ IR sections
§ non-IR sections
• The output of "ld -r" with mixed IR/non-IR objects should work with:
○ Compilers/linkers with IR support.
○ Compilers/linkers without IR support.
• Add the mixed object file has
○ IR sections
○ non-IR sections:
§ Object codes from IR sections.
§ Object codes from non-IR object files.
○ Object-only section:
§ Section name won't be generated by any tools, something like
".objectonly\004".
§ Contains non-IR object file.
§ Input is discarded after link.
• Linker action:
○ Classify each input object file:
□ If there is a ".objectonly\004" section, it is a mixed object 
file.
□ If there is a IR section, it is an IR object file.
□ Otherwise, it is a non-IR object file.
○ Relocatable link:
§ Prepare for an object-only output.
§ Prepare for a regular output.
§ For each mixed object file,
□ Add IR and non-IR sections to the regular output.
□ For object-only section:
® Extract object only file.
® Add it to the object-only output.
® Discard object-only section.
§ For each IR object file,
□ Add IR and non-IR sections to the regular output.
§ For each non-IR object file,
□ Add non-IR sections to the regular output.
□ Add non-IR sections to the object-only output.
§ Final output:
□ If there are IR objects, non-IR objects and the 
object-only
output isn't empty:
® Put the object-only output into the 
object-only section.
® Add the object-only section to the regular 
output.
□ Remove the object-only output.
○ Normal link
§ Prepare for output.
§ For each mixed object file,
□ Compile and add IR sections to the output.
□ For object-only section:
® Extract object only file.
® Add it to the output.
® Discard object-only section.
§ For each IR object file,
□ Compile and add IR sections to the output.
§ For each non-IR object file,
□ Add non-IR sections to the output.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-06 Thread H.J. Lu
On Mon, Dec 6, 2010 at 3:09 PM, Andi Kleen  wrote:
>> On Mon, Dec 6, 2010 at 2:43 PM, Andi Kleen  wrote:
 Hi,

 "ld -r" doesn't work with mixed IR/non-IR objects:

 http://www.sourceware.org/bugzilla/show_bug.cgi?id=12291
>>>
>>> There are various bugs for it in gcc bugzilla too.
>>>
 Some compilers support it. Should it be supported?
>>>
>>> Yes. I've been working on it (slim lto was the first part needed for it,
>>> without slim lto it's imho hopeless)
>>
>> Slim lto is a workaround, not a real solution.
>
> Actually I consider fat lto as a workaround for toolchain deficiencies.
> Generating everything twice and throwing one copy away is always
> inefficient. IMHO slim lto is the natural way to do LTO.

The idea is the same object file can be used with compilers
which don't support LTO.

> The only problem is that you need to fix every piece in the toolchain
> to not mess up LTO.
>
> That's why I did the gcc-ar,nm etc. wrappers. There were other problems
> too, like gcc's build system itself not handling LTO correctly (it normally
> only works because fat saves the day)
>
>> I think this support should
>> be
>> implemented in ld with help from GCC.
>
> Without slim lto you never know if a duplicate symbol is a mistake
> of the programmer or just the "fat lto" copy. Also ELF semantics
> like weak are hard if you have multiple copies.
>

It isn't easy, but doable.


-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-06 Thread Andi Kleen
> On Mon, Dec 6, 2010 at 2:43 PM, Andi Kleen  wrote:
>>> Hi,
>>>
>>> "ld -r" doesn't work with mixed IR/non-IR objects:
>>>
>>> http://www.sourceware.org/bugzilla/show_bug.cgi?id=12291
>>
>> There are various bugs for it in gcc bugzilla too.
>>
>>> Some compilers support it. Should it be supported?
>>
>> Yes. I've been working on it (slim lto was the first part needed for it,
>> without slim lto it's imho hopeless)
>
> Slim lto is a workaround, not a real solution.

Actually I consider fat lto as a workaround for toolchain deficiencies.
Generating everything twice and throwing one copy away is always
inefficient. IMHO slim lto is the natural way to do LTO.

The only problem is that you need to fix every piece in the toolchain
to not mess up LTO.

That's why I did the gcc-ar,nm etc. wrappers. There were other problems
too, like gcc's build system itself not handling LTO correctly (it normally
only works because fat saves the day)

> I think this support should
> be
> implemented in ld with help from GCC.

Without slim lto you never know if a duplicate symbol is a mistake
of the programmer or just the "fat lto" copy. Also ELF semantics
like weak are hard if you have multiple copies.

-Andi



Re: "ld -r" on mixed IR/non-IR objects (

2010-12-06 Thread H.J. Lu
On Mon, Dec 6, 2010 at 2:43 PM, Andi Kleen  wrote:
>> Hi,
>>
>> "ld -r" doesn't work with mixed IR/non-IR objects:
>>
>> http://www.sourceware.org/bugzilla/show_bug.cgi?id=12291
>
> There are various bugs for it in gcc bugzilla too.
>
>> Some compilers support it. Should it be supported?
>
> Yes. I've been working on it (slim lto was the first part needed for it,
> without slim lto it's imho hopeless)

Slim lto is a workaround, not a real solution. I think this support should be
implemented in ld with help from GCC.



-- 
H.J.


Re: "ld -r" on mixed IR/non-IR objects (

2010-12-06 Thread Andi Kleen
> Hi,
>
> "ld -r" doesn't work with mixed IR/non-IR objects:
>
> http://www.sourceware.org/bugzilla/show_bug.cgi?id=12291

There are various bugs for it in gcc bugzilla too.

> Some compilers support it. Should it be supported?

Yes. I've been working on it (slim lto was the first part needed for it,
without slim lto it's imho hopeless)

But it's going slow for various reasons. Right now I'm not optimistic
4.6 will support it, which implies it won't be able to build a linux kernel
with LTO.

-Andi



Re: "ld -r" on mixed IR/non-IR objects (

2010-12-06 Thread H. Peter Anvin
On 12/06/2010 02:30 PM, H.J. Lu wrote:
> Hi,
> 
> "ld -r" doesn't work with mixed IR/non-IR objects:
> 
> http://www.sourceware.org/bugzilla/show_bug.cgi?id=12291
> 
> Some compilers support it. Should it be supported?
> 

As we discussed in person, I think it would be user friendly to support
it, otherwise you'll break any build which uses ld -r and includes
assembly objects.

-hpa


"ld -r" on mixed IR/non-IR objects (

2010-12-06 Thread H.J. Lu
Hi,

"ld -r" doesn't work with mixed IR/non-IR objects:

http://www.sourceware.org/bugzilla/show_bug.cgi?id=12291

Some compilers support it. Should it be supported?



-- 
H.J.