Re: [OE-core] RFC: Secondary Toolchain
I'm curious to know if anyone (I certainly wouldn't be able to!) can
take a guess whether this would play nicely with external
toolchains?
In other words, if some recipe is already PROVIDES'ing
virtual/${TARGET_PREFIX}gcc etc would the correct toolchain be used
for the special packages needing the secondary toolchain?
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] RFC: Secondary Toolchain
On 10/4/12 1:15 PM, Trevor Woerner wrote:
I'm curious to know if anyone (I certainly wouldn't be able to!) can
take a guess whether this would play nicely with external
toolchains?
In other words, if some recipe is already PROVIDES'ing
virtual/${TARGET_PREFIX}gcc etc would the correct toolchain be used
for the special packages needing the secondary toolchain?
My expectations is that any existing dependency set will be managed and
maintained by the existing primary toolchain, unless a given recipe has a
specific dependency on a secondary toolchain item.
I.e. the example above will -always- be the primary toolchain from a dependency
resolution standpoint..
If the recipe adds virtual/icc, then something, such as icc, needs to exist to
provide that.
Does this seem like a reasonable behavior and expectation? (The thing to
remember is this secondary toolchain is just that.. an alternative to the
primary for specific users and NOT general purpose)
--Mark
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] RFC: Secondary Toolchain
On Thu, Oct 4, 2012 at 1:02 PM, Mark Hatle mark.ha...@windriver.com wrote:
We have an issue where we'd like to have an alternative toolchain
(assembler, linker, compiler) available for our customers to selectively
use. However, before we just go off and implement something, I'd like at
least some basic consensus on what the best practice is for doing this work.
Below is my attempt at an early proposal.
Background
Many companies have commercial / highly optimized toolchains for specific
purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc.. For
(potentially) better performance with some applications a mechanism to both
provide the hooks for the alternative toolchain as well as a way to active
it per-package is desired.
This work assumes that the third party toolchain is generally compatible
with the idea of sysroots, linking to the libc provided by OE, and generally
compatible with GNU conventions.
However, as part of the third party toolchain, it may not be GNU compatible.
This means many Open Source applications simply may not work with this
toolchain. That means that we need to have a way for a toolchain to
blacklist (or whitelist) specific recipes. This way only supported
components can be built by the user, avoiding numerous complaints.
Currently OE has a method to generate an SDK based on the GNU toolchain. We
would like to be able to also export the external toolchain along with the
SDK, effectively providing both the GNU toolchain and the third party
toolchain using the common sysroot.
We need a way to active the third party toolchain on a per-package basis.
This activation will need to use the existing sysroot, but be able to pass
different C, C++, LD, AS and other flags as specified by the third party
toolchain.
Finally third party toolchains should be implemented as layers that can
easily plug into OE.
Implementation
-
Add an OVERRIDE to specify the alternative toolchain. Can this be done
within the layer by doing something like:
OVERRIDE_append = :toolchain-${TOOLCHAIN}
TOOLCHAIN = gnu (or icc)
To activate the toolchain you would use things like:
TOOLCHAIN_pn-bash = 'icc'
To define the correct behavior for the toolchain, the following would need
to be defined (with _toolchain-toolchain):
TARGET_CPPFLAGS
TARGET_CFLAGS
TARGET_CXXFLAGS
TARGET_LDFLAGS
CC
CXX
F77
CPP
LD
CCLD
AR
AS
RANLIB
STRIP
OBJCOPY
OBJDUMP
NM
FULL_OPTIMIZATIONS
DEBUG_OPTIMIZATION
Is anyone aware of any other items that may require additional items? Will
the above actually work? Using the override of the TOOLCHAIN_… will that
actually change the override values or do we get stuck?
This seems orthogonal to actually implementing the recipe which would
procide 'icc'?
-M
Comments/suggestions appreciated!
--Mark
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] RFC: Secondary Toolchain
On 10/4/12 2:03 PM, McClintock Matthew-B29882 wrote:
On Thu, Oct 4, 2012 at 1:02 PM, Mark Hatle mark.ha...@windriver.com wrote:
We have an issue where we'd like to have an alternative toolchain
(assembler, linker, compiler) available for our customers to selectively
use. However, before we just go off and implement something, I'd like at
least some basic consensus on what the best practice is for doing this work.
Below is my attempt at an early proposal.
Background
Many companies have commercial / highly optimized toolchains for specific
purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc.. For
(potentially) better performance with some applications a mechanism to both
provide the hooks for the alternative toolchain as well as a way to active
it per-package is desired.
This work assumes that the third party toolchain is generally compatible
with the idea of sysroots, linking to the libc provided by OE, and generally
compatible with GNU conventions.
However, as part of the third party toolchain, it may not be GNU compatible.
This means many Open Source applications simply may not work with this
toolchain. That means that we need to have a way for a toolchain to
blacklist (or whitelist) specific recipes. This way only supported
components can be built by the user, avoiding numerous complaints.
Currently OE has a method to generate an SDK based on the GNU toolchain. We
would like to be able to also export the external toolchain along with the
SDK, effectively providing both the GNU toolchain and the third party
toolchain using the common sysroot.
We need a way to active the third party toolchain on a per-package basis.
This activation will need to use the existing sysroot, but be able to pass
different C, C++, LD, AS and other flags as specified by the third party
toolchain.
Finally third party toolchains should be implemented as layers that can
easily plug into OE.
Implementation
-
Add an OVERRIDE to specify the alternative toolchain. Can this be done
within the layer by doing something like:
OVERRIDE_append = :toolchain-${TOOLCHAIN}
TOOLCHAIN = gnu (or icc)
To activate the toolchain you would use things like:
TOOLCHAIN_pn-bash = 'icc'
To define the correct behavior for the toolchain, the following would need
to be defined (with _toolchain-toolchain):
TARGET_CPPFLAGS
TARGET_CFLAGS
TARGET_CXXFLAGS
TARGET_LDFLAGS
CC
CXX
F77
CPP
LD
CCLD
AR
AS
RANLIB
STRIP
OBJCOPY
OBJDUMP
NM
FULL_OPTIMIZATIONS
DEBUG_OPTIMIZATION
Is anyone aware of any other items that may require additional items? Will
the above actually work? Using the override of the TOOLCHAIN_… will that
actually change the override values or do we get stuck?
This seems orthogonal to actually implementing the recipe which would
procide 'icc'?
That is correct. I'm trying to establish a best practice for the layer
configuration, as well as general distribution/recipe configuration.
What I really don't want to see is 5 companies implementing similar
functionality and doing it in a completely incompatible way. If the variables
and override mechanism above is reasonable, then it gives people a roadmap to
get started.
--Mark
-M
Comments/suggestions appreciated!
--Mark
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] RFC: Secondary Toolchain
On Thu, Oct 4, 2012 at 11:02 AM, Mark Hatle mark.ha...@windriver.com wrote:
We have an issue where we'd like to have an alternative toolchain
(assembler, linker, compiler) available for our customers to selectively
use. However, before we just go off and implement something, I'd like at
least some basic consensus on what the best practice is for doing this work.
Below is my attempt at an early proposal.
Background
Many companies have commercial / highly optimized toolchains for specific
purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc.. For
(potentially) better performance with some applications a mechanism to both
provide the hooks for the alternative toolchain as well as a way to active
it per-package is desired.
This work assumes that the third party toolchain is generally compatible
with the idea of sysroots, linking to the libc provided by OE, and generally
compatible with GNU conventions.
However, as part of the third party toolchain, it may not be GNU compatible.
This means many Open Source applications simply may not work with this
toolchain. That means that we need to have a way for a toolchain to
blacklist (or whitelist) specific recipes. This way only supported
components can be built by the user, avoiding numerous complaints.
Currently OE has a method to generate an SDK based on the GNU toolchain. We
would like to be able to also export the external toolchain along with the
SDK, effectively providing both the GNU toolchain and the third party
toolchain using the common sysroot.
We need a way to active the third party toolchain on a per-package basis.
Per package bases is a bit iffy from my POV if you consider the compiler runtime
and C library runtime and so forth. Is it intended to just compile and
be able to
cope with the default runtimes on images or do you also intend to figure out
ways for multiple runtimes to live together.
This activation will need to use the existing sysroot, but be able to pass
different C, C++, LD, AS and other flags as specified by the third party
toolchain.
that sort of answers the above. So we are assuming that toolchain that
can interoperate
will be using GNU runtime
There are cases where you might want something like ELLCC which avoids
complete GPL runtime e.g.
I think it could be divided into runtime + tools where runtime lives
in a different layer but
essentially offering to let people use builtin C library and runtime
may not be a bad choice
Finally third party toolchains should be implemented as layers that can
easily plug into OE.
agreed. I think we need to document the interfaces in OE-Core for this
too so that
people can rely on the interface. Otherwise it will be constant pain
for keeping those
layers working together.
Implementation
-
Add an OVERRIDE to specify the alternative toolchain. Can this be done
within the layer by doing something like:
OVERRIDE_append = :toolchain-${TOOLCHAIN}
TOOLCHAIN = gnu (or icc)
To activate the toolchain you would use things like:
TOOLCHAIN_pn-bash = 'icc'
To define the correct behavior for the toolchain, the following would need
to be defined (with _toolchain-toolchain):
TARGET_CPPFLAGS
TARGET_CFLAGS
TARGET_CXXFLAGS
TARGET_LDFLAGS
CC
CXX
F77
CPP
LD
CCLD
AR
AS
RANLIB
STRIP
OBJCOPY
OBJDUMP
NM
FULL_OPTIMIZATIONS
DEBUG_OPTIMIZATION
Is anyone aware of any other items that may require additional items? Will
the above actually work? Using the override of the TOOLCHAIN_… will that
actually change the override values or do we get stuck?
Comments/suggestions appreciated!
--Mark
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] RFC: Secondary Toolchain
On 10/4/12 3:36 PM, Khem Raj wrote:
On Thu, Oct 4, 2012 at 11:02 AM, Mark Hatle mark.ha...@windriver.com wrote:
We have an issue where we'd like to have an alternative toolchain
(assembler, linker, compiler) available for our customers to selectively
use. However, before we just go off and implement something, I'd like at
least some basic consensus on what the best practice is for doing this work.
Below is my attempt at an early proposal.
Background
Many companies have commercial / highly optimized toolchains for specific
purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc.. For
(potentially) better performance with some applications a mechanism to both
provide the hooks for the alternative toolchain as well as a way to active
it per-package is desired.
This work assumes that the third party toolchain is generally compatible
with the idea of sysroots, linking to the libc provided by OE, and generally
compatible with GNU conventions.
However, as part of the third party toolchain, it may not be GNU compatible.
This means many Open Source applications simply may not work with this
toolchain. That means that we need to have a way for a toolchain to
blacklist (or whitelist) specific recipes. This way only supported
components can be built by the user, avoiding numerous complaints.
Currently OE has a method to generate an SDK based on the GNU toolchain. We
would like to be able to also export the external toolchain along with the
SDK, effectively providing both the GNU toolchain and the third party
toolchain using the common sysroot.
We need a way to active the third party toolchain on a per-package basis.
Per package bases is a bit iffy from my POV if you consider the compiler runtime
and C library runtime and so forth. Is it intended to just compile and
be able to
cope with the default runtimes on images or do you also intend to figure out
ways for multiple runtimes to live together.
This is only one runtime. You have multiple compilers all capable of producing
software compatible with the same ABI. The default (oe) compiler is used,
unless otherwise configured. The alternative(s) are used for optimization of
various items. I mentioned ICC, because it's one that I know today people are
using and it is capable of replacing gcc in many applications.
The issue is that I want to enable someone to use an alternative compiler, but I
don't want to do anything beyond enabling it if it doesn't work.
This activation will need to use the existing sysroot, but be able to pass
different C, C++, LD, AS and other flags as specified by the third party
toolchain.
that sort of answers the above. So we are assuming that toolchain that
can interoperate
will be using GNU runtime
There are cases where you might want something like ELLCC which avoids
complete GPL runtime e.g.
I think it could be divided into runtime + tools where runtime lives
in a different layer but
essentially offering to let people use builtin C library and runtime
may not be a bad choice
When the runtime changes, I consider that to be a different abi and outside the
scope of this. Thats where the machine/multilib/etc configuration items come
into play. (I.e. some architectures support an alternative compiler for bare
metal applications -- in those cases the canonical arch is no longer
-linux[-gnu]... so they would simply be treated as a different architecture.)
Finally third party toolchains should be implemented as layers that can
easily plug into OE.
agreed. I think we need to document the interfaces in OE-Core for this
too so that
people can rely on the interface. Otherwise it will be constant pain
for keeping those
layers working together.
Exactly.. I don't want people coming to me saying I want XYZ compiler, but it's
implemented for my semi-conduction specific Linux and won't work with your OSV
based Linux. That doesn't help anyone (unless they are intentionally trying to
be incompatible!) So if we can provide a functional best practice and example
of how to do this, then when people want to provide (and use) a secondary
toolchain -- it's available to them.
Implementation
-
Add an OVERRIDE to specify the alternative toolchain. Can this be done
within the layer by doing something like:
OVERRIDE_append = :toolchain-${TOOLCHAIN}
TOOLCHAIN = gnu (or icc)
To activate the toolchain you would use things like:
TOOLCHAIN_pn-bash = 'icc'
To define the correct behavior for the toolchain, the following would need
to be defined (with _toolchain-toolchain):
TARGET_CPPFLAGS
TARGET_CFLAGS
TARGET_CXXFLAGS
TARGET_LDFLAGS
CC
CXX
F77
CPP
LD
CCLD
AR
AS
RANLIB
STRIP
OBJCOPY
OBJDUMP
NM
FULL_OPTIMIZATIONS
DEBUG_OPTIMIZATION
Is anyone aware of any other items that may require additional items? Will
the above actually work? Using the override of the TOOLCHAIN_… will that
actually change the override values or do we get stuck?
Comments/suggestions
Re: [OE-core] RFC: Secondary Toolchain
On Thu, Oct 4, 2012 at 2:00 PM, Mark Hatle mark.ha...@windriver.com wrote:
On 10/4/12 3:36 PM, Khem Raj wrote:
On Thu, Oct 4, 2012 at 11:02 AM, Mark Hatle mark.ha...@windriver.com
wrote:
We have an issue where we'd like to have an alternative toolchain
(assembler, linker, compiler) available for our customers to selectively
use. However, before we just go off and implement something, I'd like at
least some basic consensus on what the best practice is for doing this
work.
Below is my attempt at an early proposal.
Background
Many companies have commercial / highly optimized toolchains for specific
purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc.. For
(potentially) better performance with some applications a mechanism to
both
provide the hooks for the alternative toolchain as well as a way to
active
it per-package is desired.
This work assumes that the third party toolchain is generally compatible
with the idea of sysroots, linking to the libc provided by OE, and
generally
compatible with GNU conventions.
However, as part of the third party toolchain, it may not be GNU
compatible.
This means many Open Source applications simply may not work with this
toolchain. That means that we need to have a way for a toolchain to
blacklist (or whitelist) specific recipes. This way only supported
components can be built by the user, avoiding numerous complaints.
Currently OE has a method to generate an SDK based on the GNU toolchain.
We
would like to be able to also export the external toolchain along with
the
SDK, effectively providing both the GNU toolchain and the third party
toolchain using the common sysroot.
We need a way to active the third party toolchain on a per-package basis.
Per package bases is a bit iffy from my POV if you consider the compiler
runtime
and C library runtime and so forth. Is it intended to just compile and
be able to
cope with the default runtimes on images or do you also intend to figure
out
ways for multiple runtimes to live together.
This is only one runtime. You have multiple compilers all capable of
producing software compatible with the same ABI. The default (oe) compiler
is used, unless otherwise configured. The alternative(s) are used for
optimization of various items. I mentioned ICC, because it's one that I
know today people are using and it is capable of replacing gcc in many
applications.
Thanks that clarifies the scope. I thought you were going for a
general solution where you
could insert any toolchain.
The issue is that I want to enable someone to use an alternative compiler,
but I don't want to do anything beyond enabling it if it doesn't work.
This activation will need to use the existing sysroot, but be able to
pass
different C, C++, LD, AS and other flags as specified by the third party
toolchain.
that sort of answers the above. So we are assuming that toolchain that
can interoperate
will be using GNU runtime
There are cases where you might want something like ELLCC which avoids
complete GPL runtime e.g.
I think it could be divided into runtime + tools where runtime lives
in a different layer but
essentially offering to let people use builtin C library and runtime
may not be a bad choice
When the runtime changes, I consider that to be a different abi and outside
the scope of this. Thats where the machine/multilib/etc configuration items
come into play. (I.e. some architectures support an alternative compiler
for bare metal applications -- in those cases the canonical arch is no
longer -linux[-gnu]... so they would simply be treated as a different
architecture.)
Finally third party toolchains should be implemented as layers that can
easily plug into OE.
agreed. I think we need to document the interfaces in OE-Core for this
too so that
people can rely on the interface. Otherwise it will be constant pain
for keeping those
layers working together.
Exactly.. I don't want people coming to me saying I want XYZ compiler, but
it's implemented for my semi-conduction specific Linux and won't work with
your OSV based Linux. That doesn't help anyone (unless they are
intentionally trying to be incompatible!) So if we can provide a functional
best practice and example of how to do this, then when people want to
provide (and use) a secondary toolchain -- it's available to them.
Implementation
-
Add an OVERRIDE to specify the alternative toolchain. Can this be done
within the layer by doing something like:
OVERRIDE_append = :toolchain-${TOOLCHAIN}
TOOLCHAIN = gnu (or icc)
To activate the toolchain you would use things like:
TOOLCHAIN_pn-bash = 'icc'
To define the correct behavior for the toolchain, the following would
need
to be defined (with _toolchain-toolchain):
TARGET_CPPFLAGS
TARGET_CFLAGS
TARGET_CXXFLAGS
TARGET_LDFLAGS
CC
CXX
F77
CPP
LD
CCLD
AR
AS
RANLIB
STRIP
OBJCOPY
Re: [OE-core] RFC: Secondary Toolchain
On Thu, 2012-10-04 at 16:00 -0500, Mark Hatle wrote:
This is only one runtime. You have multiple compilers all capable of
producing
software compatible with the same ABI. The default (oe) compiler is used,
unless otherwise configured. The alternative(s) are used for optimization of
various items. I mentioned ICC, because it's one that I know today people
are
using and it is capable of replacing gcc in many applications.
If that's the case then it doesn't seem like it ought to be necessary to
override most of the variables you mentioned. Tools like ar, ranlib and
strip are generic to a binary format and should work fine with the
output from any compiler. Also, if you're linking with GNU libc then
the chances are that you need to use one of the GNU linkers (be that
ld.bfd or gold) since I suspect most third party link editors will not
work very well. And, if you have assembly source on hand, it's probably
going to be written in GAS dialect so selecting an alternative assembler
as ${AS} is unlikely to gain much.
So, it kind of seems like the only variables you're likely to need to
change are CC, CXX and the corresponding FLAGS. I guess you might also
want to override F77 in theory though I'm not sure there are many people
using Fortran in an embedded environment.
Your proposed toolchain-icc override sounds like a reasonable enough
way to accomplish that, anyway. The only problem it doesn't seem to
solve is the use-case of everything that supports icc, I want to build
with icc; everything that doesn't support icc, I want to build with
gcc. But I'm not sure this really matters much.
p.
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] RFC: Secondary Toolchain
On 10/4/12 4:16 PM, Phil Blundell wrote:
On Thu, 2012-10-04 at 16:00 -0500, Mark Hatle wrote:
This is only one runtime. You have multiple compilers all capable of producing
software compatible with the same ABI. The default (oe) compiler is used,
unless otherwise configured. The alternative(s) are used for optimization of
various items. I mentioned ICC, because it's one that I know today people are
using and it is capable of replacing gcc in many applications.
If that's the case then it doesn't seem like it ought to be necessary to
override most of the variables you mentioned. Tools like ar, ranlib and
strip are generic to a binary format and should work fine with the
output from any compiler. Also, if you're linking with GNU libc then
the chances are that you need to use one of the GNU linkers (be that
ld.bfd or gold) since I suspect most third party link editors will not
work very well. And, if you have assembly source on hand, it's probably
going to be written in GAS dialect so selecting an alternative assembler
as ${AS} is unlikely to gain much.
I realize not everything will be overridden, and I even expect in many cases
only the CC and CXX will be defined.
I'm aware of at least one third party compiler that uses a different dialect
then GAS. And in that case the compiler and assembly would be overwritten..
(mind you anything that uses gas syntax would of course be incompatible and blow
up, which begs the question -- do you really want to use AS then?)
But I figure if we cover all of the cases then we should be good no matter what.
So, it kind of seems like the only variables you're likely to need to
change are CC, CXX and the corresponding FLAGS. I guess you might also
want to override F77 in theory though I'm not sure there are many people
using Fortran in an embedded environment.
I'm aware of one company that -was- using Fortran, I don't know if they still
are. (And they were not using the GNU F77)
Your proposed toolchain-icc override sounds like a reasonable enough
way to accomplish that, anyway. The only problem it doesn't seem to
solve is the use-case of everything that supports icc, I want to build
with icc; everything that doesn't support icc, I want to build with
gcc. But I'm not sure this really matters much.
There needs to be a way to set the TOOLCHAIN 'type' on a blacklist or whitelist
basis. I.e. in a whitelist, you simply set it to 'gnu' (or whatever), which is
more or less what I'd expect the default behavior to be. Then for every recipe
you are aware of that works -- you'd overriver the TOOLCHAIN-pn_package w/ 'alt'.
Alternatively you could do it as a blacklist.. set TOOLCHAIN = 'alt' and then
blacklist items by setting the items to 'gnu' (or whatever).
Both have advantages.. personally I'd suggest the former rather then later
approach, it's easier to support and then you can focus porting efforts on the
packages that you believe you need the most optimizations in.
--Mark
p.
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
