Re: [GSoC]Bypass assembler when generating LTO object files
On Tue, Apr 12, 2022 at 2:53 PM Iain Sandoe wrote: > > > > > On 12 Apr 2022, at 13:31, Martin Liška wrote: > > > > On 4/12/22 11:58, Richard Biener wrote: > >> On Tue, Apr 12, 2022 at 11:20 AM Jan Hubicka via Gcc > >> wrote: > >>> > >>> Hi, > > > > On 08-Apr-2022, at 6:32 PM, Jan Hubicka wrote: > > > > Ankur, > >> I was browsing the list of submitted GSoC projects this year and the > >> project regarding bypassing assembler when generating LTO object files > >> caught my eye. > > I apologize for late reply. I would be very happy to mentor this > > project. > > Thanks for the reply, but unfortunately, due to some reasons, I would not > be able to take part in GSoC this year. > But the project seems interesting and would be amazing opportunity to > learn a lot more things for me, so would it be okay if I try to give it a > go outside GSoC if no-one else picks it as their GSoC project this year ? > >>> > >>> I would be still very happy to help with that! However it would be also > >>> pity to not take part of GSoC, so if there is something I can help with > >>> on that let me know. > > >> > >> I already have a gcc built from source (sync-ed with trunk/master) and > >> launched the test-suite on it. > >> > >> I am currently in process of understanding the primilary patch > >> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and > >> experimenting with it. > >> > >> are there any other things I should be aware of (useful Doc/blog or a > >> bug tracking the project) before proceeding further ? > > > > I think it is pretty much all that exists. Basically we will need to > > implement everything that is necessary to stream out valid object file > > directly from GCC rather than going through gas. The experimental > > prototype sort of worked but it was lacking few things. > > When I try to apply that patch on my local branch ( branched from trunk > ), > it seem to be incompatible with the current working tree. Is there a > specific branch that I have to apply it to ? or is it due to the recent > file rename patch ( changing extensions from .c to .cc ) ? > > ``` > $ git apply --check bypass_asm_patch > > error: patch failed: Makefile.in:1300 > error: Makefile.in: patch does not apply > error: common.opt: No such file or directory > error: langhooks.c: No such file or directory > error: lto/Make-lang.in: No such file or directory > error: lto/lto-object.c: No such file or directory > error: lto/lto.c: No such file or directory > error: lto/lto.h: No such file or directory > error: lto-streamer.h: No such file or directory > error: toplev.c: No such file or directory > ``` > >>> > >>> I can try to update the patch, or it probably should apply to trunk > >>> checked out around the date I sent the patch. Indeed we need to change > >>> c to cc but there are likely more changes since then - most importnatly > >>> the early debug info. > >>> At I will see how easy/hard is to make the patch build with current > >>> trunk. > >> We do have ideas for the early debug with the asm-out abstraction to > >> also solve a different issue (missing simple-object support for > >> mingw/darwin). > > > > Note the debug info will be stored to a different .s file, then the file > > will be converted .o by GAS and then the bytecode will be stored to an ELF > > section of a compiled object. I've got also binutils patch when we'll > > be able to directly use the embedded section with compile.o@offset. > > Which will not work, as written, for Darwin since that is neither ELF nor > does it > use GAS - but hopefully, we can find some equivalent mechanism (there is > already > some support in the Darwin backend for switching asm context for LTO output). I think using compile.o@offset will be optional and the fallback is to extract the "file" back to a regular object file just containing debug info like we do currently but with the difference that we do not need to understand the object format since it is created "correctly" by the assembler during compile-time (and we just use the compile-object as a container to not confuse build systems). > >> Namely assemble the early debug in a different file and include the > >> resulting > >> native object in binary form in the compile output - not needing to write > >> assembly .data for that would be a good way to make this more viable. > > > > Btw. do you have any estimation how slow is GAS when we speak about debug > > info? > > How much time can we save since the latest speed-up achieved by GAS? > > > >> You might want to talk to Martin Liska for this who I think had some > >> prototype on this? > > > > I can provide a prototype if needed. > > I’d like to be in to loop from the Darwin PoV.. > thanks > Iain > > > > > Cheers, > > Martin >
Re: [GSoC]Bypass assembler when generating LTO object files
> On 12 Apr 2022, at 13:31, Martin Liška wrote: > > On 4/12/22 11:58, Richard Biener wrote: >> On Tue, Apr 12, 2022 at 11:20 AM Jan Hubicka via Gcc wrote: >>> >>> Hi, > On 08-Apr-2022, at 6:32 PM, Jan Hubicka wrote: > > Ankur, >> I was browsing the list of submitted GSoC projects this year and the >> project regarding bypassing assembler when generating LTO object files >> caught my eye. > I apologize for late reply. I would be very happy to mentor this > project. Thanks for the reply, but unfortunately, due to some reasons, I would not be able to take part in GSoC this year. But the project seems interesting and would be amazing opportunity to learn a lot more things for me, so would it be okay if I try to give it a go outside GSoC if no-one else picks it as their GSoC project this year ? >>> >>> I would be still very happy to help with that! However it would be also >>> pity to not take part of GSoC, so if there is something I can help with >>> on that let me know. >> >> I already have a gcc built from source (sync-ed with trunk/master) and >> launched the test-suite on it. >> >> I am currently in process of understanding the primilary patch >> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and >> experimenting with it. >> >> are there any other things I should be aware of (useful Doc/blog or a >> bug tracking the project) before proceeding further ? > > I think it is pretty much all that exists. Basically we will need to > implement everything that is necessary to stream out valid object file > directly from GCC rather than going through gas. The experimental > prototype sort of worked but it was lacking few things. When I try to apply that patch on my local branch ( branched from trunk ), it seem to be incompatible with the current working tree. Is there a specific branch that I have to apply it to ? or is it due to the recent file rename patch ( changing extensions from .c to .cc ) ? ``` $ git apply --check bypass_asm_patch error: patch failed: Makefile.in:1300 error: Makefile.in: patch does not apply error: common.opt: No such file or directory error: langhooks.c: No such file or directory error: lto/Make-lang.in: No such file or directory error: lto/lto-object.c: No such file or directory error: lto/lto.c: No such file or directory error: lto/lto.h: No such file or directory error: lto-streamer.h: No such file or directory error: toplev.c: No such file or directory ``` >>> >>> I can try to update the patch, or it probably should apply to trunk >>> checked out around the date I sent the patch. Indeed we need to change >>> c to cc but there are likely more changes since then - most importnatly >>> the early debug info. >>> At I will see how easy/hard is to make the patch build with current >>> trunk. >> We do have ideas for the early debug with the asm-out abstraction to >> also solve a different issue (missing simple-object support for >> mingw/darwin). > > Note the debug info will be stored to a different .s file, then the file > will be converted .o by GAS and then the bytecode will be stored to an ELF > section of a compiled object. I've got also binutils patch when we'll > be able to directly use the embedded section with compile.o@offset. Which will not work, as written, for Darwin since that is neither ELF nor does it use GAS - but hopefully, we can find some equivalent mechanism (there is already some support in the Darwin backend for switching asm context for LTO output). >> Namely assemble the early debug in a different file and include the resulting >> native object in binary form in the compile output - not needing to write >> assembly .data for that would be a good way to make this more viable. > > Btw. do you have any estimation how slow is GAS when we speak about debug > info? > How much time can we save since the latest speed-up achieved by GAS? > >> You might want to talk to Martin Liska for this who I think had some >> prototype on this? > > I can provide a prototype if needed. I’d like to be in to loop from the Darwin PoV.. thanks Iain > > Cheers, > Martin > >> Richard. >>> Honza Thanks - Ankur
Re: [GSoC]Bypass assembler when generating LTO object files
On 4/12/22 11:58, Richard Biener wrote: On Tue, Apr 12, 2022 at 11:20 AM Jan Hubicka via Gcc wrote: Hi, On 08-Apr-2022, at 6:32 PM, Jan Hubicka wrote: Ankur, I was browsing the list of submitted GSoC projects this year and the project regarding bypassing assembler when generating LTO object files caught my eye. I apologize for late reply. I would be very happy to mentor this project. Thanks for the reply, but unfortunately, due to some reasons, I would not be able to take part in GSoC this year. But the project seems interesting and would be amazing opportunity to learn a lot more things for me, so would it be okay if I try to give it a go outside GSoC if no-one else picks it as their GSoC project this year ? I would be still very happy to help with that! However it would be also pity to not take part of GSoC, so if there is something I can help with on that let me know. I already have a gcc built from source (sync-ed with trunk/master) and launched the test-suite on it. I am currently in process of understanding the primilary patch (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and experimenting with it. are there any other things I should be aware of (useful Doc/blog or a bug tracking the project) before proceeding further ? I think it is pretty much all that exists. Basically we will need to implement everything that is necessary to stream out valid object file directly from GCC rather than going through gas. The experimental prototype sort of worked but it was lacking few things. When I try to apply that patch on my local branch ( branched from trunk ), it seem to be incompatible with the current working tree. Is there a specific branch that I have to apply it to ? or is it due to the recent file rename patch ( changing extensions from .c to .cc ) ? ``` $ git apply --check bypass_asm_patch error: patch failed: Makefile.in:1300 error: Makefile.in: patch does not apply error: common.opt: No such file or directory error: langhooks.c: No such file or directory error: lto/Make-lang.in: No such file or directory error: lto/lto-object.c: No such file or directory error: lto/lto.c: No such file or directory error: lto/lto.h: No such file or directory error: lto-streamer.h: No such file or directory error: toplev.c: No such file or directory ``` I can try to update the patch, or it probably should apply to trunk checked out around the date I sent the patch. Indeed we need to change c to cc but there are likely more changes since then - most importnatly the early debug info. At I will see how easy/hard is to make the patch build with current trunk. We do have ideas for the early debug with the asm-out abstraction to also solve a different issue (missing simple-object support for mingw/darwin). Note the debug info will be stored to a different .s file, then the file will be converted .o by GAS and then the bytecode will be stored to an ELF section of a compiled object. I've got also binutils patch when we'll be able to directly use the embedded section with compile.o@offset. Namely assemble the early debug in a different file and include the resulting native object in binary form in the compile output - not needing to write assembly .data for that would be a good way to make this more viable. Btw. do you have any estimation how slow is GAS when we speak about debug info? How much time can we save since the latest speed-up achieved by GAS? You might want to talk to Martin Liska for this who I think had some prototype on this? I can provide a prototype if needed. Cheers, Martin Richard. Honza Thanks - Ankur
Re: [GSoC]Bypass assembler when generating LTO object files
On Tue, Apr 12, 2022 at 11:20 AM Jan Hubicka via Gcc wrote: > > Hi, > > > > > > > On 08-Apr-2022, at 6:32 PM, Jan Hubicka wrote: > > > > > > Ankur, > > >> I was browsing the list of submitted GSoC projects this year and the > > >> project regarding bypassing assembler when generating LTO object files > > >> caught my eye. > > > I apologize for late reply. I would be very happy to mentor this > > > project. > > > > Thanks for the reply, but unfortunately, due to some reasons, I would not > > be able to take part in GSoC this year. > > But the project seems interesting and would be amazing opportunity to > > learn a lot more things for me, so would it be okay if I try to give it a > > go outside GSoC if no-one else picks it as their GSoC project this year ? > > I would be still very happy to help with that! However it would be also > pity to not take part of GSoC, so if there is something I can help with > on that let me know. > > > > >> > > >> I already have a gcc built from source (sync-ed with trunk/master) and > > >> launched the test-suite on it. > > >> > > >> I am currently in process of understanding the primilary patch > > >> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and > > >> experimenting with it. > > >> > > >> are there any other things I should be aware of (useful Doc/blog or a > > >> bug tracking the project) before proceeding further ? > > > > > > I think it is pretty much all that exists. Basically we will need to > > > implement everything that is necessary to stream out valid object file > > > directly from GCC rather than going through gas. The experimental > > > prototype sort of worked but it was lacking few things. > > > > When I try to apply that patch on my local branch ( branched from trunk ), > > it seem to be incompatible with the current working tree. Is there a > > specific branch that I have to apply it to ? or is it due to the recent > > file rename patch ( changing extensions from .c to .cc ) ? > > > > ``` > > $ git apply --check bypass_asm_patch > > > > error: patch failed: Makefile.in:1300 > > error: Makefile.in: patch does not apply > > error: common.opt: No such file or directory > > error: langhooks.c: No such file or directory > > error: lto/Make-lang.in: No such file or directory > > error: lto/lto-object.c: No such file or directory > > error: lto/lto.c: No such file or directory > > error: lto/lto.h: No such file or directory > > error: lto-streamer.h: No such file or directory > > error: toplev.c: No such file or directory > > ``` > > I can try to update the patch, or it probably should apply to trunk > checked out around the date I sent the patch. Indeed we need to change > c to cc but there are likely more changes since then - most importnatly > the early debug info. > At I will see how easy/hard is to make the patch build with current > trunk. We do have ideas for the early debug with the asm-out abstraction to also solve a different issue (missing simple-object support for mingw/darwin). Namely assemble the early debug in a different file and include the resulting native object in binary form in the compile output - not needing to write assembly .data for that would be a good way to make this more viable. You might want to talk to Martin Liska for this who I think had some prototype on this? Richard. > Honza > > > > Thanks > > - Ankur > >
Re: [GSoC]Bypass assembler when generating LTO object files
Hi, > > > > On 08-Apr-2022, at 6:32 PM, Jan Hubicka wrote: > > > > Ankur, > >> I was browsing the list of submitted GSoC projects this year and the > >> project regarding bypassing assembler when generating LTO object files > >> caught my eye. > > I apologize for late reply. I would be very happy to mentor this > > project. > > Thanks for the reply, but unfortunately, due to some reasons, I would not > be able to take part in GSoC this year. > But the project seems interesting and would be amazing opportunity to > learn a lot more things for me, so would it be okay if I try to give it a > go outside GSoC if no-one else picks it as their GSoC project this year ? I would be still very happy to help with that! However it would be also pity to not take part of GSoC, so if there is something I can help with on that let me know. > > >> > >> I already have a gcc built from source (sync-ed with trunk/master) and > >> launched the test-suite on it. > >> > >> I am currently in process of understanding the primilary patch > >> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and > >> experimenting with it. > >> > >> are there any other things I should be aware of (useful Doc/blog or a > >> bug tracking the project) before proceeding further ? > > > > I think it is pretty much all that exists. Basically we will need to > > implement everything that is necessary to stream out valid object file > > directly from GCC rather than going through gas. The experimental > > prototype sort of worked but it was lacking few things. > > When I try to apply that patch on my local branch ( branched from trunk ), > it seem to be incompatible with the current working tree. Is there a > specific branch that I have to apply it to ? or is it due to the recent > file rename patch ( changing extensions from .c to .cc ) ? > > ``` > $ git apply --check bypass_asm_patch > > error: patch failed: Makefile.in:1300 > error: Makefile.in: patch does not apply > error: common.opt: No such file or directory > error: langhooks.c: No such file or directory > error: lto/Make-lang.in: No such file or directory > error: lto/lto-object.c: No such file or directory > error: lto/lto.c: No such file or directory > error: lto/lto.h: No such file or directory > error: lto-streamer.h: No such file or directory > error: toplev.c: No such file or directory > ``` I can try to update the patch, or it probably should apply to trunk checked out around the date I sent the patch. Indeed we need to change c to cc but there are likely more changes since then - most importnatly the early debug info. At I will see how easy/hard is to make the patch build with current trunk. Honza > > Thanks > - Ankur >
Re: [GSoC]Bypass assembler when generating LTO object files
> On 08-Apr-2022, at 6:32 PM, Jan Hubicka wrote: > > Ankur, >> I was browsing the list of submitted GSoC projects this year and the >> project regarding bypassing assembler when generating LTO object files >> caught my eye. > I apologize for late reply. I would be very happy to mentor this > project. Thanks for the reply, but unfortunately, due to some reasons, I would not be able to take part in GSoC this year. But the project seems interesting and would be amazing opportunity to learn a lot more things for me, so would it be okay if I try to give it a go outside GSoC if no-one else picks it as their GSoC project this year ? >> >> I already have a gcc built from source (sync-ed with trunk/master) and >> launched the test-suite on it. >> >> I am currently in process of understanding the primilary patch >> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and >> experimenting with it. >> >> are there any other things I should be aware of (useful Doc/blog or a >> bug tracking the project) before proceeding further ? > > I think it is pretty much all that exists. Basically we will need to > implement everything that is necessary to stream out valid object file > directly from GCC rather than going through gas. The experimental > prototype sort of worked but it was lacking few things. When I try to apply that patch on my local branch ( branched from trunk ), it seem to be incompatible with the current working tree. Is there a specific branch that I have to apply it to ? or is it due to the recent file rename patch ( changing extensions from .c to .cc ) ? ``` $ git apply --check bypass_asm_patch error: patch failed: Makefile.in:1300 error: Makefile.in: patch does not apply error: common.opt: No such file or directory error: langhooks.c: No such file or directory error: lto/Make-lang.in: No such file or directory error: lto/lto-object.c: No such file or directory error: lto/lto.c: No such file or directory error: lto/lto.h: No such file or directory error: lto-streamer.h: No such file or directory error: toplev.c: No such file or directory ``` Thanks - Ankur
Re: [GSoC]Bypass assembler when generating LTO object files
Ankur, > I was browsing the list of submitted GSoC projects this year and the > project regarding bypassing assembler when generating LTO object files > caught my eye. I apologize for late reply. I would be very happy to mentor this project. > > I already have a gcc built from source (sync-ed with trunk/master) and > launched the test-suite on it. > > I am currently in process of understanding the primilary patch > (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and > experimenting with it. > > are there any other things I should be aware of (useful Doc/blog or a > bug tracking the project) before proceeding further ? I think it is pretty much all that exists. Basically we will need to implement everything that is necessary to stream out valid object file directly from GCC rather than going through gas. The experimental prototype sort of worked but it was lacking few things. First is the production of proper object file hearder (it encodes things like architeture ELF is produced for), production of symbol table that is necessary to mark the LTO object file and also we now need a way to stream debug info (DWARF) directly to the object from dwarf2out. So I think first step would be to produce object files w/o debug info which can be consumed by unmodified linkers and then look into DWARF bytecode streaming. Honza > > I am Ankur Saini, a B.Tech CSE 3rd year student at USICT, GGSIPU india > and a former GSoC contributor at gcc ( worked on expanding gcc static > analyzer's C++ support in GSoC 2021 > [https://gist.github.com/Arsenic-ATG/8f4ac194f460dd9b2c78cf51af39afef]) > > Thanks > - Ankur
[GSoC]Bypass assembler when generating LTO object files
Hi, I was browsing the list of submitted GSoC projects this year and the project regarding bypassing assembler when generating LTO object files caught my eye. I already have a gcc built from source (sync-ed with trunk/master) and launched the test-suite on it. I am currently in process of understanding the primilary patch (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and experimenting with it. are there any other things I should be aware of (useful Doc/blog or a bug tracking the project) before proceeding further ? I am Ankur Saini, a B.Tech CSE 3rd year student at USICT, GGSIPU india and a former GSoC contributor at gcc ( worked on expanding gcc static analyzer's C++ support in GSoC 2021 [https://gist.github.com/Arsenic-ATG/8f4ac194f460dd9b2c78cf51af39afef]) Thanks - Ankur