Re: Optlink Contribution
On 2014-07-30 14:51, w0rp wrote: I think it's important to ship with a linker without requiring any further installation. Just for the record. Currently, Windows in the only platform where this is true. -- /Jacob Carlborg
Re: Optlink Contribution
On Wednesday, 30 July 2014 at 19:50:35 UTC, Jonathan Marler wrote: I like the discussion. I do want to remind everyone that OPTLINK is very fast and switching to a different linker will likely result performance hit. Wouldn't it be easier to optimize a linker written in D than tinker with optlink?
Re: Optlink Contribution
On Thursday, 31 July 2014 at 06:46:19 UTC, Kagamin wrote: On Wednesday, 30 July 2014 at 19:50:35 UTC, Jonathan Marler wrote: I like the discussion. I do want to remind everyone that OPTLINK is very fast and switching to a different linker will likely result performance hit. Wouldn't it be easier to optimize a linker written in D than tinker with optlink? No matter how much you optimize your D code, you will only ever be able to use a subset of what assembly can do. That being said, writing perfect assembly code is impossible, so we rely on the compiler to take our higher level concepts and have it decide on the best way to optimize. The beauty of higher level languages is you can make high level changes resulting an a cascade of optimizations. But the downside is sometimes changes can have cascading adverse affects as well. One of the things I like about D is that it does a much better job of allowing the programmer to tell the compiler the right information it needs to know how to optimize. I would like to see a linker written in D and see how it compares to optlink. But I would reserve making a decision on moving to another linker until I got some solid performance data. Some performance data for a D linker compiled with all three D compilers as well.
Re: Optlink Contribution
On Thursday, 31 July 2014 at 07:07:27 UTC, Jonathan Marler wrote: No matter how much you optimize your D code, you will only ever be able to use a subset of what assembly can do. Such micro-optimizations give only marginal contribution to performance, which is mostly determined by choice of algorithms. The approach to write everything in assembly with manual optimizations is obsolete for many years, at least for desktop platforms. This is also one of the reasons, why optlink is translated to C. I would like to see a linker written in D and see how it compares to optlink. But I would reserve making a decision on moving to another linker until I got some solid performance data. Some performance data for a D linker compiled with all three D compilers as well. I only wanted to say, no matter what performance data we get, the linker can be optimized by choosing faster algorithms without tinkering with assembler.
Re: Optlink Contribution
Jonathan Marler wrote in message news:xrpdzjuaxtdhyfhps...@forum.dlang.org... I like the discussion. I do want to remind everyone that OPTLINK is very fast and switching to a different linker will likely result performance hit. There are advantages to using COFF as it seems more compilers use that format and D would be more interoperable with other compilers and languages. I think every peice of software has it's own goals which determine what tradeoffs to chose. In my view, the linker should 1) always work and 2) be as fast as possible. OPTLINK passes number 2 with flying colors but is does have bugs. This is obviously due to the fact that it is written in assembly but keep in mind that the performance of the linker affects everyone who uses D. Any performance hit on the linker will be seen in every single build of any D program. Adding 1 second to link time will add 1 second to build time since this process cannot be parallelized (as far as I know). Optlink simply doesn't do enough. It doesn't support modern debug info formats. It doesn't support mixing omf and coff object files (this is probably the most requested feature, because it would allow using the stock import libraries). Worst of all, optlink is hostile to change. win32 is dying, and optlink will certainly die with it. Having a more accessible linker will let us adapt much faster.
Re: Optlink Contribution
On 7/29/2014 10:53 PM, Jonathan Marler wrote: I'm attempting to fix https://issues.dlang.org/show_bug.cgi?id=4831. I've been debugging the optlink assembly and getting familiar with the code. I have a couple questions though: 1. If I have any questions in the future about optlink who and where do I ask these questions? For now I'm posting to this forum because I'm not sure where else to send them. I'm really the only person, except a couple others who have submitted PRs against optlink. 2. Why are the assembly labels in this format: Lnumber$? I ask because I changed the labels for some functions so when I stepped through the assembly I had a better idea of what the code was doing. So I'm not sure if I should leave the new labels or go back to the old Lnumber$ labels. No known reason. 3. Is there any documentation for developers who would like to contribute to optlink? I'd like to make sure I'm following any style rules or guidelines. The fundamental problem with fixing optlink is there is essentially no test suite. This means that any fixes to it need to be surgical - as little code modified as practical, and pretty great care in doing it. Wholesale refactoring just because or for cleanup are out of the question. Wholesale reformatting is also out of the question. Once what a function does is figured out, and its inputs and outputs identified, adding comments to that effect is appreciated. Most functions in Optlink do not identify what registers are used as input, what registers are used for output, and which registers must be preserved. Some functions even return results in flags. There is no consistency. To identify these, one must look at every call of that function. Any information about developing for optlink would be helpful. I haven't found much information online. Thanks for any help, I'd like to start contributing to the D tools, its a crime to let all these bugs live on for so long when they are making adopting this language a hindrence for others. I appreciate you stepping up to make a difference on this. What I've done to fix bugs in Optlink is to first identify where in the code things go wrong. Then, I convert that section of code to C. The C code, line by line, mirrors the assembler. Then I fix the C code. This makes things much easier because I can use printf, etc., in the C code.
Re: Optlink Contribution
Making dmd generate coff would make more sense.
Re: Optlink Contribution
On Wednesday, 30 July 2014 at 06:54:52 UTC, Walter Bright wrote: The fundamental problem with fixing optlink is there is essentially no test suite. This means that any fixes to it need to be surgical - as little code modified as practical, and pretty great care in doing it. Wholesale refactoring just because or for cleanup are out of the question. Wholesale reformatting is also out of the question. The solution there is to write a comprehensive test suite. Then it can be refactored. I'm not saying it's an easy solution, it's probably a major pain, but that's what it is.
Re: Optlink Contribution
On 30/07/2014 7:03 p.m., Kagamin wrote: Making dmd generate coff would make more sense. +1 Most of the code should already be present in dmd, which makes it far crazier not to.
Re: Optlink Contribution
It looks like the easiest way to fix the bug is to change the get_filename function to support non HPFS characters. I'm guessing this was originally written to run exclusively on HPFS systems? Is this tool still suppose to support HPFS file systems? If so maybe we could add a runtime check to see what kind of fs the program is being run on (or a compile time check if we are guaranteed that each system with a different file system will use a different build, I'm guessing this is not the case.) If HPFS does not need to be supported then the fix is really easy. I also found a buffer overrun bug in the get_filename function (if your filename does not have an ending quote).
Re: Optlink Contribution
On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote: On 30/07/2014 7:03 p.m., Kagamin wrote: Making dmd generate coff would make more sense. +1 Most of the code should already be present in dmd, which makes it far crazier not to. What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmar...@puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether.
Re: Optlink Contribution
On 30/07/2014 8:58 p.m., Joakim wrote: On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote: On 30/07/2014 7:03 p.m., Kagamin wrote: Making dmd generate coff would make more sense. +1 Most of the code should already be present in dmd, which makes it far crazier not to. What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmar...@puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether. If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. If we can do this, I think Walter might go along with it. Unless of course we could convince Microsoft to have a download just for the linker. We could download that in e.g. the installer. Would be better than a full install.
Re: Optlink Contribution
On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole wrote: On 30/07/2014 8:58 p.m., Joakim wrote: On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote: On 30/07/2014 7:03 p.m., Kagamin wrote: Making dmd generate coff would make more sense. +1 Most of the code should already be present in dmd, which makes it far crazier not to. What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmar...@puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether. If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. If we can do this, I think Walter might go along with it. Unless of course we could convince Microsoft to have a download just for the linker. We could download that in e.g. the installer. Would be better than a full install. I don't think dmd comes with a COFF64 linker now, users are just told to install Visual Studio or the Windows SDK for a linker. No reason you can't do the same with COFF32. Optlink can stick around with OMF for a couple releases. I suspect nobody would use it when given the choice of COFF32 support.
Re: Optlink Contribution
On 30/07/2014 9:17 p.m., Joakim wrote: On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole wrote: On 30/07/2014 8:58 p.m., Joakim wrote: On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote: On 30/07/2014 7:03 p.m., Kagamin wrote: Making dmd generate coff would make more sense. +1 Most of the code should already be present in dmd, which makes it far crazier not to. What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmar...@puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether. If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. If we can do this, I think Walter might go along with it. Unless of course we could convince Microsoft to have a download just for the linker. We could download that in e.g. the installer. Would be better than a full install. I don't think dmd comes with a COFF64 linker now, users are just told to install Visual Studio or the Windows SDK for a linker. No reason you can't do the same with COFF32. Optlink can stick around with OMF for a couple releases. I suspect nobody would use it when given the choice of COFF32 support. I'm in agreement with you about just getting people to install it. But the problem is, its not out of the box enough for Walters liking and I'm partially agreeing with him on this. At the end of the day, we don't really want to tell new people you have to also install x. Also not to forget that we could also remove OUR implib versions of Windows Dlls. That would solve a lot of issues for people I bet.
Re: Optlink Contribution
On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole wrote: If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. GNU binutils should do.
Re: Optlink Contribution
On Wednesday, 30 July 2014 at 09:17:05 UTC, Joakim wrote: On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole wrote: On 30/07/2014 8:58 p.m., Joakim wrote: On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote: On 30/07/2014 7:03 p.m., Kagamin wrote: Making dmd generate coff would make more sense. +1 Most of the code should already be present in dmd, which makes it far crazier not to. What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmar...@puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether. If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. If we can do this, I think Walter might go along with it. Unless of course we could convince Microsoft to have a download just for the linker. We could download that in e.g. the installer. Would be better than a full install. I don't think dmd comes with a COFF64 linker now, users are just told to install Visual Studio or the Windows SDK for a linker. No reason you can't do the same with COFF32. Optlink can stick around with OMF for a couple releases. I suspect nobody would use it when given the choice of COFF32 support. I think it's important to ship with a linker without requiring any further installation. One of the things that helped me to learn D was being able to download DMD and run RDMD on Windows without installing anything else. It's not obvious to very new users who don't come from a Windows C++ background where you can get a gratis Microsoft compiler or how to configure it. If we were going to switch to COFF32, it would be a big bonus to ship a COFF32 linker in the installer.
Re: Optlink Contribution
w0rp wrote in message news:sinwmhzuvhmevqtun...@forum.dlang.org... I think it's important to ship with a linker without requiring any further installation. One of the things that helped me to learn D was being able to download DMD and run RDMD on Windows without installing anything else. It's not obvious to very new users who don't come from a Windows C++ background where you can get a gratis Microsoft compiler or how to configure it. If we were going to switch to COFF32, it would be a big bonus to ship a COFF32 linker in the installer. FWIW this was the plan when I started ylink. I wanted to make a linker that could link omf and coff together, so I could tackle the object-format and runtime-format transitions independently. It does support mscoff32 hello world now, but not much more. Until it has debug info support it's not a viable replacement for optlink unfortunately.
Re: Optlink Contribution
On Wednesday, 30 July 2014 at 13:03:30 UTC, Daniel Murphy wrote: w0rp wrote in message news:sinwmhzuvhmevqtun...@forum.dlang.org... I think it's important to ship with a linker without requiring any further installation. One of the things that helped me to learn D was being able to download DMD and run RDMD on Windows without installing anything else. It's not obvious to very new users who don't come from a Windows C++ background where you can get a gratis Microsoft compiler or how to configure it. If we were going to switch to COFF32, it would be a big bonus to ship a COFF32 linker in the installer. FWIW this was the plan when I started ylink. I wanted to make a linker that could link omf and coff together, so I could tackle the object-format and runtime-format transitions independently. It does support mscoff32 hello world now, but not much more. Until it has debug info support it's not a viable replacement for optlink unfortunately. I didn't know about ylink. The prospect of having a free software linker for D on Win32 written in D does sound attractive. I assume it would be a lot of work to make it acceptable for usage.
Re: Optlink Contribution
w0rp wrote in message news:vnaffnibgvtmqeuhz...@forum.dlang.org... I didn't know about ylink. The prospect of having a free software linker for D on Win32 written in D does sound attractive. I assume it would be a lot of work to make it acceptable for usage. I would estimate it's a smaller effort than DDMD. (win32 + omf + cv4) We really don't need to match the performance and memory usage of optlink. The best part is we can then potentially re-use parts of it inside dmd, currently dmd has its own lib generator for each format.
Re: Optlink Contribution
I don't think dmd comes with a COFF64 linker now, users are just told to install Visual Studio or the Windows SDK for a linker. No reason you can't do the same with COFF32. Optlink can stick around with OMF for a couple releases. I suspect nobody would use it when given the choice of COFF32 support. Anybody seriously programming on Windows uses VS anyway. But at least for 32 bit http://lld.llvm.org/windows_support.html could be packaged.
Re: Optlink Contribution
What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmar...@puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com Same procedure as every year.
Re: Optlink Contribution
I like the discussion. I do want to remind everyone that OPTLINK is very fast and switching to a different linker will likely result performance hit. There are advantages to using COFF as it seems more compilers use that format and D would be more interoperable with other compilers and languages. I think every peice of software has it's own goals which determine what tradeoffs to chose. In my view, the linker should 1) always work and 2) be as fast as possible. OPTLINK passes number 2 with flying colors but is does have bugs. This is obviously due to the fact that it is written in assembly but keep in mind that the performance of the linker affects everyone who uses D. Any performance hit on the linker will be seen in every single build of any D program. Adding 1 second to link time will add 1 second to build time since this process cannot be parallelized (as far as I know). That being said, if it were up to me I wouldn't abandon OPTLINK so quickly. I would spend time creating a test suite for OPTLINK, and try to comment the code a little better to encourage other developers to contribute. Then I would consider adding support in DMD to generate COFF output files for those who want to link D object files with another linker. I wouldn't make this a huge priority though as one can use an OMF to COFF converter if they really needed it.
Optlink Contribution
I'm attempting to fix https://issues.dlang.org/show_bug.cgi?id=4831. I've been debugging the optlink assembly and getting familiar with the code. I have a couple questions though: 1. If I have any questions in the future about optlink who and where do I ask these questions? For now I'm posting to this forum because I'm not sure where else to send them. 2. Why are the assembly labels in this format: Lnumber$? I ask because I changed the labels for some functions so when I stepped through the assembly I had a better idea of what the code was doing. So I'm not sure if I should leave the new labels or go back to the old Lnumber$ labels. 3. Is there any documentation for developers who would like to contribute to optlink? I'd like to make sure I'm following any style rules or guidelines. Any information about developing for optlink would be helpful. I haven't found much information online. Thanks for any help, I'd like to start contributing to the D tools, its a crime to let all these bugs live on for so long when they are making adopting this language a hindrence for others.
