Re: DMD crash, LDC crash, out of memory, no error display, etc
On Sunday, 9 April 2017 at 08:22:02 UTC, Rainer Schuetze wrote: On 08.04.2017 22:33, Nierjerson wrote: The project I am working on, which is near completion: https://github.com/IllusionSoftware/COM2D Automates COM wrapping almost completely. That is, it takes a converted COM idl file and generates a D wrapper which handles all the marshaling, invoking, creation, etc. The problem is that DMD has started crashing when trying to compile the complete set of interfaces. No error, warnings, etc. Just crashes. Interesting project. In Visual D I had to marshal a couple of function calls manually, and that has been quite painful. Automating that process would be great. A couple of notes: - you can compile dmd as a 64-bit process using the VS solution in src/vcbuild. - when built with this compiler the compile time generated string is output using about 8GB of memory. - it still crashes after that due to a stack overflow (seems to be a compiler bug). You can avoid the recursion using the -allinst command line option at the cost of slightly more memory being used (should be very little in this project) - compilation then continues but seems to be stalled. This is caused by the huge function GenerateDCOMClasses, that is still compiled into the object file even though it is just used at compile time. dmd is pretty slow compiling large funtions, you shouldn't dare enabling optimizations for these... - eventually (using about 11 GB of memory) the object file is emitted with a size of 137 MB. If possible you should avoid huge functions. Would it be feasible to make the generation a runtime tool? If I understand correctly, you run idl2d on the interface file anyway, so integration with idl2d would make sense, too. Strangely enough, I removed an inner aliasSeqOf and converted a few static if's in to if's(calling the function at compile time) and the code when from 10m generation(and usually crashing) to 10s!!! OMFG! Maybe a bit of optimization of the code and reduce some further issues. (FYI, I simply converted the foreach(m; aliasSeqOf!methods) to foreach(m; methods) and converted the static if's to if's.) Unfortunately it corrupts some code but if I can fix that then it might be a viable solution. If aliasSeqOf is that slow(60x factor, more or less) and all I'm using it for is to be able to statically iterate through an array at CT, then something is wrong with D. Either it needs a valid compile looping mechanism that doesn't require one to jump through any hoops(aliasSeqOf) or it needs to be optimized.
Re: DMD crash, LDC crash, out of memory, no error display, etc
On 4/9/2017 12:46 PM, Nierjerson wrote: I just tried compiling. dmd32 crashes optilink. dmd64 works(11m build) for the second reduced interfaces, but crashes for the ALL interfaces or the first reduced. I think D has some major problems. The code is relatively straight forward and works fine for a few interfaces. Nothing complex is happening in the process except for a little CT reflection. Please file bugzilla issues.
Re: DMD crash, LDC crash, out of memory, no error display, etc
On 4/9/2017 12:29 PM, Nierjerson wrote: Unfortunately I tried compiling again only the string output to file case and optilink crashed: checkpoint(256) Error: linker exited with status 1 with a dialog of cpu regs Please file a bugzilla issue with this, with the contents of the CPU regs at least, and hopefully the source code that reproduces it.
Re: DMD crash, LDC crash, out of memory, no error display, etc
On 4/9/2017 1:22 AM, Rainer Schuetze wrote: [...] Please file bugzilla issue(s) for what you have discovered.
Re: DMD crash, LDC crash, out of memory, no error display, etc
I just tried compiling. dmd32 crashes optilink. dmd64 works(11m build) for the second reduced interfaces, but crashes for the ALL interfaces or the first reduced. I think D has some major problems. The code is relatively straight forward and works fine for a few interfaces. Nothing complex is happening in the process except for a little CT reflection. I will try to split up the Generating function and maybe it will help. I thought someone said that it creates more symbols and is worse though.
Re: DMD crash, LDC crash, out of memory, no error display, etc
On Sunday, 9 April 2017 at 08:22:02 UTC, Rainer Schuetze wrote: On 08.04.2017 22:33, Nierjerson wrote: The project I am working on, which is near completion: https://github.com/IllusionSoftware/COM2D Automates COM wrapping almost completely. That is, it takes a converted COM idl file and generates a D wrapper which handles all the marshaling, invoking, creation, etc. The problem is that DMD has started crashing when trying to compile the complete set of interfaces. No error, warnings, etc. Just crashes. Interesting project. In Visual D I had to marshal a couple of function calls manually, and that has been quite painful. Automating that process would be great. I think so too ;) If it works out well then it could really remove the COM headache. I'm not sure how robust it is though as I have only tested it with the photoshop interface, which is dynamic, but I guess all COM interfaces have a dynamic back(e.g. can use GetIDNames, Invoke, etc). Most of the marshaling can be done completely automatic. The only thing I think that one can't get around is being able to extract the VARIANT type at runtime(because it is variant, of course ;)... but maybe a helper template function will make those cases easier. I figure for those cases, one can extend the classes and overload the functions to take arguments one likes, if they plan on using it often. Only a slight bit of work. A couple of notes: - you can compile dmd as a 64-bit process using the VS solution in src/vcbuild. Yeah, I'm using it. Helps because this project gets over 2GB very quickly ;/ - when built with this compiler the compile time generated string is output using about 8GB of memory. Which enum Interface is that? That is about what it uses for the small set. - it still crashes after that due to a stack overflow (seems to be a compiler bug). You can avoid the recursion using the -allinst command line option at the cost of slightly more memory being used (should be very little in this project) ok, I'm trying it and it seems to work. - compilation then continues but seems to be stalled. This is caused by the huge function GenerateDCOMClasses, that is still compiled into the object file even though it is just used at compile time. dmd is pretty slow compiling large funtions, you shouldn't dare enabling optimizations for these... - eventually (using about 11 GB of memory) the object file is emitted with a size of 137 MB. If possible you should avoid huge functions. Would it be feasible to make the generation a runtime tool? If I understand correctly, you run idl2d on the interface file anyway, so integration with idl2d would make sense, too. Yeah, possible. Seems it uses far less memory(hovering around ~6GB so far when it would normally be at 12GB+ and eventually error out)) I simply removed the mixin code and pragma and allowed will have it write the string output to file like it already does. I did it the other way for development because it is faster but because of the memory issues I couldn't get very far initially). I didn't occur to me to do this, I guess because I wanted the compiler to just compile it as it was ;) Kinda bugs me that it can't handle it well. Eventually it could be used as a tool to output from the idl2d the D wrapper classes. I've hacked up the idl2d output and it does expect the idl2d output to have a slightly format(hand edited). These can be modded though. I have to tidy up some things and start testing it out in the real world before I'm satisfied, but at least the proof of concept seems to work. Unfortunately I tried compiling again only the string output to file case and optilink crashed: checkpoint(256) Error: linker exited with status 1 with a dialog of cpu regs Anyways, I'll play around with the new method, it should, in theory, work as it is just writing the large code string to file rather than trying to mix in it(which, shouldn't be a big problem either as it's just the same string that is output to a file then mixed in again). Thanks.
Re: DMD crash, LDC crash, out of memory, no error display, etc
On 08.04.2017 22:33, Nierjerson wrote: The project I am working on, which is near completion: https://github.com/IllusionSoftware/COM2D Automates COM wrapping almost completely. That is, it takes a converted COM idl file and generates a D wrapper which handles all the marshaling, invoking, creation, etc. The problem is that DMD has started crashing when trying to compile the complete set of interfaces. No error, warnings, etc. Just crashes. Interesting project. In Visual D I had to marshal a couple of function calls manually, and that has been quite painful. Automating that process would be great. A couple of notes: - you can compile dmd as a 64-bit process using the VS solution in src/vcbuild. - when built with this compiler the compile time generated string is output using about 8GB of memory. - it still crashes after that due to a stack overflow (seems to be a compiler bug). You can avoid the recursion using the -allinst command line option at the cost of slightly more memory being used (should be very little in this project) - compilation then continues but seems to be stalled. This is caused by the huge function GenerateDCOMClasses, that is still compiled into the object file even though it is just used at compile time. dmd is pretty slow compiling large funtions, you shouldn't dare enabling optimizations for these... - eventually (using about 11 GB of memory) the object file is emitted with a size of 137 MB. If possible you should avoid huge functions. Would it be feasible to make the generation a runtime tool? If I understand correctly, you run idl2d on the interface file anyway, so integration with idl2d would make sense, too.
DMD crash, LDC crash, out of memory, no error display, etc
The project I am working on, which is near completion: https://github.com/IllusionSoftware/COM2D Automates COM wrapping almost completely. That is, it takes a converted COM idl file and generates a D wrapper which handles all the marshaling, invoking, creation, etc. The problem is that DMD has started crashing when trying to compile the complete set of interfaces. No error, warnings, etc. Just crashes. Before I'd get out of memory errors, which I changed everything to use string instead of wstring, which seemed to help quite a bit. LDC crashes when running the test app, which may or may not be related. It too would give an out of memory error though for the large set before. The test app works in 2 ways. First, if version != gen, it will generate the D wrappers, output them to the console via pragma, and on execution of the program, write it to gen.d. If version = gen, it uses the gen'ed version. This allows to debug the generated code, for testing, and quicker compilation. I've also gotten failed compilations without any indication why. While the code uses photoshop, it is not required to get the errors or memory issues. simply set version != go and it will disable all the photoshop call functions. The only thing that really changes the behavior(from working to out of memory to crashing) is using different "Interface" sets in main.d. Using a smaller set avoids the crash or memory issue but cannot be used properly because only that interface will generate a D wrapper which limits it's usability. Anyone willing to work on this a bit to see if they can figure out the issues? The main goal is to get ALL: //ALL enum Interfaces = ... to compile completely(there may be code generation errors, since I haven't been able to debug that setting yet because of the crashing). Once this is fixed I can then work on making the code more robust and clean it up.
