Making an .exe that executes source file inside itself.
So there has been idea I've got for around few months now: making a software which executable would contain a source file. A software that anyone could modify by opening an executable and quickly change a few lines of it, rerun an executable, see the changes. Could this be easily possible with D language, considering that sources files can be ran without long slow compilation process?
Re: Making an .exe that executes source file inside itself.
On Wednesday, April 25, 2018 19:19:58 BoQsc via Digitalmars-d-learn wrote: > So there has been idea I've got for around few months now: making > a software which executable would contain a source file. > A software that anyone could modify by opening an executable and > quickly change a few lines of it, rerun an executable, see the > changes. > > Could this be easily possible with D language, considering that > sources files can be ran without long slow compilation process? The normal way to do that is to just write a script. In the case of D, you can just use rdmd to do it. e.g. if you're on a POSIX system, just put #!/usr/bin/env rdmd at the top of your .d file and chmod it so that it's executable, and it'll run like any other script. - Jonathan M Davis
Re: Making an .exe that executes source file inside itself.
On Wednesday, 25 April 2018 at 19:43:31 UTC, Jonathan M Davis wrote: On Wednesday, April 25, 2018 19:19:58 BoQsc via Digitalmars-d-learn wrote: So there has been idea I've got for around few months now: making a software which executable would contain a source file. A software that anyone could modify by opening an executable and quickly change a few lines of it, rerun an executable, see the changes. Could this be easily possible with D language, considering that sources files can be ran without long slow compilation process? The normal way to do that is to just write a script. In the case of D, you can just use rdmd to do it. e.g. if you're on a POSIX system, just put #!/usr/bin/env rdmd at the top of your .d file and chmod it so that it's executable, and it'll run like any other script. - Jonathan M Davis Thank you Jonathan for a response. I was aware of this, and it certainly perfectly works where command line/terminal interface is the main tool to control the system, a good example would be linux/gnu distributions and macOS. However in Windows while installing D language, I noticed that .d source file extension is not associated with neither D compiler (dmd.exe) nor D script interpretator (rdmd.exe) So they can't be ran directly by clicking on those, nor they have any icons that show that these source codes are actually executable. This is a huge problem, because people that are not aware of D language migh be harder to understand that source script could be executable, at least - without help from some more experienced user.
Re: Making an .exe that executes source file inside itself.
On Wednesday, 25 April 2018 at 19:54:26 UTC, BoQsc wrote: On Wednesday, 25 April 2018 at 19:43:31 UTC, Jonathan M Davis wrote: On Wednesday, April 25, 2018 19:19:58 BoQsc via Digitalmars-d-learn wrote: So there has been idea I've got for around few months now: making a software which executable would contain a source file. A software that anyone could modify by opening an executable and quickly change a few lines of it, rerun an executable, see the changes. Could this be easily possible with D language, considering that sources files can be ran without long slow compilation process? The normal way to do that is to just write a script. In the case of D, you can just use rdmd to do it. e.g. if you're on a POSIX system, just put #!/usr/bin/env rdmd at the top of your .d file and chmod it so that it's executable, and it'll run like any other script. - Jonathan M Davis Thank you Jonathan for a response. I was aware of this, and it certainly perfectly works where command line/terminal interface is the main tool to control the system, a good example would be linux/gnu distributions and macOS. However in Windows while installing D language, I noticed that .d source file extension is not associated with neither D compiler (dmd.exe) nor D script interpretator (rdmd.exe) So they can't be ran directly by clicking on those, nor they have any icons that show that these source codes are actually executable. This is a huge problem, because people that are not aware of D language migh be harder to understand that source script could be executable, at least - without help from some more experienced user. If the purpose is to make scripts run by clicking them you can assign a file type to .d files. On Windows 10. https://www.digitaltrends.com/computing/how-to-set-default-programs-and-file-types-in-windows-10/ Doing so would make the script engine the default program instead of a text editor so you might not want to. Or maybe assign .dxe and changing the filename before running.
Re: Making an .exe that executes source file inside itself.
On Wednesday, 25 April 2018 at 19:19:58 UTC, BoQsc wrote: So there has been idea I've got for around few months now: making a software which executable would contain a source file. A software that anyone could modify by opening an executable and quickly change a few lines of it, rerun an executable, see the changes. Could this be easily possible with D language, considering that sources files can be ran without long slow compilation process? I had a similar idea a few months ago too, coincidence? The idea is that all apps have the source code embedded so that they can be debugged and modified by the user at will. Each app has a built in IDE and debugger(or can trigger an external one for space). When an error occurs it will jump to the debugger for debugging purposes. If some functionality is to be modified(e.g., change the size of a button or add some new feature) the user can trigger the IDE(hot key) to proceed. The point of having it all self contained is to allow future revision and debugging without having to worry about version issues. The point of the debugging is so the user can fix bugs that plague software. While on the whole most software may work for most users, it is very annoying when some minor bug prevents a program from working for a specific user. Many more people know how to program than ever and it will only become more and more common. Also, with such capabilities, the user can add additional functionality. It is very annoying when programs don't support simple stuff. A few lines of code could make the program much more effective for a user that wants to do something that was not thought of if they could just add it. e.g., even just display something, change the location of a widget, or modify how a text file is read so it can support a new format. One can argue that platforms such as Linux make this easy... but only marginally. If one get actually get the code to compile without having to waste many hours, one must also generally sift through the code to find the relevant parts then try and understand them. Usually because of the trial and error process is long this generally is a deterrent for "normal" people with "normal" lives. If apps are properly structured with such goals in mind then it would be of great benefit, although only a few of us like you and me have the wherewithal to extrapolate out. It would not be difficult to do(it is more about organization than about technological problem solving). With proper ide design, debugger, and such, one could make app writing self contained, so to speak. That is, the IDE and debugger are used directly to write the program instead of using the traditional methods such as vim, emacs, visual studio, coedit, etc. What is packaged in an binary are the IDE Debugger Profiling Source Code(with history, VCS included) Compiled Code(possibly optimized version) Debugging Data Scripting Generally, without debugging, the executable just runs the compiled code as if it were a normal app. When triggered, the ide/debugger can be brought up that takes over and displays the desired source code(triggers can be used). I believe the main issues for proper development would be the ability for quick code replacement(hot swapping, injection, etc) which is tricky since some code would require a full restart which will slow things down. Most of the other programs are just gluing everything together and providing certain mechanisms that allow for robustness such as being able to replace the IDE or debugger, extract the source code for repackaging. While it would be time consuming to implement well it would be very useful. There are many many times I have used programs that simply do not function correctly or are missing simple functionality that would take me a just a few minutes to write a few lines of code to fix. Having the above would make it possible to do. One could also then "upload" patches/fixes/enhancements to the cloud or download others. This way people can tailor applications to their own use rather than being stuck with what the developers(which generally are somewhat clueless as their main goal is to ship the product rather than provide all the functionality their users want).
Re: Making an .exe that executes source file inside itself.
On Wednesday, 25 April 2018 at 20:44:10 UTC, u0_a183 wrote: On Wednesday, 25 April 2018 at 19:54:26 UTC, BoQsc wrote: On Wednesday, 25 April 2018 at 19:43:31 UTC, Jonathan M Davis wrote: On Wednesday, April 25, 2018 19:19:58 BoQsc via Digitalmars-d-learn wrote: So there has been idea I've got for around few months now: making a software which executable would contain a source file. A software that anyone could modify by opening an executable and quickly change a few lines of it, rerun an executable, see the changes. Could this be easily possible with D language, considering that sources files can be ran without long slow compilation process? The normal way to do that is to just write a script. In the case of D, you can just use rdmd to do it. e.g. if you're on a POSIX system, just put #!/usr/bin/env rdmd at the top of your .d file and chmod it so that it's executable, and it'll run like any other script. - Jonathan M Davis Thank you Jonathan for a response. I was aware of this, and it certainly perfectly works where command line/terminal interface is the main tool to control the system, a good example would be linux/gnu distributions and macOS. However in Windows while installing D language, I noticed that .d source file extension is not associated with neither D compiler (dmd.exe) nor D script interpretator (rdmd.exe) So they can't be ran directly by clicking on those, nor they have any icons that show that these source codes are actually executable. This is a huge problem, because people that are not aware of D language migh be harder to understand that source script could be executable, at least - without help from some more experienced user. If the purpose is to make scripts run by clicking them you can assign a file type to .d files. On Windows 10. https://www.digitaltrends.com/computing/how-to-set-default-programs-and-file-types-in-windows-10/ Doing so would make the script engine the default program instead of a text editor so you might not want to. Or maybe assign .dxe and changing the filename before running. Executable has executable file type for a reason - it self-implies that the main function of the file - is to be executed. The problem with making source code files executable is that, source code files, might not always be monolithic executable code, but a part of a bigger interconnected program. That would lead to partily working program execution, and most of the times guess work of whether source code file was supposed to be executed. The clear alternative would be: .exe file on which, if rightclicked - would open context menu showing posibility of "edit/improve this executable". Which would not create additional problem mentioned above.
Re: Making an .exe that executes source file inside itself.
On Wednesday, 25 April 2018 at 19:19:58 UTC, BoQsc wrote: So there has been idea I've got for around few months now: making a software which executable would contain a source file. A software that anyone could modify by opening an executable and quickly change a few lines of it, rerun an executable, see the changes. Could this be easily possible with D language, considering that sources files can be ran without long slow compilation process? If it makes sense for your project, you could embed a scripting language with LuaD or PyD and rerun the script whenever it changes. Alternatively, I don't know about specifics how to implement it in D, but the key phrase you are looking for is "code hotswap" or "hot loading". It's being popularized right now in gamedev circles, to avoid slow complications of the code. The way to do it is push code into DLLs, then reload the DLL if the change is detected. Of course it's not that simple, because there are some tricky parts around DLL boundaries and you have to reload the entire state, but it is possible. In such case, rather than recompile the entire code, you can just recompile a small subset of functionality to a DLL.
Re: Making an .exe that executes source file inside itself.
On Thursday, 26 April 2018 at 06:18:25 UTC, BoQsc wrote: On Wednesday, 25 April 2018 at 20:44:10 UTC, u0_a183 wrote: On Wednesday, 25 April 2018 at 19:54:26 UTC, BoQsc wrote: On Wednesday, 25 April 2018 at 19:43:31 UTC, Jonathan M Davis wrote: On Wednesday, April 25, 2018 19:19:58 BoQsc via Digitalmars-d-learn wrote: So there has been idea I've got for around few months now: making a software which executable would contain a source file. A software that anyone could modify by opening an executable and quickly change a few lines of it, rerun an executable, see the changes. Could this be easily possible with D language, considering that sources files can be ran without long slow compilation process? The normal way to do that is to just write a script. In the case of D, you can just use rdmd to do it. e.g. if you're on a POSIX system, just put #!/usr/bin/env rdmd at the top of your .d file and chmod it so that it's executable, and it'll run like any other script. - Jonathan M Davis Thank you Jonathan for a response. I was aware of this, and it certainly perfectly works where command line/terminal interface is the main tool to control the system, a good example would be linux/gnu distributions and macOS. However in Windows while installing D language, I noticed that .d source file extension is not associated with neither D compiler (dmd.exe) nor D script interpretator (rdmd.exe) So they can't be ran directly by clicking on those, nor they have any icons that show that these source codes are actually executable. This is a huge problem, because people that are not aware of D language migh be harder to understand that source script could be executable, at least - without help from some more experienced user. If the purpose is to make scripts run by clicking them you can assign a file type to .d files. On Windows 10. https://www.digitaltrends.com/computing/how-to-set-default-programs-and-file-types-in-windows-10/ Doing so would make the script engine the default program instead of a text editor so you might not want to. Or maybe assign .dxe and changing the filename before running. Executable has executable file type for a reason - it self-implies that the main function of the file - is to be executed. The problem with making source code files executable is that, source code files, might not always be monolithic executable code, but a part of a bigger interconnected program. That would lead to partily working program execution, and most of the times guess work of whether source code file was supposed to be executed. This is wrong. All code must be included. All library source code is included. If this is infeasible then simply the binaries are included. It would be no different than dynamic linking that already exists. It doesn't create any new problems. Sure certain things would need to be worked out properly to optimize the experience, but that is obvious. One can't make everything perfect though and one shouldn't expect it... The clear alternative would be: .exe file on which, if rightclicked - would open context menu showing posibility of "edit/improve this executable". Which would not create additional problem mentioned above.
Re: Making an .exe that executes source file inside itself.
On Thursday, 26 April 2018 at 10:06:57 UTC, JN wrote: On Wednesday, 25 April 2018 at 19:19:58 UTC, BoQsc wrote: Alternatively, I don't know about specifics how to implement it in D, but the key phrase you are looking for is "code hotswap" or "hot loading". It's being popularized right now in gamedev circles, to avoid slow complications of the code. The way to do it is push code into DLLs, then reload the DLL if the change is detected. Of course it's not that simple, because there are some tricky parts around DLL boundaries and you have to reload the entire state, but it is possible. In such case, rather than recompile the entire code, you can just recompile a small subset of functionality to a DLL. - content below is just my thoughts and may or may not have anything common with reality, don't take it as truth, forget after reading, proceed on your own risk The overall process is like this: 1) make a core app that basically serves as host (whether or not put domain logic here is open question) 2) when run, scan & plug-in[*] shared libs (they must must follow well defined plugin API, ideally providing information about their requirements/dependencies and their services) 3) when library should be hot reloaded, it first serializes 'shared' data, then re-loaded with same mechanism as in 2), and deserializes data back. [*] - it has implications depending on platform, on Windows for example don't just load the libs(executing one's is locked by OS), but copy to temporary executing location first, that way one can overwrite the original libs while (a copy of) it is still running and app can be notified about file changes to perform hot reload. (AFAIK UnrealEngine does this without temporary location, instead it relies on custom build system that checks whether app is running, which will then append incremental number to the name of the resulting binary, after build is done editor replaces old library with new one. I don't know the exact notification mechanism, it could be the build system itself or just FS event handled by engine which compares known running lib names for such incremental change, both ways makes it harder to manually replace library) The one of the major points of having such libraries, after all, to be able to swap implementation without full rebuild, and better, in run time. For example in case of game engine, let's say we want to switch rendering API(say DirectX to Vulkan), in this case there is even no need to serialize anything because most(if not any) of the stuff is implementation specific, such swap could be done *almost* in real time, I mean minor freeze for few frames doesn't makes a big difference here, and really I doubt anyone would do that in a middle of a game(not in menu or pause). There is still lots of question, after all this is just example in theory, in practice however I've never yet come close enough to the practical implementation of such thing in D. WARNING: ONLY WINDOWS-SPECIFIC STUFF BELOW But now the problem. Guess what? DLL support on Windows is simply isn't there yet! AHAHAHAH (imagine some epic evil laugh here) Sure thing, it is possible to make a bunch of loose functions that operates on built-in (value) types, but does this really provides the extensibility? I can hardly imagine usefulness of such approach in any but trivial cases, just imagine Unity or Unreal Engine, or complex software like 3ds Max or Maya... It is also to some extent possible to go shady with hacks and abuses, but this is something that should not be shipped as production ready, and there is even no guarantee it will work in a future. The last option probably is to implement whole mechanism from scratch, like COM objects, D is a system language after all, but I to be honest don't have the desire to fall that low, especially because on Linux it works for years... (side note: I actually tried few simple (hacky) test cases with passing classes cross-boundary and then unloading library, seems to work on Windows, not sure about memory leaks and consequnces in long runs though)
Re: Making an .exe that executes source file inside itself.
On Friday, 27 April 2018 at 04:30:32 UTC, IntegratedDimensions wrote: On Thursday, 26 April 2018 at 06:18:25 UTC, BoQsc wrote: On Wednesday, 25 April 2018 at 20:44:10 UTC, u0_a183 wrote: On Wednesday, 25 April 2018 at 19:54:26 UTC, BoQsc wrote: On Wednesday, 25 April 2018 at 19:43:31 UTC, Jonathan M Davis wrote: [...] Thank you Jonathan for a response. I was aware of this, and it certainly perfectly works where command line/terminal interface is the main tool to control the system, a good example would be linux/gnu distributions and macOS. However in Windows while installing D language, I noticed that .d source file extension is not associated with neither D compiler (dmd.exe) nor D script interpretator (rdmd.exe) So they can't be ran directly by clicking on those, nor they have any icons that show that these source codes are actually executable. This is a huge problem, because people that are not aware of D language migh be harder to understand that source script could be executable, at least - without help from some more experienced user. If the purpose is to make scripts run by clicking them you can assign a file type to .d files. On Windows 10. https://www.digitaltrends.com/computing/how-to-set-default-programs-and-file-types-in-windows-10/ Doing so would make the script engine the default program instead of a text editor so you might not want to. Or maybe assign .dxe and changing the filename before running. Executable has executable file type for a reason - it self-implies that the main function of the file - is to be executed. The problem with making source code files executable is that, source code files, might not always be monolithic executable code, but a part of a bigger interconnected program. That would lead to partily working program execution, and most of the times guess work of whether source code file was supposed to be executed. This is wrong. All code must be included. All library source code is included. If this is infeasible then simply the binaries are included. It would be no different than dynamic linking that already exists. It doesn't create any new problems. Sure certain things would need to be worked out properly to optimize the experience, but that is obvious. One can't make everything perfect though and one shouldn't expect it... The clear alternative would be: .exe file on which, if rightclicked - would open context menu showing posibility of "edit/improve this executable". Which would not create additional problem mentioned above. Yes, all the source code and required source code to make everything work must be included. And where would all the code mentioned before should be stored? It must be stored in a an single self extracting archive, self executing archive, that is self-implying that it is an executable code, which means it can't be .d file extension, but an .exe file extension. (or any other, if anyone come up with more common or self explaining extension name) If we use the same .d file extension on this self-exectuable, self-extracting archive, it would confused people weather it is executable or some novice programmer's written module/script.
Re: Making an .exe that executes source file inside itself.
On Friday, 27 April 2018 at 14:57:34 UTC, BoQsc wrote: On Friday, 27 April 2018 at 04:30:32 UTC, IntegratedDimensions wrote: On Thursday, 26 April 2018 at 06:18:25 UTC, BoQsc wrote: On Wednesday, 25 April 2018 at 20:44:10 UTC, u0_a183 wrote: On Wednesday, 25 April 2018 at 19:54:26 UTC, BoQsc wrote: On Wednesday, 25 April 2018 at 19:43:31 UTC, Jonathan M Davis wrote: [...] Thank you Jonathan for a response. I was aware of this, and it certainly perfectly works where command line/terminal interface is the main tool to control the system, a good example would be linux/gnu distributions and macOS. However in Windows while installing D language, I noticed that .d source file extension is not associated with neither D compiler (dmd.exe) nor D script interpretator (rdmd.exe) So they can't be ran directly by clicking on those, nor they have any icons that show that these source codes are actually executable. This is a huge problem, because people that are not aware of D language migh be harder to understand that source script could be executable, at least - without help from some more experienced user. If the purpose is to make scripts run by clicking them you can assign a file type to .d files. On Windows 10. https://www.digitaltrends.com/computing/how-to-set-default-programs-and-file-types-in-windows-10/ Doing so would make the script engine the default program instead of a text editor so you might not want to. Or maybe assign .dxe and changing the filename before running. Executable has executable file type for a reason - it self-implies that the main function of the file - is to be executed. The problem with making source code files executable is that, source code files, might not always be monolithic executable code, but a part of a bigger interconnected program. That would lead to partily working program execution, and most of the times guess work of whether source code file was supposed to be executed. This is wrong. All code must be included. All library source code is included. If this is infeasible then simply the binaries are included. It would be no different than dynamic linking that already exists. It doesn't create any new problems. Sure certain things would need to be worked out properly to optimize the experience, but that is obvious. One can't make everything perfect though and one shouldn't expect it... The clear alternative would be: .exe file on which, if rightclicked - would open context menu showing posibility of "edit/improve this executable". Which would not create additional problem mentioned above. Yes, all the source code and required source code to make everything work must be included. And where would all the code mentioned before should be stored? It must be stored in a an single self extracting archive, self executing archive, that is self-implying that it is an executable code, which means it can't be .d file extension, but an .exe file extension. (or any other, if anyone come up with more common or self explaining extension name) If we use the same .d file extension on this self-executable, self-extracting archive, it would confused people weather it is executable or some novice programmer's written module/script. No, it would have to be an exe on windows. an exe is basically a self executing container. Almost all exe's only contain the binary but they can contain data. For example, zip files can be saved as self-unzipping executable. It is not a big deal to package the data in the exe and make it all work. Initially, it is not necessary to worry about. What happens is that the actual initial binary that will run will essentially decide what needs to be done and unpack everything and deal with it. You can think of the exe as a small file system that contains all the files necessary and a binary that handles whatever needs to be done which is run by the OS. Several binaries in the file would be things like the IDE/debugger and the compiled source code. If the user just wants to run the app like any normal app then the compiled source code would be extracted either to a temp exe file or to memory and ran. (so the initial process just does some unpacking and passes control) If debugging and modification is to be done then the initial process will have to do more work such as compile the source with debugging symbols, special hooks, and all that. The files would have to be extracted in memory or on disk. The ide would be ran, etc. For example, android apps are actually special zip files. All of this stuff is childs play. The real work is having a proper design that integrates everything well. Nothing would be technically challenging but just a lot of work to get everything done. I do think that apps that were designed to be used this way would need to be structured and designed in a way that makes it conducive to debugging and modifying easily and quickly.
