On 02/27/2015 11:00 AM, alister wrote:
On Sat, 28 Feb 2015 01:22:15 +1100, Chris Angelico wrote:If you're trying to use the pagefile/swapfile as if it's more memory ("I have 256MB of memory, but 10GB of swap space, so that's 10GB of memory!"), then yes, these performance considerations are huge. But suppose you need to run a program that's larger than your available RAM. On MS-DOS, sometimes you'd need to work with program overlays (a concept borrowed from older systems, but ones that I never worked on, so I'm going back no further than DOS here). You get a *massive* complexity hit the instant you start using them, whether your program would have been able to fit into memory on some systems or not. Just making it possible to have only part of your code in memory places demands on your code that you, the programmer, have to think about. With virtual memory, though, you just write your code as if it's all in memory, and some of it may, at some times, be on disk. Less code to debug = less time spent debugging. The performance question is largely immaterial (you'll be using the disk either way), but the savings on complexity are tremendous. And then when you do find yourself running on a system with enough RAM? No code changes needed, and full performance. That's where virtual memory shines. ChrisAI think there is a case for bringing back the overlay file, or at least loading larger programs in sections only loading the routines as they are required could speed up the start time of many large applications. examples libre office, I rarely need the mail merge function, the word count and may other features that could be added into the running application on demand rather than all at once. obviously with large memory & virtual mem there is no need to un-install them once loaded.
I can't say how Linux handles it (I'd like to know, but haven't needed to yet), but in Windows (NT, XP, etc), a DLL is not "loaded", but rather mapped. And it's not copied into the swapfile, it's mapped directly from the DLL. The mapping mode is "copy-on-write" which means that read=only portions are swapped directly from the DLL, on first usage, while read-write portions (eg. static/global variables, relocation modifications) are copied on first use to the swap file. I presume EXE's are done the same way, but never had a need to know.
If that's the case on the architectures you're talking about, then the problem of slow loading is not triggered by the memory usage, but by lots of initialization code. THAT's what should be deferred for seldom-used portions of code.
The main point of a working-set-tuner is to group sections of code together that are likely to be used together. To take an extreme case, all the fatal exception handlers should be positioned adjacent to each other in linear memory, as it's unlikely that any of them will be needed, and the code takes up no time or space in physical memory.
Also (in Windows), a DLL can be pre-relocated, so that it has a preferred address to be loaded into memory. If that memory is available when it gets loaded (actually mapped), then no relocation needs to happen, which saves time and swap space.
In the X86 architecture, most code is self-relocating, everything is relative. But references to other DLL's and jump tables were absolute, so they needed to be relocated at load time, when final locations were nailed down.
Perhaps the authors of bloated applications have forgotten how to do these, as the defaults in the linker puts all DLL's in the same location, meaning all but the first will need relocating. But system DLL's are (were) each given unique addresses.
On one large project, I added the build step of assigning these base addresses. Each DLL had to start on a 64k boundary, and I reserved some fractional extra space between them in case one would grow. Then every few months, we double-checked that they didn't overlap, and if necessary adjusted the start addresses. We didn't just automatically assign closest addresses, because frequently some of the DLL's would be updated independently of the others.
-- DaveA -- https://mail.python.org/mailman/listinfo/python-list
