Well, so if you want to increase the process virtual memory limits of a 32-bit process on 64-bit windows you can set the LargeAddressAware flag on the executable. While this will not give you access to a 64-bit address space, it will give you access to 4 GB per process less a few megabytes used for the trampoline (since the OS is no longer mapped into the process address space). Of course, unless you want problems the executable needs to be 32-bit clean. That means no "signed" addresses/pointers, and no diddling of the sign (or other) bits in the address field to contain some kind of overloaded information store.
SQLite is 32-bit clean. > -----Original Message----- > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] > On Behalf Of Warren Young > Sent: Thursday, 11 August, 2016 14:38 > To: SQLite mailing list > Subject: Re: [sqlite] 64-bit SQLite3.exe > > On Aug 10, 2016, at 6:32 PM, Keith Medcalf <kmedc...@dessus.com> wrote: > >> You must be talking about PAE, which is an unmitigated hack, in the > >> dirtiest sense of that word > > > > It is not a hack. It is how things work. I do not see where you get > the idea that it is a hack. > > Because I know how PAE works, and I have the technical competence to > express an informed opinion about it. > > But if you don’t want to believe me, maybe you’ll believe Linus Torvalds: > > https://cl4ssic4l.wordpress.com/2011/05/24/linus-torvalds-about-pae/ > > > non-Windows have supported physical address limits beyond 4 GB as > standard since a very long time (Linux since 2009). > > Yes, via PAE. > > If you mean something other than PAE, please give a technical reference to > what you are talking about. Like, maybe, a page in an Intel architecture > reference manual. Even a Wikipedia link would do. > > >> As you hint, some OSes allow individual apps to allocate extra RAM via > PAE > >> — UnixWare was one such — but due to the way PAE works, it can never be > >> more than 8 GiB per process at a given time. > > > > I hinted at no such thing. The original quoted paragraph said "more > than 4 GB of RAM". This has nothing to do with the per-process allocation > which is an artifact of how badly or ill-conceived the Operating System > architecture and the physical implementation of the V:R handling. Whether > the machine and OS can physically address more than 4 GB of physical RAM > has nothing whatsoever to do with the "bitedness" of the OS or the CPU, > only the width of the physical address bus and the translation tables and > hardware. > > You’re describing PAE: > > https://en.wikipedia.org/wiki/Physical_Address_Extension > > PAE required extending an IA-32 processor’s normal 32-bit address bus to > 36 bits, giving a maximum virtual address space of 64 GiB. > > PAE does not change the machine code instructions for accessing memory, > since that would require recompiling everything to allow 36-bit addresses > at the program level. This would require another incompatible Intel > instruction set, as different from IA-32 as IA-64 is. > > If you look at the GCC manuals, you will not find a “PAE mode” flag for > giving a binary with 36-bit addresses, because an IA-32 processor doesn’t > offer that addressing mode. Such a flag would be on this page in the GCC > manual: > > https://gcc.gnu.org/onlinedocs/gcc-6.1.0/gcc/x86-Options.html > > Notice that there is no PAE flag, and no -m36 flag. If you give -m64, you > get IA-64 code, which won’t run on under a 32-bit kernel, even with PAE > enabled. > > You also won’t find such a compiler flag for Visual C++: > > https://msdn.microsoft.com/en-us/library/19z1t1wy.aspx > > Though the OS kernel can use PAE to address more than 4 GiB via a 3-level > TLB scheme — as opposed to the 2-level scheme Intel used before PAE — it > doesn’t let a single program access more than 4 GiB at any given time. > > Since this whole thread is about giving a single program — sqlite3.exe — > access to more RAM, PAE doesn’t solve the OP’s problem. > > You can install 32-bit Windows Server 2012 on a PAE-aware box with 16 GiB > of RAM and run two instances of 32-bit sqlite3.exe on it, but they will > only be able to chew up half the system’s RAM between themselves, no more. > > >> Linux in particular doesn’t let individual applications use PAE to > access > >> more than 3 GiB of VM space, with the standard 3/1 user/kernel split. > >> Instead, if you have more than 4 GiB of RAM in the machine and are > running > >> a PAE kernel, it will let you have multiple programs *collectively* > using > >> more than 4 GiB of VM space. That’s not going the help the OP. > > > > And yet more of the same. You are much confused between "CPU accessing > physical RAM (the :R part)" and "processes accessing virtual RAM (the V: > part). > > Virtual memory still doesn’t solve the OP’s problem. > > You can take a PAE-supporting box with 4 GiB of physical RAM in it, > install a PAE-aware OS on it, then configure that OS for 60 GiB of swap to > give 64 GiB of total virtual memory space, but the individual programs > running under that OS *still* won’t be able to address more than 4 GiB of > virtual memory each, because the memory addressing instructions will > continue to use 32-bit pointers. > > All virtual memory does in a situation like this is allow another program > to come along and grab up to 4 GiB of virtual memory space itself. > > That doesn’t help the OP, who is running a single program — sqlite3.exe — > and needs it to address more than 4 GiB of memory. It doesn’t matter > whether you call it RAM, physical memory, or virtual memory, IA-32 simply > doesn’t solve the OP’s problem, with or without PAE. > > >> Quoting Wikipedia, “…regular application software continues to use > >> instructions with 32-bit addresses and (in a flat memory model) is > limited > >> to 4 gigabytes of virtual address space…no single regular application > can > >> access [all 64 GiB] simultaneously.” > > > > And your point is what exactly? We are not discussing per process > access to virtual memory limitations, but rather artificial physical > memory access limitations. > > I think you might be misunderstanding the quote. Low-level discussions > about an OS’s memory addressing limitations are often couched in terms of > “virtual memory,” which in that context doesn’t refer to swap space or > anything like that. It means the machine’s total addressable memory, both > swap and RAM together. > > If you take my example in the section immediately above, there is only 4 > GiB of physical RAM, but 64 GiB of virtual memory. > > The first program you run on that system won’t be able to access 4 GiB of > physical RAM because the OS kernel will have some of that tied up, but it > *can* allocate 4 GiB of virtual memory, which may be composed partly of > physical RAM and partly of swap space. > > If, while that first program is running, you run a second program on that > machine, it will still get some physical RAM because the OS kernel will > swap some of the first program’s pages out, so that both programs will end > up with considerably less than 4 GiB of physical RAM, yet both could > address 4 GiB of virtual memory at the same time. > > > You will notice that even Microsoft admits that they limit access to > physical RAM purely so they can make more money, and for no other reason > > That’s not true. According to Mark Russinovich, they tried enabling PAE > on consumer OSes during development, and it caused a bunch of poorly- > written hardware drivers to crash, so they gave up on it, rather than try > to force their hardware partners to fix them. > > Microsoft had a much easier problem to solve for the Server class OSes > because the scope of hardware is much smaller, and that hardware tends to > have much better driver support. > > If this were simply a money grab, Microsoft could have enabled PAE only > for Windows Pro or Enterprise editions, but they didn’t even do that. > > The open source OSes don’t have that problem because the projects write > most of the drivers they use, so they’re not beholden to the hardware > providers to fix their drivers for them, as Microsoft is. > > Much the same is true of OS X, which also shipped with a PAE-enabled > kernel, back before they went 64-bit-only, many years ago now. > > >> I believe the situation is essentially the same on PAE-enabled versions > of > >> Windows as on Linux. > > > > Again, you are confused between per process addressing of Virtual Memory > and CPU access of Physical Memory. > > No, I am not confused. It just isn’t relevant to the topic of this thread > that the CPU can access > 4 GiB if individual processes cannot. > sqlite3.exe is, after all, a single process. Famously so. > > >> It is also the case that most machines that shipped with 32-bit Intel > >> processors either didn’t have enough slots to allow > 4 GiB of RAM or > >> didn’t have BIOS/EFI/chipset support for that much RAM if you did have > the > >> slots for it. And why should they have done? It just adds cost with a > >> low chance that the user can make use of it, so that capability only > >> showed up in high-dollar machines. > > > > Nor did cheap motherboards wire the bus master request pin properly to > the expansion bus even though the correct wiring was part of the "IBM > compatible" specification. Just because someone may choose to purchase > low-end product does not mean that the capabilities for bus mastering (or > in the case of the present conversation, accessing more than 4 GB of > physical memory) does not validate your argument. > > Okay, let’s say PAE or some other technology you are imagining is a > solution to the OP’s problem. How is it then irrelevant if it turns out > that a huge chunk of the installed base cannot use it? You can’t expect a > lot of effort to go towards supporting technology that only a tiny slice > of the SQLite user base can use. > > >> PAE’s time is long past. 64-bit is the proper solution today. > > > > What does the size of the data bus and registers have to do with the > amount of addressable memory, either physical or virtual? There are two- > bit (bit slice) processors on the market for the past 30 years which have > exceptionally huge addressable storage (far beyond 4 GB). > > …which has nothing to do with the way the IA-32 instruction set works. > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
