I think there may be a more restrictive limit, at least on Windows. The OS must be able to find a contiguous block of virtual memory, i.e. in pagefile.sys. The paging file may not be able to grow, (depending on how it is configured) and there may not be a large enough block.
I would like to learn more about the exact situation of memory mapping files on Windows -- the above is just based on a hour of Googling and the info below. Non perl related info follows: I hit this limitation in the otherwise excellent disk indexing program Wilbur at http://wilbur.redtree.com which is free (as in beer) and open source too. (But for Windows only.) It uses memory mapped files and when one of the indexes exceeds about 500 MB it says something like "unable to map view of a file" even though my pagefile.sys is 1536 MB. The developer said: This is a system message that occurs when Wilbur is unable to memory map one of its index files and due to the way memory mapping works on Windows, I think this is normally a symptom of insufficient virtual memory space. Possible solutions might be increasing the size of your paging file (dig through the performance options on the system control panel to find this), defragmenting the disk and of course adding more real memory. But I already have a paging file of 1536 MB, which is the recommended size in windows XP (3 times physical memory of 512 MB). I also do not think that defragmenting the disk helps, except possibly if done at boot time to defrag pagefile.sys (However I did do that and it still failed.) I was able to work around the problem by putting fewer words in the index, e.g. kept the default of min length = 3 and no numbers. I also have lots more files than most people, so I think very few people will hit this limitation. Hopefully helpfully yours, Steve -- Steve Tolkin Steve . Tolkin at FMR dot COM 617-563-0516 Fidelity Investments 82 Devonshire St. V4D Boston MA 02109 There is nothing so practical as a good theory. Comments are by me, not Fidelity Investments, its subsidiaries or affiliates. -----Original Message----- From: Gyepi SAM [mailto:[EMAIL PROTECTED] Sent: Friday, November 12, 2004 7:39 AM To: [EMAIL PROTECTED] Subject: Re: [Boston.pm] transposing rows and columns in a CSV file On Fri, Nov 12, 2004 at 02:11:37AM -0500, Aaron Sherman wrote: > Seriously, while mmap is ideal in C, in Perl I would just build an array > of tell()s for each line in the file and then walk through the lines, > storing the offset of the last delimiter that I'd seen. I think mmap would be just as ideal in Perl and a lot less work too. Rather than indexing and parsing a *large* file, you must mmap and parse it. In fact, the CSV code, which was left as an exercise in you pseudo-code, would be the only code required. I should point out though that mmap has a 2GB limit on systems without 64bit support. Such systems can't store files larger than that anyhow. > Let the kernel file buffer do your heavy lifting for you. Exactly, if s/kernel file/mmap/ -Gyepi -- The convenient method is insecure and the secure method is inconvenient. --me _______________________________________________ Boston-pm mailing list [EMAIL PROTECTED] http://mail.pm.org/mailman/listinfo/boston-pm _______________________________________________ Boston-pm mailing list [EMAIL PROTECTED] http://mail.pm.org/mailman/listinfo/boston-pm
