Re: using mmap on large (> 2 Gig) files

"[EMAIL PROTECTED]" <[EMAIL PROTECTED]> writes: > Martin v. Löwis wrote: > > sturlamolden schrieb: > > > > > And why doesn't Python's mmap take an offset argument to handle large > > > files? > > > > I don't know exactly; the most likely reason is that nobody has > > contributed code to make it su

Re: using mmap on large (> 2 Gig) files

Martin v. Löwis wrote: > sturlamolden schrieb: > > > And why doesn't Python's mmap take an offset argument to handle large > > files? > > I don't know exactly; the most likely reason is that nobody has > contributed code to make it support that. That's, in turn, probably > because nobody had the pr

Re: using mmap on large (> 2 Gig) files

Paul Rubin writes: > I mean just have an interface to OS locks (Linux futex and whatever > the Windows counterpart is) and maybe also a utility function to do a > compare-and-swap in user space. There is code for spinlocks, but it allocates the lockword in the process me

Re: using mmap on large (> 2 Gig) files

Chetan <[EMAIL PROTECTED]> writes: > > Why on would you think that?! It is counterintuitive. fseek beyond > > whatever is buffered in stdio (usually no more than 1kbyte or so) > > requires a system call, while mmap is just a memory access. > And the buffer copy required with every I/O from/to the

Re: using mmap on large (> 2 Gig) files

Paul Rubin writes: > "sturlamolden" <[EMAIL PROTECTED]> writes: >> However, "memory mapping" a file by means of fseek() is probably more >> efficient than using UNIX' mmap() or Windows' >> CreateFileMapping()/MapViewOfFile(). > > Why on would you think that?! It is coun

Re: using mmap on large (> 2 Gig) files

"sturlamolden" <[EMAIL PROTECTED]> writes: > However, "memory mapping" a file by means of fseek() is probably more > efficient than using UNIX' mmap() or Windows' > CreateFileMapping()/MapViewOfFile(). Why on would you think that?! It is counterintuitive. fseek beyond whatever is buffered in std

Re: using mmap on large (> 2 Gig) files

"sturlamolden" <[EMAIL PROTECTED]> wrote: > >However, "memory mapping" a file by means of fseek() is probably more >efficient than using UNIX' mmap() or Windows' >CreateFileMapping()/MapViewOfFile(). My goodness, do I disagree with that! At least on Windows, I/O on a file mapped with MapViewOfFil

Re: using mmap on large (> 2 Gig) files

sturlamolden schrieb: > 2. The OS may be stupid. Mapping a large file may be a major slowdown > simply because the memory mapping is implemented suboptimally inside > the OS. For example it may try to load and synchronise huge portions of > the file that you don't need. Can you give an example of

Re: using mmap on large (> 2 Gig) files

sturlamolden wrote: [...] > This was an extremely stupid question on my side. I take my hat off to anyone who's prepared to admit this. We all do it, but most of us try to ignore the fact. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http:/

Re: using mmap on large (> 2 Gig) files

Martin v. Löwis wrote: Your news server doesn't honour cancel as well... > It doesn't need to, why do you think it does? This was an extremely stupid question on my side. It needs to be flushed after a write because that's how the memory pages mapping the file is synchronized with the file. Wri

Re: using mmap on large (> 2 Gig) files

Donn Cave wrote: > Wow, you're sure a wizard! Most people would need to look before > making statements like that. I know, but your news-server doesn't honour cancel messages. :) Python's mmap does indeed memory map the file into the process image. It does not fake memory mapping by means of fi

Re: using mmap on large (> 2 Gig) files

Martin v. Löwis wrote: > You know this isn't true in general. It is true for a 32-bit address > space only. Yes, but there are two other aspects: 1. Many of us use 32-bit architectures. The one who wrote the module should have considered why UNIX' mmap and Windows' MapViewOfFile takes an offset

Re: using mmap on large (> 2 Gig) files

sturlamolden schrieb: > A patch would involve an new object, say, "mmap.mmap2" that thakes the > additional offeset parameter. I don't want it to break any code > dependent on the existing "mmap.mmap" object. Also, I think mmap.mmap2 > should allow the file object to be None, and in that case retu

Re: using mmap on large (> 2 Gig) files

Fredrik Lundh wrote: > > to large to fit in a 32 address space. Thus, mmapmodule.c needs to be > > fixed before it can be used for large files. > > if you've gotten that far, maybe you could come up with a patch, instead > of stating that someone else "needs to fix it" ? I did not say "someone e

Re: using mmap on large (> 2 Gig) files

sturlamolden schrieb: > Looking at Python's source (mmapmodule.c), it seems that "mmap.mmap" > always sets the offset argument in Windows MapViewOfFile and UNIX to 0. > This means that it is always mapping from the beginning of the file. > Thus, Python's mmap module is useless for large files. This

Re: using mmap on large (> 2 Gig) files

sturlamolden schrieb: > Martin v. Löwis wrote: > >> What architecture are you on? On a 32-bit architecture, it's likely >> impossible to map in 2GiB, anyway (since it likely won't fit into >> the available address space). > > Indeed. But why does Python's memory mapping need to be flushed? It d

Re: using mmap on large (> 2 Gig) files

In article <[EMAIL PROTECTED]>, "sturlamolden" <[EMAIL PROTECTED]> wrote: ... > It seems that Python does take a length argument, but not an offset > argument (unlike the Windows' CreateFileMapping/MapViewOfFile and UNIX' > mmap), so you always map from the beginning of the file. Of course if > yo

Re: using mmap on large (> 2 Gig) files

sturlamolden wrote: > Looking at Python's source (mmapmodule.c), it seems that "mmap.mmap" > always sets the offset argument in Windows' MapViewOfFile and UNIX' > mmap to 0. This means that it is always mapping from the beginning of > the file. Thus, Python's mmap module is useless for large files

Re: using mmap on large (> 2 Gig) files

Well, compiling Python 2.5 on Solaris 10 on an x86 is no walk in the park. pyconfig.h seems to think SIZEOF_LONG is 4 and I SEGV during my build, even after modifying the Makefile and pyconfig.h. Mathew Martin v. Löwis wrote: > [EMAIL PROTECTED] schrieb: > > Anyone ever done this? It looks like P

Re: using mmap on large (> 2 Gig) files

[EMAIL PROTECTED] wrote: > Hi > Anyone ever done this? It looks like Python2.4 won't take a length arg > > 2 Gig since its not seen as an int. Looking at Python's source (mmapmodule.c), it seems that "mmap.mmap" always sets the offset argument in Windows' MapViewOfFile and UNIX' mmap to 0. This m

Re: using mmap on large (> 2 Gig) files

[EMAIL PROTECTED] wrote: > Hi > Anyone ever done this? It looks like Python2.4 won't take a length arg > > 2 Gig since its not seen as an int. Looking at Python's source (mmapmodule.c), it seems that "mmap.mmap" always sets the offset argument in Windows MapViewOfFile and UNIX to 0. This means th

Re: using mmap on large (> 2 Gig) files

[EMAIL PROTECTED] wrote: > Hi > Anyone ever done this? It looks like Python2.4 won't take a length arg > > 2 Gig since its not seen as an int. Lookin at Python's source (mmapmodule.c), it seems that "mmap.mmap" always sets the offset argument in Windows MapViewOfFile and UNIX to 0. This means tha

Re: using mmap on large (> 2 Gig) files

Martin v. Löwis wrote: > What architecture are you on? On a 32-bit architecture, it's likely > impossible to map in 2GiB, anyway (since it likely won't fit into the > available address space). Indeed. But why does Python's memory mapping need to be flushed? And why doesn't Python's mmap take an

Re: using mmap on large (> 2 Gig) files

[EMAIL PROTECTED] wrote: > Anyone ever done this? It looks like Python2.4 won't take a length arg http://docs.python.org/lib/module-mmap.html It seems that Python does take a length argument, but not an offset argument (unlike the Windows' CreateFileMapping/MapViewOfFile and UNIX' mmap), so you

Re: using mmap on large (> 2 Gig) files

Martin v. Löwis wrote: > [EMAIL PROTECTED] schrieb: >> Anyone ever done this? It looks like Python2.4 won't take a length arg >>> 2 Gig since its not seen as an int. > > What architecture are you on? On a 32-bit architecture, it's likely > impossible to map in 2GiB, anyway (since it likely won't

Re: using mmap on large (> 2 Gig) files

[EMAIL PROTECTED] schrieb: > Anyone ever done this? It looks like Python2.4 won't take a length arg >> 2 Gig since its not seen as an int. What architecture are you on? On a 32-bit architecture, it's likely impossible to map in 2GiB, anyway (since it likely won't fit into the available address sp

using mmap on large (> 2 Gig) files

Hi Anyone ever done this? It looks like Python2.4 won't take a length arg > 2 Gig since its not seen as an int. Mathew -- http://mail.python.org/mailman/listinfo/python-list