RE: Wanted: mmap or other fast IO

[Simon Marlow]

> On 16-Feb-2000, Simon Marlow <[EMAIL PROTECTED]> wrote:
> > 
> > The easiest way is to declare mmap as a foreign function 
> using foreign
> > import, then build a little wrapper around it.  
> Unfortunately you won't be
> > able to turn the resulting memory into an array (even a 
> ByteArray), since
> > these are assumed to live in GHC's heap
> 
> I'm curious: why do you make that assumption?
> 
> If I write code that defines some big array whose contents are known
> at compile time, it would be nice if ghc would allocate that array
> at compile time, in read-only memory, rather than allocating it on
> the heap at run-time.

No deep reasons I think: it just means augmenting our intermediate language
with the concept of static arrays, and propagating this through the
compiler.

Detecting the static-ness of an array is quite hard though, since there's
lots of abstraction between the original haskell array and the actual
primitives.  Haskell's "array" calls the primitive newArray# to initialise
the array, and then fills it in using foldr over the element list - we'd
have to spot that the contents are static and replace the whole thing with a
static array.

Cheers,
Simon

Re: Wanted: mmap or other fast IO

[Fergus Henderson]

On 16-Feb-2000, Simon Marlow <[EMAIL PROTECTED]> wrote:
> 
> The easiest way is to declare mmap as a foreign function using foreign
> import, then build a little wrapper around it.  Unfortunately you won't be
> able to turn the resulting memory into an array (even a ByteArray), since
> these are assumed to live in GHC's heap

I'm curious: why do you make that assumption?

If I write code that defines some big array whose contents are known
at compile time, it would be nice if ghc would allocate that array
at compile time, in read-only memory, rather than allocating it on
the heap at run-time.

-- 
Fergus Henderson <[EMAIL PROTECTED]>  |  "I have always known that the pursuit
WWW:   |  of excellence is a lethal habit"
PGP: finger [EMAIL PROTECTED]| -- the last words of T. S. Garp.

RE: Wanted: mmap or other fast IO

[Sigbjorn Finne]

Manuel M. T. Chakravarty <[EMAIL PROTECTED]> writes:
> 
> Sven Panne <[EMAIL PROTECTED]> wrote,
> 
> > [ Simply can't resist...  ;-) ]
> > 
> > Simon Marlow wrote:
> > > The easiest way is to declare mmap as a foreign function using
> > > foreign import, then build a little wrapper around it.
> > 
> > A problem will be: What Haskell types should be used for size_t and
> > off_t? Getting this done properly would require autoconf or 
> something
> > similar. Perhaps some more Haskell types (Size, Off, ...) should be
> > added to the FFI module. I'm not sure which ones exactly and if FFI
> > is the right place for it. Opinions?
> 
> I don't think that the standard FFI is the right place for
> that.  If you are programming the FFI by hand, autoconf is
> the right thing to use.  Otherwise, a tool can help.
> C->Haskell can compute the right type by analysing the C
> headers, ie, feeding it `sys/mman.h'.  H/Direct when fed
> `mman.h' might also do the right thing.
> 

I don't see why HDirect shouldn't.

This request is not a new one - having a module that provides
a Haskell mapping for the various 'abstract' types that
ISO/ANSI C defines (and POSIX.1 doesn't.) It was a bit painful
to introduce such a thing at the time (it very easily led
to mutual dep. between the 'posix' and 'misc' syslibs), but
perhaps the new library organization has untied the knot.

--sigbjorn

Re: Wanted: mmap or other fast IO

[Manuel M. T. Chakravarty]

Sven Panne <[EMAIL PROTECTED]> wrote,

> [ Simply can't resist...  ;-) ]
> 
> Simon Marlow wrote:
> > The easiest way is to declare mmap as a foreign function using
> > foreign import, then build a little wrapper around it.
> 
> A problem will be: What Haskell types should be used for size_t and
> off_t? Getting this done properly would require autoconf or something
> similar. Perhaps some more Haskell types (Size, Off, ...) should be
> added to the FFI module. I'm not sure which ones exactly and if FFI
> is the right place for it. Opinions?

I don't think that the standard FFI is the right place for
that.  If you are programming the FFI by hand, autoconf is
the right thing to use.  Otherwise, a tool can help.
C->Haskell can compute the right type by analysing the C
headers, ie, feeding it `sys/mman.h'.  H/Direct when fed
`mman.h' might also do the right thing.

It might of course be nice to provide some libraries in
hslibs/ that define access to some of the often used types
and functions.

Manuel

Re: Wanted: mmap or other fast IO

[Sven Panne]

[ Simply can't resist...  ;-) ]

Simon Marlow wrote:
> The easiest way is to declare mmap as a foreign function using
> foreign import, then build a little wrapper around it.

A problem will be: What Haskell types should be used for size_t and
off_t? Getting this done properly would require autoconf or something
similar. Perhaps some more Haskell types (Size, Off, ...) should be
added to the FFI module. I'm not sure which ones exactly and if FFI
is the right place for it. Opinions?

> [...] but you'll be able to return an Addr and use writeXXXOffAddr/
> readXXXOffAddr to access it(*).

If you are using GHC 4.06 you should better use the Storable class in
the module FFI. For an idea what a slightly higher-level module
layered upon FFI could look like, see

   http://www.informatik.uni-muenchen.de/~Sven.Panne/haskell_libs/ffi/Marshal.hs

or Manuel's

   http://www.informatik.uni-muenchen.de/~Sven.Panne/haskell_libs/ffi/C2HSMarsh.hs

from his C->HS tool. Both should be merged and improved somehow,
suggestions welcome.

> (*) the readXXXOffAddr functions are known to be broken at the
> moment (see recent discussion on ghc-bugs), we're currently working
> on fixing them.

A workaround is not using -O for modules using FFI or readXXXOffAddr.

Cheers,
   Sven
-- 
Sven PanneTel.: +49/89/2178-2235
LMU, Institut fuer Informatik FAX : +49/89/2178-2211
LFE Programmier- und Modellierungssprachen  Oettingenstr. 67
mailto:[EMAIL PROTECTED]D-80538 Muenchen
http://www.informatik.uni-muenchen.de/~Sven.Panne

RE: Wanted: mmap or other fast IO

[Simon Marlow]

Title: RE: Wanted: mmap or other fast IO






> Is there any interface to mmap(2) available? Something that 
> behaves like
> an immutable array would be great.
> 
> An mmap may have a signature like
> 
> mmap :: Ix a, ?? b => Handle -> IO (Array a b)
> 
> I've no idea what types should be allowed for b. It has to be 
> some fixed
> size type, e.g. Int, but this would require the mmapped 
> file's size to be
> a multiple of the size of Ints.


The easiest way is to declare mmap as a foreign function using foreign import, then build a little wrapper around it.  Unfortunately you won't be able to turn the resulting memory into an array (even a ByteArray), since these are assumed to live in GHC's heap, but you'll be able to return an Addr and use writeXXXOffAddr/readXXXOffAddr to access it(*).

Cheers,
    Simon


(*) the readXXXOffAddr functions are known to be broken at the moment (see recent discussion on ghc-bugs), we're currently working on fixing them.