Re: Updates to FFI spec
Alastair: (In the image processing example, images were megabytes and an expression like (x + (y * mask)) would generate 2 intermediate images (several megabytes) while doing just 2 reductions in Haskell.) Marcin: OCaml allows the programmer to specify an approximate amount of foreign memory in its wrapped C pointers. Maybe it's a good idea? We did that on the C/C++ side of things so that we could keep track of how much image memory was in use and try to trigger a GC when the amount of memory was significantly higher than the running average (or some such). I wonder why OCaml would do it on the Caml side? Maybe they include similar heuristics but as a standard part of the system instead of leaving it to each library writer to roll their own? Or could it control the amount of GC performed? If you know that all the objects you care about are in the youngest generation, you could decide to collect just the youngest generation. (Except that isn't always enough - releasing an object in the old generation might release a pointer into the new generation.) -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
Alastair Reid wrote: [snip] Region-based systems have the quite wonderful property that garbage is disposed of promptly - you don't have to wait for the next GC for the memory to be released. Which means that performGC becomes a nullop. [snip] This is not entirely true. Firstly, some region-based systems could use stacks of some sort, which might mean hanging onto something recognised as garbage until it is possible to pop the stack. Secondly, the user may think that something is garbage at the time of performGC, although the compiler is not yet able to prove that it is. I agree with those who regard it as extremely undesirable to rely on the garbage-collection algorithm to carry out important actions (like closing a window or a file), if it matters at all when they happen. I think performGC should be seen more as something like C's register storage attribute, which is a friendly hint to the compiler but nothing more. Manuel's new wording (except for the typo advices instead of advises) expresses this perfectly, so there probably isn't much point in discussing this further. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
Do you want a stronger wording on what kind of garbage collection is to be performed or do we want to keep it deliberately unspecified (ie, leave it to the individual Haskell system)? It'd be nice to say that it has to be a full GC - but I've no idea how to specify that in a non-operational (i.e., implementation dependent) way. We could insert the word 'full' and leave it to people's imaginations? -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
Alastair wrote about performGC (snipped) It'd be nice to say that it has to be a full GC - but I've no idea how to specify that in a non-operational (i.e., implementation dependent) way. I certainly don't think you should constrain implementations to be able to perform a full GC in any sense. It is possible if unlikely that someone will get along to implementing the HaskellKit, where GC is entirely dispensed with and replaced by region analysis. Even if that's not the case, I certainly hope that one of these days someone will get along to implementing a Haskell compiler that does at least some easy region analysis. Also there are probably hard-real-time GC algorithms (like Baker's treadmill) or algorithms which are close to being hard-real-time (like the train algorithm) where doing a full GC would be a major pain. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
Alastair wrote about performGC (snipped) It'd be nice to say that it has to be a full GC - but I've no idea how to specify that in a non-operational (i.e., implementation dependent) way. George Russell [EMAIL PROTECTED] writes: I certainly don't think you should constrain implementations to be able to perform a full GC in any sense. It is possible if unlikely that someone will get along to implementing the HaskellKit, where GC is entirely dispensed with and replaced by region analysis. Even if that's not the case, I certainly hope that one of these days someone will get along to implementing a Haskell compiler that does at least some easy region analysis. Region-based systems have the quite wonderful property that garbage is disposed of promptly - you don't have to wait for the next GC for the memory to be released. Which means that performGC becomes a nullop. [That said, I'm not sure that region-based analysis is all that easy in the presence of lazy evaluation since it is so critically tied to estimating lifetimes.] Also there are probably hard-real-time GC algorithms (like Baker's treadmill) or algorithms which are close to being hard-real-time (like the train algorithm) where doing a full GC would be a major pain. The desired property is that the runtime system releases all unreachable objects. To make the fine-grained GCs (i.e., those that do a little GC now and then instead of a full GC) do this, all you need is stop mutating or allocating objects and then keep invoking the GC until it gets round to where it was when you started. There's usually some kind of colouring or 'epoch' mechanism that lets you know when you're back where you started. Of course, you trash any real-time properties in the process - see recent mail by me in the archive or my old paper for ideas on how to lessen that or dream up your own ideas for how to make two real time GCs work together nicely. [Again, though, I'm not sure there's much danger of a RT GC being added to Haskell - it's too hard estimating execution time, memory usage, etc. so making your GC more predictable doesn't seem like a major win.] In short, I think the current design is perfectly adequate for interfacing current systems to C code and will extend nicely in the future as motivating examples make actual needs. For example, even if we had a finer grained version of performGC, I expect most people would find that performGC provided the functionality that they need so they'd write their own (using a finer-grained interface) if we didn't provide it for them - only those who have real time programs interfacing to Haskell would make use of the finer-grained interface. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
Alastair Reid [EMAIL PROTECTED] wrote,
George Russell [EMAIL PROTECTED] writes:
Also there are probably hard-real-time GC algorithms (like Baker's
treadmill) or algorithms which are close to being hard-real-time
(like the train algorithm) where doing a full GC would be a major
pain.
The desired property is that the runtime system releases all
unreachable objects.
I like that phrase, so I put
Finally, \code{hs\_perform\_gc()} advices the Haskell
storage manager to perform a garbage collection, where the
storage manager makes an effort to releases all
unreachable objects. This function must not be invoked
from C functions that are imported \code{unsafe} into
Haskell code nor may it be used from a finalizer.
into the spec. It's signals the intent, but still leaves an
implementation some freedom.
Manuel
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
I think the thing to do is add the existing performGC to the standard
(perhaps giving it an hs_ prefix in the process) and leave development
of an extended version of the function for when the GHC folk (or
anyone else with a generational collector) decide they want a
forcefulness argument. Come that day, we'd define:
void performGC(void) { performPartialGC(0); }
(or whatever it is you do to force a full collection).
A
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: hs_init() friends
Alastair Reid [EMAIL PROTECTED] writes: So, my proposal is to: [...] I think only GHC implements anything like this (correct me if wrong, Malcolm) and they haven't used it in the way John Meacham is interested in. At the moment, nhc98 provides a routine void haskellInit (int argc, char **argv) which collects the command-line args intended for (1) the nhc98 runtime system, and (2) the response to System.getArgs. It does not strip any arguments from the given set, on the basis that the calling C routine may be interested in the same arguments that nhc98 is interested in. Also, if the controlling C routine wants to alter the argument set before Haskell sees it, that is perfectly catered for by this interface. Regards, Malcolm P.S. The nhc98 internal names will change shortly to match those defined by the FFI spec. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Updates to FFI spec
On 12-Aug-2002, Simon Marlow [EMAIL PROTECTED] wrote: I'd be equally happy (perhaps happier) if the header file spec was removed altogether. In a sense, this would leave the Haskell part of a foreign binding even more portable, because it doesn't have to specify the names of header files which might change between platforms. This is a C interface we're talking about, right? In C, the name of the header file is part of the API. It doesn't change between different platforms unless the API changes. Specifying the header name is essential if Haskell implementations are to ever apply any type-checking to these foreign interfaces. If they don't, then in practice I think Haskell programs using the FFI are likely to be less portable, and certainly more error-prone, since they will contain type errors that may cause problems on one platform but not another. Specifying the header name is also essential for certain implementations (eg. GHC). I wan't suggesting not supplying the header file at all, just not supplying it in the foreign declaration and not defining it as part of the standard. But I take your point about the header file(s) being a proper part of the API. Cheers, Simon ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Updates to FFI spec
System.Mem.performGC does a major GC. When would a partial GC be enough? I've described the image-processing example a bunch of times. We have an external resource (e.g., memory used to store images) which is somewhat abundant and cheap but not completely free (e.g., eventually you start to swap). It is used up at a different rate than the Haskell heap so Haskell GCs don't occur at the right times to keep the cost low and we want to trigger GCs ourselves. Hmmm, the garbage collector is a black box and has its own complicated heuristics for managing memory usage, but you are describing a mechanism that depends rather heavily on certain assumed behaviours. At the least, that gives the garbage collector less flexibility to change its own behaviour, lest it invalidate the assumptions made by the external allocator. (In the image processing example, images were megabytes and an expression like (x + (y * mask)) would generate 2 intermediate images (several megabytes) while doing just 2 reductions in Haskell.) I think I'd be tempted to try to use a more predictable allocation scheme given the size of the objects involved. Perhaps arenas? How often and how hard should we GC? We can't do a full GC too often, or we'll spend a lot of time GCing, destroy our cache and cause premature promotion of Haskell objects into the old generation which will make the GC behave poorly. So if all we can do is a full GC, we'll GC rarely and use a lot of the external resource. Suppose we could collect just the allocation arena. That would be much less expensive (time taken, effect on caches, confusion of object ages) but not always effective. It would start out cheap and effective but more and more objects would slip into older generations and have to wait for a full GC. To achieve any desired tradeoff between GC cost and excess resource usage, we want a number of levels of GC: gc1, gc2, gc3, gc4, ... Each one more effective than the last and each one more expensive than the last. We'll use gc1 most often, gc2 less often, gc3 occasionally, gc4 rarely, ... But there seems to be no way to reasonably decide how often one should call these. Doesn't it depend on the garbage collector's own parameters too? I think the spec should be clarified along these lines: Header files have no impact on the semantics of a foreign call, and whether an implementation uses the header file or not is implementation-defined. Some implementations may require a header file which supplies a correct prototype for the function in order to generate correct code. I still don't like the fact that compilers are free to ignore header files. Labelling it an error instead of a change in semantics doesn't affect the fact that portability is compromised. I don't see any alternative - would you require a compiler that has only a native code generator to read header files? When there's no C compiler on the system? (this is realistic - at some point we'd like to make the via-C route in GHC completely optional, so we can ship a compiler on Windows that doesn't need to be bundled with GCC). Perhaps on GHC you should be required to register the top module in your program first, maybe something like registerModule(__stginit_Main); that way you can register multiple modules (which isn't possible at the moment, you have to have another module which imports all the others). What does that do? Is it for threading, GC, profiling, ...? Each module has a little initialisation fragment that calls all the initialisation fragments for the modules it imports. At the moment, there are two kinds of initialisation done for each module: - each foreign export is registered as a stable pointer. This prevents the garbage collector from collecting any CAFs which might be required (indirectly) by a foreign export. - when profiling, all the cost centres in the current module are initialised. It might be possible to do this using linker sets, but I haven't tried (and it would probably be highly non-portable too). Cheers, Simon ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec
At the moment, there are two kinds of initialisation done for each module: Both ELF and DLLs on Windows provide a way of specifying initializers. Or, easier yet, since the user is already using the hs_init function, you could use that. The way you'd do that in ELF is to define a special section 'hs_initializers'. Every module would contain a single object in that section: the address of the initializer. (In gcc you do this by attaching an attribute to the variable. In asm it is even easier since assemblers directly expose sections to the programmer.) The linker will do what it does with all sections: concatenate the pieces from all object files. hs_init would treat the section as an array of function pointers. I'm sure that the Windows linker must have a similar mechanism - the main trick is figuring out what name to give the sections. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Updates to FFI spec
At the moment, there are two kinds of initialisation done for each module: Both ELF and DLLs on Windows provide a way of specifying initializers. Or, easier yet, since the user is already using the hs_init function, you could use that. The way you'd do that in ELF is to define a special section 'hs_initializers'. Every module would contain a single object in that section: the address of the initializer. (In gcc you do this by attaching an attribute to the variable. In asm it is even easier since assemblers directly expose sections to the programmer.) The linker will do what it does with all sections: concatenate the pieces from all object files. hs_init would treat the section as an array of function pointers. I'm sure that the Windows linker must have a similar mechanism - the main trick is figuring out what name to give the sections. That's what I meant by a linker set :-) Cheers, Simon ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec
Alastair Reid [EMAIL PROTECTED] wrote,
For those not on the cvs mailing list:
I've applied all the changes discussed over the last 2 moniths that
received some support and no dissent.
Changes since RC5:
* Author list: changed Alastair Reid's institution
* 4.1.1: Removed [lib] from impent syntax and discussion
[..]
* 4.1.4: Removed all mention of library objects
Is SimonPJ ok with that? We added [lib] for him (and .NET).
You will need this file:
http://www.cse.unsw.edu.au/~chak/haskell/grammar.sty
to build it. (I came close to adding this file to the repo but
figured that Manuel must have a reason for not having done so
himself.)
I wanted to change some stuff first and not track it in two
CVS repos.
- I'd like to see a standard way to call the GC from C
http://www.mail-archive.com/ffi@haskell.org/msg00565.html
Note that Hugs and GHC have had this for ages except that we call the
function 'performGC' and there's no way to control how many generations
are collected.
I don't have a strong opinion on this one.
- I see the question of Function prototypes as a portability problem
waiting to happen. Either Hugs and GHC are right (you should use the
user-supplied header file or NHC is right (you should ignore the
header file). They can't both be right if we want portable code
so the report should be clear about which one is right.
(Given my druthers, I'd drop header files from the foreign import syntax
and say that you have to specify it on the command line or propose that
we standardize some variant of the GHCism {-# -include foo.h #-}. But
I'm not excited enough about it to push hard for this.)
I am still in favour of user-supplied header files and the
mechanism as it is defined in the spec right now.
- Changes to hs_init
http://www.mail-archive.com/ffi@haskell.org/msg00539.html
I am ok with that. Currently, there is a problem with the
version that is in the spec and GHC in that GHC requires an
extra argument to initialise modules. So, it all depends a
bit on how far SimonM thinks its implementable.
Cheers,
Manuel
PS: Sorry for my prolonged procrastination over these issues
and thanks for picking them up.
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec
- I'd like to see a standard way to call the GC from C
http://www.mail-archive.com/ffi@haskell.org/msg00565.html
Note that Hugs and GHC have had this for ages except that we call
the function 'performGC' and there's no way to control how many
generations are collected.
I don't have a strong opinion on this one.
Maybe SimonM and Malcolm do.
I know GHC has an interface already and I suspect that NHC will too.
GHC and Hugs both call it performGC but this doesn't match the naming
convention used in the ffi and it is expensive to use with
generational GC (since there is no way to indicate that a partial GC
may be enough).
- I see the question of Function prototypes as a portability
problem waiting to happen. Either Hugs and GHC are right (you
should use the user-supplied header file or NHC is right (you
should ignore the header file). They can't both be right if we
want portable code so the report should be clear about which one is
right.
(Given my druthers, I'd drop header files from the foreign import
syntax and say that you have to specify it on the command line or
propose that we standardize some variant of the GHCism {-# -include
foo.h #-}. But I'm not excited enough about it to push hard for
this.)
I am still in favour of user-supplied header files and the mechanism
as it is defined in the spec right now.
Malcolm and I both found it possible to interpret the spec as meaning
that the header files could be ignored. If that isn't what the report
means, the wording should be strengthened.
[My concern with specifying header files in foreign imports is twofold:
1) It suggests the existence of multiple namespaces when I very much
doubt that anyone would every implement that.
2) It is awfully repetitive because you have to mention the header files
on every foreign import just in case anyone ever did implement
multiple namespaces. (Other side of the same problem.)
I'm not arguing for a change here - just saying that I wouldn't argue
against one.]
- Changes to hs_init
http://www.mail-archive.com/ffi@haskell.org/msg00539.html
I am ok with that. Currently, there is a problem with the version
that is in the spec and GHC in that GHC requires an extra argument
to initialise modules. So, it all depends a bit on how far SimonM
thinks its implementable.
Hugs doesn't have anything resembling hs_init at the moment so I can't
say for sure. I think it can be done as a thin layer on top of the
the existing spec though so I don't think it's that hard.
--
Alastair Reid [EMAIL PROTECTED]
Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Updates to FFI spec
For those not on the cvs mailing list:
I've applied all the changes discussed over the last 2 moniths that
received some support and no dissent.
Changes since RC5:
* Author list: changed Alastair Reid's institution
* 4.1.1: Removed [lib] from impent syntax and discussion
* 4.1.3: Added parentheses round FunPtr ft to make it easier to
understand a tolerably complex type.
* 4.1.4: Removed all mention of library objects
* 6: Specified that HsBool==int in table2
Relabelled column 1 in table 3 (C symbol - CPP symbol)
Replaced 0 and 1 with HS_BOOL_FALSE/TRUE
You will need this file:
http://www.cse.unsw.edu.au/~chak/haskell/grammar.sty
to build it. (I came close to adding this file to the repo but
figured that Manuel must have a reason for not having done so
himself.)
Changes not applied:
- I really, really want to resolve the ForeignPtr issues soon.
http://www.mail-archive.com/ffi@haskell.org/msg00655.html
http://www.mail-archive.com/ffi@haskell.org/msg00544.html
http://www.mail-archive.com/ffi@haskell.org/msg00545.html
- I'd like to see a standard way to call the GC from C
http://www.mail-archive.com/ffi@haskell.org/msg00565.html
Note that Hugs and GHC have had this for ages except that we call the
function 'performGC' and there's no way to control how many generations
are collected.
- I see the question of Function prototypes as a portability problem
waiting to happen. Either Hugs and GHC are right (you should use the
user-supplied header file or NHC is right (you should ignore the
header file). They can't both be right if we want portable code
so the report should be clear about which one is right.
(Given my druthers, I'd drop header files from the foreign import syntax
and say that you have to specify it on the command line or propose that
we standardize some variant of the GHCism {-# -include foo.h #-}. But
I'm not excited enough about it to push hard for this.)
- Changes to hs_init
http://www.mail-archive.com/ffi@haskell.org/msg00539.html
--
Alastair Reid [EMAIL PROTECTED]
Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
