RE: extended foreign decls
> > > I refrained from proposing this in an earlier mail, but I've been > > > thinking about it for a long time. I *really* *really* want to be > > > able to insert small snippets of foreign code into the source of > > > a Haskell module. I *really* *really* don't want this. The current Hugs implementation of ffi could readily accomodate it because it generates a C file which you compile in the normal way but it would forever rule out an improved implementation (for use on especially well supported architectures) which doesn't need a compiler at all. One could argue that it'd be ok to support it as an optional extension on just some platforms and people would just have to be careful to use the portable subset when appropriate. Anyone who has tried to compile Linux code (intended for gcc) on some other compiler knows that people don't restrict themselves to the portable subset and, after a while, forget whether they're using extensions at all. In other words, I'd like to keep all ffi implementations identical: no non-portable extensions. [Note that the same portability argument applies to supporting unboxed pointers in the ffi:not portable and not necessary.] > Doing it in a separate tool will lose efficiency in some important > cases. If the compiler is compiling via C, then it can insert > inline C code directly in the generated code, and thus get > inlining. But I think a separate tool would have to put the C code in > a separate C file, which would prevent inlining. My group at University of Utah will be releasing "Knit" in a week or two. Knit is mostly an adaptation/implementation of Matthew Flatt's component programming model for C (see his PLDI'98 paper). But Knit can also merge multiple C files into one file, flag functions for inlining (using a simple cost model to avoid exponential code size increase), and sort function definitions so that definitions come before uses (required by gcc's inliner). If you're really that obsessed with performance, you could use Knit :-) [Knit is, of course, written in Haskell.] Or, if you want a lower-tech solution, you could define all those little functions as inline functions in a .h file and use whatever flags or directives nhc/ffi provides to say "please #include this .h file". With the latter approach, we might want to tweak GreenCard and friends to optionally flag all the internal C functions it generates as "static inline" to enhance inlining when used this way. The "static" annotation would have to be optional (i.e., controlled by an ifdef) because we'd also want to be able to compile the file by itself. Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: freeHaskellFunctionPtr
> > I would let freeHaskellFunPtr accept and ignore nullFunPtr, since I > > once had to write > > when (ptr /= nullFunPtr) $ freeHaskellFunPtr ptr > > and it can be seen as consistent with free. This sort of behaviour is comon in C but I think this is mostly because C has such a weak type system. In the Win32 library, we took some care to distinguish between values which couldn't possibly be null (whether null pointers, null window handles, null font handles or whatever) using definitions like this newtype Window = Window Addr newtype Font = Font Int32 and values which could possibly be null: type MbWindow = Maybe Window type MbFont = Maybe Font This additional information made it easier to understand function types (because a bit more of the C documentation had migrated into the Haskell type), caught programming errors early, etc. This approach wouldn't be feasible in C because C's type system isn't expressive enough to express (and check!) these types. (Well, maybe you could use unions but it would be so clumsy as to be unusable.) Which is all just my long-winded way of saying I agree with Simon that we should not do this. Even if f.e.d. could return nullFunPtr, we should not do this because we can express it better using a Maybe. Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Typing f.e.d.
> Yes, Addr is dead. So, allowing Addr in f.e.d., or anywhere > else for that matter, can only be for reasons of temporary > backward compatibility. Please note that this temporary backward compatability happens to be an important part of most ffi code currently in existence (at least, all the code that I know and/or care about). I guess you all know this but I just want to make sure that you don't break working code in the rush to move forwards. Alastair ps I have a feeling of feeping creaturism as I watch this list - a nice simple design seems to be getting more complex. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Modification to foreign import/export
> This has the advantage that when the library name is #ifdefed, the > conditional needs not to be repeated for each function. Similarly > for #included header names. Note that this can be done without separating the library name from the ffi decl: #ifdef UNIX # define LIBNAME "foo.so" #else # define LIBNAME "foo.dll" #endif foreign import ... LIBNAME ... foreign import ... LIBNAME ... foreign import ... LIBNAME ... > [...] if more than one library is specified, > an interpreter can try each of them for each function. Of course the error messages will be more confusing than they need to be... (This is true for both compilers and interpreters though the confusion is worse for interpreters because they try harder to hide the programmer from the complexity of the underlying system.) > We can give up the irrelevance of the order of declarations (which does > not apply for regular imports anyway - they must be at the top of the > module, before all local definitions). We can have some declaration > specifying the context of *following* foreign declarations. Another > such declaration can change the context later in the module. Just to be clear that I understand this argument, you're saying that: It is ok to severely break order independence in the ffi because it is already mildly broken in normal module import? This is just another source of confusion for the baffled programmer. (And more clutter for a language that already needs to go on a diet.) I'd say this is quite a bit worse than order dependencies in import decls because if they try to put an import decl in the middle of a module, they get a syntax error but if they put library decls out of order, they get a linking error (harder to understand) or they get the wrong symbol which probably has the wrong type and leads to a segmentation fault when they run it. Let's fix module import instead. > Here is a concrete proposal: > [proposal deleted] Looks like a massive extension to the ffi definition. What does all this complexity buy us? Wouldn't it be better to fix on a C/Pascal-like interface on the Haskell side and implement a series of similar ffi's for Java, C++, etc. which also target a C/Pascal-like interface? Now if I want to call Java code, I use ffi/Java to construct C-like exports for the Java library and I use ffi/Haskell to generate C-like imports for the Haskell library. On the plus side, this style of narrow-waisted architecture is much, much easier to maintain, concentrates debugging/development effort on just one interface, provides utility to people on both sides of the interface (because a C++ programmer could use the exports from the Java library too), is simpler, is more flexible, ... Could someone summarise the minus side for me? I think I missed the motivation for extending the ffi to support other languages. -- Alastair Reid ps Could I propose that any proposed new keywords be prefixed by __ or ffi_ or something like that. Adding keywords like "library" (i.e., ordinary useful words that programmers might already be using as variable names) breaks a lot of code. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Modification to foreign import/export
> > What does all this complexity buy us? > > The ability to specify #include files in a portable way, Can this not be done with one small extension instead of a much larger general purpose extension? > If a Haskell implementation needs to know from which library to take > which function, e.g. an interpreter using dlopen/dlsym, then again > this must be specified somehow. We could invent a way to encode it in > the foreign function name, but [...] The ffi spec already has a way to specify this: list it with each ffi decl. There's no need to invent anything new. I'm looking for a compelling argument that the convenience of adding all these extra declarations outweighs the costs of a more complex design, introducing more keywords into the language, etc. > When I import many stdcall functions, I must currently write stdcall > with each declaration. Can't you just add the required functionality (global library declarations, global calling convention declarations, order-dependent overriding, etc.) to hsc2hs? > Why I should be require to wrap everything in C if the compiler can > be designed to do that for me? Why not write a preprocessor that turns ffi/Java into ffi/C? Chances are that one of the existing calling conventions (i.e., stack and register layouts) and the existing set of types is going to be a pretty good fit. If you find yourself using a Haskell implementation that runs on a JVM and want to optimize it (e.g., support some Java types more directly), you can tweak ffi/Java to generate native Java or marshall some types more intelligently or whatever. A compiler flag can turn this on or off. > Surely maintaining just a C compiler is easier than maintaining both > C and Haskell compilers. So what, abandon Haskell because using C > for everything is simpler for compiler writers? I'd describe what I want as follows: Keep the ffi standard simple so that: 1) An already hairy piece of code doesn't become unmaintainable. 2) Incompatabilities between different implementations can be kept to a minimum. And the way to achieve this is to put any proposed complexity in tools rather than in the ffi. > > ps Could I propose that any proposed new keywords be prefixed by __ > > or ffi_ or something like that. [...] > > Of course they are not keywords in any context except foreign language > declaration. That's why I wrote "pseudokeywords". I hope you don't mean anything like Haskell 98's "as", "qualified" and "hiding" special identifiers? Those have to be one of the worst features in the entire Haskell syntax. Surely there's a better way? -- Alastair Reid ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Typing f.e.d.
> Alastair Reid wrote: > > [...] ps I have a feeling of feeping creaturism as I watch this list - > > a nice simple design seems to be getting more complex. > > Could you elaborate on this? The basic FFI *is* still nice and simple > IMHO. I'm thinking of: 1) Trying to support a complex language like Java in the same design (or in any design, really) My sense is that this will gradually infect the ffi design with the complexities of the other language and that this effect will be worsened by the mismatch between the two type systems. 2) Moving library specs, etc. out of the individual ffi decls. This seems to be partly motivated by supporting Java and partly by convenience and a dislike of cpp. > Mentioning plain Addr in it was a design mistake because: > >* Pointers to code and data need not have the same size. I understand this argument though I'm not sure which architectures this applies to. Alphas? >* The extra type parameter in Ptr/FunPtr tremendously helps to resolve > overloading ambiguities. >* Addr doesn't carry any type information, so there is no help at all > from the type checker, which is even worse than C. I remember briefly using the same trick in the early greencard days but gave up on it because I found myself wanting a many-to-many relationship between C types and Haskell types and using overloading and an extra type parameter I could only get a one-to-many relationship. (This is what greencard's DIS's give you.) So I've been sitting here wondering how things are turning out in practice. For example: How often do you have to create two Haskell types which really ought to be just one type just so that you can use the overloading? Is pointer arithmetic on Ptr's useful? (It's obviously useful for arrays but can you use it (safely and portably) to access fields of structs or do you have to add an explicit byte count to get predictable/safe results.) What's the preferred way to represent an opaque C data type which happens to be a pointer? With Addr, I'd use: newtype AbstractFoo = MkAbstractFoo Addr Should I use Ptr () instead? What libraries is the evolving design being tried out on (I know about your OpenGL library). > Admittedly, we have to be a little bit careful with the FFI libs I'm less concerned about the ffi libs because: 1) As a user, I only _have_ to use the low level ffi libraries: Addr, Word, Int64, ForeignObj, etc. Its nice if the high level libraries are useful too but not essential. 2) As an ffi implementer/maintainer, I only have to implement the agreed ffi interface and all the libraries on top will "just work". 3) They're not supposed to be standard so it's ok to experiment with the design a bit. 4) I don't have any code that uses them. -- Alastair Reid ps I tried to find a copy of the ffi spec on haskell.org today. After some searching, the only version I could find was more than a year old. I have a note somewhere of where the up-to-date copy is held but if anyone were to go looking for it themselves, they might have a hard time. It'd be good to sprinkle haskell.org with a few pointers to the current spec before Wednesday (which is when Hugs 2001 (including my ffi implementation) will be released. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Modification to foreign import/export
> "Alastair Reid" <[EMAIL PROTECTED]> wrote, > > > Marcin wrote: > > > This has the advantage that when the library name is #ifdefed, the > > > conditional needs not to be repeated for each function. Similarly > > > for #included header names. > > > > Note that this can be done without separating the library name from the ffi > > decl: > > [...] > > That's not a solution, because cpp is not part of Haskell. > I think, it is a good idea to make things cpp-friendly where > we can, but we must not rely on cpp to solve any problem. The point of me mail was that instead of using half the features of cpp (Marcin gave an example which used #ifdef) and extending the ffi spec to do the other half, he could use all the features of cpp (#ifdef and #define) and avoid the ffi extension. If we want to rule cpp as out of bounds (a very worthwhile but difficult goal), then we need to think about how to avoid conditional compilation too. (SimonM's mail this morning about not putting library names in source code seems to be part of this story.) Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Summary of current change suggestions
> So, as Marcin pointed out, the only use for a library object > spec for ccall is so that interpreters know which handle to > pass to dlsym(). This may be the only use but I think it's a very important use. > I am not too fond of the idea that the interpreter has to try > a dlsym() on all library objects that it did dlopen(). > Or is this maybe not too bad? I think it's just asking for trouble (for reasons given in earlier mail). -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Summary of current change suggestions
>* specifying libraries in the source isn't the right way to > go. Library names change independently of APIs, and can be > platform-specific. I like the high level goal this is heading towards and was in complete agreement with implementing it in rules like this until I realised that all that Hugs sees is the source (i.e., there's no makefile to put extra compilation info into) - so if it ain't in the source, where is it? But I like the rule... Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Summary of current change suggestions
The first time I read this syntax, I misinterpreted it. I think my misinterpretation might be interesting so I'll give it below in case anyone else likes it. > foreign library C"" "-lcurses" > foreign library java "blah" "blah" > > foreign import C"curses_function" cf :: Int -> IO Int > foreign import java "java.blah.*/foo" jf :: Int -> IO Int What I first thought this was doing was: 1) The foreign library decls _declare_ two new kinds of external function which the programmer as arbitrarily (and confusingly) chosen to call "C" and "java". 2) The foreign import declarations refer to these new kinds of external function. Those declared as "C" will use , etc. Those declared as "java" will use "blah", etc. Of course, this interpretation doesn't quite work because if "C" is just an arbitrary varid with no particular meaning, then we need to say somewhere else that "C" is to use the C calling convention. (We also need to change "C" to "c" if it is going to be a varid.) > A module can have one "foreign defaults" declaration per > calling convention, which specifies a string. > > foreign defaults ccall ": " > foreign defaults java "java." Alastair ps Just as a general remark on language design... User defined defaults are really tempting to add (my component programming language had them briefly and I am under constant pressure to add them back in) but they are kinda messy because: 1) You have to resolve (or forbid) use of multiple defaults. 2) You need a way to override the default. Do you override the whole default (includes, libraries, etc.) at once or only the bits you don't specify. The latter seems to be slightly better than the former because defaults are sorta invisible - the programmer forgets that they're there and so they forget that overriding the default will zap all the default not just the part (libraries say) that they're thinking about at that moment. 3) A small change to code in one place can have unexpected results (and it is hard to detect the consequences using typechecking or from a failed link or whatever). 4) Cutting and pasting some code from one context to another context results in very different behaviour because the defaults don't carry over. [I think every form of default behaviour in the Haskell spec has caused problems like this way out of proportion to the apparent difficulty of specifying the meaning of defaults and the difficulty of implementing it.] ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Summary of current change suggestions
> Marcin just pointed out that you can use dlopen() in a way
> that it finds symbols without a library name. That should
> do, I think.
Note that finding a marginally easier way to do the lookup doesn't address my
concern that this undirected search will cause maintenance and porting problems
by worsening an already bad situation (just which library did
Linux/solaris/freebsd/hpux/redhat6.0/redhat7.0/... choose to hide a symbol in).
In fact, I expect that having dlopen do a search through the libraries that it
thinks are relevant instead of Hugs performing a search through the list of
libraries that it knows to be relevant probably makes things worse because it
increases the gap between the programmer's mental model and what actually
happens.
Alastair
ps dlopen is not completely portable - Hugs also uses shl_{load,findsym} (HPUX
only?) and {LoadLibrary, GetProcAddress} (Win32).
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
cpp
While on the subject of preprocessors, I thought I'd mention a need I've found in distributing tools like Knit where the intended audience isn't expected to: 1) Care either way about it being implemented in Haskell 2) Have any desire to spend half a day installing lots of Haskell-related tools (greencard, c2hs, happy, etc.) on their machine before they can try out my tool. Getting them to install a GHC binary is about the limit of what I can reasonably expect. In this case, it is very useful to be able to distribute the output of all those useful support tools (greencard, c2hs, happy, etc.). But now the problem: if I achieve portability by running the input to these tools through cpp (or any other tool providing conditional compilation), then I end up with 2^N different files to distribute and complicated instructions for building the system. What would (perhaps) be nice is for the support tools to implement conditional compilation themselves and for the output file to use conditional compilation too. Alastair ps Or maybe I should just hack up a perl script that runs greencard and friends 2^N times and then packs the results (using diff, etc as appropriate) into a convenient file for shipping to users. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: cpp
> hsc2hs is in ghc-4.11. Did I miss the last 3 releases? I'm still stuck with something between 4.08.1 and 4.08.2 waiting for a debian package and freebsd port to appear for 4.08.2. :-) [Serious content of which is that I'm absolutely _not_ interested in subjecting users of my tools to the bleeding edge of Haskell tools. For example, we don't even tell people how to build Knit from source because the current instructions include installing 4.04 then using that to bootstrap 4.08.2 then building Knit.] > What would you like to obtain? An output file or files which can be used as input to cc/ghc/Hugs/etc. (i.e., which doesn't require greencard, c2hs, hsc2hs, etc.) and which contains ifdefs to handle any portability issues. > Do you mean considering every possible combination of #ifdefed > expressions? Sorry, it won't work. One of my modules contains 98 > #ifdefs in a row (generated by macros), which check availability of > errno constants. I will not distribute 316912650057057350374175801344 > variants of this file. Which is why it'd be nice if the output of the tools contained the ifdefs - a tool can often (but not always) do a more intelligent job of generating ifdefs. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: cpp
> What's Knit?
A component language for C. Announcement attached.
> Why not just distribute binaries for the platforms of interest?
People feel uncomfortable if they can't build from source themselves.
Generating binaries for a decent range of platforms is a bunch of work
(especially given the current mess in the Linux library system).
> Or runhugs scripts?
Using Hugs is attractive because it is really easy to install and very
portable. It's also too slow to use for compiling large systems or when doing
cross-component inlining.
But even if we stick with Hugs, there's a little ffi'd C code in the system -
so they still have to compile some of the code themselves.
--
Alastair
The University of Utah's Flux Research Group announces the first release
of the "Knit" component composition toolset. Knit is a new component
definition and linking language which can be used with C and assembly
code. Get Knit's open source code, examples, papers, and docs at:
http://www.cs.utah.edu/flux/knit/
Knit:
o supports components created from C and assembly code;
o supports component definitions that require little or no
modification of existing code;
o automatically schedules component initializers and finalizers;
o provides an extensible constraint system to detect subtle
errors in component composition;
o provides cross-module inlining that largely eliminates
the overheads of componentization;
o supports component hierarchies;
o supports cyclic component dependencies.
Knit can be used for any C program, but is especially well suited for
use in systems that have some of the following characteristics:
Many separate components, multiple implementations of the same
component, intricate initialization requirements, complex component
interdependencies, low-level code and embedded systems, or code that
is used in radically different configurations.
As described in our OSDI 2000 paper, we have already used Knit with the
Utah OSKit (a kit of OS components which has all these properties) and
with a suite of network router components based on MIT's "Click."
Knit is part of an ongoing R&D effort. One challenge is to balance
precision and conciseness of expression. We seek feedback, external users,
and collaborators whose experiences and insight will help evolve Knit.
To join the knit-users or knit-announce mailing lists, send email to
[EMAIL PROTECTED] with, e.g, "subscribe knit-users" in the body.
Thanks go to DARPA for their support, and thanks go to the entire Knit
crew below, but especially Alastair Reid, for the fine research,
development, and hard work that led to this Knit release. Matthew Flatt's
foundational work on Units underlies this work; Eric Eide is responsible
for the high quality of the tutorial and manual.
Kota AbeSean McDirmid
Eric Eide John Regehr
Matthew Flatt Alastair Reid
Mike Hibler Leigh Stoller
Jay Lepreau Patrick Tullmann
and Alastair says... Enjoy!
Jay Lepreau, [EMAIL PROTECTED]
Flux Research Group, School of Computing
University of Utah
RE: Modified proposal for default decls
> Many C interfaces contain a significant number of macros. And C is > one of the most widely used languages that we want to interface with. > So it seems to me that any FFI which aims to provide *good* support for > C ought to provide a way of accessing C macros. Since an explicit goal of the original ffi design was that it would not require a C compiler to implement it and since at least one implementation (STG-Hugs) doesn't require a C compiler, I'd like to see a lot of discussion before we add support for macros to the ffi. Alastair ps > However, the vast majority of C compilers > don't store the necessary type info. Since it's non-portable, it has no real bearing on this discussion but, nevertheless, people on this list might like to try compiling a C file with -g and then using 'objdump --debugging' to view the debugging information it contains. Besides variable and function types, it lists lots of typedefs and fields in struct declarations are annotated with the bit size and offset of the field. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Modified proposal for default decls
> Many C interfaces contain a significant number of macros. And C is > one of the most widely used languages that we want to interface with. > So it seems to me that any FFI which aims to provide *good* support for > C ought to provide a way of accessing C macros. [Sorry for replying twice to the same paragraph...] I just remembered that my project (Knit - a C component language) has severe problems with use of macros in header files. We are planning to solve these problems by writing a little tool that generates C function wrappers for macros. The idea is that you'd list the functions you want generated, their C type and the name of the corresponding macro and the tool would generate functions to call them and a header file to #include in place of the original file. In a way, this is like an ffi from C to CPP :-) Obviously this won't handle every possible macro but we hope it will handle enough of the common cases to be useful. The most obvious omission will be macros (like stderr) which look like variables - gcc provides a few tricks which might handle some of these problems. My project is planning to write this tool but would be more than happy if someone else did it first :-) -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: unsafePerformIO and unsafeInterleaveIO
> Should these functions be available through the standard FFI? > IMHO they should. I don't understand the question. Are you asking which modules should export them? Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: unsafePerformIO and unsafeInterleaveIO
> Without a standard way to get > at unsafePerformIO, such code would still rely on > non-standard features, which goers against the aim of > standardising the rest of the FFI. The Hugs-GHC standard libs have them exported from IOExts. Of course, the Hugs-GHC standard isn't an official Haskell library committee approved standard but, since it's something like 4 years old, I think it's a better candidate for approval than the ffi library which hasn't really stabilised yet, hasn't been whacked on by a large number of users, is much younger, etc. A ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: unsafePerformIO and unsafeInterleaveIO
> Sadly, due to lack of support on the Hugs side, the Hugs/GHC extension
> libraries have drifted apart.
Sadly true.
But any parts of the standard that haven't changed in the last 2.5 years (e.g.,
unsafe{Perform,Interleave}IO) still work - so in
this case it can be relied upon.
Alastair
ps I wonder if it'd be possible to merge them by doing this in the hslibs repository:
1) Move the STG Hugs ifdefs out of the way:
find . -name '*.[l]hs' | xargs perl -p -i -e's/__HUGS__/__STG_HUGS__/'
The __HUGS__ label made sense when I expected CLassic Hugs to be replaced by
STG-Hugs
- but it wasn't so it should be renamed.
2) Merge in the current Classic Hugs libs (using ifdef __HUGS__).
(There's a question of what to do with the Hugs libs which don't exist in
the Hugs-GHC standard. I imagine that GHC merrily puts GHC-specific files
into the repository and that'd be the thing to do with Hugs files. In each case,
a giant ifdef round the whole file would prevent it from troubling the other party.)
3) Add enough infrastructure to make it easy to checkout a working version of Hugs
based on the hslibs repository.
This would require a choice of running with -Fhscpp or using a makefile (or
whatever)
to convert all the files over so that you can run without hscpp and so that you
can build Hugs distributions. (Also requires moving files around to match Hugs'
current directory tree - no big deal.)
(It'd also mean that code freezes during the runup to a Hugs release would extend
into some of the hslibs, an expectation that other hslibs hackers would make at
least
a minimal effort to test that their changes don't accidentally break Hugs (which
requires a simple Hugs test harness for them to run now and then), etc.)
I think it'd be important to do this in an incremental way: setup some infrastructure
now, merge a few libraries now, leave the
others to be done later as time and needs dictate.
pps Should this happeen, I have a bunch of changes to the Hugs version of the Pretty
library that I'd like to commit - all to do
with strictness annotations required with unboxed types. (Or maybe Hugs should have
"strict ints" added to it - call that type Int#
and they'd behave like Int from the strictness point of view.)
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
RE: unsafePerformIO and unsafeInterleaveIO
> Alastair: > > The Hugs-GHC standard libs have them exported from IOExts. Manuel: > There is only one flaw: It doesn't work with nhc98. Huh? I thought nhc provided a reasonable subset of the Hugs-GHC libraries and, since the IO extensions are one of the first extensions people ask for, I assumed that if they provided anything they would certainly provide them. So does nhc provide them but under a different name or through a different module? Could nhc not just add them to IOExts in their next distribution? Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: FFI progress
I like this nice, simple design. I'm a little uncertain about the multiple header-file aspect but have no strong feelings either way. If most libraries could be ffi'd without the need for additional C files, the multiple header file notation would be an obvious win. As it is, many libraries I've dealt with need one or more .c files containing some support code and, in that case, it's not too big a deal to add another file. Some questions though. Suppose I have two ffi decls with differing orders of files: > foreign import ",:f" f :: > foreign import ",:f" g :: Should I report an error? Or, suppose I have: > foreign import ":f" f :: > foreign import ":g" g :: > foreign import ":h" h :: can I assume that foo.h will only be included once? can I assume that foo.h is included before bar.h? (When answering, remember that an optimising compiler might combine code from multiple modules (possibly with multiple maintainers) into a single C source file due to cross-module inlining.) -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: FFI progress
> Is anyone actually specifying > > stdcall > ccall > > calling conventions in a foreign import? hslibs/win32/Win32Dialogue.gc uses stdcall. cvs annotate says that this was written by Sigbjorn in October 1999. I'm not sure whether anyone depends on this. (The HGL definitely does not because it doesn't use dialogue boxes.) I imagine there will also be a lot of stdcall stuff in HDirect generated wrappers for COM components. Alastair ps I think "stdcall" is also known as "pascal". Don't know if that helps the story on viewing C as two languages though - you're going to have a hard time persuading people to view COM methods as though they were written in Pascal :-) ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Revised FFI syntax + typing
> Why is Word a GHC extension? Someone remind me? It's not part of the Hugs-GHC standard which is what we were working to when the ffi first came out. It's not part of that standard because it was felt at the time that bounded types with ill-defined ranges were a portability problem waiting to happen. (The alpha was still a viable machine back then and we genuinely believed that Intel were going to produce a 64 bit processor in the near future.) The fact that it's in there at all was just for backward compatability reasons. If we'd had DEPRECATE then, we might (???) even have deprecated it? -- A ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Revised FFI syntax + typing
> I don't agree that having a type of a fixed but unspecified size is evil. > You can query the sise - you don't have to assume any particular size - > and you have explicitly sized types as well. I suspect we're never going to agree about this but perhaps we can try to understand why the other feels the way they do - so I'd like to briefly (re)state why implicitly sized types disturb me so much. There's many possible ways of classifying different portability problems but one of the most important is: 1) Portability problems that can be mechanically detected. This includes missing libraries/header files, libraries and header files installed in the "wrong" place, missing functions and methods, functions with the wrong type and semantic differences so gross that even the lamest testing will detect the problem. Fixes tend to require conditional compilation. 2) Portability problems that are hard to detect mechanically. This mostly consists of subtle semantic differences that require somewhat carefully designed test cases to detect. Fixes may require conditional compilation but more usually require small subtle changes to the code to ensure that the code doesn't tickle such differences as exist. Usually all that remains is a comment which explains why something is coded in a slightly unnatural way. [If we replace "mechanically detected" with "statically detected", then even gross semantic differences would be in the second category. It makes little overall difference to the argument though.] The first kind is, in some ways, most annoying because it is very visible: the code just won't compile/link until I fix it and the fixes to make it compile remain as an ugly scar on the code. But I feel that the second kind is worse precisely because it is so hard to see: if the code has not been written with those subtle semantic issues in mind it may contain such errors; if the code is modified by someone who didn't consider those issues, it may contain errors. I agree that it is possible to write portable code with implicitly sized types but I am very concerned that it is very easy to write non-portable code containing the second type of portability problem. Using explicitly sized types doesn't avoid all overflow-related errors but at least it makes them appear more consistently and, thanks to the sheer ugliness of the names (Int32, Int8, etc.) it is harder to forget about overflow issues. -- Alastair ps On reflection, I find your example (from last week) of doing radix sorts on Ints a good example of where you would naturally want to be able to do bitwise operations on Ints. I'm not sure if I find it compelling enough to add a Bits Int instance but at least I now know of one example where one might legitimately want to. In the absence of a Bits Int instance I would have to write the code like this: -- this code to go in the same file as the code that assumes that -- Int and Int32 are isomorphic #if sizeof_Int == 32 type Intlike = Int32 toIntlike = intToInt32 -- or fromIntegral fromIntlike = int32ToInt #elif sizeof_Int == 64 type Intlike = Int64 toIntlike = ... fromIntlike = ... #else #warning ... #endif radix_sort :: [Int] -> [Int] ... rest of code converts to/from Intlike every time it does a bitop Yeah it's a bit ugly but at least it makes the assumptions it relies on fairly explicit. Even without a comment (which would be a worthwhile addition), the explicit conversions and the ifdefs make it fairly clear that we're assuming some kind of isomorphism to exist. It is unfortunate that protecting against potential portability issues is unnecessary for radix sort. To some degree we are used to that sort of thing though: there are some perfectly safe Haskell programs that the GHC/Hugs typesystem will not let me write. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: FFI Definition
> foreign export dynamic is a shortcut for importing an automatically > generated nontrivial C function, with argument type not conforming to > regular imports, which could not be written by hand. I don't get it. How do you use a function with a type like: (Int -> IO Int) -> Addr to import a C function? The closest I can get is to use foreign label to get an Addr (or FunPtr if preferred) but now I'm stuck. What's up? > So it's actually an *import*! It could be spelled as import with a > magical name - again all parameters are in the type. But it is also used for exporting so the name doesn't seem that inappropriate. Unless you're arguing that, since it brings a name into scope it can be thought of as a kind of import??? (But I don't think you are saying that.) -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
RE: Revised FFI syntax + typing
> Interesting language, but can anyone remind me what these > #if etc are supposed to me - I can't find them in the > Haskell report. They're not? But I see code like that all over the place. > There is no point in trying to fix a problem that H98 already > introduced. The H98 Committee in their infinite wisdom chose to > have types with system-dependent sizes and I think, we should be as > orthogonal as possible whether we like it or not. But H98 doesn't have Word so I don't see a requirement to add it or to provide ffi support for it. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ps As I understand it, underspecifying the size of Int in Haskell 1.0 was largely to allow compilers to use one or two bits as tags to distinguish boxed values (pointers) from unboxed values (ints, chars, etc.). In particular, this lets Haskell compilers generate Lisp code (since Lisp compilers often use that GC trick and lack the type information which would make it redundant). No current Haskell compilers use this trick (it's not very effective in lazy/polymorphic languages). Hugs comes close but, though it would be easier to provide ints with restricted range, it opts for providing a full 32 bits. (The decision to do this predates my involvement with Hugs.) The other reason for doing it is the hope of going a little bit faster on 64 bit machines. I'm not sure if there actually is a performance benefit or if it is significant. pps I'm assuming that UShort and the like have system-dependent sizes but I'm not objecting to them because it's just not possible to work with C without making the types match up - Int8 and friends don't go quite far enough towards speaking C's language. Pretty much every ffi interface I've ever written starts by figuring out the sizes of each of the major C types and selecting the corresponding explicitly sized Haskell type to represent it. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
extent strings
As the extent strings acquire more structure, I'm starting to wonder if encoding the information in strings is the right approach. What's the advantage of putting the information in a string? 1) All Haskell parsers will be able to read any ffi-ified code even if they don't support the language binding the ffi supports. But what does this achieve if the compiler backend doesn't support that ffi? 2) Avoids problems trying to forcefit Java/C/C++/... identifiers/keywords into Haskell and avoid the (ever-growing) set of Haskell keywords. But this could be done by using "" as an escape mechanism whenever there's a conflict. For example, suppose I wanted to foreign import a function called "class" using currently implemented ffi syntax. I just put "" round the offending identifier and I'm done: foreign import "class" clazz :: 3) Might be easier to generate good error messages when the language just isn't supported by that compiler. As long as the "language/calling convention" keyword appears near the start of the foreign decl, I think this is easy to do without packing everything into strings. What's the disadvantage of having highly structured strings as part of the ffi spec? There's no big technical issue that I know of but as the strings get more complex (i.e., as we start to write a grammar for them), alarm bells start to go off in my head: o Having a grammar for parsing strings seems to break the classic language structure of: lexical structure context free syntax static semantics dynamic semantics o Reminiscent of C++'s extern "C" declarations. In itself this isn't necessarily bad - but every time I see one of those decls, the word "ugly hack" springs to mind. I suspect that making these strings even more structured would make the hack look worse. o It seems simpler to have just one grammar for Haskell rather than to split it into two separate grammars. By this, I mean, two logically separate grammars (as in the current proposed syntax) instead of two physically separated grammar (as, for example, if someone were to provide a grammar for the C++ ffi binding as appendix G to the report). Is anyone else concerned about this or am I just pissing in the wind? -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Haskell FFI 1.0 vs Haskell/Direct: influence? loss of abstraction level?
> Therefore, I'd like to query the Haskell FFI mailing list for the > reasons which may have lead to drop the very declarative approach of > H/Direct in favour of the FFI 1.0, which to me, appears rather > low-level (in contrast). We wanted something that: - Was simple enough that Haskell implementers could all agree on what needed to be added to the compiler. - You could build HDirect on top of. Rather than standardizing HDirect, we chose to standardize the primitive operations on which HDirect is built. - You could build other FFI systems on top of. I think we have 5 different FFIs (built on the simple one) at the moment (raw ffi (plus extensive libraries), c2hs, hdirect, greencard and ) all with different strengths and weaknesses. - Was useful by itself - making quick little jobs and bootstrapping easy. > To make things more concrete and to show the influence of H/Direct, > here is what I dream of for the CLISP, an implementation of Common > Lisp. Only is_size() (and :guard) are missing. :language, :[in-]out > parameters, arbitrarily complex structure definitions, :alloca (vs > :malloc-free), 0-terminated arrays (c-array-max & c-array-ptr) and the > declarative style of definition have been in CLISP since 1995. The way I imagine you implementing this and making it portable across implementations is that you and fellow implementers provide one or more low level interfaces for making calls, allocating/ reading/ writing memory in various ways, etc. and then you build a portable high level library on top of this - where the code in that library is shared across implementations. That's what we've done. The extensive libraries I hinted at above are documented here (sadly with very few examples of common use): http://www.haskell.org/ghc/docs/latest/set/sec-foreign.html When we don't use one of the frontends (hdirect, c2hs, etc), the code we write has roughly the same flavour as your example (except we use Haskell overloading and higher order functions instead of macros, macros and macros :-) > Feature Request: Searchable archives at haskell.org/pipermail :) That would be nice, wouldn't it. (Hmmm, I think you can tell google to restrict its search to a particular site - so maybe we could just use google.) Hope this helps, -- Alastair Reid ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Problem with FFI?
[reply redirected to [EMAIL PROTECTED]] John Meacham <[EMAIL PROTECTED]> writes: > I may be missing something obvious here, but from the current FFI > spec it appears that it is impossible to create libraries in haskell > which are meant to be called from c code without running into > undefined behavior. The problem is in the definition of hs_init() > and hs_exit() . now, it is acceptable to have hs_init and hs_exit > called in the initialization and finalization of your library, but > the problem arrises when you link against more than one library > which is implemented in haskell, suddenly whichever library is > initialized secondly segfaults! (or whatever undefined behaviour > means.). programs could suddenly stop working when a library is > changed from a c implementation to a haskell one, which seems to be > a bad thing. Hmmm, I see what you mean. > proposed fix: allow nested calls to hs_init, hs_exit, a counter is > incremented when hs_init is called more than once, and decremented > on hs_exit. only the last call to hs_exit will actually do whatever > needs to be done. The hs_init function takes arguments - what if each call to hs_init specifies a different set of arguments? How about: hs_set_hs_argv (int argc, char *argv[]); hs_set_prog_argv (int argc, char *argv[]); hs_init (); hs_exit (); Where: hs_set_hs_argv sets command line parameters for the Haskell runtime system It can be called at most once and that call must be before the first call to hs_init Warning! The flags are not remotely portable between different Haskell runtimes. hs_set_prog_argv sets arguments seen by getProgName and getArgs It can be called at most once and that call must be before the first call to hs_init hs_init initializes the Haskell runtime It can be called multiple times and must be called before the first call to any function exported by the Haskell code. hs_exit finalizes the Haskell runtime hs_init and hs_exit are required to satisfy the usual bracketing rules: 1) At any time, the number of calls that have been made to hs_exit must be no more than the number of calls that have been made to hs_init. 2) If the number of calls to hs_exit is equal to the number of calls to hs_init, then no further calls to hs_init may be made. By the way, the name 'hs_exit' is a little confusing - I expected it to have an ExitCode argument. 'hs_fini' would better match its actual purpose. > note that this cannot be implemented by the programmer himself since > there might be several third party libraries also implemented in > haskell which an app wishes to link against. You can implement the modified API (or your modified semantics) quite readily as a little library which you link against your program. All you have to do is tweak the names a little (e.g., s/hs/HS/) to avoid name clashes. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Minor tweaks to ffi addendum
While implementing the full FFI for Hugs, I'm coming across a bunch of little things that confuse me and some wee typos. Here's a list of what I've found so far. If no-one objects in the next few days, I'll go ahead and make minor changes to the report where required. section 4.1.1: The definition of impents suggests that there must/can be spaces around the keywords 'dynamic' and 'wrapper' and that the static form can have spaces at start and end. I'm going ahead and implementing this interpretation but want to remark that it seems a bit odd. No action proposed. section 4.1.3: It took me a while to understand the explanation of Dynamic import. After a while, I realized that I should read the type as: (FunPtr ft) -> ft instead of FunPtr (ft -> ft) The current text 'FunPtr ft -> ft' does mean the former but it would be easier to read if written '(FunPtr ft) -> ft'. Proposed change: write the type as '(FunPtr ft) -> ft' Incidentally, how am I to interpret the type rules? Which of the following have the right type: foreign import ccall "dynamic" mkFun1 :: FunPtr (Int -> IO ()) -> (Int -> IO ()) type Int2 = Int foreign import ccall "dynamic" mkFun2 :: FunPtr (Int2 -> IO ()) -> (Int -> IO ()) newtype Int3 = Int3 Int foreign import ccall "dynamic" mkFun3 :: FunPtr (Int3 -> IO ()) -> (Int -> IO ()) data Int4 = Int4 Int foreign import ccall "dynamic" mkFun4 :: FunPtr (Int4 -> IO ()) -> (Int -> IO ()) type T5 = FunPtr (Int -> IO ()) -> (Int -> IO ()) foreign import ccall "dynamic" mkFun5 :: T5 Proposed change: none at present section 5.5: It's going to be really hard to implement ForeignPtr as specified. The problem is that invocation of the cleanup function is triggered by the garbage collector. I don't want the garbage collector to be recursively invoked so I don't want the garbage collector to directly invoke Haskell function. The Hugs garbage collector is not supposed to be a mutator of the heap - so putting a simple wrapper round the garbage colector won't work. And there's no mechanism outside the GC to look for cleanup functions to execute. GHC gets round this by scheduling a (preemptive) thread to execute the cleanup code. How on earth does NHC get round this? Does anyone have a suggestion for how it might be implemented in Hugs? Proposed change: none at present but I'm deeply sceptical of a design which takes a simple task (invoke a C cleanup function) and, for no discernable reason, generalizes it to the point that it requires a much more complex runtime system. section 5.6: castStablePtrToPtr and castPtrToStablePtr can break typesafety so one might want to try to make sure that any program that uses them will be forced to have the word 'unsafe' in them. Proposed change: none at present Table 2 (section 6): Is there a reason to be so coy about the concrete C types used for HsChar, HsFloat, HsDouble and HsBool? Proposed change: C symbolHaskell Symbol Constraint on concrete C type HsChar Charunsigned char HsFloat Float float HsDoubleDouble double HsBool Boolint I see no reason at all not to specify HsBool. For HsChar, I'd like to know if it is signed or unsigned. I can imagine that the vague specification is to allow for wide characters but it's not clear to me that one could write much portable code without knowing that so it seems we should pick wide or narrow characters and stick with it. (Hmmm, wonder how much code it would take to make Hugs work with wide characters?) For HsFloat/Double, I imagine we're being vague because Hugs implements Double with single precision floats and GHC on the Alpha (used to) implement Float with double precision floats. I think Hugs can be fixed (in fact, was almost all done ages ago). That leaves the question of whether we want to force Haskell implementations to follow C conventions. I'm not sure. Table 3 (section 6): Proposed change: Rename column 1 as 'Preprocessor symbol'. In the last 2 lines, change column 1 to read HS_BOOL_FALSE and HS_BOOL_TRUE. (This change merely standardizes the existing GHC fix for this problem.) -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Minor tweaks to ffi addendum
> Alternatively, we could have an explicit operation > > runFinalizers :: IO () > > which causes finalizers to run in a non-pre-emptive or > non-concurrent system. Of course, this solution is no use at all if your use of ffi is to pull a big mostly pure C library into Haskell. On two separate occasions, I've added interfaces to image processing libraries where pretty much every function was pure (it either generated a fresh image or returned an Int, Float or whatever). Images tend to be big (640x400x24bits is 3/4 Mbyte) so a few dozen image processing steps might quickly use up a lot of memory - and in all that time you don't go anywhere near the IO monad. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Minor tweaks to ffi addendum
> We have to specify the constraint that the C finalizer can't > directly or indirectly invoke any Haskell code (is this checkable at > run time?). It's the same constraint we place on 'unsafe' foreign imports. I picture a typical usage being something like this: foreign import ccall unsafe "static & free" free :: Ptr a -> IO () alloc x = do object <- malloc x newForeignPtr' object free It's easy to check correct use of the unsafe flag at runtime if we want: set a flag when you make an unsafe call and test the flag in the foreign export wrapper code. We could imagine getting the Haskell typesystem to check that newForeignPtr' is only used with unsafe free functions. It'd be a bit verbose but we could probably do it. There's not much point though since C can't do a similar job on its side. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Minor tweaks to ffi addendum
> - If we forced finalizers to be C functions because calling
> Haskell from the GC is inconvenient, then you have to add a
> constraint that the C function invoked from a finalizer can't call
> any functions foreign-exported from Haskell land. That's an
> annoying constraint to have to add, because it means that C
> libraries can't be transparent w.r.t. whether they invoke Haskell
> code or not (actually the hs_init() problem that someone else
> brought up recently also has this side-effect).
What do finalizers do?
For the most part, all they do is call other finalizers then free up
the resources they directly hold. A Haskell finalizers will usually
look like this:
void freeFoo(Foo f)
{
hs_freeStablePtr((HsStablePtr)f);
}
So finalizers have to be able to invoke some Haskell runtime functions
but don't have to be able to invoke Haskell finalization code directly.
Btw the runtime functions we found it useful to export from Hugs were:
void freeStablePtr(HsStablePtr);
void garbageCollect();
I don't think there are any others. I'm not sure if garbageCollect is
normally called by a finalizer but, if it were, I think we could
arrange for that limited form of reentrancy to work by exploiting the
fact that the GC invokes all finalizers at the end of the GC - so the
heap is in a nice clean state when it happens.
It's also possible that finalizers might 'do something' like close a
window (maybe showing an amusing animation while they do so).
1) These are much rarer.
I don't remember the last time I saw this being done. In fact, I
always considered it something of a mistake to rely on automatic
finalizers to do anything with human visible consequences (though
I seem to remember that Self or Smalltalk relied on this for
closing windows and found it very effective).
2) C programmers are already familiar with the idea that not all code
is reentrant and that you're especially likely to hit reentrancy
problems in a finalizer. So whilst it is a greater burden it is not
a new burden unlike any they normally deal with.
3) If they're using GHC, they are free to specify that their imported
finalizer is 'safe' (or threadsafe) and that 'safe' finalizer can
happily call back info the runtime.
--
Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Minor tweaks to ffi addendum
>> On two separate occasions, I've added interfaces to image
>> processing libraries where pretty much every function was pure (it
>> either generated a fresh image or returned an Int, Float or
>> whatever). Images tend to be big (640x400x24bits is 3/4 Mbyte) so
>> a few dozen image processing steps might quickly use up a lot of
>> memory - and in all that time you don't go anywhere near the IO
>> monad.
> Sure, but relying on ForeignPtrs and the garbage collector is quite
> dodgy. To support this kind of use we'd have to be much stricter
> about when finalizers are run: currently there's no guarantee about
> the promptness of running the finalizer.
We got round this by using allocators that looked like this:
static allocated = 0;
void* mymalloc(int sz)
{
void* r = malloc(sz);
allocated += size;
if (allocated > threshold) {
hugs->garbageCollect();
}
}
void myfree(void* x, int sz)
{
allocated -= sz;
free(x);
}
This let us force Haskell to give us what it could.
(The threshold mechanism was a bit more sophisticated than shown - a
range of soft and hard limits and some estimate of how many objects we
thought Haskell might be willing to give back played a part.)
Of course, with a generational collector you really want to be able to
specify how many levels to collect:
for(int depth=0; allocated > threshold; ++depth) {
ghc->garbageCollect(depth);
}
and you want the GHC to have the property that executing the sequence
garbageCollect(0);
garbageCollect(1);
garbageCollect(2);
garbageCollect(3);
garbageCollect(4);
garbageCollect(5);
costs little more than
garbageCollect(5);
Note that an interface based on asking GHC to keep GCing until it
recovers X Mb of space is no use - we're interested in freeing C
objects not Haskell objects.
Note too that I'm assuming that garbageCollect will wait until any
finalizers released by the GC have had a chance to run. (I have no
idea how this would be achieved in GHC.)
> Not to mention the fact that there's no guarantee at all about
> whether the garbage collector will detect that the ForeignPtr is
> unreachable when you think it should be.
That's true - but I have to performance tune my Haskell code just to
make sure it releases my Haskell objects so it's no hardship to have
to tune it to be sure it releases the C objects too. The reason this
approach works (and it was very successful both times I did it) was
that memory is quite abundant and virtual memory adds a useful safety
margin. If my program, uses 5 times as much memory as it has to, it
will probably still work ok. I am wary of using this approach with
less abundant resources - as people using lazy file IO have found out
to their cost (when they run out of file descriptors).
> I think for these kind of applications using ForeignPtrs isn't the
> right thing - you really want to do the allocation/deallocation
> explicitly in the IO monad.
Those kinds of applications exist - but there are many pure programs
for which ForeignPtrs work extremely well and for which it would be
painful to have to insert frequent calls to 'runFinalizers'.
--
Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: How do you represent a C pointer type?
> I have a small C interface with a C type DB. DB handles are only > administed on the C side; at the Haskell end they are a black box. > The Haskell side never sees DB values, just values of type DB * > (pointers to DB). I'm wondering how to represent this. The usual is something like: data DB-- note that we have no constructors foreign import ... :: blah -> IO (Ptr DB) > One solution would be just to represent the equivalent of DB * as > type DBP = Addr (or Ptr () or something) but then of course it might > be confused with any other Addr. I think Addr is deprecated (in favour of Ptr and FunPtr). It doesn't even appear in section 3.2. [The other type that you might expect but which was dropped was 'Word' - the reasoning being that it is an obstacle to writing portable code. 'Int' should be excluded for the same reason but that was even more controversial than omitting Word :-)] I'd say that 'Ptr ()' is discouraged partly because it's an accident waiting to happen and also because it prevents you from creating meaningful instances of Storable. > A rather more ingenious solution would be something like > newtype DBPInternal = DBPInternal Int > type DBP = Ptr (Ptr DBPInternal) > I think this might actually work in that a DBP would be marshallable > and couldn't be confused with any other type not involving > DBPInternal. Elegant it isn't, since it involves a pretend pointer > inside the Ptr just to get the types working. I think this is too ingenious for me. Not knowing about the common trick of writing a datatype with no data constructors (note: not sanctioned by H98), I can see why you might have gone for: newtype DB = DB_Dont_Use_Me Int type DBP = Ptr DB or newtype DB = DB_Dont_Use_Me Int foreign import ... :: blah -> IO (Ptr DB) the idea being that no-one would ever use the data constructor or the Int. I'm a bit puzzled by the double-layer of pointers though. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: More on Ptr types
> By the way, you lot are aware that under the old ANSI C standard at > least, pointer types can have different sizes depending on the type? > Thus a pointer to a properly-aligned 64-bit entity needs 3 less bits > than one to an 8-bit entity, and implementations are entitled to use > this. The only guarantee I can remember is that (void *) is big > enough to represent everything. Does this affect things at all? C has its faults but at least it was easy to understand what things meant (albeit in a very low-level manner) and the various implementation options. Then along come the standards organizations and everything that was simple gets complicated. I can see exactly why they did this (e.g., I'm currently working with an embedded processor that uses 16-bit addresses to index its 128K 16-bit-aligned instruction space and 12-bit addresses to index its 4K 8-bit aligned data space) but... ugh! I'm not sure whether it needs a change in the ffi implementations unless the function calling convention on some platform were to pass these pointers differently. But, let's imagine a pointer to an aligned 64K object in a 32-bit address space being stored in just 16 bits. I think the only change we'd need in the ffi spec to accomodate this is: Any casting between Ptr types (and FunPtr, StablePtr, etc.) has to apply the appropriate bit-shift (or whatever transformation is required). The simplest way to handle this may be to remove all the cast* functions from the ffi spec so that they have to be done in the C world (which understands these things). If this were simply intolerable (I must confess I don't know why we have them), I guess I'd define a ForeignCastable typeclass providing appropriate operations to cast to and from the equivalent of void*. As long as forall alpha. sizeof(HsPtr) >= sizeof(alpha*) I think the rest of the FFI spec is fine because the Haskell side never tries to do anything with a pointer - it just treats it as a curious looking bit-pattern that the C world would like it to hang onto for a while (or, in less anthropomorphic language, it is an opaque/abstract datatype). -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: More on Ptr types
If I understand you correctly, your concern is that a pointer argument/result might be passed differently depending on what it is a pointer to. We already have a special case for FunPtr because they can be different sizes from normal Ptrs (especially on a Harvard architecture). What I think you're suggesting is that we might need to make a special case for other pointer types too. This feels like a very unlikely thing to happen and I don't understand it yet so I propose the following (btw note that I don't speak for the whole FFI group). Put off acting on this until we find a platform like this. I believe that all currently supported platyforms are not like this because if they were we'd have noticed when we ported the compiler to this platform. On finding a platform like this, we can do one of: 1) Define a new calling convention ccall-plus (or whatever) which is just like the ccall convention except that the extloc strings can contain extra information to resolve the problem. 2) Require any affected functions to go through C wrappers which do the coercion. My feeling is that this will be viable because I suspect there would be something very unusual about the pointers which are passed differently (e.g., they will be pointers into I/O space or pointers to 32-bit aligned memory) so they will be rare and easy to find. 3) Find a way to add the missing type information to the Haskell type. I think this is much more than being careful to use Ptr (Ptr DB) instead of Ptr DB because I would be hugely surprised if void* and void** had different representations. Rather, I think any differences would only show between types like these: char* char[1 << 16] * char[1 << 32] * Haskell doesn't have a notion of sized types so we would deal with this either by extending Haskell (yoiks!) or by encoding the information using something like this: data Ptr8 a -- pointer to 8-bit aligned object data Ptr16 a -- pointer to 16-bit aligned object data Ptr32 a -- pointer to 32-bit aligned object data Ptr64 a -- pointer to 64-bit aligned object That is, we'd extend the FunPtr approach in some appropriate way. (Even if Haskell did have a notion of sized types, it wouldn't be enough because I'll bet there would be non-standard extensions which let C programmers override the default alignment of a type.) -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
deRefStablePtr exception?
What exception does deRefStablePtr raise when handed an invalid pointer such as 0? -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: More on Ptr types
> [snip] Unfortunately there appears to be no way this can be fixed in
> general for ANSI C without parsing the header file. For example if
> DB is an external type unknown to the Haskell compiler, and a
> function requires us to pass in a **DB, to which it writes a *DB as
> output, we cannot interface to it without knowing what sort of
> pointer a *DB is, since we cannot create a Ptr (Ptr DB) which will
> correspond to a **DB in any useful way. So we will just have to
> cross our fingers and hope it doesn't happen.
The ffi philosophy is to provide a mechanism out of which more
convenient tools can be built. There's nothing to stop an external
tool from doing this parsing.
Indeed, that's what you have to do when interfacing to something like
Xlib. Just how big an int is each of the umpteen integral types in
Xlib? Which ones are the same size on all platforms and which are
different sizes? Probably the best way to figure it out is to write a
small script along the lines of:
#include "X.h"
main() {
printf("sizeof(Display*) == %d\n", sizeof(Display*));
printf("sizeof(Window*) == %d\n", sizeof(Window*));
...
}
> It's not just a question of differently-sized words. For MMIX
> (which already has a gcc port) while pointers all have the same
> size, the bottom bits are junk (simply ignored) so compilers might
> conceivably zap these bits during casts and use them to store other
> information.
Ok, if they're passed the same way and all we have to do is respect
the bottom bits, that's easy. Don't use the cast functions provided
by the Foreign libraries and all will be cool. (We might want to do
an audit of the libraries to determine which other functions use the
cast functions.)
--
Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: deRefStablePtr exception?
Alastair: >> What exception does deRefStablePtr raise when handed an invalid >> pointer such as 0? SimonM: > That's a good point, I don't think it has any error checking at all. > Of course the only way to get an invalid stable pointer is by > calling freeStablePointer on a valid one, or by using the FFI. >From an imported C function: return (HsStablePtr)0 >From Haskell code: castPtrToStablePtr Ptr.nullPtr My reading of the description of freeStablePtr is that it is perfectly legal to pass around a (void*)0 as an HsStablePtr and to hand that to freeStablePtr and castStablePtrToPtr and expect something meaningful to happen. (My personal leanings with anything resembling a C pointer have always been to use some variant of Maybe Addr Maybe (Ptr a) Maybe (FunPtr a) Maybe (StablePtr a) if the pointer can possibly be null. It turns out that many functions assume their argument cannot be null or guarantee that their result is not null and this both catches errors and reminds the programmer to think about the null case.) > Nevertheless, deRefStablePtr should raise an exception when it finds > an invalid stable pointer. Which brings me back to my question: _which_ exception should it raise? -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
exporting functions which raise exceptions
Suppose I foreign export (or wrap) a Haskell function and that function raises an exception, calls exit or raises an error. How should this be reflected on the C side? In a batch-mode system (ghc, runhugs, nhc), we need to call a no_return function like exit but should it always be exit or should there be a way for an application to register an alternative function? (e.g., If my Basic interpreter makes a foreign function call into Haskell, I'd like an error in Haskell to bring me back to the Basic REPL not to quit my program). In an interactive system (ghci, Hugs), the right thing to do seems to be to return to the interpreters REPL. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
hs_garbage_collect
I'd like to extend the set of functions that a Haskell system provides
to C programs. As well as hs_init, hs_exit and hs_set_argv, I'd like
to have:
int hs_garbage_collect( int how_hard );
This invokes the Haskell garbage collector.
The argument and result are for use when interacting with generational
collectors.
The result is 0 if the garbage collector has done a full GC.
Increasing the size of the argument provides greater and greater
encouragement to do a full GC. Thus, this loop
int i=0; while (hs_garbage_collect(i++)) {}
is guaranteed to terminate when further GC calls will not free up any
more objects.
Subtleties:
1) What happens if the programmer writes this code instead:
int i=10; while (hs_garbage_collect(i++)) {}
^^ start deeper
It can either behave the same as if they started at 0 or
it can start by doing a hefty GC or whatever it feels like.
All that matters is that increasing the value of how_hard
makes more GC happen.
What if they write the loop backwards:
int i=10; while (hs_garbage_collect(i--)) {}
^^ start deeper^^ decrement
They'll start with a hefty collection and then do increasingly
small collections. The subsequent (smaller) collections might not
even achieve anything but they will still return zero only if there
is no more garbage to be collected.
2) On a distributed Haskell implementation, increasing the value of
how_hard must eventually cause remote machines to do a GC too.
3) It might seem simpler to just have hs_garbage_collect do a full
collection. But this would give no efficient way to write code
that garbage collects until enough C objects have been released.
Using the above interface this would be:
int freespace;
void free(void* object, int size)
{
freespace += size;
raw_free(object);
}
void* alloc(int requirement)
{
int i=0;
while (freespace < requirement) {
if (hs_garbage_collect(i++) == 0) {
return 0; // fail
}
}
return raw_alloc(requirement);
}
4) Increasing the value of how_hard must eventually cause all
ForeignPtr finalizers for free objects to be invoked. i.e., GHC
would have to keep track of whether there are any runnable
finalizer threads and run them to completion.
Because we want to garbage collect no more than we have to, we
might want the how_hard argument to behave as follows: if there are
any pending finalizers, run one of them; otherwise do either a big
or small collection.
// Note that this version ignores the how_hard argument
// but will behave correctly anyway.
int hs_garbage_collect(int how_hard)
{
static next_depth = 0;
if (pending_finalizers > 0) { // finalize what we can
run_one_finalizer_to_completion();
return 1;
}
if (next_depth == 0) {
minor_garbage_collect();
next_depth = 1;
return 1;
} else {
major_garbage_collect();
next_depth = 0;
return (pending_finalizers > 0);
}
}
--
Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/
ps Was there any consensus about changing the hs_init API as sugegsted
last week?
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: exporting functions which raise exceptions
> We discussed this before and agreed that any uncaught exception in a > call back from C to Haskell land aborts the program. If the > exceptions is to be propagated from the call back to the main > application, it is the FFI user's responsibility to arrange this. > (I believe that this is actually noted somewhere in the current > spec, if it isn't it should, though.) I couldn't see it in the spec and I'm working on the code that implements it at the moment so I thought I'd check instead of assuming or relying on a (probably imperfect) scan of the mail archives. Unless responses to my requests for clarification point to particular sections in the spec., I intend to amend the spec. to match responses. (I decided to hold off on modifications to the spec until I have a complete implementation that passes a reasonably large set of tests rather than doing it a bit at a time and getting bits wrong because I lack a complete enough picture.) -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: deRefStablePtr exception?
> But you seem to be saying that > > freeStablePtr (castPtrToStablePtr Ptr.nullPtr) > > is meaningful... which part of the spec implies that? Aha! I was reading the last sentence of the freeStablePtr description as though it applied to freeStablePtr when in fact it means that castStablePtrToPtr is guaranteed not to diverge. Well that's a lot easier to implement. btw What am I to make of the parenthetical remark (in particular it may be Ptr.nullPtr) Should I read that as a recommendation to implementors that when casting a stableptr which has already been freed then they should return nullPtr? If so, is this required or just hinted/recommended? > Either 'ErrorCall "Foreign.StablePtr.deRefStablePtr: invalid stable > pointer"', or a new exception. I don't mind which. I'll pick one and add my choice to the spec then. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: exporting functions which raise exceptions
>> I couldn't see it in the spec and I'm working on the code that > In the FFI Addendum, RC 4 > <http://www.cse.unsw.edu.au/~chak/haskell/ffi/>, Section 3.4, Ah, thank you. Hmmm, yet another 'undefined behaviour'. I think I'd prefer that as 'implementation defined'. > So, you are adding this to GHC? Hugs. I figure if that Hugs is going to use the hierarchial libraries it had better support the same ffi as GHC and NHC. > When this issue was discussed earlier, I think, SimonM pointed out > that (especially in the context of concurrency) it is hard to define > what exactly is supposed to happen with such exceptions. I'm not sure I see the difficulty - but I'll go ahead and make up my own solution (which will probably be much like what GreenCard does...) -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: exporting functions which raise exceptions
>> I'm not sure I see the difficulty - but I'll go ahead and make up >> my own solution (which will probably be much like what GreenCard >> does...) > What does GreenCard do? It does its best to return to the read eval print loop. If that fails (hmmm, not sure I understand what would cause that failure), it calls exit. A bit more error reporting seems to be called for here... -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Library archives
The current spec contains no examples of how you would specify a library archive. Which of the following are legal: foreign import ccall "[Storable_cbits.o]" readCharOffPtr :: ... foreign import ccall "[libc.so]" malloc :: ... foreign import ccall "[libc.a]" malloc :: ... foreign import ccall "[-lc]" malloc :: ... foreign import ccall "[-L/usr/lib -lc]" malloc :: ... I'm assuming that any relative filenames should be interpreted as being relative to the directory containing the .hs file. My current interpretation is that anything legal on a link line is ok (so all the above are kosher) but it may be sensible to limit it to just the first 3 forms. I'd like the spec to say what's legal and give at least one example. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Library archives
> Hence, personally, I would tend to restrict the string in square > brackets to the name of a library without a suffix. So you're proposing that it be a single library and not a list of libraries and especially not a bunch of search directories and files? This works for .h files because is is easy to merge multiple .h files into one by creating a super-file which #includes the others. But this relies on having multi-include protection in the individual (leaf-node) include files. Can we do this for libraries too? I think it would need multi-include protections and at the moment I'm struggling to see how to do it. H, what is the story if two files independently link against the same .o file or .so file? Do you get one copy of the file or two? I guess you have to get just one copy - which means that dynamic linking has to be in a single namespace which has some consequences for Hugs and GreenCard (which currently uses a separate namespace per module). > The suffix and other link options would, then, need to be supplied > by the Haskell system or via command line options to the compiler > and/or linker. I favour this, as it seems to be the most likely > scenario that makes a single foreign import declaration work on more > than one platform. This makes sense. > (I would also give the Haskell system the option of ignoring the > library spec altogether.) Hmmm, Hugs is already ignoring the header file because it doesn't seem to provide anything except headaches[details at end]. If I can ignore the lib spec too then things get much simpler but not necessarily better. The package system provided by GHC and NHC provides an alternate place where libraries could be specified. Can someone familiar with packages and with Hugs comment on whether they'd be able to replace the ffi's library spec and would make sense for typical Hugs usage. And would that meet SimonPJ's need too? -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ps The reason I'm wary of using the user-supplied header file is not cpp/C-namespace cleanliness (see recent GHC bug report) but the way that adding or removing a function prototype changes the semantics. The new Hugs ffi implementation ignores the header file and generates its own prototypes. This makes the following forms equivalent (which I consider a good thing): foreign import ccall "static" foo :: and foo :: foo = foo_dyn foo_addr foreign import ccall "& foo" foo_addr :: FunPtr foreign import ccall "dynamic" foo_dyn :: (FunPtr ) -> I think the useful and important job of checking that the ffi specs match the header file or, better, the code as compiled is best done by a separate tool. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Library archives
SimonM: > We had a long discussiong about this a while back (sorry for > sounding like a broken record!), The questions that come up as I implement the ffi are mostly things that are not clear from the spec. This is a particularily good example of that - there's no example and the explanation is brief and open to many interpretations. I could hunt through the archives and guess at what the final decision was or I could ask for clarification so that we all agree on what change is needed to the report (or I look foolish when the appropriate section of the report is pointed out to me). In this case, all I could find was a brief discussion with no final conclusion. (Or maybe I missed the conclusion - the subject lines one the messages are fairly uninformative so I skimmed old mail fairly quickly.) > where some people argued that libraries should be specified in the > FFI declaration in the source code, and some people (inc. me) argued > that libraries should be kept out of the source code altogether, on > the grounds that they were too system-dependent and it would be > impossible to write portable code without resorting to CPP. There > were other reasons, but I can't remember them all now. The more I look at it, the more I agree with you. >From the discussion, it seems that the purpose is to support .net. But I don't understand why .net concerns affect the ccall calling convention instead of the dotnet convention? Portability seems to be a major problem - even if we restrict ourselves to Unix systems, there's no sense in being able to specify libX11.so unless you can also specify the directory. Which directory will that be? Well, it varies from one system to another. Can someone come to the defence of the lib specifications because, at the moment, I see no argument for keeping them in the spec. The only thing they might be useful for is that often an ffi module needs a little file of support code and it's convenient to have a way of mentioning that file. (btw This isn't me trying to get out of implementation work: Hugs now supports the lib specifications and, as of last night, is able to load and run the HGL examples. To achieve this, I had to fix GreenCard which had clearly not been required to generate a library spec for some time since what it generated was syntactically incorrect.) > I'd be delighted if Hugs adopted a package system similar to GHC's I think I'd be happy if this happened too - but I think I'll leave that to someone else... In the meantime, I'm leaning towards the following interface for Hugs (GHCi and other interactive systems might provide a similar interface): hugs +G Foo.hs --lib="-L/usr/X11R6/lib -lX11 -lc" or, simpler still, hugs +G Foo.hs Bar.hs --lib="/usr/X11R6/lib/lib/X11.so.6 /usr/lib/libc.so.5" [The +G flag is a way of telling Hugs that it is to generate ffi code not to load and run code - so this is something that tends to be written at program install time not during normal runs.] Anything needing compiled as a result of loading Foo and Bar will be linked against these files. GHCi could do something similar but it would start by loading these files into the global namespace and then any dynamically loaded code or foreign imports would do lookups in these files. [Actually, Hugs probably would not use --lib= because Hugs traditionally uses cryptic, confusing flag names...] -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: freeHaskellFunPtr
Sven: > In a nutshell: I'm not very happy with the explicit allocation and > deallocation of the 'foreign import wrapper' thunks. They are an > artefact of the Haskell implementations and look strange in an > otherwise GC-ed world... I think they're explicitly deallocated (and, hence, explicitly allocated) because C uses explicit allocation/deallocation. The expectation is that the function pointer will be handed to C and that C code will be responsible for deallocating the function. I'm at something of a loss for how GC support would help with your example. How would Haskell's GC know that the one and only call to the wrapper had just been made? Incidentally, I'd like to add a Haskell version of freeHaskellFunPtr to the C interface: extern void hs_free_haskell_funptr( HsFunPtr x ); Of course, it can always be foreign exported (and compilers might even implement it that way) but I think we'll find that every major library exports its own copy and, since there's only so many obvious variants on the Haskell name, we'll quickly run into name conflicts which will then be resolved in the usual ugly way. I recognize that this is the top of a slippery slope which will quickly plunge us into a situation of exporting a whole bunch of functions for manipulating ffi-related datatypes. I'm not sure that's such a bad thing though. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Library archives
Alastair: >> The questions that come up as I implement the ffi are mostly things >> that are not clear from the spec. This is a particularily good >> example of that - there's no example and the explanation is brief >> and open to many interpretations. I could hunt through the >> archives and guess at what the final decision was or I could ask >> for clarification so that we all agree on what change is needed to >> the report (or I look foolish when the appropriate section of the >> report is pointed out to me). >> >> In this case, all I could find was a brief discussion with no final >> conclusion. (Or maybe I missed the conclusion - the subject lines >> one the messages are fairly uninformative so I skimmed old mail >> fairly quickly.) Simon: > See these threads, for example: > http://www.haskell.org/pipermail/ffi/2001-February/000212.html > http://www.haskell.org/pipermail/ffi/2001-May/000345.html I see a lot of discussion about header files. I see a small amount of discussion of libraries with many conflicting suggestions. I see no _conclusion_. The only conclusion is the ffi spec itself but it doesn't answer my questions. These are: 1) What is the form of a lib spec? A single library? Search paths? Omit/include file suffixes (.dll, .o etc.) 2) Is the library spec the only way of providing this information or can it also be supplied externally (e.g., on the compiler command line). The spec doesn't have to say what this way would be but it does have to say whether it exists and, I think, what can and cannot be overruled/added by this external spec. And if the answer to (2) is that there are other ways (which is what GHC has implemented), there's the important question of what if anything is special about lib specs, when would you use one way and not the other. That is: 3) What are library specs for? The only hints I got from the email archive are: 1) It might save one from calling dlsym on every namespace previously loaded. It is not clear that this is an issue though if you accept my argument from yesterday that to make C behave correctly (and also to keep static and dynamic linkers consistent with each other) you should use a single namespace (i.e., specify RTLD_GLOBAL when calling dlopen). 2) It is useful for .net - though I didn't hear why this was supposed to be relevant to people calling C. I'd like answers to these questions that are clear enough that we can agree on them and amend the spec to be clear. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ps In case it wasn't clear yesterday, I think the spec should be clear about whether C libraries live in local or global namespaces. I propose that they should be in a single global namespace. This will require no change to GHC but will require a change to Hugs' historical behaviour. ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Problem with FFI?
John Meacham <[EMAIL PROTECTED]>: > however i would make one change, only the first > calls to hs_set_hs_argv and hs_set_prog_argv are honored, after that > they are ignored, that way if multiple libraries wish to set the > arguments it wont run into undefined behavior. (since there is no > way in general to know whether hs_set_* has already been called.) Hmmm, I don't like interfaces that silently ignore arguments. How about if the functions return a bool saying whether arguments got ignored and then the caller can decide whether they care. > also, it should be made clear that it is possible to run the haskell > runtime more than once, as in > hs_init >c + haskell stuff > hs_fini >c stuff > hs_init >c + haskell stuff > hs_fini > done > so libraries which are dynamically loaded and unloaded will work. I was deliberately ruling that out. To do this we have to find every single global variable in the Haskell runtimes and provide an initializer function to set them back to their initial values. I'm a little reluctant to add it to the spec because I'm feeling lazy -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Getting constants from #defines
> I was expecting something like > > foreign import ccall unsafe "static ncurses.h ERR" cERR :: CInt > > to allow me to access the value of ERR (a #define) as cERR. In case you're wondering _why_ this doesn't work, it's because we'd like to be able to implement the ffi by dlopening a bunch of .o, .a and .so files and looking up symbols in them - with no C compiler anywhere in the picture. IIRC, this is how ghci worked at one stage - not sure if it still does. -- Alastair Reid [EMAIL PROTECTED] http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: More C interfacing issues
> Firstly, some ncurses calls return a *WINDOW and others take one as
> an argument but you never actually look at the contents of one
> yourself. I have defined "data Window = Window" and am using Ptr
> Window for the types of these. However, this means I get a warning
> about an unused constructor. Is there a better way to do this?
Hugs supports:
data Window
and I believe GHC and NHC do too. (Malcolm, SimonM: please shout if I
overstate.)
> Secondly, does anyone have any suggestions on how to provide an
> interface to this?
> void getyx(WINDOW *win, int y, int x);
> The getyx macro places the current cursor position of the given
> window in the two integer variables y and x.
> The type would probably ideally be "Ptr Window -> IO (CInt, CInt)".
The easy way is to use GreenCard (the other ffi frontends may be of
use too).
Or, you can do what GreenCard does which is to add 3 wrapper functions
more or less like this:
static int x;
static int y;
void wrap_getyx(WINDOW *win) { getyx(win,x,y); }
int getx(void) { return x; }
int gety(void) { return y; }
or what GreenCard used to do which is more or less
struct result { int x, int y };
void wrap_getyx(WINDOW *win, struct result *r) {
getyx(win,&(r->y),&(r->x));
}
getyx :: Ptr Window -> IO (CInt, CInt)"
getyx w = do
r <- alloc sizeof_result
wrap_getyx w r
x <- deref r 0
y <- deref r 1
free r
return (y,x)
or guess/check that getyx is really, really cheap to call and write two
wrappers gety and getx.
--
Alastair Reid
[EMAIL PROTECTED]
http://www.reid-consulting-uk.ltd.uk/alastair/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: More C interfacing issues
> Uh, oh! Don't even try this if you ever want to be thread-safe! :-} Yeah, I know. For what it's worth, it's not what GreenCard does with the native Hugs backend (at least, it wasn't last time I looked). Hugs primitives are allowed to return tuples of values so they do. It's a shame C doesn't support multiple return values and kinda surprising given that C is like assembler in so many other ways. -- Alastair Reid [EMAIL PROTECTED] http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: More C interfacing issues
>> > Secondly, does anyone have any suggestions on how to provide an >> > interface to this? >> >> > void getyx(WINDOW *win, int y, int x); > What's wrong with just doing this in the plain FFI? > foreign import ccall "getyx" getyx > :: Ptr Window -> Ptr CInt -> Ptr CInt -> IO () > Have I missed something? You need at least one wrapper because getyx is a macro. Note that the arguments are ints _not_ int*s. -- Alastair Reid [EMAIL PROTECTED] http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: More C interfacing issues
> Curious: how does green card know that getxy is a macro and that x > and y are out parameters... do you have to tell it? More or less. GreenCard definitions are of the form of the form: %fun foo :: ... %call ... GreenCard fills in the obvious code (i.e., result = foo(args)) if you omit the %call line. Roughly speaking, you use %call lines in the same places that you'd introduce a wrapper in greencard. So you don't tell it that it's a macro but you do write a %call line and the code in the %call line will reflect the fact that you know it is a macro. -- Alastair Reid [EMAIL PROTECTED] http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
> I've been having trouble with an imported function that returns > Int64/"long long". Haskell doesn't generate a prototype, so regular > 'int' type is assumed. Which compiler is this for? As you saw from the other replies, compilers vary in whether or not they add prototypes. GHC: yes, NHC: no, Hugs: depends which day you grabbed your copy from CVS. (I'm not sure that the ffi should allow compilers to vary in this way but it does.) -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
> The "proper" way to do it is to write the prototype yourself in a C
> header file (or use an existing header), and then use a
> compiler-specific mechanism to #include it.
Shouldn't we encourage people to use the compiler-independent
mechanism instead (and exclusively)?
That is, you put all your flags, includes, prototypes, etc. into a
header file:
foo.h:
#include "config.h"
#define ENABLE_FEATURE_X
#include
#ifdef HAVE_STDLIB_H
#include
#endif
extern void my_helper_function(int x) {...}
and then mention foo.h in all relevant foreign imports:
Foo.hs:
...
foreign import "foo.h" x :: ...
foreign import "foo.h" y :: ...
foreign import "foo.h" z :: ...
foreign import "foo.h" my_helper_function :: ...
Using this approach is easier for users and easier for tools which
generate ffi output. (I recently modified the ffi backend of
GreenCard to generate code like this instead of using the GHC-specific
mechanism SimonM described.)
They still have to use compiler-specific mechanisms to specify
#include search paths, linker options, etc.
> The nhc98 compiler does generate a prototype (ghc does not).
> However, although this works quite well in the general case,
> occasionally the generated prototype may not be compatible with the
> real C type of the function (e.g. it cannot guess the `const'
> attribute).
In a sense though, this doesn't matter because C compilers use the
same calling conventions for const arguments and non-const arguments.
Likewise, all compilers we know of pass data pointers around the same
way no matter what the size and alignment of the thing they point at.
(And if either of these properties is found to change, I believe we
would change the ffi so that it is still feasible to generate correct
machine code without being given a function prototype by the user.)
Of course, this ignores the detail that while the C compilers are
generating correct code, they may also be generating warnings about
alleged type errors - which can be a bit disconcerting.
--
Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
>> Of course, this ignores the detail that while the C compilers are >> generating correct code, they may also be generating warnings about >> alleged type errors - which can be a bit disconcerting. > It is an error, not just a warning, if the prototypes don't match. > GCC will complain loudly if there's a missing 'const' - it won't > generate any code at all. Well, if you disable the warning by giving gcc a consistent story, then the code is correct. (To give gcc a consistent story, don't #include any user or system-supplied headers and make sure gcc doesn't silently #include any of its own (as it likes to do).) A ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
Alastair:
>> (I'm not sure that the ffi should allow compilers to vary in this
>> way but it does.)
Manuel:
> The FFI doesn't really allow compilers to emit prototypes (at least
> not in general) as this leads to semantic differences eg in argument
> promotion, for which Section 4.1.5 precisely specifies the
> behaviour.
I just reread that section and can't see anything that contradicts the
idea that the FFI implementation ignores any header files provided.
For example, the phrase:
"implements calls to C functions ... as if a function prototype for
the called functions is in scope"
can be interpreted as meaning:
1) Will generate code (by generating machine code directly, by
generating C code which is then compiled, or by some other means)
which assumes the existence of a prototype _even if the user does
not supply one_.
[I see this as the most obvious interpretation of the phrase.]
2) Will generate C code which #includes any header files the user has
provided.
[I see this as a less likely interpretation but it seems to be the
one you intend?]
My reading of the whole section is that every mention of users
providing header files and function prototypes refers to compiling the
C code being called not the Haskell code doing the calling. The
examples of using foreign import don't show the use of header files
though the second example needs a header file to compile correctly
with GHC.
If the intent is that any user-provided header files must be obeyed, then
I think the spec should explicitly say:
When a header file containing a function prototype is not provided,
the function calling convention employed is undefined. It may vary
between different operating systems, between different Haskell compilers,
and between different functions.
The 'between different functions' part is because GHC #includes a lot
of header files into the code it compiles - it may well have a
prototype for the function you're calling even though you don't
provide one. Trickier yet, gcc likes to #include some files that
aren't mentioned in any of the header files that you #include. These
define functions like 'memmove' which gcc treates magically.
Also, I'm using 'function calling convention' instead of 'argument
passing convention' because some C compilers pass _and return_ small
structs (such as 2 ints) specially and I think you need prototypes at
both definition and call site to make the code work correctly.
I also think the wording could be changed to clearly separate the
notion of the user invoking a C compiler on a C library that they wish
to call from Haskell and the Haskell compiler invoking a C compiler as
part of its compilation process.
I attach my rewording of the section. Note that I am trying to make
it quite clear that an ffi declaration is not portable unless you
provide function prototypes except in the special case that your C
compiler generates the same code with or without a prototype.
\subsubsection{C Argument Promotion}
The argument passing conventions of C are dependant on whether a
function prototype for the function is in scope at the site where the
function is defined and at the sites where the function is called..
%
In particular, if no function prototype is in scope, \emph{default
argument promotion} is applied to integral and floating types.
For example, consider the following C
function definition, which lacks a prototype:
%
\begin{quote}
\begin{verbatim}
void foo (a) {
float a;
...
}
\end{verbatim}
\end{quote}
%
The lack of a prototype implies that a C compiler will apply default argument
promotion to the parameter \code{a}, and thus, \code{foo} will expect to
receive a value of type \code{double}, \emph{not} \code{float}. Hence,
any callers should call \code{foo} as though it had been defined
with type:
%
\begin{quote}
\begin{verbatim}
void foo (double a);
\end{verbatim}
\end{quote}
%
To ensure portability across different operating systems and compilers,
the user should write a header file (called \code{"header.h"}, say)
containing this prototype and should use this \code{foreign import}
declaration.
%
\begin{quote}
\begin{verbatim}
foreign import ccall "header.h" foo :: Double -> IO ()
\end{verbatim}
\end{quote}
%
(Of course, it is more reliable to use the same header file when
compiling the function being called --- but that is not always
possible.)
Just as one must be consistent in providing function prototypes
when compiling C code, one must be consistent in providing function
prototypes when compiling Haskell code.
%
That is, when a header file containing a function prototype is not
provided and the function calling convention is affected by default
argument promotion, the function calling convention employed by
foreign function declarations is undefined.
%
The calling convention employed may vary between different operating systems,
between different Haskell compilers, and between different functions.
A similar situation arises in th
Re: Generating Function Prototypes
Alastair: >> Well, if you disable the warning by giving gcc a consistent story, >> then the code is correct. (To give gcc a consistent story, don't >> #include any user or system-supplied headers and make sure gcc >> doesn't silently #include any of its own (as it likes to do).) Simon: > I don't think it's possible to completely eliminate system headers > from the transitive closure of stuff we include when compiling a .hc > file. I've just taken a look at this, and it seems that while we > can eliminate a lot of the stuff we include, there are some awkward > ones: we need gmp.h for Integer operations, and that includes > , and HsFFI.h needs to get the int-type limits > (perhaps these could be autoconf'd). Well here's a trick that makes it ok to pull in arbitrary header files which might appeal to the assembly post-processing fans amongst us. Suppose you are compiling a module containing this ffi decl. foreign import foo :: Float -> Char 1) Have the Haskell compiler generate C code like this: #include #define whatever_you_want extern HsChar ffi_generated_foo(HsChar arg1); ... put GHC-generated code here ... Using the prefix "ffi_generated_" lets us be fairly confident that there is no prototype for the function i scope. 2) In the assembly postprocessor, apply the following transformation: s/ffi_generated_foo/foo/g The only thing this will mess up is use of inline functions (which won't get inlined). extern inline functions (for which gcc generates no code at all) are a bit of a problem too but the fault lies in the person who wrote the C code for writing code which won't even compile if you turn optimisation off. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
> Errm, shouldn't that be: [...] > Or is there some other trick involved here? Sorry, yes it should - just me getting confused in translating between Haskell's f :: A -> B and C's B f(A); A ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
SimonM: > I explained (in my previous message) that GHC needs > header files and prototypes in order to generate correct foreign > calls when compiling via C. This is, IMO, a GHC-specific issue and > doesn't have anything to do with the FFI specification - although > confusion could be avoided if the spec pointed out that some > implementations might need access to header files/prototypes in > order to compile the code. I think this is a very misleading way to describe the problem. A major goal of the ffi is to provide a portable way of interfacing Haskell to C. If we regard the header file as an optional extra which some compilers need and some don't, then we have failed in that goal. The effort of producing a header file is sufficient that ffi code supplied without them will be very hard to port to ghc -fvia-C (and, as I understand it, -fvia-C is considered the more stable platform especially when doing ffi work). One might argue that the whole area of ffi is rife with portability problems because the portability problems encountered when writing, compiling and linking C code leak into the ffi. That is true to some degree but: 1) We have tools like autoconf which help make the C side of things more tractable. 2) The task of transferring a few linker flags from one makefile to another is quite small compared with the task of writing a suitable header file. 3) If I have m Haskell compilers and n platforms, then I have m*n combinations to worry about if the ffi is not portable but only n combinations to worry about if the ffi is portable between compilers. -- Alastair Reid[EMAIL PROTECTED]http://www.cs.utah.edu/~reid/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
Minor detail: > typedef long long HsInt64; Instead of writing your own definition, you can #include HsFFI.h -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
> Sure, it's a problem, a bug in GHC, whatever. But I don't see a way > around it! Make the spec say that you have to have header files containing prototypes to get correct results. It is not ideal but it is something that can be readily implemented by all Haskell compilers so we can get portable results. Of course, compilers are free to provide extensions, compiler flags, etc. which render the header files unnecessary but people using those extensions are writing non-portable code. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Generating Function Prototypes
> [...] if we were somehow able to generate prototypes then there > would be no type checking at all. Here's 4 ways to get typechecking - all admittedly harder than relying on the fact that your Haskell compiler invokes a C compiler as part of its operation (as does Hugs - not trying to put GHC down). 1) modify a C compiler to perform a consistency check; or 2) find a way of writing code which would cause an unmodified C compiler to generate type errors if the Haskell type was completely inconsistent with the C type. I haven't explored this very thoroughly but C typechecking is most fussy about assigning pointer values so code along the lines of the following might be a good place to start from: // the user-supplied header file #include "math.h" // the type the Haskell compiler generates typedef HsDouble (*sin_ptr)(HsDouble); // the consistency test sin_ptr test = &sin; We'd probably have to use #define const /**/ at the start to make the compiler 'ignore' const-ness. 3) Find a way to extract information about the calling convention from information generated by a C compiler. gcc has a number of flags that dump additional information. .stabs entries are really informative. 4) Compare the output of the C compiler. Compile a piece of code which calls the function in question first using the header file and then using the prototype implied by the ffi declaration and compare the generated code. If one pushes a float while the other pushes a double, you know you have a problem. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Proposed change to ForeignPtr
Since requiring ForeignPtr.newForeignPtr would require preemptive concurrency (see previosu discussion), I propose the following changes: 1) Add these functions: makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO ForeignObj attachForeignPtrFinalizer :: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO () 2) Remove newForeignPtr and addForeignPtrFinalizer Two variants on this proposal: 1) Reuse the old function names instead of inventing new ones. I don't care much either way. 2) Instead of removing the old functions, rename them and mark them as an optional extra. I think this would be a mistake - the GHC documentation is the right place to mention extensions which are not available on all ffi implementations. I have implemented makeForeignPtr in Hugs (I didn't do attachForeignPtrFinalizer yet because it'll take a bit longer to add that functionality and we can compile most (all?) of the hierarchial libs without it. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Proposed change to ForeignPtr
> I assume you meant > makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a) Oops, yes. > What do you expect to happen if the finaliser calls a foreign > exported function? Good question. I do not expect that to work on any platform that has difficulty implementing newForeignPtr (because you could use it to implement newForeignPtr). I don't know if it would be likely to work on GHC. I think the spec should say that it is an error or undefined depending on whether GHC supports reentrant finalizers or not. > That's a tricky one. From the standards point of view, I am > actually *very* reluctant to introduce new names. On the other > hand, reusing the old names will lead to a couple of unhappy emails > from people using the old interface again. But only a couple I conjecture. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Updates to FFI spec
For those not on the cvs mailing list:
I've applied all the changes discussed over the last 2 moniths that
received some support and no dissent.
Changes since RC5:
* Author list: changed Alastair Reid's institution
* 4.1.1: Removed [lib] from impent syntax and discussion
* 4.1.3: Added parentheses round FunPtr ft to make it easier to
understand a tolerably complex type.
* 4.1.4: Removed all mention of library objects
* 6: Specified that HsBool==int in table2
Relabelled column 1 in table 3 (C symbol -> CPP symbol)
Replaced 0 and 1 with HS_BOOL_FALSE/TRUE
You will need this file:
http://www.cse.unsw.edu.au/~chak/haskell/grammar.sty
to build it. (I came close to adding this file to the repo but
figured that Manuel must have a reason for not having done so
himself.)
Changes not applied:
- I really, really want to resolve the ForeignPtr issues soon.
http://www.mail-archive.com/ffi@haskell.org/msg00655.html
http://www.mail-archive.com/ffi@haskell.org/msg00544.html
http://www.mail-archive.com/ffi@haskell.org/msg00545.html
- I'd like to see a standard way to call the GC from C
http://www.mail-archive.com/ffi@haskell.org/msg00565.html
Note that Hugs and GHC have had this for ages except that we call the
function 'performGC' and there's no way to control how many generations
are collected.
- I see the question of Function prototypes as a portability problem
waiting to happen. Either Hugs and GHC are right (you should use the
user-supplied header file or NHC is right (you should ignore the
header file). They can't both be right if we want portable code
so the report should be clear about which one is right.
(Given my druthers, I'd drop header files from the foreign import syntax
and say that you have to specify it on the command line or propose that
we standardize some variant of the GHCism {-# -include "foo.h" #-}. But
I'm not excited enough about it to push hard for this.)
- Changes to hs_init
http://www.mail-archive.com/ffi@haskell.org/msg00539.html
--
Alastair Reid [EMAIL PROTECTED]
Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Proposed change to ForeignPtr
Alastair wrote [snip] > makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO ForeignObj > [snip] I don't understand this proposal. What is a ForeignObj? Sorry, that was a typo. The result type should be IO (ForeignPtr a) > I call a C function, which gives me a cString :: Ptr CChar, and it's > my responsibility to free it when I've finished with it. So I > convert it to a ForeignPtr: >foreignPtr <- mkForeignPtr cString > and then always refer to the pointer via foreignPtr. When > foreignPtr is garbage collected the space is freed. So how do I do > this with your proposal? With the existing spec, you would write: > foreign import free :: Ptr CChar -> IO () > > foo = do > ... > foreignPtr <- newForeignPtr cString (free cString) > ... With my proposal, you would write: > foreign import "&" free :: FunPtr (Ptr CChar -> IO ()) > > foo = do > ... > foreignPtr <- newForeignPtr cString free > ... All the rest of your code to manipulate ForeignPtrs remains the same. (Well, there's a corresponding change in addForeignFinalizer if you happen to use that.) -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Proposed change to ForeignPtr
>> > That's a tricky one. From the standards point of view, I am > >> actually *very* reluctant to introduce new names. On the other > >> hand, reusing the old names will lead to a couple of unhappy emails >> > from people using the old interface again. >> >> But only a couple I conjecture. > I read this as you would also (= like me) be in favour of keeping > the old names. Right? I'm not sure which position you're preferring here. I lean a bit towards using the old names for the new functions (the ones with free functions) and finding new names for the old functions (the ones with closure arguments). -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec
>> - I'd like to see a standard way to call the GC from C
>>
>> http://www.mail-archive.com/ffi@haskell.org/msg00565.html
>>
>> Note that Hugs and GHC have had this for ages except that we call
>> the function 'performGC' and there's no way to control how many
>> generations are collected.
> I don't have a strong opinion on this one.
Maybe SimonM and Malcolm do.
I know GHC has an interface already and I suspect that NHC will too.
GHC and Hugs both call it performGC but this doesn't match the naming
convention used in the ffi and it is expensive to use with
generational GC (since there is no way to indicate that a partial GC
may be enough).
>> - I see the question of Function prototypes as a portability
>> problem waiting to happen. Either Hugs and GHC are right (you
>> should use the user-supplied header file or NHC is right (you
>> should ignore the header file). They can't both be right if we
>> want portable code so the report should be clear about which one is
>> right.
>>
>> (Given my druthers, I'd drop header files from the foreign import
>> syntax and say that you have to specify it on the command line or
>> propose that we standardize some variant of the GHCism {-# -include
>> "foo.h" #-}. But I'm not excited enough about it to push hard for
>> this.)
> I am still in favour of user-supplied header files and the mechanism
> as it is defined in the spec right now.
Malcolm and I both found it possible to interpret the spec as meaning
that the header files could be ignored. If that isn't what the report
means, the wording should be strengthened.
[My concern with specifying header files in foreign imports is twofold:
1) It suggests the existence of multiple namespaces when I very much
doubt that anyone would every implement that.
2) It is awfully repetitive because you have to mention the header files
on every foreign import just in case anyone ever did implement
multiple namespaces. (Other side of the same problem.)
I'm not arguing for a change here - just saying that I wouldn't argue
against one.]
>> - Changes to hs_init
>>
>> http://www.mail-archive.com/ffi@haskell.org/msg00539.html
> I am ok with that. Currently, there is a problem with the version
> that is in the spec and GHC in that GHC requires an extra argument
> to initialise modules. So, it all depends a bit on how far SimonM
> thinks its implementable.
Hugs doesn't have anything resembling hs_init at the moment so I can't
say for sure. I think it can be done as a thin layer on top of the
the existing spec though so I don't think it's that hard.
--
Alastair Reid [EMAIL PROTECTED]
Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec
> System.Mem.performGC does a major GC. When would a partial GC be > enough? I've described the image-processing example a bunch of times. We have an external resource (e.g., memory used to store images) which is somewhat abundant and cheap but not completely free (e.g., eventually you start to swap). It is used up at a different rate than the Haskell heap so Haskell GCs don't occur at the right times to keep the cost low and we want to trigger GCs ourselves. (In the image processing example, images were megabytes and an expression like (x + (y * mask)) would generate 2 intermediate images (several megabytes) while doing just 2 reductions in Haskell.) How often and how hard should we GC? We can't do a full GC too often, or we'll spend a lot of time GCing, destroy our cache and cause premature promotion of Haskell objects into the old generation which will make the GC behave poorly. So if all we can do is a full GC, we'll GC rarely and use a lot of the external resource. Suppose we could collect just the allocation arena. That would be much less expensive (time taken, effect on caches, confusion of object ages) but not always effective. It would start out cheap and effective but more and more objects would slip into older generations and have to wait for a full GC. To achieve any desired tradeoff between GC cost and excess resource usage, we want a number of levels of GC: gc1, gc2, gc3, gc4, ... Each one more effective than the last and each one more expensive than the last. We'll use gc1 most often, gc2 less often, gc3 occasionally, gc4 rarely, ... It seemed important to do this 4-5 years ago when we were doing image processing with Hugs. Now, with memories much, much larger (cf. Hal Daume's 'why does my program segfault with a 4Gb heap' report) but not appreciably faster, it is even more important. > Isn't the property you really want "a full garbage collection has > been performed and all finalizers have been run to completion"? With Hugs, we used feedback between the rate of deallocation/ allocation and the rate of GCing. The feedback was easy because the GC didn't complete until all objects were freed. With GHC, it would be harder because of the delay between GC and release. What I'd do is wander round to the university library, find the section on control theory, find a good book on feedback loops and implement something from there. > I think the spec should be clarified along these lines: > Header files have no impact on the semantics of a foreign call, > and whether an implementation uses the header file or not is > implementation-defined. Some implementations may require a header > file which supplies a correct prototype for the function in order to > generate correct code. I still don't like the fact that compilers are free to ignore header files. Labelling it an error instead of a change in semantics doesn't affect the fact that portability is compromised. > I'd be equally happy (perhaps happier) if the header file spec was > removed altogether. In a sense, this would leave the Haskell part > of a foreign binding even more portable, because it doesn't have to > specify the names of header files which might change between > platforms. There are already "external" parts of a binding such as > the names of link libraries, I think I'm arguing that header files > also fall into that category. Making header files completely external seems to encourage omission of header files and non-portability. And, annoying as it is to limit ourselves to one header file, it has the nice effect that there is a file out there that Hugs can use - I don't have to go figure out how the GHC Makefiles and configure system contrive to pass the right include files on the command line: it's right there in the file for Hugs to see. (And vice-versa, of course.) > Perhaps on GHC you should be required to "register" the top module > in your program first, maybe something like > registerModule(__stginit_Main); > that way you can register multiple modules (which isn't possible at > the moment, you have to have another module which imports all the > others). What does that do? Is it for threading, GC, profiling, ...? Do I have to unregister the module later? ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Library archives
> Under .NET each DLL has its own namespace, so the [lib] spec is > needed to disambiguate. Since it's a namespace issue, I'd feel > better if on .NET the name of the C function took a different form > (perhaps .) and [lib] is removed from the spec. Isn't that just a different syntax for the same thing? The thing I don't understand here is why .net issues affect the ccall calling convention and not the dotnet calling convention? I'm totally happy with defining dotnet to be ccall plus [lib] (or lib.) specifications (much as stdcall is defined as a small delta on ccall). I know what that means and it is implementable on platforms which support dotnet. It is trying to make C fit into the .net scheme of things which causes problems. > BTW I didn't know the spec was in CVS somewhere... where exactly? haskell-report/ffi (and don't forget the grammar.sty file I pointed at in the commit message) -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec
> Hmmm, the garbage collector is a black box and has its own > complicated heuristics for managing memory usage, but you are > describing a mechanism that depends rather heavily on certain > assumed behaviours. At the least, that gives the garbage collector > less flexibility to change its own behaviour, lest it invalidate the > assumptions made by the external allocator. This is a situation where something is better than nothing. The heuristics we used with Hugs were pretty lame -I don't think they even worked as I'd intended- but they were good enough to get the job done. In fact, the application ran brilliantly: small GC pauses, little wasted memory. So, no, I don't see a heavy dependence or an extreme sensitivity to GC behaviour. >> (In the image processing example, images were megabytes and an >> expression like (x + (y * mask)) would generate 2 intermediate >> images (several megabytes) while doing just 2 reductions in >> Haskell.) > I think I'd be tempted to try to use a more predictable allocation > scheme given the size of the objects involved. Perhaps arenas? Aren't arenas a form of explicit memory management? Explicit memory management was inappropriate for this example because the objects were manipulated by non-IO operations: programmers couldn't track the lifetime of an object because it depended on the order that thunks were evaluated. Note that this is _exactly_ what MallocPtrs/ ForeignObjs/ ForeignPtrs were invented to deal with. (Indeed, the first ever application was a (different) image processing library.) > But there seems to be no way to reasonably decide how often one > should call these. Doesn't it depend on the garbage collector's own > parameters too? It probably does. It might even vary from one kind of application to another. But I don't think it is very sensitive to those parameters. The goal here is to behave tolerably (e.g., not do a full GC on a 100Mb heap 20 times per second) not optimally. [other topics answered separately] -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec
> At the moment, there are two kinds of initialisation done for each > module: Both ELF and DLLs on Windows provide a way of specifying initializers. Or, easier yet, since the user is already using the hs_init function, you could use that. The way you'd do that in ELF is to define a special section 'hs_initializers'. Every module would contain a single object in that section: the address of the initializer. (In gcc you do this by attaching an attribute to the variable. In asm it is even easier since assemblers directly expose sections to the programmer.) The linker will do what it does with all sections: concatenate the pieces from all object files. hs_init would treat the section as an array of function pointers. I'm sure that the Windows linker must have a similar mechanism - the main trick is figuring out what name to give the sections. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Cheap ForeignPtr allocation
Can you achieve the same performance gain by adding some rewrite rules? -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Proposed change to ForeignPtr
[Now that we've gotten the library specification issue out of the way, I'd like to revive discussion of this topic. Last time we discussed it, there seemed to be concensus that we would make a change but I didn't get much response when I made a concrete proposal. I'd like to resolve this promptly so that the impending Hugs release can match what the spec says (or vice-versa...).] Since requiring ForeignPtr.newForeignPtr would require preemptive concurrency (see previous discussion), I propose the following changes: 1) Add these functions: makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO (ForeignPtr a) attachForeignPtrFinalizer :: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO () It is implementation defined whether the free functions are allowed to call Haskell functions. 2) Remove newForeignPtr and addForeignPtrFinalizer [GHC can go ahead and list them as non-standard extensions] There's a minor issue about whether the old function names should be reused (leaving GHC to come up with its own names) or not. I have ceased to care either way. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Cheap ForeignPtr allocation
> Hi Folks, I'd like to propose two new functions for the ForeignPtr > interface: I'm a bit uneasy about this. It seems like decisions made on technological grounds (i.e., because they enable an optimization) tend to date rather quickly because the technology tends to change too. The functions may cover a fairly common case of using ForeignPtrs (though I've never used ForeignObjs or MallocPtrs that way) - but most code doesn't use ForeignPtrs so it's a common case of an uncommon thing. It'd be a bit more compelling if the combos made things easier (as the withFoo functions do) or raised the level of abstraction. I'm inclined to say: add it to GHC for now and prod us again once a decent body of examples builds up. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Cheap ForeignPtr allocation
> Nevertheless, I think even without the tricks I'm using in GHC, the > case where a ForeignPtr is used in conjunction with malloc()/free() > is one which is likely to be optimisable in any system with its own > memory management. I wasn't meaning so much that only GHC could take advantage of it (though I think that is true at present) but that someone might come along next week with a technique which avoids the problem altogether. > [...] using a ForeignPtr here, with free() as its finalizer, adds so > much overhead that [...] Where is the overhead coming from? Is it the cost of a C call or the cost of the standard malloc library? If the latter, I imagine that a custom allocator would have similar performance to using pinned objects. (I'm sort of assuming that pinned objects are more expensive than normal objects.) btw I don't know if it's relevant but there's an important semantic difference between allocating on the GHC heap and allocating on the C heap. The C version can be manipulated by the malloc.h functions. In particular, you can call realloc on it. Do that on the GHC heap version and... well, it won't be pretty. I don't know if this is relevant because using realloc with ForeignPtr'd memory is likely to be a delicate procedure no matter how it got allocated. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Proposed change to ForeignPtr
> I thought we have settled this already. See my appended email from > three weeks ago. The only reason I haven't changed it in the spec > yet is that I wanted to give people time to object. Nobody has. > So, I would deem this accepted (reusing the old function names btw). Sorry, yes you had. I'd failed to mark it as done in my email. I'll tweak Hugs to conform. A ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: module Data.Bits
> The FFI spec says that the operations called 'shift' and 'rotate',
> shift and rotate their argument to the right. However, the GHC
> implementation in CVS shifts and rotates to the left, and is
> documented to do so.
> Who is right? FWIW, before I read any documentation I expected the
> undecorated forms to be rightward.
They shift to the left as in C, etc.
This makes the specification come out nice and clean - you're
multiplying the number by 2^n instead of 2^{-n}.
A
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: module Data.Bits
> Alastair: it was your design originally I believe, any thoughts? Actually, I think it was Lennart's - though he might disown our variant... The change seems fine. (Actually, it feels a bit like another instance of defining the language to allow particular optimizations in particular technology... but I can only hold up progress for so long :-) -- A ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: hs_init() & friends
> So, my proposal is to: > [...] Another possibility is to say that we don't really understand the issues well enough to put in the standard yet and just comment out that section. I think only GHC implements anything like this (correct me if wrong, Malcolm) and they haven't used it in the way John Meacham is interested in. Dunno... if you think we understand it well enough, go ahead. A ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
I think the thing to do is add the existing performGC to the standard
(perhaps giving it an hs_ prefix in the process) and leave development
of an extended version of the function for when the GHC folk (or
anyone else with a generational collector) decide they want a
forcefulness argument. Come that day, we'd define:
void performGC(void) { performPartialGC(0); }
(or whatever it is you do to force a full collection).
A
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Proposed change to ForeignPtr
> I have changed this in the spec now. I attach the wording used in > the spec. Looks good. A ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: module Data.Bits
> Errm, but in C there is no unified shift operator. You have << for
> left shift and and >> for right shift, and a negative shift is
> undefined.
[blush]
>> This makes the specification come out nice and clean - you're
>> multiplying the number by 2^n instead of 2^{-n}.
> Errm, but then right shift comes out as dividing by 2^{-n}, instead
> of 2^n. For a unified shift operation, I don't think there is any
> good reason to prefer one direction over the other, since there is
> no precedent in another language (AFAIK).
I think this spec (for >0 == left shift)
shift x n = x * 2^^n
is simpler than (for >0 == right shift)
shift x n = x * 2^^(-n)
[Ok, I probably need a few from/to Fractionals added to those specs]
--
A
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Proposed change to ForeignPtr
> I think this is all a rather murky area. Consider two systems, > let's call them Haskell and Foogle, which both operate heaps and do > their own storage allocation, but also communicate over similar > FFIs. This is indeed a murky area. Especially tricky when you have two GC's (even if one is nothing more than reference counting) is the issue of cycles that span the two systems: neither side is willing to release its pointer into the other world until the other side releases its pointer. (See section 5.2 of http://www.reid-consulting-uk.ltd.uk/alastair/publications/FP94.ps.gz for more on the problem and a sketch of a fix.) > We might very reasonably have situations where fairly complex > inter-language pointers exist, so for example Haskell holds a > ForeignPtr to something in the Foogle heap; the pointed-to Foogle > object in turn references a Haskell object (presumably provided via > StablePtr). Now suppose Haskell wants to drop the ForeignPtr. Then > the logical thing for the finalizer to do is to tell Foogle that > Haskell is no longer interested in the Foogle object. This then > gives Foogle the chance on its own garbage collection to in turn > drop the Haskell StablePtr. In turn this means somehow running > StablePtr.freeStablePtr. However this scheme I don't know if that's > legal, because the Haskell finalizer you need to run "freeStablePtr" > is indirectly provoked by the initial Haskell finalizer. We should provide a C function hs_freeStablePtr and explicitly say that it is safe to call this from inside a finalizer. > Of course you would need at least reference counters (if you can > guarantee there are no cycles containing both languages) or > something more powerful otherwise, but reference counters at least > can be provided. At least half the times I've used ForeignPtrs (aka ForeignObjs aka MallocPtrs), I've had to implement my own reference counting so, yes, it does seem like it'd be good to include reference counting in the implementation somehow. I don't remember ever finding a good way to do it though. The design seems obvious enough but it seems there's always some odd little wrinkle like the object already having its own reference counting implementation or the other world wanting to talk about foo_t* (i.e., the type the external library provides) instead of hs_stablePtr_t. In particular, if I put a wrapper around an object so that I can attach a reference counter, you somehow always find yourself faced with the problem that a C library hands you a pointer to an unwrapped object and you have to try to track down the reference counter for it. Any ideas welcome. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: module Data.Bits
> Another conflict between the FFI spec and the current library > implementation: the spec says > "The function bitSize returns 0 for types that don't have a > fixed bitsize (e.g. Integer)." > whereas the current ghc implementation defines bitSize of Integer as > a runtime error. I think both are unfortunate - we should just admit that not every type has a bitsize and return a Maybe. But that doesn't seem likely to happen... It isn't clear which is the lesser of the two evils: - returning 0 is easy to catch but equally easy to carelessly forget - raising an exception is harder to catch but has the huge advantage that the failure is close to the broken code. (Does NHC support non-deterministic exceptions? Dead easy to add if not.) Hugs is silent on the issue because we never implemented Bits Integer. I think I prefer the GHC semantics. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Proposed change to ForeignPtr
> I appreciate it's not Malcolm Wallace's fault that NHC can't > implement a form of dynamic wrapper which can be used at all times. > However since I do not think this problem is going to go away, would > it not be better to put it in the specification but admit that NHC > can't do it? Otherwise the FFI specification will be very good (as > it is) if you want to call languages like C, but will be inadequate > if you want to communicate with other languages which also know > about things like concurrency and garbage-collection, such as > Haskell or even Java. I think the two of you are getting caught up in implementation details (which always seem fixable) instead of looking at fundamentals (which tend not to be). The issue is that allowing finalizers to be Haskell code or to call Haskell code forces us to add preemptive concurrency to every Haskell implementation. Hugs has cooperative concurrency but that isn't enough, it has to be preemptive. To see why, suppose Hugs is busy calculating sum [1..100] and a garbage collection finds it has to run a Haskell function, it can't run it straight away because that would preempt the original execution. Likewise, it can't run it immediately that the GC ends - that's still preemption. Hugs is probably busy evaluating a + or (:) or the like when the GC happens, what if we wait until that finishes and then run the finalizers? Maybe. The world is in a cleaner state at this point but it's still tricky. And, again, you've just added most of the machinery required for implementing preemptive concurrency. Hugs does allow context switches in the IO monad. Could we allow it there? Maybe, we could. It wouldn't do you much good though because Haskell can spend arbitrarily long running pure code so your finalizers might not get run for a long time or maybe never. So what's the objection to adding preemptive concurrency? 1) The goal was to be able to call foreign code and have it call us. It seems like there's something wrong if doing that forces you to add concurrency. 2) Preemptive concurrency is one of those things that you can't just paint on later - you really have to rewrite the whole thing from scratch because it's just too hard to find all the pieces that make the assumption that they are single-threaded. 3) Adding preemptive concurrency guarantees a steady source of hard to repeat bugs - people aren't very good at writing concurrent code. Cooperative concurrency is much easier to handle because context switches happen at times when both tasks are ready for it: the sender because it knows it is passing control over, the receiver because it put itself in a clean state before it handed over control. You ask how garbage collection can work if you can't invoke it at arbitrary times? You ask what use foreign export is if you can't call the exported function at arbitrary times? [I hope I'm paraphrasing your questions appropriately] Both questions start by assuming that you have Haskell and Foosh [or whatever your other language is called] running in separate threads (otherwise things wouldn't happen at arbitrary times). Put them in a single thread and there's no problem: you're not doing things at arbitrary times, you're doing them at quite specific times: Foosh had control and it called into Haskell or Haskell had control and it called into Foosh. In other words, the problems you raise come from wanting preemptive concurrency so it's no wonder that the ffi extension doesn't address them. We could start working on a concurrency extension but that's a different beast. That said, we probably do need a few hooks to make Haskell talk to other GCs better. See the paper I pointed at this afternoon for a sketch of what you need if you're to avoid space leaks. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Proposed change to ForeignPtr
> [snip] No, you do not really need separate threads for this problem > to occur. All you need is, say, Hugs to call a GHC-exported > function as a finalizer, in the same OS thread, GHC to run a garbage > collection during this function, and the garbage collection in turn > to want to run a Hugs finalizer, before the finalizer called from > Hugs has finished. The Hugs and GHC runtimes can talk to each other just fine (or, if they can't it's a simple oversight and well fix it). There's no problem with GHC and Hugs each telling each other that some object they own has one less pointer to it. Next time it is convenient for the runtime, it can run a GC, perhaps recognize that there's a few GHC objects it can release and it tells the GHC runtime that it can release them. The reason we can do this is because it has limited scope: just a few data structures have to be tweaked to avoid GHC coming in when Hugs' data structures are in an inconsistent state. It's quite a different matter to allow arbitrary Haskell code to be run - that means the entire runtime system and libraries have to be made reentrant. > Of course one wouldn't normally want to link GHC > from Hugs, but if even these two cannot be made to meet, I don't > know how you expect Haskell to call anything else with a reasonably > flexible GC system; it puts the kybosh on Java for example, which I > am fairly sure makes plenty of use of both callbacks and finalizers. That's fine, they can have all the finalizers they want. The finalizers can fiddle with things in the runtime system to tell the GC whatever they want. > In any case it seems to me just as dangerous to assume that the > implementation does not use OS threads, as to assume it does. The internal structure of your apps is up to you - use locks to avoid using single-threaded code in multithreaded manner. > You are effectively writing on top of the FFI document "If your > program does this perfectly reasonable combination of finalizers, it > will fall over in an undefined way should the implementation use OS > threads; furthermore there is no way around this". Basically the > fact that there is only one OS thread is an implementation detail, > not something that the user should have to think about. Programmers are used to dealing with code which is single threaded or not reentrant. It's quite common. > Is it really the case that neither NHC nor Hugs can implement a list > of actions to be taken at the first convenient point after GC has > finished without implementing the whole machinery of preemptive > concurrency? I take Malcolm Wallace's word for it that it isn't > trivial, but why do you need for example asynchronous interruption > of Haskell threads, wait queues, or time slices? Surely what you > need is some way of backing up the state upon return from GC in such > a way that you can run the queued IO actions, which may be hard but > is a long way off preemptive concurrency. The way GHC implements preemption is an optimized form of: set a bit when preemption is needed; and make sure that generated code will test that bit whenever it is in a position to perform a context switch. What you're asking Hugs and NHC to do is: add a function to a list whenever you have a finalizer to run; make sure the interpreter will test that bit whenever it is in a position to perform a context switch. It's basically the same. We don't have to mess around with signals to provide a regular timer interrupt but that's the easy bit of the code. We can probably avoid messing around with multiple C stacks. That's a significant saving but, it's the complexity of that is fairly self-contained - we could probably steal some code from some other language implementation. The cost is going over all data structures in the system making sure that operations on them are suitably atomic. One of the issues I remember from old versions of GHC was that some of the primops would do some work, allocate some memory, then finish the job. The classic error to make in that code was for the second half of the code to assume almost anything about what happened in the first half of the code: how much space is on the stack, does a table have free space in it, is this pointer into the middle of an object ok? The problem is the scope: every single data structure and every bit of code that accesses it has to be vetted and we have to keep it in mind as we maintain the code. It's a high price to pay and I don't think it's necessary (because all you really need is for the runtime systems to talk to each other in very limited ways). > If it's really impossible for NHC or Hugs to implement this, I think > I would still rather it was left to the NHC and Hugs documentation > to admit
Re: Proposed change to ForeignPtr
> BTW, having two languages with separated heaps interact is a big
> mess as soon as you can have cycles, which you usually cannot
> exclude. Alastair already pointed that out and Martin Odersky also
> has nice stories to tell about this.
Hmmm, way back in '94, my thought was that the only thing to do in the
presence of these cycles was to run the two GCs at once with a rather
intimate communication between them where one says 'I can reach X' the
other says 'ok, now I can reach Y', and eventually they both run out
of objects to trace and they can discard unreached objects.
The problem with this is that it runs into the same problems Malcolm
and I are so keen to avoid: the other language has to be able to
trigger GCs at more or less arbitrary times.
Since then, I've read a lot about non-stop concurrent GC which doesn't
need the two GCs to run simultaneously. Just thinking aloud, I wonder
if it would be any easier to implement? The communication would be
much the same ('I'm starting', 'I can reach X' and 'I'm done') but
there'd be no need to synchronize the GCs.
--
Alastair Reid [EMAIL PROTECTED]
Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
___
FFI mailing list
[EMAIL PROTECTED]
http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
> Do you want a stronger wording on what kind of garbage collection is > to be performed or do we want to keep it deliberately unspecified > (ie, leave it to the individual Haskell system)? It'd be nice to say that it has to be a full GC - but I've no idea how to specify that in a non-operational (i.e., implementation dependent) way. We could insert the word 'full' and leave it to people's imaginations? -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec: performGC
Alastair wrote about performGC (snipped) > It'd be nice to say that it has to be a full GC - but I've no idea > how to specify that in a non-operational (i.e., implementation > dependent) way. George Russell <[EMAIL PROTECTED]> writes: > I certainly don't think you should constrain implementations to be > able to perform a "full GC" in any sense. It is possible if > unlikely that someone will get along to implementing the > "HaskellKit", where GC is entirely dispensed with and replaced by > region analysis. Even if that's not the case, I certainly hope that > one of these days someone will get along to implementing a Haskell > compiler that does at least some easy region analysis. Region-based systems have the quite wonderful property that garbage is disposed of promptly - you don't have to wait for the next GC for the memory to be released. Which means that performGC becomes a nullop. [That said, I'm not sure that region-based analysis is all that easy in the presence of lazy evaluation since it is so critically tied to estimating lifetimes.] > Also there are probably hard-real-time GC algorithms (like Baker's > treadmill) or algorithms which are close to being hard-real-time > (like the train algorithm) where doing a "full GC" would be a major > pain. The desired property is that the runtime system releases all unreachable objects. To make the fine-grained GCs (i.e., those that do a little GC now and then instead of a full GC) do this, all you need is stop mutating or allocating objects and then keep invoking the GC until it gets round to where it was when you started. There's usually some kind of colouring or 'epoch' mechanism that lets you know when you're back where you started. Of course, you trash any real-time properties in the process - see recent mail by me in the archive or my old paper for ideas on how to lessen that or dream up your own ideas for how to make two real time GCs work together nicely. [Again, though, I'm not sure there's much danger of a RT GC being added to Haskell - it's too hard estimating execution time, memory usage, etc. so making your GC more predictable doesn't seem like a major win.] In short, I think the current design is perfectly adequate for interfacing current systems to C code and will extend nicely in the future as motivating examples make actual needs. For example, even if we had a finer grained version of performGC, I expect most people would find that performGC provided the functionality that they need so they'd write their own (using a finer-grained interface) if we didn't provide it for them - only those who have real time programs interfacing to Haskell would make use of the finer-grained interface. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: Updates to FFI spec
Alastair: >> (In the image processing example, images were megabytes and an >> expression like (x + (y * mask)) would generate 2 intermediate >> images (several megabytes) while doing just 2 reductions in >> Haskell.) Marcin: > OCaml allows the programmer to specify an approximate amount of > foreign memory in its wrapped C pointers. Maybe it's a good idea? We did that on the C/C++ side of things so that we could keep track of how much image memory was in use and try to trigger a GC when the amount of memory was significantly higher than the running average (or some such). I wonder why OCaml would do it on the Caml side? Maybe they include similar heuristics but as a standard part of the system instead of leaving it to each library writer to roll their own? Or could it control the amount of GC performed? If you know that all the objects you care about are in the youngest generation, you could decide to collect just the youngest generation. (Except that isn't always enough - releasing an object in the old generation might release a pointer into the new generation.) -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: ANN: H98 FFI Addendum 1.0, Release Candidate 7
> RC 7 of the FFI Addendum is now available from In adding mallocForeignPtr and friends to Hugs, I found that I needed the address of free to pass as a parameter. There's no suitable way to generate &free from MarshalAlloc.free (the obvious use of a Haskell wrapper would break the whole reason for the recent change to ForeignPtrs). Could we add &free to the export list of MarshalAlloc? foreign import ccall unsafe "stdlib.h &free" ptr_free :: FunPtr (Ptr a -> IO ()) I am currently using 'ptr_free' as the Haskell name for this pointer but I expect that a better name could be found with little effort. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: lightweight struct handling in FFI?
I made a start on writing an FFI howto at: http://www.reid-consulting-uk.ltd.uk/docs/ffi.html It is very rough, doesn't answer your specific question, has many typos and omissions, etc. but some of the links in the Tips and Tricks section (section 4) might help. To answer your question directly, I think it all depends on what you're doing. Sometimes the semantics of the operations make it easy. For example, if there is any notion of object identity (and it often isn't stated whether there is or not), you usually have to keep the master copy in the C world and not duplicate. If there is no notion of identity, no modification to the object, etc., it is often easiest to keep it on the Haskell side and copy over to the C side on each operation because it's easier. There may be a performance hit in doing this depending on the number of times it is copied in each direction but I don't think we have any numbers for a range of implementations (ghc numbers may be available). Without any effort at measurement, I'd say that using malloc is pretty slow (because C's malloc is pretty slow) and that ForeignPtr's are fairly low overhead but more than just copying a small object over. -- Alastair Reid [EMAIL PROTECTED] Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/ ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: ANN: H98 FFI Addendum 1.0, Release Candidate 7
> ... maybe I'm being stupid, but I don't see what the problem is. > Why can't mallocForeignPtr be implemented as: > mallocForeignPtrBytes size = do > r <- c_malloc (fromIntegral size) > newForeignPtr r ptr_c_free > > foreign import ccall unsafe "malloc" > c_malloc :: CInt -> Ptr a > foreign import ccall unsafe "&free" > ptr_c_free :: ForeignPtr (Ptr a -> IO ()) > i.e. it's completely independent of MarshalAlloc.malloc. If you > happen to know that MarshalAlloc.malloc is the same as C's malloc, > then you could use that instead, but you don't have to. That's fine and, of course, it can be implemented that way. I guess the issue is that if someone wanted to use MarshalAlloc.free as a finalizer they would not be able to do so. Since we don't guarantee that MarshalAlloc.malloc is "stdio.h malloc", they couldn't portably cons up a compatible &free. The relevance to mallocForeignPtr and friends is only that I happened to notice the problem while implementing them. -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
Re: addForeignPtrFinalizer
> The spec says there are no guarantees on the order in which the > finalizers are run. Doesn't this make this function almost > impossible to use? Suppose one finalizer frees the storage and the > other cleans up something it refers to. I seem to remember that Sigbjorn and Erik dealt with this problem by arranging some kind of GC-visible link between the objects so that until the second finalizer has run, the object will not be considered garbage. I can't remember the details though... Can you remind us how it worked, Sigbjorn? Was it in the heaven from hell paper? -- Alastair ___ FFI mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/ffi
