Re: using ghc with make
Hi! I almost forgot that I never responded to this, sorry.
On Tue, Apr 18, 2006 at 11:37:12AM +0100, Simon Marlow wrote:
Frederik Eaton wrote:
I have a project which currently uses Cabal, and I would like to
switch to using a plain Makefile. I have two examples of projects that use
Makefiles, darcs and jhc,
but
they both appear to hand-code the list of dependencies for
executables. The -M option to ghc should let us do this
automatically, but either because of a deficiency in GHC or in GNU
Make, that looks to be impossible or difficult. Does anyone have
experience with this?
Yes, GHC's build system works like this. We don't use --make at all
when building GHC itself. However, I want to switch to using Cabal
to build the libraries. Eventually I envisage using Cabal to build
more parts of the GHC tree.
There's certainly no reason that you can't use ghc -M with GNU make, and
that's something we will
continue to support. What problems are you having?
OK, here:
http://www.haskell.org/ghc/docs/6.4.2/html/users_guide/separate-compilation.html
There is a section
SRCS = Main.lhs Foo.lhs Bar.lhs
OBJS = Main.o Foo.o Bar.o
...
cool_pgm : $(OBJS)
rm -f $@
$(HC) -o $@ $(HC_OPTS) $(OBJS)
and below that a section:
# Inter-module dependencies
Foo.o Foo.hc Foo.s: Baz.hi # Foo imports Baz
Main.o Main.hc Main.s : Foo.hi Baz.hi # Main imports Foo and Baz
Then, the documentation explains how to get rid of the second section
using 'ghc -M'. But IIRC it doesn't explain how to get rid of the
first part, which includes information that 'ghc --make' doesn't
require. Does that make sense? I may have it wrong/backwards.
Also, it would be useful to know about dependencies on libraries.
Basically, the best possible situation would be if 'ghc' gave me a
list of everything whose {timestamp,checksum}+dependencies I should
check to make sure that an output file is up to date (but not in a
flattened form - i.e. for each file, only the *immediate* dependencies
should be listed, and those can have their own dependencies, etc.).
Even better would be to know which environment variables have been
referenced by the compiler, and which directories in a search path
mustn't have modules of a certain name (since those modules would have
otherwise been used in preference to the ones which were actually
used). Then it would be possible to just store every output file of
'ghc' in a cache in my home directory, and for instance switching
between normal and profiling builds wouldn't require recompiling
anything (or creating a separate project directory for each set of
options) as it does in the current system. I don't think this is a
very high priority, but I think that having 'ghc' duplicate more of
'make's functionality is kind of going in the wrong direction - I
think compilers and build systems need to be able to evolve
separately.
Since I'm not using the multi-compiler features of Cabal, I think it
shouldn't be too hard to get the project-specific part of my Makefile
down to the size of the .cabal file.
Sure, but you also have a quite a lot of build system infrastructure to get
right. Incedentally,
this is exactly what Cabal was meant to avoid.
- I want to be able to rebuild specific targets, rather than building
everything every time
Multiple Cabal packages tied together with a simple Makefile could do this,
no?
Yeah, but it would require me to put things in separate directories.
...
I think it would be prudent at some point to make Cabal build without --make
and to add
multiprocessor support.
It's more than just multiprocessor issues I'm thinking about - for
instance I might like to do a distributed compilation, or I might like
to cache output in a special place as I suggested above. I think it
would be very nice, someday, if 'Cabal' could provide dependency
information, or for instance a list of which commands it's going to
invoke to produce which files, so that the build system and
compiler/packager are not tied together.
Best regards,
Frederik
--
http://ofb.net/~frederik/
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
Bulat Ziganshin wrote: Thursday, April 20, 2006, 11:54:59 AM, you wrote: lazyGet can only be used to read something that was written with lazyPut. lazyPut writes the offset of the end of the serialised data at the beginning, so that lazyGet can skip over it, and subsequent gets start from the next item in the stream. the problem is what there is ONLY ONE read pointer, so it should be impossible to intersperse reading with `get` and consuming structure returned by `lazyGet`, either for BinMem or BinIO ... hmm, on the other side they don't interspersed because there is only one call to the unsafeInterleaveIO. closures returned by lazyGet just should be evaluated STRICTLY after all other `get` operation. and changing the `getAt` implementation to the following: getAt bh p = do p0 - tellBin bh seekBin bh p a - get bh seekBin bh p0 return a should omit even this restriction Ah yes, I forgot about this subtle restriction. The underlying principle is that an individual 'get' or 'lazyGet' is atomic: it traverses the whole stream before returning a result. There may be 'lazyGet' components in the result, but each one of those is atomic in the same sense. So the file pointer never needs to be set to two values simultaneously. It's a bit hacky, but it works nicely. can you recommend me paper to read about using Haskell class system? well, how about this? http://www.haskell.org/haskellwiki/Research_papers/Type_systems#Type_classes There are several papers at the level you're interested in, I think: Lennart's Implementing Haskell Overloading is a good one, and you might find Type classes in Haskell (Hall/Hammond/Peyton Jones/Wadler) useful, also Implementing Type Classes (Peterson/Jones). Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
Bulat Ziganshin wrote: 1) lazyGet/lazyPut. it's no problem to copy your implementation but i still don't understand how lazyGet should work - it share the same buffer pointer as one used in `get`. so `get` and consuming structure returned by lazyGet should interfere lazyGet can only be used to read something that was written with lazyPut. lazyPut writes the offset of the end of the serialised data at the beginning, so that lazyGet can skip over it, and subsequent gets start from the next item in the stream. btw, btw. Haskell type classes has many non-obvious peculiarities. for example, it was not easy for to understand that Haskell resolve all overloading at compile time but finds what overloaded function to call at runtime (well, i can't even describe this behavior). can you recommend me paper to read about using Haskell class system? I recommend compiling a few programs and investigating the Core with -ddump-ds and -ddump-simpl to find out what GHC really does. Overloading isn't all resolved at compile time, most of the time dictionaries of functions are passed around representing class predicates. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
On 20.04 12:06, Bulat Ziganshin wrote: my Streams library mainly consists of two parts - Streams and AltBinary. The streams part implements Handle-like interface (including such functions as vGetChar, vGetByte, vPutBuf, vSeek and so on) for various data sources - files, memory buffers, pipes, strings. m/m files support is planned but now has just preliminary implementation Having these as separate would be very nice. I think that a separately packaged AltBinary would be much easier to use for many people rather than force a dependency on the rest of Streams. - Einar Karttunen ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
Bulat Ziganshin wrote: Hello Simon, Tuesday, April 18, 2006, 3:02:20 PM, you wrote: if that is due to the time of reading .hi files, my alternative Binary library should help in some future I'd be suprised if you could improve on GHC's binary library. Using BinMem (reading/writing directly to memory), GHC's binary library is about as fast as it gets. I'm sure yours wins when dealing with files, though. sorry, Simon, but when i found 10x difference in speed (6mb/s vs 60 mb/s) it was on membufs :) although i can't say that the difference anyway will be 10 times. for so fast lib the time required to traverse lists is essential. there are also a lot of other possible problems. i just asking - whether the time required for reading these (or any other binary) files is essential for compilation speed? It's been on my todo list for a while to benchmark the various Binary libraries, since there's a consensus that we need some kind of Binary functionality in Haskell'. I said I'd be surprised if GHC's could be improved on. And indeed, I am now surprised :-) You do point out some places where it could be improved. The first thing I would do is replace the IOUArray with a ForeignPtr now, since that lets you unbox the Ptr without losing garbage collection of the memory, and retains the ability to re-allocate the storage. How does your library handle memory allocation, do you have to explicitly free the memory used for the buffer? To answer your question, reading interface files isn't a bottleneck in GHC (although improving its speed is definitely worthwhile). instance Binary Word16 where get h = do w1 - getWord8 h w2 - getWord8 h return $! ((fromIntegral w1 `shiftL` 8) .|. fromIntegral w2) first possible problem here is what getWord8 is not inlined. second - using of checked arithmetic (operations on Ints may have additional checks, unlike operations on Int#) yes, using uncheckedShiftL would be better. third - it's better to make all operations on Int and only then pack all data to Int16 constructor An Int16 is represented using an Int with sign-extension, so it should be the same. I agree that making the format independent of word size would also be good for a general purpose Binary library, although it wouldn't be helpful for GHC. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: using ghc with make
GHC might well be able to make use of such stuff too. In general, one would like to be able to treat a file much like a database, as you suggest, with binary serialisation of data structures into it. GHC's serialisation also includes a simple communing-up mechanism for leaves, especially strings. We build a kind of dictionary, to avoid repeatedly re-serialising the same string. I guess that any good binary serialisation will want to do something similar. (Or something more dynamic, a la arithmetic coding.) Simon | -Original Message- | From: [EMAIL PROTECTED] [mailto:glasgow-haskell-users- | [EMAIL PROTECTED] On Behalf Of John Meacham | Sent: 19 April 2006 00:28 | To: glasgow-haskell-users@haskell.org | Subject: Re: using ghc with make | | On Tue, Apr 18, 2006 at 10:26:18AM +0400, Bulat Ziganshin wrote: | FWIW ginsu and DrIFT both use make and don't use ghcs --make feature. I | find this can be faster once your projects grow beyond a certain size as | it takes ghc a while to figure out which files need to be rebuilt with | --make | | if that is due to the time of reading .hi files, my alternative Binary | library should help in some future | | Interesting, A big bottleneck in jhc right now is reading the (quite | large) binary ho and hl files on startup. a few things I have wanted out | of a binary library are: | | * the ability to create a hash of the structure of the underlying data |type, to verify you are reading data in the right format. | * extensible type-indexed sets (implemented hackily in Info.Binary in |jhc) | * being able to jump over unneeded data, as in go directly to the 112th |record, or the third field in a data structure without having to |slurp through everything that came before it. | * VSDB[1] style ACID updates as an option. | * VSDB style write-time optimized constant hash table. I don't mind |spending extra time when writing library files to speed up their |usage. | * mmap based reading. | | I was going to get around to writing this sometime, but perhaps there is | room for a collaborative project in there. Is your code available | somewhere bulat? | | John | | [1] VSDB is my very simple database that ensures full ACID semantics using | just the file guarentees of unix, including the weaker guarentees of | NFS. http://repetae.net/john/computer/vsdb/ | Sort of like STM on the filesystem. | | -- | John Meacham - ⑆repetae.net⑆john⑈ | ___ | Glasgow-haskell-users mailing list | Glasgow-haskell-users@haskell.org | http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
Frederik Eaton wrote: Hi all, I have a project which currently uses Cabal, and I would like to switch to using a plain Makefile. I use something like the attached Makefile. I don't have any experience with big projects though, so it could be that regenerating dependencies all the time is expensive. /Niklas ## This Makefile compiles all haskell sources files in current ## directory to an executable with the same name as the last level of ## the current directory path. Change the following variables to alter ## this behaviour EXEC= $(notdir $(shell pwd)) HSRCS = $(wildcard *.hs) HOBJS = $(addsuffix .o, $(basename $(HSRCS))) HCFLAGS = -O3 -fglasgow-exts HLFLAGS = ## Link rule $(EXEC):$(HOBJS) ghc $(HLFLAGS) -o $@ $^ ## Build rule %.o %.hi: %.hs ghc $(HCFLAGS) -c -o $(addsuffix .o, $(basename $@)) $ ## Make dependencies depend.mk: $(HSRCS) @echo Making dependencies ... ghc $(HCFLAGS) -M -optdep-f -optdepdepend.mk $^ -include depend.mk ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
Frederik Eaton wrote: I have a project which currently uses Cabal, and I would like to switch to using a plain Makefile. I have two examples of projects that use Makefiles, darcs and jhc, but they both appear to hand-code the list of dependencies for executables. The -M option to ghc should let us do this automatically, but either because of a deficiency in GHC or in GNU Make, that looks to be impossible or difficult. Does anyone have experience with this? Yes, GHC's build system works like this. We don't use --make at all when building GHC itself. However, I want to switch to using Cabal to build the libraries. Eventually I envisage using Cabal to build more parts of the GHC tree. There's certainly no reason that you can't use ghc -M with GNU make, and that's something we will continue to support. What problems are you having? Since I'm not using the multi-compiler features of Cabal, I think it shouldn't be too hard to get the project-specific part of my Makefile down to the size of the .cabal file. Sure, but you also have a quite a lot of build system infrastructure to get right. Incedentally, this is exactly what Cabal was meant to avoid. - I want to be able to rebuild specific targets, rather than building everything every time Multiple Cabal packages tied together with a simple Makefile could do this, no? - I want better dependency inference, for instance I don't want to relink every executable every time I do a build (I think this is fixed in newer versions of ghc, but not the one I use) Yes, fixed in 6.4.2 The comments about --make are interesting: ceratinly we're aware that it has a scalability problem, and it certainly isn't conducive to a frequent edit/compile/test cycle, which is one reason we don't use it in GHC. However, for a straight single-CPU build, from scratch, for a large program, and if you have a lot of memory, it is much faster than make. I don't have figures for GHC to hand, but I believe it is on the order of a factor of 2 or 3. I would expect 'make -j4' to definitely win on a 4-core box. We do have experimental patches for GHC to make --make work across multiple CPUs too, but you don't get linear speedup (single-threaded GC is one bottleneck). I think it would be prudent at some point to make Cabal build without --make and to add multiprocessor support. FWIW, I don't think 'ghc -c Foo.hs Bar.hs' goes any faster than separately compiling the two files, it doesn't cache anything between the two. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
Bulat Ziganshin wrote: Hello John, Tuesday, April 18, 2006, 3:25:35 AM, you wrote: FWIW ginsu and DrIFT both use make and don't use ghcs --make feature. I find this can be faster once your projects grow beyond a certain size as it takes ghc a while to figure out which files need to be rebuilt with --make if that is due to the time of reading .hi files, my alternative Binary library should help in some future I'd be suprised if you could improve on GHC's binary library. Using BinMem (reading/writing directly to memory), GHC's binary library is about as fast as it gets. I'm sure yours wins when dealing with files, though. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
On Tue, Apr 18, 2006 at 10:26:18AM +0400, Bulat Ziganshin wrote: FWIW ginsu and DrIFT both use make and don't use ghcs --make feature. I find this can be faster once your projects grow beyond a certain size as it takes ghc a while to figure out which files need to be rebuilt with --make if that is due to the time of reading .hi files, my alternative Binary library should help in some future Interesting, A big bottleneck in jhc right now is reading the (quite large) binary ho and hl files on startup. a few things I have wanted out of a binary library are: * the ability to create a hash of the structure of the underlying data type, to verify you are reading data in the right format. * extensible type-indexed sets (implemented hackily in Info.Binary in jhc) * being able to jump over unneeded data, as in go directly to the 112th record, or the third field in a data structure without having to slurp through everything that came before it. * VSDB[1] style ACID updates as an option. * VSDB style write-time optimized constant hash table. I don't mind spending extra time when writing library files to speed up their usage. * mmap based reading. I was going to get around to writing this sometime, but perhaps there is room for a collaborative project in there. Is your code available somewhere bulat? John [1] VSDB is my very simple database that ensures full ACID semantics using just the file guarentees of unix, including the weaker guarentees of NFS. http://repetae.net/john/computer/vsdb/ Sort of like STM on the filesystem. -- John Meacham - ⑆repetae.net⑆john⑈ ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
On Mon, Apr 17, 2006 at 10:57:32PM +0100, Frederik Eaton wrote: I have two examples of projects that use Makefiles, darcs and jhc, but they both appear to hand-code the list of dependencies for executables. The -M option to ghc should let us do this automatically, but either because of a deficiency in GHC or in GNU Make, that looks to be impossible or difficult. Does anyone have experience with this? FWIW ginsu and DrIFT both use make and don't use ghcs --make feature. I find this can be faster once your projects grow beyond a certain size as it takes ghc a while to figure out which files need to be rebuilt with --make, especially when a preprocessor is used. The main reason I have found to use --make is it makes concurrent profiling and non-profiling builds a lot easier. http://repetae.net/john/computer/ginsu/ http://repetae.net/john/computer/haskell/DrIFT/ both require an explicit 'make depend' whenever you change the import list. it would be nice if ghc could spit out the dependencies to a file as a side effect whenever using '--make' since it collects that info anyway. ginsu is particularly tricky in order to get hierarchical modules to play nice with 'automake' and 'autoconf'. it has to create symbolic links from internal files to top level ones. If better solutions exist, I'd love to hear of them. The compile / fix compiler errors cycle is an important part of the development process for me, and so I want recompilation to go as quickly as possible. By the way, the GHC user manual claims that using --make is much faster than using a Makefile, but in a test on a small program, the difference was not significant (16 seconds vs. 18 seconds). Furthermore, some build systems are able to combine targets, e.g. running ghc -c Bar.hs -c Foo.hs instead of ghc -c Bar.hs; ghc -c Foo.hs, which should eliminate the already small difference. indeed, for a project the size of jhc (~150 modules) 'make' is quite signifigantly faster than '--make'. John -- John Meacham - ⑆repetae.net⑆john⑈ ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: using ghc with make
Hi, The compile / fix compiler errors cycle is an important part of the development process for me, and so I want recompilation to go as quickly as possible. If you are not using any GHC specific features, then I find that using Hugs for compiler errors, then once Hugs is happy moving on to GHC can massively speed up development time. FWIW, I have been using this strategy for my work on Yhc (~150 modules) and get times of about 4 seconds for Hugs to load, parse, type check, and times of about 1 min for GHC --make to just do nothing, a complete build is about 15 mins. It makes fixing compile time errors much more fun. Thanks Neil ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
