Re: using ghc with make

2006-08-25 Thread Frederik Eaton
Hi! I almost forgot that I never responded to this, sorry.

On Tue, Apr 18, 2006 at 11:37:12AM +0100, Simon Marlow wrote:
 Frederik Eaton wrote:
 I have a project which currently uses Cabal, and I would like to
 switch to using a plain Makefile. I have two examples of projects that use 
 Makefiles, darcs and jhc, 
 but
 they both appear to hand-code the list of dependencies for
 executables. The -M option to ghc should let us do this
 automatically, but either because of a deficiency in GHC or in GNU
 Make, that looks to be impossible or difficult. Does anyone have
 experience with this?
 
 Yes, GHC's build system works like this. We don't use --make at all
 when building GHC itself. However, I want to switch to using Cabal
 to build the libraries. Eventually I envisage using Cabal to build
 more parts of the GHC tree.
 
 There's certainly no reason that you can't use ghc -M with GNU make, and 
 that's something we will 
 continue to support.  What problems are you having?

OK, here:

http://www.haskell.org/ghc/docs/6.4.2/html/users_guide/separate-compilation.html

There is a section


SRCS = Main.lhs Foo.lhs Bar.lhs
OBJS = Main.o   Foo.o   Bar.o

...

cool_pgm : $(OBJS)
rm -f $@
$(HC) -o $@ $(HC_OPTS) $(OBJS)


and below that a section:


# Inter-module dependencies
Foo.o Foo.hc Foo.s: Baz.hi  # Foo imports Baz
Main.o Main.hc Main.s : Foo.hi Baz.hi   # Main imports Foo and Baz


Then, the documentation explains how to get rid of the second section
using 'ghc -M'. But IIRC it doesn't explain how to get rid of the
first part, which includes information that 'ghc --make' doesn't
require. Does that make sense? I may have it wrong/backwards.

Also, it would be useful to know about dependencies on libraries. 
Basically, the best possible situation would be if 'ghc' gave me a
list of everything whose {timestamp,checksum}+dependencies I should
check to make sure that an output file is up to date (but not in a
flattened form - i.e. for each file, only the *immediate* dependencies
should be listed, and those can have their own dependencies, etc.). 
Even better would be to know which environment variables have been
referenced by the compiler, and which directories in a search path
mustn't have modules of a certain name (since those modules would have
otherwise been used in preference to the ones which were actually
used). Then it would be possible to just store every output file of
'ghc' in a cache in my home directory, and for instance switching
between normal and profiling builds wouldn't require recompiling
anything (or creating a separate project directory for each set of
options) as it does in the current system. I don't think this is a
very high priority, but I think that having 'ghc' duplicate more of
'make's functionality is kind of going in the wrong direction - I
think compilers and build systems need to be able to evolve
separately.

 Since I'm not using the multi-compiler features of Cabal, I think it
 shouldn't be too hard to get the project-specific part of my Makefile
 down to the size of the .cabal file.
 
 Sure, but you also have a quite a lot of build system infrastructure to get 
 right.  Incedentally, 
 this is exactly what Cabal was meant to avoid.
 
 - I want to be able to rebuild specific targets, rather than building
 everything every time
 
 Multiple Cabal packages tied together with a simple Makefile could do this, 
 no?

Yeah, but it would require me to put things in separate directories.

 ...
 I think it would be prudent at some point to make Cabal build without --make 
 and to add 
 multiprocessor support.

It's more than just multiprocessor issues I'm thinking about - for
instance I might like to do a distributed compilation, or I might like
to cache output in a special place as I suggested above. I think it
would be very nice, someday, if 'Cabal' could provide dependency
information, or for instance a list of which commands it's going to
invoke to produce which files, so that the build system and
compiler/packager are not tied together.

Best regards,

Frederik

-- 
http://ofb.net/~frederik/
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-21 Thread Simon Marlow

Bulat Ziganshin wrote:


Thursday, April 20, 2006, 11:54:59 AM, you wrote:

lazyGet can only be used to read something that was written with 
lazyPut.  lazyPut writes the offset of the end of the serialised data at
the beginning, so that lazyGet can skip over it, and subsequent gets 
start from the next item in the stream.


the problem is what there is ONLY ONE read pointer, so it should be
impossible to intersperse reading with `get` and consuming structure
returned by `lazyGet`, either for BinMem or BinIO

... hmm, on the other side they don't interspersed because there is
only one call to the unsafeInterleaveIO. closures returned by lazyGet
just should be evaluated STRICTLY after all other `get` operation. and
changing the `getAt` implementation to the following:

getAt bh p = do p0 - tellBin bh
seekBin bh p
a - get bh
seekBin bh p0
return a

should omit even this restriction


Ah yes, I forgot about this subtle restriction.  The underlying 
principle is that an individual 'get' or 'lazyGet' is atomic: it 
traverses the whole stream before returning a result.   There may be 
'lazyGet' components in the result, but each one of those is atomic in 
the same sense.  So the file pointer never needs to be set to two values 
simultaneously.


It's a bit hacky, but it works nicely.


can you  recommend me paper to read about using Haskell class system?


well, how about this?


http://www.haskell.org/haskellwiki/Research_papers/Type_systems#Type_classes

There are several papers at the level you're interested in, I think: 
Lennart's Implementing Haskell Overloading is a good one, and you 
might find Type classes in Haskell (Hall/Hammond/Peyton Jones/Wadler) 
useful, also Implementing Type Classes (Peterson/Jones).


Cheers,
Simon
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-20 Thread Simon Marlow

Bulat Ziganshin wrote:


1) lazyGet/lazyPut. it's no problem to copy your implementation but i
still don't understand how lazyGet should work - it share the same
buffer pointer as one used in `get`. so `get` and consuming structure
returned by lazyGet should interfere


lazyGet can only be used to read something that was written with 
lazyPut.  lazyPut writes the offset of the end of the serialised data at 
the beginning, so that lazyGet can skip over it, and subsequent gets 
start from the next item in the stream.



btw, btw. Haskell type classes has many non-obvious peculiarities. for
example, it was not easy for to understand that Haskell resolve all
overloading at compile time but finds what overloaded function to call
at runtime (well, i can't even describe this behavior). can you
recommend me paper to read about using Haskell class system?


I recommend compiling a few programs and investigating the Core with 
-ddump-ds and -ddump-simpl to find out what GHC really does.


Overloading isn't all resolved at compile time, most of the time 
dictionaries of functions are passed around representing class predicates.


Cheers,
Simon

___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-20 Thread Einar Karttunen
On 20.04 12:06, Bulat Ziganshin wrote:
 my Streams library mainly consists of two parts - Streams and
 AltBinary. The streams part implements Handle-like interface
 (including such functions as vGetChar, vGetByte, vPutBuf, vSeek and so
 on) for various data sources - files, memory buffers, pipes, strings.
 m/m files support is planned but now has just preliminary
 implementation

Having these as separate would be very nice. I think that a separately
packaged AltBinary would be much easier to use for many people rather
than force a dependency on the rest of Streams.

- Einar Karttunen

___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-19 Thread Simon Marlow

Bulat Ziganshin wrote:

Hello Simon,

Tuesday, April 18, 2006, 3:02:20 PM, you wrote:



if that is due to the time of reading .hi files, my alternative Binary
library should help in some future



I'd be suprised if you could improve on GHC's binary library.  Using 
BinMem (reading/writing directly to memory), GHC's binary library is 
about as fast as it gets.  I'm sure yours wins when dealing with files,

though.


sorry, Simon, but when i found 10x difference in speed (6mb/s vs 60
mb/s) it was on membufs :)  although i can't say that the difference
anyway will be 10 times. for so fast lib the time required to traverse
lists is essential. there are also a lot of other possible problems. i
just asking - whether the time required for reading these (or any
other binary) files is essential for compilation speed?


It's been on my todo list for a while to benchmark the various Binary 
libraries, since there's a consensus that we need some kind of Binary 
functionality in Haskell'.


I said I'd be surprised if GHC's could be improved on.  And indeed, I am 
now surprised :-)  You do point out some places where it could be 
improved.  The first thing I would do is replace the IOUArray with a 
ForeignPtr now, since that lets you unbox the Ptr without losing garbage 
collection of the memory, and retains the ability to re-allocate the 
storage.  How does your library handle memory allocation, do you have to 
explicitly free the memory used for the buffer?


To answer your question, reading interface files isn't a bottleneck in 
GHC (although improving its speed is definitely worthwhile).



instance Binary Word16 where
  get h = do
w1 - getWord8 h
w2 - getWord8 h
return $! ((fromIntegral w1 `shiftL` 8) .|. fromIntegral w2)



first possible problem here is what getWord8 is not inlined. second -
using of checked arithmetic (operations on Ints may have additional
checks, unlike operations on Int#)


yes, using uncheckedShiftL would be better.


third - it's better to make all
operations on Int and only then pack all data to Int16 constructor


An Int16 is represented using an Int with sign-extension, so it should 
be the same.


I agree that making the format independent of word size would also be 
good for a general purpose Binary library, although it wouldn't be 
helpful for GHC.


Cheers,
Simon
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


RE: using ghc with make

2006-04-19 Thread Simon Peyton-Jones
GHC might well be able to make use of such stuff too.  In general, one would 
like to be able to treat a file much like a database, as you suggest, with 
binary serialisation of data structures into it.

GHC's serialisation also includes a simple communing-up mechanism for leaves, 
especially strings.  We build a kind of dictionary, to avoid repeatedly 
re-serialising the same string.  I guess that any good binary serialisation 
will want to do something similar.  (Or something more dynamic, a la arithmetic 
coding.)

Simon

| -Original Message-
| From: [EMAIL PROTECTED] [mailto:glasgow-haskell-users-
| [EMAIL PROTECTED] On Behalf Of John Meacham
| Sent: 19 April 2006 00:28
| To: glasgow-haskell-users@haskell.org
| Subject: Re: using ghc with make
| 
| On Tue, Apr 18, 2006 at 10:26:18AM +0400, Bulat Ziganshin wrote:
|   FWIW ginsu and DrIFT both use make and don't use ghcs --make feature. I
|   find this can be faster once your projects grow beyond a certain size as
|   it takes ghc a while to figure out which files need to be rebuilt with
|   --make
| 
|  if that is due to the time of reading .hi files, my alternative Binary
|  library should help in some future
| 
| Interesting, A big bottleneck in jhc right now is reading the (quite
| large) binary ho and hl files on startup. a few things I have wanted out
| of a binary library are:
| 
|  * the ability to create a hash of the structure of the underlying data
|type, to verify you are reading data in the right format.
|  * extensible type-indexed sets (implemented hackily in Info.Binary in
|jhc)
|  * being able to jump over unneeded data, as in go directly to the 112th
|record, or the third field in a data structure without having to
|slurp through everything that came before it.
|  * VSDB[1] style ACID updates as an option.
|  * VSDB style write-time optimized constant hash table. I don't mind
|spending extra time when writing library files to speed up their
|usage.
|  * mmap based reading.
| 
| I was going to get around to writing this sometime, but perhaps there is
| room for a collaborative project in there. Is your code available
| somewhere bulat?
| 
| John
| 
| [1] VSDB is my very simple database that ensures full ACID semantics using
|   just the file guarentees of unix, including the weaker guarentees of
|   NFS. http://repetae.net/john/computer/vsdb/
|   Sort of like STM on the filesystem.
| 
| --
| John Meacham - ⑆repetae.net⑆john⑈
| ___
| Glasgow-haskell-users mailing list
| Glasgow-haskell-users@haskell.org
| http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-18 Thread Niklas Sorensson

Frederik Eaton wrote:

Hi all,

I have a project which currently uses Cabal, and I would like to
switch to using a plain Makefile. 


I use something like the attached Makefile. I don't have any experience with big projects though, so 
it could be that regenerating dependencies all the time is expensive.


/Niklas
## This Makefile compiles all haskell sources files in current
## directory to an executable with the same name as the last level of
## the current directory path. Change the following variables to alter
## this behaviour

EXEC= $(notdir $(shell pwd))
HSRCS   = $(wildcard *.hs)
HOBJS   = $(addsuffix .o, $(basename $(HSRCS)))

HCFLAGS = -O3 -fglasgow-exts
HLFLAGS = 

## Link rule
$(EXEC):$(HOBJS)
ghc $(HLFLAGS) -o $@ $^

## Build rule
%.o %.hi:   %.hs
ghc $(HCFLAGS) -c -o $(addsuffix .o, $(basename $@)) $

## Make dependencies
depend.mk: $(HSRCS)
@echo Making dependencies ...
ghc $(HCFLAGS) -M -optdep-f -optdepdepend.mk $^

-include depend.mk
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-18 Thread Simon Marlow

Frederik Eaton wrote:


I have a project which currently uses Cabal, and I would like to
switch to using a plain Makefile. 


I have two examples of projects that use Makefiles, darcs and jhc, but
they both appear to hand-code the list of dependencies for
executables. The -M option to ghc should let us do this
automatically, but either because of a deficiency in GHC or in GNU
Make, that looks to be impossible or difficult. Does anyone have
experience with this?


Yes, GHC's build system works like this.  We don't use --make at all 
when building GHC itself.  However, I want to switch to using Cabal to 
build the libraries.  Eventually I envisage using Cabal to build more 
parts of the GHC tree.


There's certainly no reason that you can't use ghc -M with GNU make, and 
that's something we will continue to support.  What problems are you having?



Since I'm not using the multi-compiler features of Cabal, I think it
shouldn't be too hard to get the project-specific part of my Makefile
down to the size of the .cabal file.


Sure, but you also have a quite a lot of build system infrastructure to 
get right.  Incedentally, this is exactly what Cabal was meant to avoid.



- I want to be able to rebuild specific targets, rather than building
everything every time


Multiple Cabal packages tied together with a simple Makefile could do 
this, no?



- I want better dependency inference, for instance I don't want to
relink every executable every time I do a build (I think this is fixed
in newer versions of ghc, but not the one I use)


Yes, fixed in 6.4.2

The comments about --make are interesting: ceratinly we're aware that it 
has a scalability problem, and it certainly isn't conducive to a 
frequent edit/compile/test cycle, which is one reason we don't use it in 
GHC.  However, for a straight single-CPU build, from scratch, for a 
large program, and if you have a lot of memory, it is much faster than 
make.  I don't have figures for GHC to hand, but I believe it is on the 
order of a factor of 2 or 3.  I would expect 'make -j4' to definitely 
win on a 4-core box.  We do have experimental patches for GHC to make 
--make work across multiple CPUs too, but you don't get linear speedup 
(single-threaded GC is one bottleneck).


I think it would be prudent at some point to make Cabal build without 
--make and to add multiprocessor support.


FWIW, I don't think 'ghc -c Foo.hs Bar.hs' goes any faster than 
separately compiling the two files, it doesn't cache anything between 
the two.


Cheers,
Simon
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-18 Thread Simon Marlow

Bulat Ziganshin wrote:

Hello John,

Tuesday, April 18, 2006, 3:25:35 AM, you wrote:



FWIW ginsu and DrIFT both use make and don't use ghcs --make feature. I
find this can be faster once your projects grow beyond a certain size as
it takes ghc a while to figure out which files need to be rebuilt with
--make



if that is due to the time of reading .hi files, my alternative Binary
library should help in some future


I'd be suprised if you could improve on GHC's binary library.  Using 
BinMem (reading/writing directly to memory), GHC's binary library is 
about as fast as it gets.  I'm sure yours wins when dealing with files, 
though.


Cheers,
Simon
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-18 Thread John Meacham
On Tue, Apr 18, 2006 at 10:26:18AM +0400, Bulat Ziganshin wrote:
  FWIW ginsu and DrIFT both use make and don't use ghcs --make feature. I
  find this can be faster once your projects grow beyond a certain size as
  it takes ghc a while to figure out which files need to be rebuilt with
  --make
 
 if that is due to the time of reading .hi files, my alternative Binary
 library should help in some future

Interesting, A big bottleneck in jhc right now is reading the (quite
large) binary ho and hl files on startup. a few things I have wanted out
of a binary library are: 

 * the ability to create a hash of the structure of the underlying data
   type, to verify you are reading data in the right format.
 * extensible type-indexed sets (implemented hackily in Info.Binary in
   jhc)
 * being able to jump over unneeded data, as in go directly to the 112th
   record, or the third field in a data structure without having to
   slurp through everything that came before it.
 * VSDB[1] style ACID updates as an option.
 * VSDB style write-time optimized constant hash table. I don't mind
   spending extra time when writing library files to speed up their
   usage.
 * mmap based reading.

I was going to get around to writing this sometime, but perhaps there is
room for a collaborative project in there. Is your code available
somewhere bulat?

John

[1] VSDB is my very simple database that ensures full ACID semantics using
  just the file guarentees of unix, including the weaker guarentees of
  NFS. http://repetae.net/john/computer/vsdb/
  Sort of like STM on the filesystem.

-- 
John Meacham - ⑆repetae.net⑆john⑈
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-17 Thread John Meacham
On Mon, Apr 17, 2006 at 10:57:32PM +0100, Frederik Eaton wrote:
 I have two examples of projects that use Makefiles, darcs and jhc, but
 they both appear to hand-code the list of dependencies for
 executables. The -M option to ghc should let us do this
 automatically, but either because of a deficiency in GHC or in GNU
 Make, that looks to be impossible or difficult. Does anyone have
 experience with this?

FWIW ginsu and DrIFT both use make and don't use ghcs --make feature. I
find this can be faster once your projects grow beyond a certain size as
it takes ghc a while to figure out which files need to be rebuilt with
--make, especially when a preprocessor is used. The main reason I have
found to use --make is it makes concurrent profiling and non-profiling
builds a lot easier.

 http://repetae.net/john/computer/ginsu/
 http://repetae.net/john/computer/haskell/DrIFT/

both require an explicit 'make depend' whenever you change the import
list.

it would be nice if ghc could spit out the dependencies to a file as a
side effect whenever using '--make' since it collects that info anyway.

ginsu is particularly tricky in order to get hierarchical modules to
play nice with 'automake' and 'autoconf'. it has to create symbolic
links from internal files to top level ones. If better solutions exist,
I'd love to hear of them.

 The compile / fix compiler errors cycle is an important part of the
 development process for me, and so I want recompilation to go as
 quickly as possible. By the way, the GHC user manual claims that using
 --make is much faster than using a Makefile, but in a test on a small
 program, the difference was not significant (16 seconds vs. 18
 seconds). Furthermore, some build systems are able to combine targets,
 e.g. running ghc -c Bar.hs -c Foo.hs instead of ghc -c Bar.hs; ghc
 -c Foo.hs, which should eliminate the already small difference.

indeed, for a project the size of jhc (~150 modules) 'make' is
quite signifigantly faster than '--make'.

John

-- 
John Meacham - ⑆repetae.net⑆john⑈
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Re: using ghc with make

2006-04-17 Thread Neil Mitchell
Hi,

 The compile / fix compiler errors cycle is an important part
 of the development process for me, and so I want
 recompilation to go as quickly as possible.

If you are not using any GHC specific features, then I find that using
Hugs for compiler errors, then once Hugs is happy moving on to GHC can
massively speed up development time.

FWIW, I have been using this strategy for my work on Yhc (~150
modules) and get times of about 4 seconds for Hugs to load, parse,
type check, and times of about 1 min for GHC --make to just do
nothing, a complete build is about 15 mins. It makes fixing compile
time errors much more fun.

Thanks

Neil
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users