Re: [Haskell-cafe] Data.Binary poor read performance

2009-02-24 Thread Felipe Lessa
On Tue, Feb 24, 2009 at 7:42 PM, jutaro wrote: > instance Eq Object where >    (ObjC a) ≡ (ObjC b) = if typeOf a ≠ typeOf b >                                then False >                                else (Just a) ≡ cast b -- can someone explain > to me why this works? In fact, can't you just s

Re: [Haskell-cafe] Data.Binary poor read performance

2009-02-24 Thread Don Stewart
jnf: > > > wren ng thornton wrote: > > > > If you have many identical strings then you will save lots by memoizing > > your strings into Integers, and then serializing that memo table and the > > integerized version of your data structure. The amount of savings > > decreases as the number of

Re: [Haskell-cafe] Data.Binary poor read performance

2009-02-24 Thread jutaro
wren ng thornton wrote: > > If you have many identical strings then you will save lots by memoizing > your strings into Integers, and then serializing that memo table and the > integerized version of your data structure. The amount of savings > decreases as the number of duplications decrease

Re: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Bertram Felgenhauer
Felipe Lessa wrote: > On Tue, Feb 24, 2009 at 4:59 AM, Don Stewart wrote: > > Looks like the Map reading/showing via association lists could do with > > further work. > > > > Anyone want to dig around in the Map instance? (There's also some patches > > for > > an alternative lazy Map serialisatio

Re: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Paulo Tanimoto
Hello, On Tue, Feb 24, 2009 at 2:36 AM, Don Stewart wrote: > This idea was the motivation for the new Seq instance, which uses > internals to build quickly. > >    Encoding to disk, the dictionary, > >        $ time ./binary /usr/share/dict/cracklib-small >        "done" >        ./binary /usr/sh

Re: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Bertram Felgenhauer
Don Stewart wrote: > dons: [...] > Just serialising straight lists of pairs, [...] > And reading them back in, > > main = do > [f] <- getArgs > m <- decode `fmap` L.readFile f > print (length (m :: [(B.ByteString,Int)])) > print "done" Well, you don't actually

Re: Re[2]: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe]Data.Binary poor read performance]

2009-02-24 Thread Claus Reinke
btw, i always thought that it should be a way to overcome any export lists and go directly to module internals. limiting export is the way to protect programmer from errors, not security feature, and it should be left to programmer to decide when he don't need it. compilers should just be able to

Export restrictions :) Re[4]: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Bulat Ziganshin
Hello Svein, Tuesday, February 24, 2009, 3:47:44 PM, you wrote: >> btw, i always thought that it should be a way to overcome any export >> lists and go directly to module internals. limiting export is the way >> to protect programmer from errors, not security feature, and it should >> be left to

Re: Re[2]: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Svein Ove Aas
2009/2/24 Bulat Ziganshin : > Hello Felipe, > > Tuesday, February 24, 2009, 11:24:19 AM, you wrote: > >> Too bad 'Map' is exported as an abstract data type and it's not >> straighforward to test this conjecture. Any ideas? > > just make a copy of its implementation to test > > btw, i always thought

Re[2]: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Bulat Ziganshin
Hello Felipe, Tuesday, February 24, 2009, 11:24:19 AM, you wrote: > Too bad 'Map' is exported as an abstract data type and it's not > straighforward to test this conjecture. Any ideas? just make a copy of its implementation to test btw, i always thought that it should be a way to overcome any e

Re: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Felipe Lessa
On Tue, Feb 24, 2009 at 5:36 AM, Don Stewart wrote: > This idea was the motivation for the new Seq instance, which uses > internals to build quickly. The problem is that fromDistinctAscList is the best we can do right now. We don't have something like (|>) that runs in O(1) time, and trying to in

Re: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Don Stewart
felipe.lessa: > On Tue, Feb 24, 2009 at 4:59 AM, Don Stewart wrote: > > Looks like the Map reading/showing via association lists could do with > > further work. > > > > Anyone want to dig around in the Map instance? (There's also some patches > > for > > an alternative lazy Map serialisation, if

Re: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Felipe Lessa
On Tue, Feb 24, 2009 at 4:59 AM, Don Stewart wrote: > Looks like the Map reading/showing via association lists could do with > further work. > > Anyone want to dig around in the Map instance? (There's also some patches for > an alternative lazy Map serialisation, if people are keen to load maps --

Re: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-24 Thread Don Stewart
dons: > wren: > > Neil Mitchell wrote: > >> 2) The storage for String seems to be raw strings, which is nice. > >> Would I get a substantial speedup by moving to bytestrings instead of > >> strings? If I hashed the strings and stored common ones in a hash > >> table is it likely to be a big win? >

Re[2]: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-23 Thread Bulat Ziganshin
Hello Duncan, Tuesday, February 24, 2009, 5:13:05 AM, you wrote: > That's actually rather surprising. The system time is negligible and the yes, this looks highly suspicious. we use 1.5 seconds for 1.3 mb file, and 0.3 seconds for 0.3 mb file, so it looks like readFile monopolize cpu here, being

Re: Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-23 Thread Duncan Coutts
On Mon, 2009-02-23 at 17:03 -0800, Don Stewart wrote: > Here's a quick demo using Data.Binary directly. [...] > $ time ./A dict +RTS -K20M > 52848 > "done" > ./A dict +RTS -K20M 1.51s user 0.06s system 99% cpu 1.582 total > > > Ok. So 1.5s to decode a 1.3M Map. There may be be

Pickling a finite map (Binary + zlib) [was: [Haskell-cafe] Data.Binary poor read performance]

2009-02-23 Thread Don Stewart
wren: > Neil Mitchell wrote: >> 2) The storage for String seems to be raw strings, which is nice. >> Would I get a substantial speedup by moving to bytestrings instead of >> strings? If I hashed the strings and stored common ones in a hash >> table is it likely to be a big win? > > Bytestrings shou

Re: [Haskell-cafe] Data.Binary poor read performance

2009-02-23 Thread wren ng thornton
Neil Mitchell wrote: 2) The storage for String seems to be raw strings, which is nice. Would I get a substantial speedup by moving to bytestrings instead of strings? If I hashed the strings and stored common ones in a hash table is it likely to be a big win? Bytestrings should help. The big win

[Haskell-cafe] Data.Binary poor read performance

2009-02-23 Thread Neil Mitchell
Hi, In an application I'm writing with Data.Binary I'm seeing very fast write performance (instant), but much slower read performance. Can you advise where I might be going wrong? The data type I'm serialising is roughly: Map String [Either (String,[String]) [(String,Int)]] A lot of the String's