Hi Gwern, The stack overflow problem is a known problem. See here:
http://www.ultraviolet.org/mail-archives/haskell-cafe.2008/msg08659.html In our project we use patched version of Data.Binary. I sent the required patches to the maintainer of Data.Binary but they were refused. I don't remember what exactly was the reason. I can't find the e-mail. I also did some changes in the library which make the serialization more compact. Without them the binary file was bigger than the textual file that we were using before. You can find the version that we use here: http://code.haskell.org/gf/src/Data/ In general I don't like to keep branches of external libraries but I am too tired to fight with different opinions and I just took the pragmatic side. Maybe this version will be useful in Yi also. Regards, Krasimir On 2/25/09, Gwern Branwen <[email protected]> wrote: > On Tue, Feb 24, 2009 at 11:36 AM, Gwern Branwen <[email protected]> wrote: > > On Tue, Feb 24, 2009 at 2:43 AM, Jean-Philippe Bernardy > > <[email protected]> wrote: > >> > >> On Tue, Feb 24, 2009 at 5:00 AM, Gwern Branwen <[email protected]> wrote: > >> > >>> After a fair bit of hacking, that turned out to be a workable > >>> solution. I just stored ArticleDB in it (don't need anything else). > >>> Had a bit of trouble working out the monads and was shortly flummoxed > >>> by the Initializable stuff > >> > >>>(until I realized that that typeclass was > >>> how the case of no-defined-variable was handled, by calling the > >>> 'initial' function) > >> > >> It would help if you documented the parts that were opaque to you; > >> a patch with haddock comments would be great. > > > > I'll see if I can do some documenting this week. It's not all that > > clear where the Haddocks would go, though - one can't really use the > > Buffer.Misc functions without putA/getA, and I'm not really sure what > > they do without those functions. Do I put the comment about what > > Initializable means by the typeclass or somewhere else? > > > >>> But the initial load is still several seconds. How would Data.Binary > >>> help there? Is it intended to be a faster version of '(read $ readFile > >>> foo) :: ArticleDB)'? > >> > >> Yep. > >> > >> Cheers, > >> -- JP. > > > > I've been looking, and it looks like that might work. At least, > > Data.Binary claims > > http://hackage.haskell.org/packages/archive/binary/0.5/doc/html/Data-Binary.html > > to have a 'Binary e => Binary (Seq e)', and I'm sure there's an > > instance for String/[Char]. I wonder whether binary-strict might not > > be better though? > > Think some sort of strictness may be required. I tried switching to > Data.Binary, and made the following changes: > > hunk ./Yi/IReader.hs 11 > +import Data.Binary > hunk ./Yi/IReader.hs 22 > -import qualified Data.ByteString.Char8 as B (pack, unpack, readFile, > writeFile) > hunk ./Yi/IReader.hs 51 > -writeDB adb = do io . forkIO . join . fmap (flip B.writeFile $ B.pack > $ show adb) $ dbLocation > +writeDB adb = do io . forkIO . join . fmap (flip encodeFile adb) $ dbLocation > hunk ./Yi/IReader.hs 56 > -readDB = io $ rddb `catch` (const $ return empty) -- May seem silly > to read in as bytestring > - where rddb = fmap B.readFile dbLocation >>= fmap (read . > B.unpack) -- and then unpack it, $ > - > -- but - is strict > +readDB = io (dbLocation >>= decodeFile) `catch` (\_ -> return empty)) > > This compiles without issue, but when one recompiles yi and runs M-x > iread, one gets a horrible error: > 'M-x ireadyi-i386-linux: /home/gwern/.yi/articles.db: openBinaryFile: > resource busy (file is locked)' > > I would think this just a lazy IO issue but for the fact that Yi seems > to loop and begin eating up as much memory as possible until the OS > kills it. (First screenshot.) > > Then I reasoned - perhaps Data.Binary has a different on-disk format > than Show, and the issue is that yi is trying to read a show-file as a > binary-file. So I went into GHCi, manually read in and parsed > articles.db, and wrote it back out with encodeFile. Sure enough, the > format was indeed different. I restarted yi, and tried again. No dice! > I get a locked error and then a 'error: stack overflow'. Oy. > > So I began bugging #haskell and Saizan gave me an ugly ByteString > recipe which worked. No idea why - my extensive tinkering in GHCi > didn't give me much of a intuition why I could do something like > 'encodeFile foo; bar <- decodeFile "foo"; foo == bar' and get a stack > overflow. > > It's incorrect - there's still an issue with file locking (the > background writes don't seem to go through). But I will admit > data.binary does seem to read articles.db pretty darn quickly. > > -- > gwern > > > > > --~--~---------~--~----~------------~-------~--~----~ Yi development mailing list [email protected] http://groups.google.com/group/yi-devel -~----------~----~----~----~------~----~------~--~---
