new i/o library
Hello Simon i'm now write some sort of new i/o library. one area where i currently lacks in comparision to the existing Handles implementation in GHC, is the asynchronous i/o operations. can you please briefly describe how this is done in GHC and partially - why the multiple buffers are used? i'm now use just one buffer, which can contain read or write data, but not both - this buffer is just flushed before switching mode of operations. am i lose something due to this simplified algorithm? moreover, i have an idea how to implement async i/o without complex burecreacy: use mmapped files, may be together with miltiple buffers. for example, we can allocate four 16kb buffers. when one buffer is filled with written data, the program unmaps it and switches to use the next buffer. i don't tested it, but OS can guess that unmapped buffer now should be asynchronously written to disk. the same for reading - when we completely read one buffer, we can unmap it, switch to the second buffer and map the third so that the OS can asynchronously fill the third buffer while we are reading second. should this work, at least on the main desktop OSes? at least, mmap/VirtualAlloc available afaik on the all ghc-supported platforms, so this should work anywhere. of course, this scheme omits async i/o on sockets in Windows -- Best regards, Bulat mailto:[EMAIL PROTECTED] ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: new i/o library
On 27.01 13:10, Bulat Ziganshin wrote: i'm now write some sort of new i/o library. one area where i currently lacks in comparision to the existing Handles implementation in GHC, is the asynchronous i/o operations. can you please briefly describe how this is done in GHC and partially - why the multiple buffers are used? One simple optimization is that you can omit all buffering with unbuffered operation. Then simply add the buffer (which is ok because Handles are mutable) if the user ever calls hLookAhead. moreover, i have an idea how to implement async i/o without complex burecreacy: use mmapped files, may be together with miltiple buffers. for example, we can allocate four 16kb buffers. when one buffer is filled with written data, the program unmaps it and switches to use the next buffer. i don't tested it, but OS can guess that unmapped buffer now should be asynchronously written to disk. the same for reading - when we completely read one buffer, we can unmap it, switch to the second buffer and map the third so that the OS can asynchronously fill the third buffer while we are reading second. should this work, at least on the main desktop OSes? Please no. There are multiple reasons to avoid mmapped files. 1) They make very few performance guarantees for reading (i.e. a Haskell thread touches memory which has not yet been read from the file causing IO and all the other Haskell threads are blocked too) 2) The time of writes is unpredictable making implementing a hFlush harder? (not sure about this) 3) Not all file descriptors will support it - i.e. we will need the read/write path in any case. 4) Mmap cannot be used for random access for arbitrary files since they may be larger than the address space. This means some kind of window needs to be implemented - and this is easily done with read/write. - Einar Karttunen ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
ghci from darcs HEAD buggy on MacOS X?
I just installed the Darwin Ports ghc-devel package, which grabs the latest from the darcs repository. It seems to build fine, but when I run ghci it seems to run and work (as far as I've tested) just fine except that it beeps continuously. Any ideas? ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re[2]: new i/o library
Hello Einar, Friday, January 27, 2006, 4:19:55 PM, you wrote: EK One simple optimization is that you can omit all buffering with EK unbuffered operation. Then simply add the buffer (which is ok EK because Handles are mutable) if the user ever calls hLookAhead. yes, i do it moreover, i have an idea how to implement async i/o without complex burecreacy: use mmapped files, may be together with miltiple buffers. for example, we can allocate four 16kb buffers. when one buffer is filled with written data, the program unmaps it and switches to use the next buffer. i don't tested it, but OS can guess that unmapped buffer now should be asynchronously written to disk. the same for reading - when we completely read one buffer, we can unmap it, switch to the second buffer and map the third so that the OS can asynchronously fill the third buffer while we are reading second. should this work, at least on the main desktop OSes? EK Please no. There are multiple reasons to avoid mmapped files. EK 1) They make very few performance guarantees for reading EK(i.e. a Haskell thread touches memory which has not yet EK been read from the file causing IO and all the other EK Haskell threads are blocked too) yes, it seems that using mmapped file may slowdown such program EK 2) The time of writes is unpredictable making implementing a EKhFlush harder? (not sure about this) i can say only about Windows - here FlushViewOfFile() do it EK 3) Not all file descriptors will support it - i.e. we will EKneed the read/write path in any case. i don't understand what you mean, can you please explain futher? EK 4) Mmap cannot be used for random access for arbitrary files EKsince they may be larger than the address space. This means EKsome kind of window needs to be implemented - and this is EKeasily done with read/write. that's not true, at least for Windows - see MapViewOfFile() -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re[2]: new i/o library
Hello Duncan, Friday, January 27, 2006, 4:00:28 PM, you wrote: moreover, i have an idea how to implement async i/o without complex burecreacy: use mmapped files, may be together with miltiple buffers. for example, we can allocate four 16kb buffers. when one buffer is filled with written data, the program unmaps it and switches to use the next buffer. i don't tested it, but OS can guess that unmapped buffer now should be asynchronously written to disk. the same for reading - when we completely read one buffer, we can unmap it, switch to the second buffer and map the third so that the OS can asynchronously fill the third buffer while we are reading second. should this work, at least on the main desktop OSes? DC On Linux an probably other unix-like OSes I don't think this would be DC any different from using read/write. DC On Linux, read and mmap use the same underlying mechanism - the page DC cache. The only difference is that with mmap you get zero-copy access to DC the page cache. However frequent mapping and unmapping may eliminate DC that advantage. Either way there is no difference in how asynchronous DC the operations are. yes, i want to save exactly this bit of performance - after i optimized all other expenses on the path of text i/o in other words, i interested in having zero-wait operation both for reading and writing, i.e. that in sequence of getChar or putChar actions there were no waits on any action - of course, if the disk is fast enough. in other words, speed of such i/o programs should be the same as if we just write these data to memory current GHC's Handle implementation uses rather complex machinery for async reading and writing, and i can even say that most part of Handle's implementation complexity is due to this async machinery. so i wanna know what exactly accomplished by this implementation and can we implement async operation much easier by using mmap? the word async is overloaded here - i'm most interested in having zero-overhead in single-threaded operation, while GHC's optimization, afair, is more about overlapping I/O in one thread with computations in another. so i'm searching for fastest and easiest-to-implement scheme. what you propose? -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: new i/o library
Bulat Ziganshin wrote: i'm now write some sort of new i/o library. one area where i currently lacks in comparision to the existing Handles implementation in GHC, is the asynchronous i/o operations. can you please briefly describe how this is done in GHC and partially - why the multiple buffers are used? Multiple buffers were introduced to cope with the semantics we wanted for hPutStr. The problem is that you don't want hPutStr to hold a lock on the Handle while it evaluates its argument list, because that could take arbitrary time. Furthermore, things like this: putStr (trace foo bar) used to cause deadlocks, because putStr holds the lock, evaluates its argument list, which causes trace to also attempt to acquire the lock on stdout, leading to deadlock. So, putStr first grabs a buffer from the Handle, then unlocks the Handle while it fills up the buffer, then it takes the lock again to write the buffer. Since another thread might try to putStr while the lock is released, we need multiple buffers. For async IO on Unix, we use non-blocking read() calls, and if read() indicates that we need to block, we send a request to the IO Manager thread (see GHC.Conc) which calls select() on behalf of all the threads waiting for I/O. For async IO on Windows, we either use the threaded RTS's blocking foreign call mechanism to invoke read(), or the non-threaded RTS has a similar mechanism internally. We ought to be using the various alternatives to select(), but we haven't got around to that yet. moreover, i have an idea how to implement async i/o without complex burecreacy: use mmapped files, may be together with miltiple buffers. I don't think we should restrict the implementation to mmap'd files, for all the reasons that Einar gave. Lots of things aren't mmapable, mainly. My vision for an I/O library is this: - a single class supporting binary input (resp. output) that is implemented by various transports: files, sockets, mmap'd files, memory and arrays. Windowed mmap is an option here too. - layers of binary filters on top of this: you could add buffering, and compression/decompression. - a layer of text translation at the top. This is more or less how the Stream-based I/O library that I was working on is structured. The binary I/O library would talk to a binary transport, perhaps with a layer of buffering, whereas text-based applications talk to the text layer. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Linking with C++ produced by Visual Studio .NET on Windows XP?
I added this to the FAQ list; please feel free to elaborate and correct. Linking to Visual Studio-generated code would be much easier if GHC were able to use Visual C++ as backend, instead of gcc (even Visual Haskell at the moment relies on gcc for C compilation). I have no idea, though, how much work it would be to make GHC able to compile via Visual C++. Cheers, Cyril ___ Simon Peyton-Jones wrote: GHC now makes it easy for all users to contribute new documentation about GHC to help other users, by adding to the GHC documentation wiki. See the Collaborative documentation heading on http://haskell.org/haskellwiki/GHC:Documentation Linking to C++ would be an ideal topic. It's a regular question, and not one of our strengths at GHC HQ! ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
