Re: [Haskell-cafe] [Conduit] weird action of leftover.
It's a bug in your implementation of takeLine I believe. It doesn't take into account that lines can span multiple chunks. When you call takeLine the first time, you get L1\n. leftover puts a chunk with exactly those contents back. When you call takeLine the second time, it gets the chunk L1\n, and your splitAt gives you back L1\n and . The is then leftover, and the next call to takeLine gets it. Your takeLine needs to include logic saying there's no newline in this chunk at all, let's get the next chunk and try that. You can look at the source to lines[1] for an example of the concept. Michael [1] http://haddocks.fpcomplete.com/fp/7.4.2/20130313-1/conduit/src/Data-Conduit-Binary.html#lines On Mon, Apr 8, 2013 at 8:44 AM, Magicloud Magiclouds magicloud.magiclo...@gmail.com wrote: Say I have code like below. If I comment the leftover in main, I got (Just L1\n, Just L2\n, Just L3\n, Just L4\n). But if I did not comment the leftover, then I got (Just L1\n, Just L1\n, Just , Just L2\n). Why is not it (Just L1\n, Just L1\n, Just L2\n, Just L3\n)? takeLine :: (Monad m) = Consumer ByteString m (Maybe ByteString) takeLine = do mBS - await case mBS of Nothing - return Nothing Just bs - case DBS.elemIndex _lf bs of Nothing - return $ Just bs Just i - do let (l, ls) = DBS.splitAt (i + 1) bs leftover ls return $ Just l main = do m - runResourceT $ sourceFile test.simple $$ (do a - takeLine leftover $ fromJust a b - takeLine c - takeLine d - takeLine return (a, b, c, d)) print m -- 竹密岂妨流水过 山高哪阻野云飞 And for G+, please use magiclouds#gmail.com. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [Conduit] weird action of leftover.
Thank you for the reply. I've learnt the code of lines. So it is because how ByteString works, that the conduit is not a stream of bytes, but chunks, right? On Tue, Apr 9, 2013 at 12:12 PM, Michael Snoyman mich...@snoyman.comwrote: It's a bug in your implementation of takeLine I believe. It doesn't take into account that lines can span multiple chunks. When you call takeLine the first time, you get L1\n. leftover puts a chunk with exactly those contents back. When you call takeLine the second time, it gets the chunk L1\n, and your splitAt gives you back L1\n and . The is then leftover, and the next call to takeLine gets it. Your takeLine needs to include logic saying there's no newline in this chunk at all, let's get the next chunk and try that. You can look at the source to lines[1] for an example of the concept. Michael [1] http://haddocks.fpcomplete.com/fp/7.4.2/20130313-1/conduit/src/Data-Conduit-Binary.html#lines On Mon, Apr 8, 2013 at 8:44 AM, Magicloud Magiclouds magicloud.magiclo...@gmail.com wrote: Say I have code like below. If I comment the leftover in main, I got (Just L1\n, Just L2\n, Just L3\n, Just L4\n). But if I did not comment the leftover, then I got (Just L1\n, Just L1\n, Just , Just L2\n). Why is not it (Just L1\n, Just L1\n, Just L2\n, Just L3\n)? takeLine :: (Monad m) = Consumer ByteString m (Maybe ByteString) takeLine = do mBS - await case mBS of Nothing - return Nothing Just bs - case DBS.elemIndex _lf bs of Nothing - return $ Just bs Just i - do let (l, ls) = DBS.splitAt (i + 1) bs leftover ls return $ Just l main = do m - runResourceT $ sourceFile test.simple $$ (do a - takeLine leftover $ fromJust a b - takeLine c - takeLine d - takeLine return (a, b, c, d)) print m -- 竹密岂妨流水过 山高哪阻野云飞 And for G+, please use magiclouds#gmail.com. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe -- 竹密岂妨流水过 山高哪阻野云飞 And for G+, please use magiclouds#gmail.com. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [Conduit] weird action of leftover.
Yes, that's a fair explanation. On Tue, Apr 9, 2013 at 7:48 AM, Magicloud Magiclouds magicloud.magiclo...@gmail.com wrote: Thank you for the reply. I've learnt the code of lines. So it is because how ByteString works, that the conduit is not a stream of bytes, but chunks, right? On Tue, Apr 9, 2013 at 12:12 PM, Michael Snoyman mich...@snoyman.comwrote: It's a bug in your implementation of takeLine I believe. It doesn't take into account that lines can span multiple chunks. When you call takeLine the first time, you get L1\n. leftover puts a chunk with exactly those contents back. When you call takeLine the second time, it gets the chunk L1\n, and your splitAt gives you back L1\n and . The is then leftover, and the next call to takeLine gets it. Your takeLine needs to include logic saying there's no newline in this chunk at all, let's get the next chunk and try that. You can look at the source to lines[1] for an example of the concept. Michael [1] http://haddocks.fpcomplete.com/fp/7.4.2/20130313-1/conduit/src/Data-Conduit-Binary.html#lines On Mon, Apr 8, 2013 at 8:44 AM, Magicloud Magiclouds magicloud.magiclo...@gmail.com wrote: Say I have code like below. If I comment the leftover in main, I got (Just L1\n, Just L2\n, Just L3\n, Just L4\n). But if I did not comment the leftover, then I got (Just L1\n, Just L1\n, Just , Just L2\n). Why is not it (Just L1\n, Just L1\n, Just L2\n, Just L3\n)? takeLine :: (Monad m) = Consumer ByteString m (Maybe ByteString) takeLine = do mBS - await case mBS of Nothing - return Nothing Just bs - case DBS.elemIndex _lf bs of Nothing - return $ Just bs Just i - do let (l, ls) = DBS.splitAt (i + 1) bs leftover ls return $ Just l main = do m - runResourceT $ sourceFile test.simple $$ (do a - takeLine leftover $ fromJust a b - takeLine c - takeLine d - takeLine return (a, b, c, d)) print m -- 竹密岂妨流水过 山高哪阻野云飞 And for G+, please use magiclouds#gmail.com. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe -- 竹密岂妨流水过 山高哪阻野云飞 And for G+, please use magiclouds#gmail.com. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] [Conduit] weird action of leftover.
Say I have code like below. If I comment the leftover in main, I got (Just L1\n, Just L2\n, Just L3\n, Just L4\n). But if I did not comment the leftover, then I got (Just L1\n, Just L1\n, Just , Just L2\n). Why is not it (Just L1\n, Just L1\n, Just L2\n, Just L3\n)? takeLine :: (Monad m) = Consumer ByteString m (Maybe ByteString) takeLine = do mBS - await case mBS of Nothing - return Nothing Just bs - case DBS.elemIndex _lf bs of Nothing - return $ Just bs Just i - do let (l, ls) = DBS.splitAt (i + 1) bs leftover ls return $ Just l main = do m - runResourceT $ sourceFile test.simple $$ (do a - takeLine leftover $ fromJust a b - takeLine c - takeLine d - takeLine return (a, b, c, d)) print m -- 竹密岂妨流水过 山高哪阻野云飞 And for G+, please use magiclouds#gmail.com. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
