Re: [Haskell-cafe] carry "state" around ....
On Sat, 2008-07-19 at 23:55 -0500, Galchin, Vasili wrote:
> yes Duncan I am trying to pass-by-value. I am familiar with
> ForeignPtr; however, I don't comprehend what you and Brandon are
> suggesting to do. Could either of you provide a code illustration or
> point at existing code to illustrate your approach?
Take a look at John Meacham's RSA example.
So at the moment you're using using Storable and a Haskell record, say:
data AIOCB = AIOCB {
...
}
and we're suggesting instead:
newtype AIOCB = AIOCB (ForeignPtr AIOCB)
then to access a member use hsc2hs:
getBlah :: AIOCB -> IO Blah
getBlah (AIOCB fptr) =
withForeignPtr fptr $ \ptr -> {# peek aiocb,blah #} ptr
So you only access the parts you need and keep the aiocb C struct
allocated on the heap (use mallocForeignPtr).
Duncan
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Help with optimization
Hi! Profiling says that my program spends 18.4 % of time (that is around three seconds) and 18.3 % of allocations in this function which is saving the rendered image to a PPM file: saveImageList :: String -> Int -> Int -> [ViewportDotColor] -> IO () saveImageList filename width height image = do B.writeFile filename file where file = B.append header bytes header = C.pack $ "P6\n" ++ show width ++ " " ++ show height ++ "\n255\n" bytes = B.pack $ concatMap (color . dealpha . (\(ViewportDotColor _ c) -> c)) image where color (VoxelColor red green blue _) = [floor $ red * 255, floor $ green * 255, floor $ blue * 255] dealpha c = addColor c (VoxelColor 1.0 1.0 1.0 1.0) -- white background For a 921615 bytes large file to save this is too much in my opinion. And I think that it consumes to much allocations. Probably it should not store intermediate lists? Any suggestions? Best regards Mitar ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] ansi2html - one program, several issues
On Sun, Jul 20, 2008 at 7:25 AM, Chaddaï Fouché <[EMAIL PROTECTED]> wrote: > > > That's exactly what I thought. But even if I remove the only 'try' I use > the > > memory consumption remains unchanged: > > It's true, but in your case your output is almost the raw input data, > which means that even without a noxious "try", you still have the > whole file in memory. Well hopefully not with your latest code, which > I would really like to see. Here is the part that actually changed: --- split c str = let (p,ps) = aux str in (p:ps) where aux [] = ([],[]) aux (x:cs) = let (xs,xss) = aux cs in if x == c then ([c],(xs:xss)) else ((x:xs),xss) splitPred :: (Eq a) => (a -> Bool) -> [a] -> [[a]] splitPred pr str = let (p,ps) = aux str in (p:ps) where aux [] = ([],[]) aux (x:cs) = let (xs,xss) = aux cs in if pr x then ([],((x:xs):xss)) else ((x:xs),xss) doOneFile :: String -> IO () doOneFile fname = do t1 <- getCurrentTime doesFileExist (fname ++ ".html") >>= \b -> if b then hPutStrLn stderr $ printf "File already processed, skipping: %s" fname else do src <- readFile fname out <- openFile (fname ++ ".html") WriteMode hSetBuffering out (BlockBuffering (Just 64000)) hPutStrLn out "" hPutStrLn out "" hPutStrLn out "" hPutStrLn out "" let extractData = \p -> case p of Right x -> x Left err -> (trace . show $ err) [] let srcSplit = splitPred (`elem`"\n") src let parsed = concatMap (extractData . parse mainParser fname) srcSplit execStateT (hPrintHtml (St id)) (out,emptyStyle) -- wypisujemy pierwszy wiersz execStateT (mapM_ hPrintHtml parsed) (out,emptyStyle) hPutStrLn out "" hPutStrLn out "" hPutStrLn out "" t2 <- getCurrentTime hPutStrLn stderr $ printf "File %s processed. It took %s. File size was %d characters." fname (show $ diffUTCTime t2 t1) (length src) hClose out -- The whole file is also attached. You will find there another (worse) implementation of split and a little bit of code similar to thread pool stuff. On Sun, Jul 20, 2008 at 8:17 AM, John Meacham <[EMAIL PROTECTED]> wrote: > On Sun, Jul 20, 2008 at 02:34:09AM +0400, Bulat Ziganshin wrote: > > i think that Parsec library should hold entire file in memory only when > > you use 'try' for whole file. otherwise it should omit data as > > proceeded > > I do not believe that is the case, since the return type of runParser > "Either ParseError a" means that before you can extract the result of > the parse from the 'Right' branch, it must evaluate whether the result > is 'Left' or 'Right' meaning it needs to parse the whole input in order > to determine whether the parse was succesful. > It's true it has to parse the whole file, but it is not true it has to reside in the memory: only the results must be there. In this case, when the result is 1-1 transformation of input, it is true. But consider this program: module Main where import Text.ParserCombinators.Parsec par = eof <|> (char 'a' >> par) alst = take 2 (repeat 'a') main = print (runParser par () "" alst) It runs in constant memory: $ ./partest.exe +RTS -sstderr C:\cygwin\home\Metharius\killer\killerPy\ansi2html\partest.exe +RTS -sstderr Right () 84,326,845,636 bytes allocated in the heap 22,428,536 bytes copied during GC 9,684 bytes maximum residency (1 sample(s)) 13,848 bytes maximum slop 1 MB total memory in use (0 MB lost due to fragmentation) Generation 0: 160845 collections, 0 parallel, 0.63s, 0.63s elapsed Generation 1: 1 collections, 0 parallel, 0.00s, 0.00s elapsed INIT time0.02s ( 0.00s elapsed) MUT time 54.31s ( 54.55s elapsed) GCtime0.63s ( 0.63s elapsed) EXIT time0.00s ( 0.00s elapsed) Total time 54.95s ( 55.17s elapsed) %GC time 1.1% (1.1% elapsed) Alloc rate1,552,176,623 bytes per MUT second Productivity 98.8% of total user, 98.4% of total elapsed Best regards Christopher Skrzętnicki ansi2html.hs Description: Binary data ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] ansi2html - one program, several issues
Dan Doel wrote: On Sunday 20 July 2008, John Meacham wrote: I do not believe that is the case, since the return type of runParser "Either ParseError a" means that before you can extract the result of the parse from the 'Right' branch, it must evaluate whether the result is 'Left' or 'Right' meaning it needs to parse the whole input in order to determine whether the parse was succesful. ... It doesn't stop it from parsing the entire file strictly. However, what it does do is prevent the parser from backtracking out of arbitrary amounts of lookahead. So, unless you use try (which allows for lookahead), when any token is consumed by the parser, it can be garbage collected (assuming the parser is the only thing pointing to the token stream). So, it consumes its input strictly, but with limited overhead (ideally using try only for some small bounded lookahead where it's needed). So with Parsec, you can keep the *input* from filling up memory, but if you do, the *result* will still take up space (e.g. Right (value)). For a simple transformation where the output is a similar string to the input, it will be just as large, so not much space is actually saved (maybe a factor of 2 -- just keeping the output, not also the input), it seems. -Isaac ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Help with optimization
Maybe your image isn't strict enough and the computations are forced when the image gets written to disc? Am 20.07.2008 um 14:13 schrieb Mitar: Hi! Profiling says that my program spends 18.4 % of time (that is around three seconds) and 18.3 % of allocations in this function which is saving the rendered image to a PPM file: saveImageList :: String -> Int -> Int -> [ViewportDotColor] -> IO () saveImageList filename width height image = do B.writeFile filename file where file = B.append header bytes header = C.pack $ "P6\n" ++ show width ++ " " ++ show height ++ "\n255\n" bytes = B.pack $ concatMap (color . dealpha . (\(ViewportDotColor _ c) -> c)) image where color (VoxelColor red green blue _) = [floor $ red * 255, floor $ green * 255, floor $ blue * 255] dealpha c = addColor c (VoxelColor 1.0 1.0 1.0 1.0) -- white background For a 921615 bytes large file to save this is too much in my opinion. And I think that it consumes to much allocations. Probably it should not store intermediate lists? Any suggestions? Best regards Mitar ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe PGP.sig Description: Signierter Teil der Nachricht ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: Re[2]: [Haskell-cafe] ansi2html - one program, several issues
I played with another approach without any parser library, just with plain
pattern matching. The idea was to create function to match all different
cases of codes. Since I already got most of the code, it was quite easy to
do. The core function consist of cases like those:
> parse ('\ESC':'[':'1':';':'4':'0':'m':rest) = modifyAndPrint (\x -> x
{ bgcol = light black })>> parse rest
> parse ('\ESC':'[':'1':';':'4':'1':'m':rest) = modifyAndPrint (\x -> x
{ bgcol = light red })>> parse rest
> parse ('\ESC':'[':'1':';':'4':'2':'m':rest) = modifyAndPrint (\x -> x
{ bgcol = light green })>> parse rest
> parse ('\ESC':'[':'1':';':'4':'3':'m':rest) = modifyAndPrint (\x -> x
{ bgcol = light yellow })>> parse rest
If you have read the old code you should recognize some parts of it here.
It should consume rather constant amount of memory. To my surprise it
consumed almost exactly the same amount of memory as the previous program.
Turns out the problematic line was this:
>hPutStrLn stderr $ printf "File %s processed. It took %s. File size was
%d characters." fname (show $ diffUTCTime t2 t1) *(length src)*
It computed length of the input file. Needless to say, because "src" was
actually the input file parsed previously, it was all hanging in the memory.
Having removed that reference to src both programs (the one that parses
input per line and the most recent one) are running in constant memory
(2Mb). This doesn't apply to the first program, which has to read whole file
before producing any output.
And the last note: the new program is also 2x faster, perhaps due to very
simple structure that is easy to optimize. It also makes sense now to use
mapMPar as it reduces run time by 30%. The full code is in attachments.
Best regards
Christopher Skrzętnicki
ansi2html.hs
Description: Binary data
ansi2html.hs
Description: Binary data
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Help with optimization
Hi! 2008/7/20 Adrian Neumann <[EMAIL PROTECTED]>: > Maybe your image isn't strict enough and the computations are forced when > the image gets written to disc? But it still takes three seconds more than displaying to the screen. This is why I am also thinking that this code is to slow. This and that when saving to a file it takes almost 25 % more allocations. So probably it is slower because of all this allocations? Mitar ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Data.Complex.magnitude slow?
On Fri, 18 Jul 2008, stefan kersten wrote: On 17.07.2008, at 21:46, Lennart Augustsson wrote: If scaleFloat and exponent are implemented with bit twiddling they can be quite fast. is there a way in ghc to 'cast' between float/int32 and double/ int64 (without going through memory)? I read this as "Is there any way to take a 32-bit float in a register and end up with a 32-bit int in a register, without going through memory, in GHC? How about the other way around? How about 64-bit floats and integers?" It is rather hard to do portably in GHC what some hardware does not do. A long time ago hardware architects decided that separating the integer and floating point register files was a Good Thing. The machine I know best is SPARC, where there are no instructions to move data directly between integer registers and floating point registers. On the other hand, there are other pressures, notably graphics. While integer and floating point registers may be kept separate, there are often "vector" instructions that let you operate on integers in the floating- point registers. (Such as the VIS instructions on SPARCs, which are technically optional.) Exploiting such instructions tends to be rather non- portable. All things considered, I wouldn't worry about it too much: the memory in question should be at the top of the stack, so should be in L1 cache, so should be pretty fast to access, and other things may be more significant. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Help with optimization
mmitar: > Hi! > > Profiling says that my program spends 18.4 % of time (that is around > three seconds) and 18.3 % of allocations in this function which is > saving the rendered image to a PPM file: > > saveImageList :: String -> Int -> Int -> [ViewportDotColor] -> IO () > saveImageList filename width height image = do > B.writeFile filename file > where file = B.append header bytes > header = C.pack $ "P6\n" ++ show width ++ " " ++ show height > ++ "\n255\n" > bytes = B.pack $ concatMap (color . dealpha . > (\(ViewportDotColor _ c) -> c)) image > where color (VoxelColor red green blue _) = [floor $ red * > 255, floor $ green * 255, floor $ blue * 255] > dealpha c = addColor c (VoxelColor 1.0 1.0 1.0 1.0) > -- white background > > For a 921615 bytes large file to save this is too much in my opinion. > And I think that it consumes to much allocations. Probably it should > not store intermediate lists? Rather than generating the bytes as a list and packing them, could you instead fill a bytestring directly? -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] carry "state" around ....
I think it may be time for a little clarity about aoicb's. From the Single Unix Specification: "The header shall define the aiocb structure which shall include AT LEAST the following members: int aio_fildes File descriptor. off_t aio_offset File offset. volatile void *aio_bufLocation of buffer. size_t aio_nbytes Length of transfer. int aio_reqprioRequest priority offset.struct sigevent aio_sigevent Signal number and value. int aio_lio_opcode Operation to be performed. " The "AT LEAST" here means that - a portable program may rely on these members being present - a portable program MUST assume that an unknown number of additional members are also present - a portable program may freely copy such a record, but may only pass it to a library function if that function is expecting to initialise it For asynchronous I/O, this means that - you can allocate an aiocb object - an aiocb passed to aio_suspend, aio_error, aio_return, or aio_cancel should have been filled in by aio_read or aio_write and should be EXACTLY THE SAME object, not a copy of it. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] GHC Error: "FATAL:Symbol _XJv_srt already defined."
Anyone know what this is about? Removing parts makes problem go away, but it's not clear what parts to remove! (that, and I wrote those parts for a reason!). bash-3.2$ ghc --version The Glorious Glasgow Haskell Compilation System, version 6.8.3 bash-3.2$ uname -a Darwin silverback-wired.home 9.4.0 Darwin Kernel Version 9.4.0: Mon Jun 9 19:30:53 PDT 2008; root:xnu-1228.5.20~1/RELEASE_I386 i386 $ rm -f DerivingError.o DerivingError.hi; ghc --make DerivingError.hs [1 of 1] Compiling DerivingError( DerivingError.hs, DerivingError.o ) /var/folders/C0/C0SledGV2RaxbU+8ZLDnVU+++TI/-Tmp-//ghc27223_0/ghc27223_0.s:6080:0: FATAL:Symbol _XxG_srt already defined. -Ron Alford DerivingError.hs Description: Binary data ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] carry "state" around ....
On 2008 Jul 20, at 21:05, Richard A. O'Keefe wrote: For asynchronous I/O, this means that - you can allocate an aiocb object - an aiocb passed to aio_suspend, aio_error, aio_return, or aio_cancel should have been filled in by aio_read or aio_write and should be EXACTLY THE SAME object, not a copy of it. Right, hence a ForeignPtr which is mostly opaque. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] [EMAIL PROTECTED] system administrator [openafs,heimdal,too many hats] [EMAIL PROTECTED] electrical and computer engineering, carnegie mellon universityKF8NH ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Data.Complex.magnitude slow?
On Jul 20, 2008, at 8:11 PM, Richard A. O'Keefe wrote: I read this as "Is there any way to take a 32-bit float in a register and end up with a 32-bit int in a register, without going through memory, in GHC? How about the other way around? How about 64-bit floats and integers?" It is rather hard to do portably in GHC what some hardware does not do. The reason to provide this as a primitive is so that code like Complex magnitude doesn't have to go through a completely opaque-to-the- compiler interface in order to extract bits from the underlying IEEE float representation. Sure, code generation can't assign it to a register, but it could be kept on the stack. Once we're peeking and poking through a Ptr (the trick that was suggested the last time this came up, too, if I remember rightly) we're sunk---GHC doesn't reason well at all about this stuff, and we need to allocate a Ptr to peek and poke. If instead we use a foreign call, again, it's completely compiler-opaque. A long time ago hardware architects decided that separating the integer and floating point register files was a Good Thing. The machine I know best is SPARC, where there are no instructions to move data directly between integer registers and floating point registers. Ironically, this is no longer true on Niagara-class (T1 and T2) SPARC machines. So nowadays this may be the only architecture that *does* provide this ability. But again, the really important thing is that the compiler can see that bits is bits and that the conversion involves no magic and no extra storage beyond a spill location on the stack. ... [stuff about SIMD snipped]... All things considered, I wouldn't worry about it too much: the memory in question should be at the top of the stack, so should be in L1 cache, so should be pretty fast to access, and other things may be more significant. But only if GHC can treat them transparently---which at the moment it evidently can't! All it'd take is a primitive from Float# -> Int32# and another from Double# -> Int64#... -Jan-Willem Maessen ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] carry "state" around ....
Thank you Duncan for your example. I will also read the RSA code.
Regards, Vasili
On Sun, Jul 20, 2008 at 6:51 AM, Duncan Coutts <[EMAIL PROTECTED]>
wrote:
>
> On Sat, 2008-07-19 at 23:55 -0500, Galchin, Vasili wrote:
> > yes Duncan I am trying to pass-by-value. I am familiar with
> > ForeignPtr; however, I don't comprehend what you and Brandon are
> > suggesting to do. Could either of you provide a code illustration or
> > point at existing code to illustrate your approach?
>
> Take a look at John Meacham's RSA example.
>
> So at the moment you're using using Storable and a Haskell record, say:
>
> data AIOCB = AIOCB {
>...
> }
>
> and we're suggesting instead:
>
> newtype AIOCB = AIOCB (ForeignPtr AIOCB)
>
> then to access a member use hsc2hs:
>
> getBlah :: AIOCB -> IO Blah
> getBlah (AIOCB fptr) =
> withForeignPtr fptr $ \ptr -> {# peek aiocb,blah #} ptr
>
> So you only access the parts you need and keep the aiocb C struct
> allocated on the heap (use mallocForeignPtr).
>
> Duncan
>
>
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Haskell code contributions
Hello, I have a password to upload Haskell code contributions but I can't find the email with the password and I have forgotten who is the "gatekeeper". ?? Regards, Vasili ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
