Re: WebUI for GHC/Haskell tooling (eventlog)

2020-09-13 Thread Evan Laforge 
Hi, I read your post back when you posted it and meant to respond, but got
distracted!  Anyway, I think the profiling tools in ghc could definitely use
some attention, so I'm glad you're looking into this!

The below is going to seem like a rant, and maybe it is in some parts, but
I mean it to be a constructive attempt to chart the gaps in documentation or
tools.  It has been observed many times that the haskell performance story is
scattered about, and many people have suggested some kind of consolidation,
which of course is always The Problem, especially for open source.  So here
I am observing that again, but there does seem to be promising movement
as people get more interested in performance, and your efforts are
encouraging.

### documentation

It would be really nice to get more complete and detailed documentation of
what the options are, and gather it into one place.  This is a disorganized
list of my own experiences:

The time units in all the profiles seem mysterious.  There's a "total time"
in the .prof file.  There's a time axis on the heap profile.  There are times
in the GC summary (INIT, MUT, ..., Total).  None of these times seem to
correspond with each other.  What do they mean?  Similarly, the "total bytes"
in the prof file doesn't seem to correspond to anything in the GC summary.
Long ago (maybe around 10 years ago) I think I intuited that the heap profile
time is CPU time, which is what foiled my attempts to separate program phases
with sleeps so I could see them.  I resorted to live profiling with ekg, and
more recently I have tried to use the eventlog and custom events for that
(eventlog2html does draw the event positions, but the feature to show the
event text didn't work for me).  Anyway, there are many tools and techniques,
but I haven't seen documentation tieing them together, along with advice and
experience reports and all that good stuff.

So I improvise.  Here is my latest attempt, for ad-hoc profile exploration:
https://github.com/elaforge/karya/blob/work/tools/run_profile.py
It fiddles with all the flags I can never remember, collects and archives the
results in a dated directory, runs all the various tools I can never remember
(ghc-prof-flamegraph has been ostensibly the most useful, but see below about
SCCs), and tries to extract a summary of the somewhat more stable numbers
(GC stats and top profile cost centers) so I can diff them.

Then there is a completely different attempt to get historical performance by
running on known inputs, with the optimized non-profiling binary, extract the
actual runtimes of various phases, and put them in a database to query later:
https://github.com/elaforge/karya/tree/work/tools/timing  This is because
I don't trust profile-built binaries to be ground truth, even if it's just
-prof and the eventlog runtime, no SCCs.

I did some work to convert event logs to the chrome tracing format:
https://github.com/elaforge/karya/blob/work/App/ConvertEventLog.hs In the end,
I didn't use the graphical tracing, but just did ad-hoc analysis of the
timestamps to see who was most expensive.  The event format is another place
where documentation would be nice, as you can see from the file, I just copy
pasted the definition out of ghc and guessed what the types mean from their
names.  This was in the ghc 8.0 era I think, and I recall that the eventlog
acquired heap data after that.  I did get it working as a replacement for
ThreadScope, and I think in general reusing a general framework that other
people maintain will work better than a custom GTK app when the maintainer
count is in the 0 to 1 range, though I recall chrome consumes JSON and trying
to get that much data through JSON hit a wall eventually.  I guess JSON
should be theoretically capable of arbitrary sizes, so presumably it was that
chrome is not optimized for large data... which might undercut the idea that
it's better to use someone else's tool.

Despite all of this, over the last 10 or so years, I have never managed to get
predictable or consistent numbers, e.g. after a ghc version change they get
dramatically worse in theory, but seem to be about the same wall clock time.
Or they steadily creep down or up over long periods where no changes should
have affected them, or there is no apparent improvement after a change that
eliminated a top SCC entry, etc. etc.  And this is without the confounding
factor of changing hardware, since I do have hardware that's unchanged from 10
years back (I'm lazy about upgrading, ok?)... though of course hardware is
confounding in general, and I haven't seen any techniques for how to control
for that.  Even on the same hardware, CPUs and OSes are quite
non-deterministic, but the best approach I've seen there is criterion-style
analysis for short benchmarks, and for long ones I just run them multiple
times and hope they are below the noise floor.

Anyway, I know all this stuff goes beyond just haskell and ghc, and is part of
the general theme that profiling and

Re: WebUI for GHC/Haskell tooling (eventlog)

2020-09-08 Thread David Eichmann 

Csaba

It's really cool to see your work. Well-Typed and Hasura have recently 
started collaborating on some tooling (have a look at the announcement 
[1]). We are planning on taking the `ghc-debug` approach that you 
touched on at the end of your blog post, Csaba. While our approaches may 
be slightly different, we should keep in contact. Perhaps we'll be able 
to benefit from each other's work.


-David E

[1] 
https://hasura.io/blog/partnering-with-well-typed-and-investing-in-the-haskell-open-source-community/


On 9/8/20 11:41 AM, Simon Peyton Jones via ghc-devs wrote:


Csaba

I don’t know anything useful about tooling, WebUI, eventlog2html, 
etc.   But I do know that these things are important, so thank you so 
much for working on them!


I assume you in touch with the (new, unified) team working on the 
Haskell IDE?


Thanks again

Simon

*From:*ghc-devs  *On Behalf Of *Csaba Hruska
*Sent:* 31 August 2020 19:30
*To:* GHC developers 
*Subject:* WebUI for GHC/Haskell tooling (eventlog)

Hello,

I've written ablog post 
<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.patreon.com%2Fposts%2F41065262=02%7C01%7Csimonpj%40microsoft.com%7C4abcd97d661e4fcbad2208d84ddbf825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637344954440665978=nhyZLT7K1zfDsa3CT%2BwOSe1DTYFQYKIWViDQiyKC7yY%3D=0> 
about my WebUI based eventlog related tool.


It is also related to eventlog2html and ghc-debug.

I'm interested in your opinion and ideas related to ghc 
debug/profiling tooling.


If you have time please read the post and it would be great to hear 
some positive or negative feedback from you. It would be great to 
discuss this topic with you.


Thanks,

Csaba


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


--
David Eichmann, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com

Registered in England & Wales, OC335890
118 Wymering Mansions, Wymering Road, London W9 2NF, England

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: WebUI for GHC/Haskell tooling (eventlog)

2020-09-08 Thread Simon Peyton Jones via ghc-devs 
Csaba

I don’t know anything useful about tooling, WebUI, eventlog2html, etc.   But I 
do know that these things are important, so thank you so much for working on 
them!

I assume you in touch with the (new, unified) team working on the Haskell IDE?

Thanks again

Simon

From: ghc-devs  On Behalf Of Csaba Hruska
Sent: 31 August 2020 19:30
To: GHC developers 
Subject: WebUI for GHC/Haskell tooling (eventlog)

Hello,

I've written a blog 
post<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.patreon.com%2Fposts%2F41065262=02%7C01%7Csimonpj%40microsoft.com%7C4abcd97d661e4fcbad2208d84ddbf825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637344954440665978=nhyZLT7K1zfDsa3CT%2BwOSe1DTYFQYKIWViDQiyKC7yY%3D=0>
 about my WebUI based eventlog related tool.
It is also related to eventlog2html and ghc-debug.
I'm interested in your opinion and ideas related to ghc debug/profiling tooling.
If you have time please read the post and it would be great to hear some 
positive or negative feedback from you. It would be great to discuss this topic 
with you.

Thanks,
Csaba
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

WebUI for GHC/Haskell tooling (eventlog)

2020-08-31 Thread Csaba Hruska 
Hello,

I've written a blog post  about my
WebUI based eventlog related tool.
It is also related to eventlog2html and ghc-debug.
I'm interested in your opinion and ideas related to ghc debug/profiling
tooling.
If you have time please read the post and it would be great to hear some
positive or negative feedback from you. It would be great to discuss this
topic with you.

Thanks,
Csaba
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs