Re: Incrementally consuming the eventlog
On Thu, Apr 28, 2011 at 11:53 PM, Donnie Jones don...@darthik.com wrote: Anyway, from your description, I don't understand how a listener would consume the eventlog incrementally? I simply meant that I want to be able to register listeners for events instead of having to parse the eventlog file after the fact. I do think it would be useful to register listeners for events. I do not think the invocation of a callback would be too much overhead, rather the action the callback performs could be a very significant overhead, such as sending eventlog data over a network connection. But, if you are willing to accept the performance loss from the callback's action to gain the event data then it seems worthwhile to me. A typical use of the callback would be to update some internal data structure of the program itself, thereby making the program self-monitoring. I've been toying with introducing log levels to the eventlog command line API so the consumer of the event log can specify the number of events it would like to receive. We could do something similar for the API e.g. registerEventListener (schedEvents .|. ioManagerEvents) (\ e - ...) Johan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Incrementally consuming the eventlog
On Fri, Apr 29, 2011 at 12:00 AM, Don Stewart don...@gmail.com wrote: I'm very interested in what the best way to get incremental event data from a running GHC process would be. Looking at the code, we flush the event buffer fairly regularly, but the event parser is currently strict. So we'd need a lazy (or incremental) parser, that'll return a list of successful event parses, then block. I suspect this mode would be supported. *My evil plan is to write a little monitoring web app that just attaches to the event stream and renders it in a useful heartbeat format* , but I need incremental parsing. A less general solution might be to have the program itself start a little web server on some port and use the API I proposed to serve JSON data with the aggregate statistics you care about. Example: main = do eventData - newIORef server - serveOn 8080 $ \ _req - readIORef eventData = sendResponse eventData registerEventListener $ \ ev - updateEventData eventData ev runNormalProgram You can wrap the creation of the webserver in a little helper function an make any program monitorable simply by doing main = withMonitoring runApp withMonitoring would take care of starting/stopping the webserver and processing events. Just a thought. Johan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Incrementally consuming the eventlog
I've put a library for incremental parsing of the event log here: http://code.haskell.org/~dons/code/ghc-events-stream/ The goal is to implement something like: http://www.erlang.org/doc/man/heart.html On Sun, May 1, 2011 at 1:44 AM, Johan Tibell johan.tib...@gmail.com wrote: On Fri, Apr 29, 2011 at 12:00 AM, Don Stewart don...@gmail.com wrote: I'm very interested in what the best way to get incremental event data from a running GHC process would be. Looking at the code, we flush the event buffer fairly regularly, but the event parser is currently strict. So we'd need a lazy (or incremental) parser, that'll return a list of successful event parses, then block. I suspect this mode would be supported. *My evil plan is to write a little monitoring web app that just attaches to the event stream and renders it in a useful heartbeat format* , but I need incremental parsing. A less general solution might be to have the program itself start a little web server on some port and use the API I proposed to serve JSON data with the aggregate statistics you care about. Example: main = do eventData - newIORef server - serveOn 8080 $ \ _req - readIORef eventData = sendResponse eventData registerEventListener $ \ ev - updateEventData eventData ev runNormalProgram You can wrap the creation of the webserver in a little helper function an make any program monitorable simply by doing main = withMonitoring runApp withMonitoring would take care of starting/stopping the webserver and processing events. Just a thought. Johan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Incrementally consuming the eventlog
On Thu, 2011-04-28 at 23:31 +0200, Johan Tibell wrote: The RTS would invoke listeners every time a new event is written. This design has many benefits: - We don't need to introduce the serialization, deserialization, and I/O overhead of first writing the eventlog to file and then parsing it again. The events are basically generated in serialised form (via C code that writes them directly into the event buffer). They never exist as Haskell data structures, or even C structures. - Programs could monitor themselves and provide debug output (e.g. via some UI component). - Users could write code that redirects the output elsewhere e.g. to a socket for remote monitoring. Would invoking a callback on each event add too big of an overhead? Yes, by orders of magnitude. In fact it's impossible because the act of invoking the callback would generate more events... :-) How about invoking the callback once every time the event buffer is full? That's much more realistic. Still, do we need the generality of pushing the event buffers through the Haskell code? For some reason it makes me slightly nervous. How about just setting which output FD the event buffers get written to. Turning all events or various classes of events on/off at runtime should be doable. The design already supports multiple classes, though currently it just has one class (the 'scheduler' class). The current design does not support fine grained filtering at the point of event generation. Those two features combined (plus control over the frequency of event buffer flushing) would be enough to implement a monitoring socket interface (web http or local unix domain socket). Making the parser in the ghc-events package incremental would be sensible and quite doable as people have already demonstrated. Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Package management
On Tue, 2011-04-26 at 14:05 -0700, Brandon Moore wrote: Based on my own misadventures and Albert Y. C. Lai's SICP (http://www.vex.net/~trebla/haskell/sicp.xhtml) it seems the that root of all install problems is that reinstalling a particular version of a particular package deletes any other existing builds of that version, even if other packages already depend on them. Deleting perfectly good versions seems to be the root of all package management problems. Yes. There are already hashes to keep incompatible builds of a package separate. Would anything break if existing packages were left alone when a new version was installed? (perhaps preferring the most recent if a package flag specifies version but not hash). That is the nix solution. It is also my favoured long term solution. The obvious difficulty is a little more trouble to manually specify packages. Are there any other problems with this idea? See nix and how it handles the configuration and policy issues thrown up by allowing multiple instances of the same version of each package. For example, they introduce the notion of a package environment which is a subset of the universe of installed packages. Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Incrementally consuming the eventlog
I've got a proof of concept event-log monitoring server and incremental parser for event streams: * http://code.haskell.org/~dons/code/ghc-events-stream/ * http://code.haskell.org/~dons/code/ghc-monitor/ Little screen shot of the snap server running, watching a Haskell process' eventlog fifo: * http://i.imgur.com/Xfr9I.png The main issue at the moment is that GHC is irregular in scheduling flusing of the event log stream, so it might be hours or days before you see any activity. This isn't useful for heartbeat style monitoring. Also, we need to break out a bit of ThreadScope to give access to its analytics (e.g. rendering time series). -- Don On Sun, May 1, 2011 at 1:51 PM, Duncan Coutts duncan.cou...@googlemail.com wrote: On Thu, 2011-04-28 at 23:31 +0200, Johan Tibell wrote: The RTS would invoke listeners every time a new event is written. This design has many benefits: - We don't need to introduce the serialization, deserialization, and I/O overhead of first writing the eventlog to file and then parsing it again. The events are basically generated in serialised form (via C code that writes them directly into the event buffer). They never exist as Haskell data structures, or even C structures. - Programs could monitor themselves and provide debug output (e.g. via some UI component). - Users could write code that redirects the output elsewhere e.g. to a socket for remote monitoring. Would invoking a callback on each event add too big of an overhead? Yes, by orders of magnitude. In fact it's impossible because the act of invoking the callback would generate more events... :-) How about invoking the callback once every time the event buffer is full? That's much more realistic. Still, do we need the generality of pushing the event buffers through the Haskell code? For some reason it makes me slightly nervous. How about just setting which output FD the event buffers get written to. Turning all events or various classes of events on/off at runtime should be doable. The design already supports multiple classes, though currently it just has one class (the 'scheduler' class). The current design does not support fine grained filtering at the point of event generation. Those two features combined (plus control over the frequency of event buffer flushing) would be enough to implement a monitoring socket interface (web http or local unix domain socket). Making the parser in the ghc-events package incremental would be sensible and quite doable as people have already demonstrated. Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Incrementally consuming the eventlog
On Thu, Apr 28, 2011 at 3:00 PM, Don Stewart don...@gmail.com wrote: So we'd need a lazy (or incremental) parser, that'll return a list of successful event parses, then block. I suspect this mode would be supported. A while ago, I hacked something together on top of the current eventlog parser that would consume an event at a time, and record the seek offset of each successful parse. If parsing failed (due to unflushed data), it would try again later. I think I might even claim that this is a somewhat sensible and parsimonious approach, but I'm drinking wine right now, so my judgment might be impaired. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Incrementally consuming the eventlog
I managed to build one on top of attoparsec's lazy parser that seems to work -- but I'd like ghc to flush a bit more regularly so I could test it better. -- Don On Sun, May 1, 2011 at 7:59 PM, Bryan O'Sullivan b...@serpentine.com wrote: On Thu, Apr 28, 2011 at 3:00 PM, Don Stewart don...@gmail.com wrote: So we'd need a lazy (or incremental) parser, that'll return a list of successful event parses, then block. I suspect this mode would be supported. A while ago, I hacked something together on top of the current eventlog parser that would consume an event at a time, and record the seek offset of each successful parse. If parsing failed (due to unflushed data), it would try again later. I think I might even claim that this is a somewhat sensible and parsimonious approach, but I'm drinking wine right now, so my judgment might be impaired. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
