Hi Chao, sounds interesting - just a couple of things that come to mind:
I this intended as debugging aid or for operational monitoring? A Crunch job is a temporary thing, to me this doesn't sound like a good match for a web service because it disappears after a (possibly short) time. Also, when multiple jobs are executed concurrently from the same machine, you can't work with a well-known port, you'd have to pick an unused port for each job. It also looks to me like this has security implications? Right now, Crunch is just a client library and we're part of Hadoop's security framework. A web service we might have to secure in some way. Regards, Matthias On Tuesday, 2013-02-26, Chao Shi wrote: > Hi Crunch Devs, > > I'm interested in adding a web status page to crunch. I'm working on a > prototype first, which simply runs a jetty server and renders the dot file > produced by DotFileWriter at browser. The dot rendering work is done by > viz.js <https://github.com/mdaines/viz.js>. It can successfully render the > plan into SVG. > > I think there are 2 issues I hit with viz.js: > > 1. The license of viz.js is unclear. It is compiled from GraphViz source > code with emscripten. GraphViz is Eclipse Public License 1.0. > > 2. viz.js is big and slow. It is a 1.4MB compressed JS. It takes 1 or 2 > seconds on my laptop to render my pipeline (30+ MRs). I think it good to > have the graph refresh frequently and show the running status of the > pipeline (i.e. whether MRs are done or not). Thus the rendering time would > be too slow. > > Another approach is to call graphviz command at server side, if viz.js is > not possible. I can't find any pure Java implementation of graphviz. > > Looking forward to your advices. > > Thanks, > Chao
