IMO Beam would greatly benefit from having a monitoring UI. Individual runners (Spark, Dataflow) have their own UI's, which could be exposed through an API as you suggest.
I also think there would be value in having a Beam-centric monitoring UI which focuses on concepts in the Beam model (PTransforms/PCollections, windows/triggers, bundles, metrics, state/timers, etc.) Dataflow's UI is pretty good, but I believe most users develop and debug locally (i.e. DirectRunner) before submitting to a cluster. On Mon, Oct 8, 2018 at 2:37 PM Winkelman, Kyle G <[email protected]> wrote: > Hello, > > > > I am interested in exposing a monitoring url through the PipelineResult > object. All of the documentation pages in the runners tab of > beam.apache.org have a section dedicated to ‘Monitoring your job’ and > most of those reference a web interface. If we were able to get a link to > this web interface, we would be able to lead non-technical users of our > application to a place where they can get some understanding of the > progress of the job. > > > > An example of this in the SparkPipelineResult would be returning the value > of the javaSparkContext.sc().uiWebUrl().get(). I haven’t looked into other > runners but if most of them have web interfaces it would make sense that > they would be able to programmatically get a url to that web interface. > > > > Looking for feedback on if this would be accepted and ways this can be > accomplished in other runners. > > > > Thanks, > > Kyle Winkelman > > > > > This e-mail, including attachments, may include confidential and/or > proprietary information, and may be used only by the person or entity > to which it is addressed. If the reader of this e-mail is not the intended > recipient or his or her authorized agent, the reader is hereby notified > that any dissemination, distribution or copying of this e-mail is > prohibited. If you have received this e-mail in error, please notify the > sender by replying to this message and delete this e-mail immediately. > -- Got feedback? tinyurl.com/swegner-feedback
