Nice! As-is, this already looks useful for making Beam accessible. Commented a bit on doc to highlight where SQL is different than Scio/Python style. I think notebooks are the perfect target. Specifically, Python and SQL on the same notebook would be amazing.
Kenn On Thu, Jun 14, 2018 at 2:04 PM Sindy Li <qiny...@google.com> wrote: > Thanks Ahmet, > > We know quite a few teams in Google are interested to run interactive Beam > pipelines, especially in Python for Machine Learning -- some are already > using it interactively in their own way. So instead of for the those teams > to develop their own version of interactive solution, we want one > repository that people can contribute to. We could also provide better > features like fast re-execution as is shown in the demo. > > Thanks, > Sindy > > On Wed, Jun 13, 2018 at 5:48 PM, Ahmet Altay <al...@google.com> wrote: > >> Thank you Sindy. >> >> I like the demo; it looks great. This would be interesting to a lot of >> users. What are your plans for moving this forward? What kind of an input >> you are looking for? >> >> Ahmet >> >> On Wed, Jun 13, 2018 at 2:32 PM, Eugene Kirpichov <kirpic...@google.com> >> wrote: >> >>> This is awesome, thanks Sindy! I hope that the questions related to >>> portability will get resolved in a way that will allow to reuse some of the >>> work for other interactive Beam experiences, including SQL as Andrew says, >>> and providing a REPL e.g. for users of Scala or other JVM-based languages. >>> >>> +Neville Li <nevi...@spotify.com> Do I remember correctly that you guys >>> had some sort of interactivity going in Scio but were looking forward to >>> Beam developing a native solution? >>> >>> On Wed, Jun 13, 2018 at 2:22 PM Sindy Li <qiny...@google.com> wrote: >>> >>>> *Thanks, Andrew!* >>>> >>>> *Here is a link to the demo on Youtube for people interested:* >>>> *https://www.youtube.com/watch?v=c5CjA1e3Cqw&feature=youtu.be >>>> <https://www.youtube.com/watch?v=c5CjA1e3Cqw&feature=youtu.be>* >>>> >>>> On Wed, Jun 13, 2018 at 1:23 PM, Andrew Pilloud <apill...@google.com> >>>> wrote: >>>> >>>>> This sounds really interesting, thanks for sharing! We've just begun >>>>> to explore making Beam SQL interactive. The Interactive Runner you've >>>>> proposed sounds like it would solve a bunch of the problems SQL faces as >>>>> well. SQL is written in Java right now, so we can't immediately reuse any >>>>> code. >>>>> >>>>> Andrew >>>>> >>>>> On Wed, Jun 13, 2018 at 11:48 AM Sindy Li <qiny...@google.com> wrote: >>>>> >>>>>> Resending after subscribing to dev list. >>>>>> >>>>>> ---------- Forwarded message ---------- >>>>>> From: Sindy Li <qiny...@google.com> >>>>>> Date: Fri, Jun 8, 2018 at 5:57 PM >>>>>> Subject: Proposing interactive beam runner >>>>>> To: dev@beam.apache.org >>>>>> Cc: Harsh Vardhan <anan...@google.com>, Chamikara Jayalath < >>>>>> chamik...@google.com>, Anand Iyer <ian...@google.com>, Robert >>>>>> Bradshaw <rober...@google.com> >>>>>> >>>>>> >>>>>> Hello, >>>>>> >>>>>> We were exploring ways to provide an interactive notebook experience >>>>>> for writing Beam Python pipelines. The design doc >>>>>> <https://docs.google.com/document/d/10bTc97GN5Wk-nhwncqNq9_XkJFVVy0WLT4gPFqP6Kmw/edit?usp=sharing> >>>>>> provides >>>>>> an overview/vision of what we would like to achieve. Pull request >>>>>> <https://github.com/apache/beam/pull/5595> provides a prototype for >>>>>> the same. The document also provides demo screen shots and >>>>>> instructions for running a demo in Jupyter. Please take a look. We >>>>>> believe >>>>>> this would be a useful addition to Beam. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> >>>>>> >>>> >> >
