"Back to my original post, I essentially want to add Flink to that list"
In that case, inside the Flink interpreter source code, everytime the input
parser encouters a ${variable} pattern, you have to access the
AngularObjectRegistry and replace the template by the actual variable value.
It is the responsibility for each interpreter to implement variable
interpolation (${var})
I did it for the Cassandra interpreter using my own syntax ( {{var}} ) :
https://github.com/apache/incubator-zeppelin/blob/master/cassandra/src/main/scala/org/apache/zeppelin/cassandra/InterpreterLogic.scala#L306-L327
"was looking through your resourcePools. I am under the impression I can use
those to pass a variable from one paragraph to another, in an akward sort
of fasion (but I may be going about it all wrong). Supposing that can be
done (or possibly is already done, but I haven't read the PRs you
listed carefully),
it would solve what I want to do for the time being."
I will create an epic to merge angular object with resource pools to keep
only one abstraction. But it doesn't solve the fundamental problem, which
is IF an interpreter wants to use variables stored in resource pool, it HAS
to implement it.
The only way we can mutualise code for variable binding is to let Zeppelin
Engine pre-process the input text bloc of each paragraph and perform
variable lookup from Resource Pool then variable replace, and after that
forward the text block to the interpreter itself.
I think it is a good idea but it would require some refactoring and may
break existing behaviors if some interpreter already implemented their own
variable template handling
"2) If we want to keep the code base compact and clean, would it be wiser
to refactor in a less Spark-centric way?"
There is nothing Spark centric here if we're talking about variable
sharing, it applies to all interpreters
On Fri, Apr 22, 2016 at 11:24 PM, Trevor Grant <trevor.d.gr...@gmail.com>
wrote:
> If I'm reading https://issues.apache.org/jira/browse/ZEPPELIN-635
> correctly- this integrates the spark, markdown, and shell interpreters.
>
> Back to my original post, I essentially want to add Flink to that list.
>
> To your point about keeping a small and managable code-base: Under the
> hood it seems like Zeppelin is a front end for Spark and oh btw, here are
> some hacks to make other stuff work too. For instance there is a lot of
> code reusage in any scala based interpreter. Wouldn't it make more sense
> to have a generic Scala interpreter and extend it for special quirks of
> each interpreter as needed, e.g. for the variable bindings of the
> particular interpreter, and loading configurations. Consider the companion
> object bug, essentially the same code had to be copy and pasted across 4
> interpreters, and the Ignite interpreter (as I recall) never even got the
> fix because of a quirk in the way the tests are written for that
> interpreter.
>
> I was looking through your resourcePools. I am under the impression I can
> use those to pass a variable from one paragraph to another, in an akward
> sort of fasion (but I may be going about it all wrong). Supposing that can
> be done (or possibly is already done, but I haven't read the PRs you listed
> carefully), it would solve what I want to do for the time being.
>
> Also consider the Python Flink I want to add to this, there will once again
> be a lot of duplication of code from the Spark Python interpreter. A
> generic Python interpreter also seems like a more reasonable approach here.
>
> So basically I've broken this conversation into two parts-
> 1) I'm trying to pass variables/object back and forth between
> Spark/Flink/Angular/etc. Please help. Seems possible but I'm having a slow
> time figuring it out
> 2) If we want to keep the code base compact and clean, would it be wiser to
> refactor in a less Spark-centric way?
>
>
>
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> *"Fortunate is he, who is able to know the causes of things." -Virgil*
>
>
> On Fri, Apr 22, 2016 at 3:41 PM, DuyHai Doan <doanduy...@gmail.com> wrote:
>
> > In this case, it is already implemented.
> >
> > Look at those merged PR:
> >
> > - https://github.com/apache/incubator-zeppelin/pull/739
> > - https://github.com/apache/incubator-zeppelin/pull/740
> > - https://github.com/apache/incubator-zeppelin/pull/741
> > - https://github.com/apache/incubator-zeppelin/pull/742
> > - https://github.com/apache/incubator-zeppelin/pull/744
> > - https://github.com/apache/incubator-zeppelin/pull/745
> > - https://github.com/apache/incubator-zeppelin/pull/832
> >
> > There is one last JIRA pending for documentation, I'll do a PR for this
> > next week: https://issues.apache.org/jira/browse/ZEPPELIN-742
> >
> > On Fri, Apr 22, 2016 at 9:52 PM, Trevor Grant <trevor.d.gr...@gmail.com>
> > wrote:
> >
> > > I want to be able to put/get/watch variables. Specifically so I can
> > > interface with AngularJS for visualizations.
> > >
> > > I've been groking the codebase trying to find a less invasive way to do
> > > this.
> > >
> > > I get wanting to keep the code base clean but sharing variables is a
> > really
> > > nice feature set and shouldn't be that hard to implement?
> > >
> > > Thoughts?
> > >
> > > Trevor Grant
> > > Data Scientist
> > > https://github.com/rawkintrevo
> > > http://stackexchange.com/users/3002022/rawkintrevo
> > > http://trevorgrant.org
> > >
> > > *"Fortunate is he, who is able to know the causes of things." -Virgil*
> > >
> > >
> > > On Fri, Apr 22, 2016 at 1:06 PM, DuyHai Doan <doanduy...@gmail.com>
> > wrote:
> > >
> > > > I think we should rather let ZeppelinContext un-modified.
> > > >
> > > > If we update ZeppelinContext for every kind of interpreter, it would
> > > become
> > > > quickly a behemoth and un-manageable.
> > > >
> > > > The reason ZeppelinContext has some support for Spark is because it's
> > > > historical. Now that the project is going to gain wider audience, we
> > > should
> > > > focus on keeping the code as cleanest and as modular as possible.
> > > >
> > > > Can you explain which feature you want to add to ZeppelinContext that
> > > will
> > > > be useful for Flink ?
> > > >
> > > >
> > > >
> > > > On Fri, Apr 22, 2016 at 7:12 PM, Trevor Grant <
> > trevor.d.gr...@gmail.com>
> > > > wrote:
> > > >
> > > > > If one were to extend the Zeppelin context for Flink, I was
> thinking
> > it
> > > > > would make the most sense to update
> > > > >
> > > > >
> ../spark/src/main/java/org/apache/zeppelin/spark/ZeppelinContext.java
> > > > >
> > > > > Any thoughts from those who are more familiar with that end of the
> > code
> > > > > base than I?
> > > > >
> > > > > Ideally we'd have a solution that extend the Zeppelin Context to
> all
> > > > > interpreters. I know y'all love Spark but there ARE others out
> > > there...
> > > > >
> > > > > Anyone have any branches / previous attempts I could check out?
> > > > >
> > > > > tg
> > > > >
> > > > >
> > > > > Trevor Grant
> > > > > Data Scientist
> > > > > https://github.com/rawkintrevo
> > > > > http://stackexchange.com/users/3002022/rawkintrevo
> > > > > http://trevorgrant.org
> > > > >
> > > > > *"Fortunate is he, who is able to know the causes of things."
> > -Virgil*
> > > > >
> > > >
> > >
> >
>
