Thanks Amit, that does clear things up!
On Thu, Jul 7, 2016 at 3:30 PM, Amit Sela <amitsel...@gmail.com> wrote:
> I don't think that the Spark runner is special, it's just the way it was
> until now and that's why I brought up the subject here.
>
> The main issue is that currently, if a user wants to write a beam app using
> the Spark runner, he'll have to provide the Spark dependencies, or he'll
> get a ClassNotFoundException (which is exactly the case for beam-examples).
> This of course happens because the Spark runner has provided dependency on
> Spark (not transitive).
>
Having provided dependencies and making the user include them in their pom
is
pretty normal, I think. We already require users to provide a slf4j logger
and
Hamcrest+Junit (if they use PAssert).
(We including all these in the examples pom.xml
<https://github.com/apache/incubator-beam/blob/master/examples/java/pom.xml#L286>
.)
I don't see any problem for a user who wants to use the Spark runner to add
these
provided deps to their pom (aka, putting them as runtime deps in examples
pom.xml).
> The Flink runner avoids this issue by having a compile dependency on flink,
> thus being transitive.
>
> By having the cluster provide them I mean that the Spark installation is
> aware of the binaries pre-deployed on the cluster and adds them to the
> classpath of the app submitted for execution on the cluster - this is
> common (AFAIK) for Spark and Spark on YARN, and vendors provide similar
> binaries, for example: spark-1.6_hadoop-2.4.0_hdp.xxx.jar (Hortonworks).
>
Makes sense. So a user submitting to a cluster would submit a jar and
command-line
options, and the cluster itself would add the provided deps.
> Putting aside our (Beam) issues, the current artifact "beam-runners-spark"
> is more suitable to run on clusters with pre-deployed binaries rather than
> a
> quick standalone execution with a single dependency that takes care of
> everything (Spark related),
great!
> but is more cumbersome for users trying to get
> going for the first time, which is not good!.
>
We should decide which experience we're trying to optimize for (I'd lean
cluster), but
I think that we should update examples pom.xml with the support.
* For cluster mode default, we would add a profile for 'local' mode
(-PsparkIncluded or something) that overrides the provided deps to be
runtime
deps instead.
* We can include switching the profile for local mode in the "getting
started" instructions.
Dan
I guess Flink uses a compile dependency for the same reason Spark uses
> provided - because it fits them - what about other runners ?
>
> Hope this clarifies some of the questions here.
>
> Thanks,
> Amit
>
> On Fri, Jul 8, 2016 at 12:52 AM Dan Halperin <dhalp...@google.com.invalid>
> wrote:
>
> > hey folks,
> >
> > In general, we should optimize for running on clusters rather than
> running
> > locally. Examples is a runner-independent module, with non-compile-time
> > deps on runners. Most runners are currently listed as being runtime deps
> --
> > it sounds like that works, for most cases, but might not be the best fit
> > for Spark.
> >
> > Q: What does dependencies being provided by the cluster mean? I'm a
> little
> > naive here, but how would a user submit a pipeline to a Spark cluster
> > without actually depending on Spark in mvn? Is it not by running the main
> > method in an example like in all other runners?
> >
> > I'd like to understand the above better, but suppose that to optimize for
> > Spark-on-a-cluster, we should default to provided deps in the examples.
> > That would be fine -- but couldn't we just make a profile for local Spark
> > that overrides the deps from provided to runtime?
> >
> > To summarize, I think we do not need new artifacts, but we could use a
> > profile for local testing if absolutely necessary.
> >
> > Thanks,
> > Dan
> >
> > On Thu, Jul 7, 2016 at 2:27 PM, Ismaël Mejía <ieme...@gmail.com> wrote:
> >
> > > Good discussion subject Amit,
> > >
> > > I let the whole beam distribution subjects continue in BEAM-320,
> however
> > > there
> > > is a not yet discussed aspect of the spark runner, the maven behavior:
> > >
> > > When you import the beam spark runner as a dependency you are obliged
> to
> > > provide
> > > your spark dependencies by hand too, in the other runners once you
> import
> > > the
> > > runner everything just works e.g. google-cloud-dataflow-runner and
> > > flink-runner. I understand the arguments for the current setup (the
> ones
> > > you
> > > mention), but I think it is more user friendly to be consistent with
> the
> > > other
> > > runners and have something that just works as the default (and solve
> the
> > > examples issue as a consequence). Anyway I think in the spark case we
> > need
> > > both, an 'spark-included' flavor and the current one that it is really
> > > useful to
> > > include the runner as a spark library dependency (like Jesse did in his
> > > video) or
> > > as a spark-package.
> > >
> > > Actually both the all-included and the runner only make sense for flink
> > too
> > > but this is a different discussion ;)
> > >
> > > What do you think about this ? What do the others think ?
> > >
> > > Ismaël
> > >
> > >
> > > On Thu, Jul 7, 2016 at 10:19 PM, Jean-Baptiste Onofré <j...@nanthrax.net
> >
> > > wrote:
> > >
> > > > No problem and good idea to discuss in the Jira.
> > > >
> > > > Actually, I started to experiment a bit beam distributions on a
> branch
> > > > (that I can share with people interested).
> > > >
> > > > Regards
> > > > JB
> > > >
> > > >
> > > > On 07/07/2016 10:12 PM, Amit Sela wrote:
> > > >
> > > >> Thanks JB, I've missed that one.
> > > >>
> > > >> I suggest we continue this in the ticket comments.
> > > >>
> > > >> Thanks,
> > > >> Amit
> > > >>
> > > >> On Thu, Jul 7, 2016 at 11:05 PM Jean-Baptiste Onofré <
> j...@nanthrax.net
> > >
> > > >> wrote:
> > > >>
> > > >> Hi Amit,
> > > >>>
> > > >>> I think your proposal is related to:
> > > >>>
> > > >>> https://issues.apache.org/jira/browse/BEAM-320
> > > >>>
> > > >>> As described in the Jira, I'm planning to provide (in dedicated
> Maven
> > > >>> modules) is a Beam distribution including:
> > > >>> - an uber jar to wrap the dependencies
> > > >>> - the underlying runtime backends
> > > >>> - etc
> > > >>>
> > > >>> Regards
> > > >>> JB
> > > >>>
> > > >>> On 07/07/2016 07:49 PM, Amit Sela wrote:
> > > >>>
> > > >>>> Hi everyone,
> > > >>>>
> > > >>>> Lately I've encountered a number of issues concerning the fact
> that
> > > the
> > > >>>> Spark runner does not package Spark along with it and forcing
> people
> > > to
> > > >>>>
> > > >>> do
> > > >>>
> > > >>>> this on their own.
> > > >>>> In addition, this seems to get in the way of having beam-examples
> > > >>>>
> > > >>> executed
> > > >>>
> > > >>>> against the Spark runner, again because it would have to add Spark
> > > >>>> dependencies.
> > > >>>>
> > > >>>> When running on a cluster (which I guess was the original goal
> > here),
> > > it
> > > >>>>
> > > >>> is
> > > >>>
> > > >>>> recommended to have Spark provided by the cluster - this makes
> sense
> > > for
> > > >>>> Spark clusters and more so for Spark + YARN clusters where you
> might
> > > >>>> have
> > > >>>> your Spark built against a specific Hadoop version or using a
> vendor
> > > >>>> distribution.
> > > >>>>
> > > >>>> In order to make the runner more accessible to new adopters, I
> > suggest
> > > >>>> to
> > > >>>> consider releasing a "spark-included" artifact as well.
> > > >>>>
> > > >>>> Thoughts ?
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Amit
> > > >>>>
> > > >>>>
> > > >>> --
> > > >>> Jean-Baptiste Onofré
> > > >>> jbono...@apache.org
> > > >>> http://blog.nanthrax.net
> > > >>> Talend - http://www.talend.com
> > > >>>
> > > >>>
> > > >>
> > > > --
> > > > Jean-Baptiste Onofré
> > > > jbono...@apache.org
> > > > http://blog.nanthrax.net
> > > > Talend - http://www.talend.com
> > > >
> > >
> >
>
