For the sake of prototyping, can you use a util that simply materializes
the intermediate result in a file system (using typeInfo input and output
formats) ?
On Tue, Apr 7, 2015 at 6:21 PM, Maximilian Michels <m...@apache.org> wrote:
> On Mon, Apr 6, 2015 at 2:37 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > BTW: Should "print()" be also an "eager" statement? I think it needs to
> be,
> > if we want to print to the driver's std out
>
>
> Yes, if we change print() to print on the Client, then it needs to execute
> eagerly.
>
> On Thu, Apr 2, 2015 at 6:59 PM, Alexander Alexandrov <
> alexander.s.alexand...@gmail.com> wrote:
>
> > I would like to run a dataflow up to a particular point and materialize
> (in
> > memory) the intermediate result. Is this possible at the moment?
> >
>
> Blocking execution mode is already implemented but the results are
> currently not cached. That means that they are lost after they are
> consumed.
>
> On Mon, Apr 6, 2015 at 2:37 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > count() and collect() need to immediately trigger an execution, because
> the
> > driver program cannot proceed otherwise. They are "eager".
> >
> > Regular sinks are "lazy", they wait until the program is triggered
> anyways.
> >
> > BTW: Should "print()" be also an "eager" statement? I think it needs to
> be,
> > if we want to print to the driver's std out.
> >
> > On Thu, Apr 2, 2015 at 5:51 PM, Aljoscha Krettek <aljos...@apache.org>
> > wrote:
> >
> > > In my opinion it should not be handled like print. The idea behind
> > > count()/collect() is that they immediately return the result which can
> > > then be used in further flink operations.
> > >
> > > Right now, this is not properly/efficiently implemented but once we
> > > have support for intermediate results on this level they start making
> > > more sense. Also, in such a case an execute would not be required
> > > after a collect()/count() if only the result of that call is required.
> > >
> > > On Thu, Apr 2, 2015 at 5:33 PM, Felix Neutatz <neut...@googlemail.com>
> > > wrote:
> > > > Hi,
> > > >
> > > > I have run the following program:
> > > >
> > > > final ExecutionEnvironment env =
> > > ExecutionEnvironment.getExecutionEnvironment();
> > > >
> > > > List l = Arrays.asList(new Tuple1<Long>(1L));
> > > > TypeInformation t = TypeInfoParser.parse("Tuple1<Long>");
> > > > DataSet<Tuple1<Long>> data = env.fromCollection(l, t);
> > > >
> > > > long value = data.count();
> > > > System.out.println(value);
> > > >
> > > > env.execute("example");
> > > >
> > > >
> > > > Since there is no "real" data sink, I get the following:
> > > > Exception in thread "main" java.lang.RuntimeException: No data sinks
> > have
> > > > been created yet. A program needs at least one sink that consumes
> data.
> > > > Examples are writing the data set or printing it.
> > > >
> > > > In my opinion, we should handle count() and collect() like print().
> > > >
> > > > What do you think?
> > > >
> > > > Best regards,
> > > >
> > > > Felix
> > >
> >
>
