Hi, there is support for cycles in Flink but the Yahoo benchmark is not making use of that feature, if I'm not completely mistaken.
Cheers, Aljoscha On Wed, 30 Nov 2016 at 09:57 Ismaël Mejía <[email protected]> wrote: > Hello, > > Shen you should probably first check the benchmark implementation at > github, I am not sure you need cycles to implement the yahoo benchmark. > Notice for example that there is a flink based implementation and AFAIK > there is no support for cycles in Flink (or at least there wasn't at the > moment they published the benchmarks). > > However if you are implementing the yahoo benchmark on Beam, that would be > a nice scenario to test the runner performance (vs the native > implementations), so it would be nice if you can share this. > > Regards, > Ismael > > On Tue, Nov 29, 2016 at 7:32 PM, Shen LI <[email protected]> wrote: > > > Hi Maria, Bobby, > > > > Thanks for the explanation. > > > > Regards, > > > > Shen > > > > On Tue, Nov 29, 2016 at 12:37 PM, Bobby Evans > <[email protected] > > > > > wrote: > > > > > In my experience almost all of the time cycles are bad and cause a lot > of > > > debugging problems. Most of the time you can implement what you want by > > > using a windowed join or group by instead. > > > - Bobby > > > > > > On Tuesday, November 29, 2016, 11:06:44 AM CST, María García Herrero > > > <[email protected]> wrote:Hi Shen, > > > > > > No. Beam pipelines are DAGs: > > > http://beam.incubator.apache.org/documentation/sdks/ > > > javadoc/0.3.0-incubating/org/apache/beam/sdk/Pipeline.html > > > Best, > > > > > > María > > > > > > On Tue, Nov 29, 2016 at 7:44 AM, Shen LI <[email protected]> wrote: > > > > > > > Hi, > > > > > > > > Can I use Beam to create a pipeline with cycles? For example, to > > > implement > > > > the Yahoo! Streaming benchmark( > > > > > https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming- > > > > computation-engines-at), > > > > can a ParDo transform consume a downstream output as a side input? > > > > > > > > Thanks, > > > > > > > > Shen > > > > > > > > > >
