Hello, thanks for the clarifying Aljoscha, you are absolutely right, sorry for my imprecision. I should do my homework and work more with flink :).
For ref, the iteration API: https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/datastream_api.html#iterations And the implementation of the benchmark that effectively does not use iterations: https://github.com/yahoo/streaming-benchmarks/blob/master/flink-benchmarks/src/main/java/flink/benchmark/AdvertisingTopologyNative.java My excuses again, Ismael On Wed, Nov 30, 2016 at 11:30 AM, Aljoscha Krettek <aljos...@apache.org> wrote: > Hi, > there is support for cycles in Flink but the Yahoo benchmark is not making > use of that feature, if I'm not completely mistaken. > > Cheers, > Aljoscha > > On Wed, 30 Nov 2016 at 09:57 Ismaël Mejía <ieme...@gmail.com> wrote: > > > Hello, > > > > Shen you should probably first check the benchmark implementation at > > github, I am not sure you need cycles to implement the yahoo benchmark. > > Notice for example that there is a flink based implementation and AFAIK > > there is no support for cycles in Flink (or at least there wasn't at the > > moment they published the benchmarks). > > > > However if you are implementing the yahoo benchmark on Beam, that would > be > > a nice scenario to test the runner performance (vs the native > > implementations), so it would be nice if you can share this. > > > > Regards, > > Ismael > > > > On Tue, Nov 29, 2016 at 7:32 PM, Shen LI <cs.she...@gmail.com> wrote: > > > > > Hi Maria, Bobby, > > > > > > Thanks for the explanation. > > > > > > Regards, > > > > > > Shen > > > > > > On Tue, Nov 29, 2016 at 12:37 PM, Bobby Evans > > <ev...@yahoo-inc.com.invalid > > > > > > > wrote: > > > > > > > In my experience almost all of the time cycles are bad and cause a > lot > > of > > > > debugging problems. Most of the time you can implement what you want > by > > > > using a windowed join or group by instead. > > > > - Bobby > > > > > > > > On Tuesday, November 29, 2016, 11:06:44 AM CST, María García Herrero > > > > <mari...@google.com.INVALID> wrote:Hi Shen, > > > > > > > > No. Beam pipelines are DAGs: > > > > http://beam.incubator.apache.org/documentation/sdks/ > > > > javadoc/0.3.0-incubating/org/apache/beam/sdk/Pipeline.html > > > > Best, > > > > > > > > María > > > > > > > > On Tue, Nov 29, 2016 at 7:44 AM, Shen LI <cs.she...@gmail.com> > wrote: > > > > > > > > > Hi, > > > > > > > > > > Can I use Beam to create a pipeline with cycles? For example, to > > > > implement > > > > > the Yahoo! Streaming benchmark( > > > > > > > https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming- > > > > > computation-engines-at), > > > > > can a ParDo transform consume a downstream output as a side input? > > > > > > > > > > Thanks, > > > > > > > > > > Shen > > > > > > > > > > > > > > >
