Don't worry, we're all constantly learning. :-) On Wed, 30 Nov 2016 at 12:00 Ismaël Mejía <[email protected]> wrote:
> Hello, thanks for the clarifying Aljoscha, you are absolutely right, sorry > for my imprecision. I should do my homework and work more with flink :). > > For ref, the iteration API: > > > https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/datastream_api.html#iterations > > And the implementation of the benchmark that effectively does not use > iterations: > > > https://github.com/yahoo/streaming-benchmarks/blob/master/flink-benchmarks/src/main/java/flink/benchmark/AdvertisingTopologyNative.java > > My excuses again, > Ismael > > On Wed, Nov 30, 2016 at 11:30 AM, Aljoscha Krettek <[email protected]> > wrote: > > > Hi, > > there is support for cycles in Flink but the Yahoo benchmark is not > making > > use of that feature, if I'm not completely mistaken. > > > > Cheers, > > Aljoscha > > > > On Wed, 30 Nov 2016 at 09:57 Ismaël Mejía <[email protected]> wrote: > > > > > Hello, > > > > > > Shen you should probably first check the benchmark implementation at > > > github, I am not sure you need cycles to implement the yahoo benchmark. > > > Notice for example that there is a flink based implementation and AFAIK > > > there is no support for cycles in Flink (or at least there wasn't at > the > > > moment they published the benchmarks). > > > > > > However if you are implementing the yahoo benchmark on Beam, that would > > be > > > a nice scenario to test the runner performance (vs the native > > > implementations), so it would be nice if you can share this. > > > > > > Regards, > > > Ismael > > > > > > On Tue, Nov 29, 2016 at 7:32 PM, Shen LI <[email protected]> wrote: > > > > > > > Hi Maria, Bobby, > > > > > > > > Thanks for the explanation. > > > > > > > > Regards, > > > > > > > > Shen > > > > > > > > On Tue, Nov 29, 2016 at 12:37 PM, Bobby Evans > > > <[email protected] > > > > > > > > > wrote: > > > > > > > > > In my experience almost all of the time cycles are bad and cause a > > lot > > > of > > > > > debugging problems. Most of the time you can implement what you > want > > by > > > > > using a windowed join or group by instead. > > > > > - Bobby > > > > > > > > > > On Tuesday, November 29, 2016, 11:06:44 AM CST, María García > Herrero > > > > > <[email protected]> wrote:Hi Shen, > > > > > > > > > > No. Beam pipelines are DAGs: > > > > > http://beam.incubator.apache.org/documentation/sdks/ > > > > > javadoc/0.3.0-incubating/org/apache/beam/sdk/Pipeline.html > > > > > Best, > > > > > > > > > > María > > > > > > > > > > On Tue, Nov 29, 2016 at 7:44 AM, Shen LI <[email protected]> > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Can I use Beam to create a pipeline with cycles? For example, to > > > > > implement > > > > > > the Yahoo! Streaming benchmark( > > > > > > > > > https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming- > > > > > > computation-engines-at), > > > > > > can a ParDo transform consume a downstream output as a side > input? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Shen > > > > > > > > > > > > > > > > > > > > >
