Sean, these are all sensible questions. As this codebase matures and is eventually committed to Flink, it makes sense to create a guide of cases where one engine would be a better fit than another.
Following the discussion in https://issues.apache.org/jira/browse/SPARK-3561, I read that it is not a goal of the Spark project to have alternative execution engines. Given that, it makes sense to not add support for Tez as an execution engine. What I think is cool in Flink is that swapping the execution engine does not in effect change the complete code path that jobs follow. All of Flink optimization is still used, and also all Flink runtime algorithms (e.g., sorting, hashing, etc) are still used to execute the operators, they just live inside Tez processors rather than Flink job vertices. On Sun, Nov 9, 2014 at 12:09 PM, Sean Owen <sro...@apache.org> wrote: > This was kind of the substance of the same conversation that happened > about Spark on Tez, which looks like it was rejected > (https://issues.apache.org/jira/browse/SPARK-3561): > > - Committing to an SPI interface is hard and imposes its own design > and runtime limitations > - Fragments / forks efforts across two implementations > - If execution engine A is good at one thing and B at another, then > you force a choice that users have difficulty making, when you could > try to improve either engine to do both > - (And in particular, the YARN elasticity thing was in theory being > improved already for Spark anyway, so wasn't as compelling) > > Of course you could make that argument against any abstraction, but > the execution engine is so core to the point of these projects that > the logic may be different. It's not the same as an execution engine > abstracting over different data layers. > > Here, maybe the SPI interface is already there and committed-to, and > there are real benefits to Tez as an alternative. I do think it will > be helpful to further articulate when you would want to use one over > the other, as even I have the same broad question. If they're the > same, what's the point? if they're not, when are they different -- is > it really an issue of resource elasticity vs maturity of the > integration? makes sense to me. > > > On Sun, Nov 9, 2014 at 10:50 AM, Flavio Pompermaier > <pomperma...@okkam.it> wrote: > > Thanks Kostas fir the reply. > > I've already read your first post and what is not fully clear to me is > the > > technical motivation of. "Tez follows design choices that are geared > > towards resource elasticity, whereas the design choices behind Flink's > > engine are geared more towards low latency querying and iterative > > processing". Since in the future I could have to choose one of the 2 I'd > > like to better understand the pros and the cons of the 2 runtimes. > > > > Thanks in advance, > > Flavio > > On Nov 9, 2014 11:02 AM, "Kostas Tzoumas" <ktzou...@apache.org> wrote: > > > >> Flavio, see my first post in this thread. > >> > >> If these differences are fine print for your application/requirements, > then > >> yes, both backends will do the same thing (distribute computation). In > that > >> case, the suggestion would be to use Flink in the normal way, as it is > >> much more mature implementation-wise than Flink-on-Tez. >
