Hi Jonathan, Thanks for your reply. I will go through the material and get back to you!
On 14 March 2018 at 11:46, Jonathan Eagles <[email protected]> wrote: > Anupam, > > I have spent some time looking over the code snippet posted and will try > my best to address your questions. But I do have a foundational question > that will help guide our continued conversations. > > - What role and feature benefits do you see Tez taking on and fulfilling > for Scope? Pig and Hive have taken the approach that Tez will be use both > construct the the DAG to and to provide the runtime execution for which to > run the DAG in a YARN environment running on a hadoop FileSystem API > compatible file system (like HDFS). > > In your reference example, I see you are overriding AbstractLogicalInput. > This is an approach that would be used to make ScopeInput a native type to > Tez as opposed to using input and output plugins to read scope input > without changing Tez code base. I wonder if this is intentional or not. If > so, I apologize for the lacking documentation that might have led you > astray. By attempting to make ScopeInput a native type to Tez, it takes a > direction different than Pig or Hive (or Flink or Cascading for that > matter). From an integration perspective, this places a very large amount > of work of developers working on Scope integration as well as possibly > making contributions troublesome. > > Once we understand the environment and role Tez is to play in Scope, we > (the community) would be happy to help guide you towards Scope integration. > > For reference, here are the links to Pig and Hive. From there have a look > These code snippets below (as well as Tez) are Apache License v2 so please > only read if that is possible. > https://github.com/apache/pig/tree/trunk/src/org/apache/pig/ > backend/hadoop/executionengine/tez/ > Hive also works in a mixed YARN/LLAP runtime environment. That might or > might now be the best example to look at unless it matches the Scope case > as well. Posting only the non-llap code path for example. > https://github.com/apache/hive/tree/master/ql/src/java/ > org/apache/hadoop/hive/ql/exec/tez/ > > Regards, > jeagles >
