Thanks for the opinion. Currently some of policy-related code(e.g. SchedulingPolicy) depend on mechanism-related code(e.g. PhysicalStage). That said, SchedulingPolicy depends on 'declaration' of PhysicalStage, not the actual 'implementation' of it. And I agree mechanism-related code should stay within nemo-runtime-* module.
Maybe we can add PhysicalStage as an interface in nemo-common and rename the current PhysicalStage to PhysicalStageImpl, implementing the PhyscalStage interface. Regards, Jangho On 06/01/2018 05:49 PM, John Yang wrote: > Sounds good to me. > > > I agree that policy-related code should go into 'nemo-common' for the > compiler-optimizer to optimize, and the runtime to execute. > I guess 'KeyExtractor' and the ir package belong to this category, and > that's why they're in 'nemo-common' > > On the other hand, mechanism-related code should stay in > 'nemo-runtime-common' and should be exposed only to the > compiler-backend/nemo. > Things like 'PhysicalStage' seems to belong to this category, as it'd be > unlikely that compiler-optimizer would want to do something with it. > > > Thanks, > John > > On Fri, Jun 1, 2018 at 3:59 PM, JangHo Seo <[email protected]> wrote: > >> Hi Nemo devs, >> >> Currently Nemo codebase is splitted into multiple Maven modules. >> While it's good for ease of maintenance and modularity, a module cannot >> reference classes and/or interfaces in other modules unless it declares >> them as dependencies in pom.xml. >> Sometimes it makes difficult to expose runtime-level behaviors as >> IR-level abstractions, when nemo-compiler-* modules cannot access the >> related definitions in nemo-runtime-* modules. >> >> A good example is [NEMO-73] 'SchedulingPolicy as Vertex-levelExecution >> Property'. The issue is to expose SchedulingPolicy to compiler passes so >> that users can configure scheduling behavior using Nemo IR. >> Unfortunately 'SchedulingPolicy' is defined in 'nemo-runtime-common', so >> compilers cannot access it and I think the proper fix is to move it to >> the 'nemo-common' module. >> Also, to move 'SchedulingPolicy' to 'nemo-common' module, lots of >> definitions in 'nemo-runtime-common' should also be moved to 'nemo-common'. >> (Because 'SchedulingPolicy' is a function of 'ExecutorRepresenter's and >> 'ExecutableTask', and the definition of 'ExecutableTask' depends on >> 'PhysicalStageEdge', 'PhysicalStage', and 'KeyRange', and so on...) >> >> The module 'nemo-runtime-common' has definitions that define 'policies', >> as well as definitions that implement 'mechanisms'. >> 'KeyRange' is an example for the former (defines how to partition the >> data in a block), and 'MessageListener' is an example for the latter >> (implements RPC). >> In my opinion, policy-related definitions should be moved to >> nemo-common, so that we can add more ExecutionProperties about >> scheduilng, dynamic optimization, state management, and so on. >> >> What do you think about this issue? >> >> Regards, >> Jangho >> >>
