Hi Thomas, Can you please elaborate on "different DAG specification can be translated into a Logical Plan using existing API"?
I don't have specific usecase in mind. This is just an idea that I wanted to share with the community on following thought process: 1. With current AppFactory impl (for JSON, properties file and populateDAG), gives us those 3 ways to construct and configure DAG. Lets say one wants to have own DAG specification format, then that's not possible as of now. 2. JSON format opens a new way of creating DAG. To me, this should a great power of apex engine that DAG specification and configuration is not limited to Java code, but even more. Making a configurable AppFactory can open discussion to even better formats/standards for specifications of DAG. 3. If JSON/Properties file format needs a change either in DAG specification/configuration, this means, the changes has to go in Apex core. Instead, if we allow overriding AppFactory, one could have more DAG specifications/conf added to Malhar making lesser changes in apex core/engine. I'm thinking more in perspective of stability of apex engine release build. 4. Recently Siyuan shared high level APIs for apex application. Having a separate AppFactory for such high level APIs can make this more modular. 5. Another example where this could be useful is Apex DataFrames shared by Capital One folks. Overall, this is just an idea and is not bound by any specific usecase. The basic idea is - Plug-ability to apex engine - Make apex platform more extensible - Reducing configuration/DAG specification related changes making apex engine more sturdy. Thanks, Chinmay. On Tue, Apr 19, 2016 at 7:56 AM, Thomas Weise <[email protected]> wrote: > Chinmay, > > Different DAG specifications can be translated into a logical plan > using the existing API. > > AppFactory is an implementation detail. Can you describe the problem > you are looking to solve? > > Thanks > > > On Mon, Apr 18, 2016 at 6:21 PM, Chinmay Kolhatkar <[email protected]> > wrote: > > Dear Community, > > > > Does apex engine allows user to override or extend existing > implementation > > of AppFactory from outside of engine? > > > > If it not, would allowing it make any required launch time enhancements > > easier and safer (for stability) rather than changing apex engine code? > > > > Another advantage I see by doing this is, user can have their own DAG > > specification format if required. > > > > Thanks, > > Chinmay. >
