#2 JSON format is already supported. #4 is an example how you can translate a specification into a logical plan without any change in core.
I think it is important to discuss this in the context of a use case. Thomas On Mon, Apr 18, 2016 at 10:42 PM, Chinmay Kolhatkar <[email protected]> wrote: > Hi Thomas, > > Can you please elaborate on "different DAG specification can be translated > into a Logical Plan using existing API"? > > I don't have specific usecase in mind. This is just an idea that I wanted > to share with the community on following thought process: > > 1. With current AppFactory impl (for JSON, properties file and > populateDAG), gives us those 3 ways to construct and configure DAG. Lets > say one wants to have own DAG specification format, then that's not > possible as of now. > > 2. JSON format opens a new way of creating DAG. To me, this should a great > power of apex engine that DAG specification and configuration is not > limited to Java code, but even more. Making a configurable AppFactory can > open discussion to even better formats/standards for specifications of DAG. > > 3. If JSON/Properties file format needs a change either in DAG > specification/configuration, this means, the changes has to go in Apex > core. Instead, if we allow overriding AppFactory, one could have more DAG > specifications/conf added to Malhar making lesser changes in apex > core/engine. I'm thinking more in perspective of stability of apex engine > release build. > > 4. Recently Siyuan shared high level APIs for apex application. Having > a separate AppFactory for such high level APIs can make this more modular. > > 5. Another example where this could be useful is Apex DataFrames shared by > Capital One folks. > > Overall, this is just an idea and is not bound by any specific usecase. > The basic idea is > - Plug-ability to apex engine > - Make apex platform more extensible > - Reducing configuration/DAG specification related changes making > apex engine more sturdy. > > > Thanks, > Chinmay. > > > > On Tue, Apr 19, 2016 at 7:56 AM, Thomas Weise <[email protected]> > wrote: > >> Chinmay, >> >> Different DAG specifications can be translated into a logical plan >> using the existing API. >> >> AppFactory is an implementation detail. Can you describe the problem >> you are looking to solve? >> >> Thanks >> >> >> On Mon, Apr 18, 2016 at 6:21 PM, Chinmay Kolhatkar <[email protected]> >> wrote: >> > Dear Community, >> > >> > Does apex engine allows user to override or extend existing >> implementation >> > of AppFactory from outside of engine? >> > >> > If it not, would allowing it make any required launch time enhancements >> > easier and safer (for stability) rather than changing apex engine code? >> > >> > Another advantage I see by doing this is, user can have their own DAG >> > specification format if required. >> > >> > Thanks, >> > Chinmay. >>
