I am thinking of a boolean property named "apex.run.simulate" On Thu, Dec 21, 2017 at 9:05 AM, Sanjay Pujare <san...@datatorrent.com> wrote:
> It is relatively easy to describe the justification for this change without > getting into the weeds and hairsplitting words. > > A DAG is built not only to launch an application but also to let a user > visualize and configure it. Currently "populateDAG" is the only method we > require application writers to implement and they implement it with the > goal of running the application. So it can use properties, configuration > and code that is really only needed if you want to "run" the DAG. > > As mentioned above a perfectly valid use case is that a platform allows a > user to construct a DAG, visualize it and then attach configuration values > to various components in the DAG, save these values as some kind of a > "configuration package" and then at a future date run the DAG with this > setup. This is consistent with the view that construction of a pipeline and > execution of the pipeline are 2 separate phases and should be delineated as > such. > > If you understand and agree with the justification we can work on improving > the original proposal. > > Sanjay > > > On Thu, Dec 21, 2017 at 8:27 AM, Vlad Rozov <vro...@apache.org> wrote: > > > "Sometimes" is not a use case. Config is not a context. > > > > Without concrete use cases the proposed change is not well justified. > > populateDAG() is supposed to populate DAG, not to record anything in an > > external system. It was a design goal for plugins. > > > > Thank you, > > > > Vlad > > > > > > On 12/20/17 02:23, Priyanka Gugale wrote: > > > >> +1 > >> Sometimes this context is required. We shouldn't change any default > >> behaviour other than making this config available. > >> > >> -Priyanka > >> > >> > >> > >> On Wed, Dec 20, 2017 at 5:32 AM, Pramod Immaneni < > pra...@datatorrent.com> > >> wrote: > >> > >> The external system recording was just an example, not a specific use > >>> case. > >>> The idea is to provide comprehensive information to populateDAG as to > the > >>> context it is being called under. It is akin to the test mode or > simulate > >>> flag that you see with various utilities. The platform cannot control > >>> what > >>> populateDAG does, even without this information, in multiple calls that > >>> you > >>> mention the application can return different DAGs by depending on > >>> any external factor such as time of day or some external variable. This > >>> is > >>> to merely provide more context information in the config. It is upto > the > >>> application to do what it wishes with it. > >>> > >>> On Tue, Dec 19, 2017 at 2:28 PM, Vlad Rozov <vro...@apache.org> wrote: > >>> > >>> -0.5: populateDAG() may be called by the platform as many times as it > >>>> needs (even in case it calls it only once now to launch an > application). > >>>> Passing different parameters to populateDAG() in simulate launch mode > >>>> and > >>>> actual launch may lead to different DAG being constructed for those > two > >>>> modes. Can't the use case you described be handled by a plugin? > >>>> > >>>> Thank you, > >>>> > >>>> Vlad > >>>> > >>>> > >>>> On 12/19/17 10:06, Sanjay Pujare wrote: > >>>> > >>>> +1 although I prefer something that is more enforceable. So I like the > >>>>> idea > >>>>> of another method but that introduces incompatibility so may be in > 4.0? > >>>>> > >>>>> On Tue, Dec 19, 2017 at 9:40 AM, Munagala Ramanath < > >>>>> amberar...@yahoo.com.invalid> wrote: > >>>>> > >>>>> +1 > >>>>> > >>>>>> Ram > >>>>>> On Tuesday, December 19, 2017, 8:33:21 AM PST, Pramod > Immaneni < > >>>>>> pra...@datatorrent.com> wrote: > >>>>>> > >>>>>> I have a mini proposal. The command get-app-package-info runs the > >>>>>> populateDAG method of an application to construct the DAG but does > not > >>>>>> actually launch the DAG. An application developer does not know in > >>>>>> > >>>>> which > >>> > >>>> context the populateDAG is being called. For example, if they are > >>>>>> recording > >>>>>> application starts in an external system from populateDAG, they will > >>>>>> > >>>>> have > >>> > >>>> false entries there. This can be solved in different ways such as > >>>>>> introducing another method in StreamingApplication or more > parameters > >>>>>> to populateDAG but a non disruptive option would be to add a > property > >>>>>> > >>>>> in > >>> > >>>> the configuration object that is passed to populateDAG to indicate if > >>>>>> > >>>>> it > >>> > >>>> is > >>>>>> simulate/test mode or real launch. An application developer can use > >>>>>> > >>>>> this > >>> > >>>> property to take the appropriate actions. > >>>>>> > >>>>>> Thanks > >>>>>> > >>>>>> > >>>>>> > >>>>>> > > >
