LGTM, we can start this.
Thanks, Lionel On Mon, Oct 22, 2018 at 8:58 PM Eugene Liu <liu...@apache.org> wrote: > I'm fine this hook proposal. > > one comment, could we define annotation to take list of hook class names > as well. > > ________________________________ > From: William Guo <gu...@apache.org> > Sent: Monday, October 22, 2018 3:42 PM > To: dev@griffin.incubator.apache.org > Subject: Re: Hooks support in JobService > > Sounds good to me. > > - application.properties would have a property containing list of hook > class names (or probably, spring bean names?), configuring subset and > order > of hook instances enabled at deployment time. > [William] I like a logic name as spring bean name. > > - Hook instantiation mechanism should create them as proper spring > beans, so that they could use same property file for configuration, and > access existing repositories and services. At the same time, only > explicitly enabled hooks should be instantiated, to avoid initializing > any > optional heavy hooks each time. > [William] We can launch our hook mechanism by leveraging spring bean > instantializtion mechanism. > > - Hooks would be implementing GriffinHook interface, and particular > implementation would use any kind of internal logic, from spring > integration to some asynchronous handling. > [William] Sounds good to me. > > - Hook interface would have single method accepting subclass of > GriffinHookEvent. Hook implementation would pick events it would like to > react on, and ignore the others. To give it some structure, events would > be > organized in hierarchy, like JobEvents, JobInstanceEvents, > MeasureEvents, > etc. > [William] Events are common parts for the Event driven flow. > > - New extension points would be added by introducing new types of events > corresponding to some point inside griffin services. Service instance > would > be creating corresponding event object, and call onEvent method on all > hook > instances defined in .properties file, according to defined order. > [William] Let's start with basic events, get it run in prototype, we can > refine this later. > > > > On Mon, Oct 22, 2018 at 2:51 PM Nick Sokolov <chemika...@gmail.com> wrote: > > > I think this is best detail I can give right now without writing any > code: > > > > - application.properties would have a property containing list of hook > > class names (or probably, spring bean names?), configuring subset and > > order > > of hook instances enabled at deployment time. > > - Hook instantiation mechanism should create them as proper spring > > beans, so that they could use same property file for configuration, > and > > access existing repositories and services. At the same time, only > > explicitly enabled hooks should be instantiated, to avoid initializing > > any > > optional heavy hooks each time. > > - Hooks would be implementing GriffinHook interface, and particular > > implementation would use any kind of internal logic, from spring > > integration to some asynchronous handling. > > - Hook interface would have single method accepting subclass of > > GriffinHookEvent. Hook implementation would pick events it would like > to > > react on, and ignore the others. To give it some structure, events > > would be > > organized in hierarchy, like JobEvents, JobInstanceEvents, > > MeasureEvents, > > etc. > > - New extension points would be added by introducing new types of > events > > corresponding to some point inside griffin services. Service instance > > would > > be creating corresponding event object, and call onEvent method on all > > hook > > instances defined in .properties file, according to defined order. > > > > Probably it's time to build some technical prototype. > > > > On Sun, Oct 21, 2018 at 6:46 PM Eugene Liu <liu...@apache.org> wrote: > > > > > Guys, > > > > > > you have agree to integration style that keeps two potential solutions, > > > Hive/Spring, leaves final decision to users. I'm fine on this > agreement. > > > > > > but do you have comprehensive design schema, I think the topic > discussion > > > is not closing in jira, we should keep in same page about > > > event/message/sync/async details. > > > > > > thanks > > > ________________________________ > > > From: William Guo <gu...@apache.org> > > > Sent: Monday, October 22, 2018 8:26 AM > > > To: dev@griffin.incubator.apache.org > > > Subject: Re: Hooks support in JobService > > > > > > Sounds good to me. > > > > > > You can implement your preferred hive-style, I can implement spring > > > integration based on your interface. > > > > > > William > > > > > > On Mon, Oct 22, 2018 at 2:10 AM Nick Sokolov <chemika...@gmail.com> > > wrote: > > > > > > > Hi William, > > > > > > > > Totally agree on keeping interface as abstract as possible. > Interfaces > > > feel > > > > like natural solution here. > > > > It should be relatively easy to provide spring integration > > implementation > > > > of hook interface. > > > > Probably we can even have default spring integration hook provided as > > an > > > > example. > > > > > > > > On Sun, Oct 21, 2018 at 6:33 AM William Guo <gu...@apache.org> > wrote: > > > > > > > > > hi Nick, > > > > > > > > > > I agree with you we need to have some lifecycle hooks for jobs. > > > > > > > > > > No matter which extension mechanism we choose at runtime, > > > > > but we can always abstract common lifecycle interfaces, and > > implements > > > > > different solutions for different environments. > > > > > > > > > > Say, at runtime someone can configure griffin extension mechanism > as > > > > > 'Hive-style' and others can configure as spring integration model. > > > > > > > > > > We cannot decide which one is super than other, but we can > implement > > > > > multiply solutions and let users decide at deployment time? > > > > > > > > > > What do you think? > > > > > > > > > > Thanks, > > > > > William > > > > > > > > > > > > > > > > > > > > On Fri, Oct 19, 2018 at 2:03 PM Nick Sokolov <chemika...@gmail.com > > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I'd like invite the community to discussion of Job object > lifecycle > > > > hooks > > > > > > design (GRIFFIN-200 < > > > https://issues.apache.org/jira/browse/GRIFFIN-200 > > > > > >). > > > > > > > > > > > > On a high level, idea is to have hooks executed triggered by > > > JobService > > > > > > before/after job is created, or before/after job is deleted, and > so > > > on. > > > > > > Main point is to provide extension mechanism for JobService, > > allowing > > > > > > third-party implementations of following features: > > > > > > > > > > > > - validating naming conventions on the jobs > > > > > > - validating cron schedules (whether job is scheduled "not too > > > > > > frequently") > > > > > > - creating dashboards for each new job and/or alerts in > > > third-party > > > > > > system (Grafana, Elastalert) > > > > > > - RBAC implementation (Hive-style, as plugin) > > > > > > - disabling alerts in third-party system if job is paused > > > > > > > > > > > > Right now several aspects are to be decided: > > > > > > > > > > > > 1. particular extention mehanism (POJO interface > implementations > > > > > > ("Hive-style"), or Spring-Integration, etc) > > > > > > 2. whether only one or multiple hooks could be configured at > > same > > > > time > > > > > > 3. execution model: whether they should run synchronously, or > > > > > > asynchronously; sequentially, or in parallel > > > > > > > > > > > > Ticket already has some discussion going, so inviting everyone to > > > Jira > > > > > > ticket, to avoid having same conversation in two places. > > > > > > > > > > > > > > > > > > > > >
