hi all, This is our new arch diagram https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
We will build milestones based on this proposal. Thanks, William On Tue, Aug 9, 2022 at 8:22 PM William Guo <gu...@apache.org> wrote: > hi > > test for new subscriber, please ignore this. > > > Thanks, > William > > On Mon, Aug 8, 2022 at 6:11 PM William Guo <gu...@apache.org> wrote: > >> hi All, >> >> Since our pmcs have reached an agreement, we will draft our feature >> roadmap pretty soon. >> >> @Willem Jiang <willem.ji...@gmail.com> >> We will keep you posted. >> >> Thanks, >> William >> >> On Mon, Aug 8, 2022 at 5:30 PM Willem Ning Jiang <ningji...@apache.org> >> wrote: >> >>> It looks like there is a new generation of Griffin. >>> @William would you wrap up a roadmap for it? >>> >>> Thanks, >>> >>> Willem >>> >>> On 2022/08/06 08:16:49 William Guo wrote: >>> > Any comments from other PMCs? >>> > >>> > We need a full discussion before we kick off our next milestone for >>> apache >>> > griffin. >>> > >>> > Thanks, >>> > William >>> > >>> > >>> > On Fri, Jul 29, 2022 at 9:55 PM john@apache <john...@apache.org> >>> wrote: >>> > >>> > > Could not agree more with Eugene's summary. >>> > > >>> > > Just had a quick meeting with William, talking his arch draft. Now we >>> > > have a clarified and clear defined goal for the new arch. >>> > > >>> > > William and I all agree that it would be great to have a simple and >>> > > clearly defined goal to go with. >>> > > >>> > > ---- >>> > > >>> > > # What Griffin should focus >>> > > >>> > > Griffin is a generic framework to enable user to measure and monitor >>> > > the data quality in an easy and extensive manner. >>> > > >>> > > # Old Griffin architecture problems >>> > > >>> > > What's the painpoints we are facing in the current edition of Griffin >>> > > from the architectural perspective? >>> > > >>> > > 1. To have an abstraction layer of query engine >>> > > The current spark implementation limited Griffin's adoption, since a >>> > > company could use other query engine in company level. It's too heavy >>> > > to set up a spark environment to run Griffin. >>> > > >>> > > 2. To support a common data quality workflow: measure-monitor-alert >>> > > In most of real scenarios, measuring is not the goal, but >>> > > measure-monitor-alert workflow. >>> > > User should be able to define an use case, including: >>> > > 1. to measure a dq metrics >>> > > 2. to define the monitoring trigger >>> > > 3. to define the alerting action >>> > > 4. to integrate the 3-steps into a single job/UoW (by scheduler) >>> > > >>> > > On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <liu...@apache.org> >>> wrote: >>> > > > >>> > > > Appreciate William post of new architecture. John & Grant had >>> raised good >>> > > > comments as well. >>> > > > >>> > > > As an evolving data quality product, we need to catch up to the >>> rolling >>> > > > forward of the big data ecosystem. I think there are points we can >>> > > > consider more. >>> > > > >>> > > > Firstly, how to absorb positive/negative experience from existent >>> jira >>> > > bugs >>> > > > and make it apply in new architecture. >>> > > > Secondly, how to highlight the improvement from the old one to the >>> new >>> > > > architecture and convince Grffin's users that the change will >>> bring more >>> > > > gains. >>> > > > Thirdly, we should consider making rule-base architecture evolving >>> as a >>> > > > long-term goal, finally becoming a de facto standard in the >>> vertical >>> > > area. >>> > > > >>> > > > Thx >>> > > > Eugene >>> > > > >>> > > > >>> > > > On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org> >>> wrote: >>> > > > >>> > > > > just created the GIP pages and moved this proposal under GIP, >>> > > > > >>> > > > > >>> > > > > >>> > > >>> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal >>> > > > > >>> > > > > Thanks, >>> > > > > William >>> > > > > >>> > > > > On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org> >>> wrote: >>> > > > > >>> > > > > > I have drafted next generation griffin architecture. >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > >>> > > >>> https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal >>> > > > > > >>> > > > > > Let's brainstorm it . >>> > > > > > >>> > > > > > Thanks, >>> > > > > > William >>> > > > > > >>> > > > > > >>> > > > > > On Sat, Jul 23, 2022 at 10:28 AM john@apache < >>> john...@apache.org> >>> > > wrote: >>> > > > > > >>> > > > > >> Hi all, >>> > > > > >> >>> > > > > >> Great to hear that we will have something new in the plan. >>> > > > > >> >>> > > > > >> It is so happy to me to see that there will be new arch draft >>> in 2 >>> > > days. >>> > > > > >> >>> > > > > >> At the same time, I would like to raise up some of my >>> ideas/comments >>> > > > > >> related to the new arch before we decided to go >>> > > > > >> >>> > > > > >> >>> > > > > >> 1. What’s the good part and bad part for the existing >>> arch? I >>> > > agree >>> > > > > >> that the current arch which were defined 4 years ago is not >>> good >>> > > > > >> enough for the ongoing requirement. So I would like to >>> recommend >>> > > that >>> > > > > >> compile a list of expected change from arch level, so we can >>> start >>> > > > > >> from there, to discuss the priority and scope. >>> > > > > >> 2. We might need to define/re-define high level principles >>> > > helping >>> > > > > >> us clarify the vision, if it is not clear enough now. We need >>> some >>> > > > > >> brain storming to collect more ideas about what griffin >>> should do >>> > > and >>> > > > > >> what should not. >>> > > > > >> 3. We need to check with what is the main pain points of >>> current >>> > > > > >> user, and try to solve them in the arch level. I will go >>> through all >>> > > > > >> the jira issues and classify them to several categories. And >>> I will >>> > > > > >> keep update the status. >>> > > > > >> >>> > > > > >> Please let me know your comments. >>> > > > > >> >>> > > > > >> On Sat, Jul 23, 2022 at 12:53 AM Grant <grant.xu...@gmail.com >>> > >>> > > wrote: >>> > > > > >> > >>> > > > > >> > Hi Willem, >>> > > > > >> > >>> > > > > >> > I am Grant Guo, PMC member of the Griffin project, apache >>> ID: >>> > > xuexu >>> > > > > >> > >>> > > > > >> > Since last December, William, me, and other PMC members >>> have been >>> > > > > >> > discussing the feasibility of the rule-based solution, >>> which could >>> > > > > bring >>> > > > > >> > Griffin to a new level in terms of usability, >>> extensibility, etc. >>> > > > > >> > >>> > > > > >> > A new architectural proposal will be posted soon. >>> > > > > >> > >>> > > > > >> > Thanks >>> > > > > >> > >>> > > > > >> > On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang < >>> > > willem.ji...@gmail.com> >>> > > > > >> wrote: >>> > > > > >> > >>> > > > > >> > > Hi, >>> > > > > >> > > >>> > > > > >> > > There has been no development activity on this project >>> for more >>> > > > > than 5 >>> > > > > >> > > months. The PMC has failed to submit several board >>> reports. >>> > > Without >>> > > > > a >>> > > > > >> > > community or anyone working on the project, there is no >>> project. >>> > > > > >> > > >>> > > > > >> > > If any of the PMC members are still active, please >>> indicate so >>> > > by >>> > > > > >> > > responding to this email. >>> > > > > >> > > >>> > > > > >> > > Thank you, all! I hope everyone is safe and healthy. >>> > > > > >> > > >>> > > > > >> > > Willem Jiang >>> > > > > >> > > >>> > > > > >> >>> > > > > > >>> > > > > >>> > > >>> > >>> >>
