Could not agree more with Eugene's summary.
Just had a quick meeting with William, talking his arch draft. Now we
have a clarified and clear defined goal for the new arch.
William and I all agree that it would be great to have a simple and
clearly defined goal to go with.
----
# What Griffin should focus
Griffin is a generic framework to enable user to measure and monitor
the data quality in an easy and extensive manner.
# Old Griffin architecture problems
What's the painpoints we are facing in the current edition of Griffin
from the architectural perspective?
1. To have an abstraction layer of query engine
The current spark implementation limited Griffin's adoption, since a
company could use other query engine in company level. It's too heavy
to set up a spark environment to run Griffin.
2. To support a common data quality workflow: measure-monitor-alert
In most of real scenarios, measuring is not the goal, but
measure-monitor-alert workflow.
User should be able to define an use case, including:
1. to measure a dq metrics
2. to define the monitoring trigger
3. to define the alerting action
4. to integrate the 3-steps into a single job/UoW (by scheduler)
On Sun, Jul 24, 2022 at 4:50 PM Eugene Law <liu...@apache.org> wrote:
>
> Appreciate William post of new architecture. John & Grant had raised good
> comments as well.
>
> As an evolving data quality product, we need to catch up to the rolling
> forward of the big data ecosystem. I think there are points we can
> consider more.
>
> Firstly, how to absorb positive/negative experience from existent jira bugs
> and make it apply in new architecture.
> Secondly, how to highlight the improvement from the old one to the new
> architecture and convince Grffin's users that the change will bring more
> gains.
> Thirdly, we should consider making rule-base architecture evolving as a
> long-term goal, finally becoming a de facto standard in the vertical area.
>
> Thx
> Eugene
>
>
> On Sun, Jul 24, 2022 at 4:00 PM William Guo <gu...@apache.org> wrote:
>
> > just created the GIP pages and moved this proposal under GIP,
> >
> >
> > https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> >
> > Thanks,
> > William
> >
> > On Sat, Jul 23, 2022 at 3:11 PM William Guo <gu...@apache.org> wrote:
> >
> > > I have drafted next generation griffin architecture.
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/GRIFFIN/Rule+Based+Architecture+Proposal
> > >
> > > Let's brainstorm it .
> > >
> > > Thanks,
> > > William
> > >
> > >
> > > On Sat, Jul 23, 2022 at 10:28 AM john@apache <john...@apache.org> wrote:
> > >
> > >> Hi all,
> > >>
> > >> Great to hear that we will have something new in the plan.
> > >>
> > >> It is so happy to me to see that there will be new arch draft in 2 days.
> > >>
> > >> At the same time, I would like to raise up some of my ideas/comments
> > >> related to the new arch before we decided to go
> > >>
> > >>
> > >> 1. What’s the good part and bad part for the existing arch? I agree
> > >> that the current arch which were defined 4 years ago is not good
> > >> enough for the ongoing requirement. So I would like to recommend that
> > >> compile a list of expected change from arch level, so we can start
> > >> from there, to discuss the priority and scope.
> > >> 2. We might need to define/re-define high level principles helping
> > >> us clarify the vision, if it is not clear enough now. We need some
> > >> brain storming to collect more ideas about what griffin should do and
> > >> what should not.
> > >> 3. We need to check with what is the main pain points of current
> > >> user, and try to solve them in the arch level. I will go through all
> > >> the jira issues and classify them to several categories. And I will
> > >> keep update the status.
> > >>
> > >> Please let me know your comments.
> > >>
> > >> On Sat, Jul 23, 2022 at 12:53 AM Grant <grant.xu...@gmail.com> wrote:
> > >> >
> > >> > Hi Willem,
> > >> >
> > >> > I am Grant Guo, PMC member of the Griffin project, apache ID: xuexu
> > >> >
> > >> > Since last December, William, me, and other PMC members have been
> > >> > discussing the feasibility of the rule-based solution, which could
> > bring
> > >> > Griffin to a new level in terms of usability, extensibility, etc.
> > >> >
> > >> > A new architectural proposal will be posted soon.
> > >> >
> > >> > Thanks
> > >> >
> > >> > On Wed, Jul 20, 2022 at 6:08 PM Willem Jiang <willem.ji...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > There has been no development activity on this project for more
> > than 5
> > >> > > months. The PMC has failed to submit several board reports. Without
> > a
> > >> > > community or anyone working on the project, there is no project.
> > >> > >
> > >> > > If any of the PMC members are still active, please indicate so by
> > >> > > responding to this email.
> > >> > >
> > >> > > Thank you, all! I hope everyone is safe and healthy.
> > >> > >
> > >> > > Willem Jiang
> > >> > >
> > >>
> > >
> >
