Meeting minutes with Nielsen:
* Discuss griffin to support filters for metastore tables or navigation assistance for table selection on UI. * Griffin provides RESTful API for backend. * Discuss griffin to support multiple source or target tables. * Discuss more supporting file types, such as parquet. * In griffin, the partition field is optional, it just helps to provide the specific part of data, it will get all the data of a table without any partition information. * Config json file provides the parameters for griffin measure calculation, you can also submit a spark job with it directly. * Currently, griffin can only reuse measure, not rule. We’ll discuss about this, if we need to support reusing rules. * Sample ratio field in config file is optional, in batch mode we don’t need to configure it. * In griffin, mapping of columns are limited, discuss to support advanced features like joining between tables , or advanced sql script. * At current, the rule parser doesn’t support customized rules, griffin has the plan to support this. //TODO document it and send it to dev list * Griffin doesn’t support metrics alert function, it posts all the metrics to elasticsearch, es supports such feature. //TODO, write a solution for it based on elastic search * In griffin, you can’t modify the exist rules or measure at current. Thanks, William ________________________________ From: William GUO <guo...@outlook.com> on behalf of William Guo <gu...@apache.org> Sent: Wednesday, August 2, 2017 10:02:15 AM To: Mara Preotescu Cc: dev@griffin.incubator.apache.org; Ananthanarayanan Ms; Kunduru, Abishek Subject: Re: Griffin support & roadmap hi mara, Are you join? Thanks, William ________________________________ From: Mara Preotescu <mara.preote...@nielsen.com> Sent: Monday, July 31, 2017 11:22:00 PM To: William Guo Cc: dev@griffin.incubator.apache.org; Ananthanarayanan Ms; Kunduru, Abishek Subject: Re: Griffin support & roadmap Hi William, Would 10:00 am CST (Beijing) work for you on Wednesday 08/02? Thanks, Mara On Sun, Jul 30, 2017 at 10:59 PM, William Guo <gu...@apache.org<mailto:gu...@apache.org>> wrote: hi Mara, We are in China, it is hard to arrange a meeting for US, CHINA, INDIA together. China day time is fine for me. Thanks, William ________________________________ From: Mara Preotescu <mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>> Sent: Monday, July 31, 2017 10:54:25 AM To: William Guo Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org> Subject: Re: Griffin support & roadmap Hi William, Either Wednesday or Thursday will work for us. Any better time working for you? What time zone are you in? I am in US ET time, a colleague of mine who I would like to join our discussion is in India, Chennai. Thanks, Mara On Sun, Jul 30, 2017 at 7:34 PM, William Guo <gu...@apache.org<mailto:gu...@apache.org>> wrote: hi Mara, Sure, We could schedule a meeting to discuss background, requirements, status and milestone. We should be fine in Wednesday or Thursday, what is your proposal? Thanks, William ________________________________ From: Mara Preotescu <mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>> Sent: Friday, July 28, 2017 7:57:45 PM To: William Guo Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org> Subject: Re: Griffin support & roadmap HI Alex, William, THANK YOU so much for your responses. Thank you for the links. And, I hope you don't mind if I'll take up your offer to contact you if needed. We are considering, here at Nielsen, using Griffin for our new Data Quality framework ... we know the project is still in the incubator but we would like give it a try and even contributing, if needed. We already install it and ran a few tests. If your time permits I would like scheduling a quick call so we could understand the current status and, most importantly if the roadmap stays as in the published documents. Thanks again, Mara On Fri, Jul 28, 2017 at 4:21 AM, William Guo <gu...@apache.org<mailto:gu...@apache.org>> wrote: hi Mara, Few links might help, you can contact us by dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org> or my personal account gu...@apache.org<mailto:gu...@apache.org> GitHub : https://github.com/apache/incubator-griffin<https://github.com/eBay/griffin> Website : https://griffin.incubator.apache.org<https://griffin.incubator.apache.org/> Contact: mailto://subscribe-...@griffin.incubator.apache.org<mailto:subscribe-...@griffin.incubator.apache.org> Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/Griffin Thanks, William ________________________________ From: Lv, Alex <lzhix...@ebay.com<mailto:lzhix...@ebay.com>> Sent: Friday, July 28, 2017 9:10:09 AM To: Mara Preotescu; Guo, William; gu...@apache.org<mailto:gu...@apache.org> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org> Subject: RE: Griffin support & roadmap <<Move Amber to BCC>> Hi Mara, Glad to hear from you, you may discuss the details with William. Thx. Best regards, Alex Lv From: Mara Preotescu [mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>] Sent: 2017年7月28日 6:14 To: Lv, Alex <lzhix...@ebay.com<mailto:lzhix...@ebay.com>>; Vaidya, Amber <amvai...@ebay.com<mailto:amvai...@ebay.com>> Subject: Griffin support & roadmap Hello Alex, Amber, I am writing you trying to reach the support for Griffin, both support e-mails for the product returned as invalid addresses (subscribe-...@griffin.incubator.apache.org<mailto:subscribe-...@griffin.incubator.apache.org>, ebay-griffin-d...@googlegroups.com<mailto:ebay-griffin-d...@googlegroups.com>). Could you please let me know who should we contact to discuss about Griffin's roadmap? We are looking, here at Nielsen, to use the Griffin framework for our DQ processes. As of today we learned, and tested, the only dimension available, Accuracy. Would you be able to share the roadmap for any other DQ dimensions availability? We are looking as well to add a few custom validations - does the tool offer any APIs that can be used for this purpose? Any information you could provide would be very, very helpful. Thank you in advance for your help and time. Mara Preotescu VP Technology, DevOps Nielsen
