Re: [DISCUSS] Apache Amoro proposal
+1. I'm happy to be one of the mentors. I have discussed with Jinsong, Nathan and the team, and am impressed by their openness and passion on improving the Amoro community through incubation. From my observation, it's a well developed community with a similar governance philosophy as the Apache Way. I believe joining Apache incubator could help the community to be more vibrant and diverse. And personally, I think having a dedicated project with the purpose to better manage Lakehouse is necessary. IMHO, compared to the traditional DBMS solutions, the existing open source Lakehouse solutions focus more on the "DB" part and lack efforts on the "MS" part. Best Regards, Yu On Fri, 23 Feb 2024 at 22:54, 周劲松 wrote: > > Hi JB， > > Yes, you can say it is an abstraction layer on top of data lake table > formats and query engines and we often call it the service layer in > Lakehouse architecture. The service layer primarily provides unified > metadata and access control, as well as common audit services, and so on. > Of course, Amoro is currently focusing on automatic optimizing, helping > users to more easily use the data lake and achieve the desired analytical > performance on it. Amoro can work with other software in the service layer > and can also extend plugins to integrate more capabilities. > > On Fri, Feb 23, 2024 at 10:18 PM Jean-Baptiste Onofré > wrote: > > > Hi Justin > > > > Even if it looks interesting, I'm not sure to understand exactly the > > purpose of the proposal. > > > > What lakehouse management system means exactly ? Is it an abstraction > > layer on top of Iceberg, Paimon + query engine powered by Flink, > > Spark, Trino ? > > > > Please let me know if you want an additional mentor, I would be happy to > > help. > > > > Thanks ! > > Regards > > JB > > > > On Fri, Feb 23, 2024 at 9:44 AM Justin Mclean > > wrote: > > > > > > Hi, > > > > > > I would like to propose a new project to the ASF incubator - Apache > > Amoro. I’m one of the mentors, but there are a lot of other people involved > > who have done all of the hard work. > > > > > > Amoro is a Lakehouse management system built on open data lake formats > > like Apache Iceberg and Apache Paimon (Incubating). Working with compute > > engines including Apache Flink, Apache Spark, and Trino, Amoro brings > > pluggable and self-managed features for Lakehouse to provide out-of-the-box > > data warehouse experience, and helps data platforms or products easily > > build infra-decoupled, stream-and-batch-fused and lake-native architecture. > > You can find the proposal here. [1] > > > > > > We are looking forward to anyone's feedback or questions. > > > > > > Thanks, > > > Justin > > > > > > [1] https://cwiki.apache.org/confluence/display/INCUBATOR/AmoroProposal > > > - > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > - > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [NOTICE] Incubation Report for February 2024
HI, The report_runbook.py script tells you most of what to do. A few more projects need t report than listed below, including some new projects that need to report monthly. I’ve also left the monthly properties as is. The report_email.py script will send an email to this list with the important dates, and who needs to report, the report_reminders.py script will send each podling a reminder email. There are some instructions here[1], but they could be improved. Kind Regards, Justin 1. https://incubator.apache.org/guides/chair.html > On 24 Feb 2024, at 1:44 pm, tison wrote: > > Hi, > > I'm trying to create an Incubation Report page for February 2024 at > [1], including the following podlings according to the report group > info that they should report in this month: > > * answer > * fury > * horaedb > * streampark > > But it still lacks information that I need some help: > > 1. What is the desired timeline and shepherd assignments? > 2. Missing IPMC level report. Perhaps the only way is going through > the mailing list? > 3. Missing other projects need a report. I guess we have a script to > list out from the podling.xml file, but I don't find it. I may write > it when I have some spare time if there is no one. I don't promise :P > > Best, > tison. > > [1] https://cwiki.apache.org/confluence/display/INCUBATOR/February2024 > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [MENTORS] March report timeline - reports due March 2023
Please ignore this as I put in the wrong year > On 25 Feb 2024, at 12:12 pm, jmcl...@apache.org wrote: > > Dates for next board report: > Wed March 01 - Podling reports due by end of day > Sun March 05 - Shepherd reviews due by end of day > Sun March 05 - Summary due by end of day > Tue March 07 - Mentor signoff due by end of day > Wed March 08 - Report submitted to Board > Wed March 15 - Board meeting > > > Expected to report this month are: > - baremaps > - kie > - marvin-ai > - nemo > - nlpcraft > - opendal > - pagespeed > - spot > - toree > - training > - uniffle > - wayang > > > Reports as usual can be submitted here [1] > > Thanks, > Justin > > > 1. https://s.apache.org/incubator-report > 1. https://cwiki.apache.org/confluence/display/INCUBATOR/Report > 1. https://cwiki.apache.org/confluence/display/INCUBATOR/March2023 > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[MENTORS] March report timeline - reports due March 2024
Dates for next board report: Wed March 06 - Podling reports due by end of day Sun March 10 - Shepherd reviews due by end of day Sun March 10 - Summary due by end of day Tue March 12 - Mentor signoff due by end of day Wed March 13 - Report submitted to Board Wed March 20 - Board meeting Expected to report this month are: - answer - baremaps - fury - gluten - horaedb - kie - nemo - resilientdb - uniffle - wayang - xtable Reports as usual can be submitted here [1] Thanks, Justin 1. https://s.apache.org/incubator-report 1. https://cwiki.apache.org/confluence/display/INCUBATOR/Report 1. https://cwiki.apache.org/confluence/display/INCUBATOR/March2024 - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[MENTORS] March report timeline - reports due March 2023
Dates for next board report: Wed March 01 - Podling reports due by end of day Sun March 05 - Shepherd reviews due by end of day Sun March 05 - Summary due by end of day Tue March 07 - Mentor signoff due by end of day Wed March 08 - Report submitted to Board Wed March 15 - Board meeting Expected to report this month are: - baremaps - kie - marvin-ai - nemo - nlpcraft - opendal - pagespeed - spot - toree - training - uniffle - wayang Reports as usual can be submitted here [1] Thanks, Justin 1. https://s.apache.org/incubator-report 1. https://cwiki.apache.org/confluence/display/INCUBATOR/Report 1. https://cwiki.apache.org/confluence/display/INCUBATOR/March2023 - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
RE: Re: [DISCUSS] Apache Amoro proposal
hi, JB As co-creator of this project, I’d love to explain more about the positioning of lakehouse management system. When discussing databases or traditional data warehouses, we often used the term DBMS (Database Management System) to describe them. Traditional databases, including MPP databases, are typically considered “out-of-box” solutions. Unlike big data systems, they don’t require various components like compute engines, data lake formats, or metadata stores. When we need a database management tool, lightweight options like Navicat are commonly used. If we further abstract the capabilities of a DBMS and map them to the modern data stack, we find that the data read/write part of a DBMS is now shared among different compute engines such as Spark, Flink, Trino, and cloud-native services like Athena. Another part of a DBMS deals with data files, index files, and metadata (also known as the information schema) maintenance. Currently, there are successful open-source and commercial projects dedicated to managing metadata, such as HiveMetastore, UnityCatalog, and more recently, Gravitino. In practice, developers often combine these projects with compute engines to optimize data files. For example, many commercial compute engines include an optimize command. Amoro, as a lakehouse management system, aims to encapsulate the maintenance and management of data lake files, index files, and metadata in a way that is transparent and easy-to-use for users. The richness of diverse computing engines is a distinctive feature of the modern data stack, opening up a multitude of possibilities for various application scenarios. Additionally, concerning the part analogous to DBMS, we aspire to have a mature system in place—one that seamlessly accommodates data written to the lakehouse by any engine, in any manner, ensuring high data availability across all other engines. For instance, when Flink writes to Iceberg, Amoro’s self-optimizing mechanism ensures efficient data analysis performance by Trino or other engines while controlling compacting costs. Additionally, Amoro handles historical data, snapshots, and orphan file cleanup in the background. By positioning Amoro in this way, we aim to provide an ‘out-of-box’ experience that feels as straightforward as traditional DBMS while keeping openness to various computing engines. At the same time, Amoro hopes to empower data product builders with a lightweight solution that integrates seamlessly into their modern data workflows. Thanks. On 2024/02/23 14:16:43 Jean-Baptiste Onofré wrote: > Hi Justin > > Even if it looks interesting, I'm not sure to understand exactly the > purpose of the proposal. > > What lakehouse management system means exactly ? Is it an abstraction > layer on top of Iceberg, Paimon + query engine powered by Flink, > Spark, Trino ? > > Please let me know if you want an additional mentor, I would be happy to help. > > Thanks ! > Regards > JB > > On Fri, Feb 23, 2024 at 9:44 AM Justin Mclean wrote: > > > > Hi, > > > > I would like to propose a new project to the ASF incubator - Apache Amoro. I’m one of the mentors, but there are a lot of other people involved who have done all of the hard work. > > > > Amoro is a Lakehouse management system built on open data lake formats like Apache Iceberg and Apache Paimon (Incubating). Working with compute engines including Apache Flink, Apache Spark, and Trino, Amoro brings pluggable and self-managed features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture. You can find the proposal here. [1] > > > > We are looking forward to anyone's feedback or questions. > > > > Thanks, > > Justin > > > > [1] https://cwiki.apache.org/confluence/display/INCUBATOR/AmoroProposal > > - > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
RE: Re: [DISCUSS] Apache Amoro proposal
+1. Looks like a good candidate with a good number of contributors already. On 2024/02/24 05:24:33 nathan ma wrote: > hi, JB > > As co-creator of this project, I’d love to explain more about the > positioning of lakehouse management system. > > When discussing databases or traditional data warehouses, we often used the > term DBMS (Database Management System) to describe them. Traditional > databases, including MPP databases, are typically considered “out-of-box” > solutions. Unlike big data systems, they don’t require various components > like compute engines, data lake formats, or metadata stores. When we need a > database management tool, lightweight options like Navicat are commonly > used. > > If we further abstract the capabilities of a DBMS and map them to the > modern data stack, we find that the data read/write part of a DBMS is now > shared among different compute engines such as Spark, Flink, Trino, and > cloud-native services like Athena. Another part of a DBMS deals with data > files, index files, and metadata (also known as the information schema) > maintenance. Currently, there are successful open-source and commercial > projects dedicated to managing metadata, such as HiveMetastore, > UnityCatalog, and more recently, Gravitino. In practice, developers often > combine these projects with compute engines to optimize data files. For > example, many commercial compute engines include an optimize command. > > Amoro, as a lakehouse management system, aims to encapsulate the > maintenance and management of data lake files, index files, and metadata in > a way that is transparent and easy-to-use for users. The richness of > diverse computing engines is a distinctive feature of the modern data > stack, opening up a multitude of possibilities for various application > scenarios. Additionally, concerning the part analogous to DBMS, we aspire > to have a mature system in place—one that seamlessly accommodates data > written to the lakehouse by any engine, in any manner, ensuring high data > availability across all other engines. For instance, when Flink writes to > Iceberg, Amoro’s self-optimizing mechanism ensures efficient data analysis > performance by Trino or other engines while controlling compacting costs. > Additionally, Amoro handles historical data, snapshots, and orphan file > cleanup in the background. > > By positioning Amoro in this way, we aim to provide an ‘out-of-box’ > experience that feels as straightforward as traditional DBMS while keeping > openness to various computing engines. At the same time, Amoro hopes to > empower data product builders with a lightweight solution that integrates > seamlessly into their modern data workflows. > > > > Thanks. > Jin Ma > > > On 2024/02/23 14:16:43 Jean-Baptiste Onofré wrote: > > Hi Justin > > > > Even if it looks interesting, I'm not sure to understand exactly the > > purpose of the proposal. > > > > What lakehouse management system means exactly ? Is it an abstraction > > layer on top of Iceberg, Paimon + query engine powered by Flink, > > Spark, Trino ? > > > > Please let me know if you want an additional mentor, I would be happy to > help. > > > > Thanks ! > > Regards > > JB > > > > On Fri, Feb 23, 2024 at 9:44 AM Justin Mclean > wrote: > > > > > > Hi, > > > > > > I would like to propose a new project to the ASF incubator - Apache > Amoro. I’m one of the mentors, but there are a lot of other people involved > who have done all of the hard work. > > > > > > Amoro is a Lakehouse management system built on open data lake formats > like Apache Iceberg and Apache Paimon (Incubating). Working with compute > engines including Apache Flink, Apache Spark, and Trino, Amoro brings > pluggable and self-managed features for Lakehouse to provide out-of-the-box > data warehouse experience, and helps data platforms or products easily > build infra-decoupled, stream-and-batch-fused and lake-native architecture. > You can find the proposal here. [1] > > > > > > We are looking forward to anyone's feedback or questions. > > > > > > Thanks, > > > Justin > > > > > > [1] https://cwiki.apache.org/confluence/display/INCUBATOR/AmoroProposal > > > - > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > - > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [NOTICE] Incubation Report for February 2024
> On Feb 24, 2024, at 9:35 AM, Dave Fisher wrote: > > > >> On Feb 23, 2024, at 6:44 PM, tison wrote: >> >> Hi, >> >> I'm trying to create an Incubation Report page for February 2024 at >> [1], including the following podlings according to the report group >> info that they should report in this month: >> >> * answer >> * fury >> * horaedb >> * streampark >> >> But it still lacks information that I need some help: >> >> 1. What is the desired timeline and shepherd assignments? >> 2. Missing IPMC level report. Perhaps the only way is going through >> the mailing list? >> 3. Missing other projects need a report. I guess we have a script to >> list out from the podling.xml file, but I don't find it. I may write >> it when I have some spare time if there is no one. I don't promise :P > > % svn co https://svn.apache.org/repos/asf/incubator/public/trunk > % python3 report_runbook.py > usage: report_runbook.py [-h] [--apache-id APACHE_ID] --month MONTH > report_runbook.py: error: the following arguments are required: —mestonth Since it is just after the February report then we are preparing March. These steps are needed first to catch up for missed months: % edit content/podlings.xml if needed % python3 clutch2.py % python3 assign_shepherds.py --year=2024 --month=3 % python3 clutch2report.py % svn add reports/report202403.txt % svn ci -m ‘Prepare for March 2024 Report' % cat reports/report202403.txt | pbcopy Now you can create the wiki page. If you have `pbcopy` then paste that into the cwiki. Please note that these runbook steps are not in a precise order. Read the whole list and proceed logically. Also, Justin has added some report_*.py scripts that may be shortcuts or additions to consider. % python3 report_runbook.py --apache-id wave --month 3 ## # This doc is meant to be a guide to preparing the incubator # report for a given month. Each section has a heading, starting # with mechanical, editorial, or Chair. If you have volunteered # to perform any of these actions, please pay attention to # those sections. ## ## # mechanical -- Send report timeline email. # # On the last Wednesday of the month, send an email to # general@incubator announcing the report timeline, using the # following content. ## March 2024 Incubator report timeline: https://cwiki.apache.org/confluence/display/INCUBATOR/March2024 Wed March 06 -- Podling reports due by end of day Sun March 10 -- Shepherd reviews due by end of day Sun March 10 -- Summary due by end of day Tue March 12 -- Mentor signoff due by end of day Wed March 13 -- Report submitted to Board Wed March 20 -- Board meeting ## # mechanical -- Send report reminders email. # # Several times between the end of the prior month and beginning # of the reporting month, send a reminder email to all reporting # podlings ## # Reminders script (no more marvin tool, follow the prompts) ./report_reminders.py ## # editorial -- Create podling summary. # # After the podling reporting deadline has passed, group # podlings into the following categories (additional categories # may be added if appropriate): # # * Still getting started at the Incubator # * Not yet ready to graduate # * No release # * Community growth # * Ready to graduate # * Did not report, expected next month # ## # For any podlings that did not report, add a "monthly" attribute # in podlings.xml. See https://s.apache.org/At0 for a sample commit. # mechanical -- Assign podlings which do not report to "monthly". [... edit podlings.xml with your editor of choice ...] svn ci -m "Assign podlings which did not report to 'monthly'." ## # editorial -- Write narrative, misc, legal, infrastructure, etc. #sections. # # After the podlings are compiled, please add the various sections # listed above. To do this, visit the incubator mail archives for # the preceeding month and review any actions performed. Were # there any code donations? Podlings renamed? Unexpected outages? ## ## # editorial -- Assemble list of releases. # # The Board report must contain a list of releases by the # Incubator project during the reporting period. This can be # assembled manually from the history of our dist area. # # The date of each release is the date it appeared in our
Re: [NOTICE] Incubation Report for February 2024
> On Feb 23, 2024, at 6:44 PM, tison wrote: > > Hi, > > I'm trying to create an Incubation Report page for February 2024 at > [1], including the following podlings according to the report group > info that they should report in this month: > > * answer > * fury > * horaedb > * streampark > > But it still lacks information that I need some help: > > 1. What is the desired timeline and shepherd assignments? > 2. Missing IPMC level report. Perhaps the only way is going through > the mailing list? > 3. Missing other projects need a report. I guess we have a script to > list out from the podling.xml file, but I don't find it. I may write > it when I have some spare time if there is no one. I don't promise :P % svn co https://svn.apache.org/repos/asf/incubator/public/trunk % python3 report_runbook.py usage: report_runbook.py [-h] [--apache-id APACHE_ID] --month MONTH report_runbook.py: error: the following arguments are required: —mestonth Best, Dave > > Best, > tison. > > [1] https://cwiki.apache.org/confluence/display/INCUBATOR/February2024 > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Graduate Apache SDAP (Incubating) as a Top Level Project
On Sat, 24 Feb 2024 at 04:03, Riley Kuttruff wrote: > > Thank you for finding those issues. I've updated the site (sdap.a.o shows the > changes but sdap.i.a.o still hasn't updated at the time I'm writing this). I > believe I've the issues you've found. Please let me know if this is not the > case. The issues I raised have been fixed, thanks. > On 2024/02/23 15:59:07 sebb wrote: > > The Downloads page has some issues: > > https://sdap.apache.org/downloads > > > > No link to KEYS file > > Links for older releases are broken > > Copyright page is 2023 > > > > On Fri, 23 Feb 2024 at 14:30, PJ Fanning wrote: > > > > > > +1 (binding) > > > > > > I had a look at the mailing lists and the community seems in a pretty > > > good state. > > > > > > As a matter of interest, are you still looking at completing the v1.2.0 > > > release [1]? If so, I could have a look at the RC over the weekend. > > > > > > [1] https://lists.apache.org/thread/vr4zf6zhg2yp41bjwvlpm1mp2nrycqcw > > > > > > On 2024/02/22 18:01:31 Riley Kuttruff wrote: > > > > Hi all, > > > > > > > > Apache SDAP joined Incubator in October 2017. In the time since, we've > > > > made significant progress towards maturing our community and our > > > > project and adopting the Apache Way. > > > > > > > > After community discussion [1][2][3], the community has voted [4] that > > > > we > > > > would like to proceed with graduation [5]. We now call upon the > > > > Incubator > > > > PMC to review and discuss our progress and would appreciate any and all > > > > feedback towards graduation. > > > > > > > > Below are some facts and project highlights from the incubation phase as > > > > well as the draft resolution: > > > > > > > > - Our community consists of 21 committers, with 2 being mentors and > > > > the remaining 19 serving as our PPMC > > > > - Several pending and planned invites to bring on new committers and/or > > > > PPMC members from additional organizations > > > > - Completed 2 releases with 2 release managers - with a 3rd release run > > > > by > > > > a 3rd release manager in progress > > > > - Our software is currently being utilized by organizations such as NASA > > > > Jet Propulsion Laboratory, NSF National Center for Atmospheric Research, > > > > Florida State University, and George Mason University in support of > > > > projects > > > > such as the NASA Sea Level Change Portal, Estimating the Circulation and > > > > Climate of the Ocean (ECCO) project, GRACE/GRACE-FO, Cloud-based > > > > Data Match-Up Service, Integrated Digital Earth Analysis System (IDEAS), > > > > and many others. > > > > - Opened 400+ PRs across 3 main code repositories, 350+ of which are > > > > merged or closed (some are pending our next release) > > > > - Maturity model self assessment [6] > > > > > > > > We have resolved all branding issues we are aware of: logo, GitHub, > > > > Website, etc > > > > > > > > We’d like to also extend a sincere thank you to our mentors, current and > > > > former for their invaluable insight and assistance with getting us to > > > > this > > > > point. > > > > > > > > Thank you, Julian, Jörn, Trevor, Lewis, Suneel, and Raphael! > > > > > > > > --- > > > > > > > > Establish the Apache SDAP Project > > > > > > > > WHEREAS, the Board of Directors deems it to be in the best interests of > > > > the Foundation and consistent with the Foundation's purpose to establish > > > > a Project Management Committee charged with the creation and maintenance > > > > of open-source software, for distribution at no charge to the public, > > > > related to an integrated data analytic center for Big Science problems. > > > > > > > > NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee > > > > (PMC), to be known as the "Apache SDAP Project", be and hereby is > > > > established pursuant to Bylaws of the Foundation; and be it further > > > > > > > > RESOLVED, that the Apache SDAP Project be and hereby is responsible > > > > for the creation and maintenance of software related to an integrated > > > > data > > > > analytic center for Big Science problems; and be it further > > > > > > > > RESOLVED, that the office of "Vice President, Apache SDAP" be and > > > > hereby is created, the person holding such office to serve at the > > > > direction of the Board of Directors as the chair of the Apache SDAP > > > > Project, and to have primary responsibility for management of the > > > > projects within the scope of responsibility of the Apache SDAP > > > > Project; and be it further > > > > > > > > RESOLVED, that the persons listed immediately below be and hereby are > > > > appointed to serve as the initial members of the Apache SDAP Project: > > > > > > > > - Edward M Armstrong > > > > - Nga Thien Chung > > > > - Thomas Cram > > > > - Frank Greguska > > > > - Thomas Huang > > > > - Julian Hyde > > > > - Joseph C. Jacob > > > > - Jason Kang > > > > - Riley Kuttruff
