+1 ( binding ) Interesting project. Please add me as a mentor to the project.
On Tue, Sep 8, 2020 at 3:26 PM Matt Casters <matt.cast...@neotechnology.com.invalid> wrote: > Hello Apache, > > Our community is eager to propose for Hop to join the Apache Incubator. > The Hop Orchestration Platform aims to help people with complex data and > metadata orchestration problems. > > Below is the complete text of the proposal but you can also find it here: > https://cwiki.apache.org/confluence/display/INCUBATOR/HopProposal > > Any help with respect to the incubation is appreciated including help from > a few more mentors to set us on the right track. On behalf of my community > I'd be happy to answer any questions you might have regarding Hop. Our > thanks go out to Max, Julian and Tom for helping us set up this proposal. > > Thanks in advance for your time! > > Best regards, > > Matt - Hop co-founder > www.project-hop.org > --- > > Abstract > ========= > Hop is short for the Hop Orchestration Platform. Written completely in Java > it aims to provide a wide range of data orchestration tools, including a > visual development environment, servers, metadata analysis, auditing > services and so on. As a platform Hop also wants to be a re-usable library > so that it can be easily re-used by other software. > > Proposal > ========= > Hop provides all the tools to build, maintain and deploy data > orchestration, ETL and data integration solutions. For example, Hop allows > you to diagram a data flow that propagates changes from a database via > Apache Kafka to a data warehouse and deploy it as an Apache Beam pipeline. > The core concepts of Hop are Pipelines and Workflows. > * Pipelines do the core data manipulation work (read, manipulate, write > data). The main items of work in pipelines are transforms. A pipeline > consists of two or more (usually many) transforms that each perform a > granular piece of work. The transforms in a pipeline run in parallel, and > together create a powerful data processing tool. > * Workflows take care of the orchestration of actions: execute pipelines, > run child workflows, environment checks, preparation, problem alerting and > so on. > If these terms sound familiar it’s because they are taken from the Apache > Beam and Apache Airflow projects. > > > The main components of the Hop platform are: > * hop-gui, a visual data orchestration IDE > * hop-run: a CLI tool to run workflows or pipelines > * hop-config: a CLI tool to configure Hop and its components > * hop-server: a light-weight web server to run and monitor workflows and > pipelines > * hop-translator: a tool for translating the various parts of the Hop tools > (i18n). > * hop-web: a thin client version of hop-gui for web browsers and mobile > devices > > > The cornerstone of the Hop platform is extensibility: all major components > of the platform are designed to be pluggable. This allows any possible > missing functionality to be created in a short amount of time. > > Background > =========== > The Hop Orchestration Platform has its origins in the Kettle community. > Kettle got acquired by Pentaho and after Pentaho’s acquisition by Hitachi > in 2015, the community struck out to solve problems less aligned with > Hitachi’s interests. > > Rationale > ========== > In the Hop community, we have always aimed to function as a meritocracy, > where contributions are accepted based on merit, and individuals gain > status in the community based on their contributions (coding and > otherwise). We’re proud to have a diverse group of people doing all the > required things in a project: development , documentation, tutorials, > architecture, testing, graphics design and much more. Bringing the project > under the Apache Software Foundation would allow us to continue and grow, > but also give our users confidence about the governance, IP status, and > future of the project. > > ASF Preparation Phase > ====================== > The very first goal of project Hop is to find a good way to cooperate on > the development across wide geographical, economical and social spectra. To > make this possible real changes were needed to a codebase which is > essentially 20 years old. Most of these changes have been tackled by now. > We think it’s fair to say that by now, Hop is a new platform even though it > shares a common background as it partly started from the Kettle code base. > Here are a few of the key focus areas we’re trying to saveguard going > forward: > * Plugins: lightweight plugins for all major functionality. This makes it > possible to extend Hop or reduce Hop in size. It also allows people to > implement or change functionality with minimal coding. In other words it > makes it easier to contribute. > * Maintain an open and responsive community where every concern, feedback > and contribution is welcome. > * Maintain a clear focus on data orchestration user requirements, not on > “industry trends” > * Documentation: we set up a version controlled “adoc” system with > automated builds which is both open, controlled and reviewed. This is > incredibly important for every Hop user and developer. > * Testing and stability: we want to massively increase stability by > implementing integration tests beyond the standard Java unit testing > because of the dynamic nature of data orchestration work. We still have a > long way to go. This work will never be finished. It’s a clear and > important goal nevertheless. > * Simplicity: things are complex enough. We follow the example of projects > like Apache Spark and Flink and so as an example “hop-run.sh” does exactly > what the name says without the need to dive into documentation. As much as > possible we make things self-evident and will re-use existing terminology. > > > For a list of the changes you can look at the monthly roundup which was > compiled since February 2020. It documents to hard work of our community > so far: > > > http://www.project-hop.org/news/roundup-2020-02/ > http://www.project-hop.org/news/roundup-2020-03/ > http://www.project-hop.org/news/roundup-2020-04/ > http://www.project-hop.org/news/roundup-2020-05/ > http://www.project-hop.org/news/roundup-2020-06/ > http://www.project-hop.org/news/roundup-2020-08/ > > > Goals > ====== > Here are a few more details and specifics of things we still want to take > on going forward: > * Add more plugin metadata to Transforms and Action plugins as well as > their supported engines. This will make it easier to refine the user > interface and make the user experience better by giving to the point > feedback on what operations are supported and required. Example metadata > to add: extra version and build information, dependencies, tags and labels > (replacing categories), documentation links, input and output capabilities, > engine capabilities and so on. > * SWT: While the Eclipse SWT project is still supported we want to make a > list of all the commonly used API calls and stick to those with our own > API. This will help the development of hop-web and allow us to possibly > more easily migrate to different user interfaces later on. > * Integration testing: every transform and action should have an > integration test before it is released to ensure quality. Java unit > testing has been proven to be insufficient in guarding against backward > compatibility, stability and functionality. We need to do better. > * Apache VFS: Hop makes extensive use of this API to handle files. As such > we want to implement the various drivers for gs://, hdfs://, s3:// through > standard Kettle plugins making it easier to choose which protocols to > support. > * Variables & Parameters: make this experience more intuitive, clean up > the underlying API and add more options to the various user interfaces > responsible for setting and passing variables and parameters. > * Make Hop-Web an integral part of the Apache Hop project removing the code > duplication (fork) we’re dealing with now. This includes the need to > improve various user interfaces which were designed for non-web clients. > * Make best practices and governance functionality an integral part of the > API of the project: > * Data sets and unit testing (already done) > * Environments and lifecycle management (partly done) > * Git support (partly done) > * Auditing and lineage > * Software policies and enforcement thereof > * Configuration management (partly done) > > > Current Status > =============== > > Meritocracy > ------------ > With Project Hop, we actively work to foster the existing community and > encourage community contributions. As of September 1st 2020 we received > over 250 pull requests and have around 600 tickets in our JIRA platform (a > lot of which were created by community members) and have active discussions > in our Mattermost chat platform with over 80 members. > > > The last half year we started to ask users on our chat chat server for > specific feedback on terminology, features and so on. It’s been a > wonderfully positive experience to have in-depth discussions on complex > issues with industry experts. We look forward to moving these discussions > and votes to an Apache mailing list. > > Community > ------------ > Hop is developed, extended and maintained by a global community of users > and developers. The Hop community is what has driven its development and > growth. > The particular past history of Hop has led to a lot of interest for the > project and already led to a number of contributions, documentation and > translations. > > Core Developers > ---------------- > We have a diverse group of core developers with people joining on a regular > basis. Matt Casters, Rodrigo Haces and David Rosenblum are part time > developers on Hop, salaried by Neo Solutions. Bart Maertens, Hans Van > Akelyen, Yannick Mols are part time Hop developers paid for by company > know.bi. Doug and Gretchen Moran were Pentaho employees but along with > Rafael Valenzuela, Dan Keeley, Jason Chu, Sergio Ramazzina and many others > they can be considered to be long time consultants and community members > for over a decade that joined the Hop community in the last year or two. > > > Alignment > ---------- > We want to anchor and safeguard our development and community building > efforts for the future. We strongly believe that as an Apache project this > can be achieved in the best possible way. The Hop project also started to > align with projects like Apache Beam, Spark and Flink in it's use of > terminology, tools, manner of configuration and so on. As mentioned > elsewhere in this document Hop is a large user of other Apache projects and > libraries and we believe that becoming an Apache project is beneficial. > Specifically for Apache Beam we believe that providing a visual pipeline > development tool can be of great value. > > Known Risks > ============ > While the current code-base of Kettle on which we have started from is > already released under the Apache Public License 2.0 proper attribution > needs to happen to Hitachi Vantara. > We have no knowledge of existing patents on any part of the Kettle > codebase. > To further reduce any risk of there even being any discussion on naming the > Hop team decided to rename the project, its tools (to be more self-evident > as well), the java API and even the main concepts (Transformations are now > called Pipelines, in line with Apache Beam naming conventions). > > Orphaned products > ------------------ > There is little risk that the project will become orphaned. The list of > active developers is large, and consists of a mix of developers who have > been working on the code for several years and recent arrivals in the > community. > > Inexperience with Open Source > ------------------------------ > The project team has a long history in open source and has contributed to > Apache licensed open source projects, mostly in the Kettle ecosystem such > as Kettle itself and the many plugins and projects surrounding it. The > experience gained there has allowed us to quickly set up all required build > tools and processes. In its fairly short history, Hop has been advocating > open source in all aspects of the project. Our submission to the Apache > Software Foundation is a logical extension of our commitment to open source > software. > > Licensing > ---------- > The original source code we started from (see below) has been open source > since december 2005, initially under the Lesser GPL but since January 2012 > all under the Apache License version 2.0. All Hop code has been scanned for > compliance with APL 2.0. We integrated Apache Rat with our build process. > > Heterogeneous Developers > ------------------------- > Hop is built, developed and maintained by a global community of > developers. Input comes from a large group of developers and users from > all over the world. At this moment over 7 companies contribute to Hop > through the developers along with a list of individuals and consultants. > > Reliance on Salaried Developers > -------------------------------- > Hop developers are a mix of volunteers, enthusiasts and people working for > an employer. There is also a group of consultants who want to be involved > in Hop because it allows them to do projects with it. They are in fact our > most important users and developers since they provide valuable feedback > from the trenches. > > Relationships with Other Apache Products > ----------------------------------------- > Hop is a heavy user of Apache software libraries. > > Apache Commons usage: > * commons-beanutils > * commons-cli > * commons-codec > * commons-collections > * commons-collections4 > * commons-compiler > * commons-compress > * commons-configuration > * commons-database-model > * commons-dbcp > * commons-digester > * commons-el > * commons-httpclient > * commons-io > * commons-lang and commons-lang3 > * commons-logging > * commons-math and commons-math3-3.5.jar > * commons-net > * commons-pool > * commons-validator > * commons-vfs2 > > > Other libraries: > * Apache Batik : for the front-end SVG drawing > * Apache Xerces (XSLT, XML processing) > > > Other usage of Apache projects related to Hop (plugins): > * Apache Avro > * Apache Beam w/ Apache Spark, Apache Flink, … > * Apache Cassandra > * Apache CouchDB > * Apache Derby > * Apache Flume > * Apache Hadoop > * Apache Hive > * Apache Kafka > * Apache Solr > * Apache Subversion > * Apache Zookeeper > > > For the build process > * Apache Maven > * Apache Jenkins > > An excessive Fascination with the Apache Brand > ----------------------------------------------- > With this proposal we are not seeking attention or publicity. Rather, we > firmly believe in Hop, visual data pipeline development and the ability to > treat the developed data pipelines (ETL) as software code. While the > original Hop code has been open source for about 15 years, we believe > putting code on GitHub can only go so far. We see the Apache community, > processes, and mission as critical for ensuring Hop is truly > community-driven, positively impactful, and innovative open source > software. We believe Hop is a great fit for the Apache Software Foundation > due to its focus on visual data processing and its relationships to > existing ASF projects. > > Documentation > ============== > Over the years, the community has contributed extensive documentation to > wiki.pentaho.com. Over time, areas of the available information have > become > incomplete or outdated. Most of this documentation has been reviewed, > updated and will be contributed to the Apache foundation with the Hop > source code. Documentation for the extensive new functionality that was > added to Hop in recent months is being written. > We consider documentation to be a core piece of the Hop platform and will > treat documentation as any other item of code. > > Initial Source > =============== > While there isn’t a Java class in Hop which is unchanged from its origins > we should mention we selected this source code to form the base of Apache > Kettle: > https://github.com/pentaho/pentaho-kettle/tree/8.2.0.7-R > > We merged various changes from the WebSpoon fork found over here: > https://github.com/HiromuHota/pentaho-kettle > > > Various community driven Kettle plugins were written to bypass bugs, slow > down code-rot and to implement missing features. They were were merged > into Hop from these locations: > https://github.com/mattcasters/kettle-debug-plugin (better debugging) > https://github.com/mattcasters/kettle-beam (Apache Beam support) > https://github.com/mattcasters/pentaho-pdi-dataset (Unit Testing) > https://github.com/mattcasters/kettle-needful-things (Bug fixes & > workarounds) > https://github.com/mattcasters/kettle-environment (Environment management) > > > The Hop repositories are currently hosted at: > https://github.com/project-hop/ > * Hop: source code for the Hop project > * Hop-doc: technical documentation for the Hop project > * Hop-website: Hop website and content repository > * Hop-docker: Docker containers, Kubernetes > > Source and Intellectual Property Submission Plan > ================================================= > The originating source code is already licensed under an Apache 2 license: > * https://github.com/pentaho/pentaho-kettle/blob/8.2.0.7-R/LICENSE.txt > * > https://github.com/HiromuHota/pentaho-kettle/blob/webspoon-8.3/LICENSE.txt > * https://github.com/mattcasters/kettle-debug-plugin/blob/master/LICENSE > * https://github.com/mattcasters/kettle-beam/blob/master/LICENSE > * > https://github.com/mattcasters/pentaho-pdi-dataset/blob/master/LICENSE.txt > * https://github.com/mattcasters/kettle-needful-things/blob/master/LICENSE > * https://github.com/mattcasters/kettle-environment/blob/master/LICENSE > > > For all contributions we have an agreement in place: > https://cla-assistant.io/project-hop/hop > > External Dependencies > ====================== > Over the course of the last year we removed non-essential dependencies as > much as possible and replaced them by interfaces and plugin types. We did > this to simplify the architecture. > It’s important to note all external dependencies are licensed under an > Apache 2.0 or Apache-compatible license. As we grow the Hop community we > will configure our build process to require and validate all contributions > and dependencies are licensed under the Apache 2.0 license or are under an > Apache-compatible license. > > Cryptography > ============= > > Required Resources > =================== > > Mailing lists > -------------- > We currently use a mix of email and Mattermost. We will migrate our > existing mailing lists to the following: > > d...@hop.incubator.apache.org > u...@hop.incubator.apache.org > priv...@hop.incubator.apache.org > comm...@hop.incubator.apache.org > > Git Repository > --------------- > The Hop code is currently in git, we’d like to keep it that way. We request > a git repository for incubator-hop with mirroring to GitHub. > > Issue Tracking > --------------- > We request the creation of an Apache-hosted JIRA. > > Jira ID: HOP > > > Other Resources > ---------------- > To allow other projects to use Hop as a library we would love to publish > artifacts on a Maven server like maven.apache.org. > > Initial Committers > =================== > * Nicholas Adment <nadm...@gmail.com> > * Hans Van Akelyen <hans.van.akel...@know.bi> > * Lokke Bruyndonckx <lokke.bruyndon...@know.bi> > * Matt Casters <matt.cast...@neo4j.com> > * Jason Chu <jianjun...@gmail.com> > * Peter Fabricius <i...@peter-fabricius.de> > * Rodrigo Haces <rodrigo.ha...@neo4j.com> > * Dave Henry <dshenr...@gmail.com> > * Hiromu Hota <hiromu.h...@gmail.com> > * Brandon Jackson <usbran...@gmail.com> > * Dan Keeley <d...@dankeeley.co.uk> > * Bart Maertens <bart.maert...@know.bi> > * Yannick Mols <yannick.m...@know.bi> > * Doug Moran <d...@dougandgretchen.com> > * Gretchen Moran <gretc...@dougandgretchen.com> > * Sergio Ramazzina <sergio.ramazz...@serasoft.it> > * Maria Carina Roldan <maria.carina.rol...@gmail.com> > * David Rosenblum <david.rosenb...@neo4j.com> > * Rafael Valenzuela <rav...@gmail.com> > > Affiliations > ============= > * Neo4J > * Matt Casters > * Rodrigo Haces > * David Rosenblum > * Know.bi > * Bart Maertens > * Hans Van Akelyen > * Lokke Bruyndonckx > * Yannick Mols > * eHealth Africa > * Doug & Gretchen Moran > * Schemetrica > * Dave Henry > * Beijing Auphi Data Co > * Jason Chu > * Serasoft Italy > * Sergio Ramazzina > * Hitachi Research > * Hiromu Hota > > > Sponsors > ========= > Champion > --------- > Maximilian Michels (m...@apache.org) > > Nominated Mentors > ------------------ > Tom Barber (magicaltr...@apache.org) > Julian Hyde (jh...@apache.org) > Maximilian Michels (m...@apache.org) > > Sponsoring Entity > ================== > The Apache Incubator >
