Hi Chris, Thanks for starting this thread, and for your contribution. My thoughts: 1. I like the idea of keeping Gobblin's connectors on a separate sub-repo, because these are fairly independent pluggable connectors. 2. They can bring in a significant amount of dependencies, which are better kept isolated from core.
As for the next steps: a. I request Gobblin committers and PMCs to review this repo to ensure it's in a shape to add to Gobblin (including code quality, structure, dependencies it brings in, etc.) b. Chris, please confirm that this contribution will be under Apache 2.0 license and the code already has the right license headers. Once, we have an affirmative answer for (a), and (b). I will coordinate with Apache Infra to set up a sub-repo for this in Gobblin, and work with Chris to bring it in. Thanks, Abhishek On Mon, Mar 22, 2021 at 5:41 PM Chris Li <[email protected]> wrote: > Proposal: > > DIL (LinkedIn internal project name) is a generic multi-stage Gobblin > connector library. The code can be accessed here: > https://github.com/linkedin/gobblin-connectors. Its core features and > high level descriptions are shared here: > https://engineering.linkedin.com/blog/2021/data-integration-library. > > Per initial discussion with members of Gobblin community, we are here > proposing a separate sub-repo for this library. > > Why: > Some thoughts/justifications of a sub-repo vs. a module in the > main Gobblin repo. > > > 1. Gobblin connectors are important part of Gobblin ecosystem, but the > development of connectors is relatively independent of Gobblin core. > 2. Gobblin connector is where open source communities can contribute > the most, and it will be growing much faster than Gobblin core. > 3. The new connector library is a comprehensive package of unique > design patterns. This is where the data integration diversity challenge > will be addressed. The importance of this code base grows by day as more > integration scenarios are becoming supported. > 4. The new connector library evolves and replaces many prior Gobblin > connectors under the “gobblin-modules” module. A separate repo will help > avoid confusion. > 5. Separating core and ecosystem modules can help improve isolation and > reduce the number of defects. > > Regards, > Chris > > > > >
