Hi Chris, I agree that connector is very important. Other Apache projects became popular mostly thank to the connectors set (I’m thinking about Apache Beam, Apache Camel, or Apache Karaf Decanter for instance). The connectors allow more users to "integrate" Gobblin in their ecosystem, so it would increase our users community. It will also increase our dev community as it’s probably easier to contribute on connector than in the Gobblin core.
About the repo vs module, there are two questions IMHO: 1. How to keep API/code sync together between Gobblin core and the connectors 2. Do we plan to have a different release cycle between core and connectors (even if it’s always possible to release a module atomically) IMHO, if we plan to do a Gobblin release including core + connectors, then a module is easier. Regards JB > Le 22 mars 2021 à 23:44, Chris Li <[email protected]> a écrit : > > Proposal: > > DIL (LinkedIn internal project name) is a generic multi-stage Gobblin > connector library. The code can be accessed here: > https://github.com/linkedin/gobblin-connectors. Its core features and high > level descriptions are shared here: > https://engineering.linkedin.com/blog/2021/data-integration-library. > > Per initial discussion with members of Gobblin community, we are here > proposing a separate sub-repo for this library. > > Why: > Some thoughts/justifications of a sub-repo vs. a module in the > main Gobblin repo. > > > 1. Gobblin connectors are important part of Gobblin ecosystem, but the > development of connectors is relatively independent of Gobblin core. > 2. Gobblin connector is where open source communities can contribute the > most, and it will be growing much faster than Gobblin core. > 3. The new connector library is a comprehensive package of unique design > patterns. This is where the data integration diversity challenge will be > addressed. The importance of this code base grows by day as more integration > scenarios are becoming supported. > 4. The new connector library evolves and replaces many prior Gobblin > connectors under the “gobblin-modules” module. A separate repo will help > avoid confusion. > 5. Separating core and ecosystem modules can help improve isolation and > reduce the number of defects. > > Regards, > Chris > > > >
