Yay! I am very enthusiastic for this progress. Big +1 from me to use this as a chance to also transition off of jira.
A couple of questions / concerns: a) when / how are we going to add additional repositories? Are we adding one whenever someone comes with a new source/sink? Do we want to have a flume-contrib or the like that things could land in initially? If we do this, can some of these components that are primarily examples live there? I’m thinking of things like flume-twitter, flume-morphline, flume-kudu, flume-legacy. b) how broad is the flume-hadoop meant to be? I presume the hbase sink(s) won’t be staying in flume-core. I would argue for an independent flume-hbase so we can use the current hbase client libraries without having to worry about Hadoop specifics. c) I share some of Tristan’s concerns on downstream consumption. Can we add in a packaging repo that initially provides some kind of minimal downstream consumable flume to deploy as well as an omnibus deploy that contains everything like we have today? d) when talking about components that can’t practically be supported, I’d like to flag up the current twitter source. It’s great for flume demos, but our current use relies on an API that is dropping out of support. Additionally, there is the newly looming possibility that there won’t be a free-for-use tier of the API to test against. > On Feb 26, 2023, at 1:08 AM, Ralph Goers <ralph.go...@dslextreme.com> wrote: > > As I mentioned last year I would like to start breaking up flume into > separate repos. There are a few reasons for this: > 1. Flume has grown so large that the CI system can no longer build it. The > jobs run out of disk space due to the large logs. > 2. The build takes a very long time to run. > 3. There are several components that can no longer be practically be > supported. > > To this end I am planning on creating the following Git repos: > flume-hadoop > flume-http > flume-irc > flume-jdbc > flume-jms > flume-kafka > flume-kudu > flume-legacy > flume-morphline > flume-scribe > flume-search > flume-spring-boot > flume-twitter > > For the time being I would propose everything else remain in the current > Flume repo. > > Note that as each of these is populated they will each need to be released, > However, most of these are fairly inactive so after the initial release they > may not need to be touched very often. > > Also, since Jira now requires new users to request us to create accounts for > them I would propose that as each of these repos are set up that they be > configured to enable GitHub Issues. > > I am looking for feedback on this but if I don’t get any I plan to start work > on this within a week or so. > > Ralph
