I lean towards option 2 with the caveat that maybe we don't have to retain every Hadoop related component when creating this separate set of components. Mainly I'm thinking that Hive has been the most problematic to maintain so maybe that is dropped all together. I think it would be unfortunate to not have publicly available HDFS processors.
On Fri, Mar 24, 2023 at 3:23 PM Matt Burgess <mattyb...@apache.org> wrote: > > As one of the small number of people that fight the battle, I like the > idea of Option 1 (full disclosure: I work for a vendor). From a > community standpoint (I'm on the PMC) I'm not strongly opposed to > Option 2 although I wouldn't want to be the one managing and releasing > the artifacts :) Having said that, unless it remained maintained and > released, I feel like it would just be a component graveyard (or maybe > more like the Apache Attic), in which case it seems unnecessary and > that's why I'm behind Option 1. Interested to hear others' thoughts of > course. > > Thanks, > Matt > > On Fri, Mar 24, 2023 at 2:07 PM Joe Witt <joe.w...@gmail.com> wrote: > > > > Team, > > > > For the full time NiFi has been in Apache we've built with support for > > various Hadoop ecosystem components like HDFS, Hive, HBase, others, > > and more recently formats/serialization modes like necessary for > > Parquet, Orc, Iceberg, etc.. > > > > All of these things however present endless challenges with > > compatibility across different versions (Hive being the most difficult > > by far), vendors (hadoop vendors, cloud vendors, etc..). And also > > super notably the incredible number of dependencies, dependency > > conflicts, inclusions/exclusions, old log libs, vulnerability updates, > > etc.. And last but certainly not least a big reason why our build has > > grown so much. > > > > We have a couple options: > > 1. Deprecate these components in NiFi 1.x and drop them entirely in > > NiFi 2.x. Leave this as a problem for vendors to deal with. NiFi > > users interacting with such components are nearly exclusively doing so > > with vendors anyway. > > > > 2. Remove the components from NiFi main code line and create a > > separate repo for 'nifi-hadoop-extensions'. We manage those > > independently and release them periodically. They would be available > > for people to grab the nars if they want to use them. We include none > > of them in the convenience binary going forward by default. > > > > 3. Change nothing. Continue to battle with the above listed items. > > This is admittedly a bit of a non-option. We can't keep spending the > > same time/energy on these we have. It is a very small number of > > people that fight this battle. > > > > Look forward to hearing thoughts on these options or others we might > > consider. > > > > Thanks
