Mathew, certainly! I've been meaning to update the blog too. Email me at mattyb...@apache.org and we'll get 'em up there :)
Thanks, Matt On Sat, Jun 15, 2024 at 2:46 PM Mathew Kiprop <mathewkipro...@gmail.com> wrote: > +1 to all subjects addressed Joe. > > I have been planning to improve developer advocacy around scripting > components. The groovy scripts have been one of my best tools to build > flows in NiFi (Rest API development , complex data structures > transformation, controlled flows scheduling…..). I’ll take this opportunity > to ask Mat Burges whether I can contribute some of my work to > http://funnifi.blogspot.com/?m=1. > > Mathew > > On Sat, 15 Jun 2024 at 21:03, Joe Witt <joew...@apache.org> wrote: > > > Team, > > > > We are seeing an increase in the unique contributors on a monthly basis > to > > NiFi and a clear and steady increase in the overall contribution rate > over > > the past 1-2 years in particular. This is awesome but we do have > important > > problems to cover and we need to make it easier to help newer > contributors > > know where to jump in and add the most value. > > > > For sure people will contribute various things just because they have a > > particular interest such as some new extension, etc.. But from a > community > > direction perspective there are things we need to continue to improve > upon > > and we want to help make the path to committership and PMC membership > more > > clear as a function of adding value to the community. > > > > So not necessarily ordered, here are some key things I've observed that > > need more attention and are among the most valuable things for the health > > of the nifi system and community. > > > > Dependency Management > > > > This is a hugely important area as we have a massive amount of > dependencies > > and this work is just plain not that fun. But if we want to present a > > beautiful landscape we need to pull weeds. We need to reduce > dependencies > > and keep things recent. This is largely accomplished today by a very > small > > group of people. We have to increase the contribution level here or > > perhaps the automation level. We have tooling that is helping now but > > still needs more attention. > > > > Vulnerability Management particular as it relates to dependencies > > > > Related to dependency management but broader. This one at times requires > > us to do more than simply maintain dependencies. It sometimes forces us > to > > change libraries and sometimes even makes maintaining > > compatibility/configuration complicated. These are among the most > critical > > and most difficult tasks to take on. This is largely done today by an > > extremely small group of people. > > > > Maintenance of extensions > > > > We have a ton of components in NiFi. Obviously not all of them really do > > require active maintenance but there is often a contribute and forget > model > > involved. For us to keep these broad range of components under community > > care they need active maintenance. I already mentioned the dependency > > factor above but the broader point here is that APIs/versions of systems > > evolve. We need to be staying up to date. There are systems like Kafka, > > Elastic, Mongo, Relational Databases, etc.. that we should always be > > supporting the latest on. These are among the most wildly popular > sources > > and destinations. We do tend to fall behind here and this is powerfully > > useful for people to contribute to. These are the things our users > > actually use NiFi for! > > > > Maintenance of things other than core nifi > > > > If you look at components ancillary to NiFi they receive far less > attention > > and effort trending toward them going away. The NiFi registry is a great > > example of this as NiFi MiNiFi Java for instance. These are things which > > are useful and indeed there is a decent user base in both cases. But > they > > require more active maintenance than they get. We need to evolve when it > > comes to security methods offered for authn/authz and that takes focus. > > > > Performance analysis and improvements targeted as a result > > > > This is done very rarely as far as I can tell outside a couple of people > > who I'm sure we could all name in a few seconds. The codebase is large > and > > Java and other parts evolve. Continually emphasizing faster and more > > efficient operations is vital to NiFi's continued growth in the > > community at large. People who spend time running NiFi in a measured > > manner with realistic flows and doing things like profiling for CPU, > > memory, etc.. and finding bottlenecks and offering solutions - absolutely > > can be high impact and valuable. Even if not at a point where a > confident > > contribution can be made, the very act of doing this level of evaluation > > and flagging concerns is valuable. > > > > Improvements related to tests both unit and integration - but also for > > users of NiFi > > > > As part of our push to NiFi 2.0 we have been running these tests far more > > often and more complete than ever before. Github Actions now pretty much > > does run all the things. We've deleted a wild amount of junk tests that > > would always fail or were spurious or required some complex local > > access/setup and therefore nobody ever ran them. We need to always > improve > > coverage but we want to make tests faster/more efficient and more > > leveraging of things like test containers would be highly valuable. > > Importantly though here I also mean to highlight we need to do a lot more > > for users of NiFi that want to create automated tests for their flows. > We > > should make it easy for them to provide a flow definition or perhaps a > DSL > > for their flow and write tests which would leverage actual nifi to > validate > > a given input results in an expected output. This is something that > would > > add a ton of value for our user base but also lead into some really > > powerful testing we can do against our own framework. > > > > Development of blogs/videos/conference presentations/social media posts > > > > Building great software by itself doesn't grow the community nearly as > much > > as if you build a strong developer advocacy game. People that create > these > > various forms of blogs/videos and conference talks about NiFi, how it > > works, and how to make it better are extremely valuable. That is often > > just as valuable or even more so than a PR review, code contribution, > etc.. > > > > Engagement in the Slack/Mailing lists > > > > Our community needs help. They ask a lot of questions about how to use > > NiFi, bugs they are hitting, etc.. Responding to these questions in a > > timely and meaningful manner is hugely valuable to them and is also > pretty > > fun for the person who helps out. > > > > Meaningful engagement acting as a pull request reviewer > > > > This one is last but certainly not least! We have a review-then-commit > > model which means we need at least one committer or PMC member to review > a > > contribution before it gets merged. Even if you're not a committer or > PMC > > member, a review is still super helpful and can save time and also puts > you > > on a great path to earning merit. Reviews are a critical part of the > code > > and overall quality of NiFi. 'LGTM' is generally not that helpful and > > while warranted for some trivial things we even see it on larger more > > meaningful commits. Not every review needs to be a back and forth affair > > of code style inputs either. There is a middle ground and you'll find it > > as you engage in more reviews! Code reviews are in far more short > supply > > than code contributions. I encourage everyone to please do at least one > > true pull request review for every contribution. In general the goal > > should be to contribute to more pull request reviews than new pull > requests > > created. If you do so this by definition means the community is growing > > and you're directly helping make that happen! > > > > Would love to see additional inputs/missing categories/other thoughts. > > Perhaps this turns into a wiki page that helps give more context to > people > > that want to find ways to add value and be on a path to committer and PMC > > membership as well. > > > > Thanks > > Joe > > >
