Pierre, Thanks for the reply, and noting the potential concern with the ability to find these components.
I think there are several ways we can address this concern, both for optional Java components, and for Python components in a separate repository. For Python components in particular, we could add direct links to published versions on the main download page, calling out their availability in the official PyPI repository. Although this would need to be denoted as a non-official release channel for Apache purposes, this is common practice in other projects, and follows the approach we already have for container images on Docker Hub. In addition to linking from the download page, we could publish the generated documentation for these components. The current process for publishing generated documentation is based on the convenience binary, but with some adjustments, we could publish the documentation for the Python components as well. This is a good prompt to start doing this for the optional Java components, and I plan to look at doing this for the next release with optional Java components. To your last question, it is worth noting that any binary releases would fall into the category of convenience builds. Initially, I think the NiFi framework release would not include the Python extension components. However, having a few short steps on installation, linked from the download page, seems like it would provide a way forward. Regards, David Handermann On Fri, Jan 26, 2024 at 1:22 AM Pierre Villard <[email protected]> wrote: > > Hi David, > > While I agree with your summary, I have a concern here which is about user > awareness of this feature. We've seen in the past: as soon as we don't > include NARs in the convenience binary, we see that users have no clue > about those NARs (and some are super powerful/useful). I agree that python > is a bit different because it requires a user action to enable it in the > first place but I still think that including the components in the > convenience binary of Apache NiFi would drive user awareness, adoption, etc. > > If we have a separated repo with its own release cycle can we imagine a > process where, when releasing Apache NiFi, it'd include whatever is the > latest version of the Python repo? Or something along those lines? > > Pierre > > Le ven. 26 janv. 2024 à 08:01, David Handermann <[email protected]> > a écrit : > > > Team, > > > > As we get closer to a full release of Apache NiFi 2.0.0, we have an > > important opportunity to set the direction for future development of > > Python-based Processors. > > > > The introduction of native Python support presents a number of new > > integration opportunities, and it also raises questions about > > maintenance and versioning. As the journey to NiFi 2.0.0 has shown, it > > requires significant effort to coordinate maintenance and > > modernization across hundreds of project modules. Although the > > internal project structure has maintained helpful separation of API > > and implementation, the current release strategy highlights the > > challenges of verifying multiple layers of changes. Introducing a new > > programming language provides greater possibilities, but also makes it > > more difficult to maintain a single repository with a single > > versioning strategy. > > > > I propose creating a new Git repository named nifi-python-extensions, > > which would have its own versioning and release process. This would > > contain the extensions now under the module of the same name in the > > NiFi repository. Having a separate repository and release process for > > Python-based extensions has the following advantages: > > > > 1. Clean separation between NiFi APIs for Python and Python-based > > Processors > > 2. Independent release cycles for Python-based Processors > > 3. Focused release verification and testing on Python-based modules > > > > These advantages can also enable more rapid iteration on Python-based > > Processors, without impacting the NiFi Framework or requiring new > > releases at that level. Although this would require a separate > > installation process for Python-based components, this could follow an > > approach similar to what is already required for optional Java-based > > components. > > > > Thanks in advance for your consideration. > > > > Regards, > > David Handermann > > Apache NiFi PMC Member > >
