[
https://issues.apache.org/jira/browse/NIFI-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated NIFI-13301:
----------------------------------------
Description:
*Objective:* Improve/enable sharing/reuse of at least two features of Apache
NiFi, so the community can have an easier time contributing their flows and/or
custom processors into a NiFi Marketplace:
* {*}Pre-Designed Flows{*}: (Approach 1) stored in a NiFi Registry or
accessible location. (Approach 2) NiFi Registry can also substituted by NiFi's
git registry client having the location of the NiFi Marketplace. Thus, we just
have a git location and NiFi uses that to find/store flows.
* *Components/Processors* built in *Java/Python* that anyone is free to use
* *End to End Full Stack Applications Powered By NiFi or MiNiFi CPP?* The
frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer, OHIF
Viewer, etc: (Maybe this one can be subscription based where users get access
if they invest in the monthly or yearly subscription?)
*Potential Solution:* Apache NiFi builds an extension point for interacting
with Extension Registries, similar to how it currently has
*{{FlowRegistryClient}}* with implementations for NiFi Registry and now GitHub,
there would be *{{ExtensionRegistryClient}}* with implementations for stuff
like Nexus, NiFi Registry, and any other vendor ones, for example maybe
Datavolo provides one, but basically in apache we don't want to build another
application, we already have NiFi Registry. (Briefly Discussed with [~bbende] )
* I will start by looking into *{{FlowRegistryClient}}* and then base
*{{ExtensionRegistryClient}}* toward that approach. If I am understanding
correctly, we would contribute an *{{ExtensionRegistryClient}}* feature into
Apache NiFi that enables NiFi to integrate with other vendors (Datavolo, H2Oai,
etc) group of processors, data flow templates and so on. I see where you are
coming from with for Apache the goal being not to build another application.
That application could be by the another and NiFi's
*{{ExtensionRegistryClient}}* hooks up to their {*}RegistryServer{*}. (Heres
the path I will take toward that goal).
{*}Project Ownership{*}: There will be a clear line that the NiFi Marketplace
is not owned by the Apache NiFi PMC, rather it will be managed and owned by a
vendor in close alignment with NiFi, which we will announce at a later time.
{*}Motivation (NiFi){*}: Some people in the community have shown interest in
having a NiFi Processor Marketplace where they can contribute their category of
processors. In particular, I saw some people interested in contributing their
NiFi python processors. This NiFi Processor marketplace could be also applied
toward the community interested in contributing custom NiFi java processors.
* {*}Extra MiNiFi C+{+}{{+}}{*}: We could even add a section for MiNiFi C+
custom processors. There is a part of the MiNiFi build process that brings in
NiFi through building the JNI extension. So, that is a way to integrate NiFi
Registry there too.
I have briefly talked with [~bbende] and [~joewitt] asking them questions on
ways to bring custom python processors into production. One of the routes was
through leveraging NiFi Registry in connection with Apache NiFi to streamline
integration. For my use case, I would leverage the NiFi Processor Marketplace
to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi Python
Processors where I focused on my master thesis AI/DL for stroke diagnosis.
*High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi
Processor Marketplace App, there are frameworks we can leverage for the UI from
ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be used for
full stack, but probably would mainly be used for the frontend portion) and
other options. in the case that we want to make something more custom to
display each category of processors. This full stack application could interact
wth the NiFi Registry, which is the central location for storage and management
of shared resources across one or more instances of NiFi and/or MiNiFi (C++).
What are some hosting options for Apache projects?
was:
*Objective:* Improve/enable sharing/reuse of at least two features of Apache
NiFi, so the community can have an easier time contributing their flows and/or
custom processors into a NiFi Marketplace:
* {*}Pre-Designed Flows{*}: (Approach 1) stored in a NiFi Registry or
accessible location. (Approach 2) NiFi Registry can also substituted by NiFi's
git registry client having the location of the NiFi Marketplace. Thus, we just
have a git location and NiFi uses that to find/store flows.
* *Components/Processors* built in *Java/Python* that anyone is free to use
* *End to End Full Stack Applications Powered By NiFi or MiNiFi CPP?* The
frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer, OHIF
Viewer, etc: (Maybe this one can be subscription based where users get access
if they invest in the monthly or yearly subscription?)
{*}Project Ownership{*}: There will be a clear line that the NiFi Marketplace
is not owned by the Apache NiFi PMC, rather it will be managed and owned by a
vendor in close alignment with NiFi, which we will announce at a later time.
{*}Motivation (NiFi){*}: Some people in the community have shown interest in
having a NiFi Processor Marketplace where they can contribute their category of
processors. In particular, I saw some people interested in contributing their
NiFi python processors. This NiFi Processor marketplace could be also applied
toward the community interested in contributing custom NiFi java processors.
* {*}Extra MiNiFi C+{+}{+}{*}: We could even add a section for MiNiFi C+
custom processors. There is a part of the MiNiFi build process that brings in
NiFi through building the JNI extension. So, that is a way to integrate NiFi
Registry there too.
I have briefly talked with [~bbende] and [~joewitt] asking them questions on
ways to bring custom python processors into production. One of the routes was
through leveraging NiFi Registry in connection with Apache NiFi to streamline
integration. For my use case, I would leverage the NiFi Processor Marketplace
to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi Python
Processors where I focused on my master thesis AI/DL for stroke diagnosis.
*High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi
Processor Marketplace App, there are frameworks we can leverage for the UI from
ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be used for
full stack, but probably would mainly be used for the frontend portion) and
other options. in the case that we want to make something more custom to
display each category of processors. This full stack application could interact
wth the NiFi Registry, which is the central location for storage and management
of shared resources across one or more instances of NiFi and/or MiNiFi (C++).
What are some hosting options for Apache projects?
> Create NiFi Processor Marketplace App Powered by NiFi Registry
> --------------------------------------------------------------
>
> Key: NIFI-13301
> URL: https://issues.apache.org/jira/browse/NIFI-13301
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions, NiFi Registry
> Environment: Linux: Ubuntu 20.04 or 22.04
> Reporter: James Guzman (Medel)
> Assignee: James Guzman (Medel)
> Priority: Minor
>
> *Objective:* Improve/enable sharing/reuse of at least two features of Apache
> NiFi, so the community can have an easier time contributing their flows
> and/or custom processors into a NiFi Marketplace:
> * {*}Pre-Designed Flows{*}: (Approach 1) stored in a NiFi Registry or
> accessible location. (Approach 2) NiFi Registry can also substituted by
> NiFi's git registry client having the location of the NiFi Marketplace.
> Thus, we just have a git location and NiFi uses that to find/store flows.
> * *Components/Processors* built in *Java/Python* that anyone is free to use
> * *End to End Full Stack Applications Powered By NiFi or MiNiFi CPP?* The
> frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer,
> OHIF Viewer, etc: (Maybe this one can be subscription based where users get
> access if they invest in the monthly or yearly subscription?)
> *Potential Solution:* Apache NiFi builds an extension point for interacting
> with Extension Registries, similar to how it currently has
> *{{FlowRegistryClient}}* with implementations for NiFi Registry and now
> GitHub, there would be *{{ExtensionRegistryClient}}* with implementations for
> stuff like Nexus, NiFi Registry, and any other vendor ones, for example maybe
> Datavolo provides one, but basically in apache we don't want to build another
> application, we already have NiFi Registry. (Briefly Discussed with [~bbende]
> )
> * I will start by looking into *{{FlowRegistryClient}}* and then base
> *{{ExtensionRegistryClient}}* toward that approach. If I am understanding
> correctly, we would contribute an *{{ExtensionRegistryClient}}* feature into
> Apache NiFi that enables NiFi to integrate with other vendors (Datavolo,
> H2Oai, etc) group of processors, data flow templates and so on. I see where
> you are coming from with for Apache the goal being not to build another
> application. That application could be by the another and NiFi's
> *{{ExtensionRegistryClient}}* hooks up to their {*}RegistryServer{*}. (Heres
> the path I will take toward that goal).
>
> {*}Project Ownership{*}: There will be a clear line that the NiFi Marketplace
> is not owned by the Apache NiFi PMC, rather it will be managed and owned by a
> vendor in close alignment with NiFi, which we will announce at a later time.
>
> {*}Motivation (NiFi){*}: Some people in the community have shown interest in
> having a NiFi Processor Marketplace where they can contribute their category
> of processors. In particular, I saw some people interested in contributing
> their NiFi python processors. This NiFi Processor marketplace could be also
> applied toward the community interested in contributing custom NiFi java
> processors.
> * {*}Extra MiNiFi C+{+}{{+}}{*}: We could even add a section for MiNiFi C+
> custom processors. There is a part of the MiNiFi build process that brings in
> NiFi through building the JNI extension. So, that is a way to integrate NiFi
> Registry there too.
> I have briefly talked with [~bbende] and [~joewitt] asking them questions on
> ways to bring custom python processors into production. One of the routes was
> through leveraging NiFi Registry in connection with Apache NiFi to streamline
> integration. For my use case, I would leverage the NiFi Processor Marketplace
> to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi
> Python Processors where I focused on my master thesis AI/DL for stroke
> diagnosis.
> *High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi
> Processor Marketplace App, there are frameworks we can leverage for the UI
> from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be
> used for full stack, but probably would mainly be used for the frontend
> portion) and other options. in the case that we want to make something more
> custom to display each category of processors. This full stack application
> could interact wth the NiFi Registry, which is the central location for
> storage and management of shared resources across one or more instances of
> NiFi and/or MiNiFi (C++). What are some hosting options for Apache projects?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
