[jira] [Commented] (NIFI-13077) On-demand Extension Provider
[ https://issues.apache.org/jira/browse/NIFI-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851852#comment-17851852 ] James Guzman (Medel) commented on NIFI-13077: - Thanks [~pvillard] for sharing this Jira ticket in the NIFI-13301 ticket. Yes many great benefits to having an On-Demand Extension Provider. One of the benefits that caught my attention as I was reading your description was the ability to create smaller versions of Apache NIFi. Both when building NiFi and MiNiFi CPP from source, I have faced multiple times where the build failed because of some external libraries that failed to download or failed to build successfully, which caused the NiFi or MiNiFi CPP to fail building. As I start working on NIFI-13301. I will keep your NIFI-13077 in mind too. I really your extra input on this area. I was talking about this area with [~bbende] and [~joewitt] . We'll most likely need to break this feature into multiple sub tasks. > On-demand Extension Provider > > > Key: NIFI-13077 > URL: https://issues.apache.org/jira/browse/NIFI-13077 > Project: Apache NiFi > Issue Type: Epic > Components: Core Framework >Reporter: Pierre Villard >Priority: Major > > We currently have the concept of *ExternalResourceProvider* with two > implementations (HDFS and NiFi Registry) that can be configured to list and > download all NARs made available in those locations. Those implementations, > if configured, would get started when NiFi starts and would download ALL of > the available NARs, plus a background thread would check every five minutes > for new NARs to be available and downloaded. > The proposal here is to have a similar concept that would focus on extensions > / components but instead of having a background thread and instead of having > all of the components downloaded, the approach would be to plug this into the > *ExtensionBuilder* and when a component cannot be instantiated (when loading > a flow definition) with locally available components, then, instead of > creating a ghost component, the Extension Providers would be queried with > specific coordinates and if the provider makes the component available, then > the NAR would be downloaded (alongside required dependencies if the NAR > depends on another NAR). > This approach already exists in the *Kafka Connect NiFi plugin* with the > class {*}ExtensionClientDefinition{*}. By adopting this approach in NiFi, > it’d be much easier to ship a much *smaller version of NiFi* and have NiFi > download the required components based on flows that are being instantiated / > deployed. > The operation of downloading the NAR would not be blocking, meaning that we > would still create a ghost component but after completion of the NAR(s) > download and the loading of the components, the flows would be fully > operational. > It might be possible to show something similar as for the Python extensions > where we show that the component is still in the process of downloading third > party dependencies. > While this is a great opportunity to reduce the size of the NiFi binary (and > associated container image), it would not be great from a user perspective > when designing flows because all of the NARs removed from the default image > would no longer be visible in the list of available components when adding, > for example, a processor to the canvas. > Longer term we could imagine that the extension providers can also implement > a listing API so that when showing the list of available components, we would > show the list of the components available locally as well as the components > available through the extensions providers. The listing of components could > add another column to indicate the source of the component. > This is something that is exposed for the Extension Bundles in the NiFi > Registry (we also have the information about the NiFi API version that has > been used for building the components so we could use this information to > only list components that should be compatible from an API standpoint - same > major version but lower or equal API version). > The immediate goal though would be to introduce the concept of > ExtensionProvider with the following APIs: > {code:java} > boolean isAvailableExtension(Coordinates) > void downloadExtension(Coordinates) > {code} > Longer term we could also consider something like: > {code:java} > List listExtensions(){code} > But we would need to figure out how a NAR can provide the information about > the components that are inside of it. The NiFi Registry provides this > information, but that would not be the case for a Maven based implementation > for example. > In nifi.properties we would have something looking like: > {code:java} > nifi.nar.extension.provider..{code} > And we would
[jira] [Updated] (NIFI-13077) On-demand Extension Provider
[ https://issues.apache.org/jira/browse/NIFI-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated NIFI-13077: Description: We currently have the concept of *ExternalResourceProvider* with two implementations (HDFS and NiFi Registry) that can be configured to list and download all NARs made available in those locations. Those implementations, if configured, would get started when NiFi starts and would download ALL of the available NARs, plus a background thread would check every five minutes for new NARs to be available and downloaded. The proposal here is to have a similar concept that would focus on extensions / components but instead of having a background thread and instead of having all of the components downloaded, the approach would be to plug this into the *ExtensionBuilder* and when a component cannot be instantiated (when loading a flow definition) with locally available components, then, instead of creating a ghost component, the Extension Providers would be queried with specific coordinates and if the provider makes the component available, then the NAR would be downloaded (alongside required dependencies if the NAR depends on another NAR). This approach already exists in the *Kafka Connect NiFi plugin* with the class {*}ExtensionClientDefinition{*}. By adopting this approach in NiFi, it’d be much easier to ship a much *smaller version of NiFi* and have NiFi download the required components based on flows that are being instantiated / deployed. The operation of downloading the NAR would not be blocking, meaning that we would still create a ghost component but after completion of the NAR(s) download and the loading of the components, the flows would be fully operational. It might be possible to show something similar as for the Python extensions where we show that the component is still in the process of downloading third party dependencies. While this is a great opportunity to reduce the size of the NiFi binary (and associated container image), it would not be great from a user perspective when designing flows because all of the NARs removed from the default image would no longer be visible in the list of available components when adding, for example, a processor to the canvas. Longer term we could imagine that the extension providers can also implement a listing API so that when showing the list of available components, we would show the list of the components available locally as well as the components available through the extensions providers. The listing of components could add another column to indicate the source of the component. This is something that is exposed for the Extension Bundles in the NiFi Registry (we also have the information about the NiFi API version that has been used for building the components so we could use this information to only list components that should be compatible from an API standpoint - same major version but lower or equal API version). The immediate goal though would be to introduce the concept of ExtensionProvider with the following APIs: {code:java} boolean isAvailableExtension(Coordinates) void downloadExtension(Coordinates) {code} Longer term we could also consider something like: {code:java} List listExtensions(){code} But we would need to figure out how a NAR can provide the information about the components that are inside of it. The NiFi Registry provides this information, but that would not be the case for a Maven based implementation for example. In nifi.properties we would have something looking like: {code:java} nifi.nar.extension.provider..{code} And we would loop through all the configured providers to find the appropriate NAR to download based on provided coordinates in the flow definition that is being instantiated (either from flow.json.gz, or an uploaded JSON flow definition, or when checking out a flow from a registry client). was: We currently have the concept of ExternalResourceProvider with two implementations (HDFS and NiFi Registry) that can be configured to list and download all NARs made available in those locations. Those implementations, if configured, would get started when NiFi starts and would download ALL of the available NARs, plus a background thread would check every five minutes for new NARs to be available and downloaded. The proposal here is to have a similar concept that would focus on extensions / components but instead of having a background thread and instead of having all of the components downloaded, the approach would be to plug this into the ExtensionBuilder and when a component cannot be instantiated (when loading a flow definition) with locally available components, then, instead of creating a ghost component, the Extension Providers would be queried with specific coordinates and if the provider makes the component available, then the NAR
[jira] [Updated] (NIFI-13077) On-demand Extension Provider
[ https://issues.apache.org/jira/browse/NIFI-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated NIFI-13077: Description: We currently have the concept of ExternalResourceProvider with two implementations (HDFS and NiFi Registry) that can be configured to list and download all NARs made available in those locations. Those implementations, if configured, would get started when NiFi starts and would download ALL of the available NARs, plus a background thread would check every five minutes for new NARs to be available and downloaded. The proposal here is to have a similar concept that would focus on extensions / components but instead of having a background thread and instead of having all of the components downloaded, the approach would be to plug this into the ExtensionBuilder and when a component cannot be instantiated (when loading a flow definition) with locally available components, then, instead of creating a ghost component, the Extension Providers would be queried with specific coordinates and if the provider makes the component available, then the NAR would be downloaded (alongside required dependencies if the NAR depends on another NAR). This approach already exists in the Kafka Connect NiFi plugin with the class {*}ExtensionClientDefinition{*}. By adopting this approach in NiFi, it’d be much easier to ship a much smaller version of NiFi and have NiFi download the required components based on flows that are being instantiated / deployed. The operation of downloading the NAR would not be blocking, meaning that we would still create a ghost component but after completion of the NAR(s) download and the loading of the components, the flows would be fully operational. It might be possible to show something similar as for the Python extensions where we show that the component is still in the process of downloading third party dependencies. While this is a great opportunity to reduce the size of the NiFi binary (and associated container image), it would not be great from a user perspective when designing flows because all of the NARs removed from the default image would no longer be visible in the list of available components when adding, for example, a processor to the canvas. Longer term we could imagine that the extension providers can also implement a listing API so that when showing the list of available components, we would show the list of the components available locally as well as the components available through the extensions providers. The listing of components could add another column to indicate the source of the component. This is something that is exposed for the Extension Bundles in the NiFi Registry (we also have the information about the NiFi API version that has been used for building the components so we could use this information to only list components that should be compatible from an API standpoint - same major version but lower or equal API version). The immediate goal though would be to introduce the concept of ExtensionProvider with the following APIs: {code:java} boolean isAvailableExtension(Coordinates) void downloadExtension(Coordinates) {code} Longer term we could also consider something like: {code:java} List listExtensions(){code} But we would need to figure out how a NAR can provide the information about the components that are inside of it. The NiFi Registry provides this information, but that would not be the case for a Maven based implementation for example. In nifi.properties we would have something looking like: {code:java} nifi.nar.extension.provider..{code} And we would loop through all the configured providers to find the appropriate NAR to download based on provided coordinates in the flow definition that is being instantiated (either from flow.json.gz, or an uploaded JSON flow definition, or when checking out a flow from a registry client). was: We currently have the concept of ExternalResourceProvider with two implementations (HDFS and NiFi Registry) that can be configured to list and download all NARs made available in those locations. Those implementations, if configured, would get started when NiFi starts and would download ALL of the available NARs, plus a background thread would check every five minutes for new NARs to be available and downloaded. The proposal here is to have a similar concept that would focus on extensions / components but instead of having a background thread and instead of having all of the components downloaded, the approach would be to plug this into the ExtensionBuilder and when a component cannot be instantiated (when loading a flow definition) with locally available components, then, instead of creating a ghost component, the Extension Providers would be queried with specific coordinates and if the provider makes the component available, then the NAR would be
[jira] [Updated] (NIFI-13301) Create ExtensionRegistryClient for External Extension Registry Interaction
[ https://issues.apache.org/jira/browse/NIFI-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated NIFI-13301: Summary: Create ExtensionRegistryClient for External Extension Registry Interaction (was: Create NiFi Processor Marketplace App Powered by NiFi Registry) > Create ExtensionRegistryClient for External Extension Registry Interaction > -- > > Key: NIFI-13301 > URL: https://issues.apache.org/jira/browse/NIFI-13301 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions, NiFi Registry > Environment: Linux: Ubuntu 20.04 or 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > > *Objective:* Improve/enable sharing/reuse of at least two features of Apache > NiFi, so the community can have an easier time contributing their flows > and/or custom processors into a NiFi Marketplace: > * {*}Pre-Designed Flows{*}: (Approach 1) stored in a NiFi Registry or > accessible location. (Approach 2) NiFi Registry can also substituted by > NiFi's git registry client having the location of the NiFi Marketplace. > Thus, we just have a git location and NiFi uses that to find/store flows. > * *Components/Processors* built in *Java/Python* that anyone is free to use > * *End to End Full Stack Applications Powered By NiFi or MiNiFi CPP?* The > frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer, > OHIF Viewer, etc: (Maybe this one can be subscription based where users get > access if they invest in the monthly or yearly subscription?) > *Potential Solution:* Apache NiFi builds an extension point for interacting > with Extension Registries, similar to how it currently has > *{{FlowRegistryClient}}* with implementations for NiFi Registry and now > GitHub, there would be *{{ExtensionRegistryClient}}* with implementations for > stuff like Nexus, NiFi Registry, and any other vendor ones, for example maybe > Datavolo provides one, but basically in apache we don't want to build another > application, we already have NiFi Registry. (Briefly Discussed with [~bbende] > ) > * I will start by looking into *{{FlowRegistryClient}}* and then base > *{{ExtensionRegistryClient}}* toward that approach. If I am understanding > correctly, we would contribute an *{{ExtensionRegistryClient}}* feature into > Apache NiFi that enables NiFi to integrate with other vendors (Datavolo, > H2Oai, etc) group of processors, data flow templates and so on. I see where > you are coming from with for Apache the goal being not to build another > application. That application could be by the another and NiFi's > *{{ExtensionRegistryClient}}* hooks up to their {*}RegistryServer{*}. (Heres > the path I will take toward that goal). > > {*}Project Ownership{*}: There will be a clear line that the NiFi Marketplace > is not owned by the Apache NiFi PMC, rather it will be managed and owned by a > vendor in close alignment with NiFi, which we will announce at a later time. > > {*}Motivation (NiFi){*}: Some people in the community have shown interest in > having a NiFi Processor Marketplace where they can contribute their category > of processors. In particular, I saw some people interested in contributing > their NiFi python processors. This NiFi Processor marketplace could be also > applied toward the community interested in contributing custom NiFi java > processors. > * {*}Extra MiNiFi C+{+}{{+}}{*}: We could even add a section for MiNiFi C+ > custom processors. There is a part of the MiNiFi build process that brings in > NiFi through building the JNI extension. So, that is a way to integrate NiFi > Registry there too. > I have briefly talked with [~bbende] and [~joewitt] asking them questions on > ways to bring custom python processors into production. One of the routes was > through leveraging NiFi Registry in connection with Apache NiFi to streamline > integration. For my use case, I would leverage the NiFi Processor Marketplace > to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi > Python Processors where I focused on my master thesis AI/DL for stroke > diagnosis. > *High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi > Processor Marketplace App, there are frameworks we can leverage for the UI > from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be > used for full stack, but probably would mainly be used for the frontend > portion) and other options. in the case that we want to make something more > custom to display each category of processors. This full stack application > could interact wth the NiFi Registry, which is the central location for > storage and management of shared resources
[jira] [Updated] (NIFI-13301) Create NiFi Processor Marketplace App Powered by NiFi Registry
[ https://issues.apache.org/jira/browse/NIFI-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated NIFI-13301: Description: *Objective:* Improve/enable sharing/reuse of at least two features of Apache NiFi, so the community can have an easier time contributing their flows and/or custom processors into a NiFi Marketplace: * {*}Pre-Designed Flows{*}: (Approach 1) stored in a NiFi Registry or accessible location. (Approach 2) NiFi Registry can also substituted by NiFi's git registry client having the location of the NiFi Marketplace. Thus, we just have a git location and NiFi uses that to find/store flows. * *Components/Processors* built in *Java/Python* that anyone is free to use * *End to End Full Stack Applications Powered By NiFi or MiNiFi CPP?* The frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer, OHIF Viewer, etc: (Maybe this one can be subscription based where users get access if they invest in the monthly or yearly subscription?) *Potential Solution:* Apache NiFi builds an extension point for interacting with Extension Registries, similar to how it currently has *{{FlowRegistryClient}}* with implementations for NiFi Registry and now GitHub, there would be *{{ExtensionRegistryClient}}* with implementations for stuff like Nexus, NiFi Registry, and any other vendor ones, for example maybe Datavolo provides one, but basically in apache we don't want to build another application, we already have NiFi Registry. (Briefly Discussed with [~bbende] ) * I will start by looking into *{{FlowRegistryClient}}* and then base *{{ExtensionRegistryClient}}* toward that approach. If I am understanding correctly, we would contribute an *{{ExtensionRegistryClient}}* feature into Apache NiFi that enables NiFi to integrate with other vendors (Datavolo, H2Oai, etc) group of processors, data flow templates and so on. I see where you are coming from with for Apache the goal being not to build another application. That application could be by the another and NiFi's *{{ExtensionRegistryClient}}* hooks up to their {*}RegistryServer{*}. (Heres the path I will take toward that goal). {*}Project Ownership{*}: There will be a clear line that the NiFi Marketplace is not owned by the Apache NiFi PMC, rather it will be managed and owned by a vendor in close alignment with NiFi, which we will announce at a later time. {*}Motivation (NiFi){*}: Some people in the community have shown interest in having a NiFi Processor Marketplace where they can contribute their category of processors. In particular, I saw some people interested in contributing their NiFi python processors. This NiFi Processor marketplace could be also applied toward the community interested in contributing custom NiFi java processors. * {*}Extra MiNiFi C+{+}{{+}}{*}: We could even add a section for MiNiFi C+ custom processors. There is a part of the MiNiFi build process that brings in NiFi through building the JNI extension. So, that is a way to integrate NiFi Registry there too. I have briefly talked with [~bbende] and [~joewitt] asking them questions on ways to bring custom python processors into production. One of the routes was through leveraging NiFi Registry in connection with Apache NiFi to streamline integration. For my use case, I would leverage the NiFi Processor Marketplace to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi Python Processors where I focused on my master thesis AI/DL for stroke diagnosis. *High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi Processor Marketplace App, there are frameworks we can leverage for the UI from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be used for full stack, but probably would mainly be used for the frontend portion) and other options. in the case that we want to make something more custom to display each category of processors. This full stack application could interact wth the NiFi Registry, which is the central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi (C++). What are some hosting options for Apache projects? was: *Objective:* Improve/enable sharing/reuse of at least two features of Apache NiFi, so the community can have an easier time contributing their flows and/or custom processors into a NiFi Marketplace: * {*}Pre-Designed Flows{*}: (Approach 1) stored in a NiFi Registry or accessible location. (Approach 2) NiFi Registry can also substituted by NiFi's git registry client having the location of the NiFi Marketplace. Thus, we just have a git location and NiFi uses that to find/store flows. * *Components/Processors* built in *Java/Python* that anyone is free to use * *End to End Full Stack Applications Powered By NiFi or MiNiFi CPP?* The frontend could be various ones from PyQT,
[jira] [Commented] (NIFI-13301) Create NiFi Processor Marketplace App Powered by NiFi Registry
[ https://issues.apache.org/jira/browse/NIFI-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850436#comment-17850436 ] James Guzman (Medel) commented on NIFI-13301: - You're welcome. [~joewitt] . I am happy to help out with this effort in the open source. We can observe some projects that have done well with community engagement in the open source for their marketplace apps. > Create NiFi Processor Marketplace App Powered by NiFi Registry > -- > > Key: NIFI-13301 > URL: https://issues.apache.org/jira/browse/NIFI-13301 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions, NiFi Registry > Environment: Linux: Ubuntu 20.04 or 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > > *Objective:* Improve/enable sharing/reuse of at least two features of Apache > NiFi, so the community can have an easier time contributing their flows > and/or custom processors into a NiFi Marketplace: > * {*}Pre-Designed Flows{*}: (Approach 1) stored in a NiFi Registry or > accessible location. (Approach 2) NiFi Registry can also substituted by > NiFi's git registry client having the location of the NiFi Marketplace. > Thus, we just have a git location and NiFi uses that to find/store flows. > * *Components/Processors* built in *Java/Python* that anyone is free to use > * *End to End Full Stack Applications Powered By NiFi or MiNiFi CPP?* The > frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer, > OHIF Viewer, etc: (Maybe this one can be subscription based where users get > access if they invest in the monthly or yearly subscription?) > {*}Project Ownership{*}: There will be a clear line that the NiFi Marketplace > is not owned by the Apache NiFi PMC, rather it will be managed and owned by a > vendor in close alignment with NiFi, which we will announce at a later time. > > {*}Motivation (NiFi){*}: Some people in the community have shown interest in > having a NiFi Processor Marketplace where they can contribute their category > of processors. In particular, I saw some people interested in contributing > their NiFi python processors. This NiFi Processor marketplace could be also > applied toward the community interested in contributing custom NiFi java > processors. > * {*}Extra MiNiFi C+{+}{+}{*}: We could even add a section for MiNiFi C+ > custom processors. There is a part of the MiNiFi build process that brings in > NiFi through building the JNI extension. So, that is a way to integrate NiFi > Registry there too. > I have briefly talked with [~bbende] and [~joewitt] asking them questions on > ways to bring custom python processors into production. One of the routes was > through leveraging NiFi Registry in connection with Apache NiFi to streamline > integration. For my use case, I would leverage the NiFi Processor Marketplace > to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi > Python Processors where I focused on my master thesis AI/DL for stroke > diagnosis. > *High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi > Processor Marketplace App, there are frameworks we can leverage for the UI > from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be > used for full stack, but probably would mainly be used for the frontend > portion) and other options. in the case that we want to make something more > custom to display each category of processors. This full stack application > could interact wth the NiFi Registry, which is the central location for > storage and management of shared resources across one or more instances of > NiFi and/or MiNiFi (C++). What are some hosting options for Apache projects? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13301) Create NiFi Processor Marketplace App Powered by NiFi Registry
[ https://issues.apache.org/jira/browse/NIFI-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated NIFI-13301: Description: *Objective:* Improve/enable sharing/reuse of at least two features of Apache NiFi, so the community can have an easier time contributing their flows and/or custom processors into a NiFi Marketplace: * {*}Pre-Designed Flows{*}: (Approach 1) stored in a NiFi Registry or accessible location. (Approach 2) NiFi Registry can also substituted by NiFi's git registry client having the location of the NiFi Marketplace. Thus, we just have a git location and NiFi uses that to find/store flows. * *Components/Processors* built in *Java/Python* that anyone is free to use * *End to End Full Stack Applications Powered By NiFi or MiNiFi CPP?* The frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer, OHIF Viewer, etc: (Maybe this one can be subscription based where users get access if they invest in the monthly or yearly subscription?) {*}Project Ownership{*}: There will be a clear line that the NiFi Marketplace is not owned by the Apache NiFi PMC, rather it will be managed and owned by a vendor in close alignment with NiFi, which we will announce at a later time. {*}Motivation (NiFi){*}: Some people in the community have shown interest in having a NiFi Processor Marketplace where they can contribute their category of processors. In particular, I saw some people interested in contributing their NiFi python processors. This NiFi Processor marketplace could be also applied toward the community interested in contributing custom NiFi java processors. * {*}Extra MiNiFi C+{+}{+}{*}: We could even add a section for MiNiFi C+ custom processors. There is a part of the MiNiFi build process that brings in NiFi through building the JNI extension. So, that is a way to integrate NiFi Registry there too. I have briefly talked with [~bbende] and [~joewitt] asking them questions on ways to bring custom python processors into production. One of the routes was through leveraging NiFi Registry in connection with Apache NiFi to streamline integration. For my use case, I would leverage the NiFi Processor Marketplace to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi Python Processors where I focused on my master thesis AI/DL for stroke diagnosis. *High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi Processor Marketplace App, there are frameworks we can leverage for the UI from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be used for full stack, but probably would mainly be used for the frontend portion) and other options. in the case that we want to make something more custom to display each category of processors. This full stack application could interact wth the NiFi Registry, which is the central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi (C++). What are some hosting options for Apache projects? was: {*}Description (NiFi){*}: Some people in the community have shown interest in having a NiFi Processor Marketplace where they can contribute their category of processors. In particular, I saw some people interested in contributing their NiFi python processors. This NiFi Processor marketplace could be also applied toward the community interested in contributing custom NiFi java processors. * {*}Extra MiNiFi C++{*}: We could even add a section for MiNiFi C++ custom processors. There is a part of the MiNiFi build process that brings in NiFi through building the JNI extension. So, that is a way to integrate NiFi Registry there too. I have briefly talked with [~bbende] and [~joewitt] asking them questions on ways to bring custom python processors into production. One of the routes was through leveraging NiFi Registry in connection with Apache NiFi to streamline integration. For my use case, I would leverage the NiFi Processor Marketplace to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi Python Processors where I focused on my master thesis AI/DL for stroke diagnosis. *High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi Processor Marketplace App, there are frameworks we can leverage for the UI from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be used for full stack, but probably would mainly be used for the frontend portion) and other options. in the case that we want to make something more custom to display each category of processors. This full stack application could interact wth the NiFi Registry, which is the central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi (C++). What are some hosting options for Apache projects? > Create NiFi Processor Marketplace App Powered by NiFi
[jira] [Created] (NIFI-13301) Create NiFi Processor Marketplace App Powered by NiFi Registry
James Guzman (Medel) created NIFI-13301: --- Summary: Create NiFi Processor Marketplace App Powered by NiFi Registry Key: NIFI-13301 URL: https://issues.apache.org/jira/browse/NIFI-13301 Project: Apache NiFi Issue Type: New Feature Components: Extensions, NiFi Registry Environment: Linux: Ubuntu 20.04 or 22.04 Reporter: James Guzman (Medel) Assignee: James Guzman (Medel) {*}Description (NiFi){*}: Some people in the community have shown interest in having a NiFi Processor Marketplace where they can contribute their category of processors. In particular, I saw some people interested in contributing their NiFi python processors. This NiFi Processor marketplace could be also applied toward the community interested in contributing custom NiFi java processors. * {*}Extra MiNiFi C++{*}: We could even add a section for MiNiFi C++ custom processors. There is a part of the MiNiFi build process that brings in NiFi through building the JNI extension. So, that is a way to integrate NiFi Registry there too. I have briefly talked with [~bbende] and [~joewitt] asking them questions on ways to bring custom python processors into production. One of the routes was through leveraging NiFi Registry in connection with Apache NiFi to streamline integration. For my use case, I would leverage the NiFi Processor Marketplace to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi Python Processors where I focused on my master thesis AI/DL for stroke diagnosis. *High Level Details of NiFi Processor Marketplace App:* For a full stack NiFi Processor Marketplace App, there are frameworks we can leverage for the UI from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be used for full stack, but probably would mainly be used for the frontend portion) and other options. in the case that we want to make something more custom to display each category of processors. This full stack application could interact wth the NiFi Registry, which is the central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi (C++). What are some hosting options for Apache projects? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} # make sure to install conan2 for your environment sudo pip install --force-reinstall -v "conan==2.3.1" # conanfile.py is in root dir of MiNiFi C++ project cd $HOME/nifi-minifi-cpp # create MINIFI_HOME env variable for binary executable minifi export MINIFI_HOME=$(pwd) # In case you need to delete all conan packages # conan remove "*" -c # install conan packages for MiNiFi C++ using conanfile.py invoking Conan conan install . --build=missing --output-folder=build_conan -pr=$HOME/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # build MiNiFi C++ using conanfile.py invoking Conan & CMake conan build . --output-folder=build_conan -pr=$HOME/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # verify we can run minifi binary executable ./build_conan/bin/minifi{code} *UPDATE (May 21, 2024):* I created a small PR for conan integration on top of Gabor Gyimesi's PR-1749 "https://github.com/apache/nifi-minifi-cpp/pull/1749; that brings in conan packages for OpenSSL, LibCurl and Zlib as alternatives to their system library counterparts when we run standalone CMake to build MiNiFi CPP. I also included steps for Maton Szasz and others to get up to speed with how to use conan v2 to build MiNiFi CPP. On top of Gabor's PR-1749, my *PR-261 (MINIFICPP-2346-P1)* has +325 -225 git diff and only 5 files changed: [https://github.com/lordgamez/nifi-minifi-cpp/pull/261] *UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the refactoring of MiNiFi CPP build infra
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} # make sure to install conan2 for your environment sudo pip install --force-reinstall -v "conan==2.3.1" # conanfile.py is in root dir of MiNiFi C++ project cd $HOME/nifi-minifi-cpp # create MINIFI_HOME env variable for binary executable minifi export MINIFI_HOME=$(pwd) # In case you need to delete all conan packages # conan remove "*" -c # install conan packages for MiNiFi C++ using conanfile.py invoking Conan conan install . --build=missing --output-folder=build_conan -pr=$HOME/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # build MiNiFi C++ using conanfile.py invoking Conan & CMake conan build . --output-folder=build_conan -pr=$HOME/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # verify we can run minifi binary executable ./build_conan/bin/minifi{code} *UPDATE (May 21, 2024):* I created a small PR for conan integration on top of Gabor Gyimesi's PR-1749 "https://github.com/apache/nifi-minifi-cpp/pull/1749; that brings in conan packages for OpenSSL, LibCurl and Zlib as alternatives to their system library counterparts when we run standalone CMake to build MiNiFi CPP. I also included steps for Maton Szasz and others to get up to speed with how to use conan v2 to build MiNiFi CPP. On top of Gabor's PR-1749, my *PR-261* has +325 -225 git diff and only 5 files changed: [https://github.com/lordgamez/nifi-minifi-cpp/pull/261] *UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the refactoring of MiNiFi CPP build infra over to conan version
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *UPDATE (May 21, 2024):* I created a small PR for conan integration on top of Gabor Gyimesi's PR-1749 "https://github.com/apache/nifi-minifi-cpp/pull/1749; that brings in conan packages for OpenSSL, LibCurl and Zlib as alternatives to their system library counterparts when we run standalone CMake to build MiNiFi CPP. I also included steps for Maton Szasz and others to get up to speed with how to use conan v2 to build MiNiFi CPP. On top of Gabor's PR-1749, my *PR-261* has +325 -225 git diff and only 5 files changed: https://github.com/lordgamez/nifi-minifi-cpp/pull/261 *UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the refactoring of MiNiFi CPP build infra over to conan version 2 is taking longer than expected. The good news is that I believe I have more than half of
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} UPDATE (May 21, 2024): As *UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the refactoring of MiNiFi CPP build infra over to conan version 2 is taking longer than expected. The good news is that I believe I have more than half of MiNiFi external libraries now managed by conan with backward compatibility for standalone CMake approach. *UPDATE (May 13, 2024)* - *[22205c9|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/22205c98cef30ee1ebe252880ae113724589bd78]:* Fetch and Pulled latest upstream main branch of parent MiNiFi to my fork main branch of MiNiFi repo; thus, merged in the latest updates from main branch of parent MiNiFi to my MINIFICPP-2346 branch. Resolved conflicts between a few CMake files coming from Date.cmake and BundledRocksdb.cmake files. For
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Affects Version/s: 0.99.0 > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0, 0.99.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log, > minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log > > Time Spent: 1h 40m > Remaining Estimate: 0h > > *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain > point, I will pull in lastest updates from MiNiFi C++ main branch on parent > level and rebase or merge into my PR. > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > Here is my draft PR for this Jira Ticket for your reference: > [https://github.com/apache/nifi-minifi-cpp/pull/1775] > Here are the commands that I run: > > {code:java} > cd $HOME/src/pipeline/nifi-minifi-cpp > # NOTE: I also plan to add smoother integration of conan commands into MiNiFI > CPP too > # For instance, I know you guys have your bootstrap scripts too > # You also have your github actions workflow scripts too > conan install . --output-folder=build > -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux > # install conan packages and build C++ for minifi using Conan & CMake, we run > this way, so conan's tc.cache_variables > # are like passing -D{minifi_option} upon running cmake generate, then > make to build MiNiFi C++ > conan build . --output-folder=build >
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the refactoring of MiNiFi CPP build infra over to conan version 2 is taking longer than expected. The good news is that I believe I have more than half of MiNiFi external libraries now managed by conan with backward compatibility for standalone CMake approach. *UPDATE (May 13, 2024)* - *[22205c9|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/22205c98cef30ee1ebe252880ae113724589bd78]:* Fetch and Pulled latest upstream main branch of parent MiNiFi to my fork main branch of MiNiFi repo; thus, merged in the latest updates from main branch of parent MiNiFi to my MINIFICPP-2346 branch. Resolved conflicts between a few CMake files coming from Date.cmake and BundledRocksdb.cmake files. For standalone CMake build appraoch,
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *UPDATE (May 13, 2024)* - [59ffefb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/59ffefb1caf662874054f55f7171692c6b9d345f]: {*}Enabled LIBRDKAFKA & GRPC_FOR_LOKI MiNiFi C++ extension using Conan & CMake{*}, *Improved Running Python Maven Executable for JNI & SFTP Test Server for conan approach* and {*}Improved standalone CMake approach for building PcapPlusPlus with its dependency on LibPcap{*}. LIBRDKAFKA required creating a librdkafka conan package passing similar CMake definitions to it to build it like we did for standalone CMake approach. GRPC_FOR_LOKI required creating a grpc conan package that uses minifi's openssl and libsystemd conan packages. In both LIBRDKAFKA & GRPC_FOR_LOKI, I updated their .cmake Bundles to account for the condition of using their conan package,
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Due Date: 19/May/24 (was: 12/May/24) > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log, > minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log > > Time Spent: 1h 20m > Remaining Estimate: 0h > > *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain > point, I will pull in lastest updates from MiNiFi C++ main branch on parent > level and rebase or merge into my PR. > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > Here is my draft PR for this Jira Ticket for your reference: > [https://github.com/apache/nifi-minifi-cpp/pull/1775] > Here are the commands that I run: > > {code:java} > cd $HOME/src/pipeline/nifi-minifi-cpp > # NOTE: I also plan to add smoother integration of conan commands into MiNiFI > CPP too > # For instance, I know you guys have your bootstrap scripts too > # You also have your github actions workflow scripts too > conan install . --output-folder=build > -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux > # install conan packages and build C++ for minifi using Conan & CMake, we run > this way, so conan's tc.cache_variables > # are like passing -D{minifi_option} upon running cmake generate, then > make to build MiNiFi C++ > conan build . --output-folder=build >
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *UPDATE (May 11, 2024)* - [c96d430|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/c96d430814a067a6a538f4324337b719cb75bf48]: Refactoring MiNiFi C++ build infrastructure adding Conan Packager support *Enabled JNI, SENSORS, USB_CAMERA, OPENCV, and SFTP,* which required creating *Maven 3.9.6 conan package* updating the conan recipe to expose the Maven Executable environment variable in CMake and *installing prebuilt Java OpenJDK 21.0.2 conan package* and creating a python wrapper run_maven.py script for calling the conan maven Java build tool to successfully build Java SFTP Test Server for SFTP extension testing and maven to build {*}Apache NiFi 1.9.0 JNI Assembly and Framework{*}; We had to go the python maven wrapper script approach since I ran into some issues calling conan maven
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *UPDATE (May 11, 2024)* - [64dd8ad|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/64dd8ad14d2ddcea5eaa12da14a0e1b8a7289bb8]: {*}Enabled Build LUA SCRIPTING, ENCRYPT CONFIG, SPLUNK, ELASTIC SEARCH, TEST PROCESSORS, GRANFANA LOKI and CONTROLLER by Conan & CMake{*}. For building these components with conan, the LUA SCRIPTING extension required me updating its Lua.cmake and Sol2.cmake files, so we can use the conan packages in CMake when USE_CONAN_PACKAGE condition is true. For lua and sol2 conan packages, I was able to just install the prebuilt binary conan packages from conancenter directly since conancenter provided the versions that match with MiNiFi C++'s standalone CMake approach's lua 5.4.6 and sol2 3.3.0 source built versions. MiNiFi controller being enabled required updating
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *UPDATE (May 9, 2024)* - {*}[63290db|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/63290db57b60054ba8ed32c0b3a7d1f87f315314]{*}: *Refactored IODBC and SOCI libraries, so they are now CMake Bundles for easier integration with Conan.* Thus, minifi-sql library now calls include(BundledIODBC), use_bundled_iodbc(...) and include(BundledSOCI) and use_bundled_soci(...). Verified I can still build MiNiFi C++ using standalone CMake. 100% (220/220) of GTESTS passed using standalone CMake. Integrated conan for building SOCI & ODBC for MiNiFi SQL extension. *[4232399|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/4232399657c9da42dff57ff2b6806aeea2485444]* *Added a SOCI conan recipe to create SOCI conan package with unixODBC conan package, so SOCI comes with targets
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *UPDATE (May 9, 2024)* - [ca3d6df|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/ca3d6df318162e2ee9d3e76e173d9f83e17d3010]: {*}Enabled MiNiFi C++ extensions GPS and COAP{*}. I added conan recipe for libcoap keeping the windows cmake patches for minifi. Since the original libcoap conan recipe's conanfile.py on conan-center-index was tailored for CMake building libcoap versions > 4.3.0, I modified the recipe, so we can build libcoap 4.2.1, which didnt have CMake support yet and thus, I updated that conanfile.py to run 'autogen.sh' to create the configure script, I passed the configure args to the configure script and then I used conan's Autotools() to run the configure script and then ran the generated make file to build libcoap 4.2.1. I updated BundledLibCoAP to account for the
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *UPDATE (May 9, 2024)* - [ca3d6df|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/ca3d6df318162e2ee9d3e76e173d9f83e17d3010]: {*}Enabled MiNiFi C++ extensions GPS and COAP{*}. I added conan recipe for libcoap keeping the windows cmake patches for minifi. Since the original libcoap conan recipe's conanfile.py on conan-center-index was tailored for CMake building libcoap versions > 4.3.0, I modified the recipe, so we can build libcoap 4.2.1, which didnt have CMake support yet and thus, I updated that conanfile.py to run 'autogen.sh' to create the configure script, I passed the configure args to the configure script and then I used conan's Autotools() to run the configure script and then ran the generated make file to build libcoap 4.2.1. I updated BundledLibCoAP to account for the
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}*{color:#172b4d}(UPDATE (May 8, 2024) -{color}*{color} *{color:#172b4d}{color}[93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:* {color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added conan recipe for liblzma (xy_utils) and libarchive keeping the patches for minifi. One area I found an issue with was integrating openssl conan package with libarchive conan package and libarchive complained about not being able to find openssl::openssl target. I did try to solve the issue by updating the libarchive conan recipe, but wasn't able to solve that openssl integration issue. I think the problem comes from libarchive CMakeLists.txt
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}*{color:#172b4d}(UPDATE (May 8, 2024) -{color} [93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:* {color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added conan recipe for liblzma (xy_utils) and libarchive keeping the patches for minifi. One area I found an issue with was integrating openssl conan package with libarchive conan package and libarchive complained about not being able to find openssl::openssl target. I did try to solve the issue by updating the libarchive conan recipe, but wasn't able to solve that openssl integration issue. I think the problem comes from libarchive CMakeLists.txt and its integration with conan
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}*(UPDATE (May 8, 2024) - [93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:* {color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added conan recipe for liblzma (xy_utils) and libarchive keeping the patches for minifi. One area I found an issue with was integrating openssl conan package with libarchive conan package and libarchive complained about not being able to find openssl::openssl target. I did try to solve the issue by updating the libarchive conan recipe, but wasn't able to solve that openssl integration issue. I think the problem comes from libarchive CMakeLists.txt and its integration with conan in the repo. I opened
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}*(UPDATE (May 8, 2024) - [93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:* {color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added conan recipe for liblzma (xy_utils) and libarchive keeping the patches for minifi. One area I found an issue with was integrating openssl conan package with libarchive conan package and libarchive complained about not being able to find openssl::openssl target. I did try to solve the issue by updating the libarchive conan recipe, but wasn't able to solve that openssl integration issue. I think the problem comes from libarchive CMakeLists.txt and its integration with conan in the repo. I opened
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}*UPDATE (May 7, 2024)* - [2460e84|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2460e846832d693c991e1da632b86ba2d4e34516]: {color:#172b4d}*Enabled OPS and OPC* MiNiFi Options, so those *MiNiFi C++ extensions* are built using *Conan CMake* and their bundled CMake dependencies use their particular *{{open62541/1.3.3@minifi/dev}}* and {*}{{mbedtls/2.16.3@minifi/dev}} conan packages{*}. I had to fix the mbedtls conan package from conan center index since it wasnt compatible when open62541 tried to integrate with mbedtls. In the thirdparty folder, you will see on line 139 where I updated libembedtls to be libmbedtls and fixing that typo allowed open62541 conan package to be able to be created with mbedtls as the encryption. I plan to make a follow up PR on conan center
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *{color:#0747a6}UPDATE (May 5, 2024) - [446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:{color}*{color:#172b4d} When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs using conan
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors, I set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility with standalone CMake approach. I got the following error coming from extension robksdb-repos/database/RocksDbInstance.cpp on
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors, I set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility with standalone CMake approach. I got the following error coming from extension robksdb-repos/database/RocksDbInstance.cpp on
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors, I set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility with standalone CMake approach. I got the following error coming from extension robksdb-repos/database/RocksDbInstance.cpp on
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors, I set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility with standalone CMake approach. I got the following error coming from extension robksdb-repos/database/RocksDbInstance.cpp on
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {*}{color:#0747a6}UPDATE (May 4, 2024) - {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}: Verified I can build MiNiFi C++ with standalone CMake like how we currently do it even after integrating conan into the build infrastructure. CMake built *MiNiFi Core, Main* and *Standard Processors* successfully. Also double checked that conan build MiNiFi still works. (Feel free to reference the logs for
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {*}{color:#0747a6}UPDATE (May 4, 2024) - {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}: Verified I can build MiNiFi C++ with standalone CMake like how we currently do it even after integrating conan into the build infrastructure. CMake built *MiNiFi Core, Main* and *Standard Processors* successfully. Also double checked that conan build MiNiFi still works. (Feel free to reference the logs for
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {*}{color:#0747a6}UPDATE (May 4, 2024) - {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}: Verified I can build MiNiFi C++ with standalone CMake like how we currently do it even after integrating conan into the build infrastructure. CMake built MiNiFi Core, Main and Standard Processors successfully. Also double checked that conan build MiNiFi still works. (Feel free to reference the logs for the
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Attachment: minifi_cpp_cmake_standalone_build_7eb8beb.log > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log, > minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log > > Time Spent: 1h > Remaining Estimate: 0h > > *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain > point, I will pull in lastest updates from MiNiFi C++ main branch on parent > level and rebase or merge into my PR. > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > Here is my draft PR for this Jira Ticket for your reference: > [https://github.com/apache/nifi-minifi-cpp/pull/1775] > * I have added *abseil* external lib dependency to conanfile.py and > integrated it into Abseil.cmake with conditional statement that checks if > USE_CONAN_PACKAGER minifi cpp option is true else checks if > USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can > began the process of integrating in conan version 2. > > Here are the commands that I run: > > {code:java} > cd $HOME/src/pipeline/nifi-minifi-cpp > # NOTE: I also plan to add smoother integration of conan commands into MiNiFI > CPP too > # For instance, I know you guys have your bootstrap scripts too > # You also have your github actions workflow scripts too > conan install .
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {*}{color:#0747a6}UPDATE (May 4, 2024) - {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}: Verified I can build MiNiFi C++ with standalone CMake like how we currently do it even after integrating conan into the build infrastructure. CMake built MiNiFi Core, Main and Standard Processors successfully. Also double checked that conan build MiNiFi still works. {color:#0747a6}{*}UPDATE (May 4, 2024)
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Attachment: minifi_cpp_conan_build_7eb8beb.log > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log, > minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log > > Time Spent: 1h > Remaining Estimate: 0h > > *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain > point, I will pull in lastest updates from MiNiFi C++ main branch on parent > level and rebase or merge into my PR. > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > Here is my draft PR for this Jira Ticket for your reference: > [https://github.com/apache/nifi-minifi-cpp/pull/1775] > * I have added *abseil* external lib dependency to conanfile.py and > integrated it into Abseil.cmake with conditional statement that checks if > USE_CONAN_PACKAGER minifi cpp option is true else checks if > USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can > began the process of integrating in conan version 2. > > Here are the commands that I run: > > {code:java} > cd $HOME/src/pipeline/nifi-minifi-cpp > # NOTE: I also plan to add smoother integration of conan commands into MiNiFI > CPP too > # For instance, I know you guys have your bootstrap scripts too > # You also have your github actions workflow scripts too > conan install .
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Attachment: minifi_cpp_conan_build_2d1b884.log > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > Attachments: minifi_cpp_conan_build_2d1b884.log > > Time Spent: 1h > Remaining Estimate: 0h > > *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain > point, I will pull in lastest updates from MiNiFi C++ main branch on parent > level and rebase or merge into my PR. > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > Here is my draft PR for this Jira Ticket for your reference: > [https://github.com/apache/nifi-minifi-cpp/pull/1775] > * I have added *abseil* external lib dependency to conanfile.py and > integrated it into Abseil.cmake with conditional statement that checks if > USE_CONAN_PACKAGER minifi cpp option is true else checks if > USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can > began the process of integrating in conan version 2. > > Here are the commands that I run: > > {code:java} > cd $HOME/src/pipeline/nifi-minifi-cpp > # NOTE: I also plan to add smoother integration of conan commands into MiNiFI > CPP too > # For instance, I know you guys have your bootstrap scripts too > # You also have your github actions workflow scripts too > conan install . --output-folder=build > -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux > #
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}{*}UPDATE (May 4, 2024) - [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color} Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi* and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*} extension with minimal set of external libraries needed. If you look at our conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a tuple
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}{*}UPDATE (May 4, 2024){*}:{color} Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi* and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*} extension with minimal set of external libraries needed. If you look at our conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a tuple of the minimal external libraries needed to build these two essential building blocks of MiNFi CPP. Next when you
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}{*}UPDATE (May 4, 2024){*}:{color} Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi* and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*} extension with minimal set of external libraries needed. If you look at our conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a list of the minimal external libraries needed to build these two essential building blocks of MiNFi CPP. Next when you
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} Tracking Prebuilt Binary Conan Packages Available on Conancenter: * 'abseil/20230125.3', * 'argparse/3.0', * 'asio/1.30.2' Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects: * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan packages
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} Tracking Prebuilt Binary Conan Packages Available on Conancenter: * 'abseil/20230125.3', * 'argparse/3.0', * 'asio/1.30.2' Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects: * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan packages
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux cd build/ cmake .. make -j $(nproc){code} Tracking Prebuilt Binary Conan Packages Available on Conancenter: * 'abseil/20230125.3', * 'argparse/3.0', * 'asio/1.30.2' Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects: * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan packages maintained by conan team * Bincrafters Bintray Repo: community-driven repo that hosts large number of pre-built conan packages for open-source libraries for free * Artifactory by JFrog: typically used as a commercial hosting service, but also provides a free tier for open-source projects. *
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux cd build/ cmake .. make -j $(nproc){code} Tracking Prebuilt Binary Conan Packages Available on Conancenter: * 'abseil/20230125.3', * 'argparse/3.0', * 'asio/1.30.2' Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects: * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan packages maintained by conan team * Bincrafters Bintray Repo: community-driven repo that hosts large number of pre-built conan packages for open-source libraries for free * Artifactory by JFrog: typically used as a commercial hosting service, but also provides a free tier for open-source projects. *
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux cd build/ cmake .. make -j $(nproc){code} Tracking Prebuilt Binary Conan Packages Available on Conancenter: * 'abseil/20230125.3', * 'argparse/3.0', * 'asio/1.30.2' Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects: * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan packages maintained by conan team * Bincrafters Bintray Repo: community-driven repo that hosts large number of pre-built conan packages for open-source libraries for free * Artifactory by JFrog: typically used as a commercial hosting service, but also provides a free tier for open-source projects. * {*}Self-Hosting{*}: While there are multiple options, I will focus on *GitHub Packages* as its a free package hosting service, which can be used to host our own conan package repos, allowing
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux cd build/ cmake .. make -j $(nproc){code} Tracking Prebuilt Binary Conan Packages Available on Conancenter: * 'abseil/20230125.3', * 'argparse/3.0', * 'asio/1.30.2' Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects: * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan packages maintained by conan team * Bincrafters Bintray Repo: community-driven repo that hosts large number of pre-built conan packages for open-source libraries for free * Artifactory by JFrog: typically used as a commercial hosting service, but also provides a free tier for open-source projects. * {*}Self-Hosting{*}: While there are multiple options, I will focus on *GitHub Packages* as its a free package hosting service, which can be used to host our own conan package repos, allowing
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux cd build/ cmake .. make -j $(nproc){code} was: *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows.
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Guzman (Medel) updated MINIFICPP-2346: Description: *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. was: *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these
[jira] [Comment Edited] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842449#comment-17842449 ] James Guzman (Medel) edited comment on MINIFICPP-2346 at 4/30/24 6:44 PM: -- I think it may be beneficial to leverage conan at the beginning for installing MiNiFi C++ external lib dependencies (conan supports Linux and Windows packages). For my medical robotics company, I created an ezconan python wrapper tool around conan for speeding up our C++ cmake build infrastructure for our medical robot. I can build that code on Linux and Windows with no issues and the rest of the software team and our other engineering teams use our build infrastructure. I wanted to try adding support for conan to the MiNiFi C++ project too since I am looking at using MiNiFi C++ to develop edge data pipelines programmatically as alternative to the yml approach. As for the vcpkg, CMake can generate those build files when we run CMake generate on Windows. Similar for Makefiles, where we run CMake generate on Linux and CMake generates them too. I will follow up on this thread with updates to my progress on conan for speeding up MiNiFi C++ builds [~szaszm] . I will keep you posted. I am also considering creating a MiNiFI C++ conan package that I can smoothly integrate MiNiFi C++ lib into other C++ projects like in my medical imaging robot. was (Author: james.medel): I think it may be beneficial to leverage conan at the beginning for installing MiNiFi C++ external lib dependencies (conan supports Linux and Windows packages). For my medical robotics company, I created an ezconan python wrapper tool around conan for speeding up our C++ cmake build infrastructure for our medical robot. I can build that code on Linux and Windows. I wanted to try adding support for conan to the MiNiFi C++ project too since I am looking at using MiNiFi C++ to develop edge data pipelines programmatically as alternative to the yml approach. As for the vcpkg, CMake can generate those build files when we run CMake generate on Windows. Similar for Makefiles, where we run CMake generate on Linux and CMake generates them too. I will follow up on this thread with updates to my progress on conan for speeding up MiNiFi C++ builds [~szaszm] . I will keep you posted. I am also considering creating a MiNiFI C++ conan package that I can smoothly integrate MiNiFi C++ lib into other C++ projects like in my medical imaging robot. > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container)
[jira] [Comment Edited] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842451#comment-17842451 ] James Guzman (Medel) edited comment on MINIFICPP-2346 at 4/30/24 5:07 PM: -- [~szaszm] Does MiNiFi C++ have a jfrog repo where we can host conan repos publicly (alternative to conancenter)? In case I need to build conan packages for MiNiFi C++ external lib dependencies and upload them as prebuilt binary conan packages to jfrog. We could create a jfrog Windows repo for MiNiFi CPP's external libs built for Windows and a jfrog Linux repo for MiNiFi CPP's external libs built for Linux. Here is Jfrog: [https://jfrog.com/] was (Author: james.medel): [~szaszm] Does MiNiFi C++ have a jfrog repo where we can host conan repos (alternative to conancenter)? In case I need to build conan packages for MiNiFi C++ external lib dependencies and upload them as prebuilt binary conan packages to jfrog. We could create a jfrog Windows repo for MiNiFi CPP's external libs built for Windows and a jfrog Linux repo for MiNiFi CPP's external libs built for Linux. Here is Jfrog: [https://jfrog.com/] > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842451#comment-17842451 ] James Guzman (Medel) edited comment on MINIFICPP-2346 at 4/30/24 5:04 PM: -- [~szaszm] Does MiNiFi C++ have a jfrog repo where we can host conan repos (alternative to conancenter)? In case I need to build conan packages for MiNiFi C++ external lib dependencies and upload them as prebuilt binary conan packages to jfrog. We could create a jfrog Windows repo for MiNiFi CPP's external libs built for Windows and a jfrog Linux repo for MiNiFi CPP's external libs built for Linux. Here is Jfrog: [https://jfrog.com/] was (Author: james.medel): [~szaszm] Does MiNiFi C++ have a jfrog repo where we can host conan repos (alternative to conancenter)? In case I need to build conan packages for MiNiFi C++ external lib dependencies and upload them as prebuilt binary conan packages to jfrog. We could create a jfrog Windows repo for MiNiFi C++'s external libs built for Windows and a jfrog Linux repo for MiNiFi C++'s external libs built for Linux. Here is Jfrog: https://jfrog.com/ > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842451#comment-17842451 ] James Guzman (Medel) commented on MINIFICPP-2346: - [~szaszm] Does MiNiFi C++ have a jfrog repo where we can host conan repos (alternative to conancenter)? In case I need to build conan packages for MiNiFi C++ external lib dependencies and upload them as prebuilt binary conan packages to jfrog. We could create a jfrog Windows repo for MiNiFi C++'s external libs built for Windows and a jfrog Linux repo for MiNiFi C++'s external libs built for Linux. Here is Jfrog: https://jfrog.com/ > Speed up MiNiFi Build with Conan package manager (C++) > -- > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 >Reporter: James Guzman (Medel) >Assignee: James Guzman (Medel) >Priority: Minor > Labels: build, conan, ubuntu > > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)