[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
--------------------------------------------
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
Tracking Prebuilt Binary Conan Packages Available on Conancenter:
* 'abseil/20230125.3',
* 'argparse/3.0',
* 'asio/1.30.2'
Tracking Conan Packages Not Available on Conancenter, so we need to use an
alternative hosting service to manage building package sources into prebuilt
binary conan packages. Here are some of the free conan hosting services for
open-source C/C++ projects:
* {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
packages maintained by conan team
* Bincrafters Bintray Repo: community-driven repo that hosts large number of
pre-built conan packages for open-source libraries for free
* Artifactory by JFrog: typically used as a commercial hosting service, but
also provides a free tier for open-source projects.
* {*}Self-Hosting{*}: While there are multiple options, I will focus on
*GitHub Packages* as its a free package hosting service, which can be used to
host our own conan package repos, allowing us to store and distribute our
pre-built conan packages without having to use 3rd party services like
Bincrafters or pay for commercial solution.
** {*}Advantage with GitHub Packages{*}: its free, integrated with our
existing GitHub workflow, allows us full control hosting our conan packages
without relying on 3rd party services, useful if we need to have specific
requirements or need flexibility in managing our package hosting.
Tracking Non Prebuilt Binary Conan Packages, which later I will create a
*GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account for
Linux, Windows, Mac, etc):
* bustache: [https://github.com/jamboree/bustache]
{*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
after I finish my work for MINFICPP-2346: While I am working on migrating
MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
PR we could do the following:
# Put the external libraries that MiNiFi CPP Core depends on into its own
*BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other names),
we can run this CMake code before adding *add_subdirectory(libminifi)*
## External libs that libminifi needs that are in MiNiFi CPP's root
CMakeLists.txt (refactor for better organization):
yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
magic_enum, RangeV3, Asio, MagicEnum
## We can also clean up the *thirdparty* folder as we migrate to conan
packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep it
MiNiFi C{+}+ code and its extension processors, etc.
# I tried building MiNiFi CPP Core just with just *minifi-standard-processors*
and the other extensions disabled, but it seems even some of MiNiFi CPP Core
depends on some of the extensions to be built with it, so it doesn't fail
building, such as Openwsman. If we don't have ENABLE_OPENWSMAN, then we get
this error when building MiNiFi CPP Core, so we should look having a build
config to build minimal MiNiFi CPP Core:
{code:java}
libminifi/src/controllers/SSLContextService.cpp:520:80: error:
‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
was:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
Tracking Prebuilt Binary Conan Packages Available on Conancenter:
* 'abseil/20230125.3',
* 'argparse/3.0',
* 'asio/1.30.2'
Tracking Conan Packages Not Available on Conancenter, so we need to use an
alternative hosting service to manage building package sources into prebuilt
binary conan packages. Here are some of the free conan hosting services for
open-source C/C++ projects:
* {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
packages maintained by conan team
* Bincrafters Bintray Repo: community-driven repo that hosts large number of
pre-built conan packages for open-source libraries for free
* Artifactory by JFrog: typically used as a commercial hosting service, but
also provides a free tier for open-source projects.
* {*}Self-Hosting{*}: While there are multiple options, I will focus on
*GitHub Packages* as its a free package hosting service, which can be used to
host our own conan package repos, allowing us to store and distribute our
pre-built conan packages without having to use 3rd party services like
Bincrafters or pay for commercial solution.
** {*}Advantage with GitHub Packages{*}: its free, integrated with our
existing GitHub workflow, allows us full control hosting our conan packages
without relying on 3rd party services, useful if we need to have specific
requirements or need flexibility in managing our package hosting.
Tracking Non Prebuilt Binary Conan Packages, which later I will create a
*GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account for
Linux, Windows, Mac, etc):
* bustache: [https://github.com/jamboree/bustache]
Some more build infra we can improve in MiNiFi CPP after I finish my work for
MINFICPP-2346: While I am working on migrating MiNiFi CPP build infrastructure
toward leveraging conan packager, for a later PR we could do the following:
# Put the external libraries that MiNiFi CPP Core depends on into its own
*BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other names),
we can run this CMake code before adding *add_subdirectory(libminifi)*
## External libs that libminifi needs that are in MiNiFi CPP's root
CMakeLists.txt (refactor for better organization):
yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
magic_enum, RangeV3, Asio, MagicEnum
## We can also clean up the *thirdparty* folder as we migrate to conan
packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep it
MiNiFi C{+}+ code and its extension processors, etc.
> Speed up MiNiFi Build with Conan package manager (C++)
> ------------------------------------------------------
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Affects Versions: 0.15.0
> Environment: Ubuntu 22.04
> Reporter: James Guzman (Medel)
> Assignee: James Guzman (Medel)
> Priority: Minor
> Labels: build, conan, ubuntu
> Time Spent: 1h
> Remaining Estimate: 0h
>
> *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
> point, I will pull in lastest updates from MiNiFi C++ main branch on parent
> level and rebase or merge into my PR.
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
> During the the build process, it clones all the external dependencies and
> then builds those external dependencies while its building MiNiFi CPP.
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail
> to download or fail to build. It would be faster to build MiNiFi C++ with the
> external dependencies already preinstalled.
>
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
> enable support for using *conan install* to install all or most of MiNiFi's
> external dependencies, using *conan install --build=missing* to build them as
> prebuilt binary conan packages, upload them to conancenter with {*}conan
> upload{*}, and then run cmake generate to generate the appropriate build
> files for the OS, then run make to build MiNiFi CPP. Also conan has really
> good support for cross platform compatibility for Linux and Windows. At this
> point because we already have most of MiNiFi CPP's external lib dependencies
> installed with conan, our CMake build will just focus on building the MiNiFi
> CPP code, enabling faster builds. Also I will need to account for backward
> compatibility support with the previous way of installing MiNiFi CPP external
> lib dependencies using CMake FetchContent_Declare(...) approach until
> conanfile.py approach covers installing all these dependencies. A valid
> verification of the MiNiFi C++ build would be to run GTESTs and if all the
> expected test cases pass, then it may be fair to conclude we would be ready
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
> script(s)):*
> # Install conan version 2 into MiNiFi C++ dev environment (my preference is
> using docker container)
> # Create a conanfile.py file at root of MiNiFi C++ project where we run
> "conan install" or "conan install --build=missing" to install MiNiFi C++
> external lib dependencies first. Then we run CMake to build MiNiFi C++.
> # Find all MiNiFi C++ find_package(...) CMake function calls and check if
> there is already supported conan packages on the conancenter, so we can
> install those packages.
> # Find all FetchContent_Declare(...) CMake function calls and check if there
> is already supported conan packages on the conancenter, so we can install
> those packages.
> # At this point we will have used conan version 2 C++ package manager to
> install almost all MiNiFi C++'s external dependencies.
> ## With Conan, we will have installed these MiNiFi external lib dependencies
> as prebuilt binary conan packages.
> ## Or we will have instructed conan to first build each C++ external lib
> dependency, upload each lib to conancenter.
> # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so
> we can then run CMake to build MiNiFi faster.
>
> Here is my draft PR for this Jira Ticket for your reference:
> [https://github.com/apache/nifi-minifi-cpp/pull/1775]
> * I have added *abseil* external lib dependency to conanfile.py and
> integrated it into Abseil.cmake with conditional statement that checks if
> USE_CONAN_PACKAGER minifi cpp option is true else checks if
> USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
> began the process of integrating in conan version 2.
>
> Here are the commands that I run:
>
> {code:java}
> cd $HOME/src/pipeline/nifi-minifi-cpp
> # NOTE: I also plan to add smoother integration of conan commands into MiNiFI
> CPP too
> # For instance, I know you guys have your bootstrap scripts too
> # You also have your github actions workflow scripts too
> conan install . --output-folder=build
> -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
> # install conan packages and build C++ for minifi using Conan & CMake, we run
> this way, so conan's tc.cache_variables
> # are like passing -D{minifi_option} upon running cmake generate, then
> make to build MiNiFi C++
> conan build . --output-folder=build
> -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
>
> Tracking Prebuilt Binary Conan Packages Available on Conancenter:
> * 'abseil/20230125.3',
> * 'argparse/3.0',
> * 'asio/1.30.2'
> Tracking Conan Packages Not Available on Conancenter, so we need to use an
> alternative hosting service to manage building package sources into prebuilt
> binary conan packages. Here are some of the free conan hosting services for
> open-source C/C++ projects:
> * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
> packages maintained by conan team
> * Bincrafters Bintray Repo: community-driven repo that hosts large number of
> pre-built conan packages for open-source libraries for free
> * Artifactory by JFrog: typically used as a commercial hosting service, but
> also provides a free tier for open-source projects.
> * {*}Self-Hosting{*}: While there are multiple options, I will focus on
> *GitHub Packages* as its a free package hosting service, which can be used to
> host our own conan package repos, allowing us to store and distribute our
> pre-built conan packages without having to use 3rd party services like
> Bincrafters or pay for commercial solution.
> ** {*}Advantage with GitHub Packages{*}: its free, integrated with our
> existing GitHub workflow, allows us full control hosting our conan packages
> without relying on 3rd party services, useful if we need to have specific
> requirements or need flexibility in managing our package hosting.
> Tracking Non Prebuilt Binary Conan Packages, which later I will create a
> *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account
> for Linux, Windows, Mac, etc):
> * bustache: [https://github.com/jamboree/bustache]
>
> {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
> after I finish my work for MINFICPP-2346: While I am working on migrating
> MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
> PR we could do the following:
> # Put the external libraries that MiNiFi CPP Core depends on into its own
> *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other
> names), we can run this CMake code before adding *add_subdirectory(libminifi)*
> ## External libs that libminifi needs that are in MiNiFi CPP's root
> CMakeLists.txt (refactor for better organization):
> yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
> gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
> magic_enum, RangeV3, Asio, MagicEnum
>
> ## We can also clean up the *thirdparty* folder as we migrate to conan
> packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep
> it MiNiFi C{+}+ code and its extension processors, etc.
> # I tried building MiNiFi CPP Core just with just
> *minifi-standard-processors* and the other extensions disabled, but it seems
> even some of MiNiFi CPP Core depends on some of the extensions to be built
> with it, so it doesn't fail building, such as Openwsman. If we don't have
> ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we
> should look having a build config to build minimal MiNiFi CPP Core:
> {code:java}
> libminifi/src/controllers/SSLContextService.cpp:520:80: error:
> ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
