[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
--------------------------------------------
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
{color:#0747a6}{*}UPDATE (May 4, 2024) -
[2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color}
Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi*
and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*}
extension with minimal set of external libraries needed. If you look at our
conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a tuple
of the minimal external libraries needed to build these two essential building
blocks of MiNFi CPP. Next when you look at conanfile.py *MiNiFiCppMain* class's
generate(self) method, you'll see I have enabled USE_CONAN_PACKAGER to ON,
SKIP_TESTS to ON, and then the essential external library dependencies that
aren needed for the *core-minifi* library and *minifiexe* binary executable,
which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, ENABLE_CIVET, ENABLE_CURL{*}. I
notice most of these external libraries are in the extensions folder and add
some dependencies on these custom extensions that are created.
{color:#ff8b00}*I think we should consider creating an extensions subfolder for
core-minifi related extensions.*{color}
Here is the output I get after building the MiNiFi CPP code, you can see MiNiFi
CPP libraries and binary executables built successfully using *CONAN version 2*
and CMake (feel free to check the attached log for the full conan build
output): [~szaszm]
{code:java}
Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list
Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT list
Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT
-- Creating API Documentation using /usr/bin/doxygen
-- Configuring done
-- Generating done
-- Build files have been written to:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py
(minifi-cpp-main/0.15.0): Running CMake.build()
conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build
"/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20
[ 3%] Built target ossp-uuid-external-build
[ 6%] Built target openwsman-external
[ 7%] Built target ossp-uuid-external
Consolidate compiler generated dependencies of target core-minifi
[ 8%] Building CXX object
libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o
[ 8%] Linking CXX shared library ../bin/libcore-minifi.so
[ 78%] Built target core-minifi
Consolidate compiler generated dependencies of target minifi-civet-extensions
Consolidate compiler generated dependencies of target minifi-openwsman
Consolidate compiler generated dependencies of target minifi-http-curl
Consolidate compiler generated dependencies of target minifiexe
[ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so
[ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so
Consolidate compiler generated dependencies of target minifi-standard-processors
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o
[ 80%] Building CXX object
minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o
[ 81%] Building CXX object
minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o
[ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so
[ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so
[ 84%] Built target minifi-civet-extensions
[ 86%] Built target minifi-http-curl
[ 87%] Built target minifi-openwsman
[100%] Built target minifi-standard-processors
[100%] Linking CXX executable ../bin/minifi
[100%] Built target minifiexe {code}
NEXT: I will verify I can build these MiNiFi CPP libraries and binary
executables using USE_CMAKE_FETCH_CONTENT option set to ON and
USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
Tracking Prebuilt Binary Conan Packages Available on Conancenter:
* 'abseil/20230125.3',
* 'argparse/3.0',
* 'asio/1.30.2'
Tracking Conan Packages Not Available on Conancenter, so we need to use an
alternative hosting service to manage building package sources into prebuilt
binary conan packages. Here are some of the free conan hosting services for
open-source C/C++ projects:
* {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
packages maintained by conan team
* Bincrafters Bintray Repo: community-driven repo that hosts large number of
pre-built conan packages for open-source libraries for free
* Artifactory by JFrog: typically used as a commercial hosting service, but
also provides a free tier for open-source projects.
* {*}Self-Hosting{*}: While there are multiple options, I will focus on
*GitHub Packages* as its a free package hosting service, which can be used to
host our own conan package repos, allowing us to store and distribute our
pre-built conan packages without having to use 3rd party services like
Bincrafters or pay for commercial solution.
** {*}Advantage with GitHub Packages{*}: its free, integrated with our
existing GitHub workflow, allows us full control hosting our conan packages
without relying on 3rd party services, useful if we need to have specific
requirements or need flexibility in managing our package hosting.
Tracking Non Prebuilt Binary Conan Packages, which later I will create a
*GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account for
Linux, Windows, Mac, etc):
* bustache: [https://github.com/jamboree/bustache]
{*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
after I finish my work for MINFICPP-2346: While I am working on migrating
MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
PR we could do the following:
# Put the external libraries that MiNiFi CPP Core depends on into its own
*BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other names),
we can run this CMake code before adding *add_subdirectory(libminifi)*
## External libs that libminifi needs that are in MiNiFi CPP's root
CMakeLists.txt (refactor for better organization):
yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
magic_enum, RangeV3, Asio, MagicEnum
## We can also clean up the *thirdparty* folder as we migrate to conan
packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep it
MiNiFi C{+}+ code and its extension processors, etc.
# I tried building MiNiFi CPP Core just with just *minifi-standard-processors*
and the other extensions disabled, but it seems even some of MiNiFi CPP Core
depends on some of the extensions to be built with it, so it doesn't fail
building, such as Openwsman. If we don't have ENABLE_OPENWSMAN, then we get
this error when building MiNiFi CPP Core, so we should look having a build
config to build minimal MiNiFi CPP Core:
{code:java}
libminifi/src/controllers/SSLContextService.cpp:520:80: error:
‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
was:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
{color:#0747a6}{*}UPDATE (May 4, 2024){*}:{color} Updated MiNiFi CPP code with
Conan & CMake to build libMiNiFi *core-minifi* and MiNiFi Main's *minifiexe*
and {*}minifi-{*}{*}standard-processors{*} extension with minimal set of
external libraries needed. If you look at our conanfile.py, you'll see
{*}minifi_core_external_libraries{*}, which is a tuple of the minimal external
libraries needed to build these two essential building blocks of MiNFi CPP.
Next when you look at conanfile.py *MiNiFiCppMain* class's generate(self)
method, you'll see I have enabled USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON,
and then the essential external library dependencies that aren needed for the
*core-minifi* library and *minifiexe* binary executable, which are
{*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, ENABLE_CIVET, ENABLE_CURL{*}. I notice
most of these external libraries are in the extensions folder and add some
dependencies on these custom extensions that are created. {color:#ff8b00}*I
think we should consider creating an extensions subfolder for core-minifi
related extensions.*{color}
Here is the output I get after building the MiNiFi CPP code, you can see MiNiFi
CPP libraries and binary executables built successfully using *CONAN version 2*
and CMake (feel free to check the attached log for the full conan build
output): [~szaszm]
{code:java}
Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list
Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT list
Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT
-- Creating API Documentation using /usr/bin/doxygen
-- Configuring done
-- Generating done
-- Build files have been written to:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py
(minifi-cpp-main/0.15.0): Running CMake.build()
conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build
"/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20
[ 3%] Built target ossp-uuid-external-build
[ 6%] Built target openwsman-external
[ 7%] Built target ossp-uuid-external
Consolidate compiler generated dependencies of target core-minifi
[ 8%] Building CXX object
libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o
[ 8%] Linking CXX shared library ../bin/libcore-minifi.so
[ 78%] Built target core-minifi
Consolidate compiler generated dependencies of target minifi-civet-extensions
Consolidate compiler generated dependencies of target minifi-openwsman
Consolidate compiler generated dependencies of target minifi-http-curl
Consolidate compiler generated dependencies of target minifiexe
[ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so
[ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so
Consolidate compiler generated dependencies of target minifi-standard-processors
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o
[ 80%] Building CXX object
minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o
[ 81%] Building CXX object
minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o
[ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so
[ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so
[ 84%] Built target minifi-civet-extensions
[ 86%] Built target minifi-http-curl
[ 87%] Built target minifi-openwsman
[100%] Built target minifi-standard-processors
[100%] Linking CXX executable ../bin/minifi
[100%] Built target minifiexe {code}
NEXT: I will verify I can build these MiNiFi CPP libraries and binary
executables using USE_CMAKE_FETCH_CONTENT option set to ON and
USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
Tracking Prebuilt Binary Conan Packages Available on Conancenter:
* 'abseil/20230125.3',
* 'argparse/3.0',
* 'asio/1.30.2'
Tracking Conan Packages Not Available on Conancenter, so we need to use an
alternative hosting service to manage building package sources into prebuilt
binary conan packages. Here are some of the free conan hosting services for
open-source C/C++ projects:
* {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
packages maintained by conan team
* Bincrafters Bintray Repo: community-driven repo that hosts large number of
pre-built conan packages for open-source libraries for free
* Artifactory by JFrog: typically used as a commercial hosting service, but
also provides a free tier for open-source projects.
* {*}Self-Hosting{*}: While there are multiple options, I will focus on
*GitHub Packages* as its a free package hosting service, which can be used to
host our own conan package repos, allowing us to store and distribute our
pre-built conan packages without having to use 3rd party services like
Bincrafters or pay for commercial solution.
** {*}Advantage with GitHub Packages{*}: its free, integrated with our
existing GitHub workflow, allows us full control hosting our conan packages
without relying on 3rd party services, useful if we need to have specific
requirements or need flexibility in managing our package hosting.
Tracking Non Prebuilt Binary Conan Packages, which later I will create a
*GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account for
Linux, Windows, Mac, etc):
* bustache: [https://github.com/jamboree/bustache]
{*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
after I finish my work for MINFICPP-2346: While I am working on migrating
MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
PR we could do the following:
# Put the external libraries that MiNiFi CPP Core depends on into its own
*BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other names),
we can run this CMake code before adding *add_subdirectory(libminifi)*
## External libs that libminifi needs that are in MiNiFi CPP's root
CMakeLists.txt (refactor for better organization):
yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
magic_enum, RangeV3, Asio, MagicEnum
## We can also clean up the *thirdparty* folder as we migrate to conan
packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep it
MiNiFi C{+}+ code and its extension processors, etc.
# I tried building MiNiFi CPP Core just with just *minifi-standard-processors*
and the other extensions disabled, but it seems even some of MiNiFi CPP Core
depends on some of the extensions to be built with it, so it doesn't fail
building, such as Openwsman. If we don't have ENABLE_OPENWSMAN, then we get
this error when building MiNiFi CPP Core, so we should look having a build
config to build minimal MiNiFi CPP Core:
{code:java}
libminifi/src/controllers/SSLContextService.cpp:520:80: error:
‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
> Speed up MiNiFi Build with Conan package manager (C++)
> ------------------------------------------------------
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Affects Versions: 0.15.0
> Environment: Ubuntu 22.04
> Reporter: James Guzman (Medel)
> Assignee: James Guzman (Medel)
> Priority: Minor
> Labels: build, conan, ubuntu
> Time Spent: 1h
> Remaining Estimate: 0h
>
> *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
> point, I will pull in lastest updates from MiNiFi C++ main branch on parent
> level and rebase or merge into my PR.
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
> During the the build process, it clones all the external dependencies and
> then builds those external dependencies while its building MiNiFi CPP.
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail
> to download or fail to build. It would be faster to build MiNiFi C++ with the
> external dependencies already preinstalled.
>
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
> enable support for using *conan install* to install all or most of MiNiFi's
> external dependencies, using *conan install --build=missing* to build them as
> prebuilt binary conan packages, upload them to conancenter with {*}conan
> upload{*}, and then run cmake generate to generate the appropriate build
> files for the OS, then run make to build MiNiFi CPP. Also conan has really
> good support for cross platform compatibility for Linux and Windows. At this
> point because we already have most of MiNiFi CPP's external lib dependencies
> installed with conan, our CMake build will just focus on building the MiNiFi
> CPP code, enabling faster builds. Also I will need to account for backward
> compatibility support with the previous way of installing MiNiFi CPP external
> lib dependencies using CMake FetchContent_Declare(...) approach until
> conanfile.py approach covers installing all these dependencies. A valid
> verification of the MiNiFi C++ build would be to run GTESTs and if all the
> expected test cases pass, then it may be fair to conclude we would be ready
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
> script(s)):*
> # Install conan version 2 into MiNiFi C++ dev environment (my preference is
> using docker container)
> # Create a conanfile.py file at root of MiNiFi C++ project where we run
> "conan install" or "conan install --build=missing" to install MiNiFi C++
> external lib dependencies first. Then we run CMake to build MiNiFi C++.
> # Find all MiNiFi C++ find_package(...) CMake function calls and check if
> there is already supported conan packages on the conancenter, so we can
> install those packages.
> # Find all FetchContent_Declare(...) CMake function calls and check if there
> is already supported conan packages on the conancenter, so we can install
> those packages.
> # At this point we will have used conan version 2 C++ package manager to
> install almost all MiNiFi C++'s external dependencies.
> ## With Conan, we will have installed these MiNiFi external lib dependencies
> as prebuilt binary conan packages.
> ## Or we will have instructed conan to first build each C++ external lib
> dependency, upload each lib to conancenter.
> # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so
> we can then run CMake to build MiNiFi faster.
>
> Here is my draft PR for this Jira Ticket for your reference:
> [https://github.com/apache/nifi-minifi-cpp/pull/1775]
> * I have added *abseil* external lib dependency to conanfile.py and
> integrated it into Abseil.cmake with conditional statement that checks if
> USE_CONAN_PACKAGER minifi cpp option is true else checks if
> USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
> began the process of integrating in conan version 2.
>
> Here are the commands that I run:
>
> {code:java}
> cd $HOME/src/pipeline/nifi-minifi-cpp
> # NOTE: I also plan to add smoother integration of conan commands into MiNiFI
> CPP too
> # For instance, I know you guys have your bootstrap scripts too
> # You also have your github actions workflow scripts too
> conan install . --output-folder=build
> -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
> # install conan packages and build C++ for minifi using Conan & CMake, we run
> this way, so conan's tc.cache_variables
> # are like passing -D{minifi_option} upon running cmake generate, then
> make to build MiNiFi C++
> conan build . --output-folder=build
> -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
>
> {color:#0747a6}{*}UPDATE (May 4, 2024) -
> [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color}
> Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi*
> and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*}
> extension with minimal set of external libraries needed. If you look at our
> conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a
> tuple of the minimal external libraries needed to build these two essential
> building blocks of MiNFi CPP. Next when you look at conanfile.py
> *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled
> USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external
> library dependencies that aren needed for the *core-minifi* library and
> *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL,
> ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are
> in the extensions folder and add some dependencies on these custom extensions
> that are created. {color:#ff8b00}*I think we should consider creating an
> extensions subfolder for core-minifi related extensions.*{color}
> Here is the output I get after building the MiNiFi CPP code, you can see
> MiNiFi CPP libraries and binary executables built successfully using *CONAN
> version 2* and CMake (feel free to check the attached log for the full conan
> build output): [~szaszm]
>
> {code:java}
> Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list
> Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT
> list
> Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT
> -- Creating API Documentation using /usr/bin/doxygen
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py
> (minifi-cpp-main/0.15.0): Running CMake.build()
> conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build
> "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20
> [ 3%] Built target ossp-uuid-external-build
> [ 6%] Built target openwsman-external
> [ 7%] Built target ossp-uuid-external
> Consolidate compiler generated dependencies of target core-minifi
> [ 8%] Building CXX object
> libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o
> [ 8%] Linking CXX shared library ../bin/libcore-minifi.so
> [ 78%] Built target core-minifi
> Consolidate compiler generated dependencies of target minifi-civet-extensions
> Consolidate compiler generated dependencies of target minifi-openwsman
> Consolidate compiler generated dependencies of target minifi-http-curl
> Consolidate compiler generated dependencies of target minifiexe
> [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so
> [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so
> Consolidate compiler generated dependencies of target
> minifi-standard-processors
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o
> [ 80%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o
> [ 81%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o
> [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so
> [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so
> [ 84%] Built target minifi-civet-extensions
> [ 86%] Built target minifi-http-curl
> [ 87%] Built target minifi-openwsman
> [100%] Built target minifi-standard-processors
> [100%] Linking CXX executable ../bin/minifi
> [100%] Built target minifiexe {code}
> NEXT: I will verify I can build these MiNiFi CPP libraries and binary
> executables using USE_CMAKE_FETCH_CONTENT option set to ON and
> USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
>
> Tracking Prebuilt Binary Conan Packages Available on Conancenter:
> * 'abseil/20230125.3',
> * 'argparse/3.0',
> * 'asio/1.30.2'
> Tracking Conan Packages Not Available on Conancenter, so we need to use an
> alternative hosting service to manage building package sources into prebuilt
> binary conan packages. Here are some of the free conan hosting services for
> open-source C/C++ projects:
> * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
> packages maintained by conan team
> * Bincrafters Bintray Repo: community-driven repo that hosts large number of
> pre-built conan packages for open-source libraries for free
> * Artifactory by JFrog: typically used as a commercial hosting service, but
> also provides a free tier for open-source projects.
> * {*}Self-Hosting{*}: While there are multiple options, I will focus on
> *GitHub Packages* as its a free package hosting service, which can be used to
> host our own conan package repos, allowing us to store and distribute our
> pre-built conan packages without having to use 3rd party services like
> Bincrafters or pay for commercial solution.
> ** {*}Advantage with GitHub Packages{*}: its free, integrated with our
> existing GitHub workflow, allows us full control hosting our conan packages
> without relying on 3rd party services, useful if we need to have specific
> requirements or need flexibility in managing our package hosting.
> Tracking Non Prebuilt Binary Conan Packages, which later I will create a
> *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account
> for Linux, Windows, Mac, etc):
> * bustache: [https://github.com/jamboree/bustache]
>
> {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
> after I finish my work for MINFICPP-2346: While I am working on migrating
> MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
> PR we could do the following:
> # Put the external libraries that MiNiFi CPP Core depends on into its own
> *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other
> names), we can run this CMake code before adding *add_subdirectory(libminifi)*
> ## External libs that libminifi needs that are in MiNiFi CPP's root
> CMakeLists.txt (refactor for better organization):
> yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
> gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
> magic_enum, RangeV3, Asio, MagicEnum
>
> ## We can also clean up the *thirdparty* folder as we migrate to conan
> packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep
> it MiNiFi C{+}+ code and its extension processors, etc.
> # I tried building MiNiFi CPP Core just with just
> *minifi-standard-processors* and the other extensions disabled, but it seems
> even some of MiNiFi CPP Core depends on some of the extensions to be built
> with it, so it doesn't fail building, such as Openwsman. If we don't have
> ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we
> should look having a build config to build minimal MiNiFi CPP Core:
> {code:java}
> libminifi/src/controllers/SSLContextService.cpp:520:80: error:
> ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
