[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
--------------------------------------------
Attachment: minifi_cpp_cmake_standalone_build_7eb8beb.log
> Speed up MiNiFi Build with Conan package manager (C++)
> ------------------------------------------------------
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Affects Versions: 0.15.0
> Environment: Ubuntu 22.04
> Reporter: James Guzman (Medel)
> Assignee: James Guzman (Medel)
> Priority: Minor
> Labels: build, conan, ubuntu
> Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log,
> minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
> point, I will pull in lastest updates from MiNiFi C++ main branch on parent
> level and rebase or merge into my PR.
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
> During the the build process, it clones all the external dependencies and
> then builds those external dependencies while its building MiNiFi CPP.
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail
> to download or fail to build. It would be faster to build MiNiFi C++ with the
> external dependencies already preinstalled.
>
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
> enable support for using *conan install* to install all or most of MiNiFi's
> external dependencies, using *conan install --build=missing* to build them as
> prebuilt binary conan packages, upload them to conancenter with {*}conan
> upload{*}, and then run cmake generate to generate the appropriate build
> files for the OS, then run make to build MiNiFi CPP. Also conan has really
> good support for cross platform compatibility for Linux and Windows. At this
> point because we already have most of MiNiFi CPP's external lib dependencies
> installed with conan, our CMake build will just focus on building the MiNiFi
> CPP code, enabling faster builds. Also I will need to account for backward
> compatibility support with the previous way of installing MiNiFi CPP external
> lib dependencies using CMake FetchContent_Declare(...) approach until
> conanfile.py approach covers installing all these dependencies. A valid
> verification of the MiNiFi C++ build would be to run GTESTs and if all the
> expected test cases pass, then it may be fair to conclude we would be ready
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
> script(s)):*
> # Install conan version 2 into MiNiFi C++ dev environment (my preference is
> using docker container)
> # Create a conanfile.py file at root of MiNiFi C++ project where we run
> "conan install" or "conan install --build=missing" to install MiNiFi C++
> external lib dependencies first. Then we run CMake to build MiNiFi C++.
> # Find all MiNiFi C++ find_package(...) CMake function calls and check if
> there is already supported conan packages on the conancenter, so we can
> install those packages.
> # Find all FetchContent_Declare(...) CMake function calls and check if there
> is already supported conan packages on the conancenter, so we can install
> those packages.
> # At this point we will have used conan version 2 C++ package manager to
> install almost all MiNiFi C++'s external dependencies.
> ## With Conan, we will have installed these MiNiFi external lib dependencies
> as prebuilt binary conan packages.
> ## Or we will have instructed conan to first build each C++ external lib
> dependency, upload each lib to conancenter.
> # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so
> we can then run CMake to build MiNiFi faster.
>
> Here is my draft PR for this Jira Ticket for your reference:
> [https://github.com/apache/nifi-minifi-cpp/pull/1775]
> * I have added *abseil* external lib dependency to conanfile.py and
> integrated it into Abseil.cmake with conditional statement that checks if
> USE_CONAN_PACKAGER minifi cpp option is true else checks if
> USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
> began the process of integrating in conan version 2.
>
> Here are the commands that I run:
>
> {code:java}
> cd $HOME/src/pipeline/nifi-minifi-cpp
> # NOTE: I also plan to add smoother integration of conan commands into MiNiFI
> CPP too
> # For instance, I know you guys have your bootstrap scripts too
> # You also have your github actions workflow scripts too
> conan install . --output-folder=build
> -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
> # install conan packages and build C++ for minifi using Conan & CMake, we run
> this way, so conan's tc.cache_variables
> # are like passing -D{minifi_option} upon running cmake generate, then
> make to build MiNiFi C++
> conan build . --output-folder=build
> -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
> {*}{color:#0747a6}UPDATE (May 4, 2024) -
> {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
> Verified I can build MiNiFi C++ with standalone CMake like how we currently
> do it even after integrating conan into the build infrastructure. CMake built
> MiNiFi Core, Main and Standard Processors successfully. Also double checked
> that conan build MiNiFi still works.
>
> {color:#0747a6}{*}UPDATE (May 4, 2024) -
> [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color}
> Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi*
> and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*}
> extension with minimal set of external libraries needed. If you look at our
> conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a
> tuple of the minimal external libraries needed to build these two essential
> building blocks of MiNFi CPP. Next when you look at conanfile.py
> *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled
> USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external
> library dependencies that aren needed for the *core-minifi* library and
> *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL,
> ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are
> in the extensions folder and add some dependencies on these custom extensions
> that are created. {color:#ff8b00}*I think we should consider creating an
> extensions subfolder for core-minifi related extensions.*{color}
> Here is the output I get after building the MiNiFi CPP code, you can see
> MiNiFi CPP libraries and binary executables built successfully using *CONAN
> version 2* and CMake (feel free to check the attached log for the full conan
> build output): [~szaszm]
>
> {code:java}
> Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list
> Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT
> list
> Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT
> -- Creating API Documentation using /usr/bin/doxygen
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py
> (minifi-cpp-main/0.15.0): Running CMake.build()
> conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build
> "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20
> [ 3%] Built target ossp-uuid-external-build
> [ 6%] Built target openwsman-external
> [ 7%] Built target ossp-uuid-external
> Consolidate compiler generated dependencies of target core-minifi
> [ 8%] Building CXX object
> libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o
> [ 8%] Linking CXX shared library ../bin/libcore-minifi.so
> [ 78%] Built target core-minifi
> Consolidate compiler generated dependencies of target minifi-civet-extensions
> Consolidate compiler generated dependencies of target minifi-openwsman
> Consolidate compiler generated dependencies of target minifi-http-curl
> Consolidate compiler generated dependencies of target minifiexe
> [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so
> [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so
> Consolidate compiler generated dependencies of target
> minifi-standard-processors
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o
> [ 80%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o
> [ 81%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o
> [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so
> [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so
> [ 84%] Built target minifi-civet-extensions
> [ 86%] Built target minifi-http-curl
> [ 87%] Built target minifi-openwsman
> [100%] Built target minifi-standard-processors
> [100%] Linking CXX executable ../bin/minifi
> [100%] Built target minifiexe {code}
> NEXT: I will verify I can build these MiNiFi CPP libraries and binary
> executables using USE_CMAKE_FETCH_CONTENT option set to ON and
> USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
>
> Tracking Prebuilt Binary Conan Packages Available on Conancenter:
> * 'abseil/20230125.3',
> * 'argparse/3.0',
> * 'asio/1.30.2'
> Tracking Conan Packages Not Available on Conancenter, so we need to use an
> alternative hosting service to manage building package sources into prebuilt
> binary conan packages. Here are some of the free conan hosting services for
> open-source C/C++ projects:
> * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
> packages maintained by conan team
> * Bincrafters Bintray Repo: community-driven repo that hosts large number of
> pre-built conan packages for open-source libraries for free
> * Artifactory by JFrog: typically used as a commercial hosting service, but
> also provides a free tier for open-source projects.
> * {*}Self-Hosting{*}: While there are multiple options, I will focus on
> *GitHub Packages* as its a free package hosting service, which can be used to
> host our own conan package repos, allowing us to store and distribute our
> pre-built conan packages without having to use 3rd party services like
> Bincrafters or pay for commercial solution.
> ** {*}Advantage with GitHub Packages{*}: its free, integrated with our
> existing GitHub workflow, allows us full control hosting our conan packages
> without relying on 3rd party services, useful if we need to have specific
> requirements or need flexibility in managing our package hosting.
> Tracking Non Prebuilt Binary Conan Packages, which later I will create a
> *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account
> for Linux, Windows, Mac, etc):
> * bustache: [https://github.com/jamboree/bustache]
>
> {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
> after I finish my work for MINFICPP-2346: While I am working on migrating
> MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
> PR we could do the following:
> # Put the external libraries that MiNiFi CPP Core depends on into its own
> *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other
> names), we can run this CMake code before adding *add_subdirectory(libminifi)*
> ## External libs that libminifi needs that are in MiNiFi CPP's root
> CMakeLists.txt (refactor for better organization):
> yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
> gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
> magic_enum, RangeV3, Asio, MagicEnum
>
> ## We can also clean up the *thirdparty* folder as we migrate to conan
> packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep
> it MiNiFi C{+}+ code and its extension processors, etc.
> # I tried building MiNiFi CPP Core just with just
> *minifi-standard-processors* and the other extensions disabled, but it seems
> even some of MiNiFi CPP Core depends on some of the extensions to be built
> with it, so it doesn't fail building, such as Openwsman. If we don't have
> ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we
> should look having a build config to build minimal MiNiFi CPP Core:
> {code:java}
> libminifi/src/controllers/SSLContextService.cpp:520:80: error:
> ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
