[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Guzman (Medel) updated MINIFICPP-2346: -------------------------------------------- Attachment: minifi_cpp_conan_build_2d1b884.log > Speed up MiNiFi Build with Conan package manager (C++) > ------------------------------------------------------ > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement > Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 > Reporter: James Guzman (Medel) > Assignee: James Guzman (Medel) > Priority: Minor > Labels: build, conan, ubuntu > Attachments: minifi_cpp_conan_build_2d1b884.log > > Time Spent: 1h > Remaining Estimate: 0h > > *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain > point, I will pull in lastest updates from MiNiFi C++ main branch on parent > level and rebase or merge into my PR. > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > Here is my draft PR for this Jira Ticket for your reference: > [https://github.com/apache/nifi-minifi-cpp/pull/1775] > * I have added *abseil* external lib dependency to conanfile.py and > integrated it into Abseil.cmake with conditional statement that checks if > USE_CONAN_PACKAGER minifi cpp option is true else checks if > USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can > began the process of integrating in conan version 2. > > Here are the commands that I run: > > {code:java} > cd $HOME/src/pipeline/nifi-minifi-cpp > # NOTE: I also plan to add smoother integration of conan commands into MiNiFI > CPP too > # For instance, I know you guys have your bootstrap scripts too > # You also have your github actions workflow scripts too > conan install . --output-folder=build > -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux > # install conan packages and build C++ for minifi using Conan & CMake, we run > this way, so conan's tc.cache_variables > # are like passing -D{minifi_option} upon running cmake generate, then > make to build MiNiFi C++ > conan build . --output-folder=build > -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} > > {color:#0747a6}{*}UPDATE (May 4, 2024) - > [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color} > Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi* > and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*} > extension with minimal set of external libraries needed. If you look at our > conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a > tuple of the minimal external libraries needed to build these two essential > building blocks of MiNFi CPP. Next when you look at conanfile.py > *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled > USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external > library dependencies that aren needed for the *core-minifi* library and > *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, > ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are > in the extensions folder and add some dependencies on these custom extensions > that are created. {color:#ff8b00}*I think we should consider creating an > extensions subfolder for core-minifi related extensions.*{color} > Here is the output I get after building the MiNiFi CPP code, you can see > MiNiFi CPP libraries and binary executables built successfully using *CONAN > version 2* and CMake (feel free to check the attached log for the full conan > build output): [~szaszm] > > {code:java} > Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list > Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT > list > Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT > -- Creating API Documentation using /usr/bin/doxygen > -- Configuring done > -- Generating done > -- Build files have been written to: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py > (minifi-cpp-main/0.15.0): Running CMake.build() > conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build > "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20 > [ 3%] Built target ossp-uuid-external-build > [ 6%] Built target openwsman-external > [ 7%] Built target ossp-uuid-external > Consolidate compiler generated dependencies of target core-minifi > [ 8%] Building CXX object > libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o > [ 8%] Linking CXX shared library ../bin/libcore-minifi.so > [ 78%] Built target core-minifi > Consolidate compiler generated dependencies of target minifi-civet-extensions > Consolidate compiler generated dependencies of target minifi-openwsman > Consolidate compiler generated dependencies of target minifi-http-curl > Consolidate compiler generated dependencies of target minifiexe > [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so > [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so > Consolidate compiler generated dependencies of target > minifi-standard-processors > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o > [ 80%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o > [ 81%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o > [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so > [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so > [ 84%] Built target minifi-civet-extensions > [ 86%] Built target minifi-http-curl > [ 87%] Built target minifi-openwsman > [100%] Built target minifi-standard-processors > [100%] Linking CXX executable ../bin/minifi > [100%] Built target minifiexe {code} > NEXT: I will verify I can build these MiNiFi CPP libraries and binary > executables using USE_CMAKE_FETCH_CONTENT option set to ON and > USE_CONAN_PACKAGER option set to OFF for the for the follow up commit. > > Tracking Prebuilt Binary Conan Packages Available on Conancenter: > * 'abseil/20230125.3', > * 'argparse/3.0', > * 'asio/1.30.2' > Tracking Conan Packages Not Available on Conancenter, so we need to use an > alternative hosting service to manage building package sources into prebuilt > binary conan packages. Here are some of the free conan hosting services for > open-source C/C++ projects: > * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan > packages maintained by conan team > * Bincrafters Bintray Repo: community-driven repo that hosts large number of > pre-built conan packages for open-source libraries for free > * Artifactory by JFrog: typically used as a commercial hosting service, but > also provides a free tier for open-source projects. > * {*}Self-Hosting{*}: While there are multiple options, I will focus on > *GitHub Packages* as its a free package hosting service, which can be used to > host our own conan package repos, allowing us to store and distribute our > pre-built conan packages without having to use 3rd party services like > Bincrafters or pay for commercial solution. > ** {*}Advantage with GitHub Packages{*}: its free, integrated with our > existing GitHub workflow, allows us full control hosting our conan packages > without relying on 3rd party services, useful if we need to have specific > requirements or need flexibility in managing our package hosting. > Tracking Non Prebuilt Binary Conan Packages, which later I will create a > *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account > for Linux, Windows, Mac, etc): > * bustache: [https://github.com/jamboree/bustache] > > {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP > after I finish my work for MINFICPP-2346: While I am working on migrating > MiNiFi CPP build infrastructure toward leveraging conan packager, for a later > PR we could do the following: > # Put the external libraries that MiNiFi CPP Core depends on into its own > *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other > names), we can run this CMake code before adding *add_subdirectory(libminifi)* > ## External libs that libminifi needs that are in MiNiFi CPP's root > CMakeLists.txt (refactor for better organization): > yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads, > gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio, > magic_enum, RangeV3, Asio, MagicEnum > > ## We can also clean up the *thirdparty* folder as we migrate to conan > packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep > it MiNiFi C{+}+ code and its extension processors, etc. > # I tried building MiNiFi CPP Core just with just > *minifi-standard-processors* and the other extensions disabled, but it seems > even some of MiNiFi CPP Core depends on some of the extensions to be built > with it, so it doesn't fail building, such as Openwsman. If we don't have > ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we > should look having a build config to build minimal MiNiFi CPP Core: > {code:java} > libminifi/src/controllers/SSLContextService.cpp:520:80: error: > ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code} -- This message was sent by Atlassian Jira (v8.20.10#820010)