[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Guzman (Medel) updated MINIFICPP-2346: -------------------------------------------- Description: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} {color:#0747a6}*UPDATE (May 7, 2024)* - [2460e84|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2460e846832d693c991e1da632b86ba2d4e34516]: {color:#172b4d}*Enabled OPS and OPC* MiNiFi Options, so those *MiNiFi C++ extensions* are built using *Conan CMake* and their bundled CMake dependencies use their particular *{{open62541/1.3.3@minifi/dev}}* and {*}{{mbedtls/2.16.3@minifi/dev}} conan packages{*}. I had to fix the mbedtls conan package from conan center index since it wasnt compatible when open62541 tried to integrate with mbedtls. In the thirdparty folder, you will see on line 139 where I updated libembedtls to be libmbedtls and fixing that typo allowed open62541 conan package to be able to be created with mbedtls as the encryption. I plan to make a follow up PR on conan center index for mbedtls conan recipe. Currently, we are able to build MiNiFi's additional extensions for OPS and OPC using conan. Verified backward compatibility with standalone CMake works, so I can also just build MiNiFi using CMake only.{color}{color} *{color:#0747a6}UPDATE (May 5, 2024) - [446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:{color}*{color:#172b4d} When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility with standalone CMake approach. I got the following error coming from extension robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I will add the fix for this issue:{color} {code:java} /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26: error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and ‘rocksdb::DBOptions’) 71 | if (db_opts_copy != db_options_) { | ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~ | | | | | rocksdb::DBOptions | rocksdb::DBOptions {code} We ran into this issue since when we created rocksdb/8.10.2 conan package, we didn't apply the 2 patches like was done in the standalone CMake approach, so I updated our conan approach where I created a rocksdb conan recipe in the nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and applied them to create the rocksdb conan package successfully. Also I do want to note that I referenced the rocksdb conan recipe from conan center index repo. After creating the rocksdb conan package, then I built MiNiFi C++ GTESTs for MiNiFi Core, MainExe, Standard-Processors and the other relevant extensions that needed to be activated for these 3 essential components using conan build. Here we have the results after running ctest in the build folder: {code:java} 192/194 Test #192: ConfigFileEncryptorTests .................... Passed 0.02 sec Start 193: ConfigFileTests 193/194 Test #193: ConfigFileTests ............................. Passed 0.01 sec Start 194: ControllerTests 194/194 Test #194: ControllerTests .............................***Failed 0.52 sec 99% tests passed, 2 tests failed out of 194 Total Test time (real) = 382.89 sec The following tests FAILED: 45 - LoggerTests (Failed) 194 - ControllerTests (Failed)Errors while running CTestOutput from these tests are in: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. {code} *We can see that 99% of the tests passed. Thus, 192/194 tests passed. This was using conan to build MiNiFi C++.* I also verified I can build{color:#172b4d} *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, BUILD_ROCKSDB to ON, ENABLE_ROCKSDB to ON to {*}build GTESTs using standalone CMake{*}. Here we can see similar results with running the MiNiFi GESTs:{color} {code:java} 192/194 Test #192: ConfigFileEncryptorTests .................... Passed 0.07 sec Start 193: ConfigFileTests 193/194 Test #193: ConfigFileTests ............................. Passed 0.08 sec Start 194: ControllerTests 194/194 Test #194: ControllerTests .............................***Failed 0.76 sec 99% tests passed, 1 tests failed out of 194 Total Test time (real) = 236.38 sec The following tests FAILED: 194 - ControllerTests (Failed)Errors while running CTestOutput from these tests are in: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. {code} We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was using standalone CMake to build MiNiFi C++. {*}{color:#0747a6}UPDATE (May 4, 2024) - {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}: Verified I can build MiNiFi C++ with standalone CMake like how we currently do it even after integrating conan into the build infrastructure. CMake built *MiNiFi Core, Main* and *Standard Processors* successfully. Also double checked that conan build MiNiFi still works. (Feel free to reference the logs for the CMake standalone build and the conan build of MiNiFi CPP) {color:#0747a6}{*}UPDATE (May 4, 2024) - [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color} Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi* and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*} extension with minimal set of external libraries needed. If you look at our conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a tuple of the minimal external libraries needed to build these two essential building blocks of MiNFi CPP. Next when you look at conanfile.py *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external library dependencies that aren needed for the *core-minifi* library and *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are in the extensions folder and add some dependencies on these custom extensions that are created. {color:#ff8b00}*I think we should consider creating an extensions subfolder for core-minifi related extensions.*{color} Here is the output I get after building the MiNiFi CPP code, you can see MiNiFi CPP libraries and binary executables built successfully using *CONAN version 2* and CMake (feel free to check the attached log for the full conan build output): [~szaszm] {code:java} Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT list Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT -- Creating API Documentation using /usr/bin/doxygen -- Configuring done -- Generating done -- Build files have been written to: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py (minifi-cpp-main/0.15.0): Running CMake.build() conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20 [ 3%] Built target ossp-uuid-external-build [ 6%] Built target openwsman-external [ 7%] Built target ossp-uuid-external Consolidate compiler generated dependencies of target core-minifi [ 8%] Building CXX object libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o [ 8%] Linking CXX shared library ../bin/libcore-minifi.so [ 78%] Built target core-minifi Consolidate compiler generated dependencies of target minifi-civet-extensions Consolidate compiler generated dependencies of target minifi-openwsman Consolidate compiler generated dependencies of target minifi-http-curl Consolidate compiler generated dependencies of target minifiexe [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so Consolidate compiler generated dependencies of target minifi-standard-processors [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o [ 80%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o [ 81%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so [ 84%] Built target minifi-civet-extensions [ 86%] Built target minifi-http-curl [ 87%] Built target minifi-openwsman [100%] Built target minifi-standard-processors [100%] Linking CXX executable ../bin/minifi [100%] Built target minifiexe {code} NEXT: I will verify I can build these MiNiFi CPP libraries and binary executables using USE_CMAKE_FETCH_CONTENT option set to ON and USE_CONAN_PACKAGER option set to OFF for the for the follow up commit. Tracking Prebuilt Binary Conan Packages Available on Conancenter: * 'abseil/20230125.3', * 'argparse/3.0', * 'asio/1.30.2' Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects: * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan packages maintained by conan team * Bincrafters Bintray Repo: community-driven repo that hosts large number of pre-built conan packages for open-source libraries for free * Artifactory by JFrog: typically used as a commercial hosting service, but also provides a free tier for open-source projects. * {*}Self-Hosting{*}: While there are multiple options, I will focus on *GitHub Packages* as its a free package hosting service, which can be used to host our own conan package repos, allowing us to store and distribute our pre-built conan packages without having to use 3rd party services like Bincrafters or pay for commercial solution. ** {*}Advantage with GitHub Packages{*}: its free, integrated with our existing GitHub workflow, allows us full control hosting our conan packages without relying on 3rd party services, useful if we need to have specific requirements or need flexibility in managing our package hosting. Tracking Non Prebuilt Binary Conan Packages, which later I will create a *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account for Linux, Windows, Mac, etc): * bustache: [https://github.com/jamboree/bustache] {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP after I finish my work for MINFICPP-2346: While I am working on migrating MiNiFi CPP build infrastructure toward leveraging conan packager, for a later PR we could do the following: # Put the external libraries that MiNiFi CPP Core depends on into its own *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other names), we can run this CMake code before adding *add_subdirectory(libminifi)* ## External libs that libminifi needs that are in MiNiFi CPP's root CMakeLists.txt (refactor for better organization): yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads, gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio, magic_enum, RangeV3, Asio, MagicEnum ## We can also clean up the *thirdparty* folder as we migrate to conan packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep it MiNiFi C{+}+ code and its extension processors, etc. # I tried building MiNiFi CPP Core just with just *minifi-standard-processors* and the other extensions disabled, but it seems even some of MiNiFi CPP Core depends on some of the extensions to be built with it, so it doesn't fail building, such as Openwsman. If we don't have ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we should look having a build config to build minimal MiNiFi CPP Core: {code:java} libminifi/src/controllers/SSLContextService.cpp:520:80: error: ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code} # I noticed that when building MiNiFi Core, MainExe, Standard-Processors and their external lib dependencies and then building the appropriate GTEST, I get a header error coming from rocksdb, but I haven't enabled rocksdb, so I think at a later point we should shift that test out, so when building these 3 core building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi Core and MainExe have a dependency for storing into that DB. Here is an example of the header error (I will ENABLE_ROCKSDB to ON): {code:java} [ 99%] Built target ControllerTests In file included from /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30, from /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10: fatal error: rocksdb/db.h: No such file or directory 26 | #include "rocksdb/db.h" {code} was: *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR. *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled. *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to enable support for using *conan install* to install all or most of MiNiFi's external dependencies, using *conan install --build=missing* to build them as prebuilt binary conan packages, upload them to conancenter with {*}conan upload{*}, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach. *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):* # Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container) # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++. # Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages. # At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies. ## With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages. ## Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter. # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster. Here is my draft PR for this Jira Ticket for your reference: [https://github.com/apache/nifi-minifi-cpp/pull/1775] * I have added *abseil* external lib dependency to conanfile.py and integrated it into Abseil.cmake with conditional statement that checks if USE_CONAN_PACKAGER minifi cpp option is true else checks if USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can began the process of integrating in conan version 2. Here are the commands that I run: {code:java} cd $HOME/src/pipeline/nifi-minifi-cpp # NOTE: I also plan to add smoother integration of conan commands into MiNiFI CPP too # For instance, I know you guys have your bootstrap scripts too # You also have your github actions workflow scripts too conan install . --output-folder=build -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # install conan packages and build C++ for minifi using Conan & CMake, we run this way, so conan's tc.cache_variables # are like passing -D{minifi_option} upon running cmake generate, then make to build MiNiFi C++ conan build . --output-folder=build -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} *{color:#0747a6}UPDATE (May 5, 2024) - [446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:{color}*{color:#172b4d} When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility with standalone CMake approach. I got the following error coming from extension robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I will add the fix for this issue:{color} {code:java} /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26: error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and ‘rocksdb::DBOptions’) 71 | if (db_opts_copy != db_options_) { | ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~ | | | | | rocksdb::DBOptions | rocksdb::DBOptions {code} We ran into this issue since when we created rocksdb/8.10.2 conan package, we didn't apply the 2 patches like was done in the standalone CMake approach, so I updated our conan approach where I created a rocksdb conan recipe in the nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and applied them to create the rocksdb conan package successfully. Also I do want to note that I referenced the rocksdb conan recipe from conan center index repo. After creating the rocksdb conan package, then I built MiNiFi C++ GTESTs for MiNiFi Core, MainExe, Standard-Processors and the other relevant extensions that needed to be activated for these 3 essential components using conan build. Here we have the results after running ctest in the build folder: {code:java} 192/194 Test #192: ConfigFileEncryptorTests .................... Passed 0.02 sec Start 193: ConfigFileTests 193/194 Test #193: ConfigFileTests ............................. Passed 0.01 sec Start 194: ControllerTests 194/194 Test #194: ControllerTests .............................***Failed 0.52 sec 99% tests passed, 2 tests failed out of 194 Total Test time (real) = 382.89 sec The following tests FAILED: 45 - LoggerTests (Failed) 194 - ControllerTests (Failed)Errors while running CTestOutput from these tests are in: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. {code} *We can see that 99% of the tests passed. Thus, 192/194 tests passed. This was using conan to build MiNiFi C++.* I also verified I can build{color:#172b4d} *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, BUILD_ROCKSDB to ON, ENABLE_ROCKSDB to ON to {*}build GTESTs using standalone CMake{*}. Here we can see similar results with running the MiNiFi GESTs:{color} {code:java} 192/194 Test #192: ConfigFileEncryptorTests .................... Passed 0.07 sec Start 193: ConfigFileTests 193/194 Test #193: ConfigFileTests ............................. Passed 0.08 sec Start 194: ControllerTests 194/194 Test #194: ControllerTests .............................***Failed 0.76 sec 99% tests passed, 1 tests failed out of 194 Total Test time (real) = 236.38 sec The following tests FAILED: 194 - ControllerTests (Failed)Errors while running CTestOutput from these tests are in: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. {code} We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was using standalone CMake to build MiNiFi C++. {*}{color:#0747a6}UPDATE (May 4, 2024) - {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}: Verified I can build MiNiFi C++ with standalone CMake like how we currently do it even after integrating conan into the build infrastructure. CMake built *MiNiFi Core, Main* and *Standard Processors* successfully. Also double checked that conan build MiNiFi still works. (Feel free to reference the logs for the CMake standalone build and the conan build of MiNiFi CPP) {color:#0747a6}{*}UPDATE (May 4, 2024) - [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color} Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi* and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*} extension with minimal set of external libraries needed. If you look at our conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a tuple of the minimal external libraries needed to build these two essential building blocks of MiNFi CPP. Next when you look at conanfile.py *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external library dependencies that aren needed for the *core-minifi* library and *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are in the extensions folder and add some dependencies on these custom extensions that are created. {color:#ff8b00}*I think we should consider creating an extensions subfolder for core-minifi related extensions.*{color} Here is the output I get after building the MiNiFi CPP code, you can see MiNiFi CPP libraries and binary executables built successfully using *CONAN version 2* and CMake (feel free to check the attached log for the full conan build output): [~szaszm] {code:java} Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT list Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT -- Creating API Documentation using /usr/bin/doxygen -- Configuring done -- Generating done -- Build files have been written to: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py (minifi-cpp-main/0.15.0): Running CMake.build() conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20 [ 3%] Built target ossp-uuid-external-build [ 6%] Built target openwsman-external [ 7%] Built target ossp-uuid-external Consolidate compiler generated dependencies of target core-minifi [ 8%] Building CXX object libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o [ 8%] Linking CXX shared library ../bin/libcore-minifi.so [ 78%] Built target core-minifi Consolidate compiler generated dependencies of target minifi-civet-extensions Consolidate compiler generated dependencies of target minifi-openwsman Consolidate compiler generated dependencies of target minifi-http-curl Consolidate compiler generated dependencies of target minifiexe [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so Consolidate compiler generated dependencies of target minifi-standard-processors [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o [ 80%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o [ 81%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so [ 84%] Built target minifi-civet-extensions [ 86%] Built target minifi-http-curl [ 87%] Built target minifi-openwsman [100%] Built target minifi-standard-processors [100%] Linking CXX executable ../bin/minifi [100%] Built target minifiexe {code} NEXT: I will verify I can build these MiNiFi CPP libraries and binary executables using USE_CMAKE_FETCH_CONTENT option set to ON and USE_CONAN_PACKAGER option set to OFF for the for the follow up commit. Tracking Prebuilt Binary Conan Packages Available on Conancenter: * 'abseil/20230125.3', * 'argparse/3.0', * 'asio/1.30.2' Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects: * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan packages maintained by conan team * Bincrafters Bintray Repo: community-driven repo that hosts large number of pre-built conan packages for open-source libraries for free * Artifactory by JFrog: typically used as a commercial hosting service, but also provides a free tier for open-source projects. * {*}Self-Hosting{*}: While there are multiple options, I will focus on *GitHub Packages* as its a free package hosting service, which can be used to host our own conan package repos, allowing us to store and distribute our pre-built conan packages without having to use 3rd party services like Bincrafters or pay for commercial solution. ** {*}Advantage with GitHub Packages{*}: its free, integrated with our existing GitHub workflow, allows us full control hosting our conan packages without relying on 3rd party services, useful if we need to have specific requirements or need flexibility in managing our package hosting. Tracking Non Prebuilt Binary Conan Packages, which later I will create a *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account for Linux, Windows, Mac, etc): * bustache: [https://github.com/jamboree/bustache] {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP after I finish my work for MINFICPP-2346: While I am working on migrating MiNiFi CPP build infrastructure toward leveraging conan packager, for a later PR we could do the following: # Put the external libraries that MiNiFi CPP Core depends on into its own *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other names), we can run this CMake code before adding *add_subdirectory(libminifi)* ## External libs that libminifi needs that are in MiNiFi CPP's root CMakeLists.txt (refactor for better organization): yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads, gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio, magic_enum, RangeV3, Asio, MagicEnum ## We can also clean up the *thirdparty* folder as we migrate to conan packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep it MiNiFi C{+}+ code and its extension processors, etc. # I tried building MiNiFi CPP Core just with just *minifi-standard-processors* and the other extensions disabled, but it seems even some of MiNiFi CPP Core depends on some of the extensions to be built with it, so it doesn't fail building, such as Openwsman. If we don't have ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we should look having a build config to build minimal MiNiFi CPP Core: {code:java} libminifi/src/controllers/SSLContextService.cpp:520:80: error: ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code} # I noticed that when building MiNiFi Core, MainExe, Standard-Processors and their external lib dependencies and then building the appropriate GTEST, I get a header error coming from rocksdb, but I haven't enabled rocksdb, so I think at a later point we should shift that test out, so when building these 3 core building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi Core and MainExe have a dependency for storing into that DB. Here is an example of the header error (I will ENABLE_ROCKSDB to ON): {code:java} [ 99%] Built target ControllerTests In file included from /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30, from /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10: fatal error: rocksdb/db.h: No such file or directory 26 | #include "rocksdb/db.h" {code} > Speed up MiNiFi Build with Conan package manager (C++) > ------------------------------------------------------ > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement > Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 > Reporter: James Guzman (Medel) > Assignee: James Guzman (Medel) > Priority: Minor > Labels: build, conan, ubuntu > Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log, > minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log > > Time Spent: 1h > Remaining Estimate: 0h > > *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain > point, I will pull in lastest updates from MiNiFi C++ main branch on parent > level and rebase or merge into my PR. > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > Here is my draft PR for this Jira Ticket for your reference: > [https://github.com/apache/nifi-minifi-cpp/pull/1775] > Here are the commands that I run: > > {code:java} > cd $HOME/src/pipeline/nifi-minifi-cpp > # NOTE: I also plan to add smoother integration of conan commands into MiNiFI > CPP too > # For instance, I know you guys have your bootstrap scripts too > # You also have your github actions workflow scripts too > conan install . --output-folder=build > -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux > # install conan packages and build C++ for minifi using Conan & CMake, we run > this way, so conan's tc.cache_variables > # are like passing -D{minifi_option} upon running cmake generate, then > make to build MiNiFi C++ > conan build . --output-folder=build > -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} > {color:#0747a6}*UPDATE (May 7, 2024)* - > [2460e84|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2460e846832d693c991e1da632b86ba2d4e34516]: > {color:#172b4d}*Enabled OPS and OPC* MiNiFi Options, so those *MiNiFi C++ > extensions* are built using *Conan CMake* and their bundled CMake > dependencies use their particular *{{open62541/1.3.3@minifi/dev}}* and > {*}{{mbedtls/2.16.3@minifi/dev}} conan packages{*}. I had to fix the mbedtls > conan package from conan center index since it wasnt compatible when > open62541 tried to integrate with mbedtls. In the thirdparty folder, you will > see on line 139 where I updated libembedtls to be libmbedtls and fixing that > typo allowed open62541 conan package to be able to be created with mbedtls as > the encryption. I plan to make a follow up PR on conan center index for > mbedtls conan recipe. Currently, we are able to build MiNiFi's additional > extensions for OPS and OPC using conan. Verified backward compatibility with > standalone CMake works, so I can also just build MiNiFi using CMake > only.{color}{color} > > *{color:#0747a6}UPDATE (May 5, 2024) - > [446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:{color}*{color:#172b4d} > When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and > *Standard-Processors* and the other relevant that needed to be activated for > these 3 essential components, I also set SKIP_TESTS to OFF, I set > ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs > using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for > conan approach while keeping backward compatibility with standalone CMake > approach. I got the following error coming from extension > robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I > will add the fix for this issue:{color} > {code:java} > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26: > error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and > ‘rocksdb::DBOptions’) > 71 | if (db_opts_copy != db_options_) { > | ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~ > | | | > | | rocksdb::DBOptions > | rocksdb::DBOptions {code} > We ran into this issue since when we created rocksdb/8.10.2 conan package, we > didn't apply the 2 patches like was done in the standalone CMake approach, so > I updated our conan approach where I created a rocksdb conan recipe in the > nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and > applied them to create the rocksdb conan package successfully. Also I do want > to note that I referenced the rocksdb conan recipe from conan center index > repo. After creating the rocksdb conan package, then I built MiNiFi C++ > GTESTs for MiNiFi Core, MainExe, Standard-Processors and the other relevant > extensions that needed to be activated for these 3 essential components using > conan build. Here we have the results after running ctest in the build folder: > {code:java} > 192/194 Test #192: ConfigFileEncryptorTests .................... Passed > 0.02 sec > Start 193: ConfigFileTests > 193/194 Test #193: ConfigFileTests ............................. Passed > 0.01 sec > Start 194: ControllerTests > 194/194 Test #194: ControllerTests .............................***Failed > 0.52 sec > 99% tests passed, 2 tests failed out of 194 > Total Test time (real) = 382.89 sec > The following tests FAILED: 45 - LoggerTests (Failed) 194 - > ControllerTests (Failed)Errors while running CTestOutput from these tests are > in: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse > "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. > {code} > *We can see that 99% of the tests passed. Thus, 192/194 tests passed. This > was using conan to build MiNiFi C++.* > > I also verified I can build{color:#172b4d} *MiNiFi C++ GTESTs for MiNiFi > Core, MainExe* and *Standard-Processors* and the other relevant that needed > to be activated for these 3 essential components, I also set SKIP_TESTS to > OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, > BUILD_ROCKSDB to ON, ENABLE_ROCKSDB to ON to {*}build GTESTs using standalone > CMake{*}. Here we can see similar results with running the MiNiFi > GESTs:{color} > {code:java} > 192/194 Test #192: ConfigFileEncryptorTests .................... Passed > 0.07 sec > Start 193: ConfigFileTests > 193/194 Test #193: ConfigFileTests ............................. Passed > 0.08 sec > Start 194: ControllerTests > 194/194 Test #194: ControllerTests .............................***Failed > 0.76 sec > 99% tests passed, 1 tests failed out of 194 > Total Test time (real) = 236.38 sec > The following tests FAILED: 194 - ControllerTests (Failed)Errors while > running CTestOutput from these tests are in: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse > "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. > {code} > We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was > using standalone CMake to build MiNiFi C++. > > {*}{color:#0747a6}UPDATE (May 4, 2024) - > {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}: > Verified I can build MiNiFi C++ with standalone CMake like how we currently > do it even after integrating conan into the build infrastructure. CMake built > *MiNiFi Core, Main* and *Standard Processors* successfully. Also double > checked that conan build MiNiFi still works. (Feel free to reference the logs > for the CMake standalone build and the conan build of MiNiFi CPP) > > {color:#0747a6}{*}UPDATE (May 4, 2024) - > [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color} > Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi* > and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*} > extension with minimal set of external libraries needed. If you look at our > conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a > tuple of the minimal external libraries needed to build these two essential > building blocks of MiNFi CPP. Next when you look at conanfile.py > *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled > USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external > library dependencies that aren needed for the *core-minifi* library and > *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, > ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are > in the extensions folder and add some dependencies on these custom extensions > that are created. {color:#ff8b00}*I think we should consider creating an > extensions subfolder for core-minifi related extensions.*{color} > Here is the output I get after building the MiNiFi CPP code, you can see > MiNiFi CPP libraries and binary executables built successfully using *CONAN > version 2* and CMake (feel free to check the attached log for the full conan > build output): [~szaszm] > > {code:java} > Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list > Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT > list > Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT > -- Creating API Documentation using /usr/bin/doxygen > -- Configuring done > -- Generating done > -- Build files have been written to: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py > (minifi-cpp-main/0.15.0): Running CMake.build() > conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build > "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20 > [ 3%] Built target ossp-uuid-external-build > [ 6%] Built target openwsman-external > [ 7%] Built target ossp-uuid-external > Consolidate compiler generated dependencies of target core-minifi > [ 8%] Building CXX object > libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o > [ 8%] Linking CXX shared library ../bin/libcore-minifi.so > [ 78%] Built target core-minifi > Consolidate compiler generated dependencies of target minifi-civet-extensions > Consolidate compiler generated dependencies of target minifi-openwsman > Consolidate compiler generated dependencies of target minifi-http-curl > Consolidate compiler generated dependencies of target minifiexe > [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so > [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so > Consolidate compiler generated dependencies of target > minifi-standard-processors > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o > [ 80%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o > [ 81%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o > [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so > [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so > [ 84%] Built target minifi-civet-extensions > [ 86%] Built target minifi-http-curl > [ 87%] Built target minifi-openwsman > [100%] Built target minifi-standard-processors > [100%] Linking CXX executable ../bin/minifi > [100%] Built target minifiexe {code} > NEXT: I will verify I can build these MiNiFi CPP libraries and binary > executables using USE_CMAKE_FETCH_CONTENT option set to ON and > USE_CONAN_PACKAGER option set to OFF for the for the follow up commit. > > Tracking Prebuilt Binary Conan Packages Available on Conancenter: > * 'abseil/20230125.3', > * 'argparse/3.0', > * 'asio/1.30.2' > Tracking Conan Packages Not Available on Conancenter, so we need to use an > alternative hosting service to manage building package sources into prebuilt > binary conan packages. Here are some of the free conan hosting services for > open-source C/C++ projects: > * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan > packages maintained by conan team > * Bincrafters Bintray Repo: community-driven repo that hosts large number of > pre-built conan packages for open-source libraries for free > * Artifactory by JFrog: typically used as a commercial hosting service, but > also provides a free tier for open-source projects. > * {*}Self-Hosting{*}: While there are multiple options, I will focus on > *GitHub Packages* as its a free package hosting service, which can be used to > host our own conan package repos, allowing us to store and distribute our > pre-built conan packages without having to use 3rd party services like > Bincrafters or pay for commercial solution. > ** {*}Advantage with GitHub Packages{*}: its free, integrated with our > existing GitHub workflow, allows us full control hosting our conan packages > without relying on 3rd party services, useful if we need to have specific > requirements or need flexibility in managing our package hosting. > Tracking Non Prebuilt Binary Conan Packages, which later I will create a > *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account > for Linux, Windows, Mac, etc): > * bustache: [https://github.com/jamboree/bustache] > > {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP > after I finish my work for MINFICPP-2346: While I am working on migrating > MiNiFi CPP build infrastructure toward leveraging conan packager, for a later > PR we could do the following: > # Put the external libraries that MiNiFi CPP Core depends on into its own > *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other > names), we can run this CMake code before adding *add_subdirectory(libminifi)* > ## External libs that libminifi needs that are in MiNiFi CPP's root > CMakeLists.txt (refactor for better organization): > yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads, > gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio, > magic_enum, RangeV3, Asio, MagicEnum > > ## We can also clean up the *thirdparty* folder as we migrate to conan > packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep > it MiNiFi C{+}+ code and its extension processors, etc. > # I tried building MiNiFi CPP Core just with just > *minifi-standard-processors* and the other extensions disabled, but it seems > even some of MiNiFi CPP Core depends on some of the extensions to be built > with it, so it doesn't fail building, such as Openwsman. If we don't have > ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we > should look having a build config to build minimal MiNiFi CPP Core: > {code:java} > libminifi/src/controllers/SSLContextService.cpp:520:80: error: > ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code} > > # I noticed that when building MiNiFi Core, MainExe, Standard-Processors and > their external lib dependencies and then building the appropriate GTEST, I > get a header error coming from rocksdb, but I haven't enabled rocksdb, so I > think at a later point we should shift that test out, so when building these > 3 core building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi > Core and MainExe have a dependency for storing into that DB. Here is an > example of the header error (I will ENABLE_ROCKSDB to ON): > {code:java} > [ 99%] Built target ControllerTests > In file included from > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30, > from > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10: > fatal error: rocksdb/db.h: No such file or directory > 26 | #include "rocksdb/db.h" {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)