[ https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Guzman (Medel) updated MINIFICPP-2346: -------------------------------------------- Due Date: 19/May/24 (was: 12/May/24) > Speed up MiNiFi Build with Conan package manager (C++) > ------------------------------------------------------ > > Key: MINIFICPP-2346 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2346 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement > Affects Versions: 0.15.0 > Environment: Ubuntu 22.04 > Reporter: James Guzman (Medel) > Assignee: James Guzman (Medel) > Priority: Minor > Labels: build, conan, ubuntu > Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log, > minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log > > Time Spent: 1h 20m > Remaining Estimate: 0h > > *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain > point, I will pull in lastest updates from MiNiFi C++ main branch on parent > level and rebase or merge into my PR. > *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. > During the the build process, it clones all the external dependencies and > then builds those external dependencies while its building MiNiFi CPP. > Sometimes the MiNiFi CPP build fails too when external lib dependencies fail > to download or fail to build. It would be faster to build MiNiFi C++ with the > external dependencies already preinstalled. > > *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to > enable support for using *conan install* to install all or most of MiNiFi's > external dependencies, using *conan install --build=missing* to build them as > prebuilt binary conan packages, upload them to conancenter with {*}conan > upload{*}, and then run cmake generate to generate the appropriate build > files for the OS, then run make to build MiNiFi CPP. Also conan has really > good support for cross platform compatibility for Linux and Windows. At this > point because we already have most of MiNiFi CPP's external lib dependencies > installed with conan, our CMake build will just focus on building the MiNiFi > CPP code, enabling faster builds. Also I will need to account for backward > compatibility support with the previous way of installing MiNiFi CPP external > lib dependencies using CMake FetchContent_Declare(...) approach until > conanfile.py approach covers installing all these dependencies. A valid > verification of the MiNiFi C++ build would be to run GTESTs and if all the > expected test cases pass, then it may be fair to conclude we would be ready > to deprecate the slow CMake FetchContent_Declare(...) approach. > > *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into > script(s)):* > # Install conan version 2 into MiNiFi C++ dev environment (my preference is > using docker container) > # Create a conanfile.py file at root of MiNiFi C++ project where we run > "conan install" or "conan install --build=missing" to install MiNiFi C++ > external lib dependencies first. Then we run CMake to build MiNiFi C++. > # Find all MiNiFi C++ find_package(...) CMake function calls and check if > there is already supported conan packages on the conancenter, so we can > install those packages. > # Find all FetchContent_Declare(...) CMake function calls and check if there > is already supported conan packages on the conancenter, so we can install > those packages. > # At this point we will have used conan version 2 C++ package manager to > install almost all MiNiFi C++'s external dependencies. > ## With Conan, we will have installed these MiNiFi external lib dependencies > as prebuilt binary conan packages. > ## Or we will have instructed conan to first build each C++ external lib > dependency, upload each lib to conancenter. > # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so > we can then run CMake to build MiNiFi faster. > > Here is my draft PR for this Jira Ticket for your reference: > [https://github.com/apache/nifi-minifi-cpp/pull/1775] > Here are the commands that I run: > > {code:java} > cd $HOME/src/pipeline/nifi-minifi-cpp > # NOTE: I also plan to add smoother integration of conan commands into MiNiFI > CPP too > # For instance, I know you guys have your bootstrap scripts too > # You also have your github actions workflow scripts too > conan install . --output-folder=build > -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux > # install conan packages and build C++ for minifi using Conan & CMake, we run > this way, so conan's tc.cache_variables > # are like passing -D{minifi_option} upon running cmake generate, then > make to build MiNiFi C++ > conan build . --output-folder=build > -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code} > > > *UPDATE (May 11, 2024)* - > [c96d430|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/c96d430814a067a6a538f4324337b719cb75bf48]: > Refactoring MiNiFi C++ build infrastructure adding Conan Packager support > *Enabled JNI, SENSORS, USB_CAMERA, OPENCV, and SFTP,* which required creating > *Maven 3.9.6 conan package* updating the conan recipe to expose the Maven > Executable environment variable in CMake and *installing prebuilt Java > OpenJDK 21.0.2 conan package* and creating a python wrapper run_maven.py > script for calling the conan maven Java build tool to successfully build Java > SFTP Test Server for SFTP extension testing and maven to build {*}Apache NiFi > 1.9.0 JNI Assembly and Framework{*}; We had to go the python maven wrapper > script approach since I ran into some issues calling conan maven executable > from CMake directly; Also added prebuilt install for libssh2 conan package > toward SFTP extensions; Thus, now the Java support for MiNiFi C++ is less > reliant on the user’s system since we can handle building this part of MiNiFi > C++ using conan; Also created an OpenCV 4.8.1 conan package updating the > conan recipe to use MiNiFi C++ conan package dependencies from libtiff, > ffmpeg and xz_utils (liblzma) and successfully build MiNiFi OpenCV extension; > Also installed prebuilt libuvc conan package for enabling USB_CAMERA. For the > conan build approach, 99% (235/236) of GTESTs passed. For the standalone > CMake build approach, 99% (234/236) of GTESTs passed. > * Also we could look at updating Apache NiFi 1.9.0 to latest 2.0 version. I > noticed we use a Java SFTP Test Server now that NiFi has native python custom > extensibility, maybe we could look at Python SFTP Test Server? > > *UPDATE (May 11, 2024)* - > [64dd8ad|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/64dd8ad14d2ddcea5eaa12da14a0e1b8a7289bb8]: > {*}Enabled Build LUA SCRIPTING, ENCRYPT CONFIG, SPLUNK, ELASTIC SEARCH, TEST > PROCESSORS, GRANFANA LOKI and CONTROLLER by Conan & CMake{*}. For building > these components with conan, the LUA SCRIPTING extension required me updating > its Lua.cmake and Sol2.cmake files, so we can use the conan packages in CMake > when USE_CONAN_PACKAGE condition is true. For lua and sol2 conan packages, I > was able to just install the prebuilt binary conan packages from conancenter > directly since conancenter provided the versions that match with MiNiFi C++'s > standalone CMake approach's lua 5.4.6 and sol2 3.3.0 source built versions. > MiNiFi controller being enabled required updating the ArgParse.cmake file, so > we could also account for the condition when we build the controller using > conan, thus I added back using the argparse conan package. After enabling > these components, I was able to successfully build them for MiNiFi using > conan. I also verified backward compatibility for building these components > in standalone CMake worked too. We achieved similar results for GTESTs where > out of 230 tests, 99% of them passed for conan build and 100% passed for > standalone CMake build. > *UPDATE (May 11, 2024)* - > [f5cfb79|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/f5cfb792bed5d94bd3c55217ed8f32e777f741d1]: > {*}Enabled Build MQTT & PCAP MiNiFi PCAP Extension by Conan & CMake{*}. > Refactored the pcap extension CMake code moving the part that downloads and > builds pcapplusplus over to its own PcapPlusPlus.cmake file. Also added conan > build condition to that .cmake file, so we can build PcapPlusPlus using conan > while still keeping backward compatibility using standalone CMake approach. > While I was able to build MiNiFi C++ with PCAP Extension enabled using > standalone CMake approach, when I built it using conan build approach, I > found pcap extension complaining about libminifi core headers and realized > include_directories(include) didnt share all the header directory paths from > libminifi to pcap extension. Thus, I created a python script under > libminifi/cmake/python to parse for the libminifi header directories > generating a include_dirs.cmake file with the LIBMINIFI_INCLUDE_DIRS CMake > variable list that gets stored in build folder and followed by > libminifi/include_dirs.cmake. That way we keep our source libminifi directory > and its subdirectories clean. Overall, I was able to build MiNiFi C++ with > MQTT and PCAP extensions enabled using conan & cmake. conan build achieved > 99% (222/223) of GTESTs passing. standalone cmake build achieved 100% of > GTESTs passing. > > *UPDATE (May 9, 2024)* - > {*}[63290db|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/63290db57b60054ba8ed32c0b3a7d1f87f315314]{*}: > *Refactored IODBC and SOCI libraries, so they are now CMake Bundles for > easier integration with Conan.* Thus, minifi-sql library now calls > include(BundledIODBC), use_bundled_iodbc(...) and include(BundledSOCI) and > use_bundled_soci(...). Verified I can still build MiNiFi C++ using standalone > CMake. 100% (220/220) of GTESTS passed using standalone CMake. Integrated > conan for building SOCI & ODBC for MiNiFi SQL extension. > *[4232399|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/4232399657c9da42dff57ff2b6806aeea2485444]* > *Added a SOCI conan recipe to create SOCI conan package with unixODBC conan > package, so SOCI comes with targets SOCI::libsoci_core and > SOCI::lib;soci_odbc.* Verified I can build MiNiFi C++'s minifi-sql extension > using conan build. However, with the conan build approach for minifi-sql > extension, I was only able to build minifi-sql as a STATIC library > succesfully. When I tried building minifi-sql as a SHARED library, I ran into > multiple undefined references to SOCI functions. I think to fix the > minifi-sql SHARED lib with conan will require making further updates to the > SOCI conanfile.py's package_info() method. After building MiNiFi with conan > including minifi-sql, I ran GTESTs and 99% of tests passed. I will look > further into minifi-sql SHARED library to bring that approach later. For now, > at least we can build minifi-sql as a STATIC library. > > *UPDATE (May 9, 2024)* - > [ca3d6df|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/ca3d6df318162e2ee9d3e76e173d9f83e17d3010]: > {*}Enabled MiNiFi C++ extensions GPS and COAP{*}. I added conan recipe for > libcoap keeping the windows cmake patches for minifi. Since the original > libcoap conan recipe's conanfile.py on conan-center-index was tailored for > CMake building libcoap versions > 4.3.0, I modified the recipe, so we can > build libcoap 4.2.1, which didnt have CMake support yet and thus, I updated > that conanfile.py to run 'autogen.sh' to create the configure script, I > passed the configure args to the configure script and then I used conan's > Autotools() to run the configure script and then ran the generated make file > to build libcoap 4.2.1. I updated BundledLibCoAP to account for the conan > way, but kept backward compatibility for standalone CMake way. For a later > update when I test on Windows, I need to add support into the libcoap 4.2.1 > conanfile.py for adding and applying the windows cmake patch to build > libcoap. For now, I am mainly testing this new conan build infra for MiNiFi > on Ubuntu 22.04 docker container and backward compatibility for standalone > CMake on Ubuntu too. 99% of GTESTs passed (215/216). > > *(UPDATE (May 8, 2024) -* > *[93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:* > {color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi > C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added > conan recipe for liblzma (xy_utils) and libarchive keeping the patches for > minifi. One area I found an issue with was integrating openssl conan package > with libarchive conan package and libarchive complained about not being able > to find openssl::openssl target. I did try to solve the issue by updating the > libarchive conan recipe, but wasn't able to solve that openssl integration > issue. I think the problem comes from libarchive CMakeLists.txt and its > integration with conan in the repo. I opened an issue on conan-center-index > to help with resolving this issue in case I get around to solving it in time. > Good news is that I was still able to create libarchive conan package with > the configuration settings similar to how we do it the standalone CMake > BundledLibArchive.cmake way just in libarchive conan recipe. I also verified > I can build these minifi extensions (LibArchive, LZMA, Procfs, Systemd) and > NANOFI using conan build and standalone CMake. The standalone CMake way still > allows us to build LibArchive with OpenSSL enabled. 213/214 (99%) GTESTs > passed after building MiNiFi C++ with conan.{color} > > *UPDATE (May 7, 2024) - > [*2460e84*|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2460e846832d693c991e1da632b86ba2d4e34516]:* > *Enabled OPS and OPC* MiNiFi Options, so those *MiNiFi C++ extensions* are > built using *Conan CMake* and their bundled CMake dependencies use their > particular *{{open62541/1.3.3@minifi/dev}}* and > {*}{{mbedtls/2.16.3@minifi/dev}} conan packages{*}. I had to fix the mbedtls > conan package from conan center index since it wasnt compatible when > open62541 tried to integrate with mbedtls. In the thirdparty folder, you will > see on line 139 where I updated libembedtls to be libmbedtls and fixing that > typo allowed open62541 conan package to be able to be created with mbedtls as > the encryption. I plan to make a follow up PR on conan center index for > mbedtls conan recipe. Currently, we are able to build MiNiFi's additional > extensions for OPS and OPC using conan. Verified backward compatibility with > standalone CMake works, so I can also just build MiNiFi using CMake only. > > *UPDATE (May 5, 2024) - > [446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:* > When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and > *Standard-Processors* and the other relevant that needed to be activated for > these 3 essential components, I also set SKIP_TESTS to OFF, I set > ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs > using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for > conan approach while keeping backward compatibility with standalone CMake > approach. I got the following error coming from extension > robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I > will add the fix for this issue: > {code:java} > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26: > error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and > ‘rocksdb::DBOptions’) > 71 | if (db_opts_copy != db_options_) { > | ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~ > | | | > | | rocksdb::DBOptions > | rocksdb::DBOptions {code} > We ran into this issue since when we created rocksdb/8.10.2 conan package, we > didn't apply the 2 patches like was done in the standalone CMake approach, so > I updated our conan approach where I created a rocksdb conan recipe in the > nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and > applied them to create the rocksdb conan package successfully. Also I do want > to note that I referenced the rocksdb conan recipe from conan center index > repo. After creating the rocksdb conan package, then I built MiNiFi C++ > GTESTs for MiNiFi Core, MainExe, Standard-Processors and the other relevant > extensions that needed to be activated for these 3 essential components using > conan build. Here we have the results after running ctest in the build folder: > {code:java} > 192/194 Test #192: ConfigFileEncryptorTests .................... Passed > 0.02 sec > Start 193: ConfigFileTests > 193/194 Test #193: ConfigFileTests ............................. Passed > 0.01 sec > Start 194: ControllerTests > 194/194 Test #194: ControllerTests .............................***Failed > 0.52 sec > 99% tests passed, 2 tests failed out of 194 > Total Test time (real) = 382.89 sec > The following tests FAILED: 45 - LoggerTests (Failed) 194 - > ControllerTests (Failed)Errors while running CTestOutput from these tests are > in: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse > "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. > {code} > *We can see that 99% of the tests passed. Thus, 192/194 tests passed. This > was using conan to build MiNiFi C++.* > > I also verified I can build *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and > *Standard-Processors* and the other relevant that needed to be activated for > these 3 essential components, I also set SKIP_TESTS to OFF, I set > ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, BUILD_ROCKSDB to ON, > ENABLE_ROCKSDB to ON to {*}build GTESTs using standalone CMake{*}. Here we > can see similar results with running the MiNiFi GESTs: > > {code:java} > 192/194 Test #192: ConfigFileEncryptorTests .................... Passed > 0.07 sec > Start 193: ConfigFileTests > 193/194 Test #193: ConfigFileTests ............................. Passed > 0.08 sec > Start 194: ControllerTests > 194/194 Test #194: ControllerTests .............................***Failed > 0.76 sec > 99% tests passed, 1 tests failed out of 194 > Total Test time (real) = 236.38 sec > The following tests FAILED: 194 - ControllerTests (Failed)Errors while > running CTestOutput from these tests are in: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse > "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. > {code} > We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was > using standalone CMake to build MiNiFi C++. > > {*}{color:#0747a6}UPDATE (May 4, 2024) - > {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}: > Verified I can build MiNiFi C++ with standalone CMake like how we currently > do it even after integrating conan into the build infrastructure. CMake built > *MiNiFi Core, Main* and *Standard Processors* successfully. Also double > checked that conan build MiNiFi still works. (Feel free to reference the logs > for the CMake standalone build and the conan build of MiNiFi CPP) > > {color:#0747a6}{*}UPDATE (May 4, 2024) - > [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color} > Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi* > and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*} > extension with minimal set of external libraries needed. If you look at our > conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a > tuple of the minimal external libraries needed to build these two essential > building blocks of MiNFi CPP. Next when you look at conanfile.py > *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled > USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external > library dependencies that aren needed for the *core-minifi* library and > *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, > ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are > in the extensions folder and add some dependencies on these custom extensions > that are created. {color:#ff8b00}*I think we should consider creating an > extensions subfolder for core-minifi related extensions.*{color} > Here is the output I get after building the MiNiFi CPP code, you can see > MiNiFi CPP libraries and binary executables built successfully using *CONAN > version 2* and CMake (feel free to check the attached log for the full conan > build output): [~szaszm] > > {code:java} > Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list > Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT > list > Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT > -- Creating API Documentation using /usr/bin/doxygen > -- Configuring done > -- Generating done > -- Build files have been written to: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py > (minifi-cpp-main/0.15.0): Running CMake.build() > conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build > "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20 > [ 3%] Built target ossp-uuid-external-build > [ 6%] Built target openwsman-external > [ 7%] Built target ossp-uuid-external > Consolidate compiler generated dependencies of target core-minifi > [ 8%] Building CXX object > libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o > [ 8%] Linking CXX shared library ../bin/libcore-minifi.so > [ 78%] Built target core-minifi > Consolidate compiler generated dependencies of target minifi-civet-extensions > Consolidate compiler generated dependencies of target minifi-openwsman > Consolidate compiler generated dependencies of target minifi-http-curl > Consolidate compiler generated dependencies of target minifiexe > [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so > [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so > Consolidate compiler generated dependencies of target > minifi-standard-processors > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o > [ 79%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o > [ 80%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o > [ 81%] Building CXX object > minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o > [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so > [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so > [ 84%] Built target minifi-civet-extensions > [ 86%] Built target minifi-http-curl > [ 87%] Built target minifi-openwsman > [100%] Built target minifi-standard-processors > [100%] Linking CXX executable ../bin/minifi > [100%] Built target minifiexe {code} > NEXT: I will verify I can build these MiNiFi CPP libraries and binary > executables using USE_CMAKE_FETCH_CONTENT option set to ON and > USE_CONAN_PACKAGER option set to OFF for the for the follow up commit. > > Tracking Prebuilt Binary Conan Packages Available on Conancenter: > * 'abseil/20230125.3', > * 'argparse/3.0', > * 'asio/1.30.2' > Tracking Conan Packages Not Available on Conancenter, so we need to use an > alternative hosting service to manage building package sources into prebuilt > binary conan packages. Here are some of the free conan hosting services for > open-source C/C++ projects: > * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan > packages maintained by conan team > * Bincrafters Bintray Repo: community-driven repo that hosts large number of > pre-built conan packages for open-source libraries for free > * Artifactory by JFrog: typically used as a commercial hosting service, but > also provides a free tier for open-source projects. > * {*}Self-Hosting{*}: While there are multiple options, I will focus on > *GitHub Packages* as its a free package hosting service, which can be used to > host our own conan package repos, allowing us to store and distribute our > pre-built conan packages without having to use 3rd party services like > Bincrafters or pay for commercial solution. > ** {*}Advantage with GitHub Packages{*}: its free, integrated with our > existing GitHub workflow, allows us full control hosting our conan packages > without relying on 3rd party services, useful if we need to have specific > requirements or need flexibility in managing our package hosting. > Tracking Non Prebuilt Binary Conan Packages, which later I will create a > *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account > for Linux, Windows, Mac, etc): > * bustache: [https://github.com/jamboree/bustache] > > {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP > after I finish my work for MINFICPP-2346: While I am working on migrating > MiNiFi CPP build infrastructure toward leveraging conan packager, for a later > PR we could do the following: > # Put the external libraries that MiNiFi CPP Core depends on into its own > *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other > names), we can run this CMake code before adding *add_subdirectory(libminifi)* > ## External libs that libminifi needs that are in MiNiFi CPP's root > CMakeLists.txt (refactor for better organization): > yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads, > gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio, > magic_enum, RangeV3, Asio, MagicEnum > > ## We can also clean up the *thirdparty* folder as we migrate to conan > packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep > it MiNiFi C{+}+ code and its extension processors, etc. > # I tried building MiNiFi CPP Core just with just > *minifi-standard-processors* and the other extensions disabled, but it seems > even some of MiNiFi CPP Core depends on some of the extensions to be built > with it, so it doesn't fail building, such as Openwsman. If we don't have > ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we > should look having a build config to build minimal MiNiFi CPP Core: > {code:java} > libminifi/src/controllers/SSLContextService.cpp:520:80: error: > ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code} > > # I noticed that when building MiNiFi Core, MainExe, Standard-Processors and > their external lib dependencies and then building the appropriate GTEST, I > get a header error coming from rocksdb, but I haven't enabled rocksdb, so I > think at a later point we should shift that test out, so when building these > 3 core building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi > Core and MainExe have a dependency for storing into that DB. Here is an > example of the header error (I will ENABLE_ROCKSDB to ON): > {code:java} > [ 99%] Built target ControllerTests > In file included from > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30, > from > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24: > /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10: > fatal error: rocksdb/db.h: No such file or directory > 26 | #include "rocksdb/db.h" {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)