[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
--------------------------------------------
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
*UPDATE (May 21, 2024):* I created a small PR for conan integration on top of
Gabor Gyimesi's PR-1749 "https://github.com/apache/nifi-minifi-cpp/pull/1749";
that brings in conan packages for OpenSSL, LibCurl and Zlib as alternatives to
their system library counterparts when we run standalone CMake to build MiNiFi
CPP. I also included steps for Maton Szasz and others to get up to speed with
how to use conan v2 to build MiNiFi CPP. On top of Gabor's PR-1749, my *PR-261*
has +325 -225 git diff and only 5 files changed:
https://github.com/lordgamez/nifi-minifi-cpp/pull/261
*UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the refactoring
of MiNiFi CPP build infra over to conan version 2 is taking longer than
expected. The good news is that I believe I have more than half of MiNiFi
external libraries now managed by conan with backward compatibility for
standalone CMake approach.
*UPDATE (May 13, 2024)* -
*[22205c9|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/22205c98cef30ee1ebe252880ae113724589bd78]:*
Fetch and Pulled latest upstream main branch of parent MiNiFi to my fork main
branch of MiNiFi repo; thus, merged in the latest updates from main branch of
parent MiNiFi to my MINIFICPP-2346 branch. Resolved conflicts between a few
CMake files coming from Date.cmake and BundledRocksdb.cmake files. For
standalone CMake build appraoch, ran GTESTs with 100% (238/238) tests passing.
I did the conan build approach and ran GTESTs with 99% (237/238) passing.
*UPDATE (May 13, 2024)* -
[59ffefb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/59ffefb1caf662874054f55f7171692c6b9d345f]:
{*}Enabled LIBRDKAFKA & GRPC_FOR_LOKI MiNiFi C++ extension using Conan &
CMake{*}, *Improved Running Python Maven Executable for JNI & SFTP Test Server
for conan approach* and {*}Improved standalone CMake approach for building
PcapPlusPlus with its dependency on LibPcap{*}. LIBRDKAFKA required creating a
librdkafka conan package passing similar CMake definitions to it to build it
like we did for standalone CMake approach. GRPC_FOR_LOKI required creating a
grpc conan package that uses minifi's openssl and libsystemd conan packages. In
both LIBRDKAFKA & GRPC_FOR_LOKI, I updated their .cmake Bundles to account for
the condition of using their conan package, so minimal changes would be needed
to their extension CMakeLists.txt file. In LIBRDKAFKA's extension
CMakeLists.txt file, it didnt really change after integrating the conan package
since I created a library alias for it similar to the standalone CMake
approach. In GRPC_FOR_LOKI's extension CMakeLists.txt, minimal changes were
made to that file mainly on the add_custom_command, target_include_directories
and target_link_libraries commands. You'll see for Grpc.cmake for conan
approach, I created similar variables to standalone CMake approach for
GRPC_CPP_PLUGIN, INCLUDE_DIR, PROTOBUF_INCLUDE_DIR, PROTOBUF_COMPILER getting
the conan related properties. For the conan side, I also made improvements to
the Python Maven Executable Wrapper script since I noticed there were times at
the beginning when running maven version test, it didn't always run it
successfully, so I updated conan maven executable path to be explicit for
passing to Python script. I then moved the location of running maven version
check into a CMake function and ran it after we included JavaMaven and
JavaOpenJDK conan packages in the MiNiFi root CMakeLists.txt file. Thus, it
succesfully runs maven version check at beginning each time. Its important to
make sure we have conan maven to successfully build NiFi JNI and Java SFTP Test
Server. Finally, on the standalone CMake approach for building PcapPlusPlus, I
noticed some issues with PCAP headers not being found when building
PcapPlusPlus, so I switched from execute_process to run the configure scripts
to using ExternalProject_Add and I also added a bundle for LibPcap since the
system LibPcap version wasn't always installed on the system. This problem with
building PcapPlusPlus dependency on LibPcap didn't happen with my conan
approach. It took a lot of time to fix the standalone CMake backward
compatibility approach, but I did it for now since we're keeping support that
approach. 99% of GTESTs ( 237/238) passed for conan.
*UPDATE (May 11, 2024)* -
[c96d430|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/c96d430814a067a6a538f4324337b719cb75bf48]:
Refactoring MiNiFi C++ build infrastructure adding Conan Packager support
*Enabled JNI, SENSORS, USB_CAMERA, OPENCV, and SFTP,* which required creating
*Maven 3.9.6 conan package* updating the conan recipe to expose the Maven
Executable environment variable in CMake and *installing prebuilt Java OpenJDK
21.0.2 conan package* and creating a python wrapper run_maven.py script for
calling the conan maven Java build tool to successfully build Java SFTP Test
Server for SFTP extension testing and maven to build {*}Apache NiFi 1.9.0 JNI
Assembly and Framework{*}; We had to go the python maven wrapper script
approach since I ran into some issues calling conan maven executable from CMake
directly; Also added prebuilt install for libssh2 conan package toward SFTP
extensions; Thus, now the Java support for MiNiFi C++ is less reliant on the
user’s system since we can handle building this part of MiNiFi C++ using conan;
Also created an OpenCV 4.8.1 conan package updating the conan recipe to use
MiNiFi C++ conan package dependencies from libtiff, ffmpeg and xz_utils
(liblzma) and successfully build MiNiFi OpenCV extension; Also installed
prebuilt libuvc conan package for enabling USB_CAMERA. For the conan build
approach, 99% (235/236) of GTESTs passed. For the standalone CMake build
approach, 99% (234/236) of GTESTs passed.
* Also we could look at updating Apache NiFi 1.9.0 to latest 2.0 version. I
noticed we use a Java SFTP Test Server now that NiFi has native python custom
extensibility, maybe we could look at Python SFTP Test Server?
*UPDATE (May 11, 2024)* -
[64dd8ad|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/64dd8ad14d2ddcea5eaa12da14a0e1b8a7289bb8]:
{*}Enabled Build LUA SCRIPTING, ENCRYPT CONFIG, SPLUNK, ELASTIC SEARCH, TEST
PROCESSORS, GRANFANA LOKI and CONTROLLER by Conan & CMake{*}. For building
these components with conan, the LUA SCRIPTING extension required me updating
its Lua.cmake and Sol2.cmake files, so we can use the conan packages in CMake
when USE_CONAN_PACKAGE condition is true. For lua and sol2 conan packages, I
was able to just install the prebuilt binary conan packages from conancenter
directly since conancenter provided the versions that match with MiNiFi C++'s
standalone CMake approach's lua 5.4.6 and sol2 3.3.0 source built versions.
MiNiFi controller being enabled required updating the ArgParse.cmake file, so
we could also account for the condition when we build the controller using
conan, thus I added back using the argparse conan package. After enabling these
components, I was able to successfully build them for MiNiFi using conan. I
also verified backward compatibility for building these components in
standalone CMake worked too. We achieved similar results for GTESTs where out
of 230 tests, 99% of them passed for conan build and 100% passed for standalone
CMake build.
*UPDATE (May 11, 2024)* -
[f5cfb79|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/f5cfb792bed5d94bd3c55217ed8f32e777f741d1]:
{*}Enabled Build MQTT & PCAP MiNiFi PCAP Extension by Conan & CMake{*}.
Refactored the pcap extension CMake code moving the part that downloads and
builds pcapplusplus over to its own PcapPlusPlus.cmake file. Also added conan
build condition to that .cmake file, so we can build PcapPlusPlus using conan
while still keeping backward compatibility using standalone CMake approach.
While I was able to build MiNiFi C++ with PCAP Extension enabled using
standalone CMake approach, when I built it using conan build approach, I found
pcap extension complaining about libminifi core headers and realized
include_directories(include) didnt share all the header directory paths from
libminifi to pcap extension. Thus, I created a python script under
libminifi/cmake/python to parse for the libminifi header directories generating
a include_dirs.cmake file with the LIBMINIFI_INCLUDE_DIRS CMake variable list
that gets stored in build folder and followed by libminifi/include_dirs.cmake.
That way we keep our source libminifi directory and its subdirectories clean.
Overall, I was able to build MiNiFi C++ with MQTT and PCAP extensions enabled
using conan & cmake. conan build achieved 99% (222/223) of GTESTs passing.
standalone cmake build achieved 100% of GTESTs passing.
*UPDATE (May 9, 2024)* -
{*}[63290db|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/63290db57b60054ba8ed32c0b3a7d1f87f315314]{*}:
*Refactored IODBC and SOCI libraries, so they are now CMake Bundles for easier
integration with Conan.* Thus, minifi-sql library now calls
include(BundledIODBC), use_bundled_iodbc(...) and include(BundledSOCI) and
use_bundled_soci(...). Verified I can still build MiNiFi C++ using standalone
CMake. 100% (220/220) of GTESTS passed using standalone CMake. Integrated conan
for building SOCI & ODBC for MiNiFi SQL extension.
*[4232399|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/4232399657c9da42dff57ff2b6806aeea2485444]*
*Added a SOCI conan recipe to create SOCI conan package with unixODBC conan
package, so SOCI comes with targets SOCI::libsoci_core and
SOCI::lib;soci_odbc.* Verified I can build MiNiFi C++'s minifi-sql extension
using conan build. However, with the conan build approach for minifi-sql
extension, I was only able to build minifi-sql as a STATIC library succesfully.
When I tried building minifi-sql as a SHARED library, I ran into multiple
undefined references to SOCI functions. I think to fix the minifi-sql SHARED
lib with conan will require making further updates to the SOCI conanfile.py's
package_info() method. After building MiNiFi with conan including minifi-sql, I
ran GTESTs and 99% of tests passed. I will look further into minifi-sql SHARED
library to bring that approach later. For now, at least we can build minifi-sql
as a STATIC library.
*UPDATE (May 9, 2024)* -
[ca3d6df|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/ca3d6df318162e2ee9d3e76e173d9f83e17d3010]:
{*}Enabled MiNiFi C++ extensions GPS and COAP{*}. I added conan recipe for
libcoap keeping the windows cmake patches for minifi. Since the original
libcoap conan recipe's conanfile.py on conan-center-index was tailored for
CMake building libcoap versions > 4.3.0, I modified the recipe, so we can build
libcoap 4.2.1, which didnt have CMake support yet and thus, I updated that
conanfile.py to run 'autogen.sh' to create the configure script, I passed the
configure args to the configure script and then I used conan's Autotools() to
run the configure script and then ran the generated make file to build libcoap
4.2.1. I updated BundledLibCoAP to account for the conan way, but kept backward
compatibility for standalone CMake way. For a later update when I test on
Windows, I need to add support into the libcoap 4.2.1 conanfile.py for adding
and applying the windows cmake patch to build libcoap. For now, I am mainly
testing this new conan build infra for MiNiFi on Ubuntu 22.04 docker container
and backward compatibility for standalone CMake on Ubuntu too. 99% of GTESTs
passed (215/216).
*(UPDATE (May 8, 2024) -*
*[93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:*
{color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi
C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added
conan recipe for liblzma (xy_utils) and libarchive keeping the patches for
minifi. One area I found an issue with was integrating openssl conan package
with libarchive conan package and libarchive complained about not being able to
find openssl::openssl target. I did try to solve the issue by updating the
libarchive conan recipe, but wasn't able to solve that openssl integration
issue. I think the problem comes from libarchive CMakeLists.txt and its
integration with conan in the repo. I opened an issue on conan-center-index to
help with resolving this issue in case I get around to solving it in time. Good
news is that I was still able to create libarchive conan package with the
configuration settings similar to how we do it the standalone CMake
BundledLibArchive.cmake way just in libarchive conan recipe. I also verified I
can build these minifi extensions (LibArchive, LZMA, Procfs, Systemd) and
NANOFI using conan build and standalone CMake. The standalone CMake way still
allows us to build LibArchive with OpenSSL enabled. 213/214 (99%) GTESTs passed
after building MiNiFi C++ with conan.{color}
*UPDATE (May 7, 2024) -
[*2460e84*|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2460e846832d693c991e1da632b86ba2d4e34516]:*
*Enabled OPS and OPC* MiNiFi Options, so those *MiNiFi C++ extensions* are
built using *Conan CMake* and their bundled CMake dependencies use their
particular *{{open62541/1.3.3@minifi/dev}}* and
{*}{{mbedtls/2.16.3@minifi/dev}} conan packages{*}. I had to fix the mbedtls
conan package from conan center index since it wasnt compatible when open62541
tried to integrate with mbedtls. In the thirdparty folder, you will see on line
139 where I updated libembedtls to be libmbedtls and fixing that typo allowed
open62541 conan package to be able to be created with mbedtls as the
encryption. I plan to make a follow up PR on conan center index for mbedtls
conan recipe. Currently, we are able to build MiNiFi's additional extensions
for OPS and OPC using conan. Verified backward compatibility with standalone
CMake works, so I can also just build MiNiFi using CMake only.
*UPDATE (May 5, 2024) -
[446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:*
When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and
*Standard-Processors* and the other relevant that needed to be activated for
these 3 essential components, I also set SKIP_TESTS to OFF, I set
ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs
using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for
conan approach while keeping backward compatibility with standalone CMake
approach. I got the following error coming from extension
robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I will
add the fix for this issue:
{code:java}
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26:
error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and
‘rocksdb::DBOptions’)
71 | if (db_opts_copy != db_options_) {
| ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~
| | |
| | rocksdb::DBOptions
| rocksdb::DBOptions {code}
We ran into this issue since when we created rocksdb/8.10.2 conan package, we
didn't apply the 2 patches like was done in the standalone CMake approach, so I
updated our conan approach where I created a rocksdb conan recipe in the
nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and applied
them to create the rocksdb conan package successfully. Also I do want to note
that I referenced the rocksdb conan recipe from conan center index repo. After
creating the rocksdb conan package, then I built MiNiFi C++ GTESTs for MiNiFi
Core, MainExe, Standard-Processors and the other relevant extensions that
needed to be activated for these 3 essential components using conan build. Here
we have the results after running ctest in the build folder:
{code:java}
192/194 Test #192: ConfigFileEncryptorTests .................... Passed
0.02 sec
Start 193: ConfigFileTests
193/194 Test #193: ConfigFileTests ............................. Passed
0.01 sec
Start 194: ControllerTests
194/194 Test #194: ControllerTests .............................***Failed
0.52 sec
99% tests passed, 2 tests failed out of 194
Total Test time (real) = 382.89 sec
The following tests FAILED: 45 - LoggerTests (Failed) 194 -
ControllerTests (Failed)Errors while running CTestOutput from these tests are
in:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse
"--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
{code}
*We can see that 99% of the tests passed. Thus, 192/194 tests passed. This was
using conan to build MiNiFi C++.*
I also verified I can build *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and
*Standard-Processors* and the other relevant that needed to be activated for
these 3 essential components, I also set SKIP_TESTS to OFF, I set
ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, BUILD_ROCKSDB to ON,
ENABLE_ROCKSDB to ON to {*}build GTESTs using standalone CMake{*}. Here we can
see similar results with running the MiNiFi GESTs:
{code:java}
192/194 Test #192: ConfigFileEncryptorTests .................... Passed
0.07 sec
Start 193: ConfigFileTests
193/194 Test #193: ConfigFileTests ............................. Passed
0.08 sec
Start 194: ControllerTests
194/194 Test #194: ControllerTests .............................***Failed
0.76 sec
99% tests passed, 1 tests failed out of 194
Total Test time (real) = 236.38 sec
The following tests FAILED: 194 - ControllerTests (Failed)Errors while
running CTestOutput from these tests are in:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse
"--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
{code}
We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was
using standalone CMake to build MiNiFi C++.
{*}{color:#0747a6}UPDATE (May 4, 2024) -
{color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
Verified I can build MiNiFi C++ with standalone CMake like how we currently do
it even after integrating conan into the build infrastructure. CMake built
*MiNiFi Core, Main* and *Standard Processors* successfully. Also double checked
that conan build MiNiFi still works. (Feel free to reference the logs for the
CMake standalone build and the conan build of MiNiFi CPP)
{color:#0747a6}{*}UPDATE (May 4, 2024) -
[2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color}
Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi*
and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*}
extension with minimal set of external libraries needed. If you look at our
conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a tuple
of the minimal external libraries needed to build these two essential building
blocks of MiNFi CPP. Next when you look at conanfile.py *MiNiFiCppMain* class's
generate(self) method, you'll see I have enabled USE_CONAN_PACKAGER to ON,
SKIP_TESTS to ON, and then the essential external library dependencies that
aren needed for the *core-minifi* library and *minifiexe* binary executable,
which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, ENABLE_CIVET, ENABLE_CURL{*}. I
notice most of these external libraries are in the extensions folder and add
some dependencies on these custom extensions that are created.
{color:#ff8b00}*I think we should consider creating an extensions subfolder for
core-minifi related extensions.*{color}
Here is the output I get after building the MiNiFi CPP code, you can see MiNiFi
CPP libraries and binary executables built successfully using *CONAN version 2*
and CMake (feel free to check the attached log for the full conan build
output): [~szaszm]
{code:java}
Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list
Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT list
Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT
-- Creating API Documentation using /usr/bin/doxygen
-- Configuring done
-- Generating done
-- Build files have been written to:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py
(minifi-cpp-main/0.15.0): Running CMake.build()
conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build
"/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20
[ 3%] Built target ossp-uuid-external-build
[ 6%] Built target openwsman-external
[ 7%] Built target ossp-uuid-external
Consolidate compiler generated dependencies of target core-minifi
[ 8%] Building CXX object
libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o
[ 8%] Linking CXX shared library ../bin/libcore-minifi.so
[ 78%] Built target core-minifi
Consolidate compiler generated dependencies of target minifi-civet-extensions
Consolidate compiler generated dependencies of target minifi-openwsman
Consolidate compiler generated dependencies of target minifi-http-curl
Consolidate compiler generated dependencies of target minifiexe
[ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so
[ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so
Consolidate compiler generated dependencies of target minifi-standard-processors
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o
[ 80%] Building CXX object
minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o
[ 81%] Building CXX object
minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o
[ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so
[ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so
[ 84%] Built target minifi-civet-extensions
[ 86%] Built target minifi-http-curl
[ 87%] Built target minifi-openwsman
[100%] Built target minifi-standard-processors
[100%] Linking CXX executable ../bin/minifi
[100%] Built target minifiexe {code}
NEXT: I will verify I can build these MiNiFi CPP libraries and binary
executables using USE_CMAKE_FETCH_CONTENT option set to ON and
USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
Tracking Prebuilt Binary Conan Packages Available on Conancenter:
* 'abseil/20230125.3',
* 'argparse/3.0',
* 'asio/1.30.2'
Tracking Conan Packages Not Available on Conancenter, so we need to use an
alternative hosting service to manage building package sources into prebuilt
binary conan packages. Here are some of the free conan hosting services for
open-source C/C++ projects:
* {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
packages maintained by conan team
* Bincrafters Bintray Repo: community-driven repo that hosts large number of
pre-built conan packages for open-source libraries for free
* Artifactory by JFrog: typically used as a commercial hosting service, but
also provides a free tier for open-source projects.
* {*}Self-Hosting{*}: While there are multiple options, I will focus on
*GitHub Packages* as its a free package hosting service, which can be used to
host our own conan package repos, allowing us to store and distribute our
pre-built conan packages without having to use 3rd party services like
Bincrafters or pay for commercial solution.
** {*}Advantage with GitHub Packages{*}: its free, integrated with our
existing GitHub workflow, allows us full control hosting our conan packages
without relying on 3rd party services, useful if we need to have specific
requirements or need flexibility in managing our package hosting.
Tracking Non Prebuilt Binary Conan Packages, which later I will create a
*GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account for
Linux, Windows, Mac, etc):
* bustache: [https://github.com/jamboree/bustache]
{*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
after I finish my work for MINFICPP-2346: While I am working on migrating
MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
PR we could do the following:
# Put the external libraries that MiNiFi CPP Core depends on into its own
*BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other names),
we can run this CMake code before adding *add_subdirectory(libminifi)*
## External libs that libminifi needs that are in MiNiFi CPP's root
CMakeLists.txt (refactor for better organization):
yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
magic_enum, RangeV3, Asio, MagicEnum
## We can also clean up the *thirdparty* folder as we migrate to conan
packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep it
MiNiFi C{+}+ code and its extension processors, etc.
# I tried building MiNiFi CPP Core just with just *minifi-standard-processors*
and the other extensions disabled, but it seems even some of MiNiFi CPP Core
depends on some of the extensions to be built with it, so it doesn't fail
building, such as Openwsman. If we don't have ENABLE_OPENWSMAN, then we get
this error when building MiNiFi CPP Core, so we should look having a build
config to build minimal MiNiFi CPP Core:
{code:java}
libminifi/src/controllers/SSLContextService.cpp:520:80: error:
‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
# I noticed that when building MiNiFi Core, MainExe, Standard-Processors and
their external lib dependencies and then building the appropriate GTEST, I get
a header error coming from rocksdb, but I haven't enabled rocksdb, so I think
at a later point we should shift that test out, so when building these 3 core
building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi Core and
MainExe have a dependency for storing into that DB. Here is an example of the
header error (I will ENABLE_ROCKSDB to ON):
{code:java}
[ 99%] Built target ControllerTests
In file included from
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30,
from
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10:
fatal error: rocksdb/db.h: No such file or directory
26 | #include "rocksdb/db.h" {code}
was:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
UPDATE (May 21, 2024): As
*UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the refactoring
of MiNiFi CPP build infra over to conan version 2 is taking longer than
expected. The good news is that I believe I have more than half of MiNiFi
external libraries now managed by conan with backward compatibility for
standalone CMake approach.
*UPDATE (May 13, 2024)* -
*[22205c9|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/22205c98cef30ee1ebe252880ae113724589bd78]:*
Fetch and Pulled latest upstream main branch of parent MiNiFi to my fork main
branch of MiNiFi repo; thus, merged in the latest updates from main branch of
parent MiNiFi to my MINIFICPP-2346 branch. Resolved conflicts between a few
CMake files coming from Date.cmake and BundledRocksdb.cmake files. For
standalone CMake build appraoch, ran GTESTs with 100% (238/238) tests passing.
I did the conan build approach and ran GTESTs with 99% (237/238) passing.
*UPDATE (May 13, 2024)* -
[59ffefb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/59ffefb1caf662874054f55f7171692c6b9d345f]:
{*}Enabled LIBRDKAFKA & GRPC_FOR_LOKI MiNiFi C++ extension using Conan &
CMake{*}, *Improved Running Python Maven Executable for JNI & SFTP Test Server
for conan approach* and {*}Improved standalone CMake approach for building
PcapPlusPlus with its dependency on LibPcap{*}. LIBRDKAFKA required creating a
librdkafka conan package passing similar CMake definitions to it to build it
like we did for standalone CMake approach. GRPC_FOR_LOKI required creating a
grpc conan package that uses minifi's openssl and libsystemd conan packages. In
both LIBRDKAFKA & GRPC_FOR_LOKI, I updated their .cmake Bundles to account for
the condition of using their conan package, so minimal changes would be needed
to their extension CMakeLists.txt file. In LIBRDKAFKA's extension
CMakeLists.txt file, it didnt really change after integrating the conan package
since I created a library alias for it similar to the standalone CMake
approach. In GRPC_FOR_LOKI's extension CMakeLists.txt, minimal changes were
made to that file mainly on the add_custom_command, target_include_directories
and target_link_libraries commands. You'll see for Grpc.cmake for conan
approach, I created similar variables to standalone CMake approach for
GRPC_CPP_PLUGIN, INCLUDE_DIR, PROTOBUF_INCLUDE_DIR, PROTOBUF_COMPILER getting
the conan related properties. For the conan side, I also made improvements to
the Python Maven Executable Wrapper script since I noticed there were times at
the beginning when running maven version test, it didn't always run it
successfully, so I updated conan maven executable path to be explicit for
passing to Python script. I then moved the location of running maven version
check into a CMake function and ran it after we included JavaMaven and
JavaOpenJDK conan packages in the MiNiFi root CMakeLists.txt file. Thus, it
succesfully runs maven version check at beginning each time. Its important to
make sure we have conan maven to successfully build NiFi JNI and Java SFTP Test
Server. Finally, on the standalone CMake approach for building PcapPlusPlus, I
noticed some issues with PCAP headers not being found when building
PcapPlusPlus, so I switched from execute_process to run the configure scripts
to using ExternalProject_Add and I also added a bundle for LibPcap since the
system LibPcap version wasn't always installed on the system. This problem with
building PcapPlusPlus dependency on LibPcap didn't happen with my conan
approach. It took a lot of time to fix the standalone CMake backward
compatibility approach, but I did it for now since we're keeping support that
approach. 99% of GTESTs ( 237/238) passed for conan.
*UPDATE (May 11, 2024)* -
[c96d430|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/c96d430814a067a6a538f4324337b719cb75bf48]:
Refactoring MiNiFi C++ build infrastructure adding Conan Packager support
*Enabled JNI, SENSORS, USB_CAMERA, OPENCV, and SFTP,* which required creating
*Maven 3.9.6 conan package* updating the conan recipe to expose the Maven
Executable environment variable in CMake and *installing prebuilt Java OpenJDK
21.0.2 conan package* and creating a python wrapper run_maven.py script for
calling the conan maven Java build tool to successfully build Java SFTP Test
Server for SFTP extension testing and maven to build {*}Apache NiFi 1.9.0 JNI
Assembly and Framework{*}; We had to go the python maven wrapper script
approach since I ran into some issues calling conan maven executable from CMake
directly; Also added prebuilt install for libssh2 conan package toward SFTP
extensions; Thus, now the Java support for MiNiFi C++ is less reliant on the
user’s system since we can handle building this part of MiNiFi C++ using conan;
Also created an OpenCV 4.8.1 conan package updating the conan recipe to use
MiNiFi C++ conan package dependencies from libtiff, ffmpeg and xz_utils
(liblzma) and successfully build MiNiFi OpenCV extension; Also installed
prebuilt libuvc conan package for enabling USB_CAMERA. For the conan build
approach, 99% (235/236) of GTESTs passed. For the standalone CMake build
approach, 99% (234/236) of GTESTs passed.
* Also we could look at updating Apache NiFi 1.9.0 to latest 2.0 version. I
noticed we use a Java SFTP Test Server now that NiFi has native python custom
extensibility, maybe we could look at Python SFTP Test Server?
*UPDATE (May 11, 2024)* -
[64dd8ad|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/64dd8ad14d2ddcea5eaa12da14a0e1b8a7289bb8]:
{*}Enabled Build LUA SCRIPTING, ENCRYPT CONFIG, SPLUNK, ELASTIC SEARCH, TEST
PROCESSORS, GRANFANA LOKI and CONTROLLER by Conan & CMake{*}. For building
these components with conan, the LUA SCRIPTING extension required me updating
its Lua.cmake and Sol2.cmake files, so we can use the conan packages in CMake
when USE_CONAN_PACKAGE condition is true. For lua and sol2 conan packages, I
was able to just install the prebuilt binary conan packages from conancenter
directly since conancenter provided the versions that match with MiNiFi C++'s
standalone CMake approach's lua 5.4.6 and sol2 3.3.0 source built versions.
MiNiFi controller being enabled required updating the ArgParse.cmake file, so
we could also account for the condition when we build the controller using
conan, thus I added back using the argparse conan package. After enabling these
components, I was able to successfully build them for MiNiFi using conan. I
also verified backward compatibility for building these components in
standalone CMake worked too. We achieved similar results for GTESTs where out
of 230 tests, 99% of them passed for conan build and 100% passed for standalone
CMake build.
*UPDATE (May 11, 2024)* -
[f5cfb79|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/f5cfb792bed5d94bd3c55217ed8f32e777f741d1]:
{*}Enabled Build MQTT & PCAP MiNiFi PCAP Extension by Conan & CMake{*}.
Refactored the pcap extension CMake code moving the part that downloads and
builds pcapplusplus over to its own PcapPlusPlus.cmake file. Also added conan
build condition to that .cmake file, so we can build PcapPlusPlus using conan
while still keeping backward compatibility using standalone CMake approach.
While I was able to build MiNiFi C++ with PCAP Extension enabled using
standalone CMake approach, when I built it using conan build approach, I found
pcap extension complaining about libminifi core headers and realized
include_directories(include) didnt share all the header directory paths from
libminifi to pcap extension. Thus, I created a python script under
libminifi/cmake/python to parse for the libminifi header directories generating
a include_dirs.cmake file with the LIBMINIFI_INCLUDE_DIRS CMake variable list
that gets stored in build folder and followed by libminifi/include_dirs.cmake.
That way we keep our source libminifi directory and its subdirectories clean.
Overall, I was able to build MiNiFi C++ with MQTT and PCAP extensions enabled
using conan & cmake. conan build achieved 99% (222/223) of GTESTs passing.
standalone cmake build achieved 100% of GTESTs passing.
*UPDATE (May 9, 2024)* -
{*}[63290db|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/63290db57b60054ba8ed32c0b3a7d1f87f315314]{*}:
*Refactored IODBC and SOCI libraries, so they are now CMake Bundles for easier
integration with Conan.* Thus, minifi-sql library now calls
include(BundledIODBC), use_bundled_iodbc(...) and include(BundledSOCI) and
use_bundled_soci(...). Verified I can still build MiNiFi C++ using standalone
CMake. 100% (220/220) of GTESTS passed using standalone CMake. Integrated conan
for building SOCI & ODBC for MiNiFi SQL extension.
*[4232399|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/4232399657c9da42dff57ff2b6806aeea2485444]*
*Added a SOCI conan recipe to create SOCI conan package with unixODBC conan
package, so SOCI comes with targets SOCI::libsoci_core and
SOCI::lib;soci_odbc.* Verified I can build MiNiFi C++'s minifi-sql extension
using conan build. However, with the conan build approach for minifi-sql
extension, I was only able to build minifi-sql as a STATIC library succesfully.
When I tried building minifi-sql as a SHARED library, I ran into multiple
undefined references to SOCI functions. I think to fix the minifi-sql SHARED
lib with conan will require making further updates to the SOCI conanfile.py's
package_info() method. After building MiNiFi with conan including minifi-sql, I
ran GTESTs and 99% of tests passed. I will look further into minifi-sql SHARED
library to bring that approach later. For now, at least we can build minifi-sql
as a STATIC library.
*UPDATE (May 9, 2024)* -
[ca3d6df|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/ca3d6df318162e2ee9d3e76e173d9f83e17d3010]:
{*}Enabled MiNiFi C++ extensions GPS and COAP{*}. I added conan recipe for
libcoap keeping the windows cmake patches for minifi. Since the original
libcoap conan recipe's conanfile.py on conan-center-index was tailored for
CMake building libcoap versions > 4.3.0, I modified the recipe, so we can build
libcoap 4.2.1, which didnt have CMake support yet and thus, I updated that
conanfile.py to run 'autogen.sh' to create the configure script, I passed the
configure args to the configure script and then I used conan's Autotools() to
run the configure script and then ran the generated make file to build libcoap
4.2.1. I updated BundledLibCoAP to account for the conan way, but kept backward
compatibility for standalone CMake way. For a later update when I test on
Windows, I need to add support into the libcoap 4.2.1 conanfile.py for adding
and applying the windows cmake patch to build libcoap. For now, I am mainly
testing this new conan build infra for MiNiFi on Ubuntu 22.04 docker container
and backward compatibility for standalone CMake on Ubuntu too. 99% of GTESTs
passed (215/216).
*(UPDATE (May 8, 2024) -*
*[93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:*
{color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi
C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added
conan recipe for liblzma (xy_utils) and libarchive keeping the patches for
minifi. One area I found an issue with was integrating openssl conan package
with libarchive conan package and libarchive complained about not being able to
find openssl::openssl target. I did try to solve the issue by updating the
libarchive conan recipe, but wasn't able to solve that openssl integration
issue. I think the problem comes from libarchive CMakeLists.txt and its
integration with conan in the repo. I opened an issue on conan-center-index to
help with resolving this issue in case I get around to solving it in time. Good
news is that I was still able to create libarchive conan package with the
configuration settings similar to how we do it the standalone CMake
BundledLibArchive.cmake way just in libarchive conan recipe. I also verified I
can build these minifi extensions (LibArchive, LZMA, Procfs, Systemd) and
NANOFI using conan build and standalone CMake. The standalone CMake way still
allows us to build LibArchive with OpenSSL enabled. 213/214 (99%) GTESTs passed
after building MiNiFi C++ with conan.{color}
*UPDATE (May 7, 2024) -
[*2460e84*|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2460e846832d693c991e1da632b86ba2d4e34516]:*
*Enabled OPS and OPC* MiNiFi Options, so those *MiNiFi C++ extensions* are
built using *Conan CMake* and their bundled CMake dependencies use their
particular *{{open62541/1.3.3@minifi/dev}}* and
{*}{{mbedtls/2.16.3@minifi/dev}} conan packages{*}. I had to fix the mbedtls
conan package from conan center index since it wasnt compatible when open62541
tried to integrate with mbedtls. In the thirdparty folder, you will see on line
139 where I updated libembedtls to be libmbedtls and fixing that typo allowed
open62541 conan package to be able to be created with mbedtls as the
encryption. I plan to make a follow up PR on conan center index for mbedtls
conan recipe. Currently, we are able to build MiNiFi's additional extensions
for OPS and OPC using conan. Verified backward compatibility with standalone
CMake works, so I can also just build MiNiFi using CMake only.
*UPDATE (May 5, 2024) -
[446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:*
When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and
*Standard-Processors* and the other relevant that needed to be activated for
these 3 essential components, I also set SKIP_TESTS to OFF, I set
ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs
using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for
conan approach while keeping backward compatibility with standalone CMake
approach. I got the following error coming from extension
robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I will
add the fix for this issue:
{code:java}
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26:
error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and
‘rocksdb::DBOptions’)
71 | if (db_opts_copy != db_options_) {
| ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~
| | |
| | rocksdb::DBOptions
| rocksdb::DBOptions {code}
We ran into this issue since when we created rocksdb/8.10.2 conan package, we
didn't apply the 2 patches like was done in the standalone CMake approach, so I
updated our conan approach where I created a rocksdb conan recipe in the
nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and applied
them to create the rocksdb conan package successfully. Also I do want to note
that I referenced the rocksdb conan recipe from conan center index repo. After
creating the rocksdb conan package, then I built MiNiFi C++ GTESTs for MiNiFi
Core, MainExe, Standard-Processors and the other relevant extensions that
needed to be activated for these 3 essential components using conan build. Here
we have the results after running ctest in the build folder:
{code:java}
192/194 Test #192: ConfigFileEncryptorTests .................... Passed
0.02 sec
Start 193: ConfigFileTests
193/194 Test #193: ConfigFileTests ............................. Passed
0.01 sec
Start 194: ControllerTests
194/194 Test #194: ControllerTests .............................***Failed
0.52 sec
99% tests passed, 2 tests failed out of 194
Total Test time (real) = 382.89 sec
The following tests FAILED: 45 - LoggerTests (Failed) 194 -
ControllerTests (Failed)Errors while running CTestOutput from these tests are
in:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse
"--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
{code}
*We can see that 99% of the tests passed. Thus, 192/194 tests passed. This was
using conan to build MiNiFi C++.*
I also verified I can build *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and
*Standard-Processors* and the other relevant that needed to be activated for
these 3 essential components, I also set SKIP_TESTS to OFF, I set
ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, BUILD_ROCKSDB to ON,
ENABLE_ROCKSDB to ON to {*}build GTESTs using standalone CMake{*}. Here we can
see similar results with running the MiNiFi GESTs:
{code:java}
192/194 Test #192: ConfigFileEncryptorTests .................... Passed
0.07 sec
Start 193: ConfigFileTests
193/194 Test #193: ConfigFileTests ............................. Passed
0.08 sec
Start 194: ControllerTests
194/194 Test #194: ControllerTests .............................***Failed
0.76 sec
99% tests passed, 1 tests failed out of 194
Total Test time (real) = 236.38 sec
The following tests FAILED: 194 - ControllerTests (Failed)Errors while
running CTestOutput from these tests are in:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse
"--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
{code}
We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was
using standalone CMake to build MiNiFi C++.
{*}{color:#0747a6}UPDATE (May 4, 2024) -
{color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
Verified I can build MiNiFi C++ with standalone CMake like how we currently do
it even after integrating conan into the build infrastructure. CMake built
*MiNiFi Core, Main* and *Standard Processors* successfully. Also double checked
that conan build MiNiFi still works. (Feel free to reference the logs for the
CMake standalone build and the conan build of MiNiFi CPP)
{color:#0747a6}{*}UPDATE (May 4, 2024) -
[2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color}
Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi*
and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*}
extension with minimal set of external libraries needed. If you look at our
conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a tuple
of the minimal external libraries needed to build these two essential building
blocks of MiNFi CPP. Next when you look at conanfile.py *MiNiFiCppMain* class's
generate(self) method, you'll see I have enabled USE_CONAN_PACKAGER to ON,
SKIP_TESTS to ON, and then the essential external library dependencies that
aren needed for the *core-minifi* library and *minifiexe* binary executable,
which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL, ENABLE_CIVET, ENABLE_CURL{*}. I
notice most of these external libraries are in the extensions folder and add
some dependencies on these custom extensions that are created.
{color:#ff8b00}*I think we should consider creating an extensions subfolder for
core-minifi related extensions.*{color}
Here is the output I get after building the MiNiFi CPP code, you can see MiNiFi
CPP libraries and binary executables built successfully using *CONAN version 2*
and CMake (feel free to check the attached log for the full conan build
output): [~szaszm]
{code:java}
Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list
Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT list
Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT
-- Creating API Documentation using /usr/bin/doxygen
-- Configuring done
-- Generating done
-- Build files have been written to:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py
(minifi-cpp-main/0.15.0): Running CMake.build()
conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build
"/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20
[ 3%] Built target ossp-uuid-external-build
[ 6%] Built target openwsman-external
[ 7%] Built target ossp-uuid-external
Consolidate compiler generated dependencies of target core-minifi
[ 8%] Building CXX object
libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o
[ 8%] Linking CXX shared library ../bin/libcore-minifi.so
[ 78%] Built target core-minifi
Consolidate compiler generated dependencies of target minifi-civet-extensions
Consolidate compiler generated dependencies of target minifi-openwsman
Consolidate compiler generated dependencies of target minifi-http-curl
Consolidate compiler generated dependencies of target minifiexe
[ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so
[ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so
Consolidate compiler generated dependencies of target minifi-standard-processors
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o
[ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o
[ 80%] Building CXX object
minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o
[ 81%] Building CXX object
minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o
[ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so
[ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so
[ 84%] Built target minifi-civet-extensions
[ 86%] Built target minifi-http-curl
[ 87%] Built target minifi-openwsman
[100%] Built target minifi-standard-processors
[100%] Linking CXX executable ../bin/minifi
[100%] Built target minifiexe {code}
NEXT: I will verify I can build these MiNiFi CPP libraries and binary
executables using USE_CMAKE_FETCH_CONTENT option set to ON and
USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
Tracking Prebuilt Binary Conan Packages Available on Conancenter:
* 'abseil/20230125.3',
* 'argparse/3.0',
* 'asio/1.30.2'
Tracking Conan Packages Not Available on Conancenter, so we need to use an
alternative hosting service to manage building package sources into prebuilt
binary conan packages. Here are some of the free conan hosting services for
open-source C/C++ projects:
* {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
packages maintained by conan team
* Bincrafters Bintray Repo: community-driven repo that hosts large number of
pre-built conan packages for open-source libraries for free
* Artifactory by JFrog: typically used as a commercial hosting service, but
also provides a free tier for open-source projects.
* {*}Self-Hosting{*}: While there are multiple options, I will focus on
*GitHub Packages* as its a free package hosting service, which can be used to
host our own conan package repos, allowing us to store and distribute our
pre-built conan packages without having to use 3rd party services like
Bincrafters or pay for commercial solution.
** {*}Advantage with GitHub Packages{*}: its free, integrated with our
existing GitHub workflow, allows us full control hosting our conan packages
without relying on 3rd party services, useful if we need to have specific
requirements or need flexibility in managing our package hosting.
Tracking Non Prebuilt Binary Conan Packages, which later I will create a
*GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account for
Linux, Windows, Mac, etc):
* bustache: [https://github.com/jamboree/bustache]
{*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
after I finish my work for MINFICPP-2346: While I am working on migrating
MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
PR we could do the following:
# Put the external libraries that MiNiFi CPP Core depends on into its own
*BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other names),
we can run this CMake code before adding *add_subdirectory(libminifi)*
## External libs that libminifi needs that are in MiNiFi CPP's root
CMakeLists.txt (refactor for better organization):
yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
magic_enum, RangeV3, Asio, MagicEnum
## We can also clean up the *thirdparty* folder as we migrate to conan
packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep it
MiNiFi C{+}+ code and its extension processors, etc.
# I tried building MiNiFi CPP Core just with just *minifi-standard-processors*
and the other extensions disabled, but it seems even some of MiNiFi CPP Core
depends on some of the extensions to be built with it, so it doesn't fail
building, such as Openwsman. If we don't have ENABLE_OPENWSMAN, then we get
this error when building MiNiFi CPP Core, so we should look having a build
config to build minimal MiNiFi CPP Core:
{code:java}
libminifi/src/controllers/SSLContextService.cpp:520:80: error:
‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
# I noticed that when building MiNiFi Core, MainExe, Standard-Processors and
their external lib dependencies and then building the appropriate GTEST, I get
a header error coming from rocksdb, but I haven't enabled rocksdb, so I think
at a later point we should shift that test out, so when building these 3 core
building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi Core and
MainExe have a dependency for storing into that DB. Here is an example of the
header error (I will ENABLE_ROCKSDB to ON):
{code:java}
[ 99%] Built target ControllerTests
In file included from
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30,
from
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10:
fatal error: rocksdb/db.h: No such file or directory
26 | #include "rocksdb/db.h" {code}
> Speed up MiNiFi Build with Conan package manager (C++)
> ------------------------------------------------------
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Affects Versions: 0.15.0, 0.99.0
> Environment: Ubuntu 22.04
> Reporter: James Guzman (Medel)
> Assignee: James Guzman (Medel)
> Priority: Minor
> Labels: build, conan, ubuntu
> Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log,
> minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
> point, I will pull in lastest updates from MiNiFi C++ main branch on parent
> level and rebase or merge into my PR.
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
> During the the build process, it clones all the external dependencies and
> then builds those external dependencies while its building MiNiFi CPP.
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail
> to download or fail to build. It would be faster to build MiNiFi C++ with the
> external dependencies already preinstalled.
>
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
> enable support for using *conan install* to install all or most of MiNiFi's
> external dependencies, using *conan install --build=missing* to build them as
> prebuilt binary conan packages, upload them to conancenter with {*}conan
> upload{*}, and then run cmake generate to generate the appropriate build
> files for the OS, then run make to build MiNiFi CPP. Also conan has really
> good support for cross platform compatibility for Linux and Windows. At this
> point because we already have most of MiNiFi CPP's external lib dependencies
> installed with conan, our CMake build will just focus on building the MiNiFi
> CPP code, enabling faster builds. Also I will need to account for backward
> compatibility support with the previous way of installing MiNiFi CPP external
> lib dependencies using CMake FetchContent_Declare(...) approach until
> conanfile.py approach covers installing all these dependencies. A valid
> verification of the MiNiFi C++ build would be to run GTESTs and if all the
> expected test cases pass, then it may be fair to conclude we would be ready
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
> script(s)):*
> # Install conan version 2 into MiNiFi C++ dev environment (my preference is
> using docker container)
> # Create a conanfile.py file at root of MiNiFi C++ project where we run
> "conan install" or "conan install --build=missing" to install MiNiFi C++
> external lib dependencies first. Then we run CMake to build MiNiFi C++.
> # Find all MiNiFi C++ find_package(...) CMake function calls and check if
> there is already supported conan packages on the conancenter, so we can
> install those packages.
> # Find all FetchContent_Declare(...) CMake function calls and check if there
> is already supported conan packages on the conancenter, so we can install
> those packages.
> # At this point we will have used conan version 2 C++ package manager to
> install almost all MiNiFi C++'s external dependencies.
> ## With Conan, we will have installed these MiNiFi external lib dependencies
> as prebuilt binary conan packages.
> ## Or we will have instructed conan to first build each C++ external lib
> dependency, upload each lib to conancenter.
> # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so
> we can then run CMake to build MiNiFi faster.
>
> Here is my draft PR for this Jira Ticket for your reference:
> [https://github.com/apache/nifi-minifi-cpp/pull/1775]
> Here are the commands that I run:
>
> {code:java}
> cd $HOME/src/pipeline/nifi-minifi-cpp
> # NOTE: I also plan to add smoother integration of conan commands into MiNiFI
> CPP too
> # For instance, I know you guys have your bootstrap scripts too
> # You also have your github actions workflow scripts too
> conan install . --output-folder=build
> -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
> # install conan packages and build C++ for minifi using Conan & CMake, we run
> this way, so conan's tc.cache_variables
> # are like passing -D{minifi_option} upon running cmake generate, then
> make to build MiNiFi C++
> conan build . --output-folder=build
> -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
>
>
> *UPDATE (May 21, 2024):* I created a small PR for conan integration on top of
> Gabor Gyimesi's PR-1749 "https://github.com/apache/nifi-minifi-cpp/pull/1749";
> that brings in conan packages for OpenSSL, LibCurl and Zlib as alternatives
> to their system library counterparts when we run standalone CMake to build
> MiNiFi CPP. I also included steps for Maton Szasz and others to get up to
> speed with how to use conan v2 to build MiNiFi CPP. On top of Gabor's
> PR-1749, my *PR-261* has +325 -225 git diff and only 5 files changed:
> https://github.com/lordgamez/nifi-minifi-cpp/pull/261
>
> *UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the
> refactoring of MiNiFi CPP build infra over to conan version 2 is taking
> longer than expected. The good news is that I believe I have more than half
> of MiNiFi external libraries now managed by conan with backward compatibility
> for standalone CMake approach.
>
> *UPDATE (May 13, 2024)* -
> *[22205c9|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/22205c98cef30ee1ebe252880ae113724589bd78]:*
> Fetch and Pulled latest upstream main branch of parent MiNiFi to my fork
> main branch of MiNiFi repo; thus, merged in the latest updates from main
> branch of parent MiNiFi to my MINIFICPP-2346 branch. Resolved conflicts
> between a few CMake files coming from Date.cmake and BundledRocksdb.cmake
> files. For standalone CMake build appraoch, ran GTESTs with 100% (238/238)
> tests passing. I did the conan build approach and ran GTESTs with 99%
> (237/238) passing.
> *UPDATE (May 13, 2024)* -
> [59ffefb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/59ffefb1caf662874054f55f7171692c6b9d345f]:
> {*}Enabled LIBRDKAFKA & GRPC_FOR_LOKI MiNiFi C++ extension using Conan &
> CMake{*}, *Improved Running Python Maven Executable for JNI & SFTP Test
> Server for conan approach* and {*}Improved standalone CMake approach for
> building PcapPlusPlus with its dependency on LibPcap{*}. LIBRDKAFKA required
> creating a librdkafka conan package passing similar CMake definitions to it
> to build it like we did for standalone CMake approach. GRPC_FOR_LOKI required
> creating a grpc conan package that uses minifi's openssl and libsystemd conan
> packages. In both LIBRDKAFKA & GRPC_FOR_LOKI, I updated their .cmake Bundles
> to account for the condition of using their conan package, so minimal changes
> would be needed to their extension CMakeLists.txt file. In LIBRDKAFKA's
> extension CMakeLists.txt file, it didnt really change after integrating the
> conan package since I created a library alias for it similar to the
> standalone CMake approach. In GRPC_FOR_LOKI's extension CMakeLists.txt,
> minimal changes were made to that file mainly on the add_custom_command,
> target_include_directories and target_link_libraries commands. You'll see for
> Grpc.cmake for conan approach, I created similar variables to standalone
> CMake approach for GRPC_CPP_PLUGIN, INCLUDE_DIR, PROTOBUF_INCLUDE_DIR,
> PROTOBUF_COMPILER getting the conan related properties. For the conan side, I
> also made improvements to the Python Maven Executable Wrapper script since I
> noticed there were times at the beginning when running maven version test, it
> didn't always run it successfully, so I updated conan maven executable path
> to be explicit for passing to Python script. I then moved the location of
> running maven version check into a CMake function and ran it after we
> included JavaMaven and JavaOpenJDK conan packages in the MiNiFi root
> CMakeLists.txt file. Thus, it succesfully runs maven version check at
> beginning each time. Its important to make sure we have conan maven to
> successfully build NiFi JNI and Java SFTP Test Server. Finally, on the
> standalone CMake approach for building PcapPlusPlus, I noticed some issues
> with PCAP headers not being found when building PcapPlusPlus, so I switched
> from execute_process to run the configure scripts to using
> ExternalProject_Add and I also added a bundle for LibPcap since the system
> LibPcap version wasn't always installed on the system. This problem with
> building PcapPlusPlus dependency on LibPcap didn't happen with my conan
> approach. It took a lot of time to fix the standalone CMake backward
> compatibility approach, but I did it for now since we're keeping support that
> approach. 99% of GTESTs ( 237/238) passed for conan.
>
> *UPDATE (May 11, 2024)* -
> [c96d430|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/c96d430814a067a6a538f4324337b719cb75bf48]:
> Refactoring MiNiFi C++ build infrastructure adding Conan Packager support
> *Enabled JNI, SENSORS, USB_CAMERA, OPENCV, and SFTP,* which required creating
> *Maven 3.9.6 conan package* updating the conan recipe to expose the Maven
> Executable environment variable in CMake and *installing prebuilt Java
> OpenJDK 21.0.2 conan package* and creating a python wrapper run_maven.py
> script for calling the conan maven Java build tool to successfully build Java
> SFTP Test Server for SFTP extension testing and maven to build {*}Apache NiFi
> 1.9.0 JNI Assembly and Framework{*}; We had to go the python maven wrapper
> script approach since I ran into some issues calling conan maven executable
> from CMake directly; Also added prebuilt install for libssh2 conan package
> toward SFTP extensions; Thus, now the Java support for MiNiFi C++ is less
> reliant on the user’s system since we can handle building this part of MiNiFi
> C++ using conan; Also created an OpenCV 4.8.1 conan package updating the
> conan recipe to use MiNiFi C++ conan package dependencies from libtiff,
> ffmpeg and xz_utils (liblzma) and successfully build MiNiFi OpenCV extension;
> Also installed prebuilt libuvc conan package for enabling USB_CAMERA. For the
> conan build approach, 99% (235/236) of GTESTs passed. For the standalone
> CMake build approach, 99% (234/236) of GTESTs passed.
> * Also we could look at updating Apache NiFi 1.9.0 to latest 2.0 version. I
> noticed we use a Java SFTP Test Server now that NiFi has native python custom
> extensibility, maybe we could look at Python SFTP Test Server?
>
> *UPDATE (May 11, 2024)* -
> [64dd8ad|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/64dd8ad14d2ddcea5eaa12da14a0e1b8a7289bb8]:
> {*}Enabled Build LUA SCRIPTING, ENCRYPT CONFIG, SPLUNK, ELASTIC SEARCH, TEST
> PROCESSORS, GRANFANA LOKI and CONTROLLER by Conan & CMake{*}. For building
> these components with conan, the LUA SCRIPTING extension required me updating
> its Lua.cmake and Sol2.cmake files, so we can use the conan packages in CMake
> when USE_CONAN_PACKAGE condition is true. For lua and sol2 conan packages, I
> was able to just install the prebuilt binary conan packages from conancenter
> directly since conancenter provided the versions that match with MiNiFi C++'s
> standalone CMake approach's lua 5.4.6 and sol2 3.3.0 source built versions.
> MiNiFi controller being enabled required updating the ArgParse.cmake file, so
> we could also account for the condition when we build the controller using
> conan, thus I added back using the argparse conan package. After enabling
> these components, I was able to successfully build them for MiNiFi using
> conan. I also verified backward compatibility for building these components
> in standalone CMake worked too. We achieved similar results for GTESTs where
> out of 230 tests, 99% of them passed for conan build and 100% passed for
> standalone CMake build.
> *UPDATE (May 11, 2024)* -
> [f5cfb79|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/f5cfb792bed5d94bd3c55217ed8f32e777f741d1]:
> {*}Enabled Build MQTT & PCAP MiNiFi PCAP Extension by Conan & CMake{*}.
> Refactored the pcap extension CMake code moving the part that downloads and
> builds pcapplusplus over to its own PcapPlusPlus.cmake file. Also added conan
> build condition to that .cmake file, so we can build PcapPlusPlus using conan
> while still keeping backward compatibility using standalone CMake approach.
> While I was able to build MiNiFi C++ with PCAP Extension enabled using
> standalone CMake approach, when I built it using conan build approach, I
> found pcap extension complaining about libminifi core headers and realized
> include_directories(include) didnt share all the header directory paths from
> libminifi to pcap extension. Thus, I created a python script under
> libminifi/cmake/python to parse for the libminifi header directories
> generating a include_dirs.cmake file with the LIBMINIFI_INCLUDE_DIRS CMake
> variable list that gets stored in build folder and followed by
> libminifi/include_dirs.cmake. That way we keep our source libminifi directory
> and its subdirectories clean. Overall, I was able to build MiNiFi C++ with
> MQTT and PCAP extensions enabled using conan & cmake. conan build achieved
> 99% (222/223) of GTESTs passing. standalone cmake build achieved 100% of
> GTESTs passing.
>
> *UPDATE (May 9, 2024)* -
> {*}[63290db|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/63290db57b60054ba8ed32c0b3a7d1f87f315314]{*}:
> *Refactored IODBC and SOCI libraries, so they are now CMake Bundles for
> easier integration with Conan.* Thus, minifi-sql library now calls
> include(BundledIODBC), use_bundled_iodbc(...) and include(BundledSOCI) and
> use_bundled_soci(...). Verified I can still build MiNiFi C++ using standalone
> CMake. 100% (220/220) of GTESTS passed using standalone CMake. Integrated
> conan for building SOCI & ODBC for MiNiFi SQL extension.
> *[4232399|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/4232399657c9da42dff57ff2b6806aeea2485444]*
> *Added a SOCI conan recipe to create SOCI conan package with unixODBC conan
> package, so SOCI comes with targets SOCI::libsoci_core and
> SOCI::lib;soci_odbc.* Verified I can build MiNiFi C++'s minifi-sql extension
> using conan build. However, with the conan build approach for minifi-sql
> extension, I was only able to build minifi-sql as a STATIC library
> succesfully. When I tried building minifi-sql as a SHARED library, I ran into
> multiple undefined references to SOCI functions. I think to fix the
> minifi-sql SHARED lib with conan will require making further updates to the
> SOCI conanfile.py's package_info() method. After building MiNiFi with conan
> including minifi-sql, I ran GTESTs and 99% of tests passed. I will look
> further into minifi-sql SHARED library to bring that approach later. For now,
> at least we can build minifi-sql as a STATIC library.
>
> *UPDATE (May 9, 2024)* -
> [ca3d6df|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/ca3d6df318162e2ee9d3e76e173d9f83e17d3010]:
> {*}Enabled MiNiFi C++ extensions GPS and COAP{*}. I added conan recipe for
> libcoap keeping the windows cmake patches for minifi. Since the original
> libcoap conan recipe's conanfile.py on conan-center-index was tailored for
> CMake building libcoap versions > 4.3.0, I modified the recipe, so we can
> build libcoap 4.2.1, which didnt have CMake support yet and thus, I updated
> that conanfile.py to run 'autogen.sh' to create the configure script, I
> passed the configure args to the configure script and then I used conan's
> Autotools() to run the configure script and then ran the generated make file
> to build libcoap 4.2.1. I updated BundledLibCoAP to account for the conan
> way, but kept backward compatibility for standalone CMake way. For a later
> update when I test on Windows, I need to add support into the libcoap 4.2.1
> conanfile.py for adding and applying the windows cmake patch to build
> libcoap. For now, I am mainly testing this new conan build infra for MiNiFi
> on Ubuntu 22.04 docker container and backward compatibility for standalone
> CMake on Ubuntu too. 99% of GTESTs passed (215/216).
>
> *(UPDATE (May 8, 2024) -*
> *[93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:*
> {color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi
> C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added
> conan recipe for liblzma (xy_utils) and libarchive keeping the patches for
> minifi. One area I found an issue with was integrating openssl conan package
> with libarchive conan package and libarchive complained about not being able
> to find openssl::openssl target. I did try to solve the issue by updating the
> libarchive conan recipe, but wasn't able to solve that openssl integration
> issue. I think the problem comes from libarchive CMakeLists.txt and its
> integration with conan in the repo. I opened an issue on conan-center-index
> to help with resolving this issue in case I get around to solving it in time.
> Good news is that I was still able to create libarchive conan package with
> the configuration settings similar to how we do it the standalone CMake
> BundledLibArchive.cmake way just in libarchive conan recipe. I also verified
> I can build these minifi extensions (LibArchive, LZMA, Procfs, Systemd) and
> NANOFI using conan build and standalone CMake. The standalone CMake way still
> allows us to build LibArchive with OpenSSL enabled. 213/214 (99%) GTESTs
> passed after building MiNiFi C++ with conan.{color}
>
> *UPDATE (May 7, 2024) -
> [*2460e84*|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2460e846832d693c991e1da632b86ba2d4e34516]:*
> *Enabled OPS and OPC* MiNiFi Options, so those *MiNiFi C++ extensions* are
> built using *Conan CMake* and their bundled CMake dependencies use their
> particular *{{open62541/1.3.3@minifi/dev}}* and
> {*}{{mbedtls/2.16.3@minifi/dev}} conan packages{*}. I had to fix the mbedtls
> conan package from conan center index since it wasnt compatible when
> open62541 tried to integrate with mbedtls. In the thirdparty folder, you will
> see on line 139 where I updated libembedtls to be libmbedtls and fixing that
> typo allowed open62541 conan package to be able to be created with mbedtls as
> the encryption. I plan to make a follow up PR on conan center index for
> mbedtls conan recipe. Currently, we are able to build MiNiFi's additional
> extensions for OPS and OPC using conan. Verified backward compatibility with
> standalone CMake works, so I can also just build MiNiFi using CMake only.
>
> *UPDATE (May 5, 2024) -
> [446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:*
> When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and
> *Standard-Processors* and the other relevant that needed to be activated for
> these 3 essential components, I also set SKIP_TESTS to OFF, I set
> ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs
> using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for
> conan approach while keeping backward compatibility with standalone CMake
> approach. I got the following error coming from extension
> robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I
> will add the fix for this issue:
> {code:java}
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26:
> error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and
> ‘rocksdb::DBOptions’)
> 71 | if (db_opts_copy != db_options_) {
> | ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~
> | | |
> | | rocksdb::DBOptions
> | rocksdb::DBOptions {code}
> We ran into this issue since when we created rocksdb/8.10.2 conan package, we
> didn't apply the 2 patches like was done in the standalone CMake approach, so
> I updated our conan approach where I created a rocksdb conan recipe in the
> nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and
> applied them to create the rocksdb conan package successfully. Also I do want
> to note that I referenced the rocksdb conan recipe from conan center index
> repo. After creating the rocksdb conan package, then I built MiNiFi C++
> GTESTs for MiNiFi Core, MainExe, Standard-Processors and the other relevant
> extensions that needed to be activated for these 3 essential components using
> conan build. Here we have the results after running ctest in the build folder:
> {code:java}
> 192/194 Test #192: ConfigFileEncryptorTests .................... Passed
> 0.02 sec
> Start 193: ConfigFileTests
> 193/194 Test #193: ConfigFileTests ............................. Passed
> 0.01 sec
> Start 194: ControllerTests
> 194/194 Test #194: ControllerTests .............................***Failed
> 0.52 sec
> 99% tests passed, 2 tests failed out of 194
> Total Test time (real) = 382.89 sec
> The following tests FAILED: 45 - LoggerTests (Failed) 194 -
> ControllerTests (Failed)Errors while running CTestOutput from these tests are
> in:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse
> "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
> {code}
> *We can see that 99% of the tests passed. Thus, 192/194 tests passed. This
> was using conan to build MiNiFi C++.*
>
> I also verified I can build *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and
> *Standard-Processors* and the other relevant that needed to be activated for
> these 3 essential components, I also set SKIP_TESTS to OFF, I set
> ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, BUILD_ROCKSDB to ON,
> ENABLE_ROCKSDB to ON to {*}build GTESTs using standalone CMake{*}. Here we
> can see similar results with running the MiNiFi GESTs:
>
> {code:java}
> 192/194 Test #192: ConfigFileEncryptorTests .................... Passed
> 0.07 sec
> Start 193: ConfigFileTests
> 193/194 Test #193: ConfigFileTests ............................. Passed
> 0.08 sec
> Start 194: ControllerTests
> 194/194 Test #194: ControllerTests .............................***Failed
> 0.76 sec
> 99% tests passed, 1 tests failed out of 194
> Total Test time (real) = 236.38 sec
> The following tests FAILED: 194 - ControllerTests (Failed)Errors while
> running CTestOutput from these tests are in:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse
> "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
> {code}
> We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was
> using standalone CMake to build MiNiFi C++.
>
> {*}{color:#0747a6}UPDATE (May 4, 2024) -
> {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
> Verified I can build MiNiFi C++ with standalone CMake like how we currently
> do it even after integrating conan into the build infrastructure. CMake built
> *MiNiFi Core, Main* and *Standard Processors* successfully. Also double
> checked that conan build MiNiFi still works. (Feel free to reference the logs
> for the CMake standalone build and the conan build of MiNiFi CPP)
>
> {color:#0747a6}{*}UPDATE (May 4, 2024) -
> [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color}
> Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi*
> and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*}
> extension with minimal set of external libraries needed. If you look at our
> conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a
> tuple of the minimal external libraries needed to build these two essential
> building blocks of MiNFi CPP. Next when you look at conanfile.py
> *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled
> USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external
> library dependencies that aren needed for the *core-minifi* library and
> *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL,
> ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are
> in the extensions folder and add some dependencies on these custom extensions
> that are created. {color:#ff8b00}*I think we should consider creating an
> extensions subfolder for core-minifi related extensions.*{color}
> Here is the output I get after building the MiNiFi CPP code, you can see
> MiNiFi CPP libraries and binary executables built successfully using *CONAN
> version 2* and CMake (feel free to check the attached log for the full conan
> build output): [~szaszm]
>
> {code:java}
> Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list
> Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT
> list
> Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT
> -- Creating API Documentation using /usr/bin/doxygen
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py
> (minifi-cpp-main/0.15.0): Running CMake.build()
> conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build
> "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20
> [ 3%] Built target ossp-uuid-external-build
> [ 6%] Built target openwsman-external
> [ 7%] Built target ossp-uuid-external
> Consolidate compiler generated dependencies of target core-minifi
> [ 8%] Building CXX object
> libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o
> [ 8%] Linking CXX shared library ../bin/libcore-minifi.so
> [ 78%] Built target core-minifi
> Consolidate compiler generated dependencies of target minifi-civet-extensions
> Consolidate compiler generated dependencies of target minifi-openwsman
> Consolidate compiler generated dependencies of target minifi-http-curl
> Consolidate compiler generated dependencies of target minifiexe
> [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so
> [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so
> Consolidate compiler generated dependencies of target
> minifi-standard-processors
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o
> [ 80%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o
> [ 81%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o
> [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so
> [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so
> [ 84%] Built target minifi-civet-extensions
> [ 86%] Built target minifi-http-curl
> [ 87%] Built target minifi-openwsman
> [100%] Built target minifi-standard-processors
> [100%] Linking CXX executable ../bin/minifi
> [100%] Built target minifiexe {code}
> NEXT: I will verify I can build these MiNiFi CPP libraries and binary
> executables using USE_CMAKE_FETCH_CONTENT option set to ON and
> USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
>
> Tracking Prebuilt Binary Conan Packages Available on Conancenter:
> * 'abseil/20230125.3',
> * 'argparse/3.0',
> * 'asio/1.30.2'
> Tracking Conan Packages Not Available on Conancenter, so we need to use an
> alternative hosting service to manage building package sources into prebuilt
> binary conan packages. Here are some of the free conan hosting services for
> open-source C/C++ projects:
> * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
> packages maintained by conan team
> * Bincrafters Bintray Repo: community-driven repo that hosts large number of
> pre-built conan packages for open-source libraries for free
> * Artifactory by JFrog: typically used as a commercial hosting service, but
> also provides a free tier for open-source projects.
> * {*}Self-Hosting{*}: While there are multiple options, I will focus on
> *GitHub Packages* as its a free package hosting service, which can be used to
> host our own conan package repos, allowing us to store and distribute our
> pre-built conan packages without having to use 3rd party services like
> Bincrafters or pay for commercial solution.
> ** {*}Advantage with GitHub Packages{*}: its free, integrated with our
> existing GitHub workflow, allows us full control hosting our conan packages
> without relying on 3rd party services, useful if we need to have specific
> requirements or need flexibility in managing our package hosting.
> Tracking Non Prebuilt Binary Conan Packages, which later I will create a
> *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account
> for Linux, Windows, Mac, etc):
> * bustache: [https://github.com/jamboree/bustache]
>
> {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
> after I finish my work for MINFICPP-2346: While I am working on migrating
> MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
> PR we could do the following:
> # Put the external libraries that MiNiFi CPP Core depends on into its own
> *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other
> names), we can run this CMake code before adding *add_subdirectory(libminifi)*
> ## External libs that libminifi needs that are in MiNiFi CPP's root
> CMakeLists.txt (refactor for better organization):
> yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
> gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
> magic_enum, RangeV3, Asio, MagicEnum
>
> ## We can also clean up the *thirdparty* folder as we migrate to conan
> packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep
> it MiNiFi C{+}+ code and its extension processors, etc.
> # I tried building MiNiFi CPP Core just with just
> *minifi-standard-processors* and the other extensions disabled, but it seems
> even some of MiNiFi CPP Core depends on some of the extensions to be built
> with it, so it doesn't fail building, such as Openwsman. If we don't have
> ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we
> should look having a build config to build minimal MiNiFi CPP Core:
> {code:java}
> libminifi/src/controllers/SSLContextService.cpp:520:80: error:
> ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
>
> # I noticed that when building MiNiFi Core, MainExe, Standard-Processors and
> their external lib dependencies and then building the appropriate GTEST, I
> get a header error coming from rocksdb, but I haven't enabled rocksdb, so I
> think at a later point we should shift that test out, so when building these
> 3 core building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi
> Core and MainExe have a dependency for storing into that DB. Here is an
> example of the header error (I will ENABLE_ROCKSDB to ON):
> {code:java}
> [ 99%] Built target ControllerTests
> In file included from
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30,
> from
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10:
> fatal error: rocksdb/db.h: No such file or directory
> 26 | #include "rocksdb/db.h" {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
