[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
*{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building
*MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the
other relevant that needed to be activated for these 3 essential components, I
also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and
ENABLE_ROCKSDB to ON to build GTESTs using conan build. I updated Bundle
Rocksdb and Find Rocksdb to account for conan approach while keeping backward
compatibility
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
*{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building
*MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and *Standard-Processors* and the
other relevant that needed to be activated for these 3 essential components, I
also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and
ENABLE_ROCKSDB to ON to build GTESTs using conan build. I updated Bundle
Rocksdb and Find Rocksdb to account for conan approach while keeping backward
compatibility
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
*{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building
MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors, I set
SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB
to ON. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach
while keeping backward compatibility with standalone CMake approach. I got the
following error coming from extension
robksdb-repos/database/RocksDbInstance.cpp on
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
*{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building
MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors, I set
SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB
to ON. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach
while keeping backward compatibility with standalone CMake approach. I got the
following error coming from extension
robksdb-repos/database/RocksDbInstance.cpp on
Re: [PR] NIFI-13139 ensure nifi-py4j-nar is included in test assembly [nifi]
joewitt commented on PR #8747: URL: https://github.com/apache/nifi/pull/8747#issuecomment-2094954148 Also fixed nifi-py4j-integration-tests which needed logback libs after the logback dependency in nifi-py4j-bridge was added. Also set one of the tests to disabled so it can be run manually. It expects a virtual env it seems. Verified the python processor in that test runs fine in a real nifi. Can we enable the nifi-py4j-integration-tests in the ci builds? anyway this commit as-is is ready to go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
*{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building
MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors, I set
SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB
to ON. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach
while keeping backward compatibility with standalone CMake approach. I got the
following error coming from extension
robksdb-repos/database/RocksDbInstance.cpp on
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
*{color:#0747a6}UPDATE (May 5, 2024) - :{color}*{color:#172b4d} When building
MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors, I set
SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB
to ON. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach
while keeping backward compatibility with standalone CMake approach. I got the
following error coming from extension
robksdb-repos/database/RocksDbInstance.cpp on
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
{*}{color:#0747a6}UPDATE (May 4, 2024) -
{color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
Verified I can build MiNiFi C++ with standalone CMake like how we currently do
it even after integrating conan into the build infrastructure. CMake built
*MiNiFi Core, Main* and *Standard Processors* successfully. Also double checked
that conan build MiNiFi still works. (Feel free to reference the logs for
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
{*}{color:#0747a6}UPDATE (May 4, 2024) -
{color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
Verified I can build MiNiFi C++ with standalone CMake like how we currently do
it even after integrating conan into the build infrastructure. CMake built
*MiNiFi Core, Main* and *Standard Processors* successfully. Also double checked
that conan build MiNiFi still works. (Feel free to reference the logs for
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
{*}{color:#0747a6}UPDATE (May 4, 2024) -
{color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
Verified I can build MiNiFi C++ with standalone CMake like how we currently do
it even after integrating conan into the build infrastructure. CMake built
MiNiFi Core, Main and Standard Processors successfully. Also double checked
that conan build MiNiFi still works. (Feel free to reference the logs for the
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Attachment: minifi_cpp_cmake_standalone_build_7eb8beb.log
> Speed up MiNiFi Build with Conan package manager (C++)
> --
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Affects Versions: 0.15.0
> Environment: Ubuntu 22.04
>Reporter: James Guzman (Medel)
>Assignee: James Guzman (Medel)
>Priority: Minor
> Labels: build, conan, ubuntu
> Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log,
> minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
> point, I will pull in lastest updates from MiNiFi C++ main branch on parent
> level and rebase or merge into my PR.
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
> During the the build process, it clones all the external dependencies and
> then builds those external dependencies while its building MiNiFi CPP.
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail
> to download or fail to build. It would be faster to build MiNiFi C++ with the
> external dependencies already preinstalled.
>
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
> enable support for using *conan install* to install all or most of MiNiFi's
> external dependencies, using *conan install --build=missing* to build them as
> prebuilt binary conan packages, upload them to conancenter with {*}conan
> upload{*}, and then run cmake generate to generate the appropriate build
> files for the OS, then run make to build MiNiFi CPP. Also conan has really
> good support for cross platform compatibility for Linux and Windows. At this
> point because we already have most of MiNiFi CPP's external lib dependencies
> installed with conan, our CMake build will just focus on building the MiNiFi
> CPP code, enabling faster builds. Also I will need to account for backward
> compatibility support with the previous way of installing MiNiFi CPP external
> lib dependencies using CMake FetchContent_Declare(...) approach until
> conanfile.py approach covers installing all these dependencies. A valid
> verification of the MiNiFi C++ build would be to run GTESTs and if all the
> expected test cases pass, then it may be fair to conclude we would be ready
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
> script(s)):*
> # Install conan version 2 into MiNiFi C++ dev environment (my preference is
> using docker container)
> # Create a conanfile.py file at root of MiNiFi C++ project where we run
> "conan install" or "conan install --build=missing" to install MiNiFi C++
> external lib dependencies first. Then we run CMake to build MiNiFi C++.
> # Find all MiNiFi C++ find_package(...) CMake function calls and check if
> there is already supported conan packages on the conancenter, so we can
> install those packages.
> # Find all FetchContent_Declare(...) CMake function calls and check if there
> is already supported conan packages on the conancenter, so we can install
> those packages.
> # At this point we will have used conan version 2 C++ package manager to
> install almost all MiNiFi C++'s external dependencies.
> ## With Conan, we will have installed these MiNiFi external lib dependencies
> as prebuilt binary conan packages.
> ## Or we will have instructed conan to first build each C++ external lib
> dependency, upload each lib to conancenter.
> # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so
> we can then run CMake to build MiNiFi faster.
>
> Here is my draft PR for this Jira Ticket for your reference:
> [https://github.com/apache/nifi-minifi-cpp/pull/1775]
> * I have added *abseil* external lib dependency to conanfile.py and
> integrated it into Abseil.cmake with conditional statement that checks if
> USE_CONAN_PACKAGER minifi cpp option is true else checks if
> USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
> began the process of integrating in conan version 2.
>
> Here are the commands that I run:
>
> {code:java}
> cd $HOME/src/pipeline/nifi-minifi-cpp
> # NOTE: I also plan to add smoother integration of conan commands into MiNiFI
> CPP too
> # For instance, I know you guys have your bootstrap scripts too
> # You also have your github actions workflow scripts too
> conan install .
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Description:
*NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
point, I will pull in lastest updates from MiNiFi C++ main branch on parent
level and rebase or merge into my PR.
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
During the the build process, it clones all the external dependencies and then
builds those external dependencies while its building MiNiFi CPP. Sometimes the
MiNiFi CPP build fails too when external lib dependencies fail to download or
fail to build. It would be faster to build MiNiFi C++ with the external
dependencies already preinstalled.
*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
enable support for using *conan install* to install all or most of MiNiFi's
external dependencies, using *conan install --build=missing* to build them as
prebuilt binary conan packages, upload them to conancenter with {*}conan
upload{*}, and then run cmake generate to generate the appropriate build files
for the OS, then run make to build MiNiFi CPP. Also conan has really good
support for cross platform compatibility for Linux and Windows. At this point
because we already have most of MiNiFi CPP's external lib dependencies
installed with conan, our CMake build will just focus on building the MiNiFi
CPP code, enabling faster builds. Also I will need to account for backward
compatibility support with the previous way of installing MiNiFi CPP external
lib dependencies using CMake FetchContent_Declare(...) approach until
conanfile.py approach covers installing all these dependencies. A valid
verification of the MiNiFi C++ build would be to run GTESTs and if all the
expected test cases pass, then it may be fair to conclude we would be ready to
deprecate the slow CMake FetchContent_Declare(...) approach.
*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
script(s)):*
# Install conan version 2 into MiNiFi C++ dev environment (my preference is
using docker container)
# Create a conanfile.py file at root of MiNiFi C++ project where we run "conan
install" or "conan install --build=missing" to install MiNiFi C++ external lib
dependencies first. Then we run CMake to build MiNiFi C++.
# Find all MiNiFi C++ find_package(...) CMake function calls and check if
there is already supported conan packages on the conancenter, so we can install
those packages.
# Find all FetchContent_Declare(...) CMake function calls and check if there
is already supported conan packages on the conancenter, so we can install those
packages.
# At this point we will have used conan version 2 C++ package manager to
install almost all MiNiFi C++'s external dependencies.
## With Conan, we will have installed these MiNiFi external lib dependencies
as prebuilt binary conan packages.
## Or we will have instructed conan to first build each C++ external lib
dependency, upload each lib to conancenter.
# Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we
can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference:
[https://github.com/apache/nifi-minifi-cpp/pull/1775]
* I have added *abseil* external lib dependency to conanfile.py and integrated
it into Abseil.cmake with conditional statement that checks if
USE_CONAN_PACKAGER minifi cpp option is true else checks if
USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
began the process of integrating in conan version 2.
Here are the commands that I run:
{code:java}
cd $HOME/src/pipeline/nifi-minifi-cpp
# NOTE: I also plan to add smoother integration of conan commands into MiNiFI
CPP too
# For instance, I know you guys have your bootstrap scripts too
# You also have your github actions workflow scripts too
conan install . --output-folder=build
-pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
# install conan packages and build C++ for minifi using Conan & CMake, we run
this way, so conan's tc.cache_variables
# are like passing -D{minifi_option} upon running cmake generate, then make
to build MiNiFi C++
conan build . --output-folder=build
-pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
{*}{color:#0747a6}UPDATE (May 4, 2024) -
{color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
Verified I can build MiNiFi C++ with standalone CMake like how we currently do
it even after integrating conan into the build infrastructure. CMake built
MiNiFi Core, Main and Standard Processors successfully. Also double checked
that conan build MiNiFi still works.
{color:#0747a6}{*}UPDATE (May 4, 2024)
[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)
[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
Attachment: minifi_cpp_conan_build_7eb8beb.log
> Speed up MiNiFi Build with Conan package manager (C++)
> --
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Affects Versions: 0.15.0
> Environment: Ubuntu 22.04
>Reporter: James Guzman (Medel)
>Assignee: James Guzman (Medel)
>Priority: Minor
> Labels: build, conan, ubuntu
> Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log,
> minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
> point, I will pull in lastest updates from MiNiFi C++ main branch on parent
> level and rebase or merge into my PR.
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
> During the the build process, it clones all the external dependencies and
> then builds those external dependencies while its building MiNiFi CPP.
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail
> to download or fail to build. It would be faster to build MiNiFi C++ with the
> external dependencies already preinstalled.
>
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
> enable support for using *conan install* to install all or most of MiNiFi's
> external dependencies, using *conan install --build=missing* to build them as
> prebuilt binary conan packages, upload them to conancenter with {*}conan
> upload{*}, and then run cmake generate to generate the appropriate build
> files for the OS, then run make to build MiNiFi CPP. Also conan has really
> good support for cross platform compatibility for Linux and Windows. At this
> point because we already have most of MiNiFi CPP's external lib dependencies
> installed with conan, our CMake build will just focus on building the MiNiFi
> CPP code, enabling faster builds. Also I will need to account for backward
> compatibility support with the previous way of installing MiNiFi CPP external
> lib dependencies using CMake FetchContent_Declare(...) approach until
> conanfile.py approach covers installing all these dependencies. A valid
> verification of the MiNiFi C++ build would be to run GTESTs and if all the
> expected test cases pass, then it may be fair to conclude we would be ready
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
> script(s)):*
> # Install conan version 2 into MiNiFi C++ dev environment (my preference is
> using docker container)
> # Create a conanfile.py file at root of MiNiFi C++ project where we run
> "conan install" or "conan install --build=missing" to install MiNiFi C++
> external lib dependencies first. Then we run CMake to build MiNiFi C++.
> # Find all MiNiFi C++ find_package(...) CMake function calls and check if
> there is already supported conan packages on the conancenter, so we can
> install those packages.
> # Find all FetchContent_Declare(...) CMake function calls and check if there
> is already supported conan packages on the conancenter, so we can install
> those packages.
> # At this point we will have used conan version 2 C++ package manager to
> install almost all MiNiFi C++'s external dependencies.
> ## With Conan, we will have installed these MiNiFi external lib dependencies
> as prebuilt binary conan packages.
> ## Or we will have instructed conan to first build each C++ external lib
> dependency, upload each lib to conancenter.
> # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so
> we can then run CMake to build MiNiFi faster.
>
> Here is my draft PR for this Jira Ticket for your reference:
> [https://github.com/apache/nifi-minifi-cpp/pull/1775]
> * I have added *abseil* external lib dependency to conanfile.py and
> integrated it into Abseil.cmake with conditional statement that checks if
> USE_CONAN_PACKAGER minifi cpp option is true else checks if
> USE_CMAKE_FETCH_CONTENT option is true for backward compatibility, so we can
> began the process of integrating in conan version 2.
>
> Here are the commands that I run:
>
> {code:java}
> cd $HOME/src/pipeline/nifi-minifi-cpp
> # NOTE: I also plan to add smoother integration of conan commands into MiNiFI
> CPP too
> # For instance, I know you guys have your bootstrap scripts too
> # You also have your github actions workflow scripts too
> conan install .
