(nifi-site) branch main updated: exclude PMC from Committers list (#83)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/main by this push: new 9d6b27e1 exclude PMC from Committers list (#83) 9d6b27e1 is described below commit 9d6b27e121aed433c57c674a2ce1d1b0c89ab8fb Author: Márton Szász AuthorDate: Thu Feb 22 19:30:08 2024 +0100 exclude PMC from Committers list (#83) --- content/community/_index.md | 2 +- themes/nifi/layouts/shortcodes/project-members.html | 8 +++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/content/community/_index.md b/content/community/_index.md index 1a061298..0b6d68dc 100644 --- a/content/community/_index.md +++ b/content/community/_index.md @@ -31,7 +31,7 @@ members on a vote open for at least 72 hours. A Committer is considered inactive can be revoked by a unanimous vote of all the active PMC members, except the member in question if they are a PMC member. -{{< project-members project="nifi" >}} +{{< project-members project="nifi" exclude-project="nifi-pmc" >}} ## Contributors diff --git a/themes/nifi/layouts/shortcodes/project-members.html b/themes/nifi/layouts/shortcodes/project-members.html index bb5c7cc3..33b4ad59 100644 --- a/themes/nifi/layouts/shortcodes/project-members.html +++ b/themes/nifi/layouts/shortcodes/project-members.html @@ -6,6 +6,13 @@ {{- $project = . -}} {{- end -}} +{{ $members := index $groups $project }} + +{{- with .Get "exclude-project" -}} + {{ $excluded := index $groups . }} + {{ $members = complement $excluded $members }} +{{- end -}} + @@ -14,7 +21,6 @@ - {{ $members := index $groups $project }} {{- range $username := $members -}} {{ $name := index $people $username }}
(nifi-minifi-cpp) branch main updated: MINIFICPP-2265 Add AttributeRollingWindow and EL nextInt
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 4a4113713 MINIFICPP-2265 Add AttributeRollingWindow and EL nextInt 4a4113713 is described below commit 4a41137135b318c9b11f3ed94bce4f1e5dfef765 Author: Marton Szasz AuthorDate: Mon Nov 20 22:10:53 2023 +0100 MINIFICPP-2265 Add AttributeRollingWindow and EL nextInt Signed-off-by: Marton Szasz Signed-off-by: Arpad Boda This closes #1703 --- EXPRESSIONS.md | 25 - PROCESSORS.md | 39 +++ extensions/expression-language/Expression.cpp | 8 ++ extensions/standard-processors/CMakeLists.txt | 3 +- extensions/standard-processors/RollingWindow.h | 72 .../processors/AttributeRollingWindow.cpp | 123 + .../processors/AttributeRollingWindow.h| 117 .../tests/unit/AttributeRollingWindowTests.cpp | 101 + .../tests/unit/RollingWindowTests.cpp | 108 ++ libminifi/include/core/AbstractProcessor.h | 58 ++ libminifi/src/core/extension/ExtensionManager.cpp | 7 +- libminifi/test/unit/AbstractProcessorTest.cpp | 114 +++ 12 files changed, 769 insertions(+), 6 deletions(-) diff --git a/EXPRESSIONS.md b/EXPRESSIONS.md index 00d6306a0..c61eaecd2 100644 --- a/EXPRESSIONS.md +++ b/EXPRESSIONS.md @@ -225,6 +225,7 @@ token, filename. - [`UUID`](#uuid) - [`literal`](#literal) - [`reverseDnsLookup`](#reversednslookup) +- [`nextInt`](#nextInt) ### Evaluating Multiple Attributes @@ -258,7 +259,6 @@ token, filename. ### Subjectless Functions -- `nextInt` - `getStateValue` ## Unsupported Features @@ -950,7 +950,7 @@ filename.txt". **Arguments**: No arguments -**Return Type**: String +**Return Type**: Number **Examples**: If the attribute "filename" has a value of "a brand new filename.txt" and the attribute "hello" does not exist, then the Expression @@ -1289,7 +1289,7 @@ found at the beginning of the Subject, the value returned will be `0`, not `1`. | - | - | | value | The value to search for in the Subject | -**Return Type**: Boolean +**Return Type**: Number **Examples**: @@ -1320,7 +1320,7 @@ found at the beginning of the Subject, the value returned will be `0`, not `1`. | - | - | | value | The value to search for in the Subject | -**Return Type**: Boolean +**Return Type**: Number **Examples**: @@ -1615,6 +1615,23 @@ more than 3 attributes whose names begin with the letter a. | `${reverseDnsLookup('::1')}` | `localhost` | | `${reverseDnsLookup('2001:4860:4860::'), 100}` | `dns.google` | +### nextInt + +**Description**: Returns a one-up value (starting at 0) and increasing over the +lifetime of the running instance of MiNiFi. This value is not persisted across restarts. +This counter is shared across all MiNiFi components, so calling this function multiple +times from one Processor will not guarantee sequential values within the context of a +particular Processor. + +**Subject Type**: No subject + +**Arguments**: No arguments + +**Return Type**: Number + +**Examples**: If the previous value returned by nextInt was 5, the Expression +${nextInt():divide(2)} obtains the next available integer (6) and divides the result +by 2, returning a value of 3. ## Evaluating Multiple Attributes diff --git a/PROCESSORS.md b/PROCESSORS.md index 6f1d93f1b..ee9c5bbd1 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -17,6 +17,7 @@ limitations under the License. - [AppendHostInfo](#AppendHostInfo) - [ApplyTemplate](#ApplyTemplate) +- [AttributeRollingWindow](#AttributeRollingWindow) - [AttributesToJSON](#AttributesToJSON) - [BinFiles](#BinFiles) - [CapturePacket](#CapturePacket) @@ -147,6 +148,44 @@ In the list below, the names of required properties appear in bold. Any other pr | success | success operational on the flow record | +## AttributeRollingWindow + +### Description + +Track a Rolling Window based on evaluating an Expression Language expression on each FlowFile. Each FlowFile will be emitted with the count of FlowFiles and total aggregate value of values processed in the current window. + +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name
(nifi-minifi-cpp) branch main updated: MINIFICPP-2271 update AWS SDK and regions, port patches
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new de5baead4 MINIFICPP-2271 update AWS SDK and regions, port patches de5baead4 is described below commit de5baead47490c1444dfa2b1d4b905f47ddac62a Author: Marton Szasz AuthorDate: Thu Dec 14 15:35:28 2023 +0100 MINIFICPP-2271 update AWS SDK and regions, port patches Signed-off-by: Marton Szasz Signed-off-by: Arpad Boda This closes #1705 --- cmake/BundledAwsSdkCpp.cmake | 14 +-- extensions/aws/processors/S3Processor.h| 17 +-- thirdparty/aws-sdk-cpp/bundle-openssl.patch| 57 ++ .../aws-sdk-cpp/c++20-compilation-fixes.patch | 12 --- thirdparty/aws-sdk-cpp/dll-export-injection.patch | 118 ++--- thirdparty/aws-sdk-cpp/openssl3-fix.patch | 54 -- 6 files changed, 129 insertions(+), 143 deletions(-) diff --git a/cmake/BundledAwsSdkCpp.cmake b/cmake/BundledAwsSdkCpp.cmake index ccdcca6ab..74c90bf6a 100644 --- a/cmake/BundledAwsSdkCpp.cmake +++ b/cmake/BundledAwsSdkCpp.cmake @@ -16,17 +16,13 @@ # under the License. function(use_bundled_libaws SOURCE_DIR BINARY_DIR) -set(PATCH_FILE1 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/c++20-compilation-fixes.patch") -set(PATCH_FILE2 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/dll-export-injection.patch") -set(PATCH_FILE3 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/shutdown-fix.patch") -set(PATCH_FILE4 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/bundle-openssl.patch") -set(PATCH_FILE5 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/openssl3-fix.patch") +set(PATCH_FILE1 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/dll-export-injection.patch") +set(PATCH_FILE2 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/shutdown-fix.patch") +set(PATCH_FILE3 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/bundle-openssl.patch") set(AWS_SDK_CPP_PATCH_COMMAND ${Bash_EXECUTABLE} -c "set -x &&\ (\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE1}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE1}\") &&\ (\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE2}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE2}\") &&\ -(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE3}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE3}\") &&\ -(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE4}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE4}\") &&\ -(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE5}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE5}\") ") +(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE3}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE3}\") ") if (WIN32) set(LIBDIR "lib") @@ -85,7 +81,7 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR) ExternalProject_Add( aws-sdk-cpp-external GIT_REPOSITORY "https://github.com/aws/aws-sdk-cpp.git"; -GIT_TAG "1.10.48" +GIT_TAG "1.11.219" UPDATE_COMMAND git submodule update --init --recursive SOURCE_DIR "${BINARY_DIR}/thirdparty/aws-sdk-cpp-src" INSTALL_DIR "${BINARY_DIR}/thirdparty/libaws-install" diff --git a/extensions/aws/processors/S3Processor.h b/extensions/aws/processors/S3Processor.h index 85e186eb2..19ff7b61e 100644 --- a/extensions/aws/processors/S3Processor.h +++ b/extensions/aws/processors/S3Processor.h @@ -48,6 +48,7 @@ inline constexpr std::string_view AP_NORTHEAST_1 = "ap-northeast-1"; inline constexpr std::string_view AP_NORTHEAST_2 = "ap-northeast-2"; inline constexpr std::string_view AP_NORTHEAST_3 = "ap-northeast-3"; inline constexpr std::string_view AP_SOUTH_1 = "ap-south-1"; +inline constexpr std::string_view AP_SOUTH_2 = "ap-south-2"; inline constexpr std::string_view AP_SOUTHEAST_1 = "ap-southeast-1"; inline constexpr std::string_view AP_SOUTHEAST_2 = "ap-southeast-2"; inline constexpr std::string_view AP_SOUTHEAST_3 = "ap-southeast-3"; @@ -55,8 +56,10 @@ inline constexpr std::string_view CA_CENTRAL_1 = "ca-central-1"; inline constexpr std::string_view CN_NORTH_1 = "cn-north-1"; inline constexpr std::string_view CN_NORTHWEST_1
(nifi-minifi-cpp) branch main updated: MINIFICPP-2274 Ship Markdown docs in binary packages
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 52ab84603 MINIFICPP-2274 Ship Markdown docs in binary packages 52ab84603 is described below commit 52ab84603045c349fa0f2e232e06a3d33754b705 Author: Marton Szasz AuthorDate: Wed Dec 13 17:10:26 2023 +0100 MINIFICPP-2274 Ship Markdown docs in binary packages Signed-off-by: Marton Szasz Signed-off-by: Arpad Boda This closes #1709 --- CMakeLists.txt | 11 +-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 5ed19b7a7..d73684e7b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -422,8 +422,11 @@ if(WIN32) file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/conf/" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/conf/") file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}") -file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/README.md" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}") file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/NOTICE" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}") +file(GLOB markdown_docs "${CMAKE_CURRENT_SOURCE_DIR}/*.md") +foreach(mddocfile ${markdown_docs}) +file(COPY "${mddocfile}" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}") +endforeach() # Determine the path of the VC Redistributable Merge Modules if (DEFINED ENV{VCToolsRedistDir}) @@ -531,7 +534,11 @@ if (NOT WIN32) COMPONENT bin) endif() -install(FILES LICENSE README.md NOTICE +install(FILES LICENSE NOTICE +DESTINATION . +COMPONENT bin) +file(GLOB markdown_docs "*.md") +install(FILES ${markdown_docs} DESTINATION . COMPONENT bin)
(nifi) branch support/nifi-1.x updated (0a67620c0c -> 44de1f2d67)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git from 0a67620c0c NIFI-12358 Corrected syntax for Java 8 add 44de1f2d67 NIFI-12377 - Deprecate support for Jython in NiFi 1.x line No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/nifi/script/ScriptingComponentHelper.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[nifi] branch support/nifi-1.x updated: NIFI-12095 This closes #7762. Increase default Xmx to 1g
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new 5c1ca14d33 NIFI-12095 This closes #7762. Increase default Xmx to 1g 5c1ca14d33 is described below commit 5c1ca14d330c928703326e46a1e24eaa6ae9022b Author: Nandor Soma Abonyi AuthorDate: Wed Sep 20 11:06:38 2023 +0200 NIFI-12095 This closes #7762. Increase default Xmx to 1g Signed-off-by: Joseph Witt --- .../nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml index 7139d02838..4f55ff9040 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/pom.xml @@ -25,8 +25,8 @@ Holds common resources used to build installers -512m -512m +1g +1g true
[nifi] branch main updated: NIFI-12114 Create separate test instance for python extensions
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 4b0c8bf6af NIFI-12114 Create separate test instance for python extensions 4b0c8bf6af is described below commit 4b0c8bf6af6eee30b0c9fc851a82e0fab45d35ab Author: Nandor Soma Abonyi AuthorDate: Fri Sep 22 16:47:27 2023 +0200 NIFI-12114 Create separate test instance for python extensions Signed-off-by: Arpad Boda This closes #7780 --- .github/workflows/system-tests.yml | 1 + nifi-system-tests/nifi-system-test-suite/pom.xml | 24 +++ .../org/apache/nifi/tests/system/NiFiSystemIT.java | 16 +- .../tests/system/python/PythonProcessorIT.java | 5 +- .../resources/conf/clustered/node1/nifi.properties | 2 +- .../resources/conf/clustered/node2/nifi.properties | 2 +- .../test/resources/conf/default/nifi.properties| 2 +- .../test/resources/conf/pythonic/bootstrap.conf| 33 .../src/test/resources/conf/pythonic/logback.xml | 214 + .../conf/{default => pythonic}/nifi.properties | 34 ++-- .../resources/conf/pythonic/state-management.xml | 32 +++ .../resources/conf/pythonic/zookeeper.properties | 45 + 12 files changed, 386 insertions(+), 24 deletions(-) diff --git a/.github/workflows/system-tests.yml b/.github/workflows/system-tests.yml index d1e81806f9..f2c4c41748 100644 --- a/.github/workflows/system-tests.yml +++ b/.github/workflows/system-tests.yml @@ -56,6 +56,7 @@ env: package verify -P integration-tests +-D include-python-integration-tests=true MAVEN_PROJECTS: >- -pl :nifi-python-framework -pl :nifi-python-extension-api diff --git a/nifi-system-tests/nifi-system-test-suite/pom.xml b/nifi-system-tests/nifi-system-test-suite/pom.xml index 92906f1c89..f173a353af 100644 --- a/nifi-system-tests/nifi-system-test-suite/pom.xml +++ b/nifi-system-tests/nifi-system-test-suite/pom.xml @@ -24,6 +24,30 @@ nifi-system-test-suite jar + + +include-python-integration-tests + + +include-python-integration-tests +!true + + + + + +maven-failsafe-plugin + + +PythonProcessorIT.java + + + + + + + + diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java index f0ef3741e2..b927fae922 100644 --- a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java +++ b/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/NiFiSystemIT.java @@ -201,7 +201,7 @@ public abstract class NiFiSystemIT implements NiFiInstanceProvider { .bootstrapConfig("src/test/resources/conf/default/bootstrap.conf") .instanceDirectory("target/standalone-instance") .overrideNifiProperties(getNifiPropertiesOverrides()) -.unpackPythonExtensions(isUnpackPythonExtensions()) +.unpackPythonExtensions(false) .build()); } @@ -211,6 +211,16 @@ public abstract class NiFiSystemIT implements NiFiInstanceProvider { "src/test/resources/conf/clustered/node2/bootstrap.conf"); } +public NiFiInstanceFactory createPythonicInstanceFactory() { +return new SpawnedStandaloneNiFiInstanceFactory( +new InstanceConfiguration.Builder() + .bootstrapConfig("src/test/resources/conf/pythonic/bootstrap.conf") +.instanceDirectory("target/pythonic-instance") +.overrideNifiProperties(getNifiPropertiesOverrides()) +.unpackPythonExtensions(true) +.build()); +} + protected String getTestName() { return testInfo.getDisplayName(); } @@ -545,10 +555,6 @@ public abstract class NiFiSystemIT implements NiFiInstanceProvider { return node2Dto; } -protected boolean isUnpackPythonExtensions() { -return false; -} - /** * Disconnects a node from the cluster * @param nodeIndex the 1-based index of the node diff --git a/nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/python/PythonProcessorIT.java b/nifi-system-tests/nifi-system-test
[nifi] branch support/nifi-1.x updated: NIFI-12010: Handle auto-commit and commit based on driver capabilities in SQL components
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch support/nifi-1.x in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/support/nifi-1.x by this push: new af8128bb5c NIFI-12010: Handle auto-commit and commit based on driver capabilities in SQL components af8128bb5c is described below commit af8128bb5c26e418d31ec38741648dd87607c33b Author: Matt Burgess AuthorDate: Wed Aug 30 23:30:22 2023 -0400 NIFI-12010: Handle auto-commit and commit based on driver capabilities in SQL components Signed-off-by: Arpad Boda This closes #7663 --- .../org/apache/nifi/admin/AuditDataSourceFactoryBean.java | 10 +- .../transaction/impl/StandardTransactionBuilder.java| 10 +- .../apache/nifi/processors/groovyx/ExecuteGroovyScript.java | 13 +++-- .../java/org/apache/nifi/hive/metastore/ScriptRunner.java | 5 - .../apache/nifi/processors/standard/AbstractExecuteSQL.java | 12 +++- .../java/org/apache/nifi/processors/standard/PutSQL.java| 7 ++- .../org/apache/nifi/processors/standard/TestExecuteSQL.java | 3 ++- .../org/apache/nifi/record/sink/db/DatabaseRecordSink.java | 9 - 8 files changed, 60 insertions(+), 9 deletions(-) diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java index 11d7c30e64..4bac1716c0 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java @@ -28,6 +28,7 @@ import java.io.File; import java.sql.Connection; import java.sql.ResultSet; import java.sql.SQLException; +import java.sql.SQLFeatureNotSupportedException; import java.sql.Statement; /** @@ -151,7 +152,14 @@ public class AuditDataSourceFactoryBean implements FactoryBean { try { // get a connection connection = connectionPool.getConnection(); -connection.setAutoCommit(false); +final boolean isAutoCommit = connection.getAutoCommit(); +if (isAutoCommit) { +try { +connection.setAutoCommit(false); +} catch (SQLFeatureNotSupportedException sfnse) { +logger.debug("setAutoCommit(false) not supported by this driver"); +} +} // create a statement for initializing the database statement = connection.createStatement(); diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java index 7d4a1fcc44..e4b12180c2 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java @@ -18,6 +18,7 @@ package org.apache.nifi.admin.service.transaction.impl; import java.sql.Connection; import java.sql.SQLException; +import java.sql.SQLFeatureNotSupportedException; import javax.sql.DataSource; import org.apache.nifi.admin.service.transaction.Transaction; import org.apache.nifi.admin.service.transaction.TransactionBuilder; @@ -35,7 +36,14 @@ public class StandardTransactionBuilder implements TransactionBuilder { try { // get a new connection Connection connection = dataSource.getConnection(); -connection.setAutoCommit(false); +final boolean isAutoCommit = connection.getAutoCommit(); +if (isAutoCommit) { +try { +connection.setAutoCommit(false); +} catch (SQLFeatureNotSupportedException sfnse) { +throw new TransactionException("setAutoCommit(false) not supported by this driver"); +} +} // create a new transaction return new StandardTransaction(connection); diff --git a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java b/nifi-nar-bundles/nifi-groovyx
[nifi] branch main updated: NIFI-12010: Handle auto-commit and commit based on driver capabilities in SQL components
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 932cfe22a3 NIFI-12010: Handle auto-commit and commit based on driver capabilities in SQL components 932cfe22a3 is described below commit 932cfe22a379ea83e11fe283a3f789115cce1607 Author: Matt Burgess AuthorDate: Wed Aug 30 23:30:22 2023 -0400 NIFI-12010: Handle auto-commit and commit based on driver capabilities in SQL components Signed-off-by: Arpad Boda This closes #7663 --- .../org/apache/nifi/admin/AuditDataSourceFactoryBean.java | 10 +- .../transaction/impl/StandardTransactionBuilder.java| 10 +- .../apache/nifi/processors/groovyx/ExecuteGroovyScript.java | 13 +++-- .../java/org/apache/nifi/hive/metastore/ScriptRunner.java | 5 - .../apache/nifi/processors/standard/AbstractExecuteSQL.java | 12 +++- .../java/org/apache/nifi/processors/standard/PutSQL.java| 7 ++- .../org/apache/nifi/processors/standard/TestExecuteSQL.java | 3 ++- .../org/apache/nifi/record/sink/db/DatabaseRecordSink.java | 9 - 8 files changed, 60 insertions(+), 9 deletions(-) diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java index 22165d2b68..040a207dcb 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/AuditDataSourceFactoryBean.java @@ -27,6 +27,7 @@ import java.io.File; import java.sql.Connection; import java.sql.ResultSet; import java.sql.SQLException; +import java.sql.SQLFeatureNotSupportedException; import java.sql.Statement; /** @@ -146,7 +147,14 @@ public class AuditDataSourceFactoryBean implements FactoryBean { try { // get a connection connection = connectionPool.getConnection(); -connection.setAutoCommit(false); +final boolean isAutoCommit = connection.getAutoCommit(); +if (isAutoCommit) { +try { +connection.setAutoCommit(false); +} catch (SQLFeatureNotSupportedException sfnse) { +logger.debug("setAutoCommit(false) not supported by this driver"); +} +} // create a statement for initializing the database statement = connection.createStatement(); diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java index 7d4a1fcc44..e4b12180c2 100644 --- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java +++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-administration/src/main/java/org/apache/nifi/admin/service/transaction/impl/StandardTransactionBuilder.java @@ -18,6 +18,7 @@ package org.apache.nifi.admin.service.transaction.impl; import java.sql.Connection; import java.sql.SQLException; +import java.sql.SQLFeatureNotSupportedException; import javax.sql.DataSource; import org.apache.nifi.admin.service.transaction.Transaction; import org.apache.nifi.admin.service.transaction.TransactionBuilder; @@ -35,7 +36,14 @@ public class StandardTransactionBuilder implements TransactionBuilder { try { // get a new connection Connection connection = dataSource.getConnection(); -connection.setAutoCommit(false); +final boolean isAutoCommit = connection.getAutoCommit(); +if (isAutoCommit) { +try { +connection.setAutoCommit(false); +} catch (SQLFeatureNotSupportedException sfnse) { +throw new TransactionException("setAutoCommit(false) not supported by this driver"); +} +} // create a new transaction return new StandardTransaction(connection); diff --git a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java b/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-proc
[nifi-minifi-cpp] branch main updated: MINIFICPP-2010 Disable systemd on non-linux remove option from bootstrap UI
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new d6912551b MINIFICPP-2010 Disable systemd on non-linux remove option from bootstrap UI d6912551b is described below commit d6912551b2e356db6ed55b7a0a58843e4b92fca8 Author: Marton Szasz AuthorDate: Fri Dec 9 16:44:56 2022 +0100 MINIFICPP-2010 Disable systemd on non-linux remove option from bootstrap UI Signed-off-by: Arpad Boda This closes #1470 --- bootstrap.sh | 10 +- bstrp_functions.sh| 8 +--- extensions/systemd/CMakeLists.txt | 2 +- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/bootstrap.sh b/bootstrap.sh index 15a887ed5..cd4f9e8b3 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -163,6 +163,12 @@ export OS_MINOR OS_REVISION=$(echo "$EVR" | cut -d. -f3) export OS_REVISION +if [[ "$OSTYPE" =~ .*linux.* ]]; then + LINUX=true +else + LINUX=false +fi + ### Verify the compiler version COMPILER_VERSION="0.0.0" @@ -328,7 +334,9 @@ add_dependency OPC_ENABLED "mbedtls" add_option AZURE_ENABLED ${TRUE} "ENABLE_AZURE" -add_option SYSTEMD_ENABLED ${TRUE} "ENABLE_SYSTEMD" +if $LINUX; then + add_option SYSTEMD_ENABLED ${TRUE} "ENABLE_SYSTEMD" +fi add_option NANOFI_ENABLED ${FALSE} "ENABLE_NANOFI" set_dependency PYTHON_ENABLED NANOFI_ENABLED diff --git a/bstrp_functions.sh b/bstrp_functions.sh index 778c5dd51..7ce885e60 100755 --- a/bstrp_functions.sh +++ b/bstrp_functions.sh @@ -380,7 +380,9 @@ show_supported_features() { echo "V. SQL Support..$(print_feature_status SQL_ENABLED)" echo "W. Openwsman Support ...$(print_feature_status OPENWSMAN_ENABLED)" echo "X. Azure Support ...$(print_feature_status AZURE_ENABLED)" - echo "Y. Systemd Support .$(print_feature_status SYSTEMD_ENABLED)" + if $LINUX; then +echo "Y. Systemd Support .$(print_feature_status SYSTEMD_ENABLED)" + fi echo "Z. NanoFi Support ..$(print_feature_status NANOFI_ENABLED)" echo "AA. Splunk Support .$(print_feature_status SPLUNK_ENABLED)" echo "AB. Kubernetes Support .$(print_feature_status KUBERNETES_ENABLED)" @@ -430,7 +432,7 @@ read_feature_options(){ ToggleFeature PYTHON_ENABLED else echo -e "${RED}Please ensure static linking is enabled for Python Support...${NO_COLOR}" && sleep 2 - fi + fi ;; n) ToggleFeature COAP_ENABLED ;; o) ToggleFeature SFTP_ENABLED ;; @@ -440,7 +442,7 @@ read_feature_options(){ v) ToggleFeature SQL_ENABLED ;; w) ToggleFeature OPENWSMAN_ENABLED ;; x) ToggleFeature AZURE_ENABLED ;; -y) ToggleFeature SYSTEMD_ENABLED ;; +y) if $LINUX; then ToggleFeature SYSTEMD_ENABLED; fi ;; z) ToggleFeature NANOFI_ENABLED ;; aa) ToggleFeature SPLUNK_ENABLED ;; ab) ToggleFeature KUBERNETES_ENABLED ;; diff --git a/extensions/systemd/CMakeLists.txt b/extensions/systemd/CMakeLists.txt index 9c1356569..9a77ee255 100644 --- a/extensions/systemd/CMakeLists.txt +++ b/extensions/systemd/CMakeLists.txt @@ -17,7 +17,7 @@ # under the License. # -if (NOT ENABLE_SYSTEMD) +if (NOT ENABLE_SYSTEMD OR NOT CMAKE_SYSTEM_NAME STREQUAL "Linux") return() endif()
[nifi] 01/01: NIFI-10833 - Fix grammar error in ListenHTTP log msg
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch NIFI-10833 in repository https://gitbox.apache.org/repos/asf/nifi.git commit ffcfbc8dc73840c1e501cad94bbcf21e41a17008 Author: Arpad Boda AuthorDate: Wed Nov 16 21:51:37 2022 +0100 NIFI-10833 - Fix grammar error in ListenHTTP log msg --- .../src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java index a35b9edc73..14a4ff3145 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java @@ -562,7 +562,7 @@ public class ListenHTTP extends AbstractSessionFactoryProcessor { for (final String id : findOldFlowFileIds(context)) { final FlowFileEntryTimeWrapper wrapper = flowFileMap.remove(id); if (wrapper != null) { -getLogger().warn("failed to received acknowledgment for HOLD with ID {} sent by {}; rolling back session", id, wrapper.getClientIP()); +getLogger().warn("failed to receive acknowledgment for HOLD with ID {} sent by {}; rolling back session", id, wrapper.getClientIP()); wrapper.session.rollback(); } }
[nifi] branch NIFI-10833 created (now ffcfbc8dc7)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch NIFI-10833 in repository https://gitbox.apache.org/repos/asf/nifi.git at ffcfbc8dc7 NIFI-10833 - Fix grammar error in ListenHTTP log msg This branch includes the following new commits: new ffcfbc8dc7 NIFI-10833 - Fix grammar error in ListenHTTP log msg The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[nifi-minifi-cpp] 01/02: MINIFICPP-1963 improve exception logging
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 9cd6d868ffb1ca98ade9a0b3756d363985c300f6 Author: Marton Szasz AuthorDate: Wed Oct 19 17:36:38 2022 +0200 MINIFICPP-1963 improve exception logging more info in SchedulingAgent::onTrigger catch-all and bump exception message log level from debug to warning Signed-off-by: Arpad Boda This closes #1436 --- extensions/libarchive/MergeContent.cpp | 5 ++- libminifi/include/Exception.h | 16 +++- libminifi/src/Exception.cpp| 41 ++ libminifi/src/FlowController.cpp | 5 ++- libminifi/src/SchedulingAgent.cpp | 11 +++-- libminifi/src/c2/C2Agent.cpp | 4 +- libminifi/src/c2/C2Client.cpp | 8 +++- libminifi/src/core/ProcessGroup.cpp| 12 +++--- libminifi/src/core/ProcessSession.cpp | 48 +++--- libminifi/src/core/Processor.cpp | 7 ++-- libminifi/src/sitetosite/RawSocketProtocol.cpp | 4 +- libminifi/src/sitetosite/SiteToSiteClient.cpp | 8 ++-- .../integration/StateTransactionalityTests.cpp | 6 +-- 13 files changed, 111 insertions(+), 64 deletions(-) diff --git a/extensions/libarchive/MergeContent.cpp b/extensions/libarchive/MergeContent.cpp index 4a2d69f7d..9af2194b7 100644 --- a/extensions/libarchive/MergeContent.cpp +++ b/extensions/libarchive/MergeContent.cpp @@ -253,8 +253,11 @@ bool MergeContent::processBin(core::ProcessContext *context, core::ProcessSessio try { mergeBin->merge(context, session, bin->getFlowFile(), *serializer, merge_flow); session->putAttribute(merge_flow, core::SpecialFlowAttribute::MIME_TYPE, mimeType); + } catch (const std::exception& ex) { +logger_->log_error("Merge Content merge catch exception, type: %s, what: %s", typeid(ex).name(), ex.what()); +return false; } catch (...) { -logger_->log_error("Merge Content merge catch exception"); +logger_->log_error("Merge Content merge catch exception, type: %s", getCurrentExceptionTypeName()); return false; } session->putAttribute(merge_flow, BinFiles::FRAGMENT_COUNT_ATTRIBUTE, std::to_string(bin->getSize())); diff --git a/libminifi/include/Exception.h b/libminifi/include/Exception.h index 634060d65..55fccefc4 100644 --- a/libminifi/include/Exception.h +++ b/libminifi/include/Exception.h @@ -17,8 +17,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -#ifndef LIBMINIFI_INCLUDE_EXCEPTION_H_ -#define LIBMINIFI_INCLUDE_EXCEPTION_H_ +#pragma once #include #include @@ -31,10 +30,7 @@ #include "utils/StringUtils.h" -namespace org { -namespace apache { -namespace nifi { -namespace minifi { +namespace org::apache::nifi::minifi { enum ExceptionType { FILE_OPERATION_EXCEPTION = 0, @@ -59,6 +55,8 @@ inline const char *ExceptionTypeToString(ExceptionType type) { return nullptr; } +std::string getCurrentExceptionTypeName(); + struct Exception : public std::runtime_error { /*! * Create a new exception @@ -92,8 +90,4 @@ struct SystemErrorException : Exception { std::error_condition error_condition_; }; -} // namespace minifi -} // namespace nifi -} // namespace apache -} // namespace org -#endif // LIBMINIFI_INCLUDE_EXCEPTION_H_ +} // namespace org::apache::nifi::minifi diff --git a/libminifi/src/Exception.cpp b/libminifi/src/Exception.cpp new file mode 100644 index 0..c9e622077 --- /dev/null +++ b/libminifi/src/Exception.cpp @@ -0,0 +1,41 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "Exception.h" + +#ifndef WIN32 +#include +#endif // !WIN32 + +#include + +namespace org::apache::nifi::minifi { +std::string getCurrentExceptionTypeName() { +#ifndef WIN32 + const std::type_info* exception_type = abi::__cxa_current_exception_type(); + if (exception_type) { +
[nifi-minifi-cpp] branch main updated (b00db7438 -> c5d58e883)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git from b00db7438 MINIFICPP-1952 Reset the callback when no longer needed new 9cd6d868f MINIFICPP-1963 improve exception logging new c5d58e883 MINIFICPP-1967 Add batch processing of lines in TailFile The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: PROCESSORS.md | 1 + extensions/libarchive/MergeContent.cpp | 5 ++- .../standard-processors/processors/TailFile.cpp| 14 ++- .../standard-processors/processors/TailFile.h | 23 --- .../tests/unit/TailFileTests.cpp | 30 ++ libminifi/include/Exception.h | 16 +++- .../{include/SwapManager.h => src/Exception.cpp} | 38 - libminifi/src/FlowController.cpp | 5 ++- libminifi/src/SchedulingAgent.cpp | 11 +++-- libminifi/src/c2/C2Agent.cpp | 4 +- libminifi/src/c2/C2Client.cpp | 8 +++- libminifi/src/core/ProcessGroup.cpp| 12 +++--- libminifi/src/core/ProcessSession.cpp | 48 +++--- libminifi/src/core/Processor.cpp | 7 ++-- libminifi/src/sitetosite/RawSocketProtocol.cpp | 4 +- libminifi/src/sitetosite/SiteToSiteClient.cpp | 8 ++-- .../integration/StateTransactionalityTests.cpp | 6 +-- 17 files changed, 141 insertions(+), 99 deletions(-) copy libminifi/{include/SwapManager.h => src/Exception.cpp} (63%)
[nifi-minifi-cpp] 02/02: MINIFICPP-1967 Add batch processing of lines in TailFile
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit c5d58e8836723ba232440d457c6ff1be811bd075 Author: Gabor Gyimesi AuthorDate: Thu Oct 20 13:13:46 2022 +0200 MINIFICPP-1967 Add batch processing of lines in TailFile Signed-off-by: Arpad Boda This closes #1439 --- PROCESSORS.md | 1 + .../standard-processors/processors/TailFile.cpp| 14 +- .../standard-processors/processors/TailFile.h | 23 ++--- .../tests/unit/TailFileTests.cpp | 30 ++ 4 files changed, 52 insertions(+), 16 deletions(-) diff --git a/PROCESSORS.md b/PROCESSORS.md index 8e7e6e328..8a0aac5de 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -2424,6 +2424,7 @@ In the list below, the names of required properties appear in bold. Any other pr | State File | TailFileState | | Specifies the file that should be used for storing state about what data has been ingested so that upon restart NiFi can resume from where it left off [...] | tail-base-directory| | | Base directory used to look for files to tail. This property is required when using Multiple file mode. Can contain expression language placeholders if Attribute Provider Service is set.**Supports Expression Language: true** [...] | **tail-mode** | Single file | Single fileMultiple file | Specifies the tail file mode. In 'Single file' mode only a single file will be watched. In 'Multiple file' mode a regex may be used. Note that in multiple file mode we will still continue to watch for rollover on the initial set of watched files. The Regex used to locate multiple files will be run during the schedule phrase. Note that if rotated files are matched by the regex, thos [...] +| **Batch Size** | 0 | | Maximum number of flowfiles emitted in a single trigger. If set to 0 all new content will be processed. [...] ### Relationships | Name| Description | diff --git a/extensions/standard-processors/processors/TailFile.cpp b/extensions/standard-processors/processors/TailFile.cpp index 7a3f20194..abfd6784b 100644 --- a/extensions/standard-processors/processors/TailFile.cpp +++ b/extensions/standard-processors/processors/TailFile.cpp @@ -135,6 +135,13 @@ const core::Property TailFile::AttributeProviderService( ->asType() ->build()); +const core::Property TailFile::BatchSize( +core::PropertyBuilder::createProperty("Batch Size") +->withDescription("Maximum number of flowfiles emitted in a single trigger. If set to 0 all new content will be processed.") +->isRequired(true) +->withDefaultValue(0) +->build()); + const core::Relationship TailFile::Success("success", "All files are routed to success"); const char *TailFile::CURRENT_STR = "CURRENT."; @@ -395,6 +402,11 @@ void TailFile::onSchedule(const std::shared_ptr &context, context->getProperty(RollingFilenamePattern.getName(), rolling_filename_pattern_glob); rolling_filename_pattern_ = utils::file::globToRegex(rolling_filename_pattern_glob); initial_start_position_ = InitialStartPositions{utils::parsePropertyWithAllowableValuesOrThrow(*context, InitialStartPosition.getName(), InitialStartPositions::values())}; + + uint32_t batch_size = 0; + if (context->getProperty(BatchSize.getName(), batch_size) && batch_size != 0) { +batch_size_ = batch_size; + } } void TailFile::parseAttributeProviderServiceProperty(core::ProcessContext& context) { @@ -784,7 +796,7 @@ void TailFile::processSingleFile(const std::shared_ptr &se FileReaderCallback file_reader{full_file_name, state.position_, delim, state.checksum_}; TailState state_copy{state}; -while (file_reader.hasMoreToRead()) { +while (file_rea
[nifi] branch main updated: NIFI-10460 GetZendesk processor
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new 0fa923159f NIFI-10460 GetZendesk processor 0fa923159f is described below commit 0fa923159f0d238e986a8dbe6edfc2acbe785f68 Author: Ferenc Kis AuthorDate: Tue Sep 6 17:10:18 2022 +0200 NIFI-10460 GetZendesk processor Signed-off-by: Arpad Boda This closes #6411 --- nifi-assembly/pom.xml | 6 + .../nifi-zendesk-bundle/nifi-zendesk-nar/pom.xml | 46 +++ .../src/main/resources/META-INF/LICENSE| 204 .../src/main/resources/META-INF/NOTICE | 39 +++ .../nifi-zendesk-processors/pom.xml| 91 ++ .../apache/nifi/processors/zendesk/GetZendesk.java | 361 + .../zendesk/ZendeskAuthenticationType.java | 74 + .../processors/zendesk/ZendeskExportMethod.java| 85 + .../nifi/processors/zendesk/ZendeskResource.java | 97 ++ .../services/org.apache.nifi.processor.Processor | 15 + .../additionalDetails.html | 71 .../nifi/processors/zendesk/GetZendeskTest.java| 267 +++ .../zendesk/ZendeskAuthenticationTypeTest.java | 41 +++ nifi-nar-bundles/nifi-zendesk-bundle/pom.xml | 33 ++ nifi-nar-bundles/pom.xml | 1 + 15 files changed, 1431 insertions(+) diff --git a/nifi-assembly/pom.xml b/nifi-assembly/pom.xml index d923e3ae75..78f67698e8 100644 --- a/nifi-assembly/pom.xml +++ b/nifi-assembly/pom.xml @@ -892,6 +892,12 @@ language governing permissions and limitations under the License. --> 1.18.0-SNAPSHOT nar + +org.apache.nifi +nifi-zendesk-nar +1.18.0-SNAPSHOT +nar + org.apache.nifi nifi-dropbox-processors-nar diff --git a/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/pom.xml b/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/pom.xml new file mode 100644 index 00..b84a6fcdfc --- /dev/null +++ b/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/pom.xml @@ -0,0 +1,46 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + + +org.apache.nifi +nifi-zendesk-bundle +1.18.0-SNAPSHOT + + +nifi-zendesk-nar +nar + +true +true + + + + +org.apache.nifi +nifi-zendesk-processors +1.18.0-SNAPSHOT + + +org.apache.nifi +nifi-standard-services-api-nar +1.18.0-SNAPSHOT +nar + + + + diff --git a/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/src/main/resources/META-INF/LICENSE b/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/src/main/resources/META-INF/LICENSE new file mode 100644 index 00..de4b130f35 --- /dev/null +++ b/nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-nar/src/main/resources/META-INF/LICENSE @@ -0,0 +1,204 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to co
[nifi] branch main updated (111c7ac0a4 -> 35fb66f50f)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git from 111c7ac0a4 NIFI-10444 FetchDropbox processor add 35fb66f50f NIFI-10403 Add processor supporting the new BigQuery Write API No new revisions were added by this update. Summary of changes: .../apache/nifi/serialization/record/Record.java | 2 +- .../nifi-gcp-bundle/nifi-gcp-processors/pom.xml| 14 + .../processors/gcp/bigquery/BigQueryUtils.java | 1 - .../nifi/processors/gcp/bigquery/PutBigQuery.java | 451 + .../processors/gcp/bigquery/PutBigQueryBatch.java | 6 +- .../gcp/bigquery/PutBigQueryStreaming.java | 20 +- .../processors/gcp/bigquery/proto/ProtoUtils.java | 66 +++ .../services/org.apache.nifi.processor.Processor | 1 + .../additionalDetails.html | 58 +++ .../processors/gcp/bigquery/AbstractBQTest.java| 26 +- .../gcp/bigquery/PutBigQueryBatchIT.java | 139 --- .../gcp/bigquery/PutBigQueryBatchTest.java | 39 +- .../processors/gcp/bigquery/PutBigQueryIT.java | 368 + .../gcp/bigquery/PutBigQueryStreamingIT.java | 306 -- .../processors/gcp/bigquery/PutBigQueryTest.java | 447 ...streaming-correct-data-with-date-formatted.json | 4 +- 16 files changed, 1439 insertions(+), 509 deletions(-) create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQuery.java create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/proto/ProtoUtils.java create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/resources/docs/org.apache.nifi.processors.gcp.bigquery.PutBigQuery/additionalDetails.html delete mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryBatchIT.java create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryIT.java delete mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryStreamingIT.java create mode 100644 nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryTest.java
[nifi] branch main updated (4b0568e6fe -> 5303aadda3)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git from 4b0568e6fe NIFI-10242 Create QueryAirtableTable processor add 5303aadda3 NIFI-10500: Improved property ordering in MQTT processors No new revisions were added by this update. Summary of changes: .../apache/nifi/processors/mqtt/ConsumeMQTT.java | 77 -- .../apache/nifi/processors/mqtt/PublishMQTT.java | 50 -- .../mqtt/common/AbstractMQTTProcessor.java | 77 +++--- .../nifi/processors/mqtt/TestConsumeMQTT.java | 4 -- 4 files changed, 98 insertions(+), 110 deletions(-)
[nifi] branch main updated: NIFI-10395 Added Apache Xalan to banned dependencies
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new d9b3257e33 NIFI-10395 Added Apache Xalan to banned dependencies new 36ae680147 Merge pull request #6336 from exceptionfactory/NIFI-10395 d9b3257e33 is described below commit d9b3257e33da272370a625efcedfd7e46f003a53 Author: exceptionfactory AuthorDate: Thu Aug 25 10:50:15 2022 -0500 NIFI-10395 Added Apache Xalan to banned dependencies --- pom.xml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pom.xml b/pom.xml index a9f3a9de52..b560c7fc7c 100644 --- a/pom.xml +++ b/pom.xml @@ -859,6 +859,8 @@ org.apache.logging.log4j:log4j-core:* commons-logging:commons-logging:* + +xalan:xalan
[nifi] branch NIFI-10255 created (now d3ec916e0a)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch NIFI-10255 in repository https://gitbox.apache.org/repos/asf/nifi.git at d3ec916e0a NIFI-10255 - Decription of original relationship of ExecuteStreamCommand processor might be misleading This branch includes the following new commits: new d3ec916e0a NIFI-10255 - Decription of original relationship of ExecuteStreamCommand processor might be misleading The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[nifi] 01/01: NIFI-10255 - Decription of original relationship of ExecuteStreamCommand processor might be misleading
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch NIFI-10255 in repository https://gitbox.apache.org/repos/asf/nifi.git commit d3ec916e0ae98305d797752e36074d228e08d681 Author: Arpad Boda AuthorDate: Thu Jul 21 01:08:52 2022 +0200 NIFI-10255 - Decription of original relationship of ExecuteStreamCommand processor might be misleading --- .../org/apache/nifi/processors/standard/ExecuteStreamCommand.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java index 2ebefcb1f1..00093bb972 100644 --- a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java +++ b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java @@ -173,7 +173,7 @@ public class ExecuteStreamCommand extends AbstractProcessor { public static final Relationship ORIGINAL_RELATIONSHIP = new Relationship.Builder() .name("original") -.description("FlowFiles that were successfully processed.") +.description("The original FlowFile will be routed. It will have new attributes detailing the result of the script execution.") .build(); public static final Relationship OUTPUT_STREAM_RELATIONSHIP = new Relationship.Builder() .name("output stream") @@ -636,4 +636,4 @@ public class ExecuteStreamCommand extends AbstractProcessor { writerThread.setDaemon(true); writerThread.start(); } -} \ No newline at end of file +}
[nifi-minifi-cpp] branch main updated: MINIFICPP-1799 Fix PutGCSObject::Key property name
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 5ef439fd9 MINIFICPP-1799 Fix PutGCSObject::Key property name 5ef439fd9 is described below commit 5ef439fd9f98cf4fe8199c513f18c4ca50e2fbeb Author: Martin Zink AuthorDate: Wed Apr 13 16:36:34 2022 +0200 MINIFICPP-1799 Fix PutGCSObject::Key property name Signed-off-by: Arpad Boda This closes #1303 --- extensions/gcp/processors/PutGCSObject.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions/gcp/processors/PutGCSObject.cpp b/extensions/gcp/processors/PutGCSObject.cpp index 45d898d9b..9fadfedfe 100644 --- a/extensions/gcp/processors/PutGCSObject.cpp +++ b/extensions/gcp/processors/PutGCSObject.cpp @@ -46,7 +46,7 @@ const core::Property PutGCSObject::Bucket( ->build()); const core::Property PutGCSObject::Key( -core::PropertyBuilder::createProperty("Name of the object.") +core::PropertyBuilder::createProperty("Key") ->withDescription("Name of the object.") ->withDefaultValue("${filename}") ->supportsExpressionLanguage(true)
[nifi-minifi-cpp] branch main updated: MINIFICPP-1771: Reworked ListenSyslog
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 4a1d35b2c MINIFICPP-1771: Reworked ListenSyslog 4a1d35b2c is described below commit 4a1d35b2cc927be464031afbed32509013e0874d Author: Martin Zink AuthorDate: Thu Mar 10 09:27:58 2022 +0100 MINIFICPP-1771: Reworked ListenSyslog Signed-off-by: Arpad Boda This closes #1294 --- LICENSE| 28 ++ NOTICE | 1 + PROCESSORS.md | 57 ++- .../CMakeLists.txt => cmake/Asio.cmake | 27 +- .../integration/features/syslog_listener.feature | 26 ++ .../minifi/core/SingleNodeDockerCluster.py | 6 + .../minifi/core/SyslogTcpClientContainer.py| 23 + .../minifi/core/SyslogUdpClientContainer.py| 23 + .../integration/minifi/processors/ListenSyslog.py | 12 + docker/test/integration/steps/steps.py | 10 + extensions/gcp/tests/PutGCSObjectTests.cpp | 40 +- extensions/standard-processors/CMakeLists.txt | 3 +- .../processors/ListenSyslog.cpp| 511 + .../standard-processors/processors/ListenSyslog.h | 326 ++--- .../tests/unit/FetchFileTests.cpp | 51 +- .../tests/unit/ListenSyslogTests.cpp | 488 .../standard-processors/tests/unit/PutUDPTests.cpp | 10 +- libminifi/include/core/PropertyValidation.h| 14 +- ...ontroller.h => SingleProcessorTestController.h} | 14 +- libminifi/test/TestBase.cpp| 2 + 20 files changed, 1118 insertions(+), 554 deletions(-) diff --git a/LICENSE b/LICENSE index 8c3a9a21d..836615ad9 100644 --- a/LICENSE +++ b/LICENSE @@ -3181,6 +3181,34 @@ FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +-- + +This project bundles 'asio' under the Boost Software License 1.0 + +Boost Software License - Version 1.0 - August 17th, 2003 + +Permission is hereby granted, free of charge, to any person or organization +obtaining a copy of the software and accompanying documentation covered by +this license (the "Software") to use, reproduce, display, distribute, +execute, and transmit the Software, and to prepare derivative works of the +Software, and to permit third-parties to whom the Software is furnished to +do so, all subject to the following: + +The copyright notices in the Software and this entire statement, including +the above license grant, this restriction and the following disclaimer, +must be included in all copies of the Software, in whole or in part, and +all derivative works of the Software, unless such copies or derivative +works are solely in the form of machine-executable object code generated by +a source language processor. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT +SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE +FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + --- This project reuses test code from TartanLlama/expected from the Public Domain or under CC0 diff --git a/NOTICE b/NOTICE index ce26b3153..0e66d48fc 100644 --- a/NOTICE +++ b/NOTICE @@ -59,6 +59,7 @@ This software includes third party software subject to the following copyrights: - date (HowardHinnant/date) - notices below - range-v3 - Eric Niebler and other contributors - expected-lite - Copyright (C) 2016-2020 Martin Moene. +- asio - Copyright (c) 2003-2022 Christopher M. Kohlhoff (chris at kohlhoff dot com) - TartanLlama/expected - public domain, thanks to Sy Brand - libyaml - Copyright (c) 2006-2016 Kirill Simonov, Copyright (c) 2017-2020 Ingy döt Net - libwebsockets - Copyright (C) 2010 - 2020 Andy Green diff --git a/PROCESSORS.md b/PROCESSORS.md index 4290a8b25..dbe84270a 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -1,4 +1,4 @@ -
[nifi-minifi-cpp] branch main updated: MINIFICPP-1708 - Route flowfiles to Failure on decompression error
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 65fad0d02 MINIFICPP-1708 - Route flowfiles to Failure on decompression error 65fad0d02 is described below commit 65fad0d02d04463181470b21c8ea290ddd61c52a Author: Adam Debreceni AuthorDate: Mon Apr 11 11:15:07 2022 +0200 MINIFICPP-1708 - Route flowfiles to Failure on decompression error Signed-off-by: Arpad Boda This closes #1298 --- extensions/libarchive/CompressContent.cpp | 17 +- .../test/archive-tests/CompressContentTests.cpp| 217 ++--- 2 files changed, 116 insertions(+), 118 deletions(-) diff --git a/extensions/libarchive/CompressContent.cpp b/extensions/libarchive/CompressContent.cpp index 421bc8d83..874eb72c0 100644 --- a/extensions/libarchive/CompressContent.cpp +++ b/extensions/libarchive/CompressContent.cpp @@ -175,7 +175,7 @@ void CompressContent::processFlowFile(const std::shared_ptr& flo fileExtension = search->second; } std::shared_ptr result = session->create(flowFile); - bool success = false; + bool success = true; if (encapsulateInTar_) { std::function&, const std::shared_ptr&)> transformer; @@ -193,9 +193,15 @@ void CompressContent::processFlowFile(const std::shared_ptr& flo transformer = [&] (const std::shared_ptr& in, const std::shared_ptr& out) -> int64_t { io::ReadArchiveStreamImpl decompressor(in); if (!decompressor.nextEntry()) { - return -1; + success = false; + return 0; // prevents a session rollback +} +auto ret = internal::pipe(&decompressor, out.get()); +if (ret < 0) { + success = false; + return 0; // prevents a session rollback } -return internal::pipe(&decompressor, out.get()); +return ret; }; } session->write(result, FunctionOutputStreamCallback([&] (const auto& out) { @@ -203,11 +209,6 @@ void CompressContent::processFlowFile(const std::shared_ptr& flo return transformer(in, out); })); })); -// TODO(adebreceni): previous attempt to handle a malformed archive were in vain -//as the session->read threw anyway rolling back the flowfile, we should correctly -//forward a malformed archive to failure -//https://issues.apache.org/jira/browse/MINIFICPP-1708 -success = true; } else { CompressContent::GzipWriteCallback callback(compressMode_, compressLevel_, flowFile, session); session->write(result, &callback); diff --git a/libminifi/test/archive-tests/CompressContentTests.cpp b/libminifi/test/archive-tests/CompressContentTests.cpp index bd1957797..6804e3a45 100644 --- a/libminifi/test/archive-tests/CompressContentTests.cpp +++ b/libminifi/test/archive-tests/CompressContentTests.cpp @@ -123,7 +123,7 @@ class CompressDecompressionTestController : public TestController{ std::shared_ptr content_repo = std::make_shared(); content_repo->initialize(std::make_shared()); -// connection from compress processor to log attribute +// connection from compress processor to success output = std::make_shared(repo, content_repo, "Output"); output->addRelationship(core::Relationship("success", "compress successful output")); output->setSource(processor.get()); @@ -135,12 +135,47 @@ class CompressDecompressionTestController : public TestController{ input->setDestinationUUID(processoruuid); processor->addConnection(input.get()); -processor->setAutoTerminatedRelationships({{"failure", ""}}); +// connection from compress processor to failure +failure_output = std::make_shared(repo, content_repo, "FailureOutput"); +failure_output->addRelationship(core::Relationship("failure", "compress failure output")); +failure_output->setSource(processor.get()); +failure_output->setSourceUUID(processoruuid); +processor->addConnection(failure_output.get()); processor->incrementActiveTasks(); processor->setScheduledState(core::ScheduledState::RUNNING); context = std::make_shared(std::make_shared(processor.get()), nullptr, repo, repo, content_repo); +helper_session = std::make_shared(context); + } + + std::shared_ptr importFlowFile(const std::string& content_path) { +std::shared_ptr flow = std::static_pointer_cast(helper_session->create()); +helper_session->import(content_path, flow, true, 0); +helper_session->flushContent(); +input->put(flow); +return flow; + } + + template + std::shared_ptr importFlowFileFrom(T&& source) { +
svn commit: r53789 - /release/nifi/KEYS
Author: aboda Date: Wed Apr 13 11:20:17 2022 New Revision: 53789 Log: Added new public key of Arpad Boda Modified: release/nifi/KEYS Modified: release/nifi/KEYS == --- release/nifi/KEYS (original) +++ release/nifi/KEYS Wed Apr 13 11:20:17 2022 @@ -1855,3 +1855,92 @@ RUN7+JEPRvIjpFFMOIlMB2Mkiwlo1Be/67zuAS0d TdbG6AMfmlsrDh8= =XV51 -END PGP PUBLIC KEY BLOCK- +pub rsa2048 2019-10-24 [SC] [expired: 2021-10-23] + EE78BB8A82A7851477AAAD0A390C1B5ADE978835 +uid [ expired] Arpad Boda +sig 3390C1B5ADE978835 2019-10-24 Arpad Boda + +pub rsa4096 2022-04-13 [SC] [expires: 2024-04-12] + D6C8A7149D47EF7AA08B35D1065668F2A58F097F +uid [ultimate] Arpad Boda +sig 3065668F2A58F097F 2022-04-13 Arpad Boda +sub rsa4096 2022-04-13 [E] [expires: 2024-04-12] +sig 065668F2A58F097F 2022-04-13 Arpad Boda + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQENBF2xedcBCADhvZxpFnZ37r3LuC4ZBNJu5OX4ORkXSWOgo7JwGb0pDGWrQlcq +ZU57fjeGvyciLPF7ICAOKDGCF2YYrRfCG2IhI4S6B+pYpIs9H9XnfoVLUOYgCQAl +VsugMlMXAnLb5MNNFEV6XLQooqvFkTtsbRcEsES6DogBeYlMoF2koJkY5W/Evpcs +96nwVH9cTtdahOx0Er3U5vNoflSsWqMXQPK+IRviaJP9f3e+pLnV2179e2TX59PV +bBwXKGwXAoKq19TFwmLAUjuHDKccynwwWenEzQ5CUiUTwtwxzp85hiVZkG07WWST +aOyGYIUnYUGRQiis4/ElXoBpS6mmld4zlbV7ABEBAAG0HUFycGFkIEJvZGEgPGFi +b2RhQGFwYWNoZS5vcmc+iQFUBBMBCgA+FiEE7ni7ioKnhRR3qq0KOQwbWt6XiDUF +Al2xedcCGwMFCQPCZwAFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQOQwbWt6X +iDXu/Qf8CS1g3ErUBwUkRPegJTjwTAxTEwyxMz2h8rYZMW2Tx55KWyRIqt0FxVqm +T7NtEeSoJbkIOcokaaOX673UmM5GJo2SaKqiBOzmYWPDiFYLHnvpfaDV9kDOSJG5 +ohhfSPGQX4x2L2GNO75h9bVSd6gITCSqq9FlyLAoaNZj5nS+1Px9ltAHpli4CQiP ++uCe3alFF7/WsYipUI5FyJ+2aUrHyua7MxNRx1SI0vky+ywqcb6O7+2BSOCkJsZS +5elSN43RfWq2+RABtSBv6kKtjxaC+FW6oCTCp6vJJvVEOA7I+rGUpP5mQHRJMtwW +2MfYBa0Byw+M+1ASE6orIjgyZPTn47kBDQRdsXnXAQgAz+nnewzYZcPEBVUQqObD +91g3sKO2PNrmW+Xql0V5li1D3s9JeufFHlkekcH6EtuvEskHrXBsqLRrCL+TlRf+ +uS+S6/+TPW8Bt0NX2UfEfV9DEKDzSnpxuzt6jDGYM6xLtDrgrJYFo/O/WGx69bnM +829Mg5DCBwLlMZ5x1gqEwFm6AKbsrqILLQSAnGP3N/FVI100aZSLe9dlTEvx/6uh +pRAUdJVNbwNr4GtQkc8PnwJkPNP8hU1yxngo613asQXSLcGOxs1TH5KPJIZq1oan +GTpyCgNYIDSSIjW6SvYEd8I6o3zbKhOGEuuRL3gJxoBWe2Y/vYO7dODXosGSk1tH +owARAQABiQE8BBgBCgAmFiEE7ni7ioKnhRR3qq0KOQwbWt6XiDUFAl2xedcCGwwF +CQPCZwAACgkQOQwbWt6XiDXhAAf/S7RThbngay8xsqtsURpVBDEpzb0a9S3HtzgM +tH0FRvqJdgGUimXoYKI6aF9cBu6d3UQx7i24pAYaMWHYXfsUg9GKXFA5UeKNfm5I +XvAljxB/dize3lAXm09DWeZR/x6eFfqixsVAAenxZz0+r3EM+H14k1HbmzZTHNtE +SPTTDAUii7UKZPuzOj0gjM9QSqY04KCyBoKzi4KXNjQM7yznpfVd0H/Gq2YZt8pd +a57suDGU8qrQdTI01RhizHCNnJWmIVLe4OcNOHQF4XnEWX1Meg24cbGjMocqLJkf +9tVXeGYnwosk65MO6kBnE98Bdsey7T1rH+PyN1ZLYP1Yiq/rbpkCDQRiVq4bARAA +tL77Y7J52fPxk0iyjeJfN826kZ+B+7CBBOPFoGQw19U5y6wimIiqg3+287YJ8k6X +AKzqSRRbyuCb6JCA2M4nOumiCBU65+m8ZfeGSLpCcmhzaInK3tFXkAdZnMzNq0V7 +QnDE4BXGXGbEmLEVorU8H9aaKuAOygDGR2ucEMzgqufsveeFiqdguMZy4bK7CBom +vjThVhMaEDwXERpMJ3d/i2LQkZXT3rGRM/uO4Pdqyb6ewM7brrAEJogWRsL8NXHR ++o/b49QwGBRqGLplBK8EUoJoSqA+q4Rkdd3wDxtOXxHRtyK8RJzrzijkxLkH4zdF +D14EgI9xs/txnVoacOS2iszkFLJYgj9fSFxu78Mp/x7Q9MpiypJSYoGPjSihinL9 +ahhcYGM3p/s0XP0LpTM+A4R8pKXNMPfdJqiR176vn3AGfYmJzSNzRenvkuZF4gFy +WdWbkBISh8XQmxNafccwN8nde9FyjIbfeiMbuHWlJL1q4YsTWTc4ZpLgWvJ1+au0 +VH/sq8Yvsf/rzu0osw/hPizqgOlNpPPZoACepQ+9jmWEcuwjr6KG6Xvg7xZPMDV/ +QnNprC7p+ouuDu5abqoDErsg2ZG749bb249TvAmdJyEug3DLKUVlCKEg3K+icSE6 +MYzX5jojT3WbxwyZAFu3orWSypfCsr7Hpy55dLu2wLUAEQEAAbQdQXJwYWQgQm9k +YSA8YWJvZGFAYXBhY2hlLm9yZz6JAlgEEwEIAEIWIQTWyKcUnUfveqCLNdEGVmjy +pY8JfwUCYlauGwIbAwUJA8JnAAULCQgHAgMiAgEGFQoJCAsCBBYCAwECHgcCF4AA +CgkQBlZo8qWPCX+PkA//feSFzxnb5VW3SrowSwWSwZ4GwhfNebxKiltlr2MpvNXi +EnSmtSX88Ir9wTxdgDPP0pyhxTQIMVn1KltX6a62h0VtNZlFsECDMFDLDapTti3J +BTfzkw2Xjw/mU5WMl/sQzn/dM+1LdVEnt/k15njpaXdxd8mxctUU3PH2p5JMbdQ+ +MpaLPeX7IPv7+cahCaBL40Q/OBCbx8SWrxjnKurmdxOvYOtRq4Y7qzV1YmAk5Nh8 +Pp0P9Us0ihJZ0/AyN0cWNp9wvuTuDubN/omkD2DknnPxh6O5g+PgjY4CHxgRCPzc +IKgBMzeuus8t7oavO1/Hpx7HRUNRZqBXd7b9WWRZ+BV3ZJLP3khaJKVR5wyLY/To +uA87Qx6DjYEabw8JAOXG+SshVTg+TpF0uBghF7dutZ1ERiA8ZA/y09CugByKbT2o +K+2RqNk8B8KMppKV0g82x/8dcVqdf7XZnRA2ckYQfUwNwGBghdEda728tfwfxzRd +XyZRYqiH885/9bAfqTKIib8yJBbCsSw+IjB3d3tCnlBxFFRPH3nK5Vv/FtAq1RbT +AFBN4PC0PxWizayAme0TB2f7iy5Ov/KvXbOUdEQqxEnhFTa5C6aTu8WBFznk23HD +aw+3KMQatOw5uSZ+xNBJfJsMbxntGOEH4kkq8DNdlNT+NoUI1lGXVewwC/22AEC5 +Ag0EYlauGwEQANMmmOFnp9peYWudW65WftqrWOtx5o4uzjJTzGe0ZUdqMlTxcxDV +I2gvqA7iAoVn4V+VKBVcEs2A9sirfYJK3XeN07SPzOzHzVhZN8ygqQ91NAQhAX+C +gkrE2BUq8/0GUGsXXK35P7aHI7ZYH0+a1k7U8Ah44qq4hp1FxWw7K6oO7/vOm9K+ +VhtjSRbPS+PT7T/15GQ0KEhaffOaP/vOC+NGVoUZPA671l+3vRef2y6OoQyGPdLi +Lu0EBSfGN94uEh4stiufZWDPY0Hxl2ZGiX2sAXDcnwzaG6bcnaz2wvKpMRfexMhJ +kQFl5KZPy77f49f4qyas3JUnp7qBstoyHw0GO+gimZridm4G3rXBpjtuR6UCp+op ++TNaUEU4FSvPN3OUHeEoN6xEl8amHTaa3NZlsW9VJFLN+U081oMUwOpmGy62zEAN +pUAkSzDn7qdHbqcZnWCtQnIwDsP4tVYCUzvVzAAqSdpfqnnrppWSbi1Vj4nK+hB9 +tiqgcGOBnuUUH24vIk6IsLuUU/zoaFzhvsux4HRrupTNpd6RIP+/CJjlF6IyE7c/ +XkmLWK0cuhOGYlWv1KEOhShLeQKVAo1/nJ0q4G/eIHjXVa559RBmZRsLjyBZ0Fp9 +9
[nifi-site] branch main updated: move szaszm to pmc
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/main by this push: new f2adfdd move szaszm to pmc new 4726411 Merge pull request #58 from szaszm/szaszm-pmc-member f2adfdd is described below commit f2adfdd3d2962ab73ed76ec0bedcb7a14f068f73 Author: Marton Szasz AuthorDate: Thu Mar 24 16:55:00 2022 + move szaszm to pmc --- src/pages/html/people.hbs | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/pages/html/people.hbs b/src/pages/html/people.hbs index dab5b66..f6d696d 100644 --- a/src/pages/html/people.hbs +++ b/src/pages/html/people.hbs @@ -194,6 +194,11 @@ title: Apache NiFi Team David Handermann + +szaszm +Márton Szász + + @@ -290,11 +295,6 @@ title: Apache NiFi Team -szaszm -Márton Szász - - - adebreceni Adam Debreceni
svn commit: r51359 - /dev/nifi/nifi-minifi-cpp/0.11.0/ /release/nifi/nifi-minifi-cpp/0.11.0/
Author: aboda Date: Mon Dec 13 10:06:29 2021 New Revision: 51359 Log: MINIFICPP-1689 Added: release/nifi/nifi-minifi-cpp/0.11.0/ - copied from r51358, dev/nifi/nifi-minifi-cpp/0.11.0/ Removed: dev/nifi/nifi-minifi-cpp/0.11.0/
[nifi-minifi-cpp] branch main updated: MINIFICPP-1666 Add Azure extension to docker minimal image
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new c0d3bb1 MINIFICPP-1666 Add Azure extension to docker minimal image c0d3bb1 is described below commit c0d3bb104f4048e7fb9499caaae6c82a50cd8866 Author: Gabor Gyimesi AuthorDate: Mon Oct 11 18:30:11 2021 +0200 MINIFICPP-1666 Add Azure extension to docker minimal image Signed-off-by: Arpad Boda This closes #1199 --- cmake/DockerConfig.cmake | 1 + 1 file changed, 1 insertion(+) diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake index 5d3c086..6520200 100644 --- a/cmake/DockerConfig.cmake +++ b/cmake/DockerConfig.cmake @@ -71,6 +71,7 @@ add_custom_target( -c ENABLE_PYTHON=OFF -c ENABLE_LIBRDKAFKA=ON -c ENABLE_AWS=ON +-c ENABLE_AZURE=ON -c DISABLE_CONTROLLER=ON -c ENABLE_SCRIPTING=OFF -c DISABLE_PYTHON_SCRIPTING=ON
[nifi-minifi-cpp] branch main updated: MINIFICPP-1633 Add ccache to docker CI builds
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 48acd26 MINIFICPP-1633 Add ccache to docker CI builds 48acd26 is described below commit 48acd26e3c07ea1bb05f91fecad601ec67bbae59 Author: Gabor Gyimesi AuthorDate: Mon Jul 26 10:17:55 2021 +0200 MINIFICPP-1633 Add ccache to docker CI builds Signed-off-by: Arpad Boda This closes #1164 --- .github/workflows/ci.yml | 15 --- .gitignore | 1 + CMakeLists.txt | 1 + cmake/DockerConfig.cmake | 1 + docker/DockerBuild.sh| 29 +++-- docker/Dockerfile| 11 +++ 6 files changed, 45 insertions(+), 13 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 29d024b..79dae16 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -258,15 +258,24 @@ jobs: docker_integration_tests: name: "Docker integration tests" runs-on: ubuntu-20.04 -timeout-minutes: 90 +timeout-minutes: 120 steps: - id: checkout uses: actions/checkout@v2 + - id: cache +uses: actions/cache@v2 +with: + path: ~/.ccache + key: docker-ccache-${{github.ref}}-${{github.sha}} + restore-keys: | +docker-ccache-${{github.ref}}- +docker-ccache-refs/heads/main - id: build run: | - ./bootstrap.sh -e -t + if [ -d ~/.ccache ]; then mv ~/.ccache .; fi + mkdir build cd build - cmake -DUSE_SHARED_LIBS= -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_JNI=OFF -DDISABLE_JEMALLOC=ON -DDISABLE_SCRIPTING=ON -DENABLE_AWS=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_AZURE=ON -DENABLE_SQL=ON .. + cmake -DUSE_SHARED_LIBS= -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_JNI=OFF -DDISABLE_JEMALLOC=ON -DDISABLE_SCRIPTING=ON -DENABLE_AWS=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_AZURE=ON -DENABLE_SQL=ON -DDOCKER_BUILD_ONLY=ON -DDOCKER_CCACHE_DUMP_LOCATION=$HOME/.ccache .. make docker - id: install_deps run: | diff --git a/.gitignore b/.gitignore index bcc7486..9dd9de9 100644 --- a/.gitignore +++ b/.gitignore @@ -77,6 +77,7 @@ __pycache__/ .vs/slnx.sqlite /.ccls-cache /.vscode +.ccache # generated files from WEL test extensions/windows-event-log/tests/custom-provider/unit-test-provider.cs diff --git a/CMakeLists.txt b/CMakeLists.txt index fd1de67..355bb6d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -28,6 +28,7 @@ set(BUILD_NUMBER "" CACHE STRING "Build number") # Base Alpine image to be used in the Docker build instead of the default Alpine image set(DOCKER_BASE_IMAGE "" CACHE STRING "Docker build Alpine base image") +set(DOCKER_CCACHE_DUMP_LOCATION "" CACHE STRING "Directory to dump ccache to after docker build for later reuse") include(CMakeDependentOption) include(CheckIncludeFile) diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake index c6059eb..d841c49 100644 --- a/cmake/DockerConfig.cmake +++ b/cmake/DockerConfig.cmake @@ -56,6 +56,7 @@ add_custom_target( -c DISABLE_PYTHON_SCRIPTING=${DISABLE_PYTHON_SCRIPTING} -c DISABLE_CONTROLLER=${DISABLE_CONTROLLER} -c DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE} +-c DOCKER_CCACHE_DUMP_LOCATION=${DOCKER_CCACHE_DUMP_LOCATION} -c BUILD_NUMBER=${BUILD_NUMBER} WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/) diff --git a/docker/DockerBuild.sh b/docker/DockerBuild.sh index 397fa8c..d6e7734 100755 --- a/docker/DockerBuild.sh +++ b/docker/DockerBuild.sh @@ -28,6 +28,7 @@ IMAGE_TAG= DUMP_LOCATION= DISTRO_NAME= BUILD_NUMBER= +DOCKER_CCACHE_DUMP_LOCATION= function usage { echo "Usage: ./DockerBuild.sh -v [additional options]" @@ -58,10 +59,10 @@ while [[ $# -gt 0 ]]; do shift ;; -v|--minifi-version) -MINIFI_VERSION="$2" -shift -shift -;; + MINIFI_VERSION="$2" + shift + shift + ;; -t|--tag) IMAGE_TAG="$2" shift @@ -84,6 +85,8 @@ while [[ $# -gt 0 ]]; do BUILD_NUMBER="${ARR[1]}" elif [ "${ARR[0]}" == "DOCKER_BASE_IMAGE" ]; then BUILD_ARGS="${BUILD_ARGS} --build-arg BASE_ALPINE_IMAGE=${ARR[1]}" +elif [ "${ARR[0]}" == "DOCKER_CCACHE_DUMP_LOCATION" ]; then + DOCKER_CCACHE_DUMP_LOCATION="${ARR[1]}" else BUILD_ARGS="${BUILD_ARGS} --build-arg ${ARR[0]}=${ARR[1]}" fi @@ -133,14 +136,28 @@ if [ -n "${BUILD_NUMBER}" ]; then TARGZ_TAG="${TARGZ_TAG}-${BUILD_NUMBER}" fi -DOCKER_COMMAND="docker build " +DOCKER_BUILD_
[nifi-minifi-cpp] branch main updated (b37d398 -> 1fb7540)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from b37d398 MINIFICPP-1607 Implement AttributesToJson processor add 1fb7540 MINIFICPP-1508 Remove the legacy Regex wrapper and use std::regex instead No new revisions were added by this update. Summary of changes: .../integration/minifi/core/DockerTestCluster.py | 6 +- encrypt-config/tests/ConfigFileEncryptorTests.cpp | 5 +- extensions/aws/s3/S3Wrapper.cpp| 11 +- extensions/http-curl/client/HTTPClient.cpp | 8 +- extensions/librdkafka/PublishKafka.cpp | 8 +- extensions/librdkafka/PublishKafka.h | 4 +- extensions/librdkafka/tests/ConsumeKafkaTests.cpp | 1 - extensions/sftp/processors/ListSFTP.cpp| 14 +- extensions/sftp/processors/ListSFTP.h | 6 +- .../standard-processors/processors/ExtractText.cpp | 16 +- .../standard-processors/processors/GetFile.cpp | 6 +- .../standard-processors/processors/TailFile.cpp| 14 +- .../tests/unit/RetryFlowFileTests.cpp | 6 +- libminifi/include/utils/HTTPUtils.h| 22 ++- libminifi/include/utils/RegexUtils.h | 85 -- libminifi/src/utils/RegexUtils.cpp | 175 - libminifi/test/unit/RegexUtilsTests.cpp| 83 -- 17 files changed, 63 insertions(+), 407 deletions(-) delete mode 100644 libminifi/include/utils/RegexUtils.h delete mode 100644 libminifi/src/utils/RegexUtils.cpp delete mode 100644 libminifi/test/unit/RegexUtilsTests.cpp
[nifi-minifi-cpp] branch main updated: MINIFICPP-1605 Always refresh AWS credentials through default credentials chain
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 2d2f300 MINIFICPP-1605 Always refresh AWS credentials through default credentials chain 2d2f300 is described below commit 2d2f3004ae42af6aa375d90ecba7a64a266e91ad Author: Gabor Gyimesi AuthorDate: Fri Jun 25 11:30:34 2021 +0200 MINIFICPP-1605 Always refresh AWS credentials through default credentials chain Signed-off-by: Arpad Boda This closes #1130 --- extensions/aws/AWSCredentialsProvider.cpp | 4 +++ extensions/aws/AWSCredentialsProvider.h| 1 + .../controllerservices/AWSCredentialsService.cpp | 2 +- .../test/aws-tests/AWSCredentialsServiceTest.cpp | 37 ++ 4 files changed, 43 insertions(+), 1 deletion(-) diff --git a/extensions/aws/AWSCredentialsProvider.cpp b/extensions/aws/AWSCredentialsProvider.cpp index 9f9f171..7bc4dbd 100644 --- a/extensions/aws/AWSCredentialsProvider.cpp +++ b/extensions/aws/AWSCredentialsProvider.cpp @@ -44,6 +44,10 @@ void AWSCredentialsProvider::setUseDefaultCredentials(bool use_default_credentia use_default_credentials_ = use_default_credentials; } +bool AWSCredentialsProvider::getUseDefaultCredentials() const { + return use_default_credentials_; +} + void AWSCredentialsProvider::setAccessKey(const std::string &access_key) { access_key_ = access_key; } diff --git a/extensions/aws/AWSCredentialsProvider.h b/extensions/aws/AWSCredentialsProvider.h index a011887..aef2c80 100644 --- a/extensions/aws/AWSCredentialsProvider.h +++ b/extensions/aws/AWSCredentialsProvider.h @@ -46,6 +46,7 @@ class AWSCredentialsProvider { void setAccessKey(const std::string &access_key); void setSecretKey(const std::string &secret_key); void setCredentialsFile(const std::string &credentials_file); + bool getUseDefaultCredentials() const; minifi::utils::optional getAWSCredentials(); private: diff --git a/extensions/aws/controllerservices/AWSCredentialsService.cpp b/extensions/aws/controllerservices/AWSCredentialsService.cpp index 4525420..b1960fa 100644 --- a/extensions/aws/controllerservices/AWSCredentialsService.cpp +++ b/extensions/aws/controllerservices/AWSCredentialsService.cpp @@ -77,7 +77,7 @@ void AWSCredentialsService::onEnable() { } minifi::utils::optional AWSCredentialsService::getAWSCredentials() { - if (!aws_credentials_ || aws_credentials_->IsExpiredOrEmpty()) { + if (aws_credentials_provider_.getUseDefaultCredentials() || !aws_credentials_ || aws_credentials_->IsExpiredOrEmpty()) { aws_credentials_ = aws_credentials_provider_.getAWSCredentials(); } return aws_credentials_; diff --git a/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp b/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp index e6d8aba..3f6167e 100644 --- a/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp +++ b/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp @@ -45,6 +45,18 @@ class AWSCredentialsServiceTestAccessor { std::shared_ptr aws_credentials_service; }; +namespace { + +void setEnvironmentCredentials(const std::string& key, const std::string& secret_key) { + #ifdef WIN32 + _putenv_s("AWS_ACCESS_KEY_ID", key.c_str()); + _putenv_s("AWS_SECRET_ACCESS_KEY", secret_key.c_str()); + #else + setenv("AWS_ACCESS_KEY_ID", key.c_str(), 1); + setenv("AWS_SECRET_ACCESS_KEY", secret_key.c_str(), 1); + #endif +} + TEST_CASE_METHOD(AWSCredentialsServiceTestAccessor, "Test expired credentials are refreshed", "[credentialRefresh]") { plan->setProperty(aws_credentials_service, "Access Key", "key"); plan->setProperty(aws_credentials_service, "Secret Key", "secret"); @@ -65,3 +77,28 @@ TEST_CASE_METHOD(AWSCredentialsServiceTestAccessor, "Test expired credentials ar // Check for credential refresh REQUIRE_FALSE(aws_credentials_impl->getAWSCredentials()->IsExpired()); } + +TEST_CASE_METHOD(AWSCredentialsServiceTestAccessor, "Test credentials from default credential chain are always refreshed", "[credentialRefresh]") { + setEnvironmentCredentials("key", "secret"); + plan->setProperty(aws_credentials_service, "Use Default Credentials", "true"); + aws_credentials_service->enable(); + assert(aws_credentials_service->getControllerServiceImplementation() != nullptr); + auto aws_credentials_impl = std::static_pointer_cast(aws_credentials_service->getControllerServiceImplementation()); + + // Check intial credentials + REQUIRE(aws_credentials_impl->getAWSCredentials()); + REQUIRE(aws_credentials_impl->getAWSCredentials()->GetAWSAccessKeyId() == "key"); + REQUIRE(aw
[nifi-minifi-cpp] branch main updated: MINIFICPP-1561 - Allow rocksdb encryption
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 5af180f MINIFICPP-1561 - Allow rocksdb encryption 5af180f is described below commit 5af180fd964b6564045b146df8473602a0ec789c Author: Adam Debreceni AuthorDate: Thu Jul 8 10:30:08 2021 +0200 MINIFICPP-1561 - Allow rocksdb encryption Signed-off-by: Arpad Boda This closes #1090 --- CONFIGURE.md | 21 encrypt-config/ConfigFile.h| 2 +- encrypt-config/ConfigFileEncryptor.h | 2 +- encrypt-config/Utils.h | 2 +- extensions/http-curl/tests/C2ConfigEncryption.cpp | 2 +- extensions/rocksdb-repos/CMakeLists.txt| 2 +- .../rocksdb-repos/DatabaseContentRepository.cpp| 16 ++- .../rocksdb-repos/DatabaseContentRepository.h | 2 + extensions/rocksdb-repos/FlowFileRepository.cpp| 41 +-- extensions/rocksdb-repos/FlowFileRepository.h | 16 ++- .../RocksDbPersistableKeyValueStoreService.cpp | 13 +- .../RocksDbPersistableKeyValueStoreService.h | 2 + extensions/rocksdb-repos/database/OpenRocksDb.cpp | 9 ++ extensions/rocksdb-repos/database/OpenRocksDb.h| 2 + .../rocksdb-repos/database/RocksDatabase.cpp | 2 +- extensions/rocksdb-repos/database/RocksDatabase.h | 2 +- .../rocksdb-repos/database/RocksDbInstance.cpp | 18 ++- .../rocksdb-repos/database/RocksDbInstance.h | 4 + extensions/rocksdb-repos/database/RocksDbUtils.h | 11 +- extensions/rocksdb-repos/database/StringAppender.h | 11 +- .../encryption/RocksDbEncryptionProvider.cpp | 127 .../RocksDbEncryptionProvider.h} | 28 ++--- libminifi/CMakeLists.txt | 2 +- libminifi/include/properties/Decryptor.h | 4 +- libminifi/include/properties/PropertiesFile.h | 2 +- .../EncryptionManager.h} | 29 +++-- .../utils/{ => crypto}/EncryptionProvider.h| 15 ++- .../include/utils/{ => crypto}/EncryptionUtils.h | 0 libminifi/include/utils/crypto/ciphers/Aes256Ecb.h | 78 .../ciphers/XSalsa20.h}| 13 +- libminifi/include/utils/file/FileSystem.h | 2 +- libminifi/src/utils/EncryptionProvider.cpp | 58 - libminifi/src/utils/crypto/EncryptionManager.cpp | 88 ++ .../utils/crypto/EncryptionProvider.cpp} | 31 ++--- .../src/utils/{ => crypto}/EncryptionUtils.cpp | 2 +- libminifi/src/utils/crypto/ciphers/Aes256Ecb.cpp | 133 + libminifi/src/utils/file/FileSystem.cpp| 2 +- libminifi/test/TestBase.h | 7 ++ libminifi/test/rocksdb-tests/EncryptionTests.cpp | 108 + .../test/rocksdb-tests/RocksDBStreamTests.cpp | 2 +- libminifi/test/rocksdb-tests/RocksDBTests.cpp | 49 +++- libminifi/test/unit/EncryptionUtilsTests.cpp | 2 +- 42 files changed, 788 insertions(+), 174 deletions(-) diff --git a/CONFIGURE.md b/CONFIGURE.md index c93352d..13c859d 100644 --- a/CONFIGURE.md +++ b/CONFIGURE.md @@ -166,6 +166,8 @@ folder. You may specify your own path in place of these defaults. nifi.flowfile.repository.directory.default=${MINIFI_HOME}/flowfile_repository nifi.database.content.repository.directory.default=${MINIFI_HOME}/content_repository + Shared database + It is also possible to use a single database to store multiple repositories with the `minifidb://` scheme. This could help with migration and centralize agent state persistence. In the scheme the final path segment designates the column family in the repository, while the preceding path indicates the directory the rocksdb database is @@ -189,6 +191,25 @@ Moreover the `"default"` name is restricted and should not be used. nifi.state.manangement.provider.local.path=minifidb://${MINIFI_HOME}/agent_state/default ^ error: "default" is restricted +### Configuring Repository encryption + +It is possible to provide rocksdb-backed repositories a key to request their +encryption. + +in conf/bootstrap.conf + nifi.flowfile.repository.encryption.key=805D7B95EF44DC27C87FFBC4DFDE376DAE604D55DB2C5496DEEF5236362DE62E +nifi.database.content.repository.encryption.key= +# nifi.state.management.provider.local.encryption.key= + +In the above configuration the first line will cause `FlowFileRepository` to use the specified `256` bit key. +The second line will trigger the generation of a random (`256` bit) key persisted back into `conf/bootstrap.conf`, which `DatabaseContentRepository` will then use for encryption. +(This way one can request e
[nifi-minifi-cpp] branch main updated (1ce101f -> 9173029)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 1ce101f MINIFICPP-1548 - Use the same rocksdb instance for multiple repositories add 9173029 MINIFICPP-1426 MSVC: Use /std:c++latest /permissive- and drop VS 2017 No new revisions were added by this update. Summary of changes: .github/workflows/ci.yml | 45 + .gitignore | 1 + CMakeLists.txt | 10 +--- README.md | 2 +- Windows.md | 11 ++-- cmake/BundledAzureSdkCpp.cmake | 8 ++- cmake/CppVersion.cmake | 42 +++ controller/CMakeLists.txt | 16 +- extensions/aws/s3/S3Wrapper.cpp| 4 +- extensions/script/CMakeLists.txt | 19 +-- extensions/sql/CMakeLists.txt | 5 +- extensions/windows-event-log/wel/JSONUtils.cpp | 2 +- extensions/windows-event-log/wel/JSONUtils.h | 2 +- libminifi/CMakeLists.txt | 59 ++ libminifi/include/utils/GeneralUtils.h | 2 +- main/CMakeLists.txt| 22 +--- main/MiNiFiWindowsService.cpp | 4 +- nanofi/CMakeLists.txt | 23 + nanofi/examples/CMakeLists.txt | 58 +++-- python/library/CMakeLists.txt | 2 +- .../fix-illegal-qualified-name-in-member.patch | 14 + win_build_vs.bat | 3 +- 22 files changed, 124 insertions(+), 230 deletions(-) create mode 100644 cmake/CppVersion.cmake create mode 100644 thirdparty/azure-sdk-cpp-for-cpp/fix-illegal-qualified-name-in-member.patch
[nifi-minifi-cpp] branch main updated (f2a0a42 -> 1ce101f)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from f2a0a42 MINIFICPP-1586-`make linter` gives errors when it is run in an unpacked release tarball add 1ce101f MINIFICPP-1548 - Use the same rocksdb instance for multiple repositories No new revisions were added by this update. Summary of changes: CONFIGURE.md | 23 +++ conf/minifi.properties | 1 + extensions/rocksdb-repos/CMakeLists.txt| 2 +- .../rocksdb-repos/DatabaseContentRepository.cpp| 24 ++- .../rocksdb-repos/DatabaseContentRepository.h | 37 +--- extensions/rocksdb-repos/FlowFileRepository.cpp| 19 +- extensions/rocksdb-repos/FlowFileRepository.h | 25 +-- extensions/rocksdb-repos/RocksDatabase.cpp | 149 --- extensions/rocksdb-repos/RocksDbStream.cpp | 2 +- extensions/rocksdb-repos/RocksDbStream.h | 6 +- .../RocksDbPersistableKeyValueStoreService.cpp | 21 +-- .../RocksDbPersistableKeyValueStoreService.h | 2 +- .../rocksdb-repos/database/ColumnHandle.cpp| 16 +- .../rocksdb-repos/database/ColumnHandle.h | 22 +-- extensions/rocksdb-repos/database/OpenRocksDb.cpp | 114 .../{RocksDatabase.h => database/OpenRocksDb.h}| 77 +++- .../rocksdb-repos/database/RocksDatabase.cpp | 91 + .../rocksdb-repos/database/RocksDatabase.h | 52 +++--- .../rocksdb-repos/database/RocksDbInstance.cpp | 169 + .../rocksdb-repos/database/RocksDbInstance.h | 71 +++ extensions/rocksdb-repos/database/RocksDbUtils.h | 80 .../rocksdb-repos/database/StringAppender.cpp | 59 ++ .../rocksdb-repos/database/StringAppender.h| 42 +++-- .../rocksdb-repos/database/WriteBatch.cpp | 25 +-- .../rocksdb-repos/database/WriteBatch.h| 32 ++-- libminifi/include/core/ProcessContext.h| 9 +- libminifi/include/properties/Configuration.h | 1 + libminifi/include/utils/GeneralUtils.h | 9 + libminifi/test/TestBase.cpp| 5 +- libminifi/test/integration/IntegrationBase.h | 5 +- .../test/rocksdb-tests/RocksDBStreamTests.cpp | 18 +- libminifi/test/rocksdb-tests/RocksDBTests.cpp | 206 + 32 files changed, 1015 insertions(+), 399 deletions(-) delete mode 100644 extensions/rocksdb-repos/RocksDatabase.cpp copy libminifi/include/utils/tls/TLSUtils.h => extensions/rocksdb-repos/database/ColumnHandle.cpp (76%) copy libminifi/include/utils/requirements/EqualityComparable.h => extensions/rocksdb-repos/database/ColumnHandle.h (71%) create mode 100644 extensions/rocksdb-repos/database/OpenRocksDb.cpp rename extensions/rocksdb-repos/{RocksDatabase.h => database/OpenRocksDb.h} (57%) create mode 100644 extensions/rocksdb-repos/database/RocksDatabase.cpp copy libminifi/include/core/logging/WindowsEventLogSink.h => extensions/rocksdb-repos/database/RocksDatabase.h (52%) create mode 100644 extensions/rocksdb-repos/database/RocksDbInstance.cpp create mode 100644 extensions/rocksdb-repos/database/RocksDbInstance.h create mode 100644 extensions/rocksdb-repos/database/RocksDbUtils.h create mode 100644 extensions/rocksdb-repos/database/StringAppender.cpp copy libminifi/include/core/TraceableResource.h => extensions/rocksdb-repos/database/StringAppender.h (56%) copy libminifi/include/utils/requirements/EqualityComparable.h => extensions/rocksdb-repos/database/WriteBatch.cpp (68%) copy libminifi/include/utils/tls/WindowsCertStoreLocation.h => extensions/rocksdb-repos/database/WriteBatch.h (67%) create mode 100644 libminifi/test/rocksdb-tests/RocksDBTests.cpp
[nifi-minifi-cpp] branch main updated: MINIFICPP-1586-`make linter` gives errors when it is run in an unpacked release tarball
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new f2a0a42 MINIFICPP-1586-`make linter` gives errors when it is run in an unpacked release tarball f2a0a42 is described below commit f2a0a42c55d5befcf7cf55b692987dbffa9d161b Author: aminadinari19 AuthorDate: Tue Jun 15 11:59:03 2021 +0200 MINIFICPP-1586-`make linter` gives errors when it is run in an unpacked release tarball MINIFICPP-1586-keeping script_dir variable MINIFICPP-1586-Removed '/' from sub-directories MINIFICPP-1586-`make linter` gives errors when it is run in an unpacked released tarball MINIFICPP-1586-new line at the end Signed-off-by: Arpad Boda --- thirdparty/google-styleguide/run_linter.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/thirdparty/google-styleguide/run_linter.py b/thirdparty/google-styleguide/run_linter.py index 1cf03a0..7f7f6fd 100644 --- a/thirdparty/google-styleguide/run_linter.py +++ b/thirdparty/google-styleguide/run_linter.py @@ -2,9 +2,7 @@ import argparse import os import cpplint -script_dir = os.path.dirname(os.path.realpath(__file__)) parser = argparse.ArgumentParser() - parser.add_argument('-i', '--includePaths', nargs="+", help='Run linter check in these directories') parser.add_argument('-q', '--quiet', action='store_true', help='Don\'t print anything if no errors are found.') args = parser.parse_args() @@ -16,7 +14,8 @@ for include_path in args.includePaths: if (".h" in file_name) or (".cpp" in file_name): list_of_files += [os.path.join(dir_path, file_name)] -repository_path = script_dir + "../../" +script_dir = os.path.dirname(os.path.realpath(__file__)) +repository_path = os.path.abspath(os.path.join(script_dir, os.pardir, os.pardir)) arg_list = list() arg_list.append("--linelength=200")
[nifi-site] branch main updated: Add Ferenc Gerlits to the list of committers
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/main by this push: new e652912 Add Ferenc Gerlits to the list of committers new 658f90a Merge pull request #46 from fgerlits/add-fgerlits-to-the-list-of-committers e652912 is described below commit e652912591e2f9bafe9befc2c631096df2d74a73 Author: Ferenc Gerlits AuthorDate: Fri Jun 11 12:48:47 2021 +0200 Add Ferenc Gerlits to the list of committers --- src/pages/html/people.hbs | 5 + 1 file changed, 5 insertions(+) diff --git a/src/pages/html/people.hbs b/src/pages/html/people.hbs index 30ec17e..8bf093c 100644 --- a/src/pages/html/people.hbs +++ b/src/pages/html/people.hbs @@ -289,6 +289,11 @@ title: Apache NiFi Team Adam Debreceni + +fgerlits +Ferenc Gerlits + +
svn commit: r48291 - /dev/nifi/nifi-minifi-cpp/0.10.0/ /release/nifi/nifi-minifi-cpp/0.10.0/
Author: aboda Date: Fri Jun 11 09:17:11 2021 New Revision: 48291 Log: MINIFICPP-1574 Promote NiFi MiNiFi C++ 0.10.0 vote artifacts to release Added: release/nifi/nifi-minifi-cpp/0.10.0/ - copied from r48290, dev/nifi/nifi-minifi-cpp/0.10.0/ Removed: dev/nifi/nifi-minifi-cpp/0.10.0/
[nifi-minifi-cpp] branch main updated: MINIFICPP-1585 Rename enum values in ConsumeKafkaTests::PublishEvent
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 4d12658 MINIFICPP-1585 Rename enum values in ConsumeKafkaTests::PublishEvent 4d12658 is described below commit 4d12658952f553f1a8285ec5224698a72e708451 Author: Ferenc Gerlits AuthorDate: Mon Jun 7 10:03:05 2021 +0200 MINIFICPP-1585 Rename enum values in ConsumeKafkaTests::PublishEvent TRANSACTION_COMMIT is defined in winnt.h as a macro, so the build fails on Windows. I have renamed it, and a few other enum values for consistency. Also added /K to the VS2019 Windows CI job, to make sure bugs like this get caught in the CI stage in the future. Signed-off-by: Arpad Boda This closes #1102 --- .github/workflows/ci.yml | 2 +- extensions/librdkafka/tests/ConsumeKafkaTests.cpp | 30 +++ 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7cafa6f..c8399dc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -154,7 +154,7 @@ jobs: run: | PATH %PATH%;C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64 PATH %PATH%;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn - win_build_vs.bat build /2019 /64 /CI /S /A /PDH /L /R + win_build_vs.bat build /2019 /64 /CI /S /A /PDH /K /L /R shell: cmd - name: test run: cd build && ctest --timeout 300 --parallel 8 -C Release --output-on-failure diff --git a/extensions/librdkafka/tests/ConsumeKafkaTests.cpp b/extensions/librdkafka/tests/ConsumeKafkaTests.cpp index c1d50f4..b011ea1 100644 --- a/extensions/librdkafka/tests/ConsumeKafkaTests.cpp +++ b/extensions/librdkafka/tests/ConsumeKafkaTests.cpp @@ -43,9 +43,9 @@ class KafkaTestProducer { public: enum class PublishEvent { PUBLISH, -TRANSACTION_START, -TRANSACTION_COMMIT, -CANCEL +BEGIN_TRANSACTION, +END_TRANSACTION, +CANCEL_TRANSACTION }; KafkaTestProducer(const std::string& kafka_brokers, const std::string& topic, const bool transactional) : logger_(logging::LoggerFactory::getLogger()) { @@ -88,15 +88,15 @@ class KafkaTestProducer { publish_message(*next_message, message_key, message_headers); std::advance(next_message, 1); break; -case PublishEvent::TRANSACTION_START: +case PublishEvent::BEGIN_TRANSACTION: logger_->log_debug("Starting new transaction..."); rd_kafka_begin_transaction(producer_.get()); break; -case PublishEvent::TRANSACTION_COMMIT: +case PublishEvent::END_TRANSACTION: logger_->log_debug("Committing transaction..."); rd_kafka_commit_transaction(producer_.get(), TRANSACTIONS_TIMEOUT_MS.count()); break; -case PublishEvent::CANCEL: +case PublishEvent::CANCEL_TRANSACTION: logger_->log_debug("Cancelling transaction..."); rd_kafka_abort_transaction(producer_.get(), TRANSACTIONS_TIMEOUT_MS.count()); } @@ -147,15 +147,15 @@ class ConsumeKafkaTest { using ExtractText = org::apache::nifi::minifi::processors::ExtractText; const KafkaTestProducer::PublishEvent PUBLISH= KafkaTestProducer::PublishEvent::PUBLISH; - const KafkaTestProducer::PublishEvent TRANSACTION_START = KafkaTestProducer::PublishEvent::TRANSACTION_START; - const KafkaTestProducer::PublishEvent TRANSACTION_COMMIT = KafkaTestProducer::PublishEvent::TRANSACTION_COMMIT; - const KafkaTestProducer::PublishEvent CANCEL = KafkaTestProducer::PublishEvent::CANCEL; + const KafkaTestProducer::PublishEvent BEGIN_TRANSACTION = KafkaTestProducer::PublishEvent::BEGIN_TRANSACTION; + const KafkaTestProducer::PublishEvent END_TRANSACTION= KafkaTestProducer::PublishEvent::END_TRANSACTION; + const KafkaTestProducer::PublishEvent CANCEL_TRANSACTION = KafkaTestProducer::PublishEvent::CANCEL_TRANSACTION; const std::vector NON_TRANSACTIONAL_MESSAGES { PUBLISH, PUBLISH }; - const std::vector SINGLE_COMMITTED_TRANSACTION { TRANSACTION_START, PUBLISH, PUBLISH, TRANSACTION_COMMIT }; - const std::vector TWO_SEPARATE_TRANSACTIONS { TRANSACTION_START, PUBLISH, TRANSACTION_COMMIT, TRANSACTION_START, PUBLISH, TRANSACTION_COMMIT }; - const std::vector NON_COMMITTED_TRANSACTION { TRANSACTION_START, PUBLISH, PUBLISH }; - const std::vector CANCELLED_TRANSACTION { TRANSACTION_START, PUBLISH, CANCEL }; + const std::vector SINGLE_COMMITTED_TRANSACTION { BEGIN_TRANSACTION, PUBLISH, PUBLISH, END_TRANSACTION }; + const std::vector TWO_SEPARATE_TRANSACTIONS { BEGIN_TRANSACTIO
[nifi-minifi-cpp] branch main updated: MINIFICPP-1581: set cmake policy CMP0096 to NEW to preserve the leading zeros in project version
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 6020496 MINIFICPP-1581: set cmake policy CMP0096 to NEW to preserve the leading zeros in project version 6020496 is described below commit 6020496ad1cd2fc236815ee78b3d9f2cee640a0c Author: Martin Zink AuthorDate: Fri Jun 4 10:24:47 2021 +0200 MINIFICPP-1581: set cmake policy CMP0096 to NEW to preserve the leading zeros in project version MINIFICPP-1581: set cmake policy for nanofi and libminifi use the buster-backports repo to install cmake 3.16.3 in debian ci job updated README.md and all cmake requirements to 3.16 use java 11 on debian buster Signed-off-by: Arpad Boda This closes #1098 --- CMakeLists.txt | 3 ++- README.md | 2 +- controller/CMakeLists.txt | 2 +- debian.sh | 4 ++-- docker/debian/Dockerfile | 6 -- extensions/ExtensionHeader.txt | 2 +- libminifi/CMakeLists.txt | 4 ++-- main/CMakeLists.txt| 2 +- nanofi/CMakeLists.txt | 3 ++- nanofi/ecu/CMakeLists.txt | 2 +- nanofi/examples/CMakeLists.txt | 2 +- 11 files changed, 18 insertions(+), 14 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 8f4d496..1ac5db7 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -17,7 +17,8 @@ # under the License. # -cmake_minimum_required(VERSION 3.11) +cmake_minimum_required(VERSION 3.16) +cmake_policy(SET CMP0096 NEW) # policy to preserve the leading zeros in PROJECT_VERSION_{MAJOR,MINOR,PATCH,TWEAK} cmake_policy(SET CMP0065 OLD) # default export policy, required for self-dlopen project(nifi-minifi-cpp VERSION 0.10.0) set(PROJECT_NAME "nifi-minifi-cpp") diff --git a/README.md b/README.md index 4e8a5f6..ecf16c4 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,7 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension ### To build Utilities -* CMake 3.11 or greater +* CMake 3.16 or greater * gcc 4.8.4 or greater * g++ 4.8.4 or greater * bison 3.0.x (3.2 has been shown to fail builds) diff --git a/controller/CMakeLists.txt b/controller/CMakeLists.txt index 6610973..5533ba7 100644 --- a/controller/CMakeLists.txt +++ b/controller/CMakeLists.txt @@ -17,7 +17,7 @@ # under the License. # -cmake_minimum_required(VERSION 3.11) +cmake_minimum_required(VERSION 3.16) include_directories(../main/ ../libminifi/include ../libminifi/include/c2 ../libminifi/include/c2/protocols/ ../libminifi/include/core/state ./libminifi/include/core/statemanagement/metrics ../libminifi/include/core/yaml ../libminifi/include/core) diff --git a/debian.sh b/debian.sh index 0df18e7..52836e9 100644 --- a/debian.sh +++ b/debian.sh @@ -67,8 +67,8 @@ build_deps(){ elif [ "$FOUND_VALUE" = "lua" ]; then INSTALLED+=("liblua5.1-0-dev") elif [ "$FOUND_VALUE" = "jnibuild" ]; then -INSTALLED+=("openjdk-8-jdk") -INSTALLED+=("openjdk-8-source") +INSTALLED+=("openjdk-11-jdk") +INSTALLED+=("openjdk-11-source") INSTALLED+=("maven") elif [ "$FOUND_VALUE" = "automake" ]; then INSTALLED+=("automake") diff --git a/docker/debian/Dockerfile b/docker/debian/Dockerfile index 1691606..e353b8b 100644 --- a/docker/debian/Dockerfile +++ b/docker/debian/Dockerfile @@ -26,9 +26,11 @@ ARG MINIFI_VERSION ENV MINIFI_BASE_DIR /opt/minifi ENV MINIFI_HOME $MINIFI_BASE_DIR/nifi-minifi-cpp-$MINIFI_VERSION -ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/ +ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/ -RUN apt-get update && apt-get install -y openjdk-11-jdk openjdk-11-source libpython3.7-dev sudo git maven libarchive13 cmake +RUN echo "deb http://deb.debian.org/debian buster-backports main" | tee -a /etc/apt/sources.list \ +&& apt-get update && apt-get install -y openjdk-11-jdk openjdk-11-source libpython3.7-dev sudo git maven libarchive13 libtool \ +&& apt-get -t buster-backports install -y cmake RUN mkdir -p $MINIFI_BASE_DIR COPY . ${MINIFI_BASE_DIR} diff --git a/extensions/ExtensionHeader.txt b/extensions/ExtensionHeader.txt index d7ac551..db240d1 100644 --- a/extensions/ExtensionHeader.txt +++ b/extensions/ExtensionHeader.txt @@ -17,7 +17,7 @@ # under the License. # -cmake_minimum_required(VERSION 3.11) +cmake_minimum_required(VERSION 3.16) include_directories(../../libminifi/include ../../libminifi/include/core) diff --git a/libminifi/CMakeLists.txt b/libminifi/CMakeLists.txt index 26bd3b3..7a4b751 100644 --- a/libminifi/CMakeLists.txt +++
[nifi-minifi-cpp] 01/02: MINIFICPP-1582 fixing build falure in centos ci job
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit e51192ed9639c04d69252fa033b444373173a00b Author: Martin Zink AuthorDate: Fri Jun 4 15:56:04 2021 +0200 MINIFICPP-1582 fixing build falure in centos ci job centos cmake symbol lookup error fix Signed-off-by: Arpad Boda This closes #1099 --- docker/centos/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docker/centos/Dockerfile b/docker/centos/Dockerfile index e54ab4c..b6284c1 100644 --- a/docker/centos/Dockerfile +++ b/docker/centos/Dockerfile @@ -27,7 +27,7 @@ ARG MINIFI_VERSION ENV MINIFI_BASE_DIR /opt/minifi ENV MINIFI_HOME $MINIFI_BASE_DIR/nifi-minifi-cpp-$MINIFI_VERSION -RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel python36-devel gcc gcc-c++ sudo git which maven make cmake +RUN yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel python36-devel gcc gcc-c++ sudo git which maven make cmake libarchive RUN mkdir -p $MINIFI_BASE_DIR COPY . ${MINIFI_BASE_DIR} @@ -40,6 +40,6 @@ RUN cd $MINIFI_BASE_DIR \ && rm -rf build \ && mkdir build \ && cd build \ - && cmake3 -DUSE_SHARED_LIBS= -DENABLE_MQTT=ON -DENABLE_LIBRDKAFKA=ON -DPORTABLE=ON -DENABLE_COAP=ON -DCMAKE_BUILD_TYPE=Release -DSKIP_TESTS=true -DENABLE_JNI=$ENABLE_JNI .. \ + && cmake -DUSE_SHARED_LIBS= -DENABLE_MQTT=ON -DENABLE_LIBRDKAFKA=ON -DPORTABLE=ON -DENABLE_COAP=ON -DCMAKE_BUILD_TYPE=Release -DSKIP_TESTS=true -DENABLE_JNI=$ENABLE_JNI .. \ && make -j$(nproc) package
[nifi-minifi-cpp] branch main updated (72e9c9e -> 33d65ed)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 72e9c9e Squashed commit of the following: new e51192e MINIFICPP-1582 fixing build falure in centos ci job new 33d65ed MINIFICPP-1583 windows build fix The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docker/centos/Dockerfile | 4 ++-- extensions/pdh/tests/PerformanceDataMonitorTests.cpp | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
[nifi-minifi-cpp] 02/02: MINIFICPP-1583 windows build fix
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 33d65edb3f005a17f3edbfd6fdcf9401605da568 Author: Martin Zink AuthorDate: Fri Jun 4 17:59:39 2021 +0200 MINIFICPP-1583 windows build fix Signed-off-by: Arpad Boda This closes #1100 --- extensions/pdh/tests/PerformanceDataMonitorTests.cpp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/extensions/pdh/tests/PerformanceDataMonitorTests.cpp b/extensions/pdh/tests/PerformanceDataMonitorTests.cpp index b3d5acd..cb49bab 100644 --- a/extensions/pdh/tests/PerformanceDataMonitorTests.cpp +++ b/extensions/pdh/tests/PerformanceDataMonitorTests.cpp @@ -250,16 +250,16 @@ TEST_CASE("PerformanceDataMonitorDecimalPlacesPropertyTest", "[performancedatamo { PerformanceDataMonitorTester tester; tester.setPerformanceMonitorProperty(PerformanceDataMonitor::DecimalPlaces, "asd"); -REQUIRE_THROWS_WITH(tester.runProcessors(), "General Operation: Invalid conversion to int64_t for asd"); +REQUIRE_THROWS_WITH(tester.runWithRetries([]{ return true; }, 1), "General Operation: Invalid conversion to int64_t for asd"); } { PerformanceDataMonitorTester tester; tester.setPerformanceMonitorProperty(PerformanceDataMonitor::DecimalPlaces, "1234586123"); -REQUIRE_THROWS_WITH(tester.runProcessors(), "Process Schedule Operation: PerformanceDataMonitor Decimal Places is out of range"); +REQUIRE_THROWS_WITH(tester.runWithRetries([]{ return true; }, 1), "Process Schedule Operation: PerformanceDataMonitor Decimal Places is out of range"); } { PerformanceDataMonitorTester tester; tester.setPerformanceMonitorProperty(PerformanceDataMonitor::DecimalPlaces, ""); -REQUIRE_NOTHROW(tester.runProcessors()); +REQUIRE_NOTHROW(tester.runWithRetries([]{ return true; }, 1)); } }
[nifi] branch main updated: [NIFI-8610] Do not reuse generic records in convertAvroToORC
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/main by this push: new f07e17b [NIFI-8610] Do not reuse generic records in convertAvroToORC f07e17b is described below commit f07e17ba74da2759213d52431e00f9d0ba5d39a5 Author: Dominik Przybysz AuthorDate: Tue May 18 10:26:58 2021 +0200 [NIFI-8610] Do not reuse generic records in convertAvroToORC [NIFI-8610] Simplify decimal test for convertAvroToORC Signed-off-by: Arpad Boda This closes #1081 --- .../nifi/processors/hive/ConvertAvroToORC.java | 5 +- .../nifi/processors/hive/TestConvertAvroToORC.java | 65 +- 2 files changed, 66 insertions(+), 4 deletions(-) diff --git a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java index b323633..d918365 100644 --- a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java +++ b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java @@ -234,9 +234,8 @@ public class ConvertAvroToORC extends AbstractProcessor { try { int recordCount = 0; -GenericRecord currRecord = null; while (reader.hasNext()) { -currRecord = reader.next(currRecord); +GenericRecord currRecord = reader.next(); List fields = currRecord.getSchema().getFields(); if (fields != null) { Object[] row = new Object[fields.size()]; @@ -284,7 +283,7 @@ public class ConvertAvroToORC extends AbstractProcessor { flowFile = session.putAttribute(flowFile, CoreAttributes.MIME_TYPE.key(), ORC_MIME_TYPE); flowFile = session.putAttribute(flowFile, CoreAttributes.FILENAME.key(), newFilename.toString()); session.transfer(flowFile, REL_SUCCESS); -session.getProvenanceReporter().modifyContent(flowFile, "Converted "+totalRecordCount.get()+" records", System.currentTimeMillis() - startTime); +session.getProvenanceReporter().modifyContent(flowFile, "Converted " + totalRecordCount.get() + " records", System.currentTimeMillis() - startTime); } catch (ProcessException | IllegalArgumentException e) { getLogger().error("Failed to convert {} from Avro to ORC due to {}; transferring to failure", new Object[]{flowFile, e}); diff --git a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestConvertAvroToORC.java b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestConvertAvroToORC.java index 9248b27..5fe6752 100644 --- a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestConvertAvroToORC.java +++ b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestConvertAvroToORC.java @@ -49,9 +49,9 @@ import org.junit.Test; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; -import java.io.InputStream; import java.io.File; import java.io.FileOutputStream; +import java.io.InputStream; import java.math.BigDecimal; import java.nio.ByteBuffer; import java.nio.charset.StandardCharsets; @@ -321,6 +321,69 @@ public class TestConvertAvroToORC { } @Test +public void test_onTrigger_complex_records_with_bigdecimals() throws Exception { + +Map mapData1 = new TreeMap() {{ +put("key1", 1.0); +put("key2", 2.0); +}}; + + +BigDecimal sampleBigDecimal1 = new BigDecimal("3500.12"); +BigDecimal sampleBigDecimal2 = new BigDecimal("0.01"); + +GenericData.Record record1 = TestNiFiOrcUtils.buildComplexAvroRecord(null, mapData1, "XYZ", 4L, Arrays.asList(100, 200), toByteBuffer(sampleBigDecimal1)); +DatumWriter writer = new GenericDatumWriter<>(record1.getSchema()); +DataFileWriter fileWriter = new DataFileWriter<>(writer); +ByteArrayOutputStream out = new ByteArrayOutputStream(); +fileWriter.create(record1.getSchema(), out); +fileWriter.append(record1); +fileWriter.append(TestNiFiOrcUtils.buildComplexAvroRecord(null, mapData1, "XYZ", 4L, Arrays.asList(100, 200), toByteBuffer(sampleBigDecimal2))); +fileWriter
[nifi-minifi-cpp] branch main updated: MINIFICPP-1568 Retry PDH processor run to fix test flakiness
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 05ae25c MINIFICPP-1568 Retry PDH processor run to fix test flakiness 05ae25c is described below commit 05ae25c1aae280767206b641f8c35d485aa85245 Author: Gabor Gyimesi AuthorDate: Wed May 26 17:05:47 2021 +0200 MINIFICPP-1568 Retry PDH processor run to fix test flakiness Signed-off-by: Arpad Boda This closes #1085 --- .../pdh/tests/PerformanceDataMonitorTests.cpp | 283 +++-- 1 file changed, 148 insertions(+), 135 deletions(-) diff --git a/extensions/pdh/tests/PerformanceDataMonitorTests.cpp b/extensions/pdh/tests/PerformanceDataMonitorTests.cpp index 7160bc5..b3d5acd 100644 --- a/extensions/pdh/tests/PerformanceDataMonitorTests.cpp +++ b/extensions/pdh/tests/PerformanceDataMonitorTests.cpp @@ -20,6 +20,7 @@ #include #include #include +#include #include "TestBase.h" #include "processors/PutFile.h" @@ -43,12 +44,18 @@ class PerformanceDataMonitorTester { plan_->setProperty(putfile_, PutFile::Directory.getName(), dir_); } - void runProcessors() { -plan_->runNextProcessor(); // PerformanceMonitor -std::this_thread::sleep_for(std::chrono::milliseconds(200)); -plan_->runCurrentProcessor(); // PerformanceMonitor -plan_->runNextProcessor(); // PutFile -plan_->runCurrentProcessor(); // PutFile + bool runWithRetries(std::function&& assertions, uint32_t max_tries = 10) { +uint32_t tries = 0; +for (uint32_t tries = 0; tries < max_tries; ++tries) { + test_controller_.runSession(plan_); + if (assertions()) { +return true; + } + plan_->reset(); + LogTestController::getInstance().reset(); + std::this_thread::sleep_for(std::chrono::milliseconds(100)); +} +return false; } void setPerformanceMonitorProperty(const core::Property& property, const std::string& value) { @@ -65,8 +72,7 @@ class PerformanceDataMonitorTester { TEST_CASE("PerformanceDataMonitorEmptyPropertiesTest", "[performancedatamonitoremptypropertiestest]") { PerformanceDataMonitorTester tester; - tester.runProcessors(); - + tester.test_controller_.runSession(tester.plan_); REQUIRE(tester.test_controller_.getLog().getInstance().contains("No valid counters for PerformanceDataMonitor", std::chrono::seconds(0))); auto created_flow_files = utils::file::FileUtils::list_dir_all(tester.dir_, tester.plan_->getLogger(), false); @@ -77,61 +83,64 @@ TEST_CASE("PerformanceDataMonitorPartiallyInvalidGroupPropertyTest", "[performan PerformanceDataMonitorTester tester; tester.setPerformanceMonitorProperty(PerformanceDataMonitor::PredefinedGroups, "Disk,CPU,Asd"); tester.setPerformanceMonitorProperty(PerformanceDataMonitor::CustomPDHCounters, "\\Invalid\\Counter,\\System\\Processes"); - tester.runProcessors(); - - REQUIRE(tester.test_controller_.getLog().getInstance().contains("Asd is not a valid predefined group", std::chrono::seconds(0))); - REQUIRE(tester.test_controller_.getLog().getInstance().contains("Error adding \\Invalid\\Counter to query", std::chrono::seconds(0))); - - uint32_t number_of_flowfiles = 0; - - auto lambda = [&number_of_flowfiles](const std::string& path, const std::string& filename) -> bool { -++number_of_flowfiles; -FILE* fp = fopen((path + utils::file::FileUtils::get_separator() + filename).c_str(), "r"); -REQUIRE(fp != nullptr); -char readBuffer[500]; -rapidjson::FileReadStream is(fp, readBuffer, sizeof(readBuffer)); -rapidjson::Document document; -document.ParseStream(is); -fclose(fp); -REQUIRE(document.IsObject()); -REQUIRE(document.HasMember("LogicalDisk")); -REQUIRE(document["LogicalDisk"].HasMember("_Total")); -REQUIRE(document["LogicalDisk"]["_Total"].HasMember("Free Megabytes")); -REQUIRE(document["System"].HasMember("Processes")); -return true; + const auto assertions = [&tester]() { +REQUIRE(tester.test_controller_.getLog().getInstance().contains("Asd is not a valid predefined group", std::chrono::seconds(0))); +REQUIRE(tester.test_controller_.getLog().getInstance().contains("Error adding \\Invalid\\Counter to query", std::chrono::seconds(0))); + +bool ff_contains_all_data = false; + +const auto lambda = [&ff_contains_all_data](const std::string& path, const std::string& filename) -> bool { + FILE* fp = fopen((path + utils::file::FileUtils::get_separator() + filename).c_str(), "r"); + REQUIRE(fp != null
[nifi-minifi-cpp] branch main updated: MINIFICPP-1577: Separate build and test steps in ubuntu-20.04 ci job
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 3c9fd89 MINIFICPP-1577: Separate build and test steps in ubuntu-20.04 ci job 3c9fd89 is described below commit 3c9fd895d413efce2e77cfcbe599ba15162d5df6 Author: Martin Zink AuthorDate: Tue Jun 1 14:23:25 2021 +0200 MINIFICPP-1577: Separate build and test steps in ubuntu-20.04 ci job Signed-off-by: Arpad Boda This closes #1095 --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 073350e..66efcfd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -239,7 +239,7 @@ jobs: echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV echo -e "127.0.0.1\t$HOSTNAME" | sudo tee -a /etc/hosts > /dev/null - name: build -run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DCMAKE_BUILD_TYPE=Release -DENABLE_BUSTACHE=ON -DENABLE_SQL=ON -DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && make -j4 VERBOSE=1 && make test ARGS="--timeout 300 -j2 --output-on-failure" +run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DCMAKE_BUILD_TYPE=Release -DENABLE_BUSTACHE=ON -DENABLE_SQL=ON -DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && make -j4 VERBOSE=1 - name: test run: cd build && make test ARGS="--timeout 300 -j2 --output-on-failure" - name: linter
[nifi-minifi-cpp] branch main updated: MINIFICPP-1578 Upgrade OS versions in docker builds
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 876906f MINIFICPP-1578 Upgrade OS versions in docker builds 876906f is described below commit 876906faffe274268a8ea417ee28b4b44f0ce8d8 Author: Ferenc Gerlits AuthorDate: Tue Jun 1 18:48:53 2021 +0200 MINIFICPP-1578 Upgrade OS versions in docker builds Signed-off-by: Arpad Boda This closes #1093 --- .gitignore | 2 +- README.md | 12 ++-- centos.sh | 4 ++-- cmake/DockerConfig.cmake| 4 ++-- docker/DockerBuild.sh | 17 + docker/centos/Dockerfile| 5 ++--- docker/debian/Dockerfile| 8 ++-- docker/fedora/Dockerfile| 2 +- docker/{xenial => focal}/Dockerfile | 5 +++-- 9 files changed, 28 insertions(+), 31 deletions(-) diff --git a/.gitignore b/.gitignore index 5926a60..60cce20 100644 --- a/.gitignore +++ b/.gitignore @@ -67,8 +67,8 @@ __pycache__/ # Ignore source files that have been placed in the docker directory during build docker/minificppsource -docker/xenial/minificppsource docker/bionic/minificppsource +docker/focal/minificppsource docker/debian/minificppsource docker/centos/minificppsource docker/fedora/minificppsource diff --git a/README.md b/README.md index 101280b..4e8a5f6 100644 --- a/README.md +++ b/README.md @@ -490,17 +490,17 @@ $ make docker-verify ``` ### Building For Other Distros -If you have docker installed on your machine you can build for CentOS 7, Fedora 29, Ubuntu 16, Ubuntu 18, and Debian 9 via our make docker commands. The following table +If you have docker installed on your machine you can build for CentOS 8, Fedora 34, Ubuntu 18.04, Ubuntu 20.04, and Debian 10 via our make docker commands. The following table provides the command to build your distro and the output file in your build directory. Since the versions are limited ( except for Ubuntu ) we output the archive based on the distro's name. | Distro | command | Output File | | - |:-| :-| -| CentOS 7 | make centos | nifi-minifi-cpp-centos-$VERSION-bin.tar.gz -| Debian 9 | make debian | nifi-minifi-cpp-debian-$VERSION-bin.tar.gz -| Fedora 29 | make fedora | nifi-minifi-cpp-fedora-$VERSION-bin.tar.gz -| Ubuntu 16 | make u16 | nifi-minifi-cpp-xenial-$VERSION-bin.tar.gz -| Ubuntu 18 | make u18 | nifi-minifi-cpp-bionic-$VERSION-bin.tar.gz +| CentOS 8 | make centos | nifi-minifi-cpp-centos-$VERSION-bin.tar.gz +| Debian 10 (buster) | make debian | nifi-minifi-cpp-debian-$VERSION-bin.tar.gz +| Fedora 34 | make fedora | nifi-minifi-cpp-fedora-$VERSION-bin.tar.gz +| Ubuntu 18.04 (bionic) | make u18 | nifi-minifi-cpp-bionic-$VERSION-bin.tar.gz +| Ubuntu 20.04 (focal) | make u20 | nifi-minifi-cpp-focal-$VERSION-bin.tar.gz ### Snapcraft diff --git a/centos.sh b/centos.sh index 7eef91e..84b0f03 100644 --- a/centos.sh +++ b/centos.sh @@ -138,9 +138,9 @@ build_deps(){ INSTALLED+=("java-1.8.0-openjdk-devel") INSTALLED+=("maven") elif [ "$FOUND_VALUE" = "python" ]; then -INSTALLED+=("python34-devel") +INSTALLED+=("python36-devel") elif [ "$FOUND_VALUE" = "lua" ]; then -INSTALLED+=("lua-devel") +INSTALLED+=("lua-libs") elif [ "$FOUND_VALUE" = "gpsd" ]; then INSTALLED+=("gpsd-devel") elif [ "$FOUND_VALUE" = "libarchive" ]; then diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake index e6d2040..5bb4107 100644 --- a/cmake/DockerConfig.cmake +++ b/cmake/DockerConfig.cmake @@ -124,7 +124,7 @@ add_custom_target( WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/) add_custom_target( -u16 +u20 COMMAND ${CMAKE_SOURCE_DIR}/docker/DockerBuild.sh -u 1000 -g 1000 @@ -132,7 +132,7 @@ add_custom_target( -i release -c ENABLE_JNI=${ENABLE_JNI} -l ${CMAKE_BINARY_DIR} --d xenial +-d focal -c BUILD_NUMBER=${BUILD_NUMBER} WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/) diff --git a/docker/DockerBuild.sh b/docker/DockerBuild.sh index 5f954d7..4641072 100755 --- a/docker/DockerBuild.sh +++ b/docker/DockerBuild.sh @@ -36,7 +36,7 @@ function usage { echo "-i, --image-type Can be release or minimal (default: release)" echo "-u, --uid User id to be used in the Docker image (def
svn commit: r48033 - /release/nifi/KEYS
Author: aboda Date: Tue Jun 1 10:57:14 2021 New Revision: 48033 Log: Adding Ferenc Gerlits and Adam Debreceni Modified: release/nifi/KEYS Modified: release/nifi/KEYS == --- release/nifi/KEYS (original) +++ release/nifi/KEYS Tue Jun 1 10:57:14 2021 @@ -1722,3 +1722,136 @@ J2LnyOSBJXeH2UleGTF/QNxXW1FzhDN75/I0EAY9 UkE= =5IZc -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2021-01-20 [SC] [expires: 2023-01-20] + EF3757EEF7D8FBD652284EC5576CB454DDD6500D +uid [ultimate] Adam Debreceni +sig 3576CB454DDD6500D 2021-01-21 Adam Debreceni +uid [ultimate] Adam Debreceni +sig 3576CB454DDD6500D 2021-01-20 Adam Debreceni +sub rsa4096 2021-01-20 [E] [expires: 2023-01-20] +sig 576CB454DDD6500D 2021-01-20 Adam Debreceni + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBGAH/CABEADM+g8UUTuAgXEqKY0jYme4SH6PmBWRbkTHIh8l6SpKBMPATOl7 +hX6Tlzlv/8JchS89CgqBSwAFAmifO7uBht3rICMnZSHzgZDHD73duC3YZpHjXxmj +zyT6CIUL2LviRU/sVUO2oVcSWPXt9rDQUJo1u+7ZrLa5zjJtW6VMceYFYlnXa21w +emEjd0oooPW9s/KOXBBTR0qGjJo6ukiiZFWH6fSOmSuzqyhrwgDZsktAsF3GsAz3 +QpR5zgEwHxBZdthGiJrrsa5I53ULTvDY6CtgwIomgKf+M6lZGKiccCZGrEIlISOG +854XKqt9Dw+vlK77unkgkQmJe/0EBcGCjIkAAQvBM/WE0yYFj9AFVciRSHj3whqa +xwOhdFUykMuNtDQSdpKd/0vyWGhGM2KweSo/7C4JAOlGE1WiiE0v1eJZNFS6vN00 +5d/O4Da0QQQpLWvkuXtsoXM02fvfUSDyBj7nks5+23cOJO5pTvG1a6oq3kQH/hVL +bpy0x/g57vYpFfy1jn0glkdTypvDCKZ3xiJScDpCr4dM0VsGWB7AxZIgULzZXOWk +5XF0VQ5dJFpciL0Zy+8Mbn7N3bUGiKOrawGyb3B3FWykpzSMW2Y9HLMZjcj60d+0 +u+kR6A+varjKDjOTAHHpL3neNJ7LTbtBFc4hnMr8oQmbD+X6J6unCoWL6wARAQAB +tC5BZGFtIERlYnJlY2VuaSA8YWRhbS5kZWJyZWNlbmlAcHJvdG9ubWFpbC5jb20+ +iQJUBBMBCAA+FiEE7zdX7vfY+9ZSKE7FV2y0VN3WUA0FAmAH/CACGwMFCQPCZwAF +CwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQV2y0VN3WUA3AJA//SVIquVis/ZxL +Kgg1tLTKY/DgTVphlDjtS4klxf9vDmyC0QmXuKIdmunmAVhijFKc/hup7NikYuSz +0+MEpeDoiCV31BW3AqVhEoX9jbHYIHCZNn0UlnZlg/EYZT3LcaN7w33pwRLT/Hzu +uOoJQIdK3WpDaTpnBehk9LaeLPJDnElgwUSjCZpp/3fBMWCA9HmuVY1UbRxEcYyP +yfTNNgq/lfmVTvjeZSdhL6ohs8AaL1S/hFMRQj3AvYUgFlvFc9zO06xIynwTRw9i +qjm4SUbJZfcmRzmPOMumWyExLK+fr2R8Gom0L028L4L8Ds787MIHlKtSOIRvxN2k +G1pTpzl9I+Vmx9oG3YdfPRa3aE3CNooEO9gPquT8kJWEsItrmqbS9YA2d/E4uYws +KZSyBoTQRf4ADvmCkXrlJwJfyfxtDZQGTC49ZucQmeqL5CVctoFvanYQeyq19t4w +v8D8L7wlCD+d9qninJzahaES1BUCNHd5/4oDKFhLJH3pLqzw08r2KLL5whsfEEdH ++Z2lZKUoWIkYqsFNV1EWib8u+lEnR2l19pWWgUEjlrSamo1RbNPAZbMX9VClyMqh +HA1K33Aq08yx5JEV5kBBtIRt5E0tUCv85JyZ7tX1cwyfSlGD66wO1gBfjgb3uemm +yATBprzJGrbvY4TXon5QqAABWOZr+Py0JkFkYW0gRGVicmVjZW5pIDxhZGVicmVj +ZW5pQGFwYWNoZS5vcmc+iQJUBBMBCAA+FiEE7zdX7vfY+9ZSKE7FV2y0VN3WUA0F +AmAJMLwCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQV2y0VN3W +UA1JGhAAjjRrZNaO4tqfJOPD3u0TOEfwjNFR2BTASMGEk7bJYYW4M131aTBUWFn/ +aBhRTgGjACwV50Hpiss05ka1F4QDBE00xj80Uk6rEld6tVxO+j38IV8wDSGmAR/H +GCLv2FGzOFB6QqvBLelIisns5Nts3+40hJx8gsH+kTLmQcPSybkySSsu2bkcNJa8 +WKONP39nqZAL9DhWqRHnSl+wwL1jwQdXpu6pjnhUu6u2LB3EKSiW3LpFIaHrL2qk +zn+02kNeeKPaRRGOVQRvyqYuL36K5Wfg1j+V1yft8QywczquRAISP/E1RkifPSZ0 +Z6CkPbFvm/FwHLfyVxbYhI0V/KXt2NMzK2Ux0P1x/dvRh0eK+EyvmYHTRQa8J05B +hmF5BL0TIkZJOx5ZvQQkS0PHGtEasCnEEHhfFzuNGA0ZIRaCzmzAFa18mSHTSLRv +tZTO0TVkA84FoNLJ4gdIiIeJFlK7Xc//GvFlcD819UdyXxmZzWYt77MC78Jfvku+ +v8lvqzq1U8ihnhUzHHicXRQrcyDAxEyC39LtDYLxjE3Svy3osKDgr6XKPJvBXfil +HvNcFXx5KRrHvdnhyQN7KjTgdZMw4r1PcUfh5oNIGh5t3Hrcgx2BZL4RXFFIQJcl +PBcQdVaH3HlTLSECZnp0mWKmLNTZGhA9yaGJfj3cuecfj+m/bb25Ag0EYAf8IAEQ +ANOdFTyfj5Snnk/pJCi9QeF+2Zndt2DV0ROdltm3xY6Gi42zxxCjn1WUkZdOwDfo +RMSUnMvNWM6/KPxa3I8ldh7ie1eiNvCagw6i6P/fy6nVrFSvmAOSNkqflNGZ2H+e +1LlhXQsmcqWlyHevSWF8vSEeFRVFdLzJOBQAO3IkWXUnY/LHA2vnq9VX4mGrLTmG ++oxO+fYYvAFKVv0EFiqNfcHepA51RrP92ETEac3seKzvcbyH0YwFWbiT7vNyjAhY +ZCKTOQXcEYYJ3/N/0Q45KeMvYwhYFP0F0XdF2igi8vLNjmpB9lOKqIrPbTouI6ER +gasMXeBxMkZQ2NNWzfP3AetjEMZJjbU2vjye9L8hJR2HJTXqBajICCKB1VIiIMwQ +AIXOCzogC6/iLDhQTiflfcmHlzeEt1yhoGtN3DsOKht8LG5VA6XV5y0LS1PTGLx2 +KVNqz9pxju8yt59Ribw9nkDBH5vA7dJgcWiOIlTHt/3A+ab78ce/iptWqyYqM4Fm +x0fQGzkFhzZzJYxMF5ofJxS7DyNW0lNGg3b7l8/LpsVR86p3gSEOeGOHQTPnLQLL +ThTpd+uLMp+zMlm/bnoUGjQNuYk7COV2owqwsqgYPIL/Q5GL0mnJ0dVJCFgjDhLk +ZrL7MU8ndClu2SuaNWljCc7UnJDXKlDLBIbdjF/dxMJxABEBAAGJAjwEGAEIACYW +IQTvN1fu99j71lIoTsVXbLRU3dZQDQUCYAf8IAIbDAUJA8JnAAAKCRBXbLRU3dZQ +DcJAD/9nK8bOALnYuqUTjb7AEY0O8R8mWdxfye8ybvm8LRy5r4MUNvdAw7Cpa5EL +32PkmQ94XA+pM2NnRyo6Q6SWBnwDdueH0JRRBScqHAp8/P3W2Xa+N4OFqw55fwBS +fxLW9PltHM7ejrU5AH65Dvs5T6MDhdrEITRwzCTt8hzRHsH2X4YxK5H8LS9WCsa+ +oG5HnX6Y2P7Buyfa1ScqlAM8CGj2gULDPkZcTpgJf11YlNS0o/430RalQyi7+r/L +F+46T8LzxfRRZrw064meLNse3BBJfamGLnS2lJmwdNgjHbL6KH1Z+2Vl6/F9i7mT +bYXWtVpG0t5kvqKBTu7XCGLOwTmVeR3FtGz6+hZPInRSBlrJ8pSNLCLVWLPQwqQd +qA0W6UdWyRMbv2egs0GmMib5iJlMkdyfXlea6d5Ha1EFp/ZEOmLCKBhgkqlxWxol +8OoFvO7EsoGiS1VpwYfXDbRV1jUy+sAM637CfNYaWFhsl/KsjOUxuhLzcI8A5dK2 +WoLJFe0eVD7F5fMv2AmlMVTsu90YwDCi3RmaYfXMqInzoInjZ31P0rqm0loJAAMm +CRzjcdrKutLHQ6z7nSayCz2cpZaN9ogoTsBc4X6Y/WlJN6XdJlYp6QDmJYp+rtoP +2rSElbZPmVnCpbKnQlp5EtPNAv9YBC744D7Efgj2uI8Q7se7CA== +=yOyR +-END PGP PUBLIC KEY BLOCK- +pub rsa4096 2021-03-22 [SC
[nifi] branch main updated (e0a1cb8 -> 96d5211)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git. from e0a1cb8 NIFI-8133 Add ability to supress null/empty values in ElasticSearchClientService for PutElasticsearchRecord output add 96d5211 Fixing keys of Adam Debreceni and Ferenc Gerlits No new revisions were added by this update. Summary of changes: KEYS | 4 1 file changed, 4 insertions(+)
[nifi-minifi-cpp] branch main updated: MINIFICPP-1575 - Make linter creation configurable
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 88deb7a MINIFICPP-1575 - Make linter creation configurable 88deb7a is described below commit 88deb7a0e4672ca3a66d9698022ae21906e1f817 Author: Adam Debreceni AuthorDate: Mon May 31 13:41:12 2021 +0200 MINIFICPP-1575 - Make linter creation configurable Signed-off-by: Arpad Boda This closes #1091 --- .github/workflows/ci.yml | 2 +- CMakeLists.txt | 1 + cmake/Extensions.cmake | 10 ++ win_build_vs.bat | 4 +++- 4 files changed, 11 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d330b4a..489fed1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -152,7 +152,7 @@ jobs: run: | PATH %PATH%;C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64 PATH %PATH%;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn - win_build_vs.bat build /2019 /64 /CI /S /A /PDH /T + win_build_vs.bat build /2019 /64 /CI /S /A /PDH /T /L shell: cmd - name: test run: cd build && ctest --timeout 300 --parallel 8 -C Release --output-on-failure diff --git a/CMakeLists.txt b/CMakeLists.txt index 8c80382..ab21b88 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -37,6 +37,7 @@ include(ExternalProject) list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake") include(WholeArchive) +option(ENABLE_LINTER "Create linter components" ON) option(SKIP_TESTS "Skips building all tests." OFF) option(PORTABLE "Instructs the compiler to remove architecture specific optimizations" ON) option(USE_SHARED_LIBS "Builds using shared libraries" ON) diff --git a/cmake/Extensions.cmake b/cmake/Extensions.cmake index 774a001..2475906 100644 --- a/cmake/Extensions.cmake +++ b/cmake/Extensions.cmake @@ -68,10 +68,12 @@ endfunction() macro(register_extension_linter target-name) - get_property(extensions GLOBAL PROPERTY EXTENSION-LINTERS) - set_property(GLOBAL APPEND PROPERTY EXTENSION-LINTERS "${target-name}") - add_custom_target(${target-name} - COMMAND python ${CMAKE_SOURCE_DIR}/thirdparty/google-styleguide/run_linter.py -q -i ${CMAKE_CURRENT_LIST_DIR}/) + if (ENABLE_LINTER) +get_property(extensions GLOBAL PROPERTY EXTENSION-LINTERS) +set_property(GLOBAL APPEND PROPERTY EXTENSION-LINTERS "${target-name}") +add_custom_target(${target-name} +COMMAND python ${CMAKE_SOURCE_DIR}/thirdparty/google-styleguide/run_linter.py -q -i ${CMAKE_CURRENT_LIST_DIR}/) + endif() endmacro() # ARGN WILL be the diff --git a/win_build_vs.bat b/win_build_vs.bat index 952e3af..b710792 100755 --- a/win_build_vs.bat +++ b/win_build_vs.bat @@ -36,6 +36,7 @@ set cpack=OFF set installer_merge_modules=OFF set strict_gsl_checks= set redist= +set build_linter=OFF set arg_counter=0 for %%x in (%*) do ( @@ -58,12 +59,13 @@ for %%x in (%*) do ( if [%%~x] EQU [/DD] set cmake_build_type=Debug if [%%~x] EQU [/CI] set "strict_gsl_checks=-DSTRICT_GSL_CHECKS=AUDIT" & set test_custom_wel_provider=ON if [%%~x] EQU [/NONFREEUCRT] set "redist=-DMSI_REDISTRIBUTE_UCRT_NONASL=ON" +if [%%~x] EQU [/L] set build_linter=ON ) mkdir %builddir% pushd %builddir%\ -cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_L [...] +cmake -G %generator% -A %build_platform% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DUSE_SHARED_L [...] IF %ERRORLEVEL% NEQ 0 EXIT /b %ERRORLEVEL% if [%cpack%] EQU [ON] ( cpack -C %cmake_build_type%
[nifi-minifi-cpp] branch main updated: MINIFICPP-1559 Add new redistributable file to the msi in VS2019 builds
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new ff31971 MINIFICPP-1559 Add new redistributable file to the msi in VS2019 builds ff31971 is described below commit ff3197165122b91b3f77a42c3036d355f3844ac6 Author: Ferenc Gerlits AuthorDate: Thu May 13 18:42:30 2021 +0200 MINIFICPP-1559 Add new redistributable file to the msi in VS2019 builds add msvcp140_atomic_wait.dll and msvcp140_codecvt_ids.dll ... and remove vcruntime140_1.dll in 32-bit VS2019 builds Signed-off-by: Arpad Boda This closes #1080 --- CMakeLists.txt | 18 +- msi/WixWin.wsi | 49 + 2 files changed, 26 insertions(+), 41 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 5fc3273..8c80382 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -671,8 +671,8 @@ if(WIN32) list(SORT VCRUNTIME_REDIST_VERSIONS) list(REVERSE VCRUNTIME_REDIST_VERSIONS) list(GET VCRUNTIME_REDIST_VERSIONS 0 VCRUNTIME_REDIST_DIR) - message("Using redist directory: ${VCRUNTIME_REDIST_DIR}") endif() + message("Using redist directory: ${VCRUNTIME_REDIST_DIR}") if (MSI_REDISTRIBUTE_UCRT_NONASL) set(UCRT_DIR_NAT "$ENV{WindowsSdkDir}Redist\\ucrt\\DLLs\\$ENV{Platform}") @@ -733,20 +733,12 @@ if(WIN32) if(CMAKE_SIZEOF_VOID_P EQUAL 8) message("Using ${VCRUNTIME_X64_REDIST_CRT_DIR} VC Redistributables") - file(COPY "${VCRUNTIME_X64_REDIST_CRT_DIR}/concrt140.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X64_REDIST_CRT_DIR}/msvcp140.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X64_REDIST_CRT_DIR}/msvcp140_1.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X64_REDIST_CRT_DIR}/msvcp140_2.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X64_REDIST_CRT_DIR}/vccorlib140.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X64_REDIST_CRT_DIR}/vcruntime140.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") + file(GLOB VCRUNTIME_X64_REDIST_CRT_FILES "${VCRUNTIME_X64_REDIST_CRT_DIR}/*.dll") + file(COPY ${VCRUNTIME_X64_REDIST_CRT_FILES} DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") elseif(CMAKE_SIZEOF_VOID_P EQUAL 4) message("Using ${VCRUNTIME_X86_REDIST_CRT_DIR} VC Redistributables") - file(COPY "${VCRUNTIME_X86_REDIST_CRT_DIR}/concrt140.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X86_REDIST_CRT_DIR}/msvcp140.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X86_REDIST_CRT_DIR}/msvcp140_1.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X86_REDIST_CRT_DIR}/msvcp140_2.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X86_REDIST_CRT_DIR}/vccorlib140.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") - file(COPY "${VCRUNTIME_X86_REDIST_CRT_DIR}/vcruntime140.dll" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") + file(GLOB VCRUNTIME_X86_REDIST_CRT_FILES "${VCRUNTIME_X86_REDIST_CRT_DIR}/*.dll") + file(COPY ${VCRUNTIME_X86_REDIST_CRT_FILES} DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/redist") else() message(FATAL_ERROR "Could not determine architecture, CMAKE_SIZEOF_VOID_P is unexpected: ${CMAKE_SIZEOF_VOID_P}") endif() diff --git a/msi/WixWin.wsi b/msi/WixWin.wsi index fbac160..411e52c 100644 --- a/msi/WixWin.wsi +++ b/msi/WixWin.wsi @@ -44,22 +44,16 @@ Licensed to the Apache Software Foundation (ASF) under one or more - - - + + + + @@ -70,7 +64,6 @@ Licensed to the Apache Software Foundation (ASF) under one or more - @@ -129,8 +122,6 @@ Licensed to the Apache Software Foundation (ASF) under one or more - - @@ -254,11 +245,8 @@ Licensed to the Apache Software Foundation (ASF) under one or more --> - - - @@ -318,12 +306,6 @@ Licensed to the Apache Software Foundation (ASF) under one or more - - - - - -
[nifi-minifi-cpp] branch main updated (cb4b2be -> d3b149c)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from cb4b2be MINIFICPP-1564 - Remove agent update add d3b149c MINIFICPP-1565 - Minor improvements to PerformanceDataMonitor No new revisions were added by this update. Summary of changes: PROCESSORS.md | 103 + extensions/pdh/PDHCounters.cpp | 12 +-- extensions/pdh/PerformanceDataMonitor.cpp | 100 extensions/pdh/PerformanceDataMonitor.h| 14 ++- .../pdh/tests/PerformanceDataMonitorTests.cpp | 23 - libminifi/include/utils/JsonCallback.h | 25 - 6 files changed, 248 insertions(+), 29 deletions(-)
[nifi-minifi-cpp] branch main updated (cde679f -> cb4b2be)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from cde679f MINIFICPP-1563 fix -Wmaybe-uninitialized warnings add cb4b2be MINIFICPP-1564 - Remove agent update No new revisions were added by this update. Summary of changes: extensions/http-curl/tests/C2UpdateAgentTest.cpp | 40 --- extensions/http-curl/tests/CMakeLists.txt| 1 - extensions/http-curl/tests/HTTPIntegrationBase.h | 22 --- libminifi/include/c2/C2Agent.h | 10 --- libminifi/src/c2/C2Agent.cpp | 84 5 files changed, 157 deletions(-) delete mode 100644 extensions/http-curl/tests/C2UpdateAgentTest.cpp
[nifi-minifi-cpp] branch main updated: MINIFICPP-1532: Create a processor to capture resource consumption metrics on Windows
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 98b9055 MINIFICPP-1532: Create a processor to capture resource consumption metrics on Windows 98b9055 is described below commit 98b9055fda485f87cdc6e161c29ed557f063846a Author: Martin Zink AuthorDate: Wed Mar 24 12:29:15 2021 +0100 MINIFICPP-1532: Create a processor to capture resource consumption metrics on Windows Updated documentation to include information about the new processor Signed-off-by: Arpad Boda This closes #1066 --- .github/workflows/ci.yml | 4 +- CMakeLists.txt | 18 +- PROCESSORS.md | 22 ++ README.md | 1 + Windows.md | 1 + extensions/pdh/CMakeLists.txt | 32 +++ extensions/pdh/MemoryConsumptionCounter.h | 70 + extensions/pdh/PDHCounters.cpp | 132 ++ extensions/pdh/PDHCounters.h | 99 +++ extensions/pdh/PerformanceDataCounter.h| 51 extensions/pdh/PerformanceDataMonitor.cpp | 284 + extensions/pdh/PerformanceDataMonitor.h| 90 +++ extensions/pdh/tests/CMakeLists.txt| 37 +++ .../pdh/tests/PerformanceDataCounterTests.cpp | 187 ++ .../pdh/tests/PerformanceDataMonitorTests.cpp | 231 + libminifi/include/utils/JsonCallback.h | 54 libminifi/include/utils/OsUtils.h | 5 + libminifi/src/utils/OsUtils.cpp| 10 + libminifi/test/unit/MemoryUsageTest.cpp| 35 ++- win_build_vs.bat | 3 +- 20 files changed, 1347 insertions(+), 19 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 83e7e86..916e6fa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -102,7 +102,7 @@ jobs: run: | PATH %PATH%;C:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x86 PATH %PATH%;C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn - win_build_vs.bat build /CI /S /A /Z + win_build_vs.bat build /CI /S /A /Z /PDH shell: cmd windows_VS2019: name: "windows-vs2019" @@ -141,7 +141,7 @@ jobs: run: | PATH %PATH%;C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64 PATH %PATH%;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn - win_build_vs.bat build /2019 /64 /CI /S /A + win_build_vs.bat build /2019 /64 /CI /S /A /PDH shell: cmd ubuntu_16_04: name: "ubuntu-16.04" diff --git a/CMakeLists.txt b/CMakeLists.txt index 6a222db..136796d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -493,7 +493,7 @@ if (ENABLE_TENSORFLOW) createExtension(TENSORFLOW-EXTENSIONS "TENSORFLOW EXTENSIONS" "This enables TensorFlow support" "extensions/tensorflow" "${TEST_DIR}/tensorflow-tests") endif() -## AWS Extentions +## AWS Extensions option(ENABLE_AWS "Enables AWS support." OFF) if (ENABLE_ALL OR ENABLE_AWS) include(BundledAwsSdkCpp) @@ -501,7 +501,15 @@ if (ENABLE_ALL OR ENABLE_AWS) createExtension(AWS-EXTENSIONS "AWS EXTENSIONS" "This enables AWS support" "extensions/aws" "${TEST_DIR}/aws-tests") endif() -## OpenCV Extesions +## PDH Extentsions +if (WIN32) + option(ENABLE_PDH "Enables PDH support." OFF) + if (ENABLE_PDH) + createExtension(PDH-EXTENSIONS "PDH EXTENSIONS" "This enables PDH support" "extensions/pdh" "extensions/pdh/tests") + endif() +endif() + +## OpenCV Extensions option(ENABLE_OPENCV "Disables the OpenCV extensions." OFF) if (ENABLE_OPENCV) createExtension(OPENCV-EXTENSIONS "OPENCV EXTENSIONS" "This enabled OpenCV support" "extensions/opencv" "extensions/opencv/tests") @@ -515,7 +523,7 @@ if (ENABLE_BUSTACHE) createExtension(BUSTACHE-EXTENSIONS "BUSTACHE EXTENSIONS" "This enables bustache functionality including ApplyTemplate." "extensions/bustache" "${TEST_DIR}/bustache-tests") endif() -## OPC Extentions +## OPC Extensions if (ENABLE_OPC) include(BundledMbedTLS) use_bundled_mbedtls(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR}) @@ -546,7 +554,7 @@ if ((ENABLE_OPENWSMAN AND NOT DISABLE_CIVET AND NOT
[nifi-minifi-cpp] branch main updated (4fc6f49 -> cf4dbd9)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 4fc6f49 MINIFICPP-1504 followup review changes new f285b88 MINIFICPP-1553 Support credential refresh in AWSCredentialsService new cf4dbd9 MINIFICPP-1554 - Log c2 response code The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../controllerservices/AWSCredentialsService.cpp | 26 ++--- .../aws/controllerservices/AWSCredentialsService.h | 14 ++--- extensions/http-curl/protocols/RESTSender.cpp | 9 ++- extensions/mqtt/protocol/MQTTC2Protocol.cpp| 2 +- .../test/aws-tests/AWSCredentialsServiceTest.cpp | 66 ++ 5 files changed, 100 insertions(+), 17 deletions(-) create mode 100644 libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp
[nifi-minifi-cpp] 02/02: MINIFICPP-1554 - Log c2 response code
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit cf4dbd926b70fb61826c77290ea12211c151555b Author: Adam Debreceni AuthorDate: Tue May 4 16:09:36 2021 +0200 MINIFICPP-1554 - Log c2 response code Signed-off-by: Arpad Boda This closes #1068 --- extensions/http-curl/protocols/RESTSender.cpp | 9 - extensions/mqtt/protocol/MQTTC2Protocol.cpp | 2 +- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/extensions/http-curl/protocols/RESTSender.cpp b/extensions/http-curl/protocols/RESTSender.cpp index 0f831e3..7f41c4a 100644 --- a/extensions/http-curl/protocols/RESTSender.cpp +++ b/extensions/http-curl/protocols/RESTSender.cpp @@ -38,7 +38,7 @@ namespace c2 { RESTSender::RESTSender(const std::string &name, const utils::Identifier &uuid) : C2Protocol(name, uuid), - logger_(logging::LoggerFactory::getLogger()) { + logger_(logging::LoggerFactory::getLogger()) { } void RESTSender::initialize(core::controller::ControllerServiceProvider* controller, const std::shared_ptr &configure) { @@ -139,6 +139,13 @@ const C2Payload RESTSender::sendPayload(const std::string url, const Direction d } bool isOkay = client.submit(); int64_t respCode = client.getResponseCode(); + const bool clientError = 400 <= respCode && respCode < 500; + const bool serverError = 500 <= respCode && respCode < 600; + if (clientError || serverError) { +logger_->log_error("Error response code '" "%" PRId64 "' from '%s'", respCode, url); + } else { +logger_->log_debug("Response code '" "%" PRId64 "' from '%s'", respCode, url); + } auto rs = client.getResponseBody(); if (isOkay && respCode) { if (payload.isRaw()) { diff --git a/extensions/mqtt/protocol/MQTTC2Protocol.cpp b/extensions/mqtt/protocol/MQTTC2Protocol.cpp index 79acf77..d50f8cb 100644 --- a/extensions/mqtt/protocol/MQTTC2Protocol.cpp +++ b/extensions/mqtt/protocol/MQTTC2Protocol.cpp @@ -25,7 +25,7 @@ namespace c2 { MQTTC2Protocol::MQTTC2Protocol(const std::string& name, const utils::Identifier& uuid) : C2Protocol(name, uuid), - logger_(logging::LoggerFactory::getLogger()) { + logger_(logging::LoggerFactory::getLogger()) { } MQTTC2Protocol::~MQTTC2Protocol() = default;
[nifi-minifi-cpp] 01/02: MINIFICPP-1553 Support credential refresh in AWSCredentialsService
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit f285b88da04c2895d69f292c95518ef6579958d3 Author: Gabor Gyimesi AuthorDate: Fri Apr 30 17:30:44 2021 +0200 MINIFICPP-1553 Support credential refresh in AWSCredentialsService Add test for credential refresh Signed-off-by: Arpad Boda This closes #1067 --- .../controllerservices/AWSCredentialsService.cpp | 26 ++--- .../aws/controllerservices/AWSCredentialsService.h | 14 ++--- .../test/aws-tests/AWSCredentialsServiceTest.cpp | 66 ++ 3 files changed, 91 insertions(+), 15 deletions(-) diff --git a/extensions/aws/controllerservices/AWSCredentialsService.cpp b/extensions/aws/controllerservices/AWSCredentialsService.cpp index 2f9477e..439f57b 100644 --- a/extensions/aws/controllerservices/AWSCredentialsService.cpp +++ b/extensions/aws/controllerservices/AWSCredentialsService.cpp @@ -60,16 +60,26 @@ void AWSCredentialsService::initialize() { } void AWSCredentialsService::onEnable() { - getProperty(AccessKey.getName(), access_key_); - getProperty(SecretKey.getName(), secret_key_); - getProperty(CredentialsFile.getName(), credentials_file_); - getProperty(UseDefaultCredentials.getName(), use_default_credentials_); + std::string value; + getProperty(AccessKey.getName(), value); + aws_credentials_provider_.setAccessKey(value); + getProperty(SecretKey.getName(), value); + aws_credentials_provider_.setSecretKey(value); + getProperty(CredentialsFile.getName(), value); + aws_credentials_provider_.setCredentialsFile(value); + bool use_default_credentials = false; + getProperty(UseDefaultCredentials.getName(), use_default_credentials); + aws_credentials_provider_.setUseDefaultCredentials(use_default_credentials); +} - aws_credentials_provider_.setAccessKey(access_key_); - aws_credentials_provider_.setSecretKey(secret_key_); - aws_credentials_provider_.setCredentialsFile(credentials_file_); - aws_credentials_provider_.setUseDefaultCredentials(use_default_credentials_); +Aws::Auth::AWSCredentials AWSCredentialsService::getAWSCredentials() { + if (aws_credentials_.IsExpiredOrEmpty()) { +cacheCredentials(); + } + return aws_credentials_; +} +void AWSCredentialsService::cacheCredentials() { auto aws_credentials_result = aws_credentials_provider_.getAWSCredentials(); if (aws_credentials_result) { aws_credentials_ = aws_credentials_result.value(); diff --git a/extensions/aws/controllerservices/AWSCredentialsService.h b/extensions/aws/controllerservices/AWSCredentialsService.h index 0551b5b..22d29fd 100644 --- a/extensions/aws/controllerservices/AWSCredentialsService.h +++ b/extensions/aws/controllerservices/AWSCredentialsService.h @@ -29,6 +29,8 @@ #include "core/logging/LoggerConfiguration.h" #include "AWSCredentialsProvider.h" +class AWSCredentialsServiceTestAccessor; + namespace org { namespace apache { namespace nifi { @@ -66,16 +68,14 @@ class AWSCredentialsService : public core::controller::ControllerService { void onEnable() override; - Aws::Auth::AWSCredentials getAWSCredentials() { -return aws_credentials_; - } + Aws::Auth::AWSCredentials getAWSCredentials(); private: + friend class ::AWSCredentialsServiceTestAccessor; + + void cacheCredentials(); + const utils::AWSInitializer& AWS_INITIALIZER = utils::AWSInitializer::get(); - std::string access_key_; - std::string secret_key_; - std::string credentials_file_; - bool use_default_credentials_ = false; Aws::Auth::AWSCredentials aws_credentials_; AWSCredentialsProvider aws_credentials_provider_; }; diff --git a/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp b/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp new file mode 100644 index 000..27e4353 --- /dev/null +++ b/libminifi/test/aws-tests/AWSCredentialsServiceTest.cpp @@ -0,0 +1,66 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include + +#include "../TestBase.h" +#include "controllerservices/AWSCredentialsService.h" +#include &q
[nifi-minifi-cpp] branch main updated: MINIFICPP-1502 Pass name and uuid arguments as const refs
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 4e47492 MINIFICPP-1502 Pass name and uuid arguments as const refs 4e47492 is described below commit 4e474928550dc08ba021359f3eff3d3e54ccd9af Author: Gabor Gyimesi AuthorDate: Mon Apr 26 12:00:27 2021 +0200 MINIFICPP-1502 Pass name and uuid arguments as const refs Signed-off-by: Arpad Boda This closes #1062 --- .../aws/controllerservices/AWSCredentialsService.h | 2 +- .../controllerservices/AzureStorageCredentialsService.h | 2 +- extensions/bustache/ApplyTemplate.h | 2 +- extensions/civetweb/processors/ListenHTTP.h | 2 +- extensions/coap/controllerservice/CoapConnector.h| 2 +- extensions/gps/GetGPS.h | 2 +- extensions/http-curl/client/HTTPClient.cpp | 2 +- extensions/http-curl/client/HTTPClient.h | 2 +- extensions/http-curl/processors/InvokeHTTP.h | 2 +- extensions/http-curl/protocols/AgentPrinter.cpp | 2 +- extensions/http-curl/protocols/AgentPrinter.h| 2 +- extensions/http-curl/protocols/RESTReceiver.cpp | 2 +- extensions/http-curl/protocols/RESTReceiver.h| 2 +- extensions/http-curl/sitetosite/HTTPProtocol.h | 2 +- extensions/http-curl/sitetosite/PeersEntity.h| 4 ++-- extensions/jni/ExecuteJavaControllerService.h| 2 +- extensions/jni/ExecuteJavaProcessor.h| 2 +- extensions/jni/JVMCreator.h | 2 +- extensions/jni/jvm/JavaControllerService.h | 2 +- extensions/libarchive/CompressContent.h | 2 +- extensions/libarchive/FocusArchiveEntry.h| 2 +- extensions/libarchive/ManipulateArchive.h| 2 +- extensions/libarchive/MergeContent.h | 2 +- extensions/libarchive/UnfocusArchiveEntry.h | 2 +- extensions/librdkafka/PublishKafka.h | 2 +- .../mqtt/controllerservice/MQTTControllerService.h | 4 ++-- extensions/mqtt/processors/AbstractMQTTProcessor.h | 2 +- extensions/mqtt/processors/ConsumeMQTT.h | 2 +- extensions/mqtt/processors/ConvertBase.h | 2 +- extensions/mqtt/processors/ConvertHeartBeat.h| 2 +- extensions/mqtt/processors/ConvertJSONAck.h | 2 +- extensions/mqtt/processors/ConvertUpdate.h | 2 +- extensions/mqtt/processors/PublishMQTT.h | 2 +- extensions/mqtt/protocol/MQTTC2Protocol.cpp | 2 +- extensions/mqtt/protocol/MQTTC2Protocol.h| 2 +- extensions/opc/include/fetchopc.h| 2 +- extensions/opc/include/opcbase.h | 2 +- extensions/opc/include/putopc.h | 2 +- extensions/opencv/CaptureRTSPFrame.h | 2 +- extensions/opencv/MotionDetector.h | 2 +- .../processors/SourceInitiatedSubscriptionListener.cpp | 2 +- .../processors/SourceInitiatedSubscriptionListener.h | 2 +- extensions/pcap/CapturePacket.h | 2 +- extensions/rocksdb-repos/DatabaseContentRepository.h | 2 +- extensions/rocksdb-repos/FlowFileRepository.h| 2 +- extensions/rocksdb-repos/ProvenanceRepository.h | 2 +- .../RocksDbPersistableKeyValueStoreService.cpp | 2 +- .../controllers/RocksDbPersistableKeyValueStoreService.h | 2 +- extensions/script/ExecuteScript.h| 2 +- extensions/script/python/ExecutePythonProcessor.h| 2 +- extensions/script/python/PyProcCreator.h | 2 +- extensions/script/python/PythonCreator.h | 2 +- extensions/sensors/GetEnvironmentalSensors.h | 2 +- extensions/sensors/GetMovementSensors.h | 2 +- extensions/sensors/SensorBase.h | 2 +- extensions/sftp/processors/FetchSFTP.cpp | 2 +- extensions/sftp/processors/FetchSFTP.h | 2 +- extensions/sftp/processors/ListSFTP.cpp | 2 +- extensions/sftp/processors/ListSFTP.h| 2 +- extensions/sftp/processors/PutSFTP.cpp | 2 +- extensions/sftp/processors/PutSFTP.h | 2 +- extensions/sftp/processors/SFTPProcessorBase.cpp | 2 +- extensions/sftp/processors/SFTPProcessorBase.h | 2 +- extensions/sql/processors/ExecuteSQL.cpp | 2 +- extensions/sql/processors/ExecuteSQL.h | 2 +- extensions/sql/processors/PutSQL.cpp | 2 +- extensions/sql/processors/PutSQL.h
[nifi-minifi-cpp] branch main updated (828fc38 -> d4cda02)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 828fc38 MINIFICPP-1510 Register and fix InvokeHTTPTests add 3f5e509 MINIFICPP-1544 Add Custom Text property to GenerateFlowFile processor add d4cda02 MINIFICPP-1504: Add Resource consumption data to heartbeats No new revisions were added by this update. Summary of changes: PROCESSORS.md | 1 + .../C2VerifyResourceConsumptionInHeartbeat.cpp | 133 +++ extensions/http-curl/tests/CMakeLists.txt | 1 + .../processors/GenerateFlowFile.cpp| 19 +++ .../processors/GenerateFlowFile.h | 1 + .../standard-processors/tests/CMakeLists.txt | 3 +- .../tests/unit/GenerateFlowFileTests.cpp | 129 +- libminifi/include/agent/build_description.h| 4 + libminifi/include/core/PropertyValue.h | 2 + libminifi/include/core/state/Value.h | 172 +-- .../include/core/state/nodes/AgentInformation.h| 142 ++- .../include/core/state/nodes/DeviceInformation.h | 165 +- libminifi/include/utils/OsUtils.h | 13 +- libminifi/include/utils/ProcessCPUUsageTracker.h | 92 ++ libminifi/include/utils/SystemCPUUsageTracker.h| 128 ++ .../include/utils/{MapUtils.h => ValueCaster.h}| 28 +-- libminifi/include/utils/ValueParser.h | 31 libminifi/src/c2/protocols/RESTProtocol.cpp| 4 + libminifi/src/core/state/Value.cpp | 1 + .../nodes/AgentInformation.cpp}| 13 +- .../src/core/state/nodes/DeviceInformation.cpp | 19 +-- libminifi/src/utils/OsUtils.cpp| 143 +--- libminifi/src/utils/ProcessCPUUsageTracker.cpp | 140 +++ libminifi/src/utils/SystemCPUUsageTracker.cpp | 190 + libminifi/test/unit/CPUUsageTest.cpp | 90 ++ libminifi/test/unit/MemoryUsageTest.cpp| 27 ++- libminifi/test/unit/ResponseNodeValueTests.cpp | 187 27 files changed, 1634 insertions(+), 244 deletions(-) create mode 100644 extensions/http-curl/tests/C2VerifyResourceConsumptionInHeartbeat.cpp create mode 100644 libminifi/include/utils/ProcessCPUUsageTracker.h create mode 100644 libminifi/include/utils/SystemCPUUsageTracker.h copy libminifi/include/utils/{MapUtils.h => ValueCaster.h} (66%) copy libminifi/src/core/{TypedValues.cpp => state/nodes/AgentInformation.cpp} (79%) copy extensions/jni/jvm/JniReferenceObjects.cpp => libminifi/src/core/state/nodes/DeviceInformation.cpp (79%) create mode 100644 libminifi/src/utils/ProcessCPUUsageTracker.cpp create mode 100644 libminifi/src/utils/SystemCPUUsageTracker.cpp create mode 100644 libminifi/test/unit/CPUUsageTest.cpp create mode 100644 libminifi/test/unit/ResponseNodeValueTests.cpp
[nifi-minifi-cpp] branch main updated (016e034 -> 6bc4b90)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 016e034 MINIFICPP-1345 Add flake8 check for python files new 870249f MINIFICPP-1032 Refactor parse_url new 6bc4b90 MINIFICPP-1547 - Change default c2 protocol The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: C2.md | 8 +- bootstrap.sh | 2 +- conf/minifi.properties | 5 +- encrypt-config/tests/resources/minifi.properties | 6 +- ...th-additional-sensitive-props.minifi.properties | 8 +- libminifi/include/RemoteProcessorGroupPort.h | 22 ++- libminifi/include/utils/HTTPClient.h | 24 +++- libminifi/src/c2/C2Agent.cpp | 24 ++-- libminifi/src/utils/HTTPClient.cpp | 155 ++--- .../test/resources/encrypted.minifi.properties | 6 +- libminifi/test/unit/HTTPUtilTests.cpp | 97 - 11 files changed, 220 insertions(+), 137 deletions(-)
[nifi-minifi-cpp] 01/02: MINIFICPP-1032 Refactor parse_url
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 870249f713219cce57f0719efa3e14c6e25e15bf Author: Ferenc Gerlits AuthorDate: Tue Apr 6 12:51:29 2021 +0200 MINIFICPP-1032 Refactor parse_url Modernize the parse_url() function and add error checking. Signed-off-by: Arpad Boda This closes #1046 --- libminifi/include/RemoteProcessorGroupPort.h | 22 ++-- libminifi/include/utils/HTTPClient.h | 24 - libminifi/src/c2/C2Agent.cpp | 16 ++- libminifi/src/utils/HTTPClient.cpp | 155 ++- libminifi/test/unit/HTTPUtilTests.cpp| 97 ++--- 5 files changed, 198 insertions(+), 116 deletions(-) diff --git a/libminifi/include/RemoteProcessorGroupPort.h b/libminifi/include/RemoteProcessorGroupPort.h index 3c33706..8598103 100644 --- a/libminifi/include/RemoteProcessorGroupPort.h +++ b/libminifi/include/RemoteProcessorGroupPort.h @@ -146,23 +146,17 @@ class RemoteProcessorGroupPort : public core::Processor { */ void setURL(std::string val) { auto urls = utils::StringUtils::split(val, ","); -for (auto url : urls) { - logger_->log_trace("Parsing %s", url); - std::string host, protocol; - int port = -1; - url = utils::StringUtils::trim(url); - utils::parse_url(&url, &host, &port, &protocol); - logger_->log_trace("Parsed -%s- %s %s, %d", url, protocol, host, port); - if (port == -1) { -if (protocol.find("https") != std::string::npos) { - port = 443; -} else if (protocol.find("http") != std::string::npos) { - port = 80; -} +for (const auto& url : urls) { + utils::URL parsed_url{utils::StringUtils::trim(url)}; + if (parsed_url.isValid()) { +logger_->log_debug("Parsed RPG URL '%s' -> '%s'", url, parsed_url.hostPort()); +nifi_instances_.push_back({parsed_url.host(), parsed_url.port(), parsed_url.protocol()}); + } else { +logger_->log_error("Could not parse RPG URL '%s'", url); } - nifi_instances_.push_back({ host, port, protocol }); } } + void setHTTPProxy(const utils::HTTPProxy &proxy) { this->proxy_ = proxy; } diff --git a/libminifi/include/utils/HTTPClient.h b/libminifi/include/utils/HTTPClient.h index 82aa55b..425c716 100644 --- a/libminifi/include/utils/HTTPClient.h +++ b/libminifi/include/utils/HTTPClient.h @@ -36,6 +36,7 @@ #include "controllers/SSLContextService.h" #include "core/Deprecated.h" #include "utils/gsl.h" +#include "utils/OptionalUtils.h" namespace org { namespace apache { @@ -362,10 +363,27 @@ class BaseHTTPClient { virtual inline bool matches(const std::string &value, const std::string &sregex) = 0; }; -extern std::string get_token(utils::BaseHTTPClient *client, std::string username, std::string password); +std::string get_token(utils::BaseHTTPClient *client, std::string username, std::string password); + +class URL { + public: + explicit URL(const std::string& url_input); + bool isValid() const { return is_valid_; } + std::string protocol() const { return protocol_; } + std::string host() const { return host_; } + int port() const; + std::string hostPort() const; + std::string toString() const; + + private: + std::string protocol_; + std::string host_; + utils::optional port_; + utils::optional path_; + bool is_valid_ = false; + std::shared_ptr logger_ = logging::LoggerFactory::getLogger(); +}; -extern void parse_url(const std::string *url, std::string *host, int *port, std::string *protocol); -extern void parse_url(const std::string *url, std::string *host, int *port, std::string *protocol, std::string *path, std::string *query); } // namespace utils } // namespace minifi } // namespace nifi diff --git a/libminifi/src/c2/C2Agent.cpp b/libminifi/src/c2/C2Agent.cpp index 0914ed5..9c914cc 100644 --- a/libminifi/src/c2/C2Agent.cpp +++ b/libminifi/src/c2/C2Agent.cpp @@ -760,22 +760,20 @@ utils::optional C2Agent::fetchFlow(const std::string& uri) const { std::stringstream adjusted_url; std::string base; if (configuration_->get(minifi::Configure::nifi_c2_flow_base_url, base)) { + base = utils::StringUtils::trim(base); adjusted_url << base; if (!utils::StringUtils::endsWith(base, "/")) { adjusted_url << "/"; } adjusted_url << uri; resolved_url = adjusted_url.str(); -} else if (configuration_->get("c2.rest.url", base)) { - std::string host, protocol; - int port = -1; - utils::parse_url(&base, &host, &port,
[nifi-minifi-cpp] 02/02: MINIFICPP-1547 - Change default c2 protocol
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 6bc4b9057ee841e170bf5087bb86ac379e82a0b8 Author: Adam Debreceni AuthorDate: Wed Apr 21 15:49:30 2021 +0200 MINIFICPP-1547 - Change default c2 protocol Signed-off-by: Arpad Boda This closes #1058 --- C2.md | 8 bootstrap.sh | 2 +- conf/minifi.properties| 5 +++-- encrypt-config/tests/resources/minifi.properties | 6 +++--- .../resources/with-additional-sensitive-props.minifi.properties | 8 libminifi/src/c2/C2Agent.cpp | 8 libminifi/test/resources/encrypted.minifi.properties | 6 +++--- 7 files changed, 22 insertions(+), 21 deletions(-) diff --git a/C2.md b/C2.md index 560b8ea..864e5202 100644 --- a/C2.md +++ b/C2.md @@ -63,11 +63,11 @@ be requested via C2 DESCRIBE manifest command. # specify classes for the AST response nifi.c2.root.classes=DeviceInfoNode,AgentInformation,FlowInformation - # specify C2 protocol -- default is CoapProtocol if not specified - #nifi.c2.agent.protocol.class=CoapProtocol - # may also use MQTT or REST - # nifi.c2.agent.protocol.class=MQTTC2Protocol + # specify C2 protocol -- default is RESTSender if not specified nifi.c2.agent.protocol.class=RESTSender + # may also use MQTT or CoapProtocol + # nifi.c2.agent.protocol.class=MQTTC2Protocol + # nifi.c2.agent.protocol.class=CoapProtocol # control c2 heartbeat interval in millisecocnds nifi.c2.agent.heartbeat.period=250 diff --git a/bootstrap.sh b/bootstrap.sh index 2401996..34fa8d1 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -292,7 +292,7 @@ add_disabled_option MQTT_ENABLED ${FALSE} "ENABLE_MQTT" add_disabled_option PYTHON_ENABLED ${FALSE} "ENABLE_PYTHON" add_dependency PYTHON_ENABLED "python" -add_disabled_option COAP_ENABLED ${TRUE} "ENABLE_COAP" +add_disabled_option COAP_ENABLED ${FALSE} "ENABLE_COAP" add_dependency COAP_ENABLED "automake" add_dependency COAP_ENABLED "autoconf" add_dependency COAP_ENABLED "libtool" diff --git a/conf/minifi.properties b/conf/minifi.properties index e5f2a1a..66dfbe9 100644 --- a/conf/minifi.properties +++ b/conf/minifi.properties @@ -65,8 +65,9 @@ nifi.content.repository.class.name=DatabaseContentRepository ## define those with missing options #nifi.c2.enable=true ## define protocol parameters -## The default is CoAP, if that extension is built. -## Alternatively, you may use RESTSender if http-curl is built +## The default is RESTSender. +## Alternatively, you may use CoapProtocol if that extension is built. +#nifi.c2.agent.protocol.class=RESTSender #nifi.c2.agent.protocol.class=CoapProtocol #nifi.c2.agent.coap.host= #nifi.c2.agent.coap.port= diff --git a/encrypt-config/tests/resources/minifi.properties b/encrypt-config/tests/resources/minifi.properties index f1ca0a2..1cf2549 100644 --- a/encrypt-config/tests/resources/minifi.properties +++ b/encrypt-config/tests/resources/minifi.properties @@ -52,10 +52,10 @@ nifi.rest.api.password=password ## define those with missing options nifi.c2.enable=true ## define protocol parameters -## The default is CoAP, if that extension is built. -## Alternatively, you may use RESTSender if http-curl is built -#nifi.c2.agent.protocol.class=CoapProtocol +## The default is RESTSender. +## Alternatively, you may use CoapProtocol if that extension is built. nifi.c2.agent.protocol.class=RESTSender +#nifi.c2.agent.protocol.class=CoapProtocol #nifi.c2.agent.coap.host= #nifi.c2.agent.coap.port= ## base URL of the c2 server, diff --git a/encrypt-config/tests/resources/with-additional-sensitive-props.minifi.properties b/encrypt-config/tests/resources/with-additional-sensitive-props.minifi.properties index d021f0f..2ca29b9 100644 --- a/encrypt-config/tests/resources/with-additional-sensitive-props.minifi.properties +++ b/encrypt-config/tests/resources/with-additional-sensitive-props.minifi.properties @@ -29,7 +29,7 @@ nifi.database.content.repository.directory.default=${MINIFI_HOME}/content_reposi nifi.remote.input.secure=true nifi.security.need.ClientAuth= -nifi.security.client.certificate= +nifi.security.client.certificate= nifi.security.client.private.key= nifi.security.client.pass.phrase=correct_horse_battery_staple nifi.security.client.ca.certificate= @@ -54,10 +54,10 @@ nifi.rest.api.password=password ## define those with missing options nifi.c2.enable=true ## define protocol parameters -## The default is CoAP, if that extension is built. -## Alternatively, you may use RESTSender if h
[nifi-minifi-cpp] 03/03: MINIFICPP-1499 - Add clcache to windows builds
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 2c51f1a53bb225b5f5e05496e339f041fd0237ef Author: Gabor Gyimesi AuthorDate: Wed Feb 17 17:39:46 2021 +0100 MINIFICPP-1499 - Add clcache to windows builds Signed-off-by: Arpad Boda This closes #1012 --- .github/workflows/ci.yml | 40 +++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4c17288..a9d6197 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -69,9 +69,19 @@ jobs: name: "windows-vs2017" runs-on: windows-2016 timeout-minutes: 120 +env: + CLCACHE_DIR: ${{ GITHUB.WORKSPACE }}\clcache steps: - id: checkout uses: actions/checkout@v2 + - id: cache +uses: actions/cache@v2 +with: + path: ${{ env.CLCACHE_DIR }} + key: windows-vs2017-clcache-${{github.ref}}-${{github.sha}} + restore-keys: | +windows-vs2017-clcache-${{github.ref}}- +windows-vs2017-clcache-refs/heads/main- - name: Setup PATH uses: microsoft/setup-msbuild@v1.0.2 - id: install-sqliteodbc-driver @@ -79,6 +89,15 @@ jobs: Invoke-WebRequest -Uri "http://www.ch-werner.de/sqliteodbc/sqliteodbc.exe"; -OutFile "sqliteodbc.exe" ./sqliteodbc.exe /S shell: powershell + - name: Setup clcache +run: | + (New-Object System.Net.WebClient).DownloadFile('https://github.com/frerich/clcache/releases/download/v4.2.0/clcache-4.2.0.zip', "$pwd\clcache.zip") + $cl_exe_dir = Split-Path -parent $(vswhere -latest -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -find VC\Tools\MSVC\**\bin\Hostx86\x86\cl.exe | select-object -last 1) + Expand-Archive -Force -Path clcache.zip -DestinationPath "$cl_exe_dir"; + move "$cl_exe_dir\cl.exe" "$cl_exe_dir\cl_original.exe" + move "$cl_exe_dir\cl.exe.config" "$cl_exe_dir\cl_original.exe.config" + move "$cl_exe_dir\clcache.exe" "$cl_exe_dir\cl.exe" + echo "CLCACHE_CL=$cl_exe_dir\cl_original.exe" >> $env:GITHUB_ENV - id: build run: | PATH %PATH%;C:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x86 @@ -88,10 +107,20 @@ jobs: windows_VS2019: name: "windows-vs2019" runs-on: windows-2019 -timeout-minutes: 90 +timeout-minutes: 120 +env: + CLCACHE_DIR: ${{ GITHUB.WORKSPACE }}\clcache steps: - id: checkout uses: actions/checkout@v2 + - id: cache +uses: actions/cache@v2 +with: + path: ${{ env.CLCACHE_DIR }} + key: windows-vs2019-clcache-${{github.ref}}-${{github.sha}} + restore-keys: | +windows-vs2019-clcache-${{github.ref}}- +windows-vs2019-clcache-refs/heads/main- - name: Setup PATH uses: microsoft/setup-msbuild@v1.0.2 - id: install-sqliteodbc-driver @@ -99,6 +128,15 @@ jobs: Invoke-WebRequest -Uri "http://www.ch-werner.de/sqliteodbc/sqliteodbc_w64.exe"; -OutFile "sqliteodbc_w64.exe" ./sqliteodbc_w64.exe /S shell: powershell + - name: Setup clcache +run: | + (New-Object System.Net.WebClient).DownloadFile('https://github.com/frerich/clcache/releases/download/v4.2.0/clcache-4.2.0.zip', "$pwd\clcache.zip") + $cl_exe_dir = Split-Path -parent $(vswhere -latest -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -find VC\Tools\MSVC\**\bin\Hostx64\x64\cl.exe | select-object -last 1) + Expand-Archive -Force -Path clcache.zip -DestinationPath "$cl_exe_dir"; + move "$cl_exe_dir\cl.exe" "$cl_exe_dir\cl_original.exe" + move "$cl_exe_dir\cl.exe.config" "$cl_exe_dir\cl_original.exe.config" + move "$cl_exe_dir\clcache.exe" "$cl_exe_dir\cl.exe" + echo "CLCACHE_CL=$cl_exe_dir\cl_original.exe" >> $env:GITHUB_ENV - id: build run: | PATH %PATH%;C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64
[nifi-minifi-cpp] branch main updated (a29b414 -> 2c51f1a)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from a29b414 MINIFICPP-1535 - Support process groups new c759c96 MINIFICPP-1539 Check if Message Key Field is really set new 88f5e04 MINIFICPP-1543 Add 'brew update' to MacOS CI jobs new 2c51f1a MINIFICPP-1499 - Add clcache to windows builds The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/ci.yml | 48 +-- extensions/librdkafka/PublishKafka.cpp| 2 +- extensions/librdkafka/tests/PublishKafkaTests.cpp | 16 3 files changed, 62 insertions(+), 4 deletions(-)
[nifi-minifi-cpp] 01/03: MINIFICPP-1539 Check if Message Key Field is really set
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit c759c96c9ab6f59a98a624dd99a9540d73d8d6b4 Author: Ferenc Gerlits AuthorDate: Fri Apr 9 18:04:47 2021 +0200 MINIFICPP-1539 Check if Message Key Field is really set Signed-off-by: Arpad Boda This closes #1049 --- extensions/librdkafka/PublishKafka.cpp| 2 +- extensions/librdkafka/tests/PublishKafkaTests.cpp | 16 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/extensions/librdkafka/PublishKafka.cpp b/extensions/librdkafka/PublishKafka.cpp index ceb76f3..b83d028 100644 --- a/extensions/librdkafka/PublishKafka.cpp +++ b/extensions/librdkafka/PublishKafka.cpp @@ -544,7 +544,7 @@ void PublishKafka::onSchedule(const std::shared_ptr &conte configureNewConnection(context); std::string message_key_field; - if (context->getProperty(MessageKeyField.getName(), message_key_field)) { + if (context->getProperty(MessageKeyField.getName(), message_key_field) && !message_key_field.empty()) { logger_->log_error("The %s property is set. This property is DEPRECATED and has no effect; please use Kafka Key instead.", MessageKeyField.getName()); } diff --git a/extensions/librdkafka/tests/PublishKafkaTests.cpp b/extensions/librdkafka/tests/PublishKafkaTests.cpp index ea1c4ef..db44eb3 100644 --- a/extensions/librdkafka/tests/PublishKafkaTests.cpp +++ b/extensions/librdkafka/tests/PublishKafkaTests.cpp @@ -190,3 +190,19 @@ TEST_CASE("PublishKafka dynamic properties can use expression language", "[Publi REQUIRE(LogTestController::getInstance().contains("PublishKafka: DynamicProperty: [retry.backoff.ms] -> [18]")); } + +TEST_CASE("PublishKafka complains if Message Key Field is set, but only if it is set", "[PublishKafka][properties]") { + PublishKafkaTestRunner test_runner; + + SECTION("Message Key Field is not set, so there is no error") { +test_runner.runProcessor(); +REQUIRE_FALSE(LogTestController::getInstance().contains("error")); + } + + SECTION("Message Key Field is set, so there is an error log") { + test_runner.setPublishKafkaProperty(processors::PublishKafka::MessageKeyField, "kafka.key"); +test_runner.runProcessor(); +REQUIRE(LogTestController::getInstance().contains("The " + processors::PublishKafka::MessageKeyField.getName() + +" property is set. This property is DEPRECATED and has no effect")); + } +}
[nifi-minifi-cpp] 02/03: MINIFICPP-1543 Add 'brew update' to MacOS CI jobs
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 88f5e04a20606f3e054582f637917d68873bcd43 Author: Ferenc Gerlits AuthorDate: Mon Apr 12 19:02:35 2021 +0200 MINIFICPP-1543 Add 'brew update' to MacOS CI jobs ... and remove packages which are already installed, because these cause `brew install` to return with a non-zero status, for some reason. Signed-off-by: Arpad Boda This closes #1051 --- .github/workflows/ci.yml | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 306ec15..4c17288 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -20,7 +20,9 @@ jobs: macos-xcode11.2.1-ccache-${{github.ref}}- macos-xcode11.2.1-ccache-refs/heads/main- - id: install_dependencies -run: brew install ossp-uuid boost flex openssl python lua@5.3 xz libssh2 ccache sqliteodbc +run: | + brew update + brew install ossp-uuid boost flex lua@5.3 ccache sqliteodbc - id: setup_env run: | echo "PATH=/usr/lib/ccache:/usr/local/opt/ccache/bin:/usr/local/opt/ccache/libexec:$PATH" >> $GITHUB_ENV @@ -50,7 +52,9 @@ jobs: macos-xcode12.0-ccache-${{github.ref}}- macos-xcode12.0-ccache-refs/heads/main- - id: install_dependencies -run: brew install ossp-uuid boost flex openssl python lua@5.3 xz libssh2 ccache sqliteodbc +run: | + brew update + brew install ossp-uuid boost flex lua@5.3 ccache sqliteodbc - id: setup_env run: | echo "PATH=/usr/lib/ccache:/usr/local/opt/ccache/bin:/usr/local/opt/ccache/libexec:$PATH" >> $GITHUB_ENV
[nifi] branch main updated (a44ab0d -> 1497a6e)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git. from a44ab0d NIFI-8397 update to simple-syslog-5424 0.0.16 add 1497a6e Update KEYS No new revisions were added by this update. Summary of changes: KEYS | 129 +++ 1 file changed, 129 insertions(+)
[nifi-minifi-cpp] branch main updated (0a3d5df -> 2b27556)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 0a3d5df MINIFICPP-1533 Update list of dependencies in README file. add 2b27556 MINIFICPP-1520 - Fix PublishKafka properties to support expression language No new revisions were added by this update. Summary of changes: PROCESSORS.md | 8 +- extensions/librdkafka/PublishKafka.cpp| 55 --- extensions/librdkafka/PublishKafka.h | 3 +- extensions/librdkafka/tests/CMakeLists.txt| 3 +- extensions/librdkafka/tests/PublishKafkaTests.cpp | 192 ++ 5 files changed, 232 insertions(+), 29 deletions(-) create mode 100644 extensions/librdkafka/tests/PublishKafkaTests.cpp
[nifi-minifi-cpp] branch main updated: MINIFICPP-1531 Use release build type for Ubuntu 20.04 job
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 4bb1fd9 MINIFICPP-1531 Use release build type for Ubuntu 20.04 job 4bb1fd9 is described below commit 4bb1fd97f5df103d5bee9a884839b24e2f77a2c5 Author: Gabor Gyimesi AuthorDate: Thu Mar 18 18:20:23 2021 +0100 MINIFICPP-1531 Use release build type for Ubuntu 20.04 job Fix exception catch warning in ListS3Tests Signed-off-by: Arpad Boda This closes #1034 --- .github/workflows/ci.yml | 4 ++-- controller/Controller.h| 2 +- extensions/http-curl/tests/HTTPHandlers.h | 2 ++ libminifi/test/aws-tests/ListS3Tests.cpp | 2 +- libminifi/test/rocksdb-tests/RepoTests.cpp | 1 + nanofi/include/sitetosite/CPeer.h | 3 +-- 6 files changed, 8 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 40f2cda..89ab136 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -159,7 +159,7 @@ jobs: sudo apt install -y ccache libfl-dev libpcap-dev libboost-all-dev echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build -run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DENABLE_BUSTACHE=ON -DENABLE_SQLITE=ON -DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && make -j4 VERBOSE=1 && make test ARGS="--timeout 300 -j2 --output-on-failure" +run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DCMAKE_BUILD_TYPE=Release -DENABLE_BUSTACHE=ON -DENABLE_SQLITE=ON -DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && make -j4 VERBOSE=1 && make test ARGS="--timeout 300 -j2 --output-on-failure" ubuntu_20_04_all_clang: name: "ubuntu-20.04-all-clang" runs-on: ubuntu-20.04 @@ -181,7 +181,7 @@ jobs: sudo apt install -y ccache libfl-dev libpcap-dev libboost-all-dev openjdk-8-jdk maven libusb-1.0-0-dev libpng-dev libgps-dev echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build -run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DENABLE_OPENWSMAN=ON -DENABLE_OPENCV=ON -DENABLE_MQTT=ON -DENABLE_GPS=ON -DENABLE_USB_CAMERA=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_OPC=ON -DENABLE_SFTP=ON -DENABLE_MQTT=ON -DENABLE_COAP=ON -DENABLE_PYTHON=ON -DENABLE_SQL=ON -DENABLE_AWS=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .. && cmake --build . --parallel 4 && make test ARGS="--timeout 300 -j8 --o [...] +run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DCMAKE_BUILD_TYPE=Release -DENABLE_OPENWSMAN=ON -DENABLE_OPENCV=ON -DENABLE_MQTT=ON -DENABLE_GPS=ON -DENABLE_USB_CAMERA=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_OPC=ON -DENABLE_SFTP=ON -DENABLE_MQTT=ON -DENABLE_COAP=ON -DENABLE_PYTHON=ON -DENABLE_SQL=ON -DENABLE_AWS=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .. && cmake --build . --parallel 4 && make test [...] ubuntu_16_04_all: name: "ubuntu-16.04-all" runs-on: ubuntu-16.04 diff --git a/controller/Controller.h b/controller/Controller.h index 6c1f77c..0544f50 100644 --- a/controller/Controller.h +++ b/controller/Controller.h @@ -146,7 +146,7 @@ int getJstacks(std::unique_ptr socket, std::ostream &out) { for (uint64_t i = 0; i < size; i++) { std::string name; - uint64_t lines; + uint64_t lines = 0; socket->read(name); socket->read(lines); for (uint64_t j = 0; j < lines; j++) { diff --git a/extensions/http-curl/tests/HTTPHandlers.h b/extensions/http-curl/tests/HTTPHandlers.h index af9b0a8..76015e4 100644 --- a/extensions/http-curl/tests/HTTPHandlers.h +++ b/extensions/http-curl/tests/HTTPHandlers.h @@ -416,6 +416,7 @@ class HeartbeatHandler : public ServerAwareHandler { } } assert(found); +(void)found; // unused in release builds } virtual void handleHeartbeat(const rapidjson::Document& root, struct mg_connection *) { @@ -431,6 +432,7 @@ class HeartbeatHandler : public ServerAwareHandler { rapidjson::Document root; rapidjson::ParseResult ok = root.Parse(post_data.data(), post_data.size()); assert(ok); + (void)ok; // unused in release builds std::string operation = root["operation"].GetString(); if (operation == "heartbeat") { handleHeartbeat(root, conn); diff --git a/libminifi/test/aws-tests/ListS3Tests.cpp b/libminifi/test/aws-tests/List
[nifi-minifi-cpp] branch main updated: MINIFICPP-1438 Fix transient failures of FlowControllerTests
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 9427bce MINIFICPP-1438 Fix transient failures of FlowControllerTests 9427bce is described below commit 9427bceeb49d6b2000c9e3ff68f0c72ef4b92f09 Author: Gabor Gyimesi AuthorDate: Thu Mar 11 10:58:01 2021 +0100 MINIFICPP-1438 Fix transient failures of FlowControllerTests Signed-off-by: Arpad Boda This closes #1030 --- libminifi/test/flow-tests/FlowControllerTests.cpp | 31 --- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/libminifi/test/flow-tests/FlowControllerTests.cpp b/libminifi/test/flow-tests/FlowControllerTests.cpp index acc7ee8..3e0b5d1 100644 --- a/libminifi/test/flow-tests/FlowControllerTests.cpp +++ b/libminifi/test/flow-tests/FlowControllerTests.cpp @@ -154,16 +154,16 @@ TEST_CASE("Flow shutdown waits for a while", "[TestFlow2]") { testController.startFlow(); // wait for the source processor to enqueue its flowFiles - auto flowFilesEnqueued = [&] {return root->getTotalFlowFileCount() == 3;}; - REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{200}, flowFilesEnqueued)); + auto flowFilesEnqueued = [&] { return root->getTotalFlowFileCount() >= 3; }; + REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{500}, flowFilesEnqueued)); - REQUIRE(sourceProc->trigger_count.load() == 1); + REQUIRE(sourceProc->trigger_count.load() >= 1); execSinkPromise.set_value(); controller->stop(); - REQUIRE(sourceProc->trigger_count.load() == 1); - REQUIRE(sinkProc->trigger_count.load() == 3); + REQUIRE(sourceProc->trigger_count.load() >= 1); + REQUIRE(sinkProc->trigger_count.load() >= 3); } TEST_CASE("Flow stopped after grace period", "[TestFlow3]") { @@ -191,16 +191,16 @@ TEST_CASE("Flow stopped after grace period", "[TestFlow3]") { testController.startFlow(); // wait for the source processor to enqueue its flowFiles - auto flowFilesEnqueued = [&] {return root->getTotalFlowFileCount() == 3;}; - REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{200}, flowFilesEnqueued)); + auto flowFilesEnqueued = [&] { return root->getTotalFlowFileCount() >= 3; }; + REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{500}, flowFilesEnqueued)); - REQUIRE(sourceProc->trigger_count.load() == 1); + REQUIRE(sourceProc->trigger_count.load() >= 1); execSinkPromise.set_value(); controller->stop(); - REQUIRE(sourceProc->trigger_count.load() == 1); - REQUIRE(sinkProc->trigger_count.load() == 1); + REQUIRE(sourceProc->trigger_count.load() >= 1); + REQUIRE(sinkProc->trigger_count.load() >= 1); } TEST_CASE("Extend the waiting period during shutdown", "[TestFlow4]") { @@ -230,10 +230,11 @@ TEST_CASE("Extend the waiting period during shutdown", "[TestFlow4]") { testController.startFlow(); // wait for the source processor to enqueue its flowFiles - auto flowFilesEnqueued = [&] {return root->getTotalFlowFileCount() == 3;}; - REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{200}, flowFilesEnqueued)); + std::size_t flow_file_count = 0; + auto flowFilesEnqueued = [&] { return root->getTotalFlowFileCount() >= 3; }; + REQUIRE(verifyWithBusyWait(std::chrono::milliseconds{500}, flowFilesEnqueued)); - REQUIRE(sourceProc->trigger_count.load() == 1); + REQUIRE(sourceProc->trigger_count.load() >= 1); std::thread shutdownThread([&]{ execSinkPromise.set_value(); @@ -257,6 +258,6 @@ TEST_CASE("Extend the waiting period during shutdown", "[TestFlow4]") { shutdownThread.join(); - REQUIRE(sourceProc->trigger_count.load() == 1); - REQUIRE(sinkProc->trigger_count.load() == 3); + REQUIRE(sourceProc->trigger_count.load() >= 1); + REQUIRE(sinkProc->trigger_count.load() >= 3); }
[nifi-minifi-cpp] branch main updated: MINIFICPP-1203 - Make cpplint ignore bison generated expression language code
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 1b44f74 MINIFICPP-1203 - Make cpplint ignore bison generated expression language code 1b44f74 is described below commit 1b44f74c956561210321bd9c9c352d87539ce026 Author: Adam Hunyadi AuthorDate: Fri Feb 19 09:25:30 2021 +0100 MINIFICPP-1203 - Make cpplint ignore bison generated expression language code Signed-off-by: Arpad Boda This closes #1010 --- extensions/expression-language/CPPLINT.cfg | 6 ++ 1 file changed, 6 insertions(+) diff --git a/extensions/expression-language/CPPLINT.cfg b/extensions/expression-language/CPPLINT.cfg new file mode 100644 index 000..344ec8c --- /dev/null +++ b/extensions/expression-language/CPPLINT.cfg @@ -0,0 +1,6 @@ +exclude_files=Scanner.cpp +exclude_files=Parser.cpp +exclude_files=Parser.hpp +exclude_files=location.hh +exclude_files=position.hh +exclude_files=stack.hh
[nifi-minifi-cpp] branch main updated: MINIFICPP-1527 Add update before install to avoid fetch errors
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new c4b29af MINIFICPP-1527 Add update before install to avoid fetch errors c4b29af is described below commit c4b29af8a528125c01b14f29106284c89fbd4ce3 Author: Gabor Gyimesi AuthorDate: Mon Mar 8 18:32:26 2021 +0100 MINIFICPP-1527 Add update before install to avoid fetch errors Signed-off-by: Arpad Boda This closes #1024 --- .github/workflows/ci.yml | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b894a30..77b105c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -108,6 +108,7 @@ jobs: ubuntu-16.04-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -154,6 +155,7 @@ jobs: ubuntu-20.04-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache libfl-dev libpcap-dev libboost-all-dev echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -198,6 +200,7 @@ jobs: debian-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -219,6 +222,7 @@ jobs: centos-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -240,6 +244,7 @@ jobs: fedora-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -261,6 +266,7 @@ jobs: ubuntu-18.04-ccache-refs/heads/main - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -282,6 +288,7 @@ jobs: ubuntu-16.04-shared-ccache-refs/heads/main- - id: install_deps run: | + sudo apt update sudo apt install -y ccache echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build @@ -296,6 +303,8 @@ jobs: - id: build run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_JNI=OFF -DDISABLE_JEMALLOC=ON -DENABLE_AWS=ON -DENABLE_LIBRDKAFKA=ON .. && make docker - id: install_deps -run: sudo apt install -y python3-virtualenv +run: | + sudo apt update + sudo apt install -y python3-virtualenv - id: test run: cd build && make docker-verify
[nifi-minifi-cpp] branch main updated: MINIFICPP-1509 - Use _stat64 on windows
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new f9279bc MINIFICPP-1509 - Use _stat64 on windows f9279bc is described below commit f9279bc47caab16de282e7f22e0045dd0df85e39 Author: Adam Debreceni AuthorDate: Wed Feb 24 14:26:54 2021 +0100 MINIFICPP-1509 - Use _stat64 on windows Signed-off-by: Arpad Boda This closes #1015 --- .../standard-processors/processors/GetFile.cpp | 59 -- .../standard-processors/processors/PutFile.cpp | 10 +--- libminifi/include/utils/file/FileUtils.h | 38 +++--- 3 files changed, 56 insertions(+), 51 deletions(-) diff --git a/extensions/standard-processors/processors/GetFile.cpp b/extensions/standard-processors/processors/GetFile.cpp index 5915866..98f2dd5 100644 --- a/extensions/standard-processors/processors/GetFile.cpp +++ b/extensions/standard-processors/processors/GetFile.cpp @@ -228,42 +228,49 @@ void GetFile::pollListing(std::queue &list, const GetFileRequest &r bool GetFile::acceptFile(std::string fullName, std::string name, const GetFileRequest &request) { logger_->log_trace("Checking file: %s", fullName); +#ifdef WIN32 + struct _stat64 statbuf; + if (_stat64(fullName.c_str(), &statbuf) != 0) { +return false; + } +#else struct stat statbuf; + if (stat(fullName.c_str(), &statbuf) != 0) { +return false; + } +#endif + uint64_t file_size = gsl::narrow(statbuf.st_size); + uint64_t modifiedTime = gsl::narrow(statbuf.st_mtime) * 1000; - if (stat(fullName.c_str(), &statbuf) == 0) { -if (request.minSize > 0 && statbuf.st_size < (int32_t) request.minSize) - return false; - -if (request.maxSize > 0 && statbuf.st_size > (int32_t) request.maxSize) - return false; + if (request.minSize > 0 && file_size < request.minSize) +return false; -uint64_t modifiedTime = ((uint64_t) (statbuf.st_mtime) * 1000); -uint64_t fileAge = utils::timeutils::getTimeMillis() - modifiedTime; -if (request.minAge > 0 && fileAge < request.minAge) - return false; -if (request.maxAge > 0 && fileAge > request.maxAge) - return false; + if (request.maxSize > 0 && file_size > request.maxSize) +return false; -if (request.ignoreHiddenFile && utils::file::FileUtils::is_hidden(fullName)) - return false; + uint64_t fileAge = utils::timeutils::getTimeMillis() - modifiedTime; + if (request.minAge > 0 && fileAge < request.minAge) +return false; + if (request.maxAge > 0 && fileAge > request.maxAge) +return false; -if (utils::file::FileUtils::access(fullName.c_str(), R_OK) != 0) - return false; + if (request.ignoreHiddenFile && utils::file::FileUtils::is_hidden(fullName)) +return false; -if (request.keepSourceFile == false && utils::file::FileUtils::access(fullName.c_str(), W_OK) != 0) - return false; + if (utils::file::FileUtils::access(fullName.c_str(), R_OK) != 0) +return false; -utils::Regex rgx(request.fileFilter); -if (!rgx.match(name)) { - return false; -} + if (request.keepSourceFile == false && utils::file::FileUtils::access(fullName.c_str(), W_OK) != 0) +return false; -metrics_->input_bytes_ += statbuf.st_size; -metrics_->accepted_files_++; -return true; + utils::Regex rgx(request.fileFilter); + if (!rgx.match(name)) { +return false; } - return false; + metrics_->input_bytes_ += file_size; + metrics_->accepted_files_++; + return true; } void GetFile::performListing(const GetFileRequest &request) { diff --git a/extensions/standard-processors/processors/PutFile.cpp b/extensions/standard-processors/processors/PutFile.cpp index 5503190..3765856 100644 --- a/extensions/standard-processors/processors/PutFile.cpp +++ b/extensions/standard-processors/processors/PutFile.cpp @@ -19,7 +19,6 @@ */ #include "PutFile.h" -#include #include #include #include @@ -149,9 +148,6 @@ void PutFile::onTrigger(core::ProcessContext *context, core::ProcessSession *ses logger_->log_debug("PutFile writing file %s into directory %s", filename, directory); - // If file exists, apply conflict resolution strategy - struct stat statResult; - if ((max_dest_files_ != -1) && utils::file::FileUtils::is_directory(directory.c_str())) { int64_t count = 0; @@ -171,7 +167,7 @@ void PutFile::onTrigger(core::ProcessContext *context, core::ProcessSession *ses } } - if (stat(destFile.c_str(), &statResult) == 0) { + if (utils::file::exists(destFile)) { logger_->log_warn("Destination file %s exists; applyi
[nifi-minifi-cpp] branch main updated (c8c4482 -> 95baec6)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from c8c4482 MINIFICPP-1487 Do not trigger the processor if the incoming queue has penalized flow files only add 95baec6 MINIFICPP-1523 - cxxopts: Add limits header No new revisions were added by this update. Summary of changes: libminifi/src/utils/StringUtils.cpp| 2 ++ thirdparty/cxxopts/include/cxxopts.hpp | 1 + 2 files changed, 3 insertions(+)
[nifi-minifi-cpp] branch main updated: MINIFICPP-1487 Do not trigger the processor if the incoming queue has penalized flow files only
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new c8c4482 MINIFICPP-1487 Do not trigger the processor if the incoming queue has penalized flow files only c8c4482 is described below commit c8c448240690e2398534c99b101e7d536a367cb7 Author: Ferenc Gerlits AuthorDate: Thu Feb 18 16:28:11 2021 +0100 MINIFICPP-1487 Do not trigger the processor if the incoming queue has penalized flow files only * make mutex_ mutable and mark const methods const * remove C-style (void) from function declarations * remove pointless comments * add a missing virtual-override pair on isYield() * add a missing 'explicit' * remove the declaration of unimplemented method getNextIncomingConnection() * Test that the processor detects correctly whether it has any work to do * Document the current behavior of Connection::poll(), to be fixed later * Use a custom queue which is aware of penalty expirations Signed-off-by: Arpad Boda This closes #1013 --- .../tests/unit/ProcessorTests.cpp | 63 libminifi/include/Connection.h | 6 +- libminifi/include/core/Processor.h | 51 +++ libminifi/include/utils/FlowFileQueue.h| 58 libminifi/src/Connection.cpp | 22 +-- libminifi/src/SchedulingAgent.cpp | 2 +- libminifi/src/core/Processor.cpp | 19 +-- libminifi/src/utils/FlowFileQueue.cpp | 103 + libminifi/test/unit/ConnectionTests.cpp| 88 +++ libminifi/test/unit/FlowFileQueueTests.cpp | 161 + libminifi/test/unit/MockClasses.h | 2 +- 11 files changed, 500 insertions(+), 75 deletions(-) diff --git a/extensions/standard-processors/tests/unit/ProcessorTests.cpp b/extensions/standard-processors/tests/unit/ProcessorTests.cpp index 5d34024..1ebcd39 100644 --- a/extensions/standard-processors/tests/unit/ProcessorTests.cpp +++ b/extensions/standard-processors/tests/unit/ProcessorTests.cpp @@ -614,3 +614,66 @@ TEST_CASE("TestRPGWithoutHostInvalidPort", "[TestRPG5]") { TEST_CASE("TestRPGValid", "[TestRPG6]") { testRPGBypass("", "8080", "8080", false); } + +TEST_CASE("A Processor detects correctly if it has incoming flow files it can process", "[isWorkAvailable]") { + LogTestController::getInstance().setDebug(); + + const auto repo = std::make_shared(); + const auto content_repo = std::make_shared(); + content_repo->initialize(std::make_shared()); + + const std::shared_ptr processor = std::make_shared("test_processor"); + const auto incoming_connection = std::make_shared(repo, content_repo, "incoming_connection"); + incoming_connection->addRelationship(core::Relationship{"success", ""}); + incoming_connection->setDestinationUUID(processor->getUUID()); + processor->addConnection(incoming_connection); + processor->initialize(); + + const auto processor_node = std::make_shared(processor); + const auto context = std::make_shared(processor_node, nullptr, repo, repo, content_repo); + const auto session_factory = std::make_shared(context); + const auto session = session_factory->createSession(); + + SECTION("Initially, the queue is empty, so there is no work available") { +REQUIRE_FALSE(processor->isWorkAvailable()); + } + + SECTION("When a non-penalized flow file is queued, there is work available") { +const auto flow_file = session->create(); +incoming_connection->put(flow_file); + +REQUIRE(processor->isWorkAvailable()); + } + + SECTION("When a penalized flow file is queued, there is no work available (until the penalty expires)") { +const auto flow_file = session->create(); +session->penalize(flow_file); +incoming_connection->put(flow_file); + +REQUIRE_FALSE(processor->isWorkAvailable()); + } + + SECTION("If there is both a penalized and a non-penalized flow file queued, there is work available") { +const auto normal_flow_file = session->create(); +incoming_connection->put(normal_flow_file); + +const auto penalized_flow_file = session->create(); +session->penalize(penalized_flow_file); +incoming_connection->put(penalized_flow_file); + +REQUIRE(processor->isWorkAvailable()); + } + + SECTION("When a penalized flow file is queued, there is work available after the penalty expires") { +processor->setPenalizationPeriodMsec(10); + +const auto flow_file = session->create(); +session-
[nifi-minifi-cpp] branch main updated: MINIFICPP-1512 update README
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 3c57dc0 MINIFICPP-1512 update README 3c57dc0 is described below commit 3c57dc065d59720a019bc5828acbfbc0b9c4d71a Author: Marton Szasz AuthorDate: Mon Mar 8 09:12:28 2021 +0100 MINIFICPP-1512 update README Signed-off-by: Arpad Boda This closes #1023 --- README.md | 93 --- 1 file changed, 54 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index cc8f421..4a4d4d7 100644 --- a/README.md +++ b/README.md @@ -27,10 +27,12 @@ MiNiFi is a child project effort of Apache NiFi. This repository is for a nativ - [System Requirements](#system-requirements) - [Bootstrapping](#bootstrapping) - [Building For Other Distros](#building-for-other-distros) - - [Cleaning](#cleaning) + - [Snapcraft](#snapcraft) + - [Installation](#installation) - [Configuring](#configuring) - [Running](#running) - [Deploying](#deploying) + - [Cleaning](#cleaning) - [Extensions](#extensions) - [Security](#security) - [Operations](#operations) @@ -73,7 +75,7 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension | Extension Set| Processors | CMAKE Flag | | - |:-| :-| | Archive Extensions| [ApplyTemplate](PROCESSORS.md#applytemplate)[CompressContent](PROCESSORS.md#compresscontent)[ManipulateArchive](PROCESSORS.md#manipulatearchive)[MergeContent](PROCESSORS.md#mergecontent)[FocusArchiveEntry](PROCESSORS.md#focusarchiveentry)[UnfocusArchiveEntry](PROCESSORS.md#unfocusarchiveentry) | -DBUILD_LIBARCHIVE=ON | -| AWS | [AWSCredentialsService](CONTROLLERS.md#awsCredentialsService)[PutS3Object](PROCESSORS.md#puts3object)[DeleteS3Object](PROCESSORS.md#deletes3object) | -DENABLE_AWS=ON | +| AWS | [AWSCredentialsService](CONTROLLERS.md#awscredentialsservice)[PutS3Object](PROCESSORS.md#puts3object)[DeleteS3Object](PROCESSORS.md#deletes3object) | -DENABLE_AWS=ON | | CivetWeb | [ListenHTTP](PROCESSORS.md#listenhttp) | -DDISABLE_CIVET=ON | | CURL | [InvokeHTTP](PROCESSORS.md#invokehttp) |-DDISABLE_CURL=ON | | GPS | GetGPS |-DENABLE_GPS=ON | @@ -88,7 +90,7 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension | SFTP | [FetchSFTP](PROCESSORS.md#fetchsftp)[ListSFTP](PROCESSORS.md#listsftp)[PutSFTP](PROCESSORS.md#putsftp) | -DENABLE_SFTP=ON | | SQL | ExecuteSQLPutSQLQueryDatabaseTable | -DENABLE_SQL=ON | | SQLite | [ExecuteSQL](PROCESSORS.md#executesql)[PutSQL](PROCESSORS.md#putsql) |-DENABLE_SQLITE=ON | -| Tensorflow | [TFApplyGraph](PROCESSORS.md#tfapplygraph)[TFConvertImageToTensor](PROCESSORS.md#tfconvertimagetotensor)[TFExtractTopLabels](PROCESSORS.md#tfextracttoplabels) |-DENABLE_TENSORFLOW=ON | +| Tensorflow | TFApplyGraphTFConvertImageToTensorTFExtractTopLabels | -DENABLE_TENSORFLOW=ON | | USB Camera | [GetUSBCamera](PROCESSORS.md#getusbcamera) | -DENABLE_USB_CAMERA=ON | | Windows Event Log (Windows only) | CollectorInitiatedSubscriptionConsumeWindowsEventLogTailEventLog | -DENABLE_WEL=ON | @@ -99,22 +101,18 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension * Build and usage currently only supports Windows, Linux and OS X environments. MiNiFi C++ can be built and run through the Windows Subsystem for Linux but we provide no support for this platform. * Provenance events generation is supported and are persisted using RocksDB. Volatile repositories can be used on systems without persistent storage. * If MiNiFi C++ is built with the OPC-UA extension enabled, it bundles [open62541](https://open62541.org/), which is available under the Mozilla Public License Version 2.0, a Category B license under [ASF 3rd party license policy](https://www.apache.org/legal/resolved.html#category-b). +* If MiNiFi C++ packaged on Windows, the resulting MSI may not be publicly redistributed under the Apache license, because it contains Microsoft redistributable DLLs, which fall under Category X of the [ASF 3rd party license policy](https://www.apache.org/legal/resolved.html#category-x). ## System Requirements ### To build Utilities -* CMake - * 3.10 or greater -* gcc - * 4.8.4 or greater -* g++ - * 4.8.4 or greater -* bison - * 3.0.x (3.2 has been shown to fail builds) -* flex - * 2.5 or greater +* CMake 3.11 or greater +* gcc 4.8.4 or greater +* g++ 4.8.4 or greater +* bison 3.0.x (3.2 has been shown to fail builds) +* flex 2.5 or greater # External Projects @@ -126,12 +124,11 @@ versions of LibreSSL, cURL, or zlib are used: * automake * libtool -**NOTE** if Lua support is enabled, then a C++ compiler with support
[nifi-minifi-cpp] 02/02: MINIFICPP-1517 - Fix integration test log cluttering
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 5b6d78d65dd1a97ee4e8e9049fd49b2ec3640e24 Author: Adam Hunyadi AuthorDate: Mon Mar 1 11:04:00 2021 +0100 MINIFICPP-1517 - Fix integration test log cluttering Signed-off-by: Arpad Boda This closes #1019 --- CMakeLists.txt | 13 - cmake/DockerConfig.cmake | 1 + docker/DockerBuild.sh| 2 ++ docker/Dockerfile| 5 +++-- 4 files changed, 18 insertions(+), 3 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index e43b850..68e1fc8 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -819,7 +819,18 @@ registerTest("encrypt-config/tests") include(BuildDocs) -include(DockerConfig) +if (ENABLE_PYTHON OR NOT DISABLE_SCRIPTING) + message(STATUS "Python and scripting extensions will disabled for the docker build as they produce many error messages on MiNiFi startup.") + set(ENABLE_PYTHON_CACHE ${ENABLE_PYTHON}) + set(DISABLE_SCRIPTING_CACHE ${DISABLE_SCRIPTING}) + set(ENABLE_PYTHON OFF) + set(DISABLE_SCRIPTING ON) # Implies DISABLE_PYTHON_SCRIPTING as well + include(DockerConfig) + set(ENABLE_PYTHON ${ENABLE_PYTHON_CACHE}) + set(DISABLE_SCRIPTING ${DISABLE_SCRIPTING_CACHE}) +else() + include(DockerConfig) +endif() if(NOT WIN32) # Create a custom build target that will run the linter. diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake index c935f27..5ae9b3b 100644 --- a/cmake/DockerConfig.cmake +++ b/cmake/DockerConfig.cmake @@ -53,6 +53,7 @@ add_custom_target( -c DISABLE_LZMA=${DISABLE_LZMA} -c DISABLE_BZIP2=${DISABLE_BZIP2} -c DISABLE_SCRIPTING=${DISABLE_SCRIPTING} +-c DISABLE_PYTHON_SCRIPTING=${DISABLE_PYTHON_SCRIPTING} -c DISABLE_CONTROLLER=${DISABLE_CONTROLLER} -c DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE} -c BUILD_NUMBER=${BUILD_NUMBER} diff --git a/docker/DockerBuild.sh b/docker/DockerBuild.sh index 51814cd..40e4701 100755 --- a/docker/DockerBuild.sh +++ b/docker/DockerBuild.sh @@ -58,6 +58,7 @@ DISABLE_LIBARCHIVE=${DISABLE_LIBARCHIVE:-} DISABLE_LZMA=${DISABLE_LZMA:-} DISABLE_BZIP2=${DISABLE_BZIP2:-} DISABLE_SCRIPTING=${DISABLE_SCRIPTING:-} +DISABLE_PYTHON_SCRIPTING=${DISABLE_PYTHON_SCRIPTING:-} DISABLE_CONTROLLER=${DISABLE_CONTROLLER:-} DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE:-} @@ -194,6 +195,7 @@ BUILD_ARGS="--build-arg UID=${UID_ARG} \ --build-arg DISABLE_LZMA=${DISABLE_LZMA} \ --build-arg DISABLE_BZIP2=${DISABLE_BZIP2} \ --build-arg DISABLE_SCRIPTING=${DISABLE_SCRIPTING} \ +--build-arg DISABLE_PYTHON_SCRIPTING=${DISABLE_PYTHON_SCRIPTING} \ --build-arg DISABLE_CONTROLLER=${DISABLE_CONTROLLER} " if [ -n "${DOCKER_BASE_IMAGE}" ]; then diff --git a/docker/Dockerfile b/docker/Dockerfile index ea6049f..20e8ce8 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -78,7 +78,7 @@ FROM build_deps AS build_minimal RUN cd ${MINIFI_BASE_DIR} \ && mkdir build \ && cd build \ - && cmake -DDISABLE_SCRIPTING=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_AWS=ON -DSKIP_TESTS=true -DCMAKE_BUILD_TYPE=MinSizeRel .. \ + && cmake -DENABLE_PYTHON=OFF -DDISABLE_SCRIPTING=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_AWS=ON -DSKIP_TESTS=true -DCMAKE_BUILD_TYPE=MinSizeRel .. \ && make -j$(nproc) package \ && tar -xzvf ${MINIFI_BASE_DIR}/build/nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz -C ${MINIFI_BASE_DIR} @@ -115,6 +115,7 @@ ARG DISABLE_LIBARCHIVE ARG DISABLE_LZMA ARG DISABLE_BZIP2 ARG DISABLE_SCRIPTING +ARG DISABLE_PYTHON_SCRIPTING ARG DISABLE_CONTROLLER RUN cd ${MINIFI_BASE_DIR} \ && mkdir build \ @@ -128,7 +129,7 @@ RUN cd ${MINIFI_BASE_DIR} \ -DDISABLE_CURL=${DISABLE_CURL} -DDISABLE_JEMALLOC=${DISABLE_JEMALLOC} -DDISABLE_CIVET=${DISABLE_CIVET} \ -DDISABLE_EXPRESSION_LANGUAGE=${DISABLE_EXPRESSION_LANGUAGE} -DDISABLE_ROCKSDB=${DISABLE_ROCKSDB} \ -DDISABLE_LIBARCHIVE=${DISABLE_LIBARCHIVE} -DDISABLE_LZMA=${DISABLE_LZMA} -DDISABLE_BZIP2=${DISABLE_BZIP2} \ --DDISABLE_SCRIPTING=${DISABLE_SCRIPTING} -DDISABLE_CONTROLLER=${DISABLE_CONTROLLER} -DCMAKE_BUILD_TYPE=Release .. \ +-DDISABLE_SCRIPTING=${DISABLE_SCRIPTING} -DDISABLE_PYTHON_SCRIPTING=${DDISABLE_PYTHON_SCRIPTING} -DDISABLE_CONTROLLER=${DISABLE_CONTROLLER} -DCMAKE_BUILD_TYPE=Release .. \ && make -j$(nproc) package \ && tar -xzvf ${MINIFI_BASE_DIR}/build/nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz -C ${MINIFI_BASE_DIR}
[nifi-minifi-cpp] branch main updated (287f184 -> 5b6d78d)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 287f184 MINIFICPP-1496 Remove ASAN from the list of extensions in bootstrap.sh new 6386078 MINIFICPP-1513 - Increase timeouts for behave tests involving http proxies, clean up logging new 5b6d78d MINIFICPP-1517 - Fix integration test log cluttering The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CMakeLists.txt | 13 +++- cmake/DockerConfig.cmake | 1 + docker/DockerBuild.sh | 2 ++ docker/DockerVerify.sh | 2 +- docker/Dockerfile | 5 +++-- .../integration/MiNiFi_integration_test_driver.py | 1 - docker/test/integration/environment.py | 1 - docker/test/integration/features/http.feature | 2 +- docker/test/integration/features/s3.feature| 6 +++--- .../integration/minifi/core/DockerTestCluster.py | 23 +++--- .../minifi/core/DockerTestDirectoryBindings.py | 1 - 11 files changed, 39 insertions(+), 18 deletions(-)
[nifi-minifi-cpp] 01/02: MINIFICPP-1513 - Increase timeouts for behave tests involving http proxies, clean up logging
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 63860782fdbb27def1d2d7fda8bd85ac91bbb74b Author: Adam Hunyadi AuthorDate: Fri Feb 26 15:34:49 2021 +0100 MINIFICPP-1513 - Increase timeouts for behave tests involving http proxies, clean up logging Signed-off-by: Arpad Boda This closes #1018 --- docker/DockerVerify.sh | 2 +- .../integration/MiNiFi_integration_test_driver.py | 1 - docker/test/integration/environment.py | 1 - docker/test/integration/features/http.feature | 2 +- docker/test/integration/features/s3.feature| 6 +++--- .../integration/minifi/core/DockerTestCluster.py | 23 +++--- .../minifi/core/DockerTestDirectoryBindings.py | 1 - 7 files changed, 21 insertions(+), 15 deletions(-) diff --git a/docker/DockerVerify.sh b/docker/DockerVerify.sh index dc897c2..afb4077 100755 --- a/docker/DockerVerify.sh +++ b/docker/DockerVerify.sh @@ -72,7 +72,7 @@ export PATH PYTHONPATH="${PYTHONPATH}:${docker_dir}/test/integration" export PYTHONPATH -BEHAVE_OPTS="-f pretty --logging-level INFO --no-capture" +BEHAVE_OPTS="-f pretty --logging-level INFO --logging-clear-handlers" cd "${docker_dir}/test/integration" exec diff --git a/docker/test/integration/MiNiFi_integration_test_driver.py b/docker/test/integration/MiNiFi_integration_test_driver.py index a1c63a0..ca5f04a 100644 --- a/docker/test/integration/MiNiFi_integration_test_driver.py +++ b/docker/test/integration/MiNiFi_integration_test_driver.py @@ -22,7 +22,6 @@ from minifi.validators.SingleFileOutputValidator import SingleFileOutputValidato class MiNiFi_integration_test(): def __init__(self, context): -logging.info("MiNiFi_integration_test init") self.test_id = str(uuid.uuid4()) self.clusters = {} diff --git a/docker/test/integration/environment.py b/docker/test/integration/environment.py index 68ddd42..4a3c12f 100644 --- a/docker/test/integration/environment.py +++ b/docker/test/integration/environment.py @@ -11,7 +11,6 @@ def raise_exception(exception): @fixture def test_driver_fixture(context): -logging.info("Integration test setup") context.test = MiNiFi_integration_test(context) yield context.test logging.info("Integration test teardown...") diff --git a/docker/test/integration/features/http.feature b/docker/test/integration/features/http.feature index 0652fd9..1f5e89d 100644 --- a/docker/test/integration/features/http.feature +++ b/docker/test/integration/features/http.feature @@ -40,7 +40,7 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And the "success" relationship of the ListenHTTP processor is connected to the PutFile When all instances start up -Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds +Then a flowfile with the content "test" is placed in the monitored directory in less than 120 seconds And no errors were generated on the "http-proxy" regarding "http://minifi-listen:8080/contentListener"; Scenario: A MiNiFi instance and transfers hashed data to another MiNiFi instance diff --git a/docker/test/integration/features/s3.feature b/docker/test/integration/features/s3.feature index 897bf18..03f0e11 100644 --- a/docker/test/integration/features/s3.feature +++ b/docker/test/integration/features/s3.feature @@ -40,7 +40,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And the http proxy server "http-proxy" is set up When all instances start up -Then a flowfile with the content "test" is placed in the monitored directory in less than 90 seconds +Then a flowfile with the content "test" is placed in the monitored directory in less than 150 seconds And the object on the "s3" s3 server is "LH_O#L|FDhttp://s3-server:9090/test_bucket/test_object_key"; @@ -102,7 +102,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server When all instances start up -Then a flowfile with the content "test" is placed in the monitored directory in less than 120 seconds +Then a flowfile with the content "test" is placed in the monitored directory in less than 150 seconds And the object bucket on the "s3" s3 server is empty And no errors were generated on the "http-proxy" regarding "http://s3-server:9090/test_bucket/test_object_key"; @@ -151,5 +151,5 @@ Feature: Sending data from MiNiFi-C++ to an AWS server When all instances start up -Then a flowfile with the content "test" is placed in the monitored directory in less than 120 s
svn commit: r46347 - /dev/nifi/nifi-minifi-cpp/0.9.0/ /release/nifi/nifi-minifi-cpp/0.9.0/
Author: aboda Date: Fri Feb 26 11:30:25 2021 New Revision: 46347 Log: MINIFICPP-1463 Promote NiFi MiNiFi C++ vote artifacts to release Added: release/nifi/nifi-minifi-cpp/0.9.0/ - copied from r46346, dev/nifi/nifi-minifi-cpp/0.9.0/ Removed: dev/nifi/nifi-minifi-cpp/0.9.0/
[nifi-minifi-cpp] branch main updated: MINIFICPP-1511 - Update documentation with Visual Studio 2019
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new a2e8068 MINIFICPP-1511 - Update documentation with Visual Studio 2019 a2e8068 is described below commit a2e80689889b41809c6110e47a87ee9c987622f1 Author: Adam Debreceni AuthorDate: Thu Feb 25 15:46:22 2021 +0100 MINIFICPP-1511 - Update documentation with Visual Studio 2019 Signed-off-by: Arpad Boda This closes #1016 --- README.md | 4 ++-- Windows.md | 8 +--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 28d39d1..cc8f421 100644 --- a/README.md +++ b/README.md @@ -192,8 +192,8 @@ On all distributions please use -DUSE_SHARED_LIBS=OFF to statically link zlib, l below. Windows - Build and Installation has been tested with Windows 10 using Visual Studio 2017. You can build - and create an MSI via the CPACK command. This requires the installation of the WiX + Build and Installation has been tested with Windows 10 using Visual Studio 2017 and Visual Studio 2019. + You can build and create an MSI via the CPACK command. This requires the installation of the WiX toolset (http://wixtoolset.org/). To do this, open up a prompt into your build directory and type 'cpack' . The CPACK command will automatically generate and provide you a path to the distributable msi file. See [Windows.md](Windows.md) for more details. diff --git a/Windows.md b/Windows.md index ab58abb..7e5223b 100644 --- a/Windows.md +++ b/Windows.md @@ -18,13 +18,13 @@ ## Requirements Apache NiFi MiNiFi C++ has been built on Window Server 2016, 2019, and Windows 10 operating systems. The project is CMake focused we suggest building via -Visual Studio 2017 or our `win_build_vs.bat` script. +Visual Studio 2017, Visual Studio 2019 or our `win_build_vs.bat` script. The project previously required OpenSSL to be installed. If you follow our build procedures, below, you will not need to install that dependency. ### Required software - - Visual Studio 2017 + - Visual Studio 2017 or Visual Studio 2019 - [CMake](https://cmake.org/download/) - [Git](https://git-scm.com/download/win) - (Optional) [WiX Toolset](https://wixtoolset.org/releases/) (only for building the MSI) @@ -68,12 +68,14 @@ After the build directory it will take optional parameters modifying the CMake c | /D | Builds RelWithDebInfo build instead of Release | | /DD | Builds Debug build instead of Release | | /CI | Sets STRICT_GSL_CHECKS to AUDIT | +| /2019 | Use the Visual Studio 2019 environment instead of the default Visual Studio 2017 | Examples: - 32-bit build with kafka, disabling tests, enabling MSI creation: `win_build_vs.bat build32 /T /K /P` - 64-bit build with JNI, with debug symbols: `win_build_vs.bat build64 /64 /J /D` -`win_build_vs.bat` requires a Visual Studio 2017 build environment to be set up. For 32-bit builds this can be achieved by using the `x86 Native Tools Command Prompt for VS 2017`, for 64-bit builds by using the `x64 Native Tools Command Prompt for VS 2017`. +`win_build_vs.bat` requires either a Visual Studio 2017 or a Visual Studio 2019 (if called with the `/2019` flag) build environment to be set up. With Visual Studio 2017 use the `x86 Native Tools Command Prompt for VS 2017` for 32-bit, or the `x64 Native Tools Command Prompt for VS 2017` for 64-bit builds. +For Visual Studio 2019 use the `x86 Native Tools Command Prompt for VS 2019`, or the `x64 Native Tools Command Prompt for VS 2019` for 32-bit and 64-bit builds respectively. ## Building directly with CMake
[nifi-minifi-cpp] branch main updated: MINIFICPP-1373 - Implement ConsumeKafka
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 6cfbf79 MINIFICPP-1373 - Implement ConsumeKafka 6cfbf79 is described below commit 6cfbf797a6c060a4781e0185a46c7cd9c4229384 Author: Adam Hunyadi AuthorDate: Fri Feb 26 10:56:02 2021 +0100 MINIFICPP-1373 - Implement ConsumeKafka Signed-off-by: Arpad Boda This closes #940 --- CMakeLists.txt | 2 +- PROCESSORS.md | 33 +- extensions/librdkafka/ConsumeKafka.cpp | 570 extensions/librdkafka/ConsumeKafka.h | 192 +++ .../{tests => docker_tests}/CMakeLists.txt | 2 +- extensions/librdkafka/rdkafka_utils.cpp| 121 + extensions/librdkafka/rdkafka_utils.h | 105 extensions/librdkafka/tests/CMakeLists.txt | 6 +- extensions/librdkafka/tests/ConsumeKafkaTests.cpp | 590 + libminifi/include/Connection.h | 4 +- libminifi/include/core/FlowFile.h | 5 +- libminifi/include/utils/GeneralUtils.h | 12 + libminifi/include/utils/ProcessorConfigUtils.h | 81 +++ libminifi/include/utils/StringUtils.h | 8 +- libminifi/src/Connection.cpp | 2 +- libminifi/src/core/FlowFile.cpp| 17 +- libminifi/src/utils/StringUtils.cpp| 17 +- libminifi/test/TestBase.cpp| 169 -- libminifi/test/TestBase.h | 51 +- libminifi/test/unit/StringUtilsTests.cpp | 10 + 20 files changed, 1908 insertions(+), 89 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index ff0024f..e43b850 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -444,7 +444,7 @@ option(ENABLE_LIBRDKAFKA "Enables the librdkafka extension." OFF) if (ENABLE_ALL OR ENABLE_LIBRDKAFKA) include(BundledLibRdKafka) use_bundled_librdkafka(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR}) - createExtension(RDKAFKA-EXTENSIONS "RDKAFKA EXTENSIONS" "This Enables librdkafka functionality including PublishKafka" "extensions/librdkafka" "extensions/librdkafka/tests") + createExtension(RDKAFKA-EXTENSIONS "RDKAFKA EXTENSIONS" "This Enables librdkafka functionality including PublishKafka" "extensions/librdkafka" "extensions/librdkafka/tests" "extensions/librdkafka/docker-tests") endif() ## Scripting extensions diff --git a/PROCESSORS.md b/PROCESSORS.md index c2cd82a..0ca385d 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -9,6 +9,7 @@ - [CapturePacket](#capturepacket) - [CaptureRTSPFrame](#capturertspframe) - [CompressContent](#compresscontent) +- [ConsumeKafka](#consumekafka) - [ConsumeMQTT](#consumemqtt) - [DeleteS3Object](#deletes3object) - [ExecuteProcess](#executeprocess) @@ -180,6 +181,37 @@ In the list below, the names of required properties appear in bold. Any other pr |failure|FlowFiles will be transferred to the failure relationship if they fail to compress/decompress| |success|FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed| +## ConsumeKafka + +### Description + +Consumes messages from Apache Kafka and transform them into MiNiFi FlowFiles. The application should make sure that the processor is triggered at regular intervals, even if no messages are expected, to serve any queued callbacks waiting to be called. Rebalancing can also only happen on trigger. +### Properties + +In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. + +| Name | Default Value | Allowable Values | Description | +| - | - | - | - | +|Duplicate Header Handling|Keep Latest|Comma-separated MergeKeep FirstKeep Latest|For headers to be added as attributes, this option specifies how to handle cases where multiple headers are present with the same key. For example in case of receiving these two headers: "Accept: text/html" and "Accept: application/xml" and we want to attach the value of "Accept" as a FlowFile attribute: - "Keep First" attaches: "Accept -> text/html" - "Keep Latest" attaches: "Accept - [...] +|**Group ID**|||A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.**Supports Expression Language: true**| +|Headers To Add As Attributes|||A comma separated list to match a
[nifi-minifi-cpp] branch main updated: MINIFICPP-1506 - Apply patch #3236 from librdkafka repository Upgrade librdkafka to 1.6.0
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new a975d23 MINIFICPP-1506 - Apply patch #3236 from librdkafka repository Upgrade librdkafka to 1.6.0 a975d23 is described below commit a975d232fa1e755b960383b7447581a36b34c3fa Author: Adam Debreceni AuthorDate: Mon Feb 22 14:38:43 2021 +0100 MINIFICPP-1506 - Apply patch #3236 from librdkafka repository Upgrade librdkafka to 1.6.0 Signed-off-by: Arpad Boda This closes #1014 --- cmake/BundledLibRdKafka.cmake| 7 +- thirdparty/librdkafka/high-cpu.patch | 185 +++ 2 files changed, 190 insertions(+), 2 deletions(-) diff --git a/cmake/BundledLibRdKafka.cmake b/cmake/BundledLibRdKafka.cmake index fe49890..bbaaf96 100644 --- a/cmake/BundledLibRdKafka.cmake +++ b/cmake/BundledLibRdKafka.cmake @@ -16,6 +16,8 @@ # under the License. function(use_bundled_librdkafka SOURCE_DIR BINARY_DIR) +set(PC "${Patch_EXECUTABLE}" -p1 -i "${SOURCE_DIR}/thirdparty/librdkafka/high-cpu.patch") + # Define byproducts if(WIN32) set(BYPRODUCT "lib/rdkafka.lib") @@ -41,10 +43,11 @@ function(use_bundled_librdkafka SOURCE_DIR BINARY_DIR) # Build project ExternalProject_Add( kafka-external -URL "https://github.com/edenhill/librdkafka/archive/v1.5.0.tar.gz"; -URL_HASH "SHA256=f7fee59fdbf1286ec23ef0b35b2dfb41031c8727c90ced6435b8cf576f23a656" +URL "https://github.com/edenhill/librdkafka/archive/v1.6.0.tar.gz"; +URL_HASH "SHA256=3130cbd391ef683dc9acf9f83fe82ff93b8730a1a34d0518e93c250929be9f6b" LIST_SEPARATOR % # This is needed for passing semicolon-separated lists CMAKE_ARGS ${LIBRDKAFKA_CMAKE_ARGS} +PATCH_COMMAND ${PC} BUILD_BYPRODUCTS "${BINARY_DIR}/thirdparty/librdkafka-install/${BYPRODUCT}" EXCLUDE_FROM_ALL TRUE ) diff --git a/thirdparty/librdkafka/high-cpu.patch b/thirdparty/librdkafka/high-cpu.patch new file mode 100644 index 000..78099ac --- /dev/null +++ b/thirdparty/librdkafka/high-cpu.patch @@ -0,0 +1,185 @@ +diff -rupN orig/src/rdkafka_broker.h patched/src/rdkafka_broker.h +--- orig/src/rdkafka_broker.h 2021-01-25 23:25:19.0 +0100 patched/src/rdkafka_broker.h 2021-02-22 16:44:21.665943600 +0100 +@@ -317,8 +317,17 @@ struct rd_kafka_broker_s { /* rd_kafka_b + rd_kafka_resp_err_t err; /**< Last error code */ + int cnt;/**< Number of identical errors */ + } rkb_last_err; ++ ++/** Recovery actions that need to be performed*/ ++uint32_t rkb_recovery_actions; + }; + ++/* ++ * Recovery actions bit flag ++ */ ++#define RKB_RECOVERY_ACTIONS_NONE 0x ++#define RKB_RECOVERY_ACTIONS_REINITIALIZE_WAKEUP_FD 0x0001 ++ + #define rd_kafka_broker_keep(rkb) rd_refcnt_add(&(rkb)->rkb_refcnt) + #define rd_kafka_broker_keep_fl(FUNC,LINE,RKB) \ + rd_refcnt_add_fl(FUNC, LINE, &(RKB)->rkb_refcnt) +@@ -450,6 +459,7 @@ rd_kafka_broker_controller_async (rd_kaf + + int rd_kafka_brokers_add0 (rd_kafka_t *rk, const char *brokerlist); + void rd_kafka_broker_set_state (rd_kafka_broker_t *rkb, int state); ++void rd_kafka_broker_set_recovery_action (rd_kafka_broker_t* rkbs, uint32_t action); + + void rd_kafka_broker_fail (rd_kafka_broker_t *rkb, +int level, rd_kafka_resp_err_t err, +diff -rupN orig/src/rdkafka_broker.c patched/src/rdkafka_broker.c +--- orig/src/rdkafka_broker.c 2021-01-25 23:25:19.0 +0100 patched/src/rdkafka_broker.c 2021-02-22 16:43:33.391455200 +0100 +@@ -272,6 +272,74 @@ int16_t rd_kafka_broker_ApiVersion_suppo + return maxver; + } + ++/** ++ * @brief Setup the wake fd for IO events ++ * ++ * @locality broker creation or reconnection ++ * @locks none ++ */ ++static void rd_kafka_broker_setup_queue_wakeup_fd(rd_kafka_broker_t* rkb) { ++int r; ++ ++/* ++ * Fd-based queue wake-ups using a non-blocking pipe. ++ * Writes are best effort, if the socket queue is full ++ * the write fails (silently) but this has no effect on latency ++ * since the POLLIN flag will already have been raised for fd. ++ */ ++rkb->rkb_wakeup_fd[0] = -1; ++rkb->rkb_wakeup_fd[1] = -1; ++if ((r = rd_pipe_nonblocking(rkb->rkb_wakeup_fd)) == -1) { ++rd_rkb_log(rkb, LOG_ERR, "WAKEUPFD", ++"Failed to setup broker queue wake-up fds: " ++"%s: disabling low-latency mode", ++rd_strerror(r)); ++ ++} ++else if (rkb->rkb_source == RD_KAFKA_INTERNAL) { ++/* nop: internal broker has no I
[nifi-minifi-cpp] branch main updated: MINIFICPP-1445 - Move docker integration tests to python behave
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new ae74606 MINIFICPP-1445 - Move docker integration tests to python behave ae74606 is described below commit ae746065319c89b6df23ef6a1bd6902306087cb8 Author: Adam Hunyadi AuthorDate: Tue Jan 12 11:56:03 2021 +0100 MINIFICPP-1445 - Move docker integration tests to python behave Signed-off-by: Arpad Boda This closes #995 --- docker/DockerVerify.sh | 25 +- docker/Dockerfile | 1 + .../integration/MiNiFi_integration_test_driver.py | 210 + docker/test/integration/README.md | 193 +--- docker/test/integration/__init__.py| 0 docker/test/integration/environment.py | 27 +++ .../features/file_system_operations.feature| 40 docker/test/integration/features/http.feature | 60 + docker/test/integration/features/https.feature | 23 ++ docker/test/integration/features/kafka.feature | 59 + docker/test/integration/features/s2s.feature | 45 docker/test/integration/features/s3.feature| 155 + docker/test/integration/minifi/__init__.py | 69 -- docker/test/integration/minifi/core/Cluster.py | 2 +- docker/test/integration/minifi/core/Connectable.py | 45 +--- .../integration/minifi/core/ControllerService.py | 1 + .../integration/minifi/core/DockerTestCluster.py | 192 .../minifi/core/DockerTestDirectoryBindings.py | 104 + .../integration/minifi/core/FileSystemObserver.py | 46 .../integration/minifi/core/OutputEventHandler.py | 15 +- docker/test/integration/minifi/core/Processor.py | 11 + .../integration/minifi/core/RemoteProcessGroup.py | 10 +- .../test/integration/minifi/core/SSL_cert_utils.py | 54 + .../minifi/core/SingleNodeDockerCluster.py | 153 ++-- .../minifi/processors/GenerateFlowFile.py | 3 +- .../test/integration/minifi/processors/GetFile.py | 2 +- .../integration/minifi/processors/HashContent.py | 8 + .../integration/minifi/processors/InvokeHTTP.py| 18 +- .../integration/minifi/processors/ListenHTTP.py| 4 +- .../minifi/processors/PublishKafkaSSL.py | 16 -- .../test/integration/minifi/processors/PutFile.py | 2 +- .../integration/minifi/processors/PutS3Object.py | 2 +- docker/test/integration/steps/steps.py | 256 + docker/test/integration/test_filesystem_ops.py | 51 docker/test/integration/test_filter_zero_file.py | 36 --- docker/test/integration/test_hash_content.py | 32 --- docker/test/integration/test_http.py | 57 - docker/test/integration/test_rdkafka.py| 98 docker/test/integration/test_s2s.py| 38 --- docker/test/integration/test_s3.py | 144 docker/test/integration/test_zero_file.py | 36 --- docker/test/test_https.py | 100 42 files changed, 1287 insertions(+), 1156 deletions(-) diff --git a/docker/DockerVerify.sh b/docker/DockerVerify.sh index f4efbc9..dc897c2 100755 --- a/docker/DockerVerify.sh +++ b/docker/DockerVerify.sh @@ -58,7 +58,8 @@ if ! command swig -version &> /dev/null; then fi pip install --upgrade \ -pytest \ +behave \ +pytimeparse \ docker \ PyYAML \ m2crypto \ @@ -71,4 +72,24 @@ export PATH PYTHONPATH="${PYTHONPATH}:${docker_dir}/test/integration" export PYTHONPATH -exec pytest -s -v "${docker_dir}"/test/integration +BEHAVE_OPTS="-f pretty --logging-level INFO --no-capture" + +cd "${docker_dir}/test/integration" +exec + behave $BEHAVE_OPTS "features/file_system_operations.feature" -n "Get and put operations run in a simple flow" && + behave $BEHAVE_OPTS "features/file_system_operations.feature" -n "PutFile does not overwrite a file that already exists" && + behave $BEHAVE_OPTS "features/s2s.feature" -n "A MiNiFi instance produces and transfers data to a NiFi instance via s2s" && + behave $BEHAVE_OPTS "features/s2s.feature" -n "Zero length files are transfered between via s2s if the \"drop empty\" connection property is false" && + behave $BEHAVE_OPTS "features/s2s.feature" -n "Zero length files are not transfered between via s2s if the \"drop empty\" connection property is true" && + behave $BEHAVE_OPTS "features/http.feature" -n "A MiNiFi instance transfe
[nifi-minifi-cpp] branch main updated: Introduce ccache in MacOS builds
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 281801f Introduce ccache in MacOS builds 281801f is described below commit 281801faff6c51037fdeabfa012ea5361baaa290 Author: Gabor Gyimesi AuthorDate: Mon Feb 15 10:00:18 2021 +0100 Introduce ccache in MacOS builds Signed-off-by: Arpad Boda This closes #1007 --- .github/workflows/ci.yml | 32 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f85f3bb..b894a30 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,14 +4,26 @@ jobs: macos_xcode_11_2_1: name: "macos-xcode11.2.1" runs-on: macos-10.15 -timeout-minutes: 60 +timeout-minutes: 90 +env: + CCACHE_BASEDIR: ${{ GITHUB.WORKSPACE }} + CCACHE_DIR: ${{ GITHUB.WORKSPACE }}/.ccache steps: - id: checkout uses: actions/checkout@v2 + - id: cache +uses: actions/cache@v2 +with: + path: ${{ env.CCACHE_DIR }} + key: macos-xcode11.2.1-ccache-${{github.ref}}-${{github.sha}} + restore-keys: | +macos-xcode11.2.1-ccache-${{github.ref}}- +macos-xcode11.2.1-ccache-refs/heads/main- - id: install_dependencies -run: brew install ossp-uuid boost flex openssl python lua@5.3 xz libssh2 +run: brew install ossp-uuid boost flex openssl python lua@5.3 xz libssh2 ccache - id: setup_env run: | + echo "PATH=/usr/lib/ccache:/usr/local/opt/ccache/bin:/usr/local/opt/ccache/libexec:$PATH" >> $GITHUB_ENV echo -e "127.0.0.1\t$HOSTNAME" | sudo tee -a /etc/hosts > /dev/null sudo xcode-select -switch /Applications/Xcode_11.2.1.app - id: build @@ -22,14 +34,26 @@ jobs: macos_xcode_12_0: name: "macos-xcode12.0" runs-on: macos-10.15 -timeout-minutes: 60 +timeout-minutes: 90 +env: + CCACHE_BASEDIR: ${{ GITHUB.WORKSPACE }} + CCACHE_DIR: ${{ GITHUB.WORKSPACE }}/.ccache steps: - id: checkout uses: actions/checkout@v2 + - id: cache +uses: actions/cache@v2 +with: + path: ${{ env.CCACHE_DIR }} + key: macos-xcode12.0-ccache-${{github.ref}}-${{github.sha}} + restore-keys: | +macos-xcode12.0-ccache-${{github.ref}}- +macos-xcode12.0-ccache-refs/heads/main- - id: install_dependencies -run: brew install ossp-uuid boost flex openssl python lua@5.3 xz libssh2 +run: brew install ossp-uuid boost flex openssl python lua@5.3 xz libssh2 ccache - id: setup_env run: | + echo "PATH=/usr/lib/ccache:/usr/local/opt/ccache/bin:/usr/local/opt/ccache/libexec:$PATH" >> $GITHUB_ENV echo -e "127.0.0.1\t$HOSTNAME" | sudo tee -a /etc/hosts > /dev/null sudo xcode-select -switch /Applications/Xcode_12.app - id: build
[nifi-minifi-cpp] branch main updated (53df4c2 -> 0d74aa4)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 53df4c2 MINIFICPP-1451: ThreadPoolAdjust test transiently failed on Windows CI add 0d74aa4 MINIFICPP-1498 Add libarchive to minimal docker image No new revisions were added by this update. Summary of changes: docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[nifi-minifi-cpp] 01/02: MINIFICPP-1491: Fixing build failures in Tensorflow extension
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 4518628bd502ce99f456283607fd419d385a Author: Martin Zink AuthorDate: Mon Feb 15 15:04:43 2021 +0100 MINIFICPP-1491: Fixing build failures in Tensorflow extension Signed-off-by: Arpad Boda This closes #1001 --- extensions/tensorflow/TFExtractTopLabels.cpp| 4 ++-- libminifi/test/tensorflow-tests/TensorFlowTests.cpp | 20 ++-- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/extensions/tensorflow/TFExtractTopLabels.cpp b/extensions/tensorflow/TFExtractTopLabels.cpp index 9bcb067..f0853c1 100644 --- a/extensions/tensorflow/TFExtractTopLabels.cpp +++ b/extensions/tensorflow/TFExtractTopLabels.cpp @@ -133,11 +133,11 @@ int64_t TFExtractTopLabels::LabelsReadCallback::process(const std::shared_ptrgetSize()) { + while (total_read < stream->size()) { auto read = stream->read(reinterpret_cast(&buf[0]), static_cast(buf_size)); for (auto i = 0; i < read; i++) { - if (buf[i] == '\n' || total_read + i == stream->getSize()) { + if (buf[i] == '\n' || total_read + i == stream->size()) { labels_->emplace_back(label.substr(0, label_size)); label_size = 0; } else { diff --git a/libminifi/test/tensorflow-tests/TensorFlowTests.cpp b/libminifi/test/tensorflow-tests/TensorFlowTests.cpp index 0334a2d..d4dc27c 100644 --- a/libminifi/test/tensorflow-tests/TensorFlowTests.cpp +++ b/libminifi/test/tensorflow-tests/TensorFlowTests.cpp @@ -45,8 +45,8 @@ TEST_CASE("TensorFlow: Apply Graph", "[tfApplyGraph]") { // NOLINT auto repo = std::make_shared(); // Define directory for input protocol buffers - std::string in_dir("/tmp/gt.XX"); - REQUIRE(testController.createTempDirectory(&in_dir[0]) != nullptr); + char in_dir_format[] = "/tmp/gt.XX"; + std::string in_dir = testController.createTempDirectory(in_dir_format); // Define input graph protocol buffer file std::string in_graph_file(in_dir); @@ -57,8 +57,8 @@ TEST_CASE("TensorFlow: Apply Graph", "[tfApplyGraph]") { // NOLINT in_tensor_file.append("/tensor.pb"); // Define directory for output protocol buffers - std::string out_dir("/tmp/gt.XX"); - REQUIRE(testController.createTempDirectory(&out_dir[0]) != nullptr); + char out_dir_format[] = "/tmp/gt.XX"; + std::string out_dir = testController.createTempDirectory(out_dir_format); // Define output tensor protocol buffer file std::string out_tensor_file(out_dir); @@ -184,16 +184,16 @@ TEST_CASE("TensorFlow: ConvertImageToTensor", "[tfConvertImageToTensor]") { // N auto repo = std::make_shared(); // Define directory for input protocol buffers - std::string in_dir("/tmp/gt.XX"); - REQUIRE(testController.createTempDirectory(&in_dir[0]) != nullptr); + char in_dir_format[] = "/tmp/gt.XX"; + std::string in_dir = testController.createTempDirectory(in_dir_format); // Define input tensor protocol buffer file std::string in_img_file(in_dir); in_img_file.append("/img"); // Define directory for output protocol buffers - std::string out_dir("/tmp/gt.XX"); - REQUIRE(testController.createTempDirectory(&out_dir[0]) != nullptr); + char out_dir_format[] = "/tmp/gt.XX"; + std::string out_dir = testController.createTempDirectory(out_dir_format); // Define output tensor protocol buffer file std::string out_tensor_file(out_dir); @@ -312,8 +312,8 @@ TEST_CASE("TensorFlow: Extract Top Labels", "[tfExtractTopLabels]") { // NOLINT auto repo = std::make_shared(); // Define directory for input protocol buffers - std::string in_dir("/tmp/gt.XX"); - REQUIRE(testController.createTempDirectory(&in_dir[0]) != nullptr); + char in_dir_format[] = "/tmp/gt.XX"; + std::string in_dir = testController.createTempDirectory(in_dir_format); // Define input labels file std::string in_labels_file(in_dir);
[nifi-minifi-cpp] 02/02: MINIFICPP-1451: ThreadPoolAdjust test transiently failed on Windows CI
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 53df4c282c58bc239fcf03d52aabc3c3b801ea69 Author: Martin Zink AuthorDate: Wed Feb 10 09:21:45 2021 +0100 MINIFICPP-1451: ThreadPoolAdjust test transiently failed on Windows CI Signed-off-by: Arpad Boda This closes #1000 --- extensions/http-curl/tests/ThreadPoolAdjust.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/extensions/http-curl/tests/ThreadPoolAdjust.cpp b/extensions/http-curl/tests/ThreadPoolAdjust.cpp index 1f55ef8..cb59aad 100644 --- a/extensions/http-curl/tests/ThreadPoolAdjust.cpp +++ b/extensions/http-curl/tests/ThreadPoolAdjust.cpp @@ -31,9 +31,11 @@ #include "processors/LogAttribute.h" #include "utils/IntegrationTestUtils.h" +constexpr uint64_t WAITTIME_MSECS = 5000; + class HttpTestHarness : public IntegrationBase { public: - HttpTestHarness() { + HttpTestHarness() : IntegrationBase(WAITTIME_MSECS) { char format[] = "/tmp/ssth.XX"; dir = testController.createTempDirectory(format); }
[nifi-minifi-cpp] branch main updated (a52faa5 -> 53df4c2)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from a52faa5 MINIFICPP-1485 Improve 'exclusive property' error message new 4518628 MINIFICPP-1491: Fixing build failures in Tensorflow extension new 53df4c2 MINIFICPP-1451: ThreadPoolAdjust test transiently failed on Windows CI The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: extensions/http-curl/tests/ThreadPoolAdjust.cpp | 4 +++- extensions/tensorflow/TFExtractTopLabels.cpp| 4 ++-- libminifi/test/tensorflow-tests/TensorFlowTests.cpp | 20 ++-- 3 files changed, 15 insertions(+), 13 deletions(-)
[nifi-minifi-cpp] branch main updated (1a41b44 -> f6d6ee7)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 1a41b44 Add build number parameter for alpine based images add f6d6ee7 MINIFICPP-1468 - Fix transient failures in mock S3 checks No new revisions were added by this update. Summary of changes: .../test/integration/minifi/core/DockerTestCluster.py | 7 ++- docker/test/integration/minifi/core/utils.py | 18 ++ 2 files changed, 24 insertions(+), 1 deletion(-) create mode 100644 docker/test/integration/minifi/core/utils.py
[nifi-minifi-cpp] branch main updated: Add build number parameter for alpine based images
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/main by this push: new 1a41b44 Add build number parameter for alpine based images 1a41b44 is described below commit 1a41b44495aa9b955b96779ff311239d8eea5f07 Author: Gabor Gyimesi AuthorDate: Tue Feb 9 10:58:09 2021 +0100 Add build number parameter for alpine based images Signed-off-by: Arpad Boda This closes #998 --- cmake/DockerConfig.cmake | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cmake/DockerConfig.cmake b/cmake/DockerConfig.cmake index 8db6c23..c935f27 100644 --- a/cmake/DockerConfig.cmake +++ b/cmake/DockerConfig.cmake @@ -55,6 +55,7 @@ add_custom_target( -c DISABLE_SCRIPTING=${DISABLE_SCRIPTING} -c DISABLE_CONTROLLER=${DISABLE_CONTROLLER} -c DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE} +-c BUILD_NUMBER=${BUILD_NUMBER} WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/) # Create minimal docker image @@ -66,6 +67,7 @@ add_custom_target( -v ${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH} -i minimal -c DOCKER_BASE_IMAGE=${DOCKER_BASE_IMAGE} +-c BUILD_NUMBER=${BUILD_NUMBER} WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/docker/) add_custom_target(
[nifi-minifi-cpp] 01/02: MINIFICPP-1460: ProcessSession should penalise flowfiles on rollback
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 44493c07f2b907463912f12065256071df4d21dd Author: Martin Zink AuthorDate: Thu Feb 4 09:26:01 2021 +0100 MINIFICPP-1460: ProcessSession should penalise flowfiles on rollback Signed-off-by: Arpad Boda This closes #991 --- libminifi/src/core/ProcessSession.cpp | 1 + libminifi/test/unit/ProcessSessionTests.cpp | 28 2 files changed, 29 insertions(+) diff --git a/libminifi/src/core/ProcessSession.cpp b/libminifi/src/core/ProcessSession.cpp index f8f885e..83ee2aa 100644 --- a/libminifi/src/core/ProcessSession.cpp +++ b/libminifi/src/core/ProcessSession.cpp @@ -791,6 +791,7 @@ void ProcessSession::rollback() { auto flowFile = it.second.modified; // restore flowFile to original state *flowFile = *it.second.snapshot; + penalize(flowFile); logger_->log_debug("ProcessSession rollback for %s, record %s, to connection %s", process_context_->getProcessorNode()->getName(), flowFile->getUUIDStr(), diff --git a/libminifi/test/unit/ProcessSessionTests.cpp b/libminifi/test/unit/ProcessSessionTests.cpp index 0847a4c..45b592c 100644 --- a/libminifi/test/unit/ProcessSessionTests.cpp +++ b/libminifi/test/unit/ProcessSessionTests.cpp @@ -74,3 +74,31 @@ TEST_CASE("ProcessSession::existsFlowFileInRelationship works", "[existsFlowFile REQUIRE(process_session.existsFlowFileInRelationship(Failure)); REQUIRE(process_session.existsFlowFileInRelationship(Success)); } + +TEST_CASE("ProcessSession::rollback penalizes affected flowfiles", "[rollback]") { + Fixture fixture; + core::ProcessSession &process_session = fixture.processSession(); + + const auto flow_file_1 = process_session.create(); + const auto flow_file_2 = process_session.create(); + const auto flow_file_3 = process_session.create(); + process_session.transfer(flow_file_1, Success); + process_session.transfer(flow_file_2, Success); + process_session.transfer(flow_file_3, Success); + process_session.commit(); + + auto next_flow_file_to_be_processed = process_session.get(); + REQUIRE(next_flow_file_to_be_processed == flow_file_1); + next_flow_file_to_be_processed = process_session.get(); + REQUIRE(next_flow_file_to_be_processed == flow_file_2); + REQUIRE_FALSE(flow_file_1->isPenalized()); + REQUIRE_FALSE(flow_file_2->isPenalized()); + REQUIRE_FALSE(flow_file_3->isPenalized()); + + process_session.rollback(); + REQUIRE(flow_file_1->isPenalized()); + REQUIRE(flow_file_2->isPenalized()); + REQUIRE_FALSE(flow_file_3->isPenalized()); + next_flow_file_to_be_processed = process_session.get(); + REQUIRE(next_flow_file_to_be_processed == flow_file_3); +}
[nifi-minifi-cpp] 02/02: MINIFICPP-1484 fix linter when not in a git tree
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit b69562aa6212cad9c0761d01936e016739a0bec6 Author: Marton Szasz AuthorDate: Mon Feb 8 17:44:45 2021 +0100 MINIFICPP-1484 fix linter when not in a git tree Signed-off-by: Arpad Boda This closes #997 --- thirdparty/google-styleguide/run_linter.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/thirdparty/google-styleguide/run_linter.sh b/thirdparty/google-styleguide/run_linter.sh index 9f79313..ca0a1ad 100755 --- a/thirdparty/google-styleguide/run_linter.sh +++ b/thirdparty/google-styleguide/run_linter.sh @@ -40,4 +40,5 @@ done HEADERS=`find $INCLUDE_DIRS -name '*.h' | sort | uniq | tr '\n' ' '` SOURCES=`find $SOURCE_DIRS -name '*.cpp' | sort | uniq | tr '\n' ' '` -python ${SCRIPT_DIR}/cpplint.py --linelength=200 ${HEADERS} ${SOURCES} +REPOSITORY="$(realpath --physical "$(dirname "$0")/../..")" +python ${SCRIPT_DIR}/cpplint.py --linelength=200 --repository="$REPOSITORY" ${HEADERS} ${SOURCES}
[nifi-minifi-cpp] branch main updated (85f05b0 -> b69562a)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 85f05b0 MINIFICPP-1455 - Fix FileStream error handling and reporting new 44493c0 MINIFICPP-1460: ProcessSession should penalise flowfiles on rollback new b69562a MINIFICPP-1484 fix linter when not in a git tree The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: libminifi/src/core/ProcessSession.cpp | 1 + libminifi/test/unit/ProcessSessionTests.cpp | 28 thirdparty/google-styleguide/run_linter.sh | 3 ++- 3 files changed, 31 insertions(+), 1 deletion(-)
[nifi-minifi-cpp] branch main updated (b977773 -> 85f05b0)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from b93 MINIFICPP-1434: Fix flaky CSite2SiteTests on macOS new 1e4c820 MINIFICPP-1477 - fix StringUtils::trim new 85f05b0 MINIFICPP-1455 - Fix FileStream error handling and reporting The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: extensions/expression-language/Expression.cpp | 5 +- libminifi/include/io/FileStream.h | 3 + libminifi/include/utils/StringUtils.h | 4 +- libminifi/include/utils/ValueParser.h | 2 +- libminifi/src/io/FileStream.cpp | 82 --- libminifi/test/TestBase.h | 4 ++ libminifi/test/unit/FileStreamTests.cpp | 81 ++ 7 files changed, 154 insertions(+), 27 deletions(-)
[nifi-minifi-cpp] 02/02: MINIFICPP-1455 - Fix FileStream error handling and reporting
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 85f05b08ed8f5d10b2745159d0f64475aa741230 Author: Martin Zink AuthorDate: Tue Jan 26 19:27:06 2021 +0100 MINIFICPP-1455 - Fix FileStream error handling and reporting Signed-off-by: Arpad Boda This closes #987 --- libminifi/include/io/FileStream.h | 3 ++ libminifi/src/io/FileStream.cpp | 82 + libminifi/test/TestBase.h | 4 ++ libminifi/test/unit/FileStreamTests.cpp | 81 4 files changed, 150 insertions(+), 20 deletions(-) diff --git a/libminifi/include/io/FileStream.h b/libminifi/include/io/FileStream.h index 54b8d59..3156a30 100644 --- a/libminifi/include/io/FileStream.h +++ b/libminifi/include/io/FileStream.h @@ -86,6 +86,9 @@ class FileStream : public io::BaseStream { */ int write(const uint8_t *value, int size) override; + private: + void seekToEndOfFile(const char* caller_error_msg); + std::mutex file_lock_; std::unique_ptr file_stream_; size_t offset_; diff --git a/libminifi/src/io/FileStream.cpp b/libminifi/src/io/FileStream.cpp index 700ec9f..20d4e6d 100644 --- a/libminifi/src/io/FileStream.cpp +++ b/libminifi/src/io/FileStream.cpp @@ -31,6 +31,19 @@ namespace nifi { namespace minifi { namespace io { +constexpr const char *FILE_OPENING_ERROR_MSG = "Error opening file: "; +constexpr const char *READ_ERROR_MSG = "Error reading from file: "; +constexpr const char *WRITE_ERROR_MSG = "Error writing to file: "; +constexpr const char *SEEK_ERROR_MSG = "Error seeking in file: "; +constexpr const char *INVALID_FILE_STREAM_ERROR_MSG = "invalid file stream"; +constexpr const char *TELLG_CALL_ERROR_MSG = "tellg call on file stream failed"; +constexpr const char *INVALID_BUFFER_ERROR_MSG = "invalid buffer"; +constexpr const char *FLUSH_CALL_ERROR_MSG = "flush call on file stream failed"; +constexpr const char *WRITE_CALL_ERROR_MSG = "write call on file stream failed"; +constexpr const char *EMPTY_MESSAGE_ERROR_MSG = "empty message"; +constexpr const char *SEEKG_CALL_ERROR_MSG = "seekg call on file stream failed"; +constexpr const char *SEEKP_CALL_ERROR_MSG = "seekp call on file stream failed"; + FileStream::FileStream(const std::string &path, bool append) : logger_(logging::LoggerFactory::getLogger()), path_(path), @@ -38,14 +51,22 @@ FileStream::FileStream(const std::string &path, bool append) file_stream_ = std::unique_ptr(new std::fstream()); if (append) { file_stream_->open(path.c_str(), std::fstream::in | std::fstream::out | std::fstream::app | std::fstream::binary); -file_stream_->seekg(0, file_stream_->end); -file_stream_->seekp(0, file_stream_->end); -std::streamoff len = file_stream_->tellg(); -length_ = len > 0 ? gsl::narrow(len) : 0; -seek(offset_); +if (file_stream_->is_open()) { + seekToEndOfFile(FILE_OPENING_ERROR_MSG); + auto len = file_stream_->tellg(); + if (len == std::streampos(-1)) +logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG << TELLG_CALL_ERROR_MSG; + length_ = len > 0 ? gsl::narrow(len) : 0; + seek(offset_); +} else { + logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG << path << " " << strerror(errno); +} } else { file_stream_->open(path.c_str(), std::fstream::out | std::fstream::binary); length_ = 0; +if (!file_stream_->is_open()) { + logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG << path << " " << strerror(errno); +} } } @@ -59,15 +80,16 @@ FileStream::FileStream(const std::string &path, uint32_t offset, bool write_enab } else { file_stream_->open(path.c_str(), std::fstream::in | std::fstream::binary); } - file_stream_->seekg(0, file_stream_->end); - file_stream_->seekp(0, file_stream_->end); - std::streamoff len = file_stream_->tellg(); - if (len > 0) { -length_ = gsl::narrow(len); + if (file_stream_->is_open()) { +seekToEndOfFile(FILE_OPENING_ERROR_MSG); +auto len = file_stream_->tellg(); +if (len == std::streampos(-1)) + logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG << TELLG_CALL_ERROR_MSG; +length_ = len > 0 ? gsl::narrow(len) : 0; +seek(offset_); } else { -length_ = 0; +logging::LOG_ERROR(logger_) << FILE_OPENING_ERROR_MSG << path << " " << strerror(errno); } - seek(offset); } void FileStream::close() { @@ -77,10 +99,16 @@ void FileStream::close() { void FileStream::seek(uint64_t offset) { std::lo
[nifi-minifi-cpp] 01/02: MINIFICPP-1477 - fix StringUtils::trim
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 1e4c820481cec72d6543fe26f40a5085184d91de Author: Marton Szasz AuthorDate: Thu Feb 4 17:04:20 2021 +0100 MINIFICPP-1477 - fix StringUtils::trim Signed-off-by: Arpad Boda This closes #990 --- extensions/expression-language/Expression.cpp | 5 + libminifi/include/utils/StringUtils.h | 4 ++-- libminifi/include/utils/ValueParser.h | 2 +- 3 files changed, 4 insertions(+), 7 deletions(-) diff --git a/extensions/expression-language/Expression.cpp b/extensions/expression-language/Expression.cpp index 545983f..5ce6f4d 100644 --- a/extensions/expression-language/Expression.cpp +++ b/extensions/expression-language/Expression.cpp @@ -840,10 +840,7 @@ Value expr_find(const std::vector &args) { #endif // EXPRESSION_LANGUAGE_USE_REGEX Value expr_trim(const std::vector &args) { - std::string result = args[0].asString(); - auto ws_front = std::find_if_not(result.begin(), result.end(), [](int c) {return std::isspace(c);}); - auto ws_back = std::find_if_not(result.rbegin(), result.rend(), [](int c) {return std::isspace(c);}).base(); - return (ws_back <= ws_front ? Value(std::string()) : Value(std::string(ws_front, ws_back))); + return Value{utils::StringUtils::trim(args[0].asString())}; } Value expr_append(const std::vector &args) { diff --git a/libminifi/include/utils/StringUtils.h b/libminifi/include/utils/StringUtils.h index 7dcfc49..54bab6c 100644 --- a/libminifi/include/utils/StringUtils.h +++ b/libminifi/include/utils/StringUtils.h @@ -99,7 +99,7 @@ class StringUtils { * @returns modified string */ static inline std::string trimLeft(std::string s) { -s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](char c) -> bool { return !isspace(c); })); +s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](unsigned char c) -> bool { return !isspace(c); })); return s; } @@ -110,7 +110,7 @@ class StringUtils { */ static inline std::string trimRight(std::string s) { -s.erase(std::find_if(s.rbegin(), s.rend(), [](char c) -> bool { return !isspace(c); }).base(), s.end()); +s.erase(std::find_if(s.rbegin(), s.rend(), [](unsigned char c) -> bool { return !isspace(c); }).base(), s.end()); return s; } diff --git a/libminifi/include/utils/ValueParser.h b/libminifi/include/utils/ValueParser.h index f7b11ba..ce4f0cc 100644 --- a/libminifi/include/utils/ValueParser.h +++ b/libminifi/include/utils/ValueParser.h @@ -148,7 +148,7 @@ class ValueParser { } void skipWhitespace() { -while (offset < str.length() && std::isspace(str[offset])) { +while (offset < str.length() && std::isspace(static_cast(str[offset]))) { ++offset; } }
[nifi-minifi-cpp] 03/05: MINIFICPP-1478 - Fix compilation errors of extensions
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit cb54b1051c67be6d9afe611d04b7448b4ddc831e Author: Gabor Gyimesi AuthorDate: Thu Feb 4 15:46:04 2021 +0100 MINIFICPP-1478 - Fix compilation errors of extensions Fix GPS extension compilation Fix SQLite compilation Fix compilation of OpenCV extension Fix openwsman compilation Fix bustache extension compilation Add extenstion dependencies to CI runners Extend timeout for 16.04-all build job Signed-off-by: Arpad Boda This closes #993 --- .github/workflows/ci.yml | 8 cmake/BundledBustache.cmake | 5 - extensions/bustache/ApplyTemplate.cpp | 2 +- extensions/gps/GetGPS.cpp | 2 +- extensions/opencv/FrameIO.h | 6 +++--- extensions/opencv/tests/CaptureRTSPFrameTest.cpp | 3 ++- .../openwsman/processors/SourceInitiatedSubscriptionListener.cpp | 4 ++-- extensions/sqlite/ExecuteSQL.cpp | 2 +- libminifi/test/bustache-tests/CMakeLists.txt | 1 - 9 files changed, 18 insertions(+), 15 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 211ec49..f85f3bb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -133,11 +133,11 @@ jobs: sudo apt install -y ccache libfl-dev libpcap-dev libboost-all-dev echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build -run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT .. && make -j4 VERBOSE=1 && make test ARGS="--timeout 300 -j2 --output-on-failure" +run: ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DENABLE_BUSTACHE=ON -DENABLE_SQLITE=ON -DENABLE_PCAP=ON -DSTRICT_GSL_CHECKS=AUDIT .. && make -j4 VERBOSE=1 && make test ARGS="--timeout 300 -j2 --output-on-failure" ubuntu_16_04_all: name: "ubuntu-16.04-all" runs-on: ubuntu-16.04 -timeout-minutes: 60 +timeout-minutes: 90 steps: - id: checkout uses: actions/checkout@v2 @@ -153,10 +153,10 @@ jobs: run: | sudo apt-add-repository -y "ppa:ubuntu-toolchain-r/test" sudo apt update - sudo apt install -y ccache openjdk-8-jdk maven + sudo apt install -y ccache openjdk-8-jdk maven libusb-1.0-0-dev libpng12-dev libgps-dev echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV - id: build -run: sudo mount tmpfs -t tmpfs /tmp && ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DENABLE_LIBRDKAFKA=ON -DENABLE_OPC=ON -DENABLE_SFTP=ON -DENABLE_MQTT=ON -DENABLE_COAP=ON -DENABLE_PYTHON=ON -DENABLE_SQL=ON -DENABLE_AWS=ON -DSTRICT_GSL_CHECKS=AUDIT .. && cmake --build . --parallel 4 && make test ARGS="--timeout 300 -j8 --output-on-failure" +run: sudo mount tmpfs -t tmpfs /tmp && ./bootstrap.sh -e -t && cd build && cmake -DUSE_SHARED_LIBS= -DENABLE_OPENWSMAN=ON -DENABLE_OPENCV=ON -DENABLE_MQTT=ON -DENABLE_GPS=ON -DENABLE_USB_CAMERA=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_OPC=ON -DENABLE_SFTP=ON -DENABLE_MQTT=ON -DENABLE_COAP=ON -DENABLE_PYTHON=ON -DENABLE_SQL=ON -DENABLE_AWS=ON -DSTRICT_GSL_CHECKS=AUDIT .. && cmake --build . --parallel 4 && make test ARGS="--timeout 300 -j8 --output-on-failure" debian: name: "debian" runs-on: ubuntu-18.04 diff --git a/cmake/BundledBustache.cmake b/cmake/BundledBustache.cmake index 4ee1804..97d9c4d 100644 --- a/cmake/BundledBustache.cmake +++ b/cmake/BundledBustache.cmake @@ -33,15 +33,18 @@ function(use_bundled_bustache SOURCE_DIR BINARY_DIR) "-DCMAKE_INSTALL_PREFIX=${BUSTACHE_BYPRODUCT_DIR}" "-DBUSTACHE_ENABLE_TESTING=OFF") +append_third_party_passthrough_args(BUSTACHE_CMAKE_ARGS "${BUSTACHE_CMAKE_ARGS}") + # Build project ExternalProject_Add( bustache-external -GIT "https://github.com/jamboree/bustache.git"; +GIT_REPOSITORY "https://github.com/jamboree/bustache.git"; GIT_TAG "42dee8ef9bbcae7e9a33500a116cfd9c314662d6" SOURCE_DIR "${BINARY_DIR}/thirdparty/bustache-src" CMAKE_ARGS ${BUSTACHE_CMAKE_ARGS} BUILD_BYPRODUCTS "${BUSTACHE_BYPRODUCT_DIR}/${BYPRODUCT}" EXCLUDE_FROM_ALL TRUE +L
[nifi-minifi-cpp] 02/05: MINIFICPP-1470 ByteArrayCallback: avoid OOB indexing during address calculation
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit a955854ab0723242079b25b254b85c4f1106affa Author: Marton Szasz AuthorDate: Tue Feb 2 16:22:39 2021 +0100 MINIFICPP-1470 ByteArrayCallback: avoid OOB indexing during address calculation Signed-off-by: Arpad Boda This closes #989 --- extensions/http-curl/client/HTTPCallback.h | 4 ++-- libminifi/include/utils/ByteArrayCallback.h | 24 2 files changed, 10 insertions(+), 18 deletions(-) diff --git a/extensions/http-curl/client/HTTPCallback.h b/extensions/http-curl/client/HTTPCallback.h index 827de10..921a8c5 100644 --- a/extensions/http-curl/client/HTTPCallback.h +++ b/extensions/http-curl/client/HTTPCallback.h @@ -120,7 +120,7 @@ class HttpStreamingCallback : public ByteInputCallBack { return ptr_ + relative_pos; } - const size_t getRemaining(size_t pos) override { + size_t getRemaining(size_t pos) override { logger_->log_trace("getRemaining(pos: %zu) called", pos); std::unique_lock lock(mutex_); @@ -128,7 +128,7 @@ class HttpStreamingCallback : public ByteInputCallBack { return total_bytes_loaded_ - pos; } - const size_t getBufferSize() override { + size_t getBufferSize() override { logger_->log_trace("getBufferSize() called"); std::unique_lock lock(mutex_); diff --git a/libminifi/include/utils/ByteArrayCallback.h b/libminifi/include/utils/ByteArrayCallback.h index cbcdb4c..84e3362 100644 --- a/libminifi/include/utils/ByteArrayCallback.h +++ b/libminifi/include/utils/ByteArrayCallback.h @@ -36,13 +36,10 @@ namespace utils { */ class ByteInputCallBack : public InputStreamCallback { public: - ByteInputCallBack() - : ptr(nullptr) { - } - - virtual ~ByteInputCallBack() = default; + ByteInputCallBack() = default; + ~ByteInputCallBack() override = default; - virtual int64_t process(const std::shared_ptr& stream) { + int64_t process(const std::shared_ptr& stream) override { stream->seek(0); if (stream->size() > 0) { @@ -51,34 +48,29 @@ class ByteInputCallBack : public InputStreamCallback { stream->read(reinterpret_cast(vec.data()), gsl::narrow(stream->size())); } -ptr = reinterpret_cast(&vec[0]); - return vec.size(); } - virtual void seek(size_t pos) { -ptr = &vec[pos]; - } + virtual void seek(size_t) { } virtual void write(std::string content) { vec.assign(content.begin(), content.end()); -ptr = &vec[0]; } virtual char *getBuffer(size_t pos) { -return &vec[pos]; +gsl_Expects(pos <= vec.size()); +return vec.data() + pos; } - virtual const size_t getRemaining(size_t pos) { + virtual size_t getRemaining(size_t pos) { return getBufferSize() - pos; } - virtual const size_t getBufferSize() { + virtual size_t getBufferSize() { return vec.size(); } private: - char *ptr; std::vector vec; };
[nifi-minifi-cpp] 01/05: MINIFICPP-1464 synchronize LICENSE and msi/LICENSE.txt
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 92575054631a31a1fd1f1f90d4ae8a2b23527e10 Author: Marton Szasz AuthorDate: Thu Jan 28 16:31:50 2021 +0100 MINIFICPP-1464 synchronize LICENSE and msi/LICENSE.txt Signed-off-by: Arpad Boda This closes #983 --- msi/LICENSE.txt | 1596 +++ 1 file changed, 1386 insertions(+), 210 deletions(-) diff --git a/msi/LICENSE.txt b/msi/LICENSE.txt index 4e0c84b..0f30eab 100644 --- a/msi/LICENSE.txt +++ b/msi/LICENSE.txt @@ -201,9 +201,15 @@ See the License for the specific language governing permissions and limitations under the License. -APACHE NIFI - MINIFI SUBCOMPONENTS: +APACHE NIFI - MINIFI C++ SUBCOMPONENTS: -The Apache NiFi - MiNiFi project contains subcomponents with separate copyright +The Apache NiFi - MiNiFi C++ project contains subcomponents licensed under the Apache License, Version 2.0: +This product bundles 'Simple-Windows-Posix-Semaphore' which is available under an ALv2 license +This project bundles 'mbedTLS' which is available under an ALv2 license +This project bundles 'RocksDB' which is available under an ALv2 license +This project bundles 'AWS SDK for C++' which is available under an ALv2 license + +The Apache NiFi - MiNiFi C++ project contains subcomponents with separate copyright notices and license terms. Your use of the source code for the these subcomponents is subject to the terms and conditions of the following licenses. @@ -238,68 +244,54 @@ This product bundles 'cpplint.py' which is available under a 3-Clause BSD Licen (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -This product bundles 'libuuid' which is available under a "3-clause BSD" license. +This product bundles 'spdlog' which is available under an MIT license. - Copyright (C) 1996, 1997 Theodore Ts'o. + Copyright (c) 2016 Alexander Dalshov. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - 1. Redistributions of source code must retain the above copyright - notice, and the entire permission notice in its entirety, - including the disclaimer of warranties. - 2. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - 3. The name of the author may not be used to endorse or promote - products derived from this software without specific prior - written permission. + Copyright (c) 2019 zvyagin.alexan...@.gmail.com - THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED - WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES - OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, ALL OF - WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE - LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT - OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR - BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF - LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE - USE OF THIS SOFTWARE, EVEN IF NOT ADVISED OF THE POSSIBILITY OF SUCH - DAMAGE. + Copyright(c) 2015 Ruslan Baratov. + + Copyright (c) 2015-2019 Gabi Melman. -This product bundles 'spdlog' which is available under an MIT license. + Copyright (c) 2016-2019 spdlog contributors. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR
[nifi-minifi-cpp] 04/05: MINIFICPP-1481 - Extract flow id from url without bucket id
This is an automated email from the ASF dual-hosted git repository. aboda pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 022afb415172618d48876eea0247e23e5bdf990c Author: Adam Debreceni AuthorDate: Mon Feb 8 14:31:07 2021 +0100 MINIFICPP-1481 - Extract flow id from url without bucket id Signed-off-by: Arpad Boda This closes #996 --- libminifi/src/core/FlowConfiguration.cpp | 13 + 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/libminifi/src/core/FlowConfiguration.cpp b/libminifi/src/core/FlowConfiguration.cpp index b007b52..232002d 100644 --- a/libminifi/src/core/FlowConfiguration.cpp +++ b/libminifi/src/core/FlowConfiguration.cpp @@ -76,14 +76,11 @@ std::unique_ptr FlowConfiguration::updateFromPayload(const s if (!url.empty() && payload != nullptr) { std::string flow_id, bucket_id; auto path_split = utils::StringUtils::split(url, "/"); -// Registry API docs: nifi.apache.org/docs/nifi-registry-docs/rest-api/index.html -// GET /buckets/{bucketId}/flows/{flowId}: Gets a flow -const auto bucket_token_found = std::find(path_split.cbegin(), path_split.cend(), "buckets"); -if (bucket_token_found != path_split.cend() && std::next(bucket_token_found) != path_split.cend()) { - bucket_id = *std::next(bucket_token_found); - const auto flows_token_found = std::find(std::next(bucket_token_found, 2), path_split.cend(), "flows"); - if (flows_token_found != path_split.cend() && std::next(flows_token_found) != path_split.cend()) { -flow_id = *std::next(flows_token_found); +for (auto it = path_split.cbegin(); it != path_split.cend(); ++it) { + if (*it == "flows" && std::next(it) != path_split.cend()) { +flow_id = *++it; + } else if (*it == "buckets" && std::next(it) != path_split.cend()) { +bucket_id = *++it; } } flow_version_->setFlowVersion(url, bucket_id, flow_id);